Execution Guided Line-by-Line Code Generation
Authors: Boaz Lavon
Shahar Katz
Lior Wolf
Venue: NeurIPS 2025 poster ()
Keywords: Program Induction
Problem Solving
Reasoning
Natural Language Processing
Primary area: deep_learning
TL;DR: We introduce EG-CFG, a code generation method that incorporates real-time execution feedback during LLM inference to guide line-by-line generation, significantly improving code accuracy.

Abstract: We present a novel approach to neural code generation that incorporates real-time execution signals into the language model generation process. While large language models (LLMs) have demonstrated impressive code generation capabilities, they typically do not utilize execution feedback during inference, a critical signal that human programmers regularly leverage. Our method, Execution-Guided Classifier-Free Guidance EG-CFG, dynamically incorporates execution signals as the model generates code, providing line-by-line feedback that guides the generation process toward executable solutions.
EG-CFG employs a multi-stage process: first, we conduct beam search to sample candidate program completions for each line; second, we extract execution signals by executing these candidates against test cases; and finally, we incorporate these signals into the prompt during generation. By maintaining consistent signals across tokens within the same line and refreshing signals at line boundaries, our approach provides coherent guidance while preserving syntactic structure. Moreover, the method naturally supports native parallelism at the task level in which multiple agents operate in parallel, exploring diverse reasoning paths and collectively generating a broad set of candidate solutions.
Our experiments across diverse coding tasks demonstrate that EG-CFG significantly improves code generation performance compared to standard approaches, achieving state-of-the-art results across various levels of complexity, from foundational problems to challenging competitive programming and data science tasks.

Execution Guided Line-by-Line Code Generation
Lior Wolf Blavatnik School of Computer Science and AI, Tel Aviv University (boazlavon@mail, shaharkatz3@mail, wolf@cs}, tau ac.il il
Boaz Lavon
Shahar Katz
Abstract
We present a novel approach to neural code generation that incorporates real-time execution signals into the language model generation process. While large lan- guage models (LLMs) have demonstrated impressive code generation capabilities, they typically do utilize execution feedback during inference, critical sig- nal that human programmers regularly leverage. Our method, Execution-Guiid Classifier-Free Guidance (EG-CFG), dynamically incorporates execution signals the model generates code, providing line-by-line feedback that guides the genera- tion process toward executable solutions. EG-CFG employs multi-stage process: first, we conduct beam search to sample candidate program completions for each line; second, we extract execution signals by executing these candidates against test cases; and finally, we incorporate these signals into the prompt during generation. By maintaining consistent signals across tokens within the same line and refreshing signals at line boundaries, our approach provides coherent guidance while preserv- ing syntactic structure. Moreover, the method naturally supports native parallelism at the task level in which multiple agents operate in parallel, exploring diverse reasoning paths and collectively generating broad set candidate solutions. Our experiments across diverse coding tasks demonstrate that EG-CFF significantly improves code generation performance compared to standard approaches, achiev- ing wtat-offthhe-at results across various levels of complexity, from foundational problems to challenging competitive programming and data science tasks. Our code is available at: https://gitii: tgggttttub..... cow/boazlavon/ aellaavoooeeeeeeee eg_
Introduction
Large language models (LLMs) have recently demonstrated remarkable code generation capabilities, significantly advancing performance in tasks such as general programming problems [I 2 3] competitive coding challenges (4) and real-world software engineering tasks [5). However, current LLM-based code generation methods primarily rely on pattern recognition derived from static representations of code rather than explicitly modeling code execution at runtime 33.. Consequently, the generated programs often appear to be correct superficially, but fail to execute correctly on actual inputs, reflecting critical gap between learned syntactic patterns and genuine executability.
7 g or self-debugging strategies 10] Recent approaches adopt multi-agent agentic workflows, explicitly employing iterative 14. However, these methods typically operate in discrete cycles: generating complete candidate solutions, executing them, and then using feedback from failures to guide subsequent attempts. Such approaches do not continuously integrate execution signals during inference, thus limiting their ability to dynamically adjust toward runtime correctness at the token level.
-rate-of-the-ar approaches typically use iterative refinement 6
refinement and collaborative feedback mechanisms [2] I3
In contrast, human programmers frequently execute incomplete code fragments quickly detect errors, assess progress, and iteratively refine their implementations based on concrete runtime outcomes, while exploring multiple candidate implementations and planning at varying levels of This iterative, real-time refinement process explicitly
detail before finalizing solutions 15 16.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

MEPP MISPPFFTT
80.6
87:2
00.5
$4.0
68.5
65.0
09.6
57.6
GP GP Bonelin LLM
GPT Basl Colattaationn
GPTT- Dhteogging
GPT Mataconiy
MapCCootes Codeer
OPT LPW
Chaude Semesto3 5.5 Baasline
Chacsl Sanoecd S
Pros
08.133
D8I-43 ManCader
088-3 McDeloogee
DS-13 LPW
DS-V9 EG-CPO
Buselow
Figure 1: MBPP & MBPP-ET performance. CG-CFG (DeepSeck-Ve V3) sets a new state-of-the-ar results.
HumanEvval Humanntvva-ET
91.5
94.5
97.0
88.0
04.5
70.2
81.7
67.7
70.1
70.1
51.0
Sesaline LLM
63P14
GPT4 Debutainng
GPI MebeGPy
GPT MapCadar
GP114
LPW
LLMMA (D0)
Coede Scmoal Quatthyfnn Flow
O8-Cinan VDodde MaDensgger
DS-99 Besedne LLM
D8:33 MapiCader
DS-V3 MGDelogaar
DS-ig
BS-CPG
Figure 2: HvaanEval & HvmanExal-ET performance. EG-CFG (DeepSeek- V3) matches the state-of- the-art on HumanEval and sets a new statc-of-the-att on HumanEval-Eii
grounds coding decisions observed execution behavior rather than relying purely on syntactic or structural reasoning [17].
Inspired by these human coding practices and by LLM exploration techniques 19 20
our proposed approach generates dynamic execution signals by explicitly sampling multiple can- didate continuations at varied completion horizons, systematically adjusting decoding temperature to encourage exploration of different reasoning paths. Executing these diverse candidates yields rich execution- -basu feedback, explicitly mirroring human iterative refinement and exploratory problem-solving processes.
Unlike methods that provide explicit correctness indicators, such as scalar pass/fail or ranking signals 4 [22), explicit verbal critiques and structured reflections on execution failures (9 20
our method provides the raw execution outcome as a soft guidance signal. This approach allows the model to autonomously interpret and integrate minimally processed feedback into its generation process, bridging significant gap between explicit cittenaallysuperriiie
reinforcement, which the model is explicitly told what runs were successful [9) and implicit self- verfiverrii,ation. where the model autonomously assesses the correctness of its own reasoning [10].
To incorporate the execution-based signal, our method utilizes Classifier-Free Guidance (CFG) 23 conditioning token-level generation decisions on the runtime outcome obtained by executing candidate code completions during inference. This approach guides the model toward solutions that are both syntactically plausible and executable, substantially improving correctness.
As depicted in Figure our Execution-Guided Classifier-Free Guidance (EG-CFG) approach achieves state-of-He seeee-ffttherrreee performance the MBPP and MBPP-ET benchmarks 3 significantly outperforming existing methods. Using the open-source DeepSeek-V33-322 model 24 EG-CFG attains 96.6% accuracy on MBPP and 73.0% on MBPP-ET, surpassing previous leading approaches such as QualityFlow [11] (94.2%), MetaGPT 25 (87.7%), and LPW [26] (84.8% on MBPP, 65.8% on MBPP. ET), all of which utilized leading closed-source models. On the HumanEval benchmark (Figure 2). EG-CFG achieves state-of-the-ar accuracy of 99.4%, matching LDB [27] and establishes new state-of-the-an on the HumanEval-ET benchmark, reaching 89.02% accuracy and surpassing LPW's 84.8% accuracy achieved with GPT-4o. Furthermore, EG-CFG establishes new state-of- ofate-of-the-an on DS- 1000 (Figure 3), a domain-specific benchmark focused on challenging data science problems, achieving 69.9% accuracy. Lastly, EG-CFG achieves a new state-of-the-a on the CodeContests benchmark (Figure 4) with 60.6% accuracy using DeepSeek-V3-ne V3-0324, demonstrating its effectiveness on challenging competitive programming problems.

---

DS-10000
68.0
57.1
64.3
45.5
38.9
OPT.3 Tumo Beshooiiiing
DPni3. Tumo Seit Detwogging
GPr-3 Tumu Seltv
Chnecd Gwend-1228 102--2288-ee Beosine LLM
DS-C Codert Beesdin LLM
Glouda 8-5 Saanel Basesin LLM
DPP 4.40 Blsasion
GP 14 Riowdie LLM
CPT-4 coN
DSA15 Beselow LLM
BS-2PG
Figure 3: DS-1000 performance. EG-CFG (DeepSeek- V3) achieves new state-of oate--o-the-arn ahe-art, surpassing GPT-4.
Conteess
66.6
41A
347
28.5
29.1
29.3
12.7
18.4
611
GPE-s
Planning
1.8.
Ansiggicall
Selt- Planning
GPTY
Anaeppical
GPT-3 P.g.5 Turbo
MayCretur
GPF-3a
Cadesim
May Credaa McTClder
CPT4 MapClater
GPT4
GPI-4 -44 L83
GP7-40 LPW
DS-13 Bowelnn LLM
08-V3 MapCadee
CodeeSin
EC-ORG
Figure 4: CodeContests performance. EG-CFG (DeepSeek- V3) sets a new State-of-theande outper- forming GPT-4 and GPT-4o methods.
The superior performance of EG-CFG MBPP-ET and HumanEval-ET highlights that our method not only generates accurate code but also significantly enhances robustness and reliability under complex and extended test scenarios. Summarizing, for dynamically generating code while executing fragments of the code and using the execution traces guide the generation process, (ii) introducing native parallelism the task level that is not achievable the sequential iterative refinement methods, (iii) using CFG in order to generate code that is conditioned the execution feedback, and (iv) obtaining the MBPP, MBPP-ET, HumanEval-Ei! DS- 1000 and CodeContests benchmarks using open-source models, outperforming previous methods that are based the leading closed-source models.
main contributions are: (i) framework
tate-of-the-ar results
Related Work
Program synthesis has long served both as a means to evaluate LLMs' capabilities [ 28 as key goal of the research community to automate code generation 229 30 Modern LLMs are typically model generates
assessed via automated benchmarks: given coding problem description, set of test cases (unit-test). In this work, focus six code generation benchmarks: MBPP 33, HumanEval [1] DS- 1000 31 and CodeContests 4, along with the extended variants MBPP-ET and HumanEval-BB ET 32 MBPP combines crowd-sourced tasks and math- based problems [33]. HumanEval includes hand-written Python challenges, DS-1000 presents challenging data science problems involving libraries like Pandas and NumPy, and CodeContests features competitive programming problems requiring advanced algorithmic reasoning.
code that is then validated
While zero-shot prompting, meaning directly querying an LLM on task, is one evaluation strategy, few-shot prompting [44, which provides a small number of input-output examples, more widely adopted. Other methods, such as Soinnoo--Touugt (CoT) K5,, leverage the model's autoregressive nature to iteratively solve such tasks by decomposing them into sub-tasks. Similar approaches, such Tree-of-Thoughh ThoThhss (21) extend CoT by exploring multiple candidate reasoning paths at higher level of granularity.
With the growing ability of LLMs to utilize external signals and tools [36], LLMs have been augmented with feedback mechanisms to improve code generation. Approaches such Reflexion [9. Program-aided Language Models (PAL) [6]. and ReAct [7] iteratively update prompts based on external tools' outputs or intermediate results. In particular, Self-Debugging [10] demonstrates how L.LMs can leverage debugging tools: the model generates code, tests it available unit-test examples, and refines its output using the debugging feedhack. Another related approach, L.W (26. employs structured two-phase process involving initial high-level plan generation followed by

---

iterative refinement and debugging guided by execution feedback. The Aulti-Granularity Debugger (MGDebugger) [37] also uses the LLM to simulate the execution of generated Python programs, employing the simulated trace as additional feedback for code refinement.
While some approaches use a single LLM solve coding problems, agentic methods employ multiple LLM instances to tackle a single task. Works such as MetaGPT [25] introduce a framework in which a given task is split into multiple procedures, each assigned to a dedicated LLM agent. AgentCoder [2] extends these multi-agent frameworks by employing both a test-designer agent and an executor agent, effectively integrating the Self-Debuggin paradigm [10] into an agentic workflow and demonstrating the benefits of execution feedback. Current agentic workflows, such as MapCoder [13] and LPW (26). typically rely on discrete cycles of sequential agent interactions or distinct refinement phases, inherently limiting their ability to fully leverage the power of parallelism within a single task. contrast, EG-CFG breaks this barrier by introducing native parallelism the task level. Multiple agents run concurrently on the same task, each with a different configuration, simultaneously exploring diverse reasoning paths without the constraints of sequential cycles, fully leveraging the power of parallelism.
Although code generation methods have advanced dramatically, they all rely on naive decoding methods such as temperature or top-p sampling [18]. In particular, execution-feedback techniques generate complete solution, execute it, and then refine the code. For example, the model prompted to produce a function, its implementation is tested against unit tests, and the resulting feedback is appended to the original prompt. This process repeats iteratively until a correct solution emerges. This approach contrasts with recent advances in LLM inference techniques, especially guidance methods such as Classifier Guidance [8 and Classifier-Fres Guidance (CFG) 23 39]. These methods condition generation on external constraints or classifier signals, directing the model to
sample tokens from multiple contrastive distributions. Although CFG has shown strong performance, its guidance signals remain static, predefined and fixed throughout the sampling process [40]
As far as we can ascertain, guidance methods such as CFG have yet to be applied at scale to dynamic, execution-driven reasoning. This work bridges this gap by demonstrating that single open- moen-source model, augmented with execution feedback and a novel token- sampligg strategy, outperforms the state-of-the-an across multiple widely adopted coding benchmarks at different complexity levels from foundational problems to challenging competitive programming and data sceince tasks. Our contribution differs from prior work in multiple key aspects, including:
1. Prior approaches generate entire code blocks (or sub-blocks) and use the execution feedback of the entire block; by contrast, our approach gradually generates the solution line-by-line by sampling and executing multiple candidate continuation programs at each step.
2. Instead of explicitly relying on unit-test pass/fail signals or direct reflections on correctness, our method incrementally constructs code fragments and leverages their execution traces as an implicit feedback signal, guiding the model without explicit external supervision.
3. Our method relies on a single prompting scheme where multiple agents perform the same task and differ only in their parameter configurations. Each agent explores diverse reasoning paths and collectively generating a broad set of candidate solutions. This enables level of parallelism that unattainable in methods where agents communicate sequentially.
4. We replace naive decoding with CFG, using innovative trace-inspired prompts for dual- distribution interpolation sampling.
3 Method
We present Reccution-Guided Classifier-Free Guidance (EG-CFG), novel inference- decoding method for neural code generation that explicitly integrates dynamic execution signals into the autoregressive generation process. A programming task r is represented by three components: (Pusk: T, Jnmme)
(1)
where Prask is a textual definition of the programming task, T = (1))7 is the set of test cases, and fname is the target Python function name. The goal is to generate executable Python code w= (wg. wi...., [1 that solves the task correctly. formally satisfying:
Execute(w* ,t)) success, Vt, ET.
We consider two distinct instruction templates po: the standard DeepSeek- Coder 3-shot prompt Al consisting of concise instructions accompanied by short illustrative examples, and an alternative

---

prompt explicitly designed to encourage the generation of step-by-step solutions with more atomic logic. See Appendix for the example prompts used.
We build the instruction prompt by incorporating the task components into the prompt template:
Pinst BUILDINTTRCCTOOPPROPPPPMM
At each step i, given a previously generated token sequence w<i, the LLM M assigns a probability distribution M (w; Pinst,we..), which is conditioned on the instruction prompt Pinst- This generation proceeds iteratively until reaching a maximum token length Naxx or an end-of-code token. We build
the instruction prompt by injecting the instruction task to the prompt template. We first run the model to produce an initial output sequence Ppre based on Pinst
u pre arg max M (w; p
Ppre Piists Wpree
pre [
(4)
The generated sequence Ppre contains reasoning tokens that precede the final executable solution. We identify the beginning of this solution block at index Esluuiion by locating special start-of-code token, which is defined in po- We then define the prefix of all preceding tokens, Ppre-sol. as:
Pere-sol
Pere [
sollution
(5)
Additionally, define an index idyn that marks the end of the final few-shot example within the instruction prompt Pina,, which is used to inject future signals into the prompt.
The prompt Psol that we pass to the LLM is constructed autoregressively by aggregating executable code tokens into Ppre-sol as formalized later in [quatton 14 of |section 3.3 This prompt contains partial solutions that progressively updated untul the solution is formed.
3.1 Dynamic Execution Feedback
Our dynamic execution feedback explicitly generates multiple candidate continuations based partially completed solution. Specifically, given Psol we generate a set of candidate continuations using beam-search decoding. Each candidate explicitly extends the current solution by d additional lines of code, capturing meaningful variations in potential solutions and providing a granular basis for execution-based guidance. Formally, given parameters specifying the number of candidates s, completion horizon d, and sampling temperature I, the beam search sampling is performed to obtain candidates.
w/, M (w:l Psol, "6y' () until CountLines(c)) d, Cj (w,, w w.
j i..... (6)
Each candidate el is generated autoregressively with a stopping condition triggered after d newline characters, representing a plausible continuation of the next d lines of code. These candidates form the basis for generating detailed execution signals used to guide subsequent inference steps.
To handle potentially invalid candidate continuations, we extract executable components via Abstract Syntax Tree (AST) parsing. Formally, for each candidate c², we apply the executable extraction function:
Executable Extraction
= Extraattxxeeutabbe(()) j j=
This extraction function iteratively attempts to parse each candidate e as follows: (1) attempt parsing e (2) if unsuccessful, append a Python pass statement to the last line and retry; (3) if still unsuccessful, iteratively remove the last line and retry. This ensures minimal modifications for syntactic validity. After extraction, we apply uniqueness filtering to remove duplicates:
= Unique (()')]-))
(8)
This AST-based verification explicitly ensures that execution guidance relies solely on syntactically valid and executable code.
Execution Feedback and Trace For each unique executable candidate e E C. we explicitly
execute it against all provided test cases T (tm |77 and record the resulting execution feedback:
e
ExtraatExeenttoooeeedbbae ,tm), Yai E C, fm T.
Our approach is agnostic to the precise structure of the execution feedback ef. In our implementa- tion, execution feedback specifically takes the form of execution traces: a structured representation of program's runtime behavior, capturing detailed step-by-step information during execution.
5

---

Specifically, we use custom debugger for structured execution traces. A trace eim is defined as sequence of N runtime events, Ek, resulting from the execution of a program e with imput tm:
e [=x]k=1 Na.m EK (Ex: les Uk: Tke (kk
(10)
Each event ER is a tuple containing the event type Es E {call, line, return, exception). the source line number lg. mappings from variable names to their values (vx) and types (Tk)), the return value Tk. and any exception details zg. This structured representation provides comprehensive insight into the correctness and behavior of executed code, serving as precise basis for dynamic feedback in our inference framework.
Dynamic Signal Aggregation The dynamic execution feedback signal is the concatenation of fixed instruction string, denoted Payn-inst: to the aggregated execution feedback. This yields the dynamic signal prompt:
Prignal [Paya-insts {(e), bm "l." "e)E),
(11)
where each tuple consists of a candidate completion ě², test case tm and its corresponding execution trace ed.m. Now we form new prompt naming this prompt Payn as dynamic signal prompt:
plyn
|p ddyn]; Psignal Psol Pdyn :]]
(12)
An example of the obtained prompt is shown in appendix Appendix
3.2 Classifier- Guidance (CFG)
Inspired by 23 39,, we utilize CFG to explicitly guide token generation by interpolating between two probability distributions: (i) an unconditional (prior) distribution based on the evolving solution prompt Psol. and (ii) conditional distribution based on dynamic signal prompt Payn which incorporates execution feedback. Formally, for each token w;, the CFG distribution is computed as:
log Mcra (w; Psyl:Pay)) = log Mw;; Psol) + y [log M Mww; Paya) log Mu;; Psa)), (13)
where y 2 0 explicitly controls the strength of guidance. Higher values of encourage the model to follow the execution-based guidance, while lower values allow greater flexibility toward the unconditional prior.
3.3 Execution-Guided Inference Loop
Our inference procedure extends standard autoregressive token generation by explicitly incorporating dynamic execution feedback via CFG. Starting from an initial prompt Ppre sal (Equation 5), we autoregressively sample tokens wi from the CFG-conditioned distribution McFa(w; Psol Payn). progressively constructing the solution sequence Psol- At each token leken-generation step, we reuse the dynamic signal Psignal (Equation IT). injecting into Psol index idyn form Payn as described in Psignal itself is regenerated only upon completing new line.
Equation 12
who = arg max McFG(w; Peot; Pōyn),
Psol
[ppre-sol, sol
(14)
3.4 Parallel Multi-Agent Execution
Given an input task as defined in Eequatiion I we launch parallel multi-agent inference process where each agent is assigned unique configuration: candidate count s, generation horizon d, sampling temperature f, instruction prompt template Po and guidance strength y. Once any agent finds a correct solution, it is immediately returned and all remaining agents are terminated. The full pseudo-code for both the execution-guiie guided inference loop and the multi-agent controller provided Appendix C
Experiments
Implementation Details We conduct our experiments using two LLMs across different parameter scales: DeepSeek-Coder- 1.3B [41), which is small enough run locally on our machines (NVIDIA GeForce RTX 2080 Ti and RTX 3090 GPUs), and a large open-source model, DeepSeek-V3- V3-0324 [24, which we use through cloud inference endpoint. We use Fireworks Al which was chosen based on two criteria: a modest cost, and the availability of a log probability output that is required to perform the CFG (Equation 13). The full code used for our experiments provided in the supplementary materials.

---

Hyperparameter Settings As explained in section 3.4 our method launches multiple parallel agents for each task. Each agent assigned a different hyperparameter configuration. The following hyper-parameter sets were used in our experiments: 8 = 3, E {0.7,0. 0.7,, 0.85, 0.95, 1.2, 1.5 E (,,3, 6, 8), 7 E (0, 0.5, 1,3). Additionally, we evaluate both po prompt templates, see section and appendix Appendix A.
Evaluation Benchmark
Our evaluations use widely-adoptee benchmarks: MBPP 3 (500 tasks) and EvmanEval [ (164 tasks), along with their extended test versions MBPP-ET and HumanEval- more challenging tasks, we also evaluate on the 1S-1000 data science benchmark [31] (1000 tasks) and the CodeContests competitive programming benchmark (using the ExecEval framework (42) ). We report accuracy: the percentage of problems passing all test cases. To rigorously test generalization and prevent overfitting to public tests, evaluations on HvmanEval, HumTnEVal-ETT, MBPP-ET, CodeContests, and DS- 1000 rely on hidden test cases inaccessible during inference. DS-1000's structure, providing only a single input-output example per problem, further tests model robustness against reliance examples.
ET 132 To assess performance
Baselines We compare our EG-CFG method against several established state-of-the-an art methods for code generation and debugging: a Baseline LLM using the few-shot template from DeepSeek-Code Coder's s evaluation [41]; MGDebugger R77, an iterative refinement method combining test-case feedback with L.L--simllttdd execution traces; MapCoder [3), which employs multi-agent interactions; QualityFlow [T], which incorporates agentic workflows for iterative enhancement; LPW (26), which utilizes structured two-phase workflow with runtime execution feedback; and CodeSim [4,, search-based method that adapts semantically similar code snippets.
4.1 Results
Using the DeepSeek-V3-ne V3-0324 model, EG-CFG achieves new saate-of-the-aee (SOTA) results across all evaluated benchmarks. On MBPP and MBPP-ET (Table T}, it surpasses prior methods using large closed-source models like GPT-4 and Claude 3.5. On HvmanEval (Table 2). it matches the state-of- the-art (LDB) at 99.4% and sets a new state-of-the-a on the challenging HumanEval-ETT variant. Furthermore, it establishes new state-of-the-arr on CodeContests TTalle 3) significantly exceeding its own baseline, MapCoder, and previous GPT-4 based approaches like LPW and CodeSim, and also sets new state-of-the-art on DS-1000 (69.9%. Table 4), outperforming CONLINE/GPT-4,
We note that across all tested baselines, the publicly available code was highly sensitive to the specific model and could not be readily applied to DeepSeek models. We invested substantial effort in debugging and adapting the code to ensure it produced meaningful results that represented each baseline method as favorably as possible. Other methods, such as QualityFlow, have not released their code, preventing us from evaluating them on the DeepSeek models. While LPW did release public implementation, we encountered substantial technical issues during the execution of their published code on the DeepSeek -Coder 1.3b model, resulting unusually low scores despite our best debugging efforts on that benchmark.
Run time A key advantage of EG-CFG is its native parallelism, design choice that aligns with the growing trend of leveraging large-scale, parallel compute in modern agentic systems. This architectural choice is key differentiator from iterative refinement methods, which are inherently sequential and therefore cannot leverage parallel compute to accelerate work on a single task. In contrast, CG- is designed to explore diverse reasoning paths simultaneously across multiple agents (as described in |section 3.4). This fundamental architectural difference makes wall-clock time the most relevant and fair metric for comparison. As can be seen in Table 5] on the MBPP benchmark, our method is more efficient than MGDebugger and competitive with MapCoder. When comparing with LPW, our method is more efficient with the smaller model but slower with the larger model.
Fairness of Comparison To demonstrate EG-CFG's qualitative advantage is not just a larger compute budget, we drastically increased baseline token usage. On MBPP SiepSeeek-3-0022- we raised MGDebugger and MapCoder retry counts 40-fold (from 5 to 200). Despite this 40x compute increase, gains were marginal: MGDebugger improved by only 6.8 percentage points, from 86.8% to 93.6% (solving 34 of its 86 prior failures). MapCoder's improvement was even smaller, at 1.6 percentage points, from 87.2% to 88.8% (solving 8 of its 64 prior failures). Both
7

---

Table 1: Performance on the MBPP and MBPP-ET benchmarks. Our proposed EG-CFG achieves a new ffate-of-thh-ar overall accuracy. The DeepSeek-Codee Coder- 1.3B and -V3-0324 results for all baselines were obtained by our study using the official implementations provided by each baseline method. The results below the double separator were collected from the respective papers. Model
Method
MBPP
MBPP-ET
Acc. (%)
RSR (%)
Acc. (%)
RSR
DeepSeek-Coder 1.3B
Baseline LLM
49.4
0.0
42,6
0.0
DeepSeek- 1.3B
EG-CFG (Ours)
83.2
66.79
59.8
29.96
DeepSeek- Coder 1.3B
MapCoder
55.2
11,46
46.2
6.27
DeepSeek-Codert 1.3B
MGDebugger 37
70.4
41.5
44.6
3.48
DeepSeck-V3-03224
Baseline LLM
82.8
0.0
64.8
0.00
SepPSeek-V3-0324
EG-CFG (Ours)
80.23
73.0
23.29
DeepScek-V3-0322
MapCoder [13
87.2
25.58
69.6
13.63
DeepSeek- SeepSeek-V3-0324
MGDebugger 37
86.8
23,25
64.8
0.00
DeepSeek V3-0334
LPW [26
84.0
6.97
65.2
1.13
GPT-4
Baseline L.LM
68.3
49.2
GPT-4
Self-Collaboration 43
78.9
62.1
GPT-4
Self-Debugging
80.6
GPT-4
MetaGPT 25
87.7
GPT-4
MapCoder 11
83.1
57.5
GPT-4o
LPW 26
84.8
65.8
CodeQwenl.5
MGDebugger 37
80.8
DeepSeek-Cooee Coder- V2-Lite
MGDebugger 37
80.0
Claude-Sonnet-3. Sonnet-3.5
Baseline LLM
88.7
Slaue--Sonnet-335
QualityFlow [11]
94.2
Table 2: Performance on the HumanEval and HumanEval-Ei benchmarks. Our proposed EG-CFG matches the state-of-the-an on HumanEval and achieves new state-of-the-art on HumanEval-ET..
Model
Method
HumanEval
HumanEval-Ei
Ace. (%)
RSR (%)
Acc. (%)
RSR (%)
DeepSeek-V3-033
Baseline LLM
82.92
0.0
79.20
0.0
DeepScek-V300324
EG-CFG (Ours)
99.4
94.04
89.02
47.21
V3-002
ManCoder 13
96.95
82.14
81.70
12.02
DeepSeek-V3-032.
MGDebugger
37
87.20
25.05
81.09
9.09
DeepStek-
L.W 26
95.12
71.42
84.74
26.63
GPT-4
Baseline LLM
67.7
50.6
GPT-4
Self-Coollborration
43
90.7
70.1
GPT-4
Self-Debugging [00
61.6
45.8
GPT-4
MetaGPT 22
85.9
GPT-4
MapCoder 13
80.5
70.1
GPT-4o
LPW 26
98.2
84.8
LLaMA 3
LDB 27
99.4
Saudee-Soneet-.5
QualityFlow [11]
98.8
CodeQwen1..
MGDebugger
37
91.5
64.1
DeepSeek- V2-Lite
MGDebugger
37
94.5
76.3
remained significantly below EG-CFG's 96.6% accuracy showing our performance gap is due the execution-guided feedback loop's 's quality, rather than the amount of computation.
Ablation study We performed an ablation study on the MBPP and MBPP-ET benchmarks to evaluate various components of our method. The results are reported in Table When omitting the beam search of Eqqation 6) which creates multiple solutions instead of single completion, the performance of the method drops and becomes much closer the baseline performance. The role of CFG is evident from the second ablation, in which value of y is used in [Equation 13) In this case, there is clear drop performance, although results are still clearly above baseline. Finally, similar drop in performance is observed when replacing the detailed execution trace used as part of the dynamic signal with a minimal execution trace, formally defined as the final event of the exccution trace [Equation 10).

---

Table 3: Performance on the CodeContests benchmark. Our proposed EG-CFG achieves new ofate-of-the-arr overall accuracy on CodeContests. The runs DeepSeek- V3-0324 are by us, and all other results are quoted from the literature. * 'The LPW results were obtained on a custom test-set; the published code was not compatible with evaluating on DeepSeek. Reported by the MapCoder
paper 13
Reported by the CodeSim paper 14.
Model
Method
Accuracy (%)
RSR (%)
DeepSeek-V3-0322
Baseline LLM
41.81
0.00
DeepSeek-V3 -33-0244
EG-CFG (Ours)
60.6
32.29
DeepSeek- V3-0324
DeepScck-V3-0324
MapCoder
50.30
14.59
DeepSeck-V3-0324
CodeSim
52.72
18.76
GPT-4o
LPW 26
34.7
GPT-4o
LDB 27
me*
29.3
GPT-4
CodeSim 14
29.1
GPT-4
MapCoder 11
28.5
GPT-4
Self-Planning
10.9
GPT-4
Analogical 45
10.9
GPT-3.5 Turbo
CodeSim [4
16.4
GPT-3.5 Turbo
MapCoder [1
12.7
GPT-3.5 Turbo
Analogical
45
7.3
GPT-3.5 Turbo
Self-Planning
6.1
MoTCoder-
MoTCoder
46
26.34
Table 4: Performance on the DS-1000 benchmark. Our proposed EG-CFG achieves new state- of-the-art overall accuracy on DS-1000. "Reported by the SelfEvolve paper [47]. ** "Reported by
DS-1000 31 official leaderboard.
Model
Method
Accuracy (%)
RSR (%)
DeepSeek- V3-0324 PepSeck-V3-0822
EG-CFG (Ours)
69.9
50.73
Baseline I.LM
38.9
0.00
GPT-4
CONLINE [48
68.0
GPT-4
Baseline L.M
60.2
GPT-3.5 Turbo
SelfEvolve [47
57.1
GPT-3.5 Turbo
Self Debugging 10
53.0
GPT-3.5 Turbo
DocPrompting 49
45.50
GPT-4o
Baseline L_M""
59.9
Claude 3.5 Sonnet
Baseline LLM"
54.3
DbepSeck-Coder V2-SFT Co-ST-V
Baseline LLM"
53.2
Owee2-77BB-nnttnen
Baseline LLM"
52.8
Conclusions
This paper introduces Execution Guiseed Classificr- Guidance (EG-CFG), a novel approach that fundamentally reframes neural code generation by incorporating real-time execution signals directly into the inference process. Our method bridges a critical gap between static pattern recognition and execution semanties by dynamically sampling candidate continuations, extracting execution traces, and leveraging these signals to guide token-level generation decisions.
The empirical results demonstrate that EG-CFG achieves new state-of-the-ate results across diverse set benchmarks. Using DeepSeek- V3-0324, it achieves 96.6% accuracy on MBPP, 73.0% on -e3pSeek-V33-0224,
MBPP-ET, 99.4% on HumanEval, 89.02% on HumanEyal-ET,, 69.90 99.9% on DS-1000, and 60.6% on CodeContests, surpassing both open-source and proprictary model- based approaches. The superior performance on both MBPP-ET and HumanEval-ET underscores the method's robustness and effectiveness generating reliable and accurate code under complex and extended test scenarios. Notably, our approach demonstrates robust performance even with smaller models, achieving 83.2% accuracy using DeepSeek-Coue Coder- 1.3B, comparable to results from substantially larger models like GPT-4. This scalability highlights the effectiveness of execution signals guiding mechanism regardless model capacity.
The EG-CFG approach offers several advantages over existing methods. Unlike discrete iterative refinement techniques that operate at coarse granularity between complete solution attempts, our method provides continuous feedback at the token level. By integrating execution signals that reflect

---

Table 5: Per-task runtime statisties (in seconds) for each model and method on MBPP.
Model
Method
Mean + SD (s)
SrepSeek-Codkee 1.3b
EG-CFG
123.23 1: 344.91
DeepSeck -odeer 1.3b
MGDebugger
495.16 - 411.07
DeepSeek- Coppeee-CCodrr 1.3b
MapCoder
121.9 213.89
DeepScek- Coder 1.3b
LPW
197.71 + 128.07
DeepSeck V33-0324
EG-CFG
271.37 271.45
DeepSeek-V3-ene V3-0324
MGDebugger MapCoder
842.24 705.19
DeepScek- --3-Scc3---333344 V3-0324
283.84 + 197.54
DeepSeek -V-beeek----3324
V3-0324
LPW
87.51 = 210.84
Table 6: Ablation results for EG-CFG on DeepSeek Coder 1.3b MBPP and MBPP-ET benchmarks.
Method
MBPP
MBPP-ET
Acc. (%)
RSR (%)
Acc. (%)
RSR (%)
EG-CFG EG-CFG, beam search
83.2
66.79
59.8
29,96
58.2
17.39
43.6
1.74
EG-CFG w/o CFG (y = 1)
75.2
50.98
48.2
9.74
EG-CFG, minimal trace
76.4
53.35
51.2
14.98
Baseline L.LM
49.4
0.0
42.6
0.0
actual runtime behavior, EG-CFG mirrors the incremental testing and debugging process that human programmers employ.
Looking forward, this work opens several promising research directions. The execution-guice guided framework could be extended more complex programming tasks requiring longer-horizon planning or multi-file interactions. Additionally, the principles of EG-CFG, which dynamically incorporate external semantie signals into generation, could benefit domains that rely grounding in systems, such as database querying, formal verification, or simulation-based tasks. More broadly, EG-CFG represents shift in generative modeling beyond static, patter-baseed generation toward responsive, context-aware methods informed by environmental interaction. As models scale and applications expand, such approaches drive the development of systems that generate high-quality outputs while reasoning about their real-world behavior, enabling more reliable and aligned generation.
Limitations
While the EG-CFG framework demonstrates significant improvements in code generation performance, several important limitations should be acknowledged.
First, the approach introduces computational overhead compared to standard inference methods. The beam search exploration, exccution of multiple candidate continuations, and dual-distribution interpo- lation in the CFG mechanism collectively increase inference time. Though our parallel execution strategy mitigates this overhead somewhat, future work should explore more efficient methods for extracting and incorporating execution signals. Second, EG-CFG's effectiveness is contingent upon the availability of executable test cases that adequately exercise the target functionality. In seal-world programming scenarios, comprehensive test cases may not always be available or easily generated by LLMs, potentially limiting the approach's 's applicability.
Finally, because our inference loop is bottom up.. it does not exploit task decomposition, strategy that has been shown improve code generation [37]. Future work could integrate our sampling strategy with iterative refinement methods, task-decomposition methods or with top-down problem-inspectie techniques [50] to achieve even better performance. Despite these limitations, EG-CFG represents significant advancement in execution- code generation and provides a solid foundation for future research addressing these challenges.
Acknowledgements
This work was supported by a Tel Aviv University Center for Al and Data Science (TAD) grant. This research was also supported by the Ministry of Innovation, Science & Technology. Israel (1001576155) and the Michael Fox Foundation (MJFF-022407). The contribution of SK is part of PhD thesis research conducted at Tel Aviv University.
10

---

References
[1] Mark Chen, Jenry Tworek, Heewoo Jun, Qining Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
[2] Juyong Jiang, Fan Wang. Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey large language models for code generation. arXiv preprint T2t1:22000005//5. 2024.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.00 07733. 2021.
[4] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 17866624) 2022.
[5] Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software enginecring: systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8):1 2024.
[6] Luyu Gao, Aman Madaan, Shuyan Zhou, Uni Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aides aided language models. In International Conference on Machine Learning, pages 10764-10799 PMLR, 2023.
[7] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. Re- act: Synergizing reasoning and acting language models. International Conference on Learning Representations (ICLR), 2023.
[8] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieher, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. 2021.
[9] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement leaming. Advances in Neural Information Processing Systems, 36:8634-8652, 2023.
[10] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, arXiv preprint xrXiv:2304.05/28, 2023.
[11] Yaojie Hu, Qiang Zhou, Qihong Chen, Xiaopeng Li, Linbo Liu, Dejiao Zhang, Amit Kachroo, Talha Oz, and Omer Tripp. Qualityflow: An agentic workflow for program synthesis controlled by Ilm quality checks. arXiv preprint urXiv:2501.1716: 2025.
[12] Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agentcoder: Multi- agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010, 2023.
[13] Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez, Mapcoder: Multi code generation for competitive problem solving. arXiv preprint arXiv:2405. 11403, 2024.
[14] Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. Codesim: Multi-agent code generation and problem solving through iimulation-driver planning and debugging. arXiv preprint arXiv:2502. 056664 2025.
[15] Thomas D LaToza and Brad A Myers. Developers ask reachability questions. In Proceedings of the 32Nd ACM/IEF International Conference on Software Engineering- Siiiineerinn-olumee Volume 1, pages 185-194, 2010.
[16] Amy J Ko, Brad A Myers, Michael Coblenz, and Htet Htet Aung. An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Transactions on software engineering, 32(12):971-987, 2006.
[17] Thomas R. G. Green and Marian Petre Usability analysis of visual programming environments: 'cognitive dimensions' framework. Journal of Visual Languages & Computing, 7(2):131-174, 1996.
[18] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi, The curious case of neural text degeneration. arXiv preprint MXXv:I904.0977 2019.
[19] Nan Wang. Yafei Liu, Chen Chen, and Haonan Lu. Genx: Mastering code and test generation with execution feedback. arXiv preprint 2412.1241, 2024.

---

[20] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint urXiv:2203.11171 2022.
[21] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809 11822, 2023.
[22] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprint rrXiv:220.10039) 2022.
[23] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598; 2022.
[24] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepscek-v3 technical report. arXiv preprint arXiv:2412. 194.9777. 2024.
[25] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng. Yuheng Cheng, Jinlin Wang. Ceyao Zhang, Zili Wang. Steven Ka Shing Yau. Zijuan Lin, al. Metagpt: Meta programming for a multi agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024.
[26] Chao Lei, Yanchuan Chang, Nir Lipovetzky, and Krista A Ehinger. boaning-driven programming: A large language model programming workflow. arXiv preprint urXiv:2411.14003 2024.
[27] Li Zhong, Zilong Wang, and Jingbo Shang. Debug like human: A large language model debugger via verifying runtime execution step-by-step. arXiv preprint arXiv:2402. 16906, 2024.
[28] Dan Hendrycks, Collin Burns, Steven Basart, Andy 7ou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[29] Richard J Waldinger and Richard CT Lec. Prow: A step toward automatic program writing. In Proceedings of the 1st international joint conference on Artificial intelligence, pages 241 242, 1969.
[30] Zohar Manna and Richard Waldinger. Toward automatic program synthesis. Communications of the ACM, 14(3):151-165, 1971.
[31] Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000; A natural and reliable benchmark for data science code generation. ArXiv, abs/2211. 11501, 2022.
[32] Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. Codescore: Evaluating code generation by learning code execution. ACM Transactions on Software Engineering and Methodology. 34 2025.
[33] Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Mathqa: Towards interpretable math word problem solving with operation-based formalisms. Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2357-22550 22672267, 2019.
Yejin Choi, and Hannaneh Hajishirzi.
[34] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
[35] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 2022.
[36] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Fric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539-68553 2023.
[37] Yuling Shi, Songsong Wang, Chengeheng Wan, and Xiaodong Gu. From code to correctness: Closing the last mile of code generation with hierarchical debugging. arXiv preprint 0rX0v:2410.01121,, 2024.
[38] Prafulla Dhariwal and Alexander Nichol, Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8743 2021.
12

---

[39] Guillaume Sanchez, Honglu Fan, Alexander Spangher. Elad Levi, Pawan Sasanka Ammanamanchi, and Stella Biderman. Stay topic with classifier-Iree guidance. arXiv preprint arXiv:2306. 17806, 2023.
[40] Guillaume Sanchez, Alexander Spangher, Honglu Fan, Elad Levi, and Stella Biderman. Stay topic with classifier-free guidance. In International Conference Machine Learning, pages 43197 432344.. PMLR, 2024.
[41] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Wu, YK Li, Deepscek-coders When the large language model meets programming the rise of code intelligence. arXiv preprint arXiv:2401. 14196, 2024.
[42] Mohammad Abdullah Matin Khan, M Saiful Bari, Do Long. Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. Xcodeeval: An execution- large scale multilingual multitask benchmark for code understanding. generation, translation and retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6766 6805, 2024.
[43] Yihong Dong. Xue Jiang. Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. ACM Transactions on Software Engineering and Methodology, 33(7):1-38, 2024.
[44] Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. Self seff-laaniing code generation with large language models. ACM Transactions Software Engineering and Methodology, 33(7):1-33 30, 2024.
[45] Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovee, Percy Liang, Ed H Chi, and Denny Zhou. Large language models analogical reasoners. arXiv preprint arX1v:2310.01714, 2023.
[46] Jingyao Li, Pengguang Chen, Bin Xia, Hong Xu, and Jiaya Jia. Motcoder: Elevating large language models with modular of thought for challenging programming tasks. arXiv preprint arXiv:2312. 2023.
[47] Shuyang Jiang, Yuhao Wang, and Yu Wang. Selfevolve: A code evolution framework via large language models. arXiv preprint arXiv:2306. 02907, 2023.
[48] Xinyi He, Jiaru Zou, Yun Lin, Mengyu Zhou, Shi Han, Zejian Yuan, and Dongmei Zhang. Conline: Complex code generation and refinement with online searching and correctness testing. CoRR, 2024.
[49] Shuyan Zhou, Uri Alon, Frank F Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. Docprompting: Generating code by retrieving the docs, arXiv preprint arXiv: 2207.05987, 2022.
[50] Shai Shalev-Shwartz, Amnon Shashua, Gal Beniamini, Yoav Levine, Or Sharir, Noam Wies, Ido Ben-Shaul, Tomer Nussbaum, and Shir Granot Peled. Artificial expert intelligence through pac-reasoning. arXiv preprint 21:12241,2,0244 2024.
13

---

A The two Pinst prompts used by our method
The two basic instruction prompts used by our method are provided in Figure 5 and Figure The former is the standard SrepSeek-Coder 3-shot prompt [41], consisting of concise instructions accompanied by short illustrative examples, and the latter is an alternative prompt explicitly designed to encourage the generation of hep-by-step solutions with more atomic logic.
14

---

Raw DeepSeck- Instrrcc Instruct Prompt for MBPP Task 395
You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, and you only answer questions related to computer science.
### Instruction:
Please refer the given examples generate python function my problem.
Examples are listed as follows:
Example 1:
>>> Problem:
Write function to find the similar elements from the given two tuple lists.
>>> Test Cases:
assert similar setaarreeeemettteC (33, 4, 5, 6), (5, 7, 4, 10)) = (4, 5)
assert similar elements (1, 2, 3, 4), (5, 4, 3, 7)) = (3, 4) assert similar_ent elements ((11, 12, 14, 13), (17, 15, 14, 13)) (13, 14)
>>> Code:
def similar_ed dataareeemmee elements (test_ tup1, test_tup2):
res tuple (est_ttpp) & set (teest_tup2)) return (res)
Example 2:
>>> Problem:
Write python function identify non-prime numbers.
Cases:
assert is is_not_prime (2) False
assert is_not_prrme (100) True
assert is_not_prrmm (35) == True
Code: import math def is_not_primett): result False
for (2, int (qtath.srr (n)) 1):
if n % 1 0: reault = True
return result
Example 3:
>>> Problem:
Write function to find the largest integers from given list of numbers using.
Test Cases: >>>
assert heap_queue Lareuue_argestteeee 35, 22, 85, 14, 65, 75, 22, 58] ,3) [85,
assert heap- largee ([25, 35, 22, 85, 14, 65, 75, 22, 58] 2) [85,
assert heap-queue__ue largest
leap_queue_aareest ([25, 35, 22, 85, 14, 65, 75, 22, 58] ,5) == [85,
Code:
import heapq as hq def heap_queue__teee ltagget (nums, n) largest_nume = hq. nlargest (n, nums)
return largest_nums
Here my problem:
>>> Problem:
Write python function to find the first non-repeated character in given string.
Cases:
assert Llast_non_rppeatinggccaaee. ("abcabc") None
assert first_non www.aoon_eeettiiggeeem character ( ("abc") == "a"
assert first_non_repeatinng character ("ababc") == "c"
Response:
Figure 5: The DeepSeck-Insten Instruct prompt used for MBPP Task 395, This prompt includes multiple solved examples followed by the target task.
15

---

Long- Lang-Instruct Prompt for MBPP Task 395 (Pina)
You are an AI programming assistant, utilizing the Deepseek Coder model, developed by Deepseek Company, ### Instruction:
Write python function find the first non-repeated character in given string.
Write Python function that satisfies the following test cases:
>>> Test Cases: ('assert cere_uoo_rppeatinngchhaeeee ("aacaaaat""""aab"" None', 'assert first_non__ent www.ruooo_eepatnngcccaaaaaaeeeee ("abc") "a",, ("ababc") = "c"']
'assert Eixat_non_reeeatinne
Your solution should be written in many lines possible. This ensures that prefizes of your function remain valid Python programs. Allowing **incremental execution and debugging**.
Write the function **step step**, progressively introducing variables and logic. Avoid using list comprehensions, lambda functions, overly compact one-liners. Instead, follow these guidelines:
Avoid list comprehensions, Incorrect:
loops instead:
return [x ** 2 for x in 1st]
Correct:
def square (1st)) squares
for num in 1st:
square__value ** 2
squares append value)
return squares
Avoid inline expressions, use variables instead
Incorrect:
def calculate, (1ength, width):
return (length width) / 2
Correct:
def caleulatte_area(onntthh (length, width): product length width area product / return area
Incorrect:
result. (p y)
Correct:
x
result. (p)end())
Incorrect:
def compute, value campute_value(a, b, c):
return (a b) (c / (a b) (a c) / (b c))
Correct:
def compute_ value campute_value(a, b, c):
termi
term2
term3 term2 term4 ccc / + c)
result = term1 (term3 + term4)
return result
Response:
Figure 6: Long-Instrue prompt for MBPP Task 395, This instruction-only prompt includes stylistic constraints that encourage traceable, step-by-step completions suitable for dynamic signal extraction.
16

---

B Example Dynamic Prompt Structure
This appendix provides concrete example illustrating the structure of the dynamic prompt Payn used by our EG-CFG method (described at the end of Section 3.1). Figure 7 beloow shows how the different components are assembled into the dynamic prompt Payn- displays the original task instruction (Pmma, followed by the dynamically injected signal (Paigna). This signal begins with the dynamic signal instruction (fayn-ins) and includes execution traces for each candidate, obtained this example by executing = candidates against a single test case (TT) 1). Finally, the figure shows the model's s partially completed response generated up to that point.
17

---

Instruction from Pins: Write python function to find the first non-repeated
character given string.
assert first_nou non, cets__non__eeeeeeee assert www..ooo_eepttcnnnccccaaaaerrr ("abcabc") None
wwwpatiin..aaaeeeeeeeeeeee ("aabc")
assert first_non_ htw.:ooo_eppaccng_ccaaaaaaaeeeee ("abc") == "a" ("ababc") "c"
first_nore seras_.on_repaaiinnccaaaeee
Dynamic Signal Instruction (Ptynina) starts at idyn Below are execution traces from running the response function after appending several possible future continuations. These continuations represent plausible ways the function might continue from its current state. They not necessarily full solutions
may be partial, exploratory, incomplete. For each candidate continuation, multiple test cases (invocations) were executed to observe its behavior under different inputs. Each entry includes:
À candidate version of the function
À specific test case used for invocation
The resulting execution
for that test
These dynamic signals can help you better understand how different plausible continuations behave at runtime, and guide you toward a more accurate solution.
Execution feedback for single test (T) = 1) and : candidates:
Function:
def First_non_ree www..repta.n..caaaatttteoeeeeee. (s) char_count = {
for char in char_count [char] char_count get (char, for char in s:
if char_count [char] == 1:
return char
return None
Invocation: first_non_ cepeating..caraccter Execution Trace:
'aabe', char_count {
char 'c² char char_ count ('at: 2, "b': 1, 'c': 1} char "b> -> count +> return 'b?
# Function:
def hrot.noo__ppetinnn.ccccrcrrrsr char_count (}
for char
count [char] chan_count. get (char, 0) 1 for char in 8: if char_count [char] == 2: return char
return None
# Invocation: first_non_ "ege.uoo...ppasiiicccaaaatttaaa Execution Trace:
'aabe', char_count {}
char 'c" count {'a': 2, >b' 1, to": 13 char la? -> count = 2 -> return 'a
### Response:
<<< ''python def prrss.oo..rpettigg.ccrrrrrroro char chur_count { for char char_coum [char] char get (char, 0) 1
Figure 7: Example of paya with injected Psignal at index idyo
18

---

C EG-CFG Pseudo-Code
To complement the formal description in Section we provide full pseudo-code for our method in this appendix. Algorithm [ ooutlnes the controller responsible for coordinating multi-agent parallel inference process, where each agent explores diverse reasoning paths using unique configuration. Each agent independently invokes the core decoding routine defined in Algorithm 2 which integrates dynamic execution feedback via classifier-free guidance (CFG). This loop incrementally generates a solution by incorporating real-time execution feedback into the prompt.
Algorithm Multi-Agent Parallel Inference Controller
Input: Task (Pusk; T, fmame) (Eq T. Model M, EG-CFG Configurations {(pu, 8, d, t,,))
I: Launch parallel agents A = (an. ft E }
2:
for all agents an with config (po, cl, t, t, in paralicl do
3: Pinst BUULDISSTRUCTTOOPPRMMII (po:
b Build instruction prompl, Equation
Pool EEE-CFGGINERRENCELLLOO 5: Execute(paa 1;) success, Vt, E 7 then
T, M, s,),,,,,
Verify solution, Equation 2
6: return
>> Return first valid solution and terminate other agents
7: end if
end
9: return mull
> Failure: No correct solution found
Algorithm 2 Execution Guidedd Classificr-Fres Guidance (EG-CFG) Inference Loop
Input: Pina; T, M, 8, d, t,,",,
1: Ppnc f- ((Pos)
Generate initial output sequence, Equation
2: Locate Ruslution, Adyn in
3: Ppe-so Pree Isalation]
Extract reasoning prefix before solution code. Equation
4: Pool Ppre-oo
Prignal < null
6: while true do
7: if Prignal null or (wi-1 exists and 'An" in ws-1) then
8:
Craw +-1
D Initialize list for executable candidates
for all each j 1,...,8do
10:
Initialize ej + while CountLines(es) < do (wk Psol, wey
11:
> Generate a candidate continuation, Equation
12:
Sample w
Append w o
14:
end while e Extrrac(xxcuutab(((()
15:
Extract executable part.
Equation
16:
Append ě to Craw end for
17:
18:
C Unique(Cravv)
Filter for unique candidate continuations, Equation Initialize list for feedback tuples
E
20:
Candidate executions against test cases run in parallel, Equation
21:
for all each è¹ E C' do
22:
for all each tno T do edm ExtractFxecutionneett Feeclaack fm)
23:
24:
Append (éé, tm, e. to end for end for
25:
26:
27:
Paimal 4-
[Polyn-inst; E]
b Construct the full dynamic signal. Equation
28:
end if
29:
Pdyn +
[paa]: I: iava]: Paignats Pool Eaan :]] arg max Mera(W Peot, Pays)
Inject signal into the prompt. Equation 12
30:
Sample next token using CFG.
31:
Append wi to Paol if wu = EOT or length(peat) 2 Nex then
D Append new token to the solution.
32:
33:
return Poot end if
34:
35:
end while
19

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research. addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes]. [No] or [NA]-
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist one of the factors their evaluation. While "[Yes]" " is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] " or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction explicitly claim state-of-the-art results on MBPP, HumanEval, and other benchmarks, which are substantiated by the experimental results in Section [ 1-5).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should elearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: Section exxpiciity discusses computational overhead, reliance on test case quality, and the lack of task decomposition.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
20

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the
implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only lested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper presents an empirical method (EG- CFG) and experimental evalua- tions; it does not introduce new theoretical theorems proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section detaiis implementation details and hyperparameter settings (s, d, 4, 7). Prompts are provided Appendix A and pseudo-code is provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken
make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same
21

---

dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The abstract includes URL to the code repository, and Section 4 states that the full code is provided in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips wttpccccpppcccccccccccccccccc.ct guwcc//gggee///e/eoo Cooeeoomiisssseeee cestsummissiooooolllo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The "Implementation Details" and "Hyperparameter Settings" in Section
specify the models, prompt templates, and specific parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail
that is necessary appreciate the results and make sense of them.
22

---

The full details can be provided either with the code, in appendix, or as supplemental
material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Table reports Mean = Standard Deviation for runtime statistics. For perfor- mance, we report standard Accuracy (%) metrics, consistent with the benchmarks used.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Section speciiiess the use of NVIDIA GeForce RTX 2080 Ti and RTX 3090 GPUs for local models and the Fireworks Al endpoint for larger models.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics Answer: [Yes]
Justification: The research involves generating code using standard open-source benchmarks and models; identify violations of the code of ethics. Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
23

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The Conclusion mentions positive impacts (enabling more reliable generation), but negative societal impacts are not explicitly discussed as the work is foundational research on code generation mechanics. Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper releases code for an inference method and utilizes existing public models; it does not release new high-risk models requiring specific safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites all used models (DeepSeek) and benchmarks (MBPP, Hu- manEval, DS- 1000, CodeContests) and utilizes them in accordance with their standard usage.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
24

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, snnerswwthcode com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces a new code repository implementing the EG-CFG method; documentation is provided in the supplementary material and the repository.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This research did not involve crowdsourcing human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This research did not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
not include any information that would break anonymity (if applicable), such as the institution conducting the review.
For initial submissions,
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Large Language Models (DeepSeek-Codees Coder//3) are central to the proposed method and experiments, descrihed in Sections
and
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy for what should or should not be described.
26


Reviews:
[{"review_id": "voJvZjionC", "reviewer": "Reviewer_FLp2", "summary": "This paper introduces a novel code generation method called Execution-Guided Classifier-Free Guidance. The core idea is to incorporate real-time execution feedback during the inference process of a language model, mimicking how human programmers iteratively run partial code to guide development.", "questions": "Refer to the weaknesses please.", "limitations": "Refer to the weaknesses please.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-writing and easy to follow.\n2. The proposed method is inference-based without introducing the additional training cost.\n3. Simple, single-agent setup makes the method easy to understand and reproduce.\n4. The performance for the evaluation on MBPP is good.\n\nWeaknesses:\n1. The evaluation is not diversity enough. It may be possible that the proposed method's generalization ability is not so clear. Is it possible to extend to some more challenging benchmark like LivecodeBench, and other coding tasks like DS-1000 (data science) and Bird (SQL)?\n2. Relies on test cases for feedback, which may limit applicability to real-world tasks lacking such supervision. Or would the methods highly rely on the high quality and reliable test cases? At least, the paper should discuss more about corresponding impact and possible limitations.\n3. Introduces significant computational overhead due to repeated execution and beam search. Would it be possible to discuss the inference time and token number compared with other baselines?", "was_revised": true, "final_justification": "My concern is well addressed, so I tend to vote for the acceptance of this paper.", "rebuttal": "Thank you for your thoughtful review and constructive feedback. \n\n## Evaluation Diversity and Generalization \nWe have substantially expanded our evaluation beyond MBPP to address concerns about benchmark diversity and broader applicability.\nOur updated experiments include the additional five benchmarks:\n- **HumanEval**: A foundational and widely adopted benchmark.\n- **MBPP-ET** and **HumanEval-ET**: Extended test suites that add 100 hidden test cases per task, offering a more rigorous evaluation of generalization and robustness - helping mitigate overfitting to visible tests.\n- **CodeContests**: A challenging competitive programming benchmark that requires multi-step reasoning and broader generalization.\n- **DS-1000**: A domain-specific benchmark targeting data science tasks. In this dataset, the problem is defined without heavily relying on test cases (one sample is shown as part of the problem prompt).\n\nNotably, the benchmarks HumanEval, CodeContests, DS-1000 contain hidden test cases which help mitigate overfitting to visible tests.\n\nOur method achieves state-of-the-art performance across all major benchmarks: MBPP (96.6%), MBPP-ET (73.0%), HumanEval (99.4%), HumanEval-ET (89.0%), CodeContests (60.6%), CodeContests (60.6%) and DS-1000 (69.9%). \n\nWe compared against every strong baseline we could find, both from PapersWithCode and directly from papers. We weren’t just relying on reported results, we actively tried to reproduce methods ourselves wherever possible. In many cases, we reran existing methods on the model (DeepSeek‑V3‑0324), to ensure a fair comparison. When code didn’t work with DeepSeek or wasn’t available, we adapted or re-implemented it, and clearly documented any limitations.\nThese results show that our method is not limited to MBPP. It generalizes effectively across diverse levels of complexity, from foundational problems to competitive programming tasks and domain-specific benchmarks - demonstrating the broader applicability of our method.\n\nWe include detailed benchmark comparison tables below (due to character limit, CodeContests table can be found in reply to fegN). These present accuracy and retry success rates (RSR) across all datasets and baselines, including both our runs on DeepSeek‑V3‑0324 and results reported in the literature.\n\n**Table**: *Performance on the MBPP and MBPP-ET benchmarks.*  \nOur proposed **EG-CFG** achieves a new state-of-the-art overall accuracy.  \nThe DeepSeek–Coder-1.3B and –V3-0324 results for all baselines were obtained by our study using the official implementations provided by each baseline method.  \nThe results below the double separator were collected from the respective papers.\n\n| **Model** | **Method** | **MBPP Acc. (%)** | **MBPP RSR (%)** | **MBPP-ET Acc. (%)** | **MBPP-ET RSR (%)** |\n|-----------|------------|-------------------|------------------|----------------------|---------------------|\n| DeepSeek-Coder 1.3B | Baseline LLM | 49.4 | 0.0 | 42.6 | 0.0 |\n| **DeepSeek-Coder 1.3B** | **EG-CFG (Ours)** | 83.2 | 66.79 | 59.8 | 29.96 |\n| DeepSeek-Coder 1.3B | MapCoder [Islam et al., 2024] | 55.2 | 11.46 | 46.2 | 6.27 |\n| DeepSeek-Coder 1.3B | MGDebugger [Shi et al., 2024] | 70.4 | 41.5 | 44.6 | 3.48 |\n| DeepSeek-V3-0324 | Baseline LLM | 82.8 | 0.0 | 64.8 | 0.0 |\n| **DeepSeek-V3-0324** | **EG-CFG (Ours)** | **96.6** | **80.23** | **73.0** | **23.29** |\n| DeepSeek-V3-0324 | MapCoder [Islam et al., 2024] | 87.2 | 25.58 | 69.6 | 13.63 |\n| DeepSeek-V3-0324 | MGDebugger [Shi et al., 2024] | 86.8 | 23.25 | 64.8 | 0.0 |\n| DeepSeek-V3-0324 | LPW [Lei et al., 2024] | 84.0 | 6.97 | 65.2 | 1.13 |\n| GPT-4 | Baseline LLM | 68.3 | - | 49.2 | - |\n| GPT-4 | Self-Collaboration [Self-Collaboration] | 78.9 | - | 62.1 | - |\n| GPT-4 | Self-Debugging [Chen et al., 2023] | 80.6 | - | - | - |\n| GPT-4 | MetaGPT [MetaGPT] | 87.7 | - | - | - |\n| GPT-4 | MapCoder [Islam et al., 2024] | 83.1 | - | 57.5 | - |\n| GPT-4o | LPW [Lei et al., 2024] | 84.8 | - | 65.8 | - |\n| CodeQwen1.5 | MGDebugger [Shi et al., 2024] | 80.8 | - | - | - |\n| DeepSeek-Coder-V2-Lite | MGDebugger [Shi et al., 2024] | 80.0 | - | - | - |\n| Claude-Sonnet-3.5 | Baseline LLM [Hu et al., 2025] | 88.7 | - | - | - |\n| Claude-Sonnet-3.5 | QualityFlow [Hu et al., 2025] | 94.2 | - | - | - |\n\n**Table**: *Performance on the HumanEval and HumanEval-ET benchmarks.*  \nOur proposed **EG-CFG** achieves a new state-of-the-art overall accuracy on HumanEval-ET.\n\n| **Model** | **Method** | **HumanEval Acc. (%)** | **HumanEval RSR (%)** | **HumanEval-ET Acc. (%)** | **HumanEval-ET RSR (%)** |\n|-----------|------------|------------------------|-----------------------|---------------------------|--------------------------|\n| DeepSeek-V3-0324 | Baseline LLM | 82.92 | 0.0 | 79.20 | 0.0 |\n| **DeepSeek-V3-0324** | **EG-CFG (Ours)** | **99.4** | **94.04** | **89.02** | **47.21** |\n| DeepSeek-V3-0324 | MapCoder [Islam et al., 2024] | 96.95 | 82.14 | 81.70 | 12.02 |\n| DeepSeek-V3-0324 | MGDebugger [Shi et al., 2024] | 87.20 | 25.05 | 81.09 | 9.09 |\n| DeepSeek-V3-0324 | LPW [Lei et al., 2024] | 95.12 | 71.42 | 84.74 | 26.63 |\n| GPT-4 | Baseline LLM | 67.7 | - | 50.6 | - |\n| GPT-4 | Self-Collaboration [Self-Collaboration] | 90.7 | - | 70.1 | - |\n| GPT-4 | Self-Debugging [Chen et al., 2023] | 61.6 | - | 45.8 | - |\n| GPT-4 | MetaGPT [MetaGPT] | 85.9 | - | - | - |\n| GPT-4 | MapCoder [Islam et al., 2024] | 80.5 | - | 70.1 | - |\n| GPT-4o | LPW [Lei et al., 2024] | 98.2 | - | 84.8 | - |\n| LLaMA 3 | LDB [Zhong et al., 2024] | 99.4 | - | - | - |\n| Claude-Sonnet-3.5 | QualityFlow [Hu et al., 2025] | 98.8 | - | - | - |\n| CodeQwen1.5 | MGDebugger [Shi et al., 2024] | 91.5 | 64.1 | - | - |\n| DeepSeek-Coder-V2-Lite | MGDebugger [Shi et al., 2024] | 94.5 | 76.3 | - | - |\n\n**Table**: *Accuracy results from the official DS-1000 leaderboard.*\n| **Model** | **Method** | **Accuracy** | **RSR** |\n|-----------|------------|--------------|---------|\n| **DeepSeek-V3-0324** | **EG-CFG (Ours)** | **69.9%** | **50.73%** |\n| DeepSeek-V3-0324 | Baseline LLM | 38.9% | 0.00% |\n| gpt-4o-2024-08-06 | Baseline LLM | 59.9% | - |\n| claude-3-5-sonnet-20240620 | Baseline LLM | 54.3% | - |\n| gpt-4-turbo-2024-04-09 | Baseline LLM | 54.0% | - |\n| deepseek-ai-deepseek-coder-V2-SFT | Baseline LLM | 53.2% | - |\n| Qwen-Qwen2-72B-Instruct | Baseline LLM | 52.8% | - |\n| deepseek-chat-V2.5 | Baseline LLM | 51.2% | - |\n| mistralai-Codestral-22B-v0.1 | Baseline LLM | 51.2% | - |\n| gpt-4-0613 | Baseline LLM | 51.0% | - |\n| gpt-4o-mini-2024-07-18 | Baseline LLM | 50.5% | - |\n\n\n## Dependence on Test Cases\nOur method is less dependent on test cases than the existing methods, since in most stages of the generation, it relies on implicit supervision, which is a central part of our method. The execution behavior is used as a soft feedback signal, interpreting outcomes without being explicitly told what is correct. This allows it to self-verify regardless of the quality of the test cases. We explicitly discuss this in the Introduction section (51-55) and highlight it as a key differentiator from prior approaches in the Related Work section (lines 141-143).\nWe note in the limitation section that the method is applied to benchmarks that have executable test cases. This is the most direct way to compare with the existing relevant methods. However, it is a soft limitation since test cases can also be generated by the model itself from the task instruction.\nFinally, in five out of six benchmarks we evaluate on (HumanEval, HumanEval-ET, MBPP-ET, CodeContests, and DS-1000), the test cases are hidden. This means the model never sees them during inference. It helps ensure that our method’s strong performance is not due to overfitting to public test cases and further supports its generalization capabilities.\nWe also note that in DS-1000, each problem includes only a single input-output example, making it impractical to rely heavily on such examples during code generation.\n\n## Computational Overhead\nFirst, native parallelization is a key strength of our method. We leverage the low price of existing APIs and the ability to parallel them, which is a growing trend of today's agents. Other baseline methods mentioned in our paper can also have access to full parallel compute, but by design, they cannot fully leverage it. They rely on sequential retries, where each iteration depends on the output of the previous one. This makes it impossible for them to parallelize inference on a single task, even if the computational resources are available. \nMoreover, in (5/6) of the benchmarks: HumanEval, HumanEval-ET, MBPP-ET, CodeContests, DS-1000 there are hidden-test cases. There are hidden test cases. This means our method cannot rely on extensive token usage to overfit the public tests in order to succeed.\nAdditionally, In order to rigorously make sure the comparison is fair, we ran trials using the DeepSeek-V3 model with two baseline methods: MGDebugger and MapCoder, on the MBPP benchmark, using a much higher number of retries which leads to significantly more extensive token usage. For MGDebugger, we increased the retry count from 5 to 200 - x40 increase, resulting in ~x40 tokens. This resulted in only an improvement of 34 out of 500 (6.8 percent), out of the 86 examples it originally got wrong. For MapCoder, we similarly increased retries from 5 to 200 (40x tokens), and saw an improvement of just 8 out of 500 (1.6 percent), out of 64 examples it originally failed.\nIn both cases, the improvement was achieved with a 40x increase in runtime and is still considerably below our performance on MBPP.\nThis supports our conclusion that the advantage of our method is not due to higher token usage, and therefore the comparison is fair. In order to clarify this issue more we will add discussion about this point in our revised version."}, {"review_id": "mKcc3B6vQP", "reviewer": "Reviewer_7xf3", "summary": "The authors present a method for generating solutions to the MBPP (mostly Basic Python Programs) dataset by forming incomplete candidate programs consisting of a small number of lines, then iteratively: executing them (after some minor syntactic repair to make them executable) while tracing the internal state, and prompting the model to further continue the programs by appending more lines to the end. Compared to other iterative generation approaches, which largely rely on debugging and editing already-completed programs, the approach focuses on executing unfinished programs and using the execution information to further extend them. They compare against several other methods and report state-of-the-art results on the MBPP dataset.", "questions": "Questions:\n- How was beam search performed with lines of code as the units for the length of the samples, since a fixed number of lines of code doens't translate into a fixed number of tokens?\n- In equation 10, what happens if some value of $\\gamma$ causes $\\log M_{CFG}(\\cdots)$ to become a positive number? I think this can happen unless $0 \\le \\gamma \\le 1$.\n- When does the loop end if a correct solution is not found (i.e. how long do you try before you give up)?\n- What happens to the initial solution $w^0_{pre}, \\cdots, w_\n- What is the relationship between $p_{sol}$ and $p_{pre}$?\n\nI found it hard to follow exactly the procedure used and would have appreciated 1) more examples of the prompts used with the models, 2) a diagram explaining the loop, 3) the overall procedure expressed as pseudocode.\n\nLine 168: \"last few example inside the template\" should be \"last few-shot example inside the template\"?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The paper reports SoTA results on a dataset that many others have attempted.\n- The method tries out an interesting combination of classifier-free guidance with iterative generation of candidate programs while providing execution feedback.\n\n## Weaknesses\n- I found the procedure in the paper a bit hard to understand, for example:\n  - Some parts (like equation 3) discuss steps as tokens during decoding; but in other parts, there is a different notion of a step which occurs due to gradual extension of canddiate solutions, which isn't as clearly described.\n  - There is an outer loop for extending solutions but it is not explicitly and clearly stated how it works, e.g. how many times it would run and what are the termination conditions.\n  - The details of the execution traces is not clearly stated.\n- The experiments are only on MBPP, which is now a few years old and superceded by other similar datasets such as LiveCodeBench. - - As I understand it, MBPP has no hidden test cases, which reduces the realism of the benchmark; the procedure outlined might have overfit to that aspect of the benchmark, considering aspects like \"we return the first correct solution identified\".\n- Because of the lack of hidden test cases, a method that tries all possible programs (given infinite time) would get 100% accuracy on MBPP. It wasn't clear to me whether the comparison to baseline methods is fair in the sense that this method might have used significantly more computation than the alternatives, or significantly more trials. Only wall-clock time with \"full parallelization\" is reported as a comparison.\n\nI would have trouble recommending the paper for acceptance until at least the clarity-related concerns are resolved, and I would suggest trying the approach on other datasets than MBPP as well.\n\n(REVISION AFTER REBUTTAL): While the clarity-related concerns are hard to evaluate, I believe the authors have significantly addressed my concerns about generalizability and inference budget.", "was_revised": true, "final_justification": "The authors addressed significant concerns in the paper regarding experimental results, which I am putting high weight on my decision to raise the score from 3 to 5. The clarity issues are hard to resolve through the rebuttal given its format, and the authors did not include the specific new language they would use in a revised paper, but I think those can be considered more minor.", "rebuttal": "Thank you for your thoughtful and detailed review. Your comments helped us identify several areas that we want to clarify and we have worked hard to improve the paper accordingly. Below, we address each of your points in turn.\n\n## Clarity Related Concerns \n> Ambiguity Between Token Steps and Candidate Extensions\n\nOn the one hand, we used token notation in Eq. (3) and (7), as these parts are related to token sampling and indexing of specific tokens. On the other hand, we chose to use the granularity of code snippets in the candidate continuation sampling in Eq. (4), because after sampling candidate continuation programs we perform deduplication and execution which are both operating at the granularity of full code snippets rather than individual tokens. For consistency and extra clarity, we will revise Eq. (4) to use token notation as well.\n\n> Outer Loop Behavior and Termination Conditions\n\nThe outer loop is the execution of each agent as an independent inference run, which terminates (returns) when the first correct solution is found. Otherwise, inference continues across the remaining agents (lines 229 to 231). Each inference run is independent and allows parallel execution (line 274). We did not describe a retry mechanism because there is no such mechanism, i.e., each agent runs exactly once. To make this clearer, we will revise the Method section to describe the outer loop execution as such and will include corresponding pseudocode.\n\n> Execution Trace Format Was Unclear\n\nOur approach is agnostic to the format of the execution feedback (line 196). This is an important part of our novelty, as we treat execution feedback as a soft guidance signal. The model interprets execution behavior on its own, without being explicitly told what is correct or relying on any specific format (lines 141–142). The structure of the execution trace used in our implementation is described in lines 197–200. To further illustrate this, we included Figure 2, which shows a concrete example of a trace. In our revision, we will formalize the trace structure so that the exact format is fully explicit.\n\n> Clarifying the Inference Procedure\n\nIn our revision, we will include full pseudocode outlining the complete procedure. We will split it into two algorithms: Algorithm 1 describes the outer loop that executes agents in parallel, each executing our core algorithm “Execution-Guided Inference Loop” (Algorithm 2). In addition, we will add more prompt examples and a diagram.\n\n## Benchmark Diversity\nWe have substantially expanded our evaluation beyond MBPP to address concerns about benchmark diversity and broader applicability.\nOur updated experiments include the additional five benchmarks:\n- **HumanEval**: A foundational and widely adopted benchmark.\n- **MBPP-ET** and **HumanEval-ET**: Extended test suites that add 100 hidden test cases per task, offering a more rigorous evaluation of generalization and robustness - helping mitigate overfitting to visible tests.\n- **CodeContests**: A challenging competitive programming benchmark that requires multi-step reasoning and broader generalization.\n- **DS-1000**: A domain-specific benchmark targeting data science tasks. In this dataset, the problem is defined without heavily relying on test cases (one sample is shown as part of the problem prompt).\n\nNotably, the benchmarks HumanEval, CodeContests, DS-1000 contain hidden test cases which help mitigate overfitting to visible tests.\n\nOur method achieves state-of-the-art performance across all major benchmarks: MBPP (96.6%), MBPP-ET (73.0%), HumanEval (99.4%), HumanEval-ET (89.0%), CodeContests (60.6%),  and DS-1000 (69.9%).\nWe compared against every strong baseline we could find, both from PapersWithCode and directly from papers. We weren’t just relying on reported results, we actively tried to reproduce methods ourselves wherever possible. In many cases, we reran existing methods on the model (DeepSeek‑V3‑0324), to ensure a fair comparison. When code didn’t work with DeepSeek or wasn’t available, we adapted or re-implemented it, and clearly documented any limitations.\n\nThese results show that our method is not limited to MBPP. It generalizes effectively across diverse levels of complexity, from foundational problems to competitive programming tasks and domain-specific benchmarks - demonstrating the broader applicability of our method.\n\nWe include detailed benchmark comparison tables below (other tables can be found in reply to fegN and FLp2). These present accuracy and retry success rates (RSR) across all datasets and baselines, including both our runs on DeepSeek‑V3‑0324 and results reported in the literature.\n\n\n## ​​Fairness of Computational Comparison\nFirst, native parallelization is a key strength of our method. We leverage the low price of existing APIs and the ability to parallel them, which is a growing trend of today's agents. Other baseline methods mentioned in our paper can also have access to full parallel compute, but by design, they cannot fully leverage it. They rely on sequential retries, where each iteration depends on the output of the previous one. This makes it impossible for them to parallelize inference on a single task, even if the computational resources are available. \nMoreover, in (5/6) of the benchmarks: HumanEval, HumanEval-ET, MBPP-ET, CodeContests, DS-1000 there are hidden-test cases. There are hidden test cases. This means our method cannot rely on extensive token usage to overfit the public tests in order to succeed.\n\nAdditionally, In order to rigorously make sure the comparison is fair, we ran trials using the DeepSeek-V3 model with two baseline methods: MGDebugger and MapCoder, on the MBPP benchmark, using a much higher number of retries which leads to significantly more extensive token usage. For MGDebugger, we increased the retry count from 5 to 200 - x40 increase, resulting in ~x40 tokens. This resulted in only an improvement of 34 out of 500 (6.8 percent), out of the 86 examples it originally got wrong. For MapCoder, we similarly increased retries from 5 to 200 (40x tokens), and saw an improvement of just 8 out of 500 (1.6 percent), out of 64 examples it originally failed.\nIn both cases, the improvement was achieved with a 40x increase in runtime and is still considerably below our performance on MBPP.\n\nThis supports our conclusion that the advantage of our method is not due to higher token usage, and therefore the comparison is fair. In order to clarify this issue more we will add discussion about this point in our revised version.\n\n## Questions \n\n> **How was beam search performed with lines of code as the units for the length of the samples, since a fixed number of lines of code doens't translate into a fixed number of tokens?**\n\nSampling is performed auto-regressively with a stopping condition based on the number of newline characters (\\n) encountered. Specifically, generation stops once the target number of lines, denoted by d, is reached. We do not assume a fixed number of tokens per line. We explicitly wrote in lines 175-176.\n\n> **In equation 10, what happens if some value of  gamma causes M_CFG to become a positive number? I think this can happen unless.**\n\nThank you for clarifying this point. For some values of gamma, the guided logits M_CFG​ can include positive values and won’t sum to a valid probability distribution. When using argmax decoding rather than sampling, normalization isn’t necessary. Softmax wouldn’t change the selected token, so the behavior of the algorithm remains the same. However, it is possible to normalize the distribution to make it compatible with the standard inference loop that allows sampling.\n\n> **When does the loop end if a correct solution is not found (i.e. how long do you try before you give up)?**\n\nWe do not use a retry mechanism within a single agent. Instead, we run multiple agents in parallel, each following a different reasoning path by varying dynamic signal parameters. As soon as one agent finds a correct solution, the others are terminated, and that solution is returned. If none succeed, no solution is reported for that task - “not found”. We can add a retry mechanism, which was shown to be beneficial to iterative refinement methods.\n\n> **What happens to the initial solution $w^0_{pre}, \\cdots, w_**\n> **What is the relationship between p_sol  and p_pre ?**\n\np_pre refers to the full model output in response to the instruction prompt p_inst. It includes both the intermediate reasoning and the solution block (which contains the “initial solution”). We define i_sol as the position of the last code-start token in p_pre, marking the beginning of the solution block. The solution prompt p_sol is constructed by taking the prefix of p_pre up to i_sol (We ignore the tokens representing “initial solution” which come after that prefix) by appending the new tokens generated by our decoding algorithm, until a code-end token is generated. The tokens generated beyond i_sol form the final solution returned by our algorithm.\n\n> **Line 168: \"last few example inside the template\" should be \"last few-shot example inside the template\"?**\n\nCorrect - fixed.\n\nIn conclusion, we sincerely thank you for your thoughtful and detailed review. We've worked hard to address each of your concerns through significant clarifications, expanded experiments, and improved presentation of our method. We hope these revisions make the contribution and impact of our work clearer. If you find the updated version compelling, we kindly ask you to consider updating your score. We truly appreciate your time and feedback."}, {"review_id": "HYLVb1CRcw", "reviewer": "Reviewer_fegN", "summary": "This work proposes a new decoding method for LMs to perform code completion tasks. Unlike prior work, which mainly focuses on having LMs loop between generating code and understanding execution logs at a multi-turn level, this method proposes incorporating execution into the decoding procedure, with LMs being provided execution signals at intermediate points in the code generation process. The authors describe the iterative procedure to extract executable code from intermediate generations, incorporate execution into the in-progress inference output, and use CFG to allow for more dynamic sampling during the decoding process. Evaluating on MBPP, the authors show superior performance over LMs equipped with more traditional execution-feedback techniques, and the ablations back up several of the design decisions made for the decoding pipeline.", "questions": "- Line 158 - p_inst is p_0 and p_task combined? Line 170 - where does p_sol fit into this whole process? This is referring to what is eventually regarded as the solution generated by the LM, which is then evaluated for correctness?\n- Figure 2 - what is the highlighted “...” character referring to?\n- Is there a reason you didn’t compare against agent scaffolds such as SWE-agent, InterCode, CodeAct, or OpenHands? It feels like those works explore the inference-from-execution-across-turns approach that maybe interesting to compare against. At this point, a lot of the code completions benchmarks (e.g. MBPP, HumanEval) feel very saturated, as a strong model like the GPT o* series or Claude 3.7/4 could zero-shot get 90%+ on the benchmark. I feel like the performance gains could be more meaningful if evaluated on a more challenging setting.\n- What is “minimal trace” in Table 3?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* I appreciate the novelty of this work, and I think it is fundamentally quite interesting. Incorporating execution feedback into the decoding process of an LM seems quite neat, and prior works have not really explored this space at all, so it is very neat to see this idea carried out with positive results.\n* I liked going through Section 3, and I thought it was written in a very clear and presentable way, where it was very straightforward how each step flowed into the next.\n* The results are strong, indicating that this approach works well for MBPP, and more generally, code completion tasks that are written in Python. I think it would be helpful to make the ablations section a bit longer and potentially explain why performance drops can be attributed to certain omissions (e.g. no beam search, CFG), but otherwise the ablations as is very succinctly demonstrate the benefit of each step of the pipeline.\n\nWeaknesses:\n* My interpretation is that this approach works exclusively for Python code right now - is this the right understanding? I think this is because of the “Executable Extraction” step, which seems to suggest that the extraction relies heavily on the `ast` library. I’m sure this may be extensible to other languages using the `tree-sitter` library or some non-Python AST construction library, but I feel this adaptation might be non-trivial?\n* Since the testbed is mainly function completions (e.g. MBPP, HumanEval), what happens if the code completions are not so “self-contained”? I think this method is quite cool, but I’m not sure how adaptable it is to in-the-wild code completion, as it feels like there’s several criteria that must be satisfied (unit tests, executable env., Python, target completion is for a function) in order for this to work.\n* Can you do evaluations with this method on other code completion benchmarks? E.g. HumanEval, LiveCodeBench, BigCodeBench? It’s impressive this method leads to a +14% performance boost (based on Table 1), but I’m curious if this carries over for other code completion settings.", "was_revised": true, "final_justification": "As I mention in my response to the authors, I maintain my current scores of 4 (rating) and 3 (confidence) in this work.\n\nThe authors demonstrate with many more evaluations that this method holds for a broad number of Python code completion benchmarks, which is great to see.\n\nI did not provide a higher score because of the specificity of this method. More complex coding related domains have emerged in the past year (e.g., SWE-bench) and multilingual code completion evaluations are readily available. For a higher score, I would have liked to see this method's generalizability to more languages and domains to justify more significant impact.", "rebuttal": "We sincerely appreciate your detailed and constructive review. Your positive remarks on the novelty of our approach were encouraging. Your suggestions helped us identify key areas for clarification and refinement of our work. We've made substantial efforts to improve the paper based on your feedback. Below, we respond to each of your comments in detail.\n\n## Executable Extraction Generality\nEG-CFG is language-agnostic. It only requires the ability to extract executable code segments, execute them, and collect execution feedback. The current implementation is specific to Python and uses the ast library for extracting executable code and a Python debugger that we modified specially for extracting the execution feedback. These can definitely be implemented in other languages. \n\n## Benchmark Diversity\nWe have substantially expanded our evaluation beyond MBPP to address concerns about benchmark diversity and broader applicability.\nOur updated experiments include the additional five benchmarks:\n- **HumanEval**: A foundational and widely adopted benchmark.\n- **MBPP-ET** and **HumanEval-ET**: Extended test suites that add 100 hidden test cases per task, offering a more rigorous evaluation of generalization and robustness - helping mitigate overfitting to visible tests.\n- **CodeContests**: A challenging competitive programming benchmark that requires multi-step reasoning and broader generalization.\n- **DS-1000**: A domain-specific benchmark targeting data science tasks. In this dataset, the problem is defined without heavily relying on test cases (one sample is shown as part of the problem prompt).\n\nNotably, the benchmarks HumanEval, CodeContests, DS-1000 contain hidden test cases which help mitigate overfitting to visible tests.\n\nOur method achieves state-of-the-art performance across all major benchmarks: MBPP (96.6%), MBPP-ET (73.0%), HumanEval (99.4%), HumanEval-ET (89.0%), CodeContests (60.6%),  and DS-1000 (69.9%). \n\nWe compared against every strong baseline we could find, both from PapersWithCode and directly from papers. We weren’t just relying on reported results, we actively tried to reproduce methods ourselves wherever possible. In many cases, we reran existing methods on the model (DeepSeek‑V3‑0324), to ensure a fair comparison. When code didn’t work with DeepSeek or wasn’t available, we adapted or re-implemented it, and clearly documented any limitations.\n\nThese results show that our method is not limited to MBPP. It generalizes effectively across diverse levels of complexity, from foundational problems to competitive programming tasks and domain-specific benchmarks - demonstrating the broader applicability of our method.\n\nWe include detailed benchmark comparison tables below (other tables can be found in reply to FLp2). These present accuracy and retry success rates (RSR) across all datasets and baselines, including both our runs on DeepSeek‑V3‑0324 and results reported in the literature.\n\n**Table**: *Performance on the CodeContests benchmark.*  \nOur proposed **EG-CFG** achieves a new state-of-the-art overall accuracy on CodeContests.  \nThe runs on DeepSeek-V3-0324 are by us, and all other results are quoted from the literature.  \n\\*LPW results were obtained on a custom test-set; the published code was not compatible with evaluating on DeepSeek.  \n\\*\\*Reported by the MapCoder paper.\n\\*\\*\\*Reported by the CodeSim paper.\n| **Model** | **Method** | **Accuracy (%)** | **RSR (%)** |\n|-----------|------------|------------------|-------------|\n| DeepSeek-V3-0324 | Baseline LLM | 41.81 | 0.00 |\n| DeepSeek-V3-0324 | **EG-CFG (Ours)** | **60.6** | **32.29** |\n| DeepSeek-V3-0324 | MapCoder [Islam et al., 2024] | 50.30 | 14.59 |\n| DeepSeek-V3-0324 | CodeSim [Islam et al., 2025] | 52.72 | 18.76 |\n| GPT-4o | LPW [Lei et al., 2024]* | 34.7 | - |\n| GPT-4o | LDB [Zhong et al., 2024]*** | 29.3 | - |\n| GPT-4 | CodeSim [Islam et al., 2025] | 29.1 | - |\n| GPT-4 | MapCoder [Islam et al., 2024] | 28.5 | - |\n| GPT-4 | Self-Planning [Jiang et al., 2024]** | 10.9 | - |\n| GPT-4 | Analogical [Yasunaga et al., 2023]** | 10.9 | - |\n| GPT-3.5 Turbo | CodeSim [Islam et al., 2025] | 16.4 | - |\n| GPT-3.5 Turbo | MapCoder [Islam et al., 2024] | 12.7 | - |\n| GPT-3.5 Turbo | Analogical [Yasunaga et al., 2023]** | 7.3 | - |\n| GPT-3.5 Turbo | Self-Planning [Jiang et al., 2024]** | 6.1 | - |\n| MoTCoder-15B | MoTCoder [Li et al., 2023] | 26.34 | - |\n\n## Extending Beyond Self-Contained Functions. \nIn order to evaluate our method we used the benchmarks that we mentioned earlier. All of them indeed build upon the assumption of a single entry point “self-contained” function setting. \nAs we explicitly wrote in the limitations section (line 326) our method does not exploit a task-decomposition strategy. We believe the core idea of guiding generation via execution feedback can be extended to more complex settings of self contained function and can be combined with task-decomposition and sequential refinements approaches (lines 327-329).\nRegarding the dependency on unit-tests, our method is less dependent on test cases than the existing methods, since in most stages of the generation, it relies on implicit supervision, which is a central part of our method. The execution behavior is used as a soft feedback signal, interpreting outcomes without being explicitly told what is correct. This allows it to self-verify regardless of the quality of the test cases. We explicitly discuss this in the Introduction section (51-55) and highlight it as a key differentiator from prior approaches in the Related Work section. \nWe note in the limitation section that the method is applied to benchmarks that have executable test cases. This is the most direct way to compare with the existing relevant methods. However, it is a soft limitation since test cases can also be generated by the model itself from the task instruction.\nWe also note that in DS-1000, each problem includes only a single input-output example, making it impractical to rely heavily on such examples during code generation.\n\n\n## Questions \n\n> **Line 158 - p_inst is p_0 and p_task combined? Line 170 - where does p_sol fit into this whole process? This is referring to what is eventually regarded as the solution generated by the LM, which is then evaluated for correctness?**\n\nCorrect. p_0 is a prompt template. We incorporate the task components (text definition of the task, public tests, function name) to the prompt template in-order to form the instruction prompt p_inst. We explicitly explain about the prompt template types in lines 160-163 and include examples of these prompt templates in the appendix A (Figures 3 and 4).\nThank you for pointing this out - in our revised paper, we will clarify that and make it more explicit.\np_pre refers to the full model output in response to the instruction prompt p_inst. It includes both the intermediate reasoning and the solution block (which contains the “initial solution”). We define i_sol as the position of the last code-start token (which is defined in p_0) in p_pre, marking the beginning of the solution block. The solution prompt p_sol is constructed by taking the prefix of p_pre up to i_sol and appending the new tokens generated by our decoding algorithm, until a code-end token is generated. The tokens generated beyond i_sol form the final solution returned by our algorithm that is evaluated for correctness.\n \n> **Figure 2 – What is the highlighted “...” character referring to?**  \n\nThe “...” in Figure 2 indicates parts of the execution trace that were omitted to simplify the figure.  \nBelow is the complete version of the trace (marked with **)\n\n Invocation:\n\nfirst_non_repeating_character(\"aabc\")\n\n Execution Trace:\n\ns = 'aabc', char_count = {}\n\nchar = 'a' -> char_count = {'a': 1}\n\n**\n\nchar = 'a' -> char_count = {'a': 2}\n\nchar = 'b' -> char_count = {'a': 2, 'b': 1}\n\nchar = 'c' -> char_count = {'a': 2, 'b': 1, 'c': 1}\n\nchar = 'a' -> count = 2 → skip\n\nchar = 'a' -> count = 2 → skip\n\n**\n\nchar = 'b' -> count = 1 → return 'b'\n\n> **Is there a reason you didn’t compare against agent scaffolds such as SWE-agent, InterCode, CodeAct, or OpenHands? It feels like those works explore the inference-from-execution-across-turns approach that maybe interesting to compare against. At this point, a lot of the code completions benchmarks (e.g. MBPP, HumanEval) feel very saturated, as a strong model like the GPT o* series or Claude 3.7/4 could zero-shot get 90%+ on the benchmark. I feel like the performance gains could be more meaningful if evaluated in a more challenging setting.**\n\nAs described in the “benchmark diversity” we addressed your point regarding evaluation in more diverse and challenging settings of code generation tasks.\nRegarding the benchmark that you proposed, as we understand, these benchmarks (SWE-Agent, InterCode, CodeAct, and OpenHands) are not purely code generation benchmarks in the traditional sense. Rather than focusing solely on evaluating a model’s coding or algorithmic skills, these benchmarks assess broader agentic abilities that reflect task-solving ability in interactive environments, such as interacting with external tools or completing multi-step workflows.\nAs we wrote in the conclusion section (lines 310-313), we believe that the core idea of guiding the inference process by the execution feedback can extend to broader settings, which involve combined textual and visual feedback from interactive environments and external programs.\n\n> **What is “minimal trace” in Table 3?**  \n\n“Minimal trace” refers to a minimal execution trace that captures only the final values and types of all variables in scope after the last line of the solution function has completed execution. We will extend the current clarification in lines 288-289. \n\nThanks again for really diving into our paper. We have done our best to improve the work based on your suggestions. If you feel the updates address your concerns, we’d truly appreciate it if you considered updating your score."}, {"review_id": "FMLbwA96Dp", "reviewer": "Reviewer_GpF1", "summary": "The paper proposes an interesting program synthesis approach called **EG-CFG**, which provides the language model with real-time, line-by-line execution feedback of current variables. During inference-time decoding, EG-CFG generates the next token by aggregating the distributions from two sources: the original prompt without inline execution feedback and a dynamic prompt enriched with inline execution feedback. The method is evaluated on a programming by examples with language instructions benchmark: the MBPP benchmark. The EG-CFG method achieves state-of-the-art accuracy performance (96.6%) on MBPP, significantly outperforming other baselines using the latest proprietary LLMs.", "questions": "- How does the computational cost (e.g. API-call tokens) of the EG-CFG method compare with other baselines? If the costs differ significantly, the comparison may not be fair.\n- The MBPP dataset contains relatively simple tasks with few variables. How does the method perform when applied to problems with more variables and inputs? This could increase the complexity of inline execution feedback and potentially result in an overly long prompt.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- **Quality:** The paper introduces a novel program synthesis approach with inline program execution feedback, mimicking how humans write and debug Python code using breakpoints. This strategy effectively increases the number of tasks successfully solved.\n- **Significance:** The proposed method achieves new state-of-the-art performance on the MBPP benchmark (96.6%), and outperforms existing approaches across both 671B and 1.3B model sizes.\n- **Clarity:** The paper is well-written and easy to follow. The method is clearly explained through both natural language and mathematical formulations, and the prompt example provides a helpful illustration.\n\n**Weakness**\n\n- **Significance:** The method is only evaluated on a single programming domain, MBPP. It would strengthen the paper to include results on more diverse and challenging datasets to demonstrate broader applicability.\n- **Quality:** Program synthesis methods often benefit from increased sampling due to the ability to verify outputs against test cases. It would be helpful to include a plot showing how the method's accuracy performance scales with greater inference-time compute (e.g., more samples).", "was_revised": true, "final_justification": "I am raising my score from 4 to 5 for the following reasons:\n\n1.\tThe program synthesis approach with inline program execution feedback is novel, effective, and elegant.\n\n2.\tThe original paper only conducted experiments on a single programming domain (MBPP), but the authors have now expanded the evaluation and achieved state-of-the-art performance across all major benchmarks: MBPP (96.6%), MBPP-ET (73.0%), HumanEval (99.4%), HumanEval-ET (89.0%), CodeContests (60.6%), and DS-1000 (69.9%). This demonstrates that the method generalizes well across diverse coding tasks.\n\n3.\tThe method remains effective on complex tasks involving more variables, longer code, and limited context windows—addressing my initial concern that it may only work on simple tasks with short code.", "rebuttal": "Thank you for your encouraging and thoughtful review. We particularly appreciate your recognition of the method’s novelty and the clarity of its presentation. Your feedback has been highly motivating, and we’ve made several meaningful revisions to further improve the paper. Below, we address your comments point by point.\n## Significance: Benchmark Diversity\nWe have substantially expanded our evaluation beyond MBPP to address concerns about benchmark diversity and broader applicability.\nOur updated experiments include the additional five benchmarks:\n- **HumanEval**: A foundational and widely adopted benchmark.\n- **MBPP-ET** and **HumanEval-ET**: Extended test suites that add 100 hidden test cases per task, offering a more rigorous evaluation of generalization and robustness - helping mitigate overfitting to visible tests.\n- **CodeContests**: A challenging competitive programming benchmark that requires multi-step reasoning and broader generalization.\n- **DS-1000**: A domain-specific benchmark targeting data science tasks. In this dataset, the problem is defined without heavily relying on test cases (one sample is shown as part of the problem prompt).\n\nNotably, the benchmarks HumanEval, CodeContests, DS-1000 contain hidden test cases which help mitigate overfitting to visible tests.\n\nOur method achieves state-of-the-art performance across all major benchmarks: MBPP (96.6%), MBPP-ET (73.0%), HumanEval (99.4%), HumanEval-ET (89.0%), CodeContests (60.6%),  and DS-1000 (69.9%). \nWe compared against every strong baseline we could find, both from PapersWithCode and directly from papers. We weren’t just relying on reported results, we actively tried to reproduce methods ourselves wherever possible. In many cases, we reran existing methods on the model (DeepSeek‑V3‑0324), to ensure a fair comparison. When code didn’t work with DeepSeek or wasn’t available, we adapted or re-implemented it, and documented any limitations.\n\nThese results show that our method is not limited to MBPP. It generalizes effectively across diverse levels of complexity, from foundational problems to competitive programming tasks and domain-specific benchmarks - demonstrating the broader applicability of our method.\nDue to character limits, we have distributed the result tables across fegN and FLp2.\n\n## Effect of More Trials on Accuracy\nThank you for the suggestion. We agree that showing how accuracy scales with increased inference-time compute would provide valuable insight into the efficiency of our method. In our revised version, we will include such a plot.\n\n## Questions\n> **How does the computational cost (e.g. API-call tokens) of the EG-CFG method compare with other baselines? If the costs differ significantly, the comparison may not be fair.**\n\nFirst, native parallelization is a key strength of our method. We leverage the low price of existing APIs and the ability to parallel them, which is a growing trend among today's agents. Other baseline methods mentioned in our paper can also have access to full parallel compute, but by design, they cannot fully leverage it. They rely on sequential retries, where each iteration depends on the output of the previous one. This makes it impossible for them to parallelize inference on a single task, even if the computational resources are available. \nMoreover, in (5/6) of the benchmarks: HumanEval, HumanEval-ET, MBPP-ET, CodeContests, DS-1000 there are hidden-test cases. There are hidden test cases. This means our method cannot rely on extensive token usage to overfit the public tests in order to succeed.\nAdditionally, In order to rigorously make sure the comparison is fair, we ran trials using the DeepSeek-V3 model with two baseline methods: MGDebugger and MapCoder, on the MBPP benchmark, using a much higher number of retries which leads to significantly more extensive token usage. For MGDebugger, we increased the retry count from 5 to 200 - x40 increase, resulting in ~x40 tokens. This resulted in only an improvement of 34 out of 500 (6.8 percent), out of the 86 examples it originally got wrong. For MapCoder, we similarly increased retries from 5 to 200 (40x tokens), and saw an improvement of just 8 out of 500 (1.6 percent), out of 64 examples it originally failed.\nIn both cases, the improvement was achieved with a 40x increase in runtime and is still considerably below our performance on MBPP.\nThis supports our conclusion that the advantage of our method is not due to higher token usage, and therefore the comparison is fair. In order to clarify this issue more we will add discussion about this point in our revised version.\n\n> **The MBPP dataset contains relatively simple tasks with few variables. How does the method perform when applied to problems with more variables and inputs? This could increase the complexity of inline execution feedback and potentially result in an overly long prompt.**\n\nOur method is completely agnostic to the number of variables in the input or whether there are variables at all in the input. This is since it is agnostic to the format of the execution feedback (line 196). It relies on implicit supervision, which is an important part of our novelty, as we treat execution feedback as a soft guidance signal. The model interprets execution behavior on its own, without being explicitly told what is correct or relying on any specific format. This allows it to self-verify regardless of the quality of the test cases. \nMoreover, we believe that in many cases, it's actually easier for the model to self-verify logic when it is broken down into smaller, more atomic steps. To encourage this, we introduced the “long-code” prompt template (see Figure 4), which encourages the model to produce longer, step-by-step solutions with simpler lines of code.\nRegarding rigorous evaluation, we have addressed your point in “Significance: Benchmark Diversity”. The benchmarks CodeContests and DS-1000 are indeed more complex, with many more inputs resulting in longer prompts, and our method still outperforms all known results on these benchmarks.\n\nWe sincerely appreciate the time and care you put into your review. Your feedback helped us improve the clarity and depth of the paper. If you feel the changes address your concerns, we would be grateful if you consider updating your score. Thank you again for your thoughtful input."}]

Author rebuttal:

Meta-review: This paper proposes Execution-Guided Classifier-Free Guidance (EG-CFG), an inference-time decoding method for code generation that integrates real-time execution feedback into the generation process. The approach is conceptually novel, lightweight, and does not require additional training. It is inspired by how human programmers iteratively run partial code to guide development and achieves state-of-the-art performance on MBPP.

The method is original and clearly presented, combining execution feedback with classifier-free guidance in a simple, reproducible manner. Reviewers appreciated the paper’s clarity and the ablations, which substantiate the contributions of different components. Empirically, the approach demonstrates significant performance improvements, outperforming strong baselines.

The main concern is the limited evaluation scope. Experiments are restricted to MBPP, which lacks hidden test cases and is somewhat dated. This raises questions about generalizability to other benchmarks (e.g., LiveCodeBench, HumanEval, BigCodeBench) and to real-world programming tasks beyond Python. Some reviewers also noted that computational overhead and certain procedural details could be discussed more explicitly.

Despite these limitations, the paper presents a creative and impactful contribution. I lean toward acceptance, as the work is timely, methodologically sound, and likely to stimulate further research in inference-time program synthesis.
Decision: Accept (poster)