Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step (2024)

Abstract: Large language models (LLMs) are leading significant progress in code generation. Beyond one-pass code generation, recent works further integrate unit tests and program verifiers into LLMs to iteratively refine the generated programs. However, these works consider the generated program as an indivisible entity, which falls short for LLMs in debugging the programs, especially when the programs contain complex logic flows and data operations. In contrast, when human developers debug programs, they typically set breakpoints and selectively examine runtime execution information. The execution flow and the intermediate variables play a crucial role in the debugging process, yet they are underutilized in the existing literature on code generation. In this study, we introduce Large Language Model Debugger (LDB), a novel debugging framework that enables LLMs to refine their generated programs with the runtime execution information. Specifically, LDB segments programs into basic blocks and tracks the values of intermediate variables after each block throughout runtime execution. This allows LLMs to concentrate on simpler code units within the overall execution flow, verify their correctness against the task description block by block, and effectively pinpoint any potential errors. Experiments demonstrate that LDB consistently enhances the baseline performance by up to 9.8% across the HumanEval, MBPP, and TransCoder benchmarks, archiving new state-of-the-art performance in code debugging for various LLM selections.

heading: 1 Introduction
text: Code generation is a critical yet challenging task that has various downstream applications, such as text-to-code generation (Chen et al., 2021; Yin and Neubig, 2017; Li et al., 2022), code translation (Roziere et al., 2020), and code autocomple-
†Corresponding authors.
tion (Li et al., 2018; Raychev et al., 2014). Recent progress in large language models (LLMs) (Li et al., 2023; Roziere et al., 2023; Achiam et al., 2023; Zhou et al., 2023a; Muennighoff et al., 2023) significantly boosts the performance of code generation and demonstrates a promising potential to be generally applied in different requirements and tasks (Shinn et al., 2023; Gu, 2023; Yuan et al., 2023). However, generating correct programs is not a one-time effort. Existing works suggest enhancing code generation through multiple sampling (Zhang et al., 2023b; Shinn et al., 2023), selfconsistency (Le et al., 2023; Huang et al., 2023a; Chen et al., 2022), and candidates ranking (Shi et al., 2022; Ni et al., 2023; Zhang et al., 2023a). Despite these advanced approaches, they still fall short on basic programming questions from the HumanEval and MBPP datasets. This underscores the limitations of single-pass program generation.
Recognizing this, a series of works have been proposed to refine the programs generated in a single pass, based on feedback from either human annotator (Chen et al., 2023a; Wu et al., 2023) or
851
LLMs themselves (Tang et al., 2023; Chen et al., 2023c). This refinement process is akin to debugging in programming practices by human developers. Chen et al. (2023c); Jiang et al. (2023) introduce unit test results and error messages to LLMs. These approaches allow LLMs to reflect on potential mistakes and generate corrected programs. Nevertheless, considering the debugging process by human developers, it is sub-optimal to solely depend on these post-execution information to debug the program, especially in cases involving complex data structures and control flows. In fact, when human developers encounter a buggy program, they do more than just collect the program’s outputs. They delve into the runtime execution to observe the execution traces1 and examine the intermediate variables by setting breakpoints. When the intermediate execution states deviate from their intention, developers pinpoint the bugs and make the corrections. This is a common workflow for well-known interactive debuggers such as GDB (Stallman et al., 1988) and PDB (Foundation, 2001).
To this end, we propose LDB, a large language model debugger that refines programs generated by LLMs using runtime execution information, emulating the debugging practices of human developers. As shown in Figure 1, feeding in a visible test case, LDB segments the execution trace into
1Refer to Appendix B for detailed introduction of execution trace, control flow graph, and basic block.
basic blocks1 based on the control flow graph1. LDB tracks the intermediate variables at the end of each basic block, similar to the breakpoints set by developers. After gathering runtime execution information, LDB queries LLMs for verdicts on each code block’s correctness and explanations of the execution flow in relation to the coding task. This approach allows language models to concentrate on simpler code units, verify intermediate states against the task description, and pinpoint potential bugs. Consequently, it effectively debugs the program and improves the quality of code generation.
We validate LDB on three code generation benchmarks, including HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) for textto-code generation, and TransCoder (Roziere et al., 2020) for code translation. We conduct experiments using the proprietary model, GPT3.5 (Achiam et al., 2023), and the open-sourced models, StarCoder (Li et al., 2023) and CodeLlama (Roziere et al., 2023). Experiments demonstrate that LDB consistently improves code generation accuracy across various LLM backbones and achieves state-of-the-art performance in debugging programs. Worth mentioning, even with the programs generated by more powerful code generators, such as GPT-4 (Achiam et al., 2023) and Reflexion (Shinn et al., 2023), LDB can still detect errors overlooked by previous advanced methods, thereby enhancing the capabilities of code generation even
further. We summarize our contribution as follows: • We propose a large language model debugger,
LDB, which takes the very first step on incorporating runtime execution information into LLMs to debug generated programs. • We leverage basic blocks in the execution traces to properly segment the programs into smaller, simpler code units, allowing LLMs to verify them step by step against users’ intention and effectively pinpoint the potential bugs. • Extensive experiments on three code generation benchmarks demonstrate the effectiveness of LDB in debugging generated programs across different LLM backbones. Reproducibility. The code will be released on Github2.

heading: 2 Large Language Model Debugger
text: Problem Formulation. We follow the problem formulation of program debugging in Chen et al. (2023c). In a code generation task, each sample can be represented as a triplet (Q,Tv, Th), where Q represents the task description, Tv and Th are visible and hidden test cases. In the text-to-code generation task, Q consists of a brief paragraph in natural language that outlines the intended goal of the task. In the code translation task, Q represents a program written in a language other than the target language, and the objective is to translate this program into the target language. A code generator is first provided with Q and Tv to generate the seed program A0. In the debugging stage, a debugger further refines A0 and finally outputs a new program A∗. A∗ will be tested using the hidden tests Th to evaluate whether it is correct or not. Th is only used in the evaluation stage and not visible during the code generation or debugging stages.

heading: 2.1 Overview
text: We show the workflow of LDB in Figure 2. We prompt a LLM to generate seed programs. This serves as the starting point of debugging if they fail any of the visible test cases. During debugging, LDB feeds in the failed visible test case to the seed program and collects the runtime execution information, including the execution trace and the runtime variable values after each basic block (Section 2.2). Then, LDB queries a LLM to verify the correctness of basic blocks step by step via comparing them with the task description (Section 2.3).
2https://github.com/FloridSleeves/LLMDebugger
Considering the block-wise runtime execution and the task description, LDB queries the LLM again to detect buggy blocks and regenerate the program. LDB repeats these three steps iteratively until the new solution pass all the visible tests, or the maximum debugging iteration is reached (Section 2.4).

heading: 2.2 Profiling
text: In the step of profiling, LDB collects the runtime execution information when running the generated program over the failed visible test cases. It collects the execution trace and then segments the trace into basic blocks to inspect intermediate variables.
Execution Traces. In static analysis, each program corresponds to a unique control flow graph (CFG) where each node in the graph is a code basic block, as shown in Figure 2. Each basic block is a straight-line sequence of code with only one entry point and one exit. The CFG represents all paths might be traversed through a program during its execution. After LDB feeding in a visible test case, the control flow goes through a sequence of basic blocks and the path is denoted as the execution trace, [B1, B2, . . . , Bn], where Bi is a basic block in the trace and n is the length of the trace.
Intermediate States. Given an execution trace [B1, B2, B3, . . . , Bn], we execute the first i blocks and collect all the variables in the scope along with their runtime values. We denote the state set as Vi = {v = v̂|v ∈ ⋃ B≤i}, where v is a variable used in the first i blocks and v̂ is its runtime value after the i-th block. We define the intermediate state after the first i blocks as (Vi−1, Bi, Vi), where Vi−1 represents the entry states of the block, Bi is the current code block to execute, and Vi provides the actual execution results after Bi.

heading: 2.3 Debugging
text: The block-wise intermediate states determined by profiling provide a comprehensive illustration for the runtime execution. In Debugging, we integrate the intermediate states into prompts and query a LLM to verify whether the basic blocks align with the intended semantics in the task description Q.
Debugging Verdicts. For each intermediate state throughout the trace, (Vi−1, Bi, Vi), the LLM is acquired to make a verdict on its correctness Di ∈ {True, False}, and elaborate the explanation Ei. If the LLM detects any buggy code block, LDB includes the message in the debugging response.
Selective Debugging. Loops and recursion are common in programming, potentially leading to extensive execution traces. If we include the lengthy traces directly in the prompt, it is highly likely to exceed the maximum token limit of LLMs. This is similar to what occurs with humans in program development. When the execution is lengthy, developers may only examine a few blocks and skip the other long and tedious execution traces when detecting bugs. Inspired by the human practice, LDB selectively samples Nb blocks from program traces to ensure the total length of runtime information within the max token limit of LLMs.
Batch Debugging In our proposed LDB, the intermediate states following each basic block are determined during the execution of the seed program with test cases. Thus, LDB can batch these states together and query the LLM for debugging verdicts. This significantly improves the token efficiency of LDB and alleviates the pitfall of repeatedly sending lengthy context to LLMs in iterative refinement (Ge et al., 2023; Hu et al., 2023). Specifically, the batch debugging query process is as follows,
{V0, B1, V1, B2, ..., Bn, Vn} LLM−−−→{(D1, E1), ..., (Dn, En)}
where Vi is the set of variables and their runtime values after the i-th blocks, Bi is the i-th block in the trace, Di is the debugging verdict from the LLM, and Ei is the corresponding explanation.

heading: 2.4 Regeneration
text: The runtime execution information helps accurately localize buggy code blocks, allowing LLMs to concentrate specifically on these areas during the regeneration process. In Regeneration, LDB collects the debugging verdicts D and explanations E, and incorporate them along with the task description Q into the prompt. Then, LDB queries the LLM again to generate the refined program. LDB iteratively runs Profiling, Debugging, and Regeneration, until the refined program passes all visible test cases, or the maximum debugging iteration is reached. We test the finalized solution A∗ using the hidden test cases Th to evaluate the performance.

heading: 3 Experiments
text: We evaluate LDB on three code generation benchmarks: HumanEval (Chen et al., 2021), TransCoder (Roziere et al., 2020), and
MBPP (Austin et al., 2021). HumanEval and MBPP are for text-to-code generation, where the task description is a brief passage outlines the intended functionality of the program to be generated. TransCoder is for code translation which requires to translate a program from C++ into Python. The task description of TransCoder consists of a C++ program to be translated. We compute Pass@1 accuracy with hidden test cases for assesment. We conduct experiments with the proprietary LLM, GPT-3.5 (turbo-0613) (Achiam et al., 2023), and the open-source LLMs, CodeLlama (34B-Instruct) (Roziere et al., 2023) and StarCoder (∼15B) (Li et al., 2023) as backbones.

heading: 3.1 Experiment Setup
text: We generate the seed programs following the same prompts and generation parameters used in our compared method Chen et al. (2023c). We set the maximum number of debugging iterations as 10. More detailed implementation details are reported in Appendix C. To obtain visible test cases for HumanEval, we extract the given visible test cases from the task description. For MBPP, we use the first test case of each problem as the visible test case and use the rest as hidden test cases. For TransCoder, we include all test cases from the dataset as visible test cases3. The experiment settings on MBPP and TransCoder are the same as the prior works (Chen et al., 2023c; Shi et al., 2022; Ni et al., 2023). After finalizing the code solution, we compute the Pass@1 accuracy with hidden test cases to evaluate the performance.

heading: 3.2 Compared Methods
text: We evaluate the seed programs and label the performance as Baseline (w/o debugger). We compare LDB against two rubber duck debugging methods from Chen et al. (2023c): Self-Debugging (+Expl.) which prompts LLMs to explain generated programs line-by-line as feedback, and SelfDebugging (+Trace) which prompts LLMs to dry run generated programs as feedback.
We reproduce the Self-Debugging methods following the instructions in Chen et al. (2023c) due to the unavailability of open-source code. When referring to “Self-Debugging”, we default to the method with higher accuracy among the two methods unless otherwise specified. Throughout the
3All test cases can be generated by running the original C++ programs, which are visible to LLMs.
evaluation process, we ensure that all debugging methods utilize the same LLM settings, visible test cases, seed programs, and prompts formats. This ensures a fair comparison and eliminates potential disruptions caused by changes in prompt formats.

heading: 3.3 Main Results
text: We compare LDB with the baseline debugging methods on HumanEval, TransCoder, and MBPP, and present the result in Table 1. We observe that LDB consistently achieves improvements of up to 9.8% on all datasets across different LLM backbones. Specifically, compared to Self-Debugging which prompts LLMs to dry run or explain the program, LDB achieves higher and more stable performance gain over the baseline by introducing the actual runtime execution information.
We attribute the advantage of LDB to the finegrained debugging feedback and the runtime information as external supplements to the LLM selfcorrection. The detailed block-level debugging responses help LLMs concentrate on the buggy areas in the program to better align the program with the task description. Moreover, as pointed out by Huang et al. (2023c), LLMs has limited self-correct reasoning abilities. Particularly in code generation, LLMs are prone to mistakes when reflecting on program execution. This is due to their inability to accurately calculate concrete variable values and predicting execution flow at branches or loops. The inaccurate feedback from LLMs could
misguide the program debugging and refinement, which explains why Self-Debugging (+Expl.) and Self-Debugging (+Trace) fail to improve the seed programs on HumanEval with StarCoder. On the contrary, LDB generates the debugging verdicts and explanations based on accurate intermediate values and execution flows, guiding the generated programs towards the correct answer.
Worth mentioning, the visible test cases are also provided to the LLM during the initial seed program generation. However, their utility in code generation is limited, as evidenced by the suboptimal performance of Baseline (w/o debugger). This aligns with our assumption that actual runtime execution information significantly helps LLMs ground their reasoning, thereby improving their ability to generate better semantically aligned code.

heading: 3.4 Results on Advanced Code Generators
text: To further demonstrate the effectiveness of LDB, we apply LDB and Self-Debugging to debug the seed programs from advanced code generators, GPT-4 (Achiam et al., 2023) and Reflexion (Shinn et al., 2023). We conduct the analysis on HumanEval as an example. We query GPT4 (1106-preview) to generate seed programs in the same setting introduced in Section 3.1. As for Reflexion, we utilize the corresponding generated programs published in the official Github repository4 as the seed programs. GPT-4 and Reflexion are considered as more powerful code generators that already achieve superior performance without any debuggers.
The results are shown in Table 3. The LLM backbone of Self-Debugging and LDB is GPT3.5 (turbo-0613), which is weaker than the code generators, GPT-4 and Reflexion. We list the performance of GPT-3.5 in the table for reference. Despite the weaker LLM backbone, both SelfDebugging and LDB can refine the programs in the debugging process. This highlights the advantage of introducing a debugging stage in code generation with LLMs. Furthermore, LDB surpasses SelfDebugging in debugging and refining programs. It can improve performance on HumanEval even further and achieve a new state-of-the-art result (95.1%) in code generation by debugging the seed programs from Reflexion. This indicates that LDB is able to examine the runtime execution and correct bugs overlooked by the advanced code generators, serving as an orthogonal supplement to current code generation techniques.

heading: 3.5 Performance vs. Debugging Iterations
text: In Figure 3, we plot the performance of LDB, Repeated Sampling, Self-Debugging (+Expl.), and Self-Debugging (+Trace) across each iteration on HumanEval using GPT-3.5. We introduce Repeated Sampling as a straightforward comparison method, where we repeatedly sample coding solutions from the program generator until the solution passes the visible test. The performance at each iteration is computed in the same way as LDB. We run these methods up to 10 iterations to examine the performance tendency. We show the performance of LDB across 20 iterations in Appendix 7 to explore the continuous improvement trend.
4https://github.com/noahshinn/reflexion
Continuous Debugging Potential of LDB In Figure 3, with increasing debugging or resampling rounds, all methods refine the seed program and improve the performance. Particularly, LDB continuously improves the performance across the debugging iterations and achieves the best debugging performance despite the slightly slow rising speed. In contrast, Self-Debugging nearly stops improving the performance after 2 iterations, as also observed in Chen et al. (2023c).
Necessity of Runtime Information. From Figure 3, we observe that the performance of SelfDebugging presents a similar trend to Repeated Sampling after 3 rounds. They both stops effectively improving the performance at an early stage (around 2 ∼ 3 iterations). This phenomenon reveals a fundamental difference between LDB and Self-Debugging. We attribute it to the limited selfcorrecting ability of LLMs, as pointed out in Huang et al. (2023c). The feedback mechanisms in SelfDebugging (self-explaining and self-tracing) in fact enhance the initial understanding of coding tasks but fail to align the task to the specific code. Therefore, the debugging performance of these methods quickly converges and then hardly improves even given more rounds of debugging. In the contrast, LDB exhibits a continuing improvement with the growth of debugging iterations. The new information from runtime execution keeps moving the models towards correct programs, which closely resembles the human debugging process.

heading: 3.6 Different Decomposition Levels of LDB
text: In Profiling (Section 2.2), we segment the runtime trace into basic blocks based on the control flow graph. A basic block only has one entry and one exit in the program execution, serving as an ideal basic unit in the runtime analysis (Sherwood et al., 2001). To explore the effectiveness and efficiency of block-level decomposition, we develop two comparative methods, LDB (line-level) and LDB (function-level), which segment the runtime trace in the granularity of lines and functions respectively. We denote the original block-level debugging method as LDB (block-level). LDB in different decomposition levels share the same architecture. They collect runtime intermediate states at the end of each code unit. The code unit can be a line in LDB (line-level), a block in LDB (blocklevel), or a function in LDB (function-level). Similarly, we adopt Selective Debugging (Section 2.3) to fit prompts into the token limits of LLMs. Please refer to Appendix E for implementation details.
Figure 4 plots the performance and average token cost per program of LDB in different decomposition levels on HumanEval using GPT-3.5. The detailed statistics are listed in Table 6 in Appendix. All three debugging methods manage to enhance the performance, demonstrating the benefits from runtime execution information. Particularly, among the three decomposition levels, LDB (block-level) achieves the highest improvement.
LDB (line-level) performs worse than LDB (block-level) even if the line-level information is more fine-grained. This may arise because linelevel decomposition leads to incomplete semantics in each code unit (i.e. a line of code). As a result,
LLMs struggles to fully understand the code units and accurately identify bugs within the program.
LDB (function-level) provides the most coarsegrained information which largely preserves the complete programs. However, the intermediate states in the function-level fail to provide detailed runtime information. Consequently, LDB (function-level) is less effective and requires approximately 8.1 iterations on average to debug a program which are much more than the other levels (see Appendix E for statistics). This explains the highest token cost of LDB (function-level).

heading: 3.7 Performance of Different Difficulty Levels
text: To evaluate the capability of LDB in debugging programs, we dive into the problems it successfully debugs in TransCoder and categorize them into three difficulty levels, Easy, Medium, and Hard. The difficulty annotation is automatically performed by GPT-4 (1106-preview) based on the canonical solutions of each problem to avoid potential subjectiveness. Figure 5 shows the improved accuracy for each difficulty level from TransCoder dataset. We observe that the performance of GPT-3.5 in code generation decreases with the increasing problem difficulty while the improvement from the debugger increases in harder problems. Particularly, LDB shows the most improvement (9.1%) on the hardlevel problems, which indicates that LDB is able to detect the non-trivial bugs and understand the complex execution flows in the harder problems.

heading: 3.8 Case Study
text: Figure 6 presents an example on HumanEval with GPT-3.5. In the example, LDB successfully fix the
the program and enable it to pass the visible and hidden test cases. This case requires the program to check two conditions: (1) numbers sorted in ascending order. (2) list does not have more than 1 duplicate of the same number. In the seed program, it checks the first condition, while mistakenly presents the second condition as not having any duplicates (lst.count(x) > 1). In BLOCK-0 to BLOCK-4, LDB makes the verdicts that each block correctly checks the first condition. For BLOCK5, LDB finds out the mistake in the condition and locates the program bugs exactly in this block. It proposes to fix this condition into lst.count(x) > 2 so that it aligns with the task description.

heading: 4 Related Work
text: Augmented Code Language Models Recent language models based on deep neural networks (Achiam et al., 2023; Roziere et al., 2023; Li et al., 2023; Nijkamp et al., 2022) demonstrate great potential for coding tasks. Despite their impressive capabilities, these models face challenges such as syntax correctness (Jin et al., 2023; Chen et al., 2023b), semantic alignment (Ni et al., 2023; Fan et al., 2023), code reliability (Zhong and Wang, 2023), and conflict merging (Zhang et al., 2022c). To address these challenges, some focus on enhancing initial code generation by leveraging multiple candidates (Shinn et al., 2023; Zhou et al., 2023a; Gu, 2023) or refining solutions based on better test cases (Zhang et al., 2023a) and selfconsistency (Chen et al., 2022; Le et al., 2023), while others train verifiers on execution results to predict solution quality (Ni et al., 2023). In contrast,
LDB enhances code generation without expanding sampling numbers complementing existing methods, while these existing code generation methods could provide better seeds for our debugger.
Feedback-based Code Refinement Generating correct solutions could require iterative refinement due to model limitations. Interactive methods like using human feedback (Chen et al., 2023a; Wu et al., 2023) are effective but labor-intensive. Alternatively, code refinement techniques (Chen et al., 2023c; Jiang et al., 2023; Hu et al., 2024) based on language models have been proposed, utilizing interpreter outputs, self-generated explanation, and other users’ submission in competition-level programming (Zhang et al., 2022b). Some train additional models for bug fixing (Pearce et al., 2023; Huang et al., 2023d; Gupta et al., 2023), while LDB utilizes debugging capabilities of original large language models. LDB follow an similar iterative refinement paradigm as previous works (Madaan et al., 2023; Zhou et al., 2023b). It leverages execution results for debugging inspired by previous works on execution-guided code generation (Chen et al., 2018; Ni et al., 2022). Additionally, agent frameworks using reinforcement learning for coding tasks (Shinn et al., 2023; Zhou et al., 2023a; Hong et al., 2023; Rasheed et al., 2024; Le et al., 2022) incorporate feedback from environments to guide actions. While these frameworks excel in searching code generation space, LDB focuses on code refinement and consistently improves performance across various initial code.
Decomposition in Reasoning Prompting methods suggest that decomposing problems aids
large language models in reasoning tasks (Wei et al., 2022; Zhou et al., 2022; Lightman et al., 2023; Dhuliawala et al., 2023; Wang et al., 2024; Cheng et al., 2022). (Zhang et al., 2022a) proposes program trunking to improve the program fix rate. Inspired by this, LDB decomposes programs into blocks, querying language models for debugging verdicts and explanation. LDB further introduces batch debugging to improve the efficiency.

heading: 5 Conclusion
text: We present LDB, a debugging framework that help LLMs refine generated programs with runtime execution information. We empirically show that LDB significantly improves code generation accuracy and achieves state-of-the-art performance in program debugging, by segmenting the programs into basic blocks and tracking the intermediate values. Experiments also reveal its unique paradigm of program debugging by using runtime information.
Limitation
LDB is a program debugging framework using large language models. Therefore, it is subjected to the limitation of existing debugging methods of human developers. The correct test cases are mandatory in LDB so that LDB can execute the program and compare the execution flow against the task description. It remains an open question in future study whether LLMs are able to do self-correct by simply looking at its intermediate execution without knowing whether the result is correct or not (a.k.a. test-case-free debugging).
Ethic Statements
This paper focuses on debugging code generated by large language models. The architecture are built upon open-source models and publicly available proprietary models. All the datasets in this paper are available online. We did not hire any human annotators in our experiments. We will release the code and datasets on https://github.com/ FloridSleeves/LLMDebugger. Therefore, we do not anticipate any major ethical concerns.

heading: Acknowledgement
text: Our work is sponsored in part by NSF CAREER Award 2239440, NSF Proto-OKN Award 2333790, as well as generous gifts from Google, Adobe, and Teradata. Any opinions, findings, and conclusions
or recommendations expressed herein are those of the authors and should not be interpreted as necessarily representing the views, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for government purposes not withstanding any copyright annotation hereon.

heading: A Performance with Better Reasoners
text: We explore the relationship between performance and the ability of debugging backbones. With better reasoners as debugging backbones, LDB achieves higher performance, which can be up to 98.2% on HumanEval with Reflexion as the seed generators and GPT-4o (2024-05-13) as the backbones.
Reflexion +LDB (GPT-3.5) +LDB (GPT-4) +LDB (GPT-4o)

heading: B Programming Language Concepts
text: Basic Block. A basic block is defined as a linear sequence of code containing a single entry point and a single exit point (Allen, 1970; Aho and Ullman, 2015). Upon executing the first instruction within a basic block, all subsequent instructions are guaranteed to execute exactly once and in sequential order. A sequence of instructions is considered a basic block if it satisfies two conditions: (1) each instruction in the sequence always executes before all subsequent instructions, and (2) there are no intervening instructions between any two instructions in the sequence (Cocke, 1970; Allen, 1970).
Control Flow Graph. The control-flow graph (CFG) (Allen, 1970; Prosser, 1959) serves as a graphical depiction of all potential paths traversed during the execution of a program. Each node within the CFG corresponds to a basic block, with directed edges representing transitions in the control flow. Typically, two special blocks are identified: the entry block, which signifies the initiation of control flow into the graph, and the exit block, where all control flow exits the graph.
Execution Traces. In this paper, execution traces are control-flow traces of the whole program (Larus, 1999). An control-flow trace of a program is a sequence of consecutively executed basic blocks within the program. It also corresponds to a path in the control flow graph from the
entry block to the exit block (Ball and Larus, 1994, 1996; Ammons and Larus, 1998).
C Implementation Details
In the debugging stage of LDB, we generate the debugging verdicts and explanation using greedy decoding with temperature T = 0 to improve the reproducibility of our experiment. The maximum number of debugging iterations is 10. We set the threshold for the number of sampled blocks and input tokens at 10 and 3,097, respectively.

heading: D Overhead Breakdown
text: The time cost of LDB is comparable to other baseline methods and all debugging steps of LDB are fully automated without any human labor, as shown in Table 4. We summarize the standard paradigm of iterative refinement methods as follows: (1) Execute the buggy program with a Python interpreter. (2) Query a LLM to generate debugging feedback. (3) Query a LLM again to regenerate a program based on the feedback. LDB, along with the baseline methods Self-Debugging all follows this paradigm. We compare the time cost of LDB and Self-Debugging in each stage on HumanEval with GPT-3.5. Compared with the baseline method, the additional time cost of LDB comes from the profiling step (Section 2.2). This step is performed by program analysis and only costs 0.09 seconds on average for programs in HumanEval. The overhead turns out to be negligible under the fluctuation of other overhead while contributing to a significant improvement in the performance as shown in Table 1 and Figure 3.

heading: E Tradeoffs in Debugging in Different Decomposition Levels
text: We conduct different level debugging following this design: (1) For LDB (line-level) debugging, we collect the intermediate states before and after each line execution. We sample the first 25 lines and last 25 lines when the line number exceeds threshold Nb = 50, which is five times of the block-level threshold. We set this number based on the previous research on the average number of instructions in a basic block (Rotenberg et al., 1996). (2) For the original LDB (block-level) debugging, we sample the first 5 blocks and the last 5 blocks when the block number exceeds threshold Nb = 10. (3) For LDB (function-level) debugging, we decompose the program on function-level,
namely we only collect the intermediate states at the entry and exit of the solution function. If the function trace exceeds the context length, we sample first 25 lines and last 25 lines to ensure same amount of code trace information with block-level and line-level. These three level of decomposition expand from fine to coarse granularity.
We show the average token cost per program and debugging turns of LDB with different granularity debugging levels in Table 5 and Table 6. Using GPT-3.5, LDB (line-level) has less token cost than LDB-Function due to less debugging turns. However, using CodeLlama, an open source model, LDB has higher token cost in the line-level debugging than the function-level debugging. The debugging turns of line-level debugging is not significantly lower than function-level debugging, which shows that CodeLlama has worse reasoning ability for line-level debugging even with more runtime execution information. Besides, line-level debugging has higher token costs for each debugging turns. Therefore, it has the highest token cost. Both GPT3.5 and CodeLlama demonstrate better efficiency and accuracy in the block-level debugging. Based on these observations, we choose block-level debugging in LDB.

heading: F Complexity Analysis of Batch Debugging
text: LDB can batch runtime information of all selected blocks together and query language models for debugging verdicts. This significantly improves the
HumanEval GPT-3.5 Avg. Turn CodeLlama Avg. Turn
token efficiency of LDB and alleviates the pitfall of repeatedly sending context to language models in iterative refinement (Hu et al., 2023). Assume the average token numbers is N for debugging each block, average block number is B for debugging, and average debugging iteration number is D. Without batch debugging, for debugging i-th block, the context length for debugging is i∗N . Therefore, debugging N blocks consumes ∑Bi=0 (i ∗ N ), approximately O(B2 ∗ N ) tokens. As a comparison, batch debugging only send one debugging message in each turn, which has context length of O(B ∗N ) tokens.

heading: G Debugging 10 More Iterations
text: As LDB shows a different trend of performance regarding debugging turns compared to the stateof-the-art methods, it is interesting to see whether
the performance continues growing after 10 debugging turns. We conduct a experiment that continue debugging until 20 debugging turns to explore the characteristic of LDB, as shown in Figure 7. The accuracy still grows after 10 debugging turns and achieves 84.1%.

heading: H Error Analysis of LDB
text: We analyze the debugging feedback of LDB in two perspectives, the bug localization accuracy and the bug categorization. We evaluate the performance of LDB in localizing the bug accurately and further specify the types of bugs detected by LDB. We use the results with GPT-3.5 as an example.
Accuracy of Bug Localization. To evaluate the correctness of debugging verdicts made by LDB, we collect the cases that LDB successfully debugs and query GPT-4 (1106-Preview) to automatically verify whether LDB correctly identifies the buggy basic block. In the prompt, we provide the task description, the seed program, the runtime information, debugging responses, and the final programs for GPT-4. Table 7 shows the analysis results on HumanEval, MBPP and TransCoder. We observe that LDB achieves an accuracy over 85% over all datasets. It has the highest accuracy on MBPP (95.5%) while performs worst on TransCoder (86.7%). This may arise because TransCoder consists of more complicated programs where the original C++ programs outlines the intention implicitly.
Bug Categorization. We further categorize the bugs detected and fixed by LDB into two categories: Syntax Errors and Semantic Errors. Syntax errors refer to the cases that violates the syntax rules of its programming language. Semantic error occurs when the code runs without producing any syntax errors, but it does not behave as intended, thus fails the test cases. We annotate the categories through GPT-4 (1106-preview) to avoid subjectiveness. Across three datasets, LDB fixes semantic errors with around 80% of the time. The reason
is that the syntax errors are less frequent given the code generation ability of current LLMs, and LDB can better debug semantic errors by introducing the detailed runtime execution information.

heading: I Extendability of LDB
text: In this paper, we have chosen Python for LDB due to its widespread popularity in software development (Huang et al., 2021). This aligns with previous studies such as Self-Debugging (Chen et al., 2023c), Reflexion (Shinn et al., 2023), and Gorilla (Patil et al., 2023), which also utilize Python to showcase the effectiveness of their methodologies.
We further elaborated on LDB’s capability to adapt to other programming languages. Importantly, LDB does not use features that are specific to Python. This makes it easy to adapt LDB for use with other programming languages by changing its program executors and analysis tools (Zhong, 2023; Huang et al., 2023b; Zhong et al., 2024). In the debugging process of LDB, the backbone LLM debugs programs based on the detailed runtime execution information, such as execution traces and intermediate variables. This runtime execution information can be similarly obtained in all different kinds of programming languages. We can use offthe-shell program analysis tools to instrument the targeted language and acquire the runtime execution information. Incorporating multilingual experiment settings, such as the HumanEval-X, remains as a future work.

heading: J Prompt Example in LDB
text: We list the prompts used in LDB in detail as follow, • The prompts used in the chat mode are shown in
Figure 8. • The prompts used in the text completion mode
are shown in Figure 9.
==================================== System ==================================== You are an expert programming assistant.
===================================== User ===================================== Complete the following task in Python. Please respond with code only. def is_sorted(lst):
''' Given a list of numbers , return whether or not they are sorted in ascending order. If list has more than 1 duplicate of the same number , return False. Assume no negative numbers and only integers.
Examples is_sorted ([5]) = True is_sorted ([1, 2, 3, 4, 5]) = True is_sorted ([1, 2, 3, 4, 5, 6, 7]) = True is_sorted ([1, 2, 2, 2, 3, 4]) = False '''
=================================== Assistant ================================== for i in range(len(lst) - 1):
if lst[i] > lst[i + 1]: return False
return not any(lst.count(x) > 1 for x in lst)
===================================== User ===================================== The code above fails the given unit test: assert is_sorted ([1, 2, 2, 3, 3, 4]) == True # Real Execution Output: False. Help me debug this.
Here is the code execution trace block by block with the intermediate variable values to debug your code. You need to answer FOR EACH BLOCK whether this code block is correct or not. If not , give an explanation on what is wrong. Please wrap your response into a JSON object that contains keys `block ` with the name of each block , key `correct ` with value False or True , and key `explanation ` with an explanation on the bug.
Example Answers: {" block ": "BLOCK -1", "correct ": "True", "explanation ": "The block initializes variable `a` and `b`."} {" block ": "BLOCK -2", "correct ": "False", "explanation ": "The block is incorrect because the code does not add the two integers together , but instead subtracts the second integer from the first. To fix this issue , we should change the operator from `-` to `+` in the return statement. This will ensure that the function returns the correct output for the given input ."}
[BLOCK -0] # lst=[1, 2, 2, 3, 3, 4] for i in range(len(lst) - 1):
if lst[i] > lst[i + 1]: # i=0 lst=[1, 2, 2, 3, 3, 4]
[BLOCK -1] # i=0 lst=[1, 2, 2, 3, 3, 4] for i in range(len(lst) - 1):
if lst[i] > lst[i + 1]: # i=1 lst=[1, 2, 2, 3, 3, 4]
[BLOCK -2] # i=1 lst=[1, 2, 2, 3, 3, 4] for i in range(len(lst) - 1):
if lst[i] > lst[i + 1]: # i=2 lst=[1, 2, 2, 3, 3, 4]
[BLOCK -3] # i=2 lst=[1, 2, 2, 3, 3, 4] for i in range(len(lst) - 1):
if lst[i] > lst[i + 1]: # i=3 lst=[1, 2, 2, 3, 3, 4]
# Write Python function to complete the task and pass the assertion tests. ### Task Start ### # These are the assertions for your function: assert find_char_long('Please move back to stream ') == ['Please ', 'move ', 'back ', 'stream ']
def find_char_long(text): """ Write a function to find all words which are at least 4 characters long in a string by using regex. """ if text == "":
return [] pat = r"\b\w{4}\b" res = re.findall(pat , text) return res
With the above function , the assertion is `find_char_long('Please move back to stream ') == ['Please ', 'move ', 'back ', 'stream ']` but the real execution output is `['move ', 'back ']`.
Debug the program trace block by block until find the incorrect block. Every block should have different feedbacks: [BLOCK -0]
# text=" Please move back to stream" if text == "":
[BLOCK -1] # text=" Please move back to stream" pat = r"\b\w{4}\b" res = re.findall(pat , text) # text=" Please move back to stream" pat="\b\w{4}\b" res=['move ', 'back '] [debug] [BLOCK -0] Feedback: CORRECT. This block is correct. It checks if the input text is empty. If the input text is empty , it returns an empty list without do regex match. [BLOCK -1] Feedback: INCORRECT. This block defines a regular expression pattern `pat ` with value r"\b\w{4}\b". However , there 's an issue with the regular expression pattern . It only matches words that are exactly 4 characters long. Therefore , the return value `_ret ` is `['move ', 'back ']`. In the task description , it asks for words which are at least 4 characters long. To fix the code , we should change `pat=r"\b\ w{4}\b"` into `pat = r"\b\w{4,}\b"`. [/ debug] Please fix the Python code. [python] import re def find_char_long(text):
""" Write a function to find all words which are at least 4 characters long in a string by using regex. """ if text == "": return [] pat = r"\b\w{4,}\b" res = re.findall(pat , text) return res
[/ python] ### Task End ###
### Task Start ### # These are the assertions for your function: assert solution ([3, 3, 3, 3, 3]) == 9
def solution(lst): """ Given a non -empty list of integers , return the sum of all of the odd elements that are in even positions.
Examples solution ([5, 8, 7, 1]) ==> 12 solution ([3, 3, 3, 3, 3]) ==> 9 solution ([30, 13, 24, 321]) ==>0 """
result = 0 for i in range(len(lst)):
if lst[i] % 2 == 0 and i % 2 == 0: result += lst[i]
return result
With the above function , the assertion is `assert solution ([3, 3, 3, 3, 3]) == 9 ` but the real execution output is ` 0`.
Debug the program trace block by block until find the incorrect block. Every block should have different feedbacks: [BLOCK -0]
# lst=[3, 3, 3, 3, 3] result = 0 for i in range(len(lst)): if lst[i] % 2 == 0 and i % 2 == 0: # lst=[3, 3, 3, 3, 3] result =0 i=0
[BLOCK -1] # lst=[3, 3, 3, 3, 3] result =0 i=0 for i in range(len(lst)):
if lst[i] % 2 == 0 and i % 2 == 0: # lst=[3, 3, 3, 3, 3] result =0 i=1
[BLOCK -2] # lst=[3, 3, 3, 3, 3] result =0 i=1 for i in range(len(lst)):
if lst[i] % 2 == 0 and i % 2 == 0: # lst=[3, 3, 3, 3, 3] result =0 i=2
[BLOCK -3] # lst=[3, 3, 3, 3, 3] result =0 i=2 for i in range(len(lst)):
if lst[i] % 2 == 0 and i % 2 == 0: # lst=[3, 3, 3, 3, 3] result =0 i=3
[BLOCK -4] # lst=[3, 3, 3, 3, 3] result =0 i=3 for i in range(len(lst)):
if lst[i] % 2 == 0 and i % 2 == 0: # lst=[3, 3, 3, 3, 3] result =0 i=4
[BLOCK -5] # lst=[3, 3, 3, 3, 3] result =0 i=4 for i in range(len(lst)): return result # lst=[3, 3, 3, 3, 3] result =0 i=4 _ret=0 [debug]

heading: Future Work
text: Incorporating multilingual experiment settings, such as the HumanEval-X, remains as a future work