LeDex: Training LLMs to Better Self-Debug and Explain Code

Authors: nan  jiang
xiaopeng  li
shiqi  wang
qiang  zhou
soneya binta hossain
baishakhi  ray
varun  kumar
xiaofei  ma
anoop  deoras
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 17:43:06

Abstract: In the domain of code generation, self-debugging is crucial. It allows LLMs to refine their generated code based on execution feedback. This is particularly important because generating correct solutions in one attempt proves challenging for complex tasks. Prior works on self-debugging mostly focus on prompting methods by providing LLMs with few-shot examples, which work poorly on small open-sourced LLMs. In this work, we propose LeDex, a training framework that significantly improves the self-debugging capability of LLMs. Intuitively, we observe that a chain of explanations on the wrong code followed by code refinement helps LLMs better analyze the wrong code and do refinement. We thus propose an automated pipeline to collect a high-quality dataset for code explanation and refinement by generating a number of explanations and refinement trajectories from the LLM itself or a larger teacher model and filtering via execution verification. We perform supervised fine-tuning (SFT) and further reinforcement learning (RL) on both success and failure trajectories with a novel reward design considering code explanation and refinement quality. SFT improves the pass@1 by up to 15.92\% and pass@10 by 9.30\% over four benchmarks. RL training brings additional up to 3.54\% improvement on pass@1 and 2.55\% improvement on pass@10. The trained LLMs show iterative refinement ability and can keep refining code continuously. Lastly, our human evaluation shows that the LLMs trained with our framework generate more useful code explanations and help developers better understand bugs in source code.

Main paper: 
 introduction: 
 Code generation has become a crucial research task to automatically generate source code based on natural language description [1] [2] [3] [4] . Although the recent Large Language Models (LLMs) have shown impressive capability in code generation, generating the correct code for a complex problem in single attempt is still challenging [5] [6] [7] [8] [9] [10] [11] [12] . This is expected because even for human developers, completing a hard programming problem might need multiple rounds of trial-and-error debugging. Self-debugging capability that allows LLMs to retrospect the incorrect code and make changes to resolve the errors is becoming increasingly important besides their code generation ability.Existing works [13, 14] investigate off-the-shelf LLMs in the scale of Codex (code-davinci-002) [1] , GPT-3.5 and GPT-4, and show that these LLMs can self-debug the wrong code they generated via prompting methods in a pipeline of code generation and self-refinement as shown in Figure 1 . The The above function gets the following error message: AssertionError: assert count_ways(8) == 153, but was 47. Please fix it.Pass Code Refinement Code Generation Execution FailThe above function gets the following error message: AssertionError: assert count_ways(8) == 153, but was 47.Please explain why the above code is wrong, and fix it. user first queries the LLM for a solution for the given programming task and the initial solution from the LLM is verified by executing them against the given unit tests. If the solution passes all the unit tests, it is considered correct. Otherwise, the user collects the unit test feedback and forms a new query to ask the LLM for a refinement. Such a process can iterate until the LLM generates a correct solution or reaches the maximum number of iterations. There are different prompt designs when asking for refinement [13] . Compared with directly asking for a refined solution (referred to as "Code Refinement" in the feedback block), asking LLMs to provide an explanation of the wrong solution and then refine it in a chain-of-thought manner (referred to as "Code Explanation and Refinement" in the feedback block) helps it to better understand the unit test feedback and increases the success rate of providing refined solutions (details in Appendix A.1).However, how to improve LLMs' self-debugging capability remains under-explored, especially given the fact that open-sourced LLMs such as StarCoder [11] and CodeLlama [12] have limited selfrefinement performance. For example, the StarCoder-15B model is only able to refine 4.43% wrong solutions for problems from the MBPP benchmark [3] , in contrast, GPT-3.5-Turbo can refine 28.90% under the same setting (details in Appendix A.1). Such limited self-refinement ability motivates the need to better train LLMs to take feedback to explain and self-refine the wrong code. Although important, an essential challenge of training LLMs to explain and refine wrong code is the lack of training data, especially high-quality code explanation data. Previous work has explored Imitation learning from Language Feedback (ILF) [15] , which trains LLMs with human-annotated explanation, yet, such an approach is not scalable and the LLMs also do not obtain the ability to explain code.In this work, we propose LEDEX, an automated pipeline to collect a high-quality dataset for code explanation and refinement by generating explanation and refinement trajectories, followed by filtering through execution verification. LEDEX then leaverages the collected data, using supervised finetuning (SFT) to significantly improve LLMs' ability to explain and refine incorrect code. Additionally, LEDEX applies reinforcement learning (RL) with a novel reward design that accounts for explanation semantics and unit test success, leading to better code explanations and corrections. In summary, this work contributes the following:• We introduce LEDEX, a scalable framework comprising automated data collection, data validation, supervised fine-tuning, and reinforcement learning with novel reward mechanisms to enhance LLMs' self-debugging capabilities, resulting in more accurate code refinements and insightful code explanations. • We experiment LEDEX on three backbones (StarCoder-15B, CodeLlama-7B, and CodeLlama-13B) using code refinements and explanations, initially collected from GPT-3.5-Turbo. Supervised fine-tuning notably boosts the models' ability to diagnose and correct faulty code, achieving up to a 15.92% improvement in pass@1 and a 9.30% increase in pass@10 across four benchmarks. • LEDEX's reinforcement learning on top of SFT, uses a novel reward function that incorporates unit test outcomes and semantic analysis of incorrect code explanations. This further enhances performance, with improvements of up to 3.54% in pass@1 and 2.55% in pass@10. • LEDEX is model-agnostic; notably, CodeLlama-7B trained on data gathered from CodeLlama-34B or even itself achieves up to 8.25% and 2.14% gains in pass@1 and pass@10, demonstrating the generalizability of the approach without reliance on GPT-3.5-Turbo.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We describe our approach and main novelties, and summarize the main results at the end of abstract and introduction section. 
approach: 
 Figure 2 shows the overview of LEDEX, including the collection of high-quality code explanation and refinement data, and the training methods. LEDEX first collects a code explanation and refinement dataset by querying from pre-trained or instruct models and verifying its responses with execution feedback to filter and obtain high-quality explanation and refinement data (steps 1 and 2 in Figure 2 , Section 2.1). Then the high-quality dataset is used for supervised fine-tuning (step 3 in Figure 2 , Section 2.2), which significantly improves the model's performance in explaining the bug and refining the code. Reinforcement learning with execution feedback is used to further guide the model to generate higher quality responses and boost the model performance (step 4 in Figure 2 , Section 2.3).Figure 2 : Overview of LEDEX.We use MBPP [3] We then run these generations against test cases to select the wrong solutions that fail any test cases.Table 1 shows the number of correct (passing all the unit tests) and wrong (failing any unit test) solutions sampled for each dataset. For each wrong solution, we need an explanation of the wrong code and a correct refinement to build the code explanation and refinement dataset. We prompt pretrained or instruction-LLMs with the problem description, wrong solution, and execution feedback (either error message or failed test case) to ask for an explanation and refinement. We experimented with GPT-3.5-Turbo, CodeLlama-34B, and CodeLlama-7B for data collection. We take GPT-3.5-Turbo as the example in this section, and an example with it is shown in Appendix A.2. We study the generalization of this data collection with different LLMs in Section 4.3. As LLMs may provide wrong explanations or refinements, we cannot blindly take them as training data. Thus, we verify the refinements by running them against the test cases again, and only those passing all the test cases are considered correct refinements. For explanation, we consider the explanations along with the correct refinements as correct. Overall, for example with GPT-3.5-Turbo, we get 13,735 correct explanations and refinements: 2,203 for MBPP, 6,419 for APPS, and 5,113 for CodeContests. This verification step is crucial to guarantee the quality of the automatically collected code explanation and refinement dataset.We form the fine-tuning data in an instruction-following format similar to StarChat [16] , where the user input is enclosed by 
 and 
, while LLM's answer is enclosed by 
 and 
 in the chat history. Moreover, to alleviate the limited amount of data, we augment the fine-tuning data by using two different instructions: providing the task description, the initial wrong code, and execution feedback, asking for (1) a refinement directly, or (2) an explanation of the wrong code and then a refinement in a chain-of-thought manner. Examples are given in Appendix A.2During supervised fine-tuning, although we include the wrong solutions as LLM's initial answer in the chat history, we do not calculate the loss for this part since we do not want the LLM to intentionally generate those wrong solutions. They are just provided as context for code explanation and refinement if the LLM indeed makes mistakes in real use cases.Reinforcement learning is widely used to further improve the quality of LLM's generated outputs [17] [18] [19] [20] . Through the RL framework, the LLM is optimized by using an algorithm to update the weights using both success and failure trajectories and maximize the rewards of its outputs. To train the fine-tuned LLMs to generate better code explanations and more correct code refinements, we design the rewards considering both parts.To train LLM to refine code, the correctness of the refinement is the main goal, which can be measured by its code similarity to the ground truth, as well as the execution result. We use CodeBLEU score as metrics for code similarity and unit test passing rate as metrics for execution results.Given a wrong solution w, the set of correct and wrong (failed) refinements are notated by R w c and R w w . For any refinement r, we calculate its CodeBLEU score and the unit test passing rate as follows:S cb (r) = 1 |R w c | rc∈R w c CodeBLEU(r, rc); Sut(r) = |Tp(r)| |T |S cb is the average CodeBLEU score between a given refinement and all the correct refinements. S ut is the fraction of the number of passed unit test cases (T p ) when running the refined code r, over the total number of unit test cases (T ) provided for this problem in the dataset.In Figure 3 , the x-axis is the scores of certain metrics, and the y-axis is the number of training data with a certain score (same for other figures in Figure 3 ). Thus, Figure 3 In our dataset, there are wrong code explanations along with code refinement, whose quality may not be perfectly reflected by the code quality. A correct code explanation may also be followed by incorrect refinement, thus, it is necessary to consider the explanation in the reward. The code explanations followed by correct refinements are treated as ground truth, notated by E w c . We calculate the average sentiment similarity [21, 22] between the explanation embedding e and corresponding embeddings in ground truth asS ex (e) = 1 |E w c | ec∈E w cCosSim(RoBERTa(e), RoBERTa(e c ))The distribution of S ex (e) is shown in Figure 3 (c).Given a pair of explanation and refinement, i.e., (e, r), the reward of the generated code refinement and the code explanation is designed as:EQUATIONThis code reward R(r) is the average of CodeBLEU score and the unit test passing rate, and since both S cb (r) and S ut (r) are scored in the range of [0, 1] , this equation makes the reward of code refinement in the range of [-5, 5] . Figure 3 (d) shows the distribution of the code refinement reward on the training dataset, which mitigates the overlap issue between correct and wrong outputs with CodeBLEU score alone as illustrated in Figure 3 (a). It also makes the reward distribution continuous, addressing the discreteness problem of only using unit test passing rate.For the design of explanation reward R(e), We observe from Figure 3 (c) that the explanation sentiment similarities of the training data mostly lie in the range of [0.4, 1.0], thus, we project the range of [0.4, 1.0] to [-5, 5] and treat 0.7 as the borderline (projected to 0 correspondingly) of good or bad explanations. Figure 3 (e) shows the distribution of the wrong code explanation reward on the training dataset. The distribution shows that there could be a good or correct code explanation followed by a wrong code refinement, where assigning a high or low reward to the entire output is not reasonable. This leads to our PPO [23] algorithm with code refinement and explanation rewards considered separately. Due to space limit, the PPO algorithm is shown in Appendix A.3.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have a limitation section (please refer to Section 5) in the paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experimental setup: 
 For supervised fine-tuning, we fine-tune three LLMs (StarCoder-15B, CodeLlama-7B, and CodeLlama-13B) using the correct initial solutions and correct refinements collected from the MBPP training set, APPS training set, and CodeContests. The model is fine-tuned for two epochs, using a batch size of 128. The optimizer is AdamW [24] with learning rate set to 2e -5 . The learning rate is adjusted using a warmup of 500 steps and then decayed following a cosine scheduler.We further train supervised fine-tuned LLMs with reinforcement learning using the PPO algorithm. The reinforcement learning training data is all the initial solutions and collected refinement on the MBPP and APPS training set. The learning rate is 2e -6 , and the batch size is set to 64. We implement reinforcement learning training based on the TRL [25] library. Both the supervised fine-tuning and reinforcement learning are conducted on 8 NVIDIA A100 GPUs, each with 40GB of memory.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: This paper does not include theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
result: 
 To evaluate the effectiveness of LEDEX, we study the following research questions (RQs) regarding code generation and code refinement capability, iterative refinement ability, approach generalizability, and the quality of the generated code explanations.We evaluate the models trained with LEDEX for their code explanation and refinement ability using four benchmarks: MBPP [3] , HumanEval [1] , MBPP + [26] , and HumanEval + [26] . We use pass@k [1] and success refinement rate as the evaluation metric. For the generation of the initial solutions, the models sample 100 solutions per task in the benchmarks (temperature set to 0.8), which are run against the provided test cases. For every incorrect solution that fails any test case, we let the models sample one refinement (and one explanation).Table 2 presents the pass@k results across four benchmarks. Overall, fine-tuning the LLMs with our curated dataset of code explanations and refinements leads to substantial improvements in both pass@1 and pass@10 for all three model architectures. For StarCoder-15B and CodeLlama-13B, HumanEval + pass@1 pass@10 pass@1 pass@10 pass@1 pass@10 pass@1 pass@10 Init. 37 RL achieves the highest pass@1 and pass@10 scores (bolded) across all four benchmarks. For CodeLlama-7B, RL achieves the best performance in seven out of eight cases, with SFT yielding the highest pass@1 score on the MBPP benchmark.For easier comparison, Table 3 summarizes the overall pass@k results on MBPP and HumanEval, along with the improvements achieved through SFT and RL. The improvements from SFT are compared to direct prompting, while the improvements from RL are relative to SFT.On MBPP and HumanEval overall, SFT boosts StarCoder-15B's pass@1 by 15.34% and pass@10 by 6.58% when directly generating code refinements. When incorporating code explanations in a chain-of-thought format, SFT further enhances StarCoder-15B's performance by 15.92% on pass@1 and 6.30% on pass@10. RL brings an additional 2.43% improvement in pass@1 and 2.55% in pass@10 for direct refinements, and a further 1.87% pass@1 and 1.85% pass@10 increase when generating both code explanations and refinements. Comparable improvements from SFT and RL are observed across the CodeLlama-7B and CodeLlama-13B models as well.On the MBPP + and HumanEval + benchmarks, which feature more rigorous test cases, respectively [26] , we observe even greater improvements from RL training on the CodeLlama models. CodeLlama-7B achieves a 1.79% increase in pass@1 and a 2.60% increase in pass@10 for refined solutions with code explanations. CodeLlama-13B shows a 3.54% improvement in pass@1 and a 1.29% improvement in pass@10 for directly generated refinements. These results demonstrate that RL training enables LLMs to produce or refine solutions that are more robust and capable of passing stricter test cases. Additional experiments and detailed case studies can be found in Appendix A.4.1, A.5.1, A.5.2, and A.5.3. LLMs have the ability to iteratively self-debug until they arrive at correct solutions. Figure 4 illustrates the overall pass@k of CodeLlama-7B across four benchmarks after up to three rounds of refinements.To simplify the figure, we plot the higher pass@k from either the "Refine" or "Expl. + Refine" approach at each refinement round for each model. Additional results on iterative refinement are provided in Appendix A.4.2.Figure 4 : Pass@k of prompting, SFT, and RL CodeLlama-7B after three iterations of refinements.Both SFT and RL consistently outperform prompting across all three refinement rounds. Even after three rounds, the prompting approach fails to match the pass@k achieved by SFT after just the first HumanEval + pass@1 pass@10 pass@1 pass@10 pass@1 pass@10 pass@1 pass@10 CodeLlama-7B Init.47 round (e.g., 47.53% vs. 56.75% in Figure 4 (a) ). These results demonstrate that LLMs trained with our pipeline possess strong iterative refinement capabilities, enabling them to achieve progressively higher pass@k with each additional round of refinement.To demonstrate the generalizability of LEDEX, particularly the independence of our data collection process from GPT-3.5-Turbo, we substitute GPT-3.5-Turbo with CodeLlama-34B for data collection.As CodeLlama-34B is a pre-trained model, we incorporate few-shot examples in the prompts to guide the generation of incorrect code explanations and refinements. All other processes remain unchanged.Table 5 presents the pass@k results for CodeLlama-7B trained on data collected from CodeLlama-34B, with Table 6 providing an overall comparison. Although SFT achieves slightly smaller improvements (around 1-3% lower than with GPT-3.5-Turbo data), it still yields notable gains in overall pass@1 and pass@10. Additionally, we observe that RL training further enhances performance on the MBPP + and HumanEval + benchmarks, with pass@1 improving by 5.28% and pass@10 by 2.24%. These results demonstrate the generalizability of LEDEX and further suggest that collecting data from a more powerful LLM can lead to better training outcomes within our framework. Additional results can be found in Appendix A.4.3.We also investigate the feasibility of using an LLM to self-bootstrap its training data, specifically by using CodeLlama-7B to generate the data that is then used for its own SFT and RL training.Table 7 presents the pass@k results of CodeLlama-7B trained with self-bootstrapped data, with Table 8 showing the overall comparison. Compared to prompting, SFT with self-bootstrapped data still delivers up to 8.25% and 2.14% improvements in pass@1 and pass@10 on MBPP and HumanEval, and up to 5.33% and 0.55% improvements on pass@1 and pass@10 on MBPP + and HumanEval + . Additionally, RL training using the self-bootstrapped data results in a further 0.71% improvement on MBPP and HumanEval, and up to a 0.78% increase on MBPP + and HumanEval + . These findings suggest that while self-bootstrapped data enables SFT to provide substantial gains over prompting, RL training offers less improvement compared to using data from stronger LLMs, such as CodeLlama-34B or GPT-3.5-Turbo.We assess whether explanations for incorrect code are useful for developers in understanding their bugs. To do this, we randomly sample 50 problems with initial incorrect solutions from the MBPP and HumanEval benchmarks and use different LLMs to generate explanations for the wrong code. HumanEval + pass@1 pass@10 pass@1 pass@10 pass@1 pass@10 pass@1 pass@10 CodeLlama-7B Init. 45 Each explanation is scored on a scale from 1 to 5, based on its correctness and helpfulness, where 1 indicates a completely incorrect or misleading explanation, and 5 denotes a correct explanation that also provides a detailed hint on how to fix the code. Both GPT-4 and human developers are used as evaluators. For GPT-4, we follow prior work [27] and prompt it to score each explanation. The results are presented in Table 9 . Both SFT and RL lead to improved explanation quality compared to prompting, with GPT-4 assigning higher scores to models trained using our approach. Notably, the gap between GPT-3.5-Turbo and the trained LLMs significantly narrows after fine-tuning.Given the time required for human evaluation, we only asked developers to rate explanations from StarCoder models and GPT-3.5-Turbo. Each explanation is scored by two developers, and their ratings are averaged. The human evaluations align with GPT-4's, confirming that SFT improves explanation quality over prompting, while RL further enhances explanations by incorporating code explanation semantics into the reward design. Detailed rubrics and examples of human evaluations can be found in Appendix A.6.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We describe the details of data collection (Section 2.1, Appendix A.2), as well as experimental setup (Section 3) in the paper. These are enough to reproduce the results described in the paper. 
limitation: 
 One potential limitation of our study is the reliance on specific large language models (LLMs) from which we collect code explanation and refinement data. Our automated framework is designed to be independent of any specific LLM, and for this study, we use GPT-3.5, CodeLlama-34B, and CodeLlama-7B itself to collect training data, and both bring significant improvement through SFT and RL. However, for future work, it would be interesting to explore the use of other LLMs, including smaller models or a mix of diverse LLMs, to gather explanation and refinement data.Additionally, our current experiments only use two types of prompts for enabling LLMs to self-debug: one that directly asks for refinement and another that first asks for an explanation of the wrong code followed by refinement. While these prompt designs have shown effectiveness, there might be better prompt strategies for self-debugging that we have not explored due to resource constraints. Exploring a broader range of prompt designs could potentially enhance the performance of our framework. Nonetheless, our proposed training framework is flexible and should be generalizable to different types of data and prompts, paving the way for future innovations in this area.6 Related work 6.1 Large language models for code Large language models (LLMs) have been widely explored across a variety of code-related tasks, including code generation [5, 8, 6, 7, 10, 9, 12, [28] [29] [30] [31] [32] [33] [34] , bug fixing [35] [36] [37] [38] , program testing [39, 40] and fuzzing [41] and so on. These models have demonstrated impressive capabilities in these domains, largely due to their strong understanding and generation abilities acquired through extensive pre- However, most existing works in this area focus primarily on improving LLMs to provide the expected output in a single round of generation. The emphasis has been on enhancing the initial output quality, minimizing the need for further modifications or iterations. This one-shot generation approach, while useful, overlooks the potential of iterative refinement, which is a crucial aspect of real-world programming where initial drafts often require multiple rounds of revision and debugging.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification:Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
self-debugging and self-refinement: 
 Existing techniques have studied the possibility of using LLMs to refine their generations. Yet, most techniques are prompting LLMs with execution results [13, 14, [42] [43] [44] [45] [46] [47] [48] for the refinement. Such prompting approaches bring limited improvement to smaller open-sourced LLMs compared to GPT-3.5. Other techniques train LLMs to self-debug. ILF [49] uses human-annotated feedback information and thus is unscalable, CYCLE [50] and Self-Edit [51] use SFT to fine-tune LLM to generate the refinement only based on the unit test execution feedback. OpenCodeInterpreter [52] and EURUS [53] construct high-quality multi-turn interaction datasets using GPT-3.5-Turbo and GPT-4 to fine-tune LLM for self-refinement.This work has four differences compared with others that train LLMs: (1) we train LLMs to generate code explanation followed by refinement, which provides additional information to users, (2) we do not require human-annotated training data but propose a scalable pipeline to automatically collect and verify data from another LLM, (3) our data collection pipeline can be generalized to open-sourced LLM or even the same LLM itself, and (4) we design novel reward functions in the RL training stage, considering both the code and explanation quality, which beings extra improvement. 
conclusion: 
 This work highlights the importance of training open-source LLMs to self-debug and introduces a scalable framework that includes automated data collection, verification, supervised fine-tuning, and reinforcement learning with novel reward designs to enhance LLMs' self-debugging capabilities.Our data collection process is model-agnostic, as demonstrated by the improvements achieved with both GPT-3.5-Turbo and CodeLlama. The data verification ensures high quality of code explanations and refinements. Fine-tuning on this data significantly boosts the LLMs' self-debugging abilities, yielding up to a 15.92% increase in pass@1, a 9.30% increase in pass@10, and a 10.15% increase in successful refinements. Reinforcement learning, utilizing our novel reward design, further enhances performance, with additional gains of up to 3.54% in pass@1, 2.55% in pass@10, and 3.62% in successful refinement rates. Comprehensive analytical experiments confirm the generalizability of our approach and demonstrate the iterative refinement capabilities of the trained models. Moreover, human evaluations indicate that the LLMs trained with our framework produce higher-quality explanations, effectively aiding developers in understanding and resolving bugs in source code.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The improvement is considered significant across different models and datasets with multiple runs. However, statistical significance test does not apply to the code generation/refinement task this paper studies. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The infrastructure we used to run the experiments are described in Section 3.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We describe the potential societal impacts in Section A.7Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The LLMs and datasets used in this paper are open-sourced and we carefully cite the original papers that produce them.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: This paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We evaluate two prompting strategies for querying LLMs to perform code refinement: one where the LLMs directly generate the refinement (Figure 5 (3.1)), and another where they first explain why the code is incorrect before generating the refinement (Figure 5 (3.2)). Running the above function returns the following error: NameError: name 'math' is not defined So the code does not pass the assertion. Please explain why the above code is wrong, and fix it. The error message "NameError: name 'math' is not defined" indicates that the code is trying to use the math module, which provides mathematical functions, but it has not been imported. Below is the fixed code. The wrong code uses the `sum()` function incorrectly. The `sum()` function expects an iterable as its argument, but in this code, the `nums` variable is already a tuple. So when you pass `[nums]` to the `sum()` function, it treats `nums` as a single element of the iterable, which is a tuple. Hence, it tries to add the tuple to an integer, resulting in a `TypeError`. Code RefinementFigure 6 provides an example of the prompt we used to collect code explanation and refinement training data from GPT-3.5 (the same approach was used for CodeLlama-34B). We set the temperature to 0.8, allowing GPT-3.5 to generate 10 code explanations and refinements per prompt. The collected data includes both the explanation of the incorrect code and the corresponding refinement. From this, we construct two formats of instruction-tuning data for supervised fine-tuning (SFT).Figure 7 (left) shows an instruction that asks LLMs to provide only the code refinement, where the instruction includes the task description, the incorrect initial solution, and execution feedback. The LLMs are trained to generate the refined solution based on this input. Figure 7 (right) presents a different instruction that asks for both the code explanation and the refinement. By incorporating both types of instruction data during training, we increase the diversity of the training set and improve the LLMs' robustness across different prompts.We modify the standard PPO algorithm to optimize the explanation and refinement generation based on their reward separately. Given an LLM (already supervise-fine-tuned) π as the policy model, a prompt x, a generation y consists of a code explanation and a refinement: y = [e, r] =At = δt + γδt+1 + . . . + γ T -t δT , δt = Rt -V (y<t, x, π) + γV (y<t+1, x, π)A t is the advantage at decoding timestamp t, T = |e| + |r| is the total length of generation output, and γ is the discount rate (a hyper-parameter set to 0.99 in our experiment). V (y <t , x, π) is the state value at generation step t given input x, which is learned and calculated by a linear layer on top of the policy model π. R t is the reward at decoding timestamp t, which is calculated as follow:R t =        R(r) -KLt(π, π ′ ) , t = T R(e) -KLt(π, π ′ ) , t = |e| KLt(π, π ′ ) ≈ log P(yt|yt-1, x, π) P(yt|yt-1, x, π ′ ) , otherwisewhere KL is the Kullback-Leibler divergence [54] between the action distribution given by the updated policy model π and the old policy model π ′ before the update.Instead of assigning both the code and explanation rewards to the entire output, we separate them by only assigning the explanation reward to the explanation portion. This avoids the issue that a low reward is assigned to a correct explanation followed by a wrong refinement, and the LLM learns to keep the correctly generated explanation part and focus on improving the incorrect refinement portion. For data where only code refinement but no explanation is generated, we use the standard design to assign the code reward to the code refinement.Following existing works, the loss of the PPO algorithm training is:L = -E T t=1 log P(y t |y <t , x, π) log P(y t |y <t , x, π ′ ) A t + αE T t=1 V (y <t , x, π) -A t + V (y <t , x, π ′ ) 2By minimizing this loss, the policy model π (the LLM under PPO training) is trained to generate explanations and refinements with higher reward but also constrained by not being distracted too much far away from the supervised-fine-tuned LLM π ′ .Table 11 shows the comparison between CodeLlama-7B trained with LEDEX, and the CodeLlama-7B trained with code generation data only. Although training with code generation enables LLMs to get comparable (or even higher pass@k on HumanEval) pass@k on one round of code generation, the LLMs cannot obtain strong self-debugging ability from code generation data. The LLM trained with code generation data can only improve the pass@k very little after self-debugging. By contrast, the LLM trained with the full collected data (code generation, code explanation, and code refinement data) gets significantly higher pass@k after refinement, showing its strong self-debugging ability.Table 11 : Pass@k on MBPP and HumanEval by CodeLlama-7B trained on code generation only, and that trained with our collected code explanation and refinement data.Approaches MBPP Humanval MBPP + HumanEval + pass@1 pass@10 pass@1 pass@10 pass@1 pass@10 pass@1 pass@10 Init.46 Figure 8 illustrates the pass@k metric when employing trained CodeLlama-13B for iterative code refinement. The findings align with those depicted in Figure 4 , that both SFT and RL consistently outperform the prompting approach in each round of refinement. Notably, even after three rounds of refinement, the prompting method fails to achieve a higher pass@k than what SFT and RL models attain after just one round. This highlights the substantial advantage of SFT and RL methods in enhancing code quality. Moreover, the SFT and RL CodeLlama-13B models demonstrate robust continuous refinement capabilities, maintaining their superior performance across multiple iterations. This consistent outperformance underscores the effectiveness of SFT and RL strategies in refining and improving code, suggesting their potential for more efficient and reliable coding practices in iterative development scenarios.Figure 8 : Pass@k of prompting, and RL CodeLlama-13B after three iterations of refinements.The pass@k of CodeLlama-13B trained with data collected from CodeLlama-34B are shown in Table 12 , where RL achieves the highest pass@1 and pass@10 in all the benchmarks. The overall results are shown in Table 13 , and SFT improves the pass@1 by up to 9.14% and pass@10 by 3.47% on the overall of MBPP and HumanEval, and improves the pass@1 by up to 8.68% and pass@10 by 3.92% on MBPP + and HumanEval + . Further RL training improves the pass@1 by up to 0.61% and pass@10 by 0.86%. The improvement brought by RL is significantly larger on MBPP + and HumanEval + (up to 6.80% higher pass@1 and 2.28% higher pass@10), which is consistent with our finding when using GPT-3.5-Turbo's training data that RL brings more considerable improvement on harder benchmarks.Table 14 presents the success refinement rates achieved by CodeLlama-13B when trained on data collected from CodeLlama-34B, averaged across four benchmarks. SFT refines 15.60% to 16.83% of incorrect solutions, outperforming the prompting approach by 3.96% to 4.86%. RL training further boosts the refinement rate, improving it by 4.78% to 7.14% over SFT. Notably, the improvement from SFT is somewhat smaller when using CodeLlama-34B's data compared to GPT-3.5-Turbo's data, likely due to the slightly lower quality of explanations and refinements generated by CodeLlama-34B. However, RL training raises the refinement rate to levels comparable to those achieved with GPT-3.5- HumanEval + pass@1 pass@10 pass@1 pass@10 pass@1 pass@10 pass@1 pass@10 Init.42 A.5 Case studies A.5.1 Correct refinements generated by SFT LLMs Figure 9 shows an example from the HumanEval benchmark for which the prompting approach fails to generate correct refinement. The bug in the initial solution is that 'nlargest' returns the largest elements in descending order, but from the example provided, one can find out that the expected output is in ascending order. The prompting approach does not work for this example, and the prompted StarCoder generates a hallucinated explanation and simply repeats the wrong solution. Actually, such simple repeats of wrong solutions are very common when using the prompting approach, which supports our motivation to train LLM to self-debug. However, after SFT training, the trained StarCoder can correctly figure out the reason for test failure and generate the correct refinement.Figure 10 shows an example from the MBPP benchmark for which only the StarCoder trained with reinforcement learning explains the wrong code correctly, pointing out that the failing test case is because "this approach does not consider the case where multiple elements have the same maximum count" (highlighted with green background), and generates the correct refinement. The SFT StarCoder is unable to diagnose the wrong code correctly.Figure 11 shows an example from HumanEval, for which both SFT and RL CodeLlama-13B generate the correct refinements that pass all the test cases from the HumanEval benchmark. Yet, the refinement from CodeLlama-13B is not fully correct, as "(x, y, z) == ((int)x, (int)y, (int)z)" is not equivalent to checking if these numbers are integers. The refinement generated by SFT CodeLLama-13B (left) fails the harder test cases in HumanEval + , i.e., AssertionError: expect any_int(3.0, The wrong code is using the `nlargest` function from the `heapq` module to find the k largest elements in the array. However, the `nlargest` function returns the largest elements in descending order, while we need the largest elements in ascending order. This is why the output is reversed. We let developers rate the explanations generated by different LLMs based on the following rubrics, which consider both the correctness of the statements in the explanations and also helpfulness of the explanation. Figure 12 shows an example of developer rating, where the prompting and SFT StarCoder are rated "1" since their explanation is wrong and not informative. yet, the RL StarCoder's explanation successfully points out that the function should return 'Not Matched!' when there is no match. Figure 13 shows another example, where the prompting StarCoder is rated "2", as the developer thinks the explanation is stating a correct fact. By contrast, SFT and RL StarCoder correctly point out that the function should check if the array is empty or not.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Justification: Code generation and code refinement are widely studied and explored, which aims to improve developers' coding experience with LLMs. There is no special concerns of ethics of the technique.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper proposes a framework to improve self-debugging for code generation tasks through a combination of SFT and RL using failed tests, their fixes coupled with explanations. The reviewers unanimously recommend its acceptance. They note strong empirical results obtained using comprehensive analysis done with various datasets and models, highlight its novelty for the reward setup and the importance of the problem. I therefore recommend its acceptance.

Reviews: review: paper_summary: paper_summary: Authors propose a framework to perform SFT and RL training to achieve superior performance in generating code with open code LMs on the self-debugging task. They leverage the test suite present in benchmarks like APPS, CodeContests to obtain execution feedback for model refinements on CodeLM generated code. The resulting dataset is then used for SFT training, followed by RL training that can also leverage failed trajectories besides the successful ones (where the model succeeded in generating a code that passes all unit tests). Authors show promising gains for open weight models in the model's capability to self-debug after training with their proposed methodology.
strength_weakness: strength_weakness: - While the experiments and analysis of the results and datasets are fairly exhaustive, I believe the choice of RL algorithms should be justified by considering or eliminating alternatives like preference optimisation using DPO or KTO. I'd suggest at least adding a discussion on the pros and cons of preference learning compared to the RL setup that the authors advocate in this work.
- Some missing baselines: teacher model (GPT-4/3.5/CodeLlama35B) performance is missing in Tables 2 and 5. Authors do not discuss the persisting gap in performance in any between the models used in creating the datasets and the performance their approach attains on the benchmarks used.
- A very relevant related work (Teaching LLMs to Self-Debug https://arxiv.org/abs/2304.05128) mentions gains in sample efficiency as one of the major benefits of performing self-debugging/refinement. I could not find a discussion or results on this aspect for the fine-tuned models presented in this paper.
questions: questions: - APPS and CodeContests are used to train the CodeLMs, but I could not find evaluation on the APPS-test or CodeContests test set. Could you explain this choice?
- Can you provide details on the number of GPU hours required in the experiments?
- Have the authors considered training a separate model using the SFT and RL techniques to solely solve code refinement for code generated from the base model?
- What do the authors think about the generality of this method to improve on attributes beyond correctness in refining code? e.g. readability, performance and secureness of generated code.
- I'm curious how the proposed approach would compare against a simple baseline where the model is SFT-trained on the final refinement collected in your training set. The current setup involves using problem description $x$ and ground truth code $y$, test suite $T$ to generate a synthetic code solution $y'_l$ that fails and is explained by $x_l$ followed by a successful refinement $y'_w$. Your approach then trains the model to generate $x_l, y'_w$ given $x, y, y'_l$. A simple baseline to compare against could involve training the model to generate $y'_w$ given $x$. This would confirm the value in framing code generation as a explanation + refinement task.
limitations: limitations: Briefly discussed by the authors in Section 5. Could be expanded to include a discussion on other aspects of code refinement not covered in the paper, and acknowledging gaps if any in performance of open models trained with this method and closed models.
paper_summary: paper_summary: This paper proposed a pipeline to obtain code explanation and refinement data from a stronger model (mainly GPT-3.5 with CodeLLaMA as ablation) to train weaker models (i.e., StarCoder-15B, CodeLLaMA-7B/13B) using SFT and RL methods. More specifically, it uses the weaker models to sample the incorrect solutions and use the stronger models to give code refinement and explanations. It also designed reinforcement learning rewards specifically for the code refinement and explanation tasks. Experiments are conducted on MBPP and HumanEval, as well as their EvalPlus versions, and results show that the both SFT and RL yields improvements over the baselines, while the improvements from RL are relatively marginal compared to those of SFT.
strength_weakness: strength_weakness: W1. Something slightly disappointing is that this work chooses to generate training data **from a stronger model**, which classifies it into the category of distillation, while the entire pipeline could have been done with the same LLM to explore the potential of self-improvement;  
W2. While the paper focus a lot on the design of the RL method (which seems to be a big part of the contribution from my understanding), the actual improvements yield by the RL methods are quite marginal. However, I do note that at least the performance does not decrease for the most cases;  
W3. There are some baselines experiments and ablations that can be added to make the evaluation part stronger (see questions below).
questions: questions: Q1. According to Olausson et al., 2023 on self-repair, after $k$ rounds of self-refinement after the initial code generation, the success rate is often less than simply pass@$k+1$. And also note that the sampling can be done in parallel while the iterative refinement can only be done sequentially. Have you compared the refinement results with pass@$k+1$ to see if self-refinement yields any benefit before / after the SFT/RL training?  
Q2. From Fig. 3(a), it seems to me that the CodeBLEU score is not a good metric as it can barely separate the correct and wrong outputs distribution-wise, is there any reason for CodeBLEU to still be factorized into the reward function despite this?  
Q3. Can you comment on the reliability of using the RoBERTa embedding for measuring the similarity of the explanations? Are there better ways to do this?   
Q4. From Tab. 2, it seems that after SFT, the "Init." performance also significantly improved, does this means that **only** training on code refinement and explanation can also improve code generation?   
Q5. (Pertain to W3) Have you tried to use the same model as the LLM to create the training data?  
Q6. The training data are created on top of APPS and CodeContest as well, but why are those two datasets not used in evaluation?  

**References**  
Olausson, Theo X., et al. "Is Self-Repair a Silver Bullet for Code Generation?." The Twelfth International Conference on Learning Representations. 2023.
limitations: limitations: N/A
paper_summary: paper_summary: The authors teach language models to better self-debug and explain code. Particularly, they utilize code explanations in the repair process where explanations are generated before refining the programs. This is accomplished via training the models with SFT and RL on data curated from different sources and model generations with test case based rejection sampling.
strength_weakness: strength_weakness: Weak results
* Benefit of explain + refine over just refine. 
The proposed explain + refine approach, associated loss on explanations does not seem to be improving performance. Infact, Expl.+Refine rows sometimes do worse than the only Refine rows. These results highlight a lack of benefit from adopting this approach. While I understand that having associated explanations is appealing and perhaps might even provide models with more inference-time-compute before performing refinements, I think the current approach or experiments does not convey that. The associated human evaluations (Table 7) also point to similar findings. 

* Optimizing single-turn vs multi-turn performance.
Finally, a lot of the instruction-tuned variants of the models used in this work claim better single-turn HumanEval and MBPP performance. I wonder how the findings would change if they started from a strong instruction-tuned model and improved its repair performance. For instance, the OpenCodeInterpreter-CL-7B model shows performance improvements of 3 points from execution feedback on HumanEval (from 72 to 75) due to strong instruction tuning data pushing pass@1 to 72. Perhaps to a larger point, the effect of repair depends on the choice of the underlying model and broader data mixture which this paper does not study.

* Choice of evaluation datasets. 
Since the authors use competition programs (like APPS or CodeContests) in their study, perhaps it is fair to also evaluate the models on competition programming datasets.

* Missing related works.
[1] is also pertinent to LLM reasoning. [2] and [3] also train and release open LLMs on repair trajectories. [3] in fact uses a similar explanation format. 

[1] Reflexion: Language Agents with Verbal Reinforcement Learning

[2] OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement

[3]  Advancing LLM Reasoning Generalists with Preference Trees
questions: questions: * Table 1 further details. I anticipate that many problems in the training sets remain unsolved post-refinement, particularly for APPS and CodeContests which are more challenging benchmarks. Can the authors list details about problems beyond number of solutions.
limitations: limitations: Limitations are discussed adequately (besides some of the above-mentioned weaknesses)
paper_summary: paper_summary: This work proposes a novel framework to enhance the self-debugging capabilities of smaller language models that do not benefit much from self-refine or other prompt-based debugging approaches. Sampling incorrect code samples produced by LMs, they pass the execution feedback on these to GPT-3.5/4 and prompt it to explain the reason for the errors and propose code refinements. Accurate refinements are used to fine-tune and create a code-correcting model. This is further enhanced with a PPO based learning from a novel reward assignment mechanism accounting for both explainability and code refinement. Overall, they demonstrate the importance of having explanations for incorrect codes and how RL can be used to enhance the debugging ability of models to show superior performance across benchmark datasets.
strength_weakness: strength_weakness: Nothing major, future evaluation on datasets like APPS etc. could be beneficial to understand the impact on harder tasks
questions: questions: - Was there any reason behind choosing the range of score to be [-5,5]?
- It seems in the bigger models tested (\geq 13B) the refine itself is quite effective, any intuitions on this scale effect?
limitations: limitations: Yes
paper_summary: paper_summary: The paper addresses the goal of self-debugging of generated code, while also explaining it. 
The approach is to: (a) sample code outputs to natural language inputs, and keep only the wrong code outputs according to unit tests; (b) sample **refinements** ("fixes") to the wrong code outputs and test those refinements using unit tests, to get both correct and incorrect refinements; (c) the authors train an LLM using SFT and RL on those correct and incorrect refinements.

The resulting trained LLM is shown to provide empirical gains, especially when allowing it to self-refine its outputs at test time, with the test-time unit tests.
strength_weakness: strength_weakness: * The paper is mostly applicative, and includes a mix of known techniques. I feel that the paper is low on conceptual novelty. The concept of self-debugging and verifying correctness using unit tests was introduced by [Chen et al., 2023](https://arxiv.org/pdf/2304.05128) (although only using prompting), and RL using the signal coming from the unit tests was done in several papers (that the authors cite). 
* It feels like there are so many different techniques and heuristics involved, that it is hard to pinpoint the exact contribution of each of them. For example: the design of the explanation score as $CosSim(RoBERTa(e), RoBERTa(ec))$, the exact hyperparameters of the reward design (e.g., $\frac{50 · S_{ex}(e) 35}{3}$), and the exact design of $S_{cb}$ using CodeBLEU. The use of explanations before fixing the code is a form of Chain-of-Thought (Wei et al., 2022), or the "feedback" in Self-Refine (Madaan et al., 2023).
* Further, teaching the models to self-debug is done using **larger models**. 
That is, the refinements are sampled from larger models such as gpt-3.5-turbo, CodeLlama-34B, and then these refinements are used to train the smaller StarCoder-15B, CodeLlama-7B, CodeLlama 13B models.
This adds a dimension of distillation (from large models to small models), and further makes it difficult to pinpoint the exact source of contribution.
* The approach is not compared to any related work, or to the teacher models themselves.
* Position in literature - Although many papers are cited, I feel that the paper does not position itself well in the literature. I do not remember exactly what did each related work do, but the paper does not help me understand the differences and its novelty compared to the related work. For example, what's the difference between this paper and [18] and [20]? They seem very similar, but the Related Work does not highlight the novelty over them.
    * Another new paper that was not cited: [Ansong et al., NExT: Teaching Large Language Models to Reason about Code Execution, ICML'2024](https://arxiv.org/pdf/2404.14662). I am not interested in the authors just citing the paper as an additional number between brackets, I am interested in discussing the actual differences and novelty over that paper.
questions: questions: 1. At test time, when evaluating the SFT/RL models: does the model see the execution results of its generated test code before refinement? That is, can the model use the unit tests of the test examples as well, or are unit tests used only at training time?
2. In the definition of $S_{cb}$ - if we rely on unit tests to verify correctness, why do we need to encourage the refinement to be similar (in terms of CodeBLEU) to **all** the possible correct refinements?
3. RL training on benchmark data may over-specialize on their specific domain, while degrading the general coding abilities of the LLM. After all, these benchmarks are only benchmarks, and over-specializing on them may hurt the usability of the model in practical use. For example, have the authors checked whether applying their SFT+RL hurts the perplexity on general code?

To summarize, I think that the paper presents strong empirical gains, but the scientific novelty is low, as the contribution is mostly applied. I thus vote for a borderline reject.
limitations: limitations: N/A

Comments: title: Response to authors
Thanks for the response
Response to authors
comment: Thank you for your thorough response!

All my concerns are now resolved and I increasedy score to 7. Good luck!
I'd like to thank the authors for the detailed response, especially the additional experiment results. 

I find the additional results interesting and promising, and I think the argument on using CodeBLEU stabilizes training makes a lot of sense.

I think adding the results / discussions from at least 1+2+3 from above would make the paper more interesting and stronger, so I hope the authors would add them in the next version of the paper. 

I have improved my score accordingly, good luck!
Thank you for following up.

We re-organized the rebuttal so that the response to each question is clear. Due to character limits, we abbreviate the reviewer’s questions, and refer some questions to our first response. Below, we focus primarily on addressing the reviewer’s new questions.

## 1. Paper novelty
Reviewer: “The paper is mostly applicative, and includes a mix of known techniques. I feel that the paper is low on conceptual novelty.”

Response:
Please refer to the “Paper novelty” in our first response.

## 2. Position in literature
Reviewer: “Position in literature - Although many papers are cited, I feel that the paper does not position itself well in the literature.”

Response:
Please refer to the “Position in literature” in our first response.

## 3. Self-taught refinement
Reviewer: “This adds a dimension of distillation (from large models to small models), and further makes it difficult to pinpoint the exact source of contribution.”

Response:
Please refer to “Self-taught refinement” in our first response.
## 4. Comparison to the teacher models or related work
Reviewer: “The approach is not compared to any related work, or to the teacher models themselves.”

Response:
The prompting baseline mentioned in our paper refers to the related work [1], and we compare with prompting method in our experiments.

We provide the comparison with the teacher models CodeLlama-34B and GPT-3.5-Turbo on the self-debugging setup in the global rebuttal pdf. Comparing it with Table 2 and Table 10 in the paper, we see that with CodeLlama-34B as the teacher, CodeLlama 7B SFT/RL achieves close to CodeLlama-34B self-debugging performance, while CodeLlama-13B SFT/RL significantly outperforms the CodeLlama-34B teacher (e.g. in HumanEval+ pass@1 56.24% vs 48.51%).

## 5. Unit Test at test time
Reviewer: “At test time, when evaluating the SFT/RL models: does the model see the execution results of its generated test code before refinement?”

Response: please refer to “Unit Test at test time” in our first response.

## 6. Reward Design
Reviewer: “why does it make sense to encourage the refinement to be similar to all the possible correct refinements?”

Response:
If we only consider one correct solution, there will be correct refinement that uses a different way but get very low CodeBLEU score. And it is also unreasonable to train the model to follow only one correct solution. Our solution does not penalize the model too much when it uses a different way to solve the problem, as long as there exist correct solutions using the similar way.

Besides, the CodeBLEU score is mainly used to densify the reward distribution, unit test is more important to separate the wrong and correct solutions.

## 7. Generalization
Reviewer: “RL training on benchmark data may over-specialize on their specific domain, while degrading the general coding abilities of the LLM.”

Response:
We have already considered the practical usage of the proposed training, and all of our experiments in the paper use rather comprehensive data, not only the self-debugging data we collected but also the original code generation data provided in the MBPP, APPS, CodeContests training set and Magicoder dataset [2] to avoid over-specialization. Besides, to avoid overfitting, we use a large batch size so the model weights are only updated for about 2000 steps..

We test our models’ perplexity on general code, e.g., 10000 samples from BigQuery Python code. The CodeLlama-7B pre-trained model’s perplexity is 1.457, the SFT model’s perplexity is 1.597, and the RL model’s perplexity is 1.599. Both are just slightly higher than the pre-trained model. **Code generation is one of the most important code tasks for LLMs and our trained model is much better than the pre-trained model on it.** 

**The SFT/RL models’ perplexity on the pretraining data being higher than the pretrained model doesn’t mean they generalize worse. The SFT (or instruction-tuned) LLMs typically have higher perplexity than the pre-trained foundation model, since they learn to follow human instructions.** We also test the instruction-tuned CodeLlama-7B released by Meta (https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf), which gets a perplexity of 1.682, even higher than ours. We generally don’t see concerns regarding the higher perplexity caused by instruction-tuning, because instruction-tuned models follow users’ instructions better and are more useful in developing AI-assistant.

Reference:

[1] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, ICLR 2024.

[2] Wei, Y., Wang, Z., Liu, J., Ding, Y., & Zhang, L. (2024). Magicoder: Empowering code generation with oss-instruct. In Forty-first International Conference on Machine Learning.
Thank you for your response.

Can you please edit your response and mention which part of my review does each part in your response refer to?

For example, I did not ask for a *comparison* to the teacher models.

I also asked:
>>2. In the definition of $S_{cb}$ - if we rely on unit tests to verify correctness, why do we need to encourage the refinement to be similar (in terms of CodeBLEU) to **all** the possible correct refinements?

The authors' response seems to be answering a different question. My emphasis was on the word "all" - why does it make sense to encourage the refinement to be similar to **all** the possible correct refinements? What if there are multiple ways to solve the same problem?

Regarding generalization, the authors mention that the perplexity increases from 1.457 to 1.597, which might be evidence that the model's coding ability is indeed hurt. How can we be convinced that the model does not over-specialize on specific domains?
Thank you for the quick response. 

> If we look at Table 2, HumanEval Pass@1 using CodeLlama-7B. With code explanation and refinement (Expl. + Refine.), prompting’s Pass@1 is 40.13%, SFT’s Pass@1 is 52.98%, RL’s Pass@1 is 55.84%. So SFT increases it by about 13%, and RL (on top of SFT) further increases it by about 3%.
> 
> And also by looking at Table 9, with only single-turn data (MBPP, APPS, CodeContests, MagiCoder’s data), the Pass@1 on HumanEval is 43.88%. SFT (with multi-turn refinement data) can still get 9% improvement.

Apologies for the misquoted numbers and thanks for the clarification. I will update my rating.
Thank you for following up.

## Human Rating
Below is the breakdown of the number of explanations having the score in each range. Although the overall average score is below 3, SFT and RL can generate 24 and 27 explanations with scores higher than 3 (50 samples in total). 

| Score | Prompt | SFT | RL | GPT-3.5-Turbo |
|-|-|-|-|-|
| 4.5 <= score | 1 | 3 | 5 | 13 |
| 4 <= score | 1 | 10 | 11 | 21 |
| 3.5 <= score | 3 | 16 | 17 | 26 |
| 3 <= score | 7 | 24 | 27 | 35|

If we look at poor explanations, the RL model generates a similar number of explanations with “score <= 1.5” compared to GPT-3.5.

| Score | Prompt | SFT | RL | GPT-3.5-Turbo |
|-|-|-|-|-|
| score == 1 | 19 | 6 | 4 | 6 |
| score <= 1.5 | 33 | 14 | 10 | 9 |
| score <= 2 | 39 | 21 | 21 | 14 |
| score <= 2.5 | 43 | 26 | 23 | 15 |

We also find that human annotators tend to be more harsh and give lower scores than GPT-4. Figures 12 and 13 include examples of explanations with “score 4”, which we think the model explains pretty well.


## SFT/RL Improvement
There seems to be some misalignment from the number you mentioned. 

If we look at Table 2, HumanEval Pass@1 using CodeLlama-7B. With code explanation and refinement (Expl. + Refine.), prompting’s Pass@1 is 40.13%, SFT’s Pass@1 is 52.98%, RL’s Pass@1 is 55.84%. So SFT increases it by about 13%, and RL (on top of SFT) further increases it by about 3%.

And also by looking at Table 9, with only single-turn data (MBPP, APPS, CodeContests, MagiCoder’s data), the Pass@1 on HumanEval is 43.88%. SFT (with multi-turn refinement data) can still get 9% improvement.

For OpenCodeInterpreter, it has been trained on high-quality data collected using GPT-4, so the base Pass@1 is already high enough (72%), which may nearly reach the model’s limit and leave less space for mult-turn SFT refinement to improve. That could be why multi-turn SFT only improves 3%. **We think this is not contradictory with our results.**

Our focus is on how to train LLM to self-debug starting from standard code generation training data (MBPP, APPS, CodeContests). Thus we only use GPT-3.5 as the teacher to collect multi-turn data. We also test by using CodeLlama-34B as the teacher (Table 5), and even using CodeLlama-7B itself to self-bootstrap multi-turn data (mentioned in global response). Our method can work without relying on GPTs.

This work is not competing on data quality, so the final results are not supposed to outperform OpenCodeInterpreter. But we would like to add OpenCodeInterpreter as related work and discuss it in our paper.
Thanks for the response. 

> The reviewer might have some misunderstanding of the explanation evaluation ... SFT/RL training significantly improves the explanation quality

From what I can see, the absolute ratings from developers is below 3 on average on a 1-5 scale. While the performance improvement from untrained model baseline is considerable, the absolute scores are still not great. 

I suspect a reason why absolute numbers are low is that not all explanations lead to refinement and the average rating might be higher in the second scenario.

>  The results in Appendix A.4.1 show that even when training with strong code instruction data, the model does not obtain self-debug capability out-of-the-box and it only refines up to 3% of its wrong solutions

Perhaps I was not clear enough -- this work depicts considerable mutliturn improvements from RL training (48% to 57% pass@1 on HumanEval for CodeLLama-7B). My concern is that if the authors start their RL training with a stronger model (say OpenCodeInterpreter-CL-7B ) which already has 70+ pass@1 on HumanEval will the RL training be as effective? For example, OpenCodeInterpreter-CL which trains models with multi-turn SFT dataset only achieves 3% improvements from 72 to 75. This makes it challenging to interpret the performance improvements achieved in this paper.