LeDex: Training LLMs to Better Self-Debug and Explain Code
Authors: Nan Jiang
Xiaopeng Li
Shiqi Wang
Qiang Zhou
Soneya Binta Hossain
Baishakhi Ray
Varun Kumar
Xiaofei Ma
Anoop Deoras
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Code Generation
Code Refinement
Code Explanation
Supervised Finetuning
Reinforcement Learning
Primary area: natural_language_processing

Abstract: In the domain of code generation, self-debugging is crucial. It allows LLMs to refine their generated code based on execution feedback. This is particularly important because generating correct solutions in one attempt proves challenging for complex tasks. Prior works on self-debugging mostly focus on prompting methods by providing LLMs with few-shot examples, which work poorly on small open-sourced LLMs. In this work, we propose LeDex, a training framework that significantly improves the self-debugging capability of LLMs. Intuitively, we observe that a chain of explanations on the wrong code followed by code refinement helps LLMs better analyze the wrong code and do refinement. We thus propose an automated pipeline to collect a high-quality dataset for code explanation and refinement by generating a number of explanations and refinement trajectories from the LLM itself or a larger teacher model and filtering via execution verification. We perform supervised fine-tuning (SFT) and further reinforcement learning (RL) on both success and failure trajectories with a novel reward design considering code explanation and refinement quality. SFT improves the pass@1 by up to 15.92\% and pass@10 by 9.30\% over four benchmarks. RL training brings additional up to 3.54\% improvement on pass@1 and 2.55\% improvement on pass@10. The trained LLMs show iterative refinement ability and can keep refining code continuously. Lastly, our human evaluation shows that the LLMs trained with our framework generate more useful code explanations and help developers better understand bugs in source code.

LEDEx: Training LLMs to Better Self-Debug and Explain Code
Nan Jiang Xiaopeng Li
Shiqi Wang
Qiang Zhou²
Soneya Binta Hossain 3+
Baishakhi Ray²
Varun Kumar
Xiaofci Ma²
Anoop Deoras
Purdue University
2ASS Labs UUnivesity of Virginia
#lang711@purrde edu (xiaopel, wshiqi w rabaisha, kuvrun, xiaaf eim, (aarrasssmmmza.ccoomm com
sh7hv@virginia.
Abstract
In the domain of code generation, self-debugging is crucial. It allows LLMs to refine their generated code based on execution feedback. This is particularly important because generating correct solutions in one attempt proves challenging for complex tasks. Prior works on self-debugging mostly focus on prompting methods by providing LLMs with few-shot examples, which work poorly small open-sourced LLMs. In this work, we propose LEDEX, training framework that significantly improves the self-debugging capability of LLMs. Intuitively, we observe that a chain of explanations on the wrong code followed by code refinement helps LLMs better analyze the wrong code and do refinement. We thus propose an automated pipeline to collect a high-quality dataset for code explanation and refinement by generating number of explanations and refinement trajectories from the LLM itself or a larger teacher model and filtering via execution verification. We perform supervised fine-tuning (SFT) and further reinforcement learning (RL) on both success and failure trajectories with a novel reward design considering code explanation and refinement quality. SFT improves the pass @ by up 15.92% and pass@ 10 by 9.30% over four benchmarks. RL training brings additional up to 3.54% improvement on pass@1 and 2.55% improvement LLMs show iterative refinement ability and can keep refining code continuously. Lastly, our human evaluation shows that the LLMs trained with our framework generate more useful code explanations and help developers better understand bugs source code.
pass @ 10. The trained
Introduction
Code generation has become a crucial research task to automatically generate source code based on natural language description ([H4). Although the recent Large Language Models (LLMs) have shown impressive capability in code generation, generating the correct code for a complex problem in single attempt still challenging [2. This is expected because even for human developers, completing a hard programming problem might need multiple rounds of trial-and-eme
debugging. delf-debugging capability that allows LLMs retrospect the incorrect code and make changes to resolve the errors is becoming increasingly important besides their code generation ability.
investigate off-the-shelf L.MMs in the scale of Codex ccode-davinci-002 [1] GPT-3355 and GPT-4, and show that these LLMS can self-debug the wrong code they generated via prompting methods in pipeline of code generation and self-refinement as shown Figure The
Existing works [13 14
WWok done while interning at AWS AI Labs
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Code Kelinement
write function to find the number of to +i11 with x
count vays(())
above function gets fullisinng
Following errer massage:
above runction gets the
Code
Execution
AssertionError:
Example:
dománoss for the given board.
Genertiin
1f return 1
return
Fall
cassert count_Nays(8) 153
Aassrrt Asssrttiontre Lonbrron:
was
but was 47,
Please exp why the wbove
aeseent count (_ 153,
code i5 krong.
count ways
says (n-1)
count (aays(8) en 153
count_44yss(-22
Piss
Code Rellaement
Code Explanation Reflnement
Task Deseription
Generated Solutions
Foedback
Figure I: Pipeline of letting LLM generate code and self-debug.
user first queries the LLM for solution for the given programming task and the initial solution from the LLM is verified by executing them against the given unit tests. If the solution passes all the unit tests, it is considered correct. Otherwise, the user collects the unit test feedback and forms new query to ask the LLM for refinement. Such process can iterate until the LLM generates a correct solution or reaches the maximum number of iterations. There are different prompt designs when asking for refinement [3. Compared with directly asking for a refined solution (referred to as "Code Refinement" in the feedback block), asking LLMs to provide an explanation of the wrong solution and then refine it in chain-of-thought manner (referred as "Code Explanation and Refinement" in the feedback block) helps it to better understand the unit test feedback and increases the success rate of providing refined solutions (details in Appendix I.
However, how to improve LLMs' self-debugging capability remains under-explored, especially given the fact that open- open-sourced LLMs such as StarCoder [T] and CodeLlama 12 have limited self- refinement performance. For example, the StarCoder-15B model is only able to refine 4.43% wrong solutions for problems from the MBPP benchmark [3], in contrast, GPT-3.5-Turbe can refine 98%90% under the same setting (details in Appendix A.I). Such limited self-refinement ability motivates the need better train I.I.Ms to take feedback explain and self-refine the wrong code. Although important, an essential challenge of training LLMs to explain and refine wrong code is the lack of training data, especially high-quality code explanation data. Previous work has explored Imitation learning from Language Feedback (ILF) [15), which trains LLMs with auman-annotated explanation, yet, such an approach is not scalable and the LLMS also do not obtain the ability to explain code.
In this work, we propose LEDEX, an automated pipeline to collect a high-qualin dataset for code explanation and refinement by generating explanation and refinement trajectories, followed by filtering through execution verification. LEDEX then leaverages the collected data, using supervised fine- tuning (SFT) to significantly improve LLMSs ability to explain and refine incorrect code. Additionally, LEDEX applies reinforcement learning (RL. with novel reward design that accounts for explanation semanties and unit test success, leading to better code explanations and corrections. summary, this work contributes the following:
We introduce LEDEX, scalable framework comprising automated data collection, data validation, supervised fine-tuning, and reinforcement learning with novel reward mechanisms to enhance LLMs' self-debugging capabilities, resulting in more accurate code refinements and insightful code explanations.
We experiment LEDEX on three backbones (StarCoder-1 155B, CodeI-lama-78, and Codel_lama 13B) using code refinements and explanations, initially collected from GPT-3.5-Turke Supervised fine-tuning notably boosts the models' ability to diagnose and correct faulty code, achieving up to a 15.92% improvement in pass @ and 9.30% increase in pass @ 10 across four benchmarks.
LEDEX'S 's reinforcement lcarning on top of SFT, uses a novel reward function that incorporates unit test outcomes and semantic analysis of incorrect code explanations. This further enhances performance, with improvements of up to 3.54% in pass @ and 2.55% in pass 10.
LEDEX is model-agnostic; notably, Jadea-lmm--7B trained on data gathered from Codel Jlama 34B or even itself achieves up to 8.25% and 2. 1.1% gains in pass @ and pass @ 10, demonstrating the generalizability of the approach without reliance on GPT-3.5-Turbo.
2 Approach
Figure 2 shows the overview LEDEX, including the collection of high-quality code explanation and refinement data, and the training methods. LEDEX first collects code explanation and refinement

---

dataset by querying from pre-trained or instruct models and verifying its responses with execution feedback to filter and obtain high-quality explanation and refinement data (steps and 2 in Figure 2
Section 2. Then the high-quality dataset is used for supervised fine-tuning (step 3 in Figure 2
Section 2.2). which significantly improves the model's performance explaining the bug and refining
the code. Reinforcement learning with execution feedback is used to further guide the model to
generate higher quality responses and boost the model performance (step in Figure 2
Section 2.3
Quarying
Solutions
Wrong
Refine.
Solutioes
Expl
Wring Solutions
Expl. Refine.
Truch
Ground
Data Collection
Wrong Solutions
Expl. Refine.
Ddl
Expl. & Refine
Correc Fay
Fuil
& Refine.
Expl. Reward
CodeBLEU Exec, Reward
Data Verification
Supervised Fine-iiuddiinnuunnn Fine-Tuning
Reinforcemment Learning Training
Figure 2: Overview of LEDEX.
2.1 Data collection and verification
We use MBPP (3] (only use the 374 problems in the training set during training), APPS 4 (only the 5,000 problems in the training set) and CodeContests (2] our base training datasets, which contain programming problems and solutions collected from various platforms. While they are helpful for training LLMs for code generation, they neither contain enough wrong solutions nor the explanation and refinement of them. To collect more wrong solutions, we prompt the pre-trained LLMs (i.c., StarCoder and CodeLlama) accordingly with 3-shot examples to sample 20 solutions (temperature set to 1.0) per problem from MBPP's training set, APPS's training set, and CodeContests. We then run these generations against test cases to select the wrong solutions that fail any test cases.
Table shows the number of correct (passing all the unit tests) and wrong (failing any unit test) solutions sampled for each dataset. For each wrong solution, we need explanation of the wrong code and a correct refinement to build the code explanation and refinement dataset. We prompt pre- trained instruction-L. LLMSs with the problem description, wrong solution, and execution feedback (either eлror message or failed test case) to ask for an explanation and refinement. We experimented with @PT-3.5-Turbbe CodeLlama-34B, and CodeLlama-711 for data collection. We take GPT-3.5- Turbo as the example in this section, and an example with it is shown in Appendix A.2 We study the generalization of this data collection with different LLMs Section 4.3
Table I: Number of unique, correct, wrong solutions sampled from pre-trained LLMs, well as the number of correct refinement generated by GPT-3.5-Turie and its refinement rate on each dataset.
Dataset
Uniqqe Solutions
#Correet Solutions
#Wmong Solutions
#Correet Refinement #Rehinement Rate
MBPP
TBPPPTraiiin Training (374)
9,500
4,706
4,794
2,203
45.95%
APPS Training (5,000)
27.736
16,372
39.21%
CodeConiest (6,627)
51,134
31,520
19,614
5,113
26.07%
As LLMs may provide wrong explanations refinements, we cannot blindly take them as training data. Thus, we verify the refinements by running them against the test cases again, and only those passing all the test cases are considered correct refinements. For explanation, we consider the explanations along with the correct refinements as correct. Overall, for example with GPT-3,5-Turbe Turbo, we get 13,735 correct explanations and refinements: 2,203 for MBPP, 6,419 for APPS, and 5,113 for CodeContests. This verification step is crucial to guarantee the quality of the automatically collected code explanation and refinement dataset.
2.2 Supervised fine-tuning
We form the fine-tuning data in an instruction-follloinng format similar to StarChat [16], where the user input is enclosed by <luser>> and </end]>, while L.LM's s answer enclosed by < assistantt and < end >> in the chat history. Moreover, to alleviate the limited amount of data, augment the
3

---

fine-tuning data by using two different instructions: providing the task description, the initial wrong code, and execution feedback, asking for (1) a refinement directly, or (2) an explanation of the wrong code and then a refinement in a chain-of chain-of-thought manner. Examples are given in Appendix A.2
During supervised fine-tuning, although include the wrong solutions as LLM's initial answer in the chat history, we do not calculate the loss for this part since we do not want the LLM to intentionally generate those wrong solutions. They are just provided as context for code explanation and refinement if the LLM indeed makes mistakes in real use cases.
2.3
Reinforcement learning
Reinforcement learning is widely used to further improve the quality of LLM'' generated outputs 17- 20 Through the RL framework, the LLM is optimized by using an algorithm to update the weights using both success and failure trajectories and maximize the rewards of its outputs. To train the fine-tuned LLMs to generate better code explanations and more correct code refinements, we design the rewards considering both parts.
2.3.1 Refinement score
To train LLM to refine code, the correctness of the refinement is the main goal, which can be measured by its code similarity to the ground truth, as well as the execution result. We use CodeBLEU score as metrics for code similarity and unit test passing rate as metrics for execution results.
Given wrong solution w, the of correct and wrong (failed) refinements notated by R. and Rw For any refinement r, we calculate its CodeBLEL score and the unit test passing rate as follows:
[Tp(r)] [T]
(rr(r
CrdeBLEU(r,,ah re); Sw,(r) TEER
Σ
R
Set is the average CodeBLEU score between given refinement and all the correct refinements. Sut is the fraction of the number of passed unit test cases (T,) when running the refined code r, over the total number of unit test cases (I) provided for this problem in the dataset.
In Figure 3] the x-axis is the scores of certain metrics, and the y-axis is the number of training data with a certain score (same for other figures in Figure 33). Thus, Figure (a) shows the frequency distribution of each score of Scb, with blue bars referring to training data with correct refinements,
and orange bars referring to that with wrong refinements. The distribution of Sue is shown in Figure (b) where the correct refinements definitely pass all the test cases and can be separated from the wrong ones.
ArOrO
19k
srrecc wrorg
COFNEC
10K:
contect wrong
Gount
4k:
6K:
20K:
4K:
10K:
0.
022
2.4
0.
0.2
055
0S Lo
[c] Explanation Similaity
0.5
0.5
Code Reward
Explain Reward
Figure 3: The CodeBLEU scores, unit test cases passing rate, sentiment similarity of wrong code explanations, final refinement code reward, and the explanation reward of the training data.
2.3.2 Explanation score
In our dataset, there are wrong code explanations along with code refinement, whose quality may not be perfectly reflected by the code quality. A correct code explanation may also be followed by incorrect refinement, thus, it is necessary to consider the explanation in the reward. The code explanations followed by correct refinements treated as ground truth, notated by E." We calculate the average sentiment similarity (21(22 between the explanation embedding and corresponding embeddings in ground truth
Saz(e) =
CosSin(RoBERIa(e)) RoBERTa(ee))
E

---

The distribution of (e)(e) is shown in Figure 3 (c).
2.3.3 Reward design
Given a pair of explanation and refinement, i.e., (e, the reward of the generated code refinement and the code explanation is designed as:
R(e) 50 Sea Sen 35
R(r) = 5- (Sa(() + (r)(r) 5;
3
This code reward R()) the average of CodeBLEU score and the unit test passing rate, and since both (rr(r) and (r) are scored in the range of [0, I], this equation makes the reward of code refinement in the range of [-5, 5]. Figure (d) shows the distribution of the code refinement reward on the training dataset, which mitigates the overlap issue between correct and wrong outputs with (a). also makes the reward distribution continuous, addressing the discreteness problem of only using unit test passing rate.
CodeBLEU score alone as illustrated in Figure 3
For the design of explanation reward R(e), We observe from Figure 3 (c) that the explanation sentiment similarities of the training data mostly lie in the range of [0.4, 1.0], thus, we project the range of [0.4, 1.0] to [-5, and treat 0.7 as the borderline (projected to 0 correspondingly) of good or bad explanations. Figure 3 (c) shows the distribution of the wrong code explanation reward on the training dataset. The distribution shows that there could be good or correct code explanation followed by wrong code refinement, where assigning high or low reward to the entire output is reasonable. This leads to our PPO [23] algorithm with code refinement and explanation rewards considered separately. Due to space limit, the PPO algorithm is shown in Appendix A.3
3 Experimental setup
For supervised fine-tuning, we fine-tune three LLMs (StarCoder- 15B, CodeLlama-7B. and CodeLlama-133h using the correct initial solutions and correct refinements collected from the MBPP training set, APPS training set, and CodeContests. The model fine-tuned for two epochs, using a batch size of 128. The optimizer is AdamW [24] with learning rate set to 2e -, The learning rate is adjusted using a warmup of 500 steps and then decayed following a cosine scheduler.
We further train supervised fine-tuned LLMs with reinforcement learning using the PPO algorithm. The reinforcement learning training data all the initial solutions and collected refinement on the MBPP and APPS training set. The learning rate is 2e-6 and the batch size is set to 64. We implement reinforcement learning training based on the TRL (25) library. Both the supervised fine-tuning and reinforcement learning are conducted on 8 NVIDIA A100 GPUs, each with 40GB of memory.
4 Result
To evaluate the effectiveness of LEDEX, we study the following research questions (RQs) regarding code generation and code refinement capability, iterative refinement ability, approach generalizability, and the quality of the generated code explanations.
4.1 RQ1: Code generation and refinement capability
We evaluate the models trained with LEDEX for their code explanation and refinement ability using four benchmarks: MBPP 3, HumanEval [ MBPP+ (26], and HumanEval* [26]. We use @ass@k [ and success refinement rate as the evaluation metric. For the generation of the initial solutions, the models sample 100 solutions per task in the benchmarks (temperature set to 0.8), which are run against the provided test cases. For every incorrect solution that fails any test case, we let the models sample one refinement (and one explanation).
4.1.1 @
Table 2 presents the pass @ k results across four benchmarks. Overall, fine-tuning the LLMs with our curated dataset of code explanations and refinements leads to substantial improvements both @ @1 I and pass @ 10 for all three model architectures. For StarCoder- 15BB and CodeLlama-
5

---

Table 2: Pass @k of initial and refined solutions on four benchmarks. Each backbone's best perfor- mance on every benchmark bolded.
Models
Approuches
MBPP
Humnavaal
MBPP
HumanEval"
pass@1
pass@
pass
pass
pass @
pass
pass @
pass 10
Init.
37.70
69.60
30.34
63.21
34,90
60.85
26.16
56.77
Prompt.
Refine
40.64
71.04
34.05
40.48
30.73
Expl. Refine
40.37
71.60
33.96
67.20
39.23
64.67
30.09
60.72
47.41
35.05
43.28
59.77
StarCoder: 15B
LEDEX SFT
Refine Expl.
$6.66
75.78
47.32
77.06
53,53
71.55
43.16
71.74
57.11
76.70
47.37
53.83
72.19
Init.
48.62
70.68
39.45
70.16
44.94
63.92
35.05
64.41
LeDEX RI.
Refine
58.00
78.12
52.25
80.80
54.13
71.71
46.11
74.07
Expl. Refine
$8.19
77.96
51.67
80.79
54.29
71.93
46.26
74.24
Init.
38.21
67.24
34.27
69.60
37.18
61.23
27.40
60.81
Prompt.
Reline
43.98
71.94
39.20
74.14
42.97
66. 86.89
31.84
65.08
Expl. Refine
43.42
72.09
40.13
74.95
42.46
67.41
32.49
66.58
Codel lama-- LEDEX SFT
Refine
$8.07
77.34
52.65
80.71
51.64
71.04
46.61
74.43
Expl. Refine
57.98
77.92
52.98
82.22
51.55
70.94
47.62
75.54
Init.
46.54
71.54
39.38
71.84
41.46
63.68
33,95
98
Reline
Expl. Refine
57.92
78.97
55.84
84.14
52,90
71.80
50.04
78.25
IniL.
42.88
70.85
37.11
74.69
38.93
62.26
30.15
66.27
Prompt.
Refine
49.68
75.85
45.78
81.07
46.22
70.14
37.62
72.68
Expl. Retine
Init.
52.43
41.65
73.61
43.67
35.29
68.49
CodeLlaama-334 13B LEDEX SFT
Retine
61.78
79.96
58.41
83:47
55.58
73:41
51.35
77.84
Expl. Reline
61.59
80.21
57.76
84.57
54.59
72.15
51.32
78.84
Init.
$1.19
73.16
45.45
74.79
45.49
62.81
39.27
69.29
Refine
Expl.
Refine
61.63
80.27
61.66
86.23
56.62
72.04
56.57
81.77
Table 3: Overall pass@k on MBPP & HumanEval and MBPP & HumanEval+ Blue or red numbers
show the improvement or deterioration: SFT compared to prompting, and RL is compared to SFT.
StarCoder-155
MBPP & Humankival
MBPP & HumanEival
LEDEX SFT
LEDEX RL
LEDEX SFT
LEDEX RL
Retine
Expl. Refine
Refine
Expl. + Retine
Refine
Expl. Retine
Expl. + Refine 51.02 72,67 +104 72,87 40.51
Retine
pasa@0
54.35 +15.34
54.71
56.58 78.78 +2.55 78.66
41280 49.64
pass 10
76.23 46.58
76,81 46.30
71.63 +7.27
72.36 49.30
CodeLlama-7B
& HomanEval
MBPP & HumanEval"
LEDEX SFT
LEDEX RL
LEDEX SFT
LEDEX RL
Retine
Expl. Refine
Refine
Expl. Refine
Refine 49.59 +11.5
Expl. Retine
Expl. + Refine 50.73 +111 51.74 +1.99
Refine
@ass
57.21 +1441
56.75 +14.14
56.62 0.60
57.41 +0.66
49.95 +11.5
6ass
78.17
78.98 46.18
79.21 +1.04
80.25 +1.27
72.42
46.27
71.81 44.74
74.03 +1.1 74.42 .2.00
CodeLlama-
MBPP & HomanEval
MBPP & HumanEval*
LEDEX SET
LEDEX RL
LEDEX SFT
LEDEX RL
Refine 60.95 12:23
Expl. Refine
Refine
Expl. Refine
Refine
Expl. Refine
Refine 57.40 +3.54 56.60 +3.24
Expl. Refine
@
60.64 +11.88
61.91 +0.96
61.64+1. +1.00
53.86 +11.14
53.26 +10.51
@ass
81.74
75.21
76.50
76.00
RL achieves the highest pass @ I and pass @ 10 scores (bolded) across all four benchmarks. For CodeLlama-7BB RL achieves the best performance in seven out of eight cases, with SFT yielding the highest pass @1 score on the MBPP benchmark.
For easier comparison, Table T the overall pass@k results MBPP and HumanEval, along with the improvements achieved through SFT and RL. The improvements from SET are compared direct prompting, while the improvements from RL are relative SFT.
MBPP and HumanEval overall, SFT boosts StarCoder- 15B's pass@) by 15.34% and pass@ 10 by 6.58% when directly generating code refinements. When incorporating code explanations in chain-of-thoughe format, SET further enhances StarCoder- 15B's performance by 15.92% on pass @ and 6.30% pass @ 10. RL brings additional 2.43% improvement in pass @ and 2.55% in

---

pass 10 for direct refinement,, and further 1.87% pass @ I and 1.85% pass @ 10 increase when generating both code explanations and refinements. Comparable improvements from SFT and RL are models as well.
observed across the CadeL-lama-78 and
On the MBPP and HumanEval+ benchmarks, which feature more rigorous test cases, respec- tively [26]. we observe even greater improvements from RL training on the CodeLlama models. CodeLlama-7B achieves a 1.79% increase in pass @ and 2.60% increase in pass @ 10 for refined solutions with code explanations. CodeLlama-Iae 13B shows a 3.54% improvement in pass @ and 1.29% improvement in pass @ 10 for directly generated refinements. These results demonstrate that RL training enables LLMs to produce or refine solutions that are more robust and capable of passing stricter cases. Additional experiments and detailed case studies can be found in Appendix A..41
A.5.I
A.5.2. and A.5.3
4.1.2
Success refinement rate
Table 4 presents the refinement success rate for each model backbone across various approaches, averaged over four benchmarks. For StarCoder- 15B, the baseline prompting method struggles, achieving only a 6.41% % to 6.90% success rate in refining incorrect initial solutions. However, after applying SFT with the high-quality dataset containing code explanations and refinement trajectories, StarCoder- 15B demonstrates a notable improvement, raising its refinement success to 16.27% to 16.56%. This increase represents significant gain of 9.37% 10.15% over the prompting baseline, showcasing the effectiveness of SFT in enhancing code refinement capabilities by leveraging targeted training data. With further RL, the refinement success for StarCoder- 15B improves even more, adding an additional 1.03% to I 1.23% over the results from SFT. This final boost highlights the complementary strengths of RL, particularly its capacity to fine-tune model behavior beyond what supervised methods can achieve.
The improvement CodeLlama-78 and CodeLlama- 13B backbones is consistent with that StarCoder- 15B, where RI. training eventually achieves the highest success refinement rate with a considerable boost of 1.81 3.62%.
Table 4: Success refinement rates over four benchmarks. Blue numbers show the improvement.
Models
Refine (%) LEDEX
Explain Refine (%)
Prompt.
LEDEX
Prompt.
LEDEX
StarCoder- 15B
6.90
16.27 49.37
17.50 -1.23
6.41
16.56 +10.15
17.59 +1.03
Codel.lama- 7B
8.65
18.14
19.95 -181
8.10
17.60
20.84 43.24
CodeLlama- 13B
11.64
18.96 47,32
22.58 -3.62
11.97
20.06 +8.00
23.50 +3.44
4.2 RQ2: Iterative refinement ability
LLMs have the ability to iteratively self-debug until they arrive at correct solutions. Figure Sitreeeillusee illuttrtee the overall pass@k of CodeLlama-7B across four benchmarks after up three rounds of refinements. To simplify the figure, we plot the higher pass @k from either the "Refine" or "Expl. Refine" approach each refinement round for each model. Additional results on iterative refinement are provided in Appendix A.4.2
paseg1
(b) passgg10
passa1
paese810
HumenEvalt
Pronpt
8FF
Procept.
SFF
Pramol
5FT
Prompt
SPT
64
823
8100
82
25
14.75
71.5
80:23
78.93
31.00
$2.39
7637
49
54.88
953
72.81
76.35
7501
67.07
43.04
20 c4
11.2
01.05
1otRefiee
tooRoline
Rofre
Sd.Sat.
fatifaliiee 2nd Retne 3rd Ratre
fxtileiie 2nd Retre Rrr
Figure 4: @ass@k of prompting, SFT, and RL CodeLlama-785 after three iterations of refinements.
Both SFT and RI. consistently outperform prompting across all three refinement rounds. Even after three rounds, the prompting approach fails to match the pass @k achieved by SFT after just the first
7

---

Table 5: @
@k of CodeLlama-7B trained with CodeLlama-34B data.
Models
Appronches
MBPP
Humanval
MBPP*
HvmanEval
pass11
pass@10
pass@
pass @ 10
pass
pass
pass @
pass@
Init.
47.01
71.18
39.24
70.00
41.64
61.95
33.07
63.58
LEDEX SFT
Refine
54,97
76.52
77.96
48.71
68.28
71.46
CodeLlamm--7B
Expl. Refine
55.05
76.63
47.34
78.11
49.15
68.35
41.49
71.76
Inii.
46.45
70.41
39.63
71.02
42.34
60.84
34.15
63.19
LEDEX RI.
Refine
54.81
76.56
48.28
80.06
53.06
70.17
47.40
73.81
Expl. Refine
55.32
76.74
48.52
79.14
53.17
48.59
75.80
Table 6: Overall pass @k on MBPP & HumanEval and MBPP & HumanEval+ trained with CodeLlama-34B's data. Blue numbers show the improvement.
CodeLlama-7B
MBPP HumanBval
MBPP HumanBval*
LEDEX SFT
LEDEX RL
LEDEX SFT
LEDEX RL
Refine
Expl. Refine
Refine
Expl. Refine
Retine
Expl. Refine
Refine
Expl. Refine
pass
30.01 +7.21 53.15 -10.5
53.20 +0.0 53.64 .044
45.62 +7.18 46.03 -763
50.76 15.1 51.31 +5.28
pass
76.88
77.42 77.33
69.57 69.74
71.65
71.98
round (e.g., 47.53% vs. 56.75% in Figure E (a)). These results demonstrate that LLMs trained with
our pipeline possess strong iterative refinement capabilities, enabling them achieve progressively higher pass@k with each additional round of refinement.
4.3
RQ3: Generalizability of approach
4.3.1 Data collection using open source LLM
To demonstrate the generalizability of LEDEX, particularly the independence of our data collection process from GPT-3.5-Turbo As CodeLlama-34B is pre-trained model, we incorporate few-shot examples in the prompts to guide the generation of incorrect code explanations and refinements. All other processes remain unchanged.
substitute GPT-3.5-Turbo with 3o44Llama-334B for data collection.
presenss the pass@ results for CodeLlama- 7B trained on data collected from CodeLlama- 34B, with Table 6 providing overall comparison. Although SFT achieves slightly smaller improvements (around 1-3% lower than with GPT-3.5- Turbo data), it still yields notable gains in overall pass @ and pass @ 10. Additionally, and HvmanEval+ benchmarks, with pass @ improving by 5.28% and pass @ 10 by 2.24%. These results demonstrate the generalizability of LEDEX and further suggest that collecting data from more powerful LLM can lead to better training outcomes within our framework. Additional results can be found in Appendix A.4.3
observe that RL training further enhances performance
the MBPP
4.3.2 Data Collection Using Self-Bootstrap
We also investigate the feasibility of using LLM to self-bootstrap its training data, specifically by using Cade7- to generate the data that then used for its own SFT and RI training.
Table presents the pass@k k results of Codel.lama-7B trained with self-hootstrapper data, with Table showing the overall comparison. Compared to prompting, SFT with solf-bootstrapped data still delivers up 8.25% and 2.14% improvements in pass @ and pass @ 10 on MBPP and HumanEval, and up to 5.33% and 0.55% improvements on pass@) and pass @ 10 on MBPP and HumanBBal+. Additionally, RL training using the self- self-bootstrapped data results in a further 0.7 % improvement on MBPP and HumanEval, and up to a 0.78% increase on MBPP* and HumanEval* These findings suggest that while self- uuffbootstrapped data enables SFT to provide substantial gains over prompting, RL training offers less improvement compared to using data from stronger LLMs, such Codel_lama-344 GPT--35-TTurbo.
4.4 RQ4: Quality of generated explanation
We assess whether explanations for incorrect code are useful for developers in understanding their bugs. To do this, we randomly sample problems with initial incorrect solutions from the MBPP and HvmanEval benchmarks and use different LLMs to generate explanations for the wrong code.

---

Table 7: Pass@k of CodeLlama-7B trained with self colf-bootstrapped data.
Models
Approaches
MBPP
Humanval
MBPP*
HumannEval
pass
pass
passr0
puss 10
puss
pass GP]
pass/l
Init.
45.83
69.24
39.85
68.83
41.75
33,25
61.50
LeDEX SFT
Refine
52.37
74,63
47.04
74.58
46.26
66.39
40.15
65.15
CodeLlama-778
Expl. Refine
51.80
74,99
45.70
74.72
45.94
65.77
39.10
67.33
Init.
46.28
68.87
39.90
69.49
41.61
61.29
33.66
62. 17
LEDEX RL
Refine
52.84
74.46
48.37
75.54
46.28
65.86
41.54
68.14
Expl. Refine
52.34
74.60
46.90
75.70
46.10
65.99
40.79
68.50
Table 8: Overall p on MBPP & HumanEval and MBPP* & HumanEval+, trained with self- bootstrapped data. Blue red numbers show the improvement deterioration.
CodeLlama-7B
MBPP HumanEval
MBPP & HumanEval'
LEDEX SFT
LEDEX RL
LEDEX SFT
LEDEX RL
Refine
Expl. Refine
Refine
Expl. Refine
Refine
Expl. Refine
Refine 44.35 66.79 40.09 67.01 +0.61
Expl. + Refine 43.94 +0.78
pass @
$1.05
50.29 +7.68
51.74
51.00 +0.1
43.77
43.16 +4.76
pass 69 10
74.62
x2.14
74.92 +2.12
74.73 +0.11 74.87 4005
66.70 +0.55
66.40 4.67
Each explanation scored on scale from 1 to 5, based on its correctness and helpfulness, where I indicates completely incorrect or misleading explanation, and 5 denotes correct explanation that also provides detailed hint on how to fix the code. Both GPT-4 and human developers are used as evaluators. For GPT-4, we follow prior work [27] and prompt it to score each explanation. The results are presented in Table Both SFT and RL. lead improved explanation quality compared to prompting, with GPT-4 assigning higher scores to models trained using our approach. Notably, the gap between GPT-3.5-Turbo and the trained LLMs significantly narrows after fine-tuning.
Given the time required for human evaluation, we only asked developers to rate explanations from StarCoder models and GPT-3.5-Turbo. Each explanation is scored by two developers, and their ratings are averaged. The human evaluations align with GPT-4's, confirming that SFT improves explanation quality over prompting, while RL further enhances explanations by incorporating code explanation semantics into the reward design. Detailed rubrics and examples of human evaluations can be found in Appendix [.
5 Limitation
One potential limitation of our study is the reliance specific large language models (LLMs) from which we collect code explanation and refinement data. Our automated framework is designed to be independent of any specific LLM, and for this study, use GPT-3.5, CodeLlama-34B,, and Codel itself to collect training data, and both bring significant improvement through SFT and RL. However, for future work, it would be interesting to explore the of other LLMs, including smaller models or a mix of diverse LLMs, to gather explanation and refinement data.
Additionally, our current experiments only use two types of prompts for enabling LLMs to self-debug: one that directly asks for refinement and another that first asks for an explanation the wrong code followed by refinement. While these prompt designs have shown effectiveness, there might be better prompt strategies for self-debugging that we have not explored due to resource constraints. Exploring broader range of prompt designs could potentially enhance the performance of our framework. Nonetheless, our proposed training framework is flexible and should be generalizable to different types of data and prompts, paving the way for future innovations in this area.
6
Related work
6.1 Large language models for code
Large language models (LLMs) have been widely explored across a variety of code-related tasks, including code generation 5 8 7
34 bug fixing 38) program testing [39 40) and fuzzing [1 and so on. These models have demonstrated impressive capabilities in these domains, largely due to their strong understanding and generation abilitics acquired through extensive pre-
10
9
12

---

Table 9: Average scores of code explanations rated by GPT-4 and developers. SC for StarCoder and CL for CadeLlama. "." refers to not applied.
Raters
Prompt. SC-15B
SET SC-15B
RI. SC-15B
Prompt. CL-7B
SFT CL-7B
RL CL-7B
Prompt. CL-13B
SFT CL-13B
RL CL-13B
Prompt. Tur-..5-Turb
GPT-4
1.90
2,92
3.18
2,00
3.00
3,10
2,54
2,62
3.24
3.60
Developers
1.73
2.60
2.76
3.21
training on vast datasets. This pre-training allows them to recognize patterns, understand context, and generate coherent and contextually relevant code snippets.
However, most existing works in this area focus primarily on improving LLMs to provide the expected output in single round of generation. The emphasis has been enhancing the initial output quality, minimizing the need for further modifications or iterations. This one-shot generation approach, while useful, overlooks the potential of iterative refinement, which is crucial aspect of real-world programming where initial drafts often require multiple rounds of revision and debugging.
6.2 Self-debugging and self-refinement
Existing techniques have studied the possibility of using LLMs to refine their generations. Yet, most techniques are prompting I.LMs with execution results [
42 48 for the refinement.
Such prompting approaches bring limited improvement to smaller open-sourced LLMs compared to GPT-3.5. Other techniques train LLMs to self-debug. ILF [49 uses human-annotated feedback information and thus is unscalable, CYCLE (50) and Self-Edit [51] use SFT to fine-tune LLM to generate the refinement only based on the unit test execution feedback. OpenCodelnterpreter [52] and EURUS (53) construct high-quality multi-turn interaction datasets using GPT-3.5-Turbe and
GPT-4 to fine- tune LLM for reffneefineenn.
This work has four differences compared with others that train LLMs: (1) we train LLMs to generate code explanation followed by refinement, which provides additional information to users, (2) we do not require human-annotated training data but propose scalable pipeline to automatically collect and verify data from another L.LM, (3) our data collection pipeline can be generalized to open-sourced LLM even the same LLM itself, and (4) we design novel reward functions in the RL training stage, considering both the code and explanation quality, which beings extra improvement.
Conclusion
This work highlights the importance of training open-source LLMs to self-debug and introduces scalable framework that includes automated data collection, verification, supervised fine-tuning, and reinforcement learning with novel reward designs to enhance LLMs' self-debugging capabilities. Our data collection process model-agnostic, as demonstrated by the improvements achieved with both GPT-3.5-Turbe and CodeLlama. The data verification ensures high quality of code explanations and refinements. Fine-tuning on this data significantly boosts the LLMs' self-debugging abilities, yielding up to 15.92% increase in pass @ 1, 9.30% increase in pass @ 10, and 10.15% increase in successful refinements. Reinforcement learning. utilizing our novel reward design, further enhances performance, with additional gains of up to 3.54% in pass @ 1, 2.55% in pass@ @ 10, and 3.62% in successful refinement rates. Comprehensive analytical experiments confirm the generalizability of approach and demonstrate the iterative refinement capabilities of the trained models. Moreover, human evaluations indicate that the LLMs trained with our framework produce higher-gu higher-quality explanations, effectively aiding developers in understanding and resolving bugs in source code.
References
[1] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Flizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain,
10

---

William Saunders, Christopher Hesse, Andrew N. Carr. Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGirew Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
[2] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser. Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang. Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.07814 2022.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.
[4] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurlPS, 2021.
[5] Yue Wang. Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In EMNLP, pages 8696-8708. Association for Computational Linguistics, 2021.
[6] Erik Nijkamp, Bo Pang. Hiroaki Hayashi, Lifu Tu, Huan Wang. Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. arXiv preprint, 2022.
[7] Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. Codegen2: Lessons for training iims on programming and natural languages. arXiv preprint, 2023.
[8] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D.Q. Bui, Junnan Li, and Steven C. H. Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint, 2023.
[9] Fried Daniel, Aghajanyan Armen, Lin Jessy, Wang Sida, Wallace Eric, Shi Freda, Zhong Ruiqi, Yih Wen-tau, Zettlemoyer Luke, and Lewis Mike. Incoder: A generative model for code infilling and synthesis, 2023.
[10] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. In KDD, 2023.
[11]
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel LaiyePoirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin. Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang. Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Naday Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt Dan- ish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! 2023.
[12] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défosse7, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilaaa: Open foundation models for code, 2023.
11

---

[13] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, 2023.
[14] Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar- Lezama. Demystifying gpt self-repair for code generation, 2023.
[15] Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R. Bowman, Kyunghyun Cho, and Ethan Perez. Improving code generation by training with natural language feedback, 2023.
[16] Lewis Tunstall, Nathan Lambert, Nazneen Rajani, Edward Beeching, Teven Le Scao, Leandro von Werra, Sheon Han, Philipp Schmid, and Alexander Rush. Creating a coding assistant with starcoder. Hugging Face Blog, 2023. https:/huggingface.com htt.g/aaa...ccoo///.eee///// www.llog/starrcaa.t..t.
[17] Rafailov Rafael, Sharma Archit, Mitchell Eric, D Manning Christopher, Ermon Stefano, and Finn Chelsea. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[18] Parshin Shojaee, Aneesh Jain, Sindhu Tipireeni, and Chandan K Reddy. Execution-based code generation using deep reinforcement learning. arXiv preprint arXiv:2301. 2023.
[19] Le Hung, Wang Yue, Deepak Gotmare Akhilesh, Savarese Silvio, and C.H. Hoi Steven. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. arXiv preprint, abs/2207.017888888 0017800 2022.
[20] Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. Rltf: Reinforcement learning from unit test feedback, 2023.
[21] all-roberta-large--1, 2024.
[22] Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2020.
[23] Schulman John, Wolski Filip, Dhariwal Prafulla, Radford Alec, and Klimov Oleg. Proximal policy optimization algorithms, 2017.
[24] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
[25] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan
Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. 2020.
[26] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. GPT really correct? rigorous evaluation of large language models for code generation. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
[27] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li. Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt- bench and chatbot arena, 2023.
[28]
DeepSeek. Deepseek coder: Let the code write itself, 2023.
[29] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[30] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
12

---

[31] Damai Dai, Chengqi Deng, Chenggang Zhao. R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng. Xingkai Yu. Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-expprts language models, 2024.
[32] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang. Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li. Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation, 2024.
[33] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code all you need, 2023.
[34] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct, 2023.
[35] Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. Impact of code language models on automated program repair. In Proceedings of the 45th International Conference on Software Engineering. ICSE 233 page 1430-1442 IEEE Press, 2023.
[36] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the era of large pre-trained language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1482-1494, 2023.
[37] Yi Wu, Nan Jiang, Hung Viet Pham, Thibaud Lutellier, Jordan Davis, Lin Tan, Petr Babkin, and Sameena Shah. How effective are neural networks for fixing security vulnerabilities. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, page 1282-1294 New York, NY, USA, 2023. Association for Computing Machinery.
[38] Soneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li, Wen-Hao Chiang, Yingjun Lyu, Hoan Nguyen, and Omer Tripp. A deep dive into large language models for automated bug localization and repair. arXiv preprint arXiv:2404.11 2024.
[39] Sungmin Kang, Juyeon Yoon, and Shin Yoo. Large language models are few-shot testers: Ex- ploring Ilm-based general bug reproduction. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 2312-2323. IEEE, 2023.
[40] Soneya Binta Hossain and Matthew Dwyer. Togll: Correct and strong test oracle generation with 1mss. arXiv preprint arXiv: 2405.3740 2024.
[41] Yinlin Deng, Chunqiu Steven Xia. Haoran Peng, Chenyuan Yang. and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the 32nd ACM SIGSOFT international symposium software testing and analysís, pages 423 435, 2023.
[42] Dong Huang, Qingwen Bu, Jie M. Zhang, Michael Luck, and Heming Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation, 2024.
[43] Lily Zhong, Zilong Wang, and Jingbo Shang. Ldb: A large language model debugger via verifying runtime execution step-by-step. 2024.
[44] Hanbin Wang, Zhenhaao Liu, Shuo Wang, Ganqu Cui, Ning Ding, Zhiyuan Liu, and Ge Yu. Intervenor: Prompting the coding ability of large language models with the interactive chain of repair, 2024.
13

---

[45] Xueyu Hu, Kun Kuang, Jiankai Sun, Hongxia Yang, and Fei Wu. Leveraging print debugging improve code generation large language models, 2024.
[46] Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt, 2024.
[47] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023.
[48] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.
[49] Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale, 2023.
[50] Yangruibo Ding, Marcus Min, Gail Kaiser, and Baishakhi Ray. Cycle: Learning to self-refine the code generation. Proc. ACM Program. Lang., 8(0OPSLA1), April 2024.
[51] Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. Self-edit: Fault-aware code editor for code generation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), pages 769-787, Toronto, Canada, 2023, Association for Computational Linguistics.
[52] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. OpenCodelnterpreter Integrating code generation with execution and refinement. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computationnll Linguistics ACI. 2024, pages 12859, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics.
[53] Lifan Yuan, Ganqu Cui, Hanbin Wang. Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng, Zhiyuan Liu, and Maosong Sun. Advancing Ilm reasoning generalists with preference trees, 2024.
[54] S. Kullback and A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79 86, 1951.
A Appendix
A.1 Limited refinement ability of open-sourced LLMs
We evaluate two prompting strategies for querying LLMs to perform code refinement: one where the LLMs directly generate the refinement (Figure (3.1)), and another where they first explain why the code is incorrect before generating the refinement (Figure (3.2)). Table reports the success rates of refinements generated by StarCoder-15B: CodeLlama- 7B. CodeLlama- 13B, and GPT-3.5 using greedy decoding. Generally, LLMs that first explain the incorrect code are more likely to generate accurate refinements. However, despite these efforts, all three open-source LLMs exhibit limited refinement capabilities. For example, StarCoder- 15B successfully refines only 4.43% to 5. 10% of incorrect code, underscoring the need for training open-source LLMs enhance their ability to explain and refine code.
Table 10: The success rate self-refinement using greedy decoding.
Models
MBPP Expl. Refine
HumanEval Expl. Refine
Refine
Refine
StarCoder-150 15B
2.58
4.43
2.04
5.10
CodeLLamma-77
CodeLLamm-115
9.80
10.20
5.88
8.24
GPT-3.5
26.24
28.90
32.32
33.33
14

---

def sun elenents(nuns): sun(nuns) forction fins
surfacearns rohe :
Seapple:
assert surfacaarce Sphereis)) 2527.455 455
nanelretr: DAP "moth" is
Saluiion Teat Frethback
mettrrr hame suih rot Deflines
Remdbrorr Hues "muln" is not oefungs
Biesse fra Sey
plrnaie eaplatin why the aboue rode arvay, aet Fiy
"Hanstiorr: tume "mitl" detines" inlicetoo the the code wiist provides natismuticed funclions, but been inportez. peliw code.
der sunfacconoa, ( reture nathip. rallas
inport carh
return nethoas madics
AS. Self-eefne
(3.)) for Wrung Solution Explanation Self-Refine
Two different prompts to ask LLM to self-refine: directly asking for refinement (left), asking an explanation of the wrong code, and then refining chain-of-thought (right).
Figure 5:
have Python prograening task to salve, but ny solution got same error ressages. tan you explain
solution wrong, and propose the cornect code for me?
#et Task tt8
Problem
Mrite function to find the sumation of tuple elements in the given tuple list.
Description
20t My wrong code dit
sun clenents(nunss::
Wrong
ros sun([nurs])
Solution
return res
Error message
TypeErrar: unsupparted operand type(s) for 4: "int" and "tuplo"
Feedback
#pe Explanation of code ###
cyour explanation to wrong code>
ven Correct code tift
syour cornect
Explanation wrong
The wrong code uses the sun()" function incorractly. The "sun()" function expects an iterable as its the variable already tuple.
but in this "sun()" function, it treats "nuns" as single elenent the iterable, which is tuple. Hence, it tries to add the tuple to an integer, resulting in Typefrror".
[[uus]
Wrong Cade Explanation
Correct code
Code Refinement
neturn res
Figure Prompt used for code explanation and refinement data collection.
A.2 Data collection and data format for training
Figure 6 provides an example of the prompt we used to collect code explanation and refinement training data from GPT-3.5 (the same approach used for 3o44L3ama-34BB We the temperature to 0.8, allowing GPT-3.5 to generate 10 code explanations and refinements per prompt. The collected data includes both the explanation of the incorrect code and the corresponding refinement. From this, we construct two formats of instruction-tuning data for supervised fine-tuning (SFT).
Figure (left) shows instruction that asks LLMs to provide only the code refinement, where the instruction includes the task description, the incorrect initial solution, and execution feedback. The LLMs trained to generate the refined solution based this input. Figure 7 (right) presents different instruction that asks for both the code explanation and the refinement. By incorporating both types of instruction data during training, increase the diversity of the training set and improve the LLMs' robustness across different prompts.
A.3 PPO algorithm with separate rewards RL
We modify the standard PPO algorithm to optimize the explanation and refinement generation based on their reward separately. Given an LLM (already supervise-fine-uuss π as the policy model, a prompt v, generation y consists of code explanation and refinement:
je,rl = (31. V2. ... M/e]+ M1c M|e411,)1/222... 3/e]- Mee|+ri
is the refinement. We then define the [d. lel is the explanation of the wrong code, and {3x 1
|e|+|r|
advantage A in the PPO algorithm as:
15

---


 hrite a function to find the sumsation of tuple elesents in the given tuple list. </end]>
<lusen|> write Function to find the summation of tuple elements in the given tuple list. </end]>
Instruction
<jassistant]>
<lassistant|> def ( res suet[ [nu[]]
def sun clesents(rues))
res sun([runs])
Initial Solution
return res
Input Instruction
return res
</end]>
<|end]>
<lusen|> Running above returns following error nessage:
juser|> Running the above code returns the
Typesrror: unsupported operand typa(s) For +: 'int" and 'tuple'
following
typetrror: unsugported operand type(s)
Sn the code does not the assention, Please:
Feedback
"intt
2. Explain the above [and]
So the code does not pass the assention.
Please fix it.
|end]>
slacsistant]]
lassistantt
The wrong code is using the "sun()" functionn incorrectly. expects
def zun cnnsentss(suuss):
res sum(nuns)
"sun()"
argunent, already tuple...
Output
return ras
def sun (nuneetsssssssss)
Refinement
<lend]>
res sun(nuns)
return res
</end]
Instruction only asking for code refinement
Instruction asking code explanation code refinement
Figure 7: Two types of instruction tuning data used in SFT and RL.
A, 6, + 78441 4
V(yet, +V ((yett,,
A, is the advantage at decoding timestamp T =le] + |rl is the total length of generation output, and y the discount rate hyper-parameter set to 0.99 our experiment). V(y=t, π is the state value at generation step given input which is learned and calculated by linear layer on top of the policy model n. R2 the reward at decoding timestamp which is calculated as follow:
R(r) - -KT(TT x')
t T
R2 R(e) - (T,(T, x')
le|
x') log P(w R)
otherwise
1,
where KL is the Dallback-Leiblee divergence [54 between the action distribution given by the updated policy model æ and the old policy model n' before the update.
Instead of assigning both the code and explanation rewards to the entire output, we separate them by only assigning the explanation reward the explanation portion. This avoids the issue that a low reward is assigned to a correct explanation followed by wrong refinement, and the LLM learns to keep the correctly generated explanation part and focus on improving the incorrect refinement portion. For data where only code refinement but no explanation generated, we use the standard design to assign the code reward to the code refinement.
Following existing works, the loss of the PPO algorithm training is:
log P( (V, T) 1' + oF Σ (Viee, (A, + (,
L = E
log P(uel
By minimizing this loss, the policy model (the LLM under PPO training) is trained to generate explanations and refinements with higher reward but also constrained by not being distracted too much far away from the supervised- snpervised-iie-tuneed LLM n'.
A.4
Additional results
A.4.1 Comparison with SFT on code generation
Table 11 shows the comparison between CodeL-amm-77B trained with LEDEX, and the CodeL trained with code generation data only. Although training with code generation enables L.LMs get comparable (or even higher pass @k on HumanEval) pass@k @k on one round of code generation, the
16

---

LLMs cannot obtain strong self- self-debugging ability from code generation data. The LLM trained with code generation data can only improve the pass@k very little after -ellgibuugg.ng.. By contrast, the LLM trained with the full collected data (code generation, code explanation, and code refinement data) gets significantly higher pass@ after refinement, showing strong self-debugging ability.
Table 11: Pass @k MBPP and HumanEval by Codel.Jam--77 trained code generation only, and that trained with our collected code explanation and refinement data.
Models
Approaches
MRPP
Humanval
MRPP
HumanFval*
pass 6
pass 10
pass@1
pass 10
pass011 1
pass
pass
pass 10
Init.
46.18
70.56
40.14
70.37
41.23
60.58
34.22
63.34
Only Code Generation Data
Reline
49.18
72.78
42,91
74.57
43.13
61.81
36.12
65.77
Expl. Refine
73.07
43.88
75.19
43.58
36.37
Init.
48.87 87
70.89
36.99
69.95
42.97
62.69
30.76
62.52
LEDEx's Full Data
Refine
58.07
77.34
52.65
80.71
51.64
71.04
46.61
74.43
Expl. Retine
57.98
77.92
52.98
82.22
$1.55
70.94
47.62
75.54
A.4.2 Iterative refinement
the pass metric when employing trained CodeI.lama-14. for iterative code refinement. The findings align with those depicted in Figure that both SFT and RI consistently outperform the prompting approach in each round of refinement. Notably, even after three rounds of refinement, the prompting method fails to achieve higher @ than what SFT and RL models attain after just one round. This highlights the substantial advantage of SFT and RL methods in enhancing code quality. Moreover, the SFT and RL CodeLlama- 13B models demonstrate robust continuous refinement capabilities, maintaining their superior performance across multiple iterations. This consistent outperformance underscores the effectiveness of SFT and RL strategies in refining and improving code, suggesting their potential for more efficient and reliable coding practices in iterative development scenarios.
(4) pass@21 on M BPP and HumanEival Proritt
(b) 10 on MBPP and Humantral Procppt
(4) paseg11 on MBPP and HumanEVatt (d) pasage on MBPP and HumanEvair Phompt
RL
Prompe
02
15.19
$3.97
10.99
83.26
13.4
66.72
50.55
70
77,85
79.22
40.7%
50.75
01.33
75A
54.27
71.05
7635
40.37
$2.18
5037
7336
71.77
4275
41.45
25.75
43.00
Wr.
te:Refre 2nd Rellss 3rd ReBiee
S61Rifin 2nd Rofino 3rd Refio
Int.Sel
16:Refnn and Redie and Redite
Int. Sot.
Txtieelne 2nd flatre 3nd falte
Figure 8: Pass@k of prompting, SFT, and RL Iada- after three iterations of refinements.
A.4.3 Generalizability
The @ of CodeLlama-133 trained with data collected from CodeL are shown in Table [2] where RL achieves the highest pass@1 @ and pass @ 10 in all the benchmarks. The overall results are shown in Table [3] and SFT improves the pass@ 1 by up to 9.14% and pass@ 10 by 3.47% on the overall of MBPP and HumanEval, and improves the pass @ by up to 8.68% and pass 10 by 3.92% on MBPP+ and HvmanEval++ Further RL training improves the pass@ by up to 0.61 % and pass @ 10 by 0.86%. The improvement brought by RI. is significantly larger on MBPP+ and HumanEval+ (up to 6.80% higher pass@) I and 2.28% higher pass 10), which is consistent with our finding when using GPT-3.5-Turbe 's training data that RL brings more considerable improvement harder benchmarks.
Table 14 presents the success refinement rates achieved by CodeLlama-133 13B when trained on data collected from CodeLlama-334: averaged across four benchmarks. SFT refines 15.60% to 16.83% of incorrect solutions, outperforming the prompting approach by 3. 3.96% to 4.86%. RL training further boosts the refinement rate, improving it by 4.78% to 7.14% over SFT. Notably, the improvement from SFT is somewhat smaller when using Codel_lama-341 likely due to the slightly lower quality of explanations and refinements generated by CodeL.lama-345 However, RL training raises the refinement rate to levels comparable to those achieved with GPT-3.5-
data compared to GPT-3.5-
's data,
17

---

Table 12: Pass @ on MBPP and HumanEval by LLMs trained with CodeLlama-344 34B's data.
Models
Approaches
MBPP
Humanval
MBPP
HumanEval**
pass@P1
pass/
puss@1
pass
GP
pass
pass 01
pass 6
42.88
37.11
62.26
66.27
Prompt.
Refine
49.68
75.85
45.78
81.07
46,22
70.14
37.62
72.68
Expl. Redine
49.97
76.39
45.90
81.18
45.77
70.48
38.36
73.84
Init.
50.61
73.14 14
44,96
76.85
44.66
63.15
37.98
69.59
CodeLlama--33B
LEDEX SET
Refine
54.15
52.87
73.50
46,96
Expl. Refine
59.11
79.56
55.04
84.75
53.97
73.66
47.73
78.17
Inii.
50.50
73.26
45.62
78.14
44.72
63.40
38.79
71.98
LEDEX RL
Refine
59.15
80.21
55.71
85.33
57.74
74.03
56.56
82.23
Expl. Refine
59.57
56.08
85.40
56.60
73.17
56.24
82.39
Table 13: Overall pass@k k MBPP & HumanEval and MBPP & HumanEval*, trained with CodeLlama-34BB data. Blue numbers show the improvement.
CodeLlama-133 13B
MBPP HumanEval
MBPP* HumamEval"
LEDEX SFT
LEDEX RL
LEDEX SFT
LEDEX RL
Refine
Expl. Refine
Refine
Expl. Refine
Refine
Expl. Refine
Refine
Expl. Refine
pass@]
$7.85 -9.13 58. 10 -9.14
58.30+0. +0.45 58.71 +0.1
50.46 +7.24 51 43 +8.68
57.26 40.80 56.45 -500
pass@110
80.61
80.84
81.47 81.47
75.09
75.5
77.37
76.92
Turbo's data, indicating that RL training can mitigate the quality differences between data generated by open-source LLMs and commercial LLMs.
A.5
Case studies
A.5.1
Correct refinements generated by SFT LLMs
Figure 9 shows an example from the HumanEval benchmark for which the prompting approach fails to generate correct refinement. The bug in the initial solution is that 'nlargeat' returns the largest elements in descending order, but from the example provided, one can find out that the expected output in ascending order. The prompting approach does not work for this example, and the prompted StarCoder generates a hallucinated explanation and simply repeats the wrong solution. Actually, such simple repeats of wrong solutions very common when using the prompting approach, which supports our motivation to train LLM to self-debug. However, after SFT training, the trained StarCoder can correctly figure out the reason for test failure and generate the correct refinement.
A.5.2 Correct refinements only generated by RL LLMs
Figure shows an example from the MBPP benchmark for which only the StarCoder trained with reinforcement learning explains the wrong code correctly, pointing out that the failing test case is be- cause "this approach does not consider the case where multiple elements have the same maximum count" (highlighted with green background), and generates the correct refinement. The SFT StarCoder is unable to diagnose the wrong code correctly.
A.5.3 Robust refinements generated by RL LLMs
Figure shows example from HumanEval, for which both SFT and RL CodeLlama- 13B generate the correct refinements that pass all the test cases from the HumanEval benchmark. Yet, the refinement from CodeLlama- 13B is not fully correct, "(x, y. 2) == ((int)x, (int)y, (int)z)" is not cquivalent to checking if these numbers are integers. The refinement generated by SFT CodeLLama- 13B (left) fails the harder test cases in HumanEval+ i... AssertionError: expect any_int 03
Table 14: Success refinement rate of different approaches over four benchmarks, trained with 34d'L'sma-34B's data. Blue numbers show the improvement.
Models
Refine (9)
Explain Retine (%)
Prompt.
LEDEX
Prompt.
LEDEX
Cama- 13B
11.64
15.60 43.96
22.74 -7.44
11.97
16.83 +4.86
21.61 +4.78
18

---

R [juser] [juser|pGiven polvenn an array are of Integers and positive integer return sorted 11st of length with the maximus k nurbers
<[user] bdiven an array are of integers and a positive integer
k, return sorted 11st of langth k with the naxtaus numbers
Exarpio
Irppu: are [-3, .,, 5], x
Input: are [-3, -4, 5],
Output: [-4, -3,
Output: [-4, -3, 5]
hote:
longth of annay will range [4, 16ee].
1. longth the areay
the range [1, 1888). 2. The elemnts in the arnay will be in the range of [-100e,
2. The elesents in the arnay will be in the nange of [-1000,
1ece].
10e0].
len(arr)
3.
len(arn)
<|end]>
</end]>
<lassistant]
<lassistant]
naxinusçarr, arr,, k):
naximan (ar,, k):
from heapy import nlargest
fron heapy import nlargest
return nlargest(K,, anr)
return nlargest anc)
<jend]>
</end)>
rlssen|s Running
code returns Following error

 Running the above returns following erron
message:
ressage:
AssortionErro:: expect naxinun((-3, -4, 5], 5) [-4, -3, S],
AssertionEnra:: expect naximun([-3, -4, 5], [-4, -3, 5]. -4]
code does pass assertion. Please:
1. Explain why the above the assertion. Please: wrong.
Explain why above code is arong.
2. Propose the correct code. </end]>
2. Propose the correct code. </and]>
<|assistant)>
<l [assistantt/>
The provided code for the maximun Function uses the heapq
The wrong code is using the "nlargest" function from the 'heap"" module to find the largest elenents in the array. However, the nlargest" function returns the largest elesents the elements why the output reversed.
sodule to Find the largest elements the array. Howwverr the heapg undule only supports finding the k largest
clesents in array.
descanding order, shill ascending orden.
It doesn't support Hinding array.
smallest elements
To fix this issue, need to use the nssallest Function froe the heapq nodule.
def naximun are, k):
from heapy import nlargest return nlargest an-] <[and}>
vaximun(ane, fron heapq import nlargest return nlargest(k, arr)
<|end]>
Incorrect Explanation and Refinement Generated Prompting StarCoder
Correct Explonation and Refinement Generated SFT StarCoder
Figure 9: Example for which prompting StarCoder fails but SFT StarCoder generates the correct explanation and refinement.
4, 7) False, but was True, since fails to realize that 3.0 is not integer. By contrast, the refinement generated by the RL CodeLlama-133 generates better refinement that passes all the test cases from HvmanEval+.
A.6 Human rating of code explanation
We let developers rate the explanations generated by different L.LMs based on the following rubrics, which consider both the correctness of the statements the explanations and also helpfulness of the explanation.
Table 15: Rubrics used for developer rating of LLMs' generated explanations.
Rates
Correctness
Totally wrong misleuding
Mostly wrong but with least one mi- correct point
Partially correct explanation with about 50% wrong
Partially correct planation with mi- nor mistakes
Totally correct planation
Helpfulness
Helpless statement that repeats the ror or is not related bug
Not very helpful but least related
Provides very vague hints about how to understand the failed test case
Provides vague Provides least clear bints about how to fix
hints about how fix the wrong code
to the bug
Figure 12 shows an example of developer rating, where the prompting and SFT StarCoder are rated "1" since their explanation wrong and not informative. yet, the RL StarCoder's explanation successfully points out that the function should return 'Not Matched! when there is no match.
Figure 13 shows another example, where the prompting StarCoder is rated "2", as the developer thinks the explanation is stating correct fact. By contrast, SFT and RI. StarCoder correctly point out that the function should check if the array is empty or not.
19

---

kluser|> l hrite function find the iten with raximat occurrences in given

 Write python function find the iten with maximur accurnences given 11st.
Exanple: assert nax, (ccmmmecccce11...33322222222.. -. <lend]s
Exemple: assert rox, occ.rrencees..2..22222.....
<|end)
<Jassistant]>
<lassistant|>
def www_occurrences(aantt
sef nax nar_occurrences (ecurr)ences(uuss)::
result (ax(s)
nuns.count)
result max(set(nuns), key nurs.count)
result


</end]>
<juser|s Running the ahove code returns the Following arron
< |jusen|> Running the above code returns the following error
nessage:
AssertionFrror: expect nax mww.praureeeee(1227701001.
Assertionfrrar: expect nww_ocuueenees(t-2577....1
"
(1,8), but was
(1.0), out was B
So the code does not pass the assertion. Please:
So the code does not the assertion. Please:
Explain the above code is wrong-
a. Explain why the above code
Propase
2.
code.
</end]>
</end|>
<|assistant]>
code occurrences function find the <lassistant|
the "nax()" function with paraneter set count'
distionary However,
elament set 'itans has naxinux find
function returns key with naximue value, not the key-value pair itself, So, the code is returning just the key Instead of the key-value patr. (nuubr))
the original 1ist. However, this approach does not der the case where rultiple eliments have the sane naxiaus count. "nax()" functian
() if d: d[r]
raximun count, which ray not correct
nunbers:
answen.
sef nax_occurrences (nunbers):
d {)
d[n] return rox(d-items(), key=larbda x: x[3]]
d[n]
</ndd>>
else:
d[n] nax_count vox(d.values())
nax nues [nun for mus, count count_diet. (lut__ictt.tees()) if count nax_count]
IF len(max, nues) return [8]
return tople (maxx_muss))
<
</end|>
Incorrect Explanation and Refinement Generated by SFT StarCoder
Correct Explanation and Refinement Generated by
StarCoder
Figure 10: Example for which the RL LLM generates the correct explanation and refinement.
function numbers equal sun the other two, and all nunbers are integors. Roturns false in any other cases.
[Jseer] Streate [trestt
<Jusen []user|>Create pCreate
Returns true if
Returns one the numbers equal
other two, and all numbers are integers. Returns false in any othen cases.
Examples
Examples
any any_int(5, 7)
any_int(5,
any Antt3, 2, 2)
any int(3,
any_int(3, -2, 1) - True
any_int(3,, -2, 1) - True
any iny_in, -2.2, 2) False
any tn,ttt3,66 -2.2, False </end|>
<|end]>
</acsistant|>
< <lassistant(>>
dof any int(x, y, 2):
cof any int(k, y, 2):
18
1F y
return True
return True
return False
<lend]>
<lend|>
</usar|s Running the above code returns the following erron

 Running the above cade returns the following error
nessage:
AssertionError: expect any inc(1.5, 5, 3.5) False, but was
AssertionErrar: expect any int(1.5, 3.5) False,
True
True
5o the code does not pass the assertion. Please:
So the code does not the assertion. Please:
Explain the above code is wrong.
a. Explain why the above code
Propose
2.
code.


<[end|>
[lassistant]>
< [
code checking if numbers intagers. It only the nurber. Houaver, task requires chacking all three numbers are intagers.
not check all the numbers equal integers. two. why the assertion iny_1,, 5, 3.5) returns True, which is incorrect.
numbers
only
y, (x, y, 2) (int(x), int(y), int(2)):
any y. y 2 all((stnstancee(uu,, int) for run [6, v, #]): return True
V.
+ or return True
oe
return False
return False
[end]>
Non-robust Refinement Generated by Prompting SFT
Robust Refinement Generated RL CodeLLama-133
IamaLI3a
Figure 11: Example for which RL LLM generates more robust refinement.
20

---

<[user] purite function search sone literals strings in string using regex.
"AssertionErron: expect check. litarals( The quick brown fox junps the dog. dog.".("bo) '''orreee' }) 'Not Matched!", , but was "Not natched!"" indicates that the code is trying to check If the string 'horse" Is In the but the string 'horse" is
Example:
assert check "heck_literals( The quick brown fox jumps over the lazy dog.'s ('fox']) }} 'Hatched!"
<|end]>
<Jassistanti>> import re
Explanation by Prompting Starr (Rating:
Beeck_ttterals(ted literals:: for literal in litorals:
wrong code handling natch for any of the literals. It is returning "Not natched!! inrediataly when there is natch, instead of checking 1iterals.
if seearch([iterallteeet)
return "Matched!"
return "Not satched!"
</end]>
<luser]> Running the above code returns the following
Explanation by SFT StarCoder (Rating: 1)
nessage:
AcsertionErrar: expect check, check_1iteee literals( 'Te quick brown fox "Not Natchedl', ,, but
Jurps the lazy dog. [''horse"]>
satched!'
The code is not handling the when there for any literals. returning natched!" In such cases, but the assertion expects "Not Matched!', This is causing assertion
5a the cade does not pass the assertion, Please: Explain why the above code is wrong.
Propose the </end]>
Task Description, Wrong Solution and Execution Feedback
Explanation by RL StarCoder (Rating:
Figure 12: Example of human rating, where RL. StarCoder generates the best explanation.
Juser] DSiven an array non-negative Integers, return copy of the given annay after sorting, given array in ascanding order if the value)
The error message "IndexErron: 1ist index of range" indicates that the code is trying to access an elerent in the array that does not exist.
will
index value,
or sort it in descending order if the sun( first index
value, last Index value) Exarples:
Explanation Prompting StarCoder (Ruting:
sort_erray(()) >>
ser)_array([5])
sort ([22_, 4, 3, 8, 1, 5])
2,
x </ndd]>
The error message "IndexErron: list index out of range" is occurring because the code is trying first and last elements of the array using array[0] and array[-1], [1], but if the array is empty, it will result in index of range
[aassstant]]>
the
aart_aarayarrray):
1f sun(array(3), arnay[-1]) x
return sorted(anray, reverse-True)
else:
return sorted(array)
Explanation SFT StarCoder (Rating: 4)
</end]>
[Jserr]> Running
code
following error
nessage:
message "IndexErron: list Index range" is trying access First
IndexError: 1ist index out of
occurs because last elements of the array using aenay[o] and "array[-1l
So
Please:
1. Explain why the above cade is wrong.
the array espty, an index out of range arron.
2. Propese the correct code. lend|>
Task Description, Wrong Solution Exccution Feedback
Explanation by RL StarCoder (Rating: 4)
Figure 13: Example of human rating, where both SFT and RL StarCoder generate good explanations.
A.7 Potential impact
Using LLMs to help coding is popular nowadays. This work proposes a technique to train LLMs to explain and self-refine code, which aims to improve developers' coding experience with LLMs. We also call for training LLMs take feedback beyond the prompting approach improve the LLMs' self-debugging ability. The technique is supposed to be on the same track as all the existing LLMS for code generation. Thus, we think no special concerns about broader impact need to be highlighted here.
21

---

NeurIPS Paper Checklist
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We describe our approach and main novelties, and summarize the main results at the end of abstract and introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have limitation section (please refer to Section in the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to vi- olations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by review- ers as grounds for rejection, a worse outcome might that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory Assumptions and Proofs
For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [NA]
22

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
We describe the details of data collection (Section 2.1 Appendix A.2 as well as experimental setup (Section 3) the paper. These are enough reproduce the results described in the paper.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also provided via detailed instructions for how to replicate the results, access hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
example, if the contribution
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way reproduce the model (e.g., with an open-soure dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
23

---

Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (btts:://nips.cc/public
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot rejected simply for including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cwweeeemmssssooneoo.o
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The datasets are described in Section 2.1 and Appendix A.2 The training details are described Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The improvement is considered significant across different models and datasets with multiple runs. However, statistical significance test does apply the code genera- tion/refinement task this paper studies.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar the standard deviation or the standard error of the mean.
24

---

It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experi- ments?
Answer: [Yes]
Justification:
The infrastructure used to run the experiments described in Section 3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Code generation and code refinement are widely studied and explored, which aims to improve developers' coding experience with LLMs. There is no special concerns of ethics of the technique.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
a The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We describe the potential societal impacts in Section A.7
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The LLMs and datasets used in this paper are open-sourced and we carefully cite the original papers that produce them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
license, copyright information, and terms use the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
assets are released,
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
26

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their
institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "sZnJKljfZu", "reviewer": "Reviewer_jfHT", "summary": "Authors propose a framework to perform SFT and RL training to achieve superior performance in generating code with open code LMs on the self-debugging task. They leverage the test suite present in benchmarks like APPS, CodeContests to obtain execution feedback for model refinements on CodeLM generated code. The resulting dataset is then used for SFT training, followed by RL training that can also leverage failed trajectories besides the successful ones (where the model succeeded in generating a code that passes all unit tests). Authors show promising gains for open weight models in the model's capability to self-debug after training with their proposed methodology.", "questions": "- APPS and CodeContests are used to train the CodeLMs, but I could not find evaluation on the APPS-test or CodeContests test set. Could you explain this choice?\n- Can you provide details on the number of GPU hours required in the experiments?\n- Have the authors considered training a separate model using the SFT and RL techniques to solely solve code refinement for code generated from the base model?\n- What do the authors think about the generality of this method to improve on attributes beyond correctness in refining code? e.g. readability, performance and secureness of generated code.\n- I'm curious how the proposed approach would compare against a simple baseline where the model is SFT-trained on the final refinement collected in your training set. The current setup involves using problem description $x$ and ground truth code $y$, test suite $T$ to generate a synthetic code solution $y'_l$ that fails and is explained by $x_l$ followed by a successful refinement $y'_w$. Your approach then trains the model to generate $x_l, y'_w$ given $x, y, y'_l$. A simple baseline to compare against could involve training the model to generate $y'_w$ given $x$. This would confirm the value in framing code generation as a explanation + refinement task.", "limitations": "Briefly discussed by the authors in Section 5. Could be expanded to include a discussion on other aspects of code refinement not covered in the paper, and acknowledging gaps if any in performance of open models trained with this method and closed models.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Authors present a strong motivation for this work (Lines 43-52) on achieving strong code generation performance with open weight models.\n- This work makes a strong contribution in the form of a framework to construct data for SFT and RL training of a model that can perform self-debugging after explaining faulty code. Authors propose a clever way of leveraging execution feedback in constructing their datasets\n- The reward construction based on environment feedback is a particularly important contribution that significantly adds to the novelty of this work. To my knowledge prior work hasn't utilised environment feedback in this manner.\n- Convincing results that confirm the utility of training open codeLMs on the task of self-debugging.", "weaknesses": "- While the experiments and analysis of the results and datasets are fairly exhaustive, I believe the choice of RL algorithms should be justified by considering or eliminating alternatives like preference optimisation using DPO or KTO. I'd suggest at least adding a discussion on the pros and cons of preference learning compared to the RL setup that the authors advocate in this work.\n- Some missing baselines: teacher model (GPT-4/3.5/CodeLlama35B) performance is missing in Tables 2 and 5. Authors do not discuss the persisting gap in performance in any between the models used in creating the datasets and the performance their approach attains on the benchmarks used.\n- A very relevant related work (Teaching LLMs to Self-Debug https://arxiv.org/abs/2304.05128) mentions gains in sample efficiency as one of the major benefits of performing self-debugging/refinement. I could not find a discussion or results on this aspect for the fine-tuned models presented in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for insightful suggestions and questions.\n\n## 1. APPS and CodeContests Evaluation\nThis is a good suggestion and we plan to add the results to the final version if accepted.\n\nBelow are the StarCoder-15B’s results on APPS and CodeContests. Results of the other two backbones can be found in our attached author response PDF file (Tables 4 and 5).\n\n| Approaches | | APPS (5000) | | CodeContests (165) | |\n|-|-|-|-|-|-|\n| | | Pass@1 | Pass@10 | Pass@1 | Pass@10 |\n| | Init. | 2.57 | 8.59 | 0.58 | **3.88** |\n| Prompt | Refine | 2.84 | 9.10 | 0.65 | 4.23 |\n| | Expl. + Ref. | 2.95 | 9.53 | 0.78 | 4.84 |\n|||||||\n| | Init. | 3.80 | 11.52 | **0.62** | 3.67 |\n| SFT | Refine | 6.89 | 17.01 | 1.16 | 5.25 |\n| | Expl. + Ref. | 6.86 | 17.18 | 1.48 | 5.94 |\n|||||||\n| | Init. | **4.50** | **13.62**  | 0.44 | 2.19 |\n| RL | Refine | **7.81** | **19.28** | **1.80**  | **6.04** |\n| | Expl. + Ref. | **8.10** | **19.75** | **1.80** | **6.37** |\n\nOn APPS and CodeContests, the prompting baseline also only refines very few solutions, and the improvements brought by refinement are marginal. However, with our SFT and RL-trained models, we see significantly stronger self-refinement ability. The final Pass@k is also about doubled from the prompting baseline.\n## 2. Preference learning (DPO/KTO) vs PPO\nPreference learning like DPO or KTO has the advantage of its simplicity without the need for a reward function or reward model. In the setup of self-debugging with execution feedback in this paper, we could construct preference data in such a way: the fix that passes unit tests is preferred over the one that fails unit tests. Such preference data could also work, but might seem less direct as execution feedback/reward is easily obtained from the execution engine, unlike the human preference setup. One advantage of our RL training is that we can assign different rewards to different parts of the sequence, i.e. the explanation reward on the explanation part and the execution reward on the generated fix part. \n\nIn this work, our focus is exploring how to train LLMs for self-debugging and bug explanation with the lack of training data. Our proposed data collection pipeline and SFT, RL (PPO-based) training do outperform existing prompting approaches significantly. Exploring alternative RL algorithms could be interesting future work in this domain.\n\n## 3. Teacher models performance\nWe provide the comparison with the teacher models in our attached author response PDF file. \n\nComparing it with Table 2 in the paper, with GPT-3.5 as the teacher, CodeLlama-13B SFT/RL cannot outperform GPT-3.5, which could be because GPT-3.5 is a much stronger teacher model.\n\nComparing it with Table 10 in the paper, with CodeLlama-34B as the teacher, CodeLlama-13B SFT/RL **sometimes outperforms the CodeLlama-34B teacher** (e.g. in HumanEval+ pass@1 56.24% vs 48.51%, and in MBPP+ pass@1 56.60% vs 53.19%).\n\n## 4. Comparison with “Teaching LLMs to Self-Debug”\n**The method in “Teaching LLMs to Self-Debug” is the prompting baseline that this paper refers to in Tables 2, 4, and 7.** We directly compare the results of our proposed method with this baseline in our experiments, and the results show that such a prompting approach cannot work as well on open-sourced LLMs.\n\nThe most notable difference is that “Teaching LLMs to Self-Debug” investigates the self-debugging of commercial LLMs by prompting, while our paper investigates how to improve open-source LLMs’ self-debugging capability via training. \n\nAnother difference is that our paper tries to **generate the explanation of the bug to help humans and LLMs better understand the reasoning**, while the paper “Teaching LLMs to Self-Debug” tries to generate an explanation for the code instead of reasoning the bug.\n\n## 5. GPU hours\nThe GPU hours for training:\n\n| Models | SFT | RL |\n|-|-|-|\n| StarCoder-15B | 320h | 192h |\n| CodeLlama-7B | 80h | 96h |\n| CodeLlama-13B | 280h | 178h |\n\nExperiments are conducted on 8 NVIDIA A100 GPUs, each with 40GB of memory\n\n## 6. Generate 𝑦𝑤′ given 𝑥\nIf we understand correctly, the suggestion is generating the correct refinement (𝑦𝑤′) given only the problem description (𝑥). We think this is essentially fine-tuning with code generation data.\n\nWe compared our approach (framing code generation as an explanation + refinement) with fine-tuning with code generation data only, and the results are shown in Appendix Table 9.\n\nWe find fine-tuning for code generation improves the initial solution, achieving comparable or sometimes higher Pass@1 than our approach. **But the model’s self-debugging ability is not improved**, and the model still cannot benefit from self-debugging. The model trained for code generation cannot successfully self-debug very often and eventually is surpassed by our approach dramatically. We hope this confirms the benefit of “framing code generation as an explanation + refinement task” over simply training a better code generation model.\n\n## 7. Separate code generation and refinement models\nTraining a separate dedicated debugging code refinement model is definitely an option. The main motivation of the paper is to improve LLM’s self-debugging capability as one part of the capabilities of LLMs, and not be over-specialized so that the training recipe can directly be incorporated in practice.\n\n## 8. Generalize to other aspects\nWe think that the method can be generalized to improve other aspects of code such as readability and secureness. The key lies in how to properly design the reward with respect to each aspect. The readability and secureness are most likely to be determined by a set of static analysis rules. Combining all aspects together to obtain a final reward is necessary and used to provide feedback to the training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "36ojv85FGq", "reviewer": "Reviewer_Dkve", "summary": "This paper proposed a pipeline to obtain code explanation and refinement data from a stronger model (mainly GPT-3.5 with CodeLLaMA as ablation) to train weaker models (i.e., StarCoder-15B, CodeLLaMA-7B/13B) using SFT and RL methods. More specifically, it uses the weaker models to sample the incorrect solutions and use the stronger models to give code refinement and explanations. It also designed reinforcement learning rewards specifically for the code refinement and explanation tasks. Experiments are conducted on MBPP and HumanEval, as well as their EvalPlus versions, and results show that the both SFT and RL yields improvements over the baselines, while the improvements from RL are relatively marginal compared to those of SFT.", "questions": "Q1. According to Olausson et al., 2023 on self-repair, after $k$ rounds of self-refinement after the initial code generation, the success rate is often less than simply pass@$k+1$. And also note that the sampling can be done in parallel while the iterative refinement can only be done sequentially. Have you compared the refinement results with pass@$k+1$ to see if self-refinement yields any benefit before / after the SFT/RL training?  \nQ2. From Fig. 3(a), it seems to me that the CodeBLEU score is not a good metric as it can barely separate the correct and wrong outputs distribution-wise, is there any reason for CodeBLEU to still be factorized into the reward function despite this?  \nQ3. Can you comment on the reliability of using the RoBERTa embedding for measuring the similarity of the explanations? Are there better ways to do this?   \nQ4. From Tab. 2, it seems that after SFT, the \"Init.\" performance also significantly improved, does this means that **only** training on code refinement and explanation can also improve code generation?   \nQ5. (Pertain to W3) Have you tried to use the same model as the LLM to create the training data?  \nQ6. The training data are created on top of APPS and CodeContest as well, but why are those two datasets not used in evaluation?  \n\n**References**  \nOlausson, Theo X., et al. \"Is Self-Repair a Silver Bullet for Code Generation?.\" The Twelfth International Conference on Learning Representations. 2023.", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. The tasks of code refinement and explanation are increasingly important due to the popularity of using language model agents for coding tasks. And this work shows a way to reliably improve the performance of LLMs in these two tasks, which may be useful for broader domains such as code editing and reasoning;  \nS2. The RL reward design and ablations could be useful for further research on using RL for code explanations and refinements;  \nS3. The paper is well-written, with clear motivations, details of methodology and comprehensive experiments.", "weaknesses": "W1. Something slightly disappointing is that this work chooses to generate training data **from a stronger model**, which classifies it into the category of distillation, while the entire pipeline could have been done with the same LLM to explore the potential of self-improvement;  \nW2. While the paper focus a lot on the design of the RL method (which seems to be a big part of the contribution from my understanding), the actual improvements yield by the RL methods are quite marginal. However, I do note that at least the performance does not decrease for the most cases;  \nW3. There are some baselines experiments and ablations that can be added to make the evaluation part stronger (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for insightful suggestions and questions.\n\n## 1. Data collected from the same model itself\nWe provide the experiment results using the synthetic data generation from the model itself for CodeLlama 7B. \n\nWe highlight some results here. Below is the CodeLlama-7B SFT/RL using the data collected from itself, evaluated on MBPP+ and HumanEval+.\n\n| Approaches | | MBPP+ | | HumanEval+ | |\n|-|-|-|-|-|-|\n| | | Pass@1 | Pass@10 | Pass@1 | Pass@10 |\n| | Init. | 37.18 | 61.23 | 27.40 | 60.81 |\n| Prompt | Refine | 42.97 | 66.89 | 31.84 | 65.08 |\n| | Expl. + Ref. | 42.46 | 67.41 | 32.49 | 66.58 |\n|||||||\n| | Init. | 41.78 | 61.77 | 33.25 | 61.50 |\n| SFT | Refine | 46.26| **66.39** | 40.15 | 67.15 |\n| | Expl. + Ref. | 45.94 | 65.77 | 39.10 | 67.33 |\n|||||||\n| | Init. | 41.61 | 61.29 | 33.66 | 62.17 |\n| RL | Refine | **46.28** | 65.86 | **41.54** | 68.14 |\n| | Expl. + Ref. | 46.10 | 65.99 | 40.79 | **68.50** |\n\nThe full results are in our attached author response PDF file, Tables 1 and 2. Results show that self-taught SFT and RL also achieve big improvements. CodeLlama-7B SFT/RL models achieve up to 5% improvement in self-debugging, compared with the baseline prompting method, and the model trained with code generation data only. Though comparing the experiments using data from CodeLlama 34B and GPT-3.5-Turbo, the improvement is less.\n\n## 2. Pass@K+1 versus refinement Pass@K\nThis is a very interesting point. We evaluate the Pass@2 of the initial solution and Pass@1 after one round of refinement.\n\nBelow is the result when the models are not trained (the prompting baseline). We do observe that Pass@2 is better than Pass@1 after refinement. **This shows that using a prompting approach to self-debug is not effective**.\n\n| Before Training | | MBPP+ | HumanEval+ |\n|-|-|-|-|\n| StarCoder-15B | Pass@2 | **45.20** | **36.38** |\n| | Expl. + Ref. Pass@1 | 39.27 | 30.09 |\n| CodeLlama-7B | Pass@2 | **46.91** | **38.17** |\n| | Expl. + Ref. Pass@1 | 42.46 | 32.49 |\n| CodeLlama-13B | Pass@2 | **48.08** | **41.52** |\n| | Expl. + Ref. Pass@1 | 45.77 | 38.36 |\n\nHowever, after the models are trained using our pipeline. The refinement Pass@1 is clearly higher than the Pass@2 of initial solutions. This shows that the model’s self-debugging performance is bad without training. The prompting approaches proposed by existing works such as (https://arxiv.org/pdf/2306.09896 and https://arxiv.org/abs/2304.05128) are not as effective on open-sourced LLMs. **This experiment further supports our motivation to train LLMs to self-debug and proves the effectiveness of our approach.**\n\n| After SFT | | MBPP+ | HumanEval+ |\n|-|-|-|-|\n| StarCoder-15B | Pass@2 | 51.19 | 39.29 |\n| | Expl. + Ref. Pass@1 | **53.83** | **43.54** |\n| CodeLlama-7B | Pass@2 | 50.93 | 40.95 |\n| | Expl. + Ref. Pass@1 | **51.55** | **47.62** |\n| CodeLlama-13B | Pass@2 | 50.93 | 44.78 |\n| | Expl. + Ref. Pass@1 | **54.59** | **51.32** |\n\n## 3. CodeBLEU in RL reward\nWe find that only using binary execution feedback as a reward does not train the model properly as the reward is too sparse. That is, a completely wrong solution will get the same reward as an almost correct solution, which hurts the RL training. Although the CodeBLEU reward is weak, we find that having it helps stabilize the training by densifying the reward distribution. This is the main reason we introduce the CodeBLEU score in the reward.\n\n## 4. Roberta embedding for text similarity\nJudging the correctness of code explanation is non-trivial. We use the Roberta model (https://huggingface.co/sentence-transformers/all-roberta-large-v1) that has been massively fine-tuned for text similarity using 1B sentence pairs.\n\nWe try to analyze the reliability of this approach as shown in Figure 3(c) in the paper. The similarity can separate the explanations that lead to correct and wrong solutions most of the time.\n\nA potential alternative approach could be using powerful LLMs such as GPT-4 to rate the code explanation, however, this is not scalable enough the handle our RL training data.\n\n## 5. Code generation improvement\nIt should be noted that the SFT training includes code generation data (provided in the original MBPP/APPS/CodeContests) + self-debug data. It could be the code generation data improves the initial solution generation. We will make this clearer in our experiment setup.\n\nWe compare our approach with fine-tuning using purely code-generation data. The results are in Appendix A4.1 Table 9 in the paper. \n\n## 6. Evaluation of APPS and CodeContests\nThis is a good suggestion and we plan to add the results to the final version if accepted.\n\nBelow are the StarCoder-15B’s results on APPS and CodeContests. Results of the other two backbones can be found in our attached author response PDF file (Tables 4 and 5). Due to the large amount of test samples in APPS (5000), we only test the other two backbones on a subset of 200 samples. We plan to complete the evaluation and add it to the paper.\n\n| Approaches | | APPS (5000) | | CodeContests (165) | |\n|-|-|-|-|-|-|\n| | | Pass@1 | Pass@10 | Pass@1 | Pass@10 |\n| | Init. | 2.57 | 8.59 | 0.58 | **3.88** |\n| Prompt | Refine | 2.84 | 9.10 | 0.65 | 4.23 |\n| | Expl. + Ref. | 2.95 | 9.53 | 0.78 | 4.84 |\n|||||||\n| | Init. | 3.80 | 11.52 | **0.62** | 3.67 |\n| SFT | Refine | 6.89 | 17.01 | 1.16 | 5.25 |\n| | Expl. + Ref. | 6.86 | 17.18 | 1.48 | 5.94 |\n|||||||\n| | Init. | **4.50** | **13.62**  | 0.44 | 2.19 |\n| RL | Refine | **7.81** | **19.28** | **1.80**  | **6.04** |\n| | Expl. + Ref. | **8.10** | **19.75** | **1.80** | **6.37** |\n\nOn APPS and CodeContests, the prompting baseline also only refines very few solutions, and the improvements brought by refinement are marginal. However, with our SFT and RL-trained models, we see significantly stronger self-refinement ability. The final Pass@k is also about doubled from the prompting baseline.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ktf8Qsnhfq", "reviewer": "Reviewer_2CSS", "summary": "The authors teach language models to better self-debug and explain code. Particularly, they utilize code explanations in the repair process where explanations are generated before refining the programs. This is accomplished via training the models with SFT and RL on data curated from different sources and model generations with test case based rejection sampling.", "questions": "* Table 1 further details. I anticipate that many problems in the training sets remain unsolved post-refinement, particularly for APPS and CodeContests which are more challenging benchmarks. Can the authors list details about problems beyond number of solutions.", "limitations": "Limitations are discussed adequately (besides some of the above-mentioned weaknesses)", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is nicely written with enough details about experiments. Improving code repair capabilities of LLMs is an important problem and the authors propose a rejection-sampling-based technique to automatically curate repair trajectories from LLMs. The associated ablations are useful and convey useful insights.", "weaknesses": "Weak results\n* Benefit of explain + refine over just refine. \nThe proposed explain + refine approach, associated loss on explanations does not seem to be improving performance. Infact, Expl.+Refine rows sometimes do worse than the only Refine rows. These results highlight a lack of benefit from adopting this approach. While I understand that having associated explanations is appealing and perhaps might even provide models with more inference-time-compute before performing refinements, I think the current approach or experiments does not convey that. The associated human evaluations (Table 7) also point to similar findings. \n\n* Optimizing single-turn vs multi-turn performance.\nFinally, a lot of the instruction-tuned variants of the models used in this work claim better single-turn HumanEval and MBPP performance. I wonder how the findings would change if they started from a strong instruction-tuned model and improved its repair performance. For instance, the OpenCodeInterpreter-CL-7B model shows performance improvements of 3 points from execution feedback on HumanEval (from 72 to 75) due to strong instruction tuning data pushing pass@1 to 72. Perhaps to a larger point, the effect of repair depends on the choice of the underlying model and broader data mixture which this paper does not study.\n\n* Choice of evaluation datasets. \nSince the authors use competition programs (like APPS or CodeContests) in their study, perhaps it is fair to also evaluate the models on competition programming datasets.\n\n* Missing related works.\n[1] is also pertinent to LLM reasoning. [2] and [3] also train and release open LLMs on repair trajectories. [3] in fact uses a similar explanation format. \n\n[1] Reflexion: Language Agents with Verbal Reinforcement Learning\n\n[2] OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement\n\n[3]  Advancing LLM Reasoning Generalists with Preference Trees", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for insightful suggestions and questions.\n\n## 1. Explanation improvement\nThe reviewer might have some misunderstanding of the explanation evaluation in Table 7. In Table 7, we performed both human evaluation and LLM Judge-based evaluation to evaluate the explanation quality before and after training. It clearly shows that the SFT/RL training significantly improves the explanation quality, by generating more correct and helpful bug explanations. Appendix A6 Table 13 lists the rubrics. \n\nWe also provide some case studies on human evaluation in Appendix A6 of the paper to show qualitatively how the finetuned model generates better explanations.\n\n## 2.Comparison with Instruction-tuning\nIn Appendix A.4.1 of the paper, we have presented the comparison of SFT on strong code instruction data with SFT on the full data we collected. The code instruction data we used includes the Magicoder dataset [ref], similar to the OpenCodeInterpreter-CL-7B the reviewer mentioned. The results in Appendix A.4.1 show that even when training with strong code instruction data, the model does not obtain self-debug capability out-of-the-box and it only refines up to 3% of its wrong solutions. And this is one of the main motivations of the proposed approach. \n\n## 3. Evaluation of APPS and CodeContests\nThis is a good suggestion and we plan to add the results to the final version if accepted.\n\nBelow are the StarCoder-15B’s results on APPS and CodeContests. Results of the other two backbones can be found in our attached author response PDF file (Tables 4 and 5). Due to the large amount of test samples in APPS (5000), we only test the other two backbones on a subset of 200 samples. We plan to complete the evaluation and add it to the paper.\n\n| Approaches | | APPS (5000) | | CodeContests (165) | |\n|-|-|-|-|-|-|\n| | | Pass@1 | Pass@10 | Pass@1 | Pass@10 |\n| | Init. | 2.57 | 8.59 | 0.58 | **3.88** |\n| Prompt | Refine | 2.84 | 9.10 | 0.65 | 4.23 |\n| | Expl. + Ref. | 2.95 | 9.53 | 0.78 | 4.84 |\n|||||||\n| | Init. | 3.80 | 11.52 | **0.62** | 3.67 |\n| SFT | Refine | 6.89 | 17.01 | 1.16 | 5.25 |\n| | Expl. + Ref. | 6.86 | 17.18 | 1.48 | 5.94 |\n|||||||\n| | Init. | **4.50** | **13.62**  | 0.44 | 2.19 |\n| RL | Refine | **7.81** | **19.28** | **1.80**  | **6.04** |\n| | Expl. + Ref. | **8.10** | **19.75** | **1.80** | **6.37** |\n\nOn APPS and CodeContests, the prompting baseline also only refines very few solutions, and the improvements brought by refinement are marginal. However, with our SFT and RL-trained models, we see significantly stronger self-refinement ability. The final Pass@k is also about doubled from the prompting baseline.\n\n## 4. Related works\nWe thank the reviewer for pointing out missing related works and providing some discussions here.\n\nIn the Reflexion paper, an LLM verbally reflects on task feedback signals and generates improvements. It is similar to the prompting method. \n\nOpenCodeInterpreter constructs a multi-turn interaction dataset that integrates execution and human feedback for code refinement using GPT-4, and shows that models on such dataset achieve good refinement performance. \n\nIn EURUS paper ([3] mentioned by the reviewer), it curates an UltraInteract dataset as an alignment dataset for complex reasoning tasks using GPT-3.5-Turbo and GPT-4, and includes multi-turn interaction trajectories with the environment and the critique. \n\nBoth of the last two papers obtain multi-turn feedback interaction dataset using close models GPT-3.5-Turbo and GPT-4, and perform finetuning. Different from them, our paper shows that even without strong LLMs like GPT-3.5-Turbo and GPT-4, we can generate synthetic data for self-debugging using much smaller open pretrained/instruct models or the same LLM itself from the additional results from this rebuttal, and they are quite effective. We also proposed a novel RL training with explanation and execution reward in our paper.\n## 5. Details on training data\nHere is a summary of the APPS dataset and the number of problems that has at least one correct solution (either correct initial solution or correct refinement from GPT-3.5-Turbo) based on difficulty levels:\n\n| | Introductory | Interview | Competition |\n|-|-|-|-|\n| Count | 2410 / 2639 | 493 / 2000 | 306 / 361 |\n\n\nFor the CodeContests dataset, the data is from multiple sources and difficulty levels are not comparable. We give the sample distribution from CodeChef with difficulty levels 1-4:\n\n| | EASY (1)  | MEDIUM (2) | HARD (3) | HARDER (4) |\n|-|-|-|-|-|\n| Count | 19 / 86 | 65 / 330 | 18 / 90 | 1 / 6 |\n\nProblems without any correct initial solutions or refinements are discarded from the training data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JwO5AHl2ch", "reviewer": "Reviewer_uGRw", "summary": "This work proposes a novel framework to enhance the self-debugging capabilities of smaller language models that do not benefit much from self-refine or other prompt-based debugging approaches. Sampling incorrect code samples produced by LMs, they pass the execution feedback on these to GPT-3.5/4 and prompt it to explain the reason for the errors and propose code refinements. Accurate refinements are used to fine-tune and create a code-correcting model. This is further enhanced with a PPO based learning from a novel reward assignment mechanism accounting for both explainability and code refinement. Overall, they demonstrate the importance of having explanations for incorrect codes and how RL can be used to enhance the debugging ability of models to show superior performance across benchmark datasets.", "questions": "- Was there any reason behind choosing the range of score to be [-5,5]?\n- It seems in the bigger models tested (\\geq 13B) the refine itself is quite effective, any intuitions on this scale effect?", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Technically solid\n- This work addresses a significant issue of little coding improvement from self-refinement prevalent in smaller LMs\n- The reward setup incorporating both code refinement and explainability in PPO is novel and shows good gains", "weaknesses": "Nothing major, future evaluation on datasets like APPS etc. could be beneficial to understand the impact on harder tasks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments.\n\n## 1. Evaluation of APPS and CodeContests\nThis is a good suggestion and we plan to add the results to the final version if accepted.\n\nBelow are the results on APPS and CodeContests. We test StarCoder on the full 5000 APPS test samples. However, due to the large amount of test samples in APPS (5000), we only test the CodeLlama-7B/13B on a subset of 200 samples. We plan to keep running to complete the evaluation and add it to the paper.\n\n| Approaches | | APPS (5000) | | CodeContests (165) | |\n|-|-|-|-|-|-|\n| | | Pass@1 | Pass@10 | Pass@1 | Pass@10 |\n| | Init. | 2.57 | 8.59 | 0.58 | **3.88** |\n| Prompt | Refine | 2.84 | 9.10 | 0.65 | 4.23 |\n| | Expl. + Ref. | 2.95 | 9.53 | 0.78 | 4.84 |\n|||||||\n| | Init. | 3.80 | 11.52 | **0.62** | 3.67 |\n| SFT | Refine | 6.89 | 17.01 | 1.16 | 5.25 |\n| | Expl. + Ref. | 6.86 | 17.18 | 1.48 | 5.94 |\n|||||||\n| | Init. | **4.50** | **13.62**  | 0.44 | 2.19 |\n| RL | Refine | **7.81** | **19.28** | **1.80**  | **6.04** |\n| | Expl. + Ref. | **8.10** | **19.75** | **1.80** | **6.37** |\n\nOn APPS and CodeContests, the prompting baseline also only refines very few solutions and the improvements brought by refinement are marginal. However, with our SFT and RL-trained models, we see significantly stronger self-refinement ability. The final Pass@k is also about doubled from the prompting baseline.\n\n## 2. Reward Range\nOur RL algorithm is based on the PPO algorithm. According to the details in Appendix A3, the rewards of each token are either the code reward, explanation reward, or the KL divergence. From our calculation of the KL divergence distribution on a subset of the training data, the trajectory tokens’ KL divergences have a minimum value of -5.22 and a maximum value of 5.37. Thus, we scale our code and explanation rewards to the range of [-5, 5], which is similar to the KL divergence value range.\n\nActually, we think as long as the code/explanation rewards are not too large or too small to cause gradient explosion, the setup should be reasonable. For example, [-1, 1], [0, 1], are also common choices of reward range.\n\n## 3. Scale Effect\nWe observed a similar scaling effect in Table 4 in Section 4.1.2 where we show the successful refinement rate of different models across prompting, SFT, and RL. We see that from CodeLlama 7B to CodeLlama 13B, the refinement rate improves by around 3% (absolute rate) across the board for all approaches. We think that larger models have better capabilities in general, and the trend applies to self-debugging capability as well. Our proposed training seems to help push the self-debugging performance closer to the model’s limit, and in general, a larger model should have a higher upper bound.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oxmy9dKTAp", "reviewer": "Reviewer_osNA", "summary": "The paper addresses the goal of self-debugging of generated code, while also explaining it. \nThe approach is to: (a) sample code outputs to natural language inputs, and keep only the wrong code outputs according to unit tests; (b) sample **refinements** (\"fixes\") to the wrong code outputs and test those refinements using unit tests, to get both correct and incorrect refinements; (c) the authors train an LLM using SFT and RL on those correct and incorrect refinements.\n\nThe resulting trained LLM is shown to provide empirical gains, especially when allowing it to self-refine its outputs at test time, with the test-time unit tests.", "questions": "1. At test time, when evaluating the SFT/RL models: does the model see the execution results of its generated test code before refinement? That is, can the model use the unit tests of the test examples as well, or are unit tests used only at training time?\n2. In the definition of $S_{cb}$ - if we rely on unit tests to verify correctness, why do we need to encourage the refinement to be similar (in terms of CodeBLEU) to **all** the possible correct refinements?\n3. RL training on benchmark data may over-specialize on their specific domain, while degrading the general coding abilities of the LLM. After all, these benchmarks are only benchmarks, and over-specializing on them may hurt the usability of the model in practical use. For example, have the authors checked whether applying their SFT+RL hurts the perplexity on general code?\n\nTo summarize, I think that the paper presents strong empirical gains, but the scientific novelty is low, as the contribution is mostly applied. I thus vote for a borderline reject.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed pipeline provide strong empirical gains\n* The authors experiment with 3 base LLMs (StarCoder-15B, CodeLlama-7B, CodeLlama 13B) and 2 \"teacher\" LLMs (gpt-3.5-turbo, CodeLlama-34B)\n* The authors experiment with multiple datasets: they train on MBPP, APPS, CodeContest, and test on HumanEval and MBPP.", "weaknesses": "* The paper is mostly applicative, and includes a mix of known techniques. I feel that the paper is low on conceptual novelty. The concept of self-debugging and verifying correctness using unit tests was introduced by [Chen et al., 2023](https://arxiv.org/pdf/2304.05128) (although only using prompting), and RL using the signal coming from the unit tests was done in several papers (that the authors cite). \n* It feels like there are so many different techniques and heuristics involved, that it is hard to pinpoint the exact contribution of each of them. For example: the design of the explanation score as $CosSim(RoBERTa(e), RoBERTa(ec))$, the exact hyperparameters of the reward design (e.g., $\\frac{50 · S_{ex}(e) 35}{3}$), and the exact design of $S_{cb}$ using CodeBLEU. The use of explanations before fixing the code is a form of Chain-of-Thought (Wei et al., 2022), or the \"feedback\" in Self-Refine (Madaan et al., 2023).\n* Further, teaching the models to self-debug is done using **larger models**. \nThat is, the refinements are sampled from larger models such as gpt-3.5-turbo, CodeLlama-34B, and then these refinements are used to train the smaller StarCoder-15B, CodeLlama-7B, CodeLlama 13B models.\nThis adds a dimension of distillation (from large models to small models), and further makes it difficult to pinpoint the exact source of contribution.\n* The approach is not compared to any related work, or to the teacher models themselves.\n* Position in literature - Although many papers are cited, I feel that the paper does not position itself well in the literature. I do not remember exactly what did each related work do, but the paper does not help me understand the differences and its novelty compared to the related work. For example, what's the difference between this paper and [18] and [20]? They seem very similar, but the Related Work does not highlight the novelty over them.\n    * Another new paper that was not cited: [Ansong et al., NExT: Teaching Large Language Models to Reason about Code Execution, ICML'2024](https://arxiv.org/pdf/2404.14662). I am not interested in the authors just citing the paper as an additional number between brackets, I am interested in discussing the actual differences and novelty over that paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for insightful suggestions and questions.\n\n## 1. Paper novelty\nWhile there are several related works on self-debugging, our paper focuses on how to improve the model’s self-debugging capability, which is important but not yet extensively investigated. \n\nWe believe \"NExT: Teaching Large Language Models to Reason about Code Execution\" is a concurrent work with us. Yet there are still differences between the NExT paper and our paper:\n1. One of the most notable differences is the proposed **RL training with explanation and execution reward**. RL training is important as it helps LLMs to learn from the failure generations as well, and the separation of explanation reward and execution reward helps LLMs learn differently about the explanations and fixes. \n2. Differences also lie in how we synthesize explanations while they focus on traces reasoning, and how we **utilize much larger training datasets such as APPS and CodeContests** to improve generalization while they primarily focus on MBPP and HumanEval as the training data. \n3. Also, they mainly conduct training experiments on PaLM 2, while we **train multiple open-sourced backbones to prove the generalizability of our approach**.\n \nWe will add a discussion on the differences with this paper in our draft, but we do not think that the contribution and novelty of our paper should be questioned based on this concurrent work.\n\n## 2. Position in literature\nAs mentioned above, our paper focuses on how to improve the model’s self-debugging capability via training. Most existing works on self-debugging focus on prompting LLMs to do self-debugging, which does not work well on open-sourced smaller LLMs as we have shown.\n\nA few related works that train LLMs as we cited in the paper:\n1. ILF requires human-annotated explanations, while we train LLMs to generate bug explanations by themselves.\n2. CYCLE and Self-Edit only train LLMs to generate refinement using SFT. We train LLMs to explain the bug which not only enhances LLMs’ reasoning but also helps developers to understand the wrong code (as our human evaluation shows). We also explored using RL to further improve the self-debugging performance.\n3. NExT, a concurrent work from ICML this year that we have discussed above.\n\nWe are also different from all these works by designing RL training. The RL training brings improvement to the strong SFT models over several baselines, leading to higher Pass@K, higher refinement rate, and better bug explanation.\n\n## 3. Self-taught refinement\nWe provide the experiment results using the synthetic data generation from the model itself for CodeLlama 7B. We highlight some results here. Below is the CodeLlama-7B SFT/RL using the data collected from itself, evaluated on MBPP+ and HumanEval+.\n\n| Approaches | | MBPP+ | | HumanEval+ | |\n|-|-|-|-|-|-|\n| | | Pass@1 | Pass@10 | Pass@1 | Pass@10 |\n| | Init. | 37.18 | 61.23 | 27.40 | 60.81 |\n| Prompt | Refine | 42.97 | 66.89 | 31.84 | 65.08 |\n| | Expl. + Ref. | 42.46 | 67.41 | 32.49 | 66.58 |\n|||||||\n| | Init. | 41.78 | 61.77 | 33.25 | 61.50 |\n| SFT | Refine | 46.26| **66.39** | 40.15 | 67.15 |\n| | Expl. + Ref. | 45.94 | 65.77 | 39.10 | 67.33 |\n|||||||\n| | Init. | 41.61 | 61.29 | 33.66 | 62.17 |\n| RL | Refine | **46.28** | 65.86 | **41.54** | 68.14 |\n| | Expl. + Ref. | 46.10 | 65.99 | 40.79 | **68.50** |\n\nThe full results are in our attached author response PDF file, Tables 1 and 2. Results show that self-taught SFT and RL also achieve big improvements. CodeLlama-7B SFT/RL models achieve up to 5% improvement in self-debugging, compared with the baseline prompting method, and the model trained with code generation data only. Though comparing the experiments using data from CodeLlama-34B and GPT-3.5-Turbo, the improvement is less.\n\n## 4. Comparison to the teacher models\nWe provide the comparison with the teacher models CodeLlama-34B and GPT-3.5-Turbo on the self-debugging setup in the global rebuttal pdf. Comparing it with Table 2 and Table 10 in the paper, we see that with CodeLlama-34B as the teacher, CodeLlama 7B SFT/RL achieves close to CodeLlama-34B self-debugging performance, while CodeLlama-13B SFT/RL significantly outperforms the CodeLlama-34B teacher (e.g. in HumanEval+ pass@1 56.24% vs 48.51%).\n\n## 5. Unit Test at test time\nThe model first generates an initial solution for a given problem description. The initial problem description contains one or more test case examples (example input and expected output).  \n\nThen the initial solution is tested against all the test cases provided for the problem. If one test case fails, the model will take the failed test case and the error message to generate refinement. \n\nThat is, at the test time, the generation of the initial solution will see some example test inputs and outputs. And the generation of refinement (and explanation) will see the exact failed test cases.\n\n## 6. Reward Design\nCodeBLEU: We find that only using binary execution feedback as a reward does not train the model properly as the reward is too sparse. That is, a completely wrong solution will get the same reward as an almost correct solution, which hurts the RL training. Although the CodeBLEU reward is weak, we find that having it helps stabilize the training by densifying the reward distribution. This is the main reason we introduce the CodeBLEU score in the reward.\n\nSimilarity: The formula for R(e) might look strange, but the main goal is to scale the majority explanation similarity (Figure 3 c in paper) to the range of [-5, 5] so that it is in the same value range as the code rewards (Figure 3 d in paper). Our reward design is based on the statistics of the training data.\n\n## 7. Generalization\nTo avoid overfitting, we use a large batch size (128) to only update the model for a few thousand steps. We test our models’ perplexity on 10000 samples from BigQuery Python code. The CodeLlama-7B pre-trained model’s perplexity is 1.457, the SFT/RL models' are 1.597 and 1.599, just slightly higher.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response

## 1. Contribution and Novelty
While there are more and more works on self-debugging, most of the existing works focus on how to prompt existing LLMs to do self-debugging. Few works investigate the self-debugging capability of LLMs and how to improve it at the time of submission. Our paper primarily focuses on how to improve the self-debugging capabilities of LLMs via **training**. 

We propose a systematic way from synthetic data generation to SFT and RL training with novel explanation and execution rewards. This is the main contribution and novelty of our paper. RL training is important as it helps LLMs to learn from both success and failure generations, and the separation of explanation reward and execution reward helps LLMs learn differently about the explanations and fixes. We also show that LLMs without the proposed training have poor self-debugging capability even if it is trained with strong code-instruction data, and the proposed method is important in improving this capability. 

While there seems to be a concurrent work from ICML 2024 on the same topic, there are significant differences and novelties in our paper, for example, the proposed RL training with explanation and execution reward, how we synthesize explanations to explain bug, how we conducted much larger scale experiments such as APPS and CodeContests, and the ablation studies we performed to showcase the importance and effectiveness of our method.

## 2. Self-taught Synthetic Data Generation
Many reviewers are interested in this. The synthetic data generation can also be done on the same model forming a self-taught manner, instead of “distillation” from stronger models. We provide additional experiment results of this kind in the attached PDF for CodeLlama-7B, where the synthetic data is generated from the model itself and it is used to fine-tune the same model. 

Experiments show that self-taught synthetic data generation and training also achieve significant improvements. CodeLlama-7B SFT model achieves up to 5% improvement in self-debugging using data generated from CodeLlama-7B, compared with the baseline prompting method and code-instruction-only model. Though comparing the experiments using data from CodeLlama-34B and GPT-3.5-Turbo, the improvement is slightly less.

## 3. Evaluation on APPS and CodeContests
Also many reviewers are interested in this. We provide evaluation results on the test set of APPS and CodeContests with StarCoder-15B below and the other two backbones in the attached PDF file (Tables 4 and 5). On APPS and CodeContests, the prompting baseline also only refines very few solutions, and the improvements brought by refinement are marginal (on APPS 2.57%->2.84%; on CodeContests 0.58%->0.78%). However, with our SFT and RL-trained models, we see significantly stronger self-refinement ability (on APPS 4.50%->8.10%; on CodeContests 0.44%->1.80%), about doubled from the prompting baseline. 


| Approaches | | APPS (5000) | | CodeContests (165) | |
|-|-|-|-|-|-|
| | | Pass@1 | Pass@10 | Pass@1 | Pass@10 |
| | Init. | 2.57 | 8.59 | 0.58 | **3.88** |
| Prompt | Refine | 2.84 | 9.10 | 0.65 | 4.23 |
| | Expl. + Ref. | 2.95 | 9.53 | 0.78 | 4.84 |
|||||||
| | Init. | 3.80 | 11.52 | **0.62** | 3.67 |
| SFT | Refine | 6.89 | 17.01 | 1.16 | 5.25 |
| | Expl. + Ref. | 6.86 | 17.18 | 1.48 | 5.94 |
|||||||
| | Init. | **4.50** | **13.62**  | 0.44 | 2.19 |
| RL | Refine | **7.81** | **19.28** | **1.80**  | **6.04** |
| | Expl. + Ref. | **8.10** | **19.75** | **1.80** | **6.37** |

## 4. Comparison with teacher model
We provide the comparison with the teacher models in our attached author response PDF file. 

Comparing it with Table 2 in the paper, with GPT-3.5 as the teacher, CodeLlama-13B SFT/RL cannot outperform GPT-3.5, which could be because GPT-3.5 is a much stronger teacher model than our backbones.

Comparing it with Table 10 in the paper, with CodeLlama-34B as the teacher, CodeLlama-13B SFT/RL **sometimes outperforms the CodeLlama-34B teacher** (e.g. in HumanEval+ pass@1 56.24% vs 48.51%, and in MBPP+ pass@1 56.60% vs 53.19%). This is non-trivial and surprising that our approach enables the smaller LLMs to outperform the teacher models sometimes.

## 5. Pass@K+1 versus refinement Pass@K
The reviewer mention a very interesting point: generation followed by one round of refinement may not be better than simply regenerating one more solution. That is, is Pass@K of generation and refinement better than Pass@K+1 of generation only?

We evaluate the Pass@2 of the initial solution and Pass@1 after one round of refinement. Below is the result when the models are not trained (the prompting baseline). We do observe that Pass@2 is better than Pass@1 after refinement. **This shows that using a prompting approach to self-debug is not effective**.

| Before Training | | MBPP+ | HumanEval+ |
|-|-|-|-|
| StarCoder-15B | Pass@2 | **45.20** | **36.38** |
| | Expl. + Ref. Pass@1 | 39.27 | 30.09 |
| CodeLlama-7B | Pass@2 | **46.91** | **38.17** |
| | Expl. + Ref. Pass@1 | 42.46 | 32.49 |
| CodeLlama-13B | Pass@2 | **48.08** | **41.52** |
| | Expl. + Ref. Pass@1 | 45.77 | 38.36 |

However, after the models are trained using our pipeline. The refinement Pass@1 is clearly higher than the Pass@2 of initial solutions. This shows that the model’s self-debugging performance is bad without training. The prompting approaches proposed by existing works such as (https://arxiv.org/pdf/2306.09896 and https://arxiv.org/abs/2304.05128) are not as effective on open-sourced LLMs. **This experiment further supports our motivation to train LLMs to self-debug and proves the effectiveness of our approach.**

| After SFT | | MBPP+ | HumanEval+ |
|-|-|-|-|
| StarCoder-15B | Pass@2 | 51.19 | 39.29 |
| | Expl. + Ref. Pass@1 | **53.83** | **43.54** |
| CodeLlama-7B | Pass@2 | 50.93 | 40.95 |
| | Expl. + Ref. Pass@1 | **51.55** | **47.62** |
| CodeLlama-13B | Pass@2 | 50.93 | 44.78 |
| | Expl. + Ref. Pass@1 | **54.59** | **51.32** |

Meta-review: The paper proposes a framework to improve self-debugging for code generation tasks through a combination of SFT and RL using failed tests, their fixes coupled with explanations. The reviewers unanimously recommend its acceptance. They note strong empirical results obtained using comprehensive analysis done with various datasets and models, highlight its novelty for the reward setup and the importance of the problem. I therefore recommend its acceptance.
Decision: Accept (poster)