Is Self-Repair a Silver Bullet for Code Generation?
Authors: Theo X. Olausson
Jeevana Priya Inala
Chenglong Wang
Jianfeng Gao
Armando Solar-Lezama
Venue: ICLR 2024 poster (2024)
Keywords: program synthesis
code generation
large language models
machine learning for code
Primary area: generative models
TL;DR: We study self-repair for code generation, finding that gains are often marginal and quite inconsistent, and offer several insights as to why.

Abstract: Large language models have shown remarkable aptitude in code generation, but still struggle to perform complex tasks. Self-repair---in which the model debugs and repairs its own code---has recently become a popular way to boost performance in these settings. However, despite its increasing popularity, existing studies of self-repair have been limited in scope; in many settings, its efficacy thus remains poorly understood. In this paper, we analyze Code Llama, GPT-3.5 and GPT-4's ability to perform self-repair on problems taken from HumanEval and APPS. We find that when the cost of carrying out repair is taken into account, performance gains are often modest, vary a lot between subsets of the data, and are sometimes not present at all. We hypothesize that this is because self-repair is bottlenecked by the model's ability to provide feedback on its own code; using a stronger model to artificially boost the quality of the feedback, we observe substantially larger performance gains. Similarly, a small-scale study in which we provide GPT-4 with feedback from human participants suggests that even for the strongest models, self-repair still lags far behind what can be achieved with human-level debugging.

Published as a conference paper at ICLR 2024
Is SELF-REPAIR A SILVER BULLET FOR CODE GENERATION?
Theo X. Olausson 1 Jeevana Priya Inala?, Chenglong Wang", Jianfeng Gao", Armando Solar-Lezama' MIT CSAIL Miicossoft Research
ABSTRACT
Large language models have shown remarkable aptitude in code generation, but still struggle perform complex tasks. Self-repair-in which the model debugs and repairs its own code-has recently become popular way to boost performance in these settings. However, despite its inereasing popularity, existing studies of self-repair have been limited in scope; in many settings, its efficacy thus remains poorly understood. In this paper, we analyze Code Llama, GPT-3.5 and GPT-4's s ability to perform self-repair on problems taken from HumanEval and APPS. We find that when the cost of carrying out repair taken into account, performance gains are often modest, vary a lot between subsets of the data, and are sometimes not present at all. We hypothesize that this is because self-repair is bottlenecked by the model's ability to provide feedback on its own code; using stronger model to artificially boost the quality of the feedback, we observe substantially larger performance gains. Similarly, small-scale study in which we provide GPT-4 with feedback from human participants suggests that even for the strongest models, self-repair still lags far behind what can be achieved with human-level debugging.
INTRODUCTION
Large language models (LLMs) have proven capable of generating code snippets from natural language specifications, but still struggle on complex coding challenges such those found in competitions and professional software engineering interviews. Recent work has sought to improve performance by leveraging self-repair (Gupta al., 2020; Le al., 2022; Chen et al., 2023b; Zhang et al., 2023), in which the model introspects and corrects mistakes in its own code. Figure shows a typical workflow. First, a program is sampled from a code generation model; this program is then run on a suite of unit tests provided as part of the specification; if the program fails any test, then the error message and the faulty program are given to a feedback generation model, which outputs a short explanation of why the code failed; finally. the feedback is passed to repair model, which generates a fixed version of the program. On the surface, this is a very attractive idea. It allows the system to overcome mistakes caused by unfortunate samples during decoding; easily incorporates feedback during the repair phase from symbolic systems such as compilers, static analysis tools, and execution engines; and mimics the trial-and-error way in which human software engineers write code.
However, it is important to remember that self-repair requires more invocations of the model, thus increasing the computational cost. In particular, whether self-repair is winning strategy or not ultimately boils down to whether you would- at an equivalent compute budget- have had a greater chance of success if you had simply drawn more code samples i.i.d. from the model and checked them against the suite of unit tests provided as part of the task. Crucially, in a competitive programming setting the efficacy of self-repair depends not only on the model's ability to generate code, which has been studied extensively in the literature, but also on its ability to identify how the code (generated by the model itself) is wrong with respect to the task specification. As far as we are aware, no previous work has studied the effect of this stage in detail.
In practice, generating feedback and producing the corrected code be done through single interaction with the model; as we will see, it can still useful to conceptually treat them as separate steps. "oorrespondence to thexooccaa 1 mi t. edu. Work partially done while T.X.O. was at Microsoft. Research. Code and data available at jithub.com/thexx//ll--rggg

---

Published as a conference paper at ICLR 2024
assert (54)
noter 10 =o
(1)
(00)
aseett 1000 y
Code
Execution
Feedback Model
Code Model
def retuzn
(2)
Given string representing the day
[7S0R1,
... 'ERI"
'B82' Lindex [2)1
the wask today. is SUN,
After many days is the next Sundey (tomorrow later)?
MON, TUE, WED, THU, FRI, or SAT.
Given input SUN, the program returned O, but the expected output was 7.
(3)
(1)
The code does not account the case where the irput is 'SNN' and the oupput should be 7. This can be fixed by removing the modulo operation.
DN2T 25375
(4)
(SXECU/ABBE)
((OON)))
assert ('MED") ")
assert SUN1))
def (5) return
17S0R1, ... 'ERI", "BA71] 1. Lindaxd
Figure 1: Self-repair with separate code and feedback models. First, a user gives a specification in the form of text and a suite of unit tests (1). Then, a code model (blue) generates a program (2). The program checked against the unit tests using a symbolic execution engine, and an error message is returned (3). In order to provide more signal to the code model, textual feedback as to why this happened is provided by a feedback model (yellow; 4). Finally, this feedback used by the code model to repair the program (5).
Contributions: In this paper, we investigate the efficacy of self-repair techniques applied to BaeeLlama--3b-insteee (Rozière et al., 2023), GPT-3.5 (Ouyang et al., 2022; OpenAI, 2022), and GPT-4 (OpenAI, 2023) for self-contained Python programming tasks. We focus evaluating the models' capacity to reflect upon, provide feedback on and debug the code. We observe that:
Self-repair is not a silver bullet: when the cost of repair is taken into account, we find several instances in which pass rates are higher or equally high with i.id. sampling (without repair), especially when the budget is small. We conjecture that this is because program generation and repair rates trend together, and many subtle factors influence which will overpower the other for given task (see Appendix C).
Self-repair is more likely to be beneficial when more of the sampling budget is spent on generating a diverse set of initial programs than on carrying out extensive repair. For example, for GPT-4 on APPS, drawing 10 samples up front and then 1 repair candidate each (up to 20 samples total) leads a pass rate 1.05x higher than pas s 820 from the same model without repair (from 63.1% to 66.1%) drawing 2 samples up front and then drawing 10 repair candidates each (up to 22 samples total) leads to a pass rate which lower than the baseline pass@22 (0.97x; 63.4% + 61.8%).
Artificially boosting the quality of the feedback significantly improves the efficacy of self-repair. We replace Code Llama's feedback with that produced by GPT-3.5 or GPT-4, and GPT-3.5's feedhack with that of GPT-4; in every case, the boosted configuration beats both the corresponding i.i.d. bascline and the corresponding self-repair configuration at all budgets. Furthermore, replacing GPT- ss own explanations with those of a human programmer improves repair significantly, increasing the fraction of repaired programs which pass the tests by factor of 1.58x (33.3% - 52.6%).
2 RELATED WORK
Program synthesis with large language models. The use of large language models for program synthesis has been studied extensively the literature (Li et al., 2022; Austin et al., 2021; Chen et al., 2021; Le et al., 2022; Fried et al., 2023; Nijkamp et al., 2023; Chowdhery et al., 2022; Touvron et al., 2023; Li et al., 2023). This literature has predominantly focused on evaluating models terms of either raw accuracy or the pass@k @@k metric (Kulal et al., 2019; Chen et al., 2021), often leveraging filtering techniques based on execution (Li et al., 2022; Shi et al., 2022) or ranking (Chen et al., 2021; Inala et al., 2022; Zhang et al., 2022) to reduce the number of samples which are considered for the final answer. Our work differs from some of the work in this literature that we assume access to the full collection of input-output examples, as is typically done in inductive synthesis (Kitzelmann, 2010;
2

---

Published as a conference paper at ICLR 2024
Polozov & Gulwani, 2015; Gulwani et al., 2017; Chen et al., 2019a; Ellis al., 2021). In particular, unlike some prior work (Li et al., 2022: Shi et al., 2022), we do not make a distinction between public tests used for filtering and private tests used to determine correctness, since our method does not involve filtering the outputs.
Code repair. Statistical and learning- -aaeed code repair has rich history in both the programming languages and machine learning communities, although it has predominantly been applied to code written by humans in software engineering context (Long & Rinard, 2016; Bader et al., 2019; Le Goues et al.. 2021; Yasunaga & Liang, 2021; Chen et al., 2019b; Mesbah et al., 2019; Wang et al., 2018). More recently, using repair as post-processing step improve code which was itself automatically synthesised has been used in the synthesis of both domain-specific languages (Gupta et al., 2020) and seneral-purposs code (Le et al., 2022; Yasunaga & Liang, 2021; 2020). Our contribution differs from most prior work in this literature in the use of textual feedback for repair, which is possible thanks to the above mentioned rise in the use of LLMs for program synthesis.
Contemporary work on LLM self-repair There is much contemporary work seeking to self-repair with LLMs, both in code generation and beyond. We now highlight a few of these works which are particularly close to ours; see Pan et al. (2023) for more complete survey of recent work in this quickly evolving field. Zhang et al. (2023) explore self-repair without natural language feedback on APPS (Hendrycks et al., 2021) using both finetuned models and prompt- based self-repair with Codex (Chen et al., 2021), InCoder (Fried et al., 2023), and CodeGen (Nijkamp et al., 2023). Notably, their framework does not consider the cost associated with feedback and repair, which presents significantly different perspective. Similarly, Chen et al. (2023b) assess Codex's 's ability to self-repair across variety of tasks, in a framework that closely resembles that which we study in this work. However, their study differs from ours in terms of the models considered and, more importantly, the research goal, as we specifically aim to investigate the significance of the textual feedback stage. Outside of code generation, self-repair has been used for wide array of purposes, including mitigating hallucinations and improving factual grounding search assistants (Peng et al., 2023) as well as code optimization and readability improvements (Madaan et al., 2023). Ultimately, we see our work, in which we investigate the significance of the textual feedback stage in particular, as being complementary to contemporary research which seeks to evaluate self-repair in a broader context; we are eager to see what the implications of our results will be in these other domains.
METHODOLOGY
3.1 SELF-REPAIR OVERVIEW
As shown in Figure 1, we model self-repair as consisting of four stages: code generation, code execution, feedback generation, and code repair. We now formally define these different stages.
Code generation. Given a specification v. programming model Mp first generates ny samples i..dd, which we denote
(p. ( igdd Mp(()
Code execution. These np code samples are then executed against a test bed. If any sample p passes all of the tests-which we denote p = gh-we stop, since satisfying program has then been found. Otherwise, we collect the error messages (es returned by the execution environment. These error messages either contain the compile/runtime error information an example input on which the program's output differs from the expected one. An example is shown in Figure (component 3).
Feedback generation. Error messages from the execution environment are usually very high-level, providing little signal for repair. Therefore, as an intermediate step, we use a feedback model to produce a more detailed explanation of what went wrong; Figure (component 4) shows an example. Formally, in this stage, we generate n, feedback strings, {fe, bi for each wrong program, Pi, as follows:
(Ai))/ itd. Mr (1i1p));
Having an explicit feedback generation step allows us to ablate this component so that we can study its significance in isolation.
We assume access to the full set of tests in executable form; see Section 5 for brief discussion on the validity of this assumption in software engineering domains.
3

---

Published as a conference paper at ICLR 2024
Code Gen
PL
Ps
fu
...
Jan
Sxpt
frynz
Repair
P113
mi
Flny1
T2
Tryln.
Tapsy?
Figure 2: repair tree begins with a specification & (root node), then grows into initial programs {p:). feedback {fig]. J. and repairs {raza).
Code repair. In the final step, for each initial program pr and feedback Sij, nt candidate repaired programs are sampled from Mp 3:
(raa) 1.d.
Mp(pipres fis)
Repair tree. We call the tree of interleaved text and programs produced by this procedure-roowe in the specification vs, then branching into initial programs pt, each of which branches into feedback fij and then repairs Tiyk-a repair tree, T (Figure 2).
Jointly sampling feedback and repair. The general framework presented above does not require the programming model and feedback model to be the same, thus allowing for the use of specialized models in the system. When Mp Mr. we jointly generate both the feedback and the repaired program a single sample from the model; see Appendix G for detailed look at how the prompt differs between this and the previous setting. Formally, we denote this as
(Uij,,u))) i.d.d Mr((ippe)))
3.2 PASSOK FOR SELF-REPAIR
In program synthesis without self-repair, performance is typically measured by pass@k (Chen et al., 2021; Kulal al., 2019)-the the probability that at least one of l: i.i.d. program samples from the model satisfies a given specification. In self-repair, program samples are drawn from the model both during the initial sample stage and during the repair stage; thus, we need to adopt pass@k k to take into account the number of samples from both stages.
In the main body of this work, we treat repair trees T as themselves forming independent samples from joint model T ~ -M (Mp Mr Mp) and define the number of programs in the tree as pprograms (T) A np + fr (or (programs(T') A np + mpn,,,,) we then compare against baseline with k = programs [T) i.i.d. samples. We believe this will make our findings most relevant practitioners, who are likely to deploy self-repairing agents with batched sampling. Appendix A repeats our experiments with two alternative evaluation strategies, in which we vary the search strategy and measure sampling cost by the total number of tokens sampled from the model to take into account the varying lengths of feedback and program samples. Importantly, although the details differ, the overall trends which we observe remain the same.
Independently generating a large amount of repair trees for each setting of the hyper- parameters quickly becomes computationally infeasible, so we plot bootstrapped estimates of the pass rates in our experiments. We first generate single very large repair tree for each task specification, with: N, 2 Mip initial program samples; N, 2 ny feedback strings per wrong program; and N, n, repair candidates per feedback string. Given setting of (np; ,,,,,,), n,),
we then sub-sample (with replacement) N, = 1000 different sub-repair-trees from this frozen dataset and average over the runs. We use N, = 50 for all experiments, and consider n. < 25 for the self-repair approaches and np I 50 for the baseline, no-repair approach. Similarly, for the feedback strings, we use N, = 25
We use the same model for both the initial code generation and the code repair, since these are fundamentally similar tasks.
4

---

Published as a conference paper at ICLR 2024
and n S 10 (except for Section 4.2, in which only consider n 1 and therefore settle for Ny = 10 instead). Finally, for the repair candidates we set N, ny = 1, since we do joint sampling of feedback and repair in most of our experiments. Estimating the pass rates in this way greatly reduces the computational cost of our experiments, since we can reuse the same initial dataset to compute the estimates for all of the various choices of nay ny, and ny--
4 EXPERIMENTS
In this section, we carry out experiments to answer the following research questions: (a) In the context of Python programming puzzles, is self-repair better than i.idd. sampling without repair for the models we consider? If so, under what prper-paraeeters is self-repair most effective? (b) Would stronger feedback model boost the model's 's repair performance? (c) Would keeping human in the loop to provide feedback unlock better repair performance even for the strongest model?
We evaluate these hypothesis for two API-served Models-GPPSS (Ouyang al., 2022; OpenAI, 2022) and GPT-4* (OpenAI, 2023)-as as well as Codel Laadellamaalll 1ama-133ba 13b-instruct (Rozière et al., 2023), a model with publicly accessible weights which consider Python programming challenges from both APPS (Hendrycks et al., 2021) and HumanEval (Chen et al., 2021); on APPS, order to keep our experiments tractable, we evaluate on a randomly chosen set of 300 tasks. 6 We implement self-repair using templated string concatenation with one- shot prompting: our prompts are given in Appendix G. Based on preliminary experiments, we set the decoding temperature to 0.8 for all models. When appropriate, we compare against a baseline without repair. This baseline, shown with a black line in the plots, is simply i.i.d. sampling from the corresponding model (c.g., GPT-4 when we explore whether GPT-4 is capable of self-repair).
be run locally on consumer-level hardware. We
4.1 SELF-REPAIR IS NOT A SILVER BULLET, BUT IMPROVES WITH DIVERSE INITIAL SAMPLES
In this subsection, we consider the setup where Mp Mp, i.e., a true self-repair setting in which a single model is used for both code/repair generation and feedback generation. To evaluate self-repair leads to better performance than no-repair, i.i.d. sampling-basee vary ny and nyr that is, the number of initial i.i.d. base samples and joint feedback, repair samples drawn from My-in the range (Mp; ffr) E 21, 5, 10, 25} x (1, 5, 10).
baseline approach, we
Figure 4 shows the results for Code LLama and GPT-3.5 on HumanEval, while Figure 3 shows the results for GPT-3.5 and GPT-4 on the more challenging APPS dataset. (We also run GPT-4 on HumanEval and CodeLlama on APPS, but defer these results to Appendix B for brevity.) In the left-hand subplots, the color of each dot indicates the number of initial samples (nz)) while its shape indicates the number of feedback-ree creairr samples (nfr). In the right hand plots, we show a heat-mat -ap with the two parameters along the axes, where the value in each cell indicates the mean pass rate with self-repair normalized by the mean pass rate of the bascline, no-repair approach when given the same budget. When the normalized mean pass rate is 1, this means that self-repair achieves the same pass rate the no-repair, baseline approach at that same sample budget; higher value (2 1) means self-repair performs better than the baseline.
On APPS, we observe marginal gains for GPT-3.5 only for the largest values of npp GPT-4, on the other hand, shows more significant improvements, beating out the baseline by up to 8%. When we break the problems down by their difficulty level (see figures in Appendix C), we find that gains are larger on harder problems: GPT-3.5 sees up to a 34% performance gain relative to the bascline on problems, for example. Meanwhile, on HumanEval we observe performance gains similar to those of GPT-4 on APPS for Code Llama (up to 10% improvement relative to the baseline), while gains for GPT-3.5 are limited as it approaches the ceiling (up to 3%).
competition-leye
'We use the frozen endpoints gpt Stt.s--t-ut000000 and gpt-4-031 5 'nttps: httpss///huugee //tuggingface co/ http///teeaaaaaaeeeeeeeeeeaaaaee Laauna/CooeLLata 3b- Instruct
'These tasks are proportionally sampled in accordance with the frequency of the different difficulty levels in the broader APPS test set: 180 interview-level questions, 60 competition-level questions, and 60 introductory- level questions. All tasks are listed in Appendix H.
'Recall that when Mp Mr, we jointly sample for n2rr pairs of feedback strings and repair programs instead of sampling them one after another (Section 3.1).
5

---

Published as a conference paper at ICLR 2024
1.0
n,m1
P
0.83
0.89
O.O.B.
0.O.B. O.O.B.
0.8
0g
Feedback-repairr (ni))
Mean pass rate
0gm
0g 10
10
0.6
Op
25
0.4
0.87
0.92
0.96
1.01
O.O.B.
0.2
0.91
0.95
0.98
1.00
1.04
10
20
30
40
50
2
5
10
25
Number of programs sampled
Initial programs (np)
(a) GPT-3.5.
1.0
0.90
0.97
0.0.B.
O.O.B.
O.O.B.
Feedback-repairs (nk)
Mean pass rate
0.0.B.
0.6
=
1.03
O.0.B.
By =
0.2
Fly 10
0.99
1.00
1.03
1.05
1.08
10
20
30
40
50
10
25
Number programs sampled
Initial programs (np)
(b) GPT-4.
Figure 3: GPT-3.5 and GPT-4 self-repair results APPS. Left: Mean pass rate vs. number of samples generated. Black line is i.i.d. sampling without repair from the same model. Note that the error bars often smaller than the markers. Right: Normalized mean pass rate relative the baseline at an equivalent budget. Cells for which the number of samples exceeds 50 marked O.O.B. (out of bounds).
From these observations, is clear that self-repair not always the best strategy when compared to a non-repair baseline with the same sample budget, especially for smaller budgets. Moreover, it is hard to predict when self-repair will be effective. In an analysis of the repair success rates (Appendix C), we find that stronger models have higher repair success rates easier tasks but at the same time, the chance of getting a correct program by resampling also increases the easier a task is. Therefore, see that program generation and repair success rates trend together, and many subtle unknown factors influence which one will overpower the other on any given domain.
While the overall efficacy of self-repair is unclear, do observe a clear trend with respect to the relationship between the increasing the number of initial programs (n,,) (i.e., moving right along the x-axis the heat maps) consistently leads to relative performance gains for all models. On the other hand, fixing n. and increasing yr (i.e.. moving up along the y-axis on the heat maps) does not appear to be worth the additional cost incurred, giving marginal gains at higher budgets and oftentimes even decreasing performance at lower budgets. This suggests that, given fixed budget, the most important factor determining whether self-repair will lead to correct program not is the diversity of the base samples that are generated up-front, rather than the diversity of the repairs sampled. Having more initial samples increases the likelihood of there being at least one program which is close the ideal program and, hence, can be successfully repaired.
Given a fixed number of feedback-repairs (n,r),
Since n,r 1 appears be the best overall choice for the hyper-parameter Nyr- next iso- late the effect of the number of initial programs, nps by exploring a denser set possible val- ues: (M, (fr) 21,2,... 24, 25} x (1). The plots shown in Figure 5 for Mp Mp

---

Published as a conference paper at ICLR 2024
1.0
0.94
1.00
O.0.B.
0.0.8. O.O.B.
Feedback-repairss (ni))
0.8
Mean pass rate
0.6
0.4
0.94
0.99
1.06
1.09
0.O.B.
=
0.2
Ry 10
0.95
1.00
1.04
1.07
1.09
20
30
40
50
10
25
Number of programs sampled
Initial programs (np)
(a) CodeLlama- 1aamma--3--aee 13b-instruct.
1.0
0.97
1.00
0.0.B.
0.O.B.
O.O.B.
Feedback-repairs (ni)
Mean pass rate
0.6
0p=1
=
1.02
0.0.B.
Fy =
0.2
Op 10
Fy 10
0.99
1.00
1.02
1.02
1.03
30
40
50
2
10
25
Number programs sampled
Initial programs (np)
(b) GPT-3.5.
Figure 4: CodeLlama- 1aaiaa------insteue and GPT-3.5 self-repair results on HumanEval. Left: Mean pass rate vs. number of samples generated. Black line is i.idd sampling without repair from the same model. Note that the error bars often smaller than the markers. Right: Normalized mean pass rate relative to the baseline at an equivalent budget. Cells for which the number samples exceeds 50 marked O.O.B. (out of bounds).
1.0
1.0
0.8
Mean pass rate
Muw Code Uomo (no ropair)
0.6
0.6
GPT-3.5 repair)
repair
Oode Uiema
Mean
0.4
0.4
My- GP7.355
My Rt am1-
Mom Code Usmes My GPT-3.5
Mum Code Uomo: GPT
CPT1-355 My car-11
10
20
30
40
50
10
20
30
40
50
Number of programs sampled
Number programs sampled
(a) HumanEval.
(b) APPS.
Figure 5: Results when ny (or yy and n,)) = 1. Shaded region shows 11 standard deviation.
(CodeLlama, GPT-3.5, GPT-4) and the bascline, no-repair approaches." We observe performance gains for both Code Ilama and GPT-3.5 on HumanFval. On APPS, only GPT-4 significantly benefits from self-repair, while both Code Ilama and GPT-3.5 mostly lag behind or match their baselines, possibly seeing some very marginal gains at high budgets. In all cases, performance gains at smaller budgets are very marginal or non-existant, but grow somewhat as the budget increases.
As GPT-3.5 is already near ceiling on HumanEval, we omit GPT-4 from this figure to reduce clutter. Note that since nyr is fixed, in these plots, there is direct correlation between np and k: h n, ng.

---

Published as a conference paper at ICLR 2024
Table 1: Success rate of repair with GPT-4's explanations vs, with those of our human participants.
Difficulty
Introductory
Interview
Competition
Overall
GPT-4 Feedback
42.64% 62.21%
19.33% 45.67%
3.67% 14.67%
33.30%
Human Feedback
52.60%
4.2 BOOSTING THE FREDBACK UNLOCKS PERFORMANCE GAINS FROM REPAIR
Next, we conduct an experiment in which we evaluate the impact of using a separate, stronger model to generate the feedback; this is to test the hypothesis that self-repair is held back by the model's inability to introspect and debug its own code. We thus set Mp to be weaker model (Code Llama on EvaanEval, Code Llama or GPT-3.5 on APPS) and Mr to be a stronger model (GPT-3.5 or GPT-4 for Code Llama on HumanEval; GPT-3.5 for Code Llama and GPT-4 for GPT-3.5 on APPS). We then vary the hyper-parameee as (Mp. Ry ny.) E {1,..... 25} x {1} x {1), similarly to the previous experiment." 10
The results for this experiment also shown in Figure 5 (Code Llama paired with GPT-3.5 in yellow; Code Llama with GPT-4 in bright green; GPT-3.5 with GPT-4 in bright blue). We observe consistent trend: on APPS, both Code Llama and GPT-3.5 now beat out both their baselines (dark green, gray) and their respective self-repair modes (purple, red). On HumanEval, the performance that Code Llama gains increases further with the strength of the feedback model; note in particular the performance that Code Llama gains when given feedback from GPT-4 (bright green line). This suggests that the textual feedhack stage itself is of crucial importance, and that improving it relieves the bottleneck in self-repair.
4.3 HUMAN FEEDBACK SIGNIHICANTLY IMPROVES THE SUCCESS RATE OF GPT-4 REPAIR
For our final experiment, we consider the effect of using an expert human programmer's feedback when performing repair with very strong models such as GPT-4. The goal of this study is not to do direct comparison between approach imposes more cognitive burden, which we do not study. Instead, our goal is to further investigate how and why feedback quality affects downstream performance in self-repair.
couman-in-the-aa
approach vs. self-repair, since a
Data collection methodology. We recruit 16 participants and collect a total of 2 human human-writter written pieces of feedback for each of 40 failing programs sampled from GPT-4. Each program is shown to two different participants, to reduce variance caused by participants skill levels and writing style. Participants were asked to spend approximately one hour on the study overall, and were compensated with a $15 gift card. This study was approved by our Institutional Review Board (IRB) and carried out exclusively through online survey, See Appendix D for more details the data collection methodology, including a complete copy of the instructions which we provide to our participants.
Quantitative analysis. Having obtained two human-written pieces of feedback for each program, we sample 25 repair candidates for each (feedback, program)-pal from GPT-4. We condition on the specification, the initial program, and the feedback string; in addition to the feedback collected from our participants, we also try two of GPT-4's own feedback strings for each program. Finally, we execute all of these candidate repairs against the test bed, and take note of how often they pass.
The results are summarized in Table 1, with a complete task-by-tase breakdown in Appendix E. We note that the overall success rate increased by over 1.58x when we replace GPT-4's 's own feedback with that of our human participants. Perhaps unsurprisingly, the relative difference increases as the problems get harder, indicating that GPT-4's ability to produce accurate and useful feedback trails further behind our human participants' when the task (and code) becomes more complex.
Qualitative analysis. We manually go through all of GPT-4's and the participants' feedback and note down whether the feedback: (a) seems, at a cursory glance, to be correct, or if it is obviously inaccurate; (b) explicitly suggests a small change to the code (e.g. "change the condition on line
10 Note that since we are now operating in a setting in which the feedback and repair stages must be separated, we have three Syper--prameters--aee ny, iinittttead of two-np, firr (Section 3.1).
To reduce cost, use Ny 10 instead of N, = 25 for this experiment (see Section 3.2).
8

---

Published as a conference paper at ICLR 2024
X"); (c) explicitly suggests a large change to the code (e.g. "frame the problem as min-cut instead of poortest-path")) (d) contains blocks of pseudocode or Python (which GPT-4's feedback never does, per experiment design); (e) expresses uncertainty (using phrases such "unsure", "it appears", etc.). 12 Examples of each category are shown in Appendix F. We find that
Almost all human-contributed feedback interleaves natural language with occasional single- statement math/code expressions; only 2/80 responses include pseudocode or explicit Python.
GPT-4's feedback is more likely to be obviously inaccurate (32/80 vs. 7/80 for the human feedback).
GPT-4 is more likely to explicitly suggest small changes (54/80 vs. 42/80 for GPT-4 and the participants, respectively; 28/48 vs. 38/73 if we filter out suggestions which are obviously incorrect), while human participants show a slightly greater tendency to suggest high-level changes (23/80 vs. 18/80 for GPT-4; 21/73 vs. 13/48 when seemingly correct).
Our human participants sometimes express uncertainty (7/80); GPT-4 never does (0/80).
This further analysis suggests that the results in Table are not due to artefacts such as our participants providing explicit code blocks which the model simply copies. Instead, the difference in performance appears to be caused by combination of more accurate feedback, a greater ability to suggest high- level, large-scale changes to the code when needed, and our participants ability to express their uncertainty (instead of confidently giving potentially inaccurate feedback).
5
LIMITATIONS
Firstly, to reduce computational cost, we pre-populate and then sub-sample from a single large repair tree to bootstrap a large number of repair trees for each setting of the hyper-pann parameters (Section 3.2). This risks introducing statistical artefacts in our analysis. To minimize this risk, we bounded np and fr far below Np and N respectively, in our self-repair experiments. Furthermore, we note that the standard deviation is very small in our experiments for all values of np and nfr (see the scatter plots in Figures 3, 4), offering increased confidence in our results.
Secondly, our experiments focus on self-containel Python programming tasks with executable unit tests. This is quite different from real-world software development tasks, where specifications are often incomplete, there long contextual dependencies, and tests unlikely to be available for each individual snippet. Future work will be required to see what role self-repair can play there: for example, whether it could resolve ambiguities in the specification, or automatic unit test synthesis techniques (Li et al., 2022; Chen et al., 2023a) could be leveraged alongside established engineering practices like Test-Driven Development (Astels, 2003) to overcome the lack of high quality tests.
Finally, our study on human data did not track how much time the participants took to debug the programs. As result, we can only evaluate the quality of the feedback (and the impact this has on repair). Further research at the intersection of Human-Compute Interaction, AI, and program synthesis is needed to explore when and how human intervention should be leveraged, as well as how programming assistants should be designed to facilitate this style of interaction.
6 CONCLUSION
We investigated self-repair for code generation, looking specifically at CodeLlama- BareLllaa---b--aae 13b-instruct, GPT-3.5 and GPT-4 on problems taken from HumanEval and APPS. In a series of experiments, we observed that (1) when the cost of carrying repair is taken into account, performance gains from self-repair are often modest, vary not only between but also within datasets, and rely on achieving sufficient diversity in the initial programs. Furthermore, by replacing the feedback stage we found that (2) substituting a weaker model's own feedback with that of a stronger model significantly improved performance. Finally, we carried out an experiment with human participants, in which we found that (3) replacing GPT-4's 's self- self-generated feedback with feedback provided by an experienced programmer increased the number of repaired programs which pass all unit tests by 1.58x. Our results suggest that self-repair is not silver bullet for code generation, and that current models are held back by their inability to reliably produce accurate and useful feedback on why the code is wrong.
We do count individual single-line statemeents/expressions such as "I 5" as pscudocode Python.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
T.X. Olausson is supported by the Defense Advanced Research Projects Agency (DARPA) under the ASKEM program, award HR001 12220042. T.X. Olausson was also supported through a position at Microsoft Research for part of the time period during which this work was carried out. A. Solar- Lezama supported by the National Science Foundation (NSF) and Intel Corporation through NSF Grant CCF:2217064. This work benefited greatly from discussion with several colleagues at Microsoft Research. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, the Defense Advanced Research Projects Agency, Intel Corporation, or Microsoft Research.
REFERENCES
Dave Astels. Test Driven Development: A Practical Guide. Prentice Hall Professional Technical Reference, 2003. ISBN 0131016490.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. Program Synthesis with Large Language Models, 2021. arXiv preprint arXiv:2108.0777 077332 abs/ 22008 07732.
Johannes Bader, Andrew Scott, Michael Pradel, and Satish Chandra. Getafix: Learning to fix bugs automatically. Proc. ACM Program. Lang-, ((OPSIL)) Oct 2019. doi: 10.1145/3360585.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian Jian-Guang Lou, and Weizhu Chen. CodeT: Code generation with generated tests. In International Conference on Learning Representations, 2023a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating Large Language Models Trained on Code, 2021. arXiv preprint arXiv:2107.03374, https: //atpis://rxvv... //rxivv. org/abs/ 2107. 03374.
Xinyun Chen, Chang Liu, and Dawn Song. Execution-Guided Neural Program Synthesis. In International Conference on Learning Representations, 2019a.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching Large Language Models to Self-Debug, 2023b. arXiv preprint arXiv:2304.05128. https: http://aavvvv..o wrg. 05128.
abs//////////// /2004.
Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. SequenceR: Sequence-to-Sequeee Learning for End-to-End Program Repair. IEEE Transaction on Software Engineering, 2019b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling Language Modeling with Pathways, 2022. arXiv preprint arXiv:2204.02311 https:
//arxiv. www./rr///aaao abs/ 2204.2222222 02311.
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua Tenenbaum. DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning. In The International Conference on Programming Language Design and Implementation, 2021.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoeer, and Mike Lewis. InCoder: A generative model for code infilling and synthesis. In International Conference on Learning Representutions, 2023.
Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. Program Synthesis. Foundations and Trends@ in Programming Languages Series. Now Publishers, 2017. ISBN 9781 9786682832....
10

---

Published as a conference paper at ICLR 2024
Kavi Gupta, Peter Ebert Christensen, Xinyun Chen, and Dawn Song. Synthesize, Execute and Debug: Learning to Repair for Neural Program Synthesis. In Advances in Neural Information Processing Systems, 2020.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring Coding Challenge Competence With APPS. In Advances in Neural Information Processing Systems, 2021.
Jeevana Priya Inala, Chenglong Wang, Mei Yang. Andres Codas, Mark Encarnación, Shuvendu Lahiri, Madanlal Musuvathi, and Jianfeng Gao. Fault-Aware Neural Code Rankers. In Advances in Neural Information Processing Systems, 2022.
Emanuel Kitzelmann. Inductive Programming: A Survey of Program Synthesis Techniques. In Approaches and Applications of Inductive Programming: Third International Workshop, 2010.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. SPoC: Search-based Pseudocode Code. In Advances in Neural Information Processing Systems, 2019.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. In Advances in Neural Information Processing Systems, 2022.
Claire Le Goues, Michael Pradel, Abhik Roychoudhury, and Satish Chandra. Automatic Program Repair. IEEE Softw., 38(4):22-27, jul 2021. ISSN 0740-7459. doi: 10.1109/MM.2221133725577.
Raymond Li, Loubna Ben Allal, Yangtian Zi Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. StarCoder: may the source be with you!, 2023. arXiv preprint arXiv:2305. 231.06660...
https: / /a//arivvv.////..///aa/////avtvv wwwpvv..org/bbaa.eo 2305. 061666666 61.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. Competition-level code generation with AlphaCode. Science, 3186662)) 2022. doi: 10.1126/siience.aag11555
Fan Long and Martin Rinard. Automatic Patch Generation by Learning Correct Code. In ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, 2016.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, al. Self-Refine: Iterative Refinement with Self-Feedback, 2023, arXiv preprint arXiv:2303.17653 https: /ttp:://aaxvvvvv.. wwtp://///aaa/////// 17651.
Ali Mesbah, Andrew Rice, Emily Johnston, Nick Glorioso, and Edward Aftandilian. DecpDelta: Learning to Repair Compilation Errrr.. In Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2019.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. CodeGinn An Open Large Language Model for Code with Multi-Turn Program Synthesis. In International Conference on Learning Representations, 2023.
OpenAl. Introducing ChatGPT, 2022. Blog post. https: http://pppaaaaaa/ea com wwtpclcg/cccccccccccccooccccccca chatgpt. [Accessed 5/17/20233.
OpenAI. GPT-4 Technical Report, 2023. arXiv preprint arXiv:2303.08877. http:::ttttsstrreeeee org/abs 2303. 08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse celf-correction strategies. arXiv preprint arXiv:2308.03 188, 2023.

---

Published as a conference paper at ICLR 2024
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. Check your facts and try again: Improv- ing large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302. 12813, 2023.
Oleksandr Polozov and Sumit Gulwani. FlashMeta: A Framework for Inductive Program Synthesis. In ACM SIGPLAN International Conference on Object-Orientee Programming, Systems, Languages, and Applications, 2015.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usumier. Thomas Scialom, and Gabriel Synnaeve. Code Llama: Open Foundation Models for Code. arXiv preprint arXiv:2308. 12950, 2023.
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. Natural Language to Code Translation with Execution. In Empirical Methods Natural Language Processing, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models, 2023. arXiv preprint arXiv:2302. 13977. https:
//arxiv. wwwpxvvvorg/aaoooooo abs 2302. 13971.
Ke Wang, Rishabh Singh, and Zhendong Su. Dynamic Neural Program Embedding for Program Repair. In International Conference on Learning Representations, 2018.
Michihiro Yasunaga and Percy Liang. Graph-based, Self-supervised Program Repair from Diagnostic Feedback. In International Conference Machine Learning, 2020.
Michihiro Yasunaga and Percy Liang. Break-It-Fix-Ii Unsupervised Learning for Program Repair. In International Conference on Machine Learning, 2021.
Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. Self-Edit: Fault-Aware Code Editor for Code Gen- eration, 2023. arXiv preprint arXiv:2305.04087 https: / aaaaaiiiivv.ooo http://awwawl../...././//// /2005. 04087.
Tianyi Zhang, Tao Yu, Tatsunori B Hashimoto, Mike Lewis. Wen-tau Yih, Daniel Fried, and Sida I. Wang. Coder Reviewer Reranking for Code Generation, 2022. arXiv preprint arXiv:22/1.16499 https://arxivv oww/orgg///oooo /2111 6490.
12

---

Published as a conference paper at ICLR 2024
Data: Task dh; sample budgets np: fif; fyy
Data: Task vi; sample budgets np; fl f2 A tuple (success, token count)
Result: A tuple (success, token count) P P [ ) - to npl: f t sum nuu tokens(p) pe P)):
Result:
0;
for i < to n. do p: f- (b);
atay(pp = & Pl) then return (True, 1);
É <- + size(p:);
end
if p: - v then return (True, t); end
R 1--- D: for p E P do
error ppog_mss(pppg v); Fp - [Mr(i: p; e) l + 0 to nyl:
e, emror msg(Ps: 15;; for j 4 to ny do Jú t- Mr(p;; p&; pi)e:); I { + (tze( for 1 to n2 do Fisk + Mp( paie fiy); t t (rajk)
1 < L con[num_tokenss
FR)::
for fe F do Rps + [M [Mp(D:pre; l itt
0 1,]
t ++ (nmm (ouu__ttees(()) r
if Fijle + v then return (True, t);
Rppl) ))
R + R Rpf
end
end end
end end end
if any([r } + R)) then return (True, t); end
return (False, t);
Algorithm 2: Generating a repair tree T, com- puting T }= & and its token count with se- quential self-repair. All operations executed serially.
return (False, 1);
Algorithm 1: Generating repair tree 7, com- puting T += sb and its token count with batched self-repair. All operations should be taken to run in parallel whenever possible.
A ALTERNATIVE EVALUATION STRATEGIES FOR SELF-REPAIR
In the main part of this paper, we chose to evaluate self-repair in terms of an adapted version of pass@k (Chen al., 2021; Kulal et al., 2019), in which a single repair tree is considered equivalent to k: np + Nip * fr samples from the baseline. This makes the results easy to digest for practitioners and scholars who are familiar with p k and prior work in this literature. However, this evaluation strategy does not account for the feedback tokens produced by the same model, which also come at cost, and so risks overemphasizing the benefits of self-repair.
In this appendix, we present and briefly discuss results in terms of two alternative evaluation strategies which address the non- con-uniform costs of program and feedback samples by comparing two dependent variables-thh pass rate and the number of tokens which had to be sampled from the model in order to achieve it-an approach which we dub pass@t. This allows us to compare not only how successful particular configuration is but also how much "work" it requires from the model.
Formally, suppose that you are given a dataset D = (waje and a chosen set of values for the hyree-parameters (Mp, MF, Top, n.). Let Tä ~ M (vaa) denote a repair tree that is sampled as described in Section 3.1 for the task dha let num nww_nnuuueeeeeeeee tocktkkesss].... denote the total number of program and feedback tokens in the repair tree; and say that T += via is true if, and only if, T has at least one leaf program that satisfies the unit tests in the specification via. Then the pass@t metric of this choice of hyper-parann sypee--prammttes is defined the expected pass rate the number of tokens which you would expect to generate with this choice of
pass8t
E
[Ta - val at t =
E [num_tokens R
SA-D (Wa)
13

---

Published as a conference paper at ICLR 2024
A.1 BATCHED PASSOT
The first variation we will consider is batched passøt. In this strategy, repair trees are assumed be generated in Algorithm I: all up initial programs sampled in parallel, then checked for correctness; if none of them pass, then all "p * " repairs of all initial programs are sampled in parallel, after which we check if any of the repairs pass. The total number of tokens sampled far is recorded at every point, and returned alongside the value Tf } t. Thus, the number of tokens which are sampled depends on both the success rate in the initial round of program generation as well as the relative verbosity of the feedback and programs. Averaging the results over all of the tasks, we get not only a mean pass rate but also mean token count, which can be plotted together as points on curve.
Figures 6, and show the results of experiments from main paper, repeated with this evaluation strategy. Note that while these plots may at first look much like those of Section they subtly different that both axes are now dependent variables (recall that in pasa@k, & is an independent variable set ahead of time). The better particular model is, the closer it would thus get to (0.0, 1.0)- ie. the top-left corner of the plot.
Broadly speaking, observe the trends as well as both Code Llama and GPT-3.5 on HumanEval; larger gains when the feedback is provided by the stronger model; typically better performance when setting ny fi.r. except for GPT-3.5 HvmanEval where performance is relatively stable across the board as is already near ceiling.
noted in Section 4: gains for GPT-4 APPS
1.0
10
0.78
0.86
O.O.B.
O.O.B. O.O.B.
0.8
Fy -
Feedback<repairs (nx)
Mean pass rate
5
Fy
10
0.79
0.86
0.94
O.O.B.
O.O.B.
0, 25
0.4
0.81
0.87
0.94
0.99
0.0.B.
0.86
0.91
0.95
0.98
0.0.B.
0.0
20.00
4000
60000
8000
10000
2
25
Mean number of tokens generated
Initial programs (np)
(a) GPT-3.5.
10
0.90
0.98
1.05
O.O.B.
O.O.B.
Feedback-repairs (nd)
0.8
Mean pass rate
0.91
0.98
1.04
1.08
0.0.B.
0.6
0,
0.4
nm=2
3
0.93
0.98
1.04
1.08
0.0.B.
n,55
0.2
0g 10
10
0.98
1.01
1.03
1.06
1.09
0u= 25
0.0
2000
4000
6000
8000
10000
25
Mean number of tokens generated
Initial programs (np)
(b) GPT-4
Figure 6: GPT-3.5 and GPT-4 self-repair results on APPS, evaluated in terms of batched pass@t. C.f. Figure 3. N.B.: The heatmaps here display the normalized mean pass rate relative to the (interpolated) baseline at an equivalent number of tokens.
14

---

Published as a conference paper at ICLR 2024
1.0
10
0.94
1.01
1.08
0.0.B.
O.O.B.
Feedback-repaiis (md)
0.8
Mean pass rate
0.93
1.01
1.08
1.11
0.0.B.
0.6
=1
=
0.4
=
Fly =
0.94
1.00
1.00
1.11
0.0.B.
5
0.2
10
1.11
25
0.0
4000
10000
1
2
5
10
25
Mean number takens generated
Initial programs (ng)
(a) CodeLlama- 13b-in---b-isisteut..
40-19
10
1.01
1.03
1.05
1.05
0.0.B.
Feedback (na)
0.8
Mean pass rate
1.02
1.04
1.05
1.05
1.04
0g-1
0.4
=
ny
1.02
1.04
1.05
1.05
1.04
0g
np 10
10
1.04
1.04
1.04
1.04
1.04
np 25
0.0
2000
4000
6000
8000
10000
25
Mean number of tokens generated
Initial programs (np)
(b) GPT-3.5.
Figure 7: Baeelllla-1---inttnnee and GPT-3.5 self-repair results on HumanEval, evaluated in terms of batched passêt. C.f. Figure 4. N.B.: The heatmaps here display the normalized mean pass rate relative to the (interpolated) baseline an equivalent number of tokens.
1.0
1.0
0.8
0.8
Mean pass rate
Mean pass rate
0.6
0.4
0.2
80
Mean number of tokens generated
Mean number of tokens generated
(a) CodeLlama and GPT-3.5 on HumanEval,
(b) GPT-3.5 and GPT-4 on APPS.
Figure 8: Batched pass@t curves for each model when ffr (or ny and n,.) 1. C.f. Figure
15

---

Published as a conference paper at ICLR 2024
A.2 SEQUENTIAL PASSET
The batched sampling approaches considered in Section and A. I.I are designed to mimic the way in which practitioners are likely deploy large-scale self-repair without user intervention. However, this quite different from the way in which end-users interact with chat-based programming assistants. This is likely take on more sequential form, where the user first receives a single program, spends some time trying to get the assistant to debug it, before finally giving up and starting over from scratch new session. One might be curious how our results extend to such setting.
In this section, we model self- self-repair as depth- first search for passing program, where the parameters np: ff: : ny are taken to bounds on the widths of each level: this shown Algorithm 2. Note that this even more tightly couples the observed pass rates and the number of tokens generated: if the pass rate is high, a passing program will quickly be found and number tokens generated will be low, and vice versa.
We again repeat the experiments from the main paper: the results are shown Figures 9, 10, 11. As before, the key trends are still discernible. However, in this setting, self-repair appears to be somewhat less beneficial; especially when the baseline pass rate already high. This is particularly visible when comparing the heatmaps in Figures and 10 to those from before (e.g.. 6, 7), well as Figure 11.
1
=
10
0.75
0.84
O.O.B. O.O.B. O.O.B.
0.8
ny =
Feedback-repairs (nt)
Mean pass rate
5
Fy
10
0.77
0.83
O.O.B.
O.O.B. O.O.B.
25
0.4
0.78
0.84
0.91
O.O.B.
O.O.B
0.84
0.88
0.92
0.96
0.0.B.
0.0
2000
4000
6000
8000
10000
25
Mean number of takens generated
Initial programs (np)
(a) GPT-3.5.
Feedback<repirss (no)
0.8
Mean pass rate
0.87
0.94
1.02
0.O.B.
0.0.B.
0.6
0.4
no=2
3
0.89
0.94
1.01
O.O.B.
0.O.B.
-
0.2
Fy 10
0.94
0.97
1.00
1.03
O.0.B.
25
0.0
2000
4000
6000
8000
10000
25
Mean number takens generated
Initial programs (np)
(b) GPT-4
Figure 9: GPT-3.5 and GPT-4 self-repair results on APPS, evaluated in terms of sequential pass@t. C.f. Figure 3. N.B.: The heatmaps here display the normalized mean pass rate relative to the (interpolated) baseline an equivalent number of tokens.
16

---

Published as a conference paper at ICLR 2024
10
0.88
O.O.B.
O.O.B. O.O.B. O.O.B.
reppaic (nc)
0.8
Mean pass rate
0.87
0.95
O.O.B.
O.O.B. O.O.B.
Py
0.4
Pr
0.88
0.94
1.02
0.0.B. 0.0.B.
0pm
0g 10
10
0.90
0.94
1.00
1.04
0.O.B.
ng 25
0.0
2000
4000
6000
8000
10000
25
Mean number of takens generated
Initial programs (ng)
(a) CodeLlama-130 13b-instruct.
O.O.B.
O.O.B.
Feedback-repainss (no)
0.8
Mean pass rate
0.6
0p11
P+
0.4
0,
=
0.95
0.98
1.00
0.O.B.
0.0.B.
5
0.2
0p
Fy 10
0.97
0.98
1.00
1.01
0.0.B.
0.0
2000
4000
6000
8000
10000
2
10
25
Mean number tokens generated
Initial programs (ng)
(b) GPT-3.5.
Figure 10: CodeLlama- 1aaa--11b--intucc and GPT-3.5 self-repair results HumanEval, evaluated in terms of sequential passêt. C.f. Figure 4. N.B.: The heatmaps here display the normalized mean pass rate relative to the (interpolated) baseline an equivalent number of tokens.
1.0
1.0
0.8
0.8
Mean pass rate
Mean pass rate
0.6
0.6
0.4
0.4
0.2
0.2
2000
4000
6000
8000
10000
2000
4000 6000 8000 10000
Mean number of tokens generated
Mean number tokens generated
(a) CodeLlama and GPT-3.5 HumanEval.
(b) GPT-3.5 and GPT-4 APPS.
Figure 11: Sequential passêt curves for each model when nyr (orny and n,,) = 1. C.f. Figure 5.
17

---

Published as a conference paper at ICLR 2024
B ADDITIONAL RESULTS: GPT-4 ON HUMANEVAL, CODE LLAMA ON APPS
1.0
10
1.00
1.01
O.O.B.
0.0.8. 0.0.B.
Feedback-repairs (nm)
Mean pass rate
1.01
1.02
1.01
0.0.8.
0.0.B.
0.6
0pm1
3
ny
0.2
10
10
1.01
25
0.0
30
40
50
2
5
10
25
Number programs sampled
Initial programs (np)
(a) GPT-4 on HumanEval.
1.0
10
0.67
0.76
O.O.B.
O.O.B. O.O.B.
ng
3
Feedback-repairs (nx)
Mean pass rate
Fy =
Fly 10
0.72
0.80
0.89
0.0.8.
0.0.B.
0.6
0.4
0.77
0.84
0.91
0.96
0.0.B.
0.2
1.03
0.0
1
2
5
10
25
Number of programs sampled
Initial programs (np)
(b) Cada 13b-instrutttttttteeee APPS.
Figure 12: Full Code Llama APPS and GPT-4 HumanEval results. Omitted from Section 4.1 for brevity.
18

---

Published as a conference paper at ICLR 2024
C ADDITIONAL RESULTS: SELF-REPAIR VS. PROBLEM DIFFICULTY
APPS problems are divided into three categories: introductory, interview and competition. This makes easy repeat our APPS experiments on problems of specific difficulty; the results are shown in Figures 13 through 16. We note that both GPT-3.5 and GPT-4 appear to benefit more from self-repair the harder the problem is. Meanwhile, Code Llama benefits less; we also note that GPT-3.5's baseline performance on APPS iitrooutryy problems (Figure 14, top) very similar to that of GPT-3.5 on HumanEval (Figure 4b), yet self-repair only appears fruitful in the latter experiment. The relationship between baseline performance and the efficacy of self-repair thus appears not be so clear cut.
One might also want to evaluate the success rate of repair without the confounding factor of how often the initial sample programs passes the tests; intuitively, we expect that harder programs should be harder to repair. Table 2 shows the fraction of repaired programs which pass the tests on APPS. Although it is important not to place too much weight on the specific numbers, since- for example- less performant model's 's initial programs might be more difficult to repair than those generated by stronger model, these results agree with our intuition.
We leave it future work investigate in detail why self-repair performance gains do not appear to trend perfectly with baseline performance; we offer the conjecture that it is due to a combination of (a) the power struggle between feedback generation and repair success (which benefit self-repair) vs. program generation success rate (which benefits i.i.d. sampling without repair); (b) the prevalence of ambiguity the natural language specification, which might affect self-repair's ability correctly identify flaws in failing program; and (c) the informativeness of the unit tests. In the meantime, as has been shown in this work, improving the model's ability to provide feedback on code (e.g. through finetuning on code explanation data) can boost the performance of self-repair.
Table 2: Repair success rates in various settings. The repair success rate is computed as number_of ofaberr___psssin__repaiii http:/aaaaobeeroooooeeeeieii
Dataset
Difficulty
Model
Repair Success Rate
CL-13b-instruct CL-133-instruct-GP1335
2.8%
5.4%
introductory
GPT-3.5
13.7%
GPT-3.5+GPT-4
29. 1%
GPT-4
28.8%
CL-13b-instruct
1.0%
1.9%
interview
GPT-3.5
4.2%
GPT-3.5+GPT-4
11.2%
APPS
GPT-4
8.7%
CL-13b-instruct
0.1%
1.--3bbiistruct-GPP3.55
0.4%
competition
GPT-3.5
1.5%
GPT-3.5+GPT-4
3.3%
GPT-4
8.6%
CL-13b-instruct
1.1%
C1--13b-instruct--PPr..3
2.2%
overall
GPT-3.5
4.7%
GPT-3.5+GPT-4 GPT-4
11.5%
10.8%
CL-13b-instruce
9. 9.1%
CL- G5--
20.1% 1%
HumanEval
CL- IL-133binstruct+GPP-
39.3%
GPT-3.5
22.4%
GPT-4
49.6%
19

---

Published as a conference paper at ICLR 2024
1.0
Dp
Pr 1
10
0.73
0.82
O.O.B.
O.O.B. 0.O.B.
0.8
Dp
Dr 3
Feedback repairs (me)
Mean pass rate
np
5
np = 10
10
5
0.77
0.86
0.94
O.O.B. 0.O.B.
0.6
25
0.96
1.01
0.2
0.87
0.93
1.00
1.03
1.08
0
10
20
30
40
50
Number of programs sampled
Initial programs (np)
1.0
ng
0.63
0.73
O.O.B.
O.O.B. 0.0.B.
0.8
Pp=
Fry
Feedback-repatt repairs (nn)
Fry 5
Mean pass rate
n2 10
Fr.
0.69
0.76
0.86
O.O.B. 0.O.B.
0.6
Пo 25
0.4
3
0.74
0.80
0.88
0.94
0.0.B.
0.2
1
0.86
0.90
0.94
0.97
1.01
0.0
20
30
40
2
10
25
Number of programs sampled
Initial programs (np)
1.0
Dp 1
10
0.56
0.56
O.O.B.
O.O.B. 0.0.B
0.8
n,
Feedback repairs (nt)
Meon pass rate
5
np 10
5
0.63
0.65
0.65
O.O.B. 0.0.B.
n, 25
0.4
3
0.2
0.87
0.84
0.0
0
10
20
30
40
50
Number of programs sampled
Initial programs (np)
Figure 13: CodeLlama- i3b-instruct results from Figure 12b (Appendix B) per APPS difficulty (row),
from top bottom: int interview, and competition.
20

---

Published as a conference paper at ICLR 2024
1.0
10
0.88
0.93
O.O.B.
O.O.B. 0.O.B.
Feedback repairs (ne)
0.8
Mean pass rate
5
0.89
0.94
0.97
O.O.B. 0.O.B.
Dp = 1
= 1
Pp
3
0.97
5
0.2
7p 10
10
0.93
0.97
0.99
1.00
1.00
n2 25
0
10
20
30
40
50
Number of programs sampled
Initial programs (np)
1.0
np=
Dr
0.79
0.85
O.O.B.
O.O.B. 0.0.B.
0.8
my=
Pi,
Feedback rebbarrk--epprssss (nn)
Mean pass rate
Pp 5
Fiy
repairs
no = 10
Di.
0.82
0.87
0.93
O.O.B.
0.0.B.
0.6
пo 25
0.4
3
0.84
0.89
0.94
0.99
0.0.B.
0.2
1
0.90
0.93
0.96
0.98
1.03
0.0
20
30
40
2
10
25
Number of programs sampled
Initial programs (np)
1.0
Dp =
= 1
10
0.88
1.03
O.O.B.
O.O.B. 0.0.B
0.8
n,
Feedback repairs (nt)
Meon pass rate
5
np 10
5
0.91
1.01
1.18
O.O.B. 0.0.B.
n, 25
0.4
3
1.02
1.17
0.2
1.01
1.13
1.34
0.0
0
10
20
30
40
50
Number of programs sampled
Initial programs (np)
Figure 14: GPT-3.5 results from Figure (Section 4.1) per APPS difficulty (row), from top to bottom: roductory interview, and competition.
21

---

Published as a conference paper at ICLR 2024
1.0
10
0.97
0.99
O.O.B.
O.O.B. 0.O.B.
Feedback repaiirs (ne)
0.8
Mean pass rate
5
0.98
1.00
1.01
O.O.B. O.O.B.
0.6
=p=1
Dly =
1.00
1.01
Mp
n
0.2
Пp 10
10
1.02
1.02
1.02
1.02
1.03
Пр 25
0
10
20
30
40
50
Number of programs sampled
Initial programs (np)
1.0
0.89
0.97
O.O.B.
O.O.B. 0.0.B.
Feedback-repaht repairs (nn)
0.8
Mean pass rate
0.90
0.96
1.03
O.0.B. 0.0.B.
0.6
=
Dry
0.4
fr 3
3
0.92
0.97
1.03
1.07
0.0.B.
fr 5
0.2
0p 10
Pry
1
0.96
0.99
1.02
1.04
1.08
Pp= 25
0.0
20
30
40
2
10
25
Number of programs sampled
Initial programs (np)
1.0
=p=1
Dr =
10
0.85
0.99
O.O.B.
O.O.B. 0.0.B
0.8
np
Feedback repairs (ne)
Meon pass rate
10 Pp
ne 10
5
0.89
1.01
1.12
O.O.B. 0.0.B.
np 25
0.4
3
1.01
1.11
0.2
1.10
1.11
1.13
0.0
0
10
20
30
40
50
Number of programs sampled
Initial programs (np)
Figure 15: GPT-4 results from Figure 3 (Section 4.1) per APPS difficulty (row), from top to bottom: roductory interview, and competition.
22

---

Published as a conference paper at ICLR 2024
1.0
0.8
Mean pass rate
0.6
0.4
0.2
20
30
40
50
Number of programs sampled
1.0
0.8
Mean pass rate
0.6
0.4
0.2
10
20
30
40
50
Number of programs sampled
1.0
0.8
Mean pass rate
0.6
0.4
0.2
10
20
30
40
50
Number of programs sampled
Figure 16: Results from Figure 5b (Section 4.2) per APPS difficulty (row), from top bottom: introductory, interview, and competition
23

---

Published as a conference paper at ICLR 2024
D HUMAN EXPERIMENT: DETAILS AND STUDY INSTRUCTIONS
We recruit 16 participants, consisting of 15 graduate students and professional machine learning engineer. Participants were told to spend approximately one hour on the study overall, and were compensated with a $15 gift card.
Participants.
Data collection. We first sample 20 tasks [ from the APPS test set; to make the data collection process less time-consuming for the participants of the study, we skew the distribution towards casier tasks (14 introductory: 3 interview; 3 competition). For each task 4hs we then sample two failing GPT-4 completions Pt.1. Pi,2, making for total of 20 2 40 programs to refine. Each participant provided with five different base programs based on their level of experience with Python and competitive programming. Programs are taken from distinct tasks; participants are never showed two different programs belonging to the same task. Participants are then asked to explain, in their own words, what the program is doing wrong. To reduce the cognitive load for participants, each program Pi.j is accompanied by the error message Cs,j and two feedback strings ft.j. sampled from GPT-4. We obtain these feedback strings by randomly sampling from the feedback- repairr pairs used in the previous experiments and removing the code block. Note that each of the 40 programs will be shown two different participants, to reduce variance caused by participants skill levels and writing style. This human data collection was approved by our Institutional Review Board (IRB) and carried out exclusively through an online survey.
Instructions. Participants were given a slide deck with instructions. The following ten images show the instructions, which include example of task shown participant:
Tasks
Your Answer
Gatup
seicty replete what dring
houphh
one bey whut de vou of Br
Tosk: Detug five incarret Py toon progrens
beral
per come revne can and no tos to acout the mour an recomssone
Oan sonn rxporeto wetrote
ei' you unmeer enrame Dooae tersentiinn andeeded No other datass beloy solleied
cas qually Buiig hert
motslts
pan papar anyton
Problem Specification
Example
Fastorae divatte
Regll ceriiiiyy
24

---

Published as a conference paper at ICLR 2024
2 Incorrect Program
3. Error Message
peggass bip cam
lapal trrugh
pegrreh parlic
4. Model Explanations
Answer Form
hel out coder the Aihosino
RERRATEDD FORANCAPIITY
These oppionation
ende eddi onler deattec toca
Study Tips
FAQ
grob your
Nr you lollingig data as wobstt??
our you the first argaama have loglt emens rot sepple
stall Only anavere
point study?
invedgpaa
models code lnebbaok
The rains difcatty Feel hea ricort your ses meseuring quicyt
Aro you ovaluaalin how useful the model explanations wore to ma?
25

---

Published as a conference paper at ICLR 2024
E HUMAN EXPERIMENT (QUANTITATIVE ANALYSIS): RESULTS PER TASK
In the table below, wc give a complete breakdown of the quantitative results presented in Section 4.3. Note that each program is associated with four different pieces of feedback: two sampled from GPT-4, and two given by our human participants. Each cell is the number of repair candidates (out of 25) that passed all the unit tests. See Section 4.3 for details, well as Appendix D for the instructions given to participants.
Task
Difficulty
Program
GPT-4 #1
GPT-4 #2
Human
Human #2
A
10
10
0
2106
interview
B
20
16
24
2673
interview
B
3
25
25
25
A
0
0
0
2923
interview
B
0
0
0
A
0
0
0
3070
competition
B
0
A
2
6
10
25
3286
competition
B
0
4
0
3754
competition
B
0
0
0
A
25
25
25
24
4182
introductory
B
25
0
25
25
A
25
3
24
23
4195
introductory
B
23
25
25
A
4
0
0
4281
introductory
B
0
A
25
0
4333
introductory
B
23
24
24
25
0
25
4347
introductory
B
0
25
25
A
25
25
25
25
4426
introductory
B
25
25
25
25
A
0
0
0
4450
introductory
B
24
22
24
A
0
4507
introductory
B
0
0
A
15
21
16
4514
introductory
B
0
25
0
A
25
0
25
4704
introductory
B
25
25
24
23
A
25
25
25
25
4741
introductory
B
25
25
25
25
A
17
25
4855
introductory
B
2
23
0
4873
introductory
B
0
0
18
A
2
25
4952
introductory
B
8
24
21
26

---

Published as a conference paper at ICLR 2024
F HUMAN EXPERIMENT (QUALITATIVE ANALYSIS): EXAMPLES
In this appendix, we give examples of each category used to classify the responses in the qualitative analysis of Section 4.3. For each category, we give example from the human participants and one from GPT-4 (when possible). Each example contains: the textual specification the task; the incorrect program; the error message (in the format was shown to participants); the feedback in question; and a short comment motivating its classification.
F.1 CATEGORY (A): OBVIOUSLY INCORRECT FEEDBACK
F.1.1 GPT-4: EXAMPLE
#+ SPECIFICATION
hotly
proposel sxap stars
elemettary
high school st controowsiial new dreaa uode proposal that bans athletle clothes in ached,, and propoza. to raise real-estate taxes to pay ter new rootball practice taci litility, and the list and is hours after the pol la have closed and xinrer has to energe!
Ir cheir desperation, the election otticials burn to you and ask you to write program to count tae vote!
-..--Ingut--e--
input consiste single case, whica
inpot contairs name of dandidate for whon vote was cast. A neme may consist miltiple words, segerated by spaces. Moras contain letters or typtens, nut no other punctuation characters. There sill be at least ends with single line containing the charactere This Line
votes cast,
the list, list con up to votes.
counted.
-..--Ctpput---
-Cotptt
candidate estained simpic absolute safority
Ithat simple majority, putpet:
cansidate), output name tais condidate! no cansisate
*Runotf!" rdon't forget to include the exclenstion nack!)
.ne--Examples -Examples--e-
Bampla Input:
Pcnny Fronklin
Marsi Gzahan
Joseph
Connie Froggate
Zenny Franklin
Connio Froggate
Stangez
Connie Froquate
3arbara Skinner
Barbara Skinner
Sample Output:
Connie Froguatt
INCORGECT
fzon collections import defacl:diet
deSsulidict candidate Input ()
while candidato 1- *e.*";
(cdes(candddace] 1=
candidate ()
cotal = sun valua.. nax_votes maxtootss.v
winner
name,
voses.itens
if ien (xinner)) sn 1 and voxevvtes vetel // 2: print (aimner [01)
else:
peint ")
#:8 ERBOR NESSAG3
swwTngut-es
Zenny Psanklin
Mar:i Gzahan
Conniw Froggatt
Connio Froquatt
Denny Pranklin
Comniw FroguaLL
Stanger
Connie Froggatt
Sarbara Skinnes
Barbars Skinner
*esPsogran Oulpui=ew
Runofft
seeRegected eue
Connie Froggatt
27

---

Published as a conference paper at ICLR 2024
The consise explaration of tha issue is: The cods checks iE the maximun vetea are strictly creater than haie oz the total votes to deternine if shore's 's mejority, Revover, it should check 12 the naximun votes than equal
total
Hff COMMENT
The issue with the code is not that the *y* should have been beet, but rathez that it not alloving for simple najorizy.
F.1.2 HUMAN: EXAMPLE
++ SPECTEICATION
Scuse and Raczoca have heap or II cards. The i-ta corr from the top has the integer A_1 writton cn Ae. these carde.
They
top heep,
will
remsiniag cards.
Sere, both Snuke and Raccoon have to take at least one card. let the the integers Snoke's cards and Raccoon's cards on and respectively. minimize Ix-V1.
They
Find the ninimom pessible value at Ix-yli.
www--Consccoattt---..
--10(99 Vieq a_t Vieq 10919]
a_1 an integer.
Input-
Irput 15 given from Standard Input in the tolloving tornat: N
aL(0)
wew.--utputt-......
Print the anzwer.
-..p-a Input 2 4
-Smpii Output-
Snuke
cemaining
x--|=-1,
mininun possible value.
#*# INCOBRECT PROGRAM
maint):
linput f)]
11st (nnp(in, ingut (p)).L(1))
Snuke_sun
Raccuon_aan sun(s)
min_diir ans (Saccoon__un))
cang=(a-1):
Snuke_sun a[i]
Racesor sun alil
diff abs (Bnake_sun Raccoon_aun)
min_Girt nin inin_dist, dicr)
print (nin_dicr)
main()
#d8 FRBOR MRSSAGR
Inguneeee
-10
saogang Oulpui=en
weweopeetedd put
#4# PEROBACK
seaull "nin_dirt" "ala (Bausoon_ cuzceapunding tie takes alll the cards, however this incorsect becruse of the constraint *Snake and Reccoon have take at leext one dard", Tosteed just setting "minudi to negative infinity initially would fix it.
init
hunan part pant was on the right track, but note that aett ing "ninudier" to inegatives infinity would cause it Lo neves be updated inaide of the for loup, and so the program would in lees always seluen nigstiive infinity.
F.2 CATEGORY (B): FEEDBACK EXPLICITLY SUGGESTS A MINOR CHANGE TO THE CODE
F.2.1 GPT-4: EXAMPLE
28

---

Published as a conference paper at ICLR 2024
6i6 SPECIFICATION
square sy-plare.
(x_4,y_4) counter clockwise order.
(Assune that the positive leaass pointa right, and the positive yrexis pointa up.) Takahashi renenbera ix 1x.1.--1) and (,22,2 but has forgot 1x.3,y and Ix (x.4ry_41, 41,
22 integez values.
x_3,y_3, 3, y.4..
shown that R_3.y_3, uniquely waist
Comalzeing
DX 1922, (8-21 Vleq 100 bully.1 (x2,2_2)
All values in inpal ase integeca.
Teput Iepul given Trom Standard Tapus the following Cormat:
x.1 1 x.2
Print x x.3.y 3 and y a5 integers, in this order.
Sem--Sample Input-rres
0.0.0.1
-..--Semple Outpat-nee -1
10,0), 10,1), (-1, (-1,0) is the tour vertices of a squaze in counter-clocoviee order.
Note that (r. k. 3...3y550-- (3-111111 10, 0x. (8.4, 4.y_4) ()-44)111,00) in not acceptes, as the vert ices are in clockwisg se order.
xl, yl, x2, y? naptint, ioput 11 Veee 151
21 X1 (92 yl) 31 0X2 XL)
pxint yo, Xd, yt1
#1# ERSOR MESSNC
>>-orogoo Output
1
eeeBxpecctd Outputeee
#f# FRBIBACK
issue codo
docsn't consider counter-clockvis. vertizes, To csiculatione.
exop the signs at (22 yl)
#*# COMMENT
Sxapping listed is concrete suggestion remain samo, vo
Since the overall structure minor change.
F.2.2 HUMAN: EXAMPLE
#ff SPECIFIGNTSCN
Given stzing S sepcesent ing the day of the week Loday. for Hoaday, Tuasday, Hadnesday, Priday, and Aftes how many daya la the next Sunday [oonoroww os lates)?
is MON, IUE, NED, 2HU,
Saturday, respertively.
Consl:
SON, MON, TUE, MEG, TEU, FRI, SAT.
Inpul.
given from Standard Inpus the following fornat:
Output
Drint
days beforw Sunday.
Sample Iaput
SAT
Sampla Output
Saturday today,
Sunday.
$26 INCORRECT PROGRAM
Input ()
I*SIN"
TUR!,
psint (17
days. indea (B)) 7)
#46 ERDOR MRSSAG3
SUN
29

---

Published as a conference paper at ICLR 2024
Poogogn Output
Exxpested Outpat
#+ FREOBNCE
The logic is incorrect. There abould be no nodulo operacor.
CONSENT
arqqescing that the nedulo oparatoz should be zenoved is a zelatively amall, but explicis, change.
F3 CATEGORY (C): FEEDBACK EXPLICITLY SUGGESTS A LARGE CHANGE TO THE CODE
F.3.1 GPT-4: EXAMPLE
#48 EPRCIFICATION
Takahashi i5 about to assemble a character sigure, consisting at parts called Part 1, Part 2, Part N and N-1 connect components, Parts ingu shable, but connecting has d_i hoics, called inserted, These heles the parte are distingui shable.
1, 2, auy Hole
connecting conponent can
Kach connecting corgonent will be inserted into two soles in different parts, ponnect ing these two parts, Te is impossibla insert multiple connecting compancats into character tigure the components are used connect parts.
complete
tolloxing properties:
Consider gragh with N vertiess corresponding the parts and t-1 undirected adeas correspending to the connecting
vertices connected ways and nake fiqure complete considezed sans the following satistied:
Then, this
connected.
every pair of toues, A uses a connecting component to connect thase holes 12 and only it uses one to connect then. number
Find
make
complete.
medclo
input
1iag Vtines
Vieg d_1 996264353
Input zcom Standard Inpuz zollowing cormat:
d_1 4.2 Aodots d_N
Output--.--
Print the anszer.
-e---Sample
-ee--Samplo Output
0cc way to nake the riqure complate is to connect Eole in Pazt and Bole 3 in Port and then connect Hols Part Bole
*ff INCOBRECT PROGRAM
del maint):
(map (int, ingut() oppst (pputt).ppii(())
M5D 998244353
[1] (8+2) for in zange (1, H+1]:
[aoi[i] fant
M00
For ki sange (1) ans dIRi] Iact]kk) & MOD fact (a-tt18-k1111 MDD
and ans CN- MDD
main()
Engut
..w Rxpected ed Outpptttoe
PRRGBACK
30

---

possiblc number or accupicd houses attor moves are Published as a conference paper at ICLR 2024
following conciae explanat issue: above calculates tozal number ways connect the parts, but it does not ensure that each cennecting camponea= is used only enee. This reselts in overcounting the nunber ways to nake the riqure complete. To 2ix the iszue, va need that connecting comporant is
Hf COMMENT
The feedback suggeata changing the code so that each connecting component is used exactly once, which does ze taken
Changing oude satisfy this part the specifion. ion awena to require najus restile.
F.3.2 HUMAN: EXAMPLE
SPECIFICATION 0h, New Year, The time to gether all your friends and reflect on the heartwerming events of the past year.
9cB triends Live in city which 9i5-th en integer coordinace
line, The 519-th triend Lives house with coordinate 5x_1 18, $x atay at 9x,18, Rach Friend allowed to move no more than once.
represented
Vor all friends 81 Vie x Mee hal halids, however, Sns, respestively).
houses with coorcinates 205 and Sna1g Hi
thoir houses
Vor exangle, let the initial posision be 3x [1, 2, 4, 418. The Finel ones then dan be 911, 3, 3, 418, $10, number occupted heuses is number
2, 315, 912, positions among 512, 1, ores,
So all criends caocse the morrss they want to pertorn, Atter that the number of occupies heuses is celeclated. maximum number occupled houses there be?
What the marimum
Ingot
singie integer Mie Me Vodot 191561
The sccond line contains 5n5 intogers SXL1, X_2, Adote, xunf (91 We X_1 Me no) the cocrdinates of the heuses
txo integees pertormed.
Input
1 2 Octpet
Irput
41888445
Octput
3 8
Input
Output
example frienda
Eviend S1_1+16, Frlend $29 staya
heuse $x_25, fxiend 938 coas RXC_-11 and feiend gees 6x_4-19. $12, 1, 316, 812, 3, 218 ue $12, 2, 4, 4]8 are also all valld cptioon Lu obiain $25 ocrapied houses.
bocupied friends go S(1,
exanple.
#*8 INCORRECT PROGRAM
From col llleti Imgort Counter
maint):
= int rinput 133
list (map (int, inguL (13)L(1)
ent Counterixt [x]
nax_houses O for range
if [pos]: houses
pocupled [3] (n for pos
if not oddugied[pos
31

---

Published as a conference paper at ICLR 2024
ppo
clir not sccupied[poss ezcupicd.pos)
elif secupied[pes escupled:pos
nin_houses sunfeccupied)
print Jiase,,, nax_hauses)
main()
ERROR NESSAGE
Input
saggan Output
eepexptted Outpateee
#!8 FREDBACK
The bode calculatea both the mininum and neximum count of homse lycorrenlly. The cocreat solut ion requires search daternine sexting the list occupied houses using sliding window to prosess indapendene segments the nundows line.
condact
nininun. This further optinized
The Tewdback suggeals conduct ing Lxew search Lo desecnine the max and min, as as way ogt inise this search.
These const itube najor changes the
is current written.
F.4 CATEGORY (D): FEEDBACK CONTAINS BLOCKS OF PSEUDOCODE OR PYTHON CODE
F.4.1
GPT-4: EXAMPLE
GPT-4's feedback never contains blocks of Python per our experiment design, in which the code block stripped (Section 4.3). We find that GPT-4 never utilizes blocks of pseudocode, either.
F.4.2 HUMAN: EXAMPLE
ET SPECIFICNISCN
Giver string S representing the day of the wook today. MOR, THE, Saturday, respertively.
foz Sunday, Hoaday, Tuasday, Hadnesday, Priday, and
Atter hov many days is the next Sunday Itonerrow or 1stes)?
ww---Consttiitt
SUN, MON, TUE, MEG, TEU, FRI, BAT.
Input
given Standard Iapus the following Cornat:
Print the
before the Sundav.
Sample Iaput-
aAI
Sample Outpat--aet
Saturday today, and Lonorzow will Bunday.
$26 INCORRECT PROGRAM
day Input
days I*SIN",
STUR!,
days. (day)
result 17 indox)
grint (result)
ERROR MESSAGE
Ergut
SUM
Program Output
- Expwcled Outpat=mm
FREDBACK
beFore returning the resulty by xe should just cbeck to see if result =-0.
just: need to addds
32

---

Published as a conference paper at ICLR 2024
#1 COMMERT
The toedback conteine klock or Pythcn code tazt the model can cimply copy in order to rix the progran.
F.5 CATEGORY (E): FEEDBACK WHICH EXPRESSES UNCERTAINTY
F.5.1 GPT-4: EXAMPLE
In the 20 2 80 examples we study, we do not find that GPT-4 ever expresses uncertainty.
F.5.2 HUMAN: EXAMPLE
*+ SPECIFICATION
kingdons
emperor intervened bring an to confll territary question comprises an 9N3 by 3N3 rectangular grid, the emperor's inxlatence ence, the two kings have withdraw their troope until no two opposing troops are in adjacent squares the nap [adjacent being heri rontal vertioal diagonal is not consideres).
long
The emperor propoces to designate certain equares the map as neutral territory, Westher king will be allow to nove rropps into these aquares, and the emperen's own Forces will patrol then to be sure observe
that both
rules.
The experor frugal does nos went to comitt more soldiers to this effort than cheolutely necessaty. life generals have narked each square of the map with the nunter of solciers zequired to secure that equare. choose whict of those
What remaine
should patrolied.
Write a progran to determine the niniman number or soldiers that the emperor will need to be deploy to guarantee that the croops at are kingdon cannot move, in one or more steps, into squares accupled second kingdom (moving harizontally vartically) without caccuntering the ampores's soldiere.
Input
begins line containing #29 integere, Fx5 She, denoting wheze 61 Vieg Ny (leq 405.
and
This is tollexed by and lines, Each Line contains txt characters, lett justicied. Thase characters will be 18', desienating positior. held cursently unsecupied position that can secured by the
B, sincle
digit, dasignating exampic,
number of soldiers. '2" would indicate that two soldiers must se deployed to that square to secure it against paseage ot other trosps. A np" indicates terrain that is Lamassible the expezoz need not commit soldiers there becsuse the kingdon
through
No 'B' xill be adfacent, horizontally or verticslly, to any 181.
There will least
and
-..--Cutput----.
2rint singla line containing inceqez denoting the niniman number ot soldiers thaz the emperoz nuat depley
is open betweon '2' positicn position, conbination of husisonia os oectical noves.
Examples
Bample Iapul:
A11111MA
AA791IIA
111861:1
11896111
1188611B
Bample Output:
#*8 INCORRECT PROGRAM
def meint):
geid [input
sarge(hi]
del bisix, 21:
dist [[flost [Ninft)]
range
dist[x] [y]
(G,, y)3 for
1, 11, 10, ny if 0 <e nr h and <= ny x and (ixt [nyl -- Flost ("in
1f gridlndd [ny]
(nxl (ny] dyl
qpepeed (inx, ny)1
12 nge oe ye gdid[e]] [uy] oo ngn;
dist Iny] dist belly) set igrid (ngl) qpeppend ny)
reluen dist
Slows (MinEt)
for catg for j. range (x)
33

---

Published as a conference paper at ICLR 2024
grid[a](11 "AT
and Bin (ans, mi for row in BCS (2) 3, *B") roz in xeM))
print (and)
m2in
6:# ERROR MESSAG3
imminputen
A11:11AA
A6:7B111A
111861:1
11886111
1128611B
=.eP:ogean Outpui
BeeBteetted Out
PREDBACK
appeaza finding in messured in the number of soldiars raquired to secure the intarmdiata pesiticas. This isa't really addrwssinng the ion all, beceuse necessazily block difforent, lsnear path. probably connected componetts, une eath for A and B modes.
"distarce" betveen position
position, whaze
the gosit iura along sume path from pacticular reaching each other from those pasitioas graph into
soma
min-cut algorithn
Towdbbcc espresses unceccainty Lvo aspecis. Pirst, the uses Indicates tha they are not completely cartain in their undarstarding of the codets cuzzent behavious by using the phrase "appoars te be* Then, they express moeeiainty in their suggest for what so do insteed, that sprobablys
nia-eat eat algosithn.
34

---

Published as a conference paper at ICLR 2024
G PROMPTS
In this appendix, we describe the prompting structure used for our experiments. To avoid repetition, we list the prompts used for the GPT models here; for CodeLlama, use identical prompts except that we replace Ppython.... " block delimiters with [PYTHON]... [/PYTHON] delimiters, since we find that this significantly improves code extraction reliability.
G.1 APPS
For initial code generation (the first sample from Mr), we use different prompts for the two types of tasks in APPS: call-based tasks, in which the desired program should take the input as a parameter to function and return the output in the function's 's return statement; and stdio-based tasks, in which inputs should be read from stdin and outputs should be written to stdout. These prompts are shown in Listing and 2, respectively. The example tasks and programs were taken from APPS' training set.
For feedback samples (i.c., samples from Mp), we use the prompt in Listing 3. This prompt contains an example in which the user provides the textual specification, the incorrect program and the error message, and the assistant generates feedback. Similarly, for repair samples (i.e., samples from Mp which follow Mp) use the prompt in Listing 4, in which the also supplies the feedback, and the assistant returns only the fixed version the program. Finally, for joint feedback-repait samples (i.e., when sampling (f,r) ~ M), we use the prompt in Listing 5. This prompt combines the prompts from Listing 3 and into one prompt, in which the assistant returns both the feedback and the fixed program. all of these prompts, the specification used was taken from APPS' training set, while the programs and the feedback were constructed manually.
Listing 1: Code generation prompt for call-based tasks.
syst.mm
You are an expert Python programmer You will ze giver a question iproblen epecitication and will generate a ccrract Pyzhan preoram that matches the spacicication and pazses all tasts. You will NCT reture within code delimiters, example: "python
snything
the progran. Fut your sixed
YOUR CODE
"USCE" QUESTION
palindrone word, phrosa, number, or other sequenes of characters which reeds came backward ICENord. Examples of numerical palindromas ara: "2932, 110011, 54322345"
number ""aun"", write function which contained within each namber. The arsay ahould susted in ascending urder and any deplicatss abould be senoved.
numerical paliadroncs
this single digit nurbera <*00 considered valld numerical palindrones.
which and with
'01C" and '02")
If "nun" contains valid palindemme,, seturn ""Ho palindronws found"". otherwise, "Hee valid" the
interer is less 10"
# Examples
palindrame (1221)
8> [22, 1221]
palindrone (34322122)
=> [22, 212, 343, 22122]
palindrone (1001331)
[33, 1001, 1331]
palindrone (1294)
palindremes found"
galindzone (11211
-> "Hol valid*
+*# Other Kata in this Seriwx:
Munezical
Palindrone 81
Ruserica]
Pelinde +1.5
Runerical
Pulindeuss #2
Murerical
indrome +3.5
Munerical Pal iodiuuu #4
Norerical Pal indrome
#5
Dse Call.
"python
def ix_pel 06)
seturn 8111-1]
35

---

Published as a conference paper at ICLR 2024
palindzone tnan) 5 12 not Lsinstance (Oun,, int1 oe nun retuzn 'Not valid"
pais oct.))
for 1, ch in cnuncrate
if ch wa "D*;
continue
j cange (2
Senta) 13
tast = #[4:1] 15 (CoSE)))
gela.ada
securn sostediint for paia) or 'No palindzones Iound!
Listing 2: Code generation prompt for stdio-based tasks.
giver quest iprobler specifioat perrect Python program that mat chea speai con and passes all texts. You HOT return
expert hython prognemeet
generate
anything except tor the progran. Fut your tixed program within code delimitere, for example: ""python
YOUR CODK HERF.
** OURSTION
Polycarp 960 different binary are binary: *0001", "12", no* *0011300",
word colled binary containe only characters '0' and "1' For
exemple, these
Polycarp wants to etter his eet at fn8 binary worde zo play game "vorde", In this game, players name words istarting fron the second) first word can be any. For exemple, those sequenes of words car named daring she ganz: "0101", "1" *10", *00", "00001".
each
with last character at
word,
Word is operation zeversing reversal becomes *1120",
characters. exampie, word "0111" "11010" atter the reversal becomes "01011",
Probably, Polycarp has such set of words that there is to put then in the order corsespondenz to the rules. In situction, he
that: the tinal words still coacaina difforent words (i.e. all vorda are uniqus) chere way put words of the tinal set of xords in the order so that the tinal sequzaee of 9as words is csasistent with the game rules.
Polycarp vants to reverse
number words. Please, help hin.
The first 1ine of the ingut contains ane integer St5 191 Vin Via 105 the number of cast cases in the input. Then 926 best cases fellow.
first of tost case contains intocar [91 the Polycarp's set, Next Saš liner contain those werdo. All of 9ns words axen't eapty and contains only characters 10" and "1", The sun of word lengtha doesn't exceed $0kedot10 All words aze different.
2\cdec10"55)
number words
Guaranieed, that the sun
Leat casea that the sun of word lenoths for all test cases in the Input doesn't exceed 54kedot10*65.
ingut dowan' exceed 32%odo_100E8, Also, guasanLeed
2rint anaver foz all of BLS Leat casea in the order tlay shore answer
case, print -1. Othervise, the 190 \1e & 'ie n:1 the nininal autber of worda in the set which abould be zeversed. The awcond line of the output should contain S&S distinct incaders the indexes of the words in the sat which should be Mords
output should contein sxt
from 815 fer you can print an enpty line). there nany answers you can print any
they appeas. 92-25 you skip this line
Example
Input
10000
3C11
01111
310
1C1
3
30000
3c001
301
0001
30001
Octput
36

---

Published as a conference paper at ICLR 2024
1.2
Use Standard Inpet Format troad "inpat write resuits "prine (1")
**semaxsial bal
"'python
for in range Hint (input (1312
Sil.tippu
20
o4a []
202 []
2009 aet.)
for in zange (a1)
impui
appendd
R[D]
X[-1]
xosa [kss.adddkk
aps.append +
k[O]
KJ 101
ozsa.add [K]
exe.append 13
kÇO]
R1-1] 10"2
2x 1-
#1 alaws
za and no and not and not 102
pa inl (-1)
contt noe
print 1xx .2)) // 2)
ozl
whil+ needs
nass [i] 1. toe [oelll 10 211
ane trpeed
need -
point
else:
print I (on
2n) // 2)
zot
shile need:
mass lors [1] (naes 10 drt
not in zo50:
and ans. appendiost))))
need -
point
Listing 3: Feedback-only prompt.
weweeaysLeneene
aro holoful progranning assistant
export Python progsammer. You helping program, The user has weillen some code, but it hea some ecrora and is not peksing the Leats. You will help the usez by giving concise (at noat 2-3 santences) textual explanation of what is wrong with the generate any
debug
fix code thenselves.
676 Q0ES7ICN
A indrote is word, ghrase, mimber, other Exaplees of
of charact which reads the backward indeunss '2332, 110011, 54322345'
For given number "or wrice a funct ion which recurns an array of all the munericel palindrones contained within each number. The ahould be sorted in ascending order and doplicates abould
In chis hala, single digit murbwea and numbers which stast or and with aucos (auch as "026" and '03") are sANDTes considered valid nunerical palindrones.
"nan" contains valid palindromes, sutarn "MHo palindrones Found"".
Otherwise, return "Hooe veligns if the input is not ar integer or is less than '0"
Rxamples
37

---

Published as a conference paper at ICLR 2024
palirdrome (1221)
-> [22, 1221]
palindrene (34322122)
22122]
palindrome (1001331)
[33, 1001, 1331]
palindrome (1294)
"No pal:ndremes tound"
palindrome (*1221")
"Mot vaild"
Otber ia this iss:
Memerical
Folindrone F1
Munerical Palindrune #1.5
Munerical Palindrona
Runecical Peldrde +3
Munesical Palindrons #3.5
Minerical Falindrone F4
Munerical
Call-Based
#*g INCORRECT PYTHCN COD2
Lpython
def is_pal securn ac::-1]
def palindrona inan)
reluen 'Not. valid!
anun)
pals x)
canoe (1
lonta) 23
text s[ief]
12 pa (Leat):
pels.addieesst)
reburn sorted lint for pals) or 'No palindrones Found"
The code above is krong and contains
Given inget "1001331"
"(10', 133', (1001',
expected outgut
1200''
wew.mastial Lanteee
The foll lowing is conci se espl wnat ion of the issue: The problen en description for states that numbers which start and with coee (auch 010" and 100") Non considered valid numerical gelindrones, Hovecer, the into account theretere returne be fixed by checking the first or last charecter ia "g" before adding the string to the set of palindsenes.
palindrore. This
Listing 4: Repair-only prompt.
holsful prograrning assistant progrem to soive a problen. The ucer has weitton cone code, but it has some ertors and is not parsing the tests. The user has spent some :ine dezupging the preeran and will provids you with conciae the code. You will program. Put your fixed progran within code delini. for example: ""pythoa
expert Python progsemerr You helping user
explanstion
fixed
YOUR CODE HERE
QUESTION
A palindrome is word, phrose, number, or other sequeaee of characters which resds the same backward zs Forward. Examples of nuterical ical palindrows ases "2332, 110011, 54322345'
given number ""per"", write function contained within each namber. The arsay ahould be sosted in ascending order and any duplicates abould be zenoved.
recurns auray
palindroncs
kata, single digit numbera numbers which stast or and zeros (such "010" 103") *o coasidered valid namerical palindranes.
"nan" contains valid palindrenes, othezwise, retorn ""Hot valio" input an integez is 70"
palindrones found"".
Rxarpl
pelindrone (1221)
=> [22, 1221]
palindrons (34322122)
22122]
palindrone (1001331)
[33, 1001, 1331]
pal Lndrone (1294)
=x "No pal indooees found
palindzone (1221*)
"Mol valid*
Other in this Series:
Nurerics] Pal indrone #1
Runesical Pal indrows #1.5
Mumerical Palinacone
Nomerical indrore #3
38

---

Published as a conference paper at ICLR 2024
Munezical Palindrone $3.5
Minerical Palindrone #4
Numerical Folindrone F5
Call-Based
INCORRECT PYTECM coD2 "python
def 1s_pel (9):
0(1:-1]
def palind:onatnnn Inan)
1e not icinstence (our, int! or nun
'Not valid'
stunn
pais sct.))
Fax i, ch in enureratt(s)) (a)
Lents)
teat *[i=f]
10 isppal (ceat):
acls.add
palindrones found'
The coda above is wrong and contains a bug, Given Lapet "1001331" the cutput van "110", "237, "1001',
"1331"1" but the expecied qutpet *[1331, 1200'", '1331']*, concise oz End with zezos (such vode above dows not Lake Lhis inco aucounl and Uheeefore reluens "00" ax vel pelindronw. This can be Fixed shecking if the rizst last charactoz ln "Q" before adding the string the set of palindranes.
following
the issJe: "00") BO7 considezed
problen deseription
which nunerical salindromes. Howaver,
that
www.aaxt www.aaciilaanteeee.
6*8 FIXED PYTHON CODB
"ppython del
securn ac::-1]
def pall indcome (nan)
intl
return 'Not valid"
aic sLL: Inun)
s)
cont inue
for j rangell (1
Senta) 1)
s(iid]
(test):
adl (t (Leat)
sortedlint for
'No palindrones tound'
Listing 5; Joint feedback-repair prompt.
You are helsful programning assistant and expert Python progssamer. You are helping user write weitton
Lesia, You help aser fizat giving voncise must sentencesl texiual explanal of what is wrong with the code. After you have pointed out what is wrond with the coda, you will then gencrate a rixed version of the progzon. Fut your zixed progran within code delimitore, for example: ** "'pyihhn
YOUR CODE
QUESTION
palindrome word, phrosa, number, oF other sequenee characters
backsard
Forward. Exarples of munes ical paliadcome ares '2332, 110011, 54322345'
given ""ou""" wrise funet which contained each number. arsay ahould be zenoved.
acray nunericel palindzones ascending ordar any duplicates
this single digit nunbera <*DDIe considered valld nunerical palindrones.
which and with
"01C" and '02")
If "cun" contains valid palindrones, seturn ""Ho palindrones found"". velignt
integer
10"
Rxanples
gelindrone (1221)
-0 [22, 1221]
palindzons (34322122)
=> [22, 212, 343, 22122]
palindrome
[33, pelindroney found"
galindrone
palinds (1121*)
=0 "Hol. valid
#46 Other Kaba in thil Swriwws
indrore #1
39

---

Published as a conference paper at ICLR 2024
Munezical Palindrone $1.5
Minerical Palindrone F2
Munerical Folindrome F3
Munerical Palindrone 63.5
Mumezical Falindsone F4
Nemerical Faiindrone F5
s*e Use Call-Based Fornat
INCORRECT PYTECN CODE
""pyyhon
def 19_sel (9)
seturn mn a[:1-1]
palindzona inan)
12 not isinstance (our, intl or nun reluzn 'Not valid"
stzinun)
pally 1) Iox i, ch in (munerate
for 1 in rance (1
lenta)) 1)
ja sfiej]
owluen soctedtint for pala) or 'No palindrones Tound" The
is weond and contains bug. Lnpet "1001331" cutput "110%, 1237, "1001", "1331"1" "" but the espected outgut was *[1331, 12002', 133311)", www.maassttttttee...
Fall lowing ue and with swcow (auch es
probl descrigt for states that and "00") are Nor cunsidesed valid munecical indronss However, coda above does not take this inzo account and therefere returns "0o" as a valid palindrons. This can be fixed checking the first last character is "g" before adding the string the set of palindeanes.
exp'
which
code below
correct version of the codo above, wheze
has fixad:
#44 FIXED PYTHON nohr:
"yyloon
pal
rwturn #[20
def indrone (nun)
Ssinstanee
Oz
reluzn 'Not valid"
pala s)t.)
for cnunerste
if ch '0';
cont inue
in range Lente) 1)
is_pal (Leat):
adsteed))
sinted For pal or 'No pal pelindrones found"
G.2 HUMANEVAL
For HumanEval, since there are no training samples to use for in-context learning, we generate the initial programs in a zero-shot manner. For feedback. repair, and joint feedback-repair samples we use the prompts in Listings 6-8. These all include one-shot example specification, program, error message and explanation; these were manually crafted based on the specification and gold standard program for HumanEval/0. Although not strietly necessary, example when doing self-repair encouraged the model to use format which we could casily separate (and analyze) the feedback and the repaired program, rather than interleaving the two. Note that the form of the error message differs slightly from APPS, since we use a different execution engine for HumanEval.
find that using in-context
Listing 6: Feedback-onty prompt.
www.may'l MamamayLLeueeeee
You are holsful programning assistant. You are helping a usez weite a program to solve a problen, The user has written code, but it has errors and is not passing the texts. You will help the user concise cenerate any code, bacause the user vanta fix
giving
the ande, will expl ana of thenselves.
$16 INCORRECT coDe
** "'ython
def agprox_ elerents (nunbers: liss (flost], thresholds flost) bools
Cleck if in given List of nunbeaa, theze ase no two murber cloass te each other
than threshold d given.
apgrox_unique wel.eiieuneeeeeetttt.....ee w
3.01, unique que, ol (11-0, CG, 2-8, 3.0, 4.0, 5.0, 2.01, 0.31
True
so0 apprex und
Falaw
40

---

Published as a conference paper at ICLR 2024
for slan aurbers:
foz clem2 in numbezs:
distance abs telan c1em21
threshold:
soturn Faise
securn 7rue
The coda does not pasa the test cases, The azzar encoracared was: "falled: agSREt approx wniqesscceenntt
2.0, 3.0], 0.51 on True: Falss In reue"
following is conciae explanation of way
seat: code fails because also
compares sicments to thenseives. This, for any iaput, the answer will be Falsc. rais can bo tixed by adding bond ion to the innes loop that shecks that the indices ace not the same.
Listing 7: Repair-only prompt.
systcm
You helptul programming arsistant
programmer, You extors and not passing progeam soiva problen. the user weitton some code, but
write
the tests. The user has spent cono time dezupging the pregran and will provida you with ccacise textual aplanatioon of what wzong with the code. You will use this explanation to cenerate fixed within code delimiters, for example: """ythsaant
the BERE
rixed
#1f INCOBRECT CCOE
"python
aspzox__nique llmentts (nunbera: [isc[float), shreshold: float)
bool:
Chack if in given list of nunbers, there are no two aumbers cleser to pach other Lhan the thisshhldd given.
3.01, 0.5)
2>> app appeo Palse ...
2.8, 3.0, 4.0, 5.0, 2.01, 0.31
for clan in aurbers:
For wlen? numbers:
abs elen21
distance thrasbald:
reburn Palow
The coda does not pasa the Lest cases. The arror uncorncered wast "falled: assert approx_aniquse clenentssss 2.0, 3,0], 0.51 as True: Falss to True" Iolluwing is cuncise eaplanet
why code failed the comparea elenenta to thenseives. Than, for any input, the answez be False. This can by adding condition fon to the inner loop that checks that the indices are not the some.
cude faile because
FIXED
python
def appcos_un _ocou__nnippp_pp hegpiggppeeeeee wleents (marbwea: liss [flost),, sheeshold: flosk) >> boul:
nee Chack if in given list of nunbers, there are no two nunbere cleser each other given. 302 approx wnw.unnau (11.0, 2.0, 3.01, 0.5)
True
300 approx weciiiieeeeeeeeee (ement 2.8, 3.0, 4.0, 5.0, 2.0), 0.31
False
Fax ids, wlen in enurezate :
(numbers):
idx2;
distance = abstelen elenZ)
if distance threehold:
return True
Listing 8: Joint feedback-repair prompt.
hulpful progr anningg progeam solva problen. The usez wsitton seme code, but has excors not passing the teats. You will help the user by first giving ponnise lst most 2-3 sentencest textual expl ipllane ion with the cude. Aftws have pointed out what is wrong with the codey will then progran, your tixed progran han YOUR coee HEReAn""
expez Python pougseames.
helpang
of shat is
generate fixed
delimibers, exaeple:
+d INCORRECT COnE
** "python
appzox_unigue aniguueeei llmentt (nurbera: Lisa]]) ilaeshhlld: float) -> bool:
Check if in given list of munbers, there are ra two numbers closer to each other than the threshold given.
approx wtt.ppeeeeeeeeetll.ooo 2.0, 3,01v
Troe
41

---

Published as a conference paper at ICLR 2024
apcco_unicoo_iiennsss 2.8, 3.0, 4.2, 5.0, 2.0), False
clan
for clem2 in numbers:
distance abs (elan elem21
relura thrashold:
seturn True
does not pasa Lest cases. arzoz orcoLncared "falled: aSSREC aapro__niqes_ccemeeee (12.2, 2.0, J.CJ, 0.51 Iruc: Falso Truc"
following is concise explanation
code failed the compates elementx to themellves. Thin, for any input, the answer be Palse. This con by adding cond fon co the inner loop that checks that the indices ace not the ReaM.
code fails becausa
FIXED
python
def pppoox_niqquu_sseooo somennss Inmnbers: [ist[fl threshold: flost) +> bool:
ne* Cleck 1f in given List of numbers, there na two numbers clowes each other
threshoid
2.0, 3.01, 0.5)
True
292 4p 2.8, 3.0, 4.9, 5.0, 2.01, 0.31
Far idx, eien enurerate trumberrss idx2, wler2
[nunbers)
idx2:
distance = abs telen =len?)
if distance Lhowsholds
SMLuen 7auM
42

---

Published as a conference paper at ICLR 2024
H APPS TASKS USED FOR OUR EVALUATIONS
These tasks were randomly sampled from APPS' test set. To avoid distribution shift, we sampled according the relative frequency of difficulties in the full dataset. We report the resulting list of tasks to aid reproducibility.
Difficulty
Tasks
introductory
'4004' '4058",
"4100"
'4182",
"4281",
"4366"
'4450', '4465', '4484', "4498".
'4507', '4514', '4544', '4544",'4533',
'4586", '4610", '4662", "4663", "4667'. '4677'. '4681', '4704', '4716',
'4741', '4750', '4786', '4787' '4801", '4855'. '4862', '4864', '4870'
*4873", '4890', '4897', "897',"4952", '4966",
"4952',
"498""
interview
'0004". '0033',00056" '0073", '0074", '0089', '0091' 0022"'
'0131', '0139", '0162", '0183" '0186' '0191', '0205'
'0249", '0253', '0268", '027"", 00:7: "0300", '030""
044 '0413'
'0427", '0434", "046"", "0496""
"0111"
"0511", '033"" '0564',
'0637', '0676' '0704', '0676'.'0704" "551"" '0579",'99"" "059""
'0728", '0757', '0765'. '0794', '0804', '0805', '0811", '0829',
'0879", '0904', '0915", '0937', '0948" '0954', '0955, '0972' '0972'
'0985". "0989", '1018", "1019", "1033'. '1046" '1076", '1133', "1133' '1140"
$1141", '1145' '1146', '1149", '1168", '1185' '1221', '1232" '1256',
'1257" '1280', '1285' '1299", '1317'
'1347' '1380', '1392', '1393'.
'1418" '1444", '1448" '1458' '1489", '1517' '1533", '1573" '1635'.
'1653", '1668" '1672', "1721", '1736', '1748", '1756", '1759' '1755'.
'1777' '1825" '1850', '1863'. '1865", '1870' '1875', '1906', '1917'.
'1956', '1962" '1967', '1976', '192"',2024",, '2049", '2062', '2092' '2093'.
'2024",
'2097', '2106' '2172' 22176' '2203', "2231" '2246', '2264' '2266'
12255'''322, '2328", "2332', '2342', '2361', '2369', '2407', "408""
'208""
'2418", '445"", '2463' "2511", '251'" '2516'. '2535', "2585', 26623'
1829",'2642) 16651",26666 2668", '2673', '2698', '2701', '2709'.
'2735', '2742', "275"'.
'2787', '2802', '2832'. '2835'. '2844', '2858' '2885' 2897', "
2932', '2945', '2973', '2980'
competition
: 3302, 3007 "307' '3077' '308'',
3099","3117", 316
"3280 '3287',
'3286',
'3335' '3353', '3355' " "3375", '3388" '340'..
"3411", """"43""""4441,,,"""""""""""""", "345", '3470'
"3484', '3548', '3557',
'3605' '3609", '3634' '3635' '3671', '3699', '3709', '3754', '3769',
'3792', '379'', "380", '3819' '3823', '384'',
'3849', '3876', '3913", '3934', '3972', '3974'
43

---

Published as a conference paper at ICLR 2024
I MORE EXAMPLES OF FEEDBACK FROM GPT-4 AND OUR HUMAN PARTICIPANTS
In this appendix, we give five more examples from the study in Section 4.3, so that the keen reader may themself compare and contrast the feedback generated by GPT-4 and by our human participants. Each listing (6- 10) contains textual specification,
message, both examples of GPT-4 feedback which were shown to the participants assigned to this task, and the responses from the participants. For each piece of feedback, also list the number of resulting repaired programs which passed all the tests (c.f. the full results table in Appendix E).
incorrec program,
Listing 9: Task 2106, Program
#48 SPRCIFICATION
There cities
old Lives. These cities located straigat line, left right s.2, ... to c_tat us ing roads. There are in one way reade, the road goes from city (L(1) to eiiy C +.1 and is ELI kiloneters long.
Peykan wants
city
Peyxan travels
consumes
tuel during
Kach city f. (except for the lest city a (nll has supply of 1.14) liters of fuel which imediately transtors transters. times.
it Peytan can stay in city This retresaes instantly nany
Iritially (at time
the old Peyxan is city and liters tuel is transterred it's enpty zeykzn"s tank eapacity uniimited. zeykan can concinus
tank rron c_1's supply. its travel it its tank i5 emptied strictly between two cities.
Find the ninimen
needs reach city c_ia).
-.---Input----
first of input contains two space-saparated integora specitics the rumber of roads between citics which is equal to E
n, 1000). valce
line ccatains spaca-saparated contains spacc-sopareted integors ag3, 2_2, .o. aa(m) (1 AL141 1300).
d22,
G
1000) and following
In the only line of the output print a single interer the miniman time requized for The old Foykan to reach city clinl fron city e.l.
Examplos
Input
2
Outpet
Input
Output
$46 INCORRECT PROGRAM
mi, naotiat, inpet O)
da 1ist (nagtink, inpul -aptit F
1ist tint, Input -)-pitt 1
fuel 2010]
win_tine
For in renge(mi)
Cuel da [2]
Fuwll
Line +
fuel
Line
print (t iinl
mein()
44

---

Published as a conference paper at ICLR 2024
FTF ERSOR NESENCE
Ingut
3
Proggron
Exxested Outpatame
1ff GPT-6 FREDBACK
The issue with the code that is does not cons idez the Lime spent in the pursent city be becoz nov ing to the overastination the ninimun time requized. This cixed updating cursent ciay beforw celoulating the fael used account ling for the tine Lakea to Lravel to the next city.
city, saacicing
the Lime spens
Numbez of passing zepairs: 7/25
GPT-4 FREDBACK
The following la concise explanation of the issue: The code incorrectly calculates the waiting tine at each city. It should nininise the waiting Lime cunsider ing the naximun fuel supply that can be collected during waiting cime. code needa updated o2 passing zopairs: 10/25
for
4i FARTICIPANT FEEDBACK 41
program doean't
Fuel pazsing throughh nisies, only of refuel. Mlanever passoss Lhrougg. a city, it should add sn sali) Lo the fuel.
actuelly stay city
Numbez of passing repairs: 10/25
PARTICIPANC FEEDBACK
The is incoerect because doean's take into account fusl each city
innadiately
transferred after entering the city lexide for the fuel received at she First aityl. As result, this cvmzeal instwa the Line needed, because assumes that the old Peykan always has Lo wall Cur they given Number of pasaing repairs: 0/25
(besides
estyl.
Listing 10: Task 3070, Program B.
#f SPECIFICATION
frea-narket, ruthlessly capitalist world train enly thing Lacentives.
Train companies are incentivised with bonuses sor high throughout, successtul journeys, and custoner catisfaction. Conversely, the companies ara disinemativised from failure via nandatory refunds for 5305 minutes
customers
Bcing a ruthless capitalist yourself, you have decided to take adventage of this genarous delay compeasation provicion.
The sefund exarded provided the same route of scations as plarned), you would still ke unable to reach your sestination in strictly less time than 9305 minutes (oz 518005 secondst, of the time you would have arrived assuning your exactly
natter combination trains you taken (provided they followed
beoked journay
Azeed with your printeut of the day's delays, and the original timetable, you mast ask yourself only one question: what is the carliest time you can book train for from station 515, in order to carn this itstive awward?
Input--...
Coe Line containing Lao intageca: BXS (81 \le N. Vle 1009), the munber of stationa, and 9N8 (81 Vie R Mle
10"59), the number of scheduled
next 586 lincs each contain integers:
SXS, the starting ing stat fon ($1 Vie Vie N-16),
986 and $28 193 Vle \ie F 854009), , the planaed daparture and arrival tines in seconds, departure
(50
864008),
dusatiin
which
Liaes delayed.
stazions aze nusbered fron 515 to 538 in the order you will visit then. Each train goes betwson stations exS and $x1 It is possible Lu change belowen Lreing instant
-Utpptt-...
Coe Line containing onw integecs the start Lime of the maillest Leain Jousney you could book in arder Lo earn your compensation, or imposatble if such Jeurney is poasible.
Examplex Sample Input 1:
2
1800 so62
2000 9200
2900 9402 1460 Sample Oui.pul 1: 1800
Sample Toput 2: 2. 1 1800 3600 1800 1300 Sample Output 2: imposaible
45

---

Published as a conference paper at ICLR 2024
INCORRECT
dEf maint):
N, M maptint, inpet D e)
train_data (ia, (1.sp13c foz
sroin_dsta.sort lenbas x: (X[0), x(1))]
tines [1e9] IN times(1)
for X, 8, 2, 2. in train_dasa:
if times [X] <e S:
nin (X
12 timas[X] [X] >- 1800:
print (inL (Linws
elses
print (* impesxible*)
main()
#18 ERBOR NESSNCE
9000
2000 9200 1600 1 2200 9400 1400
*exProgren Output
Expected Outpateme
1800
GPT-4 FREDBACK
The isxum with the code is thas is princs "t3 Instwed of the warllest possible train start time. The code needs Lo the minimun stazt Line Crom the Leain Lrain_data list Chat has valid delay fos
compensation
Number of passing repairs: 3/25
#4# GPT-4 PRETBACK #2
value of (liex(1)") which anivial to 10" and naver updated. The corsect oulpuz should starting of the earliest Leain journey that zat aftissies the condicion, To fix this, we need to track the earliest possible start time and update it If valid train journey is found.
that directll
Number or paceing repaire:
#44 PARTICIPANT BEEDBACK
This code Also, (x(1l)" in place "xl". because we want nunerical order sasting nos steinn order austing. The goal in this problen in caleuiste sor each station what the earliett-arrivinn ls and what the earl lest iest-arriving delayed path the sta resulce
timesil] instielized to sorting line "irt
and returned
place 'x10]" likeiie
undelayed path to the station tace the diffrrence between the minutes (thon to return station 1. for this path, Siven tais, first the current code based around "tines" cen rensin this is the del aylayed time. llovever, the if ststement within the for-lnop sheald be 5 sinne the updated departure time also delayed. Then "undeleyed_ times' should tracked undelayed Loop when "uncelayed tinesttimes Pinally the if emmentteeent for time print ing can use the difference between the delayed and undelayed times and see it anyl) are x 1800, Finelly need to print the carliest arriving train for that path
see
tetel dolay
earliest departure tron
Number o2 pascing repairs:
#48 PARTICIPANT VEFOBACK 42
Bexever, Firet, instesd ot cheaking it you reaca your dest instin within 30 minutes of the expected time, it checks if you reach your dest instination in niautes. fix chis, the pregram would need to keep track of the time you'd rezea your destination it keeping delays, which currencly doesn't. Second, instesd printing times[l], it should delay printing
that the larger lesces
'timos[l]', xhich is
theze
Number of pacsing repsirs: 0/25
Listing 11: Task 4333, Program A.
6Sg SPECIFICATION
aquare
ay plane. coordinates
four vertices (,, 1e.S,y
(. counter clockwise order.
(aszune that the pesitive xeaxis pointa right, and the positive yeaxss pointa up.) Takahashi renenbers (x.ly_.1) and (x22y,2), but be has forgot 3.y3.. and 4x.4.1.11 y, A.d.
Given &. Integer values.
shown that x_3,y_3, and uniquely waist
Conal cuinta
1 21. 21,19.2 Vieg 100 b (x_2,2_2
All values in inpul are integeca.
Toput:
Ingut given From Standard Iapus folloving tornat: x y. x 2
Cotputtt---
S.y. 3,x. 3.x.
and y. integers, In this order.
46

---

Published as a conference paper at ICLR 2024
--r--Sarpi Iaput
3 0.0 0
Samplc Outpat
-1
(0,1), (-2, 11, (-1,0)
vertices of in counter-clociwies
Rote (1,3, (1, not accepled, as the vert are clocksise order.
fff INCOBRECT PROGRAM
nap tink, 131
x2 (92 yt)
(x2 xi)
yl (x2
print (X2, X4, ydI
ERBOR MESSAGE
3 0.0.C.
Pmopcogzan Outpur
1
wwpeepeeted Out
#4# GPT-4 PRETBACK
cluckwiss order the vertiows, To fix this, No swap the aigns (y2 y1) and xi) calculationa.
cude thai duesn'S consides
+ Number of passing repaira: 25/25
The fall lowing la conci explanet ion the issres code abovw does not ensure that vertices are sountesmcluchen arder, It calcalates x3, yä, 28, and in the wrong urdes, which resulla in the coordinates being in clockxise order instead, to fix this, need to the csleulations for x3 and y3 y4.
Nurbez o2 passing repairs:
#4# PARTICIPANT PREDBACK 41
program calesiates vertices
Llockvice-oriennt square, ocunter-clootwisee this, 1y2 y!l and (x2 x() should be negated in the equat one.
Nunbie of passing repairs: 25/25
VEFORACK
The calculation for y3 and y4 incurrectly cubtiani the diseet ion vector y0 and yl cespectioely. each of these caees, the calculation. can be tixtd by shanging the tires subtraction to addition. Number of passing repsira: 0/25
Listing 12: Task 4347, Program B.
One day, 9n5 soople 19a6 is an even numbar) net on a plaza and made twe round dances, each round dance ccnsists of exactly 9Arrzcin) [2]5 people. Your task is to find the number of ways 6ns peopla can make dances
these consista unautly SiCrae(a) people. Bach should dancas.
exactly
Round dance dance circle consiscing of $18 or nore people. Tao round dances are andistinguishable be cransformed anothar by abeosing
(equal)
example,
dances 9[1, 3. 219, 914, 310 912, 2, 038 incistinquishaale.
For exampla, if Spa26 then tha nunber of ways is 519: one round dance consists of tha first person and the persor.
For exampia, if $pa16 then the nunber of vays is 536. Possible opticas: one round dancs 5[1,2]5, another 418; sne round dance 5[2,4]8, anothw: 8[3, round dance $[4,1]8, snothe: $[3,2]8,
numbar ways people exatslly ShEraa(ni ( 1218 people.
dances round dance consists
Inpul
The Input containk io one i integer 3n8 (32 Ve Vie 2091, 3r8 is even number.
Out put
integer
xaye
guaranteed in
5649
integer deta type.
Esamplex
Input
Octput
47

---

Published as a conference paper at ICLR 2024
Input
Octpet
Input
Octput 1260
Input
20
Output
121 1215451004098320
INCORRECT PROGRAM [actorial(t))
12 a wn reluen
elees
(ecorria((n1]
inz (input (1)
ana Eacturial 17 21 11 12
pzint (ana)
ERROR MESSAGE
Inguteee
PeoPrggnan Output
eeeBxeected Outpatewe
+4g GPT-4 PREIRACK
The Laxuu with Uhe code is that is intweer diviaion when celoulat the final answes, which down incorrectity. tix can resule to an irteger,
result
+ Nurber of passing repairs: 0/25
GPT-4 FRETRACK
The following ls cuncias explanet
the issue: oude attenpta to caloulate number ways using integer division, which may result in loss at preaision and incorrect results. To rix the sesue, change integer division to regular lar division, then round the resuls to the nearest Ineeger using "2ound(2".
Number of passing repsire: 0/25
++ FARTICIPANT FEEDBACK
so why zee 6. There en arroneous additions) division by which 18 the true resaon the wrong enswer. Number oz pacsing repsire: 25/25
problem
#+t PARTICIPANT FEEDBACK
Or Line 11, the pregran celculates tern with correct numerator, but en incorrect denominator. It divides should just divide
by (2+n), oz pacsing repaire: 25/25
Listing 13: Task 4855, Program A.
B78 SPECIFICNISON
The fackor scient ifie jousnal ia measure reflect ing the avecage namber of citat lons to ast intis using simpliried tornula calculation insact faczozeh Veraditeent [Tosel count aitations received by articles publishad jousmal)) (ltest (Toval nusbes of accicles shei)) M
published acience journais. this
Rounding is always per
exengle inpact wasting that publ 5385 acticles cited 58948 Lines is
"Journal
23.535 counding up Lo 8249.
You the editton of one ent iffe journal. You know how many artioles wonder many selenn you have bribe to cite your asticles naot evnera danands. Since monay is Light you want he the the mimins anount of sctent ists. Rech bribed acientia lst buys you single citasion.
going publi and the
pusking
speci
fautos.
Trppt
input will contain 925 integers, SAB (91 Aleg A Veg 1009), the mumber Acticl you plan to pubiiaah and $I8 161 Vleq I Vlay 1005), the inpaut factos che benees requice.
and only
first and only 1inw output should contain. ore Inceges, the ninimal scientiata you Lo bribe.
Examppess--...... Sample Inputt
48

---

Published as a conference paper at ICLR 2024
Bample Oucput 21 875
Sample Input 100
Bample Output 2:
100
#* INCORRECT PROGRAM import moth
map map(int, inpub (1)
acsdnd_citarians nath.ceilla + II 0.513
veienLists_to__be ibe aeaded_ctta.ions (wciontista_ro__ell)
#*6 ERROR MESSAGE mneingteee
Pooorog Output
15
Expestedd Outpateee
375
GPT-4
The following cuncias explanal
issae: celculation number of acientists beibe Incorrect. It should be calculated by first finding the ninimun number of citations needed: "A F', and then rabt the currert rumber of di lonss "A 11",
Nunbie passing repairs:
#4 GPT PRETBACK #2
The following is cuncias explanal iun of the issue: The cude seloulates the sequired number of scienliats
by racting Incorrectly selculates current number citations.
Narbie of passing repairs: 1/25
PARTICIPANY PREDBACK
iat variable
acneceasaty.. It's sufficient Just
Number of passing repsirs: 17/25
EARTICIPANT
The program seens have conceptual misunderaterding. Speai ly, the rumber scient sta needed to be the is equel to the number of required citations ('citat ("oitationa rewddd" flowever, the calculation ion "citation_nendd"
is incorrect it implies rounding tactor conventionally 3,49 ling fanction impact actual formula fur "citation_nnedle" should be "A.(I Number oz passing repsire: 25/25
3.51 gome
49


Reviews:
[{"review_id": "61K4YnL2Hb", "reviewer": "Reviewer_Vfe4", "summary": "This paper investigates the sample efficiency of a self-repair approach for LLM based code-generation tasks. It evaluates performance of this approach on HumanEval and APPS dataset using ColdeLLama-13b-instruct, GPT3.5, and GPT4  and provides several insights based on the results: 1) Sampling without repair can perform equally or better than self-repair in almost all sample budgets; 2) Initial sampling diversity is more critical than the diversity of repair samples; 3) Quality of feedback significantly improves the performance of self-repair.", "questions": "1. Is there a specific reason to restrict feedback and repair samples to 1 in the analysis of feedback boosting (section 4.2)? \n2. Should we expect similar results using the ‘self-debugging’ approach that uses few-shot prompts?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper provides several new insights on self-repair for code generation compared to the baseline method of sample generations without self-repair. It shows how sample budget and initial sample diversity could impact the efficiency of code generation. \n* The investigations on self-repair performance improvement only by improving feedback quality could enable more interesting future ideas.\n* Overall, the paper is well-written and easy to read. The authors did a great job in highlighting the key limitations of the analysis.", "weaknesses": "The experimental results presented in this support the claim around the limitations of self-repair. Interestingly, the findings around overall efficacy compared to baseline somewhat contradicts with the results from Chen et. al. 2023b that shows self-repair could provide significant increase in sample efficiency. Although ‘self-debuggging’ work from Chen et. al. is mentioned in the related work, I think more comparative analysis would strengthen the claim of this paper. Analysis results from more diverse code generation task including datasets other than python language would also be interesting additions to the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ooxGpLOiZc", "reviewer": "Reviewer_PNGf", "summary": "This paper investigates the ability of large language models (specifically, Code Llama, GPT-3.5, and GPT-4) to perform self-repair which refers to the model's capacity to identify and correct mistakes in its own generated code. The paper analyzes the effectiveness of self-repair in addressing code generation errors, specifically on problems sourced from HumanEval or APPS datasets. The findings suggest that the gains achieved through self-repair are often modest and vary significantly across different subsets of data. In some cases, self-repair does not result in noticeable improvements. The paper proposes that this limitation might be due to the model's ability to provide feedback on its own code, and stronger models might enhance self-repair effectiveness. Additionally, the paper explores the impact of providing the model with feedback from human participants, showing that this feedback significantly benefits the self-repair process, even for the advanced GPT-4 model. The paper offers a brief qualitative analysis to shed light on why this human feedback is valuable in improving code repair performance. Overall, the study provides insights into the challenges and potential enhancements of self-repair mechanisms in large language models for code generation tasks.", "questions": "Please explain and address the weaknesses. Otherwise, the paper is well-written and clear.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "__The paper systematically analyzes the self-repair capability of LLMs.__\n\nWhile existing works of self-repair mostly argue that self-repair is feasible for LLM, this paper is the first work to systematically study the strengths and weaknesses of self-repair. Specifically, I appreciate the systematic comparison between self-repair and the pass@k performance, though the conclusion might not be generalizable (see Weaknesses), showing that when the computation is comparable, their benefits are also similar, and also suggesting the optimal combination of these two techniques is the most effective. Also, studying the quality of feedback is also a novel perspective to analyze the self-repair capability of LLM\n\n__The paper sheds light on the weaknesses of LLM in self-repair, providing clear takeaways and indicating the potential future work in this direction.__\n\nThe paper is well written and properly organized, and it sheds light on the weaknesses of LLM's self-repair capability. Such weaknesses can be due to the lack of both data and carefully crafted training strategy for self-repair capability, indicating the future research direction of refining the existing LLM with better self-repair capability.", "weaknesses": "I am overall positive regarding this paper, and I appreciate the systematic study the paper performs to quantify LLM's self-repair capability. However, the conclusions are claimed in a strong and general tone, while the study itself is actually limited in scope for two reasons.\n\n__The scope of the study is limited to solving isolated programming challenges while ignoring real-world development.__\n\nThe study focuses completely on the programming challenges datasets, such as HumanEval and APPs. These datasets have several characteristics that are not realistic in daily development, therefore, though I appreciate the initial conclusions of this paper, these takeaways might not be applicable to a more realistic scenario. \n\nFirst, the samples in the studied datasets are provided with clear and complete problem descriptions, which are not always available in real-world programming practice. One of the main reasons that pass@k works so well in programming challenges is that the expected functionality of the program is fully revealed and clearly explained in the prompt, so the model is able to maximize the diversity within a narrowed semantic space when generating multiple sequences. However, such clearly explained prompts, as natural language, are typically not available during the ongoing development, where the developers start with a very high-level goal and eventually design modules and implement them piece by piece. During this process, the human intent is not always explicitly specified, as docstring or comment, before the code LM is prompted to complete the following code snippets. In these cases, the execution of unit tests provide meaningful feedback to specify and concretize the expected functionalities, which cannot be leveraged by the top-k generation but is valuable guidance for iterative self-repair. Therefore, though I agree with the takeaway that pass@k is comparable to, sometimes better than, self-repair in the programming challenge dataset, such observation might not be realistic for daily development and requires further study.\n\nSecond the samples in the studied datasets are mostly short and self-contained, missing the complicated data and inter-procedural dependencies. Pass@k explores the breath of each token within the sequence without directional guidance, but such breath or search space exponentially increases with the increase of the code length. The program challenges datasets contain samples mostly up to tens of lines of code, significantly underestimating the complexity of real-world software, which includes hundreds or thousands of lines of code within one single file and maintains complicated dependencies. Generating k sequences blindly without feedback may hardly fulfill the expectation due to the large search space, while execution feedback, such as the indication of a missing third-party library, helps the model quickly locate the problematic code and focus on fixing just that part. Therefore, when the complexity of the program increases, it requires further study to understand whether self-repair is equivalent as top-k generation.\n\n__It is not clear whether fine-tuning for self-repair could easily overcome the weaknesses or not.__\n\nThis paper focuses on the LLM's self-repair capability only with prompting, without optimizing the model parameters towards the self-repair capability. It is not clear whether a cheap fine-tuning could quickly enable the model's capability of understanding and leveraging the feedback efficiently. Drawing the conclusion that self-repair is not a silver bullet without trying straightforward fine-tuning might be too strong.\n\nTo conclude, I would encourage the author to consider constraining their conclusions with the study's scope and add a discussion section to mention", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DlPuDAz6J1", "reviewer": "Reviewer_LPE6", "summary": "The authors break the problem of program repair into four steps:\n\n1. (LLM) Generate an initial population of programs.\n2. Run the programs to obtain error messages.\n3. (LLM) Analyze the errors, and generate feedback as to the cause of failure.\n4. (LLM) Given the feedback, generate a repaired program.\n\nThis paper differs from prior work in part because it emphasizes the importance of step (3) -- using feedback to direct the repair.  The authors run a number of experiments with different models, explore what happens when a stronger model provides feedback to a weaker one, and also explore what happens when humans provide feedback to a model.\n\nThe authors also explicitly model the cost of repair.  Is running steps (3) and (4) more effective than simply drawing more samples from the initial population?  The answer turns out to be \"not really\".  \n\nThe experiments are well-designed and the authors make several interesting observations.  To paraphrase:\n\nA.  LLMs are currently quite bad at program repair.  The best way to get a correct program is to generate a large and diverse initial population, in the hopes that one of the initial programs is \"close enough\" to work with minimal tweaks.  Even a single repair step yields at best marginal improvements.  LLMs are thus very different from human programmers; they seem to be unable to iteratively refine and debug programs over multiple steps.  \n\nB.  The main limitation is the quality of feedback.  Given feedback from a stronger model, or from a human, LLMs show substantial gains from repair.  However, LLMs seem to have difficulty finding problems in the code that they generate.", "questions": "Your illustration of the repair-tree is interesting, and it brings to mind the idea of extending this evaluation technique to a proper tree search.  You might be able to get better results by using a value network, and focusing only on the nodes of the tree that are most promising for repair, in a manner reminiscent of evolutionary search, with the LLM as mutator. \n\nSuccessful repair attempts, preferably after several iterations of feedback and repair, could also be used to fine-tune the LLM to generate better feedback, and to generate better repair code given the feedback.  (See weaknesses, above.)\n\nHave you considered any experiments along these lines?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is well-written, and the authors formulation of the problem is insightful.  In particular, they do not merely test the accuracy of program repair in isolation, they compare it against the alternative, which is generating new initial programs from scratch.  \n\nThe experiments seem to be well done, particularly the ones which use a stronger model or human to provide feedback to a weaker model.", "weaknesses": "The authors do not attempt to fine-tune a model on the task of program repair.  Thus their experiments mainly demonstrate that LLMs, *as currently trained*, do not do a good job at the repair task.\n\nTo be fair, fine-tuning is probably out of scope for this paper, especially since some of they models they test are private, and accessible only via an API.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)