Teaching Large Language Models to Self-Debug
Authors: Xinyun Chen
Maxwell Lin
Nathanael Schärli
Denny Zhou
Venue: ICLR 2024 poster (2024)
Keywords: large language model
self-debugging
Primary area: general machine learning (i.e., none of the above)

Abstract: Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose self-debugging, which teaches a large language model to debug its predicted program. In particular, we demonstrate that self-debugging can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by leveraging code execution and explaining the generated code in natural language. Self-debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, self-debugging with code explanation consistently improves the baseline by 2-3%, and improves the prediction accuracy on problems of the hardest level by 9%. On TransCoder and MBPP where unit tests are available, self-debugging improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, self-debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10$\times$ candidate programs.

Published as a conference paper at ICLR 2024
TEACHING LARGE LANGUAGE MODELS TO SELF- DEBUG
Xinyun Chen'
Maxwell Lin?
Nathanael Schärli'
Denny Zhou'
Google DeepMind 2 UC Berkeley
(xinyunchen, schaerli, deneyozoeeooouggggglee.ccoo mx mxlingberkeley.ed
ABSTRACT
Large language models (LLMS) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solu- tion in one go becomes challenging. In this work, we propose SELF-DEBUCE DELLLDEGGGGGNG, which teaches an LLM to debug its predicted program. In particular, demon- strate that SELF-DEBUGGHE BGGGIGG can teach the LLM to perform rubber duck debug- messages,
ging; i.e., without any human feedback the code correctness the model able to identify its mistakes by investigating the execution results and explaining the generated code in natural language. SELF-DEBUGGING achieves the state-of-the-art performance on several code generation benchmarks, including Spider for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Pythor generation. On the Spider benchmark where there are no unit tests verify the code correctness, consistently improves the baseline by 2 3%, and improves the prediction accuracy on problems of the hardest level by 9%. On TransCoder and MBPP where unit tests are available, SELF-DEBUGGING improves the baseline accuracy by up 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, SELF- DEBUGGING notably improves sample efficiency, and match or outperform baseline approaches that generate more than 10x candidate programs.
with code explanation
INTRODUCTION
Code generation has been long-standing challenge with variety of applications (Yu et al. 2018
Austin et al. 2 Li et al.
2022 Roziere al.) 2020). In particular, recent large language models have demonstrated significant leap in improvement over prior deep neural networks (Chen et al. 2021a Nijkamp et al.
2023 Zheng et al. 2023 Xu et al. 2022). However, for many programming tasks, generating correct code with a single attempt is challenging. Inspired by observations that correct code is much more likely to be predicted when multiple programs are sampled from the model (Chen et al. 202 la Chowdhery et al.) 2022 Li et al. (022). one line of work has designed reranking techniques to select the best candidate from multiple samples, which typically requires tens
of samples to start with (Shi et al. 2022 Zhang et al. 2022 Ni et al. 2023 Li ct al. 2022).
Intuitively, even for human programmers, there no guarantee that the code written on the first try is always accurate. Instead of completely discarding the incorrect code, humans typically look into the code and investigate execution results, then make changes resolve the implementation errors. While some recent works show that large language models have potential for generating feedback messages critique and refine their outputs for some natural language and reasoning domains (Shinn et al. 2023 Madaan et al. 2023b; Kim et al. 2023 Nair et al. 2023 Bai et al.) 2022). prior works suggest that such large language models are not yet capable of correcting code when lacking external feedback, such unit tests human instructions (Chen et al., 2023a).
In this work, we propose SELF-DEBUGGING, where we teach the large language model to debug its own predicted code via few-shot prompting. Without any additional model training, SELF- DEBUGGING instructs the model to execute the code, then generate a feedback message based on the code and its execution result. Different from prior works utilizing human feedback for code repair, where the feedback message explains the code errors and how to fix them (Chen et al.) 2023a Austin al. 2021), SELFF- DELF-DERUGGING teaches the model to identify the implementation errors

---

Published as a conference paper at ICLR 2024
via investigating into execution results and explaining the code by itself. This debugging process is reminiscent of rubber duck debugging for human programmers, where explaining the code line-by- line in natural language rubber duck significantly boosts debugging efficiency without expert Tillustrates the full procedure of SEL BELF-DDEUGGIIG.
guidance (Hunt & Thomas 2000). Figure |
We evaluate SELFF-EEUGGING
variety of models, showing that SELF- DEBUGGING achieves the state-of-the-at performance on different types of code generation tasks. On the Spider benchmark (Yu al., 2018) for text-to-SQL generation where there are no unit tests in the problem description, with code explanation, SELF-DEBUGA numbers of initial programs, and improves the prediction accuracy on the most complicated SQL queries by 9%. On both TransCoder for code translation (Roziere et al.! 2020) and MBPP for text-to-Python generation (Austin et al. (021), utilizing unit tests along with code explanation boosts the accuracy by up to 12%, and code ex explanation alone without debugging also consistently improves the code translation performance by 3%. Meanwhile, SELF-DEBUGGIAS
consistently improves the baseline by 2 3% with different
improves sample efficiency, and can match outperform baseline models that sample more than 10x predictions. Our work indicates that besides improving their ability to generate code from scratch, teaching large language models to perform SELF- DEBUGGING without human guidance is another promising path to enhance coding capability and reduce the sampling cost required to accomplish challenging tasks.
Code exccution
TEST
Executor
genanation
Feedback
Model
Gode
Explenation
Step Cade explanation
Figure I: BELF-DEBUGGING with large language model. At each debugging step, the model first generates new code, then the code is executed and the model explains the code. The code explanation along with the execution results constitute the feedback message, based on which the model infers the code correctness and then adds this message to the feedback. The feedback message is then sent back to the model perform more debugging steps. When unit tests are not available, the feedback can be purely based code explanation.
2 PROMPTING FOR CODE GENERATION
In this section, we discuss the background on prompting for code generation using large language models. We first introduce few-shot prompting, then discuss how to select the final prediction from multiple samples based on code execution.
Few-shot prompting. Few-shot prompting aims to instruct the language model to solve a task with several input-output demonstrations (Brown al., 2020). Taking text-to-SQI. generation an example, the few-shot prompt prepends the question of interest with a list of (question, SQL.) pairs, so that when the model is asked predict subsequent tokens of the given prompt, it will follow the prompt format to generate the SQL query. Besides input-output demonstrations, we can optionally add instruction the prompt to provide high-level task description (Ouyang et al. 2022 Sanh et al. 2022 Suzgun et al. 2022). For example, in the first two steps of our shown in Figure 3] both prompts start with instructions that ask the model to generate explanations. We provide the full few-shot prompts in the appendix.
prompt
Execution-baseed code selection. Prior works demonstrate that decoding multiple samples can significantly improve the performance of large language models (Wang et al.) 2023) Shi et al. 2022). In particular, for code generation tasks, we can utilize code execution to select the final prediction (Chen et al. 2219 Li et al. 2022 Shi et al. 2022 Zhang et al. 2022 Ni al. 2023) One line of work selects the final prediction using the majority vote of execution results (Chen et al. 2019 Li al. 2022) Shi et al. (2022). while other works design reranking schemes to improve the performance (Zhang et al. 2022; Ni et al.) 2023: Yin Neubig 2019 Zeng et al. 2022). ). In this
work, when there are multiple predictions, follow the first line of work select the predicted code
2

---

Published as a conference paper at ICLR 2024
with the most frequent execution result among those that do not encounter execution errors, then apply SELF-DEBUGGING to the code.
Some code generation tasks accompanied with unit tests to specify the program execution behavior (Chen et al., 2021a] Austin et al. 202T Li et 2022; Hendrycks et al. 202T). Specifically, the unit tests are set of input-output pairs [(t 0k ]== and program P passes unit tests when P((x) = oR: Ak E {1,... }. When unit tests are presented in the problem description, we filter out programs that do not pass the unit tests before performing the execution-based majority vote.
SELF-DEBUGGING FRAMEWORK
Figure iilustratts our SELF-DERUGGIN framework for iterative debugging, where we utilize a pretrained large language model without finetuning it. One turn of SELF-DEBUGGING consists of steps: Generation, Explanation, and Feedback.
For the Generation step, given the problem description, the model predicts candidate programs.
During the Explanation step, the model is prompted to process the predictions in a semantically useful way, such as explaining the prediction in natural language, creating execution trace of the predicted code for a sample input.
Finally, for the Feedback step, feedback message concerning the code correctness is generated. This can be determined by asking the model itself, or can be generated externally from unit tests.
The debugging process terminates when the feedback message states that the prediction is correct, a maximum allowed number of debugging turns is reached.
In practice, a SELF- DEBUGGING turn does not always include all steps above. Figure GELUGDNB feedback formats that can be automatically acquired and generated.
2 presents
Staple Heliore
Explonaaiion
Dvn Pydl
comeet foom locchoo
(Rigpe Pootfori
Pyenet
[DT Nethhol
Explanztion]
The llssees ustil quetieni given number and tthe diviant grzater than
count
cppel to
oount
queeient givcn and
[Python Explanatioo)
shile ((n 4)
1
131
The variitte is soot as the divioor
stile Lin
13:
tthe diviaet
(Sipll Feodback]
The above Python inmiatioo does not the same the Cre code. Comest the
cor (oir
Pythos innslatios
Whea the loop is exccuted, the queticst the aivce surber and will be addied dhe vuriable crc.
Fallal woseet count_tre Trng.reroee, http:e/faaorrrlaaeeeeeoeee
The vanable will mattippiad by L...]
Actual Resalt: 1.5
[mooi]
Correct the mansiation.
'on' and
Revised Pyrhum
Paser the while
199
"ret aveymad onn
which evaleates 1.X.
is multilieddy by becomee 25.
(ont
The incomeot mondattion dull bocause the diviion oi Pythor setumss a floon viloo. whereaa in Cire return as integer valie, I...]
Check Bhe dition ogain 19 25)
greater equal 1. Exit the while keep, [...]
Figure 2: Examples of SELF-DEBUGGING prompts with different feedback formats for code transla- tion. See Appendix F for more explanations.
Simple feedback. The simplest form of automatic feedback is sentence that just indicates the code correctness without more detailed information, which omits the Explanation step in a full SELF-DERUGGING turn.
Unit test feedback (UT). When the problem description includes unit tests, besides utilizing code execution to check code correctness, which provides richer information for debugging. Figure presents a sample unit test feedback message. Intuitively, inspecting runtime error messages and execution results of failed unit tests also
also incorporate the execution results in the feedback.
3

---

Published as a conference paper at ICLR 2024
helps human programmers debug more effectively. In our experiments, will demonstrate that leveraging unit tests whenever available consistently boosts the SELF-DEBUGGRS performance.
Code Explanation feedback (Expl). Despite recent promising progress showing that large language models can generate critiques to avoid harmful model outputs (Ganguli et al. 2023 Bai et al. 2022)
and improve their performance
some natural language and reasoning tasks (Shinn al. 2023 Kim
let al. 2023] Saunders et al., 2022), prior work has yet to show the effectiveness of model --gnerrtted
feedback on code generation (Chen et 2023ap. On the other hand, large language models have
been shown to be capable of describing their generated problem solutions in both text (Wei et al.
2022 Kojima et al. 2022 Zhou et al. 2023) and code (Gao al., 2022: Chen et al. 2022 formats.
Inspired by these observations, instead of teaching the large language model to predict error messages, we propose SELF-DEBUGGING via explaining the generated code. This debugging process is reminiscent of rubber duck debugging. where programmer debugs code by explaining it line- line-by-ling by-line rubber duck (Hunt & Thomas 2000). By describing the code implementation and comparing to the problem description, human programmers are usually able to identify the bugs without extra guidance. Empirically, we observe that large language model can also benefit from rubber duck debugging, especially when unit tests are not available.
Execution trace feedback (Trace). Besides explaining the code itself, human programmers also often understand the code semantic meaning by simulating the execution process. Prior work on code repair has demonstrated that training the repair model on execution traces improves the debugging performance (Wang et al. 2018 Gupta et al. 2020). Therefore, when unit tests available, examine another explanation feedback format where the LLM is instructed to explain the intermediate execution steps line-by-line. Note that both the execution trace and the line-by-line explanation come from model generation instead of code execution, thus the trace feedback does not require more information than the pure code explanation feedback; i.e., no access to intermediate execution states.
4 APPLICATIONS
In our evaluation, we showcase applications of SELF-DEBUGGING to different code generation domains: text-to-SQL generation, code translation, and text-to-Python generation. First, text-to-SQL. generation represents the scenario where no unit tests are available, where the model to identify errors and justify code correctness by explaining the predicted code. In code translation, we demonstrate how to utilize the execution of unit tests to provide feedback messages. Finally, the cext-to-Python domain requires the model to infer prediction correctness when only subset of unit tests are presented in the problem description.
enables
4.1 TEXT-TO-SQ! GENERATION
The goal of text-to-SQL tasks to generate the corresponding SQL query given a question and the database information. We evaluate SFL SEIFFDDRUGGINS on the development set of the Spider benchmark Yu et al. 2018) where the unit tests are not available, thus it becomes challenging for the model to infer the correctness of the predicted SQL queries.
Figure 3 demonstrates an example to illustrate the
process. At high level, the debugging process contains 3 steps. First, prompt the model to summarize the question and infer the return type required by the question; i.e., the number of table columns in the SQL output. In the second step, the model generates the SQL explanation. Finally, the model compares the inferred SQL explanation and question description, and then predicts the correctness of the current SQL. query.
4.2 CODE TRANSLATION
Next, we apply SELF-DEBUGGING to code translation, where the goal is to translate code in one programming language into another language. In our experiments, we use the TransCoder dataset (Roziere et al. 2020), which includes a test set of parallel functions different programming languages along with unit tests. Following (Chowdhery al., (022), evaluate SELF- DEBUGGGEUG on the task of translating C++ code to Python code, with 560 problems and 10 unit tests per problem. The problem description includes both the C++ code and all unit tests.
4

---

Published as a conference paper at ICLR 2024
Step I: Question explanation
Infer the type of the question.
[Question]
Which customers have both Road" and "Shipped" order status? customer names.
[Question Explanation]
The question returns
names have both Road" and "Shipped" order question returns
column.
Step 2: Code explanation
Summarize the return type of the exccution.
[SQL]
SELECT cusiomers.csstomer name FROM customers JOIN orders ON cuslomers.customer id cuattoere
WHERE orders.order status "On Road" AND onders.onder status "Shipped"
[SQL Explanation]
t..the SQL query returns an empty table, because the "WHERE" clause contains mutually exclusive conditions.
Step Self-Deuugging with explanation
[Question]
[Explanation summary]
As in your explanation, the SQL query returns an empty tahle, because the "WHERE" clause contains mutually exclusive conditions.
The question returns column, the customer names who have order status of "On Road" and "Shipppdd". [F [Foodback]
So the SQL prediction above wrong. Please the SQL.
[Revisod SQL
SELECT customers.customer name customers
orders, customer WHERE customer name FROM customers JOIN orders ON "Shipped"
orders.order: status "On Road" INTERSECT SELECT
customers. custommrr orders.cusiomer id WHERE orders.order,
[Explanation summary #1]
As in your explanation, the SQL query returns table with I column, the customer names who have the order status "On Road" and "Shipped". The question retims column, the customer names who have order status of both "On Road" and "Shipped", So the question returns column.
[F l'eedback #1] So the SQL predictioon above is correct!
An example of SELF-DEBUGGIN prompting for text-to-SQL. generation. The model is required to predict the code, explanation and feedback. See Appendix |F more explanations.
Figure 3:
Given the availability of unit tests, we apply SELFF-DEUGGING only when the predicted Python code does not pass all the unit tests. Figure provides sample SELF-DEBUGGING prompts.
4.3 TEXT-TO- GENERATION
In addition, we evaluate SHLF-DERUGGING in the setting where only a subset of unit tests are presented in the problem description. Specifically, we perform experiments
the test set of MBPP (Austin et al. 2021). which contains 500 Python problems with text descriptions, where each problem has unit tests. We follow prior work (Shi al. 2022 Ni et al. 2023) in including the first unit test in the prompt part of the problem description, and keeping the remaining 2 unit tests hidden for full evaluation. Similar code translation, results in the feedback message, but the main difference is that the model still needs to infer the code correctness even the predicted Python code passes the given unit test.
can also utilize the unit execution
EXPERIMENTS
We evaluate SELF-DERUGGING on code-davinci--00 (Chen et al. (2021a), referred to as Codex), g. parameters. For initial code generation, when starting from one program, we perform greedy decoding with temperature
3. 5- turbo, g (OpenAl (202)), and StarCoder (Li et al. (023b) with 15.5B
O. When sampling multiple programs for problem, we set temperature т = 0.7, then we perform execution- based selection described in Section 2 For Transcoder and MBPP, also evaluated oracle baseline. which has access all unit tests and samples another program when the greedy decoding baseline makes the wrong prediction. All experiments for SBI.--DERUGGING use greedy decoding generate feedback messages and new programs. We set
5

---

Published as a conference paper at ICLR 2024
the maximum number of debugging turns to be 10, though empirically the successful debugging processes mostly end within turns. We present the full prompts for experiments in the appendix.
We evaluate BELF-DEBUGGIN against two types of code reranking baselines follows.
Models trained for the given task. The Spider benchmark contains a training set of over 10K samples, and the tfat-oofthee-ee art models mostly finetuned this training set. We compare SELF-DEBUGGING to T5-3B + N-best Reranking (Zeng et al.] (022), where the T5-3B model is specially trained for text-to-SQL. generation. Although LEVER (Ni et al. 2023) also utilizes Codex to generate candidate SQL queries, they train a verifier to select the final prediction based on execution, and thus this approach also requires extra training. For both TransCoder and MBPP benchmarks, the Sfate-of-the-ar results are all accomplished by LLMs, thus we defer the comparison to Appendix D
approaches. We compare SELF-DEBUGGING against recent approaches that also only perform prompting without any additional training. Specifically, Both MBR-Exe (Shi al. 2022) and Coder-Reviewer (Zhang et al. 2022) first generate multiple candidate programs by prompting the pretrained model. Afterward, MBR-Exec (Shi et al.] (2022) selects the program with the most common execution output, while Revie-Re (Zhang et al. 2022) selects the program by utilizing both the likelihood of the predicted code given the problem description (Coder score) and the likelihood of the problem description given the predicted code (Reviewer score).
Prompting-
5.1
MAIN RESULTS
Table I:
Comparing SELF- BELF-DEBUGGING prior ranking techniques. The baseline refers to our best results without SELF-DEBUGGING, and uses multiple samples.
(a) Results on the Spider development set.
(b) Results on MBPP dataset.
Spider (Dev)
n samples
w/ training
Prior work (Prompting with reranking)
T5-3B N-best Reranking
80.6 (n 25)
MBR-Exec
63.0 (n 25)
LEVER (Ni et al. )
81.9 (n 50)
Reviewer
66.9 (n 25)
Prompting only who debugging
LEVER
68.9 (n = 100)
74.5 (n 25)
SELF-DERUGGING (this work)
MBR-ExeC
75.2 (n 50)
Codex (baseline)
72.2 (n 10)
SELF-DEBUGGING (this work)
Simple
73.6
Codex (baseline)
81.3 (n = 32)
UT UT + Expl.
75.2
+ Expl.
84.1
75.6
First, we compare SELF-DEBUGGING to prior code reranking approaches in Table where both SELF-DEBUGGING and prior prompting- based approaches use Codex. We demonstrate that SELF- DEBUGGING consistently improves the performance.
Next, we compare different feedback formats for BELF-DEBUGGING with different LLMs in Table 2 On the Spider benchmark where unit tests are not available, simple feedback alone does not notably improve the performance, because the model typically struggles to distinguish between correct and wrong SQL queries without explanation, and does not produce meaningful changes to the initial predicted SQL queries. Note that on TransCoder and MBPP benchmarks where at least one unit test is available for SELF- SEL.F-DDBUGGE DEBUUGINGG simple feedback also utilizes the execution result to infer the code correctness even if the execution information is not presented in the feedback message, thus simple feedback still improves the model prediction for both applications. In Section 5.2.2 we further present an ablation study showing that leveraging code execution crucial for enabling a performance leap with
For all tasks, models generally benefit from richer feedback for SELF- DEBUGGING, especially when the execution information present the feedback. The oracle baseline results further justify that wrong predictions and sampling another program independently. Note that on MBPP, the oracle baseline has access to all unit tests to determine the code correctness, while DEBL-GEEBGGING only has access to unit test, as described in Section 4.3) Nevertheless, the oracle bascline still generally performs worse than SEL.F- FE.F--DRUGGING with the simple feedback, let alone other feedback types.
is more effective than completely discarding

---

Published as a conference paper at ICLR 2024
Table 2: Results of SELF-DEBUGGING with different feedback formats. The baseline corresponds the greedy decoding performance. Note that the oracle baseline generates another sample when the greedy decoding prediction wrong, assuming access all unit tests.
(b) Results on TransCoder.
(a) Results on the Spider development set.
TransCoder
Codex
GPT-3,5
GPT-4
Spider
Codex
GPT-3.5
GPT-4
StarCuder
Baseline
80.4
77.3
Baseline
77.5
71.1
73.2
64.7
Oracle Baseline
84.5
80.0
Simple
77.5
72.2
73.4
64.9
Simple
89.3
91.6
80.9
+Expl.
80.8
72.2
73.6
64.9
UT
91.6
92.7
88.8
+ Expl.
92.5
92.7
90.4
Trace.
87.9
92.3
89.5
(c) Results on MBPP
MBPP
Codex
GPT-3.5
GPT-4
StarCoder
Baseline
61.4
67.6
72.8
47.2
Oracle Baseline
65.4
72.6
77.0
50.0
Simple
68.2
70.8
78.8
50.6
UT
69.4
72.2
80.6
52.2
Expl.
69.8
74.2
80.4
52.2
Trace.
70.8
72.8
80.2
53.2
By comparing different LLMs, observe that:
Although GPT-4 is shown to be much stronger than previous GPT models on many tasks (Ope nAI )023), on Spider, both its initial SQL generation and SELFFDERUGGGIN performance much worse than Codex. One possible reason that GPT-4 is tuned for zero-shot prompting, and thus it does not follow few-shot exemplars well as Codex. Meanwhile, both GPT-3.5 and GPT-4 might not be optimized for SQL generation, and thus their zero-shot results are more than 10% worse than their few-shot counterparts.
GPT-4 significantly better than Codex and GPT-3.5 for initial Python code generation on MBPP. Meanwhile, its self-debugging performance gain is par with Codex: over 12% improvement on Transcoder, and 8% improvement on MBPP
Despite that the baseline performance of StarCoder is considerably worse than GPT models, BELF-DEBUGGING with unit test execution also offers a significant performance gain, i.c., 6% MBPP with the execution trace feedback.
5.2
ABLATION STUDIES
5.2.1
BELE-DEBUGGIN IMPROVES THE SAMPLE EFFICIENCY
90
22.3
debugging selt-debuaging
Self-debugging Codex
84.5
85.2
76.4
Spider Accuracy
86
Spider accuracy
63.9
60
82.5
02
80
17.5
20
78
Samples
Medium
Hard
(a)
(b)
Figure 4: Ablation studies on the Spider development set with Codex. (a) Accuracies with different numbers of initial samples. (b) Breakdown accuracies problems with different hardness levels.
Figure 4a demonstrates that numbers of initial samples. In particular, Spider, applying SELL--DEUGGING predictions gener-
notably improves the sample efficiency with different
7

---

Published as a conference paper at ICLR 2024
ated with greedy decoding matches the baseline accuracy using 16 samples, and SELF-DEBUGGING from samples outperforms the baseline accuracy using 32 samples. Note that typically one debug- ging turn is sufficient, and the accuracy improvement after turn is within 0.1%. Appendix E presents similar sample efficiency improvement on other benchmarks.
5.2.2 IMPORTANCE CODE EXECUTION
Table 3: Results of SELF- SELF-DEBUGGING without unit test execution.
(a) Results on Transcoder.
(b) Results on MBPP
TransCoder
Codex
GPT-3.5
GPT-4
MBPP
Codex
GPT-3.5
GPT-4
Baseline 80.4
89.1
77.3
Baseline
61.4
67.6
72.8
Simple
83.4
89.1
78.2
Simple
57.6
68.2
76.0
+ Expl.
83.9
89.1
78.0
Exppl
64.4
68.2
76.0
Trace. 83,9
89.1
78.4
Trace.
66.2
69.2
76.4
By default, we leverage unit test execution for BELI-DEBUGGING when applicable. In Table 3 we examine the scenario where the SELF- SELF-DEBUGGING process does not involve code execution Transcoder and MBPP, thus models need to fully rely on themselves to infer the code correctness as in Spider experiments. Our main findings
follows:
With Codex, SELF-DEBUGGING still improves the performance by up to 5%, and the execution trace feedback consistently improves over the simple feedback performance.
GPT-4 without unit test execution improves the MBPP accuracy by 3.6%, and the improvement other benchmarks is up to around 1%.
Compared to Codex, few-shot prompting does not notably improve the SELF-DEBUGGING performance of GPT-3.5 and GPT-4. Without unit test execution, both models tend to be overconfident their own initial predictions, while GPT-4 is better on Python generation.
In summary, code execution plays an important role SELF-DEBUGGING, however, sometimes LLMs can still improve their performance purely with self-generated feedback. In particular, execu- tion trace feedback offers consistent improvement.
5.2.3 ERROR TYPES FIXED BY SELF-DEBUGGING
To further understand how BELE-DEBUGGING improves performance, we first measure the breakdown accuracies on Spider problems with different difficulty levels, where the difficulty is annotated based on the complexity of the ground truth SQL queries. Figure [bddemossttssss that the improvement achieved by SELF-DEBUGGIN is more significant on hard problems. In particular, on extra hard problems, DELF-DERUGGING increases the accuracy by 9%. Table 4a presents a categorization on error types that are successfully fixed by VELF-DEBUGGING In general, we observe that the initial SQL queries generated by LLMs are usually not completely wrong, but they tend to make small mistakes when the questions require more complex SQL queries, e.g., missing a few WHERE conditions or SELECT columns. In this case, the LLM to identify the discrepancy between the question and the predicted SQL query, resulting in an accuracy boost for more complex tasks. See Figure 10 for a sample fix for an extra hard problem.
with code explanation facilitates
On the other hand, Transcoder and MBPP, 60%-70% successful cases fix the output mismatch error when the initial wrong code is very close to a correct one, as shown in Table 4b Specifically, on Transcoder, over 30% of successful fixes address implementation differences between different programming languages, where the issue is made more obvious with code execution. On MBPP, while 2/3 of the initial programs have problem-specific semantic errors, over 10% of the initial programs can be fixed by switching the order of function arguments and matching the argument types.
6 RELATED WORK
Code repair. Program repair is an area concerned with fixing bugs in code, where several neural networks have been developed to handle different types of bugs (Gupta et al. 2017 Yasunaga &
8

---

Published as a conference paper at ICLR 2024
Table 4: Breakdown on percentages of error types fixed by SELF-DEBUGGING.
(a) Breakdown on Spider with code code-davinee -awi-cii-iiniioe
Error type
%
Wrong WHERE conditions
25.7
(b) Breakdown on Transcoder with 11pt-3.5-turrl
Missing the DISTINCT keyword
17.1
and MBPP with gpt-4.
Wrong JOIN clauses
14.3
Wrong number of SELECT columns
11.4
Error type
Transcoder
MBPP
Wrong INTERSECT/UNION clauses
8.6
Output mismatch
61.9
69.2
Wrong aggregate functions and keywords
5.8
Runtime errors
38.1
30.8
Wrong COUNT columns
5.7
Wrong column selection
5.7
Missing nested conditions
5.7
Liang 2021; Gupta et al. 2020; Wang et al.) 2018 Le et al. 2022). While some methods train
repair models that only take the code as input (uupta et al. 2017| Yasunaga & Liang. 202T). other
approaches incorporate additional information including execution traces (Gupta et al. 2020 Wang
jet al. 2018) as well as compiler and execution errors (Yasunaga& & Liang 2020 Le et al. (2022). Our
SELF-DERUGGINO also uses additional information to aid in code repair, including execution results and self-generated code explanations. In contrast to prior work on training separate code repair model,
utilizes pretrained L.Mss, and teaches the model debug via prompting.
Prompting with feedback. Recent works have shown the great promise of RLHF-trained models to
generate critiques with prompting, which reduces harmful model outputs (Bai et al..
2022
Ganguli
et al.
2023) and improves the performance on some reasoning tasks (Shinn et al.
2023 Madaan
ct al.
2023b Kim et al. 2023 Nair et al., 2023). Reflexion (Shinn et | (003) and RCI (Kim et al.
2023 prompt an LLM powered agent to reflect and improve its actions, which utilize the oracle
reward to determine whether another trial is needed. Self-Refine (Madaan et al.) 2023b) shows that
GPT-3.5 and GPT-4 can iteratively generate feedback and revisions of its outputs to improve the final
output quality on several tasks such as text generation. Finally, DERA (Nair et al. 2023) simulates a dialogue between two agents to improve the model's predictions in the clinical domain. In this work, we focus on code generation applications, and SELF-DEBUGGIN demonstrates the effectiveness of both self- reffeeeneeted feedback and unit test feedback acquired by code execution. In particular, SELF-DEB UGGING instructs the LLM to notice prediction errors via code explanation in the style of rubber duck debugging, which enables the LLM fix prediction errors even when the unit tests not available. We also show that we can teach LLMs to perform SELF-DEBUGGING via few-shot prompting, even if the LLM is not specially tuned for zero-shot debugging.
See Appendix B for an extended discussion of related work.
7 CONCLUSION
In this work, we presented SELF-DERUGGING which enables LLMs debug code generated by themselves. In particular, we demonstrate that DEBFGGNG empowers the model to perform rubber duck debugging, so that the model can identify and fix bugs without human instructions. SELF-DEBUGGING achieves the state-of-the-art performance across several code generation domains, and notably improves sample efficiency.
Our work highlights the promise of improving the coding performance of LLMs by teaching them to debug their own predictions, instead of requiring the model generate the correct code from scratch, instructs the model to understand the code, identify the errors, and follow the error messages to fix the bugs. We consider improving the model's ability conduct all these steps as important future work. In particular, we hypothesize that better code explanation ability leads to better debugging performance. One direction to instruct the model to better describe the high- level semantic meaning of code along with the implementation details in its explanation. Another direction is to include additional debugging information in the model feedback, such as a description of potential bugs. Our preliminary results suggest that model-generater feedback messages about semantic errors do not provide additional benefits on top of line-by-ling -ine code explanation, and future work can explore techniques to predict more informative error messages.
9

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Large language models with strong coding capabilities, combined with SELF-DEBUGGING that allows automated improvement of code quality, may eventually allow for the automation the development and maintenance of large code bases in addition to the automated explanation of their functionalities and vulnerabilities. Although this can provide great productivity gains. on the flip side, the increasing reliance of large language model for real-world programming applications, crucial to systematically analyze its capabilities and limitations before widely utilizing code generated by the model.
code might also open up new vulnerabilities. Therefore, to deploy any
REPRODUCIBILITY STATEMENT
We run all experiments on public benchmarks. We use publicly accessible large language models for evaluation: API access of GPT models available at |htpps: / com/ /pii and StarCoder is an open-source LLM (Li et al. 2023b). See Appendices H I and for full prompts in experiments on Spider, Transcoder and MBPP benchmarks.
ACKNOWLEDDEMMEN
We would like to thank Jacob Austin, Quoc Le, Charles Sutton, Hanjum Dai, and Olivier Bousquet for helpful discussion and feedback.
REFERENCES
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. Multi-lingual evaluation of code generation models. In The Eleventh International Conference on Learning Representations, 2023. URL httpst //openreview. Bo
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models. arXiv preprint urXiv:2108.0725 2021.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional Harmlessness from ai feedback. arXiv preprint arXiv:2212.08077. 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel Bowman, Kyunghyun Cho, and Ethan Perez. Improving code generation by training with natural language feedback. arXiv preprint arXiv:2303. /6749, 2023a.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guant Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference Learning Representations, 2023b. URL
net/ siet/forr umr
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint AXXi:2207.00372. 2021a.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint 2rX.i:2211./2588. 2022.
10

---

Published as a conference paper at ICLR 2024
Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation.
Advances in neural information processing systems, 31, 2018.
Xinyun Chen, Chang Liu, and Dawn Song. inoccutionguidder neural program synthesis. In Interna- tional Conference on Learning Representations, 2019.
Xinyun Chen, Dawn Song, and Yuandong Tian. Latent execution for neural program synthesis beyond domain-specific languages. Advances in Neural Information Processing Systems, 34:22196- 2021b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. In International conference machine learning, pp. 990-998. PMLR, 2017.
Li Dong and Mirella Lapata. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2016.
Ahmed Elgohary. Saghar Hosseini, and Ahmed Hassan Awadallah. Speak your parser: Interactive text-to-SQL with natural language feedback. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilé Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, al. The capacity for moral self- self-correction in large language models. arXiv preprint arXiv:2302.07459 2023.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint 1exXi:2211110444 2022.
Kavi Gupta, Peter Ebert Christensen, Xinyun Chen, and Dawn Song. Synthesize, execute and debug: Learning to repair for neural program synthesis. Advances in Neural Information Processing Systems, 1111766--76955 2020.
Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. Deepfix: Fixing common language errors by deep learning. In Proceedings of the aaai conference on artificial intelligence, 2017.
Dan Hendryeks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding chal- lenge competence with APPS. In Thirty-fifth Conference on Neural Information Processing Systems Dutasets and Benchmarks Track (Round 2), 2021. URL
net/forum? 3dw
Andrew Hunt and David Thomas. The pragmatic programmer: jounneyann to master, 2000.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoye.. Learning neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017.
Srinivasan Iyer, Joannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018
Tushar Khot, Harsh Trivedi, Matthew Finlayson. Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.024000 2022.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint 2rXiv:2303.17493 2023.
11

---

Published as a conference paper at ICLR 2024
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL /otpps://ppereviii ew.
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang. Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. arXiv preprint arXiv:2302. 230.088882.. 2023.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning, Advances in Neural Information Processing Systems, 35:21314-21328, 2022.
Jinyang Li, Binyuan Hui, Reynold Cheng. Bowen Qin, Chenhao Ma. Nan Huo, Fei Huang, Wenyu Du, Luo Si, and Yongbin Li. Graphix-t5: Mixing pre-trained transformers with graph- -awae layers for text-to-sql parsing. arXiv preprint arXiv:2301.07500 2023a.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023b.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 10966662))10921000))))
2022.
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with feedback. arXiv preprint arXiv:2302.0200 02266, 2023.
Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi, Parthasarathy Ranganathan, Yiming Yang, Graham Neubig, and Amir Yazdanbakhsh. Learning performance-improving code edits.
arXiv preprint arXiv:2302.07867. 2023a.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 17651, 2023b.
Varun Nair, Elliot Schumacher, Geoffrey Tso, and Anitha Kannan. Dera: Enhancing large language model completions with dialog-enabled resolving agents. arXiv preprint arXiv:2303. 17071, 2023.
Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-codd generation with execution. arXiv preprint arXiw:2302.08-68, 2023.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang. Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint 1 14, 2021.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv: 2303.00700.. 2023.
Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Malmaud, Jacob Austin, Rishah Singh, and Michele Catasta. Measuring the impact of programming language distribution. arXiv preprint 2302::2300..193... 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL httpsss///pppeeeeeeeeeeefe/ooooe
12

---

Published as a conference paper at ICLR 2024
Baptiste Roziere, Marie Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems, 33: 20601-20611, 2020.
Baptiste Roziere, Jie Zhang, Francois Charton, Mark Harman, Gabriel Synnaeve, and Guillaume Lample. Leveraging automated unit tests for unsupervised code translation. In International Conference Learning Representations, 2022.
id=cmt=6KtR4cA
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL /RRnttss://ooenreiiie wwweeerrvvewwweeeeeeeee net/ ffm
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiuinng models for assisting human evaluators. arXiv preprint urXiv:2206.05802, 2022.
Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: Parsing incrementally for con- strained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11166, 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether tfain-of-thouggee can solve them. arXiv preprint arXiv:2210.09261, 2022.
Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Ke Wang, Rishabh Singh, and Zhendong Su. Dynamic neural program embedding for program repair. In International Conference on Learning Representations, 2018.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL httpe//////perreeeeeteeeeeeeeeee www.nnteooormmmmimmnMMm.
htttps://pperevr/ ew.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In NeurlPS, 2022. URL
nwt/ https::/openrev/ /it.eeeeeeeeee/peoo../eeee....
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations, 2023.
Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao. Large language models are reasoners with velffveriiication. arXiv preprint arXiv:2212.09500 2022.
Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoom. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium Machine Programming, pp. 1-10, 2022.
13

---

Published as a conference paper at ICLR 2024
Ziyu Yao, Yu Su, Huan Sun, and Wen-tau Yih. Model-based interactive semantic parsing: A unified framework and a text-to-SQL case study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP)IJCNIIP), 2019.
Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. International Conference on Machine Learning, pp. 10799-10808. PMLR, 2020.
Michihiro Yasunaga and Percy Liang. Break-it-fix-it: Unsupervised learning for program repair. In International Conference Machine Learning, pp. 11941-11952. PMLR, 2021.
Pengcheng Yin and Graham Neubig, Reranking for neural semantic parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, 2018.
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. CoSQL: A conversational text-to-SQL challenge towards cross cross-domain natural language interfaces to databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), 2019.
Lu Zeng, Sree Hari Krishnan Parthasarathi, and Dilek Hakkani-Tui Tur. N-best hypotheses reranking for text-to-sql systems. arXiv preprint arXiv:2210. 10668, 2022.
Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida I Wang. Coder reviewer reranking for code generation. arXiv preprint arXiv:2211. 2022.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang. Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x, 2023.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le. and Ed H. Chi. Least-to-most most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023. URL http::/://///eeeeiieeeeee
Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. Teaching algorithmic reasoning via in-context learning. arXiv preprint 0tt::2211.0906. 2022.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 2019.
A LIMITATIONS
Our work demonstrates the ability for large language models to improve code generation quality with achieves self-generated feedback and execution results. Despite the fact that performance ultimately significant performance gains on several code generation benchmarks, the bounded by the capability of the underlying large language model, and there is still large room for improvement. In particular, real-world programming tasks within large-scale projects can be more complex and include ambiguous task specifications, where both the model-generated feedback and back- mack-and-fil -and-forth interactions with users are beneficial for improving the coding performance. Meanwhile, how to scale up SELF- BELF-DEBUGGING to diagnose issues in large code bases remains important future direction, where bugs can emerge from interactions between many separate parts of the code.
14

---

Published as a conference paper at ICLR 2024
B EXTENDED RELATED WORK
Language models for code. Recent years have witnessed rapid progress in deep neural networks
for code generation (Devlin et al. 2017: Chen et al. 2019 Yu et al. 2018: Roziere et al.
2020).
While models designed and trained for specialized domains have achieved impressive performance
in various applications such as text-to-code generation (Li et al.) 2023a Wang et al.) 2020 Scholak
et al.
2021 Dong ..aptta| 2016; Iyer et al. (0188) and code translation (Chen et al. 2018 Roziere et al.
2020: 2022). latest work on large language models demonstrate that single pretrained model
can achieve the state-of-the-at performance across a wide variety of coding tasks without specialized
finetuming (Chen et al., (202 Chowdhery et al. 2022, Nijkamp et al. 2023 Zheng et al. 2023 Xu
ct al.
2022 Athiwaratkun et al.) 2023] Orlanski et al. 2023).
Despite showing the remarkable ability to follow natural language instructions, large language models still exhibit limited understanding of code execution (Austin et al. 2021 Li et al., 2022). Specifically, even when the unit tests provided in the problem description. the generated programs may still violate them (Li et al. 2022, Shi et al. 2022 Ni et al. 2023). Therefore, several approaches have been proposed to leverage code execution choose the final prediction from multiple candidates,
such as utilizing or training a language model for reranking (Zhang et al.) 2022 Ni et al. (202)), and
performing selection based on the consensus on unit test execution outputs among samples (Chen et al. 2019; 2021b Roziere et al. 2022, Shi et al., 2022, Li et al. 2022, Chen et al., (023b). In this work, our main focus to utilize and explain code execution for SELF- DEBUGGING, which improves the sample efficiency compared to utilizing execution solely for initial code generation.
Prompting techniques. Several prompting methods have been proposed to unlock latent abilities
of large language models (Wei et al., 2022; Nye et al., [02T Kojima et al. 2022, /Zhou et al.)
|023)Khot Khot et al.] [022] Zhou et al., 2022; Gao et al. [022]] Chen et al. 2022 Weng et al. 2022). Rather than prompting a model to directly produce the desired result, these methods have the model first produce useful intermediate outputs. For example, chain-of- thouhttgtt prompting asks the model to produce intermediate reasoning steps in natural language, which can be accomplished either with few-shot exemplars (Wei et al.) 2022) or in a zero-shot manner (Kojima et al. (022)) Meanwhile, several prompting techniques explicitly direct the model break down a problem into subproblems (Zhou et al., [023] Khot et al. 2022). ) For example, decomposed prompting (Khot jet al. 2022) delegates subproblems to other LLM instances that are prompted to specialize in specific tasks. Self-verification (Weng et al. (022) computes verification scores to select the final response given multiple candidate answers, where the backward verification process primarily designed for arithmetic reasoning tasks. Our prompting format of code explanation is relevant in spirit to to-hout prompting, as the line-by-line code explanation in natural language facilitates explenatiion
analysis of the code that is useful for the debugging task. On the other hand, also decomposes the debugging process into several stages, and triggers separate prompts for code explanation and feedback generation.
Training with feedback. Training with feedback to improve the outputs of large language models, both in terms of correctness and alignment with human preferences, is active research direction nowadays (Ziegler et al., 2019: Korbak et al. 2023) Ganguli et al., 2023 Bai et al., (022). One
popular technique is reinforcement learning from human feedback (RLHF) (Ziellrr et al. 2019
Ouyang et al. 2022). and RLHF-trained models have demonstrated the ability to avoid harmful outputs when instructed to do so in the prompt (Ganguli al. 2023). Constitutional AI (Bai et al. 2022) introduces another path toward training harmless models, where they use the pretrained model itself to create automated feedback for both supervised learning and RLHF: for the former, a set of principles are used to guide a language model in creating revisions of its own responses that it is then trained on, and for the latter the same principles are used to prompt a separate model for the feedback needed for RLHF
Another line of work trains a language model to refine the initial model outputs based on external feedback on prediction quality (Welleck et al. 2023 Liu et al. 2023), which improves the performance on several natural language and reasoning tasks. For code generation, a number of works have trained
models to perform code optimization (Madaan et al. 2023a), interact with users for multi-turn code generation (Yu et al.) 2219)) Yao et al. Nijkamp et al. (202). and fix generated programs based
on human feedback (Iyer al. 2017
Elgohary al. 2020
Chen et al. 2023a Le al. 2022 On
15

---

Published as a conference paper at ICLR 2024
the other hand, SELL--DBUGGINGG enables the model generate feedback messages its test time, and does not require extra training.
C SELF-DEBUGGING WITH INITIAL SAMPLES FROM DIFFERENT LLMS
Table 5: Results of SHLF-DERUGGING using gpt-3. 3---3.5-tur 5-tubb (GPT-3.5) and code code- davinci- 002 (Codex) (a) Spider; (b) Transcoder and (c) MBPP. The baseline results the highest accuracies before SELF-DEBE DELL-DEBUGGING, which are obtained by Codex for Spider, and GPT-3.5 for Transcoder and MBPP.
TransCoder
Codex
GPT-3.5
MBPP
Codex
GPT-3.5
Spider
Codex
GPT-3.5
Baseline
81,3
Baseline
89.1
Baseline
67.6
Simple
91.6
Simple
72.4
Simple
81.3
81.7
92.1
92.7
UT
73.2
72.2
+Expl.
84.1
83.0
UT + Expl.
92.7
92.7
Expl.
73.6
74.2
(a)
(b)
(c)
In Table 55 we compare the results BELL-DEBUGGING using Codex and GPT-3.5 respectively. For each benchmark, the baseline accuracy presents the best code generation performance obtained by Codex and GPT-3.5. On TransCoder and MBPP, the baseline uses GPT-3.5 with zero-shot prompting. On Spider, we observe that the performance of GPT-3.5 is significantly worse than Codex. For example, when using greedy decoding, GPT-3.5 achieves an accuracy of 59.9% with zero-shot prompting, and 71.1% with few-shot prompting, which is over 6% lower than Codex. On Spider, we utilize the initial code generation from Codex. We present GPT-3.5 SELF-DEBUGGING results using zero-shot prompting, Specifically, we used zero-shot DELF-DEBUGGING prompts which removed exemplars and adapted the instructions from few-shot prompts fit the conversation format of GPT-3.5. For example, the instruction included in the simple feedback is "Does the SQL match the question? If not, generate the fixed SQL." " for Spider.
observe that the performance with few-shot exemplars is similar.
From Tables 5b and 5c, we show that while GPT-3.5 notably outperforms Codex on generating initial code in Python, the DEBUGGEGGGG performance of Codex on par with GPT-3.5 on Transcoder. It also performs close to GPT-3.5 on MBPP, and outperforms it for both Simple and UT. In Table [a we observe that Codex also outperforms GPT-3.5 SELF- SELF-DEBUGGING for text-to-SQL generation, and code explanation again improves the performance for GPT-3.5. These results demonstrate the prompt with few-shot exemplars, while suggesting that Codex is better than GPT-3.5 at learning from few-shot exemplars with long context.
effectiveness of our SELF-DEBUGGE
D TRANSCODER COMPARISON TO BASELINE MODELS
Table compares SELF-DERUGGING to baseline models without debugging.
Table 6: TransCoder dataset (Roziere al. 2020 for C++ to Python translation.
TransCoder (pass @ 1)
w/o debugging
TransCoder
44.5
PaLM
51.8
PaLM-Coder
55.1
SEL.F- DEBUGGIN (this work)
Codex
80.4
UT + Expl.
92.5
16

---

Published as a conference paper at ICLR 2024
E MORE DISCUSSION OF SELF-DEBUGGING RESULTS
E.1 CODE TRANSLATION
For generating initial Python translation, we apply the same few-shot prompt for TransCoder (Chowdhery al. 2022). which consists of 3 exemplars (Appendix [.T] From Figure 5a again observe that the major improvement comes from the first debugging turn. Specifically, a single debugging turn with the full feedback improves over the greedy decoding accuracy by around 12%. Compared to Figure 5b, applying WELF-DEBUGGIN to greedy decoding outperforms the baseline accuracy with 5 samples, and is close the baseline accuracy with 10 samples.
Meanwhile, incorporating both unit test execution and code explanation improves the debugging performance, and we present some examples in Figures and 12 in Appendix G In addition, we demonstrate that leveraging code explanation alone without SELF-DEBUGGIAS also provides consistent performance gain of 3% for different numbers of samples, as shown in Figure 5b
TransCoder Accuracy (pass@1)
UT+Expl.
98.0
97.5
UT
97.5
TransCoder Accuracy
95.0
Simple
95.0
94.3
Codex+Expl Expl.
91.
50.0
Codex
PaLM-Coder pass@25
87.5
88.4
87.5
93.0
85.0
85.0
60.4
# Turns
# Samples
(a)
(b)
Figure 5: Ablation studies TransCoder. (a) The accuracy of SELF-DEBUGGING prompts with different numbers of debugging turns. (b) The accuracy with different numbers of initial samples. Note that Expe only prompts the model to generate explanations for the input C++ code, and does not perform SELF-DEBUGGING.
E.2 TEXT-TO-PYTION GENERATION
For initial Python code generation MBPP, apply the same 3-shot prompt (Ni al. 2023) With greedy decoding, SELFF-DERGGGNN increases the accuracy by over 8%, and leveraging both unit test execution and code explanation again consistently improves the performance.
We observe that the improvement using code explanation is less significant on MBPP. First, due to the availability of unit tests, it is much easier recognize wrong predictions from unit test feedback. Second, due to the context length limit, the number of exemplars in the prompt for SELF-DEBUGGING smaller when adding code explanation in the feedback message. Specifically, prompts with both simple and unit test feedback messages include 6 exemplars, while the prompt with both the unit test execution and code explanation has exemplars. Third. find that code explanation plays smaller role when the initial predicted code is far from the correct solution, in which case the model simply generates a new program and does not rely on code snippets in the initial prediction. See Figure [3] Appendix G for an example where SELF-DEBUGGING makes major changes to the initial code.
F EXAMPLES FOR SPIDER AND TRANSCODER
Figure and illustrate the feedback formats for Spider and Transcoder. The formats of MBPP are 9 for examples in the Spider and TransCoder benchmarks.
similar TransCoder. See Figures Aaatres@andd@1
17

---

Published as a conference paper at ICLR 2024
Step 1: Question explanation
Infer the type of the question.
[Question]
Which customers have
and 'Shipped" order status? the customer names.
[Question Explanation]
The question returns customer names who have both Road" and "Shipped" order status. So the question returns column.
customer names" returss
Step 2: Code explanation
Summarize the return type of the exccution.
[SQL]
SELECT ustomers.custure customer. name FROM customers JOIN orders ON cuskomers.csttomer id orders.custonee id
WHERE orders.order status "On Rond" OR orders.order status "Shipped"
Execution:
George
[SQL Explanation]
execution SQL query above would return table with column.
The first column, "eustemers.cusionee customer name" would contain the customer names.
With "customers JOIN orders", the table would contain the data about customers with orders.
With "WHERF orders order status - "On Road' OR orders status - "Shippod"" the table filters the records to only include customers who order status Road" or "Shipped". So the SQL query
table with column, the customer
who have the order "On Road" "Shipped".
Step 3: Self-Debuging with explanation
[Question]
[Explanation summary]
As in your explanation, the SQL query returns table with 1 column, the customer names who have the order status "On Road" "Shippod". The question returns column, the customer names who have both "On Road" and "Shipped" as order status. [F [Foedback]
So the SQL prediction above wrong, Please fix the SQL..
[Revised SQL #1
SELECT customers.customer customer name FROM customers JOIN orders ON customers.customer id custome. id WHERE orders.order status "On Road" AND unders.order statur status "Shipped"
[Explanation summary #1]
in your explanation, SQL query returns an empty table, because "WHERE" clause contains mutually exchusive conditions.
question retums column, the customer who have order of both "On Road" and "Shipped".
[Feedback #1]
So the SQL prediction above wrong. Please the [Revised SQL #2
customer orders customer id WHERE orders.order starus = "On Road" INTERSECT SELECT customers. customer name FROM custemers JOIN orders ON
SELECT
FROM
JOIN orders
cnstomers.customme id unders.customer id WHERF orders.onde status "Shipped" [Explanation summary #2]
As in your explanation, the SQL query returns table with I column, the customer namess who have the order stalus "Ou Road" and "Shipped", The question returns column, the customer names who have order status both "On Road" and "Shipped", the question returns column.
[Feedbuck #2] So SQL prediction above correct!
Figure 6: An example of SELUG-EBUGGING prompting for text-to-SQL generation. The model required predict the code, explanation and feedback. When the returned table has more than rows, only the first 2 rows are included in the prompt. Database information is omitted in the figure for clarity, and we present the full prompts in Appendix H
G EXAMPLES OF MODEL PREDICTIONS WITH SELF-DEBUGGING
See Figures TO and examples of model predictions with SELF-DEBUGGING.
18

---

Published as a conference paper at ICLR 2024
Simple Feedback
Unit Test (UT) Feedback Below are C++ programs with incorrect Python translations.
Unit Test Explanation (+Expl.) Below are C++ programs with incorrect Python translations. Explain the original code, then explain translations line by line and correct them using the provided feedback.
Below are C++ programs with incorrect Python translations.
Correct the translations using the provided feedback.
Correct the translations using the provided feedbuck.
[+++
[Original Python
Original Python]
Simple Feedback]
[UT Feedback]
[C++
RRevised Python #1
Revised Python #1
CC+ Explanation]
SSmple Feedback]
[UT Feedbuck]
Original Python]
Revised Python #2
Revised Python #2
[Python Explanation
Feedback)
Revised Python #1
[UT Feedback
Reevised Python
(C+++
[C++ Explanation]
count.. woount.traillin.zerrroe http://..rroo..fttrraaa..tttt int
The code is un implementation counting the number of
trailing zcroes in the factorial of a given number.
1;
int count 8;
return count;
The vuriable count is used to record the number of truiling zeroes.
OOriginal Python]
for int 5; 1
def count_treiling. www.tterriingggerroefeeoooee fet.rrro_eetttrillnnmerrrrrr)
The loop iterates until the quotient of the given number and i (the divisar) is greater than or equal to 1. count +- 1:
ent =
while [In / 1) 1): ont (ent + (n i))
When the loop execnted, the quotient of the given number and i will added to the variable count.
1 * return ent
[Simple Feedback]
return count:
The above Python translation does not do same thing as
Finally, the function returns the number of trailing zeroes. [Python Explanation]
the C++ code. Correct the Python translation.
[UT Feedback]
The code is implementation counting the number of trailing zeroes in the factorial of given number. ent 8:
The Python translation does not the same thing C+-
code. These are the results failed unit tests
The variable ent used record number trailing zeroes.
whether Python translation's outputs match C++
program's outputs:
i 5 variable used as divisor. while ((n 1) se 1):
Failed: assert count trailing zoroes factorial number(9)
Actual Result:
Correct the translation.
The loop iterates until the quotient of the given number and divisor) is greater than or equal 1.
[eevsed Python Win]
http:_taa.innggooeeeeeotootoos ent = whila ((n / 1) >- 1): ont = (ent + (n i)) 1 * return ont
ent (ent (a 5))
loop executed, quotient of given number and i will added to the vuriable ent.
i
The variable return ent
multiplied by
Finally, the function returns the number trailing zeroes.
Figure 7: Examples of SELF-DERUGGING prompts for code translation. Left-aligned blocks model predictions, and right-aligned blocks contain the input C++ code and feedback messages based on code execution. The full prompts are in Appendix [
19

---

Published as a conference paper at ICLR 2024
Problem description
CREATE TABLE customers (
customer_ id number
customer_ text
customer customer_details ds text.
primary key ( customer_id )
insert into customers (customer_ id, customer_name customer_details) values (1, 'Savannah', 'rerum')
CREATE TABLE orders order_id number customer_th number order_status text
date_order_pllaee time
order_de urder_detaills text primary key order_id foreign key ( customer_id id references customers customer_id
insert into orders (order_ia) id, customer_ id, order_status, date_order_ order_des (rder_details) values (1, 2, 'Shipped', '2009-82-20 15:26:19', NULL);
Question: Which customers have both Road" and "Shipped" as order status? List the customer names.
Output SQL
SQL: SELECT eestomers.custtmmee __ame customers orders customers. customer_in idd orders. customer ordersstturrsssttuu stsaussttusss "On Road"
WHERE orders.order
INTERSECT SELECT customers. customer _ame FROM customers JOIN orders ON id orders. orders.custommr__tt
WHERE orders. "Shipped"
Figure 8: An exemplar for text-to-SQL generation. The problem is taken from the Spider dataset (Yu et al. 2018). The problem description contains the database schema, and the model is required to predict the query. The prompt includes the contents of one row from each table.
20

---

Published as a conference paper at ICLR 2024
C++ Program
Python Program
string caesar_cipher ( string text, int ) {
def caesar_ ceesar_ciphertteer (text, s)
result for in range(len(text)): char text[i] if char. isuppes ()
string result ""; for int i = 8; text length )); ++ {
result += chr (nr(((((orddcha)
if isupper text i result char { int text [
65) % 26) + 65))
else:
26 65 );
result chr (( (((( ((hor
else result += char int ( text i] 97 ) % 26 + 97 );
+ s) 97) % 26) + 97))
return result
return result;
Unit Tests
assert caesar cipher ( 35225904', 2). == "ikhhkofj' (8 unit tests omi ote)
assert caesar_cipher 11',, 93) 'tt'
Figure 9; An example from the TransCoder dataset. The problem description contains the C++ program and unit tests, and the model is required predict the Python program.
21

---

Published as a conference paper at ICLR 2024
Original SQL
Question: Give the names of countries with English and French as official languages.
SELECT country. name FROM country JOIN countrylanguage ON country. code countrylangugee..ouutty.dddd
WHERE countrylanguage language "English"
INTERSECT SELECT country. name FROM country
JOIN countrylanguage ON country.com country htrperyyllaguag...cuunrttttee
WHERE countryl Language language "French"
Code explanation
The execution of the SQL query above would return a table with 1 column. The first column, "'contry.name name" would contain the country name. With 'country JOIN countrylanguage", the table would contain the data about countries and languages.
With "WHERE langguu records to only include countries with the language "English"' With "WHERE countrylanguage language 'French"' the table filters the records only include countries with the language "French''. With "INTERSECT"", the table would be filtered only include countries with both "English'' and "French" as languages.
language 'English"", the table filters the
languee
So the SQL query returns a table with 1 column, the country name of countries with both "English"' and "French" languages.
Self-debugging with explanation
As in your explanation, the SQL query returns. table with 1 column, the country name of countries with both "English") and "French'' languages. The question returns names of countries with English and French offi cial languages. So the SQL prediction above is wrong. Please fix the SQL.
SELECT country name FROM country JOIN countrylanguage ON country.code haaperylllnuugg..ouuntytyy
WHERE countrylanguage. language "English" "T"
AND wwunttyylanguge.issofoooo
INTERSECT SELECT country. name FROM country JOIN countrylanguage ON country.code
= suuntrrylanggag..couutyyyyy WHERE countrylanguage. language "French" AND countrylanguage. isofficial "T"
Figure 10: An example the Spider benchmark, where SELF-DEBUGGING fixes the prediction for an extra hard problem. Database information is omitted in the figure for clarity.
22

---

Published as a conference paper at ICLR 2024
C++
Original Python
int remainder_ numbbrrrrrs ( string
def remaiinder T__large_number nwwberubbrssu. (num):
num )
series [1, 3, 2, -1, -3, -2] series_index
int series [
1, 3, 2, 1, 3, };
result =
int series_ index 0;
for in range ( (len(num) 1), -1,
int result for ( int i num size ( 1; >= 8; i --
0;
=
digit (num[i] '0")
result += (digit * series
int digit num [1 '0"; result += digit series
peries_indexll
series_index (nerries_index
series_index ];
1) 6)
series_index = series_index
result %= 7
1
if (result 0):
result == 7;
result ((result 7) % 7)
return result
if result < (0) result =
result 7; return result;
Self self-debugging with UT feedback
The Python translation does not do the same thing as the C++ code. These are the results of one failed unit test that tests whether the Python translation's outputs match the C++ program's outputs:
Failed: assert
Actual Result: Python runtime error: Traceback (most recent call last):
File <filename>, line 29, in <module>
print _______________
File <filename?, line 22, in remainder_ Large_numbers
digit (num[i] '0¹)
TypeError: unsupported operand type(s) for -: 'str' and 'str' Correct the Python translation.
def [amainder_7__arreee Lar____aaaeee_uubbrrrrrrr seteennmmersseeeeee (num)
series [1, 3, 2, -1, -3, -2] series_indes
result
for i in (anne(()e 1), -1, -1):
digit (num([i)))) ('d)')))
result += (digit series series(series, (series_ index]) series_index ((series_inde index + 1) % 6) result $=
(result 0):
result ((result: 7) 7)
return result
Figure 11: An example where SELF-DEBUGGING with unit test feedback fixes the code translation error, while the simple feedback fails.
23

---

Published as a conference paper at ICLR 2024
C++
Baseline pass @ 25
void function, copy.oopyiinneo char sl [ 1, char
def function_copy_strrinn s2, index =8)
], int index
{
$2 index s1 index 1; if s1 [ index ] == '10' return; Kunction_copy_strin ( s1, s2, index
s2[index] s1[index] if (s1[index] '(e)):
return None
s2, index 1))
+ Expl.
The code is implementation iterative function copying given string. The character at the given index the first string will be copied the same index of the second string.
If the character at the given index in the first string is '(0', which means the end of the string, the function will be returned. Otherwise, the next character the first string will be copied the second string.
def cunction_copy_stringg s2, idx=0); s2[idx] [idx]
[(si[idx] '(0)) or ((1en(s1) 1) idx)) return Lumetiion_coy.ssrinngt.. s2, (idx 1))
An example where adding code explanation results in the correct code translation, while the baseline prompt does not. Specifically, the translation with code explanation correctly checks for the end of string.
Figure 12:
Original Python code
Prediction after self-debugging
assert encode_ list
def setade_listtnuure
1], [1, 2], [1, 31, [1, 4], [1, 4.3], [1, 5], [1, 1]
1,,,,,,,,,
count = 1
for in range len(nums)) if nums[i] == nums(i-1): count else:
Write function reflect the length encoding from list.
p.ppedd nums [1-1]])
count = 1 append (( ([count, nums[-1]])
def encode encode_list (ws.ssssssss)s res [] for in nums:
return
if i not in res:
apsend [numms i])
return res
Figure 13: An example on MBPP where the prediction after SELF-DEBUGGIN is very different from the initial code.
24

---

Published as a conference paper at ICLR 2024
H TEXT-TO-SQL GENERATION
H.1 BASELINE PROMPT (5-SHOT)
CREATE TABLE department
department_10 number
text
creation text
ranking number
sedget_in_billions billions number budget_in,
num eoyeeleleees number primary key ( department_id
insert into department teent_ttent_tne name, creation, ranking, budget_ing deddet__i_billioon nun_employees) values (1, 'State', '1789', '1' 9. 019660000998000008822222 30265. 919728S9.99999999999999999999999
CREATE TABLE head head_id id number name text
born_state text
age number
primary key ( head_id
insert into head (head_ id, name, born_state, age) values (1, (Tgee Woods','
Alabama', .er.ar,6669999999 99999999999999999999
CREATE TABLE management
department_ id number
head_id id number
temporary_acting text
primary key ( department_id
foreign key ( head_id ) references head ( head_id
foreign key ( department_id id references department ( department_ji id )
insert into management (department_ie id, head_id, id, temporary.acting) values (2,5, 'Yes")
Translate following question SQL.
Question: In which year were most departments established?
SQL: SELECT creation FROM department GROUP creation ORDER COUNT(*) DESC LIMIT 1
CREATE TABLE city
sity_id number official_name text. status text orea_km_2 number
population number
census_ranking text
primary key ( ciity_id
insert into city (city_id, official_name, status, area area_km_2. km_2, population, census rrakkigg values (1, Grand Falls/Grand-Sauls 'Town' '18.06' '5706', 636 of 5008*)
CREATE TABLE farm farm_id number year number
25

---

Published as a conference paper at ICLR 2024
total_horses number
trrking_horses number
total_cattle number,
oxen number
bulls number
cows number
pigs number
sheep_and_goate number
primary key ( farm_id
insert into farm (farm_id id, year, total_horses, warking_horses oxen, bulls, cows, pīgs, sheep_and_goad values (1, '1927' '5856.5', '3980.1 8374.5', '805.5', '316'', '3852.1', 74412.4', '7956.3')
CREATE TABLE farm__copetition competition_it id number
year number
theme text
host_city_ft number
hosts text
primary key ( competition_id )
foreign key ( host_city__dd ) references city ( city_id )
insert into Barm_copetitton (competition_ year, theme, host_city_ id,, hosts) values (1, '213'', 'Carnival N is back! ,1, 'Miley Cyrus Jared Leto and Karen Mok')
CREATE TABLE competition_recorrd (
competition_1 id number
farm_id id number
rank number
primary key ( competition id
foreign key ( farm_id references farm ( farm_id]
foreign key ( competition_id ) references farm_competition competition_ic )
)
insert into competition_recat (competition_id, farm_id, rank) values (1,8,1)
Translate the following question into SQL.
Question: Show the status of the city that has hosted the greatest number competitions.
SQL: SELECT city.status FROM city JOIN farm_competi tion ON city.city.t farm_compet' hist.hostttee.... coot_city_tee GROUP BY farm_compet tion. ORDER BY COUNT(+) DESC LIMIT 1
CREATE TABLE customers ( customer_id number customer_name text mnstomer_detaile text primary key ( customer_id
insert into customers (customer_iae id, customer _name, customer (estomer_details) values (1, 'Savannah', 'rerum")
CREATE TABLE invoices invoice_number number invoice_ datte time iivoice_detaile primary key ( invoice_number )
26

---

Published as a conference paper at ICLR 2024
) insert into invoices (invoice numbbr, invoice_ date, aewoice_detaille values (1, '1989-09-8 16:03:05', 'vitae")
CREATE TABLE orders order id number customer_id number order order_status text date_orde site_order_placee time
order_detarle text
primary key ( order_id ), foreïgn key ( customer_id references customers customer_io
)
insert into orders (order_id, customer_ id, arder_status, bate_order_placee
order_details) values (1, 2, 'Shipped', 22089-02-21 15:26:19', NULL);
CREATE TABLE products product_id number
product_name text
product_detafls text, primary key ( product_id
)
insert into products (product_id, product_name, product_details) values (1,
'fodd', NULL);
CREATE TABLE rrder_items ( prder_item_it number
product_id number
order_id number
order_item_steee
text
order_item_detaile text
primary key ( (onder_ite_id)
foreign key ( (product_id references product product_ id
foreign key ( order_id references orders { order_id
)
insert into order_items ponder_item, id, product_ia id, order order_id, order order_item_stte sir__itte_stttss, ciste__tte_dettiile values (1, 4, 6, FFinish' NULL).
CREATE TABLE shipments ( shipment_id number
order_id number
invoice_ jumbeer__mmbrrreeee number
shipmentt_trackinggnnmm text
shipment_date time
other_shipment_detaal
text
primary key ( (shipment_id)
foreign key ( mnvoice_numberr references invoices innorr
foreign key ( order_id references order order_id )
insert into shipments (shipment_ id, order_id, arroice_number, wwwpenttttrraiiinnnnnnnee 5, 13, 13452', 11983-08-15 22:34:11', NULL).
shipment_date _date, (eeeedshipmen values (1,
CREATE TABLE shipment_itene items shipment_id id number
order_item_id number
primary key (
foreign key ( shipment_id references shipments ( shipment_id )
foreign key ( order_item_id references order_items order_item___ )
)
insert into shipment_iten items (shipment_id id, order_item_tte values (11, 12)
Translate the following question into SQL.
27

---

Published as a conference paper at ICLR 2024
Question: Which customers have both "On Road" and "Shipped" order status? List the customer names.
SQL: SELECT customers www.ccuutttstccceeecc FROM customers JOIN orders customers. customer_ id orders. customer_ WHERE INTERSECT SELECT customers. www.ccuutttt__aaeeeee FROM customers JOIN orders ON customers. custemer- id orders. WHERE orders.order_status = Shipped"
"On Road"
CREATE TABLE station (
id number
text lat number
long number dock_count number
city text installation datt text
primary key ( id
insert into station (id, name, lat, long, dock_count, city, installation_date) values (2, San Jose Diridon Caltrain Station'
'San Jose', ''8/6/2013')
CREATE TABLE status
station_ id number
Mikes_avaaile lable number
docks_availabl number time text
primary key (
foreign key ( statioo_id references station id
insert into status (station_ id, bikes_ Backs_availablle time values (3,12,3,'2011506-1 22155066-22 12:46:02');
CREATE TABLE trip 1d number duration number
start_date text
start_station_nant text
start_station_id number end_date text and_station_nant _nmi text
end. end_station_1 id number
bike_id number subscription_tyye typpe text,
zip_code number
primary key ( id
insert into trip (id, duration, start_date, start_station, _atioon___mmm_______ start_state seart_station_id, end_date, end_station_nant name, end_station_ id, bike_id subscription_tyye zip_code) values (900504,394 384, '8/21/2015 3//////2211 17:03', 'Howard at 2nd', ,6,, 98/21/2015 17:10', 'San Francisco Caltrain 2 (339 Townsend) ,,69,454, 'Subscriber' ,94041);
Translate the following question into SQL.
Question: What average longitude of stations that had bike availability more than 10?
SQL: SELECT AVG(long) FROM station WHERE NOT (SELECT station_id FROM status GROUP BY station_id HAVING MAX (ix(iiies_avaiilable)) > 10)
28

---

Published as a conference paper at ICLR 2024
CREATE TABLE allergy_type allergy text
allergytype text primary key ( (allergy) )
)
insert into alyergy_type (allergy, allergytype) values "Eggs', 'food' )
CREATE TABLE has_allergy (
stuid number
allergy text
foreign key (allergy ) references allergy_type allergy )
foreign key ( stuid references student ( stuid ) )
insert into has_allergy (stuid, allergy) values 1001, 'Cat'
CREATE TABLE student (
stuid number
Iname text
fname text
age number
sex text
major number
advisor number
city_code text
primary key ( stuid )
)
insert into student (stuid, Iname, fname, age, sex, major, advisor,
city_code) values 1081, 'Smith', 'Linda', , 'F', 600, 1121, 'BAL))
Translate following question SQL.
Question: Find the number of students who are older than 18 and do not have allergy to either food animal.
SQL: SELECT COUNT(+) FROM student WHERE age 18 AND stuid NOT IN (SELECT has_allergy.stun FROM aa__alergy JOIN allergy_type ON has_allergy.alle = allergy_ wtwprryyytyye..lleeeee allergy WHERE bylergyytype.alleeyyyy. "food" OR aelergy.bype.alllrryyyyy = "animal")
<insert database schemas and the new question here>
H.2
SIMPLE FREDBACK PROMPT (9-SHOT)
CREATE TABLE department
department_ic id number
text creation text
ranking number
budget_in_billions number nun_employees number
primary key ( department_id
insert into department (depar tmenttid, name, creation, ranking, budget_in_biltee nun_employees) values (1, 'State', '1789' '1'
719666000000990088856,,, 38265. .28...999999999999999)))))
CREATE TABLE head
29

---

Published as a conference paper at ICLR 2024
head_id id number text
born_state text
age number
primary key ( head_id
insert into head (head_ id, name, dorn_state, age) values (1, 'Tiger Woods', Alabama' ,66 99999999999999988
CREATE TABLE management department. id number
head_id id number
temporary_acting text
primary key ( department_idd
foreign key ( head_id ) references head head_id
foreign key ( department_id id references department dedeprtment_id id )
insert into management
head_id, temporary.acting) values
(2,5, "Yss)
Translate the following question into SQL.
Question: which year most departments established?
SQL: SELECT creation, COUNT(*) FROM department GROUP BY creation ORDER BY (*UNT(*) DESC LIMIT
Feedback:
The SQL prediction above is wrong. Please fix the SQL.
SQL: SELECT creation FROM department GROUP BY creation ORDER BY COUNT(*)
DESC LIMIT
Feedback: The SQL prediction above is correct!
CREATE TABLE people
people_id number
sex text
name text
height number
weight number
primary key ( peeolelid
insert into people (people_id id, sex, name, height, weight) values (1, M, Hubert Henno, 188, 83);
Translate the following question into SQL.
Question:
Find the average and minimum weight for each gender.
SQL: SELECT AVG (weight)) (weiigh)) FROM people GROUP BY sex
Feedback:
The SQL prediction above is correct!
CREATE TABLE customers (
customer_ie number
customer_name text
customer customer_detarils letaills text
primary key ( customer_id
insert into customers (customer_iee id, customer_ _name, customer_detafis) values (1, 'Savannah', 'rerum")
30

---

Published as a conference paper at ICLR 2024
CREATE TABLE invoices invoice__nmmt number number invoice_date time
invoice_ denricedettilss text primary key ( iinvoice_number
insert into invoices (invoice, nnnvor,,_number, invoice_dater invoice_ values (1, '1989-89-03 16:03:05', 'vitae") ;
CREATE TABLE orders order number customer_id number order_stature text
@ate_order_place time
order_detaril text
primary key ( order_id ),
foreign key ( customer_id references customers ( customer_t
insert into orders (order_id, _id, customer_ id, order_status, date_order_placee
order_details) values (1, 2, 'Shipped', '2009 22009-02-21 15:26:19', NULL);
CREATE TABLE products productjid number
product_name text
product_detaiil text, primary key ( product_id
)
insert into products (product_id, product_name, product_details) values (1,
'food", NULL);
CREATE TABLE order_iten ( order_iten id number
product_id number
order_id number
order_item_status text
order_item_detaiil text
primary key (
foreign key ( product__d ) references product product_is id
foreign key ( order_id) references orders ( (rder_id
)
insert into order_items forder_item_11 id, product_id, id, order_id id, carrr_ite_sttttu_e pedder_tem__ee ( values (1, 4, 6, 'Finish', NULL)
CREATE TABLE shipments (
shipment_id number
order_id number
invoice_number sumriere_mmmerreeeee number
shipment__trakking_nnmtt text
shipment_date time
bitte__shipeen_detaaii text
primary key ( (shippent_idd)
foreign key ( (invoicc__nnne "rvoricenuuber') references invoices hnnobr
foreïgn key ( order_id references order order_id )
)
insert into shipments (shipment_ id, order_id, invoice number, shipment_tracke t
number, shipment_date
www.sshipmen__dettiil.. values (1,
5, 13, 13452', , '1983-08-13 22:34:11', NULL)
CREATE TABLE shipment_itee ttms ( shipment_id id number order_item_ie number primary key (
foreign key ( shipment_id references shipments ( shipment_ id
31

---

Published as a conference paper at ICLR 2024
foreign key ( order_item, references order_items order_item__tee
)
insert shipment_itene _ttmss (shipment_id, order_item_ioe values (11, 12)
Translate the following question into SQL.
Question: Which customers have both "On Road" and "Shipped" as order status? List the customer names.
SQL: SELECT customers. customer _nae FROM customers JOIN orders customers. customer id orders. customer_id WHERE orders. order_status "On Road" rrders.order _stttus "Shipped"
Feedback: The SQL prediction above is wrong. Please fix the SQL.
SQL: SELECT customers. cwttsmsrrstsssrrcccc _amee FROM customers JOIN orders ON customers. customer_ iid = orders. customer_ id WHERE orders. order_statuss status "On Road"
INTERSECT SELECT customers.
customers.customer... www.cccutttternoeeee FROM customers JOIN orders ON etomers.ccstooes. Shipped"
orders.custtmer_itt id WHERE orders.order_status
Feedback: The SQL prediction above is correct!
CREATE TABLE flight ( flno number
origin text
destination text
aid number
primary key ( flno
insert into flight (flno, origin, destination, aid) values (99.0, Los Angeles, Washington D.C., 1);
Translate the following question into SQL.
Question: Show origins of all flights with destination Honolulu.
SQL: SELECT origin FROM flight WHERE destination "HONO"
Feedback:
The SQL prediction above is wrong. Please fix the SQL.
SQL: SELECT origin FROM flight WHERE destination "Honolulu"
Feedback:
The prediction above is correct!
CREATE TABLE station (
id number name text
lat number
long number dock_count number
city text installation_date text primary key ( idd
insert into station (id, name, lat, long, dock_counts city, installation (atattioodddee))) values (2, San Jose Diridon Caltrain Station, 37. 2123777211222017110000011166661
San Jose, 8/6/2013);
CREATE TABLE status station_id number
32

---

Published as a conference paper at ICLR 2024
bikes_ aaaaalaaalllll number aackslavaillable number time text
primary key ( foreign key ( station_id ) references station id
)
insert into status (station_ie id, bikes_ availaae,,,,,,,, bocks_available, time values (3,12,3, 2015-06-02 12:46:02);
Translate the following question into SQL.
Question: What is the average longitude of stations that never had bike availability than 10?
SQL: SELECT AVG(long) FROM station WHERE id IN (SELECT station_id id FROM status WHERE bikes_available 18)
Feedback: The SQL prediction above is wrong. Please fix the SQL.
SQL: SELECT AVG(nong) FROM station WHERE NOT IN (SELECT station_id FROM status WHERE bikes_availlable > 10)
Feedback: The SQL prediction above is correct!
CREATE TABLE host
host_id number name text
nationality text.
age number
primary key ( hhsttii
insert into host (host_id, name, nationality, age) values (1, "Austin Daye"," " United States", ;43);
Translate the following question into SQL.
Question: Show the and the nationality of the oldest host.
SQL: SELECT name, nationality FROM host WHERE age (SELECT MIN(age) FROM host)
Feedback:
The SQL prediction above is wrong. Please fix the SQL.
SQL: SELECT name, nationality FROM host ORDER BY age DESC LIMIT 1
Feedback: The SQL prediction above is correct!
CREATE TABLE city (
city_id number official_name text
status text
brea_km_2 number
population number
rensus_ranking text
primary key ( (city_id)
insert into city (city_id, official_name, status, prea_km_2 population, census_ranking) values (1, Grand Falls/Grand-Sault, 'Town', '18.66,, '576'' 636 of 5608*)
CREATE TABLE farm
33

---

Published as a conference paper at ICLR 2024
farm_id number
year number
total_horses number working_horses number
total_cattle number. oxen number
bulls number
cows number
pigs number
sheep_and_goats number
primary key ( farm_id
insert into farm (farm_id, year, lotal_horses, working_horses, total_cattle oxen, bulls, cows, pigs, @meep_and_goate values (1, '192'' '5856.5" 5' '3900.1 '8374.5', '805.5', '31.6', '3852.1', '4412.4', '7956.3'
CREATE TABLE com__cometiiton competi competition_id number
year number
theme text
host_city_id number
hosts text
primary key ( competition_id
foreign key ( host_city_id references city ( city_id )
insert into carm_competition (competition id, year, theme, host_city -id hosts) values (1, '201'', 'Carnival back! ,,,, 'Miley Cyrus Jared Leto and Karen Mok')
CREATE TABLE competition_record ( competition_1 number
farm_id id number
rank number
primary key ( competition_id
foreign key ( farm_id ) references farm farm_id
foreign key ( competition_ia ) references farm_competition competition_ia
)
)
insert into competition_recodd (competition_id, id, farm_id, rank) values (1,8,1)
Translate the following question into SQL.
Question: How many di different statuses do cities have?
SQL: SELECT (statuu) FROM city
Feedback:
The SQL prediction above is wrong. Please fix the SQL.
SQL: SELECT (DISTNNIT status) FROM city
Feedback:
The SQL prediction above is correct!
CREATE TABLE user_profiles
uid number
name text
followers number primary key ( (uid) )
CREATE TABLE tweets id number
34

---

Published as a conference paper at ICLR 2024
uid number
primary key ( id
foreign key ( uid ) references user_profiles ( uid ) )
Translate the following question into SQL.
Question: Find the average number of followers for the users who had some tweets.
SQL: SELECT AVG (fooooloosr)) FROM user_profiles
Feedback:
The SQL prediction above is wrong. Please fix the SQL.
SQL: SELECT AVG (ooloooewess) FROM user_profiled WHERE uid (SELECT uid FROM tweets)
Feedback:
The SQL prediction above is correct!
CREATE TABLE trip id number duration number
bike_id number
primary key ( id
insert into trip (id, duration, bike_id) values (980504, 384, 454);
Translate following question SQL.
Question: What is the total and maximum duration for all trips with the bike id 636?
SQL: SELECT (UM(urration), (duurattion)) FROM trip WHERE bike_id 636
Feedback: The prediction above is correct!
<insert database schemas the question here> <insert original SQL here>
H.3 EXPLANATION FEEDBACK PROMPT (9-SHOT)
CREATE TABLE department
department_id number name text creation text
ranking number
fidget_in_billionn number num num_employees number
primary key ( department_id
insert into department (department_ice name, creation, ranking, in bn_biiioons, num_ son__meoyyees values (1, 'State', '1789'
budget_in_billionee 9. 77.960000000000088822,,, 36266, 39285.9999999999999999
CREATE TABLE head head_id number name text
35

---

established. So the SQL prediction above is Published as a conference paper at ICLR 2024
born_state text
age number
primary key ( head_id
)
insert into head (head_id, name, born_state, age) values (1, Tiger Woods',
Alabama' ,66 9999999999999998
CREATE TABLE management department id number
head_id number
aetoorry_actiig text
primary key ( department_id
foreîgn key ( head_id ) references head Wead_id]
foreign key ( department_id ) references department ( department_id _d )
insert into management (department_ head_id, temporary.acting) values (2,5, 'Yes')
Translate the following question into SQL.
Question: In which year most departments established?
SQL: SELECT creation, COUNT (+) FROM department GROUP BY creation ORDER BY COUNT(*) DESC LIMIT
The execution of the SQL query above would The first column, "creation" would contain the year which department was created. The second column, "COUNT would contain the number of departments created in that year. With "GROUP BY creation ORDER BY COUNT DESC", the table would be sorted in descending order by the number of departments created in each year. With "LIMIT 1", only the first (i.e., highest) result would be returned. So the SQL query returns a table with 2 columns, the year in which most departments were created, and the number of departments created in that year.
table with columns.
Feedback: in your explanation, the SQL query returns a table with columns, the year in which most departments was created, and the number of tmpnrtments created in that year. The question returns 1 column, the year in which the most departments wrong. Please fix the SQL.
SQL: SELECT creation FROM department GROUP creation ORDER BY COUNT(*) DESC LIMIT 1
The execution of the SQL query above would return table with column. The first column, "creation" would contain the year in which department created. With "GROUP creation ORDER COUNT(*) DESC", the table would be sorted descending order by the number of departments created each year. With "LIMIT 1", only the first (i.e,, highest) result would be returned. So the SQL query returns table with column, the year in which most departments were created.
Feedback: As in your explanation, the SQL query returns a table with 1 column, the year in which most departments was created. The question returns 1 column, the year in which the most departments were established. So the SQL prediction above is correct!
CREATE TABLE people people_it ie number sex text
name text height number
36

---

Published as a conference paper at ICLR 2024
weight number primary key ( people_id id
insert into people (people_id, id, sex, name, height, weight) values (1, M, Hubert Henno, 188, 83);
Translate the following question into SQL.
Question:
Find the average and minimum weight for each gender.
SQL: SELECT AVG (weighht) MIN (weeight, sex FROM people GROUP BY sex
The execution the SQL query above would return table with columns. The first column, "AVG (weightt) would contain the average weight. The second column, "MIN (weight)" would contain the minimum weight. The third column, sex" would contain the gender. With "GROUP BY sex", the table would be grouped by gender. the SQL query returns table with columns, the average weight,
minimum weight, the gender of people for each gender
Feedback: As in your explanation, the SQL query returns table with columns, average weight, the minimum weight, the gender of people for each gender. The question returns 3 columns, the average and minimum weight for each gender. So the SQL prediction above is correct!
CREATE TABLE customers ( customer_id number customer_name text
customer_details text
primary key ( customer_id )
insert into customers (customer_is id, customer_namg, customer_detaills values (1, 'Savannah', 'rerum')
CREATE TABLE invoices invoice proiice_number number invoice_date time invoice_detafls text
primary key ( iinvoice_number )
insert into invoices (invoice__umel number, invoice_date, invoice_detailyy values (1, '1989-89-03 16:03:85', 'vitae")
CREATE TABLE orders order_ie id number customer_id number order_status text
date_order_placed time order_detaile text
primary key ( order_id ),
) foreign key ( customer_it id references customers ( customer id
insert into orders forderiid, customer_id, order order_status status, date_order_plent
order_details, values (1, 2, "Shipped', '2009 02-21 15:26:19', NULL);
CREATE TABLE products product_id number product_name text product_detafls text,
primary key ( product_id )
insert products (product_id, product_name, product_details) values (1,
'food', NULL);
37

---

Published as a conference paper at ICLR 2024
CREATE TABLE order_items order_item rem_id number
product_id number
order_id number
ihter_item_sta_ee text
order_item_detaile text
primary key ( coddr_itte__e
foreign key ( product__d references product product_id id
) foreign key ( order_id references orders ( order_id
insert order_items (order_item_te ii, product_id, order_ id, Brder_itemsstane arder_item_detaalt values (1, 4, 6, 'Finish', NULL)
CREATE TABLE shipments
shipment_id number
order_id number
invoice_number number
somment_ttrckinngnubbbee text,
shipment_date time
other_shipment_detait. text
primary key ( (sshimeent_ii idd
foreign key ( sinvoice_nubbe
references invoices (
aavoices_nnuber )
foreign key ( (rder_id references order order_id
)
insert shipments (shipment_ id, order_id, invoice_number,
shipment_tracke trr__rrcckinnnubbeeee number, shipment_date other_shipment_detaaii values (1,
5, 13, '3452', '1983-08-11 08-13 13 22:34:11', NULL)
CREATE TABLE shiimmett_ itess shipment_id id number
arder_item_ite id number
primary key ( )
foreign key ( shhipment_id references shipmentss shipment_id
foreign key ( (order_item_te references order_items order_item_id
)
insert into shipment_iten items (shipment_iae id, order order_item_itee item_id) values (11, 12)
Translate the following question into SQL.
Question: Which customers have both "On Road" and "Shipped" order status? List the customer names.
SQL: SELECT customers. customer _ame FROM customers JOIN orders ON customers. customer_ite orders. customer id WHERE orders..oder_sste orders.orden -sttus "Shipped"
"On Road"
The execution of the SQL query above would column, "customers. uustomer_name would contain the customer name. With customers JOIN orders", the table would contain the data about customers with orders. In the WHERE clause, with "orders. iatatussstttss = 'On Road"" the table filters the records to only include customers who have order 'Shipped"",
empty table. The first
status "On Road", with "orders.order_stttuu the records to only include customers who have order status "Shipped", With "WHERE orders.order__stttee the table filters the records to only include customers who have both "On Road" and "Shipped" as order status the same time. This since the two are mutually exclusive. So the SQL query returns an empty table, because the "WHERE" clause contains mutually exclusive conditions.
table filters
= 'On Road' AND orders.order_staeee
'Shipped'"
possible,
Feedback: in your explanation, the query returns empty table, because the "WHERE" clause contains mutually exclusive conditions. The question returns column, the customer names who have order status of both
38

---

Published as a conference paper at ICLR 2024
"On Road" and "Shipped". So the question returns 1 column. So the SQL prediction above is wrong. Please fix the SQL.
SQL: SELECT customers, wustomerrrreeeeeeee........rr _ane FROM customers JOIN orders ON customers. customer_id id = cedero.customer..o _i WHERE onders.order_steeee
"On Road" INTERSECT SELECT customers, cearrsscsstomer_nne FROM customers JOIN orders ON customers. www.omerstttttooee orders.custtmer__ id WHERE orders.order_status Shipped"
The execution of the SQL query above would return table with column. The first column, "customers.custonn _name" would contain the customer names. With "customers JOIN orders", the table would contain the data about customers with orders. With "WHERE urders.order_staeeee table filters the recards to only include customers who have order status Road", With "WHERE ordeers.oderrsseeeee the records only include customers who have order status "Shipped". With "INTERSECT", the table would be filtered to only include customers who have both "On Road" and "Shipped" as order status. So the SQL query returns table with column, the customer names who have the order status "On Road" and "Shipped".
'On Road"", the
'Shipped"",
table filters
Feedback: As in your explanation, the SQL query returns a table with 1 column, the customer names who have the order status "On Road" and "Shipped ". The question of both "On Road" "Shipped", So SQL prediction above correct!
column, the customer
order status
CREATE TABLE flight flno number
origin text
destination text
aid number
primary key ( flno )
insert into flight (flno, origin, destination, aid) values (99.0, Los Angeles, Washington D.C., 1);
Translate following question SQL.
Question:
Show origins of all flights with destination Honolulu.
SQL: SELECT origin FROM flight WHERE destination "HONO"
The execution of the SQL query above would return empty table. The first column, "origin" would contain the origin of flights. With "WHERE destination 'HONO'",
table filters the records to only include flights whose destination is "HONO". So the SQL query returns an empty table.
Feedback: As in your explanation, the SQL query returns empty table. The question returns column, the origins of all flights with destination Honolulu. the SQL prediction above wrong. Please fix the SQL.
SQL: SELECT origin FROM flight WHERE destination = "Honolulu"
The execution of the SQL query above would return table with column. The first column, "origin" would contain the origin of flights. With "WHERE destination = 'Honolutu'", the table filters the records to only include flights whose destination is "Honolulu". with 1 column, the origin of flights with destination Hanolulu.
the SQL query returns table
Feedback: As in your explanation, the SQL query returns table with 1 column, the origin of flights with destination Honolulu. The question returns column, origins all flights with destination Honolulu. the SQL prediction above is correct!
39

---

Published as a conference paper at ICLR 2024
CREATE TABLE station ( id number name text
Lat number
long number
dock_count number
city text
installation date text
primary key ( id
insert into station (id, name, lat, long, dock_count, city, installation values (2, San Jose Diridon Caltrain Station, 37. 37.329732, 111111111117220000000000000 02, 07, San Jose, 8/6/2013);
CREATE TABLE status station_ie id number
bikes_ard likes_avaiiltee lable number docks_avarilable number
time text
primary key (
foreign key ( (stationiid references station id )
)
insert into status (station_id, id, bikes_available, docks_available, time values (3,12,3, 2015-06-02 12:46:02);
Translate the following question into SQL.
Question: What is the average longitude of stations that never had bike availability more than
SQL: SELECT AVG(long) FROM station WHERE id IN (SELECT station_id id FROM status WHERE bikes_ lable <= 10)
The execution of the SQL query above would return table with 1 column. The first column, "AvG(long)" " would contain the average longi tude. With "WHERE id (SELECT station_id FROM status WHERE bikes_ available <= 10)", the table filters the records to only include stations with or bikes available. So the SQL query returns a table with 1 column, the average longitude of stations with 18 or less bikes available.
query returns table with column, the average langitude stations with 10 or less bikes available. The question returns column, the average longitude of stations that never had bike availability more than 10. So the SQL prediction above is wrong. Please fix the SQL.
Feedback:
your explanation,
SQL: SELECT AVG(long) FROM station WHERE id NOT IN (SELECT station_ia id FROM status WHERE Likes_availabble > 10)
The execution of the SQL query above would return table with 1 column. The first column, "AvG(long)" would contain the average longi tude. With "WHERE id NOT IN (SELECT station_id FROM status WHERE bikes_avai Labii 18)", the table filters than 10 bikes available. So the SQL query returns table with column, the average longi tude of stations that do not have more than 10 bikes available
records to only include stations that not have more
Feedback: As in your explanation, the SQL query returns a table with 1 column, the average longitude of stations that do not have more than 10 bikes available. The question returns 1 column, the average longitude of stations that never had bike availability column, the average longitude of stations that never had bike availability more than 10. So the SQL prediction above is correct!
than 10. The question returns
40

---

Published as a conference paper at ICLR 2024
CREATE TABLE host ( id number
name text
nationality text
age number primary key ( host_id
insert into host (host_id, name, nationality, age) values (1, "Austin Daye"," United "aates",43);
Translate the following question into SQL.
Question:
Show the name and the nationality of the oldest host.
SQL: SELECT name, nationality FROM host WHERE age (SELECT MIN(age) FROM host)
The execution of the SQL query above would return table with columns The first column, "name" would contain the name. The second column, nationality" would contain the nationality. With "WHERE age = (SELECT MIN age) FROM host)", the table filters the records only include hosts with the minimum age, which means the youngest age. So the SQL query returns table with 2 columns, the name and the nationality of the host with the youngest age.
query returns table with columns, the name and the nationality the host with the youngest age. The question returns 2 columns, the name and the nationality of the oldest host So the SQL prediction above is wrong. Please fix the SQL.
Feedback:
your explanation,
SQL: SELECT name, nationality FROM host ORDER BY age DESC LIMIT 1
The execution of the SQL query above would return table with columns. The first column, "name" would contain the name. second column, nationality" would contain the nationality. With "ORDER age DESC", the table is sorted in descending order. With "LIMIT 1", the table only includes the first record, which is the oldest host. So the SQL query returns table with columns, the
and the nationality of the oldest host.
Feedback: As in your explanation, the SQL query returns a table with 2 columns, the name and the nationalite of the oldest host. The question returns columns, the name and the nationality of the oldest host. So the SQL prediction above correct!
CREATE TABLE city city_id number official_name text status text,
area_km number
population number
census_ranking text
primary key ( city_id
)
insert into city (city_id, official_name, status, area_km. km_2, population, census_ raakking values (1, GGrand ralls/Grand-Saule 'Town' '18.06', '5706',
636 of 5008')
CREATE TABLE farm farm_id number year number total_horses number horsing_horses number
41

---

Published as a conference paper at ICLR 2024
total_cattle number
number
bulls number
cows number
pigs number
sheep_and, _eee_aad__goas number
primary key farm_id
insert into farm (farm_id, year, total_horses, horreng_horses, oxen, bulls, cows, pigs, sheep_and_goats) values (1, '1927', 5656.5', '3960.1 '8374.5', '805.5', '31.6', '3852.1', '4412.4', '7956.3')
CREATE TABLE farm_ com__comettiion ( competition_1 _n number
year number
theme text
host_city_ji id number hosts text
primary key ( competition_id
foreign key ( host_city_id references city ( city_id
insert into farm_ farm_compett competition (competition id, year, theme, most_city_foe id, hosts) values (1," (1,'2013', 'Carnival M is back! ,,," 'Miley Cyrus Jared Leto and Karen Mok')
CREATE TABLE competition_record ( competition_1 id number
farm_id number
rank number
primary key ( competition_id
foreign key ( farm_id ) references farm ( farm_id
foreign key ( competition_ia ) references farm_ corm_competition (competition_it id )
insert into competition_record (competition_id, farm_id, rank) values (1,8,1)
Translate the following question SQL.
Question: How many different statuses do cities have?
SQL: SELECT (staaus) FROM city
The execution of the SQL query above would return table with 1 column. The first column, "COUNT (statuss) would contain the number of cities. the SQL query returns a table with column, the number cities.
Feedback: As in your explanation, the SQL query returns a table with 1 column, the number of cities. The question returns 1 column, the number of different statuses of cities. the SQL prediction above is wrong. Please fix the SQL.
SQL: SELECT CDISTNNI status) FROM city
The execution of the SQL query above would return table with column. The first column, "COUNT (DISTINCT status)" would contain the number of table with column
different statuses of cities. the SQL query the number of different statuses of cities.
Feedback: As in your explanation, the SQL query returns table with 1 column, the number of different statuses of cities. The question returns column, the number different statuses cities. the SQL prediction above is correct!
42

---

Published as a conference paper at ICLR 2024
CREATE TABLE user_profiles
uid number
name
followers number
primary key ( (uid)
CREATE TABLE tweets id number
uid number
primary key ( id
foreïgn key ( uid ) references user_profiles ( uud ) )
Translate the following question into SQL.
Question: Find the average number of followers for the users who had some tweets.
SQL: SELECT AVG(followers) FROM user_profiles
The execution the SQL query above would return table with column. first column, "AVG(followers) would contain the average number of followers. So the SQL query returns a table with 1 column, the average number of followers.
Feedback: in your explanation, the SQL query returns a table with column, the average number of followers. The question returns 1 column, the average number of followers for the users who had some tweets. So the SQL prediction above is wrong. Please fix the SQL.
SQL: SELECT AVG(followers) FROM user_profiles WHERE uid (SELECT uid FROM tweets)
The execution the SQL query above would return table with column. The first column, "AVG(fol "owess"llowrs) would contain the average number of followers. With "WHERE uid IN (SELECT uid FROM tweets)", the table filters the records only include table with co column, average number of followers of users have tweets.
who have tweets. So the SQL query returns
Feedback: As in your explanation, the SQL query returns table with 1 column, the average number of followers of users question returns column, the average number of followers for the users who had some tweets. So the SQL prediction above is correct!
tweets. The
CREATE TABLE trip ( id number duration number
bike_id id number
primary key ( idd
insert into trip (id, duration, bike_1d) values (980504, 384, 454);
Translate the following question into SQL.
Question: What is the total and maximum duration for all trips with the bike 1d 636?
SQL: SELECT (ddurattio))) (duurattio FROM trip WHERE bike_id 636
43

---

Published as a conference paper at ICLR 2024
The execution of the SQL query above would return table with columns. The first column, "SUM (durattio) would contain the sum of the duration. The second column, "MAX "Muu(tttt"""n"" would contain the maximum duration. with WHERE bike_id 636", the table filters the records to only include trips with bike ID 636. So the SQL query returns table with 2 columns, the sum of the duration and the maximum duration of trips with bike ID 636.
Feedback: in your explanation, the SQL query returns a table with columns, the sum of the duration and the maximum duration of trips with bike ID 636. The question returns columns, the total and maximum duration for all trips with the bike id 636. the SQL prediction above is correct!
<insert database schemas and the question here> <insert original SQL here>
H.4 PROMPT FOR QUESTION EXPLANATION
Infer the return type of the question.
CREATE TABLE song
song_name text
artist_name text
country text
f_id number
genre_is text,
rating number
languages text
primary key ( f_id]
insert into song (song_ name, artist_name, country, f_id, genre_is, is, rating, languages) values (Tumi robe nirobe, Shrikanta, India, 1, tagore, 8, bangla)
Question: What is the id, genre, and name of the artist for every English song ordered by ascending rating?
Answer: "What is the id" returns column, "What is the genre" returns 1 column. "What the id, genre" returns 2 columns. "What is returns 1 column. "What is the id, genre, and name" returns 3 columns, The question returns the id, genre, and name of the artist for every English song ordered by ascending rating. So the question returns 3 columns.
name"
CREATE TABLE physician ( employee_it number
name text
position text
primary key ( (employee_id
insert into physician (employee_id, name, position) values (1, John Dorian, Staff Internist);
CREATE TABLE procedures code number
name text
cost number
primary key ( code
insert procedures (code, name, cost) values (1, Reverse lhinoppooppaasy 1500.0);
44

---

Published as a conference paper at ICLR 2024
CREATE TABLE trained_in physician number
treatment text
primary key ( physiccan ) foreign key ( physician references physician ( @employee_ie id )
insert into trained_in (physician, treatment) values ;
Question: Among the procedures that cost than 1000, which specialized by physician John Wen?
Answer: "Among the procedures that cost more than 1000, which were not specialized in by physician John Wen" returns column. The question returns the procedures that cost physician John Wen. So the question returns column.
than 1098, and not specialized in by
CREATE TABLE department department_id number
name text
creation text
ranking number
budget_in_billion number num, num_emp lomeemeooyees number
primary key ( department_idd)
insert into department (department_ name, creation, ranking, num_ num__mplm_eeeeeee sem__mmllyess) values (1, State, 1789, 1,
9. .96688800999800988555,, 36265. (9888999999999999999999
CREATE TABLE head head_id number
name text
born_state text
age number
primary key ( head_id
insert into head (head_ id, name, born_state, age) values (1, Tiger Woods, Alabama, 66. 99999999999998)
CREATE TABLE management department... id number
head_id id number
temporary_acting text
primary key ( department_id
foreign key ( head_id ) references head head_id
foreign key ( department_id references department ( deppattent__id id ) )
insert into management (department_ic) head_id, temporary_acting) values (2,5, Yes)
Question:
Which department has than head at time? List the id, number of heads.
Answer: "List the id" returns column. "List the name" returns 1 column. List the id, name" returns 2 columns. "List the number of heads" returns 1 column. "List the id, name and the number of heads" returns 3 columns. The question returns the id, more than head at time. So the question returns columns.
and the number of heads the department has
Question: Show the name and number of employees for the departments managed by heads whose temporary acting value is 'Yes'?
Answer: "Show name" returns column. "Show the number of employees" returns column. "Show the name and number of employees" returns 2 columns.
45

---

Published as a conference paper at ICLR 2024
The question returns the name and number of employees for the departments managed by heads whose temporary acting value is "Yes'. So the question returns columns.
CREATE TABLE class class_code text
crs_code text
prof_num number
primary key ( class_code
foreîgn key ( prof_num references professor emp_num num
foreign key ( crs_code references course ( crs_code
insert into class (class_code, crs_code, prof_num) values (10812, ACCT-211, 105);
CREATE TABLE employee emp_num number
emp_ Lname text
emp_ fname text
emp_initial text
emp_jobcode text
primary key ( emp_num
insert into employee (emp_num, emp_ Inp_ln emp_ fap_fname, emp_initial, emp_jobcode values (198, Worley, James, F, CUST);
CREATE TABLE
crs_code text,
dept_code text
crs_description text primary key ( crs_code
insert into course ( crs_code, dept_code, crs_description) values (ACCT-211, ACCT, Accounting I);
CREATE TABLE professor (
emp_num number
dept_code text
prof_office text
primary key ( emp_num )
foreign key ( emp_nnm ) references employee @mp_nuu
insert into professor (emp_num, dept_code, prof_office ) values (103, HIST, DRE 156);
Question: Find the first names and offices of all instructors who have taught some course and also find the course description.
Answer: "Find the first names" returns column. "Find the offices" returns column. "Find the first names and offices" returns columns. "Find the first names and offices of all instructors who have taught some course" returns 2 columns. "and also find the course description" returns 1 column, "Find the first names and offices of all instructors who have taught some course and also find the returns the first names offices of instructors who have taught some course and the course description. So the question returns columns.
description" returns columns. The question
CREATE TABLE airports
apid number text
city text country text.
46

---

Published as a conference paper at ICLR 2024
elevation text primary key ( apid
Question: List the id, country, city and name of the airports ordered alphabetically by the name.
Answers "List the id" returns column. "List the country" returns column, "List the id, country" returns 2 columns. "List the city" returns 1 column, "List the id, country, city" returns 3 columns. "List the name" returns 1 column. "List the id, country, city, and name" returns columns. The question returns the id, country, city and name of the airports ordered alphabetically by the name. So the question returns columns.
CREATE TABLE student (
stuid number,
Iname text
fname text
primary key ( (tuuid) )
insert into student (stuid, Iname, fname) values (1001, Smith, Linda);
CREATE TABLE plays_games
stuid number
gameid number
hours_played number
population number
primary key ( stuid )
foreign key ( stuid ) references student ( stuid )
)
insert into plays_games (stuid, gameid, hours_played) values (1001, 1, 35);
CREATE TABLE sports_ings info ( stuid number
sport_name text
hours_per_weck number
games_played number
primary key ( (tuui )
foreign key ( stuid ) references student ( stuid
)
insert into sport_info (stuid, sport_name, hours_per_weet games_played values (1001, Athletics, 2, 5);
Question: What
the ids of all students and number of hours played?
Answer:
"What are the ids" returns column. "What are the number of hours played" returns 1 column. "What are the ids and number of hours played" returns 2 columns. The question returns the ids of students and number of hours played. So the question returns 2 columns.
Question: What is the total number of hours per week and number of games played by students under 20?
Answer: "What is the total number of hours per week" returns column. "What is the number of games played" returns 1 column. "What is the total number number of games played" returns columns. The question returns the total number hours per week and number games played by students under 20. So the question returns 2 columns.
of hours per week
CREATE TABLE journal ( journal_id number theme text
primary journal_id )
47

---

Published as a conference paper at ICLR 2024
insert into journal (journal_id, theme) values (1, Miami Dolphins);
CREATE TABLE editor
editor_id number
name text
age number
primary id editor_id
insert into editor (editor_id, name, age) values (1.0, Kamila Porczyk, 34.0)
CREATE TABLE journal_comnittee
aditor_id number
journal_id number
primary id (dditor_id
foreign key ( editor_id id references editor editorr_
foreign key ( (journal_id references journal (jurrnllidd
)
insert into journal_commitee ttee (editor_id, (ournal_id) values (1, 13);
Question: Show the names and ages of editors and the theme of journals for which they serve on committees, in ascending alphabetical order of theme. Answer: "Show the names" returns 1 column. "Show the ages" returns 1 column. "Show the returns column. "Show the names
and ages" returns columns. "Show the theme of journals"
ages editors and journals" returns 3 columns. "Show the names and ages of editors and the theme of journals for which they serve commi ttees" returns 3 columns. Show the names and ages of editors and the theme of journals for which they serve committees,
theme of
ascending alphabetical order of theme" returns columns, The question returns the names and ages of editors and the theme of journals for which they serve on committees, in ascending alphabetical order of theme. So the question returns 3 columns.
CREATE TABLE people ( people_id number sex text
name text
height number
weight number
primary key ( peeolee_id
insert into people (people_ id, sex, name, height, weight) values (1, M, Hubert Henno, 188, 83);
Question: Find the average and minimum weight for each gender. Answer: "Find the average weight" returns 1 column. "Find the minimum weight returns column. "Find the average and minimum weight" returns 2 columns. "Find the average and minimum weight for each gender" returns columns. The question returns the average and minimum weight for each gender. the question returns columns.
Question: List all info about all people.
Answer: "List info" returns all columns. The question returns all info about all people. So the question returns all columns.
CREATE TABLE student (
sturid number Iname text
fname text age number sex text
48

---

Published as a conference paper at ICLR 2024
major number
advisor number city_code text
primary key ( stuid )
insert into student (stuid, Iname, fname, age, sex, major, advisor, cody_code) values 1081, Smith, Linda, 18, F, 600, 1121, BAL);
Question: How many students are there for each major?
Answer: "How many students" returns 1 column. "How many students there for each major" returns columns. The question returns number students for each major. So the question returns 2 columns.
CREATE TABLE accounts account_id id number
customer_id number date_ wwcount_openeeeeeeeeeeecceeeeeee time account_ name text.
other_account_detaills text
primary key ( aacouuttidd
foreign key ( customer_1o id references customers customer_10
insert into accounts (account_iae id, customer_it id, date date_account, www.acccccooooooee _opeetd__ppeee___eee_e account_nand name, other_acee atther_cccont__ettil... values (1, 6, 2016-07-39 22:22:24, 338, Regular);
CREATE TABLE customers ( customer_id number dustomer_first_ne text,
customer_last_nane text
customer_address text
customer_phone text
customer_emait text
Sther_customer_detailll text
primary key ( customer_id
insert into customers (customer_ id, customer customer_first, _firrt_name,, customer_last_ same,, customer_addrest customer _chtne,r_phon,, austomer_emaill values (1, Aniyah, Feest, 55975 Theodore Estates Lake Brody, VT 57978, (673) 872-5338, fahey aetorranexammplee.. com, NULL);
Question: What are the full names of customers who have accounts? Answer: "full names" include the first and the last "What the full question returns the full names customers who have accounts. So the question returns columns.
names" returns 2 columns.
Question: What are the ids, date opened, name, and other details for all accounts?
Answer: "What are the ids" returns column. "What are the date opened" returns 1 column. "What are the ids, date opened" returns 2 columns. "What are the name" returns 1 column, "What are the ids, date opened, name" returns columns. "What the ids, date opened, name, and other details" returns columns. question returns the ids, date opened, name, and other details for all accounts. So the question returns columns.
the other details" returns 1 column. "What are
CREATE TABLE station (
id number
text
lat number long number
49

---

Published as a conference paper at ICLR 2024
dock_coum number
city text
installation_dee text
primary key ( (id)
)
insert into station (id, name, lat, long, bock_count, city, installation date) values (2, San Jose Diridon Caltrain Station, San Jose, 8/6/2013);
-1.22973,,11.2191178888 201.2228820000001110000000
CREATE TABLE status station_ie id number
bikes_available number
docks_available number
time text
primary key (
foreign key ( station_id references station id )
)
insert into status (station_id _id, bikes_available, docks_available, time)
values (3,12,3, 2015-06-02 12:46:02);
CREATE TABLE trip id number duration number
bike_id number start_date time start_station_net text
start_station id number
primary key ( id
insert into trip (id, duration, bike_id, id, start date, start_state start_station_id] values (908504, 384, 454, 8/21/2815 17:03, Howard at 2nd, 63);
name,
Question: What is name every city that has at least 15 stations how many stations does it have?
Answer: "What is the name" returns column. "What is the name of every city that has least stations" returns column. "What is the name of every city that has least 15 stations how many stations does have" returns 2 columns. The question returns the name and the number of stations of every city that has at least 15 stations. So the question returns 2 columns.
Question: Which start station had the most trips starting from August? Give me the name and id of the station.
Answer: "Give the name" returns 1 column. "Give me the id" returns column. "Give the name and id" returns 2 columns. The question returns the name id the station that had most trips starting from August. So the question returns 2 columns.
CREATE TABLE user_profiles uid number
name text
followers number primary key ( (uid]
Question: List the name and number of followers for each user, and sort the results by the number of followers in descending order.
Answer: "List the name" returns column. "List the number of followers" returns column. "List the
and number of followers" returns columns. number of followers for each user" returns columns. List the name and number of followers for each user, and sort the results by
"List
50

---

Published as a conference paper at ICLR 2024
the number of followers in descending order" returns columns. The question returns the name and number of followers for each in the descending order of the number of followers. the question returns columns.
CREATE TABLE race (
race_id number
name text text
track_id number
primary key ( race_id]
foreïgn key (track_id ) references track track_id id
insert into race (race_id, name, date, track_id) values (1, Rolex 24 At Daytona, January 26 January 27, 1);
CREATE TABLE track track_id number
name text primary key ( (rackk_d
insert track (track_id, name) values (1, Club Speedway);
track name. Answer: "Show the name" returns 1 column. "Show the date" returns 1 column.
Question: Show the name and date for each race
"Show the name and date" returns 2 columns. "Show the name and date for each race" returns columns. "Show the name and date for each race and its name" returns columns. question returns the name, date track name for each race. So the question returns 3 columns.
CREATE TABLE station ( id number
network_name text,
services text
primary key ( idd
insert into station (id, network_name, services) values (1, Amersham, Metropolitan line Chiltern Railways)
CREATE TABLE routes train_id number
station_id number
primary key ( (triinid
foreign key ( (station_id ) references station (id) )
insert into route (train_id, station_id) values (1,1);
Question: Find the number of trains for each station, well the station network name and services.
Answer: "Find the number of trains" returns 1 column. "Find the number of trains for each station" returns 2 columns, "Find the number of trains for each station, the number of trains for each station, well the station network and services" returns 3 columns. The question returns the number trains, the station network name and services for each station. So the question returns 3 columns.
well as the station network name" returns columns. "Find
CREATE TABLE manufacturers code number
51

---

Published as a conference paper at ICLR 2024
name text primary key ( code
insert into manufacturers (code, name) values (1, Sony);
CREATE TABLE products
code number
name text
manufacturer number
primary key ( code
foreign key ( manufacturer ) references manufacturer code
insert into products (code, name, manufacturer) values (1, Hard drive, 5)
Question: Find the number of products for each manufacturer, showing the name of each company.
Answer: "Find the number of products" returns column. "Find the number of products for each manufacturer" returns columns. "Find the number of products for each manufacturer, showing the name of each company" returns columns, The question returns the number of products and the company name for each manufacturer. So the question returns 2 co lumns.
<insert database schemas and the question here>
H.5 PROMPT FOR SQL EXPLANATION
Summarize the return type of the execution.
SQL: SELECT creation FROM department GROUP BY creation ORDER BY COUNT(*)
DESC LIMIT 1
Execution:
1789
Answer: The execution of SQL query above would return a table 1 column. The first column, "creation" would contain the year in which a department was created. With "GROUP BY creation ORDER BY COUNT(*) DESC", the table would be sorted in descending order the number of departments created each year. With "LIMIT 1", only the first (i.e., highest) result would returned. the SQL query returns table with column, the year in which most departments were created.
SQL: SELECT manufacturer, AVG(price) FROM products GROUP BY manufacturer
Execution:
240.0
2 150.0
Answer: The execution of the SQL query above would return a table with columns. The first column, "manufacturer" would contain the manufacturer code. The second column, "AvG(price)" " would contain the average price of each manufacturer's product. So the SQL query returns table with columns the manufacturer code, the average price each manufacturer's product.
SQL: SELECT customers. customer _name FROM customers JOIN orders ON customers. customer_id orders. customer_ie WHERE orders. wwwerrrrttttuuu "On Road" INTERSECT SELECT customers. customer. _nme FROM customers JOIN orders ON customers. customer_ite orders. customer WHERE orders.order_sst Shipped"
52

---

Published as a conference paper at ICLR 2024
Execution:
George
Answer: The execution of query would return a table column. The first column, "customers.cus wowtecccsttttaaaieeeeeeeee would contain the customer names. with "customers JOIN orders", the table would contain the data about customers with orders. With "WHERE orders.order. orders.order_status _ttadussttttsee 'On Road "", the table filters the records only include customers who have order status Road". With "WHERE arders.order_stee
"Shipped"", the table filters the records to only include customers who have order status "Shipped With "INTERSECT", the table would be filtered to only include customers who have both "On Road" and "Shipped" order status. So the SQL query returns table with column, "On Road" and "Shipped".
who have the order status
SQL: SELECT MAX(cow)), MIN(cows) FROM farm
Execution:
3987.0 2407.2
Answer: The execution of the SQL query above would return table with 2 columns. The first column, "MAX(cowS (cows)) would contain the maximum number cows. The second column, "cows" would contain the minimum number of cows. So the SQL query returns table with 2 columns, the maximum and minimum number of cows.
SQL: SELECT WWGratiig.starss movie.title FROM rating JOIN movie ON rating. mid movie. mid WHERE movie.year (SELECT MIN(year) FROM movie)
Execution:
4.5 Snow White
Answer: The execution of the SQL query above would return a table with 2 columns. The first column, "AVG(rating.starg" would contain the average rating. The second column, "movie.title" would contain the movie title. With "rating JOIN movie", the table would contain the data about ratings movies. With "WHERE movie.year = (SELECT (yery))r) FROM movie)", the table filters the records to only include movies with the minimum year, which the earliest year. So the SQL query returns table with columns, the average rating and movie title of movie with the earliest year.
SQL: SELECT course.title; course.credif FROM classroom JOIN section classroom. building aaction.buildiio AND classroom. room_ poom_n section. room_number JOIN course ON section.course. cessroom.capacity (SELECT (AXpaatc))y)) FROM classroom)
course.course_. id WHERE
Execution:
Accounting
Answer: The execution of the SQL query above would return a table with 2 columns. The first column, "course.titte would contain the course title. The second column, "course.credits" would contain the course credits. With classroom JOIN section", the table would contain the data about classrooms and sections. With "JOIN course", the table would contain courses. With "WHERE classroom.capare caasacitm = (SELECT MAX (caapaciity) FROM classroom)", the table filters the records to only include classrooms with the maximum mum capacity, which means the largest capacity. the SQL query returns table with columns, the course title the course credits the classroom with the largest capacity.
data about
SQL: SELECT model_name, ram_mib FROM chip_model ORDER BY ram_mib ASC LIMIT
1;
Execution:
X5 32.0
53

---

Published as a conference paper at ICLR 2024
Answer: The execution of the SQL query above would return table with 2 columns. The first column, "model_namer would contain the model second column, "ram_mib" would contain the size. ASC", the table would be sorted in ascending order by RAM size. with "LIMIT 1", only the first (1.e,, lowest) result would be returned. So the SQL query returns table with 2 columns, the model name and the RAM size of the chip model with the Lowest RAM size.
The
"ORDER ram_mib
SQL: SELECT official_name, status FROM city ORDER population LIMIT
Execution:
Grand Falls/Grand-Sand Sault Town Answer: The execution of the SQL query above would return a table with 2 columns. The first column, "official_name" would contain the official name. The second column, "status" would contain the status. With "ORDER BY population DESC", the table population, which means from the most number of peaple to the least number of people. With "LIMIT 1", only the first (f.e,, highest) result would be returned. the SQL query returns table with columns, the official status of the city with the largest population, which means with the most number of people.
sorted in descending order by
SQL: SELECT AVG (bbdgettmiillit FROM movie WHERE year 2000
MAX(budget_nillion) MIN(budget_milllog
Execution:
6.0 7.0 5.0
Answer: The execution of SQL query above would return a table 3 columns. The first column, "AVG(budget_nillfon) would contain the average budget. The second column, (WWX(budeet_mflllio would contain the maximum budget. The third column, "WIN(budget_million budget. With "WHERE year 2900", the table filters the records to only include movies before 2000. So the SQL query returns a table with 3 columns, the average, maximum, and minimum budget of movies before 2008.
would contain the minimum
SQL: SELECT department..epartmeen department, id, department. name, COUNT(*) FROM management JOIN department ON department. department_ id management. department_ GROUP BY department dew.ntttt.pprtttte.edmmmm HAVING COUNT(*)
Execution:
2 Treasury
Answer: The execution of the SQL query above would return a table with 3 columns, The first column, "depar Edepartment.dee depptedeetttte_tieee would contain the department ID. The second column, "department. name" would contain the number
department name. The third column, "COUNT(M)" would contain managers. With "management JOIN department", the table would contain the data about management and departments. With "GROUP BY department. department_idd, the table would be grouped department ID. With "HAVING COUNT(*) 1", the table filters the records only include departments with more than manager. So the SQL query returns table with columns, the department ID, the department name, and the number of managers of departments. wi th more than 1 manager.
SQL: SELECT accounts.custome id, customers. wwwpenerrs.sstttrrr.r. saaeeerffrrrr__rreeeeee customers. customer laa____tttam,,,,,,tt C*)NT(*) FROM accounts JOIN customers accounts. customer_id customers. customer_tte GROUP BY accounts.custome
Execution:
3
Joesph Vesta
Schaefer
6
Leuschke
54

---

Published as a conference paper at ICLR 2024
Answer: The execution of the SQL query above would return table with 4 "accounts.ccsttomr..ti
columns. The first co column, "accounts customer_id" would contain the customer
cesttommrs.customerr.... ID. second column, "customers: www.cuuurrrruuuc.. "usttmmr__first_nmme would contain the customer's first name. The third column, "customers: customermsssstttteeeee would contain the customer's last name. The fourth column, "COUNT(*)" would contain the number of accounts. With "accounts JOIN customers", the table would contain the data about accounts and customers. with "GROUP accounts .customer_ id", the table would be grouped by customer ID. So the SQL query returns a table with 4 columns, the customer ID, the customer's first name, the customer's last name, and the number of accounts of each customer.
SQL: SELECT
FROM people
Execution:
Hubert Henno | 16..10.00 1976 | 188.0 83.0
I
Dominique Daquin | 10.11.197 | 197.0 85.0
Answer: The execution of the SQL query above would return a table with 6 columns. "SELECT x" returns all columns. So the SQL query returns all information of people.
SQL: SELECT name, city, country, elevation FROM airports WHERE city "New York"
Execution:
La Guardia Airport New York United States 21
John F Kennedy International Airport New York United States 13 Answer: The execution of the SQL query above would return table with columns. The first column, "name" would contain the airport name. The second column, "city" would contain the city name. The third column, "country" would contain the country name. The fourth column, "elevation" would contain the elevation. With "WHERE city 'New York"", the table filters the records only include airports in York. So the SQL query returns table with columns, the airport name, the city name, the country name, and the elevation of airports in New York.
SQL: SELECT AVG(weight) (Ieewight), sex FROM people GROUP BY sex
Execution:
92.0 98.0
76675428571128871
82.0 F
Answer: The execution of the SQL query above would return a table with 3 columns. The first column, "AVG(weight)) would contain the average weight. The second column, "MIN (weighhh) would contain the minimum weight. The third column, "sex" would contain the gender. With "GROUP BY sex", the table would grouped by gender. So the SQL query returns a table with columns, the average weight, the minimum weight, and the gender people for each gender.
SQL: SELECT COUNT(*), city FROM employees WHERE title "IT Staff" GROUP BY city
Execution:
Lethbridge
The execution of the SQL query above would return a table with 2 columns. The first column, "COUNT would contain the number of employees. The second column, "city" would contain the city name. With "WHERE title 'IT Staff the table filters the records the title "IT Staff", With "GROUP BY city", , the table would be grouped by city. So the SQL query returns a table with 2 columns, the number of employees with the title "IT Staff", and the city name for each city.
Answer:
only include employees with
55

---

"On Road" AND Published as a conference paper at ICLR 2024
SQL: SELECT customers, www.suurrntaaaaaaeoe FROM customers JOIN orders ON customers, customer_ id
WHERE orders.order_$stt _stttus "Shipped"
Execution: None
Answer: The execution of the SQL query above would return an empty table. The first column, "customers. cettssscsstttm_rmoo
would contain the customer name. With "customers JOIN orders", the table would contain the data about customers with orders. the Road"", the table filters the records to only include customers who have order status "On Road". With "orders.order sttauusstuuussss 'Shipped"", the table filters the records to only include customers who have order status "Shipped 'On Road'
'On clause, with "orders. "orders.order_stae" order_status
". with "WHERE orders.order_stael Shipped'", the table filters the records only include customers who have both "On Road" and "Shipped" as order status at the same time. This is not possible, since the two are mutually exclusive. So the SQL query returns an empty table, because the "WHERE" clause contains mutually exclusive conditions.
orders. order_status
SQL: SELECT origin FROM flight WHERE destination "HONO"
Execution: None
Answer: The execution of the SQL query above would return an empty table.
The first column, "origin" would contain the origin of flights. With "WHERE destination "HONO"", the table filters the records to only include flights whose destination "HONO" So the SQL query returns an empty table.
<insert the new SQL here>
H.6 CHAT SIMPLE FEEDBACK PROMPT (0-SHOT)
System: You are an expert programming assistant.
User: Write the SQL query for the given question. <insert the database schema and the question here>
Assistant: KchatGPT Completion>
User: Does the SQL match the question? not, generate the fixed SQL.
Assistant: KchatGPT Completion [...]
H.7 CHAT EXPLANATION FEEDBACK PROMPT (0-SHOT)
System: You expert programming assistant.
User: Summarize the return type the SQL execution.
kinsert the database schema and the SQL here>
Assistant: KchatGPT Completion
User: Infer the return type of the question.
<insert the question here>
Assistant: KchatGPT Completion>
User: Does the SQL match the question? If not, generate the fixed SQL.
Assistant: KchatGPT Completion [...]
56

---

Published as a conference paper at ICLR 2024
1 CODE TRANSLATION
1.1 BASELINE PROMPT (3-SHOT)
The prompt for generating the initial Python translation taken from (Chowdhery et al. 2022). We set temperature 7 0.8 as in (Chowdhery et al. 2022).
Translate the following C++ program into Python [code]
[c++]
int add_ @id_1_to_a_ _od___tt___givee_numeee int { int 1;
while ( x &
<<= 1;
} x x m;
return x;
[python]
def [__1__te tog_i_tt___giiee_numeeer (x)
while (x & m):
(x m)
x (x m)
return
[C++]
unsigned int program_ for_ffffcctttiill.ottt _n_uberrrr_errrrrrrr
number ( (unsigned int )
if n ) return 1;
return program_ for for_ _f_mbbberrr__rrrrrrrr ( - )
[python]
def program
(n)
return (1 if ((n 1) or (n == 0)) else (n _a____fa___________________ ((n 1))))
[c++]
long long int sum pairwise_produce ( int )
long long int sum 8;
for ( int i 1;
i <=
i ++ ) for int 1;
j n;
j ++ ) sum sum i;
return sum;
[python]
def sum_ wwm._pairwise_producee (n)
sm = 0 for in range(1, (n + 1)):
for in range(i, (n + 1)): (sm
j})
return sm
[c++]
<insert C++ program here> [python]
57

---

Published as a conference paper at ICLR 2024
1.2 BASELINE EXPL. PROMPT (3-SHOT)
Explain the code line by line, and then translate the following program into Python
[c++]
int add _oo___to Boo_i_.t___givee_nmmme
int x {
int 1;
while ( x & ) {
m;
<<= 1;
}
= x m;
return
[/c++]
[explanation]
The code is implementation of adding 1 to given number. int =
Using the variable as mask, the initial value is
while x & {
m; <<= 1;
In the while loop, if the current digit of the given number is 1, then it will be flipped to 0, and the mask will be shifted to the next digit. This iteration will be executed until the first of the number is met.
x = x m;
The first o will be flipped to 1. Thus, 1 will be added to the given number.
return x; Finally, the function returns the incremented number. [/explanation]
[python]
def add _dd____t____give__nubbee (x):
m while (x & m):
(x m)
<<=
(x m)
return x
[/python] [c++]
unsigned int a ( unssinned n )
if ( == 0 ) return 1;
return program_ for ffcttorral_offaaammmtttt ( 1
[/c++]
[explanation
The code is an implementation of calculating the factorial of a number.
if ( return 1;
The function is defined recursively. when the given number is equal to 0, the result of the factorial is 1.
Loogram_oo__factrriaa_ooo wwl.faccorillo_____0 Otherwise, the result of factorial the product the given number and the (given number minus 1) factorial. [/explanation] [python]
return program_for
number ( _uubeerr___nabbrrrrrrreeeae
1 1
def fongram_for_facttriii of wwwbeerrraaaaa_a_aaae_____ (n)
58

---

Published as a conference paper at ICLR 2024
return (1 if ((n == 1) (n == 0)) else (n
www.faaattrral.f_oecccc for.rmmrroraacctrrial...oo [/python]
program_for for_
_nuuberrbeeerrrrrreee number ((n 1))))
[c++]
long long int sum_patrrits_prroucct ( int n )
long long int sum 8;
for int
1;
<= n;
++ ] for int = 1;
j <=
++
sum;
[/c++]
[explanation]
The code is an implementation of summing the products all the possible pairs.
Long long int The variable sum used record the result.
0;
for ( int i 1;
i <=
) for int i;
j <= <= n;
j ++
There are two nested loops. The outer loop iterates until i (the index of the first number of pair) is equal to the given number. The inner loop iterates until j (the index the second number of the pair) is equal to the given number.
sum i When the loops executed, the product of the pair will be calculated and then added to the variable sum.
return sum;
Finally, the function returns the [/explanation]
[python]
def sum_paimeeeee
www._paiwwiss_prduuctsee
sm = for in range(1, (n 1)):
for range (n 1)): sm (sm (i i))
return sm [/python]
[c++]
kinsert C++ program here>
[/c+]]
[explanation]
I.3 SIMPLE FEEDBACK PROMPT (2-SHOT)
Below C++ programs with incorrect Python translations. Correct the translations using the provided feedback.
[c++]
unsigned int program. for wwt.faaatrraao..aaaaam ____a__u_e_____berrrrrrr_e__e ( unsiggned int )
if n ) return 1; return n faatrra_foo_faatrriiit fnnn a_number (
)
59

---

Published as a conference paper at ICLR 2024
[/c++]
[python]
def program_foo ffor_fattorill,.o. _n_______ebberon
return (1 if ((n == 1)) else (n *
((n
1))))
[/python]
The above Python translation does not do the same thing as the C++ code. Correct the Python translation.
[python]
def program_ for cor_factorial_ff fft.ttorial......mmmeeeere an__mmrrr())
return (1 if ((n == 0)) else (n program_ frogram_for_facttriall la _f_a S_numberr ((n
1))))
[/python]
[c++]
long long int sun_paitriiee_rroduess ( int )
long long int sum 8;
for int
1;
<= n;
i ++ ) for int = i;
i <=
++ )
i j;
return sum;
[/c++]
[python]
def sun_pairwise_prodcctstto
sum for in range(n):
for in range (f, ((n + 1))): (sum (i f))
return sum [/python]
The above Python translation does not do the same thing the C++ code. Correct the Python translation.
[python]
def jun_pairwiis_pproductts/m (n)
sum for range (1, ((n + 1))):
for j in range ( ((n + 1))): sum = (sum (i i))
return [/python]
[c++]
kinsert C++ program here>
[/c++]
[python]
<insert original Pythan translation here>
[/python]
The above Python translation does the same thing the C++ code. Correct the Python translation. [python]
I.4 UNIT TEST FEEDBACK PROMPT (2-SHOT)
Below are C++ programs with incorrect Python translations. Correct the translations using the provided feedback. [c++]
unsigned int nuume
number ( unsigned int n )
60

---

Published as a conference paper at ICLR 2024
if n == 0 return 1; return program for www.orfaataaa.....oomoomo number ( _nmmbrrrrrrreer
)
[/c++]
[python]
def program_ for_rforrroooorrooorrrrr factorial_ of sactiial_fa___aiieaa a_number(n):
return (1 if ((n 1)) else (n program_ for__ progran_for_facttriall fttooffactrraaaof...oo htmbbrrrrrrrrrrrreee ((n
1))))
[/python]
The Python translation does
the same thing the C++ code. These
results of one failed unit test that tests whether Python
translation's outputs match the C++ program's outputs:
Failed: assert Actual Result: RecursionErrar: maximum recurston depth exceeded in comparison
==
Correct the Python translation.
[python]
def program for for_ factorial_oofff...o [_numbr()))
return (1 ((n 0)) else (n * progran_for_factoriaa... _a___aaaaaaaaraaaeea ((n
1))))
[/python]
[c++]
long long sum_pairwtse_prodeee ( int n )
long long int sum 8;
for ( int i 1;
i <=
) for int i;
j <= n;
j ++ ) sum sum * j;
return sum;
[/c++]
[python]
def sun.pairwiise_producttt//
sum =
for range(n):
for j range(i, ((n 1))): sum = (sum (1 j))
return sum [/python]
The Python translation does not do the same thing as the C++ code. These are the results of one failed unit test that tests whether the Python translation's outputs match the C++ program's outputs: sun_pairwise_prouucts[
Failed:
Actual Result: 16
Correct the Python translation.
[python]
def sum www.ppairwise_produtt/s////////
sum = 0
for in range ((n + 1))) for in range ((n 1))): sum (sum j))
return sum [/python]
[c++]
<insert C++ program here>
[/c++]
[python]
<insert original Python translation here>
[/python]
61

---

Published as a conference paper at ICLR 2024
1.5
UNIT TEST EXPL. FEEDBACK PROMPT (2-SHOT)
Below C++ programs with incorrect Python translations. Explain original code, then explain the translations line by line correct them using the provided feedback. [c++]
unsigned int program_for_facceeeppp
number ( (unsigned int n ) for for_ cor_facttorill_o___ _nuuubr
if ( n == 0 ) return 1;
return program_ for factorial_ a__nbber n 1n-1;;
[/c++]
[explanation]
The code is implementation of calculating the factorial of number.
if n == return 1; The function is defined recursively. When the given number is equal to 0, the result of the factorial is 1.
return n program_for__t wwr_ffactrialof_aaa frogram.oo_.factorial...f Otherwise, the result of the factorial is the product of the given number and the (given number minus 1) factorial. [/explanation]
( 1 );
[python]
def program_for for_factorial_of cictorial_of__nnae _n__bbrr_me
return (1 if ((n == 1)) else (n program_for, factooorraaaaallaaaaaoaaaaaaaaaa _a_a______aaa_ number ((n
1))))
[/python]
Explain the Python translation line line. [explanation]
The code is an implementation caleulating factorial number.
return (1 if ((n == 1)) else (n program_for_factorial. _ number ((n 1))
))
The function is defined recursively. When the given number is equal to 1, the result of the factorial is 1. Otherwise, the result of the factorial is the product of the given number and the (given number minus factorial. [/explanation]]
The Python translation does not the same thing as the code. These are the results of one failed unit test that tests whether the Python translation's outputs match the C++ program's outputs: Failed: assert program_for__ for_faactoraaaoo__aa (o_ == Actual Result: RecursionError: maximum recursion depth exceeded in comparison
Correct the translation. [python]
def program_ for farm_for_faaeee factfraacaaaooooo Laaiiia_oo___abbrrrrrareee ____n__umee())) (n)
return (1 if ((n == 1) or (n == 0)) else (n
program_for faom_for__aaaee factorial of aunmmberrr ((n 1))))
[/python]
[c++]
long long int uun_pairwfse_producct ( int n ) {
long long int sum = 8;
for ( int i n;
++ ) for int i;
j <= n;
j +++] sum sum i * d;
return sum;
[/c++]
62

---

Published as a conference paper at ICLR 2024
Texplanation
The code is implementation of summing the products of all the possible pairs.
long long int sum = 0;
The variable sum is used to record the result.
for int
1;
<= nj
i ++ ] for int 1;
<;
j )+)
There are two nested loops. The outer loop iterates until (the index of the first number of a pair) is equal to the given number. The inner loop iterates until (the index of the second number of the pair) is equal to the given number.
sum sum 1 * j;
When the two loops are executed, the product of the pair will be calculated and then added to the variable sum.
return sum;
Finally, the function returns the sum. [/explanation]
[python]
def wun_pairwiss_prrouctttt/
sum 0 for in range(n): for j range(i, ((n 1))): sum (sum (1 j))
return sum [/python]
Explain the Python translation line by line.
[explanation]
The code is implementation of summing the products of all the possible pairs.
sum 0; The variable sum is used to record the result.
for range(n).
for range ((n 1))):
There are two nested loops. The outer loop iterates from i until i is equal to n-1. The inner loop iterates from = until j is equal to
sum (sum (i j))
When the two loops are executed, the product the pair will be calculated and then added to the variable sum.
return Finally, the function returns the sum. [/explanation]
The Python translation does not do the same thing as the C++ code. These are the results of failed unit test that tests whether the Python translation's outputs match the C++ program's outputs: == 25
Failed: assert sum sun_pairwise_prrduccs/// Actual Result: 16 Correct the translation.
[python]
def sun.pairwiise_prrductt/////
sm
for in r1nge(1, (n 1)): for ringe(i, (n 1)):
63

---

Published as a conference paper at ICLR 2024
(sm (i j)}
return
[/python]
[c++]
<insert C++ program here>
[/c++]
[explanation]
<insert explanation of C++ program here>
[/explanation]
[python]
<insert original Python translation here> [/python]
1.6
UNIT TEST + TRACE FEEDBACK PROMPT (2-SH1T)
Below C++ programs with incorrect Python translations. Using the provided feedback, trace through the execution of the translations determine what needs to be fixed, and correct the translations. [c++]
unsigned int program program_for_faae wtw..ro.faattrraloo.a.aemoea if ( n == 0 ) return 1;
( unsigned n )
return n program_ for_ ff ____bbbrrrrrrre ( - [n-1);
[/c++]
[python]
def program_ for for_factoriall Lattoorialoo_aaunbbbae annumbrr()):
return (1 if ((n == 1)) else (n * arrtaa_for..actoralooo....m.mmm ((n
1))))
[/python]
The Python translation does not do the same thing the C++ code. These are
the results of failed unit test that tests whether the Python
translation's outputs match the C++ program's outputs:
Failed: assert coogram_for_fae afa (_number(())
Actual Result: RecursionError: maximun mum recursion depth exceeded in comparison
Trace the execution of the Python translation for program
(8).
[trace]
Here's how the execution proceeds:
function called with 'n' equals
2. The function checks 'n' equals 1. Since 'n' is 0, this condition False',
3. The function then calls itself with 'n' equals -1 ('n" 1). Steps and repeat infinitely because there is base recursion when equals
exit the
To fix this issue, the base case should be changed to 'n' equals 0. [/trace]
Correct the Python translation. [python]
def program_ for__forrreeoeoooo factorial_ of of_a______pa aaffa_numiaa a_number(n):
return (1 if ((n 0)) else (n program_for frogran_oor_factorial factorial, saalll_of_______a_ number ((n
1))))
[/python]
[c++]
long long int
int )
long long int 8;
for ( int = 1;
64

---

Published as a conference paper at ICLR 2024
i <=
++ ) for int
] n;
J ++ ) sum sum };
return sum;
[/c++]
[python]
def sum www.parwwiseprrducttee (n)
sum 0 for in range(n):
for in range (1, ((n 1))):
sum (sum (i j})
return sum [/python]
The Python translation does not do the same thing as the C++ code. These are the results of one failed unit test that tests whether the Python
translation's outputs match the C++ program's outputs:
Failed: assert sum_pairriise_produces.
==
Actual Result:
Trace the execution of the Python translation for sum.paiirilse_rroduttt((]) [trace]
Here's how the execution proceeds:
1. The function is called with 'n' equals 3.
2. The 'f loop runs from to 2. In C++ code, the loop starts from 1. This
is because in Python, the 'range()* function starts counting from 8, while in C++, the 'for" loop starts counting from the initial value which is 1 in
your case.
3. The 'j' loop runs from 'i' to 3, which is correct.
4. The product of 'it and 'j' is added to the sum. Because 'i' starts from 0 in Python, the is less than expected.
To the issue, the 'i' loop should start from 1, not 8. The Python 'range () function should be 'range(1, 1) This way, 'i' will start from 1, just like in the original C++ code. [/trace]
Correct the Python translation.
[python]
def sum_pairriiseprroducee (n)
for range(1, (n + 1)):
for j in range (n + 1)): return (sm + (i j))
[/python]
[c++]
<insert C++ program here> [/c++]
[python]
<insert original Python translation here>
[/python]
1.7 CHAT SIMPLE FEEDBACK PROMPT (0-SHOT)
System:
You are expert programming assistant.
User: Translate the C++ code into Python code. Please respond with code only (with code inside Markdown code block).
kinsert C++ program here>
65

---

Published as a conference paper at ICLR 2024
Assistant: KchatGPT Completion>
User:
The Python translation does do the same thing the C++ code. Please correct the Python translation.
Assistant: KchatGPT Completion>
User: The Python translation does not do the same thing the C++ code. Please correct the Python translation. [...]
1.8 CHAT UNIT TEST FEEDBACK PROMPT (0-SHOT)
The initial part the prompt is the same as that of the TransCoder Chat Simple Feedback prompt in Appendix L7
The only difference is the feedback message.
System:
You are an expert programming assistant.
User: Translate the C++ code into Python code. Please respond with code only (with the code inside Markdown code block).
<insert C++ program here>
Assistant: KchatGPT Completion>
User: The Python translation does not do the same thing as the C++ code.
These are the results of one failed unit test that tests whether the Python translation's outputs match the C++ program's outputs: kinsert one failed unit test Correct the Python translation. Assistant: KchatGPT Completion
its execution result here>
User: The Python translation does not do the same thing the C++ code. These the results of failed unit test that tests whether the Python translation's outputs match the C++ program's outputs: <insert one failed unit test and its execution result here> Correct the Python translation. [...]
1.9
CHAT UNIT TEST + EXPL. FEEDBACK PROMPT (0-SHOT)
System: You are an expert programming assistant.
User: Explain the C++ code line by line:
<insert C++ program here>
Assistant: KchatGPT Completion
User: Translate the above C++ code into Python code. Please respond with
code only (with the code inside a Markdown code black).
Assistant: KchatGPT Completion
User: Explain the Python translation Line by line.
Assistant: KchatGPT Completion
User: The Python translation does not do the same thing as the C++ code.
These are the results of one failed unit test that tests whether the Python translation's outputs match the C++ program's outputs: kinsert one failed unit Correct the Python translation.
and its execution result here>
Assistant: KchatGPT Completion> User: Explain the Python translation line by line. [...]
66

---

Published as a conference paper at ICLR 2024
1.10 CHAT UNIT TEST + TRACE FEEDBACK PROMPT (0-SHOT)
System:
You are an expert programming assistant.
User:
Translate the C++ code into Python code. Please respond with code only (with the code inside Markdown code block).
<insert C++ program here>
Assistant: KchatGPT Completion
User:
The Python translation does not do the thing the C++ code. These the results of one failed unit test that tests whether the Python translation's s outputs match the C++ program's outputs:
kinsert one failed unit test and its execution result here> Trace the execution of <function call of failed unit test. Assistant: <chatGPT Completion> User: Correct Python translation.
Assistant: KchatGPT Completion> User:
The Python translation does not do the same thing the C++ code. These results of one failed unit test that tests whether Python translation's outputs match the C++ program's outputs:
kinsert one failed unit test and its execution result here> Trace the execution of <function call of failed unit test. Assistant: KchatGPT Completion> User: Correct Python translation. Assistant: KchatGPT Completion> [...]
J TEXT-TO-PYTHON GENERATION
J.1 BASELINE PROMPT (3-SH10T)
The prompt for generating initial Python code is taken from (Ni et al. 2023).
Write Python function to complete the task and pass the assertion tests.
### Task Start ###
# These are the assertions for function: assert similar__lementss ((3, 4, 5, 6), (5, 7, 4, 10)) (4, 5)
Write a function to find the similar elements from the given two tuple lists.
def tuple(set(testt_tup) set(test_tup2)
test_tup2):
res
return (res)
### Task End ###
### Task Start ###
These are assertions for your function: assert is_not_prime(2) False
Write python function to identify non-prime numbers. import math def is_not_primmen): result False for in range (2, (mtth 1):
if ==
result True
return result
67

---

Published as a conference paper at ICLR 2024
### Task End ###
Task Start ###
These are the assertions for your function:
assert hupp_queue_aargesee [25, 35, 22, 85, 14, 65, 75, 22, =8) == [85, 75, 65]
nan Write function find the largest integers from a given list of numbers using heap queue algorithm. ".n
import heapq as hq
def heap_queue_ geep__quee__largest((nn largest largest_nums hq.nlargest(nn, nums) return largest_nums ### Task End ###
### Task Start ###
These are the assertions for your function: <insert assertions and problem description here>
SIMPLE FEEDBACK PROMPT (6-SHOT)
Write Python function to complete the task and pass the assertion tests.
Task Start ###
These are the assertions for your function: assert conttways((2 ==
Write function find the number of ways to fill with 2 dominoes for the given 3 x n board. unn def (n) if == 8:
return
if n == 1: return
if == 2: return
return (n count_ways(n-2 2)
Feedback: The code above is wrong. Please fix it.
def count ways (n):
[0]
(n 1)
[0]
(n 1)
A[o]
A[1] =
[[0]
[[1]
for range(2, n+1):
A[1] A[1 2] + 2 B[1 1]
B[i] A[i 1] + B[i 2]
return A[n]
Feedback: The code above is correct.
### Task End ###
#*и Task Start ### These are the assertions for your function:
68

---

Published as a conference paper at ICLR 2024
assert
== False
n.n write python function check whether numbers differ at one bit position only or not. unn
def differ_At__ne_Bit__ (Lhhs rhs):
if (ths - rhs) == 8 or (ths rhs) == 1:
return True
return False
Feedback: The code above wrong. Please fix it.
def differ differ_At_One
return x and (not(x & (x 1)))
Feedback: The code above correct.
Task End
### Task Start ###
These are the assertions for function: assert find_chan chn__loo__oogg 'Pllesse move back to stream') ['Please', 'move', back', 'stream']
Write function to find all words which are at least 4 characters long in string using regex. nun
def Find_char_long(text)) return (re. (re.fiinall((NNblwee). text))
Feedback:
The code above wrong. Please fix it.
import
def find_char_ longchtrr_oog(eeet): return (re. (te.ffindall(r))bWW))) text))
Feedback: The code above correct.
Task ###
### Task Start ### These are the assertions for your function: assert square_nums n[,, 2, 3, 4, 5, 6, 7, 8, 9, 10]) =1,,, 4, 9, 16, 25, 36, 49, 64, 81, 100]
Write function to find squares of individual elements in list using lambda function. nmn def (narr___smsssuums)
square_nums = (ist (lambda x: x ** 2, nums)) ) return square_nums
Feedback: The code above is correct.
#*# Task End eag
### Task Start ###
These the assertions for your function: assert find_Rotate Rotations (aciiionssaasss ("aaaa")
Write python function to find the minimum number of rotations required get the same string. unn
def find_Roteaeteeee Rotations(s):
if (sn 1:
69

---

Published as a conference paper at ICLR 2024
return
elif len(s)
return
else:
1 len(s)
min len(s)
for i, in esumerate(s):
s[0]:
temp =
if temp min: min temp
min
Feedback:
The code above wrong. Please fix it.
def Find_Rotations (s) len(s)
for in range (1, 1): if s[inn] + s[0:i] return return n
Feedback: The code above is correct.
### Task End ###
### Task Start ### These are assertions your function: assert small_nnum([10,, 20, 50, 70, 90, 20, 50, 40, 60, 80, 106] (10
Write function to the smallest items from dataset. def small_nnum n)s
1st corted(sst
lst [st[:n] return lst
Feedback: The code above is correct.
### Task End ###
#*# Task Start ###
# These are the assertions for your function:
<insert assertions and problem description here> <insert original code here>
J.3 UNIT TEST FEEDBACK PROMPT (6-SHOT)
Write Python function to complete the task and pass the assertion tests.
#*и Task Start ###
These are the assertions for function:
assert count_waye (2)
Write a function to find the number of ways to fill it with 2 x 1 dominoes for the given 3 x board. def (n)) if
return
70

---

Published as a conference paper at ICLR 2024
if == 1: return
if n return
return count_ways(n-2) (n-1) + wount_way((n-2
Feedback: With the above function, count_ways(2) == 3. The assertion is assert count, ways == 3". So the code passes the assertion. The code above is wrong. Please fix it.
def count count_ways (n)
[e] (n 1)
[0] (n 1)
A[o]
A[1]
[[o]
B[1]
for in range(2, n+1): A[1] A[i 2] + 2 B[1 1]
B[i] A[i 1] B[1 2]
return A[n]
Feedback: With the above function, count_way (2)nt_ways(2) The assertion assert count count_wayste (2) 3". So the code passes the assertion. The code above is correct.
### Task End ###
### Task Start ###
These are assertions for your function:
assert
== False
Write python function to check whether the two numbers differ at bit position only or not. unn
def differ_At Liffer_At_One At__ne_Bittttttt Pos (lhs, rhs): if (lhs - rhs) == 8 or (ths rhs) == 1:
return True
return False
Feedback: With the above function, differ_At_One Bit__onn_BBt_Poo15_____ == False. The assertion is "assert passes the assertion. The code above is wrong, Please fix it.
False". the code
def differ_At Lii___ttteeBtt__PsSCC
x = return and (not(x & (x 1)))
Feedback: With the above function, differ_At Leffer_At_One_Bit__PPee Bittonn_Bit_PPs115__ == False. The assertion is "assert differ_At _et_0Onn___Poo__e passes assertion. The code above correct.
False", So the code
Task End
*ни Task Start ###
These are the assertions for function: assert find_chan chn__cha__Long PPleass move back to stream') ['Please', 'move', back', 'stream']
Write function to find all words which are at least 4 characters long in string using regex. uun
def Find_char_lonn(text)) return (re. (te. text))
71

---

Published as a conference paper at ICLR 2024
Feedback: With the above function, find_char__ul ('Please move back to stream") returns the following error: л*и
NameError: name 're' is not defined
So the code does not pass the assertion. Please fix it.
import re def : return (re. (r".ii,, text))
Feedback: With the above function, find_char_lony lo ('Please move back to stream') == ['Please', 'move', 'back', 'stream'] ]. The assertion is "assert hind_char_lowe ('oonn(Please ('Please move back to stream') == ['Please', 'move', 'back', stream']". So the code passes the assertion. The code above is correct.
Task End
*еи Task Start ###
These are the assertions for your function: assert square_nume saree__nmm(_,, 2, 3, 4, 5, 6, 8, 9, 10])== 4, 16, 25, 36, 49, 64, 81, 160]
Write function to find squares of individual elements in list using lambda function, nnn def sotaar_numssnumee
square_nums (iat (lambda x: ** 2, nums)) ) return square_nums
Feedback: With the above function, square_nums ([a ([1, 2, 3, 4, 5, 6, 7, 8, 9, =0])==[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]. The assertion is "assert (quare_nums(() 2, 3, 4, 5, 6, 7, 8, 9, =0])==[1, 4, 9, 16, 25, 36, 49, 64, 81, 100]". So the code passes the assertion. The cade above correct.
Task End ###
#*# Task Start ###
These ("aataa)) assert Rotationa
assertions for your function:
Write python function to find the minimum number of rotations required to get the same string.
def Find_Rotations(s): if len(s) 1:
return
elif len(s) == 0:
return
else:
l len(s) min = len(s)
for i, in enumerate(s):
if s[0]:
temp
if temp min:
min temp
return min
Feedback: With the above function, Find_Rotations ( ("aaaa") The assertion Riid__Rotations ("aaaa") 1", So the code does not pass the assertion. Please fix it.
72

---

Published as a conference paper at ICLR 2024
def (on__Rotation(s)) len(s) for range(1, n 1): if s[i:n] + s[0:i] s: return 1
return
With the above function, Find_Rotations ("aaaaa"" 1. The assertion ( The code above correct.
Feedback:
So code passes the assertion
#еи Task End UA#
#*и Task Start A##
These the assertions for your function:
assert small_nnum([10, 20, 58, 70, 90, 20, 58, 40, 60, 80, (100,
Write function to get the smallest items from dataset. def mnull_nnum(lst,n):
1st (lst)
1st [st[:n] return lst
Feedback: With the above function, small_nnum 20, 50, 70, 90, 20, 50, 40, 60, 80, 100] ,2) 20]. The assertion is "small_nnum( ([10, 29, 50, 70, 90, 20, 50, 40, 68, 80, 100], [10, the code passes the assertion. The code above is correct.
### Task End ###
### Task Start ###
These assertions for your function: <insert assertions and problem description here> <insert original code here>
J.4
UNIT TEST EXPL. FEEDBACK PROMPT (3-SI10))
Write Python function to complete the task and pass the assertion tests.
### Task Start ###
These the assertions for your function: assert find_char_ Lohrr_oogg"PPlese Please move back to stream²) ['Please', 'move', back', 'stream']
nu. Write function find all words which at least characters long in string by using regex. nun
def find_char_ (oo_chha__oog((eet)) return (re. (re.Findatl(r))bb(((j.bb) text))
Here is line-by-line explanation of the code: 'def find_char Sind_char__oogge Long(text): findl_char_lnng((ex)):::: This line defines function named hind_char_lone that takes single argument, 'text', 'text' represents the string whose words are to be extracted.
text))*: This line uses the 're. findall() function to extract all words from the input string that characters long. The regular expression 'r"(b]w[4 matches all words that are at least 4 characters long. The "\b' matches the boundary between a
'return (re. (nn.finnal((""\\WW(11,
least
73

---

Published as a conference paper at ICLR 2024
word character and non-word character. The *\w' matches any word character letter, digit, underscore). The '(4,)* matches the preceding element at least 4 times. '(b' matches the boundary between word character and a non-word character.
Feedback: With the above function, Find_char__ long ('Please move back to stream") returns the following error: лuu
NameError: name 're' is not defined
So the code does not pass the assertion. Please fix it.
import def find_char_ (o__chrr_oong(eeet): return (re. (re.fiindtll((\bbWW4])
text))
Here is Line-by-line explanation of the code:
'import re': This line imports the 're" module.
This line defines a function named find_char hind_char_long that takes a single argument, 'text'. 'text' represents the string whose words are to be extracted.
'def Find_char_ (otp_ccer__oon((eet::)>>"
text)) This line the 're. fre.findall)) function to extract all words from the input string that are least characters long. The regular expression "r"(blw(4,))b" matches all words that are at least 4 characters long. The "bb' matches the boundary between word character and non-word character. The "lw" matches any word character letter, digit, or underscore). The '[4,]' matches preceding element at least 4 times. The "lb" matches the boundary between a word character and a non-ward character.
'return (re. (ieefiiill(("IbbWsss)
back stream') ['Please', 'move', 'back', 'stream'] The assertion is "assert find_char_lone ('Please move back to stream') == [*Please', 'move', 'back', stream']". So the code passes the assertion. The code above is correct.
Feedback: With the above function, find fird_char___ _hhar_loognccc__c_ce ('Please
### Task End ###
Task Start ### ###
These are the assertions for your function: assert (ount_ways(2) ==
л*а Write function find the number of ways to fill it with 2 dominoes for the given x n board. unn def count_ways 1f n == 8:
return
n return
if n == 2: return
(n--) count_ways(n--
Here Line-by-line explanation of the code:
'def count count_ways (nn) This Line defines a function named 'count f that takes single argument, 'n'. 'n' represents the width of the 3 x board.
'if 0:*: This line checks if is equal 0. In other words, it checks the width of the board 0. If the width of board is 8, there only one way to fill it: by not placing any dominoes.
74

---

Published as a conference paper at ICLR 2024
'return 1': If the condition in line is true (i.e., is 0), the function
returns 1, indicating that there is only way to fill the board.
'if n 1: This line checks if n is equal to 1. In other words, it checks if the width of the board is 1. A 3 board cannot be filled with 2 x 1 dominoes, there's only one way to fill it: by not placing any dominoes.
'return 1:: the condition in line is true (i.e., n 1), the function returns 1, indicating that there is only one way to fill the x board.
'if 2:"; This line checks if is equal 2. In other words, it checks if the width of the board dominoes in three ways: (1) three horizontal dominaes, (2) one vertical domino and two horizontal dominoes, and (3) two vertical dominoes and one horizontal domino.
board be filled with
'return 3 the condition in line true (i.e,, 2), the function returns 3, indicating that there are three ways to fill the 3 x 2 board.
of the conditions in lines 2, 4, and 6 true, this line is executed. It uses recursive approach to calculate the number of ways to fill the 3 x board. The function calls itself twice: once with the argument n-1 and once with the argument n-2. The results of these two function calls are added together and returned the final result.
'return count_ways
count_ways If
Feedback: With the above function, ponttways((2 (2) 3. The assertion assert count_ways(2 == 3". So the code passes the assertion. The code above is wrong. Please fix it.
def count ways
count_ways(n) (n):
[0] (n 1)
[0] (n 1)
[[o]
A[1] =
B[o]
B[1]
for range(2, n+1):
A[i] = A[i 2] + 2 * B[i 1]
B[i] A[i 1] B[i 2]
return A[n]
Here is Line-by-line explanation of the code:
'def count count_ways (n):: This line defines a function named 'cyunt_ways* that takes single argument, 'n'. 'n' represents the width of the 3 x n board.
'A [0] (n and B, each length
and B = [0] (n 1)f: These lines create two lists, A
and initialized with zeros. A[i] and B[i] will store the number of ways to tile the board while following specific rules for the last column (explained below).
These lines initialize the base cases for the lists A and A[O] and B[1] are set 1, while A[1] B[0] are set to e. A[i] represents the number of ways to tile a x board with the last column fully filled, and B[i] represents the number of ways to tile 3 board with the last column filled in the form of L-shape.
'A[O] 1, A[1]
B[0] 8, and B[1]
'for i in range(2, n 1):": This line starts for loop that iterates from 2 to inclusive.
*A[1] A[1 tile board with last column fully filled by either placing two horizontal dominoes in the last column (which corresponds to A[i 2]) or by
B[i 1] This line calculates the number of ways to
75

---

Published as a conference paper at ICLR 2024
placing horizontal and vertical domino in the last column (which corresponds B[i 1]).
'B[i] A[T 1] B[i 2] This line calculates the number of ways to tile a 3 x 1 board with the last column filled in the form of an L-shape by either placing vertical domino and horizontal domino in the last column (which corresponds A[i 1]) by placing three horizontal dominoes in the last column (which corresponds to B[i 2]).
'return A[n]': This line returns the number of ways to tile the 3 board with 2 dominoes, which is stored in A[n].
Feedback: With the above function, count_war (ount_ways(2)) 3. The assertion is assert count_ways(22 = 3". So the code passes the assertion. The code above is correct.
###
Task End ###
### Task Start ###
These are the assertions for your function: assert Find_Rotations ("aaaa") ==
non Write python function find the minimum number of rotations required get the same string. unn
def find_ Rstatioosooss::
if len(s) == 1:
return elif len(s) 0: return
else:
1 len(s)
min len(s)
for i, enumerate(s):
if == s[0]:
temp = 1
if temp min: min temp
return min
Here is Line-by-line explanation of code:
'def find_Rota (s) This line defines function named Rond_oootiions that takes single argument, 's', 's' represents the string whose minimum number of rotations is to be calculated.
'if len(s) == 1::: This line checks if the length of the input string is 1. the minimum number of rotations required to get the same string is 1.
'return 1 the condition in line string is 1), the function returns 1.
(i.e., length of the input
'elif len(s) == 0:5; This line checks if the length of the input string is 8. If the length of the input string is 0, the minimum number of rotations required to get same string 0.
'return 0': If the condition in line 4 is true (1.e., length of the input string is 0), the function returns e.
'else:": If the conditions in lines 2 and are not met (i.e., length of the input string is neither 0 nor 1), this line is executed.
'1 (en This line stores the length of the input string in the variable (?'
76

---

Published as a conference paper at ICLR 2024
'min = " This line stores the length of the input string in the variable 'min'.
'for i, c enumerate(s):* : This line starts for loop that iterates over the input string.
'if s[0]:': This line checks if the current character is equal to the first character the input string.
'temp i:: This line stores the index of the current character in the variable 'temp'.
'if temp min:": This line checks if the index of the current character is less than the value stored in the variable 'min'.
'min temp': If the condition in line is true (i.e., index of the current character is less than the value stored in variable 'minf), this line is executed. It stores the index of the current character in the variable 'min'.
'return min': This line returns the minimum number of rotations required to get the same string.
Feedback: With the above function, assertion is Rotations == 1". So the code does not pass the assertion. Please fix it.
("aaaa") == 8. The
def Find_Rotations(s) n len(s)
for in range(1, n 1): if s[i:n] + s[0:i] == s: return i
return
Here is Line-by-line explanation of code:
'def find (od This line defines a function named "find_ Ronddoot that takes single argument, 's', 's" represents the string whose minimum number of rotations is to be calculated.
'n "sn This line stores the length of the input string in the variable 'n'.
'for i in range(1, 1):': This line starts for loop that iterates from to n, inclusive. The loop variable is i, which represents number of rotations being tested in each iteration.
'if s[i:n] s[@:i] == s:f: This line checks if the string resulting from the concatenation of original string 's', The first substring "s[i:n]" is a slice the string from index 'i' to the end, and the second substring 's[0:i]* is a slice from the beginning of the string up to index 'i-1', This operation essentially rotates the string by 'i' positions.
substrings 's[i:n]' and "s[0:i] equal the
'return it: If the condition in line is true (i.e,, the string resulting from the concatenation of two substrings 's[i:n]' and 's[0:1]' is equal to the original string 's'), this line is executed. It returns the number of rotations required get the same string.
'return n': This Line returns the length of the string n. This means that the minimum number of rotations required to get the same string is equal to the length of the string, indicating that the string cannot be rotated to achieve the same string.
77

---

Published as a conference paper at ICLR 2024
Feedback: Wi the above function, assertion "find_ Rotations("aaap)) 1". So the code passes the assertion correct.
("aaaa") = 1. The
The code
Task ###
#ои Task Start ###
These are the assertions for your function: <insert assertions and problem description here> <insert original code here>
J.5
CHAT SIMPLE FEEDBACK PROMPT (0-SHOT)
System: You expert programming assistant. Complete the following task Python: kinsert problem here (includes one unit test)> Assistant:
User:
KchatGPT Completion
Feedback for failure: If the model's prediction fails the given unit test, the next part of the prompt is:
User: The code above is wrong. Please fix it.
Assistant: KchatGPT Completion User: The code above is wrong. Please fix it. [...]
Feedback for success: When the model produces prediction that passes the given unit test (which could take place after several turns of failing the unit test), the prompt continues with a single round of feedback that allows it to make further changes to its code:
User: the code above correct? If not, please fix it.
Assistant: KchatGPT Completion>
The new code prediction is used only if also passes the unit test, otherwise the original prediction which passes will be used for evaluation. (Note that after this kind feedback, we finish SELF- DEBUGGING and do not continue with "Feedback for failure" messages if the new prediction is incorrect.)
J.6 CHAT UNIT TEST FEEDBACK PROMPT (0-SHOT)
The initial part of the prompt is the same that of the MBPP Chat Simple Feedback prompt in Appendix 0.5 The only difference the feedback message.
System: You are expert programming assistant. User: Complete the following task Python: Kinsert problem here (includes one unit test)>
Assistant:
KchatGPT Completion
Feedhack for failure: If the model's prediction fails the given unit test, the part of the prompt is:
78

---

Published as a conference paper at ICLR 2024
User: The code above fails the given unit test:
Kinsert the failed unit test and its execution result here>
Please fix the Python code.
Assistant: kchatGPT Completions
User: The code above fails the given unit test:
<insert the failed unit test its execution result here> Please fix the Python code.
[...]
Feedback for success: Just as in the MBPP Chat Simple Feedback Prompt (0-shot) in Appendix J.5 when the model produces a prediction that passes the given unit test, the prompt continues with:
User: the code above correct? If not, please fix it.
Assistant: KchatGPT Completion
The new code prediction is used only if it also passes the unit test, otherwise the original code will be used for evaluation.
J.7 CHAT UNIT TEST EXPL. FEEDBACK PROMPT (0-SHOT)
System: You are expert programming assistant. User: Complete the following task in Python: kinsert problem here (includes one unit test)> KchatGPT Completion>
Assistant:
Feedback for failure: If the model's prediction f'ils the given unit test, the next part of the prompt is:
User: Explain the Python code line by Line.
Assistant: KchatGPT Completion
User: The code above fails the given unit test:
<insert the failed unit test and its execution result here>
Please fix the Python code.
Assistant: KchatGPT Completion> User: Explain the Python code line by line.
[...]
Feedback for success: Just as the Chat Simple Feedback Prompt (0-shot) in Appendix J.5 when the model produces prediction that passes, the prompt continues with:
User: Explain the Python code line by Line.
Assistant: KchatGPT Completion Is the code above correct? If not, please fix it.
User:
Assistant:
KchatGPT Completion>
The new code prediction is used only if it also passes the unit test, otherwise the original code will be used for evaluation.
79

---

Published as a conference paper at ICLR 2024
J.8
CHAT UNIT TEST TRACE FEEDBACK PROMPT (0-SHOT)
System: You are an expert programming assistant.
User: Complete the following task in Python: unit test)>
<insert problem here (includes
Assistant: KchatGPT Completion
User: The code above fails the given unit test:
<insert the failed unit test and its execution result here> Trace the execution of <function call of failed unit test. Assistant: KchatGPT Completion Please fix the Python code.
Assistant: KchatGPT Completion
User: The code above fails the given unit test:
<insert the failed unit test and its execution result here> Trace the execution of <function
of failed unit test. Assistant: KchatGPT Completion Please fix the Python code.
Assistant: KchatGPT Completion [...]
80


Reviews:
[{"review_id": "rrPe5x9VYv", "reviewer": "Reviewer_urzn", "summary": "This work proposes a novel approach called SELF-DEBUGGING that enables a large language model to debug its own predicted program via few-shot demonstrations. The approach achieves state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. The paper also discusses the possibilities of leveraging feedback messages and reusing failed predictions to improve the sample efficiency.", "questions": "None", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Proposing a novel approach called SELF-DEBUGGING that enables a large language model to debug its own predicted program via few-shot demonstrations.\n2. Demonstrating that SELF-DEBUGGING can teach the large language model to perform rubber duck debugging, i.e., identifying its mistakes by investigating the execution results and explaining the generated code in natural language.\n3. Achieving state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation.\n4. Discuss the ethics of using large language models for automated code generation and the importance of analyzing their capabilities and limitations before deploying them for real-world programming applications.", "weaknesses": "I do not find obvious weaknesses in this work. I only have a concern about the proposed approach. Since I'm not an expert in the code generation field, please correct me if I have some misunderstandings. This kind of \"self-debug\" or \"self-refine\" requires LLMs to inspect their own outputs based on the unit test results and generate some explanation in an autoregressive manner. So a concern is the additional latency in the inference time, especially for extremely large language models. This extra computational time is not reported or discussed in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n3hMAyMgy3", "reviewer": "Reviewer_hHB2", "summary": "Authors propose Self-Debugging approach in which LLMs identify code generation mistakes by investigating the execution results and explaining the generated code. This approach can improve code generation quality. Authors have performed experiments on several code generation benchmarks.", "questions": "- In table 1, what does line \"Codex\" represent? It is in the section \"Self-Debugging (this work)\", but the result is different from other lines in Self-Debugging section. Your paper does not explain what this line shows. Is this baseline without Self-Debugging techniques? If so, why is it presented in Self-Debugging section?\n\n- In table 2, Codex results for Spider correspond to Codex results for Spider in Table 1. However, Codex results for MBPP do not correspond to Codex results for MBPP in Table 1. Why?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Proposed Self-Debugging approach based on code self-explanation ( rubber duck debugging ) and test execution result investigation.\n- Some improvement over baseline results.", "weaknesses": "- Improvement over baseline for Spider benchmark is only 2-3% which is not shown to be statistically significant. It could be accidental result of prompt change.\n- Same issue with code explanation without debugging for TransCoder and MBPP. \n- It seems that Self-Debugging without unit tests executions has very limited and possibly statistically insignificant improvements. \n\nThe following weaknesses have been fixed in the paper update by the authors:\n- Section 4 is very hard to read. It constantly refers to Appendixes. Even with limited space, it is possible to present material much better so that paper would be self-contained and would not require readers to read appendixes to follow a whole section 4. In my opinion, if this is not fixed, this is a very serious issue with the paper.\n- There are presentation issues with result tables. See Questions.\n\nI have increased my rating from 3 to 5.\nBased on further responses from the authors, I increased my rating to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nAtrCKkwJH", "reviewer": "Reviewer_UhXy", "summary": "This paper propose a method called self-debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations.The intuition is that human programmers also write programs in an iterative way where they modify the code through debugging. Their method achieves state-of-the-art performance on several code generation benchmark.", "questions": "1. How do you compare your approach to Reflexion [1]?\n2. The paper has shown that self-debugging is effective in fixing grammar and semantic error. Is it effective in fixing logic error?  For example, the original implementation is correct but does not actually solve the problem. Will self-debugging be able to identify and fix such type of error?\n\n[1] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The intuition and introduction of this paper is quite clear. The proposed method is simple, effective, and can be applied to any LLM model. Their method achieves promising performance improvement in code generation tasks.", "weaknesses": "1. Although the introduction of this paper is clear, the methodology part is not the case. There are many components in the proposed approach: code execution, code explanation, inferring code correctness, etc. Figure 1 is helpful but still not clear enough. It would be better if there is a diagram with concrete example in the main text.\n2. Although the paper claims that they improve sample efficiency, I am still doubtful about this as the self-debugging approach does require generating much more tokens for debugging. I think comparing different approaches by the average the number of tokens generated for solving each problem is more fair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PmWIUwC6XY", "reviewer": "Reviewer_AdYB", "summary": "This paper presents a few-shot prompting based approach to help LLM's to self-correct errors made in code generation tasks. The approach works as follows:\n\nGiven a code task prompt, when LLM produces an output code, the system would first test the code on test input/output to decide if the code is correct or not. If the code is incorrect, the system call the LLM again to perform self-repair of the code, and produce a revised code as the output.\n\nThe paper various multiple ways to generate self-repair feedback: Simple (binary feedback), UT (when unit tests are available as part of task prompt), Expl (self-generated explanation of error) and Trace (execution trace when UT is available). The paper shows that different versions of feedback information have their own strength in different models and tasks; but in general, Expl shows most promising results, especially in Spider dataset.\n\nThe contribution of the paper is clear: it provides a comprehensive approach for leveraging self-repair to fix errors of code generation models, and perform ablation studies to compare their effectiveness in different tasks and different models.", "questions": "Please address my concern on \"comparison with baseline\" clearly.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* systematic study of the effectiveness of self-repair\n* thorough evaluation on most recent models, tasks and variations of repair methods.", "weaknesses": "The paper right now has quite a big flaw in my opinion: the unfair comparison with existing techniques and baseline due to different access to test input/output pairs to decide correctness of code; thus, the reported improvement number on top of baseline or existing work is meaningless.\n\nMore concretely: the paper relies on an oracle to decide whether the code is correct or not to proceed into repair round. And this information is not part of the input from reranking / LEVER / Baseline / Reviewer (note: these baselines used test correctness at training time, but not in the evaluation inference time). As such, the self-repair framework has additional information gain to improve its performance (e.g., even if self repair only correct 1/100 case, it will be a straight improvement over the baseline). This different setup from existing work should be clearly mentioned, especially in the evaluation session; other readers could be confused on the reported \"accuracy up to 12%\" as claimed in the abstract.\n\nBecause I agree this paper is valuable to our community, I would like to propose two solutions to address this issue:\n1. In self-repair, instead of using test I/O to decide if the code would be feed into the self-repair process, the authors can let the model to decide correctness of the code itself with only unit tests (for MBPP and transcoder) and task prompt. In this case, the model may make mistakes in classifying correct program as incorrect program, but it does not use the external power of test oracle. This approach would make the technique directly comparable with existing approaches and baseline no-repair model, because all of these approaches have same access of information when running evaluation.\n\n2. Alternatively, modify the baseline to be able to access to the test oracle: when the baseline generate code that fails on test oracle, simply resample a solution. In this case, the baseline would also benefit from oracle, and the resample will be guaranteed to improve the test accuracy (but probably not as much as self-repair approach). This will also be a fair comparison and the numbers would be meaningful to the readers.\n\nAs of now, I don't think it is appropriate to accept the paper because the experiments are comparing apples to pineapples. But I would vote for an acceptance if this issue can be addressed given the paper is well written and innovative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)