LLM-Assisted Code Cleaning For Training Accurate Code Generators
Authors: Naman Jain
Tianjun Zhang
Wei-Lin Chiang
Joseph E. Gonzalez
Koushik Sen
Ion Stoica
Venue: ICLR 2024 poster (2024)
Keywords: Code Generation
Data Quality
Synthetic Data
Primary area: generative models
TL;DR: We highlight importance of code style for improving data quality

Abstract: Natural language to code generation is an important application area of LLMs and has received wide attention from the community. 
The majority of relevant studies have exclusively concentrated on increasing the quantity and functional correctness of training sets while disregarding other stylistic elements of programs. More recently, data quality has garnered a lot of interest and multiple works have showcased its importance for improving performance. In this work, we investigate data quality for code and find that making the code more structured and readable leads to improved code generation performance of the system. We build a novel data-cleaning pipeline that uses these principles to transform existing programs by 1.) renaming variables, 2.) modularizing and decomposing complex code into smaller helper sub-functions, and 3.) inserting natural-language based planning annotations. We evaluate our approach on two challenging algorithmic code generation benchmarks and find that fine-tuning CodeLLaMa-7B on our transformed programs improves the performance by up to \textbf{30\%} compared to fine-tuning on the original dataset. Additionally, we demonstrate improved performance from using a smaller amount of higher-quality data, finding that a model fine-tuned on the entire original dataset is outperformed by a model trained on one-eighth of our cleaned dataset. Even in comparison to closed-source models, our models outperform the much larger AlphaCode models.

Published as a conference paper at ICLR 2024
LLM ASSISTE CODE CLEANING FOR TRAINING AC- CURATE CODE GENERATORS
Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E. Gonzalez, Koushik Sen & Ion Stoica University of California, Berkeley
(naman_jatl Sain,, t junzz we ch ang,, jegonzal ksen, istoica @berre edu
ABSTRACT
Natural language to code generation is an important application area of LLMs and has received wide attention from the community. The majority of relevant studies have exclusively concentrated on increasing the quantity and functional correct- ness of training sets while disregarding other stylistic elements of programs. More recently, data quality has garnered lot of interest and multiple works have show- cased its importance for improving performance. In this work, we investigate data quality for code and find that making the code more structured and readable leads to improved code generation performance of the system. We build a novel data-cleaning pipeline that uses these principles transform existing programs by 1.) renaming variables, 2.) modularizing and decomposing complex code into smaller helper fub-functions, and 3.) inserting natural-language based plans via LLM based transformations. We evaluate our approach on two challenging algo- rithmic code generation benchmarks and find that fine-tuning CODELLAMA-77 on our transformed modularized programs improves the performance by up to 30 % compared to fine-tuming on the original dataset. Additionally, we demonstrate im- proved performance from using a smaller amount of higher-quality data, finding that model fine-tuned on the entire original dataset is outperformed by a model trained on 15% of our cleaned dataset. Even in comparison closed-soures mod- els, our models outperform the much larger ALPHACODE models (Li et al. 2022)
INTRODUCTION
Natural language to code generation has witnessed considerable advances in recent years with the advent of large language models (LLMs for brevity). These advances primarily arise from train- ing on large web-scale data and are measured based the functional correctness of the programs. Thus, other aspects like readability, structuring, and styling and how they affect training and data quality are largely ignored by these works. On the flip side, many recent works have demonstrated the effectiveness of training on higher quality data during both pre-training (Li et al. 2023d) and fine-tuning (Zhou al. 2023 Cao al. 2023) phases. Even within the code-generation domain, Gunasckar et al. (2023) demonstrated the benefits of training on a "textbook" quality dataset, gener- ated synthetically using the GPT-3.5-TURBB model (Ouyang et al. (022).. However, these works do not provide an understanding of the factors that actually improve the data quality.
In this work, we show that using programs following good programming practices and allowing for more readability leads to improved code generation performance compared to using programs that do not follow these practices. We use these insights to build a novel automated code data- cleaning pipeline that transforms programs while maintaining functional correctness using input- output examples. In contrast to prior works that curate high quality datasets by directly generating new data using LLMs. here translate existing datasets into their parallel cleaned versions while identifying attributes that actually improve data quality.
We use LLMS to perform the transformations used in our data-cleaning approach. We demonstrate that instruction-tumm models can take a user- user-identified attribute of data quality as a natural language instruction and perform the transformation accurately. Our approach leverages the disparity in diffi- culty between generating a solution and editing an existing one. Therefore, it is particularly effective in domains where the existing model struggles generate correct solution but can effectively edit

---

Published as a conference paper at ICLR 2024
Original Program
Renamed Program
inttiinout()),, E112)
run_vertices, colors a Inteipput)),, E d5s, erges, grape tisttrangetn)). U.
ess,, edges, di tisterangetn)l, U.
Tes--ces oquala checking
for ranpećn 1):
rangelmux vertices
fyy eaplint, (ppit()-split(j)
v a eaplint, snput()-aplittd) cstera(e] colors dau[v]
clvis
var
orogram to descriptive, meaningful and consistent
while duu popt),, prpll dlul-arpendie arppenstt
white v = douletges, popti,, dsuledges.pop()
Variable Renaning
graph [s], al.-aaeedddy
graph sspeensts)
def bfstx]:
def bistel:
current_vertizesst)))
printibrstorsiel) //
prantlatsterrsi0 //
Mooutarizetion
Truncated Problem Statement
734-E)
painted black white them have the color. poteettn operation changes color of vertioes such that verioce the shorleet path from to have the same color. Find the mir iium operations required
There vertices
Cerion the
Planning Comment
Modularized Program
main()
reed edd_eedss
This is. the axin functing ion of the prosree.
takes inout using read, edges, the biss
grash
Find,
while verter diu(vertex): vertex dau lvertoxl retur versex
read_ edpeet)
function reads
def menge_ treestr,, V, douts U, v, deu ...
nerge.tteeee treesta, v, This
Plan Annotation
the
bus build_prophie sil.annhhnun..vrttieer colors, merge_ pew_treess..... (r..st...)
Generate natural language descript lowing
fovd_root lex,
return graph
functions the program
def brsck, ner_verices,, graph):
5. bustd_graph (auu vertice,,,,, coters, edges)
graph
find reot neth...
msind]:
num vertices, on colors, edges = read read edges() edges paillddgraphhurs vertices, colors,
graph)
This function implamnts traadth-farst-searee
(rrrt ...312/21
The overview of code cleaning approach. We apply instruction-tuned L.LSs to transform existing datasets by providing natural language prompts and use input-output examples to maintain function equivalence between original and transformed programs. Our cleaning approach works in three steps. The top-left figure depicts the original program from the dataset. This program first undergoes variable renaming (top-right figure). Next, the renamed program decomposed into constituent sub-funce fonfffnnttinsss and converted into modularized program (bottom-right figure). Finally, we generate a natural-language plan from the modularized program by summarizing the functions in top-down manner (bottom-left figure). This plan is prepended to the program as comment. The middle-left figure presents the truncated problem statement
Figure 1:
given solution. We perform data-cleaning transformations in three iterations: 1) renaming variables 2) modularizing complex code into subfunctions, and 3) adding planning annotations.
Figure D provides an overview of our approach. Notice that the variable renaming step at the top adjusts the variable names to be contextually relevant (e.g. a to root_u and to graph). The mod- ularization step (depicted on the right) identifies and decomposes the original program into several smaller subfunctions such as find root, merge trees, build graph, etc. It then implements these subroutines and assembles the modular program. Finally, our planning step (depicted at the bottom) constructs plan by summarizing functions in top-down fashion (starting from the main).
We evaluate our approach in a niche, yet challenging, domain of algorithmic code generation. The goal is to generate a program for given problem statement. The task is challenging because it requires both high-level algorithmic reasoning and low-level coding and evaluated using a strict functional correctness metric. We use two well-known algorithmic code generation benchmarks, namely APPS (Hendrycks et al. 2021) and CONTESTSSS (Li et al. 2022). We transform the cor- responding programs in the training sets and obtain parallel datasets from cleaning approach. Additionally, we utilize onput-output examples to maintain functional equivalence between the orig- inal and transformed programs. We qualitatively analyze the generated dataset and find that it uses
2

---

Published as a conference paper at ICLR 2024
smaller helper sub-functions, each often implementing standard algorithm or key program fune- tionality, and provide more in depth findings in Section [4.] We further assess the impact of the transformed datasets on the performance CODELLAMA-77 model on the various collected datasets. Our findings reveal that the model fine- tuned on our modularized dataset outperforms the model fine-tuned on the functionally equivalent original dataset by up to 30% Beyond, performance improvement, we also demonstrate that im- proving data quality improves the data efficiency. In particular, model fine-tuned on the entire original dataset is outperformed by a model trained on just 15% of our cleaned dataset.
our downstream code generation task. We fine-tune
We next study improving planning in a supervised learning setup similar prior works (Fu al, 2023 Li et al., 2023b). While we observe limited improvements in planning, we disentangle plan- ning vs coding capabilities and find that our fine-tuned model is capable of using gold-annotated plans, extracted from the ground-truth solutions to accurately generate solutions for the complex programs. This highlights planning for complex problems remaining a key bottleneck that does not seem to improve by merely increasing training datasets. Finally, in comparison to existing baselines, our fine-tuned models outperform the larger ALPHACODE (Li et al. 2022) models.
METHODOLOGY
In this section, present general data transformation approach and then instantiate it for per- forming code data cleaning.
2.1 TRANSFORMATIONS FOR DATA CLEANING
Given dataset DD consisting of N instances d,, such that, D [d.) To achieve desired data cleaning specification, the user additionally provides a data-cleaning instruction I, which highlights an attribute that needs be modified. Optionally, we also use an oracle equivalence checker (O) which ensures that the transformed data instance d, consistent with the original input based on some desired metric. For example, we can use edit-distance or functional equivalence based on input-output examples as our oracle checker.
We use pre-trained language model (denoted by .M) generate the transformed instance (d,) by prompting the model with the transformation instruction (Z) and the original answer (y). We can perform either zero-shot or few-shot prompting for performing the data cleaning operation. Finally, we extract the instance d, generated by M, and apply our oracle equivalence checker (O) to ensure consistency with the original data. O(d,, d.) O, i.e., the oracle reports a failure, we reject the generated output and retry the example within sampling budget.
While our transformation approach does not provide any guarantees about the quality of the per- formed transformation and relies LLMs, empirically observe that instruction-tuned LLMS can perform various unstructured data cleaning steps quite effectively. We provide a detailed anal- ysis of the generated outputs for algorithmic code generation setting in Section 4.1 Finally, in accordance with existing literature on prompting LLMs, we found that using simple and precise, low-level instructions improves the performance and accuracy of the models in performing the oper- ations. Thus, for complex data cleaning operations (refactoring), we find improvements by breaking it down and performing multiple operations iteratively (renaming followed by modularization).
2.2 CODE AATA-CLEANING
We apply our transformations-biee based data cleaning approach to programming data. Coding requires both low-level programming and high-level reasoning or planning skills. Therefore, we propose three-step cleaning pipeline that improves the readability and program structuring targeting the low- level coding skills and inserts natural-language based plans data targeting the high-level reasoning skills. Our steps are detailed below.
1. Rename variables. This step renames the variables in the program, making them descrip- tive and casier to follow. Figure [ top provides example of this transformation.
2. Modularize functions. Problem decomposition has been identified as a key approach for improving the reasoning capabilities of models (Zhou al., 2022 Wang et al. 2023). We
3

---

Published as a conference paper at ICLR 2024
split
APPS-INTRODUCTORY
APPS-INTERVIEW
APPS-COMPETIIION
CODE-CONTESTS
train
42
1247
361
7132
Problems count
702
2699
309
165
train
200
Tests count
39
200
Solutions count
train
18394
5060
98582
Table 1: Details about the number of problems, the median number of test cases per problem, and the number of solutions in the APPS and CODE-CONTESTS datasets.
identify program decompositions and transform the program by extracting their functional- ity into smaller helper functions. Figure right
right provides an example of this transformation.
3. Plan annotations. This step summarizes the helper functions in the already modularized program and prepends it to the programs in the form of a natural language plan. These nat- ural language descriptions are analogous to prompting approaches that are used for solving reasoning problems like chain-of-thougle prompting (Wei et al. (022), parsel (Zelikman et al. 2023), etc. Figure bottom provides an example of this transformation.
Additionally, while performing these transformations, we use the test cases provided the dataset to construct oracle equivalence checker (O). It ensures that our transformed programs maintain functional equivalence to the original program.
3 EXPERIMENTAL SETUP
In this section, we detail our experimental setup and implementation. Section outlines the bench- marks and metrics used for the algorithmic code generation task, while Sections 3.2 and 3.3 delve into the specifics of our code cleaning approach and fine-tuning experiments respectively.
3.1 BENCHMARKS
We use two standard algorithmic code generation benchmarks, APPS and CODE-CONTESTS The benchmarks provide a collection of problem statements described in natural language and corre- sponding test cases. The goal is to generate a program that successfully solves the problem. The evaluation performed using strict functional correctness metric.
APPS (Hendrycks et al. 2021). This benchmark includes 10,000 problems, evenly split between training and test sets. sourced from multiple open-access competitive programming websites. It is further divided into IIPPS-NTRRDUCTRRY APPS-INTERVIEW, and based on problem difficulty. this study, we only consider problems sourced from a subset of the competition websites based on the number of test cases provided.
subsets
CODE-CONTESTS (Li al., 2022). This benchmark includes 13,328 problems in the training set and 165 problems in the test set. We only use subset of the training split that includes python solutions satisfying the provided test cases. Additionally, since the training set provides over perform LSH based near-deduplication the solutions and
hundred solutions per problem, limit them to a maximum 25 solutions per problem.
Table 1 and Appendix A provide further details about our final datasets.
Metries. We assess the code generation performance of the models using the PAsS@ K metric (Kulal al. 2019 Chen et al. 2021). which evaluates the functional correctness of generated programs. For each problem, we generate solutions (where N 2 2K) and compute the expected number of scenarios in which the problem is solved at least once when sub-selecting a random sample of K solutions. We vary K in (1, 10, 25} for APPS dataset and [1, 10, 100) for the CODE-CONTESTS benchmark. We present more details about sampling hyperparameters in Appendix
3.2 DATA TRANSFORMATIONS
We apply data transformation approach on the APPS and CODE-CONTESTE datasets. Unless specified otherwise, we use GPT-3.5-TURBO as our default language model M to perform the trans- formations and default temperature 0.3. In case of failure, retry up 5 iterations. We

---

Published as a conference paper at ICLR 2024
Dataset
Notation
Applied On
Transformation Instruction (I)
Buse
Deriginal
Rename
Penaaee
Doviginat
Rename the variobles program to be descriptive, meaningful, consistent Refactor the above program making more modudrr with smaller and meaningful helper functions with good descriptive names for the helper functions
Modularize
D moddulrr
D Dresuame
Planming
Dmoduler
Generate notural languoge description for the following functions in the program
Table 2: Transformed datasets generated by our code cleaning approach. For each transformation, we have provided the corresponding notation, the transformation instruction used perform the cleaning step and the dataset the transformation was applied on.
obtain three parallel datasets at the end of our cleaning process, one for each of renaming, modu- larization, and planning (note that the transformations are applied sequentially). T a summary of the generated datasets along with the instructions used to generate them. We provide complete details about the transformations in Appendix B
We also simulate simple direct synthetic data generation approach somewhat similar to Gunasekar iet al. (2023). Specifically, we generate solutions for the training problems using the OPT-3.5-TURBO model. We use in-context learning with the two-shot prompt examples selected from our D modular dataset. To ensure diverse solutions, we use three distinct few-shot examples and generate eight so- lutions for every prompt temperature of 0.5. Additionally, based on the ground truth test cases provided in the dataset to ensure we are not training on incorrect programs. Since it resembles distillation-like setup, we refer this dataset as D Disstill.
filter the solutions for correctness
3.3 EXPERIMENT DETAILS
To evaluate the quality of the transformed datasets, we measure how they impact the test benchmark accuracy. We study both in-context learning and fine-tuning using examples from our datasets.
Models. We use the CODELLAMA-78 model (Rozière et al. 2023 in all our experiments (referred as CL-7B ahead). We use the model checkpoint from huggingfact and perform batched inference
through vLLM (Kwon al., 2023), necessary for computing the PASS @ K metric. We also present the numbers from CODE DAVINCI-002 and GPT-3.5-TURBO whenever available.
In-context learning. We select two question-answwe pairs from the Doriginal and D Poodulor training sets as our in-context learning example. For a fair comparison between the two evaluations, we use the same problem and corresponding solutions from the two datasets examples. The examples are combined with appropriate delimiters and the model is then prompted with a new problem. Note that these in-context learning examples increase the sequence length by over 2,000 tokens and considerably slow the inference.
Fine- Tunnng We perform full fine-tuning over the base CL.-BB model the different datasets. We train the models for two epochs on the APPS dataset and one epoch on the TODE-CONTEST dataset using a 5e learning rate and an effective batch size of 256 on A6000 GPUs.
EXPERIMENTAL RESULTS
We present our experimental results in this section. Section 4. first provides a qualitative overview of the transformed programs and Section 4.2 presents the main code generation results.
4.1 ANALYSIS OF THE TRANSFORMED PROGRAMS
Data statistics. For the CODE-CONTESTS dataset, out of 98,582 programs extracted from the original dataset (Doriginat), we can successfully transform 92,675 (94.0%) into our modularized dataset (D (Dmodwar)). We obtain similar success rates for the APPS dataset (details deferred to the appendix). On the contrary, the distilled dataset (Daistsit) which is constructed by generating solutions directly using GPT-3.5-TURBO only finds a correct solution for about 50% of the problems.
Analysis of the transformed programs. We find that our transformation approach decomposes the original programs by inserting three new functions on a median (~2.6 functions on average). get
https: /tgggiugggig ng a aace.ac... co/ code lama/ /ode//ooeliii 1. amma 1a-ma---b-h- 7b-hf
Model generations were obtained from Chen et al. 2022a)
5

---

Published as a conference paper at ICLR 2024
APPS INTRODUCTORY
HUPS-INTEVVIEW
PASS@1 1
PA
PA
PASS@1
PASS 10
P5
In-context Learning
CL-7B + Doriginal
14.2
29.2
38.4
1.8
7.3
10.4
CL-7B Doddulrr
17.5
30.1
39.7
2.2
8.6
12.3
+3.3
+0.9
+1.3
+0.4
+1.3
+1.9
Fine-tuning
CL-7B + Doriginal
18.7
34.4
40.2
3.4
9.7
13.6
CL.7B + modduar
22.7
36.9
42.6
11.0
15.0
+4.0
+2.5
+2,4
+0.8
+1.3
+1.4
CL-7B Palenming
22.1
37.1
43.8
3.7
10.5
14.8
CI-7BB Dreamme
19.2
36.6
42.9
4.0
10.7
14.6
CI.77B Daistill
21.1
35.3
40.5
4.1
10.8
14.5
Closed models
DAVINCI-002
22.1
50.2
58.7
16.8
23.8
Table 3: Results on APPS dataset. We use the CODELLAMA-7B model (referred to CL-7B) under in-context learning and fine-tuning. We use samples from the original and our transformed datasets and find that our cleaned datasets improve the performance of the model by over 20%. The green highlighted numbers depict the improvements obtained from using Doodular (over Dorigina!). Simi- larly, using D Drename and D pianning also provide improvements, usually lesser than using D modular-
a better understanding of the decomposition, we cluster the functions using their function names and signatures. We find that these helper functions often implement key program logic, standard algorithms, and utilities like handling inputs, outputs, and orchestrating the main function. Interest- ingly, also find that the helper functions in implementations. For example, the top five most frequent helper functions, d's, bui 1d graph, god, dp, and b binary_sed searyysserhhhh occur in about 3-8% of the problems. Additionally, we qualitatively analyze a hundred random samples from Deriginal and D Doodulrr datasets to determine the quality of performed transformations. Figures mations. We find that most of the transformations are meaningful. They improve the readability of the programs and also find suitable decomposition for the program logic encoded in the control flow as examples). However, in some cases, the generated helper functions can have improper names (caiculato_mal cacttt__mx____ mat__aaa_aoooooaeeeeeeaaaeaeeeee in Figure II or complex implementations copied directly from the original program
often reused across problems, with small variations
II in the appendix provide examples of such transfor-
(see Figures
in Figure 12). Additionally, for simpler programs (Figure 13 the entire program functionality can be implemented the decomposition does not provide any extra information. Finally, et al., 2023) evaluation to quantitatively assess the transformations regards to their meaningul-- ness and about the consistency of original and transformed programs. Appendix C.I presents the comprehensive setup. We find that over 99% of the transformations are regarded as helpful which only 3-5% of examples are judged as can do better. Similarly, 99.4% of the transformed programs are judged as consistent with the original programs. More detailed evaluation results in Table 6
a single function and GPT-4 judge (Zheng
Unlike, generated code, we cannot constrain check the generated natural language plans. Thus, find that sometimes the plans can be imprecise and vary in detail. While using stronger pretrained model like GPT-4 could alleviate some of these issues, believe this will be good avenue for applying something analogous to process supervision (Lightman et al. 2023
4.2 MAIN RESULTS
Tables and provide primary results APPS and CODE-CONTESTS datasets respectively. We defer the results for the OPPS-COMPETIIIO subset to Appendix and and highlight our findings below.
4.2. EFFECT OF MODULARIZATION
We find that our data-cleaning approach improves the performance of the model on both APPS and CODE CONTESTS datasets in both in-context learning and fine-tuning settings.
CODE-CONTESTS
Result soureed from Li
2022
'Result sourced from Zhang ct al. (202bb
Result sourced from
al. 2023c

---

Published as a conference paper at ICLR 2024
CODE CRDE-CONTESTS
PAss 10
P5ss@25
PAss@ 100
In-context Learning
CI-77 D Doriginal
5.1
6.5
7.2
CODE-CUNTESTS-PLAN
CL-7B D Doodulrr
4.9
6.6
9.3
-0.2
+0.1
+2.1
PASS
25
PASS@ 100
Fine-tuning
CL-7B + D Dariginal
6.5
9.5
15.0
CL-7B + + Pariginull
5
6.4
10.9
CL-18 Pmodstur
8.8
11.8
17.8
CI-7B modular
6.1
8.3
12.4
CL-7B Palanainn
6.9
10.5
15.4
+1.1
+1.9
+1.5
CL-7B D
17.9
22.3
28.1
CL-7B + Dalenning
5.3
7.0
10.8
+9.1
+10.5
+11.3
CL-7B Drename
4.7
6.3
10.5
Closed models
Table 5: Effect of using ground-truth plans,
ALPHACODE-99 9D
5.0
7.0
10.0
We disentangle the high-level reasoning coding capabilities by extracting ground-truth plans from solutions corresponding the test problems. We find significant improvement in the performance on the MODE-CONTESTE -PAAN dataset, indicating that the model trained on the Pplanning dataset while incapable of building correct plans, can follow such plans accurately.
ALPHACODE-41B
5.0
7.0
10.0
Cone-DAvINci-002
3.0
7.5
GPT-3,5-rURM(
18.2
+ BRAINSTORM
29.3
Table 4: Result on the CODE-CONTESTS dataset. Similar to findings on the APPS dataset, we find that our data cleaning approach generally improves the performance with modularization working particularly well while planning and re- naming providing marginal to improvements.
In-context Learning. We first evaluate the performance of the model when provided with parallel two-shot in-context learning examples from D Poriginal and Dmodular datasets each. We find that the PASS@1 1 improves from 14.2 to 17.5 (a 23% relative improvement) on the APPS-INTRODUCTORY dataset and PASS @ 100 improves from 7.2 to 9.3 (a 29% relative improvement) on the CODE- CONTESTS dataset. These results indicate that more readablity and better-struuttured coding is helpful to the model in solving more problems.
Fine-tuning. Next, fine-tune the model on the D Doriginat and D Moddulr datasets and again find strong performance improvements from our transformation approach. Specifically, on the APPS- INTRODUCTORY dataset, the @Ass@1 improves from 18.7 22.7 (a 23% relative improvement). Similarly, the CODE-CONTEST dataset P metric improves from 6.4 to 8.4 (30% relative improvement). These results cement our above findings about the effect of cleaning the data.
Interestingly, we also note that fine-tuning only provides modest improvements over the in-context leamning performance. We hypothesize that this is due to the challenging nature of our task.
4.2.2 EFFECT OF PLANNING ANNOTATIONS
Prior work has demonstrated considerable successes in improving reasoning in LLMS (uue et al. 2023 Magister et al.) 2022 Fu et al., 2023) by performing supervised learning natural language reasoning or planning steps. We perform similar experiment, fine-tuning the model on D Pptanning dataset consisting of plans generated by our approach top of D Doodular- We find that planning only provides a modest improvement over the Pmodutar dataset (PAsS@25 improved from 42.6 to 43.9 the APPS-INTRODUCTONN dataset) or often improvements at all.
Upon inspection of the generated solutions, we find that often the generated plans are imprecise or incorrect, highlighting that planning still remains bottleneck. To disentangle the high-level planning from the coding component, we analyze the performance of the model when provided with ground-truth plans on the our data transformation approach on the test set (similar to how D Plaaning training set was created). Table provides results on this subset of 109 problems from the we were able to extract the ground truth plans (since some problems don't have valid python solutions). While our model trained on the Dplanning dataset is incapable of synthesizing new plans, it can follow the generated plans correctly. All metrics improve significantly, e.g. PAsS @ 100 improving from 17.8 to 1, I, well over the performance of GPT-3.5 TURBBO, a much larger model!
dataset. We extract these ground-truth plans by applying
dataset for which
'Nte that the in-context examples add over 2,000 tokens the prefix and lead to much slower decoding
7

---

Published as a conference paper at ICLR 2024
def r Impus O (..p2ii1
22
del Menoom_ahiiie Lo (grid); sov_indicas
swLuen grid
PASS@1
whith col.umo (grid):
selumngind
16
rwturn grid
main(1:
14
Dont
grid geadggr
Devio
grid ranove_abil te Igrid; grid
tw_.oellnoeeeeeeeee innitgridd print_ Igriid)
5000
10000
15000
20000
Number of training samples
Effect of quality on data-efficiency of the model. Finetuning on 12.5% of clean Dmodular dataset results in similar performance as finetuning on the entire Doriginal dataset.
Figure 3:
Figure 2:
Example of program gener- ated by our model trained on the 'D modullr dataset. It solves the problem by using helper functions acting rows and columns.
Our mixed results raise critical questions for future work on improving planning in LLMs. In par- ticular, poor performance might be attributed to any imprecision in automatically generated plans. Future data euration techniques that filter augment this imprecision would be valueable. Alterna- tively, the supervised learning paradigm followed in this work might be insufficient for models to generalize planning in complex domains. Future work can explore alternative learning algorithms, possibly over our modularization approach which naturally decomposes programs.
4.2.3 ABLATIONS
Effect of data size. Beyond improving the quality of the resulting model, data quality is also attributed to improving the data efficiency. We evaluate this aspect by fine-tuning different fractions of D Doriginal and D Poodular datasets and find similar results. Figure 3 presents the performance of the model as a function of training set size. As shown in the figure, training on just 15% D modulor dataset achieves similar PASS@ as fine-tuning on the entire D Doriginal-
model on
Effect of renaming. We variable renaming evaluate the performance of the model fine-tuned only on the Drename dataset and find that renaming provides some performance improvements when compared to fine-tuning example, PASS @ improved from 17.2 to 19.1 on APPS-INTRODUCTORY However, renaming still performs worse in comparison fine-tuning on the Dmoduler. This highlights that beyond just readable code, functional decomposition is also a key aspect of improving our performance.
intermediate step in our cleaning process. We
Doriginal dataset. For
Cleaning Transformations vs Distillation. We compare our transformation approach with direct distillation baseline where we directly generate solutions using GPT-3.5
referred to as the Daistatl " This corresponds to various LLM instruction or fine-tuning approaches (Xu et al.) 2023 Li et al. 2023b) providing strong baseline for data cleaning. On the APPS-INTRODUCTOON dataset, we find that fine-tuning on the D moodlar dataset achieves better performance compared to the Daistill dataset demonstrating the advantage of cleaning over the generation bascline.
Choice transformation model. To evaluate how the choice of transformation model affects performance, we use the GPT-4-TURBO model to transform setup in Appendix C.3). GPT-4-TURBO, a stronger model, performs the transformations successfully and the resulting model trained on this version of the modularized dataset achieves even higher accuracy. For instance, @ 10 improves from 33.0 when using Dmodutar constructed with GPT- 3.5-TUrro to 34.3 when using the Poodular constructed with GPT-4-TURRO (full results in Table 8
subset of the training set (detailed
4.2.4 COMPARISON TO OTHER BASELINES
Beyond CL-7B, fine-tuned models outperform strong baselines like ALPHACODE on the CODE- CONTESTS dataset but still lag behind larger CODE-DAVINCI-ON and GPT-3.5-TURBO models.
NNote that generate these solutions using in-context examples from the D, modular dataset
8

---

Published as a conference paper at ICLR 2024
4.2.5 CASE STUDY OF GENERATED MODULARIZED PROGRAM
provides an example of a program correctly generated by a model fine-tuned on our D modular dataset. The problem requires removing rows and columns containing cells with certain attributes (i.e., if the cell is white) The modularized solution correctly identifies the steps required solve the problem and implements them separate helper functions, providing readable code.
Figure 2
5 RELATED WORK
Instruction tuning. Instruction tuning refers to the process of finetuning a base pretrained LLM to perform general-purposs tasks and follow instructions. Recent works, [hou al. (2023); Cao (ct al. (2023); Chen et al. (2023a) have demonstrated that a small high-quality instruction corpus is sufficient for achieving good instruction tuning performance. Here, perform task-specific fine- tuning of LLMS and observe similar performance improvements.
Synthetic data for LLMS. Recent works have explored using synthetic datasets for general- purpose or task-specific finetuning of LLMS. These approaches work by generating synthetic datasets from a strong LLM (like GPT-3.5-TURBO or GPT-4) using set of existing tasks (Taori
et al. 2023 Chiang et al. 2023) or generating new tasks using self-instruct (Wang et al.] (002) or
evol-instruct (Xu al.) 2023) approaches. This has been also applied for task-specific finetuning
in common-sense reasoning (West et al., (202)), text-summarization (Sclar et al. 2022). mathemat-
ical reasoning (Luo al., 2023a] Yue et al.] (023), tool use (Patil al. 2023). coding (Luo al.
2023b), and general- purpose reasoning Li et al. (2023b): Zelikman et al. (2022).
More specifically, (uue et (2023) curates diverse corpus of mathematies problems with chain-of- thought or program-of-thought (Chen et al. 2022b) annotations for mathematical reasoning analo- gous to our plans. Gunnaseaar et al. (2023) proposed pre-training models on programming "text- books" generated synthetically from GPT-3.5-TURBO [Haluptzok et al.] (2023) similarly generates programming puzzles and corresponding solutions from language models. Our work also studies cu- rating synthetic data for
space. However, instead of directly generating data using LLMS, we identify good programming patterns and clean existing datasets using them.
Algorithmic Code Generation. Code generation is a broad domain and is covered in Appendix D We only discuss pertinent algorithmic code generation works here. [enddrycks et al. (2021) released the APPS dataset while Li et al. (2022) released the CODE-CONTESTS dataset with the ALPHACODe models. Zhang et al. (2023c) proposed a mookaeea-seeachh based decoding algorithm for improving reasoning in LLMS and is orthogonal proposed CODET and ALGO, that use generated tests to re-rank the generated solutions. Zelikman al. (2023) proposed the paRSel approach where they first generate a plan in a vroblem-specification
our work. Chen al.] (2022a); Zhang al. (2023b)
language and then generate a program using it. Li et al.] (2023a) also study disentangling the
planning and coding for closed source LLMS, similar to our experiments on open models.
DISCUSSION AND CONCLUSION
Traditionally, data quality has been linked to functional correctness, ignoring the rich stylistic as- pects differing across programs. In this work, we demonstrate that these aspects like readability, and program structuring actually impact the performance of the trained model on downstream tasks and thus also contribute to data quality, perhaps in relation to amenability to autoregressive modeling. Next, we proposed a novel data-cleaning pipeline demonstrating that LLMs can be used for trans- forming existing datasets to improve their quality based on user-instructions and oracle equivalence checker. While our evaluations focused on the algorithmic code generation task, we believe that this approach would also be useful for other domains for improving data quality as well. In particular, even in the absence of symbolic checkers (like test cases), we believe that there is an opportunity to use learned "oracles" for ensuring consistency and quality in other domains akin to how used in Sclar et al.] (2022). Finally, beyond improving algorithmic code generation, we believe our mod- ularization approach can be beneficial for general software engineering use cases (test generation, debugging, verification) where modularity is beneficial. A key limitation is that this work relies upon proprictary models for data transformation. With access to stronger open models, we believe approach can also be applied in self-training setup using oracle equivalence checkers.

---

Published as a conference paper at ICLR 2024
Acknowledgement This work was supported in part by NSF grants CCF-1900968 and by SKY Lab industrial sponsors and affiliates Astronomer, Google, IBM, Intel. Lacework, Mi- crosoft, Mohamed Bin Zayed University of Artificial Intelligence, Nexla, Samsung SDS, Uber, and VMware and finally generously provided research credits from Vast.ai. Any opinions, findings, con- clusions, recommendations in this paper are solely those of the authors and do not necessarily reflect the position of the sponsors. Additionally, we thank Alex Gu, Manish Shetty, and anonymous reviewers for helpful discussion and feedback on the paper.
REFERENCES
Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D C, Arun Iyer, Suresh Parthasarathy, Sriram Rajamani, B. Ashok, and Shashank Shet. Codeplan: Repository-level coding using Ilms and planning. In Neural Information Processing Systems Workshop on Foundation Models for 2023.
Decision Making
Yihan Cao, Yanbin Kang, and Lichao Sun. Instruction mining: High-quality instruction data selec- tion for large language models. arXiv preprint arXiv:2307.06299 2023.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guame Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprint arXiv:2207. 10397, 2022a.
Hao Chen, Yiming Zhang, Qi Zhang, Hantao Yang, Xiaomeng Hu, Xuetao Ma, Yifan Yanggong, and Junbo Zhao. Maybe only 0.5% data needed: preliminary exploration of low training data instruction tuning. arXiv preprint arXiv:2305.09246 2023a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374. 2021.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12558, 2022b.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05//8, 2023b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
7/ //Imsys..rr//// /Lmmyys.orr/blog/22223333-----
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. arXiv preprint arXiv:2301.12726 2023.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorn,, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, al. Textbooks are all you need. arXiv preprint arXiv:2306. 1664,, 2023.
Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. Language models can teach them selves program better. The Eleventh International Conference on Learning Representattonss 2023.
Dan Hendryeks, Steven Basart, Sauray Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurlPS, 2021.
Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang. Li Li, Xiapu Luo, David Lo. John Grundy, and Haoyu Wang. Large language models for software enginecring: A systematic litera- ture review. arXiv preprint arXiv:2308. 10620, 2023.
Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy. Sriram Rajamani, and Rahul Sharma. Jigsaw: Large language models meet program synthesis. In ICSE 2022.
10

---

Published as a conference paper at ICLR 2024
Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. Self Self-planning code generation with large language model. arXiv preprint 0rXiv:2333.06689 2023.
Darren Key, Wen-Ding Li, and Kevin Ellis. speak, you verify: Toward trustworthy neural program synthesis. arXiv preprint arXiv:2210.0088 10.0088,, 2022.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-based based pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/2211. 112221111150 1,011,
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Hoi. CodeRL: Master- ing code generation through pretrained models and deep reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Hung Le, Hailin Chen, Amrita Saha, Akash Gokul, Doyen Sahoo, and Shafiq Joty. Codechain: To- wards modular code generation through chain self-revisions with representative sub-modules. arXiv preprint arXiv:2310.08992, 2023.
Jierui Li, Szymon Tworkowski, Yingying Wu, and Raymond Mooney, Explaining competitive-level programming solutions using lims. arXiv preprint arXiv:2307.05337, 2023a.
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. Symbolic chain-of tf-in-of- distillation: Small models can also "think" step-by-step. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2665-2679, Toronto, Canada, July 2023b. Association for Computational Linguistics.
Xin-Ye Ye Li, Jiang-Tian Xue, Zheng Xie, and Ming Li. Think outside the code: Brainstorming boosts large language models code generation. arXiv preprint arXiv:2305.10677 2023c.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05460 05463, 2023d.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092-1999 2022.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.200500 2023.
Chao Liu, Xuanlin Bao, Hongyu Zhang, Neng Zhang. Haibo Hu, Xiaohong Zhang, and Meng Yan. Improving chatgpt prompt for code generation. arXiv preprint 2xXi:22055008666 2023a.
Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. Ritf: Rein- forcement learning from unit test feedback, 2023b.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.0m 095883, 211583,2000 2023a.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint ar arXiv:2306.0885 08568, 2023b.

---

Published as a conference paper at ICLR 2024
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. arXiv preprint arXiv:2212.08410, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744 2022.
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305. 15334, 2023.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Mclanie Sclar, Peter West, Sachin Kumar, Yulia Tsvetkov, and Yejin Choi. Referee: Reference- free sentence summarization with sharper controllability through symbolic knowledge distillation. 2022.
arXiv preprint arXiv:22
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Re- flexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K. Reddy. Execution- code gen- eration using deep reinforcement learning. Transactions on Machine Learning Research, 2023.
ISSN 2835-8856.
Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. Repository-level prompt generation for large language models of code. International Conference on Machine Learning, pp. 31693- 31715, PMLR, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305. 04091, 2023.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-tthougee prompting elicits reasoning in large language models. Advances in 2022.
Neural Information Processing Systems,
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang. Liwei Jiang. Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. In Proceedings the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 4602-462. Seattle, United States, July 2022. Association for Computational Linguistics.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua How- land, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sut- ton. Natural language to code generation in interactive data science notebooks. Jordan GoyddGraben and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 126-172 173, Toronto, Canada, July 2023. Association for Computational Linguisties.
Anna Rogers,
12

---

Published as a conference paper at ICLR 2024
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STar:
Bootstrapping reasoning with rea- soning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah Goodman, and Nick Haber. Parsel: A (de-
) compositional framework for algorithmic reasoning with language models. arXiv preprint 2Xiv:2212.105611 2023.
Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303. /2570, 2023a.
Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. Algo: Synthesizing algorithmic programs with generated oracle verifiers. arXiv preprint arXiv:2305. 14591, 2023b.
Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua Tenenbaum, and Chuang Gan. Planning with large language models for code generation. arXiv preprint arXiv:2303.05510, 2023c.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Daniel Fried Zhiruo Wang, Shuyan Zhou and Graham Neubig. Execution-bass based evaluation for open-domain code generation. 2022.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206 2023.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 10625, 2022.
Terry Yue Zhuo Large language models are state-of-the-ar evaluators of code generation, arXiv preprint arXiv:2304. 14311233444.17,,, 2023.
13

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL SETUP
APPS benchmark. Since some of the problems the APPS dataset are sourced from websites that provide insufficient or absent test cases, we filter the problems from those platforms. Specifically, we only retain problems from the code forces, codechef, and at coder competition websites. This also removes the disparity/domain shift between the training and test splits which has been observed as an issue in the APPS dataset in prior works (Section 4.1 in [i et al. (2023ch). While we considerably reduced the size of our training set, our test still quite close to the set containing around 3800 problems instead of the default 5000.
CODE-CONTESTS benchmark. The original CODE-CONTESTE benchmark consists of 13,328 prob- lems in the training dataset. We restrict the dataset to only problems with valid python solutions that pass the test cases. Next, since the original dataset provides over 100 solutions per problem, we perform minhash-based deduplication from gaoya and retain a maximum of 25 solutions per problem. This results in about 7k problems in the training set spanning about 98.5k solutions. We do not perform any filtering on the test set.
the solutions (hash size=64, num bands=60, band size=5)
Additionally, we note that some of the provided solutions in both APPS and CODE- CONTESTS datasets do not pass the test cases. These cases are sometimes caused by incorrect programs, of- ten because of scraping solutions with the wrong language. However, more often problems in these datasets support multiple correct solutions (for instance solutions to a problem can return a list of elements in any order). The ground truth test cases only check for single correct solution and thus result in many solutions failing the test cases. We retain such samples for the smaller APPS dataset and instead check whether the transformed program behavior is similar to the original program.
Metrics We use the PASS@ to perform our evaluations. We perform nucleus sampling using VLLM with p 0.95. We outline the default sampling configurations used for computing the metrics
PASS We use a sampling budget (N) 10 and temperature 0.1.
2. PASS 10 We use sampling budget (N) 50 and temperature 0.6.
3. PASS @ 25- We use sampling budget (N) 50 and temperature 0.6.
4. Pass@ 100 We use sampling budget (N) 200 and temperature 0.8.
Finetuning details We finetune the CODELLAMA-78 model using deepspeed huggingface trainer. We use the following training configuration for our main experiments
Training Parameters
Values
LR
5e
Epochs
or 2 depending on the dataset 256 (combing grad. accumulation)
Batch Size
Dtype
6
14

---

Published as a conference paper at ICLR 2024
B CODE TRANSFORMATIONS IMPLEMENTATION
We implement our code transformation approach using zero-shot prompting with GPT-3.S-TURRO model. After transformation, we extract the generated code and evaluate its functional correctness using the provided test cases, In case the program does not pass, we retry the process with up to maximum of attempts. In our experience, instruction-tume models can follow precise commands and transform programs very well.
B. RENAMING
We use the following prompt to perform renaming.
QUESTION:
[prohlem_statemen
ANSWER:
"ppthhn
(solution)
Rename the variables the program to be descriptive meaningful, and consistent. Do not change the original semanties of the program. Enclose the program within backticks shown above and remember to use descriptive variable names.
B.2 MODULARIZATION
Unlike renaming, perform two rounds of modularization in case the generated program consists of long function implementations (hinting that the function can be decomposed further). We use the following prompt to perform the first round modularization
QUESTION:
pproblem_ prrollem_statement
ANSWER:
"'python
greaaed_sollution
Refactor the
program. Follow guidelines
make the program more modular with smaller and meaningful helper functions
good deseriptive names for the helper functions
have entry function called 'main() 'main() is called inside "if name__.
main__..
Do not change the original semantics of the program significantly and no need to perform optimizations. Enclose the program within backticks shown
Next, in case the modularized program contains function with the number of lines greater than 20, we further prompt the model while signaling which functions to further decompose. This occurs in about 20-40% of modularized solutions and we use the following prompt.
QUESTION:
pproblem_ statemm)s
ANSWER:
... "python
pondullarize_ssotuttino
Refactor the above program by modularizing it and brenking down long and complex functions into smaller meaningful helper functions. Particularly re refactor and decompose the following funetion(s) into smaller helper functions [function_ string] Only return the refactored program enclosed in backticks as shown above.
B.3 PLANNING
We the following prompt generate natural language plans
15

---

Published as a conference paper at ICLR 2024
QUESTION:
prohlemmsstatement
ANSWER:
'.' "'python
poooulaaizeddsolttionn
...
Generate summary for the following Functions and classes in the program within four lines descriptive helpful for understanding the program (however concise in four lines).
each. The summaries
The functions and classes are [list_of fust_offfuuntiinn wlnffnnntonnnt names]
Follow the provided format for the summaries while being informative and concise. Enclose the signatures
as
16

---

Published as a conference paper at ICLR 2024
C ADDITIONAL RESULTS
C.I GPT-4 JUDGE EVALUATION FOR TRANSFORMATIONS
quantitative evidence of the improvements made from our transformation approach. However, since the transformations are free-form code generation, we rely using GPT-4 as judge, an evaluation approach gaining popularity for evaluating free-form language out- puts (Zheng et al. 2023; Zhuo 2023). Specifically, we ask the language model to answer whether the modularized refactored code has better variable names, better function decomposition, and con- sistent with the original program. The model can provide answers on key of 1-3 from comparison questions and for the consistency question. The following prompt depicts our approach
We here present
SYSTEM PROMPT:
Please act as an impartial judge and evaluate the code refactoring below. You need to evaluate whether the refactored program uses better and correct variable names. refactors the implementation into correct smaller helper functions and consistency with the original program. Your evaluation should based on correctnes and helpfulness of the refacioring in better understanding the prablem and also if it is still consistent with the original program i.c. it follows similar program logic and algorithm.
For evaluating variable names and function decomposition, please give score from to where means the refactoring is not helpful at all 2 means the refactoring is somewhat helpful and means the refactoring is very helpful. Example format
often assessing whether the variable names are more descriptive and meaningful and correctly refleet the variable 's purpose
Variable reasoning: [[rassniig for the variable
Variable names: [[]]] or [[2]] or [[3]]
Function decomposition reasaning: [[reasoning for the decomposition whether some function is toa long, passibility to perform further abstractions, choice of abstractions helper function names]]
often assessing
Function decompesition: I[I]] or [[21] or 11311
Far evaluating cansistency. please give score of 0 if the refactored program is not consistent with the original program and 1
consistent. Example format
Consistency reasoning: [[reasoning for the consistency score often assessing whether the refactored program follows similar program logic and algorithm the original program ]] Consistency: 11011 or II1II
QUESTION:
pproblem_statement
ORIGINAL SOLUTION:
(solution)
SOLUTION:
(solution)
While this evaluation might portray certain subtle biases, we believe it still provides signal to assess the quality of the transformations. To reduce costs, we the GPT-4 as judge evaluation to 1000 problems in the APPS dataset GPT-4 followed the proposed format for 998 solutions and present results on them in 6 Results demonstrate that most of the applied transformations are meaningful while remaining consistent with the original ground truth solutions.
Score distribution
Average
Variable names
[3 967, : 28, 1: 3)
2.96
Function decomposition Consistency
{3 : 938, 2: 59, 1:
2.93
{1 994, 0 : 4}
0.994
Table 6: GPT-4 judge evaluation for quality of 998 transformed examples. We compared the original and unmodified solution using GPT-4 for variable names used, function decomposition, and consistency of the modified and original solution. Results demonstrate that most of the trans- formations are successful and meaningful while being consistent with the original program.
Our prompt spans around 1.5-2k tokens including problem, original, and refactored programs leading to high costs
17

---

Published as a conference paper at ICLR 2024
To get better insights into GPT-4 evaluations, we look the examples which receive lower scores. The scores and associated reasoning appear meaningful. For example, in Figure [1] the modularized program is already significantly more readable than original renamed program. However, GPT-4 identifies that the ca 1 cu ate peteeeerrutaaiiii immuaaioon certtoo__eveeeeee _even and ca cul aoupateeperree @ate_permutatee ieerruaaiionneeet functions virtually the same and can be abstracted further. Note that this transformation is artifiact of the fact that original program consisted of same program logic distributed across two far apart if conditions. Similarly, in Figure 15| GPT-4 identified some unmodified variable names like while acknowledging other improvements such as sky to heights giving it rating of 2. The rating of provided when the transformation does modify any variable names or does not decompose existing functions, as evidenced by score distributed, a rare occurrence. Indeed, often examples marked with a rating actually improve upon orignal code non-trivial ways
helper
C.2 COMPETMPETNN RESULTS
We present the results on APPS-COMPETIIN dataset here.
APPS-COMPETITION
PASS
PASS@ 10
PASS @ 100
Fine-tuning
CL.-BB + Doriginal
0.2
1.7
3.1
CL-7B + Doodular
0.5
2.3
3.2
+0.3
+0.6
+0.1
CODE DAVINCI-002
0.3
2.9
5.7
Table 7: Results on the APPS-COMPETITION dataset.
C.3 ABLATION ON CHOICE OF MODEL
We use GPT-3.5-TURBO as default model for performing the transformations in the main exper- iments since it provides nice balance between the accuracy and cost of performing the transfor- mations. Here, to demonstrate the generality of our approach we perform an ablation by replacing the transformation model with GPT-4-TURBO. Since this model is about 8-10x expensive than GPT-3.5-TURRO, we perform this ablation on a subset of 5k programs sampled from the dataset.
Experimental setup. We repeat the renaming and modularization steps described in Section 3.2 using the GPT-4-TURBO model. We call the resulting transformed dataset as Dimodutarr Next, fairly compare the resulting dataset with the original and modularized dataset generated using GPT- 3.5-TURBO, we sample the corresponding parallel original and transformed programs and call them modular datasets.
D Doriginal and D
APPS-INTRODUCTORY
PASS@
PASS@ 10
PASS @ 100
CL-7B D Doriginal
16.3
31.6
37.6
CI.-7B D3.5
18.8
33.0
38.2
moduler
CL-7B D4 moduler
19.4
34.3
40.0
+0.6
+1.3
+1.8
Table 8: Ablation on the choice of model used for performing the transformations. modular represents the dataset generated using GPT-3.5-TURBO and D4 modular represents the dataset gener- ated using GPT-4- We find that the performance of the model trained on the D modular dataset better than the model trained on the 33.5 p modular dataset.
Dis.5
10 "Crriously, GPT-4 sometimes returned rating of 2.5 instead of integer 2 or 3. We rounded it to 2, thus making our evaluation harsher!
18

---

Published as a conference paper at ICLR 2024
D ADDITIONAL RELATED WORK
Code LLMs have been used for multiple domains in various lines of approaches. Here, we present
few key approaches and recommend the reader to Hou al. (2023) for detailed survey. Chen et al. (202T) released the CODE-DAd D---DAIIIIO0002 model and evaluate for code generation. Since then,
have been used for variety of domains such as data science (Jain et al. Lai et 2022] Yin
et al. 2023), APIs (Zhiruo Wang & Neubig, [022] Patil et al. (023). and repositories (Zhang et al.
2023a Bain al. Shrivvaaava et al. 2023). (e et al. 2022, Shojace et al. 2023] Liu al.
2023b use reinforcement learning with compilation/execution feedback to fine-tune code LLMMS for (algorithmic) code generation task.
Other works have approached code generation from different fronts, exploring planning (Jiang et al.
2023), repair (Chen et al., 2023b Shinn et al. 2023), test generation (Key et al. 2022. Chen al,
2022a), and prompt optimization (Liu et al. 2023a). Le et al. (2023) proposed prompting-baset
approach for modular code-generane generrtteen.
19

---

Published as a conference paper at ICLR 2024
E EXAMPLES OF TRANSFORMED PROGRAM
del inaplin
... ITRUNGATER;;
reburn books, run_sell, inok_info
group by gente
infol:
ca0g genre I[]
for price, gente in book
books/gerre- (eg.rr--ll..geeeeee -appel spriice benke_
def sort books, by price (genre_booksi: booksl:
genre. (reverse-Tzue)
return genre_ books
def maan
cet selculate
prices booksi:
impore aye
genre
inputreyo stwin. reodiiree
genre. caneeti (9, 0)
for genre in gerre books:
kerpp Cint, input (1
for in rarge len (ggnnne)) genreli] genre [1-2] 2x
ab- tinp toput 10 Ve For [0]xn]
return genre books
cet selculate gerre dp [0] (cun_sell-1)
ge in [0]+10]
(appenda (al cor z,b in ab]
[g] cj.rot (reverse-True) for a in (10)
tor genze in genre_Rocte:
dp2 [3] (nom sella1)
in (10)
sange [ier (genzol):
:or ir range trum sell-1-1);
dp2[1-1] max s111
in range lenteicl11:
gorre(i))
dp dp2
atc! [101111+1110]]1112-4-443
return (dp)
(K-1
cet mein
nun bcoks, aur_soll, book_info road_input
tor a in zange (10)
penro_bocke group_books btcpoo_booo_eooeebeeee_e_ebbeeee benge tnon_books,
dp2= (kk)
book_infs)
(lan
sors_bocks_by prics pooks_
ter zarge 1K+1-1)
genre_ bocke crieriato_porther
[1 enax 401
qerke_bookk)
max mox_purchssc_prles
dpodp2
calculate_nnxe
gentr_bbokk)
print (nax 1d
print (max_parcacso_prte))
mair.(1
moin()
(a) Original program
(b) Transformed program
Figure 4: Original and transformed programs
20

---

Published as a conference paper at ICLR 2024
def Fastorrs(g)
faclors
while isi <- 93
11 3
appoora.ppemd()
g//5 1- is
(reverse=froe)
facloca.pop()
For
range Cint tingut 133
return factors
Uin,, inpat 0 apl de
def jal__atsestt_diiii
factora):
P
largest divisor for in factors:
1.
dividend_
dividend copy
8:
divident.ooy //-
it q 8
0:
Largest_aivison max divieeeet_diviooor
Fector.agpend(i g//i 1- apprar/))))
dividend largest__diviau:
def ma 13
int Linpat (1)
ractor.eort (reverse-True) Fantor-pop()
rangert):
Pr q naptima irput (.split
factors gat__actoorigg (g)
tactor:
largest_divien facsora) peint [Lorgert_civiooc
Largest_diviison
while a
d //- 1
mein.
print
main
(a) Original program
(b) Transformed program
Figure 5: Original and transformed programs
21

---

Published as a conference paper at ICLR 2024
Gef pi.
calculate audalo initialize naneratoz
denoninatos
numerator decini recor
fuz i in range(s):
1)1 denoninator [deroninator
zuluen (nureraoo pow lemnoninnneo, 2, p1)
det iouypod marliee (ranboss, d, b) check it minber ie good nunser while number 1-
if number
mumber 10
return Pallee
number f/= 10
zuluzn Teue
BI: aniiiallin nuteratio and ceneminator
def cuunt__xxell meceelletinnnneeeslll b, al:
and
10 +.7 denoninator
in tange(e):
nom trum ta 131 P den Iden (1 131 4.9 (our (den,
fur in range 11: cum_ot_digite
B11 P 6, renap linc, ingot splitt()) andr0
12 1 1- 0:
g-10s*947 947
Inunerator
don-1
donominstor Idenoninator 1) \
For range (n+11:
bew.bbbaiie In-13
If ix_gand umber (sun of digits, b) and inumerstor poy:
nome Inum (n-
deroninator, p1) p1 8
Jen (dons 11)149
an-True
it st101-s and #b10!-b: am-Faice
6Gt main
5, map (int, input (1 sp1it result count excel lent numbers (a, print (result)
break
s//-10
1t am:
and lans+ Lou (dea, p-2, $91 print (anstp)
12 name main
main
Original program
(b) Transformed program
Figure 6: Original and transformed programs
22

---

Published as a conference paper at ICLR 2024
Impore sect
senerato_palnddzon ()
pe I indrowe II
Duz range C,, 1C002):
Ia:-1l:
pel indromee oppenddduun)
eMbuen pal Indroma
closest indas biswel bbsee (swl indeunes, it obs in - pa indrores (index 113 <e abe in pal pnlindrows [ndde]]:
indrone (pwl
anlindonnss [indox 11
set return pa indrones [Endex)
impozt biseos
For
range (1, 30002):
pel indromes ppneeace_ppeiiine
it str (trmstitttr (11 [1-1] irev. (3)
int (input (1 ossest palindrone Find closestt indrone indrumes, print (closest (closest_peiisrrrees pelindrome)
nwint (1)
inde bbisset___eeee (ewxys)
abs tn-rev ind-1]] ind)
print trev ind-111
wlaws
ic
mwinn's
peint (ndd))
(a) Original program
(b) Transformed program
Figure 7: Original and transformed programs
23

---

Published as a conference paper at ICLR 2024
def Find divisors (max incervaas
diviscra
divisoz rance (1, max_interva111)) If nax_interval divizor =- 3:
divisora
del dn caloulate pureteeeiiiinn (iniaaalll divisues):
current. index
interval intervals:
div (oueeea index) interval:
current index 4-
mim i- divisons(eurreet__
interval
return min. sum
inn ann-iin Einpas (11
[int Cinpul (1) For: in cange (00]
def main 1)
Haput
intervels [int Cirgut (1) For sanget mam (em_nnerooneees)
ans 0
elae:
intervel s.gort
11
nun_selssr.ones
min_sum
For x in range(), yells
se:
if 9 R.append
nax_interval intervels] 1]
max interval)
our anx for in T: while Ricur] and B(cue]
sum on late min, sun nteevals,
divisors)
(min
name
print Carsl
main
(a) Original program
(b) Transformed program
Figure 8: Original and transformed programs
24

---

Published as a conference paper at ICLR 2024
From oo eeat fons import defaullddiet
rond Ivertex, dau) whi vertex I= deu [vertex]
verlex da. [ae_[vee_ex]
def Lnegg_. tu, dau)
rost_u Find
rock_a rout_
deal root_y) v] dsolv] root_u wi:
def mein (1:
collections inport defaulldiet
dau[zuoi_u]]
dau[u]
Ty crl Gigect (.), inpat (1 [1:2]
das,, möges, 11st (range (0), 1I, defaultdict
def bui burahgg trum, vest voet colors, eöges):
(lish]
deu list (range (nun_vert graph defaulediot fle)
range
0, v. naptiint iopas 01
u, v edgess:
u ----- 1
i2 colore(a] -- (M)
deo)
culorsiu] colors[v]:
o, deulel, dsc
Idep [ cappen [vsalv)) graph Iden [den/v]] (alauvlll.ppp... append
while deslal:
daa[a]
rebuzn dau,
deu(b)
def bff [x, nun vertices, graph);
1: < 2: dsulb] deulv]
next verticer [X]
available [True]
vertices
doute] dcu(e]
while next_verticos:
4- current_verts next_vert
eiges.aggenddCh
next_vertices,
for U. enumerate Idea)) Idea):
for y in curcert vertices:
dou - dsu[v]
availablely] Falls
while deuueedgesppog D., (1
graphly)
available jneighhoee
app))apppend(2 (p))
next sestt_eatticss.ppeedaa (raiphbbbrr))
return it eise corrent_verticce(00]
LES (X)
moin 1)
nxt, avail, t [X], [True] n.
nun, vertices int (input (1)
while nxt:
colors 0
oue, EXt,
range reunereettiiioon
for in caz:
G, v = map maр (inz, input (spp1it)))))))
avail Folse dlylt
-- 1
avail[y]: nxt appond lyl
osgcs.append(1e, VI1
dou, graph graph (oun_rerioor colors, edrea)
return if else cur[0]
print (Dfs nur_verticas, craph), num_rersices, graph)
posnt 10:1
15 nome 7oin
12 nanc mzin
mein
Original program
(b) Transformed program
Figure 9: Original and transformed programs
25

---

Published as a conference paper at ICLR 2024
Impore
caloclata_distaaccc (gzaph, start): Tentgragh)
distancea [ fur range to 111
visited [False
range
visited(start] True
stack []
distances latares//sttttl.
heeggg.heagppeh tstpph start)
whill stack:
current rode heepg heappop For: neighbor in graph cour node]]
viai Led sighhoe)
import heapg
visited Inei (ighoo)) True
anteeneed (eeeed/eightten [neighoor) distancesl
def dea (graphi stari):
cadeeni_node?
(graph)
teopg-haegguhhieee return distancws
reighbor]
dist D.
cargeta 133
visites = [Falce for range in + 19.
risiteelatart, True
def
Find possible book, locat ions
nep(int, inpal st.sir it
cististart
attected_soo settiements 11st (mag Hint, input D.
heepq- hegggghheeuushttteeeeeee start]
stript) ((t))appltt ()11)
while stacks
graph II] for sange 1)] range
heapg v. graph [e]:
6, nepting, ingot O D.stri pattrppp.ppppiiii
12 not visited(y]; visited(v)
graph(b]- [b). appence) www.ppleeee..... loceee
[v] tesp..acsppuchtttack vi
attected actected_attiimeree graph)
recurn dist
def da calpllaee possible locat ions actacted settienent,, graph): distances os calen ate distances Igraph, max nax_distance, y sini_nax_distaace (ctoeuusstaccoctsscanccs setscted_settiemeee
couution
Dy T, nag
meplint, Hint, input D.stripp -split (1)
list (int, input st.pprppfi ()11
(U)
range
(graph,
3, nap (int, input (.stripp)) (p.ppitt)))
max_distance v tind_nax (ww.dd.tttccccdsssceees
praphlal.aspenddel graphibl spgeeddl
atrected_sett set___ssttttmttt)
distances csleollll late distances (graph,
(grapa,
roturn count_pocsibie___. losstionssoosonniiiioooo
distancer__p 0, cistances_y)
nax_distance =
@ic__nxx_disttaeee iistanesstiittaaco
attected acteoted_etttLeeeettt. settienents)
for 1 in p:
max_distance
1t dist [1] max dix_ddstancc: distance diet(i]
-1
settlement attected, eeentte
[pttannneptt) nax_distance:
nox_distance clstanccs u seztlemant
diatu Igrash, 6)
nax
for 1 in p:
daf counk_poasible atw.aibia__ooctooaaaaoooaa (a, d, distances_ar distances_v): porsitle_iocttions
in distulil max_distance: aix_aietance dictu[s]
ranca E 11:
1: distancas_uli) and
cisty disigraph, ?
dictances_ <- d:
possible_iosatons pessibie_lossionns
for in sange 1);
if 0 distu[a] <u d and <- discv[i] <e
daf maint):
possitic_iocation
Fird Shw.e
print
pesntlactccted)
main''
(1lution:1
maint)
(a) Original program
(b) Transformed program
Figure 10: Original and transformed programs
26

---

Published as a conference paper at ICLR 2024
def Find dividers (eath_lergthh:
dividers diviser
while divisor diviaoe <e peli_lwogghe
1f path_leacth divisoz 0:
siviiders.ergeeeeeivviiee while patillenghh divisos -- 0:
patt_lenge //= divisor
divisor
Limpat 111 dividere 13 -2
if path length > 1:
dividers, agpend (path_)engeh)
whille
nr
return dividers
IF n. D:
dividecs.-pponte
def oa powoollateneeeeooooo (each_)) Find dividers lenge (dividera)
while a n //-1
return dividere
se:
def maintls
dividers. dovidees..eppeolto
gethglenggh length int (in
calculase www.eee_____oooeeeeeeeeeeeeeo colors lengh print (nex colorr)
print (B) elees
Original program
(b) Transformed program
Figure 11: Original and transformed programs
27

---

Published as a conference paper at ICLR 2024
Import
aount__eqquencea Inan,-owas, nur_days): mod 10x.917
dpusene clty.onn. [10] (nun]Lowarll) For
La toun__owas-l):
"p_oniqqe__ii noout [[0] (nm_,twwasl))
in range (nain Luwns11]]
minecctty count
day cange (nun)daya):
ap same city. coy_tttn22 [10]+ Inun (non_towne4))
for in range (oun__towxxll]]
dpuunique_si 2youm for in range (num towns+]]]
[[3] (oun_)oenat))
For i in rarge [,, mon_towwsslly 1):
Jos wwweeotty...
coun_towast--]]
dgLxam pity [1]
unique_city wttpeee_ctttoooooeeeeeeee
dp unique count [+][3]
impost sys resdline sys.etdin.
dpp wtt.m//sty...mmm2200.
dp same www._otttyyoyyllllcmc [31
1+5one i+some_city_com wwty_cccyymmm.
ned
N, X nip(ink, reed)ine(( 10a.S47
dp wwt.iggeeeittyonnn... 2[]]]]
op unuunnqueeecc city countecnnt22 (1]gg) 3+1
dpecc [:0] [1 (6-1) For range (8F1)
irique_city count txame city count)
dpus [19]-1841) for in zange (M+11]
dpecci:l [91
inique_nity [--]
For in range (6)
it ton_ttom----1:
opecat 1|0]+ (8+13 for in range (N+11)
op unique city (o
6pox [10] 1817) For in (N
(dp uni eng_ungeeeello wtw.eciityccuuurrl 2 Inun
(1nge X+11:
(-n)=janiuu__city wttp://cccoeeeeooooooo..o.o mod
for 1 in range (M+1-11:
keoc [3scc[1]
(1))
city_cottnt [d[:] in
dpsec2 (1]]g) (1 iskeco)
dp. city count2]
nod
dp sp__niguee__mm dp_unirue_city_comm city_couue
[d[:] foz in
(p]12(1)13] 1spus2(3]131 (xus+
keoo);
nod
[p-cs] [dpscc2
roturn eg__oos_ccsy_comm mun_ttwaee
mod kusi
12 M-i-j:
dat main
opus2 (3+1]
[d]
autldays
input s)litt))
count_ perranttegeecceeeooo (nUN_tOMnS, ner_days! print (result)
apsc: [d]:] for in dpscc2] idlil in
print (dpscc [M] [0])
main
(a) Original program
(b) Transformed program
Figure 12: Original and transformed programs
28

---

Published as a conference paper at ICLR 2024
def celoplate beauty (num, (nun_poiints, segnents, segnenta):
Langhh [[] (nnncppaoint
taf lengthh]] =1
fuz 1 in range run_ovini
[tast_iength(j1 for segneats
1]+101
Lail Jaii__ang mas (Lmap) 12
spine length (Ter (segnents iill ilrstail lergghhil
range non_pointsel11
cuburn max (apinnneeeg
def ma (eint)
mungpeinis, son_aegeenta (initt) For
topur 2 st
n.m [ine tat For Engut O (ppit D1
segnenta = [i:ll for i range fh, nanupoints +1)]
awg
[1=[] For in (yye1)))
range
range segmenta)s
4,4s [int E33 Far topar W -topl
pointl, point2 [int (i)
inpat dI.
segla. .append (b)
splis
=eglb" "pppendee
segnents/poingg sgpend (point21
Leil
segnent appoinn -apeedd (point))
tailll]
result calcuiste besuty inun_pointe,
For sr range 12
mm segnent segmenta)
Leep [sa21 For in seg[-]]-(0
pr int (seauls)
ta:115] (bemp2+1
terp [Ten (weg(1) wblll [H] range (er
name main
print (maxiteepp)
(a) Original program
(b) Transformed program
Figure 13: Original and transformed programs
29

---

Published as a conference paper at ICLR 2024
def Findd permotatior
C:
1f k
// oo
17 21
1: in //
cubarn List Crange (1, 11)
elses
permutat calcula www.eeeentatineeoo (i,
wiif roturn permection
wlsu
in /1
21 a In 17 1 13
zeluen Hone
retuen lia Irange 23,
151
e.se:
permutation caliall pernut add
retuen permutation
def osloulate perrmaaa
count, www.aaa..oon.eeeee.... Jon k11 0, 10 For rangein)), while k 0: +-
For: range (int (input (11)
cx 1 (rppt(.))l)tt
it (n - or k03-11h//21+-2)
for in range count, -11:
11.
1ixi -11
4-11
for in range rongell,rt]):
1.2 count kl. F count 1,
priat
toz 1 in rance (C):
print
co (at(()))) (, tor in zangein)
return
e0000:
5-
selcelate__emmeeeeeee ott._pprttttttooootttt.o.o K1: k, count, D, 1, x
count k-ntp ccoutt11 for 1 in range -n)
11 //
0, 0, 10 foz in rance (6)1, while
1[x]=1 ix+-2
1[2+count 1 +k],p a-couast2 / for in zange in):
K, count k + count toz in rançe (E, count, -11:
1e11:E]==01:
-[i]mp
p+11
k
for La 1:
1:2
count 1 kl. count 1,
pria:
in
print
1151
clae:
111]
inin-=1:
peint 11(k==0) print(-1)
in= or (03/t///22100//2
1111
doE main 1)
p-int
int (imput (1)
aLi2ik=m(an-onn
for range
map (1nt, input D (pllit
axia=(a,"* "an"","err"
permutacion pawnlpapettaiio I8, k]
print
12 parmutation is not Hone: print (spermusstion)
X, e. in sance [n]1,1
(ar11)//2 1,0,0,00
print (-1)
white (8>011
count countr
Original program
(b) Transformed program
Figure 14: Original and transformed programs
30

---

Published as a conference paper at ICLR 2024
Import
opunt_ (heighta)
out
change 0
False
whill
e Ien (heightts)
15 beights <e heights 1l
import eys
change lallllllt (heighes,
cuta change
nome inn lagn atd nee 131 ott di read fin itplit skyel (nag Hinys0??
if to del h+ightx(1] Falaw
regereve (1
cutxee
elae:
changeeC
Lpdaato heights (height.. 13
change
return cute
whi cillen Osky1: sky
deloulate wew.aaateeehnneeeeeeeeee (hiight,,
for 5 in (ang
change
Falze
if coskkyly (1 Jir
Cux
in range 1, 1, 11:
heighteld] heighes 5x1 break
else:
change heights (heights(t)
changebeskyy chargersky(E)
i2 change heightoli]]
t-True
change heighteli)
break
frue Prue break return change
cut coth-ne
hwwghts
skgli)
rangeri -1,
F-Fo1
12 heipatsl]] heighte 31: break
1--1
else:
slea:
acights(s) hoighcs
tor i in range 1i-2,-1,-1) if (4)
daf maint):
break
stdin. bodin.readiine
neights imopiitt, aye aye.stdin, readline spis:
1+-1
heights. reverse
changen0
(rutt
print (outs)
main
(a) Original program
(b) Transformed program
Figure 15: Original and transformed programs
31


Reviews:
[{"review_id": "uxoSd1oQlk", "reviewer": "Reviewer_MNi4", "summary": "This work explores applying a proposed data-cleaning pipeline (1: renaming variables, 2: refactoring into helper functions, and 3: inserting natural language comments to guide generation) to two major datasets (APPS and CodeContests). Fine-tuning a CodeLLAMA 7B model on these cleaned datasets dramatically improves fine tuning efficiency (requiring 8x less data to match the same performance) and improves accuracy a decent amount (often 1.2x-1.3x). They don't find the 3rd form of refactoring (introducing planning-based comments) to yield improvements, and through an additional experiment they narrow this down to be due to the inability of the model to generate good plans (as opposed to its ability to follow the plans).", "questions": "- In table 3 theres one missing entry – Pass@1 APPS Interview Distill. Where is it?\n\n- I don't understand the \"distill\" baseline dataset laid out at the end of 3.2, and referenced at various points\n    - My best guess is that you're doing synthetic data generation to generate a new dataset by prompting with few-shot examples from the modular dataset? Is it generating both the test cases *and* the solutions to them? Or are the test cases taken from somewhere and then its just generating solutions? \n\n- Two relevant pieces of work on synthetic data generation of code for LLMs are the Self Taught Reasoner (STaR) (Zelikman et al 2022) and Language Models Can Teach Themselves to Program Better (Haluptzok et al 2022). Those would be relevant to reference under the \"Synthetic data for LLMS\" section of Related Work, and could also relate to the distillation (though as mentioned before, I understand the distill baseline less).\n\nLow level confusing things:\n- Given that the 30% relative improvement is in Table 4, it's confusing that the caption of Table 3 brings up the 30% statistic (led to me spending a while trying to figure out which two numbers divide to get 30%, which is none in table 3)\n- Note: missing period in last paragraph of Section 1 right before \"Next\"\n- Typo at end of 2.1 with random sentence ending: \"steps quite effectively. effective in generating high-quality outputs.\"\n- The sentence \"We obtain three *parallel* datasets at the end of our cleaning process, one for each of renaming, modularization, and planning\" and in particular the world \"parallel\" is a bit misleading since at least to me it suggests that each dataset comes from applying a single transformation to the original dataset independent of the others, but actually the 3 transformations build on each other. This is clarified by Table 2 but would be helpful to have in the text as well.\n- Table 4 isn't actually labelled \"Table 4\" anywhere (since there's no shared caption)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This is a beautifully written paper, quite easy to follow and at just the right level of detail.\n - APPS and CodeContests are two very standard datasets so those were a great choice.\n - The experiment in Table 4b around ground truth plans is a very nice little experiment, I appreciate the inclusion of that\n- The data efficiency results in Figure 3 are quite good, showing 8x less finetuning data is needed to achieve the same Pass@1 when finetuning on the refactored programs.\n- I appreciate the smaller scale observations/insights on LLM prompting, which I think are generally nice thing to include in these sorts of conference papers for the community, e.g. \"Finally, in accordance with existing literature on prompting LLMS, we found that using simple and precise, low-level instructions improves the performance and accu- racy of the models in performing the operatons. Thus, for complex data cleaning operations, we find improvements by breaking it down and performing multiple operations iteratively.\"\n - The main results (Table 3 and 4a) are decent (not incredible, but reasonable in my opinion).", "weaknesses": "- The improvements in Table 3 are okay, not huge but still noticeable.\n- The planning results are also modest, but this is interesting in its own right, and the analysis of how ground truth plans would help considerably is a good way to isolate much of the problem to the plan creation rather than plan execution.\n- While most of the paper was easy to read, I was quite unclear on the distillation dataset baseline – see the Questions section for details\n- For more minor / easily fixed weaknesses see Questions section", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "waxiGu5uTK", "reviewer": "Reviewer_FtQG", "summary": "The paper shows that improving the “quality” of a code dataset can improve the performance of a CodeLlama7B model fine-tuned on that dataset. Specifically, for every source file in a dataset, the authors propose to use a __second__ instruction-tuned language model (gpt-3.5-turbo) to perform three types of transformations in order:\n1) rename variables to have semantically meaningful names,\n2) “modularize” the code by breaking up large chunks of code into smaller functions,\n3) prepend a “plan” before the code that summarizes the role of each individual function.\n\nThe authors use their synthetic dataset to a) provide few-shot examples to CodeLlama7B for in-context learning, b) fine-tune CodeLlama7B. Through this process, they show modest improvements in pass@k on the APPS and CodeContest data sets.", "questions": "1. Comments on Fig 1:\n    - In the renaming step, the variable `n` is not renamed everywhere. I understand that the actual LLM output can have mistakes, but maybe for an explanatory diagram this could be avoided.\n    - Instead of “a -> root_u,  b -> root_v, …” as the text above the arrow, it would be clearer to show the natural language instruction that you provided (“Rename the variables in the program to be…”). As it stands currently, it looks like the renaming (“a -> root_u,  b -> root_v, …”) is the __input__ to the model. Same comment for the modularization and planning steps too.\n\n2.  It’s not immediately obvious how the natural language “plans” are used (my initial understanding was that they are provided as a docstring for each function). Would be nice to clarify in Fig 1 that they are __prepended__ to the program as a comment.\n\n3. In the APPs benchmark, do you consider all problems from “codeforces”, “codechef” and “atcoder” ? Or is there some further filtering done after that ? If further filtering has been done, can you please clarify what procedure has been followed?\n\n4. What does this line “These cases are sometimes due to incorrect programs but more …. while only a single test solution is provided” mean ?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed idea is simple and an interesting approach to re-format data using a neural approach. Especially given the fact that the domain is code, transformations can be verified using an oracle - which in this case is a set of test cases. \n\n- The paper validates that by fine-tuning an LLM on a smaller good quality dataset, it is possible to achieve better/equal results than fine-tuning on a larger lower quality dataset - something that has been pointed out by many other papers in different context. \n\n- The approach can be used as inspiration for other domains where the LLM is capable of editing an existing solution but incapable of generating an entirely new solution. However, the efficacy of this approach might be significantly impacted by the presence/absence of an oracle.", "weaknesses": "- The experiment section of the paper reports numbers on the subsets of two datasets. It would be nice to clearly outline the filtering criteria that are used for each dataset. I have certain questions regarding this in the “Clarifications section”. \n\n- In many of the tables, there has been no reference or explanation to numbers that show a negative effect on results. For example, on the Code-Contest dataset, D_rename works worse than the baseline. It would be nice if the authors could be candid about this in their writing.\n\n- Additionally, I believe that the CL-7B + D_distill number is missing in table 4(a). Can you please include that number in the rebuttal ?\n\n- Overall, the effect of planning information is a mixed bag, and the conclusions are slightly confusing. For example, from the sentence, “Upon inspection of the generated solutions, we find that often the generated plans are imprecise or incorrect, highlighting that planning still remains a bottleneck.” – I am confused by this statement because I am unsure if this is because of the drawbacks of CodeLlama or a drawback of the paper’s approach. Overall I am unsure if D_planning actually supports the paper’s claim.  \n\n- The improvements on Code-Contests seem to be not as effective as would be expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qlSlHg0YZw", "reviewer": "Reviewer_DhCg", "summary": "This paper investigates data quality for code generation and finds that making the code more structured and readable leads to improved code generation performance. The authors build a data-cleaning pipeline to transform existing programs by 1.) renaming variables, 2.) modularizing and decomposing complex code into smaller helper sub-functions, and 3.) inserting natural-language based planning annotations. Experiments on two algorithmic code generation benchmarks indicate that fine-tuning on the transformed programs improves the code generation performance compared to fine-tuning on the original dataset.", "questions": ". How is the quality of the transformed code? \n\n. Does the effectiveness of the proposed model depend on LLMs such as ChatGPT?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": ". A nice idea to enhance code quality for code generation.\n\n. A series of experiments were carried out to evaluate the effectiveness of the proposed method. \n\n. The paper is easy to follow.", "weaknesses": "1. The proposed code transformations (i.e., renaming, modularizaing, and annotations) are a bit simple, and rely heavily on the capability of ChatGPT (GPT-3.5-TURBO).  It is unknown whether the proposed method can be utilized for other, more powerful LLM-based code generation systems. For example, in Section 4.2.2, the authors acknowledge that the poor performance obtained on the planning dataset may stem from the model's inability to generate accurate annotations, indicating that the effectiveness of the proposed model depends on LLMs. \n2. The authors propose two steps of data cleaning (i.e., renaming, modularizaing) to the original source code. However, the authors did not validate the quality of the transformed code (the authors only tested whether the transformed code can be consistent to the original data). For example, whether the variable names actually became clearer and more readable after cleaning, and whether the model accurately segmented the code into modules. Hence, it is unknown whether such simple transformation process can enhance the quality of the training data.\n3. As observed from the experimental results, the improvement of the proposed method could be insignificant and inconsistent. Some negative instances can occasionally be observed such as the $CL-7B + D_{modular}$ in in-context learning and the $CL-7B + D_{rename}$ in fine-tuning on the CODE-CONTESTS dataset. Unfortunately, the authors did not provide an explanation for the decline in these results, which may undermine the method's validity.\n4. Code generation could be achieved by prompting a general LLM such as ChatGPT directly as well. Also, it has been found that by improving the prompts, the code generation performance can be improved. The authors may discuss this approach to accurate code generation: Liu et al., Improving ChatGPT Prompt for Code Generation, https://arxiv.org/abs/2305.08360", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)