CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules
Authors: Hung Le
Hailin Chen
Amrita Saha
Akash Gokul
Doyen Sahoo
Shafiq Joty
Venue: ICLR 2024 poster (2024)
Keywords: code generation
language model
self-revision
program synthesis
modularity
llm
competitive programming
clustering
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce CodeChain, a novel framework for language models to generate modular codes through a chain of self-revisions conditioned by representative sub-module components.

Abstract: Large Language Models (LLMs) have already become quite proficient at solving simpler programming tasks like those in HumanEval or MBPP benchmarks. However, solving more complex and competitive programming tasks is still quite challenging for these models - possibly due to their tendency to generate solutions as monolithic code blocks instead of decomposing them into logical sub-tasks and sub-modules. On the other hand, experienced programmers instinctively write modularized code with abstraction for solving complex tasks, often reusing previously developed modules. To address this gap, we propose CodeChain, a novel framework for inference that elicits modularized code generation through a chain of self-revisions, each being guided by some representative sub-modules generated in previous iterations. Concretely, CodeChain first instructs the LLM to generate modularized codes through chain-of-thought prompting. Then it applies a chain of self-revisions by iterating the two steps: 1) extracting and clustering the generated sub-modules and selecting the cluster representatives as the more generic and re-usable implementations, and 2) augmenting the original chain-of-thought prompt with these selected module-implementations and instructing the LLM to re-generate new modularized solutions. We find that by naturally encouraging the LLM to reuse the previously developed and verified sub-modules, CodeChain can significantly boost both modularity as well as correctness of the generated solutions, achieving relative pass@1 improvements of 35\% on APPS and 76\% on CodeContests. It is shown to be effective on both OpenAI LLMs as well as open-sourced LLMs like WizardCoder. We also conduct comprehensive ablation studies with different methods of prompting, number of clusters, model sizes, program qualities, etc., to provide useful insights that underpin CodeChain's success.

Published as a conference paper at ICLR 2024
CODECHAIN: TOWARDS MODULAR CODE GENERA- TION THROUGH CHAIN OF SELF-REVISION WITH REP- RESENTATIVE SUB-MODULES
Hung Le, Hailin Chen, Amrita Saha, Akash Gokul, Doyen Sahoo, Shafiq Joty Salesforce Research
(hungle, hailin. chen, amrita. saha a @swlessorre.com com
ABSTRACT
Large Language Models (LLMs) have already become quite proficient at solv- ing simpler programming tasks like those in HumanEval or MBPP benchmarks. However, solving more complex and competitive programming tasks is still quite challenging for these models possibly due their tendency generate solutions as monolithic code blocks instead of decomposing them into logical sub-tasks and sub-modules. On the other hand, experienced programmers instinctively write modularized code with abstraction for solving complex tasks, often reusing pre- viously developed modules. To address this gap, we propose CodeChain, a novel framework for inference that elicits modularized code generation through chain of self-revisions, each being guided by some representative aob-modules generated in previous iterations. Concretely, CodeChain first instructs the LLM to generate modularized codes through chain-of-thought prompting. Then it applies a chain of self-revisions by iterating the steps: 1) extracting and clustering the gen- erated sub-modules and selecting the cluster representatives as the more generic and re-usable implementations, and 2) augmenting the original chain-of-thougle prompt with these selected poddule-implementaiion re- g--ennrrtte new modularized solutions. We find that by naturally encouraging the LLM to reuse the previously developed and verified sub-modules, CodeChain can significantly boost both modularity as well as correctness of the generated solutions, achieving relative pass @1 1 improvements of 35% APPS and 76% on CodeContests. It is shown to be effective on both OpenAI LLMs as well as open- sourced LLMs like WizardCoder. We also conduct comprehensive ablation studies with different methods of prompting, number of clusters, model sizes, program qualities, etc., to provide useful insights that underpin CodeChain's success
and instructing the LLM to
INTRODUCTION
It has been long-standing goal in Al to develop systems that can generate executable and functionally correct computer programs to solve complex problems (Manna & Waldinger 1971). recent years,
we have witnessed unprecedented progress in this field, specifically with the remarkable success of large pretrained language models or LLMs (Koubaa 2023 Wang & Komatsuzaki 202 Radford et al. 2019). Originally developed for natural languages, these models have been extended with
combination of code and text modeling capabilities (Rozière et al. 2023 Black et al. 2021 Chen
let al. 2021), resulting in good performance in code generation from natural language problem description (Li et al. 2023 Luo et al. 20233 Wang et al. (2023). However, when evaluated on highly complex coding tasks, the current SoTA models still cannot match a skillful developer (Hendrycks)
et al.
2021
Li et al. 2022 Shinn et al. 2023), mostly due to their naive generation approach.
Most prior approaches with LLMs adopt naive generation method in which the models would typically generate the code solution as a single monolithic block of code instead of decomposing the task into logical sub-tasks. Another limit of this naive generation approach is that the models would simply generate a large number of solutions independently, with the hope that one of the solutions
https: htttp:://gitub..coo weweallaooraare forceA Ressarcch/Codehhaii

---

Published as a conference paper at ICLR 2024
Problem Description
Public test cases
Private test cases
length beauthul
substring of string becomes beausitul...
Imput:
Input:
paindroms...
5 4rrbbaccbn 3ta2
10
Bknabchabsbabbaat
cost string minimum
51n2
21m2
You are given siring of lnnght ft..
operotions
Outgut:
Outgat:
You have to answer quertes calculate the cost of the substring of the string from |_ --th to (ithh posiion, indusive....
compare/
+/
+
for final
analyze
3/3
Developer
reuse
Figure 1: [Top] An example code generation task from CodeContests (Li al. 2022) where the problem description and public test cases are provided inputs to the model. [Down] We illustrate a typical problem-solvine process which a developer attempts to solve the problem iteratively, revising and reusing parts of their previously developed codes until satisfied.
would pass all the private test cases (Chen et al. 2021 Li et al. 2023 Austin et al. 2021). More
recently, Li al. (002); ); Chen et al. (2023b): Zhang et al. (2023b) propose to sablsample output programs using some forms of feedback from the public test results. However, these approaches assume that the sub-sampled programs could pass the private test cases, even without revising or
debugging the programs. Some recent works like (Zhang et al.) 2023a Olausson al. 2023 Le
ct al. 2022 Chen et al., 2023c a Shinn et al.) 2023) have addressed this by performing self-revision with LLMs, utilizing feedbacks such compiler error messages, test outcomes, and natural language explanation to improve the generated solutions. However, these approaches limit to using only independent feedback from individual solutions, neglecting potential collective insights from all generation samples or their sub-components.
On the other hand, in today's agile development environment, experienced developers fully familiar with the concept of modularity in programming. Given a problem, they would instinctively write solutions that modularized by high-level logical sub-tasks and sub-modules. The developers would then keep testing and analyzing their implementations, altering modular components from their previously developed solutions to efficiently improve their final solutions (see Figure [ Inspired by this problem-solving process, we propose CodeChain, novel inference framework to improve code generation in LLMs through chain of sub-module based self-revisions (see Figure 2
Specifically, in CodeChain, to incorporate modularity in code generation, we first introduce chain- of-thought prompting to instruct LLMs to decompose their solutions into modular segments. Each modular segment represents abstract function that is intended for high-level logical sub-task. To leverage this modularity in programs, we propose to further improve the generation process through a chain of self-revisions each of which is conditioned by set of sampled sub-modee uu-mmodules as follows: (i) we first extract the sub-modules found in generated programs and group them into clusters. Within each cluster, we sample the centroid sub-modules and treat them representative and reusable code parts for self-revision. (ii) We then augment the original chain-of-thoughh prompt with these selected sub-modules and instruct LLMs generate new modularized solutions. With this approach, LLMs can receive the collective insights from modular components of all past generation samples to improve their future generations, imitating the problem-solvingg process of an experienced developer.
Our experiments show that CodeChain can significantly boost LLM performance and achieve SoTA performance on challenging code tasks in APPS (Hendrycks al. (021I) and CodeContests (Li et al. 2022). Concretely, CodeChain improves the average pass@/ performance by more than 35% on APPS and 76% on CodeContests. We also observe consistent improvements for both OpenAl LLMs as well as open-sourced LLMs such as WizardCoder (Luo et al., (023). We further conducted comprehensive ablation studies, including analysis in single vs. multi-step revisions, feedback types, number of clusters, etc., and derived useful insights behind CodeChain's 's success.
2 RELATED WORK
Broadly related to work is the research of large Transformer-basee language models (LLMs)
(Koubaa 2023 Brown et al.) 2020 Radford et al. 2019 Wang & Komatsuzaki 2021 Touvron et al.
2023a Originally designed for natural language processing, these models have been extended to
2

---

Published as a conference paper at ICLR 2024
Chain of Self-revisions
Evaluation
+/
Step1: CoT prompting
Pretrained Code LLM
Modularized solutions
Private test
Problem
Step6: Re-generate
Step2:
Step3:
Description
Reusing/Adapting selected sub-modules
Filter by public
Extract modular functions
cases
Encode
Step4:
def func (x)
Clustering
Funcsion....
aob-modules
octputellat
Step5: Representative sub-modules
Embedding space
Figure 2: An overview of CodeChain: a pretrained LLM is first instructed with chain-of-thought prompting generate set of modularized solutions. Generated sub-modules then extracted from potentially correct solutions and grouped into different semantic clusters. The cluster centroids are selected as representative aub-modules to condition the next self-revision round. The model is instructed to reuse or adapt these modules into its revised solutions.
learn from large-scale code data and become proficient in understanding contexts and generating outputs in programming languages (Rozière et al. 2023
Chen et al. 2021 Li et al.) 2023 Gunasekar
al. 2023 Wang al., [023] Nijkamp
al. 2023).
Complementing the long-standing code
generation research (Gulwani et al., 2012 Kurach et al.
2015 Devlin et al. 2017 Parisotto et al.
2016), LLMs can generate programs of more general purpose programming languages, correctly
following programming syntactic rules (Lu al. 2021 Clement et al.
2020 and solving simple
coding problems with reasonable accuracy (Lai et al. 2022 Chen et al.,
2021 Austin et al. [021).
In more direct relevance to work is the recent line of work for improving code generation qualities through output feedback. Chen et al. (202T) introduced simple filtering approach by selecting only output samples that successfully pass the public test cases. AlphaCode (Li et al. 2022). CodeTT (Chen et al. 2023b), and MBR-Exec (Shi et al. 2022) proposed to generate more test cases and use more sophisticated rule-based methods rank generation samples by their execution behaviors. LEVER (Ni et al. 2023), Coder-Reviewer (Zhang et al.) 2023b) and Code Rankers (Inala et al. 2022) follow similar principle but introduce more model- ranking methods.
Recently, more related works have been proposed to boost generation quality through iterative self- revisions. Zhang al. (2023a) utilizes test outcomes from public test cases form feedback for models to self-revise their codes. Self-correct (Welleck et al. 2023) and CodeRL (Le et al.) 2022) introduce secondary models to predict the correctness output programs and revise them accordingly. Self-debug (Chen et al.) (023c).. Sef-refine (Madaan et al., (023), and Reflexion (Shinn ctt al. 2023) propose to facilitate better code revision with synthetic natural language explanation or reflection self-generated by LLMs. Self-repair (Olausson al. 2023) and ILF (Chen et al.) (023a) follow similar strategy but highlight the use of natural language explanation provided by human experts. Different from prior approaches, we propose to generate modularized programs and sequentially revise these programs using more representative and reusable sub-module programs (please Appendix Aor more systematic comparison).
CODECHAIN FRAMEWORK
3.1 CODE GENERATION TASK
We treat code generation as sequence-lo-sequennc task, which consists of problem description as an input sequence and an output sequence of a flattened solution program: W (i....., why) with V. Typically, language model generates code sequence by autoregressively sampling
3

---

Published as a conference paper at ICLR 2024
Chain-of-thought prompting for code generation
"Instruction"
Develop well-structured Python solution the provided problem that obeys the constraints and passes the example cases, Ensure modularity and considering potential edge cases and failures. Start by outlining required code modules, including function headers and signatures. Subsequently, proceed to implement each module to create the final code.
simpler terms, create clean and organized Python solution for given problem. Break it down into smaller parts (modules) with clear function names and input/output specifications. Once the structure ready, write the code for each module to complete the solution.
Figure 3: An example of CoT prompting for code generation in CodeChain. The model is required to first outline the solution in terms of sub-module signatures, each of which is intended for solving high-level sub-task in the final solution. The model is then required to implement these sub- -odulls and combine them into final solution (see Appendix [F] for a full version of the prompt).
tokens Wi from the parameterized conditional distribution pa(-1t01:-11 D). Generated codes evaluated against (private) test cases to check the execution correctness (Hendrycks et al. 2021 Chen al. 2021 Li et al. 2022). The test cases comprise set of input-output pairs {(e,, 0, 11 An output program W is correct when (ig) = 0j for all j E J). If the problem description contains some test cases, we treat these as public test cases: {( o'n ) (usually M < J).
Models have the option use these public test cases to improve its generation.
3.2 MODULAR CODE GENERATION WITH COT PROMPTING
LLMs, especially the instruction-tuna ones, can follow complex natural language instructions describing novel unseen tasks (uuyag et al. 2022 Touvron et al. 2023b Wang et al. 2023). They have shown remarkable performance in many reasoning-basse tasks when they are instructed to solve problem step-by-step. i.e., chain-of tffthhugttt (CoT)) prompting (Zhou et al. 2023 Wei et al.) 2022, Kojima et al. 2022). We propose to adapt this technique to generate codes by instructing the models to first outline the required sub-modules, generating only their function headers and docstrings describing their intended usage. The model is then instructed to implement the modules and ultimately combine them into final solution. Following this generation scheme, the output distributions:
define
S. pa po in ~ po ($:),
sub-modules, including the function headers and docstrings (1)
tokens final solution
where {5:} is the set of sub-modules outlined by the model. We append the instruction with one-shot demonstration. Figure presents one example of the instruction prompt.
As illustrated further Figure in the Appendix, this technique encourages the model decompose program into natural boundaries, e.g., sub-modules, similarly to how developer often tackles challenging coding task by breaking solution into modular components. Though this is a more pragmatic style of code-development, empirically we have found that this prompting approach can adversely impact the correctness of the generated end-to-end solutions (shown later in Table 33). This is expected as most of the current L.LMs are not pretrained to generate perfectly functioning modularized programs. To address this, we introduce Chain of Self-Revisions which allows the LLM to iteratively revise solution by re-using adapting some of the representative sub-modules from the previous iterations. Further, we also establish empirically that our self revision technique indeed benefits more from this modularized style of code generation.
3.3 SELECT REPRESENTATIVE MODULLS ACROSS MULTIPLE SAMPLES
Prior studies have demonstrated the benefits of generating multiple samples and selecting the best ones based on different ranking or scoring schemes (Li et al. 2022 Chen et al.) 2023b Zhang et al. 2023b). A common approach is to simply select the representative candidates based their execution results on the public test cases (Li et al. 2022 Chen et al. 202I). However, all prior methods only select end-to-end program candidates. On challenging coding tasks, is extremely to obtain such
4

---

Published as a conference paper at ICLR 2024
program-level correctness and the selected candidates still likely to fail when tested on private test cases. Thus, we propose to perform selection at sub-module level instead of program level.
Specifically, given a generation budget of N samples, extract and combine the set of sub-modules across all samples $ {{S; } for all n E {1, N), where {S; In is the set of sub-module in the n-th generated sample. We then perform K-mean clustering on this set of sub-modules to group them into K clusters. For each of these clusters, we then extract "centroid" (representative) sub-module Cn that is closest the true centroid of the cluster in the embedding space:
O₁ = arg min |SS pk
(3)
where Sh is the embedding representation of sub-module Š; in cluster k and is the centroid of cluster By selecting these "centroid" sub- modules, we can sample the most semantically representative and re-usable functions across all samples. Note that in cases where public test cases are available, one can filter out any failed samples first before further applying our selection method.
3.4 IMPROVE CODE GENERATION WITH CHAIN OF SELF-REVISIONS
Prior approaches improved code generation by regener- ating code conditioned by different types of feedback, ranging from compiler error messages to natural language
Self-revise prompting with
representative sub-modules
explanation of the output programs (Chen et al.] 2023a
Madaan al. [2033 Chen al.) 2023c Shinn al. 2023 Le et al.j 2022). However, these methods focus on the feedback extracted only per individual generation sample.
"Instruction"
related utility Python functions, reuse adapt them as much as possible into your solution (create new unique functions needed)...
...Given
We propose to utilize new type of feedback in the form of clustered sub-modules extracted from all the N gener- ated samples (as described in Sec. 3.3). Augmenting our original CoT prompt with the implementations of these representative sub-modules LLM to re-use or adapt these functions when generating the code conditioned on that prompt in the subsequent vision rounds. Specifically, revision round R, the output token is sampled from the conditional distribution:
### TASK:
<eproblem>>
### RELEVANT FUNCTIONS:
<<gub-modules>>
explicitly encourage the
### RESPONSE:
Figure 4: An example of prompting to self-revise programs. The original in- struction from CoT prompting (Fig. 3 is combined with this instruction and the (4)
we ~ pal
(..),, (.--1,D)) (R
model is provided with set of repre- sentative sub-mot modules selected from previously generated sam- ples. Please refer to Appendix [F for full version of the prompt.
where ČR- (CR-1) =k=1 is the set of all centroid sub- modules from the previous generation round R 1. In
round Rt the new sub-modules are regenerated by the conditional probability (revised version of Eq. M]:
SR
po po CR-II D)
(5)
We enable this self-revision procedure by prompting the LLM with an additional instruction. Figure presents an example of the prompt with the new instruction. This style of self-revision with selective sub-modules is reminiscent of the code reuse process. In today's 's agile code-development environment, developers typically re-use or adapt snippets of previously developed code order to program more modularly, accurately, and efficiently. Inspired by this process and combined with our representative sub-module selection method, our CodeChain framework allows the LLM to iteratively improve their generations more efficiently through chain of reuse- reuse-based self-revisions
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Benchmarks. We demonstrate the efficacy of CodeChain on challenging code generation tasks, specifically, on two major benchmarks: APPS (Hendrycks et al.] 2021), and CodeContests (Li et al..
5

---

Published as a conference paper at ICLR 2024
Table APPS test results: results with f for models finetuned on APPS training data
(a) Performance by pass @ (%)
Model
Size
Introductory
Interview
Competition
All
Codex
12B
4.14
0.14
0.02
0.92
CodeTT
770M
6.60
1.03
0.30
2.00
CodeRL+CodeTT5 i
770M
7.08
1.86
0.75
2.69
text-davinci-002
7.48
Self=edit-text-daviniieee
7.94
code-davinci-000
29.30
6.40
2.50
10.20
WizardCoder
15B
26.04
4.21
0.81
7.90
CodeChain WizardCoder
15B
26.29
7.49
3.75
10.50
GPT3.5
48.00
19.42
5.42
22.33
GodeChain+GPT3.5
54.50
28.11
12.38
30.24
(b) Performance by pass I (%) with outputs filtered by public/synthetic tests
Model
Size
Filtering
Introductory
Interview
Competition
All
Codex
naive
22.78
2.64
3.04
6.75
CodeRI++CodeT5
770M
naive
17.17
6.78
4.88
8.48
code-davinci-002
naive
43.60
13.30
7.00
18.10
code-davinci-002
CodeT
47.30
14.30
6.20
19.28
GPT3.5
CodeT 61.52
30.57
9.46
32.54
God3
CodeT
62.72
32.96
15.08
35.34
2022). A majority of test samples from these benchmarks are curated from competitive programming platforms such as Codeforces making them an appropriate bed to evaluate our approach. Please refer to Appendix C Cnd 6 for more details of the benchmarks.
Evaluation. We followed (Hendrycks et al. 2021 Chen al.. 2021 al., 2022 and evaluated the models using the passing rate metric pass @ k, defined as the percentage of problems solved by using & generated programs per problem. We focused mainly on pass@@ this work and followed (Chen et al. 2021 to calculate the normalized passing rate given a generation budget of N outputs per problem. apply CodeChain, we fixed the budget each generation/revision round to generation samples per problem. After the first round of direct generation, we let the models self- revise generated codes for up to rounds of revision. On APPS and CodeContests, we reported the results the test split following the best self-revision round performance the validation set. Across all benchmarks, we fixed the one-shot sample in CoT prompting and revision prompting. We randomly selected this one-shot sample from the APPS training split (see Appendix G
20
Base language models. We applied CodeChain to both open-sourced and closed-sourced pretrained LLMs, including OpenAl's G35 and GPT4 (Koubaa 2023). and WizardCoder (Luo et al.) (023). We evaluated different versions of WizardCoder, with model sizes ranging from IB to 34B parameters.
WizardCoder models are instruction-tume from strong foundational code LLMs, including StarCoder
(Li et al. 2023) and Code LLaMA (Rozière et al. 2023). For OpenAl models, we obtained the generation samples by prompting through the public API access For WizardCoder,
utilized the
HuggingFace-hoste hostee model parameters (Wolf et al. 2019) and vLLM (Kwon et al.) 2023) to generate programs. We adopted default temperature 0.6 to generate output tokens and a max output length of 2048 tokens. Finally, to fairly compare L.MM generation capabilities, we chose to use StarEncoder (Li et al. 2023) to embed sampled sub-modules throughout all experiments.
4.2 EXPERIMENTAL RESULTS
Results on APPS. We compare our approach with prior LLM baselines like Codex (Chen et al. 2021), CodeT5 (Wang al. 2021), and code-davinci. as well code-revision methods such as Self-edit (Zhang el al. 2023a). CodeRL (Wang et al.) 202T Le et al. 2022) and Self-repair (Olausson al. 2023). Table shows that CodeChain, when applied with base LLMs such as GPT3.5 and WizardCoder 15B, can achieve significant performance gains by the pass @ Specifically, CodeChain
hhttps://codefot com/
gpt-3.5-turbo-166 16k and gpt-4 httph:tpp//oc.c....m
/ttt:://aaaatomm httss:::/ppattoooo forn.o com/ doc/ /mcdel s htwp://meeee///eiiieeeeeee overview

---

Published as a conference paper at ICLR 2024
Table Comparison with Self-repair: following Olausson al. (2023), we reported the results on the same subset of 20 samples on APPS test split using GPT3.5 and GPT4 as base models. Please refer Table for the full list of this subset.
Model
Feedback source
Introductory
Interview
Competition
All
Self-repair+GPT4
GPT4
42.64
19.33
3.67
33.30
Self-repair+GPT4
Human
62.21
45.67
14.67
52.60
GPT3.5
30,00
18.33
0.00
23.75
CodeChain+GPP335
Sub-modules
31.67
27.86
0.00
26.35
GPT4
42.86
18.33
13.33
34.75
GodeChain+GPT4
Sub-modules 71.07
55.00
23.33
61.50
Table APPS validation results by pass / (%). we tested CodeChain+GPP335 for self-revision round by aspects: prompting. filtering by public tests, and sampling methods for revision (R: random, C: centroid, P: whole programs, and M: sub-modules).
CoT prompting
filter by public tests
Sampling for revision
Introductory
Interview
Competition
All
39.00
26.50
12.50
26.00
R-P
12,40
2.00
0.61
5.00
C-P
23.27
9.00
3.80
12.02
C-P
45.20
28.03
9.80
27.68
33.50
23.70
10.10
22.43
R-P
24.40
18.80
9.20
17.47
C-P
31.33
23.70
10.10
21.71
C-P
45.50
33.17
11.80
30.16
R-M
49.30
36.90
12.40
32.87
C-M
52.00
38.83
14.50
35.11
can achieve 10.50% pass@1 @1 with WizardCoder the base model, and 30.24% pass@1 @1 with OpenAI GPT3.5 as the base model, establishing a new SoTA result on APPS. Previous works (Chen et al. 2021 Li al. 2022) introduced additional performance results by filtering out generation samples that fail public tests and computed pass@k & on the filtered set. In this work, we followed the setup proposed by CodeT (Chen et al., (023b) which utilized more advanced filtering with synthetic test cases (see Appendix F for the prompt we used to generate test cases). Table 1b shows that when evaluated on filtered code samples, our CodeChain+GPT3.. can achieve So results across all levels of problem difficulty with average of 35.34% pass @
From Table la when compared with related approaches such as Self-edit and CodeRL, we observed significant relative performance gains when using CodeChain. In Table 2 following Olausson et al. (2023), to compare with Self-repair, we evaluated our approach over the same test subset of 20 samples (14/3/3 samples of http://tttyyinteree/////pppiitte as base models. We observed that CodeChain can improve the performance with both base models, with more significant gains using GPT4. Specifically, CodeChain+GPP4 can achieve SoTA result of 61.50% pass@ I on average, even outperforming Self-repair+GPT4 with human feedback.
level), using both GPT3.5 and GPT4
Analysis on single-round self-revision. To understand the benefits of CodeChain, we conducted experiments with different variants on the validation split of APPS. Table presents the results on single-round self-revision by main aspects: prompting. filtering by public tests, and sampling meth- ods for conditional revisions. First, we observed that without self-revisions (i.e. direct generation), CoT prompting actually negatively affects the model performance as compared to normal prompting. This observation might be due to the fact that pretrained L.MMs are not designed to generate perfectly modularized solutions (they were pretrained on public Github codes without filtering for modularity). observe that the modularized approach is better, achieving
However, after applying self-revision, better performance gains than non modularizzd solutions.
Secondly, we found that the best strategy select representative codes for conditional revision is through clustering. This method can reduce noisy data points and create better form feedback to improve the generated codes. Finally, observed that clustering alone is not sufficient select the optimal representative samples. Additional filtering by public tests is needed to first shift the output distribution to more likely correct samples before clustering the outputs. To avoid the need for public
7

---

Published as a conference paper at ICLR 2024
CodeChain Introductory
Self-debug (UT)
Self-debug (expl)
Foxed
Increasing order
Dynamic
Depreasing order
Interview
Competition
1.8
60.0
44,0
22.0
1.6
52.5
38.0
17.8
1.4
1.2
32.0
37.5
26.0
9.3
Revision Round
30.0
20.0
5.0
Figure 6: we tested CodeChain++PP33.5 different setups of cluster numbers and reported the average relative pass@1 I im- provements from direct generation (round 0).
3
3
Revision Round
Revision Round
Revision Round
Figure 5: APPS validation results with chain of self-revisions: we tested CodeChain+GPT3.. 5.5 for self-revision rounds and reported pass@) / in each problem difficulty level. Using GPT3.5 as base model, we compared with related approaches, including Self-debug (with unit test (UT) feedback or explanation (expl)) (Chen et al. 2023c) and Reflexion (Shinn et al. 2023).
test cases, suggest exploring better embedding models that can group output samples not just by their programming semantics but also by their functional correctness.
Analysis chain of self-revisions. To analyze the trend of model performance over chain of self- revisions, we monitored the passing rates of direct generation and 5 subsequent self-revision rounds. Figure 5 presents relatively consistent improvements in all levels of problem difficulties, with optimal performance gain obtained in revision round and slight performance drops in round One possible reason for these performance drops is that the selected output samples become overfitting the small set of available public test cases, negatively affecting the passing rates of subsequently revised codes on more extensive private hidden test-suite.
1B
O
7B
3B
15B
18.0
13.5
Secondly, we also observed that on different levels of problem diffi- culties, CodeChain has different rates of performance improvement. Specifically, we found that more challenging problems (i.e. compe- tition and interview level) benefit more from CodeChain than basic problems (i.e. introductory level). Similar observations can be seen on open-sourced WizardCoder (Luo et al. 2023), with clearer perfor- mance trends on 7B, 15B, and 34B model sizes (see Figure 7
9.0
4.5
0.0
1
2
3
4
5
Revision Round
Analysis by types of feedback. In Figure 5 we also observed that CodeChain achieve better performance than other related self- revision approaches using other types of feedback, such as test out- comes with natural language explanations (Chen et al. 2023c) reflection (Shinn et al. 2023). Note that CodeChain can be comple- mented with other self-revision approaches such as Self-debug by combining different feedback types and selecting more diverse and representative sub-modules, even on generation samples that initially fail public tests.
Figure 7: APPS valida- tion pass@1 results of WizardCoder- to 34B. The dotted lines are direct generation results.
Analysis by number of representative sub-modules. One hyper-parameter of CodeChain is the number of clusters each round of self-revision. We experimented with different scheme: (i) fixed number of clusters across all rounds to K; (ii) decreasing order number of clusters: [K;}
(K, K 1,.... 1): (iii) increasing order number of clusters: (K,) K, + 1....:: (iv) dynamic number of clusters based on the silhouette coefficients (Rousseeuw) 1987). We selected K = 5 for all experiments. From Figure 6) we observed that the best approach to set the number clusters is to follow decreasing order. This scheme offers the models more diverse centroid sub-modules in the beginning with a larger number of clusters. Towards subsequent revision rounds, a smaller number of clusters is more beneficial as the sampled sub-modules become more and more closely semantically similar over time. We found that this scheme is reminiscent of the model training paradigm moving from exploration exploitation, the models become more confident in their generation.
8

---

Published as a conference paper at ICLR 2024
GPT3.5
CodeChain
Val
Test
Wizardooder
CodeChain
Model
Size
Filtering
pass
pass
pass
pass
16.0
code-davincci0012
1.00
WizardCoder
15B
1.11
3.18
1.98
3.27
12.0
CodeChain
15H
2.35
3.29
2.48
3.30
GPT3.5
6.81
16.23
5.82
11.16
8.0
CodeCCain
12.86
16.91
10.27
14.11
code-daavinc--022
CodeT
3.20
GPT3.5
CodeT
17.30
11.34
0.0
+CodeChain
CodeT
17.91
13.75
Revision Round
Figure 8: CodeContests results by pass@1 (%): we report the results of CodeChain using WizardCoder-1555 15B and GPT3.5 as base models. Left: test and validation results. Right: valida- tion results over sequential self-revision rounds. The dotted lines are direct generation results.
Results on CodeContests. Figure presents the results of CodeChain with WizardCoder- 15B and GPT3.5 the base models. We observed that both pass@1 and p CodeChain can achieve significant performance gains as compared to direct generation on the corresponding base models. Applying additional filtering method (Chen al.) 2023b), CodeChain+GPT3..5 can achieve the SoTA results of 13.75% pass / on the test split. As op- posed APPS where optimal performance was reached revision round 4, from this valida- tion results we noted that the performance kept improving till the final revision round. Differ- from APPS, we used the official public test cases available in the CodeContests benchmark. These test cases are generally more diverse than the ones we manually extracted in APPS, and hence, make the re- vised codes less overfitting even in the 5th revision round.
Direct Gen
CodaChain RO
CodeChain
90
Qualitative Results. To understand the modularity and
reusability of CodeChain generation, we conducted exper- iments to evaluate these qualities on randomly sampled generated programs. Specifically, we prompted GPT4 with instructions to rate output samples following a Likert scale from 0 to 5 where is the highest score for optimally mod- ular/ reusable programs. Please refer to Appendix F for a full version the prompt. In this experiment, we reused the GPT3.5 generated samples for the set of 20 random test tasks from Table 2 Figure shows the distribution of output samples by Likert scores in each quality. We observed that when using CodeChain, GPT3.5 is more likely to generate programs with high levels of modularity and reusability, with the majority of outputs rated to on the Likert scale. This is significantly higher than the conventional direct generation approach, with about 80% of time generating non-modular non-reusable codes (i.e. score 0). For additional experimental results and qualitative examples of CodeChain, please refer to Appendix and
67.5
67.5
45
45
22.5
Modularity Score
Reusability Score
Distribution of output sam- ples (%) by code qualities in the APPS test subset. We obtained the qualitative scores by prompting GPT4 with specific evaluation instructions.
Figure 9:
CONCLUSION
We present CodeChain, a novel inference framework to improve code generation through chain of self-revisions and sampling representative sub-modules. In CodeChain, we introduce chain- of-thought prompting to generate more modularized programs, which creates natural boundaries for the models to sample parts the solutions for reuse and revision. each revision step, we iterate between selecting representative sub-modules and augmenting thain-of-though prompting with these selected sub-modules. Our experiments indicate the significant performance improvement of CodeChain when using OpenAl GPT open-soureed WizardCoder the base models, achieving new SoTA results on APPS and CodeContests benchmarks. We provided comprehensive ablation studies understand the contributing factors behind CodeChain's outstanding results.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language 2021.
models. arXiv preprint arXiv:2108.077
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autore- gressive language modeling with mesh-tensorflow. URL https://doi. org/10.5281/zemodd. 5297715, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,0020
Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel Bowman, Kyunghyun Cho, and Ethan Perez. Improving code generation by training with natural language feedback. arXiv preprint arXiv:2303. 16749, 2023a.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, 2023b. URL
net/ fortt//orum?/ -it:tii-kwwttttttt Cmu 9c
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint wxXiv:210.033744 2021.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint urXiv:2304.088 /28, 2023c.
Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. PyMT5: multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lunguage Processing (EMNLP), pp. 9052-9065, Online, November 2020. Association for Computational Linguis-
tics. doi: 18.18////v www....emmlmmmiin...mm URL htt.tssss/acaaaathooooyoooo
emmnpp p- main. 728
Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. In International conference machine learning, pp. 990-998. PMLR, 2017.
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou CodeBERT: A pre-trained model for pro- gramming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp- 1536-1556 1547, Online, November 2020. Association for Computational Linguistics... doi: 10.18653/v 1/2020 1/2020.nnndiigg--s-eeeeae mondinns-emnl..13. 139. URL htttps:/aaclanthholoo..oooo Findings emn P. 139
Sumit Gulwani, William R Harris, and Rishabh Singh. Spreadsheet data manipulation using examples. Communications of the ACM, 9555)9971155, 2012. 2012.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks all you need. arXiv preprint arXiv:2306. 1644, 2023.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps. NeurIPS, 2021.
Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnación, Shuvendu K Lahiri, Madanlal Musuvathi, and Jianfeng Gao. Fault-aware neural code rankers. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. httt.htts:///poot
ew. net/forum? htt.rrmmmmid.......nnpobb..
10

---

Published as a conference paper at ICLR 2024
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213 2022.
Anis Koubaa. Gpt-4 vs. gpt-3.5: A concise showdown. 2023.
Karol Kurach, Marcin Andrychowicz, and Ilya Sutskever. Neural random-access machines, arXiv preprint M.Xiv:5111.06399 2015.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/22 11011, 2022.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning, Advances in Neural Information Processing Systems, 35:21314-2113 2022.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint wrXiv:2305.06666 2023.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. arXiv preprint 0rXiv:2203.07814, 2022.
Shuai Lu. Daya Guo, Shuo Ren, Junjie Huang. Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang. Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng. Shengyu Fu, and Shujie Liu. Codexglue: A machine learning benchmark dataset for code understanding and generation. In NeurlPS Datasets and Benchmarks, 2021.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardeoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.0855 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprini arXiv:2303. 17651, 2023.
Zohar Manna and Richard Waldinger. Toward automatic program synthesis. Commun. ACM, 14(3):151-165, mar 1971. ISSN 0001-0782. doi: 10.11457362566332256 URL https
362568
Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, pp. 26106-26128. PMLR, 2023.
Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. Codegen2: Lessons for training Ilms on programming and natural languages. arXiv preprint ar 005:02300..0.309 2023.
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. Demystifying gpt self-repair for code generation. arXiv preprint arXiv:2306.094: 099966 2023.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint 250 2022.

---

Published as a conference paper at ICLR 2024
Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. Neuro-symbolic program synthesis. arXiv preprint arXiv:1611.0185; 2016.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, nla Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Peter J Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65 66, 1987.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zeutlemoyer, and Sida I. Wang. Natural language code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3533-3546, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v 1/2022.cmnlpmmaii
231. URL atttps://acee htttpss//ac.anntooooyyooooo org/ 2022 emn.p-ma wnw.mainni..... 231.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0900
2023b.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https: //gi thub. com/k / www.m//iinofiiooomoo fo wttp:///essttttaarsssaaaa
jax May 2021.
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In EMNLP (1), pp. Association for Computational Linguistics, 2021.
Yue Wang. Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922. 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. thain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24833 2022.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and
Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh International net. forum?
Conference on Learning Representations, 2023. URL
Id=hH36Je@ZDaO
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-ar natural language processing. arXiv preprint 1rXiv:1910.0377,, 2019.
Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. Self-edit: Fault-aware code editor for code generation. arXiv preprint arXiv:2305.000 040877 2023a.
Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida Wang. Coder reviewer reranking for code generation. In International Conference on Machine Learning, pp. 41832-41846. PMLR, 2023b.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023, URL
12

---

Published as a conference paper at ICLR 2024
A COMPARISON WITH RELATED METHODS
Table 4: A comparison of CodeChain and related approaches by 4 aspects: (i) code execution: whether the method utilizes execution outcomes output programs on public/synthetic test cases; (ii) representative samples: whether the method sub-samples outputs for evaluation/ revision. (iii) supervision free: whether the method requires model to be finetuned on specialized tasks, such as correctness prediction or bug detection. (iv) iterative revision: whether the method allows model to self-revise output programs multiple times.
Code execution
Representative samples
Supervision free
Iterative revision
Approach
CodeRanker (Inala et al. 2022
LEVER (Ni
2023
Coder-Reviewerr (hang et al. 2023b
AlphaCode (Li al. 2022
MBR-Exec (Shi et al. 2022
4
CodeT (Chen et al. 2023b)
Self-correct (Welleck al.
2023)
ILF (Chen et al.) (2023a)
CodeRL (Le et al. 2022
Self-edit (Zhang et al. 2003aa
Self-refine (Madaan al. 2024
Self-debug (Chen et al. 2023c
Self-repair (Olausson al. 2023
Reflexion (Shinn et al. 2023)
CodeChain (ours)
For systematic comparison between CodeChain and related approaches, please refer to Table 4
B DEMONSTRATION OF MODULARIZED CODE GENERATION
Please refer to Figure 10 for an illustration modularized code generation.
count_ www.tttttttt.. (oords)
(1ab)
Descriptions
duf main and (ro edal
sord[-1]]
bisasy
moddle
def reserzze words (verde, revereed indices): reserzed words
der reverzee worde (worde, reversed pewerrec_indioeer indiows)
end-to-end solution
Step 1: Outline sub-modules
Step Implement sub-modules and combine them into solution
Figure 10: An example of modularized code generation: first, the model is required to outline sub- modules needed, each of which consists of function header and doestring describing the intended use. Subsequently, the model implements each module fully in code and integrates them as parts of the complete final solution.
C ADDITIONAL DETAILS ON BENCHMARKS
Table 6 presents the summary of the benchmarks used. Note that APPS, the original benchmark does not include a specific validation split, we randomly selected samples from the original training split and reported validation results this set. We sampled 50 samples in each of
13

---

Published as a conference paper at ICLR 2024
Table 5: List of problem ids create APPS subset. We followed et al. (2023) to test on the same subset of 20 samples. In total, we selected 14/3/3 hntoolccory/neeeee////opttttoooo samples from the original APPS test split.
Problem level Problem IDs
Introductory
4182, 4195, 4281, 4333, 4347, 4426, 4450, 4507, 4704, 4741, 4855, 4873, 4952 2106, 2673, 2923
Intervicw
Competition
3070, 3286, 3754
Table 6: A summary of APPS and CodeContests: f are statistics reported by Li ct al. 2022)
# Public
# Private
Benchmarks
Val
Test
test cases
test cases
APPS (Hendrycks al. 2021
150
5000
1.98
20.99
CodeContests (Li et al.
2022)
117
165
2.00
203.70
the 3 levels of difficulty: Introductory, Interview, and Competition. For reproducibility, included the specific problem ids of this set Table 7
We also reported the average number of public and private test cases in each benchmark. Note that on APPS, as the original benchmark did not officially include public test cases, we extracted example input-output pairs from problem descriptions using rule-based method. We then treated the extracted input-output pairs as public test cases.
D ADDITIONAL EXPERIMENTAL RESULTS
Analysis on chain of self-revisions Figure 11 and 12 show the clear performance gains of CodeChain, using GPT3.5 and WizardCoder-3413 the base models, over 5 rounds of revisions. Specifically on APPS, we found that model performance generally peaks at serf-revision round 4 (over 1.6x/ 2x performance improvement on average on GPT3.5/ WizardCoder). There is slight performance drop in round 5 with GPT3.5 model while the performance is quite stable on WizardCoder. Secondly, we also found that the rates of performance gains are quite different on different levels of problem difficulty. The best improvement from CodeChain is on competition-level problems (with over 2x/ 5x performance gain on GPT3.5/ WizardCoder). This observation indicates that for this type of problem, LLMs can benefit more from leveraging representative modules form of hint to revise the generated programs.
In Figure we reported the relative performance gains of CodeChain over multiple rounds of revisions when applying additional filtering on output samples. We observed that compared to direct generation (i.e. round 0), CodeChain can improve the performance of pass @ by 1.4x filtered outputs, using GPT3.5 as the base model. This observation indicates that CodeChain can complement the line of research works for filtering sub-sampling output code generations (Chen et al. 2023b)
Li et al. 2022 Ni et al. 2023 Shi et al. 2022 by letting the models revise and improve the outputs iteratively.
Analysis by filtering tests and filtering methods. To understand the impacts of using test cases for filtering in CodeChain, we conducted ablation experiments with different types of test cases: abblc/private/synthttie test cases. The synthetic test cases are generated by prompting GPT3.5 with example test cases (see Appendix for the full version of this prompt). On synthetic test cases, we experimented with sampling generation outputs from the largest clusters (similar to Li et al. (2022) or by following some consensus of test outcomes among all generation samples (similar to Chen et al.) (2023b)).
As can be seen in Table g when using synthetic test cases, CodeT sampling approach Chen et al. (2023b) can achieve better performance than other sampling approaches. Both the conventional filtering approach (complete pass) and AlphaCode (sampling from the largest clusters) (Li et al. 2022) led to performance drops, possibly due the noisy filtered outputs from imperfect synthetic tests. CodeT, this data noise is addressed through a more sophisticated grouping and sampling method to select better program candidates. However, all results from synthetic tests not good
14

---

Published as a conference paper at ICLR 2024
Introductory
Interview
Competition
All
Pass@1 (96)
pass@1 improvement
60
2.2
30
1.6
15
1
0
1
3
4
Revision Round
Revision Round
Figure 11: APPS validation results with chain self-revisions: we tested CodeChain+GPT3.5 for self-revision rounds and reported pass @ / results. We also reported the relative performance gains from direct generation (i.e. round 0). Note that the left chart, the dotted lines represent normal direct generation of non-monn mon-modularized solutions.
Introductory A Interview
Competition * All pass@1 improvement
Pass@1 (96)
22.5
15
3
7.5
2
0
2
3
4
5
0
2
5
Revision Round
Revision Round
Figure 12:
APPS validation results with chain of self-revisions: for 5 self-revision rounds and reported pass@1 I results. We
tested
CodeChain+ CodeChain-WizrrCCodee333 also reported the relative performance gains from direct generation (i.c. round 0). Note that the left chart, the dotted lines represent normal direct generation of non- non-modularized solutions.
Introductory
Interview
Competition
All
Filtered pass01 improvement
2
4
5
Revision Round
Figure 13: APPS validation filtered pass @ / results with chain of self-revisions: we report the relative performance gains of CodeChain+GPP3 when filtering output samples by synthetic test cases (similarly to CodeT (Chen et al. 2023b)) on all generation/ revision rounds.
15

---

Published as a conference paper at ICLR 2024
Table 7: List of problem ids to create APPS validation split. In total, we selected 50/50/50
hntooducor//merrre////mptttooooo samples from the original APPS training split.
Problem level
Problem IDs
Introductory
2361, 2362, 2363, 2364, 2365, 2366, 2367, 2368, 2369, 2370, 2371, 2372, 2373, 2374,
2375, 2376, 2377, 2378, 2379, 2380, 2381, 2382, 2383, 2384, 2385, 2386, 2387, 2389, 2390, 2391, 2392, 2393, 2394, 2395, 2396, 2397, 2398, 2400, 2401, 2402, 2403, 2404,
2405, 2406, 2407, 2408, 2409, 2410, 2411, 2412
Interview
0, 2, 4, 6, 7, 8, 10, II, 12, 13, 14, 15,
17, 18, 20, 23, 24, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 44, 46, 47, 48, 49, 50
Competition
2000, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 201O, 2011, 2012, 2013, 2014, 2015,
2016, 2017, 2019, 2020, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2032, 2033, 2034, 2036, 2037, 2038, 2039, 2040, 2041, 2042, 2043, 2045, 2046, 2048,
2049, 2050, 2051, 2052, 2053, 2054, 2055, 2056
Table 8: Ablation results APPS validation split by pass @ 1(%): report the results of CodeChain+GPT3.5 using public/private/syntttiic test cases to filter for generation samples be- fore applying grouping the sub-modules into clusters. For synthetic test cases, we experimented with sampling outputs from the largest cluster (following Li et (2022) or sampling outputs by some consensus of test outcomes (following Chen et al. (023b)).
Filtering Tests
Filter by
Introductory
Interview
Competition
All
33.50
23.70
10.10
22.43
Synthetic
All Passed
33,50 (0.0)
23,70 (0.0)
10.10 (0.0)
22.43 (0.0)
Synthetic
A AlphaCode
33.15 (-0.4)
20.98 (-2.7)
12.00 (+1.9)
22.04 (-0.4)
Synthetie
CodeT
39.40 (45.9)
23.60 (-0.1)
12.27 (42.2)
25.09 (42.7)
Public All Passed
52.00 (+18.5)
38.83 (+15.1)
14.50 (+4.4)
35.11 (+12.7)
Private
All Passed
55.70 (+22.2)
37.60 (+13.9)
18.90 (+8.8)
37.40 (+15.0)
as ones with public test cases available in problem descriptions. This observation indicates that CodeChain is quite sensitive to the correctness filtering and has to rely on high-quality test cases. Finally, as expected, we found that the best performance is obtained by filtering against private test cases which can cover more corner cases and lead to better filtered programs for self-revisions.
Analysis by different embedding models. Table presents the results of CodeChain+GiPT3. when using different embedding methods: StarCoder (Li et al. 2023). CodeTT5+ (Wang et al. 2003), and CodeBert (Feng et al.] 2020). We found that CodeT can outperform CodeBert in all levels of problem difficulty, especially on the introductory-level problems. Among all embedding methods, using StarCoder with CodeChain can achieve the best performance overall. We noted that one major advantage of StarCoder is the ability to encode long-contex: programs, which we found quite frequently in many challenging coding problems in APPS and CodeContests.
Using public tests as a proxy to select optimal revision step. In all our experiments, we selected revised generation samples during test time on the best revision round we found by validation results. However, in practice, we propose to use the public tests as a proxy to gauge model performance throughout the chain of self-revisions. Figure 14 shows that on APPS, performance on public tests is often correlated well with the performance on private test cases. Specifically, the model peaked consistently at revision round in both sets of test cases. Therefore, in practice, even without validation set, we can still apply CodeChain using public test results as a stopping signal to select the best version of revised programs.
Results on Lee(CodeHardGym. Finally, we attempted to apply CodeChain on the recent code generation benchmark LeetCodehardGGym (Shinn et al., 2023). including a set of 39 challenging coding tasks extracted from LeetCode. Figure 15 shows that GPT4+CodeChain can significantly improve the performance of using GPT4 alone. Different from APPS, we observed that the model achieves the best performance revision round 3, achieving close to 7% pass @ 7. Note that our experimental setup here is quite different from the original setup in (Shinn et al. 2023). We could only test 39/40 original test problems due to erroneous test data, Also, in (Shinn et al.) 2002),, the public test suite was synthetically created by L.LMs and validated by AST. In our setup, simply used the available public tests extracted
problem
withdrawn from the released benchmark
16

---

Published as a conference paper at ICLR 2024
Table 9: Ablation results on APPS validation split by pass ((%): we report the results of CodeChain+GPP335 when clustering sub-modules by different embedding methods.
Embedding Model
Introductory
Interview
Competition
All
33.50
23.70
10.10
22.43
CodeBert
50.73 (+17.2)
34.80 (+11.1)
13,80 (+3.7)
33.11 (+10.7)
CodeTS+
53.00 (+19.5)
35.37 (+11.7)
13.10 (+3.0)
33.82 (+11.4)
StarCoder
52.00 (+18.5)
38.83 (+15.1)
14.50 (+4.4)
35.11 (+12.7)
from problem descriptions. Nevertheless, we still observed consistent benefits CodeChain in this benchmark, complementing other related results of CodeChain in APPS and CodeContests.
Public teats
Private
GPT4
CodeChain
8.0
82
39.0
79
36.8
6.0
76
34.5
4.0
2.0
73
32.3
70
30.0
3
2 Revision Round
Revision Round
Figure 14: APPS validation results by pass@) 1 (%) on public vs. private test cases
Figure 15: Horrdyyde pass@ results
using GPT4+CodeChain revision rounds.
Direct Gen
CodeChain RO CodeChain R5
CodeChain R3
80
90
60
67.5
40
45
20
22.5
0
0
2
3
4
5
2
5
Modularity Score
Reusability Score
Figure 16: Distribution of output samples (%) by code qualities in the APPS test subset: we obtained the qualitative scores by conducting survey with human judges to evaluate the qualities of generation samples of CodeChain (revision round 0 to 5) and direct generation. Please refer Figure and for copy of the survey guideline and example survey question.
Human evaluation results. We conducted survey with human judges to estimate how generation samples from CodeChain aligns with human preference by two qualities: reusability and modularity, We referred to the set of instruction we used to prompt GPT-4 for automatic evaluation and reused them to create the survey guideline. The human judges are required to follow a Likert scale from 0 to 5 where 5 is the highest score for optimally modularized/ resuable programs. We reused the same subset of samples from GPT-4 evaluation (20 random test tasks from Table so that we can fairly compare with GPT-4 based results. Figure 16 shows the distribution of output samples by
17

---

Published as a conference paper at ICLR 2024
Likert scores in each quality. We observed that compared to the GPT-4 based results (Figure the human evaluation results CodeChain are slightly different with more even distributions across the upper range (score 3 5 in both modularity and reusability). However, still observed consistent improvement when comparing CodeChain against the traditional direct generation approach. Please refer to Figure 17 and T8 for copy of the survey instruction/guidellne to human judges and example survey question.
Final Evaluation for CodeChain
Judging Modularity:
A code is modular if it is decomposed into logical atomic sub-modules each handling different sub-tasks of
the given task. Sub-modules are functions that invoked from the 'main' method.
Based your judgement give rating Likert Scale from [0-5] to the given solution.
Modularity Ratings:
Rating Optimally Modular If the candidate solution is optimally modular
Rating to 4: Less Modular If the solution could have been more modular or the
sub-modules are not
atomic enough
Rating O: Not Modular If the solution is flat block of code without any decomposition into sub-modules
Judging Re-usability:
sub-module is re-usable if is generic enough that its implementation can be re-used as- for other
problems for this domain. If they have any specific characteristic part of the implementation that only particular this given task and less likely appear in other tasks, then the sub- module
is not re-usable. Only the candidate solution is at least somewhat modular and there exists at least one sub-module, for each sub-module give a rating on Likert Scale from [0-5] its re-usability based on the
above
judgement. If you rate it less than 5, provide the justification.
Re-usability Ratings:
Rating the function generic and reusable other problems
Rating to 4: the function somewhat generic and might be reusable only in problems quite similar to the given task
Rating the function is not at all generic reusable other problems If the candidate solution is not modular such reusability rating is not needed.
Last, note that 'main' function not sub-module.
Figure 17: Survey guideline for human evaluation: adapted the instructions used for GPT4- based evaluation to create a survey guideline for our human evaluation experiments. For each generation sample (either from CodeChain or direct generation baseline), human judges are required to give rate it from 0 to 5 on 2 specific code qualities: modularity and reusability.
EXAMPLE GENERATION SAMPLES
For the input problem, presented different generation samples generated by GPT4 and CodeChain+GPT4. In each generation sample, we appended the evaluation generated by GPT4 regarding the modularity and reusability of the output program. Through automatic evaluation
18

---

Published as a conference paper at ICLR 2024
def read read_input():
x1, X2, y2 map(inn,, snput().split())
x1, yl, X2, y2
def calculate_squaa square_vertices(11, y1, x2, y2):
(y2 y1)
y3 y2 (X2 x1)
X4 x1 [y2 yl)
y4 yl
10
return X3, y3, X4, y4
11
def print_ outtt_ou y3, y4):
13
print(x3, y3, x4, y4)
14
15
def solve jolve_square():
16
x1, y1, x2, y2 read input()
17
x3, y3, x4, y4 = ca calculate square square_ (x1,, y1, x2, y2) orint_output(x3, y3,
18
19
20 solve_square()
3
5
Modularity
O
O
O
O
0
O
Re-usability
O
O
O
Figure 18: An example survey question with generation sample and the choices a human judge can choose rate the code (0 5) by its modularity and reusability qualities.
metrics, we found that the generation samples gradually became more modularized following the chain of revisions. In general, the generation samples also more reusable compared to the normal direct generation approach (without CodeChain). Through manual inspection, we observed that CodeChain can indeed generate better programs, properly modularized and well explained with function docstrings.
E.1 EXAMPLE OF INTRODUCTORY-LEVES PROBLEM AND CODECHAIN GENERATION SAMPLES
Listing I: An example of problem description from APPS (problem ID 4195)
Today, the nemorable AcDoder Beginner Contest takes place. On this occasion, Takahashi would Like to give
an intzger to Ringo.
conceat is Abcodax Beginnoz Contest 160, Ringo ba happy given intoger that can be divided by 190 exactly D times. Find the B-th emallest integer that weuld. make Riago happy.
nana
O,
an integor betveen and 100 (inclusive).
Input-
Input is given from Standard Iapuz in the following fornat: D M
-utp-ttt---
Print the N-th emcllest intoger that can be divided by 100 exactly tines.
se---Sappl Iaput-rees
Samplo Output
The Incegers that can be divided by 100 exactly tines (that 1s, not divicihie by 190) aze as sollows: 1, 2, 3, 5, 7,
Thas, the
intogar that Ringo happy
Listing 2: An example of output generated by direct generation with GPT4
hpython
19

---

Published as a conference paper at ICLR 2024
(int, inpac cap1sc (pap1c
12 100:
X -- (100x>0
This problam zan be salved by simple math. Python, ** is used to caiculate the power number.
Therefore, 100 the power of is the fastor by which we multiply n. Bevevex, 1f La 100, we count the 100th numbor beczuse itsoir is excluded Linea.
be divided by 103 exastly
wvaluat the above outgut sodulacity and ceusability Modulazity
Ra-usability
The bode is not modular, Bo, ow lab rating not applicalle.
Listing 3: An example of output generated by GPT4 with CoT prompting (revision round of CodeChain)
"nnndole def (ttn_omnleeet__nteee000) R):
the snallest integer can
exactly
b (iol): posilive intager or 2-
(int): positive intecar
and
Oucpul:
... result: the N-th smallest incegez that can be divided by 100 exactly times.
STEP 2: GENERATE PYTRCN CODR
*. "python Nti.amallet__nnoeeeeee
if x -- 100:
N t= 1
(200
oeluen result
meptint, inpat (1 sptit mall_
This code warks by mull iglliing 100 ea raised Lo the powez of D Limea with input number N. IC is 100, Lnurrowet aince needs be the contender for 100rs divisible number.
H4 OpenAt eval ustion of the shove output nodularity and reuseaility Modularity
#48 Bhe uxebill liy nch_aanlll integees Juali The Zunot apecific the hand and genes enough to reused ir other problemt len. While the pei naig' mill plyying by certain power number cou potent fally be applied in other problens, the hardeodedd vel uea and the apecific edge case nakes it. leas likely to directly reusable.
Listing 4: An example of output generated by GPT4 with CodeChain self-revision (revision round 3)
SUB KODULE:
'module
input
bescription: This lon geia Inguta Tron Uhe Jome/sldan
Input: None
Oueput:
fini): Divisibllity with
x [inc]: The NEb texm needed
'moduls
del muniee (D,
Description: Celmulate Rth happy number acvording to the viven divisibility rules Inpute
Hint)s itvisibllity with 100 factor
(int)s The Nth needed
Ouiput:
inn the Y-th ans lesst integer that can be divided by 100 exactly B. times.
20

---

Published as a conference paper at ICLR 2024
"nodole
def peint_aucyet (cumber):
beseristion: function
Input:
nuncor lint): the coerespancing number,
Oucpuc:
None
STEP GENBRATE PYTRON
"python
def inp_l O1:
D, X
())
swourn 1D,
def hapoy_nunbez (D, R): 100:
securn 100+.3
def (runber): peint (nurbez]
get repot
autler happy__ nambes (D,
(nusber))
*48 OgenAT eval ust fon of the shove output by nodularity and reusebllig lity 636 Modulasily
$26 Bu usability
inguts (Jest f Fieation fons This Functioo is sonewtaa, generia and night be used in other plaaces that of space separated integer inputa. fixed hance vesy rerusable.
require
the number inpats the type
heppy numbert 2 (Josti Fieet This Funct ion is lees generio and it is specific to sal ve the numbey_ problen due it's speci Fie logic that won't encountered in most other problem ecenarios.1
prine outputs (Just fons This funct ion generren i you can print any runber with it, but it Limited to incegers only, if we have other date types ve could nos be abls to atlllaw this Funation appropriately..)
Listing 5: An example of output generated by GPT4 with CodeChain self-revision (revision round 5)
STEP
sUa-80DULes:
def ser_ingut(11 (1:
..*
Description: function geua Inputs Sxon sowi/aldin.
Input: Hone
Output:
(inl) Divisibllity with 100 Castos
[inc]: tezm. needed
def print_outgut (runbes);
Descrigtions This lonctioo peintx the
Input:
nanber (int): the ourrespand number.
Oui:ut:
Nona
print [nurbsez]
"podule
def find_nnh_iinteee
nee
Descrivtion: function compuces analleat integoz that by exactly tines.
Input:
D. (ink): Divisibllity with 100 factos
(ine)s
Ourput:
nunbee rink): the mumber that would nake Bingo happy ne*
STEP 2: GEURBATT: PYTHIO CODR
"pythoo
peet..nputt. (1:
21

---

Published as a conference paper at ICLR 2024
Dy naptiat, Znput (1 securn 1D, N)
def
(runber):
Peint (nunbez)
def sind_nthh ID, NI:
x =n 190:
securn 100 or
gui_insut aunber nind intoqur (D, print_ csippto (nunber)
4.5 OpenAI evaluation fon of the above outgut by nodulacity and reusability
300ND 5
Modulacity
44# Tee uxability 5
prin:_output:
nundin lesl intgee 1. Jus cinationn This Fomition is vway sgecific to the poubleen at hand and needa to neet vary apacific conditions, thus its reusability la linited. Hovever, it could be caed in sinilas problens glight Ficstiioo
E.2 EXAMPLE OF COMPETITION-LEVER PROBLEM AND CODECHAIN GENERATION SAMPLES
Listing 6: An example of problem description from APPS (problem ID 3286)
Your friend Edvin loves lenquages. personal isvocrite you not tamilsan with it, it language transzormation sheze every consonant doubled, put in-between, The rest is kept the same. For exerple,, the word "ear" would be transtormed into "cocaroe".
spezking Robber Language.
you how avesome it is. Edvin is sare that nobody nut him and you speas the Rebber Lenguage, so he sacrypted. it using this simple technique, and wrote it down on note, Acter trying the password and realicing it wozk, you found many ways password night have
was going to you paseword
favounite
Edvin drunk darine transformation.
find
originally.
You are given an encrypted password which Edvin has attereted to traaslate into the Robber languago. Your dssernine how wrong, wae when scanning through transforn some consonants asiac the zuis above. Edvin is not detarministie in his faslures, A.a. if he niszed to tranctoun consonant occe, it not necessary that to would mise it i2 it accured again lator the atring.
produce
You assuns that wish eyes, sometines
only thing
A voxel is considered to be one of the letters "a", te", *1" "o" and "a", A consonans is any other letter.
Input concists et tne encrypted pazswerd, resuiting aa 610000005 characters, It will selely consist of lowez case letrera a-2,
Eavins tranciation. wozd can long
Octput should consist of one nunber, number ways password cronsisted it. Since this number may be haze, print the remainder modelo 519003095.
-xampposs---.
Sample Input
caz
Bampla Output 1:
Bample Input 2:
opows
Bample ducput
Listing 7: An example of output generated by direct generation with GPT4
"pytton
ingut (3
lwnisa
rin, *ot, *ofl)
dp
[0] (mil)
nod
10xx6+9
dp10]
dpIt]
for range n 1)
aii-21 vew sti-2]
s(i-11 e15-2) sti-1) rots
dpfi] dpfi-1]
22

---

Published as a conference paper at ICLR 2024
dptil dp[1-1] + 4p[1-2] dpti] t- ncd
print (dp [n]]
OpwnAI
absow outgut by nodulacity ceusabllity
138 Modulazity
Ra-usabiliey
The solution 10 not meduloz, henze re-usability roting is not applicable.
Listing 8: An example of output generated by GPT4 with CoT prompting (revision round of CodeChain)
STEP 1: GENERATE SU3-POPULES:
"noddll
wowel(th)
beseription: This function zecurns Toue if Uhe specified Cheracles is vowel, Palse otherwine.
Outputr
[booll: Trpe it the character is vovel, Falss othervise. .***
def count pwwstble.ee...oooooo..oooo original, words horr))
Description: tunction counts possible original worde that could have into the Language
Input:
word (str): Robber Language Nors.
count (intl: count pascible criginal worde.
STEP 2: GENERATE PYTВCH CoD2
""python
det 1s (sb)
['a", ref, for, fu'l
seturn
def wou.t_oooraa.....orrcaaaeeeeeee.
leniword)
de (M+1)
[p[0]
for in cange(1, 8+11:
dp[i-1]
not is_vowsl [word[1-1]] and Nord[a-1] vord[1-2] word]a-3] tors
op11] (p(1-2)
dp[-] to MOD
word input st.strip
print (count ctwpstt_pcccbbl..ottppooeeeeeeee (vord)
This Bython functicn
follcws:
It counts the number of ways xe con fern the Robber lznguags word by taking a synanis programning approact. The 'dp" last is initialized with zezoes with 'ds;0]" set to "1', Then for each character in the word, bafore
precsdine character la not. fu', If theae conditions are mw, counts as possable place vhere consonani sould have been akippad and adds 'dati-2] to 'dpill', Pinally, it returns "dp[l]" whtch contains the tosal runber of possilie original words. The modelo operaticn is used to keep the result from ovestlowing the bit intager
it
consonact
616 OpwnAI wvaluat Zon of Lhe abovw outgut by nadulacity and ruusability Modulasity
Ratiing:
#f8 Re-usability
usability
Listing 9: An example of output generated by GPT4 with CodeChain self-revision (revision round 3)
sua KODULeS:
Since te given two relevent funct ions "split_word and "oount pooeet__osssill www.pppw.ee....mm hibbluppeee www.oooorr keep theaw tonctions solution. The "instialize wew.iiiieieeeeeeeeeeeeeeee indicoo can gl global verrabl es.
23

---

Published as a conference paper at ICLR 2024
"nnodoe
det h
n*e
Description; fuaction splito
se eiements that can transiated (atz)s word needs spiit.
Rebber langusge.
Input:
Oulpul:
apritaardd [list]: The 1ist ot atrings, whare each acring zepresenta possible original letzer
"nnodle
def initial iec. wow..
nee
Description: fonction inlsss the vouela nodulo varisbles.
Oucpuc:
vovals (sat) A set of vowels sodulo The nodula
*. "nodule
tat___oooolloooooeeeeee (iatd_
bescription: This function counta the numbez of posalble password that night have produced the original
Ispul:
Hird [list): 1ist with elemants from the word
Ourput:
(intls The nanber of possible passwuuda modula 1000009
1* "pytthon
def epl word (
split_aord_liat
whi le < Ten [wrrd]: bendwrrd
if 3. len (ward) and wordfi fia and word fofe
split list.apee (1:14313
word aitt__iitt.geeedd..e (
DeLuEn werd
def bount possibll pesswords (word_ Hist, vovela, module)s
og [ono
(p(0) 1 Len bword 1set [0]]
or word Liet in woveln else
for in renge(), (), Ten [ 1ss Jen [ioss____ist
[1t in vovele:
op(1] dpli
elaez
api1] = idpli 11 8 moduld dpl-1]
det initialize wwwelooooelleeoooooo and [ return setil'af, e", fe', 's', fo', (uf]), 1003009
soive_tetee wwwp_ttckkkkoooooo :xorsi:
itlit word List split (verd))
vovele, madulo initialire wtte:aiite_oooclleeooooeo voxcl and__
(eplit_worr_llee
nodulo)
word input ()
print (solve__teee back (werd)
...
Ir the tinal sode, runction [= uses "split word()" to break down the word inte possible original letters, wetzissize_vowel___ao. and__iddd to act the voxel set and nodulo values, Finally it uses original
coun__oosttie_pssoooree/ crieuiste
#t OpenAI evaluation of the above output by modularity and reusanility
Modularity
#18 Ra-usability
spli-_word:
initialize, pegei__and_nosulac Justisication: Functions setting constants re-used, but function nana and content are vary specidis to the task. 2. Justirication: It specific to current problen, i: is specitic to "Robbez Language! problan.
solve_task: custifications function ovarall problan specific not se-usable other problens.
24

---

Published as a conference paper at ICLR 2024
Listing 10: An example of oulput generated by GPT4 with CodeChain self-revision (revision round 5)
arEP GENERATE SU3 :US-EO:ULE::
"nnodol
dct 1s_yoNsi(zh)
Description: function checks whether charactez vevel.
Inpuk:
ch (stz): A single chazacter atring.
Output:
(bool): "Izue" if the chesacles
Valls" otherviae.
.. "nodule
spl hordd)
Description: This function splits the word into 11st of elements that can ba translated into the originel
Input:
word (stx): The encrypted passwore.
Ou:pul:
List accings, aprit_aord_ist
zepresenta possible original letcer.
"nodole
count pount_.pwwibiii www.ppibbiiillleeeee..... Lot.
bescri Fonz This functioo countx the number of mach element in the Hast could translated into original letlers.
Input:
apl aundd list [1ittl: liss of strings, where each string is an element thas could possibly be Leanalated inco usiginal letlez.
Ourput:
passibill biilities (int) The total mumber says the password night have looked originally.
STEP
"python
def vo_eo (ch)
'aeiou'
def yglit_anrddoorde twrd)]
epl ielit,son 1ist
whill (word]
it + 2. < len tords and not is vowel (word(i) and word 1] fot and word[i] -- wordid + 211
split_ lat. aggend (oord (i 1513
else:
split word, Hist (woddd
split
def possibilitiesssppll.. word letli possibilitics modulo 1003009
for element in split word list:
icniciement
(e_voncl
pocsibilities
possibi lities module
seturn pesciatiities
word input
split word split word (xord: possibilities coun__oastbiliiiee (ept print (psceppbabllllttieee
This solution nakes use of the "split_wozd' and "is_vowol" solovant functions. The pouns_posttbiliieer problen. any
divide the into processes consonent, Then, single character and is consorant, it means it could either have been an original consonant that Esvin sidn't process consonart that Esvin processed correctly.
and specitic
originsl aseuning
#+ OpentI ovalustion of the above output by modularity and reusantlity #: Modulasity
# Re-usabilitt
cs_consonent: 5, Justitication: function to caess if given character is consonant or not is generic manigulacion
wide variety
function involving genoric than 'is_conzonant" because spacitically
prosessing,
splis_wozd: 3, custifications designed to split string in the special xey that is relevant for the Rebber Lenguage, Boviver, it night be useful in sinilax tasks involving spacially fomatted string decoding.
25

---

Published as a conference paper at ICLR 2024
solve_task: 2, suatifiiations This function vazy apecific curzent task unlikely reused in other pzoblens as it implements the Leqie for deriving possibls original passwords fron the encrypted pasoward. in Babber lznguage.
F PROMPTS WITH INSTRUCTION
Listing 11: CoT prompting with instruction to generate modularized code. <question guide> is replaced with instructions for the model to follow either standard input streams or call-based functions.
-Inatruction
Develop vell-structured Python aclution
constrainca and the exampie cases. Ensure nedelarity considering potential adga cascs railures. Start by outlining the sequired code madules, including function headeza and aignatures. Subsequently, proseed to implement each module to create the final coda.
provided prebiem
aimplex Lezna, create
organized Fychoa solucion the given problen. Break down iato smaller parts imoduiasi with clear function nanes and input/outpu: speciiications, Cace the structore la seady, wrile the actuel unde for each nodule to cumplete the sulu.ion. code needs esquaation iondidr.. Please ssap your code using
Exampla
#26 TASK:
<sexample preillnn>>
RESPONSS:
STEP GENERATE suB KODULES:
ccexample genarated sub-nodalasp>
STEP 2: GENERA"F PYTICH cODR
oSexample generaled uode?>
#48 Exangle
$36 TASK:
*** ARSPONSR;
Listing 12: self-revision prompting with instruction revise and generate modularized code. eques- tion gondguddd> is replaced with instructions for the model to follow either standard input streams or call-based functions. <sub modules> replaced with representative sub-modules selected by CodeChain framework.
Develop aoll-structured Python aclution for provided problem that obeya conatrainca passes exemple test cases. Ensure mode larilyrity and cony dering potentis) edge raxes and failures, viven set of related utillsy Python Junot Lay Lo reuse adapt then much as possible into your salution eutlining recuized code modiles, Including functtion headers and signatures, Subseguent yy, proceed to implement each module to create the uode.
tcreate unique functions aeaded).
simpler terne,
organized Pyshon solut for given problen. stalles parta (nndulest with cleas function names and suput/ouipe gieatfiaaionn. Gate atzucture is ready, wrice the actual code for each module to complet ete the solusion,
code nweda esqueat coqpaati guidoss Please usap your
using
** Exangie
**6 TASK:
<sexample proislen>
noduleat0
#** RESPONSE:
STEP (GFURRATP: cUn-NNDUE::::
osexample generaled
STEP 2: GEVERATE PYTRON co0g
ccexample generated ed codedo
Example
*4* TASK:
<<new problemss
*** RELEVANT FUNCTTONS:
*** RESPONSE:
Listing 13: prompt to generate synthetic test cases to use as additional public tests. <example tests is replaced with any available test cases extracted in the problem description. We expect the model to follow similar formats to the example test cases and continue to generate up to 20 test cases in total.
$26 Quwstions
Generate 20
ingub and potpot paire exenple
cases for the fell following problen:
esproblenc>
71easw reaponse followw lowing this Fornats
Test 15
26

---

Published as a conference paper at ICLR 2024
Input: <input>
soxpecced outger>
#f teat 2:
#F Answer:
<cexampi__eett
Listing 14: prompt evaluate code generation samples by their modularity and reusability qualities.
You arw an expeet sofivare enganeed and you always eaphasize on welt ing noduler reusable codles, any given task desoription.
You
teak descrigt fan and candidate solation in Form of python code for it, You have jodge code:
following aspeava
+4 budging Mocularrity:
cose mocular it decomposed into logical atomic sub-modules handling different
of
given tass, Sub-modules runctions are invoked lased your fudigement give racing on tikert Scale from [0-5] to the given ton.
'main'
Modularity Ratings:
Ret-ing Optinelly Modular
candidate solution optinelly Rating to 4: lese Hodular It the sclution could have been more nedeler or it the sub-nodeles are not stonic enough
Rating Not Nodular
the sclution
block
without decompsittion into sub-nediles
*+# budging Re-usability:
5. cub-nodule is re-usable it it is generic cnouch that its implementation can be re-used as-is for other problens for this domain. they is only partiauler so given is not ro-arsblo.
specicic characteristie likely to appear
the implementation taske,
only it the cansidate solstion is at least sonewhat modular and there existe at least one suo-module,, for rating likert Seale from [0-5] its re-usabil lty based on the above
each sub-modu :udgemcnt. If you rate
then provice the justitication.
3o-usabillty Retings:
Rating 5: cunction generic and rousable
problens
Rating
generic might be reusaale only in problens quite the
civen task
Rating 01 the sunction is not all generic or reusable in other prebieme
nedular rousability rating is aot A150, 'main' sunction is not sub-mocuic.
candidate sclution
Your should
medularity and re-usebility ratings. Judging correctness cansidate solution oz impleneating correct solution for the tass
Your overall response should follox the format:
++ Nodularity
sacing
#f# he-usebility
<sub-modulel>: <rating:0-5 sating, Justificatioa: Juscification if nat-reusable. <sub-moduic2ht <zating ...>
saub-noduleB>> Seating ...
#!8 TASK:
CANDIDATE
exCODE>>
#!# RESPONSS:
G ONE-SHOT EXAMPLES
Listing 15: one-shot example input for normal code generation prompting or prompting to generate modularized solutions
*4 Exergle
Polycarp has 9n8 di iffrent bi hary words. word cal led binary contains only characters for and (1r, For exemple, these words binerys *0301*, *12M, age 10011100*.
Polycarp want's to offer his sen of $ně hinary words to play gene fvords", In this game, players name words and math next word (stacting from the second) start with the lest charact of the previous surd. cirst
any, For example, these sequence of words car named auring the ganes "010.", "10*, *00*, "0000!".
27

---

Published as a conference paper at ICLR 2024
Word reversal is the operation of zeversing the ordar of the cherseters. For exampie, the word "0111" afcer the reversal becomes "I110", the word "11010" after the reversal becomes *01011",
Probably, Polycarp has
act ef words game ruics. In this situction, he wants to reverse some words cron his oct so that: the tinal set ot 8nš wozds still concaina differeat vords (1.e. all words are unique); cheze is to put all worda of the cinal
tacre
way to pat than tha order corsesponden: to the
words the order that the rinal sequence
words coasistent with the
game rules.
Polycarp wants to zeverse minimal number of words. Please, help hin.
Input
The firat 1ina of the input
integer SE5 \io via 10"45) Ingut. Then best dasea follow.
the
The first line of a test case contains one interar $a8 (81 Vie n. Vie 21edec10-59) -- the number of words In wards eaplly characters and The san word lengtha dowan't exceed $4kodot10"68, All words aze different.
Polyuarp'
only
Guarantend, that the of Ens for all teat casea in the ingut dowan's exceed $2%odo 10"55. Also, guaranteed that
lengths for
dossn't exceed 54kedoc19*66.
-..-- -ot---Cuputt---
Psint answer all of BLS Leat casea the ardez tlay appeas.
If there is answer for the teat
print -1, Otharkize, the first Line of the should contain ske the mininal cander
180 Vie "le of the outpet should contain S&S distinct intequra the indaxes be reversed. Nordl are nnnerated from $15 to $n8 in the order they appeans If Sk-0s you can skip this Line [or you print eaply line). IE theck
in the which should vorda
aed. The secund
set which should
anavers you print of ther.
Esample 14
Irgut
0001
1000
01.0
00,00
301
00001
Output
Listing 16: one-shot example input for code revision
Exarple IASK:
Polyuarp has $u8 different binasy words, wurd cal hinasy if it cuntaios only charactees for and rir, For example, these worda are binaey: 10001*, *1s", *g* and 10011100*.
Polycarp want Lo offer his wel of Su8 binaey words to play gane faurda", and each next word tatasting from the second) nust start with the last character of the previous word. The first word can be Por exanple, these sequence of words be named during the "ol0"", 11*, *10*, *90*, "00001".
game playees nane
Word reversel la the operation ion of reversing the order of the cheranters. For examp le the word PO111# after the ceomssal becuses #1110", the ward 111010 aftes the ceversal bequnex *01011".
Probebly, Palyyarp has such set of words that there no way to put then the order correspondent to the gans sules, In this sicast he vanta to zeversu some words from his set so that: the final ael of ans words still contains di sifferent words liee all worda uni final sequence
there is words
all words with the
of finel game zules.
wards
Polycarp vants to reverse mininal nunber of words. Please, help hin.
28

---

Published as a conference paper at ICLR 2024
Input
The first line et the input contains cne integer 9E9 191 Alc Vie 101491 the nunber cest cases in the follow.
Than
The first 1ine of a test ease contains one integer 9a8 191 Vic n Vie 21cdet10ns9) the number of words in the Polycarp'a set. Next 8as lines concain these verda. Al1 of Ens worda azentt empty and contains only ienoths docen't exceed 64xcdot10*6e. A11 words differant.
charactors 10'
Gcaranceed, that the sun of 8n5 for all test cases in the input doesn't exceed 92hedo:10 Also, ouarantaed that the zun of werd lenoths for all test sases in the inpet socsa't excced 54'esot10"65.
-...--utptt-....................
Print
of 926
appeas.
If chere is no answer for the Less case, print -1, Ocharkise, the fiest line of the output should contaia 8&5 190 Vio & via c51 -- the ninimal cunber of vords in the set which should be reversed. The sacond line of outgut be seversed. Horda ase numerated inct integers line for you can peint an anpty 1ine). I2 there are nany answers you can print any of then.
indeses
which should they appeaz. If 9k-28 you akip
508 the
Examplo-
Tepul.
3001
1000
00111
31.0
1C1
3C00C
00001
0C1
3001
Output
0
*+* RELEVANT FURCTIONS:
des weveroe_Nords istring):
description: Function roVCERES caca word given string.
input:
string istr):: the input string.
output:
severeed_string (stx)'s reversed string each
seturn ".Join(e[:1-11 for in string.eplit(" '))
"nnddee
det saant__stat__o__come (Nords)
new
Description:
counts number that start with charactor.
Input:
words (list): A 1ist of binary words.
Output:
stact_
dicsionary contairing end_count idefauitdict): A dictionary containira the count of vords that end with each charactor.
charecter.
collections dewauitdiitt (iat)
end_count collecioon.ettuudditt
for verd in worde: start_ocunt [word[0]] +s
end_esunt (wodd]]]] +-
start_ocunt, end_count
Listing 17: one-shot example output for normal code generation
#4 ARSPONSR:
hoython
29

---

Published as a conference paper at ICLR 2024
zange tiat (input (1)1
int (input 1)7
maad
22
6223 set.1)
2005 sct.)
Input
mass.appendtee (K)
1f k[O] =- for and X[-1] -- '1 addd
app.apdedd
20 to
elif K(O] 'I' and K1-13 an 1o12
append (1
oz +=
wlif k[0] for and K r01
oo 1- 1
it zu and oc and not and not 201 (-13
continus
elee:
if
print
//
and
need
Esc oxt // 2
1
need:
2Xx nass [sos [i] Hen (xos [3] 11) Dex
if asa not in uzsat
need
print nt Ixans]
wlaw:
print 201 // and
need
lox
not // 2
sox mead [aze [i]
(nass 11) 2ex
it 2na not in 20581 aps.ap toxe
nood
print Txans]
Listing 18: one-shot example output for modularized code generation/revision
sU3-KODULES:
"moddle
deL count_ stade_ (worda)
nee
Description:
counts number that start with each charactor.
Input:
words (11st): A 11st of binary worda.
Ouiput:
stant_count (dataulidiet) diecionazy contairiag count of words acart with each character. end_count idefauitdiet): & dictionary containira the count of vorda that end with each character.
'moduls
def ceversa_vorda tworda, indicee]_idiics]:
n..
Deseription: function reverues specitied Cied worda the given list.
Input:
wards (1ist) A List of binary worde.
(1iat)s list indicws indicat words zwcecaed.
Oucput:
reversed uords (list) A new liss of words wish the speci fied words reversed.
STEP 2: GENERATE PYTECN COD2 < "python
isgort cullectioons
def count start and chers (words):
stast_count: collect oos dle deefallldd diet (int) end_esent collections.
cint)
word words:
30

---

Published as a conference paper at ICLR 2024
stazt_ [wozd[0;]
end_couat [word[-1]] +o
return start_count, end_count
NordEsa_NOEds fwords, Ioverscs_indicaa::
sovcrsed_xore II] for i, vord in (aumerate(xorda))
se:
www.
else: wwe.rcodierde..apadddooo
def cesv__tsskhoraal
stact_ cnd_c coun._stari sinn (worda)
cheracters di
for chat in atart_
if lastarart_count [char] enf_ccunt [char]] 1:
iverrsd_indiics
Tar char in wi
diffesence
[chas] end_count [chax]]
revorse_count diffezence
if ama [tihes) oui- [shar]:
Indices [1 for i, wozd enunezate (words) if wozd, schat
))
indices [1 for i, sord in enunerate (words) it aozd,andwwith (chaz)) indices exteee (indl [ev[:seevere_ccon
soversed_ noverse_norda (worda, inversed_idiion
cwveraed len (revecaddd
imi (inpul (1)
itl:
iinptt
words
for in (a1g wprdk (1)
total_ceversed, reversed__eee
solve_tack
print (total reversed)
if reveraed I- print (vxeveroed__eodd))
31


Reviews:
[{"review_id": "RGm0ocLeUJ", "reviewer": "Reviewer_3rJJ", "summary": "This paper proposes the CodeChain approach for code generation in LLM (Large Language Model) sub-modules, aiming to enhance the modularity and accuracy of the resulting code.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured and presents its arguments in a clear fashion. \n\nThe prompt method described is uncomplicated yet efficacious. \n\nThe experimental evaluation encompasses both closed-source and open-source models, providing a comprehensive analysis.", "weaknesses": "It is unclear whether the method can improve the quality of code generation specifically for Codellama models.\n\nThe potential of CodeChain to bolster problem-solving capabilities in other domains, such as mathematics, is not established.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8WSTKY7VT3", "reviewer": "Reviewer_sbrc", "summary": "The paper proposes CodeChain a method for prompting LLMs to generate modular code and reuse the generated submodules in subsequent iterations of prompting. The generated submodules are extracted and clustered to find representative and reusable components for iterations of self-revision. Experimental results on the APPS and CodeContests demonstrates that CodeChain significantly improves pass@1 metric when compared to several prior approaches.", "questions": "* The authors observe that the training datasets do not filter for modularity. Have the authors tried filtering the training dataset for modular code and do light-weight fine-tuning with parameter efficient tuning methods?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The CodeChain prompting technique proposed in the paper is intuitive and easy to utilize. Designing a prompting strategy for inducing and leverage modular functions seems novel. \n* The experimental results on the selected benchmarks show the effectiveness of the prompting strategy relative to several prior methods.\n* Studies on the impact of different clustering strategies, embedding choices and revision sampling are informative.\n* Overall CodeChain seems like a good prompting strategy which is simple and effective.", "weaknesses": "* The clustering strategy seems to add only a small performance improvement to the overall approach. Randomly picking the modules also seems to do reasonably well (Table 3). One experiment that would be helpful is adding all submodules instead of randomly picking the generated modules to include in future revisions. Would this eliminate the need for clustering. \n\n* It is unclear how much the prompting strategy is sensitive to specific wording of the prompt and alternative formulations. Did the authors try multiple variants of the prompt and if so what was the variance and sensitivity of the results. Is it possible that there are prompts which could instruct the model to do revisions in a single shot? Could this improve direct generation? \n\n * The evaluation largely relies on two benchmarks. It would be good to see the evaluation extend to some of the appropriate subsets in https://github.com/bigcode-project/bigcode-evaluation-harness", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fA9YanDVuj", "reviewer": "Reviewer_hL3y", "summary": "This paper proposes a modular code generation approach for complex programming tasks. CodeChain extracts and clusters generated sub-modules, selects representative implementations and instructs the LLM to generate new solutions using these selected modules. Experimental results show that CodeChain significantly improves modularity and correctness, achieving relative pass@1 improvements of 35% on APPS and 76% on CodeContests. The framework works well with both OpenAI LLMs and open-sourced LLMs like WizardCoder. The paper also includes comprehensive ablation studies that provide insights into CodeChain's performance.", "questions": "In section 3.2, the author mentioned, \"We append the instruction with a one-shot demonstration.\", which part of the referenced figure (Figure 3) or the appendix (Appendix F) is the one-shot demonstration? How is this demonstration being constructed?\n\n\n\n=== Post-rebuttal ====\nThanks for the detailed clarification and the additional evaluation. While I still have concerns regarding the novelty and the proper evaluation with GPT-4 (in terms of human evaluation), I think this paper is a good contribution to the community. I increased my score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* SoTA performance on code generation benchmarks\n* CodeChain works well across LLMs (GPT-3.5, GPT-4 and WizardCoder)\n* The paper is well-written and is easy to understand.\n* The authors provided extensive ablation.", "weaknesses": "* Correctness/Soundness: \n(1) Programs generated by CodeChain are with high levels of modality and reusability on Likert scale judging by GPT-4 prompt. It is unclear how this evaluation align with human preference.\n(2) The effectiveness of sub-module generation is unclear. \n(3) The analysis on the chain of self-revisions sees a slight performance drop in the 5th iteration, which hints at the limitation of self-revise prompting.\n\n* Novelty/Originality: \n1. Using CoT to generate demonstrations and choosing representative demonstrations has been explored in [1]. The novelty of this work lies in employing this idea for code generation, which is incremental.\n2. Missing citation: [2] explores a similar idea of decomposing source code into components. \n3. The idea of utilizing LLMs' ability to self-revise has been studied in [3,4] and more. \n\n* Writing could be improved: There are multiple references to an Appendix section (e.g., Appendix F) without clarifying which figure/prompt is being referred to.  \n\nReferences\n[1] Automatic Chain of Thought Prompting in Large Language Models, ICLR 2023\n[2] Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation, ICLM 2023\n[3] Large Language Models Can Self-Improve, ICLR 2023\n[4] CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W8PZQcCjvr", "reviewer": "Reviewer_dD56", "summary": "This paper proposes a new inference framework for modularised code generation using a large language model. Existing LLM tends to generate monolithic code blocks while a complex coding task usually needs to be broken down into multiple sub-tasks with each addressed by a sub-module. Motivated by the modular programming discipline, this paper proposes to iteratively ask LLM to generate code solutions with sub-modules and add the potentially valid sub-modules to the prompts for LLM’s generation at the next round. The sub-module selection is conducted by test case filtering and clustering.  The proposed framework yielded nontrivial improvements over the direct generation baselines.", "questions": "In table 2, why is the result of GPT4 (All: 34.75) is even bettrer than Self-repair+GPT4 with GPT4 as the feedback source (33.30)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In general, this paper follows the self-revision practice to improve the quality of code generated by LLM in a trial-and-error manner. However, instead of solely focusing on the functional correctness of the code, this paper raises the concern about LLM’s ability to decompose a complex coding task into multiple sub-tasks so as to generate reliable and reusable sub-modules. Although both task decomposition and chain-of-thoughts have been widely studied in different NLP applications, they are delicately adapted to fit the modular programming discipline in order to benefit code generation. Empowering LLM to produce modular code is not only helpful in improving the code’s reliability but also important for reducing the cost of subsequent manual maintenance. So this work’s originality and significance are considerable to me.", "weaknesses": "+ Sub-module filtering by public tests: given that the objects to be filtered are sub-modules generated by LLMs but we actually have no idea how the LLMs will decompose the target task/module into what sub-tasks/sub-modules. Then how can we compose the tests for the unknown sub-modules?\n\n+ Four schemes for deciding the number of clusters are investigated. However, different tasks are of different complexity and it is difficult to decompose them into a fixed number of sub-tasks. Then is it possible that LLMs may generate some sub-modules which are never used in the task-level solution? If yes, then what are the effects of those sub-modules?\n\n+ Kmeans are selected for clustering by why? Will density-based clustering like DBSCAN work better given it doesn’t need to specify the number of clusters as a prior?\n\n+ As shown in Figure 5 and Figure 7, I failed to observe a consistent pattern to help me decide the optimal round of self-revision for CodeChain. The authors explained that the performance degradation at the 5th round is because of overfitting to the public test sets but this claim lacks support.\n\n+ The title of the y-axis is missing in charts like Figures, 6, 7", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)