CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Authors: Erik Nijkamp
Bo Pang
Hiroaki Hayashi
Lifu Tu
Huan Wang
Yingbo Zhou
Silvio Savarese
Caiming Xiong
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Program synthesis
multi-turn generation
code generation
large language models
generative models

Abstract: Program synthesis strives to generate a computer program as a solution to a given problem specification, expressed with input-output examples or natural language descriptions. The prevalence of large language models advances the state-of-the-art for program synthesis, though limited training resources and data impede open access to such models. To democratize this, we train and release a family of large language models up to 16.1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER. We show the utility of the trained model by demonstrating that it is competitive with the previous state-of-the-art on zero-shot Python code generation on HumanEval. We further investigate the multi-step paradigm for program synthesis, where a single program is factorized into multiple prompts specifying subproblems. To this end, we construct an open benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse problem sets that are factorized into multi-turn prompts. Our analysis on MTPB shows that the same intent provided to CODEGEN in multi-turn fashion significantly improves program synthesis over that provided as a single turn. We make the training library JAXFORMER and model checkpoints available as open source contribution: https://github.com/salesforce/CodeGen.

Published as a conference paper at ICLR 2023
CODEGEN: AN OPEN LARGE LANGUAGE MODEL FOR CODE WITH MULTI-TURN PROGRAM SYNTHESIS
Erik Nijkamp; Bo Pang, Hiroaki Hayashi,
Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, Caiming Xiong
Salesforce Research
ABSTRACT
Program synthesis strives to generate a computer program as a solution to a given problem specification, expressed with input-outpul examples or natural language descriptions. The prevalence of large language models advances the state-of-the- for program synthesis, though limited training resources and data impede open access to such models. To democratize this, we train and release a family of large language models up to 16. parameters, called CODEGEN, on natural language and programming language data, and open source the training library JA XFORMER. We show the utility of the trained model by demonstrating that is competitive with the previous state-of-ther the- on zero-shot Python code generation on HvmanEval. We further investigate the multi-step paradigm for program synthesis, where a single program is factorized into multiple prompts specifying subproblems. To this end, we construct an open benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse problem sets that are factorized into multi-turn prompts. Our analysis on MTPB shows that the same intent provided to CODEGEN multi- turn fashion significantly improves program synthesis over that provided as single turn. We make the training library JAXFORMER and model checkpoints available open source contribution: https://giithb. http://itii..mmm//sssssssf/eeeee
INTRODUCTION
Creating a program has typically involved a human entering code by hand. The goal of program synthesis is to automate the coding process, and generate a computer program that satisfies the user's specified intent. Some have called it the holy grail of computer science (Manna & Waldinger I97I Gulwani et al. 2017). Successful program synthesis would not only improve the productivity of experienced programmers but also make programming accessible to wider audience.
Two key challenges arise when striving to achieve program synthesis: (1) the intractability of the search space, and (2) the difficulty of properly specifying user intent. To maintain expressive search space, one needs a large search space, which poses challenges in efficient search. Previous work (Joshi et al., 2002: Panchekha et al. 2015; Cheung et al. 2013) leverages domain-specific language to restrict the search space: however, this limits the applicability of synthesized programs. On the contrary, while being widely applicable, general- general-purposs purpose programming languages (e.g.. C, Python) introduce an even larger search space for possible programs. To navigate through the enormous program space, formulate the task as language modeling, learning conditional distribution of the next token given preceding tokens and leverage transformers (Vaswani et al. 2017) and large-scale self-supervised pre-training. This approach has seen success across modalities (Devlin al. 2019)
Lewis et al., 2020 Dosovitskiy et al.) 2021). Likewise, prior works have developed pre- trainedd language models for programming language understanding (Kanade et al.) 2020 Feng al., 2020).
To realize program synthesis successfully, users must employ some means to communicate their intent to the models such as a logical expression (which specifies a logical relation between inputs
Equal contribution.
Correspondence to: Erik Nijkamp (erik (erik mijkamp@@ @aaeesasraeeeeomn Bo Pang (b. (b.pang @ www.e@aaaleleeoeomoooomoo .co).. Hiroaki Hayashi (hiroakihayashiie salessoreosssssssc.cooomm Yingho Zhou kynngoo/houuxaalesfssce..mmm Caiming Xiong (exiong salee.orccc.ommm

---

Published as a conference paper at ICLR 2023
and outputs of a program), pseudo-code, input-output examples, or verbalized specifications in natural language. On the one hand, a complete formal specification enjoys the exact specifications of user intent but may require domain expertise and effort from users to translate the intent to such a form. On the other hand, specification merely based on input-output examples is less costly but may under-specify the intent, leading inaccurate solutions. Previous work has benefited from various methods and their combinations as the input to program synthesis models, including pscudo- code (Kulal et al. 2019), part of program and its documentation (Chen al., 202T), or natural language paragraph with input-ouipu examples (Hendrycks et al. 2021). However, we argue that a truly user-friendly form of intent natural language text.
To overcome these challenges, we propose a multi-turn program synthesis approach, where user communicates with the synthesis system by progressively providing specifications natural language while receiving responses from the system in the form of synthesized subprograms, such that the user together with the system complete the program in multiple steps. The following two considerations motivate this approach.
First, we speculate that factorizing a potentially long and complicated specification into multiple steps would ease the understanding by a model and hence enhance program synthesis. In the multi-turn approach, a model can focus on the specification associated with one subprogram and avoid arduously tracking the complicated dependency among subprograms. This effectively reduces the search space besides the convenience of specifying user intent. Indeed, our speculations are confirmed in our experiments with higher quality synthesized programs through the multi-turn approach.
Second, code exhibits a weak pattern of interleaved natural and programming language, which may be exploitable. Such a pattern is formed by programmers who explain the functionality of a program with comments. With the language modeling objective, we hypothesize that the interleaving pattern provides a supervision signal for the model to generate programs given natural language descriptions over multiple turns. The signal is highly noisy or weak, because only a subset of data would exhibit such a pattem, comments may be inaccurate or uninformative, and some of them may even be placed at an irrelevant position. However, up-scaling the model and data size might overcome such weak supervision, allowing the model to develop multi-turn program synthesis capacity. This enables user intent to be expressed in multiple turns, that is, the intent can be decomposed and fulfilled part by part while each turn can easily be expressed in natural language.
In this work, we develop multi-turn programming benchmark to measure the models' capacity for multi-turn program synthesis. To solve a problem in the benchmark, a model needs to synthesize program in multiple steps with user who specifies the intent in each turn in natural language. Please refer to Figure 1 for an example where the model synthesizes a program to extract the user name of an email address. Performance on the benchmark is measured by pass rate on spprtt-wrttten test cases. To the best of our knowledge, this is the first multi-turn program synthesis benchmark, which allows quantitative analysis of multi-turn program synthesis. With the emergence of multi-turn program synthesis capacity in large language models that benefits problem-solving, we believe this benchmark will foster future research in program synthesis.
Our Contributions Our work shares the basic idca of adopting language models for program synthesis with the recent and concurrent efforts (Chen al. 202T Austin al. 202T Li et al. 2022)
with single-turn user intent specification. In addition, we contribute with respect to four aspects:
We study multi-tum program synthesis emerging in autoregressive models under scaling laws. We leverage this capacity to introduce a multi-turn program synthesis paradigm.
We investigate its properties quantitatively with a novel multi-turn programming benchmark]
We will open source model checkpoint and the custom training library: JAXFORMER
For program synthesis, no large-scale models competitive with Codex are available as open-source. This hinders progress, given that the expensive compute resources required to train these models are only accessible to limited number of institutions. Our open source contribution allows a wide range of researchers to study and advance these models, which may greatly facilitate research progress.
BBnchharkk: http://////ttttii//////eeeeeeeer
"Checkpoints: httpss//ggihuaa..mm//lleefffrfrc
"Training/httts:/// http:////tttth.coo//aaatfttttter
2

---

Published as a conference paper at ICLR 2023
2 MODEL TRAINING
To evaluate the emergence of multi-turn programming capabilities under scaling laws, we adopt stan- dard transformer-based autoregressive language models, varying (1) the number of model parameters (350M, 2.7B, 6.I 16.1B) and (2) the number of tokens of programming languages in the training corpora. For scaling the training, custom library JAXFORMER for TPU-v4 hardware was developed and will be released as open-source, including the trained model weights.
2.1 DATASETS
The family of CODEGEN models is trained sequentially on three datasets: THEPILE, BIGQUERY, and BIGPYTHION.
The natural language dataset THEPILE an 825.18 18 GiB English text corpus collected by Gao et al. (2020) for language modeling (MIT license). The dataset is constructed from 22 diverse high-quality subsets, one of which is programming language data collected from GitHub repositories with >100 stars that constitute 7.6% of the dataset. Since the majority of THEPILE is English text, the resulting models are called as natural language CODEGEN models (CoDEGEN-NL).
The multi-lingua dataset BIGQUERY is subset of Google's ' publicly available BigQuery dataset, which consists of code (under open- sourcs license) in multiple programming languages. For the multi- lingual training, the following 6 programming languages are chosen: C, C++, Go, Java, JavaScript, and Python. Thus, we refer to models trained on the BIGQUERY as multi-lingual CODEGEN models (CODEGEN-MUULT))
The mono-lingual dataset BIGPYTHON contains a large amount of data in the programming language, Python. We have compiled public, non-personal information from GitHub consisting of permissively licensed Python code in October 2021. Consequently, we refer to models trained on BIGPYTHION as mono-lingual CODEGEN models COODGGEN-MONO)
The pre-processing follows: (1) filtering, (2) deduplication, (3) tokenization, (4) shuffling, and (5) concatenation. For details on THEPILE, we refer Gao al. (2020). For BIGQUERY and BIGPYTHON, we refer to Appendix A Table summarizes the statistics of the training corpora.
2.2 MODELS
The CODEGEN models are in the form of autoregressive transformers with next-token prediction language modeling as the learning objective trained on natural language corpus and programming language data curated from GitHub. The models are trained in various sizes with 350M, 2.7B, 6. BB, and 16.1 parameters. The first three configurations allow for direct comparison with open-sourced large language models trained on text corpus, GPT-NEO (350M, 2.7B) (Black et al.) 2021) and GPT-J (6B) (Wang & Komatsuzaki 1021). See Table 6 in Appendix A for model specifications.
The CODEGEN models are trained in a sequential nature over datasets. CODEGEN-NL is first trained on THEPIL.. CODEGEN-MUS MUL.I is initialized from CODEGEN-NL and trained on BIGQUERY. Finally CODEGEN-MONE is initialized from GRDEGEN-MULTY and trained on BIGPYTHON.
The emergence of program synthesis conditional on descriptions in natural language may stem from the size of the models and data, training objective, and nature of the training data itself. This is called emergence since we do not explicitly train the model on comment-code pairs Similar phenomena are observed in wide range of natural language tasks where large-scale unsupervised language model can solve unseen tasks in a zero-shot fashion (Brown et al. 2020). The emergence phenomena or surprising zero-shot generalization is often attributed to the large scale of the model and the data.
While our focus is not to reveal the underlying mechanism on why program synthesis capacity emerges from simple language modeling, we make an attempt to provide an explanation given the nature of our modeling approach and the training data. The data consists of regular code from GitHub (without manual selection), for which some data exhibits pattern of interleaved natural and programming language, which we believe provides a noisy supervision signal for the program synthesis capacity due the next-token prediction training objective. However, such a data pattern is highly noisy and weak, because only a subset of data exhibits such a pattern, e.g., comments may be inaccurate uninformative, and some of them may even be placed at an irrelevant
emphasize that
3

---

Published as a conference paper at ICLR 2023
pass @ [%]
Model
k
100
GPT-NEO 350M
0.85
2.55
5.95
GPT-NEO 2.7B
6,41
11.27
21.37
GPT-J 6B
11.62
15.74
27.74
CODEX 300M
13.17
20.37
36.27
CODEX 2.5B
21.36
35,42
59.50
CODEX 12B
28.81
46.81
72.31
code-cushman-001
33.5
54.3
77.4
code-davinci-001* code-duvinci-002*
39.0
60.6
84.1
47.0
74.9
92.1
CODEGEN-NL 350M
2.12
7.38
CODEGEN-NI. 2.7B
6.70
14.15
22.84
CODEGEN-NL 6.1B
10.43
18.36
29.85
CODEGEN-NL 16.1B
14.24
23.46
38.33
CODEGEN-MULT 350M
6.67
10.61
16.84
CODEGEN- 2.7B
14.51
24.67
38.56
CODEGEN-MULTI 6.1
18.16
28.71
44.85
CoDEGEN-MULTI I6.
18.32
32.07
50.80
CODEGEN-MONO 350M
12.76
23.11
35.19
CODEGEN-MONO 2.7B
23.70
36.64
57.01
CODEGEN- 6.1B
26.13
42.29
65.82
CODEGEN- MONO 16.1B
29.28
49.86
75.00
Table Evaluation results on the HumanFval benchmark. Each p (where k E {1, 10, 100))
for each model is computed with three sampling temperatures (t {0.2, 0.6, 0.8)) and the highest one among the three are displayed, which follows the evaluation procedure in Chen et al. (202T).
Results for the model marked with are from Chen al. (2022).
position. Therefore, believe two main factors contribute to the program synthesis capacity: 1) large scale of model size and data size and 2) noisy signal in training data.
The scaling of such LLMS requires data and model parallelism. To address these requirements, training library JAXFORMER hnttps://githubb.com com/sa http://saasssorec///oorreree efficient training on Google's TPU-v4 hardware. We refer to Appendix (A for further details on the technical implementation and sharding schemes. Table 6 summarizes the
was developed for
SINGLE-TURI EVALUATION
We first evaluate our CODEGEN using existing program synthesis benchmark: HumanEval (MIT license) (Chen et al. 202T). HumanEyal contains 164 hand-written Python programming problems. Each problem provides prompt with descriptions of the function be generated, function signature, and example test cases in the form of assertions. The model needs to complete function given the prompt such that it can pass all provided test cases, thus measuring the performance by functional correctness. Since a user intent is specified a single prompt and provided to the model once, we regard the evaluation HumanEval evaluation which we introduce in the next section. Following Chen et al. (2021). we recruit nucleus sampling (Holtzman al. 2020 with top-p where p 0.95.
single-turn evaluation, to distinguish it from the multi-turn
3.1 HUMANEVAL PERFORMANCE SCALES AS FUNCTION OF MODEL SIZE AND DATA SIZE
We compare our models to the Codex models (Chen et al. 2021), which demonstrate the of the-art performance on Hvma Moreover, our models are compared to open-sourced large language models, GPT-NEO (Black et al. 202 and GPT-J (Wang & Komatsuzaki 2021). These are trained on THEPILE (Gao et al. 2020). and thus similar to our CODEGEN-NL models, in terms of training data and model size. All models
evaluated with temperature t {0.2, 0.6, 0.8), and we compute pass @ & where E {1, 10, 100) for each model. For direct comparison to the results by Chen al. 2021) choose the temperature that yields the best-performing pass @ & for each
4

---

Published as a conference paper at ICLR 2023
CODEGEN-MONE
350M
2.7B
6.1B
16.
Pass
3.78 = 0.23
3.66
0.14
3.35 + 0.13
3.12
0.11
Non-Pass
5.18 = 0.19
4.37 - 0.18
3.88 + 0.13
3.40
0.11
Table 2: Average prompt perplexity* (# standard error) of CODEGEN-MONO models on pass and non-pass problems.
k. The results of our models and baselines are summarized in Table Our CODEGEN-NL models (350M, 2.7B, 6.1B) outperform or perform on par with the respective GPT-NEO and GPT-J models. Further training CODEGEN-N -I on multilingual programming language data (BiGQUeRY) leads to CODEGEN-MULTI, The multilingual CODEGEN models outperform the models trained on THEPILE (GPT-NEO, GPT-J, CODEGEN-NL) by large margin. We then finetune CODEGEN-MULTI on Python-only dataset (BIGPYTHON), resulting in CODEGEN-MONG The program synthesis capacity improved substantially. Therefore, the Python program synthesis capacity enhances as the amount of Python training data increases. For almost all models, expected, increasing the size of the model improves overall performance.
Our Python-monolingual CODEGEN models have competitive or improved performance, compared to the current sratt-of-the-eee models, Codex. CODEGEN-MONE when & 100 but outperforms it when & E {1, 10). While it is only half the size, our CODEGEN- MONO B demonstrates pass @ k scores approaching those of the best-performing Codex, CODEX 12B. Our largest model CODEGEN-Menn -MONo 16. 1B is competitive outperforms it depending on k.
2.7B underperforms CODEX 2.5B
3.2 BETTER USER INTENT UNDERSTANDING YIELDS BETTER SYNTHESIZED PROGRAMS
The success of a program synthesis system highly depends on how well it understands user intent. When the system is based a language model, the perplexity of problem prompts provides proxy for the system's 's understanding of user intent specifications. A low perplexity of an intent specification under model indicates that this intent specification is compatible with the knowledge learned by the model from the training data, We investigate whether better prompt understanding, with lower prompt perplexity
proxy, leads more functionally accurate programs.
We partition all problems into pass versus non-pass ones. A pass problem is one that at least one sample from 200 samples passes all test cases, while for a non-pass problem none of the 200 samples pass all test cases. We compute the average perplexity of the problem prompts of the pass problems and that of the non-pass ones, based on samples from CODEGEN-MONO models. The results are displayed in Table Appendix F for the results on CODEGEN-NL and CODEGEN-MULTII The prompts of the pass problems have lower perplexity than those of the non-pass ones. This finding implies that program synthesis is more likely to be successful when the user intent specification is understood better by the model. Indeed, some training data contains interleaved sequences natural language comments and programs, where the comments describe the functionality of the following program. We thus speculate that user intent specifications similar to such a pattern would be better understood by the model, and hence lead better program synthesis. Inspired by this pattern, propose to specify user intent in multiple turns such that the model focus on a partial problem at a time, which would make user intent understanding by the model easier.
MULTI-TURN EVALUATION
In this section, we propose and study a multi-step program synthesis paradigm where program synthesis is decomposed into multiple steps and the system synthesizes subprogram in each step. To examine such paradigm, we first develop a Multi-Turn Programming Benchmark (MTPB). MTPB consists of 115 problems written by experts, each of which includes multi-step descriptions in natural language (prompt). To solve a problem, a model needs to synthesize functionally correct subprograms (1) following the description at the current step and (2) considering descriptions and synthesized subprograms at previous steps (e.g., correct backreference of functions and/or variables defined in the previous steps). An illustrative example is shown in Figure [
5

---

Published as a conference paper at ICLR 2023
Discourse
Generation
Input abc www.ggeennpppaia.
Samppe
Import define reguar oxprasaion that masches araal
Tum
Tum
Tum
P1
62
Tum
PM
F4
"(opprt) stow
Tum
Pb
P4
2%
Concatenate
Evaluation
Mose
Ps
Ranowe sucer sutetig daring Fon the symbol from "adereae"
Ripert oetiv reputa aquassio tua satore
de X3 ripte no fot tuutte 7o0000
sdoras adtnas addraa 1arthhgh1
autrrray cartary the cutool
Chandding dee
suatec with verrenscc feethrr
Model
print
Expested Ougutt
Taze
"abc
Actual Output
An illustrative example for the Multi-Turn Programming Benchmark, performing the task of extracting the user name of an email address. ! Each problem consists prompts p; and unit tests, where some prompts include templates (i.e. {input}) that are filled with test case inputs before it is fed to the model. In the displayed example, the input a string containing abc bbbc.xy@@exmmpl. com, which replaces { {input} in P2, and the expected output is abc xyz. 2 Our model conditions on the concatenation of interleaved past prompts and generated responses. Generated responses from each turn are concatenated and executed, where the output compared to the answer.
Figure 1:
4.1 BENCIMARK CONSTRUCTION
We (4 authors) start by defining a set of 115 problems requiring a diverse range of programming knowledge, including math, arryy operations, string manipulations, algorithmss, data science, and problems that require other knowledge, such that the number of problems in each category is roughly balanced For each problem, construct triplet consisting of multi-turn prompts P. test case inputs 7, and test case outputs O. Multi-turn prompts P are designed following the two constraints: (1) the problem is decomposed into more turns, (2) single turn cannot be attributed to solving the problem. For example, implementing a linear regression model could be phrased as "Perform linear regression on x and y"" Since the main task is fully expressed in this prompt, understanding this prompt is sufficient to perform the task. We avoid such cases via manual inspection and distribute problem-solvee solving over turns. Together with the prompts, we task the problem author to prepare 5 sets of test case inputs 7 and outputs O to evaluate model outputs with functional correctness. To reduce wrongly rewarding false positive solutions that give meaningless programs but pass the tests, we examine and revise such cases ensure the test quality.
Unlike HumanEval for which models are expected to complete partially defined function, MTPB problems only provide the prompts, thereby models have to generate the solution from scratch While the free-form generation may allow for more potential solutions, the lack of an entry point to provide test inputs makes it challenging to test the generated code diverse test cases. To overcome this challenge, we instead embed test case inputs within prompts. Specifically, prompts written with Python's formatted string" where input values substituted for the variable name when a specific test case is applied to the problem. For example, a prompt, "Define a string named 's'
PProblem writing was performed a closed book format, i.e. we are not allowed to consult with online resources while writing the problems.
See Appendix Dor complete listing.
"To guide sampling in Python, we prefix the prompt with: # Import libraries. import numpy as htw.lllsis.ttmlfsssriiiii

---

Published as a conference paper at ICLR 2023
Pass Rate [%]
Data
Model
350M
2.7B
6.1B
16.
THEPULE THEPILE
GPT-NEO GPT-J
0.79
8.17
18.86
CODEGEN-NL
0.23
15.31
19.37
30.33
BIGQUERY
CODEGEN-MULTI
4.09
20.82
25.51
26.27
BIGPYTHON
MODO
16.98
38.72
43.52
47.34
code-cushman-000
56.77
code-davinci-001
55.28
code-davinci-0022
59,86
Table 3: Evaluation results on the Multi-Tury Programming Benchmark. The multi-turn program synthesis performance varies as function of model size (columns) and code data size (rows).
PPL.
Pass Rate" [%]
Prompt
350M
2.7B
6.1B
16.1B
350M
2.7B
6.1B
16.11
Single-Turn
13.92 1 1.89
11.67 1: 1.46
10.58 1: 1.20
10.25 1: 0.99
5.75
25.43
28.48
38.74
Multi-Tum
10.09 + 0.62
8.90 + 0.52
8.18 + 0.43
8.05 + 0.43
16.98
38.72
43.52
47.34
Table 4:
Comparison between multi- and concatenated single-turn specifications on perplexity (PPL) and program synthesis performance (as measured by pass rate) under CODEGEN-MONO models.
with the value [var].", together with a test case input var 'Hello' will be formatted into "Define a string named 's' with the value 'Hello'. Also see I in Figure for an example.
4.2 EXECUTION ENVIRONMENT AND SOLUTION EVALUATION
For execution, the history of pairs of prompts and generated completions concatenated into a self-contained program (see 3 in Figure for an example). The program is then executed in an isolated Python environment following the single-turn HvmanEval benchmark (Chen al.) (02II) However, the problems in HumanEval are constructed in such a way that known function signature completed, thus invocation of the generated code under of functional unit tests is trivial. In our multi-turn case, no such entry point (or return value) is guaranteed to be generated. To circumvent the issue of missing return signature (or value), the last prompt of the multi-turn problems in MTPB is always specified to print out the resulting state to the terminal. Then, the benchmark execution environment overloads the Python print print(args) function and stores args on a stack. If the sampled code for the last prompt of problem does not include the print() statement, which is valid convention to print on the terminal in Python or specifically Jupyter notebooks, then the AST of the generated code will be mutated inject invocation of print(). Finally, type-relaxed equivalence check (e.g., an implicit conversion between lists and tuples) of args against the predefined gold output of the problem is performed determine test failure success.
4.3 MULTI-STEP PROGRAMMINE CAPACITY SCALES WITH MODEL SIZE AND DATA SIZE
In this analysis, we investigate how the model size and data size affect the program synthesis capacity in multi-turn paradigm. In the MTPB, each problem has test cases and we sample 40 samples for each test case with each model, based on which the pass rate is computed for each problem. The MTPB evaluation results (average pass rate) for our CODEGEN models, baselines, and OpenAl Codex model are shown in Table 3 Clearly, the performance on the MTPB improves as a function of the model size and data size. This suggests that the capacity of multi-step program synthesis scales as a function of the model size and data size. The models are simply trained with an autoregressive language modeling objective. While the model and the data scale up, multi-tum program synthesis capacity emerges, that is, the capacity to synthesize programs in a multi-turn turtuuu fashion.
"Accessed November 10th, 2022.

---

Published as a conference paper at ICLR 2023
Diferrooinn PitereooinnPeeett Parttor
Modiar
13-
10
30095
2.73
6.1
16J8
Narther of Medil Paramatoon
Difference in average pass-rate of problems in single-tur and multi-tury formulation over levels of problem difficulty. The improvement sizable for most model sizes and difficulty levels, except for easy problems with larger models.
Figure 2:
4.4 BETTER USER SPECIFICATION UNDERSTANDING WITH MULTI-TURN FACTORIZATION
We hypothesize that multi-turn factorization enhances the model's understanding of user intent specifications, which in turn lead higher program synthesis capacity. To test this hypothesis, we form a single-tun turgee-turrn counterpart of multi-turn specifications by concatenating each specification into a single turn. As discussed Section 3.2 we adopt the prompt perplexity as proxy for user intent understanding. Thus, we compare the perplexity of the multi-turn prompts and that of the concatenated single-turn prompts under the four CODEGEN-MONO models.
The average perplexity (see Appendix for the calculation details) over all the problems in the MTPB displayed in the left panel of Table 4 For all models, the single-turn specification has a higher average perplexity than the multi-turn specification. It implies that the multi-turn user specifications can be better understood by the models. We notice that the average perplexity for both multi- turn and single-tur intent specifications under larger models is slightly lower than that under smaller models, indicating that the larger ones understand the user intent better than the smaller ones.
We compare the program synthesis pass rate with the multi-turn prompts that with the concatenated single-turn prompts. The results are shown in the right panel of Table 4 Multi-turn specifications lead close to more than 10 percentage points over single-turn specifications for all model sizes. Together with the perplexity analysis above, it appears that factorizing a user specification into multiple steps and leveraging the emerged capacity of large language models allow them digest the specification more easily and synthesize programs more successfully.
Furthermore, we categorize the problems by difficulty level based on their average pass rates ("hard" with less than 30%, "casy" with larger than 70%), and cxamine the interaction effect between difficulty level and model size on the improvement by multi-turn factorization. See the results in Figure 2 Across almost all model sizes and difficulty levels, multi-turn prompts lead to significant improvement over single-turn prompts and most improvements are nearly or higher than percentage points. Interestingly, the larger models (6.1D and 16. IB) are invariant to multi-turn factorization for easy problems (see the two short bars, 0.19% and -0.25%. in Figure 2). This implies that when the problems can be easily understood by the model (due to the combined effect of casiness of the problems and the high capacity of larger models), it is not necessary or beneficial to factorize the specifications. This is in fact consistent with our motivating assumption that factorizing complicated specifications would ease problem understanding and improve program synthesis.
4.5 QUALITATIVE EXAMPLES
To further understand the differences in model behavior over model sizes, we examine cases where large models have contrasting performances smaller models. We specifically select problems for which CODEGEN- -MNOO 16. IB and CODEGEN-MONO 2. show a significant discrepancy in performance. On problems where CODEGEN- MONO 16. IB performed significantly worse compared to CODEGEN-MONO 2.7B, we observe that the larger model becomes inflexible due to taking the prompt literally. For example, initializing a number always results in an integer, despite the prompt asking to cast into string (Figure 3), or the "return" keyword in prompt triggers function definition while the intent is to directly generate an executable program (Figure 4 However in general, larger-scale models overcome mistakes due prompt misinterpretation by smaller models, including assigning multiple variables at the same time (Figure 5) or understanding the concept of any comparison (Figure (p).
8

---

Published as a conference paper at ICLR 2023
5 RELATED WORK
Program Synthesis While program synthesis has a long history, two inherent challenges remain unsolved: (1) intractability of the program space and (2) difficulty in accurately expressing user
intent (Manna & Waldinger 1971; Gulwani et al. 2017). A large body of prior research attempted to
address (1) by exploring methods like stochastic search techniques (Parisotto al., 2017 Schkufza
et al. ) and deductive top-down search (Gulwani
PPllozo & Gulwani 2015). However,
the scalability of these approaches is still limited. User intent can be expressed with various methods:
formal logical specifications, input-output examples, and natural language descriptions. Complete and formal specifications require too much effort, while informal ones like input-output examples often under-specify problems (Gulwani 201T). Well-learned conditional distribution and language understanding capacity owing to the large-scale model and data allows for efficient solutions for these two challenges. Several works investigate converting conversational intents into programmable representations, such as SQL (Yu et al.] 2019a|b) or dataflow graph (Andrees et al. 2020). Our proposed benchmark requires the generation of Python, which is more general and complex.
Large Language Models Transformers capture dependency among sequence elements through attention mechanism (Bahdanau et al. (0144) and are highly scalable. has been successfully applied to natural language processing (Devlin et al. 2019] Lewis et al.. [220]] Raffel et al. (022)) computer vision (Dosovitskiy et al., 2021). and many other areas (Oord et al. 2018| Jumper et al. 2021). Prior works, such as CuBERT (Kanade et al. 2020), CodeBERT (Feng et al. (020). PyMT5 (Clement et al., 2020). and CodeTS (Wang et al. 2021 have applied transformers towards code understanding but these mostly focus on code retrieval, classification, and program repair. Several recent and concurrent
efforts explore using large language models for program synthesis (Chen et al., 2021 Austin et al.
2021: Li et al. 2022 Fried et al., 2022) and its effectiveness /Vaithilingam al. [022] While they focus on generating code in single turn, we propose to factorize the specifications into multiple turns
and demonstrate that it is highly effective to improve synthesis quality. It is worth pointing out that Austin et al. (202T) explored refining the code in multiple iterations, but it is essentially a single-turn approach since a complete program is produced in every single turn. Prompting pre- pre-trained language models with intermediate information to improve task performance has attracted interest (Nye et al.) 202T
Wei al. 2022). Our proposed MTPB also allows the model to leverage past turns as context.
Benchmarks for Program Synthesis To quantitatively evaluate program synthesis models, several benchmarks have been proposed with different input forms. A popular imput forms include preceding code in the same line (Raychev et al. (2066), pseudo-code (Kulal et al.] (019). docstring and function signature (Chen et al., (021). or problem description (Hendrycks et al. (022)) In most of those cases, only directly relevant input information is given the model. In contrast, few previous works instantiate benchmarks that measure the ability to generate programs given surrounding program context beyond the target program, such as variables and other methods (lyer et al., 2018)) or alternating "cells" of preceding code and text blocks (Agashe et al. 2019), while the primary Tocus to generate the target program itself. We propose a new benchmark that requires a progressive generation of subprograms through multi-turn prompts.
6 CONCLUSION
We study program synthesis with large causal language models trained on large corpora of code data. The capacity to understand long context and generate coherent responses emerges from the simple language modeling as the model size and data size scale up. Leveraging this capacity and observing that better user intent understanding leads to better program synthesis, we propose a multi-step program synthesis approach in which program synthesis is achieved through a multi-turn specification and code generation. Moreover, we develop the Multi-Turn Programming Benchmark (MTPB) to investigate our models' capacity synthesizing programs in such a multi-step paradigm. Our experiments show that the multi-step program synthesis capacity scales as a function of the model size and data size. The intent specifications, which are specified in multiple steps, are digested more easily by the models and lead to more accurate program synthesis. We open-source the training code and the model checkpoints to facilitate future research and practical applications in this area.

---

Published as a conference paper at ICLR 2023
BROADER IMPACT AND ETHICAL CONSIDERATIONS
All variants of CODEGEN are firstly pre-trained on the Pile, which includes a small portion of profane language. Focusing on the GitHub data that best aligns our expected use case of program synthesis, Gao et al. (020) report that 0.1 % of the data contained profane language, and has sentiment biases against gender and certain religious groups. Thus, while did not observe in our samples, CODEGEN may generate such content as well. In addition to risks on natural language outputs (e.g. docstrings), generated programs may include vulnerabilities and safety concerns, which are not remedied in this work. Models should not be used in applications until being treated for these risks.
REFERENCES
Rajas Agashe, Srinivasan Iyer, and Luke Zettlemoyer. Juice: A large scale distantly supervised dataset for open domain context-based code generation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 5436-5446, 2019.
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Fisner, et al. Task-oriented dialogue as dataflow synthesis. Transactions of the Association for Computational Linguistics, 8:556-571, 2020.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.00 07733, 2021.
Dzmitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint urXiv: 1409.043, 2014.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with fess-Tensor,,ow, March 2021. URL https://doi.orr you use this software, please cite using these metadata.
10. 50.5221/zeoodo. 5297715
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://github.. hoom/google/aar
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.1033) 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.00372. 2021.
Alvin Cheung, Armando Solar-Lezama, and Samuel Madden. Optimizing database-backed applica- tions with query synthesis. ACM SIGPLAN Notices, 48(6):3-14, 2013.
Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. Pymt5: multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9052-9065, 2020.
Jacob Devlin, Ming-Wee Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18693/v1/N19. 1423. URL https 7/ac //aclannhology.orgg//// 1423
10

---

Published as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. URL httph:tt////peeeeiiiiiii http///wwwweeeee//ommmmmddvdddda
net wtte/eooouuuu
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguisties: EMNLP 2020, pp. 1536-1547, 2020.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.059900 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint urXiv:2101.00027, 2020.
Sumit Gulwani. Automating string processing in spreadsheets using input-output examples. ACM
Sigplan Notices, 46(1):317-330) 2011.
Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. Program synthesis. Foundations and Trends@ in Programming Languages, 4(1-2):1-119, 2017.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL httpp:://oeeerriiee.teefforrrrrp 5D9360ZH315
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. ICLR, 2020. URL httptt://oppereeiiieeeee wttp:/ooruuiidddyyyeyyrryuu
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1643-1652. Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/11DD18-11999 URL /RR/sttts://aallattooo.o org/
D18-118 1192
Rajeev Joshi, Greg Nelson, and Keith Randall. Denali: A goal-directed superoptimizer. ACM SIGPLAN Notices, 37(5):304-33 2002.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-555) 3599, 2021.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. International Conference on Machine Learning, pp. 5110-5121. PMLR, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. hRtp::///////i /bss/ 1412, 6980
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising eequence-to-sequeee pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871 -8800 2020.

---

Published as a conference paper at ICLR 2023
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume. Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukeuoglu, and Oriol Vinyals. Competition-level code generation with alphacode, Feb 2022.
Zohar Manna and Richard J Waldinger. Toward automatic program synthesis. Communications of the ACM, 14(3):111166,, 1971.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.0011 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint /rXiv:/807003748 2018.
Pavel Panchekha, Alex Sanehez-Stem, James Wilcox, and Zachary Tatlock. Automatically improving accuracy for floating point expressions. ACM SIGPIAN Notices, 10(6)) 2015.
Emilio Parisotto, Abdel rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. Neuro-symbolic program synthesis. In ICLR (Poster), 2017. URL httpss//oppeereeeee
net/forummiddrrJooWFcc.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318. PMLR, 2013.
Oleksandr Polozov and Sumit Gulwani. Flashmeta: A framework for inductive program synthe- sis. In Proceedings of the 2015 ACM SIGPLAN International Conference on Ppject-Oriented Programming, Systems, Languages, and Applications, pp. 107-126, 2015.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking. Storage and Analysis, pp. 1-16. IEEE, 2020.
Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. ACM SIGPLAN Notices, 51(10))731-747, 2016.
Eric Schkufza, Rahul Sharma, and Alex Aiken. Stochastic superoptimization. ACM SIGARCH Computer Architecture News, 41 (11(13300-311111 2013.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, arXiv preprint 1xXiv:2104.09864 2021.
Priyan Vaithilingam, Tianyi Zhang, and Flena I. Glassman. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-7, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-60088 2017.
Ben Wang and Aran Komatsuzaki. GPT.I- A 6 Billion Parameter Autoregressive Language Model.
toansssorrttttarrrrrrrrreeomreee jax May 2021.
12

---

Published as a conference paper at ICLR 2023
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. Codet5: Identifier-aware unified pre- trained encoder-decoder models for code understanding and generation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. CoSQL: A conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)) pp. 1962-1979, Hong Kong, China, November 2019a. Association for Computational Linguisties. doi: 10.18653/v 1/D19-1211 URL http://acclatthooo...oooooo htt.:/oooooo..o////-1199900099 D
Tao Yu, Rui Zhang. Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang. Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev. SParC: Cross-domain semantic parsing in context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 451 1-4523, Florence, Italy, July 2019b. Association for Computational Linguistics. doi: 10.18653/v 1/P -1--1999999 1443. URL https: http://aaaaantn www.hhooooy...r......... 1443
13

---

Published as a conference paper at ICLR 2023
A MODEL TRAINING
To evaluate the emergence of multi-tur program synthesis capabilities under scaling laws, we adopt standard transformer-based autoregressive language models, varying (1) the number of model parameters (350M, 2.7B, 6.I 16.IB) and (2) the number of tokens of programming languages in the training corpora. For scaling the models, custom library JAXFORMER for training large language models on TPU-v4 hardware was developed and will be released as open source, including the trained model weights.
A.I DATASETS
Dataset
Language
Raw Size
Final Size
Final Tokens
Natural Language
825.18 GiB
1159.04 GiB
354.7B
THEPILE
Code
95.16 GiB
95.16 GiB
31.6B
C
1772.1 GiB
48.9 GiB
19.7B
C++
205.5 GiB
69.9 GiB
25.5B
Go
256.4 GiB
21.4 GiB
9.6B
BIGQUERY
Java
335.1 GiB
120.3 GiB
35.4B
JavaScript
1282.3 GiB
24.7 GiB
9.7B
Python
196.8 GiB
55.9 GiB
19.3B
BIGPYTHON
Python
5558.1 GiB
217.3 GiB
71.7B
Table 5: Approximate statistics for training corpora along the pre- pre-processing steps.
For each dataset, the pre- pre-processing shares the following steps: (1) filtering, (2) deduplication, (3) tokenization, (4) shuffling, and (5) concatenation. For details on THEPILE, we refer to Gao et al. (2020). For BIGQUERY and BIGPYTHON, in (1) files are filtered by file extension, and files with average lines length of <100 characters, a maximum line length of 1, 000, and >90% of the characters being decimal hexadecimal digits are removed. For (2), exact duplicates based on their SHA-256 hash are removed, which amounts to substantial portion of the raw data due to forks and copies of repositories. For (3), the BPE vocabulary of GPT-2 is extended by special tokens representing repeating tokens of tabs and white spaces. In the multi-lingual setting of BIGQUERY, a prefix is prepended to indicate the name the programming language. For (4), each year of data randomly shuffled. For (5), sequences are concatenated to fill the context length of 2, 048 tokens with special token as a separator. Table summarizes the statistics of the training corpora.
CODEGEN-NL models randomly initialized and trained on THEPILE. CODEGEN-MULTI models are initialized from CODEGEN- NL and then trained on the BIGQUERY. CODEGEN-MONN models initialized from CODEGEN-MULTI and then trained on BIGPYTHON.
A.2 MODELS
Our models are autoregressive transformers with the regular next-token prediction language modeling as the learning objective. The family of CODEGEN models is trained in various sizes with 350M, 2.7B. B. and 16.1B parameters. The first three configurations allow for direct comparison with open- sourced large language models trained on text corpus, GPT-NEO (350M, 2.7B) (Black et al. (021) and GPT-J (6B) (Wang& Komatsuzaki 2021). See Table f in Appendix A for model specifications.
The architecture follows a standard transformer decoder with left-to-right causal masking, For the positional encoding, we adopt rotary position embedding (Su et al., 2022]. For the forward pass, we execute the self-attention and feed-forward circuits in parallel for improved communication overhead following Wang & Komatsuzaki (2021), that is, F++1 = 2g + mlp(In(12 + attn(In(x;)))) is altered to .I+1 It + attn(In(22 22)) mlnx)) for which the computation of self-attention attn(). and ford-forward, mlp(), with layer-norm, In(), is simultancous. The architecture and hyper parameter choices were optimized specifically for the hardware layout of TPU-v4.
hyper-parameter
14

---

Published as a conference paper at ICLR 2023
Model
Dataset
Hlyper-parameee
350M
2.7B
6.1B
16.
Number of layers
20
32
33
34
Number of heads
32
16
Dimensions per head
64
80
256
256
CODEGEN
Context length
2,048
2,048
2,048
2,048
Batch size
500k
1M
2M
2M
Weight decay
0.1
0.1
Learning rate
3.0e-1
1.6e
1.2e -
0.9e
CODEGEN-NL.
THEPuLE
Warm-up steps
3k
3k
Warm-up/ Total steps
350k
350k
350k
350k
Learning rate
1 1.8--4 4
0.8e-4 4
0.4e-1p
0.5e
CODEGEN-MUITI
BIGQUERY
Warm-up steps
3k
3k
3k
3k
Total steps
150k
150k
150k
150k
Learning rate Warm-up steps
1.8e-4
0.8e-4
0.4e-4
0.5c
CODEGEN- MONO
BIGPYTHON
3k
3k
3k
3k
Total steps
150k
150k
150k
150k
Table 6: Hyper-paramen for model specification and optimization for the family of CODEGEN models.
A.3 TRAINING
The scaling of large language models requires data and model parallelism. Google's TPU-v4 hardware with high-speed toroidal mesh interconnect naturally allows for efficient parallelism. To efficiently utilize the hardware, the training of the models is implemented in JAX (Bradbury et al. 2018 011). For parallel evaluation JAX the pjat( operator adopted. The operator enables a paradigm named p
Bongle-progran multiple-de data (SPMD) code, which refers to a parallelism technique where the same computation is run on different input data in parallel on different devices (T)) Specifically, pját() is the API exposed for the XLA SPMD partitioner in JAX, which allows given function be evaluated in parallel with equivalent semantics over a logical mesh of compute.
Our library JAXFORMER recruits designated coordinator node orchestrate the cluster of TPU- m with custom TCP/IP protocol. For data parallelism, the coordinator partitions a batch and distributes the partitions the individual TPU-VMs. For model parallelism,
V
schemes for the sharding of model parameters are supported: (1) Matra-TP-VMM. where parameters are sharded across MXU core inside physical TPU-v4 board and replicated across boards following Shoeybi et al.) (211)): Wang & Komatsuzaki (02II; (2) Aite--PP-VMM. where parameters are sharded across
TPU-v4 boards and activations are replicated following Rajbhandari et al. (2020).
Both intra- -P-ra-TPU-VM and inter-TPU-VX. sharding schemes are implemented based on specific pjit() logical mesh specification (r, p, c) with r replicas of the parameters, p partitions of the parameters, and logical cores per board over ni, TPU boards with each n. logical cores such that 1xp p ni and r X p X nı X Ne.
The intra-TPU-VM scheme is adopted for models of size of less or equal to 6B parameters, the total amount model and optimizer parameters fit into the combined HBM memory of a single TPU-v4 board. For instance, TPU-v4-512 slice with n4 64 and n. would be configured as (r, (rPp.c) c) = (64,1, 1). That is, the parameters are being replicated across = 64 boards with p total inter-board partitions and intra-board parallelism across 4 logical chips. In this configuration, the mean gradient is accumulated across boards via with_sharding sethhsssarddngccosttaaaleee constraint(tt) effectively emulating the behavior of the xmap( operator.
Specificallly Trt hag amc 4ogical virtual MXU cores).
15

---

Published as a conference paper at ICLR 2023
The inter-TPU-VM. scheme is adopted for models exceeding the size of 6B parameters for which the model and optimizer parameters have to be sharded across TPU-v4 boards. For instance, a TPU-v4-512 slice with nt, 64 and n. 4 would be configured as pr. c) (1, 64, 4). For larger slices such as TPU-v4-1022 with 74, = 128, one may introduce redundancy in the parameter sharding, e-g., (r: p. c) (2, 64,4), 4). In this configuration, the activations replicated across boards via with_sharding constraint(). Moreover, (r, p, c) allows for backwards compatibility for the logical hardware layout transition from TPU-v3 with c 8 to TPU-v4 with c 4 by adjusting p without the need for re-sharding.
For the optimization, Table summarrzes the syper-parameters. We adopt the Adam (Kingma Ba 2015) optimizer with (81, P2. (0.9, 0.999, 1e-08) and global gradient norm clipping Pascanu al. 2013) of 1.0. The learning rate function over time follows GPT-3 (Brown et al. 2020) with warm-up steps and cosine annealing. In summary, we mainly adopted the GPT-3 reference configurations with minor variations accounting for TPU optimizations. We did have the compute capacity to optimize these hyper-parameters further.
B PASS @ & ESTIMATOR
We use the unbiased estimator proposed in Chen et al. (2021) compute pass@k. For each task, n 2 samples are sampled. particular, we use n = 200 and k S 100. Suppose e number of correct samples, among the samples, which pass all the unit tests. Then the unbiased estimator is defined as follows:
pass Biss@A
Eproblems
R
Directly computing this estimator is numerically unstable. We use the numerically stable numpy implementation introduced by Chen et al. (202T).
C TYPE-RELAXED EQUIVALENCE CHECK FOR MТPB EVALUATION
perform the following type-relaxation before assessing the equivalence between model outputs and the expected outputs.
Convert numpy arrays into correspondingly typed lists of standard types (e.g. np. in.int32 will be cast to int).
pandas series are converted and compared in numpy array format.
For the rest, model outputs are cast into the type of gold standard outputs.
Floating numbers are compared with 1e-6 as the tolerance threshold.
16

---

Published as a conference paper at ICLR 2023
D LIST OF MTPB PROBLEMS
Problem Name
Problem Description
Category
Sandwich string
Append a string the middle of another string.
string
Normalize integer list
Normalize a list of positive integers and print formatted percentages. Convert units of time.
math math
Convert time
Squared Fihonacci
Print the squared Fihonacci numbers.
math array
Compare counts
Compare the count of positive and negative numbers in given list. Construct and compute the mean of a pandas DataFrame. Solve the fizz buzz problem.
Pandas mean
D.S.
Fizz buzz
Algo.
Bi-grams
Print the bi-grams of sentence.
string
Top note
Print the name with top note out a dictionary.
dict
Hex to binary
Convert hex to binary and reverse.
math
Invert diet
Detect an inversion of a given dictionary.
diet
Class definition
Create POJO class.
class
Longest number
Print the longest number.
math
Linear regression
Fit linear regression model with specified function and sk-learn.
D.S.
Encrypt and decrypt
Rotate alphabet for encryption, then reverse the operation.
Algo.
Dedup custom objects
Implement a class with hash and obtain a count unique objects.
class
Drunken python
Convert between integer and string without using built-in functions. Encode string into morse code given its conversion rule.
string
Morse code
Algo.
Two-sum
Implement the two-sum prohlem on a given input pair.
Algo.
k-means
Implement and run k-means on sampled points.
D.S.
Even odd sum
Print the sum of even and odd numbers in a list.
math
Shift zeros
Move all the Calculate the bootstrap 95% confidence interval of an array. Sum the even digits between two numbers.
a list the right.
array
Bootstrap 95% CI
D.S.
Sum even digits
math
Min-max diff
Compute the difference between max and min numbers in list. Print the sorted, case-insensitive unique characters of string. Compare and print the longer string given strings. Sum numbers before and after the decimal point of float. Count the number of vowels in string. Compute the factorial of Finds the maximum range of triangle's third edge. Compute the factorial and remainder when divided. Sum the angles in a polygon.
array
Distinct chars
string
Longer string
string
Sum float digits
math
Count vowels
string
Factorial
math
Max edge triangle
math
Factorial & remainder
math
Sum polygon angles
math
Sum string numbers
Add together two numbers represented in string. Sum the range from the minimum the maximum of a list.
string
Min-max sum
array
Vowel overlap
Find the number of overlapped vowels of two words.
string
Sum negative
Calculate the sum of negative numbers in list.
math
Load dataset
Load from a file and print statistics.
D.S.
Char length list
Return list of non punctuation character lengths from words. Convert six hexadecimal digit string into list of RGB values. Check certain element the majority of given list.
string
Hex to RGB
math
Majority vote
array
Week later
Print formatted date week later given a date.
string
Sorted word weights
Check if the list of word weights (sum of ASCII values) are sorted. Sum pairs of adjacent digits until the number is palindrome.
math
Create Palindrome
string
Simulate Backspace
Apply the backspace characters in string and print the modified. split into train and test set. Sum the integers a (min, max) range that don't 't appear a list. Find if string contains digits.
string
Data manipulation
Manipulate pandas DataFrame
D.S.
Sum non-overlap
array
Detect digits
array
Cascading functions
Scquentially invoke function objects in given list. Pluralize duplicated words in list. Given relative altitudes. find the highest altitude Truncate a sentence so that it contains k words Find the elements that appear one time in an array Remove all the occurrences of element in arryy Check whether the sum an array equal to given value
math
Pluralize duplicates
dict
Highest altitude
array
Truncate words
array
Single element
array
Remove elements
array
Check array sum
array
Table 7: Problems in MTPB, showing the problem to 55. D.S. and Algo. refers to data science and algorithm.
17

---

Published as a conference paper at ICLR 2023
Problem Name
Problem Description
Category
Merge sorted lists
Merge sorted lists into one
Algo.
Maximum subarray
Find the max contiguous subarray and return sum Find the largest integer but smaller than the square root Find the longest word in word list Sum all the unique numbers a list Compute the diagonal sum matrix Check condition number of matrix is less than a threshold Compute matrix multiplication sum of two matrices Compare two matrix determinants Compute the log of exponential input Find the nearest points the origin Find the longest common prefix of two strings Find duplicates in list
Algo.
Max square rool integer
Algo.
Longest word
Algo.
Sum unique elements
Algo.
Diagonal sum
D.S.
Matrix condition number
D.S.
Matrix multiplication sum
D.S.
Matrix determinant
D.S.
Log-sum-exp
D.S.
K nearest points
array
Longest common prefix
Algo.
Duplicate elements
array
First unique character
Find the first non-repeating character in string Find uncommon words in two sentences Compute the average word length of a sentence Compare the character frequencies in two strings Reverse string
Algo.
Uncommon words
Algo.
Average words length
Algo.
Compare char freq
string
Reverse string
string
Square Sum diff
Difference between the square of and the of squares Compute the cosine similarity between two vectors Compare vector distances to the origin
math
Cosine sim
math
Vector distance
math
Smallest standard dev.
Find the smaller standard deviation given two lists Find the smaller mean given two lists Compute coefficient of variation given list Compute the 1.1 norm given a Compute z-statistic given a list Move all negative elements in list to the end Remove alphabetical characters in string Find the largest norm among n-dimensional points Given two arrays (pred, gold), calculate the score Add spaces before capital letters
D.S.
Smallest means
D.S.
Coefficient of variation norm
D.S.
D.S.
Z-statistic
D.S.
Move negatives
array
Remove alphabets
string
Largest norm
D.S.
F1 score
D.S.
Add Space
string
Remove outlier
Remove data points in the tail (2sigma) of normal distribution Convert values into categorical variables Group items an array using a provided function Given an array of "prices", find the max profit Sum of all position indices where value appear
D.S.
Convert to categorical
D.S.
Group by key
array
Max stock profit
array
Sum positions
array
Find missing num
Find missing number given list and Common numbers among rows a matrix Ohtain sum of Collatz sequence starting from given number Name the location of a "ball" after cup swapping
number
array
Common num matrix
array
Sum Collatz
Algo.
Cup swap
Algo.
Reverse digits
Reverse digits in number with stack
Algo.
Calculate arrows
Calculate arrowheads left and right
Algo.
Check interval num
Check if interval (max-min) included in list
Algo.
Length encoding
Encode a string by converting repeated chars with counts Use regex to match email addresses and remove special chars Print out the second largest element in arry Return largest prefix sum an array Find the element which is closest to and print the distance Find the max length contiguous subarray with unique characters Ohtain the frequency of the most frequent character Find the length of longest palindrome substring
string
Convert email
string
Second largest
array
Largest prefix sum Closest element to zero Consecutive unique char Highest frequency char Longest palindrome
array
array
string
string
string
Count primes
Calculate prime numbers in a range Rotate an array to the right k steps Check if an array can be split into two sets with equal sums
Algo.
Rotate array
Algo.
Partition equal sets
Algo.
Square root integer
Compute the integer part of square root
math
Plus
Return the digits after integer added by
math
Check square sum
Check whether one integer is sum two square numbers Determine whether standard deviation is less than
math
Compare standard dev.
D.S.
Matrix size
Calculate the sum of row and column numbers Calculate the difference between mean and median for array
D.S.
Diff' mean and median
D.S.
Table Problems in MTPB, showing the problem 56 to 115. D.S. and Algo. refers to data science and algorithm.
18

---

Published as a conference paper at ICLR 2023
E PERPLEXITY COMPUTATION FOR SINGLE- AND MULTI-TURN PROMPTS
Suppose {p: is the set prompts for a given problem, and {8, are the n sub-programt syn- thesized by model Pe. Suppose C;-1 |p $1; ... Pi-1; 8i-11 where : indicates concatenation, the conditional probability of p; is Prob; = Pa (p-j)c--))) and then the perplexity for the multi-turn prompts is computed
n log Prob;
PPLMukd-turn exp
Σ
(2)
m
where m the total number of tokens of all prompts {pr 1- Suppose [pi: :.:....... Pv: 8n). then its probability is Prob (a(c), and the the perplexity for the single-turn prompts is computed as
PP PPPLSingle-turn
exp (- m
log Prob)
F PERPLEXITY COMPARISON FOR CODEGEN-NL AND CODEGEN-MULTT
CODEGEN-NL
350M
2.7B
6.1B
Pass
4.53
3.25
2.78
Non-Pass
4.96
3.87
3.65
Table 9: Average prompt perplexity+ of CODEGEN-NL models on pass and non-pass problems.
CODEGEN-MEN
350M
2.7B
6.1B
Pass
4.78
3.82
3.82
Non-Pass
5.64
4.85
4.80
Table 10: Average prompt perplexity+ of CODEGEN-MULTI models pass and non-pass problems.
G ADDITIONAL BENCHMARK RESULTS
Model
pass
pass
pass @ 100
CODEGEN-NL 350M
0.96
19.91
CODeGEN-NL 2.7B
5.34
24.63
48.95
CODEGEN-NL
8.15
55.27
CODEGEN-NL. 16.1B
10.92
38.43
62.76
CODEGEN-MULTI 350M
7.46
24.18
46.37
CoDeGEN-MULLI 2.7B
18.06
45.80
65.34
CODEGEN-MULTI 6.
18.35
47.27
67.92
CODEGEN-MULTI 16. 16.1B
20.94
51.61
70.02
CODEGEN-MONO 350M
14.59
41.49
63.00
CoDeGEN-MONo 2.7B
27.31
59.19
74.24
CODEGEN-MONO 6.1B
32,48
64.20
76.81
CODEGEN-MONG 16.1B
35.28
67.32
80.09
INCODeR 6B
21.30
46.50
66.20
code-cushman-O00
45.90
66.90
79.90
code-davinci-001
51.80
72.80
84.10
code-davinci-002
58.10
76.70
84,50
Table 11: Pass on Mostly Basic Python Problems (MBPP).
We also evaluated our models on Mostly Basic Python Problems (MBPP) (Austin et al. 2021). The results are displayed in Table TT Following Chen al. (2022). sampled programs from the
19

---

Published as a conference paper at ICLR 2023
sanitized MBPP for all of our models, with 100 and temperature= 0.8. The last four are from the aforementioned paper. In general we observe the consistent trend improving the performance different versions (NL, Multi, Mono), with our largest CODEGEN-MONE approaching the results from code-cushman-000 While we do not know whether any of OpenAl models is the "Codex 12B" reported in Chen al. (202T), we believe model achieves reasonable results on MBPP as well. We also note that our CODEGEN-MONE 6. B. significantly outperformed INCODER 6B.
16.1B
H GENERATED SAMPLES
H.1 CASES WHEERE CODEGEN-MONT 16. IB UNDER-PERFORMS
Import libraries. import numpy as np
Import libraries.
import numpy as np
Initialize variable "stack" with empty list, and "num" with string.
Initialize variable "stack" with empty list, and "num" with string.
stack = () num "108"
stack [] 100
For each chracter in "num", append the character to "stack".
For each chracter in "num", append the character to "stack".
for i in range(len(nun)): (puceeappnddn(m[]])
for i in num: appek.))
Assign an empty string to variable result", and concatenate characters popped from the last element "stack" to "result" until "stack" empty.
Assign an empty string to variable result", and concatenate characters popped from the last element "stack"
to "result" until "stack" empty.
result
result while len(stack)
while !en(stack)!= result stack.pop()
result pop((
Cast "result" as integer and print it out.
Cast "result" as integer and print it out.
(rint())t(result))
print(int(resultt)
CODEGEN-MONO 2.7B
CODEGEN-MONO 16. IB
Figure 3: Generated samples for "Reverse digits" problem. While being simplistic, we often observe that CODEGEN- MONO 16. assumes the type when assigning a number a variable. Here on the right, despite being explicitly specified to initialize number string, the larger model fails to do so. This causes an error in the following turn when the code attempts iterate over num.
20

---

Published as a conference paper at ICLR 2023
Import libraries.
Import libraries.
import numpy as nр
import numpy as np
Assign the matrix "[[3, 2], [2, a]]* variable named "my_matrix".
Assign the matrix "[(3, 2], [2, 3]]" variable named "my my_matrix""
mamarriiix... [[s, 21, [2, 3]]
my jimmtriix
[[3, 2], [2, 3]]
Assign the number "1" variable named "t"
Assign the number "1" variable named 't"
==1
Compute the condition number my_matrix and store as result.
Compute the condition number of my_matrix and store as result.
result np. canaalgg.codmmmmmmttaat matrix)
condition_ number np. conduayy
Check whether result smaller than If yes, return "True", otherwise return "False".
Check whether the result smaller than
If yes, return "True", otherwise return "False".
if result
def check brack_condition ceanddttoonnuberr(eeee number cumber(condition ponbbbtion_nuuberr t):
print( print((Truee) (TTrue")
else:
if condition_ _imbborr_ < t:
return True
else:
False
CODEGEN-MONO 2.7B
CODEGEN-MONO 16. BB
Figure 4: Generated samples for "Matrix condition number" problem. Both models generate programs same scope until the last turn, where CODEGEN MoNO 16. 1B generates the form of function definition. Noticeably, the larger model generates return keyword that appears in the prompt, but it only valid within a function.
21

---

Published as a conference paper at ICLR 2023
H.2 CASES WHERE CODEGEN-MONE 16. IB OUTPERFORMS
Import libraries.
Import libraries.
import numpy as np
import numpy as np
Define a variable "pos" with "B", "swap" with ('AC', "C
Define a variable "pos" with "B", "swap" with ['AC', "cA'].
pos ['B', "swap']
pos "B" swap ['AC', 'CA']
write function "move" that takes two strings and y input, and replace any appearance of in with an empty string, then return y.
write function "move" that takes two strings and y input, and replace any appearance of in with an empty string, then return y.
def move(x,y):
return y.replace(x,"))
def movelx, v): return y.replace(x, "*)
For each element in "swap", if it contains "pos", call "move" on "pos" and the current element and store the result to "pos".
For each element in "swap", if it contains "pos", call "move" on "pos" and the current element and store the result "pos".
def swap(swap):
for i in range(Len(swap)): if swap[i] 'pos':
for i in swap:
if in i: pos move(pos,
swap[i] move(pos(o),pog pos()))
return swap
Print out "pos". print(pos)
Print out "pos".
print(pos)
CODEGEN-MONO 2.7B
CODEGEN-MONO IB
Figure 5: Generated samples for "Cup swap" problem. Here, CODEGEN-MONO 2.7B mistakenly treats "swap" as a part value t o be assigned, while MNDEGEN-MONE 16. 1B parses the intent correctly and assigns variables in parallel.
22

---

Published as a conference paper at ICLR 2023
Import libraries.
Import libraries.
import numpy
import numpy
Initialize the variable named lsta with a list ('abc', 'abcze"].
Initialize the variable named lsti with list ['abc', 'abcie"].
lst1 ['abe', 'abc1o']
lst1 ['abc', 'abc1o']
Create a function called num_ing cin_in_st(() to check whether a string contains a number
Create a function called num num_in_str() to check whether a string contains a number
def num str(string):
def num num_in___ain se_iin_stte in_str(s):
for i in string: if i.isnumeric(): return True
return any(i.isdigit() for in s)
call the function num_in_str() to find strings lsta that have numbers and assign them to list named lst2
else: return False
Call the function sum_in_str() to find strings in 1stı that have numbers and assign them to list named lst2
1st2 [i for i in 1st1 if in str(t)
Print out 1st2
lst2 []
print(lst2)
for lsti:
if num iim_in_str(i;: aste.append
Print out lst2
print(lst2)
CODEGEN-MONO 2.7B
CODEGEN-MONO 16. 1B
Figure 6: Generated samples for "Detect digits" problem. CODEGEN-MONO 2.7B fails to implement the num in_str, resulting in checking only the first character. In contrast, CODEGEN- MoNo 16.1B successfully utilizes any function scan all the characters in the given string.
ADDITIONAL ANALYSES MTPB
We conducted additional analyses to illustrate the relationship generated program length and pass rate and showed the results in Figure 7 Figure 8 and Figure The relationship between generated
program length and prompt length is shown in Figure 10
23

---

Published as a conference paper at ICLR 2023
1.0
0.4
0.2
0.0
100
200
300
400
500
Figure Maximum Length of Completion versus Pass Rate.
0.8
0.7
0.6
0.5
Pase-aate
0.3
02
15.00
47.14
121.84
350.78
Average completion
Figure 8: Maximum Length of Completion versus Pass Rate.
08
0.3
0.
011
0.
110
23
Iverage length completion
Figure 9: Maximum Length of Completion versus Pass Rate.
24

---

Published as a conference paper at ICLR 2023
Token count relationship between prompts and generated programs from BodeGen-16B-Mono
y=x
250
Turn
200
150
140
120
Number of tokens in generated program
100
80
60
20
0
20
40
60
80
100
Number of tokens in prompt
Figure 10: Prompt Length versus Generated Program Length.
25


Reviews:
[{"review_id": "jaByUUjSdS", "reviewer": "Reviewer_JBoh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents new family of open-source code large language models called CodeGen of size upto 16B parameters. This model outperforms Codex 12B model on the HumanEval dataset. The paper also presents a new multi-turn programming dataset where the specification for a single task is divided into multiple sub-tasks so that the model can solve each sub-task independently to solve the full task.    \n", "summary_of_the_review": "The contribution with regards to the CodeGen model is helpful for the community (since it is open-sourced and performs similar to Codex 12B model), but there is limited novelty there. \n\nThe MTPB dataset contribution needs more explanation regarding its practical use cases and its limited capability to handle more complex programs. \n\nPOST REBUTTAL: most of my questions are answered and I am increasing my score to a 8.", "strength_and_weaknesses": "**Strengths:**\n- CodeGen model performs comparatively to Codex (12B) model and is open-sourced (unlike Codex). So it should help facilitate future research in this area of AI for code. \n\n- The results on how various pretraining datasets impacts the coding performance of the models is very interesting. \n\n**Weaknesses:**\n- CodeGen model (single-turn) is only evaluated on a single dataset with only 164 problems. Why is the model not evaluated on other publicly available coding datasets such as MBPP and APPS? \n\n- While the multi-turn dataset helps the models generate more correct code, I am not sure about its practical use cases. Would a user be able to specify the sub-tasks in the required format (such as defining appropriate variables) as in Figure 1? \n\n- How does multi-turn task break down work for programs with control flow structures (for e.g. write code for binary search)? From the examples, it appears that the MTPB dataset treats small loops and branches as a single sub-task, but how can a user specify sub-tasks within a loop/branch?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hS1y6g-lEM", "reviewer": "Reviewer_6Afu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper trains large language models in sequence on three datasets, both natural language and programming language, and publishes the training library, JAXFORMER. The largest model outperforms the codex model on the HumanEval benchmark. The experiments show that the passing problem has lower prompt perplexity on the HumanEval benchmark. Some training data contains interleaved sequences of natural language comments and programs. For example, programmers often explain the functionality of programs with comments written with code. This work hypothesizes that the pattern provides weak supervision of multi-turn code generation. To further investigate this, the paper proposes the multi-turn code generation framework and constructs an open-source benchmark, Multi-Turn Programming Benchmark, to evaluate the proposed framework. For each turn in the generation, the model condition on the concatenation of interleaved past prompts and generated responses. Multi-turn generation specification invariantly outperforms single-turn specification. The evaluation result on the constructed dataset supports the proposed hypothesis. The performance difference in multi-turn and single-turn specification over levels of difficulty and models shows that the improvement is sizable except for easy problems with larger models.", "summary_of_the_review": "I vote for accepting this paper because its idea is original and contributes to this area while some improvement can be made (see above).", "strength_and_weaknesses": "## Paper strengths and contributions\n**Motivation and intuition**\nThe motivation for multi-turn code generation is convincing. The supervision of comments and programs is helpful for multi-turn program synthesis.\n\n**Novelty**\nThe idea of utilizing weak supervision of interleaved patterns is intuitive and convincing. This paper presents an effective way to make use of this idea.\n\n**Technical contribution**\nCodegen for program synthesis seems effective, especially when the user wants to generate pieces of code from input/output examples or natural language descriptions.\n\n**Clarity**\nThe overall writing is clear. The authors utilize figures and tables well to illustrate the ideas. Figure 1 clearly shows the code generation process.\n\n**Related work**\nThe authors clearly describe the related prior works from both the perspectives of program synthesis, large language models, and benchmarks for program synthesis.\n\n**Experimental results**\nThe presentation of the experimental results is clear. Mainly, Table 4 provides understandable results showing that multi-turn specifications achieve better performance compared to single-turn specifications.\n\n**Reproducibility**\nI believe reproducing the results is possible given the clear description provided in the main paper and the appendix.\n\n## Paper weaknesses and questions\n\n**Code comment analysis** \nSome programmers like to write comments, while some are not. Also, different programmers write comments very differently. I suggest the authors investigate the effect of such differences. One very simple way could be to analyze the occurrence frequency of interleaved natural and programming patterns in the dataset. While it is hard to formally define meaningful comments, it would be insightful to at least calculate the document frequency of interleaved natural and programming language.\n\n**Experiment setup**\nIn Table 3, the paper only compares the proposed method against GPT-Neo and GPT-J, which is not sufficient. The result will stand out to compare against Codex, the state-of-the-art program synthesis model.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "R0ICL2zVI0", "reviewer": "Reviewer_wk6b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents several large autoregressive language models trained using large datasets scraped from the web, and in particular datasets of code from GitHub. The authors evaluate the models on tasks of generating code from natural language. First, the authors show state-of-the-art results on the HumanEval dataset and report that larger models also demonstrate lower perplexity on the prompt (the natural language program specification). The authors then describe the Multi-Turn Programming Benchmark (MTPB), in which the natural language intents are interleaved with code and each intent describes what the immediately subsequent code should do. Compared to a baseline where all the natural language is provided to the model concatenated together, the multi-turn version shows improvement.", "summary_of_the_review": "I think the model snapshots from the paper are a highly useful contribution to the community and I based my recommendation largely on that point.", "strength_and_weaknesses": "# Strengths\n- The paper reports state-of-the-art results for the HumanEval benchmark for code generation from natural language specifications.\n- The authors provide downloadable checkpoints for the trained models, which can be a useful resource for the community.\n- The paper introduces a multi-turn natural language to code dataset where the natural language instructions are interleaved with the intended generated code. \n\n# Weaknesses\n- The model is evaluated only on one existing dataset, HumanEval, and not other similar datasets like APPS or MBPP.\n- While the work represents a significant expenditure of resources, the amount of technical novelty is relatively low, and some of the new aspects in the construction of the model itself are not studied or described as much as they could be:\n  - The \\textsc{BigPython} dataset seems responsible for much of the gains in the work, but the description of it in the paper is limited to \"We have compiled public, non-personal information from GitHub consisting of permissively licensed Python code in October 2021\".\n  - There is significant space dedicated to describing the TPU-based training setup with JAX, but not much about the reasoning behind it.\n- The paper doesn't have much experiments about hyperparameters and training data selection (e.g. the order in which the datasets were used for training), or further information/justification for the values used. \n- There is no release of training datasets.\n- The deduplication methodology used during dataset preprocessing may be too weak. Only removing exact duplicate files may still leave many files which are very similar (for example, varying only in some comments at the top of the file). For more discussion on another code dataset, see https://twitter.com/miltos1/status/1497126435261083649\n- Considering that copies of the HumanEval dataset may have been posted to GitHub, the authors should investigate the possibility of training dataset contamination.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FIzfCljkVP", "reviewer": "Reviewer_9c66", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A family of large decoder-only Transformer models fine-tuned on code data are presented and released for downstream use.\nA new \"multi-turn programming benchmark\" (MTPB) is presented as well, in which models have to generate larger programs step-by-step, with natural language prompts describing each step.\nThe new models (and some baselines) are evaluated on code generation tasks. First, results on the HumanEval dataset show that the new models are competitive with the state of the art.\nResults on the new MTPB dataset again indicate that the newly-trained models are competitive, and show that the new setting allows all models to synthesize more complex programs.", "summary_of_the_review": "The core contribution here is the release of the CodeGen models, for which alone I'd recommend acceptance of the paper. I find the MTPB dataset to be largely uninteresting, as it relies on very fine-grained natural language supervision. It's not clear to me whether it adds value over the Django dataset of Oda et al.\n", "strength_and_weaknesses": "* (+) Training and releasing the models is in itself forms a substantial contribution to the field, enabling research groups with more limited resources to experiment on top of sound base models of significant sizes.\n* (+) The multi-turn dataset is a valuable contribution towards practical neural program synthesis applications.\n* (~) MTPB does not allow for \"going back\" (i.e., a user indicating that a generated response is wrong & clarifying their prompt).\n* (-) MTPB (from a cursory qualitative evaluation of examples) seems to rely on very fine-grained prompts. In many examples in App F, the prompts are substantially longer than the required code.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%