Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents several large autoregressive language models trained using large datasets scraped from the web, and in particular datasets of code from GitHub. The authors evaluate the models on tasks of generating code from natural language. First, the authors show state-of-the-art results on the HumanEval dataset and report that larger models also demonstrate lower perplexity on the prompt (the natural language program specification). The authors then describe the Multi-Turn Programming Benchmark (MTPB), in which the natural language intents are interleaved with code and each intent describes what the immediately subsequent code should do. Compared to a baseline where all the natural language is provided to the model concatenated together, the multi-turn version shows improvement.
strength_and_weaknesses: # Strengths
- The paper reports state-of-the-art results for the HumanEval benchmark for code generation from natural language specifications.
- The authors provide downloadable checkpoints for the trained models, which can be a useful resource for the community.
- The paper introduces a multi-turn natural language to code dataset where the natural language instructions are interleaved with the intended generated code. 

# Weaknesses
- The model is evaluated only on one existing dataset, HumanEval, and not other similar datasets like APPS or MBPP.
- While the work represents a significant expenditure of resources, the amount of technical novelty is relatively low, and some of the new aspects in the construction of the model itself are not studied or described as much as they could be:
  - The \textsc{BigPython} dataset seems responsible for much of the gains in the work, but the description of it in the paper is limited to "We have compiled public, non-personal information from GitHub consisting of permissively licensed Python code in October 2021".
  - There is significant space dedicated to describing the TPU-based training setup with JAX, but not much about the reasoning behind it.
- The paper doesn't have much experiments about hyperparameters and training data selection (e.g. the order in which the datasets were used for training), or further information/justification for the values used. 
- There is no release of training datasets.
- The deduplication methodology used during dataset preprocessing may be too weak. Only removing exact duplicate files may still leave many files which are very similar (for example, varying only in some comments at the top of the file). For more discussion on another code dataset, see https://twitter.com/miltos1/status/1497126435261083649
- Considering that copies of the HumanEval dataset may have been posted to GitHub, the authors should investigate the possibility of training dataset contamination.  
clarity,_quality,_novelty_and_reproducibility: # Clarity
I did not realize until quite late that the $\textsc{CodeGen-Multi}$ and $\textsc{CodeGen-Mono}$ models are trained on, respectively, The Pile plus the BigQuery dataset, and The Pile plus BigQuery plus BigPython. That fact is only mentioned in Section 4.3 and in the appendix, and not where the models are initially described (Section 2.1 and 2.2). This can be clearer in Section 2.

# Originality
The multi-turn dataset introduced in the paper seems to have quite a bit of similarity to SPoC (from Kulal et al., 2019), although SPoC is generally at a lower level of granularity. A more detailed discussion about it would have been helpful.

# Quality
- The full model performs well on the HumanEval dataset, and performance scales with increasing data and model size, showing that the model training worked successfully. 
- It is unclear why Table 2 only has information about $\textsc{CodeGen-Mono}$ and not the other models. More data here would be more convincing.
- For the multi-turn dialogue data, the instructions for subsequent turns only make sense if the previous turns generated correct code (or at least code in line with the authors' intent). As such, it would be useful to see how well the model does when all the output for all previous turns are correct.
summary_of_the_review: I think the model snapshots from the paper are a highly useful contribution to the community and I based my recommendation largely on that point.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: A family of large decoder-only Transformer models fine-tuned on code data are presented and released for downstream use.
A new "multi-turn programming benchmark" (MTPB) is presented as well, in which models have to generate larger programs step-by-step, with natural language prompts describing each step.
The new models (and some baselines) are evaluated on code generation tasks. First, results on the HumanEval dataset show that the new models are competitive with the state of the art.
Results on the new MTPB dataset again indicate that the newly-trained models are competitive, and show that the new setting allows all models to synthesize more complex programs.
strength_and_weaknesses: * (+) Training and releasing the models is in itself forms a substantial contribution to the field, enabling research groups with more limited resources to experiment on top of sound base models of significant sizes.
* (+) The multi-turn dataset is a valuable contribution towards practical neural program synthesis applications.
* (~) MTPB does not allow for "going back" (i.e., a user indicating that a generated response is wrong & clarifying their prompt).
* (-) MTPB (from a cursory qualitative evaluation of examples) seems to rely on very fine-grained prompts. In many examples in App F, the prompts are substantially longer than the required code.

clarity,_quality,_novelty_and_reproducibility: The paper is overall clear and well-written. 

Appendix B contains most technical details to make a re-implementation believable, and the source code release supports this assumption.

* (Q1) Will the BigPy dataset be released?
* (Q2) Will MTPB be released?
* (Q3) MTPB: What is the longest intermediate piece of code required in a turn?
* (Q4) MTPB: Did you evaluate how realistic it is to assume that users can provide such close guidance, but not write the code themselves?

Minor comments:
* page 3: "models are called call the models as"
* page 3: "While it is not our focus ..." - this sentence is not grammatical, and I have no idea what point is being made here.
* Fig 1 (nit): the solution produced by the model is wrong (email addresses are much complicated than this), but on par with how most humans would solve the problem.
* Sect. 4.3: this repeats earlier (more complete) descriptions of the used models - this is not necessary.
* page 4: "performance on the MTPB improves as a function of the model size and data size" - the latter claim is unsupported, as the additional data is also more specialized to the problem setting (i.e., more Python-specific). Please rephrase.
summary_of_the_review: The core contribution here is the release of the CodeGen models, for which alone I'd recommend acceptance of the paper. I find the MTPB dataset to be largely uninteresting, as it relies on very fine-grained natural language supervision. It's not clear to me whether it adds value over the Django dataset of Oda et al.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents new family of open-source code large language models called CodeGen of size upto 16B parameters. This model outperforms Codex 12B model on the HumanEval dataset. The paper also presents a new multi-turn programming dataset where the specification for a single task is divided into multiple sub-tasks so that the model can solve each sub-task independently to solve the full task.    

strength_and_weaknesses: **Strengths:**
- CodeGen model performs comparatively to Codex (12B) model and is open-sourced (unlike Codex). So it should help facilitate future research in this area of AI for code. 

- The results on how various pretraining datasets impacts the coding performance of the models is very interesting. 

**Weaknesses:**
- CodeGen model (single-turn) is only evaluated on a single dataset with only 164 problems. Why is the model not evaluated on other publicly available coding datasets such as MBPP and APPS? 

- While the multi-turn dataset helps the models generate more correct code, I am not sure about its practical use cases. Would a user be able to specify the sub-tasks in the required format (such as defining appropriate variables) as in Figure 1? 

- How does multi-turn task break down work for programs with control flow structures (for e.g. write code for binary search)? From the examples, it appears that the MTPB dataset treats small loops and branches as a single sub-task, but how can a user specify sub-tasks within a loop/branch?
clarity,_quality,_novelty_and_reproducibility: The paper is fairly well-written and reproducible with the models being publicly available. Novelty is limited since there are already several pre-training/dataset collection papers in this area. 

For the MTPB dataset, I don't fully understand the part about embedding the test case inputs within the prompts. Why was this approach chosen over HumanEval's style of learning to complete given a function definition? How do you deal with multiple test cases? How does inference and evaluation work with multiple test cases?  

It will be useful to list Codex Davinci's performance in Table 1 (for completeness, even though it is a much bigger model). Similarly, it would be interesting to look at Codex's performance on the MTPB dataset. 

summary_of_the_review: The contribution with regards to the CodeGen model is helpful for the community (since it is open-sourced and performs similar to Codex 12B model), but there is limited novelty there. 

The MTPB dataset contribution needs more explanation regarding its practical use cases and its limited capability to handle more complex programs. 

POST REBUTTAL: most of my questions are answered and I am increasing my score to a 8.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper trains large language models in sequence on three datasets, both natural language and programming language, and publishes the training library, JAXFORMER. The largest model outperforms the codex model on the HumanEval benchmark. The experiments show that the passing problem has lower prompt perplexity on the HumanEval benchmark. Some training data contains interleaved sequences of natural language comments and programs. For example, programmers often explain the functionality of programs with comments written with code. This work hypothesizes that the pattern provides weak supervision of multi-turn code generation. To further investigate this, the paper proposes the multi-turn code generation framework and constructs an open-source benchmark, Multi-Turn Programming Benchmark, to evaluate the proposed framework. For each turn in the generation, the model condition on the concatenation of interleaved past prompts and generated responses. Multi-turn generation specification invariantly outperforms single-turn specification. The evaluation result on the constructed dataset supports the proposed hypothesis. The performance difference in multi-turn and single-turn specification over levels of difficulty and models shows that the improvement is sizable except for easy problems with larger models.
strength_and_weaknesses: ## Paper strengths and contributions
**Motivation and intuition**
The motivation for multi-turn code generation is convincing. The supervision of comments and programs is helpful for multi-turn program synthesis.

**Novelty**
The idea of utilizing weak supervision of interleaved patterns is intuitive and convincing. This paper presents an effective way to make use of this idea.

**Technical contribution**
Codegen for program synthesis seems effective, especially when the user wants to generate pieces of code from input/output examples or natural language descriptions.

**Clarity**
The overall writing is clear. The authors utilize figures and tables well to illustrate the ideas. Figure 1 clearly shows the code generation process.

**Related work**
The authors clearly describe the related prior works from both the perspectives of program synthesis, large language models, and benchmarks for program synthesis.

**Experimental results**
The presentation of the experimental results is clear. Mainly, Table 4 provides understandable results showing that multi-turn specifications achieve better performance compared to single-turn specifications.

**Reproducibility**
I believe reproducing the results is possible given the clear description provided in the main paper and the appendix.

## Paper weaknesses and questions

**Code comment analysis** 
Some programmers like to write comments, while some are not. Also, different programmers write comments very differently. I suggest the authors investigate the effect of such differences. One very simple way could be to analyze the occurrence frequency of interleaved natural and programming patterns in the dataset. While it is hard to formally define meaningful comments, it would be insightful to at least calculate the document frequency of interleaved natural and programming language.

**Experiment setup**
In Table 3, the paper only compares the proposed method against GPT-Neo and GPT-J, which is not sufficient. The result will stand out to compare against Codex, the state-of-the-art program synthesis model.
clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: I vote for accepting this paper because its idea is original and contributes to this area while some improvement can be made (see above).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a family of LMs and LLMs (up to 16B parameters) trained on code generation and to be released. The impact of the release of such backbones to the community should be consequent: they are not the sole existing models for codegen, but they fill a niche between larger (and thus harder to productionize / serve for inference) and smaller (codegen) LMs. The ablations and the dataset cleaning (descriptions) could be improved further. In particular, there should be a study (it is not easy) of training dataset / eval dataset contamination, which the authors acknowledge in the rebuttal. The authors introduce a new benchmark Multi-Turn Programming Benchmark (MTPB) of 115 problem sets, with multi-turn prompts, which seems interesting. It was debated during review and rebuttal and will include strong baselines to compare on. This can be considered as a clear contribution of the paper. The evaluations that are comparable to prior work are adequate but could be improved: only on one existing dataset, HumanEval. Overall, the paper will be a strong addition to the ICLR conference.
justification_for_why_not_higher_score: There are some minor flaws, which have been partially addressed but not solved during the rebuttal period.
justification_for_why_not_lower_score: This is a sizeable contribution to the community, there is no incorrectness, we should not downplay useful contributions for lack of novelty, so this should get accepted. If wanted, the SAC can lower to Accept (Poster), but I think having this paper as a spotlight would interest enough of the ICLR community.
Decision: Accept: notable-top-25%