Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The current work presents InCoder, an alternative approach to general CLM or seq-to-seq code generation where the training regime is changed to allow code generation in arbitrary subsections of the code.
Experimentation seems appropriate, including HumanEval, DocString generation, return type prediction and variable name prediction showing model capabilities.

strength_and_weaknesses: Strengths 
A really good paper
The idea is clever, simple and straightforward. It provides a functionality that general GPT-like or full transformer models don’t.
Experimentation seems appropriate. Authors make a great effort not sticking with standard benchmarks which might not be the ideal scenario for InCoder

Weaknesses
The training regime comes with a cost and the Left to right generation, specifically on Table 11

clarity,_quality,_novelty_and_reproducibility: Table 11 might have included CODEX-2.5B (in addition to the 12B). The performance of Codex 2.5B in HumanEval surpass InCoder, but as stated in the weaknesses section it is expected given the change in the training regime

summary_of_the_review: A good idea, well implemented and wrapped up in a quality paper
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a mask infilling method to train language model for program synthesis with the ability of code infilling. The method randomly selects a span and replace it with a mask token, and place the span after the sequence as the target. The proposed method is novel, however the masking method is also used in the literature of T5 pretraining. Experiments show that the mask infilling does not degrade left-to-right performance, while significantly improve insertion capability.
strength_and_weaknesses: Strength:
- The paper is well-written. The method is well explained with examples. Experiment results are extensive with analysis.
- The proposed method is a simple but effective masking objective. Zero-shot performance demonstrates the effectiveness for infilling line completion task, as well as other related prediction tasks.


Weakness:
- From the main text, I cannot infer whether the training is done 100% with mask infilling or there are some probability that normal left-to-right data is used for training without any masking. If it is 100% mask infilling, it seems surprising to me that left-to-right is not affected at all. I would expect normal left-to-right data exist in training, like the concurrent work fill-in-the-middle (FIM) training (Bavarian et al., 2022).
- Some training details seem missing, for example, how many training tokens and number of iterations the model has gone through during the training process.
- Though there is an <EOM> token indicating the end of generation, there is possibility that the generated code after <MASK:0> will contain overlapping content as the actual right context. Did you see any such issue exist? How do you resolve such issue?
clarity,_quality,_novelty_and_reproducibility: This paper is well-writtern and is of high quality. The proposed method is novel. And I expect that the results should be easy to reproduce.
summary_of_the_review: Overall, I think this is a good paper with novel method proposed and well-performed experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a code synthesis model that is trained to perform both autoregressive left-to-right generation as well as infilling via added mask tokens. The experimental results show improved performance on various infilling related code generation tasks.
strength_and_weaknesses: The paper is well written and easy to read. The approach described tackles fundamental challenges in autoregressive generation by training  models that are capable of filling the masked regions. This ability is necessary for tasks such return type prediction and doc string generation where the output depends on both left and right context. The experimental results are also encouraging as for various tasks shown, the model is able to perform better with less model size (e.g., compared to code-davinci-001).

The paper, however, does not discuss full code generation as it focuses primarily on infilling experiments, so its competitiveness on more challenging tasks is not known. With the ability of infill and the original motivation that "Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined.", I'd be very interested to see how the model performs on full code generation as well as devising an ability to detect and re-write portions of the code when solving those tasks.
clarity,_quality,_novelty_and_reproducibility: The paper is very clear.

I think the algorithmic novelty is predictable in a sense that masking and filling mask regions have been used in other context, but it is interesting to see its results in code infilling tasks.

The paper uses custom dataset generation steps, so without exact details of those, it would be challenging to reproduce the paper.
summary_of_the_review: Despite not evaluating on full code generation task, the paper has enough experimental results to support itself for infilling tasks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach to perform code infilling that supports multiple insertion points. The approach is by defining an infilling format that describes the missing code locations and the code to be infilled. The training approach is by the traditional autoregressive next token prediction loss. This paper indeed proposes a simple technique that seems to work reasonably well.
strength_and_weaknesses: Pros:

This paper shows that a simple method works. 

Cons:

There is no baseline against the traditional left to right approach. That is, with a given amount of data / compute this work uses, what would be the performance if we use these resources for left to right completion alone? This is important because the amount of data that the paper uses as well as the training resources are slightly different from other papers, therefore, it would be good to see how much this amount of data/compute yield in terms of left to right completion. Note that this could be different from the left-to-right performance that is obtained from the infilling method proposed. That goal here would be to compare if this approach results in performance drop due to infilling or not, and if so how much. A similar work by Bavarian 2022 shows that with a similar approach but only with 1 infilling spot, there seems to be no degradation, but this work is sufficiently different that it is not clear if the results by Bavarian 2022 also applies here.

clarity,_quality,_novelty_and_reproducibility: Clarity: The approach proposed seems clearly presented. However, in the figure, I would not call these 'zero-shot' inference since the model is trained specifically to do such infilling, correct?

Quality: Missing an important comparison, as mentioned above. However, there are a lot of important evaluations (single line, multi line etc, as well as entire scope + execution based evaluation). 

Novelty: Moderate, but has a good amount of impact.

Reproducibility: This work releases model weights, so it seems reproducible (even though the code release on the inference is slightly lacking).
summary_of_the_review: Nice contribution, but lacking thorough comparison to the normal baseline / how much performance drops compared to normal left to right approach. However, the multiple infilling spots is a plus.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents an approach to learn an in-filling model for code, by training with the causal masking setting proposed in (Aghajanyan et al., 2022a).  After training the model can be used to in-fill given left and right context significantly improving its performance on a variety of code modeling tasks, including in-filling code, generating documentation given code, type inference and variable name prediction.

The proposed idea is simple and straight-forward, and comes with a range of benefits.  The paper is well-written.  All reviewers liked the paper and favored acceptance.  The fact that the causal masking setting was proposed in prior work diminishes the novelty of this work somewhat, but it is still a good paper and I fully support its acceptance, given its appropriate fit with many code related tasks and good performance.
justification_for_why_not_higher_score: The key training setting was proposed in an earlier work.
justification_for_why_not_lower_score: The proposed idea was simple, straight-forward and led to good performance on a variety of tasks.  This is the kind of paper that we want to accept and see at ICLR.
Decision: Accept: notable-top-25%