InCoder: A Generative Model for Code Infilling and Synthesis
Authors: Daniel Fried
Armen Aghajanyan
Jessy Lin
Sida Wang
Eric Wallace
Freda Shi
Ruiqi Zhong
Scott Yih
Luke Zettlemoyer
Mike Lewis
Venue: ICLR 2023 notable top 25% (2023)
Keywords: code generation
program synthesis
language to code

Abstract: Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined. We introduce InCoder, a unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via masking and infilling). InCoder is trained to generate code files from a large corpus of permissively licensed code, where regions of code have been randomly masked and moved to the end of each file, allowing code infilling with bidirectional context. Our model is the first large generative code model that is able to infill arbitrary regions of code, which we evaluate in a zero-shot setting on challenging tasks such as type inference, comment generation, and variable re-naming. We find that the ability to condition on bidirectional context substantially improves performance on these tasks, while still performing comparably on standard program synthesis benchmarks in comparison to left-to-right only models pretrained at similar scale. Our models and code will be publicly released.

Published as a conference paper at ICLR 2023
INCODER: A GENERATIVE MODEL FOR CODE INFILLING AND SYNTHESIS
Daniel Fried 910
Armen Aghajanyan +0
Jessy Lin
Sida Wang ?
Eric Wallace
Freda Shi
Ruiqi Zhong
Wen-tau Yih
Luke Zettlemoyer of
Mike Lewis"
Facebook Al Research o
University of Washington
UC Berkeley"
TTI-Chicago Carnegic Mellon University"
dfried@cs. cmu. edu, {armenag, mike com
ABSTRACT
Code seldom written in a single left-to-right pass and instead repeatedly edited and refined. We introduce INCODER, unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via masking and infilling). InCoder is trained generate code files from large corpus of per- missively licensed code, where regions of code have been randomly masked and moved to the end of each file, allowing code infilling with bidirectional context. Our model is the first large generative code model that is able to infill arbitrary re- gions of code, which we evaluate in a zero-shot setting on challenging tasks such type inference, comment generation, and variable re-naming. We find that the ability to condition on bidirectional context substantially improves performance these tasks, while still performing comparably standard program synthesis benchmarks in comparison to left-to-right only models pretrained at similar scale. Our models and code are publicly released]
INTRODUCTION
Large language models trained on vast repositories of code have demonstrated remarkable progress in neural program synthesis and related tasks (Chen et al.) 2021a| Austin et al.] [021] Xu et al., 2022 Nijkamp et al. [2022] Chowdhery et al., 2022). However, such models generate code left- to-right, which makes them less directly applicable to many ubiquitous code editing tasks, such as fixing bugs, adding comments, or re- naming variables. We introduce INCODER, unified model for program synthesis and editing. Like prior work, INCODER is trained to maximize the likelihood of corpus of code. However, we adopt a causal masking objective (Aghajanyan et al., (2022a), allowing INCODER to infill blocks of code conditioned on arbitrary left and right contexts.
More specifically, we learn to infill by randomly replacing spans of code with sentinel token and moving them the end of the sequence (Figure 1] top). The model is trained to predict all tokens in the complete sequence in this permuted ordering. During inference, spans with sentinel tokens, prompting the model with the new sequence, and having it generate new tokens to replace the masked spans (Figure I bottom). Because the model can also trivially generate without sentinel tokens, the result is a unified approach for both program synthesis (via left-to-right generation) and editing (via infilling).
can edit code by replacing
We evaluate performance on a range of zero-shot code infilling tasks (Sec. 4). both new and from existing work, including challenging cases such as type prediction, variable re-naming, com- ment generation, and completing missing lines of code. Zero-shot infilling with bidirectional con- text substantially outperforms approaches based left-to-right-only models, and several tasks obtains performance comparable to state-of-the-ar models fine-tuned on the tasks. Ablation experi- ments (Sec. (5) show that this does not come at the cost of left-to-right generation ability; our causal masking model achieves similar performance to a standard language model on program synthesis benchmarks (Chen al. 2021a Austin al. 2021) despite its more general training objective.
Equal contribution

---

Published as a conference paper at ICLR 2023
Training
Original Document
Masked Document
def cosnt conn__yrrdcctleeeeee strp Pictistr, int)
def coust, (iVlonne Str1 oP DiCtEste. 18515
-*nCouet the narter accorrences of each werd the file, THe th oeennt. "r") f:
"-"Ceunt the rimber eccurrences each Hore in the file
with opertfilerams. 'r1) as ft
porc_saani
<PASCI ward, counts:
for line
vort courtsfverd) in
vord Line, spliitte
else:
ware_ceents vond_counts (worol
vors countstuend)
rature courte
<HASR:0> uard, (0
vont_conts//vrrd
Lire
raturn coursssurss
Tlar sp <Fat>
Zero-shot Inference
Type Inforence
Variable Name Prodiction
crsiennee DIClEstr, --"Count the "r's
file."
werd_counts
jeond_coust:
sore_conntstennd
verd]
Docstring Generation
Mutti Region Infillnn
def count velde:t str) + Bictjesr, Ind]:
Free collections lepert Counter
Counts the ruaoer of cocarrences of each sore to the gives file
""rooutt Uhe nanter accorrences of each ward the fite. wores
sarae "Ileramn: The rase of the file to court.
cretura: stetrenare masging wonte to the suaber of securrences
of Fi18S
for Tine rites
whth Stencf "r') f: werd ccunts +.
Hords www.nddiiie
return Deunterrvenda)
for Lire 14 11
for wore in Thne SpL If vord coutsco connt leore) +
rise:
serd (eno_
retern count
At training time (top), our causal masking objective samples one more spans of code in training documents (in the upper left figure, a single span) and moves these spans to the end of the document, with their original location denoted by special mask sentinel tokens. An autoregressive language model is trained to produce these entire masked documents, allowing it to learn to generate insertion text conditioned on bidirectional context. At inference time (bottom), we can perform a variety of code editing and infilling tasks in a zero-shot fashion by inserting mask tokens at desired locations and allowing the model to generate code to insert there. All examples shown are real out- puts from our INCODER-6. 6.7B model, with the regions inserted by the model highlighted in orange.
Figure 1:
2 INFILLING AND SYNTHESIS VIA CAUSAL MASKING
Neural models for code generation have either utilized left-to-right (causal) autoregressive lan- guage modeling objective (Brown et al., 2020 Chen et al., (021a)) or, as BERT does, a masked lan- guage modeling objective (Devlin et al. 2019 Feng al. 2020). Both approaches have strengths and weaknesses. Causal models only condition on context to the left of the generated tokens, thus preventing infilling, but they can autoregressively generate entire documents. On the other hand, masked language models can condition on both the left and right contexts to infill a masked region, however, their training objective typically limited to generating only about 15% of document. In this paper, we adopt the recently proposed causal masking objective (Aghajanyan et al. 2022a), which aims to combine the strengths of both causal and masked language models.
2.1 TRAINING
At training time, the causal masking procedure samples a number of spans of contiguous tokens in each document to mask (Figure I] top left). We sample the number of spans from a Poisson distribution with mean of one, truncated to the support [1, 256], so that there are typically a small
2

---

Published as a conference paper at ICLR 2023
number of spans (with single span around 50% of the time), but the distribution has long tail (up to 256 spans). Each span's 's endpoints are sampled uniformly from the length of the document and the set of sampled spans is rejected and resampled if any spans overlap.
Once spans are sampled, each span & is replaced with a special mask sentinel token, <Mask :>. The sequence of tokens in the span is then moved to the end of the document (Figure top right), with the mask sentinel token prepended and a special end-of-mask token <EOM> token appended. In other words, when mask token appears for the first time in the left-to-right ordering, it marks the location the span was removed from; when it appears for the second time, it marks the start of the moved span text. More formally, assume we have a document D with N tokens, and we have sampled one span Span = Daj. Let Left be the left context Dos and Right be the right context D Dj:N- Then, we maximize the log probability of the masked document:
log P[[Left; <Mask: 0>; Right; <Mask: Span; <EOM>])
(I)
where: ; denotes sequence concatenation. If more than span sampled, each would be similarly appended at the end of the document in order. As in standard left-to-right generative language modeling, we compute the probability of the sequence auto-regressively and train the model using cross entropy loss on all tokens except the mask sentinel tokens <Mask: k>, so that the model does not generate these tokens during inference.
2.2 INFERENCE
During inference, the model can either be used for left-to-right generation in the standard way (by sampling autoregressively from the model, without using any special tokens), can insert code at arbitrary locations in an existing document by inserting a <Mask: k> tokens at the desired location(s) and continuing generation at the end of the document. Assuming for simplicity of notation that we want to insert text at only a single location, we can generate span to insert between the location's 's Left and Right context sequences by sampling tokens autoregressively from the distribution
P( [Left; <Mask: 0>; Right; <Mask: 0>])
until either an <EOM> token is generated or a task-dependent stopping criterion is achieved When applicd to code, this allows us to perform tasks that benefit from the bidirectional context a zero-shot fashion, shown in Figure I] bottom. For example, generation conditioned on both the left context (function signature) and right context (function im- plementation). We can also infill multiple dependent regions, e.g., generate imports required by function that the model is generating. See Section B.2 for details, including multi-region infilling.
perform Python docstring
SeepSection
MODELS
Our primary model is INCODER-6.7B, 6.7B Transformer (Vaswani et al. (017) language model. We use the same architecture as the dense 6.7B models described in Artetxe et al. 2)21); the Fairseq architecture description can be found Table in the appendix. All experiments use this model unless stated otherwise (we train smaller models for comparison in Section 5).
To train our models, we collect a corpus of (1) public code with permissive, non-copyleft, open- source licenses from GitHub and GitLab and (2) StackOverflow questions, answers, and comments. Our primary focus in this paper is on the Python language. but we also include code files from 28 total languages and StackOverflow content from all available languages. We decontaminate our pre-training corpus by removing all datasets which we use in our evaluation experiments. See Section A. for details. Our final pre-training corpus contains a total of 159 GB of code, 52 GB of it in Python, and a total of 57 GB of content from StackOverflow. See Figure for size by language.
INFILLING EXPERIMENTS
Our primary evaluation is performing zero-shot infilling for diverse set of tasks: inserting lines of code, predicting function return types, generating docstrings, renaming variables, and inserting missing code tokens. We formulate each task filling in one more masked-out regions of code.
In practice, use variation of this method that inserts an extra sentinel token into the context to counter hias infill length caused by the Transformer's fixed context window size. See SecfScction B.2
3

---

Published as a conference paper at ICLR 2023
To evaluate how INCODER benefits from bidirectional context when generating infills, compare three different inference methods: the causal masking inference procedure described in Section 2 standard left-to-right generation approach (left-to-right single), and left-to-right generation and reranking approach (left-to-right reranking). Since our model is also able to generate left-to-right, can compare all three inference methods using the same INCODER-6.78 model and thus avoid any confounding effects due to a change in the model. For all three inference methods. we obtain generations from the model using top-p (nucleus) sampling (Holtzman al. 2020) with p 0.95
and a temperature tuned for each task and inference method using the task's 's development data g
Left-to-right single. This baseline does all. It generates a single completion for the location by conditioning on the left context and sampling tokens autoregressively from the model P(- Left) until task-specific stop condition is reached (e.g., for comment generation, when a comment comment-ending ending delimiter is produced).
the context to the right of the masked location at
Left-to-right reranking. This baseline uses only the left context to propose candidates to infill the blank, but uses both the left and right contexts to choose among these candidates. Concretely, we first generate K possible completions for the blank region, Span, ... Span following the same procedure as left-to-right single, using K 10 unless otherwise specified. We then evaluate each candidate by substituting it into the blank and scoring the completed document. We use either total log probability of the completed document log P[[Left; Spang; Right]) or, following Chen et al. (202Ta), log probability averaged across the number of tokens in the completed document. We select between these two scoring methods for each task using performance on the task's development data.
4.1 INFILLING LINES OF CODE (HUMANEVAL)
We create an infilling benchmark for complete lines of code from the HumanFval dataset (Chen al., [021a). This dataset provides comment descriptions of functions paired with a canonical implementation of each function and several input-ouiput pairs that the function should pass. Hu- manEval was introduced as a benchmark for the synthesis of entire Python functions; we evaluate models this original synthesis setting in Section C.6) We use this dataset because it affords functional testing of completed code (as opposed to relying solely on an evaluation of the code sur- face form), which is particularly important when infilling longer regions that have more potential ways to be completed correctly. We construct two infilling tasks from the dataset, for single lines and multiple lines:
Single-line infilling. In this task, we mask out each non-blank line of code in the canonical func- tion implementation turn (creating examples for a function with N non-blank lines). The task is to generate a single-line completion for the blank conditioned on the natural language description of the function and the code lines before and after the blank. We evaluate using (1) pass rate: the rate at which the completed function passes all of the function's input-output pairs (i.c., analogous to the pass@) metric from Chen et al. (221aa) and (2) exact match: percentage of times that the completed lines exactly match the masked lines in the canonical implementation. Performance is averaged across all examples generated for all programs in the dataset.
Multi-line infilling, This task is constructed in the same way as single-line infilling above but allows each masked region to contain multiple lines of code, creating N x (N + 1)/2 examples for a function with N non-blant lines. We again evaluate completions using pass rate and exact match, averaged across all infilling examples.
Inference details. To choose when to end the infill produced by our inference methods, we trun- cate the candidates generated by the left-to-right (L-R) baselines to the actual number of lines in the blanked-out region. For our causal-masked (CM) infilling method, we end the infill when the model generates the <EOM> token. For the L-R single and CM infilling methods, a temperature of 0.2. For the L-R rerank method, we use a temperature of 0.8 to sample K = 10 candidates and rescore with the total log probability of the completed function.
sample using
*For all generation experiments, we prefix prompts with meta-data indicating the code generated should be Python; see Section A.3) for meta-data details.
4

---

Published as a conference paper at ICLR 2023
Method
Pass Rate
Exact Match
Method
Pass Rate
Exact Match
L-R single
48.2
38.7
L-R single
24.9
15.8
L-R reranking
54.9
44.1
reranking
28.2
17.6
CM infilling
69.0
56.3
CM infilling
38.6
20.6
PLBART
41.6
PLBART
code-cushman-00)
53.1
42.0
code-cushman-000
30.8
17.4
code-davinci-001
63.0
56.0
code-davinci-001
37.8
19.8
(a) Single-line infilling.
(b) Multi-line infilling.
Table On our single- and multi-line code infilling benchmarks that we construct from HumanEval, our causal- mmakedd (CM) approach obtains substantial improvements over left-to-right single candi- date and left-to-right reranking baselines in both function test pass rate and exact match.
Single-Line Infilling
Multi-Line Infilling
0.8
CM Infilling
Single
0.7
0.6
Reranking
Rate
0.6
Pass Rate
0.5
0.5
0.4
0.4
0.3
0.3
CM Infilling
L-R Single
0.2
0.2
L-R Reranking
0.1
0.1
0.2
0.6
0.8
Fraction of Lines in Right Context
Fraction of Lines Right Context
Figure 2: Infilling pass rate by the fraction of the function's lines which are provided to the right of the region that must be infilled, for single-line infilling (left) and multi-line infilling (right). Shaded regions give 95% confidence intervals, estimated using bootstrap resampling. Our causal-masked (CM) infilling method, blue, consistently outperforms both of the left-to-right (L-R) baselines, with larger gains more right-sided context becomes available (the right side of both graphs)
Results. Table shows the results for the single-line (left) and multi-line settings (right). In both settings, CM infilling improves substantially over the L-R single baseline and the L-R reranking baseline. Note that these results computed by averaging over all examples, which includes masked regions at all positions functions (including the beginning, when no left context is avail- able, and end, when right context is available). [Figure 2 shows finer-grained comparison, where we group examples by the fraction of lines in the canonical function which are contained in the context the right of the infill. The CM infilling method sees larger improvements over the L-R baselines more right-sided context becomes available (i.e., when the blanked region occurs earlier in the function)
We also compare against two alternate zero-shot methods for incorporating right-sided context: (1) mecoder-decoder code model trained with denoising infilling objective (PLBART, A.e.ARR,,haadd Ahmad al. 2021), and (2) templated prompting of large left-to-right generative code models (the cushman-001 and davinci-001 Codex models available through OpenAl's API). See [Section C.I for details these experiments InCoder outperforms all models in both single-line and multi-line infilling, de- spite having lower performance in left-to-right generation than Codex (seel Table IT}, demonstrating that causal masking training benefits infilling.
FFor multi-line infilling, performance increases with increasing amounts of right context, as having large right context implies that fewer lines are available be removed when constructing infill examples, the resulting examples are often easier infill. "PL BART outputs tokenized eode, which prevents from computing meaningful exact match metrics.

---

Published as a conference paper at ICLR 2023
Method
BLEU
Ours: L-R single
16.05
Ours: L-R reranking
17.14
Ours:
Causal-masked infilling
18.27
RoBERTa (Finetuned)
18.14
CodeBERT (Finetuned)
19.06
PLBART (Finctuned)
19.30
CodeT5 (Finctuned)
20.36
Table 2: CodeXGLUE Python Docstring generation BLEU scores. Our model evaluated a zero-shot setting, with fine-tuning for docstring generation, but it approaches the performance of pretrained code models that are fine-tuned on the task's s 250K examples (bottom block).
4.2 DOCSTRING GENERATION (CODEXGLUE)
We next evaluate documentation string (docstring) generation, where models must generate a nat- ural language docstring that summarizes Python code snippet. Right context may be particularly useful for docstring generation, as conditioning on the function body can allow models to generate more informative descriptions. Prior neural code generation models are fine-tuned on supervised docstring-cods code pairs to perform this task (e.g., Clement et 2020 Chen et al. [021a Lu et al. 2021 Ahmad et 2021), however we evaluate our model zero-shot, with no explicit supervision.
We use the CodeXGLUE code-to-text docstring generation task (Lu et al. 2021), which is con- structed from CodeSearchNet (Husain al., 2019), consisting of doestring-code pairs scraped from publicly available GitHub repositories. The L-R single candidate baseline is prompted with the function signature in the left context preceding the docstring. The CM infilling and L-R reranking methods also observe the right context, consisting of the function body.
We compare models following the original automatic evaluation setup for the task. In Table 2 we report smoothed 4-gram BLEU scores for all models, using the reference docstrings provided in the dataset. These references have been preprocessed to strip extraneous content (e.g., argument definitions) from the original scraped docstrings. We use greedy generation for the CM infilling and L-R single candidate generation methods and sample K 10 candidates at temperature 0.8 with average log probability scoring for the L-RR reranking method (selected by tuning the validation set of the task). For all inference methods, we stop generation if the model generates a newline. We also include the performance of the supervised baseline from the CodeXGLUE paper: an encoder- decoder model with a Code encoder fine-tuned on ~ 250K training examples from the dataset. Our zero-shot performance approaches the performance of the fine-tuned CodeBERT model.
4.3 RETURN TYPE PREDICTION
Predicting return type hints for Python functions is a challenging structured generation task (see Figure I, "type inference"). We evaluate on two datasets: one we construct from CodeXGLUE and the dataset from TypeWriter OSS (Pradel al. 2020
CodeXGLUE. We develop benchmark for return type prediction using the same Python CodeXGLUE dataset used in the code-to-text (docstring generation) task. We run an abstract syntax tree (AST) processor on all functions in the development and test sets of this dataset to (1) identify functions with PEP 484 return type hint annotation that is not None and (2) remove all other type hints (e.g., for function arguments and variable declarations) from the function. This leaves 232 functions in the development and 469 functions in the test set.
The task to condition on the function signature and body and predict the type hint. We compare the type hints predicted by our various methods the annotated type hint in the original function. using exact match accuracy on the normalized type hint
htttps://peps prttson..yoo...mmmm org/ htce/ppp-040 0484/
We normalize each type hint by first parsing the type AST and then un parsinsiin the AST to surface form string, then compute exact match these surface forms. We note that this metric is somewhat noisy, given

---

Published as a conference paper at ICLR 2023
Method
Accuracy
Method
Precision
Recall
F1
Left-to-right single
12.0
Ours: Left-to-right single
30.8
30.8
30.8
Left-to-right reranking
12.4
Ours: Left-to-right reranking
33.3
33.3
33.3
Causal-masked infilling
58.1
Ours: Causal-masked infilling
59.2
59.2
59.2
(a) Results on the test set of the benchmark that we construct from CodeXGI.UF.
TypeWriter (Supervised)
54.9
43.2
48.3
(b) Results on subset of the TypeWriter's OSS datasct (Pradel et al. (200)). We include examples from which we were able to obtain source files, successfully ex- tract functions and types, that have non-None return type hints, and that were not included in our model's training data.
Table Results for predicting Python function return type hints on two datasets. We see substantial improvements from causal masked infilling over baseline methods using left- left-to-right inference.
To compare our three generation methods, we stop generation when : is generated, which ends the type hint and signals the start of the function body. We tune inference hyperparameters on the development set, and we use temperature of 0.2 for left-to-right-sinnle. 0.8 for left-to-right reranking, and greedy generation for causal masked infilling. Results on the test set are given in Table 3al Conditioning on the right context (i.c., the function body) gives some benefit in the left- to-right reranking setting, but gives substantial improvement via our causal masked infilling.
TyptWriter OSS. Some recent work has developed supervised machine learning approaches for predicting type annotations for dynamically-typed languages including Python (Xu et al.. 2016 Allamanis et al., 2020 Pradel et al., 2020) and TypeScript (Hellendoomn et al., 2018] Wei et al,, 2020 Jesse et al. 2021) We compare our zero-shot model one such approach for Python, Type- Writer (Pradel et al. 2020). which combines a neural architecture for type hint prediction with a search-based incremental type validation procedure.
To compare to the supervised TypeWriter approach, we obtain its predictions on the open-source software (OSS) dataset used in that work (Pradel et al. (2020), consisting of Python functions from GitHub. Unfortunately, we could not evaluate on their full evaluation set since much of it was in- cluded in our model's training data. We filter to instances that were not included our training data, for which we were able to obtain files and extract functions and types from via AST parsing, and which have non-NONE return type hints. This leaves 2,092 examples (about 12% of their evaluation set). We otherwise emulate their exact setup, which allows our model to condition on file imports, the function body, and the function signature to predict return type hints. We use the same inference hyperparameters as we did for CodeXGLUE type hint prediction.
We present our results in two tables:
Table 3b containing metrics across non-None types, and Ta-
ble 10 the Appendix, which includes None types as well (following Pradel et al. 2020) We again see benefits from causal masked infilling's s ability to condition on the function body when generating return types, and find that our zero-shot model outperforms the supervised TypeWrite Writer model.
4.4 VARIABLE NAME PREDICTION
Variable name prediction is a less-constrained code generation task that requires modeling bidirec- tional context. We again use the test set from the CodexGlue code-to-text task (docstring generation) and run an AST transform to isolate and either mask all the occurrences of the variable name (in- filling) take the left-most context from the first variable name (left-to-right mode). In the infilling setting, given that we generate the number of masks equivalent to the number of times a variable seen, we select the most common prediction as our singular prediction. Furthermore, we only evaluate the set of variable names containing four or more characters. For our re-ranking, we con- sider candidate set of 25 variables. We present our results in Table 4 We again see substantial benefits from using both left and right context: left-to-right reranking and causal- infilling both outperform the left-to-right single baseline (which uses only the left context). Causal-masker
that numma--annotatee type hints can be inaccurate, and that exact match does not reason about type unification or equivalence (c.g., there is partial credit given for predicting ontionaa[sstr rather than Uni NNnne, str]).
"Our model makes a prediction for every example, has identical precision, recall, and F1 scores.

---

Published as a conference paper at ICLR 2023
Method
Accuracy
Left-to-right single
18.4
Left-to-right reranking
23.5
Causal-masked infilling
30.6
Table 4: Results the variable renaming benchmark that construct from CodeXGLUF. Our model benefits from using the right-sided context in selecting (I--R reranking and CM infilling) and proposing (CM infilling) variable names.
infilling substantially on the left-to-right reranking method, demonstrating the value of conditioning on the right context when proposing candidate completions.
ABLATION EXPERIMENTS
For analysis of the effects of training model with causal masking (rather than the standard language modeling objective, as well as model size and the training data, we train several variations of our model. We compare model pass @ I scores the HumanEval (Chen et al. 2021a) and MBPP (Austin et al. 2021) left-to-right synthesis benchmarks, with results in Table
Objective. Comparing 1.3B parameter models trained on the same training data with the causal masked (CM) objective (row 2) and the standard left-to-right language modeling (LM) objective (row 3). we see that the causal-masiee and MBPP tasks in pass @ score. This provides further evidence that causal masking training does hurt the model's s ability to perform standard left-to-right generation, least the 1.3B parameter scale, in line with the findings of Bavarian et al. (2022).
model obtains slightly higher performance on the HumanEval
Model size. With data fixed, increasing model size consistently improves performance (comparing the 6.7B and 1.3B CM models in rows and 2, and the 1.3B and 2.3B LM models in rows 3 and 6).
Effeets of data. We compare models trained on our entire dataset of multiple code languages and StackOverflow (multi lang SO, deseribed in [Section A.I) to data ablations that train only on Python code files and StackOverflow (Python + SO) and only Python code files (Python). We find that training multiple languages gives slight reduction in performance evaluations. However, comparing rows 4 and 5, we see that including StackOverflow data in training substantially improves performance on both HumanEval and MBPP. This suggests that future work on generative code models for or other corpora that mix natural language and code as training data.
these Python
synthesis tasks should consider using StackOverflow
QUALITATIVE EXAMPLES
We show a variety of qualitative examples from model in Section D.2 in both the infilling and left-to-right generation modes: docstring generation, metadata conditioning, class attribute inference from class usage, comment-conditioned code editing, StackOverflow title and tag generation, and zero-shot bidirectional translation of technical jargon between Chinese and English.
Size (B)
Training Data
Data Size
Train Tokens
Train Compute
HumanEval Pass 1
MBPP Pass@
Obj.
1)
6.7
CM
multi lang SO
204 GB
52 B
3,0Z
15
19.4
2)
1.3
CM
multi lang SO
204 GB
52 B
0.6 Z
8
10.9
3)
1.3
LM
multi lang SO
204 GB
52 B
0.6
8.9
4)
1.3
LM
Python SO
104 GB
25
0.3
9.8
5)
1.3
LM
Python
49 GB
11 B
0.1
6.1
6)
2.3
LM
multi lang So
204 GB
52 B
1.1
12.7
Table 5: Ablation results, comparing model performance on the Python portion validation set held out from our training corpora as well as the HumanEval and MBPP benchmarks. We compare models by size (in billions of parameters), objective (causal masked, CM, versus standard left-to- right language modeling, LM), training data, and total amount of compute in training (in zettaflops).
8

---

Published as a conference paper at ICLR 2023
7 RELATED WORK
Language Models for Code There has been a flurry of recent work on training large-scale neural language models on source code. Existing models differ in their architectural design and training objectives, e.g.. decoder-only language models (Austin et al.) [022 Chen et al.
2021a a Izadi et al,,
2022
Xu et al., 2022; Nijkamp et al. (022), encoder-only masked language models (Feng et al.
2020
Kanade et al. (2220), and encoder-decoder models (Ahmad et al. [022]] Li et al. 2022] Roziere
al.
2021 Wang al. 202T). Decoder-only language models have grown in popularity
they
can perform zero-shot program synthesis by generating in a left-to-right fashion. On the other hand, InCoder is a decoder-only mausally-masker language model that can infill arbitrary spans of text. This allows the model to perform program synthesis and many other code infilling tasks.
Infilling Models Many weal-world applications require infilling sequences using left and right con- text, e.g., editing sentences (Shih et al. 2019), restoring ancient text (Assael et al. (2019), and fixing bugs source code, Unfortunately, standard stantardleff-to-rigglh left-to-right language models cannot directly infill text, and popular masked language models are mainly trained to infill very short spans (Chan al. 2019 Devlin et al.] 2019; Raffel et al.] 2020; Roziere et al. (021]. Recent work addresses this by chang- ing model architectures, inference procedures, and training objectives (Aghajanyan
al. 2022a
Stern et af. 2019: West et al. 2021 Aghajanyan et al., 2022b). Most related to our approach h is the work of Donahue et al. 2020 and CM3 (Aghajanyan et 20222,,,a), who train left-to- right language models to fill in masked token segments of varying lengths; and the work Alon et al. (2020), who train an infilling-capable, AST-structured generative model of code on a smaller scale. In addition, concurrent to our work, OpenAl developed
in-the-middly (FIM) training objective similar to the causal masking objective we use, trained code models with it, and evaluated on the HumanEval infilling tasks we introduce here (Bavarian et al.) 2022). Similar to our findings in [Section 5] they find that the infilling capability does not adversely affect left-to-right performance. Our objective, in contrast, allows infilling multiple regions of code, and we demonstrate the benefits of infilling across a broader range natural programming tasks.
Machine Learning for Code Assistance There is an extensive literature on using machine learn- ing models to aid human programmers. This includes methods to infer variable types (Pradel et al.
Wei et al., 2020), generate unit tests (Fraser & Arcuri| (011), repair programs (Gupta al., Yasunaga & Liang 2020; Chen et al., 2021c| Yasunaga & Liang 2021), and verify program correctness (Ryan al.] 2020). Our model can infill arbitrary spans of code, allowing it complete many of these tasks, as well as perform standard left-to-right generation, in single approach.
2020
2017
Machine Learning for Program Synthesis Program synthesis approaches directly generate pro- grams from specification of functionality (Gulwani et al. 2017). Such models work by taking e.g., input-output examples (Balog et al., [017]] Gulwani]
2011] Chen al., 2021b Bavishi al., 2019). partial implementations (Solar-Lezama Lezama et al. 2006), or natural language descriptions Zeele & Mooney [996] Yu et al., [018 Yin al.] 2018 Kulal al.) 2019) Chen et al. 2021a) of the desired program as input. Our InCoder model differs from this past work as it can both synthesize and infill arbitrary spans of code, conditioning on natural language and partial implementations.
CONCLUSION
We demonstrated that using causal masking objective when training generative model of code enables strong zero-shot performance on many challenging and practical code infilling and editing tasks. The model's additional infilling capability does not appear to harm its ability to do standard left-to-right generation: ablation and comparison experiments show that our causal- masked models have comparable performance to similarly-resourced models on standard left- left-to-right language-to- code synthesis benchmarks. Looking forward, we expect our model performance to continue to increase with more parameters, data, and training steps (Kaplan et al.) 2020 Henighan et al. 2020).
Moreover, fine-tuning would allow our models to be better able to condition on natural language instructions and other indications of human intent (Zhong et al. 2021 Wei et al., 2022 Ouyang
et al. 2022). Finally, model lays foundation for future work supervised infilling & editing via model fine-tuning, as well as performing iterative decoding, where the model can be used to refine its own output (Ghazvininejad al. 2019).

---

Published as a conference paper at ICLR 2023
REFERENCES
Rajas Agashe, Srinivasan lyer, and Luke Zettlemoyer. JulCe: A large scale distantly supervised dataset for open domain context-based code generation. Proceedings of EMNLP, 2019. URL
https://aclanthoooggy 1rt.hh/0ooo..gr//111554444
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. CM3: A causal masked multimodal model of the Internet. arXiv preprint 2rXiv:2001.07200, 2022a.
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke Zettlemoyer. HTLM: Hyper-text pre-training and prompting of language models. In ICLR, 2022b.
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for program understanding and generation. In NAACL, 2021.
Miltiadis Allamanis. The adverse effects of code duplication in machine learning models of code. In SPLASH, 2019.
Miltiadis Allamanis, Earl T Barr, Soline Ducousso, and Zheng Gao. Typilus: Neural type hints. In PLDI, 2020.
Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In ICML, pp. 245 256.. PMLR, 2020.
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victo- ria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:21 10684, 2021.
Yannis M. Assael, Thea Sommerschield, and Prag. Restoring ancient text using deep learning: case study on Greek epigraphy. In EMNLP, 2019.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2 108. 07732, 2021.
Mandeep Baines, Shruti Bhosale, Vittorio Caggiano, Naman Goyal, Siddharth Goyal, Myle Ott, Benjamin Lefaudeux, Vitaliy Liptchinsky, Mike Rabbat, Sam Sheiffer, Anjali Sridhar, and Min Xu. FairScale: A general purpose modular PyTorch library for high performance and large scale 2021.
training. https:
com/ ceceebbookreseerc//aaaiiii
Matej Balog, Alexander L. Gaunt, Mare Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. DeepCoder: Learning to write programs. In ICLR, 2017.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint wwx.v:2207.14255. 2022.
Rohan Bavishi, Caroline Lemieux, Roy Fox, Koushik Sen, and Ion Stoica. AutoPandas: neural- backed generators for program synthesis. PACMPL, 2019.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivan- shu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. 2022.
Burton H Bloom. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 1970.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. In NeurlPS, 2020.
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. KERMIT: Genera- tive insertion-based modeling for sequences. arXiv preprint arXiv: 2xXiv/9900 1906.01644 2019.
10

---

Published as a conference paper at ICLR 2023
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021a.
Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, and Denny Zhou. SpreadsheerCoder: Formula prediction from semi-structured context. In ICML, b021b.
Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Monperrus Martin. SequenceR: Sequuence-to-equeeeee learning for end-to-end program repair. IEEE Transactions on Software Engineering, 2021c.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.03311, 2022.
Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. PyMT5: multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pp. 9052-9065, Online, November 2020. Association for Computational Lin- guistics. doi: 10.116533v1/2020.coom www.:000eemmlp--pmmm main.728. URL https://aclantholle http:/hhooogoooggg/aeeoo org/2020.
emnlp-main moin.main... 728
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Chris Donahue, Mina Lee, and Percy Liang. Enabling language models to fill the blanks. InAC 2020.
Zhangyin Feng. Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. CodeBERT: A pre-trained model for programming and natural languages. EMNLP Findings, 2020.
Gordon Fraser and Andrea Arcuri. EvoSuite: Automatic test suite generation for object-oriented software. In ACM SIGSOFT, 2011.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-Predict: Parallel decoding of conditional masked language models. In EMNLP, 2019.
Sumit Gulwani. Automating string processing in spreadsheets using input-output examples. ACM SIGPLAN Notices, 2011.
Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. Program synthesis. In Foundations and Trends in Programming Languages, 2017.
Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. DeepFix: Fixing common C lan- guage errors by deep learning. In AAAI, 2017.
Vincent J Hellendoorn, Christian Bird, Earl Barr, and Miltiadis Allamanis. Deep learning type inference. In ESEC/FSE 2018, 2018.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.1477 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case neural text degeneration. In ICLR, 2020.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv: 190,:0999909933 2019.
11

---

Published as a conference paper at ICLR 2023
Maliheh Izadi, Roberta Gismondi, and Georgios Gousios. CodeFill: Multi-token code completion by jointly learning from structure and naming sequences. In ICSE, 2022.
Kevin Jesse, Premkumar T. Devanbu, and Toufique Ahmed. Learning type annotation: Is big data enough? In ESEC/FSE, 2021.
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In ICML, 2020.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. arXiv preprint arXiv:2202.065333 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini arXiv:2001.0836 2020.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. CTRL: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1999.05888, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. SPoC: Search-based pseudocode code. In NeurIPS, 2019.
Katherine Lee. Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In ACL, 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequencceto--sqquenee pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with AlphaCode. arXiv preprint arXiv:2203.08814 2022.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. CodeXGlue: A machine learning bench- mark dataset for code understanding and generation. In NeurlPS, 2021.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. arXiv preprint arXiv:2203. 13474, 2022.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. Fairseq: fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038, 2019,
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. arXiv preprint ar arXiv:2203.0213: 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
Michael Pradel, Georgios Gousios, Jason Liu, and Satish Chandra. Type Neural type predic- tion with search-based validation. In ACM SIGSOFT, 2020.
12

---

Published as a conference paper at ICLR 2023
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. Technical report. OpenAl, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. In JMLR, 2020.
Ronald Rivest. RFC1321: The MDS message-digest algorithm, 1992.
Baptiste Roziere, Marie-Anne Lachaux, Mare Szafraniec, and Guillaume Lample. DOBF: A deob- fuscation pre-training objective for programming languages. In NewrIPS, 2021.
Gabriel Ryan, Justin Wong, Jianan Yao, Ronghui Gu, and Suman Sekhar Jana. CLN2INV: Learning loop invariants with continuous logic networks. In ICLR, 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, Berlin, Germany, 2016. URL http://ss///ccaaatloloooooo/////
Yong-Siang Shih, Wei-Cheng Chang, and Yiming Yang. XL--Eitor: Post-editing sentences with XLNet. arXiv preprint 19Xiv:1910. /0479, 2019.
Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, and Vijay Saraswat. Combi- natorial sketching for finite programs. In ASPLOS, 2006.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. In ICML, 2019.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tzi Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.055: 220399 2022.
Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Natural Language Processing with Trans- formers. O' O'Reilly Media, Inc., 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention all you need. NewrlPS, 2017.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. httpss//ggtthbbo.oo////gggo///// htaprr/ooooommmttttaaarrrrrrrerr
transformer-jay. May 2021.
Yue Wang, Weishi Wang. Shafiq Joty, and Steven CH Hoi. CodeT5: Identifier-awarr unified pre- trained encoder-decoder models for code understanding and generation. In EMNLP, 2021.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
Jiay Wei, Maruth Goyal, Greg Durrett, and Isil Dillig. LambdaNet: Probabilistic type inference using graph neural networks. ICLR, 2020.
Peter West, Ximing Lu, Ari Holtzman, Chandra Bhagavatula, Jena Hwang, and Yejin Choi. Re- flective decoding: Beyond unidirectional generation with off-the-shelf language models. In ACL, 2021.
Frank F Xu, Uri Alon, Graham Neubig, and Vincent J Hellendoom. A systematic evaluation of large language models of code. arXiv preprint 13119 2022.
Zhaogui Xu, Xiangyu Zhang, Lin Chen, Kexin Pei, and Baowen Xu. Python probabilistic type inference with natural language support. In SIGSOFT, 2016.
Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. ICML. PMLR, 2020.
Michihiro Yasunaga and Percy Liang. Break-it-fix-itt Unsupervised learning for program repair, In ICMI, 2021.
13

---

Published as a conference paper at ICLR 2023
Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. Learning to mine aligned code and natural language pairs from stack overflow. ACM MSR, 2018.
Tao Yu, Rui Zhang, Kai-Chou Yang. Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma,
Irene Z Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to- SQL task. In EMNLP. 2018.
John M Zelle and Raymond Mooney. Learning to parse database queries using inductive logic programming. In AAAI, 1996.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. In NeurlPS, 2019.
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In EMNLP, 2021.
A DATA
A.1
CODE DATA
Sources. We obtained code files and repository metadata from GitHub and GitLab through the sites" public APIS over period ending December 9th, 2021. We obtained approximately 670,000 public non-fork repositories which GitHub/GitLab detected as containing primarily Python, JavaScript, Jupyter Notebook files, and with either an MIT, Apache 2.0, BSD-2, BSD-3 clause license. We included all code from a list of 28 languages (determined by file extension) contained in these repositories Since Python files can also be contained in con-majority-Pythee repositories, we also included all other Python and Jupyter files obtainable through the GitHub archive on BigQuery that we did not already obtain from GitHub directly We preprocess Jupyter notebooks by includ- ing all text and code (with Markdown formatting removed from text cells), with cells demarcated by XML-style tags (see Section A.3)
TD
Deduplication. Recent work has shown that deduplicating training data can improve model per- formance and reduce the risk of memorizing training data (Allamanis) 2019 Lee et al. 2022 Kand- pal al. 2022). Our deduplication scheme removes code files using exact match on the sequence
of alphanumeric tokens in the file This removed approximately 75% of the corpus by file size (reducing from TB to 250 GB) as there numerous duplicated repositories, library dependen- cies included as source files, and common boilerplate code files (e.g., for Python web frameworks). We also use regular expressions to detect email addresses in the code files and replace them with dummy address to reduce the risks of the model memorizing real email addresses or hallucinating fake ones.
To ensure that our code generation models can be evaluated on several current code generation benchmarks, we perform data decontamination: removing overlap between our training data and the evaluation sets of these benchmarks. We remove any repositories contained in
Decontamination.
'ee include source files from C, C++, CSS, C#. Common Lisp, Dart, Forth, Go, HTML, Haskell, Java, JavaScript, Julia, Jupyter, Kotlin, Lua, Matlab, PHP, Perl, Python, R, Ruby, Rust, SQL, Scala, Shell, Swift, and TypeScript, although the great majority of files are Python and JavaScript. See[Tigure use
10 (Ww
github-on onruub-on-bal bittuu--onbbigqurr--aa of the above permissive licenses.
We only include repositories with one
11 We implement match checking using Bloom filter Bloom 1970 whose keys are: the file extension, number of tokens in the file, and an MD5 hash Rivest (192) the sequence of tokens, which is highly
accurate at identifying files with exactly matching token sequences.
We replace detected email addresses with removed@example.com.
14

---

Published as a conference paper at ICLR 2023
the validation and test sets of CodeSearchNee (Husain al. 2019), these are used to construct validation and test sets for several of the tasks in CodeXGLLUE (Lu al. 2021
Filtering. Our filtering similar to past work on generative models of code Chen et al. a); Nijkamp et al. ); Xu al. (2022): remove files that contain any line longer than 3000 tokens or an average line length greater than 100 tokens, have less than 40% of their characters being alphanumeric underscores, appear be automatically generated, which we determine using substring match on a small number of phrases produced by automatic code and documentation
generation systems Python files.
Our decontamination and filtering steps together remove roughly 10% of
A.2 STACKOVERFLOW
The second component of our corpus consists of questions, answers, and comments from Stack- Overflow. The Pile (Gao et 2020), which used train recent generative code models that we compare to in Section 5| also contains these questions and answers but does not contain com- ments. We include all questions that have least one answer, up to ten answers with non-negative score (sorted by score) per question, and to five comments per question/answer. Qualitatively, we find that comments, together with the infilling ability of the model, allow our model have some capability to do interactive code editing guided by language (see Figure 11)
A.3 METADATA
We include metadata the code files and StackOverflow questions/answen. directly in training data to allow attribueecondittonee generation (Keskar et al., Zeller et al., (019) and attribute prediction. For code file data, our attributes are the code filename, the file extension (as proxy for language), the file source (GitHub or GitLab), and, for GitHub repositories, the number of stars binned into six buckets To allow this metadata to be optional when performing left- to-right prompting of the model, we insert each attribute the beginning of its document with a probability of 50% (allowing the model learn metadata conditioning); otherwise, the end of its document (allowing metadata prediction). See[Figure 6a and Figure 6b for examples. For StackOverflow, and the number of votes for each question and answer, binned in the same way as repository stars. We insert comments directly after the questions answers they written for. See Figure 6c for examples.
insert it
metadata attributes are the question tags for the topic (e.g.. python, django)
A.4 TOKENIZATION
To increase the amount of context that our code model can condition on, the length of documents that the model can generate, and the efficiency of training and inference, we train a byte-level BPE tokenizer (Sennrich al. (016); Radford et al. (2019). We allow tokens to extend across whitespace (excluding newline characters) so that common code idioms (e.g.. import numpy as пр) are rep- resented single tokens in the vocabulary. This substantially improves the tokenizer's efficiency reducing the total number tokens required to encode our training corpus by 45% relative to the byte-level BPE tokenizer and vocabulary of GPT-22
15

---

Published as a conference paper at ICLR 2023
Total File Size (GB)
40
30
python
javascrips
html
C++
java
jupyter notebook
typescripo
go
CSS
php
pisql
c#
ruby
sql
shell
objective-c
perl
scala
rust
Figure 3: Code corpus composition (after deduplication and filtering) by total file size for the most common languages, as determined by file extension.
A.5 CORPUS STATISTICS
See Figure 3) for plot showing code corpus composition (after deduplication and filtering) by total file size for the most common languages, as determined by file extension.
B MODEL AND INFERENCE DETAILS
B.1 MODEL.
Parameter
INCODER-IL3B
INCODER-6.77
decoder-embed -iim
2048
4096
dlcoddrr-output-dim
2048
4096
decoder-input-din
2048
4096
decoder-fft- ffh-cmbed-dim
8192
16384
decoder-layen
24
32
True
True
decoder- heads
32
share-dccoder popuuttouput--nne embed
True
Truc
Geooder-learned
False
False
Table 6: Fairseq architecture hyperparameters for our INCODER models.
Our primary model INCODER-6, 7B, 6.7B Transformer Vaswani et al. (0117) language model. We the same architecture the dense 6.7B models described in Artetxe al. (202T): the Fairseq architecture description can be found in Table 6 INCODER-66 7B was trained on 248 V100 GPUs for
We also search for occurrences of code from the HumanEval Chen al. 2021a dataset using substring match. but did not find any matches our training set. The solutions to the problems this dataset, the time that we obtained our files from GitHub, were only contained in ISON files. We additionally removed repositories in the evaluation sets of the JulCe tasks (Agashe et al. 2019), although we did not evaluate our model on these tasks in this present work.
OOur filters target files automatically generated by the Protocol Buffer Compiler, Django, Epydoc, Pdoc, Sphinx, MkDocs, MathJax.
15 We size bins using an inverse binary logarithmic scale, so that bucket corresponds to the repositories with star counts up the 50th percentile, bucket corresponds to the 50th to 75th percentiles, and bucket 5 to those in the 97th percentile and above.
16

---

Published as a conference paper at ICLR 2023
18-
InCoder-68 InCoder-18
Validation Perplexity
EvaanEval Pass@]
16-
14-
12-
8-
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Fraction Training Data
Fraction of Training
Figure Loss curves show that perplexity is still improving after one epoch and that perplexity improves substantially with a larger model size. This suggests that increasing epochs, data size, or model size would improve performance.
Figure 5: Performance of INCODER-6.7B on the HumanEval left-to-right synthesis bench- mark generally increases over the course of pre- training. We plot line of best fit along with 95% confidence interval via bootstrap resam- pling.
24 days. We perform one epoch on the training data, using each training document exactly once. Our implementation utilized the causal masking implementation (Aghajanyan et al. 2022a) available in Fairseq (Ott et al. 2019), with the underlying library being Py PyTorc (Paszke et al. 2019). Our per- GPU batch size 1.0 and used the Adam optimizer with 81 = 0.9, 32 = 0.98 (Kingma & Ba 2015). For our learning rate scheduler, we use the built-in polynomial decay learning rate scheduler available in iaszaz al.) (211)) with 1500 warmup updates. Fairscale was used for improving memory efficiency through tully sharding model states (Baines et al., 202T).
with maximum token sequence length of 2048. We clip all gradient norms to
We compare the validation perplexity of the 6B parameter model and smaller 1.3B parameter model (see (secionction for details on the training of this 1.3B model) in Figure 4| showing comparable scaling laws to those reported by Aghajanyan et al. |Aghajanyan et al. (2022a). Our models have also not yet saturated and would benefit from further training; we report the performance of the 6. model on the HumanEval Python function synthesis benchmark (Chen et al. (222 (see|Section C.6 for a description of this benchmark) and see a consistent increase in performance over the course of training (Figure 5).
B.2 INFERENCE DETAILS
In practice, to generate single infill we sample from the distribution P(-
[Left; <Mask:0>; Right; <Mask: 12; <Mask: where we insert an artificial <Mask: 1> token. Not inserting <Mask: 1> gives an implicit size hint the model that the <Mask: 0> token should be expanded to fill the rest of the 2048 token context window. Instead, inserting a <Mask: 1> token indicates the model that some amount of the document is omitted after the right context. We found that including this substantially improved the ability of the model to predict <EOM> appropriately when generating an infill for <Mask: See Aghajanyan al. (2022a) for more.
More generally, when inserting at multiple locations, we condition on the document with mul- tiple mask sentinel tokens inserted and a final mask token appended. For example, to insert at two locations we use [A; <Mask: 0>; C; <Mask: 1>; E: <Mask:2>]] and infill the masks in order, appending the appropriate <Mask:k> sentinel tokens to signal the start of generation for the next span, i.e. the completed document for two insertion locations is represented by [A; <Mask: 0>; C; <Mask: 1>; E; <Mask: 2>; <Mask:0>; B; <EOM>; <Mask: D; <EOM>], where regions B and D have been infilled.
17

---

Published as a conference paper at ICLR 2023
C EXPERIMENTAL DETAILS AND SUPPLEMENTARY RESULTS
C.I INFILLING COMPARISONS
We describe our adaptation of models from prior work the zero-shot infilling setting, for the experiments described in Section 4
Encoder-decoder (PLBART). We use PLBART-Large (Ahmad al. 2021), ) an encoder-decoder model trained on code (including 220GB of Python) using a BART (Lewis et al. (019) masked denoising objective. We pre- and post-process each HumanEval infilling example as needed for PLBART: represent each example a stream of space-separated tokens (as identified by Python's ' built-in lexer) with newlines and indentations replaced by control characters, and use <mask> token represent the line to be infilled. We extract the infilled region from the output by searching for the longest suffix of the left context contained in the output, and (as in our left-to-right baselines) take the ground-truth number of lines following this left context suffix the infill.
Left-to-right with templated prompting (Codex). We perform zero-shot prompting on the Codex cude-cushma--001 and code-davinci- OpenAl API models using the following tem- plate:
[code before the infill mask] <INFILL> [code after the infill mask] # Complete the above code by replacing the <INFILL> tag.
[code before the infill mask]
We take [code after the infill mask] as the indicator of completion.
C.2 CODE CLOZE (CODEXGLUE)
CodeXGLUE cloze is created from CodeSearchNet to evaluate CodeBERT and consists of short natural language description followed by code in several programming languages. We evaluate on the max/min subtask, where the model has to decide if the given mask should be filled with either max min. Since there are only two options in this task, we can closely compare the causal-masked infilling and left-to-right setups by scoring both options and selecting the sequence with the highest likelihood.
Table 7 contains the main results. Using the causal-masker infill format with a single token (con- taining min/max) as the masked region (CM infill-token) performs better than using just the left context, but not as well as scoring the entire sequence left to right. Masking a larger region (CM infill-region), containing the left prefix and 10 right-side tokens in the masked region, performs comparably to scoring the whole sequence. Infill region length and tokenization can affect the per- formance, see C.3 for more details and more comparisons.
Method
Python
JavaScript
Ruby
Go
Java
PHP
Left-to-right single
76.9
77.6
65.8
70.4
74.1
77.1
to-i-to- reranking
87.9
90.1
76.3
92.8
91.7
90.4
CM infill-token
81.8
73.9
81.6
95.4
77.6
87.0
CM infill-region
86.2
91.2
78.9
94.7
89.8
91.4
CodeBERT
82.2
86.4
86.8
90.8
90.5
88.2
Table 7: Accuracy on the CodeXGLUE max/min cloze task. We compare four different inference methods. Left-to-right single: scoring with left-to-right ordering using only the left context and the completion (containing max or min); Left-to-right reranking: scoring with left-to-right ordering using the left context, completion, and right context: CM mnfill-token: causal masking scoring, using only a single token (containing max or min) as the infill, CM infill-reiio:: causal masking scoring that additionally contains 10 tokens from the right side context.
18

---

Published as a conference paper at ICLR 2023
Note that comparing the scores of the sequences, which differ in their infills, with the left-to-right setup is more computationally expensive than with the CM infilling setup, as the Transformer in- termediate activations can be cached and shared across identical sequence prefixes, and in the CM infill setup all sequence differences occur at the ends.
C.3 CLOZE AND SINGLE TOKEN INFILL DETAILS
Python
Javascript
Ruby
Go
Java
PHP
Left-break
72.4
72.1
68.4
71.7
74.1
76.9
Left-token
76.9
77.6
65.8
70.4
74.1
77.1
Left-region
84.2
88.6
73.7
85.5
87.6
87.0
Left-right-break
77.9
79.4
63.2
89.5
82.0
85.3
Left-right
87.9
90.1
76.3
92.8
91.7
90.4
Infill-break
79.1
83.1
84.2
90.1
84.0
85.3
Infill-ooken
81.8
73.9
81.6
95.4
77.6
87.0
Infill-region
86.2
91.2
78.9
94.7
89.8
91.4
CodeBERT 82.2
86.4
86.8
90.8
90.5
88.2
Codex
93.6
93.4
94.7
99.3
95.0
94.3
Table 8: Accuracy on CodeXGLUE cloze max/min. Left: scoring using only the left context, Left- right: score the whole program. Infill: score the infilling sequence, -region: include left context and 10 tokens from the right. -break: break tokenization on the infilled token. Codex : version code-davinci-o0
OpenAl's Codex model, as accessed through their API. Information on the training data for this model is unclear, and it may contain portions of CodeSearchNet (which contains this task's evaluation set).
As shown in Table 8 breaking tokenization (-break) on infill decreases the performance using all scoring methods. For example, whereas Math. max was single token the full sequence, the sequence is broken into Math., max, and for infilling. Infilling with the original tokenization increases the performance slightly, but does not match full left-right scoring. We suspect this is because the model was not trained infilling single tokens, unlike CodeBERT. A way to fix this to include a larger region on the left and a few more tokens on the right. This will only slightly increase the scoring complexity. To show that model uses the right context, compare it with scoring the left-only model. More precisely, the sequences being scored are
Left-to-right single:
[Left; Token]
Left-to-right reranking:
[Left; Token; Right]
Infill-token:
[Left; <Mask:0>; 0>; Right; 12; <Mask:0>; 0>; Token; <EOSS>]
Left-region:
[Left; Token; Right[:
Infill-region:
<Mask:0>; Right[10:]; <Mask: 12; <Mask: 02; Left; Token; Right[: 10]; <EOSs>]
C.4 COMPARISON TO OPENAI'S CODE API
We evaluate OpenAl's proprietary code-davinci system, accessed through their API, on our single-line HumanEval infilling task, with results given in Table 9, There is limited public informa- tion about this system, including on its training data or procedure (although Bavarian al. 2022 describes their FIM objective early research that helps power the model), how it performs in- fills, or whether any postprocessing is done on model outputs, but we report its performance to help gauge the difficulty of our new task. For both code iiteeddvvinii--e- and our INCODER-6 7B model, conditioning on right-sided context improves performance, with the most substantial improvements from infilling.
19

---

Published as a conference paper at ICLR 2023
Model
Inference
Pass Rate
Exact Match
INCODER-6.7B
Left-to-right single
48.2
38.7
INCODER-6.7B
Left-to-right reranking
54.9
44.1
INCODER-6.72
Infilling
69.0
56.3
node-davinci-0de
Left-to-right single
63.7
48.4
code-davinci-002
Left-to-right reranking
71.8
52.0
code-davinci-000 Infilling
87.4
69.6
Table We evaluate OpenAl's proprietary code-davinci-P0 -eddaainiiii222 system, accessed through their API, on our single-line HumanEval infilling task. Although no information currently public about this system, its training data or procedure, how it performs infills, or whether any postprocessing is done on model outputs, we report performance help gauge the difficulty our new task. For both code-davincis -002 and our INCODER-6.76 model, conditioning right-sided context improves performance, with the most substantial improvements from infilling.
C.5 ADDITIONAL TYPE HINT PREDICTION SETTING
Our results in Section 4.3 filtered out functions from the TypeWriter prediction which had return type hint of None, as these type hints are are overrepresented the dataset in compared to naturally-occurring code, due to the filtering process used to construct it. For a closer comparison to the setting used in the original TypeWriter paper, we present results including these functions in Table 10 Given TypeWriter's static analysis capabilities, and the overrepresentation types in this evaluation set, we add a simple post-processing step (return checks) that predicts None the function does not have any non-trivial return statements, which captures some of the effect of TypeWrite Wyitersser's analysis capabilities. In all settings, our zero-shot infill approach outperforms the left-to-right baselines, and obtains performance comparable to the supervised TypeWriter approach when return checks are used.
None
Method
Precision
Recall
Fl
Ours: Left-to-right single
20.0
20.0
20.0
Ours: Left-to-right rerank
24.2
24.2
24.2
Ours: Infill
46.8
46.8
46.8
Ours: beft-to-righh single + Return checks
63.2
63.2
63.2
Ours: Left-to-right rerank + Return checks
64.3
64.3
64.3
Ours:
Infill Return checks
76.7
76.7
76.7
Writer (Supervised)
78.8
69.9
74.1
Table 10: Return type hint prediction results on the 25% subset of Type OSS dataset where were able obtain source files, extract functions and types from, and that were not contained in our model's training set. Given an overrepresentation of functions with None in this dataset, and the static analysis capabilities of TypeWriter, also give results using simple post-procsssing step that predicts None if the function does not have any non-trivial return statements.
C.6 COMPARISON TO LEFT-TO-RES -IGGHT GENERATIVE MODELS CODE SYNTHESIS
We compare to past published work on generative code models the HumanEval (Chen et al. 2021ah and MBPP (Austin et al., 202T) benchmarks, which require models to condition on natural language descriptions (docstrings) to produce Python programs (typically single function), and evaluates overall functional accuracy (pass rate) across examples using several test cases for each program.
We evaluate our INCODER-6:. 6.70 model in zero-shot evaluation on both of these benchmarks. For HumanEval, follow past work by prompting with function signatures and doestring descrip- tions, sample 200 candidate program completions, and compute pass@ 1, pass@ 10, and pass@ 100 using the unbiased sampling estimator of Chen al. (Chen et al. 2021a). For MBPP, which
20

---

Published as a conference paper at ICLR 2023
Model
Size (8)
Python Code (GB)
Other Code (GB)
Other (GBB)
Code License
InEl??
HE: #1
@r 10
HE:
@ 100
HE:
MBPP
61
Released
CodePurro Tunstall et 2022
1,5
50
None 90
Nome None 730
4,0
8.7
17.9
PolyCoder
16
GPT-1 Wang < Komarsizaki 202
6
11.6
15.7
27.7
Chen
102
52
Permissive
27.8
GPT-NcoXX Bt 2022
20
6
90
730
15.4
25.6
41.2
CodeCien-Momn
62
1200
18.2
28.7
CodeGen-Mono Nijkamp a 2002
279
375
1200
26.1
42.3
65.8
GooleGen-Mona Kijappp 2i1e3
16.1
279
375
1200
29.3
49.9
75.0
Unreleased
LaMDA (Austin et al. 2021 Thoppiaan
137
None
None
???
14.0
47.3
14,8
Alph [irer 21122
54
660
None
17.1
28.2
45.3
Codex-2.56 2,58 et 2021
2.5
180
Nonc
-500
21.4
35.4
59.5
Codex
Chen
28.8
PaLM-Coder (Chowdiitt atal. 2022
540
-20
-200
-4000
Permissive
36.0
88.4
47.0
Table 11: A comparison of our INCODER-6.7B model to published code generation systems using pass rates @ K candidates sampled on the HumanEval and MBPP benchmarks. All models decoder-only transformer models. A "Permissive" code license indicates models trained on only open-source repositories with non-copyleft licenses. The GPT-J, GPT-NeoX, and CodeGen models are pre-trained on The Pile (Gao et al.) (020), which contains a portion of GitHub code without any license filtering, including 6 GB of Python. Although the LaMDD model does not train code repositories. its training corpus includes 18 B tokens of code from web documents. The total file size of the LaMDA corpus was not reported, but it contains 2.8 T tokens total. We estimate the corpus size for PaLM using the reported size of the code data and the token counts per section of the corpus.
does not include function signatures, we prompt only with the docstring description and compute pass @1 1 (Chowdhery al. 2022) using single candidate [6] We top-p sampling with p 0.95, with a temperature of 0.2 for pass@ and 0.8 for pass 10 and pass @ 100.
We compare INCODER-6.7B model to models from past work (which have all been left-to- right only) in Table 11 giving the model size and training data summary statisties as reported (or estimated, in cases when paper only reports token counts, as tokenizer efficiencies vary) in these papers. While differences in details of the Transformer model architectures, datasets, and training procedures across papers and experimental setups make rigorous comparison impossible, we note that our model achieves roughly comparable performance on the HumanEval metrics to CodeGen- Multi Nijjkamp et al. 2022). which is also ~6B parameter model trained on roughly the same amount of Python code, as well as AlphaCode's s 1.1B decoder-only model (Li et al. 2022 which also uses similar amount of Python training data.
While this setting is not directly comparable to the three-shot setting where the models of Austin et al. (Austin et al, 2)21) and Chowdhery et al. (Chowdhery et al. 2022) performed best, we found that our model did not benefit from additional examples in the prompt, which we attribute much smaller size of our model (6.7B, versus 137B 540B parameters) and the sensitivity of in-context learning model scale.
21

---

Published as a conference paper at ICLR 2023
D EXAMPLES
D.1 METADATA EXAMPLES
SyATE
File sctn-fpyrt pyython
<teet?
Tats notebook dewonstrates ustin sctkit Tears to perfora PCA.
</text>
Snatplotlib
</eett>
<cell>
contscoots:: wor__contsteend
Inport
marplotllb, pyplot
clsc:
worE_contsteerd
reture NOTE_COEEES Fileremersrrnngatt11..... datersn4
(a) Metadata for code includes the file exten- sion, source thub or gitlab), filename, and binned number of stars for GitHub repositories (in lngarithmically-sized bins numbered 5, see Section A.3)
(b) In addition to the standard metadata used for all other code files, Jupyter Notehook metadata in- cludes the kernel type (in this instance, Python) well as the type the cells the notebook (either code or text).
tags-python.co ango
Django How to display form with input
I'm trying to create simple web application that can allow user to enter usernane and password..
Here's what I have so far: <code>
from django.shortcuss shortcuts Inport render
...
</code>
Can you
<// dscorem3 |>
Why aren't you using the djanga authentication library? c
<| dscore=4 |>
You need to add your form to the template: <code>
...
tags=python,cm djang
(c) Metadata attributes for StackOverflow include question tags and discretized scores questions and answers.
Figure 6: Examples of metadata attributes included the training data to allow attribute-conditioned generation and attribute prediction. To allow both generation and prediction. attributes randomly included either at the beginning of the document or at the end (with probability 0.5 each). Attributes occur in random order to allow arbitrary orderings at inference time.
22

---

Published as a conference paper at ICLR 2023
D.2 EXAMPLE MODEL OUTPUTS
def get coononnnnsee]
conn_ conn_config self._get get__oon_____eee conn_params_( () snowflake.c snowflake.comnector.com return conn
(a) Reference doestring: Returns a snowflaae.comnection object. Model docstring: Establishes connection to the Snowflake cluster.
def sprint *colors):
.*. return ((conttt)))teeet N33 formatt" ("o"att""". (ooiji(sstt for color in colors, RESET, context=text))) if $S_ANSI_ITA TERMINAL and colors else text
(b) Reference docstring: Format text with color or other effects into ANSI escaped string. Model docstring: Prints a string with ANSI color codes.
Example docstring generations for the CodeXGLUE code-to-text dataset. Captions for each example give the reference human-written doestring and the output from our INCODER-6.7B model with causal-masked infilling. The model generates doestrings zero-shot by inserting text between the comment delimiters.
Figure 7:
fite exce py
count the words in all estes in the current directory Ieport Ieport sys
hoad
notn():
getcodd)
for filencee in os. iis
titename, endswtthi" path joiniced. filenzee)
with spençfners)) as infile: intile:
words LencLinee splic())
in (vards)
nane main()
mzin
Figure 8: Meta-data conditioning file extensions for Python (left) and Shell (right) allows com- pleting the same text comment as either a Python script or a pipelined bash command, respectively. Regions highlighted in orange are left-to-right generations from INCODER-6.7. 6.7B model.
23

---

Published as a conference paper at ICLR 2023
class Person:
def init (self.nanl age. gender):
self name-name
self ageeage
self gender=gender
Person 18, "Male")
Figure 9: Given the beginning of a class definition and right-sided context of the class being used, the model is able to infer plausible attribute names for the class (c.g., "Eren" is likely to be a name, 18 is age, "Male" is the gender.) The region highlighted in orange is an infill generation from INCODER-6.1. model.
Original Code
Inseri comment and reganenerate
cata, target data. data. (device), target.tt(ddevcc) optinizer.zero zero_rradd)
data. data. to(device). (ddevie)) intzer. sero_grad() nutput model(cata) F.nl1_ (autppts target) backwaack))dd(
output model(data)
(outbuu,, target) baskarack
cata, target = data data. to(devce)))) to(gete.)( ootinizer..ero_.ree
data, target data. to(device). torget.t aptimiizer.zer_gnaa zero_gra()
output (odel(data) lass (eut_uuss target) backwarr)
output nodel
criterion SmoStoothLllosss) loss criterion(output, target) loss.backward()
Figure 10: By inserting comment (# use Huber loss) in the code and replacing the line after with mask token, a user can prompt the model to in-fill a region of code with a contextually- appropriate replacement. Lines in orange in-fill generations from our INCODER-6.. 6.7B model.
24

---

Published as a conference paper at ICLR 2023
Generate conditioned question
need to write Python function called "snake_to_camee to_camel",, Here's description: "Write function to convert snake case string to camel case string."
<I q tags=python |>
<| |>
You str.title:
<code>
>>> my__saae_case'.. .ittle(
MySnakeCase"
</code>
</
tags=python dscore=3
Add comment
need write Python function called "snake camel', Here description: "Write function convert snake case string to camel case string."
<I q tags=python |>
<| a |>
You can use str.title title:
<code>
>>>
"my_ snake_casse.. .tttee
'MySnakeCase'
</code>
</ tags=python dscore=3
<| |>
Close, but you need to replace the underscores with an empty string.
<|/ c |>
|>
Thanks, edited!
</ |>
Regenerate infill conditioned on comment
need to write Python function called tatee_to_cmeel's Here's description: "Write function to convert snake case string to camel case string."
<| q tags=python }>
<| a |>
<code>
def snake_ta ttc_ttteo_elleeeeeeeeeee (name):
return (iieex for x in name. split(''')
</code>
tags=python dscare=3 |>
<| |>
Close, but need to replace the underscores with an empty string.
<|/ c |>
<| c |>
Thanks, edited!
<//
Figure 1I: Pretraining on StackOverflow, and our model's infilling capability, allows to perform zero-shot interactive refinement of function. In the first example, the model has generated the orange region conditioned on the user description. comment specifying refinement to the function, and selected the text for the model to replace. In the third example, the orange region has been infilled by the model.
the second example, the user has added
25

---

Published as a conference paper at ICLR 2023
tags=tensorflox, gPU |>
How to limit GPU memory usage in TensorFlow?
I am interested in Limiting the GPU memory usage 1n Tensor Low. I work an environment which computational resources are shared. i.e., have few server machines equipped with few Nvidia Tittn X GPUs each.
For small to moderate s size models, the 12 GB of the Tittn X is usually enough for 2-3 people to run training concurrently on the same GPU. If models are small enough that single model does not take full advantage of all the computational units of the GPU, this can actually result in a speedup compared with running one training process after the other. Even in cases where the concurrent access to the GPU does slow down the individual training time. it is still nice to have the flexibility of having multiple users simultaneously train the GPU.
The problem with TensorFlow is that, by default, it allocates full amount of available GPU memory when it launched. Even for small two- neural network, I that all 12 GB of the GPU memory is used Is there way to make TensorFlow only allocate, say, GB of GPU memory. if one knows that this is enough for given model?
q dscore=3
Figure 12:
Question tag and title prediction from the text StackOverflow question. Regions highlighted in orange infill generations from our INCODER-6.77 7B model.
, bubble sort
 dynamic programming
 quick sort
#u data structure
word embedding
Figure 13: Zhot bidirectional technical jargon translation between Chinese and English. Re- gions in orange are infill generations from our INCODER-66 6.7B model.
26


Reviews:
[{"review_id": "xKRUnqBJkc", "reviewer": "Reviewer_PWMt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The current work presents InCoder, an alternative approach to general CLM or seq-to-seq code generation where the training regime is changed to allow code generation in arbitrary subsections of the code.\nExperimentation seems appropriate, including HumanEval, DocString generation, return type prediction and variable name prediction showing model capabilities.\n", "summary_of_the_review": "A good idea, well implemented and wrapped up in a quality paper\n", "strength_and_weaknesses": "Strengths \nA really good paper\nThe idea is clever, simple and straightforward. It provides a functionality that general GPT-like or full transformer models don’t.\nExperimentation seems appropriate. Authors make a great effort not sticking with standard benchmarks which might not be the ideal scenario for InCoder\n\nWeaknesses\nThe training regime comes with a cost and the Left to right generation, specifically on Table 11\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nQ3roIgNaV", "reviewer": "Reviewer_7zUC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a mask infilling method to train language model for program synthesis with the ability of code infilling. The method randomly selects a span and replace it with a mask token, and place the span after the sequence as the target. The proposed method is novel, however the masking method is also used in the literature of T5 pretraining. Experiments show that the mask infilling does not degrade left-to-right performance, while significantly improve insertion capability.", "summary_of_the_review": "Overall, I think this is a good paper with novel method proposed and well-performed experiments.", "strength_and_weaknesses": "Strength:\n- The paper is well-written. The method is well explained with examples. Experiment results are extensive with analysis.\n- The proposed method is a simple but effective masking objective. Zero-shot performance demonstrates the effectiveness for infilling line completion task, as well as other related prediction tasks.\n\n\nWeakness:\n- From the main text, I cannot infer whether the training is done 100% with mask infilling or there are some probability that normal left-to-right data is used for training without any masking. If it is 100% mask infilling, it seems surprising to me that left-to-right is not affected at all. I would expect normal left-to-right data exist in training, like the concurrent work fill-in-the-middle (FIM) training (Bavarian et al., 2022).\n- Some training details seem missing, for example, how many training tokens and number of iterations the model has gone through during the training process.\n- Though there is an <EOM> token indicating the end of generation, there is possibility that the generated code after <MASK:0> will contain overlapping content as the actual right context. Did you see any such issue exist? How do you resolve such issue?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gKGjCGaCmQ", "reviewer": "Reviewer_bKjD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a code synthesis model that is trained to perform both autoregressive left-to-right generation as well as infilling via added mask tokens. The experimental results show improved performance on various infilling related code generation tasks.", "summary_of_the_review": "Despite not evaluating on full code generation task, the paper has enough experimental results to support itself for infilling tasks.", "strength_and_weaknesses": "The paper is well written and easy to read. The approach described tackles fundamental challenges in autoregressive generation by training  models that are capable of filling the masked regions. This ability is necessary for tasks such return type prediction and doc string generation where the output depends on both left and right context. The experimental results are also encouraging as for various tasks shown, the model is able to perform better with less model size (e.g., compared to code-davinci-001).\n\nThe paper, however, does not discuss full code generation as it focuses primarily on infilling experiments, so its competitiveness on more challenging tasks is not known. With the ability of infill and the original motivation that \"Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined.\", I'd be very interested to see how the model performs on full code generation as well as devising an ability to detect and re-write portions of the code when solving those tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "evVp_0hs12", "reviewer": "Reviewer_CKVi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach to perform code infilling that supports multiple insertion points. The approach is by defining an infilling format that describes the missing code locations and the code to be infilled. The training approach is by the traditional autoregressive next token prediction loss. This paper indeed proposes a simple technique that seems to work reasonably well.", "summary_of_the_review": "Nice contribution, but lacking thorough comparison to the normal baseline / how much performance drops compared to normal left to right approach. However, the multiple infilling spots is a plus.", "strength_and_weaknesses": "Pros:\n\nThis paper shows that a simple method works. \n\nCons:\n\nThere is no baseline against the traditional left to right approach. That is, with a given amount of data / compute this work uses, what would be the performance if we use these resources for left to right completion alone? This is important because the amount of data that the paper uses as well as the training resources are slightly different from other papers, therefore, it would be good to see how much this amount of data/compute yield in terms of left to right completion. Note that this could be different from the left-to-right performance that is obtained from the infilling method proposed. That goal here would be to compare if this approach results in performance drop due to infilling or not, and if so how much. A similar work by Bavarian 2022 shows that with a similar approach but only with 1 infilling spot, there seems to be no degradation, but this work is sufficiently different that it is not clear if the results by Bavarian 2022 also applies here.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%