Incoder: A Generative Model for CodeInﬁlling and Synthesis
Presented by: Levent Toksoz

Motivation
Large Language Models that are trained on large code repositories shows great
potential in neural program synthesis and other code related tasks
However most of these models are designed to generate code left to rightThus they are not very effective at code editing tasks like ﬁxing bugs, adding
comments or re-naming variables
The ones that utilize Masked Language Modeling are mostly designed to inﬁll very
short spans

Motivation
Also this type of generation (left to right) is not natural for a human - typically we
do not write a code in one left to right pass
Having ability to go back and edit some parts of the code is importantHence they introduce Incoder - a model that is capable of inﬁlling arbitrary regionsof codeIncoderAim of the Incoder is to combine the strengths of both MLM and casual model
Want to take MLM`s ability to condition both left and right context (which isespecially helpful for code inﬁlling) andCasual Model`s ability to autoregressively generate entire documents
Thus they introduce casual masking procedureCasual Masking
At Training
Sample number of spans of contiguous tokens in each document from a Poisson
distribution
Sample the length of each span uniformly from the length of the document (makesure spans do not overlap)
Replace each span k with special mask sentinel token (<Mask:k>)

Casual Masking
Then move the sequence of tokens that are in the span to the end of thedocument and mark the beginning of this span with the mask sentinel token andend with the end of mask token (<EOM>)
By doing this we want to maximize the log probability of the masked document:Casual Masking
Thus computing the probability of the sequence auto-regressively
Model is trained on cross entropy loss on all tokens except the mask sentineltokens (<Mask: k>) since we do not want the model to generate these tokens
during the inferenceCasual Masking

Casual Masking
At Inference
Model can either used to generate left to right by sampling autoregressively
Or to insert code at any desired location by placing <Mask:k> token in thatlocation and doing generation at the end of the document to get bidirectional
context

Training
Data: open source public libraries from Github and Gitlab (mainly python),StackOverﬂow questions/answers/comments and code
Code ﬁles are ﬁltered by removing the ones that have duplicates and any lines
longer than 3000 tokens or average line length greater than 100 tokens

Training
Metadata is also included for code ﬁles (ﬁle name, ﬁle extension, ﬁle source,
number of stars) and for stackoverﬂow data (question tags and number of votes)
Code is tokenized by using byte-level BPE tokenizerTokens are allowed to extend over white spaces to represent certain programming
idioms as one token like import numpy as np

Training
Model architecture is based on Fairseq architecture and trained on 248 V100
GPUs for 24 days with Adam optimizer and polynomially decaying learning rate

Training
Even after 24 days of training they claim that their model is not yet saturated

Experiments
Mainly two types of experiments are performed: Inﬁlling Experiments and CodeSynthesis Experiments
Inﬁlling Experiments are conducted to test the zero-shot inﬁlling ability of themodel on the following programming tasks : 1- Inﬁlling Lines of Code 2- Docstring
generation 3- Code Cloze 4- Return Type Prediction 5- Variable name prediction

Experiments
Compared their approach with two other techniques: standard left to rightgeneration and left-to-right generation with reranking
Left-to-right generation with reranking essentially uses the left context to propose
candidates to inﬁll the blank and both right and left context to rank it

Inﬁlling Lines of Code Results
Benchmarks are constructed from HumanEval
HumanEval has descriptions of functions paired with their implementation andseveral input–output test pairs
Both Multi-line inﬁlling and Single-line inﬁlling is analyzed using Exact Match andPass rate (rate at which the completed function passes all of the function’s
input–output pairs) metrics

Inﬁlling Lines of Code Results

Inﬁlling Lines of Code Results

Docstring Generation
CodexGLUE code to text benchmark is used (aim is to generate a natural language
docstring that summarizes a Python code snippet )Code Cloze
CodexGLUE cloze set is used. (It is made of short natural language description
along with their code in several programming languages)
Aim is to predict whether mask should be ﬁlled with max or minReturn Type and Variable Name Prediction
Benchmarks are created from the CodexGlue

Code Synthesis
Evaluates zero shot performance of Incoder on Human Eval and MBPP
benchmarks (aim is to condition on docstrings to generate python code)
Pass Rate on provided test suites is used as a evaluation metricCode Synthesis

Strengths
Proposes a novel casual masking approach that combines the strength of both
MLM and causal models
It has strong zero-shot performance on most of the editing and code inﬁlling andtasks
Provides functionality to edit any arbitrary region of the code which is often
neglected by the other large scale transformer based models

Limitations
Mostly focuses on code editing inﬁlling, code synthesis performance of the model
is not that well explored
Full potential of the model is not measured since even after 24 days theirvalidation performance was increasing
Future work: Supervised inﬁlling via model ﬁne-tuning and or iterative decodingwith model reﬁning its own output

References
Fried, Daniel, et al. "Incoder: A generative model for code inﬁlling and synthesis."
arXiv preprint arXiv:2204.05999 (2022).