Pre-trained Contextual Embedding of Source Code
Authors: Aditya Kanade
Petros Maniatis
Gogul Balakrishnan
Kensen Shi
Venue:  (2020)

Abstract: The source code of a program not only serves as a formal description of an executable task, but it also serves to communicate developer intent in a human-readable form. To facilitate this, developers use meaningful identifier names and natural-language documentation. This makes it possible to successfully apply sequence-modeling approaches, shown to be effective in natural-language processing, to source code. A major advancement in natural-language understanding has been the use of pre-trained token embeddings; BERT and other works have further shown that pre-trained contextual embeddings can be extremely powerful and can be finetuned effectively for a variety of downstream supervised tasks. Inspired by these developments, we present the first attempt to replicate this success on source code. We curate a massive corpus of Python programs from GitHub to pre-train a BERT model, which we call Code Understanding BERT (CuBERT). We also pre-train Word2Vec embeddings on the same dataset. We create a benchmark of five classification tasks and compare finetuned CuBERT against sequence models trained with and without the Word2Vec embeddings. Our results show that CuBERT outperforms the baseline methods by a margin of 2.9-22%. We also show its superiority when finetuned with smaller datasets, and over fewer epochs.

Under review as a conference paper at ICLR 2020
PRE-TRAINED CONTEXTUAL EMBEDDING OF SOURCE CODE
A Anonymour authors
Paper under double-blind review
ABSTRACT
The source code of a program not only serves as a formal description of an ex- ecutable task, but it also serves to communicate developer intent in a human- readable form. To facilitate this, developers use meaningful identifier names and natural-language documentation. This makes it possible to successfully apply sequence-modeling approaches, shown to be effective in natural-language pro- cessing, to source code. A major advancement in natural-language understanding has been the of pre-trained token embeddings; BERT and other works have further shown that pre- pre-trained contextual embeddings can be extremely powerful and can be finetuned effectively for a variety of downstream supervised tasks. In- spired by these developments, we present the first attempt to replicate this success source code. We curate a massive corpus of Python programs from GitHub pre-train a BERT model, which we call Code Understanding BERT (CuBERT). We also pre-train Word2 embeddings on the same dataset. We create bench- mark of five classification tasks and compare finetuned CuBERT against sequence models trained with and without the Word2 embeddings. Our results show that CuBERT outperforms the baseline methods by margin of 2.9-22%. We also show its superiority when finetuned with smaller datasets, and over fewer epochs. We further evaluate CuBERT's s effectiveness on a joint classification, localization and repair task involving prediction of two pointers.
INTRODUCTION
Modern software development places a high value on writing clean and readable code. This helps other developers understand the author's intent so that they can maintain and extend the code. De- velopers use meaningful identifier names and natural-language documentation to make this hap- pen (Martin 2008). As result, source code contains substantial information that can be exploited by machine-learnin algorithms. Sequence modeling on source code has been shown to be success- ful in variety of software-end wengiwrrenngineeiiggg tasks, such as code completion (Hindle et al.] 2012) Raychev let al. 2014), source code to pseudocode mapping (Oda et al. 2015). APl-sequence prediction (Gu 2016), program repair (Pu al., 20166; Gupta al. 2017). and natural language code mapping (Iyer et al., 2018), among others.
The distributed vector representations of tokens, called token (or word) embeddings, are a crucial component of neural methods for sequence modeling. Learning useful embeddings in a supervised setting with limited data is often difficult. Therefore, many unsupervised learning approaches have been proposed to take advantage of large amounts of unlabeled data that are more readily available. This has resulted in ever more useful pre-trae trained token embeddings (Mikolov et al. 2013a| Pen- nington et al. 2014). However, the subtle differences in the meaning of token in varying contexts are lost when each word is associated with a single representation. Recent techniques for learning contextual embeddings (McCann et al. 2017 Peters al.) [0188] Radford et al., 2018 2019 Devlin (ct al. 2019: Yang et al. (011)) provide ways to compute representations of tokens based on their surrounding context, and have shown significant accuracy improvements in downstream tasks, with only a small number task-specific parameters.
Inspired by the success of pre-trained contextual embeddings for natural languages, we present the first attempt to apply the underlying techniques to source code. In particular, BERT (Devlin et al. 2019) produces bidirectional Transformer encoder (Vaswani et al. 2017 by training it to

---

Under review as a conference paper at ICLR 2020
predict values of masked tokens and whether two sentences follow each other in natural discourse. The pre-trained model can be finetuned for downstream supervised tasks and has been shown to number of NLP benchmarks. In this work, we derive contextual
produce state-of-the art results emhedding of source code by training a BERT model on source code. We call our model CuBERT, short for Code Understanding BERT.
In order to achieve this, we curate a massive corpus of Python programs collected from GitHub. GitHub projects are known contain large amount of duplicate code. To avoid biasing the model to such duplicated code, we perform deduplication using the method of (Allamanis] (2018). The resulting corpus has 6.6M unique files with a total of 2 billion words. We also train Word2 embeddings (Mikolov et al. 2013a| b). namely, continuous bag-of-words (CBOW) and Skipgram embeddings, on the same corpus. For evaluating CuBERT, we create a benchmark of five classifi- cation tasks, ranging from classification of source code according to presence or absense of certain classes of bugs, to mismatch between a function's s natural language description and its body, to pre- dicting the right kind of exception to catch for given code fragment. These tasks are motivated by prior work this space, but unfortunately, the associated datasets come from different languages and varied sources. We want to ensure that there is no overlap between pre-training and finetuning datasets, and that all of the tasks are defined on Python code. We therefore create new datasets for the five tasks after carefully separating the pre-training and finetuning corpora. To evaluate Cu- BERT's effectiveness on a more complex task, we create a task for joint classification, localization and repair of variable misuse bugs (Vasic al. 2019), which involves predicting two pointers.
We finetune CuBERT on each of the classification tasks and compare the results with multi-layered bidirectional LSTM (Hochreiter & Schmidhuber, 1997) models. We train the LSTM models from scratch and also using pre-trainined Word2Vec embeddings. Our results show that CuBERT con- sistently outperforms these baseline models by 2.9 9-22% across the tasks. We perform number of additional studies by varying the sampling strategies used for training Word2 models, by varying program lengths, and by comparing against Transformer models trained from scratch. In addition, we also show that CuBERT can be finetuned effectively using only 33% of the task-specific labeled data and with only 2 epochs, and that it attains results competitive to the baseline models trained with the full datasets and much larger number of epochs. CuBERT when finetuned on the variable misuse localization and repair task, produces high classification, localization and localization+repair accuracies. The contributions of this paper are as follows:
We present the first attempt at pre-training a BERT contextual embedding of source code.
We show the efficacy of the pre-trained contextual embedding on five classification tasks. Our results show that the finetuned models outperform the baseline LSTM models sup- ported by Verd2 embeddings, and Transformers trained from scratch. Further, the finetuning works well even for smaller datasets and fewer training epochs. We also evalu- ate CuBERT on multi-headed pointer prediction task.
We plan to make the models and datasets publicly available for use by others.
2 RELATED WORK
Given the abundance of natural-language text, and the relative difficulty of obtaining labeled data, much effort has been devoted to using large corpora to learn about language in an unsupervised fash- ion, before trying to focus on tasks with small labeled training datasets. Word2Vec (Mikolov al. 2013a|b) b) computed word embeddings based on word co-occurrence and proximity. but the same em- bedding used regardless of the context. The continued advances in word embeddings (Pennington et al. 2014) led to publicly released pre-trained embeddings, used in variety of tasks.
To deal with varying word context, contextual word embeddings were developed (MeCann et al., 2017 Peters et al.) 2018 Radford et al.) 2018 2019). in which an embedding is learned for the context of a word in a particular sentence, namely the sequence of words preceding it and possibly following it. BERT (Devlin et al. 2019) improved natural-language pre-training by using de- noising autoencoder. Instead of learning a language model, which is inherently sequential, BERT optimizes for predicting noised word within a sentence. Such prediction instances are generated by choosing a word position and either keeping it unchanged, removing the word, or replacing the word with random wrong word. It also pre-trains with the objective of predicting whether two sentences
2

---

Under review as a conference paper at ICLR 2020
can be next to each other. These pre-training objectives, along with the use of Transformer-basee architecture, gave BERT an accuracy boost in a number of NLP tasks over the state-of-the-aa BERT has been improved upon in various ways, including modifying training objectives, utilizing ensembles, combining attention with autoregression (Yang et al. 2019), and expanding pre-training corpora and time (Liu al.) (019). However, the main architecture of BERT seems hold up as the state-of-the-arr as of this writing.
In the space of programming languages, attempts have been made to learn embeddings in the context of specific ooftware-engineent
tasks. These include embeddings of variable and method identifiers using local and global context (Allamanis et al. 2015). abstract syntax trees or ASTs (Mou et al. 2016), paths in ASTs (Alon et al. al.]]2019). memory heap graphs (Li et al. 2016), and ASTs enriched with data flow information (Allamanis et al. 2018). These approaches require analyzing source code beyond simple tokenization. In this work, we derive a pre-trained contextual embedding of tokenized source code without explicitly modeling source spdessecii information, and show that the resulting embedding can be effectively finetuned for downstream tasks.
EXPERIMENTAL SETUP
3.1 CODE CORPUS FOR FINETUNING TASKS
We use the ETH Py150 corpus Raychev et al., 2016) to generate datasets for the finetuning tasks. The ETH 150 corpus consists of 150K Python files from GitHub, and is partitioned into a training split (100K files) and a test split (50K files). We held out 10K files from the training split as a validation split. We deduplicated the dataset in the fashion of Allamanis (2018), resulting in slightly smaller dataset of 85K, 9.5K, and 47K files in train, validation, and test, respectively.
3.2 THE GITHUB PYTHONN -RE-TRAINGNG CODE CORPUS
We used the public GitHub repository hosted on Google's BigQuery platform (the g withuubreepo dataset under BigQuery's 's public-data project, bigquery-pubblle --t-ppubdlddaa..... We extracted all files ending in py, under open-source, redistributable licenses. removed symbolic links, and retained only files reported to be in the refs maatte branch. This resulted in about 16. IM files.
To avoid duplication between pre-training and finetuning data, we removed files that had high sim- ilarity to the files in the ETH Py P5150 dataset, using the method of Allamanni (2018). In particular, two files are considered similar to each other if the Jaccard similarity between the sets of tokens (identifiers and string literals) is above 0.8 and in addition, it is above 0.7 for multi-sets of tokens. This brought the dataset to 13.5M files. We then further deduplicated the remaining files, by clus- tering them into equivalence classes holding similar files according to the same similarity metric, and keeping only one exemplar per equivalence class. This helps avoid biasing the pre-trained em- bedding. Finally, we removed files that could not be tokenized. In the end, we were left with 6.6M Python files containing over 2 billion words. This is our Python pre-training code corpus.
3.3 SOURCE CODE MODELING
We first tokenize a Python program using the standard Python tokenizer (the tokenize package). We leave language keywords intact and produce special tokens for syntactic elements that have either no string representation (e.g.. DEDENT tokens, which occur when a nested program scope concludes), or ambiguous interpretation (e.g., new line characters inside string literals, at the logical end of Python statement, or in the middle of a Python statement result in distinct special tokens). We split identifiers according to common heuristic rules (e-g., snake or Camel case). Finally, we split string literals using heuristic rules, on whitespace characters, and special characters. We limit all thus produced tokens to a maximum length of 15 characters. We call this the program vocabulary. Our Python pre-training code corpus contained 10.2M unique tokens, including 12 reserved tokens.
We greedily compress the program vocabulary into a subword vocabulary (Schuster & Nakajima 2012) using the SubwordTexx Encoder from the Tensor2Tense project (Vaswani et al. 2018)), resulting in slightly over 50K tokens. All words in the program vocabulary can be losslessly encoded using one or more of the subword tokens.
3

---

Under review as a conference paper at ICLR 2020
We encode programs first into program tokens, as described above, and then encode those tokens one by one in the subword vocabulary. The objective of this encoding scheme is to preserve syntactically meaningful boundaries of tokens. For example, the identifier "snake case" could be encoded as "sna ke ca se", preserving the snake case split of its characters, even the subtoken "e c" were very popular in the corpus; the latter encoding might result in smaller representation but would lose the intent of the programmer in using a snake-case identifier. Similarly, "i-0" may be encoded as separate tokens i, and O,
be very frequent in the corpus, but we still force ensuring that we preserve the distinction between operators and operands. Both the BERT model and the Vec embeddings are built on the subword vocabulary.
3.4 FINETUNING TASKS
To evaluate CuBERT, we design five classification tasks and multi-headed pointer task. These are motivated by prior work, but unfortunately, the associated datasets come from different languages and varied sources. We want the tasks to be on Python code, and for accurate results, we ensure that there is no overlap between pre-training and finetuning datasets. We therefore create all the tasks on the ETH Py150 corpus (see Section B.I). As discussed in Section 3.2 we ensure that there is no duplication between this and the pre-training corpus. We hope that our datasets for these tasks will be useful to others as well. The finetuning tasks are described below. more detailed discussion is presented in Appendix A
Variable Misuse Classification Allamanis al. (2018) observed that developers may mistakenly use an incorrect variable in the place of correct one. These mistakes may occur when developers copy-paste similar code but forget to rename all occurrences of variables from the original fragment, or when there are similar variable names in contexts that can be confused with each other. These can be subtle errors that remain undetected during compilation. The task by Allamanni et al. (2018) is to predict correct variable name at location within function and was devised on C# programs. We take the classification version restated by Vasic et al. (2019). wherein, given a function, the task is to predict whether there is a variable misuse at some location in the function, without specifying a particular location to consider. In this setting, the classifier has to consider all variables and their usages to make the decision. In order to create negative (buggy) examples, replace variable use at some location with another variable that is defined within the function.
Wrong Binary Operator Pradel & Sen (018)) proposed the task of detecting whether binary operator in a given expression is correct. They use features extracted from limited surrounding context. We use the entire function with the goal of detecting whether any binary operator in the function is incorrect. The negative examples are created by randomly replacing some binary operator with another type-compatible operator.
Swapped Operand Pradel Sen (2018) propose the wrong binary operand task where a variable or constant is used incorrectly in an expression, but that task is quite similar to the variable misuse task we already use. We therefore define another class of operand errors where the operands of non- commutative binary operators are swapped. The operands can be arbitrary subexpressions, and are not restricted to be just variables or constants. To simplify example generation, we restrict examples for this task to those in which the binary operator and its operands all fit within single line.
Dunction-Doestring Mismatch Developers are encouraged write descriptive docstrings to ex- plain the functionality and usage of functions. This provides parallel corpora between code and natural language sentences that have been used for machine translation between the two (Barone & Sennrich 2017). detecting uninformative docstrings (Louis et al. 2018) and to evaluate their utility to provide supervision in neural code search (Cambronero et al. 2019). We prepare a sentence-pair classification problem where the function and its docstring form two distinct sentences. Similar to the other finetuning tasks, we use the ETH 150 corpus to create this dataset. The positive ex- pairs We create negative examples by replacing
amples come from the correct function-docstrin correct docstrings with docstrings of other functions, randomly chosen from the dataset. For this task, the existing docstring is removed from the function body.
4

---

Under review as a conference paper at ICLR 2020
Train
Validation
Test
Variable Misuse Classification
796020
8192 (86810)
429854
Wrong Binary Operator
537244
8192 (59112)
293872
Swapped Operand
276116
8192 (30818)
152248
Conecion-Docstrrin
391049
8192 (44029)
213269
Exception Type
21694
2459
(2459)
12036
Variable Misuse Localization and Repair
796020
8192 (86810)
429854
Table 1: Benchmark finetuning datasets. Note that for validation, we have subsampled the original datasets (in parentheses) down to 8192 examples, except for exception classification, which only had 2459 validation examples, all of which are included.
Exception Type While it is possible to write generic exception handlers (e.g., "except Exception" Python), it is considered good coding practice to catch and handle the precise exceptions that be raised by code fragment. We identified the 20 most common exception types from the GitHub dataset, excluding the catch-all Exception intiiinn on (full list in Table Given a function with except clause for one of these exception types, we replace the exception with a special "hole" token. The task is the multi-class classification problem of predicting the original exception type.
Variable Misuse Localization and Repair As an instance of a non-classification task, we con- sider the joint classification, localization and repair version of the variable misuse task from Vasic jet al. (2019). Given a function, the task is to predict one pointer (called the localization pointer) identify variable misuse location and another pointer (called the repair pointer) identify variable from the same function that is the right one to use at the faulty location. The model is also trained classify functions that do not contain any variable misuse as bug-free by making the localization pointer point to special location in the function. We create negative examples using the same method as used in the Variable Misuse Classification task.
Table lists the sizes of the resulting benchmark datasets extracted from the (deduplicated) ETH Py150 corpus. The Exception Type task contains fewer examples than the other tasks, since exam- ples for this task only come from functions that catch one of the chosen 20 exception types.
3.5 BERT FOR SOURCE CODE
The BERT model (Devlin et al. 2019) consists of multi-layered Transformer encoder. It trained with two tasks: (1) to predict the correct tokens in a fraction of all positions, some of which have been replaced with incorrect tokens or the special [MASK] token (the Masked Language Model task) and (2) predict whether the two sentences separated by the special [SEP] token follow each other in some natural discourse (the Next Sentence Prediction task). Thus, each example consists of one (for MLM) or (for NSP) senfences, where sentence is the concatenation of contiguous lines from the source corpus, sized to fit the target example length. To ensure that every sentence treated in multiple instances of both MLM and NSP, BERT by default duplicates the corpus 10 times, and generates independently derived examples from each duplicate. With 50% probability, the second example sentence comes from random document (for NSP). With 15% probability, token is chosen for an MLM prediction (up to 20 per example), and from those chosen, 80% are masked, 10% are left undisturbed, and 10% are replaced with a random token.
CuBERT is similarly formulated, but CuBERT sentence is a logical code line, defined by the Python standard. Intuitively, a logical code line the shortest sequence of consecutive lines that may constitute legal statement, c.g., it has correctly matching parentheses. We count example lengths by counting the subword tokens of both sentences (see Section! 3.3).
We train the BERT Large model, consisting of 24 layers with 16 attention heads and hidden size of 1024 units. Sentences are created by parsing our pre- pre-training dataset. Task-specific classifiers pass the embedding of special offet-lf [CLS] token through feedforward and softmax layers. For the pointer prediction task, the pointer is computed over the sequence of outputs generated by the last layer of the BERT model.
5

---

Under review as a conference paper at ICLR 2020
3.6 BASELINES
3.6.1
WORD2VEC
We train Word2Vee models using the same pre-training corpus the BERT model. To maintain parity, we generate the dataset for Word2Vec using the same pipeline as BERT but by disabling masking and generation of negative examples for NSP. The dataset is generated without any dupli- cation. We train both CBOW and Skipgram models using GenSim (Rehūřek & Sojka 2010). To deal with the large vocabulary, we use negative sampling and hierarchical softmax (Mikolov et al. 2013a b) to train the two versions. In all, we obtain four Word2 Vec embeddings.
3.6.2 BIDIRECTIONAL LSTM AND TRANSFORMER
In order obtain context-sensitive encodings of input sequences for the finetuning tasks, we multi-layered bidirectional LSTMs (Hochreiter & Schmidhuber] 1997) (BiLSTMs). These are initialized with the pre-trained Word2Vec embeddings. Additionally, to further evaluate whether LSTMs alone are sufficient without pre-training. we try initializing the BiLSTM with an embedding matrix that is trained from scratch. We also trained Transformer models (Vaswani et al. (017)) for our finetuning tasks. We used BERT's own Transformer implementation, to ensure comparability of results. For comparison with prior work, we use the unidirectional LSTM and pointer model from (Vasic et al. 2019) for the Variable Misuse Localization and Repair task.
4 EXPERIMENTAL RESULTS
4.1 TRAINING DETAILS
As stated above, CuBERT's dataset generation duplicates the corpus 10 times, whereas Word2 is trained without duplication. To compensate for this difference, we trained Werd2 for 10 epochs and CuBERT for I epoch. We pre-train CuBERT with the default configuration of the BERT Large model. For sequences of length 128, 256 and 512, we use batch sizes of 8192, 4096 and 2048 respectively. For Word2 when training with negative samples, we choose 10 negative samples. The embedding sizes for all the pre-trainn pre-trained models are set at 1024.
For the baseline BiLSTM models, we did extensive experimentation on the Variable Misuse task by varying the number of layers (1-3) and the number of hidden units (128, 256, 512). We also tried LSTM output dropout probability 10.1, 0.5), optimizers (Adam (Kingma & Ba 2014) and Ada- Grad (Duchi et al., 201T),, and learning rates (Ie-3, 1e-4, 1e-5). The most promising combination was a 3-layered BiLSTM with 512 hidden units per layer, LSTM output dropout probability of 0.1 and Adam optimizer with learning rate of le-3. We use this set of parameters for all the tasks ex- cept the Exception Type task. Due to the much smaller dataset size of the latter (Table [T]. did a separate search and chose a single-layer BiLSTM with 256 hidden units. We used the batch size of 8192 for the larger tasks and 64 for the Exception Type task. For the baseline Transformer mod- els, we originally attempted to train a Transformer model of the same configuration as CuBERT. However, the size of our training dataset seemed too small to train that large a Transformer. Instead, we performed a hyperparameter search over transformer layers (1-6), hidden units (128, 256, 512), learning rates (5e-5, le-4, 5e-4, le-3) and batch sizes (64, 256, 1024, 2048, 4096, 8192) on the Variable Misuse task. The best architecture (4 layers, 512 hidden units, 16 attention heads, learning rate of 5e-4, batch size of 4096) used for all the tasks except the Exception Type task. A separate experimentation for the smaller Exception Type dataset resulted in the best configuration of 3 layers, 512 hidden units, 16 attention heads, learning rate of 5e-5, and batch size of 2048.
Finally, for our baseline pointer model (referred to LSTM+pointer below) we searched over the following hyperparameter choices: hidden sizes of 512 and 1024, token embedding sizes of 512 and 1024, learning rates of 0.1, 0.01, and 0.001, and the AdaGrad and Gradient Descent optimizers. In contrast to the original work, we generated one pair of buggy/bug-ffe than one per variable use, per function, which would bias towards longer functions), use CuBERT's subword wwwwoordtokkeizeed vocabulary of 50K subtokens (rather than a limited full-token vocabulary, which leaves many tokens out of vocabulary).
examples per function (rather

---

Under review as a conference paper at ICLR 2020
Setting
Misuse
Operator
Operand
Docstring
Exception
From scratch
76.05%
82.00%
87.77%
78.43%
40.37%
77.66%
84.42%
88.66%
89.13%
48.85%
BiLSTM (100 epochs)
CBOW
hs
77.01
84. 44.11% 11%
89.69%
86.74%
46.73%
71.58%
83.06%
87.67%
84.69%
48.54%
Skipgram
hs 77.21%
83.06%
89.01 %
82.56%
49.68%
2 epochs
90.09%
85.15%
88.67%
95.81 11%
52.38%
CuBERT
epochs
92.73%
88.43%
88.67%
95.81%
62.55%
20 epochs
94.61%
90.24%
92.56%
96.85%
71.74%
Transformer (100 epochs) 79.37%
78.66%
86.21% %
91.10%
48.60%
Table 2: Test accuracies of finetuned CuBERT against BiLSTM (with and without Word2 embeddings) and Transformer trained from seratch on the classification tasks. "ns" and "hs" re- spectively refer to negative sampling and hierarchical softmax settings used for training CBOW and Skipgram models. "From scratch" refers to training with freshly initialized token embeddings, that is, without pre-trained Word2 embeddings.
4.2 RESEARCH QUESTIONS
We set out to answer the following research questions. We will address each with our results.
1. Do contextual embeddings help with source-code analysis tasks, when pre-trained on an unlabeled code corpus? We compare CuBERT to BiLSTM models with and without pre- trained Word2Vec embeddings on the classification tasks (Section 4.3).
2. Does finetuning actually help, or is the Transformer model behind CuBERT the main power behind the approach? We compare finetuned CuBERT models Transformer-basee mod- els trained from scratch on the classification tasks (Section 4.4b.
3. How does the performance of CuBERT on the classification tasks scale with the amount of labeled training data? We compare the performance of finetuned CuBERT models when finetuning with one third, two thirds, or the full training dataset for each task (Section 4.5)
4. How does example length affect the benefits of CuBERT? We compare finetuning perfor- mance for different example lengths on the classification tasks (Section 4.6).
5. How does CuBERT perform on complex tasks? We implemented and finetuned model for multi-headed pointer prediction task, namely, the Variable-Misuse Localization and Repair task (Section 4.7 We compare it to the model from Vasic et al. (2019).
all the results are presented for sequences of length 512. We give examples of classification instances in Appendix[B] Baand include visualizations attention weights for them.
Except for Section 4.6
4.3 CONTEXTUAL VS. WORD EMBEDDINGS
The purpose of this analysis is to understand how much pre-trained contextual embeddings help, compared word embeddings. For each classification task, we trained BiLSTM models starting with each of our baseline Word2Vec embeddings, namely, continuous bag of words (CBOW) and Skipgram trained with negative sampling or hierarchical softmax. In all the models, the Word2 Vec embeddings can be refined during training. Within the first 100 epochs, the performance of the BiL.TMM models stopped improving. The best model weights per task were selected by finding the minimum validation loss the corresponding dataset (Table I over the first 100 epochs. On the CuBERT side, we finetuned the pre-trained model for 20 epochs, with similar model selection.
The resulting test-split accuracies are shown in Table CuBERT consistently outperforms BiLSTM (with the best task-wise Word2Vec configuration) on all tasks, by a margin of 2.9.22%. Thus, the pre-trained contextual embedding provides superior results even with a smaller budget of 20 epochs, compared to the 100 epochs used for BiLSTMs. The Exception Type classification task is an interesting case since it has order of magnitude less training data than the other tasks (see

---

Under review as a conference paper at ICLR 2020
Best of # Epochs
Train Fraction
Misuse
Operator
Operand
Docstring
Exception
100%
90.09%
85.15%
88.67%
95.81 %
52.38%
2
66%
89.52%
83.26%
88.66%
95.17%
34.70%
33%
88.64%
82.28%
87.45%
95.29%
26.87%
100%
92.73%
88.43%
88.67%
95.81%
62.55%
10
66%
92.06%
87 .6.6%
90.39%
95.64%
64.59%
33%
91.23%
84.44%
87.45%
95.48%
54.22%
100%
94.61% %
90.24%
92.56%
96.85%
71.74%
20
66%
94.19%
89.36% 87.67%
92.01 %
96.17% 96.37%
70.11%
33%
93.54%
91.30%
67.72%
Table Effects of reducing training-spi split size on finetuning performance on the classification tasks.
Table D.. The difference between the performance of BiLSTM and CuBERT is the highest for this task. Thus, finetuning is of much value for tasks with limited labeled training data.
We analyzed the performance of CuBERT with the reduced finetuning budget of only 2 and 10 epochs (see Table 2). Except for the Operand task, CuBERT outperforms BiLSTM within 2 fine- tuning epochs. On the Operand task, the performance difference between CuBERT with finetuning epochs and BiL.TTM is about 1%. For the rest of the tasks, CuBERT with only 2 finetun- ing epochs outperforms BiL.STM (with the best task-wise Word2Vec configuration) by a margin of 0.7-12%. This shows that CuBERT can reach accuracies that are comparable to or better than those of BiLSTMs trained with Word2Vec embeddings within only a few epochs.
10
We also trained the BiLSTM models from scratch, that is, without using the Word2Vec embeddings. The results are shown in the first row of Table 2 Compared to those, the use of Word2 embed- dings performs better by margin of 1.5-10.5% Though single Word2Vec configuration is the best, CBOW trained with negative sampling gives the most consistent results overall.
4.4 TRANSFORMER ALL YOU NEED?
One may wonder if CuBERT's promising results derive more from using a Transformer-base model for its classification tasks, and less from the actual, unsupervised pre-training. Here we compare our results on the classification tasks Transformer-bass based model trained from scratch, i.e., without the benefit of pre-trained embedding. All the models were trained for 100 epochs during which their performance stopped improving. We selected the best model per task using least validation loss. As seen from the last row of Table the performance of CuBERT is substantially higher than the Transformer models trained from scratch. We therefore conclude that pre-training is crucial to CuBERT's success.
4.5
THE EFFECTS OF LITTLE SUPERVISION
The big draw of unsupervised pre-training followed by finetuning is that some tasks have small labeled datasets. We study here how CuBERT fares when the size of its training split is reduced. We sampled uniformly the training split of ETH Py150 down to 2/3rds and 1/3rd of its original size. and produced training datasets for each of the classification tasks from each sub-split. We then finetuned the pre-trained CuBERT model with each of the 3 different training splits. Validation and testing were done with the same original datasets. Table shows the results.
The Function Docstring task seems robust to the reduction of the training dataset, both early and late in the finetuning process (that is, within 2 vs. 20 epochs), whereas the Exception Classification task heavily impacted by the dataset reduction, given that it has relatively few training examples to begin with. Interestingly enough, for some tasks, even finetuning for only 2 epochs and only using third of the training data outperforms the baselines. For example, for both Variable Misuse and Function Docstring, CuBERT at epochs and 1/3rd training data outperforms the BiLSTM with Word2 and the Transformer baselines.
8

---

Under review as a conference paper at ICLR 2020
Length
Misuse
Operator
Operand
Docstring
Exception
128
85,89%
77.92%
77.17%
97.10%
55.95%
256
92.69%
86.52% 90.24%
87.26%
97.08% 96.85%
65.38%
512
94.61% %
92.56%
71.74%
Table Best out of 20 epochs of finetuning, for three example lengths, on the classification tasks.
Model
Setting
True Positive
Classification Accuracy
Localization Accuracy
Loc+Repair Accuracy
LSTM+pointer
100 epochs
81.63%
78.76%
63.83%
56.37%
epochs
97.18%
89.37%
79.05%
75.84%
Coinerr
10 epochs
94.94%
93.05% %
88.52%
85.91 %
20 epochs
96.83%
94.85 %
91.11 %
89.35 %
Comparison of the finetuned CuBERT+poinde model and the LSTM+pointer model from Vasic et al. (2019) on the variable misuse localization and repair task.
Table 5:
4.6 THE EFFECTS REDUCING CONTEXT
Context size especially useful in code tasks, given that some relevant information may lie many "sentences" away from its locus of interest. Here we study how reducing the context length (i.e., the length of the examples used pre-train and finetune) affects performance. We produce data with shorter example lengths by following the standard BERT mechanism. Table shows the results.
Although context seems be important most tasks, the Function Docstring task seems improve with reduced context. This may be because the task primarily depends on comparison between the docstring and the function signature, and including more context dilutes the model's focus.
For comparison, we also evaluated the BiLSTM model on sequences of length 128 and 256 for the Variable Misuse task. We obtained accuracies of 71.34% and 73.63% respectively, which are lower than the best BiL.TMM accuracy sequence length 512 and also lower than the accuracies of CuBERT for the corresponding lengths (see Table
4.7 EVALUATION ON MULTI-HEADED POINTER PREDICTION TASK
We now discuss the results of finetuning CuBERT to predict the localization and repair pointers for the variable misuse task. For this task, we implement the multi-headed pointer model from Vasic al. (209)) top of CuBERT. The baseline consists of the same pointer model tional LSTM as used in Vasic et af. (2019). We refer to these respectively as CuBERT+poine LSTM+pointer models, respectively. Due limitations of space, we omit the details of the pointer model and refer the reader to the above paper. As reported in Section 4 of Vasic et al. (2019). to enable comparison with an enumerative approach, the evaluation was performed only on 12K test examples. In comparison, we report the numbers on all 430K test examples (Table I for both the models.
unidirec-
and
Similar to other tasks, we trained the baseline model for 100 epochs and finetuned CuBERT for up 20 epochs. Jlives the results along the same metrics as Vasic et al. (2019). The metries are defined as follows: 1) True Positive is the percentage of bug-free functions classified as bug-free. 2) Classification Accuracy the percentage of correctly classified examples (between bug-free and buggy). 3) Localization Accuracy is the percentage of buggy examples for which the localization pointer correctly identifies the bug location. 4) Localization-Reppla Accuracy is the precentage of buggy examples for which both the localization and repair pointers make correct predictions. As seen from Ta the CuBERT+pointer model outperforms the LSTM+pointer model consistently across all the metrics, and within 2 and 10 epochs.

---

Under review as a conference paper at ICLR 2020
5 CONCLUSIONS AND FUTURE WORK
We present the first attempt at pre-trained contextual embedding of source code by training a BERT model, called CuBERT, which we finetuned on five classification tasks and compared against BiL- STM with Word2 embeddings and Transformer models. As a more challenging task, we also evaluated CuBERT on a multi-headed pointer prediction task. CuBERT outperformed the baseline models consistently. We evaluated CuBERT with less data and fewer epochs, highlighting the bene- fits pre-training on a massive, unsupervised code corpus. We see this as a promising step towards source-code understanding, and plan to explore its utility other programming languages and tasks. We use a tokenized representation of source code and leave it to the underlying Transformer model to infer any structural interactions between the input tokens through its self-attention mechanism. However, the literature on deep learning for source code also demonstrates the utility of encoding explicit structural information such as data-flow information (Allamanis et al. 2018). The Trans- former model has been extended to incorporate explicitly provided structural information (Shaw al., 2018). Using such relation- aware Transformers for pre-training source code representations will be an important direction to explore in future.
REFERENCES
Miltiadis Allamanis. The adverse effects of code duplication in machine learning models of code. CoRR, abs/1812 112.06669,, 2018.
1812. 04469
Miltiadis Allamanis, Earl Barr, Christian Bird, and Charles Sutton. Suggesting accurate method and class names. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pp. 38-49, New York, NY, USA, 2015, ACM. ISBN 978-1- 4503-3675-8. doi: 10. 111444/776660527777888888 URL http:://oi: acm 0776805.2786822
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning represent programs with graphs. In International Conference on Learning Representations, 2018.
Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. Code2vec: Learning distributed rep- resentations of code. Proc. ACM Program. Lang, :(POPL):40:1440:2 January 2019. ISSN 2475-1421. doi: 10.1145/3290353, URL httt: /oi.. acm. org/10..gooooooo 1145/ 3290353
Antonio Valerio Miceli Barone and Rico Sennrich. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv: 2017.
Jose Cambronero, Hongyu Li, Seohyun Kim, Koushik Sen, and Satish Chandra. When deep learning met code search. arXiv preprint arXiv:1905.03813 2019.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Vi Vi'egas, and Martin Wattenberg. Visualizing and measuring the geometry of bert. ArXiv, 0bbs/190600277 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-144 1423. URL https 7/ w/www.co wwwweb. org/ http://wwwwogggattotoooeeeeeeeee http://aaaatoooog///---------
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(1u))2121-2159, 2011.
Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. Deep api learning. In Proceed- ings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software En- gineering, FSE 2016, pp. 631-642, New York, NY, USA, 2016. ACM. ISBN 978- --53---48-84- URL
6. doi: 10. 00.114//25002002955000
http: //0::///oo. doi. acm. 12t...0.....////922222
2950334
10

---

Under review as a conference paper at ICLR 2020
Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. Deepfix: Fixing common lan- guage eπrors by deep learning. In Proceedings of the Thirty-First AAAI Conference on Artifi- cial Intelligence, AAAI 17, pp. 1345-1351. AAAI Press, 2017. WtRLhtt::// /tt./alcccccm///// org/
citat
ion. oon cfm?
3298436
A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu, On the naturalness software. 2012
34th International Conference on Software Engineering (ICSE), pp. 837-847, June 2012. doi: 00.1109//CSE.2212.2221
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 98):
1780, November 1997. ISSN 0899-7667. doi: 10.1 1e1116/6....19999 1997.9.8.77355. doi org.oorg/00 10. 1162//eco...oo 1997. 8.......877......
Srinivasan lyer, loannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. In Proceedings the 2018 Conference on Empirical Methods in Nat- ural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 1643-1652,
2018. URL
www. aclweb. org/ anthooogyoooii 1192/
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard S. Zemel. Gated graph sequence neural networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016. URL httt:://aara
org/ abs/ 1511. 05493
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11699 1662,, 2019, /
abs 1907.10700 abs.//91..111.. 11692]
Annic Louis, Santanu Kumar Dash, Earl T Barr, and Charles Sutton. Deep learning to detect redun- dant method comments, arXiv preprint arXiv: 0666.4416, 2018
Robert C. Martin. Clean Code: A Handbook of Agile Software Crafismanship. Prentice Hall PTR. Upper Saddle River, NJ, USA, edition, 2008 ISBN 0132350882, 9780132350884
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6294-6303 Curran Associates, Inc., 2017. URL httt::/ttpppeeeeeeee nips cc/paper/ pdf
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word repre- sentations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scotisdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013a. URL
http://arxiva wwtp:iii.rrg/aas.. abs 1301.11120 3781
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 1131199 Curran Associates, Inc., 2013b.
Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. Convolutional neural networks over tree structures for programming language processing. In Proceedings of the Thirtieth AAAI Con- ference on Artificial Intelligence, AAAI'1,, pp. 1287-1293. AAAI Press, 2016. URL http:
7/dl. acm.
cm 1id 3016002,
Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. Learning to generate pseudo-code from source code using statistical machine translation (t), In 2015 30th IEEE/ACM International Conference on Automated Software Engi- neering (ASE), pp. 574-584. IEEE, 2015.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In In EMNLP, 2014.
11

---

Under review as a conference paper at ICLR 2020
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL-LIT, pp. 2018.
Michael Pradel and Koushik Sen. Deepbugs: A learning approach to name-based bug detection. Proc. ACM Program. Lang. 1400 October 2018. ISSN 2475-1421. doi: 10.1145/3276517. URL http:hhtttooooo.. acm org/10. 114553325513 30t.......0111111113333117
Yewen Pu, Karthik Narasimhan, Armando Solar-Lezama, and Regina Barzilay. Sk.p: A neural program corrector for moocs. In Companion Proceedings of the 2016 ACM SIGPLAN Inter- national Conference on Systems, Programming, Languages and Applications: Software for Hu- manity, SPLASH Companion 2016, pp. 39-40, New York, NY. USA, 2016. ACM ISBN 978- 1-4503-4437-1 doi: 10. 10.11455/984443.2288889 29:4043 2989222
URL http: acm. org 1145/
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training. URL https://s3-u--wex-22 amuzonaws. com/openai- understanding paper. pdf. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language mod- els. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '14, pp. 419-428, New York, NY, USA, 2014. ACM. ISBN 978- 1-4503-2784-8. doi: 10. (0.11445/594291.55443) 0594291.2594322
URL http: //o:: acm. org 12///10.111 10.
Veselin Raychev, Pavol Bielik, and Martin T. Vechev. Probabilistic model for code with decision trees. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriee Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, part of SPLASH 2016, Amsterdam, The Netherlands, October 30- November 4, 2016, pp. 731-747, 2016.
Radim Rehüřek and Petr Sojka. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pp. 45-50, Valletta, Malta, May 2010. ELRA. http://iss wwnii c2/ http:///pwwpttttccceeooocc /tw.:o////8493/ee
Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In International Confer- ence on Acoustics, Speech and Signal Processing, pp. 5149-5152, 2012.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- 2018.
tions. arXiv preprint arXiv:1803.022
Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh Singh. Neural program repair by jointly learning to localize and repair. CoRR, abs/1904.01120, 2019. URL http:
//arxiv. wwciiv.rrrgaaaa abs///b///// 904.00400 01 720
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017. URL all- all-you-neeee. pdf
http::/ nips. wwt.ee//ppppa http///pppp////1///// htt.://11aaatttiiiooooo
Ashish Vaswani, Samy Bengio, Eugene Brevdo, François Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. Tensor2tensor for neural machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americus, AMTA 2018, Boston, MA, USA, March 17-21, 2018 Volume 1: Research Papers, pp. 193-199, 2018. URL https: www. wwwwwb.ww........... org anthology W18- 1819/
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237, 2019. http: www.aawwwv....raaaaommmo abs/ 6906. 08237
12

---

Under review as a conference paper at ICLR 2020
Train
Exception Type
Test
Validation
100%
66%
33%
ValueError
2324
477
4058
2715
1344
KeyError
2240
453
4009
2566
1271
AttributeError
1657
311
2895
1896
876
TypeError
913
187
1747
1175
564
OSError
891
164
1641
1106
543
I0Error
865
168
1560
1046
560
ImportError
776
202
1372
935
471
IndexError
694
153
1197
813
408
DoesNotExist
6
2
0
0
KeyboardInterrupt
287
590
408
223
StopIteration
307
69
488
302
155
AssertionBrror
177
32
397
276
158
SystemExit
139
23
264
173
101
Runt.imEError
128
36
299
203
104
НТTPError
59
13
119
80
35
UnicodetecodeBrror
151
24
251
173
82
Not.ImplemennteEErro
127
27
222
136
52
ValidationError
95
15
172
121
58
objectboesNotExist:
105
213
142
64
NameError
95
19
197
124
56
Table 6: Example counts class for the Exception Type task, broken down into the dataset splits. We show separately the 100% train dataset, as well as its 33% and 66% subsamples used in the ablations.
A DATA PREPARATION FOR FINETUNING TASKS
A.1 LABEL FREQUENCIES
All four of our binary-classification finetuning tasks had an equal number of buggy and bug-free examples. The Exception task, which is a multi-class classification task. had different number of examples class (i.e., exception types). We show the breakdown of example counts per label for our finetuning dataset splits in T
A.2 FINETUNING TASK DATASETS
In this section, we describe in detail how we produced our finetuning datasets (Section 3.4
common primitive in all our data generation is splitting a Python module into functions. We do this by parsing the Python file and identifying function definitions in the Abstract Syntax Tree that have no other function definition between themselves and the root of the tree. The resulting functions include functions defined at module scope, but also methods of classes and subclasses. Not included are functions defined within other function and method bodies, or methods of classes that are, themselves, defined within other function or methods bodies.
We do filter functions by length, although task aask-specffic data generation may filter out some func- tions (see below). When generating examples for fixed-length pre-training or finetuning model, we prune all examples to the maximum target sequence length (in this paper we consider 128, 256, and 512 subtokenized sequence lengths). Note that if synthetically generated buggy/bug-fee pair differs only at a location beyond the target length (say on the 600-th subtoken), we still retain both examples. For instance, for the Variable Misuse Localization and Repair task, retain both buggy and bug-free examples, even the error and/or repair locations beyond the end of the maximum target length.
example
13

---

Under review as a conference paper at ICLR 2020
Commutative
Non-Commutative
Arithmetic
+,
4,4, %
Comparison Membership Boolean
I=, 18, is not
<, in, not in
< >=
and, or
Table 7: Binary operators.
A.2.1 REPRODUCIBLE DATA GENERATION
We make pseudorandom choices various stages in finetuning data generation. It was impor- tant to design pseudorandomness mechanism that gave (a) reproducible data generation, (b) non- deterministic choices drawn from the uniform distribution, and (c) order independence. Order in- dependence is important because our data generation is done in distributed fashion (using Apache Beam), so different pseudorandom number generator state machines are used by each distributed worker.
More specifically, pscudorandomness. is computed based on an experiment-wice seed, but is inde- pendent of the order in which examples generated. Specifically, to make a pscudorandom choice about a function, we hash (using MD5) the seed and the function data (its source code and metadata about its provenance), and use the resulting hash as a uniform pseudorandom value from the func- tion, for whatever needs the data generator has (e.g., in choosing one of multiple choices). In that way, the same function will always result in the same choices given a seed, regardless of the order in which each function is processed, resulting in reproducible dataset generation.
To choose among multiple choices, we hash the function's 's pseudorandom value along with all choices (sorted in canonical order) and use the digest to compute an index within the list of choice. Note that given two choices over different candidates but for the same function, independent de- cisions will be drawn. We also use such order-independent pseudorandomnes: when subsampling datasets (e.g., to generate the validation datasets). In those cases, we hash sample with the seed, as above, and turn the resulting digest into pseudorandom number in [0, 1]. which can be used to decide given a target sampling rate.
A.2.2 VARIABLE MISUSE CLASSIFICATION
A variable use is any mention of a variable in a load scope. This includes a variable that appears in the right-hand side of an assignment, or field dereference. We regard as defined all variables mentioned either in the formal arguments of a function definition, or on the right-hand side of an assignment. We do include in our defined variables those declared in module scope (i.e., globals).
To decide whether to generate examples from a function, we parse it, and collect all variable-use locations, and all define variables, described above. We discard the function if it has variable uses, or if it defines fewer than two variables; if there is only one variable defined, the problem of detecting variable misuse is moot. For any function that do not discard, we generate buggy and a bug-free example, as described next.
To generate buggy example from function, we choose one variable use pseudorandomly (see above how multiple-choice decisions are done), and replace its current occupant with different pseudorandomly-chen chooeen variable defined in the function (with a separate multiple-choice decision).
A.2.3 WRONG BINARY OPERATOR
This task considers both commutative and non-commutative binary operators (unlike the Swapped- Argument Classification task), See Table for the full list, and note that we have excluded relatively infrequent operators, e.g., the Python integer division operator //.
If function has no binary operators, it is discarded. Otherwise, it is used to generate bug-free example, and a single buggy example as follows: one of the operators is chosen pseudorandomly (as described above), and different operator chosen to replace it in the same of the Table 7 So, for instance, a buggy example would only swap == with is, but not with not in, which would not type-check if we performed static type inference on Python.
14

---

Under review as a conference paper at ICLR 2020
We take appropriate ensure the code parses after bug is introduced. For instance, if swap the operator in the expression 1==2 with is, we ensure that there is space between the tokens (i.e., 1 is rather than the incorrect 1 is 2), even though it was not needed before.
A.2.4 SWAPPED OPERAND
Since this task targets swapping the arguments of binary operators, we only consider non- commutative operators from Table
Functions without eligible operators are discarded, and the choice of the operator to mutate a function, well the choice of buggy operator use, done above, but limiting choices only on non-commutative operators.
To avoid complications due to format changes, only consider expressions that fit in single line (in contrast to the Wrong Binary Operator Classification task). We also do not consider expressions that look the same after swapping (e.g., a a).
A.2.5
D MISMATCH
In Python, a function docstring is a string literal that directly follows the function signature and before the main function body. Whereas in other common programming languages, the function documentation is a comment, in Python it is an actual, semantically meaningful string literal.
We discard functions that have no docstring from this dataset, or functions that have an empty docstring. We split the rest into the function definition without the docstring, and the docstring summary (i.e., the first line text from its docstring), discarding the rest of the doestring.
We create bug-free examples by pairing function with its own docstring summary.
To create buggy examples, we pair every function with another function's 's docstring summary, ac- cording to global pseudorandom permutation of all functions: for all i, we combine the i-th func- tion (without its docstring) with the P;-th function's doestring summary, where P a pseudorandom permutation, under given seed. We discard pairings in which == Pit], but for the seeds chose, no such pathological permuted pairings occurred.
A. EXCEPTION TYPE
Note that, unlike all other tasks, this task has notion of buggy bug-free free examples.
We discard functions that do not have any except clauses in them.
For the rest, we collect all locations holding exception types within except clauses, and choose one of those locations to query the model for classification. Note that single except clause may hold comma- senaaate list of exception types, and the same type may appear in multiple locations within a function. Once a location is chosen, we replace with special HCLE.... token, and create classification example that pairs the function (with the masked exception location) with the true label (the removed exception type).
The count of examples per exception type be found in Table
A.2.7 VARIABLE MISUSE LOCALIZATION AND REPAIR
The dataset for this task is identical to that for the Variable Misuse Classification task (Sec- tion A.2.2p. However, unlike the classification task, examples contain more features relevant to localization and repair. Specifically, in addition to the token sequence describing the program, we also extract number of boolean input masks:
A candidates mask, which marks True all tokens holding variable, which can therefore be either the location of bug, or the location of repair. The first position is always candidate, since may be used to indicate a bug program.
A targets mask, which marks as True all tokens holding the correct variable, for buggy examples. Note that the correct variable may appear in multiple locations in function,
15

---

Under review as a conference paper at ICLR 2020
Attention at layer 23
[CLS]
resize
sIlf
event
NEWLINE
event
100ly
200m
DEDENT
[SEP]
[CLS]
restte
200m
[SEP]
INDENT
DEDENT
NCWLNN
def on_resize (self, event)
event. appeytapplyyzoom/
Figure 1: Variable Misuse Example. In the code snippet, 'event. apply __zoom! should actually be 'self apply-zoom' The CuBERT variable-misuse model correctly predicts that the code has an error. As seen from the attention map, the query tokens are attending to the second occurrence of the 'event' token in the snippet, which corresponds to the incorrect variable usage.
therefore this mask may have multiple True positions. Bug-free examples have an all-False targets mask.
An error-location mask, which marks as True the location where the bug occurs (for buggy examples) or the first location (for bug free examples).
All the masks mark True some of the locations that hold variables. Because many variables subtokenized into multiple tokens, if variable is to be marked as True the corresponding mask, only mark True its first subtoken, keeping trailing subtokens as False.
B ATTENTION VISUALIZATIONS
In this section, provide sample code snippets used test the different classification tasks. Fur- ther, Figures [ show visualizations of the attention matrix of the last layer of the finetuned Cu- BERT model (Coenen al. 2019) for the code snippets. In the visualization, the Y-axis shows the query tokens and X-axis shows the tokens being attended to. The attention weight between pair of tokens is the maximum of the weights assigned by the multi-head attention mechanism. The color changes from dark to light weight changes from 0 to 1.
16

---

Under review as a conference paper at ICLR 2020
Attention at layer 23
[CLS]
01
sel
isinstance
and
other
NEWLINE
self
value
DEDENT
return
not
NEWLINE
(SEP)
ethor
ather
Ink
def gt_ (self, other):
if isinstance (other, int) and other==0: self.get_valuee return other is not sels
Figure 2: Wrong Operator Example. In this code snippet, 'other is not self' should actu- ally be 'other < self'. The CuBERT wrong- wrong-binary-opeer binaryooperatoor model correctly predicts that the code snippet has error. As seen from the attention map, the query tokens all attending the incorrect operator 'is'.
17

---

Under review as a conference paper at ICLR 2020
Attention at layer 23
[CLS]
INDENT
def
ds
model
NEWLINE
INDENT
return
ds
registry
model
DEDENT
DEDENT
NEWLINE
[SEP]
[CLS]
contains
ds
registry
model
[SEP]
INDENT
NEWLINE
INDENT
DEDENT
DEDENT
NEWLINE
def _ontainnn (cls, modell)
return cls._registry in model
Figure 3: Swapped Operand Example. In this code snippet, the return statement should be 'model cls .registry'' The swapped-operand model correctly predicts that the code snippet has an error. The query tokens are paying substantial attention to 'in' and the second occurrence of 'model' in the snippet.
18

---

Under review as a conference paper at ICLR 2020
Attention at layer
[CLS]
initial
data
NEWLINE
INDENT
Sum
Kernel
DEDENT
Gat
initall
[SDP]
COV
returm
Sum
Kermel
CON
ISEP)
NEW LIL
NEWLING INDENT
GEDENT DEWILING
Docstring: Get form initial data."
Function:
def add__ssedd (self, cov)
return SunOfKernel (self, cov)
Figure 4: Function Docstring Example. The CuBERT function- -nnttoondocstring model correctly predicts that the docstring wrong for this code snippet. Note that most of the query tokens are attending to the tokens in the docstring.
19

---

Under review as a conference paper at ICLR 2020
Attention at layer 23
[CL5]
INDENT
subprocess
return
200
NEWUNE DEDENT
NEWLINE
retorn
sonty
NEWLINE
[SEP]
True
success
False
400
(SE7)
NEW INE
NEWLINE
TEWEUNE
NEWLING
try:
subprocess. call (hook_value) return jsoni fy (ruu)) 200 except HOLE as 8:
return jsonify fy (success=False,
(eror=stx 400
Figure Exception Classification Example. For this code snippet, the CuBERT exception- classification model correctly predicts '..HOLE_.' 'OsError'. The model's attention matrix also shows that HOLE... is attending to 'subprocess', which is indicative of an OS-related error.
20


Reviews:
[{"review_id": "SklbqFhatB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a transformer based approach to address a number of problems for machine learning from code. Then, the paper adds BERT-based pretraining to significantly improve on the results. The paper is nicely written and easy to follow, but with relatively thin contributions besides the large amount of experiments.\n\nInitially, I was quite positive on the entire paper, but as I read it in more details, I got less convinced that this is something I want to see at ICLR. First, there are no good baselines. BiLSTM gets quite high accuracy on most of the tasks, which is unexpected because most prior works show that the tasks benefit from some semantic understanding of code. I cannot relate any of the numbers with a previous work. Right now, I even have reduced confidence that the authors do not have bugs in the tasks or the reported numbers. Then, for the Operator and Operand tasks, BiLSTM is also doing impressively well (these tasks were done differently in prior works). Interestingly, things get reversed on the last two tasks. Given that most of the experiments are not the same as in any previous paper, I would strongly appreciate if much more details are given in the appendix. In fact, the appendix right now does not have much useful information besides praising how good job the attention is doing. What would be needed is information on how many samples were generated, how large were they, was any data thrown out? Table 1 is a good start, but it actually raises questions. You split the Python dataset into separate functions like Vasic et al and the number of functions 2x higher (counting the artificial samples, I guess), did you put a limit on how large functions you consider? 250 tokens was the limit of Vasic et al. To which of the tasks in the Vasic et al paper can I relate? Is the BiLSTM on the level of that work or it is substantially worse or better? Also, are the results coming from the paper SOTA or uncomparable to other works?\n\nFive tasks are evaluated, which is impressive. This is one reason I want to see the results published. The problem is also quite important. The experiments that show the effect on reduced labelled data are quite important and interesting - in fact, for many tasks, we can start curating datasets and having model working on small data will be crucial. However, I think the paper needs more work before it is something to present, cite or build upon.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hkgs8EAk5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThis paper presents a BERT-inspired pretraining/finetuning setup for source code tasks. It collects a corpus of\nunlabeled Python files for BERT pretraining, designs or adopts 5 tasks on established smaller-scale Python corpora, and\nadjusts the BERT model to tokenize and encode source code snippets appropriately.\n\n# Strengths\n\n* The idea of applying the pretraining/finetuning paradigm to program analysis tasks makes sense, and has been\n  informally attempted by multiple groups in the community in 2019. This is the first high-quality submission to a\n  top-tier ML conference I've seen on the subject, though.\n* The authors exercised commendable care and diligence in preparing the training data, adopting BERT to source code\n  inputs, and ensuring correctness of the experimental setup. I appreciated all the provided details on tokenization\n  (Section 3.3), deduplication (Sections 3.1-3.2), and task setup (Section 3.5). This should become a technical standard\n  in the community.\n* The paper is written clearly and concisely, and is generally a pleasure to read.\n\n# Weaknesses\n\nI have a gripe with the authors' choice to ignore program structure (e.g. abstract syntax trees) or features (e.g.\ntypes) in their program representation. Without this extra information (easily available from a compiler/interpreter\nAPI) the pipeline is not substantially different from the original NLP pipeline of BERT et al. The main program-related\nrepresentation insight comes in tokenization (Section 3.3) and the definition of \"sentences\". To repeat, I appreciate\nthe effort the authors put in making tokenization appropriate for BERT processing of source code, but this is a drop in\nthe bucket compared to the all the other program-related features the work is leaving off the table. Programs are not\nnatural language.\nThe argument that source code analysis would \"pass on the burden ... to downstream tasks\" (Page 3) is odd. First, most\ndownstream tasks of interest occur in the settings where this analysis is already available: IDEs, code review\nassistants, linters, etc. Second, one often needs program analysis to even define downstream tasks in the first place --\nfor example, determining whether function arguments are swapped required detecting a function call and boundaries of its\narguments, thus parsing the program!\n\nThis work obtains (and nicely analyzes) impressive results obtained by applying CuBERT. However, it does not put the\nresults in context with prior work based on structured program representations. Without this, it is difficult to say\nwhether the improvement comes from pretraining or from the language model simply learning a better \"parsed\"\nrepresentation of an input program from all the unlabeled corpus. If it's the latter, one might argue that supplying the\nmodel with structured program features explicitly might eliminate much of the need for the unlabeled corpus.\nI personally think that there will still be a gap between pretraining and finetuning even with structured program\nfeatures simply due to the sheer volume of available data, which, as the authors showed, is crucial for good\ngeneralization of Transformer. However, this still needs to be shown empirically.\n\nThe \"Function-Docstring Mismatch\" task, as presented, seems too easy. If the distractors (negative examples) are truly\nchosen at random, most of them are going to use obviously different vocabulary from the original function signature (as\nFigure 4 demonstrates). A well designed task would somehow bias the sampling toward subtle distractors such as `get` vs.\n`set` docstrings, but this seems challenging.\nThis also explains why the task is not influenced as much by reduction of training data (Table 3).\n\nThe Next Sentence Prediction pretraining task, as adapted for CuBERT, seems too difficult, in contrast. If the paired\nsentences (i.e. code lines) are chosen at random, the model would lack most of the context required to make a decision\nabout the logical relationship between them, such as which variables are defined and available in context, which\nfunctionality is being implemented, etc. I wonder, can the authors experiment with pretraining CuBERT only with the\nMasked Language Model task? Will it worsen the results substantially or at all?\n\n# Questions\n\nSection 3.2: \"similar files according to the same similarity metric...\"\nWhat are these metrics?\n\nWhat is the fraction of positive/negative examples in the constructed finetuning datasets?\n\nWhat is the motivation for making Variable Misuse and Wrong Operator/Operand into a simple classification tasks instead\nof the original (more useful) correction task?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BygAMHy5_H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a BERT-based pre-training for source code related task. By pre-training on BERT-like models on source code and finetuning on a set of 5 tasks, the authors show good performance improvements over non-pretrained models. The authors make a series of ablation studies showing that pre-training is indeed useful.\n\nOverall, I find this work relevant and interesting, albeit somewhat unsurprising. Nevertheless, I see no reason to reject this paper. To make my \"weak accept\" to a \"strong accept\" I would like to see experiments on more tasks, preferably more complex tasks. For example, such tasks could include (a) variable naming (b) method naming (c) docstring prediction/summarization (d) language modeling/autocompletion. I believe it's unclear to the reader if pre-training is also helpful for any of those tasks too and adding such experiments would significantly strengthen the paper.\n\nSome clarifications comments/questions to the authors:\n\n* I would insist that the authors rename the \"Variable Misuse\" task to \"Variable Misuse Localization\". To my understanding the current model points to the misused variable (if any), but does not attempt to suggest a fix. This tackles only a part of the task discussed in Vasic et al. (2019), Allamanis et al (2018) and this might confuse readers who want to compare with those works.\n\n* For the Function-Docstring Mismatch task (Section 3.4):\n    * It's unclear to me which dataset is used. Is it the Py150 dataset or the Barone & Sennrich (2017)?\n    * I believe that the citations [a], [b] would be appropriate here.\n\n* Overall, for the all the tasks except from \"Exception Type\", there is a replicability issue: Since the authors manually mutate the code (e.g. introduce a variable misuse, swap an operand), for anyone to compare directly, they would need access to the mutated samples. I would strongly encourage the authors to provide more details on how they create mutated samples and (eventually) the source code that achieves that.\n\n* For the Variable Misuse, Wrong Binary Operator, Swapped Operand tasks. There are a few things that need to be clarified:\n   * How long is each code snippet? One would expect that the longer the code snippet the harder the task. Do the authors pass a whole function?\n   * What is the proportion of positive/negative examples in each task?\n\n\n\n[a] Cambronero, Jose, et al. \"When Deep Learning Met Code Search.\" arXiv preprint arXiv:1905.03813 (2019).\n[b] Louis, Annie, et al. \"Deep learning to detect redundant method comments.\" arXiv preprint arXiv:1806.04616 (2018).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents  CuBERT (Code Understanding BERT), which is BERT-inspired pretraining/finetuning setup, for source code contextual embedding. The embedding results are tested on classification tasks to demonstrate the effectiveness of CuBERT. 

This is an interesting application paper that extends existing models to source code analysis. The authors did a good job at motivating the applications, describing the proposed models and discussing the experiments. The authors also agree to share all the datasets and source code so that the experiment results can be replicated and compared with by other researchers. 

One major concern is the lack of strong baselines. All reviewers are concerned about this issue. The paper could lead to a good publication in the future if the issues can be addressed. 
Decision: Reject