Title: SCELMO: SOURCE CODE EMBEDDINGS FROM LANGUAGE MODELS

Abstract: Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair. Contextual embeddings are common in natural language processing but have not been previously applied in software engineering. We introduce a new set of deep contextualized word representations for computer programs based on language models. We train a set of embeddings using the ELMo (embeddings from language models) framework of Peters et al (2018) . We investigate whether these embeddings are effective when fine-tuned for the downstream task of bug detection. We show that even a low-dimensional embedding trained on a relatively small corpus of programs can improve a state-of-the-art machine learning system for bug detection.

1. INTRODUCTION:

Learning rich representations for source code is an open problem that has the potential to enable software engineering and development tools. Some work on machine learning for source code has used hand engineered features (Long & Rinard, 2016, e.g.) , but designing and implementing such features can be tedious and error-prone. For this reason, other work considers the task of learning a representation of source code from data (Allamanis et al., 2018a) . Many models of source code are based on learned representations called embeddings, which transform words into a continuous vector space (Mikolov et al., 2013) . Currently in software engineering (SE) researchers have used static embeddings (Harer et al., 2018; White et al., 2019; Pradel & Sen, 2018) , which map a word to the same vector regardless of its context. However, recent work in natural language processing (NLP) has found that contextual embeddings can lead to better performance (Peters et al., 2018; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019) . Contextualized embeddings assign a different vector to a word based on the context it is used. For NLP this has the advantage that it can model phenomena like polysemy. A natural question to ask is if these methods would also be beneficial for learning better SE representations.

In this paper, we introduce a new set of contextual embeddings for source code. Contextual embeddings have several potential modelling advantages that are specifically suited to modelling source code:

• Surrounding names contain important information about an identifier. For example, for a variable name, surrounding tokens might include functions that take that variable as an argument or assignments to the variable. These tokens provide indirect information about possible values the variable could take, and so should affect its representation. Even keywords can have very different meanings based on their context. For instance, a private function is not the same as a private variable or a private class (in the case of Java / C++). • Contextual embeddings assign a different representation to a variable each time it is used in the program. By doing this, they can potentially capture how a variable's value evolves through the program execution. • Contextual embeddings enable the use of transfer learning. Pre-training a large neural language model and querying it for contextualized representations while simultaneously fine-tuning for the specific task is a very effective technique for supervised tasks for which there is a small amount of supervised data available. As a result only a small model needs to be fine-tuned atop the pre-trained model, without the need for task-specific architectures nor the need of training a large model for each task separately.

In this paper, we highlight the potential of contextual code embeddings for program repair. Automatically finding bugs in code is an important open problem in SE. Even simple bugs can be hard to spot and repair. A promising approach to this end is name-based bug detection, introduced by DeepBugs (Pradel & Sen, 2018) . The current state-of-the-art in name-based bug detection relies on static representations from Word2Vec (Mikolov et al., 2013) to learn a classifier that distinguishes correct from incorrect code for a specific bug pattern. We introduce a new set of contextualized embeddings for code and explore its usefulness on the task of name-based bug detection. Our method significantly outperforms DeepBugs as well as other static representations methods on both the DeepBugs dataset as well as a new previously unused test set of JavaScript projects. We release our implementation and representations as they could lead to improvements in a great variety of SE tasks.

2. RELATED WORK:

Unsupervised static word embeddings have been extensively used to improve the accuracy of supervised tasks in NLP (Turian et al., 2010) . Notable examples of such methods are Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) . However, the above models learn only a single context-independent word representation. To overcome this problem some models (Wieting et al., 2016; Bojanowski et al., 2017) enhance the representations with subword information, which can also somewhat deal with out-of-vocabulary words. Another approach is to learn a different representation for every word sense (Neelakantan et al., 2014) but this requires knowing the set of word senses in advance. More recent methods overcome the above issues by learning contextualized embeddings. Melamud et al. (2016) encode the context surrounding a pivot word using a bidirectional LSTM. Peters et al. (2018) use a deep bidirectional LSTM, learning word embeddings as functions of its internal states, calling the method Embeddings using Language Models (ELMo). We discuss ELMo in detail in Section 3. Devlin et al. (2018) introduced bidirectional encoder representations from transformers (BERT). This method learns pre-trained contextual embeddings by jointly conditioning on left and right context via an attention mechanism.

Program repair is an important task in software engineering and programming languages. For a detailed review see Monperrus (2018); Gazzola et al. (2019) . Many recent program repair methods are based on machine learning. Yin et al. (2018) learn to represent code edits using a gated graph neural network (GGNN) (Li et al., 2016) . Allamanis et al. (2018b) learn to identify a particular class of bugs called variable misuse bugs, using a GGNN. Chen et al. (2019) introduce SequenceR which learns to transform buggy lines into fixed ones via machine translation. Our work is orthogonal to these approaches and can be used as input in other models.

Finally, our work is also related to code representation methods many of which have also been used in program repair. Harer et al. (2018) learn Word2Vec embeddings for C/C++ tokens to predict software vulnerabilities. White et al. (2019) learn Word2Vec embeddings for Java tokens and utilize them in program repair. Alon et al. (2019) learn code embeddings using abstract syntax tree paths. A more detailed overview can be found in (Allamanis et al., 2018a; Chen & Monperrus, 2019) .

3. EMBEDDINGS FROM LANGUAGE MODELS (ELMO):

ELMo (Peters et al., 2018) computes word embeddings from the hidden states of a language model. Consequently, the embeddings of each token depend on its context of the input sequence, even out-of-vocabulary (OOV) tokens have effective input representations. In this section, we briefly describe the ELMo embeddings.

The first step is that a neural language model is trained to maximize the likelihood of a training corpus. The architecture used by ELMo a bidirectional LSTM with L layers and character convolutions in the input layer. Let the input be a sequence of tokens (t 1 , ...t N ). For each token t k , denote by x LM k the input representation from the character convolution. Consequently, this representation passes through L layers of forward and backward LSTMs. Then each layer j ∈ {1, ..., L} of the forward LSTM computes a hidden state --→ h LM k,j , and likewise the hidden states of the backward LSTM are denoted by ←-h LM k,j . The parameters for the token representation and for the output softmax layer are tied for both directions, while different parameters are learned for each direction of the LSTMs.

After the language model has been trained, we can use it within another downstream task by combining the hidden states of the language model from each LSTM layer. This process is called ELMo. For each token t k of a sentence in the test set, the language model computes 2L + 1 hidden states, one in each direction for each layer, and then the input layer. To make the following more compact, we can write these as h LM k,0 = x LM k for the input layer, and then

h LM k,j = [ --→ h LM k,j , ←-- h LM k,j

] for all of the other layers. The set of these vectors is

EQUATION

To create the final representation that is fed to downstream tasks, ELMo collapses the set of representations into a single vector E k for token t k . A simplistic approach is to only select the top layer, so that E k = h LM k,L . A more general one, which we use in this work, is to combine the layers via fine-tuned task specific weights s = (s 1 . . . s L ) for every layer. Then we can compute the embedding for token k as

EQUATION

where γ is an additional scalar parameter that scales the entire vector. In our experiments we did not performed finetuning and thus used equal weights s j = 1/(L + 1) for each layer and γ = 1. However, our implementation also supports all the aforementioned ways of collapsing the set of representations.

A potential drawback of the method is that it still utilizes a softmax output layer with a fixed vocabulary that does not scale effectively and it still predicts UNK for OOV tokens which may have a negative effect on the representations.

4. SOURCE CODE ELMO:

We describe Source Code ELMo (SCELMo), which trains ELMo on corpora of source code. However, we note that normally ELMo models in other domains are able to effectively utilize much larger representations. The code was tokenized using the esprima JavaScript tokenizer1 . For training the ELMo model we used a corpus of 150,000 JavaScript Files (Raychev et al. 2016) consisting of various open-source projects. This corpus has previously been used on several tasks (Raychev et al., 2016; Pradel & Sen, 2018; Bavishi et al., 2018) . We applied the patch released by Allamanis et al. (2018a) to filter out code duplication as this phenomenon was shown on this and other corpora to result in inflation of performance metrics. This resulted in 64750 training files and 33229 validation files. Since the validation set contains files from the same projects as the train the contained instances might be too similar and unrealistic overestimating. To address this we also created a test set of 500 random JavaScript projects sampled from the top 20,000 open-source JavaScript projects as of May 2019. The test corpus has not been previously utilized in previous work and is a better reflection of the performance of the learned bug detectors. Lastly, it is important to know what the performance of the method will be if we do not have access to training data from the projects on which we would like to find bugs. This is common in practice for many real case scenarios. For training the ELMo model, we use an embedding size of 100 features for each of the forward and backward LSTMs so that each layer sums up to 200 features.

5. CONTEXTUAL EMBEDDINGS FOR PROGRAM REPAIR:

In this section, we describe how contextual embeddings can be incorporated within a recent machine learning-based bug detection system, the DeepBugs system of Pradel & Sen (2018) . In the first part of this section, we give background about the DeepBugs system, and then we describe how we incorporate SCELMo within DeepBugs. DeepBugs treats the problem of finding a bug as a classification problem. The system considers a set of specific bug types, which are small mistakes that might be made in a program, such as swapping two arguments. For each bug type, DeepBugs trains a binary classifier that takes a program statement as input and predicts whether the statement contains that type of bug. At test time, this classifier can be run for every statement in the program to attempt to detect bugs.

In order to train the model both examples of correct and incorrect (buggy) code are necessary. DeepBugs treats the existing code as correct and randomly mutates it to obtain buggy code. To obtain training examples, we extract all expressions from the source code which are either the function calls with exactly two arguments and all binary expressions. To create instances of buggy code we mutate each of the correct instances. As such, arguments in function calls are swapped, the binary operator in binary expressions is replaced with another random one, and finally randomly either the left or the right operand is replaced by another random binary operand that appears in the same file. Then the classification task is a binary task to predict whether the instance is correct, i.e., it comes from the original code, or whether it is buggy, i.e. it was one of the randomly mutated examples. The validation and test sets are mutated in the same way as the training set. The split between correct and buggy instances has 50/50 class distribution as for each original code instance exactly one mutated buggy counterpart is created.

The architecture for the classifier is a feedforward network with a single hidden layer of 200 dimensions with Relu activations and a sigmoid output layer. For both the input and hidden layers a dropout of 0.2. The network was trained in all experiments for 10 epochs with a batch size of 50 and the RMSProp optimizer. We note that for maintaining a consistent comparison with DeepBugs we kept all the above parameters as well as the optimizer's parameters fixed to the values reported in Pradel & Sen (2018) . Tuning these parameters would probably result in at least a small performance increase for our method. In our experiments, we consider three bug types that address a set of common programming mistakes: swapped arguments of function calls, using the wrong binary operator and using an incorrect binary operand in a binary expression. The methodology can easily be applied to other bug types. Figure 1 illustrates an example of each of the three bug types.

5.1. INPUT TO THE CLASSIFIER:

A key question is how a statement from the source code is converted into a feature vector that can be used within the classifier. DeepBugs uses a set of heuristics that, given a statement and a bug type, return a sequence of identifiers from the statement that are most likely to be relevant. For instance, for the call to setTimeout in Listing 1 the following sequence of identifiers would be extracted: [setTimeout, delay, function]. A detailed description of the heuristics is available in Appendix A.

These heuristics result in a sequence of program identifiers. These are converted to continuous vectors using word embeddings, concatenated, and this is the input to the classifier. DeepBugs uses Word2Vec embeddings trained on a corpus of code. In our experiments, we train classifiers using three different types of word embeddings. First, we kept the 10,000 most frequent identifiers/literals and assigned to each of them a random embedding of 200 features. Second, to reproduce the results of Pradel & Sen (2018) , we use the CBOW variant of Word2Vec to learn representations consisting of 200 features for the 10,000 most frequent identifiers/literals. Finally, we train a FastText embeddings (Bojanowski et al., 2017) on the training set to learn identifier embeddings that contain subword information. The subwords used by FastText are all the character trigrams that appear in the training corpus. Identifiers are therefore composed of multiple subwords. To represent an identifier, we sum the embeddings of each of its subwords and summing them up. This allows the identifier embeddings to contain information about the structure and morphology of identifiers. This also allows the FastText embeddings, unlike the Word2Vec ones, to represent OOV words as a combination of character trigrams.

Note that DeepBugs can detect bugs only in statements that do not contain OOV (out-of-vocabulary) identifiers, because its Word2Vec embeddings cannot extract features for OOV names. Instead our implementation does not skip such instances. Since the original work discarded any instances that contain OOV identifiers we neither know how the method performs on such instances nor how often those appear in the utilized dataset of DeepBugs. Moreover, Deep-Bugs supported only a specific subset of AST nodes and skipped the rest. For example if a call's argument is a complex expression consisting of other expressions then the call would be skipped. However, we expanded the implementation to support all kinds of AST nodes and to not skip instances with nested expressions as discussed in Appendix A. We note that we still skip an instance if one of its main parts (e.g., a function call's argument) is a complex expression longer than 1,000 characters as such expressions might be overly long to reason about. We investigated two variants of the bug detection model, which query SCELMo in different ways to get features for the classifier. The first utilizes the heuristic of Section A to extract a small set of identifiers or literals that represent the code piece. For example, for an incorrect binary operand instance we extract one identifier or literal for the left and right operands respectively, and we also extract its binary operator. Then, those are concatenated to form a query to the network. In the case of function calls we extract the identifier corresponding to the name of the called function, one identifier or literal for the first and second argument respectively and an identifiers for the expression on which the function is called. We also add the appropriate syntax tokens (a '.' if necessary, ',' between the two arguments, and left and right parentheses) to create a query that resembles a function call. This baseline approach creates simplistic fixed size queries for the network but does not utilize its full potential since the queries do not necessarily resemble actual code, nor correct code similar to the sequences in the training set for the embeddings. We will refer to this baseline as No-Context ELMo.

Our proposed method, we compute SCELMo embeddings to the language model all the tokens of the instances for which we need representations. Valid instances are functions calls that contain exactly two arguments and binary expressions. To create a fixed-size representation we extract only the features corresponding a fixed set of tokens. Specifically, for functions calls we use the representations corresponding to the first token of the expression on which the function is called, the function name, the first token of the first argument and the first token of the second argument.

While, for binary expressions we use those of the first token of the left operand, the binary operator, and the first token of the right operand. Since the representations contain contextual information, the returned vectors can capture information about the rest of the tokens in the code sequence.

6. RESULTS:

We next discuss the experiments we performed and their corresponding results. We measured the performance of the three baselines as well as those of non-contextual ELMO and SCELMO. Measuring the performance of non-contextual ELMO allows us to evaluate how much improvement is due to specifics of the language model architecture, such as the character convolutional layer which can handle OOVs, and how much is due to the contextual information itself.

6.1. PERFORMANCE ON VALIDATION SET:

In our first experiment we evaluate the performance of the methods in tasks where training data from the same projects are available. The evaluation performed in this experiment gives a good estimation of how our method performs compared to the previous state-of-the-art technique of DeepBugs. One main difference however is that the evaluation now also includes instances which contain OOV. As a consequence the bug detections tasks are harder than those presented by Pradel & Sen (2018) as their evaluation does not include in both the training and validation set any instance for which an extracted identifier is OOV. Table 1 illustrates the performance of the baselines and our models. As one would expect the FastText baseline improves over Word2Vec for all bug types due to the subword information. Moreover, our model SCELMo massively outperforms all other methods. Lastly, even no-context ELMo the heuristic version of SCELMo that does not utilize contextual information at test time outperforms the baseline methods showcasing how powerful the pretrained representations are.

6.2. INCLUDING COMPLEX EXPRESSIONS:

In our next experiment we also included instances that contain elements that are complex or nested expressions. For instance, in the original work if one the arguments of a function call or one of the operands of a binary expression is an expression consisting of other expressions then the instance would not be included in the dataset. Several AST node types such as a NewExpression node or an ObjectExpression were not supported. Figure 2 a few examples of instances that would be previously skipped2 . Such instances were skipped by Pradel & Sen (2018) and not included in their results. We do note though that we still skip very long expressions that contain more than 1000 tokens.

// First argument is binary expression doComputation(x + find_min(components), callback); // Second argument is an unsupported node factory.test(simulator, new Car('Eagle', 'Talon TSi', 1993)); Similarly to the previous experiment SCELMo significantly outperforms all other models. This is evident in Table 2 . Lastly, we clarify that the results of this section should not be directly compared to those of the previous one as for this experiment the training set is also larger.

6.3. EXTERNAL TEST EVALUATION:

The last experiment's objective is to showcase how the various models would perform on unseen projects as this better illustrates the generalizability of the techniques. The configuration utilized is identical to that of the previous section. By looking at Table 3 one can notice that the baselines have a major drop in performance. This is a common finding in machine learning models of code, namely, that applying a trained model to a new software project is much more difficult than to a new file in the same project. In contrast, SCELMo offers up to 15% improvement in accuracy compared to Word2Vec baseline. In fact, impressively enough SCELMo on the external test set is better than the evaluation set one of the baselines.

6.4. OOV STATISTICS:

In order to better understand the above results we measured the OOV rate of the basic elements of the code instances appearing in the dataset. Here the OOV rate is calculated based on the vocabulary of 10000 entries utilized by the Word2Vec and random baseline models. These are illustrated in Tables 4 and 5 . We measured the OOV rates for both the version of the dataset used in Section 6.4, which we call Train and Validation, and that used in Section 6.2, which we call Extended Train and Extended Validation.

Tables 4 and 5 describe the OOV rates for different parts of the expression types that are considered by the DeepBugs bug detector. A detailed description of the identifiers extraction heuristic can be found in Appendix A. We first focus on the swapped arguments bug pattern and consider all of the method call that have exactly two arguments. Each method call contains the function name, a name of the first argument, a name of the second argument, and a base object. The base object is the identifier that would be extracted from the expression (if such an expression exists) on which the function is called. For instance, from the following expression: window.navigator.userAgent.indexOf("Chrome"), userAgent would be extracted as the base object. Table 4 shows for each of the components how often they are OOV.

In the expanded version of the dataset if one of the arguments is a complex expression then it is converted into a name based on the heuristic described in Section A. The resulting statistics contain valuable information as for instance, it is almost impossible for the Word2Vec baseline to reason about a swap arguments bug if the identifiers extracted for both arguments are OOV.

In a similar manner for the incorrect operand and operator bug patterns we consider all the binary operations. Each binary expression consists of a left and right operand and a name is extracted for each of them. For each operand we also measured the frequency with which the operand corresponds to certain common types such as identifier, literal or a ThisExpression. Although related work (Pradel & Sen, 2018; Allamanis et al., 2018b; Vasic et al., 2019) has shown that there is great potential for embedding based neural bug finders, the evaluation has mostly focused on synthetic bugs introduced by mutating the original code. However, there is no strong indication that the synthetic bugs correlate to real ones, apart from a small study of the top 50 warnings for each bug type produced by DeepBugs. A good example is the mutation operation utilized for the incorrect binary operator bug. A lot of the introduced bug instances could result in syntactic errors. This can potentially create a classifier with a high bias towards correlating buggy code to syntactically incorrect code, thus hindering the model's ability to generalize on real bugs. Ideally, in an industrial environment we would like the resulting models to achieve a false positive rate of less than 10 % (Sadowski et al., 2015) . Sadly, high true positive rates are not to be expected as well since static bug detectors were shown to be able to detect less than 5% of bugs Habib & Pradel, 2018) contained in the Defects4J corpus (Just et al., 2014) and less than 12% in a single-statement bugs corpus (Karampatsis & Sutton, 2019) . We note that in the second case the static analysis tool is given credit by reported any warning for the buggy line, so the actual percentage might lower than the reported one.

We next make a first step on investigating the practical usefulness of our methods by applying the classifiers of the previous section on a small corpus of real JavaScript bugs. However, we think that this is a very hard yet interesting problem that should be carefully examined in future work. In order to mine a corpus of real bug changes we used the methodology described in (Karampatsis & Sutton, 2019) . We note that we adapted their implementation to utilize the Rhino JavaScript parser3 . Their methodology extracts bug fixing commits and filters them to only keep those that contain small single-statement changes. Finally, it classifies each pair of modified statements by whether the fit a set of mutation patterns. The resulting dataset is shown in Table 6 . Upon acceptance of the paper we will release this dataset along with our implementation, the rest of data used, and the learned representations.

Finally, we queried the DeepBugs and SCELMo with each buggy instance as well as its fixed variant and measured the percentage of correctly classified instances for each of the two categories. We also ignored any instances for which the JavaScript parser utilized for both failed to extract an AST. We classified as bugs any instances that were assigned a probability to be a bug > 75%. In an actual system this threshold should ideally be tuned on a validation set.

Table 7 suggests that there might indeed be some potential for future practical applications of neural bug finding techniques. Both are able to uncover some of the bugs. However, the results also suggest that careful tuning of the predictions threshold might be necessary, especially if we take into account the industrial need to comply with a low false positive rate (FPR). For instance, raising SCELMo's prediction threshold to 80% for the swap arguments bug results in finding only 3.34% of the bugs but correctly classifying 100% of the repaired function calls, thus achieving 0.0% false positive rate. Moreover, since SCELMo could not uncover any of the real binary operator bugs, future work could investigate the effect of utilizing different mutation strategies for the purpose of artificial bug-induction. Future work could also investigate if fine-tuning on small set of real bugs could result in more robust classifiers.

8. CONCLUSION:

We have presented SCELMo, which is to our knowledge the first language-model based contextual embeddings for source code. Contextual embeddings have many potential advantages for source code, because surrounding tokens can indirectly provide information about tokens, e.g. about likely values of variables. We highlight the utility of SCELMo embeddings by using them within a recent state-of-the-art machine learning based bug detector. The SCELMo embeddings yield a dramatic improvement in the synthetic bug detection performance benchmark, especially on lines of code that contain out-of-vocabulary tokens and complex expressions that can cause difficulty for the method. We also showed and discussed the performance of the resulting bug detectors on a dataset of real bugs raising useful insights for future work.

:

https://esprima.org/

The AST is extracted using the acorn parser https://github.com/acornjs/acorn

https://github.com/mozilla/rhino

Figures and Tables:

Figure 1: Bug type examples.

Figure 2: Examples of instances that would be skipped by DeepBugs.

Comparison of ELMo versus non-contextual embeddings for bug detection on a validation set of projects. Data is restricted to expressions that contain only single names.

Table content: <table><tr><td/><td colspan="5">Random Word2Vec FastText No-Context ELMo SCELMo</td></tr><tr><td>Swapped Arguments</td><td>86.18%</td><td>87.38%</td><td>89.55%</td><td>90.02%</td><td>92.11%</td></tr><tr><td colspan="2">Wrong Binary Operator 90.47%</td><td>91.05%</td><td>91.11%</td><td>92.47%</td><td>100.00%</td></tr><tr><td colspan="2">Wrong Binary Operand 75.56%</td><td>77.06%</td><td>79.74%</td><td>81.71%</td><td>84.23%</td></tr><tr><td colspan="3">5.2 CONNECTING SCELMO TO THE BUG DETECTOR</td><td/><td/><td/></tr></table>

Comparison of SCELMo versus static embeddings on bug detection on a validation set of projects. Complex expressions are included in this validation set.

Table content: <table><tr><td/><td colspan="5">Random Word2Vec FastText No-Context ELMo SCELMo</td></tr><tr><td>Swapped Arguments</td><td>86.37%</td><td>87.68%</td><td>90.37%</td><td>90.83%</td><td>92.27%</td></tr><tr><td colspan="2">Wrong Binary Operator 91.12%</td><td>91.68%</td><td>91.92%</td><td>92.75%</td><td>100.00%</td></tr><tr><td colspan="2">Wrong Binary Operand 72.73%</td><td>74.31%</td><td>77.41%</td><td>79.65%</td><td>87.10%</td></tr></table>

Comparison of SCELMo versus static embeddings on bug detection on an external test set of 500 JavaScript projects.

Table content: <table><tr><td/><td colspan="5">Random Word2Vec FastText No-Context ELMo SCELMo</td></tr><tr><td>Swapped Arguments</td><td>75.79%</td><td>78.22%</td><td>79.40%</td><td>81.37%</td><td>84.25%</td></tr><tr><td colspan="2">Wrong Binary Operator 82.95%</td><td>85.54%</td><td>83.15%</td><td>86.54%</td><td>99.99%</td></tr><tr><td colspan="2">Wrong Binary Operand 67.46%</td><td>69.50%</td><td>72.55%</td><td>75.74%</td><td>83.59%</td></tr></table>

OOV statistics for calls with exactly two arguments (Swapped arguments instances). The statistics are calculated on variants of the DeepBugs dataset.

Table content: <table><tr><td/><td>Train</td><td colspan="3">Expanded Train Validation Expanded Validation</td></tr><tr><td>Two Arguments Calls</td><td>574656</td><td>888526</td><td>289061</td><td>453486</td></tr><tr><td>Calls Missing Base Object</td><td>25.07%</td><td>28.63%</td><td>25.63%</td><td>28.80%</td></tr><tr><td>Base Object Missing or OOV</td><td>34.56%</td><td>37.38%</td><td>35.57%</td><td>38.07%</td></tr><tr><td>Function Name OOV</td><td>20.69%</td><td>17.07%</td><td>20.33%</td><td>16.94%</td></tr><tr><td>First Argument OOV</td><td>31.01%</td><td>36.99%</td><td>31.64%</td><td>37.15%</td></tr><tr><td>Second Argument OOV</td><td>27.25%</td><td>22.86%</td><td>27.94%</td><td>23.49%</td></tr><tr><td>Both Arguments OOV</td><td>11.33%</td><td>9.57%</td><td>11.96%</td><td>10.16%</td></tr><tr><td>Base and Function Name OOV</td><td>10.20%</td><td>8.32%</td><td>10.39%</td><td>8.61%</td></tr><tr><td>Base and Arguments OOV</td><td>4.21%</td><td>3.31%</td><td>4.88%</td><td>3.77%</td></tr><tr><td colspan="2">Function Name and Arguments OOV 2.86%</td><td>2.26%</td><td>2.85%</td><td>2.28%</td></tr><tr><td>All Elements OOV</td><td>1.53%</td><td>1.18%</td><td>1.61%</td><td>1.27%</td></tr></table>

OOV statistics for binary operations.

Table content: <table><tr><td/><td>Train</td><td colspan="3">Expanded Train Validation Expanded Validation</td></tr><tr><td>Binary Operations</td><td>1075175</td><td>1578776</td><td>540823</td><td>797108</td></tr><tr><td>Left Operand OOV</td><td>25.40%</td><td>28.84%</td><td>26.04%</td><td>29.55%</td></tr><tr><td>Right Operand OOV</td><td>20.37%</td><td>23.98%</td><td>20.74%</td><td>24.55%</td></tr><tr><td>Both Operands OOV</td><td>7.82%</td><td>11.29%</td><td>8.24%</td><td>11.88%</td></tr><tr><td>Unknown Left Operand Type</td><td>83.36%</td><td>87.80%</td><td>83.14%</td><td>87.74%</td></tr><tr><td>Unknown Right Operand Type</td><td>48.48%</td><td>47.23%</td><td>48.47%</td><td>47.05%</td></tr><tr><td colspan="2">Both Operand Types Unknown 33.34%</td><td>36.06%</td><td>33.20%</td><td>35.87%</td></tr><tr><td>All OOV or Unknown</td><td>3.59%</td><td>4.03%</td><td>3.81%</td><td>4.3%</td></tr><tr><td colspan="3">7 IS NEURAL BUG-FINDING USEFUL IN PRACTICE?</td><td/><td/></tr></table>

Real bug mined instances.

Table content: <table><tr><td/><td colspan="3">Swapped Arguments Wrong Binary Operator Wrong Binary Operand</td></tr><tr><td>Mined Instances</td><td>303</td><td>80</td><td>1007</td></tr></table>

Real bug identification task recall and false positive rate (FPR).

Table content: <table><tr><td/><td colspan="4">Word2Vec-Recall Word2Vec-FPR SCELMo-Recall SCELMo-FPR</td></tr><tr><td>Swapped Arguments</td><td>3.34%</td><td>0.33%</td><td>49.67%</td><td>33.78%</td></tr><tr><td>Wrong Binary Operator</td><td>8.95%</td><td>7.70%</td><td>0.00%</td><td>0.00%</td></tr><tr><td>Wrong Binary Operand</td><td>11.99%</td><td>12.11%</td><td>15.81%</td><td>14.34%</td></tr><tr><td>(</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
The paper proposes to use ELMO embeddings to improve the precision on the first step of the DeepBugs tasks defined by Pradel and Sen (2018). This first step is an artificial problem created by taking real programs (with and without bugs, but assuming almost all of them are correct) and introducing bugs of certain type into the programs. Then, a classifier needs to distinguish between the real and the artificial set. This classifier is then to be used as a checker for anomalies in code and the anomalies are reported as bugs, however the paper skips this second step and only reports results on the first classification problem. Technically, the paper improve this first step of DeepBugs by using a standard variant of ELMO. The evaluation is detailed, but the results are unsurprising. The paper simply tech-transfers the idea from NLP to Code. If this work is accepted at the conference, I cannot imagine an interesting presentation or a poster that simply cites the changed numbers. Did we expect ELMO to be worse than more naive or random embeddings? The work and its results heavily peg on the DeepBugs and increases the precision of its first step by a significant margin, but does not show getting any more useful results. In fact, on one task (Wrong Binary Operator), SCELmo gets to 100% accuracy. This means it will never report any bugs, whereas DeepBugs seems to be performing best on exactly this kind of reports with its weaker model. I would recommend the authors to either work on showing practical usefulness of the technique, showing something for the full bugfinding task (not merely the first, artificial part), or to investigate if (or how) the idea to add bug-introducing changes to a code dataset is conceptually flawed for bugfinding (as this idea is widely used by several other works like Allamanis et al 2018b or by https://arxiv.org/abs/1904.01720 which also don*t get to practical tools ). There seems to be some indication of this by the reported 100% accuracy, but right now this remains completely uninvestigated. Minor issues: Listing 3: Opernad -> Operand Page 5. There is no Table 6.1