SCELMo: Source Code Embeddings from Language Models
Authors: Rafael - Michael Karampatsis
Charles Sutton
Venue:  (2020)
Keywords: Transfer Learning
Pretraining
Program Repair

Abstract: Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair. 
Contextual embeddings are common in natural language processing but have not been previously applied in software engineering.
We introduce a new set of deep contextualized word representations for computer programs based on language models.
We train a set of embeddings using the ELMo (embeddings from language models) framework of Peters et al (2018).
We investigate whether these embeddings are effective when fine-tuned for the downstream task of bug detection.
We show that even a low-dimensional embedding trained on a relatively small corpus of programs can improve a state-of-the-art machine learning system for bug detection.

Under review as a conference paper at ICLR 2020
SOURCE CODE EMBEDDINGS FROM LANGUAGE MODELS
SCELMo:
Anonymous authors Paper under double-blind review
ABSTRACT
Continuous embeddings of tokens in computer programs have been used to support a variety of software development tools, including readability, code search, and program repair. Contextual embeddings are common in natural language processing but have not been previously applied in software engincering. We introduce a new set of deep contextualized word representations for com- puter programs based on language models. We train set of embeddings using the ELMo (em- beddings from language models) framework of Peters al (2018). We investigate whether these embeddings are effective when fine-tuned for the downstream task of bug detection. We show that even a oowdimensional embedding trained on a relatively small corpus of programs can improve a state-of-the-art machine learning system for bug detection.
1 INTRODUCTION
Learning rich representations for source code is an open problem that has the potential to enable software engineering and development tools. Some work machine learning for source code has used hand engineered features (Long & | c.g.), but designing and implementing such features can be tedious and error-prone. For this reason, other work considers the task of learning representation of source code from data (Allamanis et al.) 2018a). Many models of source code are based on learned representations called embeddings, which transform words into a continuous vector space (Mikolov et al., 2013). Currently in software engineering (SE) researchers have used static embeddings (Harer et al. 2018 White et al., 2019; Pradel & Sen 2018). which map a word to the same vector regardless of its context. However, recent work in natural language processing (NLP) has found that contextual embeddings can lead to better performance (Peters et al. 2018, Devlin et al., 2018 Yang et al., 2019 Liu et al. 2019). Contextualized embeddings assign a different vector to a word based on the context it is used. For NLP this has the advantage that it can model phenomena like polysemy. A natural question to ask is if these methods would also be beneficial for learning better SE representations.
In this paper, introduce new set of contextual embeddings for source code. Contextual embeddings have several potential modelling advantages that are specifically suited to modelling source code:
Surrounding names contain important information about an identifier. For example, for a variable name, surrounding tokens might include functions that take that variable as an argument or assignments to the variable. These tokens provide indirect information about possible values the variable could take, and so should affect its representation. Even keywords instance, a private function is not the same as a private variable or a private class (in the case Java C++).
have very different meanings based on their context. For
Contextual embeddings assign a different representation to variable each time it is used in the program. By doing this, they can potentially capture how variable's 's value evolves through the program execution.
Contextual embeddings enable the use of transfer learning. Pre-training large neural language model and querying for contextualized representations while simultaneously fine-tuning for the specific task is a very effective technique for supervised tasks for which there is small amount of supervised data available. As a result only small model needs to be fine-tuned atop the pre-trained model, without the need for task-specific architectures nor the need of training large model for each task separately.
In this paper, we highlight the potential of contextual code embeddings for program repair. Automatically finding bugs in code is an important open problem in SE. Even simple bugs can be hard to spot and repair. A promising approach to this end is name-based bug detection, introduced by DeepBugs (Pradel & Sen. 2018). The current sate-of-the-arr in name-based bug detection relies on static representations from Word2 (Mikolov et al.) (013)) to learn a classifier that distinguishes correct from incorrect code for specific bug pattern. We introduce a new set of contextualized

---

Under review as a conference paper at ICLR 2020
embeddings for code and explore its usefulness on the task of hame-based bug detection. Our method significantly outperforms DeepBugs as well as other static representations methods on both the DeepBugs dataset as well as a new previously unused test set of JavaSeript projects. We release our implementation and representations as they could lead to improvements in great variety of SE tasks.
2 RELATED WORK
Unsupervised static word embeddings have been extensively used to improve the accuracy of supervised tasks in NLP (Turian et al., 2010). Notable examples of such methods are Word2Vec (Mikolov et al. 2013) and GloVe (Pennington et 2014]. However, the above models learn only a single context-independent word representation. To overcome this problem some models (Wieting al. 2016 Bojanowski et al., 2017) enhance the representations with subword information, which can also somewhat deal with out-of-vocabulary words. Another approach is to learn a different representation for every word sense (Neelakantan et al., 2014) but this requires knowing the set of word senses in advance. More recent methods overcome the above issues by learning contextualized embeddings, [Melamud et al. (2016) encode the context surrounding a pivot word using a bidirectional LSTM. |Peters et al. (2018) use a deep bidirectional LSTM, learning word embeddings as functions of its internal states, calling the method Embeddings using Language Models (ELMo). We discuss ELMo detail in Section Devlin et al. (2018) introduced bidirectional encoder representations from transformers (BERT). This method learns pre-trained contextual embeddings by jointly conditioning on left and right context via an attention mechanism.
Program repair is an important task in software engineering and programming languages. For detailed review see Monperrus (2018); Gazzola et al. (2019). Many recent program repair methods are based on machine learning. Yin et al. (2018) learn represent code edits using gated graph neural network (GGNN) (Li et al., (016). Allamanis et al. (2018b) learn to identify a particular class of bugs called variable misuse bugs, using a GGNN. Chen et al. (2019) introduce SequenceR which learns to transform buggy lines into fixed ones via machine translation. Our work is orthogonal to these approaches and can be used as input in other models.
Finally, our work is also related code representation methods many of which have also been used in program repair. Harer et al. (2018) learn Word2Vec embeddings for C/C++ tokens to predict software vulnerabilities. White et al. (2019) learn Word2Vec embeddings for Java tokens and utilize them in program repair. [Alon et al. (2019) learn code embeddings using abstract syntax tree paths. A more detailed overview can be found in (Allamanis et al. 201 Sa
Chen
& Monperrus 2019).
3 EMBEDDINGS FROM LANGUAGE MODELS (ELMO)
ELMo (Pteess et al. 2018) computes word embeddings from the hidden states of a language model. Consequently, the embeddings of each token depend on its context of the input sequence, even out-of-vocabulary (OOV) tokens have effective input representations. In this section, we briefly describe the ELMo embeddings.
The first step is that neural language model is trained to maximize the likelihood of a training corpus. The architecture used by ELMo a bidirectional LSTM with L layers and character convolutions in the input layer. Let the input be sequence of tokens (t.)...tv). For each token tu. denote by f the input representation from the character convolution. Consequently, this representation passes through L layers of forward and backward LSTMs. Then
each layer j E (I,...,,) }} of the forward LSTM computes a hidden state f and likewise the hidden states of the
backward LSTM are denoted by h.M. The parameters for the token representation and for the output softmax layer are tied for both directions, while different parameters are learned for each direction of the LSTMs.
After the language model has been trained, we can use it within another downstream task by combining the hidden states of the language model from each LSTM layer. This process is called ELMo. For each token ta of sentence in the test set, the language model computes 2L + 1 hidden states, one in each direction for each layer, and then the input layer. To make the following more compact, we can write these h k.0 z
LM J.M for the input layer, and then
h kJ
LM La I.M for all of the other layers. The set of these vectors is [ k,g
LM 1 = 0, T.}. R 1
(1)
To create the final representation that is fed to downstream tasks, ELMo collapses the set of representations into f.M A more general
single vector E, for token tg. A simplistic approach is to only select the top layer, so that E, = h hgL
one, which we use in this work, is to combine the layers via fine-tuned task specific weights s = (81 (s1...8t) 81) for every
2

---

Under review as a conference paper at ICLR 2020
layer. Then we can compute the embedding for token ic as
LM ER syhkS
(2)
where y is an additional scalar parameter that scales the entire vector. In our experiments we did not performed fine- tuning and thus used equal weights sj 1/ + 1) for each layer and y = 1. However, our implementation also supports all the aforementioned ways of collapsing the set of representations.
A potential drawback of the method is that it still utilizes softmax output layer with a fixed vocabulary that does not scale effectively and it still predicts UNK for OOV tokens which may have a negative effect on the representations.
4 SOURCE CODE ELMO
We describe Source Code ELMo (SCELMo), which trains ELMo on corpora of source code. However, we note that normally ELMo models in other domains are able to effectively utilize much larger representations. The code was tokenized using the esprima JavaScript tokenizer]
For training the ELMo model we used a corpus of 150,000 JavaScript Files (Raychev et al. 2016) consisting of various open-source projects. This corpus has previously been used on several tasks (Raychev et al. 2016 Pradel & Sen 2018 Bavishi et al.) 2018). We applied the patch released by Alllamanis al.] (2018a) to filter out code duplication to result in inflation of performance metries. This resulted in 64750 training files and 33229 validation files. Since the validation set contains files from the same projects as the train the contained instances might be too similar and unrealistic overestimating. To address this we also created a test set of 500 random JavaScript projects sampled from the top 20,000 open- seen-source JavaScript projects as of May 2019. The test corpus has not been previously utilized in previous work and is a better reflection of the performance of the learned bug detectors. Lastly, it is important to know what the performance of the method will if we do not have access to training data from the projects on which we would like to find bugs. This is common in practice for many real case scenarios. For training the ELMo model, we use an embedding size of 100 features for each of the forward and backward LSTMs so that each layer sums up to 200 features.
this phenomenon was shown this and other corpora
5 CONTEXTUAL EMBEDDINGS FOR PROGRAM REPAIR
In this section, we describe how contextual embeddings can be incorporated within a recent machine saaring-bassd bug detection system, the DeepBugs system Pradel & Sen (018)). In the first part of this section, we give background about the DeepBugs system, and then describe how we incorporate SCELMo within DeepBugs. DeepBugs treats the problem of finding a bug as a classification problem. The system considers a set of specific bug types, which are small mistakes that might be made in a program, such as swapping two arguments. For each bug type, DeepBugs trains binary classifier that takes a program statement as input and predicts whether the statement contains that type of bug. At test time, this classifier can be run for every statement in the program to attempt to detect bugs.
In order to train the model both examples of correct and incorrect (buggy) code are necessary. DeepBugs treats the existing code as correct and randomly mutates to obtain buggy code. To obtain training examples, we extract all expressions from the source code which are either the function calls with exactly two arguments and all binary expressions. To create instances of buggy code we mutate each of the correct instances. As such, arguments in function calls are swapped, the binary operator in binary expressions is replaced with another random one, and finally randomly either the left or the right operand replaced by another random binary operand that appears in the same file. Then the classification task is a binary task to predict whether the instance is correct, i.c., it comes from the original code, or whether it is buggy, i.c. it was one of the randomly mutated examples. The validation and test sets are mutated in the same way the training set. The split between correct and buggy instances has 50/50 class distribution as for each original code instance exactly one mutated buggy counterpart is created.
The architecture for the classifier is a feedforward network with single hidden layer of 200 dimensions with Relu activations and sigmoid output layer. For both the input and hidden layers a dropout of 0.2. The network was trained in all experiments for 10 epochs with batch size of 50 and the RMSProp optimizer. We note that for maintaining consistent comparison with DeepBugs we kept all the above parameters as well as the optimizer's 's parameters fixed to the values reported in Pradel & Sen (2018). Tuning these parameters would probably result in at least a small performance increase for our method.
https://esprima.co
3

---

Under review as a conference paper at ICLR 2020
7/ Argument order is lnversed.
2 delay 1000;
setTimeout (delay, function() 1 // Function should first. logMessage (msqvalue);
5 ;
Listing Swapped Arguments Bug
instead of was used.
2 var F. new Pronisc()) ();
3 if (promises === null s6 promises. length === 0)
4 p.done (error, result);
5
Listing 2: Incorrect Binary Operator
1 7/ Call to .lengh is missing.
2
if { index < matrix
3 do wo__smething ()
Listing 3: Incorrect Binary Operand
Figure 1: Bug type examples.
In our experiments, we consider three bug types that address set of common programming mistakes: swapped argu- ments of function calls, using the wrong binary operator and using an incorrect binary operand in a binary expression. The methodology can easily be applied to other bug types. Figure iillustrates an example of each of the three bug types.
5.1 INPUT TO THE CLASSIFIER
A key question is how a statement from the source code is converted into a feature vector that can be used within the classifier. DeepBugs
set of heuristies that, given statement and bug type, return sequence of identifiers from the statement that are most likely to be relevant. For instance, for the call to serlTimeuut in Listing the following sequence of identifiers would be extracted: [serTimeout, delay, function]. A detailed description of the heuristics is available in Appendix A
These heuristics result in sequence of program identifiers. These are converted continuous vectors using word embeddings, concatenated, and this is the input to the classifier. DeepBugs uses Word2 embeddings trained on corpus of code. In our experiments, we train classifiers using three different types of word embeddings. First, we kept the 10,000 most frequent identifiers/iterals and assigned to each of them a random embedding of 200 features. Second, to reproduce the results of Pradel & Sen (2018), we use the CBOW variant of Word2Vec learn representations consisting of 200 features for the 10,000 most frequent identifiens/itterlls. Finally, we train a FastText embeddings (Bojanowski et al. 2017) on the training set to Iearn identifier embeddings that contain subword information. The therefore
subwords used by FastText are all the character trigrams that appear in the training corpus. Identifiers composed multiple subwords. To represent an identifier, we sum the embeddings of each of its subwords and summing them up. This allows the identifier embeddings to contain information about the structure and morphology of identifiers. This also allows the FastText embeddings, unlike the Word2 Vec ones, to represent OOV words as a combination of character trigrams.
Note that DeepBugs can detect bugs only in statements that do not contain OOV (out-of-vocabulary) identifiers, be- cause its Word2Vec embeddings cannot extract features for O0V names. Instead implementation does not skip such instances. Since the original work discarded any instances that contain OOV identifiers we neither know how the method performs on such instances nor how often those appear in the utilized dataset of DeepBugs. Moreover, Deep- Bugs supported only specific subset of AST nodes and skipped the rest, For example if call's argument is complex expression consisting of other expressions then the call would be skipped. However, we expanded the implementation to support all kinds of AST nodes and to not skip instances with nested expressions discussed in Appendix A We note that we still skip an instance if one of its main parts (e.g., a function call's argument) complex expression longer than 1,000 characters such expressions might be overly long to reason about.
4

---

Under review as a conference paper at ICLR 2020
Table I: Comparison of ELMo versus non-contextual embeddings for bug detection validation set of projects. Data is restricted to expressions that contain only single names.
Random
Word2Vec
FastText
SCELMo No-Context ELMo
Swapped Arguments 86.18%
87.38%
89.55%
90.02%
92. 11%
Wrong Binary Operator
90.47%
91.05%
91.11%
92.47%
100.00%
Wrong Binary Operand
75.56%
77.06%
79.74%
81.71%
84.23%
5.2 CONNECTING SCELMO TO THE BUG DETECTOR
We investigated two variants of the bug detection model, which query SCELMo in different ways to get features for the classifier. The first utilizes the heuristic of Section A to extract small set of identifiers or literals that represent the code piece. For example, for an incorrect binary operand instance we extract one identifier or literal for the left and right operands respectively, and we also extract its binary operator. Then, those concatenated to form a query to the network. In the case of function calls we extract the identifier corresponding to the name of the called function, one identifier or literal for the first and second argument respectively and an identifiers for the expression on which the function is called. We also add the appropriate syntax tokens (a if necessary. ",' between the two arguments, and left and right parentheses) create query that resembles function call. This baseline approach creates simplistic fixed size queries for the network but does not utilize its full potential since the queries do not necessarily resemble actual code, nor correct code similar to the sequences in the training set for the embeddings. We will refer this baseline as No-Context ELMo.
Our proposed method, we compute SCELMo embeddings to the language model all the tokens of the instances for which we need representations. Valid instances are functions calls that contain exactly two arguments and binary expressions. To create fixed-size representation we extract only the features corresponding fixed set of tokens, Specifically, for functions calls we use the representations corresponding to the first token of the expression the function is called, the function name, the first token of the first argument and the first token of the second argument. While, for binary expressions of the right operand. Since the representations contain contextual information, the returned vectors can capture information about the rest of the tokens in the code sequence.
which
use those of the first token of the left operand, the binary operator, and the first token
6 RESULTS
We next discuss the experiments we performed and their corresponding results. We measured the performance of the three baselines as well as those of non-contextual ELMO and SCELMO. Measuring the performance of non-contextual ELMO allows us to evaluate how much improvement due to specifics of the language model architecture, such as the character convolutional layer which can handle OOVs, and how much is due to the contextual information itself.
6.1 PERFORMANCE ON VALIDATION SET
In our first experiment we evaluate the performance of the methods in tasks where training data from the same projects are available. The evaluation performed in this experiment gives a good estimation of how our method performs com- pared to the previous state-of-the-ant technique of DeepBugs. One main difference however is that the evaluation now also includes instances which contain OOV. As consequence the bug detections tasks are harder than those presented by |Praeel & Sen| (2018) their evaluation does not include in both the training and validation set any instance for which an extracted identifierin is OOV. Table illustraae the performance of the baselines and our models. As one would expect the FastText baseline improves over Word2Vec for all bug types due to the subword information. Moreover, our model SCELMo massively outperforms all other methods. Lastly, even no-context ELMo the heuristic version of SCELMo that does not utilize contextual information at test time outperforms the baseline methods showcasing how powerful the pretrained representations are.
6.2 INCLUDING COMPLEX EXPRESSIONS
In our next experiment we also included instances that contain elements that are complex or nested expressions. For instance, in the original work if one the arguments of a function call or one of the operands of binary expression is an expression consisting of other expressions then the instance would not be included in the dataset. Several AST node
5

---

Under review as a conference paper at ICLR 2020
Table 2: Comparison of SCELMo versus static embeddings on bug detection validation set of projects. Complex expressions are included this validation set.
Random
Word2Vec
FastText
No-Context ELMo
SCELMo
Swapped Arguments 86.37%
87.68%
90.37%
90.83%
92.27%
Wrong Binary Operator
91.12%
91.68% 74.31 %
91.92%
92.75%
100.00%
Wrong Binary Operand
72.73%
77.41%
79.65%
87.10%
Table 3: Comparison of SCELMo versus static embeddings on bug detection an external test set of 500 JavaScript projects.
Random
Word2
FastText
SCELMo No-Context ELMo
Swapped Arguments 75.79%
78.22%
79.40%
81.37%
84.25%
Wrong Binary Operator
82.95%
85.54%
83.15%
86.54%
99.99%
Wrong Binary Operand
67.46%
69.50%
72.55%
75.74%
83.59%
types such as a NewExpressi node an ObjectExpresseeee were not supported. Figure 2 a few examples of instances that would previously skipped P Such instances were skipped by Pradel Sen (2018) 8) and not included in their results. We do note though that we still skip very long expressions that contain more than 1000 tokens.
// First argument is binary expression
2 doComputation find min (comporents), callback);
Second argunent unsupported node
2 Factory.tes test (simul ator, new Car Bagle', Tallo rsi', 1993) )
Figure 2: Examples of instances that would be skipped by DeepBugs.
Similarly to the previous experiment SCELMo significantly outperforms all other models. This is evident Table Lastly, we clarify that the results of this section should not be directly compared to those of the previous one as for this experiment the training set also larger.
6.3 EXTERNAL TEST EVALUATION
The last experiment's objective showcase how the various models would perform on unseen projects this better illustrates the generalizability of the techniques. The configuration utilized is identical to that of the previous section. By looking Table 3 one can notice that the basclines have major drop performance. This a common finding in machine learning models of code, namely, that applying trained model to new software project is much more difficult than to new file in the same project. contrast, SCELMo offers up to 15% improvement in accuracy compared to Verd2Vec baseline. In fact, impressively enough SCELMo on the external test set is better than the evaluation set one of the baselines.
6.4 OOV STATISTICS
In order to better understand the above results we measured the OOV rate of the basic elements of the code instances appearing in the dataset. Here the OOV rate is calculated based on the vocabulary of 10000 entries utilized by the Word2Vec and random baseline models. These are illustrated in Tables [ measured OOV rates for both the version of the dataset used in Section 6.44 which we call Train and Validation, and that used in Section 6.2 which we call Extended Train and Extended Validation.
Tables 4 and describe the OOV rates for different parts of the expression types that considered by the DeepBugs bug detector. detailed description of the identifiers extraction heuristic can found in Appendix A
first focus
'The AST is extracted using the acorn parser htt.tt////aiaaaa..comaoo////////
6

---

Under review as a conference paper at ICLR 2020
on the swapped arguments bug pattern and consider all of the method call that have exactly two arguments. Each method call contains the function name, a name of the first argument, name of the second argument, and base object. The base object is the identifier that would be extracted from the expression (if such expression exists) which the function is called. For instance, from the following expression: window, www.naaaaagtoossser.gggeo ien.eeee "Chrome") userAgent would be extracted the base object. Table shows for each of the components how often they are OOV, In the expanded version the dataset if one the arguments a complex expression then it is converted into name based the heuristic described in Section A The resulting statisties contain valuable information as for instance, it is almost impossible for the Word2 Vec baseline to reason about a swap arguments bug if the identifiers extracted for both arguments OOV.
In a similar manner for the incorrect operand and operator bug patterns we consider all the binary operations. Each binary expression consists of left and right operand and is extracted for each of them. For each operand we also measured the frequency with which the operand corresponds to certain common types such as identifier, literal or Exisxxprssioo.
Table 4: OOV statistics for calls with exactly two arguments (Swapped arguments instances). The statistics are calculated on variants of the DeepBugs dataset.
Train
Expanded Train
Validation
Expanded Validation
Two Arguments Calls
574656
888526
289061
453486
Calls Missing Base Object
25.07%
28.63%
25.63%
28.80%
Base Object Missing or OOV
34.56%
37.38%
35.57%
38.07%
Function Name OOV
20.69%
17.07%
20.33%
16.94%
First Argument OOV
31.01%
36.99%
31.64%
37.15%
Second Argument OOV
27.25%
22.86%
27.94%
23.49%
Both Arguments OOV
11.33%
9.57%
11.96%
10.16%
Base and Function Name OOV
10.20%
8.32%
10.39%
8.61%
Base and Arguments OOV
4.21%
3.31%
4.88%
3.77%
Function Name and Arguments OOV
2.86%
2.26%
2.85%
2.28%
All Elements OOV
1.53%
1.18%
1.61%
1.27%
Table 5: OOV statistics for binary operations.
Train
Expanded Train
Validation
Expanded Validation
Binary Operations
1075175
1578776
540823
797108
Left Operand OOV
25.40%
28.84%
26.04%
29.55%
Right Operand OOV
20.37%
23.98%
20.74%
24.55%
Both Operands OOV Unknown Left Operand Type
7.82%
11.29%
8.24%
11.88%
83.36%
87.80%
83. 14%
87.74%
Unknown Right Operand Type
48.48%
47.23%
48.47%
47.05%
Both Operand Types Unknown
33.34%
36.06%
33.20%
35.87%
All OOV Unknown
3.59%
4.03%
3.81%
4.3%
7 IS NEURAL BUG-FINDING USEFUL IN PRACTICE?
Although related work (Pradel & Sen 2018: Allamanis et al. 2018h Vasic et al. 2019) has shown that there great potential for embedding based neural bug finders, the evaluation has mostly focused synthetic bugs introduced by mutating the original code. However, there is no strong indication that the synthetic bugs correlate to real ones, apart from small study of the top 50 warnings for each bug type produced by DeepBugs. A good example is the mutation operation utilized for the incorrect binary operator bug. A lot of the introduced bug instances could result in syntactic enrors. This can potentially create classifier with a high bias towards correlating buggy code syntactically incorrect code, thus hindering the model's s ability generalize on real bugs. Ideally, in an industrial environment we would like the resulting models to achieve a false positive rate of less than 10 (Sadowski et al.) (015). Sadly, high true positive rates are not to be expected as well since static bug detectors were shown to be able to detect less than 5% of bugs
7

---

Under review as a conference paper at ICLR 2020
Table 6: Real bug mined instances.
S Swapped Arguments
Wrong Binary Operator
Wrong Binary Operand
Mined Instances
303
80
1007
Table 7: Real bug identification task recall and false positive rate (FPR).
Word2 Rec-R
Word2Vec-FPR
SCELMo-Recal SCELMo-FPR
Swapped Arguments 3.34%
0.33%
49.67%
33.78%
Wrong Binary Operator
8.95%
7.70%
0.00%
0.00%
Wrong Binary Operand
11.99%
12.11%
15.81%
14.34%
(Habib & Pradel| 2018) contained in the Defects4J corpus (Just et al.) 2014) and less than 12% in single-statement bugs corpus (Karampatsis Sutton 2019). We note that in the second case the static analysis tool is given credit by reported any warning for the buggy line, so the actual percentage might lower than the reported one.
We next make a first step on investigating the practical usefulness of our methods by applying the classifiers of the previous section problem that should be carefully examined future work. In order to mine a corpus of real bug changes we used the methodology described in (Karampatsis & Sutton 2019). We note that we adapted their implementation to utilize the Rhino JavaScript ( Their methodology extracts bug fixing commits and filters them to only keep those that contain small single-statement changes. Finally, it classifies each pair of modified statements by whether the fit set of mutation patterns. The resulting dataset is shown in Table 6] Upon acceptance of the paper we will release this dataset along with our implementation, the rest of data used, and the learned representations.
small corpus of real JavaScript bugs. However, we think that this is a very hard yet interesting
Finally, we queried the DeepBugs and SCELMO with each buggy instance as well as its fixed variant and measured the percentage of correctly classified instances for each of the two categories. We also ignored any instances for which the JavaScript parser utilized for both failed to extract an AST. We classified as bugs any instances that were assigned probability to be a bug > 75%. In an actual system this threshold should ideally be tuned on validation set.
Table 7 suggests that there might indeed be some potential for future practical applications of neural bug finding techniques. Both are able to uncover some of the bugs. However, the results also suggest that careful tuning of the predictions threshold might be necessary, especially if we take into account the industrial need to comply with low false positive rate (FPR). For instance, raising SCELMo's prediction threshold to 80% for the swap arguments bug results in finding only 3.34% of the bugs but correctly classifying 100% of the repaired function calls, thus achieving 0.0% false positive rate. Moreover, since SCELMo could not uncover any of the real binary operator bugs, future work could investigate the effect of utilizing different mutation strategies for the purpose of artificial bug- induction. Future work could also investigate if fine-tuning on small set of real bugs could result in more robust classifiers.
8 CONCLUSION
We have presented SCELMo, which to our knowledge the first manguage-mode based contextual embeddings for source code. Contextual embeddings have many potential advantages for source code, because surrounding tokens can indirectly provide information about tokens, e.g. about likely values of variables. We highlight the utility of SCELMo embeddings by using them within a recent state-of-thh-an machine learning based bug detector. The SCELMo em- beddings yield a dramatic improvement in the synthetic bug detection performance benchmark, especially on lines of code that contain out-of-vocabulary tokens and complex expressions that can cause difficulty for the method. We also showed and discussed the performance of the resulting bug detectors on a dataset of real bugs raising useful insights for future work.
attp://gttubb.cc//ooill////..
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness, ACM Comput. Surv., 51(4):81:1-81:37.u July 2018a. ISSN 0360-0300. doi: 10.1145/3212695,
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs.
International Conference Learning Representations, 2018b. htt.:ttss//peeeeeeeeeeeeee/e///e
1d=BJOFETxR
Uni Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. Code2vec: Learning distributed representations of code, Proc. ACM Program. Lang., 3(POPL)):0:1 1-40:29, January 2019. ISSN 2475-1421. doi: 10. 10.1145/3290353, URL
http://dde acm. httpe/oggg..1.///// 0tt.:o/.............114143993330
Rohan Bavishi, Michael Pradel, and Koushik Sen. Contex(2name: deep learning-based appronac to infer natural
variable names from usage contexts. CoRR, abs/1809.05199, 2018. URL
05193
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword informa- tion. Transactions of the Association for Computational Linguistics, 5:135-146, 2017. doi: 10.1162/hacLa.00003
URL ftttp:://wwit ac acwwese org/ antaalottootnlaaae 10110
Zimin Chen and Martin Monperrus. A literature study of embeddings on source code. CoRR, abs/1904.03061, 2019. 2019,,
URL httt::///aaxi....rrr//////0904 03061
Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. Sequencer: Sequeencetoo-sequenee learning for end-to-end program repair. CoRR, abs/1901.01808, 2019. URL
attp://arxiv htte://azav.cccr/aaas///aee
01808
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional trans- formers for language understanding. CoRR, abs/1810.04405, 2018. URL http:://axiii org/ abs/1810.
04805
L. Gazzola, D. Micucci, and L. Mariani. Automatic software repair: A survey. IEEE Transactions on Software Engineering, 45(01):34 67, jan 2019. ISSN 1939-3520. doi: 10. (0.1109//SE.20177755013
Andrew Habib and Michael Pradel. How many of all bugs do we find? study of static bug detectors. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, pp. 317-328.
New York, NY, USA, 2018. ACM. ISBN 978-1-4503-5977-5.
doi: 10. 10.1145/3231147.2238211 URL http:
7/doi. acm. o wrg. /0. 111111131111111 3238213
Jacob A. Harer, Louis Y. Kim, Rebecca L. Russell, Onur Ozdemir, Leonard R. Kosta, Akshay Rangamani, Lei H. Hamilton, Gabriel I. Centeno, Jonathan R. Key, Paul M. Ellingwood, Mare W. McConley, Jeffrey M. Opper, Sang Peter Chin, and Tomo Lazovich. Automated software vulnerability detection with machine learning. CoRR, abs/1803.04499 2018. URL http:://aaaae http/aawwiv.g.aaaaaaa/ea/eaeee/e wtgg/ba///b///////////////////// 04497
René Just, Darioush Jalali, and Michael D. Ernst. Defects4j: A database of cxisting faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, pp. 437-440, New York, NY, USA, 2014. ACM. ISBN 2628055. htt /ww.:///iiiii. doi. org / 1111114/26111 211.5//211138861 0384. 2628055
doi: 10.1145/261034.
Rafhel-Michael Karampatsis and Charles Sutton. How Often Do Single- Bugs Occur? The Many Dataset. arXiv preprint arXiv: 1905.1334,, 2019. http:///wwaaaaaaaoo.oooooo//eaaa
Yujia Li, Richard Zemel, Mare Brockschmidt, and Daniel Tarlow. Gated graph sequence neural networks. In Proceed-
ings of ICLR 16, April 2016. URL http:///wwwwic...m... www.osiiott..ooom..ommomowwo...t com/ en us- httpau//seeeeaacccppulll.tttt
gated- gateed-grap---ssqeerreeeeenna
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Oner Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: robustly optimized BERT pretraining approach. CoRR, abs/1907. 1692, 2019. URL htth:ht::/ /aaxiiivv../ argg/bbs/190..11
9

---

Under review as a conference paper at ICLR 2020
Fan Long and Martin Rinard. Automatic patch generation by learning correct code. In Proceedings of the 43rd Annual ACM JGGPLAN-SIGCCT Symposium on Principles of Programming Languages, POPI. '16, pp. 298-312, New York, NY, USA, 2016. ACM. ISBN 978- 778-1-4503-35492
doi: 10. 10.11455233761422337777 URL http:
7/doi or rg 00t...1111//1111111 28376117
Oren Melamud, Jacob Goldberger, and Ido Dagan. context2vec: Learning generic context embedding with bidirec- tional LSTM. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 51-61, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/w1/K16- 1006. URLhttte /wwww//wwww..///////////www/./.. ac web. org/ wntaaooogg/lll--111
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems
26, pp. 3111-3119. Curran Associates, Inc., 2013. URL http: http:://aaatt /ttp://papppa//eeeeeeee nips. www..iipps.pppppppeeeepeppp cc/paper/ 5021 distributeed representations of words and phrases and their compositiona påf
Martin Monperrus. Automatic software repair: A bibliography. ACM Comput. Surv., 51(1))17:11177 January 2018. ISSN 0360-0300. doi: 10. 10.1145/3105906 URL httphh/to acm. org/10.../.............. 1145/111159900
Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. Efficient non-parametric estimation of multiple embeddings per word in vector space. In Proceedings of the 2014 Conference Empirical Methods
in Natural Language Processing (EMNLP), pp. 1059-1069, Doha, Qatar, October 2014. Association for Computa- tional Linguistics. doi: 10.3115/11/D14-111
URL https:/ / www. acwweb. org baierg/anthhn www.ttantoooooooooooooooo /D14 1113
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532- 1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D144116 URL https:// www. acwweb. org/ anth antthooocy//44-1111
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 2227- 2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/N188 1202.
URL HRRLattps:// www www.ww.www..c.. /rg/ http://nttoooooolloaoaoo
Michael Pradel and Koushik Sen. Deepbugs: A learning approach to name- name-based bug detection. Proc. ACM Program. Lang., 2 October 2018. ISSN 2475-1421. doi: 10. 10.1145/3276517. helphttp://ae /oii.
acm. wwt.ooo0111111//755555535
Veselin Raychev, Pavol Bielik, Martin Vechev, and Andreas Krause. Learning programs from noisy data. In Proceed- ings of the 43rd Annual ACM SIGPLAN-SIGAL Symposium on Principles of Programming Languages, POPL '16 pp. 761-774, New York, NY, USA, 2016. ACM. ISBN 978-1-45003-354-2 doi: 10.1145/2837611.283777 URL attp: acm. .tt.............11/////33111111.
Caitlin Sadowski, Jeffrey van Gogh, Ciera Jaspan, Emma Söderberg, and Collin Winter. Tricorder: Building a program
analysis ecosystem. In Proceedings of the 37th International Conference on Software Engineering Volume 1, ICSE
"15 pp. 598 608, Piscataway, NJ, USA, 2015. IEEE Press. ISBN 978- 17799-1934- URL http: //tt:://l.... acm. org/ citttiaiinnnn
www.ccmmmmmm2c. org.ciiaatioo..ff221.222 8744. 2818828
Joseph Turian, Lev Ratinov, and Yoshua Bengio. Word representations: A simple and general method for semi- supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL '10, pp. 384-394, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics. URL http: 7/dl.acm wttggiitation. www.ecc1mmmm1.........
Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh singh. Neural program repair by jointly learning to localize and repair. In International Conference Learning Representations, 2019. URL https:
oetaar/reweeenneeefooooooooooooo 21dmmmm1111111yyygo0omm / //openreview net hwt./oorrtr//e
Martin White, Michele Tufano, Matias Martinez, Martin Monperrus, and Denys Poshyvanyk. Sorting and transforming program repair ingredients via deep learning code similarities. pp. 479-490, 02 2019. doi: 10. 109/ 2019. 8668043.
10

---

Under review as a conference paper at ICLR 2020
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Charagram: Embedding words and sentences via character n-grams. Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pp. 1504-1515, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 18.63833/v1 D16 1157. URL URLhttps:: www. ac www..errgganhooogg.....oooo
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. XLNet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237, 2019. URL http:
//axx/l..rrg/ab//1188.0000 08237
Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Mare Brockschmidt, and Alexander L. Gaunt. Learning to
represent edits. CoRR, abs/1810. 2018. URL htt.:ttp//aax.v.oo////////////// 13337
11

---

Under review as a conference paper at ICLR 2020
A NAME EXTRACTION HEURISTIC
In order for DeepBugs to operate is necessary to extract identifiers or literals for each expression part of the statement. The bug detector for swapped arguments utilizes the following elements of the function call:
Base Object: The expression on which the function is called. The called function.
Callee:
Argument I:
The expression consisting the first argument of the called function.
Argument 2: The expression consisting the first argument of the called function.
Similarly the bug detectors for incorrect binary operators and operands utilize the following elements of the binary expression:
Binary Operator:
The binary operator utilized in the expression.
Left Operand: The left operand of the binary expression.
Right Operand: The right operand of the binary expression.
We next describe the extraction heuristic, which is shared by all the bug detectors. The heuristic takes as input a node n representing an expression and returns name(n) based on the following rules:
Identifier: return its name.
Literal: return its value.
this expression: return this.
Update expression with argument x: return name(z).
Member expression accessing property pt return name(p).
Member expression accessing property boselp): return name(base).
Call expression (ase.calleee...): return nallecal
Property node If n.key does not exist return come(n.valuss If name(n.key) does not exist return name(n.value). Otherwise randomly return either name(n. value) or name()n.key).
Binary expression with left operand / and right operand Run the heuristic on both and retrieve name(l) and name(()) If name(()) does not exist return name(r). If name(r) does not exist return mame(l). Otherwise randomly return either name(1) ir name(r).
Logical expression with left operand and right operand r: Run the heuristic both and to retrieve name(l) and nome((). If name(l) does not exist return name(r). If name(() does not exist return name(l). Otherwise randomly return either name(() ir name(r).
Assignment expression with left operand and right operand Run the heuristic both and to re- trieve name(1) and name(r). If name(I) does not exist return name(r). If name(r) does not exist return name(l). Otherwise, randomly return either name(l) ir name(r).
Unary expression with argument u Return name(u).
Array expression with elements 4 For all 4 that name(l) exists randomly choose one of them and return mlme(1.)
Conditional expression with operands e, and r: Randomly choose of c, r for which exists and return its name.
Function expression: return f'unction.
Object expression: return {.
New expression with a constructor function call e: return name(c).
All random decisions follow a uniform distribution.
12


Reviews:
[{"review_id": "rke1ZgtatB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use ELMO embeddings to improve the precision on the first step of the DeepBugs tasks defined by Pradel and Sen (2018). This first step is an artificial problem created by taking real programs (with and without bugs, but assuming almost all of them are correct) and introducing bugs of certain type into the programs. Then, a classifier needs to distinguish between the real and the artificial set. This classifier is then to be used as a checker for anomalies in code and the anomalies are reported as bugs, however the paper skips this second step and only reports results on the first classification problem.\n\nTechnically, the paper improve this first step of DeepBugs by using a standard variant of ELMO. The evaluation is detailed, but the results are unsurprising. The paper simply tech-transfers the idea from NLP to Code. If this work is accepted at the conference, I cannot imagine an interesting presentation or a poster that simply cites the changed numbers. Did we expect ELMO to be worse than more naive or random embeddings?\n\nThe work and its results heavily peg on the DeepBugs and increases the precision of its first step by a significant margin, but does not show getting any more useful results.  In fact, on one task (Wrong Binary Operator), SCELmo gets to 100% accuracy. This means it will never report any bugs, whereas DeepBugs seems to be performing best on exactly this kind of reports with its weaker model.\n\nI would recommend the authors to either work on showing practical usefulness of the technique, showing something for the full bugfinding task (not merely the first, artificial part), or to investigate if (or how) the idea to add bug-introducing changes to a code dataset is conceptually flawed for bugfinding (as this idea is widely used by several other works like Allamanis et al 2018b or by https://arxiv.org/abs/1904.01720 which also don't get to practical tools ). There seems to be some indication of this by the reported 100% accuracy, but right now this remains completely uninvestigated. \n\nMinor issues:\nListing 3: Opernad -> Operand\nPage 5. There is no Table 6.1\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgzrYrRFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper leverage recent advances of ELMo in context embedding and apply it in the source code embedding. With the help of ELMo, source embedding can take the three benefits: (1)  Surrounding names provide indirect information about possible values the variable could take; (2) an variable’s value evolves through the program execution can be captured; (3) open a gate for the reuse of the ptr-trained model. To evaluate the effectiveness of the proposed approach, authors conduct experiments on the downstream task of the bug detection. \nPros:\n1. This work study an interesting problem, which is challenging to solve.\n2. The application and combination of different techniques in this paper are smart.\n3. The experiment results show better performance of contextual embedding based method compared with non-contextual embedding based methods.\nCons:\n1. It is a good application of known techniques, but the novelty is limited.\n2. It is suggested to evaluate the effectiveness of the proposed approach on various source code analysis task such as variable misuse.\n3. It is suggested to compare with other state-of-the-art baseline methods, e.g. BERT.\n4. In the end of the introduction section, the authors claim that \"we release our implementation and representation...\". However, implementation, representation and dataset are missing.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyly6N2g5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an embedding method for source code tokens, which is based on contextual word representation, particularly is based on the method of ELMo. The learned representation is evaluated on the task of bug detection, with promising performance.\n\nStrengths:\nThe paper addresses an important and impactful problem. The solution designed for this problem seems very reasonable. Experiments are useful and reasonable and the experimental results are promising and in the favor of the paper.\nThe paper is well written and clear.\n\nWeaknesses:\n- The data used (in particular the method of buggy code generation applied) seems very specific.  It would be interesting to know the performance of the method on real bugs. \n- The paper is a bit low in technicality. \n\nDecision: Accept\nI think this paper is overall a good work and can open direction of research even beyond the scope of the paper, for example  in combining learning and reasoning, or in source code generation with adversarial models.\n\nMinor: \n- Since compilers can spot errors in code completely, it would be useful to motivate the advantage of learning for bug detection\n- The table referrals in the body of the paper contains wrong table numbers in Sections 6.1, 6.2, 6.3.\n- The incorrect Binary Operator example in Listing 2 does not seem to be a well justified bug. It could be a correct piece of code for a different purpose.\n- which use -> which we use", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper improves DeepBugs by borrowing the NLP method ELMo as new representations. The effectiveness of the embedding is investigated using the downstream task of bug detection. 

Two reviewers reject the paper for two main concerns:
1 The novelty of the paper is not strong enough for ICLR as this paper mainly uses a standard context embedding technique from NLP.
2 The experimental results are not convincing enough and more comprehensive evaluation are needed. 

Overall, this novelty of this paper does not meet the standard of ICLR.

Decision: Reject