SCELMo: Source Code Embeddings from Language Models

1 INTRODUCTION . Learning rich representations for source code is an open problem that has the potential to enable software engineering and development tools . Some work on machine learning for source code has used hand engineered features ( Long & Rinard , 2016 , e.g . ) , but designing and implementing such features can be tedious and error-prone . For this reason , other work considers the task of learning a representation of source code from data ( Allamanis et al. , 2018a ) . Many models of source code are based on learned representations called embeddings , which transform words into a continuous vector space ( Mikolov et al. , 2013 ) . Currently in software engineering ( SE ) researchers have used static embeddings ( Harer et al. , 2018 ; White et al. , 2019 ; Pradel & Sen , 2018 ) , which map a word to the same vector regardless of its context . However , recent work in natural language processing ( NLP ) has found that contextual embeddings can lead to better performance ( Peters et al. , 2018 ; Devlin et al. , 2018 ; Yang et al. , 2019 ; Liu et al. , 2019 ) . Contextualized embeddings assign a different vector to a word based on the context it is used . For NLP this has the advantage that it can model phenomena like polysemy . A natural question to ask is if these methods would also be beneficial for learning better SE representations . In this paper , we introduce a new set of contextual embeddings for source code . Contextual embeddings have several potential modelling advantages that are specifically suited to modelling source code : • Surrounding names contain important information about an identifier . For example , for a variable name , surrounding tokens might include functions that take that variable as an argument or assignments to the variable . These tokens provide indirect information about possible values the variable could take , and so should affect its representation . Even keywords can have very different meanings based on their context . For instance , a private function is not the same as a private variable or a private class ( in the case of Java / C++ ) . • Contextual embeddings assign a different representation to a variable each time it is used in the program . By doing this , they can potentially capture how a variable ’ s value evolves through the program execution . • Contextual embeddings enable the use of transfer learning . Pre-training a large neural language model and querying it for contextualized representations while simultaneously fine-tuning for the specific task is a very effective technique for supervised tasks for which there is a small amount of supervised data available . As a result only a small model needs to be fine-tuned atop the pre-trained model , without the need for task-specific architectures nor the need of training a large model for each task separately . In this paper , we highlight the potential of contextual code embeddings for program repair . Automatically finding bugs in code is an important open problem in SE . Even simple bugs can be hard to spot and repair . A promising approach to this end is name-based bug detection , introduced by DeepBugs ( Pradel & Sen , 2018 ) . The current state-of-the-art in name-based bug detection relies on static representations from Word2Vec ( Mikolov et al. , 2013 ) to learn a classifier that distinguishes correct from incorrect code for a specific bug pattern . We introduce a new set of contextualized embeddings for code and explore its usefulness on the task of name-based bug detection . Our method significantly outperforms DeepBugs as well as other static representations methods on both the DeepBugs dataset as well as a new previously unused test set of JavaScript projects . We release our implementation and representations as they could lead to improvements in a great variety of SE tasks . 2 RELATED WORK . Unsupervised static word embeddings have been extensively used to improve the accuracy of supervised tasks in NLP ( Turian et al. , 2010 ) . Notable examples of such methods are Word2Vec ( Mikolov et al. , 2013 ) and GloVe ( Pennington et al. , 2014 ) . However , the above models learn only a single context-independent word representation . To overcome this problem some models ( Wieting et al. , 2016 ; Bojanowski et al. , 2017 ) enhance the representations with subword information , which can also somewhat deal with out-of-vocabulary words . Another approach is to learn a different representation for every word sense ( Neelakantan et al. , 2014 ) but this requires knowing the set of word senses in advance . More recent methods overcome the above issues by learning contextualized embeddings . Melamud et al . ( 2016 ) encode the context surrounding a pivot word using a bidirectional LSTM . Peters et al . ( 2018 ) use a deep bidirectional LSTM , learning word embeddings as functions of its internal states , calling the method Embeddings using Language Models ( ELMo ) . We discuss ELMo in detail in Section 3 . Devlin et al . ( 2018 ) introduced bidirectional encoder representations from transformers ( BERT ) . This method learns pre-trained contextual embeddings by jointly conditioning on left and right context via an attention mechanism . Program repair is an important task in software engineering and programming languages . For a detailed review see Monperrus ( 2018 ) ; Gazzola et al . ( 2019 ) . Many recent program repair methods are based on machine learning . Yin et al . ( 2018 ) learn to represent code edits using a gated graph neural network ( GGNN ) ( Li et al. , 2016 ) . Allamanis et al . ( 2018b ) learn to identify a particular class of bugs called variable misuse bugs , using a GGNN . Chen et al . ( 2019 ) introduce SequenceR which learns to transform buggy lines into fixed ones via machine translation . Our work is orthogonal to these approaches and can be used as input in other models . Finally , our work is also related to code representation methods many of which have also been used in program repair . Harer et al . ( 2018 ) learn Word2Vec embeddings for C/C++ tokens to predict software vulnerabilities . White et al . ( 2019 ) learn Word2Vec embeddings for Java tokens and utilize them in program repair . Alon et al . ( 2019 ) learn code embeddings using abstract syntax tree paths . A more detailed overview can be found in ( Allamanis et al. , 2018a ; Chen & Monperrus , 2019 ) . 3 EMBEDDINGS FROM LANGUAGE MODELS ( ELMO ) . ELMo ( Peters et al. , 2018 ) computes word embeddings from the hidden states of a language model . Consequently , the embeddings of each token depend on its context of the input sequence , even out-of-vocabulary ( OOV ) tokens have effective input representations . In this section , we briefly describe the ELMo embeddings . The first step is that a neural language model is trained to maximize the likelihood of a training corpus . The architecture used by ELMo a bidirectional LSTM with L layers and character convolutions in the input layer . Let the input be a sequence of tokens ( t1 , ... tN ) . For each token tk , denote by xLMk the input representation from the character convolution . Consequently , this representation passes through L layers of forward and backward LSTMs . Then each layer j ∈ { 1 , ... , L } of the forward LSTM computes a hidden state −−→ hLMk , j , and likewise the hidden states of the backward LSTM are denoted by ←−− hLMk , j . The parameters for the token representation and for the output softmax layer are tied for both directions , while different parameters are learned for each direction of the LSTMs . After the language model has been trained , we can use it within another downstream task by combining the hidden states of the language model from each LSTM layer . This process is called ELMo . For each token tk of a sentence in the test set , the language model computes 2L + 1 hidden states , one in each direction for each layer , and then the input layer . To make the following more compact , we can write these as hLMk,0 = x LM k for the input layer , and then hLMk , j = [ −−→ hLMk , j , ←−− hLMk , j ] for all of the other layers . The set of these vectors is Rk = { hLMk , j |j = 0 , ... , L } . ( 1 ) To create the final representation that is fed to downstream tasks , ELMo collapses the set of representations into a single vector Ek for token tk . A simplistic approach is to only select the top layer , so that Ek = hLMk , L . A more general one , which we use in this work , is to combine the layers via fine-tuned task specific weights s = ( s1 . . . sL ) for every layer . Then we can compute the embedding for token k as Ek = γ L∑ j=0 sjh LM k , j , ( 2 ) where γ is an additional scalar parameter that scales the entire vector . In our experiments we did not performed finetuning and thus used equal weights sj = 1/ ( L + 1 ) for each layer and γ = 1 . However , our implementation also supports all the aforementioned ways of collapsing the set of representations . A potential drawback of the method is that it still utilizes a softmax output layer with a fixed vocabulary that does not scale effectively and it still predicts UNK for OOV tokens which may have a negative effect on the representations . 4 SOURCE CODE ELMO . We describe Source Code ELMo ( SCELMo ) , which trains ELMo on corpora of source code . However , we note that normally ELMo models in other domains are able to effectively utilize much larger representations . The code was tokenized using the esprima JavaScript tokenizer1 . For training the ELMo model we used a corpus of 150,000 JavaScript Files ( Raychev et al . 2016 ) consisting of various open-source projects . This corpus has previously been used on several tasks ( Raychev et al. , 2016 ; Pradel & Sen , 2018 ; Bavishi et al. , 2018 ) . We applied the patch released by Allamanis et al . ( 2018a ) to filter out code duplication as this phenomenon was shown on this and other corpora to result in inflation of performance metrics . This resulted in 64750 training files and 33229 validation files . Since the validation set contains files from the same projects as the train the contained instances might be too similar and unrealistic overestimating . To address this we also created a test set of 500 random JavaScript projects sampled from the top 20,000 open-source JavaScript projects as of May 2019 . The test corpus has not been previously utilized in previous work and is a better reflection of the performance of the learned bug detectors . Lastly , it is important to know what the performance of the method will be if we do not have access to training data from the projects on which we would like to find bugs . This is common in practice for many real case scenarios . For training the ELMo model , we use an embedding size of 100 features for each of the forward and backward LSTMs so that each layer sums up to 200 features .

Summary: The paper proposes to use ELMO embeddings to improve the precision on the first step of the DeepBugs tasks defined by Pradel and Sen (2018). This first step is an artificial problem created by taking real programs (with and without bugs, but assuming almost all of them are correct) and introducing bugs of certain type into the programs. Then, a classifier needs to distinguish between the real and the artificial set. This classifier is then to be used as a checker for anomalies in code and the anomalies are reported as bugs, however the paper skips this second step and only reports results on the first classification problem.