日本ソフトウェア科学会第 36 回大会 (2019 年度) 講演論文集

ASTToken2Vec: An Embedding Method for
Neural Code Completion
Li Dongfang

Masuhara Hidehiko

Code completion systems help programmers to write code more eﬃciently and to reduce typographical errors by automatically suggesting the code fragment that the programmers likely to write next. This work
attempts to increase prediction performance of an LSTM-based code completion system proposed by Chang
Liu et al. by proposing a new embedding method (a vector representation) for AST nodes. This method iscalled ASTToken2Vec, similar to Word2Vec, which trains a neural network by using context information to
give a vector representation of an AST node. We integrate our embedding method with an LSTM modeland evaluate its prediction performance on a JavaScript AST dataset generated from open-source programs
containing a total of 150,000 JavaScript ﬁles.

1 Introduction
Code completion systems are one of the most useful and indispensable tools in modern integrated
development environments (IDEs). They help programmers writing code faster with less typographical errors. An eﬀective code completion system
can predict what kind of code programmers want
to write and generate a code suggestion list automatically.
Traditional code completion systems are not intelligent enough. Most of them are based on statistical methods and suggest code only by simple
term frequency which often relatively have a higher
error rate. Another problem is that they rely on
static types (like Eclipse for Java) to ﬁlter out candidates. However, the dependency on typing information limits their applicability to widely used indynamically typed languages like Python.etc.
With the rapid development of deep learningtechniques, more and more research pays attention
to applying deep learning models, especially recurrent neural networks (RNNs) to automatic code
completion. RNN models achieved excellent performance in the natural language processing (NLP)
李 東方, 増原 英彦, 東京工業大学数理計算科学系, Dept.
of Mathematical and Computing Science, Tokyo Institute of Technology.

ﬁeld. Intuitively, a programming language can beconsidered as a special language just with stricter
semantic and syntax. Hence, it is straightforward
to apply RNNs to programming languages tasks
like code completion and may have an exciting performance.
In this paper, we propose an embedding method
for abstract syntax trees (ASTs) nodes called ASTToken2Vec to improve the prediction performanceof the deep learning-based code completion. It consists of a four-layer neural network for encoding thecontext information of ASTs to generate semanticbased embedding representation vectors that contain more knowledge hidden behind ASTs. We use
this ASTToken2Vec model as a pre-trained modeland integrate it with an LSTM based model as acode completion system to predict next tokens. We
name our integration model as AT2V-LSTM modeland the overview of our model illustrates in Figure
1.
As an experiment, we implement both ASTToken2Vec embedding and AT2V-LSTM modelwith a JavaScript AST dataset [10] collected fromopen-source programs containing a total of 150,000
JavaScript ﬁles. We visualize representation vectors of several terminal tokens and evaluate theperformance of predicting next tokens by AT2VLSTM integration model From the evaluation andresults analysis, we conclude that the ASTTo-Fig. 1

Overview of ASTToken2Vec LSTM
integration model

ken2Vec method is able to generate semantic-based
representation vectors of AST nodes and the AT2VLSTM integrated model predicts tokens more correctly.

2 Background
Hindle et al. [3] explore how to use a widelyadopted n-gram statistical language model for code
completion and provide empirical evidence which
is able to prove that programming language codeis even more repetitive than natural languages.
Nguyen et al. [6] propose generative models of natural source code with hierarchical structure and a
distributed representation of source code element.
They also leverage compiler logic and abstractions
to improve their generative models. Tung et al. [8]extends the state-of-the-art n-gram approach whichis called SLAMC by incorporating semantic information into code tokens.
Comparing with n-gram model, a statistical nonparametric Bayesian probabilistic tree-based system [1] is proposed by Allamanis et al. which isable to extract code idioms from the existing written code ﬁles. Liang et al. [4] focus on learning programs for multiple related tasks with a few training
samples and propose a nonparametric hierarchical
Bayesian model which is able to share the statisticalinformation across multiple tasks for code completion.
Bielik et al. [2] introduce a generative model
for code called probabilistic higher-order grammar(PHOG). This model is able to capture the richcontext information between tokens by allowing
conditioning of a production rule. Raychev et al.
[9] create a domain-speciﬁc language (DSL) overabstract syntax trees (ASTs) called TGEN which
can encode an AST to a speciﬁc language context.
They also propose a special decision tree called
DEEP3 which can make code predictions leveraging the AST context encoded by the TGEN model.
Raychev et al. [11] and White et al. [12] explore
how to apply RNN models on sequences of tokens
to facilitate the task of code completion. Chang etal. [5] propose several LSTM-based models for code
completion with an AST dataset and they leverage the ASTs by converting ASTs to sequences oftraining samples. Their work gives us inspiration
about how to convert an AST to a sequence andhow to train an LSTM model with AST dataset.

3 Abstract Syntax Tree
Abstract Syntax Trees (AST) are a tree structurethat represents structural information of programs.
It is widely used in code completion engines.
There are two kinds of a node in ASTs, nonterminal nodes and terminal nodes. A non-terminal
node has a children node list denoting all its children nodes. For example, in JavaScript, a nonterminal token could be “FunctionDeclaration”,“VariableDeclarator”, etc. These non-terminal tokens declare what kind of functions or variables
speciﬁed in the program. Other kinds of nonterminal tokens represent more knowledge about
the structure and logical judgment of a program
like “ForStatement”, “IfStatement”, “WhileStatement”, etc.
A terminal node has “value” to represent thevalue of terminals and does not have any children
node. In JavaScript, for instance, the type of terminal tokens could be “LiteralString”, “LiteralNumber”, “Identiﬁer” etc. Due to programmers can
specify any strings, variables and function’s namein a program, it is obvious that there are inﬁnite
possibilities for terminal tokens.
Due to ASTs contain more semantic knowledge
about source code, we use ASTs as the basic data to
generate two sets of training samples for our models’ training.

4 ASTToken2Vec Embedding
ASTToken2Vec model is an embedding model for
AST nodes. It is inspired by the embedding method
for natural languages, namely Word2Vec [7]. It
trains a four-layers neural network to generate
the semantic-based representation vectors of AST
nodes which enable the LSTM-based code completion model to leverage more structural knowledge
to predict next tokens. In order to do that, we givea basic hypothesis of ASTToken2Vec which is sameas Word2Vec’s: we assume that if two nodes in an
AST have a similar context, the meaning of these
two nodes also has a high-level similarity. We speciﬁed the surrounding non-terminal nodes of a target
node as the non-terminal context and surrounding
terminal nodes as the terminal context. The details of contexts is explained in subsection4. 2 andsubsection4. 3.
4. 1 Model Architecture
The ASTToken2Vec model is a four layers neuralnetwork contains one input layer, one single hidden layer, and two output layers. Due to there
are two kinds of tokens non-terminal and terminal
in ASTs, we design two application variant of ourembedding model: ASTToken2Vec for terminal token abbreviated as TT2V and ASTToken2Vec fora non-terminal token called NT2V.
The input layer is the one-hot encoding representation vector of a non-terminal node for the NT2V
model and a terminal node for the TT2V model.
The ASTToken2Vec model has two output layers:
non-terminal output layer for non-terminal context
representation and terminal output layer representing the terminal context of the input node. The
length of these two output layers is equal to thesize of the terminal vocabulary and non-terminal
vocabulary separately. We use the values in thehidden layer as the embedding vectors to represent
the input AST node and the length of the hidden
layer is a hyperparameter D speciﬁed by users. In
the training phase, we calculate a joint loss function
to update the model.
Figure 2(1) illustrates the architecture of the
NT2V model and Figure 2(2) is the architectrueof TT2V model.

Fig. 2

Architecture of NT2V (1) and TT2V(2)

4. 2 Embedding for Non-terminals
The ASTToken2Vec model for encoding nonterminals to vectors is abbreviated as NT2V. It
employs both the terminal context and the nonterminal context of a target non-terminal to generate embedding representation vectors. A training tuple for NT2V contains three elements: (target non-terminal, non-terminal context, terminal
context) where the target non-terminal is the nonterminal token which the NT2V model generates
embedding vector for. The second and third elements are lists representing the contexts of the input target non-terminal. We deﬁne these two contexts as follows.
4. 2. 1 Non-terminal context
The non-terminal context for a non-terminal token means the surrounding non-terminal nodes of
it in an AST. Concretely, we deﬁne the n parent
non-terminal nodes of a target non-terminal and its
all non-terminal children nodes as its non-terminal
context. Here, n is a hyper-parameter which declares the scope of the parent non-terminal context
employed by the NT2V model. If n is relatively
small, it means NT2V model does not consider thesurrounding non-terminal tokens which are far fromthe target node as the non-terminal context.
4. 2. 2 Terminal context
We deﬁne the terminal context of a target nonterminal as all its children terminal nodes. If a
non-terminal node does not have any terminal children nodes (all its children nodes are non-terminal
tokens or it does not have any children), we use a
special terminal token: TT-EMPTY to declare an
empty terminal context for it.
We use an example of partial AST to illustratethe contexts of a non-terminal in the Figure 3(1).
The nodes whose name starts with “NT” are nonterminal nodes and nodes whose name starts with
“TT” represent terminal nodes. In this AST, We
assume a target non-terminal node “NT-4” which
is surrounded by an oval. Non-terminal nodes surrounded by a rectangle are the non-terminal context of the target node including “NT-2” and “NT1”. Terminal nodes: “TT-1”, “TT-2” and “TT-3”
which have an underline mean the terminal context of the target node “NT-4”. Hyper-parametern here is speciﬁed as two.

Fig. 3

Non-terminal and terminal context for

non-terminal nodes (1) and terminal nodes (2)4. 3 Embedding for Terminals
The ASTToken2Vec model for terminal tokens’
embedding vectors generation is abbreviated as
TT2V. Similar with NT2V model, the structure oftraining tuples for TT2V is (target terminal, nonterminal context, terminal context) NT2V model
leverage both non-terminal context and terminal
context to generate the embedding vectors for thetarget terminal. The non-terminal context and terminal context of a target terminal are described as
below.
4. 3. 1 Non-terminal context
Because there is no child node of a terminal
node, we only consider the non-terminal context ofa terminal as only its n parent non-terminal nodes.
Same with NT2V, hyperparameter n is used to deﬁne the scope of non-terminal context.
4. 3. 2 Terminal context
We deﬁne the terminal context of a target terminal node as m neighbor terminal nodes in an AST.
Neighbor terminal nodes of a target terminal meanits sibling terminals which have the same parent

non-terminal with the target terminal. Here m is a
hyperparameter to specify the size of the terminal
context. A relatively small m means TT2V model
does not consider too many surrounding terminal
nodes as the terminal context. If a terminal node
does not have any neighbor terminal nodes whichmeans its parent node only has one single terminal
child, in this case, we use a special terminal node:
TT-EMPTY to represent an empty terminal context.
The partial AST in the Figure 3(2) shows a concrete example of what is contexts for terminals. We
specify a target terminal node “TT-2” which is surrounded by an oval. Non-terminal nodes: “NT-2”
and “NT-4” which are emphasized by a rectangle,
represent the non-terminal context and terminal
nodes “TT-1” and “TT-3” are the terminal context
of the target node. In this example, hyperparameter n is speciﬁed as two and m is equal to one.
4. 4 Joint Loss Function
Due to there are two output layers in our ASTToken2Vec model, we design a joint loss function combining the non-terminal context output and terminal context output.
There are three parts of the loss function calculation. Lossnt is used to represent the loss of nonterminal context output. The loss of terminal context output is represented by Losstt . Both of them
are multi-labels loss calculations because there are
more than one surrounding tokens as the context.
And Losstotal is the ﬁnal joint loss function for ourmodel’s training.
Lossnt = −

N
∑

i
i
(ynt−context
× log(ŷnt−context
))

i=1

(1)
Equation1 is the Lossnt calculation formulawhich is a log loss function. Concretely, for an
input token x, ASTToken2Vec model calculates
the non-terminal context output as ŷnt−contextiand ynt−context
represents its ground-truth nonterminal context. N is the size of the non-terminal
vocabulary.
Losstt = −

M
∑

j
j
(ytt−context
× log(ŷtt−context
))

j=1

(2)
Equation2 illustrates the formula of Losstt calcu-

lation which also a log loss function. ŷtt−context isthe terminal output of our model. and ytt−context
is the ground truth label of the terminal context.
M is the size of terminal vocabulary.

5. 2 Model Architecture
The architecture of our integration model is illustrated in the Figure 4. It contains one input layer,
one LSTM layer, and an output layer.

Losstotal = α ∗ Lossnt + (1 − α) × Losstt (3)Equation3 is the joint loss function combining
Lossnt and Losstt . We utilize a hyperparameter
α whose range is from zero to one to adjust the importance between the loss of non-terminal context
output Lossnt and terminal context output Losstt .

5 AT2V-LSTM Integration
We integrate a basic LSTM model with our ASTToken2Vec embedding method. This integration
model is called AT2V-LSTM which is able to leverage the semantic-based information extracted by
ASTToken2Vec embedding and predict the next tokens as code completion.
5. 1 Sequences of Training Samples
In order to train the linear-structured LSTM
model, we convert ASTs to sequences of training
samples. Basically, we ﬁrst convert an AST to aleft-child-right-sibling (LC-RS) binary tree. Then,
we transform this LC-RS binary into a complete
binary tree by padding a special non-terminal node
NT-EMPTY. Next, we apply a deep-ﬁrst in-order
traversal on this complete binary tree to generate
a visiting sequence of training samples. There are
four elements in a sample: (non-terminal, terminal, node-or-leaf, right-or-left). Concretely, whena non-terminal node is visited, it is considered
as a target non-terminal and the ﬁrst element in
the sample. The second element: terminal is thechildren terminals of the target non-terminal. If
the target non-terminal does not have any terminal child, we use a speciﬁed terminal token TTEMPTY to represent its empty child. The last twoelements: node-or-leaf is used to declare whether
the non-terminal is a leaf or not in the complete binary tree and right-or-left represents the position
relationship between the target non-terminal andits parent node. These two elements are used toreconstruct the predicting AST from the sequenceof training samples.

Fig. 4

The architectrue of AT2V-LSTM model

5. 2. 1 Input layer
The input layer is a combination layer of therepresentation vectors of feeding elements andthese vectors are initialized by ASTToken2Vec
embedding. The input is a sequence of training samples and one sample has four elements:(Ni , Ti , N Li , RLi ) where Ni is a non-terminal, Ti isa terminal, N Li is the type information of Ni and
RLi is the side information of Ni . All these four
elements are encoded by one-hot encoding and multiply embedding matrices as below:
Inputi = Concat(A·Ni +B ·Ti ,C ·N Li +D·RLi )(4)
where A, B are the embedding matrices for nonterminals, terminals and initialized by embedding
vectors generated by ASTToken2Vec. A is a K×VN
matrix and the shape of matrix B is K × VT . K
is the length of embedding vectors. VN and VT are
the size of non-terminal vocabulary and terminal
vocabulary respectively. C, D are embedding matrices for type information, and side information.
5. 2. 2 LSTM layer
The LSTM layer receives the embedding vectors
from the input layer as xi and takes the output
ht−1 and hidden state ct−1 from the previous stateof LSTM layer. Then, the LSTM layer computes
three operating gates: forget gate, update gate andoutput gate to calculate a new hidden state as ht
and the new output as ct .

5. 2. 3 Output layer
The output layer has four trainable matrices
as the linear mapping between the output of the
LSTM layer and the prediction. There are four
instances our model predicting: next non-terminal
Ni+1 , next terminal Ti+1 , type and side information of the next non-terminal, N Li+1 and RLi+1 .
The formula of output layer is as below:
Pi = sof tmax(W × hi + b)
(5)
where P is the prediction of next training sample
including pn , pt , pnl and prl representing the prediction of next non-terminal, next terminal, the sideand type information of pn . W s are four trainablematrices for linear mapping and hi is the output of
the LSTM layer. The softmax function returns thepossibility of the next tokens predicting.

6 Experiments
6. 1 Dataset Details
The data we use for both the ASTToken2Vec embedding model training and AT2V-LSTM integration model training is from the same dataset which
is a JavaScript AST dataset provided by Raychev etal. [10]. There are 100,000 ASTs as training dataset
and 50,000 ASTs as evaluation dataset.
6. 1. 1 Non-terminal Vocabulary
There are 44 diﬀerent kinds of non-terminal tokens speciﬁed by the JavaScript programming language grammar. Base on these 44 non-terminals,
we add two more bits of information: whether the
non-terminal token has a child token; whether thisnon-terminal has a right sibling or not. These twobits care more about the surrounding context of
non-terminals and make the task of non-terminal
predict become more challenge. This adjunctionis also used in the previous work [5] [10]. There
are 98 kinds of bits-information combination nonterminal tokens in total as the elements in the
non-terminal vocabulary including the special nonterminal token:NT-EMPTY we use as a padding
token to build a complete binary tree from an AST.
6. 1. 2 Terminal Vocabulary
Theoretically, there are inﬁnite kinds of terminal
tokens may be included in programs. So, we use
the idea of Word of Bag to specify the terminal
vocabulary. Concretely, we sort all terminal tokens appearing in the training dataset by their fre-quencies of occurrence. Then we choose the 50,000
most frequent terminal tokens as the vocabulary of
the terminal. For infrequent terminal tokens (outof our terminal vocabulary bag), we use a special
terminal token UNK to represent these terminals.
In total, we have 50,002 tokens in the vocabularyof the terminal including particular terminal UNK
and TT-EMPTY which is used to represent a childterminal for a non-terminal who does not have aterminal child.
6. 2 Experiment of ASTToken2Vec
6. 2. 1 Training details
We implement ASTToken2Vec models to generate embedding vectors for both non-terminals andterminals. We deﬁne the size of the hidden layer D
is equal to 1,000 so that the length of embedding
vectors is 1,000. We specify the adjuster α in the
joint loss function as 0.6. We use the Adam optimization algorithm with the learning rate of 0.002
to train models. The size of the training batch for
ASTToken2Vec is b = 100 and the training epoch
is e = 10.
6. 2. 2 Visualization
We visualize the representation vectors of several terminal tokens to show the performance of
ASTToken2Vec. We ﬁrst apply principal component analysis (PCA) algorithm to these vectors to
reduce the dimension from 1000-d to 2-d. Then we
normalize the 2-d vectors with min-max normalization so that the entire range of values of elements
is -2 to 2.
We pick up several terminal tokens to visualize,
the visualization is shown in Figure 5. Terminal
token Identiﬁers are blue, LiteralNumbers are represented by purple, Property terminals are green
token and red tokens are LiteralString in the ﬁgure.
In the ﬁgure, there are several clusters like: “literal string cluster”, “property cluster” which represent diﬀerent types of terminals. We also ﬁnd thateven tokens are the same type, if their meaning is
diﬀerent, they are still not in the same cluster. For
example, even the type of “append” and “value”
are Property, they are still far from each other because “append” is a property which can add someelements to a container in the most cases. However“value” usually is a member in a class without some
operation functionality of a container. Another ex-Models
Basic LSTM
AT2V-LSTM
Table 1

Top 1 accuracy
83.5 ± 0.2%
85.2 ± 0.2%

Top 3 accuracy
92.6 ± 0.2%
94.4 ± 0.2%

Non-terminal evaluation accuracy

integration model is a little higher than the accuracy of the basic LSTM model. The evaluation result illustrated in the table1 also shows that theaccuracy of the integration model is 1.5% higher
than the basic baseline model.

Fig. 5The visualization of embedding vectors

ample is “length” and “userName”. Due to they
have a diﬀerent meaning, they are not in the samecluster even they are all Identiﬁers
We also calculate the cosine similarity between
these embedding vectors and the result meets ourconclusion of visualization.
6. 3 Experiment of AT2V-LSTM
6. 3. 1 Training details
We implement two models for code completion:
basic LSTM model and our AT2V-LSTM integration model to compare the predicting performance.
We use Adam optimization algorithm to train ourmodel with base learning rate 0.0025 and it multiplies 0.9 every epoch as learning rate decay. We clip
the gradient which is more than 6 to 6 and less than
-6 to -6 to avoid the gradient exploding problem.
We specify the time sequence s = 50 and the batch
size is b = 100, therefore, there are s × b = 5000
training instances for one training batch. We train
two models e = 10 epochs.
6. 3. 2 Next Non-terminal Prediction
Valid accuracy curve of the next non-terminal token prediction during the training phase is illustrated in Figure 6. The blue curve represents thevalidation accuracy of the basic LSTM model andthe orange curve is our AT2V-LSTM integration
model.
From the validation accuracy curves, we ﬁnd thenon-terminal prediction accuracy of AT2V-LSTM

Fig. 6

Validation accuracy for non-terminal

prediction during the training phase

6. 3. 3 Next Terminal Prediction
Figure 7 illustrates the validation accuracy curvefor next terminal token prediction during the training phase. Orange curve represents our AT2VLSTM integration model and the blue curve is theaccuracy of the basic LSTM model. The evaluation
accuracy for the terminal in the test phase is shownin Tabel2. From both the evaluation result, we canﬁnd that the AT2V-LSTM integration model has abetter performance with the predicting accuracy of
78.9% than the basic baseline model 75.8%.

Fig. 7

Validation accuracy for terminal

prediction during the training phaseModels
Basic LSTM
AT2V-LSTM
Table 2

Top 3 accuracy
87.7 ± 0.2%
89.2 ± 0.2%

Terminal evaluation accuracy

ModelsBasic LSTM
AT2V-LSTM
Table 3

Top 1 accuracy
75.8 ± 0.2%
78.9 ± 0.2%

Type accuracy
97.6 ± 0.2%
97.8 ± 0.2%

Side accuracy
94.8 ± 0.2%
95.1 ± 0.2%

Type and side evaluation accuracy

6. 3. 4 Next Token Information Prediction
The evaluation result of the type (non-leaf orleaf) and the side (right-child or left-child) is shownin the Table3. We can ﬁnd that both two models
achieve a good performance that the accuracy oftype information is near to 97% and side information is near to 95%.
6. 3. 5 Uncommon terminal repeation
We analyze the possible reasons that may cause
our integration model to work better for some evaluation cases.

hole. The expecting token is “Property pageY”.
The basic model predicts terminal “UNK” whichmeans this model consider the token appearing in
the hole is a quite uncommon terminal. However,
our AT2V-LSTM model gives a correct prediction.
From the programming habits and the statistic ofthe dataset, the “property pageY” is an uncommon
terminal only appears in several ﬁles but repeats
many times in one ﬁle. Due to it is uncommon, thebasic LSTM model is hard o ltearn enough informatnio and give an incorrect prediction. On thecontrary, the AT2V-LSTM integration model canleverage the semantic information of “pageY” extracted by the ASTToken2Vec embedding and givea correct prediction.

7 Conclusion
In this paper, we propose an embedding method
for AST nodes called ASTToken2Vec. We integrate
it with a basic LSTM model to build an integration
model called AT2V-LSTM. From the results of ourexperiments, we conclude that the ASTToken2Vec
model is able to generate the semantic-based embedding representation vectors for tokens. These
embedding vectors enable our AT2V-LSTM integration model to leverage more semantic knowledge
hidden behind ASTs and to complete code witha higher possibility of predicting next tokens correctly.
References

Fig. 8

Code snippets for prediction result
analysis

The code snippet in the ﬁgure8 is a test case andmodels predict what token should be ﬁlled in the[ 1 ] Allamanis, M. and Sutton, C.: Mining idioms
from source code, Proceedings of the 22nd ACM
SIGSOFT International Symposium on Foundations of Software Engineering, ACM, 2014,
pp. 472–483.
[ 2 ] Bielik, P., Raychev, V., and Vechev, M.: PHOG:
probabilistic model for code, International Conference on Machine Learning, 2016, pp. 2933–2942.
[ 3 ] Hindle, A., Barr, E. T., Su, Z., Gabel, M., andDevanbu, P.: On the naturalness of software, 2012
34th International Conference on Software Engineering (ICSE), IEEE, 2012, pp. 837–847.
[ 4 ] Liang, P., Jordan, M. I., and Klein, D.: Learning
programs: A hierarchical Bayesian approach, Proceedings of the 27th International Conference on
Machine Learning (ICML-10), 2010, pp. 639–646.
[ 5 ] Liu, C., Wang, X., Shin, R., Gonzalez, J. E., andSong, D.: Neural code completion, (2016).
[ 6 ] Maddison, C. J. and Tarlow, D.: Structured
Generative Models of Natural Source Code, CoRR,

Vol. abs/1401.0514(2014).
[ 7 ] Mikolov, T., Sutskever, I., Chen, K., Corrado,G., and Dean, J.: Distributed Representationsof Words and Phrases and their Compositionality,
CoRR, Vol. abs/1310.4546(2013).
[ 8 ] Nguyen, T. T., Nguyen, A. T., Nguyen, H. A.,
and Nguyen, T. N.: A Statistical Semantic Language Model for Source Code, Proceedings of the2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, New York, NY,
USA, ACM, 2013, pp. 532–542.
[ 9 ] Raychev, V., Bielik, P., and Vechev, M.: Probabilistic model for code with decision trees, ACM
SIGPLAN Notices, Vol. 51, No. 10, ACM, 2016,

pp. 731–747.
[10] Raychev, V., Bielik, P., Vechev, M., and Krause,A.: Learning Programs from Noisy Data, SIGPLAN
Not., Vol. 51, No. 1(2016), pp. 761–774.
[11] Raychev, V., Vechev, M., and Yahav, E.: Codecompletion with statistical language models, Acm
Sigplan Notices, Vol. 49, No. 6, ACM, 2014,
pp. 419–428.
[12] White, M., Vendome, C., Linares-Vásquez, M.,
and Poshyvanyk, D.: Toward deep learning software repositories, Proceedings of the 12th Working
Conference on Mining Software Repositories, IEEE
Press, 2015, pp. 334–345.