Open Vocabulary Learning on Source Code with a Graph-Structured Cache

1 INTRODUCTION . Computer program source code is an abundant and accessible form of data from which machine learning algorithms could learn to perform many useful software development tasks , including variable name suggestion , code completion , bug finding , security vulnerability identification , code quality assessment , or automated test generation . But despite the similarities between natural language and source code , deep learning methods for Natural Language Processing ( NLP ) have not been straightforward to apply to learning problems on source code ( Allamanis et al. , 2017 ) . There are many reasons for this , but two central ones are : 1 . Code ’ s syntactic structure is unlike natural language . While code contains natural language words and phrases in order to be human–readable , code is not meant to be a read like a natural language text . Code is written in a rigid syntax with delimiters that may open and close dozens of lines apart ; it consists in great part of references to faraway lines and different files ; and it describes computations that proceed in an order often quite distinct from its written order . 2 . Code is written using an open vocabulary . Natural language is mostly composed of words from a large but closed ( a.k.a . fixed–size and unchanging ) vocabulary . Standard NLP methods can thus perform well by fixing a large vocabulary of words before training , and labeling the few words they encounter outside this vocabulary as “ unknown ” . But in code every new variable , class , or method declared requires a name , and this abundance of names leads to the use of many obscure words : abbreviations , brand names , technical terms , etc.1 A model must be able to reason about these newly–coined words to understand code . The second of these issues is significant . To give one indication : 28 % of variable names contain out–of–vocabulary words in the test set we use in our experiments below . But more broadly , the open vocabulary issue in code is an acute example of a fundamental challenge in machine learning : 1We use the terminology that a name in source code is a sequence of words , split on CamelCase or snake case . E.g . the method name addItemToList is composed of the words add , item , to , and list . We also use the term variable in its slightly broader sense to refer to any user–named language construct , including function parameter , method , class , and field names , in addition to declared variable names . how to build models that can reason over unbounded domains of entities , sometimes called “ open– set ” learning . Despite this , the open vocabulary issue in source code has received relatively little attention in prior work . The first issue , in contrast , has been the focus of much prior work . A common strategy in these works is to represent source code as an Abstract Syntax Tree ( AST ) rather than as linear text . Once in this graph–structured format , code can be passed as input to models like Recursive Neural Networks or Graph Neural Networks ( GNNs ) that can , in principle , exploit the relational structure of their inputs and avoid the difficulties of reading code in linear order ( Allamanis et al. , 2018 ) . Our contribution : In this paper we extend such AST–based models for source code in order to address the open vocabulary issue . We do so by introducing a Graph–Structured Cache ( GSC ) to handle out–of–vocabulary words . The GSC represents vocabulary words as additional nodes in the AST as they are encountered and connects them with the edges to where they are used in the code . We then process the AST+GSC with a GNN to produce outputs . See Figure 1 . We empirically evaluated the utility of a Graph–Structured Cache on two tasks : a code completion ( a.k.a . fill–in–the–blank ) task and a variable naming task . We found that using a GSC improved performance on both tasks at the cost of an approximately 30 % increase in training time . More precisely : even when using hyperparameters optimized for the baseline model , adding a GSC to a baseline model improved its accuracy by at least 7 % on the fill–in–the–blank task and 103 % on the variable naming task . We also report a number of ablation results in which we carefully demonstrate the relative importance of each model component to a model ’ s performance . 2 PRIOR WORK . REPRESENTING CODE AS A GRAPH Given their prominence in the study of programming languages , Abstract Syntax Trees ( ASTs ) and parse trees are a natural choice for representing code and have been used extensively . Often models that operate on source code consume ASTs by linearizing them ( usually with a depth–first traversal ) as in Amodio et al . ( 2017 ) , Liu et al . ( 2017 ) , or Li et al . ( 2017 ) or by using AST paths as input features as in Alon et al . ( 2018 ) , but they can also be processed by deep learning models that take graphs as input , as in White et al . ( 2016 ) and Chen et al . ( 2018 ) who use Recursive Neural Networks ( RveNNs ) ( Goller & Kuchler , 1996 ) on ASTs . RveNNs are models that operate on tree– topology graphs , and have been used extensively for language modeling ( Socher et al. , 2013 ) and on domains similar to source code , like mathematical expressions ( Zaremba et al. , 2014 ; Arabshahi et al. , 2018 ) . They can be considered a special case of Message Passing Neural Networks ( MPNNs ) in the framework of Gilmer et al . ( 2017 ) : in this analogy RveNNs are to Belief Propagation as MPNNs are to Loopy Belief Propagation . They can also be considered a special case of Graph Networks in the framework of Battaglia et al . ( 2018 ) . ASTs also serve as a natural basis for models that generate code as output , as in Maddison & Tarlow ( 2014 ) , Yin & Neubig ( 2017 ) , Rabinovich et al . ( 2017 ) , Chen et al . ( 2018 ) , and Brockschmidt et al . ( 2018 ) . Data–flow graphs are another type of graphical representation of source code with a long history ( Krinke , 2001 ) , and they have occasionally been used to featurize source code for machine learning ( Chae et al. , 2017 ) . Most closely related to our work is the work of Allamanis et al . ( 2018 ) , on which our model is heavily based . Allamanis et al . ( 2018 ) combine the data–flow graph and AST representation strategies for source code by representing code as an AST augmented with extra labeled edges indicating semantic information like data– and control–flow between variables . These augmentations yield a directed multigraph rather than just a tree,2 so in Allamanis et al . ( 2018 ) a variety of MPNN called a Gated Graph Neural Network ( GGNN ) ( Li et al. , 2016 ) is used to consume the Augmented AST and produce an output for a supervised learning task . Graph-based models that are not based on ASTs are also sometimes used for analyzing source code , like Conditional Random Fields for joint variable name prediction in ( Raychev et al. , 2015 ) . 2This multigraph was referred to as a Program Graph in Allamanis et al . ( 2017 ) and is called an Augmented AST herein . REASONING ABOUT OPEN SETS The question of how to gracefully reason over an open vocabulary is longstanding in NLP . Character–level embeddings are a typical way deep learning models handle this issue , whether used on their own as in Kim et al . ( 2016 ) , or in conjunction with word–level embedding Recurrent Neural Networks ( RNNs ) as in Luong & Manning ( 2016 ) , or in conjunction with an n–gram model as in Bojanowski et al . ( 2017 ) . Another approach is to learn new word embeddings on–the–fly from context ( Kobayashi et al. , 2016 ) . Caching novel words , as we do in our model , is yet another strategy ( Grave et al. , 2017 ) and has been used to augment N–gram models for analyzing source code ( Hellendoorn & Devanbu , 2017 ) . In terms of producing outputs over variable–sized input and outputs , also known as open–set learning , attention-based pointer mechanisms were introduced in Vinyals et al . ( 2015 ) and have been used for tasks on code , e.g . in Bhoopchand et al . ( 2016 ) . Such methods have been used to great effect in NLP in e.g . Gulcehre et al . ( 2016 ) and Merity et al . ( 2017 ) . The latter ’ s pointer sentinel mixture model is the direct inspiration for the readout function we use in the Variable Naming task below . Using graphs to represent arbitrary collections of entities and their relationships for processing by deep networks has been widely used ( Johnson , 2017 ; Bansal et al. , 2017 ; Pham et al. , 2018 ; Lu et al. , 2017 ) , but to our knowledge we are the first to use a graph–building strategy for reasoning ( at train and test time ) about an open vocabulary of words . 3 PRELIMINARIES . 3.1 ABSTRACT SYNTAX TREES . An Abstract Syntax Tree ( AST ) is a graph — specifically an ordered tree with labeled nodes — that is a representation of some written computer source code . There is a 1–to–1 relationship between source code and an AST of that source code , modulo comments and whitespace in the written source code . Typically the leaves of an AST correspond to the tokens written in the source code , like variable and method names , while the non–leaf nodes represent syntactic language constructs like function calls or class definitions . The specific node labels and construction rules of ASTs can differ between or within languages . The first step in Figure 1 shows an example . 3.2 GRAPH NEURAL NETWORKS . The term Graph Neural Network ( GNN ) refers to any deep , differentiable model that takes graphs as input . Many GNNs have been presented in the literature , and several nomenclatures have been proposed for describing the computations they perform , in particular in Gilmer et al . ( 2017 ) and Battaglia et al . ( 2018 ) . Here we give a brief recapitulation of supervised learning with GNNs using the Message Passing Neural Network framework from Gilmer et al . ( 2017 ) . A GNN is trained using pairs ( G , y ) whereG = ( V , E ) is a graph defined by its vertices V and edges E , and y is a label . y can be any sort of mathematical object : scalar , vector , another graph , etc . In the most general case , each graph in the dataset can be a directed multigraph , each with a different number of nodes and different connectivity . In each graph , each vertex v ∈ V has associated features xv , and each edge ( v , w ) ∈ E has features evw . A GNN produces a prediction ŷ for the label y of a graph G = ( V , E ) by the following procedure : 1 . A function S is used to initialize a hidden state vector h0v for each vertex v ∈ V as a function of the vertex ’ s features ( e.g. , if the xv are words , S could be a word embedding function ) : h0v = S ( xv ) 2 . For each round t out of T total rounds : ( a ) Each vertex v ∈ V receives the vector mt+1v , which is the sum of “ messages ” from its neighbors , each produced by a function Mt : mt+1v = ∑ w∈neighbors of v Mt ( h t v , h t w , evw ) . ( b ) Each vertex v ∈ V updates its hidden state based on the message it received via a function Ut : ht+1v = Ut ( h t v , m t+1 v ) . 3 . A function R , the “ readout function ” , produces a prediction based on the hidden states generated during the message passing ( usually just those at from time T ) : ŷ = R ( { htv|v ∈ V , t ∈ 1 , ... , T } ) . GNNs differ in how they implement S , Mt , Ut , and R. But all these functions are differentiable and most are parameterized , so the model is trainable via stochastic gradient descent of a loss function on y and ŷ .

Summary: The submission presents an extension to the Allamanis et al ICLR'18 paper on learning from programs as graphs. The core contribution is the idea of introducing extra nodes and edges into the graph that correspond to (potentially rare) subwords used in the analyzed program code. Experiments show that this extended graph leads to better performance on two tasks, compared to a wide range of baseline methods.