Global Relational Models of Source Code

Models of code can learn distributed representations of a program ’ s syntax and semantics to predict many non-trivial properties of a program . Recent state-ofthe-art models leverage highly structured representations of programs , such as trees , graphs and paths therein ( e.g. , data-flow relations ) , which are precise and abundantly available for code . This provides a strong inductive bias towards semantically meaningful relations , yielding more generalizable representations than classical sequence-based models . Unfortunately , these models primarily rely on graph-based message passing to represent relations in code , which makes them de facto local due to the high cost of message-passing steps , quite in contrast to modern , global sequence-based models , such as the Transformer . In this work , we bridge this divide between global and structured models by introducing two new hybrid model families that are both global and incorporate structural bias : Graph Sandwiches , which wrap traditional ( gated ) graph message-passing layers in sequential message-passing layers ; and Graph Relational Embedding Attention Transformers ( GREAT for short ) , which bias traditional Transformers with relational information from graph edge types . By studying a popular , non-trivial program repair task , variable-misuse identification , we explore the relative merits of traditional and hybrid model families for code representation . Starting with a graph-based model that already improves upon the prior state-of-the-art for this task by 20 % , we show that our proposed hybrid models improve an additional 10–15 % , while training both faster and using fewer parameters . 1 INTRODUCTION . Well-trained models of source code can learn complex properties of a program , such as its implicit type structure ( Hellendoorn et al. , 2018 ) , naming conventions ( Allamanis et al. , 2015 ) , and potential bugs and repairs ( Vasic et al. , 2019 ) . This requires learning to represent a program ’ s latent , semantic properties based on its source . Initial representations of source code relied on sequential models from natural-language processing , such as n-gram language models ( Hindle et al. , 2012 ; Allamanis & Sutton , 2013 ; Hellendoorn & Devanbu , 2017 ) and Recurrent Neural Networks ( RNNs ) ( White et al. , 2015 ) , but these models struggle to capture the complexity of source code . Source code is rich in structured information , such as a program ’ s abstract syntax tree , data and control flow . Allamanis et al . ( 2018b ) proposed to model some of this structure directly , providing a powerful inductive bias towards semantically meaningful relations in the code . Their Gated Graph Neural Network ( GGNN ) model for embedding programs was shown to learn better , more generalizable representations faster than classical RNN-based sequence models . However , the debate on effective modeling of code is far from settled . Graph neural networks typically rely on synchronous message passing , which makes them inherently local , requiring many iterations of message passing to aggregate information from distant parts of the code . However , state-of-the-art graph neural networks for code often use as few as eight message-passing iterations ( Allamanis et al. , 2018b ; Fernandes et al. , 2018 ) , primarily for computational reasons : program graphs can be very large , and training time grows linearly with the number of message passes . This is in contrast to , e.g. , Transformer models ( Vaswani et al. , 2017 ) , which allow program-wide information flow at every step , yet lack the powerful inductive bias from knowing the code ’ s structure . This leads us to a basic research question : is there a fundamental dichotomy between global , unstructured and local , structured models ? Our answer is an emphatic no . Our starting point is the sequence-to-pointer model of Vasic et al . ( 2019 ) , which is state-of-the-art for the task of localizing and repairing a particular type of bug . As a sequence model , their architecture can ( at least potentially ) propagate information globally , but it lacks access to the known semantic structure of code . To this end , we replace the sequence encoder of Vasic et al . ( 2019 ) with a GGNN , yielding a new graph-to-mutlihead-pointer model . Remarkably , this model alone yields a 20 % improvement over the state of the art , though at the cost of being significantly larger than the sequence model . Motivated by this result , we propose two new families of models that efficiently combine longerdistance information , such as the sequence model can represent , with the semantic structural information available to the GGNN . One family , the Graph Sandwich , alternates between message passing and sequential information flow through a chain of nodes within the graph ; the other , the Graph Relational Embedding Attention Transformer ( GREAT ) , generalizes the relative position embeddings in Transformers by Shaw et al . ( 2018 ) to convey structural relations instead . We show that our proposed model families outperform all prior results , as well as our new , already stronger baseline by an additional 10 % each , while training both substantially faster and using fewer parameters . 2 RELATED WORK . Distributed Representation of Programs : There has been increasing interest in modeling source code using machine learning ( Allamanis et al. , 2018a ) . Hindle et al . ( 2012 ) model programs as sequences of tokens and use an n-gram model for predicting code completions . Raychev et al . ( 2015 ) use conditional random fields ( CRFs ) to predict program properties over a set of pairwise program features obtained from the program ’ s dependency graph . Many approaches use neural language models to embed programs as sequences of tokens ( Bhoopchand et al. , 2016 ; White et al. , 2015 ) . Some techniques leverage the ASTs of programs in tree-structured recurrent models ( Piech et al. , 2015 ; Parisotto et al. , 2016 ; Chen et al. , 2018 ) . code2vec ( Alon et al. , 2018 ) and code2seq ( Alon et al. , 2019 ) model programs as a weighted combination of a set of leaf-to-leaf paths in the abstract syntax tree . Finally , Allamanis et al . ( 2018b ) proposed using GGNNs for embedding program graphs consisting of ASTs together with control-flow and data-flow edges . Some recent models of code embed run-time information of programs , e.g. , program traces , besides syntactic information ( Wang et al. , 2018 ) . In this paper , we explore the space of combining sequence-based and graph-based representations of programs , as well as introduce a Transformer-based model with additional program-edge information to learn program representations . Fernandes et al . ( 2018 ) also combine an RNN and a GNN architecture , achieving slight improvements over a GGNN . However , they only consider a single RNN layer inserted at the start ; we include larger and more diverse hybrids , as well as entirely different combinations of structural and sequential features . Neural Program Repair : Automatically generating fixes to repair program bugs is an active field of research with many proposed approaches based on genetic programming , program analysis and formal methods , and machine learning ( Monperrus , 2018 ; Gazzola et al. , 2019 ) . In this paper , we focus on a specific class of repair task called VarMisuse as proposed by Allamanis et al . ( 2018b ) , who use a graph-based embedding of programs to predict the most likely variable at each variable-use location and generate a repair prediction whenever the predicted variable is different from the one present , using an enumerative approach . Vasic et al . ( 2019 ) improved this approach by jointly predicting both the bug and repair locations using a two-headed pointer mechanism . Our multi-headed pointer graph , graph-sandwich and GREAT models significantly outperform these approaches . 3 SEMI-STRUCTURED MODELS OF SOURCE CODE . Models of code have so far either been structured ( GNNs ) or unstructured ( RNNs , Transformers ) . Considering the graph-based models ’ substantially superior performance compared to RNNs despite their locality limitations , we may ask : to what extent could global information help GNNs , and to what extent could structural features help sequence-based models ? 3.1 MODELS . We address the questions of combining local and global information with two families of models . Graph-sandwich Models Let T = 〈t1 , t2 , · · · , tn〉 denote a program ’ s token sequence and G = ( V , E ) denote the corresponding program graph , where V is the set of node vertices and E is the list of edge sets for different edge types . In both graph and sequence based models , the nodes v maintain a state vector h ( v ) that is initialized with initial node embedding x ( v ) ∈ RD . In a GGNN layer , messages of type k are sent from each node v ∈ V to its neighbors computed as m ( v ) k = LinearLayerk ( h ( v ) ) . After the message passing step , the set of messages at each node are aggregated as m ( v ) = Σek ( u , v ) ∈E m ( u ) k . Finally , the state vector of a node v is updated as h ( v ) new = GRU ( m ( v ) , h ( v ) ) . In an RNN layer , the state of a node v ( corresponding to a terminal leaf token ti in T ) is updated as h ( v ) new = f ( tv , h ( ti−1 ) ) . A Transformer compute tv → qt , kt , vt , corresponding to a query , key and value for each token.1 Each token then computes its attention to all other tokens using eij = ( qikj > ) / √ N ,2 which can be soft-maxed to yield attention probabilities aij = exp ( eij ) /Σ exp ( ei , : ) . Our first class of models follows from the observation that T ⊆ V ; i.e. , the source code tokens used by sequence models , like RNNs , are by definition also nodes in the program graph , so GGNNs update their state with every message pass . We can thus envision a combined model that uses each of these as a building block ; for instance , assuming initial node features x ( v ) ∈ RD , the formula [ RNN , GGNN ( 3 ) , RNN ] describes a model in which we first run an RNN on all tokens ∈ T ( in lexical ordering ) , then , using these as initial states for v ∈ T while using the default node-type embeddings for all other nodes , run three message passing steps using a GGNN , after which we again gather the nodes corresponding to T and update their state with an additional RNN pass . The resulting family of models alternates GGNN-style message passing operations and layers of sequence-based models . By varying the number and size of sequential layers and blocks of GGNNstyle message passing , this variant particularly provides insight into the first question above ( how can global information help GNNs ? ) , by showing the transition in performance potential of models that increasingly incorporate sequential features . We refer to this class of models as sandwich models . Graph Relational Embedding Attention Transformer The above family of models still rely on explicit message passing for their structural bias , thereby only indirectly combining structural and global information to the model . We may wish to instead directly encode structural bias into a sequence-based model , which requires a relaxation of the ‘ hard ’ inductive bias from the GGNN . For Transformer-based architectures , Shaw et al . ( 2018 ) show that relational features can be incorporated directly into the attention function by changing the attention computation to:3 eij = ( qi + bij ) kj > / √ N where qi and kj correspond to the query and key vectors as described above , bij is an added bias term for the specific attention weight between tokens i and j , and N is the per-head attention dimension . In our case , we compute bij = W > e e + be , where We ∈ RN , be ∈ R , and e ∈ RN is an embedding of the edge type connecting nodes i and j , if any . If multiple edge types are present between two nodes , the resulting biases are simply added . We name this model GREAT , for Graph Relational Embedding Attention Transformer .

Summary: The paper proposes improvements on existing probabilistic models for code that predicts and repairs variable misuses. This is a variant of the task, proposed by Vasic et al. The task takes a dataset of python functions, introduces errors in these functions and makes a classifier that would identify what errors were introduced and effectively reconstruct the original code.