code2seq: Generating Sequences from Structured Representations of Code

The ability to generate natural language sequences from source code snippets has a variety of applications such as code summarization , documentation , and retrieval . Sequence-to-sequence ( seq2seq ) models , adopted from neural machine translation ( NMT ) , have achieved state-of-the-art performance on these tasks by treating source code as a sequence of tokens . We present CODE2SEQ : an alternative approach that leverages the syntactic structure of programming languages to better encode source code . Our model represents a code snippet as the set of compositional paths in its abstract syntax tree ( AST ) and uses attention to select the relevant paths while decoding . We demonstrate the effectiveness of our approach for two tasks , two programming languages , and four datasets of up to 16M examples . Our model significantly outperforms previous models that were specifically designed for programming languages , as well as state-of-the-art NMT models . An online demo of our model is available at http : //code2seq.org . Our code , data and trained models are available at http : //github.com/tech-srl/code2seq . 1 INTRODUCTION . Modeling the relation between source code and natural language can be used for automatic code summarization ( Allamanis et al. , 2016 ) , documentation ( Iyer et al. , 2016 ) , retrieval ( Allamanis et al. , 2015b ) , and even generation ( Balog et al. , 2017 ; Rabinovich et al. , 2017 ; Yin and Neubig , 2017 ; Devlin et al. , 2017 ; Murali et al. , 2017 ; Brockschmidt et al. , 2019 ) . In this work , we consider the general problem of generating a natural language sequence from a given snippet of source code . A direct approach is to frame the problem as a machine translation problem , where the source sentence is the sequence of tokens in the code and the target sentence is a corresponding natural language sequence . This approach allows one to apply state-of-the-art neural machine translation ( NMT ) models from the sequence-to-sequence ( seq2seq ) paradigm ( Sutskever et al. , 2014 ; Cho et al. , 2014 ; Bahdanau et al. , 2014 ; Luong et al. , 2015 ; Vaswani et al. , 2017 ) , yielding state-ofthe-art performance on various code captioning and documentation benchmarks ( Iyer et al. , 2016 ; Allamanis et al. , 2016 ; Loyola et al. , 2017 ) despite having extremely long source sequences . We present an alternative approach for encoding source code that leverages the syntactic structure of programming languages : CODE2SEQ . We represent a given code snippet as a set of compositional paths over its abstract syntax tree ( AST ) , where each path is compressed to a fixed-length vector using LSTMs ( Hochreiter and Schmidhuber , 1997 ) . During decoding , CODE2SEQ attends over a different weighted average of the path-vectors to produce each output token , much like NMT models attend over token representations in the source sentence . We show the effectiveness of our code2seq model on two tasks : ( 1 ) code summarization ( Figure 1a ) , where we predict a Java method ’ s name given its body , and ( 2 ) code captioning ( Figure 1b ) , where we predict a natural language sentence that describes a given C # snippet . On both tasks , our CODE2SEQ model outperforms models that were explicitly designed for code , such as the model of Allamanis et al . ( 2016 ) and CodeNN ( Iyer et al. , 2016 ) , as well as TreeLSTMs ( Tai et al. , 2015 ) and state-of-the-art NMT models ( Luong et al. , 2015 ; Vaswani et al. , 2017 ) . To examine the importance of each component of the model , we conduct a thorough ablation study . In particular , we show the importance of structural encoding of code , by showing how our model yields a significant improvement over an ablation that uses only token-level information without syntactic paths . To the best of our knowledge , this is the first work to directly use paths in the abstract syntax tree for end-to-end generation of sequences . 2 REPRESENTING CODE AS AST PATHS . An Abstract Syntax Tree ( AST ) uniquely represents a source code snippet in a given language and grammar . The leaves of the tree are called terminals , and usually refer to user-defined values which represent identifiers and names from the code . The non-leaf nodes are called nonterminals and represent a restricted set of structures in the language , e.g. , loops , expressions , and variable declarations . For example , Figure 2c shows a partial AST for the code snippet of Figure 2a . Names ( such as num ) and types ( such as int ) are represented as values of terminals ; syntactic structures such as variable declaration ( VarDec ) and a do-while loop ( DoStmt ) are represented as nonterminals . Given the AST of a code snippet , we consider all pairwise paths between terminals , and represent them as sequences of terminal and nonterminal nodes . We then use these paths with their terminals ’ values to represent the code snippet itself . For example , consider the two Java methods of Figure 2 . Both of these methods count occurrences of a character in a string . They have exactly the same functionality , although a different implementation , and therefore different surface forms . If these snippets are encoded as sequences of tokens , the recurring patterns that suggest the common method name might be overlooked . However , a structural observation reveals syntactic paths that are common to both methods , and differ only in a single node of a Do-while statement versus a For statement . This example shows the effectiveness of a syntactic encoding of code . Such an encoder can generalize much better to unseen examples because the AST normalizes a lot of the surface form variance . Since our encoding is compositional , the encoder can generalize even if the paths are not identical ( e.g. , a For node in one path and a While in the other ) . Since a code snippet can contain an arbitrary number of such paths , we sample k paths as the representation of the code snippet . To avoid bias , k new paths are sampled afresh in every training iteration . In Section 5 we show that this runtime-sampling provides regularization and improves results compared to sampling the same k paths for each example in advance . Formally , we use C to denote a given snippet of code . Every training iteration , k pairs of terminals are uniformly sampled from within the AST of C. Each pair of terminals ( vi1 , v i li ) implies a single path between them : vi1v i 2 ... v i li . Finally , the input code example is represented as a set of these k random AST paths : { ( v11v 1 2 ... v 1 l1 ) , ... , ( vk1v k 2 ... v k lk ) } , where lj is the length of the jth path . 3 MODEL ARCHITECTURE . Our model follows the standard encoder-decoder architecture for NMT ( Section 3.1 ) , with the significant difference that the encoder does not read the input as a flat sequence of tokens . Instead , the encoder creates a vector representation for each AST path separately ( Section 3.2 ) . The decoder then attends over the encoded AST paths ( rather than the encoded tokens ) while generating the target sequence . Our model is illustrated in Figure 3 . 3.1 ENCODER-DECODER FRAMEWORK . Contemporary NMT models are largely based on an encoder-decoder architecture ( Cho et al. , 2014 ; Sutskever et al. , 2014 ; Luong et al. , 2015 ; Bahdanau et al. , 2014 ) , where the encoder maps an input sequence of tokens x = ( x1 , ... , xn ) to a sequence of continuous representations z = ( z1 , ... , zn ) . Given z , the decoder then generates a sequence of output tokens y = ( y1 , ... , ym ) one token at a time , hence modeling the conditional probability : p ( y1 , ... , ym|x1 , ... , xn ) . At each decoding step , the probability of the next target token depends on the previously generated token , and can therefore be factorized as : p ( y1 , ... , ym|x1 , ... , xn ) = m∏ j=1 p ( yj |y < j , z1 , ... , zn ) In attention-based models , at each time step t in the decoding phase , a context vector ct is computed by attending over the elements in z using the decoding state ht , typically computed by an LSTM . αt = softmax ( htWaz ) ct = n∑ i αtizi The context vector ct and the decoding state ht are then combined to predict the current target token yt . Previous work differs in the way the context vector is computed and in the way it is combined with the current decoding state . A standard approach ( Luong et al. , 2015 ) is to pass ct and ht through a multi-layer perceptron ( MLP ) and then predict the probability of the next token using softmax : p ( yt|y < t , z1 , ... , zn ) = softmax ( Wstanh ( Wc [ ct ; ht ] ) ) 3.2 AST ENCODER . Given a set of AST paths { x1 , ... , xk } , our goal is to create a vector representation zi for each path xi = v i 1v i 2 ... v i li . We represent each path separately using a bi-directional LSTM to encode the path , and sub-token embeddings to capture the compositional nature of the terminals ’ values ( the tokens ) . Path Representation Each AST path is composed of nodes and their child indices from a limited vocabulary of up to 364 symbols . We represent each node using a learned embedding matrixEnodes and then encode the entire sequence using the final states of a bi-directional LSTM : h1 , ... , hl = LSTM ( E nodes v1 , ... , E nodes vl ) encode path ( v1 ... vl ) = [ h → l ; h ← 1 ] Token Representation The first and last node of an AST path are terminals whose values are tokens in the code . Following Allamanis et al . ( 2015a ; 2016 ) , we split code tokens into subtokens ; for example , a token with the value ArrayList will be decomposed into Array and List . This is somewhat analogous to byte-pair encoding in NMT ( Sennrich et al. , 2016 ) , although in the case of programming languages , coding conventions such as camel notation provide us with an explicit partition of each token . We use a learned embedding matrix Esubtokens to represent each subtoken , and then sum the subtoken vectors to represent the full token : encode token ( w ) = ∑ s∈split ( w ) Esubtokenss The LSTM decoder may also predict subtokens at each step ( e.g . when generating method names ) , although the decoder ’ s subtoken embedding matrix will be different . Combined Representation To represent the path x = v1 ... vl , we concatenate the path ’ s representation with the token representations of each terminal node , and apply a fully-connected layer : z = tanh ( Win [ encode path ( v1 ... vl ) ; encode token ( value ( v1 ) ) ; encode token ( value ( vl ) ) ] ) where value is the mapping of a terminal node to its associated value , and Win is a ( 2dpath + 2dtoken ) × dhidden matrix . Decoder Start State To provide the decoder with an initial state , we average the combined representations of all the k paths in the given example : h0 = 1 k k∑ i=1 zi Unlike typical encoder-decoder models , the order of the input random paths is not taken into account . Each path is encoded separately and the combined representations are aggregated with mean pooling to initialize the decoder ’ s state . This represents the given source code as a set of random paths . Attention Finally , the decoder generates the output sequence while attending over all of the combined representations z1 , ... zk , similarly to the way that seq2seq models attend over the source symbols . The attention mechanism is used to dynamically select the distribution over these k combined representations while decoding , just as an NMT model would attend over the encoded source tokens .

Summary: The authors present a method for generating sequences from code. To achieve this, they parse the code and produce a syntax tree. Then, they enumerate paths in the tree along leaf nodes. Each path is encoded via an bidirectional LSTM and a (sub)token-level LSTM decoder with attention over the paths is used to produce the output sequence.  The authors compare their model with other models and show that it outperforms them on two code-to-sequence tasks. An ablation study shows how different components affect the model's performance.