Neural Networks for Modeling Source Code Edits

1 INTRODUCTION . Source code repositories are in a state of continuous change , as new features are implemented , bugs are fixed , and refactorings are made . At any given time , a developer will approach a code base and make changes with one or more intents in mind . The main question in this work is how to observe a past sequence of edits and then predict what future edits will be made . This is an important problem because a core challenge in building better developer tools is understanding the intent behind a developer ’ s actions . It is also an interesting research challenge , because edit patterns can not be understood only in terms of the content of the edits ( what was inserted or deleted ) or the result of the edit ( the state of the code after applying the edit ) . An edit needs to be understood in terms of the relationship of the change to the state where it was made , and accurately modeling a sequence of edits requires learning a representation of the past edits that allows the model to generalize the pattern and predict future edits . As an example , consider Figure 1 . We show two possible edit sequences , denoted as History A and History B . Both sequences have the same state of code after two edits ( State 2 ) , but History A is in the process of adding an extra argument to the foo function , and History B is in the process of removing the second argument from the foo function . Based on observing the initial state ( State 0 ) and the sequence of edits ( Edits 1 & 2 ) , we would like our models to be able to predict Edit 3 . In the case of History A , the specific value to insert is ambiguous , but the fact that there is an insertion at that location should come with reasonably high confidence . The main challenge in modeling sequences of edits is in how to develop good representations that will both capture the required information about intent , as well as scale gracefully with the length of the sequence . We consider two representations of edits that we call explicit and implicit representations . The explicit representation explicitly instantiates the state resulting from each edit in the sequence , while the implicit representation instantiates a full initial state and then subsequent edits in a more compact diff-like representation . On the explicit representation we consider a hierarchical recurrent pointer network model as a strong but computationally expensive baseline . On the implicit representation , we consider a vanilla sequence-to-sequence model , and a two-headed attention-based model with a pointer network head for producing edit positions and a content head for producing edit contents . These models demonstrate tradeoffs that arise from different problem formulations and inform design decisions for future models of edit sequences . On carefully designed synthetic data and a large-scale dataset of fine-grained edits to Python source code , we evaluate the scalability and accuracy of the models on their ability to observe a sequence of past edits and then predict future edits . We show that the two-headed attention model is particularly well-suited to achieving high accuracy , well-calibrated confidence , and good scalability on the real data , which makes us optimistic about future prospects of developer tools that learn to extract intent from developers as they make edits to large , real code bases . In total , this work formalizes the problem of learning from and predicting edit sequences , provides an initial exploration of model space , and demonstrates applicability to the real-world problem of learning from edits that developers make to source code . 2 PROBLEM FORMULATION . Implicit vs . Explicit Data Representation . The first question is how to represent edit sequence data . We define two data formats having different tradeoffs . The explicit format ( Figure 2 ( a ) ) represents an edit sequence as a sequence of sequences of tokens in a 2D grid . The inner sequences index over tokens in a file , and the outer sequence indexes over time . The task is to consume the first t rows and predict the position and content of the edit made at time t. The implicit format ( Figure 2 ( b ) ) represents the initial state as a sequence of tokens and the edits as a sequence of ( position , content ) pairs . The explicit representation is conceptually simple and is relatively easy to build accurate models on top of . The downside is that processing full sequences at each timestep is expensive and leads to poor scaling . Conversely , it is easy to build scalable models on the implicit representation , but it is challenging to recognize more complicated patterns of edits and generalize well . We show experimentally in Section 6 that baseline explicit models do not scale well to large datasets of long sequences and baseline implicit models are not able to generalize well on more challenging edit sequences . Ideally we would like a model that operates on the implicit data format but generalizes well . In Section 4 we develop such a model , and in Section 6 we show that it achieves the best of both worlds in terms of scalability and generalization ability . Notation . A state s is a sequence of discrete tokens s = ( s0 , . . . , sM ) with sm ∈ V , and V is a given vocabulary . An edit e ( t ) = ( p ( t ) , c ( t ) ) is a pair of position p ( t ) ∈ N and content c ( t ) ∈ { DELETE } ∪ V . An edit sequence ( which we also call an instance ) is an initial state s ( 0 ) along with a sequence of edits e = ( e ( 1 ) , . . . , e ( T ) ) . We can also refer to the implied sequence of states ( s ( 0 ) , . . . , s ( T ) ) , where s ( t ) is the state that results from applying e ( t ) to s ( t−1 ) . There are two representations for position . In the explicit representation , p ( t ) = p ( t ) e is the index of the token in s ( t−1 ) that c ( t ) should be inserted after . If c ( t ) is a DELETE symbol , then p ( t ) e is the index of the token in s ( t−1 ) that should be deleted . In the implicit representation , we assign indices 0 , . . . , M to tokens in the initial state and indices M + 1 , . . . , M + T to the edits that are made ; i.e. , if a token is inserted by e ( t ) then it is given an index of M + t. The position p ( t ) = p ( t ) i is the index of the token that the edit should happen after . For example , see Figure 2 ( b ) . The second edit is to insert a B after the B that was inserted in the first edit . The index of the first B is 6 , and the position p ( 7 ) i = 6 , denoting that the second B should be inserted after the first B . Problem Statement . We can now state the learning problems . The goal in the explicit problem is to learn a model that maximizes the likelihood of e ( t ) given s ( 0 ) , . . . , s ( t−1 ) and the implicit problem is to learn a model that maximizes the likelihood of e ( t ) given s ( 0 ) , e ( 1 ) , . . . , e ( t−1 ) for all t . 3 BASELINE MODELS . Baseline Explicit Model . The baseline explicit model is a two-level Long Short-Term Memory ( LSTM ) neural network ( Hochreiter & Schmidhuber , 1997 ) similar to hierarchical RNN models like in Serban et al . ( 2016 ) . We refer to the hidden vector for the mth token in timestep t as h ( t , m ) ∈ RD . In the simplest version of the model , the first level LSTM encodes each state sequence s ( t ) in parallel and produces hidden states h ( t,0 ) , . . . , h ( t , M ) . The second level LSTM takes as input the sequence of h ( 0 , M ) , . . . , h ( T , M ) and produces hidden state h̃ ( t ) ∈ RD and output state o ( t ) ∈ RD for each time step . We predict the distribution over c ( t+1 ) as softmax ( W ( out ) o ( t ) ) , where W ( out ) ∈ R ( |V|+1 ) ×D . To predict p ( t+1 ) e , we use a pointer network construction ( Vinyals et al. , 2015 ) where α ( t , m ) = 〈 o ( t ) , h ( t , m ) 〉 and P ( p ( t+1 ) e = m ) ∝ expα ( t , m ) . A useful elaboration is to provide a second path of context for the first level LSTM by concatenating the previous value of the token in the same position as an input . That is , we provide ( s ( t−1 ) m , s ( t ) m ) as input at position ( t , m ) , letting s ( −1 ) m be a special padding symbol . See Figure 3 ( a ) for an illustration . Baseline Implicit Model . The natural application of the sequence-to-sequence framework is to consume the initial state s ( 0 ) in the encoder and produce the sequence of ( p ( t ) i , c ( t ) ) pairs in the decoder . The encoder is a standard LSTM . The decoder is slightly non-standard because each action is a pair . To deal with pairs as inputs , we concatenate an embedding of p ( t ) i with an embedding of c ( t ) . To produce pairs as outputs , we predict position and then content given position . Formally , for position inputs , we embed each integer m ∈ { 0 , . . . , M + T − 1 } by taking the mth column of a learned matrix W ( p-in ) ∈ RD× ( M+T ) . These are concatenated with content embeddings to produce inputs to the decoder , yielding hidden state h ( t ) at step t of the decoder . To predict position , we use a matrix W ( p-out ) ∈ R ( M+T ) ×D and define the distribution over position p ( t+1 ) i as softmax ( W ( p-out ) h ( t ) ) . Content is predicted by concatenating h ( t ) with W ( p-in ) : ,p ( t+1 ) , the embedding of the ground truth position , to get h̃ ( t ) , and then the predicted distribution over content c ( t+1 ) is softmax ( W ( c-out ) h̃ ( t ) ) where W ( c-out ) ∈ R ( |V|+1 ) ×D maps hidden states to content logits .

Summary: The paper provides good baselines for predicting edits in a text (evaluated on source code) learned from a history of changes. This is an interesting problem that has not beed systematically studied in literature, with the exception of several sporadic works from the software engineering community. Fully predicting the code text writing process as opposed to the code itself is an interesting task with possible big impact, IF the accuracy of this edit model manages to significantly outperform simple left-to-right code text prediction techniques.