Title: LEARNING TO REPRESENT EDITS

Abstract: We introduce the problem of learning distributed representations of edits. By combining a "neural editor" with an "edit encoder", our models learn to represent the salient information of an edit and can be used to apply edits to new inputs. We experiment on natural language and source code edit data. Our evaluation yields promising results that suggest that our neural network models learn to capture the structure and semantics of edits. We hope that this interesting task and data source will inspire other researchers to work further on this problem.

1. INTRODUCTION:

One great advantage of electronic storage of documents is the ease with which we can edit them, and edits are performed in a wide variety of contents. For example, right before a conference deadline, papers worldwide are finalized and polished, often involving common fixes for grammar, clarity and style. Would it be possible to automatically extract rules from these common edits? Similarly, program source code is constantly changed to implement new features, follow best practices and fix bugs. With the widespread deployment of (implicit) version control systems, these edits are quickly archived, creating a major data stream that we can learn from.

In this work, we study the problem of learning distributed representations of edits. We only look at small edits with simple semantics that are more likely to appear often and do not consider larger edits; i.e., we consider "add definite articles" rather than "rewrite act 2, scene 3." Concretely, we focus on two questions: i) Can we group semantically equivalent edits together, so that we can automatically recognize common edit patterns? ii) Can we automatically transfer edits from one context to another? A solution to the first question would yield a practical tool for copy editors and programmers alike, automatically identifying the most common changes. By leveraging tools from program synthesis, such groups of edits could be turned into interpretable rules and scripts (Rolim et al., 2017) . When there is no simple hard rule explaining how to apply an edit, an answer to the second question would be of great use, e.g., to automatically rewrite natural language following some stylistic rule.

We propose to handle edit data in an autoencoder-style framework, in which an "edit encoder" f ∆ is trained to compute a representation of an edit x -→ x + , and a "neural editor" α is trained to construct x + from the edit representation and x -. This framework ensures that the edit representation is semantically meaningful, and a sufficiently strong neural editor allows this representation to not be specific to the changed element. We experiment with various neural architectures that can learn to represent and apply edits and hope to direct the attention of the research community to this new and interesting data source, leading to better datasets and stronger models.

Briefly, the contributions of our paper are: (a) in Sect. 2, we present a new and important machine learning task on learning representations of edits (b) we present a family of x-: var greenList = trivia==null ? null : trivia.Select(t=> t.UnderlyingNode);

x+: var greenList = trivia?.Select(t=> t.UnderlyingNode); Edit 1

x -: me = ((ue!=null) ? ue.Operand : null) as MemberExpression;

x + : me = ue?.Operand as MemberExpression; Edit 2 Edit Representation f∆(x-, x+) ∈ R n Neural Editor α(x -, f∆(x-, x+))

Figure 1 : Given an edit (Edit 1) of x -to x + , f ∆ computes an edit representation vector. Using that representation vector the neural editor α applies the same edit to a new x -. The code snippets shown here are real code change examples from the roslyn open-source compiler project. models that capture the structure of edits and compute efficient representations in Sect. 3 (c) we create a new source code edit dataset, and release the data extraction code at https://github.com/Microsoft/msrc-dpu-learning-to-represent-edits and the data at http://www.cs.cmu.edu/ ˜pengchey/githubedits.zip. (d) we perform a set of experiments on the learned edit representations in Sect. 4 for natural language text and source code and present promising empirical evidence that our models succeed in capturing the semantics of edits.

2. TASK:

In this work, we are interested in learning to represent and apply edits on discrete sequential or structured data, such as text or source code parse trees1 . Figure 1 gives a graphical overview of the task, described precisely below.

Edit Representation Given a dataset of edits {x

(i) -→ x (i) + } N i=1 , where x (i)

-is the original version of some object and x (i) + its edited form (see upper half of Figure 1 for an example), our goal is to learn a representation function f ∆ that maps an edit operation x -→ x + to a real-valued edit representation f ∆ (x -, x + ) ∈ R n . A desired quality of f ∆ is for the computed edit representations to have the property that semantically similar edits have nearby representations in R n . Having distributed representations also allows other interesting downstream tasks, e.g., unsupervised clustering and visualization of similar edits from large-scale data (e.g. the GitHub commit stream), which would be useful for developing human-assistance toolkits for discovering and extracting emerging edit patterns (e.g. new bug fixes or emerging "best practices" of coding).

Neural Editor Given an edit representation function f ∆ , we want to learn to apply edits in a new context. This can be achieved by learning a neural editor α that accepts an edit representation f ∆ (x -, x + ) and a new input x -and generates x + .2 This is illustrated in the lower half of Figure 1 .

3. MODEL:

We cast the edit representation problem as an autoencoding task, where we aim to minimize the reconstruction error of α for the edited version x + given the edit representation f ∆ (x -, x + ) and the original version x -. By limiting the capacity of f ∆ 's output and allowing the model to freely use information about x -, we are introducing a "bottleneck" that forces the overall framework to not simply treat f ∆ (x -, x + ) as an encoder of x + . The main difference from traditional autoencoders is that in our setup, an optimal solution requires to re-use as much information as possible from x - to make the most of the capacity of f ∆ . Formally, given a probabilistic editor function P α such as a neural network and a dataset {x

(i) -→ x (i) + } N i=1

, we seek to minimize the negative likelihood loss

L = - 1 N i log P α (x + | x -, f ∆ (x -, x + )).

Note that this loss function can be interpreted in two ways: (1) as a conditional autoencoder that encodes the salient information of an edit, given x -and (2) as an encoder-decoder model that encodes x -and decodes x + conditioned on the edit representation f ∆ (x -, x + ). In the rest of this section, we discuss our methods to model P α and f ∆ as neural networks.

3.1. NEURAL EDITOR:

As discussed above, α should use as much information as possible from x -, and hence, an encoderdecoder architecture with the ability to copy from the input is most appropriate. As we are primarily interested in edits on text and source code in this work, we explored two architectures: a sequenceto-sequence model for text, and a graph-to-tree model for source code, whose known semantics we can leverage both on the encoder as well as on the decoder side. Other classes of edits, for example, image manipulation, would most likely be better served by convolutional neural models.

Sequence-to-Sequence Neural Editor First, we consider a standard sequence-to-sequence model with attention (over the tokens of x -). The architecture of our sequence-to-sequence model is similar to that of Luong et al. (2015) , with the difference that we use a bidirectional LSTM in the encoder and a token-level copying mechanism (Vinyals et al., 2015) that directly copies tokens into the decoded sequence. Whereas in standard sequence-to-sequence models the decoder is initialized with the representation computed by the encoder, we initialize it with the concatenation of encoder output and the edit representation. We also feed the edit representation as input to the decoder LSTM at each decoding time step. This allows the LSTM decoder to take the edit representation into consideration while generating the output sequence.

Graph-to-Tree Neural Editor Our second model aims to take advantage of the additional structure of x -and x + . To achieve this, we combine a graph-based encoder with a tree-based decoder. We use T (x) to denote a tree representation of an element, e.g., the abstract syntax tree (AST) of a fragment of source code. We extend T (x) into a graph form G(x) by encoding additional relationships (e.g., the "next token" relationship between terminal nodes, etc.) (see Figure 2 (a)). To encode the elements of G(x -) into vector representations, we use a gated graph neural network (GGNN) (Li et al., 2015) . Similarly to recurrent neural networks for sequences (such as biRNNs), GGNNs compute a representation for each node in the graph, which can be used in the attention mechanisms of a decoder. Additionally, we use them to obtain a representation of the full input x -, by computing their weighted average following the strategy of Gilmer et al. (2017) (i.e., computing a score for each node, normalizing scores with a softmax, and using the resulting values as weights).

Our tree decoder follows the semantic parsing model of Yin & Neubig (2018) , which sequentially generate a tree T (x + ) as a series of expansion actions a 1 . . . a N . The probability of taking an action is modeled as p(a t | a <t , s), where s is the input (a sequence of words in the original semantic parsing setting) and a <t is the partial tree that has been generated so far. The model of Yin & Neubig (2018) mainly uses two types of actions: EXPANDR expands the current non-terminal using a grammar rule, and GENTERM generates a terminal token from a vocabulary or copies a token from s3 . The dependence on the partial tree a <t is modeled by an LSTM cell which is used to maintain state throughout the generation procedure. Additionally, the LSTM receives the decoder state used to pick the action at the parent node as an additional input ("parent-feeding"). This process illustrated in Figure 2(b) .

AssignStmt FieldAccess v . F = Expr x + x AssignStmt u = Expr x + x ↔ v u - . ∅ - F ∅ = = = = x x = + + = x x (a) (b)

We extend this model to our setting by replacing the input sequence s by x -; concretely, we condition the decoder on the graph-level representation computed for G(x -). Additionally, we use the change representation f ∆ (•) as an additional input to the LSTM initial state and at every decoding step.

Based on the observation that edits to source code often manipulate the syntax tree by moving expressions around (e.g. by nesting statements in a conditional, or renaming a function while keeping its arguments), we extend the decoding model of Yin & Neubig (2018) by adding a facility to copy entire subtrees from the input. For this, we add a decoder action TREECP. This action is similar to standard copying mechanism known from pointer networks (Vinyals et al., 2015) , but instead of copying only a single token, it copies the whole subtree pointed to.

However, adding the TREECP action means that there are many correct generation sequences for a target tree. This problem appears in token-copying as well, but can be easily circumvented by marginalizing over all correct choices at each generation step (by normalizing the probability distribution over allowed actions to sum up those that have the same effect). In the subtree-copying setting, the lengths of action sequences representing different choices may differ. In our implementation we handle this problem during training by simpling picking the generation sequence that greedily selecting TREECP actions.

3.2. EDIT REPRESENTATION:

To compute a useful edit representation, a model needs to focus on the differences between x -and x + . A risk in our framework is that f ∆ degenerates into an encoder for x + , turning α into a decoder.

To avoid this, we need to follow the standard autoencoder trick, i.e. it is important to limit the capacity of the result of f ∆ by generating the edit representation in a low-dimensional space R N . This acts as a bottleneck and encodes only the information that is needed to reconstruct x + from x -. We again experimented with both sequence-based and graph-based representations of edits.

Sequence Encoding of Edits Given x -(resp. x + ) as sequence of tokens t ), we can use a standard (deterministic) diffing algorithm to compute an alignment of tokens in the two sequences. We then use extra symbols ∅ for padding, + for additions,for deletions, ↔ for replacements, and = for unchanged tokens to generate a single sequence representing both x -and x + . This is illustrated in Figure 3(a) . By embedding the three entries in each element of the sequence separately and concatenating their representation, they can be fed into a standard sequence encoder whose final state is our desired edit representation. In this work, we use a biLSTM.

Graph Encoding of Edits:

As in the graph-to-tree neural editor, we represent x -and x + as trees T (x -) and T (x + ). We combine these trees into a graph representation G(x -→ x + ) by merging both trees into one graph, using "Removed", "Added" and "Replaced" edges. To connect the two trees, we compute the same alignment as in the sequence case, connecting leaves that are the same and each replaced leaf to its replacement. We also propagate this information up in the trees, i.e., two inner nodes are connected by "=" edges if all their descendants are connected by "=" edges. This is illustrated in Figure 3(b) . Finally, we also use the same "+" / "-" / "↔" / "=" tags for the initial node representation, computing it as the concatenation of the string label (i.e. token or nonterminal name) and the embedding of the tag. To obtain an edit representation, we use a GGNN unrolled for a fixed number of timesteps and again use the weighted averaging strategy of Gilmer et al. (2017) .

4. EVALUATION:

Evaluating an unsupervised representation learning method is challenging, especially for a newly defined task. Here, we aim to evaluate the quality of the learned edit representations with a series of qualitative and quantitative metrics on natural language and source code.

4.1. DATASETS AND CONFIGURATION:

Natural Language Edits We use the WikiAtomicEdits (Faruqui et al., 2018) dataset of pairs of short edits on Wikipedia articles. We sampled 1040K edits from the English insertion portion of the dataset and split the samples into 1000K/20K/20K train-valid-test sets.

Source Code Edits To obtain a dataset for source code, we clone a set of 54 C# projects on GitHub and collected a GitHubEdits dataset (see Appendix A for more information). We selected all changes in the projects that are no more than 3 lines long and whose surrounding 3 lines of code before and after the edited lines have not been changed, ensuring that the edits are separate and short. We then parsed the two versions of the source code and take as x -and x + the code that belongs to the top-most AST node that contains the edited lines. Finally, we remove trivial changes such variable renaming, changes within comments or formatting changes. Overall, this yields 111 724 edit samples. For each edit we run a simple C# analysis to detect all variables and normalize variable names such that each unique variable within x -and x + has a unique normalized name V0, V1, etc. This step is necessary to avoid the sparsity of data induced by the variety of different identifier naming schemes. We split the dataset into 91,372 / 10,176 / 10,176 samples as train/valid/test sets.

Additionally, we introduce a labeled dataset of source code edits by using C# "fixers". Fixers are small tools built on top of the C# compiler, used to perform common refactoring and modernization tasks (e.g., using new syntactic sugar). We selected 16 of these fixers and ran them on 6 C# projects to generate a small C#Fixers dataset of 2,878 edit pairs with known semantics. We present descriptions and examples of each fixer in Appendix A.

Configuration Throughout the evaluation we use a fixed size of 512 for edit representations. The size of word embeddings and hidden states of encoding LSTMs is 128. The dimensionality of the decoding LSTM is set to 256. Details of model configuration can be found in Sect. A.

When generating the target x + , our neural editor model can optionally take as input the context of the original input x -(e.g., the preceding and succeeding code segments surrounding x -), whose information could be useful for predicting x + . For example, in source code edits the updated code snippet x + may reuse variables defined in the preceding snippet. In our code experiments, we use a standard bidirectional LSTM network to encode the tokenized 3 lines of code before and after x - as context. The encoded context is used to initialize the decoder, and as an additional source for the pointer network to copy tokens from.

4.2. QUALITY OF EDIT REPRESENTATIONS:

First, we study the ability of our models to encode edits in a semantically meaningful way. Visualizing Edits on Fixers Data In a first experiment, we train our sequential neural editor model on our GitHubEdits data and then compute representations for the edits generated by the C# fixers. A t-SNE visualization (Maaten & Hinton, 2008) of the encodings is shown in Figure 4 . For this visualization, we randomly selected 100 examples from the edits of each fixer (if that fixer has more than 100 samples) and discarded fixer categories with less than 40 examples. Readers are referred to Appendix A for detailed descriptions of each fixer category. We find that our model produces dense clusters for simple or distinctive code edits, e.g. fixer RCS1089 (using the ++ or --unary operators instead of a binary operator (e.g., i = i + 1 → i++), and fixer CA2007 (adding .ConfigureAwait(false) for await statements). We also analyzed cases where (1) the edit examples from the same fixer are scattered, or (2) the clusters of different fixers overlap with each other. For example, the fixer RCS1077 covers 12 different aspects of optimizing LINQ method calls (e.g., type casting, counting, etc.), and hence its edits are scattered. On the other hand, fixers RCS1146 and RCS1206 yield overlapping clusters, as both fixers change code to use the ?. operator. Fixers RCS1207 (change a lambda to a method group, e.g. foo(x=>bar(x)) → foo(bar)) and RCS1021 (simplify lambda expressions, e.g. foo(x=>{return 4;}) → foo(x=>4)) are similar, as both inline lambda expressions in two different ways. Analysis yields that the representation is highly dependent on surface tokens. For instance, IDE004 (removing redundant type casts, e.g. (int)2 → 2) and RCS1207 (removing explicit argument lists) yield overlapping clusters, as both involve deleting identifiers wrapped by parentheses.

Human Evaluation on Encoding Natural Language WikiAtomicEdits In a second experiment, we test how well neighborhoods in edit representation space correspond to semantic similarity. We computed the five nearest neighbors of 200 randomly sampled seed edits from our training set, using both our trained sequence-to-sequence editing model with sequential edit encoder, as well as a simple bag-of-words baseline based on TF-IDF scores. We then rated the quality of the retrieved neighbors on a scale of 0 ("unrelated edit"), 1 ("similar edit") and 2 ("semantically or syntactically same edit"). Details of the annotation schema is included in Sect. E. We show the (normalized) discounted cumulative gain (DCG, Manning et al. (2008) ) for the two models at the top of Tab. 1 (higher is better). The relevance scores indicate that our neural model clearly outperforms the simplistic baseline. Tab. 1 also presents two example edits with their nearest neighbors. Example 1 shows that the neural edit models succeeded in representing syntactically and semantically similar edits, while the bag-of-words baseline relies purely on surface token overlap. Interestingly, we also observed that the edit representations learned by the neural editing model on WikiAtomicEdits are somewhat sensitive to position, i.e. the position of the inserted tokens in both the seed edit and the nearest neighbors is similar. This is illustrated in Example 2, where the second ("senegalese striker") and the third ("republican incumbent") nearest neighbors returned by the neural model have similar editing positions as the seed edit, while they are semantically diverse.

4.3. EDIT ENCODER PERFORMANCE:

To evaluate the performance of our two edit encoders discussed in Sect. 3.2 and disentangle it from the choice of neural editor, we train various combinations of our neural editor model and manually evaluate the quality of the edit representation. More specifically, we trained our neural editor models on GitHubEdits and randomly sampled 200 seed edits and computed their 3 nearest neighbors using each end-to-end model. We then rated the resulting groups using the same 0-2 scale as above. The resulting relevance scores are shown in Tab. 2.

Bag of Words Model:

Seq2Seq -Seq Edit Encoder DCG/NDCG@5 9.3 / 67.3% 13.5 / 90.3% DCG@5 (by edit size) 1: 14.7 2-3: 10.8 >3: 5.4 1: 16.2 2-3: 12.9 >3: 12.4 Example 1 daniel james nava ( born february 22 , 1983 ) is an american professional baseball outfielder nava is only the fourth player in mlb history to hit a grand slam in his first major league at bat and the second to do it on the first pitch . NN-1 he batted .302 with 73 steals , and received a september call -up to the major leagues as an outfielder .

arthur ray briles ( born december 3 , 1955 ) is a former american football coach and his most recent head coaching position was at baylor university , a position he held from the 2008 season through the 2015 season . NN-2 he played as an outfielder for the hanshin tigers .

jonathan david disalvatore ( born march 30 , 1981 ) is a professional ice hockey he was selected by the san jose sharks in the 4th round ( 104th overall ) of the 2000 nhl entry draft . NN-3 in 2012 , his senior at oak mountain , dahl had a .412 batting average , 34 runs batted in ( rbis ) , and 18 stolen bases as an outfielder .

professor paul talalay ( born march 31 , 1923 ) is the john jacob abel distinguished service professor of pharmacology and director of the laboratory for molecular sciences at johns hopkins school of medicine in baltimore . Example 2 she , along with her follow artist carolyn mase studied with impressionist landscape painter john henry twachtman at the art students league of new york . NN-1 his brother was draughtsman william daniell and his uncle was landscape painter thomas daniell .

the first painting was a portrait of a young girl , emerantia van beresteyn , the sister of the landscape painter nicolaes van beresteyn , the later founder of half of this hofje . NN-2 william james linton ( december 7 , 1812 -december 29 , 1897 ) was an english -born american wood engraver , landscape painter , political reformer and author of memoirs , novels , poetry and non-fiction .

he was the club 's top scorer with 22 goals in all competitions , one more than senegalese striker lamine diarra , who left the club at the end of the season .

NN-3 early on , hopper modeled his style after chase and french impressionist masters douard manet and edgar degas .

caforio " aggressively attacked " his opponent , republican incumbent steve knight , for his delayed response to the leak . Comparing the sequential edit encoders trained with Seq2Seq and Graph2Tree editors, we found that the edit encoder trained with the Graph2Tree objective performs better. We hypothesize that this is because the Graph2Tree editor better captures structural-level information about an edit. For instance, Example 1 in Tab. 3 removes explicit type casting. The Seq2Seq editor has difficulty distinguishing this type of edit, confusing it with changes of lambda expressions to method groups (1st and 2nd nearest neighbors) since both two types of edits involve removing paired parentheses.

Surprisingly, we found that the graph-based edit encoder does not outperform the sequence-based encoder. However, we observe that the graph edit encoder in many cases tends to better capture high-level and abstract structural edit patterns. Example 2 in Tab. 3 showcases a seed edit that swaps two consecutive declarations, which corresponds to swapping the intermediate Expression nodes representing each statement on the underlying AST. In this case, the graph edit encoder is capable of grouping semantically similar edits, while it seems to be more difficult for the sequential encoder 

Seq2Seq -Seq Edit Encoder:

x -: V0.Debug(() => LITERAL);

x + : V0.Debug(LITERAL);

x -: V0.Debug(() => LITERAL);

x + : V0.Debug(LITERAL);

x -: V0.WriteCompressedInteger((uint)V1);

x + : V0.WriteCompressedInteger(V1);

Graph2Tree -Seq Edit Encoder x -: V0.WriteCompressedInteger((uint)V1);

x + : V0.WriteCompressedInteger(V1);

x -: V0.WriteCompressedInteger((uint)V1);

x + : V0.WriteCompressedInteger(V1);

x -: V0.WriteCompressedInteger((uint)V1);

x + : V0.WriteCompressedInteger(V1);

Graph2Tree -Graph Edit Encoder:

x -: V0.UpdateLastRead(this.V1);

x + : V0.UpdateLastRead(V1);

x -: V0.UpdateLastWrite(this.V1);

x + : V0.UpdateLastWrite(V1);

x -: V0.Append(this.V1);

x + : V0.Append(V1); x -: int V0 = V1; int V2 = V3;

x + : int V2 = V3; int V0 = V1;

Graph2Tree -Graph Edit Encoder x -: RetryConfig V0; string V1;

x + : string V1; RetryConfig V0;

x -: int V0 = V1; int V2 = V3;

x + : int V2 = V3; int V0 = V1;

x -: double V0= -1; double V1= -1;

x + : double V1= -1; double V0= -1; encoder to capture the edit pattern. On the other hand, we found that the graph edit encoder often fails to capture simpler, lexical level edits (e.g., Example 1). This might suggest that terminal node information is not effectively propagated, an interesting issue worth future investigation.

4.4. PRECISION OF NEURAL EDITORS:

Finally, we evaluate the performance of our end-to-end system by predicting the edited input x + given x -and the edit representation. We are interested in answering two research questions: First, how well can our neural editors generate x + given the gold-standard edit representation f ∆ (x -, x + )? Second, and perhaps more interestingly, can we use the representation of a similar edit f ∆ (x -, x + ) to generate x + by applying that edit to x -(i.e. x+ = α(x -, f ∆ (x -, x + )))?

To answer the first question, we trained our neural editor models on the WikiAtomicEdits and the GitHubEdits dataset, and evaluate the performance of encoding and applying edits on test sets. For completeness, we also evaluated the performance of our neural editor models with a simple "Bag-of-Edits" edit encoding scheme, where f ∆ (x -, x + ) is modeled as the concatenation of two vectors, each representing the sum of the embeddings of added and deleted tokens in the edit, respectively. This edit encoding method is reminiscent of the model used in Guu et al. (2017) for solving a different task of language modeling by marginalizing over latent edits, which we will elaborate in Sect. 5. Tab. 4 lists the evaluation results. With our proposed sequence-and graph-based edit encoders, our neural editor models achieve reasonable end-to-end performance, surpassing systems using bag-of-edits representations. This is because many edits are context-sensitive and position-sensitive, requiring edit representation models that go beyond the bag-of-edits scheme to capture those effects (more analysis is included in Appendix B). Interestingly, on the GitHubEdits dataset, we find that the Seq2Seq editor with sequential edit encoder registers the best performance. However, it should be noted that in this set of experiments, we encode the gold-standard edit f ∆ (x -, x + ) to predict x + . As we will show later, better performance with the gold-standard edit does not necessarily imply better (more generalizable) edit representation. Nevertheless, we hypothesize that the higher accuracy of the Seq2Seq edit is due to the fact that a significant proportion of edits in this dataset is small and primarily syntactically simple. Indeed we find that 69% of test examples have a token-level edit distance of less than 5.

To answer the second question, we use the trained neural editors from the previous experiment, and test their performance in a "one-shot" transfer learning scenario. Specifically, we use our high-quality C#Fixers dataset, and for each fixer category F of semantically similar edits, we randomly select a seed edit {x -→ x + } ∈ F, and use its edit representation f ∆ (x -, x + ) to predict the updated code for all examples in F, i.e., we have

x+ = α(x -, f ∆ (x -, x + )), ∀ {x -→ x + } ∈ F.

This task is highly non-trivial, since a fixer category could contain more than hundreds of edit examples collected from different C# projects. Therefore, it requires the edit representations to generalize and transfer well, while being invariant of local lexical information like specific method names. To make the experimental evaluation more robust to noise, for each fixer category F, we randomly sample 10 seed edit pairs {x -→ x + }, compute their edit representations and use them to predict the edited version of the examples in F and evaluate accuracy of predicting the exact final version. We then report the best score among the 10 seed representations as the performance metric on F.

Tab. 5 summarizes the results and also reports the upper bound performance when using the goldstandard edit representation f ∆ (x -, x + ) to predict x + , and an approximation of the "lower bound" accuracies using pre-trained Seq2Seq and Graph2Tree models without edit encoders. We found that our neural Graph2Tree editor with the sequential edit encoder significantly outperforms the Seq2Seq editor, even though Seq2Seq performs better when using gold-standard edit representations. This suggest that the edit representations learned with the Graph2Tree model generalize better, especially for edits discussed in Sect. 4.2 that involve syntactic variations like RCS1021 (lambda expression simplification, 7.8% vs. 30.7% for Seq2Seq and Graph2Tree), and RCS1207 (change lambdas to method groups, 7.1% vs. 26.2%). Interestingly, we also observe that Seq2Seq outperforms the Graph2Tree model for edits with trivial surface edit sequences, where the Graph2Tree model does not have a clear advantage. For example, on RCS1015 (use nameof operator, e.g. Exception("x") → Exception(nameof(x))), the accuracies for Seq2Seq and Graph2Tree are 40.0% (14/35) and 28.6% (10/35), resp. We include more analysis of the results in Appendix C.

5. RELATED WORK:

Edits have recently been considered in NLP, as they represent interesting linguistic phenomena in language modeling and discourse (Faruqui et al., 2018; Yang et al., 2017a) . Specifically, Guu et al. (2017) present a generative model of natural language sentences via editing prototypes. Our work shares with Guu et al. (2017) in that (1) the posterior edit encoding model in Guu et al. ( 2017) is similar to our baseline "bag-of-edits" encoder in Sec. 4.4, and (2) the sequence-to-sequence sentence generation model given the prototype and edit representation is reminiscent of our Seq2Seq editor. In contrast, our work directly focuses on discriminative learning of representing edits and applying the learned edits for both sequential (NL) and structured (code) data. Another similar line of research is "retrieve-and-edit" models for text generation (Hashimoto et al., 2018) , where given an input data x, the target prediction y is generated by editing a similar target y that is retrieved based on the similarity between its source x and the input x. While these models typically require an "editor" component to generate the output by exploiting the difference between similar inputs, they usually use the simpler bag-of-edits representations (Wu et al., 2019) , or implicitly capture it via end-to-end neural networks (Contractor et al., 2018) . To our best knowledge, there is not any related work that classifies or otherwise explicitly represents the differences over similar input, with the exception of differential recurrent neural networks used for action recognition in videos (Veeriah et al., 2015; Zhuang et al., 2018) . This is a substantially different task, as the data includes a temporal component as well.

Source code edits are a widely studied artifact. Specialized software, such as git, is widely used to store source code revision histories. Nguyen et al. ( 2013) studied the repetitiveness of source code changes by identifying identical types of changes using a deterministic differencing tool. In contrast, we employ on a neural network to cluster similar changes together. Rolim et al. (2017) use such clusters to synthesize small programs that perform the edit. The approach is based on Rolim et al. (2018) extract manually designed syntactic features from code and cluster over multiple changes to find repeatable edit rules. Similarly, Paletov et al. (2018) extract syntactic features specifically targeting edits in cryptography API protocols. In this work, we try to avoid hand-designed features and allow a neural network to learn the relevant aspects of a change by directly giving as input the original and final version of a changed code snippet.

Modeling tree generation with machine learning is an old problem that has been widely studied in NLP. Starting with Maddison & Tarlow (2014) , code generation has also been considered as a tree generation problem. Close to our work is the decoder of Yin & Neubig (2017) which we use as the basis of our decoder. The work of Chen et al. ( 2018) is also related, since it provides a tree-to-tree model, but focuses on learning a single translation tasks and cannot be used directly to represent multiple types of edits. Both Yin & Neubig (2017) and Chen et al. (2018) have copying mechanism for single tokens, but our subtree copying mechanism is novel.

Autoencoders (see Goodfellow et al. (2016) for an overview) have a long history in machine learning.

Variational autoencoders (Kingma & Welling, 2013) are similar to autoencoders but instead of focusing on the learned representation, they aim to create accurate generative probabilistic models. Most (variational) autoencoders focus on encoding images but there have been works that autoencode sequences, such as text (Dai & Le, 2015; Bowman et al., 2015; Yang et al., 2017b) and graphs (Simonovsky & Komodakis, 2018; Liu et al., 2018) . Conditional variational autoencoders (Sohn et al., 2015) have a related form to our model (with the exception of the KL term), but are studied as generative models, whereas we are primarily interested in the edit representation.

6. DISCUSSION & CONCLUSIONS:

In this work, we presented the problem of learning distributed representation of edits. We believe that the dataset of edits is highly relevant and should be studied in more detail. While we have presented a set of initial models and metrics on the problem and obtained some first promising results, further development in both of these areas is needed. We hope that our work inspires others to work on this interesting problem in the future.

A DATASETS AND CONFIGURATION:

WikiAtomicEdits We randomly sampled 1040K insertion examples from the English portion of WikiAtomicEdits (Faruqui et al., 2018) dataset, with a train, development and test splits of 1000K, 20K and 20K.

GitHubEdits We cloned the top 54 C# GitHub repositories based on their popularity (Tab. 8). For each commit in the master branch, we collect the previous and updated versions of the source code, and extract all consecutive lines of edits that are smaller than three lines, and with at least three preceding and successive lines that have not been changed. We then filter trivial changes such as variable and identifier renaming, and changes happened within comments. We also limit the number of tokens for each edit to be smaller than 100, and down-sample edits whose frequency is larger than 30. Finally, we split the dataset by commit ids, ensuring that there are no edits in the training and testing (development) sets coming from the same commit. Tab. 6 lists some statistics of the dataset. C#Fixers We selected 16 C# fixers from Roslyn4 and Roslynator5 , and ran them on 6 C# projects to generate a small, high-quality C# fixers dataset of 2 878 edit pairs with known semantics. Table 7 lists the detailed descriptions for each fixer category. And more information can be found at https://github.com/JosefPihrt/Roslynator/blob/master/ src/Analyzers/README.md.

Network Configuration:

Throughout the experiments, we use a fixed edit representation size of 512. The dimensionality of word embedding, the hidden states of the encoder LSTMs, as well as the gated graph neural network is 128, while the decoder LSTM uses a larger hidden size of 256. For the graph-based edit encoder, we used a two-layer graph neural network, with 5 information propagation steps at each layer. During training, we performed early stopping, and choose the best model based on perplexity scores on development set. During testing, we decode a target element x + using a beam size of 5. 

B CLUSTERING EXPERIMENTS:

To qualitatively evaluate the quality of the learned edit representations. We use the models trained on the WikiAtomicEdits and GitHubEdits datasets to cluster natural language and code edits. We run K-Means clustering algorithm on 0.5 million sampled edits from WikiAtomicEdits, and all 90K code edits from GitHubEdits, producing 50 000 and 20 000 clusters for each dataset.

Tab. 9 and Tab. 10 list some example clusters on WikiAtomicEdits and GitHub datasets, respectively. Due to the size of clusters, we omit out-liners and present distinctive examples from each cluster. On the WikiAtomicEdits dataset, we found clusters whose examples are semantically and syntactically similar. More interestingly, on the source code data, we find representative clusters that relate to idiomatic patterns and best practices of programming. The clustering results produced by our model would be useful for programming synthesis toolkits to generate interpretable code refractory rules, which we leave as interesting future work.

Finally, we remark that the clustering results indicate that the encoding of edits is context-sensitive and position-sensitive for both natural language and source code data. For instance, the WikiAtomicEdits examples we present in Tab. 9 clearly indicate that semantically similar insertions also share similar editing positions. This is even more visible in code edits (Tab. 10). For instance, in the first example in Tab. 10, Equal() can be changed to Empty() only in the Assert namespace (i.e., the context).

These examples demonstrate that it is important for an edit encoder to capture the contextual and positional information in edits, a property that cannot be captured by simple "bag-of-edits" edit representation methods. Description Add a parenthetical expression also ... as to modify the subject 1. mid-state regional airport , also known as mid-state airport , is a small airport on in rush township , centre county in pennsylvania in the united states . 2. islamic culture , also known as saracenic culture , is a term primarily used in secular academia to describe the cultural practices common to historically islamic peoples . 3. birds of prey , also known as raptors , are birds that hunt for food primarily via flight , using their keen senses , especially vision . 4. tetyana styazhkina , also written as tetyana stiajkina , ( ; born april 10 , 1977 ) is a ukrainian cycle racer who rides for the chirio forno d'asolo team . 5. acid jazz , also known as club jazz , is a musical genre that combines elements of jazz , soul , funk , disco and hip hop .

Description Specify location using a prepositional phrase.

1. the douro fully enters portuguese territory just after the confluence with the gueda river ; once the douro enters portugal , major population centres are less frequent along the river . 2. mochou lake and mochou lake park are located at 35 hanzhongmen da jie in the jianye district of nanjing , china , west to qinhuai river . 3. reiner gamma is an albedo feature that is located on the oceanus procellarum , to the west of the reiner crater on the moon . 4. she made a brief return to the screen in " parrish " ( 1961 ) , playing the supporting role of mother which received little attention by the press . 5. he was involved in a few storylines , including one where he broke his toe and had a heart attack after he was pushed by a mugger in the market .

Description Add positional or temporal clause 1. at the time ajax and hercules were trapped behind a landslide at the gaillard cut , both were working to clear the landslide .

2.:

at the docks , hikaru attempts to befriend the tiger , but finds that it dislikes humans .

3.:

about the second , i do know they exist , but the question is whether they are considered a genre outside of japan . 4.

in the battle , shirou uses his reality marble , unlimited blade works and defeats gilgamesh . 5.

in the game , red is a curious 11 -year -old boy from pallet town . x-V0 = ((V1!= null) ? V1.Operand : null) as MemberExpression; x+ V0 = V1?.Operand as MemberExpression;

xstring V0 = V1 == null ? null : V1.GetType().Name; x+ string V0 = V1?.GetType().Name;

xvar V0 = V1 == null ? null : V1(V2). 

C BREAK-DOWN ANALYSIS OF TRANSFER LEARNING RESULTS:

Tab. 11 lists the detailed evaluation results for the transfer learning experiments discussed in Sect. 4.4. We refer readers to Tab. 7 for detailed descriptions of each fixer category. The neural Graph2Tree editor outperforms the Seq2Seq editor (both with sequential edit encoders) on 10 out of 16 fixer categories. However, we found that there are categories where both end-to-end system under-performs, even though the upper-bound accuracy is high (e.g. RCS1077, RCSRCS1197, RCS1207, RCS1032). While improving the generalization ability of the neural editor models to achieve better transfer learning performance is an important future work, we remark that this task is indeed non-trivial. First, some fixer categories cover a broad range of similar edits, which could not be captured by a single seed edit. xSecond, some categories contain syntactically or semantically complex refactoring rules. For instance, RCS1207 converts method groups into anonymous functions, involving changing multiple positions of the source code, which might not be trivially captured by the sequential edit encoder from a single example edit. Additionally, RCS1197 requires reasoning about a chain of expressions. It turns sb.Append(s1 + s2 + . . . + sN) into sb.Append(s1).Append(s2).[. . .]Append(sN)), which our current models are unable to reason about. More interestingly, we found that there are cases where the edits are syntactically simple, but could be semantically more difficult to learn. For instance, RCS1032 is about removing redundant parentheses from expressions. Although the edit pattern might seem to be syntactically simple at the AST level (replacing a ParethesizedExpressionSyntax node by its child node), determining which pair of parentheses is actually redundant in an expression (e.g. (a + b) * (c / d)) is semantically non-trivial to learn from a single edit example. We believe that further advances in (general) learning from source code are required to correctly handle these cases.

D IMPACT OF TRAINING SET SIZE:

To evaluate the data efficiency of our proposed approach, we tested the end-to-end performance of our neural editor model (Sect. 4.4, Tab. 4) with varying amount of training data. Tab. 12 lists the results. We found both Graph2Tree and Seq2Seq editors are relatively data efficient. They registered around 90% of the accuracies achieved using the full training set with only 60% of the training data.

E DETAILS OF HUMAN EVALUATION:

As discussed in Sect. 4.2, we performed human evaluation to rate the qualities of neighboring edits given a seed edit. The annotation instructions on GithubEdits and WikiAtomicEdits datasets are listed below. The annotation was carried out by three authors of this paper, and we anonymized the source of systems that generated the output. The three-way Fleiss' kappa inter-rater agreement is κ = 0.55, which shows moderate agreement (Artstein & Poesio, 2008) , an agreement level that is also used in other annotation tasks in NLP (Faruqui & Das, 2018) . The changed constituents in the seed edit and the neighboring edit are applied to the similar positions of the original sentence, serving the same syntactic and semantic role. For example, Examples

• Seed Edit x -var V0 = V1.Where(V2 => V2.Name == LITERAL).Single();

x + var V0 = V1.Single(V2=> V2.Name == LITERAL);

• Neighbor x -var V0 = V1.GetMembers().Where(V2 => V2.Kind == SymbolKind.Property).Single();

x + var V0 = V1.GetMembers().Single(V2 => V2.Kind == SymbolKind.Property);

• Seed Edit x -Type V0 = V1 == null ? typeof(object) : V1.GetType();

x + Type V0 = V1?.GetType() ?? typeof(object);

• Neighbor x -string V0 = V1 == null ? string.Empty : VAR1.ToString();

x + string V0 = V1?.ToString() ?? string.Empty;

• Seed Edit x -Assert.True(Directory.Exists(V0) == V1);

x + Assert.Equal(Directory.Exists(V0), V1);

• Neighbor x -Assert.True(V0.GetString(V0.GetBytes(LITERAL)) == V1.ContainingAssembly.Identity.CultureName);

x + Assert.Equal(V0.GetString(VAR0.GetBytes(LITERAL)), V1.ContainingAssembly.Identity.CultureName);

Rating 1 Syntactically or Semantically Related

The seed and neighboring edits share functionally or syntactically similar patterns.

Examples:

The following edit is a related edit of the first example above, as it applies the same simplification (.Where(COND).Func() to .Func(COND)), but for FirstOrDefault instead of Single:

• Seed Edit x -var V0 = V1.Where(V2 => V2.Name == LITERAL).Single();

x + var V0 = V1.Single(V2=> V2.Name == LITERAL);

• Neighbor x -var V0 = V1.Where(V2 => V3.ReportsTo == V2.EmployeeID).FirstOrDefault();

x + var V0 = V1.FirstOrDefault(V2 => V3.ReportsTo == V2.EmployeeID);

The following edit is a related edit of the second example above, as it also replaces a ternary expression for null checking with the ?. and ?? operators:

• Seed Edit x -Type V0 = V1 == null ? typeof(object) : V1.GetType();

x + Type V0 = V1?.GetType() ?? typeof(object);

• Neighbor x -var V0 = V1 != null ? V1.ToList() : new List<TextSpan>();

x + var V0 = V1?.ToList() ?? new List<TextSpan>();

We also considered pairs such as the following related, since they share similar syntactic structure

• Seed Edit x -V0.State = V1;

x + V0.SetState(VAR1);

• Neighbor x -V0.Quantity = V1;

x + V0.SetQuantity(V1);

Rating 0 Not Related

The seed and neighboring edits are not related based on the above criteria. the original old bay house , home of the chief factor , still exists and is now part of the fort vermilion national historic site . the population was 6,400 at the 2010 census and is part of the st. louis metropolitan area .

Rating 1 Syntactically Related:

The changed constituents in the seed and the neighboring edit are applied to the similar positions of the original sentence, and they play similar syntactic roles. This includes examples like adding a disfunction, adding a complement, prepositional clause or other syntactic constructs with similar phrases or language structures. For example,

Seed Edit:

Neighbor the douro fully enters portuguese territory just after the confluence with the gueda river ; once the douro enters portugal , major population centres are less frequent along the river .

she made a brief return to the screen in " parrish " ( 1961 ) , playing the supporting role of mother which received little attention by the press .

when they found it , they discovered a group of pagumon living there instead who immediately proceeded to treat the digidestined as honored guests , saying that pagumon are the fresh form of koromon .

in 2012 slote and his baseball book " jake " were the subject of an espn ( 30 for 30 ) short documentary in which slote describes his writing process and reads from the book , saying it is his best writing .

the aircraft was intended to be certified and supplied as a complete ready -to -flyaircraft for the flight training and aerial work markets .

in june reinforcements finally did arrive when provincial and militia units from new york , new jersey , and new hampshire were sent up from fort edward by general daniel webb .

Rating 0 Not Related:

The seed and neighboring edits are not related based on the above criteria.

:

Existing editing systems, e.g. the grammar checker in text editors and code refactoring module in IDEs, are powered by domain-specific, manually crafted rules, while we aim for a data-driven, domain-agnostic approach.

We leave the problem of identifying which edit representation f∆(x-, x+) to apply to x -as interesting future work.

EXPANDR corresponds to the APPLYCONSTR action in the original model ofYin & Neubig (2018). There is also a REDUCE action which marks the end of expanding a non-terminal with non-deterministic number of child nodes. SeeYin & Neubig (2018) for details.

http://roslyn.io

https://github.com/JosefPihrt/Roslynator

Figures and Tables:

Figure 2: (a) Graph representation of statement u = x + x. Rectangular (resp. rounded) nodes denote tokens (resp. non-terminals). (b) Sequence of tree decoding steps yielding x + x -23, where x + x is copied (using the TREECP action) from the context graph in (a).

Figure 3: Sequence (a) and graph (b) representation of edit of v.F = x + x to u = x + x.

Figure 4: t-SNE visualization of edits from 13 C# fixers, where point color indicates the fixer. Labels indicate the id of the fixer, see main text.

ToArray(); x+ var V0 = V1?.Invoke(V2).ToArray(); Description Optimize LINQ queries x-var V0 = V1.Customers.Where(V2 => V2.CustomerID == LITERAL) .FirstOrDefault(); x+ var V0 = V1.Customers .FirstOrDefault(V2 => V2.CustomerID == LITERAL); x-var V0 = V1.TypeConverters.Where(V2 => V2.CanConvertTo(V3, V1)) .FirstOrDefault(); x+ var V0 = V1.TypeConverters .FirstOrDefault(V2 => V2.CanConvertTo(V3, V1)); x-var V0 = this.V1.Where(V2 => V2.CanDeserialize(V3)) .FirstOrDefault(); x+ var V0 = this.V1.FirstOrDefault(V2 => V2.CanDeserialize(V3)); x-var V0 = V1.Where(V2 => V2.Item1 == V3 && V2.Item2 == V4) .FirstOrDefault(); x+ var V0 = V1.FirstOrDefault(V2 => V2.Item1 == V3 && V2.Item2 == V4); Description Change from Add function to indexer. x-V0.Add(V1.key, V1.V2); x+ V0[V1.key] = V1.V2; x-V0.Add(V1.Id, V2); x+ V0[V1.Id] = V2; x-V0.Add(V1.Etag, V1); x+ V0[V1.Etag] = V1; x-V0.Add(V1.V2, V3.Merge(V1.V4)); x+ V0[V1.V2] = V3.Merge(V1.V4);

Natural language human evaluation results and 3 nearest neighbors. Inserted text marked. Example 1 neural editing model returns syntactically and semantically similar edits. Example 2 Neural edit representations are sensitive to position.

Table content: <table/>

Relevance scores of human evaluation on GitHubEdits data. Acc.@1 denotes the ratio that the 1-nearest neighbor has a score 2.

Table content: <table><tr><td>Model</td><td colspan="3">DCG@3 NDCG@3 (%) Acc.@1 (%)</td></tr><tr><td>BoW</td><td>7.77</td><td>75.99</td><td>58.46</td></tr><tr><td>Seq2Seq -Seq Edit Encoder</td><td>10.09</td><td>90.05</td><td>75.90</td></tr><tr><td>Graph2Tree -Seq Edit Encoder</td><td>10.56</td><td>91.40</td><td>79.49</td></tr><tr><td>Graph2Tree -Graph Edit Encoder</td><td>9.44</td><td>86.20</td><td>72.31</td></tr></table>

Two example source code edits and their nearest neighbors based on the edit representations computed by each model.

Table content: <table><tr><td>Example 1</td></tr><tr><td>x -: V0.SendSelectSoundRequest((int)V1);</td></tr><tr><td>x + : V0.SendSelectSoundRequest(V1);</td></tr></table>

Test performance of different neural editors.

Table content: <table><tr><td>Model</td><td colspan="3">Acc.@1 (%) Recall@5 (%) PPL per token</td></tr><tr><td>GitHubEdits</td><td/><td/><td/></tr><tr><td>Seq2Seq -Bag-of-Edits Encoder</td><td>44.05</td><td>54.97</td><td>1.4808</td></tr><tr><td>Seq2Seq -Seq Edit Encoder</td><td>59.63</td><td>65.46</td><td>1.2792</td></tr><tr><td>Graph2Tree -Bag-of-Edits Encoder</td><td>40.66</td><td>49.42</td><td>1.5058</td></tr><tr><td>Graph2Tree -Seq Edit Encoder</td><td>57.49</td><td>62.94</td><td>1.3043</td></tr><tr><td>Graph2Tree -Graph Edit Encoder</td><td>48.05</td><td>56.51</td><td>1.3712</td></tr><tr><td>WikiAtomicEdits</td><td/><td/><td/></tr><tr><td>Seq2Seq -Bag-of-Edits Encoder</td><td>23.73</td><td>43.47</td><td>1.3730</td></tr><tr><td>Seq2Seq -Seq Edit Encoder</td><td>72.94</td><td>76.53</td><td>1.0527</td></tr></table>

Transfer learning results on C# fixers data, averaged across all fixer categories.

Table content: <table><tr><td colspan="3">Model Acc.(%) Acc.  Seq2Seq -Seq Edit Encoder 38.35 77.67</td><td>41.50</td><td>83.84</td></tr><tr><td>Graph2Tree -Seq Edit Encoder</td><td>49.21</td><td>77.30</td><td>51.93</td><td>81.77</td></tr><tr><td>Baselines (no edit encoding)</td><td/><td/><td/><td/></tr><tr><td>Seq2Seq w/o Edit Encoder</td><td>7.07</td><td>-</td><td>14.29</td><td>-</td></tr><tr><td>Graph2Tree w/o Edit Encoder</td><td>8.81</td><td>-</td><td>11.90</td><td>-</td></tr><tr><td/><td/><td/><td/><td/></tr></table>

Statistics of the GitHubEdits Dataset Average Num. Tokens in x - Average size of AST for T (x -) 28.5 Average size of AST for T (x + ) 29.4

Table content: <table><tr><td/><td>16.4</td></tr><tr><td>Average Num. Tokens in x +</td><td>17.0</td></tr><tr><td>Average Edit Distance</td><td>5.0</td></tr></table>

Descriptions of fixer categories in C#Fixers dataset

Table content: <table><tr><td>Fixer ID</td><td>Description</td><td colspan="4">Num. Edits Example</td></tr><tr><td>CA2007</td><td colspan="2">apply .ConfigureAwait(false) to await statements</td><td>1051</td><td colspan="2">x-: await Console.WriteAsync() x+: await Console.WriteAsync()</td></tr><tr><td/><td/><td/><td/><td/><td>.ConfigureAwait(false)</td></tr><tr><td colspan="2">IDE0004 Cast is redundant</td><td/><td>53</td><td colspan="2">x-: var x = 1; var b = (int)x; x+: var x = 1; var b = x;</td></tr><tr><td colspan="2">RCS1015 Use nameof operator</td><td/><td>35</td><td colspan="2">x-: Exception("parameter"); x+: Exception(nameof(parameter));</td></tr><tr><td/><td/><td/><td/><td/><td>var x = items.Select(f =&gt;</td></tr><tr><td colspan="2">RCS1021 Simplify lambda expression</td><td/><td>411</td><td>x-:</td><td>{</td><td>return f.ToString();</td></tr><tr><td/><td/><td/><td/><td/><td>});</td></tr><tr><td/><td/><td/><td/><td colspan="2">x+: var x = items.Select( f =&gt; f.ToString());</td></tr><tr><td colspan="2">RCS1032 Remove redundant parentheses</td><td/><td>24</td><td colspan="2">x-: if ((x)) {} x-: if (x) {}</td></tr><tr><td colspan="2">RCS1058 Use compound assignment</td><td/><td>43</td><td colspan="2">x-: i = i + 2; x+: i += 2;</td></tr><tr><td colspan="2">RCS1077 Optimize LINQ method call</td><td/><td>200</td><td colspan="2">x-: items.Where(f =&gt; Foo(f)).Any(); x+: items.Any(f =&gt; Foo(f));</td></tr><tr><td>RCS1089</td><td>Use --/++ operator instead of assignment</td><td/><td>75</td><td colspan="2">x-: i = i + 1; x+: i += 1;</td></tr><tr><td colspan="2">RCS1097 Remove redundant ToString call</td><td/><td>20</td><td colspan="2">x-: var x = s.ToString(); x+: var x = s;</td></tr><tr><td colspan="2">RCS1118 Mark local variable as const</td><td/><td>477</td><td colspan="2">x-: string s = "a"; string s2 = s + "b";</td></tr><tr><td/><td/><td/><td/><td colspan="2">x+: const string s = "a"; string s2 = s + "b";</td></tr><tr><td>RCS1123</td><td>Add parentheses according to operator precedence</td><td/><td>109</td><td colspan="2">x-: if (x || y &amp;&amp; z) {} x+: if (x || (y &amp;&amp; z) ) {}</td></tr><tr><td colspan="2">RCS1146 Use conditional access</td><td/><td>71</td><td colspan="2">x-: x != null &amp;&amp; x.StartsWith("a"); x+: x?.StartsWith("a");</td></tr><tr><td>RCS1197</td><td>Optimize call of StringBuilder's Append/AppendLine</td><td/><td>95</td><td colspan="2">x-: sb.Append(s + "x"); x+: sb.Append(s).Append("x");</td></tr><tr><td colspan="3">RCS1202 Avoid NullReferenceException</td><td>56</td><td colspan="2">x-: items.First().ToString(); x+: items?.First().ToString();</td></tr><tr><td>RCS1206</td><td>Use conditional access instead of conditional expression</td><td/><td>116</td><td colspan="2">x-: int i = (x != null) ? x.Value.GetHashCode() : 0; x+: int i = x?.GetHashCode() ?? 0;</td></tr><tr><td>RCS1207</td><td>Use method group instead of anonymous function</td><td/><td>42</td><td colspan="2">x-: items.Select(f =&gt; Foo(f));</td></tr></table>

Our C# GitHub dataset projects

Table content: <table><tr><td>Name</td><td>GitHub Id</td><td>Description</td></tr><tr><td>acat</td><td>intel/acat</td><td>Assistive Context-Aware Toolkit</td></tr><tr><td>akka.net</td><td>akka/akka.net</td><td>Distributed Actors</td></tr><tr><td>aspnetboilerplate</td><td>aspnetboilerplate/aspnetboilerplate</td><td>ASP.NET boilerplate</td></tr><tr><td>AutoMapper</td><td>AutoMaper/AutoMapper</td><td>Object-Object Mapper</td></tr><tr><td>BotBuilder</td><td>Microsoft/BotBuilder</td><td>Bot Framework</td></tr><tr><td>CefSharp</td><td>cefsharp/CefSharp</td><td>Chromium Embedded Framework Bindings</td></tr><tr><td>choco</td><td>chocolatey/choco</td><td>package mananger</td></tr><tr><td>cli</td><td>dotnet/cli</td><td>.NET CLI Tools</td></tr><tr><td>CodeHub</td><td>CodeHubApp/CodeHub</td><td>iOS application</td></tr><tr><td>coreclr</td><td>dotnet/coreclr</td><td>.NET Framework</td></tr><tr><td>corefx</td><td>dotnet/corefx</td><td>.NET FOundational Libraries</td></tr><tr><td>dapper</td><td>StackExchange/Dapper</td><td>Object Mapper</td></tr><tr><td>dnSpy</td><td>0xd4d/dnSpy</td><td>.NET debugger and assembly editor</td></tr><tr><td>duplicati</td><td>duplicati/duplicati</td><td>Encrypted Cloud Backups</td></tr><tr><td>EntityFramework</td><td>aspnet/EntityFramework</td><td>Object-Relational Mapper</td></tr><tr><td>EntityFrameworkCore</td><td>aspnet/EntityFrameworkCore</td><td>Object-Relational Mapper -Core</td></tr><tr><td>FluentValidation</td><td>JeremySkinner/FluentValidation</td><td>Validation Rules</td></tr><tr><td>framework</td><td>accord-net/framework</td><td>ML, CV Framework</td></tr><tr><td>GVFS</td><td>Microsoft/VFSForGit</td><td>Git Virual File System</td></tr><tr><td>Hangfire</td><td>HangfireIO/Hangfire</td><td>Background job library</td></tr><tr><td>ILSpy</td><td>icsharpcode/ILSpy</td><td>Decompiler</td></tr><tr><td>JavaScriptServices</td><td>aspnet/JavaScriptServices</td><td>ASP.NET JS Services</td></tr><tr><td>MahApps.Metro</td><td>MahApps/MahApps.Metro</td><td>WPF Framework</td></tr><tr><td>MaterialDesignInXamlToolkit</td><td>MaterialDesignInXamlToolkit/ Materi-</td><td>Design XAML &amp; WPF</td></tr><tr><td/><td>alDesignInXamlToolkit</td><td/></tr><tr><td>mono</td><td>mono/mono</td><td>.NET implementation</td></tr><tr><td>monodevelop</td><td>mono/monodevelop</td><td>IDE</td></tr><tr><td>MonoGame</td><td>MonoGame/MonoGame</td><td>Game Framework</td></tr><tr><td>msbuild</td><td>Microsoft/msbuild</td><td>Build Tool</td></tr><tr><td>Mvc</td><td>aspnet/Mvc</td><td>MVC Framework</td></tr><tr><td>Nancy</td><td>NancyFx/Nancy</td><td>HTTP based services</td></tr><tr><td>Newtonsoft.Json</td><td>JamesNK/Newtonsoft.Json</td><td>JSON framework</td></tr><tr><td>NLog</td><td>NLog/NLog</td><td>Loggin for .NET</td></tr><tr><td>OpenLiveWriter</td><td>OpenLiveWriter/ OpenLiveWriter</td><td>Text editor</td></tr><tr><td>OpenRA</td><td>OpenRA/OpenRA</td><td>Strategy Game Engine</td></tr><tr><td>Opserver</td><td>opserver/Opserver</td><td>Monitoring System</td></tr><tr><td>orleans</td><td>dotnet/orleans</td><td>Distributed Virtual Actors</td></tr><tr><td>PowerShell</td><td>PowerShell/PowerShell</td><td>Command Line</td></tr><tr><td>Psychson</td><td>brandonlw/Psychson</td><td>Firmware</td></tr><tr><td>PushSharp</td><td>Redth/PushSharp</td><td>Push Notifications</td></tr><tr><td>ravendb</td><td>ravendb/ravendb</td><td>Database</td></tr><tr><td>ReactiveUI</td><td>reactiveui/ReactiveUI</td><td>Reactive MVC Framework</td></tr><tr><td>RestSharp</td><td>restsharp/RestSharp</td><td>HTTP/REST Client</td></tr><tr><td>roslyn</td><td>dotnet/roslyn</td><td>.NET Compiler</td></tr><tr><td>Rx.NET</td><td>dotnet/reactive</td><td>Reactive extensions.</td></tr><tr><td>ServiceStack</td><td>ServiceStack/ServiceStack</td><td>Web Service Framework</td></tr><tr><td>shadowsocks-windows</td><td>shadowsocks/ shadowsocks-windows</td><td>Cryptography</td></tr><tr><td>ShareX</td><td>ShareX/ShareX</td><td>Screen Recorder</td></tr><tr><td>SignalR</td><td>SignalR/SignalR</td><td>Real-time web framework</td></tr><tr><td>Sonarr</td><td>Sonarr/Sonarr</td><td>PVR</td></tr><tr><td>SpaceEngineers</td><td>KeenSoftwareHouse/ SpaceEngineers</td><td>Game</td></tr><tr><td>SparkleShare</td><td>hbons/SparkleShare</td><td>File Sharing</td></tr><tr><td>StackExchange.Redis</td><td>StackExchange/ StackExchange.Redis</td><td>Redis Client</td></tr><tr><td>WaveFunctionCollapse</td><td>mxgmn/ WaveFunctionCollapse</td><td>Bitmap/tilemap Generator</td></tr><tr><td>Wox</td><td>Wox-launcher/Wox</td><td>Launcher</td></tr></table>

Example clusters on WikiAtomicEdits data using representations learned by a neural Seq2Seq editor with sequential edit encoder Description Add a person's middle name 1. isaiah marcus rankin ( born 22 may 1978 in london ) is an english professional footballer currently playing for stevenage borough . 2. audrey kathleen brown ( born 24 may , 1913 ) is a british athlete who competed mainly in the 100 metres . 3. alice edith rumph was a painter , etcher , and teacher . 4. mark larry taufua is an australian professional rugby league player . 5. monique edith lamoureux ( born july 3 , 1989 ) is an american ice hockey player .

Table content: <table/>

Example clusters on GithubEdits data using representations learned by a Graph2Tree editor with sequential edit encoder. Locally defined variable names are canonicalized.

Table content: <table><tr><td>Description Switch from Assert.Equal to Assert.Empty</td></tr><tr><td>x-Assert.Equal(0, V0.ProjectIds.Count);</td></tr><tr><td>x+ Assert.Empty(V0.ProjectIds);</td></tr><tr><td>x-Assert.Equal(0, V0.ProjectReferences.Count());</td></tr><tr><td>x+ Assert.Empty(V0.ProjectReferences);</td></tr><tr><td>x-Assert.Equal(0, V0.TrustedSelectionPaths.Count);</td></tr><tr><td>x+ Assert.Empty(V0.TrustedSelectionPaths);</td></tr><tr><td>x-Assert.Equal(0, V0.Count);</td></tr><tr><td>x+ Assert.Empty(V0);</td></tr><tr><td>x-Assert.Equal(0, V0.Messages.Count);</td></tr><tr><td>x+ Assert.Empty(V0.Messages);</td></tr><tr><td>Description Use conditional access</td></tr><tr><td>x-Type V0 = V1 == null ? null : V1.GetType();</td></tr><tr><td>x+ Type V0 = V1?.GetType();</td></tr></table>

Break-down performance results on the transfer learning task. See Tab. 7 for descriptions of each fixer category.

Table content: <table/>

Test performance of end-to-end experiments with varying amount of training data.

Table content: <table><tr><td colspan="4">Training Set Size Acc.@1 (%) Recall@5 (%) PPL per token</td></tr><tr><td>GitHubEdits</td><td/><td/><td/></tr><tr><td colspan="2">Graph2Tree -Seq Edit Encoder</td><td/><td/></tr><tr><td>20%</td><td>43.88</td><td>50.53</td><td>1.5703</td></tr><tr><td>40%</td><td>50.44</td><td>56.63</td><td>1.4152</td></tr><tr><td>60%</td><td>53.78</td><td>60.00</td><td>1.3720</td></tr><tr><td>80%</td><td>55.51</td><td>60.85</td><td>1.3392</td></tr><tr><td>100%</td><td>57.49</td><td>62.94</td><td>1.3043</td></tr><tr><td>WikiAtomicEdits</td><td/><td/><td/></tr><tr><td>Seq2Seq -Seq Edit Encoder</td><td/><td/><td/></tr><tr><td>20%</td><td>42.87</td><td>48.24</td><td>1.4123</td></tr><tr><td>40%</td><td>57.72</td><td>62.31</td><td>1.1812</td></tr><tr><td>60%</td><td>65.22</td><td>69.62</td><td>1.1070</td></tr><tr><td>80%</td><td>68.44</td><td>73.34</td><td>1.0751</td></tr><tr><td>100%</td><td>72.94</td><td>76.53</td><td>1.0527</td></tr></table>

Annotation Instruction for GitHubEdits DataRating 2 Semantically and Syntactically Equivalent

Table content: <table/>

Annotation Instruction for WikiAtomEdits Data Rating 2 Semantically and Syntactically Equivalent The changed constituents in the seed edit and the neighboring edit are applied to the similar positions of the original sentence, serving the same syntactic and semantic role. For example, Seed Edit Neighbor chaz guest ( born 1961 ) was born in niagra falls , . . . , a decorated hero in wwii in europe , including the purple heart . randal l. schwartz ( born november 22 , 1961 ) , also known as merlyn , is an american author , system administrator and programming consultant. he was elected to donegal county council for sinn fin in 1979 , and held his seat until his death at age 56 . davis graduated from high school in january 1947 , immediately enrolling at wittenberg college in rural ohio at age 17 . dror feiler served as a paratrooper in the israel defense forces . nagaur fort -sandy fort ; centrally located ; 2nd century old ; witnessed many battles ; lofty walls & spacious campus ; having many palaces & temples inside .

Table content: <table/>

Peer review (iclr2019):
 The authors state nicely and clearly the main contributions they see in their work (Intro, last paragraph). Specifically the state the paper: 1) present a new and important machine learning task, 2) present a family of models that capture the structure of edits and compute efficient representations, 3) create a new source code edit dataset, 4) perform a set of experiments on the learned edit representations and present promising empirical evidence that the models succeed in capturing the semantics of edits. We decided to organize this review by commenting on the above-stated contributions one at a time: “A new and important machine learning task” Regarding “new task”: PRO: We are unfamiliar with past work which presents this precise task; the task is new. Section 5 makes a good case for the novelty of this work. CON: None. Regarding “important task”: PRO: The authors motivate the task with tantalizing prospective applications-- automatically editing text and code, e.g. for grammar, clarity, and style. Conceptualizing edits as NLP objects of interest that can be concretely represented, clustered, and used for prediction is an advance. CON: Many text editors, office suites, and coding IDEs already include features which automatically suggest or apply edits for grammar, clarity, and style. The authors do not describe shortcomings in existing tools that might be better addressed using distributed representations of edits. Consequently, the significance of the proposed contribution is unclear. “A family of models that capture the structure of edits and compute efficient representations” Regarding “a family of models”: PRO: The family of models presented by the authors clearly generalizes: such models may be utilized for computational experiments on datasets and edit types beyond those specifically utilized in this evaluation. The authors apply well-utilized neural network architectures that may be trained and applied to large datasets. The architecture of the neural editor permits evaluation of the degree to which the editor successfully predicts the correct edit given a pre-edit input and a known representation of a similar edit. CON: The authors do not propose any scheme under which edit representations might be utilized for automatically editing text or code when an edit very similar to the desired edit is not already known and its representation available as input. Hence, we find the authors do not sufficiently motivate the input scheme of their neural editor. The input scheme of the neural editor makes trivial the case in which no edit is needed, as the editor would learn during training that the output x+ should be the same as the input x- when the representation of the “zero edit” is given as input. While the authors discuss the importance of “bottlenecking” the edit encoder so that it does not simply learn to encode the desired output x+, they do not concretely demonstrate that the edit encoder has done otherwise in the final experiments. Related to that: If the authors aimed to actually solve automated edits in text/code then it seems crucial their data contained "negative examples" i.e. segments which require no edits. In such an evaluation one would test also when the algorithm introduces unnecessary/erroneous edits. Regarding “capture structure of edits”: PRO: The authors present evidence that edit encoders tightly cluster relatively simple edits which involve adding or removing common tokens. The authors present evidence that relatively simple edits completed automatically by a “fixer” often cluster together, i.e. a known signal is retained in clustering. The authors present evidence that the nearest neighbors of edits in an edit-representation space often are semantically or structurally similar, as judged by human annotators. Section 4.3 includes interesting observations comparing edit patterns better captured by the graph or seq edit encoders. CON: The details of the human annotation tasks which generated the numerical results in Tables 1 and 2 are unclear: were unbiased third parties utilized? Were the edits stripped of their source-encoder label when evaluated? Objectively, what separates an “unrelated” from a “similar” edit, and what separates a “similar” from a “same” edit? Did multiple human annotators undertake this task in parallel, and what was their overall concordance (e.g. “intercoder reliability”)? Without concrete answers to these questions, the validity and significance of the DCG/NDCG results reported in Tables 1 and 2 are unclear. It is not clear from the two examples given in Table 1 that the three nearest neighbors embedded by the Seq encoder are “better”, i.e. overall more semantically and/or syntactically similar to the example edit, than those embedded by the Bag of Words model. It is unclear which specific aspects of “edit structure” are better captured by the Seq encoder than the Bag of Words model. The overall structure of Tables 1 and 2 is awkward, with concrete numerical results dominated by a spatially large section containing a small number of examples. “create a new source code edit dataset” PRO: The authors create a new source code edit dataset, an important contribution to the study of this new task. CON: Minor: is the provided dataset large enough to do more than simple experiments? See note below on sample size. “present promising empirical evidence that the models succeed in capturing the semantics of edits” PRO: The experiment results show how frequently the end-to-end system successfully predicted the correct edit given a pre-edit input and a known representation of a similar edit. Gold standard accuracies of more than 70%, and averaged transfer learning accuracies of more than 30%, suggest that this system shows promise for capturing the semantics of edits. CON: Due to concerns expressed above about the model design and evaluation of the edit representations, it remains unclear to what degree the models succeed in capturing the semantics of edits. Table 11 shows dramatic variation in success levels across fixer ID in the transfer learning task, yet the authors do not propose ways their end-to-end system might be adjusted to address areas of weak performance. The authors do not discuss the impact of training set size on their evaluation metrics. The authors do not discuss the degree to which their model training task would scale to larger language datasets such as those needed for the motivating applications. ############## Based on the authors* response, revisions, and disucssions we have updated the review and the score.