Learning to Model Editing Processes

1 INTRODUCTION . Revising and editing are a central part of the the human creative worflow , with most original content ( e.g . art , books , articles , source code ) being developed not in a single iteration , but in many iterations with each more refined than the last . How can we model these editing processes from inception to completion ? In this paper , we attempt to provide a first answer to this question , specifically focusing on generation of sequential data such as natural language documents or source code . Most current work on language generation tasks such as machine translation ( Vaswani et al. , 2017 ) , language modeling ( Baevski & Auli , 2018 ) , or summarization ( See et al. , 2017 ) generates the target sentence or document in a single pass ( usually from left to right ) . There has been a reasonable amount of work that can generate edits to existing sequences for the purposes of post-editing , grammatical error correction ( Omelianchuk et al. , 2020 ) , text style transfer ( Mallinson et al. , 2020 ; Malmi et al. , 2020 ; Reid & Zhong , 2021 ) , sentence fusion ( Malmi et al. , 2019 ) , or machine translation ( Gu et al. , 2019 ) . However , these works all 1 ) model only a single editing step and 2 ) do not fully define a model of incrementally editing a document from a blank slate to the final text , and thus do not stand in for the one-pass generative models of sequences described above . In this context , we propose the task of modeling editing processes in which we look to explicitly model the likelihood of the entire process of revising a document to a polished form . In particular , and in contrast to previous works on modeling edits , we hypothesize that in order to edit more accurately , instead of simply learning to predict the next revision given the current revision , we should have context of multiple previous revisions when deciding when and how to edit the document next . Given the novelty of this task , this paper simultaneously proposes 1 ) both baseline and novel models for the task , 2 ) evaluation methodology , and 3 ) evaluation datasets that can be used to compare model efficacy . The proposed multi-step editing model predicts discrete edit operations ( Levenshtein , 1966 ) to enable progressive refinement as shown in Figure 1 , rather than framing sequence editing as a sequence to sequence task ( Reid & Zhong , 2021 ; Faltings et al. , 2021 ) . In the figure , for each step of the editing process discrete operations ( insert , replace , delete , keep ) are predicted and then actions ( such as generating a replaced span ) are performed based on this . This has two benefits : 1 ) it allows the model to scale well with respect to input sequence length , and 2 ) allows us to make substantial changes with fewer actions ( Grangier & Auli , 2018 ) . We use these edit operations to condition a semi-autoregressive model that is able to insert and replace multiple spans at once . Combined with 1All data and code will be open-sourced upon acceptance . an encoder that is able to quickly specify which spans of text need to be changed and how , this allows for considerable changes to be made to the text ( including insertion , deletion , re-ordering , and replacement ) in a relatively simple and cheap manner . Furthermore , this allows us to disentangle how likely the model is to operate ( replace , delete , etc ... ) on a given span , and how likely the model thinks the generated text for a given span is . As we are modeling editing processes , and hypothesize that context from edits applied to the sequence are helpful , we propose a method for edit-aware sequence compression which can compress sequences into their edit operations and use relative edit positional embeddings to specify the position of edits relative to each other . To evaluate the efficacy of proposed and baseline models , we propose an evaluation metric , edit perplexity ( ePPL ) , which combines both of these ideas for a unified metric to measure edit likelihood . However we can easily disentangle this into operation perplexity ( oPPL ) and generative perplexity ( gPPL ) which measures the likelihood of performing a given operation on a span , versus the likelihood of newly generated text for the next revision . Given that the task of modeling natural editing processes in itself is novel , we collect new datasets to study this behavior ; WIKIREVISIONS and CODEREVISIONS . These datasets , in the code and natural language domains respectively , cover over 2.5M and 2.3M natural sequential revisions . We train and evaluate our proposed models on these datasets . Our experimental results demonstrate that edit perplexity , generative perplexity , and operation perplexity are all reduced when one models edits with previous changes/edits as context . We also qualitatively demonstrate the ability of the model to generate natural edit sequences , and demonstrate the utility of the learned representations on downstream tasks of commit message generation ( Loyola et al. , 2017 ) and edit intention classification ( Yang et al. , 2017 ) . 2 PROBLEM DEFINITION . Let X = { x0 , x1 , . . . , xN } be a series of N versions of a document , where the ith revised document is denoted by xi . x0 represents an initial state ( generally the null string ) , and xN represents the current state of the edited document . The probability of this series of document versions occurring can be decomposed as p ( X ) = N∏ i=1 p ( xi|xi−10 ) , ( 1 ) where xi−10 : = x0 , . . . , xi−1 ( similarly below ) . The right hand side is the likelihood of the transformation of the previous document version xi−1 to the current document version xi given the previous revision history x < i . We refer to the likelihood of the whole revision process as the edit likelihood , and judge learned models based on their ability to achieve high edit likelihood on held-out data . Note that standard generative models ( specifically language models ; LMs ) calculate the probability of only the final version p ( xN ) , whereas the proposed formulation calculates the probability of the entire sequence of document edits . It nonetheless could be used to calculate the final version ’ s likelihood by treating the editing process as latent and marginalizing over it p ( xN ) = ∑ { X̃=x̃N1 |x̃N=xN } p ( X̃ ) . ( 2 ) Thus our formulation , in contrast to previous single-step models of edits ( Yin et al. , 2019b ; Malmi et al. , 2019 ; Reid & Zhong , 2021 ) , can also be used to define a generative model over single documents . It is also worth noting that the final document likelihood is lower-bounded by the edit likelihood ; i.e . p ( xN ) ≥ p ( X ) . 3 MODELING EDITING PROCESSES . In this section , we now describe our approach to actually modeling these sequences of edits through ( 1 ) a decomposition of the modeling process into a sequential process of modeling edit operations then actual edits , and ( 2 ) neural model of modeling these operations and edits . 3.1 MODELING OPERATIONS AND OPERATION-CONDITIONED EDITS . While the probability p ( xi|xi−10 ) of the next document given all previous document versions could theoretically be modeled with a single neural sequence model , this is infeasible computationally ( and likely infeasible from learning perspective as well ) . To simplify this problem , we employ the n-th order Markov assumption , assuming that the probability of the next document is conditioned only on the previous n documents p ( xi|xi−1i−n ) . This probability could be modeled directly , and in fact in the case of n = 1 this becomes analogous to the single-step editing problem tackled by previous work ( Yin et al. , 2019b ; Malmi et al. , 2019 ; Reid & Zhong , 2021 ; Faltings et al. , 2021 ) . To our knowledge , no previous work has modeled natural editing processes with n > 1 . However , in the interest of both efficiency and efficacy , we take an alternative approach where we first predict a set of edit operations ei , and then predict the next document version based on the previous documents and these edit operations : p ( xi|xi−1i−n ) ≈ p ( xi , ei|x i−1 i−n ) ( 3 ) = p ( xi|ei , xi−1i−n ) p ( ei|x i−1 i−n ) . ( 4 ) The first approximation becomes an equality when the edit operations can be deterministically derived from xi and xi−1 , i.e . p ( ei|xi , xi−1 ) = 1 , as is the case described below . Edit Operations . We base the edit operations in e on those calculated by the Levenshtein algorithm ( Levenshtein , 1966 ) , including token-level insertions , deletions , and substitutions . These are expressed as four operations insert , delete , keep , and replace denoted by { INSERT , DELETE , KEEP , REPLACE } . For multi-word insertions and replacements , e.g . a replacement of a contiguous span of words , we apply the the same REPLACE label to all tokens in this span . An example of each operation is shown in Figure 1 . Decomposed Edit Likelihood . We can then re-define our previous formulation of edit likelihood : P ( xN1 ) = N∏ i=1 p ( xi|ei , xi−1i−n ) p ( ei|x i−1 i−n ) , ( 5 ) and analogously define edit log-likelihood Lxe : = logP ( xN1 ) = N∑ i=1 log p ( xi|ei , xi−1i−n ) + log p ( ei|x i−1 i−n ) . ( 6 ) We can further decompose this into only the components corresponding to the edit operations Le : =∑N i=1 log p ( ei|x i−1 i−n ) , or the operation-conditioned edits Lx|e : = ∑N i=1 log p ( xi|ei , x i−1 i−n ) , both of which we will utilize for devising evaluation metrics in Section 5.2 below . 3.2 EDITPRO . In this section , we propose a model of multi-step editing processes , EDITPRO , which is based on a semi-autoregressive edit-conditioned encoder-decoder model with a Transformer ( Vaswani et al. , 2017 ) . The model ( depicted in Figure 2 ) contains three main components : ( 1 ) an edit encoder , ( 2 ) an operation classifier and ( 3 ) an insertion-replacement decoder . Edit Encoder . The encoder fenc takes in a document version xi−1 and feeds it through multiple self-attention and feedforward layers ( Vaswani et al. , 2017 ) to produce contextual representations for each token . In the case that we perform variable-order edit modeling , we use cross-attention to feed in representations of previous edit steps . For models where n > 1 , we feed in n− 1 additional edit sequences – we describe this process after describing our methods for edit sequence prediction . Edit Operation Prediction . We use an autoregressive tagger , using a single Transformer layer with a causal attention mask , that models the probability of each edit in edit operation sequence e = eM1 from left to right , p ( ej |e j−1 1 ) . Notably , we also performed preliminary experiments with a tagger that predicts operations independently , but found it was heavily biased towards the KEEP operation as most words are kept in any single document revision , and thus did not produce coherent multi-word edit sequences when sampling sequences of edits . Generating Replacements and Insertions . When editing , given our four Levenshtein operations ( INSERT , REPLACE , KEEP , DELETE ) , two of them — INSERT and REPLACE— entail generation of new content conditioned on the current revision of the document . Given our predicted edit operations e , we propose a semi-autoregressive model with a Transformer decoder that can decode multiple spans in parallel for efficiency purposes . Each edit span contains the following properties : it has a start index ( denoted by sstart ) , end index ( denoted by send ) , and an operation type ( denoted by stype ) . Note that these can be simply be extracted by looking at contiguous spans of a certain type in an edit ( e.g . REPLACE for descended from→ domesticated descendant of in Figure 1 ) . We use a mean pooling operation to aggregate the contextual vectors produced by fenc ( x ) into span representation x̂s : x̂s = 1 send − sstart send∑ t=sstart fenc ( x ) t ( 7 ) We then update the span representation x̂s by taking the sum of the appropriate operation embedding for the span type and the current span representation and feed it to a multi-layer perceptron with an intermediate non-linearity : x̂s ← MLP ( Wop ( e ) s+x̂s ) , where Wop denotes an embedding matrix for each operation . x̂s is then used to initialize the < s > token for the decoder span to further condition the generative process . Encoding Edit History . As we look to investigate variable order edit modeling over long sequences of text , we need a way to be able to represent edits in a way useful for predicting the next editing steps . Previous work ( Yin et al. , 2019a ; Marrese-Taylor et al. , 2021 ; Yao et al. , 2021 ) has focused largely on learning a single-vector representation for edits which is compressed but limited in expres- siveness . One the other hand , a perhaps more intuitive way taken from common Transformer-based ( Vaswani et al. , 2017 ) models would be to use cross-attention between all n previous documents , which is more expressive but prohibitively expensive when n is scaled upwards . Instead , we make a compromise between the above approaches , leveraging predicted edits ei−1i−n to compress the sequence and their derived spans ( as discussed above ) . Given each of these spans , we compute the edit-compressed sequence , composed of a sequence of vector representations with each vector representing a different span . For each span in each of the previous revisions in xi−1i−n ) , we mean pool the encoder ( pre-edit ) and the decoder ( post-edit ) representations for that span . We then sum this representation with the operation representing its edit operation and feed it into an MLP . Once we have done this for each span , we sum a learned relative edit positional embedding , where we learn an embedding matrix where each index in the matrix represents each order . This is to specify the order of the previous edits . Finally , we compose these into a sequence and treat that as the “ edit-compressed ” sequence representation for that edit . Turning Pre-trained Encoder-Decoder Models into Editors . Despite the fact that our model introduces both an edit prediction and a semi-autoregressive component , it is easy to finetune a pretrained language model into an editor with our method as it uses vanilla Transformer layers as a backbone . We perform this by batching various spans and their conditioning variables together and training the model to adapt to decode these in parallel .

Summary: Existing edit-based models condition on a single sentence or document. This paper argues that such frameworks do not represent processes well in which the same document undergoes multiple iterations of revisions, for example the edit histories of Wikipedia articles or source code. The proposed system is a two-stage model that first labels each token in the next revision with one of four edit labels, and then employs a seq2seq second stage model to predict the output tokens for generative labels such as INSERT and REPLACE. Both stages do not only condition on the previous revision but on the n latest revisions. Gains in perplexity are shown, as well as improvements of two downstream tasks: "edit-conditioned generation" (generating meta information to edits such as comments) and "edit-conditioned classification" (1-way semantic intent classification).