Learning to Model Editing Processes

1 INTRODUCTION . Revising and editing are a central part of the the human creative worflow , with most original content ( e.g . art , books , articles , source code ) being developed not in a single iteration , but in many iterations with each more refined than the last . How can we model these editing processes from inception to completion ? In this paper , we attempt to provide a first answer to this question , specifically focusing on generation of sequential data such as natural language documents or source code . Most current work on language generation tasks such as machine translation ( Vaswani et al. , 2017 ) , language modeling ( Baevski & Auli , 2018 ) , or summarization ( See et al. , 2017 ) generates the target sentence or document in a single pass ( usually from left to right ) . There has been a reasonable amount of work that can generate edits to existing sequences for the purposes of post-editing , grammatical error correction ( Omelianchuk et al. , 2020 ) , text style transfer ( Mallinson et al. , 2020 ; Malmi et al. , 2020 ; Reid & Zhong , 2021 ) , sentence fusion ( Malmi et al. , 2019 ) , or machine translation ( Gu et al. , 2019 ) . However , these works all 1 ) model only a single editing step and 2 ) do not fully define a model of incrementally editing a document from a blank slate to the final text , and thus do not stand in for the one-pass generative models of sequences described above . In this context , we propose the task of modeling editing processes in which we look to explicitly model the likelihood of the entire process of revising a document to a polished form . In particular , and in contrast to previous works on modeling edits , we hypothesize that in order to edit more accurately , instead of simply learning to predict the next revision given the current revision , we should have context of multiple previous revisions when deciding when and how to edit the document next . Given the novelty of this task , this paper simultaneously proposes 1 ) both baseline and novel models for the task , 2 ) evaluation methodology , and 3 ) evaluation datasets that can be used to compare model efficacy . The proposed multi-step editing model predicts discrete edit operations ( Levenshtein , 1966 ) to enable progressive refinement as shown in Figure 1 , rather than framing sequence editing as a sequence to sequence task ( Reid & Zhong , 2021 ; Faltings et al. , 2021 ) . In the figure , for each step of the editing process discrete operations ( insert , replace , delete , keep ) are predicted and then actions ( such as generating a replaced span ) are performed based on this . This has two benefits : 1 ) it allows the model to scale well with respect to input sequence length , and 2 ) allows us to make substantial changes with fewer actions ( Grangier & Auli , 2018 ) . We use these edit operations to condition a semi-autoregressive model that is able to insert and replace multiple spans at once . Combined with 1All data and code will be open-sourced upon acceptance . an encoder that is able to quickly specify which spans of text need to be changed and how , this allows for considerable changes to be made to the text ( including insertion , deletion , re-ordering , and replacement ) in a relatively simple and cheap manner . Furthermore , this allows us to disentangle how likely the model is to operate ( replace , delete , etc ... ) on a given span , and how likely the model thinks the generated text for a given span is . As we are modeling editing processes , and hypothesize that context from edits applied to the sequence are helpful , we propose a method for edit-aware sequence compression which can compress sequences into their edit operations and use relative edit positional embeddings to specify the position of edits relative to each other . To evaluate the efficacy of proposed and baseline models , we propose an evaluation metric , edit perplexity ( ePPL ) , which combines both of these ideas for a unified metric to measure edit likelihood . However we can easily disentangle this into operation perplexity ( oPPL ) and generative perplexity ( gPPL ) which measures the likelihood of performing a given operation on a span , versus the likelihood of newly generated text for the next revision . Given that the task of modeling natural editing processes in itself is novel , we collect new datasets to study this behavior ; WIKIREVISIONS and CODEREVISIONS . These datasets , in the code and natural language domains respectively , cover over 2.5M and 2.3M natural sequential revisions . We train and evaluate our proposed models on these datasets . Our experimental results demonstrate that edit perplexity , generative perplexity , and operation perplexity are all reduced when one models edits with previous changes/edits as context . We also qualitatively demonstrate the ability of the model to generate natural edit sequences , and demonstrate the utility of the learned representations on downstream tasks of commit message generation ( Loyola et al. , 2017 ) and edit intention classification ( Yang et al. , 2017 ) . 2 PROBLEM DEFINITION . Let X = { x0 , x1 , . . . , xN } be a series of N versions of a document , where the ith revised document is denoted by xi . x0 represents an initial state ( generally the null string ) , and xN represents the current state of the edited document . The probability of this series of document versions occurring can be decomposed as p ( X ) = N∏ i=1 p ( xi|xi−10 ) , ( 1 ) where xi−10 : = x0 , . . . , xi−1 ( similarly below ) . The right hand side is the likelihood of the transformation of the previous document version xi−1 to the current document version xi given the previous revision history x < i . We refer to the likelihood of the whole revision process as the edit likelihood , and judge learned models based on their ability to achieve high edit likelihood on held-out data . Note that standard generative models ( specifically language models ; LMs ) calculate the probability of only the final version p ( xN ) , whereas the proposed formulation calculates the probability of the entire sequence of document edits . It nonetheless could be used to calculate the final version ’ s likelihood by treating the editing process as latent and marginalizing over it p ( xN ) = ∑ { X̃=x̃N1 |x̃N=xN } p ( X̃ ) . ( 2 ) Thus our formulation , in contrast to previous single-step models of edits ( Yin et al. , 2019b ; Malmi et al. , 2019 ; Reid & Zhong , 2021 ) , can also be used to define a generative model over single documents . It is also worth noting that the final document likelihood is lower-bounded by the edit likelihood ; i.e . p ( xN ) ≥ p ( X ) . 3 MODELING EDITING PROCESSES . In this section , we now describe our approach to actually modeling these sequences of edits through ( 1 ) a decomposition of the modeling process into a sequential process of modeling edit operations then actual edits , and ( 2 ) neural model of modeling these operations and edits . 3.1 MODELING OPERATIONS AND OPERATION-CONDITIONED EDITS . While the probability p ( xi|xi−10 ) of the next document given all previous document versions could theoretically be modeled with a single neural sequence model , this is infeasible computationally ( and likely infeasible from learning perspective as well ) . To simplify this problem , we employ the n-th order Markov assumption , assuming that the probability of the next document is conditioned only on the previous n documents p ( xi|xi−1i−n ) . This probability could be modeled directly , and in fact in the case of n = 1 this becomes analogous to the single-step editing problem tackled by previous work ( Yin et al. , 2019b ; Malmi et al. , 2019 ; Reid & Zhong , 2021 ; Faltings et al. , 2021 ) . To our knowledge , no previous work has modeled natural editing processes with n > 1 . However , in the interest of both efficiency and efficacy , we take an alternative approach where we first predict a set of edit operations ei , and then predict the next document version based on the previous documents and these edit operations : p ( xi|xi−1i−n ) ≈ p ( xi , ei|x i−1 i−n ) ( 3 ) = p ( xi|ei , xi−1i−n ) p ( ei|x i−1 i−n ) . ( 4 ) The first approximation becomes an equality when the edit operations can be deterministically derived from xi and xi−1 , i.e . p ( ei|xi , xi−1 ) = 1 , as is the case described below . Edit Operations . We base the edit operations in e on those calculated by the Levenshtein algorithm ( Levenshtein , 1966 ) , including token-level insertions , deletions , and substitutions . These are expressed as four operations insert , delete , keep , and replace denoted by { INSERT , DELETE , KEEP , REPLACE } . For multi-word insertions and replacements , e.g . a replacement of a contiguous span of words , we apply the the same REPLACE label to all tokens in this span . An example of each operation is shown in Figure 1 . Decomposed Edit Likelihood . We can then re-define our previous formulation of edit likelihood : P ( xN1 ) = N∏ i=1 p ( xi|ei , xi−1i−n ) p ( ei|x i−1 i−n ) , ( 5 ) and analogously define edit log-likelihood Lxe : = logP ( xN1 ) = N∑ i=1 log p ( xi|ei , xi−1i−n ) + log p ( ei|x i−1 i−n ) . ( 6 ) We can further decompose this into only the components corresponding to the edit operations Le : =∑N i=1 log p ( ei|x i−1 i−n ) , or the operation-conditioned edits Lx|e : = ∑N i=1 log p ( xi|ei , x i−1 i−n ) , both of which we will utilize for devising evaluation metrics in Section 5.2 below . 3.2 EDITPRO . In this section , we propose a model of multi-step editing processes , EDITPRO , which is based on a semi-autoregressive edit-conditioned encoder-decoder model with a Transformer ( Vaswani et al. , 2017 ) . The model ( depicted in Figure 2 ) contains three main components : ( 1 ) an edit encoder , ( 2 ) an operation classifier and ( 3 ) an insertion-replacement decoder . Edit Encoder . The encoder fenc takes in a document version xi−1 and feeds it through multiple self-attention and feedforward layers ( Vaswani et al. , 2017 ) to produce contextual representations for each token . In the case that we perform variable-order edit modeling , we use cross-attention to feed in representations of previous edit steps . For models where n > 1 , we feed in n− 1 additional edit sequences – we describe this process after describing our methods for edit sequence prediction . Edit Operation Prediction . We use an autoregressive tagger , using a single Transformer layer with a causal attention mask , that models the probability of each edit in edit operation sequence e = eM1 from left to right , p ( ej |e j−1 1 ) . Notably , we also performed preliminary experiments with a tagger that predicts operations independently , but found it was heavily biased towards the KEEP operation as most words are kept in any single document revision , and thus did not produce coherent multi-word edit sequences when sampling sequences of edits . Generating Replacements and Insertions . When editing , given our four Levenshtein operations ( INSERT , REPLACE , KEEP , DELETE ) , two of them — INSERT and REPLACE— entail generation of new content conditioned on the current revision of the document . Given our predicted edit operations e , we propose a semi-autoregressive model with a Transformer decoder that can decode multiple spans in parallel for efficiency purposes . Each edit span contains the following properties : it has a start index ( denoted by sstart ) , end index ( denoted by send ) , and an operation type ( denoted by stype ) . Note that these can be simply be extracted by looking at contiguous spans of a certain type in an edit ( e.g . REPLACE for descended from→ domesticated descendant of in Figure 1 ) . We use a mean pooling operation to aggregate the contextual vectors produced by fenc ( x ) into span representation x̂s : x̂s = 1 send − sstart send∑ t=sstart fenc ( x ) t ( 7 ) We then update the span representation x̂s by taking the sum of the appropriate operation embedding for the span type and the current span representation and feed it to a multi-layer perceptron with an intermediate non-linearity : x̂s ← MLP ( Wop ( e ) s+x̂s ) , where Wop denotes an embedding matrix for each operation . x̂s is then used to initialize the < s > token for the decoder span to further condition the generative process . Encoding Edit History . As we look to investigate variable order edit modeling over long sequences of text , we need a way to be able to represent edits in a way useful for predicting the next editing steps . Previous work ( Yin et al. , 2019a ; Marrese-Taylor et al. , 2021 ; Yao et al. , 2021 ) has focused largely on learning a single-vector representation for edits which is compressed but limited in expres- siveness . One the other hand , a perhaps more intuitive way taken from common Transformer-based ( Vaswani et al. , 2017 ) models would be to use cross-attention between all n previous documents , which is more expressive but prohibitively expensive when n is scaled upwards . Instead , we make a compromise between the above approaches , leveraging predicted edits ei−1i−n to compress the sequence and their derived spans ( as discussed above ) . Given each of these spans , we compute the edit-compressed sequence , composed of a sequence of vector representations with each vector representing a different span . For each span in each of the previous revisions in xi−1i−n ) , we mean pool the encoder ( pre-edit ) and the decoder ( post-edit ) representations for that span . We then sum this representation with the operation representing its edit operation and feed it into an MLP . Once we have done this for each span , we sum a learned relative edit positional embedding , where we learn an embedding matrix where each index in the matrix represents each order . This is to specify the order of the previous edits . Finally , we compose these into a sequence and treat that as the “ edit-compressed ” sequence representation for that edit . Turning Pre-trained Encoder-Decoder Models into Editors . Despite the fact that our model introduces both an edit prediction and a semi-autoregressive component , it is easy to finetune a pretrained language model into an editor with our method as it uses vanilla Transformer layers as a backbone . We perform this by batching various spans and their conditioning variables together and training the model to adapt to decode these in parallel .

Summary: This paper proposes a new task of modeling editing processes to model the whole process of iteratively generating sequences. To tackle the new challenge, the authors propose a conceptual framework to describe the likelihood of multi-step edits and describe neural models that can learn a generative model of sequences based on these multi-step edits. The Experimental results show that modeling editing processes improve performance compared to previous single-step models of edits on related downstream tasks and the proposed task.