SlowGuess/NAACL_2021_OCR huggingface dataset:
Felix Faltings $^{\diamond, *}$  Michel Galley $^{\spadesuit}$  Gerald Hintz $^{\spadesuit}$  Chris Brockett $^{\spadesuit}$  Chris Quirk $^{\spadesuit}$  Jianfeng Gao $^{\spadesuit}$  Bill Dolan $^{\spadesuit}$
$\diamond$ Department of Computer Science, ETH Zürich  $\clubsuit$ Microsoft Research
fafelix@student.ethz.ch
{mgalley,gehint,chrisbkt,chrisq,jfgao,billdol}@microsoft.com
A prevailing paradigm in neural text generation is one-shot generation, where text is produced in a single step. The one-shot setting is inadequate, however, when the constraints the user wishes to impose on the generated text are dynamic, especially when authoring longer documents. We address this limitation with an interactive text generation setting in which the user interacts with the system by issuing commands to edit existing text. To this end, we propose a novel text editing task, and introduce WikiDocEdits, a dataset of single-sentence edits extracted from Wikipedia revision histories. We show that our Interactive Editor, a transformer-based model trained on this dataset, outperforms baselines and obtains positive results in both automatic and human evaluations. We present empirical and qualitative analyses of this model's performance. $^1$
A long-standing goal of natural language processing research has been to generate long-form text (Lebowitz, 1985; Fan et al., 2018; Rashkin et al., 2020). Recent large generative language models such as GPT-2 (Radford et al., 2019), and GPT-3 (Brown et al., 2020), demonstrate an impressive ability to generate fluent text, but their outputs are difficult to control beyond a prompt, and they manifest a tendency to hallucinate facts (Wiseman et al., 2017). Much recent work has thus focused on making such models more controllable (Keskar et al., 2019; Hu et al., 2017; Zhang et al., 2020; Dathathri et al., 2019), and factually grounded (Guu et al., 2020; Liu et al., 2018b).
Figure 1: An illustration of our interactive text generation setting. This is an example generated by our model. The blue panels represent the text being edited, taken from the document shown on the right. The orange panels represent user edit commands. The model grounds edits in query results from a commercial search engine.
Most such work only considers a one-shot generation setting. Given a set of inputs, which may be a prompt, a control code (Keskar et al., 2019), or a table of data (Liu et al., 2018b) for example, the system generates text in a single step. Humans, though, often produce text through an evolutionary process involving multiple draft-edit cycles. This is not simply because they make mistakes when writing, but because they may require multiple iterations to help them shape and even make sense of what they want to express (Pirolli and Card, 2005). For example, consider a user writing an article about Barack Obama. They might start with a simple sentence such as "Barack Obama was the 44th President of the United States". Next, they may wish to expand on that sentence, adding information, or rephrasing it to integrate it better with
* Work done at Microsoft Research.
<sup>1</sup>All our code (including code to recreate our data) and pre-trained models will be made available at: http://microsoft.com/research/project/interactive-document-generation
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5259-5274 June 6-11, 2021. ©2021 Association for Computational Linguistics
the text. Replicating this process in software will mean allowing users to adjust their requirements in response to model outputs. Even an error-free system that meets all of a user's initial requirements does not obviate the need for iteration, since those constraints are themselves dynamic. While this work focuses on text, we also note that these arguments extend to other settings where a system must generate a complex, structured object for a user, such as image or code generation.
The purpose of this paper is to bring into view the task of controllable text editing, as a step beyond one-shot generation towards interactive document generation. A full interactive document generation system will likely comprise multiple components, possibly including one-shot generation to create a first draft. Editing is crucial to interactivity because it allows users to change previously generated text to fit their dynamic constraints. This is a stateful operation, where the state is the current version of the document, as opposed to stateless recasting of text from scratch using a one-shot model. While services like Grammarly or MS Word already offer rewriting suggestions, they mainly focus on syntactic or stylistic edits such as paraphrases (Gupta et al., 2018). In this work, we are interested in a broader range of edits, particularly those that add or remove content, or change the meaning of text. Figure 1 illustrates this editing setting with an example from our trained model, where a user produces a sentence about Barack Obama over multiple edits.
In sum, we make the following contributions: We introduce a challenging new text editing task, wherein a model must learn to edit text in response to a user command, while drawing on grounding to avoid problems of hallucination (Wiseman et al., 2017). To accompany this task, we release an open-source dataset of sentence-level edits extracted from Wikipedia, including editor comments, which we leverage as natural language commands, together with pre-retrieved grounding documents. We show that a transformer-based editing model trained on our data outperforms "parrot" and GPT-2 baselines, and obtains competitive results compared to gold-standard edits in human evaluations. We then perform an empirical analysis of our model's performance, showing the importance of the command and grounding, and the varying difficulty of edits in our dataset.
2 Text Editing Task
We now formalize our text editing task. Let  $D$  be a document,  $q$  a user command $^2$ , and  $\mathcal{G}$  some appropriate form of grounding. Moreover, let  $D'$  be an edited version of  $D$ . Then our task is, given a dataset of edits  $\mathcal{D} = \{(D_0, q_0, \mathcal{G}_0, D_0'), \dots, (D_N, q_N, \mathcal{G}_N, D_N')\}$ , learn to produce document  $D'$ , given  $D, q$ , and  $\mathcal{G}$ .
Note that while previous work on text editing usually only considers  $D$  as input, we include both a form of control  $q$  and grounding  $\mathcal{G}$ . The command is needed because otherwise the type of edit to be made is undefined, while the grounding provides external knowledge needed to make an edit.
In our specific instance of this task, we will only consider sentence-level edits. More formally, we consider edits  $D \to D'$ , where  $D$  and  $D'$  differ only on a single sentence  $s \in D$ , respectively  $s' \in D'$ . While, in general, edits can vary in complexity from document-level to character-level changes, sentences are a natural way to break down text into relatively independent units of meaning, so it makes sense to edit text one sentence at a time. More complex, document-level edits can be seen as a composition of multiple sentence-level edits.
Additionally, we will consider user commands  $q$  written in natural language, e.g., "add years in office". The command could also take other forms, such as a categorical variable, but natural language allows for the greatest flexibility in specifying what the edit should accomplish. Moreover, natural language commands are a good fit for our model, which we will initialize with pretrained language model weights. For similar reasons, we will also consider corpora of text snippets as our grounding  $\mathcal{G}$ . Alternatively, the grounding could also consist of structured data such as tables or graphs. In a real user scenario, this grounding might be supplied by the user, or retrieved on the fly. For our dataset, we pre-retrieve groundings by querying a commercial search engine.
To accompany our text editing task we present a novel dataset of nearly 12 million sentence-level edits, WikiDocEdits. These edits were extracted from the revision histories in the February 1, 2020
2This notation reflects that the edit command is analogous to a query in a retrieval or QA setting in that it expresses a form of user intent.
dump of English Wikipedia.3
For a given Wikipedia page, a revision consists of a source and target text, corresponding to the old and new versions of the page. Each revision is also accompanied by an editor comment, which we will use as a proxy for the user command. For a given revision, we split the source and target texts into sentences and then attempt to match the sentences between source and target. For efficiency, we only look at a  $k$ -sentence neighborhood. Unmatched sentences are candidates for edits. A source sentence  $s$  and target sentence  $t$  form an edit pair  $s \rightarrow t$  if  $f(s,t) > \epsilon$ , where  $f$  is sentence-level BLEU<sup>4</sup> without smoothing and  $\epsilon = 0.1$  in our case. If an unmatched source sentence does not form an edit pair with any target sentence, we consider it to be a sentence deletion. This can also be thought of as matching to an empty sentence. We identify sentence insertions in an analogous manner. Importantly, we only consider revisions that contain a single sentence-level edit. Otherwise, the editor comment that accompanies each revision may only describe one of the possibly many sentence-level edits. See appendix A for a detailed description of our processing pipeline.
We retrieve grounding snippets for the edits in our dataset by querying a commercial search engine. In order to formulate a query for a given edit, we combine the relevant page and section titles with keywords from the target sentence. While the target sentence is not available at test time, we make the assumption that in a real user scenario the relevant grounding would be provided by the user.
We retrieve the top 200 returned web page results and only keep the preview snippets returned by the search engine as the grounding corpus.6
Because Wikipedia, as well as several clones, often appear in search engine results, we check for 4-gram overlap between the target sentence and each grounding snippet, removing any snippet with more than  $50\%$  overlap. Finally, we rerank<sup>7</sup> the retrieved snippets using an information extraction score, and merge the ranked snippets to take the first  $N = 512$  tokens.
Statistic	Percentiles	Mean
25%	50%	75%
Sentence length	16	23	31	25.25
Diff length	2	3	9	7.27
Comment length	2	3	7	5.20
Table 1: Summary statistics of WikiDocEdits. All statistics were computed on a  $1\%$  subsample of the data. Lengths reported in number of words. The diff length corresponds to the number of words, inserted or deleted, affected by a given edit.
3.2 Data Analysis
We now provide an overview of our dataset. From 667 dump files in the February  $1^{\text{st}}$  2020 dump of Wikipedia, we extract 11,850,786 edits, and take a  $1\%$  sample of 118,818 edits to run our analyses. Table 1 presents summary statistics for our data, and in the following, we break down the edits by edit type, and present some examples. See also appendix D for an analysis of the quality of the retrieved grounding.
Fluency and Content Edits We are interested in the distribution of different edit types within our dataset. In particular, we want to distinguish between fluency edits, which only affect the grammar or structure of a sentence, and content edits, which change the meaning of a sentence. We can lean on previous work to categorize edits on Wikipedia. Yang et al. (2017) create 13 edit intention categories, and train a classifier to label revisions according to the categories. We apply their classifier to our data, and group their 13 categories into "fluency", "content", or "other" edits, as reported in table 2. With the caveat that the edits were labelled automatically using a trained classifier, we see that, while fluency edits make up the majority of the edits in our data, a large proportion are content edits.
Examples Table 3 presents some examples from our data. These were chosen to illustrate a variety of edits. The first example shows an elaboration edit, appending new information to the end of a sentence. The second example is a simple typo fix, while the third is changing a fact. Finally, the last example is a more complex edit to reword a sentence. We can see that there is a large variety of edits in our dataset. See table 11 in the appendix for more examples.
3Downloadable from https://dumps.wikimedia.org/.
4We use BLEU-4 in all experiments of this paper.
See appendix B for how we identify keywords.
<sup>6</sup>We also experimented with retrieving and parsing the HTML pages from the search but this did not lead to better end-to-end performance than just using the snippets.
See appendix C for details on reranking.
Group	Labels	%
Fluency	Refactoring,
Wikification,
Point-of-view	57.00
Content	Fact-update,
Elaboration,
Clarification	24.77
Other	Unlabeled,
Counter-vandalism,
Vandalism,
Process,
Disam-biguation	26.65
Table 2: Breakdown of edits by grouped intention labels. See Table 10 in the appendix for a breakdown by intention label instead of group. The percentages do not total 100 because edits can have multiple labels.
Figure 2: An illustration of our model. The inputs to the encoder are sequences of tokens separated by  $\langle \mathrm{sep} \rangle$  tokens, represented by the vertical bars in the figure.
We formalize our model, which we refer to as Interactive Editor, as a standard auto-regressive sequence to sequence model. Because our data only contains single-sentence edits, we assume that the sentence to be edited in the source document is given as an input to the model.
Given a source sentence  $s \in D$ , the context around  $s$ , which we will refer to as  $D$  by abuse of notation, a user command  $q$ , a grounding corpus  $\mathcal{G}$ , and a candidate target sentence  $s'$ , the model,  $f$ , computes
$$
\begin{array}{l} f (s, s ^ {\prime}, D, q, \mathcal {G}) = P (s ^ {\prime} | s, D, q, \mathcal {G}) \\ = \prod_ {i} P (s _ {i} ^ {\prime} | s _ {<   i} ^ {\prime}, s, D, q, \mathcal {G}), \\ \end{array}
$$
where  $s_{<i}' = \{s_0', \dots, s_{i-1}'\}$  are the tokens preceding  $s_i'$  in  $s'$ .
We use the same encoder-decoder architecture as T5 (Raffel et al., 2020) and initialize our model with pretrained language model weights. The
encoder-decoder architecture allows us to perform full attention over the inputs  $s, D, q,$  and  $\mathcal{G}$ , while the decoder allows us to auto-regressively generate  $s'$ . Meanwhile, initializing with pretrained weights has been shown to achieve state-of-the-art results on many NLP tasks (Raffel et al., 2020).
In order to adapt T5 for our task, we represent all our inputs as sequences of tokens. We then concatenate these sequences together using separator tokens, truncating and padding them to fixed lengths. This is straightforward since all our inputs are text. See fig. 2 for reference. We also use the standard cross-entropy loss to train.
We train our model on a subset of  $\sim 1,020\mathrm{K}$  edits from WikiDocEdits. We use a training/validation/test split of  $1,000\mathrm{K} / 10\mathrm{K} / 10\mathrm{K}$  edits, and train for 3 epochs with a fixed learning rate of 0.0001, and a batch size of 128. We use the T5-base implementation from Huggingface (Wolf et al., 2020), and finetune all weights in the model. We validate every 200 steps and select the model with the lowest validation loss.
For inference we use beam search with a beam width of 5, and keep the 5 highest ranked candidates, excluding any generation that parrots the source as this corresponds to making no edits.
Metrics We consider several metrics to evaluate our model. One natural metric to consider is BLEU ((Papineni et al., 2002)). BLEU shows high correlation with human judgement on machine translation (Papineni et al., 2002; Doddington, 2002). While this should not a priori transfer to evaluating different tasks, our task in fact bears a high similarity to machine translation because of how the output is constrained by the inputs. If, for example, the source sentence in an English to German translation task is "Sally met Lucy", the German translation must in some way mention Sally and Lucy. Similarly, in our task, if the source sentence is "Barack Obama was the 44th President of the United States", and the command is "add birth date", the edit must somehow mention a birth date somewhere. Thus, in our setting, BLEU makes sense as a metric since in principle a good model output should not deviate too far from the reference. We use macro-averaged
Comment	added class of '13
Source	Krishna attended Dartmouth College where she was a double major in govern-ment and French.
Target	Krishna attended Dartmouth College where she was a double major in govern-ment and French and graduated in the class of '13.
Comment	sp
Source	Mountain State is currently seeing alternative accreditation by the Commis-sion on Collegiate Nursing Education.
Target	Mountain State is currently seeking alternative accreditation by the Com-mission on Collegiate Nursing Education.
Comment	correct year of marriage (did not fit NSW records)
Source	He married Margaret Frances Prowse Shaw in Sydney in 1874.
Target	He married Margaret Frances Prowse Shaw in Sydney in 1871.
Comment	Rephrasing
Source	Entitled "It Feels Like Home (Re Invented) Tour 2011", it contained his songs and remakes of Alliage hits.
Target	Entitled "It Feels Like Home (Re Invented) Tour 2011", it included many remakes of Alliage hits as well as some of his newer songs.
Table 3: Example edits from WikiDocEdits. The edited portions are highlighted in bold.
sentence-level BLEU with epsilon smoothing and equally weighted  $n$ -grams, with  $n$  up to 4.
One issue with BLEU is that the source and target sentences in our task are already very similar, so a model that simply parrots back the source sentence could achieve an unduly high score. Therefore, we also evaluate model outputs by comparing the word-level edits made by the model against the reference, where a word-level edit is a tuple of an operation, either insertion or deletion, a position, and a word. For example, in the edit "Barack Obama was the \(44^{\text{th}}\) President of the United States" \(\rightarrow\) "Barack Obama, born August \(4^{\text{th}}\) 1961, was the \(44^{\text{th}}\) President of the United States", the set of word edits would look like \(\{(\text{insert}, 2, "\), ), (\text{insert}, 3, "\text{born}"), \ldots\}\). Now, denote the set of word edits between two sentences \(a\) and \(b\) as \(\mathrm{WE}(a, b)\). Then, with \(s\) the source sentence, \(s'\) the reference target sentence and \(h\) the target sentence generated by the model, we compute the precision
$$
P _ {\mathrm {W E}} (s ^ {\prime}, h, s) = \frac {| \mathrm {W E} (s ^ {\prime} , s) \cap \mathrm {W E} (h , s) |}{| \mathrm {W E} (h , s) |},
$$
recall,
$$
R _ {\mathrm {W E}} (s ^ {\prime}, h, s) = \frac {| \mathrm {W E} (s ^ {\prime} , s) \cap \mathrm {W E} (h , s) |}{| \mathrm {W E} (s ^ {\prime} , s) |},
$$
and F1 score,
$$
F _ {1, \mathrm {W E}} (s ^ {\prime}, h, s) = 2 \cdot \frac {P _ {\mathrm {W E}} \cdot R _ {\mathrm {W E}}}{P _ {\mathrm {W E}} + R _ {\mathrm {W E}}}.
$$
Finally, we compute sentence-level accuracy, which reports the proportion of edits for which the model output exactly matched the reference.
Baselines We use two baselines to compare our model to. First, we consider the parrot baseline that simply outputs the source sentence as is. The second baseline attempts to delete the source sentence and replace it with a new sentence. We use a pretrained GPT-2 model (Radford et al., 2019) that generates a sentence given the left context.
Table 5 presents our main results. Notice that the parrot baseline is able to achieve a considerably high BLEU score, as expected, while the GPT-2 baseline surprisingly achieves a high word edit recall score. Our interactive neural editor model is able to beat both baselines across all metrics, as would be expected. Even on a harsh metric like accuracy our model achieves a nontrivial score, although we suspect most of the edits that the model gets exactly right are fluency edits. See table 6 for
Comment	Added more marriage info.
Reference edit	Johnson married Group 1 Crew member Manwell Reyes in 2011.
Johnson married Group 1 Crew member Manwell Reyes on June 11, 2011
in Half Moon Bay, California.
Model edit	Johnson married Group 1 Crew member Manwell Reyes in 2011.
Johnson married Group 1 Crew member Manwell Reyes in 2011 in a
ceremony at Half Moon Bay, California.
Comment	another minor addition
Reference edit	They are more frequent than primary brain tumors.
They are more frequent than primary brain tumors, and are mainly a
problem in adults, though children may also have secondary tumors.
Model edit	They are more frequent than primary brain tumors.
Secondary brain tumors are more frequent than primary brain tumors.
Table 4: Example outputs from Interactive Editor for two edits from the test data. The edit shown is the top-ranked generation from beam search, excluding the parrot generation if it occurs. The grounding and context are omitted here for brevity.
Model	Acc.	Word Edit	BLEU
R	P	F1
Baselines:
Parrot baseline	0	0	0	0	0.67
GPT-2	0	0.38	0.05	0.07	0.00
Ablations:
Only source	0.17	0.24	0.26	0.23	0.62
No command	0.20	0.28	0.31	0.28	0.65
No grounding	0.18	0.24	0.28	0.24	0.64
Our system:
Interactive	0.30	0.41	0.44	0.41	0.70
Table 5: Evaluation of our model (Interactive Editor) against baselines and ablations.
a breakdown by edit type, and table 4 for example model outputs.
Ablations The middle rows of Table 5 show the results for three ablations of our model. The first ablation removes everything but the source sentence  $s$ . This is similar to the paraphrase setting (Gupta et al., 2018), and the editing setting in Faruqui et al. (2018) and Yin et al. (2018). We can see that including the context, grounding, and command as additional inputs yields significant improvements over only using the source sentence. We can also see from the second ablation that the commands are a crucial element in the model's performance. This is not surprising since
without a command the model must guess what type of edit to make. Similarly, the model without grounding performs considerably worse than the full model, showing that the grounding is equally important as the command. Surprisingly, the last two ablations perform only marginally better than the first, meaning that removing the grounding in addition to the commands, or vice-versa, does not lead to a large drop in performance. This seems to suggest a synergistic effect between the command and the grounding, which makes sense since the model would not know what to do with the grounding without a command, and likewise, the model would not have access to the right information without the grounding, even if it knew what to edit from the command.
Breakdown by edit type The results of our full model are broken down by edit intention labels in Table 6. The columns report the same metrics as in our main table of results, with the exception of S-BLEU, which reports the BLEU score between the source sentence and target, and the last column, which reports the number of test edits that were classified into each category. With the caveat that intention labels come from an automatic classifier and not human annotation, we can observe that our model has varying performance across different types of edits. The model performs very well on fluency edits, but worse on content edits. This comes at no surprise given that fluency ed
Intention Category	Acc.	Word Edit	BLEU	S-BLEU	#Edits
P	R	F1
Fluency	0.36	0.49	0.47	0.46	0.76	0.73	6244
Content	0.10	0.24	0.19	0.20	0.42	0.38	2792
Other	0.29	0.45	0.41	0.41	0.74	0.72	3027
Table 6: Breakdown of results by intention category for our full model. The categories are the same as in table 2.
Task	Preference (%)
Reference	Neutral	Interactive
Command	41.00	31.71	27.29
Grounding	29.14	34.86	36.00