SlowGuess/NAACL_2021_OCR huggingface dataset:
its should be easier as they usually correct minor mistakes, which a language model should be able to detect from pretraining. Content edits, on the other hand, require pulling the correct information from the grounding and incorporating it in the correct manner into the sentence. The S-BLEU scores confirm this since the source sentences in the fluency examples are much more similar to the target sentences than for the content edits. In fact, when looking at the absolute improvement of the BLEU over the S-BLEU scores, the model performs equally well on both types of edits.
We conducted two rounds of human evaluations, each time across 200 examples from our test set. Annotators were crowd sourced, and each example was rated by seven judges for a total of 1400 judgements.[8]
Command and Grounding In our first round of human evaluations we compared our model's top output from beam search to the reference edit. There were two tasks. In the first task, we asked judges to choose which system better accomplished the command  $q$ . In the second, we asked which system was more faithful to the grounding  $\mathcal{G}$ . Table 7 presents the results. Although there is a clear preference for the Reference edits in the command-related task,  $59\%$  of judgments suggest that Interactive Editor may be equal to or better
Table 7: Human Evaluation: judging preferences for our system (Interactive Editor) vs. human references.
System A	System B
Full +	3.45	2.55	Ablated +
Full -	3.33	3.12	Ablated -
Full +	3.45	3.33	Full -
Ablated -	3.12	2.55	Ablated +
Table 8: Human Evaluation: comparisons between absolute evaluations of different settings. Raters were asked whether edits were satisfactory. 0 corresponds to strong disagreement, and 5 to strong agreement. Systems are given by model (full or with the comment ablated), and whether the command was shown to the raters (+ or -). Bolded numbers indicate significant difference with  $p < 0.0125$ .
than the reference. In the grounding task, Interactive Editor demonstrates good correspondence with the background material.10 Judges were further asked whether the retrieved grounding was relevant to the context  $D$ :  $92.86\%$  of judgments recorded the grounding as either "Somewhat relevant" or "Very relevant".
Absolute Scoring We also evaluated the overall quality of model outputs. We considered our full model, and our ablated model that only takes the source sentence as input. We also considered showing and hiding the edit commands, for a total of 4 settings. For a given setting, raters were asked whether they found each of the top 3 model outputs satisfactory. Table 8 presents the results for the top model outputs, with bootstrapped p-values for pairwise comparisons. We use a Bonferroni corrected  $\alpha = 0.0125$  to determine significance. Note that our full model outperforms our ablated model in the first two comparisons. Inter
The high percentage of Neutral judgments here may be partially attributable to other factors. Majority Neutral judgments are observed for approximately  $65\%$  of those examples that received at least one Neutral judgment. This suggests many commands may not be readily interpretable to judges.
10 Appendix E presents some additional automatic metrics to measure the faithfulness of the model to the grounding.
The annotators were remunerated at a rate above the prevailing Seattle minimum wage at the time.
estingly, the difference is smaller when the raters are not shown the commands. Additionally, only the ablated model is rated differently depending on whether the commands are shown. This is to be expected since the ablated model is not likely to be faithful to the commands. In addition to reporting the mean scores from the raters, we can also look at the number of examples where at least one of the top model outputs was found satisfactory by human judges (i.e. scored higher than 3). We find that, when showing the edit commands, at least one of the outputs from our full model was satisfactory in  $85.83\%$  of cases versus  $60.17\%$  for the ablated model.
Text	Geoff Hinton is an English tennis player.
Command	fix profession
Text	Geoffrey Hinton is a computer science professor at the University of Toronto.
Command	add nationality
Text	Geoffrey Hinton is an English-Canadian computer science professor at the University of Toronto.
Command	add birthdate
Text	Geoffrey Hinton (born 1946) is an English-Canadian computer science professor at the University of Toronto.
Command	add most famous work
Text	Geoffrey Hinton (born 1946) is an English-Canadian computer science professor at the University of Toronto. Geof-frey Hinton is most famous for his work on artificial neural networks.
Table 9: An example of a multi-turn interaction with our model. At each turn, the edit was chosen among the top 3 outputs returned by beam-search. See table 12 in the appendix for the grounding used in this example.
This paper focuses on the task of editing individual sentences, which we believe to be a challenging task for NLP, as it involves making nuanced changes to text according to natural language commands. We also believe this task has
useful applications, particularly in speech-to-text scenarios, where it may be more convenient to speak out a command rather than edit the text directly. However, we also wish to emphasize that this task is a step towards a larger goal of interactive document generation, and that there are many interesting future directions to explore in this space. While this paper has focused on single interactions (i.e. making isolated edits to text), it would be worth modeling multiple interactions between the user and model. One can imagine that there may be a natural order in which to make edits, such as adding information at the start, and fine-tuning the language at the end. It is an open question whether or not a model could learn this. For illustration, table 9 gives an example of using our model to make several edits in order to create a sentence. Ultimately, this may look more like a dialogue than a sequence of commands coming from the user. Additionally, it would also be interesting to look at other settings where a model must generate a complex, structured object for a user, such as code, or images. We hope that our text editing task, as a first step, can demonstrate the potential for interactive generation systems, and that it will encourage the community to pursue more ideas in this space.
Grounded Generation Large language models can generate fluent text (Radford et al., 2019; Brown et al., 2020; Raffel et al., 2020), but they have a tendency to hallucinate facts (Wiseman et al., 2017). Thus, several works have explored using various forms of grounding to enable models to generate factually consistent texts (Koncel-Kedziorski et al., 2019; Liu et al., 2018b; Prabhumoye et al., 2019; Liu et al., 2018a; Guu et al., 2020). Our work uses grounding to ensure that edits are factually correct, although our task differs from previous work because of the user command, which requires specific information to be retrieved from the grounding during generation.
Controllable Generation While grounding can be seen as a way to implicitly control the contents of generated text, other works have explored more explicit forms of control. Hokamp and Liu (2017) and Zhang et al. (2020) use lexical constraints, while Keskar et al. (2019) and Dathathri et al. (2019) control higher level attributes of text, such as style, tone, or topic. Our task instead
uses natural language commands, which can flexibly express different types of constraints, ranging from low-level lexical ones, to high-level topical ones. In this sense, we can also draw the parallel to dialog response generation (Ghazvininejad et al., 2018; Dinan et al., 2018), task-oriented dialog (Gao et al., 2018), or open domain question answering (Min et al., 2019; Chen et al., 2017), that also involve user responses or queries, although these tasks are not concerned with text generation in the context of document creation.
Story Generation The task of Document Generation considered in our work bears similarity with work on generating long-form narratives (Jain et al., 2017). While earlier work in Story Generation focused more on plan-based architectures (Lebowitz, 1985), more recent work moved towards end-to-end approaches (Fan et al., 2018) allowing generation to be unconstrained and creative. As narratives are often aimed at particular goals expressed in terms of outlines and plans, much of the literature in Story Generation is framed as a form of controllable generation, using storylines (Peng et al., 2018), events (Martin et al., 2017; Harrison et al., 2017), plot words or word skeletons (Xu et al., 2018; Ippolito et al., 2019), plans (Yao et al., 2019), story ending (Tambwekar et al., 2019), and outlines (Rashkin et al., 2020) as various forms of constraints. Our work takes a significantly different approach, as we treat document or story generation as an iterative process that allows a human to generate a full document from scratch, but also allows constraints to be more dynamic (e.g., add nationality in Table 9 only if the system missed that the first time).
Text Editing Several previous works have focused on text editing. Guu et al. (2018) generate sentences by editing prototypes taken from their training corpus, although they use editing only as a means for language modeling. Wu et al. (2019) expand upon Guu et al. (2018)'s setting, but for dialog. More related to our own setting, Faruqui et al. (2018) propose WikiAtomicEdits, a dataset of edits crawled from Wikipedia. However, they consider a much narrower definition of edits than our data does. Yin et al. (2018) use WikiAtomicEdits and propose the task of learning to represent edits, which Marrese-Taylor et al. (2020) expand using a variational approach. In contrast, we are more interested in generating edits rather than repre
senting them. Related to Wikipedia data, Pryzant et al. (2020) also used Wikipedia revision histories to learn to debias text, whereas we considered general edits. Iso et al. (2020) propose a fact-based text editing task, but they do not consider control or other types of edits. Another related task to text editing is text paraphrasing (Gupta et al., 2018), however paraphrasing usually conserves the meaning of a sentence. While the edits we consider include meaning-preserving edits, we are mostly interested in edits that affect meaning.
In this work we argued that text generation should be interactive, and, as a means towards that end, we proposed a general text editing task, where a system must edit a document in response to a user command. In our specific instance of the task we considered single-sentence edits, and we crawled a dataset of several million edits from Wikipedia that included commands, in the form of editor comments, as well as grounding documents. We then showed that training a transformer-based model on our data, while initializing with pretrained language model weights, yields encouraging results on both automatic and human evaluations. Additionally, our ablation studies showed the crucial role played by the user command and grounding. Breaking down our results by types of edits, we saw that our model not only performs well on easier fluency edits, but also on much harder content edits. Finally, we discussed future research directions for interactive document generation, as well as possible extensions to other domains such as images or code.
The authors would like to thank Thomas Hofmann, as well as Sudha Rao, Matt Richardson, Zhang Li, Kosh Narayanan, and Chandra Chikkareddy for their helpful suggestions.
Giuseppe Attardi. 2015. WikiExtractor. https://github.com/attardi/wikiextractor.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc.
T. Brown, B. Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, P. Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, G. Krüger, Tom Henighan, R. Child, Aditya Ramesh, D. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, E. Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, J. Clark, Christopher Berner, Sam McCandlish, A. Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. ArXiv, abs/2005.14165.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proc. of ACL.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. In Proc. of ICLR.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. In Proc. of ICLR.
George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram cooccurrence statistics. In Proceedings of the second international conference on Human Language Technology Research, pages 138-145.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proc. of ACL, pages 889-898.
Manaal Faruqui, Ellie Pavlick, Ian Tenney, and Dipanjan Das. 2018. Wikipatomic edits: A multilingual corpus of wikipedia edits for modeling language and discourse. In Proc. of EMNLP, pages 305-315.
Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural approaches to conversational ai. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1371-1374.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, W. Dolan, Jianfeng Gao, Wen tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Proc. of AAAI.
A. Gupta, A. Agarwal, Prawaan Singh, and P. Rai. 2018. A deep generative framework for paraphrase generation. In Proc. of AAAI.
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-augmented language model pre-training. ArXiv, abs/2002.08909.
B. Harrison, Christopher Purdy, and Mark O. Riedl. 2017. Toward automated story generation with markov chain monte carlo methods and deep neural networks. In AIIDE Workshops.
Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid beam search. In Proc. of ACL, pages 1535-1546.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In Proc. of ICML, pages 1587-1596.
Daphne Ippolito, David Grangier, Chris Callison-Burch, and Douglas Eck. 2019. Unsupervised hierarchical story infilling. In Proceedings of the First Workshop on Narrative Understanding.
Hayate Iso, Chao Qiao, and Hang Li. 2020. Fact-based text editing. In Proc. of ACL, pages 171-182.
Parag Jain, Priyanka Agrawal, Abhijit Mishra, Mohak Sukhwani, Anirban Laha, and Karthik Sankaranayanan. 2017. Story generation from sequence of independent short descriptions. arXiv preprint arXiv:1707.05501.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. ArXiv, abs/1909.05858.
T. Kiss and J. Strunk. 2006. Unsupervised multilingual sentence boundary detection. Computational Linguistics, 32:485-525.
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text generation from knowledge graphs with graph transformers. In Proc. of NAACL-HLT.
M. Lebowitz. 1985. Story-telling as planning and learning. Poetics, 14:483-502.
Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018a. Generating wikipedia by summarizing long sequences. In Proc. of ICLR.
T. Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Z. Sui. 2018b. Table-to-text generation by structure-aware seq2seq learning. In Proc. of AAAI.
Edison Marrese-Taylor, Machel Reid, and Y. Matsuo. 2020. Variational inference for learning representations of natural language edits. ArXiv, abs/2004.09143.
Lara J Martin, Prithviraj Ammanabrolu, Xinyu Wang, William Hancock, Shruti Singh, Brent Harrison, and Mark O Riedl. 2017. Event representations for automated story generation with deep neural nets. arXiv preprint arXiv:1706.01331.
Sewon Min, Danqi Chen, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019. Knowledge guided text retrieval and reading for open domain question answering. ArXiv, abs/1911.03868.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. of ACL.
Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. 2018. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pages 43-49.
Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis.
Shrimai Prabhumoye, Chris Quirk, and Michel Galley. 2019. Towards content transfer through grounded text generation. In Proc. of NAACL-HLT, pages 2622-2632.
Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. 2020. Automatically neutralizing subjective bias in text. In Proc. of AAAI, volume 34, pages 480-489.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, Open AI.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. PlotMachines: Outline-conditioned generation with dynamic plot state tracking. In Proc. of EMNLP.
Pradyumna Tambwekar, Murtaza Dhuliawala, Lara J Martin, Animesh Mehta, Brent Harrison, and Mark O Riedl. 2019. Controllable neural story plot generation via reward shaping. In Proc. of IJCAI, pages 5982-5988. AAAI Press.
Sam Joshua Wiseman, Stuart Merrill Shieber, and Alexander Sasha Matthew Rush. 2017. Challenges in data-to-document generation. In Proc. of EMNLP. Association for Computational Linguistics.
Thomas Wolf, Julien Chaumont, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, et al. 2020. Transformers: State-of-the-art natural language processing. In Proc. of EMNLP, pages 38-45.
Yu Wu, Furu Wei, Shaohan Huang, Yunli Wang, Zhoujun Li, and Ming Zhou. 2019. Response generation by context-aware prototype editing. In Proc. of AAAI, volume 33, pages 7281-7288.
Jingjing Xu, Xuancheng Ren, Yi Zhang, Qi Zeng, Xiaoyan Cai, and Xu Sun. 2018. A skeleton-based model for promoting coherence among sentences in narrative story generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4306-4315, Brussels, Belgium. Association for Computational Linguistics.
Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2017. Identifying semantic edit intentions from revisions in wikipedia. In Proc. of EMNLP, pages 2000-2010.
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Proc. of AAAI, volume 33, pages 7378-7385.
Pengcheng Yin, Graham Neubig, Miltiadis Allamanis, Marc Brockschmidt, and Alexander L Gaunt. 2018. Learning to represent edits. In Proc. of ICLR.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019a. Bertscore: Evaluating text generation with bert. In Proc. of ICLR.
Xizhe Zhang, Dheeraj Rajagopal, Michael Gamon, Sujay Kumar Jauhar, and Chang-Tien Lu. 2019b. Modeling the relationship between user comments and edits in document revision. In Proc. of EMNLP-IJCNLP.
Yizhe Zhang, G. Wang, C. Li, Zhe Gan, Chris Brockett, and B. Dolan. 2020. Pointer: Constrained text generation via insertion-based generative pre-training. ArXiv, abs/2005.00558.
A Data Processing pipeline
This section describes our pipeline to obtain atomic edits from Wikipedia revisions in more detail. We start by filtering the revisions in the data. In particular, following (Zhang et al., 2019b), we only keep revisions that affect a single section, and we exclude revisions that do not contain an editor comment. We also exclude certain page types like talk or user pages.
We then strip the Wikipedia markup in the retrieved text, using the WikiExtractor script (Attardi, 2015). This removes most markup and Wikimedia templates from the text. Because the markup language used on Wikipedia is not completely formalized<sup>11</sup>, and because malformed markup often appears in intermediate versions of Wikipedia pages, there is no guarantee that we can remove all the markup from the text.
We then split each section into sentences using the Punkt sentence tokenizer (Kiss and Strunk, 2006) provided in the NLTK python package (Bird et al., 2009).
After splitting into sentences, we attempt to match the sentences from the pre-edit (source) document to the sentences in the post-edit (target) document. Unmatched sentences will be candidates for edits. Similarly to (Faruqui et al., 2018), for each sentence  $s_i$  in the source document, we only look at the target sentences  $\{t_{i-k}, \dots, t_i, \dots, t_{i+k}\}$ , with  $k = 20$ . This avoids the quadratic complexity of looking at all matches.
We then filter out revisions that contain more than one sentence-level edit to ensure that the comment is relevant. If there is a single unmatched source, respectively target, sentence, we consider it a sentence deletion, respectively insertion. Because we do not look at all matches between source and target sentences, a sentence may remain unmatched if, in the target document, it was moved more than  $k$  sentences away compared to the source document. Thus we only keep a sentence insertion or deletion if the total number of source and target sentences differ by one. If there are both an unmatched source sentence  $s$  and target sentence  $t$ , we consider them to form an edit  $s \rightarrow t$  if  $f(s,t) > \epsilon$ , where  $f$  is the BLEU score and  $\epsilon = 0.1$ .
As a final step, we filter out edits that involve sentences with markup punctuation. We have
found that this helps remedy the shortfalls of the markup removal step, since it often leaves behind markup symbols. While there may be valid sentences that use markup punctuation, we do not expect them to make up a significant part of the data, nor do we expect them to be significantly different from regular sentences, except for their use of unusual punctuation.
B Grounding Search Query Construction
For a given edit, we combine the relevant page and section titles with keywords from the target sentence to construct a query that we use to retrieve grounding from a commercial search engine. In order to identify keywords we look at document frequency
$$
\operatorname {d f} (w) = \frac {| \{D \in \mathcal {D} \mid w \in D \} |}{| \mathcal {D} |},
$$
where  $\mathcal{D}$  is a sample of 500,000 Wikipedia articles taken from the TensorFlow Wikipedia dataset.12 We consider words  $w$  with  $\mathrm{df}(w) < 0.01$  to be keywords.
C Grounding Document Reranking
Because the combined length of the grounding snippets we retrieve far exceeds the capacity of our model, we rerank the retrieved snippets using an information extraction score. We then merge the ranked snippets and take only the first  $N = 512$  tokens. Following (Liu et al., 2018a) we use tfidf scores to rerank. For a given edit  $s \to s'$ , with retrieved grounding documents  $\mathcal{G}$ , the information extraction score of snippet  $G \in \mathcal{G}$  is
$$
\operatorname {s c o r e} (G) = \sum_ {w \in s ^ {\prime}} \operatorname {t f - i d f} (w, G),
$$
where the tfidf score of word  $w$  is