SlowGuess/NAACL_2022_OCR huggingface dataset:
For the T5 baselines, we find that adding evidence to the input results significant increase in all metrics, demonstrating that using the evidence is crucial to obtaining good performance.
EDiT5 obtains additional  $3 - 5\%$  absolute increase in all performance metrics compared to T5, establishing EDT5 as a strong baseline for future systems to be compared against. The reduction of unsupported entity tokens implies that EDT5 hallucinates less frequently than T5 models. Results are provided for different model sizes to illustrate how performance scales with parameter counts.
Example Output An example EDIT5 output is provided in Figure 4, and additional outputs in Appendix E. The examples illustrate important features of the task. In Figure 4 the goal is to update the Wikipedia article for Holli Sullivan to reflect her new role of Secretary of State of Indiana. In the reference, this information is reflected in an updated version of the first sentence as well as in a newly added last sentence. An additional sentence is added after the first sentence paraphrasing the introduction of the source article, which describes Sullivan's previous position as a member of the Indiana House of Representatives.
In the EDIT5 output for this example, information is only added at the end of the article. While the model correctly states that Sullivan was appointed to be Secretary of State by Governor Eric Holcomb, as well as includes additional context surrounding Sullivan's appointment that is paraphrased from the evidence, there are some issues with the output. First, because the first sentence of the article is not updated there is conflicting information about Sullivan's current position. Second, the added sentence hallucinates that Sullivan was appointed in January 2020 when she was actually
ROUGE
1	2	L
Copy Source	78.1	69.3	75.0
T5-Large	57.0	44.2	49.5
Edit5-Large	78.6	69.1	72.7
Table 5: ROUGE Scores Are Insensitive to Edits.
appointed in March 2021, a fact that directly appears in the evidence.
Categorizing Errors To better understand the types of errors made by EDIT5, we review a random sample of 100 of its predictions on the gold evaluation data and categorize them as either: grounded updates, meaning all generated claims are supported, ungrounded updates, meaning at least one unsupported claim appears in the output, or no updates, meaning the model did not predict any updates. For grounded updates we additionally keep track of how many updates include additional content not present in the ground truth update, or are missing content that appears in the ground truth update. For ungrounded updates we track whether an incorrect number/date appears in the update, the model distorted evidence, i.e., paraphrased or combined claims in the evidence in a way that changed their meaning, or hallucinated new claims unrelated to the evidence.
The results of this analysis are presented in Table 4b. We find that EDiT5 makes no mistakes on half of the examples, however a substantial portion of these updates had some issue with content selection. Of the incorrect updates, the most common mistake was incorrect numbers and dates, followed by hallucinations, and finally distorted evidence. This suggests that improving numeracy could be a fruitful line of study in future work on this task.
ROUGE is Problematic We provide ROUGE F-scores for each of the baseline models on the gold evaluation data in Table 5. In contrast to the previous results, we find that the simple copy source baseline attains a strong score of 77.4 despite making no updates. This is better than the T5 baseline results and comparable to the EDIT5 results. This illustrates the importance of evaluating on updates rather than the whole text.
Silver Data is Useful for Evaluation The results in Section 3.3 demonstrate high agreement between the silver and gold evaluation data which begs the question: can silver data be used in place
Holli Sullivan is an American politician who serves in the Indiana House of Representatives as a member of the Republican Party. In 2014 the district 78 seat for state Representative was vacated by Suzanne Crouch, who had been appointed state Auditor. ...Text omitted to save space... In 2017, she co-authored House Bill 1002, which provided for a long term plan for sustaining roads and bridges in Indiana including a phase-in shift of all gas tax to be dedicated to a dedicated infrastructure fund. That same session she authored a bill which created a strategic plan to reduce cervical cancer.
Secretary of State of Indiana List of Secretaries of State
#	Name	Took Office	Left Office
62	Holli Sullivan	March 16, 2021	-
Secretary of State of Indiana Introduction
The current office holder is Holli Sullivan, who was appointed by Governor Eric Holcomb to serve out the term of former Secretary of State Connie Lawson, who announced in February 2021 that she planned on resigning from office.
Holli Sullivan is an American politician who is the 62nd and current secretary of state of Indiana since March 2021. As a member of the Republican Party, she previously represented the 78th district in the Indiana House of Representatives from 2014 to 2021. ..Copied text... In 2021, Holli was named the 62nd Secretary of State of Indiana by Governor Eric Holcomb.
EDIT5
Copied text... In January 2020 Representative Sullivan was appointed by Governor Eric Holcomb to serve out the term of former Secretary of State Connie Lawson, who announced in February 2021 that she planned on resigning from office.
Figure 4: Example EDIT5 Output vs Ground Truth. Color coding indicates alignment between the new information and the edits. EDIT5 updates the original article by paraphrasing sentences from the textual evidence, however misses relevant information in the table, and generates an incorrect date.
UpdateROUGE	Entity	Unsup. tokens
1	2	L	Prec.	Rec.
100.0	100.0	94.3	75.4	92.8	92.8
of gold data for evaluation? To answer this, we measure the Spearman rank correlation between the gold baseline results in Table 4a and silver baseline results (provided in Table A2 of the Appendix to save space). Rank correlations for each of the metrics are shown in Table 6. Overall we find high rank correlation for each of the metrics, which suggests silver evaluation performance is a reliable indicator of gold performance. Thus, models whose pretraining data overlaps FRUIT-WIKI may be evaluated and compared on data produced by running our pipeline on future Wikipedia snapshots without requiring further human evaluation.
Controllability The improvement we obtained from EDiT5 over T5 implies that more controls can be added into the model. In this section we investigate whether additional control provided by the users can improve the overall generations. We follow Keskar et al. (2019) and Narayan et al. (2021), and provide more detailed instruction by adding control codes, i.e., special tokens, to the input that
Table 6: Spearman Rank Correlation Between Gold and Silver Performance Metrics.
UpdateROUGE	Entity	Unsup.
1	2	L	Prec.	Rec.	Tokens
Edit5	46.3	32.4	39.6	67.2	53.1	1.54
Control	57.6	42.1	50.2	70.5	64.5	2.42
Table 7: Controllability. Using control codes that indicate which sentences to delete, add or edit, and which evidence to use, can greatly improve generation.
instruct the model whether to add, copy, edit or remove a sentence, as well as which evidence to use when making an addition or edit. We use the target text to provide oracle labels for the control code, and see if the Edit5 can take advantage of the codes. Example inputs and predictions are provided in Figure A6 of the Appendix.
Results on the gold evaluation data are provided in Table 7. Including oracle control codes in the input produces a substantial  $10\%$  absolute improvement in all metrics besides unsupported tokens. This demonstrates that increased user control has the potential to produce updates that more closely resemble the desired output.
Early work on writing assistants largely focuses on grammar error correction; for a survey see Wang et al. (2020). Neural models have expanded the capabilities of writing assistants to solve a wider
variety of tasks including: autocompletion (Chen et al., 2019), and following rhetorical directives such as paraphrasing, elaborating, etc. (Sun et al., 2021). In this work, we seek to expand these capabilities further to producing grounded updates, which has been previously studied by Kang et al. (2019), however only for post-modifier generation.
As our primary focus is on writing grounded updates to Wikipedia articles, our work is closely related to existing works on Wikipedia article generation, which generally uses one of two approaches: data-to-text generation (Lebret et al., 2016; Bao et al., 2018; Parikh et al., 2020; Chen et al., 2021; Cheng et al., 2020), or multi-document summarization (Banerjee and Mitra, 2016; Liu et al., 2018; Shi et al., 2021). In particular, the hyperlink-based approach for associating evidence to articles is directly inspired by these works, and our annotation procedure for removing unsupported text directly draws from Parikh et al. (2020).
Determining which facts contradict claims in the existing article is a central topic of work on fact extraction and verification (Thorne et al., 2018). Recently, Schuster et al. (2021) introduced the VITAMIN-C dataset of factual revisions to Wikipedia articles and the task of factually consistent generation. This work differs from FRUIT in that it only focuses on sentences and does not require adding new facts or content selection.
Our work is also related to the TAC 2008 Update Summarization Task (Dang and Owczarzak, 2008), which involves summarizing information about a topic that does not overlap with an existing summary, instead of updating an existing summary to reflect new information.
In this work we introduced FRUIT, a novel text generation task where the goal is to update an article to reflect new information about its subject. To enable research on this task, we formulated a pipeline for extracting weakly supervised training and evaluation data from pairs of Wikipedia snapshots, and collected data for the years 2019-2020 and 2020-2021, as well as human annotated gold evaluation data. We additionally provided results for several strong baselines, that demonstrate both the feasibility of this task, as well as strong correlation between gold and distantly supervised data evaluation performance that establishes the trustworthiness of future data produced using our pipeline.
This work lays the foundation for future research into making faithful updates to entries in textual knowledge bases. One limitation of this work is that the metrics we use to evaluate faithfulness are all entity-centric, and thus may overstate the performance of models whose edits include the correct entities but misspecify the relations between them or other facets of the evidence. Accordingly, one promising direction for future work on this task is to develop more robust metrics for measuring faithfulness.
An additional promising direction for future work is to consider open settings of evidence collection, where other forms of updated information such as excerpts from news articles could be used to justify edits in place of updates to other entries in the same knowledge base. Relatedly, we also recommend studying this task in streaming settings, where updates arrive in sequential fashion, in addition to the batch setting considered in this work.
Finally, there are a number of promising directions for improving model performance on this task. In particular, copy mechanisms (See et al., 2017) have been widely used in data-to-text tasks (Wiseman et al., 2017), and may help mitigate issues such as the mistranscribed dates we saw in Section 5.
We would like to thank Kelvin Guu, Kenton Lee, Kristina Toutanova, and the anonymous reviewers for their helpful feedback. This work is funded in part by the DARPA MCS program under Contract No. N660011924033. Additionally, Robert is supported in part by the Irvine Initiative in AI, Law, and Society fellowship.
This paper introduces a dataset and system for updating an existing piece of text to incorporate information from external evidence. Depending on the veracity of the external evidence, systems for solving this task could potentially be abused by bad actors to spread misinformation.
Sumit Asthana and Aaron Halfaker. 2018. With few eyes, all hoaxes are deep. Proc. ACM Hum.-Comput. Interact., 2(CSCW).
Siddhartha Banerjee and Prasenjit Mitra. 2016. Wiki-write: Generating wikipedia articles automatically. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 2740-2746. IJCAI/AAAI Press.
Junwei Bao, Duyu Tang, Nan Duan, Zhao Yan, Yuhua Lv, Ming Zhou, and Tiejun Zhao. 2018. Table-to-text: Describing table region with natural language. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5020-5027. AAAI Press.
Mia Xu Chen, Benjamin N. Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yi-nan Wang, Andrew M. Dai, Zhifeng Chen, Timothy Sohn, and Yonghui Wu. 2019. Gmail smart compose: Real-time assisted writing. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pages 2287-2295. ACM.
Mingda Chen, Sam Wiseman, and Kevin Gimpel. 2021. WikiTableT: A large-scale data-to-text dataset for generating Wikipedia article sections. In Findings of the Association for Computational Linguistics: ACLIJCNLP 2021, pages 193-209, Online. Association for Computational Linguistics.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dólár, and C. Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. ArXiv, abs/1504.00325.
Liying Cheng, Dekun Wu, Lidong Bing, Yan Zhang, Zhanming Jie, Wei Lu, and Luo Si. 2020. ENT-DESC: Entity description generation by exploring knowledge graph. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1187-1197, Online. Association for Computational Linguistics.
Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the tac 2008 update summarization task. Theory and Applications of Categories.
Johannes Daxenberger and Iryna Gurevych. 2012. A corpus-based study of edit categories in featured and non-featured Wikipedia articles. In Proceedings of COLING 2012, pages 711-726, Mumbai, India. The COLING 2012 Organizing Committee.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrievalaugmented language model pre-training.
Jun Seok Kang, Robert Logan, Zewei Chu, Yang Chen, Dheeru Dua, Kevin Gimpel, Sameer Singh, and Niranjan Balasubramanian. 2019. PoMo: Generating entity-specific post-modifiers in context. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 826-838, Minneapolis, Minnesota. Association for Computational Linguistics.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. ArXiv, abs/1909.05858.
Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4940-4957, Online. Association for Computational Linguistics.
Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203-1213, Austin, Texas. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052-7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Shashi Narayan, Yao Zhao, Joshua Maynez, Gonccalo Simoes, and Ryan T. McDonald. 2021. Planning with entity chains for abstractive summarization. ArXiv, abs/2104.07606.
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and
Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173-1186, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Sascha Rothe, Joshua Maynez, and Shashi Narayan. 2021. A thorough evaluation of task-specific pretraining for summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 140-145, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624-643, Online. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4603-4611. PMLR.
Weijia Shi, Mandar Joshi, and Luke Zettlemoyer. 2021. DESCGEN: A distantly supervised dataset for generating entity descriptions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 415-427, Online. Association for Computational Linguistics.
Simeng Sun, Wenlong Zhao, Varun Manjunatha, Rajiv Jain, Vlad Morariu, Franck Dernoncourt, Balaji Vasan Srinivasan, and Mohit Iyyer. 2021. IGA: An intent-guided authoring assistant. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5972-5985, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks.
In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERIFICATION. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Yu Wang, Yuelin Wang, Jie Liu, and Zhuo Liu. 2020. A comprehensive survey of grammar error correction. ArXiv, abs/2005.06600.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. ArXiv, abs/2006.14799.
We perform an ablation study to measure the impact of the modifications made to the target output of EDiT5. The results are provided in Table A1. We observe that both the diff format and including reference tokens have a positive impact on the evaluation metrics, with reference tokens having the larger impact.
UpdateROUGE	Entity	Unsupp.
1	2	L	Prec.	Rec.	Tokens
EDIT5	46.3	32.4	39.6	67.2	53.1	1.54
- Diff	45.5	31.7	39.1	66.8	50.8	1.66
- Ref.	45.1	31.6	38.8	66.3	50.7	1.89
B Model Training Details
Optimizer: AdaFactor (Shazeer and Stern, 2018), Batch Size: 128, Learning Rate: 1e-3, Dropout Rate: 0.1, Training Iterations: 30,000. Training performed on a cluster of 16 2nd generation TPUs for  $< 3\mathrm{B}$  param models, and 32 TPUS for 3B parameter models.
C Silver Baseline Results
Table A1: Edit5 Ablations.
UpdateROUGE	Target Entity	Evid.
1	2	L	P	R	Acc
T5-Large + Evid.	26.8	15.9	22.3	56.3	29.8	2.33
39.2	27.3	34.2	66.9	42.4	1.63
Edit5
Small	37.8	24.9	32.6	61.4	41.2	1.53
Base	42.8	28.7	36.4	60.5	49.2	2.32
Large	42.7	29.9	37.2	66.1	47.5	1.47
3B	43.8	31.5	38.6	68.4	48.6	1.53
Table A2: Baseline Results on Silver Evaluation Data.
D Input and Output Formats