SlowGuess/NAACL_2022_OCR huggingface dataset:
Robert L. Logan IV*†1 Alexandre Passos²
Sameer Singh $^{1}$  Ming-Wei Chang $^{\dagger,2}$
<sup>1</sup>University of California, Irvine <sup>2</sup>Google Research
{rlogan,sameer}@uci.edu
{apassos,mingweichang}@google.com
Textual knowledge bases such as Wikipedia require considerable effort to keep up to date and consistent. While automated writing assistants could potentially ease this burden, the problem of suggesting edits grounded in external knowledge has been under-explored. In this paper, we introduce the novel generation task of faithfully reflecting updated information in text (FRUIT) where the goal is to update an existing article given new evidence. We release the FRUIT-WIKI dataset, a collection of over 170K distantly supervised data produced from pairs of Wikipedia snapshots, along with our data generation pipeline and a gold evaluation set of 914 instances whose edits are guaranteed to be supported by the evidence. We provide benchmark results for popular generation systems as well as EDIT5—a T5-based approach tailored to editing we introduce that establishes the state of the art. Our analysis shows that developing models that can update articles faithfully requires new capabilities for neural generation models, and opens doors to many new applications.<sup>1</sup>
Information changes on a constant basis. Every day, athletes are traded to new teams, and musicians and actors produce new albums and TV shows. Maintaining textual knowledge bases to keep track of these changes requires considerable community effort. For instance, a team of 120K volunteer editors make 120 edits to English Wikipedia every minute, and write 600 new articles a day. As the knowledge base grows, the amount of maintenance effort is compounded by the need to keep the knowledge
base consistent; e.g., each edit may render information in one of the existing  $6.3\mathrm{M}+$  articles obsolete.
Assistive writing technologies have the potential to substantially reduce the burden of keeping text corpora up to date and consistent. However, existing work has mainly focused on correcting grammar (Wang et al., 2020), reducing repetitive typing (Chen et al., 2019), and following rhetorical directives (Sun et al., 2021), whereas the problem of producing edits grounded in external knowledge has received little attention (Kang et al., 2019). In contrast, numerous works have developed systems for distilling external knowledge into text (e.g., Wikipedia article generation) by treating the problem as multi-document summarization (Liu et al., 2018; Shi et al., 2021) or data-to-text generation (Bao et al., 2018; Parikh et al., 2020). However, these systems are not useful for updating existing texts as they can only generate text from scratch.
To help endow writing assistants with grounded editing capabilities, we introduce the novel generation task of faithfully reflecting updated information in text (FRUIT), where the goal is to incorporate new information into an existing piece of text. An illustration is provided in Figure 1. Given an outdated Wikipedia article and collection of new information about the article's subject, FRUIT requires updating the existing text so that it is consistent with the new information, as well as adding text to reflect new salient facts, e.g., in Figure 1, the first sentence is updated to reflect that Tom Kristensson now drives in the Junior World Championship, and new sentences are added to reflect his achievements in 2019 and 2020.
FRUIT presents several unique challenges. First, unlike many generation tasks, models cannot obtain good performance by solely relying on their parametric world knowledge. Whenever the provided evidence contradicts parametric knowledge, the model must prefer the evidence, which recent work has shown is difficult for pretrained language
<sup>1</sup>Our data and code are available at:
https://github.com/google-research/
language/tree/master/language/fruit.
2https://en.wikipedia.org/wiki/
Wikipedia:Statistics
Human Language Technologies, pages 3670 - 3686
Output: Updated Article
Input: New Information
Figure 1: Illustration of the FRUIT task. An outdated original article and relevant new information are provided as inputs, and the goal is to generate the updated article. In this example, the original article about Tom Kristensson was written in 2020, and the new information is comprised of updated information about Tom Kristensson that has been added to other Wikipedia articles between 2020 and 2021. Given these inputs, the goal is to produce the updated 2021 version of article. Models need to identify the relevant supporting facts (orange and teal) to generate faithful updates while ignoring superfluous information (grey).
models (Krishna et al., 2021; Longpre et al., 2021). Second, the generated text needs to be faithful to both the original article and the new evidence, except when evidence invalidates information in the existing article. Finally, this task requires models to jointly read and analyze evidence from both textual and tabular sources and determine which is relevant and which can be ignored, thus combining challenging aspects of both multi-document summarization and data-to-text generation.
To facilitate research on this task, we release the FRUIT-WIKI dataset, a collection of over 170K distantly supervised ("silver") update-evidence pairs. This dataset is produced by comparing pairs of English Wikipedia snapshots to identify updates to an article between two snapshots, and associating information from the other articles that supports these updates under a distant supervision assumption. As there is no guarantee that updates in the later Wikipedia snapshots can be supported by the collected evidence, we also collect a "gold" evaluation set of 914 human annotated update-evidence pairs where unsupported claims have been removed without disturbing fluency. We train and validate our models using silver data and then evaluate the final performance using gold data.
We establish initial benchmark results for a number of trivial and neural sequence-to-sequence base
lines. We also introduce EDIT5, a T5-based model specially adapted for grounded editing, which establishes state-of-the-art performance on FRUIT-WIKI. Through an extensive set of analyses, we identify a number of failure modes needed to be improved upon in order to obtain better performance on FRUIT-WIKI, as well as other interesting topics for future work on this task. We additionally release our data collection pipeline to allow researchers to produce data from future Wikipedia snapshots and other languages, which we show to produce high-quality silver data.
2 The FRUIT Task
In this section we introduce the task of faithfully reflecting updated information in text (FRUIT). Given an input piece of text focused on a topic or event, along with a collection of potentially new information about the subject of the text, the goal is to update the input text to reflect the new information. A concrete illustration of the task is provided in Figure 1. The original piece of text along with its updates are shown on the left, while the new information is shown on the right.
Formally, we assume access to pair of texts,  $A^t$  and  $A^{t'}$ , pertaining to a given subject, written at
times  $t$  and  $t'$  (respectively). In addition, we assume access to a set of new information, a.k.a., evidence,  $\mathcal{E}^{t\rightarrow t'} = \{E_1,\ldots ,E_{|\mathcal{E}|}\}$ , mentioning the subject written between times  $t$  and  $t'$ . As is shown in Figure 1, the evidence can contain structured objects (e.g., excerpts from tables) as well as unstructured text. Given  $A^t$  and  $\mathcal{E}^{t\to t'}$  the goal is produce the updated text  $A^{t'}$ .
Successful completion of this task requires a number of complex and inter-related reasoning capabilities. For one, models must be able to identify which evidence contradicts existing portions of the source article, and which evidence introduces new salient information about the subject in order to correctly choose whether to alter the existing text vs. add new text. For example, in Figure 1 the first sentence is updated to reflect that Tom Kristensson now races in a different competition, whereas new sentences are added describing his achievements in the years 2019 and 2020. Models must also be able to determine whether a given piece of evidence should be used at all, i.e., perform content selection. For example, in Figure 1, the number of rounds won by Kristenssen appears in the evidence but does not correspond to any piece of updated text. Although some evidence may not appear in the updated article, the converse is not true, the system should aim to generate an updated article where all the updates are faithful to the evidence.
In this section we introduce important considerations for evaluating FRUIT systems.
Evaluate on Updated Text There is often considerable overlap between the original and updated text. As we will see in Section 5 this poses a challenge for standard evaluation metrics like ROUGE (Lin, 2004) as systems can achieve high scores without making any updates. In this work, we propose to evaluate FRUIT systems using an alternative metric, UpdateROUGE, that only considers updated sentences instead full texts. For example, in Figure 1, the reference for UpdateROUGE only consists of the first and last two sentences.
Evaluate Faithfulness Ensuring that generations faithfully reflect information in the evidence and updated article is crucial. However measuring faithfulness of generations is an active area of research (Celikyilmaz et al., 2020) and adapting existing metrics to the FRUIT task is non-trivial.
As a simple proxy for faithfulness, we choose to measure the token overlap between named entities appearing in the generation and the target article/evidence, where entities are identified using the named entity recognizer used by Guu et al. (2020) to perform salient span masking. We specifically introduce the following measurements:
1.Unsupported Entity Tokens. This metric shows the average number of entity tokens appearing in generated updates that do not appear in the source article or evidence. This is intended to capture the overall amount of unfaithful text, focusing on entities, where higher numbers indicate less faithfulness.
2. Entity Precision and Recall. These metrics capture the overlap of mentions between the generated updates and the target. Entity precision measures the fraction of entity tokens appearing in the generated updates that appear in target entities, whereas entity recall measures the fraction of entity tokens in the target that appear in the entities in generated updates. The latter is similar to UpdateROUGE but only evaluated on entities, and thus, potentially less sensitive to paraphrasing.
Parametric Knowledge Consideration FRUIT systems should incorporate information from the provided evidence into the update, and not information that happened to be present during training or pretraining. In this work we attempt to address this by evaluating models only on updates that were made to the text after the data used to pretrain and finetune the model was collected. As this setup precludes evaluating models trained after 2020 on FRUIT-WIKI, we release our data collection pipeline so that researchers can produce evaluation datasets from future versions of Wikipedia.
3 Dataset Collection and Analysis
As discussed in the introduction, keeping track of new information and then updating articles to reflect that information requires a massive amount of manual effort. Thus, in order to scalably collect sufficient data for training and evaluating FRUIT systems, some amount of automation is likely required. In this section we introduce the FRUIT-WIKI dataset and associated data collection pipeline, which allows the automatic collection of high-quality training and evaluation data for FRUIT from pairs of Wikipedia snapshots.
Train	Test
Silver	Gold
Years	'19-'20	'20-'21	'20-'21
Articles	114K	54K	914
Edits	407K	182K	3.0K
Subst. Edits	135K	62K	1.3K
Evidence	720K	315K	7.7K
Content Sel.	93K	42K	913
3.1 Pipeline
Our data collection pipeline produces distantly annotated training and evaluation data from pairs of Wikipedia snapshots. We will refer to the earlier snapshot as the source snapshot, and the later snapshot as the target snapshot.
Step 1. Collect Article Updates We compute the diff between the introductory sections of articles appearing in both the source and target snapshot to identify all of the material that has been updated (which will serve as  $A^t$  and  $A^{t'}$ ). We also compute the diff between the non-introductory sections of articles to find new mentions of the subjects of other articles (which will serve as  $\mathcal{E}^{t\to t'}$ ). These mentions can take the form of sentences in the text, as well as new table rows and list entries. Entities are disambiguated using Wikipedia hyperlinks.
Step 2. Filter Stylistic Updates A large number of edits to Wikipedia are stylistic (Daxenberger and Gurevych, 2012), and are therefore irrelevant to our task. In the next step of the pipeline, we attempt to filter articles that have only been superficially edited by keeping only those where at least one new added entity appears in the target snapshot.
Step 3. Identify Supporting Evidence In the last step of our pipeline, we seek to determine which pieces of evidence in  $\mathcal{E}^{t\to t'}$  justify each of the updated sentences in  $A^{t'}$ . To do so, we make the following distant supervision assumption: an updated sentence  $a\in A^{t'}$  containing an added entity  $s^\prime$  is substantiated by a piece of evidence  $E\in \mathcal{E}^{t\to t^{\prime}}$  only if  $s^\prime$  is also mentioned in  $E$ . The accuracy of the annotations produced by this assumption will be measured in Section 3.3.
Our pipeline is implemented using Apache Beam,3 to allow for distributed processing. We
Table 1: Dataset Statistics. We use  $10\%$  of the training data as our validation data.
UpdateROUGE	Entity
1	2	L	Prec.	Recall
87.4	84.6	87.1	91.8	94.6
Table 2: Inter-Annotator Agreement.
plan on releasing the code upon publication to enable other users to produce FRUIT data from future Wikipedia snapshots, as well as languages other than English.
3.2 FRUIT-WIKI
We run our pipeline on English Wikipedia snapshots from Nov. 20, 2019 to Nov. 20, 2020 to produce the training dataset, and from Nov. 20, 2020 to June 1, 2021 to produce the evaluation dataset. Detailed statistics are provided in Table 1. On average, there are around 3 to 4 updates per article, and around 7 pieces of associated evidence. About  $80\%$  of updates require some form of content selection, i.e., ignoring some evidence, when performing updates.
We find that only a third of the updates are substantiated by one or more pieces of evidence according to our distant supervision assumption. Thus, the remaining updates are either: a) superficial changes to the source article, or b) additions of new claims that are unsupported with respect to the collected evidence. The latter is a particular issue as these claims can cause the model to learn to hallucinate during training, and should be impossible for the model to guess during evaluation. Through the usage of human annotations and carefully selected evaluation metrics we will study the extent to which this is an issue throughout the rest of the paper.
We categorize articles in our dataset using the Wikipedia Foundation's topic model (Asthana and Halfaker, 2018). The distribution of topics is displayed in Figure 2. We find that the majority (approximately  $50\%$ ) of updates deal with cultural topics (e.g., sports, media, personal biographies), and geographic entities (e.g., countries, states) which intuitively are likely to be affected by current events while there are few updates to STEM- and history-related articles.
3.3 Gold Evaluation Data
To address the issue of unsupported claims during evaluation, we hired a team of 9 annotators to produce a "gold" evaluation subset of our test dataset.
<sup>3</sup>https://beam.apache.org/
Figure 2: Topic Distribution.
We collect annotations for 914 update-evidence pairs where each instance is corrected to ensure that all of the updates are supported. For the remainder of the paper we will refer to the distantly supervised test dataset annotations as "silver".
Annotation Process For each instance, annotators were shown the source article, evidence, and a marked up copy of the target article. In the marked up article, each updated sentence was highlighted and prefixed with reference labels to the supporting evidence identified by our pipeline. The correction process proceeded in two steps. In the first step, annotators were asked to highlight all of the unsupported claims and incorrect reference labels in the target article. In the second step, annotators were then asked to remove the unsupported text and minimally update the article to preserve fluency.
Annotators attended an initial 30 minute training and were provided regular feedback from the authors during the early stages of annotation. To ensure data quality, an additional annotator was hired with the sole job of checking the other annotator's work and correcting their mistakes. In total annotators spent roughly 500 hours on annotation. The annotation interface and a completed annotation are shown in Figure A7 in the Appendix.
Agreement We measure annotator agreement using a subset of 100 instances that were annotated by multiple annotators. Following Chen et al. (2015) and Shi et al. (2021), we quantify agreement by computing the evaluation metrics described in Section 2.2. The results are provided in Table 2. We observe high inter-annotator agreement with all scores in the 80s and 90s.
Analysis Statistics for the gold evaluation dataset are provided in Table 1. Overall, they closely resemble the statistics for the distantly supervised
UpdateROUGE	Entity	Reference Agreement
1	2	L	Prec.	Recall
83.7	81.2	83.4	90.4	100.0	84.5
Table 3: Gold and Silver Annotation Agreement. Quality of Silver Annotations by using the Gold as the reference.
data with one exception: the fraction of substantiated updates has increased.
To measure the quality of our silver data, we reapply the approach used to measure inter-annotator agreement to compute agreement between the gold and silver annotations. We also measure the reference agreement, i.e., the fraction of reference labels kept by the annotators. Results are provided in Table 3. We find that agreement is high with most scores in the 80s, a strong indication that the data produced by our pipeline is high quality. In particular, the high UpdateROUGE scores provide further evidence that only a small amount of the updated text in the weakly supervised data is unsupported, while the high reference agreement indicates that our distant supervision assumption is usually accurate.
In this section we introduce baseline methods to establish initial benchmark results on FRUIT-WIKI. We consider trivial approaches that copy task inputs, as well as T5, a neural sequence-to-sequence baseline which has shown strong performance on related tasks such as summarization (Raffel et al., 2020; Rothe et al., 2021) We additionally introduce EDIT5, a variant of T5 that produces a sequence of edits instead of the entire updated text, and employs additional tweaks to improve performance.
4.1 Copy Baselines
The first set of baselines we introduce are trivial methods that merely copy the input. We consider two variants:
- Copy Source: Generates a copy of the source article, and
- Copy Source + Evidence: Generates a copy of the source article concatenated with the evidence. Our evaluation metrics only apply to unstructured text, however the evidence may contain structured tables. In order to convert these tables to text, we apply a conventional linearization scheme (Lebret
UpdateROUGE	Entity	Unsup.
1	2	L	Prec.	Recall	Tokens
Copy Source	0.0	0.0	0.0	0.0	0.0	0.00
+ All Evidence	18.8	6.9	12.0	37.9	64.9*	0.00
T5-Large	31.1	18.4	24.4	52.7	44.9	2.67
+ Evidence Input	44.3	29.4	36.8	62.2	50.7	2.34
Edit5-Small	41.2	27.3	35.3	62.4	44.9	1.71
Edit5-Base	47.0	32.1	39.7	62.2	54.9	2.28
Edit5-Large	46.3	32.4	39.6	67.2	53.1	1.54
Edit5-3B	47.4	34.0	41.1	69.9	52.5	1.58
* Entity recall is not 100% for the Copy Source + All Evidence baseline due to lexical variation in entity mentions.
Grounded Updates	50
Additional Content	15
Missing Content	22
Ungrounded Updates	35
Number/Date	21
Distorted Evidence	11
Hallucination	14
No Updates	14
Table 4: (a) Model Results on Gold Evaluation Data. EDiT5 outperforms T5 models in all metrics. (b) Error Analysis for EDiT5-3B. We find that the model makes correct, grounded updates on  $50\%$  of the inspected articles. For incorrect updates, ungrounded numbers/dates are one of the main sources of error.
et al., 2016; Wiseman et al., 2017) that separates table entries using row and column delimiters.
4.2 T5
T5 (Raffel et al., 2020) is a pretrained sequence-to-sequence (Sutskever et al., 2014) model based on the transformer architecture (Vaswani et al., 2017). Similar to the previous section we experiment with two variants:
- T5: Only includes the source article in its input,
- T5 + Evidence Inputs: Includes both the source article and evidence in the input.
Tabular inputs are linearized using the same approach described in the previous section. Experiments are performed using the JAX-based T5X library. $^{4}$  Hyperparameters and additional training details are described in Appendix B.
4.3 EDIT5
Lastly, we introduce EDIT5, which improves upon the T5-based approach described in the previous section through the usage of a compressed output format that removes the need to write the entire update from scratch and encourages content planning. The output is modified in two ways:
First, as the majority of text in the target article is copied from the source, we replace any copied sentence with a single copy token identifying the sentence, e.g., if the second sentence is copied it is replaced by the token [2]. Similar to a copy mechanism (See et al., 2017), this allows the model to dedicate less capacity to repeating sequences from the input. As the resulting output resembles
(2) Tom Krister Kristensson (born 30 April 1991) is a Swedish rally driver, who drives in the Junior World Championship. [1] [2] (1) In the 2019 season of JWRC, Tom finished second behind Jan Solans. (2) The next season he went on to become the 2020 Junior World Rally champion.
Figure 3: EDIT5 Output Format. Instead of generating the fully updated text, EDIT5 generates sequences of edited sentences, copy tokens (e.g., [2], which means copy the second sentence), and reference tokens (e.g., (1), which means the following sentence should use the first piece of evidence).
that produced by the diff data comparison utility, we refer to this as a diff-formatted output.
Second, before each update we insert a sequence of reference tokens identifying the pieces of evidence that support the update, e.g., if the first and third piece of evidence in  $\mathcal{E}^{t\rightarrow t^{\prime}}$  support an update then the update is prefaced by (1) (3). This approach, inspired by the use of entity chains for summarization (Narayan et al., 2021), trains the model to plan which references to use before generating an update. These reference tokens are removed from the output text of the model prior to computing the evaluation metrics.
An example of the EDIT5 output format is provided in Figure 3, and a comparison to the T5 output format is provided in Appendix D. Training details and hyperparameters match the setup described in Section 4.2.
<sup>4</sup>https://github.com/google-research/t5x
Baseline results on the gold evaluation data are provided in Table 4a, and ablation results are provided in Appendix A. In general, we find that the copy baselines perform worse than T5 and T5 performs worse than EDT5. Notably, the copy source baseline rightfully scores zero on all metrics, while we will later find that it obtains a high ROUGE score.
Although our models are trained on silver data, they still obtain good performance on the gold evaluation set. This shows the high quality of our silver data collection pipeline, and T5's ability to generate reasonable updates based on the evidence.