SlowGuess/NAACL_2022_OCR huggingface dataset:
Alexander Spangher $^{1}$ , Xiang Ren $^{1}$ , Jonathan May $^{1}$ , Nanyun Peng $^{2}$
$^{2}$ Computer Science Department, University of California, Los Angeles {spangher, xiangren, jonmay}@usc.edu, violetpeng@cs.ucla.edu
News article revision histories provide clues to narrative and factual evolution in news articles. To facilitate analysis of this evolution, we present the first publicly available dataset of news revision histories, NewsEdits. Our dataset is large-scale and multilingual; it contains 1.2 million articles with 4.6 million versions from over 22 English- and French-language newspaper sources based in three countries, spanning 15 years of coverage (2006-2021).
We define article-level edit actions: Addition, Deletion, Edit and Refactor, and develop a high-accuracy extraction algorithm to identify these actions. To underscore the factual nature of many edit actions, we conduct analyses showing that added and deleted sentences are more likely to contain updating events, main content and quotes than unchanged sentences.
Finally, to explore whether edit actions are predictable, we introduce three novel tasks aimed at predicting actions performed during version updates. We show that these tasks are possible for expert humans but are challenging for large NLP models. We hope this can spur research in narrative framing and help provide predictive tools for journalists chasing breaking news.
Revision histories gathered from various natural language domains like Wikipedia (Grundkiewicz and Junczys-Dowmunt, 2014), Wikihow (Faruqui et al., 2018) and student learner essays (Zhang and Litman, 2015) have primarily been studied to explore stylistic changes, such as grammatical error correction (Shah et al., 2020) and argumentation design (Afrin et al., 2020). However, deeper questions about content updates and narrative evolution are underexplored: Which facts are uncertain and likely to be changed? Which events are likely to
Figure 1: We identify sentence-level operations – Edit, Addition, Deletion and Refactor – between two versions of a news article (merges, shown here, and splits are a special cases of Edits). We propose tasks aimed at predicting these operations on article versions. We characterize aspects of additions, deletions and edits. We hope NewsEdits can contribute to research on narrative and factual development patterns.
update? What voices and perspectives are needed to complete a narrative?
Existing edits corpora do not address these questions due to the nature of previously studied domains: as shown in Yang et al. (2017), the distribution of edits in other domains, like Wikipedia, tend to focus on syntax or style edits. In this work, we introduce a novel domain for revision histories, news article revision histories which, we show, covers the updating of events. Many edits in news either (1) incorporate new information (2) update events or (3) broaden perspectives (Section 3).
Our dataset, NewsEdits, contains 1.2 million articles with 4.6 million versions. We develop a document-level view for studying revisions and define four edit actions to characterize changes between versions: sentence Addition, Deletion, Edit and Refactor (i.e. the sentence is moved within a document). We introduce algorithms for identifying these actions. We count over 40 million Edits, Additions, Deletions or Refactors in NewsEdits.
We argue that news is an important, practical medium to study questions about narrative, factual
<sup>1</sup>We release the dataset and all code used in modeling and evaluation: https://github.com/isi-nlp/NewsEdits.git
Human Language Technologies, pages 127 - 157
and stylistic development. This is because, we hypothesize, there are consistent patterns in the way articles update in the breaking news cycle (Usher, 2018). To prove this hypothesis, we show that updates are predictable. We design three tasks: (1) "predict whether an article will be updated," (2) "predict how much of an article will be updated," (3) "predict sentence-level edit actions." We show that current large language model (LLM)-based predictors provide a strong baseline above random guessing in most tasks, though expert human journalists perform significantly better. Our insights are twofold: (a) article updates are predictable and follow common patterns which humans are able to discern (b) significant modeling progress is needed to address the questions outlined above. See Section 4.6 for more details.
Finally, we show that the NewsEdits dataset can bring value to a number of specific, ongoing research directions: event-temporal relation extraction (Ning et al., 2018; Han et al., 2019a), article link prediction (Shahaf and Guestrin, 2010), fact-guided updates (Shah et al., 2020), misinformation detection (Appelman and Hettinga, 2015), headline generation (Shen et al., 2017) and author attribution (Savoy, 2013), as well as numerous directions in computational journalism (Cohen et al., 2011; Spangher et al., 2020) and communications fields (Spangher et al., 2021b).
1. We introduce NewsEdits, the first public academic corpus of news revision histories.
2. We develop a document-level view of structural edits and introduce a highly scalable sentence-matching algorithm to label sentences in our dataset as Addition, Deletion, Edit, Refactor. We use these labels to conduct analyses characterizing these operations.
3. We introduce three novel prediction tasks to assess reasoning about whether and how an article will change. We show that current large language models perform poorly compared with expert human judgement.
2 The NewsEdits Dataset
NewsEdits is a dataset of 1.2 million articles and 4.6 million versions. In Section 2.1, we discuss the sources from which we gathered our dataset. In Section 2.2, we discuss the categories of edit-actions designed to characterize changes between versions, and in Section 2.3, we discuss the algorithm we
Figure 2: Number of versions per article, by outlet.
built to identify these edit-actions.
We collect a dataset of news article versions. An article is defined by a unique URL, while a version is one publication (of many) to that same URL. We combine data from two online sources that monitor news article updates: NewsSniffer and Twitter accounts powered by DiffEngine. These sources were chosen because, together, they tracked most major U.S., British and Canadian news outlets (Kirchhoff, 2010). Our corpus consists of article versions from 22 media outlets over a 15-year timescale (2006-2021), including The New York Times, Washington Post and Associated Press. Although the median number of updates per article is 2, as shown in Figure 2, this varies depending on the outlet. More dataset details in Appendix E.
2.2 Edit-Action Operations
Since we are interested in how an entire news article updates between versions, we focus on sentence edits (document-level actions), not word edits (sentence-level actions). Identifying that sentences are added and deleted (vs. updated), can help us study the degree of change an edit introduces in the article (Daxenberger and Gurevych, 2012, 2013; Fong and Biuk-Aghai, 2010).
Thus, we define the following sentence-level edit-actions, shown in Figure 1: Addition, Deletion, Edit and Refactor. Additions should contain novel information and Deletions should remove information from the article. Edits should be substantially similar except for syntactic changes, rephrased and minimally changed or updated information. Special cases of the Edit operation result in sentences that are merged or split without substantial changes. See Section 2.3 for more details.
Refactors are intentionally moved in an article.4
$^{2}$ https://www.newssniffer.co.uk/
<sup>3</sup>https://github.com/DocNow/diffengine
As an example, in Figure 1, the addition of Sentence 2 in version  $t + 1$  shifts Sentences 3, 4, 5 down. These are not refactors, just incidental moves caused by other operations. However, Sentences 5, 6 in version  $t$  are shifted upwards in
BERT-Based	Subsequence Matching	BLEU-Based
Method	F1-Score	Method	F1-Score	Method	F1-Score
Hungarian	TB-mini	88.5	ngram-1	86.0	BLEU-1	86.7
TB-medium	88.7	ngram-2	88.7	BLEU-2	89.2
RB-base	88.6	ngram-3	88.5	BLEU-3	88.8
Max	TB-mini	89.0	ngram-4	88.2	BLEU-1,2	88.8
TB-medium	89.5	BLEU-1,2,3	89.1
RB-base	89.4
Table 1: F1 scores on validation data for matching algorithms. Left-hand group shows embedding-based methods (TinyBert (TB) and RoBERTa (RB)) with Maximum or Hungarian matching. Middle group shows ngram methods. Right-hand group shows BLEU for different ngram weightings (1,2 and 1,2,3 are uniform weightings over unigrams, bigrams and trigrams).
Refactors are important because, based on the inverse pyramid (Pöttker, 2003) of article structure, sentences that are higher in an article are more important (Scanlan, 2003). Thus, Refactors give us insight into the changing importance of sentences in a narrative.
2.3 Edit-Action Extraction
To extract these edit-actions, we need to be able to construct a bipartite graph linking sentences between two versions of an article (example graph shown in Figure 1). If an edge exists between a sentence in one version and a sentence in the other, the sentence is an Edit (or Unchanged). If no edge exists, the sentence is an Addition (if the sentence exists in the newer version only) or Deletion (if it exists in the older version only). We identify Refactors based on an algorithm we develop: in short, we identify a minimal set of edges in the graph which causes all observed edge-crossings. For details on this algorithm, see Appendix F.
In order to construct this bipartite graph, we need a scalable, effective, sentence-similarity algorithm. There is a wide body of research in assessing sentence-similarity (Quan et al., 2019; Abujar et al., 2019; Yao et al., 2018; Chen et al., 2018). However, many of these algorithms measure symmetric sentence-similarity. As shown in Figure 1, two sentences from the old version can be merged in the new version. The symmetric similarity between these three sentences would be low, leading us to label the old sentences as Deletions and the new one an Addition, even if they were minimally edited
(for concrete examples, see Table 14). This violates our tag definitions (Section 2.2). So, we need to measure one-way similarity between sentences, allowing us to label merged and split sentences as Edits. Our algorithm is an asymmetrical version of the maximum alignment metric described by Kajiwara and Komachi (2016):
$$
\operatorname {S i m} _ {\text {a s y m}} (x, y) = \frac {1}{| x |} \sum_ {i = 1} ^ {| x |} \max  _ {j} \phi (x _ {i}, y _ {j})
$$
where  $\phi (x_i,y_j)\coloneqq$  similarity between words  $x_{i}$  in sentence  $x$  and  $y_{j}$  in sentence  $y$ .
We test several word-similarity functions,  $\phi$ . The first uses a simple lexical overlap, where  $\phi(x_i, y_j) = 1$  if lemma  $(x_i) = \text{lemma}(y_j)$  and 0 otherwise. The second uses word-embeddings, where  $\phi(x_i, y_j) = \text{Emb}(x_i) \cdot \text{Emb}(y_j)$ , and  $\text{Emb}(x_i)$  is the embedding derived from a pretrained language model (Jiao et al., 2020; Liu et al., 2019).
Each  $\phi$  function assesses word-similarity; the next two methods use  $\phi$  to assess sentence similarity. Maximum alignment counts the number of word-matches between two sentences, allowing many-to-many word-matches between sentences. Hungarian matching (Kuhn, 1955) is similar, except it only allows one-to-one matches. We compare these with BLEU variations (Papineni et al., 2002), which have been used previously to assess sentence similarity (Faruqui et al., 2018).
2.4 Edit-Action Extraction Quality
Although our sentence-similarity algorithm is unsupervised, we need to collect ground-truth data in order to set hyperparameters (i.e. the similarity threshold above which sentences are considered a match) and evaluate different algorithms. To
7We extend this to non-overlapping ngram matches.
version $_{t+1}$ , which is movement that is not caused by other operations. We label this as a Refactor.
An inverse pyramid narrative structure is when the most crucial information, or purpose of the story, is presented first (Scanlan, 2003).
$^{6}$ E.g. "ipsum, Lorem"  $\rightarrow$  "ipsum; and Lorem". Conversely, one sentence can also be split.
Total Num.	% of Sents.
Edits	26.6 mil.	17.6 %
Additions	10.2 mil.	6.8 %
Deletions	5.4 mil.	3.6 %
Refactors	1.6 mil.	1.1 %
(a) Edit-actions by version (% total in version).
Figure 3: Dynamics of edit actions.
(b) Edit-actions by sentence position  $(\%)$ .
do this, we manually identify sentence matches in 280 documents. We asked two expert annotators to identify matches if sentences are nearly the same, they contain the same information but are stylistically different, or if they have substantial overlap in meaning and narrative function. See Appendix G for more details on the annotation task. We use  $50\%$  of these human-annotated labels to set hyperparameters, and  $50\%$  to evaluate match predictions, shown in Table 1. Maximum Alignment with TinyBERT-medium embeddings (Jiao et al., 2020) (Max-TB-medium) performs best. $^{8}$
3 Exploratory Analysis
We extract all edit actions in our dataset using methods described in the previous section. Statistics on the total number of operations are shown in Table 2. In this section, we analyze Additions, Deletions and Edits to explore when, how and why these edit-actions are made and the clues this provides as to why articles are updated. We leave a descriptive analysis of Refactors to future work.
Insight #1: Timing and location of additions, deletions and edits reflect patterns of breaking news and inverse pyramid article structure. How do editing operations evolve from earlier to later versions, and where do they occur in the news article?
In Figure 3a, we show that edit-actions in an article's early versions are primarily adding or updating information: new articles tend to have roughly  $20\%$  of their sentences edited,  $10\%$  added and few deleted. This fits a pattern of breaking news lifecy
Table 2: Summary Statistics for Sentence Operations
Add.	Del.	Unchang.
Contains Event	38.5	39.3	31.4
Contains Quote	48.4	50.0	39.2
Discourse: Main	4.4	4.9	3.6
Discourse: Cause	29.0	30.2	23.6
Discourse: Distant	63.5	61.4	68.1
Table 3: % Additions, Deletions or Unchanged sentences that contain Events or Quotes, or have news discourse role: Main (main events), Cause (immediate context) or Distant (history, analysis).  $F < .01$ ,  $n = 7,368,634$ .
cles: an event occurs, reporters publish a short draft quickly, and then they update as new information is learned (Hansen et al., 1994; Lewis and Cushion, 2009). We further observe, as is demonstrated in Figure 6 in the appendix, that updates occur rapidly: outlets known for breaking news have a median article-update time of  $< 2$  hours.
An article's later lifecycle, we see, is determined by churn:  $\approx 5\%$  of sentences are added and  $5\%$  are deleted every version. As seen in Figure 3b, additions and edits are more likely to occur in the beginning of an article, while deletions are more likely at the end, indicating newer information is prioritized in an inverse pyramid structural fashion.
Insight #2: Additions and deletions are more likely to contain fact-patterns associated with breaking news (quotes, events, or main ideas) than unchanged sentences. In the previous section, we showed that the timing and position of edit-actions reflects breaking news scenarios. To provide further clues about the semantics of edit-actions, we sample Additions, Deletions and unchanged sentences and study the kinds of information contained in these sentences. We study three different fact-patterns associated with breaking news: events, quotes and main ideas (Ekström et al., 2021; Usher, 2018). To measure the prevalence of these fact-patterns, we sample 200,000 documents (7 million sentences) from our corpus and run an event-extraction pipeline (Ma et al., 2021), quote-detection pipeline (Spangher et al., 2020), and news discourse model (Spangher et al., 2021a). As shown in Table 3, we find added and deleted sentences have significantly more events, quotes and Main-Idea and Cause discourse than unchanged sentences. (See Appendix B for more details.)
Insight #3: Edited sentences often contain updating events. The analyses in the previous sec
8For more details and examples, see Appendix F.
$^{9}$ E.g. Associated Press, New York Times and Wash. Post
Event Chains
attack, killed), (injured, killed), (shot, dead), (shot, killed), (attack, injured), (injured, died), (election, won), (meeting, talks), (talks, meeting), (elections, election), (war, conflict)
Table 4: Selection of top event extracted from edited sentence pairs across article versions.
tions have established that edit-actions both are positioned in the article in ways that resemble, and contain information that is described by, breaking news epistemologies (Ekström et al., 2021). A remaining question is whether the edit-actions change fact-patterns themselves, rather than simply changing the style or other attributes of sentences.
One way to measure this is to explore whether edit-actions update the events in a story (Han et al., 2019b). We focus on pairs of edited sentences. We randomly sample Edits from documents in our corpus ( $n = 432$ , 329 pairs) and extract events using Ma et al. (2021)'s model. We find that edited sentence pairs are more likely to contain events ( $43.5\%$ ) than unchanged sentences ( $31.4\%$ ). Further, we find that  $37.1\%$  of edited sentences with events contain different across versions. We give a sample of pairs in Table 4. This shows that many within sentence operations update events.
Taken together, we have shown in this analysis that factual updates drive many of the edit operations that we have constructed to describe NewsEdits revision histories. Next, we will measure how predictable these update patterns are.
4 Predictive Analysis on NewsEdits
As shown in Section 3, many edit-actions show breaking news patterns, which Usher (2018) observed follow common update patterns. Now, we explore how predictable these operations are, to address whether future work on the fundamental research questions addressed in Section 1 around narrative design is feasible.
In this section, we outline three tasks that involve predicting the future states of articles based on the current state. These tasks, we hypothesize, outline several modeling challenges: (1) identify indicators of uncertainty used in news writing<sup>10</sup> (Ekström et al., 2021), (2) identify informational incompleteness, like source representation (Spangher et al., 2020) and (3) identify prototypical event patterns (Wu et al., 2022). These are all strategies that ex
pert human evaluators used when performing our tasks (Section 4.6). The tasks range from easier to harder, based on the sparsity of the data available for each task and the dimensionality of the prediction. We show that they are predictable but present a challenge for current language modeling approaches: expert humans perform these tasks much more accurately than LLM-based baselines.
In addition to serving a model-probing and data-explanatory purpose, these tasks are also practical: journalists told us in interviews that being able to perform these predictive tasks could help newsrooms allocate reporting resources in a breaking news scenario.[11]
4.1 Task Description and Training Data Construction
We now describe our tasks. For all three tasks, we focus on breaking news by filtering NewsEdits down to short articles (# sents  $\in$  [5, 15]) with low version number (<20) from select outlets.[12]
Task 1: Will this document update? Given the text of an article at version  $\nu$ , predict if  $\exists \nu + 1$ . This probes whether the model can learn a high-level notion of change, irrespective of the fact that different edit-actions have different consequences for the information presented in a news article.
For Task 1,  $y = 1$  if a newer version of an article was published and 0 otherwise. We sample 100,000 short article versions from NewsEdits, balancing across length, version number, and  $y$ .
Task 2: How much will it update? Given the text of an article at version  $\nu$ , predict in the next version how many Additions, Deletions, Edits, Refactors will occur. This moves beyond Task #1 and requires the model to learn more about how each edit-action category changes an article.
For Task 2,  $y =$  counts of sentence-level labels (Num. Additions, Num. Deletions, Num. Refactors, Num. Edits) described in the previous sections, aggregated per document. Each count is binned: [0,1), [0,3), [3,∞) and is predicted separately as a multiclass classification problem. We sample 150,000 short article versions balancing for sources, length and version number.
Task 3: How will it update? For each sentence in version  $\nu$ , predict whether: (1) the sentence itself
11 See Appendix A for more details.
12The New York Times, Associated Press, Washington Post, BBC, Independent, Guardian and Reuters were used, as they are more known for breaking news (Usher, 2018). See Appendix E for more details.
10E.g. "Police to release details of the investigation."
Figure 4: Architecture diagram for the model used for our tasks. Word-embeddings are averaged using Self-Attention to form sentence-vectors. A minimal transformer layer is used to contextualize these vectors (+Contextual Layer). In Tasks 1 and 2, self-attention is used to generate a document-embedding vector.
will change (i.e. it will be a Deletion or Edit) (2) a Refactor will occur (i.e. it will be moved either up or down in the document) or (3) an Addition will occur (i.e. either above or below the sentence). This task, which we hypothesize is the hardest task, requires the model to reason specifically about the informational components of each sentence and understand nuance about structure and form in a news article (i.e. like the inverse pyramid structure (Pöttker, 2003)).
For Task 3,  $y =$  individual sentence-level labels. Labels are derived for the following subtasks mentioned above: (1) Sentence Operations is a categorical label comprising: [Deletion, Edit, Unchanged], expressed as a one-hot vector. (2) Refactor is a categorical label comprising: [Up, Down, Unchanged], also expressed as a one-hot vector. (3) Addition Above and Addition Below are each binary labels expressing whether  $>1$  sentences was added above or below the target sentence. Because some sentences had Additions above and below, we chose to model this subtask as two separate classification tasks. We sample 100,000 short article versions, balancing for sources, length and version number.
For each task, the input  $X$  is a document represented as a sequence of sentences. For each evaluation set, we sample  $4k$  documents balancing for class labels (some labels are highly imbalanced and cannot be balanced).
4.2 Modeling
We benchmark our tasks using a RoBERTa-based architecture shown in Figure 4. Spangher et al. (2021a) showed that a RoBERTa-based architec
ture (Liu et al., 2019) with a contextualization layer outperformed other LLM-based architectures like Reimers and Gurevych (2019) for document-level understanding tasks (further insight given in Section 4.6).
In our model, each sentence from document  $d$  is fed into a pretrained RoBERTa. Base model<sup>13</sup> to obtain contextualized word embeddings. The word embeddings are then averaged using self-attention, creating sentence vectors. For Task 3, these vectors are then used directly for sentence-level predictions. For Tasks 1 and 2 these vectors are condensed further, using self-attention, into a single document vector which is then used for document-level predictions. The sentence vectors are optionally contextualized to incorporate knowledge of surrounding sentences, using a small Transformer layer<sup>14</sup> (+Contextualized in Tables 5, 6, 7).