SlowGuess/NAACL_2022_OCR huggingface dataset:
For deriving sentence embeddings, we tested several different methods. We tested both using the <sep> token from RoBERTa and averaging the word-embeddings of each word-piece, as in Spangher et al. (2021a), but found that a third method—using self-attention over the word embeddings, or a learned, weighted average—performed the best. We concatenated a sentence-level positional embedding vector, as in Spangher et al. (2021a), with a max cutoff of 40 positional embeddings (i.e. every sentence with an index greater than 40 was assigned the same vector.)
Here, we give additional details on the dataset, starting with relevant analyses and ending with technical details that should guide the user on how to access our dataset.
33We achieve a macro F1-score of .67 on validation data using the architecture described in Spangher et al. (2021a).
U.S. Politics (topic 0)	U.K. Politics (topic 2)	Police Crime (topic 5)	Aviation (topic 6)	Tragedy (topic 7)	War (topic 9)	Criminals Crime (topic 12)	School (topic 13)	Violence Crime (topic 18)
mr	government	police	people	family	killed	court	school	police
president	party	man	airport	died	people	year	year	officers
trump	mr	old	plane	hospital	attack	old	world	people
minister	labour	year	aircraft	old	al	mr	new	area
prime	council	arrested	reported	man	forces	man	people	incident
house	minister	woman	agency	service	attacks	murder	city	local
donald	leader	officers	officials	rescue	group	police	time	scene
obama	new	men	news	year	military	years	years	shot
white	people	suspicion	air	police	city	told	day	shooting
new	secretary	london	flight	death	security	guilty	event	injured
Table 9: Topic Model: Top Topics, selected on the bases of the number of documents they are most-expressed in. Labels are assigned by the researchers post-hoc. Several topics appear to be subsets of a broader Crime topic: we note the superclass Crime in parentheses. The specific Crime topic mentioned in the main body is the Violence topic (Topic 18)
(a) Distribution over days per update, group 1. Median across all sources in this group is 21 days.
(b) Distribution over days per update, group 2. Median across all sources in this group is .9 days
(c) Distribution over days per update, group 3. Median across all sources in this group is .35 days, or 8.4 hours
Figure 6: Average time between version updates. We break sources into four primary groups with similar update distributions.
(d) Distribution over days per update, group 4. Median across all sources in this group is .05 days, or 1.33 hours.
(a) Distributions over discourse tags, by article length.
(c) Odds of discourse element in added sentences. odds  $= p(d|v,a) / p(d|v!,a)$
Figure 7: Dynamics of news discourse composition size across time.  $d$  refers to discourse label,  $\nu$  refers to version and  $a$ ,  $del$  refers to is-added, is_deleted
(b) Odds of discourse element, by version. odds  $= p(d|\nu) / p(d! \nu)$ .
(d) Odds of discourse element in deleted sentences. odds  $= p(d|v, del) / p(d|v, !del)$ .
E.1 Additional Analysis
E.1.1 Amount of time between Versions
The amount of time between republication of an article varies widely across news outlets, and has a large role in determining what kinds of stories are being republished. As can be seen in Figure 6, we group sources into 4 categories: (1) Figure 6a, those that update articles over weeks (tabloids and magazines), (2) Figure 6b, those that update articles on a daily basis, on median, (3) Figure 6c, those that update 2-3 times a day, and (4) Figure 6d, those that update hourly, or breaking news outlets.
We are especially interested in rapid updates, because, by limits imposed by this timescale on how
much information can be gathered by journalists, these updates are more likely to contain single units of information, updates and quotes. Thus, in our experiments, we focus on The New York Times, Independent, Associated Press, Washington Post, and BBC. We also include Guardian and Reuters because they typically compete directly with the previously mentioned outlets in terms of content and style, even if they do not publish as frequently.
E.1.2 Discourse Across Time
We are interested in the dynamics of articles over time. Although this analysis is still ongoing, we seek to understand how, as the article grows through time, the types of information included in it changes. We show in Figure 7a and 7b that in
Unchanged	said, trump, people, president, concerns, government, year
Add/Del	says, senate, law, death, wednesday, monday, tuesday
Table 10: Top Words in Additions/Deletions vs. top words in unchanged sentences.
later versions and longer articles<sup>34</sup> sentences are dominated by Distant discourse.
Interestingly, later versions are also more likely to have Main and Cause discourse added. Based on our annotator interviews, we surmise that this is because, for breaking news, a journalist is frequently trying to assess the causes behind the story. In early drafts, we also see Main sentences being removed. This is due to, as the story is updating in early versions, the Main event is most likely to be changing.
E.1.3 Top Words
Top Words: We characterize added and deleted sentences by their word usage in Table 10. Words indicating present-tense, recent updates are more likely: day-names like "Monday" or "Tuesday" and the present-tense verb "says" (compared with the past-tense "said" in unchanged sentences).
E.1.4 Collection of Corrections, Authorship
To identify instances of Corrections in added sentences, we used the following lexicon:
"was corrected", "revised", "clarification", "earlier error", "version", "article"
Here are some examples of corrections:
- CORRECTION: An earlier version of this story ascribed to Nate spokesman Brig Gen Carsten Jacobsen comments suggesting that after Saturday's shooting, people would have to be "looking over their shoulders" in Afghan ministries.
- CORRECTION 19 November 2012: An earlier version of this story incorrectly referred to "gargoyles", not "spires".
- Correction 7 March 2012: An earlier version of this story mistakenly said Rushbrook's car had been travelling at  $140\mathrm{mph}$  at the time of the crash.
To identify instances of Contributor Lines, we use the following lexicon:
"reporting by", "additional reporting", "contributed reporting", "editing by"
Here are some examples of contributor lines:
Additional reporting by Simon Browning.
- 'The article relied heavily on reporting by Reuters and the BBC, and it cited Reuters in saying that during a visit in October 1989 by Pope John Paul II to South Korea, China had prevented the pope's airplane from flying through Chinese airspace.
- The revelation comes after reporting by The New York Times last week showing that the head of communications at the N.I.H.'s parent agency, the Department of Health and Human Services, also accused federal scientists of using the coronavirus to try to defeat Mr. Trump.
Additional reporting by Daniel Strauss in Richmond, Virginia, Richard Luscombe in West Palm Beach, Florida, and Ed Pilkington in Essex Junction, Vermont.
E.2 Dataset Tables and Fields
Our dataset is released in a set of 5 SQLite tables. Three of them are primary data tables, and two are summary-statistic tables. Our primary data tables are: articles, sentence_diffs, word_diffs; the first two of which are shown in Tables 12a and 12b (word_diffs shares a similar structure with sentence_diffs). We compile two summary statistics tables to cache statistics from sentence_diffs and word_diffs; they calculate metrics such as NUM_SENTENCES_ADDED and NUM_SENTENCES_REMOVE per article.[35]
The sentence_diffs data table's schema is shown in Table 12 and some column-abbreviated sample rows are shown in Table 14. As can be seen, the diffs are calculated and organized on a sentence-level. Each row shows a comparison of sentences between two adjacent versions of the same article.[36] Every row in sentence_diffs contains index columns: SOURCE, A_ID, VERSION_OLD, and VERSION Newly. These columns can be used to uniquely map each row in sentence_diffs to two rows in article.[37]
34 Version Number has spearman's correlation  $r = .335$  with article length.
35 These summary statistic tables make it convenient to, say, filter sentence_diffs in order to train a model on all articles that have one sentence added; or all articles that have no sentences removed.
$^{36}\mathrm{So}$ , for instance, article A, with versions 1, 2 where each version has sentences i, ii, iii, would have 3 rows (assuming sentences were similar): A.1-2.i, A.1-2.ii, A.1-2.iii.
37One mapping for sentence_diffs.VERSION_OLD  $=$  article.VERSION_ID and one mapping for sentence_diffs.VERSION_NEW  $\equiv$  article.VERSION_ID.
Source	# Articles	# Versions	Start	End	Ctry.	Lang.	Coll.
BBC	307,616	1,244,490	2006-08	2021-01	U.K.	En.	NS
Guardian	231,252	852,324	2012-01	2021-01	U.K.	En.	NS
Nytimes	87,556	395,643	2012-08	2020-12	U.S.	En.	NS
Telegraph	78,619	124,128	2017-01	2018-09	U.K.	En.	NS
Fox	78,566	117,171	2017-01	2018-09	U.S.	En.	DE
CNN	58,569	117,202	2017-01	2018-09	U.S.	En.	DE
Independent	55,009	158,881	2014-01	2018-05	U.K.	En.	NS
CBC	54,012	387,292	2017-08	2018-09	Ca.	En.	DE
Dailymail	50,639	166,260	2017-01	2018-09	U.K.	En.	DE
BBC	42,797	99,082	2017-01	2018-09	U.K.	En.	DE
La Presse	40,978	73,447	2017-08	2018-09	Ca.	Fr-Ca.	DE
Toronto Star	33,523	310,112	2017-08	2018-07	Ca.	En.	DE
Globemail	32,552	91,820	2017-08	2018-09	Ca.	En.	DE
Reuters	31,359	143,303	2017-01	2018-09	U.K.	En.	DE
National Post	22,934	63,085	2017-08	2018-09	Ca.	En.	DE
Associated Press	22,381	97,314	2017-01	2018-09	U.S.	En.	DE
Washington Post	19,184	68,612	2014-01	2020-07	U.S.	En.	NS
Toronto Sun	19,121	46,353	2017-08	2018-09	Ca.	En.	DE
Calgary Herald	7,728	33,427	2017-08	2018-09	Ca.	En.	DE
The Rebel	4,344	19,383	2017-08	2018-09	Ca.	En.	DE
Canada Land	65	101	2017-12	2018-09	Ca.	En.	DE
Table 11: A summary of the number of total number of articles and versions for different media outlets which comprise our dataset. Also shown is the original collection that they were derived from (DE for DiffEngine, and NS from NewsSniffer), and the date-ranges during which articles from each outlet were collected.
E.3 TAG columns in sentence_diffs
The columns TAG_OLD and TAG Newly in sentence_diffs have specific meaning: how to transform from version to its adjacent version. In other words, TAG_OLD conveys where to find SENT_OLD in VERSION Newly and whether to change it, whereas TAG Newly does the same for SENT Newly in VERSION OLD.
More concretely, consider the examples in Table 14b, 14a and 14c. As can be seen, each tag is 3-part and has the following components. Component 1 can be either M, A, or R. M means that the sentence in the current version was Matched with a sentence in the adjacent version, A means that a sentence was Added to the new version and R means the sentence was Removed from the old version.[38] Component 2 is only present for Matched sentences, and refers to the index or indices of the sentence(s) in the adjacent version.[39] Additionally, Component 3 is also only present if the sentence is Matched. It can be either C or U. C refers to whether the matched sentence was Changed and U to whether it was Unchanged.
Although not shown or described in detail, all M sentences have corresponding entry-matches in
word_diffs table, which has a similar schema and tagging aim.
A user might use these tags in the following ways:
1. To compare only atomic edits, as in Faruqui et al. (2018), a user could filter sentence_diffs to sentences where M.C is in TAG_OLD (or equivalently, TAG Newly). Then, they would join TAG_OLD Component_2 with SENTENCE_ID. Finally, they would select SENT_OLD, SENT Newly.40
2. To view only refactorings, or when a sentence is moved from one location in the article to another, a user could filter sentence_diffs to only sentences containing M..U and follow a similar join process as in use-case 1.
3. To model which sentences might be added, i.e.  $p(\text{sentence}_i \in \text{article}_{t+1} | \text{sentence}_i \notin \text{article}_t)$ , a user would select all sentences in SENT_OLD, and all sentences in SENT Newly where A is in TAG Newly.
4. To model the inverse of use-case 3, i.e. which sentences would be removed, or  $p(\text{sentence}_i \notin \text{article}_{t+1} | \text{sentence}_i \in \text{article}_t)$ , a user would select all sentences in SENT_NEW, and all sentences in SENT_OLD where R is in TAG_OLD.
38 i.e. an Added row is not present in the old version and a Removed row is not present in the new version. They have essentially the same meaning and we could have condensed notation, but we felt this was more intuitive.
39I.e. in TAG_OLD, the index refers to the SENTENCE_ID of SENT_NEW
or simply look in the word_diffs table.
Column Name	Type	Column Name	Type	Column Name	Type
SOURCE
A_ID
VERSION_ID	index
index
index	TITLE
URL
TEXT	text
text
text	CREATED
archive_url
NUM_VERSIONS	text
text
int
(a) DB schema for the article table. SOURCE, A_ID and VERSION_ID are the primary key columns.
Column Name	Type	Column Name	Type	Column Name	Type
SOURCE	index	V新加_ID	index	TAG新加	text
A_ID	index	SENTENCE_ID	index	SENT新加	text
V新加_ID	index	SENT OLD	text	TAG新加	text
(b) DB schema for the sentence_diffs table (word_diffs is similar). Table compares version pairs of articles. The rows in the table are on the sentence-level; V_OLD_ID refers to the index of the old version, V Newly_ID refers to the index of the new version. TAG_OLD gives information for how to transition from the old version to the new version; TAG Newly is the inverse.
Table 12: Schemas for two databases central to our content organization scheme.
E.4 Comparison With Other Edits Corpora
Here, we give a tabular comparison with other edits corpora, showing our
In this section, we give further examples further justify our asymmetrical sentence-matching algorithm. The examples shown in Tables 14b, 14a and 14c illustrate our requirements. The first example, shown in Table 14b, occurs when a sentence is edited syntactically, but its meaning does not change.[42] So, we need our sentence-matching algorithm to use a sentence-similarity measure that considers semantic changes and does not consider surface-level changes. The second example, shown in Table 14a, occurs when a sentence is split (or inversely, two sentences are merged.) Thus, we need our sentence matching algorithm to consider many-to-one matchings for sentences. The third example, shown in Table 14c, occurs when sentence-order is rearranged, arbitrarily, throughout a piece. Finally, we need our sentence-matching algorithm to perform all pairwise comparisons of sentences.
F.1 Refactors
To identify which sentences were intentionally moved rather than moved as a consequence of other document-level changes, we develop an iterative algorithm based on the idea that a refactor is an intentional sentence movement that creates an edge-crossing. Algorithm 2 gives our algorithm.
In English, our algorithm represents sentence
input :Article versions  $v_{old}, v_{new}$ , Match Threshold  $T$
output: maps  $m_{old\to new}$ ,  $m_{old} \gets m_{new}$  initialize;
$m_{old\to new},m_{old\leftarrow new} = \{\} ,\{\} ;$
// match  $\nu_{old} \rightarrow \nu_{new}$
for  $(i,s_i)\in v_{old}$  do
$$
d = \max  _ {s _ {j} \in v _ {\text {n e w}}} \operatorname {S i m} _ {\text {a s y m}} \left(s _ {i}, s _ {j}\right)
$$
$$
j = \operatorname {a r g m a x} _ {s _ {j} \in v _ {\text {n e w}}} \operatorname {S i m} _ {\text {a s y m}} \left(s _ {i}, s _ {j}\right)
$$
$$
m _ {o l d \rightarrow n e w} [ i ] = j \times \mathbb {1} [ d > T ]
$$
// match  $v_{old} \gets v_{new}$
for  $(j,s_j)\in v_{new}$  do
$$
d = \max  _ {s _ {i} \in v _ {\text {o l d}}} \operatorname {S i m} _ {\text {a s y m}} \left(s _ {j}, s _ {i}\right)
$$
$$
i = \arg \max  _ {s _ {i} \in v _ {\text {o l d}}} \operatorname {S i m} _ {\text {a s y m}} (s _ {j}, s _ {i})
$$
$$
m _ {o l d \leftarrow n e w} [ j ] = i \times \mathbb {1} [ d > T ]
$$
Algorithm 1: Asymmetrical sentence-matching algorithm. Input  $v_{old}$ ,  $v_{new}$  are lists of sentences, and output is an index mapper. If a sentence maps to 0 (i.e.  $d < T$ ), there is no match.  $Sim_{asym}$  is described in text.
matches between two article versions as a bipartite graph. We use a Binary Tree to recursively find all edge crossings in that graph. This idea is based off of the solution for an SPOJ challenge problem: https://www.spoj.com/problems/MSE06H/.43 We extend this problem to return the set of all edge crossings, not just the crossing number.
Then, we filter edge crossings to a candidate
42 Syntactic changes: synonyms are used, or phrasing is condensed, but substantially new information is not added
43Solution given here: https://github.com/akhiluanandh/SPOJ/blob/master/MSE06H.cpp.
Corpus	# Revisions	Language	Source	Goal
WiKed Error Corpus	12 million changed sentences	English	Wikipedia	Grammatical Error Correction (GEC)
WikiAtomic-Edits	43 million “atomic edits”41	8 languages	Wikipedia	Language Modeling
WiCoPaCo	70,000 changed sentences	French	Wikipedia	GEC and Sentence paraphrasing
WikiHow-ToImprove	2.7 million changed sentences	English	WikiHow	Version prediction, article improvement
NewsEdits	36.1 million changed sentences, 21.7 million added sentences, 14.2 million removed sentences. 72 million atomic edits.	English and French	22 media outlets	Language modeling, event sequencing, computational journalism
Table 13: A comparison of natural language revision history corpora.
set, applying the following conditions in order and stopping when there is only one edge crossing left: (1) edges that have the most number of crossings (2) edges that extend the most distance or (3) edges that move upwards. In most cases, we only apply the first and then the second conditions. In very rare cases, we apply all three. In rarer cases, we apply all three and still have multiple candidate edges. In those cases, we just choose the first edge in the candidate set. We continue removing edges until we have no more crossings.
G Annotation-Task Descriptions
G.1 Task: Sentence Matching
We give our annotators the following instructions:
The goal of this exercise is to help us identify sentences in an article-rewrite that contain substantially new information. To do this, you will identify which sentences match between two versions of an article.
Two sentences match if:
1. They are nearly the same, word-forword.
2. They convey the same information but are stylistically different.
3. They have slightly different information but have substantial overlap in meaning and narrative function.
Examples of Option 3 include (please see the "Examples" section for real examples):
1. Updating events.
- (Ex) The man was presumed missing.  $\rightarrow$  The man was found in his home.
- (Ex) The death count was at 23.  $\rightarrow$  50 were found dead.
- (Ex) The senators are still negotiating the details.  $\rightarrow$  The senators have reached a deal.
2. An improved analysis.
- (Ex) The president is likely seeking improved relations.  $\rightarrow$  The president is likely hoping that hardliners will give way to moderates, improving relations.
- (Ex) The storm, a Category IV, is expected to hit Texas.  $\rightarrow$  The storm, downgraded to Category III, is projected to stay mainly in the Gulf.
- (Ex) Analysts widely think the shock will be temporary.  $\rightarrow$  The shock, caused by widespread shipping delays, might last into December, but will ultimately subside.
3. A quote that is very similar or serves the same purpose.
- (Ex) "We knew we had to get it done." said Senator Murphy.  $\rightarrow$  "At the end of the day, no one could leave until we had a deal" said Senator Harris.
- (Ex) "It was gripping." said the by
Sent
Idx	Old
Tag	Old Version	New Version	New
Tag
1	M
1
C	The Bundesbank would only refer to an in-
terview Mr. Weidmann gave to Der
Spiegel magazine last week, in which he
said, “I can do my job best by staying
in office.”	The Bundesbank would only refer to an in-
terview published in Der Spiegel maga-
zine last week, in which Mr. Weidmann
said, “I can carry out my duty best if
I remain in office.”	M
1
C
(a) Demo 1: Word-Level atomic edit corrections applied when a sentence-level match is found, using the difflib Python library.
Sent
Idx	Old
Tag	Old Version	New Version	New
Tag
1	M
1 2
C	DALLAS—Ebola patient Thomas Eric Dun-
can told his fiancée the day he was diagnosed last week that he regrets exposing her to the deadly virus and had he known he was carrying Ebola, he would have “preferred to stay in Liberia and died than bring this to you,” a family friend said	DALLAS—Ebola patient Thomas Eric Dun-
can told his fiancée the day he was diagnosed last week that he regrets exposing her to the deadly virus.	M
1
2	Had he known he was carrying Ebola, he would have “preferred to stay in Liberia and died than bring this to you,” a family friend said.	M
1
C
(b) Demo 2: A sentence that is split results in the addition of a new sentence, but is matched with the previous dependent clause. Minimal word-level edits are applied.
Sent
Idx	Old
Tag	Old Version	New Version	New
Tag
1	M
2
U	“The mother, this was the first time seeing her son since he got to the States.	“She has not seen him for 12 years, and the first time she saw him was through a monitor,” said Lloyd.	M
2
U
2	M
1
U	She has not seen him for 12 years, and the first time she saw him was through a monitor,” said Lloyd.	“The mother, this was the first time seeing her son since he got to the States.”	M
1
U
3	“She wept, and wept, and wept.”	A
(c) Demo 3: Two features shown: (1) Refactoring, or order-swapping, makes sentences appear as though they have been deleted and then added. Swapped sentences are matched through their tags. (2) The last sentence is a newly added sentence and is not matched with any other sentence.
Table 14: Here we show demos of three tricky edge-cases and how our tagging scheme handles them. Old Tag annotates a Old Version relative to changes in the New Version (or “converts” the Old Version to the New Version). New Tag is the inverse. Tag components: Component 1: M, A, R. Whether the sentence is Matched, Added, or Removed. Component 2: Index. If Matched, what is the index of the sentence in version that it is matched to. Component 3: C, U. If Matched, is the sentence Changed or Unchanged.
stander.  $\rightarrow$  "I couldn't stop watching." said a moviegoer.
Two sentences do not match if:
1. They contain substantially different information.
2. They serve different narrative functions.
3. There is a much better match for one sentence somewhere else in the document.
- Two sentences might match even if they are in different parts of the document.
- One sentence can match with multiple other sentences, because that sentence might be split up into multiple sentences, each with similar information as parts of the original.
- Sentences don't have to match.
- Substantially new information, perspectives or narrative tools might be added in a new version.
- Substantially old information, perspectives or narrative tools might be removed from an old version.
input :Sentence matches,i.e. edges  $e$  between doc  $i$  and doc  $j$  ,as a list of tuples:  $e_i = (s_{i1},s_{i2}),e_j = (s_{j1},s_{j2})\ldots$
output: Minimal set of edges  $r$  that, when removed, eliminate all crossings.
// Subroutine identifies all edge crossings in  $e'$  and returns mapping  $c = \{e_i \to [e_j, e_k \dots], e_j \to \dots\}$  from each edge to all its crossings.
$c =$  getEdgeCrossings(e)
while  $|c| > 0$  do
Algorithm 2: Identifying Refactors. We define refactors as the minimal set of edge crossings in a bipartite graph which, when removed, remove all edge crossings.
Annotators completed the task by drawing lines between sentences in different versions of an article. An example is shown in Figure 8. We use highlighting to show when non overlapping sequences in the inbox, using simple lexical overlap. If the user mouses over a text block, they can see which words do no match between all textblocks on the other side. Although this might bias them towards our lexical matching algorithms, we do not see them beaking TB-medium. This was very helpful for reducing the cognitive overload of the task.
G.2 Task: Edit Actions