SlowGuess/NAACL_2022_OCR huggingface dataset:
We experiment with the following variations. For Task 2, we train with less data ( $n = 30,000$  version pairs) and more data ( $n = 150,000$  version pairs), balanced as described in Section 4.1, to test whether a larger dataset would help the models generalize better. We also experiment, for all tasks, with freezing the bottom 6 layers of the RoBERTa architecture (+Partially Frozen) to probe whether pretrained knowledge is helpful for these tasks. Additionally, we experiment giving the version number of the older version as an additional input feature alongside the text of the document (+Version).
Finally, for Tasks 2 and 3, we attempt to jointly model all subtasks using separate prediction heads for each subtask but sharing all other layers. We use uniform loss weighting between the tasks. Spangher et al. (2021a) showed that various document-level understanding tasks could benefit by being modeled jointly. For our tasks, we hypothesize that decisions around one operation might affect another: i.e. if a writer deletes many sentences in one draft they might also add sentences, so we test whether jointly modeling has a positive effect.
We do not consider any feature engineering on the input text, like performing event extraction (Ma et al., 2021), even though results in Section 3 show that certain types of edit-actions are more likely to contain events. We wish to establish a strong baseline and test whether models can learn salient features on their own. For more discussion on mod
13We used Wolf et al. (2020)'s version, found here https://huggingface.co/roberta-base.
14 Specifically, we initialize a 2-layer, 2-headed GPT2 transformer block to perform autoregressive contextualization.
Num. Additions	Num. Deletions	Num. Edits	Num. Refactors
Mac F1	Mic F1	Mac F1	Mic F1	Mac F1	Mic F1	Mac F1	Mic F1
Most Popular	19.8	25.0	25.6	47.8	21.9	32.0	39.2	64.5
Random	32.5	33.9	30.2	36.4	31.7	35.1	25.8	35.1
Baseline (n = 30,000)	22.1	27.9	25.6	46.5	21.4	30.6	35.2	64.5
(n = 150,000)	29.7	36.3	25.7	48.1	22.4	32.8	39.2	64.6
+Partially Frozen	52.2	54.0	44.8	59.0	49.3	53.1	44.3	65.6
+Contextual	50.7	52.2	41.0	57.4	50.8	54.8	45.0	64.3
+Version	52.0	54.5	45.3	59.8	49.9	53.7	43.8	63.1
+Multitask	46.7	50.2	28.2	48.4	42.1	49.5	40.3	55.1
Human	66.4	69.3	64.6	67.5	65.9	75.6	71.3	70.7
Table 5: Task 2 Benchmarks: Baseline model performance for document-level update tasks. Counts of Added, Deleted,Edited and Refactored sentences are binned into roughly equal-sized "low" ([0,1) sentences), "medium"([0,3) sentences), "high" ([3,∞) sentences) bins. Macro and Micro F1 calculated across bins. (Scores shown are median of 1,000 bootstrap resamples of the evaluation dataset.)
Additions	Sentence Operations	Refactors
Above (F1)	Below (F1)	Mac. F1	Mic. F1	Mac. F1	Mic. F1
Most Popular	0.0	0.00	18.1	20.2	34.7	53.3
Random	11.8	14.4	28.0	38.3	24.7	34.7
Baseline	8.3	0.1	36.5	61.9	35.2	54.2
+Partially Frozen	3.5	0.0	35.4	60.9	35.4	54.6
+Version	0.1	0.0	30.3	59.0	41.6	57.2
+Multitask.	0.0	0.0	27.5	57.8	39.5	54.8
Human	38.6	46.7	63.8	63.5	45.6	91.5
Table 6: Task 3 Benchmarks: Baseline model performance for sentence-Level tasks. Addition tasks are: "Was a sentence added below the target sentence?", "Was a sentence added above the target sentence?" Sentence Operations columns are three operations that occur on the target sentence: "Deletion", "Editing", "Unchanged". Refactor is binned into whether the target sentence is "Moved Up", "Moved Down" or "Unchanged". (Scores shown are median of 1,000 bootstrap resamples of the evaluation dataset.)
F1	F1
Most Popular	56.6	Baseline	60.8
Random	50.6	+Partially Frozen	66.0
Human	80.1	+Contextual	61.7
+Version	77.6
Table 7: Task 1 Benchmarks: Baseline model performance for next-version prediction task. Label is binary. (Scores are median of 1,000 bootstrap resamples of the evaluation dataset.)
eling choices and hyperparameter values, see Appendix D.
4.3 Human Performance
To evaluate how well human editors agree on edits, we design two human evaluation tasks and recruit 5 journalists with  $\geq 1$  year of editing experience at major U.S. and international media outlets.
Evaluation Task 1: We show users the text of an article and ask them whether or not there will be an update. Collectively, they annotate 100 articles. After completing each round, they are shown the
true labels. This evaluates Task 1.
Evaluation Task 2: We show users the sentences of an article, and they are able to move sentences, mark them as deleted or edited, and add sentence-blocks above or below sentences. They are not asked to write any text, only mark the high-level actions of "I would add a sentence," etc. Collectively they annotate 350 news articles. After each annotation, they see what edits actually happened. The raw output evaluates Task 3 and we aggregate their actions for each article to evaluate Task 2. They are instructed to use their expert intuition and they are interviewed afterwards on the strategies used to make these predictions. (See Appendix G for task guidelines and interviews).
As shown in Tables 5, 6, and 7, model-performance indicates that our tasks do range from easier (Task 1) to harder (Task 3). While our models show improvements above Random, and Most Popular
Topic (↑)	F1	Topic (↓)	F1	y (Add)	F1
U.S. Pol.	38.1	Local Pol.	66.8	[0, 1)	16.2
Business	48.4	War	61.8	[1, 5)	59.7
U.K. Pol.	50.4	Crime	58.3	[5, 100)	0.9
Table 8: Error Analysis: LDA (first two columns): Documents belonging to some topics are easier to predict than others. By label (last column): medium-range growth is easier to predict.
in almost all subtasks, a notable exception is Task 3's Addition subtasks, where the models do not clearly beat Random. We note that this was also the most difficult subtask for human evaluators.
We observe that  $+$  Partially Frozen increases performance on Task 2, boosting performance in all subtasks by  $\approx 10$  points. In contrast, it does not increase performance on Task 3, perhaps indicating that the subtasks in Task 3 are difficult for the current LLM paradigm. Although adding version embeddings (+Version) boosts performance for Task 1, it does not seem to measurably increase performance for the other tasks. Finally, performing Task 2 and 3 as multitask learning problems decreases performance for all subtasks.
In contrast, human evaluators beat model performance across tasks, most consistently in Task 2, with on average performance 20 F1-score points above Baseline models. On Task 3, human performance also is high relative to model performance. We observe that, despite Additions in Task 3 being the hardest task, as judged by human and model performance, humans showed a  $\approx 40$  point increase above model performance. Humans are also better at correctly identifying minority classes, with a wider performance gap seen for Macro F1 scores (i.e. see Sentence Operations, where the majority of sentences are unchanged).
4.5 Error Analysis
We perform an error analysis on the Task 2 task and find that there are several categories of edits that are easier to predict than others. We run Latent Dirichlet allocation on 40,000 articles, shown in Table 8.[15] We assign documents to their highest topic and find that articles covering certain news topics (like War) update in a much more predictable pattern than others (like Business), with a spread of over 26 F1-score points. Further, we find that certain edit-patterns are easier to differentiate, like articles that grow between 1-5 sentences (Table 8). This
show us ways to select for subsets of our dataset that are more standard in their update patterns.
The class imbalance of this dataset (Table 2) results in the Most Popular scoring highly. To mitigate this, we evaluate on balanced datasets. Class imbalanced training approaches (Li et al., 2020; Spangher et al., 2021a) might be of further help.
4.6 Evaluator Interviews
To better understand the process involved with successful human annotation, we conducted evaluator interviews. We noticed that evaluators first identified whether the main news event was still occurring, or if it was in the past. For the former, they tried to predict when the event would update.[16] For the latter, they considered discourse components to determine if an article was narratively complete and analyzed the specificity of the quotes.[17] They determined where to add information in the story based on structural analysis, and stressed the importance of the inverse pyramid for informational uncertainty: information later in an article had more uncertainty; if confirmed, it would be moved up in later versions.[18] Finally, they considered the emotional salience of events; if a sentence described an event causing harm, it would be moved up.[19]
Clearly, these tasks demand strong world-knowledge and common sense, as well and high-level discourse, structural and narrative awareness.[20] Combining these different forms of reasoning, our results show, is challenging for current language models, which, for many subtasks, perform worse than guessing. +Multitask performance actually decreases performance for both Task 2 and Task 3, indicating that these models learn features that do not generalize across subtasks. This contrasts with what our evaluators said: their decision to delete sentences often used the same reasoning as, and were dependent on, their decisions to add.
However, we see potential for improvement in these tasks. Current LLMs have been shown to identify common arcs in story-telling (Boyd et al., 2020), identify event-sequences (Han et al., 2019b) and reason about discourse structures (Spangher
16 The longer the timespan, the more information they predicted would be added between drafts.
$^{17}$ E.g. Generic quotes, say a public announcement, would be updated with specific, eye-witness quotes.
18One evaluator called this a "buried cause".
19See Appendix G for full interviews.
20Evaluators told us they "thought like the AP." The AP, or the Associated Press, has a styleguide (Goldstein, 1953) that many outlets use to guide their writing.
15Topic words shown in Appendix C.
et al., 2021a; Li et al., 2021). Further, for the ROCStories challenge, which presents four sentences and tasks the model with predicting the fifth (Mostafazadeh et al., 2017, 2016), LLMs have been shown to perform scene reconstruction (Tian et al., 2020b), story planning (Yao et al., 2019; Peng et al., 2018), and structural common sense reasoning (Chen et al., 2019). These are all aspects of reasoning that our evaluators told us they relied on. Narrative arcs in journalism are often standard and structured (Neiger and Tenenboim-Weinblatt, 2016), so we see potential for improvement.
A significant contribution of this work, we feel, is the introduction of a large corpus of news edits into revision-history research and the framing of questions around sentence-level edit-actions. Despite the centrality of news writing in NLP (Marcus et al., 1993; Carlson et al., 2003; Pustejovsky et al., 2003; Walker et al., 2006), we know of no academic corpus of news revision histories. Two works that analyze news edits to predict article quality (Tamori et al., 2017; Hitomi et al., 2017) do not release their datasets.[21] WikiNews[22] articles and editor-annotations have been used for document summarization (Bravo-Marquez and Manriquez, 2012), timeline synthesis (Zhang and Wan, 2017; Minard et al., 2016), word-identification (Yimam et al., 2017) and entity salience (Wu et al., 2020). However, we are not aware of any work using WikiNews revision histories. We did not include WikiNews because its collaborative community edits differ from professional news edits.
Since at least 2006, internet activists have tracked changes made to major digital news articles (Herrmann, 2006). NewsDiffs.org, NewsSniffer and DiffEngine are platforms which researchers have used to study instances of gender and racial bias in article drafts,[23] (Brisbane, 2012; Burke, 2016; Jones and Neubert, 2017; Fass and Main, 2014) shifting portrayals of social events, (Johnson et al., 2016) and lack of media transparency (Gourarie, 2015). These tools collect article versions from RSS feeds and the Internet Archive. Major newspa
pers $^{24}$  and thousands of government websites $^{25}$  are being analyzed. We use DiffEngine and NewsSniffer to construct NewsEdits.
Wikipow (Anthonio et al., 2020; Bhat et al., 2020) and Source Code Diffs (Tan and Bockisch, 2019; Shen et al., 2019; Tsantalis et al., 2018; Silva and Valente, 2017; Marrese-Taylor et al., 2020; Xu et al., 2019) use revision histories from domains and for purposes different than ours. Many tasks have benefited from studying Wikipedia Revisions, like text simplification (Yatskar et al., 2010), textual entailment (Zanzotto and Pennacchiotti, 2010), discourse learning (Daxenberger and Gurevych, 2013) and grammatical error correction (Faruqui et al., 2018). However, most tasks focus on word-level edit operations to explore sentence-level changes. Ours focuses on sentence-level operations to explore document-level changes. Research in Student Learner Essays focuses on editing revisions made during essay-writing (Leacock et al., 2010; Wang et al., 2020; Zhang, 2020; Zhang and Litman, 2015). Researchers categorize the intention and effects of each edit (Zhang et al., 2017; Afrin et al., 2020), but do not try to predict edits.
In this work, we have introduced the first large-scale dataset of news edits, extracted edit-actions, and shown that many were fact-based. We showed that edit-actions are predictable by experts but challenging for current LM-backed classifiers. Going forward, we will develop a schema describing the types of edits. We are inspired by the Wikipedia Intentions schema developed by Yang et al. (2017), and are working in collaboration with journalists to further clarify the differences. This development will help to clarify the nature of these edits as well as focus further directions of inquiry.
We are grateful to Amanda Stent, Sz-Rung Shiang, Gabriel Kahn, Casey Williams, Meg Robbins, I-Hung Hsu, Mozhdeh Gheini, Jiao Sun and our anonymous reviewers for invaluable feedback. Spangher is grateful for Bloomberg for supporting this research with a PhD fellowship. May is supported by DARPA Contract FA8750-19-2-0500.
$^{24}$ https://twitter.com/i/lists/821699483088076802
25https://envirodatagov.org/federal-environmental-web-tracker-about-page/
$^{21}$  Datasets could not be released due to copyright infringement, according to the authors in response to our inquiry.
22https://en.wikipedia.org/wiki/Main_Page
23http://www.newsdiffs.org/ diff/192021/192137/www.nytimes. com/2013/03/31/science/space/ yvonne-brill-rocket-scientist-dies-at-88.html
8.1 Dataset
We received permission from the original owners of the datasets, NewsSniffer and DiffEngine. Both sources are shared under strong sharing licenses. NewsSniffer is released under an AGPL-3.0 License,[26] which is a strong "CopyLeft" license. DiffEngine is released under an Attribution-NoDerivatives 4.0 International license.[27]
Our use is within the bounds of intended use given in writing by the original dataset creators, and is within the scope of their licensing.
8.2 Privacy
We believe that there are no adverse privacy implications in this dataset. The dataset comprises news articles that were already published in the public domain with the expectation of widespread distribution. We did not engage in any concerted effort to assess whether information within the dataset was libelious, slanderous or otherwise unprotected speech. We instructed annotators to be aware that this was a possibility and to report to us if they saw anything, but we did not receive any reports. We discuss this more below.
8.3 Limitations and Risks
The primary theoretical limitation in our work is that we did not include a robust non-Western language source; indeed, our only two languages were English and French. We tried to obtain sources in non-Western newspapers and reached out to a number of activists that use the DiffEngine platform to collect news outside of the Western world, including activists from Russia and Brazil. Unfortunately, we were not able to get a responses.
Thus, this work should be viewed with that important caveat. We cannot assume a priori that all cultures necessarily follow this approach to breaking news and indeed all of the theoretical works that we cite in justifying our directions also focus on English-language newspapers. We provide documentation in the Appendix about the language, source, timeline and size of each media outlet that we use in this dataset.
One possible risk is that some of the information contained in earlier versions of news articles was updated or removed for the express purpose that it
was potentially unprotected speech: libel, slander, etc. We discussed this with the original authors of NewsSniffer and DiffEngine. During their years of operation, neither author has received any requests to take versions down. Furthermore, instances of First Amendment lawsuits where the plaintiff was successful in challenging content are rare in the U.S. We are not as familiar with the guidelines of protected speech in other countries.
Another risk we see is the misuse of this work on edits for the purpose of disparaging and denigrating media outlets. Many of these news tracker websites have been used for noble purposes (e.g. holding newspapers accountable for when they make stylistic edits or try to update without giving notice). But we live in a political environment that is often hostile to the core democracy-preserving role of the media. We focus on fact-based updates and hope that this resource is not used to unnecessarily find fault with media outlets.
8.4 Computational Resources
The experiments in our paper require computational resources. All our models run on a single 30GB NVIDIA V100 GPU, along with storage and CPU capabilities provided by AWS. While our experiments do not need to leverage model or data parallelism, we still recognize that not all researchers have access to this resource level.
We use Huggingface RoBERTa-base models for our predictive tasks, and release the code of all the custom architectures that we construct at https://github.com/isi-nlp/NewsEdits.git. Our models do not exceed 300 million parameters.
8.5 Annotators
We recruited annotators from professional journalism networks like the NICAR listserve.[28] All the annotators consented to annotate as part of the experiment, and were paid $1 per task, above the highest minimum wage in the U.S. Of our five annotators, three are based in large U.S. cities, one lives in a small U.S. city and one lives in a large Brazilian city. Four annotators identify as white and one identifies as Latinx. Four annotators identify as male and one identifies as female. This data collection process is covered under a university IRB. We do not publish personal details about the annotations, and their interviews were given with consent
28 https://www.ire.org/training/conferences/nicar-2021/
$^{26}$ https://opensource.org/licenses/AGPL-3.0
27 https://creativecommons.org/licenses/by-nd/4.0/
and full awareness that they would be published in full.
Sheikh Abujar, Mahmudul Hasan, and Syed Akhter Hossain. 2019. Sentence similarity estimation for text summarization using deep learning. In Proceedings of the 2nd International Conference on Data Engineering and Communication Technology, pages 155-164. Springer.
Tazin Afrin, Elaine Lin Wang, Diane Litman, Lindsay Clare Matsumura, and Richard Correnti. 2020. Annotation and classification of evidence and reasoning revisions in argumentative writing. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 75-84.
Talita Anthonio, Irshad Bhat, and Michael Roth. 2020. wikiHowToImprove: A resource and analyses on edits in instructional texts. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5721-5729, Marseille, France. European Language Resources Association.
Alyssa Appelman and Kirstie Hettinga. 2015. Do news corrections affect credibility? not necessarily. *Newspaper Research Journal*, 36(4):415-425.
Irshad Bhat, Talita Anthonio, and Michael Roth. 2020. Towards modeling revision requirements in wikiHow instructions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8407-8414, Online. Association for Computational Linguistics.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993-1022.
Ryan L Boyd, Kate G Blackburn, and James W Pennebaker. 2020. The narrative arc: Revealing core narrative structures through text analysis. Science advances, 6(32):eaba2196.
Felipe Bravo-Marquez and Manuel Manriquez. 2012. A zipf-like distant supervision approach for multi-document summarization using wiknews articles. In International Symposium on String Processing and Information Retrieval, pages 143-154. Springer.
Arthur S. Brisbane. 2012. Insider's view of changes, from outside. The New York Times.
Austin Burke. 2016. Newsdiffs: A tool for tracking changes to online news articles - vr research - public records research: Opposition research.
Lynn Carlson, Daniel Marcu, and Mary Ellen Okuowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Current and new directions in discourse and dialogue, pages 85-112. Springer.
Jiaao Chen, Jianshu Chen, and Zhou Yu. 2019. Incorporating structured commonsense knowledge in
story completion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6244-6251.
Qingyu Chen, Sun Kim, W John Wilbur, and Zhiyong Lu. 2018. Sentence similarity measures revisited: ranking sentences in pubmed documents. In Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, pages 531-532.
Prafulla Kumar Choubey, Aaron Lee, Ruihong Huang, and Lu Wang. 2020. Discourse as a function of event: Profiling discourse structure in news articles around the main event. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5374-5386, Online. Association for Computational Linguistics.
Sarah Cohen, James T Hamilton, and Fred Turner. 2011. Computational journalism. Communications of the ACM, 54(10):66-71.
Johannes Daxenberger and Iryna Gurevych. 2012. A corpus-based study of edit categories in featured and non-featured Wikipedia articles. In Proceedings of COLING 2012, pages 711-726, Mumbai, India. The COLING 2012 Organizing Committee.
Johannes Daxenberger and Iryna Gurevych. 2013. Automatically classifying edit categories in wikipedia revisions. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 578-589.
Mats Ekstrom, Amanda Ramsalv, and Oscar Westlund. 2021. The epistemologies of breaking news. Journalism Studies, 22(2):174-192.
Manaal Faruqui, Ellie Pavlick, Ian Tenney, and Dipanjan Das. 2018. WikiAtomicEdits: A multilingual corpus of Wikipedia edits for modeling language and discourse. pages 305-315.
John Fass and Angus Main. 2014. Revealing the news: How online news changes without you noticing. Digital Journalism, 2(3):366-382.
Emilio Ferrara. 2017. Disinformation and social bot operations in the run up to the 2017 french presidential election. arXiv preprint arXiv:1707.00086.
Peter Kin-Fong Fong and Robert P Biuk-Aghai. 2010. What did they do? deriving high-level edit histories in wikis. In Proceedings of the 6th International Symposium on Wikis and Open Collaboration, pages 1-10.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Norm Goldstein. 1953. The Associate Press Rules Regulations and General Orders.