SlowGuess/NAACL_2022_OCR huggingface dataset:
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. 2022. TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models. abs/2204.14211.
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2021. Towards Continual Knowledge Learning of Language Models.
Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal Analysis of Language through Neural Language Models. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science.
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. 2021. Mind the Gap: Assessing Temporal Generalization in Neural Language Models. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS).
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual knowledge in gpt. arXiv preprint arXiv:2202.05262.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2021. Fast Model Editing at Scale. abs/2110.11309.
Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, and Greg Durrett. 2021. CREAM: A Dataset for Commonsense Reasoning over Entity Knowledge. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track.
Denis Paperno, Germán Kruszewski, Angeliki Lazari-dou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and R. Fernández. 2016. The lambada dataset: Word prediction requiring a broad discourse context. *ArXiv*, abs/1606.06031.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Roktäschel, and Sebastian
Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1-67.
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable Neural Networks. In International Conference on Learning Representations (ICLR).
Derry Tanti Wijaya and Reyyan Yeniterzi. 2011. Understanding semantic change of words over centuries. In International Workshop on Detecting and Exploiting Cultural Diversity on the Social Web, DETECT.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Michael J.Q. Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371-7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363.
A Examples of ECBD Sentences
See Table 4 for examples of masked sentences in the ECBD data.
B Perplexity per year
See Table 5 for a more fine-grained view of the results in Table 3.
C Perplexity per span type
See Table 6 for a breakdown of the perplexity that T5 achieves on different types of spans, showing that random spans are generally higher perplexity than NP spans but that adding definitions can help both.
D Recall@10
LMs can be evaluated on recall@10, i.e., a binary score indicating if model's top ten predictions contains the gold masked span  $m_y$ . For T5, we first generate sequences using beam search (we choose beam size = 100 in our experiments). Then we take the top ten unique sequences and extract the text spans between <extra_id_0> and <extra_id_1> as predictions. Table 7 reports recall@10 on each subset. Table 8 list recall@10 per span type for each subset.
We only explore recall on T5, since it is not obvious how to compute it for the other two models. For BART, we can extract the predicted span by aligning the model's prediction with the gold context, assuming that it starts to copy from the input right context at some point. However, in some cases, we found that the generated right context does not match with the gold right context; it's unclear how to be handle this. For GPT-Neo, since it is a left-to-right LM, extracting the predicted span would require conditioning on the span length, which is information that T5 does not have access to. As a result, we do not report recall@10 for these models.
E Data Licensing
The Wikipedia text we used is licensed under CC BY-SA. Our use of Wikipedia, constructing a dataset which we will make publicly available under the same license, is consistent with the terms of the license.
All experiments were conducted using an NVIDIA Quadro RTX 8000. We only evaluate existing mod
Masked Sentence	Span Type	Origin Year
At 18:00 UTC on August 16, after Grace exited the Dominican Republic, [MASK] were lifted.
Answer: “all tropical storm watches”	NP	2021
AirTags can be [MASK] the Find My app.
Answer: “interacted with using”	RANDOM	2021
British tabloid “The Sun” is credited with the first headline use of ‘Megxit’ on [MASK] 2020.
Answer: “9 January”	NP	2020
The iPhone SE features an [MASK] a glass front and back.
Answer: “aluminum frame, paired with”	RANDOM	2020
The GPT-2 model has [MASK], and was trained on a dataset of 8 million web pages.
Answer: “1.5 billion parameters”	NP	2019
The epicenter of the 2019 Albania earthquake [MASK] kilometers from Tirana to the Northwest.
Answer: “was about 30”	RANDOM	2019
On November 12, 2019, Maverick City Music released [MASK], "Maverick City, Vol. 2".
Answer: “their follow-up EP”	NP	2018
Austin FC are the operators of a newly-[MASK].
Answer: “built stadium at McKalla Place”	RANDOM	2018
The first quarter of Super Bowl LI was [MASK] with each team punting twice.
Answer: “a scoreless defensive match”	NP	2017
Hurricane Irma was the top Google searched term in [MASK] in 2017.
Answer: “the U.S. and globally”	RANDOM	2017
Table 4: Examples selected from the 2017-2021 subsets of ECBD.
els on our datasets and did not do any finetuning. One evaluation experiment typically takes 15 minutes to complete. For T5 experiments, we use Hugging Face's Transformer package (Wolf et al., 2020).
POPULAR	2017	2018	2019	2020	2021
Type: seq-to-seq	T5 Large	Size: 770M
ORIGINAL	13.02	15.28	14.78	16.43	19.81	18.60
NO ENT	18.28	22.28	21.70	23.35	28.41	23.26
RANDOM DEF.	12.10	14.56	13.54	15.10	17.42	17.17
DEFINITION	11.04	12.27	10.76	12.34	14.07	12.61
Δ(ORIG. → RAND.)	-0.92	-0.72	-1.24	-1.33	-2.39	-1.43
Δ(ORIG. → DEF.)	-1.98	-3.01	-4.02	-4.09	-5.74	-5.99
Type: seq-to-seq	BART Large	Size: 406M
ORIGINAL	22.70	22.74	19.52	21.00	28.03	30.53
NO ENT	33.33	33.58	28.25	29.67	39.56	38.57
RANDOM DEF.	27.69	27.11	23.80	25.96	32.41	36.86
DEFINITION	21.01	18.97	16.58	17.35	22.12	21.72
Δ(ORIG. → RAND.)	+4.99	+4.37	+4.28	+4.96	+4.38	+6.33
Δ(ORIG. → DEF.)	-1.69	-3.77	-2.94	-3.65	-5.91	-8.81
Type: left-to-right	GPT-Neo	Size: 1.3B
ORIGINAL	28.61	28.91	27.55	26.63	33.15	33.81
NO ENT	54.01	52.88	53.95	46.44	53.89	57.61
RANDOM DEF.	39.46	41.75	43.15	37.41	45.30	47.32
DEFINITION	23.19	20.47	18.00	18.68	22.17	22.69
Δ(ORIG. → RAND.)	+10.85	+12.84	+15.6	+10.78	+12.15	+13.51
Δ(ORIG. → DEF.)	-5.42	-8.44	-9.55	-7.95	-10.98	-11.12
Table 5: Results of T5, BART, and GPT-Neo on the test set, showing perplexity  $(\downarrow)$  for each subset.
Input Type	2017	2018	2019	2020	2021
NP	RAND	NP	RAND	NP	RAND	NP	RAND	NP	RAND
ORIGINAL	5.86	7.33	5.81	7.51	6.11	7.29	5.92	7.63	6.23	7.31
NO ENT	5.90	8.02	5.78	8.56	5.99	8.31	6.75	9.36	7.28	9.21
RANDOM DEF.	5.59	6.60	5.54	6.84	5.77	6.60	5.70	6.98	6.01	6.65
DEFINITION	4.96	5.98	4.98	6.02	5.12	5.85	5.14	6.13	5.13	5.82
Δ(ORIG. → RAND.)	-0.27	-0.73	-0.27	-0.67	-0.34	-0.69	-0.22	-0.65	-0.22	-0.66
Δ(ORIG. → DEF.)	-0.90	-1.35	-0.83	-1.49	-0.99	-1.44	-0.78	-1.50	-1.10	-1.49
Table 6: Results of T5 model (pre-trained with data from 2019) on the dev set with perplexity  $\left( \downarrow \right)$  per span type.
POPULAR	2017	2018	2019	2020	2021
Type: seq-to-seq	T5 Large	Size: 770M
ORIGINAL	28.2	25.4	27.4	27.7	20.8	23.0
No ENT	23.8	21.6	23.2	23.7	19.5	21.5
RANDOM DEF.	28.4	24.3	28.5	26.8	21.4	23.2
DEFINITION	29.3	28.4	31.8	28.2	24.8	26.1
Δ(ORIG. → RAND.)	+0.2	-1.1	+1.1	-0.9	+0.6	+0.2
Δ(ORIG. → DEF.)	+1.1	+3.0	+4.4	+0.5	+4.0	+3.1
Table 7: Results of T5 on the test set, showing recall@10 (↑) for each subset.
Input Type	2017	2018	2019	2020	2021
NP	RAND	NP	RAND	NP	RAND	NP	RAND	NP	RAND
Type: seq-to-seq	T5 Large	Size: 770M
ORIGINAL	30.3	20.0	31.8	20.2	29.3	22.0	30.1	19.8	29.3	21.6
NO ENT	27.2	18.8	28.1	16.7	26.2	18.1	26.8	16.7	25.9	18.2
RANDOM DEF.	31.8	20.8	32.8	19.9	29.8	21.6	31.3	20.5	29.5	21.6
DEFINITION	34.1	22.8	35.9	22.8	33.0	24.9	33.7	23.0	32.7	25.2
Δ(ORIG. → RAND.)	+1.5	+0.8	+1.0	-0.3	+0.5	-0.4	+1.2	+0.7	+0.2	+0.0
Δ(ORIG. → DEF.	+3.8	+2.8	+4.1	+2.6	+3.7	+2.9	+3.6	+3.2	+3.4	+3.6
Table 8: Results of T5 model (pre-trained with data from 2019) on the dev set with recall@10 (↑) per span type.