SlowGuess/NAACL_2022_OCR huggingface dataset:
The data consists of the text excerpts written between the years 1798 and 1963. The mean publication year of the text excerpts is 1891. Excerpts between the years 1833 and 1925 make up about  $96\%$  of the data in the train set (cf. Figure 2a), but only  $85\%$  in the dev and test sets, which are more uniform (due to balancing described in Section 4,
10https://gonito.net/challenge/challenging-america-geo-prediction
11https://gonito.net/challenge/challenging-america-word-gap-prediction
(a) Excerpt counts vs. publication dates in train set.
(c) Excerpt counts vs. publication dates in dev/test set.
Figure 2: Statistics for the RetroTemp challenge
(b) Average excerpt length vs. publication dates in train set.
(d) Average excerpt length vs. publication dates in dev/test set.
cf. Figure 2c). There are 432000 excerpts in the train set, 10500 in the dev set and 8500 in the test set. These numbers are consistent across the challenges. The average excerpt length is 1745 characters with 323.8 words, each one containing from 150 words up to 583 words.
The length of each text in the excerpts seems to have a negative correlation with publication date – the later the text was published, the shorter snippet text (on average) it contains (see Figure 2b and 2d).
Strong baselines for all three tasks are available at the Gonito evaluation platform. The baselines (see Tables 2 and 3) include, for each model, its score in the appropriate metric as well as the Git SHA1 reference code in the Gonito benchmark (in curly brackets). Reference codes can be used to access any of the baseline solutions at http: //gonito.net/q.
We distinguish between self-contained submissions, which use only data provided in the task, and non-self-contained submissions, which use external data, e.g. publicly available pre-trained transformers. Our baselines take into account only textual features.
More detailed analysis of the baseline performance is given in Appendix C. The current top performing models have the most difficulty with
texts which (1) are older, (2) contain OCR noise, (3) come from less popular locations (especially, in the west).
8.1 RetroTemp and RetroGeo
The baseline solutions for RetroTemp and RetroGeo were prepared similarly. RetroGeo requires two values (latitude and longitude) – we treat them separately and train two separate regression models for them.
For the self-contained models we provide the mean value from the train test, the linear regression based on TF-IDF and the BiLSTM (bidirectional long short-term memory) method.
For non-self-contained submissions, we incorporate RoBERTa (Liu et al., 2019) models released in two versions: base (125M params) and large (355M params). The output features are averaged, and the linear layer is added on top of this. Both RoBERTa and the linear layer were fine-tuned during training.
The best self-contained models are BiLSTM submissions in both tasks. Non-self-contained submissions result in much higher scores than self-contained models. In both tasks, RoBERTa-large with linear layer provides better results than RoBERTa-base.
For the RetroTemp challenge we also provide results obtained with the RoBERTa model pre
trained from scratch (see Section 5). Even though the model without time-related prefix was used, the results are significantly better than the original RoBERTa Base: the confidence intervals obtained with bootstrap sampling are, respectively,  $10.81 \pm 0.21$  and  $12.10 \pm 0.22$  (single runs are reported).
Hyperparameter setup is described in Appendix B.
8.2 RetroGap
For non-self-contained submissions, we applied RoBERTa in base and large version without any fine-tuning. Since standard RoBERTa training does not incorporate any data, but text, we did not include temporal metadata during inference.
For self-contained submissions, we applied RoBERTa Challam base both in version with a date and without a date.
RoBERTa ChallAm base with date is better than RoBERTa ChallAm base without date. This means the incorporation of temporal metadata has a positive impact on the MLM task. Both self-contained submissions are better than the standard RoBERTa base, so our models trained on historical data performs better than models trained on regular data if the same base model size is considered. Since we did not train RoBERTa ChallAm large, we cannot confirm this holds true, when it comes to large RoBERTa models. The standard RoBERTa large is the best performing model, so in this case, a larger model is better even if not trained on the data from different domain.
9 Ethical issues
We share the data from Chronicling America, following the statement of the Library of Congress: "The Library of Congress believes that the newspapers in Chronicling America are in the public domain or have no known copyright restrictions."12
Historical texts from American newspapers may be discriminatory, either explicitly or implicitly, particularly regarding race and gender. Recent years have seen research on the detection of discriminatory texts. In (Xia et al., 2020) adversarial training is used to mitigate racial bias. In (Field and Tsvetkov, 2020) the authors "take an unsupervised approach to identifying gender bias against women at a comment level and present a model that can
surface text likely to contain bias." The most recent experiments on the topic ((Caselli et al., 2021), (Aluru et al., 2020)) result in re-trained BERT models for abusive language detection in English. We use one of them, DeHateBERT (Aluru et al., 2020), to detect the abusive texts in the ChallAm dataset. We tagged items that either (1) are marked as abusive speech by DeHateBERT with the probability greater than 0.75 or (2) contain words from a list of blocked words. The fraction of detected texts was  $2.04 - 2.40\%$  (depending on the challenge and set). The tags along with the probabilities are available in the hate-speech-info.tsv files for each test directory.
Note that temporal and geospatial metadata might constitute useful features in future work on better detection of hate speech in historical texts.
10 Conclusions
This paper has introduced a challenge based on OCR excerpts from the Chronicling America portal. The challenge consists of three tasks: guessing the publication date, guessing the publication location, and filling a gap with a word. We propose baseline solutions for all three tasks.
Chronicling America is an ongoing project, as we define our challenge in such a way that it can easily evolve in parallel with the development of Chronicling America. Firstly, any new materials appearing on the portal can be automatically incorporated into our challenge. Secondly, the challenge is open for five yet undefined ML tasks.
This work was partially supported by the Cyfrowa Infrastruktura Badawcza dla Humanistyki i Nauk o Sztuce DARIAH-PL project (POIR.04.02.00-00-D006/20).
Sai Saket Aluru, Binny Mathew, Punyajoy Saha, and Animesh Mukherjee. 2020. Deep learning models for multilingual hate speech detection. *ArXiv preprint*, abs/2004.06465.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric
<sup>12</sup>https://chroniclingamerica.loc.gov/about
Table 2: Baseline results for the RetroTemp/Geo challenges. * indicates non-self-contained models.
Model	RetroTemp	RetroGeo
git ref	RMSE	git ref	Haversine
mean from train	{fbf19b}	31.50	{766824}	1321.47
tfidf with linear regression	{63c8d4}	17.11	{8acd61}	2199.36
BiLSTM	{f7d7ed}	13.95	{d3d376}	972.71
RoBERTa Base + linear layer*	{1159e6}	12.07	{08412c}	827.13
RoBERTa Large + linear layer*	{2e79c8}	8.15	{7a21dc}	651.20
RoBERTa ChallAm Base + linear layer*	{d0cff4}	10.80	—
Table 3: Baseline results for the RetroGap challenge. * indicates non-self-contained models.
Model	git ref	Perplexity
RoBERTa base (no fine-tune)	{166e03}	72.10
RoBERTa large (no fine-tune)	{bf5171}	52.58
RoBERTa ChallAm Base (without date)*	{f96da0}	56.64
RoBERTa ChallAm Base (with date)*	{3ebfc0}	53.76
Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Tommaso Caselli, Valerio Basile, Jelena Mitrovic, and Michael Granitzer. 2021. HateBERT: Retraining BERT for abusive language detection in English. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 17-25, Online. Association for Computational Linguistics.
Mehmet Cibaroglu. 2019. Post-truth in social media. 6:87-99.
Maribeth Clark. 2014. A survey of online digital newspaper and genealogy archives: Resources, cost, and access. Journal of the Society for American Music, 8:277-283.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and
William W. Cohen. 2021. Time-aware language models as temporal knowledge bases.
Rui Dong and David Smith. 2018. Multi-input attention for unsupervised OCR correction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2363-2372, Melbourne, Australia. Association for Computational Linguistics.
Anjalie Field and Yulia Tsvetkov. 2020. Unsupervised discovery of implicit gender bias. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 596-608, Online. Association for Computational Linguistics.
Filip Gralinski, Rafal Jaworski, Łukasz Borchmann, and Piotr Wierzchoń. 2016. Gonito.net - open platform for research competition, cooperation and reproducibility. In Antonio Branco, Nicoletta Calzolari, and Khalid Choukri, editors, Proceedings of the 4REAL Workshop: Workshop on Research Results Reproducibility and Resources Citation in Science and Technology of Language, pages 13-20.
Filip Gralinski and Piotr Wierzchoń. 2018. RetroC—A Corpus for Evaluating Temporal Classifiers. In Human Language Technology. Challenges for Computer Science and Linguistics. 7th Language and Technology Conference, LTC 2015, pages 101-111. Springer.
Filip Graliński, Anna Wróblewska, Tomasz Stanisławek, Kamil Grabowski, and Tomasz Górecki. 2019. GEval: Tool for debugging NLP datasets and models. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural
Networks for NLP, pages 254-262, Florence, Italy.  
Association for Computational Linguistics.
Filip Graliński. 2013. Polish digital libraries as a text corpus. In Proceedings of 6th Language & Technology Conference, pages 509-513, Poznan. Fundacja Uniwersytetu im. Adama Mickiewicza.
Filip Graliński. 2017. (Temporal) language models as a competitive challenge. In Proceedings of the 8th Language & Technology Conference, pages 141-146. Fundacja Uniwersytetu im. Adama Mickiewicza w Poznaniu.
Filip Gralinski. 2019. Against the Arrow of Time. Theory and Practice of Mining Massive Corpora of Polish Historical Texts for Linguistic and Historical Research. Wydawnictwo Naukowe UAM, Poznan.
Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar. 2019. ICDAR2019 competition on scanned receipt OCR and information extraction. 2019 International Conference on Document Analysis and Recognition (ICDAR).
Benjamin Lee, Jaime Mears, Eileen Jakeway, Meghan Ferriter, Chris Adams, Nathan Yarasavage, Deborah Thomas, Kate Zwaard, and Daniel Weld. 2020. The newspaper navigator dataset: Extracting and analyzing visual content from 16 million historic newspaper pages in Chronicling America.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv preprint, abs/1907.11692.
Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K Gray, Joseph P Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, et al. 2011. Quantitative analysis of culture using millions of digitized books. science, 331(6014):176-182.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Thi-Tuyet-Hai Nguyen, Adam Jatowt, Mickael Coustaty, Nhu-Van Nguyen, and Antoine Doucet. 2019. Deep statistical analysis of OCR errors for effective post-OCR processing. In Proceedings of the 18th Joint Conference on Digital Libraries, JCDL '19, page 29-38. IEEE Press.
Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1756-1765, Vancouver, Canada. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. *ArXiv preprint*, abs/1910.10683.
Guy D. Rosin, Ido Guy, and Kira Radinsky. 2021. Time masking for temporal language models.
Zejiang Shen, Kaixuan Zhang, and Melissa Dell. 2020. A large dataset of historical Japanese documents with complex layouts. ArXiv preprint, abs/2004.08686.
Tomasz Stanisławek, Filip Graliński, Anna Wróblewska, Dawid Lipinski, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, and Przemysław Biecek. 2021. Kleister: Key information extraction datasets involving long documents with complex layouts. In Document Analysis and Recognition - ICDAR 2021, pages 564-579, Cham. Springer International Publishing.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020. Demoting racial bias in hate speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 7-14, Online. Association for Computational Linguistics.
A Procedure for selecting text excerpts
The OCR text follows the newspaper layout, which is defined by the following entities: page, column, line. Each entity has  $x_0$ ,  $y_0$ ,  $x_1$ ,  $y_1$  coordinates of text in the DjVu document. Still, various errors may occur in the OCR newspaper layout (e.g. two columns may be split into one). We intend to select only excerpts which preserve the correct output. To this end, we select only excerpts that fulfill the following conditions:
1. There are between 150 and 600 text tokens in the excerpt. The tokens are words separated by whitespaces.
2. The  $y$  coordinates of each line are below the  $y$  coordinates of the previous line.
3. The  $x_0$  coordinate of each line does not differ by more than  $15\%$  from the  $x_0$  coordinate of the previous line.
4. The  $x_{1}$  coordinate is not shifted to the right more than  $15\%$  from the  $x_{1}$  coordinate of the previous line.
If the newspaper edition contains no such excerpts, we reject it. If there is more than one such excerpt, we select one excerpt using a stable pseudo-random procedure based on the newspaper edition ID.
This procedure produces text excerpts with images consisting of OCR texts only. The excerpts are downsized to reduce the size to an appropriate degree to maintain good quality. We do not pre-process images in any other way, so excerpts may have different sizes, height-to-width ratios, and colors.
B Hyperparameter setup
Hyperparameters were determined on the development set, training on a limited number of examples. In particular, for fine-tuning RoBERTa models the following hyperparameters were used:
- optimizer: AdamW
learning rate: 0.000001
- batch size: 4
early-stopping patience: 3
warm-up steps: 10000
C Analysis of the best baselines
See Table 4 and 5 for the list of top 30 features correlating most with, respectively, the worst and bad results in ChallAm challenges (as returned by the GEval tool with the option -worst-features -numerical-features (Gralinski et al., 2019)). The features are tokens within the input (in:), expected output (exp:) and the actual output (out:), or numerical features such as high/low value ( $:=+/:=-$ ) or length/shortness of a text ( $:+/\#:-\#$ ).
As can be seen the bottleneck for the current best model is due to:
- old texts (:= - in RetroTemp),
- OCR noise (cf. short words such  $n_i$ ,  $ol$ ,  $j$  or punctuation marks likely to be introduced by OCR misrecognitions),
- less popular publication locations (especially far west).
Obviously, year references (1902, 1904) make it easy to guess the publication texts (in RetroTemp), whereas in RetroGap some non-content words such as the, and, of are easy to guess for the language model (even if their garbaged form, e.g. ot, ol, needs to be accounted for in the probability distribution).
Table 4: Features highly correlating with bad results
RetroTemp	RetroGeo	RetroGap
exp:=-	exp:={{+}}	exp:={{+}}
in(Text):;	in(Text):=+	exp:,
in(Text]:nold	exp:-100.445882	exp:.
in(Text):ni	exp:39.78373	out:.
in(Text):she	exp:-115.763123	out:-
out:=-	exp:40.832421	in<LeftContext>:n
in(Text):"	exp:-93.101503	out:,
in(Text):aim	exp:44.950404	out:,
in(Text):sav-	exp:-112.730038	out:'
in(Text):ii	exp:46.395761	out:*
in(Text):rifle	exp:-97.337545	in<RightContext>:*
in(Text):hut	exp:37.692236	in<LeftContext>:\(>\)
in(Text):!	exp:-76.062727	out:={{-}}
in(Text):guilt	exp:39.697887	in<RightContext>:\(>\)
in(Text):nLeave	exp:-106.487287	in<LeftContext>:i
in(Text):ol	exp:31.760037	out:!!
in(Text):cold	exp:-81.772437	exp:,
in(Text):contemplate	exp:24.562557	in<LeftContext>:*
in(Text):nI	exp:-71.880373	in<RightContext>:l
in(Text):thee	exp:44.814771	out:"
in(Text):Ben-	out:={{+}}	out:|
in(Text):1945	exp:-135.313889	in<LeftContext>:l
in(Text):God	exp:59.458333	out:1
in(Text):it	exp:-112.077346	exp:"
in(Text):noi	exp:33.448587	in<LeftContext>:<
in(Text):man's	exp:-122.330062	in<LeftContext>:-
in(Text):Roman	exp:47.603832	in<RightContext>:\(>\)
in(Text):I	exp:-112.942369	out:i
in(Text):Henry	exp:46.128794	out:j
in(Text):nford	exp:-90.184225	in<LeftContext>:e
Table 5: Features highly correlating with good results
RetroTemp	RetroGeo	RetroGap
in(Text):Democratic	exp:44.007274	out:Of
in(Text):defeat	exp:-80.85675	out:The
in(Text):Secretary	exp:40.900892	out:ana
in(Text):notice	exp:-77.804161	out:aud
in(Text):July	exp:39.4301	out:by
in(Text):General	exp:-79.96021	out:cf
in(Text):1904	exp:37.274532	out:end
in(Text):cent	exp:-82.137089	out:for
in(Text):of	exp:38.844525	out:he
in(Text):are	exp:-77.859581	out:in
in(Text):will	exp:39.289184	out:io
in(Text):1902	exp:-80.344534	out:lo
in(Text):against	exp:39.280645	out:mat
in(Text):nbeen	exp:-81.929558	out:of
in(Text):Minnesota	exp:33.789577	out:ol
in(Text):1903	exp:-77.321601	out:or
in(Text):Judicial	exp:37.506699	out:ot
in(Text):President	exp:-73.986614	out:tc
in(Text):June	exp:-77.036646	out:te
in(Text):to	exp:-77.047023	out:th
in(Text):for	exp:-77.090248	out:tha
in(Text):hereby	exp:-77.43428	out:that
in(Text):States	exp:-80.720915	out:the
in(Text):United	exp:37.538509	out:this
in(Text):nLouisiana	exp:38.80511	out:tho
in(Text):county	exp:38.81476	out:tie
in(Text):State	exp:38.894955	out:tile
in(Text):Is	exp:40.063962	out:to
in(Text):cash	exp:40.730646	out:tu
in(Text):In	out:-158.09514	out:und