SlowGuess/NAACL_2022_OCR huggingface dataset:
Jakub Pokrywka
Adam Mickiewicz University
Filip Graliński
Krzysztof Jassem
Karol Kaczmarek
Adam Mickiewicz University Applica.ai
Krzysztof Jurkiewicz
Piotr Wierzchon
The aim of the paper is to apply, for historical texts, the methodology used commonly to solve various NLP tasks defined for contemporary data, i.e. pre-train and fine-tune large Transformer models. This paper introduces an ML challenge, named Challenging America (ChallAm), based on OCR-ed excerpts from historical newspapers collected from the Chronicling America portal. ChallAm provides a dataset of clippings, labeled with metadata on their origin, and paired with their textual contents retrieved by an OCR tool. Three, publicly available, ML tasks are defined in the challenge: to determine the article date, to detect the location of the issue, and to deduce a word in a text gap (cloze test). Strong baselines are provided for all three ChallAm tasks. In particular, we pretrained a RoBERTa model from scratch from the historical texts. We also discuss the issues of discrimination and hate-speech present in the historical American texts.
The dominant approach in the design of current NLP solutions is (pre-)training a large neural language model, usually applying a Transformer architecture, such as GPT-2, RoBERTa or T5, and fine-tuning the model for specific tasks (Devlin et al., 2019; Raffel et al., 2019). The solutions are evaluated on benchmarks such as GLUE (Wang et al., 2019b) or SuperGLUE (Wang et al., 2019a), which allow comparing the performance of various methods designed for the same purpose. An important feature of a good NLP benchmark is the clear separation between train and test sets. This requirement prevents data contamination, when the model (pre-)trained on huge data might have "seen" the test set in some form.
The expansion of digital information is proceeding in two directions on the temporal axis. In the forward direction, new data are made publicly available on the Internet every second. What is less
obvious is that, in the backward direction, older and older historical documents are digitized and disseminated publicly.
To the best of our knowledge, our paper introduces the first benchmark which serves to use and evaluate the "pre-train and fine-tune scenario" applied to a massive collection of historical texts.
The very idea of building language models on historical data is not new. The Google Ngram Viewer (Michel et al., 2011) is based on large amounts of texts from digitized books. The corpus as a whole is not open for the NLP community – only raw n-gram statistics are available. The temporal information is crude (at best, the year of publication is given) and the corpus is heterogeneous (in fact, it is a dump of digitized books of any origin).
In our research, we use one of the richest sources of homogeneous historical documents, Chroni-cling America, a collection of digitized newspapers that cover the publication period of over 300 years (with significant coverage of 150 years), and design an NLP benchmark that may open new opportunities for the modeling of the historical language.
Recently, time-aware language models such as Temporal T5 (Dhingra et al., 2021) and TempoBERT (Rosin et al., 2021) have been proposed. They focus on modern texts dated yearly, whereas we extend language modeling towards both longer time scales and more fine-grained (daily) resolution, using massive amounts of historical texts.
The contribution of this paper is as follows:
- We extracted a large corpus of English historical texts that may serve to pre-train historical language models (Section 5).
These are the main features of the corpus:
- the corpus size is 74 GB (201 GB of total raw text), which is comparable with
Findings of the Association for Computational Linguistics: NAACL 2022, pages 737 - 749
contemporary text data for training massive language models, such as GPT-2, RoBERTa or T5;
- the corpus is free of spam and noisy data (although the quality of OCR processing varies);
- texts are dated with a daily resolution, hence a new dimension of time (on a fine-grained level) can be introduced into language modeling;
- the whole corpus is made publicly available;
- Based on selected excerpts from Chronicling America, we define a suite of challenges (named Challanging America, or ChallAm in short) with three ML tasks combining layout recognition, information extraction and semantic inference (Section 7). We hope that ChallAm will give rise to a historical equivalent of the GLUE (Wang et al., 2019b) or SuperGLUE (Wang et al., 2019a) benchmarks.
- In particular, we provide a tool for the intrinsic evaluation of language models based on a word-gap task, which calculates the model perplexity in a comparative scenario (the tool may be used in competitive shared tasks) (Section 7.3).
- We propose a "future-proof" methodology for the creation of NLP challenges: a challenge is automatically updated whenever the underlying corpus is enriched (Section 4).
- We introduce a method for data preparation that prevents data contamination (Section 4).
- We train base Transformer (RoBERTa) models for historical texts (Section 5). The models are trained on texts spanning 100 years, dated with a daily resolution.
- We provide strong baselines for three ChronAm challenges (Section 8).
- We take under consideration the issue of discrimination and hate speech in the historical American texts. To this end we have applied up-to date methods to tag the abusive content from the data (Section 9).
2 Related Machine Learning datasets and challenges
This section concerns ML challenges which deliver labeled OCR documents as training data, a definition of the processing task, and an evaluation environment to estimate the performance of uploaded solutions. More often than not, such challenges concern either layout recognition (localization of layout elements) or Key Information Extraction (finding, in a document, precisely specified business-actionable pieces of information). Layout recognition in Japanese historical texts is described in (Shen et al., 2020). The authors use deep learning-based approaches to detect seven types of layout element categories: Page Frame, Text Region, Text Row, Title Region, etc. Some Key Information Extraction tasks are presented in (Stanisławek et al., 2021). The two datasets described there contain, respectively, NDA documents and financial reports from charity organizations. The tasks for the datasets consist in detecting data points, such as effective dates, interested parties, charity address, income, spending. The authors provide several baseline solutions for the two tasks, which apply up-to-date methods, pointing out that there is still room for improvement in the KIE research area. A challenge that comprises both layout recognition and KIE is presented in (Huang et al., 2019) – the challenge is opened for the recognition of OCR-scanned receipts. In this competition (named ICDAR2019) three tasks are set up: Scanned Receipt Text Localization, Scanned Receipt OCR, and Key Information Extraction from Scanned Receipts.
A common feature of the above-mentioned challenges is the goal of retrieving information that is explicit in the data (a text fragment or layout coordinates). Our tasks in ChallAm go a step further: the goal is to infer the information from the OCR image rather than just retrieve it.
Similar challenges for two out of the three tasks introduced in this paper have been proposed before for the Polish language:
- a challenge for temporal identification (Gralinski and Wierzchoń, 2018); the challenge was based on a set of texts coming from Polish digital libraries, dated between the years 1814 and 2013;
- a challenge for “filling the gap” (RetroGap) (Graliński, 2017) with the same training
set as above.
The training sets for those challenges were purely textual. Here, we introduce the challenges with the addition of original images (clippings), though we do not use graphical features in baselines yet.
3 Chronicling America
In 2005 a partnership between the National Endowment for the Humanities and the Library of Congress launched the National Digital Newspaper Program, to develop a database of digitized documents with easy access. The result of this 15-year effort is Chronicling America – a website<sup>1</sup> which provides access to selected digitized newspapers, published from 1690 to the present. The collection includes approximately 140 000 bibliographic title entries and 600 000 library holdings records, converted to the MARCXML format. The portal supports an API which allows accessing of the data in various ways, such as the JSON format, BulkData (bulk access to data) or Linked Data,<sup>2</sup> or searching of the database with the OpenSearch protocol.<sup>3</sup>. The accessibility of data in various forms makes Chronicling America a valuable source for the creation of datasets and benchmarks.
The portal serves as a resource for various research activities. Cultural historians may track performances and events of their interest in a resource which is easily and openly accessible, as opposed to commercial databases or "relatively small collections of cultural heritage organizations whose online resources are isolated and difficult to search" (Clark, 2014). The database enables searching for the first historical usages of word terms. For instance, thanks to the Chronicling America portal, it was discovered in (Cibaroglu, 2019) that the term "fake news" was first used in 1889 in the Polish newspaper Ameryka.
The resource is helpful in research aiming to improve the output of the OCR process. The authors of (Nguyen et al., 2019) study OCR errors occurring in several digital databases – including Chronicling America – and compare them with human-generated misspellings. The research results in several suggestions for the design of OCR post-processing methods. The implementation of an unsupervised approach in the correction of OCR
documents is described in (Dong and Smith, 2018). Two million issues from the Chronicling America collection of historic U.S. newspapers are used in a sequence-to-sequence model with attention.
Chronicling America is a type of digitized resource that may be of wide use for both humanities and computational research. We prepared datasets and challenges based on the data from the Chronicling America resource. We hope that our initiative will bring about research that will facilitate the development of ML-based processing tools, and consequently increase access to digitized resources for the humanities.
An example of an ML tool based on Chronicling America is described in (Lee et al., 2020). The task was to predict bounding boxes around various types of visual content: photographs, illustrations, comics, editorial cartoons, maps, headlines and advertisements. The training set was crowd-sourced and included over  $48\mathrm{K}$  bounding boxes for seven classes. Using a pre-trained Faster-RCNN detection object, the researchers achieved an average accuracy of  $63.4\%$ . Both the training set and the model weights file are publicly available. Still, it is difficult to estimate the value of the results achieved without any comparison with other models trained on the same data.
In our proposal we go a step further. We provide and make freely available training data from Chronicling America for three ML tasks. For each task we develop and share baseline solutions. Alternative solutions can be submitted to the Gonito $^{4}$  evaluation platform (Gralinski et al., 2016, 2019) to be evaluated automatically and compared against our baselines.
4 Data processing
The PDF files were downloaded from Chronicling America and processed using a pipeline primarily developed for extracting texts from Polish digital libraries (Gralinski, 2013, 2019). Firstly, the metadata (including URL addresses for PDF files) were extracted by a custom web crawler and then normalized; for instance, titles were normalized using regular expressions (e.g. The Bismarck tribune. [volume], May 31, 1921 was normalized to THE BISMARCK TRIBUNE). Secondly, the PDF files were downloaded and the English texts were processed into DjVu files (as this is the target format
<sup>1</sup>https://chroniclingamerica.loc.gov
$^{2}$ https://www.w3.org/standards/semanticweb/data
<sup>3</sup>https://opensearch.org/
4https://gonito.net
Table 1: Statistics for the raw data obtained from the Chronicling America website
Documents with metadata obtained	1 877 363
...in English	1 705 008
...downloaded	1 683 836
...processed into DjVu files	1 665 093
for the pipeline) using the pdf2djvu tool<sup>5</sup>. The original OCR text layer was retained (the files were not re-OCR red, even though, in some cases, the quality of OCR was low).
Table 1 shows a summary of the data obtained at each processing step. Two factors were responsible for the fact that not  $100\%$  of files were retained at each phase: (1) issues in the processing procedures (e.g. download failures due to random network problems or errors in the PDF-to-DjVu procedure that might be handled later); (2) some files are simply yet to be finally processed in the ongoing procedure.
The procedure is executed in a continuous manner to allow the future processing of new files that are yet to be digitized and made public by the Chronicling America initiative. This solution requires a future-proof procedure for splitting and preparing data for machine-learning challenges. For instance, the assignment of documents to the training, development and test sets should not change when the raw data set is expanded. Such a procedure is described in Section 6.
5 Data for unsupervised training
The state of the art in most NLP tasks is obtained by training a neural-network language model on a large collection of texts in an unsupervised manner and fine-tuning the model on a given downstream task. At present, the most popular architectures for language models are Transformer (Devlin et al., 2019) models (earlier, e.g. Word2vec (Mikolov et al., 2013) or LSTM models (Peters et al., 2017)). The data on which such models are trained are almost always modern Internet texts. The high volume of texts available at Chronicling America, on the other hand, makes it possible to train large Transformer models for historical texts.
Using a pre-trained language model on a downstream task bears the risk of data contamination - the model might have been trained on the task
test set and this might give it an unfair edge (see (Brown et al., 2020) for a study of data contamination in the case of the GPT-3 model when used for popular English NLP test sets). This issue should be taken into account from the very beginning. In our case, we release<sup>6</sup> a dump of all Chronicling America texts (for pre-training language models), but limited only to the  $50\%$  of texts that would be assigned to the training set (according to the MD5 hash). This dump contains all the texts, not just the excerpts described in Section 6.2. As the size of the dump is 74.0G characters, it is on par with the text material used to train, for instance, the GPT-2 model.
We also release a RoBERTa Base ChallAm model trained on the text corpus. The model was trained from scratch, i.e. it was not based on the weights of the original RoBERTa model (Liu et al., 2019). The BPE dictionary was also induced anew.
Two versions of the RoBERTa ChallAm model were prepared: one<sup>7</sup> was trained with temporal metadata encoded as a prefix of the form year: YYYY, month: MM, day: DD, weekday: WD, another,<sup>8</sup> for comparison, without such a prefix. The ChallAm models have the same number of parameters as the original RoBERTa Base (125M). Each model was trained on two Tesla V100 32GB GPUs for 9 days.
6 Procedure for preparing challenges
We created a pipeline that can generate various machine learning challenges. The pipeline input should consist of DjVu image files, text (OCR image), and metadata. Our main goals are to keep a clear distinction between dataset splits and to assure the reproducibility of the pipeline. This allows potential improvement to current challenges and the generation of new challenges without dataset leaks in the future. We achieved this by employing stable pseudo-randomness by calculating an MD5 hash on a given ID and taking the modulo remainder from integers from certain preset intervals. These pseudo-random assignments are not dependent on any library, platform, or programming language (using a fixed seed for the pseudo-random
$^{6}$ https://gonito.net/get/data/challenging-america-full-train-dump-2021-10-26.tsv.xz
<sup>7</sup>http://gonito.net/get/data/roberta-ch-allam-base-with-date-1325000.zip
<sup>8</sup>http://gonito.net/get/data/roberta-ch-allam-base-without-date-1325000.zip
<sup>5</sup>http://jwilk.net/software/pdf2djvu
generator might not give the same guarantees as using MD5 hashes), so they are easy to reproduce.
This procedure is crucial to make sure that challenges are future-proof, i.e.:
- when the challenges are re-generated on the same Chronicling America files, exactly the same results are obtained (including text and image excerpts; see Section 6.2);
- when the challenges are re-generated on a larger set of files (e.g. when new files are digitized for the Chronicling America project), the assignments of existing items to the train/dev/test sets will not change.
6.1 Dataset structure
All three of our machine learning challenges consist of training (train), development (dev), and test sets. Each document in each set consists of excerpts from a newspaper edition. One newspaper edition provides a maximum of one excerpt. Excerpts in the datasets are available as both a cropped PNG file from the newspaper scan (a "clipping") and its OCR text. This makes it possible to employ image features in machine learning models (e.g. font features, paper quality). A solution might even disregard the existing OCR text layer and re-OCR the clipping or just employ an end-to-end model. (The OCR layer is given as it is, with no manual correction done – this is to simulate realistic conditions in which a downstream task is to be performed without a perfect text layer.)
Sometimes additional metadata are given. For the train and dev datasets, we provide the expected data. For the test dataset, the expected data are not released. These data are used by the Gonito evaluation platform during submission evaluation. All newspaper and edition IDs are encoded to prevent participants from checking the newspaper edition in the Chronicling America database. The train and dev data may consist of all documents which meet our criteria for text excerpts, so the data may be unbalanced with respect to publishing years and locations. We tried to balance the test sets as regards the years of publication (the year-prediction and word-gap challenges) or locations (the geo-prediction challenge), though it is not always possible due to large imbalances in the original material.
6.2 Selecting text excerpts
The details of the procedure for selection of text excerpts is given in Appendix A. A sample excerpt is
shown in Figure 1a. Note that excerpts are selected using a stable pseudo-random procedure based on the newspaper edition ID (similarly to the way the train/dev/test split is done, see Section 6.3).
6.3 Train/dev/test split
Each newspaper has its newspaper ID (i.e. normalized title, as described in Section 4), and each newspaper edition has its newspaper edition ID. We separate newspapers within datasets, so for instance, if one newspaper edition is assigned to the dev set, all editions of that newspaper are assigned to the dev set. All challenges share common train and dev datasets and no challenges share the same test set. This prevents one from checking expected data from other challenges. The set splits are as follows:  $50\%$  for train,  $10\%$  for dev,  $5\%$  for each challenge test set. This makes it possible to generate eight challenges with different test sets. In other words, there is room for another five challenges in the future (again this is consistent with the “future-proof” principle of the whole endeavor).
7 Challenging America tasks
In this section, we describe the three tasks defined in the challenge. They are released on the Gonito evaluation platform, which enables the calculation of metrics both offline and online, as well as the submission of solutions. An example of text from an excerpt given in those tasks is shown in Figure 1b.
7.1 RetroTemp
This<sup>9</sup> is a temporal classification task. Given a normalized newspaper title and a text excerpt, the task is to predict the publishing date. The date should be given in fractional year format (e.g. 1 June 1918 is represented as the number 1918.4137, and 31 December 1870 as 1870.9973).
Hence, solutions to the challenge should predict the publication date with the greatest precision possible (i.e. day if possible). The fractional format will make it easy to accommodate even more precise timestamps, for example, if modern Internet texts (e.g. tweets) are to be added to the dataset.
Due to the regression nature of the problem, the evaluation metric is RMSE (root mean square error).
$^{9}$ https://gonito.net/challenge/challenging-america-year-prediction
(a) An excerpt.
(b) Fragment of a text from an excerpt.
Figure 1: An example of an excerpt
Perhaps one of the most interesting political developments in the political history of California is that which has been disclosed as a result of the quarrel of Leland Stanford and Collis P. Hunt- ington, of the Southern and Central Pa- cific Railways, and which has been sup- pressed as to details, after the scandal has embraced a whole continent. It is probable that much matter for good will ultimately result from this and other indecent developments. Prior to the ar- rival of Mr. Huntington on this Coast the people of California were in danger of being deluged in a stream of adula- tion directed towards Senator Stanford. Although Stanford notoriously purchased his seat in the United States Senate, and although bis purchase of that seat, considering his obligations to Senator Sargent, was a matter of never to be forgottoa treachery, the toad- eaters of the might}’ Senator are intent upon having censers swung in his ...
The motivation behind the RetroTemp challenge is to design tools that may help supplement the missing metadata for historical texts (the older the document, the more often it is not labeled with a time stamp). Even if all documents in a collection are time-stamped, such tools may be useful for finding errors and anomalies in metadata.
7.2 RetroGeo
The task<sup>10</sup> is to predict the place where the newspaper was published, given a normalized newspaper title, text excerpt, and publishing date in fractional year format. The expected format is the latitude and longitude. In the evaluation the distance on the sphere between output and expected data is calculated using the haversine formula, and the mean value of errors is reported.
The motivation for the task (besides the supplementation of missing or wrong data) is to allow research on news propagation. Even if a news article is labeled with the localization of its issue, an automatic tool may infer that it was originally published somewhere else.
7.3 RetroGap
This<sup>11</sup> is a task for language modeling. The middle word of an excerpt is removed in the input document (in both text and image), and the task is to predict the removed word, given the normalized newspaper title, the text excerpt, and the publishing date in fractional year format (in other words, it is a cloze task). The output should contain a probability distribution for the removed word (not just a word or a single probability). The metric is perplexity; PerplexityHashed, to be precise, as implemented in the GEval evaluation tool (Graliński et al., 2019), the modification is analogous to LogLossHashed in (Graliński, 2017), its goal is to ensure proper evaluation in the competitive (shared-task) setup (i.e. avoid self-reported probabilities and ensure objective comparison of all reported solutions, including out-of-vocabulary words).
7.4 Statistics