SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Melissa Dell $^{1,2*}$ , Jacob Carlson $^{1}$ , Tom Bryan $^{1}$ , Emily Silcock $^{1}$ , Abhishek Arora $^{1}$ , Zejiang Shen $^{3}$ , Luca D'Amico-Wong $^{1}$ , Quan Le $^{4}$ , Pablo Querubin $^{2,5}$ , Leander Heldring $^{6}$
<sup>3</sup>Massachusetts Institute of Technology; Cambridge, MA, USA.
4Princeton University; Princeton, NJ, USA.
$^{5}$ New York University; New York, NY, USA.
$^{6}$ Kellogg School of Management, Northwestern University, Evanston, IL, USA.
Existing full text datasets of U.S. public domain newspapers do not recognize the often complex layouts of newspaper scans, and as a result the digitized content scrambles texts from articles, headlines, captions, advertisements, and other layout regions. OCR quality can also be low. This study develops a novel, deep learning pipeline for extracting full article texts from newspaper images and applies it to the nearly 20 million scans in Library of Congress's public domain Chronicling America collection. The pipeline includes layout detection, legibility classification, custom OCR, and association of article texts spanning multiple bounding boxes. To achieve high scalability, it is built with efficient architectures designed for mobile phones. The resulting American Stories dataset provides high quality data that could be used for pre-training a large language model to achieve better understanding of historical English and historical world knowledge. The dataset could also be added to the external database of a retrieval-augmented language model to make historical information - ranging from interpretations of political events to minutiae about the lives of people's ancestors - more widely accessible. Furthermore, structured article texts facilitate using transformer-based methods for popular social science applications like topic classification, detection of reproduced content, and news story clustering. Finally, American Stories provides a massive silver quality dataset for innovating multimodal layout analysis models and other multimodal applications.
Historical local newspapers provide a massive repository of texts about American communities and their inhabitants that can elucidate topics ranging from semantic change to political polarization to the construction of national and cultural identities to the minutiae of the daily lives of people's ancestors. Given the enormous breadth and depth of content, historical newspapers have been widely studied, yet existing open source U.S. newspaper datasets have significant limitations that complicate the extent to which modern deep learning methods can leverage and liberate their content.
Library of Congress's Chronicling America project [19] is the primary public domain historical U.S. newspaper dataset. It consists of around 20 million historical newspaper scans and their corresponding digitized texts. Its content is concentrated before 1925, as this content has entered the public
domain. Chronicling America does not recognize oftentimes complex newspaper layouts, and so digitized texts are provided at the page level, often scrambling headlines, articles, advertisements, captions, and other content regions together. Because a non-trivial share of the underlying scans are illegible, incoherent texts are prevalent, with illegibility varying across space and time. This complicates applying natural language processing (NLP) and statistical methods, and the data are not of sufficient quality to use for training a large language model to achieve a better understanding of historical English and historical world knowledge.
To address these limitations, we develop a pipeline for cheaply extracting high quality digitized article texts and layout regions from newspaper scans. First, layout detection predicts the coordinates and classes of content regions - e.g. articles, headlines, bylines, advertisements, pictures, etc. - using object detection methods [34]. Then, an image classifier removes illegible text bounding boxes. We next digitize the text regions using a novel optical character recognition (OCR) architecture that yields highly scalable, accurate results within our constrained budget. The focus on cost effectiveness makes the pipeline accessible to others with limited budgets who would like to digitize massive historical document collections. Finally, we associate headline, byline, and article bounding boxes. We do not process foreign language newspapers, as off-the-shelf OCR tends to perform poorly on the diverse languages and scripts.
The resulting American Stories (Structured text on reporting in every state) dataset contains 1.14B content regions. The dataset has extensive geographic coverage across all states and has content dating as far back as the 17th century, although the bulk of content comes from the early 20th century. Figure 1 shows the distribution of scans across years and states. The vast majority of American Stories is older than the 72 year rule that the U.S. government uses to release personal information (e.g., from the census) into the public domain.
(a) Scans Across Time
(b) Scans Across Space
Figure 1: Scans in the Chronicling America database across time and space.
We show that the pipeline produces accurate predictions. The resulting texts could be used for historical language model training, or added to an external database of a retrieval augmented language model to facilitate the study of topics ranging from international events to family history. The layouts and corresponding texts could provide a massive silver quality dataset for applications like multimodal layout analysis and classification. The American Stories dataset also yields significantly better performance on social science analyses than the Chronicling America OCR and allows analyses that would be impossible without structured article texts. For example, we cluster article embeddings to detect which stories (e.g., Pancho Villa Expedition, 1916) received the most coverage each year.
The rest of this study is organized as follows. Section 2 discussed related literature and Section 3 describes the American Stories dataset. Section 4 outlines the digitization pipeline, and Section 5 evaluates the quality of the outputs. Section 6 discusses applications and Section 7 considers limitations and recommended usage.
2 Related Literature
Public domain newspaper datasets exist for many countries, but typically pipelines are proprietary, as the norm is to outsource digitization to a private company. Commercial newspaper databases
likewise do not disclose their pipelines, resulting in a dearth of open-source methods. The most closely related work to American Stories is the open-source Newspaper Navigator dataset [15]. The main, and crucial, difference between American Stories and [15] is that [15] does not detect bounding boxes of articles. [15] focuses on 7 classes of visual content: headlines, photographs, illustrations, maps, comics, editorial cartoons, and advertisements. Distinguishing articles enables legibility classification, application of custom OCR, and association of articles across bounding boxes. These tasks are at the core of our contribution, and enable the usefulness of our contribution for downstream applications. In addition, the OCR in [15] is limited to Chronicling America's OCR, which we show leads to a quality deterioration.
While we focus exclusively on texts where the entire newspaper is indisputably in the public domain (typically because it was published more than 95 years ago), it is worth noting that our pipeline could help address some of the copyright issues that have limited the public availability of historical newspapers. Outside of the nation's most widely circulated newspapers, it was rare for local papers to publish with a copyright notice or renew their copyright, required formalities until the latter half of the 20th century. Hence, the majority of local papers well into the 20th century are off-copyright. Yet these papers might sometimes print copyrighted content by third parties - e.g., frequently comics, rarely ads, and occasional runs of syndicated fiction [23]. Individual news articles did not have their copyrights renewed, as copyrighting yesterday's news lacked commercial value. Detecting individual content regions - e.g., so that ads and comics could be cropped out and fictional texts removed with a classifier - is a prerequisite for removing content potentially under copyright. Some copyright experts [23] have suggested this as a way forward for making historical newspapers more accessible.
3 Dataset
Table 1 describes American Stories, totaling 1.14 billion content region bounding boxes. Headlines, images, bylines, and captions are OCR'ed if legible. The dataset contains 3,313 tokens per page on average, making the full dataset 65.6 billion tokens.
(1)
Total
Boxes	(2)
Articles	(3)
Text Bounding Boxes	(4)
Citations	(5)
Bylines	(6)
Images	(7)
Ads	(8)
Tables	(9)
Mastheads
Legible	-	335M	368M	9.7M	14.7M	-	-	-	-	-
Illegible	-	26M	27M	0.9M	2.5M	-	-	-	-	-
Borderline	-	77M	22M	1.3M	1.2M	-	-	-	-	-
Total	1.14B	438M	417M	11.9M	18.4M	9.1M	221M	16.3M	4.9M
Table 1: American Stories dataset statistics.
American Stories provides the classes and coordinates for all content regions. Using the provided metadata, it is straightforward for users to link the coordinates with the original scans, which can be downloaded through the Library of Congress's website. We do not OCR ads because they oftentimes have unusual fonts and complex layouts, including scene text and complex tables with pricing or schedule information, complicating OCR. The table class includes tabular article data, e.g. sporting rosters. We do not transcribe these because accurately detecting and harmonizing the diversity of table layouts is challenging with current technology. Newspaper headers are not OCR'ed because most of their information is contained in the metadata. Mastheads - which contain subscription information - and page numbers often used very small fonts and hence are disproportionately likely to be illegible; page numbers can be inferred from the metadata.
Each text region is classified as legible, illegible, or borderline, with examples of each category shown in Figure 2. Borderline forms the grey zone between clearly legible and clearly illegible texts and is OCR'ed. Users can remove it if they would like to limit to the highest quality texts.
Substantial shares of illegible content can degrade language model training and bias downstream applications. For example, it is common for social scientists to construct data based on the presence of keyword terms [9], assigning a positive outcome if the term is present and a zero outcome otherwise. Illegibility can bias downstream analyses if it is correlated with an underlying unobserved factor. Figure 3 shows that illegibility is correlated with space and time, and hence likely to be cor
Figure 2: Examples of legibility classification, as predicted by our trained model.
related with unobserved factors. Using our data, researchers can remove papers with high illegibility rates if desired and more realistically assess selection into the database.
(a) Illegibility Across Time
(b) Illegibility Across Space
Figure 3: Illegible articles in the Chronicling America database across time and space.
American Stories has a Creative Commons CC-BY license, to encourage widespread use. The data are available on the Hugging Face Hub<sup>1</sup>. The raw files are in a json format, and the Hugging Face repo comes with a setup script that easily allows people to download both raw and parsed data to facilitate language modeling and computational social science applications. The supplementary materials and the Readme on the dataset repository provide a detailed usage guide.
Overview: The American Stories pipeline consists of four steps: layout/line detection, legibility classification, OCR, and article association. The pipeline is available on  $\mathrm{G h i t b u^{2}}$
The pipeline's modularity offers several advantages. Theoretically, localization (of layouts, lines, words, and characters) and recognition (of characters, akin to classification) may rely on different features of the image, suggesting modularity [31]. Practically, there are vast differences in the number of labels required for training each component of the pipeline. Modularity also leads to architectural simplicity. Swapping in different encoders is straightforward, which makes the pipeline more customizable and future-proof. Off-the-shelf models performing a single pipeline task - like Segment Anything [14] or an OCR engine - would be straightforward to swap in.
<sup>1</sup>https://huggingface.co/datasets/dell-research-harvard/AmericanStories
$^{2}$ https://github.com/dell-research-harvard/AmericanStories
The American Stories pipeline uses architectures designed for mobile phones - Yolo v8 [34] and MobileNet v3 [10] - because the accuracy hit relative to much larger models was very modest and deployment costs were over an order of magnitude lower. If budget is not a concern, it would be straightforward to swap in a vision transformer, (e.g., [4, 1, 21]) and a two stage object detection framework (e.g., [2]), variations that [3] examine quantitatively on Chronicling America.
The pipeline was run on Azure F-series CPU nodes. Training used an Nvidia A6000 GPU card. Details are described in the Supplementary Materials.
Layout and Line Detection: Layout region coordinates and classes are detected using Yolo v8 [34], with Figure 4 showing examples. We train the layout detection model on 2,202 labeled newspaper images, consisting of 48,874 layout objects, with an average of 22 layout objects per page. All annotations for the pipeline were created by undergraduate research assistants, with scans selected randomly and using active learning [27]. To develop a general purpose model, we annotated scans of public domain and off-copyright newspapers from throughout the 19th and 20th centuries. Scans from Chronicling America comprise  $13\%$  of the training sample.
Figure 4: Variety of newspaper layouts with our layout detection pipeline outputs overlayed.
For content regions that are OCR'ed, we detect individual lines within each region using Yolo v8, as lines are the input to OCR. The model is trained on 4,000 synthetic and 373 annotated line crops. Yolo v8, like other object detection frameworks, takes square images as inputs. When the aspect ratios of content regions differ significantly from squares, downstream OCR performance tends to be adversely affected. Hence, for content regions with an aspect ratio greater than 2:1 (twice as tall as wide), we split the layout region (with overlap), run line detection over each separate box, and then run non-maximum suppression jointly over the resulting predictions. For the same reason, before sending lines to OCR, we split lines with an aspect ratio below 1:30 (thirty times wider than tall).
Legibility Classification: We classify headline, article, byline, and caption regions as legible, illegible, or borderline (see Figure 2) using an image classifier with a Mobilenet v3 backbone that is trained on 1,094 double-labeled content region crops.
OCR: We aimed to deploy a highly accurate digitization pipeline within a constrained cloud compute budget of \(60,000 USD. As documented in detail in the supplementary materials, existing OCR solutions did not meet these requirements. Commercial solutions were far too costly, and TrOCR Base [18] - an open-source transformer sequence-to-sequence OCR that produced accurate results - was nearly fifty times more costly to deploy on cloud CPUs than our budget. More scalable open-source OCR engines were noisy even when fine-tuned on newspaper annotations.
We met our accuracy and cost objectives with the EfficientOCR framework [3]. EfficientOCR uses deep learning-based object detection methods to localize individual characters and/or words in an image. Character/word recognition is modeled as a character/word-level image retrieval problem, using a vision encoder contrastively trained on character/word crops, largely created through augmenting digital fonts. At inference time, character/word embeddings are decoded to text in parallel by retrieving their nearest neighbor from an offline index of exemplar character or word embeddings, created by rendering character/word images with a digital font. Distances are computed using cosine similarity with a Facebook Artificial Intelligence Similarly Search (FAISS) backend [11].
Switching between character and word recognition is important. There are many terms that would not appear in even a very large dictionary, as narrow newspaper columns imply that hyphenated words at the end of lines are common.<sup>3</sup> The texts also have a long-tailed distribution of proper nouns and antiquated acronyms and words. Hence, at inference time, whenever a word crop is below a tuned cosine similarity threshold of 0.82 from its nearest neighbor in the offline word embedding index, we instead apply character-level EfficientOCR to individual character crops within the word. The supplementary materials provide a detailed description of training and deployment.
Content Association: We associate headlines together (if spanning multiple boxes), associate them with bylines (if present), and with the first article bounding box, using rule-based methods that exploit the position of article and byline bounding boxes relative to headlines (see the Supplementary Materials). Rule-based methods perform less well for articles spanning multiple columns or pages, and language understanding is required. However, we find on a labeled sample that only around  $3.8\%$  of articles span multiple article bounding boxes, and only  $0.2\%$  of articles span multiple pages, meaning there is little scope for gains from neural methods. (Articles spanning multiple columns and pages become more common after the Chronicling America period, as the price of paper falls and font size increases.) Since these cases are rare and our compute budget is limited, we do not associate multiple article bounding boxes together for this release, but may do so in the future, using the RoBERTa cross-encoder method developed in [29].
5 Pipeline Evaluation
Measurement: We use four carefully constructed datasets to evaluate the pipeline:
- Full page scans: Two student annotators labeled layout regions and hand-entered the texts for 10 full page scans, resolving all discrepancies by hand. The set consists of 597 content regions and 196,655 characters. It allows us to evaluate the end-to-end pipeline, which requires transcribed full-page scans since layout analysis is applied at the page level. We further use this sample for evaluating content association. It contains 214 headline-article bounding box pairs.
- Transcribed day per decade sample: To evaluate line detection and OCR on highly diverse content, we hand-transcribe a randomly selected sample of 50 lines for each decade between 1850-1920. During this period, printing and archival technology changes significantly. Examples of these textlines, with their accompanied EffOCR transcriptions, are shown in the supplemental materials as Table 1.
- Transcriptions of randomly selected lines from Carlson et al. [3]: This sample is used to report comparisons to other object detection frameworks, backbones, and OCR engines. These comparisons are taken from [3], which develops EfficientOCR. This sample includes 64 textlines drawn randomly from random scans in the Chronicling America collection.
- Legibility sample: We evaluate legibility on a randomly selected (from legibility training data), double labeled sample of 100 bounding boxes, 50 headlines and 50 articles. Transcriptions are not included, as we cannot create character labels for illegible content.
We measure pipeline accuracy with the character error rate (CER), defined as the Levenshtein distance [16] between the end-to-end digitized content and the ground truth, divided by the length of the ground truth. OCR is similarly evaluated by the character error rate on ground truth layout and line annotations (and hence does not include transcriptions errors induced by errors in the layout predictions). We also examine non-word rate, as it does not require costly-to-create labeled data so can be evaluated on a much larger sample, though it is more difficult to interpret. To measure
<sup>3</sup>Our word dictionary consists of words rendered three times each: with all lowercase characters, all uppercase characters (common in headlines), and capitalized. We select words by first taking the top 25,000 words from a modern dictionary that ranks word frequency [5]. We remove 3,999 words that never appear in a sample of all Chronicling America scans from one-day-per decade (1850s-1920s), OCR'ed with character EfficientOCR. Inspection revealed that these were overwhelmingly words related to modern concepts like computing and modern medicine. We then added the 500 words that most frequently appear in this sample but were not frequent modern words - mostly consisting of terms from antiquated domains like traditional medicine - and also added numbers, contractions, state and month abbreviations, and punctuation, for a total of 22,230 total terms.
the quality of layout and line detection, we use mean average precision (mAP@50:95), as well as decomposing what share of the overall character error rate is due to layout and line detection errors. For full article association, we focus on confusability between legible and illegible scans. Finally, we evaluate content association using F1.
Overall Pipeline Evaluation: The end-to-end CER is 0.051 (Table 2), showing the pipeline's high overall accuracy. Some of the errors - e.g. confusing commas and periods are particularly prevalent - are straightforward to fix in post-processing. When we apply a lightweight spellchecker [5], the CER falls to 0.044. Spellchecking produces a slight increase in CER in headlines, likely due to a higher concentration of proper nouns.
(1) Overall	(2) Headlines	(3) Articles	(4) Ads
Mean Average Precision	63.48	88.64	91.31	78.4
Overall CER (Spellchecked)	.044	.092	.038	-
Overall CER	.051	.089	.049	-
CER from OCR	.043	.071	.039	-
CER from layout detection	.012	.018	.010	-
Article Association F1	97.0	-	-	-
Table 2: Pipeline evaluation on ten labeled scans. CER is the character error rate, decomposed into errors from OCR and from layout detection. Article association F1 evaluates the association of headlines with each other and the first article bounding box. Spellchecking is applied after EffOCR, using [5].
Figure 5 plots the non-word rate at the scan level on a day-per-decade sample, where non-word rate measures the share of terms not in a lengthy dictionary with 82,765 terms [5]. Even with a perfect OCR, the non-word rate could be appreciable, due to hyphenated words at the end of rows, acronyms, proper nouns, and antiquated terms. The American Stories distribution is concentrated well to the left of Chronicling America's distribution, underscoring the quality of our texts. Differences in the non-word rate between Chronicling America and American Stories are likely to reflect both differences in OCR quality and differences in filtering content based on legibility and content region type, as Chronicling America digitizes all the texts it can localize on the page.
Figure 5: Non-Word Rate Distributions of American Stories and Chronicling America.
Layout and Line Detection Evaluation: mAP, reported in Table 2, is high, particularly for the classes of central interest such as headlines (88.64) and articles (91.31). Confusing ads for articles - as they often look similar in this period - is relatively common. The ads that look like articles tend to OCR well and have natural language texts, so these errors are also tolerable. The overwhelming majority of content regions in the ten labeled scans are articles, headlines, and advertisements, as photographs - a rarity in this period - do not appear. The mAP for line detection is 86.20.