SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also decompose the overall CER into errors due to OCR and errors due to line and layout detection. When the layout model fails to detect text regions, misclassifies them as another category such as ads that isn't OCR'ed, or when line detection misses or crops lines, this increases the CER. The CER due to layout and line detection errors is 0.012.
Legibility Classification Evaluation: We want to avoid classifying legible texts as illegible and vice versa, with the borderline class existing to encompass the grey zone between these two categories. In a random sample of labeled texts - containing 81 legible texts - none of the legible texts are misclassified as illegible. Of 16 illegible texts, only one is misclassified as legible. To further evaluate this, we ran all content regions from a one-day-per-decade sample through OCR. The nonword rate is more than three times higher for illegibly classified content as compared to legibly classified content, and more than twice as high for illegibly classified content as for borderline content.
OCR Evaluation: Table 2 reports the CER from running OCR on ground truth layouts and lines. It is 0.043. The supplementary materials provide a much more detailed analysis of OCR quality. Evaluation on a day per decade sample shows that CER ranges from  $8.9\%$  (1850s) to  $1.8\%$  (1910s), with the bulk of content concentrated in the later period where scans are less challenging. The supplementary materials also document that EffOCR [3] best meets our accuracy and cost requirements, by comparing to a variety of open-source and proprietary OCR engines.
Content Association Evaluation: We achieve an F1 of 97 for associating headlines with articles on our labeled evaluation dataset. We associate all bylines correctly.
Language Modeling: American Stories provides a massive amount of text that can be used to continue pre-training large transformer language models, helping a language model to develop a better understanding of 19th and early 20th century English and greater world knowledge about the past. Moreover, a retrieval augmented language model (e.g. [12]), combined with longer context windows, could provide a valuable tool for retrieving and summarizing vast information, whether it be perspectives on paradigm-shifting historical events or the minutiae of the daily lives of people's ancestors. American Stories also provides extensive content for studying semantic change.
Multimodal Classification: The layouts and texts in American Stories comprise a rich multimodal dataset, providing vast silver quality data that could be used for developing novel methods for multimodal layout analysis or multimodal classification - e.g., with image-caption pairs.
Topic Classification: Topic classification of texts in historical newspapers is a common social science application, with the literature overwhelmingly using keyword searches to measure topics [9]. To evaluate how American Stories can facilitate topic classification, relative to the existing Chronicling America page-level OCR, we use neural and sparse methods to classify whether a randomly selected set of content is about politics, a frequently covered topic. We use a RoBERTa large [20] classifier, applied to articles in American Stories and chunks of Chronicling America (the page OCR is significantly longer than the context window). The training data were randomly sampled at the article level and contain 2418 articles. The development set contains 15 randomly selected full scans (498 articles) and the test set contains 62 randomly selected scans (1473 articles). We also consider keywords, using two different approaches for selecting these: keyword mining on the training set and asking ChatGPT, with careful prompting. All methods are described in the supplementary materials.
American Stories supports article level classification, whereas Chronicling America only supports page level classification. A page is a positive example in the ground truth if any of the articles are on topic, and article or chunk predictions can be aggregated to page level predictions using the same definition. Retrieval at the scan level tends to retrieve a lot of extraneous content, as typically only part of the page contains articles about politics.
On structured article texts, neural methods outperform keyword methods by a wide margin (F1 of 83.6 versus 58.1). All methods perform better at the page level, as there is a much lower chance of false negatives, with the best performance by a wide margin coming from applying neural methods to the American Stories corpus.
Content Dissemination Networks: Reproduced content is of considerable interest to social scientists [7], but detecting it can be challenging due to OCR noise and abridgement. We evaluate this task on a full-day sample of labeled front pages from March 1, 1916, a random day that consists of 113 reproduced articles (the median article is reproduced twice) as well as 1,994 singleton articles.
Topic classification	Reproduced content
Neural F1 (1)	Sparse Mining (2)	GPT (3)	Neural ARI (4)	Sparse Viral Texts (5)
Article Level
American Stories	83.6	56.4	58.1	75.1	32.3
Page Level
American Stories	96.0	82.8	79.6	86.2	74.6
Chronicling America	83.3	83.7	79.6	-	71.4
To detect reproduced content at the article level with American Stories, we deploy the pre-trained neural model from [28]. They contrastively tuned a Sentence BERT model [26, 32] on a large, hand-annotated sample of paired reproduced articles from a later period. At inference time, article representations are clustered using single linkage clustering to detect reproduced content. To make the American Stories result comparable with Chronicling America, we amalgamate the predictions to the page level. A page-pair is counted as positive if the pages have any article in common, making the task easier.
For detecting reproduced content with the Chronicling America full page scans, where we lack the article texts required for the neural method, we deploy the sparse methods from Viral Texts [30]. Viral Texts was designed specifically for detecting reproduced texts in Chronicling America's noisy page-level OCR by looking for overlapping  $n$ -gram spans. We also apply Viral texts to American Stories at the article and page level. Table 3 reports the adjusted rand index (ARI) for these specifications. The Viral Texts method gives slightly better results on American Stories, but the real advantage of American Stories is that the article structure allows the use of a neural method, which dominates the sparse method by nearly 12 percentage points at the page level.
News Story Clustering: The structured nature of American Stories allows for further article-level clustering of content. As a demonstration of this, we show how articles can be grouped into news stories, with different articles that are part of the same unfolding news story clustering together. This prediction is not possible with the unstructured page content in Chronicling America.
Table 3: Comparing American Stories to Library of Congress's Chronicling America.
Year	Biggest story	Year	Biggest story
1885	Death of General Grant	1903	Panama Canal Treaty
1886	Southwest Railroad Strike	1904	Russo-Japanese War
1887	Vatican supports Knights of Labor	1905	Russo-Japanese Peace Process
1888	Rail strikes	1906	Hepburn Railroad Rate Bill
1889	Samoan Crisis	1907	Mining accidents
1890	1893 World's Fair planning	1908	Taft presidential victory
1891	New Orleans Lynchings	1909	Race to the North Pole
1892	Homestead Steel Strike	1910	Rail strikes
1893	World's Fair, Chicago	1911	Canadian Reciprocity Bill
1894	Wilson-Gorman Tariff Act	1912	Republican National Convention (Taft v Roosevelt)
1895	British occupation of Toronto, Nicaragua	1913	Underwood-Simmons Tariff Act
1896	Bimetallism Movement	1914	World War I
1897	Coal Miners' Strike	1915	World War I
1898	Cuban War of Independence	1916	Pancho Villa Expedition
1899	Philippine-American War	1917	World War I
1900	Anglo-Boer War	1918	World War I
1901	U.S. Steel Recognition Strike	1919	Treaty of Versailles
1902	Anthracite Coal Strike	1920	Rail strikes
Table 4: Largest news story in each year, 1885-1920.
To create these clusters, we fine tune a contrastive biencoder on modern news texts, scraped from allsides.com, a website which amalgamates different representations of the same news story from different news outlets. Full details of the data, model and training are given in the supplementary materials. We ran this model over all de-duplicated front page articles from 1885-1920. We read 20 random articles in the largest cluster for each year and named the story which the articles in the cluster refer to. Table 4 shows the largest news story cluster by year.
This application demonstrates one of the many ways that the structured article texts in American Stories can be used to unlock new ways of studying historical and social questions.
In addition to these tasks, image-caption pairs from layout analysis could be used for training and assessing image captioning models, visual question-answering models, cross-modal retrieval models, image retrieval models, and multi-modal understanding (e.g., representation learning) models. American Stories could also be leveraged to substantially lower the costs of creating new benchmark datasets for other common ML tasks, e.g., image and text classification, image retrieval, and named entity recognition.
7 Limitations and Recommended Usage
American Stories contains historical language, that reflects the semantics and cultural biases of the time. This is a distinguishing feature, that is core to many potential applications. We do not filter texts that use antiquated terms or that may be considered offensive, as this would invalidate the use of the dataset for studying semantic change and historical contexts. At the same time, this makes American Stories less suited for tasks that require texts that fully conform to current cultural standards or semantic norms. For these reasons, we recommend against the use of American Stories for training generative models. While the OCR is high quality, American Stories is also not well-suited to tasks requiring fully clean texts. Rather, American Stories can be used for a wide variety of applications, ranging from elucidating social science questions to training a historically-oriented language model to exploring world and family history. It also provides a modular pipeline that can be customized for other document collections and scaled cheaply, offering a blueprint for liberating large-scale historical text corpora.
Funding was provided by the Harvard Data Science Initiative, Harvard Catalyst, the Ken Griffin Harvard Economics Fund, and Microsoft Azure compute credits. We thank Anoushka Ashwin, Domenick Clark Regina, Chloe Combes, Will Cox, Connor Fagal, Brevin Franklin, Prabhav Kamojjhala, Zachary Lee, Roberto Lopez-Irrisarry, Elan Pelegri, Krishna Prasad Srinivasan, and Sarah Strohecker for research assistance.
[1] ALI, A., TOUVRON, H., CARON, M., BOJANOWSKI, P., Douze, M., Joulin, A., LAPTEV, I., NEVEROVA, N., SYNNAEVE, G., VERBEEK, J., ET AL. Xcit: Cross-covariance image transformers. Advances in neural information processing systems 34 (2021).
[2] CAI, Z., AND VASCONCELOS, N. Cascade r-cnn: Delving into high quality object detection. Proceedings of the IEEE conference on computer vision and pattern recognition (2018), 6154-6162.
[3] CARLSON, J., BRYAN, T., AND DELL, M. EfficientOCR for building a diverse digital history. arXiv preprint arXiv:2304.02737 (2023).
[4] DOSOVITSKIY, A., BEYER, L., KOLESNIKOV, A., WEISSENBORN, D., ZHAI, X., UNTERTHINER, T., DEHGHANI, M., MINDERER, M., HEIGOLD, G., GELLY, S., ET AL. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020).
[5] GArBE, W. SymSpell, June 2012.
[6] GEBRU, T., MORGENSTERN, J., VECCHIONE, B., VAUGHAN, J. W., WALLACH, H., AU2, H. D. I., AND CRAWFORD, K. Datasheets for datasets, 2021.
[7] GUARNERI, J. Newsprint Metropolis. University of Chicago Press, 2017.
[8] HADSELL, R., CHOPRA, S., AND LECUN, Y. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) (2006), vol. 2, IEEE, pp. 1735-1742.
[9] HANLON, W. W., AND BEACH, B. Historical newspaper data: A researcher's guide and toolkit, 2022.
[10] HOWARD, A., SANDLER, M., CHU, G., CHEN, L.-C., CHEN, B., TAN, M., WANG, W., ZHU, Y., PANG, R., VASUDEVAN, V., ET AL. Searching for mobilenetv3. Proceedings of the IEEE/CVF international conference on computer vision (2019), 1314-1324.
[11] JOHNSON, J., Douze, M., AND JEGOU, H. Billion-scale similarity search with gpus. IEEE Transactions on Big Data 7, 3 (2019), 535-547.
[12] KHATTAB, O., SANTHANAM, K., LI, X. L., HALL, D., LIANG, P., POTTs, C., AND ZAHARIA, M. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp. arXiv preprint arXiv:2212.14024 (2022).
[13] KHOSLA, P., TETERWAK, P., WANG, C., SARNA, A., TIAN, Y., ISOLA, P., MASCHINOT, A., LIU, C., AND KRISHNAN, D. Supervised contrastive learning. arXiv preprint arXiv:2004.11362 (2020).
[14] KIRILLOV, A., MINTUN, E., RAVI, N., MAO, H., ROLLAND, C., GUSTAFSON, L., XIAO, T., WHITEHEAD, S., BERG, A. C., LO, W.-Y., ET AL. Segment anything. arXiv preprint arXiv:2304.02643 (2023).
[15] LEE, B. C. G., MEARS, J., JAKEWAY, E., FERRITER, M., ADAMS, C., YARASAVAGE, N., THOMAS, D., ZWAARD, K., AND WELD, D. S. The newspaper navigator dataset: extracting headlines and visual content from 16 million historic newspaper pages in chronicling america. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (2020), pp. 3055-3062.
[16] LEvenshtein, V. I., ET AL. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady (1966), vol. 10, Soviet Union, pp. 707-710.
Supplementary Materials
Model Details
American Stories deploys a modular pipeline to digitize historical newspapers. This section provides details for each component of the pipeline.
Layout Detection
To detect articles, headlines, ads, and other content regions in a newspaper scan, we deploy YOLOv8 (Medium) [34], initialized from the officially released YOLOv8m pretrained checkpoint. We train for 100 epochs on 2,202 labeled newspaper scans with 48,874 total layout objects, using default YOLOv8 hyperparameters except: {imgsz: 1280, iou: 0.2, max_DET: 500}. The final model achieved a 0.91 mAP50:95 on article bounding boxes and a 0.84 mAP50:95 on headline bounding boxes. We decreased the confidence threshold to 0.1 to increase article and headline recall.
Legibility Classification
Text image bounding boxes are classified as legible, borderline, or illegible, using MobileNetV3 (Small) [10] initialized from the PyTorch Image Models ("timm") [35] pretrained checkpoint. We train for 50 epochs on 979 labeled article, headline, and image caption examples. 678 of the labeled examples were legible, 192 borderline, and 109 illegible. The model was trained with weighted Cross Entropy Loss: weights [2.0, 1.0, 1.0] for legible, borderline, and illegible classes, respectively. The following specifications were used: {resolution: 256, learning rate: 2e-3}. The learning rate was multiplied by 0.1 every twenty epochs.
Text Line Detection
Line bounding boxes are detected using YOLOv8 (Small) [34] initialized from the official YOLOv8s pretrained checkpoint. We train first for 100 epochs on 4000 synthetically generated articles, with default YOLOv8 hyperparameters. After synthetic training, the model was additionally trained for 50 epochs on 373 hand-annotated article and headline crops, with default YOLOv8 hyperparameters except for the following: {resolution: 640, initial learning rate: 0.02, final learning rate: 0.002}.
Word and Character Localization
Words and characters are detected using YOLOv8 (Small) [34], initialized from the official YOLOv8s pretrained checkpoint. We train first for 100 epochs on 8000 synthetically generated textlines with default YOLOv8 hyperparameters. After synthetic training, the model was additionally trained for 100 epochs on 684 hand-annotated text line images, with default hyperparameters except for the following: {resolution: 640, initial learning rate: 0.02, final learning rate: 0.001}. Each hand-annotated line image was replicated three times with random augmentations along three axes: random rotation between  $-1^{\circ}$  and  $1^{\circ}$ , random image brightness shift from -30 to  $30\%$ , and randomly applied blur at the 0-4px level. On average, text line examples contained 4.3 words and 23.6 characters.
Word Recognition
Building upon the architecture in [3], we train word recognition as a nearest neighbor image retrieval problem. As described in the main text, the training dataset for the model consists of digital renders of words created using 43 fonts, silver quality data from the target dataset created by applying the EffOCR-C (Small) model from [3] to a random sample of days, and a small number of randomly selected hand labeled word crops. We limited the number of crops with model-generated labels to 20 - so each word can have 0-20 silver-quality crops depending upon its frequency of occurrence in our random sample. This limit is binding for common words, e.g., "the".
The recognizer is trained using the Supervised Contrastive ("SupCon") loss function [13], a generalization of the InfoNCE loss [24] that allows for multiple positive and negative pairs for a given anchor. In particular, we work with the "outside" SupCon loss formulation
$$
\mathcal {L} _ {\text {o u t}} ^ {\sup } = \sum_ {i \in I} \mathcal {L} _ {\text {o u t}, i} ^ {\sup } = \sum_ {i \in I} \frac {- 1}{| P (i) |} \sum_ {p \in P (i)} \log \frac {\exp \left(\boldsymbol {z} _ {i} \cdot \boldsymbol {z} _ {p} / \tau\right)}{\sum_ {a \in A (i)} \exp \left(\boldsymbol {z} _ {i} \cdot \boldsymbol {z} _ {a} / \tau\right)}
$$
as implemented in PyTorch Metric Learning [22], where  $\tau$  is a temperature parameter,  $i$  indexes a sample in a "multiviewed" batch (in this case multiple fonts/augmentations of the same word),  $P(i)$  is the set of indices of all positives in the multiviewed batch that are distinct from  $i$ ,  $A(i)$  is the set of all indices excluding  $i$ , and  $z$  is an embedding of a sample in the batch [13].
To create training batches for the recognizer, we use a custom  $m$  per class sampling algorithm without replacement, adapted from the PyTorch Metric Learning repository [22]. The  $m$  word variants for each class (word) are drawn from both target documents and augmented digital fonts. We select  $m = 4$  and the batch size is 1024, meaning 4 styles of each of 256 different words appear in each batch. For training without hard negatives, we define an epoch as letting the model see each word (case-sensitive) exactly  $m = 4$  times. Sampling for each class occurs without replacement until all variants are exhausted.
In order to converge faster with limited compute, we also implement offline-hard negative mining, batching similar negatives and their corresponding positive anchors together - thus making the contrasts between the positive and negative pairs within a batch especially informative. To create hard negative sets, we render each word using a reference font (Noto-Serif Regular) and embed it to create a reference index. We find  $k = 8$  nearest neighbors for each word using this index and the model trained without hard negatives, which yields sets of 8 words that have a similar appearance when rendered with the reference font. We use only the reference font to create these sets because using crops corresponding to all 43 fonts for each word is computationally costly and creates more hard negative sets than we can use in training. We also use each word crop from the target dataset (both silver quality annotations generated with model predictions and gold quality human-annotated predictions) to create hard negative sets. Hence, the total number of hard-negative sets equals the number of words in our dictionary (generated with the reference font) plus the number of word crops from the newspaper data in the training set.
Each hard negative set contains 8 words, with  $m = 4$  views per word, which means we can fit 32 randomly sampled hard negative sets within each batch. An epoch is defined as seeing each hard negative set once. Since the number of synthetic views of an image is much larger than the number of target newspaper crops, whenever newspaper crops are available we force the  $m$  views of a word to contain an equal number of synthetic and target crops.
We use a MobileNetV3 (Small) encoder pre-trained on ImageNet1k sourced from the timm [35] library, more specifically, the model mobilenetv3_small_050. We use 0.1 as the temperature for SupCon loss and AdamW as the optimizer with Pytorch [25] defaults for all parameters other than weight decay (5e-4) and learning rate. We used Cosine Annealing with Warm Restarts as the learning rate scheduler with a maximum learning rate of  $2e - 3$ , a minimum learning rate of 0, time to first restart  $(T_{0})$  as the number of batches in an epoch, and restart factor,  $T_{mult}$  of 2 using the implementation provided in Pytorch.
While fonts and newspaper crops for each word act as an augmentation on the skeleton of the word, we also add more image-level transformations to improve generalization. These include Affine transformation (only slight translation and scaling allowed), Random Color Jitter, Random Autocontrast, Random Gaussian Blurring, Random Grayscale, Random Solarize, Random Sharpness, Random Invert, Random Equalize, Random Posterize and Randomly erasing a small number of pixels of the image. Additionally, we pad the word to make the image square while preserving the aspect ratio of the word render. We do not use common augmentations like Random Cropping or Center Cropping, to avoid destroying too much information.
The model trained without hard negatives was trained for 50 epochs and with hard negatives, it was trained for 40 epochs. For selecting the best checkpoint, we use 1-CER (OCR Character Error Rate) as the validation metric on the validation set from [3]. We chose the model that performed best in terms of CER when detecting only words on the validation set. This means that if a word is outside
of our dictionary, it is forcefully matched to the nearest neighbor in the dictionary. The best model achieved a CER of  $4.9\%$  with word-only recognition.
At inference time, words are recognized by retrieving their nearest neighbor from the offline embedding index created with the reference font, using a Facebook Artificial Intelligence Similarity Search backend [11]. The code to train the model and generate training data, as well as the model checkpoints, are made available on our GitHub repo.
When the nearest neighbor to an embedded word crop in the offline word embedding index is below a cosine similarity threshold of 0.82, we default to character-level recognition. We use the EffOCR-C (Small) model that is developed in [3] for character recognition.
Content Association
This step associates headlines, bylines, and article bounding boxes. We use rule-based methods that exploit the position of article and byline bounding boxes relative to headlines. Specifically, we associated a headline bounding box with an article bounding box if they overlap vertically by more than  $1\%$  of the page width, and the bottom of the headline is no greater than  $10\%$  of the page height above the top of the article, and no greater than  $2\%$  of the page height below the top of the article. If multiple article bounding boxes satisfy these rules for a given headline, then we take the highest. The same rules are used to associate bylines.
Pipeline Evaluation
As discussed in the main paper, we evaluate the data processing pipeline in an end-to-end fashion, as well as evaluating individual sections, particularly OCR. Here we provide additional details on those evaluations.
OCR Evaluation