SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Processing 20 million scans required a cost-effective OCR solution, and downstream tasks require highly accurate OCR. We compared custom, open-source, and commercial OCR solutions by accuracy, speed, and cost to determine our final architecture. Character Error Rate measurements were made on two separate validation datasets:
- CER [3] Error rate on a dataset of 64 randomly selected Chronicling America textlines, sampled from the entire collection. Textlines were randomly sampled from random scans, then cropped and transcribed. This dataset and its construction is described in detail in [3].
- CER Day-Per-Decade Error rate on a sample of 225 total textlines, sampled from all scans in the Chronicling America collection published on March 1st of years ending in "6," from 1856-1926. Unlike the above sample from [3], this sample is balanced across the time periods the predominate the Chronicling America collection. 25 textlines were sampled randomly from random pages published on each of the days. A selection of textlines from this collection, along with their EffOCR transcriptions, are shown in Figure S-1. This dataset is designed to be much more challenging than the first, weighting older, harder to read scans more heavily despite their relative scarcity in the Chronicling America collection.
Comparisons are listed in Table S-1. Training procedures for EffOCR-Word are described above. See [3] for training procedures, initialization checkpoints, and additional details on training and evaluating comparison models.
Of the options we examined, EffOCR-Word (Small) was the clear best option, providing a Character Error Rate under  $5\%$  on the hardest evaluation set while offering the cheapest rate per line on an Microsoft Azure Fs4v2 instance.
<sup>4</sup>https://github.com/dell-research-harvard/AmericanStories.
Model/Engine	Seq2Seq?	Transformer?	Pretraining	Parameters	CER [3]	CER Day-Per-Decade	Lines Per Second	Cost Per 10K Lines
EffOCR-C (Base)	×	×	from scratch	112.5 M	0.023	0.062	0.27	$1.77
EffOCR-C (Small)	×	×	from scratch	9.3 M	0.028	0.080	7.28	$0.06
EffOCR-T (Base)	×	✓	from scratch	101.8 M	0.022	0.059	0.17	$2.80
EffOCR-Word (Small)	×	×	from scratch	10.6 M	0.015	0.043	11.60	$0.04
Google Cloud Vision OCR	?	?	off-the-shelf	?	0.005	0.019	?	$15.00
Tesseract OCR (Best)	✓	×	off-the-shelf	1.4 M	0.106	0.170	2.43	$0.19
EasyOCR CRNN	✓	×	off-the-shelf	3.8 M	0.170	0.274	10.75	$0.04
fine-tuned	0.036	0.157
from scratch	0.131	-
PaddleOCR SVTR	×	×	off-the-shelf	11 M	0.304	7.36	$0.06
fine-tuned	0.103
from scratch	0.104
TrOCR (Base)	✓	✓	off-the-shelf	334 M	0.015	0.038	0.23	$2.02
fine-tuned	0.013	0.027
from scratch	0.809	-
TrOCR (Small)	✓	✓	off-the-shelf	62 M	0.039	0.121	0.53	$0.90
fine-tuned	0.075	0.091
from scratch	0.773	-
Table S-1: Chronicling America Results and Comparisons. This table reports the performance of different OCR architectures, off-the-shelf (without fine-tuning on target data), fine-tuned on the Chronicling America training set from initialization on the best public, pre-trained OCR checkpoint, and pre-trained from scratch on a consistent, standardized set of synthetic text lines and then fine-tuned on the Chronicling America training set. “?” indicates that the field is unknown due to the proprietary nature of the architecture. Inference speeds are based on an extrapolation from inference speeds measured for EffOCR-Word (Small) to digitize the entire Chronicling America collection using cloud compute.
WASHINGTON, April 1—Ambas-	WASHINGTON, April 1 Ambas-
FORT WORTH JITNEYS OUIT	FORT WORTH JITNEYS OUIT
General Plan 5-4-31	General Plan 5-4-31
State of Tennessee	State of Tennessee
A non-Federal project to furnish free home assistance	A non-Federal project to furnish free home assistance
SEED DISTRIBUTION	SEED DISTRIBUTION
Iron. Steel and Tin Workers	Iron, Steel and Tin Workers
ADVERSE REPORTS ON DEMENT'S NOMINATION.	ADVERSE REPORTS ON DEMENTS NOMINATION
IMPROVEMENT IS SHOWN	IMPROVEMENT IS SHOWN
Figure S-1: Examples of textlines in the Day-Per-Decade evaluation set. Image crops are shown on the left, with their corresponding EffOCR transcriptions (using the final model set used in the American Stories processing pipeline) on the right.
Legibility classification was tested on a set of 100 image crops (50 articles and 50 headlines) sampled randomly from the 1,094-image legibility training set. All legibility images were double-entered. Since the goal was to be cautious in classifying images as illegible, where annotators disagreed the more legible of the two labels was used.
Annotators were instructed to use the following definitions for legibility labeling:
- Legible: Greater than  $95\%$  of words in an image readable without context from adjacent words.
- Borderline: Between 50 and  $95\%$  of words in an image readable without context from adjacent words.
- Illegible: Less than  $50\%$  of words in an image readable without context from adjacent words.
Inter-annotator agreement was  $91\%$  between the two annotators. A sample of annotator discrepancies is presented in Figure S-2
(a) Legible/Borderline
(b) Borderline/Illegible
(c) Legible/Borderline
(d) Borderline/Illegible
Figure S-2: Examples of Inter-Annotator disagreement in legibility labeling. Images are labeled "Class 1"/"Class 2" to show the two labeled classes. In cases of disagreement, the more legible of the two labels was used for training and evaluation.
The paper presents multiple applications that can be facilitated by the American Stories dataset. This section provides details for each application given.
To evaluate topic classification, we focused on the topic of politics. As we evaluate at both the scan and article level, for development and test sets we sampled full scans (all articles on the same scan). We took a random sample of up to three front page scans from each election year in our sample. These scans were double-labelled by student research assistants and incongruences were discussed and resolved. We place  $20\%$  of these (15 scans, 498 articles) into a development set and the remaining 62 scans (1473 articles) into the test set. Training data was sampled at the article level, rather than the scan level, from the same population of front page articles in election years. Training
data was single-labelled by the same research assistants. A sample were double-labelled to check for consistency and they agreed on the labelling in  $93\%$  of cases.
To evaluate neural methods, we finetune RoBERTa large [20] on the training set for ten epochs, with a batch size of 16, and a learning rate of 2e-6.
For evaluation of sparse methods, we use two different methods to select keywords. First, we use the test and evaluation sets to mine keywords. We use TF-IDF to pull words and bigrams that are most commonly found in train set articles about politics, but not found in off-topic articles. We take the top 40 words and bigrams and then sequentially pick those that maximise F1 on the evaluation set, until there is no remaining keyword that increases F1. Using this technique, the mined keywords were: vote, election, republican, committee, united, party, president, congress.
Second, we prompted Chat-GPT to produce keywords. We used the prompt: "You have a large number of 19th century US newspaper articles. You wish to classify these on whether they are about politics or not. The only way that you can do this is by checking whether they match any of a list of keywords or keyphrases. You can search for these keywords or phrases in each article, and if it matches any of them it will be classified as about politics, but if it does not match any, it will not. Please provide a list of keywords and phrases that will correctly classify as many of the articles that are about politics as possible, with a minimal number of off-topic articles classified as on topic" and received the following keywords: President, Congress, Election, Senator, Representative, Governor, Democratic Party, Whig Party, Republican Party, Suffrage, Legislation, Lawmakers, Government, Policy, Bill, Campaign, Debate, Vote, Political Convention, Public Office, Political Reform, Impeachment.
Using these lists of keywords, we consider any article to be predicted as on topic if it contains any of these keywords. We do not take case into account.
The structured data in American Stories allows us to classify at the article level, a significant advantage. However, for comparison with Chronicling America, we also evaluate the same methods at the scan level. A scan is counted as on topic if any article on that page is on topic. For neural methods on Chronicling America, we chunk the text into passages of 256 tokens, as the page OCR is significantly longer than the context window.
Content Dissemination Networks
To detect reproduced content, we also compare neural and sparse methods. In this case, the neural methods are only possible with American Stories, whereas sparse methods are possible with both American Stories and Chronicling America.
We evaluate these methods on all front pages from March 1, 1916, a randomly selected day. A single day is chosen because reproduced content tends to be published around the same time, so a single day will have a far higher number of reproduced articles than a random selection of front pages across time. On this day, American Stories contains 114 scans, with 2,354 articles. 1,994 of these were not reproduced, while 360 were reproduced. These 360 comprise 113 distinct articles, with the median reproduced article being reprinted 2 times.
For the neural method, we use the pre-trained neural model from [28]. This is a contrastively-trained bi-encoder finetuned from the MPNet Sentence BERT model [26, 32] on a large, hand-annotated sample of pairs of reproduced historical newspaper articles. [28] find this biencoder is marginally improved by running a cross-encoder over the outputs, but we do not reproduce these results as the cross-encoder is computationally costly for a small gain in performance. They also find that this fine-tuned biencoder model outperforms more generic semantic textual similarity models (eg. [26] by up to 20 percentage points. Thus the model is chosen to maximise accuracy, within a reasonable compute budget. At inference time, article representations are clustered using single linkage clustering to detect reproduced content, and spurious links are pruned using community detection. We use the same distance threshold as in [28].
To enable a comparison to Chronicling America, where the content is only available at a page level, we amalgamate these results by page. A page-pair is counted as positive if they have any article in common. Nonetheless, the page-level evaluation of the neural method requires the data to be split into articles. It cannot be run over the unstructured text in Chronicling America.
Therefore for detecting reproduced content in Chronicling America, where we do not have article texts, we deploy the sparse methods from Viral Texts [30]. Viral Texts was designed specifically for detecting reproduced texts in Chronicling America's noisy page-level OCR by looking for overlapping  $n$ -gram spans. To compare this method between American Stories and Chronicling America, we also run it over the articles in American Stories and then amalgamate these results at the page level.
Finally, we deploy the locality-sensitive hashing (LSH) specification from [28] to evaluate the performance of sparse methods on American Stories, using the same parameters. In particular we do this because the Viral Texts method is not designed to be run at the article level. As expected, we find that LSH performs better than Viral Texts at the article level, but both methods perform significantly worse than the neural methods, in line with the findings of [28].
Finally, we demonstrate that the content in American Stories can be clustered into news stories, following the same story between newspapers and across time. To create clusters of stories, we use a contrastively-trained biencoder.
We train this biencoder using data from allsides.com, a modern news website which shows how the same story is written by different newspapers. Articles on allsides.com are truncated. Groups of articles on the same story on allsides.com were used to create positive pairs. For negatives, we used the fact that each article is labelled with various tags, and also that we know which news source each article came from. For each article we take the article from the same news source, with different topic tags, that had the largest cosine similarity, using the biencoder before finetuning. In the cases where there were no articles with different tags from the same news source, we use articles from a news source which is lifted with the same political leaning. The specification that an article has different tags is important for making sure that articles are actual negatives.
Overall this gave 26,194 unique articles, with 18,382 positive pairs and 18,445 negative pairs. We featurized the data as "headline [sep] article" and we finetuned the biencoder from [28] as in experiments we found that this outperformed finetuning an MPNet Sentence BERT model [26, 32] directly. We optimised hyperparameters using hyperband [17]. The best model was trained fro 9 epochs, on a single GPU, with a batch size of 32, a warm up percent of 0.392. We optimised online contrastive loss [8], with and a loss margin of 0.497.
At inference time, we cluster using single-linkage clustering, with a cosine similarity threshold of 0.92. We control cluster size using leiden community detection [33]. We deduplicate the content using the method outlined in the section above. We take all articles that are reprinted at least five times, and run same story clustering over a year at a time.
Dataset details
Dataset URL
The dataset can be found at https://huggingface.co/datasets/dell-research-harvard/AmericanStories.
This dataset has structured metadata following schema.org, and is readily discoverable.5
Training labels for the individual models detailed in this paper are also available, and can be found at https://huggingface.co/datasets/dell-research-harvard/AmericanStoriesTraining.
The DOI for this dataset is: 10.57967/hf/0757.
See https://search.google.com/test/rich-results/result?id=esZkoGgfOsLlnkrvwx9nSQ for full metadata.
The dataset has a Creative Commons CC-BY license.
Dataset usage
The dataset is hosted on Hugging Face. Each year in the dataset is divided into a distinct file. The dataset can be easily downloaded using the datasets library:
As the dataset is very large, files for specific years can be downloaded by specifying them or users can download all data for all years. Additionally, we provide two options for the output type. The first contains data at the article level, with features like newspaper name, page number, edition, date, headline, byline, and article text. The second contains data at the scan level. It contains information including the scan metadata; all detected content regions like articles, photographs, and adverts; legibility information, and bounding box coordinates.
from datasets import load_dataset
# Download data for the year 1809 at the associated article level (Default)
dataset = load_dataset("dell-research-harvard/AmericanStories",
	"subset_years",
	year_list=['1809","1810"]
)
# Download and process data for all years at the article level
dataset = load_dataset("dell-research-harvard/AmericanStories",
	"all_years")
# Download and process data for 1809 at the scan level
dataset = load_dataset("dell-research-harvard/AmericanStories",
	"subset_years_content_regions",
	year_list=['1809"]
)
# Download ad process data for all years at the scan level
dataset = load_dataset("dell-research-harvard/AmericanStories",
	"all_years_content Regions")
Users can find more information on accessing the dataset using the dataset card on Hugging Face.
Author statement
We bear all responsibility in case of violation of rights.
Maintenance Plan
We have chosen to host the dataset on huggingface as this ensures long-term access and preservation of the dataset.
Dataset documentation and intended uses
We follow the datasheets for datasets template [6]. Additionally, we have completed the dataset card on Hugging Face which can be accessed using the link to the dataset on Hugging Face hub  $^{6}$
Moreover, we have included our responses to The Machine Learning Reproducibility Checklist as outlined in table S-2.
<sup>6</sup>https://huggingface.co/datasets/dell-research-harvard/AmericanStories
For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.
The dataset was created to provide researchers with a large, high-quality corpus of structured and transcribed newspaper article texts from historical local American newspapers. These texts provide a massive repository of information about topics ranging from political polarization to the construction of national and cultural identities to the minutiae of the daily lives of people's ancestors. The dataset will be useful to a wide variety of researchers including historians, other social scientists, and NLP practitioners.
Who created this dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
The dataset was created by a team of researchers at Harvard University, New York University, Northwestern Kellogg School, MIT, and Princeton University, led by Melissa Dell.
Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.
Funding was provided by the Harvard Data Science Initiative, compute credits that Microsoft Azure provided to the Harvard Data Science Initiative, Harvard Catalyst, and the Harvard Economics Department Ken Griffin Fund for Research on Development Economics and Political Economy.
Any other comments?
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.
Dataset instances are detected content regions in newspaper page scans from the Library of Congress's Chronicling America collection. In the cases of article, headline, image caption, and byline regions, a text transcription is included if the page is written in English.
How many instances are there in total (of each type, if appropriate)?
Version 0.1.0 of American Stories contains 402 million content regions, 294 million of which include a text transcription.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
The Version 1.0 of the dataset will contain all possible instances. Version 0.1.0 contains approximately  $40\%$  of all instances as of 6/7/23.
What data does each instance consist of? "Raw" data (e.g., unprocessed text or images) or features? In either case, please provide a description.
Each instance includes: a unique content region id, its detected class (ARTICLE, HEADLINE, CAPTION, BYLINE, IMAGE, AD, TABLE, HEADER, PAGE NUMBER, or MASTHEAD), and the pixel coordinates of the newspaper page bounding box for the identified region. If the content region is classified as ARTICLE, HEADLINE, CAPTION, or BYLINE, the transcribed text is also provided.
Is there a label or target associated with each instance? If so, please provide a description.
Content regions are labeled by their model predicted class. Text article transcriptions have no label.
Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g. because it was unavailable). This does not include intentionally removed information but might include, e.g., redacted text.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit.
All articles and other content regions include metadata that can definitively determine relationships to other content regions. For example, two articles with the same lccn (newspaper identifier), edition, and page number are from the same newspaper page scan.
Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.
There are no recommended splits.
Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.
Layout detection, OCR, and article association all introduce noise.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
The provided text data are self-contained. Some applications could require downloading the original scans, which are at https://chroniclingamerica.loc.gov/. All scans are freely available and in the public domain.
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals non-public communications)? If so, please provide a description.
The dataset is drawn entirely from image scans in the public domain that are freely available for download from the Library of Congress's website.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.
Texts in the dataset reflect attitudes and values of a large, diverse group of newspaper editors and writers in the period they were written (1790-1960) and include content that may be considered offensive for a variety of reasons.
Does the dataset relate to people? If not, you may skip the remaining questions in this section.
Yes. The dataset contains news about people.
Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.
It may be possible to infer certain characters about individuals covered in the news historically from the data. The authors of the dataset do not identify any subpopulations.
Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.
If an individual appeared in the news during this period, then texts may contain their name and other information. In some cases, it may be possible to link individuals to information on ancestry websites or Wikipedia (in the case of prominent historical figures). We do not attempt to do so in this paper.
Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.
The data are drawn entirely from newspaper scans in the public domain.
Collection Process