SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Emily Silcock<sup>1</sup>, Abhishek Arora<sup>1</sup>, Melissa Dell<sup>1,2*</sup>
<sup>1</sup>Harvard University; Cambridge, MA, USA.
$^{2}$ National Bureau of Economic Research; Cambridge, MA, USA.
*Corresponding author: melissadell@fas.harvard.edu.
A diversity of tasks use language models trained on semantic similarity data. While there are a variety of datasets that capture semantic similarity, they are either constructed from modern web data or are relatively small datasets created in the past decade by human annotators. This study utilizes a novel source, newly digitized articles from off-copyright, local U.S. newspapers, to assemble a massive-scale semantic similarity dataset spanning 70 years from 1920 to 1989 and containing nearly 400M positive semantic similarity pairs. Historically, around half of articles in U.S. local newspapers came from newswires like the Associated Press. While local papers reproduced articles from the newswire, they wrote their own headlines, which form abstractive summaries of the associated articles. We associate articles and their headlines by exploiting document layouts and language understanding. We then use deep neural methods to detect which articles are from the same underlying source, in the presence of substantial noise and abridgement. The headlines of reproduced articles form positive semantic similarity pairs. The resulting publicly available HEADLINES dataset is significantly larger than most existing semantic similarity datasets and covers a much longer span of time. It will facilitate the application of contrastively trained semantic similarity models to a variety of tasks, including the study of semantic change across space and time.
Transformer language models contrastively trained on large-scale semantic similarity datasets are integral to a variety of applications in natural language processing (NLP). Contrastive training is often motivated by the anisotropic geometry of pre-trained transformer models like BERT [5], which complicates working with their hidden representations. Representations of low frequency words are pushed outwards on the hypersphere, the sparsity of low frequency words violates convexity, and the distance between embeddings is correlated with lexical similarity. This leads to poor alignment between semantically similar texts and poor performance when individual term representations are pooled to create a representation for longer texts [21]. Contrastive training reduces anisotropy [25].
A variety of semantic similarity datasets have been used for contrastive training [18]. Many of these datasets are relatively small, and the bulk of the larger datasets are created from recent web texts; e.g., positive pairs are drawn from the texts in an online comment thread or from questions marked as duplicates in a forum. To provide a semantic similarity dataset that spans a much longer length of time and a vast diversity of topics, this study develops HEADLINES (Historical Enormous-Scale Abstractive DupLIcate News Summaries), a massive dataset containing nearly 400 million high quality semantic similarity pairs drawn from 70 years of off copyright U.S. newspapers. Historically, around half of content in the many thousands of local newspapers across the U.S. was taken from centralized sources such as the Associated Press wire [8]. Local newspapers reprinted wire articles
but wrote their own headlines, which form abstractive summaries of the articles. Headlines written by different papers to describe the same wire article form positive semantic similarity pairs.
To construct HEADLINES, we digitize front pages of off-copyright local newspapers, localizing and OCRing individual content regions like headlines and articles. The headlines, bylines, and article texts that form full articles span multiple bounding boxes - often arranged with complex layouts - and we associate them using a model that combines layout information and language understanding [14]. Then, we use neural methods from [23] to accurately predict which articles come from the same underlying source, in the presence of noise and abridgement. HEADLINES allows us to leverage the collective writings of many thousands of local editors across the U.S., spanning much of the 20th century, to create a massive, high-quality semantic similarity dataset. HEADLINES captures semantic similarity with minimal noise, as positive pairs summarize the same underlying texts.
This study is organized as follows. Section 2 describes HEADLINES, and Section 3 relates it to existing datasets. Section 4 describes and evaluates the methods used for dataset construction, Section 5 benchmarks the dataset, and Section 6 discusses limitations and intended usage.
2 Dataset Description
HEADLINES contains 393,635,650 positive headline pairs from off-copyright newspapers. Figure 1 plots the distribution of content by state.
Figure 1: Geographic variation in source of headlines
Dataset statistics by decade are provided in Table 1. Content declines sharply in the late 1970s, due to a major copyright law change effective January 1, 1978.
Decade	Headline Count	Cluster Count	Positive Pair Count	Word Count	Words Per Headline	Line Count	Lines Per Headline	Character Error Rate
1920s	4,889,942	1,032,108	28,928,226	68,486,589	14.0	18,983,014	3.9	4.3%
1930s	5,519,472	1,126,566	37,529,084	75,210,423	13.6	21,905,153	4.0	3.7%
1940s	6,026,940	1,005,342	62,397,004	61,629,003	10.2	19,538,729	3.2	2.4%
1950s	7,530,810	1,192,858	100,527,238	61,127,313	8.1	20,823,786	2.8	2.3%
1960s	6,533,071	926,819	108,415,279	46,640,311	7.1	16,408,148	2.5	3.7%
1970s	3,664,201	585,782	52,981,097	24,472,831	6.7	7,829,510	2.1	3.2%
1980s	703,052	170,507	2,857,722	5,161,537	7.3	1,502,893	2.1	1.5%
Total	34,867,488	6,039,982	393,635,650	342,728,007	9.8	106,991,233	3.1
Table 1: Descriptive statistics of HEADLINES.
The supplementary materials summarize copyright law for works first published in the United States. The newspapers in HEADLINES are off-copyright because they were published without a copyright notice or did not renew their copyright, required formalities at the time. Far from being an oversight, it was rare historically to copyright news, outside the nation's most widely circulated papers. The headlines in our dataset were written by editors at these local papers, and hence are in the public domain and anyone can legally use or reference them without permission.
It is possible that a newspaper not itself under copyright could reproduce copywritten content from some third party - the most prevalent example of this is comics - but this does not pose a problem for HEADLINES, since the dataset is built around the locally written headlines that describe the same
wire articles. If we were to accidentally include a syndicated headline, it would be dropped by our post-processing, since we drop headline pairs within a Levenshtein edit distance threshold of each other. It is also worth noting that a detailed search of U.S. copyright catalogs by [19] did not turn up a single instance of a wire service copyrighting their articles. (Even if they had, however, it would not pose a problem for headlines, since they were written locally.)
Figure 2 shows examples of semantic similarity pairs.
Figure 2: Semantic similarity examples, showing article image crops and OCR'ed headlines.
We quantify variation in HEADLINES across years, using a measure reminiscent of Earth Mover distance. This measure computes how much each text in a query dataset (e.g., 1920 headlines) would have to change (in embedding space) to have the same representation as the closest text in a key dataset (e.g., 1930 headlines).
Specifically, we first take a random sample of 10,000 texts per year. For year  $j$ , we embed texts  $t_{1j} \ldots t_{10,000j}$  using all-mpnet-base-v2. We choose MPNet because it has been shown to perform well across a variety of embedding datasets and tasks [18]. For each of these  $t_{ij}$ , we compute the most similar embedding in year  $k$ , measured by cosine similarity. This gives us a vector of similarity measures  $s_{1jk} \ldots s_{10,000jk}$ , that for each text in year  $j$  measure proximity to the most similar text in year  $k$ . We average these similarities to calculate  $SIM_{jk}$ .<sup>1</sup> Figure 3, which plots the  $SIM_{jk}$ , shows that similarity increases with temporal proximity. The dark square towards the upper left is World War 2, during which newspapers coverage was more homogeneous due to the centrality of the war.
Figure 3: Average similarity between different years of HEADLINES.
<sup>1</sup>We end in 1977, as we have very few texts after this date.
HEADLINES is useful for training and evaluating models that aim to capture abstractive similarity, whether using the embeddings for tasks like clustering, nearest neighbor retrieval, or semantic search [18]. Because it contains chronological content over a long span of time, it can be used to evaluate dynamic language models for processing continuously evolving content [1, 15], as well as how large language models can be adapted to process historical content [17, 4, 16]. Likewise, it can be used to train or evaluate models that predict the region or year a text was written [20]. In addition, it is useful for training models and developing benchmarks for a number of downstream tasks, such as topic classification of vast historical and archival documents, which have traditionally been classified by hand. This is an extremely labor-intensive process, and as a result many historical archives and news collections remain largely unclassified. Similarly, it could facilitate creating a large scale dataset to measure term-level semantic change, complementing existing smaller-scale SemEval tasks.
HEADLINES has a Creative Commons CC-BY license, to encourage widespread use, and is available on Huggingface.2
3 Existing Semantic Similarity Datasets
There is a dense literature on semantic similarity, with datasets covering diverse types of textual similarity and varying greatly in size. The focus of HEADLINES on semantic similarity in historical texts sets it apart from other widely used datasets. It also dwarfs the size of most existing datasets, aggregating the collective work of 20th century newspapers editors, from towns across the U.S. Its paired headlines summarize the same text, rather than being related by other forms of similarity frequently captured by datasets, such as being in the same conversation thread or answering a corresponding question.
One related class of semantic similarity datasets consists of duplicate questions from web platforms, e.g., questions tagged by users as duplicates from WikiAnswers (77.4 million positive pairs) [6], duplicate stack exchange questions (around 304,000 duplicate title pairs) [2], and duplicate quora questions (around 400,000 duplicate pairs) [12]. Alternatively, MS COCO [3] used Amazon's Mechanical Turk to collect five captions for each image in the dataset, resulting in around 828,000 positive caption pairs. In Flickr [26], 317,695 positive semantic similarity pairs describe around 32,000 underlying images. Like HEADLINES, positive pairs in these datasets refer to the same underlying content, but are describing an image rather than providing an abstractive summary of a longer text. In future work, HEADLINES could be expanded to include caption pairs describing the same underlying photo wire image, as local papers frequently wrote their own captions.
Online comment threads have also been used to train semantic similarity models. For example, the massive scale Reddit Comments [11] draws positive semantic similarity pairs from Reddit conversation threads between 2016 and 2018, providing 726.5 million positive pairs. Semantic similarity between comments in an online thread reflects conversational similarity, to the extent the thread stays on topic, rather than abstractive similarity. Likewise, question-answer and natural-language inference datasets are widely used for semantic similarity training. While other datasets exploit abstractive summaries - e.g., Semantic Scholar (S2ORC) has been used to create semantic similarity pairs of the titles and abstracts of papers that cite each other - to our knowledge there are not large-scale datasets with abstractive summaries of the same underlying texts.
A wide variety of text embedding datasets have been combined into the Massive Text Embedding Benchmark (MTEB) [18], which evaluates 8 embedding tasks on 58 datasets covering 112 languages. We measure the similarity between HEADLINES and the English datasets in MTEB, using the Earth Mover-style distance, described in Section 2. As above, we first take a random sample of (up to) 10,000 texts from each decade of HEADLINES, as well as each of the English datasets in MTEB (if the dataset contains fewer than 10K texts, we use the full dataset and limit the comparison dataset to the same number of randomly selected texts). For dataset  $j$ , we embed texts  $t_{1j} \ldots t_{10,000j}$ . For each of these  $t_{ij}$ , we compute the most similar embedding in dataset  $k$ , averaging these across all texts in  $j$  to compute  $SIM_{jk}$ .  $SIM_{jk}$  need not be symmetric. Suppose dataset  $j$  is highly homogeneous, whereas dataset  $k$  is heterogeneous.  $SIM_{jk}$  may be high, because the similar embeddings in homogeneous dataset  $j$  are close to a subset of embeddings in dataset  $k$ . On the other hand,  $SIM_{kj}$  may be low, because most texts in dataset  $k$  are dissimilar from texts in homogeneous dataset  $j$ .
$^{2}$ https://huggingface.co/datasets/dell-research-harvard/headlines-semantic-similarity
Dataset sizes are drawn, when applicable, from a table documenting the training of Sentence BERT [21].
Figure 4: Average similarity between HEADLINES (by decade) and the English datasets in the Massive Text Embedding Benchmark. The similarity measure is described in the text.
Figure 4 shows the entire similarity matrix between HEADLINES and the English datasets in MTEB; rows are the query dataset and columns are the key. The style of the figure was adapted from [18]. The average similarity between HEADLINES and MTEB datasets is 31.4, whereas the average similarity between MTEB datasets and HEADLINES is 33.5, supporting our supposition that HEADLINES is diverse on average relative to other benchmarks. This average max similarity shows that there is ample information in HEADLINES not contained in existing datasets.
Table 2 shows examples where the nearest text to a headline in an MTEB dataset is highly similar, versus of average similarity. In some cases, highly similar texts in fact have a different meaning, but the limited context in the MTEB datasets makes this difficult to capture.
Headline	Highly similar texts	Similarity
"Inflation Cuts are Questioned"	Reddit: “Today FOMC Resumed Meeting Inflation getting out of hand... Maybe... Maybe Not”	0.55
“Bear Bites Off Arm of Child”	StackExchange: “How to handle animal AI biting and holding onto the character...”	0.46
“British Cruiser Reported Sunk”	Twitter: “That's It, Britain is Sunk”	0.61
“Will Free Press Dance to Government Tune”	Twitter: “Donald Trump v. a free press”	0.60
“Partitioning Plan Unsatisfactory, Ike Declares”	Ubuntu Questions: “Partitioning Issues”	0.51
Headline	Average similarity texts	Similarity
“Reds Knot Strong Tie”	ArXiv: “Knots and Polytopes”	0.34
“SHOWERS PROMISED TO END HEAT WAVE Two Deaths From Heat Over Weekend”	StackOverflow: “how to annotate heatmap with text in matplotlib?”	0.27
“Salary Boost Due For Some On Labor Day”	Twitter: “Glassdoor will now tell you if you're being underpaid”	0.31
“40 Old Ladies Now In Senate Says Rogers”	Quora: “Is 19 young?”	0.28
Table 2: This table shows similarities between example texts.
4 Dataset Construction and Evaluation
4.1 Digitization
We digitized front pages from off-copyright newspapers spanning 1920-1989. We recognize layouts using Mask RCNN [10] and OCR the texts. We transcribed the headlines using Tesseract. The digitization was performed using Azure F-Series CPU nodes.
We evaluate this OCR on a hand annotated sample of 300 headlines per decade. Table 1 reports the character error rate, defined as the Levenshtein distance between the transcribed text and the ground truth, normalized by the length of the ground truth. As expected, OCR quality improves over time, as there is less damage from aging and fewer unusual fonts.
4.2 Article Association
Newspaper articles have complex and irregular layouts that can span multiple columns (Figure 5).
(a) The green article is notcontiguous and it is unclear if the second part continues the purple, green, or even blue articles if the text is not used.
(b) The green article is nested inside the blue article, which causes rule-based and image-based models to either associate the green and blue articles together, or miss the second half of the blue article.
(c) The smallest rectangle that contains the green article also contains the purple and red articles.
Figure 5: Articles that are misassociated with rule-based or image-based methods
We associate the (potentially multiple) headline bounding boxes with the (potentially multiple) article bounding boxes and byline boxes that comprise a single article using a combination of layout information and language understanding. A rule-based approach using the document layouts gets many of the associations correct, but misses some difficult cases where article bounding boxes are arranged in complex layouts. Language understanding can be used to associate such articles but must be robust to noise, from errors in layout detection (e.g. from cropping part of a content bounding box or adding part of the line below) and from OCR character recognition errors.
Hand annotating a sufficiently large training dataset would have been infeasibly costly. Instead, we devise a set of rules that - while recall is relatively low - have precision above 0.99, as measured on an evaluation set of 3,803 labeled bounding boxes. The algorithm exploits the positioning of article bounding boxes relative to headline boxes (as in Figure 6), first grouping an article bounding box with a headline bounding box if the rules are met and then associating all article bounding boxes grouped with the same headline together. Since precision is above 0.99, the rule generates nearly perfect silver-quality training data.
Figure 6: Illustration of article association pipeline
To train a language model to predict whether article box B follows box A, we embed the first and last 64 tokens of the texts in boxes B and A, respectively, with a RoBERTa base model [14]. The pair is positive when B follows A. The training set includes 12,769 positive associated pairs, with training details described in the supplementary materials. At inference time, we first associate texts using the rule-based approach, described in Figure 6, which has extremely high precision. To improve recall, we then apply the RoBERTa cross-encoder to remaining article boxes that could plausibly be associated, given their coordinates. Texts cannot be followed by a text that appears to the left, as layouts always proceed from left to right, so these combinations are not considered.
(1) F1	(2) Recall	(3) Precision
Full Article Association	93.7	88.3	99.7
Table 3: This table evaluates the full article association model.
We evaluate this method on a hand-labeled dataset of 214 scans. Full details of this dataset are given in the appendix. Table 3 evaluates recall, precision and F1 for associated articles. The F1 of 93.7
is high, and precision is extremely high. Errors typically occur when there is an error in the layout analysis or when contents are very similar, e.g., grouping multiple obituaries into a single article.
4.3 Detecting Reproduced Content
Accurately detecting reproduced content can be challenging, as articles were often heavily abridged by local papers to fit within their space constraints and errors in OCR or article association can add significant noise. Table 4 shows examples of reproduced articles.
SAN FRANCISCO (#—Presi-dent Eisenhower was reported ready to open the U. N.'s 10th anniversary session today with an important policy declaration.
This word came from informed sources as the Big Four foreign ministers prepared for a private huddle tonight to plan the meeting of their chiefs of government in Geneva July 18.
The President was scheduled to speak at 6 p.m., EDT. Some diplomats believed his speech would have special significance, coming as it does just a month before the top level talks.
The first direct contact between Soviet Foreign Minister V. M. Molotov and the Western leaders was established last night at a dinner given by Colombian Ambassador Eduardo Zuleta Angel There was no information as tc whether any serious discussion: were held, but the two sides dic have a chance to sense the at-]mosphere which will prevail to-]night.. Diplomatic sources reported th Russian likely would press for : Declaration of some sort by the ! 10th anniversary meeting
SAN FRANCISCO #President Eisenhower was reported ready to open the U_N's 10th anniversary session today with an important policy declaration.
Some diplomats believed Eisenhower's speech would have special significance, coming as it does just a month before the top-level talks! The first direct contact between (Soviet Foreign Minister V. M.) Molotov and the Western leaders! was established last night "at a dinner given by Colombian Ami bassador Eduardo Zulcta Angel.
! Diplomatic sources reported the Russians likely would press for a declaration of some sort by the 11th anniversary meeting.
"SY MAA  
MARKKRKRELSUN  
SAN FRANCISCO  
#8—President Eisenhower was reported ready to open the U.N.'s 10th anniversary session today with an important! policy declaration,
Some diplomats believed Eisen.hower's speech would have special! significance, coming as it does just a month before the top-level talks.
Mulles to Bo Host The first direct contact between Soviet Foreign Minister V.M. Molotov and the Western leaders was established last night at a dinner given by Colombian Ambassador Eduardo Zuleta Angel.
Diplomatic sources reported the Russians likely would press for a declaration of some sort by the 10th anniversary meeting.
Table 4: Examples of reproduced articles. Additions are highlighted, and OCR errors are underlined.
We use the model developed by [23], who show that a contrastively trained neural MPNet bi-encoder - combined with single linkage clustering of article representations - accurately and cheaply detects reproduced content. This bi-encoder is contrastively trained on a hand-labeled dataset (detailed in the appendix) to create similar representations of articles from the same wire source and dissimilar representations of articles from different underlying sources, using S-BERT's online contrastive loss [9] implementation.
We run clustering on the article embeddings by year over all years in our sample. In post-processing, we use a simple set of rules exploiting the dates of articles within clusters to remove content like weather forecasts and legal notices, that are highly formulaic and sometimes cluster together when they contain very similar content (e.g. a similar 5-day forecast) but did not actually come from the same underlying source. We remove all headline pairs that are below a Levenshtein edit distance, normalized by the min length in the pair, of 0.1, to remove pairs that are exact duplicates up to OCR noise. Training and inference were performed on an A6000 GPU card. More details are provided in the supplementary materials.
To evaluate how well the model detects reproduced content, we use a labeled sample of all front page articles appearing in the newspaper corpus for three days in the 1930s and 1970s, taken from [23]. This sample consists of 54,996 positive reproduced article pairs and 100,914,159 negative pairs. The
large-scale labeled evaluation dataset was generated using the above pipeline, so the evaluation is inclusive of any errors that result from upstream layout detection, OCR, or article association errors.