A Massive Scale Semantic Similarity Dataset of Historical English
Authors: Emily Silcock
Abhishek Arora
Melissa Dell
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: semantic similarity
semantic change

Abstract: A diversity of tasks use language models trained on semantic similarity data. While there are a variety of datasets that capture semantic similarity, they are either constructed from modern web data or are relatively small datasets created in the past decade by human annotators. This study utilizes a novel source, newly digitized articles from off-copyright, local U.S. newspapers, to assemble a massive-scale semantic similarity dataset spanning 70 years from 1920 to 1989 and containing nearly 400M positive semantic similarity pairs. Historically, around half of articles in U.S. local newspapers came from newswires like the Associated Press. While local papers reproduced articles from the newswire, they wrote their own headlines, which form abstractive summaries of the associated articles. We associate articles and their headlines by exploiting document layouts and language understanding. We then use deep neural methods to detect which articles are from the same underlying source, in the presence of substantial noise and abridgement. The headlines of reproduced articles form positive semantic similarity pairs. The resulting publicly available HEADLINES dataset is significantly larger than most existing semantic similarity datasets and covers a much longer span of time. It will facilitate the application of contrastively trained semantic similarity models to a variety of tasks, including the study of semantic change across space and time.

A Massive Scale Semantic Similarity Dataset of Historical English
Emily Silcock', Abhishek Arora', Melissa Dell
Harvard University; Cambridge, MA, USA.
National Bureau of Economic Research; Cambridge, MA, USA.
*Corresponding author:
melissadell
Abstract
A diversity of tasks use language models trained on semantic similarity data. While there are variety of datasets that capture semantic similarity, they are either constructed from modern web data are relatively small datasets created in the past decade by human annotators. This study utilizes a novel source, newly digitized articles from off-copyright, local U.S. newspapers, to assemble a massive-scale semantic similarity dataset spanning 70 years from 1920 to 1989 and containing nearly 400M positive semantic similarity pairs. Historically, around half of articles in U.S. local newspapers came from newswires like the Associated Press. While local papers reproduced articles from the newswire, they wrote their own headlines, which form abstractive summaries of the associated articles. We associate articles and their headlines by exploiting document layouts and language understanding. We then deep neural methods to detect which articles are from the same underlying source, in the presence of substantial noise and abridgement. The headlines of reproduced articles form positive semantic similarity pairs. The resulting publicly available HEADLINES dataset is significantly larger than most existing semantic similarity datasets and covers much longer span of time. It will facilitate the application of contrastively trained semantic similarity models to a variety of tasks, including the study of semantic change across space and time,
1 Introduction
Transformer language models contrastively trained on large-scale semantic similarity datasets are integral to variety applications in natural language processing (NLP). Contrastive training is often motivated by the anisotropic geometry of pre-trained transformer models like BERT (5), which complicates working with their hidden representations. Representations of low frequency words are pushed outwards on the hypersphere, the sparsity of low frequency words violates convexity, and the distance between embeddings is correlated with lexical similarity. This leads to poor alignment between semantically similar texts and poor performance when individual term representations pooled to create representation for longer texts 21 Contrastive training reduces anisotropy [25].
A variety of semantic similarity datasets have been used for contrastive training [8] Many of these datasets are relatively small, and the bulk of the larger datasets are created from recent web texts: egg. positive pairs are drawn from the texts in an online comment thread from questions marked as duplicates a forum. To provide semantic similarity dataset that spans much longer length of time and vast diversity of topics, this study develops HEADLINES (Historical Enormous-Serr Abstractive IcpLIcate News Summaries), a massive dataset containing nearly 400 million high quality semantic similarity pairs drawn from 70 years of off copyright U.S. newspapers. Historically, around half of content in the many thousands of local newspapers across the U.S. was taken from centralized sources such as the Associated Press wire 8 Local newspapers reprinted wire articles
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

but wrote their own headlines, which form abstractive summaries of the articles. Headlines written by different papers describe the same wire article form positive semantic similarity pairs.
To construct HEADLINES, we digitize front pages of off-copyright local newspapers, localizing and OCRing individual content regions like headlines and articles. The headlines, bylines, and article texts that form full articles span multiple bounding boxes often arranged with complex layouts and we associate them using model that combines layout information and language understanding 14]. Then, we use neural methods from 23 to accurately predict which articles come from the same underlying source, in the presence of noise and abridgement. HEADLINES allows us leverage the collective writings of many thousands of local editors across the U.S., spanning much of the 20th century, create a massive, sigh-qualtty semantic similarity dataset. HEADLINES captures semantic similarity with minimal noise, as positive pairs summarize the same underlying texts.
This study is organized follows. Section Decctiondessrinng describes HEADLINES, and Section 3 relates it existing datasets. Section 4 describes and evaluates the methods used for dataset construction, Section benchmarks the dataset, and Section discusse limitations and intended usage.
Dataset Description
HEADLINES contains 393,635,650 positive headline pairs from off-copyright newspapers. Figure plots the distribution of content by state.
Figure 1: Geographic variation in source headlines
Dataset statistics by decade are provided in Table ! Content declines sharply in the late 1970s, due major copyright law change effective January 1978.
Decade
Headline Count
Cluster Count
Positive Pair Count
Word
Words Per Headline
Line Count
Lines Per Headline
Character Froo Rute
Count
1920s
4,889.943
1,032. 108
28,928.2. 226
68.486.589
14.0
18.983.014
3.9
4.3%
1930s
5,519,472
1,126,560
37.529.06 008
75,210.423
13.6
21,905,15:
4.0
3,7%
1940s
6,026,940
1,05,342
62,397.004
61,629,003
10.2
19.538,729
3.2
2.4%
1950s
7,530,810
1,192,888
100,527.238
61,127.313
8.1
20,823,786
2.8
2.3%
1960s
6,533,071
926.819
105.415.279
46.640,311
16,408, 148
2.5
3.7%
1970s
3,664,201
585, 785,
52,981 ,097
24,472.831
6,7
7,829,510
2.1
3,2%
1980s
703.052
170.507
2.857,722
5,161,537
1,502, 893
2.1
Total
34,867,488
6,039,982
393.635,650
342,728,001
9.8
106,991,233
3.1
Table 1: Descriptive statistics of HEADLINES.
The supplementary materials summarize copyright law for works first published in the United States. The newspapers HEADLINES are off-copyright because they were published without a copyright notice did not renew their copyright, required formalities at the time. Far from being an oversight, was rare historically to copyright news, outside the nation's most widely circulated papers. The headlines in dataset were written by editors these local papers, and hence are in the public domain and anyone can legally use or reference them without permission.
It is possible that a newspaper not itself under copyright could reproduce copywritten content from some third party the most prevalent example of this is comics but this does not pose a problem for HEADLINES, since the dataset is built around the locally written headlines that describe the same

---

wire articles. If we were to accidentally include a syndicated headline, it would be dropped by our post-processing, since we drop headline pairs within Levenshtein edit distance threshold of each other. It is also worth noting that a detailed search of U.S. copyright catalogs by [19 did not turn up a single instance of a wire service copyrighting their articles. (Even they had, however, would not pose problem for headlines, since they were written locally.)
Figure shows examples of semantic similarity pairs.
GENERAL
WRANGEL IS IN BAD TANGLE
FIVK A DAY 2. ear Old W Yon't Quit Cigurettes
Boy 2, Just Has to Have Cigarettes Daily
WRANGEL
L LOSING
GROUND
Wist Contacl Collanes
PROCIA
the por
ioeal dive
VIRANII IS
Buaral Josae Hoope Hote
Won't Quil Ovarcties
nont broak lie of kusi.
SANUGAA WANTOUL LOGENO
peatss les hokke
CAcuR Ouerra OF salirna
Flasts Tvens Sacck
Hove Ogaretion Duily
Figure 2: Semantic similarity examples, showing article image crops and OCR' ed headlines.
We quantify variation in HEADLINES across years, using a measure reminiscent of Earth Mover distance. This measure computes how much each text in query dataset (e.g.. 1920 headlines) would have to change (in embedding space) to have the same representation as the closest text in a key dataset (e.g.. 1930 headlines).
Specifically, we first take random sample of 10,000 texts per year. For year j. we embed texts t1y...11,,00 ,0,00, 000g using all bal-mpnet-base-v2. We choose MPNet because it has been shown to perform well across a variety of embedding datasets and tasks [18]. For each these taj. we compute the most similar embedding in year k, measured by cosine similarity. This gives us vector of similarity measures $10.k-..10..00jkk that for each text in year j measure proximity to the most similar text in year kt We average these similarities to calculate S1 jk Figure 3 which plots the S1M shows that similarity increases with temporal proximity. The dark square towards the upper left is World War 2, during which newspapers coverage was more homogeneous due to the centrality of the war.
2.58
0.54
3.52
Figure 3: Average similarity between different years of HEADLINES.
We end in 1977, we have very few texts after this date.

---

HEADLINES is useful for training and evaluating models that aim to capture abstractive similarity, whether using the embeddings for tasks like clustering, nearest neighbor retrieval, or semantic search 18 Because it contains chronological content over a long span of time, it can be used to evaluate dynamic language models for processing continuously evolving content [ 15], as well as how large language models can be adapted to process historical content 117 4 [6] Likewise, it can be used to train or evaluate models that predict the region or year a text was written 20] In addition, it is useful for training models and developing benchmarks for a number of downstream tasks, such topic classification of vast historical and archival documents, which have traditionally been classified by hand. This is extremely labor-intensive process, and result many historical archives and news collections remain largely unclassified. Similarly, it could facilitate creating a large scale dataset to measure term-level semantic change, complementing existing smaller-scale SemEval tasks.
HEADLINES has a Creative Commons CC-BY license, to encourage widespread use, and is available on Huggingface 2
3 Existing Semantic Similarity Datasets
There is a dense literature on semantic similarity, with datasets covering diverse types of textual similarity and varying greatly in size. The foeus of HEADLINES on semantic similarity in historical texts sets it apart from other widely used datasets. It also dwarfs the size of most existing datasets, aggregating the collective work of 20th century newspapers editors, from towns across the U.S. Its paired headlines summarize the same text, rather than being related by other forms of similarity frequently captured by datasets, such as being in the same conversation thread or answering corresponding question.
One related class of semantic similarity datasets consists of duplicate questions from web platforms, e.g. questions tagged by users as duplicates from WikiAnswers (77.4 .44 million positive pairs) [6], duplicate stack exchange questions (around 304.000 duplicate title pairs) (2) and duplicate quora questions (around 400,000 duplicate pairs) [2] Alternatively, MS COCO 3 used Amazon's 's Mechanical Turk to collect five captions for each image in the dataset, resulting in around 828,000 positive caption pairs. In Flickr (26.. 317.695 positive semantic similarity pairs describe around 32,000 underlying images. Like HEADLINES, positive pairs in these datasets refer to the same underlying content, but are describing an image rather than providing an abstractive summary of a longer text. In future work, HEADLINES could be expanded include caption pairs describing the same underlying photo wire image, as local papers frequently wrote their own captions.
Online comment threads have also been used to train semantic similarity models. For example, the massive scale Reddit Comments [T]] draws positive semantic similarity pairs from Reddit conversation threads between 2016 and 2018, providing 726.5 million positive pairs. Semantic similarity between comments in an online thread reflects conversational similarity, to the extent the thread stays on topic, rather than abstractive similarity. Likewise, question-anse answer and natural- language inference datasets are widely used for semantic similarity training. While other datasets exploit abstractive summaries- e.g., Semantic Scholar (S2ORC) has been used to create semantic similarity pairs of the titles and abstracts of papers that cite each other to our knowledge there are not large-scale datasets with abstractive summaries of the same underlying texts.
A wide variety of text embedding datasets have been combined into the Massive Text Embedding Benchmark (MTEB) 18), which evaluates 8 embedding tasks on 58 datasets covering 112 languages. We measure the similarity between HEADLINES and the English datasets in MTEB, using the Earth Mover-style distance, described in Section 2 As above, we first take random sample of (up to) 10,000 texts from each decade of HEADLINES, as well as each of the English datasets in MTEB (if the dataset contains fewer than 10K texts, use the full dataset and limit the comparison dataset to the same number of randomly selected texts). For dataset j. we embed texts (1j...110.000 10.000-- For each of these lij. we compute the most similar embedding in dataset k; averaging these across all texts in j to compute S MM STM Mst. need not be symmetric. Suppose dataset j is highly homogeneous, whereas dataset k is heterogeneous. SIM, may be high, because the similar embeddings in homogeneous dataset j are close to a subset of embeddings in dataset k. On the other hand, S1M2; may be low, because most texts in dataset i are dissimilar from texts in homogeneous dataset j.
? hhtps://huginggface. htt.ggginnnnnccootttttttttteeeee coldatasets/deell-eeeene haw.r//eeeeeeeecaaaaaaaaahhhhiie havaachhedtiiessseaattiiiiimiiii
3 DDtaset sizes are drawn, when applicable, from table documenting the training of Sentence BERT 21

---

0.7
0.4
0.2
Figure 4: Average similarity between HEADLINES (by decade) and the English datasets in the Massive Text Embedding Benchmark. The similarity measure is described in the text.
Figure shows the entire similarity matrix between HEADLINES and the English datasets in MTEB; rows are the query dataset and columns are the key. The style of the figure was adapted from [8]. The average similarity between HEADLINES and MTEB datasets is 31.4, whereas the average similarity between MTEB datasets and HEADLINES is 33.5, supporting our supposition that HEADLINES is diverse average relative to other benchmarks. This average max similarity shows that there is ample information in HEADLINES not contained in existing datasets.
Table 2 shows examples where the nearest text headline in MTEB dataset is highly similar, versus of average similarity. some cases, highly similar texts in fact have different meaning, but the limited context in the MTEB datasets makes this difficult to capture.
Headline
Highly similar
Similarity
"Inflation are Questioned"
"Today FOMC Resumed Meeting Inflation getting of hand... Maybe... Maybe Not"
Reddit:
0.55
"Bear Bites Off Amn of Child"
StackExchange: "How handle animal Al biting
0.46
and holding the character...
"British Cruiser Reported Sunk"
Twitter: "That's 's It, Britain Sunk"
0.61
"Will Free Press Dance Government Tune" "Partitionngg Plan Unsatisfactory, lke Declares"
Twitter: "Donald Trump free press"
0.60
Ubantu Questions: "Partitioinng Issues"
0.51
Headline
Average similarity
Similarity
Strong Tie"
ArXiv:
Polytopes"
"SHOWERS PROMISED TO END HEAT WAVE Two Deaths From Heat Over Weekend"
StackOverflow: "how annotate heatmap with
0.27
text in matplotib??
"Salary Boost Due For Some On Labor Day"
Twitter: "Glassdoor will now tell you if youre heing underpaid"
0.31
Ladies Now In Senate Says Rogers"
Quora: "Is 19 young?"
0.28
Table 2: This table shows similarities between example texts.
5

---

4 Dataset Construction and Evaluation
4.1 Digitization
We digitized front pages from cofyrihttt newspapers spanning 1920-1989. We recognize layouts using Mask RCNN [0] and OCR the texts. We transcribed the headlines using Tesseract. The digitization was performed using Azure F-Series CPU nodes.
We evaluate this OCR on hand annotated sample of 300 headlines per decade. Table I the character error rate, defined as the Levenshtein distance between the transcribed text and the ground truth, normalized by the length of the ground truth. As expected, OCR quality improves over time, as there is less damage from aging and fewer unusual fonts.
4.2 Article Association
Newspaper articles have complex and irregular layouts that can span multiple columns (Figure 5
Senior Citizen
Coolidge
Programs Might
Budget Approved
Get Fund Boost
Border Patrol Raiding Short-Pay Bracero Users
Reagan Favored Nixon Shos
Crange Draws
Fire Damage Sleeping Box
Scho
for Salute As footiio Force
Trustees
Named
Road Band Bection Puttiuns Started
Areand: Appear First in Smeel Towas
City's Low Books Will be Trimmed
Varned Schemer
Today's
No Prote In Closin
CHEVROUT
INSIDE Vitd in DISPATCH
(b) The green article is nested in- side the blue article, which causes rule-based and image-based mod- els to either associate the green and blue articles together, or miss the second half of the blue article.
(a) The green article is not- contiguous and is unclear if the second part continues the purple, green, or even blue articles if the text is not used.
(c) The smallest rectangle that contains the green article also con- tains the purple and red articles.
Figure 5: Articles that are misassociated with rule-based or image-based methods
We associate the (potentially multiple) headline bounding boxes with the (potentially multiple) article bounding boxes and byline boxes that comprise single article using combination of layout information and language understanding. A rule-based approach using the document layouts gets many of the associations correct, but misses some difficult cases where article bounding boxes are arranged in complex layouts. Language understanding can be used to associate such articles but must be robust to noise, from errors in layout detection (e.g. from cropping part of content bounding box or adding part of the line below) and from OCR character recognition errors.
Hand annotating sufficiently large training dataset would have been infeasibly costly. Instead, we devise a set of rules that- while recall is relatively low have precision above 0.99, as measured on an evaluation set of 3,803 labeled bounding boxes. The algorithm exploits the positioning of article bounding boxes relative to headline boxes (as Figure 6)), first grouping an article bounding box with a headline bounding box if the rules are met and then associating all article bounding boxes grouped with the same headline together. Since precision is above 0.99, the rule generates nearly perfect silver-quality training data.

---

PIEacE NEWS.TRBUNK TRBUNE
News- TERS-THIUUNE
PORT PIEZE NEWS-TTRKCEE TRIKNEE
Ikee Gots Push-Button
like
Gets Push-Button Button Power
Fight
Puzh Power To Fight
(a) Article bounding boxes that are under the same headline and at the bottom of the page are used to create training data.
(b) At inference time, all bound- ing boxes that are directly under the same headline are associated with cach other.
(c) Any other article with head- line directly above not com- pared, leaving only a few orphans that are left to be associated.
Pour PIEaCE News TRISUNe
FORT PuEace News T
FORT PIEace Nows. TRMMITT
Push-Button Power To Fight
Gets Push Button Power To Fight
Gets Push-Button Power To Fight
(d) This orphan (green bounding box) is compared with the bound- ing box above it. In this case, is a separate article without head- line so it is not associated.
(c) Orphans are also compared with bounding boxes to the right. This orphan (green bounding box) is associated with the bounding box directly above.
(f) The final orphan is compared with two columns to the right, as sometimes articles skip columns. If further columns to right ex- ist, they are not compared.
Figure 6: Illustration of article association pipeline
To train language model to predict whether article box B follows box A, embed the first and last 64 tokens of the texts in boxes and A, respectively, with a RoBERTa base model [14]. The pair is positive when B follows A. The training set includes 12,769 positive associated pairs, with training details described in the supplementary materials. At inference time, we first associate texts using the rule-based approach, described in Figure 6. which has extremely high precision. To improve recall, we then apply the RoBERTa cross-encoder to remaining article boxes that could plausibly be associated, given their coordinates. Texts cannot be followed by a text that appears the left, as layouts always proceed from left to right, so these combinations are not considered.
(1)
(2)
(3) Precision
Fl
Recall
Full Article Association
93.7
88.3
99.7
Table 3: This table evaluates the full article association model.
We evaluate this method on a hand-labeled dataset of 214 scans. Full details of this dataset are given in the appendix. Table evaluates recall, precision and Fl for associated articles. The Fl of 93.7
7

---

is high, and precision is extremely high. Errors typically occur when there is an error in the layout analysis or when contents are very similar, egg., grouping multiple obituaries into a single article.
4.3 Detecting Reproduced Content
Accurately detecting reproduced content can be challenging, as articles were often heavily abridged by local papers to fit within their space constraints and errors in OCR or article association can add significant noise. Table 4 shows examples of reproduced articles.
FRANCISCO of Press-dent Eisenhower er was_ reported ready to open the N's 10th nivessury session today with an impoecant poliey declaration.
SAN FRANCISCO
"SY MAA
IKET TO OPEN UN 10TH ANNIVERSARY
Big Four Ministers To Meet Tonight San Francisco
g President Eisenbower was reported ready to ogen the U_N's 10th anniversary session
Ike's s Talk Will Open U. N. Meet
MARKRKKELSUU
SAN FRANCISCO
President Eises-
hower was repocted
today with an important policy deciaration.
ready to open the U.N's
Presidett Set To Speak:
10th anniversary session today with importan! policy doelaationn
This word from forsed sourees as Big Four foreign minis- ters prepared for privase
This word cume
Tenight
formed sources
Ministers
Big Four foreiga minis-
Arrange Private
This word came from in- formed sources the Big Four foreign minis- less geepased for privatc huddle tonight to plan the meeting of their chiefs govemment Geneva July
with
sons prepared for a private
Huddle Tenight
hodkle touight plan chiets
huddle tnoight plan meeting their of governmen in Geneva July 18.
meeting of govemmmnt in Geneva July 18.
aae reperted maat to
andt E217 teaee die
The Presideat was sehedl uled speak
Some diplomats believed Eisenhowr's speach would have special significancs, coming as does just month before the top-level takks The first direct contact berween (Sovier Foreign Minister Molotoe and the Wostern
for your booen
poosed
p.m. EDE Some diplocants be- leved his speech would
mood
Same diplomats believed
taen dilemor beleved Eoen
would
Eisen hower' speech
special significunce,
would
special!
coming as does just a mouth before the top level tilks.
The diss beleenn
significanes coming does just a month bofor the top-lavel talks. Mulles Hos
leades
The flsst direct between Savier Foneign Minister V. M. Molatov and the Westem lead- establisked last
Doleme meates
The first direct contact hetween Sovier Foreign
leaders was established last night "at a'dinner
lits encrrmery diveine
Minister V. M. Molntov snd the Westemn lead- as was established night at dinser given by
given by Coloomhian Anl bassidor Eduardo Zulena Angel.
night at a diuner given
by Colombian Ambas-
sador Fauaddo Zuleta An-
I Diplomatic somness reposted the "Rossims likely would press for deciaration of some sodt by the 1100 anniversary meeting
Colcerbian Ambasuadee Edhardo Zuleta Augel. Diplomatic sources reportad the Russians likely would press for 'declanaion by anniverssey meeting.
There
motion as to whether any
seriouss discussions were
held but the two sides die
the 'mmssphere which
will prevail I
Diplomatic sources reported the Russians likely would press for : '"Alclaaation of some sort by the ! 10th unniversary meeting
Table 4: Examples of reproduced articles. Additions are highlighted, and OCR errors are underlined.
We use the model developed by 233, who show that a contrastively trained neural MPNet bi-encoder combined with single linkage clustering of article representations accurately and cheaply detects reproduced content. This bi-encoder is contrastively trained on hand-labeled dataset (detailed in the appendix) to create similar representations of articles from the same wire source and dissimilar representations of articles from different underlying sources, using S-BERT's online contrastive loss 9 implementation.
We run clustering on the article embeddings by year over all years in our sample. In post-processing. we use a simple set of rules exploiting the dates of articles within clusters to remove content like weather forecasts and legal notices, that are highly formulaic and sometimes cluster together when they contain very similar content (e.g. a similar 5-day forecast) but did not actually come from the same underlying source. We remove all headline pairs that are below a Levenshtein edit distance, normalized by the min length in the pair, of 0.1, to remove pairs that are exact duplicates up to OCR noise. Training and inference were performed on an A6000 GPU card. More details are provided in the supplementary materials.
To evaluate how well the model detects reproduced content, we use a labeled sample of all front page articles appearing in the newspaper corpus for three days in the 1930s and 1970s, taken from (23]. This sample consists of 54,996 positive reproduced article pairs and 100,914, negative pairs. The
8

---

large-scale labeled evaluation dataset was generated using the above pipeline, so the evaluation is inclusive of any errors that result from upstream layout detection, OCR, or article association errors.
The neural bi-encoder methods achieve a high adjusted rand index (ART) of 91.5, compared to 73.7 for optimal local sensitive hashing specification, chosen the validation set. This shows that neural methods substantially outperform commonly used sparse methods for detecting reproduced content. The neural bi-encoder is slightly outperformed by adding re-ranking step that uses a neural cross-encoder on the best bi-encoder matches (ARI of 93.7). We do not implement this method because the pross-encodee doesn't It scale well. In contrast, the bi-encoder pipeline can be scaled to 10 million articles on single GPU in matter of hours, using a FAISS [3 backend.
Neural
Non-Neural
Most scalable
Bi-encoder (91.5)
LSH (73.7)
Less scalable
Re-ranking (93.7) N-gram overlap (75.0)
Table 5: The numbers parentheses are the Adjusted Rand Index for four different models bi-encoder, "re-ranking" strategy that combines a bi- and cross-encoder, locally sensitive hashing (LSH), and N N-gram overlap. Hyperparameters were chosen on the NEWS-COPY validation set, and all models were evaluated on the NEWS-COPY test set.
An error analysis provided in (23. Errors typically consist of articles about the same story from different wire services (e.g. the Associated Press and the United Press) or updates to story as new events unfolded. Both types of errors will plausibly still lead to informative semantic similarity pairs.
Benchmarking
We benchmark HEADLINES using variety of different language models and the MTEB clustering task. This task embeds texts using different base language models and then uses k the number of clusters the ground truth data - for k-means clustering. Following MTEB, we score the model using the v-measure (22). We should note that real-world problems are often framed as clustering tasks- rather than as classification tasks because & is unknown. By using & from the ground truth, makes the task easier. Nevertheless, examine this task to allow for comparison with the rest of the literature.
0.80
Model Name
0.78
MinLN-L112
MiniNN-L12-ma 11mil--12-mmlitiinggaall
0.76
MPNet
MPNat- Muttt--utiilnguall
STS Large
ST5-XL
0.74
STS XXL
coCondeneermmmmrro
Glave
Komninos
GTR-Large
0.70
GIR-XL
GTRXXXL
bi-ttase
0.68
20
30
40
50
60
70
Year (19XX)
Figure 7: This figure benchmarks HEADLINES on the MTEB clustering task. The x-axis shows the year that the sample was taken from and the y-axis gives the v-measure.
Figure 7 plots the results of this benchmarking exercise. MTEB benchmarks clustering on Arxiv, Bioarxiv, Medarxiv, Reddit, StackExchange, and Twenty Newsgroups. Texts are labeled with their classification (c.g., fields like Computer Vision for the Arxiv datasets; the subreddit for Reddit). The

---

best average v-score across these datasets, from MPNet, is 43.69. The best average v-score across decades for HEADLINES, from ST5-XXL, around 78. This difference is likely reflect, at least in part, that our cluster labels are less noisy, since texts in the same cluster summarize the same content. In contrast, titles of Reddit posts in the same subreddit may be only loosely linked to each other, and many could be within the domain of another subreddit cluster. While user happened to post in one subreddit, another user could have reasonably made the same titled post in a different subreddit. The under-identification of the clustering tasks for some texts in the MTEB datasets is suggested by the very low v-scores across state-of-the-an language models. Overall, this suggests the high quality of clusters in HEADLINES relative many web text datasets. Yet there is still ample scope for improvement to the tate-of--thearr model.
Limitations and Recommended Usage
HEADLINES contains some transcription errors. For working with historical texts, these are more a feature than a bug, as most historical texts are transcribed and also contain various OCR errors. Training a model on transcribed texts likely makes it more robust to transcription errors inference time. However, researchers requiring completely clean texts should seek another corpus.
HEADLINES contains historical language, that reflects the semantics and cultural biases of many thousands local newspaper editors. This a distinguishing feature of HEADLINES, that core to many potential applications. We do not attempt filter texts with antiquated terms or that may be considered offensive, as this would invalidate the use of the dataset for studying semantic change and historical contexts. At the same time, this makes HEADLINES less suited for tasks that require texts that fully conform to current cultural standards or semantic norms. For these reasons, we recommend against the use of HEADLINES for training generative models. Rather, with nearly 400M positive semantic similarity pairs spanning much of the 20th century, it can plausibly play an important role in facilitating the application of large language models historical texts.
10

---

Acknowledgements
Funding was provided by the Harvard Data Science Initiative, Harvard Catalyst, and Microsoft Azure compute credits. We thank Luca Amico-Worp for excellent research assistance.
References
[1] AMBA HOMBATAH, S., CHEN, T., ZHANG, M., BENDERSKY, M., AND NAJORK, M. Dy- namic language models for continuously evolving content. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (2021), pp. 2514-2524.
[2] BERT, S. stackexchange, 2021.
[3] CHEN, X., FANG, H., LIN, T.-Y., VEDANTAM, R., GUPTA, S., DOLLÁR, P., AND ZIT-
NICK, C. L.. Mierosoft coco captions: Data collection and evaluation server. arXiv preprini
arXiv:1504.00322 (2015).
[4] EHRMANN, M., HAMDI, A., PONTES, E. L.,, ROMANELLO, M., AND DOUCET, A. Named entity recognition and classification in historical documents: A survey, ACM Computing Surveys (2021).
[5] ETHAYARAJH, K. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. arXiv preprint
(2019). [6] FADER, A., ZETTLEMOYER, L., AND ETZIONI, O. Open question answering curated and extracted knowledge bases. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (2014), pp. 1156-1165.
[7] FALKNER, S., KLEIN, A., AND HUTTER, F. BOHB: Robust and efficient hyperparameter optimization at scale. In Proceedings of the 35th International Conference on Machine Learning (10-15 Jul 2018 (0, J. Dy and A. Krause, Eds., vol. 80 of Proceedings of Machine Learning Research, PMLR, pp. 1437-1446.
[8] GUARNERI, J. Newsprint Metropolis. University of Chicago Press, 2017.
[9] HADSELL, R., CHOPRA, S., AND LECUN, Y. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR' 066 (2006), vol. 2, IEEE, pp. 1735-1742.
[10] HE, K., GRIOXARI, G., DOLLÁR, P., AND GIRSHICK, R. Mask r-cnn. Proceedings of the IEEE international conference on computer vision (2017), 2961-2969.
[11] HENDERSON, M., BUDZIANOWSKI P., CASANUEVA, I., COOPE, S., GERZ, D., KUMAR, G., MRKŠIĆ, N., SPITHOURAKIS, G., SU, P.-H., VULIĆ, I., ET AL. A repository of conversational datasets. In Proceedings of the First Workshop NLP for Conversational Al (2019), pp. 1-10.
[12] IYER, S., DANDEKAR, N., AND CSERNAL, K. First quora dataset release: Question pairs, 2017.
[13] JOHNSON, J., DOUZE, M., AND JÉGOU, H. Billion-scale similarity search with gpus. IEEE Transactions Big Data 7, 3 (2019), 535-547.
[14] LIU, Y., ОтТ, M., GOYAL, N., DU, J., JOSHI, M., CHEN,, D., LEVY, O., LEWIS, M., ZETTLEMOYER, L., AND STOYANOV, V. Roberta: robustly optimized bert pretraining approach. arXiv preprint 1907.11977.11999 (2019).
[15] LOUREIRO, D., BARBIERI, F., NEVES, L., ANKE, L. E., AND CAMACIIO-COLLADOS Timelms: Diachronic language models from twitter. arXiv preprint urXiv:2202.000 03829 (2022).
[16] MANIAVACAS, E., AND FONTEYN, L. Macberth: Development and evaluation of a historically pre-trained language model for english (1450-1950) In Proceedings of the Workshop Natural
Language Processing for Digital Humanities (2021), pp. 23-36.
[17] MANIAVACAS, E., AND FONTEYN, L. Adapting vs. pre-training language models for historical languages. Journal of Data Mining & Digital Humanities, Digital humanities in languages (2022).
[18] MUENNIGHOFF, N., TAZI, N., MAGNE, L., AND REIMERS, N. Mteb: Massive text embedding benchmark. arXiv preprint at arXiv:2210.07316 (2022).

---

[19] OCKERBLOOM, J. M. Everybody's 's library questions: Newspaper copyrights, notices, and renewals, 2019.
[20] RASTAS, I., RYAN, Y. C., TITHONEN, I., QARAEI, M., REPO, L.,, BABBAR, R., MÄKELÄ, E., TOLONEN, M., AND GINTER, F. Explainable publication year prediction of eighteenth century texts with the bert model. In Proceedings of the 3rd Workshop Computational Approaches to Historical Language Change (2022), pp. 68-77.
[21] REIMERS, N., AND GUREVYCH, Sentence-bert. Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084 (2019).
[22] ROSENBERG, A., AND HIRSCHBERG, J. V-measure: A conditional entropy-based external 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CONLL) (2007), pp. 410-420.
cluster evaluation measure. In Proceedings of the 2007
[23] SILCOCK, E., D' D'AMICO-WWon L., YANG, AND DELL, M. Noise-robust de-duplication at scale. International Conference on Learning Representations (2023).
[24] SONG, K., TAN, X., QIN, T., LU, J., AND LIU, T.-Y. Mpnet: Masked and permuted pre- training for language understanding. Advances in Neural Information Processing Systems 33 (2020), 16857-16867.
[25] WANG, T., AND ISOLA, P. Understanding contrastive representation learning through alignment and uniformity the hypersphere. International Conference Machine Learning 119 (2020), 9929-9939.
[26] YOUNG, P., LAI, A., HODOSH, M., AND HOCKENMAIER, From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2 (2014), 67-78.
12

---

Appendices
A-1 Methods to Associate Articles
Figure 6 in the main text illustrates the full article association procedure.
First, we used a rule-based algorithm using associate article bounding boxes that are under the same headline, as these are part of the same article with extremely high probability. Algorithm I gives pseudocode for this method. We set the parameters as Ps 100, Pr 20, PB = 50.
For training data, where we want article pairs that are not only part of the same article. but also where they appear in the given order, we further narrow down the pairs. Specifically, we use only those pairs which are horizontally next to each other, and which have no other bounding boxes below them, as for these pairs, we can guarantee that the pair of bounding follow directly after one another (whereas for other article bounding boxes that share headline, there may be a third bounding box in between). Algorithm 2 shows pseudocode for this procedure, and we used Pe 5, and it is further illustrated in panel A of figure 6 in the main text.
For hard negatives, we used article boxes under the same headline in reverse reading order (right to left). For standard negatives, we took pairs of articles on the same page, where B was above and to the left of A, as articles do not read from right left. One twelfth of our training data were positive pairs, another twelfth were hard negative pairs and the remainder were standard negative pairs. This outperformed more balanced training sample.
We use this dataset to finetune a cross-encoder using a RoBERTa base model 14] We used a tenth of our training data
Bayesian search algorithm 7 to find optimal hyperparameters (limited compute prevented us from running this search with the full dataset), which led to a learning rate of 1. 1.7e-5, with a batch size of 64 and 29.57% warm up. We trained for 26 epochs with an Adam W optimizer, and optimize a binary cross-entropy loss.
We evaluate these methods and 1955. These scans were labeled by a highly-trained trained undergraduate research assistant. Summary statistics of this dataset are given in table A-l and evaluation results are given in the main text.
hand-labeled dataset of 214 scans, randomly selected from 1968
Scan count
Article bounding hoxes
Headline hounding boxes
Article-article associations
214
3,803
2,805
1,851
Table A-I: Descriptive statisties of article association training data.
A-2 Methods to Detect Reproduced Content
To detect reproduced content, use the contrastively trained bi-encoder model developed by (3). which is trained to learn similar representations for reproduced articles and dissimilar representations for non-reproduced articles. This model is based an S-BERT MPNET model 21 24 and is fine- tuned on a labelled dataset of articles from the same underlying wire source, using S S-BERT's online contrastive loss [9] implementation, with 0.2 margin and cosine similarity metric. The learning rate is 2e-5 with 100% warm up and a batch size of 32. It uses an AdamW optimizer, and the model is trained for 16 epochs. This bi-encoder is trained and evaluated hand-labeled dataset, which is detailed in A-2 The results of this evaluation are given in the main
the distance
text.
To create clusters from the bi-encoder embeddings, we use highly scalable single-linkage clustering, with cosine similarity threshold of 0.94. We build graph using articles nodes, and add edges if the cosine similarity is above this threshold. As edge weights we use the negative exponential of the difference in dates (in days) between the two articles. We then apply Leiden community detection to the graph to control false positive edges that can otherwise merge disparate groups of articles.
We further remove clusters that have over 50 articles and contain articles with greater than five different dates. We also remove clusters that contain over 50 articles, when the number of articles is more than double the number of unique newspapers from which these articles are sourced. This
13

---

Algorithm Rule-based association of article bounding boxes
INPUT:
bi,..., bn E B: set of bounding boxes that appear on the same scan, with their coordinates, denoted le right(b;), top(b;), bottom((,)) and type (headline, article, byline etc.), denoted type(bb)).
PARAMETERS:
W: width of scan
H: height of sean
Ps: fraction of width, for creating side margin
Pr: fraction of height, for creating top margin
Pn: fraction of height, for creating bottom margin
OUTPUT: Article ArticlePairs = (b,, bj) E Blb;,b, by predicted to part of same full article and (bpe (ype(b;)) article]
1: Initialise: Ms = W/Ps, the side margin, MB = H/PB. the bottom margin, Mf =
H/Pr. the top margin, MatchedH eadlines O. Araccle { Article.AticleePairs
2: for all by in B where type(bo) is article do
3: Create Bu C where:
All bounding boxes of type byline removed
by is removed
All bounding boxes are removed that do share least Ms of the horizontal axis
All bounding boxes are removed whose bottom is more than Mr below the top of bo
All bounding boxes are removed whose bottom is more than Mr above the top of by
4:
if
Bo is not empty then
5:
Let 61 be the element of Bo that has the lowest bottom coordinate if type(b1) ) is headline then M aatchedHeadlingg H Article Pairs Headline Pairs U {(bo, b1))
6:
7:
MatchedH U {b1}
8:
end 10: end if
9:
11: end for
12: for all by in MatchedHl eadlines do
Let H1 C Headline ArticlePairr
pairs that contain that headline, by
14: if H1 has at least two elements then
15:
Let A be all the bounding boxes of type article from the pairs Hi Let C' be all combinations of 2 elements of A Article.rticce
16:
17:
= Article U
18: end if
19:
end for
Algorithm 2 Selection of ordered article pairs
INPUT: Articceerr B, from algorithm
PARAMETERS:
Pe: fraction of column width, for creating margin
OUTPUT: OrderedPairs C Article Article:
Initialise: OrderedPairs = O
2: for in Article
do 3: Let pr be the element of p with the furthest left coordinate
4: Let pr be the other element
5: if le fef((p)) not further to the right of right(pr) than widthh(:)/Fec then if there no other bounding boxes below pl then
6:
7:
OrderedPairs = OrderedPairs U {p}
8: end if
9: end
10: end for
removes clusters of content that are correctly clustered in the sense of being based on the same underlying source, but are not useful for the HEADLINES dataset. For example, advertisement
14

---

Positives Pairs
Negative Pairs
Reproduced Articles
Singleton Articles
Total Articles
Training Data
Training Validation
36,291
37,637 3,246
891
7,728
3,042
20
283
Full Day Evaluation
Validation
28,547 54,996
12,409,031
447
2,162
4,988
Test
100,914.159
1,236
8,046
14,211
Full Dataset
122,876
113,364.073
2,594
10,208
27,210
Table A-2: Summary statistics of training and evaluation data for detecting duplicate content.
(misclassified an article due an article-like appearance) might be repeated by the same newspaper on multiple different dates and would be removed by these rules, or weather forecasts can be very near duplicates across space and time, forming large clusters.
A-3 Summary of Copyright Law for Works Published in the United States
Date of Publication
Conditions
Copyright Term
Public Domain Anytime
Works prepared by an officer/employee of the U.S. Govermeent their official duties
Nosc
Before 1928
None
None. Copyright expired.
through 1978 March 1989
a copyright notice
comply required formalities
Published without notice and without subsequent registration within years
None. Fallure comply with required formalities
1928 through 1963
Published with motice but copyright was not renewed
Nome. Copyright expired
Copyrighted
Published without notice, but with sabsequent registration within years
700 (95) years
author (corporate author)
1928 through 1963
Published with notice and the copyright was renewed
95 years after publication
1964 through 1977
Published with notice
95 years after publication
March 1989
Crented after 1977 and published with notice
70 years the death author (corporate author) years after creation. earlier
March
before 1978 published with notice the specified period
greater specified provious calry or 31 December 2047
March through
Created after 1977
(95) years or years after creation, if earlier
(corporato author)
March through
Created before 1978 and first published this period
The the specified
entry
or 31 December 2047
None
(95) years
(corporate author) or 120 years after creation, earlier
Table A-3: This table summarizes U.S. copyright law, based on similar table produced by the Cornell libraries. For concision, focus on works initially published in the United States. A variety of other cases are also covered at uthttps::/guudees I wwwnell.eee..oooooo www.eeel..e./ooprriiie........e.
Library.cornell.cce.
15


Reviews:
[{"review_id": "VW0dgv3DOk", "reviewer": "Reviewer_ZfNZ", "summary": "", "questions": "", "limitations": "no", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "Data Source and Coverage: Utilizing newly digitized articles from off-copyright, local U.S. newspapers spanning nearly 70 years (1920-1989) is a significant strength. The dataset covers a large time span, providing researchers with the opportunity to study semantic change over time and understand historical linguistic patterns.\n\nData Quality: The authors have taken care to address potential noise and abridgement in the dataset due to OCR and article association errors. The rule-based approach with high precision and the use of RoBERTa bi-encoder for detecting reproduced content demonstrate a thoughtful and rigorous data processing methodology.\n\nPotential for Research: The paper discusses how the dataset can be used for a variety of tasks, including contrastively trained semantic similarity models, geographic analysis based on authors' locations, and studying semantic change across space and time. This highlights the dataset's potential for diverse and meaningful research applications.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "NvzUUTl6JY", "reviewer": "Reviewer_boGz", "summary": "", "questions": "", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Overall, an interesting dataset that would be very useful for the field. Decent coverage of related work and fairly clear writing.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "B7pUqV6i7y", "reviewer": "Reviewer_d3Qx", "summary": "", "questions": "", "limitations": "There is no discussion about the limitations of the paper.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "This size of the dataset is quite large compared to the existing datasets, and it is prepared on historical English.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "CqM3Lt16Mb", "reviewer": "Reviewer_eGCu", "summary": "", "questions": "", "limitations": "Authors have not discussed limitations of the methods used. \nWould also recommend discussing the copyright issue, even though the newspapers are off-copyright. The point being, that curating articles at scale this way needs to be accompanied by a stronger statement about intended usage. For example, if models are to be trained based on this, does the copyright status still hold?", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Large dataset with newspaper articles and headlines across a wide variety of topics.\nIntroduce a great strategy towards identifying news article bodies at scale.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "u7bIIPkma1", "reviewer": "Reviewer_oH6a", "summary": "", "questions": "", "limitations": "The authors have an have not addressed the limitations clearly. There is no defined section as \"Limitations\" . However, the authors recognizes that the OCR method can result into errors that cause grammatical and spelling issues which may affect the context of the article.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1- The paper is well-written and easy to follow.  \n2 - The dataset is innovative and large. \n3 - The dataset is accessible and publicly available", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper presents HEADLINES a very large diachronic dataset including nearly 400,000 semantic similarity documents pairs. I don’t think the limitations identified by the reviewers would justify the rejection of this paper, which can be an interesting contribution to NeurIPS and to the NLP community. The authors provided a thorough reply to the reviewers and will surely be able to improve the final version of the paper.
Decision: Accept (Poster)