SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The neural bi-encoder methods achieve a high adjusted rand index (ARI) of 91.5, compared to 73.7 for an optimal local sensitive hashing specification, chosen on the validation set. This shows that our neural methods substantially outperform commonly used sparse methods for detecting reproduced content. The neural bi-encoder is slightly outperformed by adding a re-ranking step that uses a neural cross-encoder on the best bi-encoder matches (ARI of 93.7). We do not implement this method because the cross-encoder doesn't scale well. In contrast, the bi-encoder pipeline can be scaled to 10 million articles on a single GPU in a matter of hours, using a FAISS [13] backend.
Neural	Non-Neural
Most scalable	Bi-encoder (91.5)	LSH (73.7)
Less scalable	Re-ranking (93.7)	N-gram overlap (75.0)
Table 5: The numbers in parentheses are the Adjusted Rand Index for four different models - a bi-encoder, a "re-ranking" strategy that combines a bi- and cross-encoder, locally sensitive hashing (LSH), and  $N$ -gram overlap. Hyperparameters were chosen on the NEWS-COPY validation set, and all models were evaluated on the NEWS-COPY test set.
An error analysis is provided in [23]. Errors typically consist of articles about the same story from different wire services (e.g. the Associated Press and the United Press) or updates to a story as new events unfolded. Both types of errors will plausibly still lead to informative semantic similarity pairs.
5 Benchmarking
We benchmark HEADLINES using a variety of different language models and the MTEB clustering task. This task embeds texts using different base language models and then uses  $k$  - the number of clusters in the ground truth data - for k-means clustering. Following MTEB, we score the model using the v-measure [22]. We should note that real-world problems are often framed as clustering tasks - rather than as classification tasks - because  $k$  is unknown. By using  $k$  from the ground truth, it makes the task easier. Nevertheless, we examine this task to allow for comparison with the rest of the literature.
Figure 7: This figure benchmarks HEADLINES on the MTEB clustering task. The x-axis shows the year that the sample was taken from and the y-axis gives the v-measure.
Figure 7 plots the results of this benchmarking exercise. MTEB benchmarks clustering on Arxiv, Bioarxiv, Medarxiv, Reddit,StackExchange,and Twenty Newsgroups. Texts are labeled with their classification (e.g., fields like ComputerVision for the Arxiv datasets; the subreddit for Reddit). The
best average v-score across these datasets, from MPNet, is 43.69. The best average v-score across decades for HEADLINES, from ST5-XXL, is around 78. This difference is likely to reflect, at least in part, that our cluster labels are less noisy, since texts in the same cluster summarize the same content. In contrast, titles of Reddit posts in the same subreddit may be only loosely linked to each other, and many could be within the domain of another subreddit cluster. While a user happened to post in one subreddit, another user could have reasonably made the same titled post in a different subreddit. The under-identification of the clustering tasks for some texts in the MTEB datasets is suggested by the very low v-scores across state-of-the-art language models. Overall, this suggests the high quality of clusters in HEADLINES relative to many web text datasets. Yet there is still ample scope for improvement to the state-of-the-art model.
6 Limitations and Recommended Usage
HEADLINES contains some transcription errors. For working with historical texts, these are more a feature than a bug, as most historical texts are transcribed and also contain various OCR errors. Training a model on transcribed texts likely makes it more robust to transcription errors at inference time. However, researchers requiring completely clean texts should seek another corpus.
HEADLINES contains historical language, that reflects the semantics and cultural biases of many thousands of local newspaper editors. This is a distinguishing feature of HEADLINES, that is core to many potential applications. We do not attempt to filter texts with antiquated terms or that may be considered offensive, as this would invalidate the use of the dataset for studying semantic change and historical contexts. At the same time, this makes HEADLINES less suited for tasks that require texts that fully conform to current cultural standards or semantic norms. For these reasons, we recommend against the use of HEADLINES for training generative models. Rather, with nearly 400M positive semantic similarity pairs spanning much of the 20th century, it can plausibly play an important role in facilitating the application of large language models to historical texts.
Funding was provided by the Harvard Data Science Initiative, Harvard Catalyst, and Microsoft Azure compute credits. We thank Luca D'Amico-Wong for excellent research assistance.
[19] OCKERBLOOM, J. M. Everybody's library questions: Newspaper copyrights, notices, and renewals, 2019.
[20] RASTAS, I., RYAN, Y. C., TIIHONEN, I., QARAEI, M., REPO, L., BABBAR, R., MÄKELÄ, E., TOLONEN, M., AND GINTER, F. Explainable publication year prediction of eighteenth century texts with the bert model. In Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change (2022), pp. 68-77.
[21] REIMERS, N., AND GUREVYCH, I. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019).
[22] ROSENBERG, A., AND HIRsCHBERG, J. V-measure: A conditional entropy-based external cluster evaluation measure. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL) (2007), pp. 410-420.
[23] SILCOCK, E., D'AMICO-WONG, L., YANG, J., AND DELL, M. Noise-robust de-duplication at scale. International Conference on Learning Representations (2023).
[24] SONG, K., TAN, X., QIN, T., LU, J., AND LIU, T.-Y. Mpnet: Masked and permuted pretraining for language understanding. Advances in Neural Information Processing Systems 33 (2020), 16857-16867.
[25] WANG, T., AND ISOLA, P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. International Conference on Machine Learning 119 (2020), 9929-9939.
[26] YOUNG, P., LAI, A., HODOSH, M., AND HOCKENMAIER, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2 (2014), 67-78.
A-1 Methods to Associate Articles
Figure 6 in the main text illustrates the full article association procedure.
First, we used a rule-based algorithm using associate article bounding boxes that are under the same headline, as these are part of the same article with extremely high probability. Algorithm 1 gives pseudocode for this method. We set the parameters as  $P_{S} = 100$ ,  $P_{T} = 20$ ,  $P_{B} = 50$ .
For training data, where we want article pairs that are not only part of the same article, but also where they appear in the given order, we further narrow down the pairs. Specifically, we use only those pairs which are horizontally next to each other, and which have no other bounding boxes below them, as for these pairs, we can guarantee that the pair of bounding follows directly after one another (whereas for other article bounding boxes that share a headline, there may be a third bounding box in between). Algorithm 2 shows pseudocode for this procedure, and we used  $P_{C} = 5$ , and it is further illustrated in panel A of figure 6 in the main text.
For hard negatives, we used article boxes under the same headline in reverse reading order (right to left). For standard negatives, we took pairs of articles on the same page, where B was above and to the left of A, as articles do not read from right to left. One twelfth of our training data were positive pairs, another twelfth were hard negative pairs and the remainder were standard negative pairs. This outperformed a more balanced training sample.
We use this dataset to finetune a cross-encoder using a RoBERTa base model [14]. We used a Bayesian search algorithm [7] to find optimal hyperparameters on one tenth of our training data (limited compute prevented us from running this search with the full dataset), which led to a learning rate of 1.7e-5, with a batch size of 64 and  $29.57\%$  warm up. We trained for 26 epochs with an AdamW optimizer, and optimize a binary cross-entropy loss.
We evaluate these methods on a hand-labeled dataset of 214 scans, randomly selected from 1968 and 1955. These scans were labeled by a highly-trained undergraduate research assistant. Summary statistics of this dataset are given in table A-1 and evaluation results are given in the main text.
Scan count	Article bounding boxes	Headline bounding boxes	Article-article associations
214	3,803	2,805	1,851
Table A-1: Descriptive statistics of article association training data.
A-2 Methods to Detect Reproduced Content
To detect reproduced content, we use the contrastively trained bi-encoder model developed by [23], which is trained to learn similar representations for reproduced articles and dissimilar representations for non-reproduced articles. This model is based on an S-BERT MPNET model [21, 24] and is fine-tuned on a hand-labelled dataset of articles from the same underlying wire source, using S-BERT's online contrastive loss [9] implementation, with a 0.2 margin and cosine similarity as the distance metric. The learning rate is 2e-5 with  $100\%$  warm up and a batch size of 32. It uses an AdamW optimizer, and the model is trained for 16 epochs. This bi-encoder is trained and evaluated on a hand-labeled dataset, which is detailed in A-2. The results of this evaluation are given in the main text.
To create clusters from the bi-encoder embeddings, we use highly scalable single-linkage clustering, with a cosine similarity threshold of 0.94. We build a graph using articles as nodes, and add edges if the cosine similarity is above this threshold. As edge weights we use the negative exponential of the difference in dates (in days) between the two articles. We then apply Leiden community detection to the graph to control false positive edges that can otherwise merge disparate groups of articles.
We further remove clusters that have over 50 articles and contain articles with greater than five different dates. We also remove clusters that contain over 50 articles, when the number of articles is more than double the number of unique newspapers from which these articles are sourced. This
Algorithm 1 Rule-based association of article bounding boxes
INPUT:  $b_{1},\ldots ,b_{n}\in B$  : set of bounding boxes that appear on the same scan, with their coordinates, denoted left  $(b_{i})$  right  $(b_{i})$  top  $(b_{i})$  bottom  $(b_{i})$  , and type (headline, article, byline etc.), denoted type  $(b_{i})$    
PARAMETERS: W: width of scan H: height of scan  $P_S$  : fraction of width, for creating side margin  $P_T$  : fraction of height, for creating top margin  $P_B$  : fraction of height, for creating bottom margin   
OUTPUT: ArticleArticlePairs =  $\{(b_i,b_j)\in B\times B|b_i,b_j$  predicted to be part of the same full article and type  $(b_{i}) =$  type  $(b_{j}) =$  article} 1: Initialise:  $M_S = W / P_S$  , the side margin,  $M_B = H / P_B$  , the bottom margin,  $M_T =$ $H / P_T$  , the top margin, MatchedHeadlines  $= \{\}$  , HeadlineArticlePairs  $= \{\}$  ArticleArticlePairs  $= \{\}$  2: for all  $b_{0}$  in B where type(b0) is article do 3: Create  $B_0\subset B$  where: - All bounding boxes of type byline are removed -  $b_{0}$  is removed - All bounding boxes are removed that do not share at least  $M_S$  of the horizontal axis - All bounding boxes are removed whose bottom is more than  $M_B$  below the top of  $b_{0}$  - All bounding boxes are removed whose bottom is more than  $M_T$  above the top of  $b_{0}$  4: if  $B_0$  is not empty then 5: Let  $b_{1}$  be the element of  $B_{0}$  that has the lowest bottom coordinate 6: if type  $(b_{1})$  is headline then 7: MatchedHeadlines  $=$  MatchedHeadlines  $\cup \{b_1\}$  8: HeadlineArticlePairs  $=$  HeadlineArticlePairs  $\cup \{(b_0,b_1)\}$  9: end if 10: end if 11: end for 12: for all  $b_{h}$  in MatchedHeadlines do 13: Let  $H_{1}\subset$  HeadlineArticlePairs be all pairs that contain that headline,  $b_{h}$  14: if  $H_{1}$  has at least two elements then 15: Let A be all the bounding boxes of type article from the pairs in  $H_{1}$  16: Let C be all combinations of 2 elements of A 17: ArticleArticlePairs  $=$  ArticleArticlePairs  $\cup C$  18: end if 19: end for
Algorithm 2 Selection of ordered article pairs
INPUT: ArticleArticlePairs, B, from algorithm 1.  
PARAMETERS:  
 $P_C$ : fraction of column width, for creating margin  
OUTPUT: OrderedPairs  $\subset$  ArticleArticlePairs  
1: Initialise: OrderedPairs = {}  
2: for  $p$  in ArticleArticlePairs do  
3: Let  $p_l$  be the element of  $p$  with the furthest left coordinate  
4: Let  $p_r$  be the other element  
5: if  $left(p_r)$  is not further to the right of  $right(p_l)$  than  $width(p_l) / F_C$  then  
6: if there are no other bounding boxes below  $p_l$  then  
7: OrderedPairs = OrderedPairs  $\cup$  {p}  
8: end if  
9: end if  
10: end for
removes clusters of content that are correctly clustered in the sense of being based on the same underlying source, but are not useful for the HEADLINES dataset. For example, an advertisement
Positives Pairs	Negative Pairs	Reproduced Articles	Singleton Articles	Total Articles
Training Data
Training	36,291	37,637	891	-	7,728
Validation	3,042	3,246	20	-	283
Full Day Evaluation
Validation	28,547	12,409,031	447	2,162	4,988
Test	54,996	100,914,159	1,236	8,046	14,211
Full Dataset	122,876	113,364,073	2,594	10,208	27,210
Table A-2: Summary statistics of training and evaluation data for detecting duplicate content.
(misclassified as an article due to an article-like appearance) might be repeated by the same newspaper on multiple different dates and would be removed by these rules, or weather forecasts can be very near duplicates across space and time, forming large clusters.
A-3 A Summary of Copyright Law for Works Published in the United States
Date of Publication	Conditions	Copyright Term
Public Domain
Anytime	Works prepared by an officer/employee of the U.S. Government as part of their official duties	None
Before 1928	None	None. Copyright expired.
1928 through 1977	Published without a copyright notice	None. Failure to comply with required formalities
1978 to 1 March 1989	Published without notice and without subsequent registration within 5 years	None. Failure to comply with required formalities
1928 through 1963	Published with notice but copyright was not renewed	None. Copyright expired
Copyrighted
1978 to 1 March 1989	Published without notice, but with subsequent registration within 5 years	70 (95) years after the death of author (corporate author)
1928 through 1963	Published with notice and the copyright was renewed	95 years after publication
1964 through 1977	Published with notice	95 years after publication
1978 to 1 March 1989	Created after 1977 and published with notice	70 (95) years after the death of author (corporate author) or 120 years after creation, if earlier
1978 to 1 March 1989	Created before 1978 and first published with notice in the specified period	The greater of the term specified in the previous entry or 31 December 2047
From 1 March 1989 through 2002	Created after 1977	70 (95) years after the death of author (corporate author) or 120 years after creation, if earlier
From 1 March 1989 through 2002	Created before 1978 and first published in this period	The greater of the term specified in the previous entry or 31 December 2047
After 2002	None	70 (95) years after the death of author (corporate author) or 120 years after creation, if earlier
Table A-3: This table summarizes U.S. copyright law, based on a similar table produced by the Cornell libraries. For concision, we focus on works initially published in the United States. A variety of other cases are also covered at https://guides.library.cornell.edu/copyright.