Chronicling Germany: An Annotated Historical Newspaper Dataset

Authors: Christian Schultze, Niklas Kerkfeld, Kara Kuebart, Princilia Weber, Moritz Wolter, Felix Selgert
Primary area: datasets and benchmarks
Keywords: historic newspaper processing, digital history, computer vision

Abstract:
The correct detection of dense article layout and the recognition of characters in historical newspaper pages remains a challenging requirement for Natural Language Processing (NLP) and machine learning applications on historical newspapers in the field of digital history. Digital newspaper portals for historic Germany typically provide Optical Character Recognition (OCR) text, albeit of varying quality. Unfortunately, layout information is often missing, limiting this rich source’s scope. Our dataset is designed to enable the training of layout and OCR modells for historic German-language newspapers. The Chronicling Germany dataset contains 693 annotated historical newspaper pages from the time period between 1852 and 1924. The paper presents a processing pipeline and establishes baseline results on in- and out-of-domain test data using this pipeline. Both our dataset and the corresponding baseline code are freely available online. This work creates a starting point for future research in the field of digital history and historic German language newspaper processing. Furthermore, it provides the opportunity to study a low-resource task in computer vision.

Decision: reject
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730500303373

thread_type:
official_review

<thread_object>
creator: reviewer_uuca

<content>
summary:
This paper presents an annotated historical newspaper dataset that was written in German between the 19 and 20th centuries.

The dataset contains 693 pages where each page is manually labelled with both transcriptions of the text and layout elements, such as background, table, etc. 

The authors also provided baseline methods assess the layout segmentation, OCR recognition accuracies as well as the performance of the full pipeline.

The authors hope that researchers could build upon the Chronicling Germany dataset to improve historical newspaper processing methods.

strengths:
* Chronicling Germany dataset introduces OCR as well as polygon layout analysis problem for historical newspaper scans
* Compared to Dell 2024, Chronicling Germany appears to have much more complex layouts. This could introduce a new line of problems in historic document analysis 
* The paper is well written and easy to follow

weaknesses:
* The authors claim that modern human readers will struggle reading the contents of Chronicling Germany however, only provided examples of font differences. As layout analysis is also the primary focus of this dataset, it will be nice if the authors could also focus more on the difficulties for modern readers to understand the layout differences.
* The authors did not provide information regarding the image sizes or DPI. This could help researchers evaluate the usefulness of the dataset.
* The benchmark methods doesn’t seem very well justified. Have you explored how existing pipelines trained on modern newspaper perform on your dataset? 
* There is no information about the legibility of the data

questions:
* As the authors mentioned that comprehending the pages is not trivial for modern German readers, why is the reading order within scope of this dataset?
* Follow up: how was the reading order automatically assigned and evaluated to have “satisfactory results”?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730531111136

thread_type:
official_review

<thread_object>
creator: reviewer_cclu

<content>
summary:
This paper addresses the challenges of article layout detection and text recognition in historical German newspaper pages. The authors present the "Chronicling Germany" dataset, containing 693 annotated pages from 1852 to 1924, and establish a baseline pipeline for layout detection and OCR tasks. The authors aim to address the challenges of article layout detection and text recognition in historical German newspaper pages, which are essential for NLP and machine learning in digital history. They validate the model's performance on an out-of-distribution set of 112 pages from 1785-1866. Both the dataset and the baseline code are publicly available.

strengths:
The paper makes a significant contribution by providing a new, large, and annotated dataset of 693 high-resolution historical newspaper pages in German Fraktur, addressing a crucial gap in resources for processing German-language historical documents.

weaknesses:
The paper does not cite significant existing projects that contributed to newspaper recognition: 
- Newseye  https://www.newseye.eu/open-science/research-datasets/
- Impresso  https://impresso-project.ch/outputs/datasets/

While the OCR-D project, which develops advanced tools for processing German historical documents https://ocr-d.de/en/, is mentioned in the annex, it should be highlighted in the main text 

The study does not leverage high-performance OCR systems such as Pero-OCR https://pero-ocr.fit.vutbr.cz/, which could have enhanced baseline OCR results. This choice limits the impact of the paper’s findings, as more modern and effective systems are overlooked.

The paper’s reliance on an OCR system that requires baseline detection is questionable. These systems are typically designed for non-horizontal or curved text, which is rare in newspaper layouts. A retrained OCR model, especially one leveraging pre-existing models on platforms like HuggingFace https://huggingface.co/Teklia/pylaia-newseye-austrian , would likely have been more suitable.


The use of a relatively older UNet model for layout detection is debatable, given that more recent and effective models like YOLO-based architectures outperform UNet in similar tasks. Table 3's comparison, which references Dell et al., highlights these limitations, suggesting the paper's choice of model may not be optimal. As a result, the evaluation of layout analysis show poor generalization capabilities, as reflected in Table 3. 

The full evaluation of the pipeline is limited; the paper only reports a Character Error Rate of 6% without providing further details. Furthermore, article separation is not assessed, making the results appear preliminary and potentially incomplete. Comprehensive evaluation metrics are needed for a stronger validation of the approach.


The legend for Table 3 lacks clarity, especially regarding what is meant by "F1 Score in distribution"

questions:
No specific question.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642230609

thread_type:
official_review

<thread_object>
creator: reviewer_fkw8

<content>
summary:
This paper presents a new dataset called "Chronicling Germany", consisting of 693 annotated historical German newspaper pages from 1852 to 1924. The dataset includes layout annotations and ground truth text transcriptions. The authors establish baseline results for layout detection, text line recognition, and OCR tasks using the dataset. They also test generalization on an out-of-distribution test set.

strengths:
This paper presents a new historical German newspaper dataset, providing layout information and text line annotations, and also offers some baselines for layout detection and OCR tasks.

weaknesses:
1. The novelty and contribution of this paper is limited. The main contribution of this paper is the historical German newspaper dataset itself. However, compared to existing datasets [1][2][3], this dataset does not have significant uniqueness in dataset size and diversity.
[1] Christian Clausner, Christos Papadopoulos, Stefan Pletschacher, and Apostolos Antonacopoulos.
The enp image and ground truth dataset of historical newspapers. In 2015 13th International
Conference on Document Analysis and Recognition (ICDAR), pp. 931–935. IEEE, 2015.
[2] UB-Mannheim. Ground truth for neue zürcher zeitung black letter period. https://github.
com/UB-Mannheim/NZZ-black-letter-ground-truth, 2023a.
[3] UB-Mannheim. reichsanzeiger gt. https://github.com/UB-Mannheim/reichsanzeiger-gt/, 2023b.

2. Lack of quantitative comparison with existing datasets, which fails to show the superiority of the Chronicling Germany dataset.

3. Lack of detailed assessment of the annotation quality, such as the review of annotation consistency.

4. Section 5 does not provide detailed results of testing on OOD data, which is insufficient to reflect the generalizability of the pipeline.

5. The paper incorrectly used the ICLR 2024 template instead of the ICLR 2025 one.

questions:
1. In comparison to other German historical newspaper datasets, how does the quantitative aspects of your dataset fare in terms of size, annotation quality, and diversity?

2. Have the authros explored employing more advanced baseline methods for your tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661741633

thread_type:
official_review

<thread_object>
creator: reviewer_kwsv

<content>
summary:
This paper presents the Chronicling Germany dataset. It consists in an annotated dataset of historical newspapers in German language. The dataset has been constructed after 1,500 hours of human annotation of the layout components. It consists of 693 pages. The dataset also includes 1,900 individually annotated advertisements. According to the authors, it the largest fully annotated collection of historic German newspaper pages. The motivation of creating this dataset is that the problem of historical newspapers understanding lacks of enough data, in particular in some languages like German. The newspapers of the dataset have some singular features. Among others, the particular use of the Fraktur font, the presence of some characters and combinations, and the dense layout.
In addition to the dataset, the paper presents a  processing pipeline for layout analysis and text recognition, and experimentally evaluates the pipeline on in- and out-of-domain test data.

strengths:
A useful dataset for the community of digital humanities, covering a gap of low resources data. The dataset has been rigorously constructed, with layout annotations. In addition to the language, the documents in the dataset have some particularities that make it interesting to tackle language-independent document layout analysis and recognition problems.

The complementary pipeline processing that is presented is a good way to illustrate the value of the dataset, comparing the processing with other datasets.

weaknesses:
Annotated data is important for the scientific community. But presenting a new dataset in a conference requires a solid justification on how useful is this dataset in contributing in the progress of the state of the art in the main problems addressed by the community. In this case, an annotated historical newspaper dataset is not in the mainstream of the representation learning community nor the scope of the conference. Its interest is addressed to a marginal audience of the representation learning community.

questions:
I believe that the contribution of this dataset is interesting, however as indicated above, it is addressed to a narrow audience, considering the scope of the conference. I am open to be persuaded on the relevance of this contribution in the context of the representation learning area. Beyond the interest in the problem of historical document layout analysis and recognition, from a wider perspective, authors should identify other points that make this dataset interesting for a larger audience.

Comparison with the pipeline developed by Dell et al. (2024) analyzed in table 3. It is not clear to me if there is a crossfold validation, i.e. if the  proposed pipeline is tested with the American Stories dataset, as well as the Dell et al. pipeline is tested on the proposed German dataset. It would be good to have different datasets and different methods for the comparison.

======================== AFTER THE REBUTTAL

After interacting with the authors, and looking at the other reviewers' revisions, I will keep the score of the first review. I thank the authors for their clarifying responses, and the effort to consider the comments. Authors provided interesting new material, and I encourage them to include in a potential revised version of the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732126889859

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall rebuttal

comment:
Dear editorial Team,

Thank you for the constructive comments and insightful feedback. We were happy to read that reviewers found the paper well written (UUca), the dataset new (FKw8) as well as rigorously constructed (KWSV), presenting an interesting to tackle document-layout analysis problem (KWSV), while closing a crucial gap in resources (CCLu).
Furthermore, UUca noticed that this dataset contained pages with a more complex layout than those in the dataset by Dell et al. (2024), which opens up new research avenues to the community.

Naturally our reviewers also voiced questions and concerns, we have done our best to address these individually and summarize the most important points here.

### Chronicling Germany Data-Set size in comparison to related work.

Reviewers asked about the overall motivation for the collection of the data in the first place, as well as its relation to prior art. To address both, we have added a new table (Table 2) to the paper, which we repeat here:


|Dataset                                                                 | Pages                                   |
|----------------------------------------------------------------|---------------------------------------|
|Chronicling Germany (ours)                                              | **693**                                     |
|Europeana  [1]                                                          | 528                                     |
|News Eye Finnish [7]                                                    | 200                                     |
|Deutscher Reichsanzeiger und Preußischer Staatsanzeiger  [2]            | 197                                     |
|News Eye French [8]                                                     | 183                                     |
|Neue Züricher Zeitung (impresso) [3, 4, 6]                              | 167                                     |
|News Eye Austrian [9]                                                   | 158                                     |
|News Eye Competition [5]                                                | 100  (50 simple track, 50 complex track)|



[1] http://dataset.primaresearch.org/www/assets/papers/ICDAR2015_Clausner_ENPDataset.pdf

[2] https://github.com/UB-Mannheim/reichsanzeiger-gt/

[3] https://zenodo.org/records/3333627

[4] https://github.com/UB-Mannheim/NZZ-black-letter-ground-truth

[5] https://zenodo.org/records/4943582

[6] Ströbel, Phillip and Clematide, Simon Improving OCR of Black Letter in Historical Newspapers: The Unreasonable Effectiveness of HTR Models on Low-Resolution Images, https://www.zora.uzh.ch/id/eprint/177164/1/
Improving_OCR_of_Black_Letter_in_Historical_Newspapers_The_Unreasonable_Effecti.pdf

[7] NewsEye / READ AS training dataset from Finnish Newspapers (19th C.) https://zenodo.org/records/5654858

[8] NewsEye / READ AS training dataset from French Newspapers (19th, early 20th C.) https://zenodo.org/records/5654841

[9] NewsEye / READ AS training dataset from Austrian Newspapers (19th, early 20th C.) https://zenodo.org/records/5654907
</content>
</thread_object>

</thread>
<thread>
creation date:
1750021484325

thread_type:
public_comment

<thread_object>
creator: ~moritz_wolter1

<content>
title:
Paper available

comment:
We want to thank everyone for their feedback. The final version of this paper is now available at https://openreview.net/pdf?id=YHDfqvtye9 .
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper discusses the challenges of detecting layouts and recognizing characters in historical German newspapers for in terms of Natural Language Processing and Machine Learning. The authors propose and discuss the *Chronicling Germany Dataset*, which consists of 693 annotated German newspaper pages from 1852 to 1924. The paper additionally discusses baselines to address the challenges of historical document understanding by training layout and OCR models, proposing a processing pipeline, and establishing baseline results on both in- and out-of-domain test data.

Reviewers are almost unanimous in recognizing the importance of annotated datasets of the type proposed for historical document analysis, and also recognize the gap in the current offering of German-language resources of this type. However, the papers (as reviewers note) offers very little in terms of scientific contributions to the broader machine and representation learning communities. While the proposed pipeline illustrates the utility of the proposed dataset, it is based on consolidates, well-known models and techniques and thus also does not represent a novel contribution. Finally, the proposed dataset -- while it does address a lack of German-language resources -- is comparable in terms of size and variety with existing historical document resources as pointed out by multiple reviewers.

The consensus that emerged from the discussion phase is that this paper is not well-aligned with the ICLR community due to the lack of significant scientific contributions related to machine and representation learning and thus the decision is to reject. The authors are encouraged to look for a better-aligned venue, for example the NeurIPS dataset track.