# ARletta. Open-Source Handwritten Text Recognition Models for Historic Dutch

Lith Lefranc, Ilja van Damme, Thibault Clérice, Mike Kestemont

► To cite this version:

Lith Lefranc, Ilja van Damme, Thibault Clérice, Mike Kestemont. ARletta. Open-Source Handwritten Text Recognition Models for Historic Dutch. Journal of Open Humanities Data, 2024, 10 (43), pp.1-7.
10.5334/johd.225. hal-04890226

HAL Id: hal-04890226

https://hal.science/hal-04890226v1

Submitted on 16 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
<figure><img src="image_2.png" /></figure>

# ARletta. Open-Source
Handwritten Text
Recognition Models for
Historic Dutch

DATA PAPER

]u[ ubiquity press

LITH LEFRANC ID

ILJA VAN DAMME ID

THIBAULT CLÉRICE ID

MIKE KESTEMONT ID

## ABSTRACT

We release ARletta, a series of open-source models for the automated transcription of historic Dutch-language handwritten sources, which has remained a desideratum in the scholarly community until now. All models presented were trained on publicly available data using the open-source *kraken* engine. Our endeavor focuses on the digitization of a large-scale collection of local police reports (1876–1945). Additionally, we include a supermodel trained on the union of other Dutch-language datasets (extending back to the 17th century) which we hope will be useful as a foundational model for future projects. Our results demonstrate performance that is competitive with proprietary software solutions.

## CORRESPONDING AUTHOR:

Lith Lefranc

Centre for Urban History,
University of Antwerp,
Antwerp, BE; Antwerp Centre
for Digital Humanities and
Literary Criticism, University
of Antwerp, Antwerp, BE

Lith.Lefranc@UAntwerpen.be

## KEYWORDS:

handwritten text recognition;
layout analysis; line
segmentation; historic Dutch

## TO CITE THIS ARTICLE:

Lefranc, L., Van Damme, I.,
Clérice, T., & Kestemont, M.
(2024). ARletta. Open-Source
Handwritten Text Recognition
Models for Historic Dutch.
*Journal of Open Humanities
Data*, 10: 43, pp. 1–7. DOI:
https://doi.org/10.5334/
johd.225

<footnote>*Author affiliations can be found in the back matter of this article</footnote>
# (1) OVERVIEW

## REPOSITORY LOCATION

https://doi.org/10.5281/zenodo.11191457

## CONTEXT

Our need for a Dutch-language model for handwritten text recognition (HTR) arose in the context of a data-driven research project based on the historic incident books ('gebeurtenisboeken') of Antwerp, Belgium.<fnref n="1" /> In these books, local policemen reported meticulously on their neighborhood patrols and documented many aspects of urban street life (see example in Figure 1 and Table 1).

<figure><img src="image_3.png" /></figure>

Lefranc et al.
Journal of Open
Humanities Data
DOI: 10.5334/johd.225

<table><caption>**Figure 1** Example of an incident (FelixArchief, 731#1604, p. 211).</caption><tr><td>1 Mei 1885<br>Mr Sergoynne<br>à verbaliser<br>D</td><td>Om 9 1/2 uren 's avonds heb ik gezien dat de meid uit het huis n° 57 der Leopoldt matten uitklopte voor hare woning tegen den muer van het gasthuis toen ik er naartoe ging liep zy binnen en is niet meer buiten gekomen (zij veroorzaakte eenen over vloed van stof)<br>Broes, Dymphna, 16 jaar, geb. te Minderhout, meid<br>Wonende Leopold str. 57.<br>Comme cette fille est ci jeune et qu'elle savait pas mieux nous n'y avons pas donné suite [onleesbare handtekening]</td></tr><tr><td>1 May 1885<br>Mr Sergoynne<br>draw up an<br>official report<br>D</td><td>At 9 1/2 hours in the evening I saw that the maid from the house N° 57 in the Leopold street was beating rugs against the wall of the guesthouse when I came over she ran away inside the house and didn't come outside any more (she caused a flood of dust).<br>Broes, Dymphna, 16 years, born in Minderhout, maid<br>residing in Leopoldstr 57.<br>Because she was so young and she didn't know better, we didn't follow up on this. [ unreadable signature-mark]</td></tr></table>

<caption>**Table 1** Transcription of Figure 1 and translation to English (FelixArchief, 731#1604, p. 211).</caption>

For our research project, we scanned 271 incident books (87,594 pages containing text) preserved by the Antwerp City Archives, ranging from 1876 up to 1945. A ground truth training set of 15,000 words is generally considered adequate for obtaining satisfying HTR results (Muehlberger et al., 2019). This proved to be insufficient in the case of the incident books, as the pages contain many different handwriting styles and display irregularities in layout. Additionally, the incidents feature French passages as a consequence of Belgian language policies at that time. Because existing solutions fell short, we developed new models for layout analysis, line segmentation, and transcription through the open-source *kraken* engine, only using publicly available data.

# (2) METHOD

## STEPS

### Assets

Our project depended on the following assets:

* **Manu:** At the end of 2023, no models were publicly available in the *kraken* model repository (Kiessling, 2020) for the automated transcription of handwritten sources in historic Dutch, except the multilingual medieval CATMuS model that was partially trained on Middle Dutch (Clérice et al., 2023). Of all the models on HTR-united, *Manu McFrench* came closest to our Antwerp focus material because it was trained on French materials with a similar temporal scope (Chagué et al., 2023).

<footnote n="1">¹ PhD research project on nocturnal street life in Antwerp (1876–1940) through data-driven methods by Lith Lefranc under the supervision of Ilja Van Damme and Mike Kestemont.</footnote>
* *IJssberg:* Two publicly available training sets created by the Dutch national archives seemed valuable to us because they share the same language, historical period, and administrative character (Keijser, 2020):

○ VOC: 4,735 pages of the 17th and 18th century East-Asia Company archive

○ Notarial: 1,615 pages of 19th-century Dutch notarial deeds

* *Antwerp incident books:* We prepared a ground truth dataset from the Antwerp incident books. One expert (PhD student) annotated an initial subset of the data, consisting of 271 randomly selected pages.<fnref n="2" /> This subset was later supplemented with 3444 pages annotated by students.<fnref n="3" /> The student-contributed materials are much larger in scope than the materials annotated by the expert, but the former's quality is unfortunately much less consistent.

<figure><img src="image_4.png" /><figcaption>**Figure 2** Visualization of the training process.</figcaption></figure>

Layout analysis and line segmentation

In terms of layout, the Antwerp material contains two region types of interest: marginal text (marginalia) and blocks of main text (paragraph) (see Figure 3).<fnref n="4" /> Using the YALTAi library (Clérice, 2023), we trained a YOLOv8 model to automatically extract these regions from a page and limit the line segmentation to these regions. For the line segmentation, kraken's default BLLA model produced acceptable results, but we lightly finetuned it on the Antwerp data to alleviate some ad hoc issues in the data. Regarding layout, we kept all baselines in the original datasets but repolygonized them to guarantee their compatibility with the kraken framework. Baselines are annotated with straightforward geometric lines in HTR and are fairly portable across different platforms. For HTR, the baselines are used to determine a suitable polygon around lines, maximizing the inclusion of ascenders and descenders of characters, while minimizing the overlap with glyphs from neighbouring lines. The polygons which result from this process nevertheless tend to be quite irregular in shape and are much less portable across different transcription engines. The repolygonization step for kraken was therefore essential (cf. Pinche, 2023; see Figures 2 and 3).

All transcriptions were encoded as UTF-8 and we deliberately chose NFKD normalization, meaning that complex glyphs, such as those involving diacritics, were decomposed into their constituent glyphs. The glyph `<é>`, for instance, could be represented in Unicode as a single glyph (U+00E9) or the combination of simplex `<e>` (U+0065) and the combining acute accent (U+0301). We wanted to capture the intuition that the prediction `<e>` (U+0065) for `<é>` would still be a relatively better prediction than a wholly unrelated character.

<footnote n="2">2 This random selection was stratified, in that each incident book is represented by at least a single page.</footnote>

<footnote n="3">3 These were not randomly selected, but consecutive pages from incident books were chosen in function of the students' research assignments: FelixArchief, 450#58, 731#1592, 731#1596, 731#1598, 731#1612, 731#1613, 731#1614, 731#1626, 731#1639, MA#17871, MA#28935, MA#28937, MA#30824, MA#30825, MA#30827, MA#30828, MA#30844. The students had to annotate the sources as part of an undergraduate module at the University of Antwerp on learning how to transcribe and use historical sources.</footnote>

<footnote n="4">4 We applied the standardized region nomenclature as proposed in the SegmOnto vocabulary (Gabay et al., 2023).</footnote>
<figure><img src="image_5.png" /></figure>

Lefranc et al.
Journal of Open
Humanities Data
DOI: 10.5334/johd.225

<caption>**Figure 3** An example of a page with annotated text regions (green) and baselines (pink) (FelixArchief, MA#17612, p. 85).</caption>

## Handwritten text recognition

Our project focused on the HTR for the Antwerp incident books. Given the considerable diachronic spread of the available training material, we experimented with the following strategies:

- **Finetuning**: we sequentially finetuned a series of models, where we started with the oldest materials and gradually proceeded to the more recent periods. We indicate such a pretraining workflow using arrows; e.g. “Manu : VOC → Notarial”

- **Union**: we experimented with joining datasets, through training on the union of the available materials. We indicate such a union approach using + signs: e.g. “Manu : VOC + Notarial”

- **Scratch**: we trained on a subset of the available calibration assets, without making use of a foundation model but relied on randomly initialized model weights.<fnref n="5" />

Models were trained on a single GPU (GeForce RTX 2080 Ti) using a learning rate of 10⁻⁴, a batch size of 12 and early stopping in combination with learning rate reduction (for a maximum of 50 epochs and a patience of 3 epochs). When finetuning models, the backbone was frozen during the initial 5K steps; we applied a warmup of 10K steps. We evaluate our models through the character accuracy rate measure (CAR). Test scores were calculated through the CERberus tool (see Table 3), which offers options for relaxed evaluations (Haverals & Kestemont, 2023; Haverals, 2024). For all datasets, we split off a random, unstratified validation set (10% of the available data) at the level of individual pages; the remaining 90% of the pages were used for training models. For the Antwerp dataset, a dedicated test set of 101 pages was created (annotated by the expert only) on which all models were evaluated (see Table 2).

<footnote n="5">5 In the Variable-size Graph Specification Language (VGSL), that network can be expressed as: `[1,120,0,1 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,13,32 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 Mp2,2 Cr3,9,64 Do0.1,2 S1(1x0)1,3 Lbx200 Do0.1,2 Lbx200 Do0.1,2 Lbx200 Do]`.</footnote>
<table><thead><tr><th>DATASET</th><th>SUBSET</th><th>PAGES</th><th>REG-IONS</th><th>LINES</th><th>LINE LENGTH*</th><th>CHAR-ACTERS</th><th>VOCA-BULARY**</th></tr></thead><tbody><tr><td rowspan="2">VOC</td><td>train</td><td>4261</td><td>6079</td><td>132611</td><td>36.37 (±17.84)</td><td>4823321</td><td>120</td></tr><tr><td>valid</td><td>474</td><td>655</td><td>15154</td><td>35.79 (±17.57)</td><td>542321</td><td>101</td></tr><tr><td rowspan="2">Notarial</td><td>train</td><td>1453</td><td>3624</td><td>92003</td><td>35.03 (±20.03)</td><td>3222690</td><td>107</td></tr><tr><td>valid</td><td>162</td><td>377</td><td>9554</td><td>35.78 (±18.67)</td><td>341841</td><td>96</td></tr><tr><td rowspan="2">Antw-expert</td><td>train</td><td>243</td><td>1828</td><td>10766</td><td>30.63 (±17.10)</td><td>329806</td><td>89</td></tr><tr><td>valid</td><td>28</td><td>209</td><td>1260</td><td>29.59 (±16.77)</td><td>37288</td><td>83</td></tr><tr><td rowspan="2">Antw-students</td><td>train</td><td>3099</td><td>27496</td><td>145387</td><td>29.00 (±17.76)</td><td>4216129</td><td>118</td></tr><tr><td>valid</td><td>345</td><td>3089</td><td>16196</td><td>28.92 (±17.85)</td><td>468359</td><td>106</td></tr><tr><td>Antw-test</td><td>test</td><td>101</td><td>715</td><td>4628</td><td>30.18 (±17.09)</td><td>139658</td><td>90</td></tr></tbody></table>

<table><thead><tr><th>MODEL</th><th></th><th>VALIDATION*</th><th>ANTW-TEST</th><th>ANTW-TEST (RELAXED)**</th></tr></thead><tbody><tr><td>Manu (base model)</td><td></td><td>NA</td><td>70.51%</td><td>72.24%</td></tr><tr><td>Manu : VOC</td><td></td><td>94.36%</td><td>76.57%</td><td>78.95%</td></tr><tr><td>Manu : VOC → Notarial</td><td></td><td>95.68%</td><td>83.04%</td><td>85.22%</td></tr><tr><td>Manu : VOC → Notarial → Antw<sub>expert</sub></td><td></td><td>91.47%</td><td>90.01%</td><td>91.57%</td></tr><tr><td>Manu : VOC → Notarial → Antw<sub>expert</sub> + Antw<sub>student</sub></td><td></td><td>89.97%</td><td>92.58%</td><td>93.97%</td></tr><tr><td>Antw<sub>expert</sub> + Antw<sub>student</sub> (scratch)</td><td></td><td>89.05%</td><td>91.54%</td><td>92.88%</td></tr><tr><td>Manu : VOC + Notarial + Antw<sub>expert</sub> + Antw<sub>student</sub> (super model)</td><td></td><td>92.78%</td><td>92.31%</td><td>93.69%</td></tr></tbody></table>

In the finetuned model stack, we observe a gradual improvement in CAR as the materials come closer in time to the Antwerp dataset (see Table 3). Combining the expert and students' annotations in the Antwerp materials is beneficial, despite the lower transcription quality of the latter. This benefit must come from the sheer size of the students' annotations. The finetuning approach, finally, outperforms training from scratch, as well as the grand model on the Antwerp test set, but not by a large margin. The super model, however, still performed acceptably, suggesting its potential usefulness as a foundation model for other applications in this domain in the future.

## (3) DATASET DESCRIPTION

### REPOSITORY NAME

Zenodo

### OBJECT NAME

ARletta

### FORMAT NAMES AND VERSIONS

data: jpg, xml, txt; models: pt; ml.model; software: py

### CREATION DATES

2021.10.01–2024.04.15

### DATASET CREATORS

Training data: Lith Lefranc, Oliver Bogaerts, Louise Ledegen; Models: Mike Kestemont

Lefranc et al.
Journal of Open
Humanities Data
DOI: 10.5334/johd.225

5

<caption>**Table 2** Train-validation splits for each dataset.</caption>

* Line length expressed in characters (including whitespace). Mean and standard deviation are reported.

** Number of unique characters in the character vocabulary.

<caption>**Table 3** HTR training results (CAR).</caption>

*Note that the validation scores cannot be directly compared across model rows below, but are still useful because they give the reader a sense of in-domain model fit.

**Exclusion of whitespace, punctuation and capital letters.
## LANGUAGE

English

## LICENSE

CC-BY-NC-SA

## PUBLICATION DATE

2024.05.16

## (4) REUSE POTENTIAL

The layout and HTR models are applicable to images of serial historical sources containing (Dutch-language) handwritten text. They are particularly tailored to administrative reports from the nineteenth and twentieth centuries whose formatting breaks down into margin and main body text. The baseline segmentation and HTR models can be applied to images of these sources through the *kraken* engine or through the *eScriptorium* platform (Kiessling et al., 2019). The text region model can only be adopted within YOLO/YALTAi. All of the annotated data can be used for training new models. We released the filenames in our training and validation splits to ensure replicability.

## ACKNOWLEDGEMENTS

Thanks to Wouter Haverals, Jef Peeraer, Felix Archief and the students of the course *Historische Oefeningen 2* (2021–2023, University of Antwerp) supervised by Ilja Van Damme, Wout Saelens, Egon Bauwelink, and Lith Lefranc.

## FUNDING INFORMATION

This dataset results from a research project funded by University of Antwerp (GOA FFB200403).

## COMPETING INTERESTS

The authors have no competing interests to declare.

## AUTHOR INFORMATIONS

- Lith Lefranc: Conceptualization; Data Curation; Formal Analysis; Project Administration; Resources; Validation; Writing – original draft

- Ilja Van Damme: Conceptualization; Funding Acquisition; Supervision; Writing – review & editing

- Thibault Clérice: Software

- Mike Kestemont: Conceptualization; Data Curation; Formal Analysis; Funding Acquisition; Methodology; Software; Supervision; Validation; Writing – original draft

## AUTHOR AFFILIATIONS

Lith Lefranc [ID](orcid.org/0000-0002-6860-552X)

Centre for Urban History, University of Antwerp, Antwerp, BE; Antwerp Centre for Digital Humanities and Literary Criticism, University of Antwerp, Antwerp, BE

Ilja Van Damme [ID](orcid.org/0000-0001-8614-2277)

Centre for Urban History, University of Antwerp, Antwerp, BE

Thibault Clérice [ID](orcid.org/0000-0003-1852-9204)

National Institute for Research in Digital Science and Technology, Paris, FR

Mike Kestemont [ID](orcid.org/0000-0003-3590-693X)

Antwerp Centre for Digital Humanities and Literary Criticism, University of Antwerp, Antwerp, BE
REFERENCES

Chagué, A., Clérice, T., Norindr, J., Humeau, M., Davoury, B., et al. (2023). Manu McFrench, from zero to hero: impact of using a generic handwriting recognition model for smaller datasets. *Digital Humanities 2023: Collaboration as Opportunity, Alliance of Digital Humanities Organizations*. Graz, Austria. Retrieved from https://inria.hal.science/hal-04094241/document (last accessed: 18 April 2024).

Clérice, T., Vlachou-Efstathiou, M., & Chagué, A. (2023). CREMMA Medii Aevi: Literary manuscript text recognition in Latin. *Journal of Open Humanities Data*, 9(4), 1–19. DOI: https://doi.org/10.5334/johd.97

Clérice, T. (2023). You Actually Look Twice At it (YALTAi): using an object detection approach instead of region segmentation within the Kraken engine. *Journal of Data Mining and Digital Humanities*. DOI: https://doi.org/10.46298/jdmdh.9806

Gabay, S., Pinche, A., Christensen, K., Camps, J.-B., & Carboni, N. (2023). SegmOnto, A Controlled Vocabulary to Describe the Layout of Pages. Retrieved from https://segmonto.github.io/ (last accessed: 18 April 2024).

Haverals, W. (2024). WHaverals/CERberus: v1.0.1. Zenodo. DOI: https://doi.org/10.5281/zenodo.10668052

Haverals, W., & Kestemont, M. (2023). Handwritten Text Recognition Applied to the Manuscript Production of the Carthusian Monastery of Herne in the Fourteenth Century [Conference presentation]. *Digital Humanities 2023: Collaboration as Opportunity, Alliance of Digital Humanities Organizations*. Graz, Austria. DOI: https://doi.org/10.5281/zenodo.8107391

Keijser, L. (2020). Scans and transcriptions of the VOC and the Haarlem notarial deeds archives. *Zenodo*. DOI: https://doi.org/10.5281/zenodo.3613666 (last accessed: 18 April 2024).

Kiessling, B., Tissot, R., Stokes, P., & Ezra, D. S. B. (2019). eScriptorium: An Open Source Platform for Historical Document Analysis. *2019 International Conference on Document Analysis and Recognition Workshops (ICDARW)*. Sydney, Australia. DOI: https://doi.org/10.1109/ICDARW.2019.10032

Kiessling, B. (2020). A Modular Region and Text Line Layout Analysis System [Conference presentation]. *17th International Conference on Frontiers in Handwriting Recognition (ICFHR)*. Dortmund, Germany. DOI: https://doi.org/10.1109/ICFHR2020.2020.00064

Muehlberger, G., Seaward, L., Terras, M., Ares Oliviera, V., Bosch, V., et al. (2019). Transforming scholarship in the archives through handwritten text recognition: Transkribus as a case study. *Journal of Documentation*, 7(5), 954–976. DOI: https://doi.org/10.1108/JD-07-2018-0114

Pinche, A. (2023). Generic HTR Models for Medieval Manuscripts. The CREMMALab Project. *Journal of Data Mining and Digital Humanities*. DOI: https://doi.org/10.46298/jdmdh.10252

TO CITE THIS ARTICLE:

Lefranc, L., Van Damme, I., Clérice, T., & Kestemont, M. (2024). ARletta. Open-Source Handwritten Text Recognition Models for Historic Dutch. *Journal of Open Humanities Data*, 10: 43, pp. 1–7. DOI: https://doi.org/10.5334/johd.225

Submitted: 21 May 2024

Accepted: 19 June 2024

Published: 11 July 2024

COPYRIGHT:

© 2024 The Author(s). This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC-BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. See http://creativecommons.org/licenses/by/4.0/.

*Journal of Open Humanities Data* is a peer-reviewed open access journal published by Ubiquity Press.