# Multicentury Handwritten Text Recognition (HTR) model for historical documents from 17th to 20th centuries -Using TrOCR

Marttila Riikka, Joska Sanna, Lipsanen Mikko, Föhr Atte, Jokipii Ilkka

<figure><img src="image_1.png" /></figure>

To cite this version:

Marttila Riikka, Joska Sanna, Lipsanen Mikko, Föhr Atte, Jokipii Ilkka. Multicentury Handwritten Text Recognition (HTR) model for historical documents from 17th to 20th centuries -Using TrOCR. 2025. hal-05130154

HAL Id: hal-05130154

https://hal.science/hal-05130154v1

Preprint submitted on 11 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Multicentury Handwritten Text Recognition
(HTR) model for historical documents from
17th to 20th centuries – Using TrOCR

Marttila Riikka¹, Joska Sanna¹, Lipsanen Mikko¹, Föhr Atte¹, Jokipii Ilkka¹

¹ National Archives of Finland

firstname.lastname@kansallisarkisto.fi

**Abstract**

Handwritten texts and documents represent large potential for academic research. The process of digitisation, followed by the transformation of these handwritten materials into machine type format, stands to significantly enhance the accessibility and usability for research. The aim of this paper is to compare the performance of various handwritten text recognition (HTR) models developed specifically for individual centuries against a multicentury HTR model trained on a comprehensive dataset from 1600s to 1900s. All models are trained and fine-tuned using Transformer-based Optical Character Recognition (TrOCR) model as a base model.

The goal of this study is to determine whether a multicentury HTR model could learn to recognise the handwritten text from different centuries as well as both Swedish and Finnish languages. Maintaining only one model would reduce computational and storage resource requirements and facilitate simpler system integrations. It was also investigated whether a model could be trained with both handwritten and typewritten documents. The historical documents in the National Archives of Finland are mainly written in either Finnish or Swedish. With a well-performing multicentury HTR model, the National Archives of Finland can achieve its strategic goal of making archival material into data that is both searchable and usable for modern digital research. Modern technologies, such as large language models (LLM) combined with recognised historical documents through a retrieval-augmented generation (RAG) system, open significant potential for utilising historical texts. This includes applications like chatbots, data search and text generation methods.

**Keywords:** deep learning, digital humanities, handwritten text recognition, historical research, historical documents, machine learning, optical character recognition, transformer models, computer vision

**Introduction**

Handwritten historical documents remain an important part of research materials in different fields even today. HTR enables the accurate transcription of handwritten historical sources, making them machine-readable and accessible for large-scale analysis. The technology expands access to underrepresented sources, such as letters and diaries bringing previously marginalized historical voices within the digital archival corpus (Nockels et al. 2024). When historical documents are digitised, modern text analysis methods can be applied to make them easier to use and analyse. However, text must first be recognised from images containing text and converted into machine-readable form. Without the advancements in open-source machine learning methods and computational power in recent years, handwritten text recognition on a larger scale would not have been possible (Muehlberger et al. 2019).
A research project conducted at the National Archives of Finland utilises the pretrained Transformer-based Optical Character Recognition (TrOCR) model developed by Microsoft (Li et al. 2022). It combines an image Transformer encoder and a text Transformer decoder for optical character recognition, replacing traditional CNN- and RNN-based approaches and eliminating the need for additional language models for post-processing accuracy. TrOCR is pre-trained on synthetic data and fine-tuned on human-labeled datasets, demonstrating superior performance on both printed and handwritten text recognition tasks.

In this research, the performance of various HTR models developed specifically for the handwriting styles of individual centuries is compared against a multicentury HTR model trained on a comprehensive dataset from the 1600s to 1900s. The goal is to train HTR models to perform with sufficient accuracy on documents in both Finnish and Swedish languages. An additional goal was to discover the possibility of training a 1900s model with both handwritten and typewritten documents. A significant part of the typewritten documents includes handwritten notes and many handwritten documents, such as different forms, include typewritten text. But current open-source optical character recognition (OCR) tools, such as Tesseract (Smith 2007) and PaddleOCR (Li et al. 2022) are not trained to recognise handwritten text. Additionally, many of the documents include typewritten tables filled with handwritten text. It would be highly beneficial to develop a recognition model that can handle both handwritten and typewritten text. It would eliminate the need for a separate text style detection step in the pipeline, as all text could be processed by a single model.

One of the strategic goals of The National Archives of Finland is to bring historically significant archival material into data that is more searchable and usable than before for modern digital research. The National Archives' online services enhance the usability of this data and data packages are also provided for researchers to use and analyse. Deep learning technologies, such as high-performing HTR models, provide solutions for making handwritten historical documents more accessible and usable. The data has the potential to serve as source materials in many academic fields, such as historical or linguistic research or social science, and beyond (Lahtinen & Katajisto 2020, Paju et al. 2020). In addition to academic research, enhanced usability and easier access to archival materials benefit the public in various ways.

## Data

This research has access to tens of thousands of pages of annotated data. Annotation here refers to the transcription of texts and the marking of text lines. The number of lines per page varies typically ranging from 10 to 50. The total number of text lines used for model training process exceeds one million (n=1,207,628). All documents need to be transcribed, and text areas and text lines must be labelled as part of data preprocessing. To increase the amount of training data, weaker models were used for pre-annotating the texts, which were then corrected by humans. The text lines were annotated using the Label Studio tool (Tkachenko et al. 2020-2025). Label Studio outputs provide text lines as coordinates, which allows for the separation of text lines from page images and their matching with corresponding transcriptions. These text line images and transcriptions were then used for training and fine-tuning the TrOCR model.

The available data include court records from the 17th to 19th centuries, diaries and letters from the 18th to 19th centuries and census records and word diaries from the 20th century. Additionally, there were many smaller collections of historical documents. In Table 1 the number of row images is separated based on whether the text is written in Finnish or Swedish. As shown in Table 1, the Finnish language documents become more common from the late 19th century onward. Before that, most documents were written in Swedish, as Finnish had not yet developed into a written or official language. All typewritten documents used in this research were from 20th century (number of line images=48000).
<table><caption>Table 1 Amount of row images by language in dataset</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="4">Language</th><th rowspan="2">Total</th></tr><tr><th>Finnish</th><th>%</th><th>Swedish</th><th>%</th></tr></thead><tbody><tr><td rowspan="4">Data from</td><td>1600s</td><td>0</td><td>0</td><td>93092</td><td>100</td><td>93092</td></tr><tr><td>1700s</td><td>0</td><td>0</td><td>411339</td><td>100</td><td>411339</td></tr><tr><td>1800s</td><td>124014</td><td>21</td><td>456607</td><td>79</td><td>580621</td></tr><tr><td>1900s*</td><td>119023</td><td>97</td><td>3553</td><td>3</td><td>122576</td></tr><tr><td></td><td>1600-1900**</td><td>243037</td><td>20</td><td>964591</td><td>80</td><td>1207628</td></tr></tbody></table>

*Including typewritten line images (n=60000)

**Data is used in the training of the multicentury HTR model

## Methods

Before the development of the multicentury HTR model, various Optical Character Recognition (OCR) models, such as Tesseract (tesseract 2023) and PaddleOCR (PaddlePaddle 2023), were explored. However, the strength of OCR models lies naturally in typewritten text. Different open-source technologies were extensively tested and various data augmentation methods and image preprocessing techniques, such as blurring and noise reduction, were examined. Multi-model OCR (MMOCR) (Kuang et al. 2021) developed by a large developer community and Microsoft's Transformer-based OCR (TrOCR) were the most successful in recognising handwritten text. When comparing the MMOCR's Self-Attention Text Recognition Network (SATRN) model with the Microsoft/trocr-large-handwritten model distributed via Hugging Face (Microsoft 2021), the conclusion was that the SATRN model performed better in terms of computational power than the TrOCR model. However, the TrOCR model was prioritised for its accuracy, as it exhibited fewer character errors in the test data, generalised better to different datasets and performed well on both Finnish and Swedish language data with a single model. Additionally, a smaller version of the TrOCR model is available which also helps to reduce computational resource requirements. In the TrOCR model, no image blurring or noise reduction preprocessing techniques are needed.

Both the Microsoft/trocr-large-handwritten and Microsoft/trocr-small-handwritten models were tested. The trocr-large-handwritten model is more complex and accurate, making it better suited for handling challenging handwriting. It requires more computational resources. The trocr-small-handwritten model has significantly fewer parameters and is lighter and faster, but it is less accurate with more complex handwriting. At the National Archives of Finland, the multicentury HTR model was trained using the trocr-large-handwritten model, primarily on handwritten text that uses basic Latin characters (A-Z, a-z) and includes Nordic special characters (å, ä, ö). It has not been trained on non-Latin alphabets, such as Chinese characters, Cyrillic script, or other writing systems like Arabic or Hebrew. The model may not generalize well to any other languages than Finnish, Swedish or English. The multicentury HTR model is published in the HuggingFace (https://huggingface.co/Kansallisarkisto/multicentury-htr-model).

Before extracting row images from the page images, the data was randomly divided into training (80%), validation (10%), and test (10%) datasets. The training dataset was used for training the HTR model, while the validation set was automatically used by TrOCR for model validation to identify the best model configuration. The test dataset was used to compare different models against each other. The test data had been randomly selected and weighted across different centuries and languages (Swedish and Finnish) to ensure a sufficiently representative sample from each century and both languages. Different models are compared, and model accuracy was evaluated using the Character
Error Rate (CER) value. The CER metric indicated how many character errors the model makes per 100 characters:

$$CER = (I + D + S) / N$$

where

I = Number of character insertions

D = Number of character deletions

S = Number of character substitutions

N = Total number of characters in the reference text

Table 2 compiles the number of row images from different centuries. The quantities are presented separately for training, testing and validation datasets.

<table><caption>Table 2: Number of row images in used dataset</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="5">Dataset</th></tr><tr><th>Train</th><th>Validation</th><th>Test</th><th>Total</th><th>%</th></tr></thead><tbody><tr><td rowspan="5">Data from</td><td>1600s</td><td>74341</td><td>9489</td><td>9262</td><td>93092</td><td>8</td></tr><tr><td>1700s</td><td>327431</td><td>40497</td><td>43411</td><td>411339</td><td>34</td></tr><tr><td>1800s</td><td>464507</td><td>57343</td><td>58771</td><td>580621</td><td>48</td></tr><tr><td>1900s handwritten<br>typewritten</td><td>49649<br>48000</td><td>6921<br>6000</td><td>6006<br>6000</td><td>62576<br>60000</td><td>5<br>5</td></tr><tr><td>from 1600s to<br>1900s*</td><td>963928</td><td>120250</td><td>123450</td><td>1207628</td><td>100</td></tr></tbody></table>

*Data is used in the training of the multicentury HTR model

The multicentury HTR model was trained by using both local hardware resources at the National Archives of Finland as well as external resources from the Finnish IT Center of Science (CSC). Training the model requires significant processing power which was provided by NVIDIA RTX A6000 Graphics Processing Units (GPUs) acquired by the National Archives of Finland. When operating at full capacity, these GPUs consume approximately 300W of power each. Additionally, CSC's Mahti supercomputing environment was used, offering access to NVIDIA Ampere A100 GPUs.

Training the multicentury HTR model (14 epochs) takes approximately 72 hours using four GPUs with following hyperparameters:

* Evaluation strategy: epoch

* Train batch size per device: 16

* Learning rate: 1e-5

* Scheduler: linear

* Warmup steps: 500

* Optimizer: AdamW

* Number of epochs: 14

* FP16 mixed precision training: True

* Half precision backend: cuda amp

Furthermore, to support data processing, a separate model was trained to detect text lines and areas from digitized images using a YOLOv8 model 8 (Jocher et al., 2023). The first step of the HTR
pipeline (Figure 1) is to detect text regions and text lines from the image. The text lines are then converted into temporary line images and fed into the text recognition model.

<figure><img src="image_3.png" /><figcaption>Figure 1 HTR pipeline</figcaption></figure>

The second step is to detect text regions from image. To arrange the recognised handwritten text in the right reading order, text regions are needed to be detected. The coordinates of the text lines inside these text regions were compared to ensure correct ordering. Each line is then associated with the closest text region to ensure the reading order within each text region is accurate. Figure 2 shows an example page in original image format before any object detection (a), after text area detection (b) and after text line detection (c).

<figure><img src="image_4.png" /><figcaption>Figure 2 Example of image with detected region and line bounding boxes.</figcaption></figure>

Both the text region and text line detection steps use models trained with YOLOv8. This research focuses specifically on the HTR part of the pipeline. Table 3 summarises the amount of data used for fine-tuning the models. It includes the number of annotated text regions used for text region detection, the number of annotated lines used for line detection, as well as the number of line images and their corresponding transcribed text used for training the HTR model.
<table><caption>Table 3 The number of annotations used (text areas/text lines/row images)</caption><thead><tr><th rowspan="2"></th><th colspan="4">Dataset</th></tr><tr><th>Train</th><th>Validation</th><th>Test</th><th>Total</th></tr></thead><tbody><tr><th>Region detection</th><td>3598</td><td>450</td><td>450</td><td>4500</td></tr><tr><th>Line detection</th><td>7344</td><td>920</td><td>920</td><td>9184</td></tr><tr><th>Text Recognition</th><td>963928</td><td>120250</td><td>123450</td><td>1207628</td></tr></tbody></table>

## Results

The CER of the current multicentury HTR model is 2.9 (Table 4). Over 80% of the training data is from the 1700s to 1800s which is shown in the model's performance (CER 2.6-2.9). The style of handwriting does not change at any specific timing point and the data from the end of 1800s is quite similar in style to data from 1900s. The CER of the test dataset from 20th century is 3.3. The highest CER is in the test dataset from 17th century (5.5). One reason might be the poorer scanning quality of the very old paper, but getting more training data would most likely improve accuracy.

<table><caption>Table 4 Comparison of CER across different models and test datasets</caption><thead><tr><th rowspan="2"></th><th colspan="5">Test dataset from</th></tr><tr><th>1600s (CER)</th><th>1700s (CER)</th><th>1800s (CER)</th><th>1900s (CER)</th><th>1600-1900 (CER)</th></tr></thead><tbody><tr><th>multicentury HTR model*</th><td><strong>5.5</strong></td><td><strong>2.9</strong></td><td><strong>2.6</strong></td><td><strong>3.3</strong></td><td><strong>2.9</strong></td></tr><tr><th>1600s model</th><td>5.4</td><td>3.8</td><td>4.5</td><td>7.1</td><td>4.9</td></tr><tr><th>1700s model</th><td>5.5</td><td>2.9</td><td>2.6</td><td>3.3</td><td>3.0</td></tr><tr><th>1800s model</th><td>5.5</td><td>2.9</td><td>2.6</td><td>3.3</td><td>3.0</td></tr><tr><th>1900s model</th><td>5.5</td><td>2.9</td><td>2.6</td><td>3.3</td><td>3.0</td></tr></tbody></table>

The accuracy of the century specific models is not better than the accuracy of the multicentury model, or at least the accuracy of the multicentury model is good enough to eliminate the need to maintain century specific models. Given that its accuracy was consistently sufficient across test dataset from all centuries the development effort was subsequently directed toward the multicentury HTR model. In Figures 3-6 examples of documents from different centuries are presented as line images, transcriptions and HTR outputs using multicentury HTR model.

<footnote>*Data from 1600s to 1900s is used in the training of the multicentury HTR model</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 3 Row images with predicted and original annotated text. Examples from 17th century.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 4 Row images with predicted and original annotated text. Examples from 18th century.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 5 Row images with predicted and original annotated text. Examples from 19th century.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 6 Row images with predicted and original annotated text. Examples from 20th century.</figcaption></figure>

Conclusion

The outcomes of the study are significant. The dataset used for model training and validation included a diverse array of data from numerous scribes across different centuries. It was unexpected that the same model could learn both Swedish and Finnish languages so effectively. This removes the need for the system to classify the language of the text in the image separately, which enhances the efficiency of the planned pipeline, making it easier to optimise and maintain.

In practice we have found that when working with large datasets, for example court records and war diaries, it is useful to fine-tune a multicentury HTR model with a small portion of the larger dataset and then apply the fine-tuned model to process the larger volume of data. The results of the century-specific models compared in this study support this approach as well. However, the results with multicentury HTR model are so promising that fine-tuning for smaller datasets does not seem warranted.

With 420 shelf kilometers of material to digitise the multicentury HTR model enables the National Archives of Finland to convert historically significant cultural heritage into typewritten form. This makes the materials more accessible for reading and studying, as well as for broader use and exploitation.

Like all machine learning models, handwriting recognition models can reflect biases from their training data. However, the model has been trained to read text from images and does not have a creative process for generating new content. Therefore, the biases of the model in this context refer to character errors or inaccuracies. It is important to distinguish this from the generative AI's ability to produce text based on its training data and the associated disinformation bias.

During the project various modern large language model (LLM) tools were tested to postprocess recognised historical handwritten texts. When dealing with historical or legal documents the accuracy of every character and word is crucial. If a LLM misinterprets the context, the content of the original document can change significantly. Instances of model hallucination have also been identified, where text is generated from an entirely different context. Leveraging large language models in combination with a retrieval-augmented generation (RAG) system which utilises historical data from HTR-processed documents offers significant potential for utilising historical documents (Lewis et al. 2021). This approach enables the combination of information from various sources such as archives to provide clear descriptions and summaries. Furthermore, by integrating Model Context Protocol (MCP) servers, it becomes possible to create structured, secure interfaces to diverse archival
databases and document repositories (Hou et al. 2025). MCP servers can expose tools that allow
language models to dynamically retrieve, combine, filter, and interpret historical texts, enhancing the
accuracy and contextual relevance of generated outputs.

## Acknowledgements

We would like to thank everyone who participated in the project with special thanks to all the
annotators as well as the volunteers and researchers specialising in historical handwriting who wrote
the transcriptions. Furthermore, the collaboration with CSC is gratefully acknowledged.

## References

Hou, X., Zhao, Y., Wang, S., & Wang, H. (2025). *Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions* (arXiv:2503.23278v2).
https://doi.org/10.48550/arXiv.2503.23278

Jocher, G., Chaurasia, A., & Qiu, J. (2023). YOLO by Ultralytics (Version 8.0.0) [Computer software]. https://github.com/ultralytics/ultralytics

Kuang, Z., Sun, H., Li, Z., Yue, X., Lin, T. H., Chen, J., Wei, H., Zhu, Y., Gao, T., Zhang, W., Chen, K., Zhang, W., & Lin, D. (2021). *MMOCR: A comprehensive toolbox for text detection, recognition and understanding*. In *Proceedings of the 29th ACM International Conference on Multimedia (MM '21)*. ACM. https://doi.org/10.1145/3474085.3478328

Lahtinen, A. & Katajisto, K. (2020). *Handwritten text recognition opens up a treasure trove of information on Finnish society*. In *J. Nuorteva & P. Happonen (Eds.), The National Archives of Finland Strategy 2025: Perspectives for the Future* (pp.18-19).

Lee, J., Park, S., Baek, J., Oh, S. J., Kim, S., Lee, H. (2021). *On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention*. arXiv:1910.04396v1

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Riedel, S. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. *Advances in Neural Information Processing Systems*, 33. https://arxiv.org/abs/2005.11401

Li, C., Liu, W., Guo, R., Yin, X., Jiang, K., Du, Y., Du, Y., Zhu, L., Lai, B., Hu, X., Yu, D., & Ma, Y. (2022). *PP-OCRv3: More attempts for the improvement of ultra lightweight OCR system* (arXiv:2206.03001v2) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2206.03001

Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z., Wei, F. (2022). *TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models*. arXiv: 2109.10282v5

Microsoft. (2021). *TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models*. Hugging Face. https://huggingface.co/microsoft/trocr-large-handwritten

Muehlberger, G., Seaward, L., Terras, M., Ares Oliveira, S., Bosch, V., Bryan, M., Colutto, S., Déjean, H., Diem, M., Fiel, S., Gatos, B., Greinoecker, A., Grüning, T., Hackl, G., Haukkovaara, V., Heyer, G., Hirvonen, L., Hodel, T., Jokinen, M., ... Zagoris, K. (2019). *Transforming scholarship in the archives through handwritten text recognition: Transkribus as a case study*. *Journal of Documentation*, Vol. 75 No. 5, pp. 954-976. https://doi.org/10.1108/JD-07-2018-0114

Nockels, J., Gooding, P., & Terras, M. (2024). *The implications of handwritten text recognition for accessing the past at scale*. *Journal of Documentation*, 80(7), 148-167
Paju, P., Oiva, M., & Fridlund, M. (2020). Digital and distant histories: Emergent approaches within the new digital history. In M. Fridlund, M. Oiva, & P. Paju (Eds.), *Digital histories: Emergent approaches within the new digital history* (pp. 3–18). Helsinki: Helsinki University Press.
https://doi.org/10.33134/HUP-5-1

Smith, R. (2007). An overview of the Tesseract OCR engine. In *Proceedings of the Ninth International Conference on Document Analysis and Recognition (ICDAR '07)*. IEEE.

Tkachenko, M., Malyuk, M., Holmanyuk, A., & Liubimov, N. (2020–2025). *Label Studio: Data labeling software* [Computer software]. GitHub. Retrieved from https://github.com/HumanSignal/label-studio