# Large Multilingual ATR Models and Humanities Practice

Benjamin Kiessling

► To cite this version:

Benjamin Kiessling. Large Multilingual ATR Models and Humanities Practice. 2025 Workshop SCOOP - Source Codes of the Past, Institute of Advanced Study, Jun 2025, Princeton NJ, United States. hal-05150070

HAL Id: hal-05150070

https://inria.hal.science/hal-05150070v1

Submitted on 8 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Large Multilingual ATR Models and Humanities
Practice

Benjamin Kiessling

June 10, 2025
* Automatic Text Recognition has become a staple technology in the Historical Sciences.

* Incremental progress in layout analysis and text recognition in combination with increased diffusion of tooling among scholars has led to most material being accessible to retrodigitization.

* ATR pipelines themselves have not changed substantially and most open ATR packages still employ recurrent line recognizers trained with CTC loss.
State of ATR

* These pipelines mesh well with certain scholarly practices in the Humanities.
  o can easily be fine-tuned on very small datasets

* Their limited generalization power requires large model zoos to achieve coverage.

* Multi-language and -script models are difficult to train.

* Text recognition requires accurate line segmentation.

<figure><img src="image_2.png" /></figure>
# Humanities Practices in ATR Deployment

* Transcription is an editorial process involving editorial choices.

* Same document transcribed in a multitude of ways depending on scientific purpose.

in allem vnserm Trübsal/das wir auch können
trösten/die da sind in allerley Trübsal mit dem
Troft/damit wir von GOTT getröstet werden/
Hochgelobet vnd geliebet in Ewigkeit/

Retaining ligatures, distinction between
long (f) and round (s), no normalization
of u and v

in allem vnserm Trübsal/ das wir auch können
trösten/ die da sind in allerley Trübsal mit dem
Troft/damit wir von GOTT getröstet werden/
Hochgelobet vnd geliebet in Ewigkeit/

Modernization of u-v, s-forms, umlauts,
and hyphenation, dissolution of ligatures

in allem unserm Trübsal/ das wir auch können
trösten/ die da sind in allerley Trübsal mit dem
Trost/damit wir von GOTT getröstet werden/
Hochgelobet und geliebet in Ewigkeit/
# Humanities Practices in ATR Deployment

* Datasets are not interchangeable and reusable without common transcription guidelines (CATMus, ...).

* Renders the notion of generalized models specific to domains.

* Alignment with specific transcription norms is necessary functionality.
Page-wise Recognition of Text

* Party is a WIP open text recognition method that generalizes well across a large number of languages and writing systems.

* Designed to avoid ML techniques that run counter to Humanities practices.

* Is intended to squash multiple recurrent issues

* non-trivial model selection

* strong linkage between segmentation and recognition methods

* degradation of recognition with imperfect segmentation

* BiDi troubles that require user-awareness during manual transcription
Page-wise Recognition of Text

<figure><img src="image_3.png" /></figure>
# Page-wise Recognition of Text - Decoder

* Decoder is a Llama model trained from scratch on an OSCAR subset with "tokenization-free" byte tokenization.

* Predicts and ingests UTF-8 stream:

* very compact token embedding layer

* open vocabulary

* adaptation through simple fine-tuning

<figure><img src="image_4.png" /></figure>
Page-wise Recognition of Text

* A base model pretrained on 46 languages with different writing systems (Arabic, Chinese, Hebrew, Greek, Syriac, Latin, ...) generally performs well above 90% character accuracy. This includes print, handwriting, lithography.

* Pretrained on 660k historical documents, 300k born-digital prints.
Page-wise Recognition of Text

* Transcription guidelines were notably *not* aligned before training.

* While for some languages the employed datasets are (mostly) uniform, e.g. classical Chinese and German, for others (Hebrew-script languages) there is considerable variation.

* It is probably not feasible to ensure transcription consistency for the dataset sizes required to train such gigantic models.

* Similar issues seen with Transkribus super models.
* Fine-tuning to ensure coherent transcription is necessary for many applications.

* RNN+CTC style recognizer allow fine-tuning with small dataset sizes (<10 pages) and on commodity hardware.

* Party models have 40×the parameters of kraken recognition models.

* With some tricks we can push the GPU memory consumption to 5Gb.
Page-wise Recognition of Text

* Dataset size to fine-tune for a particular script is large.

* Small datasets (<100 pages) tend to overfit severely.

* Excludes low resource material and many Humanities research questions.

* Integrate prototypical network layer to allow few-shot learning.

<figure><img src="image_5.png" /></figure>
Questions?