# Kraken - A Universal Text Recognizer for the Humanities

Benjamin Kiessling

<figure><img src="image_1.png" /></figure>

## To cite this version:

Benjamin Kiessling. Kraken - A Universal Text Recognizer for the Humanities. Digital Humanities 2019, Jul 2019, Utrecht, Netherlands. 10.34894/Z9G2EX. hal-04936936

HAL Id: hal-04936936

https://hal.science/hal-04936936v1

Submitted on 17 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Kraken - an Universal Text Recognizer for the Humanities

Benjamin Kiessling (benjamin.kiessling@psl.eu), Université PSL, France; Leipzig University

## 1. Introduction

Retrodigitization of both printed and handwritten material is a common prerequisite for a diverse range of research questions in the humanities. While optical character recognition on printed texts is widely considered to be fundamentally solved in academia, with the most commonly used paradigm (Graves et al., 2006) dating back to 2006, this hasn't translated into increased availability of adaptable, libre-licensed OCR engines to the technically inclined humanities scholar.

The nature of the material of interest commands a platform that can be altered with minimum effort to achieve optimal recognition accuracy; uncommon scripts, historical languages, complex or archaic page layout, and non-paper writing surfaces are rarely satisfactorily addressed by off-the-shelf commercial solutions. In addition, an open system ameliorates the severe resource constraints of humanities research by enabling sharing of artifacts, such as training data and recognition models, inaccessible with proprietary OCR technology.

## 2. Kraken

The Kraken text recognition engine is an extensively rewritten fork of the OCRopus system. It can be used both for handwriting and printed text recognition, is easily (re-)trainable, and great care has been taken to eliminate implicit assumptions on content and layout that complicate the processing of non-Latin and non-modern works.

Thus Kraken has been extended with features and interfaces enabling the processing of most scripts, among them full Unicode right-to-left, bidirectional, and vertical writing support, script detection, and multiscript recognition. Processing of scripts not included in Unicode is also possible through a simple JSON interface to the codec mapping numerical model outputs to characters. The same interface provides facilities for efficient recognition of large logographic scripts.

Output includes fine-grained bounding boxes down to the character level that may be used to quickly acquire a large number of samples from a corpus to assist in paleographic research. Kraken implements a flexible output serialization scheme utilizing a simple templating language. Templates are available for the most commonly used formats ALTO, hOCR, TEI, and abbyyXML.

While including implementations of all the subprocesses needed in a text recognition pipeline, most functional blocks can be accessed separately on the command line, allowing flexible substitution of specially optimized methods. A stable programming interface allows total customization and integration into other software packages.

## 3. Recognition

<figure><img src="image_3.png" /><figcaption>Figure 1. Network architecture (H: sequence height, W: sequence length, C: alphabet size)</figcaption></figure>

The recognition engine operates as a segmentation-less sequence classifier using an artificial neural network to map an image of a single line of text, the input sequence, into a sequence of characters, the
output sequence. The artificial neural network employed is a hybrid convolutional and recurrent neural network trained with the CTC loss function (Graves et al., 2006) that reduces training data requirements to line-level transcriptions (Figure 3). Regularization is mainly provided by dropout (Hinton et al., 2012) after both convolutional and recurrent layers. User intervention in determining training duration and model selection is largely eliminated through early stopping.

Specialized networks, e.g. for particularly complex scripts, can be assembled from building blocks with a simple network specification language although the default architecture shown in Figure 1 is suitable for the vast majority of applications.

Processing of dictionaries and library catalogues with extensive semantic markup such as italic, underlining, and bolding, is also possible through specially prepared training data.

## 4. Layout Analysis and Script Detection

<figure><img src="image_4.png" /></figure>
<figure><img src="image_5.png" /><figcaption>Figure 2. Sample output of the trainable segmentation method.</figcaption></figure>

Kraken's layout analysis extracts text lines from an input image for later processing by the recognition engine. Apart from a basic segmenter taken from OCRopus a trainable line extractor is in the process of being implemented. Full trainability of layout analysis is of utmost importance to a truly universal OCR system, as text layout and its semantics varies widely across time and space, e.g. hand-crafted methods for printed Latin text are unlikely to work reliably on Arabic text or manuscripts with extensive interlinear annotation.

The trainable layout analysis module consists of a two-step instance segmentation method: an initial seed-labelling network operates on the whole page labelling the area between baseline and mean of each line. As the output of the network is a probability of each pixel belonging to a baseline it is binarized using hysteresis thresholding after smoothing with a gaussian filter. The binarized image is then skeletonized and end point are extracted with a discrete convolution. Finally, the vectorized baseline between the endpoints is rectified and a variable environment calculated based on the distance of connected components from the labelled area is extracted.

The seed-labelling network is a modified U-net (Ronneberger et al., 2015) on the basis of a 34-layer residual network (He et al., 2016) pretrained on ImageNet.

Preliminary results on a page from a publicly available dataset of Arabic and Persian manuscripts (Kiessling et al., 2019) can be seen in Figure 2.

Script detection, the basis for multi-script support in the recognizer, is implemented as a segmentation- less sequence classification problem, similar to text recognition. Instead of assigning a unique label to each code point or grapheme cluster we assign all code points of a particular script the same label. The network is then trained to output the correct sequence of script labels (Figure 3). The output sequence is then used to split the line into single-script runs that can be classified with monolingual recognition models (Figure 4).

ويقول رئيس شركة U. S. Steel : انالا اؤمن بالدين. فالدين له الميزة الغير المسرة

ويقول رئيس شركة U. S. Steel : «انا لا اؤمن بالدين. فالدين له الميزة الغير المسرة
000000200000200000020020000002220000002000020020002222122122111112000020000200000

<caption>Figure 3. Original and modified ground truth (top: original line, middle: transcription, bottom: assigned script classes)</caption>

5. Results

<table>
  <thead>
    <tr>
      <th></th>
      <th>Mean character accuracy</th>
      <th>Standard deviation</th>
      <th>Maximum accuracy</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Prints<br>Arabic (Kiessling et al.,<br>2017)</td>
      <td>99.5%</td>
      <td>0.05</td>
      <td>99.6%</td>
    </tr>
    <tr>
      <td>Persian <sup>1</sup></td>
      <td>98.3%</td>
      <td>0.33</td>
      <td>98.7%</td>
    </tr>
    <tr>
      <td>Syriac <sup>2</sup></td>
      <td>98.7%</td>
      <td>0.38</td>
      <td>99.2%</td>
    </tr>
    <tr>
      <td>Polytonic Greek <sup>3</sup></td>
      <td>99.2%</td>
      <td>0.26</td>
      <td>99.6%</td>
    </tr>
  </tbody>
</table>
<table><tr><td>Latin (Springmann et al., 2018)</td><td>98.8%</td><td>0.09</td><td>99.3%</td></tr><tr><td>Latin incunabula (Springmann et al., 2018)</td><td>99.0%</td><td>0.11</td><td>99.2%</td></tr><tr><td>Fraktur (Springmann et al., 2018)</td><td>99.0%</td><td>0.31</td><td>99.3%</td></tr><tr><td>Cyrillic</td><td>99.3%</td><td>0.15</td><td>99.6%</td></tr><tr><td>Manuscripts</td><td></td><td></td><td></td></tr><tr><td>Hebrew <sup>4</sup></td><td>96.9%</td><td>-</td><td>-</td></tr><tr><td>Medieval Latin <sup>5</sup></td><td>98.2%</td><td>-</td><td>-</td></tr></table>
اجواء شتى. وهذا سبب من اسباب الغموض اللغوي وعدم تضمين اللفظ معناه المحدود الوضعي، والداعي الى تفوق الفن على الطبيعة ليصبح ضرباً من ضروب المخدرات. وكذا فلا يصور الشيء بكامله وانما يكتفى بالاشارة الى بعض اجزائه.

وخاصة الاميجاء قائمة على امكانياته، لان الاميجاء كالعمة يوسع الافتراضات ويطلق المخيلة والشعر والتأمل. فالاميجاء يحوك أوفاً من الحيوط الماونة حول مغزل النفس، او قل انه انتظار شي. سيحدث؟ وفي الانتظار لذة لا تعرفها الحقيقة الواقعة.

وكأنما تنقسم الالفاظ الى نثرية وشعرية. والشعرية هي الغرض. ومن هذا القبيل فن التصوير الزيتي، حيث يولد انسجام الالوان، واظلالها، وادهان بعض الخطوط والاجزاء، في نفس المتمتع، جوا الامحائيا شبيهاً بتأثير الالفاظ الشعرية. وفن «رمبراندت» هو من هذا الباب، حيث تحمل اللوحة اليك ما ينطلق وراء الخطوط

والايوان، وفي هذا الاميجاء يقول أحدهم:

«Deux choses sont également requises : l'une est certaine somme de complexité ou plus proprement de combinaison; l'autre une certaine quantité d'esprit suggestif, quelque chose comme un courant souterrain de pensée non visible, indéfinie... c'est l'excès dans l'expansion du sens qui ne sait être qu'insinué.» (۱)

والمشترع ملارمه قول صراح بهذا الصدد: «ان ما شيد من صروح، والبحر، والوجه الانساني، متعة لا يؤديها الوصف، بل يحملها الاميجاء (۲). واذن في الاميجاء، غنى للفكر «لان فعل اللغة الامحائية الرئيسي هو توليد مجازي ذكرية وشعرية، تازحت ام تفرعت» (۳)».

والشعر الرمزي بتحديده شحنة الامحائية. «ففي هذا الاميجاء، يتراءكم السكون ويحفظ ويتند. هذا ما يبدوا في ابتسامة «الجوكوندا» وفي «اصع يوحنا» المرفوعة في تمثال دي فنشي. ثم ان الاميجاء، بثابة عصا الجوة الموسيقية. فكما ان سيلاً من الموسيقى يتدفق تحت شارته، كذا تنبجس الدنيا الداخلية بانفعال اللفظة الامحائية.

<caption>Figure 4. Sample output of the script detection on a bilingual French/Arabic page. Note that Eastern Arabic are always classified as Latin text</caption>

<footnote>(۱) Bremond, La Poésie pure, p. 118.</footnote>

<footnote>(۲) Mallarmé, Divagations, p. 245.</footnote>

<footnote>(۳) Paulhan, La Double Fontc. du lang. P. 169.</footnote>
Kraken has been used on a wide variety of writing systems, achieving uniformly high character accuracy (CER). Sample accuracies for a diverse set of scripts spanning across multiple centuries of printing are shown in Table 1.

As a special use case we evaluated recognition of text and emphasis in a mixed English and romanized Arabic library catalog on a training set of 350 lines (50 lines in the validation set) resulting in an averaged CER of 99.3% (σ=0.16) over 10 runs with 95.38% CER on cursive and text with increased spacing (σ=1.46). When using only emphasized text accuracy as the stopping criterium mean accuracy rises to 99.03% (σ=0.28).

# Appendix A

## Bibliography

1. Graves, A., Fernández, S., Gomez, F. and Schmidhuber, J. (2006). Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. *Proceedings of the 23rd International Conference on Machine Learning*. ACM, pp. 369–376.

2. He, K., Zhang, X., Ren, S. and Sun, J. (2016). Deep residual learning for image recognition. *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*. pp. 770–778.

3. Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R. R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. *ArXiv Preprint ArXiv:1207.0580*.

4. Kiessling, B., Miller, M. T., Maxim, G., Savant, S. B. and others (2017). Important New Developments in Arabographic Optical Character Recognition (OCR). *Al-Uṣūr Al-Wusṭā*, **25**: 1–13.

5. Kiessling, B., Stoekl Ben Ezra, Daniel and Miller, Matthew Thomas (2019). BADAM: A Public Dataset for Baseline Detection in Arabic-script Manuscripts.

6. Ronneberger, O., Fischer, P. and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. *International Conference on Medical Image Computing and Computer-Assisted Intervention*. Springer, pp. 234–241.

7. Springmann, U., Reul, C., Dipper, S. and Baiter, J. (2018). Ground Truth for training OCR engines on historical documents in German Fraktur and Early Modern Latin. *ArXiv Preprint ArXiv:1809.05501*.

## Notes

1. Mid-20th century printing

2. Late-19th century printing in Serā form

3. Late-19th century printing

4. Midrash Tanhuma, BNF Héb 150

5. Josephus Latinus, Bamberg 78 with augmentation