<figure><img src="image_1.png" /></figure>

HAL
open science

# Towards automatic TEI encoding via layout analysis

Juliette Janes, Ariane Pinche, Claire Jahan, Simon Gabay

► To cite this version:

Juliette Janes, Ariane Pinche, Claire Jahan, Simon Gabay. Towards automatic TEI encoding via layout analysis. Fantastic future 21, 3rd International Conference on Artificial Intelligence for Libraries, Archives and Museums, AI for Libraries, Archives, and Museums (ai4lam), Dec 2021, Paris, France. hal-03527287

HAL Id: hal-03527287

https://hal.science/hal-03527287

Submitted on 15 Jan 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Towards automatic TEI encoding via layout
analysis

Juliette Janes, Ariane Pinche, and Claire Jahan

École nationale des chartes | PSL (France)

Simon Gabay

Université de Genève (Switzerland)

The forefront of research on textual documents (may they be manuscripts and prints) is slowly moving from text recognition to automatic encoding. Quickly transforming images into XML-TEI documents is therefore the next important obstacle that needs to be tackled to offer enhanced mining options to digital libraries users.

<figure><img src="image_3.png" /><figcaption>Figure 1: La Bruyère, *Les Caractères de Théophraste*, 1688, p. 53. In black the page number, in red a headpiece, in brown the title, in cyan the text, in blue the drop capital, in yellow the signature, in grey a marginal note, in green noise.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 2: Example of a page with zones automatically recognised: Molière, *L'Ecole des Maris*, 1661, p. 1. In yellow the page number, in purple the headpiece, in pink the text, in blue the drop capital, in green the signature.</figcaption></figure>
If extensive research has been carried on complicated and highly standardised layouts such as manuscripts (Gabay, Rondeau Du Noyer, et al. 2020) or exhibition catalogues (Gabay, Topalov, et al. 2021) with dedicated tools like GROBID (Khemakhem 2020), more simple documents still await for an easily trainable and implementable solution. Most of the pages contain simple elements such as page numbers, marginal notes, signatures... (cf. fig. 1) which all have a different status than the actual body of the text, and therefore need to be disentangled from one another. For instance, finding all the mentions of Andromache in the play of the same name is a nonsense if we do not separate running titles (in which the name of the eponym heroins appear) from the dialogues.

Recent experiments show that a standardised description of the page and state-of-the-art models for layout analysis could offer a limited, yet efficient solution (cf. fig. 2) to numerous scholars for two reasons. On the one hand powerful open-source OCR engines are now available (Kiessling 2019) via user-friendly interfaces (Kiessling et al. 2019): it will become easier to train models tailored to one's need in a near future. On the other hand research institutions are now investing in OCR infrastructures (Gabay 2021; Romary et al. 2021), which will accelerate the production of data locally, without the direct support of GLAMs.

Using a common vocabulary to annotate zones called SegmOnto<fnref n="1" /> (Gabay, Camps, et al. 2021) (that is still evolving), we have developed a generic workflow to analyse the layout, OCRise the text, and convert the ALTO output into minimally encoded TEI files (cf. table 1). This workflow is currently being tested on three different datasets: one of medieval manuscripts (cf. table 5), one of 17th c. literary prints (cf. table 4), and one of 19th c. catalogues (cf. table 6).

<table><caption>Table 1: SegmOnto zones and their corresponding TEI element.</caption><thead><tr><td>Segmonto zone</td><td>Corresponding TEI element</td></tr></thead><tbody><tr><td>Damage</td><td>&lt;damage&gt;</td></tr><tr><td>Decoration</td><td>&lt;figure&gt;</td></tr><tr><td>DropCapital</td><td>&lt;hi&gt;</td></tr><tr><td>Figure</td><td>&lt;figure&gt;</td></tr><tr><td>Main</td><td>&lt;p&gt;</td></tr><tr><td>Margin</td><td>&lt;note&gt;</td></tr><tr><td>MusicNotation</td><td>&lt;figure&gt;</td></tr><tr><td>Numbering</td><td>&lt;fw type="Numbering"&gt;</td></tr><tr><td>RunningTitle</td><td>&lt;fw type="RunningTitle"&gt;</td></tr><tr><td>Seal</td><td>&lt;figure&gt;</td></tr><tr><td>Signatures</td><td>&lt;fw type="Signatures"&gt;</td></tr><tr><td>Stamp</td><td>&lt;figure&gt;</td></tr><tr><td>Table</td><td>&lt;table&gt;</td></tr><tr><td>Title</td><td>&lt;p&gt;</td></tr></tbody></table>

The key aspect of the workflow being zones detection, we have experimented two neural

architectures for the layout analysis, and possible combinations of data to increase the efficiency. One model for each set has been trained, then two combinations: the medieval and the 17th c. data on the one hand, the 17th c. data and the 19th c. on the other hand (cf. table 2).

A second experiment has redimensioned the image taken as input in the

<footnote n="1">¹https://segmonto.github.io.</footnote>
<table><caption>Table 2: Architecture 1: efficiency of complex models measured with Mean Intersection-Over-Union (MIoU), frequency-weighted intersection over union (FWIoU), Mean accuracy (MAcc) and accuracy (Acc). Combination 1 gathers Medieval data and 17th c., Combination 2 gathers 17th c. and 19th c. data.</caption><thead><tr><th></th><th>Medieval data</th><th>17<sup>th</sup> c.</th><th>19<sup>th</sup> c.</th><th>Combination 1</th><th>Combination 2</th></tr></thead><tbody><tr><th>MIoU</th><td>0.6017</td><td>0.1965</td><td>0.2092</td><td>0.4883</td><td>0.1964</td></tr><tr><th>FWIoU</th><td>0.7445</td><td>0.7794</td><td>0.7157</td><td>0.7537</td><td>0.7322</td></tr><tr><th>MAcc</th><td>0.9805</td><td>0.9846</td><td>0.9613</td><td>0.9886</td><td>0.9834</td></tr><tr><th>Acc</th><td>0.9805</td><td>0.9846</td><td>0.9613</td><td>0.9886</td><td>0.9834</td></tr></tbody></table>

VGSL specs (1 200→1 800)<fnref n="2" />, which significantly improves the results for both main categories (cf. table 3): intersection over union (i.e. zone detection) and accuracy (i.e. zone classification).

<table><caption>Table 3: Architecture 2.</caption><thead><tr><th></th><th>Medieval data</th><th>17<sup>th</sup> c.</th><th>19<sup>th</sup> c.</th><th>Combination 1</th><th>Combination 2</th></tr></thead><tbody><tr><th>MIoU</th><td>0.6127</td><td>0.1685</td><td>0.3662</td><td>0.5386</td><td>0.3269</td></tr><tr><th>FWIoU</th><td>0.7754</td><td>0.8002</td><td>0.7244</td><td>0.7845</td><td>0.7821</td></tr><tr><th>MAcc</th><td>0.9905</td><td>0.9886</td><td>0.9651</td><td>0.9917</td><td>0.9869</td></tr><tr><th>Acc</th><td>0.9905</td><td>0.9886</td><td>0.9651</td><td>0.9917</td><td>0.9869</td></tr></tbody></table>

Lower scores for medieval data and 19th c. can easily explained by the relatively smaller size of the training set and the greater complexity of the layout. If the results obtained with combined datasets are hard to interpret without the existence of a test for layout analysis, scores do not suffer a significant drop: the combination seems to be relatively efficient.

## Acknowledgements

This paper would not have been possible without the help of Thibault Clérice and Jean-Baptiste Camps (ENC), the SegmOnto technical and semantic working groups, the Kraken and eScriptorium development teams.

## References

* Gabay, S. (2021). *Formes Numérisées et Détection Unifiée des Écritures*. https://www.unige.ch/lettres/humanites-numeriques/index.php/?cID=188.

<footnote n="2">²the architecture of the second training is therefore the following1,1800,0,3 Cr7,7,64,2,2 Gn32 Cr3,3,128,2,2 Gn32 Cr3,3,128 Gn32 Cr3,3,256 Gn32 Lbx32 Lby32 Cr1,1,32 Gn32 Lby32 Lbx32.</footnote>
Gabay, S., J.-B. Camps, et al. (Sept. 2021). "SegmOnto: common vocabulary and practices for analysing the layout of manuscripts (and more)". In: *Proceedings of the 1st International Workshop on Computational Paleography, IWCP@ICDAR 2021*. 1st International Workshop on Computational Paleography IWCP. Lecture Notes in Computer Science. Lausanne (Switzerland): Springer.

Gabay, S., L. Rondeau Du Noyer, et al. (July 2020). "Quantifying the Unknown: How many manuscripts of the marquise de Sévigné still exist?" In: *Digital Humanities DH2020*. DH2020 Book of Abstracts. Ottawa, Canada: ADHO. URL: https://hal.archives-ouvertes.fr/hal-02898929 (visited on 11/23/2020).

Gabay, S., B. Topalov, et al. (Sept. 2021). "Automating ArtL@s - extracting data from exhibition catalogues". In: *Digital Humanities DH2020*. EADH2021 Book of Abstracts. Krasnoyarsk, Russia: ADHO. URL: REPLACE.

Khemakhem, M. (2020). "Standard-based Lexical Models for Automatically Structured Dictionaries". PhD thesis. Paris: INRIA.

Kiessling, B. (July 2019). "Kraken - an Universal Text Recognizer for the Humanities". In: *Digital Humanities 2019 Conference Abstracts*. Digital Humanities 2019 Conference. Utrecht, The Netherlands: Alliance of Digital Humanities Organizations (ADHO). URL: https://dev.clariah.nl/files/dh2019/boa/0673.html (visited on 03/16/2020).

Kiessling, B. et al. (Sept. 2019). "eScriptorium: An Open Source Platform for Historical Document Analysis". In: *2019 International Conference on Document Analysis and Recognition Workshops (ICDARW)*. 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW). Vol. 2, pp. 19–19. DOI: 10.1109/ICDARW.2019.10032.

Romary, L. et al. (2021). *CREMMA - Consortium pour la Reconnaissance d'Écriture Manuscrite des Matériaux Anciens*. https://www.dim-map.fr/projets-soutenus/cremma/.
# Appendices

## Examples

<figure><img src="image_5.png" /><figcaption>Figure 3: *Manuscrit, BnF fr. 412, f°21, 13th c.*</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 4: *Molière, L'Ecole des maris, Paris, G. de Luyne, 1661*</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 5: *Annuaire-almanach du commerce, de l'industrie..., 1898*</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 6: *Catalogue de l'exposition annuelle du musée de Rouen, 1860*</figcaption></figure>
Datasets

<table><caption>Table 4: 17th century prints dataset description (always with zones Main, Decoration, DropCapital, Numbering, RunningTitle, Signatures, Stamp and always with lines Default and DropCapitalLine).</caption><thead><tr><th>Prints' title and author</th><th>Gallica's ark</th><th>Date</th><th>No. of pages</th><th>Prints' specificities</th></tr></thead><tbody><tr><td>Lettres du Sieur Balzac, Jean-Louis Balzac</td><td>btv1b86262420</td><td>1624</td><td>57</td><td>None</td></tr><tr><td>Méduse, Claude Boyer</td><td>bpt6k311844g</td><td>1697</td><td>34</td><td>No Damage zone</td></tr><tr><td>Les caractères de Théophraste, Jean de la Bruyère</td><td>btv1b86070385</td><td>1688</td><td>36</td><td>No Damage zone</td></tr><tr><td>Histoire amoureuse des Gaules, Bussy-Rabutin</td><td>btv1b8623309s</td><td>1665</td><td>32</td><td>No RunningTitle zone</td></tr><tr><td>Le théâtre de P. Corneille, Pierre Corneille</td><td>bpt6k10403751</td><td>1664</td><td>47</td><td>None</td></tr><tr><td>Discours de la méthode, René Descartes</td><td>btv1b86069594</td><td>1637</td><td>14</td><td>None</td></tr><tr><td>La princesse de Clèves, Madame de La Fayette</td><td>btv1b8610820b</td><td>1678</td><td>53</td><td>No Margin zone</td></tr><tr><td>La Mariane, Tristan L'Hermite</td><td>bpt6k1511072f</td><td>1639</td><td>31</td><td>No Title zone</td></tr><tr><td>L'Escole des femmes, Molière</td><td>btv1b8610785b</td><td>1663</td><td>37</td><td>No Damage and Margin zones</td></tr><tr><td>George Dandin, ou le Mary confondu, Molière</td><td>btv1b8610793w</td><td>1669</td><td>66</td><td>No Margin zone</td></tr><tr><td>Dom García de Navarre, Molière</td><td>+Z258398909<sup>3</sup></td><td>1694</td><td>20</td><td>None</td></tr><tr><td>Statira, Nicolas Pradon</td><td>bpt6k8416272</td><td>1680</td><td>38</td><td>No Rubric line</td></tr><tr><td>Athénais, Nicolas Pradon</td><td>bpt6k857200c</td><td>1697</td><td>34</td><td>None</td></tr><tr><td>Les plaideurs, Jean Racine</td><td>btv1b8610811c</td><td>1669</td><td>37</td><td>No Rubric line</td></tr><tr><td>Oeuvres de Racine. Tome Premier, Jean Racine</td><td>bpt6k9905809</td><td>1676</td><td>19</td><td>No Damage zone</td></tr><tr><td>Oeuvres de Racine. Tome Second, Jean Racine</td><td>bpt6k990581p</td><td>1676</td><td>46</td><td>None</td></tr></tbody></table>

<table><caption>Table 5: Medieval dataset description</caption><thead><tr><th>Manuscript ID</th><th>Date</th><th>No. of pages</th><th>No. of columns</th><th>Running Title</th><th>Drop Capital</th></tr></thead><tbody><tr><td>BuF, Arsenal 3516</td><td>13th</td><td>10</td><td>4</td><td>No</td><td>Yes</td></tr><tr><td>BuF, ms fr. 22549</td><td>14th</td><td>3</td><td>3</td><td>No</td><td>Yes</td></tr><tr><td>BuF, ms fr. 24428</td><td>13th</td><td>20</td><td>2</td><td>No</td><td>Yes</td></tr><tr><td>BuF, ms fr. 412</td><td>13th</td><td>49</td><td>2</td><td>Yes</td><td>Yes</td></tr><tr><td>BuF, ms fr. 844</td><td>13th</td><td>18</td><td>2</td><td>No</td><td>Yes</td></tr><tr><td>Cologny, bodmer, 168</td><td>13th</td><td>22</td><td>2</td><td>No</td><td>Yes</td></tr><tr><td>Vaticane, Reg. Lat., 1616</td><td>14th</td><td>41</td><td>1</td><td>No</td><td>Yes</td></tr></tbody></table>
<table><caption>Table 6: Catalogs dataset description</caption><thead><tr><th>Prints' title</th><th>Type</th><th>Date</th><th>No. of pages</th><th>No. of columns</th><th>Other zones</th></tr></thead><tbody><tr><td>Annuaire-almanach du commerce, de l'industrie...</td><td>Annuaire</td><td>1898</td><td>50</td><td>2</td><td>Numbering</td></tr><tr><td>Exposition des œuvres de M. Courbet à l'École des Beaux Arts</td><td>Exhibition</td><td>1882</td><td>19</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue des œuvres exposées, Société des Indépendants</td><td>Exhibition</td><td>1892</td><td>5</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue des œuvres exposées, Société des Indépendants</td><td>Exhibition</td><td>1913</td><td>7</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue des peintures, sculptures et miniatures, Palais du Luxembourg</td><td>Exhibition</td><td>1923</td><td>5</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue des peintures, sculptures et miniatures, Palais du Luxembourg</td><td>Exhibition</td><td>1818</td><td>2</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de l'exposition des Beaux-Arts de Nancy</td><td>Exhibition</td><td>1867</td><td>5</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de l'exposition des Beaux-Arts de Nancy</td><td>Exhibition</td><td>1843</td><td>5</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de l'exposition des Beaux-Arts de Nancy</td><td>Exhibition</td><td>1849</td><td>5</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de l'exposition de la biennale de Paris</td><td>Exhibition</td><td>1892</td><td>5</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de l'exposition de la biennale de Paris</td><td>Exhibition</td><td>1961</td><td>5</td><td>1</td><td>Running Title</td></tr><tr><td>Catalogue de l'exposition de la biennale de Paris</td><td>Exhibition</td><td>1965</td><td>4</td><td>1</td><td>Numbering Running Title</td></tr><tr><td>Catalogue de l'exposition de la biennale de Paris</td><td>Exhibition</td><td>1969</td><td>5</td><td>1</td><td>Numbering Running Title</td></tr><tr><td>Catalogue de l'exposition du Musée des Beaux Arts de Rouen</td><td>Exhibition</td><td>1853</td><td>5</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de l'exposition du Musée des Beaux Arts de Rouen</td><td>Exhibition</td><td>1869</td><td>7</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de l'exposition du Musée des Beaux Arts de Rouen</td><td>Exhibition</td><td>1888</td><td>7</td><td>1</td><td>Numbering Running Title</td></tr><tr><td>Catalogue de la Biennale de Sao Paulo</td><td>Exhibition</td><td>1951</td><td>6</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de la Biennale de Sao Paulo</td><td>Exhibition</td><td>1972</td><td>5</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de l'exposition de la société des amis des arts de Strasbourg</td><td>Exhibition</td><td>1884</td><td>15</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de l'exposition de la société des amis des arts de Strasbourg</td><td>Exhibition</td><td>1895</td><td>5</td><td>1</td><td>Numbering Running Title</td></tr><tr><td>Catalogue de l'exposition de la Biennale de Venise</td><td>Exhibition</td><td>1905</td><td>3</td><td>1</td><td>Numbering Running Title</td></tr><tr><td>Catalogue de l'exposition de la Biennale de Venise</td><td>Exhibition</td><td>1920</td><td>5</td><td>1</td><td>Numbering Running Title</td></tr><tr><td>Revue des Autographes</td><td>Manuscripts</td><td>1870</td><td>5</td><td>1</td><td>Numbering</td></tr><tr><td>Revue des Autographes</td><td>Manuscripts</td><td>1871</td><td>6</td><td>1</td><td>Numbering</td></tr><tr><td>Revue des Autographes</td><td>Manuscripts</td><td>1873</td><td>4</td><td>1</td><td>Numbering</td></tr><tr><td>Revue des Autographes</td><td>Manuscripts</td><td>1877</td><td>4</td><td>1</td><td>Numbering</td></tr><tr><td>Revue des Autographes</td><td>Manuscripts</td><td>1880</td><td>2</td><td>1</td><td>Numbering</td></tr><tr><td>Revue des Autographes</td><td>Manuscripts</td><td>1881</td><td>2</td><td>1</td><td>Numbering</td></tr><tr><td>Revue des Autographes</td><td>Manuscripts</td><td>1883</td><td>5</td><td>1</td><td>Numbering</td></tr><tr><td>Revue des Autographes</td><td>Manuscripts</td><td>1885</td><td>2</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de Ventes de manuscrits Charavay</td><td>Manuscripts</td><td>1845</td><td>6</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de ventes de manuscrits Laverdet</td><td>Manuscripts</td><td>1856</td><td>4</td><td>1</td><td>Numbering Title</td></tr><tr><td>Catalogue de vente de manuscrits Charavay</td><td>Manuscripts</td><td>1857</td><td>6</td><td>1</td><td>Numbering Title</td></tr><tr><td>Catalogue de vente de manuscrits Charavay</td><td>Manuscripts</td><td>1866</td><td>7</td><td>1</td><td>Numbering Title Stamp</td></tr><tr><td>Catalogue de vente de manuscrits Bovet</td><td>Manuscripts</td><td>1887</td><td>14</td><td>1</td><td>Numbering Running Title Figure</td></tr><tr><td>Catalogue de vente de manuscrits Charavay</td><td>Manuscripts</td><td>1857</td><td>7</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de vente de manuscrits Bovet</td><td>Manuscripts</td><td>1899</td><td>6</td><td>1</td><td>Numbering</td></tr><tr><td>Catalogue de vente de manuscrits Kra</td><td>Manuscripts</td><td>1912</td><td>9</td><td>2</td><td>Numbering</td></tr><tr><td>Catalogue de vente de manuscrits Kra</td><td>Manuscripts</td><td>1919</td><td>8</td><td>1</td><td>Numbering</td></tr></tbody></table>