<figure><img src="image_1.png" /></figure>

HAL
open science

# FAIRer transcriptions: HTR-United and the possibility of a common for training data

Alix Chagué

► To cite this version:

Alix Chagué. FAIRer transcriptions: HTR-United and the possibility of a common for training data. Horizons of digital philology, Università degli Studi di Napoli Federico II; Université de Montréal, Apr 2024, Naples, Italy. hal-04697566

HAL Id: hal-04697566

https://inria.hal.science/hal-04697566v1

Submitted on 13 Sep 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# FAIRer transcriptions:
HTR-United and the possibility
of a common for training data

Alix Chagué - alix.chague@inria.fr

(Inria; UdeM; EPHE)

Naples - April 16, 2024

Horizons of digital philology
# Foreword

This presentation was partly inspired
from Thibault Clérice's introduction to
HTR with eScriptorium.
<figure><img src="image_3.png" /></figure>

# WHAT ARE WE TALKING ABOUT?
## SOME DEFINITIONS

<figure><img src="image_4.png" /></figure>
# What is text recognition?

- HTR : Handwritten Text Recognition

- OCR : Optical Character Recognition

## Key points to understand how it works:

1. Predict a textual content

2. From the image of a document (page)

3. Thanks to IA trained by humans

4. In an iterative process involving

a. human interventions

b. automatic/computation steps

<figure><img src="image_5.png" /></figure>
# Processing manuscript vs. prints

## OCR (for prints)

- <2% character error

- Commercial software (FineReader, Google Cloud Vision, etc.) or open source software (Tesseract 4)

- Language-specific generic models ready to be used, rely on fonts

## HTR (for manuscripts)

- 5-10% of character error

- Commercial software (Transkribus, ...) or open source software (Kraken / eScriptorium, ...)

- Require creating data to build a corpus-specific model
# Processing manuscript vs. prints

## OCR (for prints)

- <2% character error

- Commercial software (FineReader, Google Cloud Vision, etc.) or open source software (Tesseract 4)

- Language-specific generic models ready to be used, rely on fonts

## HTR (for manuscripts)

- 5-10% of character error

- Commercial software (Transkribus, ...) or open source software (Kraken / eScriptorium, ...)

- Require creating data to build a corpus-specific model

<figure><img src="image_6.png" /></figure>
# Creating data and training a model

<figure><img src="image_7.png" /></figure>
<figure><img src="image_8.png" /></figure>

MEASURING THE
ERROS IN
AUTOMATIC
TRANSCRIPTION

<figure><img src="image_9.png" /></figure>
# Measuring the success of a Text Recognition model

- Metrics are a useful (but imperfect) way to assess the success of HTR

- Understanding how we evaluate an HTR models helps understanding how it operates and what to expect from using HTR

- When assessing the performance of an HTR model:

- Character Error Rate

- Word Error Rate

- Computed by comparing a **reference text** and the model's **prediction** (ideally using the same segmentation)

- A model should not be tested on the data used to train it!
# Types of errors

Categories to compute metrics:

- Substitution : STEAM → STEAL

- Deletion : STEAM → TEAM

- Insertion : STEAM → STREAM

But there's more subtlety to it!

- Non-word errors (STEAN) vs. real-word errors

- Case-errors (StEAM) ; diacritic errors (STÉAM) ;

- Punctuation errors ; whitespace errors ; digits

- Many errors are caused by an imperfect segmentation (missing end of line, etc.)

$$CER = \frac{S + D + I}{N}$$

<figure><img src="image_10.png" /></figure>
<figure><img src="image_11.png" /></figure>

TRANSCRIBING IS
NOT SELF
EVIDENT

<figure><img src="image_12.png" /></figure>
# Text Recognition is a Computer Vision task

- Relies on visual clues

- Doesn't *understand* the text, only sees it

- Produces what it is trained to produce
# What is textual content?

## le Part Mirabean

Sauf le part Mirabean carle la Seme.
Et nos amans, faut-il qu'il m'ait sonnerie?
T'en j'en veuant, t'ayant après la peine.

Vieme le nuit, some l'heure,
tis j'aus i'en vait, j's demeure.

tis muirs dans les muirs, restan face à face
Tandis que son le part de son ha, paie
Da éternel, regardless l'an he n'axe.

Vieme le nuit, some l'heure
tis j'aus i'en vait, j's demeure.

t'amant i'en va comme cette eau courante,
t'amant i'en va; comme la vie et l'onte
Et comme l'Experance et violente!

Vieme le nuit, some l'heure
tis j'aus i'en vait, j's demeure

Pavent le j'aus et pavent les xemains,
Mitoys, paie, m'les amans revenuont;
Sauf le part Mirabean carle la Seme

Vieme le nuit, some l'heure
tis j'aus i'en vait, j's demeure!

15
# What is textual content?

- the main body of text (different paragraphs, title)

<figure><img src="image_13.png" /></figure>
# What is textual content?

- the main body of text (different paragraphs, title)

- the handwritten page number?

<figure><img src="image_14.png" /></figure>

## the Part Mirabeau

Sauf le part Mirabeau carle la Seme.
Et nos amans, faut-il qu'il m'ait sonnerie?

T'en j'en veuant, t'ayant, après la peine.

Vieme le mut, some l'heure,
tis j'aus i'en vait, j's demeure.

tis muirs dans les muirs, n'ont face à face
Tandis que son le part de son ha, paie
Da l'eternel, regardless l'an he n'laive.

Vieme le mut, some l'heure
tis j'aus i'en vait, j's demeure.

t'amant s'en va comme cette eau courante,
T'amant i'en va; comme la vie et l'onte
Et comme l'Experance et violente!

Vieme le mut, some l'heure
tis j'aus i'en vait, j's demeure

Pavent le j'aus et pavent les xemains,
M't'ouy, paie, m'les amours reviennent!

Sauf le part Mirabeau carle la Seme

Vieme le mut, some l'heure,
tis j'aus i'en vait, j's demeure!
# What is textual content?

- the main body of text (different paragraphs, title)

- the handwritten page number?

- the pre-printed page number?
# What is textual content?

- the main body of text (different paragraphs, title)

- the handwritten page number?

- the pre-printed page number?

- the manual page/inventory number?

<figure><img src="image_15.png" /></figure>

the Part Mirabean

Sauf le part Mirabean carle la Seme.
Et nos amans, faut-il qu'il m'ait sonnième !

T'en j'en veuant toujours après la peine.

Virome le mut, some l'heure,
tis j'aus i'en vait, j's demeure.

tis marais dans les mains, n'ont face à face
Tandis que son le part de son bras, paixé

Da éternel, regardless l'an he n'love.

Virome le mut, some l'heure
tis j'aus i'en vait, j's demeure.

t'amais s'en va comme cette eau courante,
t'amais i'en vait, comme la vie et l'onte

Et comme l'Experance et violente !

Virome le mut, some l'heure
tis j'aus i'en vait, j's demeure

Parcet le j'aus et parcet les xemains,
Mitoys, paixé, m'les amais revenuamt!

Sauf le part Mirabean carle la Seme

Virome le mut, some l'heure,
tis j'aus i'en vait, j's demeure !
# What is textual content?

- the main body of text (different paragraphs, title)

- the handwritten page number?

- the pre-printed page number?

- the manual page/inventory number?

- the text in the library stamp?

the Part Mirabean

Sauf le part Mirabean carle la Seme.
Et nos amans, faut-il qu'il m'ait sonnerie?

T'en j'en veuant, j'en j'en veuant, j'en demeure.

Vieme le mut, some l'heume,
tis j'ans i'en vaut, j'en demeure.

tis marais dans les marais, n'ont pas face à face
Tandis que son le part de son ha, paixé.

Da éternel, regardless l'an he n'pace.

Vieme le mut, some l'heume,
tis j'ans i'en vaut, j'en demeure.

t'aman s'en va comme cette eau courante,
t'aman s'en va, comme la vie et l'onte

Et comme l'Experance et violente !

Vieme le mut, some l'heume
tis j'ans i'en vaut, j'en demeure

Pavent le jans et pavent les xemains,
Mitoys, paixé, m'les amans revenuont!

Sauf le part Mirabean carle la Seme

Vieme le mut, some l'heume,
tis j'ans i'en vaut, j'en demeure !

<figure><img src="image_16.png" /></figure>
# What is textual content?

- the main body of text (different paragraphs, title)

- the handwritten page number?

- the pre-printed page number?

- the manual page/inventory number?

- the text in the library stamp?

- the partially illegible text?

the Part Mirabean

Sauf le part Mirabean carle la Seme.
Et nos amans, faut-il qu'il m'ait sonnerie?

T'en j'en veu, tout jeux, après la peine.

Virome le mut, some l'heure,
tis j'ans i'en vait, j's demene.

t's marais dans les mains, n'ont face à face
Tandis que son le part de son bras, paix
Da l'eternel, regardless l'an he n'love.

Virome le mut, some l'heure
tis j'ans i'en vait, j's demene.

t'aman s'en va comme cette eau courante,
t'aman s'en va, comme la vie et l'onte

Et comme l'Experance et violente !

Virome le mut, some l'heure
tis j'ans i'en vait, j's demene

Pavent le jans et pavent les xemains,
Mitoys, paix, m'les amans revenuont!

Sauf le part Mirabean carle la Seme

Virome le mut, some l'heure,
tis j'ans i'en vait, j's demene !
# What is textual content?

- the main body of text (different paragraphs, title)

- the handwritten page number?

- the pre-printed page number?

- the manual page/inventory number?

- the text in the library stamp?

- the partially illegible text?

- the credit line from the online library?

1

the Part Mirabean

15

Sauf le part Mirabean carle la Seme.
Et nos amans, faut-il qu'il m'ait sonneme?
T'en j'en veuant t'oyans apres la peine.

Vieme le mut, some l'heume,
tis j'ans i'en vait, j's dememe.

tis marais dans les mains, n'ont face à face
Tandis que son le part de son bras, paie
Da l'eternel, regardless l'an he n'la voie.

Vieme le mut, some l'heume
tis j'ans i'en vait, j's dememe.

t'amant s'en va comme cette eau courante,
t'amant i'en va; comme la vie et l'onte
Et comme l'Experience et violente!

Vieme le mut, some l'heume
tis j'ans i'en vait, j's dememe.

Parent le jans et parent les xemains,
Mitoys, parei, w'les amans reviennent,
Sauf le part Mirabean carle la Seme.

Vieme le mut, some l'heume,
tis j'ans i'en vait, j's dememe.!
# What is textual content?

- the main body of text (different paragraphs, title)

- the handwritten page number?

- the pre-printed page number?

- the manual page/inventory number?

- the text in the library stamp?

- the partially illegible text?

- the credit line from the online library?

- underlines? paragraph brackets?

<figure><img src="image_17.png" /></figure>
## How do we transcribe?

- A **Mademoiselle** Louise Trudelle

- A **Mlle** Louise Trudelle

- A **Mlle** Louise Trudelle

- A **Mlle** Louise Trudelle

- A **M^lle** Louise Trudelle

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

- Corresponding **Sec'y**.

- Corresponding **Secretary**.
## How do we transcribe?

<figure><img src="image_20.png" /></figure>

- the **changed/s** charges

- the charges

- the **changed/s** charges

- the [[**changed/s**]] charges

- the [**||**] charges

- the XXX charges
# Key takeaways

Transcribing is not self-evident!

- Transcription paradigm (regularized/graphemic/graphetic)

- Choice of characters to render different phenomenons

- Transcription software usually don't include text formatting

In any case: document your choices, even more so if you are a team of 1 transcribers!!
<figure><img src="image_21.png" /></figure>

# TWO EXAMPLES OF ATTEMPTS TO HOMOGENIZE PRACTICES

<figure><img src="image_22.png" /></figure>
# Community: transcription rules

Harmonization of data allows to exchange data and models!

How to ensure that?

- Define transcription methods adapted to a research problem AND to machine learning capacities

- Define the targeted precision/accuracy

- Define a set of characters to render the source and document choices

- Use recommendations from existing projects

- **CATMuS** : https://catmus-guidelines.github.io

<figure><img src="image_23.png" /></figure>
# Comunity: layout description

SegmOnto provides guidelines and a controlled vocabulary to describe zones and lines on a page!

- **Lines**: DefaultLine, InterlinearLine

- **Pages**: MainZone, MarginZone, DropCapitalZone

- **Customization syntax**: Name:subtype#number

- **Documentation**: https://segmonto.github.io/

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>
<figure><img src="image_26.png" /></figure>

# WHAT SOLUTION OFFERS HTR-UNITED?

<figure><img src="image_27.png" /></figure>
# Why is HTR-United important?

- Need to rely on pre-existing models and pre-existing data

- There is no centralized data warehouse for (gold) HTR datasets

- They are rarely "FAIR":

- Hard to **Find** and not always **Accessible**

- uncertain formats and varying annotations rules

- unclear **Reuse** conditions

<figure><img src="image_28.png" /></figure>

Findable

<figure><img src="image_29.png" /></figure>

Accessible

<figure><img src="image_30.png" /></figure>

Interoperable

<figure><img src="image_31.png" /></figure>

Reusable

https://www.go-fair.org/
fair-principles/
# Make your data findable!

- Emphasize key aspect of sharing data:

- Data quality and use of standards

- Documentation

- Citation

- Reusability

<figure><img src="image_32.png" /></figure>
# HTR-United in details

- Editors: Alix Chagué & Thibault Clérice

- A catalog browsable by humans and by machines

- Low tech environment for easy maintenance

- An ecosystem of recommendations and tools to ensure data quality

- A catalog fed by data creators

- Data is published wherever they want

- A form is filled online to create a structured description

- The description is reviewed by editors, then added to the catalog

- HTR-United focuses on datasets, not on transcription models
# Content of HTR-United (as of April 2024)

- 83 datasets created by at least 38 different projects

- 21 languages (a lot of French (43) and Latin (18)) for 8 scripts (mostly Latin)

- Handwritten *and* printed documents, mixed or not

- A handwriting-period coverage spanning from 800 to 2023

- Created with at least 6 different software (eScriptorium, Kraken, Transkribus ...)

- more than 43M characters, 1M lines or 11K images
Content of HTR-United (as of April 2024)

<figure><img src="image_33.png" /></figure>
# Content of HTR-United (as of April 2024)

<figure><img src="image_34.png" /></figure>
Content of HTR-United (as of April 2024)

<figure><img src="image_35.png" /></figure>
Content of HTR-United (as of April 2024)

<figure><img src="image_36.png" /></figure>
Let's browse the catalog!

https://htr-united.github.io/
Key takeaway

- HTR technology gains from users forming a community

- learn faster

- avoid mistakes

- homogenize practices

- HTR-United intends to facilitate the creation of this community

- reliable data

- community-organized tool

- transparent process
Thank you!