# ATR: What can eScriptorium do for you?

Alix Chagué, Floriane Chiffoleau

## ► To cite this version:

Alix Chagué, Floriane Chiffoleau. ATR: What can eScriptorium do for you?. 2023. [hal-04247827](https://hal.science/hal-04247827v1).

HAL Id: hal-04247827

https://hal.science/hal-04247827v1

Preprint submitted on 18 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# ATR: What can eScriptorium do for you?

Alix Chagué, Floriane Chiffoleau

From Source to Full Text: Workshop on Using Automatic Text Recognition (ATR)

September 8th 2023

<figure><img src="image_2.png" /></figure>

ALMAnaCH project-team

Inria

Le Mans
Université
Summary of
the
presentation

1. Introduction to **ATR software**

2. Guided tour of **eScriptorium**

3. The essence of a good model: **training data**

4. Fantastic **models** (and where to find them)

5. **Predict and assess**: the cornerstone of ATR

6. **Exercise**: Getting acquainted with
**eScriptorium**

7. The **eScriptorium documentation**: a helping hand for users

8. **Resources**
# Introduction to ATR software
Introduction of
the ATR
software

- ATR or Automatic Text Recognition is the process of transforming a digitized document into a machine-readable text, through the use of **segmentation** and **text recognition** tools

- Various software for ATR with some specificities:

- Proprietary or open source

- Printed documents, handwritten documents or both

- Charged, freemium, free

- Example of open-source, semi-free OCR engine:

- Tesseract (no interface)

- Transkribus (interface) + PyLaia (transcription engine)

- eScriptorium (interface) + Kraken (transcription engine)
# Guided tour of eScriptorium
# Guided tour of eScriptorium

* eScriptorium is an open-source software (web application), for transcribing textual documents, developed by the research team Scripta (PSL).

* It provides a **graphical user interface** (GUI) for:

* **document management**,

* **layout annotation (aka segmentation)**,

* **transcribing** (manually or automatically),

* and **training models**.

* and more!

* It uses **Kraken** (Benjamin Kiessling - PSL) as a transcription engine. Kraken is language-agnostic and fully open source.

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>
<figure><img src="image_5.png" /></figure>

# eScriptorium

eScriptorium: A Digital Text Production Pipeline for Print and Handwritten Texts using machine learning techniques.

<figure><img src="image_6.png" /></figure>

## Automatic Transcription

Apply OCR/HTR to images of printed and handwritten documents using shared open models.

<figure><img src="image_7.png" /></figure>

## Manual transcription

Make use of an ergonomic user interface leveraging modern browser technology to edit segmentations and transcriptions.

<figure><img src="image_8.png" /></figure>

## Train Models

Create new models or finetune existing ones to improve automatic recognition.

<figure><img src="image_9.png" /></figure>

## Import/Export

Import and export models and texts transcriptions in a variety of formats. Access data through a full REST API.
<figure><img src="image_10.png" /></figure>

Sandbox project

Documents

Reports

Get overall metrics on the project's content

Create new Document

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

Filters
[ ] demo

Dynamically
filter the list of
documents

Select

Preview

Name

Owner

Last modified

Image count

Tags

Actions

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /><figcaption>Gallery of Fashion (01/1800)</figcaption></figure>

demo_tutorial

Sept. 1, 2023

5 images.

<figure><img src="image_15.png" /></figure>

demo

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /><figcaption>Another project</figcaption></figure>

demo_tutorial

Sept. 4, 2023

8 images.

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

**Projects** contain **documents**,
which contain images
(or **document parts**)

Edit the document
(transcribe, add images,
add metadata, etc...)
<figure><img src="image_23.png" /></figure>

eScriptorium

Home

Contact

My Projects

My Models

Hello Demo

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

Description

Ontology

Images

Edit

Models

Reports

Guillaume Apollinaire. Alcools.

<figure><img src="image_26.png" /></figure>

Drop images here or click to upload.

<figure><img src="image_27.png" /></figure>

Image import

Access to my
other projects

Access to my created
or uploaded models

Select all

Unselect all

Selected 0/141

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>

Import

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

Export

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

100%

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>

<figure><img src="image_49.png" /></figure>

<figure><img src="image_50.png" /></figure>

Train

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

<figure><img src="image_54.png" /></figure>

<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>

Binarize

<figure><img src="image_58.png" /></figure>

Segment

<figure><img src="image_59.png" /></figure>

Transcribe

<figure><img src="image_60.png" /></figure>

<figure><img src="image_61.png" /></figure>

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

Status of the segmentation and transcription
<figure><img src="image_64.png" /></figure>

eScriptorium

Home

Contact

My Projects

My Models

Hello Demo

<figure><img src="image_65.png" /></figure>

<figure><img src="image_66.png" /></figure>

Description

Ontology

Images

Edit

Models

Reports

Guillaume Apollinaire. Alcools.

Page 5 - 4_ca4b2_default.jpg - (2618x3308) - 860.91 KB

manual

<figure><img src="image_67.png" /></figure>

<figure><img src="image_68.png" /></figure>

<figure><img src="image_69.png" /></figure>

<figure><img src="image_70.png" /></figure>

<figure><img src="image_71.png" /></figure>

<figure><img src="image_72.png" /></figure>

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>

<figure><img src="image_75.png" /></figure>

<figure><img src="image_76.png" /></figure>

<figure><img src="image_77.png" /></figure>

<figure><img src="image_78.png" /></figure>

<figure><img src="image_79.png" /></figure>

<figure><img src="image_80.png" /></figure>

<figure><img src="image_81.png" /></figure>

Different
transcriptions
available

kraken:model_24

kraken:model_41

kraken:model_5

kraken:generic_lectaurep_26

kraken:lectaurep_base_best

kraken:generic_lectaurep_26_v3

<figure><img src="image_82.png" /></figure>

✓ manual

Segmentation

Transcription

1

<figure><img src="image_83.png" /></figure>

<figure><img src="image_84.png" /></figure>

<figure><img src="image_85.png" /></figure>

<figure><img src="image_86.png" /></figure>

<figure><img src="image_87.png" /></figure>

<figure><img src="image_88.png" /></figure>
ATR software are similar because the workflow is close

<figure><img src="image_89.png" /><figcaption>Transkribus App (09/2023)</figcaption></figure>
# The essence of a good model: training data
# The essence of a good model: training data

* What is training data ? → Sets of images and their exact transcription (for text recognition) or layout structure (for layout recognition), used to train models

* There are two categories of training data:

**GOLD CORPUS**

manual annotation (layout or text) or corrected prediction

**SILVER CORPUS**

automatic prediction (with a model) without correction
The essence of
a good model:
training data

Three formats for the training data:

Sets of ALTO XML files and their
images

Sets of PAGE XML files and their
images

Sets of text line images and their
transcription

In each case, same filename, different
extension (xml/txt, jpg/png/tiff)
## Example of a line from an ALTO XML

```xml
<TextLine ID="eSc_line_3b2a0fba"
         BASELINE="703 102 1507 108" HPOS="701.0" VPOS="73.0" WIDTH="806.0" HEIGHT="69.0">
    <Shape><Polygon POINTS="703 102 701 135 1052 129 1080 139 1082 139 1084 139 1085 139 1106 129 1124 140 1126 140 1128 140 1129 140 1144 140 1267 133 1413 142 1432 142 1433 142 1439 142 1481 133 1501 142 1507 108 1505 76 703 73 703 102"/></Shape>
    <String CONTENT="Clermont-Créans (Sarthe) 15 Aout 1914" HPOS="701.0" VPOS="73.0" WIDTH="806.0" HEIGHT="69.0"></String>
</TextLine>
```

## Example of a line from a PAGE XML

```xml
<TextLine id="eSc_line_3b2a0fba" >
    <Coords points="703,102 701,135 1052,129 1080,139 1082,139 1084,139 1085,139 1106,129 1124,140 1126,140 1128,140 1129,140 1144,140 1267,133 1413,142 1432,142 1433,142 1439,142 1481,133 1501,142 1507,108 1505,76 703,73 703,102"/>
    <Baseline points="703,102 1507,108"/>
    <TextEquiv>
        <Unicode>Clermont-Créans (Sarthe) 15 Aout 1914</Unicode>
    </TextEquiv>
</TextLine>
```

Clermont-Créans (Sarthe) 15 Aout 1914

Clermont-Créans (Sarthe) 15 Aout 1914

Set of a text line image and its transcription
# The essence of a good model: training data

* Additionally to your own training data, you can find other training data online, such as thanks to the *HTR-United* catalog

* It contains metadata on training datasets available for the creation of transcription or segmentation models

<figure><img src="image_90.png" /></figure>

<figure><img src="image_91.png" /></figure>
Fantastic models (and where to find them)
# Fantastic models (and where to find them)

<figure><img src="image_92.png" /></figure>

What is a model? → It is a file that has been trained to recognize certain types of patterns.

<figure><img src="image_93.png" /></figure>

A model can be developed for 2 types of process

<figure><img src="image_94.png" /></figure>

**Layout recognition** → Process that determines the constituents of an image and locates the regions and lines of the document where data have been written

<figure><img src="image_95.png" /></figure>

**Text recognition** → Process that recognizes the characters in document lines and produces a machine-readable text

<figure><img src="image_96.png" /></figure>

Specific to the software that produced them (no standardization)

<figure><img src="image_97.png" /><figcaption>blla.mlmodel</figcaption></figure>

<figure><img src="image_98.png" /><figcaption>CREMMA-Medie
val_best_V1-0-0.
mlmodel</figcaption></figure>
## Fantastic models (and where to find them)

### Where to find them ?

<figure><img src="image_99.png" /></figure>

#### For Kraken and eScriptorium

<figure><img src="image_100.png" /></figure>

- In the tab "My models" on eScriptorium → it contains public models, available for every user and the models the user uploaded/trained themself

<figure><img src="image_101.png" /></figure>

- In the repository "OCR/HTR model" created on Zenodo

<figure><img src="image_102.png" /></figure>

#### For other software

<figure><img src="image_103.png" /></figure>

- Inside software, but not downloadable (Transkribus, FineReader, etc.) (most common case)

<figure><img src="image_104.png" /></figure>

- Along with software source code (Tesseract)
<figure><img src="image_105.png" /></figure>

<figure><img src="image_106.png" /></figure>

My Models

Upload a model

<table><thead><tr><th></th><th>Role</th><th>Script</th><th>Size</th><th>Trained from</th><th>Training Status</th><th>Accuracy</th><th>Errors</th><th>Right</th></tr></thead><tbody><tr><td>generic_lectau_nfd_rs_26</td><td>Recognize</td><td></td><td>15.4 MB</td><td></td><td>✓</td><td>82.5%</td><td>-</td><td>User</td></tr><tr><td>Generic CREMMA Model for Medieval Manuscripts (DOI 10.5281/zenodo.7631619)</td><td>Recognize</td><td>Latin</td><td>21.7 MB</td><td>Q</td><td>✓</td><td>91.3%</td><td>-</td><td>Public</td></tr><tr><td>GalliCorpora+ (French Early Modern Print) (DOI 10.5281/zenodo.7410359)</td><td>Recognize</td><td>Latin</td><td>15.5 MB</td><td>Q</td><td>✓</td><td>96.6%</td><td>-</td><td>Public</td></tr><tr><td>HTR-United-Manu McFrench V1 (DOI 10.5281/zenodo.6657808)</td><td>Recognize</td><td>Latin</td><td>15.4 MB</td><td>Q</td><td>✓</td><td>90.6%</td><td>-</td><td>Public</td></tr><tr><td>HTR-United-Manu McFrench V3 (DOI 10.5281/zenodo.6657808)</td><td>Recognize</td><td>Latin</td><td>21.7 MB</td><td>Q</td><td>✓</td><td>90.3%</td><td>-</td><td>Public</td></tr><tr><td>finetune_modelpec_9360_NFC</td><td>Recognize</td><td></td><td>16.6 MB</td><td></td><td>✓</td><td>93.6%</td><td>-</td><td>Owner</td></tr><tr><td>modelpec_9378_NFC</td><td>Recognize</td><td></td><td>16.6 MB</td><td></td><td>✓</td><td>93.8%</td><td>-</td><td>Owner</td></tr></tbody></table>

<figure><img src="image_107.png" /></figure>

<figure><img src="image_108.png" /></figure>

<figure><img src="image_109.png" /></figure>

<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

<figure><img src="image_112.png" /></figure>

<figure><img src="image_113.png" /></figure>
<figure><img src="image_114.png" /></figure>

# OCR/HTR model repository

## Recent uploads

<figure><img src="image_115.png" /></figure>

<figure><img src="image_116.png" /></figure>

### Transcription model for Lucien Peraire's handwriting (French, 20th century)

<figure><img src="image_117.png" /></figure>

Alix Chagué;

This model was trained on the peraire-ground-truth dataset (v.2.0.0) and using Manu McFrench as a base model. The peraire-ground-truth dataset contains documents written in French by Lucien Peraire in the late 1920s and in the late 1960s. For more details on the dataset and the transcription guidelines

Uploading on July 28, 2023

View

<figure><img src="image_118.png" /></figure>

### HTR model for German manuscripts trained from several datasets

<figure><img src="image_119.png" /></figure>

Stefan Weil;

This German handwriting recognition model has been trained with ground truth from different sources. It is based on a

View

<figure><img src="image_120.png" /></figure>

### New upload

## OCR/HTR model repository

A repository of OCR/HTR models with metadata assisting in model selection.

**Curated by:**

bkiessling

**Curation policy:**

We're accepting any inclusion request with a valid metadata file. This will eventually be automated when the Zenodo REST API supports community curation.

**Created:**

February 26, 2019

**Harvesting API:**

OAI-PMH Interface
<figure><img src="image_121.png" /></figure>
<figure><img src="image_122.png" /></figure>
Predict and assess: the cornerstone
of ATR
# Predict the transcription

<figure><img src="image_123.png" /></figure>

A prediction is the act of using a model to generate (or predict) a layout recognition or a text recognition using an image and a segmentation/transcription model.

<figure><img src="image_124.png" /><figcaption>Layout</figcaption></figure>

<figure><img src="image_125.png" /><figcaption>Text</figcaption></figure>

<figure><img src="image_126.png" /></figure>
Assess the transcription

How to assess?

Eyeballing the result (Can I read it?
Does it look gibberish?)

Using metrics

CER : Character Error Rate

WER : Word Error Rate

How to go deeper?

KaMI-Lib or CERberus: ways to nuance
the evaluation (caps, digits, diacritics,
etc.)

How to obtain your metrics (from the
example below)

CER → 25 characters, 4 incorrect
characters, CER = 16% (4/25 * 100)

WER → 5 words, 2 incorrect words,
WER = 40% (2/5 * 100)

LA SIGNATURE de la PAIX .

LA SIGTCNATURBE de la PAITX .
Exercise: Getting acquainted with eScriptorium
Exercise: Getting
acquainted with
eScriptorium

1. Go to the website: [escriptorium.inria.fr](https://escriptorium.inria.fr/)

2. Login in eScriptorium:

- Username: apinche_formation

- Password: training1234

3. Go to the project: Pinche_FormationDHI

4. Create a new document

- Name: "workshopdhi_" + your last name
  → example: "workshopdhi_chiffoleau"

- Parameters: "Latin", "Baseline" and "Left to right"

5. Take a look at the models already present on the account

- "My models"
# The eScriptorium documentation: a helping hand for users
# The eScriptorium documentation: a helping hand for users

* The tutorial aims to provide a **set of (written) walkthroughs detailing eScriptorium's features**, for example:

* Managing projects and collections of documents,

* Learning how to segment, annotate and transcribe a text image,

* Instructions for training models.

The tutorial is available at https://escriptorium-tutorial.readthedocs.io

All collaborations are welcomed: https://github.com/elix-tz/escriptorium-tutorial

<figure><img src="image_127.png" /></figure>
<figure><img src="image_128.png" /></figure>

EScriptorium
DOCUMENTATION

Search docs

Home

Contribute to the documentation

About this documentation

QUICK-START

Quick-start

FAQ

WALKTHROUGH

Import data

Automatic prediction

Manual segmentation

Manual transcription

Manual annotation

<figure><img src="image_129.png" /></figure>

Home

<figure><img src="image_130.png" /></figure>

Edit on GitHub

Next

<figure><img src="image_131.png" /></figure>

eScriptorium is a web application offering a workspace to manage the various steps of a transcription campaign. These steps can involve manual or automatic processes and be applied to printed documents or handwritten ones. The application uses Kraken as a segmentation/transcription engine. Since its beginning in 2019, the SCRIPTA PSL research group is responsible for its creation and development.

You can find more information about eScriptorium and the context of its production in:

* Stokes, P., B. Kiessling, D. Stökl Ben Ezra, R. Tissot, and E. H. Gargem. "The EScriptorium VRE for Manuscript Cultures." Edited by Claire Clivaz and Garrick V. Allen. *Classics@ Journal, Ancient Manuscripts and Virtual Research Environments, 18* (2021). https://classics-at.chs.harvard.edu/classics18-stokes-kiessling-stokl-ben-ezra-tissot-gargem/.

The goal of this documentation is to facilitate learning how to use the application as beginners or advanced users.

Quick start
Resources
Resources

☐ eScriptorium: https://escriptorium.inria.fr/

☐ eScriptorium documentation:
https://escriptorium.readthedocs.io

☐ Kraken: https://kraken.re/main/index.html

☐ OCR/HTR models:
https://zenodo.org/communities/ocr_models

☐ CERberus:
https://github.com/WHaverals/CERberus

☐ HTR-United: https://htr-united.github.io/

☐ KaMILib:
https://huggingface.co/spaces/Iterriel/kami-app

☐ PyLaia: https://github.com/jpuigcerver/PyLaia

☐ Tesseract:
https://github.com/tesseract-ocr/tesseract

☐ Transkribus: https://app.transkribus.eu/
Thank you for your attention

Any questions ?

Contact: alix.chague[at]inria.fr
floriane.chiffoleau[at]inria.fr