# An accessible and transparent pipeline for publishing historical egodocuments

Alix Chagué, Floriane Chiffoleau

► To cite this version:

Alix Chagué, Floriane Chiffoleau. An accessible and transparent pipeline for publishing historical egodocuments. WPIP21 - What's Past is Prologue: The NewsEye International Conference, Mar 2021, Virtual, Austria. hal-03173038

HAL Id: hal-03173038

https://hal.science/hal-03173038

Submitted on 18 Mar 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# An accessible and transparent pipeline for publishing historical egodocuments

What's Past is Prologue : The NewsEye International Conference
17 March 2021

DAHN Project
Alix Chagué - Floriane Chiffoleau
Research and Development Engineers at Inria

<figure><img src="image_2.png" /></figure>
# WHY A PIPELINE ?

* **Black box:**

➤ does everything we want; but no interception of the data, no customization;

<figure><img src="image_3.png" /></figure>

* **Scattered toolbox:**

➤ risk of obsolescence and discontinuity; portability of data not guaranteed;

<figure><img src="image_4.png" /></figure>

* **Pipeline:**

➤ clear steps, compatible softwares, extensive documentation

➤ data integrity, user in control

<figure><img src="image_5.png" /></figure>
# THE DAHN PROJECT

## "Dispositif de soutien à l'Archivistique et aux Humanités Numériques"

* **Members:** Inria + EHESS + University of Le Mans | MESRI;

* Scientific digital edition program for archival corpus;

* **Goal:** facilitate the digitization of data extracted from archival collections and their dissemination to the public in the form of digital documents and/or as online editions

* ALMANACH in charge of developing and/or enhancing the tools that are part of the pipeline.
# THE CORPUS: PAUL D'ESTOURNELLES DE CONSTANT CORRESPONDENCE

## ❖ The context:

* War reports

* Exchange of opinions between two pacifists

## ❖ The corpus, a perfect test subject:

* Typewritten documents: easier OCR, no HTR, faster work

* Voluminous corpus: 1500 letters, 430 from the war period alone (from April 15, 1914 to November 19, 1918), one-to-several tens pages

<figure><img src="image_6.png" /><figcaption>Distribution of pages per letter</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>The sender: Paul d'Estournelles de Constant</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>The recipient: Nicholas Murray Butler</figcaption></figure>
# THE CORPUS: DIGITIZATION OF THE CORRESPONDENCE

An heterogeneous corpus of digitizations:

<figure><img src="image_9.png" /></figure>

* Manual photo shooting of the corpus, by a researcher of the project, prior to its creation: medium quality images, case of blurred lines, cropped images and/or incomplete pages

<figure><img src="image_10.png" /></figure>

* Institutional and exhaustive digitization campaign with proper material by the institution where the corpus is kept: high quality images

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>
THE PIPELINE

<figure><img src="image_14.png" /></figure>
# THE OCR ENGINE: A CAPSTONE

<figure><img src="image_15.png" /></figure>

## Powered by eScriptorium and Kraken

* eScriptorium: web interface for collaborative and automatic transcription projects;

* Historical ties between ALMAnaCH and SCRIPTA PSL (project team behind eScriptorium);

* Intuitive platform and user-friendly on multiple features;

* Entirely open-source and up-to-date formats;

<figure><img src="image_16.png" /></figure>

## Guiding principles:

* Every step of the pipeline relies on open-source software or services;

* Compliance with the FAIR principles.

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

PSL ⭐
UNIVERSITÉ PARIS

SCRIPTA
hist·ire et pratiques de l'écrit

F

indable

A

ccessible

I

nteroperable

R

eusable

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>
THE PIPELINE STEP BY STEP: SEGMENTATION

* Segmentation and layout annotation with a system of lines and zones tagging coupled with an ontology;

* Possible to build your own modelization and pass it on to a model;

* Integration of the TEI framework early-on;

* Participation of few members of the DAHN project in SegmOnto, a working group aiming at creating a general TEI-based ontology for HTR projects.

<figure><img src="image_23.png" /></figure>

Example of an
image annotated
for Segmonto
# THE PIPELINE STEP BY STEP: TRANSCRIPTION

<figure><img src="image_24.png" /></figure>

## Training of a transcription model

<figure><img src="image_25.png" /></figure>

Ground truth for the first model: 84 pages (10 letters) → not efficient enough

<figure><img src="image_26.png" /></figure>

Construction of a dataset more representative: focus on specific parts of text (capital letters, numbers, narrow text) - about 100 of additional pages → 92,74% accuracy

<figure><img src="image_27.png" /></figure>

Fine-tuning a pre-existing model (“tapuscorpus”, trained on varied typewritten docs): resulting model potentially more capable of generalization → 93,90% accuracy

<figure><img src="image_28.png" /></figure>

## A model capable to render meaningful formatting of the text:

<figure><img src="image_29.png" /></figure>

Handwritten annotation → no recognition required, a double ‘££’ for every handwritten word

<figure><img src="image_30.png" /></figure>

Deletions → a ‘€’ at the start of the deletion and at the end, even if when spanning over several words

<figure><img src="image_31.png" /></figure>

un €stimulant€ de misè

<figure><img src="image_32.png" /></figure>

££ ££ ££ ££ ££ ££ ££
# THE PIPELINE STEP BY STEP: POST-PROCESS CORRECTIONS

* Export/import with eScriptorium

* Spell-checking:

* Python script and `pyspellchecker` module

* Generation of Python dictionaries for correction

* Correction of texts

* Python script and find/replace

* Manual correction of remaining errors

* Goal → implementation of this method in the eScriptorium API

<figure><img src="image_33.png" /></figure>
# THE PIPELINE STEP BY STEP: ENCODING

## Header

* General encoding of the header, with manuscript description for the corpus

* Correspondence data in the `<profileDesc>`

## Body

* Default text structure (titles, paragraphs), changes in text (addition and deletion), difficulties of the corpus (unclear, gaps) and named entities (person, place, organization)

* Correspondence part:

* Easy tags (opener, closer, address, signatures, etc.)

* Assistance available for difficult tags: TEI mailing-list, [Correspondence Special Interest Group](#)

## Helping tools for the encoding

* Script for the encoding of metadata in the header, using an inventory with essential information

* Script for the encoding of the body, using regular expressions
# THE PIPELINE STEP BY STEP: PUBLICATION

<figure><img src="image_34.png" /></figure>

## TEI Publisher

<figure><img src="image_35.png" /></figure>

From exist-db, an open source software project for databases built on XML technology

<figure><img src="image_36.png" /></figure>

Transformation files (ODD) and templates (HTML)

<figure><img src="image_37.png" /></figure>

Presentation by collections

<figure><img src="image_38.png" /></figure>

## Application for ego documents

<figure><img src="image_39.png" /></figure>

Text and image opposed

<figure><img src="image_40.png" /></figure>

Metadata available with a specific button

<figure><img src="image_41.png" /></figure>

ODD containing all the tags related to ego documents

<figure><img src="image_42.png" /></figure>

One corpus = one collection

<figure><img src="image_43.png" /></figure>

tei
Publisher

existdb

<figure><img src="image_44.png" /></figure>

### Collection de corpus d'égodocuments

#### Documents

Trier par

Titre

Filtrer selon

Titre

Filter

<figure><img src="image_45.png" /></figure>

SUPPRIMER

<figure><img src="image_46.png" /></figure>

#### Correspondance de d'Estournelles de Constant

Ce dossier contient le corpus et les index de la correspondance de Paul d'Estournelles de Constant, ainsi que l'histoire du corpus et des informations à propos du projet.

<figure><img src="image_47.png" /></figure>

#### Correspondance des Intellectuels Berlinois

Ce dossier contient le corpus et les index de la correspondance des intellectuels berlinois de 1800 à 1830.
FAVORING THE USE OF THE PIPELINE: DOCUMENTATION

* Initial documentation provided with the software, standards and framework

* Self-produced documentation

* Comprehensive series of guidelines for the encoding of ego documents (ODD)

* Jupyter notebooks and markdown files commenting the use of Python scripts (encoding and correction)

* Provision of the data from the steps of the pipeline on the dedicated repository
GENERALIZATION AND MODULARITY

* Corpus of the _Berlin intellectuals_, correspondence of the start of the 19th century

* Modularity: Plug data in without following the pipeline from the beginning

* Generalization: No limit of space and time for the encoding and publication

* Sustainability

* Dissemination of the data in useful formats

* Ground truth available in _HTR-United_

<figure><img src="image_48.png" /></figure>
TO CONCLUDE: KEY TAKEAWAYS

* Scenario fully documented and based on open-source and standards to avoid blackbox and/or scattered toolbox

* Adaptability of the input and output formats of softwares

* Prerequisites to use the pipeline:

* Digitized images

* A transcription system such as eScriptorium/Kraken

* A TEI modelisation of the structure of your documents

* A server to deploy a TEI Publisher application
THANK YOU FOR YOUR ATTENTION

CONTACTS

alix.chaque@inria.fr

floriane.chiffoleau@inria.fr

LINKS

Repository of the project: https://github.com/FloChiff/DAHNProject

eScriptorium: http://traces6.paris.inria.fr/