# Do (colored) backgrounds matter? An experiment on artificially augmented ground truth for handwritten text recognition applied to historical manuscripts

Alix Chagué, Hugo Scheithauer

► To cite this version:

Alix Chagué, Hugo Scheithauer. Do (colored) backgrounds matter? An experiment on artificially augmented ground truth for handwritten text recognition applied to historical manuscripts. Congress 2024 of the Humanities and Social Sciences Sustaining Shared Futures, CSDH/SCHN, Jun 2024, Montréal, Canada. hal-04622805

HAL Id: hal-04622805

https://inria.hal.science/hal-04622805

Submitted on 24 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Do (colored) backgrounds matter?

of a variable day depth. The effect of
day depths upon the surfaceing of various
animals has been reviewed elsewhere.
Anomalies. In split of the variability

of a variable day depth. The effect of
day depths upon the surfaceing of various
animals has been reviewed elsewhere..
Anomalies. In spite of the variability
of migrational behaviour, some kinds
of anomalies may be recognized.
Verbal movement occurs in some cases

An experiment on artificially
augmented ground truth for
handwritten text recognition
applied to historical manuscripts

Alix Chagué (Inria, UdeM, EPHE)
Hugo Scheithauer (Inria, EPHE)

<figure><img src="image_1.png" /></figure>
# Do (colored) backgrounds matter?

of a variable day depth. The effect of
day depths upon the surfaceing of various
animals has been reviewed elsewhere.
Anomalies. In split of the variability

of a variable day depth. The effect of
day depths upon the surfaceing of various
animals has been reviewed elsewhere..
Anomalies. In spite of the variability
of migrational behaviour, some kinds
of anomalies may be recognized.
Vertical movement occurs in some forms

An experiment on artificially
augmented ground truth for
handwritten text recognition
applied to historical manuscripts

Alix Chagué (Inria, UdeM, EPHE)
Hugo Scheithauer (Inria, EPHE)

<figure><img src="image_2.png" /></figure>
# Handwritten Text Recognition (HTR)

* supervised machine learning

* retrieve text from manuscripts

* requires annotated data

* data production is a challenge

* variety of scripts

* homogeneity of annotations

* availability and findability

* reusing datasets is important

TEXT

<figure><img src="image_2.png" /></figure>

MODEL

<figure><img src="image_3.png" /></figure>

DATA
# Handwritten Text Recognition (HTR)

* supervised machine learning

* retrieve text from manuscripts

* requires annotated data

* data production is a challenge

* variety of scripts

* homogeneity of annotations

* availability and findability

* reusing datasets is important

TEXT

<figure><img src="image_3.png" /></figure>

MODEL

<figure><img src="image_4.png" /></figure>

DATA
# The IAM Dataset

We used the IAM Dataset for our experiment:

* created by the IAM institute at the University of Bern between 1999 and 2002

* 13K annotated text lines written in English

* 657 different writers

* distributed in the form of words, text lines, sentences or pages

* black ink on white background (grayscale images)
# The IAM Dataset

We used the IAM Dataset for our experiment:

* created by the IAM institute at the University of Bern between 1999 and 2002

* 13K annotated text lines written in English

* 657 different writers

* distributed in the form of words, text lines, sentences or pages

* black ink on white background (grayscale images)
# The IAM Dataset & historical documents

don't address me

IAM Dataset

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

18th / 20th centuries

* can we make datasets like IAM look more like historical documents?

* binarization is no longer necessary and recent historical document digitizations are rarely black and white

* can we upgrade the IAM Dataset instead of downgrading historical datasets? (artificial colorization/decolorization of the images)

* is it just a question of color or are the accidents in the background also meaningful?
# The IAM Dataset & historical documents

don't address me

IAM Dataset

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

18th / 20th centuries

* can we make datasets like IAM look more like historical documents?

* binarization is no longer necessary and recent historical document digitizations are rarely black and white

* can we upgrade the IAM Dataset instead of downgrading historical datasets? (artificial colorization/decolorization of the images)

* is it just a question of color or are the accidents in the background also meaningful?
# Our questions and contribution

* can we make the IAM Dataset useful for historical HTR?

* are colored text lines more efficient to train HTR models rather than grayscale images?

* is it more environmentally cost-effective to use poorer grayscale images rather than rich colored images?

Additionally:

* we proposed a simple image manipulation method to add colored backgrounds

* we created an open dataset of real blank page documents
# Our questions and contribution

* can we make the IAM Dataset useful for historical HTR?

* are colored text lines more efficient to train HTR models rather than grayscale images?

* is it more environmentally cost-effective to use poorer grayscale images rather than rich colored images?

Additionally:

* we proposed a simple image manipulation method to add colored backgrounds

* we created an open dataset of real blank page documents
# Our method

* augment a dataset with a simple image manipulation: add a richer realistic background

* train model on the original/enhanced IAM Dataset

* test resulting models on the IAM Dataset test set

* further test resulting models on real historical data

<figure><img src="image_6.png" /></figure>
# Our method

* augment a dataset with a simple image manipulation: add a richer realistic background

* train model on the original/enhanced IAM Dataset

* test resulting models on the IAM Dataset test set

* further test resulting models on real historical data

<figure><img src="image_7.png" /></figure>
<figure><img src="image_7.png" /></figure>

# Dataset of real blank pages: Gallic album

* we create a dataset of real blank pages, found in Gallica

* 111 blank images, in color or not

* freely accessible: github.com/HugoSchtr/Gallicalbum

* useful to artificially colorize text images

* useful for other tasks like Layout Detection (no layout)
<figure><img src="image_8.png" /></figure>

# Dataset of real blank pages: Gallic album

* we create a dataset of real blank pages, found in Gallica

* 111 blank images, in color or not

* freely accessible: github.com/HugoSchtr/Gallicalbum

* useful to artificially colorize text images

* useful for other tasks like Layout Detection (no layout)
Enhanced IAM-Dataset

A **MOVE** to stop Mr. Gaitskell from **MOVE** to stop Mr. Gaitskell from

a01-000u-00.png – A **MOVE** to stop Mr Gaitskell from

appear to "prop up" an out-dated institution. appear to "prop up" an out-dated institution

a01-003x-08.png – appear to "prop up" an out-date institution

The film version of Miss Shelagh Delaney's play The film version of Miss Shelagh Delaney's play

c03-000c-00.png – The film version of Miss Shelagh Delaney's play

the great advantages to be derived from this the great advantages to be derived from this

c03-000c-05.png – the great advantages to be derived from this

The journey has been against me, as there has The journey has been against me, as there has

g06-018a-05.png – The journey has been against me, as there has

objects of beauty or interest. Vittoria, in the objects of beauty or interest. Vittoria, in the

k04-022-04.png – objects of beauty or interest. Vittoria, on the

fringe of the party, caught snatches of this fringe of the party, caught snatches of this

k04-022-05.png – fringe of the party, caught snatches of this
# Enhanced IAM-Dataset

<figure><img src="image_9.png" /><figcaption>a01-000u-00.png - A MOVE to stop Mr Gaitskell from</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>a01-003x-08.png - appear to "pop up" an out-date institution</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>c03-000c-00.png - The film version of Miss Shelagh Delaney's play</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>c03-000c-05.png - the great advantages to be derived from this</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>g06-018a-05.png - The journey has been against me, as there has</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>k04-022-04.png - objects of beauty or interest. Vittoria, on the</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>k04-022-05.png - fringe of the party, caught snatches of this</figcaption></figure>
Testing on real historical data

* 2 historical datasets in English covering a period from 1850 to 1950

* listed in the HTR-United catalog and reusable (licence)

* annotation rules compatible with IAM's

<figure><img src="image_8.png" /><figcaption>Joseph Hooker HTR (1850-1911)</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Univ. of Denver Jewish Consumptives Relief Society
Medical Records (1900-1950)</figcaption></figure>
Testing on real historical data

- 2 historical datasets in English covering a period from 1850 to 1950

- listed in the HTR-United catalog and reusable (licence)

- annotation rules compatible with IAM's

<figure><img src="image_16.png" /><figcaption>Joseph Hooker HTR (1850-1911)</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Univ. of Denver Jewish Consumptives Relief Society
Medical Records (1900-1950)</figcaption></figure>
# Experimentation set-up

* Kraken v 4.3.0

* default training architecture

* 120 pixels resize, 3 CNNs + 3Bi-LSTM + 1 linear layer

* learning rate of 1e-4 and batch size of 1

* two configs for the number of training epochs:

* fixed (100)

* early stopping (stops when learning curves reaches a plateau)

* train / validation / test follows the official IAM split

* test different combinations of original/enhanced train set and validation sets

<figure><img src="image_10.png" /></figure>

<table><thead><tr><td>Set</td><td>N text lines</td><td>N writers</td></tr></thead><tbody><tr><td>train</td><td>6161</td><td>283</td></tr><tr><td>val 1</td><td>900</td><td>46</td></tr><tr><td>val 2</td><td>940</td><td>43</td></tr><tr><td>test</td><td>1861</td><td>128</td></tr><tr><td>Total</td><td>9862</td><td>500</td></tr></tbody></table>
# Experimentation set-up

* Kraken v 4.3.0

* default training architecture

* 120 pixels resize, 3 CNNs + 3Bi-LSTM + 1 linear layer

* learning rate of 1e-4 and batch size of 1

* two configs for the number of training epochs:

* fixed (100)

* early stopping (stops when learning curves reaches a plateau)

* train / validation / test follows the official IAM split

* test different combinations of original/enhanced train set and validation sets

<figure><img src="image_18.png" /></figure>

<table><thead><tr><td>Set</td><td>N text lines</td><td>N writers</td></tr></thead><tbody><tr><td>train</td><td>6161</td><td>283</td></tr><tr><td>val 1</td><td>900</td><td>46</td></tr><tr><td>val 2</td><td>940</td><td>43</td></tr><tr><td>test</td><td>1861</td><td>128</td></tr><tr><td>Total</td><td>9862</td><td>500</td></tr></tbody></table>
<table><thead><tr><th>Train set</th><th>Validation set</th><th>N epochs</th><th>IAM grayscale test set</th><th>IAM enhanced test set</th><th>Historical test set</th><th>Training time</th></tr></thead><tbody><tr><td>IAM grayscale</td><td>IAM grayscale</td><td>100</td><td>69,8</td><td>29,7</td><td>4,1</td><td>01:14:33</td></tr><tr><td>IAM grayscale</td><td>IAM grayscale</td><td>87</td><td>70,1</td><td>30,6</td><td>7,6</td><td>01:10:19</td></tr><tr><td>IAM grayscale</td><td>IAM Enhanced</td><td>100</td><td>69,4</td><td>26,5</td><td>0,3</td><td>01:14:58</td></tr><tr><td>IAM grayscale</td><td>IAM Enhanced</td><td>62</td><td>66,4</td><td>27,9</td><td>11,1</td><td>00:51:30</td></tr><tr><td>IAM Enhanced</td><td>IAM grayscale</td><td>100</td><td>47,8</td><td>60,2</td><td>-4,9</td><td>01:16:52</td></tr><tr><td>IAM Enhanced</td><td>IAM grayscale</td><td>72</td><td>44,1</td><td>58,7</td><td>11,0</td><td>01:04:54</td></tr><tr><td>IAM Enhanced</td><td>IAM Enhanced</td><td>100</td><td>51,0</td><td>62,1</td><td>8,3</td><td>01:15:30</td></tr><tr><td>IAM Enhanced</td><td>IAM Enhanced</td><td>194</td><td>52,0</td><td>64,9</td><td>7,1</td><td>02:40:32</td></tr><tr><td>Manu McFrench Model V1</td><td>/</td><td>/</td><td>65,6</td><td>52,4</td><td>59,6</td><td>/</td></tr><tr><td>Manu McFondue</td><td>/</td><td>/</td><td>63,5</td><td>54,9</td><td>60,4</td><td>/</td></tr></tbody></table>

<table><tr><td>accuracy scale</td><td>0%</td><td>50%</td><td>100%</td></tr></table>

Our results
<table><thead><tr><th>Train set</th><th>Validation set</th><th>N epochs</th><th>IAM grayscale test set</th><th>IAM enhanced test set</th><th>Historical test set</th><th>Training time</th></tr></thead><tbody><tr><td>IAM grayscale</td><td>IAM grayscale</td><td>100</td><td>69,8</td><td>29,7</td><td>4,1</td><td>01:14:33</td></tr><tr><td>IAM grayscale</td><td>IAM grayscale</td><td>87</td><td>70,1</td><td>30,6</td><td>7,6</td><td>01:10:19</td></tr><tr><td>IAM grayscale</td><td>IAM Enhanced</td><td>100</td><td>69,4</td><td>26,5</td><td>0,3</td><td>01:14:58</td></tr><tr><td>IAM grayscale</td><td>IAM Enhanced</td><td>62</td><td>66,4</td><td>27,9</td><td>11,1</td><td>00:51:30</td></tr><tr><td>IAM Enhanced</td><td>IAM grayscale</td><td>100</td><td>47,8</td><td>60,2</td><td>-4,9</td><td>01:16:52</td></tr><tr><td>IAM Enhanced</td><td>IAM grayscale</td><td>72</td><td>44,1</td><td>58,7</td><td>11,0</td><td>01:04:54</td></tr><tr><td>IAM Enhanced</td><td>IAM Enhanced</td><td>100</td><td>51,0</td><td>62,1</td><td>8,3</td><td>01:15:30</td></tr><tr><td>IAM Enhanced</td><td>IAM Enhanced</td><td>194</td><td>52,0</td><td>64,9</td><td>7,1</td><td>02:40:32</td></tr><tr><td>Manu McFrench Model V1</td><td>/</td><td>/</td><td>65,6</td><td>52,4</td><td>59,6</td><td>/</td></tr><tr><td>Manu McFondue</td><td>/</td><td>/</td><td>63,5</td><td>54,9</td><td>60,4</td><td>/</td></tr></tbody></table>

<table><tr><td>accuracy scale</td><td>0%</td><td>50%</td><td>100%</td></tr></table>

Our results
# Discussions of the results

* probably not enough data to train an efficient model out of domain with the chosen architecture

* Manu McFrench and Manu McFondue are much better, but not good enough in zero-shot scenario on the historical testset

* models trained only on the original dataset = significant loss of accuracy on colored images (-40pts)

* models trained on colored images = minor loss of accuracy on original dataset (-10pts)

* colorful background seem to have an impact during prediction and not during training

* adding a colored background might augment the minimum amount of training data required
# Discussions of the results

* probably not enough data to train an efficient model out of domain with the chosen architecture

* Manu McFrench and Manu McFondue are much better, but not good enough in zero-shot scenario on the historical testset

* models trained only on the original dataset = significant loss of accuracy on colored images (-40pts)

* models trained on colored images = minor loss of accuracy on original dataset (-10pts)

* colorful background seem to have an impact during prediction and not during training

* adding a colored background might augment the minimum amount of training data required
# Perspectives to push the experiment further

* increase the quantity of data

* control the quality of the annotation in the historical test sets

* test with other historical datasets

* increase the batch size during training

* mix grayscale data and enhanced data in the train and validation sets

* Could we reach more stable scores when playing on these parameters?

* Can we reach 90% of accuracy on the IAM test set?
# Perspectives to push the experiment further

* increase the quantity of data

* control the quality of the annotation in the historical test sets

* test with other historical datasets

* increase the batch size during training

* mix grayscale data and enhanced data in the train and validation sets

* Could we reach more stable scores when playing on these parameters?

* Can we reach 90% of accuracy on the IAM test set?
# Final discussion on the environmental costs

* artificially enhancing IAM dataset increased its size by a factor of 6

* Train set: 325 Mb → 1960 Mb

* Validation set: 52 Mb → 287 Mb

* Test set: 107 Mb → 611 Mb

* minor impact on compilation time but not on training time (fixed epoch)

* estimations from Green Algorithms' Calculator ( calculator.green-algorithms.org ):

<table><thead><tr><th></th><th>Compilation time</th><th>Training time</th><th>Total time</th><th>Carbon footprint (CO2e)</th><th>Energy needed (kWH)</th></tr></thead><tbody><tr><th>All grayscale</th><td>1m27s</td><td>1h14m33s</td><td>1h16m0s</td><td>109,17</td><td>2,13</td></tr><tr><th>All enhanced</th><td>2m14s</td><td>1h15m30s</td><td>1h17m44s</td><td>112,04</td><td>2,18</td></tr></tbody></table>
# Final discussion on the environmental costs

* artificially enhancing IAM dataset increased its size by a factor of 6

* Train set: 325 Mb → 1960 Mb

* Validation set: 52 Mb → 287 Mb

* Test set: 107 Mb → 611 Mb

* minor impact on compilation time but not on training time (fixed epoch)

* estimations from Green Algorithms' Calculator ( calculator.green-algorithms.org ):

<table><thead><tr><th></th><th>Compilation time</th><th>Training time</th><th>Total time</th><th>Carbon footprint (CO2e)</th><th>Energy needed (kWH)</th></tr></thead><tbody><tr><th>All grayscale</th><td>1m27s</td><td>1h14m33s</td><td>1h16m0s</td><td>109,17</td><td>2,13</td></tr><tr><th>All enhanced</th><td>2m14s</td><td>1h15m30s</td><td>1h17m44s</td><td>112,04</td><td>2,18</td></tr></tbody></table>
Key takeaways

* we created a dataset of blank pages

* we successfully applied our enhancement technique to the IAM dataset

* we need to push our experimentations further to get more meaningful results

* we can already find that colorful background add noise during prediction but not during really training

* enhancing the dataset has a limited impact on the environmental cost of training a transcription model on the IAM Dataset
Key takeaways

* we created a dataset of blank pages

* we successfully applied our enhancement technique to the IAM dataset

* we need to push our experimentations further to get more meaningful results

* we can already find that colorful background add noise during prediction but not during really training

* enhancing the dataset has a limited impact on the environmental cost of training a transcription model on the IAM Dataset
Dataset references:

- U.-V. Marti, H. Bunke, *The IAM-database: An English sentence database for offline handwriting recognition*, International Journal on Document Analysis and Recognition 5 (2002) 39–46. doi:10.1007/s100320200071.

- K. Pham, *University of Denver Collections as Data - HTR Train and Validation Set JCRS_2020_5_27, 2020*. doi:10.5281/zenodo.4243023.

- J. Schaefer, K. Ross-Jones, A. Litvine, *Joseph Hooker HTR*, 2023.

- A. Chagué, H. Scheithauer, *Gallic album*, 2023.

Thank you!

And many thanks to Jennifer Carrow for her feedback
on this presentation!
Dataset references:

- U.-V. Marti, H. Bunke, *The IAM-database: An English sentence database for offline handwriting recognition*, International Journal on Document Analysis and Recognition 5 (2002) 39–46. doi:10.1007/s100320200071.

- K. Pham, *University of Denver Collections as Data - HTR Train and Validation Set JCRS_2020_5_27, 2020*. doi:10.5281/zenodo.4243023.

- J. Schaefer, K. Ross-Jones, A. Litvine, *Joseph Hooker HTR*, 2023.

- A. Chagué, H. Scheithauer, *Gallic album*, 2023.

Thank you!

And many thanks to Jennifer Carrow for her feedback
on this presentation!