# They're veGAN but they almost taste the same: generating simili-manuscripts with artificial intelligence

Chahan Vidal-Gorène, Jean-Baptiste Camps

## ► To cite this version:

Chahan Vidal-Gorène, Jean-Baptiste Camps. They're veGAN but they almost taste the same: generating simili-manuscripts with artificial intelligence. Digital Humanities 2023: Collaboration as opportunity, Alliance of Digital Humanities Organisations, Jul 2023, Graz, Austria. hal-04086940

HAL Id: hal-04086940

https://enc.hal.science/hal-04086940

Submitted on 18 May 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>
# They're veGAN but they almost taste the same: generating simili-manuscripts with artificial intelligence*

Chahan Vidal-Gorène¹,² and Jean-Baptiste Camps¹

¹École nationale des chartes, Université PSL, France

²Calfa, France

## 1 Introduction

The aim of our research is to artificially generate fake historical manuscripts using generative adversarial networks, better known as GANs (Goodfellow et al., 2014). In this neural network architecture, a generator network is put in competition with a discriminator (e.g., the first one generates images that seem authentic, and the second tries to guess if they are or not).

At this stage, these experiments pursue two objectives: evaluating feasibility of generating realistic fake manuscripts under certain conditions of layout, script, or date, and creating artificial data for HTR training, as is done for printed materials with synthetic data (Etter et al., 2019). Ground truth creation is indeed a time consuming task, in particular for ancient languages for which we still lack specialists able to manually annotate documents. GAN appears as a relevant answer to this challenge, as they reach very convincing results in different scenarii. State-of-the-art results rely on a style-transfer approach (Karras et al., 2018), in which a generator try to map on a targeted dataset unsupervised learned features from an other. Such an approach has already been successfully applied to historical manuscripts to create realistic Latin manuscripts (Vögtlin et al., 2021) with very constrained styles in training (page-level), or for contemporary cursive hand-writings (Fogel et al., 2020) (line-level with a semi-supervised approach).

This short paper investigates the feasibility of the line-level approach for historical manuscripts.

## 2 Datasets

We focus on two different manuscript traditions. The first dataset is composed of 8 Armenian manuscripts in a very regular *bolorgir* script (Stone et al., 2002, Vidal-Gorène and Decours-Perez, 2021), from the 14th to the 18th century. Dataset composition is described in table 1. The Old French dataset is composed of data extracted from the *CREMMA-MEDIEVAL* dataset (Pinche, 2022), from which 6324 lines of a 13th century manuscript in *textualis* Latin script were selected (ms. BnF, fr. 412).

Data annotation has been made on eScriptorium (Kiessling et al., 2019) for Old French manuscripts and Calfa Vision (Vidal-Gorène et al., 2021) for Armenian manuscripts.

<footnote>* This research was funded by the project "Artificial Pasts: lost texts and manuscripts that never were" (Paris Artificial Intelligence Research Institute - PRAIRIE, SHS call).</footnote>
<table><caption>Table 1: Composition of the Armenian dataset. The manuscript identifiers follow the ID system of the Index of Digitized Armenian Manuscripts (Vidal-Gorène et al., 2022)</caption><thead><tr><th>Manuscript*</th><th>Language</th><th>Date</th><th>Script</th><th>Lines</th></tr></thead><tbody><tr><td>M1</td><td>Classical Armenian</td><td>1632-1633</td><td>bolorgir</td><td>512</td></tr><tr><td>M6</td><td>Classical Armenian</td><td>16th</td><td>bolorgir</td><td>819</td></tr><tr><td>M982</td><td>Classical Armenian</td><td>1460</td><td>bolorgir</td><td>3.035</td></tr><tr><td>MAF52</td><td>Classical Armenian</td><td>15th-16th</td><td>bolorgir</td><td>954</td></tr><tr><td>MAF54</td><td>Classical Armenian</td><td>16th</td><td>bolorgir</td><td>1.253</td></tr><tr><td>MAF62</td><td>Classical Armenian</td><td>18th</td><td>bolorgir</td><td>522</td></tr><tr><td>TBI122</td><td>Classical Armenian</td><td>17th</td><td>bolorgir</td><td>1.209</td></tr></tbody><tfoot><tr><td colspan="4">TOTAL</td><td>8.304</td></tr></tfoot></table>

## 3 Method

We mainly follow the ScrabbleGAN approach (Fogel et al., 2020) and code<fnref n="1" />, using a generator-discriminator combined with a HTR architecture to assist the discrimination process. Instead of training the GAN to generate characters that are joined to create a sentence (the “scrabble approach”), we investigate the relevance to focus on strokes, that are then combined to create characters and words. The choice of the stroke level is motivated by the nature of the scripts envisioned. Indeed, working at the character level, which might be very suitable for printed materials, could result in generated text with too neatly separated, or even artificially juxtaposed, letters, without the ligatures and stroke fusions that are encountered in manuscripts. The recogniser architecture has been reduced compared to the original paper, as we need less abstraction to distinguish the constituent strokes of the letters<fnref n="1" />.

<figure><img src="image_2.png" /><figcaption>Figure 1: Recogniser architecture, with 3 CNN layers less compared to the original paper. State-of-the-art recogniser generally include LSTM layers, but too high accuracy would lead here to a less realistic generation, as the recogniser would recognise the text even if image is heavily noisy.</figcaption></figure>

Data originally consist in image and ALTO XML pairs. For these experiments, we have applied

<footnote n="1">¹ https://github.com/amzn/convolutional-handwriting-gan</footnote>
two *modi operandi*:

1. with preprocessing:

a) binarisation of images;

b) extraction of lines patch, basically generating a line image - txt pair, cropping the polygon of the line in a white bbox, with baseline adjustment;

c) Chocomufin of text for Old French (Clérice and Pinche, 2021)

2. without preprocessing: we only perform the line extraction step with baseline adjustment, keeping the mask of the line.

4 Results and discussion

Generated images are described below (fig. 2 and 3).

For Armenian, the artificial images, both binarized or non-preprocessed, are very convincing examples of *bolorgir* script, and are fully legible for an expert (and hard to differentiate from an actual manuscript image).

For Old French, the results for the binarized version are less enticing, but nonetheless inter-
esting. In particular, the model seems to insert a spurious character before or after each letter:
e.g. the last word of the last line in fig. 2 – that should be *mengier* – actually reads something as
*mienigierie*. This might be a way to simplify the work of the recogniser by signalling the end of
individual characters.

<figure><img src="image_3.png" /><figcaption>Figure 2: Examples of binarized GT and generated images for Classical Armenian and Old French</figcaption></figure>

Generated color lines, both in Armenian and Old French, are even more convincing for the
expert. The presence of the background and all variations of intensity in the ink reinforce the
credibility of the images generated. This is an initially counter-intuitive result. The very wide
variety of backgrounds (papyrus, parchment, paper ; colorimetry, etc.) nevertheless leads to a
defect in the GAN which produces a result with a somewhat blurred appearance, with a slightly
dripping color see fig. 3, images 2 and 5). Some of the very rare letters are also less well generated,
and produce noise in the colorimetry of the line (see fig. 3, image 6). From a qualitative point
of view, it is not possible to identify whether an Armenian line is true or not, except on the
question of the abbreviative signs (*badiw*, horizontal line above the lines) drawn randomly (above
unabbreviated words). This abbreviation sign is naturally considered as background and therefore
generated randomly, because it is not present in the transcription provided.

It is interesting to note that the GANs also reproduce the presence of a black random mask
around the text, mask initially generated during the extraction of the lines from the ALTO files.
This may appear as noise, however it therefore multiplies the possible scenarios in the composition
of the masks and may also be a mean of reinforcing the robustness of the recognition.

For now, the recogniser proves too efficient, recognising even images with no human-readable
text, and preventing the generator from producing fully historically plausible images. As the
recogniser is able to recognise the text even if lines are not realistic, the generator seems to be
penalised and stops its attempts to create more realistic results.
<figure><img src="image_4.png" /><figcaption>Figure 3: Examples of colour generated images for Classical Armenian and Old French</figcaption></figure>

At this stage, generated images with their corresponding transcription can already be used as a relevant data augmentation. We have led a very short experiment with the Armenian dataset. We have divided our dataset in 4 subsets : train (827 real lines), val (1703 real lines), test in-domain (3772 real lines) and test out-of-domain (2002 lines) and have performed 4 trainings:

1. default: train set only

2. default+gan250: train set + 250 generated lines

3. default+gan500: train set + 500 generated lines

4. default+gan1000: train set + 1000 generated lines

<figure><img src="image_5.png" /><figcaption>Figure 4: Val accuracy during training</figcaption></figure>

For artificial data generation in Armenian, we have used text from chronicles really far from training real manuscripts (mainly Gospels). This addition of a new vocabulary and sentences is maybe the reason of the increase of HTR performance on out-of-domain tasks.

Training curves (Fig. 4) show that we significantly speed-up training with fake images, as we could have with a bigger training dataset. CER achieved by default model and default+gan500
model are respectively 6.13% and 5.63% on in-domain test, and 22.72% and 10.79% on out-of-domain test.

In this scenario of an under-resourced script, the data augmentation, in particular with out-of-domain words, therefore appears relevant. We have not been able to conduct similar experiments at this stage for Old French, for which we have a large number of abbreviative signs that are very poorly covered, and therefore poorly generated by the GAN. The main advantage of the method seems to be the qualitative dimension of the data augmentation carried out, which is not just a simple and random transformation of the image. If the final accuracy reached is the same, the model with GAN converges faster – in terms of number of epochs – and demonstrates its relevance in a complete out-of-domain situation. A 100% GAN model, on the other hand, does not yet seem relevant.

# 5 Future research

Current experiments highlight the feasibility and the relevance of using GAN to create fake historical manuscripts. We show that state-of-the-art systems can be transferred to historical data, and intend in the future to reduce the recogniser abilities to increase image generation. We also plan to generate complete pages instead of single lines. The approach presented has the disadvantage of requiring an already well-balanced dataset to be able to generate all the characters in a relevant way. A complementary style transfer for poorly endowed characters might have its relevance.

For now, we do not, strictly speaking, offer a evaluation of the (pseudo-)authenticity of generated lines, which is by any means a question difficult to approach. Yet, the verisimilitude of the generated lines for the expert upon direct inspection, on one hand, and the improve in HTR performance (meaning that the HTR models found some measure of information of historical relevance in the generated lines), on the other, offer two complementary approaches to this evaluation. In the future, both approaches could be extended.

# References

Thibault Clérice and Ariane Pinche. Choco-Mufin, a tool for controlling characters used in OCR and HTR projects, 9 2021. URL https://github.com/PonteIneptique/choco-mufin.

David Etter, Stephen Rawls, Cameron Carpenter, and Gregory Sell. A synthetic recipe for ocr. In *2019 International Conference on Document Analysis and Recognition (ICDAR)*, pages 864–869. IEEE, 2019.

Sharon Fogel, Hadar Averbuch-Elor, Sarel Cohen, Shai Mazor, and Roee Litman. Scrabblegan: Semi-supervised varying length handwritten text generation, 2020. URL https://arxiv.org/abs/2003.10557.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In *Proceedings of Advances in Neural Information Processing Systems 27 (NIPS 2014)*, 2014.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2018. URL https://arxiv.org/abs/1812.04948.

Benjamin Kiessling, Robin Tissot, Peter Stokes, and Daniel Stökl Ben Ezra. eScriptorium: an
open source platform for historical document analysis. In *2019 International Conference on Document Analysis and Recognition Workshops (ICDARW)*, volume 2, pages 19–19. IEEE, 2019.

Ariane Pinche. CREMMA Medieval, an Old French dataset for HTR and segmentation, 2022. URL https://github.com/htR-United/cremma-medieval.

Michael E Stone, Dickran Kouymjian, and Henning Lehmann. *Album of Armenian Paleography*. Aarhus University Press Aarhus, 2002.

Chahan Vidal-Gorène and Aliénor Decours-Perez. A computational approach of armenian paleography. In *International Conference on Document Analysis and Recognition*, pages 295–305. Springer, 2021.

Chahan Vidal-Gorène, Boris Dupin, Aliénor Decours-Perez, and Thomas Riccioli. A modular and automated annotation platform for handwritings: evaluation on under-resourced languages. In *International Conference on Document Analysis and Recognition*, pages 507–522. Springer, 2021.

Chahan Vidal-Gorène, Anush Sargsyan, and Emmanuel Van Elverdinghe. Index of digitized armenian manuscripts, July 2022. URL https://doi.org/10.5281/zenodo.6894290.

Lars Vögtlin, Manuel Drazyk, Vinaychandran Pondenkandath, Michele Alberti, and Rolf Ingold. Generating Synthetic Handwritten Historical Documents With OCR Constrained GANs. In *2021 16th IAPR International Conference on Document Analysis and Recognition (ICDAR)*, Lausanne, Switzerland, sep 2021.