Automatic Transcription of Handwritten Old Occitan Language
Authors: Esteban Garces Arias
Vallari Pai
Matthias Schöffel
Christian Heumann
Matthias Aßenmacher
Venue: EMNLP 2023 Main (2023)
Keywords: handwritten text recognition
low-resource languages
transformer
computer vision
natural language processing
TL;DR: we achieve state-of-the-art performance on the transcription of handwritten old occitan text

Abstract: While existing neural network-based approaches have shown promising results in Handwritten Text Recognition (HTR) for high-resource languages and standardized/machine-written text, their application to low-resource languages often presents challenges, resulting in reduced effectiveness. In this paper, we propose an innovative HTR approach that leverages the Transformer architecture for recognizing handwritten Old Occitan language. Given the limited availability of data, which comprises only word pairs of graphical variants and lemmas, we develop and rely on elaborate data augmentation techniques for both text and image data. Our model combines a custom-trained Swin image encoder with a BERT text decoder, which we pre-train using a large-scale augmented synthetic data set and fine-tune on the small human-labeled data set. Experimental results reveal that our approach surpasses the performance of current state-of-the-art models for Old Occitan HTR, including open-source Transformer-based models such as a fine-tuned TrOCR and commercial applications like Google Cloud Vision. To nurture further research and development, we make our models, data sets, and code publicly available.

Automatic Transcription of Handwritten Old Occitan Language
Esteban Garces Arias'
Vallari Pai
Matthias Schöffel
Christian Heumann"
Matthias Aßenmacher
Department of Statistics, LMU, Munich, Germany
Bavarian Academy of Sciences, BAdW, Munich, Germany
Munich Center for Machine Learning (MCML), LMU, Munich, Germany
(esteban.garcesarian) garcesar ias, chris, matriassttttiissssta uni ami--munchenn.d Valari.Pai@caree 1mu.de
matthias. scwoef
Abstract
French, it lacks comprehensive collections of dig- itized manuscripts with scanned images and an- notated corpora, which essential resources for conducting detailed morpho-syntactic or syntactic analyses (Scrivner and Kübler, 2012).
While existing neural network-basse ap- proaches have shown promising results in Handwritten Text Recognition (HTR) for high- resource languages and standardized/machine written text, their application to low-resource languages often presents challenges, resulting reduced effectiivnness. In this paper, we pro- pose innovative HTR approach that lever- ages the Transformer architecture for recogniz- ing handwritten Old Occitan language. Given the limited availability of data, which com- prises only word pairs of graphical variants and lemmas, we develop and rely on elab- orate data augmentation techniques for both text and image data. Our model combines a custom- trained Swin image encoder with BERT text decoder, which pre-train using a large-scale augmented synthetic data set and fine-tune the small human-labeled data set. Experimental results reveal that our approach surpasses the performance of current state-of- the-art models for Old Occitan HTR, including open-source Transformer-based models such fine-tuned TrOCR and commercial appli- cations like Google Cloud Vision. To nurture further research and development, we make our models, data sets, and code publicly available: https://hugging: co/ modda
More recently, however, there have been efforts to ramp up the availability of resources for this language. An example of this is the creation of a digital version of the Old Occitan dictionary!, led by team of researchers at the Bavarian Academy of Sciences. The project aims to establish open- access database for scientific research, featuring curated collection of vocabulary. A crucial step for this is the digitization of handwritten material of (non-standardized) words. For our research, we had access to a collection of 600,000 handwrit- ten cards (93% of these being unlabeled) contain- ing graphical variants alongside their respective lemmas. In this work, explore and combinc Transformer-based architectures with data augmen- tation techniques specifically tailored address the challenges posed by HTR for low-resource lan- guages. The experimental outcomes demonstrate the effectiveness of our method, as it achieves state- of of-the-art (SOTA) results in Old Occitan.
Contributions This work provides the following contributions:
1 Introduction
Old Occitan, also known as Old Provençal, was a language widely spoken in the 11th-16th cen- turies, in particular in southern France, northeast- ern Spain, and northwestern Italy. It occupies prominent position in both the linguistic and cul- tural legacy of Romance languages, primarily due its role as precursor to French lyric and the dissemination of works by Troubadours through- out Europe. Despite its well-established historical importance, the linguistic research of Old Occi- tan remains relatively limited. In contrast to Old
1. We propose a Transformer-based HTR ap- proach using encoder-decoder architec- ture, effectively addressing limitations in low- resource languages through data augmenta- tion techniques.
We conduct a comparative analysis of various architectures, data augmentation methods, and decoding strategics.
Dictionmaire de occcitan médiéval
http: / //www//ww//////////// dom- en- 1i Ligne, de/

---

3. We extensively review existing open-source and closed-source OCR and HTR tools, and benchmark our model against them. Our model achieves state-of-the-arr (SOTA) results on the Old Occitan data set.
Furthermore, researchers have explored the ap- plication of Transformer-basee models low- resource languages. For instance, Ströbel et al. (2022) successfully fine-tuned a TrOCR instance for handwritten Medieval Latin and surpassed the widely-used Transkribus (Kahle et al., 2017), a commercial platform for historical document tran- scription. The results highlight the potential of custom Transformer-based models to address HTR challenges in languages with a limited amount of available data.
4. We publish our codebase and the data sets: https://huggingg soiis
2
Related Work
A comparative study by Michael et al. (2019) exam- ined various attention mechanisms and positional encodings to address the alignment between in- put and output sequences in the context of HTR, a field of study that was traditionally dominated by Recurrent Neural Network (RNN) encoders combined with Connectionist Temporal Classifi- cation (CTC) decoders (Bluche and Messina, 2017; Graves al., 2006; Pham et al., 2014).
3
Data Preparation
3.1 Handwritten cards
Researchers have continuously compiled the Old Occitan card database used in this study since around 1960. Each card follows prototypical structure, as exemplified in Figure 1, consisting of 'GRAPHICAL VARIANT > LEMMA' written in uppercase letters against a blue background. The handwritten text, on average, spans 15.7 characters and typically comprises two words: the graphical variant (left side) with average length of 7.5 characters, and the lemma (the standardized entry in the Old Occitan dictionary; right side) with an average length of 7.2 characters. The two words are always separated by a one-character arrow sign, denoting the reference from the graphical variant to the lemma
Subsequently, a systematic literature review con- ducted by Memon et al. (2020) examined HTR and OCR research between 2000 and 2018. Most of the reported approaches employed architectures based on Convolutional Recurrent Neural Network (CRNN), Long Short-Term Memory (LSTM) mod- els, and CTC, applied to well-studied languages such as English, Chinese, Urdu, and Arabic.
With the popularization of the Transformer model by Vaswani et al. (2017) and its multi- modal applications, other approaches deviating from the conventional Convolutional Neural Net- work (CNN) and RNN architectures have emerged and shown SOTA performances in HTR. One no- table example is Transformer OCR (TrOCR) (Li et al., 2021), which adopts Transformer model its backbone. This end-to-end HTR model con- sists of a vision encoder and autoregressive text decoder. Subsequently, Barrere et al. (2022) pro- posed a more compact Transformer-based archi- tecture that employs different visual feature em- bedding techniques and combines CTC and cross- entropy loss during training. Both Transformer- based approaches achieved competitive results on the widely-used IAM data set (Marti and Bunke, 2002), which contains overall 115,320 handwrit- ten English words on 13,353 images. Addition- ally, Diaz et al. (2021) compared various encoder- decoder models and found that using self-attention in the encoder and CTC-trained decoder enriched with a language model yielded SOTA performance on the same data set.
The labeled data consist of 41,634 samples containing 39,554 unique graphical variants and 15,852 unique lemmas. This results in an average of 2.5 graphical variants per lemma. The annota- tions are provided in plain text format and consist of a set comprising 73 distinct characters. These characters primarily include the 26 letters of the Latin alphabet, diacritics from the French alpha- bet, and digits. Additionally, the labeled material contains punctuation symbols, question and excla- mation marks, parentheses, brackets, and phonetic notation. comprehensive overview of the charac- ter inventory, including their absolute and relative frequencies, can be found in Table Appendix C.
The majority of the images in the data set hold text that is exclusively written the upper half of the card, presented in clear and legible manner. However, notable portion of the cards exhibits text spread across multiple lines, accompanied by corrections, additional side notes, bent corners, or other irregular marks that may have arisen during

---

BAÇA
SENET EAAO
-ENNE- - PARON
ALEGREE2A
ALEGRESSA
ALCERE2A
(a) Original image
(b) Pre-processed image
SERVOYSA, 9 CERVEM 4.
BATINA a BATEUAR
Figure 2: Pre processing of an original Old Occitan card, after cropping, enhancement of contrast, sharpness and brightness, and binarization.
format used in the EMNIST data set (Cohen et al., 2017). which features white text a black back- ground. An example of our image pre-processing is shown in Figure 2. A comprehensive list of the pre-proeessing parameters is shown in Table Appendix B.
DALMA BALMATOE
POINHIER PONNER
"DANNATGE
Ruis "affn. (dlli wolline
Corpus pre-processing First, we create a com- pilation of different relevant corpora: We combine excerpts from Histoire de langue provençale: à Avignon du XIIe au XIXe siècle (Pansier, 1974) with three Old Occitan corpora publicly available The corpora are composed of documents covering liter- ature, lyric, law, ecclesiastic narrative, and adminis- tration texts from 1050- 15500 A detailed overview of the corpora used (name, genre, and number of tokens) is shown in Table 11 in Appendix E. Sec- ond, apply additional pre-processing steps ensure the eleanliness and continuity of the text. Misplaced empty spaces, such as those found af- ter apostrophes or between words and punctuation marks, are removed. Additionally, we eliminate line breaks to create uninterrupted text. Finally, the set of unique words is extracted and transformed into upper-case format. These words are then combined with every other word to generate syn- thetic word pairs, separated by an arrow symbol to emulate the structure of 'GRAPHICAL VARI- ANT + LEMMA'. This procedure results the creation of over 82 million synthetic word pairs, from which synthetic labels were generated for data augmentation purposes. Table 12 in Appendix E provides an example of this process. It is worth noting that initially, our approach involved the gen- eration of word pairs with maximum Levenshtein distance of two, as this distance closely aligned with the median value observed in our training la- bels. However, we subsequently decided to remove
Figure 1: Examples of (original) cards with Old Occitan text. Standard cases are depicted in the first row, while different exceptions are displayed in images below.
the scanning process. Various examples illustrat- ing these variations are depicted in the lower part Figure The average dimensions (width x height) of card are 1235 x 1390 pixels, with a print resolution of 300 dpi.
3.2 Data pre-processing
We pre-process the data set enhance model per- formance, enabling data augmentation and syn- thetic image generation.
Image prepprocsssing Firstly, the images un- dergo a cropping process where only the upper 40% is retained for the text recognition task, con- sidering that the relevant text is always located in the upper section of the card, and the lower 60% of the image might contain noise in the form of irregu- lar annotations. Secondly, contrast, sharpness, and brightness enhancements are applied to improve the legibility of the text. Lastly, the images are binarized. To ensure compatibility for data aug- mentation, we align this binarized variant with the
Phttp: //cl, indiana. edu::aedddeeea edu: w080/anniss-eeeee/eee/ee.aa momis--gii-..m g.i-3.6.4 0.

---

this constraint increase not only the quantity and variability of word pairs but also the potential gen- eralization of our model, in particular when applied to slightly different data. noteworthy result of this modification can be observed in the case of special characters such as A, Ô, and French dia- critics, which experienced a significant increase in examples, growing from just few dozen to thou- sands.
The generation process involves utilizing real training labels (or synthetic labels from the aug- mented corpus) as input. For each character these labels, a random image is sampled from the extended EMNIST character inventory. Concate- nating these images produces the synthetic output image. To replicate the variability observed in real Old Occitan handwritten text, random components introduced, such as varying image sizes, num- ber of lines, rotation angles, dilation factors, and pixel distances between characters, allowing for overlapping and irregular spacing. This approach enables the generation an average of 37.5 images per second. An example of the generated images is depicted in Figure 3.
3.3 Data Augmentation
Real images We consider two steps to augment the real pre-processed images: random rotation and dilation. The parameters are depicted Table 10 while selected examples augmented images are displayed in Figure 5, both in Appendix D.
For reproducibility purposes, comprchensive overview of the image generation parameters is provided in Table 13, while the steps are outlined in Algorithm 1. The code for image generation, along with the extended version of EMNIST, is publicly accessible".
ALEGRESSA
ALEGREZA
Final data set Around 7% of the handwritten material was labeled, amounting to 41,634 samples. Among these, we use 80% (33,308) for training, 10% (4,163) for validation, and the remaining 10% (4,163) for testing. To expand the training data, we generate synthetic images. Specifically, we generate 180,000 images during the model selec- tion phase. For enhanced pre-training of the best- performing model, we generate additional 720,000 images, resulting a total of 900,000 images.
(a) Real pre- processed image
AL EG RESS A AL FGR EZA
4LEGRESS h
Model selection This stage involves exploring various combinations of vision models as encoders and language models as decoders. The pre-training, fine-tuning and evaluation configura- tions can be found in Table
Stage 1:
PLEGREZA
AL GRESS AL GREZA
Stage 2: Final model training Once the optimal encoder-decoder combination was identified, we enhanced the data augmentation for (a) pre-training, with more synthetic images, and (b) fine-tuning, incorporating random rotation and dilation of real images, as described in Table 1.
(b) Synthetic images generated with EMNIST
Figure 3: Examples of pre-processed original images and synthetic images generated from the EMNIST data set and the training labels. Synthetic images were also generated from a synthetic corpus (cf. Sec. 3.2).
Synthetic images To increase the available train- ing data, we expand the EMNIST data set by in- corporating special characters frequently encoun- tered in our cards, including the arrow (-+) and the cedilla (C). This data set, the training labels, and the pre-processed corpus described in Section 3.2, are used to generate synthetic images.
Experimental Setup
Model selection We explore combinations of four vision encoder models: BETT (Bao al., 2021). DeiT (Touvron et al., 2021). VIT (Doso- vitskiy et al., 2021), and Swin (Liu al., 2021)
Public GitHub repository

---

Stage (Model selection) 180,000
4.1 Model Architectures
(Final model training)
Pro-training
900,000
Tokenizer We employ byte-level BPE (Byte Pair Encoding) tokenization (Sennrich al., 2016) train tokenizer using our training labels. This technique involves itcratively merging the most fre- quent pairs of bytes until a predetermined limit is reached, or no further merges are possible. our case, we set the number of output tokens to 73 (cf. Table 9) to match the number of charac- ters observed in our labels (note that in this non- standardized, low-resource language, the vocabu- lary of words is not entirely known). The tokenizer required 82 encodings for classification, padding, end-of-sentencee and unknown tokens, as well as for the observed characters, given that some them require more than a single integer for encod- ing (cf. Table 14 in Appendix G).
(w/ synthetic images) Fine-tuning real images)
33,308
33, random rotations and dilations, et. Table 10)
Evaluation
4,163 (Validation)
4,163
(w/ real images)
(Test)
Total
217,471
1,137,319
Table I: Data configuration for the two stages of this research; Model selection and final model training.
with two language decoders: GPT-2 (Radford et al., 2019) and BERT (Devlin et al., 2019). To inves- tigate the impact of synthetic data, we train each combination using three different training setups: synthetic data only ("pre-train",, real data only ("only real"), and fine-tuning the pre-trained model with real data ("fine-tune"), This resulted in a total of x 2 x 3 = 21 experiments.
Image encoder We use the Swin Transformer (27.5M parameters), extension of VIT (86.4M parameters), to enhance efficiency and enable cross- window connections. Swin incorporates hierarchi- cal feature maps and shifted window attention, lim- iting self-attention to non-overlapping local win- dows. Like ViT, BEiT, and DeiT, the image is di- vided into patches and projected into embeddings. However, Swin uses smaller patches to avoid high computational costs. Each patch is assigned to a self-attention window for local processing, reduc- ing the complexity of the self-attention To capture features across windows, the self-attention window shifted, allowing for the processing of separated image regions. This hierarchical structure enables lower blocks to handle fine-grained information while upper layers operate on merged visual rep- resentations. Our model uses newly initialized Swin encoder with pre-trained image processor hen--bus--pathhhd-idddoo////
To evaluate the performance of the models, we used our validation data set. Tables 5 and 7 provide a comprehensive overview of the model combina- tions and the corresponding training regimes.
Final model training After selecting the best- performing model the validation set, hanced data augmentation and incorporate 900,000 synthetic images in total during pre- pre-training. Seven random rotations and dilations were applied to the real images, as outlined Table 1.
To assess the effectiveness of our optimized model, conducted a benchmarking study. We compared its performance on the test set against popular open-source tools that support handwritten Occitan. In addition, we included a commercial alternative, Google Cloud Vision*, known for its proven high performance in practical applications (Thammarak et al., 2022). A short description of the models is presented in Table 2.
Text decoder We utilize BERT-based decoder architecture (1 (114.MM parameters) for our text de- coding task. This decoder employs an autoregres- sive language modeling objective, and the proba- bilities over the vocabulary are calculated using the softmax function. During decoding, we do not apply n-gram repetition penalties. This choice is motivated by the presence of repetitive patterns in our handwritten material, as indicated by the me- dian Levenshtein distance (Levenshtein et al., 1966) of two between graphical variants and the lemmas. To effectively capture the textual structure the task, we choose to train our decoder from scratch.
Open- Source
tuned
w/ our data
RasyOCR (Jaided, 2020)
Yes
ResNetsCTC
No
Tesserecr OCR (Ooms, 2023)
Yes
CNN-I.STMs
No
PaldeoOCR et 2020)
CRNN-CTC
RIS-LSTM (lle et al., 2016)
Yes
RewNeIB-CNN--LSTA
Yes
et
Transformer
Unknown
Google Cloud Vision
No
No
(commercial)
Characteristics of models used for benchmark- Table 2:
ing. featuring open-source and commercial tools.

---

Focader
Dheoder
Training Selop
GPU Unilization
Training Ramtime
Inforunce
Examples/
CRR (Wiighte))
Correctly predicted
3199
252
GPT-2
Ouly Real
31.4
14.8
1,718.
2.42
0.406
6.11%
BeiT
Fine Tune
Pre-Train
32.3
16.7
182
5/
1,5111
1,5663
2.52
507
0.876
0.213
21,815
01.1%
28.5
6.23
1,6608
3,06
0.273
14.1%
GPT-2
Ouly Real
27.5
18.6
1,88399
2.47
0.041
73.216
Delf
Fine-Tune
29.8
28.1
6.2
1,988
2.45
3.20
0.021
12,496
82.8%5
16.5
1,361.7
Fine Tune
27.4
6.3
1,370.1
3.06
0.021
81.918
Pre Train
41.5
19.0
1,1933
2.57
0.988
6.016
CIP-22
Only Rea
10.0
158
1,732.1
2.40
0.079
70.45%
VIr
Fine Tune
Prc-Train
30.7
6.1
1,8577
2.51
0.355
0.022
K3.75
7.2%%
BERT
Only
26.0
1,3619
3.05
0.023
$3.4
Fine-Tune
28.9
5.9
1,371.1
3.04
0.018
86.35
Pte Train
10.9
200
1,01.3
2.00
0.316
9.016
GPT2
Ouly Real
22.2
19.9
1,4488
2.39
0.126
M0.615
Fine Tune
26.0
1,6188
2.50
0.020
84.2%
BERT
Only
21.3
21.0
1,407.7
2.96
0.026
81.1%
Fine Tune
22.9
6.5
1,4003
297
0.016
87.0%
Table Results for all model combinations trained synthetic and/or real data. The best results highlighted in bold. All experiments were performed with GPU NVIDIA Tesla V100 (16 GB).
This approach enables us prioritize the unique characteristics of the text and utilize a customized tokenizer specifically trained to excel at a granular character level. Additionally, we explore the use of GPT-2 (114.3M parameters) as an alternative to the BERT-based decoder.
summarized in Tables 3 and For reproducibility. Table 7 contains a list of hyperparameter values used during training.
Decoding strategy We considered beam search strategy and experimented with different beam width values: (greedy), 4, 10, 15, 30, and 50. The additional parameters for natural language genera- tion are summarized in Table 8 in Appendix B.
Performance metries We assess the models ing multiple metrics, including weighted Character Error Rate (CER) and the percentage of correctly predicted labels. Additionally, we measure training runtime, GPU utilization (%), total number of train- able parameters, and inference speed in labels per second to gauge resource usage and complexity.
Results
5.1 Model selection
The CER is computed by summing up edit oper- ations and dividing by the label length.
The results of the comparative experiments are pre- sented in Table 3, demonstrating that the combina- tion of Swin and BERT yields the best outcomes across various evaluation metrics. These metrics include weighted CER (0.016), percentage of cor- rectly predicted labels (87.0%), lowest runtime (6.5 hours), GPU utilization (22.9%), and the number of parameters (1 (142M, second only to Swin + GPT-2). Regarding the inference speed, the model predicts 2.97 labels/second on average, not far away from the fastest model (BEiT + BERT, with 3.06 pre- dicted labels/second). The results also highlight the effectiveness of using synthetic data, as it de- creases the weighted CER from 0.026 ("only real") to 0.016 ("fine-tune") and increases the percentage of correctly predicted labels from 81.1% % ("only real") to 87.0% ("fine-tune").
CER + D+
S+D+I
N = D+
(1)
where S is the number of substitutions, D is the number of deletions, / is number of insertions, is the number of correct characters. and N is ber of characters in the label. To account for the varying length of the labels, which range from 4 to 40 characters with a standard deviation of 4.27, we utilize the weighted CER.
EIL 1 CER.
W WeightedCEI
(2)
Σ 4
where 1; is the number of characters of label i, and CER,, is the CER for example i, 1,...,m. The observed values for all defined metrics are
Simillar trends of improvements when using synthetic data are observed for the majority of model combinations.

---

5.2 Benchmarking against HTR & OCR tools We further enhance data augmentation the Swin BERT model, as outlined in Table 1. Subse- quently, we compare the performance of our model with open-source and closed-souree alternatives. The benchmarking results, presented in Figure 4, demonstrate the superior performance of our model with respect to the competing tools.
ever, as the number of text lines increased up three, the model's performance declined, resulting in a mean weighted CER of 161 and a correct prediction rate of 71 71.4%.
Additionally, the quality of annotations proved be an important factor affecting performance. For standard images, the model achieved an average weighted CER of 0.004 and accurately predicted 97.0% of the labels. However, when dealing with images containing annotation errors or noise, the performance decreased to a mean weighted CER of 0.043, with correct prediction rate of 81.2%. It is important note that these observations were based on a limited number of examples, with ir- regular annotation quality and three lines of text comprising 14 and 138 instances, respectively. Ad- ditionally, it should be mentioned that a median CER value of zero was observed in images featur- ing three lines of text and in those with annotation irregularities. This suggests that although these particular image types present challenge to the model, it exhibits proficient capability in handling the majority of such exceptions.
Our approach achieves better results across all key evaluation metries, including CER (0.004), weighted CER (0.005), and the percentage of cor- rectly predicted labels (96.5%). Additionally, our model exhibits enhanced stability and robustness, with the lowest standard deviation (0.029) among all competing models. Remarkably, our model out- performs fine-tuned Transformer- based alternatives like TrOCR and fine-tuned conventional alterna- tives like RI8+LSTM. Additionally, it surpasses the performance of videly-adopted OCR and HTR tools designed to handle handwritten Occitan.
Note that the fine-tuned TrOCR, despite having a relatively high weighted CER (0.083), achieves 94% share of correctly predicted labels. However, the remaining 6% (incorrect) predictions exhibit remarkably high CER values, shown by the large standard deviation. We attribute this to the fact that it was pre-trained with millions of printed and hand- written material, which included sequences that were considerably longer than those found in the smaller Old Occitan dataset. On the other hand, the more conventional fine-tuned RI8+LSTM model shows relatively low weighted CER values (0.043) but a rather constrained proportion of accurately predicted labels (57.2%), especially when com- pared to the Transformer-based alternatives.
Further, we observe that in cases where predic- tions were correct, the label length averaged 15.7 characters, while incorrect predictions had erage label length of 18.3 characters. Additionally, since the impact of beam width on the decoding quality was low, we decided to use greedy search as a decoding strategy for efficiency. Finally, we observed that unsurprisingly, higher weighted CER values were associated with less frequently appear- ing characters.
5.4 Ablation study
In order to assess the influence of data augmenta- tion and synthetic data independent of each other, the Swin + BERT model trained solely real images as baseline for our ablations. We systematically introduce the individual steps and finally combine them for our final model (cf. Tab. 4). The number of synthetic images appears to be the most influential factor, highlighted in bold. Specifically, incorporating 900,000 synthetic exam- ples resulted in reduction of the weighted CER by 0.02 and in an increase of correctly predicted labels by 13.2 percentage points. The impact of random rotation and dilation on real images was found be comparably low, improving the weighted CER by 0.002 and the share of correctly predicted labels by 2.4 percentage points. Note that the effect of
5.3 Error analysis
To comprehensively assess the strengths and weak- nesses of our model, we conducted an error analy- sis across various dimensions, including the num- ber of handwritten lines, annotation quality, decod- ing strategy, label length, and character represen- tation in the labels (cf. Figures in Appendix H for visualization). Moreover, we highlight the ten worst performances and the top ten performances complex images (Figures 12 and 13 in Appendix H). Overall, the results suggest a negative correla- tion between the number of text lines and the model performance. When processing a single line of text, the model achieved a mean weighted CER of 0.003 and accurately predicted 97. 1% of the labels. How-

---

Model benchmarking
CER over 163 test examples
1.00
0.75
00.000
0.25
EasyOCR
Tesseract OCR
PacdleDCR
Google Cloud Vision R18+LSTM (fne-suned) TrOCR (fne-tuned)
Ours
Modw
Archiledtre
Meen
MeantWeighed
Median
Max
SalDwe
Cornectiy predicted label (%)
EasyOCR
ResNett CTC
0.246
0.377
021
4.00
0.200
6.2
Tesseract OGR
CNN LSTMM
6,183
0.178
0.13
7:18
0,255
17.7
PaddeOCR
GRNN CTC
0,206
0.197
0.10
3.00
0.244
20.8
Google
(commordia))
R18-LSTM luned)
ResNECt-CNN-LSTM
0.061
TIOOR (ina-tuned)
Transformer
0,970
0.70
04.0
Oure
Transformer
0.004
0.00
0.00
0.53
0129
98.5
Figure 4: Comparison of test set performance between model and other OCR and HTR tools supporting handwritten Occitan language. To aid visualization, the y-axis is limited to values between 0.0 and 1.0. Thus it is worth noting certain models may have CER values greater than 1.0 (which are cut off here).
the initial 180,000 images accounts for half of the increase observed for the 900,000 synthetic images.
from 91.6% to 92.3% (cf. Fig. 15 in Appendix I). This data set can also be accessed publicly https://huggingface co/misoda
CER (Weighted)
Correctly predicted (%)
Architecture: Swin + BERT
Discussion and Outlook
Only Real
81.1%
Only Real (w/ augmentation)
0.024
83.5%
HTR encounters substantial challenges in low- resource languages, hindering language diversity and inclusivity. These challenges arise from di- verse handwriting styles, variable image quality, the absence of standardized vocabulary, and lim- ited training data. In response, we propose tomized end-to-end tool specifically designed for the Old Occitan dictionary. However, our approach can also serve as a blueprint for similar systems for other (extremely) low-resource languages.
w/ Synthetic (180.000)
0.016
87.0%
w/ Synthetic (900,000)
0.006
94.3%
Final model
0.005
96.5%
Table 4: Impact of (amount of) synthetic data and aug-
mentation (most beneficial step highlighted in bold).
5.5 Inspecting domain shift
To assess the generalization capabilities of our model, conduct tests on different Old Occ- itan data set (cf. Fig. 14 in Appendix I). This data set consists of 316 images containing upper-case single lemmas with mean length of 7.6 charac- ters and without any graphical variants or arrows. The text is written on green, yellow, or red back- grounds. The predictions of our model undergo automatic post-processing to remove any predicted arrows any predicted text after arrow. The results exemplify good generalization of our model in this relatively similar data set, with weighted CER values ranging from 0.010 to 0.011 and a percentage of correctly predicted labels ranging
To further enhance the performance and capa- bilities of our model, there are several key areas that can addressed. Firstly, investigating penal- ization techniques the context of low-resource languages is a promising path to prevent overfitting, improve model generalization, and reduce poten- tial biases in the predictions (Steiner et al., 2021) 1). Secondly, considering smaller architectures, such as DistilBERT (Sanh al., 2019), is strategy worth exploring. Our results have shown that rela- tively small models achieve the best performances on the Old Occitan data set, indicating that these architectures can offer improved efficiency with-

---

out compromising performance. Additionally, in- corporating SOTA segmentation techniques, such as YOLOv8 (Jocher et al., 2023), SAM (Kirillov et al., 2023), and SEEM (Zou et al., 2023), can potentially bolster the encoding efficiency of the model. This strategy may prove beneficial in sce- narios where data availability is constrained, aiding the training process and potentially enhancing per- formance. Finally, we would like to explore further data augmentation techniques, such as GAN-based approaches, as described by Guan et al. (2020), or grid-based distortion augmentation, as proposed by Wigington et al. (2017), in combination with our own method.
consistently recognize these cases accurately. To address the model's weaknesses, we suggest fur- ther data augmentation to enhance its performance in underrepresented classes.
Secondly, the model generalization potential seems limited due to its training a specific data structure. The training data primarily consists of upperease text and predominantly includes word pairs with an arrow in between. This specialization may hinder the ability to generalize and accurately process other types of text and language structures. However, it is worth noting that the model has shown good performance (after post-processing) on different Old Occitan data set, demonstrated in Figure 15.
7 Conclusion
Furthermore, it is important to acknowledge that, except for TrOCR and R R18+LSMM, the tools used for comparison in this study were not specif- ically fine-tuned on the same data set as our pro- posed Transformer-basee approach. This distinc- tion raises questions about how the performance of these tools would compare when applied to the same data. The benchmarking conducted in this study includes tools specialized OCR that also support HTR. A more comprehensive comparison would involve tools specialized in HTR and fine- tuning them on our own data. Finally, it is im- portant to note that our approach has not been ex- plored for other low-resource languages. There- fore, its performance, when applied to languages beyond the scope of our study, remains uncertain. However, other experiments have shown the good performance of similar Transformer brasse models, such as their application on Medieval Latin (Ströbel et al., 2022).
In this paper, we present Transformer-bast based model for the HTR of Old Occitan, a low-resource language. Our approach involves training various configurations of vision encoders in conjunction with language decoders. Through our experiments, we have determined that the most effective combi- nation is a Swin + BERT model, as it achieves su- perior performance in terms of weighted CER and share of correctly predicted labels. Furthermore, it has fewer parameters, requires less training time, and utilizes fewer computational resources. To im- prove the performance of our model, we applied elaborate data augmentation techniques, reducing the weighted CER from 0.026 to 0.005 and im- proving the share of correctly predicted labels from 81.1% to 96.5%. As a result, our approach sur- passes various open-source and commercial HTR and OCR tools. During the evaluation of our best model, we observe that it tends to be less effective on images that contain long character sequences, annotation irregularities, multiple lines of text, and rare characters. However, it demonstrates com- petency in handling complex cases (cf. Fig. 13 in Appendix H). Overall, our approach presents a promising HTR solution for Old Occitan, and nurtures further research to explore applications to other languages and data sets.
Ethics Statement
We affirm that our research adheres the ACL Ethics Policy. This work involves the use of pub- licly available datasets and does not involve human subjects or any personally identifiable information. We declare that we have no conflicts of interest that could potentially influence the outcomes, interpre- tations, or conclusions of this research. All funding sources supporting this study are acknowledged in the acknowledgments section. We have made our best effort document our methodology, experi- ments, and results accurately and are committed sharing our code, data, and other relevant resources to foster reproducibility and further advancements in research.
Limitations
Our approach has several limitations that can be addressed improve its efficiency further. Firstly, the character recognition performance shows high variability, especially for less frequently observed classes, such as the equal sign, brackets, apostro- phes, and French diacritics. The model struggles to

---

Acknowledgements
Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Wei- wei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang. Qingqing Dang, and Haoshuang Wang. 2020. Pp-ocr: A practical ultra lightweight ocr system.
We wish to extend our thanks to the Bavarian Academy of Sciences and the ALMA project for their support and granting us access to the hand- written material. We would like to express ap- preciation to Paula Ruppert for her invaluable assis- lance in preparing and curating the dataset, and to Meiningwei Li for his essential technical support while evaluating CNN+LSTM- basedd approaches. This work has received partial funding from the Deutsche Forschungsgemeinschat (DFG, German Research Foundation) as part of BERD@NFD) under grant number 460037581.
Alex Graves, Santiago Fernández, Faustino J. Gomez, and Jürgen Schmidhuber. 2006. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd international conference Machine learn- ing.
Mingyang Guan, Haisong Ding, Kai Chen, and Qiang Huo. 2020. Improving handwritten ocr with aug- mented text line images synthesized from online handwriting samples by style-conditioned gan. In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 151-156. IEEE
References
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2 (xKiv:21000082: 08254.
Killian Barrere, Yann Soullard, Aurélie Lemaitre, and Bertrand Coüasnon. 2022. A Light Transformer- Based Architecture for Handwritten Text Recognition, pages 275-290.
AI Jaided, 2020. Easyocr. Retrieved October, 9:2020.
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. 2023. YOLO by Ultralytics.
Théodore Bluche and Ronaldo O. Messina. 2017. Gated convolutional recurrent neural networks for multilin- gual handwriting recognition. 2017 14th IAPR In- ternational Conference Document Analysis and Recognition (ICDAR), 01:646-65 651
Philip Kahle, Sebastian Colutto, Günter Hackl, and Gün ter Mühlberger. 2017. Transkribus service plat- form for transcription, recognition and retrieval of historical documents. In 2017 14th IAPR Interna- tional Conference on Document Analysis and Recog- nition (ICDAR), volume 04, pages 19-24.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. 2017. Emnist: Extending mnist handwritten letters. In 2017 international joint conference on neural networks (JJCN), pages 2921 2926. IEEE
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment anything.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedingy of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Vladimir I Levenshtein al. 1966. Binary codes capa- ble of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pages 707-710. Soviet Union.
Minghao Li, Tengchao Lx, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 2021. Trocr: Transformer- basse optical character recognition with pre-trained models.
Daniel Hernandez Diaz, Siyang Qin, R. Reeve In- gle, Yasuhisa Fujii, and Alessandro Bissacco. 2021. Rethinking text line recognition models. CoRR, abs/2104.07787.
Ze Liu, Yutong Lin, Yue Cao, Han HU, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer. Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
U-V Marti and Horst Bunke. 2002. The iam-database: english sentence database for offline handwrit- ing recognition. International Journal on Document Analysis and Recognition, 5:39-46.

---

Jamshed Memon, Maira Sami, and Rizwan Ahmed Khan. 2020. Handwritten optical character recog- nition (oer): A comprehensive systematic literature review (slr).
Hugo Touvron, Matthieu Cord, Matthijs Douze, Fran- cisco Massa, Alexandre Sablayrolles, and Hervé Jé- gou. 2021. Training data-efficient image transform- distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR.
Johannes Michael, Roger Labahn, Tobias Grüning, and Jochen Zöllner. 2019. Evaluating sequence-to- sequence models for handwritten text recognition. 2019 International Conference Document Analy- sis and Recognition (ICDAR), pages 1286-1293.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez. Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- lems, NIPS' 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Jeroen Ooms. 2023. tesseract: Open Source OCR En- (website) (devel).
gine, Https://docs. www.ooeccssoossseeeee wwwpessi..ogg/tesssssseeeeee http:///ghhhbbooooo///ppps///sss
Curtis Wigington, Seth Stewart, Brian Davis, Bill Bar- rett, Brian Price, and Scott Cohen. 2017. Data augmentation for recognition of handwritten words and lines using cnn-lstm network. In 2017 14th IAPR international conference document analysis and recognition (ICDAR), volume I, pages 639-645. IEEE.
P. Pansier. 1974. Histoire de la langue provençale: à Avignon du XIle au XIXe siècle. Number Bd. in Histoire de langue provençale: à Avignon du XIle XIXe siècle. Slatkine Reprints.
Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. 2014. Dropout improves re- current neural networks for handwriting recognition. In 2014 14th International Conference on Frontiers Handwriting Recognition, pages 285-290.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. 2023. Segment everything everywhere all at once.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019, Language models are unsupervised multitask learners.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. In NeurlPS EMC2 Workshop.
Olga Scrivner and Sandra Kübler. 2012. Building an old occitan corpus via cross-language transfer. In KONVENS, pages 392-4001
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (ooluee 1: Long Papers), pages 7725-125, Berlin, Germany, Association for Computational Lin- guistics.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. 2021. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106. 10270.
Phillip Strübel, Simon Clematide, Martin Volk, and Tobias Hodel. 2022. Transformer-baset htr for his- torical documents.
Karanrat Thammarak. Prateep Kongkla, Yaowarat Sirisathitkul, and Sarun Intakosum, 2022. Compar- ative analysis of tesseract and google cloud vision for thai vehicle registration certificate. International Journal of Electrical and Computer Engineering, 22: 1849-1858.

---

Appendix
A Experimental setup
Encoder
Decoder
# Parameters Eucoder Decoder
Training Setup
Examples
Epochs
Pre-Train
180,000
85.7M 114.3M
Only
Fine-Tune
213,308
BEIT
Pre-Train
180,000
BERT
85.7M 114.5M
Only Real
33,308
Fine-Tune
213.308
Pre-Train
180,000
25
GPT-2
86,4M 114.3M
Only Real
33,308
50
Fine-Tune
213,308
50
Pre-Train
180,000
25
BERT
86.4M 114.5M
Only Real
33,308
Fine-Tune
213,308
Pre-Train
180,000
GPT-2
86.4M 114.3M
Only Real
33,308
213,308
VIT
Pre-Irain
180,000
BERT
86.4M 114.5M
Only Real
33,308
Fine-Tune
213,308
Pre-Train
180,000
GPT-2
27.5M 114.3M
Only Real
33,308
Fine-Tune
213,308
50
Pre Train
180,000
25
BERT
27.5M + 114.5M
Only Real 33,308
50
Fine-Tune 213,308
Table 5: Combination of different vision encoder and language decoder models trained synthetic and/or real data. All the experiments were performed with GPU NVIDIA Tesla V100 (16 GB).
B Hyperparameters
Parameters for Image Pre-processing
Parameter
Value
Cropping
Contrast Factor
Sharpness Factor
Brightness Factor
3
Table 6: Parameters for image pre- neepproesssing. We used the defaults from the PIL (9.2.0) library for all non-reported
values.
Parameters for Training with and without Augmentation
Parameter
Value
Seed
Optimizer
AdamW
Encoder
(BEiT, DeïT, ViT, Swin) {GPT-2, BERT)
Decoder
Batch Size (Train, Validation & Test)
48
Table 7: Parameters for training to reproduce our results. For all non-reported values, we used the defaults from the transformers (4.25. library.

---

Parameters for Natural Language Generation
Parameter
Value
Max Length
200
Early Stopping
True
No Repeat Ngram Size
100
Number of Beams
(1. 4, 10, 15, 30, 50)
Table 8: Parameters for natural language generation. We used the defaults from the transformers (4.25. library for all non-reported values.
C Character Inventory
#
Character
Frequency
Relative Frequency (%)
#
Character
Frequency
Relative Frequency (%)
A
88295
12,6753
38
104
0.0149
2
@
83268
11.9536
39
103
0.0148
3
E
77563
11.1346
40
77
0.0111
R
61007
8.7579
41
56
0.0080
5
S
46037
6.6089
42
45
0,0065
I
45298
6.5028
43
43
0.0062
7
N
39317
5,6442
44
Ó
43
0.0062
8
O
33621
4.8265
45
39
0.0056
L
26316
3.7778
46
30
0.0043
10
T
26061
3,7412
47
28
0,0040
D
21790
3.1281
0.0027
12
M
21058
3,0230
49
19
0.0027
13
C
21052
3,0221
50
18
0.0026
14
B
16061
2,3057
51
0.0017
15
U
16004
2,2975
52
9
0,0013
16
15598
2,2392
0.0009
17
F
12039
1,7283
54
4
0.0006
18
G
10935
1.5698
55
4
0.0006
19
H
9348
1.3420
0.0006
20
Z
8583
1.2321
57
3
0.0004
21
V
4771
0.6849
58
3
0.0004
Y
3693
0.5302
0.0004
23
2608
0.3744
60
3
0,0004
24
Q
1458
0.2093
61
0.0004
25
654
0.0939
62
0.0004
26
514
0.0738
63
A
2
0.0003
27
X
477
0.0685
64
0.0003
28
2
373
0,0535
65
0.0001
29
372
0.0534
66
0.0001
30
Ç
371
0.0533
67
0.0001
31
280
0.0402
68
0.0001
32
280
0.0402
0.0001
33
190
0.0273
70
0.0001
34
I
189
0.0271
71
0.0001
35
171
0.0245
72
0.0001
36
125
0.0179
73
W
0.0001
37
É
121
0.0174
Table 9; Character frequency and relative frequency observed in the labels.

---

D Augmentation of real images
BAB77 SHE A CAPTICHE
BABTISHE SHE > BAPTISHE CHE
BAB71 SHE > BAPTISHE
BAB71 CHE > BAPTICHE
(a) Original image
(b) Pre-processed and augmented images
Figure Examples of data augmentation on real images. Random dilations and rotations are applied to the pre-processed images.
Parameter
Value
Seed
42
Rotation angle
[-4, -3, 2,
Dilation factor
[1, 3,
Table 10: Parameters for data augmentation of real images. used the defaults from the PIL (9.2.0) library for all non-reported values.

---

E
Generation of synthetic corpus
Name
Genre
# Tokens
Lines -992
Literary Prose in verse (13th-century)
6,878
Flamenca (lines 993-2133)
Literary Prose in verse (13th-century)
8,132
Boece
Verse (10th-century)
7,054
Pansier
Literary Prose and Administrative Texts (10th to 15th-century)
49,067
Total
71,131
Table 1I: Compilation of Old Occitan corpora for data augmentation purposes.
Real corpus (Excerpt) Carta audir descebre ne kanonegue oi maiso maison sanct adenant nom castel honor adenant castel territorio...
Synthetic corpus (Excerpt)
CARTA@AUDIR
DESTEEEE
CARTA @NE
CARTA&ALTRE
CARTA&KANONEFGUR
CARTA@OI
@AATO
CARTA&MAISON
CANTA&SANCT
CARTAGADENANT
CARTA@NOM
CARTARCASTEL
CARTA
CARTA
CAENANAD
CARTA@CASTEL
CARTA
AUDIR@CARTA
DESCERE
AUDIR@NE
AUDIR
KUDIR@KANONEGUE
AUDIR@OI
AUDIR&MANE
AUDIR@MAISON
AUDIR&
ADENANT
AUDIR@NOM
AUDIR@CASTEL
HUDIR
AUDIR@AVER
AUDIR&ADENANT
AUDIR@CASTEL
CASTEL&HONOR
CASTELGAVER
CASTEL&A ADENANT
CASTEL ATRRRTTRRO
TERRITORIO@CARTA
TERRITORIO@
"ERRITORIO@DEESEERRE
TERRITORIO@NE TERRITORIOSALTRE
TERRITORIORKKOONEGUEE
TERRITORIO@01 TEERITORIO@MAISO
TERRITORIO
TERRITORIO@SANCT
TERRITORRIO
TERRITORIO@@NO
TERRITORIO@CAATEEL
TERRITORIOOHONOOR
TERRITOORIQ@AVER TERRITORIO*ADENANT
Table 12: Example of synthetic corpus generated from random combination of Old Occitan words in uppercase. The sign represents an arrow linking the graphical variant with its lemma.

---

F
Generation of synthetic images
Parameter
Value
Seed
42
Pixel distance
10,1,223,,55,,,,,,55
(between subsequent characters)
Image height
3,42, (...4,,,)"miindimm. -imiinddmm_erttiaall
Image width
(1,2, ""min_dim_horinee siagghh_sineeimaeeee lingge -2, 2. 5, [False, True)
Starting point first character
Rotation angle
Dilation factor Multiline
Table 13: Parameters for generation of synthetic images. We used the defaults from the PIL (9.2.0) library for all non-reported values.
Generation of synthetic images
1: function SINGLE Bingle_linE_ IMG(encoded_ _word, pixel_data labels, mncoding_dicctonaryy
2: Initialize an empty area for the image, based on the length of the encoded_word.
3: for number in encoded_ww _oor do:
4: Select randomly a row from labels matching with number.
5: Fill the respective empty area with the EMNIST image from the pixel_data row.
6: Add random shifts to emulate overlapping and variable distance between characters.
7: end for
8: Normalize the image and return it.
9: end function
function WORUS2ING(teRt__srrnng,,
pixel_data, labels,
pncodingg.ictiionary
multiline_flag
12:
Assign unknown characters the text_string with a special token [UNK]. Encode the text_string with the encoding_ dictionary.
13:
14:
if multiline_ lfllinneegagggga == False is specified then
15:
Call the SINGLE L.INE_IIGG IMG function with the encoded word and return the resulting image.
16:
else
17:
Create two encoded word lists for the two lines. Call the SINGLE BINGLL_LINE___L L___LIE___MMM_ IMG function for each line and stack the resulting images vertically. Reshape the final image the appropriate dimensions.
20:
end if
21:
Apply random rotation and dilation.
22:
Convert the modified image to an array and return it.
23:
end function
Algorithm 1: Generation of synthetic images for data augmentation.

---

G
Character Encoding with byte-level BPE Tokenizer
#
Token
Encoding
Token
Encoding
cls_token
[0]
40
N
[39]
2
pad_token token
[1]
41
O
[40]
3
eos cos_token
[2]
42
P
[41]
[3]
43
Q
[42]
5
[4]
44
R
[43]
6
[5]
45
S
[44]
7
[6]
46
T
[45]
8
[7]
47
U
[46]
9
[8]
48
V
[47]
10
19]
49
W
[48]
11
[10]
50
X
[49]
[11]
51
[50]
13
[12]
52
Z
[51]
[13]
53
[52]
15
[14]
54
1
[53]
16
[15]
55
[54]
17
[16]
56
A
[58, 631
18
[17]
A
[58, 64]
A
19
[18]
[58, 65]
20
[19]
59
[58, 66)
21
[201
60
CCC
[58, 681
22
[21]
61
[58, 69)
23
[22]
62
É
[58, 70]
24
[23]
63
É
[58, 71]
25
[24]
64
[58, 72]
26
[25]
65
[58, 73J
i
27
[26]
66
[58, 74]
28
[27]
67
Ñ
[58, 75]
29
C
[28]
68
Ó
[58. 76)
30
[29]
Ó
[58. 77J
31
[30]
70
Ô
[58, 78]
32
F
[31]
71
Ö
[58, 79]
33
G
[32]
72
Ú
[58, 80)
34
H
[33]
73
Ú
[58, 81]
35
[34]
74
Y
[59, 56]
36
[35]
75
Ñ
[60, 57, 67]
37
[36]
[61, 63, 55]
38
[37]
77
[62]
39
[38]
Table 14: Four encodings are required for classification, padding, seneeenee and unknown tokens. Furthermore. observed 73 characters in our labeled material. With byte-level BPE approach, the tokenizer required 82 encodings (integers from 0 to 81) encode the aforementioned 77 tokens.

---

H Error analysis
by
CER sue 1123 Inc compin.
0.75
0.20
Lines
Pormpii
Mear
Mewe/Weegtrwi
Medan
Sin
Max
BtDey
Correctly predicsed labels (M)
940
0:008
0:008
3.59
0.645
94.9
Figure 6: Test set performance our model by the number of handwritten lines.
performance by annotation quality
celt over comrpes
0.25
Imegee
Quality
Yragleer Centers on ere paedd venicoe
Duslty
Stamplex
Mean
Meentweeghee
Median
Mr
Nax
StdDw
Correctly predicted labelo (%)
Imegurar
133
0.230
0:43
C488
0.255
B1.2
Figure Test set performance of our model by annotation quality.

---

Model performance by width
GER over (163 excrppee
0.73
GEB
0.23
Beam width
Veanitolggoed
Gorrectlyy
03406
0,00620
0.53
00426
0,00649
0.53
0.0377
66
00424
0.00643
0.53
0.0211
5
0,00427
0,00647
0.55
0.0313
66.5
0.00409
0.53
0.0298
Figure 8: Test set performance by beam width.
Label length distribution by prediction class
Messsrrd 4161 lest examples
Label length
Prodiction class
Calegery
Examples
Median
StdDav
5.510
Figure 9: Label length distribution of correct and incorrect predictions.

---

Mean Weighted CER Label Length
0.4
0.3
Weighted
02
0.1
19
32
114
265
147
60
51
249
311
125
10
15
20
25
30
Label Length
The numbers represent the label frequency per length
Figure 10: Mean Weighted CER by label length.
Moan Weighted CER vs. Character froquency labels
0.
03
Mean Weighted CER
02
Ez Uot J 88'
9.
1000
2000 Count
3000
4000
'A-Z special
Figure 11: Mean Weighted CER by character representation in the labels.

---

(a)
(f)
*BETA
* ABETA
FARIZEU TARISTIBU
FARISTEU
EISSALABETA
DEISSALARETAR
(b)
(g]
APONCH
PONH
DESPERSEBUDAMENT -DESPERCABUDAMEN
@ A PONCH
-EU-
(c)
(h)
-3 RECEBRE
BALESTRE
BALESTIER BALESTA
(ARCEBO -S -NE)
(d)
(i)
ATREMPAMENT men) )
ARTAL (WALTAL)
AuTAL
- ATEM?RA RA DAMEN N
i)
SALIER -ISSUGUA
BARROYLH -B-BARROLH
SSALIER-ENUGA
LY S.V. VERROLH
Image
Label
Prediction
CER
(a)
*BETA@ -ABETA RISSAA--ISSALAABAAAA DRISSALABEE ABETAR
*BETA PEETTA ABETAR ABETAR
0.52
(b)
A & PONH
APONCH & PONH
0.48
(c)
ARCEBOSNEIARCEBO S .N) @RECEBRE
ARCEBOSNE RECEBRE
0.47
(d)
CREEMPANENTEEATEEMPP.MMEEE @ ATEMPRADAMEN
ATREMPAMENTe DEMMER
0.38
(c)
BARROYLH@ BARROLH -LY S.V. VERROLH
BARROILHG BARROLHER
0.53
(
FARIZBU & FARISTIEU CESTTIIUUFARSSTE FARISTEU
FARISTIEU FARRSTIIU
0.35
(g)
DESPERSEBUDDMENT & DESPERC- EU-BU BU-BUDAMEN
RESPERSEBUDAMEN & DESPERSAMEN
0.23
BALESTRE@ BALESTIER BALLESAA
BALESTRÉ BALESTIER
0.35
ARTALI ALTAL AUTAL
ARTAL AUTAL
0.42
i
SALIER-ISSUGUA SAALLSSALLSLLLUSEE
SALIER
0.36
Figure 12: Worst predictions (based CER) the Old Occitan

---

(a)
SOUTEMACHEUTE
FATTLLAAEEN-TACHILLA MEN
MALLTE-AANEUTA
(b)
SOBRE-AAONDOZATTEN
BATALHAYRI2
BATALHAI-
RITE
- SOORARONDO2AATE
(c)
(h)
(NSINUATTION))
[ANONSIAMEN
ANONCIAMEN
ENSINUACION
(d)
(i)
*DESANAHORAT (Pn) DEZENAMORAT
PASSO-CORDO PASSACORDA
(e)
i
FREITOR - REFECTOR Rn
PPRROOPANNNNS PAROPIANAL
Image
Lahel
Prediction
CER
(a)
SOUTTEMALHEUTEESSS SOLTAAAAN-AEEETE MANLEUTA
SOUTEMALLHHUTTESSOLLIY MAA.LLALUUTTTTSSLLLLAAAAAALAUUU MANLEUTA
(b)
SOBRE-AONDOZAMENTOO SOBRABONDOZAMEN
0
(c)
INSINIAATIOIN
0
(d)
IDEEXNAMMORATBDDEENNNOO
0
(e)
"PARROPIANANSGPPPPPPPP
APXARROPANNNNSGPPRPPPPPAAN
0
(f)
FATTIHHAMENNFRRCHILAAAAW
KATTILHAMENNPPACCHHLAAAAA
(8)
BATALHAYRIZE BATAIZAI--IZZ
BATALHAYRIZZG BATTI
(h)
( JANONSIAMENJ@ ANONCIAMEN
(ANONSIAMENIê ANONCIAMEN
PASSO-CORDO PASSAC
(
*FRETTOR @REFECTOR
ORRETTORRRFFCCTI
Figure 13: Best ten predictions (based on CER) complex images from Old Occitan test set (complexity here refers to images including multiple lines, irregular annotations, and/or characters, such parentheses and hyphens).

---

I Performance on different Old Occitan data set
GAVET
GAZARD ON ADOR
GEISSARIA, 2
and
Figure 14: Examples images from a different Old Occitan data set, comprising 316 images with simple lemmas and different background colors.
Model performance on Old Occitan data sets
GE1
00.00
Projeci
Data set
Date
Doampee
Mean
MeeneWeggroe
Medien
Min
Max
SkiDev
Correctly predicted labels
Blus (DOM Projecti
4165
0:004
0.008
0.590
0.029
se
Yellw
0.010
0.250
Green
179
0.014
0.01*
0,250
0.048
51,8
Figure 15: Test set performance comparison of our model on the Old Occitan data set (DOM Project) vs. different Old Occitan data set, comprising 316 images with simple lemmas different background colors. predictions underwent automatic post-processing step, consisting of keeping all the generated tokens until the first arrow was predicted.

---

List of Acronyms
BEIT
Bidirectional Encoder representation for Image Transformers
BERT
Bidirectional Encoder Representations from Transformers
BPE
Byte Pair Encoding
CER
Character Error Rate
CNN
Convolutional Neural Network
CRNN
Convolutional Recurrent Neural Network
CTC
Connectionist Temporal Classification
DeiT
Data-efficient image Transformer
DistilBERT
Distilled version of BERT
DOM
Dictionnaire de l'occitan médiéval (Old Occitan dictionary)
GPT-2
Generative Pre-trained Transformer 2
HTR
Handwritten Text Recognition
LSTM
Long Short-Term Memory
NLP
Natural Language Processing
OCR
Optical Character Recognition
RNN
Recurrent Neural Network
SAM
Segment Anything Model
SEEM
Segment Everything Everywhere with Multi-modal prompts all at once
SOTA
State-of-the-Art
Swin
Shifted Window Transformer
TrOCR
Transformer OCR
VIT
Vision Transformer
YOLO
You Only Look Once


Reviews:
[{"review_id": "w3FASJOxgs", "reviewer": "Reviewer_GHMV", "summary": "", "questions": "Is the fine-tune  TrOCR model trained on same data as other  (Swin + BERT) model, if not then what's the reasons  and how is this comparison fair ?", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the time and effort you have put into reviewing our work. Below, we answer your questions, address your concerns, and try to clear up possible misunderstandings. We will do this point by point, first copying the respective statement from your review and then providing our answer/clarification, we hope this makes reading our rebuttal for you as convenient as possible.\n\n- _Paper Topic And Main Contributions:_ This paper presents a data augmentation technique [...] 4 vision encoders and 2 textual encoders.\n    - _Our Answer:_ Data augmentation is indeed a fundamental step, but not the focus of the paper. Our core contribution is that we evaluate eight combinations of encoder-decoder architectures as a first step, and provide insights through our experiments into the benefits of a combination of a Swin image encoder with a BERT-based language decoder (trained from scratch) in the context of a low-resource language. In a second, auxiliary step, we enhance its performance through different data augmentation techniques and provide insights about its performance compared to well-established open-source and commercial alternatives. Finally, we hope to contribute to the HTR community by making our model, datasets, and code publicly available on GitHub/Hugging Face.\n\n- _Reasons To Accept:_ The final model [...] outperform open-source solutions.\n    - _Our Answer:_ One important further point that we would like to add is that we also outperform a well-established closed-source solution (Google Cloud Vision, see Table 2 and Figure 4) that has proven its effectiveness in the context of low-resource languages.\n\n- _Reasons To Reject 1:_ Hard to identify the core novelty proposed in the technique: \n    - _Our Answer:_ It is correct that we do not introduce a novel architecture. Instead, the novelty of our works lies in the insights gained through exhaustive experiments of model combinations (Table 3) and benchmarking (Figure 4) in the specific context of this particularly low-resource language. In our ablation study, we also measure the contribution of straightforward vs. more elaborate data augmentation techniques to the performance of our best model (Table 4). Additionally, we believe that the novel resources (curated data, high-performing models, and code) we make publicly available will be a contribution to the community (in particular Digital Humanities and Linguistics).\n\n- _Reasons To Reject 2:_ it seems, that smart data augmentation for this language is a core contribution. However, all the proposed image transformations are well-known [...] hard to contextualize the contribution.\n    - Our Answer: These data augmentation techniques play a valuable role in addressing data scarcity and enhancing our models' performance, though they are not the central focus of our work and therefore we did not include a comprehensive benchmarking of this particular step. Nevertheless, the ablation study (Table 4) provides a simple comparison of straightforward techniques such as dilation and rotation against the more elaborate generation of EMNIST-based synthetic images. This contrast highlights the notable advantages of the latter, particularly as the volume of synthetic images increases.\nStill, we are aligned in the view that delving into and evaluating alternative data augmentation techniques (like GANs or Random Warp Grid Distortion -as suggested by Wigington et al.), is an important step to gauge their impact on model performance and we will consider this in our future research.\n\n- _Reasons To Reject 3:_ In terms of the model all 4 vision encoders and 2 text encoders are used off the shelve, and no new architectural changes are proposed [...] Though there are some other native HTR transformer-based models mentioned by authors like Barrere et al. (2022), the authors don't compare with those.\n    - _Our Answer:_ In addition to our response to \"Reason to Reject 1\", we would like to add that one of our main objectives is to compare the models (as they were published, with their base configuration) and highlight the best mix (with the help of data augmentation) in the context of a low-resource language. As part of our future research, we intend to explore and propose more efficient architectures, including an optimized fusion of the encoder and decoder as well as customized loss functions to balance the information coming from both images and text.\nRegarding the \"Light transformer\" model introduced by Barrere et al. (2022): To the best of our knowledge, the model isn't accessible to the public, thus preventing its inclusion in our comparative analyses. Nevertheless, we have contacted the authors and we plan to include it in the final version upon a positive response from the authors.\n\n- _Reasons To Reject 4:_ Lastly seems like fine-tuned TrOCR [...] is not fine-tuned [...], in such case lower performance is expected. Why doesn't the author fine-tune TrOCR on the same data and test against the best (Swin + BERT) model\n    - _Our Answer:_ This is a very crucial point, and we would like to clarify that we DO FINE-TUNE the TrOCR on the same data as our best-performing model to ensure a fair comparison. The results are reported in the Benchmarking section, along with other architectures (Figure 4).\nIn lines 433 – 443, we discuss the performance of the fine-tuned TrOCR and speculate on the reasons for the performance differences (e.g. we presume that the effect of pre-training with millions of images of English sentences could be related to the generation of rather long predictions - even after fine-tuning with Old Occitan data). We will make sure that this is conveyed with more clarity in the final version of the paper. \n\n- _Questions For The Authors:_ Is the fine-tune TrOCR model trained on same data as other (Swin + BERT) model, if not then what's the reasons and how is this comparison fair?\n    - _Our Answer:_ As stated above, we have indeed fine-tuned the TrOCR using the same dataset as our most successful model. The outcomes of this fine-tuning process have been detailed in the Benchmarking section and visually presented in Figure 4. As a side comment: The performance of TrOCR (without finetuning) in Old Occitan (or in many other languages different from English) is remarkably low, the predictions are usually random English words that have almost no connection to the input handwritten text. This can be verified under the following link: https://huggingface.co/spaces/nielsr/TrOCR-handwritten\n\n- _Reproducibility: 3_\n    - _Our Answer:_  As stated in the paper, our model, datasets, and code are already available in the anonymous GitHub repository (link in the Abstract, line 28), and will be publicly available on Hugging Face upon publication. The appendix also provides a comprehensive list of hyperparameters and settings and all the scripts are available and ready to use. We hope to have cleared the uncertainties on that end and would be grateful if you could take these clarifications into account when reassessing the level of reproducibility of our work.\n\n- _Soundness: 2_\n    - _Our Answer:_ We hope to have answered all the questions to your satisfaction and also to have clarified your concerns about the fairness in our experiments and benchmarking, in particular with regards to the fine-tuning of TrOCR with the same data as our best-performing model. We would greatly appreciate it if you could consider our clarifications while reassessing the soundness of our work. We hold the belief that a rating of 2 (“...major technical/methodological problems”) may have potentially been influenced by the ambiguities surrounding the fairness of our methodology.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper encompasses an exhaustive benchmarking of the performance of  HTR recognition for Old Occitan language using smart data augmentation to deal with scarcity. The final model also shows significance gains from the augmentation as compared with the real data itself helping the model trained with this data to outperform open-source solutions.", "reasons_to_reject": "Hard to identify the core novelty proposed in the technique:\n- it seems, that smart data augmentation for this language is a core contribution. However, all the proposed image transformations are well-known and widely used. Any novel augmentation specific (to this language or any HTR dataset) needs to be clearly stated and compared. Since the paper doesn't compare with any other augmentation (generic to vision or HTR) method, hard to contextualize the contribution.\n\n- In terms of the model all 4 vision encoders and 2 text encoders are used off the shelve, and no new architectural changes are proposed (including the fusion of encoders). Though there are some other native HTR transformer-based models mentioned by authors like Barrere et al. (2022), the authors don't compare with those.  \n\n- Lastly seems like fine-tuned TrOCR which is the closest transformer baseline compared with is not fine-tuned on the target domain dataset (as from line 433-443, author explain domain mismatch), in such case lower performance is expected. Why doesn't the author fine-tune  TrOCR on the same data and test against the best (Swin + BERT) model", "questions_to_authors": "Is the fine-tune  TrOCR model trained on same data as other  (Swin + BERT) model, if not then what's the reasons  and how is this comparison fair ?", "paper_topic_and_main_contributions": "This paper presents a data augmentation technique for the Old Occitan language and benchmarks a transformer model trained with this data against some SOTA HTR models. These transformers employ and test a combination 4 vision encoders and 2 textual encoders.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dBo14CLj8X", "reviewer": "Reviewer_udXX", "summary": "", "questions": "Is the dataset being made public?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the time and the effort you have put into reviewing our work. Below, we answer your questions, address your concerns and try to clear up possible misunderstandings. We will do this point by point, first copying the respective statement from your review and then providing our answer/clarification, we hope this makes reading our rebuttal for you as convenient as possible.\n\n- _Paper Topic And Main Contributions:_ Old Occitan is a language spoken in [...] uses a image encoder followed by a BERT-based decoder. The algorithm is also compared with GPT-2.\n    - _Our Answer:_ This is a very good summary of our work, thank you very much. Please find our comments about reproducibility below :)\n\n- _Reasons To Reject:_ The scope [...] quite limited. It will be good to extend it to handwritten documents.\n    - _Our Answer:_ This is a valid point. However, we would like to mention that the model has shown good performance (after automatic postprocessing) on a further Old Occitan dataset that only contains single words (Figure 14, in the Appendix), demonstrating its effectiveness under a slight domain shift (Figure 15, in the Appendix). In future work, we plan to explore the potential of our approach to handwritten documents of Old Occitan and other languages, and our aim is to effectively expand its capabilities to a wider range. We hope that this work is an additional step for this under-resourced language, paving the way for further advancements.\n\n- _Questions For The Authors:_ Is the dataset being made public? \n    - _Our Answer:_ Yes, our model, datasets, and code are already available in the anonymous GitHub repository (link in the Abstract, line 28), and will be publicly available on Hugging Face upon publication. The appendix also provides a comprehensive list of hyperparameters and settings and all the scripts are available and ready to use.\n\n- _Reproducibility: 3_ \n    - _Our Answer:_ As stated before, all of our data, models, and code are already available in an anonymous GitHub repository and will be made publicly available on GitHub and Hugging Face. We hope to have cleared the uncertainties on that end and we would be very grateful if you could take these clarifications into account when reassessing the degree of reproducibility of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\nThe paper is well motivated and well written. The dataset is described well. Results from the approach is promising.", "reasons_to_reject": "\nThe scope of the handwritten text (capital handwritten letters, two words per card) is quite limited. It will be good to extend it to handwritten documents.\n", "questions_to_authors": "Is the dataset being made public?", "paper_topic_and_main_contributions": "Old Occitan is a language spoken in 11th-16th centuries in France, Spain and Italy. Authors propose a method for OCR of this language from handwritten text and show that their approach beats the state of the art systems.\nData collection and augmentation details are well described although it is not clear if it is made public.\nAlgorithm uses a image encoder followed by a BERT-based decoder. The algorithm is also compared with GPT-2.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HKuALiIbD6", "reviewer": "Reviewer_6eLD", "summary": "", "questions": "Question A: What software did you use for the image processing (lines 183 to 195). PIL for everything?\n\nQuestion B: You say you trained a tokenizer by setting the output tokens to 73. The decoders work on the subword levels, though. How do you justify this decision? --> by continuing reading I kind of got the answer. Ok. But still, why train your decoder from scratch and not use the pre-trained one? Have you tried employing a pre-trained decoder?\n\nQuestion C: This might be provocative (but this is somehow intended): Would you agree that you push your methods quite far for a rather small dataset? Don't you think that similar results would be possible with less effort?\n\nQuestion D: Would you expect better results starting from a French pre-trained LM as a decoder (like CamemBERT?)?\n\nQuestion E: How big is the variation in the handwriting? Do you know how many people have participated in creating the cards?\n\nQuestion F: Would you think it's worthwhile to include random warp grid distortion as a pre-processing step? C. Wigington, S. Stewart, B. Davis, B. Barrett, B. Price, and S. Cohen, ‘‘Data augmentation for recognition of handwritten words and lines using a CNN-LSTM Network,’\n\nQuestion G: Did you use some kind of layout recognition?\n\nQuestion H: I find parts of your synthetic data generation process counter-intuitive. The surface form and the lemma are very very close in maybe 95%(?) of the cases, i.e. there is a certain overlap. With Levenshtein you could even put a number to how close most surface forms and lemmas are. When you create your synthetic examples, they are quite off, most of the times, e.g. CARTA@AUDIR. Wouldn't you think that performance could be enhanced by creating examples which are closer to the real data?\n\n\n\n\n\n\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the time and effort you have put into reviewing our work. Below, we answer your questions, address your concerns, and try to clear up possible misunderstandings. We will do this point by point, first copying the respective statement from your review and then providing our answer/clarification, we hope this makes reading our rebuttal for you as convenient as possible.\n\n- _Paper Topic And Main Contributions:_ This paper presents (in great detail) how to conduct HTR [...], thus allowing for a comparison against other systems.\n    - _Our Answer:_ This is a very good summary of our work, thank you very much.\n\n- _Slight Weakness 1:_ I find the CER graphs (violin plots, I believe?) hard to read.\n    - _Our Answer:_ We will try our best to improve the presentation and description of these plots in the final version of the paper. If you have any concrete suggestions about how we could present them more comprehensively, we are happy to adjust them accordingly. If your comment only concerns the degree of legibility (i.e. figure - or text - is too small?), we will definitely improve this!\n\n- _Slight Weakness 2:_ The authors mentioned this themselves, but they didn't give conventional methods the same chance as they did for Transformers. I would like to see this corrected for the final versions.\n    - _Our Answer:_ This is an interesting point that we would like to explore. Our decision was based on the promising performance of Transformer-based approaches, but we will add a comparison with a conventional method (we believe you refer to CNN+RNN-based methods?). We already looked into this a little and might include something like these options: https://link.springer.com/chapter/10.1007/978-3-031-06555-2_34, https://github.com/mittagessen/kraken or other CNN+LSTM alternatives in the final version.\n\n- _Slight Weakness 3:_ As hinted at in Question H: the process of synthetic data generation is a bit counter-intuitive.\n    - _Our Answer:_ Please see the answer to question H :-)\n\n*Questions For The Authors:*\n\n- _Question A:_ We employed several libraries for image manipulation: PIL modules for cropping, enhancement of contrast and sharpness, rotation and dilations, cv2 for enhancement of brightness, and numpy for managing images as arrays.\n\n- _Question B:_ Tokenizer: This was motivated by the lack of a known set of words and by the observed special characters linked to the specific structure of this project, such as numbers, French diacritics, arrows, punctuation marks, etc. Additionally, we considered that it would be easier for the model to predict fewer classes and provide the model with more examples per class (partially through data augmentation). For this reason, we specified the byte-level BPE to be on a character level, as illustrated in Table 14.\nPre-trained decoder: In previous work (recognition of handwritten lemmas), we pre-trained our language decoder but the results showed a very marginal improvement (and sometimes even a negative effect on model performance). We presume that the contribution of a pre-trained decoder could be more evident in the context of handwritten documents where the context, grammar, and language structure might play a more relevant role, or when the data used during pre-training are similar to the data of the downstream task. An additional factor to be considered is that Old Occitan is a non-standardized language. This is exemplified by the existence of graphical variants/surface forms, so probably a pre-trained decoder would struggle with these (minor) deviations from the lemmas.\n\n- _Question C:_ The selection of different steps stemmed from several factors: the notable performance of Transformer-based architectures in HTR, the potential to merge diverse image encoders with language decoders (motivating the question about the best fit for our setting), and the limited availability of data (motivating different data augmentation efforts). Nevertheless, we believe that there is room for exploring numerous simplifications in forthcoming projects. For instance, we point to the study by Barrere et al. (2022) as an illustration, where they introduced a considerably smaller architecture (a 6.9 million parameters Transformer), yielding marginally inferior results compared to their baseline (a 100 million parameters Transformer).\n\n- _Question D:_ This could be interesting to explore. French and Occitan are close languages that share numerous words and characters, and a pre-trained language model such as CamemBERT could be beneficial to our project. We also consider incorporating resources from Catalan, as it shares many similarities with Old Occitan. As stated above, our tests with pre-trained language decoders have yielded mixed outcomes in the past. We presume that this observed uncertainty around the contribution of pre-trained decoders can be explained by the fact that the data used during pre-training still exerts a large influence even after fine-tuning for our (sometimes very specific, different, and data-constrained) downstream task. We intend to study this in more detail. Thank you for your suggestions!\n\n- _Question E:_ We don't have a record, but from our labeling process we have observed around 15 participants. Regarding the overall variation, we would say that over 80% of the cards follow the structure depicted in the first row of Figure 1, and the primary sources of variability identified are: stroke thickness, ink color, font size, image size, scanning quality, noise/errors, and number of lines with text.\n\n- _Question F:_ This seems to be a promising step that we could include in future work, thank you! During data augmentation, we tried to generate word pairs that were not necessarily present in the original data to increase the variability of the training examples. However, applying a random warp grid distortion step to both existing and new (synthetic) word pairs could be definitely beneficial. The paper also highlights good results of RWGD across multi-author datasets of different languages, centuries, and various degrees of legibility, so we will consider this approach in future projects.\n\n- _Question G:_ We didn't include this step in our method due to the rather simple and clean structure of our data, but will probably need it in future projects, where the handwritten text requires recognition and segmentation. Layout recognition could be also a beneficial step in this project in order to accurately identify the relevant parts of the images (in particular for the more challenging cards (with multiple lines of text, including annotation errors), where the model struggles to correctly distinguish the graphical variants/surface forms from the lemmas).\n\n- _Question H:_ This was indeed our first approach, we started generating word pairs for words that shared a maximum Levenshtein distance of 2 (as this was the observed median distance in the training material). However, we then moved to a more flexible approach to increase the number (and variability) of word pairs. Generating word pairs based on the Levenshtein distance might generate word pairs that are closer to the original material, but it also represents an additional constraint that reduces the number of available examples per character, which is critical to their accurate prediction, as visualized in Figure 11 (Appendix).\nAs we didn't observe any performance decrease, we kept this more flexible strategy. Maybe this flexibility can lead to better generalization on slightly different data (but still of a similar structure) while not harming performance for our use case.\n\n- _Typos Grammar Style And Presentation Improvements:_ Thank you very much again for such detailed and helpful comments, we will work through all of them and adjust our paper accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strong points of the paper are:\n- careful experimental design and execution\n- new insights with the use of Swin Transformers and training language models from scratch\n- an extensive description of procedure, and further analysis provided in the appendix\n- the treatment of a low-resource language and the prove that with little material, good results are attainable\n\nAs such the DH and linguistics community (esp. lexicographers, I would assume), strongly benefit from work as presented by the authors. I enjoyed the read a lot!", "reasons_to_reject": "There are no reasons to reject the paper, but slight weaknesses:\n- I find the CER graphs (violin plots, I believe?) hard to read.\n- The authors mentioned this themselves, but they didn't give conventional methods the same chance as they did for Transformers. I would like to see this corrected for the final versions.\n- As hinted at in Question H: the process of synthetic data generation is a bit counter-intuitive.", "questions_to_authors": "Question A: What software did you use for the image processing (lines 183 to 195). PIL for everything?\n\nQuestion B: You say you trained a tokenizer by setting the output tokens to 73. The decoders work on the subword levels, though. How do you justify this decision? --> by continuing reading I kind of got the answer. Ok. But still, why train your decoder from scratch and not use the pre-trained one? Have you tried employing a pre-trained decoder?\n\nQuestion C: This might be provocative (but this is somehow intended): Would you agree that you push your methods quite far for a rather small dataset? Don't you think that similar results would be possible with less effort?\n\nQuestion D: Would you expect better results starting from a French pre-trained LM as a decoder (like CamemBERT?)?\n\nQuestion E: How big is the variation in the handwriting? Do you know how many people have participated in creating the cards?\n\nQuestion F: Would you think it's worthwhile to include random warp grid distortion as a pre-processing step? C. Wigington, S. Stewart, B. Davis, B. Barrett, B. Price, and S. Cohen, ‘‘Data augmentation for recognition of handwritten words and lines using a CNN-LSTM Network,’\n\nQuestion G: Did you use some kind of layout recognition?\n\nQuestion H: I find parts of your synthetic data generation process counter-intuitive. The surface form and the lemma are very very close in maybe 95%(?) of the cases, i.e. there is a certain overlap. With Levenshtein you could even put a number to how close most surface forms and lemmas are. When you create your synthetic examples, they are quite off, most of the times, e.g. CARTA@AUDIR. Wouldn't you think that performance could be enhanced by creating examples which are closer to the real data?\n\n\n\n\n\n\n", "paper_topic_and_main_contributions": "This paper presents (in great detail) how to conduct HTR for cards containing Old Occitan surface forms and the corresponding lemma(s). The authors go a long way from elaborate pre-processing methods to synthetic data generation. Moreover, the use Transformer-based architectures to train reliable models which are capable of deciphering what is written on the cards. A strong point is the carefully planned experimental design, as well as the execution of the experiments. The paper contributes novel insights and propagates the use of Swin Transformers for HTR. They compare their method to many others, thus allowing for a comparison against other systems.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I include general questions here. I do not expect a detailed answer for them, but are thought as food for thought for the authors. Some comments are high-level and I leave it to the authors if they want to consider them. Some are just common thoughts.\n\nTitle: Transcription ... Handwritten ... Language\n\n009: Transformer\n\n023: Transformer-based --> should be capitalised everywhere\n\n049: dictionary,\\footnote{} --> also for other occurrences\n\n144: I would call this surface form instead of graphical variant\n\n157: set instead of vocabulary\n\nFigure 1: I struggled a bit with the term \"Old Occitan cards\". The cards you show contain Old Occitan cards but are not from the Odl Occitan period.\n\n185: How do you make sure that you don't cut off text?\n\n201: put link in a footnote\n\n206: It would be nice to know how many lines the different corpora contain\n\nSection 3.3: maybe provide sources which noted that this kind of data augmentation helped\n\n\n\n\n\n"}}]

Author rebuttal:

Meta-review: The paper is well motivated and well written. The dataset is described well. The authors have considered an exhaustive set of approaches in the experimentation setup and have conducted a detailed analysis. There are new and possibly reusable insights with the use of Swin Transformers and training language models from scratch. 

Some points to be improved:
- Add conventional methods like CNN+LSTM in the experimental analsysis
- Discuss negative results as well on  the synthetic data generation process  
- Hard to identify the core novelty proposed in the technique:
Decision: Accept-Main