Detecting Erroneously Recognized Handwritten Byzantine Text
Authors: John Pavlopoulos
Vasiliki Kougia
Paraskevi Platanou
Holger Essler
Venue: EMNLP 2023 Findings (2023)
Keywords: text classification
error detection
handwritten text recognition
TL;DR: A benchmark of incorrect HTR output detection in Byzantine Greek shows the promising nature of a model pre-trained on modern- and then on ancient-Greek

Abstract: Handwritten text recognition (HTR) yields textual output that comprises errors, which are considerably more compared to that of recognised printed (OCRed) text. Post-correcting methods can eliminate such errors but may also introduce errors. In this study, we investigate the issues arising from this reality in Byzantine Greek. We investigate the properties of the texts that lead post-correction systems to this adversarial behaviour and we experiment with text classification systems that learn to detect incorrect recognition output. A large masked language model, pre-trained in modern and fine-tuned in Byzantine Greek, achieves an Average Precision score of 95%. The score improves to 97% when using a model that is pre-trained in modern and then in ancient Greek, the two language forms Byzantine Greek combines elements from. A century-based analysis shows that the advantage of the classifier that is further-pre-trained in ancient Greek concerns texts of older centuries. The application of this classifier before a neural post-corrector on HTRed text reduced significantly the post-correction mistakes.

Detecting Erroneous Handwritten Byzantine Text Recognition
John Pavlopoulos
Vasiliki Kougia
Department of Informatics
Faculty of Computer Science
Athens University of Economics and Business University of Vienna, Austria
Greece anni gr
UniVie Doctoral School Computer Science vasiliki. kougia@univie. ac. at
Paraskevi Platanou
Holger Essler
Department of Philology, School of Philosophy
Department of Humanities Ca" Foscari University of Venice, Italy holger essler@uivee...... it
University of Athens, Greece
pplatanou@phil.. www.hh.ii.o..g.r........
Abstract
Handwritten text recognition (HTR) yields tex- tual output that comprises errors, which considerably more compared to that of recog- nised printed (OCRed) text. Post-correcting methods can eliminate such errors but may also introduce errors. In this study, we investigate the issues arising from this reality Byzan- tine Greek. We investigate the properties of the texts that lead post-correction systems to this adversarial behaviour and experiment with text classification systems that learn to detect incorrect recognition output. large masked language model, pre-trained in modern and fine-tuned in Byzantine Greek, achieves an Average Precision score of 95%. The score im- proves to 97% when using model that is pre- trained in modern and then in ancient Greek, the two language forms Byzantine Greek com- bines elements from. A century-based analy- sis shows that the advantage of classifier that is further-po pre--re-tinnt trained in ancient Greek con- cerns texts of older centuries. The application of this classifier before neural post-corrector on HTRed text reduced significantly the post- correction mistakes.
Figure 1: Number of HTRed lines according to their CER as presented by Pavlopoulos et al. (2023)
transcribed; and the progress of HTR for historical manuscripts, which struggles with low resources. This why researchers from Digital Humanities have started encouraging the development of Nat- ural Language Processing (NLP) solutions, in an attempt to assist the experts toward this goal, since improving the accuracy of recognition will be excit- ing for the disciplines struggling with vast corpora and little man power (McGillivray et al., 2020).
1 Introduction
1.1 Background
Handwritten Text Recognition (HTR) concerns the automated transcription a handwritten text, ex- tracting its computerised form from an image. Ap- plying HTR on an old manuscript can lead to a high character error rate (CER), often due to the lack of training data (Pavlopoulos et al., 2023). In the real world, this erroneous HTR output is post-corrected by experts, who get involved, typically, in tedious and time-consuming task. Delaying the delivery of manually curated data, hinders both: the preserva- tion of digitised manuscripts that have not yet been
Although detecting erroneous recognised output is not a novel task (Chiron et al., 2017; Amrhein and Clematide, 2018; Rigaud et al., 2019; Jatowt et al., 2019; Nguyen et al., 2020; Schaefer and Neudecker, 2020), existing published studies fo- cus on printed material (i.e., OCRed). Handwritten text, on the other hand, increases the challenge in recognition compared to printed material and hence, leads to more errors. The different hands, the greater time period, morphology and spelling

---

Transeription
Recognition
HTR system working the character level is likely examining morphological aspects and often dis- regards semantic and pragmatic levels. vowel replacement function addressing the morphological level, however, may omit important semantic as- pects and relationships between sentence elements at the syntactic level and thus, misinterpret the case. For example, the line "λevtIov Õιςwway soutóv" was HTRed "Revtov vie A neu- ral corrector, then, which received as input the HTRed output, yielded "Nevtovv öie goutov" (Pavlopoulos et al., 2023), correctly separating the two last words but incorrectly removing the accent. In this way and in this instance, post-correcting involves the risk of introducing errors by trans- forming probably correct output cases.
a. Toy Noówy
Ton RoÖdy
b. Tov zoB0s
ax0206866
a. eyyιvouêva uh aBevvvvte εууιvoυêva zddn un oBevvivre
zdrin
eYYιvoμeva Toan μ aßevvóvte tyyIvoμév8τá00 paze ofævmooTe teş @umupfSoua
b.
a. tes @utoplCovar
Tòv QuvTehvvv 2Add xod ò ōid Tec
tôv Quı=λova dzAd xot ó ōta Tec teaeurupíooßo: xba áqueehoux xddd
b. Teq @ut
Tàv kumehōva and zod ó öid TEς
xoxxa a ÖsoTeC
Table 1: Synthetic transcription (umaan-generate)) and recognition system--gnnerated output with few (a) and many (b) errors. Finding the differences between the transcription and recognition is easier for the three examples shown in red (b).
1.2 The focus of the study
This study quantifies the hazards of the above- mentioned adversarial post-correction, by focus- ing on the Byzantine Greek language. The devel- opment of this language entails a comprehensive examination of written language throughout the centuries under study, In essence, Byzantine Greek emerged as an academic style known Atticised Greek, incorporating religious topics. As time pro- gressed, elements of contemporary spoken Greek gradually became integrated into the language (Hor- rocks, 2014). As the results of our study also show, the combination of pre-trained data on Modern Greek, resembling what was considered as spoken language back then, and Ancient Greek, including the above-mentioned Atticised Greek, is beneficial for text classifiers detecting incorrect recognition output. Furthermore, by covering period of more than seven centuries, the already challenging nature of recognition (Platanou et al., 2022) is increased. hence allowing more diverse material for our investigation on post-correction. ErTors in hand- written text recognition are often due to different hands (scribes) and scripts that evolved for many centuries. Printed text, on the other hand, exists for a much narrower period, there are no hands to cause similar errors," and clities. As we show in this work, lower recognition error rate means more difficult classification of erroneous lines. By contrast, recognised handwritten text often hard to parse even for humans, which makes an easier classification challenge and a much more difficult
Tw N TE T A
Figure 2: Image of the handwritten text 'täv rоδωy' in the recognised text, the third character was misspelled, written as cta ("r)) instead of (Table 1, 1st row)
(see Section 3), can explain the greater diversity of error rates (Fig. I). The recognition quality, how- ever, is related to the detection quality of erroneous output. To give an example, the detection of an er- roncous recognition is casier when the recognition output is scrambled compared with single incor- rect character. This is shown in Table 1, where classifying recognised text (as erroneous/not) is easier for the human eye in (b) compared to that of (a). We hypothesise that this applies also to models.
We argue that detecting erroneously recognised text can improve automatic error correction. Cur- rent approaches to post-correcting the often flawed recognised text, disregard the likelihood of flaw- less recognition, assuming that the correct text can be simply copied (e.g., with an encoder-decoder- based correction). Consequently, they are faced with the possibility of introducing, instead of mitigating, errors that would be absent if post- correction hadn't been applied. Taking the case of a misspelled article (e.g., see Fig. 2), for in- stance, there is variety of letters to choose from, for replacement, always accordingly to the declen- sion and the case of the noun. Even if grammati- cal rules are set (e.g., in a rule-based system), am
'Fonts pose similar problem, yet less challenging when compared to hands, due to their non-standaddised form and high variation.

---

error-correction task, as shown in Table 1.
guage models to detect the erroneous output in an unsupervised manner. On the other hand, although we consider that recognition errors are overall more for handwritten text compared to printed material, the quality of recognition can vary significantly for the former (Hodel et al., 2021), as is also shown in Fig. 1, and does not always come with a high error rate. Therefore, the detection of erroneously recognised handwritten text is also related to that for printed text, regarding their low error output.
1.3 The contributions
We explore the extent to which classification to flawless/flawed texts can improve the post- correction of recognised output, by avoiding adver- sarial errors introduced when applied to flawless texts. We benchmark machine- and deep-learning- based text classifiers, showing that Transformer that is pre-trained in modern Greek and then fine- tuned to detect erroneously recognised Byzantine Greek (10th- 16th CE) text achieves an Average Precision of 95%. When the same Transformer is further pre-trained in ancient Greek prior to fine-tuning, the score increases to 97%. We quan- tify the benefits of employing our text classifier in the real world, by training MTS (Xue et al., 2021) encoder-decoder error-corrector, which al- tered 80% of the flawless texts. If our best- performing classifier had been applied prior cor- rection, the percentage would have been decreased to 2% with a trade-off of 2% texts being incorrectly classified as flawless.
Erroneous recognised printed text detection
Initial work using rule-based approaches could not address all errors, and was followed by machine and deep learning approaches (Lyu et al., 2021). Standard approaches include support vector ma- chines on top of n-gram-based features (Dannélls and Virk, 2021; Virk et al., 2021). Jatowt et al. (2019) used a gradient tree boosting binary classi- fier top of various features, including character and word n- n-grams, part-of-speech token frequency based on automatically created resources.
Amrhein and Clematide (2018) performed de- tailed analyses and experiments of error detection and post-correction with various statistical and neu- ral machine tranti-ttrnnlttion methods. In general, they found that the former perform better in error correc- tion, while the latter models are better in error de- tection. However, they concluded that there is not single method that works best on all datasets while the results are highly affected by the data a model is trained on. Furthermore, they suggested that post-correction of recognised printed text should focus more on the improvement of error detection. Nguyen et al. (2020) used BERT pre-trained on named-entity recognition for token classification in order to perform error detection. After subtoken tokenization they obtain Glove or Fasttext word embeddings, which are combined with segment and positional embeddings and given as input to BERT (Devlin et al., 2018). The hidden states are fed to dense layer on top that classifies each token as erroneous or not. Although interesting, token- level detection is not suited for high-error settings.
The better performance of Transformer that is not pre-trained only in modern Greek can be explained by the fact that Byzantine Greek com- bines language elements from both, ancient and modern Greek. Since our study covers a broad his- torical period, we separated the evaluation dataset based on the century in order to investigate this hypothesis. Our results show that the detection of erroncous recognition depends on the chronology of the manuscript and that further pre-training in ancient Greek benefits the classification of lines from manuscripts in the following three centuries: 10thCF, 1tCC,, and 13thCF. By following a simi- lar segmentation basse evaluation method for the error rate level, instead of the century, we show that higher error rates make an easier classification goal. By contrast, the problem is harder for low error rates, when flawed and flawless texts look alike.
2 Related work
Our work focuses on the text classification of recog- nised handwritten text. In the absence of published studies for handwritten text, however, we sum- marise work focused on the recognition of printed text. Printed and handwritten materials differ, be- cause the latter often yields scrambled recognition output (Table 1). This is probably what motivated Ströbel et al. (2022) in using the perplexity of lan-
The ICDAR challenge
ICDAR organised competition focused on post- correcting text recognised from newspapers, shop- ping receipts, and other printed sources, including two subtasks: error detection, i.e., to detect the position and the length of the errors, and error cor- rection (Chiron et al., 2017; 17; Rigaud et al., 2019).

---

In 2017, the best-performing method for the error detection subtask employed probabilistic character error models and weighted finite-state transducers while in 2019, the best was a pre-trained multilin- gual BERT with convolutional and fully-connected layers on top that classified each sub-token as er- roneous or not. Schaefer and Neudecker (2020) followed a similar classification scheme, but fo- cused characters and not sub-tokens. Then, by excluding lines with at least one classified character from post-correction, they showed that false alter- ation (i.c., of correctly OCRed lines) by an encoder- decoder model could be avoided. Our work shows that casting this problem as text classification task can lead to high accuracy for handwritten text, al- though similar benefits may apply to printed text.
20
10
13
12
3.3
14
35 16
CENTURY
Figure 3: Violinplot of CER per century
that diversity of errors is produced, from very few to many per line (Fig. 1). This fact makes this dataset optimal for our training purposes, because need a diversity of enrors to train fiers. 5 Furthermore, the errors are not uniformly distributed across centuries (Fig. 3), motivating a time/CER- segmentation for evaluation pur- poses (discussed in Section 4.2).
classi-
3 Data
The written language of the manuscripts and pa- pyri used in this study is Byzantine Greek. Within these texts, morphological categories such as the optative, the pluperfect, and the perfect have disap- peared, while others such as the dative case have gradually decreased. Infinitives and participles are still there the texts, as reminiscents of the classi- cal tradition, encouraging one to treat the language as unique variant, different from modern Greek. There are several spelling conventions that deviate from the older orthographic rules while the ancient punctuation signs are still in use, albeit not always with the same function. Therefore, following Clas- sical and preceding Modern Greek, this language can be considered combination of the two.
Our classification dataset consists of the 1,875 texts (lines), treated as the flaw- less negative (0) class, and the respective system- transcribed texts, treated as the flawed, positive (I) class. 6 This resulted total an almost perfectly balanced dataset of 3,744 text lines, which we split to train, development and test subsets by using a 80/10/10 ratio. The average length of the transcrip- tion is 40.10 characters (6.83 words), and that of the HTR output is 37.09 characters (5.89 words).
4 Experimental analysis
The transcriptions and the HTR output were provided by the organisers of the recent HTREC challenge." 4 which regarded the automated correc- tion of HTR errors in Byzantine Greek (Pavlopou- los et al., 2023). Selected images of handwritten text in respective manuscripts (10th to 16th CE) were transcribed by human expert. The train- ing dataset of the challenge consists of 1,875 lines of transcribed text by a human expert and by an HTR model (examples shown in Table 2). Tran- skribus (Kahle et al., 2017) was used by the organ- isers to produce the system transcriptions, trained on seven images one per century. Our investiga- tion of the CER of the opted HTR model reveals
We opted for machine and deep learning bench- marks, with different transfer learning settings.?
Machine learning benchmarks
Machine learning methods include a variant of Naive Bayes and Support Vector Machines (NB- SVM), which is often used as a bascline for text classification (Wang and Manning, 2012). Also, we experimented with Random Forest and a multi- layer perceptron (MLP) using character n-grams
Fewer errors will harm class balance while similar errors will harm generalisation of the classifiers.
"Six lines excluded from the positive class, because perfect transcription was achieved.
'The difference lies using sigmoid function (or on top of the text representation, instead of one per character. Alsso known as Medieval Greek or simply Byzantine. 'ht //www. arooowd. com/ hh///callla mengee/ htrec-2822 2822
"All experiments were performed using Google's Colabo- ratory, using a Tesla T4 GPU card. Our large language models comprise million parameters.
"We used the implementation of KTRAIN with default pa- rameters (Maiya, 2020).

---

Transcription by expert
IITR output byyevopewwterrrroouevweee alharie zanpes
ěÝyιvoυéva nódn μř, aßevvvvte GMtá fी, exkóae (the born-in passions not extinguishing but
the release)
:05 [Soo toh xax eavrool zolāă yiveria auyxapaby (of the life of themselves many happening forgive)
tou f ou tou xakeavtoig nolkayivertaa auxyup 6v
tec Sutttplooa töv Quteλλvvv 2Ad xat. à ötd TEc (- set on fire the vineyard but and the due to the)
dANaxO aiuaaeettvv sutupiCou 6oça
Table Human (on the left) and HTR-based (on the right) transcription (lines) from the 'Commentary on Isaiah' by Basil the Great (fol. 75r, Oxford, Bodleian Library MS. Barocci 102)
1, 5), term-frequency inverse document fre- quency, and default parameters otherwise."
AP
AUC
F1
F1
Random
0.52
0.50
0.49
0.47
NBSVM
0.66
0.65
0.60
0.51
Deep learning benchmarks
Forest
0.64
0.65
0.64
0.50
Deep learning methods comprise a recurrent neu- ral network using Gated Recurrent Units (GRU) and trainable word embeddings of 300 dimen- sions, originally pre-trained on (modern) Greek (Joulin et al., 2017)." Also, experimented with the state-of-the-art in text classification, which is BERT (Devlin al., 2018) and its variants (Li et al., 2022; Minaee et al., 2021). In specific, we also fine-tuned two BERT models pre-trained in Greek. The first (GreekBERT) is pre-trained in modern Greek (Koutsikakis et al., 2020) while the second extends the first by being further pre-trained in ancient Greck (Singh et al., 2021)."!
MLP
0.79
0.79
0.73
0.69
GRU
0.79
0.79
0.68
0.71
GreekBERT:M
0.95
0.94
0.88
0.88
GreckBERT.M+A
0.97
0.97
0.90
0.91
Table 3: Evaluation with Average Precision (AP), AUC, for the classification of HTRed texts to flawed (+) or flawless (-) transcription. Random (uniform) classifica- tion was used as a baseline. GreekBERT was pre- pre-trained modern (:M) or modern and ancient Greek (:A).
outperformed in all metrics by GreckBERT. Two GreekBERT models were used, one pre-trained in modern Greek (:M) and one in modern and then in ancient Greek (:M+A). Byzantine Greek, already discussed (Section 3), combines language elements found in both ancient and modern Greek. This is probably why GreckBERT.M+A. which was pre- trained in modern and then in ancient Greek, was better compared to GreekBERT.M, which was pre- trained only in modern Greek. 13
Evaluation measures
We opted for the Fl score per class, the Area un- der the Receiver Operating Characteristic Curve (AUC), and Average Precision (AP) that is pre- ferred in class-imbalanced settings. All evaluation measures were implemented in SCIKIT-LEARN. 12
4.1 Text classification benchmark
4.2 Evaluation per century
Table 3 shows the results of all our text classifi- cation benchmarks, including random baseline (responding uniformly) that draws the lower perfor- mance limit. NBSVM is the worst machine learn- ing model, followed by Forest and MLP. The latter achieves promising results, close to GRU (better in AP and F1 ). GRU, on the other hand, is clearly
We separated the lines of the test data based on the century of the manuscript that they came from. For each century, then, we evaluated the two Transform- ers, GreckBERT.M and GreckBERT.M+A. As can be seen in Fig. 4, GreekBERT.M+A is better the 10th (five percent units) and 11th century CE, with models performing similarly on the 12th and after the 13th century, when both models achieve a
We verified consistency of this finding by repeating the experiments three times, reporting an average AUC of 0.97 over 0.94 (st.d. lower than 0.05).
html
Ancieet Greek texts, which can used for pre-training. are much fewer in number compared to publicly available modern Greek data. Hence, the option of pre-training directly in ancient Greek was not considered viable. Phttps: / http///sckkktttt Meartteeen.noa org/ org.stablel
"Tor statistical significance, we sampled twenty lines per century, repeating the sampling process ten times. We focus on Average Precision, but similar findings were observed with the other metrics.

---

0.950
0.925
0.900
0.875
Model
0.850
GreekBERTIM
GreekBERT:A
10
13
14
15
16
Figure 4: Average Precision per CE GreekBERT.M, pretrained on modern Greek (solid), and GreekBERT.M+A, further pre- trained on ancient Greek (dashed) error bands show the 95% bootstrap confidence intervals
higher score. On the 13th century, the performance of both models drops, but considerably more for GreekBERT:M. The 14th, 15th and 16th century were more challenging in HTR (Platanou et al., 2022), which could explain the better performance the classifiers. That is, more errors the recog- nised output could potentially help the classifier distinguish between the two classes (correct/not). i5 Based on the previous observations, we report three findings:
hard to parse even by humans. This characteristic. however, makes them easier to distinguish from flawless lines, which comprise errors from the recognition.
Low-CER lines (i.e., lines with CER of zero to twenty) lead to different picture, since Greek- BERT:M+A is consistently better, overall, than GreekBERT.M. Low CER means small differences between the texts hewless-flawedd which makes the classification task more difficult. In this zone, further pre-training on ancient Greek had a bigger impact, compared to zones with higher error rate.
F1. The classification performance depends the chronology of the manuscript.
F2. Older manuscripts better handled by fur- ther pre-training on ancient Greek.
Discussion
In this section, first, assess the possibly adver- sarial nature of post-correction. Then, we describe synthetic error detection application, followed by a century-based error analysis we performed.
F3. Lines from recent manuscripts pose an easier classification challenge to both GreekBERT models.
4.3 Evaluation per error zone
5.1 Substantiality assessment
Our dataset is balanced regarding whether the recognition output is erroneous (positive class) or not, yet it remains imbalanced regarding the CER level (see Fig. 1). In order to assess the two Trans- formers across different input types, we rolled window of ten units, from low to high CER values, and we assessed the models per window (Fig. 5). 16 High-CER lines (i.e., lines with CER of thirty or more) appear to set an easy classification task to both models. Performance increases the CER increases. This is reasonable considering that lines with a high CER comprise many errors and
Error-correction systems may introduce errors. even when applied to flawless HTRed texts. To assess the substantiality of this claim, we ex- perimented with a neural encoder-decoder post- corrector. In specific, we employed a multilingual variant of the T5 Text-to-Ten Text Transfer Transformer (Raffel et al., 2019), called MT5 (Xue et al., 2020), which was pre-trained languages." 17 MT5 has been used for Grammat- ical Error Correction in modern Greek and sets strong baseline for HTR Error Correction (Ko- ree and Pavlopoulos, 2022). We fine-tuned this model on the 1,875 training instances of the HTREC
dataset covering 101
It remains unclear authors whether more hands were present during these centuries, explaining performance the HIT the classification models, this was another reason; e.g., writing became more cursive over time. We sampled thirty lines, repeating sampling ten times.
L'ehe Greek language was on the 20th position with 43 billion tokens extracted from 42 million pages

---

0.225
0.875
0.850
Model
GreekBERT:M
0.825
GreekBERT:A
10
20
50
x01
Figure 5: Average Precision of GreekBERT:M, pretrained on modern Greek (blue, solid), and GreekBERT further pre pre-trained on ancient Greek (orange, dashed), using a rolling window over CER (left edge shown horizon- tally) error bands depict 95% bootstrap confidence intervals
challenge, by encoding the system- and decod- ing the numan-transsriptions 18 This model is neural eπror correction baseline that reduced the CER in 19 out of the 180 HTRed texts in the offi cial HTREC2022 evaluation set. We applied this corrector the respective 180 transcriptions of these data (the ground truth evaluation instances in HTREC2022) and found that 144 (80%) were mis- takenly altered by our MT5 corrector. For example, the line "avv touç urōous ortives outs apyupти napa" was altered to "aiv touç μπбоuu or TIveç oUts aрrорtι napa' (word division, 4th word). Similarly, "yevans (in poveuars μnn zλ=' was altered to "XhEvoEus (m) poveuaεuς (m) xhænt'. Most of these adversarial mistakes (all but the top three of Table 4) would have been avoided if our A classifier had been applied prior to the application of this neural corrector. Con- versely, only four (i.e., the rest in Table 4) would have been mistakenly found as correct from the system-transeribed instances.
on the flawed system recognitions (the same lines were used). The human transcriptions were cor- rectly classified as flawless, all except three (1.7% error rate), which are shown in the top three rows of Table 4. The system recognitions were also rectly classified flawed, all except four (2.3% eтror rate), shown in the last four rows of Table 4.
5.3 Error analysis
An error detection by experts in Table 4,19 showed that no clue (indicating an error) was found for the first line (except from the rare "púhaxoxovd6ce" with double accent). As is shown in Figure 6 (a), which depicts the attention on the subwords of this line, this indeed the network's 's focus.
On the second line, word 'tquan)rrjv'" was sidered the outcome of misspelling of the word 'vouuorruur"' matching the algorithm's decision. In this case, was the ending of the word that at- tracted attention (Fig. 6), but the network focused more on the article "tav' preceding the word rather than on the word itself. The article is in plural form, not matching a word in singular. Apparently, 'va- μanλffu" is not a misspelling, but part of the word 'iouuorrxutuxoo'' that was corrupted.
5.2 A synthetic application of error detection We experimented with the HTREC2022 evaluation dataset, which was used to assess the participating systems. This dataset comprises 180 lines that were transcribed from nine images. Out of these lines, one had zero CER achieved by the HTR model (perfect transcription), which was correctly classi- fied by GreekBERT:M+A model. We then per- formed two analyses. first by applying our model on the flawless human transcriptions (as if a tran- scription was the HTR output) and then by applying
In the last four lines, that were classified as cor- rect, several flaws were raised, indicating failure of the algorithm. For example, word divisions, such as in the case of "wazza" instcad "coç xai'; added characters, such as in the case of 'ebuyns' instead of 'elxrs'; character replacements, such as in the case of '(xerelav' instead of "uxeteizv".
"WW used the SimpleTS implementation thttps:// gi thtbb. carly stopping (patience of 2 epochs), 15 tokens max. length, batch size of 4, and best loss found at the 12th epoch (2.64).
with 100 epochs,
No access was given to the manuscript or the human tran- scription, for fair comparison with the setting of the algorithm.

---

TRANSCRIPTIONS AND HTR OUTPUT Eyd 6 del Twe pιxxóλov0ó9 eiuı
GREEKBERT.MAA
GT
(I am always readily following)
F
N
la xau YεδεωV EX tov axubaa tov vauamArra (- and Gedeon from the dogs -)
N
ōpqnar eiç xot YoVUTεтл (having run to and falling on the knee)
F
N
Tulau Tæς Xeipaç enavto
(having put the hands them)
F
N
Exerelary óбεστoTZ, xol suμevio dxcvvov rhv tñς npos etuyns (---and favourably listen to the of the prayer)
Łyevóyny ic. apon thiços μsó weyeλffnn εXysssooytyyoog (became---. ------)
N
F
waxx the auvexous διō0mxxaaa TOUS 30x ofof the continuous instruction -)
N
F
Table HTRed lines and transcriptions that GreekBERT.M+A mistakenly predicted flawed (F) or not (N) the ground truth (GT) on the right shows whether there was an error or not
Error patterns
"p". The way the model interprets this structure can be seen in the following case; 'pēp' èn' abthy to where "'pép" consists of one letter (q) and one ligature (ep) is read by the model as "openxx tyy kçw'. The model fails reading correctly the letter combination and returns only the letter "p' and one of the two letters involved in the 'ep" combination (p), which is actually the one written in the posi- tion next to the letter "q'. The same instance shows clearly that there is tendency towards adding char- acters because of different drawing representations of characters found in this data group. The letter ( can be considered such a case. In the previous example, we can see that the word "to" is read by the model "vço"" This is because, in this instance. the letter 'os" has the shape of laid eight. The same letter has the usual shape in other cases in the text of this century, which was quite confusing for the model and led the model to decide that should be a combination of both "a" and 'w', since the former is quite close to the half laid-eight shape.
Next, we proceed with an analysis of the recogni- tion errors, using data from three centuries. We opt for 16th CE. the century on which both our models perform considerably well (Fig. 4). Next, we fo- cus on the 14th CE, going back two centuries, but still observing good performance by both models. Last is shown the 13th CE, where GreekBERT:M performs much worse than GreekBERT:M+A.
16th CE This is the data group with the high- est average CER (see Figure 3). What seems to be challenging this data group is the fact that there is variety of punctuation marks which are sometimes read by the model as letters because of their shape. An illustrative example of this case is the word "Tλoū which is read by the model "zoovoo". This is due to comma following the word "xhoūç' which is very similar the last letter of the word (ç). There are also many cases of ligatures in which the model fails to recognise both letters, especially the one written on the top of the ligature, such as the letter '+' in the combination "Tp' as one can see in the following instance; 'ö nutpic" read 'wa tTt.':"..
13th CE The data group that is easiest to read for the model seems to be the 13th century one. this group, errors appear mainly because of special letter shapes and abbreviations. Such letter shape is that of the letter "s' which is written as the letter '6' with added line pointing upwards. A char- acteristic example of the failure of the recognition task is this one; 'ěoo'' read as 'ows'. Here the model has clearly classified the " as '8'. Re- garding the abbreviations representing character groups, one can find among others the following
14th CE The data from the 14th century show the second highest CER average (see Figure 3). This partly because of great number of ligatures appearing this data group. A common ligature among this group is the one which represents the group of the letters 'e" and "p". In this ligature happens that the letter 'e' is written above the letter

---

EYW
CEE
T
quAa
# Koooou
Doc
E1
#
êy ö del noag pulazddou0ós equ
(b) Àot xou yeōsuv toy axultov tov va
0pt
#Nuww
**vv
(c) Sorntdd ELę YOVUEETY
Figure 6: The attention of GreckBERT (darker is more) for the three top (missclassified) lines of Table 4 the symbol used to concatenate subword with the preceding one during tokenisation (ōрхιиоv öpa #ttlwv)
one: "av'. In the case of "oüx layuaxv', where the last two letters are represented by the abbreviation, the model reads "oúxoay'' where "ws has replaced the abbreviation because of similarity in shape.
fine-tuned, had classified the input first. Directions of future work comprise the develop- ment of larger dataset, which, although not easily developed in our domain of focus, it could poten- tially allow us draw more conclusions and lead to further findings. Furthermore, we plan to extend our dataset with calibrated evaluation subsets, or- der to gain more perspective on the performance of systems for specific error types, including ones that occur naturally and are not meant to be corrected.
5.4 Language and script generalisation
Our work can potentially inspire analogous pre- training in other languages with long documented history. To provide an example: Old, Middle, and Modern English and Japanese (Lyovin, 1997, pp. 2) can qualify for pre-training on both Old and Modern data, as in the Greek language in The scripts results are based on, however, very common both Greek and Latin manuscripts while common handwritten character shapes. e.g.. epsilon (E) (Chambers and Chambers, 1891, pp. 703), may illuminate error patterns in Latin scripts.
Limitations
Our study is focused on Byzantine Greek, from the 10th to the 16th century. Our hy- pothesis, that text classification could assist the post correction of HTRed output, should be studied for more languages and covering a longer period to draw general conclusions.
6
Conclusion
By contrast to OCR, where low CER values are reported, HTRed output can significantly differ from the human transcription. For ex- ample, correct text may be misplaced (e.g.. due to mistaken word division) or by the in- troduction of an extra character (e.g., the last word of the fifth line in Table 4). Hence, the creation of ground truth at the character level is very challenging in this context, hindering experimentation the token or the charac- ter level, as in OCR (Schulz and Kuhn, 2017; Schaefer and Neudecker, 2020).
This work showed that detecting erroncous recog- nised lines improves the accuracy of handwritten Byzantine text recognition, aiming to assist the dis- ciplines struggling with vast corpora and little man power. By contrast to printed text, errors in hand- written text vary greatly, some of which ier and others difficult catch. Our experiments showed that two BERT models, one pre-trained modern and the other pre-trained modern then ancient Greek perform well for the task of erroneous handwritten recognised text classifi- cation. The latter performs better, especially for manuscripts from older centuries and for lines with higher recognition enror rate. By using neural text- to-text Transformer for error correction along with our best performing classifier, versarial behaviour is not uncommon in the neural corrector's output which could have been signifi- cantly limited if a BERT classifier, pre-trane mr-itrained and
Sensitivity analysis has not been performed. experiments to verify the validity of our findings regarding GreckBERT, we have not experimented with cross validation over the centuries, as in (Pla- tanou et al., 2022). Training, however, an algo- rithm on texts from specific centuries, then as-
Although repeated
found that ad-

---

sessing on ones from previous or future ones, could potentially reveal interesting patterns.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Pu- pers, pages 427-431 Association for Computational Linguisties.
Diplomatic transcriptions may lead to naturalyy errors (i.e., ones the editors would want to keep, to stay as close to the original as possible), which our proposed classifiers could and probably would classify as incorrect recognition output. This limitation could be bypassed if the proposed systems are used complementary with an expert, not trying to substitute one. For example, the expert could view only lines the classifier labelled as incorrect, bypassing or acknowledging errors which were meant to be there.
Philip Kahle, Sebastian Colutto, Günter Hackl, and Gün- ter Mühlberger. 2017. Transkribus service plat- form for transcription, recognition and retrieval of historical documents. In 2017 14th IAPR Interna- rional Conference on Document Analysis and Recog- nition (ICDAR), volume 04, pages 19-24.
Katerina Korre and John Pavlopoulos. 2022. Enrich- ing grammatical error correction resources for Mod- ern Greek. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 4984-4991, Marseille, France. European Language Resources Association.
John Koutsikakis, Ilias Chalkidis, Prodromos Malaka- siotis, and Ion Androutsopoulos. 2020. Greek-bert: The greeks visiting sesame street. In 11th Hellenic Conference on Artificial Intelligncce pages 110-117.
References
Chantal Amrhein and Simon Clematide. 2018. Super- vised ocr error detection and correction using statisti- cal and neural machine translation methods. Journal for Language Technology and Computational Lin- guistics (JLCL), 33(1):49-76. 76.
Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S Yu, and Lifang He. 2022. A survey text classification: From traditional to deep learning. ACM Transactions on Intelligent Sys- tems and Technology (TIST), 13(2):1-41.
William Chambers and Robert Chambers. 1891. Cham- bers' Encyclopadia: A Dictionary of Universal Knowledge, volume 8. W. & R. Chambers.
Anatole Lyovin. 1997. An Introduction to the Lan- guages of the World. Oxford University Press, USA.
Guillaume Chiron, Antoine Doucet, Mickaël Coustaty, and Jean- dean-Philippe Moreux. 2017. lcdar2017 compe- tition on post-ocr text correction. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 1, pages 1423-1428, Kyoto, Japan.
Lijun Lyu, Maria Koutraki, Martin Krickl, and Besnik Fetahu. 2021. Neural oCT post-hoc correction of his- torical corpora. Transactions of the Association for Computational Linguisticss pages 479-493.
Arun Maiya. 2020.
ktrain: low-code library for augmented machine learning. arXiv preprint arXiv:2004. 10703.
Dana Dannélls and Shafqat Virk. 2021. A supervised machine learning approach for post-ocr error detec- tion for historical text. In Swedish Language Tech- nology Conference and NLP4CALL, pages 13-20.
Barbara McGillivray, Thierry Poibeau, and Pablo Ruiz Fabo. 2020. Digital humanities and natural language processing:"je t'aime... moi non plus".
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1xXXi::880..4000
Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Nar- jes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2021. Deep learning- baaedd text classification: a com- prehensive review. ACM Computing Surveys (CSUR). 54(3):1-00
Tobias Mathias Hodel, David Selim Schoch, Christa Schneider, and Jake Purcell. 2021. General models for handwritten text recognition: feasibility and state- of-the art. german kurrent as an example. Journal of open humanities data, 7013)11110.
Thi Tuyet Hai Nguyen, Adam Jatowt, Nhu- Van Nguyen, Mickael Coustaty, and Antoine Doucet. 2020. Neural machine translation with bert for post-oer error detec- tion and correction. In Proceedings of the ACM/IEEE joint conference on digital libraries in 2020, pages 333-336.
Geoffrey Horrocks. 2014. Greek: A History of the Language and its Speakers. John Wiley & Sons.
Adam Jatowt, Mickaël Coustaty, Nhu- Van Nguyen, An- toine Doucet, et al. 2019. Post-oer error detection by generating plausible candidates. In 2019 Interna- tional Conference on Document Analysis and Recog- nition (ICDAR), pages 876-881, Sydney, Australia.
John Pavlopoulos, Vasiliki Kougia, Paraskevi Platanou, Stepan Shabalin, Konstantina Liagkou, Emmanouil Papadatos, Holger Essler, Jean- -aptistt Camps, and Franz Fischer. 2023. Error correcting htr'ed byzan- tine text.

---

Paraskevi Platanou, John Pavlopoulos, and Georgios Papaioannou. 2022. Handwritten paleographic greek text recognition: A century-based approach. In Pro- ceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6585-6589.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019, Exploring the limits of transfer learning with unitied text-to-text trans- former. arXiv preprint arXiv: 1410.1991000688
Christophe Rigaud, Antoine Doucet, Mickaël Coustaty, and Jean-Phile San-PPilippe Moreux. 2019. Iedar 2019 com- petition on post-ocr text correction. In 2019 interna- tional conference document amalysis and recogni- tion (ICDAR), pages 1588-1593, Sydney, Australia.
Robin Schaefer and Clemens Neudecker. 2020. two- step approach for automatic ocr post-correction. In Proceedings of the The 4th Joint SIGHUM Workshop Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 52-57, held online.
Sarah Schulz and Jonas Kuhn. 2017.
domain-tailored ocr post-correction In Proceedings of the 2017 Conference Empirical Methods in Nat- ural Language Processing, pages 2716-2726, Copen- hagen, Denmark.
Pranaydeep Singh, Gorik Rutten, and Els Lefever. 2021. pilot study for bert language modelling and mor- phological analysis for ancient and medieval greek. The 5th Joint SIGHUM Workshop on Computa- tional Linguistics for Cultural Heritage, Social Sci- ences, Humanities and Literature (LaTeCH-CIff.. 2021).
Phillip Benjamin Suröhel, Simon Clematide, Mar- tin Volk, Raphael Schwitter, Tobias Hodel, and David Schoch. 2022. Evaluation of htr models without ground truth material. arXiv preprint arXiv:2201.06/70.
Shafqat Mumtaz Virk, Dana Dannélls, and Azam Sheikh Muhammad. 2021. A novel machine learning based approach for post-ocr eтror detection. Proceed- ings of the International Conference on Recent Ad- in Natural Language Processing (RANLP 2021), pages 1463-1470, Varna, Bulgaria.
Sida Wang and Christopher Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic clas- sification. In Proceedings of the 50th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 90-94, Jeju Island, Korea, Association for Computational Linguistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel, 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.1193. 19944.


Reviews:
[{"review_id": "PpuzpoO82d", "reviewer": "Reviewer_ubKi", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the invested time and feedback. Please read our response below:\n\n> R1. The number of samples in the classification datasets is small ~2k samples, which is hard to determine the significance of the approach.\n\nIn text classification settings, this size is not prohibitively small (see for example [1]) while we would like to note that larger datasets are not easily developed in our domain of focus (please see the 2nd reason to accept by `Reviewer PgK8`). We acknowledge that a larger dataset could potentially allow us to draw more conclusions and lead to further findings, hence we will propose it as future work.\n\n[1] Ravi Shekhar, Mladen Karan, and Matthew Purver. 2022. CoRAL: a Context-aware Croatian Abusive Language Dataset. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 217–225.\n\n> R2. The approach is not compared with any baseline methods in the literature.\n\nThis remark is not entirely clear to us. Besides the two Transformers and the machine learning algorithms we trained and assessed, we aren’t aware of another suitable baseline nor could we find one in literature. If the reviewer could provide more feedback, we could elaborate more.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a method for enhancing the output of the OCR by 2%. The paper provides an in-depth analysis of the errors.", "reasons_to_reject": "- R1. The number of samples in the classification datasets is small ~2k samples, which is hard to determine the significance of the approach.\n- R2. The approach is not compared with any baseline methods in the literature.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper provides an approach for post-correction of the outputs from OCR in the Byzantine language. They showed that finetuning a masked language model on this language enhances the performance by 2%.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zLQKMhGjpj", "reviewer": "Reviewer_PgK8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and the suggestions. Please read our response per point below:\n\n> Some form of explicit ablation study might help create more perspective to view and solve this problem.\n\nIn future work, we plan to extend our dataset with calibrated evaluation subsets, in order to gain more perspective on the performance of systems for specific error types. We could not think of another ablation toward this goal but we could elaborate more if we misread this comment. \n\n> Section on point-of-view about generalizability of the findings to other languages or scripts might help.\n\nCorrect. We mentioned this as a limitation of our work, but we will add the following paragraph in Section 5 to further address this point:\n*Our work can potentially inspire analogous pre-training in other languages with a long documented history. To provide an example, Old/Middle/Modern English and Japanese [1] can qualify for pre-training on Old-Modern data, as the Greek language in our case does. The scripts our results are based on, however, are very common both in Greek and Latin manuscripts while common handwritten character shapes, e.g., epsilon (E) [2], may illuminate error patterns in Latin scripts.*\n\n[1] Lyovin, A.V., Kessler, B. and Leben, W.R. (2017). \"An introduction to the languages of the world\". Oxford: Oxford university press., pp.2\n\n[2] \"Chamber’s Encyclopaedia; A dictionary of universal knowledge\". London: W. & R. Chambers., 1894, pp.703 \n\n> Details around error type can also shed light on generalizability and may also help set new standards for post-correction process.\n\nOur work provided an extensive error analysis (Section 5.3), which included error patterns (please see also the 2nd reason to accept by `Reviewer nm3Z`). In future work, however, we plan to study more error types, such as the ones that occur naturally and are not meant to be corrected. We will clarify this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Paper is well written and technically sound.\n- Simple yet direct application & improvement in an area where lot of corpora is present but very less humans to look into.\n- Looking through the lens of temporal aspect (proposed century-based approach) is very novel. \n- The paper proposal could have some implications for other languages and scripts which exhibit similar patterns of variations over time.\n", "reasons_to_reject": "- Some form of explicit ablation study might help create more perspective to view and solve this problem.\n- Section on point-of-view about generalizability of the findings to other languages or scripts might help.\n- Details around error type can also shed light on generalizability and may also help set new standards for post-correction process.\n\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors provide a study of post-correction of Handwritten Text Recognition (HTR ) errors, with a focus on Byzantine Greek from the 10th to the 16th century. They also call out the findings related to classification task of detecting erroneously recognized lines. They use two BERT models to shore how pre-training on just modern Greek vs modern + ancient Greek differ in performance of classification task. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6hnXeQenrq", "reviewer": "Reviewer_nm3Z", "summary": "", "questions": "Question A: You say early on that you dover 7 centuries. How does Byzantine Greek change over this time? Shouldn't you include 2 or 3 sentences around line 097? If it changes, how do you think these changes affect the performance of models? --> after having read on, you elaborate on this in Section 3. However, the question about the impact these changes had remains.\n\nQuestion B: For editions of texts, editors usually want to stay as close to the original as possible. That is, during writing, errors naturally occur. How would you make sure that automatic correction does not touch such words? And, more importantly for your work: a classifier would probably label such a word as error, when indeed the model provided the \"correct\" transcription. How would you treat such cases? --> again, after having read on, you hint at this in Section 5.1. Still, a qualitative analysis of the errors introduced is missing. I think this would be something your contribution would profit from!\n\nQuestion C: How much do the hands differ in the dataset? Could the variance of performance of the HTR system be due to the greater variance of the hands?\n\n\n\n\n\n\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate the constructive feedback and thoughtful suggestions provided by the reviewer. We commit to revise our work accordingly. Please read our thoughts on the raised concerns and how we addressed them below:\n\n> Question A: You say early on that you dover 7 centuries. How does Byzantine Greek change over this time? Shouldn't you include 2 or 3 sentences around line 097? If it changes, how do you think these changes affect the performance of models? --> after having read on, you elaborate on this in Section 3. However, the question about the impact these changes had remains.\n\nCorrect. We will add the following in line 096: \n\n“The development of the Byzantine Greek language entails a comprehensive examination of written language throughout the centuries under study. In essence, Byzantine Greek emerged as an academic style known as Atticised Greek, incorporating religious topics. As time progressed, elements of contemporary spoken Greek gradually became integrated into the language [1]. As the results of our study also show, the combination of pre-trained data on Modern Greek, resembling what was considered as spoken language back then, and Ancient Greek, including the above-mentioned Atticised Greek, is beneficial for text classifiers detecting incorrect recognition output.”\n\n[1] Horrocks, Geoffrey. 2010. \"Greek: A History of the Language and its Speakers\". Second Edition. Malden, MA: Wiley-Blackwell., pp.220-221\n\n> Question B: For editions of texts, editors usually want to stay as close to the original as possible. That is, during writing, errors naturally occur. How would you make sure that automatic correction does not touch such words? And, more importantly for your work: a classifier would probably label such a word as error, when indeed the model provided the \"correct\" transcription. How would you treat such cases? --> again, after having read on, you hint at this in Section 5.1. Still, a qualitative analysis of the errors introduced is missing. I think this would be something your contribution would profit from!\n\nCorrect. Diplomatic transcriptions may lead indeed to such errors, which our proposed classifiers could and probably would classify as incorrect recognition output. This is exactly why the proposed systems should be used complementary with an expert and not trying to substitute them. For example, the expert could view only lines the classifier labelled as incorrect, bypassing or acknowledging errors which were meant to be there. This is a limitation which we will include in our manuscript. \n\n> Question C: How much do the hands differ in the dataset? Could the variance of performance of the HTR system be due to the greater variance of the hands?\n\nThe 14th, 15th and 16th century were more challenging in HTR [2], which might explain the better performance of the classifiers (Figure 4). That is, more errors in the recognised output could potentially help the classifier distinguish between the two classes (correct/not). Regarding whether more hands were present during these centuries, however, explaining the performance of the HTR and classification models, remains unclear to us and a material of future investigation. For example, another possible explanation is that writing became more cursive over time. We will clarify this. \n\n[2] Paraskevi Platanou, John Pavlopoulos, and Georgios Papaioannou. 2022. Handwritten Paleographic Greek Text Recognition: A Century-Based Approach. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6585–6589, Marseille, France. European Language Resources Association.\n\n\n### Missing references\n\nWe added Hodel et al. and we will also add Springmann et al. \n\n\n### Typos Grammar Style And Presentation Improvements\n\nWe are grateful for the analytical feedback and we will fix all the points suggested. Please find below our response for specific comments:\n\n> 033: ideally yes, but this not always the case!\n\nWe will add the word “typically”.\n\n> 040-046: I would like to have some backup for this claim (it is true, but maybe you mention some refernces where this is actually the case).\n\nWe will add the following citation:\n\n[3] McGillivray, Barbara, Thierry Poibeau, and Pablo Ruiz. \"Digital humanities and natural language processing:“Je t’aime... Moi non plus”.\" Digital Humanities Quarterly 14, no. 2 (2020).\n\n> Figure 2: should maybe be Figure 1. Moreover, change \\citep to \\cite in caption. Also, what is the y-axis? # of docs?\n\nThe vertical axis indeed shows the number of lines (we will clarify) and we will consider moving our claim that “HTR output does not always come with a high error rate” up to the introduction along with the figure.\n\n> 082-085: that is a strong claim. any backup? what about long distance relationships? what about attention-based models that take huge contexts into account?\n\nThis argument regards rule-based systems, lacking attentional or more generally neural components. It was used explicitly in order to provide an example regarding how errors may be introduced during post-correction. We will clarify this to avoid confusion.\n\n> 085-091: maybe you introduce an example for this.\n\nWe will add the following example:\n\n\"The line “λεντιον διεζωσεν εαυτόν” was HTRed as “λεντιον διεζωσενεαυτόν”. A neural corrector, then, which received as input the HTRed output, yielded “λεντιον διεζωσεν εαυτoν” (Pavlopoulos et al., 2023), correctly separating the two last words but incorrectly removing the accent.\"\n\n> 104: no clitics. and: there might not be hands, but different fonts\n\nCorrect, we will add the lack of clitics. Also, we will add a footnote clarifying that fonts pose a similar problem yet less challenging when compared to hands, due to their non-standardised form and high variation. \n\n> 161: here you say HTR does not always come with high error rates, while in 030 you say in can lead to high error rates and in 054 you say it leads to more errors. this is somewhat inconsistent. Maybe cite Hodel et al. (see missing references) for an indicator for what is possible for HTR and what is regarded as good, mediocre or bad in terms of CER.\n\nWe consider that recognition errors are overall more for handwritten text compared to printed materials. Also, Figure 2 shows that there is a greater error variety across lines, yielding lines of varying quality, from good to bad according to Hodel et al (2021). We will add the missing reference and consider rewording to avoid confusion. \n\n> 159: as far as I know, Ströbel et al. evaluated on HTR and not printed texts\n\nIndeed, Ströbel et al. worked on HTR. This is why we mention their work, after commenting in the previous sentence (lines 156-158) that handwritten material often yields scrambled recognition output. However, their work concerns the study of metrics for evaluating and ranking HTR models, which is different from the focus of our work and the related work we discuss.\n\n> 348: thirty or more what, errors per line?\n\nLines with CER of 30+ (we will clarify)\n\n> 356: zero to twenty what?\n\nLines with CER of 20- (we will clarify)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main reasons to accept the paper are:\n- the careful analysis of the dataset with the help of the results the authors attained\n- the analysis of error patterns\n- the discovery of time-period-based performance differences\n- the fact alone that work on Byzantine is presented, which is not so frequent\n\nI enjoyed the read and found it interesting.", "reasons_to_reject": "The paper has some flaws:\n- I'm missing the read thread through the paper. I could not always follow the arguments without re-reading and some claims need to be substantiated! (see comments)\n- We could attest the paper some kind of lack-of-novelty, but the usefulness of the approach and the evaluation in a new (the Byzantine) setting make up for this. \n- I would have expected some more resources which have tackled the problem in the past.", "questions_to_authors": "Question A: You say early on that you dover 7 centuries. How does Byzantine Greek change over this time? Shouldn't you include 2 or 3 sentences around line 097? If it changes, how do you think these changes affect the performance of models? --> after having read on, you elaborate on this in Section 3. However, the question about the impact these changes had remains.\n\nQuestion B: For editions of texts, editors usually want to stay as close to the original as possible. That is, during writing, errors naturally occur. How would you make sure that automatic correction does not touch such words? And, more importantly for your work: a classifier would probably label such a word as error, when indeed the model provided the \"correct\" transcription. How would you treat such cases? --> again, after having read on, you hint at this in Section 5.1. Still, a qualitative analysis of the errors introduced is missing. I think this would be something your contribution would profit from!\n\nQuestion C: How much do the hands differ in the dataset? Could the variance of performance of the HTR system be due to the greater variance of the hands?\n\n\n\n\n\n\n", "paper_topic_and_main_contributions": "This paper evaluates the use of classifiers to detect erroneously recognised text from HTR models that have been applied to Byzantine text. The authors show that Transformer-based classifiers exceed in this task and can be a great help to evaluate automatically recognised texts. As such, the paper makes an important contribution to the field of the Digital Humanities, and especially in the realm of digitisation procedures. One of the most interesting findings is that the performance of classifiers differs based on the century a text had been written in. This shows that the LMs that are behind the classifier (here BERT) are sensitive to language change, or rather, it implies that while training HTR systems, data from several periods should be included.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "161: Hodel et al. https://openhumanitiesdata.metajnl.com/articles/10.5334/johd.46\n\n166: maybe include Springmann et al. https://arxiv.org/pdf/1606.05157.pdf\n\n\n\n\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I include general questions here. I do not expect a detailed answer for them, but are thought as food for thought for the authors. Some comments are high-level and I leave it to the authors if they want to consider them. Some are just common thoughts.\n\n015: precision with respect to detecting incorrect output?\n\n029: computerised --> I'd prefer digitised\n\n030: HTR to an\n\n033: ideally yes, but this not always the case!\n\n036: delete :\n\n038: delete ;\n\n040: from the\n\n043: delete ,\n\n040-046: I would like to have some backup for this claim (it is true, but maybe you mention some refernces where this is actually the case).\n\nFigure 2: should maybe be Figure 1. Moreover, change \\citep to \\cite in caption. Also, what is the y-axis? # of docs?\n\nFIgure 1: maybe use an image with higher resolution?\n\n071: delete ,\n\n075: introducing instead of mitigating  errors that ...\n\n078: delete e.g.,\n\n079: rather choose characters instead of letters and delete ,\n\n082-085: that is a strong claim. any backup? what about long distance relationships? what about attention-based models that take huge contexts into account?\n\n085-091: maybe you introduce an example for this.\n\n104: no clitics. and: there might not be hands, but different fonts\n\n161: here you say HTR does not always come with high error rates, while in 030 you say in can lead to high error rates and in 054 you say it leads to more errors. this is somewhat inconsistent. Maybe cite Hodel et al. (see missing references) for an indicator for what is possible for HTR and what is regarded as good, mediocre or bad in terms of CER.\n\n159: as far as I know, Ströbel et al. evaluated on HTR and not printed texts\n\n214: instead of\n\nfootnote 1: or so --> not very scientific\n\n290: More specifically,\n\nTable 3: call it Random Forest, same in 307\n\n311: pre-trained on -> several instances\n\nFigure 4: name of y-axis missing, same for Figure 5\n\n348: thirty or more what, errors per line?\n\n352: for humans\n\n356: zero to twenty what?\n\n365: section we first assess ...\n\n459: characters\n\nin general: speak of characters instead of letters\n\n481: reading the character combination correctly\n\n\n\n\n"}}]

Author rebuttal:

Meta-review: The reviewers agreed that the paper tackles an under-resourced problem by means of thorough experiments and detailed dataset and error analyses. Criticism was centered around the generalizability of results, the small size of the dataset and a lack of novelty, which is made up for by the usefulness of the approach and the new Byzantine setting.
Decision: Accept-Findings