SlowGuess/NAACL_2022_OCR huggingface dataset:
Nikolai Vogler
nvogler@ucsd.edu
Matthew Thomas Miller
Roshan Institute for Persian Studies
tmiller@umd.edu
Jonathan Parkes Allen
jallen22@umd.edu
Taylor Berg-Kirkpatrick
tberg@ucsd.edu
We present a self-supervised pre-training approach for learning rich visual language representations for both handwritten and printed historical document transcription. After supervised fine-tuning of our pre-trained encoder representations for low-resource document transcription on two languages, (1) a heterogeneous set of handwritten Islamicate manuscript images and (2) early modern English printed documents, we show a meaningful improvement in recognition accuracy over the same supervised model trained from scratch with as few as 30 line image transcriptions for training. Our masked language model-style pre-training strategy, where the model is trained to be able to identify the true masked visual representation from distractors sampled from within the same line, encourages learning robust contextualized language representations invariant to scribal writing style and printing noise present across documents.
Document transcription is the task of converting images of handwritten or printed text into a symbolic form suitable for indexing, searching, and computational analysis. Historical documents, whether they were (re)produced via handwriting or the early printing press, confound current statistical document transcription models due to (1) extremely varied style and content across domains,
He was a second time indicted, for that he, on the 17th of December, about the hour of two in the night, the dwelling-house of Ralph Verney, E.g.: commonly called Earl Verney, of the kingdom of Ireland, did break and eper, and healing one fliver cap, value 20 s. one silver spoon, value 20 s. fix silver plates, value 12 l. three silver chocolate flands, value 30 a. fix silver tea spoons,
Figure 1: Example page image crops from an Islamicate manuscript dated to 1842 (Top, ref: Leiden Or. 669), showcasing its dense, visual complexity with extensive marginalia, and printed proceedings of London's Old Bailey Courthouse (Bottom, c.  $18^{\text{th}}$  century) (Shoemaker, 2005).
(2) the presence of noise, and (3) a dearth of labeled data.
First, historical printed documents, such as books produced from early modern England (c.  $16^{\mathrm{th}} - 18^{\mathrm{th}}$  centuries; bottom of Fig. 1), use nonstandardized spacing and fonts (Shoemaker, 2005) and can contain code-switching that confuses language models (Garrette et al., 2015). However, this variation pales in comparison to their handwritten counterparts. For instance, pre-modern Islamicate manuscripts (i.e., Persian and Arabic
<sup>1</sup>We use the generic term document transcription to refer to both the task of optical character recognition (OCR), which is typically reserved for printed documents, and handwritten text recognition (HTR) for manuscripts.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 206 - 216
handwritten documents from c.  $7^{\text{th}} - 19^{\text{th}}$  centuries; top of Fig. 1), differ in script family, scribal handwriting style, and symbol inventory/vocabulary. As a result, a large degradation in performance is observed when evaluating HTR models on unseen manuscripts (Jaramillo et al., 2018).
Production and imaging noise also present a problem for historical document transcription models. Whether it be uneven inkling from a printing press, inconsistent text baselines, or holes resulting from insect damage to ancient pages, techniques must be designed to cope with the noise (Berg-Kirkpatrick and Klein, 2014; Goyal et al., 2020).
While neural networks have a demonstrated capability to model complex data distributions, they typically require large amounts of supervised training data to do so, which is infeasible for historical documents. Unsupervised, non-neural transcription models with fewer parameters alleviate the need to create labeled data (Berg-Kirkpatrick et al., 2013), but struggle with complex handwriting variation. For Islamicate manuscripts, ground truth transcription often requires paleography experts to decipher the ancient writing systems as they appear in each scribal writing style.
In this paper, we propose a self-supervised learning framework designed to overcome these three challenges presented by historical documents. Inspired by the astounding success of self-supervised pre-training techniques for masked language modeling (MLM) in NLP (Devlin et al., 2019), visual models (Chen et al., 2020; Radford et al., 2021), and speech recognition (Baevski et al., 2020), our approach pre-trains a neural text line-image encoder by learning to distinguish masked regions of unlabeled line images from other distractor regions. Specifically, our contribution is the following:
- we show that the recent pre-train/fine-tune paradigm is particularly advantageous for low-resource historical document transcription, obtaining large improvements in both printed and handwritten documents in both English and Arabic-script languages.
- we motivate the self-supervised contrastive loss for document transcription through the lens of "lacuna reconstruction", where blank parts of a document called lacuna must be inferred by human readers.
In doing so, we argue that our approach to pretraining implicitly incentivizes the model to discover and encode discrete character classes in its internal representations, while ignoring style differences occurring in lines using different fonts or languages, or authored by other scribes.
Masked Pre-training Our approach to self-supervised pre-training follows a growing body of work in both NLP and speech that leverages mask-predict objectives for learning useful, task-agnostic language representations from unlabeled data. In the self-supervised pre-train/supervised fine-tune paradigm, these representations can then be updated on the task of interest using in-domain labeled data. Past work covers learning representations for NLP from monolingual and multilingual text (Devlin et al., 2019; Yang et al., 2019), speech (Baevski et al., 2019; Jiang et al., 2019; Song et al., 2020; Wang et al., 2020), and images grounded with text (Radford et al., 2021). Representations can be learned through either reconstruction-type objectives (Jiang et al., 2019; Song et al., 2020; Wang et al., 2020) or probabilistic contrastive loss functions (Oord et al., 2018; Baevski et al., 2019, 2020). Most similar to our work is the speech recognition system wav2vec2.0 (Baevski et al., 2020), which uses the same two phase training setup with a self-supervised contrastive loss during pre-training and Connectionist Temporal Classification (CTC) loss on transcribed speech data during fine-tuning. Talnikar et al. (2020) presents that the self-supervised loss regularizes the supervised loss during joint learning of both objectives. Follow up work has shown the usefulness of the pre-trained speech representations for exploring speech variation (Bartelds et al., 2020). In this paper, we show that the same learning paradigm can also be successfully applied to very low resource document transcription settings.
Islamicate HTR While machine recognition of handwritten, historic English/German documents can range from  $5 - 12\%$  character error rate (CER) on a sufficient amount of in-sample manuscript training data (Sánchez et al., 2019), performance on Arabic-script languages is much more challenging, leading to substantially higher
CER. Pre-modern Islamicate manuscripts (i.e., Persian and Arabic handwritten documents from c.  $7^{\text{th}} - 19^{\text{th}}$  centuries), often differ in script family, scribal handwriting style, and symbol inventory/vocabulary. In the top of Figure 1, we present an extreme example of some of the problematic visual variation that can be observed. Even a model trained in a supervised fashion on such a complex document sees a large degradation in performance when evaluating HTR models on unseen manuscripts (Jaramillo et al., 2018). Until recently, OCR performance on Arabic-script printed texts was still poor, typically above  $25\%$  CER (Alghamdi and Teahan, 2017), which is too high for downstream users (i.e., researchers and librarians).
Recent studies involving Islamicate manuscripts found that state-of-the-art systems are only able to achieve 40 to mid-20% CER using proprietary software (e.g., Google Cloud Vision, RDI, Transkribus) (Clausner et al., 2018; Keinan-Schoonbaert, 2020, 2019). However, results from these studies only report in-domain performance—an unrealistic scenario where considerable amounts of labeled data can be obtained to enable both training and testing on the same manuscript. In contrast, out-of-domain performance tends to suffer considerably, supported by Romanov et al. (2017)'s study of neural OCR for printed Arabic-script documents. Our work aims to address such performance issues for both in-domain and out-of-domain Islamicate HTR settings by learning general, content-rich pre-trained language representations from large amounts of heterogeneous unlabeled data.
Historical OCR Closely related to manuscript transcription, OCR is another task involving language recognition from images. However, OCR operates on documents that have been printed by a machine with regular, re-used character fonts exhibiting much less superficial glyph variation than human handwriting. OCR is far from a solved problem in the case of documents printed on early modern (c.  $16^{\mathrm{th}} - 18^{\mathrm{th}}$  centuries; see bottom of Fig. 1), movable-type printing presses, where humans would manually set metal type casts with non-standard spacing and fonts (Shoemaker, 2005). In this setting, inkling noise and historical font shapes confuse OCR models trained on modern, computer-generated documents (Arlitsch and Herbert, 2004). Berg-Kirkpatrick et al. (2013)'s Ocular
explicitly uses a generative probabilistic model inspired by historical printing processes to model such noise. Later work has extended it to handle more typesetting noise (Garrette et al., 2015), and produce both diplomatic and normalized transcriptions (Garrette and Alpert-Abrams, 2016). Separately, OCR post-correction models have been proposed to resolve OCR outputs in historical documents (Hämäläinen and Hengchen, 2019; Dong and Smith, 2018) and other low-resource settings (Rijhwani et al., 2020, 2021). In contrast, our approach pre-trains the visual language recognition model's encoder, which produces better contextualized representations in order to reduce the amount of errors the model itself makes. Unlike Ocular, our proposed method does not use a language model and is not fully unsupervised as we require 1–3 pages of transcribed data for learning to transcribe during fine-tuning.
When human readers encounter a lacuna, a blank information gap in a portion of a book or manuscript, they must infer its latent meaning using nearby context like in a cloze test (Taylor, 1953). We argue that the most useful information for inference lies in the ability to reason about the identities of the missing characters in the lacuna using the identities of the surrounding characters. Indeed, MLM-style pre-training techniques are also motivated by the idea of the cloze test, and recent research indicates that language representations learned through the prediction of missing content using surrounding sentential context are useful for many downstream tasks (Devlin et al., 2019; Clark et al., 2019, 2020). Our approach combines the ideas of lacuna inference and masked pre-training to provide a useful learning signal for downstream historical document transcription, a setting with massive digitized collections but few transcribed examples.
Specifically, we introduce a self-supervised pretraining method that randomly masks lacuna-like regions of document line images and learns to reconstruct them by distinguishing them from nearby line image segments, or foils. While lacuna can be reconstructed in a generative way, we find that a discriminative contrastive loss works better in practice. By leveraging a diverse set of unlabeled data
Figure 2: Our proposed two-stage approach for low-resource document transcription first pre-trains a line image encoder using a self-supervised contrastive loss on unlabeled data (left), followed by a fine-tuning phase, in which the pre-trained encoder learns to transcribe 1-3 pages of supervised data using a CTC loss (right).
for pre-training, the model is forced to infer the identities of masked text regions in the presence of scribal writing variation or typesetting noise ubiquitous in historical documents. In the next sections, we describe our model/masking strategy in detail.
In Figure 2, we show our two-stage pre-train/finetune modeling approach. First, we describe the document line image encoder that is shared between stages. For simplicity of description, we assume that each document line image,  $X$ , is  $n$  pixels tall and  $m$  pixels wide, and that pixels are binary-valued. Thus, the space of input text line images can be denoted as  $\mathcal{X} = \{0,1\}^{n\times m}$ . We first process the input with a convolutional feature extractor,  $f:\mathcal{X}\mapsto \mathcal{H}$ , that maps the input,  $X$ , to an encoding matrix,  $H$ , using a deep convolutional neural network followed by a reshaping of the image height dimension into the channels dimension. Next, a contextual encoder,  $g:\mathcal{H}\mapsto \mathcal{C}$ , computes a contextualized representation matrix,  $C$ , from  $H$  using a neural sequence model, parameterized by a bidirectional LSTM (Hochreiter and
Schmidhuber, 1997). We describe both the design of  $f$ , which determines the output size of the convolutional encoding space  $\mathcal{H}$ , and  $g$  in Section 5.1. Together, both the convolutional and contextual layers form the encoder of text line images used for downstream document transcription. Ideally,  $f$  will capture the underlying visual appearance of distinct character classes, while  $g$  will discover linguistic correlations between these classes.
3.2 Masking
During pre-training, we replace randomly sampled, non-overlapping segments of  $H$  with a learned mask embedding vector prior to computing contextualized representation matrix  $C$ . We train the model to distinguish the masked region from a foil using the contrastive loss presented in Section 3.3.
3.3 Pre-training Objective
We use the following self-supervised contrastive loss whose variants have demonstrated success in self-supervised representation learning (Oord et al.,
2018; Baevski et al., 2020):
$$
\mathcal {L} _ {U} (c _ {t}) = - \log \frac {\exp \left(s (c _ {t} , h _ {t})\right)}{\sum_ {t ^ {\prime}} \exp \left(s (c _ {t} , h _ {t ^ {\prime}})\right)}
$$
Here,  $c_{t}$  (depicted in Figure 2) is the contextual encoder's output representation of the masked line image at position  $t$ . Similarly,  $h_{t}$  (also depicted in Figure 2) is the convolutional encoder's output representation of the masked region itself. Further,  $s(c,h)$  represents a scoring function that computes the similarity between representation vectors  $c$  and  $h$ . We use the cosine similarity similar to Baevski et al. (2020), but compute it using only raw vectors, instead of the raw vectors and quantized vectors. The cross-entropy loss requires the model to distinguish the representation of the true masked region,  $h_{t}$ , from distractor representations: the convolutional encodings of other segments,  $h_{t'}$  with  $t' \neq t$ .
3.4 Fine-tuning Objective
After learning pre-trained representations, we add the randomly initialized, fully connected character vocabulary projection layer to the top of our context encoder network (top right of Fig. 2) and perform supervised training using the Connectionist Temporal Classification (CTC) objective (Graves et al., 2006; Graves, 2012; Baevski et al., 2020) with transcribed data. CTC is a commonly used loss function for supervised training in speech and handwriting recognition systems. In this case, CTC is used to marginalize over all monotonic alignments between the sequence of input visual representations and the observed ground truth output sequence of characters.
In this section, we describe the unlabeled pretraining and labeled fine-tuning/testing datasets used in our experiments. Representative line images from five of the datasets are exhibited in Figure 3.
4.1 Islamicate Manuscripts
First, we introduce a variety of pre-modern Islamicate manuscript datasets (i.e., Persian and Arabic handwritten documents from c.  $7^{\text{th}} - 19^{\text{th}}$  centuries) selected for both their uniquely different domain
Figure 3: Assortment of cropped, grayscale line images from a selection of our datasets, as extracted by annotators. From top to bottom, RASM 2019 (Keinan-Schoonbaert, 2020), Attar-Mubhij, Huliyya, Trove (Holley, 2010), Old Bailey (Shoemaker, 2005). The Islamicate line images are shown pre-binarization, while the English line images come binarized.
content (e.g., scientific to legal to religious) and their visually distinct scribal handwriting style. All but the first pre-train dataset are professionally transcribed by Islamicate manuscript scholars.
HMML Pre-train Through a collaboration with the Hill Museum and Manuscript Library (HMML), we obtain about 100 early modern, mostly Syrian,  $\mathrm{nashk}^2$  manuscripts dating from 1600-1775 with some vowels, but with ornamentally voweled texts excluded (i.e., texts in which every single vowel and orthographic feature is included, usually for ornamental reasons). We filter out manuscripts with extensive marginalia, figures, or tables, though some marginal notes and other elements (e.g., seals, interlinears) are still present. This results in a dataset containing roughly 750,000 unlabeled line images.
HMML Fine-tune We obtain transcriptions for 115 line images from a 4-page held-out subset of the HMML Pre-train dataset. This dataset is designed for in-domain fine-tuning/testing experiments with our pre-trained models.
RASM 2019 For the ICDAR 2019 Competition on Recognition of Historical Arabic Scientific Manuscripts, the British Library released 2,164 transcribed line images from scientific manuscripts written in various scribal hands (Keinan-Schoonbaert, 2020). RASM 2019 has become a popular benchmark for Arabic-script
2https://en.wikipedia.org/wiki/Naskh_(script)
handwriting recognition due to its relatively large amount of supervised data for the task.
Attar-Mubhij An Arabic-language legal text with 190 transcribed line images obtained from OpenITI.3
Huliyya A 229-line Persian, nata'liq<sup>4</sup> devotional text written by an early modern scholar containing mostly prayers (also obtained from Open-ITI).
4.2 Early Modern English Printed Works
Next, we describe several English book and newspaper datasets used in our experiments that were originally printed in early modern England and Australia.
EEBO Pre-train We harvest 750,000 unlabeled line images from a randomly sampled collection of document images from Early English Books Online (EEBO),<sup>5</sup> which contains "almost every work printed in the British Isles and North America, as well as works in English printed elsewhere from 1470-1700."
Trove A dataset of historic Australian newspapers (c. 1803-1954) from the National Library of Australia (Holley, 2010). We use the manually transcribed version totaling 450 lines (Berg-Kirkpatrick et al., 2013).
Old Bailey A manually transcribed set of 20 documents printed 1716-1906, consisting of 30 lines per document, taken from Berg-Kirkpatrick and Klein (2014). Shoemaker (2005) compiled the documents, which describe proceedings of London's Old Bailey Courthouse.
4.3 Line Extraction
Since our model processes individual line images of a document, we use Kiessling (2020)'s line extraction method to automatically segment page images into their component text line images for at-scale collection of the pre-training datasets. We find and discard poorly extracted line images outside an empirically determined pixel width-to-height ratio range of 6-23.
$^{3}$ https://openiti.org  
 $^{4}$ https://en.wikipedia.org/wiki/Nastaliq  
 $^{5}$ https://www.proquest.com/eebo
In this section, we describe our experimental setup, including architectural details and hyperparameters for the neural line image encoder, pre-train/finetune specifics, dataset splits, and the baseline systems we compare against.
Encoder For all experiments, we binarize the line images and scale them to a height of 96 pixels, but allow them to vary in width. We base our CNN architecture on the Kraken OCR system (Kiessling, 2019): two rectangular  $4 \times 2$  kernels first process the input image, each followed by a Leaky ReLU activation and Group Norm. Two max pooling operations are applied, one before and one after the final  $3 \times 3$  convolutional layer kernel, with kernel sizes/strides of  $4 \times 2 / 1 \times 2$  for both. The first kernel uses a stride of  $4 \times 2$  and the final two both use  $1 \times 1$ . The convolutional hidden dimensions are 64, 128, and 256. We use a 3-layer BiLSTM for our contextual encoder with a hidden size of 512. This results in 6,408,000 trainable parameters. Models are implemented in PyTorch (Paszke et al., 2019) and Fairseq (Ott et al., 2019). Code is available at https://github.com/nvog/lacuna.
Pre-training During pre-training, we perform a grid search over masking probability and length using 75k lines of data and select the best model based on lowest fine-tuned CER on HMML Finetune. We determine  $p = 0.5 / p = 0.65$  to perform best for Islamicate manuscript/English print with a non-overlapping segment length of 12 time steps. We ensure that 8 time steps are between each non-overlapping segment. A maximum of 100 time steps are sampled and used as foils in the denominator of the loss from Sec. 3.3. We use the same learning rate scheduler and Adam optimizer from Baevski et al. (2020) that warms up for the first  $8\%$  of updates to a learning rate of 5e-4 and linearly decays it afterwards. Models are pre-trained for 3-5 days on 4 RTX 2080 Ti cards.
Fine-tuning During fine-tuning, we use a tri-stage learning rate schedule with the Adam optimizer, which warms up the learning rate to 5e-4 during the first  $10\%$  of updates and decays it linearly by a factor of 0.05 for the final  $50\%$  of training. We only update the fully connected layer for
System	Test Dataset CER (↓)
HMML-F	RASM	Attar-Mubhij	Huliyya
Google Cloud OCR	49.0	57.0	61.2	71.4
30 Lines for Supervised Fine-tuning
# Lines Pre-train	Fine-tune/Test Dataset CER (↓)
HMML-F	RASM	Attar-Mubhij	Huliyya
0	51.0	68.9	60.4	70.3
75k	22.7	46.1	30.4	52.9
750k	14.8	36.2	23.7	45.5
90 Lines for Supervised Fine-tuning
# Lines Pre-train	Fine-tune/Test Dataset CER (↓)
HMML-F	RASM	Attar-Mubhij	Huliyya
0	36.9	61.7	36.8	52.5
75k	15.2	34.4	20.8	37.5
750k	10.0	25.9	15.0	28.3
Table 1: Document transcription results on Islamicate manuscripts. Character error rate (CER) is reported on held-out test sets introduced in Section 4.1. For baselines, we compare against the current Google Cloud OCR via the API, and the state-of-the-art, neural network-based architecture from Kraken (Kiessling, 2019), which does not use self-supervised pre-training (i.e., 0 lines pre-train). With access to the same amount of 30 and 90 lines of supervised fine-tuning data as this system, our proposed self-supervised pre-training regime (using  $75\mathrm{k}$  and  $750\mathrm{k}$  lines of unlabeled manuscript data) shows a large improvement across all datasets.
the first 200 epochs of training and then proceed to update the contextual encoder as well. These optimization choices are inspired by Baevski et al. (2020). We use a small batch size of 8 and train for a maximum of 700 epochs with the CTC loss (Sec. 3.4). We use greedy decoding after removing the CTC's blank token and do not use any external language model. For Islamicate manuscript experiments we perform NFD unicode normalization. Character Error Rate (CER) is computed using Kraken OCR (Kiessling, 2019).
5.2 Fine-tune/Test Splits