SlowGuess/NAACL_2022_OCR huggingface dataset:
For Islamicate manuscript datasets, we hold out  $10\%$  of RASM 2019 for testing and the final page each of HMML Fine-tune, Attar-Mubhij, and Huliyya. For English print datasets, we use the same test splits as Berg-Kirkpatrick and Klein (2014) for fair comparison and fine-tune on the validation set of each dataset.
For our first baseline, we use the proprietary Google Cloud OCR API (Fujii et al., 2017; Ingle et al., 2019), which provides state-of-the-art results on multilingual handwritten and printed modern documents. In contrast to our system's unlabeled pre-training procedure for historical documents, this system uses synthesized handwriting strokes and data perturbation to obtain more supervised data for improved performance.
For our second baseline, we use the popular, state-of-the-art open-source Kraken OCR (Kiessling, 2019), which consists of a CNN-LSTM encoder trained in a supervised fashion with the same segmentation-free Connectionist Temporal Classification (Graves et al., 2006) loss function we use during our method's fine-tuning stage. We provide the encoder's implementation details in Section 5.1.
For early modern English print experiments, we also compare to the fully unsupervised Ocular (Berg-Kirkpatrick et al., 2013), which is a generative probabilistic model purpose-built for the historical printing process, yet unable to handle complex glyph variation observed in handwriting.
In this section, we present document transcription results for both Islamicate manuscripts and early modern English works introduced in Section 4. We compare performance against supervised and unsupervised prior work, and investigate the impact of pre-training/fine-tuning dataset sizes.
6.1 Islamicate Manuscripts
In Table 1, we present single-run supervised finetuning results on in-domain subsets of each dataset limited to 30 and 90 lines for low-resource setting evaluation. These two settings are roughly equivalent to 1 and 3 pages of transcribed data for each manuscript. Each row represents a dif
Baselines
System	Test Dataset CER (↓)
Trove	Old Bailey
Google Tesseract	37.5	-
ABBYY FineReader	22.9	-
Ocular	14.9	14.9
Ocular Beam	12.9	10.9
Ocular Beam-SV	11.2	10.3
Google Cloud OCR	13.3	8.5
30 Lines for Supervised Fine-tuning
# Lines Pre-train	Test Dataset CER (↓)
Trove	Old Bailey
0	70.5	60.0
75k	20.3	26.5
750k	19.6	12.2
90 Lines for Supervised Fine-tuning
# Lines Pre-train	Test Dataset CER (↓)
Trove	Old Bailey
0	38.7	28.6
75k	12.2	9.4
750k	10.4	7.6
Table 2: Document transcription results on early modern English printed works. Character error rate (CER) is reported on held-out test sets introduced in Section 4.2. First 5 baselines are taken from Berg-Kirkpatrick and Klein (2014). Similar to Table 1, supervised data is limited to 30 and 90 line settings.
ferent set of encoder parameters, which we use to initialize the fine-tuning experiments. The  $0 \#$  lines pre-train row represents a randomly initialized Kraken-style encoder, while  $75k$  and  $750k$  settings use the encoder parameters pre-trained with our lacuna reconstruction objective on different orders of magnitude of unlabeled HMML Pre-train line images. We also compare to the Google Cloud OCR introduced in Section 5.3.
The first thing we can observe is the extremely high character error rates for both the commercial Google Cloud OCR system and the randomly initialized 0k pre-train models, especially in the 30-line setting. Access to about 2 more pages of data (in the 90-line setting) improves results for this setting in the Arabic-language legal text Attar
Mubhij, but does not seem to help much for RASM 2019, a larger collection of scientific manuscripts. This is probably due to the higher amount of diversity in content and style in this benchmark dataset for Arabic-language HTR. Seemingly, without any signal from pre-training and only tens of lines of transcribed data, the model is unable to learn a sufficient visual encoder for the large variety of scribal hands and scripts observed in the manuscripts (examples shown in Fig. 3). Pre-training on just  $75\mathrm{k}$  lines halves the error rate for Attar-Mubhij in the 30-line setting. Furthermore,  $750\mathrm{k}$  pre-train reduces the Attar-Mubhij CER from 60.4 to 23.7.
The HMML Fine-tune dataset (HMML-F in Table 1) has the largest relative error rate difference between the pre-trained models and models without pre-training. Errors are reduced by about  $55\%$  for  $75\mathrm{k} - 30$ ,  $70\%$  for  $750\mathrm{k} - 30$ ,  $58\%$  for  $75\mathrm{k} - 90$ , and  $73\%$  for  $750\mathrm{k} - 90$ , which is at least 10 points higher than other datasets on average. Since manuscripts in HMML-F are sourced from the same library as the HMML Pre-train dataset, the results suggest that in-domain pre-training data provides an advantage over the other documents from different collections. Regardless, our approach's improved performance on 30-line settings compared to the supervised 90-line results trained from scratch across all datasets is impressive and shows promising generalization ability.
6.2 Early Modern English Printed Works
In Table 2, we present supervised fine-tuning results on in-domain subsets of each dataset limited to the same 30 and 90 line settings as in the Islamicate manuscript experiments. Our first observation is that the randomly initialized encoder from the 0-line pre-train setting sees a much larger improvement from 30 to 90 lines of supervised fine-tuning data than the Islamicate manuscript experiments. We speculate this is due to the more similar and repeated glyph shapes on printed data compared to handwritten data, which makes learning of the visual encoder easier. Still, pre-training the visual encoder cuts CER across both datasets, though we do see a slightly bigger relative error rate reduction when fine-tuning on Trove versus Old Bailey.
In Figures 4 & 5, we show comparisons across predicted transcriptions from different systems and datasets for illustrative purposes. First, we observe
Line image:	so far as the Serpentine within forty or fifty yards—it was not near Apaley
Ground truth:	so far as the Serpentine within forty or fifty yards-it was not near Apaley
Google Cloud:	Bo far as the Nerpentine within forty or fifty yards-it WAB Dot near Apaley
0k Pre-train:	tshr othe rcpensin witlin forty or fifty yardl--it wn tot mor Amlen
75k Pre-train:	to sar as the Perpentine witlin forty or fifty yards—it was not near Aptley
750k Pre-train:	so far as the Serpentine within forty or fifty yards-it was not near Apaley
Figure 4: Comparison of results on the Old Bailey test set with errors highlighted. Pre-trained results are from the 90-line fine-tuning setting.
Line image:	sale are given by the Sportsman's "Special
Ground truth:	sale are given by the Sportsman's "Special
Google Cloud:	Bale are giveu by the Spmrtsinan's "Special
0k Pre-train:	ule are givemn by the fpvrruzon peceinl
75k Pre-train:	sale ars given by the Sportsmon's "Special
750k Pre-train:	sale are given by the Sportsmon's "Special
Figure 5: Comparison of results on the Trove test set with errors highlighted. Pre-trained results are from the 90-line fine-tuning setting.
that Google Cloud OCR, the best baseline system on Old Bailey, consistently struggles with inkling variation. For example, the bleeding ink on the initial 's' of each line image is mistaken for a 'B', the 'n' in 'not' in Fig. 4 is mistaken for a 'D' due to the subtle connection of the glyph's legs from overinking, and the 'm' in 'Sportsman' in Fig 5 is confused for the characters 'in' because of underinking. However, the 0k pre-train baseline clearly makes the most insertion/deletion/substitution errors since it must learn how to transcribe noisy line images from a randomly initialized encoder using only 90 transcribed line images for supervised parameter learning. Initializing the visual encoder with parameters learned from our self-supervised regime on 75k unlabeled line images from EEBO reduces a lot of these nonsensical errors to only superficial glyph recognition issues. By increasing the pre-training amount by an order of magnitude to 750k, we obtain our best results. Future work could integrate a language model during decoding to address the unlikely sequences of characters/words still output by our best system, like the words 'Apaley' and 'Sportsmon'.
In this paper, we proposed a two-phase pretrain/fine-tune approach for document transcript
tion and applied it to historical documents in low-resource settings. Our pre-training strategy, inspired by reconstructing missing information, or lacuna, in documents uses hundreds of thousands of unlabeled line images to learn rich visual language representations. After supervised fine-tuning on tens of transcribed line images, we showed large character error rate reduction on Islamicate manuscripts exhibiting major script and style variation and we improved over several state-of-the-art OCR systems on early modern English printed works. We estimate that our approach could save human annotators significant amounts of time and enable more distant readings of library collections.
While more accurate transcription of printed and handwritten documents in low-resource settings can expand research access for language and history scholars, it could also potentially facilitate government surveillance of marginalized communities. Separately, bad actors could more easily scan and digitize document images containing sensitive information and use them for nefarious purposes.
This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE-2038238. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. The project is also funded in part by the NSF under grants 1618044 and 1936155.
Mansoor Alghamdi and William Teahan. 2017. Experimental evaluation of arabicOCR systems. *PSU Research Review*, 1(3):229-241.
Kenning Arlitsch and John Herbert. 2004. Microfilm, paper, andOCR: Issues in newspaper digitization. the utah digital newspapers program.
Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. 2019. Effectiveness of self-supervised pretraining for speech recognition. arXiv preprint arXiv:1911.03912.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33.
Martijn Bartelds, Wietse de Vries, Faraz Sanal, Caitlin Richter, Mark Liberman, and Martijn Wieling. 2020. Neural representations for modeling variation in english speech. arXiv preprint arXiv:2011.12649.
Taylor Berg-Kirkpatrick, Greg Durrett, and Dan Klein. 2013. Unsupervised transcription of historical documents. In ACL.
Taylor Berg-Kirkpatrick and Dan Klein. 2014. Improved typesetting models for historicalOCR. In ACL.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2019. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Pre-training transformers as energy-based cloze models. arXiv preprint arXiv:2012.08561.
C. Clausner, A. Antonacopoulos, N. Mcgregor, and D. Wilson-Nunn. 2018. Icfhr 2018 competition on recognition of historical arabic scientific manuscripts - rasm2018. In 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 471-476.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Rui Dong and David A Smith. 2018. Multi-input attention for unsupervisedOCR correction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2363-2372.
Yasuhisa Fujii, Karel Driesen, Jonathan Baccash, Ash Hurst, and Ashok C Popat. 2017. Sequence-to-label script identification for multilingualOCR. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 1, pages 161-168. IEEE.
Dan Garrette and Hannah Alpert-Abrams. 2016. An unsupervised model of orthographic variation for historical document transcription. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 467-472.
Dan Garrette, Hannah Alpert-Abrams, Taylor Berg-Kirkpatrick, and Dan Klein. 2015. Unsupervised code-switching for multilingual historical document transcription. In *NAACL*.
Kartik Goyal, Chris Dyer, Christopher Warren, Max G'Sell, and Taylor Berg-Kirkpatrick. 2020. A probabilistic generative model for typographical analysis of early modern printing. In Proceedings of 2020 Annual Conference of the Association for Computational Linguistics.
Alex Graves. 2012. Offline arabic handwriting recognition with multidimensional recurrent neural networks. In Guide to OCR for Arabic scripts, pages 297-313. Springer.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376.
Mika Hämäläinen and Simon Hengchen. 2019. From the paft to the finiture: a fully automatic nmt and word
embeddings method forOCR post-correction. arXiv preprint arXiv:1910.05535.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Rose Holley. 2010. Trove: Innovation in access to information in australia. Ariadne, (64).
R Reeve Ingle, Yasuhisa Fujii, Thomas Deselaers, Jonathan Baccash, and Ashok C Popat. 2019. A scalable handwritten text recognition system. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 17-24. IEEE.
Jose Carlos Aradillas Jaramillo, Juan Jose Murillo-Fuentes, and Pablo M Olmos. 2018. Boosting handwriting text recognition in small databases with transfer learning. In 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 429-434. IEEE.
Dongwei Jiang, Xiaoning Lei, Wubo Li, Ne Luo, Yuxuan Hu, Wei Zou, and Xiangang Li. 2019. Improving transformer-based speech recognition using unsupervised pre-training. arXiv e-prints, pages arXiv-1910.
Adi Keinan-Schoonbaert. 2019. Using transkribus for arabic handwritten text recognition. *British Library Digital Scholarship Blog*.
Adi Keinan-Schoonbaert. 2020. Results of the rasm2019 competition on recognition of historical arabic scientific manuscripts. British Library Digital Scholarship Blog.
Benjamin Kiessling. 2019. Kraken-an universal text recognizer for the humanities. Proceedings of the  $DH$ .
Benjamin Kiessling. 2020. A modular region and text line layout analysis system. In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 313-318. IEEE.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairyseq: A fast, extensible toolkit for sequence modeling.* arXiv preprint arXiv:1904.01038.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8037.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020.
Shruti Rijhwani, Antonios Anastasopoulos, and Graham Neubig. 2020. Ocr post correction for endangered language texts. arXiv preprint arXiv:2011.05402.
Shruti Rijhwani, Daisy Rosenblum, Antonios Anastasopoulos, and Graham Neubig. 2021. Lexically aware semi-supervised learning forOCR post-correction. Transactions of the Association for Computational Linguistics, 9:1285-1302.
Maxim Romanov, Matthew Thomas Miller, Sarah Bowen Savant, and Benjamin Kiessling. 2017. Important new developments in arabographic optical character recognition (ocr). arXiv preprint arXiv:1703.09550.
Robert Shoemaker. 2005. Digital london: Creating a searchable web of interlinked sources on eighteenth century London. Program.
Xingchen Song, Guangsen Wang, Yiheng Huang, Zhiyong Wu, Dan Su, and Helen Meng. 2020. Speech-Xlnet: Unsupervised acoustic model pretraining for self-attention networks. Proc. Interspeech 2020, pages 3765-3769.
Joan Andreu Sánchez, Verónica Romero, Alejandro H. Toselli, Mauricio Villegas, and Enrique Vidal. 2019. A set of benchmarks for handwritten text recognition on historical documents). Pattern Recognition, 94:122-134.
Chaitanya Talnikar, Tatiana Likhomanenko, Ronan Collobert, and Gabriel Synnaeve. 2020. Joint masked cpc and ctc training for asr. arXiv e-prints, pages arXiv-2011.
Wilson L Taylor. 1953. "cloze procedure": A new tool for measuring readability. Journalism quarterly, 30(4):415-433.
Weiran Wang, Qingming Tang, and Karen Livescu. 2020. Unsupervised pre-training of bidirectional speech encoders via masked reconstruction. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6889-6893. IEEE.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems, 32:5753-5763.