SlowGuess/ACL_2020_OCR huggingface dataset:
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, pages 282-289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270, San Diego, California. Association for Computational Linguistics.
Vladimir Iosifovich Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics-Doklady, 10(8).
Nuno Mamede, Jorge Baptista, and Francisco Dias. 2016. Automated anonymization of text documents. In 2016 IEEE Congress on Evolutionary Computation (CEC), pages 1287-1294. IEEE.
Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. 2017. Adversarial training methods for semi-supervised text classification. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings.
Marcin Namysl and Iuliu Konya. 2019. Efficient, lexicon-free OCR using deep learning. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 295-301.
Clemens Neudecker. 2016. An open corpus for named entity recognition in historic newspapers. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 4348-4352, Portoorž, Slovenia. European Language Resources Association (ELRA).
Carolina Parada, Mark Dredze, and Frederick Jelinek. 2011. OOV sensitive named-entity recognition in speech. In INTERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association, Florence, Italy, August 27-31, 2011, pages 2085-2088.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages
2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Aleksandra Piktus, Necati Bora Edizel, Piotr Bojanowski, Edouard Grave, Rui Ferreira, and Fabrizio Silvestri. 2019. Misspelling oblivious word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3226-3234, Minneapolis, Minnesota. Association for Computational Linguistics.
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147-155, Boulder, Colorado. Association for Computational Linguistics.
Graham Rawlinson. 2007. The significance of letter position in word recognition. IEEE Aerospace and Electronic Systems Magazine, 22(1):26-27.
C. Rigaud, A. Doucet, M. Coustaty, and J. Moreux. 2019. ICDAR 2019 competition on post-OCR text correction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1588-1593.
Allen Schmaltz, Yoon Kim, Alexander Rush, and Stuart Shieber. 2017. Adapting sequence models for sentence correction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2807-2813, Copenhagen, Denmark. Association for Computational Linguistics.
Sarah Schulz and Jonas Kuhn. 2017. Multi-modular domain-tailored OCR post-correction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2716-2726, Copenhagen, Denmark. Association for Computational Linguistics.
Ray Smith. 2007. An overview of the Tesseract OCR engine. In Proceedings of the Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), volume 2, pages 629-633.
Matthias Sperber, Jan Niehues, and Alex Waibel. 2017. Toward robust neural machine translation for noisy input sequences. In The International Workshop on Spoken Language Translation (IWSLT), Tokyo, Japan.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Learning Representations, ICLR 2014.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
D. Wemhoener, I. Z. Yalniz, and R. Manmatha. 2013. Creating an improved version using noisy OCR from multiple editions. In 2013 12th International Conference on Document Analysis and Recognition, pages 160-164.
Michihiro Yasunaga, Jungo Kasai, and Dragomir Radev. 2018. Robust multilingual part-of-speech tagging via adversarial training. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 976-986, New Orleans, Louisiana. Association for Computational Linguistics.
Stephan Zheng, Yang Song, Thomas Leung, and Ian J. Goodfellow. 2016. Improving the robustness of deep neural networks via stability training. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 4480-4488.
A Noise Model - Supplementary Materials
In this section, we present the extended description of our vanilla noise model introduced in §3.1. Let  $P_{edit} = \eta / 3$  be the probability of performing a single character edit operation (insertion, deletion, or substitution) that replaces the source character  $c$  with a noisy character  $\tilde{c}$ , where  $\tilde{c} \neq c$ . Equation (1) defines the vanilla error distribution, which we use at training time:
$$
P (\tilde {c} | c) = \left\{ \begin{array}{l} \frac {P _ {e d i t}}{| \Sigma \backslash \{\varepsilon \} |}, \text {i f} c = \varepsilon \text {a n d} \tilde {c} \neq \varepsilon . (1 \mathrm {a}) \\ 1 - P _ {e d i t}, \text {i f} c = \varepsilon \text {a n d} \tilde {c} = \varepsilon . (1 \mathrm {b}) \\ \frac {P _ {e d i t}}{| \Sigma \backslash \{c , \varepsilon \} |}, \text {i f} c \neq \varepsilon \text {a n d} \tilde {c} \neq c. (1 \mathrm {c}) \\ P _ {e d i t}, \text {i f} c \neq \varepsilon \text {a n d} \tilde {c} = \varepsilon . (1 \mathrm {d}) \\ 1 - 2 P _ {e d i t}, \text {i f} c \neq \varepsilon \text {a n d} \tilde {c} = c. (1 \mathrm {e}) \end{array} \right.
$$
It consists of the following components:
(a) The insertion probability  $P_{ins}(\tilde{c}|\varepsilon)$  in eq. (1a). It describes how likely it is to insert a nonempty character  $\tilde{c} \neq \varepsilon$  and it is uniform over the set of all characters from the alphabet  $\Sigma$ , except the  $\varepsilon$  symbol.
(b) The keep  $\varepsilon$  probability  $P_{\text{keep}}(\varepsilon|\varepsilon)$  in eq. (1b).
(c) The substitution probability  $P_{subst}(\tilde{c}|c)$  in eq. (1c). It is uniform over the set of all characters from the alphabet  $\Sigma$ , except the source character  $c$  and the  $\varepsilon$  symbol.
(d) The deletion probability  $P_{del}(\varepsilon |c)$  in eq. (1d).
(e) The keep probability  $P_{\text{keep}}(c|c)$  in eq. (1e).
Equations (1a) and (1b) correspond to the row in the character confusion matrix  $\Gamma$ , where  $c = \varepsilon$  and form a valid probability distribution:
$$
P_{\text{keep}}(\varepsilon |\varepsilon) + \sum_{\tilde{c}\in \Sigma \setminus \{\varepsilon \}}P_{ins}(\tilde{c} |c) = 1.
$$
Similarly, eqs. (1c) to (1e) correspond to the rows in the character confusion matrix  $\Gamma$ , where  $c \in \Sigma \backslash \{\varepsilon\}$ , and are also valid probability distributions:
$$
P _ {d e l} (\varepsilon | c) + P _ {k e e p} (c | c) + \sum_ {\tilde {c} \in \Sigma \backslash \{c, \varepsilon \}} P _ {s u b s t} (\tilde {c} | c) = 1
$$
Finally, for comparison, we present visualizations of the confusion matrices used in our vanilla (Figure 7a) and OCR error models (Figure 7b).
B Extended evaluation results
B.1 Sensitivity Analysis
In this section, we present the extended version of our sensitivity study (§4.3). Figure 8 summarizes the results on the synthetic data distribution with various test- and training-time noise levels ( $\eta_{test}$  and  $\eta_{train}$ , respectively) and weighting factors  $\alpha$ . We noticed a similar trend as in our initial analysis. As the level of noise  $\eta_{test}$  increases, the overall accuracy decreases, but this trend is less pronounced for  $\alpha \neq 0$ . At the same time, the gap between the models trained with and without our auxiliary objectives becomes larger.
B.2 Qualitative Analysis
In this section, we compared the outputs generated by the baseline models trained with and without our auxiliary training objectives (Table 2). We found that the NAT method improved robustness to capitalization errors (the first and the fourth row in Table 2a) as well as to substitutions (the second, the third and the fifth row in Table 2a and the first, the second, the fourth and the fifth row in Table 2b), deletions (the fifth row in Table 2a) and insertions
of characters (the third and the fifth row in Table 2b). Moreover, it better recognized the semantics of the sentence in the third row of Table 2a, where the location name was creatively rewritten (Brazland instead of Brazil).
We present the detailed hyper-parameters of the sequence labeling model  $f(x)$  used in our experiments (§4). Note that dropout was applied both before and after the LSTM layer (Table 3).
Parameter name	Parameter value
Tagging schema	BIOES
Mini batch size	32
Max. epochs	100
LSTM # hidden layers	1
LSTM # hidden units	256
Optimizer	SGD
Initial learning rate	0.1
Learning rate anneal factor	0.5
Minimum learning rate	0.0001
Word dropout level	0.05
Variational dropout level	0.5
Patience	3
Table 3: Hyper-parameters of the sequence labeling model  $f\left( x\right)$  used in our experiments.
D Data Set Statistics and Estimated Error Rates
In this section, we present the detailed statistics of the data sets used in our NER experiments (Table 4). Following Akbik et al. (2018), we used the revisited version of German CoNLL 2003, which was prepared in 2006 and is believed to be more accurate, as the previous version was done by non-native speakers<sup>14</sup>. Moreover, we used only the inner layer of annotation for GermEval 2014.
Finally, in Table 5, we report estimated error rates for all data sets and all noising procedures used in our experiments.
14The revisited annotations are available on the official website of the CoNLL 2003 shared task: https://www. clips.uantwerpen.be/conll2003/ner/.
1.	Reference result	7-1 Raul <B-PER> Gonzalez <E-PER> 7-1 Juan <B-PER> Pizzi <E-PER>
NAT output	7-1 raul <B-PER> gonzalez <E-PER> 7-1 juan <B-PER> Pizzi <E-PER>
Baseline output	7-1 raul gonzalez <S-PER> 7-1 juan Pizzi <S-PER>
2.	Reference result	6. Heidi <B-PER> Zurbriggen <E-PER> (Switzerland <S-LOC>) 153
NAT output	6. Heidi <B-PER> Zurbriggen <E-PER> (switzerland <S-LOC>) 153
Baseline output	6. Heidi <B-PER> Zurbriggen <E-PER> (switzerland) 153
3.	Reference result	Plastic surgery gets boost in Brazil <S-LOC> .
NAT output	Plastic surgury hets boost is Brazil <S-LOC> .
Baseline output	Plastic surgury hets boost is Brazil <S-PER> .
4.	Reference result	Waltraud <B-PER> Zimmer <E-PER>, Rödermark-Ober-Roden <S-LOC>
NAT output	Waltraud <B-PER> zimmer <E-PER>, Rödermark-Ober-Roden <S-LOC>
Baseline output	Waltraud <S-PER> zimmer, Rödermark-Ober-Roden <S-LOC>
5.	Reference result	Deutschland <S-LOC> ist noch nicht Teil der Reiseroute . "
NAT output	Deutschland <S-LOC> is nach rich Teil der Reiseroute . "
Baseline output	Deutschland <S-PER> is nach rich Teil der Reiseroute . "
(a) Misspellings.
Reference result	Hapoel <B-ORG> Jerusalem <E-ORG> 12 4 1 7 10 18 13
1. NAT output	Hapoel <B-ORG> lerusalem <E-ORG> I2 A 1 7 10 18 13
Baseline output	Hapoel <S-ORG> lerusalem I2 A 1 7 10 18 13
Reference result	SOCER - SPANISH <S-MISC> FIRST DIVISION RESULT / STANDINGS .
2. NAT output	SOCER - SPANlSH <S-MISC> FIRST DIVISOW RESULT / STA'DINGS .
Baseline output	SOCER - SPANlSH <S-PER> FIRST DIVISOW RESULT / STA'DINGS .
Reference result	EU <S-ORG>, Poland <S-LOC> agree on oil import tariffs .
3. NAT output	EU <S-ORG>, Po'land <S-LOC> agree on oil import tariffs .
Baseline output	EU <S-ORG>, Po'land <S-ORG> agree on oil import tariffs .
Reference result	Schlamm scheint zu halten - Yahoo <B-ORG> ! <E-ORG>
4. NAT output	Schlamm scheint zu halten - Yahoo0 <B-ORG> ! <E-ORG>
Baseline output	Schlamm scheint zu halten - Yahoo0 <S-PER> !
Reference result	Fachverband <B-ORG> für <I-ORG> Hauswirtschaft <E-ORG> :
5. NAT output	Fachverbandi <B-ORG> für <I-ORG> Hauswi'tschaTt <E-ORG> :
Baseline output	Fachverbandi für Hauswi'tschaTt :
(b) OCR errors.
Table 2: Outputs produced by the models trained with and without our auxiliary NAT objectives (NAT output and Baseline output, respectively). We demonstrate examples that contain misspellings and OCR errors, where the models trained with the auxiliary NAT objectives correctly recognized all tags, while the baseline models either misclassified or completely missed some entities.
Train	Dev	Test	Total
Sentences	14,041	3,250	3,453	20744
Tokens	203,621	51,362	46,435	301418
PER	6,600	1,842	1,617	10059
LOC	7,140	1,837	1,668	10645
ORG	6,321	1,341	1,661	9323
MISC	3,438	922	702	5062
(a) English CoNLL 2003.
Train	Dev	Test	Total
Sentences	12,705	3,068	3,160	18933
Tokens	207,484	51,645	52,098	311227
PER	2,801	1,409	1,210	5420
LOC	4,273	1,216	1,051	6540
ORG	2,154	1,090	584	3828
MISC	780	216	206	1202
(b) German CoNLL 2003.
Train	Dev	Test	Total
Sentences	24,000	2,200	5,100	31300
Tokens	452,853	41,653	96,499	591005
PER	7,679	711	1,639	10029
PER-deriv	62	2	11	75
PER-part	184	18	44	246
LOC	8,281	763	1,706	10750
LOC-deriv	2,808	235	561	3604
LOC-part	513	52	109	674
ORG	5,255	496	1,150	6901
ORG-deriv	41	3	8	52
ORG-part	805	91	172	1068
MISC	3,024	269	697	3990
MISC-deriv	236	16	39	291
MISC-part	190	18	42	250
(c) GermEval 2014.
OCR noise	Mis-spellings†	Mis-spellings‡
English CoNLL 2003	8.9%	16.5%	9.8%
German CoNLL 2003	9.0%	8.3%	8.0%
GermEval 2014	9.3%	8.6%	8.2%
(a) Character Error Rates.
OCR noise	Mis-spellings†	Mis-spellings‡
English CoNLL 2003	35.6%	55.4%	48.3%
German CoNLL 2003	39.5%	26.5%	45.5%
GermEval 2014	41.2%	27.0%	47.9%
(b) Word Error Rates.
Table 5: Error rate estimation for different noise distributions. OCR noise is modeled with the character confusion matrix, whereas misspellings are induced using look-up tables released by Belinkov and Bisk (2018)† and Piktus et al. (2019)‡.
Table 4: Statistics of the data sets used in our NER experiments (§4). We present statistics of the training (Train) development (Dev) and test (Test) sets, including the number of sentences, tokens, and entities: person names (PER), locations (LOC), organizations (ORG) and miscellaneous (MISC). The GermEval 2014 data set defines two additional fine-grained sub-labels: "-part" and "-deriv" that mark derivation and compound words, respectively, which stand in direct relation to Named Entities.
(a) Vanilla error distribution used at training time  $(\eta = 20\%)$ .
(b) Real error distribution estimated from a large document corpus using the Tesseract OCR engine.
Figure 7: Confusion matrices for the vanilla and the OCR error distributions. Each cell represents  $P(\tilde{c} | c)$ . The rows correspond to the original characters  $c$  and the columns represent the perturbed characters  $\tilde{c}$ . In this example, we include all symbols from the alphabet of the English CoNLL 2003 data set. The vanilla noise model assigns equal probability to all substitution errors, while the OCR error model is biased towards substitutions of characters with similar shapes like "I"→"1", "$"→"5", "0"→"0" or ",","→". Moreover, the vanilla model assumes that the deletion of a character  $c$  is as likely as the sum of substitution probabilities with all non-empty symbols:  $P_{del}(\varepsilon | c) = \sum_{\tilde{c} \in \Sigma \setminus \{\varepsilon\}} P_{subst}(\tilde{c} | c)$ .
(a) Data augmentation objective (synthetic noise:  $\eta_{test} = 1\%$
(c) Data augmentation objective (synthetic noise:  $\eta_{test} = 5\%$
(e) Data augmentation objective (synthetic noise:  $\eta_{test} = 10\%$
Figure 8: Extended results of our sensitivity analysis on the English CoNLL 2003 test data (§B.1). Each figure presents the results of models trained using one of our auxiliary training objectives on the original data perturbed with various levels of synthetic noise. The bar marked as "OCR" represents a model trained using the OCR noise distribution. Other bars correspond to models trained using synthetic noise distribution and different hyperparameters  $(\alpha, \eta_{train})$ .
(b) Stability training (synthetic noise:  $\eta_{test} = 1\%$
(d) Stability training objective(synthetic noise:  $\eta_{test} = 5\%$
(g) Data augmentation objective (synthetic noise:  $\eta_{test} = 20\%$ )
(f) Stability training objective (synthetic noise:  $\eta_{test} = 10\%$
(h) Stability training objective (synthetic noise:  $\eta_{test} = 20\%$ )