SlowGuess/ACL_2020_OCR huggingface dataset:
We evaluated the impact of our hyper-parameters on the sequence labeling accuracy using the English CoNLL 2003 data set. We trained multiple models with different amounts of noise  $\eta_{train}$  and different weighting factors  $\alpha$ . We chose the FLAIR + GloVe model as our baseline because it achieved the best results in the preliminary analysis (§4.2) and showed good performance, which enabled us to perform extensive experiments.
Figure 5 summarizes the results of the sensitivity experiment. The models trained with our auxiliary objectives mostly preserved or even improved accuracy on the original data compared to the baseline model  $(\alpha = 0)$ . Moreover, they significantly outperformed the baseline on data perturbed with natural noise. The best accuracy was achieved for  $\eta_{train}$  from 10 to  $30\%$ , which roughly corresponds to the label-preserving noise range. Similar to Heigold et al. (2018) and Cheng et al. (2019), we conclude that a non-zero noise level induced during training  $(\eta_{train} > 0)$  always yields improvements on noisy input data when compared with the models trained exclusively on clean data. The best choice of  $\alpha$  was in the range from 0.5 to 2.0.  $\alpha = 5.0$  exhibited lower performance on the original data. Moreover, the models trained on the real error distribution demonstrated at most slightly better performance, which indicates that the exact noise distribution does not necessarily have to be known at training time<sup>13</sup>.
To quantify improvements provided by our approach, we measured sequence labeling accuracy on the subsets of data with different levels of perturbation, i.e., we divided input tokens based on edit distance to their clean counterparts. Moreover, we partitioned the data by named entity class to assess the impact of noise on recognition of different entity types. For this experiment, we used both the test and the development parts of the English CoNLL 2003 data set and induced OCR errors with our noisig procedure.
Figure 6 presents the results for the baseline and the proposed methods. It can be seen that our ap
13 Nevertheless, the aspect of mimicking an empirical noise distribution requires more thoughtful analysis, and therefore we leave to future work.
<sup>8</sup>We used FLAIR v0.4.2.
<sup>9</sup>We experimented with a pre-processing step that used a spell checking module, but it did not provide any benefits and even decreased accuracy on the original data. Therefore we did not consider it a viable solution for this problem.
10We present data set statistics and sample outputs from our system in the appendix.
This choice was motivated by the availability of pretrained embedding models in the FLAIR framework.
<sup>12</sup>We did not replicate the exact results from the original papers because we did not use development sets for training, and our approach is feature-based, as we did not fine-tune embeddings on the target task.
Data set	Model	Train loss	Original data	OCR errors	Misspellings†	Misspellings‡
English CoNLL 2003	FLAIR + GloVe	L0	92.05	76.44±0.45	75.09±0.48	87.57±0.10
Laugm	92.56 (+0.51)	84.79±0.23 (+8.35)	83.57±0.43 (+8.48)	90.50±0.08 (+2.93)
Lstabil	91.99 (-0.06)	84.39±0.37 (+7.95)	82.43±0.23 (+7.34)	90.19±0.14 (+2.62)
BERT	L0	90.91	68.23±0.39	65.65±0.31	85.07±0.15
Laugm	90.84 (-0.07)	79.34±0.32 (+11.11)	75.44±0.28 (+9.79)	86.21±0.24 (+1.14)
Lstabil	90.95 (+0.04)	78.22±0.17 (+9.99)	73.46±0.34 (+7.81)	86.52±0.12 (+1.45)
ELMo	L0	92.16	72.90±0.50	70.99±0.17	88.59±0.19
Laugm	91.85 (-0.31)	84.09±0.18 (+11.19)	82.33±0.40 (+11.34)	89.50±0.16 (+0.91)
Lstabil	91.78 (-0.38)	83.86±0.11 (+10.96)	81.47±0.29 (+10.48)	89.49±0.15 (+0.90)
GloVe + Char	L0	90.26	71.15±0.51	70.91±0.39	87.14±0.07
Laugm	90.83 (+0.57)	81.09±0.47 (+9.94)	79.47±0.24 (+8.56)	88.82±0.06 (+1.68)
Lstabil	90.21 (-0.05)	80.33±0.29 (+9.18)	78.07±0.23 (+7.16)	88.47±0.13 (+1.33)
German CoNLL 2003	FLAIR + Wiki	L0	86.13	66.93±0.49	78.06±0.13	80.72±0.23
Laugm	86.46 (+0.33)	75.90±0.63 (+8.97)	83.23±0.14 (+5.17)	84.01±0.27 (+3.29)
Lstabil	86.33 (+0.20)	75.08±0.29 (+8.15)	82.60±0.21 (+4.54)	84.12±0.26 (+3.40)
Wiki + Char	L0	82.20	59.15±0.76	75.27±0.31	71.45±0.15
Laugm	82.62 (+0.42)	67.67±0.75 (+8.52)	78.48±0.24 (+3.21)	79.14±0.31 (+7.69)
Lstabil	82.18 (-0.02)	67.72±0.63 (+8.57)	77.59±0.12 (+2.32)	79.33±0.39 (+7.88)
Germ-Eval 2014	FLAIR + Wiki	L0	85.05	58.64±0.51	67.96±0.23	68.64±0.28
Laugm	84.84 (-0.21)	72.02±0.24 (+13.38)	78.59±0.11 (+10.63)	81.55±0.12 (+12.91)
Lstabil	84.43 (-0.62)	70.15±0.27 (+11.51)	75.67±0.16 (+7.71)	79.31±0.32 (+10.67)
Wiki + Char	L0	80.32	52.48±0.31	61.99±0.35	54.86±0.15
Laugm	80.68 (+0.36)	63.74±0.31 (+11.26)	70.83±0.09 (+8.84)	75.66±0.11 (+20.80)
Lstabil	80.00 (-0.32)	62.29±0.35 (+9.81)	68.23±0.23 (+6.24)	72.40±0.29 (+17.54)
Table 1: Evaluation results on the CoNLL 2003 and the GermEval 2014 test sets. We report results on the original data, as well as on its noisy copies with OCR errors and two types of misspellings released by Belinkov and Bisk (2018)† and Piktus et al. (2019)‡.  $\mathcal{L}_0$  is the standard training objective.  $\mathcal{L}_{\text{augm}}$  and  $\mathcal{L}_{\text{stabil}}$  are the data augmentation and the stability objectives, respectively. We report mean F1 scores with standard deviations from five experiments and mean differences against the standard objective (in parentheses).
(a) Data augmentation objective (original test data).
(c) Data augmentation objective (tested on OCR errors)
(b) Stability training objective (original test data).
(d) Stability training objective (tested on OCR errors)
Figure 5: Sensitivity analysis performed on the English CoNLL 2003 test set (§4.3). Each figure presents the results of models trained using one of our auxiliary training objectives on either original data or its variant perturbed with OCR errors. The bar marked as "OCR" represents a model trained using the OCR noise distribution. Other bars correspond to models trained using synthetic noise distribution and different hyper-parameters ( $\alpha$ ,  $\eta_{train}$ ).
proach achieved significant error reduction across all perturbation levels and all entity types. Moreover, by narrowing down the analysis to perturbed tokens, we discovered that the baseline model was particularly sensitive to noisy tokens from the LOC and the MISC categories. Our approach considerably reduced this negative effect. Furthermore, as the stability training worked slightly better on the LOC class and the data augmentation was more accurate on the ORG type, we argue that both methods could be combined to enhance overall sequence labeling accuracy further. Note that even if the particular token was not perturbed, its context could be noisy, which would explain the fact that our approach provided improvements even for tokens without perturbations.
Improving robustness has been receiving increasing attention in the NLP community. The most relevant research was conducted in the NMT domain.
Noise-additive data augmentation A natural strategy to improve robustness to noise is to augment the training data with samples perturbed using a similar noise model. Heigold et al. (2018) demonstrated that the noisy input substantially degrades the accuracy of models trained on clean data. They used word scrambling, as well as character flips and swaps as their noise model, and achieved the best results under matched training and test noise conditions. Belinkov and Bisk (2018) reported significant degradation in the performance of NMT systems on noisy input. They built a look-up table of possible lexical replacements from Wikipedia edit histories and used it as a natural source of the noise. Robustness to noise was only achieved by training with the same distribution—at the expense of performance degradation on other types of noise. In contrast, our method performed well on natural noise at test time by using a simplified synthetic noise model during training. Karpukhin et al. (2019) pointed out that existing NMT approaches are very sensitive to spelling mistakes and proposed to augment training samples with random character deletions, insertions, substitutions, and swaps. They showed improved robustness to natural noise, represented by frequent corrections in Wikipedia edit logs, without diminishing performance on the original data. However, not every word in the vocabulary has a corresponding misspelling. Therefore, even when noise is applied
(a) Divided by the edit distance value.
(b) Divided by the entity class (clean tokens).
(c) Divided by the entity class (perturbed tokens).
Figure 6: Error analysis results on the English CoNLL 2003 data set with OCR noise. We presented the results of the FLAIR + GloVe model trained with the standard and the proposed objectives. The data was divided into the subsets based on the edit distance of a token to its original counterpart and its named entity class. The latter group was further partitioned into the clean and the perturbed tokens. The error rate is the percentage of tokens with misrecognized entity class labels.
at the maximum rate, only a subset of tokens is perturbed (20-50%, depending on the language). In contrast, we used a confusion matrix, which is better suited to model statistical error distribution and can be applied to all tokens, not only those present in the corresponding look-up tables.
Robust representations Another method to improve robustness is to design a representation that is less sensitive to noisy input. Zheng et al. (2016) presented a general method to stabilize model predictions against small input distortions. Cheng et al. (2018) continued their work and developed the adversarial stability training method for NMT by adding a discriminator term to the objective func
tion. They combined data augmentation and stability objectives, while we evaluated both methods separately and provided evaluation results on natural noise distribution. Piktus et al. (2019) learned representation that embeds misspelled words close to their correct variants. Their Misspelling Oblivious Embeddings (MOE) model jointly optimizes two loss functions, each of which iterates over a separate data set (a corpus of text and a set of misspelling/correction pairs) during training. In contrast, our method does not depend on any additional resources and uses a simplified error distribution during training.
Adversarial learning Adversarial attacks seek to mislead the neural models by feeding them with adversarial examples (Szegedy et al., 2014). In a white-box attack scenario (Goodfellow et al., 2015; Ebrahimi et al., 2018) we assume that the attacker has access to the model parameters, in contrast to the black-box scenario (Alzantot et al., 2018; Gao et al., 2018), where the attacker can only sample model predictions on given examples. Adversarial training (Miyato et al., 2017; Yasunaga et al., 2018), on the other hand, aims to improve the robustness of the neural models by utilizing adversarial examples during training.
The impact of noisy input data In the context of ASR, Parada et al. (2011) observed that named entities are often OOV tokens, and therefore they cause more recognition errors. In the document processing field, Alex and Burns (2014) studied NER performed on several digitized historical text collections and showed that OCR errors have a significant impact on the accuracy of the downstream task. Namysl and Konya (2019) examined the efficiency of modern OCR engines and showed that although the OCR technology was more advanced than several years ago when many historical archives were digitized (Kim and Cassidy, 2015; Neudecker, 2016), the most widely used engines still had difficulties with non-standard or lower quality input.
Spelling- and post-OCR correction. A natural method of handling erroneous text is to correct it before feeding it to the downstream task. Most popular post-correction techniques include correction candidates ranking (Fivez et al., 2017; Flor et al., 2019), noisy channel modeling (Brill and Moore, 2000; Duan and Hsu, 2011), voting (Wemhoener et al., 2013), sequence to sequence models (Afli
et al., 2016; Schmaltz et al., 2017) and hybrid systems (Schulz and Kuhn, 2017).
In this paper, we have taken a different approach and attempted to make our models robust without relying on prior error correction, which, in case of OCR errors, is still far from being solved (Chiron et al., 2017; Rigaud et al., 2019).
In this paper, we investigated the difference in accuracy between sequence labeling performed on clean and noisy text (§2.3). We formulated the noisy sequence labeling problem (§2.2) and introduced a model that can be used to estimate the real noise distribution (§3.1). We developed the noise induction procedure that simulates the real noisy input (§3.2). We proposed two noise-aware training methods that boost sequence labeling accuracy on the perturbed text: (i) Our data augmentation approach uses a mixture of clean and noisy examples during training to make the model resistant to erroneous input (§3.3). (ii) Our stability training algorithm encourages output similarity for the original and the perturbed input, which helps the model to build a noise invariant latent representation (§3.4). Our experiments confirmed that NAT consistently improved efficiency of popular sequence labeling models on data perturbed with different error distributions, preserving accuracy on the original input (§4). Moreover, we avoided expensive re-training of embeddings on noisy data sources by employing existing text representations. We conclude that NAT makes existing models applicable beyond the idealized scenarios. It may support an automatic correction method that uses recognized entity types to narrow the list of feasible correction candidates. Another application is data anonymization (Mamede et al., 2016).
Future work will involve improvements in the proposed noise model to study the importance of fidelity to real-world error patterns. Moreover, we plan to evaluate NAT on other real noise distributions (e.g., from ASR) and other sequence labeling tasks to support our claims further.
We would like to thank the reviewers for the time they invested in evaluating our paper and for their insightful remarks and valuable suggestions.
Haithem Afli, Zhengwei Qiu, Andy Way, and Páraic Sheridan. 2016. Using SMT for OCR error correction of historical texts. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 962-966, Portož, Slovenia. European Language Resources Association (ELRA).
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the-art NLP. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54-59, Minneapolis, Minnesota. Association for Computational Linguistics.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638-1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Beatrice Alex and John Burns. 2014. Estimating and rating the quality of optically character recognised text. In Proceedings of the First International Conference on Digital Access to Textual Cultural Heritage, DATeCH '14, pages 97-102, New York, NY, USA. ACM.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018. Adversarial training for multi-context joint entity and relation extraction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2830-2836, Brussels, Belgium. Association for Computational Linguistics.
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings.
Darina Benikova, Chris Biemann, and Marc Reznicek. 2014. NoSta-d named entity annotation for German: Guidelines and dataset. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014), pages 2524-2531, Reykjavik, Iceland. European Languages Resources Association (ELRA).
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with
subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Eric Brill and Robert C. Moore. 2000. An improved error model for noisy channel spelling correction. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 286-293, Hong Kong. Association for Computational Linguistics.
Pin-Jung Chen, I-Hung Hsu, Yi Yao Huang, and Hung-Yi Lee. 2017. Mitigating the impact of speech recognition errors on chatbot using sequence-to-sequence model. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2017, Okinawa, Japan, December 16-20, 2017, pages 497-503.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust neural machine translation with doubly adversarial inputs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4324-4333, Florence, Italy. Association for Computational Linguistics.
Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018. Towards robust neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1756-1766, Melbourne, Australia. Association for Computational Linguistics.
G. Chiron, A. Doucet, M. Coustaty, and J. Moreux. 2017. ICDAR2017 competition on post-OCR text correction. In 2017 14th IAPR International Conference on Document Analysis and Recognition (IC-DAR), volume 01, pages 1423-1428.
Jason P.C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4:357-370.
Leon Derczynski, Alan Ritter, Sam Clark, and Kalina Bontcheva. 2013. Twitter part-of-speech tagging for all: Overcoming sparse and noisy data. In Proceedings of the International Conference on Recent Advances in Natural Language Processing RANLP 2013, pages 198-206, Hissar, Bulgaria. INCOMA Ltd. Shoumen, BULGARIA.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Huizhong Duan and Bo-June (Paul) Hsu. 2011. Online spelling correction for query completion. WWW 2011.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36, Melbourne, Australia. Association for Computational Linguistics.
Pieter Fivez, Simon Šuster, and Walter Daelemans. 2017. Unsupervised context-sensitive spelling correction of clinical free-text with word and character n-gram embeddings. In *BioNLP* 2017, pages 143–148, Vancouver, Canada., Association for Computational Linguistics.
Michael Flor, Michael Fried, and Alla Rozovskaya. 2019. A benchmark corpus of English misspellings and a minimally-supervised model for spelling correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 76-86, Florence, Italy. Association for Computational Linguistics.
J. Gao, J. Lanchantin, M. L. Soffa, and Y. Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pages 50-56.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, ICLR 2015.
Georg Heigold, Stalin Varanasi, Günter Neumann, and Josef van Genabith. 2018. How robust are character-based word embeddings in tagging and MT against wrod scrambling or randdm nouse? In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Papers), pages 68-80, Boston, MA. Association for Machine Translation in the Americas.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. Computing Research Repository, arXiv:1508.01991.
Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. 2019. Training on synthetic noise improves robustness to natural noise in machine translation. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 42-47, Hong Kong, China. Association for Computational Linguistics.
Sunghwan Mac Kim and Steve Cassidy. 2015. Finding names in trove: Named entity recognition for Australian historical newspapers. In Proceedings of the Australasian Language Technology Association Workshop 2015, pages 57-65, Parramatta, Australia.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS'12, pages 1097-1105, USA. Curran Associates Inc.