SlowGuess/ACL_2020_OCR huggingface dataset:
Marcin Namysl $^{1,2}$ , Sven Behnke $^{1,2}$ , and Joachim Kohler $^{1}$
<sup>1</sup> Fraunhofer IAIS, Sankt Augustin, Germany
$^{2}$  Autonomous Intelligent Systems, Computer Science Institute VI, University of Bonn, Germany {Marcin.Namysl, Sven.Behnke, Joachim Koehler}@iais.fraunhofer.de
Sequence labeling systems should perform reliably not only under ideal conditions but also with corrupted inputs—as these systems often process user-generated text or follow an error-prone upstream component. To this end, we formulate the noisy sequence labeling problem, where the input may undergo an unknown noisng process and propose two Noise-Aware Training (NAT) objectives that improve robustness of sequence labeling performed on perturbed input: Our data augmentation method trains a neural model using a mixture of clean and noisy samples, whereas our stability training algorithm encourages the model to create a noise-invariant latent representation. We employ a vanilla noise model at training time. For evaluation, we use both the original data and its variants perturbed with real OCR errors and misspellings. Extensive experiments on English and German named entity recognition benchmarks confirmed that NAT consistently improved robustness of popular sequence labeling models, preserving accuracy on the original input. We make our code and data publicly available for the research community.
Sequence labeling systems are generally trained on clean text, although in real-world scenarios, they often follow an error-prone upstream component, such as Optical Character Recognition (OCR; Neudecker, 2016) or Automatic Speech Recognition (ASR; Parada et al., 2011). Sequence labeling is also often performed on user-generated text, which may contain spelling mistakes or typos (Derczynski et al., 2013). Errors introduced in an upstream task are propagated downstream, diminishing the performance of the end-to-end system (Alex and Burns, 2014). While humans can easily cope with typos, misspellings, and the complete omission of letters when reading (Rawlinson,
Figure 1: An example of a labeling error on a slightly perturbed sentence. Our noise-aware methods correctly predicted the location (LOC) label for the first word, as opposed to the standard approach, which misclassified it as an organization (ORG). We complement the example with a high-level idea of our noise-aware training, where the original sentence and its noisy variant are passed together through the system. The final loss is computed based on both sets of features, which improves robustness to the input perturbations.
2007), most Natural Language Processing (NLP) systems fail when processing corrupted or noisy text (Belinkov and Bisk, 2018). Although this problem is not new to NLP, only a few works addressed it explicitly (Piktus et al., 2019; Karpukhin et al., 2019). Other methods must rely on the noise that occurs naturally in the training data.
In this work, we are concerned with the performance difference of sequence labeling performed on clean and noisy input. Is it possible to narrow the gap between these two domains and design an approach that is transferable to different noise distributions at test time? Inspired by recent research in computer vision (Zheng et al., 2016), Neural Machine Translation (NMT; Cheng et al., 2018), and ASR (Sperber et al., 2017), we propose two Noise-Aware Training (NAT) objectives that improve the accuracy of sequence labeling performed on noisy input without reducing efficiency on the original data. Figure 1 illustrates the problem and our approach.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1501-1517 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
- We formulate a noisy sequence labeling problem, where the input undergoes an unknown noisiging process (§2.2), and we introduce a model to estimate the real error distribution (§3.1). Moreover, we simulate real noisy input with a novel noise induction procedure (§3.2).
- We propose a data augmentation algorithm (§3.3) that directly induces noise in the input data to perform training of the neural model using a mixture of noisy and clean samples.
- We implement a stability training method (Zheng et al., 2016), adapted to the sequence labeling scenario, which explicitly addresses the noisy input data problem by encouraging the model to produce a noise-invariant latent representation (§3.4).
- We evaluate our methods on real OCR errors and misspellings against state-of-the-art baseline models (Peters et al., 2018; Akbik et al., 2018; Devlin et al., 2019) and demonstrate the effectiveness of our approach ( $\S 4$ ).
- To support future research in this area and to make our experiments reproducible, we make our code and data publicly available<sup>1</sup>.
2.1 Neural Sequence Labeling
Figure 2 presents a typical architecture for the neural sequence labeling problem. We will refer to the sequence labeling system as  $F(x; \theta)$ , abbreviated as  $F(x)^2$ , where  $x = (x_1, \ldots, x_N)$  is a tokenized input sentence of length  $N$ , and  $\theta$  represents all learnable parameters of the system.  $F(x)$  takes  $x$  as input and outputs the probability distribution over the class labels  $y(x)$  as well as the final sequence of labels  $\hat{y} = (\hat{y}_1, \dots, \hat{y}_N)$ .
Either a softmax model (Chiu and Nichols, 2016) or a Conditional Random Field (CRF; Lample et al., 2016) can be used to model the output distribution over the class labels  $y(x)$  from the logits  $l(x)$ , i.e., non-normalized predictions, and to output the final sequence of labels  $\hat{y}$ . As a labeled entity can span several consecutive tokens within a sentence,
special tagging schemes are often employed for decoding, e.g., BIOES, where the Beginning, Inside, Outside, End-of-entity and Single-tag-entity subtags are also distinguished (Ratinov and Roth, 2009). This method introduces strong dependencies between subsequent labels, which are modeled explicitly by a CRF (Lafferty et al., 2001) that produces the most likely sequence of labels.
Figure 2: Neural sequence labeling architecture. In the standard scenario (§2.1), the original sentence  $x$  is fed as input to the sequence labeling system  $F(x)$ . Token embeddings  $e(x)$  are retrieved from the corresponding look-up table and fed to the sequence labeling model  $f(x)$ , which outputs latent feature vectors  $h(x)$ . The latent vectors are then projected to the class logits  $l(x)$ , which are used as input to the decoding model (softmax or CRF) that outputs the distribution over the class labels  $y(x)$  and the final sequence of labels  $\hat{y}$ . In a real-world scenario (§2.2), the input sentence undergoes an unknown noisig process  $\Gamma$ , and the perturbed sentence  $\tilde{x}$  is fed to  $F(x)$ .
2.2 Noisy Neural Sequence Labeling
Similar to human readers, sequence labeling should perform reliably both in ideal and sub-optimal conditions. Unfortunately, this is rarely the case. User-generated text is a rich source of informal language containing misspellings, typos, or scrambled words (Derczynski et al., 2013). Noise can also be introduced in an upstream task, like OCR (Alex and Burns, 2014) or ASR (Chen et al., 2017), causing the errors to be propagated downstream.
$^{1}$ NAT repository on GitHub: https://github.com/mnamysl/nat-acl2020  
 $^{2}$ We drop the  $\theta$  parameter for brevity in the remaining of the paper. Nonetheless, we still assume that all components of  $F(x; \theta)$  and all expressions derived from it also depend on  $\theta$ .
To include the noise present on the source side of  $F(x)$ , we can modify its definition accordingly (Figure 2). Let us assume that the input sentence  $x$  is additionally subjected to some unknown noising process  $\Gamma = P(\tilde{x}_i|x_i)$ , where  $x_{i}$  is the original  $i$ -th token, and  $\tilde{x}_i$  is its distorted equivalent. Let  $\mathcal{V}$  be the vocabulary of tokens and  $\tilde{\nu}$  be a set of all finite character sequences over an alphabet  $\Sigma$ .  $\Gamma$  is known as the noisy channel matrix (Brill and Moore, 2000) and can be constructed by estimating the probability  $P(\tilde{x}_i|x_i)$  of each distorted token  $\tilde{x}_i$  given the intended token  $x_{i}$  for every  $x_{i}\in \mathcal{V}$  and  $\tilde{x}_i\in \tilde{\nu}$ .
2.3 Named Entity Recognition
We study the effectiveness of state-of-the-art Named Entity Recognition (NER) systems in handling imperfect input data. NER can be considered as a special case of the sequence labeling problem, where the goal is to locate all named entity mentions in unstructured text and to classify them into pre-defined categories, e.g., person names, organizations, and locations (Tjong Kim Sang and De Meulder, 2003). NER systems are often trained on the clean text. Consequently, they exhibit degraded performance in real-world scenarios where the transcriptions are produced by the previous upstream component, such as OCR or ASR (§2.2), which results in a detrimental mismatch between the training and the test conditions. Our goal is to improve the robustness of sequence labeling performed on data from noisy sources, without deteriorating performance on the original data. We assume that the source sequence of tokens  $x$  may contain errors. However, the noisng process is generally label-preserving, i.e., the level of noise is not significant enough to affect the corresponding labels<sup>3</sup>. It follows that the noisy token  $\tilde{x}_i$  inherits the ground-truth label  $y_i$  from the underlying original token  $x_i$ .
3 Noise-Aware Training
3.1 Noise Model
To model the noise, we use the character-level noisy channel matrix  $\Gamma$ , which we will refer to as the character confusion matrix (§2.2).
Natural noise We can estimate the natural error distribution by calculating the alignments between the pairs  $(\tilde{x},x)\in \mathcal{P}$  of noisy and clean sentences
using the Levenshtein distance metric (Levenshtein, 1966), where  $\mathcal{P}$  is a corpus of paired noisy and manually corrected sentences (§2.2). The allowed edit operations include insertions, deletions, and substitutions of characters. We can model insertions and deletions by introducing an additional symbol  $\varepsilon$  into the character confusion matrix. The probability of insertion and deletion can then be formulated as  $P_{ins}(c|\varepsilon)$  and  $P_{del}(\varepsilon |c)$ , where  $c$  is a character to be inserted or deleted, respectively.
Synthetic noise  $\mathcal{P}$  is usually laborious to obtain. Moreover, the exact modeling of noise might be impractical, and it is often difficult to accurately estimate the exact noise distribution to be encountered at test time. Such distributions may depend on, e.g., the OCR engine used to digitize the documents. Therefore, we keep the estimated natural error distribution for evaluation and use a simplified synthetic error model for training. We assume that all types of edit operations are equally likely:
$$
\sum_ {\tilde {c} \in \Sigma \backslash \{\varepsilon \}} P _ {i n s} (\tilde {c} | \varepsilon) = P _ {d e l} (\varepsilon | c) = \sum_ {\tilde {c} \in \Sigma \backslash \{c, \varepsilon \}} P _ {s u b s t} (\tilde {c} | c),
$$
where  $c$  and  $\tilde{c}$  are the original and the perturbed characters, respectively. Moreover,  $P_{ins}$  and  $P_{subst}$  are uniform over the set of allowed insertion and substitution candidates, respectively. We use the hyper-parameter  $\eta$  to control the amount of noise to be induced with this method<sup>4</sup>.
3.2 Noise Induction
Ideally, we would use the noisy sentences annotated with named entity labels for training our sequence labeling models. Unfortunately, such data is scarce. On the other hand, labeled clean text corpora are widely available (Tjong Kim Sang and De Meulder, 2003; Benikova et al., 2014). Hence, we propose to use the standard NER corpora and to induce noise into the input tokens during training synthetically.
In contrast to the image domain, which is continuous, the text domain is discrete, and we cannot directly apply continuous perturbations for written language. Although some works applied distortions at the level of embeddings (Miyato et al., 2017; Yasunaga et al., 2018; Bekoulis et al., 2018), we do not have a good intuition how it changes the meaning of the underlying textual input. Instead, we apply our noise induction procedure to generate distorted copies of the input. For every
<sup>4</sup>We describe the details of our vanilla error model along with the examples of confusion matrices in the appendix.
3Moreover, a human reader should be able to infer the correct label  $y_{i}$  from the token  $\tilde{x}_i$  and its context. We assume that this corresponds to a character error rate of  $\leq 20\%$
input sentence  $x$ , we independently perturb each token  $x_{i} = (c_{1}, \ldots, c_{K})$ , where  $K$  is the length of  $x_{i}$ , with the following procedure (Figure 3):
(1) We insert the  $\varepsilon$  symbol before the first and after every character of  $x_{i}$  to get an extended token  $x_{i}^{\prime} = (\varepsilon, c_{1}, \varepsilon, \ldots, \varepsilon, c_{K}, \varepsilon)$ .
(2) For every character  $c_k^\prime$  of  $x_{i}^{\prime}$ , we sample the replacement character  $\tilde{c}_k^\prime$  from the corresponding probability distribution  $P(\tilde{c}_k^\prime |c_k^\prime)$ , which can be obtained by taking a row of the character confusion matrix that corresponds to  $c_k^\prime$ . As a result, we get a noisy version of the extended input token  $\tilde{x}_i^\prime$ .
(3) We remove all  $\varepsilon$  symbols from  $\tilde{x}_i^{\prime}$  and collapse the remaining characters to obtain a noisy token  $\tilde{x}_i$ .
Figure 3: Illustration of our noise induction procedure. Three examples correspond to insertion, deletion, and substitution errors.  $x_{i}$ ,  $x_{i}^{\prime}$ ,  $\tilde{x}_i^\prime$ , and  $\tilde{x}_i$  are the original, extended, extended noisy, and noisy tokens, respectively.
3.3 Data Augmentation Method
We can improve robustness to noise at test time by introducing various forms of artificial noise during training. We distinct regularization methods like dropout (Srivastava et al., 2014) and task-specific data augmentation that transforms the data to resemble noisy input. The latter technique was successfully applied in other domains, including computer vision (Krizhevsky et al., 2012) and speech recognition (Sperber et al., 2017).
During training, we artificially induce noise into the original sentences using the algorithm described in §3.2 and train our models using a mixture of clean and noisy sentences. Let  $\mathcal{L}_0(x,y;\theta)$  be the standard training objective for the sequence labeling problem, where  $x$  is the input sentence,  $y$  is the corresponding ground-truth sequence of labels, and  $\theta$  represents the parameters of  $F(x)$ . We define our composite loss function as follows:
$$
\mathcal {L} _ {a u g m} (x, \tilde {x}, y; \theta) = \mathcal {L} _ {0} (x, y; \theta) + \alpha \mathcal {L} _ {0} (\tilde {x}, y; \theta),
$$
where  $\tilde{x}$  is the perturbed sentence, and  $\alpha$  is a weight of the noisy loss component.  $\mathcal{L}_{\text{augm}}$  is a weighted sum of standard losses calculated using clean and noisy sentences. Intuitively, the model that would optimize  $\mathcal{L}_{\text{augm}}$  should be more robust to imperfect input data, retaining the ability to perform well on clean input. Figure 4a presents a schematic visualization of our data augmentation approach.
3.4 Stability Training Method
Zheng et al. (2016) pointed out the output instability issues of deep neural networks. They proposed a training method to stabilize deep networks against small input perturbations and applied it to the tasks of near-duplicate image detection, similar-image ranking, and image classification. Inspired by their idea, we adapt the stability training method to the natural language scenario.
Our goal is to stabilize the outputs  $y(x)$  of a sequence labeling system against small input perturbations, which can be thought of as flattening  $y(x)$  in a close neighborhood of any input sentence  $x$ . When a perturbed copy  $\tilde{x}$  is close to  $x$ , then  $y(\tilde{x})$  should also be close to  $y(x)$ . Given the standard training objective  $\mathcal{L}_0(x,y;\theta)$ , the original input sentence  $x$ , its perturbed copy  $\tilde{x}$  and the sequence of ground-truth labels  $y$ , we can define the stability training objective  $\mathcal{L}_{\text{stabil}}$  as follows:
$$
\mathcal {L} _ {\text {s t a b i l}} (x, \tilde {x}, y; \theta) = \mathcal {L} _ {0} (x, y; \theta) + \alpha \mathcal {L} _ {\text {s i m}} (x, \tilde {x}; \theta),
$$
$$
\mathcal {L} _ {s i m} (x, \tilde {x}; \theta) = \mathcal {D} \left(y (x), y (\tilde {x})\right),
$$
where  $\mathcal{L}_{sim}$  encourages the similarity of the model outputs for both  $x$  and  $\tilde{x}$ ,  $\mathcal{D}$  is a task-specific feature distance measure, and  $\alpha$  balances the strength of the similarity objective. Let  $R(x)$  and  $Q(\tilde{x})$  be the discrete probability distributions obtained by calculating the softmax function over the logits  $l(x)$  for  $x$  and  $\tilde{x}$ , respectively:
$$
R (x) = P (y \mid x) = \operatorname {s o f t m a x} (l (x)),
$$
$$
Q (\tilde {x}) = P (y | \tilde {x}) = \operatorname {s o f t m a x} \left(l (\tilde {x})\right).
$$
We model  $\mathcal{D}$  as Kullback-Leibler divergence  $(\mathcal{D}_{KL})$ , which measures the correspondence between the likelihood of the original and the perturbed input:
$$
\mathcal {L} _ {s i m} (x, \tilde {x}; \theta) = \sum_ {i} \mathcal {D} _ {K L} \left(R \left(x _ {i}\right) \| Q \left(\tilde {x} _ {i}\right)\right),
$$
$$
\mathcal {D} _ {K L} \big (R (x) \| Q (\tilde {x}) \big) = \sum_ {j} P (y _ {j} | x) \log \frac {P (y _ {j} | x)}{P (y _ {j} | \tilde {x})},
$$
(a) Data augmentation training objective  $\mathcal{L}_{\text{augm}}$
(b) Stability training objective  $\mathcal{L}_{\text{stabil}}$
Figure 4: Schema of our auxiliary training objectives.  $x$ ,  $\tilde{x}$  are the original and the perturbed inputs, respectively, that are fed to the sequence labeling system  $F(x)$ .  $\Gamma$  represents a noisig process.  $y(x)$  and  $y(\tilde{x})$  are the output distributions over the entity classes for  $x$  and  $\tilde{x}$ , respectively.  $\mathcal{L}_0$  is the standard training objective.  $\mathcal{L}_{\text{augm}}$  combines  $\mathcal{L}_0$  computed on both outputs from  $F(x)$ .  $\mathcal{L}_{\text{stabil}}$  fuses  $\mathcal{L}_0$  calculated on the original input with the similarity objective  $\mathcal{L}_{\text{sim}}$ .
where  $i, j$  are the token, and the class label indices, respectively. Figure 4b summarizes the main idea of our stability training method.
A critical difference between the data augmentation and the stability training method is that the latter does not use noisy samples for the original task, but only for the stability objective<sup>5</sup>. Furthermore, both methods need perturbed copies of the input samples, which results in longer training time but could be ameliorated by fine-tuning the existing model for a few epochs<sup>6</sup>.
Model architecture We used a BiLSTM-CRF architecture (Huang et al., 2015) with a single Bidirectional Long-Short Term Memory (BiLSTM) layer and 256 hidden units in both directions for  $f(x)$  in all experiments. We considered four different text representations  $e(x)$ , which were used to achieve state-of-the-art results on the studied data set and should also be able to handle misspelled text and out-of-vocabulary (OOV) tokens:
- FLAIR (Akbik et al., 2018) learns a Bidi
rectional Language Model (BiLM) using an LSTM network to represent any sequence of characters. We used settings recommended by the authors and combined FLAIR with GloVe (Pennington et al., 2014;  $FLAIR + GloVe$ ) for English and Wikipedia FastText embeddings (Bojanowski et al., 2017;  $FLAIR + Wiki$ ) for German.
- BERT (Devlin et al., 2019) employs a Transformer encoder to learn a BiLM from large unlabeled text corpora and sub-word units to represent textual tokens. We use the  $\mathrm{BERT}_{\mathrm{BASE}}$  model in our experiments.
- ELMo (Peters et al., 2018) utilizes a linear combination of hidden state vectors derived from a BiLSTM word language model trained on a large text corpus.
- Glove/Wiki + Char is a combination of pretrained word embeddings (GloVe for English and Wikipedia FastText for German) and randomly initialized character embeddings (Lample et al., 2016).
Training We trained the sequence labeling model  $f(x)$  and the final CRF decoding layer on top of the pre-trained embedding vectors  $e(x)$ , which were fixed during training, except for the character embeddings (Figure 2). We used a mixture of the original data and its perturbed copies generated from the synthetic noise distribution (§3.1) with our noise induction procedure (§3.2). We kept most of the hyper-parameters consistent with Akbik et al. (2018)<sup>7</sup>. We trained our models for at most 100 epochs and used early stopping based on the development set performance, measured as an average F1 score of clean and noisy samples. Furthermore, we used the development sets of each benchmark data set for validation only and not for training.
Performance measures We measured the entity-level micro average F1 score on the test set to compare the results of different models. We evaluated on both the original and the perturbed data using various natural error distributions. We induced OCR errors based on the character confusion matrix  $\Gamma$  (§3.2) that was gathered on a large document corpus (Namysl and Konya, 2019) using the Tesseract OCR engine (Smith, 2007). Moreover, we employed two sets of misspellings released by
Both objectives could be combined and used together. However, our goal is to study their impact on robustness separately, and we leave further exploration to future work.
<sup>6</sup>We did not explore this setting in this paper, leaving such optimization to future work.
<sup>7</sup>We list the detailed hyper-parameters in the appendix.
Belinkov and Bisk (2018) and Piktus et al. (2019). Following the authors, we replaced every original token with the corresponding misspelled variant, sampling uniformly among available replacement candidates. We present the estimated error rates of text that is produced with these noise induction procedures in Table 5 in the appendix. As the evaluation with noisy data leads to some variance in the final scores, we repeated all experiments five times and reported mean and standard deviation.
Implementation We implemented our models using the FLAIR framework (Akbik et al., 2019). We extended their sequence labeling model by integrating our auxiliary training objectives (§3.3, §3.4). Nonetheless, our approach is universal and can be implemented in any other sequence labeling framework.
4.2 Sequence Labeling on Noisy Data
To validate our approach, we trained the baseline models with and without our auxiliary loss objectives  $(\S 3.3, \S 3.4)^9$ . We used the CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003) and the GermEval 2014 (Benikova et al., 2014) data sets in this setup<sup>10</sup>. The baselines utilized GloVe vectors coupled with FLAIR and character embeddings (FLAIR + GloVe, GloVe + Char), BERT, and ELMo embeddings for English. For German, we employed Wikipedia FastText vectors paired with FLAIR and character embeddings (FLAIR + Wiki, Wiki + Char)<sup>11</sup>. We used a label-preserving training setup ( $\alpha = 1.0$ ,  $\eta_{train} = 10\%$ ).
Table 1 presents the results of this experiment<sup>12</sup>. We found that our auxiliary training objectives boosted accuracy on noisy input data for all baseline models and both languages. At the same time, they preserved accuracy for the original input. The data augmentation objective seemed to perform slightly better than the stability objective. However, the chosen hyper-parameter values were rather ar
bitrary, as our goal was to prove the utility and the flexibility of both objectives.