SlowGuess/ACL_2020_OCR huggingface dataset:
Peter Makarov
SimonClematide
Institute of Computational Linguistics
University of Zurich, Switzerland
makarov@cl.uzh.ch
simon.clematide@cl.uzh.ch
Historical text normalization, the task of mapping historical word forms to their modern counterparts, has recently attracted a lot of interest (Bollmann, 2019; Tang et al., 2018; Lusetti et al., 2018; Bollmann et al., 2018; Robertson and Goldwater, 2018; Bollmann et al., 2017; Korchagina, 2017). Yet, virtually all approaches suffer from the two limitations: 1) They consider a fully supervised setup, often with impractically large manually normalized datasets; 2) Normalization happens on words in isolation. By utilizing a simple generative normalization model and obtaining powerful contextualization from the target-side language model, we train accurate models with unlabeled historical data. In realistic training scenarios, our approach often leads to reduction in manually normalized data at the same accuracy levels.
Text normalization is the task of mapping texts written in some non-standard variety of language  $L$  (a dialect or an earlier diachronic form) to some standardized form, typically the official modern standard variety of  $L$  (Table 1). Examples include the normalization of informal English-language tweets (Han and Baldwin, 2011); quasiphonetic transcriptions of dialectal Swiss German (Samardžić et al., 2015); and historical documents such as religious texts in 15th-century Icelandic (Bollmann et al., 2011; Pettersson et al., 2013b; Ljubesić et al., 2016, inter alia).
Text can to a large extent be normalized by replacing non-standard words with their standard counterparts. Because of this often-made assumption, this task is also known as "lexical" or "spelling normalization" (Han and Baldwin, 2011; Tang et al., 2018).
There has been a lot of interest in historical and dialectal text normalization over the past years. Earlier works attempt type-level normalization by way of search for standardized words (Pettersson et al., 2013a; Bollmann, 2012). More recently, the dominant approach casts the problem as probabilistic type-level character transduction. Most commonly, a fully-supervised machine translation system transduces words in isolation (Bollmann, 2019). The use of context is limited to employing a target-side language model for an improved, contextualized decoding (Ljubesic et al., 2016; Etxeberria et al., 2016; Jurish, 2010).
In this paper, we develop simple approaches to semi-supervised contextualized text normalization. On the example of historical text normalization, we show that one can reduce the amount of supervision by leveraging unlabeled historical text and utilizing context at training. Our methods build on familiar techniques for semi-supervised learning such as generative modeling and expectation-maximization and unify previous work (search, noisy channel, contextualized decoding, neural character-level transduction) in a simple setup.
We experimentally validate the strength of our models on a suite of historical datasets. In addition to the token-level supervision scenario, we show benefits of a more economic supervision by a word-type normalization dictionary.
2 Historical text normalization
Most normalization approaches attempt to learn a function from historical to modern word types without taking context into consideration. This is based on the observation that morpho-syntactic differences between the non-standard and standard varieties (e.g. in word order, grammatical case distinctions) are negligible and normalization ambi
7284
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7284-7295 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
wermuttfafft	in	die	ohren	getropfft	tödtet	die	würme	darinnen
wermutsaft	in	die	ohren	getropft	tötet	die	würmer	darin
wormwood juice	in	the	ears	dripped	kills	the	worms	inside
Table 1: Historical text normalization. An excerpt from the RIDGES corpus of 15th-17th century German scientific writing (Odebrecht et al., 2017). Top=Early New High German, middle=Modern Standard German.
guity is often not very high. Some earlier works cast text normalization as search over standardized word forms (Pettersson et al., 2013a; Bollmann, 2012). Hand-crafted rules or a string-distance metric (Levenshtein, 1966) with parameters estimated from the labeled data are used to retrieve best matching standard candidates.
Another line of work follows a principled probabilistic solution: a noisy channel model (Shannon, 1948), which consists of a channel  $p(\mathbf{x} \mid \mathbf{y})$  and a language model  $p(\mathbf{y})$  (Jurish, 2010; Pettersson et al., 2013b; Samardžić et al., 2015; Etxeberria et al., 2016; Ljubesić et al., 2016; Scherrer and Ljubesić, 2016). The channel model operates at the character level and takes the form of either a character alignment model (Brown et al., 1993) or a weighted finite-state transducer (WFST, Mohri, 1997). Channel model parameters are estimated from a manually normalized corpus. The language model is often trained on external target-side data. Some works perform normalization of words in context. Jurish (2010) and Etxeberria et al. (2016) decode sequences of historical word tokens by combining a character-level channel with a word-level language model  $p(\mathbf{y}_{1:m})$ . Scherrer and Ljubesić (2016) learn a character alignment model directly over untokenized segments of historical texts.
Numerous neural approaches to text normalization (Tang et al., 2018; Lusetti et al., 2018; Bollmann et al., 2018; Robertson and Goldwater, 2018; Bollmann et al., 2017; Korchagina, 2017) learn a discriminative model  $p(\mathbf{y} \mid \mathbf{x})$  parameterized with some generic encoder-decoder neural network—that performs the traditional character-level transduction of isolated words. The models are trained in a supervised fashion on a lot of manually labeled data. For example, Tang et al. (2018) train on tens of thousands of labeled pairs, including for varieties that share more than  $75\%$  of their vocabularies. Except Lusetti et al. (2018), who use a target-side language model to rerank base model hypotheses in context, no other approach in this group uses context in any way.
3 The role of context
If non-standard language exhibits normalization ambiguity, one would expect contextualization to reduce it. For example, historical German "defz" in the RIDGES corpus (Odebrecht et al., 2017) normalizes to three modern word types: "das", "des" (various forms of the definite article), and "dessen" (relative pronoun whose). Knowing the context (e.g. whether the historical word occurs clause-initially or before a neuter noun) would help normalize "defz" correctly. As suggested by Ljubesic et al. (2016), the accuracy of the oracle that normalizes words in isolation by always selecting their most frequent normalization upper bounds the accuracy of non-contextual systems.
Many historical normalization corpora do not have high normalization ambiguity (Table 3). The upper bound on accuracy for non-contextual normalization is 97.0 on average  $(\pm 0.02)$  and is above 92.4 for every historical language that we study here, indicating that lexical normalization is a very reasonable strategy.
Even if context may sometimes not be necessary for adequately solving the task in a fully supervised manner, we would expect contextualization to lead to more accurate unsupervised and semi-supervised generative models.
4.1 Contextualized generative model
We start off with a generative model in the form of a noisy channel over sequences of words (Eq. 1). The channel model factorizes over non-standard words, and a non-standard word  $\mathbf{x}_i$  depends only on the corresponding standardized word  $\mathbf{y}_i$ . The simple structure of our model follows from the lexical normalization assumption.
$$
p _ {\theta} \left(\mathbf {x} _ {1: m}, \mathbf {y} _ {1: m}\right) \approx \overbrace {p (\mathbf {y} _ {1 : m})} ^ {\text {h i s t o r i c a l}} \overbrace {\prod_ {i = 1} ^ {m} p _ {\theta} \left(\mathbf {x} _ {i} \mid \mathbf {y} _ {i}\right)} ^ {\text {l a n g u a g e m o d e l}} \tag {1}
$$
Compared to a discriminative model, which would directly capture the mapping from non
standard word sequences  $\mathbf{x}_{1:m}$  to standardized  $\mathbf{y}_{1:m}$  without having to account for how non-standard data arise, this model offers some important advantages. First, it can be trained by maximizing marginal likelihood  $p(\mathbf{x}_{1:m})$ , which leads to semi-supervised learning. Second, we can use a language model estimated from arbitrary external text.
The only model parameters are the parameters  $\theta$  of the channel model  $p_{\theta}(\mathbf{x}_i\mid \mathbf{y}_i)$ . The parameters of the language model  $p(\mathbf{y}_{1:m})$  are held fixed.
4.2 Neural channel
The channel model  $p(\mathbf{x}_i\mid \mathbf{y}_i)$  stochastically maps standardized words to non-standard words. Any type-level normalization model from  $\S 2$  can be applied here (in the reverse direction from the normalization task).
For our experiments, we use the neural transducer of Makarov and Clematide (2018) as it has been shown to perform strongly on morphological character transduction tasks. Parameterized with a recurrent encoder and decoder, the model defines a conditional distribution over edits  $p_{\theta}(\mathbf{x}, \mathbf{a} \mid \mathbf{y}) = \prod_{j=1}^{|\mathbf{a}|} p_{\theta}(a_j \mid a_{1:j-1}, \mathbf{y})$ , where  $\mathbf{y} = y_1, \ldots, y_{|\mathbf{y}|}$  is a standardized word as a character sequence and  $\mathbf{a} = a_1 \ldots a_{|\mathbf{a}|}$  an edit action sequence. Using this model as a channel requires computing the marginal likelihood  $p_{\theta}(\mathbf{x} \mid \mathbf{y})$ , which is intractable due to the recurrent decoder. We approximate  $p_{\theta}(\mathbf{x} \mid \mathbf{y})$  by  $p_{\theta}(\mathbf{x}, \mathbf{a}^* \mid \mathbf{y})$ , where  $\mathbf{a}^*$  is a minimum cost edit action sequence from  $\mathbf{y}$  to  $\mathbf{x}$ . This works well in practice as the network produces a highly peaked distribution with most probability mass placed on minimum cost edit action sequences.
4.3 Language model
We consider two language model factorizations, which lead to different learning approaches.
Neural HMM. If the language model is an  $n$ -gram language model
$$
p \left(\mathbf {y} _ {1: m}\right) \approx \prod_ {i = 1} ^ {m + 1} p \left(\mathbf {y} _ {i} \mid \mathbf {y} _ {i - n + 1: i - 1}\right), \tag {2}
$$
the overall generative model has the form of an  $n$ -gram Hidden Markov Model (HMM) with transition probabilities given by the language model and emission probabilities by the channel. HMM has been proposed for this problem before but with
different parameterizations (Jurish, 2010; Etxeberria et al., 2016). For simplicity, we use count-based language models in the experiments. Full neural parametarization can be achieved with  $n$ -gram feedforward neural language models (Bengio et al., 2003).
RNN LM-based model. Our second language model is a word-level recurrent neural language model (RRN-LM, Mikolov, 2012). It does not make any independence assumptions, which increases expressive power yet precludes exact inference in the generative model.
4.4 Expectation-maximization
Let  $U$  be a set of unlabeled non-standard sentences,  $V_{\mathbf{x}}$  the set of non-standard word types in  $U$ , and  $V_{\mathbf{y}}$  the vocabulary of the standardized variety. In the unsupervised case, we train by maximizing the marginal likelihood of  $U$  with respect to the channel parameters  $\theta$ :
$$
L _ {U} (U, \theta) = \sum_ {\mathbf {x} _ {1: m} \in U} \log \sum_ {\mathbf {y} _ {1: m} \in V _ {\mathbf {y}} ^ {m}} p _ {\theta} (\mathbf {x} _ {1: m}, \mathbf {y} _ {1: m}) \tag {3}
$$
For an  $n$ -gram neural HMM, this can be solved using generalized expectation-maximization (GEM, Neal and Hinton, 1998; Berg-Kirkpatrick et al., 2010). We compute the E-step with the forward-backward algorithm. In the M-step, given the posterior  $p(\mathbf{y} \mid \mathbf{x})$  for each non-standard word type  $\mathbf{x}$ , we maximize the following objective with respect to  $\theta$  with a variant of stochastic gradient ascent:
$$
L _ {M} (U, \theta) = \sum_ {\mathbf {x} \in V _ {\mathbf {x}}} \sum_ {\mathbf {y} \in V _ {\mathbf {y}}} p (\mathbf {y} \mid \mathbf {x}) \log p _ {\theta} (\mathbf {x} \mid \mathbf {y}) \tag {4}
$$
GEM provably increases the marginal likelihood.
We train the RNN LM-based model with hard expectation-maximization (hard EM, Samdani et al., 2012). This is a simple alternative to approximate inference. Hard EM does not guarantee to increase the marginal likelihood, but often works in practice (Spitkovsky et al., 2010). The difference from GEM is the E-step. To compute it, we decode  $U$  with beam search. Let  $B = \bigcup_{\mathbf{x}_{1:m} \in U} \{\mathbf{y}_{1:m} \in V_{\mathbf{y}}^{m} \mid \mathbf{y}_{1:m}$  is in the beam for  $\mathbf{x}_{1:m}\}$ . We set the posterior  $p(\mathbf{Y} = \mathbf{y} \mid \mathbf{X} = \mathbf{x})$  to be proportional to the sum of the probabilities of sentence-wise normalizations from  $B$  where  $\mathbf{x}$  gets normalized as  $\mathbf{y}$ .
7286
Semi-supervised training. We linearly combine the maximum likelihood (MLE) of the set  $S = \{(\mathbf{x},\mathbf{y})_i\}_{i = 1}^n$  of labeled normalization pairs with the marginal likelihood of  $U$  (Eq. 3):
$$
L (S, U, \theta) = \sum_ {(\mathbf {x}, \mathbf {y}) \in S} \log p _ {\theta} (\mathbf {x} \mid \mathbf {y}) + \lambda L _ {U} (U, \theta) \tag {5}
$$
$\lambda \geq 0$  controls how much information from  $U$  flows into the parameter update. The difference from the unsupervised case is that the M-step computes Eq. 4 scaled with  $\lambda$  and the MLE of  $S$ .
In practice, we initially pretrain the channel on the labeled data and then move to full semi-supervised training with some non-zero  $\lambda$  fixed for the rest of training.
4.5 Proposal of standardized candidates
Candidate set heuristic. Performing EM with the full modern vocabulary  $V_{\mathbf{y}}$  as the set of possible normalization candidates is vastly impractical: The forward-backward algorithm runs in  $O(m|V_{\mathbf{y}}|^{n})$  time. In related tasks, this has lead to training heuristics such as iterative EM (Ravi and Knight, 2011). To keep this computation manageable, we propose generating a candidate set  $C(\mathbf{x})$  of  $k$  modern words for each non-standard word  $\mathbf{x}$ . To this end, we use approximate nearest neighbor search with edit distance (Hulden, 2009). The algorithm efficiently searches through an FST, which encodes a part of the vocabulary, with the  $A^{*}$  search. We encode different word frequency bands of the vocabulary as separate FSTs, which we search in parallel. We rerank the candidates taking into account non-standard and standardized words' relative frequencies (see Appendix). Thus, all summations and maximizations over  $V_{\mathbf{y}}$  are performed over the reduced set  $C(\mathbf{x})$ .
Our heuristic embodies normalization by search (§2) and could be replaced with a more informed search and reranking algorithm (Bollmann, 2012; Baron and Rayson, 2008).
Candidate generation with direct model. The candidate set heuristic is too restrictive. It is hard to achieve perfect coverage at manageable candidate set sizes (e.g. if  $\mathbf{x}$  and target  $\mathbf{y}$  have no characters in common as e.g. historical German "fy"  $\mapsto$  "sie" (they)). Worse still, this approach completely fails if the target  $\mathbf{y}$  does not appear in the corpus. This could be because the corpus is small (e.g. most Wikipedias); rich morphology
Algorithm 1 GEM training (§4.4)
Full version uses restarts and candidate pruning (see Appendix).
Input: Unlabeled dataset  $U$ , labeled dataset  $S$ , development set, number of modern candidates  $k$  to generate, number of EM epochs  $K$ , mixture parameter  $\lambda$  combining the unsupervised and supervised objectives.
1: Compute  $k$  candidates  $C(\mathbf{x})$  for each non-standard word type  $\mathbf{x} \in V_{\mathbf{x}}$  from  $U$  (by either method in § 4.5).
2: Randomly initialize channel parameters  $\theta^{(0)}$ .
3: if labeled dataset  $S \neq \emptyset$  then pretrain  $\theta^{(0)}$  on  $S$ .
4: for epoch  $t \gets 1$  to  $K$  do
5:  $E$  -step:
6:  $Q \gets \mathbf{0}^{k \times |V_{\mathbf{x}}|}$
7: Compute channel scores  $p_{\theta(t-1)}(\mathbf{x} \mid \mathbf{y})$  for all  $\mathbf{x} \in V_{\mathbf{x}}$  and  $\mathbf{y} \in C(\mathbf{x})$ . (Use uniform scores if  $t = 1$  and  $S = \emptyset$ .)
8: for non-standard word sequence  $\mathbf{x}_{1:m} \in U$  do
9: Run forward-backward or beam search (§4.4) to compute each word's posterior  $p(\mathbf{Y}_i \mid \mathbf{x}_{1:m})$ .
10: for position  $i\gets 1$  to  $m$  do
11:  $Q(\mathbf{y},\mathbf{x}_i)\gets Q(\mathbf{y},\mathbf{x}_i) + p(\mathbf{Y}_i = \mathbf{y}\mid \mathbf{x}_{1:m})$  for all  $\mathbf{y}\in C(\mathbf{x}_i)$
12:Normalize:  $p(\mathbf{y}\mid \mathbf{x})\gets Q(\mathbf{y},\mathbf{x}) / \sum_{\mathbf{y}}Q(\mathbf{y},\mathbf{x})$  for all  $\mathbf{x}\in V_{\mathbf{x}}$  and  $\mathbf{y}\in C(\mathbf{x})$
13: M-step:
14: Start training from  $\theta^{(t - 1)}$  and use  $p(\mathbf{y}|\mathbf{x})$  in unsupervised objective  $L_{M}(U,\theta)$  (Eq. 4):
15:  $\theta^{(t)}\gets \operatorname *{argmax}_{\theta}\sum_{(\mathbf{x},\mathbf{y})\in S}\log p_{\theta}(\mathbf{x}\mid \mathbf{y}) + \lambda L_M(U,\theta)$
16: return  $\theta^{(t)}$  leading to best accuracy on development set
or orthographic conventions lead to a vast number of word types (e.g. Hungarian); or the target word is not even attested in the standardized variety (e.g. "czuhant")  $\mapsto$ $^*$  "zehant" (immediately) in the Anselm historical German corpus (Krasselt et al., 2015)). We, therefore, also consider candidate generation with a direct model  $q_{\phi}(\mathbf{y}\mid \mathbf{x})$
We bootstrap a direct model from a contextualized generative model. We fit it by minimizing the cross-entropy of the direct model relative to the posterior of the generative model  $p(\mathbf{y} \mid \mathbf{x})$ . For a semi-supervised generative model, this combines with the MLE of the labeled set  $S(\kappa \geq 0)$ :
$$
\begin{array}{l} L (S, U, \phi) = \sum_ {(\mathbf {x}, \mathbf {y}) \in S} \log q _ {\phi} (\mathbf {y} \mid \mathbf {x}) \\ + \kappa \sum_ {\mathbf {x} \in V _ {\mathbf {x}}} \sum_ {\mathbf {y} \in V _ {\mathbf {y}}} p (\mathbf {y} \mid \mathbf {x}) \log q _ {\phi} (\mathbf {y} \mid \mathbf {x}), \tag {6} \\ \end{array}
$$
Any type-level normalization model from prior work could be used here. We choose the direct model to be a neural transducer, like the channel. It generates candidates using beam search.
7287
4.6 Prediction and reranking
We consider two ways of sentence-wise decoding with our generative models. The first uses the maximum a posteriori (MAP) decision rule, which finds a normalization that maximizes the posterior  $p(\mathbf{y}_{1:m} \mid \mathbf{x}_{1:m})$ . Depending on the factorization of the language model, we solve this exactly (with the Viterbi algorithm) or approximately (with beam search).
The other approach is to learn a reranker model on the development set. The model rescores sentence hypotheses  $\hat{\mathbf{y}}_{1:m}$  generated by the base model (with k-best Viterbi or beam search). It uses rich non-local features—the hypothesis' scores under a word- and character-level RNN language models—as well as length-normalized base model score, mean out-of-vocabulary rate and edit distance from  $\mathbf{x}_{1:m}$  (see Appendix). We implement a PRO reranker (Hopkins and May, 2011) that uses hamming loss.
For our experiments, we use eight datasets compiled by various researchers (Pettersson, 2016; Ljubesic et al., 2016; Bollmann, 2018) from historical corpora (Tables 2 and 3).<sup>1</sup>
Seven languages are Indo-European: Germanic (English, German, Icelandic, and Swedish), Romance (Portuguese and Spanish), and Slavic (Slovene). Additionally, we experiment with Hungarian, a Finno-Ugric language. From the Slovene dataset, we only use the collection of the older and more challenging texts in the Bohoric alphabet.
The data are of different genres (letters, religious and scientific writings). The earliest texts are in 14th-c. Middle English. In many datasets, the proportion of identity normalizations is substantial. The smallest word overlap is in the Hungarian data  $(18\%)$ , the largest is in English  $(75\%)$ .
All corpora are tokenized and aligned at the segment and token level. For some datasets, either segments do not coincide with grammatical sentences, or the data have no segment boundaries at all (e.g. Hungarian or Icelandic). In such cases, to make input amenable to training with context, we resort to sentence splitting on punctuation marks.
Algorithm 2 MAP decoding or reranking (§4.6)
Input: Non-standard word sequence  $\mathbf{x}_{1:m}$ , number of modern candidates  $c$  to generate, number of sentence hypotheses  $k$  to generate.
1: for  $i\gets 1$  to  $m$  do
2: Compute  $c$  candidates  $C(\mathbf{x}_i)$  (§4.5).
3: Compute channel scores  $p(\mathbf{x}_i \mid \mathbf{y})$  for all  $\mathbf{y} \in C(\mathbf{x}_i)$ .
4: if MAP decoding then
5: Decode using Viterbi or beam search:
6:  $\mathbf{y}_{1:m}^{*}\gets \underset {\mathbf{y}_{1:m}\in V_{\mathbf{y}}^{m}}{\mathrm{argmax}}p(\mathbf{x}_{1:m},\mathbf{y}_{1:m})$
7: else
8: Produce  $k$  hypotheses using  $k$ -best Viterbi or beam:
9:  $R(\mathbf{x}_{1:m}) \leftarrow \underset{R \subseteq V_{\mathbf{y}}^{m} \text{s.t.} |R| = k}{\arg \max} \sum_{\mathbf{y}_{1:m} \in R} p(\mathbf{x}_{1:m}, \mathbf{y}_{1:m})$
10: Rerank using a linear model with non-local features:
11:  $\mathbf{y}_{1:m}^{*}\gets \operatorname *{argmax}_{\hat{\mathbf{y}}_{1:m}\in R(\mathbf{x}_{1:m})}\psi^{\top}g(\mathbf{x}_{1:m},\hat{\mathbf{y}}_{1:m})$
12: return best hypothesis  $\mathbf{y}_{1:m}^{*}$
We also split very long segments to ensure the maximum segment length of fifty words.
Token alignment is largely one-to-one, with rare exceptions. Clitization and set phrases (e.g. German "muftu"  $\mapsto$  "musst du" (you must), "aller handt"  $\mapsto$  "allerhand" (every kind of)) are common causes for many-to-one alignments, which our models fail to capture properly.
State-of-the-art. We compare our models to the strongest models for historical text normalization:
- the Norma tool (Bollmann, 2012), which implements search over standardized candidates; and
- the character-level statistical machine translation model (cSMT, Ljubesic et al., 2016), which uses the Moses toolkit (Koehn et al., 2007). This approach estimates a character  $n$ -gram language model on external data and fits a MERT reranker model (Och, 2003) on the development set.
According to Bollmann (2019), Norma performs best in the low-resource setting ( $\leq 500$  labeled tokens), and cSMT should be preferred in all other data conditions. Norma's strong performance in the low-resource scenario derives from the fact that searching for candidates can be fairly easy for some languages e.g. English. The reranker trained on the development set is key to cSMT's strength.
Realistic low-resource setting. Our contextualized models are particularly appealing when labeled data are limited to at most a couple of thousand annotated word pairs. This would be the most common application scenario in practice, and approaches requiring tens of thousands of training
<sup>1</sup>The datasets are featured in the large-scale study of Bollmann (2019), who conveniently provides most data in a unified format at https://github.com/coastalcph/histnorm/.
7288
period	train	dev	reference
de	1482–1652	41.9	9.7	Odebrecht et al. (2017)
en	1386–1698	147.8	16.3	Markus (1999)
es	15th–19th c.	97.3	11.7	Vaamonde (2017)
is	15th c.	49.6	6.1	Rögnvaldsson et al. (2012)
pt	15th–19th c.	222.5	26.7	Vaamonde (2017)
hu	1440–1541	134.0	16.7	Simon (2014)
sl	1750–1840s	50.0	5.8	Erjavec (2012)
sv	1527–1812	24.5	2.2	Fiebranz et al. (2011)