SlowGuess/ACL_2020_OCR huggingface dataset:
samples would be ruled out as unrealistic. We, therefore, experiment with small labeled training set sizes  $n$  ranging from 500 to  $5\mathrm{K}$ . Additionally, we consider the unsupervised scenario ( $n = 0$ ), which might be less relevant practically (even a small amount of labeled data might lead to substantial improvement) but allows us to demonstrate most directly the advantage of our approach.
To keep the experiments close to the real-life application scenario (Kann et al., 2019), we additionally cap the size of the development set at  $2\mathrm{K}$  tokens. Otherwise, we require that the development set have 500 tokens more than the labeled set  $S$  to ensure that we validate on not too small a number of word types (e.g. at  $1\mathrm{K}$  tokens, we get only about 600 word types on average).
Finally, the unlabeled set  $U$  comprises non-standard word sequences from all the remaining non-test data. Our sampled development sets are much smaller compared to the original development set from the official data split. Not to waste any data, we also include the historical part of the rest of the original development set into the unlabeled set  $U$ . The labeled training set  $S$  is sampled uniformly at random from  $U$  with targets.
Semi-supervised training with type-level normalization dictionary. Supervision by type-level dictionary (as opposed to token-level annotations) is a simple and effective way of reducing the amount of manually labeled data (Garrette et al., 2013). We simulate type-level normalization dictionary construction by selecting  $d$  most frequent non-standard word types from the original training set. We build a labeled set  $S$  by pairing them with the most frequent standard word types that they normalize to. We experiment on German and Slovene. We use a development set of 500 tokens.
Experimental setup. We use Wikipedia dumps for training language models and the candidate set
Table 2: Historical normalization datasets. Train and development set sizes in thousands of tokens.
I	NC	H	Seg	C@50	C@150
de	43.8	95.6	.155	Y	91.0	94.2
en	74.9	98.0	.087	Y	94.7	96.3
es	72.9	97.2	.125	Y	94.3	95.8
hu	17.6	98.0	.075	N	78.1	81.2
is	46.7	92.4	.213	N	84.3	86.2
pt	65.3	97.4	.129	Y	92.3	94.7
sl	41.1	98.3	.057	N	90.7	92.1
sv	59.9	99.2	.026	Y	89.8	91.5
avg	52.8	97.0	.108	89.2	91.5
Table 3: Historical normalization datasets (cont.). I=proportion of identity normalizations, NC=accuracy of the non-contextual oracle that selects the most frequent normalization for each historical word, H=normalization entropy, Seg=whether the dataset is sentence-segmented (Y=yes, N=no), C@z=C(x) coverage of the dataset at  $z$  standard candidates per historical word type x. All statistics are computed on the official training sets.
heuristic. For the neural HMM, we fit count-based bigram language models using KenLM (Heafield et al., 2013). All RNN language models are parameterized with a Long Short-Term Memory cell (Hochreiter and Schmidhuber, 1997) and use dropout regularization (Zaremba et al., 2014). The HMMs use  $C(\mathbf{x})$  of 150 candidates, the RNN LM-based models use 50 candidates.
We train for 15 iterations of EM setting  $\lambda = \kappa = 0.8$  throughout. We optimize the neural channel with mini-batched AdaDelta (Zeiler, 2012).
We set the beam size of the RNN LM-based models to four for both final decoding and the E-step. For reranking, the base HMMs output  $150\mathrm{k}$  best sentence hypotheses and the RNN LM-based models output the beam. The reranker models are trained with the perceptron algorithm.
The direct models are trained with AdaDelta. We decode them with beam search and rerank the beam with a PRO reranker using the channel and direct model scores and relative frequency as features. We use the top two reranked candidates as the new candidate set. We refer the reader to the Appendix for further details on training.
We train Norma and cSMT on our data splits using the training settings suggested by the authors.
The semi-supervised contextualized approach results in consistent improvements for most languages and labeled data sizes (Tables 4 and 5).
7289
de	en	es	hu	is	pt	sl	sv	avg
identity	44.36	75.29	73.40	17.53	47.62	65.19	40.74	58.59	52.84
best supervised	88.22	95.24	95.02	91.70	87.31	95.18	93.30	91.13	92.14
n = 0
neural HMM	77.49	87.94	88.75	68.25	77.44	82.63	80.84	75.82	79.89
+rerank	81.02	89.92	89.29	68.66	77.09	84.92	83.52	77.19	81.45
+direct+rerank	79.28	89.21	89.71	70.54	78.91	83.81	84.67	79.88	82.00
RNN LM-based	80.70	90.47	87.04	57.86	73.20	82.95	81.01	78.30	78.94
+rerank	80.80	90.18	86.69	57.91	73.75	83.32	82.55	78.16	79.17
+direct+rerank	81.15	91.06	88.16	63.75	79.87	84.27	85.08	81.61	81.87
n = 500
Norma (Bollmann, 2012)	74.00	84.24	86.41	62.57	76.56	81.62	78.04	77.77	77.65
cSMT (Ljubešić et al., 2016)	76.28	85.17	88.88	68.31	79.00	83.00	83.67	81.66	80.75
neural HMM	80.34	88.97	90.34	69.00	78.38	86.88	85.23	80.08	82.40
+rerank	82.89	90.81	91.00	69.88	78.68	88.53	85.75	81.62	83.64
+direct+rerank	82.55	90.48	91.55	71.04	80.90	87.75	86.68	84.55	84.44
RNN LM-based	82.33	91.12	89.78	62.50	75.93	85.84	83.42	80.88	81.48
+rerank	82.42	91.19	90.14	62.21	75.93	86.01	83.85	81.08	81.60
+direct+rerank	83.23	91.66	90.18	67.70	81.63	87.26	87.54	84.11	84.16
n = 1,000
Norma (Bollmann, 2012)	75.52	85.27	87.94	64.84	77.49	83.56	79.16	79.35	79.14
cSMT (Ljubešić et al., 2016)	78.91	86.89	90.44	70.35	80.32	85.14	85.53	84.82	82.85
neural HMM	80.91	89.51	90.82	70.55	79.86	87.52	85.11	81.95	83.28
+rerank	83.34	91.18	92.28	70.95	79.92	89.06	85.81	83.19	84.47
+direct+rerank	82.68	91.18	92.15	73.17	82.38	88.98	87.32	85.78	85.45
RNN LM-based	82.84	91.35	90.45	65.02	77.80	87.03	83.53	82.77	82.60
+rerank	83.14	91.23	90.49	64.73	78.68	87.41	84.77	83.33	82.97
+direct+rerank	83.39	91.76	91.22	69.13	81.22	87.65	87.60	85.86	84.73
Table 4: Test set results for unsupervised and semi-supervised (500 and 1,000 tokens) settings. Best results within each category are highlighted in bold and, where applicable, are statistically significant compared to cSMT ( $p < 0.05$ , McNemar's test). Best supervised results are quoted from Bollmann (2019).
Compared to cSMT, an average error reduction ranges from  $19\%$  ( $n = 500$ ) to almost  $3\%$  ( $n = 5\mathrm{K}$ ) or  $8\%$  excluding Hungarian, the language on which the models perform worst. Reranking provides an important boost (almost  $5\%$  error reduction compared to the base model, and almost  $8\%$  for neural HMMs), and bootstrapping direct model candidates results in even better performance (almost  $14\%$  error reduction).
Unsupervised case. Remarkably, with no labeled training data (and only a 500-token labeled development set), the best configuration achieves  $88.4\%$  of the top scores reported for fully supervised models (Table 2 of Bollmann (2019)). It outperforms the Norma baseline trained on  $n = 1\mathrm{K}$  labeled samples, reducing its error by almost  $4\%$ .
Effects of unlabeled dataset size. We typically see strong performance for languages where the unlabeled dataset  $U$  is large ( $\approx$  official training and development sets together, Table 2). This in-
cludes English, that shows little ambiguity (Table 3) and so would be expected to profit less from contextualization.
Effects of the modern corpus and preprocessing. The size and coverage of the Wikipedia dump (Table 3) for Icelandic and particularly Hungarian degrade the models' performance and are likely the key reason why cSMT outperforms all contextual models for Hungarian as the labeled dataset increases ( $n = 2.5\mathrm{K}$  and  $n = 5\mathrm{K}$ ), despite the large amount of unlabeled Hungarian text. The RNN LM-based models are hit worst due to the poorest coverage. The lack of original segment boundaries (Table 3, Icelandic is only partially segmented) further exacerbates performance.
Remarkably, the overall approach works despite language models and candidate sets using out-of-domain standardized data. Leveraging in-domain data such as collections of literary works from the time period of the source historical text could lead to even better performance (Berg-Kirkpatrick
de	en	es	hu	is	pt	sl	sv	avg	avg\hu
identity	44.36	75.29	73.40	17.53	47.62	65.19	40.74	58.59	52.84	57.87
best supervised	88.22	95.24	95.02	91.70	87.31	95.18	93.30	91.13	92.14	92.19
n = 2.5K
cSMT (Ljubesić et al., 2016)	82.08	88.66	91.47	75.85	82.06	87.82	88.32	87.28	85.56	86.95
neural HMM	82.16	89.72	92.07	70.73	81.00	88.54	85.56	82.38	84.02	85.92
+rerank	84.64	91.58	92.96	71.52	80.77	89.99	86.48	84.44	85.30	87.27
+direct+rerank	84.85	91.48	93.26	74.34	81.86	90.02	88.04	86.80	86.33	88.04
RNN LM-based	83.55	91.79	92.11	67.70	79.19	88.38	85.01	84.06	83.97	86.30
+rerank	83.87	91.84	92.56	67.94	79.64	88.36	85.68	84.31	84.27	86.61
+direct+rerank	84.83	92.16	92.57	72.45	82.91	89.54	†88.12	86.22	86.10	88.05
direct model (from RNN LM)	83.59	91.39	91.79	72.57	83.40	89.75	87.92	†86.94	85.92	87.83
n = 5K
cSMT (Ljubesić et al., 2016)	83.50	90.02	92.52	79.16	83.09	89.83	89.40	†88.51	87.04	88.16
neural HMM	83.15	89.51	92.74	71.92	80.88	89.38	86.52	83.75	84.73	86.56
+rerank	85.19	91.37	93.85	72.72	81.05	90.99	87.52	85.28	86.00	87.89
+direct+rerank	85.76	91.90	94.24	75.74	83.42	90.79	†89.18	88.29	87.41	89.08
RNN LM-based	84.50	92.11	92.64	69.03	79.79	89.07	85.51	85.01	84.71	86.95
+rerank	85.04	92.15	92.36	69.34	79.97	89.48	86.13	85.49	85.00	87.23
+direct+rerank	85.82	92.41	92.32	74.39	83.77	89.86	88.93	87.71	86.90	88.69
direct model (from RNN LM)	85.03	91.84	92.54	75.86	83.97	90.21	88.48	88.56	87.06	88.66
et al., 2013).
Candidate generation with direct model. Generating candidates with the direct model leads to large gains for languages with poor coverage (Icelandic and Hungarian RNN LM-based models see an average error reduction of over  $20\%$  and  $14\%$  respectively). At larger labeled dataset sizes (Table 5), bootstrapping a direct model and reranking its output without context becomes an effective strategy (Icelandic, Portuguese).
Normalization ambiguity. We would expect languages with higher normalization ambiguity to profit from contextualization (Ljubesic et al., 2016). German, Portuguese, and Spanish gain even in the most competitive semi-supervised  $5\mathrm{K}$  condition, consistent with the amount of ambiguity they exhibit (Table 3). Losses and modest gains are observed for languages with the lowest ambiguity rates (Slovene, Swedish).
We look at the accuracies on unambiguous and ambiguous normalizations (Figure 1). The contextual model consistently outperforms cSMT on ambiguous tokens, often by a wide margin and even when cSMT is better overall (Slovene). An extreme case is German at  $n = 5\mathrm{K}$ , where the two approaches perform similarly on unambiguous tokens, yet cSMT suffers considerably on ambiguous ones (38% error reduction by the neural
HMM). German ranks second by normalization ambiguity (Table 3).
Type-level normalization dictionary. We observe gains equivalent to using a token-level dataset of at least double the dictionary size (Table 6). Slovene profits a lot from dictionary supervision, with 1K-type model performing close to the 5K-token model.
Table 5: Test set results for semi-supervised setting (2,500 and 5,000 tokens). Best results within each category are in bold. The differences between cSMT and the best of the proposed models are statistically significant ( $p < 0.05$ , McNemar's test) unless marked with  $\dagger$ . Best supervised results are quoted from Bollmann (2019).
n	de	sl
250	82.08	85.43
1K	83.29	86.33
RNN LM-based	neural HMM
Table 6: Test set results for supervision by type-level normalization dictionary.
Shortcomings of the approach. The general problem of our approach, as well as most approaches that we build on, is reliance on gold tokenization. Overall, we have faced minor issues with tokenization (one notable example is Swedish where  $0.6\%$  of the target-side test data are words with a colon for which we fail to retrieve candidates from Wikipedia). Tokenization remains a challenge for normalization of unpreprocessed non-standard data (Berg-Kirkpatrick et al., 2013).
7291
Figure 1: Test set performance breakdown by unambiguous and ambiguous tokens in  $n = 1\mathrm{K}$  (top) and  $n = 5\mathrm{K}$  (bottom) semi-supervised conditions. Comparisons are between neural HMM (+direct+erank, green) and cSMT (Ljubesic et al., 2016, violet). Ambiguity (=whether a historical word normalizes into more than one standard word type) is computed on the official training data.
7 Future work
Clearly, one can simultaneously use both methods of candidate generation (§4.5). We leave it for future work to verify whether this leads to an improved performance.
Computing the posterior  $p(\mathbf{y} \mid \mathbf{x})$  in both generative models is hard, which is why we are forced to reduce the number of admissible candidates  $\mathbf{y}$  and, in the case of the RNN LM-based model, approximate the posterior with maximization. This problem can be addressed in a principled way by using variational inference (Jordan et al., 1999), a framework for approximate inference that deals with intractable distributions. We leave it for future work to validate its effectiveness for this problem.
As noted earlier, it is a simplification to assume that non-standard text is tokenized. Being able to normalize across token boundaries (by merging multiple non-standard tokens or splitting one into multiple standardized ones) is crucial for tackling real-world text normalization tasks and related problems such as optical character recognition error correction. An appealing direction for future work would be developing a joint model for text tokenization and normalization. One family of latent-variable models that would be suitable for this task are segmental recurrent neural networks (SRNNs, Kong et al., 2016). SRNNs explicitly model input segmentation and have been successfully applied to online handwriting recognition, Chinese word segmentation, joint word segmentation and part-of-speech tagging (Kong et al., 2016; Kawakami et al., 2019).
This paper proposes semi-supervised contextual normalization of non-standard text. We focus on historical data, which has gained attention in the digital humanities community over the past years. We develop simple contextualized generative neural models that we train with expectation-maximization. By leveraging unlabeled data and accessing context at training time, we train accurate models with fewer manually normalized training samples. No labeled training data are necessary to achieve  $88.4\%$  of the best published performance that uses full training sets. Strong gains are observed for most of the considered languages across realistic low-resource settings (up to 5K labeled training tokens). The techniques developed here readily apply to other types of normalization data (e.g. informal, dialectal). We will make our implementation publicly available.2
We thank the reviewers for helpful comments and Eva Pettersson and Manfred Markus for their help with the English dataset. This work has been supported by the Swiss National Science Foundation under grant CR-SII5_173719.
Alistair Baron and Paul Rayson. 2008. VARD2: A tool for dealing with spelling variation in historical corpora. In Postgraduate conference in corpus linguistics.
$^{2}$ https://github.com/ZurichNLP/ acl2020-historical-text-normalization
7292
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research.
Taylor Berg-Kirkpatrick, Alexandre Bouchard-Côte, John DeNero, and Dan Klein. 2010. Painless unsupervised learning with features. In NAACL-HLT.
Taylor Berg-Kirkpatrick, Greg Durrett, and Dan Klein. 2013. Unsupervised transcription of historical documents. In ACL.
Marcel Bollmann. 2012. (semi-)automatic normalization of historical texts using distance measures and the Norma tool. In Second Workshop on Annotation of Corpora for Research in the Humanities (ACRH-2).
Marcel Bollmann. 2018. Normalization of historical texts with neural network models. Sprachwissenschaftliches Institut, Ruhr-Universität.
Marcel Bollmann. 2019. A large-scale comparison of historical text normalization systems. In *NAACL-HLT*.
Marcel Bollmann, Joachim Bingel, and Anders Søgaard. 2017. Learning attention for historical text normalization by learning to pronounce. In ACL.
Marcel Bollmann, Florian Petran, and Stefanie Dipper. 2011. Rule-based normalization of historical texts. In Proceedings of the Workshop on Language Technologies for Digital Humanities and Cultural Heritage (LaTeCH).
Marcel Bollmann, Anders Søgaard, and Joachim Bingel. 2018. Multi-task learning for historical text normalization: Size matters. In Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP.
Peter F. Brown, Stephen A. Della-Pietra, Vincent J. Della-Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation. Computational Linguistics.
Tomaz Erjavec. 2012. The goo300k corpus of historical Slovenia. In LREC.
Izaskun Etxeberria, Inaki Alegia, Larraitz Uria, and Mans Hulden. 2016. Evaluating the noisy channel model for the normalization of historical texts: Basque, Spanish and Slovene. In LREC.
Rosemarie Fiebranz, Erik Lindberg, Jonas Lindstrom, and Maria Ågren. 2011. Making verbs count: the research project 'Gender and Work' and its methodology. Scandinavian Economic History Review, 59(3).
Dan Garrette, Jason Mielens, and Jason Baldridge. 2013. Real-world semi-supervised learning of posttaggers for low-resource languages. In ACL.
Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a #twitter. In ACL.
Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H Clark, and Philipp Koehn. 2013. Scalable modified kneser-ney language model estimation. In ACL.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural computation*, 9(8).
Mark Hopkins and Jonathan May. 2011. Tuning as ranking. In EMNLP.
Mans Hulden. 2009. Fast approximate string matching with finite automata. Procesamento del languaje natural, 43.
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. 1999. An introduction to variational methods for graphical models. Machine learning.
Bryan Jurish. 2010. More than words: Using token context to improve canonicalization of historical german. Journal For Language Technology And Computational Linguistics.
Katharina Kann, Kyunghyun Cho, and Samuel R Bowman. 2019. Towards realistic practices in low-resource natural language processing: The development set. In EMNLP.
Kazuya Kawakami, Chris Dyer, and Phil Blunsom. 2019. Learning to discover, ground and use words with segmental neural language models. In ACL.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In ACL.
Lingpeng Kong, Chris Dyer, and Noah A Smith. 2016. Segmental recurrent neural networks. In ICLR.
Natalia Korchagina. 2017. Normalizing medieval german texts: from rules to deep learning. In Proceedings of the NoDaLiDa 2017 Workshop on Processing Historical Language.
Julia Krasselt, Marcel Bollmann, Stefanie Dipper, and Florian Petran. 2015. Guidelines für die Normalisierung historischer deutscher Texte / Guidelines for Normalizing Historical German Texts. Bochumer Linguistische Arbeitsberichte: 15.
Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics doklady, 10(8).
Nikola Ljubesic, Katja Zupan, Darja Fiser, and Tomaz Erjavec. 2016. Normalising Slovene data: historical texts vs. user-generated content. In KONVENS.
Massimo Lusetti, Tatyana Ruzsics, Anne Gohring, Tanja Samardžić, and Elisabeth Stark. 2018. Encoder-decoder methods for text normalization. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial).
7293
Peter Makarov and Simon Clematide. 2018. Neural transition-based string transduction for limited-resource setting in morphology. In COLING.
Manfred Markus. 1999. Manual of ICAMET (Innsbruck Computer Archive of Machine-Readable English Texts). Leopold-Franzens-Universitat Innsbruck.
Tomáš Mikolov. 2012. Statistical language models based on neural networks. Ph.D. thesis, Brno University of Technology, Czech Republic.
Mehryar Mohri. 1997. Finite-state transducers in language and speech processing. Computational linguistics.
Radford M Neal and Geoffrey E Hinton. 1998. A view of the em algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models. Springer.
Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In ACL.
Carolin Odebrecht, Malte Belz, Amir Zeldes, Anke Lüdeling, and Thomas Krause. 2017. RIDGES Herbology: designing a diachronic multi-layer corpus. Language Resources and Evaluation, 51(3).
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. Journal of machine learning research.
Eva Pettersson. 2016. Spelling normalisation and linguistic analysis of historical text for information extraction. Ph.D. thesis, Acta Universitatis Upsaliensis.
Eva Pettersson, Beata Megyesi, and Joakim Nivre. 2013a. Normalisation of historical text using context-sensitive weighted Levenshtein distance and compound splitting. In NODALIDA.
Eva Pettersson, Beata Megyesi, and Jörg Tiedemann. 2013b. An SMT approach to automatic annotation of historical text. In Proceedings of the Nodalida Workshop on Computational Historical Linguistics.
Sujith Ravi and Kevin Knight. 2011. Deciphering foreign language. In ACL-HLT.
Alexander Robertson and Sharon Goldwater. 2018. Evaluating historical text normalization systems: How well do they generalize? In *NAACL*.
Eiríkur Rögnvaldsson, Anton Karl Ingason, Einar Freyr Sigurdhsson, and Joel Wallenberg. 2012. The Icelandic parsed historical corpus (icepahc). In LREC.
Tanja Samardžić, Yves Scherrer, and Elvira Glaser. 2015. Normalising orthographic and dialectal variants for the automatic processing of swiss german.
In Language and Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics.
Rajhans Samdani, Ming-Wei Chang, and Dan Roth. 2012. Unified expectation maximization. In NAACL-HLT.
Yves Scherrer and Nikola Ljubesic. 2016. Automatic normalisation of the Swiss German ArchiMob corpus using character-level machine translation. In KONVENS.
Claude Elwood Shannon. 1948. A mathematical theory of communication. Bell system technical journal.
Eszter Simon. 2014. Corpus building from Old Hungarian codices. In The evolution of functional left peripheries in Hungarian syntax.
Valentin I Spitkovsky, Hiyan Alshawi, Daniel Jurafsky, and Christopher D Manning. 2010. Viterbi training improves unsupervised dependency parsing. In CoNLL.
Gongbo Tang, Fabienne Cap, Eva Pettersson, and Joakim Nivre. 2018. An evaluation of neural machine translation models on historical spelling normalization. In COLING.
Gael Vaamonde. 2017. Userguide for digital edition of texts in PS Post Scripture.
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329.
Matthew D Zeiler. 2012. ADADELTA: an adaptive learning rate method. arXiv:1212.5701.
Candidate set generation. We extract the text from the Wikipedia dumps with Wikietractor,3 tokenize it with multifit,4 and preprocess the tokens using the procedure of Bollmann (2019). We remove all words containing characters that have a relative frequency  $\leq 0.0001$ . We partition the types into frequency bins with upper limits of  $1,2,5,10,100,10^{3},10^{4},10^{5},10^{6},\infty$ . For each frequency bin, we extract an initial candidate list based on minimum edit distance,  $\operatorname{ED}(h,m)$ . For the computation of edit distance, we ignore all diacritical marks of the letters (by ignoring composing characters according to the Unicode standard). We rerank our modern candidates based on a frequency ratio  $f_{h,m} = \frac{\#h}{\#\mathfrak{m}}$  that punishes rare