SlowGuess/ACL_2020_OCR huggingface dataset:
Maria Ryskina $^{1}$  Matthew R. Gormley $^{2}$  Taylor Berg-Kirkpatrick $^{3}$
$^{2}$ Machine Learning Department, Carnegie Mellon University
<sup>3</sup>Computer Science and Engineering, University of California, San Diego
{mryskina,mgormley}@cs.cmu.edu tberg@eng.ucsd.edu
Informal romanization is an idiosyncratic process used by humans in informal digital communication to encode non-Latin script languages into Latin character sets found on common keyboards. Character substitution choices differ between users but have been shown to be governed by the same main principles observed across a variety of languages—namely, character pairs are often associated through phonetic or visual similarity. We propose a noisy-channel WFST cascade model for deciphering the original non-Latin script from observed romanized text in an unsupervised fashion. We train our model directly on romanized data from two languages: Egyptian Arabic and Russian. We demonstrate that adding inductive bias through phonetic and visual priors on character mappings substantially improves the model's performance on both languages, yielding results much closer to the supervised skyline. Finally, we introduce a new dataset of romanized Russian, collected from a Russian social network website and partially annotated for our experiments.
Written online communication poses a number of challenges for natural language processing systems, including the presence of neologisms, code-switching, and the use of non-standard orthography. One notable example of orthographic variation in social media is informal romanization<sup>2</sup>—speakers of languages written in non-Latin alphabets encoding their messages in Latin characters, for convenience or due to technical constraints (improper rendering of native script or keyboard
Figure 1: Example transliterations of a Russian word xopoio [horo'so, 'good'] (middle) based on phonetic (top) and visual (bottom) similarity, with character alignments displayed. The phonetic-visual dichotomy gives rise to one-to-many mappings such as  $u / \int / \rightarrow sh / w$ .
[Phonetically romanized]
[UnderlyingCyrillic]
[Visually romanized]
layout incompatibility). An example of such a sentence can be found in Figure 2. Unlike named entity transliteration where the change of script represents the change of language, here Latin characters serve as an intermediate symbolic representation to be decoded by another speaker of the same source language, calling for a completely different transliteration mechanism: instead of expressing the pronunciation of the word according to the phonetic rules of another language, informal transliteration can be viewed as a substitution cipher, where each source character is replaced with a similar Latin character.
In this paper, we focus on decoding informally. romanized texts back into their original scripts. We view the task as a decipherment problem and propose an unsupervised approach, which allows us to save annotation effort since parallel data for informal transliteration does not occur naturally. We propose a weighted finite-state transducer (WFST) cascade model that learns to decode informal romanization without parallel text, relying only on transliterated data and a language model over the original orthography. We test it on two languages, Egyptian Arabic and Russian, collecting our own dataset of romanized Russian from a Russian social network website vk.com.
1The code and data are available at https://github. com/ryskina/romanization-decipherment
2Our focus on informal transliteration excludes formal settings such as pinyin for Mandarin where transliteration conventions are well established.
8308
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8308-8319 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
4to mowet bit' ly4we?	[Romanized]
ЧTo может 6ыть Лучиме?	[Latent Cyrillic]
Čto možet byt' lučse?	[Scientific]
/ʃto 'moʒit bit' 'lutʃfi/	[IPA]
What can be better?	[Translated]
Figure 2: Example of an informally romanized sentence from the dataset presented in this paper, containing a many-to-one mapping  $\mathcal{H} / u \to w$ . Scientific transliteration, broad phonetic transcription, and translation are not included in the dataset and are presented for illustration only.
Since informal transliteration is not standardized, converting romanized text back to its original orthography requires reasoning about the specific user's transliteration preferences and handling many-to-one (Figure 2) and one-to-many (Figure 1) character mappings, which is beyond traditional rule-based converters. Although user behaviors vary, there are two dominant patterns in informal romanization that have been observed independently across different languages, such as Russian (Paulsen, 2014), dialectal Arabic (Darwish, 2014) or Greek (Chalamandaris et al., 2006):
Phonetic similarity: Users represent source characters with Latin characters or digraphs associated with similar phonemes (e.g.  $\mathcal{M} / \mathrm{m} / \rightarrow m$ ,  $\mathcal{N} / 1 / \rightarrow l$  in Figure 2). This substitution method requires implicitly tying the Latin characters to a phonetic system of an intermediate language (typically, English).
Visual similarity: Users replace source characters with similar-looking symbols (e.g.  $\sqrt[4]{y / u} \rightarrow y$  in Figure 2). Visual similarity choices often involve numerals, especially when the corresponding source language phoneme has no English equivalent (e.g. Arabic  $\varepsilon / \mathfrak{S} / \rightarrow 3$ ).
Taking that consistency across languages into account, we show that incorporating these style patterns into our model as priors on the emission parameters—also constructed from naturally occurring resources—improves the decoding accuracy on both languages. We compare the proposed unsupervised WFST model with a supervised WFST, an unsupervised neural architecture, and commercial systems for decoding romanized Russian (translit) and Arabic (Arabizi). Our unsupervised WFST outperforms the unsupervised neural baseline on both languages.
Prior work on informal transliteration uses supervised approaches with character substitution rules either manually defined or learned from automatically extracted character alignments (Darwish, 2014; Chalamandaris et al., 2004). Typically, such approaches are pipelined: they produce candidate transliterations and rerank them using modules encoding knowledge of the source language, such as morphological analyzers or word-level language models (Al-Badrashiny et al., 2014; Eskander et al., 2014). Supervised finite-state approaches have also been explored (Wolf-Sonkin et al., 2019; Hellsten et al., 2017); these WFST cascade models are similar to the one we propose, but they encode a different set of assumptions about the transliteration process due to being designed for abugida scripts (using consonant-vowel syllables as units) rather than alphabets. To our knowledge, there is no prior unsupervised work on this problem.
Named entity transliteration, a task closely related to ours, is better explored, but there is little unsupervised work on this task as well. In particular, Ravi and Knight (2009) propose a fully unsupervised version of the WFST approach introduced by Knight and Graehl (1998), reframing the task as a decipherment problem and learning cross-lingual phoneme mappings from monolingual data. We take a similar path, although it should be noted that named entity transliteration methods cannot be straightforwardly adapted to our task due to the different nature of the transliteration choices. The goal of the standard transliteration task is to communicate the pronunciation of a sequence in the source language (SL) to a speaker of the target language (TL) by rendering it appropriately in the TL alphabet; in contrast, informal romanization emerges in communication between SL speakers only, and TL is not specified. If we picked any specific Latinscript language to represent TL (e.g. English, which is often used to ground phonetic substitutions), many of the informally Romanized sequences would still not conform to its pronunciation rules: the transliteration process is character-level rather than phoneme-level and does not take possible TL digraphs into account (e.g. Russian  $\frac{cx}{\mathrm{sx}} \rightarrow sh$ ), and it often involves eclectic visual substitution choices such as numerals or punctuation.
8309
tion (e.g. Arabic  $\rightarrow$  ta7t, Russian [dla, 'for']  $\rightarrow$  dl9I).
Finally, another relevant task is translating between closely related languages, possibly written in different scripts. An approach similar to ours is proposed by Pourdamghani and Knight (2017). They also take an unsupervised decipher-ment approach: the cipher model, parameterized as a WFST, is trained to encode the source language character sequences into the target language alphabet as part of a character-level noisy-channel model, and at decoding time it is composed with a word-level language model of the source language. Recently, the unsupervised neural architectures (Lample et al., 2018, 2019) have also been used for related language translation and similar decipherment tasks (He et al., 2020), and we extend one of these neural models to our characterlevel setup to serve as a baseline  $(\S 5)$ .
We train a character-based noisy-channel model that transforms a character sequence  $o$  in the native alphabet of the language into a sequence of Latin characters  $l$ , and use it to decode the romanized sequence  $l$  back into the original orthography. Our proposed model is composed of separate transition and emission components as discussed in §3.1, similarly to an HMM. However, an HMM assumes a one-to-one alignment between the characters of the observed and the latent sequences, which is not true for our task. One original script character can be aligned to two consecutive Latin characters or vice versa: for example, when a phoneme is represented with a single symbol on one side but with a digraph on the other (Figure 1), or when a character is omitted on one side but explicitly written on the other (e.g. short vowels not written in unvocalized Arabic but written in transliteration, or the Russian soft sign  $b$  representing palatalization being often omitted in the romanized version). To handle those alignments, we introduce insertions and deletions into the emission model and modify the emission transducer to limit the number of consecutive insertions and deletions. In our experiments, we compare the performance of the model with and without informative phonetic and visual similarity priors described in §3.2.
If we view the process of romanization as encoding a source sequence  $o$  into Latin characters, we can consider each observation  $l$  to have originated via  $o$  being generated from a distribution  $p(o)$  and then transformed to Latin script according to another distribution  $p(l|o)$ . We can write the probability of the observed Latin sequence as:
$$
p (l) = \sum_ {o} p (o; \gamma) \cdot p (l | o; \theta) \cdot p _ {\text {p r i o r}} (\theta ; \alpha) \tag {1}
$$
The first two terms in (1) correspond to the probabilities under the transition model (the language model trained on the original orthography) and the emission model respectively. The third term represents the prior distribution on the emission model parameters through which we introduce human knowledge into the model. Our goal is to learn the parameters  $\theta$  of the emission distribution with the transition parameters  $\gamma$  being fixed.
We parameterize the emission and transition distributions as weighted finite-state transducers (WFSTs):
Transition WFSA The n-gram weighted finite-state acceptor (WFSA)  $T$  represents a character-level n-gram language model of the language in the native script, producing the native alphabet character sequence  $o$  with the probability  $p(o; \gamma)$ . We use the parameterization of Allauzen et al. (2003), with the states encoding conditioning history, arcs weighted by n-gram probabilities, and failure transitions representing backoffs. The role of  $T$  is to inform the model of what well-formed text in the original orthography looks like; its parameters  $\gamma$  are learned from a separate corpus and kept fixed during the rest of the training.
Emission WFST The emission WFST  $S$  transduces the original script sequence  $o$  to a Latin sequence  $l$  with the probability  $p(l|o;\theta)$ . Since there can be multiple paths through  $S$  that correspond to the input-output pair  $(o,l)$ , this probability is summed over all such paths (i.e. is a marginal over all possible monotonic character alignments):
$$
p (l \mid o; \theta) = \sum_ {e} p (l, e \mid o; \theta) \tag {2}
$$
We view each path  $e$  as a sequence of edit operations: substitutions of original characters with Latin ones  $(c_{o} \rightarrow c_{l})$ , insertions of Latin characters  $(\epsilon \rightarrow c_{l})$ , and deletions of original characters  $(c_{o} \rightarrow \epsilon)$ . Each arc in  $S$  corresponds to one
3The square brackets following a foreign word show its linguistic transliteration (using the scientific and the Buckwalter schemas for Russian and Arabic respectively) and its English translation.
of the possible edit operations; an arc representing the edit  $c_{o}\rightarrow c_{l}$  is characterized by the input label  $c_{o}$ , the output label  $c_{l}$ , and the weight  $-\log p(c_l|c_o;\theta)$ . The emission parameters  $\theta$  are the multinomial conditional probabilities of the edit operations  $p(c_{l}|c_{o})$ ; we learn  $\theta$  using the algorithm described in §3.3.
3.2 Phonetic and visual priors
To inform the model of which pairs of symbols are close in the phonetic or visual space, we introduce the priors on the emission parameters, increasing the probability of an original alphabet character being substituted by a similar Latin one. Rather than attempting to operationalize the notions of phonetic or visual similarity, we choose to read the likely mappings between symbols off human-compiled resources that use the same underlying principle: phonetic keyboard layouts and visually confusable symbol lists. Examples of mappings that we encode as priors can be found in Table 1.
Phonetic similarity Since we think of the informal romanization as a cipher, we aim to capture the phonetic similarity between characters based on association rather than on the actual graphemeto-phoneme mappings in specific words. We approximate it using phonetic keyboard layouts, one-to-one mappings built to bring together "similar-sounding" characters in different alphabets. We take the character pairs from a union of multiple layouts for each language, two for Arabic4 and four for Russian.5 The main drawback of using keyboard layouts is that they require every character to have a Latin counterpart, so some mappings will inevitably be arbitrary; we compensate for this effect by averaging over several layouts.
Visual similarity The strongest example of visual character similarity would be homoglyphs—symbols from different alphabets represented by the same glyph, such as Cyrillic  $a$  and Latin  $a$ . The fact that homoglyph pairs can be made indistinguishable in certain fonts has been exploited in phishing attacks, e.g. when Latin characters are replaced by virtually identical Cyrillic ones (Gabrilovich and Gontmakher, 2002). This led the Unicode Consortium to publish a list of symbols and symbol combinations similar enough to be po
Original	Latin
Phon.	Vis.
p /r/	r	p
ð /b/	b	b, 6
θ /v/	v, w	b
\( \vartheta \)/w, u:, o:/	w, u	—
\( \grave{c} \)/x/	k, x	—
Table 1: Example Cyrillic-Latin and Arabic-Latin mappings encoded in the visual and phonetic priors respectively.
tentially confusing to the human eye (referred to as confusibles). This list contains not only exact homoglyphs but also strongly homoglyphic pairs such as Cyrillic  $IO$  and Latin  $lO$ .
We construct a visual prior for the Russian model from all Cyrillic-Latin symbol pairs in the Unicode confusibles list.7 Although this list does not cover more complex visual associations used in informal romanization, such as partial similarity (Arabic Alif with Hamza i  $\rightarrow$  2 due to Hamza  $\pmb{\varepsilon}$  resembling an inverted 2) or similarity conditioned on a transformation such as reflection (Russian  $\lambda \to \nu$ ), it makes a sensible starting point. However, this restrictive definition of visual similarity does not allow us to create a visual prior for Arabic—the two scripts are dissimilar enough that the confusibles list does not contain any Arabic-Latin character pairs. Proposing a more nuanced definition of visual similarity for Arabic and the associated prior is left for future work.
We incorporate these mappings into the model as Dirichlet priors on the emission parameters:  $\theta \sim \mathrm{Dir}(\alpha)$ , where each dimension of the parameter  $\alpha$  corresponds to a character pair  $(c_{o}, c_{l})$ , and the corresponding element of  $\alpha$  is set to the number of times these symbols are mapped to each other in the predefined mapping set.
We learn the emission WFST parameters in an unsupervised fashion, observing only the Latin side of the training instances. The marginal likelihood of a romanized sequence  $l$  can be computed by
$^{6}$ https://www.unicode.org/Public/security/latest/confusables.txt
In our parameterization, we cannot introduce a mapping from one to multiple symbols or vice versa, so we map all possible pairs instead:  $(\infty, lo) \to (\infty, l)$ ,  $(\infty, o)$ .
4http://arabic.omaralzabir.com/, https://thomasplagwitz.com/2013/01/06/imrans-phonetic-keyboard-for-arabic/5http://winrus.com/kbd_e.htm
8311
Figure 3: Schematic of the emission WFST with limited delay (here, up to 2) with states labeled by their delay values.  $*_{o}$  and  $*_{l}$  represent an arbitrary original or Latin symbol respectively. Weights of the arcs are omitted for clarity; weights with the same input-output label pairs are tied.
summing over the weights of all paths through a lattice obtained by composing  $T \circ S \circ A(l)$ . Here  $A(l)$  is an unweighted acceptor of  $l$ , which, when composed with a lattice, constrains all paths through the lattice to produce  $l$  as the output sequence. The expectation-maximization (EM) algorithm is commonly used to maximize marginal likelihood; however, the size of the lattice would make the computation prohibitively slow. We combine online learning (Liang and Klein, 2009) and curriculum learning (Bengio et al., 2009) to achieve faster convergence, as described in §3.3.1.
3.3.1 Unsupervised learning
We use a version of the stepwise EM algorithm described by Liang and Klein (2009), reminiscent of the stochastic gradient descent in the space of the sufficient statistics. Training data is split into mini-batches, and after processing each minibatch we update the overall vector of the sufficient statistics  $\mu$  and re-estimate the parameters based on the updated vector. The update is performed by interpolating between the current value of the overall vector and the vector of sufficient statistics  $s_k$  collected from the  $k$ -th mini-batch:  $\mu^{(k+1)} \gets (1 - \eta_k) \mu^{(k)} + \eta_k s_k$ . The stepsize is gradually decreased, causing the model to make smaller changes to the parameters as the learning stabilizes. Following Liang and Klein (2009), we set it to  $\eta_k = (k+2)^{-\beta}$ .
However, if the mini-batch contains long sequences, summing over all paths in the corresponding lattices could still take a long time. As we know, the character substitutions are not arbitrary: each original alphabet symbols is likely to be mapped to only a few Latin characters, which means that most of the paths through the lattice would have very low probabilities. We prune the improbable arcs in the emission WFST while training on batches of shorter sentences. Doing this eliminates up to  $66\%$  and up to  $76\%$  of the emission arcs for Arabic and Russian respectively.
We discourage excessive use of insertions and deletions by keeping the corresponding probabiliti
ties low at the early stages of training: during the first several updates, we freeze the deletion probabilities at a small initial value and disable insertions completely to keep the model locally normalized. We also iteratively increase the language model order as learning progresses. Once most of the emission WFST arcs have been pruned, we can afford to compose it with a larger language model WFST without the size of the resulting lattice rendering the computation impractical. The two steps of the EM algorithm are performed as follows:
E-step At the E-step we compute the sufficient statistics for updating  $\theta$ , which in our case would be the expected number of traversals of each of the emission WFST arcs. For ease of bookkeeping, we compute those expectations using finite-state methods in the expectation semiring (Eisner, 2002). Summing over all paths in the lattice is usually performed via shortest distance computation in log semiring; in the expectation semiring, we augment the weight of each arc with a basis vector, where the only non-zero element corresponds to the index of the emission edit operation associated with the arc (i.e. the input-output label pair). This way the shortest distance algorithm yields not only the marginal likelihood but also the vector of the sufficient statistics for the input sequence.
To speed up the shortest distance computation, we shrink the lattice by limiting delay of all paths through the emission WFST. Delay of a path is defined as the difference between the number of the epsilon labels on the input and output sides of the path. Figure 3 shows the schema of the emission WFST where delay is limited. Substitutions are performed without a state change, and each deletion or insertion arc transitions to the next or previous state respectively. When the first (last) state is reached, further deletions (insertions) are no longer allowed.
M-step The M-step then corresponds to simply re-estimating  $\theta$  by appropriately normalizing the obtained expected counts.
Arabic	Russian
Sent.	Char.	Sent.	Char.
LM train	49K	935K	307K	111M
Train	5K	104K	5K	319K
Validation	301	8K	227	15K
Test	1K	20K	1K	72K
Table 2: Splits of the Arabic and Russian data used in our experiments. All Arabic data comes from the LDC BOLT Phase 2 corpus, in which all sentences are annotated with their transliteration into the Arabic script. For the experiments on Russian, the language model is trained on a section of the Taiga corpus, and the train, validation, and test portions are collected by the authors; only the validation and test sentences are annotated.
3.3.2 Supervised learning
We also compare the performance of our model with the same model trained in a supervised way, using the annotated portion of the data that contains parallel  $o$  and  $l$  sequences. In the supervised case we can additionally constrain the lattice with an acceptor of the original orthography sequence:  $A(o)\circ T\circ S\circ A(l)$ . However, the alignment between the symbols in  $o$  and  $l$  is still latent. To optimize this marginal likelihood we still employ the EM algorithm. As this constrained lattice is much smaller, we can run the standard EM without the modifications discussed in §3.3.1.
3.4 Decoding
Inference at test time is also performed using finite-state methods and closely resembles the E-step of the unsupervised learning: given a Latin sequence  $l$ , we construct the machine  $T \circ S \circ A(l)$  in the tropical semiring and run the shortest path algorithm to obtain the most probable path  $\hat{e}$ ; the source sequence  $\hat{o}$  is read off the obtained path.
Here we discuss the data used to train the unsupervised model. Unlike Arabizi, which has been explored in prior work due to its popularity in the modern online community, a dataset of informally romanized Russian was not available, so we collect and partially annotate our own dataset from the Russian social network vk . com.
4.1 Arabic