SlowGuess/ACL_2020_OCR huggingface dataset:
Preprocessing We generate a set of 270 candidate transliterations of 26 Russian words to use as queries. However, many of the produced combinations are highly unlikely and yield no results, and some happen to share the spelling with words in other languages (most often other Slavic languages that use Latin script, such as Polish). We scrape public posts on user and group pages, retaining only the information about which posts were authored by the same user, and manually go over the collected set to filter out coincidental results. We additionally preprocess the collected data by normalizing punctuation and removing non-ASCII characters and emoji. We also replace all substrings of the same character repeated more than twice to only two repetitions, as suggested by Darwish et al. (2012), since these repetitions are more likely to be a written expression of emotion than to be explained by the underlying Russian sentence. The same preprocessing steps are applied to the original script side of the data (the annotations and the monolingual language model training corpus) as well.
Annotation guidelines While transliterating, annotators perform orthographic normalization wherever possible, correcting typos and errors in word boundaries; grammatical errors are not corrected. Tokens that do not require transliteration (foreign words, emoticons) or ones that annotator fails to identify (proper names, badly misspellled words) are removed from the romanized sentence and not transliterated. Although it means that some of the test set sentences will not exactly represent the original romanized sequence, it will help us ensure that we are only testing our model's ability to transliterate rather than make word-by-word normalization decisions.
In addition, 200 of the validation sequences are dually annotated to measure the inter-annotator agreement. We evaluate it using character error rate (CER; edit distance between the two sequences normalized by the length of the reference sequence), the same metric we use to evaluate the model's performance. In this case, since neither of the annotations is the ground truth, we compute CER in both directions and average. Despite the discrepancies caused by the annotators deleting unknown words at their discretion, average CER is only 0.014, which indicates a very high level of agreement.
B Hyperparameter settings
WFST model The Witten-Bell smoothing parameter for the language model is set to 10, and the relative entropy pruning threshold is  $10^{-5}$  for the trigram model and  $2 \cdot 10^{-5}$  for higher-order models. Unsupervised training is performed in batches of size 10 and the language model order is increased every 100 batches. While training with the bigram model, we disallow insertions and freeze all the deletion probabilities at  $e^{-100}$ . The EM stepsize decay rate is  $\beta = 0.9$ . The emission arc pruning threshold is gradually decreased from 5 to 4.5 (in the negative log probability space). We perform multiple random restarts for each experiment, initializing the emission distribution to uniform plus random noise.
UNMT baseline Our unsupervised neural baseline uses a single-layer LSTM with hidden state size 512 for both the encoder and the decoder. The embedding dimension is set to 128. For the denoising autoencoding loss, we adopt the default noise model and hyperparameters as described by Lample et al. (2018). The autoencoding loss is annealed over the first 3 epochs.
We tune the maximum training sequence length (controlling how much training data is used) and the maximum allowed decoding length by optimizing the validation set CER. In our case, the maximum output length is important because the evaluation metric penalizes the discrepancy in length between the prediction and the reference; we observe the best results when setting it to 40 characters for Arabic and 180 for Russian. At training time, we filter out sequences longer than 100 characters for either language, which constitute  $1\%$  of the available Arabic training data (both the Arabic-only LM training set and the Latin-only training set combined) but almost  $70\%$  of the Russian data. Surprisingly, the Russian model trained on the remaining  $30\%$  achieves better results than the one trained on the full data; we hypothesize that the improvement comes from having a more balanced training set, since the full data is heavily skewed towards the Cyrillic side (LM training set) otherwise (see Table 2).
8319