SlowGuess/ACL_2020_OCR huggingface dataset:
Now we will analyze embedding spaces learned by different models. We take embeddings learned by models trained with BPE and BPE-dropout and for each token look at the closest neighbors in the corresponding embedding space. Figure 6 shows several examples. In contrast to BPE, nearest neighbours of a token in the embedding space of BPE-dropout are often tokens that share sequences of characters with the original token. To verify this observation quantitatively, we computed character 4-gram precision of top-10 neighbors: the proportion of those 4-grams of the top-10 closest neighbors which are present among 4-grams of the original token. As expected, embeddings of BPE-dropout have higher character 4-gram precision (0.29) compared to the precision of BPE (0.18).
This also relates to the study by Gong et al. (2018). For several tasks, they analyze the em
withdraw	resul	meeting	olec	comptroll
BPE	BPE-dropout	BPE	BPE-dropout	BPE	BPE-dropout	BPE	BPE-dropout	BPE	BPE-dropout
aimed	withd	undert	result	meetings	meetings	olecular	molec	icial	comptrollership
molecules	withdrawal	checkl	results	meet	meet	molecules	olecular	supervis	comptroller
aromatic	withdraw	maastr	resulting	session	eting	ljubl	molecule	&	troll
specialties	withdrawn	&	resulted	conference	me	zona	molecular	subcomm	control
publishers	withdraw	unisp	ults	met	etings	choler	molecules	yugosl	controller
chain	withdrawals	phili	res	workshop	met	oler	aec	trigg	controlled
americ	withdrawing	ζ	resultant	meet	meet	ospheric	oler	sophistic	controllers
chron	dra	preca	ult	sessions	session	olar	tolu	obstac	control
eager	retire	prosecut	ul	convened	et	elic	omet	reag	control
ighty	reti	tali	outcome	reunion	conference	ochlor	olip	entals	controls
Figure 6: Examples of nearest neighbours in the source embedding space of models trained with BPE and BPE-dropout. Models trained on WMT14 En-Fr (4m).
(a) BPE
Figure 7: Visualization of source embeddings. Models trained on WMT14 En-Fr (4m).
(b) BPE-dropout
bedding space learned by a model. The authors find that while a popular token usually has semantically related neighbors, a rare word usually does not: a vast majority of closest neighbors of rare words are rare words. To confirm this, we reduce dimensionality of embeddings by SVD and visualize (Figure 7). For the model trained with BPE, rare tokens are in general separated from the rest; for the model trained with BPE-dropout, this is not the case. While to alleviate this issue Gong et al. (2018) propose to use adversarial training for embedding layers, we showed that a trained with BPE-dropout model does not have this problem.
6.3 Robustness to misspelled input
Models trained with BPE-dropout better learn compositionality of words and the meaning of subwords, which suggests that these models have to be more robust to noise. We verify this by measuring the translation quality of models on a test set augmented with synthetic misspellings. We augment the source side of a test set by modifying each word with the probability of  $10\%$  by applying one of the predefined operations. The operations we consider are (1) removal of one character from a word, (2) insertion of a random character into a word, (3) substitution of a character in a word with a random one. This augmentation produces words
source	BPE	BPE-dropout	diff
En-De
original	27.41	28.01	+0.6
misspelled	24.45	26.03	+1.58
De-En
original	32.69	34.19	+1.5
misspelled	29.71	32.03	+2.32
En-Fr (4m)
original	33.38	33.85	+0.47
misspelled	30.30	32.13	+1.83
En-Fr (16m)
original	34.37	34.82	+0.45
misspelled	31.23	32.94	+1.71
Table 5: BLEU scores for models trained on WMT14 dataset evaluated given the original and misspelled source. For En-Fr trained on  ${16}\mathrm{\;m}$  sentence pairs,  ${BPE}$  -dropout was used only on the source side (Section 5.2).
with the edit distance of 1 from the unmodified words. Edit distance is commonly used to model misspellings (Brill and Moore, 2000; Ahmad and Kondrak, 2005; Pinter et al., 2017).
Table 5 shows the translation quality of the models trained on WMT 14 dataset when given the original source and augmented with misspellings. We deliberately chose large datasets, where improvements from using BPE-dropout are smaller. We can see that while for the original test sets the improvements from using BPE-dropout are usually modest, for misspelled test set the improvements are a lot larger: 1.6-2.3 BLEU. This is especially interesting since models have not been exposed to misspellings during training. Therefore, even for large datasets using BPE-dropout can result in substantially better quality for practical applications where input is likely to be noisy.
Closest to our work in motivation is the work by Kudo (2018), who introduced the subword regularization framework multiple segmentation candidates and a new segmentation algorithm. Other segmentation algorithms include Creutz and Lagus (2006), Schuster and Nakajima (2012), Chittnis and DeNero (2015), Kunchukuttan and Bhattacharyya (2016), Wu and Zhao (2018), Banerjee and Bhattacharyya (2018).
Regularization techniques are widely used for training deep neural networks. Among regularizations applied to a network weights the most popular are Dropout (Srivastava et al., 2014) and  $L_{2}$  regularization. Data augmentation techniques in natural language processing include dropping tokens at random positions or swapping tokens at close positions (Iyyer et al., 2015; Artetxe et al., 2018; Lample et al., 2018), replacing tokens at random positions with a placeholder token (Xie et al., 2017), replacing tokens at random positions with a token sampled from some distribution (e.g., based on token frequency or a language model) (Fadaee et al., 2017; Xie et al., 2017; Kobayashi, 2018). While BPE-dropout can be thought of as a regularization, our motivation is not to make a model robust by injecting noise. By exposing a model to different segmentations, we want to teach it to better understand the composition of words as well as subwords, and make it more flexible in the choice of segmentation during inference.
Several works study how translation quality depends on a level of granularity of a segmentation (Cherry et al., 2018; Kreutzer and Sokolov, 2018; Ding et al., 2019). Cherry et al. (2018) show that trained long enough character-level models tend to have better quality, but it comes with the increase of computational cost for both training and inference. Kreutzer and Sokolov (2018) find that, given flexibility in choosing segmentation level, the model prefers to operate on (almost) character level. Ding et al. (2019) explore the effect of BPE vocabulary size and find that it is better to use small vocabulary for low-resource setting and large vocabulary for a high-resource setting. Following these observations, in our experiments we use different vocabulary size depending on a dataset size to ensure the strongest baselines.
We introduce BPE-dropout - simple and effective subword regularization, which operates within the standard BPE framework. The only difference from BPE is how a word is segmented during model training: BPE-dropout randomly drops some merges from the BPE merge table, which results in different segmentations for the same word. Models trained with BPE-dropout (1) outperform BPE and the previous subword regularization on a wide range of translation tasks, (2) have better quality of learned embeddings, (3) are more robust to noisy input. Future research directions include adaptive dropout rates for different merges and an in-depth analysis of other pathologies in learned token embeddings for different segmentations.
We thank anonymous reviewers for the helpful feedback, Rico Sennrich for valuable comments on the first version of this paper, and Yandex Machine Translation team for discussions and inspiration.
Farooq Ahmad and Grzegorz Kondrak. 2005. Learning a spelling error model from search query logs. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 955-962, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. Unsupervised statistical machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3632-3642, Brussels, Belgium. Association for Computational Linguistics.
Tamali Banerjee and Pushpak Bhattacharyya. 2018. Meaningless yet meaningful: Morphology grounded subword-level NMT. In Proceedings of the Second Workshop on Subword/Character LEvel Models, pages 55-60, New Orleans. Association for Computational Linguistics.
Loic Barrault, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Muller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1-61, Florence, Italy. Association for Computational Linguistics.
Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 272-303, Belgium, Brussels. Association for Computational Linguistics.
Eric Brill and Robert C. Moore. 2000. An improved error model for noisy channel spelling correction. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 286-293, Hong Kong. Association for Computational Linguistics.
Colin Cherry, George Foster, Ankur Bapna, Orhan First, and Wolfgang Macherey. 2018. Revisiting character-based neural machine translation with capacity and compression. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4295-4305, Brussels, Belgium. Association for Computational Linguistics.
Rohan Chitnis and John DeNero. 2015. Variable-length word encodings for neural translation models. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2088-2093, Lisbon, Portugal. Association for Computational Linguistics.
Mathias Creutz and Krista Lagus. 2006. Morfessor in the morpho challenge. In Proceedings of the PASCAL Challenge Workshop on Unsupervised Segmentation of Words into Morphemes, pages 12-17. Cite-seer.
Shuoyang Ding, Adithya Renduchintala, and Kevin Duh. 2019. A call for prudent choice of subword merge operations in neural machine translation. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 204-213, Dublin, Ireland. European Association for Machine Translation.
Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 567-573, Vancouver, Canada. Association for Computational Linguistics.
Chengyue Gong, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2018. Frage: Frequency-agnostic word representation. In Advances in neural information processing systems, pages 1334-1345.
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages
1681-1691, Beijing, China. Association for Computational Linguistics.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 452-457, New Orleans, Louisiana. Association for Computational Linguistics.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing.
Julia Kreutzer and Artem Sokolov. 2018. Learning to segment inputs for nmt favors character-level processing. In Proceedings of the 15th International Workshop on Spoken Language Translation.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Anoop Kunchukuttan and Pushpak Bhattacharyya. 2016. Orthographic syllable as basic unit for SMT between related languages. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1912-1917, Austin, Texas. Association for Computational Linguistics.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations.
Myle Ott, Michael Auli, David Grangier, and MarcAurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning, pages 3956-3965.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. 2017. Mimicking word embeddings using subword RNNs. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 102-112, Copenhagen, Denmark. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on
Machine Translation: Research Papers, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149-5152. IEEE.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Rico Sennrich and Biao Zhang. 2019. Revisiting low-resource neural machine translation: A case study. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 211-221, Florence, Italy. Association for Computational Linguistics.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, Los Angeles.
Yingting Wu and Hai Zhao. 2018. Finding better subword segmentation for neural machine translation. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, pages 53-64. Springer.
Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lvy, Aiming Nie, Dan Jurafsky, and Andrew Y. Ng. 2017. Data-noising as smoothing in neural network language models. In International Conference on Learning Representations.
A Training time
Table 6 shows number of training batches for the experiments in Section 5.1 (Table 2), Table 7 — for the experiments in Section 5.2 (Table 3).
In the main text, all models were trained (and evaluated) on lowercased data. Here we provide results of the models trained and evaluated without lower case (Table 8).
BPE	Kudo (2018)	BPE-dropout
IWSLT15
En-Vi	23	26	36
Vi-En	23	29	33
En-Zh	30	29	43
Zh-En	39	51	100
IWSLT17
En-Fr	36	45	60
Fr-En	32	46	85
En-Ar	30	60	62
Ar-En	41	51	59
WMT14
En-De	468	450	501
De-En	447	442	525
ASPEC
En-Ja	280	165	462
Ja-En	239	144	576
Table 6: Number of thousands of training batches for the experiments from Table 2.
BPE	BPE-dropout
src-only	dst-only	both
250k	47	53	53	85
500k	160	210	250	320
1m	30	114	67	180
4m	100	321	180	600
16m	345	345	-	400
Table 7: Number of thousands of training batches for the experiments from Table 3. Note that we use batch size 4k tokens for small corpora (250k and 500k) and 32k tokens for large corpora (1m, 4m and 16m).
BPE	BPE-dropout
IWSLT15
En-Vi	31.44	32.70
Vi-En	32.19	33.22
IWSLT17
En-Fr	38.79	39.83
Fr-En	38.06	38.60
En-Ar	14.30	15.20
Ar-En	31.56	33.00
Table 8: BLEU scores. Bold indicates the best score; differences with the baselines are statistically significant (with  $p$ -value of 0.05). (Statistical significance is computed via bootstrapping (Koehn, 2004).)