SlowGuess/ACL_2020_OCR huggingface dataset:
listed in the BPE merge table. By contrast DPE segments "carts" into "cart" + "s". We attribute the linguistic characteristics of the DPE segments to the fact that DPE conditions the segmentation of a target word on the source sentence and the previous tokens of the target sentence, as opposed to BPE, which mainly makes use of frequency of subwords, without any context.
DPE generally identifies and leverages some linguistic properties, e.g., plural, antonym, normalization, verb tenses, etc. However, BPE tends to deliver less linguistically plausible segmentations, possibly due to its greedy nature and the lack of context. We believe this phenomenon needs further investigation, i.e., the contribution of source vs. target context in DPE segmentations, and a quantitative evaluation of linguistic nature of word fragments produced by DPE. We will leave this to future work.
Conditional Subword Segmentation. One of our hypothesis for the effectiveness of subword segmentation with DPE is that it conditions the segmentation of the target on the source language. To verify this hypothesis, we train mixed character-subword Transformer solely on the target language sentences in the bilingual training corpus using the language model training objective. This is in contrast to the mixed character-subword model used in the DPE segmentation of the main results in Table 2, where the model is conditioned on the source language and trained on the sentence pairs using a conditional language model training objective. Once the mixed character-subword Transformer language model is trained, it is then used to segment the target sentence of the bilingual corpus in the pre-processing step before a translation model is trained.
Table 5 shows the results. It compares the unconditional language model (LM) DPE vs the conditional DPE for segmenting the target language, where we use BPE dropout for segmenting the source language. We observe that without the information from the source, LM DPE is on-par to BPE, and is significantly outperformed by conditional DPE. This observation confirms our hypothesis that segmentation in NMT should be source-dependent.
We are further interested in analyzing the differences of the target language segmentation depending on the source language. For this analysis,
BPE	DPE (ours)
recognises	recognise + s
advocates	advocate + s
eurozone	euro + zone
underlines	underline + s
strengthens	strengthen + s
entrepreneurship	entrepreneur + ship
acknowledges	acknowledge + s
11.30	11 + .30
wines	wine + s
pres + ently	present + ly
f + illled	fill + ed
endors + ement	endorse + ment
blo + c	bl + oc
cru + cially	crucial + ly
eval + uations	evaluation + s
tre + es	tr + ees
tick + ets	tick + et + s
predic + table	predict + able
multilater + alism	multilateral + ism
rat + ings	rating + s
predic + ted	predict + ed
mo + tives	motiv + es
reinfor + ces	reinforce + s
pro + tocols	protocol + s
pro + gressively	progressive + ly
sk + ill	ski + ll
preva + ils	prevail + s
decent + ralisation	decent + ral + isation
sto + red	stor + ed
influ + enz + a	influen + za
margin + alised	marginal + ised
12.00	12 + .00
sta + ying	stay + ing
intens + ity	intensi + ty
rec + ast	re + cast
guid + eline	guide + line
emb + arked	embark + ed
out + lines	outline + s
scen + ari + os	scenario + s
n + ative	na + tive
ma + ture	ma + ture
preven + tative	prevent + ative
hom + eland	home + land
bat + hing	bath + ing
endang + ered	endanger + ed
cont + inen + tal	continent + al
t + enth	ten + th
vul + n + era + bility	vul + ner + ability
realis + ing	real + ising
t + ighter	tight + er
Table 4: Word fragments obtained by BPE vs. DPE. The most frequent words that resulted in a disagreement between BPE and DPE segmentations on  $Et \rightarrow En$  are shown.
we filtered out a multilingual parallel corpus from WMT, which contains parallel sentences in three languages English, Estonian and Romanian. That is, for each English sentence we have the corresponding sentences in Et and Ro. We then trained two DPE segmentation models for the translation tasks of  $\mathrm{Et}\rightarrow \mathrm{En}$  and  $\mathrm{Ro}\rightarrow \mathrm{En}$ , where English is the target language. Figure 4 shows when conditioning
Source Target	BPE drop BPE drop	BPE drop LM DPE	BPE drop DPE
En→Ro	28.07	28.07	28.66
En→Hu	12.94	12.87	13.36
Ro→En	32.56	32.57	32.99
Hu→En	16.61	16.41	17.05
Figure 4: Disagreement of DPE segments between Et-En and Ro-En over English vocabulary
the segmentation of the target on different source languages, DPE can lead to different segmentations even for an identical multi-parallel corpus. The differences are more significant for low frequency words.
Another aspect of DPE segmentation method is its dependency on the segmentation of the source. As mentioned, we segment the target sentence on the fly using our mixed character-subword model given a randomized segmentation of the source produced by BPE dropout. That means during the training of the NMT model where we use BPE dropout for the source sentence, the corresponding target sentence may get a different DPE segmentation given the randomized segmentation of the source sentence. We are interested in the effectiveness of the target segmentation if we commit to a fixed DPE segmentation conditioned on the BPE segmentation of the input. Table 6 shows the results. We observe that there is a marginal drop when using the fixed DPE, which indicates that the encoder can benefit from a stochastic segmentation, while the decoder prefers a deterministic segmentation corresponding to the segmentation of the source.
DPE vs BPE. We are interested to compare the effectiveness of DPE versus BPE for the target, given BPE dropout as the same segmentation
Table 5: DPE-LM learns a segmentation of the target based on language modelling, which is not conditioned on the source language.
Source Target	BPE drop DPE Fixed	BPE drop DPE On The Fly
En→Ro	28.58	28.66
En→Hu	13.14	13.36
En→Et	18.51	18.80
Ro→En	32.73	32.99
Hu→En	16.82	17.05
Et→En	24.37	24.62
Table 6: "DPE Fixed" obtains a fixed segmentation of the target sentence given the BPE-segmented source sentence, whereas "DPE On The Fly" obtain the best segmentation of the target sentence given a randomized segmentation of the source produced by BPE dropout.
Source Target	BPE drop BPE	BPE drop BPE drop	BPE drop DPE
En→Ro	28.04	28.07	28.66
En→Et	18.09	18.20	18.80
Ro→En	32.40	32.56	32.99
Et→En	23.52	23.65	24.62
Table 7: BLEU score of different segmentation methods for the target.
method for the source. Table 7 shows the results. As observed, target segmentation with DPE consistently outperforms BPE, leading to up to .9 BLEU score improvements. We further note that using BPE dropout on the target has a similar performance to BPE, and it is consistently outperformed by DPE.
We further analyze the segmentations produced by DPE vs BPE. Figure 5 shows the percentage of the target words which have different segmentation with BPE and DPE, for different word frequency bands in  $\mathrm{En} \rightarrow \mathrm{Et}$  translation task. We observe that for Estonian words whose occurrence is up to 5 in the training set, the disagreement rate between DPE and BPE is  $64\%$ . The disagreement rate decreases as we go to words in higher frequency bands. This may imply that the main difference between the relatively large BLEU score difference between BPE and DPE is due to their different segmentation mainly for low-frequency words.
We further plot the distribution of BLEU scores by the length of target sentences. As shown in Figure 6, DPE demonstrates much better gains on the longer sentences, compared with the BPE version.
Figure 5: Disagreement of segments between BPE and DPE over Estonian vocabulary.
Figure 6: BLEU scores of BPE vs DPE by the lengths of sentences for  $\mathrm{En}\rightarrow \mathrm{Et}$
This paper introduces Dynamic Programming Encoding in order to incorporate the information of the source language into subword segmentation of the target language. Our approach utilizes dynamic programming for marginalizing the latent segmentations when training, and inferring the highest probability segmentation when tokenizing. Our comprehensive experiments show impressive improvements compared to state-of-the-art segmentation methods in NMT, i.e., BPE and its stochastic variant BPE dropout.
We would like to thank the anonymous reviewers, Taku Kudo, Colin Cherry and George Foster for their comments and suggestions on this work. The computational resources of this work are supported by the Google Cloud Platform (GCP), and by the Multi-modal Australian ScienceS Imaging and Visualisation Environment (MASSIVE) (www.massive.org.au). This material is partly based on research sponsored by Air Force Research Laboratory and DARPA under agreement number FA8750-19-2-0501. The U.S. Government is au
thorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. *ICLR*.
William Chan, Chitwan Sahara, Geoffrey Hinton, Mohammad Norouzi, and Navdeep Jaitly. 2020. *Imputer: Sequence modelling via imputation and dynamic programming*. arXiv:2002.08926.
William Chan, Yu Zhang, Quoc Le, and Navdeep Jaitly. 2016. Latent sequence decompositions. arXiv preprint arXiv:1610.03035.
Colin Cherry, George Foster, Ankur Bapna, Orhan First, and Wolfgang Macherey. 2018. Revisiting character-based neural machine translation with capacity and compression. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4295-4305.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724-1734.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Sebastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1-10.
Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75.
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5:365-378.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1173-1182.
Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramón Fernández, Silvio Amir, Luis Marujo, and Tiago Luís. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1520-1530.
Minh-Thang Luong and Christopher D Manning. 2016. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1054-1063.
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2019. Bpe-dropout: Simple and effective subword regularization. arXiv:1910.13267.
Sara Sabour, William Chan, and Mohammad Norouzi. 2018. Optimal completion distillation for sequence learning. arXiv preprint arXiv:1810.01398.
Chitwan Sahara, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. Nonautoregressive machine translation with latent alignments. arXiv:2004.07437.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1715-1725.
I Sutskever, O Vinyals, and QV Le. 2014. Sequence to sequence learning with neural networks. NIPS.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NIPS.
Ekaterina Vylomova, Trevor Cohn, Xuanli He, and Gholamreza Haffari. 2017. Word representation models for morphologically rich languages in neural machine translation. Proceedings of the First Workshop on Subword and Character Level Models in NLP.
Chong Wang, Yining Wang, Po-Sen Huang, Abdelrahman Mohamed, Dengyong Zhou, and Li Deng. 2017. Sequence modeling via segmentations. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3674-3683. JMLR.org.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144.