SlowGuess/NAACL_2021_OCR huggingface dataset:
We study the methods of how to produce and consider global information for NMT. Since we have discussed three successful directions, we compare our method with the baselines of Transformer (Vaswani et al., 2017), generative NMT including GNMT (Shah and Barber, 2018) and MirrorGNMT (Zheng et al., 2020), Deliberation (Xia et al., 2017) and Soft-prototype (Wang et al., 2019). Meanwhile, we have introduced some additional parameters to the model, which is the same as the comparable models. Therefore, we evaluate not only the performance but also the inference efficiency. The comparison of the inference efficiency is based on the inference speed of the vanilla big-Transformer. Besides, we reconfigure MirrorGNMT and GNMT to big-Transformer settings, and we additionally reimplement Soft-prototype on English  $\rightarrow$  Romanian. Table 1 presents the performance of our model and the baselines on the training dataset. We summarize the results that:
- Competitive Translation Quality Our method outperforms the baselines of big-Transformer and GNMT on all the language pairs. Compared to state-of-the-art models, our model gains competitive performance on all the language pairs.
- Clear Advantage in Inference Efficiency Besides competitive performance on all the language pairs, our model has a clear advantage in the comparison of inference efficiency. Specifically, GNMT, Mirror-GNMT and Deliberation introduce computational complexity to the decoder that needs more than 1 iteration to consider a translation (+80% additional time at least), and Soft-prototype increases the computational complexity on the encoder side (+34% additional time). However, our method only introduces a generator to the model so that the computational complexity in the encoder and the decoder is the same as in vanilla big-Transformer, which results in an efficient inferring and an almost free draft (only +5% additional time).
- Improvement from EM-like process We re
6During our test, generative NMT including GNMT and Mirror-GNMT always need 2-3 iterations for the EM-like process, and Deliberation needs a constant of 2 iterations.
<sup>3</sup>http://www.statmt.org/wmt16/translation-task.html <sup>4</sup>https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-BLEU.perl <sup>5</sup>lr = peak_lr × min(1, step/warm_up) × (max(step, warm_up))⁻⁰.⁵, where warm_up = 3000 and peak_lr = 0.05.
newtest2014	newtest2016	Speed
Model	Fr → En	En → Fr	De → En	En → De	Ro → En	En → Ro
Transformer (Vaswani et al., 2017), DM baseline	42.21	41.85	32.11	28.40	34.01	32.83	1 ×
GNMT (Shah and Barber, 2018), GM baseline	42.67	42.22	32.54	28.81	34.52	33.34	2.1 ×
Transformer + Deliberation (Xia et al., 2017)	42.58	29.11	1.8 ×
Mirror-GNMT (Zheng et al., 2020)	33.11	29.22	34.91	33.87	2.7 ×
Transformer + Soft-prototype (Wang et al., 2019)	42.99	29.46	34.12	1.34 ×
our method	42.94	42.73	33.03	29.20	34.89	33.82	1.05 ×
our method with EM-like process	43.05	42.97	33.31	29.49	35.09	34.18	2.42 ×
Table 1: Performance of our method. Our method is competitive on translation quality and has a clear advantage in inference efficiency. DM baseline: discriminative model baseline. GM baseline: generative model baseline.
port a result obtained by employing the EM-like process for our model in the last row. Although there is noticeable room for improvement, it degrades the inference efficiency significantly so that we do not suggest such a combination. We will discuss this result and integration in §6.2 Drafting with EM-like process.
6.2 Drafting with EM-like process
In the most discussion of this work, we sample  $z$  from  $q_{z}(z|s,G(s))$  in a one-shot generative style for the sake of inference efficiency. The previous evaluation shows that such an idea is feasible. Meanwhile, our model shares some properties with generative NMT, which makes us interested in the integration with the EM-like process for the sake of the best translation quality only.
In this scenario, we have two steps to translate  $X$ :
1. We sample a semantic draft  $z$  from  $q_{z}(z|s, G(s))$  and gain a possible translation  $Y'$ .
2. We then sample a new semantic draft  $z'$  from  $q_{z}(z|s, t')$  to predict a possible and new translation  $Y''$ , where  $t' = \frac{1}{M'} \sum_{k=0}^{M'} Enc(Y')_k$  and  $M'$  is the length of  $Y'$ .
The second step can be repeated to maximize a lower bound on  $\log(p_{nmt}(Y|X))$ . We observe some improvements from employing the EM-like process, reporting the result in the last row of Table 1 that we achieve the best performance on all the language pairs. However, most significantly, the translation converges at  $2 \sim 3$  iterations that increase the inference time by  $137\%$  (from  $1.05 \times$  to  $2.42 \times$ ). Concretely, the model needs to re-encode the last translation to obtain a new draft and re-decode the new draft to generate a new translation, e.g., re-encode  $Y'$  to obtain  $Enc(Y')$  and its  $t'$ , resample the draft  $z'$  from  $q_z(z|s,t')$  and re-decode
the aggregation of  $Enc(X)$  and  $z'$ . Thus, we suggest the one-shot generative style in practice.
Additionally, we realize that in this case the improvement may come from not only the re-sampled draft but also the adaptation of two ideas: 1) "double encoding" in Soft-prototype (Wang et al., 2019) because we encode the previously complete translation/prototype for the next translation; 2) "double decoding" in Deliberation (Xia et al., 2017) because we make more than one complete translation. We will justify the significance of the draft in §6.3 Test for Draft and §6.4 Draft Reliance Test.
6.3 Test for Draft
We are interested in whether the draft does indeed provide useful semantics/global information. In the last section, the improvement from the EM-like process can intuitively show the effect of the draft because a better-quality draft re-sampled from the last translation continuously improves the performance, but the improvement may only come from "double encoding" and "double decoding". Therefore, we conduct a test to demonstrate that the generative draft learns the desired semantics.
In this test, we share the same missing word translation task with GNMT (Shah and Barber, 2018). Concretely, the model is forced to give a translation based on the draft heavily. We share the same settings that each word has a  $30\%$  chance of being missing independently. Note that we do not conduct this experiment for Deliberation (Xia et al., 2017) and Soft-prototype (Wang et al., 2019) because such discriminative models do not sample semantics from the semantic space. Table 2 shows the test result on training dataset German  $\leftrightarrow$  English and test dataset newstest2014. We observe that our model outperforms GNMT and achieves competitive performance to Mirror-GNMT (Zheng et al., 2020). Specifically, compared to GNMT, our method trains a multilingual encoder and a cross-lingual generator to encourage shared semantics for the semantic space.
Model	newtest2014
De → En	noisy De → En	En → De	noisy En → De
GNMT (Shah and Barber, 2018)	32.54	23.28	28.81	19.93
Mirror-GNMT (Zheng et al., 2020)	33.11	24.37	29.22	20.74
our method	33.03	23.93	29.20	20.35
our method with EM-like process	33.31	24.21	29.49	20.52
Table 2: Performance of missing word translation. Our method is competitive in this scenario even without integrating language modeling to recover noisy input.
GNMT	Mirror-GNMT	our method	+ EM-like process
DKL	5.73	6.92	6.65	7.03
Table 3: Draft Reliance Test. The average value of  $D_{KL} = D_{KL}q_z(z|X,Y)||p(z)$  on test set German  $\leftrightarrow$  English. Higher value indicates higher reliance on the semantic draft or the latent variable.
Compared to Mirror-GNMT, which gains the improvement from the simultaneously used LM (language model) and back-translation technic (Sennrich et al., 2016a), our model is not integrated with LM to counter noisy input so that Mirror-GNMT gains slightly better performance. We leave the integration with denoising language modeling (Vincent, 2010) for future experiments.
6.4 Draft Reliance Test
We have demonstrated that the semantic draft is useful for the translation task. We further indicate how much the model relies on the semantic draft. Since the objective function Eq.3 is the same as in GNMT (Shah and Barber, 2018) and Mirror-GNMT (Zheng et al., 2020), we report a comparison on the term of  $D_{KL} = D_{KL}q_z(z|X,Y)||p(z)$ , presenting the result in Table 3. The test is conducted on training dataset German  $\leftrightarrow$  English and test dataset newtest2014 by averaging the value of  $D_{KL} = D_{KL}q_z(z|X,Y)||p(z)$ . Our method relies on the semantic draft (or the latent variable from the semantic space) heavier than GNMT does. With the EM-like process, the reliance is higher than Mirror-GNMT.
6.5 Necessity of Semantic Space and  $G$
Although the semantic draft does indeed provide useful global information in §6.3 Test for Draft and §6.4 Draft Reliance Test, we still question the necessity of the semantic space because  $G(s) \approx t$  and  $s \approx t$ . In other words, we can simply process  $G(s)$  or  $s$  to the decoder, which can provide global information for decoding potentially. To justify, we train the model on training dataset German ↔ English and test dataset newtest2014 with 4 dif
ferent types of draft based on the framework  $Dec(Enc(X),draft)$  ..
- We use our full-packaged model  $\text{draft} = z$ , where  $z$  comes from  $q_{z}(z|s, G(s))$ .
-  $\text{draft} = G(s)$  is set for translation to test the significance of the semantic space.
- To test the significance of  $G$ , we set  $\text{draft} = z'$ , where  $z'$  comes from  $q_{z'}(z'|s)$ .
- We test both the significance of  $G$  and the semantic space by setting  $\text{draft} = s$  for translation.
Besides the difference of draft, all the other configurations are the same for this test. We report the result in Table 4, and our observations are that:
- According to "row 2 vs row 4", we can see the significance of the cross-lingual generator  $G$ .
- "row 3 vs row 4" indicates the significance of the semantic space.
- When focusing on "row 2 vs row 3",  $G$  improves general translation performance (column 2&4), and the semantic space improves noisy translation (column 3&5)
We intuitively conclude that the semantic space and the cross-lingual generator  $G$  can further smooth and regularize the semantic for decoding, similar to that is found in GNMT (Shah and Barber, 2018) and (Bowman et al., 2016). Moreover, the cross-lingual generator  $G$  can only restore a coarse semantic so that the model cannot only rely on  $G(s)$  to maintain translation quality when testing in the missing word translation task generally.
6.6 Improvement from Non-parallel Data
We have mentioned the multilingual property of the encoder in our design, using the NMT encoder to process  $X$  and  $Y$ . As reported in multilingual BERT (Devlin et al., 2019; Karthikeyan et al., 2020;
newstest2014
draft type	De → En	noisy De → En	En → De	noisy En → De
\( q_z(z|s,G(s)) \)	33.03	23.93	29.20	20.35
\( G(s) \)	32.85	21.82	29.03	17.97
\( q_{z'}(z'|s) \)	32.74	22.34	28.91	19.14
s	32.15	20.92	28.49	17.11
Table 4: Performance with/without semantic space or/and generator.
Model	newtest2016
Ro → En	En → Ro
Transformer + XLM + non-parallel (Lample and Conneau, 2019)	35.30	34.11
Mirror-GNMT + non-parallel(Zheng et al., 2020)	37.54	35.93
Transformer + Soft-prototype + non-parallel (Wang et al., 2019)	38.05	36.41
our method	34.89	33.82
our method + non-parallel	37.42	35.77
our method with EM-like process + non-parallel	38.19	36.53
Table 5: Performance of training with additional non-parallel data. The performance of our method is competitive, significantly improved by non-parallel data.
Wu and Dredze, 2019), sharing encoder for non-parallel sentences in different languages can still build shared semantic space implicitly. This leads us to experiment with that we can jointly train the encoder with the objective of multilingual BERT. We then train on a relative low-resource language pair *Romanian*  $\leftrightarrow$  English, and we use additional monolingual data *News Crawl articles* 2015 from WMT 2016 to jointly train the multilingual encoder with the objective of multilingual BERT. In Table 5, we report competitive results, and the performance is significantly improved by simultaneously using non-parallel data. Note that, when training on non-parallel data, we can pre-train the multilingual encoder with the BERT objective instead of joint training. We leave this idea for further experiments.
Translation quality can be further improved by global information from the target sentence. Although there have been three feasible solutions, successful methods do not consider inference efficiency carefully, which leads to high cost in inferring. In this work, we present a method/framework to improve the performance of NMT. We sample a semantic draft from semantic space that the decoder can consider the semantic draft to obtain the required global information with high efficiency in inferring. Our empirical study shows that, compared to previously successful methods, our method can achieve competitive performance and has a clear advantage in inference efficiency. Since we do not change the architecture of the NMT model, our model can be further improved by employing pre
training (Lample and Conneau, 2019; Devlin et al., 2019; Radford et al., 2018), back-translation (Sennrich et al., 2016a) and other finetuning methods with non-parallel data. And, our model can also be used in unsupervised NMT (Artetxe et al., 2018; Lample et al., 2018). We leave all these experiments for future work.
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pages 265-283.
Nicholas Altieri and David Duvenaud. 2015. Variational Inference with Gradient Flows. NIPS Workshop, 37:3-6.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings.
Ond rej Bojar, Rajen Chatterjee, Christian Feder-mann, Yvette Graham, Barry Haddow, Matthias
Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation, pages 131-198, Berlin, Germany. Association for Computational Linguistics.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale's Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA. Association for Computational Linguistics.
Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bryan Eikema and Wilker Aziz. 2019. Auto-encoding variational neural machine translation. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 124-141, Florence, Italy. Association for Computational Linguistics.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1243-1252. PMLR.
Alex Graves. 2013. Generating Sequences With Recurrent Neural Networks. arXiv preprint arXiv:1308.0850.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735-1780.
Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
K Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings.
Diederik P Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings.
Diederik P Kingma and Max Welling. 2014. Autoencoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014 - Conference Track Proceedings.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. In Advances in neural information processing systems.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only. In 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings.
Mauro Mezzini. 2018. Empirical study on label smoothing in neural networks. In WSCG 2018 - Short papers proceedings.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook FAIR's WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation, pages 314-319, Florence, Italy. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1278-1286, Beijing, China. PMLR.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Harshil Shah and David Barber. 2018. Generative neural machine translation. In Advances in Neural Information Processing Systems, volume 2018-Decem, pages 1346-1355.
Leslie N Smith. 2017. Cyclic learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pages 464-472. IEEE.
Jinsong Su, Shan Wu, Deyi Xiong, Yaojie Lu, Xianpei Han, and Biao Zhang. 2018. Variational recurrent neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, volume 4, pages 3104-3112.
Ashish Vaswani, Google Brain, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in neural information processing systems, pages 5998-6008.
Pascal Vincent. 2010. Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. Journal of Machine Learning Research, 11:3371-3408.
Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, CengZiang Zhai, and Tie-Yan Liu. 2019. Neural Machine Translation with Soft Prototype. In Advances in Neural Information Processing Systems.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 833-844, Hong Kong, China. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyls, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv, abs/1609.08144.
Yingce Xia, Fei Tian, Lijun Wu, Jianxin Lin, Tao Qin, Nenghai Yu, and Tie-Yan Liu. 2017. Deliberation Networks: Sequence Generation Beyond One-Pass Decoding. In Advances in Neural Information Processing Systems, pages 1784-1794. Curran Associates, Inc.
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. 2016. Variational neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 521-530, Austin, Texas. Association for Computational Linguistics.
Zaixiang Zheng, Hao Zhou, Shujian Huang, Lei Li, Xin-Yu Dai, and Jiajun Chen. 2020. MIRRORGENERATIVE NEURAL MACHINE TRANSLATION. In 8th International Conference on Learning Representations, ICLR 2020 - Conference Track Proceedings.