SlowGuess/ACL_2020_OCR huggingface dataset:
All of our models build on Transformer. For WMT14 De-En and WMT16 Ro-En, we use the base configuration (Vaswani et al., 2017): 6 blocks, with 512-dimensional embedding, 2048-dimensional feed-forward network, and 8 attention heads. For FLoRes (low-resource) and MTNT (low-resource and noisy), we use a smaller Transformer: 4 layers, 256-dimensional embedding, 1024-dimensional inner layers, and 4 attention heads. Input and output embeddings are shared between the inference network and decoder. We use  $T = 4$  categorical latent variables of dimension 16 (found by grid search on the dev set). Auxiliary bag-of-words predictions are combined with the decoder prediction with  $\lambda = 0.1$ . We optimize using Adam (Kingma and Ba, 2015) with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.98$ ,  $\epsilon = 1\mathrm{E - }8$ , weight decay of 0.001, and the warmup and learning rate schedule of Ott et al. (2018b). All models are trained on 8 NVIDIA V100 GPUs with 32K tokens per mini-batch. We train WMT14 De-En with 200k updates and others with 100k updates. We do not use early stopping.
We employ joint BPE vocabularies. The sizes are  $32\mathrm{k}$  for En-De and En-Ro;  $30\mathrm{k}$  for Fr-En; and  $3\mathrm{k}$  for Si-En. We also use a word dropout rate of 0.4 during training of all models, which is complementary to our approach.
We found the default initialization in the FAIRSEQ NMT toolkit was effective; we did not need to explore several initializations to avoid degenerate models.
Model	DKL	Iqφ(z, x)	Iqφ(z, y)	NLL
DCVAE + KLA	0.001	0.001	4.2E-6	3.17
Our model	0.17	0.18	0.31	3.16
Table 1: Our model mitigates posterior collapse. The KL value refers to  $\mathrm{D}_{\mathrm{KL}}(q_{\phi}(\boldsymbol {z}\mid \boldsymbol {x},\boldsymbol {y})\parallel p_{\theta}(\boldsymbol {z}\mid \boldsymbol {x}))$  for DCVAE and  $\mathrm{D}_{\mathrm{KL}}(q_{\phi}(\boldsymbol {z}\mid \boldsymbol {y})\parallel p_{\theta}(\boldsymbol {z}\mid \boldsymbol {x}))$  for our model.
5.4 Preventing Posterior Collapse
We compare our model to a standard DCVAE lacking the new objective. We report four metrics of posterior collapse on the validation set of WMT Ro-En:
1. Kullback-Leibler divergence (KL).
2. Mutual information between the latent variable and the source:  $\mathrm{I}_{q_{\phi}}(\pmb {z};\pmb {x})$
3. Mutual information between the latent variable and the target:  $\mathrm{I}_{q_{\phi}}(z; \mathbf{y})$ .
4. Negative conditional log-likelihood (NLL) per token.
Table 1 shows that when using standard DCVAE ELBO, even with the common practice of KL annealing (KLA), both the KL loss and mutual information settle to almost 0 which is consistent with the analysis in Equation 5.
We also plot the progression of  $\mathrm{D}_{\mathrm{KL}}$ ,  $\mathrm{I}_{q_{\phi}}(\boldsymbol {z};\boldsymbol {x})$  and  $\operatorname{I}_{q_{\phi}}(\boldsymbol {z};\boldsymbol {y})$  during training in Figure 2. The posterior collapse of the baseline model is apparent: both  $\mathrm{D}_{\mathrm{KL}}$  mutual information terms drop to 0 at the beginning of training as a result ELBO's design. On the other hand, our model, without using any annealing schedule, effectively increases mutual information and prevents KL loss from settling to a degenerate solution early on.
5.5 Translation Quality
We report corpus-level BLEU (Papineni et al., 2002) $^6$  on the test sets where the translations are generated by sampling each  $z_{k}$  with soft-assignment (vs. argmax).
Supervised Learning on Parallel Data First, we evaluate our model's performance when trained with parallel data on standard WMT datasets. Table 2 shows that our model consistently outperforms both VNMT and DCVAE models—which
<sup>6</sup>We use detokenized SacreBLEU (Post, 2018).
8517
Figure 2: Row (A): comparison of KL and mutual information between baseline (DCVAE, solid triangle, orange color) and our model (solid circle, teal color). Rows (B) and (C): ablation study on relative contribution from MICVAE and BoW. All metrics are computed on the WMT16 Ro-En validation set over the course of 140k training updates.
Model	WMT16	WMT14
Ro-En	En-Ro	De-En	En-De
VNMT	34.20	34.27	30.35	25.84
DCVAE	34.16	34.51	29.76	25.46
Our model	34.76	34.97	31.39	26.42
Non-latent	34.73	34.54	30.89	26.36
require ad-hoc KL annealing—while on par with a strong Transformer baseline.
Semi-supervised with Source-side Monolingual Data Leveraging monolingual data is a common practice to improve low resource NMT. One popular approach uses target-side monolingual data through "backtranslation" as a data augmentation, but how to effectively leverage source-side monolingual data is an open challenge (Sennrich et al.,
Table 2: BLEU score on WMT benchmarks. Best result on each dataset is in bold. Our model provides minor gains  $(\leq 0.5$  points) over the standard Transformer, not degrading like VNMT and DCVAE. Alongside improvements in semi-supervised or noisy settings, this suggests that there is no BLEU compromise in choosing this model.
Model	Fr–En	En–Fr
Non-latent	26.7	24.8
DCVAE	26.4	26.1
+ source mono	27.3	26.4
Our model	28.6	26.3
+ source mono	29.8	26.7
Table 3: Translation performance (BLEU) of utilizing source-side monolingual data. Best result on each data condition (with and without monolingual data) is bold.
2016a; Zhang and Zong, 2016; Wu et al., 2019). We use the joint training objective described in Equation 14. To have a fair comparison, we also extend VNMT and DCVAE with the same joint training algorithm, i.e., the newly added monolingual data is used to train their corresponding sequence encoder and inference network with standard VAE ELBO. That is, the only difference is that our model was trained to promote mutual information  $\mathrm{I}_{q_{\phi}}(z,x)$  and  $\mathrm{I}_{q_{\phi}}(z,y)$ . As shown in Table 3, by doing so the proposed model brings larger gains during semi-supervised learning with source-side monolingual data.
8518
Figure 3: BLEU when increasing the number of noisy parallel sentences (ranked by Zipporah) in training, Si-En.
Robustness to Noisy Data While high-quality parallel data is scarce for low-resource language pairs, weakly aligned sentence pairs can be mined from massive unpaired data such as Paracrawl. We evaluate our model's performance when augmenting the training set with increasingly noisy parallel data filtered by Zipporah (Xu and Koehn, 2017). Because VNMT and DCVAE underperform our proposal in previous experiments, we omit them from this experiment. Figure 3 shows the results in the Sinhala-English direction. Our model always outperforms standard Transformer, which struggles as more (and noisier) data is added. The gap grows from +1.2 to +4.7 BLEU.
Ablation Study How do the different ingredients of our proposed approach contribute to preventing posterior collapse and improving translation quality? We explore two variants of the proposed model: 1) modified ELBO only: only adding mutual information term to the training objective, while without gradients from  $\mathcal{L}_{\mathrm{BoW}}$ , 2) BoW only: which is equivalent to DCVAE combined with BoW decoder.
First, we perform the same collapse metrics evaluation as in Table 1. Figure 2(B) suggests that by explicitly adding mutual information term back to the training objective, both  $\mathrm{I}_{q_{\phi}}(z; \boldsymbol{x})$  and  $\mathrm{I}_{q_{\phi}}(z; \boldsymbol{y})$  are effectively raised, while the remaining aggregated KL term is still optimized to zero. Such behavior is consistent with the analysis revealed
Model	De-En (3.9M)	Ro-En (608K)
BoW and LMICVAE	31.4	34.8
BoW only	31.1	34.2
Table 4: Ablation study on translation quality (BLEU). The information-infused loss function provides additional performance over the DCVAE with a bag-of-words decoder.
in Equation 5. On the other hand, regularizing  $z$  with the BoW decoder only, shown in Figure 2(C), is very effective in preventing KL vanishing as well as increasing mutual information. When two approaches are combined, as was shown in Figure 2(A), the model retains higher mutual information for both  $\mathrm{I}_{q_{\phi}}(z; \boldsymbol{x})$  and  $\mathrm{I}_{q_{\phi}}(z; \boldsymbol{y})$ .
Next, we see whether the difference in mutual information yields different translation quality. We compare two models: BoW only (Figure 2(C)) and both (Figure 2(A)), on WMT14 De-En and WMT16 Ro-En test sets. Table 4 shows the difference matters more in a low-data regime.
Analysis of Outputs Delving into model predictions helps us understand how our model outperforms the others. We examined erroneous 1-best predictions on the Ro-En data. We provide salient examples of phenomena we identified in Table 5. (Naturally, as the Ro-En score differences are not dramatic, the predictions are largely similar.)
Several examples support the fact that our model has more fluent and accurate translations than the baseline or VNMT. VNMT often struggles by introducing disfluent words, and both VNMT and Transformer select justifiable but incorrect words. For instance, in our second example, the gender and animacy of the possessor are not specified in Romanian. Our model selects a more plausible pronoun for this context.
Analysis of Latent Variables Finally, we probe whether different latent variables encode different information. We random sample 100 sentences from two test sets of distinct domains, MTNT (Reddit comments) and WMT (news) with 50 sentences each. We plot the  $t$ -SNE projection of their corresponding samples  $z_{k}$  inferred from  $\Phi_{k}$ ,  $k = 1,2,3,4$  respectively. Figure 4 suggests that different latent variables learn to organize the data in different manners, but there was no clear signal that any of them exclusively specialize in encoding a domain label. We leave a thorough analysis of
<sup>7</sup>https://paracrawl.eu/
8519
Source: ma intristeaza foarte tare .
Reference: that really saddens me .
Base: i am very saddened .
VNMT: i am saddened very loudly . (Wrong sense of tare)
Ours: i am very saddened .
Source: cred ca executia sa este gresita .
Reference: i believe his execution is wrong .
Base: i believe that its execution is wrong .
VNMT: i believe that its execution is wrong .
Ours: i believe that his execution is wrong .
Source: da , chinatown
Reference: yes , chinatown
Base: yes , chinatown
VNMT: yes , thin .
Ours: yes , chinatown
Source: nu stiu cine va fi propus pentru aceasta functie .
Reference: i do not know who will be proposed for this position .
Base: i do not know who will be proposed for this function .
VNMT: i do not know who will be proposed for this function .
Ours: i do not know who will be proposed for this position .
Source: recrutarea , o prioritate tot mai mare pentru companii
Reference: recruitment , a growing priority for companies
Base: recruitment , an increasing priority for companies
VNMT: recruitment , [article missing] increasing priority for companies
Ours: recruitment , a growing priority for companies
Table 5: Translation examples from the baseline Transformer, VNMT, and our model. Disfluent words or absences are in red, and slightly incorrect lexical choice is in blue. Romanian diacritics have been stripped.
Figure 4:  $t$ -SNE visualization of  $z_{k}$ ,  $k = 1,2,3,4$  samples from 100 sentences from two datasets with distinct domains, MTNT (orchid) and WMT news (green).
their information specialization to future work.
Unlike most prior work in (conditional) text generation, we tackle posterior collapse without requiring an annealing schedule (Bowman et al., 2016; Sønderby et al., 2016; Kim et al., 2018), a weakened decoder (Gulrajani et al., 2017), or a restricted variational family (Razavi et al., 2019).
Unlike Ma et al. (2018), who also employ bag-of-words as an NMT objective, our BoW decoder only sees the latent variable  $z$ , not the encoder states. Conversely, unlike Weng et al. (2017), our generative decoder has access to both the latent variable and the encoder states; bag-of-words prediction is handled by separate parameters.
VNMT (Zhang et al., 2016) applies CVAE with Gaussian priors to conditional text generation. VRNMT (Su et al., 2018) extends VNMT, mod
eling the translation process in greater granularity. Both needed manually designed annealing schedules to increase KL loss and avoid posterior collapse. Discrete latent variables have been applied to NMT (Kaiser et al., 2017; Gu et al., 2018; Shen et al., 2019), without variational inference or addressing posterior collapse. Approaches to stop posterior collapse include aggressively trained inference networks (He et al., 2019), skip connections (Dieng et al., 2019), and expressive priors (Tomczak and Welling, 2018; Razavi et al., 2019).
Unlike our conditional approach, Shah and Barber (2018) jointly model the source and target text in a generative fashion. Their EM-based inference is more computationally expensive than our amortized variational inference. Eikema and Aziz (2019) also present a generative (joint) model relying on autoencoding; they condition the source text  $\pmb{x}$  on the latent variable  $\pmb{z}$ . Finally, Schulz et al. (2018), like us, value mutual information between the data and the latent variable. While they motivate KL annealing using mutual information, we show that the annealing is unnecessary.
We have presented a conditional generative model with latent variables whose distribution is learned with variation inference, then evaluated it in machine translation. Our approach does not require an annealing schedule or a hamstrung decoder to avoid posterior collapse. Instead, by providing a new analysis of the conditional VAE objective to improve it in a principled way and incorporating an auxiliary decoding objective, we measurably prevented posterior collapse.
As a result, our model has outperformed previous variational NMT models in terms of translation quality, and is comparable to non-latent Transformer on standard WMT Ro  $\leftrightarrow$  En and De  $\leftrightarrow$  En datasets. Furthermore, the proposed method has improved robustness in dealing with uncertainty in data, including exploiting source-side monolingual data as well as training with noisy parallel data.
We thank Alexandra DeLucia, Chu-Cheng Lin, Hongyuan Mei, Kenton Murray, Guanghui Qin, and João Sedoc (alphabetically) for remarks on the exposition.
Léon Bottou and Yann L. Cun. 2004. Large scale online learning. In Advances in Neural Information Processing Systems 16, pages 217-224. Curran Associates, Inc.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21, Berlin, Germany. Association for Computational Linguistics.
Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, Jennifer C. Lai, and Robert L. Mercer. 1992. An estimate of an upper bound for the entropy of English. Computational Linguistics, 18(1):31-40.
Xi Chen, Diederik P. Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2017. Variational lossy autoencoder. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Semisupervised learning for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1965-1974, Berlin, Germany. Association for Computational Linguistics.
Thomas M. Cover and Joy A. Thomas. 2006. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley, New York, NY, USA.
Anna Currey, Antonio Valerio Miceli Barone, and Kenneth Heafield. 2017. Copied monolingual data improves low-resource neural machine translation. In Proceedings of the Second Conference on Machine Translation, pages 148-156, Copenhagen, Denmark. Association for Computational Linguistics.
Adji B. Dieng, Yoon Kim, Alexander M. Rush, and David M. Blei. 2019. Avoiding latent variable collapse with generative skip models. In Proceedings of Machine Learning Research, volume 89 of Proceedings of Machine Learning Research, pages 2397-2405.
Bryan Eikema and Wilker Aziz. 2019. Auto-encoding variational neural machine translation. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 124–141, Florence, Italy. Association for Computational Linguistics.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In 6th
International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. Open-Review.net.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vázquez, and Aaron C. Courville. 2017. Pixelvae: A latent variable model for natural images. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc'Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali-English and Sinhala-English. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6098-6111, Hong Kong, China. Association for Computational Linguistics.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Lagging inference networks and posterior collapse in variational autoencoders. In ICLR.
Matthew D. Hoffman and Matthew J. Johnson. 2016. ELBO surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, volume 1.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with Gumbel-Softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings.
Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. 2017. One model to learn them all. CoRR, abs/1706.05137v1.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. In Proceedings of the Workshop on Continuouss Vector Space Models and their Compositionality, pages 119-126, Sofia, Bulgaria. Association for Computational Linguistics.
Huda Khayrallah and Philipp Koehn. 2018. On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 74-83, Melbourne, Australia. Association for Computational Linguistics.
Yoon Kim, Sam Wiseman, Andrew Miller, David Sontag, and Alexander Rush. 2018. Semi-amortized variational autoencoders. In Proceedings of the
8521
35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2678-2687, Stockholm, Sweden. PMLR.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Adam Lopez and Matt Post. 2013. Beyond bitext: Five open problems in machine translation. In Twenty Years of Bitext.
Shuming Ma, Xu Sun, Yizhong Wang, and Junyang Lin. 2018. Bag-of-words as target for neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 332-338, Melbourne, Australia. Association for Computational Linguistics.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The concrete distribution: A continuous relaxation of discrete random variables. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Paul Michel and Graham Neubig. 2018. MTNT: A testbed for machine translation of noisy text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 543-553, Brussels, Belgium. Association for Computational Linguistics.
Radford M. Neal and Geoffrey E. Hinton. 1998. A view of the EM algorithm that justifies incremental, sparse, and other variants. In Michael I. Jordan, editor, Learning in Graphical Models, pages 355-368. Springer Netherlands, Dordrecht.
Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018a. Analyzing uncertainty in neural machine translation. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 3953-3962. PMLR.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018b. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1-9, Belgium, Brussels. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
Ali Razavi, Aaron van den Oord, Ben Poole, and Oriol Vinyals. 2019. Preventing posterior collapse with delta-vaes. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1530-1538. JMLR.org.
Philip Schulz, Wilker Aziz, and Trevor Cohn. 2018. A stochastic decoder for neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1243-1252, Melbourne, Australia. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Harshil Shah and David Barber. 2018. Generative neural machine translation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 1346-1355. Curran Associates, Inc.
Tianxiao Shen, Myle Ott, Michael Auli, and Marc'Aurelio Ranzato. 2019. Mixture models for diverse machine translation: Tricks of the trade. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5719-5728. PMLR.
8522