SlowGuess/ACL_2020_OCR huggingface dataset:
Alexander A. Alemi, Ben Poole, Ian Fischer, Joshua V. Dillon, Rif A. Saurous, and Kevin Murphy. 2018. Fixing a broken ELBO. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, pages 159-168.
Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Representation Learning (ICLR), San Diego, USA.
Hareesh Bahuleyan, Lili Mou, Olga Vechtomova, and Pascal Poupart. 2018. Variational attention for sequence-to-sequence models. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1672-1682, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomczak, and Max Welling. 2018. Sylvester normalizing flows for variational inference. In *Conference on Uncertainty in Artificial Intelligence (UAI)*, Tel Aviv-Yafo, Israel.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21, Berlin, Germany. Association for Computational Linguistics.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. 2017. Density estimation using Real NVP.
Sergey Edunov. 2019. https://github.com/pytorch/fairseq/issues/506#issuecomment-464411433.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In *Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 489–500, Brussels, Belgium.
Bryan Eikema and Wilker Aziz. 2018. Auto-encoding variational neural machine translation. arXiv: 1807.10564v1.
Jonas Gehring, Michael Auli, David Grangier, and Yann N. Dauphin. 2017a. A convolutional encoder model for neural machine translation. In Conference of the Association for Computational Linguistics (ACL), Vancouver, Canada.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017b. Convolutional sequence to sequence learning. In International Conference on Machine Learning (ICML), Sydney, Australia.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317-1327, Austin, Texas. Association for Computational Linguistics.
Diederik P. Kingma and Max Welling. 2014. Autoencoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Arya D. McCarthy, Xian Li, Jiatao Gu, and Ning Dong. 2019. Improved variational neural machine translation by promoting mutual information. https://arxiv.org/abs/1909.09237.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
Artidoro Pagnoni, Kevin Liu, and Shangyan Li. 2018. Conditional variational autoencoder for neural machine translation. CoRR, abs/1812.04405.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Belgium, Brussels. Association for Computational Linguistics.
Victor Prokhorov, Ehsan Shareghi, Yingzhen Li, Mohammad Taher Pilehvar, and Nigel Collier. 2019. On the importance of the Kullback-Leibler divergence term in variational autoencoders for text generation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 118-127, Hong Kong. Association for Computational Linguistics.
Michael Przystupa, Mark Schmidt, and Muhammad Abdul-Mageed. 2019. Investigating the impact of normalizing flows on latent variable machine translation. In Proceedings of The Workshop on Invertible Neural Nets and Normalizing Flows. International Conference of Machine Learning.
Danilo Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. In International Conference on Machine Learning (ICML), pages 1530-1538, Lille, France.
Harshil Shah and David Barber. 2018. Generative neural machine translation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 1346-1355. Curran Associates, Inc.
Jinsong Su, Shan Wu, Deyi Xiong, Yaojie Lu, Xianpei Han, and Biao Zhang. 2018. Variational recurrent neural machine translation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications
7776
of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5488-5495.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Neural Information Processing Systems Conference (NIPS), pages 5998-6008, Long Beach, USA.
Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations (ICLR), New Orleans, USA.
Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. 2016. Variational neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 521-530, Austin, Texas. Association for Computational Linguistics.
A Word dropout
We investigate the effect of different dropout rate and summarize the results in Table 3. In particular, we take the VNMT baseline with Gaussian latent variable for IWSLT (row 3 in Table 1) and for WMT (row 12 in Table 1). As shown, word dropout is important for both setup but it is more so for IWSLT. It seems that tasks with low resources benefit more from word dropout. We also observe that above certain rate, word dropout hurts the performance.
Dropout rate	0.0	0.1	0.2	0.3
IWSLT	34.4	35.7	35.8	35.6
WMT	29.0	29.1	28.8	28.7
B Latent Dimension
We report the results of varying the dimension of latent variable  $(D)$  in Table 4. For this study, we use the VNMT baseline with Gaussian latent variable in IWSLT condition (row 3 in Table 1). Our experiments suggest that the latent dimension between 64 and 128 is optimal. The same conclusion holds for the WMT condition.
Table 3: Results of different dropout rate for IWSLT and WMT setup. The best results are in bold.
D	8	16	32	64	128	256
BLEU	35.6	35.5	35.4	35.7	35.8	35.4
C Normalizing Flow Configuration
In the Experimental Results section, we report the accuracy for our models with 4 flows. In Table 5, we conduct experiments varying the number of flows for the IWSLT condition. Our baseline (num flows=0) is an NMT model with word dropout, which performs on par with the static  $Z$  experiment reported in Table 1's row 3. These results suggest that increasing the number of flows improves accuracy, but the gain diminishes after 4 flows. The results are consistent for all normalizing flows that we considered. We also conduct experiments with employing more flows, but unfortunately, we observe either unstable training or lower accuracy.
Table 4: Results of different dropout rate for IWSLT. The best results are in bold.
Num
Flows	PF	SF
(M=8)	CL
0	35.3
1	35.8	35.6	35.8
2	35.7	35.5	35.8
3	36.0	35.9	35.7
4	36.1	36.0	36.1
5	35.9	36.1	35.9
6	35.8	36.0	35.9
In Table 6, we conduct experiments varying the number of orthogonal columns  $(M)$  in our Sylvester normalizing flows (SF) experiments. As shown, increasing  $M$  improves the accuracy up to  $M = 24$ . We see no additional gain from employing more additional orthogonal columns beyond 24. In Table 1, we report  $M = 8$ , because it introduces the least number of additional parameters.
Table 5: Translation accuracy of VNMT models employing various number of flows in the IWSLT condition. The best results are in bold.
M	2	4	8	16	24	32
BLEU	35.7	35.5	36.0	36.0	36.2	35.9
Table 6: Results of different number of orthogonal columns for SF. The best results are in bold.