SlowGuess/ACL_2020_OCR huggingface dataset:
Jason R. Smith, Herve Saint-Amand, Magdalena Plamada, Philipp Koehn, Chris Callison-Burch, and Adam Lopez. 2013. Dirt cheap web-scale parallel text from the common crawl. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1374-1383, Sofia, Bulgaria. Association for Computational Linguistics.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. 2015. Learning structured output representation using deep conditional generative models. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 3483-3491. Curran Associates, Inc.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. 2016. Ladder variational autoencoders. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 3738-3746. Curran Associates, Inc.
Jinsong Su, Shan Wu, Deyi Xiong, Yaojie Lu, Xianpei Han, and Biao Zhang. 2018. Variational recurrent neural machine translation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5488-5495. AAAI Press.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104-3112. Curran Associates, Inc.
Christoph Tillmann and Hermann Ney. 2003. Word reordering and a dynamic programming beam search algorithm for statistical machine translation. Computational Linguistics, 29(1):97-133.
Jakub M. Tomczak and Max Welling. 2018. VAE with a VampPrior. In International Conference on Artificial Intelligence and Statistics, AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, volume 84 of Proceedings of Machine Learning Research, pages 1214-1223. PMLR.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Rongxiang Weng, Shujian Huang, Zaixiang Zheng, Xinyu Dai, and Jiajun Chen. 2017. Neural ma
chine translation with word predictions. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 136-145, Copenhagen, Denmark. Association for Computational Linguistics.
Lijun Wu, Yiren Wang, Yingce Xia, Tao QIN, Jianhuang Lai, and Tie-Yan Liu. 2019. Exploiting monolingual data at scale for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4205-4215, Hong Kong, China. Association for Computational Linguistics.
Hainan Xu and Philipp Koehn. 2017. Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2945-2950, Copenhagen, Denmark. Association for Computational Linguistics.
Yilin Yang, Liang Huang, and Mingbo Ma. 2018a. Breaking the beam search curse: A study of (re-)scoring methods and stopping criteria for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3054-3059, Brussels, Belgium. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018b. Breaking the softmax bottleneck: A high-rank RNN language model. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Biao Zhang, Deyi Xiong, Jinsong Su, Qun Liu, Rongrong Ji, Hong Duan, and Min Zhang. 2016. Variational neural discourse relation recognizer. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 382-391, Austin, Texas. Association for Computational Linguistics.
Jiajun Zhang and Chengqing Zong. 2016. Exploiting source-side monolingual data in neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1535-1545, Austin, Texas. Association for Computational Linguistics.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. 2019. Infovae: Balancing learning and inference in variational autoencoders. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 5885-5892. AAAI Press.
8523
Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. 2018. Unsupervised discrete sentence representation learning for interpretable neural dialog generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1098-1107, Melbourne, Australia. Association for Computational Linguistics.
8524
A Derivation of Equation 5
To prove the decomposition of the conditional VAE's regularization term into a mutual information term and a KL divergence term, we introduce a random variable  $\ell$  representing an index into the training data; it uniquely identifies  $(\pmb{x}^{(\ell)},\pmb{y}^{(\ell)})$ . This alteration is "entirely algebraic" (Hoffman and Johnson, 2016) while making our process both more compact and more interpretable.
$$
q (\ell , \boldsymbol {z}) \triangleq q (\ell) q (\boldsymbol {z} \mid \ell) \quad q (\boldsymbol {z} \mid \ell) \triangleq q (\boldsymbol {z} \mid \boldsymbol {x} ^ {(\ell)}, \boldsymbol {y} ^ {(\ell)}) \quad q (\ell) \triangleq \frac {1}{L}
$$
$$
p (\ell , \boldsymbol {z}) \triangleq p (\ell) p (\boldsymbol {z} \mid \ell) \quad p (\boldsymbol {z} \mid \ell) \triangleq p (\boldsymbol {z}) \quad p (\ell) \triangleq \frac {1}{L}
$$
We define the marginals  $p(z)$  and  $q(z)$  as the aggregated posterior (Tomczak and Welling, 2018) and aggregated approximate posterior (Hoffman and Johnson, 2016). (This allows the independence assumption above.) Moving forward will require just a bit of information theory: the definitions of entropy and mutual information. For these, we direct the reader to the text of Cover and Thomas (2006).
Given these definitions, the regularization term of the ELBO objective may be expressed as
$$
\mathbb {E} _ {\ell} \left[ \mathrm {D} _ {\mathrm {K L}} \left(q (\boldsymbol {z} \mid \boldsymbol {x}, \boldsymbol {y}) \parallel p (\boldsymbol {z} \mid \boldsymbol {x})\right) \right] = \sum_ {\ell} \frac {1}{L} q (\boldsymbol {z} \mid \boldsymbol {x}, \boldsymbol {y}) \log \frac {q (\boldsymbol {z} \mid \boldsymbol {x} , \boldsymbol {y})}{p (\boldsymbol {z} \mid \boldsymbol {x})}.
$$
We may now multiply the numerator and denominator by  $\frac{1}{L}$  and use its equivalence to  $p(\ell)$  and  $q(\ell)$ .
$$
= \sum_ {\ell} q (\ell , z) \log \frac {q (\ell , z)}{p (\ell , z)}
$$
Factoring then gives us two log terms.
$$
= \sum_ {\ell} q (\ell , z) \left[ \log \frac {q (z)}{p (z)} + \log \frac {q (\ell \mid z)}{p (\ell)} \right]
$$
We then distribute the weighted sum.
$$
= \mathrm {D} _ {\mathrm {K L}} (q (\boldsymbol {z}) \| p (\boldsymbol {z})) + \mathbb {E} _ {q (\boldsymbol {z})} \left[ \mathrm {D} _ {\mathrm {K L}} (q (\ell | \boldsymbol {z}) | p (\ell)) \right]
$$
Because of how we defined  $p(\ell)$ , we expand the second term and factor out the constant  $\mathrm{H}(p(\ell)) = \log L$ .
$$
= \mathrm {D} _ {\mathrm {K L}} (q (\boldsymbol {z}) \| p (\boldsymbol {z})) + \log L - \mathbb {E} _ {q (\boldsymbol {z})} [ \mathrm {H} (q (\ell | \boldsymbol {z})) ]
$$
Finally, we arrive at the result from Equation 5 by using  $\log L = \mathrm{H}(q(\ell))$ .
$$
= \mathrm {D} _ {\mathrm {K L}} (q (z) \parallel p (z)) + \mathrm {I} _ {q} (\ell ; z).
$$