# Informative Language Encoding by Variational Autoencoders Using Transformer
## 1 Introduction  
A variational autoencoder (VAE) [1] has been applied in numerous NLP tasks, including language modeling [2] and semi-supervised text classification [3]. The most prominent component of a VAE in language modeling is the statistical use of latent representation, which aims to contain holistic and informative features in texts such as styles, topics, and semantic features. With this latent representation, samples from its prior distribution can generate diverse and exquisite sentences [2].  
Due to the inherent auto-regressive nature of texts, an auto-regressive decoder such as a recurrent neural network (RNN) and a long short-term memory (LSTM) [4] are also widely used, and a few integrated models with RNN and VAE have also been proposed [2]. Previous VAE-driven language modeling approaches use encoders and decoders in the form of RNN [2] as well as the fusion of RNN and Transformer [5, 6].  
However, the mere use of such auto-regressive decoders, besides failing to include informative latent encodings, induces a so-called posterior collapse that makes latent representations useless [2, 7]. To mitigate this problem, several techniques are proposed in the literature, such as updating inference networks and generative networks in imbalanced ways [7], refactoring loss functions [8], changing loss functions [9], and adopting Kullback-Leibler (KL) annealing [2, 10].  
Transformer has produced state-of-the-art outcomes, becoming a default choice in various natural language tasks, including generative language modeling [11] and discriminative language understanding [12]. Its excellent performance is due to a self-attention mechanism that captures contextual information from entire sequences. Thus, our viewis that the integration of Transformer with a variational autoencoder, which captures the entire sequence and statistical inference, will be beneficial for various natural language tasks.  
To the best of our knowledge, no such models that make use of a Transformer architecture internally coupled with a VAE to build a new language modeling approach have been proposed and tested. In addition, from a computational viewpoint, Transformer has several benefits in comparison with RNNs, such as mitigating vanishing-gradient issues and the ability to handle sequential operations and parallelization [13]. Thus, building a language representation model in VAE solely with Transformer may capture the exquisite holistic features of sentences with parallel computing. Furthermore, a careful adoption of Transformer in our model may bring forth sufficient performance in various NLP tasks without a severe adaptation of the model.  
To sum up, this paper makes the following contributions: (1) We provide a novel Transformer model inherently coupled with a variational autoencoder, which we call a variational autoencoder Transformer (VAE-Transformer), for language modeling; (2) We implement the VAE-Transformer model with KL annealing techniques and perform experiments involving real-life datasets with different sentence lengths. With the results, we verify that the model, which produces more informative embedding representations, is better than previous RNN-based models in reconstruction and representation learning.