# Informative Language Encoding by Variational Autoencoders Using Transformer
## 4 Experiments  
First, we compare the proposed method with a few existing methods in language modeling. In this work, we consider three baseline models: (1) LSTM with a variational autoencoder [2], denoted by VAE-LSTM, which uses a variational autoencoder where both the encoder and decoder are LSTM; (2) an LSTM language model, denoted by LSTM, which uses an LSTM for the prediction of next tokens by previous tokens; (3) Transformer language modeling, denoted by Transformer-Decoder, which uses only the Transformer decoder without an encoder-decoder attention layer in the same way as [11]. We denote the proposed method as VAE-Transformer. We also utilize the variational autoencoder with the pre-trained model GPT2 [16] to see how a pre-trained model works in a variational autoencoder in terms of language modeling. The encoder and the decoder of this model are the Transformer encoder and pre-trained model GPT2, respectively. This model is used for conditional story generation in language modeling, and we give the same inputs to the encoder and the decoder as those used in our model. We denote this model as VAE-GPT. We show the parameter settings for the tested models in Table 1.  
For the comparison, we used four datasets, which are publicly available and widely adopted as benchmark datasets in language modeling: Pen Tree Bank (PTB) [19], Stanford Sentiment Treebank (SST2) [20], Twitter US Airline Sentiment (AIR), and WikiText2. In short, PTB is a dataset for part-of-speech tagging in NLP, selected from Wall Street Journal. SST2 is a collection of movie reviews with two labels (positive, negative). AIR consists of tweets for US airlines with three labels (positive, neutral, negative). We, however, just use two labels (positive, negative) in the following experiments. Lastly, WikiText2 is a subset of wikipedia data, commonly used for language modeling. To evaluate the performance of language modeling, we adopt two evaluation approaches: intrinsic and extrinsic. The intrinsic evaluation considers two aspects of the generated sentences, which are the output of the decoder. The first one is reconstruction, which refers to how well the generated sentences are reconstructed compared with the input sentences. We represent the ability of reconstruction as a perplexity measure [2]. The definition of perplexity (PPL), shown in Equation (7), is related to a measure of how well the probability distribution of a language model predicts a text sequence as reciprocal to the averaged log probability of words. Thus, the smaller the perplexity measure for a language model, the better the model:  
\[ \begin{array}{ll}\mathit{PPL}(\mathbf{x})&=P(x_{1}x_{2}\dots x_{N})^{-\frac{1}{N }}=\exp^{-l},\\ \mathit{where}\ l&=\frac{1}{N}\log P(x_{1}x_{2}\dots x_{N}).\end{array}  \tag{7}\]  
The second aspect considered by the intrinsic evaluation is how well representation learning is accomplished on the basis of a KL divergence measure since the performance of representation learning [2]. Kullback-Leibler divergence (KL) in Equation (2), being a statistical distance, is a measure of how one probability distribution is different from another reference probability distribution. In particular, if a language model produces a low reconstruction error and a high KL divergence, it means that the latent variable, \(z\)