# Informative Language Encoding by Variational Autoencoders Using Transformer
## 4 Experiments
### Results
#### 4.1.2 Extrinsic Evaluation  
In the SST2 dataset, the VAE-Transformer using linear annealing yielded the best result, but in order to achieve consistency with the results of [2; 10], we used cyclic annealing models for extrinsic evaluation. Similarly, in pre-trained models, we fine-tuned just the encoder part of the variational autoencoders in the pre-trained language model and chose  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**PTB**} & \multicolumn{3}{c}{**Wikitext2**} & \multicolumn{3}{c}{**SST2**} & \multicolumn{3}{c}{**AIR**} \\ \hline
**Model** & **PPL** & **KL** & **MI** & **PPL** & **KL** & **MI** & **PPL** & **KL** & **MI** & **PPL** & **KL** & **MI** \\ \hline VAE+ Transformer+_Cycle_ & _75.65_ & **4.40** & **2.45** & 121.71 & 4.27 & 1.96 & 411.40 & _0.21_ & _0.11_ & _83.70_ & _0.70_ & **0.62** \\ \hline VAE+Transformer+_Linear_ & _82.01_ & _2.50_ & _1.54_ & _120.16_ & **5.29** & **2.26** & _408.62_ & 0.18 & 0.08 & 85.92 & 0.01 & 0.01 \\ \hline VAE+Transformer & 91.52 & 0.00 & 0.00 & 128.25 & 0.16 & 0.09 & 413.77 & 0.06 & 0.02 & 88.65 & 0.00 & 0.00 \\ \hline Transformer Decoder & 118.69 & & & 170.42 & & & 456.60 & & & 100.90 & & \\ \hline VAE+LSTM+_Cycle_ & 88.64 & 0.90 & 0.81 & 133.81 & 0.55 & 0.38 & 659.85 & 0.00 & 0.00 & 142.72 & 0.01 & 0.01 \\ \hline VAE+LSTM+_Linear_ & 92.08 & 0.01 & 0.01 & 134.61 & 0.01 & 0.01 & 629.05 & 0.00 & 0.00 & 137.53 & 0.00 & 0.00 \\ \hline VAE+LSTM & 92.04 & 0.00 & 0.00 & 132.23 & 0.03 & 0.02 & 656.12 & 0.00 & 0.00 & 147.00 & 0.00 & 0.00 \\ \hline LSTM LM & 102.44 & & & 151.86 & & & 646.63 & & & 152.25 & & \\ \hline VAE+GPT & **54.23** & 0.01 & 0.00 & **79.10** & 0.02 & 0.01 & **358.90** & **1.57** & **0.74** & **83.44** & **0.86** & _0.52_ \\ \hline \hline \end{tabular}
\end{table}
Table 2: Language modeling evaluation in terms of perplexity (PPL), KL divergence (KL), and mutual information (MI). _Linear_ means KL annealing with a linear function, and _Cycle_ means KL annealing with a cyclic function. Bold-faced numbers and italic ones represent the best and the second best, respectively.  
Figure 9: Learning curves in training the PTB dataset.  
the best model using the validation data. The model was optimized by the AdamW optimizer [24] with cosine annealing [25]. We used an initial learning rate of \(1\times 10^{-4}\), with a final learning rate of \(1\times 10^{-7}\) that restarted every 10 epochs for all datasets. We set the momentum parameters at 0.9 and 0.999 and the weight decay at 0.1. The batch size was 32, and we trained the classification model for extrinsic evaluation in 20 epochs. The result is shown in Table 3. For all the datasets, it can be seen that the proposed model shows better performance than VAE-LSTM in cross entropy. This result demonstrates that the latent variable of the proposed model is more informative and fit for classification tasks than that of VAE-LSTM. In addition, we can observe that VAE-GPT has the largest cross entropy, which implies that with the powerful and complex decoder, the latent variables have low sentence representation capability because of posterior collapse.  
With intrinsic evaluation and extrinsic evaluation, our model surpassed the other tested models. Overall, the experimental results show that the Transformer-based models are better than RNN-based model in reconstruction and representation learning, and better at avoiding posterior collapse than the RNN-based models.