# Informative Language Encoding by Variational Autoencoders Using Transformer
## 4 Experiments
### Results
#### 4.1.3 Ablation Study  
We also checked the consistency of the proposed model by performing experiments with different latent dimensions in intrinsic evaluation. We show the results in Table 4 for latent dimension 32. The results for latent dimension 32 are similar with the results for latent dimension 64 in language modeling and representation learning. Because of the size of the model, practical scores are different. However, both results show that the variational autoencoder for Transformer reached the best results in every dataset. These results show that our proposed model is superior to the other models in language modeling and representation learning. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**PTB**} & \multicolumn{4}{c}{**Wikitext2**} & \multicolumn{4}{c}{**SST2**} & \multicolumn{4}{c}{**AIR**} \\ \hline
**Model** & **PPL** & **KL** & **MI** & **PPL** & **KL** & **MI** & **PPL** & **KL** & **MI** & **PPL** & **KL** & **MI** \\ \hline VAE+ Transformer+Cycle & **75.17** & **4.78** & **2.58** & _125.78_ & _3.67_ & _1.73_ & **301.83** & **1.02** & **0.90** & **71.40** & **2.64** & **2.14** \\ \hline VAE+Transformer+Linear & _81.49_ & _2.70_ & _1.65_ & **122.17** & **5.48** & **2.33** & _312.05_ & _0.01_ & _0.01_ & _77.95_ & _0.33_ & _0.30_ \\ \hline VAE+Transformer & 92.23 & 0.00 & 0.00 & 128.46 & 1.28 & 0.68 & 321.14 & 0.01 & 0.00 & 80.23 & 0.00 & 0.00 \\ \hline Transformer Decoder & 96.79 & & & 160.97 & & & 358.63 & & & 79.08 & & \\ \hline VAE+LSTM+Cycle & 88.31 & 0.91 & 0.82 & 131.67 & 1.38 & 0.82 & 504.19 & 0.00 & 0.00 & 118.02 & 0.03 & 0.03 \\ \hline VAE+LSTM+Linear & 90.81 & 0.01 & 0.00 & 136.60 & 0.00 & 0.00 & 503.56 & 0.00 & 0.00 & 115.16 & 0.00 & 0.00 \\ \hline VAE+LSTM & 92.79 & 0.00 & 0.00 & 134.02 & 0.00 & 0.00 & 547.01 & 0.00 & 0.00 & 121.89 & 0.00 & 0.00 \\ \hline LSTM LM & 93.30 & & & 141.90 & & & 486.67 & & & 123.43 & & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Language modeling evaluation. These models have a latent variable whose dimension is 32. Bold-faced and italic numbers represent the best and the second best, respectively. \begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**SST2**} & \multicolumn{4}{c}{**AIR**} \\ \cline{2-13}
**Model** & \multicolumn{2}{c}{**Train**} & \multicolumn{2}{c}{**Validation**} & \multicolumn{2}{c}{**Test**} & \multicolumn{2}{c}{**Train**} & \multicolumn{2}{c}{**Validation**} & \multicolumn{2}{c}{**Test**} \\ \cline{2-13}  & **BCE** & **Acc** & **BCE** & **Acc** & **BCE** & **Acc** & **BCE** & **Acc** & **BCE** & **Acc** & **BCE** & **Acc** \\ \hline VAE+ Transformer+Cycle & 0.630 & 0.689 & 0.641 & 0.661 & **0.618** & **0.700** & 0.492 & 0.793 & 0.459 & 0.797 & **0.483** & 0.800 \\ \hline VAE+LSTM+Cycle & 0.638 & 0.682 & 0.649 & 0.652 & 0.631 & 0.694 & 0.547 & 0.794 & 0.543 & 0.800 & 0.542 & **0.801** \\ \hline VAE+GPT & 0.634 & 0.670 & 0.650 & 0.646 & 0.633 & 0.677 & 0.522 & 0.787 & 0.517 & 0.789 & 0.525 & 0.798 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The results of classification of the SST2 and AIR datasets. BCE represents binary cross entropy, and Acc represents accuracy. Moreover, we can see the effect of LSTM in the model by replacing it with GRU. We show the results in Tables 5 and 6, with the proposed model evaluated by VAE-GRU. The results show that VAE without annealing produces a score of almost zero in KL divergence and Mutual information. This is because of posterior collapse, after which the latent variable becomes a useless representation. It can be observed that the GRU-based model achieved the best performance in the PTB and Wikitext2 datasets, while the Transformer-based model was the best in the SST2 and AIR datasets in the language modeling task. As for the model's computation time, we measured the time required to calculate forward passes in training the data, which is shown in Table 7. We conducted the experiments on both PTB and Wikitext2, whose volumes were substantially large.