# Informative Language Encoding by Variational Autoencoders Using Transformer
## 4 Experiments
### Results
#### 4.1.3 Ablation Study
To check the parallelization of the model, we sized up the model in terms of the parameters. As shown in Table 7, when the model's dimension is 128, the differences between VAE-Transformer, VAE-LSTM, and VAE-GRU are indiscernible even though VAE-GRU is the fastest. However, when the dimension is 1024, the differences between VAE-Transformer, VAE-LSTM, and VAE-GRU become discernible. Even though VAE-Transformer has the largest number of parameters, as shown in Table 8, VAE-Transformer is the fastest. It shows that Transformer-based variational autoencoders possess great parallelization as compared to RNN-based variational autoencoders. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline  & \multicolumn{6}{c}{**SST2**} & \multicolumn{6}{c}{**AIR**} \\ \cline{2-13}
**Model** & \multicolumn{2}{c}{**Train**} & \multicolumn{2}{c}{**Validation**} & \multicolumn{2}{c}{**Test**} & \multicolumn{2}{c}{**Train**} & \multicolumn{2}{c}{**Validation**} & \multicolumn{2}{c}{**Test**} \\ \cline{2-13}  & **BCE** & **Acc** & **BCE** & **Acc** & **BCE** & **Acc** & **BCE** & **Acc** & **BCE** & **Acc** & **BCE** & **Acc** \\ \hline VAE+ Transformer+Cycle & 0.630 & 0.689 & 0.641 & 0.661 & **0.618** & **0.700** & 0.492 & 0.793 & 0.459 & 0.797 & **0.483** & 0.800 \\ \hline VAE+LSTM+Cycle & 0.638 & 0.682 & 0.649 & 0.652 & 0.631 & 0.694 & 0.547 & 0.794 & 0.543 & 0.800 & 0.542 & **0.801** \\ \hline VAE+GRU+Cycle & 0.630 & 0.689 & 0.643 & 0.661 & 0.619 & 0.700 & 0.535 & 0.794 & 0.528 & 0.800 & 0.590 & 0.801 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The results of the classification of the SST2 and AIR datasets. BCE represents binary cross entropy, and Acc represents accuracy. These models have a latent variable whose dimension is 64. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**PTB**} & \multicolumn{6}{c}{**Wikitext2**} & \multicolumn{2}{c}{**SST2**} & \multicolumn{2}{c}{**AIR**} \\ \hline
**Model** & **PPL** & **KL** & **MI** & **PPL** & **KL** & **MI** & **PPL** & **KL** & **MI** & **PPL** & **KL** & **MI** \\ \hline VAE+ Transformer+Cycle & _75.17_ & _4.78_ & _2.58_ & 125.78 & 3.67 & 1.73 & **301.83** & **1.02** & **0.90** & **71.40** & **2.64** & **2.14** \\ \hline VAE+Transformer+Linear & 81.49 & 2.70 & 1.65 & 122.17 & 5.48 & 2.33 & _312.05_ & _0.01_ & _0.01_ & _77.95_ & _0.33_ & _0.30_ \\ \hline VAE+Transformer & 92.23 & 0.00 & 0.00 & 128.46 & 1.28 & 0.68 & 321.14 & 0.01 & 0.00 & 80.23 & 0.00 & 0.00 \\ \hline Transformer Decoder & 96.79 & \multicolumn{6}{c}{160.97} & \multicolumn{6}{c}{358.63} & \multicolumn{6}{c}{79.08} \\ \hline VAE+LSTM+Cycle & 88.31 & 0.91 & 0.82 & 131.67 & 1.38 & 0.82 & 504.19 & 0.00 & 0.00 & 118.02 & 0.03 & 0.03 \\ \hline VAE+LSTM+Linear & 90.81 & 0.01 & 0.00 & 136.60 & 0.00 & 0.00 & 503.56 & 0.00 & 0.00 & 115.16 & 0.00 & 0.00 \\ \hline VAE+LSTM & 92.79 & 0.00 & 0.00 & 134.02 & 0.00 & 0.00 & 547.01 & 0.00 & 0.00 & 121.89 & 0.00 & 0.00 \\ \hline LSTM LM & 93.30 & \multicolumn{6}{c}{141.90} & \multicolumn{6}{c}{486.67} & \multicolumn{6}{c}{123.43} \\ \hline VAE+GRU+Cycle & **69.88** & **5.99** & **2.94** & **110.96** & **9.34** & **2.88** & 381.45 & 0.09 & 0.08 & 99.00 & 0.29 & 0.24 \\ \hline VAE+GRU+Linear & 76.33 & 3.57 & 2.07 & _110.15_ & _9.08_ & _2.87_ & 394.33 & 0.00 & 0.00 & 102.11 & 0.03 & 0.02 \\ \hline VAE+GRU & 86.74 & 0.70 & 0.46 & 123.66 & 0.68 & 0.23 & 376.84 & 0.00 & 0.00 & 95.96 & 0.00 & 0.00 \\ \hline GRU LM & 89.92 & \multicolumn{6}{c}{123.56} & \multicolumn{6}{c}{376.72} & \multicolumn{6}{c}{96.60} \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Language modeling evaluation. These models have a latent variable whose dimension is 32. Bold-faced and italic numbers represent the best and the second best, respectively. In the classification task, the GRU-based model achieved the best results in terms of computational time. One needs to be wary of concluding that the proposed model is inferior to the GRU-based models. Naturally, the size of a GRU-based model is smaller than Transformer- and LSTM-based models.