# Informative Language Encoding by Variational Autoencoders Using Transformer
## 5 Conclusions  
This paper introduced the use of a Transformer-based variational autoencoder for sentences. We compared the model with an RNN-based variational autoencoder, Transformer decoder, and LSTM. Our model trained input datasets faster than the RNN-based model, and our model was better than the RNN-based model in terms of reconstruction and representation learning. Furthermore, we used our model in a classification task. The results show that the latent variables of our model exceeded those of the RNN-based model, which implies the ability of the model to learn the holistic features of input sentences. Moreover, we also infer that with a strong and complex decoder such as the pre-trained model GPT, posterior collapse might occur. In future work, we hope to combine our model of state-of-the-art pre-trained word embeddings such as GPT [11], BERT [12], and BART [26] with techniques that reduce posterior collapse. In this work, we used the traditional loss function and its KL annealing-equipped version in a variational autoencoder to address  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Data** & \multicolumn{2}{c}{**PTB**} & \multicolumn{2}{c}{**Wiktext2**} \\ \hline Model’s dimension & 128 & 1024 & 128 & 1024 \\ \hline VAE+Transformer & 14.51 & **85.84** & 23.12 & **142.48** \\ \hline VAE+LSTM & 12.64 & 137.36 & 19.05 & 227.65 \\ \hline VAE+GRU & **11.29** & 109.78 & **16.44** & 182.79 \\ \hline Transformer & 11.39 & 63.09 & 19.84 & 106.70 \\ \hline LSTM & 9.89 & 74.37 & 15.14 & 119.79 \\ \hline GRU & 9.44 & 60.44 & 14.81 & 104.28 \\ \hline \hline \end{tabular}
\end{table}
Table 7: The results for the language modeling computation time (in seconds) in the PTB and Wikitext2 datasets.  
\begin{table}
\begin{tabular}{c c c} \hline \hline  & **Model’s Dimension** & **Number of Parameters** \\ \hline \multirow{2}{*}{VAE+Transformer} & 128 & 12,763,706 \\ \cline{2-3}  & 1024 & 118,395,834 \\ \hline \multirow{2}{*}{VAE+LSTM} & 128 & 12,756,922 \\ \cline{2-3}  & 1024 & 87,515,578 \\ \hline \multirow{2}{*}{VAE+GRU} & 128 & 12,526,010 \\ \cline{2-3}  & 1024 & 75,575,738 \\ \hline \multirow{2}{*}{Transformer} & 128 & 8,272,186 \\ \cline{2-3}  & 1024 & 71,461,690 \\ \hline \multirow{2}{*}{LSTM} & 128 & 8,108,346 \\ \cline{2-3}  & 1024 & 48,315,450 \\ \hline \multirow{2}{*}{GRU} & 128 & 8,042,298 \\ \cline{2-3}  & 1024 & 45,034,554 \\ \hline \hline \end{tabular}
\end{table}
Table 8: The number of parameters in Table 7.  
the issue of posterior collapse. We also aimed to compare various techniques, including other loss functions, to alleviate posterior collapse in further studies.  
Investigation, C.O.; Resources, G.L.; Supervision, K.L. All authors have read and agreed to the published version of the manuscript.  
This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2020R1F1A1076278). This work was also supported by 'Human Resources Program in Energy Technology' of the Korea Institute of Energy Technology Evaluation and Planning (KETEP), granted financial resource from the Ministry of Trade, Industry & Energy, Republic of Korea (No. 20204010600090).  
The authors declare that there are no conflict of interest regarding publication of this paper.