# Informative Language Encoding by Variational Autoencoders Using Transformer
## 4 Experiments
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **Parameters** & **Value** \\ \hline \multirow{8}{*}{VAE-Transformer} & model dim & 128 \\  & hidden dim & 512 \\  & heads & 2 \\  & layers & 2 \\  & dropout & 0.1 \\  & activation function & GeLU \\  & number of parameters & 12,763,706 \\ \hline \multirow{8}{*}{VAE-LSTM} & model dim & 128 \\  & encoder bidirectional & TRUE \\  & dropout & 0.1 \\  & number of parameters & 12,526,010 \\ \hline \multirow{8}{*}{VAE-GRU} & model dim & 128 \\  & layers & 2 \\  & encoder bidirectional & TRUE \\  & number of parameters & 12,756,922 \\ \hline \multirow{8}{*}{Transformer-Decoder} & model dim & 128 \\  & hidden dim & 512 \\  & heads & 2 \\  & layers & 2 \\  & dropout & 0.1 \\  & activation function & GeLU \\  & number of parameters & 8,272,186 \\ \hline \multirow{8}{*}{LSTM} & model dim & 128 \\  & layers & 2 \\  & dropout & 0.1 \\  & number of parameters & 8,108,346 \\ \hline \multirow{8}{*}{GRU} & model dim & 128 \\  & layers & 2 \\  & dropout & 0.1 \\  & number of parameters & 8,042,298 \\ \hline \multirow{8}{*}{VAE-GPT} & model dim & 768 \\  & encoder layers & 2 \\  & decoder layers & 12 \\  & dropout & 0.1 \\  & number of parameters & 207,349,971 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameter settings for the tested models. represents the sentence data quite well while possibly well reflecting informative features such as topics or styles from the input sentences. Furthermore, to specify how the latent variables of a language model represent data, we also include mutual information (MI) [21]. Mutual information is a measure of how much two random variables are related. We calculate the mutual information with latent variable \(z\) and input variable \(x\). When KL divergence and mutual information are close to zero, it means that the latent variable \(z\), unable to contain valuable information from input sentences, causes the model to degenerate as conventional language modeling does. Encoders that bring forth latent variable \(z\) with useful information from input sentences will have non-zero KL divergence and mutual information. [2] In addition, we include the extrinsic evaluation as an extension of intrinsic evaluation, especially for representation learning. If representation learning works well, the latent variable \(z\) contains an ample amount of useful information for the input sentences. To implement the usefulness of \(z\) in representation learning, we use a sentiment classification task with \(z\) for the SST2 and AIR datasets. As the classification in previous language models [11; 12] proceeds with pre-training and fine-tuning after training a model for language modeling with input data, we similarly fine-tune the model for classification with the same input data, following the idea of transfer learning. For a fair comparison, we update linear classifier module parameters in the classification model, excluding VAE encoder parameters. We use latent variable \(z\) for classification, following the idea of transfer learning. Figure 7 illustrates the adopted procedure of extrinsic evaluation in classification. For pre-processing input texts in the experiments, we use a sentence-piece tokenizer with a vocabulary size of 30,522, similar to the procedure used for BERT [22]. For VAE-GPT, however, we use a byte pair encoding tokenizer that is used in GPT2. We use a fixed sequence length (including \(<BOS>\) and \(<EOS>\) tokens). When an input sentence is larger than the fixed length, it is truncated. When it is less than the length, it is zero-padded. We use a fixed sequence length of 64 for the PTB, SST2, and AIR datasets and 256 for the Wikitext2 dataset. The parameters of the tested models are shown in Table 1. For fair testing, we make the total number of parameters for the VAE-Transformer and VAE-LSTM models as close as possible. We also make those for the Transformer-Decoder and LSTM as close as possible.