# Informative Language Encoding by Variational Autoencoders Using Transformer
## 3 The Proposed Model, VAE-Transformer  
The proposed VAE-Transformer model combines Transformer with a variational autoencoder for both the encoder and the decoder, as shown in Figure 6. This model remains mostly the same as the original Transformer. However, the proposed model differs from the original model in that a variational autoencoder is plugged in to connect its encoder output with its encoder-decoder structure so as to apply and adjust variational effects in the decoding process. It is worth comparing the VAE-based Transformer with the original Transformer and VAE-equipped RNNs. In addition, we believe that the embedding by the VAE-based Transformer will be diverse while maintaining an effective representation of input texts, such that the proposed model may be potentially useful in further NLP tasks such as summarization, machine translation, among others.  
The encoder remains the same as in Transformer, except for the connection of the encoder output to the decoder by VAE. hlln order to create a latent variable \(\mathbf{z}\), we attach an additional neural network after the last layer of the encoder. This involves \(\mathbf{\mu}\) and \(\sigma\), which are the distribution parameters of the latent variable \(\mathbf{z}\), which is the same as in [1] and  
Figure 6: Illustration of a Transformer-based variational autoencoder. \(O\) is the output of the encoder layer, and the subscripts correspond with the input sentences. The latent-decoder attention layer is indicated with a red line. The latent variable \(\mathbf{z}\) from \(<CLS>\) is indicated by the blue line.  
shown as a blue box in Figure 6. In addition, we revise the encoder-decoder attention in the Transformer decoder so that the latent variable \(\mathbf{z}\), instead of the output of the encoder, feeds the attention module. The rationale of this structure is that it is quite close to the encoder-decoder attention in Transformer, therefore allowing us to take advantage of the original Transformer decoder structure. Thus, by removing the previous encoder-decoder attention, we provide more attention to the latent variable and the output of the decoder. We denote this structure as a latent-decoder attention after the feed-forward network, displayed as the red box in Figure 6.  
We define the abbreviations for the tokens in use: token \(<CLS>\) means the beginning of a sentence, and token \(<SEP>\) means the end of a sentence. Using the encoder, we generate the output of the input sequence \(X=[<CLS>,x_{1},x_{2},\ldots,x_{n}]\), in which the output of \(<CLS>\) represents the summary of \(X\), which is similar to BERT [12]. Using a neural network, as the blue box shows in Figure 6, we convert the output of \(<CLS>\) in the encoder into its stochastically sampled representation, \(\mathbf{z}^{\prime}\), by a Gaussian distribution, with the mean \(\mathbf{\mu}_{z}\) and variance \(\mathbf{\sigma_{z}}\) of the latent variable \(\mathbf{z}\). Because of the indifferentiable sampling process, we use a reparametrization trick that outsources the sampling process to the additional variable, whose distribution is \(N(\mathbf{0},\mathbf{I})\):  
\[\mathbf{z}^{\prime}=\mathbf{\mu}_{z}+\mathbf{\sigma}_{z}\odot\mathbf{\epsilon},\ \ \mathbf{ \epsilon}\sim N(\mathbf{0},\mathbf{I}),\]  
where \(\odot\) is the element-wise product [1]. Using the decoder, we generate the embeddings, \(\mathbf{z}^{\prime}\), of the input sequence, \(Y=[<CLS>,x_{1},x_{2},\ldots,x_{n}]\), and process them in the masked self-attention layer. We process the samples of latent variable \(\mathbf{z}\) and the output of the masked self-attention layer in the latent-decoder attention layer, similar to the process performed in the encoder-decoder attention layer of the original Transformer decoder. Lastly, we predict the token \(x_{t}\) based on \(\mathbf{x}_{<t}\) and \(\mathbf{z}\). We obtain the loss of the entire output sequence by fitting it onto \([x_{1},x_{2},\ldots,x_{n},<SEP>]\).