# Informative Language Encoding by Variational Autoencoders Using Transformer
## 2 Preliminaries  
In this section, we briefly explain the models of variational autoencoders and Transformer to propose a new integrated model that combines the two. **Variational Autoencoder**. A VAE model is a deep generative model using latent variables, as shown in Figure 1[1]. VAE models assume that observable data derive from latent (hidden) variables that follow a simple distribution, usually Gaussian. Suppose that an observed data point \(\mathbf{x}\) is a high-dimensional random vector, and that latent variable \(\mathbf{z}\) is in a relatively small dimensional space. The log-likelihood of data \(\mathbf{x}\) can be expressed as follows:  
\[ \begin{split}\log p_{\theta}(\mathbf{x})&=\log\int_{ \mathbf{z}}p_{\theta}(\mathbf{x},\mathbf{z})dz\geq\int_{\mathbf{z}}q_{\phi}(\mathbf{z}|\mathbf{x})\log \frac{p_{\theta}(\mathbf{x},\mathbf{z})}{q_{\phi}(\mathbf{z}|\mathbf{x})}dz\\ &=\int_{\mathbf{z}}q_{\phi}(\mathbf{z}|\mathbf{x})\log(\frac{p_{\theta}(\mathbf{ z}|\mathbf{z})p(\mathbf{z})}{q_{\phi}(\mathbf{z}|\mathbf{x})})dz\\ &=E_{\mathbf{z}\sim q_{\phi}(\mathbf{z}|\mathbf{x})}[\log(p_{\theta}(\mathbf{x}| \mathbf{z}))]-KL(q_{\phi}(\mathbf{z}|\mathbf{x})||p(\mathbf{z})):=ELBO,\end{split}  \tag{1}\]  
where \(p_{\theta}(\cdot)\) and \(q_{\phi}(\cdot)\) are the probability density functions of \(\mathbf{x}\) and \(\mathbf{z}\), respectively, and the Kullback-Leibler divergence is \(KL(p||q)=\sum_{x}p(x)\log(\frac{p(x)}{q(x)})\). The right-hand side of the equation above is the lower bound of the log-likelihood of data \(\mathbf{x}\), which is called an evidence lower bound (ELBO). Noticeably, by including inference model \(q_{\phi}(\mathbf{z}|\mathbf{x})\) and  
Figure 1: Variational autoencoder. generative model \(p_{\theta}(\mathbf{x},\mathbf{z})\), ELBO can be expressed as the sum of two terms by encoder \(q_{\phi}(\mathbf{z}|\mathbf{x})\) and decoder \(p_{\theta}(\mathbf{x}|\mathbf{z})\), as shown in Equation (2) and Figure 1. Typically, we assume that \(q_{\phi}(\mathbf{z}|\mathbf{x})\) is Gaussian, \(N(\mathbf{\mu},\sigma^{2}\mathbf{I})\), with dimension \(k\), and that \(p(\mathbf{z})\) is also Gaussian, \(N(\mathbf{0},\mathbf{I})\), with dimension \(k\). Under this assumption, we can rewrite ELBO as follows:  
\[E_{\mathbf{z}\sim q_{\phi}(\mathbf{z}|\mathbf{x})}[\log(p_{\theta}(\mathbf{x}|\mathbf{z}))]+\frac{ k}{2}\log(|\sigma^{2}\mathbf{I}|)+\frac{k}{2}-\frac{1}{2}\mathbf{Tr}(\sigma^{2}\mathbf{I})- \frac{1}{2}\mathbf{\mu}^{\text{T}}\mathbf{\mu}. \tag{2}\]  
Using a neural network model of parameters \(\phi\) and \(\theta\), we estimate the parameters by maximizing ELBO, which is equivalent to maximizing the log-likelihood. The VA model can be applied to language modeling for text generation and text embeddings. To generate a sentence, \(\mathbf{x}\), of length \(T\), a language model generates each token, \(x_{t}\), \(t\geq 1\), conditioned on the previous tokens:  
\[p(\mathbf{x})=\prod_{t=1}^{T}p(x_{t}|\mathbf{x}_{<t}), \tag{3}\]  
where \(x_{t}\) represents the word (or unit) token at position \(t\), and \(\mathbf{x}_{<t}\) represents word tokens before position \(t\). In other words, \(x_{<t}\) means \(x_{1},x_{2},\dots,x_{t-1}\) for \(t>1\) and a given fixed token, <BOS>, for \(t=1\), with \(p(x_{1}|x_{<1})=p(x_{1})\). Using the auto-regressive characteristics of Equation (3), we express ELBO as follows:  
\[E_{\mathbf{z}\sim q_{\phi}(\mathbf{z}|\mathbf{x})}[\sum_{t=1}^{T}\log(p_{\theta}(x_{t}| \mathbf{z},\mathbf{x}_{<t}))]+\frac{1}{2}\log(|\sigma^{2}\mathbf{I}|)+\frac{k}{2}-\frac{1} {2}\mathbf{Tr}(\sigma^{2}\mathbf{I})-\frac{1}{2}\mathbf{\mu}^{\text{T}}\mathbf{\mu} \tag{4}\]  
In a variational autoencoder for language modeling, both an encoder and a decoder use auto-regressive models such as LSTM [4] and GRU [14]. As depicted in Figure 2, decoders are conditioned on latent variables designed to capture global features such as style, topic, and high-level syntactic features [2].