# Informative Language Encoding by Variational Autoencoders Using Transformer
## 2 Preliminaries
However, a variational autoencoder model using an auto-regressive decoder easily falls into local optima, which is called a posterior collapse or KL vanishing [2; 7]. This phenomenon occurs because the model training implemented to maximize ELBO often finishes at an early step, with latent variables \(\mathbf{z}\) failing to contain meaningful information about input sequences and the KL divergence term falling to zero. When reconstructing \(x_{t}\) with the latent variable \(\mathbf{z}\) from such a training process, the model just relies on \(\mathbf{x}_{<t}\), as depicted in Figure 3 by the red line [7]. Figure 3: Illustration of a posterior collapse. Figure 2: Illustration of a variational autoencoder for language modeling. <BOS> is the beginning of string tokens, and <EOS> is the end of string tokens  
To alleviate this phenomenon, numerous techniques such as updating inference networks and generative networks in imbalanced ways [7], refactoring loss functions [8], changing loss functions [9], and KL annealing [2,10] have been proposed. Because of its fast convergence coupled with low hardware capacity, we adhere to KL annealing techniques for our experiment in this research work [2,10]. **Transformer. ** Transformer is a sequence-to-sequence model that removes recurrence and adopts self-attention [13]. Transformer architectures have become the state of the art in various NLP tasks such as translation, summarization, and classification. Moreover, the architecture is used in various pre-trained language models such as BERT [12] and GPT [11], which have accomplished top performance in numerous NLP tasks. Self-attention and its multi-head version are crucial in Transformer. When a sentence is injected into the Transformer architecture and then subsequently passed to the embedding layer, it is summed with positional encoding, which gives position information to the output of the embedding layer. Next, the output is projected to multiple sets of query, key, and value. Attention includes a mapping from a query (Q), along with a key (K) and a value (V), to an output as a weighted sum of values in which the weight, also called the attention score, is the similarity between query and key. The similarity is a scaled dot product between query and key, normalized by the dimension of key, \(d_{k}\), as in Equation (5), where \(Q\in\mathbb{R}^{s_{1}\times d_{k}},K\in\mathbb{R}^{s_{2}\times d_{k}},V\in \mathbb{R}^{s_{2}\times d_{v}}\), \(s_{1}\) and \(s_{2}\) are the lengths of the sequence in query and key (or value), respectively, and \(d_{q}\), \(d_{k}\), and \(d_{v}\) are the dimensions of query, key, and value, respectively:  
\[Attention(Q,K,V)=Softmax(QK^{T}/\sqrt{d_{k}})V. \tag{5}\]  
As an extension of attention, self-attention is an attention mechanism in which the query, key, and value are from same source. In addition, the Transformer mechanism adopts a multi-head attention, a mechanism which projects query, key, and value into multiple and different spaces and applies attention functions, as shown in Figure 4. Equation (6) shows its process as a concatenation of multiple attention outputs, in which the linear projections are \(W_{i}^{Q}\in\mathbb{R}^{d_{model}\times d_{q}},W_{i}^{K}\in\mathbb{R}^{d_{model} \times d_{k}},W_{i}^{V}\in\mathbb{R}^{d_{model}\times d_{q}}\), and \(W^{o}\in\mathbb{R}^{hd_{c}\times d_{model}}\), where \(d_{model}\) is the dimension of the model and \(h\) is the number of heads. Multi-head attention brings forth various views of query, key, and value. The total number of heads, which is a hyperparameter, is often set in practice by an associated performance measure. \[ \begin{array}{rl}Multi-Head(Q,K,V)&=Concat(head_{(1)},\ldots,head_{(h)})W^{o},\\ head_{i}&=Attention(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V}).\end{array}  \tag{6}\]  
The architecture includes an encoder composed of two sub-layers, a multi-head self-attention layer, and a feed-forward neural network layer.