Title: VARIATIONAL AUTOENCODERS FOR TEXT MODELING WITHOUT WEAKENING THE DECODER

Abstract: Previous work (Bowman

1. INTRODUCTION:

Research into generative models for text is an important field in natural language processing (NLP) and various models have been historically proposed. Although supervised learning with recurrent neural networks is the predominant way to construct generative language models (Sutskever et al., 2014; Wu et al., 2017; Vaswani et al., 2017) , auto-regressive word-by-word sequence generation is not good at capturing interpretable representations of text or controlling text generation with global features (Bowman et al., 2015) . In order to generate sentences conditioned on probabilistic latent variables, Bowman et al. (2015) proposed Variational Autoencoders (VAEs) (Kingma & Welling, 2013) for sentences. However, some serious problems that prevent training of the model have been reported.

The problem that has been mainly discussed in previous papers is called "posterior collapse" (van den Oord et al., 2017) . Because decoders for textual VAEs are trained with "teacher forcing" (Williams & Zipser, 1989) , they can be trained to some extent without relying on latent variables. As a result, the KL term of the optimization function (Equation 1) converges to zero and encoder input is ignored (Bowman et al., 2015) . Successful textual VAEs have solved this problem by handicapping the decoder so the model is forced to utilize latent variables (Bowman et al., 2015; Yang et al., 2017) . However, we believe that weakening the capacity of the decoder may lower the quality of generated texts and requires careful hyper-parameter turning to find the proper capacity. Therefore, we take a different approach.

We focus on two overlooked problems. First, previous research fails to address the problem inherent to the structure of VAEs. The fundamental cause of posterior collapse (apart from teacher forcing) is the existence of a suboptimal local minimum for the KL term. Second, although existing models use a LSTM as the encoder, it is known that this simple model is not sufficient for text generation tasks (Bahdanau et al., 2014; Luong et al., 2015; Vaswani et al., 2017) . In this work, we propose a new architecture for textual VAEs with two modifications to solve these problems.

First, we use a multimodal prior distribution and an unimodal posterior distribution to eliminate the explicit minima of ignoring the encoder (Chapter 3.2). Multimodal prior distributions for VAEs have been proposed recently for image and video tasks (Johnson et al., 2016; Dilokthanakul et al., 2016) . Specifically, our model uses a Gaussian Mixture distribution as prior distribution which is trained with the method proposed by Tomczak & Welling (2017) . Second, we modify the encoder (Chapter 3.3). We empirically compare a number of existing encoders and adopt a combination of two. The first is the recently proposed method of embedding text into fixed-size variables using the attention mechanism (Lin et al., 2017) . Although this method was originally proposed for classification tasks, we show this encoder is also effective at text generation tasks. The second is a a Bag-of-Words encoding of input text to help the encoder. It has been reported that a simple Bag-of-Words encoding is effective at embedding the semantic content of a sentence (Pagliardini et al., 2018) . Our experiments show that the modified encoder produces improved results only when other parts of the model are modifed as well to stabilize training. Additionally, our results imply that the self-attention encoder captures grammatical structure and Bag-of-Words captures semantic content.

Finally, to help the model acquire meaningful latent variables without weakening the decoder, we add multi-task learning (Chapter 3.4). We find that a simple sub-task of predicting words included in the text significantly improves the quality of output text. It should be noted that this task does not cause posterior collapse as it does not require teacher forcing.

With these modifications, our model outperforms baselines on BLEU score, showing that generated texts are well conditioned on information from the encoder (Chapter 4.3). Additionally, we show that each component of the multimodal prior distribution captures grammatical or contextual features and improves interpretability of the global features (Chapter 4.5). et al. (2015) is the first work to apply VAEs to language modeling. They identify the problem of posterior collapse for textual VAEs and propose the usage of word dropout and KL annealing. Miao et al. (2015) models text as Bag-of-Words with VAEs. This is part of the motivation behind the usage of Bag-of-Words for textual VAEs. Yang et al. (2017) hypothesize that posterior collapse can be prevented by controlling the capacity of the decoder and propose a model with a dilated CNN decoder which allows changing the effective filter size. Semeniuta et al. (2017) use a deconvolutional layer without teacher forcing to force the model into using information from the encoder.

2. RELATED WORK:



Bowman:

Our use of a multimodal prior distribution is inspired by previous works which try to modify prior distributions of VAEs. Johnson et al. (2016) and Dilokthanakul et al. (2016) apply a VAE with Gaussian Mixture prior distribution to video and clustering, respectively. Tomczak & Welling (2017) propose the construction of a prior distribution from a mixture of posterior distributions of some trainable pseudo-inputs.

Another recent proposal to restrict the latent variables is to use discrete latent variables (Rolfe, 2017; van den Oord et al., 2017) . Some discrete autoencoder models for text modeling has been proposed (Kaiser & Bengio, 2018; Kaiser et al., 2018) . While some results show promise, discretization such as Gumbel-Softmax (Jang et al., 2016) and Vector Quantization (van den Oord et al., 2017) is required to train discrete autoencoders with gradient descent as the gradient of discrete hidden state cannot be calculated directly. A multimodal prior distribution can be regarded as a smoothed autoencoder model with discrete latent variables (Dilokthanakul et al., 2016) without a requirement for discretization.

3. MODEL:

3.1 VARIATIONAL AUTOENCODER FOR TEXT GENERATION

3.1.1. VARIATIONAL AUTOENCODER:

A RNN language model is trained to learn a probability distribution of the next word x t conditioned on all previous words x 1 , x 2 , . . . , x t-1 (Mikolov et al., 2010) . A language model conditioned on a deterministic latent vector z (such as input text representation) has been proposed as well (Sutskever et al., 2014) :

p(x|z) = T t=1 p(x t |x 1 , x 2 , . . . , x t-1 , z)

Although these models can be regarded as a generative model with auto-regressive sampling, they cannot capture interpretable probabilistic structures of global features. Bowman et al. (2015) propose a new language model which explicitly captures probabilistic latent variables of global features with Variational Autoencoders (Kingma & Welling, 2013) .

Variational Autoencoders (VAEs) are one way to construct a generative model based on neural networks, which learns Variational Bayes through gradient decent. A VAE has an encoder q φ (z|x) and a decoder p θ (x|z) each parameterized by a neural network. In many cases, a standard Gaussian distribution is used for the prior distribution of the latent vector p(z) and a Gaussian distribution is used for q φ (z|x). Instead of directly maximizing the intractable marginal probability p(x) = p(z)p θ (x|z)dz, we maximize the evidence lower bound:

log p(x) ≥ E q φ (z|x) [log p θ (x|z)] -KL(q φ (z|x)|p(z))

(1) = L ELBO As the model samples from q φ (z|x), the reparameterization trick (Kingma & Welling, 2013 ) can be used to train the model with gradient descent. Previous work on textual VAEs (Bowman et al., 2015; Yang et al., 2017) simply applied this model to sequence-to-sequence text generation models (Figure 1a ).

3.1.2. PROBLEMS OF VAES FOR TEXT GENERATION:

Recent works (Bowman et al., 2015; Yang et al., 2017) have identified several obstacles for training VAEs for text generation. One of the largest problems, referred to as "posterior collapse" (van den Oord et al., 2017) , is that training textual VAEs often drives the second term of Equation 1 (KL term) close to zero (Bowman et al., 2015) . When the KL term becomes zero, no information from the input text is reflected on latent variables since q φ (z|x) and p(z) are identical. This is an undesirable outcome since latent variables are expected to capture a meaningful representation of input to generate conditional output. However, to aid stabilization, the previous ground truth word is given to the decoder each time during training (teacher forcing (Williams & Zipser, 1989) ). As this technique is applied to textual VAEs as well, a simple language model based on LSTM can be trained without information from the decoder and cause posterior collapse. In order to solve this problem, previous methods try to weaken the decoder to force the model to use information from the encoder. However, weakening the capacity of the decoder is not an ideal strategy since it can lower the quality of generated text and requires additional hyper-parameters specifying decoder capacity. In this paper, we propose three modifications to the model and successfully improve upon textual VAEs without restricting the capacity of the decoder. These modifications are explained in the following chapters: 3.2, 3.3, and 3.4.

3.2. MULTIMODAL PRIOR DISTRIBUTION FOR VAE:

In typical VAEs, a standard normal distribution N (0, 1) is used as the prior distribution p(z) and a normal distribution N (µ, σ 2 ) is used as the posterior distribution q φ (z|x). Although this model is also used for previous textual VAE models (Bowman et al., 2015; Yang et al., 2017) , there is a trivial local minimum p(z) = q φ (z|x) which makes KL(q φ (z|x)|p(z)) in Equation 1zero, manifesting in what is referred to as posterior collapse. Roughly speaking, we can avoid this if q φ (z|x) cannot be identical to p(z). One simple way to achieve this is to use a multimodal distribution as the prior distribution p(z) and an unimodal distribution as the posterior distribution q φ (z|x). This idea is motivated by recently proposed VAE models with a multimodal prior distribution for image and video generation (Johnson et al., 2016; Dilokthanakul et al., 2016) . We provide further explanation in Appendix A and discuss that modification for the decoder is not necessary if the problems in prior distribution is fixed.

The problem with using a multimodal distribution as a prior for VAEs is deciding on what kind of distribution to use. Models which learn a multimodal prior distribution along with other parts of the VAE have been recently proposed (Johnson et al., 2016; Dilokthanakul et al., 2016; Tomczak & Welling, 2017) . One successful model uses a multimodal prior distribution of a variational mixture of posteriors prior (VampPrior) (Tomczak & Welling, 2017) . VampPrior VAEs have multiple trainable pseudo-inputs u k and regard the mixture of the posterior distributions of the pseudo-inputs 1 K K i=1 q φ (z|u k ) as the prior distribution (K is a pre-defined number of pseudo-inputs). Pseudoinputs are trained at the same time as the other components of the VAE. Although pseudo-inputs have the same size as the input image for the VAE in the original work (Tomczak & Welling, 2017) , we use pseudo-inputs which are projected onto µ and σ directly (Figure 2 ).

In our experiments, we find a multimodal prior distribution performs unsupervised clustering and each component of multimodal prior distribution captures specific features of a sentence. Moreover, the components themselves also form clusters, creating a hierarchical structure within the representation space (Chapter 4.5).

3.3. ENCODER:

Existing models of textual VAEs use a simple LSTM as an encoder (Bowman et al., 2015; Yang et al., 2017) . However, recent research into text generation has found that simple LSTMs do not have enough capacity to encode information from the whole text. Motivated by the results of our experiments (Chapter 3.3), we propose concatenating the representation from the self-attention encoder and Bag-of-Words information. Ideally, self-attention encodes grammatical structure and Bag-of-Words encodes overall meaning. Our experiments imply our model is successful in this kind of division of roles (Chapter 4.4).

3.3.1. SELF-ATTENTION ENCODER:

The attention mechanism (Bahdanau et al., 2014; Luong et al., 2015) is a popular model to encode text with LSTMs. Since VAEs are models with fixed size probabilistic latent variables, this mechanism with variable size representation cannot be applied directly. Therefore, we use a recently proposed method called self-attention (Lin et al., 2017 ) (Figure 3 ), an effective model to embed text into a fixed size vector representation for classification tasks using an attention mechanism. Our experiments show that embedded representations from self-attention are useful for text generation.

The self-attention model uses hidden states of bi-directional LSTM h 1 , . . . , h n with variable length. To acquire a fixed sized representation m s , hidden states are summarized with attention weights m s = n i=1 a si h i . Attention weights are calculated by using a weight matrix W 1 with shape d-by-2u (u is the size of a hidden state of bi-directional LSTM and d is a hyper-parameter) and a vector w s2 with size d:

(a s1 , . . . , a sn ) = softmax(w s2 tanh(W 1 H T ))

Here H is a n-by-2u matrix of the hidden states H = (h 1 , . . . , h n ). To get richer information, r different weights (r is a hyper-parameter) are calculated with a r-by-d weight matrix W 2 = (w 12 , . . . , w r2 ) in the model:

A = softmax(W 2 tanh(W 1 H T ))

Here the softmax is performed along the second dimension. Finally, a fixed sized representation is acquired by M = AH. We simply flatten the matrix M into a representation vector. All parameters are trained with gradient descent.

3.3.2. BAG-OF-WORDS INPUT:

Previous research shows the effectiveness of Bag-of-Words in NLP tasks such as text classification (Hill et al., 2016) . Because the difficulty of encoding the content of the input sentence with LSTM is known, we propose using a simple Bag-of-Words input to encode the content of the sentence for text generation tasks. Also, since VAEs are trained in a stochastic manner, it is difficult to train the encoder. Since Bag-of-Words input is much easier to train compared to LSTMs and self-attention encoders, it will help stabilize training. We simply summarize word representation of all words in the input text and project this vector with a linear layer.

3.4. MULTI-TASK LEARNING:

In NLP deep learning tasks, some methods to improve the performance of the main task with multitask learning has been reported. For example, multi-lingual training even improves the result of each language in translation task (Dong et al., 2015) and sub-task of phone recognition improves the result of speech recognition (Toshniwal et al., 2017) . One of the effects of multi-task learning is said that it enables to acquire better intermediate representations (Liu et al., 2015) . Also, a recently proposed model to encode chemical structure with VAEs show that multi-task learning improves the quality of embedded representation (Rafael et al., 2018) .

To address the largest problem of VAEs for text, the difficulty in learning meaningful latent variables, we propose using multi-task learning in our model. However, using additional information such as 

4.2. COMPARISON OF ENCODERS IN NON-VAE TASKS:

We compare a self-attention encoder (Lin et al., 2017) , a LSTM encoder, and a Bag-of-Words encoder with tasks to embed a text into 128 sized vector and show the results in Table 1 . First, we compare the models on a sequence-to-sequence autoencoder model. We show that the self-attention encoder works best in terms of BLEU score. However, we find that the self-attention encoder has a higher false negative rate compared to even a simple LSTM at the task of predicting the words in an input text. From this result, we hypothesize that the self-attention encoder is good at acquiring the structure of a sentence or focusing on specific information but is not good at embedding all the information in a sentence. From these results, we decided to use self-attention and Bag-of-Words for our encoder.

4.3. LANGUAGE MODELING RESULTS:

The results for language modeling are shown in Table 2 . We report the reconstruction loss (negative log likelihood) of text, KL divergence and BLEU of textual VAEs. The results show that multi-task learning and a multimodal prior distribution in isolation both improve the model. On the other hand, changing the encoder in isolation has no influence on results. Note that this is not the case for non-VAE models. However, when multi-task learning is also used, incorporating Bag-of-Words input (the first modification of the encoder) improves the score. Moreover, when we use a multimodal prior distribution, the self-attention encoder, the second modification of the encoder, outperforms the LSTM encoder. This result implies that it is difficult to train the encoder (especially the self-attention encoder) of VAEs unlesss the overall model is improved as well. Therefore, when other parts of the model are improved in tandem and training becomes more stable, the improved ability of the encoder is utilized. Finally, our model with all modifications (the last line) outperforms baselines by a significant margin.

4.4. ANALYSIS OF TWO TYPES OF THE ENCODERS:

Our model uses self-attention and Bag-of-Words as the encoder. We show the results which imply that self-attention acquires grammatical structure and Bag-of-Words provides semantic content. what is the penalty between drugs and battery in ? SA is it true that australia likes war to update their new improve weapons ? BoW definition of traditional education the death penalty and violence in a community output is it true that a good alternative to get a degree of education ? is it possible to death penalty in the world to death penalty ?

Table 3 : Sampling from the posterior distribution of our model when different input is given to the self-attention and Bag-of-Words encoders. "SA" is a sentence given to self-attention encoder and "BoW" is a sentence given to the Bag-of-Words encoder. For details, see Chapter 4.4. For more samples, see Table 7 in Appendix D.

First, to see the relationship between these two encoders, we analyze generated sentences when different sentences are provided to self-attention and Bag-of-Words encoder. We show examples of the results in Table 3 . Generated sentences in Table 3 have similar grammatical structure to the input of the self-attention encoder and nouns in the sentences are strongly affected by the Bag-of-Words encoder.

Moreover, by looking into the attention weights of the self-attention encoder, we can see which parts of a sentence the encoder focuses on as shown by Lin et al. (2017) . We show the maximum attention weight for each word in Figure 4 . We can see that the self-attention encoder assigns a larger weight to words which determine the structure of a sentence such as interrogatives and prepositions rather than nouns. In addition, attention weights are similar between sentences which share grammatical structure even when nouns or word lengths differ.

4.5. INTERPRETABILITY OF MULTIMODAL PRIOR DISTRIBUTION:

We show our model properly acquires a representation of sentences and a multimodal prior distribution helps us interpret acquired representation with unsupervised clustering. By sampling from each component, we can see that our model successfully performs clustering. We find that sentences allocated to to components respectively have one of at least two things in common: grammatical structure or topic. For sentences sampled from components, please see Table 8 in Appendix D. We can see several clusters of components. For further analysis of cluster 1 and 2, see Chapter 4.5, Table 4 .

We show a new method to interpret the global structure of the acquired representation space. We analyze the representation space by visualizing the means of 100 components in the multimodal prior distribution of our model with t-SNE (Maaten & Hinton, 2008) and show the result in Figure 5 .

In addition to the fact that each component clusters together, we now see the clusters themselves form into larger clusters, creating a hierarchical relationship. We take a further look into two clear clusters indicated in Figure 5 . First, we sample from component 38, 56, and 94 in cluster 1 and show the result in Table 4 . From the sampled sentences, we can see that components in cluster 1 share grammatical structure "[interrogative] can I [verb]" and each component has its own topics (computer, politics, culture). On the other hand, components in cluster 2 share the topics (politics or human relationship) and each component has its own grammatical structure. Also, from Figure 5 , components 52, 31, and 37 seem to be on the circle in this order and we can see the continuous changes of grammatical structure in this order. Thus, we can observe that our model acquires a hierarchical structure of sentences and the structure can be easily interpreted through analysis of components in the multimodal prior distribution.

As models with multimodal distributions are relatively new, we hope methods to control multimodal prior distribution are investigated further in future works. However, we emphasize that our result is already impressive since without a multiomodal prior, extensive search with sampling or additional labels is required to interpret the structure of acquired text representation. A multimodal prior distribution makes it much easier to understand the structure of the representation space though analysis of components of the distribution. Table 4 : Samples from components of the prior distribution from cluster 1 (above) and 2 (below) in Figure 5 . Components in cluster 1 share grammatical structure and components in cluster 2 share topics. Please see Chapter 4.5 for more details.

5. CONCLUSION:

and increases hyper-parameters. We show (i) multimodal prior distribution, (ii) improvement of the encoder and (iii) multi-task learning can improve the model with a simple LSTM decoder.

A THEORETICAL ANALYSIS OF MULTIMODAL PRIOR DISTRIBUTION:

We show theoretical justification for a multimodal prior distribution as a solution for posterior collapse. We use the equivalent objective for ELBO (Equation 1) by Zhao et al. (2017) :

EQUATION

where p D (x) is the data distribution, p θ (x) is the marginal distribution p θ (x) = p(z)p θ (x|z)dz, and

C is E p D (x) [log p D (x)],

which does not depend on any parameters. This objective can be minimized to zero without utilizing latent variables under the assumption that (i) the decoder is sufficiently flexible and (ii) the posterior distribution can be trained so p(z) = q φ (z|x) (Zhao et al., 2017) . This nature of ELBO causes posterior collapse in VAEs. There are two simple ways to break these assumptions.

First, if the capacity of the decoder is restricted, assumption (i) cannot be satisfied. This is the theoretical underpinning for previous approaches used in textual VAEs (Bowman et al., 2015; Yang et al., 2017) which restrict the capacity of the decoder. However, as previously discussed, weakening the decoder is undesirable. Additionally, hyper-parameter search is required to strike a balance between the two terms if the KL term is not modified as well.

Therefore, we propose to break the assumption (ii) with a multimodal prior distribution. When the prior distribution p(z) is a multimodal distribution and the posterior distribution q φ (z|x) is an unimodal distribution, there is no way to satisfy p(z) = q φ (z|x). Moreover, there will be multiple minima for KL(q φ (z|x)|p(z)). When Kullback-Leibler divergence KL(q|p) between the Gaussian mixture distribution p and the normal distribution q is minimized (here we assume that q is trainable), q will be allocated to one component of p since KL(q|p) = q(z) log q(z) p(z) dz becomes larger when p is assigned a low probability in an area where q is assigned a high probability (Figure 6 ). In such a formulation, there is no clear global minima for the KL term and the posterior distribution is not forced to ignore information from the encoder.

We propose a hypothesis that the modification of the decoder is not necessary if multimodal prior distribution is used. In practice, it is natural to assume that training the decoder so p D (x) = p θ (x|z) for all z is much harder than to make KL(q φ (z|x)|p(z)) = 0. Under the assumption, the model will be trained so KL(q φ (z|x)|p(z)) = 0 as the first step and this condition force the decoder to be trained so p D (x) = p θ (x|z) for all z when there is no modification of the model. Although this is the opposite way from the explanation by Zhao et al. (2017) , this process is more natural in practice since it is easy to train prior distribution. Therefore, if we modify the model to avoid KL(q φ (z|x)|p(z)) = 0, the decoder will not try to satisfy p D (x) = p θ (x|z) for all z but learn the conditioned distribution for each z. Government) . As test and validation dataset, we use 10,000 sentences each. We set the maximum length of a sentence to 60 words (ignore the rest of the sentence, the average length of the original sentences is 38.12 words) and use the most common 40,000 words for this experiment.

B.2 MODEL CONFIGURATIONS:

Our model uses self-attention and Bag-of-Words in the encoder and a LSTM for the decoder. The size of the hidden state of LSTM is 256 for both for LSTM and self-attention. The size of the word embedding is 256 and the size of the latent variables is 128. For the self-attention encoder, we use d = 350 and r = 30. In accordance with (Bowman et al., 2015; Yang et al., 2017) , we feed the latent variables on every step of the decoder LSTM by concatenating it with the word embedding. We applied 0.4 word dropout for input text to the decoder for our model and the model from Bowman et al. (2015) . In this paper, we modify the model without restricting the capacity of the decoder. However, the method used by Bowman et al. (2015) called word dropout, which was originally proposed to weaken the decoder, is now seen as a method of smoothing (Xie et al., 2017) . As this method is also effective and harmless for non-VAE text generation task, we use word dropout for our model. In addition, we pretrain the encoder and the decoder with sequence-to-sequence text generation for our multi-prior distribution model. Note that it was impossible to pretain decoders for previous models since it can result in posterior collapse.

For multi-prior distribution, we compare 4 numbers of components [50, 100, 500, 2000] and found that performance is not sensitive to this hyperparameter, although a larger number of components results in a slightly better score (Table 5 ). As using a prior distribution with many components leads to overfitting, over-regularization, and high computational complexity (Tomczak & Welling, 2017), we report the score of a prior distribution with 500 components and analyze the acquired representation space with 100 components for ease of analysis.

We compare our model with two models proposed by Bowman et al. (2015) and Yang et al. (2017) . Basically, we use the same configurations for these models. For the model of Yang et al. (2017) , we use the SCMM-VAE model in the original paper and pretrain the encoder.

We use Adam (Kingma & Ba, 2014) for the optimizer. According to our experiments, setting the learning rate to 5 × 10 -4 and β 1 to 0.5 performs the best. For KL weight annealing, we set the initial weight for the KL term to be 0 and increase it linearly to 1 until epoch 30. After KL weight annealing, we train for 80 epochs with learning rate decay (0.95 for every epoch). , 2015) , they denotes the LSTM initialized with an autoencoder and a language model. The methods of semisupervised learning with VAEs use the same scheme as (Yang et al., 2017) . LSTM is a simple supervised model.

The structure of semi-supervised models using VAEs is taken from Yang et al. (2017) . We use the topic of a sentence from the dataset as a label and feed the encoded representation from the encoder to the discriminator. We report the results of semi-supervised learning in Table 6 . Our models do not differ from semi-supervised learning baselines. This result can be understood because this semi-supervised learning assumes that label information is helpful or necessary to generate proper sentences. Our experiments show that our model both is conditioned by the encoder and also generates proper sentences without labels. This is consistent with the reasoning from Yang et al.

(2017) that the best models for language modeling and semi-supervised learning are different.

D ADDITIONAL SAMPLING RESULTS:



D.1 QUALITATIVE ANALYSIS OF TWO TYPES OF ENCODERS:

We show additional samples for Table 3 what happens to the death penalty for a day ? SA is it true that australia likes war to update their new improve weapons ? BoW definition of traditional education the death penalty and violence in a community is it true that a good alternative to get a degree of education ? is it possible to death penalty in the world to death penalty ? is it true that the age of people to change lots of traditional ? is there a place to get a peaceful death penalty in the world ? is it a good idea of having a 100 % of education ? are there any place in the city to make a death penalty ? Table 7 : Sampling from posterior distribution of our model when different texts are input to selfattention and Bag-of-Words of the encoder. "SA" is a sentence given to self-attention encoder and "BoW" is a sentence to Bag-of-Words encoder. For detail, see Chapter 4.4. 1 is it true that the only way to provide the holy rabbit through the same answer ? is it possible to go to a police officer ? is it possible to have to pay for her home 22 does any body really work with their own marketing cards ? does anyone have a good website ? does anyone know how to get rid of them ? 76 what is the best way to get money from this year ? what is the best way to download a satellite background ? what is the best way to keep my boyfriend ? 60 who will win the world cup in france ? where was the first place in the nba ? is anyone proud of the world cup <UNK>? 68 i can not put my script in the java script , how do i format the <UNK>? my script is not working ! windows media player ? does anyone have a good thing to learn english ? how do i search ? 80 do n't you think the president is the worst president of the us is the liberal , the jewish religion will be cut the food to do ? who is the president of the united states ? Table 8 : Samples from components of prior distribution. Component 1, 22, and 76 generate sentences with common structure. On the other hand, component 60, 68, and 83 generate structurally diverse sentences on the same topics (computer, sports). <UNK>is a word not in the dictionary. For detail, see Chapter 4.5.

D.2 COMPONENTS OF MULTI-PRIOR DISTRIBUTION:

We report text from 6 components of a multimodal prior distribution from our model in Table 8 . We found two types of features allocated for components. The first one is grammatical structure. Components 1, 22, and 77 in Table 8 each generate similarly structured sentences: sentences from component 1 begin with "it is true that" or "it is possible to", sentences from component 22 begin with "does anyone (anybody)", and sentences from component 77 begin with "what is the best way to". This result is straightforward to interpret as properly acquiring grammatical structure will lower reconstruction loss. More interestingly, sentences generated the next type of components, namely components 60, 68, and 83 are each on the same topic. Sentences generated from component 60 are about sports, those from component 68 are about computer (music), and those from component 83 are about politics. However, these sentences do not share grammatical structure and generate sentences with diverse structures.

:



Figures and Tables:

(a) The overall architecture of existing models. (b) The overall architecture of our model. In the encoder, hidden states of the self-attention Encoder and BoW are concatenated. The decoder estimates BoW of the input text from the latent variables as a sub-task in addition to generating text. In our model, the prior distribution of the latent variables is a Gaussian mixture model.

Figure 1: The overall architecture of existing models and our model.

Figure 2: VampPrior VAE. h is a representation of input text from the encoder. u 1 , . . . , u K are pseudo-inputs. This model is our modified version of VampPrior VAE. The red and green arrows represent neural networks with shared weights.

Figure 3: A self-attention encoder. This model encodes variable length input into a fixed length representation using an attention mechanism. The fixed length representation is acquired by summing up the hidden states of the bi-directional LSTM based on attention weights. Attention weights a s1 , . . . , a sn are calculated by (a s1 , . . . , a sn ) = softmax(w s2 tanh(W 1 H T )).

Figure 4: Visualized attention weight of the self-attention encoder. We show the maximum attention weight for each word. Darker red represents a larger attention weight. Interrogatives and prepositions are assigned larger weights compared to nouns.

Figure 5: The mean of each component of the multimodal prior distribution, visualized with t-SNE.We can see several clusters of components. For further analysis of cluster 1 and 2, see Chapter 4.5, Table4.

Figure6: This Gaussian mixture distribution (blue line) has two equally weighted components with means 2.0 and -2.0, and variance 1.0. The distribution (green line) to the left is N (2.0, 0.3) and to the right is N (0, 5). The Kullback-Leibler divergence KL(q|p) between Gaussian mixture distribution p and normal distribution q of the left image is 1.4, and the right image is 12.4.

A comparison of encoder models. BLEU corresponds to the BLEU scores of non-VAE sequence generation tasks. FN corresponds to the false negative rate for the prediction task of words in input text. SA denotes self-attention Encoder and BoW denotes Bag-of-Words Encoder. grammatical properties or labels is not desirable for language modeling with textual VAEs. We find that the simple task of predicting words in output text can help the model improve the quality of output text. Additionally, this sub-task will alleviate the problem of posterior collapse since it does not contain auto-regressive structure which in turn requires training with teacher forcing.We compare our model with two models proposed byBowman et al. (2015) andYang et al. (2017). Basically, we use the same configurations for these models. For the model ofYang et al. (2017), we use a SCMM-VAE model in the original paper and pretrain the encoder. For the multimodal prior distribution model, we report the score of a prior distribution with 500 components and analyze the acquired representation space with one with 100 components for ease of analysis. We use 100,000 sentences from a scale document dataset "Yahoo! Answers Comprehensive Questions and Answers version 1.0" for training to acquire the results. For details of the dataset and model parameters, see Appendix B.

Table content: <table><tr><td>model</td><td>SA</td><td colspan="2">LSTM BoW SA+BoW</td></tr><tr><td colspan="3">BLEU 25.11 23.48 22.08</td><td>34.48</td></tr><tr><td>FN</td><td colspan="2">69.74 51.35 34.55</td><td>28.62</td></tr><tr><td>4 EXPERIMENTS</td><td/><td/></tr><tr><td>4.1 SETTINGS</td><td/><td/></tr></table>

Language modeling results. SA denotes self-attention and DCNN denotes Dilated CNN. BoW is Bag-of-Words input and MT is multi task learning (Bag-of-Words prediction task). The model of the first row is(Bowman et al., 2015), and the model of the second row is(Yang et al., 2017).

Table content: <table><tr><td/><td colspan="5">Encoder BoW Decoder MT Prior</td><td>Text</td><td>KLD BLEU</td></tr><tr><td/><td>LSTM</td><td/><td>LSTM</td><td/><td>Uni</td><td colspan="2">218.92 15.28</td><td>12.98</td></tr><tr><td/><td>LSTM</td><td/><td>DCNN</td><td/><td>Uni</td><td colspan="2">276.58 12.64</td><td>11.92</td></tr><tr><td/><td>SA</td><td/><td>LSTM</td><td/><td>Uni</td><td colspan="2">218.04 15.46</td><td>13.13</td></tr><tr><td/><td>SA</td><td>o</td><td>LSTM</td><td/><td>Uni</td><td colspan="2">214.93 16.92</td><td>12.95</td></tr><tr><td/><td>LSTM</td><td/><td>LSTM</td><td>o</td><td>Uni</td><td colspan="2">216.05 30.39</td><td>15.76</td></tr><tr><td/><td>SA</td><td/><td>LSTM</td><td>o</td><td>Uni</td><td colspan="2">213.15 30.71</td><td>15.65</td></tr><tr><td/><td>LSTM</td><td>o</td><td>LSTM</td><td>o</td><td>Uni</td><td colspan="2">200.60 41.73</td><td>17.89</td></tr><tr><td/><td>SA</td><td>o</td><td>LSTM</td><td>o</td><td>Uni</td><td colspan="2">203.92 41.12</td><td>17.69</td></tr><tr><td/><td>LSTM</td><td/><td>LSTM</td><td/><td colspan="3">Multi 196.02 24.51</td><td>13.84</td></tr><tr><td/><td>LSTM</td><td/><td>DCNN</td><td/><td colspan="3">Multi 214.40 27.08</td><td>13.99</td></tr><tr><td/><td>SA</td><td/><td>LSTM</td><td/><td colspan="3">Multi 208.31 18.21</td><td>13.51</td></tr><tr><td/><td>LSTM</td><td>o</td><td>LSTM</td><td>o</td><td colspan="3">Multi 203.72 41.01</td><td>18.28</td></tr><tr><td/><td>SA</td><td>o</td><td>LSTM</td><td>o</td><td colspan="3">Multi 193.89 48.99</td><td>19.20</td></tr><tr><td>SA</td><td colspan="6">what is the importance of computer in data processing ?</td></tr><tr><td colspan="5">BoW definition of traditional education</td><td/><td colspan="2">the death penalty and violence in a com-</td></tr><tr><td/><td/><td/><td/><td/><td/><td>munity</td></tr><tr><td colspan="6">output what are the definition of environmental</td><td/></tr><tr><td/><td colspan="3">education for education ?</td><td/><td/><td/></tr></table>

Comparison of BLEU scores from multimodal prior distribution model with different numbers of components.

Table content: <table><tr><td colspan="4">Encoder BoW Decoder MT</td><td>50</td><td>100</td><td>500 2000</td></tr><tr><td>LSTM</td><td>o</td><td>LSTM</td><td>o</td><td colspan="2">17.98 18.24 18.28 18.51</td></tr><tr><td>SA</td><td>o</td><td>LSTM</td><td>o</td><td colspan="2">18.79 18.74 19.20 19.32</td></tr></table>

This analysis motivates us to modify textual VAE without weakening the capacity of the decoder. The results of our experiments are consistent with this hypothesis.

Table content: <table><tr><td>B EXPERIMENTS SETTING</td></tr><tr><td>B.1 DATASET</td></tr><tr><td>We use the large scale document dataset "Yahoo! Answers Comprehensive Questions and Answers</td></tr><tr><td>version 1.0". From the Yahoo! Answer dataset, we pick up 100,000 sentences randomly from 10</td></tr><tr><td>topics (Society &amp; Culture, Science &amp; Mathematics, Health, Education &amp; Reference, Computers &amp;</td></tr><tr><td>Internet, Sports, Business &amp; Finance, Entertainment &amp; Music, Family &amp; Relationships, Politics &amp;</td></tr></table>

Semi-supervised learning. LM-LSTM and SA-LSTM come from (Dai & Le

Table content: <table><tr><td colspan="3">C SEMI-SUPERVISED LEARNING</td><td/><td/><td/><td/></tr><tr><td>Model</td><td colspan="6">Encoder BoW Decoder MT Prior 100</td><td>500</td></tr><tr><td>LSTM</td><td>LSTM</td><td/><td>-</td><td/><td>-</td><td colspan="2">12.10 18.50</td></tr><tr><td>LM-LSTM</td><td>LSTM</td><td/><td>-</td><td/><td>-</td><td colspan="2">31.21 41.51</td></tr><tr><td>SA-LSTM</td><td>LSTM</td><td/><td>-</td><td/><td>-</td><td colspan="2">20.38 36.55</td></tr><tr><td/><td>LSTM</td><td/><td>LSTM</td><td/><td>Uni</td><td colspan="2">38.90 53.25</td></tr><tr><td>VAE</td><td>LSTM SA</td><td>o</td><td>DCNN LSTM</td><td>o</td><td>Uni Uni</td><td colspan="2">23.19 50.14 38.84 54.69</td></tr><tr><td/><td>SA</td><td>o</td><td>LSTM</td><td>o</td><td colspan="3">Multi 34.99 54.20</td></tr></table>

in Table7. Please see Chapter 4.4 for detailed explanation.

Table content: <table><tr><td>SA</td><td colspan="2">what is the importance of computer in data processing ?</td></tr><tr><td colspan="2">BoW definition of traditional education</td><td>the death penalty and violence in a com-</td></tr><tr><td/><td/><td>munity</td></tr><tr><td/><td>what are the definition of environmental</td><td>what is the penalty between drugs and</td></tr><tr><td/><td>education for education ?</td><td>battery in ?</td></tr><tr><td/><td>what does the definition of a consumer</td><td>what is the pros and cons in a deadly</td></tr><tr><td/><td>solution ?</td><td>nation ?</td></tr><tr><td/><td>how are your definition of education</td></tr><tr><td/><td>system ?</td></tr></table>

Peer review (iclr2019):
 This paper focuses on applying variational autoencoders to text data. A well-known problem in this domain is posterior collapse, where the posterior "collapses" to an uninformative prior distribution and the decoder becomes a standard language model which does not utilize the latent code. This paper assembles a few tricks to improve VAEs for text, including using a self-attention-based encoder, a mixture-of-Gaussians prior on the latent code, and a "multi-task" loss which attempts to reconstruct a BoW representation in addition to the original sentence. Some experiments are carried out on the Yelp dataset, including some qualitative study. The issue of posterior collapse is important and modeling the global structure of text with a latent variable is a valuable endeavor. This paper provides some promising results in that direction. My main issue with the paper is that it is primarily the combination of a few pre-existing tricks to the text-VAE problem. Bringing together multiple tricks and showing that they can enable significant progress is potentially valuable, but only if the results are obviously better or they represent a big step forward in an important application (for example, showing convincing sentence interpolations, attribute vector arithmetic, autoencoding of very long text sequences, etc). This paper does not demonstrate any sufficiently strong result; instead, it just provides some limited evaluation that things get a little better along with some basic qualitative analysis of potential benefits of using a mixture prior. Apart from this, the paper claims that combining these tricks avoids "weakening the decoder" but uses a decoder with word dropout in all experiments, and also claims they decrease the number of hyperparameters, which is not shown in any rigorous way. The proposed additions also are described in insufficient detail - how are the bag of word representations constructed? How are they predicted? What is the additional loss term? How is it combined to the standard reconstruction loss for the autoregressive decoder? Etc. Finally, it is missing discussion (and comparison, as baselines) to a great deal of prior work on avoiding posterior collapse. As-is, the paper is not strong enough for publication. Below, I go into a bit more detail about specific issues and suggestions. Specific comments: - The paper blames posterior collapse on teacher forcing ("Because decoders for textual VAEs are trained with “teacher forcing” (Williams & Zipser, 1989), they can be trained to some extent without relying on latent variables.") This is a misleading claim. Even with teacher forcing, it could potentially be advantageous to the model to utilize the latent (recall that the latent can capture information about the entire input sequence, so p(x_t | x_1, ... x_{t - 1}) is less informative than p(x_t | x_1, ... x_{t - 1})), but because of the architecture/learning dynamics/hyperparameters/etc the model tends to choose to "ignore" z. This is an independent problem of teacher forcing. The main issue caused by teacher forcing is exposure bias, which is a related but different issue. Teacher forcing is not necessary for training latent variable models with autoregressive decoders, it just happens to work well because it*s a remarkably good search strategy. - "existing models use a LSTM as the encoder, it is known that this simple model is not sufficient for text generation tasks (Bahdanau et al., 2014; Luong et al., 2015; Vaswani et al., 2017)." I assume you are citing these papers because they deal with attention and show that attention improves sequence-to-sequence models. The fact that attention helps is because the decoder is provided with additional conditioning information, not because the encoder is different (in Bahdanau and Luong the decoder remains a simple LSTM). - There are many missing related works (and baselines) for avoiding posterior collapse, such as "Avoiding Latent Variable Collapse with Generative Skip Models", "Adversarially Regularized Autoencoders", "Variational Lossy Autoencoder", etc. - Your equation for p(x|z) at the beginning of 3.1.1 is a bit odd because you are talking about a deterministic z, and using it to point out that z might not learn good global features. However, when z is a stochastic latent variable the equation would be exactly the same. Writing it in this way does not make this distinction clear. A deterministic z can indeed capture interpretable global structure (see Figure 2 of Sutskever et al. 2014). If you want to claim that using a stochastic z may _better_ capture global structure, I would recommend making this argument from an information theoretic perspective (as was done in "Variaional Lossy Autoencoder", for example). - That weaking the decoder "requires additional hyper-parameters specifying decoder capacity" is not really a compelling arugment in your paper since your approach also requires additional hyperparameters (number of mixture components, coefficients for additional loss terms, etc.) - You call posterior collapse a "trivial local minimum". Are you sure it*s a local minimum? This implies that the gradient of the loss w.r.t. every one of the parameters is zero at this point. I think you are using "local minimum" in a loose sense; please don*t do this. - What are pseudo-inputs? Do you basically mean they are model parameters which are updated w.r.t. the loss function? - "Recent research into text generation has found that simple LSTMs do not have enough capacity to encode information from the whole text." Citation needed! - I think what you are calling "self-attention" is actually the "feed-forward attention mechanism" from "Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems". - "We simply summarize word representation of all words in the input text and project this vector with a linear layer." I don*t think this adequately describes what you are doing. How is this summarization done? Via a histogram of the words which are present? An average of the words* embedding vectors? Same with the multi-task part. What exactly are you predicting? How? - I presume your BLEU scores in Table 1 are reflecting the BLEU score for the input vs. the reconstructed output. Please clarify. - In the appendix it is written that "We applied 0.4 word dropout for input text to the decoder for our model and the model from Bowman et al. (2015)." This is definitely "weakening the decoder" - that is the primary motivation for word dropout. I don*t think you can claim your approach can avoid weakening the decoder if all of your results are with word dropout.