Venue: NAACL 2019

# Topic-Guided Variational Autoencoders for Text Generation

Wenlin Wang\({}^{1}\), Zhe Gan\({}^{2}\), Hongteng Xu\({}^{1,3}\), Ruiyi Zhang\({}^{1}\), Guoyin Wang\({}^{1}\), Dinghan Shen\({}^{1}\), Changyou Chen\({}^{4}\), Lawrence Carin\({}^{1}\)

\({}^{1}\)Duke University, \({}^{2}\)Microsoft Dynamics 365 AI Research,

\({}^{3}\)Infinia ML, Inc, \({}^{4}\)University at Buffalo

{wenlin.wang, ruiyi.zhang, gw60, ds448, lcarin}@duke.edu zhe.gan@microsoft.com, hongteng.xu@infiniaml.com changyou@buffalo.edu

###### Abstract

We propose a topic-guided variational autoencoder (TGVAE) model for text generation. Distinct from existing variational autoencoder (VAE) based approaches, which assume a simple Gaussian prior for the latent code, our model specifies the prior as a Gaussian mixture model (GMM) parametrized by a neural topic module. Each mixture component corresponds to a latent topic, which provides guidance to generate sentences under the topic. The neural topic module and the VAE-based neural sequence module in our model are learned jointly. In particular, a sequence of invertible Householder transformations is applied to endow the approximate posterior of the latent code with high flexibility during model inference. Experimental results show that our TGVAE outperforms alternative approaches on both unconditional and conditional text generation, which can generate semantically-meaningful sentences with various topics.

## 1 Introduction

Text generation plays an important role in various natural language processing (NLP) applications, such as machine translation Cho et al. (2014); Sutskever et al. (2014), dialogue generation Li et al. (2017a), and text summarization Nallapati et al. (2016); Rush et al. (2015). As a competitive solution to this task, the variational autoencoder (VAE) Kingma and Welling (2013); Rezende et al. (2014) has been widely used in text-generation systems Bowman et al. (2015); Hu et al. (2017); Serban et al. (2017). In particular, VAE defines a generative model that propagates latent codes drawn from a simple prior through a decoder to manifest data samples. The generative model is further augmented with an inference network, that feeds observed data samples through an encoder to yield a distribution on the corresponding latent codes.

Compared with other potential methods, \(e.g.\), those based on generative adversarial networks (GANs) Yu et al. (2017); Guo et al. (2017); Zhang et al. (2017b, 2018); Chen et al. (2018), VAE is of particular interest when one desires not only text generation, but also the capacity to infer meaningful latent codes from text. Ideally, semantically-meaningful latent codes can provide high-level guidance while generating sentences. For example, when generating text, the vocabulary could potentially be narrowed down if the input latent code corresponds to a certain topic (_e.g._, the word "military" is unlikely to appear in a sports-related document).

However, in practice this desirable property is not fully achieved by existing VAE-based text generative models, because of the following two challenges. First, the sentences in documents may associate with different semantic information (_e.g._, topic, sentiment, etc.) while the latent codes of existing VAE-based text generative models often employ a simple Gaussian prior, which cannot indicate the semantic structure among sentences and may reduce the generative power of the decoder. Although some variants of VAE try to impose some structure on the latent codes Jiang et al. (2016); Dilothanakul et al. (2016), they are often designed with pre-defined parameter settings without incorporating semantic meanings into the latent codes, which may lead to over-regularization Dilokthanakul et al. (2016).

The second issue associated with VAE-based text generation is "posterior collapse," first identified in Bowman et al. (2015). With a strong auto-regressive decoder network (_e.g._, LSTM), the model tends to ignore the information from the latent code and merely depends on previous generated tokens for prediction. Several strategies are proposed to mitigate this problem, including making the decoder network less auto-regressive (\(i.e.\)

using less conditional information while generating each word) (Yang et al., 2017; Shen et al., 2017a), or bridging the amortization gap (between the log-likelihood and the ELBO) using semi-amortized inference networks (Kim et al., 2018). However, these methods mitigate the issue by weakening the conditional dependency on the decoder, which may fail to generate high-quality continuous sentences.

To overcome the two problems mentioned above, we propose a topic-guided variational autoencoder (TGVAE) model, permitting text generation with designated topic guidance. As illustrated in Figure 1(a), TGVAE specifies a Gaussian mixture model (GMM) as the prior of the latent code, where each mixture component corresponds to a topic. The GMM is learnable based on a neural topic model -- the mean and diagonal covariance of each mixture component is parameterized by the corresponding topic. Accordingly, the degree to which each component of the GMM is used to generate the latent code and the corresponding sentence is tied to the usage of the topics. In the inference phase, we initialize the latent code from a GMM generated via the encoder, and apply the invertiable Householder transformation (Bischof and Sun, 1994; Sun and Bischof, 1995) to derive the latent code with high flexibility and low complexity.

As shown in Figure 1(b), besides unconditional text generation, the proposed model can be extended for conditional text generation, _i.e.,_ abstractive text summarization (Nallapati et al., 2016) with an attention module. By injecting the topics learned by our model (semantic information), we are able to make better use of the source document and improve a sequence-to-sequence summarization model (Sutskever et al., 2014).

We highlight the contributions of our model as follows: (\(i\)) A new Topic-Guided VAE (TGVAE) model is proposed for text generation with designated topic guidance. (\(ii\)) For the model inference, Householder flow is introduced to transform a relatively simple mixture distribution into an arbitrarily flexible approximate posterior, achieving powerful approximate posterior inference. (\(iii\)) Experiments for both unconditional and conditional text generation demonstrate the effectiveness of the proposed approach.

## 2 Model

The proposed TGVAE, as illustrated in Figure 1(a), consists of two modules: a neural topic model (NTM) and a neural sequence model (NSM). The NTM aims to capture long-range semantic meaning across the document, while the NSM is designed to generate a sentence with designated topic guidance.

### Neural Topic Model

Let \({\mathbf{d}}\in\mathbb{Z}^{D}_{+}\) denote the bag-of-words representation of a document, with \(\mathbb{Z}_{+}\) denoting non-negative integers. \(D\) is the vocabulary size, and each element of \({\mathbf{d}}\) reflects a count of the number of times the corresponding word occurs in the document. Let \(a_{n}\) represent the topic assignment for word \(w_{n}\). Following Miao et al. (2017), a Gaussian random vector is passed through a softmax function to parameterize the multinomial document topic distributions. Specifically, the generative process

Figure 1: Illustration of the proposed Topic-Guided Variational Autoencoder (TGVAE) for text generation. (a) For generation (the red arrows), the topics inferred from a neural topic model are used to guide a Gaussian mixture prior of the latent code, which is further fed into the decoder to generate a sentence. For inference (the black arrows), the sentence is encoded into a vector and then propagated through the Householder flow to obtain the approximate posterior. (b) An attention module is further added for text summarization. The same neural topic model is also applied, but omitted here for simplicity of illustration. “LT” denotes a linear transformation.

 

of the NTM is

\[\mathbf{\theta} \sim\mathcal{N}(0,\mathbf{I}), \quad\mathbf{t}=g(\mathbf{\theta})\,, \tag{1}\] \[a_{n} \sim\text{Discrete}(\mathbf{t}), w_{n} \sim\text{Discrete}(\mathbf{\beta}_{a_{n}})\,,\]

where \(\mathcal{N}(0,\mathbf{I})\) is an isotropic Gaussian distribution, \(g(\cdot)\) is a transformation function that maps sample \(\mathbf{\theta}\) to the topic embedding \(\mathbf{t}\), defined here as \(g(\mathbf{\theta})=\text{softmax}(\hat{\mathbf{W}}\mathbf{\theta}+\hat{\mathbf{b}})\), where \(\hat{\mathbf{W}}\) and \(\hat{\mathbf{b}}\) are trainable parameters; \(\mathbf{\beta}_{a_{n}}\) represents the distribution over words for topic \(a_{n}\); \(n\in[1,N_{d}]\), and \(N_{d}\) is the number of words in the document. The marginal likelihood for document \(\mathbf{d}\) is:

\[p(\mathbf{d}|\mathbf{\beta}) =\int_{\mathbf{t}}p(\mathbf{t})\prod_{n}\sum_{a_{n}}p(w_{n}|\mathbf{\beta}_{a _{n}})p(a_{n}|\mathbf{t})dt \tag{2}\] \[=\int_{\mathbf{t}}p(\mathbf{t})\prod_{n}p(w_{n}|\mathbf{\beta},\mathbf{t})dt\] \[=\int_{\mathbf{t}}p(\mathbf{t})p(\mathbf{d}|\mathbf{\beta},\mathbf{t})d\mathbf{t}=\int_{ \mathbf{\theta}}p(\mathbf{\theta})p(\mathbf{d}|\mathbf{\beta},\mathbf{\theta})d\mathbf{\theta}\,.\]

The second equation in (2) holds because we can marginalize out the sampled topic words \(a_{n}\) by

\[p(w_{n}|\mathbf{\beta},\mathbf{t})=\sum_{a_{n}}p(w_{n}|\mathbf{\beta}_{a_{n}})p(a_{n}|\mathbf{ t})=\mathbf{\beta}\mathbf{t}\,, \tag{3}\]

where \(\mathbf{\beta}=\{\mathbf{\beta}_{i}\}_{i=1}^{T}\) are trainable parameters of the decoder; \(T\) is the number of topics and each \(\mathbf{\beta}_{i}\in\mathbb{R}^{D}\) is a topic distribution over words (all elements of \(\mathbf{\beta}_{i}\) are nonnegative, and sum to one).

### Neural Sequence Model

Our neural sequence model for text generation is built upon the VAE proposed in Bowman et al. (2015). Specifically, a continuous latent code \(\mathbf{z}\) is first generated from some prior distribution \(p(\mathbf{z})\), based on which the text sequence \(\mathbf{y}\) is then generated from a conditional distribution \(p(\mathbf{y}|\mathbf{z})\) parameterized by a neural network (often called the decoder). Since the model incorporates a latent variable \(\mathbf{z}\) that modulates the entire generation of the sentence, it should be able to capture the high-level source of variation in the data.

Topic-Guided Gaussian Mixture PriorThe aforementioned intuition is hard to be captured by a standard VAE, simply imposing a Gaussian prior on top of \(\mathbf{z}\), since the semantic information associated with a document intrinsically contains different subgroups (such as topics, sentiment, etc.). In our model, we consider incorporating the topic information into latent variables. Our model assumes each \(\mathbf{z}\) is drawn from a _topic-dependent_ GMM, that is,

\[p(\mathbf{z}|\mathbf{\beta},\mathbf{t}) =\sum\nolimits_{i=1}^{T}t_{i}\mathcal{N}(\mathbf{\mu}(\mathbf{\beta}_{i}) ,\mathbf{\sigma}^{2}(\mathbf{\beta}_{i}))\] \[\mathbf{\mu}(\mathbf{\beta}_{i}) =f_{\mu}(\mathbf{\beta}_{i})\] \[\mathbf{\sigma}^{2}(\mathbf{\beta}_{i}) =\text{diag}(\exp\left(f_{\sigma}(\mathbf{\beta}_{i})\right))\,, \tag{4}\]

where \(t_{i}\) is the usage of topic \(i\) in a document and \(\mathbf{\beta}_{i}\) is the \(i\)-th topic distribution over words. Both of them are inherited from the NTM discussed above. Both \(f_{\mu}(\cdot)\) and \(f_{\sigma}(\cdot)\) are implemented as feedforward neural networks, with trainable parameters \(\mathbf{W}_{\mu}\) and \(\mathbf{W}_{\sigma}\), respectively. Compared with a normal GMM prior that sets each mixture component to be \(\mathcal{N}(0,\mathbf{I})\), the proposed topic guided GMM prior provides semantic meaning for each mixture component, and hence makes the model more interpretable and controllable for text generation.

DecoderThe likelihood of a word sequence \(\mathbf{y}=\{y_{m}\}_{m=1}^{M}\) conditioned on the latent code \(\mathbf{z}\) is defined as:

\[p(\mathbf{y}|\mathbf{z}) =p(y_{1}|\mathbf{z})\prod\nolimits_{m=2}^{M}p(y_{m}|y_{1:m-1},\mathbf{ z})\] \[=p(y_{1}|\mathbf{z})\prod\nolimits_{m=2}^{M}p(y_{m}|\mathbf{h}_{m})\,, \tag{5}\]

where the conditional probability of each word \(y_{m}\) given all the previous words \(y_{1:m-1}\) and the latent code \(\mathbf{z}\) is defined through the hidden state \(\mathbf{h}_{m}\): \(\mathbf{h}_{m}=f(\mathbf{h}_{m-1},y_{m-1},\mathbf{z})\), where the function \(f(\cdot)\) is implemented as a Gated Recurrent Unit (GRU) cell Cho et al. (2014) in our experiments.

## 3 Inference

The proposed model (see Figure 1(a)) takes the bag-of-words as input and embeds a document into a topic vector. The topic vector is then used to reconstruct the bag-of-words input, and the learned topic distribution over words is used to model a topic-dependent prior to generate a sentence in the VAE setup. Specifically, the joint marginal likelihood can be written as:

\[p(\mathbf{y},\mathbf{d}|\mathbf{\beta})=\int_{\mathbf{\theta}}\int_{\mathbf{z}}p( \mathbf{\theta})p(\mathbf{d}|\mathbf{\beta},\mathbf{\theta})\\ \cdot p(\mathbf{z}|\mathbf{\beta},\mathbf{\theta})p(\mathbf{y}|\mathbf{z})\ d\mathbf{ \theta}d\mathbf{z}\,. \tag{6}\]

Since direct optimization of (6) is intractable, autoencoding variational Bayes is employed Kingma and Welling (2013). Denote \(q(\mathbf{\theta}|\mathbf{d})\) and \(q(\mathbf{z}|\mathbf{y})\) as the variational distributions for \(\mathbf{\theta}\) and \(\mathbf{z}\), respectively. The variational objective function, also 

called the evidence lower bound (ELBO), is constructed as

\[{\cal L}= \mathbb{E}_{q_{Q}(\mathbf{\theta}|\mathbf{d})}\left[\log p(\mathbf{d}|\mathbf{\beta} ,\mathbf{\theta})\right]-\text{KL}\left(q(\mathbf{\theta}|\mathbf{d})||p(\mathbf{\theta})\right)+ \tag{7}\] \[\underbrace{\mathbb{E}_{q(\mathbf{z}|\mathbf{y})}\left[\log p(\mathbf{y}|\bm {z})\right]-\mathbb{E}_{q(\mathbf{\theta}|\mathbf{d})}\left[\text{KL}\left(q(\mathbf{z}| \mathbf{y})||p(\mathbf{z}|\mathbf{\beta},\mathbf{\theta})\right)\right]}_{\text{neural sequence model},{\cal L}_{s}}.\]

By assuming

\[q(\mathbf{\theta}|\mathbf{d})=\mathcal{N}(\mathbf{\theta}|g_{\mu}(\mathbf{d}),\text{diag}( \exp\left(g_{\sigma}(\mathbf{d})\right))),\]

where both \(g_{\mu}(\cdot)\) and \(g_{\sigma}(\cdot)\) are implemented as feed-forward neural networks, the re-parameterization trick Kingma and Welling (2013) can be applied directly to build an unbiased and low-variance gradient estimator for the \({\cal L}_{t}\) term in (7). Below, we discuss in detail how to approximate the \({\cal L}_{s}\) term in (7) and infer an arbitrarily complex posterior for \(\mathbf{z}\). Note that \(\mathbf{z}\) is henceforth represented as \(\mathbf{z}_{K}\) in preparation for the introduction of Householder flows.

### Household Flow for Approximate Posterior

Householder flow Zhang et al. (2017a); Tomczak and Welling (2016) is a volume-preserving normalizing flow Rezende Mohamed (2015), capable of constructing an arbitrarily complex posterior \(q_{K}(\mathbf{z}_{K}|\mathbf{y})\) from an initial random variable \(\mathbf{z}_{0}\) with distribution \(q_{0}\), by composing a sequence of invertible mappings, _i.e._, \(\mathbf{z}_{K}=f_{K}\circ\cdots\diamond f_{2}\circ f_{1}(\mathbf{z}_{0})\). By repeatedly applying the chain rule and using the property of Jacobians of invertible functions, \(q_{K}(\mathbf{z}_{K}|\mathbf{y})\) is expressed as:

\[\log q_{K}(\mathbf{z}_{K}|\mathbf{y})=\log q_{0}(\mathbf{z}_{0}|\mathbf{y})-\sum ^{K}_{k=1}\log\Big{|}\det\frac{\partial f_{k}}{\partial\mathbf{z}_{k-1}}\Big{|}, \tag{8}\]

where \(|\det\frac{\partial f_{k}}{\partial\mathbf{z}_{k-1}}|\) is the absolute value of the Jacobian determinant. Therefore, the \({\cal L}_{s}\) term in (7) may be rewritten as

\[\mathbb{E}_{q_{0}(\mathbf{z}_{0}|\mathbf{y})}[\log p(\mathbf{y}|\mathbf{z}_{K})] +\sum^{K}_{k=1}\log\Big{|}\det\frac{\partial f_{k}}{\partial\mathbf{ z}_{k-1}}\Big{|}\] \[-\mathbb{E}_{q(\mathbf{\theta}|\mathbf{d})}\left[\text{KL}(q_{0}(\mathbf{z}_ {0}|\mathbf{y})||p(\mathbf{z}_{K}|\mathbf{\beta},\mathbf{\theta}))\right]~{}. \tag{9}\]

Here \(q_{0}(\mathbf{z}_{0}|\mathbf{y})\) is also specified as a GMM, _i.e._, \(q_{0}(\mathbf{z}_{0}|\mathbf{y})=\sum^{T}_{i=1}\pi_{i}(\mathbf{y})\mathcal{N}(\mathbf{\mu}_{i} (\mathbf{y}),\mathbf{\sigma}^{2}_{i}(\mathbf{y}))\). As illustrated in Figure 1(a), \(\mathbf{y}\) is first represented as a hidden vector \(\mathbf{h}\), by encoding the text sequence with an RNN. Based on this, the mixture probabilities \(\mathbf{\pi}\), the means and diagonal covariances of all the mixture components are all produced by an encoder network, which is a linear layer with the input \(\mathbf{h}\). In (9), the first term can be considered as the reconstruction error, while the remaining two terms act as regularizers, the tractability of which is important for the whole framework.

KL Divergence between two GMMsSince both the prior \(p(\mathbf{z}_{K}|\mathbf{\beta},\mathbf{\theta})\) and the initial density \(q_{0}(\mathbf{z}_{0}|\mathbf{y})\) for the posterior are GMMs, the calculation of the third term in (9) requires the KL divergence between two GMMs. Though no closed-form solutions exist, the KL divergence has an explicit upper bound Dilokthanakul et al. (2016), shown in Proposition 1.

**Proposition 1**.: _For any two mixture densities \(p=\sum^{n}_{i=1}\pi_{i}g_{i}\) and \(\hat{p}=\sum^{n}_{i=1}\hat{\pi}_{i}\hat{g}_{i}\), their KL divergence is upper-bounded by_

\[\text{KL}\left(p||\hat{p}\right)\leq\text{KL}\left(\pi||\hat{\pi}\right)+\sum ^{n}_{i=1}\pi_{i}\text{KL}\left(g_{i}||\hat{g_{i}}\right)~{}, \tag{10}\]

_where equality holds if and only if \(\frac{\pi_{i}g_{i}}{\sum^{n}_{i=1}\pi_{i}g_{i}}=\frac{\hat{\pi}g_{i}}{\sum^{n}_ {i=1}\hat{\pi}g_{i}}\)._

Proof.: With the log-sum inequality

\[\text{KL}\left(p||\hat{p}\right) =\int\left(\sum_{i}\pi_{i}g_{i}\right)\log\frac{\sum_{i}\pi_{i}g_{ i}}{\sum_{i}\pi\hat{g_{i}}}\] \[\leq\int\sum_{i}\pi_{i}g_{i}\log\frac{\pi_{i}g_{i}}{\hat{\pi}g_{ i}}\] \[=\sum_{i}\pi_{i}\log\frac{\pi_{i}}{\hat{\pi}}+\sum_{i}\pi_{i}\int g _{i}\log\frac{g_{i}}{\hat{g_{i}}}\] \[=\text{KL}(\pi||\hat{\pi})+\sum_{i}\pi_{i}\text{KL}(g_{i}||\hat{g _{i}})\,. \tag{11}\]

Since the KL divergence between two Gaussian distributions has a closed-form expression, the upper bound of the KL divergence between two GMMs can be readily calculated. Accordingly, the third term in (9) is upper bounded as

\[{\cal U}_{KL}=\mathbb{E}_{q(\mathbf{\theta}|\mathbf{d})}\big{[}\text{KL} \left(\mathbf{\pi}(\mathbf{y})||\mathbf{t}\right) \tag{12}\] \[+\sum^{T}_{i=1}\pi_{i}(\mathbf{y})\text{KL}\left(\mathcal{N}(\mathbf{\mu}_ {i}(\mathbf{y}),\mathbf{\sigma}^{2}_{i}(\mathbf{y})||\mathcal{N}(\mathbf{\mu}(\mathbf{\beta}_{i}), \mathbf{\sigma}^{2}(\mathbf{\beta}_{i}))\right)\big{]},\]

where the expectation \(\mathbb{E}_{q(\mathbf{\theta}|\mathbf{d})}[\cdot]\) can be approximated by a sample from \(q(\mathbf{\theta}|\mathbf{d})\).

Householder FlowHouseholder flow Tomczak and Welling (2016) is a series of Householder transformations, defined as follows. For a given vector \(\mathbf{z}_{k-1}\), the reflection hyperplane can be defined by a Householder vector \(\mathbf{v}_{t}\) that is orthogonal to the hyperplane. The reflection of this point about the hyperplane is

\[\mathbf{z}_{k}=\left(\mathbf{I}-2\frac{\mathbf{v}_{k}\mathbf{v}_{k}^{T}}{||\mathbf{v}_{k}||^{2}} \right)\mathbf{z}_{k-1}=\mathbf{H}_{k}\mathbf{z}_{k-1}\,,\] (13) 

where \({\bf H}_{k}={\bf I}-2\frac{p_{k}{\mathbf{v}}_{k}^{T}}{\|{\mathbf{v}}_{k}\|^{2}}\) is called the _Householder matrix_. An important property of the _Householder matrix_ is that the absolute value of the Jacobian determinant is equal to 1, therefore \(\sum_{k=1}^{K}\log\Big{|}\det\frac{\partial f_{k}}{\partial{\mathbf{z}}_{k-1}} \Big{|}=\sum_{k=1}^{K}\log|\det{\bf H}_{k}|=0\), significantly simplifying the computation of the lower bound in (9). For \(k=1,\ldots,K\), the vector \({\mathbf{v}}_{k}\) is produced by a linear layer with the input \({\mathbf{v}}_{k-1}\), where \({\mathbf{v}}_{0}={\mathbf{h}}\) is the last hidden vector of the encoder RNN that encodes the sentence \({\mathbf{y}}\).

Finally, by combining (7), (9) and (12), the ELBO can be rewritten as

\[\mathcal{L}\geq\mathcal{L}_{t}+\mathbb{E}_{q_{0}({\mathbf{z}}_{0}|{\mathbf{y}})}[\log p ({\mathbf{y}}|{\mathbf{z}}_{K})]-\mathcal{U}_{KL}\,. \tag{14}\]

### Extension to text summarization

When extending our model to text summarization, we are interested in modeling \(p({\mathbf{y}},{\mathbf{d}}|{\mathbf{x}})\), where \(({\mathbf{x}},{\mathbf{y}})\) denotes the document-summary pair, and \({\mathbf{d}}\) denotes the bag-of-words of the input document. The marginal likelihood can be written as \(p({\mathbf{y}},{\mathbf{d}}|{\mathbf{x}})=\int_{\theta}\int_{z}p({\mathbf{\theta}})p({\mathbf{d}}| {\mathbf{\beta}},{\mathbf{\theta}})p({\mathbf{z}}|{\mathbf{\beta}},{\mathbf{\theta}})p({\mathbf{y}}|{ \mathbf{x}},{\mathbf{z}})\,d{\mathbf{\theta}}d{\mathbf{z}}\). Assume the approximate posterior of \({\mathbf{z}}\) is only dependent on \({\mathbf{x}}\), _i.e._, \(q({\mathbf{z}}|{\mathbf{x}})\) is proposed as the variational distribution for \({\mathbf{z}}\). The ELBO is then constructed as

\[\mathcal{L}= \mathcal{L}_{t}+\mathbb{E}_{q({\mathbf{z}}|{\mathbf{x}})}\left[\log p({ \mathbf{y}}|{\mathbf{x}},{\mathbf{z}})\right]\] \[-\mathbb{E}_{q({\mathbf{\theta}}|{\mathbf{d}})}\left[\text{KL}\left(q({ \mathbf{z}}|{\mathbf{x}})||p({\mathbf{z}}|{\mathbf{\beta}},{\mathbf{\theta}})\right)\right]\,, \tag{15}\]

where \(\mathcal{L}_{t}\) is the same as used in (7). The main difference when compared with unconditional text generation lies in the usage of \(p({\mathbf{y}}|{\mathbf{x}},{\mathbf{z}})\) and \(q({\mathbf{z}}|{\mathbf{x}})\), illustrated in Figure 1(b). The generation of \({\mathbf{y}}\) given \({\mathbf{x}}\) is not only dependent on a standard Seq2Seq model with attention (Nallapati et al., 2016), but also affected by \({\mathbf{z}}\) (_i.e._, \({\mathbf{z}}_{K}\)), which provides the high-level topic guidance.

### Diversity Regularizer for NTM

Redundancy in inferred topics is a common issue existing in general topic models. In order to address this, it is straightforward to regularize the row-wise distance between paired topics to diversify the topics. Following Xie et al. (2015); Miao et al. (2017), we apply a topic diversity regularization while carrying out the inference.

Specifically, the distance between a pair of topics is measured by their cosine distance \(a({\mathbf{\beta}}_{i},{\mathbf{\beta}}_{j})=\arccos\Big{(}\frac{|{\mathbf{\beta}}_{i}{ \mathbf{\beta}}_{j}|}{|{\mathbf{\beta}}_{i}|\,{}_{2}\,|{\mathbf{\beta}}_{j}|\,{}_{2}}\Big{)}\). The mean angle of all pairs of \(T\) topics is \(\phi=\frac{1}{T^{2}}\sum_{i}\sum_{j}a({\mathbf{\beta}}_{i},{\mathbf{\beta}}_{j})\), and the variance is \(\nu=\frac{1}{T^{2}}\sum_{i}\sum_{j}(a({\mathbf{\beta}}_{i},{\mathbf{\beta}}_{j})-\phi)^ {2}\). Finally, the topic-diversity regularization is defined as \(R=\phi-\nu\).

## 4 Related Work

The VAE was proposed by Kingma and Welling (2013), and since then, it has been applied successfully in a variety of applications (Gregor et al., 2015; Kingma et al., 2014; Chen et al., 2017; Wang et al., 2018; Shen et al., 2018). Focusing on text generation, the methods in Miao et al. (2017, 2016); Srivastava and Sutton (2017) represent texts as bag-of-words, and Bowman et al. (2015) proposed the usage of an RNN as the encoder and decoder, and found some negative results. In order to improve the performance, different convolutional designs (Semenitua et al., 2017; Shen et al., 2017a; Yang et al., 2017) have been proposed. A VAE variant was further developed in Hu et al. (2017) to control the sentiment and tense of generated sentences. Additionally, the VAE has also been considered for conditional text generation tasks, including machine translation (Zhang et al., 2016), image captioning (Pu et al., 2016), dialogue generation (Serban et al., 2017; Shen et al., 2017b; Zhao et al., 2017) and text summarization (Li et al., 2017b; Miao and Blunsom, 2016). In particular, distinct from the above works, we propose the usage of a topic-dependent prior to explicitly incorporate topic guidance into the text-generation framework.

The idea of using learned topics to improve NLP tasks has been explored previously, including methods combining topic and neural language models (Ahn et al., 2016; Dieng et al., 2016; Lau et al., 2017; Mikolov and Zweig, 2012; Wang et al., 2017), as well as leveraging topic and word embeddings (Liu et al., 2015; Xu et al., 2018). Distinct from them, we propose the use of topics to guide the prior of a VAE, rather than only the language model (_i.e._, the decoder in a VAE setup). This provides more flexibility in text modeling and also the ability to infer the posterior on latent codes, which could be useful for visualization and downstream tasks.

Neural abstractive summarization was pioneered in Rush et al. (2015), and it was followed and extended by Chopra et al. (2016). Currently the RNN-based encoder-decoder framework with attention (Nallapati et al., 2016; See et al., 2017) remains popular in this area. Attention models typ 

ically work as a keyword detector, which is similar to topic modeling in spirit. This fact motivated us to extend our topic-guided VAE model to text summarization.

## 5 Experiments

We evaluate our TGVAE on text generation and text summarization, and interpret its improvements both quantitatively and qualitatively.

### Text Generation

**Dataset** We conduct experiments on three publicly available corpora: APNEWS, IMDB and BNC.1 APNEWS2 is a collection of Associated Press news articles from 2009 to 2016. IMDB is a set of movie reviews collected by Maas et al. (2011), and BNC (BNC Consortium, 2007) is the written portion of the British National Corpus, which contains excerpts from journals, books, letters, essays, memoranda, news and other types of text. For the three corpora, we tokenize the words and sentences, lowercase all word tokens, and filter out word tokens that occur less than 10 times. For the topic model, we remove stop words in the documents and exclude the top \(0.1\%\) most frequent words and also words that appear less than 100 documents. A summary statistics is provided in Table 1.

Footnote 1: These three datasets can be downloaded from [https://github.com/jhlau/topically-driven-language-model](https://github.com/jhlau/topically-driven-language-model).

Footnote 2: [https://www.ap.org/en-gb/](https://www.ap.org/en-gb/)

**Evaluation** We first compare the perplexity of our neural sequence model with a variety of baselines. Further, we evaluate BLEU scores on the generated sentences, noted as _test_-BLEU and _self_-BLEU. _test_-BLEU (higher is better) evaluates the quality of generated sentences using a group of real test-set sentences as the reference, and _self_-BLEU (lower is better) mainly measures the diversity of generated samples Zhu et al. (2018).

**Setup** For the neural topic model (NTM), we consider a 2-layer feed-forward neural network to model \(q({\mathbf{\theta}}|{\mathbf{d}})\), with 256 hidden units in each layer; ReLU is used as the activation function. The hyperparameter \(\lambda\) for the neural topic model diversity regularizer is fixed to \(0.1\) across all the experiments. All the sentences in the paragraph are used to obtain the bag-of-words presentation \({\mathbf{d}}\). The maximum number of words in a paragraph is set to 300. For the neural sequence model (NSM), we use bidirectional-GRU as the encoder and a standard GRU as the decoder. The hidden state of our GRU is fixed to \(600\) across all the three corpora. For the input sequence, we fix the sequence length to 30. In order to avoid overfitting, dropout with a rate of \(0.4\) is used in each GRU layer.

**Baseline** We test the proposed method with different numbers of topics (components in GMM) and different numbers of Householder flows (\(i.e.\), \(K\)), and compare it with six baselines: (_i_) a standard language model (LM); (_ii_) a standard variational RNN auto-encoder (VAE); (_iii_) a Gaussian prior-based VAE with Householder Flow (VAE+HF); (_iv_) a standard LSTM language model with LDA as additional feature (LDA+LSTM); (_v_) Topic-RNN Dieng et al. (2016), a joint learning framework which learns a topic model and a language model simultaneously; (_vi_) TDLM Lau et al. (2017), a joint learning framework which learns a convolutional based topic model and a language model simultaneously.

**Results** The results in Table 3 show that the models trained with a VAE and its Householder extension does not outperform a well-optimized language model, and the KL term tends to be annealed with the increase of \(K\). In comparison, our TGVAE achieves a lower perplexity upper bound, with a relative larger \(\mathcal{U}_{KL}\). We attribute the improvements to our topic guided GMM model design, which provides additional topical clustering information in the latent space; the Householder flow also boosts the posterior inference for our TGVAE. We also observe consistent improvements with the number of topics, which demonstrates the efficiency of our TGVAE.

To verify the generative power of our TGVAE, we generate samples from our _topic-dependent_ prior and compare various methods on the BLEU scores in Table 2. With the increase of topic numbers, our TGVAE yields consistently better _self_-BLEU and a boost over _test_-BLEU relative to standard VAE models. We also show a group of sampled sentences drawn from a portion of topics in Table 5. Our TGVAE is able to generate diverse sentences under topic guidance. When generating sentences under a mixture of topics, we draw multiple samples from the GMM and take \({\mathbf{z}}\) as the averaged sample.

Though this paper focuses on generating coherent topic-specific sentences rather than the learned topics themselves, we also evaluate the topic coherence Lau et al. (2017) to show the rationality of our joint learning framework. We compute topic coher 

ence using normalized PMI (NPMI). In practice, we average topic coherence over the top 5/10/15/20 topic words. To aggregate topic coherence score, we further average the coherence scores over topics. Results are summarized in Table 4.

### Text Summarization

**Dataset** We further test our model for text summarization on two popular datasets. First, we follow the same setup as in Rush et al. (2015) and consider the Gigawords corpus3, which consists of \(3.8\)M training pair samples, \(190\)K validation samples and 1,951 test samples for evaluation. An input-summary pair consists of the first sentence and the headline of the source articles. We also evaluate various models on the DUC-2004 test set4, which has \(500\) news articles. Different from Gigawords, each article in DUC-2004 is paired with four expert-generated reference summaries. The length of each summary is limited to \(75\) bytes.

Footnote 4: [http://duc.nist.gov/duc2004](http://duc.nist.gov/duc2004)

**Evaluation** We evaluate the performance of our model with the ROUGE score Lin (2004), which counts the number of overlapping content between the generated summaries and the reference summaries, _e.g._, overlapped n-grams. Following practice, we use F-measures of ROUGE-1 (RF-1), ROUGE-2 (RF-2) and ROUGE-L (RF-L) for Gigawords and Recall measures of ROUGE-1 (RR-1), ROUGE-2 (RR-2) and ROUGE-L (RR-L) for DUC-2004.

**Setup** We have a similar data tokenization as we

\begin{table}
\begin{tabular}{c|c c|c c c|c c c|c c c} \hline \multirow{2}{*}{**Dataset**} & \multicolumn{2}{c|}{**Vocabulary**} & \multicolumn{3}{c|}{**Training**} & \multicolumn{3}{c|}{**Development**} & \multicolumn{3}{c}{**Testing**} \\  & LM & TM & \# Docs & \# Sents & \# Tokens & \# Docs & \# Sensts & \# Tokes & \# Bocs & \# Sents & \# Tokens \\ \hline APNEWS & \(32,400\) & \(7,790\) & \(50K\) & \(0.7M\) & \(15M\) & \(2K\) & \(27.4K\) & \(0.6M\) & \(2K\) & \(26.3K\) & \(0.6M\) \\ \hline IMDB & \(34,256\) & \(8,713\) & \(75K\) & \(0.9M\) & \(20M\) & \(12.5K\) & \(0.2M\) & \(0.3M\) & \(12.5K\) & \(0.2M\) & \(0.3M\) \\ \hline BNC & \(41,370\) & \(9,741\) & \(15K\) & \(0.8M\) & \(18M\) & \(1K\) & \(44K\) & \(1M\) & \(1K\) & \(52K\) & \(1M\) \\ \hline \end{tabular}
\end{table}
Table 1: Summary statistics for APNEWS, IMDB and BNC.

\begin{table}
\begin{tabular}{c|c c c|c c c|c c c} \hline \multirow{2}{*}{Metric} & \multirow{2}{*}{Methods} & \multicolumn{3}{c|}{APNEWS} & \multicolumn{3}{c|}{IMDB} & \multicolumn{3}{c}{BNC} \\  & PTL & KL & PPTL & KL & PPL & KL \\ \hline LM & 62.79 & – & 03.38 & – & 100.07 & — \\ LDA+LSTM & 57.05 & – & 69.58 & – & 96.42 & – \\ Topic-RNN & 56.77 & – & 68.74 & – & 94.66 & – \\ TDLM & 35.00 & – & 63.67 & – & 91.42 & – \\ \hline VAE & \(\leq\)758.9 & 1.78 & 86.16 & 2.78 & \(\leq\)105.10 & 0.13 \\ VAE+HF (KL=1) & \(\leq\)752.98 & 1.32 & \(\leq\)840.6 & 1.83 & \(<\)105.13 & 0.31 \\ VAE+HF (K=10) & \(\leq\)716.00 & 0.83 & \(<\)383.67 & 1.51 & \(<\)104.82 & 0.17 \\ \hline TGVAE (K=0, T=10) & \(\leq\)56.12 & 2.73 & \(<\)62.99 & 3.99 & \(<\)592.32 & 3.40 \\ TGVAE (K=1, T=10) & \(\leq\)56.58 & 2.70 & \(>\)62.12 & 2.38 & \(<\)591.7 & 3.32 \\ TGVAE (K=10, T=10) & \(\leq\)55.77 & 2.69 & \(<\)62.22 & 3.94 & \(<\)91.9 & 2.99 \\ TGVAE (K=10, T=30) & \(\leq\)51.27 & 3.62 & \(<\)59.45 & 4.62 & \(<\)88.34 & 3.82 \\ TGVAE (K=10, T=50) & \(\leq\)**48.73** & 3.55 & \(<\)**57.11** & 5.02 & \(<\)**57.86** & 4.57 \\ \hline \end{tabular}
\end{table}
Table 2: _test_-BLEU (higher is better) and _self_-BLEU (lower is better) scores over three corpora.

\begin{table}
\begin{tabular}{c|c c c|c c c|c c c c} \hline \multirow{2}{*}{Methods} & \multicolumn{3}{c|}{APNEWS} & \multicolumn{3}{c|}{IMDB} & \multicolumn{3}{c}{BNC} \\  & PTL & KL & PPL & KL & PPL & KL \\ \hline LM & 62.79 & – & 03.38 & – & 100.07 & — \\ LDA+LSTM & 57.05 & – & 69.58 & – & 96.42 & – \\ Topic-RNN & 56.77 & – & 68.74 & – & 94.66 & – \\ TDLM & 35.00 & – & 63.67 & – & 91.42 & – \\ \hline VAE & \(\leq\)758.9 & 1.78 & 86.16 & 2.78 & \(\leq\)105.10 & 0.13 \\ VAE+HF (KL=1) & \(\leq\)752.98 & 1.32 & \(\leq\)840.6 & 1.83 & \(<\)105.13 & 0.31 \\ VAE+HF (K=10) & \(\leq\)716.00 & 0.83 & \(<\)383.67 & 1.51 & \(<\)104.82 & 0.17 \\ \hline TGVAE (K=0, T=10) & \(\leq\)56.12 & 2.73 & \(<\)62.99 & 3.99 & \(<\)592.32 & 3.40 \\ TGVAE (K=1, T=10) & \(\leq\)56.58 & 2.70 & \(>\)62.12 & 2.38 & \(<\)591.7 & 3.32 \\ TGVAE (K=10, T=10) & \(\leq\)55.77 & 2.69 & \(<\)62.22 & 3.94 & \(<\)91.9 & 2.99 \\ TGVAE (K=10, T=30) & \(\leq\)51.27 & 3.62 & \(<\)59.45 & 4.62 & \(<\)88.34 & 3.82 \\ TGVAE (K=10, T=50) & \(\leq\)**48.73** & 3.55 & \(<\)**57.11** & 5.02 & \(<\)**57.86** & 4.57 \\ \hline \end{tabular}
\end{table}
Table 3: Perplexity and averaged KL scores over three corpora. KL in our TGVAE denotes \(\mathcal{U}_{KL}\) in Eqn. (12).

 

have in text generation. Additionally, for the vocabulary, we count the frequency of words in both the source article the target summary, and maintain the top 30,000 tokens as the source article and target summary vocabulary. For the NTM, we further remove top \(0.3\%\) words and infrequent words to get a topic model vocabulary in size of \(8000\). For the NTM, we follow the same setup as our text generation. In the NSM, we keep using bidirectional-GRU as the encoder and a standard GRU as the decoder. The hidden state is fixed to \(400\). An attention mechanism Bahdanau et al. (2015) is applied in our sequence-to-sequence model.

**Baseline** We compare our method with the following alternatives: (_i_) a standard sequence-to-sequence model with attention Bahdanau et al. (2015) (Seq2Seq); (_ii_) a model similar to our TGVAE, but without the usage of the topic-dependent prior and Householder flow Var-Seq2Seq); and (_iii_) a model similar to our TGVAE, but without the usage of the topic dependent prior (Var-Seq2Seq

\begin{table}
\begin{tabular}{l|c c c c c c} \hline \multicolumn{7}{l|}{**Data**} & \multicolumn{2}{c|}{**Topic**} \\  & education & * the commission has approved a bill that would make state funding available for the city ’s new school . & * the feline did n’t survive fence hangers at the lake & * the play founde chefee was not a comb\textgreater{}, ‘smomb\textgreater{}’s ruling and that the state ’s highest court has been convicted or first-degree murder. & * forexasters may they ”re still trying to see the national weather service watch for the latest forecast for fridley evening . & * the hopes are lydepew nor coworkers 9 million \\ APNEWS & weather & * procedures vary below the jacplot now receive sofa 9 million & * the albums have professor hours said memory that the state’s open court claims it takes an emotional manner about issuing child moderters based on religion. & * animal–machine medicine & * the study says the annual surface department and others are sort to make similar cases to the virus in the zoo. & * after watching movie, there is a great documentary about the war in the years of the Israeli war . & * the entire animation was great at times as tho readings of Disney favorings. & * the show would have warranted for Zo episodes and it does help immediately & * the streak the other party income ’s cuink\textgreater{} husband (carried) / () as this past , ab+tr for the road ) . & * howlers showed up to let be as much as the move “is last scene where three were comk> pills to nth the cumk\textgreater{} . & * the movie should a bottle as the worst movie is here over seen. & * the raymé favorite childhood is that rocker chakes the character in jane’s way, playing the one with hannlet. & * her thinking\text@cases```
`
`````` drugs drug as the problem although both economically ill patients arising from a local job will be in traumatic changes. & * in traumatic changes & * theraims change\text@cases{}\end{tabular} & * the stoy sales six given to five students ’ award off : out at a laboratory after the three watts of the hours travelling in and chairman store the bank of the cursko-stittie. & * schoolchildren will either go or brake to church in his place every year in the savvy . & 

HF).

**Results** The results in Table 7 show that our TGVAE achieves better performance than a variety of strong baseline methods on both Gigawords and DUC-2004, demonstrating the practical value of our model. It is worthwhile to note that recently several much more complex CNN/RNN architectures have been proposed for abstract text summarization, such as SEASS Zhou et al. (2017), ConvS2S Gehring et al. (2017), and ReinforcedConvS2S Wang et al. (2018a). In this work, we focus on a relatively simple RNN architecture for fair comparison. In such a way, we are able to conclude that the improvements on the results are mainly from our topic-guided text generation strategy. As can be seen, though the Var-Seq2Seq model achieves comparable performance with the standard Seq2Seq model, the usage of Householder flow for more flexible posterior inference boosts the performance. Additionally, by combining the proposed topic-dependent prior and Householder flow, we yield further performance improvements, demonstrating the importance of topic guidance for text summarization.

To demonstrate the readability and diversity of the generated summaries, we present typical examples in Table 6. The words in blue are the topic words that appear in the source article but do not exist in the reference, while the words in red are neither in the reference nor in the source article. When the topic information is provided, our model is able to generate semantically-meaningful words which may not even exist in the reference summaries and the source articles. Additionally, with our topic-guided model, we can always generate a summary with meaningful initial words. These phenomena imply that our model supplies more insightful semantic information to improve the quality of generated summaries.

Finally, to demonstrate that our TGVAE learns interpretable topic-dependent GMM priors, we draw multiple samples from each mixture component and visualize them with t-SNE Maaten and Hinton (2008). As can be seen from Figure 2, we have learned a group of separable _topic-dependent_ components. Each component is clustered and also maintains semantic meaning in the latent space, _e.g._, the clusters corresponding to the topic "stock" and "finance" are close to each other, while the clusters for "finance" and "disease" are far away from each other. Additionally, to understand the topic model we have learned, we provide the top 5 words for 10 randomly chosen topics on each dataset (the boldface word is the topic name summarized by us), as shown in Table 8.

## 6 Conclusion

A novel text generator is developed, combining a VAE-based neural sequence model with a neural topic model. The model is an extension of conditional VAEs in the framework of unsupervised learning, in which the topics are extracted from the data with clustering structure rather than predefined labels. An effective inference method based on Householder flow is designed to encourage the complexity and the diversity of the learned topics. Experimental results are encouraging, across multiple NLP tasks.

\begin{table}
\begin{tabular}{l|c c c c c c c c c c} \hline \hline
**Dataset** & **education** & **animal** & **crime** & **winter** & **lottery** & **terrortium** & **law** & **art** & **transportation** & **market** \\  & students & animals & murder & weather & mega & synia & lawsuit & album & airlines & zacks \\  & education & dogs & first-degree & corevesses & lottery & iran & appeals & music & rail & crocts \\ APNEWS & schools & zero & shooting & winds & powerful & millasm & jservices & film & transit & earnings \\  & math & bar & sentenced & rain & gambling & argon & constitutional & songs & bridge & revenue \\  & teachers & wildlife & gamthor & snow & jackpot & kora & judge & comedy & airport & income \\ \cline{2-13}  & war & children & **episode** & **name** & **decide** & **ethic** & action & **borrer** & **negative** & **Japanese** \\ \cline{2-13}  & aircraft & chemical & season & carrot!fonders & holm & porm & human & horror & stupd & mile \\  & president & musical & episode & statanyk & poster & unfurning & king & zeolite & hornlike & lutorova \\  & war & beutics & sandler & gible & mitheit & srs & chin & verwerd & swks & sadika \\  & military & mneticals & cartoons & powell & book & gay & i & candynn & waste & amine \\  & soldiers & diavey & jokes & hultor & aganda & geroc & junja & dracula & scary & takashi \\ \hline  & medical & education & religion & entertainment & If & Law & fédicities & crime & **ports** & **environment** \\ \cline{2-13}  & patients & award & church & film & unix & tax & bedrooms & police & corp & nuclear \\  & gastric & discipline & god & video & corp & comn & hotel & killed & league & emission \\  & cells & economic & art & album & software & lamont & restaurant & arrested & striker & dioxide \\  & oesophageal & research & rheological & comedy & server & council & rooms & soliders & steam & pollution \\  & mucosa & imitation & religions & movie & bin & penioneters & situated & murder & goal & warning \\ \hline  & **terrorti** & **crime** & **finance** & **sprits** & **lav** & **stock** & **auto** & **dicase** & **globalization** & **pellites** \\ \cline{2-13}  & polacin & wounding & teel & scored & sentence & spring & motor & flu & nuclear & repulsion \\  & antfa & skilled & kk & ruchonus & guilty & index & amunder & decease & es & mexio \\  & yaser & roadside & gold & points & crimes & prices & toyga & virus & dpark & democrats \\  & abbas & injuring & opcc & champion & court & tites & auto & bird & nuto & bank \\  & ixaraki & crashed & qpc & beats & connected & stock & food & health & bilateral & presidential \\ \hline \hline \end{tabular}
\end{table}
Table 8: 10 topics learned from our model on APNEWS, IMDB, BNC and Gigawords.

 



## References

* Ahn et al. (2016) Sungjin Ahn, Heeyoul Choi, Tanel Paramaaa, and Yoshua Bengio. 2016. A neural knowledge language model. _arXiv preprint arXiv:1608.00318_.
* Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In _ICLR_.
* Bischof and Sun (1994) Christian H Bischof and Xiaobai Sun. 1994. On orthogonal block elimination. _Preprint MCS-P450-0794, Mathematics and Computer Science Division, Argonne National Laboratory_.
* Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. _JMLR_.
* Consortium (2007) BNC BNC Consortium. 2007. The British National Corpus, version 3 (BNC XML Edition). _Distributed by Bodleian Libraries, University of Oxford, on behalf of the BNC Consortium_.
* Bowman et al. (2015) Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2015. Generating sentences from a continuous space. _arXiv preprint arXiv:1511.06349_.
* Chen et al. (2017) Changyouu Chen, Chunyuan Li, Liqun Chen, Wenlin Wang, Yuchen Pu, and Lawrence Carin. 2017. Continuous-time flows for efficient inference and density estimation. _arXiv preprint arXiv:1709.01179_.
* Chen et al. (2018) Liqun Chen, Shuyang Dai, Chenyang Tao, Haichao Zhang, Zhe Gan, Dinghan Shen, Yizhe Zhang, Guoyin Wang, Ruiyi Zhang, and Lawrence Carin. 2018. Adversarial text generation via featuremover's distance. In _NeurIPS_.
* Cho et al. (2014) Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dznitty Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. _arXiv preprint arXiv:1406.1078_.
* Chopra et al. (2016) Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In _NAACL_.
* Dieng et al. (2016) Adji B Dieng, Chong Wang, Jianfeng Gao, and John Paisley. 2016. Topicrnn: A recurrent neural network with long-range semantic dependency. _arXiv preprint arXiv:1611.01702_.
* Dilokthanakul et al. (2016) Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. 2016. Deep unsupervised clustering with Gaussian mixture variational autoencoders. _arXiv preprint arXiv:1611.02648_.
* Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. _arXiv preprint arXiv:1705.03122_.
* Gregor et al. (2015) Karol Gregor, Ivvo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. 2015. DRAW: A recurrent neural network for image generation. _arXiv preprint arXiv:1502.04623_.
* Guo et al. (2017) Jiaixian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. 2017. Long text generation via adversarial training with leaked information. _arXiv preprint arXiv:1709.08624_.
* Hu et al. (2017) Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In _ICML_.
* Jiang et al. (2016) Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. 2016. Variational deep embedding: An unsupervised and generative approach to clustering. _arXiv preprint arXiv:1611.05148_.
* Kim et al. (2018) Yoon Kim, Sam Wiseman, Andrew C Miller, David Sontag, and Alexander M Rush. 2018. Semi-Amortized variational autoencoders. _arXiv preprint arXiv:1802.02550_.
* Kingma et al. (2014) Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In _NIPS_.
* Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013. Autoencoding variational Bayes. _arXiv preprint arXiv:1312.6114_.
* Lau et al. (2017) Jey Han Lau, Timothy Baldwin, and Trevor Cohn. 2017. Topically driven neural language model. _arXiv preprint arXiv:1704.08012_.
* Li et al. (2017a) Jiwei Li, Will Monroe, Tianlin Shi, Sebastien Jean, Alan Ritter, and Dan Jurafsky. 2017a. Adversarial learning for neural dialogue generation. _arXiv preprint arXiv:1701.06547_.
* Li et al. (2017b) Piji Li, Wai Lam, Lidong Bing, and Zihao Wang. 2017b. Deep recurrent generative decoder for abstractive text summarization. _arXiv preprint arXiv:1708.00625_.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. _Text Summarization Branches Out_.
* Liu et al. (2015) Yang Liu, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2015. Topical word embeddings. In _AAAI_.
* Maas et al. (2011) Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In _ACL_.
* van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. _JMLR_.
* Miao and Blunsom (2016) Yishu Miao and Phil Blunsom. 2016. Language as a latent variable: Discrete generative models for sentence compression. _arXiv preprint arXiv:1609.07317_

Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017. Discovering discrete latent topics with neural variational inference. _arXiv preprint arXiv:1706.00359_.
* Miao et al. (2016) Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In _ICML_.
* Mikolov and Zweig (2012) Tomas Mikolov and Geoffrey Zweig. 2012. Context dependent recurrent neural network language model. _SLT_.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. _arXiv preprint arXiv:1602.06023_.
* Pu et al. (2016) Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens, and Lawrence Carin. 2016. Variational autoencoder for deep learning of images, labels and captions. In _NIPS_.
* Rezende and Mohamed (2015) Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. _arXiv preprint arXiv:1505.05770_.
* Rezende et al. (2014) Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. _arXiv preprint arXiv:1401.4082_.
* Rush et al. (2015) Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. _arXiv preprint arXiv:1509.00685_.
* See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. _arXiv preprint arXiv:1704.04368_.
* Semeniuta et al. (2017) Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. 2017. A hybrid convolutional variational autoencoder for text generation. _arXiv preprint arXiv:1702.02390_.
* Serban et al. (2017) Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In _AAAI_.
* Shen et al. (2018) Dinghan Shen, Qinliang Su, Paidamoyo Chapfuwa, Wenlin Wang, Guoyin Wang, Lawrence Carin, and Ricardo Henao. 2018. Nash: Toward end-to-end neural architecture for generative semantic hashing. _arXiv preprint arXiv:1805.05361_.
* Shen et al. (2017a) Dinghan Shen, Yizhe Zhang, Ricardo Henao, Qinliang Su, and Lawrence Carin. 2017a. Deconvolutional latent-variable model for text sequence matching. _arXiv preprint arXiv:1709.07109_.
* Shen et al. (2017b) Xiaoyu Shen, Hui Su, Yanran Li, Wenjie Li, Shuzi Niu, Yang Zhao, Akiko Aizawa, and Guoping Long. 2017b. A conditional variational framework for dialog generation. _arXiv preprint arXiv:1705.00316_.
* Srivastava and Sutton (2017) Akash Srivastava and Charles Sutton. 2017. Autoencoding variational inference for topic models. _arXiv preprint arXiv:1703.01488_.
* Sun and Bischof (1995) Xiaobai Sun and Christian Bischof. 1995. A basis-kernel representation of orthogonal matrices. _SIAM journal on matrix analysis and applications_.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _NIPS_.
* Tomczak and Welling (2016) Jakub M Tomczak and Max Welling. 2016. Improving variational auto-encoders using householder flow. _arXiv preprint arXiv:1611.09630_.
* Wang et al. (2018a) Li Wang, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du. 2018a. A reinforced topici-aware convolutional sequence-to-sequence model for abstractive text summarization. _arXiv preprint arXiv:1805.03616_.
* Wang et al. (2017) Wenlin Wang, Zhe Gan, Wenqi Wang, Dinghan Shen, Jiaji Huang, Wei Ping, Sanjeev Satheesh, and Lawrence Carin. 2017. Topic compositional neural language model. _arXiv preprint arXiv:1712.09783_.
* Wang et al. (2018b) Wenlin Wang, Yuchen Pu, Vinay Kumar Verma, Kai Fan, Yizhe Zhang, Changyou Chen, Piyush Rai, and Lawrence Carin. 2018b. Zero-shot learning via class-conditioned deep generative models. In _AAAI_.
* Xie et al. (2015) Pengtao Xie, Yuntian Deng, and Eric Xing. 2015. Diversifying restricted Boltzmann machine for document modeling. In _KDD_.
* Xu et al. (2018) Hongteng Xu, Wenlin Wang, Wei Liu, and Lawrence Carin. 2018. Distilled wasserstein learning for word embedding and topic modeling. In _NIPS_.
* Yang et al. (2017)Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. _arXiv preprint arXiv:1702.08139_.
* Yu et al. (2017) Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. SeqGAN: Sequence generative adversarial nets with policy gradient. In _AAAI_.
* Zhang et al. (2016)Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. 2016. Variational neural machine translation. _arXiv preprint arXiv:1605.07869_.
* Zhang et al. (2018) Ruiyi Zhang, Changyou Chen, Zhe Gan, Wenlin Wang, Liqun Chen, Dinghan Shen, Guoying Wang, and Lawrence Carin. 2018. Sequence generation with guider network. _arXiv preprint arXiv:1811.00696_.
* Zhang et al. (2017a)Ruiyi Zhang, Chunyuan Li, Changyou Chen, and Lawrence Carin. 2017a. Learning structural weight uncertainty for sequential decision-making. _arXiv preprint arXiv:1801.00085_.
* Zhang et al. (2017b) Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. 2017b. Adversarial feature matching for text generation. _arXiv preprint arXiv:1706.03850_ 

Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. _arXiv preprint arXiv:1703.10960_.
* Zhou et al. (2017) Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. 2017. Selective encoding for abstractive sentence summarization. _arXiv preprint arXiv:1704.07073_.
* Zhu et al. (2018) Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. _arXiv preprint arXiv:1802.01886_.