Year: 2018

Abstract:
Variational autoencoders (VAEs) have received much attention recently as an endto-end architecture for text generation. Existing methods primarily focus on synthesizing relatively short sentences (with less than twenty words). In this paper, we propose a novel framework, hierarchically-structured variational autoencoders (hier-VAE), for generating long and coherent units of text. To enhance the model’s plan-ahead ability, intermediate sentence representations are introduced into the generative networks to guide the word-level predictions. To alleviate the typical optimization challenges associated with textual VAEs, we further employ a hierarchy of stochastic layers between the encoder and decoder networks. Extensive experiments are conducted to evaluate the proposed method, where hier-VAE is shown to make effective use of the latent codes and achieve lower perplexity relative to language models. Moreover, the generated samples from hier-VAE also exhibit superior quality according to both automatic and human evaluations.

heading: 1 INTRODUCTION
text: Probabilistic generative models for text have received considerable attention recently, in part because of their ability to leverage abundant unlabeled data and learn text representations with strong generalization abilities. Combined with the flexibility of deep neural networks, generative modeling provides a powerful and effective framework to estimate the underlying probability distribution of sentences or documents, and to potentially capture the rich sequential information inherent in natural language. In this work, we focus on one specific type of generative model: the variational autoencoder (VAE) (Kingma & Welling, 2013).
The VAE maps a text sequence into a continuous latent variable, or code, via an inference (encoder) network; a generative (decoder) network is utilized to reconstruct the input sentence, conditioned on samples from the latent code (via its posterior distribution). The inference and generative networks, parametrized by deep neural networks, can be optimized jointly by maximizing a
variational lower bound of the marginal distribution of the training corpus. The advantages of leveraging the neural variational inference (NVI) framework have been demonstrated on a wide range of natural language processing (NLP) tasks (Bowman et al., 2016; Miao et al., 2016; Yang et al., 2017; Serban et al., 2017; Shah & Barber, 2018; Kim et al., 2018; Kaiser et al., 2018; Bahuleyan et al., 2018; Deng et al., 2018). Although NVI-based methods have demonstrated success in modeling (generating) relatively short sentences (typically less than twenty words), there are several challenges that need to be addressed to facilitate long text generation with deep latent-variable models.
The first challenge originates from the nature of recurrent neural networks (RNNs), which have become the cornerstone for text generation. RNNs are typically trained with gradient descent, using backpropagation through time (BPTT) (Rumelhart et al., 1986). However, it is very difficult to scale such a training paradigm to long sequences, due to vanishing or exploding gradients (Hochreiter & Schmidhuber, 1997). Moreover, the information of the entire generated sequence needs to be stored in the RNN intermediate hidden states, which could be very demanding for RNNs with the increasing length of generated texts. As shown in Table 1, the sentence sampled from a vanilla textVAE model (Bowman et al., 2016) tends to repeat itself as the sequence proceeds, leading to inferior
generation results. In addition to the issue of repetitiveness, to generate globally-coherent long text sequences, it is desirable that both the higher-level abstract features and lower-level concrete details (e.g., specific word choices) of longer texts can be leveraged by the generative network. This intuition is also hard to capture with a single-layer (flat) RNN-based generative network.
Another underlying challenge of generating long text relates to the “posterior collapse” issue inherent in training variational autoencoders with a strong autoregressive decoder (Bowman et al., 2016; Yang et al., 2017; Semeniuta et al., 2017; Shen et al., 2018). Bowman et al. (2016) found that while employing an LSTM-based VAE for text, the posterior distribution of latent codes tends to match the prior distribution regardless of the input sequence (the KL divergence between the two distributions is very close to zero). Consequently, the information from the latent variable is not leveraged by the generative network (Bowman et al., 2016). To mitigate this issue, several strategies have been proposed (see optimization challenges in Section 2) to make the decoder less autoregressive; thus, less contextual information is utilized by the decoder network (Yang et al., 2017; Shen et al., 2018). However, reducing the autoregressive power of the decoder to make the latent variable more informative can be suboptimal from a text-generation perspective, since the coherence of the generated paragraphs may be sacrificed.
Motivated by these observations, we propose a hierarchically-structured variational autoencoder (hier-VAE), a novel variational approach for generating long sequences of text. To improve the model’s plan-ahead capability for capturing long-term dependency, our approach exploits a hierarchical LSTM decoder as the generative network. Specifically, a set of intermediate sentence-level representations are introduced during the decoding process (via a sentence-level LSTM network), providing additional semantic information while making word-level predictions.
To ameliorate the posterior collapse issue associated with training a VAE for text, we further propose leveraging a hierarchy of latent variables between the convolutional inference networks and recurrent generative networks. With multiple stochastic layers, where the prior of each latent variable is inferred from data, rather than fixed (as in a standard VAE setting (Kingma & Welling, 2013)), the sampled latent codes at the bottom level are endowed with more flexibility to abstract meaningful features from the input sequences. We evaluate the proposed hier-VAE comprehensively on language modeling, generic (unconditional) text generation, and conditional generation tasks. The proposed model demonstrates substantial improvement relative to several baseline methods, in terms of perplexity on language modeling and quality of generated samples (measured by both the BLEU statistics and human evaluation). We further generalize our method to the conditional generation scenario to demonstrate the versatility of the proposed method.

heading: 2 RELATED WORK
text: VAE for text generation The variational autoencoder, trained under the neural variational inference (NVI) framework, has been widely used for generating text sequences (Bowman et al., 2016; Yang et al., 2017; Semeniuta et al., 2017; Zhao et al., 2017). By encouraging the latent feature space to match a prior distribution within an encoder-decoder architecture, the learned latent variable could potentially encode high-level semantic features and serve as a global representation during the decoding process (Bowman et al., 2016). The generated results are also endowed with better diversity due to the sampling procedure of the latent codes (Zhao et al., 2017). However, existing works have mostly focused on generating one sentence (or multiple sentences with at most twenty words in total). The task of generating relatively longer units of text has been less explored. Our proposed architecture seeks to extend the NVI framework for long-form text generation by exploiting the hierarchical nature of a paragraph (multi-sentence text sequences), where the outputs from a sentence-level LSTM are utilized to guide word-level predictions.
Optimization challenges with text-VAEs The “posterior collapsing” issue associated with training text-VAEs was first outlined by (Bowman et al., 2016), where they have utilized KL divergence annealing and word dropout strategies to mitigate the optimization challenge. However, their resulting method still does not outperform a pure language model in terms of the final perplexity. Yang et al. (2017) argue that the small KL term relates to the strong autoregressive nature of an LSTM generative network, and they proposed to utilize a dilated CNN as a decoder to improve the informativeness of the latent variable. (Zhao et al., 2018b) proposed to augment the VAE training objective with an additional mutual information term. Unfortunately, such a strategy results in an intractable integral in the case where the latent variables are continuous. Our work tries to tackle the “posterior
collapse” issue from two perspectives: 1) more flexible priors are assumed over the latent variables (learned from the data); 2) the hierarchical structure within a paragraph is taken into account, so that the latent variables can focus less on the local information (e.g., word-level coherence) but more on the global features.
Hierarchical structures in NLP Natural language is inherently organized in a hierarchical manner (characters form a word, words form a sentence, sentences form a paragraph, paragraphs from a document, etc.). Yang et al. (2016) employed hierarchical LSTM encoders at the word- and sentence-level along with an attention mechanism to learn document representations. Li et al. (2015) proposed a hierarchical autoencoder to reconstruct long paragraph of texts. One method conceptually similar to our work is that of (Serban et al., 2017), which produces a stochastic latent variable for each sentence during decoding. In contrast, our model encodes the entire paragraph into one single latent variable. As a result, the latent variable learned in our model relates more to the global semantic information of a paragraph, whereas those in (Serban et al., 2017) mainly contain the local information of a specific sentence. Therefore, their model is not suitable for tasks such as latent space interpolation. Our approach bears close resemblance to the VHCR model (Park et al., 2018). However, there exist several key differences that make our work unique: i) both latent variables in our hier-VAE are designed to contain global information. More importantly, although the local/utterance variables are generated from the global latent variable in VHCR, the priors for the two sets of latent variables are both fixed as standard diagonal-covariance Gaussian. In contrast, the prior of the bottom-level latent variable in our model is learned from the data (thus more flexible relative to a fixed prior), which exhibits promising results in terms of mitigating the “posterior collapse” issue (see Table 2); ii) the underlying data distribution of the entire paragraph is captured in the bottomlevel latent variable in hier-VAE. While in the setup of VHCR, the responses are modeled/generated conditioned on both the latent variables and the contexts. Therefore, the (global) latent variable learned by our model should contain more information. Modeling hierarchical structures has been shown to be beneficial for generating others types of data as well, such as images (Sønderby et al., 2016; Gulrajani et al., 2016), music (Roberts et al., 2018), or speech (Hsu et al., 2017).

heading: 3 HIERARCHICALLY-STRUCTURED VAE MODEL
text: Our goal is to build a model that can generate human-generated-like sentences conditioned on random samples from the (prior) latent space. The proposed method, as illustrated in Figure 1, is built upon the VAE framework introduced in (Bowman et al., 2016), and consists of a hierarchical CNN inference network (encoder) and a hierarchical LSTM generative network (decoder). The encoder attempts to extract features at different semantic levels, which are then leveraged by the decoder to allow long-term planning while synthesizing long-form text. We define long-form text as sequences of sentences (such as a Yelp review on a business, or an abstract of an article) that are arranged in a logical manner and usually follow a definite plan for development.

heading: 3.1 VARIATIONAL AUTOENCODER
text: Recurrent neural network (RNN) language models (Mikolov et al., 2011), which predict each token conditioned on the entire history of previously generated tokens, have been widely employed for unsupervised generative modeling of natural language. Although effective, they typically do not explicitly capture the global (high-level) features (e.g., topic or style properties) of a paragraph or sentence. Motivated by this, the variational autoencoder (VAE) was introduced by (Bowman et al., 2016) as an alternative for generative text modeling, where a stochastic latent variable is augmented as the additional information to modulate the sequential generation process. Let x denote a text sequence, which consists of L tokens, i.e., x1, x2, ..., xL. A continuous latent code z is first sampled from the prior distribution p(z) (a multivariate Gaussian prior is typically chosen), and then the text sequence x is generated, conditioned on z, via a generative (decoder) network. Therefore, the generating distribution can be written as:
pθ(x|z) = pθ(x1|z) L∏ t=2 pθ(xt|x1, x2, ..., xt−1, z) (1)
where θ are the parameters of the generative network (to be learned). Since the information from latent variable z is incorporated during the entire generation process, the high-level properties of the corresponding paragraph or sentence could be better leveraged (Bowman et al., 2016; Yang et al., 2017). To estimate the parameters θ, one would ideally maximize the marginal distribution
Lower-level CNN encoder
Higher-level CNN encoder
z1
z2
z1
z2
…. ….
Sentence-level LSTM decoder
Word-level LSTM decoder ….
KL losses
Figure 1: Schematic diagram of the proposed hierarchically-structured VAE.
p(x) = ∫
p(z)p(x|z)dz. However, computing this marginal is intractable in most cases of practical interest. Instead, we maximize a variational lower bound, as typically employed in the VAE framework (Kingma & Welling, 2013): Lvae = Eqφ(z|x)[log pθ(x|z)] − DKL(qφ(z|x)||p(z)). Note that qφ(z|x) is employed to approximate the true posterior, which is usually called the inference or encoder network (parameterized by φ). The Kullback-Leibler (KL) divergence term DKL(qφ(z|x)||p(z)), which can be written in closed-form (Kingma & Welling, 2013), encourages the approximate posterior distribution qφ(z|x) to be close to the prior p(z). Note that the VAE prior is often taken to be a diagonal-covariance Gaussian for convenience. Although the NVI framework has been shown effective for various NLP tasks, directly applying it to long text generation is still challenging. The reasons are two-fold: (1) the lack of a long-term planning mechanism, which is critical for generating semantically-coherent long texts (Serdyuk et al., 2017); (2) the posterior collapse issue. Concerning (2), it was demonstrated in (Bowman et al., 2016) that due to the autoregressive nature of the RNN, the decoder tends to ignore the information from z entirely, resulting in an extremely small KL term.

heading: 3.2 HIERARCHICAL LSTM-BASED GENERATIVE NETWORK
text: To improve the plan-ahead ability of the generative network, we introduce intermediate sentence representations to facilitate the sequential generation process. Instead of directly making word-level predictions merely conditioned on the semantic information from z, a series of plan vectors are first generated based upon z with a sentence-level LSTM decoder. The hypothesis underlying this design is that explicitly incorporating the knowledge of (inherently hierarchical) paragraph structure is beneficial for the model to capture sentence-level coherence and potentially mitigate repetitiveness. Intuitively, while predicting each token, the information from both previously generated words and sentence-level representations can be utilized (see Figure 1).
Assume that the input paragraph consists of M sentences, and the t-th sentence contains Nt words (for t = 1, 2, ...,M ). To generate the plan vectors, the sampled latent code z is first sent through a one-layer multi-layered perceptron (MLP), with ReLU activation functions, to obtain the starting state of the sentence-level LSTM decoder. Subsequently, the representations for each sentence, the plan vectors are inferred in a sequential manner: hst = LSTM
sent(hst−1, z), for t = 1, 2, 3, ...,M . Note that z is fed to the higher-level LSTM at every time step to predict the sentence representations. This is because the latent code can be regarded as a paragraph-level abstraction, and thus is informative for determining the semantics of each generated subsequence.
The generated sentence-level plan vectors are then passed through a word-level LSTM decoder to generate the words for each sentence. Similar to the sentence-level decoder, the corresponding plan vector, hst , is concatenated with the word embedding of the previous word and fed to the LSTM at every time step (see Figure 1). Let wt,i denote the i-th token in the t-th sentence (teacher forcing is employed during training, and we use greedy decoding at test time). This process can be expressed as (for t = 1, 2, ...,M and i = 1, 2, 3, ..., Nt):
hwt,i = LSTM word(hwt,i−1,h s t ,We[wt,i−1]), (2)
p(wt,i|wt,<i,hst ) = softmax(V h w t,i), (3)
hwt,0 is inferred from the corresponding plan vector via an MLP layer. Weight matrix V is used for computing a distribution over words, and We are word embeddings to be learned. For each sentence, once the special END token is generated, the word-level LSTM stops decoding (note that each sentence is padded with an END token at the preprocessing step). The parameters of the word-level decoder for each sentence are shared, and are trained jointly with the parameters of sentence-level LSTM.
Inspired by Chen et al. (2016), we hypothesize that with the hierarchical decoder networks described above, the latent codes are encouraged to abstract more global and high-level semantic information from the paragraph. While relatively more detailed information (e.g., word-level (local) coherence) can be captured via the word- and sentence-level LSTM networks, the latent variables can focus more on the global information (e.g., paragraph-level semantic features) (Chen et al., 2016). Empirically, we also found that this strategy helps the generative network to make more effective use of the latent variable z (indicated by a larger KL loss term).

heading: 3.3 HIERARCHICALLY-STRUCTURED LATENT VARIABLES
text: Due to the use of an autoregressive LSTM decoder for word-level predictions, the problem of “posterior collapse”, where the model tends to ignore the latent codes while decoding, may still exist. In this work, we consider one effective way to mitigate this problem by making less restrictive assumptions regarding the prior distribution of the latent variable. Specifically, we propose to leverage a hierarchy of latent variables for text modeling. As shown in Figure 1, the inference network goes upward through each latent variable to infer their posterior distributions, while the generative network samples downward to obtain the prior distributions over the latent variables. In this manner, the prior distribution of latent variable at the bottom is inferred from the top-layer latent codes, rather than fixed (as in a standard VAE model). Consequently, the model is endowed with more flexibility to abstract useful high-level features (Gulrajani et al., 2016), which can then be leveraged by the hierarchical LSTM network. Without loss of generality, here we choose to employ a two-layer hierarchy of latent variables, where the bottom and top ones are denoted as z1 and z2, respectively. Note that this framework can be extended easily to more latent-variable layers. Empirically, we found that the two-layer setup already gives rise to much more informative latent codes, relative to a standard VAE with one stochastic layer.
The posterior distributions over the latent variables are assumed to be conditionally independent given the input x. Thus, the joint posterior distribution of the two latent variables can be written as:
qφ(z1, z2|x) = qφ(z2|x)qφ(z1|x) (4)
Concerning the generative network, the latent variable at the bottom is sampled conditioning on the one at the top. Thus, we have:
pθ(z1, z2) = pθ(z2)pθ(z1|z2) (5)
To optimize the parameters of the inference and generative networks, the second term in the VAE objective, DKL(qφ(z|x)||p(z)), can be regarded as the KL divergence between the joint posterior and prior distributions of the two latent variables. Thus, the variational lower bound can be written as (given the assumptions in (4) and (5)):
Lvae = Eqφ(z1|x)[log pθ(x|z1)]−DKL(qφ(z1, z2|x)||pθ(z1, z2)) (6)
DKL = ∫ z1,z2 qφ(z2|x)qφ(z1|x) log qφ(z2|x)qφ(z1|x) pθ(z2)pθ(z1|z2) dz1dz2
= ∫ z1,z2 [qφ(z2|x)qφ(z1|x) log qφ(z1|x) pθ(z1|z2) + qφ(z2|x)qφ(z1|x) log qφ(z2|x) pθ(z2) ]dz1dz2 = Eqφ(z2|x)[DKL(qφ(z1|x)||pθ(z1|z2))] +DKL(qφ(z2|x)||pθ(z2)) (7)
Note that the left-hand side of (7) is the abbreviation of DKL(qφ(z1, z2|x)||p(z1, z2)). Given the Gaussian assumption for both the prior and posterior distributions, both KL divergence terms can be written in closed-form.

heading: 4 EXPERIMENTS

heading: 4.1 EXPERIMENTAL SETUPS
text: Datasets We evaluate the proposed hierarchically-structured VAE on both generic (unconditional) long text generation and conditional paragraph generation (with additional text input as auxiliary information). For the former, we employ two datasets: Yelp Reviews (Zhang et al., 2015) and ArXiv Abstracts (Celikyilmaz et al., 2018). For the conditional-generation case, we consider the task of synthesizing a paper abstract (which typically includes several sentences) conditioned on
the paper title (with the ArXiv Abstract dataset). We employ a hierarchical CNN architecture as the inference/encoder network. Specifically, a sentence-level CNN encoder is first applied to each sentence to obtain a fixed-length vector. Afterwards, a paragraph-level CNN encoder is utilized to aggregate the vectors with respect to all sentences. Details of the dataset, the experimental setup and model architectures are provided in the Supplementary Materials (SM).
Model specification To abstract meaningful representations from the input sentences, we employ a hierarchical CNN architecture as the inference/encoder network. Specifically, a sentencelevel CNN encoder is first applied to each sentence to obtain a fixed-length vector. Afterwards, a paragraph-level CNN encoder is utilized to aggregate the vectors with respect to all sentences. In the single-variable hier-VAE, the paragraph feature vector is fed into linear layers to infer the mean and variance of the latent variable z. For the double-variable case, the feature vector is further transformed with two MLP layers, and then is used to compute the mean and variance of the top-level latent variable. The dimension of latent variable z is set to 300. The dimensions of both the sentence-level and word-level LSTM decoders are set to 512. For the generative networks, to infer the bottom-level latent variable (i.e., modeling p(z1|z2)), we first feed the sampled latent codes from z2 to two MLP layers, which is followed by two linear transformation to infer the mean and variance of z1, respectively.
The model is trained using Adam (Kingma & Ba, 2014) with a learning rate of 3 × 10−4 for all parameters, with a decay rate of 0.99 for every 3000 iterations. Dropout (Srivastava et al., 2014) is employed on both word embedding and latent variable layers, with rates selected from {0.3, 0.5, 0.8} on the validation set. We set the mini-batch size to 128. Following (Bowman et al., 2016) we adopt the KL cost annealing strategy to stabilize training: the KL cost term is increased linearly to 1 until 10,000 iterations. All experiments are implemented in Tensorflow (Abadi et al., 2016), using one NVIDIA GeForce GTX TITAN X GPU with 12GB memory.
Baselines For the language modeling experiments, we implement several baselines: language model with a flat LSTM decoder (flat-LM), VAE with a flat LSTM decoder (flat-VAE), language model with a hierarchical LSTM decoder (hier-LM). For generic text generation, we further consider two recently proposed generative models as the baselines: Adversarial Autoencoders (AAE) (Makhzani et al., 2015) and Adversarially-Regularized Autoencoders (ARAE) (Zhao et al., 2018a). Instead of penalizing the KL divergence term, AAE introduces a discriminator network to match the prior and posterior distributions of the latent variable. Further, AARE proposes to learn a more flexible prior by assuming no specific forms of the latent space. For the proposed hierarchicallystructured VAE, we implement two variants, one with a single latent variable and another with double latent variables, denoted as hier-VAE-S and hier-VAE-D, respectively. Our code will be released to encourage future research.

heading: 4.2 LANGUAGE MODELING RESULTS
text: We first evaluate our method on language modeling task using Yelp and ArXiv datasets, where we report the negative log likelihood (NLL) and perplexity (PPL). Following (Bowman et al., 2016; Yang et al., 2017; Kim et al., 2018), we utilize the KL loss term to measure the extent of “posterior collapse”. For this experiment we use the following baselines: language model with a flat LSTM decoder, VAE with a flat LSTM decoder, language model with a hierarchical LSTM decoder, VAE with a hierarchical LSTM decoder. As shown in Figure 2, on the Yelp dataset, the standard textVAE with a flat LSTM decoder has a KL divergence term very close to zero, indicating that the generative model makes neligible use of the information from latent variable z. Consequently, the flat-VAE model obtains slightly worse NNL and PPL relative to a flat LSTM-based language model. In contrast, with a hierarchical LSTM decoder, the KL divergence cost term becomes much larger, demonstrating that the VAE model tends to leverage more information from the latent variable in the decoding stage. Further, the PPL of hier-VAE is also decreased from 47.9 to 46.6 (compared with a LM with hierarchical decoder), further showing that the sampled latent codes has indeed helped make word-level predictions.
Moreover, with a hierarchy of latent variables, the hier-VAE-D model exhibits an even larger KL divergence cost term (increased from 3.6 to 6.8) than that with a single latent variable, indicating that more information from the latent variable has been utilized by the generative network. This may be attributed to the fact that the latent variable priors of the hier-VAE-D model are inferred from the data, rather than a fixed standard Gaussian distribution. As a result, the model is endowed with more flexibility to encode informative semantic features in the latent variables, yet matching their
posterior distributions to the corresponding priors. More importantly, by effectively exploiting the sampled latent codes, hier-VAE-D achieves the best PPL results on both datasets (on ArXiv dataset our hierarchical decoder outperforms the LM by reducing the PPL from 58.1 down to 54.3).

heading: 4.3 GENERIC TEXT GENERATION
text: We further evaluate our method by examining the quality of generated paragraphs. In this regard, we randomly sample 1000 latent codes and obtain the generated text sequences by feeding them to the trained generative networks. We employ corpus-level BLEU score (Papineni et al., 2002)
to quantitatively evaluate the generated paragraphs. Specifically, we follow the strategy in (Yu et al., 2017; Zhang et al., 2017) and use the entire test set as the reference. The BLEU scores for the 1000 generated sentences are averaged to obtain the final score for each model. As shown in Table 5, VAE tends to be a stronger baseline for paragraph generation, exhibiting higher corpus-level BLEU scores than both AAE and ARAE. This observation is consistent with the results shown in (Cı́fka et al., 2018). The VAE with hierarchical decoder again demonstrates better BLEU scores than the one with a flat decoder, indicating that the plan-ahead mechanism associated with the hierarchical decoding process indeed benefits the sampling quality. Moreover, hier-VAE-D exhibits slightly better results than hier-VAE-S. We attribute this to the more flexible prior distribution of hier-VAE-D, which improves the ability of the inference networks to extract semantic features from a paragraph and thus yields more informative latent codes.
Table 3 shows an example generated text from hier-VAE-H and flat-VAE-H. Compared to the our hierarchical model, using flat decoders with flat VAE architecture exibits n-gram repetitions as well as less variations in the descriptions. The hieararhical model, on the other hand, contains more information with less repetitions (word or semantic semantic repetitions) indicating a more coherent generated text.
The Continuity of Latent Space Following (Bowman et al., 2016), we further measure the continuity of the learned latent space. Specifically, two points are randomly sampled from the prior latent space (denoted as A and B, respectively). Sentences are generated based on the equidistant intermediate points along the linear trajectory between A and B. As shown in Table 4, these intermediate samples are all realistic-looking reviews that are syntactically and semantically reasonable, demonstrating the smoothness of the learned VAE latent space. Interestingly, we even observe that the generated sentences gradually transit from positive to negative sentiment along the linear trajectory. To validate that the sentences are not generated by simply retrieving the training data, we further find the closest instance, among the entire training set, for each generated review. We demonstrate the details of the results in the supplementary material (Table 15).
Diversity of Generated Paragraphs We also evaluate the diversity of random samples from a trained model, since one model might generate realistic-looking sentences while suffering from severe mode collapse (i.e., low diversity). Three metrics are employed to measure the diversity of generated paragraphs: Self-BLEU scores (Zhu et al., 2018), unique n-grams (Fedus et al., 2018) and entropy score (Zhang et al., 2018). Specifically, for a set of sampled sentences, the Self-BLEU
metric calculates the BLEU score of each sample with respect to all other samples as the reference (the numbers over all samples are then averaged); the unique score computes the percentage of unique n-grams within all the generated reviews; and the entropy score measures how evenly the empirical n-gram distribution is for a given sentence, which does not depend on the size of testing data, as opposed to unique scores. Note that all three metrics are the lower, the better.
Here, we randomly sample 1000 reviews from each model, and the corresponding results are shown in Table 6. Note that a small self-BLEU score must be accompanied with a large BLEU score to justify the effectiveness of a model, i.e., being able to generate realistic-looking as well as diverse samples. Among all the VAE variants, hier-VAE-D shows the smallest BLEU score and largest unique n-grams percentage, further demonstrating the advantages of making both the generative networks and latent variables hierarchical. As for AAE and ARAE, although they exhibit better diversity according to both metrics, their corpus-level BLEU scores are much worse relative to hier-VAE-D. Thus,
we leverage human evaluation for further comparison.
color of each point indicates the category it belongs to. The embeddings of the same label are indeed very close in the 2-D plot, while those with different labels are relatively farther away from each other. Additionally, the embeddings of the abstracts belonging to the High Energy Physics and Nuclear topics are meshed, which is expected since these two topics are semantically highly related.
Note that the model is trained in a totally unsupervised manner, and the inference network is able to extract meaningful global patterns from the input paragraph.
Attribute Vector Arithmetic To further investigate the latent space’s structure, we conduct an experiment to alter the sentiments of reviews with attribute vector. Specifically, on the Yelp Review dataset, we first obtain the sampled latent codes for all reviews with positive sentiment (among the entire training set), and calculate the corresponding mean latent vector. The mean latent vector for all negative reviews are computed in the same manner. The two vectors are then subtracted (i.e., positive mean vector minus negative mean vector) to obtain the “sentiment attribute vector”. For evaluation, we randomly sample 1000 reviews with negative sentiment and add the “sentiment attribute vector” to their latent codes. The manipulated latent vectors are then fed to the hierarchical decoder to produce the transferred sentences (which should hopefully convey positive sentiment).
As shown in Table 8, the original sentences have been successfully manipulated to positive sentiment with the simple attribute vector operation. However, the specific contents of the reviews are not fully retained. One interesting future direction is to decouple the style and content of long-form texts to allow content-preserving attribute manipulation. We further employed a CNN sentiment classifier to evaluate the sentiment of manipulated sentences. The classifier is trained on the entire training set and achieves a test accuracy of 94.2%. With this pre-trained classifier, 83.4% of the transferred reviews are judged to be positive-sentiment, indicating that “attribute vector arithmetic” consistently produces the intended manipulation of sentiment.
Human Evaluation We conducted human evaluation using Amazon Mechanical Turk to assess the coherence and non-redundancy of the texts generated from our models in comparison to the baselines, which is difficult to measure based on automated metrics. Given a pair of generated reviews, the judges are asked to select their preferences (no difference between the two reviews is also an option) according to the following four evaluation criteria: fluency & grammar (G), consistency (C), non-redundancy (N), and overall (O). Details of the evaluation are provided in the supplementary material.
As shown in Table 9, hier-VAE generates superior human-looking samples compared to flat-VAE on the Yelp Reviews dataset. Even though both models underperfom when compared against the ground-truth real re-
views, hier-VAE was rated higher in comparison to flat-VAE (raters find hier-VAE closer to humangenerated than the flat-VAE) in all the criteria evaluation criteria. We further compare our methods against AAE (the same data preprocessing steps and hyperparameters are employed). The results show that hier-VAE again produces more grammatically-correct and semantically-coherent samples relative to the AAE baseline.

heading: 4.4 CONDITIONAL PARAGRAPH GENERATION
text: We further evaluate the proposed VAE model on a conditional generation task. Specifically, we consider the task of generating abstract of a paper based on the corresponding title. The same arXiv dataset is utilized, where title and abstract are given as paired text sequences. The title is used as input of the inference network. For the generative network, instead of reconstructing the same input (i.e., title), the paper abstract is employed as the target for decoding. We compare the hier-VAE-D model against hier-LM. We observe that the hier-VAE-D model achieves a test perplexity of 55.7 (with a KL term of 2.57), which is smaller that the test perplexity of hier-LM (58.1). This indicates that the information from the title has indeed been leveraged by the generative network to facilitate the decoding process. In table 10 we show two generated samples from the hier-VAE-D model.

heading: 4.5 ANALYSIS

heading: 4.5.1 THE ARCHITECTURE OF ENCODER NETWORKS
text: To investigate the impact of encoder networks on the VAE’s performance, we further conduct an ablation study, where the herarchical CNN encoder in the hier-VAE model is replaced with a flat CNN encoder or a hierarchical LSTM encoder, respectively. The corresponding results are shown in Table 11. It
can be observed that the model with a flat CNN encoder yields worst (largest) perplexity, suggesting that it is beneficial to make the encoder hierarchical. Additionally, hierarchical CNN encoder exhibits slightly better results than hierarchical LSTM encoder according to our experiments.

heading: 4.5.2 MITIGATING “POSTERIOR COLLAPSE” ISSUE
text: To resolve the “posterior collapse” issue of training textual VAEs, Park et al. (2018) also introduced a strategy called utterance drop (u.d). Specifically, they proposed to weaken the autoregressive power of hierarchical RNNs by dropping the utterance encoder vector with a certain probability. To investigate the effec-
tiveness of their method relative to our strategy of employing a hierarchy of latent variables, we further conduct a comparative study. Particularly, we utilize hier-VAE-S as the baseline model and apply the two strategies to it respectively. The corresponding results on language modeling (Yelp dataset) are shown in Table 12. Their u.d strategy indeed allows better usage of the latent variable (indicated by a larger KL divergence value). However, the NLL of the language model becomes even worse, possibly due to the weakening of the decoder during training (similar observations have also been reported in Table 2 of (Park et al., 2018)). In contrast, our hierarchical prior strategy yields larger KL terms as well as lower NNL value, indicating the advantage of our strategy to mitigate the “posterior collapse” issue.

heading: 5 CONCLUSION
text: We introduced a hierarchically-structured variational autoencoder for long text generation. A hierarchical LSTM generative network is employed, which models the semantic coherence at both the word- and sentence-levels. hierarchy of stochastic layers is further utilized, where the priors of the latent variables are learned from the data. Consequently, more informative latent codes are manifested, indicated by a larger KL loss term yet smaller variational lower bound. The generated samples from the proposed model also exhibit superior quality relative to those from several baseline methods (according to automatic metrics). Human evaluations further demonstrate that the samples from hier-VAE are less repetitive and more semantically-consistent.

heading: A DATASETS DETAILS
text: In the following, we provide details of dataset preprocessing as well as the experimental setups used in the experiments. For both Yelp Reviews and ArXiv Abstracts datasets, we truncate the original paragraph to the first five sentences (split by punctuation marks including comma, period and point symbols), where each sentence contains at most 25 words. Therefore, each paragraph has at most 125 words. We further remove those sentences that contain of less than 30 words. The statistics of both datasets are detailed in Table 13. Note that the average length of paragraphs considered here are much larger than previous generative models for text (Bowman et al., 2016; Yu et al., 2017; Hu et al., 2017; Zhang et al., 2017), since these works considered text sequences that contain only one sentence with at most twenty words.
B ADDITIONAL GENERATED SAMPLES FROM hier-VAE-D VS flat-VAE
We provide additional examples for the comparison between hier-VAE-D vs flat-VAE in Table 14, as a continuation of Table 3.

heading: C RETRIEVED CLOSEST TRAINING INSTANCES OF GENERATED SAMPLES (YELP REVIEWS DATASET)
text: We provide samples of retrieved instances from the Yelp Review training dataset which are closest to the generated samples. Table 15 shows the closest training samples of each generated Yelp review. The first column indicates the intermediate generated sentences produced from linear transition from a point A to another point B in the prior latent space. The second column on the right are the real sentences retrieved from the training set that are closest to the ones generated on the left (determined by BLEU-2 score). We can see that the retrieved training data is quite different from the generated samples, indicating that our model is indeed generating samples that it has never seen during training.

heading: D HUMAN EVALUATION SETUP AND DETAILS
text: Some properties of the generated paragraphs, such as (topic) coherence or non-redundancy, can not be easily measured by automated metrics. Therefore, we further conduct human evaluation based on 100 samples randomly generated by each model (the models are trained on the Yelp Reviews dataset for this evaluation). We consider flat-VAE, adversarial autoencoders (AAE) and real samples from the test set to compare with our proposed hier-VAE-D model. The same hyperparameters are employed for the different model variants to ensure fair comparison. We evaluate the quality of these generated samples with a blind heads-up comparison using Amazon Mechanical Turk. Given a pair of generated reviews, the judges are asked to select their preferences (“no difference between the two reviews” is also an option) according to the following 4 evaluation criteria: (1) fluency & grammar, the one that is more grammatically correct and fluent; (2) consistency, the one that depicts a sequence of topics and events that is more consistent; (3) non-redundancy, the one that is better at non-redundancy (if a review repeats itself, this can be taken into account); and (4) overall, the one that more effectively communicates reasonable content. These different criteria help to quantify the impact of the hierarchical structures employed in our model, while the non-redundancy and consistency metrics could be especially correlated with the model’s plan-ahead abilities. The generated paragraphs are presented to the judges in a random order and they are not told the source of the samples. Each sample is rated by three judges and the results are averaged across all samples and judges.

heading: E MORE SAMPLES ON ATTRIBUTE VECTOR ARITHMETIC
text: We provide more samples for sentiment manipulation, where we intend to alter sentiment of negative Yelp reviews with “attribute vector arithmetic”, as a continuation of Table 8.

Review summaries:
This paper proposed a hierarchical generative model for generating long text. The authors use a hierarchical LSTM decoder to first generate sentence-level representations; then based on the representation of each sentence, a word-level LSTM decoder is utilized to generate a sequence of words in this sentence. In addition, they use multiple layers of latent variables to address the posterior collapse issue.