Hierarchically-Structured Variational Autoencoders for Long Text Generation

1 INTRODUCTION . Probabilistic generative models for text have received considerable attention recently , in part because of their ability to leverage abundant unlabeled data and learn text representations with strong generalization abilities . Combined with the flexibility of deep neural networks , generative modeling provides a powerful and effective framework to estimate the underlying probability distribution of sentences or documents , and to potentially capture the rich sequential information inherent in natural language . In this work , we focus on one specific type of generative model : the variational autoencoder ( VAE ) ( Kingma & Welling , 2013 ) . The VAE maps a text sequence into a continuous latent variable , or code , via an inference ( encoder ) network ; a generative ( decoder ) network is utilized to reconstruct the input sentence , conditioned on samples from the latent code ( via its posterior distribution ) . The inference and generative networks , parametrized by deep neural networks , can be optimized jointly by maximizing a variational lower bound of the marginal distribution of the training corpus . The advantages of leveraging the neural variational inference ( NVI ) framework have been demonstrated on a wide range of natural language processing ( NLP ) tasks ( Bowman et al. , 2016 ; Miao et al. , 2016 ; Yang et al. , 2017 ; Serban et al. , 2017 ; Shah & Barber , 2018 ; Kim et al. , 2018 ; Kaiser et al. , 2018 ; Bahuleyan et al. , 2018 ; Deng et al. , 2018 ) . Although NVI-based methods have demonstrated success in modeling ( generating ) relatively short sentences ( typically less than twenty words ) , there are several challenges that need to be addressed to facilitate long text generation with deep latent-variable models . The first challenge originates from the nature of recurrent neural networks ( RNNs ) , which have become the cornerstone for text generation . RNNs are typically trained with gradient descent , using backpropagation through time ( BPTT ) ( Rumelhart et al. , 1986 ) . However , it is very difficult to scale such a training paradigm to long sequences , due to vanishing or exploding gradients ( Hochreiter & Schmidhuber , 1997 ) . Moreover , the information of the entire generated sequence needs to be stored in the RNN intermediate hidden states , which could be very demanding for RNNs with the increasing length of generated texts . As shown in Table 1 , the sentence sampled from a vanilla textVAE model ( Bowman et al. , 2016 ) tends to repeat itself as the sequence proceeds , leading to inferior generation results . In addition to the issue of repetitiveness , to generate globally-coherent long text sequences , it is desirable that both the higher-level abstract features and lower-level concrete details ( e.g. , specific word choices ) of longer texts can be leveraged by the generative network . This intuition is also hard to capture with a single-layer ( flat ) RNN-based generative network . Another underlying challenge of generating long text relates to the “ posterior collapse ” issue inherent in training variational autoencoders with a strong autoregressive decoder ( Bowman et al. , 2016 ; Yang et al. , 2017 ; Semeniuta et al. , 2017 ; Shen et al. , 2018 ) . Bowman et al . ( 2016 ) found that while employing an LSTM-based VAE for text , the posterior distribution of latent codes tends to match the prior distribution regardless of the input sequence ( the KL divergence between the two distributions is very close to zero ) . Consequently , the information from the latent variable is not leveraged by the generative network ( Bowman et al. , 2016 ) . To mitigate this issue , several strategies have been proposed ( see optimization challenges in Section 2 ) to make the decoder less autoregressive ; thus , less contextual information is utilized by the decoder network ( Yang et al. , 2017 ; Shen et al. , 2018 ) . However , reducing the autoregressive power of the decoder to make the latent variable more informative can be suboptimal from a text-generation perspective , since the coherence of the generated paragraphs may be sacrificed . Motivated by these observations , we propose a hierarchically-structured variational autoencoder ( hier-VAE ) , a novel variational approach for generating long sequences of text . To improve the model ’ s plan-ahead capability for capturing long-term dependency , our approach exploits a hierarchical LSTM decoder as the generative network . Specifically , a set of intermediate sentence-level representations are introduced during the decoding process ( via a sentence-level LSTM network ) , providing additional semantic information while making word-level predictions . To ameliorate the posterior collapse issue associated with training a VAE for text , we further propose leveraging a hierarchy of latent variables between the convolutional inference networks and recurrent generative networks . With multiple stochastic layers , where the prior of each latent variable is inferred from data , rather than fixed ( as in a standard VAE setting ( Kingma & Welling , 2013 ) ) , the sampled latent codes at the bottom level are endowed with more flexibility to abstract meaningful features from the input sequences . We evaluate the proposed hier-VAE comprehensively on language modeling , generic ( unconditional ) text generation , and conditional generation tasks . The proposed model demonstrates substantial improvement relative to several baseline methods , in terms of perplexity on language modeling and quality of generated samples ( measured by both the BLEU statistics and human evaluation ) . We further generalize our method to the conditional generation scenario to demonstrate the versatility of the proposed method . 2 RELATED WORK . VAE for text generation The variational autoencoder , trained under the neural variational inference ( NVI ) framework , has been widely used for generating text sequences ( Bowman et al. , 2016 ; Yang et al. , 2017 ; Semeniuta et al. , 2017 ; Zhao et al. , 2017 ) . By encouraging the latent feature space to match a prior distribution within an encoder-decoder architecture , the learned latent variable could potentially encode high-level semantic features and serve as a global representation during the decoding process ( Bowman et al. , 2016 ) . The generated results are also endowed with better diversity due to the sampling procedure of the latent codes ( Zhao et al. , 2017 ) . However , existing works have mostly focused on generating one sentence ( or multiple sentences with at most twenty words in total ) . The task of generating relatively longer units of text has been less explored . Our proposed architecture seeks to extend the NVI framework for long-form text generation by exploiting the hierarchical nature of a paragraph ( multi-sentence text sequences ) , where the outputs from a sentence-level LSTM are utilized to guide word-level predictions . Optimization challenges with text-VAEs The “ posterior collapsing ” issue associated with training text-VAEs was first outlined by ( Bowman et al. , 2016 ) , where they have utilized KL divergence annealing and word dropout strategies to mitigate the optimization challenge . However , their resulting method still does not outperform a pure language model in terms of the final perplexity . Yang et al . ( 2017 ) argue that the small KL term relates to the strong autoregressive nature of an LSTM generative network , and they proposed to utilize a dilated CNN as a decoder to improve the informativeness of the latent variable . ( Zhao et al. , 2018b ) proposed to augment the VAE training objective with an additional mutual information term . Unfortunately , such a strategy results in an intractable integral in the case where the latent variables are continuous . Our work tries to tackle the “ posterior collapse ” issue from two perspectives : 1 ) more flexible priors are assumed over the latent variables ( learned from the data ) ; 2 ) the hierarchical structure within a paragraph is taken into account , so that the latent variables can focus less on the local information ( e.g. , word-level coherence ) but more on the global features . Hierarchical structures in NLP Natural language is inherently organized in a hierarchical manner ( characters form a word , words form a sentence , sentences form a paragraph , paragraphs from a document , etc. ) . Yang et al . ( 2016 ) employed hierarchical LSTM encoders at the word- and sentence-level along with an attention mechanism to learn document representations . Li et al . ( 2015 ) proposed a hierarchical autoencoder to reconstruct long paragraph of texts . One method conceptually similar to our work is that of ( Serban et al. , 2017 ) , which produces a stochastic latent variable for each sentence during decoding . In contrast , our model encodes the entire paragraph into one single latent variable . As a result , the latent variable learned in our model relates more to the global semantic information of a paragraph , whereas those in ( Serban et al. , 2017 ) mainly contain the local information of a specific sentence . Therefore , their model is not suitable for tasks such as latent space interpolation . Our approach bears close resemblance to the VHCR model ( Park et al. , 2018 ) . However , there exist several key differences that make our work unique : i ) both latent variables in our hier-VAE are designed to contain global information . More importantly , although the local/utterance variables are generated from the global latent variable in VHCR , the priors for the two sets of latent variables are both fixed as standard diagonal-covariance Gaussian . In contrast , the prior of the bottom-level latent variable in our model is learned from the data ( thus more flexible relative to a fixed prior ) , which exhibits promising results in terms of mitigating the “ posterior collapse ” issue ( see Table 2 ) ; ii ) the underlying data distribution of the entire paragraph is captured in the bottomlevel latent variable in hier-VAE . While in the setup of VHCR , the responses are modeled/generated conditioned on both the latent variables and the contexts . Therefore , the ( global ) latent variable learned by our model should contain more information . Modeling hierarchical structures has been shown to be beneficial for generating others types of data as well , such as images ( Sønderby et al. , 2016 ; Gulrajani et al. , 2016 ) , music ( Roberts et al. , 2018 ) , or speech ( Hsu et al. , 2017 ) . 3 HIERARCHICALLY-STRUCTURED VAE MODEL . Our goal is to build a model that can generate human-generated-like sentences conditioned on random samples from the ( prior ) latent space . The proposed method , as illustrated in Figure 1 , is built upon the VAE framework introduced in ( Bowman et al. , 2016 ) , and consists of a hierarchical CNN inference network ( encoder ) and a hierarchical LSTM generative network ( decoder ) . The encoder attempts to extract features at different semantic levels , which are then leveraged by the decoder to allow long-term planning while synthesizing long-form text . We define long-form text as sequences of sentences ( such as a Yelp review on a business , or an abstract of an article ) that are arranged in a logical manner and usually follow a definite plan for development . 3.1 VARIATIONAL AUTOENCODER . Recurrent neural network ( RNN ) language models ( Mikolov et al. , 2011 ) , which predict each token conditioned on the entire history of previously generated tokens , have been widely employed for unsupervised generative modeling of natural language . Although effective , they typically do not explicitly capture the global ( high-level ) features ( e.g. , topic or style properties ) of a paragraph or sentence . Motivated by this , the variational autoencoder ( VAE ) was introduced by ( Bowman et al. , 2016 ) as an alternative for generative text modeling , where a stochastic latent variable is augmented as the additional information to modulate the sequential generation process . Let x denote a text sequence , which consists of L tokens , i.e. , x1 , x2 , ... , xL . A continuous latent code z is first sampled from the prior distribution p ( z ) ( a multivariate Gaussian prior is typically chosen ) , and then the text sequence x is generated , conditioned on z , via a generative ( decoder ) network . Therefore , the generating distribution can be written as : pθ ( x|z ) = pθ ( x1|z ) L∏ t=2 pθ ( xt|x1 , x2 , ... , xt−1 , z ) ( 1 ) where θ are the parameters of the generative network ( to be learned ) . Since the information from latent variable z is incorporated during the entire generation process , the high-level properties of the corresponding paragraph or sentence could be better leveraged ( Bowman et al. , 2016 ; Yang et al. , 2017 ) . To estimate the parameters θ , one would ideally maximize the marginal distribution Lower-level CNN encoder Higher-level CNN encoder z1 z2 z1 z2 … . … . Sentence-level LSTM decoder Word-level LSTM decoder … . KL losses Figure 1 : Schematic diagram of the proposed hierarchically-structured VAE . p ( x ) = ∫ p ( z ) p ( x|z ) dz . However , computing this marginal is intractable in most cases of practical interest . Instead , we maximize a variational lower bound , as typically employed in the VAE framework ( Kingma & Welling , 2013 ) : Lvae = Eqφ ( z|x ) [ log pθ ( x|z ) ] − DKL ( qφ ( z|x ) ||p ( z ) ) . Note that qφ ( z|x ) is employed to approximate the true posterior , which is usually called the inference or encoder network ( parameterized by φ ) . The Kullback-Leibler ( KL ) divergence term DKL ( qφ ( z|x ) ||p ( z ) ) , which can be written in closed-form ( Kingma & Welling , 2013 ) , encourages the approximate posterior distribution qφ ( z|x ) to be close to the prior p ( z ) . Note that the VAE prior is often taken to be a diagonal-covariance Gaussian for convenience . Although the NVI framework has been shown effective for various NLP tasks , directly applying it to long text generation is still challenging . The reasons are two-fold : ( 1 ) the lack of a long-term planning mechanism , which is critical for generating semantically-coherent long texts ( Serdyuk et al. , 2017 ) ; ( 2 ) the posterior collapse issue . Concerning ( 2 ) , it was demonstrated in ( Bowman et al. , 2016 ) that due to the autoregressive nature of the RNN , the decoder tends to ignore the information from z entirely , resulting in an extremely small KL term .

Summary: This paper proposed a hierarchical generative model for generating long text. The authors use a hierarchical LSTM decoder to first generate sentence-level representations; then based on the representation of each sentence, a word-level LSTM decoder is utilized to generate a sequence of words in this sentence. In addition, they use multiple layers of latent variables to address the posterior collapse issue.