TopicGAN: Unsupervised Text Generation from Explainable Latent Topics

1 INTRODUCTION . Recently , deep generative models ( Goodfellow et al. , 2014 ; Kingma & Welling , 2013 ; Makhzani et al. , 2015 ) have achieved a great success on generating realistic images , videos and audio . Learning discrete representations of data and then generating data from the discovered representations have been increasingly studied , because the obtained discrete representations can benefit unsupervised learning ( Chen et al. , 2016 ; van den Oord et al. , 2017 ) , semi-supervised learning ( Odena et al. , 2016 ) , and few-shot learning . However , it remains extremely challenging for generating texts and learning interpretable discrete representations of texts due to the discrete , sparse and high dimensional properties of textual data . Arjovsky et al . ( 2017 ) mentioned that the original generative adversarial network ( GAN ) fails to generate discrete data due to the gradient vanishing problem . Wasserstein distance has been proposed to more precisely measure the distance between real and fake distributions and therefore tackles the gradient vanishing problem ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) . Another obstacle for generating discrete data is the non-differentiable function when generating words , which makes the gradients unable to be backpropagated from the discriminator to the generator . With the help of reinforcement learning , the generator is able to maximize the scores from the discriminator when the gradient is not able to flow from discriminator ( Yu et al. , 2017 ; Li et al. , 2017 ; Fedus et al. , 2018 ) . In natural language processing ( NLP ) , learning representations of texts is shown useful for unsupervised language understanding . While learning continuous text representations has been widely studied ( Kiros et al. , 2015 ; Arora et al. , 2017 ; Logeswaran & Lee , 2018 ) , learning high-level discrete representations has been explored by fewer work ( Zhao et al. , 2018 ; Miao et al. , 2016 ) . In order to take non-differentiable discrete variables as latent representations of an auto-encoder , some special methods such as Gumbel-Softmax ( Jang et al. , 2016 ) or vector quantisation ( van den Oord et al. , 2017 ) were applied by the prior work . Furthermore , because textual data is high dimensional and has rich but sometimes noisy information such as stop words , it is challenging to learn useful discrete representations of texts . To mitigate the difficulty of text generation and text discrete representation learning , this paper proposes TopicGAN , which simplifies those two problems by a two-step progressive generation . The idea of dividing the generation process into a pipeline yielded impressive results on high resolution image generation ( Zhang et al. , 2016 ; Karras et al. , 2018 ) . The progressive generation is a natural way to generate texts considering how human writes texts . When writing articles , human first considers the context of the texts and then organizes the context with correct grammar . Hence , in this work , we split text generation into two step , one is context generation and another is text generation from context with correct grammar . In the first step , we use topic model to discover the latent topics and use bag-of-words generator to produce bag-of-words according to the discovered latent topics and continuous noise . In the second step , based on the generated bag-of-words , we decode a sequential text by a recurrent neural network ( RNN ) . We utilize InfoGAN ( Chen et al. , 2016 ) to discover the latent topics and generate a bag of topical words without supervision in the first step , where the categorical classifier of InfoGAN can be considered as a topic model , which is able to discover latent topics of documents . We show that our model can yield explainable topics and outperform previous topic models such as latent Dirichlet allocation ( LDA ) ( Blei et al. , 2003 ) or variational topic model ( Miao et al. , 2016 ) for unsupervised document classification . Topic modeling can be applied to many applications , including extractive text summarization ( Titov & McDonald , 2008 ) , document retrieval ( Wei & Croft , 2006 ) or unsupervised classification . Also , unlike previous topic models that did not consider the word correlation during generating words , our method is able to regularize the correlation between words by the discriminator so that it yields more reasonable bag-of-words and produces high-quality texts . The contributions of this work are three-fold : • We propose two-step progressive text generation , which aligns well with the nature of text generation . • Our model is able to discover explainable topics by a topic classifier and achieves promising performance on unsupervised learning . • Compared to previous topic models , the proposed TopicGAN is able to capture the correlation between words , and thus performs better generation results . 2 RELATED WORK . Text Generation via GAN Prior work attempted at generating texts using GAN ( Yu et al. , 2017 ; Che et al. , 2017 ; Li et al. , 2017 ; Liu et al. , 2018 ) , and there are two main directions . One is to tackle the gradient vanishing problem in the original GAN , where JensenShannon divergence to evaluate the discrete real data distribution and the continuous generated distribution . By using Wasserstein distance , the discrete and continuous distributions can be properly measured . However , the Wasserstein distance requires the discriminator to be a Lipschitz continuous function ; therefore some restrictions including weight-clipping ( Arjovsky et al. , 2017 ) , gradient penalty ( Gulrajani et al. , 2017 ) are imposed on the discriminator . Another direction is to feed sampled discrete words from generated distribution to the discriminator . While the sampling operation is non-differentiable , reinforcement learning is applied to optimize the score from the discriminator . Some designed reward functions , such as Monte Carlo tree search ( MCTS ) ( Yu et al. , 2017 ) , are proposed to evaluate the generated word for each time step ( Lin et al. , 2017 ; Fedus et al. , 2018 ; Li et al. , 2017 ) . Our proposed progressive two-step generation framework can be easily combined with any current adversarial text generation models , because we can choose one those current work as our final jointly optimization method . Our framework effectively facilitates the training process of text generation by decomposing the task into two subproblems . InfoGAN InfoGAN has shown impressive performance for learning disentangled representations of images in an unsupervised manner ( Chen et al. , 2016 ) . The original GAN generates images from a continuous noise z , while each dimension of the noise does not contain disentangled features of generated images . To learn semantically meaningful representations , InfoGAN maximizes the mutual information between input code c and the generated output G ( z , c ) . However , maximizing the mutual information is intractable , because it requires the access of P ( c | G ( z , c ) ) . Based on variational information maximization , Chen et al . ( 2016 ) used an auxiliary distribution Q ( c | G ( z , c ) ) to approximate P ( c | G ( z , c ) ) . The auxiliary function Q can be a neural network that can be jointly optimized : min G , Q max D Ex∼Pdata [ log ( D ( x ) ) ] + Ez∼Pz , c∼Pc [ log ( 1−D ( G ( z , c ) ) ) − λQ ( c | G ( z , c ) ) ] , ( 1 ) where Pdata is the real data distribution , Pz is the noise distribution , and Pc is the code distribution . The code c can be either continuous or categorical . In our work , the code c is set to be categorical , and the categorical classifier Q becomes a topic model . Therefore , we call the third term in ( 1 ) categorical loss . 3 TOPICGAN . When using GAN for natural language generation , the generator has difficulty generating text with reasonable context and correct grammar simultaneously . In addition , when feeding a sequential text to the InfoGAN categorical code classifier , due to its complex structure , the classifier fails to discover meaningful discrete information . The key idea of this work is that we divide text generation into two steps : 1 ) generating bag-of-words that can roughly represent the topical information , and then 2 ) generating sequential texts from the learned topical words . As shown in Figure 1 , given a discrete topic c and a continuous noise z as the input , the bag-of-words generator Gbow generates topical words . After obtaining topical words , the sequence generator Gseq generates texts according to topical words . 3.1 GENERATING TOPICAL WORDS . The upper part of Figure 1 illustrates how our model generate topical words , where there are a bag-of-words generator Gbow , a bag-of-words discriminator Dbow , a topic model Q , and a noise predictor E. • Bag-of-words generator Gbow It takes a discrete one-hot topic code c and a continuous noise z as the input , and manages to generate bag-of-words that captures the input topical information and are indistinguishable from the bag-of-words of real texts . Here the bag-of-words is a binary vector generating from sigmoid function , where each dimension indicates whether a single word in the dictionary exists in the text . • Bag-of-words discriminator Dbow It takes the bag-of-words vector as its input and distinguishes whether it is generated or human-written . • Topic model Q It is a categorical topic code classifier that is implemented by a matrix , considering that a linear model is easier to interpret the generated bag-of-words . • Noise predictor E It focuses on predicting the noise that can reconstruct the input bag-of-words by Gbow . Similar to Chen et al . ( 2016 ) , we apply ( 1 ) to train our generator Gbow , discriminator Dbow , and the topic model Q . However , it is difficult to generate discrete bag-of-words due to the gradient vanishing problem , we apply WGAN ( Arjovsky et al. , 2017 ) loss to train Gbow and Dbow . In addition , during training , there is a severe mode collapse issue within the same topic . That is , given the same topic code , the generator ignores the continuous noise and output the same topical words . The reason is that outputting the same bag of words for the generator is the optimal solution to maximize the mutual information between the discrete input topic code and its output . To tackle this issue , we clip the probability from topic model Q in ( 1 ) to a specific range α , and rewrite the ( 1 ) to : min G , Q max D Ex∼Pdata [ Dbow ( x ) ] − Ez∼Pz , c∼Pc [ Dbow ( Gbow ( z , c ) ) ] − λEz∼Pz , c∼Pc [ min ( Q ( c | Gbow ( z , c ) ) , α ) ] . ( 2 ) , where x is the binary bag-of-words vector of real text . Here , to constrain the Dbow to be Lipschitz function , we apply gradient penalty ( Gulrajani et al. , 2017 ) to Dbow . In addition , we apply batchnormalization to alleviate the mode collapse problem ( Ioffe & Szegedy , 2015 ) . In order to obtain better results , an auto-encoder is included in the optimization procedure ( Larsen et al. , 2015 ; Huang et al. , 2018 ) . Here we use binary cross entropy loss as reconstruction loss function : min G , Q , E Ex∼Pdata [ −x ∗ log ( Gbow ( Q ( x ) , E ( x ) ) ) + ( 1− x ) ∗ log ( 1−Gbow ( Q ( x ) , E ( x ) ) ) ] ( 3 ) , where the topic classifier Q and the noise predicor E encode real text bag-of-words x into the discrete code and continuous noise respectively . We train 2 and 3 alternately .

Summary: This paper presents a topic model based on adversarial training. Specifically, the paper adopts the framework of InfoGAN to generates the bag-of-words of a document and the latent codes in InfoGAN correspond to the latent topics in topic modelling. In addition to the above framework, to make the model work better, several add-ons are also proposed, combining autoencoder, loss clipping, and a generative model to generate text sequences based on the bag-of-words.