Generative Adversarial Nets for Multiple Text Corpora

1 Introduction . Generative adversarial nets ( GAN ) ( Goodfellow et al. , 2014 ) belong to a class of generative models which are trainable and can generate artificial data examples similar to the existing ones . In a GAN model , there are two sub-models simultaneously trained : a generative model G from which artificial data examples can be sampled , and a discriminative model D which classifies real data examples and artificial ones from G. By training G to maximize its generation power , and training D to minimize the generation power of G , so that ideally there will be no difference between the true and artificial examples , a minimax problem can be established . The GAN model has been shown to closely replicate a number of image data sets , such as MNIST , Toronto Face Database ( TFD ) , CIFAR-10 , SVHN , and ImageNet ( Goodfellow et al. , 2014 ; Salimans et al. , 2016 ) . The GAN model has been extended to text data in a number of ways . For instance , Zhang et al . ( 2016 ) applied a long-short term memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) generator and approximated discretization to generate text data . Moreover , Li et al . ( 2017 ) applied the GAN model to generate dialogues , i.e . pairs of questions and answers . Meanwhile , the GAN model can also be applied to generate bag-of-words embeddings of text data , which focus more on key terms in a text document rather than the original document itself . Glover ( 2016 ) provided such a model with the energy-based GAN ( Zhao et al. , 2017 ) . To the best of our knowledge , there has been no literature on applying the GAN model to multiple corpora of text data . Multi-class GANs have been proposed ( Liu & Tuzel , 2016 ; Mirza & Osindero , 2014 ) , but a class in multi-class classification is not the same as multiple corpora . Because knowing the underlying corpus membership of each text document can provide better information on how the text documents are organized , and documents from the same corpus are expected to share similar topics or key words , considering the membership information can benefit the training of a text model from a supervised perspective . We consider two problems associated with training multi-corpus text data : ( 1 ) Given a separate set of word Mikolov et al . ( 2013b ) , how to obtain a better set of cross-corpus word embeddings from them ? ( 2 ) How to incorporate the generation of document embeddings from different corpora in a single GAN model ? For the first problem , we train a GAN model which discriminates documents represented by different word embeddings , and train the cross-corpus word embedding so that it is similar to each existing word embedding per corpus . For the second problem , we train a GAN model which considers both crosscorpus and per-corpus “ topics ” in the generator , and applies a discriminator which considers each original and artificial document corpus . We also show that with sufficient training , the distribution of the artificial document embeddings is equivalent to the original ones . Our work has the following contributions : ( 1 ) we extend GANs to multiple corpora of text data , ( 2 ) we provide applications of GANs to finetune word embeddings and to create robust document embeddings , and ( 3 ) we establish theoretical convergence results of the multi-class GAN model . Section 2 reviews existing GAN models related to this paper . Section 3 describes the GAN models on training cross-corpus word embeddings and generating document embeddings for each corpora , and explains the associated algorithms . Section 4 presents the results of the two models on text data sets , and transfers them to supervised learning . Section 5 summarizes the results and concludes the paper . 2 Literature Review . In a GAN model , we assume that the data examples x are drawn from a distribution px ( · ) , and the artificial data examples G ( z ) : = G ( z , θg ) are transformed from the noise distribution z ∼ pz ( · ) . The binary classifier D ( · ) outputs the probability of a data example ( or an artificial one ) being an original one . Because the probabilistic structure of a GAN can be unstable to train , the Wasserstein GAN ( Arjovsky et al. , 2017 ) is proposed which applies a 1-Lipschitz function as a discriminator . We note that in many circumstances , data sets are obtained with supervised labels or categories , which can add explanatory power to unsupervised models such as the GAN . For instance , the CoGAN ( Liu & Tuzel , 2016 ) considers pairs of data examples from different categories , and the weights of the first few layers ( i.e . close to z ) are tied . Mirza & Osindero ( 2014 ) proposed the conditional GAN where the generator G and the discriminator D depend on the class label y. Salimans et al . ( 2016 ) applied the class labels for semi-supervised learning with an additional artificial class . However , all these models consider only images and do not produce word or document embeddings , therefore being different from our models . For generating real text , Zhang et al . ( 2016 ) proposed textGAN in which the generator has an LSTM form , and a uni-dimensional convolutional neural network ( Collobert et al. , 2011 ; Kim , 2014 ) is applied as the discriminator . Also , a weighted softmax function is applied to make the argmax function differentiable . The focus of our work is to summarize information from longer documents , so we apply document embeddings such as the tf-idf to represent the documents rather than to generate real text . For generating bag-of-words embeddings of text , Glover ( 2016 ) proposed a GAN model with the mean squared error of a de-noising autoencoder as the discriminator , and the output x is the one-hot word embedding of a document . Our models are different from this model because we consider tf-idf document embeddings for multiple text corpora in the deGAN model ( Section 3.2 ) , and weGAN ( Section 3.1 ) can be applied to produce word embeddings . Also , we focus on robustness based on several corpora , while Glover ( 2016 ) assumed a single corpus . For extracting word embeddings given text data , Mikolov et al . ( 2013a ) proposed the word2vec model , for which there are two variations : the continuous bag-of-words ( cBoW ) model ( Mikolov et al. , 2013a ) , where the neighboring words are used to predict the appearance of each word ; the skip-gram model , where each neighboring word is used individually for prediction . In GloVe ( Pennington et al. , 2014 ) , a bilinear regression model is trained on the log of the word co-occurrence matrix . In these models , the weights associated with each word are used as the embedding . For obtaining document embeddings , the para2vec model ( Le & Mikolov , 2014 ) adds per-paragraph vectors to train word2vec-type models , so that the vectors can be used as embeddings for each paragraph . A simpler approach by taking the average of the embeddings of each word in a document and output the document embedding is exhibited in Socher et al . ( 2013 ) . 3 Models and Algorithms . Suppose we have a number of different corpora C1 , . . . , CM , which for example can be based on different categories or sentiments of text documents . We suppose that Cm = { dm1 , . . . , dmnm } , m = 1 , . . . , M , where each dmi represents a document . The words in all corpora are collected in a dictionary , and indexed from 1 to V . We name the GAN model to train cross-corpus word embeddings as “ weGAN , ” where “ we ” stands for “ word embeddings , ” and the GAN model to generate document embeddings for multiple corpora as “ deGAN , ” where “ de ” stands for “ document embeddings . ” 3.1 weGAN : Training cross-corpus word embeddings . We assume that for each corpora Cm , we are given word embeddings for each word vm1 , . . . , vmV ∈ Rd , where d is the dimension of each word embedding . We are also given a classification task on documents that is represented by a parametric model C taking document embeddings as feature vectors . We construct a GAN model which combines different sets of word embeddings Vm : = { vmi } Vi=1 , m = 1 , . . . , M , into a single set of word embeddings G : = { v0i } Vi=1 . Note that V1 , . . . , VM are given but G is trained . Here we consider G as the generator , and the goal of the discriminator is to distinguish documents represented by the original embeddings V1 , . . . , VM and the same documents represented by the new embeddings G. Next we describe how the documents are represented by a set of embeddings V1 , . . . , VM and G. For each document dmi , we define its document embedding with Vm as gmi : = f ( dmi , Vm ) , where f ( · ) can be any mapping . Similarly , we define the document embedding of dmi with G as follows , with G = { v0j } Vj=1 trainable fG ( dmi ) : = f ( dmi , G ) . In a typical example , word embeddings would be based on word2vec or GLoVe . Function f can be based on tf-idf , i.e . f ( dmi , V ) = ∑V j=1 t m ij v m j where vmj is the word embedding of the j-th word in the m-th corpus Cm and tmi = ( tmi1 , . . . , tmiV ) is the tf-idf representation of the i-th document dmi in the m-th corpus Cm . To train the GAN model , we consider the following minimax problem min C , G max D { M∑ m=1 nm∑ i=1 [ log ( D ( gmi ) ) + log ( 1− D ( fG ( dmi ) ) ) − log ( eTkmi C ( fG ( d m i ) ) ) ] } , ( 1 ) where D is a discriminator of whether a document is original or artificial . Here kmi is the label of document dmi with respect to classifier C , and ekmi is a unit vector with only the k m i -th component being one and all other components being zeros . Note that log ( eTkmi C ( fG ( d m i ) ) ) is equivalent to KL ( ekmi ‖C ( fG ( d m i ) ) ) , but we use the former notation due to its brevity . The intuition of problem ( 3 ) is explained as follows . First we consider a discriminator D which is a feedforward neural network ( FFNN ) with binary outcomes , and classifies the document embeddings { fG ( dmi ) } nm i=1 M m=1 against the original document embeddings { gmi } nm i=1 M m=1 . Discriminator D minimizes this classification error , i.e . it maximizes the log-likelihood of { fG ( dmi ) } nm i=1 M m=1 having label 0 and { gmi } nm i=1 M m=1 having label 1 . This corresponds to M∑ m=1 nm∑ i=1 [ log ( D ( gmi ) ) + log ( 1−D ( fG ( dmi ) ) ) ] . ( 2 ) For the generator G , we wish to minimize ( 3 ) against G so that we can apply the minimax strategy , and the combined word embeddings G would resemble each set of word embeddings V1 , . . . , VM . Meanwhile , we also consider classifier C with K outcomes , and associates dmi with label kmi , so that the generator G can learn from the document labeling in a semi-supervised way . If the classifier C outputs a K-dimensional softmax probability vector , we minimize the following against G , which corresponds to ( 3 ) given D and C : M∑ m=1 nm∑ i=1 [ log ( 1−D ( fG ( dmi ) ) ) − log ( eTkmi C ( fG ( d m i ) ) ) ] . ( 3 ) For the classifier C , we also minimize its negative log-likelihood − M∑ m=1 nm∑ i=1 log ( eTkmi C ( fG ( d m i ) ) ) . ( 4 ) Assembling ( 4-6 ) together , we retrieve the original minimax problem ( 3 ) . We train the discriminator and the classifier , { D , C } , and the combined embeddings G according to ( 4-6 ) iteratively for a fixed number of epochs with the stochastic gradient descent algorithm , until the discrimination and classification errors become stable . Figure 1 illustrates the weGAN model . The algorithm for weGAN is summarized in Algorithm 1 in the appendix . 𝒞1 𝒞2 ⋯ 𝒞𝑀 𝑑1 1 ⋯ 𝑑𝑛1 1 𝑑1 2 ⋯ 𝑑𝑛2 2 ⋯ 𝑑1 𝑀 ⋯ 𝑑𝑛𝑀 𝑀 𝑔1 1 , … , 𝑔𝑛1 1 corpora documents 𝒱1 word embeddings document embeddings 𝒟discriminator 𝐹𝑟𝑜𝑚 { 𝒱1 , … , 𝒱𝑀 } 𝑜𝑟 𝒢 𝒞classifier output 𝒱2 𝑔1 2 , … , 𝑔𝑛2 2 𝒱𝑀 𝑔1 𝑀 , … , 𝑔𝑛𝑀 𝑀 𝑡𝑛𝑀 𝑀 { 𝑑1 1 , … , 𝑑𝑛𝑀 𝑀 } 𝒢 { 𝑓𝒢 𝑑1 1 , … , 𝑓𝒢 ( 𝑑𝑛𝑀 𝑀 ) } Label distribution Figure 1 : Model structure of weGAN . 𝒞1 𝒞2 ⋯ 𝒞𝑀 𝑑1 1 ⋯ 𝑑𝑛1 1 𝑑1 2 ⋯ 𝑑𝑛2 2 ⋯ 𝑑1 𝑀 ⋯ 𝑑𝑛𝑀 𝑀 corpora document embeddings 𝒟discriminator 𝐹𝑟𝑜𝑚 𝒞1 , 𝒞2 , … , 𝒞𝑀 𝑜𝑟 { 𝒢1 , … , 𝒢𝑀 } output 𝒢1 ( 𝑛 ) generators 𝑊ℎ 0 , 𝑊𝑜 0 𝒢1 𝒢𝑀𝒢2 noise𝑛 𝒢2 ( 𝑛 ) 𝒢𝑀 ( 𝑛 ) generated embeddings 𝑊ℎ 1 , 𝑊𝑜 1 𝑊ℎ 2 , 𝑊𝑜 2 𝑊ℎ 𝑀 , 𝑊𝑜 𝑀 Figure 2 : Model structure of deGAN .

Summary: The paper proposes to use Generative Adversarial Networks (GANs) in the context of natural language processing and introduces two models for generating document embeddings. The first model, weGAN, aggregates multiple sets of single-corpus word2vec embeddings into one set of cross-corpus word representations; document embeddings are a function of these updated word embeddings. The second model, deGAN, side-steps word-level embeddings and directly generates document-level representations. For both models, the real examples come from word2vec and tf-idf, while the artificial examples are the ones generated by the network. The authors show that their document embeddings are better than the word2vec/tf-idf baseline at clustering documents according to the corpus they originate from.