Generative Adversarial Nets for Multiple Text Corpora

1 Introduction . Generative adversarial nets ( GAN ) ( Goodfellow et al. , 2014 ) belong to a class of generative models which are trainable and can generate artificial data examples similar to the existing ones . In a GAN model , there are two sub-models simultaneously trained : a generative model G from which artificial data examples can be sampled , and a discriminative model D which classifies real data examples and artificial ones from G. By training G to maximize its generation power , and training D to minimize the generation power of G , so that ideally there will be no difference between the true and artificial examples , a minimax problem can be established . The GAN model has been shown to closely replicate a number of image data sets , such as MNIST , Toronto Face Database ( TFD ) , CIFAR-10 , SVHN , and ImageNet ( Goodfellow et al. , 2014 ; Salimans et al. , 2016 ) . The GAN model has been extended to text data in a number of ways . For instance , Zhang et al . ( 2016 ) applied a long-short term memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) generator and approximated discretization to generate text data . Moreover , Li et al . ( 2017 ) applied the GAN model to generate dialogues , i.e . pairs of questions and answers . Meanwhile , the GAN model can also be applied to generate bag-of-words embeddings of text data , which focus more on key terms in a text document rather than the original document itself . Glover ( 2016 ) provided such a model with the energy-based GAN ( Zhao et al. , 2017 ) . To the best of our knowledge , there has been no literature on applying the GAN model to multiple corpora of text data . Multi-class GANs have been proposed ( Liu & Tuzel , 2016 ; Mirza & Osindero , 2014 ) , but a class in multi-class classification is not the same as multiple corpora . Because knowing the underlying corpus membership of each text document can provide better information on how the text documents are organized , and documents from the same corpus are expected to share similar topics or key words , considering the membership information can benefit the training of a text model from a supervised perspective . We consider two problems associated with training multi-corpus text data : ( 1 ) Given a separate set of word Mikolov et al . ( 2013b ) , how to obtain a better set of cross-corpus word embeddings from them ? ( 2 ) How to incorporate the generation of document embeddings from different corpora in a single GAN model ? For the first problem , we train a GAN model which discriminates documents represented by different word embeddings , and train the cross-corpus word embedding so that it is similar to each existing word embedding per corpus . For the second problem , we train a GAN model which considers both crosscorpus and per-corpus “ topics ” in the generator , and applies a discriminator which considers each original and artificial document corpus . We also show that with sufficient training , the distribution of the artificial document embeddings is equivalent to the original ones . Our work has the following contributions : ( 1 ) we extend GANs to multiple corpora of text data , ( 2 ) we provide applications of GANs to finetune word embeddings and to create robust document embeddings , and ( 3 ) we establish theoretical convergence results of the multi-class GAN model . Section 2 reviews existing GAN models related to this paper . Section 3 describes the GAN models on training cross-corpus word embeddings and generating document embeddings for each corpora , and explains the associated algorithms . Section 4 presents the results of the two models on text data sets , and transfers them to supervised learning . Section 5 summarizes the results and concludes the paper . 2 Literature Review . In a GAN model , we assume that the data examples x are drawn from a distribution px ( · ) , and the artificial data examples G ( z ) : = G ( z , θg ) are transformed from the noise distribution z ∼ pz ( · ) . The binary classifier D ( · ) outputs the probability of a data example ( or an artificial one ) being an original one . Because the probabilistic structure of a GAN can be unstable to train , the Wasserstein GAN ( Arjovsky et al. , 2017 ) is proposed which applies a 1-Lipschitz function as a discriminator . We note that in many circumstances , data sets are obtained with supervised labels or categories , which can add explanatory power to unsupervised models such as the GAN . For instance , the CoGAN ( Liu & Tuzel , 2016 ) considers pairs of data examples from different categories , and the weights of the first few layers ( i.e . close to z ) are tied . Mirza & Osindero ( 2014 ) proposed the conditional GAN where the generator G and the discriminator D depend on the class label y. Salimans et al . ( 2016 ) applied the class labels for semi-supervised learning with an additional artificial class . However , all these models consider only images and do not produce word or document embeddings , therefore being different from our models . For generating real text , Zhang et al . ( 2016 ) proposed textGAN in which the generator has an LSTM form , and a uni-dimensional convolutional neural network ( Collobert et al. , 2011 ; Kim , 2014 ) is applied as the discriminator . Also , a weighted softmax function is applied to make the argmax function differentiable . The focus of our work is to summarize information from longer documents , so we apply document embeddings such as the tf-idf to represent the documents rather than to generate real text . For generating bag-of-words embeddings of text , Glover ( 2016 ) proposed a GAN model with the mean squared error of a de-noising autoencoder as the discriminator , and the output x is the one-hot word embedding of a document . Our models are different from this model because we consider tf-idf document embeddings for multiple text corpora in the deGAN model ( Section 3.2 ) , and weGAN ( Section 3.1 ) can be applied to produce word embeddings . Also , we focus on robustness based on several corpora , while Glover ( 2016 ) assumed a single corpus . For extracting word embeddings given text data , Mikolov et al . ( 2013a ) proposed the word2vec model , for which there are two variations : the continuous bag-of-words ( cBoW ) model ( Mikolov et al. , 2013a ) , where the neighboring words are used to predict the appearance of each word ; the skip-gram model , where each neighboring word is used individually for prediction . In GloVe ( Pennington et al. , 2014 ) , a bilinear regression model is trained on the log of the word co-occurrence matrix . In these models , the weights associated with each word are used as the embedding . For obtaining document embeddings , the para2vec model ( Le & Mikolov , 2014 ) adds per-paragraph vectors to train word2vec-type models , so that the vectors can be used as embeddings for each paragraph . A simpler approach by taking the average of the embeddings of each word in a document and output the document embedding is exhibited in Socher et al . ( 2013 ) . 3 Models and Algorithms . Suppose we have a number of different corpora C1 , . . . , CM , which for example can be based on different categories or sentiments of text documents . We suppose that Cm = { dm1 , . . . , dmnm } , m = 1 , . . . , M , where each dmi represents a document . The words in all corpora are collected in a dictionary , and indexed from 1 to V . We name the GAN model to train cross-corpus word embeddings as “ weGAN , ” where “ we ” stands for “ word embeddings , ” and the GAN model to generate document embeddings for multiple corpora as “ deGAN , ” where “ de ” stands for “ document embeddings . ” 3.1 weGAN : Training cross-corpus word embeddings . We assume that for each corpora Cm , we are given word embeddings for each word vm1 , . . . , vmV ∈ Rd , where d is the dimension of each word embedding . We are also given a classification task on documents that is represented by a parametric model C taking document embeddings as feature vectors . We construct a GAN model which combines different sets of word embeddings Vm : = { vmi } Vi=1 , m = 1 , . . . , M , into a single set of word embeddings G : = { v0i } Vi=1 . Note that V1 , . . . , VM are given but G is trained . Here we consider G as the generator , and the goal of the discriminator is to distinguish documents represented by the original embeddings V1 , . . . , VM and the same documents represented by the new embeddings G. Next we describe how the documents are represented by a set of embeddings V1 , . . . , VM and G. For each document dmi , we define its document embedding with Vm as gmi : = f ( dmi , Vm ) , where f ( · ) can be any mapping . Similarly , we define the document embedding of dmi with G as follows , with G = { v0j } Vj=1 trainable fG ( dmi ) : = f ( dmi , G ) . In a typical example , word embeddings would be based on word2vec or GLoVe . Function f can be based on tf-idf , i.e . f ( dmi , V ) = ∑V j=1 t m ij v m j where vmj is the word embedding of the j-th word in the m-th corpus Cm and tmi = ( tmi1 , . . . , tmiV ) is the tf-idf representation of the i-th document dmi in the m-th corpus Cm . To train the GAN model , we consider the following minimax problem min C , G max D { M∑ m=1 nm∑ i=1 [ log ( D ( gmi ) ) + log ( 1− D ( fG ( dmi ) ) ) − log ( eTkmi C ( fG ( d m i ) ) ) ] } , ( 1 ) where D is a discriminator of whether a document is original or artificial . Here kmi is the label of document dmi with respect to classifier C , and ekmi is a unit vector with only the k m i -th component being one and all other components being zeros . Note that log ( eTkmi C ( fG ( d m i ) ) ) is equivalent to KL ( ekmi ‖C ( fG ( d m i ) ) ) , but we use the former notation due to its brevity . The intuition of problem ( 3 ) is explained as follows . First we consider a discriminator D which is a feedforward neural network ( FFNN ) with binary outcomes , and classifies the document embeddings { fG ( dmi ) } nm i=1 M m=1 against the original document embeddings { gmi } nm i=1 M m=1 . Discriminator D minimizes this classification error , i.e . it maximizes the log-likelihood of { fG ( dmi ) } nm i=1 M m=1 having label 0 and { gmi } nm i=1 M m=1 having label 1 . This corresponds to M∑ m=1 nm∑ i=1 [ log ( D ( gmi ) ) + log ( 1−D ( fG ( dmi ) ) ) ] . ( 2 ) For the generator G , we wish to minimize ( 3 ) against G so that we can apply the minimax strategy , and the combined word embeddings G would resemble each set of word embeddings V1 , . . . , VM . Meanwhile , we also consider classifier C with K outcomes , and associates dmi with label kmi , so that the generator G can learn from the document labeling in a semi-supervised way . If the classifier C outputs a K-dimensional softmax probability vector , we minimize the following against G , which corresponds to ( 3 ) given D and C : M∑ m=1 nm∑ i=1 [ log ( 1−D ( fG ( dmi ) ) ) − log ( eTkmi C ( fG ( d m i ) ) ) ] . ( 3 ) For the classifier C , we also minimize its negative log-likelihood − M∑ m=1 nm∑ i=1 log ( eTkmi C ( fG ( d m i ) ) ) . ( 4 ) Assembling ( 4-6 ) together , we retrieve the original minimax problem ( 3 ) . We train the discriminator and the classifier , { D , C } , and the combined embeddings G according to ( 4-6 ) iteratively for a fixed number of epochs with the stochastic gradient descent algorithm , until the discrimination and classification errors become stable . Figure 1 illustrates the weGAN model . The algorithm for weGAN is summarized in Algorithm 1 in the appendix . 𝒞1 𝒞2 ⋯ 𝒞𝑀 𝑑1 1 ⋯ 𝑑𝑛1 1 𝑑1 2 ⋯ 𝑑𝑛2 2 ⋯ 𝑑1 𝑀 ⋯ 𝑑𝑛𝑀 𝑀 𝑔1 1 , … , 𝑔𝑛1 1 corpora documents 𝒱1 word embeddings document embeddings 𝒟discriminator 𝐹𝑟𝑜𝑚 { 𝒱1 , … , 𝒱𝑀 } 𝑜𝑟 𝒢 𝒞classifier output 𝒱2 𝑔1 2 , … , 𝑔𝑛2 2 𝒱𝑀 𝑔1 𝑀 , … , 𝑔𝑛𝑀 𝑀 𝑡𝑛𝑀 𝑀 { 𝑑1 1 , … , 𝑑𝑛𝑀 𝑀 } 𝒢 { 𝑓𝒢 𝑑1 1 , … , 𝑓𝒢 ( 𝑑𝑛𝑀 𝑀 ) } Label distribution Figure 1 : Model structure of weGAN . 𝒞1 𝒞2 ⋯ 𝒞𝑀 𝑑1 1 ⋯ 𝑑𝑛1 1 𝑑1 2 ⋯ 𝑑𝑛2 2 ⋯ 𝑑1 𝑀 ⋯ 𝑑𝑛𝑀 𝑀 corpora document embeddings 𝒟discriminator 𝐹𝑟𝑜𝑚 𝒞1 , 𝒞2 , … , 𝒞𝑀 𝑜𝑟 { 𝒢1 , … , 𝒢𝑀 } output 𝒢1 ( 𝑛 ) generators 𝑊ℎ 0 , 𝑊𝑜 0 𝒢1 𝒢𝑀𝒢2 noise𝑛 𝒢2 ( 𝑛 ) 𝒢𝑀 ( 𝑛 ) generated embeddings 𝑊ℎ 1 , 𝑊𝑜 1 𝑊ℎ 2 , 𝑊𝑜 2 𝑊ℎ 𝑀 , 𝑊𝑜 𝑀 Figure 2 : Model structure of deGAN .

Summary: The paper proposes extensions of Generative Adversarial Networks to modeling multiple text corpora. Concretely, the paper looks at two problems: 1) given independently pretrained word embeddings from K corpora, finding a common word embdding, 2) extracting document representations from a discriminator of a GAN trained to generate tf-idf vectors. Preliminary experiments show that the proposed approaches outperform baseline classifiers trained with word2vec.