Introduction
Generative adversarial nets (GAN) (Goodfellow et al., 2014) belong to a class of generative models which are trainable and can generate artificial data examples similar to the existing ones. In a GAN model, there are two sub-models simultaneously trained: a generative model INLINEFORM0 from which artificial data examples can be sampled, and a discriminative model INLINEFORM1 which classifies real data examples and artificial ones from INLINEFORM2 . By training INLINEFORM3 to maximize its generation power, and training INLINEFORM4 to minimize the generation power of INLINEFORM5 , so that ideally there will be no difference between the true and artificial examples, a minimax problem can be established. The GAN model has been shown to closely replicate a number of image data sets, such as MNIST, Toronto Face Database (TFD), CIFAR-10, SVHN, and ImageNet (Goodfellow et al., 2014; Salimans et al. 2016).
The GAN model has been extended to text data in a number of ways. For instance, Zhang et al. (2016) applied a long-short term memory (Hochreiter and Schmidhuber, 1997) generator and approximated discretization to generate text data. Moreover, Li et al. (2017) applied the GAN model to generate dialogues, i.e. pairs of questions and answers. Meanwhile, the GAN model can also be applied to generate bag-of-words embeddings of text data, which focus more on key terms in a text document rather than the original document itself. Glover (2016) provided such a model with the energy-based GAN (Zhao et al., 2017).
To the best of our knowledge, there has been no literature on applying the GAN model to multiple corpora of text data. Multi-class GANs (Liu and Tuzel, 2016; Mirza and Osindero, 2014) have been proposed, but a class in multi-class classification is not the same as multiple corpora. Because knowing the underlying corpus membership of each text document can provide better information on how the text documents are organized, and documents from the same corpus are expected to share similar topics or key words, considering the membership information can benefit the training of a text model from a supervised perspective. We consider two problems associated with training multi-corpus text data: (1) Given a separate set of word embeddings from each corpus, such as the word2vec embeddings (Mikolov et al., 2013), how to obtain a better set of cross-corpus word embeddings from them? (2) How to incorporate the generation of document embeddings from different corpora in a single GAN model?
For the first problem, we train a GAN model which discriminates documents represented by different word embeddings, and train the cross-corpus word embedding so that it is similar to each existing word embedding per corpus. For the second problem, we train a GAN model which considers both cross-corpus and per-corpus “topics” in the generator, and applies a discriminator which considers each original and artificial document corpus. We also show that with sufficient training, the distribution of the artificial document embeddings is equivalent to the original ones. Our work has the following contributions: (1) we extend GANs to multiple corpora of text data, (2) we provide applications of GANs to finetune word embeddings and to create robust document embeddings, and (3) we establish theoretical convergence results of the multi-class GAN model.
Section 2 reviews existing GAN models related to this paper. Section 3 describes the GAN models on training cross-corpus word embeddings and generating document embeddings for each corpora, and explains the associated algorithms. Section 4 presents the results of the two models on text data sets, and transfers them to supervised learning. Section 5 summarizes the results and concludes the paper.
Literature Review
In a GAN model, we assume that the data examples INLINEFORM0 are drawn from a distribution INLINEFORM1 , and the artificial data examples INLINEFORM2 are transformed from the noise distribution INLINEFORM3 . The binary classifier INLINEFORM4 outputs the probability of a data example (or an artificial one) being an original one. We consider the following minimax problem DISPLAYFORM0
With sufficient training, it is shown in Goodfellow et al. (2014) that the distribution of artificial data examples INLINEFORM0 is eventually equivalent to the data distribution INLINEFORM1 , i.e. INLINEFORM2 .
Because the probabilistic structure of a GAN can be unstable to train, the Wasserstein GAN (Arjovsky et al., 2017) is proposed which applies a 1-Lipschitz function as a discriminator. In a Wasserstein GAN, we consider the following minimax problem DISPLAYFORM0
These GANs are for the general purpose of learning the data distribution in an unsupervised way and creating perturbed data examples resembling the original ones. We note that in many circumstances, data sets are obtained with supervised labels or categories, which can add explanatory power to unsupervised models such as the GAN. We summarize such GANs because a corpus can be potentially treated as a class. The main difference is that classes are purely for the task of classification while we are interested in embeddings that can be used for any supervised or unsupervised task.
For instance, the CoGAN (Liu and Tuzel, 2016) considers pairs of data examples from different categories as follows INLINEFORM0
where the weights of the first few layers of INLINEFORM0 and INLINEFORM1 (i.e. close to INLINEFORM2 ) are tied. Mirza and Osindero (2014) proposed the conditional GAN where the generator INLINEFORM3 and the discriminator INLINEFORM4 depend on the class label INLINEFORM5 . While these GANs generate samples resembling different classes, other variations of GANs apply the class labels for semi-supervised learning. For instance, Salimans et al. (2016) proposed the following objective DISPLAYFORM0
where INLINEFORM0 has INLINEFORM1 classes plus the INLINEFORM2 -th artificial class. Similar models can be found in Odena (2016), the CatGAN in Springenberg (2016), and the LSGAN in Mao et al. (2017). However, all these models consider only images and do not produce word or document embeddings, therefore being different from our models.
For generating real text, Zhang et al. (2016) proposed textGAN in which the generator has the following form, DISPLAYFORM0
where INLINEFORM0 is the noise vector, INLINEFORM1 is the generated sentence, INLINEFORM2 are the words, and INLINEFORM3 . A uni-dimensional convolutional neural network (Collobert et al, 2011; Kim, 2014) is applied as the discriminator. Also, a weighted softmax function is applied to make the argmax function differentiable. With textGAN, sentences such as “we show the efficacy of our new solvers, making it up to identify the optimal random vector...” can be generated. Similar models can also be found in Wang et al. (2016), Press et al. (2017), and Rajeswar et al. (2017). The focus of our work is to summarize information from longer documents, so we apply document embeddings such as the tf-idf to represent the documents rather than to generate real text.
For generating bag-of-words embeddings of text, Glover (2016) proposed the following model DISPLAYFORM0
and INLINEFORM0 is the mean squared error of a de-noising autoencoder, and INLINEFORM1 is the one-hot word embedding of a document. Our models are different from this model because we consider tf-idf document embeddings for multiple text corpora in the deGAN model (Section 3.2), and weGAN (Section 3.1) can be applied to produce word embeddings. Also, we focus on robustness based on several corpora, while Glover (2016) assumed a single corpus.
For extracting word embeddings given text data, Mikolov et al. (2013) proposed the word2vec model, for which there are two variations: the continuous bag-of-words (cBoW) model (Mikolov et al., 2013b), where the neighboring words are used to predict the appearance of each word; the skip-gram model, where each neighboring word is used individually for prediction. In GloVe (Pennington et al., 2013), a bilinear regression model is trained on the log of the word co-occurrence matrix. In these models, the weights associated with each word are used as the embedding. For obtaining document embeddings, the para2vec model (Le and Mikolov, 2014) adds per-paragraph vectors to train word2vec-type models, so that the vectors can be used as embeddings for each paragraph. A simpler approach by taking the average of the embeddings of each word in a document and output the document embedding is exhibited in Socher et al. (2013).
Models and Algorithms
Suppose we have a number of different corpora INLINEFORM0 , which for example can be based on different categories or sentiments of text documents. We suppose that INLINEFORM1 , INLINEFORM2 , where each INLINEFORM3 represents a document. The words in all corpora are collected in a dictionary, and indexed from 1 to INLINEFORM4 . We name the GAN model to train cross-corpus word embeddings as “weGAN,” where “we” stands for “word embeddings,” and the GAN model to generate document embeddings for multiple corpora as “deGAN,” where “de” stands for “document embeddings.”
weGAN: Training cross-corpus word embeddings
We assume that for each corpora INLINEFORM0 , we are given word embeddings for each word INLINEFORM1 , where INLINEFORM2 is the dimension of each word embedding. We are also given a classification task on documents that is represented by a parametric model INLINEFORM3 taking document embeddings as feature vectors. We construct a GAN model which combines different sets of word embeddings INLINEFORM4 , INLINEFORM5 , into a single set of word embeddings INLINEFORM6 . Note that INLINEFORM7 are given but INLINEFORM8 is trained. Here we consider INLINEFORM9 as the generator, and the goal of the discriminator is to distinguish documents represented by the original embeddings INLINEFORM10 and the same documents represented by the new embeddings INLINEFORM11 .
Next we describe how the documents are represented by a set of embeddings INLINEFORM0 and INLINEFORM1 . For each document INLINEFORM2 , we define its document embedding with INLINEFORM3 as follows, DISPLAYFORM0
where INLINEFORM0 can be any mapping. Similarly, we define the document embedding of INLINEFORM1 with INLINEFORM2 as follows, with INLINEFORM3 trainable DISPLAYFORM0
In a typical example, word embeddings would be based on word2vec or GLoVe. Function INLINEFORM0 can be based on tf-idf, i.e. INLINEFORM1 where INLINEFORM2 is the word embedding of the INLINEFORM3 -th word in the INLINEFORM4 -th corpus INLINEFORM5 and INLINEFORM6 is the tf-idf representation of the INLINEFORM7 -th document INLINEFORM8 in the INLINEFORM9 -th corpus INLINEFORM10 .
To train the GAN model, we consider the following minimax problem DISPLAYFORM0
where INLINEFORM0 is a discriminator of whether a document is original or artificial. Here INLINEFORM1 is the label of document INLINEFORM2 with respect to classifier INLINEFORM3 , and INLINEFORM4 is a unit vector with only the INLINEFORM5 -th component being one and all other components being zeros. Note that INLINEFORM6 is equivalent to INLINEFORM7 , but we use the former notation due to its brevity.
The intuition of problem (8) is explained as follows. First we consider a discriminator INLINEFORM0 which is a feedforward neural network (FFNN) with binary outcomes, and classifies the document embeddings INLINEFORM1 against the original document embeddings INLINEFORM2 . Discriminator INLINEFORM3 minimizes this classification error, i.e. it maximizes the log-likelihood of INLINEFORM4 having label 0 and INLINEFORM5 having label 1. This corresponds to DISPLAYFORM0
For the generator INLINEFORM0 , we wish to minimize (8) against INLINEFORM1 so that we can apply the minimax strategy, and the combined word embeddings INLINEFORM2 would resemble each set of word embeddings INLINEFORM3 . Meanwhile, we also consider classifier INLINEFORM4 with INLINEFORM5 outcomes, and associates INLINEFORM6 with label INLINEFORM7 , so that the generator INLINEFORM8 can learn from the document labeling in a semi-supervised way.
If the classifier INLINEFORM0 outputs a INLINEFORM1 -dimensional softmax probability vector, we minimize the following against INLINEFORM2 , which corresponds to (8) given INLINEFORM3 and INLINEFORM4 : DISPLAYFORM0
For the classifier INLINEFORM0 , we also minimize its negative log-likelihood DISPLAYFORM0
Assembling (9-11) together, we retrieve the original minimax problem (8).
We train the discriminator and the classifier, INLINEFORM0 , and the combined embeddings INLINEFORM1 according to (9-11) iteratively for a fixed number of epochs with the stochastic gradient descent algorithm, until the discrimination and classification errors become stable.
The algorithm for weGAN is summarized in Algorithm 1, and Figure 1 illustrates the weGAN model.
Algorithm 1. Train INLINEFORM0 based on INLINEFORM1 from all corpora INLINEFORM2 . Randomly initialize the weights and biases of the classifier INLINEFORM3 and discriminator INLINEFORM4 . Until maximum number of iterations reached Update INLINEFORM5 and INLINEFORM6 according to (9) and (11) given a mini-batch INLINEFORM7 of training examples INLINEFORM8 . Update INLINEFORM9 according to (10) given a mini-batch INLINEFORM10 of training examples INLINEFORM11 . Output INLINEFORM12 as the cross-corpus word embeddings.
deGAN: Generating document embeddings for multi-corpus text data
In this section, our goal is to generate document embeddings which would resemble real document embeddings in each corpus INLINEFORM0 , INLINEFORM1 . We construct INLINEFORM2 generators, INLINEFORM3 so that INLINEFORM4 generate artificial examples in corpus INLINEFORM5 . As in Section 3.1, there is a certain document embedding such as tf-idf, bag-of-words, or para2vec. Let INLINEFORM6 . We initialize a noise vector INLINEFORM7 , where INLINEFORM8 , and INLINEFORM9 is any noise distribution.
For a generator INLINEFORM0 represented by its parameters, we first map the noise vector INLINEFORM1 to the hidden layer, which represents different topics. We consider two hidden vectors, INLINEFORM2 for general topics and INLINEFORM3 for specific topics per corpus, DISPLAYFORM0
Here INLINEFORM0 represents a nonlinear activation function. In this model, the bias term can be ignored in order to prevent the “mode collapse” problem of the generator. Having the hidden vectors, we then map them to the generated document embedding with another activation function INLINEFORM1 , DISPLAYFORM0
To summarize, we may represent the process from noise to the document embedding as follows, DISPLAYFORM0
Given the generated document embeddings INLINEFORM0 , we consider the following minimax problem to train the generator INLINEFORM1 and the discriminator INLINEFORM2 : INLINEFORM3 INLINEFORM4
Here we assume that any document embedding INLINEFORM0 in corpus INLINEFORM1 is a sample with respect to the probability density INLINEFORM2 . Note that when INLINEFORM3 , the discriminator part of our model is equivalent to the original GAN model.
To explain (15), first we consider the discriminator INLINEFORM0 . Because there are multiple corpora of text documents, here we consider INLINEFORM1 categories as output of INLINEFORM2 , from which categories INLINEFORM3 represent the original corpora INLINEFORM4 , and categories INLINEFORM5 represent the generated document embeddings (e.g. bag-of-words) from INLINEFORM6 . Assume the discriminator INLINEFORM7 , a feedforward neural network, outputs the distribution of a text document being in each category. We maximize the log-likelihood of each document being in the correct category against INLINEFORM8 DISPLAYFORM0
Such a classifier does not only classifies text documents into different categories, but also considers INLINEFORM0 “fake” categories from the generators. When training the generators INLINEFORM1 , we minimize the following which makes a comparison between the INLINEFORM2 -th and INLINEFORM3 -th categories DISPLAYFORM0
The intuition of (17) is that for each generated document embedding INLINEFORM0 , we need to decrease INLINEFORM1 , which is the probability of the generated embedding being correctly classified, and increase INLINEFORM2 , which is the probability of the generated embedding being classified into the target corpus INLINEFORM3 . The ratio in (17) reflects these two properties.
We iteratively train (16) and (17) until the classification error of INLINEFORM0 becomes stable. The algorithm for deGAN is summarized in Algorithm 2, and Figure 2 illustrates the deGAN model..
Algorithm 2. Randomly initialize the weights of INLINEFORM0 . Initialize the discriminator INLINEFORM1 with the weights of the first layer (which takes document embeddings as the input) initialized by word embeddings, and other parameters randomly initialized. Until maximum number of iterations reached Update INLINEFORM2 according to (16) given a mini-batch of training examples INLINEFORM3 and samples from noise INLINEFORM4 . Update INLINEFORM5 according to (17) given a mini-batch of training examples INLINEFORM6 and samples form noise INLINEFORM7 . Output INLINEFORM8 as generators of document embeddings and INLINEFORM9 as a corpus classifier.
We next show that from (15), the distributions of the document embeddings from the optimal INLINEFORM0 are equal to the data distributions of INLINEFORM1 , which is a generalization of Goodfellow et al. (2014) to the multi-corpus scenario.
Proposition 1. Let us assume that the random variables INLINEFORM0 are continuous with probability density INLINEFORM1 which have bounded support INLINEFORM2 ; INLINEFORM3 is a continuous random variable with bounded support and activations INLINEFORM4 and INLINEFORM5 are continuous; and that INLINEFORM6 are solutions to (15). Then INLINEFORM7 , the probability density of the document embeddings from INLINEFORM8 , INLINEFORM9 , are equal to INLINEFORM10 .
Proof. Since INLINEFORM0 is bounded, all of the integrals exhibited next are well-defined and finite. Since INLINEFORM1 , INLINEFORM2 , and INLINEFORM3 are continuous, it follows that for any parameters, INLINEFORM4 is a continuous random variable with probability density INLINEFORM5 with finite support.
From the first line of (15), INLINEFORM0
This problem reduces to INLINEFORM0 subject to INLINEFORM1 , the solution of which is INLINEFORM2 , INLINEFORM3 . Therefore, the solution to (18) is DISPLAYFORM0
We then obtain from the second line of (15) that INLINEFORM0
From non-negativity of the Kullback-Leibler divergence, we conclude that INLINEFORM0
Experiments
In the experiments, we consider four data sets, two of them newly created and the remaining two already public: CNN, TIME, 20 Newsgroups, and Reuters-21578. The code and the two new data sets are available at github.com/baiyangwang/emgan. For the pre-processing of all the documents, we transformed all characters to lower case, stemmed the documents, and ran the word2vec model on each corpora to obtain word embeddings with a size of 300. In all subsequent models, we only consider the most frequent INLINEFORM0 words across all corpora in a data set.
The document embedding in weGAN is the tf-idf weighted word embedding transformed by the INLINEFORM0 activation, i.e. DISPLAYFORM0
For deGAN, we use INLINEFORM0 -normalized tf-idf as the document embedding because it is easier to interpret than the transformed embedding in (20).
For weGAN, the cross-corpus word embeddings are initialized with the word2vec model trained from all documents. For training our models, we apply a learning rate which increases linearly from INLINEFORM0 to INLINEFORM1 and train the models for 100 epochs with a batch size of 50 per corpus. The classifier INLINEFORM2 has a single hidden layer with 50 hidden nodes, and the discriminator with a single hidden layer INLINEFORM3 has 10 hidden nodes. All these parameters have been optimized. For the labels INLINEFORM4 in (8), we apply corpus membership of each document.
For the noise distribution INLINEFORM0 for deGAN, we apply the uniform distribution INLINEFORM1 . In (14) for deGAN, INLINEFORM2 and INLINEFORM3 so that the model outputs document embedding vectors which are comparable to INLINEFORM4 -normalized tf-idf vectors for each document. For the discriminator INLINEFORM5 of deGAN, we apply the word2vec embeddings based on all corpora to initialize its first layer, followed by another hidden layer of 50 nodes. For the discriminator INLINEFORM6 , we apply a learning rate of INLINEFORM7 , and for the generator INLINEFORM8 , we apply a learning rate of INLINEFORM9 , because the initial training phase of deGAN can be unstable. We also apply a batch size of 50 per corpus. For the softmax layers of deGAN, we initialize them with the log of the topic-word matrix in latent Dirichlet allocation (LDA) (Blei et al., 2003) in order to provide intuitive estimates.
For weGAN, we consider two metrics for comparing the embeddings trained from weGAN and those trained from all documents: (1) applying the document embeddings to cluster the documents into INLINEFORM0 clusters with the K-means algorithm, and calculating the Rand index (RI) (Rand, 1971) against the original corpus membership; (2) finetuning the classifier INLINEFORM1 and comparing the classification error against an FFNN of the same structure initialized with word2vec (w2v). For deGAN, we compare the performance of finetuning the discriminator of deGAN for document classification, and the performance of the same FFNN. Each supervised model is trained for 500 epochs and the validation data set is used to choose the best epoch.
The CNN data set
In the CNN data set, we collected all news links on www.cnn.com in the GDELT 1.0 Event Database from April 1st, 2013 to July 7, 2017. We then collected the news articles from the links, and kept those belonging to the three largest categories: “politics,” “world,” and “US.” We then divided these documents into INLINEFORM0 training documents, from which INLINEFORM1 validation documents are held out, and INLINEFORM2 testing documents.
We hypothesize that because weGAN takes into account document labels in a semi-supervised way, the embeddings trained from weGAN can better incorporate the labeling information and therefore, produce document embeddings which are better separated. The results are shown in Table 1 and averaged over 5 randomized runs. Performing the Welch's t-test, both changes after weGAN training are statistically significant at a INLINEFORM0 significance level. Because the Rand index captures matching accuracy, we observe from the Table 1 that weGAN tends to improve both metrics.
Meanwhile, we also wish to observe the spatial structure of the trained embeddings, which can be explored by the synonyms of each word measured by the cosine similarity. On average, the top 10 synonyms of each word differ by INLINEFORM0 word after weGAN training, and INLINEFORM1 of all words have different top 10 synonyms after training. Therefore, weGAN tends to provide small adjustments rather than structural changes. Table 2 lists the 10 most similar terms of three terms, “Obama,” “Trump,” and “U.S.,” before and after weGAN training, ordered by cosine similarity.
We observe from Table 2 that for “Obama,” ”Trump” and “Tillerson” are more similar after weGAN training, which means that the structure of the weGAN embeddings can be more up-to-date. For “Trump,” we observe that “Clinton” is not among the synonyms before, but is after, which shows that the synonyms after are more relevant. For “U.S.,” we observe that after training, “American” replaces “British” in the list of synonyms, which is also more relevant.
We next discuss deGAN. In Table 3, we compare the performance of finetuning the discriminator of deGAN for document classification, and the performance of the FFNN initialized with word2vec. The change is also statistically significant at the INLINEFORM0 level. From Table 3, we observe that deGAN improves the accuracy of supervised learning.
To compare the generated samples from deGAN with the original bag-of-words, we randomly select one record in each original and artificial corpus. The records are represented by the most frequent words sorted by frequency in descending order where the stop words are removed. The bag-of-words embeddings are shown in Table 4.
From Table 4, we observe that the bag-of-words embeddings of the original documents tend to contain more name entities, while those of the artificial deGAN documents tend to be more general. There are many additional examples not shown here with observed artificial bag-of-words embeddings having many name entities such as “Turkey,” “ISIS,” etc. from generated documents, e.g. “Syria eventually ISIS U.S. details jet aircraft October video extremist...”
We also perform dimensional reduction using t-SNE (van der Maaten and Hinton, 2008), and plot 100 random samples from each original or artificial category. The original samples are shown in red and the generated ones are shown in blue in Figure 3. We do not further distinguish the categories because there is no clear distinction between the three original corpora, “politics,” “world,” and “US.” The results are shown in Figure 3.
We observe that the original and artificial examples are generally mixed together and not well separable, which means that the artificial examples are similar to the original ones. However, we also observe that the artificial samples tend to be more centered and have no outliers (represented by the outermost red oval).
The TIME data set
In the TIME data set, we collected all news links on time.com in the GDELT 1.0 Event Database from April 1st, 2013 to July 7, 2017. We then collected the news articles from the links, and kept those belonging to the five largest categories: “Entertainment,” “Ideas,” “Politics,” “US,” and “World.” We divided these documents into INLINEFORM0 training documents, from which INLINEFORM1 validation documents are held out, and INLINEFORM2 testing documents.
Table 5 compares the clustering results of word2vec and weGAN, and the classification accuracy of an FFNN initialized with word2vec, finetuned weGAN, and finetuned deGAN. The results in Table 5 are the counterparts of Table 1 and Table 3 for the TIME data set. The differences are also significant at the INLINEFORM0 level.
From Table 5, we observe that both GAN models yield improved performance of supervised learning. For weGAN, on an average, the top 10 synonyms of each word differ by INLINEFORM0 word after weGAN training, and INLINEFORM1 of all words have different top 10 synonyms after training. We also compare the synonyms of the same common words, “Obama,” “Trump,” and “U.S.,” which are listed in Table 6.
In the TIME data set, for “Obama,” “Reagan” is ranked slightly higher as an American president. For “Trump,” “Bush” and “Sanders” are ranked higher as American presidents or candidates. For “U.S.,” we note that “Pentagon” is ranked higher after weGAN training, which we think is also reasonable because the term is closely related to the U.S. government.
For deGAN, we also compare the original and artificial samples in terms of the highest probability words. Table 7 shows one record for each category.
From Table 7, we observe that the produced bag-of-words are generally alike, and the words in the same sample are related to each other to some extent.
We also perform dimensional reduction using t-SNE for 100 examples per corpus and plot them in Figure 4. We observe that the points are generated mixed but deGAN cannot reproduce the outliers.
The 20 Newsgroups data set
The 20 Newsgroups data set is a collection of news documents with 20 categories. To reduce the number of categories so that the GAN models are more compact and have more samples per corpus, we grouped the documents into 6 super-categories: “religion,” “computer,” “cars,” “sport,” “science,” and “politics” (“misc” is ignored because of its noisiness). We considered each super-category as a different corpora. We then divided these documents into INLINEFORM0 training documents, from which INLINEFORM1 validation documents are held out, and INLINEFORM2 testing documents. We train weGAN and deGAN in the the beginning of Section 4, except that we use a learning rate of INLINEFORM3 for the discriminator in deGAN to stabilize the cost function. Table 8 compares the clustering results of word2vec and weGAN, and the classification accuracy of the FFNN initialized with word2vec, finetuned weGAN, and finetuned deGAN. All comparisons are statistically significant at the INLINEFORM4 level. The other results are similar to the previous two data sets and are thereby omitted here.
The Reuters-21578 data set
The Reuters-21578 data set is a collection of newswire articles. Because the data set is highly skewed, we considered the eight categories with more than 100 training documents: “earn,” “acq,” “crude,” “trade,” “money-fx,” “interest,” “money-supply,” and “ship.” We then divided these documents into INLINEFORM0 training documents, from which 692 validation documents are held out, and INLINEFORM1 testing documents. We train weGAN and deGAN in the same way as in the 20 Newsgroups data set. Table 9 compares the clustering results of word2vec and weGAN, and the classification accuracy of the FFNN initialized with word2vec, finetuned weGAN, and finetuned deGAN. All comparisons are statistically significant at the INLINEFORM2 level except the Rand index. The other results are similar to the CNN and TIME data sets and are thereby omitted here.
Conclusion
In this paper, we have demonstrated the application of the GAN model on text data with multiple corpora. We have shown that the GAN model is not only able to generate images, but also able to refine word embeddings and generate document embeddings. Such models can better learn the inner structure of multi-corpus text data, and also benefit supervised learning. The improvements in supervised learning are not large but statistically significant. The weGAN model outperforms deGAN in terms of supervised learning for 3 out of 4 data sets, and is thereby recommended. The synonyms from weGAN also tend to be more relevant than the original word2vec model. The t-SNE plots show that our generated document embeddings are similarly distributed as the original ones.
Reference
M. Arjovsky, S. Chintala, and L. Bottou. (2017). Wasserstein GAN. arXiv:1701.07875.
D. Blei, A. Ng, and M. Jordan. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research. 3:993-1022.
R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. (2011). Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research. 12:2493-2537.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. (2014). Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27 (NIPS 2014).
J. Glover. (2016). Modeling documents with Generative Adversarial Networks. In Workshop on Adversarial Training (NIPS 2016).
S. Hochreiter and J. Schmidhuber. (1997). Long Short-term Memory. In Neural Computation, 9:1735-1780.
Y. Kim. Convolutional Neural Networks for Sentence Classification. (2014). In The 2014 Conference on Empirical Methods on Natural Language Processing (EMNLP 2014).
Q. Le and T. Mikolov. (2014). Distributed Representations of Sentences and Documents. In Proceedings of the 31st International Conference on Machine Learning (ICML 2014).
J. Li, W. Monroe, T. Shi, A. Ritter, and D. Jurafsky. (2017). Adversarial Learning for Neural Dialogue Generation. arXiv:1701.06547.
M.-Y. Liu, and O. Tuzel. (2016). Coupled Generative Adversarial Networks. In Advances in Neural Information Processing Systems 29 (NIPS 2016).
X. Mao, Q. Li, H. Xie, R. Lau, Z. Wang, and S. Smolley. (2017). Least Squares Generative Adversarial Networks. arXiv:1611.04076.
T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. (2013). Distributed Embeddings of Words and Phrases and Their Compositionality. In Advances in Neural Information Processing Systems 26 (NIPS 2013).
T. Mikolov, K. Chen, G. Corrado, and J. Dean. (2013b). Efficient Estimation of Word Representations in Vector Space. In Workshop (ICLR 2013).
M. Mirza, S. Osindero. (2014). Conditional Generative Adversarial Nets. arXiv:1411.1784.
A. Odena. (2016). Semi-supervised Learning with Generative Adversarial Networks. arXiv:1606. 01583.
J. Pennington, R. Socher, and C. Manning. Glove: Global vectors for word representation. (2014). In Empirical Methods in Natural Language Processing (EMNLP 2014).
O. Press, A. Bar, B. Bogin, J. Berant, and L. Wolf. (2017). Language Generation with Recurrent Generative Adversarial Networks without Pre-training. In 1st Workshop on Subword and Character level models in NLP (EMNLP 2017).
S. Rajeswar, S. Subramanian, F. Dutil, C. Pal, and A. Courville. (2017). Adversarial Generation of Natural Language. arXiv:1705.10929.
W. Rand. (1971). Objective Criteria for the Evaluation of Clustering Methods. Journal of the American Statistical Association, 66:846-850.
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen. (2016). Improved Techniques for Training GANs. In Advances in Neural Information Processing Systems 29 (NIPS 2016).
R. Socher, A. Perelygin, Alex, J. Wu, J. Chuang, C. Manning, A. Ng, and C. Potts. (2013) Recursive deep models for semantic compositionality over a sentiment treebank. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2013).
J. Springenberg. (2016). Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks. In 4th International Conference on Learning embeddings (ICLR 2016).
L. van der Maaten, and G. Hinton. (2008). Visualizing Data using t-SNE. Journal of Machine Learning Research, 9:2579-2605.
B. Wang, K. Liu, and J. Zhao. (2016). Conditional Generative Adversarial Networks for Commonsense Machine Comprehension. In Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17).
Y. Zhang, Z. Gan, and L. Carin. (2016). Generating Text via Adversarial Training. In Workshop on Adversarial Training (NIPS 2016).
J. Zhao, M. Mathieu, and Y. LeCun. (2017). Energy-based Generative Adversarial Networks. In 5th International Conference on Learning embeddings (ICLR 2017).

Question: Which GAN do they use?
Answer: We construct a GAN model which combines different sets of word embeddings INLINEFORM4 , INLINEFORM5 , into a single set of word embeddings INLINEFORM6 . 

weGAN, deGAN

Question: Do they evaluate grammaticality of generated text?
Answer: No

Question: Which corpora do they use?
Answer: CNN, TIME, 20 Newsgroups, and Reuters-21578