Year: 2019

Abstract:
Generative adversarial nets (GANs) have been successfully applied to the artificial generation of image data. In terms of text data, much has been done on the artificial generation of natural language from a single corpus. We consider multiple text corpora as the input data, for which there can be two applications of GANs: (1) the creation of consistent cross-corpus word embeddings given different word embeddings per corpus; (2) the generation of robust bag-of-words document embeddings for each corpora. We demonstrate our GAN models on real-world text data sets from different corpora, and show that embeddings from both models lead to improvements in supervised learning problems.

heading: 1 Introduction
text: Generative adversarial nets (GAN) (Goodfellow et al., 2014) belong to a class of generative models which are trainable and can generate artificial data examples similar to the existing ones. In a GAN model, there are two sub-models simultaneously trained: a generative model G from which artificial data examples can be sampled, and a discriminative model D which classifies real data examples and artificial ones from G. By training G to maximize its generation power, and training D to minimize the generation power of G, so that ideally there will be no difference between the true and artificial examples, a minimax problem can be established. The GAN model has been shown to closely replicate a number of image data sets, such as MNIST, Toronto Face Database (TFD), CIFAR-10, SVHN, and ImageNet (Goodfellow et al., 2014; Salimans et al., 2016).
The GAN model has been extended to text data in a number of ways. For instance, Zhang et al. (2016) applied a long-short term memory (LSTM) (Hochreiter & Schmidhuber, 1997) generator and approximated discretization to generate text data. Moreover, Li et al. (2017) applied the GAN model to generate dialogues, i.e. pairs of questions and answers. Meanwhile, the GAN model can also be applied to generate bag-of-words embeddings of text data, which focus more on key terms in a text document rather than the original document itself. Glover (2016) provided such a model with the energy-based GAN (Zhao et al., 2017).
To the best of our knowledge, there has been no literature on applying the GAN model to multiple corpora of text data. Multi-class GANs have been proposed (Liu & Tuzel, 2016; Mirza & Osindero, 2014), but a class in multi-class classification is not the same as multiple corpora. Because knowing the underlying corpus membership of each text document can provide better information on how the text documents are organized, and documents from the same corpus are expected to share similar topics or key words, considering the membership information can benefit the training of a text model from a supervised perspective. We consider two problems associated with training multi-corpus text data: (1) Given a separate set of word Mikolov et al. (2013b), how to obtain a better set of cross-corpus word embeddings from them? (2) How to incorporate the generation of document embeddings from different corpora in a single GAN model?
For the first problem, we train a GAN model which discriminates documents represented by different word embeddings, and train the cross-corpus word embedding so that it is similar to each existing word embedding per corpus. For the second problem, we train a GAN model which considers both crosscorpus and per-corpus “topics” in the generator, and applies a discriminator which considers each original and artificial document corpus. We also show that with sufficient training, the distribution of
the artificial document embeddings is equivalent to the original ones. Our work has the following contributions: (1) we extend GANs to multiple corpora of text data, (2) we provide applications of GANs to finetune word embeddings and to create robust document embeddings, and (3) we establish theoretical convergence results of the multi-class GAN model.
Section 2 reviews existing GAN models related to this paper. Section 3 describes the GAN models on training cross-corpus word embeddings and generating document embeddings for each corpora, and explains the associated algorithms. Section 4 presents the results of the two models on text data sets, and transfers them to supervised learning. Section 5 summarizes the results and concludes the paper.

heading: 2 Literature Review
text: In a GAN model, we assume that the data examples x are drawn from a distribution px(·), and the artificial data examples G(z) := G(z, θg) are transformed from the noise distribution z ∼ pz(·). The binary classifier D(·) outputs the probability of a data example (or an artificial one) being an original one. Because the probabilistic structure of a GAN can be unstable to train, the Wasserstein GAN (Arjovsky et al., 2017) is proposed which applies a 1-Lipschitz function as a discriminator.
We note that in many circumstances, data sets are obtained with supervised labels or categories, which can add explanatory power to unsupervised models such as the GAN. For instance, the CoGAN (Liu & Tuzel, 2016) considers pairs of data examples from different categories, and the weights of the first few layers (i.e. close to z) are tied. Mirza & Osindero (2014) proposed the conditional GAN where the generator G and the discriminator D depend on the class label y. Salimans et al. (2016) applied the class labels for semi-supervised learning with an additional artificial class. However, all these models consider only images and do not produce word or document embeddings, therefore being different from our models.
For generating real text, Zhang et al. (2016) proposed textGAN in which the generator has an LSTM form, and a uni-dimensional convolutional neural network (Collobert et al., 2011; Kim, 2014) is applied as the discriminator. Also, a weighted softmax function is applied to make the argmax function differentiable. The focus of our work is to summarize information from longer documents, so we apply document embeddings such as the tf-idf to represent the documents rather than to generate real text.
For generating bag-of-words embeddings of text, Glover (2016) proposed a GAN model with the mean squared error of a de-noising autoencoder as the discriminator, and the output x is the one-hot word embedding of a document. Our models are different from this model because we consider tf-idf document embeddings for multiple text corpora in the deGAN model (Section 3.2), and weGAN (Section 3.1) can be applied to produce word embeddings. Also, we focus on robustness based on several corpora, while Glover (2016) assumed a single corpus.
For extracting word embeddings given text data, Mikolov et al. (2013a) proposed the word2vec model, for which there are two variations: the continuous bag-of-words (cBoW) model (Mikolov et al., 2013a), where the neighboring words are used to predict the appearance of each word; the skip-gram model, where each neighboring word is used individually for prediction. In GloVe (Pennington et al., 2014), a bilinear regression model is trained on the log of the word co-occurrence matrix. In these models, the weights associated with each word are used as the embedding. For obtaining document embeddings, the para2vec model (Le & Mikolov, 2014) adds per-paragraph vectors to train word2vec-type models, so that the vectors can be used as embeddings for each paragraph. A simpler approach by taking the average of the embeddings of each word in a document and output the document embedding is exhibited in Socher et al. (2013).

heading: 3 Models and Algorithms
text: Suppose we have a number of different corpora C1, . . . , CM , which for example can be based on different categories or sentiments of text documents. We suppose that Cm = {dm1 , . . . , dmnm}, m = 1, . . . ,M , where each dmi represents a document. The words in all corpora are collected in a
dictionary, and indexed from 1 to V . We name the GAN model to train cross-corpus word embeddings as “weGAN,” where “we” stands for “word embeddings,” and the GAN model to generate document embeddings for multiple corpora as “deGAN,” where “de” stands for “document embeddings.”

heading: 3.1 weGAN: Training cross-corpus word embeddings
text: We assume that for each corpora Cm, we are given word embeddings for each word vm1 , . . . , vmV ∈ Rd, where d is the dimension of each word embedding. We are also given a classification task on documents that is represented by a parametric model C taking document embeddings as feature vectors. We construct a GAN model which combines different sets of word embeddings Vm := {vmi }Vi=1, m = 1, . . . ,M , into a single set of word embeddings G := {v0i }Vi=1. Note that V1, . . . ,VM are given but G is trained. Here we consider G as the generator, and the goal of the discriminator is to distinguish documents represented by the original embeddings V1, . . . ,VM and the same documents represented by the new embeddings G. Next we describe how the documents are represented by a set of embeddings V1, . . . ,VM and G. For each document dmi , we define its document embedding with Vm as gmi := f(dmi ,Vm), where f(·) can be any mapping. Similarly, we define the document embedding of dmi with G as follows, with G = {v0j }Vj=1 trainable fG(dmi ) := f(dmi ,G). In a typical example, word embeddings would be based on word2vec or GLoVe. Function f can be based on tf-idf, i.e. f(dmi ,V) = ∑V j=1 t m ij v m j where vmj is the word embedding of the j-th word in the m-th corpus Cm and tmi = (tmi1, . . . , tmiV ) is the tf-idf representation of the i-th document dmi in the m-th corpus Cm. To train the GAN model, we consider the following minimax problem
min C,G max D
{ M∑
m=1 nm∑ i=1 [log(D(gmi )) + log(1− D(fG(dmi )))− log(eTkmi C(fG(d m i )))] } , (1)
where D is a discriminator of whether a document is original or artificial. Here kmi is the label of document dmi with respect to classifier C, and ekmi is a unit vector with only the k m i -th component being one and all other components being zeros. Note that log(eTkmi C(fG(d m i ))) is equivalent to KL(ekmi ‖C(fG(d m i ))), but we use the former notation due to its brevity.
The intuition of problem (3) is explained as follows. First we consider a discriminator D which is a feedforward neural network (FFNN) with binary outcomes, and classifies the document embeddings {fG(dmi )} nm i=1 M m=1 against the original document embeddings {gmi } nm i=1 M m=1. Discriminator D minimizes this classification error, i.e. it maximizes the log-likelihood of {fG(dmi )} nm i=1 M m=1 having label 0 and {gmi } nm i=1 M m=1 having label 1. This corresponds to
M∑ m=1 nm∑ i=1 [log(D(gmi )) + log(1−D(fG(dmi )))] . (2)
For the generator G, we wish to minimize (3) against G so that we can apply the minimax strategy, and the combined word embeddings G would resemble each set of word embeddings V1, . . . ,VM . Meanwhile, we also consider classifier C with K outcomes, and associates dmi with label kmi , so that the generator G can learn from the document labeling in a semi-supervised way. If the classifier C outputs a K-dimensional softmax probability vector, we minimize the following against G, which corresponds to (3) given D and C:
M∑ m=1 nm∑ i=1 [log(1−D(fG(dmi )))− log(eTkmi C(fG(d m i ))) ] . (3)
For the classifier C, we also minimize its negative log-likelihood
− M∑
m=1 nm∑ i=1 log(eTkmi C(fG(d m i ))). (4)
Assembling (4-6) together, we retrieve the original minimax problem (3).
We train the discriminator and the classifier, {D, C}, and the combined embeddings G according to (4-6) iteratively for a fixed number of epochs with the stochastic gradient descent algorithm, until the discrimination and classification errors become stable.
Figure 1 illustrates the weGAN model. The algorithm for weGAN is summarized in Algorithm 1 in the appendix.
𝒞1 𝒞2 ⋯ 𝒞𝑀
𝑑1 1 ⋯ 𝑑𝑛1 1 𝑑1 2 ⋯ 𝑑𝑛2 2 ⋯ 𝑑1 𝑀 ⋯ 𝑑𝑛𝑀 𝑀
𝑔1 1, … , 𝑔𝑛1 1
corpora
documents
𝒱1 word
embeddings
document
embeddings
𝒟discriminator
𝐹𝑟𝑜𝑚 {𝒱1, … , 𝒱𝑀} 𝑜𝑟 𝒢
𝒞classifier
output
𝒱2
𝑔1 2, … , 𝑔𝑛2 2
𝒱𝑀
𝑔1 𝑀 , … , 𝑔𝑛𝑀 𝑀
𝑡𝑛𝑀 𝑀{𝑑1 1 , … , 𝑑𝑛𝑀 𝑀 }
𝒢
{𝑓𝒢 𝑑1 1 , … , 𝑓𝒢(𝑑𝑛𝑀 𝑀 )}
Label distribution
Figure 1: Model structure of weGAN.
𝒞1 𝒞2 ⋯ 𝒞𝑀
𝑑1 1 ⋯ 𝑑𝑛1 1 𝑑1 2 ⋯ 𝑑𝑛2 2 ⋯ 𝑑1 𝑀 ⋯ 𝑑𝑛𝑀 𝑀
corpora
document
embeddings
𝒟discriminator
𝐹𝑟𝑜𝑚 𝒞1, 𝒞2, … , 𝒞𝑀 𝑜𝑟 {𝒢1, … , 𝒢𝑀}output
𝒢1(𝑛)
generators
𝑊ℎ 0,𝑊𝑜 0
𝒢1 𝒢𝑀𝒢2
noise𝑛
𝒢2(𝑛) 𝒢𝑀(𝑛)generated embeddings
𝑊ℎ 1,𝑊𝑜 1 𝑊ℎ 2,𝑊𝑜 2 𝑊ℎ 𝑀,𝑊𝑜 𝑀
Figure 2: Model structure of deGAN.

heading: 3.2 deGAN: Generating document embeddings for multi-corpus text data
text: In this section, our goal is to generate document embeddings which would resemble real document embeddings in each corpus Cm, m = 1, . . . ,M . We construct M generators, G1, . . . ,GM so that Gm generate artificial examples in corpus Cm. As in Section 3.1, there is a certain document embedding such as tf-idf, bag-of-words, or para2vec. Let G = {G1, . . . ,GM}. We initialize a noise vector n = (n1, . . . , ndn) ∈ Rdn , where n1, . . . , ndn
iid∼ N , and N is any noise distribution. For a generator Gm = {Wmh ,W 0h ,Wmo ,W 0o } represented by its parameters, we first map the noise vector n to the hidden layer, which represents different topics. We consider two hidden vectors, h0 for general topics and hm for specific topics per corpus, hm = a1(Wmh n), h 0 = a1(W 0 hn). Here a1(·) represents a nonlinear activation function. In this model, the bias term can be ignored in order to prevent the “mode collapse” problem of the generator. Having the hidden vectors, we then map them to the generated document embedding with another activation function a2(·),
om = a2(W m o h m + w0oh 0). (5)
To summarize, we may represent the process from noise to the document embedding as Gm(n) = a2(W m o a1(W m h n) + w 0 oa1(W 0 hn)). Given the generated document embeddings G1(n), . . . ,GM (n), we consider the following minimax problem to train the generator G and the discriminator D:
min G M∑ m=1 En log { eTM+mD∗G(Gm(n))/ [eTM+mD∗G(Gm(n)) + eTmD∗G(Gm(n))] } , (6)
where
D∗G ∈ argmaxD M∑ m=1 Edm∼pm [ log(eTmD(dm)) ] + M∑ m=1 En[log(e T M+mD(Gm(n)))]. (7)
Here we assume that any document embedding dm in corpus Cm is a sample with respect to the probability density pm. Note that when M = 1, the discriminator part of our model is equivalent to the original GAN model.
To explain (10), first we consider the discriminator D. Because there are multiple corpora of text documents, here we consider 2M categories as output ofD, from which categories 1, . . . ,M represent the original corpora C1, . . . , CM , and categories M + 1, . . . , 2M represent the generated document embeddings (e.g. bag-of-words) from G1, . . . ,GM . Assume the discriminator D, a feedforward neural network, outputs the distribution of a text document being in each category. We maximize the
log-likelihood of each document being in the correct category against D M∑
m=1
Epm [ log(eTmD(dm)) ] + M∑ m=1 En[log(e T M+mD(Gm(n)))]. (8)
Such a classifier does not only classifies text documents into different categories, but also considers M “fake” categories from the generators. When training the generators G1, . . . ,GM , we minimize the following which makes a comparison between the m-th and (M +m)-th categories
M∑ m=1 En log eTM+mD(Gm(n)) eTM+mD(Gm(n)) + eTmD(Gm(n)) . (9)
The intuition of (13) is that for each generated document embedding Gm(n), we need to decrease eTM+mD(Gm(n)), which is the probability of the generated embedding being correctly classified, and increase eTmD(Gm(n)), which is the probability of the generated embedding being classified into the target corpus Cm. The ratio in (13) reflects these two properties. We iteratively train (12) and (13) until the classification error ofD becomes stable. Figure 2 illustrates the deGAN model. The algorithm for deGAN is summarized in Algorithm 2 in the appendix.
We next show that from (10), the distributions of the document embeddings from the optimal G1, . . . ,GM are equal to the data distributions of C1, . . . , CM , which is a generalization of Goodfellow et al. (2014) to the multi-corpus scenario. The proof of Proposition 1 is in the appendix.
Proposition 1. Let us assume that the random variables d1, . . . , dM are continuous with probability density p1, . . . , pM which have bounded support X ; n is a continuous random variable with bounded support and activations a1 and a2 are continuous; and that G1∗, . . . ,GM∗ are solutions to (10). Then q∗1 , . . . , q ∗ M , the probability density of the document embeddings from G1∗, . . ., GM∗, are equal to p1, . . . , pM .

heading: 4 Experiments
text: In the experiments, we consider four data sets, two of them newly created and the remaining two already public: CNN, TIME, 20 Newsgroups (in the appendix), and Reuters-21578 (in the appendix). The code and the two new data sets are available at github.com/deeplearning2018/emgan. For the pre-processing of all the documents, we transformed all characters to lower case, stemmed the documents, and ran the word2vec model on each corpora to obtain word embeddings with a size of 300. In all subsequent models, we only consider the most frequent 5,000 words across all corpora in a data set.
The document embedding in weGAN is the tf-idf weighted word embedding transformed by the tanh activation, i.e. f(dmi ,Vm) = tanh (∑V j=1 t m ij v m j ) . For deGAN, we use L1-normalized tf-idf as the document embedding because it is easier to interpret than the transformed embedding in (20).
For weGAN, the cross-corpus word embeddings are initialized with the word2vec model trained from all documents. For training our models, we apply a learning rate which increases linearly from 0.01 to 1.0 and train the models for 100 epochs with a batch size of 50 per corpus. The classifier C has a single hidden layer with 50 hidden nodes, and the discriminator with a single hidden layer D has 10 hidden nodes. All these parameters have been optimized. For the labels kmi in (8), we apply corpus membership of each document.
For the noise distribution N for deGAN, we apply the uniform distribution U(−1, 1). In (14) for deGAN, a1 = tanh and a2 = softmax so that the model outputs document embedding vectors which are comparable to L1-normalized tf-idf vectors for each document. For the discriminator D of deGAN, we apply the word2vec embeddings based on all corpora to initialize its first layer, followed by another hidden layer of 50 nodes. For the discriminator D, we apply a learning rate of 0.1, and for the generator G, we apply a learning rate of 0.001, because the initial training phase of deGAN can be unstable. We also apply a batch size of 50 per corpus. For the softmax layers of deGAN, we initialize them with the log of the topic-word matrix in latent Dirichlet allocation (LDA) (Blei et al., 2003) in order to provide intuitive estimates.
w2v-RI weGAN-RI mean 67.88% 68.45%
sd. 0.02% 0.01% w2v-accuracy weGAN-accuracy mean 92.05% 92.36% sd. 0.06% 0.03%
Table 1: A comparison between word2vec and weGAN in terms of the Rand index and the classification accuracy for the CNN data set.
Obama w2v Bush Trump Kerry Abe Netanyahu Rouhani Erdogan he Karzai Tillerson Obama weGAN Trump Bush Kerry Abe Netanyahu Erdogan Tillerson he Carter Rouhani Trump w2v Obama Pence Erdogan Bush Duterte he Sanders Macron Christie Tillerson Trump weGAN Obama Pence Bush Christie Sanders Clinton Erdogan Tillerson Macron
Duterte U.S. w2v US Pentagon United Iranian NATO Turkish Qatar Iran British UAE U.S. weGAN US Pentagon United Iranian
NATO Turkish Iran Qatar American UAE
Table 2: Synonyms of “Obama,” “Trump,” and “U.S.” before and after weGAN training for the CNN data set.
For weGAN, we consider two metrics for comparing the embeddings trained from weGAN and those trained from all documents: (1) applying the document embeddings to cluster the documents into M clusters with the K-means algorithm, and calculating the Rand index (RI) (Rand, 1971) against the original corpus membership; (2) finetuning the classifier C and comparing the classification error against an FFNN of the same structure initialized with word2vec (w2v). For deGAN, we compare the performance of finetuning the discriminator of deGAN for document classification, and the performance of the same FFNN. Each supervised model is trained for 500 epochs and the validation data set is used to choose the best epoch.

heading: 4.1 The CNN data set
text: In the CNN data set, we collected all news links on www.cnn.com in the GDELT 1.0 Event Database from April 1st, 2013 to July 7, 2017. We then collected the news articles from the links, and kept those belonging to the three largest categories: “politics,” “world,” and “US.” We then divided these documents into 21,674 training documents, from which 2,708 validation documents are held out, and 5,420 testing documents.
We hypothesize that because weGAN takes into account document labels in a semi-supervised way, the embeddings trained from weGAN can better incorporate the labeling information and therefore, produce document embeddings which are better separated. The results are shown in Table 1 and averaged over 5 randomized runs. Performing the Welch’s t-test, both changes after weGAN training are statistically significant at a 0.05 significance level. Because the Rand index captures matching accuracy, we observe from Table 1 that weGAN tends to improve both metrics.
Meanwhile, we also wish to observe the spatial structure of the trained embeddings, which can be explored by the synonyms of each word measured by the cosine similarity. On average, the top 10 synonyms of each word differ by 0.22 word after weGAN training, and 20.7% of all words have different top 10 synonyms after training. Therefore, weGAN tends to provide small adjustments rather than structural changes. Table 2 lists the 10 most similar terms of three terms, “Obama,” “Trump,” and “U.S.,” before and after weGAN training, ordered by cosine similarity.
We observe from Table 2 that for “Obama,” ”Trump” and “Tillerson” are more similar after weGAN training, which means that the structure of the weGAN embeddings can be more up-to-date. For “Trump,” we observe that “Clinton” is not among the synonyms before, but is after, which shows that the synonyms after are more relevant. For “U.S.,” we observe that after training, “American” replaces “British” in the list of synonyms, which is also more relevant.
We next discuss deGAN. In Table 3, we compare the performance of finetuning the discriminator of deGAN for document classification, and the performance of the FFNN initialized with word2vec. The change is also statistically significant at the 0.05 level. From Table 3, we observe that deGAN improves the accuracy of supervised learning.
US original climate change year study according says country temperatures average US deGAN area efforts volunteers town
weapons shot local nearly department also
Table 4: Bag-of-words representations of original and artificial text in the CNN data set.
To compare the generated samples from deGAN with the original bag-of-words, we randomly select one record in each original and artificial corpus. The records are represented by the most frequent words sorted by frequency in descending order where the stop words are removed. The bag-of-words embeddings are shown in Table 4.
From Table 4, we observe that the bag-of-words embeddings of the original documents tend to contain more name entities, while those of the artificial deGAN documents tend to be more general. There are many additional examples not shown here with observed artificial bag-of-words embeddings having many name entities such as “Turkey,” “ISIS,” etc. from generated documents, e.g. “Syria eventually ISIS U.S. details jet aircraft October video extremist...”
10 5 0 5 10 10
5
0
5
10
Figure 3: 2-d representation of original (red) and artificial (blue) examples in the CNN data set.
15 10 5 0 5 10 15 15
10
5
0
5
10
15
Figure 4: 2-d representation of original (red) and artificial (blue) examples in the TIME data set.
We also perform dimensional reduction using t-SNE (van der Maaten & Hinton, 2008), and plot 100 random samples from each original or artificial category. The original samples are shown in red and the generated ones are shown in blue in Figure 3. We do not further distinguish the categories because there is no clear distinction between the three original corpora, “politics,” “world,” and “US.”
We observe that the original and artificial examples are generally mixed together and not well separable, which means that the artificial examples are similar to the original ones. However, we also observe that the artificial samples tend to be more centered and have no outliers (represented by the outermost red oval).

heading: 4.2 The TIME data set
text: In the TIME data set, we collected all news links on time.com in the GDELT 1.0 Event Database from April 1st, 2013 to July 7, 2017. We then collected the news articles from the links, and kept
those belonging to the five largest categories: “Entertainment,” “Ideas,” “Politics,” “US,” and “World.” We divided these documents into 12,286 training documents, from which 1,535 validation documents are held out, and 3,075 testing documents.
Table 5 compares the clustering results of word2vec and weGAN, and the classification accuracy of an FFNN initialized with word2vec, finetuned weGAN, and finetuned deGAN. The results in Table 5 are the counterparts of Table 1 and Table 3 for the TIME data set. The differences are also significant at the 0.05 level.
Obama w2v Trump Bush Xi Erdogan Rouhani Reagan Hollande Duterte Abe Jokowi Obama weGAN Trump Bush Xi Erdogan Reagan Rouhani Hollande Abe Jokowi Duterte Trump w2v Obama Erdogan Rubio Duterte Bush Putin Sanders Xi Macron Pence Trump weGAN Obama Erdogan Rubio Bush Sanders Putin Duterte Xi Macron Pence
U.S. w2v NATO Iran Japan Pentagon Russia Pakistan Tehran EU Ukrainian Moscow U.S. weGAN NATO Pentagon Iran Japan Russia Tehran Pakistan EU Ukrainian Moscow
Table 6: Synonyms of “Obama,” “Trump,” and “U.S.” before and after weGAN training for the TIME data set.
From Table 5, we observe that both GAN models yield improved performance of supervised learning. For weGAN, on an average, the top 10 synonyms of each word differ by 0.27 word after weGAN training, and 24.8% of all words have different top 10 synonyms after training. We also compare the synonyms of the same common words, “Obama,” “Trump,” and “U.S.,” which are listed in Table 6. In the TIME data set, for “Obama,” “Reagan” is ranked slightly higher as an American president. For “Trump,” “Bush” and “Sanders” are ranked higher as American presidents or candidates. For “U.S.,” we note that “Pentagon” is ranked higher after weGAN training, which we think is also reasonable because the term is closely related to the U.S. government.
For deGAN, we also compare the original and artificial samples in terms of the highest probability words, which is shown in Table 7. We also perform dimensional reduction using t-SNE for 100 examples per corpus and plot them in Figure 4. All these figures and tables show results similar to Section 4.1.

heading: A Algorithms for weGAN and deGAN
text: Algorithm 1 (for weGAN).
1. Train G based on f from all corpora C1, . . . , CM . 2. Randomly initialize the weights and biases of the classifier C and discriminator D. 3. Until maximum number of iterations reached
(a) Update C and D according to (4) and (6) given a mini-batch S1 of training examples {dmi }i,m.
(b) Update G according to (5) given a mini-batch S2 of training examples {dmi }i,m. 4. Output G as the cross-corpus word embeddings.
Algorithm 2 (for deGAN).
1. Randomly initialize the weights of G1, . . . ,GM . 2. Initialize the discriminator D with the weights of the first layer (which takes document
embeddings as the input) initialized by word embeddings, and other parameters randomly initialized.
3. Until maximum number of iterations reached (a) Update D according to (12) given a mini-batch of training examples dmi and samples
from noise n. (b) Update G1, . . . ,GM according to (13) given a mini-batch of training examples dmi and
samples form noise n. 4. Output G1, . . . ,GM as generators of document embeddings and D as a corpus classifier.

heading: B Proof of Proposition 1
text: Since X is bounded, all of the integrals exhibited next are well-defined and finite. Since n, a1, and a2 are continuous, it follows that for any parameters, Gm(n) is a continuous random variable with probability density qm with finite support. From (11),
D∗G = argmaxD
{ M∑
m=1
∫ pm(x) log(e T mD(x))dx
+ M∑ m=1 ∫ qm(x) log(e T M+mD(x))dx }
= argmax D {∫ M∑ m=1 pm(x) log(e T mD(x))
+ M∑ m=1 qm(x) log(e T M+mD(x))dx
} . (10)
This problem reduces to
max b1,...,bm M∑ m=1 am log bm subject to M∑ m=1 bm = 1, (11)
the solution of which is b∗m = am/ ∑M m=1 am, m = 1, . . . ,M . Therefore, the solution to (15) is
D∗G(x) = (p1(x), . . . , pM (x), q1(x), . . . , qM (x))∑M m=1 pm(x) + ∑M m=1 qm(x) . (12)
We then obtain from (10) that
q∗1 , . . . , q ∗ M ∈ arg min
q1,...,qM M∑ m=1 ∫ qm(x) · log [ qm(x) qm(x) + pm(x) ] dx
= arg min q1,...,qM
−M log 2 + M∑
m=1
∫ qm(x) log [ qm(x)
(qm(x) + pm(x))/2
] dx
= arg min q1,...,qM
−M log 2 + M∑
m=1
KL(qm‖(qm + pm)/2).
From non-negativity of the Kullback-Leibler divergence, we conclude that
(q∗1 , . . . , q ∗ M ) = (p1, . . . , pM ).

heading: C The 20 Newsgroups data set
text: The 20 Newsgroups data set is a collection of news documents with 20 categories. To reduce the number of categories so that the GAN models are more compact and have more samples per corpus, we grouped the documents into 6 super-categories: “religion,” “computer,” “cars,” “sport,” “science,” and “politics” (“misc” is ignored because of its noisiness). We considered each super-category as a different corpora. We then divided these documents into 10,708 training documents, from which 1,335 validation documents are held out, and 7,134 testing documents. We train weGAN and deGAN in the the beginning of Section 4, except that we use a learning rate of 0.01 for the discriminator in deGAN to stabilize the cost function. Table 8 compares the clustering results of word2vec and weGAN, and the classification accuracy of the FFNN initialized with word2vec, finetuned weGAN, and finetuned deGAN. All comparisons are statistically significant at the 0.05 level. The other results are similar to the previous two data sets and are thereby omitted here.

heading: D The Reuters-21578 data set
text: The Reuters-21578 data set is a collection of newswire articles. Because the data set is highly skewed, we considered the eight categories with more than 100 training documents: “earn,” “acq,” “crude,” “trade,” “money-fx,” “interest,” “money-supply,” and “ship.” We then divided these documents into 5,497 training documents, from which 692 validation documents are held out, and 2,207 testing documents. We train weGAN and deGAN in the same way as in the 20 Newsgroups data set. Table 9 compares the clustering results of word2vec and weGAN, and the classification accuracy of the FFNN initialized with word2vec, finetuned weGAN, and finetuned deGAN. All comparisons are statistically significant at the 0.05 level except the Rand index. The other results are similar to the CNN and TIME data sets and are thereby omitted here.

Review summaries:
The paper proposes extensions of Generative Adversarial Networks to modeling multiple text corpora. Concretely, the paper looks at two problems: 1) given independently pretrained word embeddings from K corpora, finding a common word embdding, 2) extracting document representations from a discriminator of a GAN trained to generate tf-idf vectors. Preliminary experiments show that the proposed approaches outperform baseline classifiers trained with word2vec.

The paper proposes to use Generative Adversarial Networks (GANs) in the context of natural language processing and introduces two models for generating document embeddings. The first model, weGAN, aggregates multiple sets of single-corpus word2vec embeddings into one set of cross-corpus word representations; document embeddings are a function of these updated word embeddings. The second model, deGAN, side-steps word-level embeddings and directly generates document-level representations. For both models, the real examples come from word2vec and tf-idf, while the artificial examples are the ones generated by the network. The authors show that their document embeddings are better than the word2vec/tf-idf baseline at clustering documents according to the corpus they originate from.