Generative Adversarial Nets for Multiple Text Corpora
Authors: Diego Klabjan
Baiyang Wang
Venue:  (2020)
Keywords: GAN
NLP
embeddings

Abstract: Generative adversarial nets (GANs) have been successfully applied to the artificial generation of image data. In terms of text data, much has been done on the artificial generation of natural language from a single corpus. We consider multiple text corpora as the input data, for which there can be two applications of GANs: (1) the creation of consistent cross-corpus word embeddings given different word embeddings per corpus; (2) the generation of robust bag-of-words document embeddings for each corpora. We demonstrate our GAN models on real-world text data sets from different corpora, and show that embeddings from both models lead to improvements in supervised learning problems.

Under review as a conference paper at ICLR 2020
GENERATIVE ADVERSARIAL NETS FOR MULTIPLE TEXT CORPORA
Anonymous authors Paper under double-blind review
ABSTRACT
Generative adversarial nets (GANs) have been successfully applied the artificial generation of image data. In terms of text data, much has been done on the artificial generation of natural language from single corpus. We consider multiple text corpora as the input data, for which there can be two applications of GANs: (1) the creation of consistent cross-corpus word embeddings given different word embeddings per corpus; (2) the generation of robust bag-of-words document em- beddings for each corpora. We demonstrate our GAN models on real-world text data sets from different corpora, and show that embeddings from both models lead improvements in supervised learning problems.
Introduction
Generative adversarial nets (GAN) (Goodfellow al. 2014) belong class of generative models which are trainable and can generate artificial data examples similar to the existing ones. In a GAN model, there are two sub-models simultaneously trained: a generative model & from which artificial data examples can be sampled, and discriminative model 2 which classifies real data examples and artificial ones from G. By training G maximize its generation power, and training D to minimize the generation power of G, so that ideally there will be no difference between the true and artificial examples, minimax problem can be established. The GAN model has been shown to closely replicate number of image data sets, such as MNIST, Toronto Face Database (TFD), CIFAR-10, SVHN, and ImageNet (Goodfellow et al.) 2014 Salimans et al. 2016).
The GAN model has been extended to text data in number of ways. For instance, Zhang et al. (2016) applied long-short term memory (LSTM) (Hochreiter & Schmidhuber 1997) generator and approximated discretization to generate text data. Moreover, Li et (2017) applied the GAN model to generate dialogues, i.e. pairs of questions and answers. Meanwhile, the GAN model can also be applied to generate Pag-of-worde embeddings of text data, which focus more key terms in a text document rather than the original document itself. Glover (2016) provided such a model with the energy-based GAN (Zhao et al. 2017).
To the best of our knowledge, there has been no literature on applying the GAN model to multiple corpora of text data. Multi-class GANs have been proposed (Liu & Tuzel 2016 Mirza & Osindero 2014), but a class in multi-class classification is not the same as multiple corpora. Because knowing the underlying corpus membership of each text document can provide better information on how the text documents are organized, and documents from the same corpus are expected to share similar topics or key words, considering the membership information can benefit the training of text model from supervised perspective. We consider two problems associated with training multi-corpus text data: (1) Given separate set of word Mikolov et al. (2013b), how obtain a better set of cross-corpus word embeddings from them? (2) How to incorporate the generation of document embeddings from different corpora a single GAN model?
For the first problem, we train GAN model which discriminates documents represented by different word embeddings, and train the cross-corpus word embedding so that it is similar to each existing word embedding per corpus. For the second problem, train a GAN model which considers both cross- corpus and per-corpus "topics" in the generator, and applies a discriminator which considers each original and artificial document corpus. We also show that with sufficient training, the distribution of

---

Under review as a conference paper at ICLR 2020
the artificial document embeddings is equivalent to the original ones. Our work has the following contributions: (1) we extend GANs to multiple corpora of text data, (2) we provide applications of GANs to finetune word embeddings and to create robust document embeddings, and (3) we establish theoretical convergence results of the multi-class GAN model.
Section 2 reviews existing GAN models related to this paper. Section 3 describes the GAN models on training cross-corpus word embeddings and generating document embeddings for each corpora, and explains the associated algorithms. Section 4 presents the results of the two models on text data sets, and transfers them to supervised learning. Section 5 summarizes the results and concludes the paper.
2 Literature Review
In a GAN model, we assume that the data examples x are drawn from a distribution pxx()), and the
artificial data examples G(z) (lz. 0g) are transformed from the noise distribution z ~ p>(+). (-). The binary classifier (.) outputs the probability of a data example (or an artificial one) being an original one. Because the probabilistic structure of a GAN can be unstable to train, the Wasserstein GAN (Arjovsky et al. 2017) is proposed which applies a 1-Lipschitz function as a discriminator.
We note that in many cireumstances, data sets are obtained with supervised labels categories, which can add explanatory power to unsupervised models such as the GAN. For instance, the CoGAN (Liu & Tuzel 2016) considers pairs of data examples from different categories, and the weights of the first few layers (i.e. close to are tied. Mirza & Osindero (2014) proposed the conditional GAN where generator G and the discriminator DD depend on the class label y- [Salimans al. (2016) applied the class labels for semi-supen supervisd learning with additional artificial class. However, all these models consider only images and do not produce word or document embeddings, therefore being different from our models.
For generating real text, Zhang et al. (2016) proposed textGAN which the generator has an LSTM form, and uni-dimensional convolutional neural network (Collobert et al. 2011 Kim 2014) is applied as the discriminator. Also, a weighted softmax function is applied to make the argmax function differentiable. The focus of our work is summarize information from longer documents, so we apply document embeddings such as the tf-idf to represent the documents rather than to generate real text.
For generating bag-of-words embeddings of text, Glover (2016) proposed a GAN model with the mean squared error of de-noising autoencoder the discriminator, and the output x is the one-hot word embedding of document. Our models are different from this model because we consider tf-idf document embeddings for multiple text corpora the deGAN model (Section 3.2), and weGAN (Section 3.1) can be applied to produce word embeddings. Also, we focus on robustness based on several corpora, while Glover (2016) assumed single corpus.
For extracting word embeddings given text data, Mikolov et al. (2013a) proposed the word2vec model, for which there are two variations: the continuous -oo-dsswords (cBoW) model (Mikolov al., 2013a), where the neighboring words are used to predict the appearance of each word; the skip-gram GloVe (Pennington
model, where each neighboring word used individually for prediction. et al. 2014), bilinear regression model is trained the log of the word co-occurrence matrix. In these models, the weights associated with each word are used as the embedding. For obtaining document embeddings, the para2vee model (Le & Mikolov (014)) adds per-paragraph vectors to train word2vec-type models, so that the vectors can be used as embeddings for each paragraph. A simpler approach by taking the average of the embeddings of each word in document and output the document embedding is exhibited in Socher et al. (2013).
3 Models and Algorithms
Suppose have number of different corpora Cl.....C CMM which for example can be based on different categories or sentiments of text documents. We suppose that Cm {dT" ...., d 1, 1,...; M, where each d'e represents document. The words in all corpora are collected in
2

---

Under review as a conference paper at ICLR 2020
dictionary, and indexed from 1 to V. We name the GAN model to train cooss- word embeddings as "weGAN," where "we" stands for "word embeddings." and the GAN model to generate document embeddings for multiple corpora as "deGAN," where "de" stands for "document embeddings."
3.1 weGAN: Training cross-corpus word embeddings
We assume that for each corpora C",, we are given word embeddings for each word ui" vy E Rd where the dimension of each word embedding. We are also given classification task on documents that is represented by parametric model C taking document embeddings as feature vectors. We construct a GAN model which combines different sets word embeddings 1 : {o:" LV m
M, into single set of word embeddings g == {E²) Ji=1 Note that V ..... V are given but & is trained. Here we consider & as the generator, and the goal of the discriminator is to distinguish documents represented by the original embeddings V"1 ,...; VM and the same documents represented by the new embeddings G.
Next we describe how the documents are represented by a set of embeddings V1 ..... VM and g. For each document d we define its document embedding with y.t g,"h :- f(d Vmn,, where f(-) can any mapping. Similarly, we define the document embedding of d with G as follows, with G { trainable fo(d"") == f(d 9). In typical example, word embeddings would be based word2vec or GLoVe. Function f can be based on tf-idf, i.e. f(d V) =E "7 where o is the word embedding of the j-th word in the m-th corpus Cm and t.t (111..... the tf-idf representation of the i-th document d,"h in the m-th corpus CmM.
................................ tVV
To train the GAN model, we consider the following minimax problem
M
min max C D
[og((D()))) + log(1- D(So(dT))) log log(ez ((s(d)))
(I)
where discriminator of whether a document original or artificial. Here km the label of document d'e with respect to classifier C, and ex:" is unit vector with only the kmm-thh component being one and all other components being zeros. Note that L [[C(f((d"")) ))), but we use the former notation due to its brevity.
is equivalent to
The intuition of problem (3) is explained as follows. First we consider discriminator D which is feedforward neural network (FFNN) with binary outcomes, and classifies the document embeddings m=1 against the original document embeddings {g,"" 1 == m=1- Discriminator D mini- mizes this classification error, i.c. it maximizes the log- likelihood of (dg He m=1 having label
{ff )m M
M
M
0 and [g Jiml m=1 having label 1. This corresponds to
M
M
[og(((())) + log(1 - D(fg(d
(2)
For the generator G, we wish minimize (3) against that we can apply the minimax strategy, and the combined word embeddings G would resemble each set of word embeddings V Vi...., V Meanwhile, we also consider classifier with K outcomes, and associates d with label kT so that the generator G can learn from the document labeling in semi-supervised way.
the classifier outputs K-dimensional softmax probability vector, we minimize the following against G, which corresponds to (3) given D and C:
M
[log(1 D) Cf
(3)
For the classifier C, we also minimize its negative log likelihodd
M
log T C(Jg )))
(4)
m=1
1=1
Assembling (4-6) together, retrieve the original minimax problem (3).
3

---

Assume the discriminator D, feedforward Under review as a conference paper at ICLR 2020
We train the discriminator and the classifier, {D, C), and the combined embeddings & according to (4-6) iteratively for a fixed number epochs with the stochastic gradient descent algorithm, until the discrimination and classification errors become stable.
Figure Iillustrates the weGAN model. The algorithm for weGAN summarized in Algorithm in the appendix.
corpora
corpara W
documenls
d5
di
do
W.. W.
W W
10 W
word ambaddings
y1
generaioos embookinggs generaled embeddiings
9
document embeddings
Flonn 06.
Ba(d) -... www.--olleee I.
F (m)
dicctminator
classifier
discriminator
output
From (V' parorr
Label distrbutton
ourgut
From C2
Cx
Figure 1:
Model structure of weGAN.
Figure 2: Model structure of deGAN.
3.2 deGAN: Generating document embeddings for multi-corpus text data
In this section, our goal is to generate document embeddings which would resemble real document embeddings each corpus C"" m 1. 1..... .M. We construct M generators, g'...... GM so that Gmd generate artificial examples in corpus C"". As in Section 3.1, there is certain document embedding such as if-idf, bag-of-words, or para2vee. Let G = {91.... 9M}. We initialize noise vector (M1, (..,..., )d.)) ) Rd=, where ; iid N, and A is any noise distribution.
For generator 9Tm = {Wrr, W vector n the hidden layer, which represents different topics. We consider two hidden vectors, ho for general topics and hT for specific topics per corpus, hm = aj (WT"") h aj (WPn). Here a1( represents nonlinear activation function. In this model, the bias term can be ignored order to prevent the "mode collapse" problem of the generator. Having the hidden vectors, we then map them to the generated document embedding with another activation function a2(-).
hir WmT, Wo represented by its parameters, we first map the noise
0z ("""h" + n1
(5)
To summarize, we may represent the process from noise to the document embedding as GTd (n)
a2 (W aj (W7"' n) + uga,( (WFn)) Given the generated document embeddings 9 (n)...., ............ g gM))..
consider the following minimax problem train the generator G and the discriminator D:
M
min 9
En log fcre+m (((""(())) I DG(D" + e (n)))
(6)
where
DG E arg max
Ex pmm
[log(e D(d
Eo D(g"(n))))
(7)
Here we assume that any document embedding d corpus Cm is sample with respect to the probability density Pm- Note that when M 1, the discriminator part of our model is equivalent to the original GAN model.
To explain (10), first we consider the discriminator D. Because there are multiple corpora of text documents, here we consider 2M categories as output of D, from which categories 1,..., M represent the original corpora C¹ ..... CMM and categories M + 1,... 2MM represent the generated document embeddings (e.g. bag-of- bag-of-words) from ".....,
neural network, outputs the distribution of text document being in each category. We maximize the
4

---

Under review as a conference paper at ICLR 2020
log- eegllikeihood of each document being in the correct category against D
M
pp ((d")) +
Σ E, D(0 m==
(8)
Such classifier does not only classifies text documents into different categories, but also considers M "fake" categories from the generators. When training the generators G ..... G we minimize the following which makes a comparison between the m-th and (M + m))th categories
M En log
Mm D(g (n))
MM+m ((G ((G(())
The intuition of (13) is that for each generated document embedding g (12, we need to decrease +m D(Gm((n) which is the probability of the generated embedding being correctly classified, and increase e Dig"(n)), which is the probability of the generated embedding being classified into the target corpus Cm. The ratio in (13) reflects these two properties.
We iteratively train (12) and (13) until the classification error of 'D becomes stable. Figure 2 Dillstrraes the deGAN model. The algorithm for deGAN summarized in Algorithm 2 in the appendix.
We next show that from (10), the distributions of the document embeddings from the optimal gl, 3.... Gd are equal to the data distributions of C1 .....y . C.M, which is generalization Goodeeloow ct al. (2014) to the multi-corpus scenario. The proof of Proposition is in the appendix.
Proposition 1. Let us assume that the random variables d ...., d' continuous with probability density .I:.... PM which have bounded support X: n is a continuous random variable with bounded support and activations aj and d2 are continuous; and that G :.... GA*t solutions (10). Then
qi Ihr. the probability density of the document embeddings from G¹ ..... GM, are equal to P......PM..
4 Experiments
In the experiments, consider four data sets, two of them newly created and the remaining two already public: CNN, TIME, 20 Newsgroups (in the appendix), and Reuters-21578 (in the appendix). The code and the two new data sets are available at tttthubbcom//deepeaamnn2200 200m:m/mnnnn For the pre-processing of all the documents, we transformed all characters to lower case, stemmed the documents, and ran the word2vee model on each corpora to obtain word embeddings with size of 300. In all subsequent models, we only consider the most frequent 5,000 words across all corpora in a data set.
The document embedding in weGAN is the tf-idf weighted word embedding transformed by the tanh activation, i.c. f(d,", Vmm) = tanh (E)= 4m by oT)) For deGAN, we use L normalized tf-idf as the
document embedding because it is easier to interpret than the transformed embedding in (20).
For weGAN, the cross-corpus word embeddings are initialized with the word2vec model trained from all documents. For training our models, we apply learning rate which increases linearly from 0.01 to 1.0 and train the models for 100 epochs with a batch size of 50 per corpus. The classifier C has single hidden layer with 50 hidden nodes, and the discriminator with a single hidden layer D has 10 hidden nodes. All these parameters have been optimized. For the labels k in (8), we apply corpus membership of each document.
For the noise distribution N for deGAN, we apply the uniform distribution U-,,,1) In (14) for deGAN, a1 = tanh and 0₂ = sof tmar so that the model outputs document embedding vectors which are comparable to I, nooomazal tf-idf vectors for each document. For the discriminator 'D of deGAN, we apply the word2vec embeddings based on all corpora to initialize its first layer, followed by another hidden layer of 50 nodes. For the discriminator D, we apply learning rate of 0. and for the generator G, we apply a learning rate of 0.001, because the initial training phase of deGAN can be unstable. We also apply batch size of 50 per corpus. For the softmax layers of deGAN, initialize them with the log of the word matrix in latent Dirichlet allocation (LDA) Blei et al. 2003) in order to provide intuitive estimates.
5

---

Under review as a conference paper at ICLR 2020
Obama w2v Bush Trump Kerry Abe Ne- tanyahu Rouhani Erdogan he Karzai Tillerson
Obama weGAN Trump Bush Kerry Abe Ne-
w2v-RI
weGAN-RI
tanyahu Erdogan Tillerson he Carter Rouhani Trump w2v Obama Pence Erdogan Bush
mean
67.88%
68.45 %
sd.
0.02%
0.01%
Duterte he Sanders Macron Christie Tillerson
w2v W2v-acuracyy
weGAN accuracy
Trump weGAN Obama Pence Bush Christie
mean
92.05%
92.36%
Sanders Clinton Erdogan Tillerson Macron
sd.
0.06%
0.03%
Duterte
U.S. w2v US Pentagon United Iranian
Table 1: A comparison between word2vec and weGAN in terms of the Rand index and the classification accuracy for the CNN data set.
NATO Turkish Qatar Iran British UAE U.S. weGAN US Pentagon United Iranian NATO Turkish Iran Qatar American UAE
Table Synonyms of "Obama," "Trump." and "U.S." before and after weGAN training for the CNN data set.
For weGAN, we consider two metrics for comparing the embeddings trained from WeGAN and those trained from all documents: (1) applying the document embeddings to cluster the documents into M clusters with the K-means algorithm, and calculating the Rand index (RI) (Rand 1971) against the original corpus membership: (2) finetuning the classifier and comparing the classification error against an FFNN of the same structure initialized with word2vec (w2v). For deGAN, compare the performance of finetuning the discriminator of deGAN for document classification, and the performance of the same FFNN. Each supervised model is trained for 500 epochs and the validation data set is used to choose the best epoch.
4.1 The CNN data set
In the CNN data set, we collected all news links on www.cnn.com in the GDELT 1.0 Event Database from April 1st, 2013 to July 7, 2017. We then collected the news articles from the links, and kept those belonging to the three largest categories: "politics," "world," and "US." We then divided these documents into 21,674 training documents, from which 2,708 validation documents are held out, and 5,420 testing documents.
We hypothesize that because weGAN takes into account document labels in a semi-supervised way, the embeddings trained from weGAN can better incorporate the labeling information and therefore, produce document embeddings which are better separated. The results are shown in Table [ and averaged over 5 randomized runs. Performing the Welch's t-test, both changes after weGAN training are statistically significant at 0.05 significance level. Because the Rand index captures matching accuracy, we observe from Table [ that weGAN tends to improve both metrics.
Meanwhile, we also wish to observe the spatial structure of the trained embeddings. which can be explored by the synonyms of each word measured by the cosine similarity. On average, the top 10 synonyms of each word differ by 0.22 word after weGAN training, and 20.7% of all words have different top synonyms after training. Therefore, weGAN tends to provide small adjustments rather than structural changes. Table lists the 10 most similar terms of three terms, "Obama," "Trump," " and "U.S."" " before and after weGAN training, ordered by cosine similarity.
We observe from Table that for "Obama," "Trump" and "Tillerson" are more similar after weGAN training, which means that the structure the weGAN embeddings can be more up-to-date. For "Trump." observe that "Clinton" is not among the synonyms before, but is after, which shows that the synonyms after are more relevant. For "U.S.," " we observe that after training, "American" replaces "British" in the list of synonyms, which is also more relevant.
We next discuss deGAN. Table we compare the performance of finetuning the discriminator of deGAN for document classification, and the performance of the FFNN initialized with word2vec. The change is also statistically significant at the 0.05 level. From Table improves the accuracy of supervised learning.
we observe that deGAN

---

Under review as a conference paper at ICLR 2020
politics original India US Carter defense Indian relationship China said military two politics deGAN year US meeting used read security along building worth foreign world original Turkey Turkish Attack ISIS said Kurdish Erdogan group bomb report world deGAN cut company get lot made code could Steve items may road block phone US original climate change year study according says country temperatures average US deGAN area efforts volunteers town weapons shot local nearly department also
W2v-accuracy
accurayyy
mean sd.
92.05%
92.29% %
0.06%
0.09%
Table 3: A comparison between word2vec and deGAN in terms of the accuracy for the CNN data set.
Table 4: Bag-of-words representations of orig- inal and artificial text in the CNN data set.
To compare the generated samples from deGAN with the original ffg-of-words, we randomly select one record in each original and artificial corpus. The records are represented by the most frequent words sorted by frequency in descending order where the stop words are removed. The bag-of-worde embeddings are shown in Table
From T we observe that the bag-of- bag-of-words words embeddings of the original documents tend to contain more name entities, while those the artificial deGAN documents tend be more general. There are many additional examples not shown here with observed artificial rag-of-words embeddings having
antition much "Treee SISTe
from 15 Anamrmantt
"Cuvin monntelll.
iist...
5
-10
-10
15
-30
Figure 3: 2-d representation of original (red) and artificial (blue) examples in the CNN data set.
Figure 4: 2-d representation of original (red) and artificial (blue) examples in the TIME data set.
We also perform dimensional reduction using t-SNE (yan der Maaten & Hinton 2008), and plot 100 random samples from each original or artificial category. The original samples shown in red and the generated ones are shown in blue in Figure 3. We do not further distinguish the categories because there is clear distinction between the three original corpora, "politics," "world," and "US."
We observe that the original and artificial examples are generally mixed together and not well separable, which means that the artificial examples similar to the original ones. However, we also observe that the artificial samples tend to be more centered and have no outliers (represented by the outermost red oval).
4.2
The TIME data set
In the TIME data set, we collected all news links on time.com in the GDELT 1.0 Event Database from April 1st, 2013 July 7, 2017. We then collected the articles from the links, and kept.
7

---

Charleston many Under review as a conference paper at ICLR 2020
those belonging the five largest categories: "Entertainment," "Ideas,, "Plitts"" "Politics," "US," and "World." We divided these documents into 12,286 training documents, from which 1,535 validation documents held out, and 3,075 testing documents.
Table 5 compares the clustering results word2vee and weGAN, and the classification accuracy of an FFNN initialized with word2vec, finetuned weGAN, and finetuned deGAN. The results in Table are the counterparts Table and Table 3 for the TIME data set. The differences are also significant the 0.05 level.
Obama w2v
Trump Bush Xi Erdogan Rouhani Reagan Hollande Duterte Abe Jokowi
Obama weGAN
Trump Bush Xi Erdogan Reagan Rouhani Hollande Abe Jokowi Duterte w2v Obama Erdogan Rubio Duter- te Bush Putin Sanders Xi Macron Pence
w2v-RI
wcGAN-RI
mean
70.96%
71.14%
Trump
sd.
0.02%
0.02%
w2v-accur.
OGGAN-accur
deGAN-accur.
Trump weGAN Obama Erdogan Rubio Bush Sanders Putin Duterte Xi Macron Pence U.S. w2v NATO Iran Japan Pentagon
83.79%
84.76%
85.38%
0.17%
0.08%
0.11%
Russia Pakistan Tehran EU Ukrainian Moscow U.S. weGAN NATO Pentagon Iran Japan
Table 5: A comparison between word2vec, weGAN, and deGAN in terms of the Rand index and the classification accuracy for the TIME data set.
Russia Tehran Pakistan EU Ukrainian Moscow
Table 6: Synonyms of "Obama," "Trump," " and "U.S." before and after weGAN training for the TIME data set.
we observe that both GAN models yield improved performance of supervised learning. For weGAN, on an average, the top 10 synonyms of each word differ by 0.27 word after weGAN training, and 24.8% of all words have different top 10 synonyms after training. We also compare the synonyms of the same common words, "Obama," "Trump." and "U.S.," which are listed Table 6 In the TIME data set, for "Obama," "Reagan" is ranked slightly higher an American president. For "Trump," "Bush" and "Sanders" are ranked higher as American presidents or candidates. For "U.S.," we note that "Pentagon" is ranked higher after weGAN training, which we think also reasonable because the term is closely related to the U.S. government.
From Table 5
For deGAN, we also compare the original and artificial samples in terms of the highest probability words, which is shown in Table 7 We also perform dimensional reduction using t-SNE for 100 examples per corpus and plot them in Figure All these figures and tables show results similar to Section 4.1.
Entertainment original show London attack people proud open going right according way home music actor Michael John song going meeting James produced pop vocal American would service national young country year serve security world city project part development grand bear often west new status high agents Politics original Assange embassy BBC Swedish charges told authorities officials Sweden
Politics deGAN
members present national committee party Paul sign Trump removed brief
Entertainment deGAN
US
original
Carolina South funeral hand wrote political words Davis head board man relationship recent Sunday stone fire wrote
Ideas original
US
deGAN
gay well Erdogan Turkey
Ideas deGAN
world original
political power government two leaders minister AKP
world deGAN suffering like know old violence local daily young interest three first man
Table 7: Bag-of-words representations of original and artificial text in the TIME data set.
8

---

Under review as a conference paper at ICLR 2020
References
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. Technical report, 2017.
David M. Blei, Andrew Y. Ng. and Michael I. Jordan. Latent dirichlet allocation, Journal of Machine Learning Research, 3:993-1022, 2003.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493-2537, 2011.
John Glover. Modeling documents with generative adversarial networks. In Workshop on Adversarial Training (NIPS 201 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27. 2014.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), 2014.
Quoc V. and Tomas Mikolov. Distributed representations of sentences and documents. In Proceedings of the 31st International Conference Machine Learning (ICML 2014), 2014.
Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural
dialogue generation. Technical report, 2017.
Ming- Yu Liu and Oncel Tuzel. Coupled generative adversarial networks. In Advances in Neural Information Processing Systems 29 (NIPS 2016), 2016.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. In Workshop (ICLR 2013), 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26 (NIPS 2013), Lake Tahoe, Nevada, USA, 2013b.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. Technical report, 2014.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, 2014.
William M. Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66:846-850 1971.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems 29 (NIPS 2016). 2016.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013), 2013.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 2008
Yizhe Zhang, Zhe Gan, and Lawrence Carin. Generating text via adversarial training. Workshop Adversarial Training (NIPS 2016), 2016.

---

Under review as a conference paper at ICLR 2020
Junbo Jake Zhao, Michaël Mathieu, and Yann LeCun. Energy-based generative adversarial networks.
In Sth International Conference on Learning embeddings (ICLR 2017), 2017.
A Algorithms for weGAN and deGAN
Algorithm (for weGAN).
1. Train g based on f from all corpora .l....,ch
2. Randomly initialize the weights and biases of the classifier and discriminator D,
3. Until maximum number of iterations reached
(a) Update C and 'D according to (4) and (6) given a mini-batch Si of training examples {d.t Jimm
(b) Update G according to (5) given mini-batch S₂ of training examples {d.")
4. Output 9 as the cross-corpus word embeddings.
Algorithm 2 (for deGAN).
1. Randomly initialize the weights of g'.....
2. Initialize the discriminator D with the weights of the first layer (which takes document embeddings as the input) initialized by word embeddings, and other parameters randomly initialized.
3. Until maximum number of iterations reached
(a) Update D according to (12) given a mini-batch of training examples di" and samples from noise n.
(b) Update g "'....,g* .M. according to (13) given a mini-batch training examples d!" and samples form noise
4. Output gI...., GM as generators of document embeddings and 'D corpus classifier.
B Proof of Proposition
Since X' is bounded, all of the integrals exhibited next are well-defined and finite, Since n, d2, and ag continuous, it follows that for any parameters, GTD (n) is continuous random variable with probability density Ym with finite support. From (11),
M Σ
D = arg max D
Pm (x) log(e
+
Om (x) log(e D(x))dt
Σ Pm (x) D(
arg max
M
Um(o) log(e ((x)))ds
(10)
m=1
This problem reduces
max b by.....om
am log bm subject bm
(11)
the solution of which is bn, 0m/ DM am, I,..., M. Therefore, the solution to (15) is (pr(x), ..... PM (1)) (.)....... ... (MM())
(x)
(12)
ΣM 1m=1 (m) + EM 4m (x)
10

---

Under review as a conference paper at ICLR 2020
We then obtain from (10) that
M
/
4m (x)
:...... Iho E arg min
Σ m==
(m(z)-1u log
4m (x) + pm(())
M
9m (m(
arg
min qM
M log 2 +
gm (x) log
dr
(9m (x) Pm (z))
m=1
arg
min Pa
M log
Σ
KL 11 Pm)
m==1
From non-negativity of the Kullback-Leibler divergence, we conclude that
(oi.....9.)) - (PI....,PM) PM.)
C The 20 Newsgroups data set
The 20 Newsgroups data set is collection of news documents with 20 categories. To reduce the number of categories that the GAN models are more compact and have more samples per corpus, we grouped the documents into 6 super-categories: "religion." "computer," "cars," "sport," "science," and "politics" ("mise" ignored because of its noisiness). We considered each super-category as a different corpora. We then divided these documents into 10,708 training documents, from which 1,335 validation documents are held out, and 7,134 testing documents. We train weGAN and deGAN in the the beginning of Section 4, except that we use learning rate of 0.01 for the discriminator in deGAN to stabilize the cost function. Table S compares the clustering results of word2vec and weGAN, and the classification accuracy of the FFNN initialized with word2vec, finetuned weGAN, and finetuned deGAN. All comparisons are statistically significant at the 0.05 level. The other results similar to the previous two data sets and are thereby omitted here.
w2v-RI
weGAN-RI RI
mean
76.14%
76.74%
sd.
0.02%
0.08%
wav-accur.
weGAN-accuu
deGAN-accur.
87.34%
89.90 %
89.32%
0.04%
0.02%
0.15%
Table A comparison between word2vee, weGAN, and deGAN terms of the Rand index and the classification accuracy for the 20 Newsgroups data set.
D
The Reuters-21577 data set
The Reuters-21578 data set is a collection of newswire articles. Because the data set is highly skewed, we considered the eight categories with more than 100 training documents: "earn," "acq," "crude." " "trade," "money-xx; "interest," "money-supply." and "ship." We then divided these documents into 5,497 training documents, from which 692 validation documents are held out, and 2,207 testing documents. We train weGAN and deGAN in the same way in the 20 Newsgroups data set. Table 9 compares the clustering results of word2vec and weGAN, and the classification accuracy of the FFNN initialized with word2vec, finetuned weGAN, and finetuned deGAN. All comparisons statistically significant at the 0.05 level except the Rand index. The other results are similar to the CNN and TIME data sets and thereby omitted here.

---

Under review as a conference paper at ICLR 2020
w2v-RI
weGAN-RI
mean
71.28%
71.43%
sd.
0,26%
0.07%
w2v-accur:
weGAN-accur.
deGANNaccurr
92.86%
95.10% %
94.86%
0.09%
0.10%
0.10%
Table 9: A comparison between word2vec, weGAN, and deGAN terms of the Rand index and the classification accuracy for the Reuters-21578 data set.
12


Reviews:
[{"review_id": "S1eUbIW3YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes two models: (1) A model to learn word embeddings from different corpus. (2) A model to generate robust bag-of-words document embeddings. This topic may not be novel enough, considering the current development in GAN and domain adaptation. Some questions:\n-\tFor the word embedding model, why it is called GAN? It seems that nothing is generated. \n-\tFurther, for the word embedding model, the notation is over-complicated. This model is very similar to the domain adversarial network (DANN). This is my understanding and please correct me if I am inaccurate: A classifier is trained to distinguish the domains/corpus, while the word encoder tries to learn shared features across domains/corpus. The sub- and super-scripts are confusing. \n-\tFor the deGAN, only generating bag-of-words may not be that attracting. Further, the idea is also similar to many existing works in domain adaptation. Each domain/corpus has unique components, while all domains/corpus also share similar features. The final output is just a combination of the shared components and the unique features. This idea is straightforward. It would be much better if the author can generate natural language, instead of just bag-of-words. \n-\tPlease fix the formatting issues. For instance, Figure 3 and Figure 4 cover the text. ‘To explain (10), first we consider…’. I think here should be (6).\n-\tMore baseline competitors are preferred. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1gUv3cg5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper proposes extensions of Generative Adversarial Networks to modeling multiple text corpora. Concretely, the paper looks at two problems: 1) given independently pretrained word embeddings from K corpora, finding a common word embdding, 2) extracting document representations from a discriminator of a GAN trained to generate tf-idf vectors. Preliminary experiments show that the proposed approaches outperform baseline classifiers trained with word2vec.\n\n\nStrengths\n+ The paper clearly mentions all the experimental details\n+ The paper has a nice set of qualitative examples that probe what the proposed model is learning\n\n\nWeaknesses\n\n* It is not clear what problem the paper is trying to solve. Is the goal to get better document representations, in which case why do we think that using a GAN is a good idea. Learning a generative model to use representations for a downstream task seems like a pretty roundabout way of doing things (if we dont care about generating anything in the first place). Further, if better document representations are desired then the paper should compare to works like Bert (Devlin et.al. [1]).\n\n* Sec. 3.1: Calling the proposed weGAN model a GAN seems a bit inconsistent/ wrong. The proposed model is not a generative adversarial network, there is no sampling of a noise or a notion of a generative distribution. The real data, similar to the image case is a bunch of samples from the data distribution (where the stochasticity comes not from the word embeddings but from the tf-idf of thte document), but the generator also uses the tf-idf representation, so essentially both the generator as well as the real world data use the document as an input. Hence it feels like a somewhat odd model formulation, which would be nice to clarify / explain.\n\n* Further, it is not clear why one would want to train “cross-corpus” embeddings in the way describeed in Sec. 3. 1. Why not just train word2vec on the union of all the corpora (instead of per-corpus) and use it as the word representation? What are the conditions in which one would not want to do the common word representation?\n\n* Sec. 3.2: Why not formulate modeling multiple corpora in the spirit of CoGAN (Liu and Tuzel, 2016), by getting M discriminators (potentially with parameter sharing) to solve binary classification tasks as opposed to the 2M way classification problem? Atleast a comparison to an approach like this seems warranted. Further, it is not clear why one would want to generate tf-idf vectors. It seems like all the experiments are around representation learning and classification as opposed to generating or evaluating tf-idf vectors for documents. \n\n* As a side note, for the experiments concerning deGAN vs the baseline, it would be nice to check what happens when the last classification layer is not initialized with LDA parameters. Since that is an additional source of information. \n\nReferences\n[1]: Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” arXiv [cs.CL]. arXiv. http://arxiv.org/abs/1810.04805.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "H1eZpcn45H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use Generative Adversarial Networks (GANs) in the context of natural language processing and introduces two models for generating document embeddings. The first model, weGAN, aggregates multiple sets of single-corpus word2vec embeddings into one set of cross-corpus word representations; document embeddings are a function of these updated word embeddings. The second model, deGAN, side-steps word-level embeddings and directly generates document-level representations. For both models, the real examples come from word2vec and tf-idf, while the artificial examples are the ones generated by the network. The authors show that their document embeddings are better than the word2vec/tf-idf baseline at clustering documents according to the corpus they originate from.\n\nWhile the context in which GANs are used is novel and creative, the reasons for rejection outweigh the positives. The main reason is the fact that the paper is outdated by two/three years. There is no mention of the shift towards contextual embeddings that has been happening in the last few years (InferSent, ELMo, BERT) and that became ubiquitous in NLP.  Instead, all comparisons are against word2vec (published in 2013) that is no longer a relevant baseline, even for non-contextual word embeddings. In fact, the newest cited papers are from 2017. The Github link in the paper was last updated on 23rd December 2017. My objection is not that this work was started a long time ago, but that it has not been updated in years. In the lack of proper comparison against more novel techniques, it is hard to estimate whether the reported gains over word2vec are meaningful.\n\nAnother argument for rejection is the fact that the experimental section does not make a convincing case that the positive results matter beyond the very specific datasets and tasks selected.\n\nBefore commenting in more detail on the selection of datasets, tasks and metrics, I want to document my understanding of the experimental setup; I found it non-trivial to understand what constitutes a \"single corpus\" in the experiments. My mental model is the following: the paper operates on four datasets (CNN, TIME, 20 Newsgroups, and Reuters-21578). These are four different experiments, and the four datasets never interact. The cross-corpus property of the model is exercised within each of these datasets, which have a natural grouping of topics; for instance, CNN is divided into \"US\", \"world\", and \"politics\". In other words, a cross-corpus CNN model is trained on three corpora: US, world, and politics. With this mental model of the experimental setup, the experiments raise a series of questions:\n\n1) Do the selected datasets / classification tasks matter in the grad scheme of NLP?\nThe end-task is to classify the corpus of origin for each document (for instance, for the CNN experiment, the model has to decide between US, world, and politics). If I understand correctly, the weGAN model jointly trained its word embeddings with this objective. As a final step, the classifier was additionally fine-tuned on this task. So it is not very surprising that weGAN is better at clustering documents than word2vec, since it was explicitly trained to do so. It would have been more informative to see weGAN performing better on a task that was solely fine-tuned on (e.g. some sentiment classification task that was not part of the training objective when embeddings were trained). In that case, one could argue that the enhanced embeddings can be used as an out-of-the box tool for a lot of different tasks. Another option would be evaluating on the same document membership task, but on datasets that were not seen during training.\n\nAdditionally, the fact that the authors evaluated on *new* datasets with a *new* task distances them even further from any potential comparisons against prior work. Even for the two already-existing datasets with results included in the appendix, there are no references to previous SoTA.\n\n2) Are the reported positive results meaningful?\nAnother problem with not showing results from previous work is that it is hard to understand whether the gains over the word2vec baseline are meaningful. For instance, the classification accuracy for the CNN corpus is increased from 92.05% (word2vec) to 92.29% (deGAN) and 92.36% (weGAN). The authors show that this is *statistically significant*, but is it meaningful? Is a 0.24% or 0.31% increase in accuracy worth the increased complexity of the model? For reference, what ratio of gains/computational cost would BERT incur?\n\n3) What is the message of qualitative evaluation?\nThe authors offer qualitative analysis of their model. For instance, Table 2 shows the top 10 synonyms for words \"Obama\", \"Trump\" and \"US\", as produced by word2vec and weGAN. The lists of synonyms are extremely similar and don't seem to convey much more information besides the fact that weGAN makes minimal changes. I find the particular examples that the authors choose to underline the superiority of their algorithm unconvincing. For instance, the fact that, in the list of synonyms for US, the word British (selected by word2vec as the 9th closest word) is turned into American (on the same 9th position) loses its significance when you also consider the fact that weGAN did not derank any of the arguably less similar terms from higher positions (e.g. Turkish remains the 6th most similar term to US).\n\nOther nit-picks:\n- Formatting issues: In Sections 3.1 and 3.2, the numerical references to equations are broken. Figures 3 and 4 cover the text above. Tables 2, 4, 6, 7 are hard to read. It almost feels like the paper was updated from one conference format to another, without the necessary Latex adjustments.\n- The paper has no \"Conclusion\" section\n- Improper use of singular / plural of \"corpus\" / \"corpora\"\n\nTo end on a more positive note, here are some reasons why I did appreciate the paper:\n- The usage of GANs in this context is very creative; the models do not generate *raw* text, but rather intermediate representations; this might be a promising direction for using GANs in NLP.\n- The description of the models in sections 3.1 and 3.2 is rigorous, with useful explanations that break down the reasoning behind the loss function choices.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The general consensus amongst the reviewers is that this paper is not quite ready for publication. The reviewers raised several issues with your paper, which I hope will help you as you work towards finding a home for this work.
Decision: Reject