Title: GENERATIVE ADVERSARIAL NETS FOR MULTIPLE TEXT CORPORA

Abstract: Generative adversarial nets (GANs) have been successfully applied to the artificial generation of image data. In terms of text data, much has been done on the artificial generation of natural language from a single corpus. We consider multiple text corpora as the input data, for which there can be two applications of GANs: (1) the creation of consistent cross-corpus word embeddings given different word embeddings per corpus; (2) the generation of robust bag-of-words document embeddings for each corpora. We demonstrate our GAN models on real-world text data sets from different corpora, and show that embeddings from both models lead to improvements in supervised learning problems.

1. Introduction:

Generative adversarial nets (GAN) (Goodfellow et al., 2014) belong to a class of generative models which are trainable and can generate artificial data examples similar to the existing ones. In a GAN model, there are two sub-models simultaneously trained: a generative model G from which artificial data examples can be sampled, and a discriminative model D which classifies real data examples and artificial ones from G. By training G to maximize its generation power, and training D to minimize the generation power of G, so that ideally there will be no difference between the true and artificial examples, a minimax problem can be established. The GAN model has been shown to closely replicate a number of image data sets, such as MNIST, Toronto Face Database (TFD), CIFAR-10, SVHN, and ImageNet (Goodfellow et al., 2014; Salimans et al., 2016) .

The GAN model has been extended to text data in a number of ways. For instance, Zhang et al. (2016) applied a long-short term memory (LSTM) (Hochreiter & Schmidhuber, 1997) generator and approximated discretization to generate text data. Moreover, Li et al. (2017) applied the GAN model to generate dialogues, i.e. pairs of questions and answers. Meanwhile, the GAN model can also be applied to generate bag-of-words embeddings of text data, which focus more on key terms in a text document rather than the original document itself. Glover (2016) provided such a model with the energy-based GAN (Zhao et al., 2017) .

To the best of our knowledge, there has been no literature on applying the GAN model to multiple corpora of text data. Multi-class GANs have been proposed (Liu & Tuzel, 2016; Mirza & Osindero, 2014) , but a class in multi-class classification is not the same as multiple corpora. Because knowing the underlying corpus membership of each text document can provide better information on how the text documents are organized, and documents from the same corpus are expected to share similar topics or key words, considering the membership information can benefit the training of a text model from a supervised perspective. We consider two problems associated with training multi-corpus text data: (1) Given a separate set of word Mikolov et al. (2013b) , how to obtain a better set of cross-corpus word embeddings from them? (2) How to incorporate the generation of document embeddings from different corpora in a single GAN model?

For the first problem, we train a GAN model which discriminates documents represented by different word embeddings, and train the cross-corpus word embedding so that it is similar to each existing word embedding per corpus. For the second problem, we train a GAN model which considers both crosscorpus and per-corpus "topics" in the generator, and applies a discriminator which considers each original and artificial document corpus. We also show that with sufficient training, the distribution of the artificial document embeddings is equivalent to the original ones. Our work has the following contributions: (1) we extend GANs to multiple corpora of text data, (2) we provide applications of GANs to finetune word embeddings and to create robust document embeddings, and (3) we establish theoretical convergence results of the multi-class GAN model. Section 2 reviews existing GAN models related to this paper. Section 3 describes the GAN models on training cross-corpus word embeddings and generating document embeddings for each corpora, and explains the associated algorithms. Section 4 presents the results of the two models on text data sets, and transfers them to supervised learning. Section 5 summarizes the results and concludes the paper.

2. Literature Review:

In a GAN model, we assume that the data examples x are drawn from a distribution p x (•), and the artificial data examples G(z) := G(z, θ g ) are transformed from the noise distribution z ∼ p z (•). The binary classifier D(•) outputs the probability of a data example (or an artificial one) being an original one. Because the probabilistic structure of a GAN can be unstable to train, the Wasserstein GAN (Arjovsky et al., 2017) is proposed which applies a 1-Lipschitz function as a discriminator.

We note that in many circumstances, data sets are obtained with supervised labels or categories, which can add explanatory power to unsupervised models such as the GAN. For instance, the CoGAN (Liu & Tuzel, 2016) considers pairs of data examples from different categories, and the weights of the first few layers (i.e. close to z) are tied. Mirza & Osindero (2014) proposed the conditional GAN where the generator G and the discriminator D depend on the class label y. Salimans et al. (2016) applied the class labels for semi-supervised learning with an additional artificial class. However, all these models consider only images and do not produce word or document embeddings, therefore being different from our models.

For generating real text, Zhang et al. (2016) proposed textGAN in which the generator has an LSTM form, and a uni-dimensional convolutional neural network (Collobert et al., 2011; Kim, 2014) is applied as the discriminator. Also, a weighted softmax function is applied to make the argmax function differentiable. The focus of our work is to summarize information from longer documents, so we apply document embeddings such as the tf-idf to represent the documents rather than to generate real text.

For generating bag-of-words embeddings of text, Glover (2016) proposed a GAN model with the mean squared error of a de-noising autoencoder as the discriminator, and the output x is the one-hot word embedding of a document. Our models are different from this model because we consider tf-idf document embeddings for multiple text corpora in the deGAN model (Section 3.2), and weGAN (Section 3.1) can be applied to produce word embeddings. Also, we focus on robustness based on several corpora, while Glover (2016) assumed a single corpus.

For extracting word embeddings given text data, Mikolov et al. (2013a) proposed the word2vec model, for which there are two variations: the continuous bag-of-words (cBoW) model (Mikolov et al., 2013a) , where the neighboring words are used to predict the appearance of each word; the skip-gram model, where each neighboring word is used individually for prediction. In GloVe (Pennington et al., 2014) , a bilinear regression model is trained on the log of the word co-occurrence matrix. In these models, the weights associated with each word are used as the embedding. For obtaining document embeddings, the para2vec model (Le & Mikolov, 2014) adds per-paragraph vectors to train word2vec-type models, so that the vectors can be used as embeddings for each paragraph. A simpler approach by taking the average of the embeddings of each word in a document and output the document embedding is exhibited in Socher et al. (2013) .

3. Models and Algorithms:

Suppose we have a number of different corpora C 1 , . . . , C M , which for example can be based on different categories or sentiments of text documents. We suppose that C m = {d m 1 , . . . , d m nm }, m = 1, . . . , M , where each d m i represents a document. The words in all corpora are collected in a dictionary, and indexed from 1 to V . We name the GAN model to train cross-corpus word embeddings as "weGAN," where "we" stands for "word embeddings," and the GAN model to generate document embeddings for multiple corpora as "deGAN," where "de" stands for "document embeddings."

3.1. weGAN: Training cross-corpus word embeddings:

We assume that for each corpora C m , we are given word embeddings for each word v m 1 , . . . , v m V ∈ R d , where d is the dimension of each word embedding. We are also given a classification task on documents that is represented by a parametric model C taking document embeddings as feature vectors. We construct a GAN model which combines different sets of word embeddings

V m := {v m i } V i=1 , m = 1, . . . , M , into a single set of word embeddings G := {v 0 i } V i=1 . Note that V 1 , . . . , V M are given but G is trained.

Here we consider G as the generator, and the goal of the discriminator is to distinguish documents represented by the original embeddings V 1 , . . . , V M and the same documents represented by the new embeddings G.

Next we describe how the documents are represented by a set of embeddings V 1 , . . . , V M and G. For each document d m i , we define its document embedding with V m as g m i := f (d m i , V m ), where f (•) can be any mapping. Similarly, we define the document embedding of d m i with G as follows, with

G = {v 0 j } V j=1 trainable f G (d m i ) := f (d m i , G).

In a typical example, word embeddings would be based on word2vec or GLoVe. Function f can be based on tf-idf, i.e. f (

d m i , V) = V j=1 t m ij v m j

where v m j is the word embedding of the j-th word in the m-th corpus C m and t

m i = (t m i1 , . . . , t m iV ) is the tf-idf representation of the i-th document d m i in the m-th corpus C m .

To train the GAN model, we consider the following minimax problem

EQUATION

where D is a discriminator of whether a document is original or artificial. Here k m i is the label of document d m i with respect to classifier C, and e k m i is a unit vector with only the k m i -th component being one and all other components being zeros. Note that log(e

T k m i C(f G (d m i ))) is equivalent to KL(e k m i C(f G (d m i ))

), but we use the former notation due to its brevity. The intuition of problem (3) is explained as follows. First we consider a discriminator D which is a feedforward neural network (FFNN) with binary outcomes, and classifies the document embeddings

EQUATION

For the generator G, we wish to minimize (3) against G so that we can apply the minimax strategy, and the combined word embeddings G would resemble each set of word embeddings V 1 , . . . , V M . Meanwhile, we also consider classifier C with K outcomes, and associates d m i with label k m i , so that the generator G can learn from the document labeling in a semi-supervised way.

If the classifier C outputs a K-dimensional softmax probability vector, we minimize the following against G, which corresponds to (3) given D and C:

EQUATION

For the classifier C, we also minimize its negative log-likelihood

EQUATION

Assembling (4-6) together, we retrieve the original minimax problem (3).

We train the discriminator and the classifier, {D, C}, and the combined embeddings G according to (4-6) iteratively for a fixed number of epochs with the stochastic gradient descent algorithm, until the discrimination and classification errors become stable.

Figure 1 illustrates the weGAN model. The algorithm for weGAN is summarized in Algorithm 1 in the appendix. 

𝒞 1 𝒞 2 ⋯ 𝒞 𝑀 𝑑 1 1 ⋯ 𝑑 𝑛1 1 𝑑 1 2 ⋯ 𝑑 𝑛2 2 ⋯ 𝑑 1 𝑀 ⋯ 𝑑 𝑛𝑀 𝑀 𝑔 1 1 , … , 𝑔 𝑛1 1 corpora documents 𝒱 1 word embeddings document embeddings 𝒟 discriminator 𝐹𝑟𝑜𝑚 {𝒱 1 , … , 𝒱 𝑀 } 𝑜𝑟 𝒢 𝒞 classifier output 𝒱 2 𝑔 1 2 , … , 𝑔 𝑛2 2 𝒱 𝑀 𝑔 1 𝑀 , … , 𝑔 𝑛𝑀 𝑀 𝑡 𝑛𝑀 𝑀 {𝑑 1 1 , … , 𝑑 𝑛𝑀 𝑀 } 𝒢 {𝑓 𝒢 𝑑 1 1 , … , 𝑓 𝒢 (𝑑 𝑛𝑀 𝑀 )} Label distribution Figure 1: Model structure of weGAN. 𝒞 1 𝒞 2 ⋯ 𝒞 𝑀 𝑑 1 1 ⋯ 𝑑 𝑛1 1 𝑑 1 2 ⋯ 𝑑 𝑛2 2 ⋯ 𝑑 1 𝑀 ⋯ 𝑑 𝑛𝑀 𝑀 corpora document embeddings 𝒟 discriminator 𝐹𝑟𝑜𝑚 𝒞 1 , 𝒞 2 , … , 𝒞 𝑀 𝑜𝑟 {𝒢 1 , … , 𝒢 𝑀 } output 𝒢 1 (𝑛) generators 𝑊 ℎ 0 , 𝑊 𝑜 0 𝒢 1 𝒢 𝑀 𝒢 2 noise 𝑛 𝒢 2 (𝑛) 𝒢 𝑀 (𝑛) generated embeddings 𝑊 ℎ 1 , 𝑊 𝑜 1 𝑊 ℎ 2 , 𝑊 𝑜 2 𝑊 ℎ 𝑀 ,

3.2. deGAN: Generating document embeddings for multi-corpus text data:

In this section, our goal is to generate document embeddings which would resemble real document embeddings in each corpus C m , m = 1, . . . , M . We construct M generators, G 1 , . . . , G M so that G m generate artificial examples in corpus C m . As in Section 3.1, there is a certain document embedding such as tf-idf, bag-of-words, or para2vec. Let G = {G 1 , . . . , G M }. We initialize a noise vector n = (n 1 , . . . , n dn ) ∈ R dn , where n 1 , . . . , n dn iid ∼ N , and N is any noise distribution.

For a generator G m = {W m h , W 0 h , W m o , W 0 o } represented by its parameters, we first map the noise vector n to the hidden layer, which represents different topics. We consider two hidden vectors, h 0 for general topics and h m for specific topics per corpus,

h m = a 1 (W m h n), h 0 = a 1 (W 0 h n).

Here a 1 (•) represents a nonlinear activation function. In this model, the bias term can be ignored in order to prevent the "mode collapse" problem of the generator. Having the hidden vectors, we then map them to the generated document embedding with another activation function a 2 (•),

EQUATION

To summarize, we may represent the process from noise to the document embedding as

G m (n) = a 2 (W m o a 1 (W m h n) + w 0 o a 1 (W 0 h n)).

Given the generated document embeddings G 1 (n), . . . , G M (n), we consider the following minimax problem to train the generator G and the discriminator D:

EQUATION

where

EQUATION

Here we assume that any document embedding d m in corpus C m is a sample with respect to the probability density p m . Note that when M = 1, the discriminator part of our model is equivalent to the original GAN model.

To explain (10), first we consider the discriminator D. Because there are multiple corpora of text documents, here we consider 2M categories as output of D, from which categories 1, . . . , M represent the original corpora C 1 , . . . , C M , and categories M + 1, . . . , 2M represent the generated document embeddings (e.g. bag-of-words) from G 1 , . . . , G M . Assume the discriminator D, a feedforward neural network, outputs the distribution of a text document being in each category. We maximize the log-likelihood of each document being in the correct category against

EQUATION

Such a classifier does not only classifies text documents into different categories, but also considers M "fake" categories from the generators. When training the generators G 1 , . . . , G M , we minimize the following which makes a comparison between the m-th and (M + m)-th categories

EQUATION

)

The intuition of ( 13) is that for each generated document embedding G m (n), we need to decrease e T M +m D(G m (n)), which is the probability of the generated embedding being correctly classified, and increase e T m D(G m (n)), which is the probability of the generated embedding being classified into the target corpus C m . The ratio in (13) reflects these two properties.

We iteratively train ( 12) and ( 13) until the classification error of D becomes stable. Figure 2 illustrates the deGAN model. The algorithm for deGAN is summarized in Algorithm 2 in the appendix.

We next show that from (10), the distributions of the document embeddings from the optimal G 1 , . . . , G M are equal to the data distributions of C 1 , . . . , C M , which is a generalization of Goodfellow et al. (2014) to the multi-corpus scenario. The proof of Proposition 1 is in the appendix.

Proposition 1. Let us assume that the random variables d 1 , . . . , d M are continuous with probability density p 1 , . . . , p M which have bounded support X ; n is a continuous random variable with bounded support and activations a 1 and a 2 are continuous; and that G 1 * , . . . , G M * are solutions to (10). Then q * 1 , . . . , q * M , the probability density of the document embeddings from G 1 * , . . ., G M * , are equal to p 1 , . . . , p M .

4. Experiments:

In the experiments, we consider four data sets, two of them newly created and the remaining two already public: CNN, TIME, 20 Newsgroups (in the appendix), and Reuters-21578 (in the appendix). The code and the two new data sets are available at github.com/deeplearning2018/emgan. For the pre-processing of all the documents, we transformed all characters to lower case, stemmed the documents, and ran the word2vec model on each corpora to obtain word embeddings with a size of 300. In all subsequent models, we only consider the most frequent 5,000 words across all corpora in a data set.

The document embedding in weGAN is the tf-idf weighted word embedding transformed by the tanh activation, i.e. f (

d m i , V m ) = tanh V j=1 t m ij v m j .

For deGAN, we use L 1 -normalized tf-idf as the document embedding because it is easier to interpret than the transformed embedding in (20).

For weGAN, the cross-corpus word embeddings are initialized with the word2vec model trained from all documents. For training our models, we apply a learning rate which increases linearly from 0.01 to 1.0 and train the models for 100 epochs with a batch size of 50 per corpus. The classifier C has a single hidden layer with 50 hidden nodes, and the discriminator with a single hidden layer D has 10 hidden nodes. All these parameters have been optimized. For the labels k m i in (8), we apply corpus membership of each document.

For the noise distribution N for deGAN, we apply the uniform distribution U (-1, 1). In ( 14) for deGAN, a 1 = tanh and a 2 = sof tmax so that the model outputs document embedding vectors which are comparable to L 1 -normalized tf-idf vectors for each document. For the discriminator D of deGAN, we apply the word2vec embeddings based on all corpora to initialize its first layer, followed by another hidden layer of 50 nodes. For the discriminator D, we apply a learning rate of 0.1, and for the generator G, we apply a learning rate of 0.001, because the initial training phase of deGAN can be unstable. We also apply a batch size of 50 per corpus. For the softmax layers of deGAN, we initialize them with the log of the topic-word matrix in latent Dirichlet allocation (LDA) (Blei et al., 2003) For weGAN, we consider two metrics for comparing the embeddings trained from weGAN and those trained from all documents: (1) applying the document embeddings to cluster the documents into M clusters with the K-means algorithm, and calculating the Rand index (RI) (Rand, 1971) against the original corpus membership;

(2) finetuning the classifier C and comparing the classification error against an FFNN of the same structure initialized with word2vec (w2v). For deGAN, we compare the performance of finetuning the discriminator of deGAN for document classification, and the performance of the same FFNN. Each supervised model is trained for 500 epochs and the validation data set is used to choose the best epoch.

4.1. The CNN data set:

In the CNN data set, we collected all news links on www.cnn.com in the GDELT 1.0 Event Database from April 1st, 2013 to July 7, 2017. We then collected the news articles from the links, and kept those belonging to the three largest categories: "politics," "world," and "US." We then divided these documents into 21,674 training documents, from which 2,708 validation documents are held out, and 5,420 testing documents.

We hypothesize that because weGAN takes into account document labels in a semi-supervised way, the embeddings trained from weGAN can better incorporate the labeling information and therefore, produce document embeddings which are better separated. The results are shown in Table 1 and averaged over 5 randomized runs. Performing the Welch's t-test, both changes after weGAN training are statistically significant at a 0.05 significance level. Because the Rand index captures matching accuracy, we observe from Table 1 that weGAN tends to improve both metrics.

Meanwhile, we also wish to observe the spatial structure of the trained embeddings, which can be explored by the synonyms of each word measured by the cosine similarity. On average, the top 10 synonyms of each word differ by 0.22 word after weGAN training, and 20.7% of all words have different top 10 synonyms after training. Therefore, weGAN tends to provide small adjustments rather than structural changes. Table 2 lists the 10 most similar terms of three terms, "Obama," "Trump," and "U.S.," before and after weGAN training, ordered by cosine similarity.

We observe from Table 2 that for "Obama," "Trump" and "Tillerson" are more similar after weGAN training, which means that the structure of the weGAN embeddings can be more up-to-date. For "Trump," we observe that "Clinton" is not among the synonyms before, but is after, which shows that the synonyms after are more relevant. For "U.S.," we observe that after training, "American" replaces "British" in the list of synonyms, which is also more relevant.

We next discuss deGAN. In Table 3 , we compare the performance of finetuning the discriminator of deGAN for document classification, and the performance of the FFNN initialized with word2vec. The change is also statistically significant at the 0.05 level. From Table 3 , we observe that deGAN improves the accuracy of supervised learning.

w2v-accuracy deGAN-accuracy mean 92.05% 92.29% sd.

0.06% 0.09% To compare the generated samples from deGAN with the original bag-of-words, we randomly select one record in each original and artificial corpus. The records are represented by the most frequent words sorted by frequency in descending order where the stop words are removed. The bag-of-words embeddings are shown in Table 4 .

From Table 4 , we observe that the bag-of-words embeddings of the original documents tend to contain more name entities, while those of the artificial deGAN documents tend to be more general. There are many additional examples not shown here with observed artificial bag-of-words embeddings having many name entities such as "Turkey," "ISIS," etc. from generated documents, e.g. "Syria eventually ISIS U.S. details jet aircraft October video extremist..." We also perform dimensional reduction using t-SNE (van der Maaten & Hinton, 2008) , and plot 100 random samples from each original or artificial category. The original samples are shown in red and the generated ones are shown in blue in Figure 3 . We do not further distinguish the categories because there is no clear distinction between the three original corpora, "politics," "world," and "US."

We observe that the original and artificial examples are generally mixed together and not well separable, which means that the artificial examples are similar to the original ones. However, we also observe that the artificial samples tend to be more centered and have no outliers (represented by the outermost red oval).

4.2. The TIME data set:

In the TIME data set, we collected all news links on time.com in the GDELT 1.0 Event Database from April 1st, 2013 to July 7, 2017. We then collected the news articles from the links, and kept those belonging to the five largest categories: "Entertainment," "Ideas," "Politics," "US," and "World." We divided these documents into 12,286 training documents, from which 1,535 validation documents are held out, and 3,075 testing documents.

Table 5 compares the clustering results of word2vec and weGAN, and the classification accuracy of an FFNN initialized with word2vec, finetuned weGAN, and finetuned deGAN. The results in Table 5 are the counterparts of Table 1 and Table 3 From Table 5 , we observe that both GAN models yield improved performance of supervised learning. For weGAN, on an average, the top 10 synonyms of each word differ by 0.27 word after weGAN training, and 24.8% of all words have different top 10 synonyms after training. We also compare the synonyms of the same common words, "Obama," "Trump," and "U.S.," which are listed in Table 6 . In the TIME data set, for "Obama," "Reagan" is ranked slightly higher as an American president. For "Trump," "Bush" and "Sanders" are ranked higher as American presidents or candidates. For "U.S.," we note that "Pentagon" is ranked higher after weGAN training, which we think is also reasonable because the term is closely related to the U.S. government.

For deGAN, we also compare the original and artificial samples in terms of the highest probability words, which is shown in Table 7 . We also perform dimensional reduction using t-SNE for 100 examples per corpus and plot them in Figure 4 . All these figures and tables show results similar to Section 4.1. 

A Algorithms for weGAN and deGAN:

Algorithm 1 (for weGAN).

1. Train G based on f from all corpora C 1 , . . . , C M .

2. Randomly initialize the weights and biases of the classifier C and discriminator D. 

B Proof of Proposition 1:

Since X is bounded, all of the integrals exhibited next are well-defined and finite. Since n, a 1 , and a 2 are continuous, it follows that for any parameters, G m (n) is a continuous random variable with probability density q m with finite support. From ( 11 

EQUATION

)

We then obtain from (10) that q * 1 , . . . , q * M ∈ arg min q1,...,q M M m=1 q m (x) • log q m (x) q m (x) + p m (x) dx = arg min q1,...,q M -M log 2 + M m=1 q m (x) log q m (x) (q m (x) + p m (x))/2 dx = arg min q1,...,q M -M log 2 + M m=1 KL(q m (q m + p m )/2).

From non-negativity of the Kullback-Leibler divergence, we conclude that (q * 1 , . . . , q * M ) = (p 1 , . . . , p M ).

C The 20 Newsgroups data set:

The 20 Newsgroups data set is a collection of news documents with 20 categories. To reduce the number of categories so that the GAN models are more compact and have more samples per corpus, we grouped the documents into 6 super-categories: "religion," "computer," "cars," "sport," "science," and "politics" ("misc" is ignored because of its noisiness). We considered each super-category as a different corpora. We then divided these documents into 10,708 training documents, from which 1,335 validation documents are held out, and 7,134 testing documents. We train weGAN and deGAN in the the beginning of Section 4, except that we use a learning rate of 0.01 for the discriminator in deGAN to stabilize the cost function. 

D The Reuters-21578 data set:

The Reuters-21578 data set is a collection of newswire articles. Because the data set is highly skewed, we considered the eight categories with more than 100 training documents: "earn," "acq," "crude," "trade," "money-fx," "interest," "money-supply," and "ship." We then divided these documents into 5,497 training documents, from which 692 validation documents are held out, and 2,207 testing documents. We train weGAN and deGAN in the same way as in the 20 Newsgroups data set. 

:



Figures and Tables:

Figure 2: Model structure of deGAN.

Figure 3: 2-d representation of original (red) and artificial (blue) examples in the CNN data set.

Until maximum number of iterations reached (a) Update C and D according to (4) and (6) given a mini-batch S 1 of training examples {d m i } i,m . (b) Update G according to (5) given a mini-batch S 2 of training examples {d m i } i,m . 4. Output G as the cross-corpus word embeddings. Algorithm 2 (for deGAN). 1. Randomly initialize the weights of G 1 , . . . , G M . 2. Initialize the discriminator D with the weights of the first layer (which takes document embeddings as the input) initialized by word embeddings, and other parameters randomly initialized. 3. Until maximum number of iterations reached (a) Update D according to (12) given a mini-batch of training examples d m i and samples from noise n. (b) Update G 1 , . . . , G M according to (13) given a mini-batch of training examples d m i and samples form noise n. 4. Output G 1 , . . . , G M as generators of document embeddings and D as a corpus classifier.

m log b m subject to M m=1 b m = 1, (11) the solution of which is b * m = a m / M m=1 a m , m = 1, . . . , M . Therefore, the solution to (15) is

in order to provide intuitive estimates.

Table content: <table><tr><td/><td/><td/><td colspan="2">Obama w2v</td><td>Bush Trump Kerry Abe Ne-</td></tr><tr><td/><td/><td/><td colspan="3">tanyahu Rouhani Erdogan he Karzai Tillerson</td></tr><tr><td/><td/><td/><td colspan="3">Obama weGAN Trump Bush Kerry Abe Ne-</td></tr><tr><td/><td>w2v-RI</td><td>weGAN-RI</td><td colspan="3">tanyahu Erdogan Tillerson he Carter Rouhani</td></tr><tr><td>mean</td><td>67.88%</td><td>68.45%</td><td>Trump</td><td colspan="2">w2v Obama Pence Erdogan Bush</td></tr><tr><td>sd.</td><td>0.02%</td><td>0.01%</td><td colspan="3">Duterte he Sanders Macron Christie Tillerson</td></tr><tr><td/><td colspan="2">w2v-accuracy weGAN-accuracy</td><td colspan="3">Trump weGAN Obama Pence Bush Christie</td></tr><tr><td>mean</td><td>92.05%</td><td>92.36%</td><td colspan="3">Sanders Clinton Erdogan Tillerson Macron</td></tr><tr><td>sd.</td><td>0.06%</td><td>0.03%</td><td/><td/><td>Duterte</td></tr><tr><td/><td/><td/><td>U.S.</td><td>w2v</td><td>US Pentagon United Iranian</td></tr><tr><td colspan="3">Table 1: A comparison between word2vec</td><td/><td colspan="2">NATO Turkish Qatar Iran British UAE</td></tr><tr><td colspan="3">and weGAN in terms of the Rand index and</td><td colspan="3">U.S. weGAN US Pentagon United Iranian</td></tr><tr><td colspan="3">the classification accuracy for the CNN data</td><td colspan="3">NATO Turkish Iran Qatar American UAE</td></tr><tr><td>set.</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td colspan="3">Table 2: Synonyms of "Obama," "Trump,"</td></tr><tr><td/><td/><td/><td colspan="3">and "U.S." before and after weGAN training</td></tr><tr><td/><td/><td/><td colspan="3">for the CNN data set.</td></tr></table>



Table content: <table><tr><td/><td>politics original</td><td>India US Carter defense</td></tr><tr><td/><td colspan="2">Indian relationship China said military two</td></tr><tr><td/><td colspan="2">politics deGAN year US meeting used read</td></tr><tr><td/><td colspan="2">security along building worth foreign</td></tr><tr><td/><td colspan="2">world original Turkey Turkish Attack ISIS</td></tr><tr><td/><td colspan="2">said Kurdish Erdogan group bomb report</td></tr><tr><td/><td colspan="2">world deGAN cut company get lot made</td></tr><tr><td/><td colspan="2">code could Steve items may road block phone</td></tr><tr><td>: A comparison between word2vec</td><td colspan="2">US original climate change year study</td></tr><tr><td>and deGAN in terms of the accuracy for the</td><td colspan="2">according says country temperatures average</td></tr><tr><td>CNN data set.</td><td colspan="2">US deGAN area efforts volunteers town</td></tr><tr><td/><td colspan="2">weapons shot local nearly department also</td></tr></table>

Bag-of-words representations of original and artificial text in the CNN data set.

Table content: <table/>

for the TIME data set. The differences are also significant at the 0.05 level. Synonyms of "Obama," "Trump,"and "U.S." before and after weGAN training for the TIME data set.

Table content: <table><tr><td/><td>Obama w2v</td><td>Trump Bush Xi Erdogan</td></tr><tr><td/><td colspan="2">Rouhani Reagan Hollande Duterte Abe Jokowi</td></tr><tr><td>w2v-RI 70.96% 0.02% w2v-accur. weGAN-accur. deGAN-accur. weGAN-RI mean 71.14% sd. 0.02% 83.79% 84.76% 85.38% 0.17% 0.08% 0.11%</td><td colspan="2">Obama weGAN Reagan Rouhani Hollande Abe Jokowi Duterte Trump Bush Xi Erdogan Trump w2v Obama Erdogan Rubio Duter-te Bush Putin Sanders Xi Macron Pence Trump weGAN Obama Erdogan Rubio Bush Sanders Putin Duterte Xi Macron Pence U.S. w2v NATO Iran Japan Pentagon</td></tr><tr><td>Table 5: A comparison between word2vec, weGAN, and deGAN in terms of the Rand index and the classification accuracy for the</td><td colspan="2">Russia Pakistan Tehran EU Ukrainian Moscow U.S. weGAN NATO Pentagon Iran Japan Russia Tehran Pakistan EU Ukrainian Moscow</td></tr><tr><td>TIME data set.</td><td/></tr></table>

Bag-of-words representations of original and artificial text in the TIME data set.Junbo Jake Zhao, Michaël Mathieu, and Yann LeCun. Energy-based generative adversarial networks.In 5th International Conference on Learning embeddings (ICLR 2017), 2017.

Table content: <table><tr><td colspan="2">Entertainment original</td><td>show London attack</td><td>Politics deGAN</td><td>members present</td></tr><tr><td colspan="3">people proud open going right according way</td><td colspan="2">national committee party Paul sign Trump</td></tr><tr><td/><td/><td>home</td><td/><td>removed brief</td></tr><tr><td colspan="2">Entertainment deGAN</td><td>music actor Michael</td><td>US original</td><td>Charleston many</td></tr><tr><td colspan="3">John song going meeting James produced pop</td><td colspan="2">Carolina South funeral hand wrote political</td></tr><tr><td/><td/><td>vocal</td><td/><td>words</td></tr><tr><td>Ideas</td><td>original</td><td>American would</td><td>US deGAN</td><td>Davis head board</td></tr><tr><td colspan="3">service national young country year serve</td><td colspan="2">man relationship recent Sunday stone fire wrote</td></tr><tr><td/><td/><td>security world</td><td/><td>gay well</td></tr><tr><td>Ideas</td><td>deGAN</td><td>city project part</td><td>world original</td><td>Erdogan Turkey</td></tr><tr><td colspan="3">development grand bear often west new status</td><td colspan="2">political power government two leaders</td></tr><tr><td/><td/><td>high agents</td><td/><td>minister AKP</td></tr><tr><td>Politics</td><td colspan="2">original Assange embassy BBC</td><td>world deGAN</td><td>suffering like know old</td></tr><tr><td colspan="3">Swedish charges told authorities officials</td><td colspan="2">violence local daily young interest three first</td></tr><tr><td/><td/><td>Sweden</td><td/><td>man</td></tr></table>

Table8compares the clustering results of word2vec and weGAN, and the classification accuracy of the FFNN initialized with word2vec, finetuned weGAN, and finetuned deGAN. All comparisons are statistically significant at the 0.05 level. The other results are similar to the previous two data sets and are thereby omitted here. A comparison between word2vec, weGAN, and deGAN in terms of the Rand index and the classification accuracy for the 20 Newsgroups data set.

Table content: <table><tr><td/><td>w2v-RI</td><td>weGAN-RI</td></tr><tr><td>mean</td><td>76.14%</td><td>76.74%</td></tr><tr><td>sd.</td><td>0.02%</td><td>0.08%</td></tr><tr><td colspan="3">w2v-accur. weGAN-accur. deGAN-accur.</td></tr><tr><td>87.34%</td><td>89.90%</td><td>89.32%</td></tr><tr><td>0.04%</td><td>0.02%</td><td>0.15%</td></tr></table>

Table9compares the clustering results of word2vec and weGAN, and the classification accuracy of the FFNN initialized with word2vec, finetuned weGAN, and finetuned deGAN. All comparisons are statistically significant at the 0.05 level except the Rand index. The other results are similar to the CNN and TIME data sets and are thereby omitted here. A comparison between word2vec, weGAN, and deGAN in terms of the Rand index and the classification accuracy for the Reuters-21578 data set.

Table content: <table><tr><td/><td>w2v-RI</td><td>weGAN-RI</td></tr><tr><td>mean</td><td>71.28%</td><td>71.43%</td></tr><tr><td>sd.</td><td>0.26%</td><td>0.07%</td></tr><tr><td colspan="3">w2v-accur. weGAN-accur. deGAN-accur.</td></tr><tr><td>92.86%</td><td>95.10%</td><td>94.86%</td></tr><tr><td>0.09%</td><td>0.10%</td><td>0.10%</td></tr></table>

Peer review (iclr2020):
The paper proposes to use Generative Adversarial Networks (GANs) in the context of natural language processing and introduces two models for generating document embeddings. The first model, weGAN, aggregates multiple sets of single-corpus word2vec embeddings into one set of cross-corpus word representations; document embeddings are a function of these updated word embeddings. The second model, deGAN, side-steps word-level embeddings and directly generates document-level representations. For both models, the real examples come from word2vec and tf-idf, while the artificial examples are the ones generated by the network. The authors show that their document embeddings are better than the word2vec/tf-idf baseline at clustering documents according to the corpus they originate from. While the context in which GANs are used is novel and creative, the reasons for rejection outweigh the positives. The main reason is the fact that the paper is outdated by two/three years. There is no mention of the shift towards contextual embeddings that has been happening in the last few years (InferSent, ELMo, BERT) and that became ubiquitous in NLP. Instead, all comparisons are against word2vec (published in 2013) that is no longer a relevant baseline, even for non-contextual word embeddings. In fact, the newest cited papers are from 2017. The Github link in the paper was last updated on 23rd December 2017. My objection is not that this work was started a long time ago, but that it has not been updated in years. In the lack of proper comparison against more novel techniques, it is hard to estimate whether the reported gains over word2vec are meaningful. Another argument for rejection is the fact that the experimental section does not make a convincing case that the positive results matter beyond the very specific datasets and tasks selected. Before commenting in more detail on the selection of datasets, tasks and metrics, I want to document my understanding of the experimental setup; I found it non-trivial to understand what constitutes a *single corpus* in the experiments. My mental model is the following: the paper operates on four datasets (CNN, TIME, 20 Newsgroups, and Reuters-21578). These are four different experiments, and the four datasets never interact. The cross-corpus property of the model is exercised within each of these datasets, which have a natural grouping of topics; for instance, CNN is divided into *US*, *world*, and *politics*. In other words, a cross-corpus CNN model is trained on three corpora: US, world, and politics. With this mental model of the experimental setup, the experiments raise a series of questions: 1) Do the selected datasets / classification tasks matter in the grad scheme of NLP? The end-task is to classify the corpus of origin for each document (for instance, for the CNN experiment, the model has to decide between US, world, and politics). If I understand correctly, the weGAN model jointly trained its word embeddings with this objective. As a final step, the classifier was additionally fine-tuned on this task. So it is not very surprising that weGAN is better at clustering documents than word2vec, since it was explicitly trained to do so. It would have been more informative to see weGAN performing better on a task that was solely fine-tuned on (e.g. some sentiment classification task that was not part of the training objective when embeddings were trained). In that case, one could argue that the enhanced embeddings can be used as an out-of-the box tool for a lot of different tasks. Another option would be evaluating on the same document membership task, but on datasets that were not seen during training. Additionally, the fact that the authors evaluated on *new* datasets with a *new* task distances them even further from any potential comparisons against prior work. Even for the two already-existing datasets with results included in the appendix, there are no references to previous SoTA. 2) Are the reported positive results meaningful? Another problem with not showing results from previous work is that it is hard to understand whether the gains over the word2vec baseline are meaningful. For instance, the classification accuracy for the CNN corpus is increased from 92.05% (word2vec) to 92.29% (deGAN) and 92.36% (weGAN). The authors show that this is *statistically significant*, but is it meaningful? Is a 0.24% or 0.31% increase in accuracy worth the increased complexity of the model? For reference, what ratio of gains/computational cost would BERT incur? 3) What is the message of qualitative evaluation? The authors offer qualitative analysis of their model. For instance, Table 2 shows the top 10 synonyms for words *Obama*, *Trump* and *US*, as produced by word2vec and weGAN. The lists of synonyms are extremely similar and don*t seem to convey much more information besides the fact that weGAN makes minimal changes. I find the particular examples that the authors choose to underline the superiority of their algorithm unconvincing. For instance, the fact that, in the list of synonyms for US, the word British (selected by word2vec as the 9th closest word) is turned into American (on the same 9th position) loses its significance when you also consider the fact that weGAN did not derank any of the arguably less similar terms from higher positions (e.g. Turkish remains the 6th most similar term to US). Other nit-picks: - Formatting issues: In Sections 3.1 and 3.2, the numerical references to equations are broken. Figures 3 and 4 cover the text above. Tables 2, 4, 6, 7 are hard to read. It almost feels like the paper was updated from one conference format to another, without the necessary Latex adjustments. - The paper has no *Conclusion* section - Improper use of singular / plural of *corpus* / *corpora* To end on a more positive note, here are some reasons why I did appreciate the paper: - The usage of GANs in this context is very creative; the models do not generate *raw* text, but rather intermediate representations; this might be a promising direction for using GANs in NLP. - The description of the models in sections 3.1 and 3.2 is rigorous, with useful explanations that break down the reasoning behind the loss function choices.