A Quality-Diversity Controllable GAN for Text Generation
Authors: Xingyu Lou
Kaihe Xu
Zhongliang Li
Tian Xia
Shaojun Wang
Jing Xiao
Venue:  (2020)
Keywords: text generation
GAN
quality-diversity
generalized Jensen-Shannon divergence

Abstract: Text generation is a critical and difficult natural language processing task. Maximum likelihood estimate (MLE) based models have been arguably suffered from exposure bias in the inference stage and thus varieties of language generative adversarial networks (GANs) bypassing this problem have emerged. However, recent study has demonstrated that MLE models can constantly outperform GANs models over quality-diversity space under several metrics. In this paper, we propose a quality-diversity controllable language GAN.

adversarial Under review as a conference paper at ICLR 2020
A QUALITY-DIVERSITY CONTROLLABLE GAN FOR TEXT GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Text generation is a critical and difficult natural language processing task. Maxi- mum likelihood estimate (MLE) based models have been arguably suffered from exposure bias in the inference stage and thus varieties of language generative ad- versarial networks (GANs) bypassing this problem have emerged. However, re- cent study has demonstrated that MLE models can constantly outperform GANs models over quality-diversil space under several metrics. In this paper, we pro- pose quality-diversity controllable language GAN. We illustrate intimate con- nection between forward/reverse KL divergences and our principled approach, and demonstrate that our proposed model can achieve quality-diversity control by adjusting a hyper-parameter. Through extensive experiments, we show that the proposed model outperforms MLE models and other GANs variants on synthetic data and is more competitive on real-world tasks such as COCO Image Captions and EMNLP2017 WMT News datasets.
INTRODUCTION
Text generation is of the most challenging and widely used natural language processing task and it serve as a critical module on numerous practical applications. In recent years, neural autoregres- sive models have shown superiority on various natural language generation (NLG) tasks (Zhang & Lapata 2014 Gal & Ghahramani] 2016 Du et al., Fiorini & Lu 2018). The most common method of training these models is the maximum likelihood estimation (MLE) which maximize the log predictive likelihood of each true token in the training sequence given the previous observed to- kens. However, MIE models have been arguably suffered from exposure hias (Bengio et al.) 2015] Ranzato et al.) 2016) which means discrepancy between training and inference stages due to the fact that at inference time, the model predicts next token based on the sequence generated by itself which may not occur in training set. In order to overcome this problem, scheduled sampling (Ben- gio et al. 2015) and professor forcing (Lamb et al. 2016) are proposed, but scheduled sampling proven be fundamentally inconsistent (Huszár 2015) and professor forcing
method similar to GANs (Goodfellow et al., 2014).
Varieties of generative adversarial networks (GANs) (Goodfellow al., 2014) models for text gen- eration have emerged as alternatives to MLE models with the hope of not suffering from exposure bias. However, due to the discrete nature, GANs have difficulty in text generation since the gradients from discriminator can not be passed to generator explicitly and thus most these GANs based mod- els (Yu al.) 2017: Guo al. 2018 Lin et al.) 2017 Fedus al. 2018 Xu et al. 2018) treat text generation sequential decision making process (Bachman & Precup 2015) and utilize policy gradient (Williams) 1992) overcome this difficulty. Despite the impressive achievements, they inevitably have problem of high variance due to the use of reinforcement learning (RL) strategies.
Caccia et al. 2018) have made several surprising observations and argued that, through simple temperature sweeping at inference stage, MLE models can constantly outperform these RL-based GANs models by a large margin over quality-diversity space under several metrics. They attribute this to the fact that the negative impact coming with RL is severer than so-called exposure bias caused by MLE. A RL-free approach to train GANs for text generation without resorting to an explicit neural network the discriminator, named No Neural Network the Discriminator GAN (N3DGAN), is proposed by (Li et al. (2019) and shows state-of of------theettee art results on both synthetic and real datasets under quality-only metrics.

---

Under review as a conference paper at ICLR 2020
We propose quality-diversity controllable GAN through principle approach which is gener- alization of N3DGAN (Li et al. 2019). This generalization has intimate connections with direct optimization of both forward and inverse KullbackLeibler (KL) divergences with respect to pirical distribution and model distribution. As shown in (Huszár, 2015), minimizing forward KI. divergence often leads models that overgeneralize, and sometimes produce samples that are very unlikely under true distribution and thus it is good at diversity. Minimizing the reverse KL diver- gence instead will try to avoid any behaviour that unlikely under true distribution the cost of ignoring modes of true distribution completely, thus it is good at quality. Through this principled ap- proach, we demonstrate that our proposed model can control the quality- trade-off, that is, the degree to which the model depends on forward and reverse KL divergences respectively through a single parpeeear
The key contributions of our work are three-folds: (1) We propose simple but effective GANs model for text generation which can control the quality-diversity trade-off through a single hyper- parameter at training stage. (2) Different from most existing GANs variants, our proposed model, as a generalization of N3DGAN, does not need to rely on MLE pre-training and is RL-free. (3) Through extensive experiments, we show that our proposed model outperform MLE based model and other GANs variants over quality- space the synthetic-date tasks and are more competitive to MLE model than other GANs variants in the real-world tasks.
2 RELATED WORK
Huszár (2015) analyzes the difference between optimizing forward and reverse KI. divergences, and suggests to use a generalized Sensen-Shannon divergence to replace MLE, but he doesn't 't real- ize its relation to the mini-max game's 's value function well as tractability of directly optimizing the generalized Jensen-Shan Sensen-Shannon divergence on training data, and morcover he doesn't 't conduct any experiments.
Caccia et al. (0188) try to navigate the search and generator rejection sampling (Ott et al., 2018) methods, unfortunately all these methods can only navigate the quality-diversity space at inference stage.
space through temperature sweep, beam
Li al. 2019) propose N3DGAN which is RL-free and doesn't 't need additional maximum likeli- hood pre-training, but they only report results on quality metrics but not on diversity.
et 2019) propose cooperative training (CoT) approach for text generation which dont have to rely on RL and MLE for pre-training, instead they optimize an approximate Jensen-Shannon divergence. It is done by introducing a mediator and dropping the term in the gradient of generator which behaves like RL. In the synthetic-date experiments, they perform similarly but slightly worse than MLE model (Caccia et al. 2018).
de Masson d' Autume et al. (2019) combine existing techniques such as large batch size, dense rewards and discriminator regularization to train language GANs from scratch without the need for MLE trainiain But they still have to use RL which is essentially the same as most language GANs (Yu et al.) [2117] Guo et al.] 2018 Lin et al., 2017 Fedus et al., 2018 Xu et al., 2018), and they don't conduct any experiment on synthetic data which has oracle metrics to validate its effectiveness.
A QUALITY-DIVERSIT CONTROLLABLE GAN
In the well-known adversarial modeling framework Goodfellow al. (2014), there are two play- ers, one is called generator represented by a differentiable function G and another one is called discriminator represented by a differentiable function D. They approximate the data distribution Pdata through an alternative mini-max game between these two components. The discriminator D trained to discriminate the samples drawn from the generator distribution PG from the real data, while the generator is trained to fool the discriminator so that it cannot distinguish between the sam- ples from PG and Pdota- Goodfellow et al. (2014) proved that if the generator has enough capacity to cover data distribution Pdata- then optimizing the mini-max game is equivalent to directly opti- mizing the Jensen-Shannol Shannon divergence. Unfortunately Goodfellow et al. (2014) didn't 't distinguish the real data either population (true but unknown) distribution empirical distribution.
2

---

Under review as a conference paper at ICLR 2020
Li al. 2019) notice the distinction and they explicitly use the empirical distribution in their ap- proach. They argue that when an alternative mini-max optimization procedure is performed for the value function, closed form solution for the discriminator exists in the maximization step, thus an explicit neural network model for the diseriminator not needed. Moreover optimizing the mini-max game is equivalent to directly optimizing the Jensen-Shannon divergence (JSD) between the generator's s distribution and the empirical distribution over the training data, where the corre- sponding KL divergence related the generator's distribution and the empirical distribution over the training data is reduced to the sum over the training data. This means that sampling from the generator over the entire space as well as the policy-gradinn algorithm to compute the gradient of the generator can be eliminated. Optimizing the JSD becomes computationally tractable to train the generator for text generation.
We now propose quality-diversity controllable GAN which is generalization of N3DGAN LLi al., 2019) and illustrate connections the forward KI. divergence and the reverse KI. diver- gence as stated below.
Since the samples in the training data are limited, we use E [0, 1] to denote the ratio of labeled samples discriminator receives from the generator and the training data, and let the discriminator D and the generator G play the following two-player minimax game with a generalized distinguisha- bility game value function V D):
min max V (G. D) Tr Ez--io((>) [log Dxx)] + (1 TT Eo-na((> [ogg(1 D(x))]
(I)
where Pdata (I) denotes the empirical distribution over training set C. {I1, ,IN) with N sam- ples, and
if E otherwise
Paaa((3))
is equivalent to optimize generalized Gensen-Shamme diver- gence (Huszár 2015) between Pdata and pc, which formulated follows
We will then show that optimize ①
JSD (fant (x) || pa (1))
Dr.. (panta
(x) TTaat (x: (1- (1-r))a(z)) (1 - Drr (ar |l TT (x)) (1 - (x))
and control the quality-diversity trade-off through single parameter æ.
When optimizing the value function V (G, for any given generator G, the optimal discriminator (z) is
Proposition 1.
Ban (x)
if o E C otherwise
D2(2)=
Fran (7) xpa(()
and the value function at the optimal discriminator D becomes
Va (a((a)))
x. ,, (x))
Peato (x)
(x(x)
Σ Pama log
+ (1 - ④) Σpo(#) po(x) (r) log
x. n
x TP (x) + (1 x) x2)
T (x)
+ - (x)
sEC
zEC
+ (1 - TT) log(1 -w)
Σ PG(x)
#EC
Given any generator G(z), the training criterion for the discriminator D(x) is maximize the value function V= D) in and could be re-written as.
Proof:
V= D) = Σ
Jama log D(x) + (1 - π) Σ
Pa(x) log(1 - D(x))
sEC
sEC
(3)
+ (1 - T)
(o( log(1 D(x))
=Ec
3

---

Under review as a conference paper at ICLR 2020
Taking derivative with respect to (z)), we have
aV, D)
Panl
(x)) (1 T) pa(2)
if e C otherwise
(1-D(w))
(4)
@D(x)
πT) [()))
p((
Thus, the optimal solution for ((x) is
Ben (x)
if E
De(() =
*Dau
(1 (2)
(5)
otherwise
Plug 5 into B
we have
DGG,D&
Pdaaa (x)
po(x)
Pduta log
(11-
- x)
Σ (x) (c))<:loggg
TPamal (x) + (1 (x)
TPaata (x) + (1 T)Pa(>))
sEC
ZEC
+ (1 - πT) log(1 T) Σ pc(x) + log π
(2)
(6)
sEC
sEC
The generalized JSD can be written
JSD (Pdaaa (2) ) px(x))
n DKL (Feaaa (2) TTdataa (2) + (1 +11 + (1)-7)DDL(plll TPdaaa (x) + (1 ((1-T)pa' ()))
Pdlata (2)
PG(x))
Σ Pdmta log xEC
+ (1 (1-x)) Σ
(((()
log T (I) 17 TT) (x))
TTPetaa (2:) (1 TTPG((2)
2EC
+ (1 TT) log(1 T) Σ pa( (1 T) log(1 m)
FEC
Because the last terms on the right side of Equation (6) and Equation both are constant, thus
equation (2) is established.
Proposition 1 shows that optimizing the mini-mas max game of equation (1) is equivalent to optimizing equation (2) which is over the training data, thus it is tractable and the model parameter 0 can be updated directly using stochastic gradient descent (SGD) or its variants.
Proposition 2. When π is converging to O, optimizing (G is equivalent to optimizing the forward KL divergence DKL (pa
(x)), i.c. optimizing maximum log-likelihood; and
when æ is converging to 1, optimizing (G is equivalent to optimizing the reverse KI. divergence DKL (pG()) ) Adata (z)).
I-x
Proof: Consider the Equation when + 0, the second and third terms of V=(.DD) is equivalent to and thus we have
x
lim (G,
lim
Σ
Priaaa log
Paau
Drr. (psmaa (x:) (G)))
(8)
TDamaa (x) + (1 2))
Zec
The same can be proved that in the limit of π + 1, V.G.D) is equivalent to reverse KL divergence DKL (xa()) Paaaa (x)).
1-x
This proposition shows an intimate connection optimizing the objective VG D) to those the forward and reserve KL divergences. Thus we can control the quality-diversity trade-off for text generation through single hyper-parameter æ..
EXPERIMENTAL STUDY
4.1 DATASETS
The experiments consist of two parts: synthetic-data experiments and real-world experiments. We two datasets, COCO Image Captions dataset and EMNLP2017 WMT News dataset in real- world experiments since they have become common benchmarks for text generation (Guo et al.
2018
Caccia al. 2018 Montahaei et al. 2019)
4

---

Under review as a conference paper at ICLR 2020
4.1.1 SYNTHETIC DATASET
In synthetic-data experiments, order to test the validity of our proposed model we conduct simulated test similar to Yu et al. (2017). Following them, oracle model to generate the real data distribution Poracle (z ... It-1) to stimulate the real- world structured sequences. We use the oracle model generate 10,000 sequences of length 20 and 40 respectively as our training set and another two 5,000 sequences are generated as our validation set and test set. Both of them have vocab size of 5,000.
random initialized LSTM the
4.1.2 COCO IMAGE CAPTIONS DATASET
For COCO Image Captions dataset (Chen al. 2015), use the same pre- pre-processing method as (Guo et al. 2018), but we randomly select 5,000 samples from training set as our validation set. The resulted dataset consist of a training with 75,000 samples, validation set with 5,000 samples and a test set with 5,000 samples. The vocab size is 4,838 and the maximum length of captions is 32.
4.1.3 EMNLP2017 WMT NEWS DATASET
EMNLP2017 WMT News BBoarr et al. 2017) is collection of news texts for the machine trans- lations task, among a version of this dataset for English Corpus containing more than 600,000 sen- tences, we drop the sentences having more than 3 words with less than 400 frequency and keep only the sentences that have between 20 and 40 words. We replace these most infrequent words with token unk > Next, randomly sample 200,000 sentences the training set, 10,000 sentences as the validation set and 10,000 as the test set. The resulted vocab size is 3,980.
4.2 EVALUATION METRICS
is still difficult to accurately evaluate the effects text generation models using existing metrics since the effects consist of several aspect such consistency, diversity and fluency. Thus, in order to be consistent, we choose evaluation metrics according to previous work (Semeniuta et al. 2018 Caccia et al. 2018 de Masson d' Autume et al. 2019 Montahaci al. 2019). We use negative log likelihood for synthetic-date experiments and use n-gram based metrics and Fréchet Distance (FD) for real-data experiments.
4.2.1 NEGATIVE LOG LIKELIHOOD
In the work of Yu et al. 2017). the authors recommend to use the oracle negative log likelihood (NLLorecte) which is an oracle quality metric for synthetic-data experiments. However, only use this metric will be misleading since it is blind to diversity. Thus, similar to [Caccia al. (2018), we additionally use a metric NLLeest to represent diversity that calculated by taking as input the sentences generated by oracle model to the language model we're re interested in.
4.2.2 N-GRAM BASED METRICS
We use the local metrics BLEU-5 (Papineni al. 2002 and Self BLEU-S (huu et al. 2018 to cap- ture quality and diversity respectively. While BLEU based metrics is considered to have limitations that only capture local consistency and fail capture semantie variations (Semeniuta et al.) 2018 Caccia et al. 2018: de Masson d' Autume et al. 2019), they are still one of the most widely used and simple metrics for text generation tasks.
4.2.3 FRÉCHET BERT DISTANCE
Fréchet Inception Distance (FID) is originally proposed to computes the distance between two Gaus- sian distributions of the features of real images and generated images extracted by Inception network (Heusel et al. 2017). We follow Montahaei al. (2019) utilize BERT (Devlin et al. 2019), pow- Molllo/Mootahee)
erful unsupervised pre- pre-training model for text, to replace the Inception network to capture quality and diversity of generated sentences. This metric named as Fréchet BERT Distance (FBD).
5

---

Under review as a conference paper at ICLR 2020
We do not MS-Jaccard and Language Model Score (LMS) because MS-Jaccard is also gram based metric like BLEU and it is highly correlated with FBD (Montahaei et al. 2019). and LMS relies on training new models and have an inherent bias favoring MLE language they are trained using the same criteria (de Masson d' Autume et al. 2019).
laEaaguagge models, since
4.3 IMPLEMENTATION DETAILS
We choose hypee-parameter according to the result validation set and report all metrics test set. Our models are implemented with Tensorflow (Abadi et al. (0116) using an NVIDIA V100 GPU. The hidden size, embedding size and batch size we used 32, 32 and 64 respectively which are kept the same as Texygen framework (Zhu et al. 2018) for fair comparison. We terminate the training of all models based on NLLoraste in synthetic data experiments and based on their BLEU-5 scores on validation set in real-data experiments.
SDOCAS
SECCIN
MLLex
ALL
NCLored
(a) Synthetic data with length 20
(b) Synthetic data with length 40
Figure I: Temperature sweep curves for the synthetic-dee data experiments of sentence length 20 (left) and 40 (right) over NLLtest NLI-ovacle space (lower is better for both metries)
SCOGAN
M.Len
NCLommee
(a) Synthetic data with length 20
(b) Synthetic data with length 40
Figure 2:
Frontier curves of our models and three baseline models. Left is better and down is better. The models trained via proposed adversarial objective perform better compared with MLE model and other GANs variants.
4.4 EXPERIMENTAL RESULTS
In this section, report the experimental results synthetic-data experiments and real-world experiments. We compare our proposed models with three baseline models: MLE, SeqGAN (Yu al. 2017), LeakGAN (Guo al. 2018) in all the experiments. Similar Caccia al. (018).

---

Under review as a conference paper at ICLR 2020
0.
OURO
98
Self BUBLLSS
Self
0.7
0355
0.50
0.45
0.40
0.25
0.30
0.25
00.70
23455
0..00
-3.05
0.50
-2.45
-0.40
-0.25
-030
Negative BLEU-5
Negative BLEU-5
(a) COCO Image Captions dataset
(b) EMNLP2017 WMT News dataset
Figure 3: Negative BLEU-5 plotted against Self BLEU-5 for COCO Image Captions (left) and EMNLP2017 WMT News (right) test set. Lower is better for both metrics.
Temperature
Models
0.7
0.8
0.9
1.0
1.1
1.2
MLE SEQGAN
4.247
4.125
4.064
4.067
4.141
4.346
4.783
4.778
4.811
4.957
5.171
5.425
LEAKGAN
5.585
5.580
5.692
5.197
5.441
5.702
OURS π -
4.231
4.128
4.076
4.088
4.162
4.325
OURS =
4.398
4.267
4.113
4.065
4.031
4.129
OURS I 0.5
4.443
4.234
4.122
4.053
4.024
4.147
TRAINING SET
3.581
Table 1: FBD scores COCO Image Captions test set at different temperatures.
we conduct sweep of temperatures in inference stage evaluate the trained models over quality- diversity space. We fixed the hyper-parameter æ and sweep the temperatures that we can get different temperature curves for our proposed models trained with different т. Samples from our proposed model and baseline models are shown in Appendix According the generated samples, we can observe that our proposed model outperforms SeqGAN and LeakGAN, and when æ - 1, can be seen that the samples are more focused some of the topics and have good quality, while when ı - 1, the topics of the samples are more diverse and have better diversity.
4.4.1 SYNTHETIC EXPERIMENTS
In the synthetic-data experiments, we use NLI-oracle and NLL as our quality and diversity met- ries respectively. Both of them are the lower the better. Lowering the temperature make the model tend to produce high quality but similar sentences, while raising the temperature makes the model tend generate diverse but relatively random sentences. We compute all metrics different tem- peratures for each model to draw sweep curves in divvety-d space.
Figure/ 1 depicts the temperature sweep curves of sequence length 20 and 40 respectively over NLL.eat
space. From this figure, we can observe that when + 0, the curve of our proposed model basically overlaps with the curve of the MLE model which shows the correctness of proposition 2. Besides, we can also observe that our models outperform other GANs variants by a large margin and perform much better than MLE on quality metric at the cost of relatively small losses diversity metric. It is worth noting that our models do not need additional maximum like- lihood pre-trat pre-training and reinforcement learning and thus it is more faster in training than traditional GANs variants. As can be seen from Figure 2, the frontier curve obtained from a family of temper- ature sweep curves of our models outperforms the curves of MLE, SeqGAN and LeakGAN in most cases.
7

---

Under review as a conference paper at ICLR 2020
Temperature
Models
0.7
0.8
0.9
1.0
1.1
1.2
MLE
6.263
5.524
5.067
4.816
5.204
5.726
SEQGAN
7.423
6.674
6.321
5.932
6.268
6.814
LEAKGAN
7.632
6.962
6.510
6.176
6.560
7.138
OURS +
6.260
5.513
5.082
4.803
5.153
5,733
OURS π 0.1
6.784
6,127
5.663
5,341
5.754
6.244
OURS 0.5
6.660
6.004
5.542
5.241
5.640
6.155
TRAINING SET
3.317
Table 2: FBD scores on EMNLP2017 WMT News test set at different temperatures.
4.4.2 REAL-DATA EXPERIMENTS
Figure 3 reports negative BLEU-5 versus Self BLEU-5 metrics on COCO Image Captions and EMNLP2017 WMT News test sets for the models trained via proposed objective, MLE model and other GANs variants, Table | and show the FBD scores on these two test sets respectively for the models and training set.
For COCO Image Captions task, as can be seen from Figure 3(a) and Table when parameer 0.1 and diversity spaces and outperform MLE model in terms of quality metric, which may be because compared with maximum likelihood, our models can properly optimize both forward KI. diver- gence D (aama (x) || (x)) and reverse KL divergence x (Pa(x) || Pdata and compared with other GANs variants, we do not rely on reinforcement learning. When hyper-parameter ग 0, demonstrate that the performance of our model is al- most identical the MLE model which is consistent with proposition and better than other GANs variants..
0.5, models outperform other GANs variants under different quality-
the result shown in Figure 3(a) and Table [
For EMNLP2017 WMT News task, as shown in Figure 3(b) and Table 2 when 0.1 and π 0.5, although our models does not perform as well as the MLE model, they are significantly better than other GANs variants and more competitive to MLE model.Consisten. with the conclusions obtained from the synthetic data experiments and the COCO experiment, the performances of our proposed model and the MLE model are almost equal when π tends to zero. Compared with COCO task, the difference between the FBD scores of the models the News task and training set is slightly large which may be because EMNLP2017 WMT News is relatively more difficult task.
CONCLUSION
This research proposes quality-diversii controllable GAN which is generalization of N3DGAN and it can achieve quality-diversit control through a single hyper- syper-parameter æ. When converges to O, it is equivalent to directly optimizing forward KI. divergence; and when converges to 1, equivalent to directly optimizing reverse KI. divergence. Through experiments in synthetic- data and real-data scenarios, verify the convergence and the controllability, and we show that our proposed model outperforms MLE model and other GANs variants over quality-diversity space in the synthetic-data experiments, and in the real-data experiments, it is more competitive to MLE model than other GANs variants under several metrics. As can be seen from the experimental results, depending the task and purpose, such as get higher quality text more diverse text, we should choose different hyper-parameter и to get better results on what we are interested in.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Va- sudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In Proceedings of USENIX, 2016.
8

---

Under review as a conference paper at ICLR 2020
Philip Bachman and Doina Precup. Data generation sequential decision making. In Proceedings of NIPS, 2015.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Proceedings of NIPS, 2015.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shu-jian Huang, Matthias Huck, Philipp Koehn, Qun Liu, and Varvara Logacheva. Findings of the 2017 Conference on Machine Translation (WMT17). In Proceedings of the Second Conference on Machine Translation, 2017.
M. Caccia, L. Caccia, W. Fedus, H. Larochelle, J. Pineau, and L. Charlin. Language GANs falling short. In Neural Information Processing Systems Workshop on Critiquing and Correcting Trends in Machine Learning, 2018.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/ 10b//5504.00355, 2015.
Cyprien de Masson d' Autume, Mihaela Rosca, Jack W. Rae, and Shakir Mohamed. Training lan- guage gans from scratch. CoRR, abs/1905.09922. 2019.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings NAACL-IILT, 2019.
Xinya Du, Junru Shao, and Claire Cardie. Learning to ask: Neural question generation for reading comprehension. In Proceedings of ACL, 2017.
William Fedus, Ian Goodfellow, and Andrew M. Dai. MaskGAN; Better text generation via filling in the In The 6th International Conference on Learning Representations, 2018.
Nicolas Fiorini and Zhiyong Lu. Personalized neural language models for real-world query auto completion. Proceedings of NAACL-HLT, 2018.
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. Proceedings of NIPS, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of NIPS, 2014.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. Proceedings of AAAI, 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Proceedings of NIPS, 2017.
Ferene Huszár. How (not) to train your generative model: Scheduled sampling, likelihood, adver- sary? arXiv preprint arXiv:1511.05001, 2015.
Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron Courville, and Yoshua Bengio. Professor foreing: A new algorithm for training recurrent networks. Proceedings of NIPS, 2016.
Zhongliang Li, Tian Xia, Xingyu Lou, Kaihe Xu, Shaojun Wang, and Jing Xiao. Adversarial dis- crete sequence generation without explicit neural networks as discriminators. In Proceedings of AISTATS, 2019.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-Tine Sun. Adversarial ranking for language generation. In Proceedings of NIPS. 2017.
Sidi Lu, Lantao Yu, Siyuan Feng, Yaoming Zhu, and Weinan Zhang. Cot: Cooperative training for generative modeling of discrete data. In Proceedings of ICMI., 2019.

---

Under review as a conference paper at ICLR 2020
Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. Jointly measuring diver- sity and quality in text generation models. Proceedings of NAACL-HLT, 2019.
Myle Ott, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. Analyzing uncertainty in neural machine translation. In Proceedings of ICML, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL., 2002.
Marc' Aureloo Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level train- ing with recurrent neural networks. In The 4th International Conference on Learning Represen- fations, 2016.
Stanislau Semeniuta, Aliaksei Severyn, and Sylvain Gelly. On accurate evaluation of gans for lan- guage generation. CoRR, abs/ 1sss.00664496. 2018.
Ronald J Williams. Simple statistical gradient-followwe algorithms for connectionist reinforcement learning. Machine learning, 1992.
Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. Diversity-promotin gan: A cross-entropy based generative adversarial network for diversified text generation. In Proceedings of EMNLP, 2018.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. In Proceedings of AAAI, 2017.
Xingxing Zhang and Mirella Lapata. Chinese poetry generation with recurrent neural networks. In Proceedings of EMNLP, 2014.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang. and Yong Yu. Texygen: A benchmarking platform for text generation models. In Proceedings of SIGIR, 2018.
10

---

Under review as a conference paper at ICLR 2020
A SAMPLES
Sources
Example
OURS x -0
person is sitting on top of a wooden table
A boy with blue kite the beach holding his child
little girl holding a kite in a grassy field.
A young boy flying a kite in front of colorful kite.
man wearing a girl holding plant racer in the air.
A girl with a large body of people flying kites in the sky
Two people standing in a living room with food in the open..
A tall dog laying on top of a tennis court.
A couple of people standing next to a field playing objects wade.
man and boy jumping front of a large chair.
OURS π -
A couple of people standing on a beach.
young child flying a kite sitting on a beach.
A girl is standing on a lush green field.
A group of people on top of a sandy beach.
A crowd of people flying kite in snow slope
person is flying a kite laying on a sandy heach
A person is holding kite in garden
Two women sitting on top of a sandy beach.
group of people in field.
A man standing on the beach next to the water.
MLE
woman in a wide picture holding kite with a kite.
A woman in a living room next to large window.
man using a cell phone holding a kite.
A tennis player is flying kite in a field.
man and a girl take picture to fly a kite.
A man with child in park while woman watches.
Three people and children sitting on chairs, talking on a cellphone.
crowd of people playing kites on top of machine.
Two men sitting in a green room next to each other.
There skate board on his beach with tennis sky
SeqGAN
A group of people on the beach that are uncooked
A man is sitting in a chocolate to he held on the beach.
Two men are the kite in the distance.
man is brushing his teeth on top of water
A group of zebra is seen on video game
A group of guys standing at the beach during a frisbee.
Two device with chairs and two ready for women stands to it
The computer is playing a very steel skate games space.
man is using her pass
A group of men standing in sky next to dark building.
LeakGAN
television brushing in a stair mat at the table
beautiful baseball player holding bat is traveling in the ocean and water.
Two young men standing near a phone at five coffee next to sidewalk.
A blue dog sitting on top of kite at living room.
A group of people standing with her kite in the beach.
big train with checkered cake. keyboard and mouse.
A person wearing a hat holding wii boat holding two chairs.
man sitting next to a window
An old man holding a tennis ball while upside his racket in the sky by the sky. here boat on a living room on a table with a tennis ball.
Table 3: Samples taken from models on COCO Image Captions
11

---

Under review as a conference paper at ICLR 2020
Sources
Example
OURS π +
in fact, this in the past, at least in the european union, given the public nuclear program result of the macroeconomic junkg. in the world, he lost an annual funds rate is even less than the government s single bank planning.
the main factors view: preserving the us and central banks will be junk; the current global financial crisis, and is no longer
at this time, the and the eurozone s junk; would reduce the majority of policies that still driven by both sides for junk,. the democratic party will be able to junkı, the junk; of the american and political and cultural opportunity to fix the security council.
there has become time for iran and damaging only in the region where the us junk; us economy would be competitive.
after all, this is the junky of the lessons about the junkg. would become the region seeking to challenge to the national culture.
true economic crisis in a growing global scale so much less than they have benefited from emerging economies.
however the budget deficits is
but they are in the crisis in other words, and will have junk;, the emergence of junk; of the eu and north america.
in the wake the transformation of the media. there is question who would be junk; in the past 30 years.
OURS π +1
in the rest of the election. however the us will not at the difficulty of climate change.
more important should be the junk; of europe and used in modern history and all the chinese central banks shown to junk; the european union and the global financial crisis.
with the global crisis the us administration leaders must act fast in the junk; of the global financial system.
how many of the europeans could bring it to be in the common way dominate the financial crisis.
in fact, it not at the same time, there is junkg by the west, and the global financial crisis.
however particularly important to explain the chinese government.
this would be able to junk; its debts, it is supposed to junkę the international economy and junk; violence.
the costs of the junk; will be increasing
the financial crisis
the world politics is likely to accept the expense of others in the crisis.
Table 4: Samples taken from our proposed model on EMNLP 2017 News
12

---

Under review as a conference paper at ICLR 2020
Sources MLE
Example
they pushing for long as 18 % of chinese without the currency support of the new middle east strengthening their imports
the government relationship with american power was junk; to become crucial tool create a serious threat from israel and the junk; of the way. for example, however, the rest of the world is we are the junky of the junky that markets must be able to success.
in the fourth quarter of 2011 the largest of the world s periphery economies created the world peace in the junk; of 2007
the us was unable to its willingness to tackle the financial crisis, the new government must be addressed by the start to junk; and the future
indeed, the integrity of the world is not about whether the eu has to be junk; by junkg as well as the iranian constitution, as well as their problem. the us policy is that would have not seen the dollar as the us junk; asia and the world economy.
moreover, , the eu inflation is more likely take more improved than about the us and the benefits of economic recovery. a key question is that junk; should be on a new framework for a junki
from the american government.
this that the russian economy is just one-third of the value of I billion annually a source of global trade.
SeqGAN
like the junk? of the center for the most of the russian crisis. the junk; of the uk is right to ensure that the will junk-- -
it said that it says junk;, and could not take significant impact on oil. in asia, political has been wrong the world bank, like what is the most important time.
in the recent crisis, is so that something else about most of the parties. what is important for observers that high and how he could junk; to support the education system.
we could develop a relatively little chance to fundamental classes, it is likely to junk; its response to the yen.
on the asian financial system, like all of the west bank, it is what could not junk; to put up its foreign junki.
in short, unprecedented increase in the ecb would allow junk; opportunity to rise to junk; for the euro
manage the junk;, of the economic crisis, and the administration should increase their lives.
LeakGiN
but gradually. the candidates must have to the excessive junkg and that the old conflict can be made for the necessary housing. both of the tax increase are on an opportunity to come to wonder that iraq is not little scenario.
junky to recognize the need for greater responsibility for junk; concerning its foreign program.
finally. junki, much of there is no more risky to achieve the goal. it is likely to be reduced to make domestic risk to stimulate inflation in the national cooperation.
the relationship between a financial system is associated raise income inequality
but any recent problem that made such events as much by junkı up between global investment and strategic costs. for radical institutions to keep the access to minimize the biggest impact of an independent world..
the junk; process of capital requirements must be given that, or junkę and from the decision to guide global security.
it is nothing that hope that, lack of trust, such as a freedom of law.
Table 5: Samples taken from baseline models on EMNLP 2017 News
13


Reviews:
[{"review_id": "H1e4ESkYKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a GAN-based text generation approach, where the authors propose to directly optimize a weighted version of JSD replacing p_data with its empirical distribution. I find the theoretical analysis of the approach confusing, thus would like to get clarification from the authors. The experiments largely rely on automatic evaluation, which is known to be unreliable for text generation. I'd like to see human evaluation of the generated sentences, and at least some example outputs should be shown (even if it's cherry-picked). Given that both the theory and the empirical results are not solid in the current version, I intend to reject the submission.\n\n=== Theoretical analysis ===\n1. Main question: based on Equation 2, the optimal solution p_G^* is the empirical data distribution. It's unclear if p_G^* goes to the real data distribution when N goes to infinity.\n2. Given the definition of the generalized JSD in Equation y, when \\pi = 0 and \\pi = 1, JSD_\\pi is both 0. How does it control the balance between forward and reverse KL? I'm also wondering what's the connection between Proposition 2 and the interpolation between forward and reverse KL (which is implied in the text).\n3. In the proof of Proposition 2, what is H? Also in Equation 8, second line, how does the second term disappear? Would be good to have complete proof in the appendix.\n\n=== Empirical results ===\n1. The description of NLL_test and NLL_oracle is very brief. Could you specify what are the language model and data used in each case?\n2. In Table 2, all numbers are pretty close, are they significantly different? It would be really helpful to show some qualitative results as well.\n3. Is there evidence that \\pi is controlling the tradeoff between quality and diversity? From the experiments it's mainly controlled by the temperature.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BygMPlX3Kr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a method (loss function) for training GAN model for generation of discrete text token generation. The aim of this loss method to control the trade off between quality vs diversity while generating the text data.\n\nFor example,\nif original sentence is \"The company ’ s shares rose 1 . 5 percent to 1 . 81 percent , the highest since the end of the year .\" and the output is \"The company ’ s shares rose 1 . 5 percent to 1 . 81 percent , the highest since the end of the year .\" then the quality of generation is high and diversity is low. \n\nPros:\n1. The paper is very well written, with importance to the smaller details. It is a very good read for even people who are new to this problem. Especially, I appreciate the part where authors took efforts in writing why a few metrics are not used!\n2. The motivation is good and also contributions are explicitly written. The details of the approach are provided clearly.\n3. The experiments are provided in two different datasets and also the experiments support the two major claims in the paper.\n\nCons:\n1. The primary concern with this submission is the novelty. The Proposition 1 of using forward-backward JSD based divergence has already been proposed in Li et al. (2019). Also, Li et al. (2019) proposes the entire contribution of this paper. The only difference is the introduction of \\pi, which controls the percentage of labelled data to be considered between the generated data and original data. Basically, Li et al. (2019) is a specific case of this paper where \\pi = 0.5. Thus, I would consider this paper as one additional experiment in Li et al. (2019) and not a whole paper as such.\n2. Also, in the formulation in Eqn 2, the proposition 2 becomes a direct observation when \\pi becomes 0 or \\pi becomes 1. I would not call this as a proposition but a mere observation of Eqn 2.\n3. Thus, taking away proposition 1 (already proposed in Li et al. (2019)) and proposition 2 (which is a mere observation) I do not find any novelty in this paper.\n4. From an experiments perspective, Li et al. (2019) performed experiments in 4 datasets: Chinese Poems, MS COCO captions, Obama Speech, and EMNLP2017 WMT news. However, in this paper, results are shown in only two datasets - MS COCO captions and EMNLP2017 WMT news. Was it because that this paper was submitted in haste and/or the results in the other two datasets are not compelling enough to share?\n5. The result analysis are poor - the authors have shown only the numbers in the tables, while the interpretation on these numbers and the discussion is left to reviewers discretion. Also, there are no generated examples that the authors are showing in either of the datasets. The authors should further discuss and analyze the results, show generated examples, and explain success and failure cases and the reasons behind them.\n\nOverall, I find the novelty and the experimental analysis of the paper, very weak.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1xIAMv8Yr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed to use KL and reversed KL as its new objective function for text generation GAN training.\nHowever, this paper missed a lot important references. Basically, the authors only compare results with seqGAN and leakGAN. MaliGAN (https://arxiv.org/pdf/1702.07983.pdf), TextGAN (https://arxiv.org/pdf/1706.03850.pdf), etc. \nAlso, KL + reversed KL training method for GAN framework is first proposed in Symmetric VAE (https://arxiv.org/abs/1709.01846), and the Proposition 2 basically are the same as the Symmetric VAE paper.\n\nTherefore, I think this work is lack of novelty, and still need more time to work on.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a method (loss function) for training GAN model for generation of discrete text token generation. The aim of this loss method to control the trade off between quality vs diversity while generating the text data.

The paper is generally well written, but the experimental section is not overly good: Interpretation of the results is missing; error bars are missing. 
Decision: Reject