The Detection of Distributional Discrepancy for Text Generation
Authors: Xingyuan Chen
Ping Cai
Peng Jin
Haokun Du
Hongjun Wang
Xinyu Dai
Jiajun Chen
Venue:  (2020)

Abstract: The text generated by neural language models is not as good as the real text. This means that their distributions are different. Generative Adversarial Nets (GAN) are used to alleviate it. However, some researchers argue that GAN variants do not work at all. When both sample quality (such as Bleu) and sample diversity (such as self-Bleu) are taken into account, the GAN variants even are worse than a well-adjusted language model. But, Bleu and self-Bleu can not precisely measure this distributional discrepancy. In fact, how to measure the distributional discrepancy between real text and generated text is still an open problem. In this paper, we theoretically propose two metric functions to measure the distributional difference between real text and generated text. Besides that, a method is put forward to estimate them. First, we evaluate language model with these two functions and find the difference is huge. Then, we try several methods to use the detected discrepancy signal to improve the generator. However the difference becomes even bigger than before. Experimenting on two existing language GANs, the distributional discrepancy between real text and generated text increases with more adversarial learning rounds. It demonstrates both of these language GANs fail. 

Under review as a conference paper at ICLR 2020
THE DETECTION OF DISTRIBUTIONAL DISCREPANCY FOR TEXT GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
The text generated by neural language models is not as good as the real text. This means that their distributions are different. Generative Adversarial Nets (GAN) are used to alleviate it. However, some researchers argue that GAN variants do not work at all. When both sample quality (such as Bleu) and sample diversity (such as self-Bleu) are taken into account, the GAN variants even are worse than a well-adjusted language model. But, Bleu and self-Bleu can not precisely measure this distributional discrepancy. In fact, how to measure the distributional discrep- ancy between real text and generated text is still an open problem. In this paper, we theoretically propose two metric functions to measure the distributional differ- ence between real text and generated text. Besides that, method put forward to estimate them. First, we evaluate language model with these two functions and find the difference is huge. Then, we try several methods to use the detected discrepancy signal to improve the generator. However the difference becomes even bigger than before. Experimenting on two existing language GANs, the dis- tributional discrepancy between real text and generated text increases with more adversarial learning rounds. It demonstrates both of these language GANs fail.
INTRODUCTION
Text generation by neural language models (LM), such as LSTM (Hochreiter & Schmidhuber 1997) have given rise to much progress and are now used to dialogue generation (Li et al. (2017), machine translation (Wu et al. 2016) and image caption (Xu et al. 2015). However, the generated sentences are still poor in semantics or global coherence, even not perfect grammatically speaking (Caccia let al. 2019).
means that the discrepancy between generated text and real text is large. One reason is the ar- chitecture and parameters' number of LM itself (Radford et al. 2019: Santoro al. 2018). Many researchers attribute it to the exposure bias (Bengio et al.] (015)) because the LM is trained with maximum likelihood estimate (MLE) and predicts the next word conditioned words from the ground-truth during training. But it only conditions on the words generated by itself during refer- ence.
Statistically, this discrepancy means the two distributional functions of real texts and generated texts different. Reducing this distributional difference may be practicable way to improve text generation.
Some researchers try reduce this difference with GAN (Goodfellow et al. 2014). They use discriminator to detect the discrepancy between real samples and generated samples, and feed the signal back to upgrade the generator (a LM). In order solve the non-differential issue that arises by the need to handle discrete tokens, reinforcement learning (RL) (Williams) 1992) is adapted by SeqGAN (Yu et al. 2017). RankGAN (Lin et al. (207)) and LeakGAN (Guo et al. 2018). The Gumble-Softmax is also introduced by GSGAN (Jag et al. (017) and RelGAN (Nie et al. 2019) to solve this issue. These language GANs pre-train both the generator (G) and the discriminator (D) before adversarial l During adversarial Icarning, for each round, the G is trained several epochs and then, the D is trained tens of epochs. Learning stops when the model converges. Furthermore, considering the generated texts' quality and diversity simultaneously (Shi al. 2018),
An exception is RelGAN which needs pre-train D.

---

Under review as a conference paper at ICLR 2020
MaskGAN (Fedus et al. 2018). DpGAN et al. 2018), FMGAN (Chen et al. 2018) and ReIGAN Nie et al. 2019) are proposed. They evaluate the generated text with Bleu and self Bleu (Zhu et al., 2018) LM socre and reverse LM score (Cifka et al. 2018). and claim these GANs improve the
performance of generator.
However recently questions have been rasied over these claims. [Semeniuta et al. (2018) and Cac- cia et al. (219)) showed that via more precise experiments and evaluation, these considered GAN variants defeated by a well-adjusted language model. d' d'Autume al. (2019) trained language GANs from scratch, nevertheless, they only achieve the "comparable" performance against LM. He al. (219) quantifies the exposure bias and concludes is either percent lower in performance or indistinguishabl..
All the aforementioned methods treat GAN black box for evaluation. For those language GANs, there are several critical issues such whether the D detects the discrepancy or not; the detected discrepancy is severe or not, the signals from D could improve the generator or not are still un- clear. In this paper, try to solve these problems via investigating GAN in both pre-training and adversarial learning process. Theoretically analysing the signal from D, we obtain two met- ric functions to measure the distributional difference. With these two functions, we first measure the difference between the real text and the generated text by a MLE-trained language model (pre- train). Second, we try some methods update generator with feedback signal from D, then, these metric functions to evaluate the updated generator. Finally, we analysis the existing language GANs during the adversarial learning with these two functions. All the code and data could be find https:/gittub.comm/
Our contributions are as follows:
We propose two metric functions measure the distributional difference between real text and generated text. Besides that, method is put forward to estimate them.
Evaluated using these two functions, a number of experiment show there is an obvious discrepancy between the real text and the generated text even when is generated by well-adjusted language model.
Although this discrepancy could be detected by D, the feedhack signal from D can not improve G using existing methods.
Experimenting two existing language GANs, SeqGAN and ReIGAN, the distributional discrepancy between real text and generated text increases with more adversarial learning rounds. It demonstrates both of these language GANs fail.
METHOD
In GAN, the generator Go implicitly defines probability distribution (1 mimic the real data distribution (z))
min max ((Do, Gg)= Exx)((x) [og))(())) Eo log (D(11)((=)))
Go D3
We define D& detect the discrepancy between (x and pa(x). We optimize D& as follow,
(oygD((e) + En [oyg( Da(xt))
max V(D Go) maa Enwoy
Do
Do
Assuming (z)z) is the optimal solution for given 0, according to Goodfellow et al. 2014), there will be,
Pa(x) Pa(x) + (x(
(3)
We obtain metric functions to measure this discrepancy.
2

---

Under review as a conference paper at ICLR 2020
if pa(x) 2 px(x) Dz(x) 2 0.5, D2(x) 0.5,
üf pa(x) < ps
With it, the integration of density function could be transformed into statistic equation. Based on that, could get way which will be described in next, compute the precise discrepancy.
2.1 APPROXIMATE DISCREPANCY
Let,
pa(
(x
(a(x)= Pa(x) po(x)
(0(x)
(5)
Pa(œ) + Po(a)
So, (a(x) p(r comes from real d)ta|z). (0(z) = p(x comes from generated datalz), qa(a) + (x) 1. With equation 5 could get constraint and an approximated measure function of distributional function. Figure aa) illustrates the relationship between qo (x) and qa(z).
Let.
Ud px (1)()))
vig (x) (Dz
They are two statistic equations who are the expectation of the D predictions on real and on generated text respectively. According to the above equation, it easy get following equation.
Ua + Un] = 0.5
gives constraint for D& converging Dr We should take this constraint into account when estimating the ideal function Dr From equation 3 the process of optimizing the discriminator is make u, big and make small. So, we could estimate the distributional discrepancy according to following function.
Intuitively, using and ug, we get metric function to measure the discrepancy between (a and pa(x).
d,
(8)
We call it approximate discrepancy. It is the subtraction of the average of well-trained discriminator (denoted as D.,) makes the predictions real samples and on generated samples. reflects the discrepancy between these two sets to some degree. From equation 5 and 6
we get
equation g
d_ = / [aa(t) -
-0a)) (o()))
= Bempe(()) [qa(x)) - [x))
Figure (a) illustrates the discrepancy between two distributional functions (x) and qa(x). Both of them are systematic to the line of q = 0.5. But is not a complete measure because there is not only positive part but also negative part. A complete metric function is shown in next section.
2.2 ABSOLUTE DISCREPANCY
In order to precisely measure the discrepancy, we define d..
1 dg = 2
|pa(x) (x
(10)
3

---

Under review as a conference paper at ICLR 2020
posals
0.4
0.9
0.35
0.8
0.7
0.25
0.5
0.2
0.4
0.15
0.3
0.1
2/00
0.2
M
2100
0.1
2
2
(a) Approximate discrepancy illustration.
(b) Precise discrepancy illustration.
Figure I: The illustration of two measures. In (a), the yellow area denotes the negative and the green one denotes the positive. In (b), the half of the shadow area equals the result of equation I1 Therefore, the range of it is O ~ 2. Bigger value means more discrepancy.
The range of this function ~ 1. The bigger its value is, the discrepancy is more. When its value is zero, it means pa(x)) = po(x), namely there is discrepancy. Fortunately, this function could be estimated by statistic method which is described by following equation. The proof is shown in appendix A.
d, 2
1
E zxpp(()
2>pp(()) z>pa(x) (1) E (w) (1) (=)=)00 (7(+)50.5 D1(x)
(1)
(1)
(11)
(1(+)
With equation 11 could precisely estimate the discrepancy between (e and pa(x). Assuming the classification precision of D a, then the error rate is b a. According to equation d, = a b. So, the discrepancy between pa(x) and pa(æ) equals the classification precision D minus its enror rate.
2.3 USING (x) IMPROVE G3
Given an instance x generated by Gg, if (* is larger, it means the possibility of a is real data is larger. For an instance (x) 0.8, there will be po(x) < Pa(x) according to equation 3 So, we should update Go to make the probability density p=(z) increase. It may improve the performance of Gg. Based on this, we could select out some generated instances by the value of D² (x)) to update the generator. In fact, find it helpful use the faked samples whose score are higher assigned by D Experiment 4.3 shows the results.
IMPLEMENT PROCEDURE
The optimal function D is an ideal function which can only be statistically estimated by approxi- mated function. We could design function D& and sample from real data and generated data, then train Dg according to equation 2 When it convergences, we get D3. Do is the approximated func- tion of Dr The degree of approximation mainly determined by three factors: the structure and the parameters' number of D., the volume of training data, and the settings of syper-pprramters.
Based on the above analysis, we get two metric functions to measure the distributional discrepancy between dataset A and B. The specific implement procedure is follow,

---

Under review as a conference paper at ICLR 2020
Step 1: Design discriminator De-
The set A and B are respectively divided into training set DivainA and D DrroinB. validation set DaevA and Dev and test set Diest and Drest The partition should be equal an amount of instances as possible for classification training.
Step 2:
Step3: Dø & is optimized with DtrainA and DtrainB according to the equation 2 Validated
with D deod and De we could judge whether D convergences or not and then get Dø-
Step4: According to equation 8 and with two test datasets, we could estimate the discrepancy of two distributional density functions between dataset A and B. d, denotes the absolute discrepancy and da denotes the approximate discrepancy respectively.
Generally speaking, there should be d, S dg. Because D can not be obtained, it is hard to get the degree of the approximation de to dz. Many research results have shown that the discriminators with deep neural networks very powerful, and even exceed human performance on some tasks such as image classification (He et al. 2016) and text classification (Kim] 2014). So, if the D with CNN and attention mechanism is well trained, Dr will be meaningful approximation of D Therefore, we could obtain the meaningful approximation of d, and do via Dos
4 EXPERIMENT
We select out SeqGAN and RelGAN representatives for experiment and the benchmark datasets are also the same as theirs. Then, we show that the well trained discriminator / could measure the discrepancy between the real and generated texts, and then point out the existing GAN-based methods do not work. Finally, a third party discriminator is used to evaluate the performance of adversarial learning with the increment of training iterations.
4.1 DATASETS AND MODEL SETTINGS
Both SeqGAN and ReIGAN are experimented on relative short sentences (COCO image caption) P and long sentences dataset (EMNLP2017 WMT news) [ length is about 11 words. There are total 4,682 word types and the longest sentence consists of 37 words. Both the training and test data contain 10,000 sentences. For the latter, the average length of sentences is about 20 words. There are total 5,255 word types and the longest sentence consisted of 51 words. All training data, about 280 thousand sentences, is used and 10,000 sentences in test data. According to section 3, each test data divide into two parts. Half is validation set and the rest half is test set. We always generate the same amount of sentences to compare with the two test datasets respectively.
For the former dataset, sentences average
For these two models, all hyper-parameters including word embedding size, learning rate and dropout are set the same as their papers. For RelGAN, the standard GAN loss function (the non- saturating version) adapted because the relative standard loss which is used in (Nie et 2019) does not meet the constraints of equation 7 But, when measure ReIGAN's discrepancy during the adversarial stage, own loss funetion is still relative standard loss. critical byper-parameters, temperature, is set 100 which is the best result in their paper. During the process of training D) to obtain, we always train 10,000 epochs and observe its performance on validation dataset.
4.2 THE DISTRIBUTIONAL DIFFERENCE IN PRE-TRAIN
We estimate the distributional difference caused by the MLE-based generators. We first train the generator for N epochs and then train D until it converges (it is trained 10 thousand epochs). For example, following (Nie et al. 2019), we train G 150 epochs, and at that time, the PPL is the smallest value measured by validation set. Then, D& is trained following the procedure in section Figure shows the results on EMNLP dataset. From this figure, we can see that (1) Dg convergences after
hhtt:://cocoddatasctoorg/ http://www.saamt.nngwwmtl
5

---

Under review as a conference paper at ICLR 2020
about 5,000 epochs. (2) There is always ua + ug everywhere. (3) When it convergence, the ut 0.7 and ug is 0.3. More results are illustrated in appendix B.1.
0.9
The discriminator" prediction
0.9
0.8
approx
0.8
Discrepancy
0.6
0.6
0.6
0.5
0.3
0.2
0.2
4000
8000
training epochs for D
training cpochs for D
(a) The discrepancy between validation set and gen- crated data. The orange lines denote the absolute dis- crepancy and the blue lines denote the approximate discrepancy.
(b) The discriminator's 's prediction. The orange lines denote the predictions validation and the blue lines denote on generated data.
Figure 2: The results of pre-train SeqGAN' generator 80 epochs on EMNLP dataset. All the pale lines denote batch instances' discrepancy and the curve is the exponential moving average on this sampled batch for each epoch.
Considering the smoothed value on one hatch rather than the prediction on the whole data, we use the convergence discrininator to predict the all validation data and generated data | Table summaries the discrepancy across two models and two datasets. It reflects the difference between real text and generated text is huge.
Table 1: The discrepancy across two models and two datasets in pre-train. For both da and d,, lower better.
Model
Dataset
dg
da
Accuracy
COCO
0.42
0.44
0.71
SeqGAN
EMNLP
0.57
0.47
0.78
COCO
0.64
0.45
0.82
ReIGAN
EMNLP
0.52
0.31
0.76
4.3 COULD THE DETECTED DISCREPANCY BY D& IMPROVE THE GENERATOR?
In this section, we will explore whether the above discrepancy detected by Do in pre-train could improve G. It should be noted that the D& is well pre-trained. We select out the best pre-train epochs for G. It is updated according to the signals from the Dz In order to verify the effect of those feedback signals, we generate many instances rather only several batch-size ones are used to adjust generator's parameters.
Then, fixing G, we re-train with 10,000 epochs to get a new convergence discriminator, named D' for computing two distributional functions according to equation 9 and Unfortunately, in the view absolute discrepancy or approximated discrepancy, the discrepancy always overpass the original value which is computed in pre-train. This demonstrates that the generator is not improved yet. Figure illustrates a comparison. More experimental results are shown in appendix B.2.
Besides following (Zhu et al.) 2018), We propose new methods train G. Rather than all the gen- erated instances are used to update G, only the ones who are assigned relative low scores by D
AAcorrding section we sample generated instances as much instances.

---

Under review as a conference paper at ICLR 2020
absolute discropancy
2.4
-4.2
traiming epochs
spproximmtee discrepancy
2.4
0.0
20000
training epochs
Figure 3: The compare of discrepancy between pre-train and the generator updated with the feedback signals from D, which is obtained from pre-train. The vertical dash line represents the end of pre-training. SeqGAN's generator is pre-trained 80 epochs and the dataset is COCO.
used. We denote it HW. The reason is that think the higher score instances maybe more informative than the lower ones. Regretfully, both of two methods fail. Table Ilist the discrepancy across two datasets. It again demonstrated that the absolute measure is necessary. Appendix B.3 show the results of several thresholds are set for selecting out more informative generated instances.
Table 2: The compare between the absolute discrepancy in pre-training and G updated by D& feedback signal. #samples denotes the amount of the generated data is used for updating the G. For example, 2S means the generated instances is two time as the amount of test data. Random denotes the existing way but the other instances whose score are between 0.3 and 0.5 assigned by D, are selected out.
the results according to HW. < 0.3 0.5 means the generated
Dataset
COCO
EMNLP
pre- Tre-Train
0.42
0.57
#samples
0.IS
0.5S
IS
2S
5S
0.1S
0.5S
IS
2S
5S
random
0.52
0.45
0.54
0.53
0.55
0.68
0.68
0.67
0.67
0.67
< 0.3
0.60
0.58
0.54
0.54
0.72
0.77
0.76
0.73
0.76
0.73
0.3 0.5
0.44
0.56
0.58
0.60
0.55
0.68
0.67
0.66
0.63
0.66
0.5 0.9
0.52
0.51
0.47
0.34
0.58
0.66
0.57
0.60
0.64
0.62
2 0.9
0.68
0.46
0.50
0.44
0.58
0.66
0.60
0.59
0.60
0.62
4.4 THIRD PARTY DISCRIMINATOR EVALUATES THESE LANGUAGE GANs
In order to evaluate different adversarial learning's GANs, which a clone of the discriminator in its counterpart language GAN except parameters' value. Given round, ated text at this round. Then, two distributional functions are computed according to its prediction. Figure shows the result. In the view of both approximate discrepancy and absolute discrepancy,
third party discriminator Da
train Dy many epochs (making sure it convergence) with real text and the gener-

---

Under review as a conference paper at ICLR 2020
the difference of the distribution real text and generated text does not decrease with more adver- sarial learning rounds are adapted. Once again, it shows that the way of the existing language GANs could not improve text generation.
0.9
0.9
discrepency
discrepancy
0.7
0.6
0.6
0.4
0.4
absolatc approcimais
0.3
apprstmme
0.3
100 200 500 700 R001000
30 100 400 600 700000 5001
adverrarial rounds
adversarial rounds
(a) SeqGAN
(b) ReIGAN
Figure 4: A third party discriminator evaluates two GANs' performance variety on COCO.
5 RELATED WORK
Many GAN-based models are proposed to improve traditional neural language model. SeqGAN Yu al. 2017) is the first one and try by to attack the non-differential issue by resorting RL. By applying policy gradient (uttonn et al. 2000) method, it optimizes the LSTM generator with rewards received through Monte Carlo (MC) sampling. Many researchers, such as RankGAN (Lin
2017).
MailGAN (Tong et al. 2017) follow this way although its in-effective in MC search. The Rrefree model, for an example GSGAN (Jang et al., 2017), contains applying continuous approximating softmax function and working latent continuous space directly. TexIGAN (Salimans et al. 2016) adds Maximum Mean Discrepancy to the original objective of GAN based on feature matching. RelGAN (Nic et al. 2019) is state-of-the-art model which uses relation memory (Santoro et al. 2018), which al lowing for interactions between memory slots by using the self-attention mechanism Vaswani al.) 2017). We select out SeqGAN and ReIGAN representatives for experiment. The results show that the adversarial learning does not work in both of them.
Caccia al. 2019 first argues the current evaluation measures correlate with human judgment (Cifka et al.) 2018) is treacherous. They furthermore propose temperature sweep which evaluates model many temperature settings rather than only one. By using this metric, they find well- adjust language model could beat those considered language GANs. Semeniuta et al. (018)) and He al. (2019) also argues GAN-based models are weak than LM, because they observe the impact of exposure bias is not severe. He et al. (2019) furtherly quantify the exposure bias by using conditional distribution. Neither designing better metric nor showing the weakness of the language GANs, we try to investigate language GANs in mechanism and quantify the discrepancy between real texts and generated texts both pre-train and adversarial learning.
CONCLUSION AND FUTURE WORK
We present two directly metric functions to measure the discrepancy between real text and generated text. It must be noted that they are independent of any text generation method including GANs- based. Numerous experiments show that this discrepancy dose exist. We try some methods to update the parameters of generator according the detected discrepancy signals. Unfortunately, the distributional difference between real data and generated data does not decrease. It is hard to improve generator with these signals. Finally, We use third part discriminator to evaluate the effectiveness of GAN and find with more adversarial learning epochs, the discrepancy increase rather than decreasing. It shows the existing language GANs do work at all.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In NeurIPS, 2015.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. arXiv preprint arXiv: 1 1.02549, 2019.
Liqun Chen, Shuyang Dai, Chenyang Tao, Dinghan Shen, Zhe Gan, Haichao Zhang, Yizhe Zhang, and Lawrence Carin. Adversarial text generation via Srature-mover 's distance. NeurIPS, 2018.
Ondrej Cifka, Aliaksei Severyn, Enrique Alfonseca, and Katja Filippova. Eval all, trust a few, do wrong to none: Comparing sentence generation models. CoRR, Abs/1804.07972, 2018. URL
Cyprien d' Autume, Mihaela Rosca, Jack Rae, and Shakir Mohamed. Training language gans from scratch. In NewrIPS, 2019.
William Fedus, lan Goodfellow, and Andrew M. Dai. Maskgan: Better text generation via filling in the ------ In ICLR, 2018.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Xu Bing, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
Jiaxian Guo, Sidi Lu, Cai Han, Weinan Zhang, and Jun Wang. Long text generation via adversarial training with leaked information. AAAI, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Tianxing He, Jingzhao Zhang, Zhiming Zhou, and James Glass. Quantifying exposure bias for neural language generation. arXiv preprint arXiv:1905.1061 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzation with gumbel- softmaxx In ICLR, 2017.
Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014.
Jiwei Li, Will Monroe, Tianlin Shi, Sébatian Jean, Alan Ritter, and Jurafsky Dan. Adversarial learning for neural dialogue generation. In EMNLP, 2017.
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming Ting Sun. Adversarial ranking for language generation. arXiv preprint arXiv: 700. 10929, 2017.
Weili Nie, Narodytska Nina, and Ankit Patel. Relgan: Relational generative adversarial networks for text generation. In ICLR, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, 2019.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NewrlPS, 2016.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rac, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. In NeurIPS, 2018.
Stanislau Semeniuta, Aliaksei Severyn, and Sylvain Gelly. On accurate evaluation of gans for lan- guage generation. arXiv preprint arXiv:1806.049.. 2018
Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Toward diverse text generation with inverse reinforcement learning. In IJCAI, pp. 4361-4366 2018.

---

Under review as a conference paper at ICLR 2020
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NeurIPS, pp. 1057-1063, 2000.
Che Tong, Yanran Li, Ruixiang Zhang, R Devon Hjelm, and Yoshua Bengio. Maximun-likelihood augmented discrete generative adversarial networks. arXiv preprint arXiv: 17X.::/702.07983, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Ronald J Williams. Simple statistical learning. Machine learning, 8(3-4)2229-25 1992.
algorithms for connectionist reinforcement
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Cao Yuan, Gao Qin, and Klaus Macherey. Google's ' neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 2016.
Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. Diversity-promoting gan: A cross-entropy based generative adversarial network for diversified text generation. EMNLP, 2018.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. arXiv preprini 1rXiv:/50..03000 2015.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI, 2017.
Yaoming Zhu, Sidi Lu, Zheng Lei, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yu Yong. Texygen: A benchmarking platform for text generation models, In SIGIR, 2018
10

---

Under review as a conference paper at ICLR 2020
A
FORMULA INDUCTION
We show the derivation of Equation 11
1
/ (pa(z) po(x))dz
d
2
1
/
(pa(z)-Po(aa) - po( (po=)) dz + Snacessnntts (polx) - p)(x))) dz
2
pa(o)2pp()))
1
/
pp(())2p(()) po(x))d f !pstercpptos pa(x)dr
Pa(x)ddo+
polx)do
2
pa(O)22p(1))
pp((e)<po((o)
E xmpa(x)
+ E xmpp(()) (1) - E xmpa(
(1) - EE z>pala)
(1)
2
palm)2po(z)
Re(z)<po(1)
pa(2)22o(m))
<p(())
1
E Pe)pp(()
(1)
R Pxppp(()
(1)
Riewpa(x) (w) (1) E (p))
(1)
2
220.5
x00.
220.5
005
(12)
where =
Pa(ar)
(m)

---

Under review as a conference paper at ICLR 2020
B APPENDIX B
B.1
THE REST THREE EXAMPLES IN PRE-TRAIN.
0.9
absulute
The discriminator prediction
0.9
0.8
0.8
Discreponcy
0.6
0.5
0.5
0.3
0.2
0.2
0.1
training epochs for D
training cpochs for
abeol Lte
discriminator's 's prediction
vox lidec on
0.8
0.7
0.7
Discrepaney
0.6
0.6
0.4
0.4
0.3
0.3
0.0
0.0
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
training epochs
training epochs for
0.9
The discriminator" 8 prediction
0.9
0.8
appr on wote
0.8
generat on
0.7
0.7
Diserepancy
0.5
0.5
0.4
0.4
0.2
0.1
0.1
0.0
2000
4000
6002
18000
10000
2002
4000
6583
8000
10020
training epochs for D
training epochs for D
Figure 5: The results of pre-train SeqGAN' generator 80 epochs COCO dataset (upper), RelGAN on EMNLP (middle) and ReIGAN on COCO(below). All the pale lines denote batch instances' discrepancy and the curve is the exponential moving average on this sampled batch for each epoch.
12

---

Under review as a conference paper at ICLR 2020
B.2
THE DISCREPANCY COMPARE ON EMNLP BETWEEN PRE-TRAINING AND UPDATED LM.
absolute discrepancy
0.6
0.4
0.2
0.0
-0.2
2500
5000
7500
10000
12500
15000
17500
20000
training epochs
approxxmaaee discrepancy
0.6
0.4
2500
5000
10000
12500
15000
20000
training epochs
Figure The compare of discrepancy between pre-train and the generator updated with the feedback signals from D2 which is obtained from pre-train. The vertical dash line represents the end of pre-training. SeqGAN" generator is pre-trained 80 epochs and the dataset is EMNLP.
B.3 THE COMPARE OF THE APPROXIMATE DISCREPANCY BETWEEN RANDOM AND HW.
Table 3: The compare between the approximate discrepancy in pre-training and G is updated by
D& 's feedhack signal. #samples denotes the amount of the generated data is used for updating the G. For example, 2S means the generated instances is two time as the amount of test data. Random denotes the existing way but the other row are the results according HW. < 0.3 0.5 means the generated instances whose score are between 0.3 and 0.5 assigned by D, are selected out.
Dataset
COCO
EMNLP
pre- Train
0.44
0.47
#samples
0.15
0.5S
IS
2S
5S
0.1S
0.5S
IS
2S
5S
random
0.57
0.50
0.58
0.58
0.60
0.55
0.52
0.50
0.52
0.53
< 0.3
0.65
0.62
0.59
0.60
0.75
0.64
0.62
0.61
0.61
0.61
0.3 0.5
0.49
0.60
0.62
0.64
0.56
0.51
0.50
0.49
0.49
0.49
0.5 0.9
0.57
0.55
0.51
0.58
0.62
0.49
0.46
0.45
0.45
0.44
2 0.9
0.55
0.55
0.50
0.61
0.47
0.47
0.45
0.45
0.45
0.46
13

---

Under review as a conference paper at ICLR 2020
C GENERATED SENTENCES ON COCO IMAGE CAPTIONS DATASET
Table Generated sentences who are scored 0.9 or higher by Do at the end of pre-training. Obvi- ously, they are better the next sentences listed in table 5
the ground and sink are under the window of pink covered pot. a toilet sitting next to a toilet next green organized kitchen.
the bathroom contains mop that rolls of flowers next to the vanity do two sinks
a vehicle trailer meeting a corner of street.
a small dog is sitting in purse with text.
two men are riding an outside of the window of a street
a porcelain tea pot by window
a humongous jumbo jet with chain chair on the ground
large white bus some luggage onto one the other smiling front tire. a wooden ship with rusted heater flying high in sky.
the side outside with giraffe doors
the bathroom has a pedestal table with pots on bathroom with wooden cabinets and washer, curtain and bottles growing on the flooring. the side of a person looks at a woman on the back seat
a bottle of whiskey in the bathroom with broken toilet
a ramp extends to the top of an air mattress.
a photograph of palm wall in the glass lap top.
a smiley face stand with 4 per gallon..
several bicycles parked stuffed
display using it.
two dogs top of a a white airplane flying above someone with his friend parked behind it terminal. a bathroom water from shower with cobbler and pink tile walls.
an amusement table..
a toilet is bathroom with multiple monitors paper.
a red fire hydrant displaying the woman in park and many headlights.
a tiled bathroom with a claw view of toilet and soap dispenser a group of ripe bananas in their hands on the back seat.
a motorcycle parked next to stone building next to the road an airplane flies low to people aboard a boat with a cemetery a frisbee. a bath room with toilet behind it
a homemade cake in home bathroom has sink a, shower, and window.
a truck is shown as the parking lot corner
a bathroom is lit with foods such as to a web job graze.
porcelain toilet in blue toilet seat a simple white bathroom with a large white fridge, and shower a large delta passenger airplane flying through cloudy sky a bathroom with a mirror, dishwasher dishwasher, and tub/shower. a blue plane is walking as seen all towards the usual number of tonic. two adorable chubby dogs in a group of pots
four airplanes sitting on top of a building in the blue sky
horse is racing on his motorcycle while maintenance are walking next to the river
an image of a old bathroom with bottle of spirits suggest tuscan decor, laptop an outdoor art bus is facing toward another snowboarder
a kitchen filled with in tables and two computer monitors...
this is an image of motorcycle parked down by bench. a crowd of people sitting on a bench next to street
an electronic cat in mens restroom next to multiple dark and pans along under screen table.
two woman in her phone next to a vehicle..
a person riding motor moped at playground..
two dirty dog standing in front of a car.
a white bathtub sits next mirror in a small closed.
14

---

Under review as a conference paper at ICLR 2020
Table 5: Generated sentences who are scored 0.1 or lower by Do at the end of pre-training, Obvi-
ously, they are better the next sentences listed in table.
a white metal structure with a backpack on top.. a bath tub next to the toilet in middle of nice plate.
rams lights and need onto the outside a zoo
airplane parked the ground in front of building.
a white vw car passing in front of a car
an airplane flying over airport terminal.
two people riding bikes on an outdoor from police officer.
an old propeller airplane parked off.
a man standing down by side on street.
a image of toilet, mirror and painting on the wall
two toilets walking lipstick television
two doves sit on a bathroom counter with wooden cabinets and a bucket. . this enclosure has white toilet next to the sink. the dining area is wide two people riding her back to carve above them
two men stand with all line of street using indoors
a boy wearing travel and his motorcycle outside town outside.
person brushing her teeth with her hair holding stove.
a white bathtub sitting in a bathroom with no cabinets.
a bath room with counter preparing food
a man holding up market stand by the ground
a kitchen has large round glass serving and cabinets..
a woman standing outside of a black car
a bathroom has an island in the middle granite glass a large passenger jet flying over airport next to the street.
a tiger cat sitting on top of a window looking very clean.
bicyclists holding flip phone next the aircraft.
a walk opened in the bathroom with a jungle theme.
the sink has white appliances with child.
a kitchen with a chrome toilet next to bathtub.
man standing in blue park looking sidewalk next a brown horse against group of electrical boxes. an old parked motorcycle with its kickstand down to
messy bathroom with tub, in the sink and the mirror with privacy.
a an open kitchen tucked in public kitchen.
a white brown kitchen with bowls on counter top counter.
a woman standing in red liquid under a small water fountain.
a view above the toilet roll sink with a stove
a couple of chefs up in a kitchen preparing food
a bathroom with a toilet, and mirrors from its reflection in a large sink.
a bath room with tub and refrigerator.
a group of traffic light with people skiing in the mist two dogs are huddled together screen corner on wall
wild animals grazing in the center of a blue sky. woman holds spoon by off road
a kitchen with toiletries in the just darts. motor sandwich and ride turn across a road
a man crossing a traffic in an oriental city
women turning her phone food prepares food
two stuffed animals are laptop. dishwasher, and sink..
a toilet in front of window a white room.
large woman. on snowboards at an open purse an image of a sink trash can
15


Reviews:
[{"review_id": "SJlBb0_QYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an estimator to quantify the difference in distributions between real and generated text based on a classifier that discriminates between real vs generated text.  The methodology is however not particularly well motivated and the experiments do not convince me that this proposed measure is superior to other reasonable choices.  Overall, the writing also contains many grammatical errors and confusing at places.\n\nMajor Comments:\n\n- There are tons of other existing measures of distributional discrepancy that could be applied to this same problem.  Some would be classical approaches (eg. Kullback-Leibler or other f-divergence based on estimated densities, Maximum Mean Discrepancy based on a specific text kernel, etc) while others would be highly related to this work through their use of a classifier.  Here's just a few examples: \n\ni) Lopez-Paz & Oquab (2018). \"Revisiting Classifier Two-Sample Tests\n\": https://arxiv.org/abs/1610.06545 \nii) the Wasserstein critic in Wasserstein-GAN\niii) Sugiyama et al (2012). \"Density Ratio Estimation in Machine Learning\"\n\nGiven all these existing methods (I am sure there are many more), it is unclear to me why the estimator proposed in this paper should be better. The authors need to clarify this both intuitively and empirically via comparison experiments (theoretical comparisons would be nice to see as well).\n\n- The authors are proposing a measure of discrepancy, which is essentially useful as a two-sample statistical test.  As such, the authors should demonstrate a power analysis of their test to detect differences between real vs generated text and show this new test is better than tests based on existing discrepancy measures.\n\n- The authors claim training a generator to minimize their proposed divergence is superior to a standard language GAN. However, the method to achieve this is quite convoluted, and straightforward generator training to minimize D_phi does not appear to work (the authors do not say why either).\n\n\nMinor Comments:\n\n- x needs to be defined before equation (1). \n\n- It is mathematically incorrect to talk about probability density functions when dealing with discrete text. Rather these should be referred to as probability mass functions, likelihoods, or distributions (not \"distributional function\" either). \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HyxQWDWtYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes two metrics to measure the discrepancy between generated text and real text, based on the discriminator score in GANs. Empirically, it shows that text generated by current text generation methods is still far from human-generated text, as measured by the proposed metric. The writing is a bit rough so sometimes it's hard to figure out what has been done. It's also unclear how the proposed metrics compare to simply using the discriminator for evaluation. Therefore, I'm inclined to reject the current submission.\n\nApproach:\n- The proposed metric essentially relies on the learned discriminator to measure the closeness of generated text vs real text, based on the strong assumption that the learned discriminator is near-optimal. It has been previously shown that learning a classifier from generated and real text does not generalize well (Lowe et al, 2017, Chaganty et al, 2018).\n- What's the advantage of the proposed metric, compared to existing ones, e.g. KL divergence, total variation etc.?\n\nExperiments:\n- What's the accuracy of the learned discriminators? The discrepancy could be due to both data difference and classification error.\n\nMinor:\nBleu -> BLEU\n\nReference:\nTowards an automatic turing test: Learning to evaluate dialogue responses. R. Lowe, M. Noseworthy, I. V. Serban, N. Angelard- Gontier, Y. Bengio, and J. Pineau. 2017.\nThe price of debiasing automatic metrics in natural language evaluation. A. Chaganty, S. Mussmann, and P. Liang. 2018. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1e5DF0aKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper argues that text generated by existing neural language models are not as good as real text and proposes two metric functions to measure the distributional difference between real text and generated text. The proposed metrics are tried on language GANs but fail to produce any improvement.\n\nMajor issues:\n\nThis manuscript is poorly organized and the introduction is not well-written. It’s true that generating text from random noise vector remains a challenging problem, but sequence-to-sequence models for machine translation and question answering have achieved tremendous successes. The description in the first paragraph about neural language models is not accurate. \n\nThere are numerous grammar issues and mis-spellings. For e.g., pp. 1: “RelGAN which needs not...”, pp. 2: “We analysis…”, “could be find…”, pp 3: “equation 8” should be “equation 9”...\n\nThe proposed metrics are also questionable. Eq. 3 on page 2 holds for any x sampled from the distribution, not just for a single data point. To test the effectiveness of a good metric, extensive experiments on toy datasets such as MNIST, CIFAR10, and synthetic datasets should be conducted. This paper mixes text generation and proposed metrics together. The claimed failure experiments make the proposed metrics even more questionable.\n\nIn summary, the presentation and the organization of this paper should be significantly improved for submission. The proposed metrics are questionable and should be thoroughly tested on synthetic and toy datasets before deploying it for text generation.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a novel metric to detect distributional discrepancy for text generation models and argue that these can be used to explain the failure of GANs for language generation tasks. The reviewers found significant deficiencies with the paper, including:

1) Numerous grammatical errors and typos, that make it difficult to read the paper.

2) Mischarcterization of prior work on neural language models, and failure to compare with standard distributional discrepancy measures studied in prior work (KL, total variation, Wasserstein etc.). Further, the necessity of the complicated procedure derived by the authors is not well-justified.

3) Failure to run experiments on standard banchmarks for image generation (which are much better studied applications of GANs) and confirm the superiority of the proposed metrics relative to standard baselines. 

The reviewers were agreed on the rejection decision and the authors did not participate in the rebuttal phase.

I therefore recommend rejection.
Decision: Reject