Title: THE DETECTION OF DISTRIBUTIONAL DISCREPANCY FOR TEXT GENERATION Anonymous authors Paper under double-blind review

Abstract: The text generated by neural language models is not as good as the real text. This means that their distributions are different. Generative Adversarial Nets (GAN) are used to alleviate it. However, some researchers argue that GAN variants do not work at all. When both sample quality (such as Bleu) and sample diversity (such as self-Bleu) are taken into account, the GAN variants even are worse than a well-adjusted language model. But, Bleu and self-Bleu can not precisely measure this distributional discrepancy. In fact, how to measure the distributional discrepancy between real text and generated text is still an open problem. In this paper, we theoretically propose two metric functions to measure the distributional difference between real text and generated text. Besides that, a method is put forward to estimate them. First, we evaluate language model with these two functions and find the difference is huge. Then, we try several methods to use the detected discrepancy signal to improve the generator. However the difference becomes even bigger than before. Experimenting on two existing language GANs, the distributional discrepancy between real text and generated text increases with more adversarial learning rounds. It demonstrates both of these language GANs fail.

1. INTRODUCTION:

Text generation by neural language models (LM), such as LSTM (Hochreiter & Schmidhuber, 1997) have given rise to much progress and are now used to dialogue generation (Li et al., 2017) , machine translation (Wu et al., 2016) and image caption (Xu et al., 2015) . However, the generated sentences are still poor in semantics or global coherence, even not perfect grammatically speaking (Caccia et al., 2019) .

It means that the discrepancy between generated text and real text is large. One reason is the architecture and parameters' number of LM itself (Radford et al., 2019; Santoro et al., 2018) . Many researchers attribute it to the exposure bias (Bengio et al., 2015) because the LM is trained with a maximum likelihood estimate (MLE) and predicts the next word conditioned on words from the ground-truth during training. But it only conditions on the words generated by itself during reference.

Statistically, this discrepancy means the two distributional functions of real texts and generated texts is different. Reducing this distributional difference may be a practicable way to improve text generation.

Some researchers try to reduce this difference with GAN (Goodfellow et al., 2014) . They use a discriminator to detect the discrepancy between real samples and generated samples, and feed the signal back to upgrade the generator (a LM). In order to solve the non-differential issue that arises by the need to handle discrete tokens, reinforcement learning (RL) (Williams, 1992) is adapted by SeqGAN (Yu et al., 2017) , RankGAN (Lin et al., 2017) , and LeakGAN (Guo et al., 2018) . The Gumble-Softmax is also introduced by GSGAN (Jang et al., 2017) and RelGAN (Nie et al., 2019) to solve this issue. These language GANs pre-train both the generator (G) and the discriminator (D) before adversarial learning1 . During adversarial learning, for each round, the G is trained several epochs and then, the D is trained tens of epochs. Learning stops when the model converges. Furthermore, considering the generated texts' quality and diversity simultaneously (Shi et al., 2018) , MaskGAN (Fedus et al., 2018) , DpGAN (Xu et al., 2018) , FMGAN (Chen et al., 2018) and RelGAN (Nie et al., 2019) are proposed. They evaluate the generated text with Bleu and self-Bleu (Zhu et al., 2018) or LM socre and reverse LM score (Cífka et al., 2018) , and claim these GANs improve the performance of generator.

However recently questions have been rasied over these claims. Semeniuta et al. (2018) and Caccia et al. (2019) showed that via more precise experiments and evaluation, these considered GAN variants are defeated by a well-adjusted language model . d 'Autume et al. (2019) trained language GANs from scratch, nevertheless, they only achieve the "comparable" performance against LM. He et al. (2019) quantifies the exposure bias and concludes it is either 3 percent lower in performance or indistinguishable.

All the aforementioned methods treat GAN as a black box for evaluation. For those language GANs, there are several critical issues such as whether the D detects the discrepancy or not; the detected discrepancy is severe or not, the signals from D could improve the generator or not are still unclear. In this paper, we try to solve these problems via investigating GAN in both pre-training and the adversarial learning process. Theoretically analysing the signal from D, we obtain two metric functions to measure the distributional difference. With these two functions, we first measure the difference between the real text and the generated text by a MLE-trained language model (pretrain). Second, we try some methods to update generator with feedback signal from D, then, we use these metric functions to evaluate the updated generator. Finally, we analysis the existing language GANs during the adversarial learning with these two functions. All the code and data could be find https://github.com/.

Our contributions are as follows:

• We propose two metric functions to measure the distributional difference between real text and generated text. Besides that, a method is put forward to estimate them. • Evaluated using these two functions, a number of experiment show there is an obvious discrepancy between the real text and the generated text even when it is generated by a well-adjusted language model. • Although this discrepancy could be detected by D, the feedback signal from D can not improve G using existing methods. • Experimenting on two existing language GANs, SeqGAN and RelGAN, the distributional discrepancy between real text and generated text increases with more adversarial learning rounds. It demonstrates both of these language GANs fail.

2. METHOD:

In GAN, the generator G θ implicitly defines a probability distribution p θ (x) to mimic the real data distribution p d (x).

min

EQUATION

We define D φ to detect the discrepancy between p θ (x) and p d (x). We optimize D φ as follow,

EQUATION

Assuming D * φ (x) is the optimal solution for a given θ, according to (Goodfellow et al., 2014) , there will be,

EQUATION

We obtain two metric functions to measure this discrepancy.

EQUATION

With it, the integration of density function could be transformed into statistic equation. Based on that, we could get a way which will be described in next, to compute the precise discrepancy.

2.1. APPROXIMATE DISCREPANCY:

Let,

EQUATION

So, q d (x) = p(x comes from real data|x), q θ (x) = p(x comes from generated data|x), q d (x) + q θ (x) = 1. With equation 5, we could get a constraint and an approximated measure function of distributional function. Figure 1 (a) illustrates the relationship between q θ (x) and q d (x).

Let,

EQUATION

They are two statistic equations who are the expectation of the D * φ 's predictions on real text and on generated text respectively. According to the above equation, it is easy to get following equation.

EQUATION

It gives a constraint for D φ converging to D * φ . We should take this constraint into account when estimating the ideal function D * φ . From equation 3, the process of optimizing the discriminator is make u d big and make u θ small. So, we could estimate the distributional discrepancy according to the following function.

Intuitively, using u d and u θ , we get a metric function to measure the discrepancy between p θ (x) and p d (x),

d a = u d -u θ (8)

We call it approximate discrepancy. It is the subtraction of the average score of a well-trained discriminator (denoted as Dφ ) makes the predictions on real samples and on generated samples. It reflects the discrepancy between these two sets to some degree. From equation 5 and 6, we get equation 8,

EQUATION

Figure 1 (a) illustrates the discrepancy between two distributional functions q θ (x) and q d (x). Both of them are systematic to the line of q = 0.5. But it is not a complete measure because there is not only a positive part but also a negative part. A complete metric function is shown in next section.

2.2. ABSOLUTE DISCREPANCY:

In order to precisely measure the discrepancy, we define d s , The range of this function is 0 ∼ 1. The bigger of its value is, the discrepancy is more. When its value is zero, it means p d (x) ≡ p θ (x), namely there is no discrepancy. Fortunately, this function could be estimated by statistic method which is described by following equation. The proof is shown in appendix A.

EQUATION

d s = 1 2 E x∼p d (x) D * φ (x)>0.5 1 -E x∼p d (x) D * φ (x)≤0.5 1 + E x∼p θ (x) D * φ (x)≤0.5 1 -E x∼p θ (x) D * φ (x)>0.5 1 (11)

With equation 11, we could more precisely estimate the discrepancy between p θ (x) and p d (x).

Assuming the classification precision of D * φ is a, then the error rate is b = 1 -a. According to equation 11, d s = a -b. So, the discrepancy between p θ (x) and p d (x) equals the classification precision of D * φ minus its error rate.

2.3. USING D:

* φ (x) TO IMPROVE G θ Given an instance x generated by G θ , if D * φ (x)

is larger, it means the possibility of x is real data is larger. For an instance D * φ (x) = 0.8, there will be p θ (x) < p d (x) according to equation 3. So, we should update G θ to make the probability density p θ (x) increase. It may improve the performance of G θ . Based on this, we could select out some generated instances by the value of D * φ (x) to update the generator. In fact, we find it helpful to use the faked samples whose score are higher assigned by D * φ . Experiment 4.3 shows the results.

3. IMPLEMENT PROCEDURE:

The optimal function D * φ is an ideal function which can only be statistically estimated by approximated function. We could design a function D φ and sample from real data and generated data, then train D φ according to equation 2. When it convergences, we get Dφ . Dφ is the approximated function of D * φ . The degree of approximation is mainly determined by three factors: the structure and the parameters' number of D φ , the volume of training data, and the settings of hyper-parameters.

Based on the above analysis, we get two metric functions to measure the distributional discrepancy between dataset A and B. The specific implement procedure is as follow,

Step 1: Design a discriminator D φ .

Step 2: The set A and B are respectively divided into training set D trainA and D trainB , validation set D devA and D devB , and test set D testA and D testB . The partition should be as equal an amount of instances as possible for classification training.

Step3: D φ is optimized with D trainA and D trainB according to the equation 2. Validated with D devA and D devB , we could judge whether D φ convergences or not and then get Dφ .

Step4: According to equation 8 and 11, with two test datasets, we could estimate the discrepancy of two distributional density functions between dataset A and B. ds denotes the absolute discrepancy and da denotes the approximate discrepancy respectively.

Generally speaking, there should be d s ≤ ds . Because D * φ can not be obtained, it is hard to get the degree of the approximation d s to ds . Many research results have shown that the discriminators with deep neural networks are very powerful, and can even exceed human performance on some tasks such as image classification (He et al., 2016 ) and text classification (Kim, 2014) . So, if the D φ with CNN and attention mechanism is well trained, Dφ will be a meaningful approximation of D * φ . Therefore, we could obtain the meaningful approximation of d s and d a via Dφ .

4. EXPERIMENT:

We select out SeqGAN and RelGAN as representatives for experiment and the benchmark datasets are also the same as theirs. Then, we show that the well trained discriminator D could measure the discrepancy between the real and generated texts, and then point out the existing GAN-based methods do not work. Finally, a third party discriminator is used to evaluate the performance of adversarial learning with the increment of training iterations.

4.1. DATASETS AND MODEL SETTINGS:

Both SeqGAN and RelGAN are experimented on relative short sentences (COCO image caption)2 and long sentences dataset (EMNLP2017 WMT news)3 . For the former dataset, sentences average length is about 11 words. There are total 4,682 word types and the longest sentence consists of 37 words. Both the training and test data contain 10,000 sentences. For the latter, the average length of sentences is about 20 words. There are total 5,255 word types and the longest sentence is consisted of 51 words. All training data, about 280 thousand sentences, is used and 10,000 sentences in test data. According to section 3, each test data divide into two parts. Half is validation set and the rest half is test set. We always generate the same amount of sentences to compare with the two test datasets respectively. For these two models, all hyper-parameters including word embedding size, learning rate and dropout are set the same as their papers. For RelGAN, the standard GAN loss function (the nonsaturating version) is adapted because the relative standard loss which is used in (Nie et al., 2019) does not meet the constraints of equation 7. But, when measure RelGAN's discrepancy during the adversarial stage, it own loss function is still relative standard loss. A critical hyper-parameters, temperature, is set 100 which is the best result in their paper. During the process of training D to obtain, we always train D 10,000 epochs and observe its performance on validation dataset.

4.2. THE DISTRIBUTIONAL DIFFERENCE IN PRE-TRAIN:

We estimate the distributional difference caused by the MLE-based generators. We first train the generator for N epochs and then train D until it converges (it is trained 10 thousand epochs). For example, following (Nie et al., 2019) , we train G 150 epochs, and at that time, the PPL is the smallest value measured by validation set. Then, D φ is trained following the procedure in section 3. Figure 2 shows the results on EMNLP dataset. From this figure, we can see that (1) D φ convergences after about 5,000 epochs. (2) There is always u d + u θ ≈ 1 everywhere. (3) When it convergence, the u d is 0.7 and u θ is 0.3. More results are illustrated in appendix B.1. Considering the smoothed value on one batch rather than the prediction on the whole data, we use the convergence discriminator to predict on the all validation data and generated data4 . Table x summaries the discrepancy across two models and two datasets. It reflects the difference between real text and generated text is huge. In this section, we will explore whether the above discrepancy detected by Dφ in pre-train could improve G. It should be noted that the Dφ is well pre-trained. We select out the best pre-train epochs for G. It is updated according to the signals from the Dφ . In order to verify the effect of those feedback signals, we generate many instances rather only several batch-size ones are used to adjust generator's parameters.

WUDLQLQJHSRFKVIRU' 'LVFUHSDQF\:

Then, fixing G, we re-train D with 10,000 epochs to get a new convergence discriminator, named D φ , for computing two distributional functions according to equation 9 and 11. Unfortunately, in the view of absolute discrepancy or approximated discrepancy, the discrepancy always overpass the original value which is computed in pre-train. This demonstrates that the generator is not improved yet. Figure 3 are used. We denote it as HW. The reason is that we think the higher score instances maybe more informative than the lower ones. Regretfully, both of two methods fail. Table 3 list the discrepancy across two datasets. It again demonstrated that the absolute measure is necessary. Appendix B.3 show the results of several thresholds are set for selecting out more informative generated instances.

Table 2 : The compare between the absolute discrepancy in pre-training and G is updated by Dφ 's feedback signal. #samples denotes the amount of the generated data is used for updating the G. For example, 2S means the generated instances is two time as the amount of test data. Random denotes the existing way but the other row are the results according to HW. < 0.3 -0.5 means the generated instances whose score are between 0.3 and 0.5 assigned by D, are selected out.

Dataset:

COCO EMNLP pre-Train 0.42 0.57 #samples 0.1S 0.5S 1S 2S 5S 0.1S 0.5S 1S 2S 5S random 0.52 0.45 0.54 0.53 0.55 0.68 0.68 0.67 0.67 0.67 < 0.3 0.60 0.58 0.54 0.54 0.72 0.77 0.76 0.73 0.76 0.73 0.3 -0.5 0.44 0.56 0.58 0.60 0.55 0.68 0.67 0.66 0.63 0.66 0.5 -0.9 0.52 0.51 0.47 0.54 0.58 0.66 0.57 0.60 0.64 0.62 ≥ 0.9 0.68 0.46 0.50 0.44 0.58 0.66 0.60 0.59 0.60 0.62

4.4. A THIRD PARTY DISCRIMINATOR EVALUATES THESE LANGUAGE GANS:

In order to evaluate different adversarial learning's GANs, we use a third party discriminator D 3 which is a clone of the discriminator in its counterpart language GAN except parameters' value. Given a round, we train D 3 many epochs (making sure it convergence) with real text and the generated text at this round. Then, two distributional functions are computed according to its prediction.

Figure 4 shows the result. In the view of both approximate discrepancy and absolute discrepancy, the difference of the distribution on real text and generated text does not decrease with more adversarial learning rounds are adapted. Once again, it shows that the way of the existing language GANs could not improve text generation. 

DGYHUVDULDOURXQGV:



5. RELATED WORK:

Many GAN-based models are proposed to improve traditional neural language model. SeqGAN (Yu et al., 2017) is the first one and try by to attack the non-differential issue by resorting RL. By applying policy gradient (Sutton et al., 2000) method, it optimizes the LSTM generator with rewards received through Monte Carlo (MC) sampling. Many researchers, such as RankGAN (Lin et al., 2017) , MailGAN (Tong et al., 2017) follow this way although its in-effective in MC search. The RL-free model, for an example GSGAN (Jang et al., 2017) , contains applying continuous approximating softmax function and working on latent continuous space directly. TextGAN (Salimans et al., 2016) adds Maximum Mean Discrepancy to the original objective of GAN based on feature matching. RelGAN (Nie et al., 2019) is a state-of-the-art model which uses relation memory (Santoro et al., 2018) , which allowing for interactions between memory slots by using the self-attention mechanism (Vaswani et al., 2017) . We select out SeqGAN and RelGAN as representatives for experiment. The results show that the adversarial learning does not work in both of them. Caccia et al. (2019) first argues the current evaluation measures correlate with human judgment (Cífka et al., 2018) is treacherous. They furthermore propose temperature sweep which evaluates model at many temperature settings rather than only one. By using this metric, they find a welladjust language model could beat those considered language GANs. Semeniuta et al. (2018) and He et al. ( 2019) also argues GAN-based models are weak than LM, because they observe the impact of exposure bias is not severe. He et al. (2019) furtherly quantify the exposure bias by using conditional distribution. Neither designing a better metric nor showing the weakness of the language GANs, we try to investigate language GANs in mechanism and quantify the discrepancy between real texts and generated texts both in pre-train and adversarial learning.

6. CONCLUSION AND FUTURE WORK:

We present two directly metric functions to measure the discrepancy between real text and generated text. It must be noted that they are independent of any text generation method including GANsbased. Numerous experiments show that this discrepancy dose exist. We try some methods to update the parameters of generator according to the detected discrepancy signals. Unfortunately, the distributional difference between real data and generated data does not decrease. It is hard to improve generator with these signals. Finally, We use a third part discriminator to evaluate the effectiveness of GAN and find with more adversarial learning epochs, the discrepancy increase rather than decreasing. It shows the existing language GANs do not work at all.

A FORMULA INDUCTION:

We show the derivation of Equation 11,

EQUATION

where z = 

:

An exception is RelGAN which needs not pre-train D.

http://cocodataset.org/

http://www.statmt.org/wmt17/

According to section 3, we sample generated instances as much as test instances.

Figures and Tables:

(a) Approximate discrepancy illustration. (b) Precise discrepancy illustration.

Figure 1: The illustration of two measures. In (a), the yellow area denotes the negative and the green one denotes the positive. In (b), the half of the shadow area equals to the result of equation 11. Therefore, the range of it is 0 ∼ 2. Bigger value means more discrepancy.

The discrepancy between validation set and generated data. The orange lines denote the absolute discrepancy and the blue lines denote the approximate discrepancy. The discriminator's prediction. The orange lines denote the predictions on validation set and the blue lines denote on generated data.

Figure2: The results of pre-train SeqGAN' generator 80 epochs on EMNLP dataset. All the pale lines denote batch instances' discrepancy and the curve is the exponential moving average on this sampled batch for each epoch.

Figure 3: The compare of discrepancy between pre-train and the generator is updated with the feedback signals from Dφ which is obtained from pre-train. The vertical dash line represents the end of pre-training. SeqGAN's generator is pre-trained 80 epochs and the dataset is COCO.

Figure 4: A third party discriminator evaluates two GANs' performance variety on COCO.

Figure 5: The results of pre-train SeqGAN' generator 80 epochs on COCO dataset (upper), RelGAN on EMNLP (middle) and RelGAN on COCO(below). All the pale lines denote batch instances' discrepancy and the curve is the exponential moving average on this sampled batch for each epoch.

The discrepancy across two models and two datasets in pre-train. For both d a and d s , lower is better.

Table content: <table><tr><td>Model</td><td>Dataset</td><td>ds</td><td>da</td><td>Accuracy</td></tr><tr><td>SeqGAN</td><td colspan="3">COCO EMNLP 0.57 0.47 0.42 0.44</td><td>0.71 0.78</td></tr><tr><td>RelGAN</td><td colspan="3">COCO EMNLP 0.52 0.31 0.64 0.45</td><td>0.82 0.76</td></tr><tr><td colspan="5">4.3 COULD THE DETECTED DISCREPANCY BY Dφ IMPROVE THE GENERATOR?</td></tr></table>

The compare between the approximate discrepancy in pre-training and G is updated by Dφ 's feedback signal. #samples denotes the amount of the generated data is used for updating the G. For example, 2S means the generated instances is two time as the amount of test data. Random denotes the existing way but the other row are the results according to HW. < 0.3 -0.5 means the generated instances whose score are between 0.3 and 0.5 assigned by D, are selected out. -0.5 0.49 0.60 0.62 0.64 0.56 0.51 0.50 0.49 0.49 0.49 0.5 -0.9 0.57 0.55 0.51 0.58 0.62 0.49 0.46 0.45 0.45 0.44 ≥ 0.9 0.55 0.55 0.50 0.61 0.47 0.47 0.45 0.45 0.45 0.46

Table content: <table><tr><td>Dataset</td><td/><td>COCO</td><td/><td/><td/><td>EMNLP</td><td/></tr><tr><td>pre-Train</td><td/><td>0.44</td><td/><td/><td/><td>0.47</td><td/></tr><tr><td colspan="2">#samples 0.1S 0.5S</td><td>1S</td><td>2S</td><td>5S</td><td>0.1S 0.5S</td><td>1S</td><td>2S</td><td>5S</td></tr><tr><td>random</td><td colspan="8">0.57 0.50 0.58 0.58 0.60 0.55 0.52 0.50 0.52 0.53</td></tr><tr><td>&lt; 0.3</td><td colspan="8">0.65 0.62 0.59 0.60 0.75 0.64 0.62 0.61 0.61 0.61</td></tr><tr><td>0.3</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
This paper proposes an estimator to quantify the difference in distributions between real and generated text based on a classifier that discriminates between real vs generated text. The methodology is however not particularly well motivated and the experiments do not convince me that this proposed measure is superior to other reasonable choices. Overall, the writing also contains many grammatical errors and confusing at places. Major Comments: - There are tons of other existing measures of distributional discrepancy that could be applied to this same problem. Some would be classical approaches (eg. Kullback-Leibler or other f-divergence based on estimated densities, Maximum Mean Discrepancy based on a specific text kernel, etc) while others would be highly related to this work through their use of a classifier. Here*s just a few examples: i) Lopez-Paz & Oquab (2018). *Revisiting Classifier Two-Sample Tests *: https://arxiv.org/abs/1610.06545 ii) the Wasserstein critic in Wasserstein-GAN iii) Sugiyama et al (2012). *Density Ratio Estimation in Machine Learning* Given all these existing methods (I am sure there are many more), it is unclear to me why the estimator proposed in this paper should be better. The authors need to clarify this both intuitively and empirically via comparison experiments (theoretical comparisons would be nice to see as well). - The authors are proposing a measure of discrepancy, which is essentially useful as a two-sample statistical test. As such, the authors should demonstrate a power analysis of their test to detect differences between real vs generated text and show this new test is better than tests based on existing discrepancy measures. - The authors claim training a generator to minimize their proposed divergence is superior to a standard language GAN. However, the method to achieve this is quite convoluted, and straightforward generator training to minimize D_phi does not appear to work (the authors do not say why either). Minor Comments: - x needs to be defined before equation (1). - It is mathematically incorrect to talk about probability density functions when dealing with discrete text. Rather these should be referred to as probability mass functions, likelihoods, or distributions (not *distributional function* either).