Year: 2019

Abstract:
The text generated by neural language models is not as good as the real text. This means that their distributions are different. Generative Adversarial Nets (GAN) are used to alleviate it. However, some researchers argue that GAN variants do not work at all. When both sample quality (such as Bleu) and sample diversity (such as self-Bleu) are taken into account, the GAN variants even are worse than a well-adjusted language model. But, Bleu and self-Bleu can not precisely measure this distributional discrepancy. In fact, how to measure the distributional discrepancy between real text and generated text is still an open problem. In this paper, we theoretically propose two metric functions to measure the distributional difference between real text and generated text. Besides that, a method is put forward to estimate them. First, we evaluate language model with these two functions and find the difference is huge. Then, we try several methods to use the detected discrepancy signal to improve the generator. However the difference becomes even bigger than before. Experimenting on two existing language GANs, the distributional discrepancy between real text and generated text increases with more adversarial learning rounds. It demonstrates both of these language GANs fail.

heading: 1 INTRODUCTION
text: Text generation by neural language models (LM), such as LSTM (Hochreiter & Schmidhuber, 1997) have given rise to much progress and are now used to dialogue generation (Li et al., 2017), machine translation (Wu et al., 2016) and image caption (Xu et al., 2015). However, the generated sentences are still poor in semantics or global coherence, even not perfect grammatically speaking (Caccia et al., 2019).
It means that the discrepancy between generated text and real text is large. One reason is the architecture and parameters’ number of LM itself (Radford et al., 2019; Santoro et al., 2018). Many researchers attribute it to the exposure bias (Bengio et al., 2015) because the LM is trained with a maximum likelihood estimate (MLE) and predicts the next word conditioned on words from the ground-truth during training. But it only conditions on the words generated by itself during reference.
Statistically, this discrepancy means the two distributional functions of real texts and generated texts is different. Reducing this distributional difference may be a practicable way to improve text generation.
Some researchers try to reduce this difference with GAN (Goodfellow et al., 2014). They use a discriminator to detect the discrepancy between real samples and generated samples, and feed the signal back to upgrade the generator (a LM). In order to solve the non-differential issue that arises by the need to handle discrete tokens, reinforcement learning (RL) (Williams, 1992) is adapted by SeqGAN (Yu et al., 2017), RankGAN (Lin et al., 2017), and LeakGAN (Guo et al., 2018). The Gumble-Softmax is also introduced by GSGAN (Jang et al., 2017) and RelGAN (Nie et al., 2019) to solve this issue. These language GANs pre-train both the generator (G) and the discriminator (D) before adversarial learning1. During adversarial learning, for each round, the G is trained several epochs and then, the D is trained tens of epochs. Learning stops when the model converges. Furthermore, considering the generated texts’ quality and diversity simultaneously (Shi et al., 2018),
1An exception is RelGAN which needs not pre-train D.
MaskGAN (Fedus et al., 2018), DpGAN (Xu et al., 2018), FMGAN (Chen et al., 2018) and RelGAN (Nie et al., 2019) are proposed. They evaluate the generated text with Bleu and self-Bleu (Zhu et al., 2018) or LM socre and reverse LM score (Cı́fka et al., 2018), and claim these GANs improve the performance of generator.
However recently questions have been rasied over these claims. Semeniuta et al. (2018) and Caccia et al. (2019) showed that via more precise experiments and evaluation, these considered GAN variants are defeated by a well-adjusted language model . d’Autume et al. (2019) trained language GANs from scratch, nevertheless, they only achieve the ”comparable” performance against LM. He et al. (2019) quantifies the exposure bias and concludes it is either 3 percent lower in performance or indistinguishable.
All the aforementioned methods treat GAN as a black box for evaluation. For those language GANs, there are several critical issues such as whether the D detects the discrepancy or not; the detected discrepancy is severe or not, the signals from D could improve the generator or not are still unclear. In this paper, we try to solve these problems via investigating GAN in both pre-training and the adversarial learning process. Theoretically analysing the signal from D, we obtain two metric functions to measure the distributional difference. With these two functions, we first measure the difference between the real text and the generated text by a MLE-trained language model (pretrain). Second, we try some methods to update generator with feedback signal from D, then, we use these metric functions to evaluate the updated generator. Finally, we analysis the existing language GANs during the adversarial learning with these two functions. All the code and data could be find https://github.com/.
Our contributions are as follows:
• We propose two metric functions to measure the distributional difference between real text and generated text. Besides that, a method is put forward to estimate them.
• Evaluated using these two functions, a number of experiment show there is an obvious discrepancy between the real text and the generated text even when it is generated by a well-adjusted language model.
• Although this discrepancy could be detected by D, the feedback signal from D can not improve G using existing methods.
• Experimenting on two existing language GANs, SeqGAN and RelGAN, the distributional discrepancy between real text and generated text increases with more adversarial learning rounds. It demonstrates both of these language GANs fail.

heading: 2 METHOD
text: In GAN, the generator Gθ implicitly defines a probability distribution pθ(x) to mimic the real data distribution pd(x).
min Gθ max Dφ
V (Dφ, Gθ) = Ex∼pd(x) [ logDφ(x) ] + Ex∼pθ(x) [ log ( 1−Dφ(x) )] (1)
We define Dφ to detect the discrepancy between pθ(x) and pd(x). We optimize Dφ as follow,
max Dφ V (Dφ, Gθ) = max Dφ
Ex∼pd [ logDφ(x) ] + Ex∼pθ [ log ( 1−Dφ(x) )] (2)
Assuming D∗φ(x) is the optimal solution for a given θ, according to (Goodfellow et al., 2014), there will be,
D∗φ(x) = pd(x)
pd(x) + pθ(x) (3)
We obtain two metric functions to measure this discrepancy.
D ∗ φ(x) ≥ 0.5, iif pd(x) ≥ pθ(x)
D∗φ(x) < 0.5, iif pd(x) < pθ(x) (4)
With it, the integration of density function could be transformed into statistic equation. Based on that, we could get a way which will be described in next, to compute the precise discrepancy.

heading: 2.1 APPROXIMATE DISCREPANCY
text: Let,
qd(x) = pd(x)
pd(x) + pθ(x) qθ(x) =
pθ(x)
pd(x) + pθ(x) (5)
So, qd(x) = p(x comes from real data|x), qθ(x) = p(x comes from generated data|x), qd(x) + qθ(x) = 1. With equation 5, we could get a constraint and an approximated measure function of distributional function. Figure 1(a) illustrates the relationship between qθ(x) and qd(x).
Let, ud = Ex∼pd(x) ( D∗φ(x) ) uθ = Ex∼pθ(x) ( D∗φ(x) ) (6)
They are two statistic equations who are the expectation of the D∗φ’s predictions on real text and on generated text respectively. According to the above equation, it is easy to get following equation.
1
2
[ ud + uθ ] = 0.5 (7)
It gives a constraint for Dφ converging to D∗φ. We should take this constraint into account when estimating the ideal function D∗φ. From equation 3, the process of optimizing the discriminator is make ud big and make uθ small. So, we could estimate the distributional discrepancy according to the following function.
Intuitively, using ud and uθ, we get a metric function to measure the discrepancy between pθ(x) and pd(x),
da = ud − uθ (8)
We call it approximate discrepancy. It is the subtraction of the average score of a well-trained discriminator (denoted as D̂φ) makes the predictions on real samples and on generated samples. It reflects the discrepancy between these two sets to some degree. From equation 5 and 6, we get equation 8,
da = ∫ [ qd(x)− qθ(x) ] pd(x)dx = Ex∼pd(x) [ qd(x)− qθ(x) ] (9)
Figure 1 (a) illustrates the discrepancy between two distributional functions qθ(x) and qd(x). Both of them are systematic to the line of q = 0.5. But it is not a complete measure because there is not only a positive part but also a negative part. A complete metric function is shown in next section.

heading: 2.2 ABSOLUTE DISCREPANCY
text: In order to precisely measure the discrepancy, we define ds,
ds = 1
2
∫ ∣∣pd(x)− pθ(x)∣∣dx (10)
The range of this function is 0 ∼ 1. The bigger of its value is, the discrepancy is more. When its value is zero, it means pd(x) ≡ pθ(x), namely there is no discrepancy. Fortunately, this function could be estimated by statistic method which is described by following equation. The proof is shown in appendix A.
ds = 1
2 [ E x∼pd(x) D∗φ(x)>0.5 ( 1 ) − E x∼pd(x) D∗φ(x)≤0.5 ( 1 ) + E x∼pθ(x) D∗φ(x)≤0.5 ( 1 ) − E x∼pθ(x) D∗φ(x)>0.5 ( 1 )]
(11)
With equation 11, we could more precisely estimate the discrepancy between pθ(x) and pd(x). Assuming the classification precision of D∗φ is a, then the error rate is b = 1 − a. According to equation 11, ds = a − b. So, the discrepancy between pθ(x) and pd(x) equals the classification precision of D∗φ minus its error rate.
2.3 USING D∗φ(x) TO IMPROVE Gθ
Given an instance x generated by Gθ, if D∗φ(x) is larger, it means the possibility of x is real data is larger. For an instance D∗φ(x) = 0.8, there will be pθ(x) < pd(x) according to equation 3. So, we should update Gθ to make the probability density pθ(x) increase. It may improve the performance of Gθ. Based on this, we could select out some generated instances by the value of D∗φ(x) to update the generator. In fact, we find it helpful to use the faked samples whose score are higher assigned by D∗φ. Experiment 4.3 shows the results.

heading: 3 IMPLEMENT PROCEDURE
text: The optimal function D∗φ is an ideal function which can only be statistically estimated by approximated function. We could design a function Dφ and sample from real data and generated data, then train Dφ according to equation 2. When it convergences, we get D̂φ. D̂φ is the approximated function of D∗φ. The degree of approximation is mainly determined by three factors: the structure and the parameters’ number of Dφ, the volume of training data, and the settings of hyper-parameters.
Based on the above analysis, we get two metric functions to measure the distributional discrepancy between dataset A and B. The specific implement procedure is as follow,
Step 1: Design a discriminator Dφ.
Step 2: The set A and B are respectively divided into training set DtrainA and DtrainB , validation set DdevA and DdevB , and test set DtestA and DtestB . The partition should be as equal an amount of instances as possible for classification training.
Step3: Dφ is optimized with DtrainA and DtrainB according to the equation 2. Validated with DdevA and DdevB , we could judge whether Dφ convergences or not and then get D̂φ.
Step4: According to equation 8 and 11, with two test datasets, we could estimate the discrepancy of two distributional density functions between dataset A and B. d̂s denotes the absolute discrepancy and d̂a denotes the approximate discrepancy respectively.
Generally speaking, there should be ds ≤ d̂s. Because D∗φ can not be obtained, it is hard to get the degree of the approximation ds to d̂s. Many research results have shown that the discriminators with deep neural networks are very powerful, and can even exceed human performance on some tasks such as image classification (He et al., 2016) and text classification (Kim, 2014). So, if the Dφ with CNN and attention mechanism is well trained, D̂φ will be a meaningful approximation of D∗φ. Therefore, we could obtain the meaningful approximation of ds and da via D̂φ.

heading: 4 EXPERIMENT
text: We select out SeqGAN and RelGAN as representatives for experiment and the benchmark datasets are also the same as theirs. Then, we show that the well trained discriminator D could measure the discrepancy between the real and generated texts, and then point out the existing GAN-based methods do not work. Finally, a third party discriminator is used to evaluate the performance of adversarial learning with the increment of training iterations.

heading: 4.1 DATASETS AND MODEL SETTINGS
text: Both SeqGAN and RelGAN are experimented on relative short sentences (COCO image caption) 2 and long sentences dataset (EMNLP2017 WMT news) 3. For the former dataset, sentences average length is about 11 words. There are total 4,682 word types and the longest sentence consists of 37 words. Both the training and test data contain 10,000 sentences. For the latter, the average length of sentences is about 20 words. There are total 5,255 word types and the longest sentence is consisted of 51 words. All training data, about 280 thousand sentences, is used and 10,000 sentences in test data. According to section 3, each test data divide into two parts. Half is validation set and the rest half is test set. We always generate the same amount of sentences to compare with the two test datasets respectively.
For these two models, all hyper-parameters including word embedding size, learning rate and dropout are set the same as their papers. For RelGAN, the standard GAN loss function (the nonsaturating version) is adapted because the relative standard loss which is used in (Nie et al., 2019) does not meet the constraints of equation 7. But, when measure RelGAN’s discrepancy during the adversarial stage, it own loss function is still relative standard loss. A critical hyper-parameters, temperature, is set 100 which is the best result in their paper. During the process of training D to obtain, we always train D 10,000 epochs and observe its performance on validation dataset.

heading: 4.2 THE DISTRIBUTIONAL DIFFERENCE IN PRE-TRAIN
text: We estimate the distributional difference caused by the MLE-based generators. We first train the generator for N epochs and then train D until it converges (it is trained 10 thousand epochs). For example, following (Nie et al., 2019), we trainG 150 epochs, and at that time, the PPL is the smallest value measured by validation set. Then, Dφ is trained following the procedure in section 3. Figure 2 shows the results on EMNLP dataset. From this figure, we can see that (1) Dφ convergences after
2http://cocodataset.org/ 3http://www.statmt.org/wmt17/
about 5,000 epochs. (2) There is always ud + uθ ≈ 1 everywhere. (3) When it convergence, the ud is 0.7 and uθ is 0.3. More results are illustrated in appendix B.1.
Considering the smoothed value on one batch rather than the prediction on the whole data, we use the convergence discriminator to predict on the all validation data and generated data 4. Table x summaries the discrepancy across two models and two datasets. It reflects the difference between real text and generated text is huge.
4.3 COULD THE DETECTED DISCREPANCY BY D̂φ IMPROVE THE GENERATOR?
In this section, we will explore whether the above discrepancy detected by D̂φ in pre-train could improve G. It should be noted that the D̂φ is well pre-trained. We select out the best pre-train epochs for G. It is updated according to the signals from the D̂φ. In order to verify the effect of those feedback signals, we generate many instances rather only several batch-size ones are used to adjust generator’s parameters.
Then, fixing G, we re-train D with 10,000 epochs to get a new convergence discriminator, named D̂′φ, for computing two distributional functions according to equation 9 and 11. Unfortunately, in the view of absolute discrepancy or approximated discrepancy, the discrepancy always overpass the original value which is computed in pre-train. This demonstrates that the generator is not improved yet. Figure 3 illustrates a comparison. More experimental results are shown in appendix B.2.
Besides following (Zhu et al., 2018), We propose a new methods train G. Rather than all the generated instances are used to update G, only the ones who are assigned relative low scores by D
4According to section 3, we sample generated instances as much as test instances.
are used. We denote it as HW. The reason is that we think the higher score instances maybe more informative than the lower ones. Regretfully, both of two methods fail. Table 3 list the discrepancy across two datasets. It again demonstrated that the absolute measure is necessary. Appendix B.3 show the results of several thresholds are set for selecting out more informative generated instances.

heading: 4.4 A THIRD PARTY DISCRIMINATOR EVALUATES THESE LANGUAGE GANS
text: In order to evaluate different adversarial learning’s GANs, we use a third party discriminator D3 which is a clone of the discriminator in its counterpart language GAN except parameters’ value. Given a round, we train D3 many epochs (making sure it convergence) with real text and the generated text at this round. Then, two distributional functions are computed according to its prediction. Figure 4 shows the result. In the view of both approximate discrepancy and absolute discrepancy,
the difference of the distribution on real text and generated text does not decrease with more adversarial learning rounds are adapted. Once again, it shows that the way of the existing language GANs could not improve text generation.

heading: 5 RELATED WORK
text: Many GAN-based models are proposed to improve traditional neural language model. SeqGAN (Yu et al., 2017) is the first one and try by to attack the non-differential issue by resorting RL. By applying policy gradient (Sutton et al., 2000) method, it optimizes the LSTM generator with rewards received through Monte Carlo (MC) sampling. Many researchers, such as RankGAN (Lin et al., 2017), MailGAN (Tong et al., 2017) follow this way although its in-effective in MC search. The RL-free model, for an example GSGAN (Jang et al., 2017), contains applying continuous approximating softmax function and working on latent continuous space directly. TextGAN (Salimans et al., 2016) adds Maximum Mean Discrepancy to the original objective of GAN based on feature matching. RelGAN (Nie et al., 2019) is a state-of-the-art model which uses relation memory (Santoro et al., 2018), which allowing for interactions between memory slots by using the self-attention mechanism (Vaswani et al., 2017). We select out SeqGAN and RelGAN as representatives for experiment. The results show that the adversarial learning does not work in both of them.
Caccia et al. (2019) first argues the current evaluation measures correlate with human judgment (Cı́fka et al., 2018) is treacherous. They furthermore propose temperature sweep which evaluates model at many temperature settings rather than only one. By using this metric, they find a welladjust language model could beat those considered language GANs. Semeniuta et al. (2018) and He et al. (2019) also argues GAN-based models are weak than LM, because they observe the impact of exposure bias is not severe. He et al. (2019) furtherly quantify the exposure bias by using conditional distribution. Neither designing a better metric nor showing the weakness of the language GANs, we try to investigate language GANs in mechanism and quantify the discrepancy between real texts and generated texts both in pre-train and adversarial learning.

heading: 6 CONCLUSION AND FUTURE WORK
text: We present two directly metric functions to measure the discrepancy between real text and generated text. It must be noted that they are independent of any text generation method including GANsbased. Numerous experiments show that this discrepancy dose exist. We try some methods to update the parameters of generator according to the detected discrepancy signals. Unfortunately, the distributional difference between real data and generated data does not decrease. It is hard to improve generator with these signals. Finally, We use a third part discriminator to evaluate the effectiveness of GAN and find with more adversarial learning epochs, the discrepancy increase rather than decreasing. It shows the existing language GANs do not work at all.

heading: A FORMULA INDUCTION
text: We show the derivation of Equation 11,
ds = 1
2 ∫ ∣∣pd(x)− pθ(x)∣∣dx = 1
2 [ ∫ pd(x)≥pθ(x) ( pd(x)− pθ(x) ) dx+ ∫ pd(x)<pθ(x) ( pθ(x)− pd(x) ) dx ] = 1
2 [ ∫ pd(x)≥pθ(x) pd(x)dx+ ∫ pd(x)<pθ(x) pθ(x)dx− ∫ pd(x)≥pθ(x) pθ(x)dx− ∫ pd(x)<pθ(x) pd(x)dx ] = 1
2 [ E x∼pd(x) pd(x)≥pθ(x) ( 1 ) + E x∼pθ(x) pd(x)<pθ(x) ( 1 ) − E x∼pθ(x) pd(x)≥pθ(x) ( 1 ) − E x∼pd(x) pd(x)<pθ(x) ( 1 )]
= 1
2
[ Ex∼pd(x)
z≥0.5
( 1 ) + Ex∼pθ(x)
z<0.5
( 1 ) − Ex∼pθ(x)
z≥0.5
( 1 ) − Ex∼pd(x)
z<0.5
( 1 )]
(12)
where z = pd(x)pd(x)+pθ(x) .

heading: B APPENDIX B
text: B.1 THE REST THREE EXAMPLES IN PRE-TRAIN.
B.2 THE DISCREPANCY COMPARE ON EMNLP BETWEEN PRE-TRAINING AND UPDATED LM.
B.3 THE COMPARE OF THE APPROXIMATE DISCREPANCY BETWEEN RANDOM AND HW.

heading: C GENERATED SENTENCES ON COCO IMAGE CAPTIONS DATASET

Review summaries:
This paper proposes an estimator to quantify the difference in distributions between real and generated text based on a classifier that discriminates between real vs generated text.  The methodology is however not particularly well motivated and the experiments do not convince me that this proposed measure is superior to other reasonable choices.  Overall, the writing also contains many grammatical errors and confusing at places.

This paper proposes two metrics to measure the discrepancy between generated text and real text, based on the discriminator score in GANs. Empirically, it shows that text generated by current text generation methods is still far from human-generated text, as measured by the proposed metric. The writing is a bit rough so sometimes it's hard to figure out what has been done. It's also unclear how the proposed metrics compare to simply using the discriminator for evaluation. Therefore, I'm inclined to reject the current submission.