The Detection of Distributional Discrepancy for Text Generation

1 INTRODUCTION . Text generation by neural language models ( LM ) , such as LSTM ( Hochreiter & Schmidhuber , 1997 ) have given rise to much progress and are now used to dialogue generation ( Li et al. , 2017 ) , machine translation ( Wu et al. , 2016 ) and image caption ( Xu et al. , 2015 ) . However , the generated sentences are still poor in semantics or global coherence , even not perfect grammatically speaking ( Caccia et al. , 2019 ) . It means that the discrepancy between generated text and real text is large . One reason is the architecture and parameters ’ number of LM itself ( Radford et al. , 2019 ; Santoro et al. , 2018 ) . Many researchers attribute it to the exposure bias ( Bengio et al. , 2015 ) because the LM is trained with a maximum likelihood estimate ( MLE ) and predicts the next word conditioned on words from the ground-truth during training . But it only conditions on the words generated by itself during reference . Statistically , this discrepancy means the two distributional functions of real texts and generated texts is different . Reducing this distributional difference may be a practicable way to improve text generation . Some researchers try to reduce this difference with GAN ( Goodfellow et al. , 2014 ) . They use a discriminator to detect the discrepancy between real samples and generated samples , and feed the signal back to upgrade the generator ( a LM ) . In order to solve the non-differential issue that arises by the need to handle discrete tokens , reinforcement learning ( RL ) ( Williams , 1992 ) is adapted by SeqGAN ( Yu et al. , 2017 ) , RankGAN ( Lin et al. , 2017 ) , and LeakGAN ( Guo et al. , 2018 ) . The Gumble-Softmax is also introduced by GSGAN ( Jang et al. , 2017 ) and RelGAN ( Nie et al. , 2019 ) to solve this issue . These language GANs pre-train both the generator ( G ) and the discriminator ( D ) before adversarial learning1 . During adversarial learning , for each round , the G is trained several epochs and then , the D is trained tens of epochs . Learning stops when the model converges . Furthermore , considering the generated texts ’ quality and diversity simultaneously ( Shi et al. , 2018 ) , 1An exception is RelGAN which needs not pre-train D. MaskGAN ( Fedus et al. , 2018 ) , DpGAN ( Xu et al. , 2018 ) , FMGAN ( Chen et al. , 2018 ) and RelGAN ( Nie et al. , 2019 ) are proposed . They evaluate the generated text with Bleu and self-Bleu ( Zhu et al. , 2018 ) or LM socre and reverse LM score ( Cı́fka et al. , 2018 ) , and claim these GANs improve the performance of generator . However recently questions have been rasied over these claims . Semeniuta et al . ( 2018 ) and Caccia et al . ( 2019 ) showed that via more precise experiments and evaluation , these considered GAN variants are defeated by a well-adjusted language model . d ’ Autume et al . ( 2019 ) trained language GANs from scratch , nevertheless , they only achieve the ” comparable ” performance against LM . He et al . ( 2019 ) quantifies the exposure bias and concludes it is either 3 percent lower in performance or indistinguishable . All the aforementioned methods treat GAN as a black box for evaluation . For those language GANs , there are several critical issues such as whether the D detects the discrepancy or not ; the detected discrepancy is severe or not , the signals from D could improve the generator or not are still unclear . In this paper , we try to solve these problems via investigating GAN in both pre-training and the adversarial learning process . Theoretically analysing the signal from D , we obtain two metric functions to measure the distributional difference . With these two functions , we first measure the difference between the real text and the generated text by a MLE-trained language model ( pretrain ) . Second , we try some methods to update generator with feedback signal from D , then , we use these metric functions to evaluate the updated generator . Finally , we analysis the existing language GANs during the adversarial learning with these two functions . All the code and data could be find https : //github.com/ . Our contributions are as follows : • We propose two metric functions to measure the distributional difference between real text and generated text . Besides that , a method is put forward to estimate them . • Evaluated using these two functions , a number of experiment show there is an obvious discrepancy between the real text and the generated text even when it is generated by a well-adjusted language model . • Although this discrepancy could be detected by D , the feedback signal from D can not improve G using existing methods . • Experimenting on two existing language GANs , SeqGAN and RelGAN , the distributional discrepancy between real text and generated text increases with more adversarial learning rounds . It demonstrates both of these language GANs fail . 2 METHOD . In GAN , the generator Gθ implicitly defines a probability distribution pθ ( x ) to mimic the real data distribution pd ( x ) . min Gθ max Dφ V ( Dφ , Gθ ) = Ex∼pd ( x ) [ logDφ ( x ) ] + Ex∼pθ ( x ) [ log ( 1−Dφ ( x ) ) ] ( 1 ) We define Dφ to detect the discrepancy between pθ ( x ) and pd ( x ) . We optimize Dφ as follow , max Dφ V ( Dφ , Gθ ) = max Dφ Ex∼pd [ logDφ ( x ) ] + Ex∼pθ [ log ( 1−Dφ ( x ) ) ] ( 2 ) Assuming D∗φ ( x ) is the optimal solution for a given θ , according to ( Goodfellow et al. , 2014 ) , there will be , D∗φ ( x ) = pd ( x ) pd ( x ) + pθ ( x ) ( 3 ) We obtain two metric functions to measure this discrepancy . D ∗ φ ( x ) ≥ 0.5 , iif pd ( x ) ≥ pθ ( x ) D∗φ ( x ) < 0.5 , iif pd ( x ) < pθ ( x ) ( 4 ) With it , the integration of density function could be transformed into statistic equation . Based on that , we could get a way which will be described in next , to compute the precise discrepancy . 2.1 APPROXIMATE DISCREPANCY . Let , qd ( x ) = pd ( x ) pd ( x ) + pθ ( x ) qθ ( x ) = pθ ( x ) pd ( x ) + pθ ( x ) ( 5 ) So , qd ( x ) = p ( x comes from real data|x ) , qθ ( x ) = p ( x comes from generated data|x ) , qd ( x ) + qθ ( x ) = 1 . With equation 5 , we could get a constraint and an approximated measure function of distributional function . Figure 1 ( a ) illustrates the relationship between qθ ( x ) and qd ( x ) . Let , ud = Ex∼pd ( x ) ( D∗φ ( x ) ) uθ = Ex∼pθ ( x ) ( D∗φ ( x ) ) ( 6 ) They are two statistic equations who are the expectation of the D∗φ ’ s predictions on real text and on generated text respectively . According to the above equation , it is easy to get following equation . 1 2 [ ud + uθ ] = 0.5 ( 7 ) It gives a constraint for Dφ converging to D∗φ . We should take this constraint into account when estimating the ideal function D∗φ . From equation 3 , the process of optimizing the discriminator is make ud big and make uθ small . So , we could estimate the distributional discrepancy according to the following function . Intuitively , using ud and uθ , we get a metric function to measure the discrepancy between pθ ( x ) and pd ( x ) , da = ud − uθ ( 8 ) We call it approximate discrepancy . It is the subtraction of the average score of a well-trained discriminator ( denoted as D̂φ ) makes the predictions on real samples and on generated samples . It reflects the discrepancy between these two sets to some degree . From equation 5 and 6 , we get equation 8 , da = ∫ [ qd ( x ) − qθ ( x ) ] pd ( x ) dx = Ex∼pd ( x ) [ qd ( x ) − qθ ( x ) ] ( 9 ) Figure 1 ( a ) illustrates the discrepancy between two distributional functions qθ ( x ) and qd ( x ) . Both of them are systematic to the line of q = 0.5 . But it is not a complete measure because there is not only a positive part but also a negative part . A complete metric function is shown in next section . 2.2 ABSOLUTE DISCREPANCY . In order to precisely measure the discrepancy , we define ds , ds = 1 2 ∫ ∣∣pd ( x ) − pθ ( x ) ∣∣dx ( 10 ) The range of this function is 0 ∼ 1 . The bigger of its value is , the discrepancy is more . When its value is zero , it means pd ( x ) ≡ pθ ( x ) , namely there is no discrepancy . Fortunately , this function could be estimated by statistic method which is described by following equation . The proof is shown in appendix A. ds = 1 2 [ E x∼pd ( x ) D∗φ ( x ) > 0.5 ( 1 ) − E x∼pd ( x ) D∗φ ( x ) ≤0.5 ( 1 ) + E x∼pθ ( x ) D∗φ ( x ) ≤0.5 ( 1 ) − E x∼pθ ( x ) D∗φ ( x ) > 0.5 ( 1 ) ] ( 11 ) With equation 11 , we could more precisely estimate the discrepancy between pθ ( x ) and pd ( x ) . Assuming the classification precision of D∗φ is a , then the error rate is b = 1 − a . According to equation 11 , ds = a − b . So , the discrepancy between pθ ( x ) and pd ( x ) equals the classification precision of D∗φ minus its error rate . 2.3 USING D∗φ ( x ) TO IMPROVE Gθ Given an instance x generated by Gθ , if D∗φ ( x ) is larger , it means the possibility of x is real data is larger . For an instance D∗φ ( x ) = 0.8 , there will be pθ ( x ) < pd ( x ) according to equation 3 . So , we should update Gθ to make the probability density pθ ( x ) increase . It may improve the performance of Gθ . Based on this , we could select out some generated instances by the value of D∗φ ( x ) to update the generator . In fact , we find it helpful to use the faked samples whose score are higher assigned by D∗φ . Experiment 4.3 shows the results .

Summary: This paper proposes two metrics to measure the discrepancy between generated text and real text, based on the discriminator score in GANs. Empirically, it shows that text generated by current text generation methods is still far from human-generated text, as measured by the proposed metric. The writing is a bit rough so sometimes it's hard to figure out what has been done. It's also unclear how the proposed metrics compare to simply using the discriminator for evaluation. Therefore, I'm inclined to reject the current submission.