Language GANs Falling Short

1 INTRODUCTION . Generating fluent natural language is a central aim in Natural Language Processing ( NLP ) . Transformer architectures ( Vaswani et al. , 2017 ) with hundreds of millions or billions of parameters regularly reestablish state-of-the-art on held-out validation perplexities , however , the generated samples can still often lack coherence . It was hypothesized that exposure bias , differences in the training and inference procedures , is the root cause for poor sample quality . Therefore , numerous GANs ( Goodfellow et al. , 2014 ) for text generation were proposed since they do not suffer from exposure bias due a training objective that directly seeks to improve sample quality ( to a learned critic ) . While many of the early GAN variants demonstrated improvements in sample quality ( Yu et al. , 2017 ; Guo et al. , 2017 ) , they often ignored loss of sample diversity . This is not only a theoretical concern because it is well-documented that GANs exhibit mode collapse ( Che et al. , 2016 ; Salimans et al. , 2016 ) where the generator produces a subset of the ” modes ” in the training data , reducing ∗Authors contributed equally 1Code to reproduce experiments is available at github.com/pclucas14/GansFallingShort diversity . Evaluating generated samples remains challenging ( Liu et al. , 2016 ; Zhao et al. , 2017 ) , but Cı́fka et al . ( 2018 ) recommends assessing generated samples along two dimensions : 1 ) the quality of each sentence ; 2 ) the diversity across sentences . However , this too is problematic because now the assessment relies on two measures which can make it difficult to compare models . Without further information about the relative trade-off between the two dimensions it is impossible to claim which is superior ( Figure 1 , left subplot ) . In this work , we build on the natural relationship between a model ’ s softmax temperature and the resultant quality-diversity trade-off : lower temperatures generate less diverse , higher-quality samples ; higher temperatures increase the entropy of the distribution and produce more diverse , lower-quality samples . We propose a temperature sweep as a computationally efficient method to characterize the quality-diversity trade-off of different models . By explicitly controlling the temperature we remove a potential source of bias ( models may have different ” implicit ” temperatures ) and obtain a more complete understanding of each model ’ s behavior ( Figure 1 middle and right ) . Additionally , we address that temperature modulation changes the entropy of the conditional distributions , as opposed to changing the entropy of the joint distribution . We explore other ways to navigate the quality-diversity space with less bias , including stochastic beam search and generator rejection . Although these methods provide small gains in the quality-diversity trade-off they have other computational limitations . Despite the dizzying array of text-GAN variants and algorithms ( Yu et al. , 2017 ; Che et al. , 2017 ; Lin et al. , 2017 ; Zhang et al. , 2017 ; Guo et al. , 2017 ; Fedus et al. , 2018 ; Lu et al. , 2018a ; Shi et al. , 2018 ; Xu et al. , 2018 ; Chen et al. , 2018 ; Nie et al. , 2019 ; d ’ Autume et al. , 2019 ) our conclusions using temperature sweeps are clear : MLE models still dominate GANs . According to all metrics we studied , changing the temperature of MLE models at generation time leads to a better qualitydiversity trade-off compared to GANs . This evaluation technique provides a definitive boundary for future GAN research and our hope is that it will help the NLP community accurately assess natural-language generation progress . 2 ADVERSARIAL TEXT GENERATION . GANs are implicit generative models learned via a competition between a generator network Gθ and a discriminator network Dφ . The generator network Gθ produces samples from a probability distribution pmodel ( x ) . The discriminator Dφ ( x ) attempts to distinguish whether an input value x is real ( training data ) or generated . Mathematically , the GAN objective can be formulated as a minimax game L = min θ max φ Ex∼pdata [ logDφ ( x ) ] + Ex∼Gθ [ 1− logDφ ( x ) ] ( 1 ) GANs were first proposed for continuous domains since their training procedure differentiates through the discriminator into the generator . However , modeling text , which is both discrete and typically modeled sequentially requires a challenging adaptation for GANs which , in their original formulation , were built upon “ one-shot ” continuous data . Discrete ( sequential ) data require an alternative approach . Yu et al . ( 2017 ) estimate the gradient to the generator via REINFORCE policy gradients ( Williams , 1992 ) . In their formulation , the discriminator evaluates full sequences . Therefore , to provide error attribution earlier for incomplete sequences and to reduce the variance of gradients , they perform k Monte-Carlo rollouts until the sentence is completed . Yu et al . ( 2017 ) advertise their model using two tasks which we argue ( with hindsight ) are flawed . First , they introduce a synthetic evaluation procedure where the underlying data distribution P is known and can be queried . By representing P with an LSTM ( referred to as an oracle in the literature ) , they directly compute the likelihood of samples drawn from a generative model Gθ . The issue is that they benchmark models against each other on this likelihood alone , i.e. , the diagnostic is entirely blind to diversity . For example , a model that always outputs the same highly likely sequence would easily outperform other potentially superior models . For real data , there was no agreed-upon metric to evaluate the quality of unconditional NLG at the time . This led the authors to propose a new metric , Corpus-level BLEU , which computes the fraction of n-grams in a sample that appears in a reference corpus . Again , this metric is agnostic to diversity . Generating a single good sentence over and over will give a perfect BLEU score . This paper sparked a stream of works that adopt their evaluation framework . Notably , RankGAN ( Lin et al. , 2017 ) , MaliGAN ( Che et al. , 2017 ) , TextGAN ( Zhang et al. , 2017 ) , LeakGAN ( Guo et al. , 2017 ) and IRL-GAN ( Shi et al. , 2018 ) were proposed soon after . All but one of the aforementioned papers evaluate sample quality alone . As a remedy , Zhu et al . ( 2018 ) propose a metric that compares a generated sentence with a corpus of generated sentences , called Self-BLEU . They , along with Lu et al . ( 2018b ) , provide an extensive comparison of GANs using quality ( negative BLEU ) and diversity ( Self-Bleu ) . However , it is not clear which algorithm is superior , as evidenced by Figure 2 , because no model simultaneously outperforms the other on both metrics . It is now standard for language GANs to evaluate simultaneously quality and diversity . GANs Trained without RL Reinforcement Learning ( RL ) is often difficult to optimize , unstable , and sensitive to hyperparameters . Because of this , GAN-variants have recently been proposed that eschew RL-techniques in favor of fully-differentiable objectives . Cooperative training ( CoT ) ( Lu et al. , 2018a ) and feature mover GAN ( FM-GAN ) Chen et al . ( 2018 ) are notable approaches . Research in both RL and non-RL GANs is still very active ( e.g. , Xu et al . ( 2018 ) ; Nie et al . ( 2019 ) ; d ’ Autume et al . ( 2019 ) ; Li et al . ( 2019 ) ; Gagnon-Marchand et al . ( 2019 ) ) and so our review is not exhaustive . 3 TEMPERATURE SWEEP : TOWARDS ROBUST NLG EVALUATION . During natural language generation , a single spurious sampled token can lead to an overall lowquality and incoherent sequence . In other words , a high-entropy conditional distribution may result in poor sample quality at inference . To address this problem , one can modulate the entropy of Gθ ( xt | x1 : t−1 ) with a Boltzmann temperature parameter α ( Ackley et al. , 1988 ) . If ot is the gen- erator ’ s pre-logit activation and W is the word embedding matrix , then the conditional distribution of the generator is given by Gθ ( xt | x1 : t−1 ) = softmax ( ot ·W/α ) . Decreasing α below 1.0 will increase ot and thus decrease the entropy ofG ’ s conditional probability . Temperature tuning naturally moves the model in quality/diversity space . We demonstrate this in Table 1 . The evaluation protocol for NLG explained in Section 2 is to compare models with respect to both quality and diversity metrics . Often this results in a situation where it is impossible to tell which model is superior , as shown in Figure 2 and further exemplified in Figure 1 ( Left ) . One can then control the quality-diversity trade-off of autoregressive text generators using temperature . We can leverage this tool to design a new evaluation framework that shines a light on the real performance of each model . More precisely , we propose to generate samples at s temperatures for each model in order to compute temperature curves in quality-diversity space . This is exemplified in Figure 1 . We refer to this procedure as temperature sweep . This new way of evaluating NLG models allows practitioners to answer questions such as : which model to use if interested in high quality ( or diversity ) samples ? Does a new model improve upon others in the quality/diversity space or is it just reducing the entropy of the distribution ? It could also be leveraged as a cross-validation tool e.g. , early-stop once the best temperature curve is achieved according to a heuristic . In the next section we show , using temperature sweep , that MLE models consistently outperform the new proposed GAN variants everywhere in the quality-diversity space . MLE performs equally on synthetic data to CoT and outperforms it on real data , whilst being computationally and algorithmically less complicated . Our results are further validated in independent follow-up works ( d ’ Autume et al. , 2019 ; Alihosseini et al. , 2019 ) where temperature sweeps show MLE outperforming GANs . To implement temperature sweep we have taken advantage of the fact that our model factorizes the joint distribution over an observation as a product of conditional distributions over single tokens given all previous tokens ( this is a property of most autoregressive neural networks ) . We change the temperature of these conditionals , which is straightforward to implement . However , this is different from changing the temperature of the joint probability distribution . The samples generated in this way are biased towards having lower entropy at the beginning of sentences compared to samples obtained by changing the entropy of the joint distribution . Changing the entropy of the joint distribution quickly becomes intractable with respect to vocabulary size and/or the total number of timesteps . We show below some entropy reducing techniques are less biased than changing the temperature of the conditionals and might achieve a better quality-diversity trade-off . However , we argue that theses methods aren ’ t suitable for our evaluation protocol due to computational inefficiency ( see Section 5.4 ) . Stochastic Beam Search is a popular approximate decoding technique . It is computationally expensive but leads to higher-likelihood samples than greedy decoding . Here , we focus on its stochastic version ( not to be confused with the locally optimal one local beam search ) . In Stochastic beam search with beam size k , the k most likely hypotheses ( sampled so far ) are kept at every decoding steps . In Appendix A , we detail this technique and explain how it is less biased than temperature tuning regarding the reduction of the joint distribution ’ s entropy . Finally , the more traditional local ( deterministic ) beam search removes all sample diversity and is not useful in our setting . Generator Rejection Sampling works as follow : generate some sentences ; compute their likelihood under the generator ’ s own distribution ; accept/reject the sample given a threshold . The threshold enables the practitioner to modulate the quality-diversity trade-off . In Appendix B , we explain how it is less biased than temperature tuning , why it is computationally expensive , and how the discriminator can be used in rejection sampling .

Summary: Recently many language GAN papers have been published to overcome the so called exposure bias, and demonstrated improvements  in natural language generation in terms of sample quality, some works propose to assess the generation in terms of diversity, however, quality and diversity are two conflicting measures that are hard to meet. This paper is a groundbreaking work that proposes receiver operating curve or Pareto optimality for quality and diversity measures, and shows that simple temperature sweeping in MLE generates the best quality-diversity curves than all language GAN models through comprehensive experiments. It points out a good target that language GANs should aims at.