Language GANs Falling Short

Generating high-quality text with sufficient diversity is essential for a wide range of Natural Language Generation (NLG) tasks. Maximum-Likelihood (MLE) models trained with teacher forcing have consistently been reported as weak baselines, where poor performance is attributed to exposure bias; at inference time, the model is fed its own prediction instead of a ground-truth token, which can lead to accumulating errors and poor samples. This line of reasoning has led to an outbreak of adversarial based approaches for NLG, on the account that GANs do not suffer from exposure bias. In this work, we make several surprising observations which contradict common beliefs. First, we revisit the canonical evaluation framework for NLG, and point out fundamental flaws with quality-only evaluation: we show that one can outperform such metrics using a simple, well-known temperature parameter to artificially reduce the entropy of the model’s conditional distributions. Second, we leverage the control over the quality / diversity tradeoff given by this parameter to evaluate models over the whole quality-diversity spectrum, and find MLE models constantly outperform the proposed GAN variants over the whole quality-diversity space. Our results have several implications: 1) The impact of exposure bias on sample quality is less severe than previously thought, 2) temperature tuning provides a better quality / diversity trade off than adversarial training, while being easier to train, easier to cross-validate, and less computationally expensive. 1 Recent Developments in NLG GANs are an instance of generative models based on a competition between a generator network Gθ and a discriminator network Dφ. The generator network Gθ represents a probability distribution pmodel(x). The discriminator Dφ(x) attempts to distinguish whether an input value x is real (came from the training data) or fake (came from the generator). Mathematically, the GAN objective can be formulated as a minimax game min θ max φ Ex∼pdata [logDφ(x)] + Ex∼Gθ [1− logDφ(x)]. Preprint. Work in progress. ar X iv :1 81 1. 02 54 9v 1 [ cs .C L ] 6 N ov 2 01 8 GANs originally were applied on continuous data like images. This is because the training procedure relied on backpropagation through the discriminator into the generator. Discrete (sequential) data require an alternative approach. [Yu et al., 2017] estimate the gradient to the generator via REINFORCE policy gradients [Williams, 1992]. In their formulation, the discriminator evaluates full sequences. Therefore, to provide error attribution earlier for incomplete sequences and to reduce the variance of gradients they perform k Monte-Carlo rollouts until the sentence is completed. [Yu et al., 2017] advertise their model using two tasks which we argue (with hindsight) are flawed. First, they introduce a synthetic evaluation procedure where the underlying data distribution P is known and can be queried. By representing P with an LSTM (referred to as an oracle in the literature) they directly compute the likelihood of samples drawn from a generative model Gθ. The problem is they benchmark models against each other on this likelihood alone, i.e., the diagnostic is completely blind to diversity. For example, a model that always outputs the same highly likely sequence would easily outperform other potentially superior models. For real data, there was no agreed upon metric to evaluate the quality of unconditional NLG at the time. This led the authors to propose a new metric, Corpus-level BLEU, which computes the fraction of n-grams in a sample that appear in a reference corpus. Again, this metric is agnostic to diversity. Generating a single good sentence over and over will gives a perfect BLEU score. 0.3 0.2 0.1 Negative BLEU-5 0.2 0.3 0.4 0.5