Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper analyzes why GANs underperform MLE in natural language generation tasks. The authors argue that minimizing KL-divergence like MLE is a more efficient approach compared with minimizing the same distinguishability criteria in adversarial models. The authors also propose that minimizing distinguishability can be regarded as boosting likelihood for certain families of models including n-gram models and neural networks with a softmax output layer.
strengths_and_weaknesses: Strengths:

1. It’s essential and meaningful to give a theoretical analysis on why text GANs fall short. The authors try to connect the minimization of distinguishability and the boosting of the MLE training objective, which is an interesting perspective.

Weaknesses:

1. The organization of this paper should be largely improved. It’s hard for me to follow the detailed derivation because there is always a gap between consecutive formulas. Also, as a paper focusing on text GANs, most parts are about general GANs which are not directly related to texts. For example, the authors mention GANs based on the n-gram model in Section 3. But the sequential formulation of q(x) where $x=w_1 w_2 \cdots w_t$ is not applied to the following derivation in Section 4 and 5. I feel that many parts of this paper are used to analyze general GANs (such image GANs), which are divergent from the title.
2. As mentioned in Section 2, the non-differentiable problem of text GANs is always solved by policy gradient or Gumbel-Softmax approximation. I wonder whether the following theoretical analysis consider this step because it plays an important role in the performance of text GANs from the existing works.
3. Since this paper proposes a specific algorithm (i.e., Algorithm 1 in Section 6), the authors should conduct an experiment (at least on synthetic data) to show the effectiveness of the proposed algorithm.
questions: I have included my questions in the weaknesses part.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The submission broadly aims at providing arguments for why GANs have not seen much success in sequential generation, as in language, but maximum likelihood models have. 

The submission is entirely theoretical, demonstrating that for a reasonable notion of “distinguishability” of model samples from true samples, one can derive a method to minimize the empirical negative log-likelihood using the output of a classifier that can discriminate (with some effectiveness at least) the model samples from true samples. This derivation connects “distinguishability” (for example, predictions from an adversarial discriminator) with log-likelihood maximization. For sequential models, one can adapt the same procedure to conditional distributions over sequence items, yielding an algorithm reminiscent of boosting, where (possibly-) weak “distinguishers” can be used at each stage to further minimize negative log-likelihood. A run-time analysis of this algorithm is provided.
strengths_and_weaknesses: I think some of the phrasing might be a bit misleading to some readers. The abstract suggests that the difference between maximizing likelihood for an explicit model and training an implicit model such as a GAN is “largely artificial”. Having read the paper, I do not find the equivalence this claim seemed to promise: it turns out that the main argument tying the two together is that one can use a differentiating-signal between two distributions to improve log-likelihood for an explicit model. L49-51 claims that GAN training is a “roundabout way of maximizing likelihood on observed data”, but in my reading, the submission does not really substantiate this — it is only shown that a current log-likelihood may be improved if one uses an effective adversary’s predictions to push down probability density at points distinguishable from the true samples by the adversary. In my view, this does not really imply that GAN training is maximum likelihood in disguise in any meaningful sense as seems suggestive in the submission, rather it only suggests that maximum likelihood training may be conducted by using (even weak) adversaries.

The lack of empirical support is somewhat disappointing, after deriving the algorithm. Algorithm 1 seems implementable, as long as one can design a $g$ that can handle multi-dimensional inputs (or $N$ of them), since Lemma 3 seems to suggest that the normalization is doable by only summing over $|\mathcal{V}|$ terms. Is there something I’m missing, in terms of practical application, which would explain the lack of empirical validation? Without experimental support, it is unclear if the connection in the paper is actionable: does it buy us anything? The conclusion suggests that the takeaway is that “in applications where it is natural to fit models by minimizing log-loss, it is indeed likely to be a more direct and efficient means of fitting a model”. So is the recommendation that one ought to use “distinguishers” to optimize NLL using the developed procedure? Or that we should directly attempt to optimize NLLs? What would be an example of an unnatural circumstance for fitting models with MLE (recall that for images, flow-based explicit models have been approaching GANs promisingly closely in terms of sample quality, for example [1]; the auto-regressive model PARTI from Google is highly convincing)?

It would be nice to have some clearer characterization of what is meant by “similar power” in F and Q.

In general, the question the submission seems to aim to tackle at a high-level (based on the title at the very least) might be too ambitious to settle. With all the moving parts in designing and training a model, such as architectural choices, optimization issues, issues of mode-collapse in generation, and the often unanticipated way they mesh  together (or don’t), the meta-aspects guiding efforts invested by the community on certain approaches over others depending on publicized initial results, pin-pointing reasons for why one class of methods might have seen less success than others is difficult unless a clear technical reason is identified. In my opinion, this submission does not really identify any such issue that clearly informs us why GAN training might be “overkill for NLP”. Once again, recall how the dominant narrative of “GANs are best for image generation, and nothing else comes close” has changed significantly in recent times to “flow, diffusion, and autoregressive models are starting to look really really good!”. 

Typo in L349: “to” —> “two”


Some related work that are relevant:

 — L144-151 does not take into account variants of GANs, such as the WGAN-GP [2], that do not suffer from gradients vanishing on discrete-spaces. More generally, how does the discussion in the paper relate to cases like WGANs where the critic outputs are not in [0, 1]?

 — FlowGAN [3] is one relevant work. They report that, in their framework, optimizing for likelihood in a hybrid model results in poor sample quality but good likelihoods, and conversely, using adversarial training results in opposite trends. Related to this observation are the broader observations in [4, 5] about how improved likelihoods need not correspond to improved sample quality in practice for high-dimensional data. 

Overall, 

 — While the paper seems to have decent elements of originality in it,

 — and is very clearly written,

 — but does not seem to deliver sufficiently in order to be of much practical significance at the current stage. 


[1] Glow, Kingma and Dhariwal, 2018

[2] WGAN-GP, Gulrajani et al., 2017

[3] FlowGAN, Grover et al., 2018

[4] Locally-connected transformations for deep GMMs, van den Oörd and Dambre, 2015

[5] A note on the evaluation of generative models, Theis et al., 2016.
questions: 1. Why is Algorithm 1 empirically non-validatable at this stage?

2. How does the recommendation in the paper to perform NLL optimization relate to observations in [4, 5]?

3. Can the method be modified to accommodate unbounded critics?
limitations: The authors have discussed potential negative societal impacts.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper argues that training a GAN, which minimizes distinguishability between a learned and actual distribution, and maximizing likelihood, are often equivalent for NLP tasks. They do this by (i) giving a case where minimizing distinguishability is not the same as maximizing likelihood due to limitations in the set of possible generates, then (ii) showing how training a GAN is equivalent to maximizing likelihood for n-gram models, then (iii) showing how training distinguishing reduces log loss, and finally (iv) showing a polynomial-time reductions from a distinguishing distribution to an MLE for sequence models.
strengths_and_weaknesses: Strengths
* The paper gives good theoretical motivation for an empirical phenomenon that, to the best of my knowledge, wasn't well understood. In particular, previous explanations argued that 
* The exposition in the paper is great; the authors help motivate with simple cases where distinguishability + MLE are different, then show how they're the same in an intuitive and realistic case, then provide general theory
* The explicit polynomial time reduction seems like a strong theoretical contribution.  

Weaknesses
* The main weakness is the lack of empirical validation; in particular, it would've been nice to see if GANs actually do end up at the MLE in simple cases, and whether the rate at which they do is slower, as argued in the paper. 
questions: * [140-141]: Is it the case that likelihood-based models generated text that could be easily distinguished from humans? I think fake-news detection, etc. has been an issue for at least 5 years? 
* What is novel about the next-token distinguisher? Is this not a direct application of the standard GAN objective for classification to sequence modeling? 
limitations: N/A.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors show maximizing likelihood is effectively minimizing distinguishability for log-linear Q. Authors suggest a polynomial-time reduction from likelihood maximization to next-token distinguishability. If the distinguisher has an advantage over a threshold, then a generator network with lower log-loss can be constructed accordingly. Since in sequential domains (eg, text) the likelihood is easier to compute, one might prefer to use MLE in the first place. 
strengths_and_weaknesses: # Strengths

It’s well known that GANs don’t work so well in the domain of text compared to MLE, yet a thorough investigation is still lacking. The paper offers new and detailed insights into this phenomenon. Previous works explain this as the consequences of having discrete tokens, being not differentiable, sparse reward, optimization challenges, etc. This paper chose a different angle by showing how GAN and MLE are actually closely related.

The usage of “distinguishability” to analyze GAN is new.

The general reduction is useful and solid. It’s good to see asymptotic analysis.

The argument is backed with mathematics derivations. An essential algorithm is given. Assumptions are clearly given before claims.

The final conclusion is straightforward and clear.

Overall this paper is working on a topic with significance. People do care why GANs can/cannot work on text. This paper is making contributions and casting new insights into this problem.

# Weaknesses

The introduction of “distinguishability” is abrupt to me. See Q2.

The relation of distinguishability, likelihood(KL divergence) and JS divergence should be discussed further. See Q2 Q3.

It seems this paper focuses on next-token level GANs and not sentence level GAN. See Q4.

“GANs are overkill for NLP” seems too strong a claim, unless you can be very sure that you cover all types of GAN in NLP.

In general, I'm positive for this paper, but I still have some concerns that need to be addressed.


questions: Q1: 

What’s the difference between “distinguisher” and “discriminator”? If they’re the same, maybe authors should unify the terms.

Q2: 

It’s known that MLE minimizes KL divergence and GAN minimizes JS divergence, but this paper assumes GAN minimizes distinguishability. On line 70, “Motivated by this observation, numerous adversarial approaches to approximating q have been attempted to minimize distinguishability d(q)”. Is it an observation, assumption or statement? Can you prove that GAN is indeed minimizing distinguishability (e.g., with what kind of loss for generator/discriminator, using what kind objective, and give the optimal value for D)?

Q3: 

When you say “distinguishable” or “distinguishability”, is it symmetrical (like JS) or asymmetrical (like KL)? Do “q is distinguishable from p” and “p is distinguishable from q” mean the same thing? 

If it’s asymmetrical (like KL), then it’s not surprising that maximizing likelihood and minimizing distinguishability yield the same convergence in most cases. But I don’t think GANs are using an asymmetrical objective (because discriminator treats real and fake samples equally).

If it’s symmetrical, then the example about ages you give in line 72-82 does not seem right to me. “A smaller m < 100 would yield less distinguishable samples”. Any q with m < 100 assigns zero probability to ages over 100, hence real samples (with 100+ ages) are very distinguishable from samples in q.  

Actually in the ages example, MLE(KL) and GAN(JS) both yields m=119 as the optimal point.

Q4: 

Does your argument work on next-token level GAN only? By next-token level, I mean the generator takes sentence prefix as input, and output the distribution for the next token, and the discriminator takes in prefix + next_token, and predicts a score for only the next_token (assuming prefix is always real). 

Another set of GAN is sentence-level, where discriminator takes in a whole sentence and predicts only one score. This kind of GAN suffers from sparse reward (with RL) or sampling difficulties. 

On next-token level, it’s no news GAN is almost identical as MLE. So the conclusion is not surprising. However the mathematics and proofs in the paper is valuable.

Q5: 

I still wonder what makes the fundamental difference between GAN in text and vision. Is it because text models are sequential (so likelihood is easy to compute)? Let’s say if you have an autoregressive image generator which generates small pictures (say 8x8) pixel after pixel, and each pixel is an int in [0,255]. Will your conclusion apply to this kind of sequential model? Will MLE work better than GAN?

Correct me if I am wrong or misunderstand your point.
limitations: I can completely understand this is a theoretical paper, but any experiment (even on toy/synthetic datasets) will further confirm the arguments made by this paper and examine to what extend all the assumptions stand.
review_title: null

Meta-review: metareview: In the context of text generation, the paper gives a theoretical argument that GAN objectives are equivalent to maximum-likelihood training when the generator and discriminator families are 'paired'. Reviewers generally felt that the perspective was interesting (broM, jtPN, UUT1) and the theory was insightful (jtPN, UUT1). Reviewer vzAW raises the concern that the original draft of this paper overclaimed throughout, but I feel this has been addressed well enough in a revision. Reviewers broM and vzAW felt empirical validation was lacking, but since the paper's focus is clearly theoretical I don't see this as preventing acceptance. Overall this paper is borderline but I feel that it's interesting enough to merit acceptance despite flaws.

confidence: Less certain
Decision: Accept