Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an explanation for mode collapse in the original GAN with the log -D objective for the generator (dubbed the non-saturating GAN or NS-GAN for short). The paper takes the approach of comparing the gradient of the generator objective for the original GAN with cross-entropy loss (dubbed the minimax GAN or MM-GAN for short) and the log -D variant. The key observation is that the difference between the gradient of the generator objective of MM-GAN and NS-GAN is that MM-GAN has a factor of D_p(G(z,\theta)), whereas NS-GAN has a factor of 1-D_p(G(z,\theta)), where D_p(G(z,\theta)) is the output of the discriminator on a sample generated from z. Hence, the terms in the MM-GAN gradient that appear fake to the discriminator are downweighted, whereas they are upweighted in the NS-GAN gradient. Because the samples from modes that are already overrepresented are likely declared fake by the discriminator, the contribution to the generator gradient is dominated by these samples in NS-GAN. 

Strengths:
This observation is very nice, intuitive and simple and is new to my knowledge. It sheds light on the weaknesses of the log -D variant of GANs.  

Weaknesses:
Paper title is broader than what the paper shows - the proposes explanation only applies to GANs with the log -D generator objective and does not apply to other GAN variants, e.g.: original GAN (with cross-entropy generator objective), WGAN, LSGAN etc.

The argument is not presented clearly, and sometimes observations with no obvious logical relationship to the claim are mentioned. At other times, it is unclear what the point is. For example:

On pg. 3, in the first paragraph under sect. 2.3, it is mentioned that "the minibatch used to update G will have more samples from O since they are generated more often". It is unclear how the number of samples from O in the minibatch could cause a difference between MM-GAN and NS-GAN - this observation is true for both MM-GAN and NS-GAN!

In the next paragraph, the paper mentioned that the generator gradient "is only locally informative" - again this is true for all GANs. How is this relevant for the argument made in the paper?

Then it mentioned that "generated samples give rise to conflicting gradients" - it is unclear what conflicting gradients mean. In what sense are the gradients "conflicting"?

In the next paragraph, the paper mentioned "NS-GAN struggles to discover new modes: g(x) ≈ 0 ⇒ 1 − D_p(x) ≈ 0" - the connection between the claim about the difficulty of discovering new modes and the equations should be explained more clearly. Also, the implication in the equations would not be true if r(x) ≈ 0. 

Also, for eq. (7), technically the optimal discriminator is only uniquely defined at the real data points (see Sinn & Rawat, AISTATS 2018) because the GAN is trained on a finite sample. The optimal discriminator result in (Goodfellow et al., 2014) assumes access to the true data distribution, or equivalently an infinite stream of samples from the distribution. So technically the paper's claim on the discovery of new modes cannot be justified by "g(x) ≈ 0 ⇒ 1 − D_p(x) ≈ 0", because D_p(x) could take on any value at locations other than the data points. Similarly, on pg. 4, in the paragraph below eq. (9), because D_l^{opt} could take on arbitrary values at positions other than the data points, the claim that D_{opt} = ±\infty is inaccurate - it is in fact not uniquely defined for points that are not real data points. Though because this is a common mistake in the literature, I'd be fine with the addition of a note that explains this caveat before presenting the theoretical argument (without insisting on a fundamental solution to this issue). 

Sect. 2.4 on "MM-GAN Interaction with ADAM" is not very mathematically rigorous and relies primarily on an assumption that the value of the logits of the discriminator approaches the optimum linearly. It's unclear if this assumption is actually justified, since the gradient of the cross-entropy loss w.r.t the logits tapers off on the extreme ends of the logits, so the logits should approach the optimum more and more slowly as they become larger in magnitude. 

Also, in eq. 11, the left size is vector graphics, whereas the right side is a scalar. 

Under sect. 3.1, the JSD between class predictions assume each class contains only one mode and cannot detect intra-class mode collapse. This caveat should be prominently posted. 
review_title: Review for Paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper reexamines the original (MM) and the non-saturating (NS) GAN objective.  The authors show that the gradients of the respective objectives just differ from a scaling factor depending on the discriminator's output for generated samples. While the scaling factor for the MM gradient is responsible for the well known vanishing gradient if the discriminator is optimal, the scaling factor of the NS gradient counteracts this saturation effect. However, on the other side, the NS scaling factor introduces a mode dropping effect and the inability of the learning dynamci to discover new modes. The authors show additionally that the NS minibatch gradient is the weighted sum of the single sample MM gradients with respective scaling factors as weights. These scaling factors avoid saturating, however, alter the direction of the resulting minibatch gradient. To counteract the change of the gradient direction the authors propose to summarize the sample scaling factors into one scalar for the batch gradient which preserves the non-saturating behaviour of the NS and the gradient direction of the MM objective. The new GAN objective is called MM-nsat.  Additionally the authors discuss the non-saturating effect of the ADAM beta2 parameter for the MM-GAN Generator. 

Experiments: A ring of Gaussians experiment shows the mode dropping effect of the NS-GAN. Training a 4-layer network on MNIST shows demonstrates the vanishing gradient effect on MM-GAN and the counteracting effect with a smaller ADAM beta2 parameter applied to the Generator. Also on MNIST MM-nsat was compared with NS showing a lower FID and Jensen-Shannon divergence between real and generated class distributions for MM-nsat. On the Cat 128x128 dataset mode collapse was visually shown for NS. 
On MNIST, CIFAR10 and Cat 128x128 MM-nsat was compared with NS, Hinge and LS-GAN outperforming all of them based on the FID.

Pros: The theoretical insight why NS-GAN suffers from mode collapse is novel and interesting. The experiments are convincing and extensive.

Cons: The proposed objective is not novel [1][2], however, it is derived directly from the original MM-GAN objective and is better theoretically motivated.  

For completeness, it would be interesting to see the experiments in section K with the pair MM-nsat/MM as well. 

[1] R Devon Hjelm, Athul Paul Jacob, Tong Che, Adam Trischler, Kyunghyun Cho, and Yoshua Bengio. Boundaryseeking generative adversarial networks, 2018
[2] Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, and Eric P. Xing. On unifying deep generative models. CoRR,
abs/1706.00550, 2017. URL http://arxiv.org/abs/1706.00550

review_title: IW-GAN revisited

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors present a simple estimator which approximates the gradient of the "true" generator loss in GANs using the gradients from the more commonly used non-saturating generator loss. They present results on toy data and small image datasets like CIFAR to show that the method leads to stabler training and does not suffer from mode collapse as badly.

Had this paper appeared at ICLR in, say, 2016 or 2017, I think it would have been a welcome addition to the literature on "tricks to improve GAN training", which was fairly small at that point. Since then, the field has absolutely exploded, to a degree that I find somewhat baffling, as GANs have no practical applications beyond generating pretty pictures. In addition, recent advances to VAE training (building on some of these GAN tricks like spectral normalization) have massively improved the quality of VAE samples, so the need for GANs continues to diminish. Nevertheless, it is not my job to judge whether a given paper is trendy or not - only to judge whether it is technically correct and up to the standards of publication. On that front, I think it is a decent paper - the method is quite simple, but the results on both mixtures of Gaussians and CIFAR and other image datasets do seem like an improvement. I would have appreciated experiments on a larger dataset like CIFAR or high-res CelebA. I also would have appreciated a more rigorous comparison against the plethora of GAN training tricks that have appeared in the last 5 years - for instance, competitive gradient descent (Schaefer and Anandkumar, 2019) to name just one. As it is, I worry this paper will disappear in a flood of other similar papers without more rigorous comparisons - but then, that may be for the field to judge after it is published.

A few stylistic points:
* Please increase the size of the axis labels in Fig 1. They are unreadable unless zoomed in very far on a screen.
review_title: Bayesians hate this one weird trick for improving GAN training!

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This work proposes that many common issues with GAN methods are based on the weighting of the samples given to the generator’s objective function. They focus on a study of the original GAN objective proposed in Goodfellow et al. where the generator’s objective is the negative of the discriminators objective. The GAN community quickly observed that when the discriminator outperforms the generator with this objective, the saturating nature of the sigmoid function causes the gradients to vanish for the generator’s objective. For this reason, a new objective (NS-GAN) was proposed which modifies the generator’s objective to alleviate this gradient vanishing issue.  The authors argue that this modified objective is to blame for a number of common issues with GAN methods -- most notably the mode-dropping issue. The authors present theory that backs up these claims and they propose a new generator training objective which re-weights the gradients of the generator objective to have the same average magnitude as NS-GAN but have the same relative magnitudes of the original GAN objective. 

The authors demonstrate the impact of their new loss function in a series of quantitative and qualitative settings. 


Strong areas:

I am a very big fan of work that questions standard assumptions that are taken almost as fact within our community. When GANs were originally proposed, most researchers saw the NS-GAN objective as a strict improvement over the MM-GAN objective and moved on. This work does a great job to demonstrate that NS-GAN is most definitely not “superior” to MM-GAN and may possibly be a worse choice of objective function. This change may seem insignificant on the surface and the authors here clearly show that it has an impact -- one large enough to consider if it should be used at all. To aid in their claims, the authors provide clear and concise explanations backed up by easy to understand theory. Particularly interesting to me was the argument that the popular Adam optimizer does not alleviate the saturation issues found with MM-GAN even though my own intuition on the optimizer tells us that it should help with issues like this. 

Weaknesses:

The empirical results presented appear promising. The proposed approaches (MM-Unit and MM-NSAT) considerably outperform the NS-GAN objectives. My biggest issues are with the quality of the baselines. I am not an expert in the GAN field, but from inspecting the Conv-4 model with spectral normalization on CIFAR10, it appears like the best performing model achieves an FID of approx 42 and the worst model gets around 48 (figure 8, bottom right). While this difference is notable and the error-bars indicate it is statistically significant, I am concerned because these numbers seem to considerably underperform prior work. The original paper on spectral normalization for GANs reports an FID of 29.3 for standard CNNs. This model uses the NS-GAN objective (to my knowledge) so, I am confused as to why the authors did not simply replicate their setup -- especially since the standard CNN model proposed in the original spectral norm paper can be trained with reasonable compute and the authors have released code. If I am misunderstanding something about the experiments, please do let me know, but I am confused by this choice.

I also took some issues with the D-JS-CD score proposed to score the class distribution of generated samples. There have been a number of proposed metrics like this such as IS and classifier score (https://arxiv.org/abs/1905.10887). I understand that this score (unlike classifier score) does not rely on a conditional model, but if a new score is to be proposed and it is used to convince the reader that a new method performs well, then it should be applied to some baseline models. Pre-trained models from prior GAN papers could be used to obtain these scores. 

As a researcher from a different field, my main concern about the experimental results is that the results are quite far from the current state-of-the-field. State of the art results are by no means required for publication in this venue, but the baseline models presented here perform much worse than they have been shown to in prior work. Since a near 20-point increase in FID can be achieved with a few tweaks to the NS-GAN objective (SN-GAN), I am left wondering if the presented improvement from the MM-NSAT objective will vanish once those improvements are applied or if it will still hold. Since this is not shown, then I am uncertain of the significance of the observations made in this work. 

Some more nit-picky issues:

The text in the figures is too small and near impossible to read. I would present all of the results in Figure 8 in a table instead. There is no information gained by seeing loss curves. A table would save much more space and allow the readers to more easily compare this work to other works. In Figure 7, I find “best” and “worst” picks by qualitative methods to be somewhat unconvincing. You should show the highest and lowest FID or just show random samples.

My recommendation:

I am not an active member of the GAN community so I am more than willing to accept if my recommendation goes against more senior folks who work in that field. 

I found this to be an interesting work that provided a non-trivial insight, backed it up with clear and easy-to-follow theory and demonstrated their observations held on some medium-scale experiments. My biggest issues come from my reservations about the experimental results. The baseline NS-GAN with spectral normalization presented in this work greatly underperforms previously published methods that use the same objective. The difference between the presented baseline and the proposed method is smaller than the difference in performance between the presented baseline and previously published methods with the baseline objective. 

These discrepancies give me sufficient doubt where I am not certain that the insights of this work provide a sufficient improvement when combined with architectural improvements and proper parameter tuning. 

For these reasons, I am advocating against acceptance of this work but making it clear that I think this paper is borderline.

If the experimental setup was more in line with prior work and the same trend in results held, then I would be more likely to recommend acceptance of this paper. 

review_title: An interesting approach, questionable experiments

Meta-review: I think we did learn something new from this paper, and I think the reviewers all seem to agree with this.
The observation you make about the objective seems correct and interesting (though reviewers and ACs do sometime miss errors), but I have the following complaints that keep me from recommending acceptance:

1. The theory seems right, but in practice, all sorts of GANs with all sorts of objective functions experience "mode collapse", 
so it doesn't seem like the issue you point out w/ the NS-GAN objective can be the whole story. 
However, we generally don't ask of a paper that it tells the whole story all in one go...

2. I do think the experiments are somewhat poorly done (compared to those for say the median paper about GANs that gets accepted to one of these conferences). Moreover, many people have made similar experimental claims to the ones that are in this paper that haven't held up on more complicated data sets, so I tend to apply more scrutiny to such claims when they're only evaluated on smaller tasks.

3. There have, as R3 points out, been a huge number of papers proposing tricks for training GANs, and some of them work really well. 
What I'm missing from this paper is an exploration of the relationship between your observation and those (mostly ad-hoc) tricks. 
Does your observation explain why those tricks are necessary? 
Does it explain why some existing trick works as well as it does?
If your observation is totally orthogonal to existing tricks, can you get much better performance on a challenging data set by using it?
I don't feel like I got satisfactory answers to those questions.

All this being said, the paper was borderline, and I think if you dealt with some of the complaints above you would have a pretty good shot of getting a revised version accepted at another major machine learning conference.
Decision: Reject