Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper does a good job of clearly articulating a problem in contemporary training of GANs, coming up with an intuitive solution via regularizers in addition to optimizing only the discriminator score, and conducting clever experiments to show that the regularizers have the intended effect. 

There are recent related and improved GAN variants (ALI, VAEGAN, potentially others), which are included in qualitative comparisons, but not quantitative. It would be interesting to see whether these other types of modified GANs already make some progress in addressing the missing modes problem. If code is available for those methods, the paper could be strengthened a lot by running the mode-missing benchmarks on them (even if it turns out that a "competing" method can get a better result in some cases).

The experiments on digits and faces are good for validating the proposed regularizers. However, if the authors can show better results on CIFAR-10, ImageNet, MS-COCO or some other more diverse and challenging dataset, I would be more convinced of the value of the proposed method. 


review_title: Clearly identifies and attacks a key problem in GANs

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

This paper proposes several regularization objective such as "geometric regularizer" and "mode regularizer" to stabilize the training of GAN models. Specifically, these regularizes are proposed to alleviate the mode-missing behaviors of GANs.

Review:

I think this is an interesting paper that discusses the mode-missing behavior of GANs and proposes new evaluation metric to evaluate this behavior. However, the core ideas of this paper are not very innovative to me. Specifically, there has been a lot of papers that combine GAN with an autoencoder and the settings of this paper is very similar to the other papers such as Larsen et al. As I pointed out in my pre-review comments, in the Larsen et al. both the geometric regularizer and model regularizer has been proposed in the context of VAEs and the way they are used is essentially the same as this paper. I understand the argument of the authors that the VAEGAN is a VAE that is regularized by GAN and in this paper the main generative model is a GAN that is regularized by an autoencoder, but at the end of the day, both the models are combining the autoencoder and GAN in a pretty much same way, and to me the resulting model is not very different. I also understand the other argument of the authors that Larsen et al is using VAE while this paper is using an autoencoder, but I am still not convinced how this paper outperforms the VAEGAN by just removing the KL term of the VAE. I do like that this paper looks at the autoencoder objective as a way to alleviate the missing mode problem of GANs, but I think that alone does not have enough originality to carry the paper.

As pointed out in the public comments by other people, I also suggest that the authors do an extensive comparison of this work and Larsen et al. in terms of missing mode, sample quality and quantitative performances such as inception score.
review_title: Review

final_score: rating: null
review_content: How is the model trained? Section 3.3 refers to a "manifold step" and a "diffusion step" but only describes them informally. Do the steps consist of one step of gradient descent, or perhaps something else?
review_title: Training pseudocode

final_score: rating: null
review_content: What do the authors think is the main difference between this work and Larsen et al? Specifically, both the geometric regularizer (autoencoding cost in Larsen et al) and the mode regularizer (Discriminating based on samples from p(z) and q(z|x) in Larsen et al) have been proposed in the context of VAEs in Larsen et al and I feel the way they are used is essentially the same as this paper.
review_title: pre-review question

final_score: rating: 7: Good paper, accept
review_content: The authors identify two very valid problem of mode-missing in Generative Adversarial Networks, explain their intuitions as to why these problems occur and propose ways to remedy it. The first problem is about the discriminator becoming too good (close to 0 on fake, and 1 on real data) and providing 0 gradients to the generator. The second problem is that GANs are prone to missing modes of the data generating distribution entirely. The authors propose two regularization techniques to address these problems: Geometric Metrics Regularizer and Mode Regularizer

Overall, I felt that this is a good paper, providing a good analysis of the problems and proposing sensible solutions - if lacking solid from-first-principles motivation for the particular choices made. My other critique is the focus on manifolds, almost completely disregarding the probability density on the manifold - see my detailed comment below.

Detailed comments on the Geometric Metrics Regularizer: The motivation for this is to provide a way to measure and penalize distance between two degenerate probability distributions concentrated on non-overlapping manifolds, those of the generator and of the real data. There are different ways one could go about measuring difference between two manifolds or probability distributions concentrated on manifolds, for example:

- projection heuristic: measure the average distance between each point x on manifold A and the corresponding nearest point on manifold B (let’s call it the projection of x onto B).
- earth mover’s distance: establish a smooth mapping between the two manifolds that maps denser areas on manifold A to nearby denser areas of manifold B, and measure the average distance between corresponding pairs.

The two heuristics are similar but while the earth mover distance is a divergence measure for distributions, the projection heuristic only measures the divergence of the manifolds, disregarding the distributions in question.
The authors propose measuring the average distance between a point on the real data manifold and a point it gets mapped to by the composition of the encoder and the generator. While E○G will map to the generative manifold, it is unclear to me if they would map to a high-probability region on that manifold, so this probably doesn’t implement anything like Earth Mover’s Distance. On this note, I have just remembered seeing this before: https://github.com/danielvarga/earth-moving-generative-net As the encoder is trained so that E○G(x) is close to x on average, it feels like a variant of the projection heuristic above. Would the authors agree with this assessment?
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes two regularization approaches for training GAN, aiming to provide stronger gradient signal to move the generated distribution to data distribution and to avoid the generated distribution from getting trapped in only one or a few modes of the data distribution.  The presented approaches are entirely based on some intuitive arguments. As such intuitions are interesting, likely useful, and deserve further exploration in a broader context, they stay as heuristics as this point.  The paper will benefit from more rigorous theoretical justification of the presented approaches. 
review_title: Good intuition while lacking analysis

Meta-review: This paper presents some intuitions about why GAN training is difficult, and proposes a somewhat remedy that seems to help. 
 
 Pros
  - a detailed explanation of the motivation
  - side experiments to demonstrate the properties of the new method
 
 Cons
  - There are already several closely-related approaches, namely VAEGAN, with similar motivations and procedures. This paper argues that their method is conceptually distinct from the VAEGAN, but I and other reviewers found the arguments underdeveloped unconvincing.
  - Most of the evaluations are qualitative.
 
 The search space of reasonable modifications to the GAN objective is so large that either a more principled or systematic (as in, do more comparison experiments) exploration of the space is necessary. 

Given the importance of the topic, the PCs still believe this paper should be accepted, but we encourage the authors to further revise their paper to address the remaining outstanding issues.
Decision: Accept (Poster)