Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The submission describes an empirical study regarding the training performance
of GANs; more specifically, it aims to present empirical evidence that the
theory of divergence minimization is more a tool to understand the outcome of
training (i.e. Nash equillibrium) than a necessary condition to be enforce
during training itself.

The work focuses on studying "non-saturating" GANs, using the modified generator
objective function proposed by Goodfellow et al. in their seminal GAN paper, and
aims to show increased capabilities of this variant, compared to the "standard"
minimax formulation. Since most theory around divergence minimization is based
on the unmodified loss function for generator G, the experiments carried out in
the submission might yield somewhat surprising results compared the theory.

If I may summarize the key takeaways from Sections 5.4 and 6, they are:
- GAN training remains difficult and good results are not guaranteed (2nd bullet
  point);
- Gradient penalties work in all settings, but why is not completely clear;
- NS-GANs + GPs seems to be best sample-generating combination, and faster than
  WGAN-GP.
- Some of the used metrics can detect mode collapse.

The submission's (counter-)claims are served by example (cf. Figure 2, or Figure
3 description, last sentence), and mostly relate to statements made in the WGAN
paper (Arjovsky et al., 2017).

As a purely empirical study, it poses more new and open questions on GAN
optimization than it is able to answer; providing theoretical answers is
deferred to future studies. This is not necessarily a bad thing, since the
extensive experiments (both "toy" and "real") are well-designed, convincing and
comprehensible. Novel combinations of GAN formulations (non-saturating with
gradient penalties) are evaluated to disentangle the effects of formulation
changes.

Overall, this work is providing useful experimental insights, clearly motivating
further study.

review_title: Well-written experimental study; light on theory; poses new questions and aims to answer some

final_score: rating: 7: Good paper, accept
review_content: This paper answers recent critiques about ``standard GAN'' that were recently formulated to motivate variants based on other losses, in particular using ideas from optimal transport.  It makes main points
1) ``standard GAN'' is an ill-defined term that may refer to two different learning criteria, with different properties
2) though the non-saturating variant (see Eq. 3) of ``standard GAN'' may converge towards a minimum of the Jensen-Shannon divergence, it does not mean that the minimization process follows gradients of the Jensen-Shannon divergence (and conversely, following gradient paths of the Jensen-Shannon divergence may not converge towards a minimum, but this was rather the point of the previous critiques about ``standard GAN''). 
3) the penalization strategies introduced for ``non-standard GAN'' with specific motivations, may also apply successfully to the ``standard GAN'', improving robustness, thereby helping to set hyperparameters.
Note that item 2) is relevant in many other setups in the deep learning framework and is often overlooked.

Overall, I believe that the paper provides enough material to substantiate these claims, even if the message could be better delivered. In particular, the writing is sometimes ambiguous (e.g. in Section 2.3, the reader who did not follow the recent developments on the subject on arXiv will have difficulties to rebuild the cross-references between authors, acronyms and formulae). The answers to the critiques referenced in the 
 paper are convincing, though I must admit that I don't know how crucial it is to answer these critics, since it is difficult to assess wether they reached or will reach a large audience.

Details:
- p. 4 please do not qualify KL as a distance metric 
- Section 4.3: "Every GAN variant was trained for 200000 iterations, and 5 discriminator updates were done for each generator update" is ambiguous: what is exactly meant by "iteration" (and sometimes step elsewhere)? 
- Section 4.3: the performance measure is not relevant regarding distributions. The l2 distance is somewhat OK for means, but it makes little sense for covariance matrices. 
review_title: Good point

final_score: rating: 4: Ok but not good enough - rejection
review_content: Quality: The authors study non-saturating GANs and the effect of two penalized gradient approaches. The authors consider a number of thought experiments to demonstrate their observations and validate these on real data experiments. 

Clarity: The paper is well-written and clear. The authors could be more concise when reporting results. I would suggest keeping the main results in the main body and move extended results to an appendix.

Originality: The authors demonstrate experimentally that there is a benefit of using non-saturating GANs. More specifically, the provide empirical evidence that they can fit problems where Jensen-Shannon divergence fails. They also show experimentally that penalized gradients stabilize the learning process.

Significance: The problems the authors consider is worth exploring further. The authors describe their finding in the appropriate level of details and demonstrate their findings experimentally. However, publishing this  work is in my opinion premature for the following reasons:

- The authors do not provide further evidence of why non-saturating GANs perform better or under which mathematical conditions (non-saturating) GANs will be able to handle cases where distribution manifolds do not overlap;
- The authors show empirically the positive effect of penalized gradients, but do not provide an explanation grounded in theory;
- The authors do not provide practical recommendations how to set-up GANs and not that these findings did not lead to a bullet-proof recipe to train them.


review_title: A great first study

Meta-review: AnonReviewers 2 and AnonReviewer 3 rated the paper highly, with AR3 even upgrading their score.  AnonReviewer1 was less generous:

" Overall, it is a good empirical study, raising a healthy set of questions. In this regard, the paper is worth accepting. However, I am still uncomfortable with the lack of answers and given that the revision does not include the additional discussion and experiments promised in the rebuttal, I will stay with my evaluation."

The authors have promised to produce the discussion and new experiments. Given the nature of both (1: the discussion is already outline in the response and 2: the experiments are straightforward to run), I'm inclined to accept the paper because it represents a solid body of empirical work.
Decision: Accept (Poster)