Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary
========
The authors present a new regularization term, inspired from game theory, which encourages the discriminator's gradient to have a norm equal to one. This leads to reduce the number of local minima, so that the behavior of the optimization scheme gets closer to the optimization of a zero-sum games with convex-concave functions.


Clarity
======
Overall, the paper is clear and well-written. However, the authors should motivate better the regularization introduced in  section 2.3.


Originality
=========
The idea is novel and interesting. In addition, it is easy to implement it for any GANs since it requires only an additional regularization term. Moreover, the numerical experiments are in favor of the proposed method.


Comments
=========
- Why should the norm of the gradient should to be equal to 1 and not another value? Is this possible to improve the performance if we put an additional hyper-parameter instead?

- Are the performances greatly impacted by other value of lambda and c (the suggested parameter values are lambda = c = 10)?

- As mentioned in the paper, the regularization affects the modeling performance. Maybe the authors should add a comparison between different regularization parameters to illustrate the real impact of lambda and c on the performance.

- GANs performance is usually worse on very big dataset such as Imagenet. Does this regularization trick makes their performance better?



Post-rebuttal comments
---------------------------------

I modified my review score, according to the problems raised by Reviewer 1 and 3. Despite the idea looks pretty simple and present some advantages, the authors should go deeper in the analysis, especially because the idea is not so novel.
review_title: A simple regularization term for training GANs is introduced, with good numerical performance.

final_score: rating: 3: Clear rejection
review_content: This paper contains a collection of ideas about Generative Adversarial Networks (GAN) but it is very hard for me to get the main point of this paper. I am not saying ideas are not interesting, but I think the author needs to choose the main point of the paper, and should focus on delivering in-depth studies on the main point.

1. On the game theoretic interpretations 

The paper, Generative Adversarial Nets, NIPS 2014, already presented the game theoretic interpretations to GANs, so it's hard for me to think what's new in the section. Best response dynamics is not used in the conventional GAN training, because it's very hard to find the global optimal of inner minimization and outer maximization.
The convergence of online primal-dual gradient descent method in the minimax game is already well-known, but this analysis cannot be applied to the usual GAN setting because the objective is not convex-concave. I found this analysis would be very interesting if the authors can find the toy example when GAN becomes convex-concave by using different model parameterizations and/or different f-divergence, and conduct various studies on the convergence and stability on this problem.

I also found that the hypothesis on the model collapsing has very limited connection to the convex-concave case. It is OK to form the hypothesis and present an interesting research direction, but in order to make this as a main point of the paper, the author should provide more rigorous arguments or experimental studies instead of jumping to the hypothesis in two sentences. For example, if the authors can provide the toy example where GAN becomes convex-concave vs. non-convex-concave case, and how the loss function shape or gradient dynamics are changing, that will provide very valuable insights on the problem. 

2. DRAGAN

As open commenters pointed out, I found it's difficult to find why we want to make the norm of the gradient to 1.
Why not 2? why not 1/2? Why 1 is very special?
In the WGAN paper, the gradient is clipped to a number less than 1, because it is a sufficient condition to being 1-Lipshitz, but this paper provides no justification on this number.
It's OK not to have the theoretical answers to the questions but in that case the authors should provide ablation experiments. For example, sweeping gradient norm target from 10^-3, 10^-2, 10^-1, 1.0, 10.0, etc and their impact on the performance.
Also scheduling regularization parameter like reducing the size of lambda exponentially would be interesting as well.
Most of those studies won't be necessary if the theory is sound. However, since this paper does not provide a justification on the magic number "1", I think it's better to include some form of ablation studies.

Note that the item 1 and item 2 are not strongly related to each other, and can be two separate papers. I recommend to choose one direction and provide in-depth study on one topic. Currently, this paper tries to present interesting ideas without very deep investigations, and I cannot recommend this paper to be published.

review_title: Lack of the main point

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper addresses the well-known stability problem encountered when training GANs. As many other papers, they suggest adding a regularization penalty on the discriminator which penalizes the gradient with respect to the data, effectively linearizing the data manifold.

Relevance: Although I think some of the empirical results provided in the paper are interesting, I doubt the scientific contribution of this paper is significant. First of all, the penalty the author suggest is the same as the one suggest by Gulrajani for Wasserstein GAN (there the motivation behind this penalty comes from the optimal transport plan). In this paper, the author apply the same penalty to the GAN objective with the alternative update rule which is also a lower-bound for the Wasserstein distance.

Justification: The authors justify the choice of their regularization saying it linearizes the objective along the data manifold and claim it reduces the number of non-optimal fixed points. This might be true in the data space but the GAN objective is optimized over the parameter space and it is therefore not clear to me their argument hold w.r.t to the network parameters. Can you please comment on this?

Regularizing the generator: Can the authors motivate their choice for regularizing the discriminator only, and not the generator? Following their reasoning of linearizing the objective, the same argument should apply to the generator.

Comparison to existing work: This is not the first paper that suggests adding a regularization. Given that the theoretical aspect of the paper are rather weak, I would at least expect a comparison to existing regularization methods, e.g.
Stabilizing training of generative adversarial networks through regularization. NIPS, 2017

Choice of hyper-parameters: The authors say that the suggested value for lambda is 10. Can you comment on the choice of this parameter and how it affect the results? Have you tried  annealing lambda? This is a common procedure in optimization (see e.g. homotopy or continuation methods).

Bogonet score: I very much like the experiment where the authors select 100 different architectures to compare their method against the vanilla GAN approach. I here have 2 questions:
- Did you do a deeper examination of your results, e.g. was there some architectures for which none of the method performed well?
- Did you try to run this experiment on other datasets?

review_title: Rather incremental work, I doubt the scientific contribution is significant

Meta-review: Pros:
The proposed regularization for GAN training is interesting and simple to implement.

Cons:
- Reviewers agree that the methodology is incremental over the WGAN with gradient penalty and the modification is not well motivated.
- Experimental results do not clearly demonstrate the benefits of the proposed algorithm and the paper also lacks comparisons with related works.
GIven the pros/cons, the committee feels the paper is not ready for acceptance in its current state.
Decision: Reject