Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper provides a unified theoretical framework for regularizing GAN losses. It accounts for most regularization technics especially spectral normalization and gradient penalty and explains how those two methods are in fact complementary. So far this was only observed experimentally but without any theoretical insight. The result goes beyond that as the criterion could be applied to general convex cost functional.
The main general theorem is Theorem 1 which states 3 conditions on the optimal critic and 2 others on the generator. The paper is mainly concerned by the conditions on the optimal critic and show that the first 2 conditions can be achieved by the Spectral normalization, while the last one can be achieved by some gradient penalty.
The paper is clearly written, well structured and pleasant to read.
I have the following two remarks:
	- Proposition 8 provides a way to ensure condition 2 holds (beta-smoothness). It requires spectral normalization and smooth activation functions. In practice, while the spectral normalization is important, the choice of the activation is not in general 1-smooth (Leaky-relu for instance). Does it really matter in practice? 
	Some illustrative experiments could be beneficial to better understand what's happening.
	- Is it that hard to obtain generators that satisfy condition G1 and G2, it seems to be a natural consequence on the regularity of the mapping f? If that is the case, it might be worth better explaining how this is challenging.
	
Limitations: The paper considers only the setting where the optimal critic is reached and therefore it is still unclear if the analysis carries on to the training procedures used in practice (non-optimal critic). The authors recognize this limitation and leave it for future work.

Overall, I feel that the paper provides good insights on what regularization is important for training gans and why. For that reason, I think this paper should be accepted.


------------------------------------------------------------------------------------------------------------
Revision:

I think the paper provides a good theoretical contribution in terms of interpreting many of the tricks used for improving GAN training. In fact the paper also suggests some new regularization methods (prop 13 for conditions D3)  which would constrain the RKHS norm of the critic. The authors show how it is related to gradient penalty, in a particular case, but the result also suggests something more general. For instance [1], consider an abstract RKHS space containing deep networks and provide an upper-bound on the rkhs norm of such networks in terms of the spectral norm of their weights and a lower-bound in terms of its Lipschitz constant. 

I do agree with reviewer 1 that a better discussion of the connection to [2] should be included since that paper was interested in  ensuring weak continuity of the loss, which can be thought of as  a first requirement to get more regularity of the cost functional.

I still think the paper is worth being accepted and raised my score to 8 as I think the authors addressed the major concerns that were raised. 

[1] A. Bietti, G. Mialon, D. Chen, and J. Mairal. A Kernel Perspective for Regularizing Deep Neural Networks.
[2] Michael Arbel, Dougal Sutherland, Mikołaj Binkowski, and Arthur Gretton. On gradient regularizers for MMD GANs. In Advances in Neural Information Processing Systems, pp. 6700–6710, 2018.






review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The work studies the relationship between the stability and the smoothness of GANs based on the proposition which was proposed by Bertsekas . It explains many nontrivial empirical observations when one is training GANs, including both of the necessities of the spectral normalization and the gradient penalty, in a theoretical perspective. And the work points out that most common GAN losses do not satisfy the all of the smoothness conditions, thereby corroborating their empirical instability. Meanwhile, it develops regularization techniques that enforce the smoothness conditions, which can lead to stability of the GAN.

Pros
1. The paper theoretically gives a reasonable explanation of why applying a gradient penalty together spectral norm seems to improve performance of generator.
2.  The proofs of the theorems and the propositions in this paper are gorgeous and beautiful.

Cons
1. As the paper concludes, in practice, it is impossible to let the generator be trained after the discriminator attain theoretical optimal. As a paper which topic is about the training process of GAN, it is better to account for real situation.

2. The experiment section is too simple and lacks of persuasiveness. The main theorem only gives the sufficiency of those conditions. I think it’s necessary to give an example which can imply that anyone condition is essential.

3. Proposition 9, Proposition 12 and Equation (7) show the equivalence between the condition (D3) and the existence of the regularization term of the reproducing kernel Hilbert space norm of the discriminator. But after this, the paper uses the first order term of the expansion in Proposition 13 to substitute $\|\psi\|_{H}^2$. The condition (D3) doesn't necessarily still hold if only adding the gradient penalty term to the objective function. Why it can be supposed that the first order term of the expansion plays a leading role in penalizing ? Isn't it unconvincing to explain the necessity of the gradient penalty from the perspective of making the condition (D3) true?
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The paper provides new theoretical view on GAN regularisation. However, it lacks proper empirical evaluation and makes an impression of a work in progress. Furthermore, the conclusions lead mostly to common techniques that have already been studied. 

Pros:
- Theorem 1 provides sufficient conditions for convergence of generator gradients to zero (under assumption of optimal discriminators).
- New view on combining loss functions and regularizers via inf-convolutions.
- Clarification of a difference between gradient penalties and spectal normalization.

Cons:
- No evaluation with respect to any reasonable GAN setting.
- Proposed regularization technique combines existing methods and does not actually propose new ones.
- The main insights of sections 4 and 5 are trivial, like enforcing Lipshitzness of optimal discriminator by optimization of only Lipshitz discriminators.
- It is unclear wheather proposed solutions are practical, e.g. use of smooth activation functions may be costly and may lead to vanishing gradients. Again, experiments would be desired.
- Same combination of regularization techniques (gradient penalty, spectral norm and MMD loss) has been studied by [1] in various forms (Gradient-Constrained MMD, Scaled MMD). However, there is no discussion of similarities and differences between these works. 
- Submission's main text is 10 pages long without sufficient reasons for that (figures, tables).

Detailed comments:
(1) End of Section 4: 'Theorem 1 also suggests that applying only Lipschitz constraints is not enough to stabilize GANs'. Theorem 1 is not 'iff', so Lipshitz constraint *may be* not enough.
(2) Section 6 concludes that penalization of discriminators RKHS's norm is required. It is unclear, however, why discriminator function would belong to such space.
(3) In Appendix B authors say, in the context of WGAN, that 'The Lipschitz constraint on the discriminator is typically enforced by spectral normalization (Miyato et al., 2018), (...)'. This setting fails, as stated earlier in the Introduction.
(4) It seems there is conceptual misundersting of what MMD-GANs are in Appendix B. Authors say 'Despite their names, MMD-GANs (Li et al., 2017a; Arbel et al., 2018) typically do not directly minimize the MMD but instead an adversarial version of the MMD'. GANs by definition are adversarial, while optimization against MMD alone is not. Hence, it is *according to their names*, not 'despite'. 
Generator losses implied by MMD-GANs under assumption of optimal discriminators, have been termed 'Optimized MMD' [1] and studied earlier in [2].
(5) Given (4), The Table 2. includes MMD as a GAN loss, although authors probably refer to the properties of non-adversarial Generative Moment Matching Networks [3].


[1] Michael Arbel, Dougal Sutherland, Mikołaj Binkowski, and Arthur Gretton. On gradient regularizers for MMD GANs. In Advances in Neural Information Processing Systems, pp. 6700–6710, 2018.
[2] B. K. Sriperumbudur, K. Fukumizu, A. Gretton, G. R. G. Lanckriet, and B. Schölkopf. “Kernel choice and classifiability for RKHS embeddings of probability distributions.” NIPS. 2009
[3] Yujia Li, Kevin Swersky, Richard Zemel, "Generative Moment Matching Networks", ICML 2015.
review_title: Official Blind Review #1

Meta-review: The paper provides a theoretical study of what regularizations should be used in GAN training and why. The main focus is that the conditions on the discriminator that need to be enforced, to get the Lipshitz property of the corresponding function that is optimized for the generator. Quite a few theorems and propositions are provided. As noted by Reviewer3, this adds insight to well-known techniques: the Reviewer1 rightfully notes that this does not lead to any practical conclusion. 
Moreover, then training of GANs never goes to the optimal discriminator, that could be a weak point; rather than it proceeds in the alternating fashion, and then evolution is governed by the spectra of the local Jacobian (which is briefly mentioned). This is mentioned in future work, but it is not clear at all if the results here can be helpful (or can be generalized).
 At some point of the paper it gets to "more theorems mode" which make it not so easy and motivating to read. 
The theoretical results at the quantitative level are very interesting.  I have looked for a long time on Figure 1: does this support the claims? First my impression was it does not (there are better FID scores for larger learning rates). But in the end, I think it supports: the convergence for a smaller that $\gamma_0$ learning rate to the same FID indicated the convergence to the same local minima (probably). This is perfectly fine. Oscillations afterwards move us to a stochastic region, where FID oscillates. So, the theory has at least minor confirmation. 


Decision: Accept (Poster)