Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes "spectral normalization" -- constraining the spectral norm of the weights of each layer -- as a way to stabilize GAN training by in effect bounding the Lipschitz constant of the discriminator function. The paper derives efficient approximations for the spectral norm, as well as an analysis of its gradient. Experimental results on CIFAR-10 and STL-10 show improved Inception scores and FID scores using this method compared to other baselines and other weight normalization methods.

Overall, this is a well-written paper that tackles an important open problem in training GANs using a well-motivated and relatively simple approach. The experimental results seem solid and seem to support the authors' claims. I agree with the anonymous reviewer that connections (and differences) to related work should be made clearer. Like the anonymous commenter, I also initially thought that the proposed "spectral normalization " is basically the same as "spectral norm regularization", but given the authors' feedback on this I think the differences should be made more explicit in the paper.

Overall this seems to represent a strong step forward in improving the training of GANs, and I strongly recommend this paper for publication.

Small Nits: 

Section 4: "In order to evaluate the efficacy of our experiment": I think you mean "approach".

There are a few colloquial English usages which made me smile, e.g. 
 * Sec 4.1.1. "As we prophesied ...", and in the paragraph below 
 * "... is a tad slower ...".
review_title: Nice step forward in improving training of GANs

final_score: rating: 7: Good paper, accept
review_content: The paper is motivated by the fact that in GAN training, it is beneficial to constrain the Lipschitz continuity of the discriminator. The authors observe that the product of spectral norm of gradients per each layer serves as a good approximation of the overall Lipschitz continuity of the entire discriminating network, and propose gradient based methods to optimize a "spectrally normalized" objective.

I think the methodology presented in this paper is neat and the experimental results are encouraging. However, I do have some comments on the presentation of the paper:

1. Using power method to approximate matrix largest singular value is a very old idea, and I think the authors should cite some more classical references in addition to (Yoshida and Miyato). For example,

Matrix Analysis, book by Bhatia
Matrix computation, book by Golub and Van Loan.

Some recent work in theory of (noisy) power method might also be helpful and should be cited, for example,
https://arxiv.org/abs/1311.2495

2. I think the matrix spectral norm is not really differentiable; hence the gradients the authors calculate in the paper should really be subgradients. Please clarify this.

3. It should be noted that even with the product of gradient norm, the resulting normalizer is still only an upper bound on the actual Lipschitz constant of the discriminator. Can the authors give some empirical evidence showing that this approximation is much better than previous approximations, such as L2 norms of gradient rows which appear to be much easier to optimize?
review_title: Paper review: the methodology is neat, but presentation can be improved

final_score: rating: 7: Good paper, accept
review_content: This paper borrows the classic idea of spectral regularization, recently applied to deep learning by Yoshida and Miyato (2017) and use it to normalize GAN objectives. The ensuing GAN, coined SN-GAN, essentially ensures the Lipschitz property of the discriminator. This Lipschitz property has already been proposed by recent methods and has showed some success. However,  the authors here argue that spectral normalization is more powerful; it allows for models of higher rank (more non-zero singular values) which implies a more powerful discriminator and eventually more accurate generator. This is demonstrated in comparison to weight normalization in Figure 4. The experimental results are very good and give strong support for the proposed normalization.


While the main idea is not new to machine learning (or deep learning), to the best of my knowledge it has not been applied on GANs. The paper is overall well written (though check Comment 3 below), it covers the related work well and it includes an insightful discussion about the importance of high rank models. I am recommending acceptance, though I anticipate to see a more rounded evaluation of the exact mechanism under which SN improves over the state of the art. More details in the comments below.

Comments:
1. One concern about this paper is that it doesn’t fully answer the reasons why this normalization works better. I found the discussion about rank to be very intuitive, however this intuition is not fully tested.  Figure 4 reports layer spectra for SN and WN. The authors claim that other methods, like (Arjovsky et al. 2017) also suffer from the same rank deficiency. I would like to see the same spectra included. 
2. Continuing on the previous point: maybe there is another mechanism at play beyond just rank that give SN its apparent edge? One way to test the rank hypothesis and better explain this method is to run a couple of truncated-SN experiments. What happens if you run your SN but truncate its spectrum after every iteration in order to make it comparable to the rank of WN? Do you get comparable inception scores? Or does SN still win?
3. Section 4 needs some careful editing for language and grammar.

review_title: Standard idea, great results 

Meta-review: This paper presents impressive results on scaling GANs to ILSVRC2012 dataset containing a large number of classes. To achieve this, the authors propose "spectral normalization" to normalize weights and stabilize training which turns out to help in overcoming mode collapse issues.  The presented methodology is principled and well written. The authors did a good job in addressing reviewer's comments and added more comparative results on related approaches to demonstrate the superiority of the proposed methodology. The reviewers agree that this is a great step towards improving the training of GANs.  I recommend acceptance.
Decision: Accept (Oral)