Title: Spectral Normalization for Generative Adversarial Networks
Authors: Takeru Miyato
Toshiki Kataoka
Masanori Koyama
Yuichi Yoshida
Venue: ICLR 2018
Keywords: Generative Adversarial Networks
Deep Generative Models
Unsupervised Learning

Abstract: One of the challenges in the study of generative adversarial networks is the instability of its training. 
In this paper, we propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator.
Our new normalization technique is computationally light and easy to incorporate into existing implementations. 
We tested the efficacy of spectral normalization on CIFAR10, STL-10, and ILSVRC2012 dataset, and we experimentally confirmed that spectrally normalized GANs (SN-GANs) is capable of generating images of better or equal quality relative to the previous training stabilization techniques. 

TL;DR: We propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator of GANs.

Review discussion:
This paper borrows the classic idea of spectral regularization, recently applied to deep learning by Yoshida and Miyato (2017) and use it to normalize GAN objectives. The ensuing GAN, coined SN-GAN, essentially ensures the Lipschitz property of the discriminator. This Lipschitz property has already been proposed by recent methods and has showed some success.
However,  the authors here argue that spectral normalization is more powerful; it allows for models of higher rank (more non-zero singular values) which implies a more powerful discriminator and eventually more accurate generator. This is demonstrated in comparison to weight normalization in Figure 4. The experimental results are very good and give strong support for the proposed normalization.
While the main idea is not new to machine learning (or deep learning), to the best of my knowledge it has not been applied on GANs. The paper is overall well written (though check Comment 3 below), it covers the related work well and it includes an insightful discussion about the importance of high rank models. I am recommending acceptance, though I anticipate to see a more rounded evaluation of the exact mechanism under which SN improves over the state of the art. More details in the comments below.
Comments:
1. One concern about this paper is that it doesn’t fully answer the reasons why this normalization works better. I found the discussion about rank to be very intuitive, however this intuition is not fully tested.  Figure 4 reports layer spectra for SN and WN. The authors claim that other methods, like (Arjovsky et al. 2017) also suffer from the same rank deficiency. I would like to see the same spectra included.
2. Continuing on the previous point: maybe there is another mechanism at play beyond just rank that give SN its apparent edge? One way to test the rank hypothesis and better explain this method is to run a couple of truncated-SN experiments. What happens if you run your SN but truncate its spectrum after every iteration in order to make it comparable to the rank of WN? Do you get comparable inception scores? Or does SN still win?
3. Section 4 needs some careful editing for language and grammar.
Thank you so much for the review!
>This paper borrows the classic idea of spectral regularization, recently applied to deep learning by Yoshida and Miyato (2017) and use it to normalize GAN objectives.
Thank you very much for the comments;  we however would like to remind that the spectral normalization presented in this paper is very much different from spectral ‘norm’ regularization introduced in Yoshida and Miyato (2017).
Also,  unlike what we refer to as  “gradient penalty method”,  we are not regularizing the objective function in any means, so “normalize GAN objectives“ is an inaccurate keyword for our paper.
We are still using the same objective function as the classic GAN; we are just looking for the candidate discriminator from the normalized set of functions.
We will emphasize these points in the revised manuscript, since these confusions seem to be recurring issues.
>1.  The authors claim that other methods, like (Arjovsky et al. 2017) also suffer from the same rank deficiency. I would like to see the same spectra included.
Thanks for the suggestion. We plan to test with the weight clipping method  (Arjovsky et al. 2017) and report the results in the revised manuscript.
>2. Continuing on the previous point: maybe there is another mechanism at play beyond just rank that give SN its apparent edge? One way to test the rank hypothesis and better explain this method is to run a couple of truncated-SN experiments. What happens if you run your SN but truncate its spectrum after every iteration in order to make it comparable to the rank of WN? Do you get comparable inception scores? Or does SN still win?
That sounds like a good suggestion. Should there be ample rooms in the computational resource and time, we might try the experiment with CIFAR 10.
>3. Section 4 needs some careful editing for language and grammar.
Thanks, we will proofread the document once again.
This paper proposes "spectral normalization" -- constraining the spectral norm of the weights of each layer -- as a way to stabilize GAN training by in effect bounding the Lipschitz constant of the discriminator function. The paper derives efficient approximations for the spectral norm, as well as an analysis of its gradient. Experimental results on CIFAR-10 and STL-10 show improved Inception scores and FID scores using this method compared to other baselines and other weight normalization methods.
Overall, this is a well-written paper that tackles an important open problem in training GANs using a well-motivated and relatively simple approach. The experimental results seem solid and seem to support the authors' claims. I agree with the anonymous reviewer that connections (and differences) to related work should be made clearer.
Like the anonymous commenter, I also initially thought that the proposed "spectral normalization " is basically the same as "spectral norm regularization", but given the authors' feedback on this I think the differences should be made more explicit in the paper.
Overall this seems to represent a strong step forward in improving the training of GANs, and I strongly recommend this paper for publication.
Small Nits:
Section 4: "In order to evaluate the efficacy of our experiment": I think you mean "approach".
There are a few colloquial English usages which made me smile, e.g.
* Sec 4.1.1. "As we prophesied ...", and in the paragraph below
* "... is a tad slower ...".
Thank you so much for the review!
>I also initially thought that the proposed "spectral normalization " is basically the same as "spectral norm regularization", but given the authors' feedback on this I think the differences should be made more explicit in the paper.
Thanks for the suggestion; we will emphasize the difference between spectral norm regularization and our spectral normalization in the revised manuscript.
And thanks for pointing out the colloquialism, we will relax it :-)
The paper is motivated by the fact that in GAN training, it is beneficial to constrain the Lipschitz continuity of the discriminator. The authors observe that the product of spectral norm of gradients per each layer serves as a good approximation of the overall Lipschitz continuity of the entire discriminating network, and propose gradient based methods to optimize a "spectrally normalized" objective.
I think the methodology presented in this paper is neat and the experimental results are encouraging.
However, I do have some comments on the presentation of the paper:
1. Using power method to approximate matrix largest singular value is a very old idea, and I think the authors should cite some more classical references in addition to (Yoshida and Miyato).
For example,
Matrix Analysis, book by Bhatia
Matrix computation, book by Golub and Van Loan.
Some recent work in theory of (noisy) power method might also be helpful and should be cited, for example,
https://arxiv.org/abs/1311.2495
2. I think the matrix spectral norm is not really differentiable;
hence the gradients the authors calculate in the paper should really be subgradients. Please clarify this.
3.
It should be noted that even with the product of gradient norm, the resulting normalizer is still only an upper bound on the actual Lipschitz constant of the discriminator. Can the authors give some empirical evidence showing that this approximation is much better than previous approximations, such as L2 norms of gradient rows which appear to be much easier to optimize?
Thank you so much for the review!
>The authors observe that the product of spectral norm of gradients per each layer serves as a good approximation of the overall Lipschitz continuity of the entire discriminating network, and propose gradient based methods to optimize a "spectrally normalized" objective.
Thank you very much for the comments;
however we would like to emphasize that we are controlling the spectral norm of the operators, not their gradient.
Also, unlike what we refer to as  “gradient penalty method”,  we are not modifying the objective function in any means. We are still using the same objective function as the classic GAN; we are just looking for the candidate discriminator from the normalized set of functions.
> 1. I think the authors should cite some more classical references in addition to (Yoshida and Miyato).
Thanks for the remark, and yes we should have cited some of the classic references. We will add them to the revised manuscripts.
>2. I think the matrix spectral norm is not really differentiable;
hence the gradients the authors calculate in the paper should really be subgradients. Please clarify this.
Indeed,  when the spectrum has multiplicities, we would be looking at subgradients, and technically we should have said so.
However, the probability of this happening is zero (almost surely), and we assumed we can continue discussions without giving considerations to such events.  We will make note of this fact in the revised version.  Thanks!
>3.  Can the authors give some empirical evidence showing that this approximation is much better than previous approximations, such as L2 norms of gradient rows which appear to be much easier to optimize?
We would like to remind that the gest of our paper is not about the accuracy of the Lipschitz constant;  we do not intend to claim that our spectral normalization better controls the Lipschitz constant than the gradient penalty method.
As we claim in Section 3, an advantage of our normalization over the gradient penalty based method (WGAN-GP) is that we can control the Lipschitz constant even outside the neighborhoods of the observed datapoint.
Furthermore, spectral normalization can be carried out with less computational cost. Please see the discussions in the designated section for more detail.
We owe great thanks to all reviewers for helpful comments toward improving our manuscripts.
We revised our manuscript based on the reviewer’s comments (including the ones that were visible to us by mistake because of the administrator’s technical problem) and uploaded the revision.
Firstly,  we conducted additional comparative study against orthonormal regularization, and showed the advantage of our algorithm over the orthonormal regularization.
Secondly,  we responded to the AnonReviewer2’s comment by running still another experiment with weight clipping (Arjovsky et al. 2017) and compared the results on CIFAR10 and STL10.
We confirmed that, as we have noted in Section 3, weight clipping also suffered from the rank degeneracy and its performance turned out to be much worse than our spectral normalization.
Thirdly, for the ImageNet, we re-calculated the inception scores for all methods using the original tensorflow implementation and replaced the scores on the table, because we were using Chainer instead of Tensorflow exclusively for the ImageNet results, and there were some numerical variations. The newly computed values do not affect any of our claims regarding the advantages and the superiority of our algorithm.
The code for reproducing the results in this paper has been uploaded at
https://github.com/pfnet-research/sngan_projection.
Also we have uploaded other materials (pretrainied models, generated images and movies) at https://drive.google.com/drive/folders/1GnDuF02F3a_zNEwiA74DnaG7OQ3-Co3N.
Please go to the links if you are interested in our work.

Meta-review:
role: PC
data: comment: This paper presents impressive results on scaling GANs to ILSVRC2012 dataset containing a large number of classes. To achieve this, the authors propose "spectral normalization" to normalize weights and stabilize training which turns out to help in overcoming mode collapse issues.  The presented methodology is principled and well written. The authors did a good job in addressing reviewer's comments and added more comparative results on related approaches to demonstrate the superiority of the proposed methodology. The reviewers agree that this is a great step towards improving the training of GANs.  I recommend acceptance.

Decision: Accept (Oral)