Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This submission analyzes spectral normalization (SN), an widely used and effective method for improving GANs, from the viewpoint of its effect on the gradients of the parameters of the networks (as opposed to the gradients with regards to their inputs).
Doing so, it shows that SN controls for vanishing and exploding gradients in the discriminator. It also provides a possible explanation as to why the usual implementation of SN is more effective than its "formally correct" counterpart.
Finally, drawing links with the popular LeCun initialization, it proposes an improved variant of Spectral Normalization.




main_review: Originality:
Analyzing gradients is a natural and fairly well-studied way to consider spectral normalization. However, by specifically studying the weights of the discriminator, this work is able to find novel, surprising, results and applications, including links with initialization methods. While the related work paragraph is succinct, much of the actual related work is discussed thoroughly in the relevant sections of the paper.

Quality:
The theoretical analysis seems of good quality. Experiments are comprehensive on multiple relevant datasets. The modifications for SN proposed in section 5 are less convincing. While theoretically strong, they seem to only result in a marginal (and not always consistent) improvement in practice.

Clarity:
The submission is clearly written and well organized.

Significance:
The paper provides some new understanding of Spectral Normalization, a crucial component of GANs. The findings are interesting theoretically and could lead to future improvements. Importantly, the paper opens a new direction for future work.
limitations_and_societal_impact: Limitations are openly discussed.
Potential societal impacts are adequately adressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper offers a more insightful analysis on Spectral Normalization (SN) and shows that SN controls two important failure modes of GAN: exploding and vanishing gradients. Specifically, the paper provides the theories to prove that SN upper-bounds the gradients and controls the variance of weights closely to LeCun initialization. Furthermore, given the new understanding of the connection between SN and LeCun initialization, the paper proposes a simple modification of SN to control both the variances of internal outputs and the variance of backpropagation gradients, which mimics the way of Xavier initialization. It also introduces new scaling weights, motivated by Kaming initialization, to obtain better performance in practice. The paper provides ablation studies and experiments on multiple benchmark datasets.
main_review: Pros
The paper is well-written, and contributions are verified with empirical experiments on benchmark datasets.  Understanding GAN is important, and this paper offers a useful contribution.

Cons
The analysis of gradient exploding and gradient vanishing have experimented on two different datasets: MNIST and CIFAR-10. Do both issues happen on the same dataset? It is good to verify these problems on other datasets as well.

BSN failed on the ILSVRC2012 dataset, do the authors have any insights why? 

What GAN objective function is used in the implementation? In my experience, SN (Miyato et al.,) does work only well with hinge loss objectives. Do authors have any insights into that from the analysis? Also, does the improved SN proposed work well with other loss functions? 

limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper provides insights towards understanding the stability of spectral normalization GANs (SNGANs), and proposes a new framework (BSSN) based on the proposed theoretical insights.
main_review: Overall this is a well-written paper, it provides two theoretical insights on the stability of SNGAN: SN prevents exploding (in Sec. 3) and vanishing (in Sec. 4) gradients. Based on the theoretical analysis, this paper also proposes the BSSN framework that incorporates the theoretical insights. See detailed comments/questions below:

1) In Theorem 1, how often does the constraint of $\Pi_{i=1}^Lc_i=1$ hold in practice? Perhaps the author can provide some examples/justifications of this assumption?

2) Comparing Theorem 2 and Theorem 3, it seems that the author believe that the BSSN is better than the SN because the BSSN can controls the variance at the scale of $\frac{2}{(n_i+m_i)}$ while SN controls the variance at the scale of $\frac{1}{\max\{n_i,m_i\}}$, is there any theoretical justification that $\frac{2}{(n_i+m_i)}$ is better than $\frac{1}{\max\{n_i,m_i\}}$?

3) The reviewer appreciates the way that the authors present their theoretical results: each proposed theorem comes along with experiments that justifying their observations (Fig. 2-3 for Thm. 1 in Section 3; Fig. 4-6 for Thm. 2 in Section 4; and Fig. 7 for Thm. 3 in Section 5).

4) The reviewer is not an expert in GANs experiments, but the reviewer finds the proposed BSSN framework indeed performs better under both FID and IS, and the numerical experiments seem reasonable and promising.
limitations_and_societal_impact: Perhaps the authors could improve the paper by answering/clarifying questions 1) and 2) mentioned in the main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper analyzes the benefits of the spectral norm on GAN: preventing gradient exploding and gradient vanishing. They prove that SN can control the variance of weights, which can prevent gradient vanishing, following the logic of LeCun initialization. They further propose a bidirectional SN, which is an improvement based on SN, following the logic of Xariver initialization. They conduct experiments on multiple datasets with DCGAN structure.
main_review: Originality (+): To my best knowledge, this work is the first one that connects SN with LeCun initialization and formally proves that SN can control the variance of weights and prevent gradient vanishing. But preventing gradient explosion is relatively well-known and less surprising. Overall, it can provide people a more theoretical understanding of SN.

Quality: 

Pros:

1. I think the proof itself is sound, but I have some questions about Theorem 2 (see cons).

2. Plots are clear and informative. 

3. I think it's a complete piece of work.

Cons: 

1. Theorem 2 doesn't provide a direct bound or analysis on the gradient, while Theorem 1 does. Instead, they analyze the variance of weights and justify the gradient vanishing in a roundabout way.

2. In Theorem 2, there is a gap between assumption and real GAN training: LeCun init assumes the activation layers are symmetrical, while GAN usually uses LeakReLU, which lowers the variance of the input and then weakens the story.

3. The lower bound in Theorem 2 could be far smaller than 1: 1/(16K^2 * max{m,n} * log(min{m,n})  (line 636). This means the variance of activation will get smaller and smaller. This situation will get worse with the effect of unsymmetrical activation layers. So from this bound, I cannot see the variance gets preserved, and the training can get saturated actually.

4. From Fig. 4, the gradient norm does get preserved. So I concern if Theorem 2 is the correct way to prove this. 

Clarity: The submission is clearly written and easy to follow.

Significance: 

Pros: 

1. I think the message that SN can prevent gradient explosion and vanishing is significant and the message itself should be correct by a large chance. 

2. The proposed BSN and BSSN can improve the results of SN.

Cons: Besides the concerns mentioned above, the experiments have some flaws that may make researchers hesitate to use the proposed methods in their work.

1. They only show the success with DCGAN structure, but not with more complex structures (e.g. ResNet and BigGAN) for the majority of the experiments. In the only experiment with the ResNet structure, BSN doesn't show the advantage. 

2. Tunning the weight scaling seems too cumbersome (i.e. pick the best from [0.7, 0.8, 0.9, 1.0, 1.2, 1.4, 1.6])

I will raise my score if concerns get addressed.
limitations_and_societal_impact: The paper mentions the limits and leaves them in future work.
review_title: null

Meta-review: The submission discusses the benefits of spectral normalization for GAN training. Initial reviewer assessment was mostly positive while one reviewer was a bit more concerned. After a discussion with the authors the reviewer appreciated the contributions while remaining unconvinced that controlling the variance via Lecun initialization is the key for SN's success. AC thinks this paper provides interesting insights that could spur future research.
Decision: Accept (Poster)