Why Spectral Normalization Stabilizes GANs: Analysis and Improvements
Authors: Zinan Lin
Vyas Sekar
Giulia Fanti
Venue: NeurIPS 2021 Poster (2021)
Keywords: spectral normalization
generative adversarial networks

Abstract: Spectral normalization (SN) is a widely-used technique for improving the stability and sample quality of Generative Adversarial Networks (GANs). However, current understanding of SN's efficacy is limited. In this work, we show that SN controls two important failure modes of GAN training: exploding and vanishing gradients. Our proofs illustrate a (perhaps unintentional) connection with the successful LeCun initialization. This connection helps to explain why the most popular implementation of SN for GANs requires no hyper-parameter tuning, whereas stricter implementations of SN have poor empirical performance out-of-the-box. Unlike LeCun initialization which only controls gradient vanishing at the beginning of training, SN preserves this property throughout training. Building on this theoretical understanding, we propose a new spectral normalization technique: Bidirectional Scaled Spectral Normalization (BSSN), which incorporates insights from later improvements to LeCun initialization: Xavier initialization and Kaiming initialization. Theoretically, we show that BSSN gives better gradient control than SN. Empirically, we demonstrate that it outperforms SN in sample quality and training stability on several benchmark datasets.

Why Spectral Normalization Stabilizes GANs: Analysis and Improvements
Zinan Lin
Vyas Sekar
Giulia Fanti
Carnegie Mellon University Pittsburgh, PA 15213
Carnegie Mellon University Pittsburgh, PA 15213 vsekar@andrew. cmu edu
Carnegie Mellon University Pittsburgh, PA 15213 gf gfanti@andrer cmu edu
sinanl@andrew. cmu edu
Abstract
Spectral normalization (SN) 30 is widely-used technique for improving the stability and sample quality Generative Adversarial Networks (GANs). However, current understanding of SN's efficacy is limited. In this work, show that SN controls two important failure modes of GAN training: exploding and vanishing gradients. Our proofs illustrate (perhaps unintentional) connection with the cessful LeCun initialization
(25] This connection helps to explain why the most popular implementation of SN for GANs
300 requires no hyper-paramete tuning,
whereas stricter implementations of SN [15 12 have poor empirical performance out-of-the-box. Unlike LeCun initialization which only controls gradient vanishing at the beginning of training, SN preserves this property throughout training. Build- propose new spectral normalization
ing on this theoretical understanding, technique: Bidirectional Scaled Spectral Normalization (BSSN), which incorpo- rates insights from later improvements LeCun initialization: Xavier initialization [33 and Kaiming initialization [17]. Theoretically, we show that BSSN gives better gradient control than SN. Empirically, we demonstrate that it outperforms SN in sample quality and training stability on several benchmark datasets.
1 Introduction
Generative adversarial networks (GANs) are state-of-the-arr deep generative models, perhaps best known for their ability to produce high-resolution, photorealistic images 14]. The objective of GANs is to produce random samples from target data distribution, given only access to an initial set of training samples. This is achieved by learning two functions: generator G. which maps random input noise to generated sample, and discriminator D, which tries to classify input samples as either real (i.e., from the training dataset) or fake (i.e., produced by the generator). practice, these functions are implemented by deep neural networks (DNNs), and the competing generator and discriminator are trained in an alternating process known as adversarial training. Theoretically, given enough data and model capacity, GANs converge to the true underlying data distribution [14).
Although GANs have been very successful in improving the sample quality of data-driven generative models 22 g
their adversarial training also contributes to instability. That is, small prper-parammetr changes and even randomness in the optimization can cause training fail. Many approaches have been proposed for improving the stability of GANs, including different architectures [38
22 8). loss One of
functions 2 16 50), and various types of regulaariztiion//ommaiiaaiii
(30
9
most successful proposals to date is called spectral normalization (SN) 30
15
12
SN forces each
layer of the discriminator to have unit spectral norm during training. This has the effect of controlling
the Lipschitz constant of the discriminator, which is empirically observed improve the stability of
GAN training [30]. Despite the successful applications SN 28 53 2 remains unclear precisely why this specific normalization is effective.
52 31 27 to date,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Explains
SN
LeCun initialization
Prior initialization techniques
Our proposed BSSN
Bidirectional normalization Molivates:
Xavier initialization
Scaling
Motivates
Kaiming initialization:
Figure I: The interesting connections we find between spectral normalizations and prior initialization techniques: (I) LeCun initialization [25] can help explain why SN avoids vanishing gradients; (2)
Motivated by newer initialization techniques [1 17]. propose BSSN to further improve SN.
In this paper, we show that SN controls two important failure modes of GAN training: exploding gradients and vanishing gradients. These problems are well- to cause instability in GANs 3 8,, leading either bad local minima or stalled training prior to convergence. We make three primary contributions:
(I) Analysis of why SN avoids exploding gradients ($ 3 Poorly-chosen architectures and hyper- parameters, as well as randomness during training, can amplify the effects of large gradients on training instability, ultimately leading to generalization error in the learned discriminator. We theoretically prove that SN upper bounds gradients during GAN training, mitigating these effects.
(2) Analysis of why SN avoids vanishing gradients IS 4V. Small gradients during training known to cause GANs (and other DNNs) to converge to had models [25 The well-known LeCun initialization, first proposed over two decades ago, mitigates this effect by carefully choosing the variance of the initial weights (25]. We prove theoretically that SN controls the variance of weights in way that closely parallels LeCun initialization. Whereas LeCun initialization only controls the gradient vanishing problem at the beginning of training, we show empirically that SN preserves this property throughout training. Our analysis also explains why a strict implementation of SN [12] has poor out-of-the-box performance on GANs and requires additional tuning to avoid the vanishing gradient problem, whereas the implementation of SN in 30 requires no tuning.
(3) Improving SN with the above theoretical insights 5). Given this new understanding of the connections between SN and LeCun initialization, we propose Bidirectional Scaled Spectral Normal- ization (BSSN), a simple modification of SN that combines two key insights (Fig. T (a) It introduces novel bidirectional spectral normalization inspired by Xavier initialization, which improved on LeCun initialization by controlling not only the variances of internal outputs, but also the variance of backpropagated gradients [13. We theoretically prove that BSSN mimics Xavier initialization to give better gradient control than SN. (b) BSSN introduces a new scaling of weights inspired by Kaiming initialization, a newer initialization technique that has better performance practice [17.. We show that BSSN achieve better sample quality and training stability than SN on several benchmark datasets, including CIFARIO, 10, STL10, CelebA, and ImageNet.
Note that our goal in this work is not to propose the best normalization or regularization technique for training GANs; there has been substantial work in this area already with promising results 48
Our goal is instead to understand why SN has been so effective, and evaluate whether these insights can inform more effective alternatives. Our theoretical and empirical results suggest that gradient control may play significant role in this story (though they need not be the only factor explaining SN's success).
34
20 11.
2 Background and Preliminaries
The instability of GANs is believed to be predominantly caused by poor discriminator learning 2 40 We therefore focus in this work on the discriminator, and the effects of SN on discriminator learning. We adopt the same model as [30]. Consider a discriminator with L internal layers:
Do(x) = az o ww a1.-1 o 1rre o... 01 o ho. (2:)
(1)
{wi, W2,...,, the weights; a, (i = 1...., L - 1) is the activation function in the i-th layer, which is usually element-wiss ReLU or leaky ReLU in GANs [14]. az is the activation function for the last layer, which is sigmoid for the vanilla GAN 14 and identity for WGAN-GP [16]; l.o is the linear transformation in i-th layer, which is usually fully-connected or a convolutional neural network [14 38). Like prior work on the
where x denotes the input to the discriminator and 0
theoretical analysis of (spectral) normalization [30 12 42 20) we do not model bias terms.

---

Lipschitz regularization and spectral normalization. Prior work has shown that regularizing the Lipschitz constant of the discriminator ||Dollaap improves the stability of GANs [6 50]. For
example, WGAN-GP [66 adds a gradient penalty (IIV 1)² to the loss function, where I. = oz + (1 - (z) and a ~ Uniform (0, 1) to ensure that the Lipschitz constant of the discriminator bounded by 1.
Spectral normalization (SN) takes a different approach. For fully connected layers (i.c., Lw, (x) = w; it regularizes the weights w; to ensure that spectral norm ||m:|] al = 1 for all E [1, L), where the spectral norm |wallsp is defined as the largest singular value of wt- This bounds the Lipschitz constant
of the discriminator since || > I1 I1 Ilaa II ||a Ilp S II ||wslllp II [eell..ppp V 1, as I| Ilai I || sp and ||a: 10p S 1 for networks with (leaky) ReL.U as activation functions for the internal layers and identity/sigmoid as the activation function for the last layer [30]. Prior work has theoretically connected the generalization gap of neural networks to the product of the spectral norms of the layers
32
20
48. These insights led to multiple implementations of spectral
normalization [12 15 51
30
20
48 with the implementation of [30] achieving particular success
on GANs. SN can be viewed as special case of more general techniques for enhancing the stability of neural network training by controlling the entire spectrum of network's 's input-output Jacobian 34] weight matrices [20]. or learned embeddings [48].
In practice, spectral normalization [12 (30] is implemented by dividing the weight matrix w; by its w; spectral norm: uu w.y: where u, and U2 are the left/right singular vectors of w/ corresponding to its largest singular value. As observed by Gouk et al. [1], there are two approaches in the SN literature for instantiating the matrix w/ for convolutional neural networks (CNNs). In a CNN, since convolution is a linear operation, convolutional layers can equivalently be written as a multiplication by am expanded weight matrix W; that derived from the raw weights w/, Hence in principle, spectral normalization should normalize each convolutional layer by ||x; sp [2). We call this canonical normalization SN as it controls the spectral norm of the convolution layer.
However, the spectral normalization that is known outperform other regularization techniques and improves training stability for GANs (30), which we call SNw, does not implement SN in a strict sense. Instead, it uses [ X (Cy kuk)) I1.;: that is, it first reshapes the convolution kernel
w, E Roott into a matrix dē of shape Cout x (Cin and then normalizes with the spectral norm / where Gin is the number of input channels, Cout is the number of output channels, hi is the kernel width, and hA is the kernel height. Miyato et al. showed that their implementation implicitly penalizes wi from being too sensitive in one specific direction (30)- However, this does not explain why SNw is more stable than other Lipschitz regularization techniques, and as observed in (15). it is unclear how SN relates to SN Despite this, SN has empirically been immensely successful in stabilizing the training of GANs [8] 28 [33 21 52 p1 Even more puzzling, we show in S that the canonical approach SN has comparatively poor out-of-the- box performance when training GANs.
Hence, two questions arise: (1) Why is SN so successful stabilizing the training of GANs? (2) Why is SN, proposed by 30 so much more effective than the canonical SN
related to two well-known phenomena: vanishing and exploding gradients. These terms describe a problem in which gradients either grow or shrink rapidly during training 66 35 36 7) and they are known to be closely related the instability of GANs 2 8. We provide an example to illustrate how vanishing or exploding gradients cause training instability in GANs in App.
In this work, show that both questions
3 Exploding Gradients
In this section, we show that spectral normalization prevents gradient explosion by bounding the gradients of the discriminator. Moreover, we show that the common choice to normalize all layers equally achieves the tightest upper bound for a restricted class of discriminators. We use O E R" to denote vector containing all elements (wi,..,, Wr.). In the following analysis, we assume linear transformations are fully-connected layers bn., (x) wyzr as in 30). though the same analysis can be applied to convolutional layers. Following prior work on the theoretical analysis of (spectral) normalization [30 12 42
we assume no bias in the network (i.c., Eq. (I) for simplicity.
3

---

To highlight the effects of the spectral norm of each layer on the gradient and simplify the exposition, we will compute gradients with respect w/
wy in the following discussion. In reality,
a." meye,
gradients are computed with respect to w;; we defer this discussion to App. C where we show the relevant extension.
How SN controls exploding gradients. The following proposition shows that under this simplifying assumption, spectral normalization controls the magnitudes of the gradients of the discriminator with respect to 0. Notice that simply controlling the Lipschitz constant of the discriminator (e.g,, as in WGAN (2]) does not imply this property; instead ensures small (sub)gradients with respect the input, æ.
Proposition 1 (Upper bound of gradient's ' Frobenius norm for spectral normalization). If |/0 ll S 1 [aolleip: and the norm of the overall
for all E [1, I,, then we have || ar Do(x))lr- S || I ||
gradient can be bounded by V>)l(>))] S vL ||>||| IT ||a:le
(Proof in App. AV. Note that under the assumption that internal activation functions are ReLU or leaky ReLU, if the activation function for the last layer is identity (e.g., for WGAN-GP [16]), the above bounds can be simplified to V|V [x) I ||>|| and [Va S vL ||=]| and if the activation for the last layer is sigmoid (e.g., for vanilla GAN (4), the above bounds become [|V,, wV,,Dag (p S 0.25 ||=||| and [D)) < 0.25v L x to limit the norm of the Hessian, which we defer to App. D
A comparable bound can also be found
The bound in Prop. I has significant effect in practice. Fig. 2] shows the norm of the gradient for each layer of GAN trained on MNIST with and without spectral nor- malization. Without spectral normalization, some layers have extremely large gradients throughout training, which makes the overall gradient large. With spectral normal- ization, the gradients of all layers are upper bounded shown in Prop. I We see similar results in other datasets and network architectures (App.
60
Norm gradient Direali
50
40
30
20
18 Jover
Eaeer
Dover
Exyer
Optimal spectral norm allocation. Common implemen- tations of SN advocate setting the spectral norm of each layer to the same value [30 [2]. However, the following proposition states that can set the spectral norms of different layers to different constants, without changing the network's behavior the input samples, long the product of the spectral norm bounds is the same.
Iterations
Figure 2: Gradient norms of each dis- criminator layer in MNIST.
0.. 00.. o bwy and D/ Proposition For any discriminator Dg- = dz, o lwg o a 9...
ar, o log wr, aal-1 o leg, 1 o... o legnng where the internal activation functions {ai) L-1 are ReLU or leaky ReLU, and positive constant scalars C1,.., CL satisfy that I C2 = 1, we have
D" 2nh (z)((z) Fr
D₂(x) = D2(()) Vx and
Vx, Vn Z+
Fx.
(Proof in App. By. Given this observation, it is natural to ask if there is any benefit to setting the spectral norms each layer equal. turns out that the answer is yes, under some assumptions that appear to approximately hold in practice. Let
W.. Do(x)lle
[1r5 |uslll
(ReLU, leaky ReLU) W.j e [..1).
A De dz. o kwy e... a an bun
(2)
By Do(+))
This intuitively describes the set of all discriminators for which scaling up the weight of one layer proportionally increases the gradient norm of all other layers; the definition of this is motivated by our upper bound on the gradient norm (App. A). The following theorem shows that when optimizing over set D, choosing every layer to have the same spectral norm gives the smallest possible gradient norm, for a given set of parameters.
Theorem 1. Consider a given set of discriminator parameters = {Wi,... WL). For a vector 1/1. denote the geometric mean (C ,ar...,,C } we denote 0. & (cowell Let d II ||w,|l//'

---

of the spectral norms of the weights. Then we have
A |
arg
min
[V, Do, (x)
[wlllp
Dog ED, 11 q= ORR
(Proof App. E.. The key constraint in this theorem is that we optimize only over discriminators in set 'D in Eq.. (2) To show that this constraint is realistic (i.e., SN GAN
y=x
discriminator optimization tends to choose models in D), we trained layers on MNIST, computing the ratios of the gradient norms at each layer and the ratios of the spectral norms, as dictated by Eq. (2) We computed these ratios at different epochs during training,
Ratio of gradiens norm
10a
layer 2 /layerl
layer 3 /layer1
-nommaalyy GAN with four hidden
layer 4 / /layerl
10a
well as for different randomly- selected rescalings of the spectral normalization vector Each point in Fig. 3 represents the results averaged over 64 real samples at specific epoch of training for a given (random) c. Vertical series of points are from different epochs of the same run, therefore their ratio of spectral norms the same. The fact that most of the points near the diagonal line suggests that training naturally favors discriminators that are in or near D; we confirm
10
10-1
10"
10a
Inverse ratio of spectral norm
Figure 3: Ratio of gradient norm v.s. in- verse ratio of spectral norm in MNIST.
this intuition in other experimental settings in App. K| This observation, combined with Thm. suggests that it is better to force the spectral norms of every layer to be equal. Hence, existing SN implementations 30 12 chose the correct, uniform normalization across layers upper bound discriminator's s gradients.
Implications other normallzattion/egulaizzzz section can also be used to show the same results for other normaaization/rguulrizaaooo techniques that control the spectral norm of weights, like weight normalization [4] and orthogonal regularization [9] However, these techniques do not necessarily exhibit the more important properties proved in [S for SN, and have some known drawbacks (see more discussion in 6).
techniques. Note that the analysis in this
Vanishing Gradients
An equally troublesome failure mode of GAN training is vanishing gradients [2]. Prior work has proposed new objective functions mitigate this problem 3 [6], but these approaches not Fig. IT). In this section, we show that SN also helps to control vanishing gradients.
fully effective
How SN controls vanishing gradients. Gradients tend to vanish for two reasons. First, gradients vanish when the objective function saturates 255 2]. which often associated with function pa- rameters growing too large. Common loss functions (e.g., hinge loss) and activation functions (e.g.. sigmoid, tanh) saturate for inputs of large magnitude. Large parameters tend amplify the inputs to the activation functions and/or loss functions, causing saturation. Second, gradients vanish when function parameters (and hence, internal outputs) grow too small. This is because backpropagated gradients are scaled by the function parameters (App. A).
These insights motivated the LeCun initialization technique 225 The key idea is that prevent gradients from vanishing, we must ensure that the outputs of each neuron do not vanish or explode. If the inputs to a neural unit are uncorrelated random variables with variance then to ensure that the unit's 's output also has variance (approximately) 1, the weight parameters should be zero-mean random variables with variance of L. where nı; denote the fan-in (number incoming connections) of layer (25]. Hence, LeCun initialization prevents gradient vanishing by controlling the variance of the individual parameters. In the following theorem, we show that SN enforces similar condition.
Theorem 2 (Parameter variance of SN). For a matrix A E B with i.i.d. entries 0ij from symmetric distribution with zero mean (e.g., zero-mean Gaussian or uniform), we have
Var
R2g
(3)
||||
m
5

---

Narm of graatemncal
10-
Inceptoo score
Sta (Mya atlll
Sh, flyya et al:
Stow Farris st
5km thrrii al)
brappton
Stiw Parris dibaaae 1.75)
Moeas Parris and jeove:1..00
(M. #OALL
100000
200000
300000
400000
100000
200000
300000
400000
1.5 2.0
Reradion
Rerations
Figure 4: Inception score of dif- Figure 5: Gradient norms of dif-
Figure 6: Inception score of
ferent SN variants CIFAR1O. ferent SN variants in CIFARIO.
scaled SN in CIFARIO.
Furthermore, if m, n 2 2 and max {m, n 2 3, and aij are from a zero-mean Gaussian, we have
max TI logi (og( S Var (TAIIT))
max mm.
where I is a constant which does not depend on m.
(Proof in App. FJ. Hence, spectral normalization forces scales inversely with max n). The proof relies on a characterization of extreme values of random vectors drawn from the surface of a high-dimensional unit ball. Many fully-connected, feed-forward neural networks have a fixed width across hidden layers, max (m, n) corresponds precisely the fan-in of any neuron in a hidden layer, implying that SN has an effect like LeCun initialization.
parameters to have a variance that
Why SN works better than SN In CNN, the interpretation of max m, n) depends on how SN is implemented. Recall that the implementation SN by (30] does not strictly implement SN, but a variant that normalizes by the spectral norm of w;; = w; Cour kakn
architectures like DCGAN (38) the larger dimension of w; for hidden layers tends to be Cin kin which is exactly the fan-in. This means that SN gets the right variance for hidden layers in CNN.
Perhaps surprisingly, we find empirically that the strict implementation SNcon of 12] does nor prevent gradient vanishing. Figs. 4 and [ shows the gradients of SN vanishing when trained on CIFAR10, leading to comparatively poor inception score, whereas the gradients of SN, remain stable. To understand this phenomenon, recall that SN normalizes by the spectral norm of an expanded matrix ui derived from wy. Thm. does not hold for wē; since its entries are not i.i.d. (even initialization); hence it cannot be used to explain this effect. However, Corollary in 47 shows that || < || sp S a ||rii|lap where is a constant only depends on kernel size, input size, and stride size of the convolution operation. This result has two implications:
45
also deduced special case of the second inequality.)
(1) || sp I a || Although SNw does not strictly normalize the matrix with the actual spectral norm of the layer, it does upper bound the spectral norm of the layer. Therefore, all our analysis in & still applies for SN, by changing the spectral norm constant from to a ||:i:|l.pp This means that SNw still prevent gradient explosion.
(2) || I Ilpp This implies that SN normalizes by a factor that least as large as SN In fact, we observe empirically that [| llgp is strictly larger than ||rū;]|, during training (App. L.3). This means that for the same w,, discriminator using SN will have smaller outputs than the discriminator using SN.. We hypothesize that the different scalings explain why SN has vanishing gradients but SN. does not.
To confirm this hypothesis, for SNw and SNcome, propose to multiply all the normalized weights by a scaling factor s, which is fixed throughout the training. Fig. shows that SN seems to be a shifted version of SN SNcomv with 1.75 has similar inception score (Fig. 4) to SN,, as well as similar gradients (ig.. 5) and parameter variances (App. L.4) throughout training. This, combined with Thm. 2) suggests that SNw inherently finds the correct scaling for the problem, whereas "proper" spectral normalization SN requires additional hyper-parameter tuning.
SN has good parameter variances throughout training. Our theoretical analysis only applies at initialization, when the parameters are selected randomly. However, unlike LeCun initialization which only controls the variance at initialization, we find empirically that that for SN appears to hold throughout training (Fig. 7). As a comparison, if trained without SN, the variance increases and the gradient decreases, which makes sample quality bad (App. L.2). This explains why in practice GANs trained with SN are stable throughout training. Formally extending our theoretical analysis to apply throughout training requires more complicated techniques, which we defer to future work.

---

Layer
Layer
Layer
Layer
Parameter varlance
0.01
0.00
Layer
Layer
Layer
Empirical
0.01
Theoretical
bound
0.00
200k
400k
200k
400k
200k
400k
Iterations
Figure 7: Parameter variances throughout training in CIFARIO. The blue lines show the parameter variances of different layers when SN is applied, and the original line shows our theoretical upper bound given in Eq. (3)
Extensions of Spectral Normalization
Given the above theoretical insights, we propose an extension of spectral normalization called Bidirectional Scaled Spectral Normalization (BSSN). It combines two key ideas: bidirectional normalization and weight scaling.
5.1 Bidirectional Normalization
Glorot and Bengio [3] built on the intuition of LeCun 25 to design an improved initialization, commonly called Xavier inilialization. Their key observation was that to limit gradient vanishing (and explosion), is not enough to control only feed-forward outputs; we should also control the variance of backpropagated gradients. Let nı, m. denote the fan-in and fan-out of layer i. (In fully-connected layers, n, = m = the width layer i.) Whereas LeCun chooses initial parameters with variance I. Glorot and Bengio choose them with variance 22, m. compromise between 1 (to control n,*
output variance) and 1. (to control variance of backpropagated gradients).
The first component of BSSN is Bidirectional Spectral Normalization (BSN), which applies a similar intuition to improve the spectral normalization of Miyato et al. [30]. For fully connected layers,
BSN keeps the normalization the same SN,, (30 For convolution layers, instead of normalizing
oooo x(e (.)
w x(Coue kw 5,)
by || Sout x (cinkhhks)) Isp we normalize by ow
where
We (oo kn) 1l is the spectral norm of the reshaped convolution kernel of dimension Can X ko))) For calculating these spectral norms, we use the same power iteration method in 30 The following theorem gives the theoretical explanation.
Theorem (Parameter variance of BSN). For convolutional kernel RCCue with i.i.d. entries Wij from a symmetric distribution with zero mean (e.g. zero-mean Gaussian or uniform) where kn 2 max
Cout Cin Cout
J.
and ow defined as above, we have
Wli
Var
Tw
Can kn + Couthiwkn
2 3, and Wij are from a zero-mean Gaussian distribution, there exists a constant T. that does not depend Cáns Cout; kin, Kn, such that
Furthermore, if Cins Cout 2 2 and Cin k Coukkaki
L
Wey
2
S Var
Cinkingki log(Cout) + Coutkunk log(cin)
Ten
Cin Kin Coutkynk
(Proof in App. G.. In convolution layers, n4 = bin and m, ko Therefore, BSN sets the variance of parameters to scale as -.-m;; as dictated by Xavier initialization. Moreover, BSN naturally inherits the benefits of SN discussed in (c.g., controlling variance throughout training).
2

---

5.2
Weight Scaling
The second component of BSSN is multiply all the normalized weights by constant scaling factor (i.e., as we did in Fig. 6). We call the combination of BSN and this weight scaling Bidirectional Scaled Spectral Normalization (BSSN). Note that scaling can also be applied independently to SN, which we call Scaled Spectral Normalization (SSN). The scaling is motivated by the following reasons.
(I) The analysis in LeCun and Xavier initialization assumes that the activation functions are linear, which is not true in practice. More recently, Kaiming initialization effect of non-lineat activations [17]. The result is that we should set the variance of parameters to be 2/(1 + a²) times the ones in LeCun Xavier initialization, where a is the negative slope of leaky ReLU. This suggests the importance of constant scaling.
proposed to include the
(2) However, we found that the scaling constants proposed in LeCun/Kaiming initialization do not always perform well for GANs. Even more surprisingly, there are multiple modes of good scaling. Fig. shows the sample quality of LeCun initialization with different scaling criminator. We see that there are at least two good modes of scaling: one at around 0.2 and the other at around 1.2. This phenomenon cannot be explained by the analysis in LeCun/Kaiming initialization.
Inceptiin score
6-
the dis-
LeCun initialiization
SN (Miyato
10-1
10a
Scale
Recall that SN has similar properties as LeCun initializa- tion (S 4). Interestingly, we see that SSN also has two good modes of scaling Fig. 8) Although the best scaling con- stants for LeCun initialization and SN are very different, there indeed exists an interesting mode correspondence in terms of parameter variances (App. Mp. We hypothesize that the shift of good scaling from Kaiming initialization here could result from adversarial training, and defer the theoretical analysis to future work. These results highlight the need for a separate scaling factor.
Figure 8: Inception score of SSN and scaled LeCun initialization in CIFARIO. with mean and standard error of the best score during training across mul- tiple runs.
(3) The bounds in Thm. 2 and Thm. 30only imply that in SN and BSN the order parameter variance w.r.t. the network size is correct, but constant scaling is unknown.
5.3
Results
In this section we verify the effectiveness of BSSN with extensive experiments. The code for reproducing the results is at https://githhbb
30 already compares SN with many other regularization techniques like WGAN-GP [6]. batch normalization 19), layer normalization [4. weight normalization [41), and orthogonal regularization [9. and SN is shown outperform them all. Therefore, compare BSSN with SN. To isolate the effects of the two components proposed in BSSN, we also compare against bidirectional normalization without scaling (BSN) and scaling without bidirectional normalization (SSN).
We conduct experiments across different public (non-sensitive) datasets (from low-resolution to high-resolution) and network architectures (from standard CNN to ResNets). More specifically, we conducts experiments on CIFARIO, 10, STL10, CclebA, and ImageNet (ISSVRC2012), following the same settings in [30] All experimental details are attached in Apps. N to
S The results are in Table
BSN v.S. SN (showing the effect of bidirectional normalization $ 5.1). (1) By comparing BSN with SN in Table I we can see that BSN outperforms SN by large margin in all metrics except ILSVRC2012 (discussed later). (2) More importantly, the superiority of BSN stable across syper-parameters. In App. N] we vary the learning rates (Ag. a.) and momentum parameters of generator and discriminator, and the number of discriminator updates per generator update (Mais). We see that BSN consistently outperforms SN in most of the cases. (3) Moreover, BSN is more stable in the entire training process. We see that as training proceeds, the sample quality of SN often drops, whereas the sample quality of BSN appears to monotonically increase FFgg. 9. more in Apps. P to R
8

---

CIFARIO
STL10
CelebA
ILSVRC2012
IS
FID
IS
FID
FID <
IS
FID J
Real data
11.26
9.70
26.70
10.17
4.44
197.37
15.62
SN
7.12
31.43
0.90
9.05
0.05
44.35
1 0.54
9.43
= 0.09
12.84
0.33
75.06 +: 2.38
SSN
7.38 1: 0.06
29.31
0.23
9.28
0.03
43.52
0.26
8.50
0.20
12.84
1 0.33
73.21 1: 1.92
BSN
7.54 0.04
26.94
0.58
9.25
0.01
42.98
+: 0.54
9.05 0.13
1.77 + 0.13
265.20 + 19.01
BSSN
7.54 0.04 26.94
0.58
9.25
0.01
42,90
0.17
9.05 + 0.13
13.23 0.16
69.04 + 1.46
Table I: Inception score (IS) and FID CIFARIO, STLIO, CelebA, and ILSVRC2012. The last three rows are proposed in this work. with BSSN representing our final proposal combination of BSN and SSN. Each experiment conducted with random seeds except that the last three rows on ILSVRC2012 is conducted with 3 random seeds. Mean and standard eπrоr across these random seeds are reported. We follow the common practice of excluding IS in CelebA as the inception network pretrained on ImageNet, which is different from CelebA, Bold font marks best numbers in a column.
BSN generally outperforms SN in final sample quality (i.e., the end of training), but also in peak sample quality. I.e., BSN stabilizes the training process, which is the purpose of SN (and BSN).
SSN v.s. SN (showing the effect of scaling S 5.2) Com- paring SSN with SN in Table I we see that scaling consis- tently improves (or has the same metric) in all cases. This verifies our intuition in S 5.2 that the inherent scaling in SN is not optimal, and extra constant scaling is needed for best results.
7.0
Inception coore
6.5
BSSN v.s. BSN (showing the effect of scaling S 5.2). By comparing BSSN with BSN in (Table T] we see that some cases the optimal scale of BSN happens to be (e.g.. in CI- FAR10), but in other cases, scaling is critical. For example, in ILSVRC2012, BSN without any scaling has the same gradient vanishing problem we observe for SN [12] in
5.5
5.0
BSS
100000
200000
300000
400000
Iterations
Figure 9: Inception score in CIFARIO. The results averaged over 5 random seeds, with ag = 0.0001, a, = 0.0001, Vidis 1.
$ 4 which causes bad sample quality. BSSN successfully solves the gradient vanishing problem and achieves the best sample quality.
Additional results. Due to space constraints, we defer other supplementary results (e.g., generated images, train- ing curves, more comparisons and analysis) to co/Apss. NotoS
Summary. In summary, both designs we proposed can effectively stabilize training and achieve better sample quality. Combining them together, BSSN achieves the best sample quality in most cases. This demonstrates the practical value of the theoretical insights in $ and
6 Discussion
Related work. Related to our upper bound, [42] shows that batch normalization (BN) reduces the Hessian's 's scale along the gradient direction, making gradients more predictive. Given Propp. can apply this reasoning to explain why
GANs are robust to different learning rates shown in (30 However, our insights regarding gradient vanishing are the more surprising It is unclear whether BN similarly controls vanishing gradients.
result, and not discussed in 42
In parallel to this work, some other approaches have been proposed to improve SN. For example, M finds out that even with SN, the condition numbers of the weights can still be large, which causes instability. To solve the issue, they borrow the insights from linear algebra and propose preconditioning layers to improve the condition numbers and promote stability.
Other reasons contributing to the stability of SN. This paper presents one possible reason (i.e., SN avoids exploding and vanishing gradients), and shows such a correlation through theoretical and empirical analysis. However, there could exist many other parallel factors. For example, (30 points out that SN could speed up training by encouraging the weights to be updated along directions orthogonal to itself. Our results do not shed light on these orthogonal hypotheses.

---

Implications on other normalizatoon/regllrizzaiii techniques. As discussed in S 3 other normalization techniques like weight normalization [4] and orthogonal regularization 9 also control the maximum gradient norm, shown by a simple extension of our results. However, they perform worse in practice [30]. We hypothesize two reasons: (1) They may not have the more important properties proved in S 4 For example, the official implementation of orthogonal regularization on CNN kernels [I] gives a parameter variance of ejnks." which is larger than the one in SNw: (2) They have known drawbacks like promoting less effective features [30]. Extending our analysis framework to explain these differences would be an interesting future work.
Future directions. Our results suggest that SN stabilizes GANs by controlling exploding and vanishing gradients in the discriminator. However, our analysis also applies to the training of any feed-forward neural network. This may explain why SN also helps train generators [ and neural networks more broadly [12 [1] [5. We focus on GANs because SN seems to disproportionately benefit them (3). Carefully understanding why is an interesting direction for future work.
Related to the weight initialization and training dynamics, recent work [37 43 has shown that Gaussian weights or ReLU activations cannot achieve dynamical isometry (all singular values of the network Jacobian near 1), desired property for training stability. Orthogonal weight initialization may be better at achieving the goal. We considered Gaussian weights and ReLU activations as they are the predominant implementations in GANs, but studying other networks may be useful too.
Ethics. Although our work focuses understanding GAN fundamentals, some downstream applications of GANs (e.g., spreading misinformation) are ethically problematic. We believe there is still value to studying GANs, which can also enable positive applications like better synthetic data.
Acknowledgements
This work was supported part by faculty research awards from Google, JP Morgan Chase, and the Sloan Foundation, as well as gift grants from Cisco and Siemens AG. This research was sponsored in part by National Science Foundation Convergence Accelerator award 2040675 and the U.S. Army Combat Capabilities Development Command Army Research Laboratory and was accomplished under Cooperative Agreement Number W91 INF-132-00045 (ARL Cyber Security CRA). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Combat Capabilities Development Command Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on. Zinan Lin acknowledges the support of the Siemens Makers Fellowship. the CMU Presidential Fellowship. and the Cylab Presidential Fellowship. This work used the Extreme Science and Engineering Discovery Environment (XSEDE) [46), which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges system (33, which is supported by NSF award number ACI- 1C1-644, the Pittsburgh Supercomputing Center (PSC).
References
[1] ajbrock. Neural photo editor. 2017.
Editor
[2] Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adver- sarial networks, 2017.
[3] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint 1701:07701.0777 2017.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607v:6007.000 2016.
[5] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. for neural networks. In Advances in Neural Information Processing Systems, pages 6240-6249, 2017.
margin bounds
[6] Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions neural networks, 5(2):157-166, 1994.
10

---

[7] Jeremy Bernstein, Arash Vahdat, Yisong Yue, and Ming- Yu Liu. On the distance between two neural networks and the stability of learning, arXiv preprint arXiv:2002.03432, 2020.
[8] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:/809.11000 2018.
[9] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. arXiv preprint arXiv: 160.00060, 2016.
[10] Adam Coates, Andrew Ng. and Honglak Lec. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-2233 2011.
[11] Tiantian Fang, Alex Schwing, and Ruoyu Sun. Precondition layer and its use for (gan Js, 2021.
[12] Farzan Farnia, Jesse M Zhang, and David Tse. Generalizable adversarial training via spectral normalization. arXiv preprint 1.I1.18111.044533 2018.
[13] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 256, 2010.
[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[15] Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael Cree. Regularisation of neural networks by enforcing lipschitz continuity. arXiv preprint arXiv: 04368, 2018.
[16] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in neural information processing systems, pages 5767-5777, 2017.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026- 1034, 2015.
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, pages 6626 6637,6667,55 2017.
[19] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint
2015.
[20] Haoming Jiang, Zhehui Chen, Minshuo Chen, Feng Liu, Dingding Wang, and Tuo Zhao. On computation and generalization of generative adversarial networks under spectrum control. In International Conference on Learning Representations, 2019.
[21] Alexia Jolicoeur-Martt Martineau. The relativistic discriminator: a key element missing from standard gan. arXiv preprint arXiv:1807.00733 2018.
[22] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint drXiv:1710.10196, 2017.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[24] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009.
[25] Yann LeCun, Leon Bottou, Genevieve B. Orr, and Klaus Robert Müller. Efficient BackProp. pages 9-50. Springer Berlin Heidelberg, Berlin, Heidelberg, 1998.
[26] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
11

---

[27] Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523 2018.
[28] Zinan Lin, Kiran Koshy Thekumparampil. Giulia Fanti, and Sewoong Oh. Infogan-cr: Disen- tangling generative adversarial networks with contrastive regularizers. ICML, 2020.
[29] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
[30] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: #rXXi:/802.05957, 2018.
[31] Takeru Miyato and Masanori Koyama. cgans with projection discriminator. arXiv preprint arXiv:1802.05637, 2018.
[32] Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalizee margin bounds for neural networks. arXiv preprint arXiv: 101017777000000 2017.
[33] Nicholas A. Nystrom, Michael Levine, Ralph Z.. Roskies, and Ray Scott. Bridges: A uniquely flexible hpe resource for new communities and data analytics. In Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure, XSEDE 15, pages 1-3:-,, New York, NY, USA, 2015, ACM
[34] Augustus Odena, Jacob Buckman, Catherine Olsson, Tom Brown, Christopher Olah, Colin Raffel, and Ian Goodfellow. Is generator conditioning causally related to gan performance? In International Conference on Machine Learning. pages 3849-3858. 2018.
[35] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient problem. CoRR, abs/1211.5063, 2:417, 2012.
[36] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference machine learning, pages 1310-1318, 2013.
[37] Jeffrey Pennington, Samuel S Schoenholz, and Surya Ganguli. Resurrecting the sigmoid deep learning through dynamical isometry: theory and practice. arXiv preprint arXiv: /to:://11.04777 1.04735, 2017.
[38] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06444 2015.
[39] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211 2015.
[40] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pages 2234-2242, 2016.
[41] Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterizztion to accelerate training of deep neural networks. In Advances in neural information processing systems, pages 901 909, 2016.
[42] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? In Advances in Neural Information Processing Systems, pages 2483-2493, 2018.
[43] Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint 131..613 2013.
[44] Yoav Seginer. The expected norm of random matrices. Combinatorics, Probability and Computing, 9(2):149-166, 2000.
12

---

[45] Sahil Singla and Soheil Feizi. Fantastic four: Differentiable and efficient bounds singular values of convolution layers. In International Conference on Learning Representations. 2021.
[46] J. Towns, T. Cockerill, M. Dahan, I. Foster, K. Gaither, A. Grimshaw, V. Hazlewood, S. Lathrop, D. Lifka, G. D. Peterson, R. Roskies, J. R. Scott, and N. Wilkins-Diehh Xsede: Accelerating scientific discovery. Computing in Science Engineering. 16(5):62-74, 2014.
[47] Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training; Scalable certification of perturbation invariance for deep neural networks. In Advances in Neural Information Processing Systems, pages 6541-6551 2018.
[48] Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, and Quanquan Gu. Improving neural language generation with spectrum control. International Conference on Learning Representations, 2019.
[49] David Warde-Farley and Yoshua Bengio. Improving generative adversarial networks with denoising feature matching. 2016.
[50] Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. Improving the improved train- ing wasserstein gans: A consistency term and effectt. arXiv preprint arXiv: 180::0003004 2018.
[51] Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generaliz- ability of deep learning. arXiv preprint arXiv: 705. /705. 10941, 2017.
[52] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. Proceedings of the IEEE International Conference on Computer Vision, pages 4471 2019.
[53] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint
2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See S 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] See 6
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See S and
(b) Did you include complete proofs of all theoretical results? [Yes] See App. A through
App. G
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL.)? [Yes] See https://github. http:tttuu...oooooooooeoeeo jommfjxmmmlma /BSN
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
were chosen)? [Yes] See App. through App. S as well as the code in https
//github. ".githuu.coo/// jfnmffxmmlzze jxmlzn/// BSN
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Tables [ 8
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See[App.
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
13

---

(a) If your work uses existing assets, did you cite the creators? [Yes] See App. H
(b) Did you mention the license of the assets? [Yes]
See https: / ttt///gttttt github..c.. com/
f jjmlzz/BBSW for code license
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
See the supplementary material
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We specify that our experiments are run on public datasets in
$ 5.3
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] We specify in 5.3 that public datasets do not contain sensitive data.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "mMkjS3ScRxc", "reviewer": "Reviewer_zyay", "summary": "The paper offers a more insightful analysis on Spectral Normalization (SN) and shows that SN controls two important failure modes of GAN: exploding and vanishing gradients. Specifically, the paper provides the theories to prove that SN upper-bounds the gradients and controls the variance of weights closely to LeCun initialization. Furthermore, given the new understanding of the connection between SN and LeCun initialization, the paper proposes a simple modification of SN to control both the variances of internal outputs and the variance of backpropagation gradients, which mimics the way of Xavier initialization. It also introduces new scaling weights, motivated by Kaming initialization, to obtain better performance in practice. The paper provides ablation studies and experiments on multiple benchmark datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\nThe paper is well-written, and contributions are verified with empirical experiments on benchmark datasets.  Understanding GAN is important, and this paper offers a useful contribution.\n\nCons\nThe analysis of gradient exploding and gradient vanishing have experimented on two different datasets: MNIST and CIFAR-10. Do both issues happen on the same dataset? It is good to verify these problems on other datasets as well.\n\nBSN failed on the ILSVRC2012 dataset, do the authors have any insights why? \n\nWhat GAN objective function is used in the implementation? In my experience, SN (Miyato et al.,) does work only well with hinge loss objectives. Do authors have any insights into that from the analysis? Also, does the improved SN proposed work well with other loss functions? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "Four hours"}, "extra_text": {}}, {"review_id": "VU4otRKb81W", "reviewer": "Reviewer_NDw4", "summary": "This paper provides insights towards understanding the stability of spectral normalization GANs (SNGANs), and proposes a new framework (BSSN) based on the proposed theoretical insights.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this is a well-written paper, it provides two theoretical insights on the stability of SNGAN: SN prevents exploding (in Sec. 3) and vanishing (in Sec. 4) gradients. Based on the theoretical analysis, this paper also proposes the BSSN framework that incorporates the theoretical insights. See detailed comments/questions below:\n\n1) In Theorem 1, how often does the constraint of $\\Pi_{i=1}^Lc_i=1$ hold in practice? Perhaps the author can provide some examples/justifications of this assumption?\n\n2) Comparing Theorem 2 and Theorem 3, it seems that the author believe that the BSSN is better than the SN because the BSSN can controls the variance at the scale of $\\frac{2}{(n_i+m_i)}$ while SN controls the variance at the scale of $\\frac{1}{\\max\\{n_i,m_i\\}}$, is there any theoretical justification that $\\frac{2}{(n_i+m_i)}$ is better than $\\frac{1}{\\max\\{n_i,m_i\\}}$?\n\n3) The reviewer appreciates the way that the authors present their theoretical results: each proposed theorem comes along with experiments that justifying their observations (Fig. 2-3 for Thm. 1 in Section 3; Fig. 4-6 for Thm. 2 in Section 4; and Fig. 7 for Thm. 3 in Section 5).\n\n4) The reviewer is not an expert in GANs experiments, but the reviewer finds the proposed BSSN framework indeed performs better under both FID and IS, and the numerical experiments seem reasonable and promising.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Perhaps the authors could improve the paper by answering/clarifying questions 1) and 2) mentioned in the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "Pt2nMsJt8v0", "reviewer": "Reviewer_oz6n", "summary": "This submission analyzes spectral normalization (SN), an widely used and effective method for improving GANs, from the viewpoint of its effect on the gradients of the parameters of the networks (as opposed to the gradients with regards to their inputs).\nDoing so, it shows that SN controls for vanishing and exploding gradients in the discriminator. It also provides a possible explanation as to why the usual implementation of SN is more effective than its \"formally correct\" counterpart.\nFinally, drawing links with the popular LeCun initialization, it proposes an improved variant of Spectral Normalization.\n\n\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nAnalyzing gradients is a natural and fairly well-studied way to consider spectral normalization. However, by specifically studying the weights of the discriminator, this work is able to find novel, surprising, results and applications, including links with initialization methods. While the related work paragraph is succinct, much of the actual related work is discussed thoroughly in the relevant sections of the paper.\n\nQuality:\nThe theoretical analysis seems of good quality. Experiments are comprehensive on multiple relevant datasets. The modifications for SN proposed in section 5 are less convincing. While theoretically strong, they seem to only result in a marginal (and not always consistent) improvement in practice.\n\nClarity:\nThe submission is clearly written and well organized.\n\nSignificance:\nThe paper provides some new understanding of Spectral Normalization, a crucial component of GANs. The findings are interesting theoretically and could lead to future improvements. Importantly, the paper opens a new direction for future work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are openly discussed.\nPotential societal impacts are adequately adressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "2YBzFsHx4jg", "reviewer": "Reviewer_8Yem", "summary": "The paper analyzes the benefits of the spectral norm on GAN: preventing gradient exploding and gradient vanishing. They prove that SN can control the variance of weights, which can prevent gradient vanishing, following the logic of LeCun initialization. They further propose a bidirectional SN, which is an improvement based on SN, following the logic of Xariver initialization. They conduct experiments on multiple datasets with DCGAN structure.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality (+): To my best knowledge, this work is the first one that connects SN with LeCun initialization and formally proves that SN can control the variance of weights and prevent gradient vanishing. But preventing gradient explosion is relatively well-known and less surprising. Overall, it can provide people a more theoretical understanding of SN.\n\nQuality: \n\nPros:\n\n1. I think the proof itself is sound, but I have some questions about Theorem 2 (see cons).\n\n2. Plots are clear and informative. \n\n3. I think it's a complete piece of work.\n\nCons: \n\n1. Theorem 2 doesn't provide a direct bound or analysis on the gradient, while Theorem 1 does. Instead, they analyze the variance of weights and justify the gradient vanishing in a roundabout way.\n\n2. In Theorem 2, there is a gap between assumption and real GAN training: LeCun init assumes the activation layers are symmetrical, while GAN usually uses LeakReLU, which lowers the variance of the input and then weakens the story.\n\n3. The lower bound in Theorem 2 could be far smaller than 1: 1/(16K^2 * max{m,n} * log(min{m,n})  (line 636). This means the variance of activation will get smaller and smaller. This situation will get worse with the effect of unsymmetrical activation layers. So from this bound, I cannot see the variance gets preserved, and the training can get saturated actually.\n\n4. From Fig. 4, the gradient norm does get preserved. So I concern if Theorem 2 is the correct way to prove this. \n\nClarity: The submission is clearly written and easy to follow.\n\nSignificance: \n\nPros: \n\n1. I think the message that SN can prevent gradient explosion and vanishing is significant and the message itself should be correct by a large chance. \n\n2. The proposed BSN and BSSN can improve the results of SN.\n\nCons: Besides the concerns mentioned above, the experiments have some flaws that may make researchers hesitate to use the proposed methods in their work.\n\n1. They only show the success with DCGAN structure, but not with more complex structures (e.g. ResNet and BigGAN) for the majority of the experiments. In the only experiment with the ResNet structure, BSN doesn't show the advantage. \n\n2. Tunning the weight scaling seems too cumbersome (i.e. pick the best from [0.7, 0.8, 0.9, 1.0, 1.2, 1.4, 1.6])\n\nI will raise my score if concerns get addressed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper mentions the limits and leaves them in future work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission discusses the benefits of spectral normalization for GAN training. Initial reviewer assessment was mostly positive while one reviewer was a bit more concerned. After a discussion with the authors the reviewer appreciated the contributions while remaining unconvinced that controlling the variance via Lecun initialization is the key for SN's success. AC thinks this paper provides interesting insights that could spur future research.
Decision: Accept (Poster)