Why Spectral Normalization Stabilizes GANs: Analysis and Improvements
Year: 2021

Abstract:
Spectral normalization (SN) [30] is a widely-used technique for improving the stability and sample quality of Generative Adversarial Networks (GANs). However, current understanding of SN’s efficacy is limited. In this work, we show that SN controls two important failure modes of GAN training: exploding and vanishing gradients. Our proofs illustrate a (perhaps unintentional) connection with the successful LeCun initialization [25]. This connection helps to explain why the most popular implementation of SN for GANs [30] requires no hyper-parameter tuning, whereas stricter implementations of SN [15, 12] have poor empirical performance out-of-the-box. Unlike LeCun initialization which only controls gradient vanishing at the beginning of training, SN preserves this property throughout training. Building on this theoretical understanding, we propose a new spectral normalization technique: Bidirectional Scaled Spectral Normalization (BSSN), which incorporates insights from later improvements to LeCun initialization: Xavier initialization [13] and Kaiming initialization [17]. Theoretically, we show that BSSN gives better gradient control than SN. Empirically, we demonstrate that it outperforms SN in sample quality and training stability on several benchmark datasets.

heading: 1 Introduction
text: Generative adversarial networks (GANs) are state-of-the-art deep generative models, perhaps best known for their ability to produce high-resolution, photorealistic images [14]. The objective of GANs is to produce random samples from a target data distribution, given only access to an initial set of training samples. This is achieved by learning two functions: a generator G, which maps random input noise to a generated sample, and a discriminator D, which tries to classify input samples as either real (i.e., from the training dataset) or fake (i.e., produced by the generator). In practice, these functions are implemented by deep neural networks (DNNs), and the competing generator and discriminator are trained in an alternating process known as adversarial training. Theoretically, given enough data and model capacity, GANs converge to the true underlying data distribution [14].
Although GANs have been very successful in improving the sample quality of data-driven generative models [22, 8], their adversarial training also contributes to instability. That is, small hyper-parameter changes and even randomness in the optimization can cause training to fail. Many approaches have been proposed for improving the stability of GANs, including different architectures [38, 22, 8], loss functions [2, 3, 16, 50], and various types of regularizations/normalizations [30, 9, 41]. One of the most successful proposals to date is called spectral normalization (SN) [30, 15, 12]. SN forces each layer of the discriminator to have unit spectral norm during training. This has the effect of controlling the Lipschitz constant of the discriminator, which is empirically observed to improve the stability of GAN training [30]. Despite the successful applications of SN [8, 28, 53, 21, 52, 31, 27], to date, it remains unclear precisely why this specific normalization is so effective.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
In this paper, we show that SN controls two important failure modes of GAN training: exploding gradients and vanishing gradients. These problems are well-known to cause instability in GANs [3, 8], leading either to bad local minima or stalled training prior to convergence. We make three primary contributions:
(1) Analysis of why SN avoids exploding gradients (§ 3). Poorly-chosen architectures and hyperparameters, as well as randomness during training, can amplify the effects of large gradients on training instability, ultimately leading to generalization error in the learned discriminator. We theoretically prove that SN upper bounds gradients during GAN training, mitigating these effects.
(2) Analysis of why SN avoids vanishing gradients (§ 4). Small gradients during training are known to cause GANs (and other DNNs) to converge to bad models [25, 3]. The well-known LeCun initialization, first proposed over two decades ago, mitigates this effect by carefully choosing the variance of the initial weights [25]. We prove theoretically that SN controls the variance of weights in a way that closely parallels LeCun initialization. Whereas LeCun initialization only controls the gradient vanishing problem at the beginning of training, we show empirically that SN preserves this property throughout training. Our analysis also explains why a strict implementation of SN [12] has poor out-of-the-box performance on GANs and requires additional tuning to avoid the vanishing gradient problem, whereas the implementation of SN in [30] requires no tuning.
(3) Improving SN with the above theoretical insights (§ 5). Given this new understanding of the connections between SN and LeCun initialization, we propose Bidirectional Scaled Spectral Normalization (BSSN), a simple modification of SN that combines two key insights (Fig. 1): (a) It introduces a novel bidirectional spectral normalization inspired by Xavier initialization, which improved on LeCun initialization by controlling not only the variances of internal outputs, but also the variance of backpropagated gradients [13]. We theoretically prove that BSSN mimics Xavier initialization to give better gradient control than SN. (b) BSSN introduces a new scaling of weights inspired by Kaiming initialization, a newer initialization technique that has better performance in practice [17]. We show that BSSN achieve better sample quality and training stability than SN on several benchmark datasets, including CIFAR10, STL10, CelebA, and ImageNet.
Note that our goal in this work is not to propose the best normalization or regularization technique for training GANs; there has been substantial work in this area already with promising results [34, 48, 20, 11]. Our goal is instead to understand why SN has been so effective, and evaluate whether these insights can inform more effective alternatives. Our theoretical and empirical results suggest that gradient control may play a significant role in this story (though they need not be the only factor explaining SN’s success).

heading: 2 Background and Preliminaries
text: The instability of GANs is believed to be predominantly caused by poor discriminator learning [2, 40]. We therefore focus in this work on the discriminator, and the effects of SN on discriminator learning. We adopt the same model as [30]. Consider a discriminator with L internal layers:
Dθ(x) = aL ◦ lwL ◦ aL−1 ◦ lwL−1 ◦ . . . ◦ a1 ◦ lw1(x) (1)
where x denotes the input to the discriminator and θ = {w1, w2, ..., wL} the weights; ai (i = 1, ..., L− 1) is the activation function in the i-th layer, which is usually element-wise ReLU or leaky ReLU in GANs [14]. aL is the activation function for the last layer, which is sigmoid for the vanilla GAN [14] and identity for WGAN-GP [16]; lwi is the linear transformation in i-th layer, which is usually fully-connected or a convolutional neural network [14, 38]. Like prior work on the theoretical analysis of (spectral) normalization [30, 12, 42, 20], we do not model bias terms.
Lipschitz regularization and spectral normalization. Prior work has shown that regularizing the Lipschitz constant of the discriminator ‖Dθ‖Lip improves the stability of GANs [3, 16, 50]. For example, WGAN-GP [16] adds a gradient penalty (‖∇Dθ(x̃)‖ − 1)2 to the loss function, where x̃ = αx + (1 − α)G(z) and α ∼ Uniform (0, 1) to ensure that the Lipschitz constant of the discriminator is bounded by 1.
Spectral normalization (SN) takes a different approach. For fully connected layers (i.e., lwi(x) = wix), it regularizes the weightswi to ensure that spectral norm ‖wi‖sp = 1 for all i ∈ [1, L], where the spectral norm ‖wi‖sp is defined as the largest singular value of wi. This bounds the Lipschitz constant of the discriminator since ‖Dθ‖Lip ≤ ∏L i=1 ‖lwi‖Lip · ∏L i=1 ‖ai‖Lip ≤ ∏L i=1 ‖wi‖sp · ∏L i=1 ‖ai‖Lip ≤ 1, as ‖lwi‖Lip ≤ ‖wi‖sp and ‖ai‖Lip ≤ 1 for networks with (leaky) ReLU as activation functions for the internal layers and identity/sigmoid as the activation function for the last layer [30]. Prior work has theoretically connected the generalization gap of neural networks to the product of the spectral norms of the layers [5, 32, 20, 48]. These insights led to multiple implementations of spectral normalization [12, 15, 51, 30, 20, 48], with the implementation of [30] achieving particular success on GANs. SN can be viewed as a special case of more general techniques for enhancing the stability of neural network training by controlling the entire spectrum of a network’s input-output Jacobian [34], weight matrices [20], or learned embeddings [48].
In practice, spectral normalization [12, 30] is implemented by dividing the weight matrix wi by its spectral norm: wi
uTi wivi , where ui and vi are the left/right singular vectors of wi corresponding to
its largest singular value. As observed by Gouk et al. [15], there are two approaches in the SN literature for instantiating the matrix wi for convolutional neural networks (CNNs). In a CNN, since convolution is a linear operation, convolutional layers can equivalently be written as a multiplication by an expanded weight matrix w̃i that is derived from the raw weights wi. Hence in principle, spectral normalization should normalize each convolutional layer by ‖w̃i‖sp [15, 12]. We call this canonical normalization SNConv as it controls the spectral norm of the convolution layer.
However, the spectral normalization that is known to outperform other regularization techniques and improves training stability for GANs [30], which we call SNw, does not implement SN in a strict sense. Instead, it uses ∥∥wicout×(cinkwkh)∥∥sp; that is, it first reshapes the convolution kernel wi ∈ Rcoutcinkwkh into a matrix ŵi of shape cout× (cinkwkh), and then normalizes with the spectral norm ‖ŵi‖sp, where cin is the number of input channels, cout is the number of output channels, kw is the kernel width, and kh is the kernel height. Miyato et al. showed that their implementation implicitly penalizes wi from being too sensitive in one specific direction [30]. However, this does not explain why SNw is more stable than other Lipschitz regularization techniques, and as observed in [15], it is unclear how SNw relates to SNConv. Despite this, SNw has empirically been immensely successful in stabilizing the training of GANs [8, 28, 53, 21, 52, 31, 27]. Even more puzzling, we show in § 4 that the canonical approach SNConv has comparatively poor out-of-the-box performance when training GANs.
Hence, two questions arise: (1) Why is SN so successful at stabilizing the training of GANs? (2) Why is SNw proposed by [30] so much more effective than the canonical SNConv?
In this work, we show that both questions are related to two well-known phenomena: vanishing and exploding gradients. These terms describe a problem in which gradients either grow or shrink rapidly during training [6, 35, 36, 7], and they are known to be closely related to the instability of GANs [2, 8]. We provide an example to illustrate how vanishing or exploding gradients cause training instability in GANs in App. I.

heading: 3 Exploding Gradients
text: In this section, we show that spectral normalization prevents gradient explosion by bounding the gradients of the discriminator. Moreover, we show that the common choice to normalize all layers equally achieves the tightest upper bound for a restricted class of discriminators. We use θ ∈ Rd to denote a vector containing all elements in {w1, ..., wL}. In the following analysis, we assume linear transformations are fully-connected layers lwi(x) = wix as in [30], though the same analysis can be applied to convolutional layers. Following prior work on the theoretical analysis of (spectral) normalization [30, 12, 42], we assume no bias in the network (i.e., Eq. (1)) for simplicity.
To highlight the effects of the spectral norm of each layer on the gradient and simplify the exposition, we will compute gradients with respect to w′i =
wi uTi wivi in the following discussion. In reality, gradients are computed with respect to wi; we defer this discussion to App. C, where we show the relevant extension.
How SN controls exploding gradients. The following proposition shows that under this simplifying assumption, spectral normalization controls the magnitudes of the gradients of the discriminator with respect to θ. Notice that simply controlling the Lipschitz constant of the discriminator (e.g., as in WGAN [2]) does not imply this property; it instead ensures small (sub)gradients with respect to the input, x. Proposition 1 (Upper bound of gradient’s Frobenius norm for spectral normalization). If ‖wi‖sp ≤ 1 for all i ∈ [1, L], then we have ‖∇wtDθ(x)‖F ≤ ‖x‖ ∏L i=1 ‖ai‖Lip , and the norm of the overall gradient can be bounded by ‖∇θDθ(x)‖F ≤ √ L ‖x‖ ∏L i=1 ‖ai‖Lip .
(Proof in App. A). Note that under the assumption that internal activation functions are ReLU or leaky ReLU, if the activation function for the last layer is identity (e.g., for WGAN-GP [16]), the above bounds can be simplified to ‖∇wtDθ(x)‖F ≤ ‖x‖ and ‖∇θDθ(x)‖ ≤ √ L ‖x‖, and if the activation for the last layer is sigmoid (e.g., for vanilla GAN [14]), the above bounds become ‖∇wtDθ(x)‖F ≤ 0.25 ‖x‖ and ‖∇θDθ(x)‖ ≤ 0.25 √ L ‖x‖. A comparable bound can also be found to limit the norm of the Hessian, which we defer to App. D.
The bound in Prop. 1 has a significant effect in practice. Fig. 2 shows the norm of the gradient for each layer of a GAN trained on MNIST with and without spectral normalization. Without spectral normalization, some layers have extremely large gradients throughout training, which makes the overall gradient large. With spectral normalization, the gradients of all layers are upper bounded as shown in Prop. 1. We see similar results in other datasets and network architectures (App. J).
Optimal spectral norm allocation. Common implementations of SN advocate setting the spectral norm of each layer to the same value [30, 12]. However, the following proposition states that we can set the spectral norms of different layers to different constants, without changing the network’s behavior on the input samples, as long as the product of the spectral norm bounds is the same.
Proposition 2. For any discriminator Dθ = aL ◦ lwL ◦ aL−1 ◦ lwL−1 ◦ . . . ◦ a1 ◦ lw1 and D′θ = aL ◦ lcL·wL ◦ aL−1 ◦ lcL−1·wL−1 ◦ . . . ◦ a1 ◦ lc1·w1 where the internal activation functions {ai} L−1 i=1
are ReLU or leaky ReLU, and positive constant scalars c1, ..., cL satisfy that ∏L i=1 ci = 1, we have
Dθ(x) = D ′ θ(x) ∀x and
∂nDθ(x) ∂xn = ∂nD′θ(x) ∂xn ∀x, ∀n ∈ Z+ .
(Proof in App. B). Given this observation, it is natural to ask if there is any benefit to setting the spectral norms of each layer equal. It turns out that the answer is yes, under some assumptions that appear to approximately hold in practice. Let
D , { Dθ = aL ◦ lwL ◦ . . . ◦ a1 ◦ lw1 :
‖∇wiDθ(x)‖F∥∥∇wjDθ(x)∥∥F = ‖wj‖sp ‖wi‖sp
, ai ∈ {ReLU, leaky ReLU} ∀i, j ∈ [1, L] } . (2)
This intuitively describes the set of all discriminators for which scaling up the weight of one layer proportionally increases the gradient norm of all other layers; the definition of this set is motivated by our upper bound on the gradient norm (App. A). The following theorem shows that when optimizing over set D, choosing every layer to have the same spectral norm gives the smallest possible gradient norm, for a given set of parameters. Theorem 1. Consider a given set of discriminator parameters θ = {w1, ..., wL}. For a vector c = {c1, . . . , cL}, we denote θc , {ctwt}Lt=1. Let λθ = ∏L i=1 ‖wi‖ 1/L sp denote the geometric mean
of the spectral norms of the weights. Then we have{ λθ ‖w1‖sp , . . . , λθ ‖wL‖sp } = arg min c: Dθc∈D, ∏L i=1 ci=1, ci∈R+ ‖∇θcDθc(x)‖F
(Proof in App. E). The key constraint in this theorem is that we optimize only over discriminators in set D in Eq. (2). To show that this constraint is realistic (i.e., SN GAN discriminator optimization tends to choose models in D), we trained a spectrally-normalized GAN with four hidden layers on MNIST, computing the ratios of the gradient norms at each layer and the ratios of the spectral norms, as dictated by Eq. (2). We computed these ratios at different epochs during training, as well as for different randomlyselected rescalings of the spectral normalization vector c. Each point in Fig. 3 represents the results averaged over 64 real samples at a specific epoch of training for a given (random) c. Vertical series of points are from different epochs of the same run, therefore their ratio of spectral norms is the same. The fact that most of the points are near the diagonal line suggests that training naturally favors discriminators that are in or near D; we confirm
this intuition in other experimental settings in App. K. This observation, combined with Thm. 1, suggests that it is better to force the spectral norms of every layer to be equal. Hence, existing SN implementations [30, 12] chose the correct, uniform normalization across layers to upper bound discriminator’s gradients.
Implications on other normalization/regularization techniques. Note that the analysis in this section can also be used to show the same results for other normalization/regularization techniques that control the spectral norm of weights, like weight normalization [41] and orthogonal regularization [9]. However, these techniques do not necessarily exhibit the more important properties proved in § 4 for SN, and have some known drawbacks (see more discussion in § 6).

heading: 4 Vanishing Gradients
text: An equally troublesome failure mode of GAN training is vanishing gradients [2]. Prior work has proposed new objective functions to mitigate this problem [2, 3, 16], but these approaches are not fully effective (Fig. 11). In this section, we show that SN also helps to control vanishing gradients.
How SN controls vanishing gradients. Gradients tend to vanish for two reasons. First, gradients vanish when the objective function saturates [25, 2], which is often associated with function parameters growing too large. Common loss functions (e.g., hinge loss) and activation functions (e.g., sigmoid, tanh) saturate for inputs of large magnitude. Large parameters tend to amplify the inputs to the activation functions and/or loss functions, causing saturation. Second, gradients vanish when function parameters (and hence, internal outputs) grow too small. This is because backpropagated gradients are scaled by the function parameters (App. A).
These insights motivated the LeCun initialization technique [25]. The key idea is that to prevent gradients from vanishing, we must ensure that the outputs of each neuron do not vanish or explode. If the inputs to a neural unit are uncorrelated random variables with variance 1, then to ensure that the unit’s output also has variance (approximately) 1, the weight parameters should be zero-mean random variables with variance of 1ni , where ni denote the fan-in (number of incoming connections) of layer i [25]. Hence, LeCun initialization prevents gradient vanishing by controlling the variance of the individual parameters. In the following theorem, we show that SN enforces a similar condition.
Theorem 2 (Parameter variance of SN). For a matrix A ∈ Rm×n with i.i.d. entries aij from a symmetric distribution with zero mean (e.g., zero-mean Gaussian or uniform), we have
Var (
aij ‖A‖sp ) ≤ 1max{m,n} . (3)
0 100000 200000 300000 400000 Iterations
2
4
6
In ce
pt io
n sc
or e
SNw (Miyato et al.) SNConv (Farnia et al.) SNConv (Farnia et al.) (scale=1.75)
Figure 4: Inception score of different SN variants in CIFAR10. 0 100000 200000 300000 400000 Iterations
100
101
102
No rm
o f g
ra di
en t o f D( re al )
SNw (Miyato et al.) SNConv (Farnia et al.) SNConv (Farnia et al.) (scale=1.75)
Figure 5: Gradient norms of different SN variants in CIFAR10. 0.5 1.0 1.5 2.0 3.0 4.0 Scale
Furthermore, if m,n ≥ 2 and max {m,n} ≥ 3, and aij are from a zero-mean Gaussian, we have
L max{m,n} log(min{m,n}) ≤ Var ( aij ‖A‖sp ) ≤ 1max{m,n} ,
where L is a constant which does not depend on m,n.
(Proof in App. F). Hence, spectral normalization forces zero-mean parameters to have a variance that scales inversely with max{m,n}. The proof relies on a characterization of extreme values of random vectors drawn from the surface of a high-dimensional unit ball. Many fully-connected, feed-forward neural networks have a fixed width across hidden layers, so max{m,n} corresponds precisely to the fan-in of any neuron in a hidden layer, implying that SN has an effect like LeCun initialization.
Why SNw works better than SNConv. In a CNN, the interpretation of max{m,n} depends on how SN is implemented. Recall that the implementation SNw by [30] does not strictly implement SN, but a variant that normalizes by the spectral norm of ŵi = wicout×(cinkwkh). In architectures like DCGAN [38], the larger dimension of ŵi for hidden layers tends to be cinkwkh, which is exactly the fan-in. This means that SN gets the right variance for hidden layers in CNN.
Perhaps surprisingly, we find empirically that the strict implementation SNConv of [12] does not prevent gradient vanishing. Figs. 4 and 5 shows the gradients of SNConv vanishing when trained on CIFAR10, leading to a comparatively poor inception score, whereas the gradients of SNw remain stable. To understand this phenomenon, recall that SNConv normalizes by the spectral norm of an expanded matrix w̃i derived from wi. Thm. 2 does not hold for w̃i since its entries are not i.i.d. (even at initialization); hence it cannot be used to explain this effect. However, Corollary 1 in [47] shows that ‖ŵi‖sp ≤ ‖w̃i‖sp ≤ α ‖ŵi‖sp, where α is a constant only depends on kernel size, input size, and stride size of the convolution operation. ([45] also deduced a special case of the second inequality.) This result has two implications:
(1) ‖w̃i‖sp ≤ α ‖ŵi‖sp: Although SNw does not strictly normalize the matrix with the actual spectral norm of the layer, it does upper bound the spectral norm of the layer. Therefore, all our analysis in § 3 still applies for SNw by changing the spectral norm constant from 1 to α ‖ŵi‖sp. This means that SNw can still prevent gradient explosion.
(2) ‖ŵi‖sp ≤ ‖w̃i‖sp: This implies that SNConv normalizes by a factor that is at least as large as SNw. In fact, we observe empirically that ‖w̃i‖sp is strictly larger than ‖ŵi‖sp during training (App. L.3). This means that for the same wi, a discriminator using SNConv will have smaller outputs than the discriminator using SNw. We hypothesize that the different scalings explain why SNConv has vanishing gradients but SNw does not.
To confirm this hypothesis, for SNw and SNConv, we propose to multiply all the normalized weights by a scaling factor s, which is fixed throughout the training. Fig. 6 shows that SNConv seems to be a shifted version of SNw. SNConv with s = 1.75 has similar inception score (Fig. 4) to SNw, as well as similar gradients (Fig. 5) and parameter variances (App. L.4) throughout training. This, combined with Thm. 2, suggests that SNw inherently finds the correct scaling for the problem, whereas “proper" spectral normalization SNConv requires additional hyper-parameter tuning.
SN has good parameter variances throughout training. Our theoretical analysis only applies at initialization, when the parameters are selected randomly. However, unlike LeCun initialization which only controls the variance at initialization, we find empirically that Eq. (3) for SN appears to hold throughout training (Fig. 7). As a comparison, if trained without SN, the variance increases and the gradient decreases, which makes sample quality bad (App. L.2). This explains why in practice GANs trained with SN are stable throughout training. Formally extending our theoretical analysis to apply throughout training requires more complicated techniques, which we defer to future work.

heading: 5 Extensions of Spectral Normalization
text: Given the above theoretical insights, we propose an extension of spectral normalization called Bidirectional Scaled Spectral Normalization (BSSN). It combines two key ideas: bidirectional normalization and weight scaling.

heading: 5.1 Bidirectional Normalization
text: Glorot and Bengio [13] built on the intuition of LeCun [25] to design an improved initialization, commonly called Xavier initialization. Their key observation was that to limit gradient vanishing (and explosion), it is not enough to control only feed-forward outputs; we should also control the variance of backpropagated gradients. Let ni,mi denote the fan-in and fan-out of layer i. (In fully-connected layers, ni = mi−1 = the width of layer i.) Whereas LeCun chooses initial parameters with variance 1 ni , Glorot and Bengio choose them with variance 2ni+mi , a compromise between 1 ni
(to control output variance) and 1mi (to control variance of backpropagated gradients).
The first component of BSSN is Bidirectional Spectral Normalization (BSN), which applies a similar intuition to improve the spectral normalization of Miyato et al. [30]. For fully connected layers, BSN keeps the normalization the same as SNw [30]. For convolution layers, instead of normalizing
by ∥∥wcout×(cinkwkh)∥∥sp, we normalize by σw , ∥∥∥wcout×(cinkwkh)∥∥∥ sp + ∥∥∥wcin×(coutkwkh)∥∥∥ sp
2 , where∥∥wcin×(coutkwkh)∥∥sp is the spectral norm of the reshaped convolution kernel of dimension cin × (coutkwkh). For calculating these two spectral norms, we use the same power iteration method in [30]. The following theorem gives the theoretical explanation.
Theorem 3 (Parameter variance of BSN). For a convolutional kernel w ∈ Rcoutcinkwkh with i.i.d. entries wij from a symmetric distribution with zero mean (e.g. zero-mean Gaussian or uniform) where kwkh ≥ max { cout cin , cincout } , and σw defined as above, we have
Var ( wij σw ) ≤ 2 cinkwkh + coutkwkh .
Furthermore, if cin, cout ≥ 2 and cinkwkh, coutkwkh ≥ 3, and wij are from a zero-mean Gaussian distribution, there exists a constant L that does not depend on cin, cout, kw, kh such that
L
cinkwkh log(cout) + coutkwkh log(cin) ≤ Var ( wij σw ) ≤ 2 cinkwkh + coutkwkh .
(Proof in App. G). In convolution layers, ni = cinkwkh and mi = coutkwkh. Therefore, BSN sets the variance of parameters to scale as 2ni+mi , as dictated by Xavier initialization. Moreover, BSN naturally inherits the benefits of SN discussed in § 4 (e.g., controlling variance throughout training).

heading: 5.2 Weight Scaling
text: The second component of BSSN is to multiply all the normalized weights by a constant scaling factor (i.e., as we did in Fig. 6). We call the combination of BSN and this weight scaling Bidirectional Scaled Spectral Normalization (BSSN). Note that scaling can also be applied independently to SN, which we call Scaled Spectral Normalization (SSN). The scaling is motivated by the following reasons.
(1) The analysis in LeCun and Xavier initialization assumes that the activation functions are linear, which is not true in practice. More recently, Kaiming initialization was proposed to include the effect of non-linear activations [17]. The result is that we should set the variance of parameters to be 2/(1 + a2) times the ones in LeCun or Xavier initialization, where a is the negative slope of leaky ReLU. This suggests the importance of a constant scaling.
(2) However, we found that the scaling constants proposed in LeCun/Kaiming initialization do not always perform well for GANs. Even more surprisingly, there are multiple modes of good scaling. Fig. 8 shows the sample quality of LeCun initialization with different scaling on the discriminator. We see that there are at least two good modes of scaling: one at around 0.2 and the other at around 1.2. This phenomenon cannot be explained by the analysis in LeCun/Kaiming initialization.
Recall that SN has similar properties as LeCun initialization (§ 4). Interestingly, we see that SSN also has two good modes of scaling (Fig. 8). Although the best scaling constants for LeCun initialization and SN are very different, there indeed exists an interesting mode correspondence in terms of parameter variances (App. M). We hypothesize that the shift of good scaling from Kaiming initialization we see here could result from adversarial training, and
defer the theoretical analysis to future work. These results highlight the need for a separate scaling factor.
(3) The bounds in Thm. 2 and Thm. 3 only imply that in SN and BSN the order of parameter variance w.r.t. the network size is correct, but constant scaling is unknown.

heading: 5.3 Results
text: In this section we verify the effectiveness of BSSN with extensive experiments. The code for reproducing the results is at https://github.com/fjxmlzn/BSN.
[30] already compares SN with many other regularization techniques like WGAN-GP [16], batch normalization [19], layer normalization [4], weight normalization [41], and orthogonal regularization [9], and SN is shown to outperform them all. Therefore, we compare BSSN with SN. To isolate the effects of the two components proposed in BSSN, we also compare against bidirectional normalization without scaling (BSN) and scaling without bidirectional normalization (SSN).
We conduct experiments across different public (non-sensitive) datasets (from low-resolution to high-resolution) and network architectures (from standard CNN to ResNets). More specifically, we conducts experiments on CIFAR10, STL10, CelebA, and ImageNet (ILSVRC2012), following the same settings in [30]. All experimental details are attached in Apps. N to S. The results are in Table 1.
BSN v.s. SN (showing the effect of bidirectional normalization § 5.1). (1) By comparing BSN with SN in Table 1, we can see that BSN outperforms SN by a large margin in all metrics except in ILSVRC2012 (discussed later). (2) More importantly, the superiority of BSN is stable across hyper-parameters. In App. N, we vary the learning rates (αg, αd) and momentum parameters of generator and discriminator, and the number of discriminator updates per generator update (ndis). We see that BSN consistently outperforms SN in most of the cases. (3) Moreover, BSN is more stable in the entire training process. We see that as training proceeds, the sample quality of SN often drops, whereas the sample quality of BSN appears to monotonically increase (Fig. 9, more in Apps. P to R).
BSN generally outperforms SN in final sample quality (i.e., at the end of training), but also in peak sample quality. I.e., BSN stabilizes the training process, which is the purpose of SN (and BSN).
SSN v.s. SN (showing the effect of scaling § 5.2). Comparing SSN with SN in Table 1, we see that scaling consistently improves (or has the same metric) in all cases. This verifies our intuition in § 5.2 that the inherent scaling in SN is not optimal, and an extra constant scaling is needed for best results.
BSSN v.s. BSN (showing the effect of scaling § 5.2). By comparing BSSN with BSN in Table 1, we see that in some cases the optimal scale of BSN happens to be 1 (e.g., in CIFAR10), but in other cases, scaling is critical. For example, in ILSVRC2012, BSN without any scaling has the same gradient vanishing problem we observe for SNConv [12] in § 4, which causes bad sample quality. BSSN successfully solves the gradient vanishing problem and achieves the best sample quality.
Additional results. Due to space constraints, we defer other supplementary results (e.g., generated images, training curves, more comparisons and analysis) to Apps. N to S.
Summary. In summary, both designs we proposed can effectively stabilize training and achieve better sample quality. Combining them together, BSSN achieves the best sample quality in most cases. This demonstrates the practical value of the theoretical insights in § 3 and 4.

heading: 6 Discussion
text: Related work. Related to our upper bound, [42] shows that batch normalization (BN) reduces the Hessian’s scale along the gradient direction, making gradients more predictive. Given Prop. 1, we can apply this reasoning to explain why spectrally-normalized GANs are robust to different learning rates as shown in [30]. However, our insights regarding gradient vanishing are the more surprising result, and not discussed in [42]. It is unclear whether BN similarly controls vanishing gradients.
In parallel to this work, some other approaches have been proposed to improve SN. For example, [11] finds out that even with SN, the condition numbers of the weights can still be large, which causes instability. To solve the issue, they borrow the insights from linear algebra and propose preconditioning layers to improve the condition numbers and promote stability.
Other reasons contributing to the stability of SN. This paper presents one possible reason (i.e., SN avoids exploding and vanishing gradients), and shows such a correlation through theoretical and empirical analysis. However, there could exist many other parallel factors. For example, [30] points out that SN could speed up training by encouraging the weights to be updated along directions orthogonal to itself. Our results do not shed light on these orthogonal hypotheses.
Implications on other normalization/regularization techniques. As discussed in § 3, other normalization techniques like weight normalization [41] and orthogonal regularization [9] also control the maximum gradient norm, shown by a simple extension of our results. However, they perform worse in practice [30]. We hypothesize two reasons: (1) They may not have the more important properties proved in § 4. For example, the official implementation of orthogonal regularization on CNN kernels [1] gives a parameter variance of 1cinkw , which is larger than the one in SNw; (2) They have known drawbacks like promoting less effective features [30]. Extending our analysis framework to explain these differences would be an interesting future work.
Future directions. Our results suggest that SN stabilizes GANs by controlling exploding and vanishing gradients in the discriminator. However, our analysis also applies to the training of any feed-forward neural network. This may explain why SN also helps train generators [53, 8] and neural networks more broadly [12, 15, 51]. We focus on GANs because SN seems to disproportionately benefit them [30]. Carefully understanding why is an interesting direction for future work.
Related to the weight initialization and training dynamics, recent work [37, 43] has shown that Gaussian weights or ReLU activations cannot achieve dynamical isometry (all singular values of the network Jacobian near 1), a desired property for training stability. Orthogonal weight initialization may be better at achieving the goal. We considered Gaussian weights and ReLU activations as they are the predominant implementations in GANs, but studying other networks may be useful too.
Ethics. Although our work focuses on understanding GAN fundamentals, some downstream applications of GANs (e.g., spreading misinformation) are ethically problematic. We believe there is still value to studying GANs, which can also enable positive applications like better synthetic data.

heading: Acknowledgements
text: This work was supported in part by faculty research awards from Google, JP Morgan Chase, and the Sloan Foundation, as well as gift grants from Cisco and Siemens AG. This research was sponsored in part by National Science Foundation Convergence Accelerator award 2040675 and the U.S. Army Combat Capabilities Development Command Army Research Laboratory and was accomplished under Cooperative Agreement Number W911NF-13-2-0045 (ARL Cyber Security CRA). The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Combat Capabilities Development Command Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on. Zinan Lin acknowledges the support of the Siemens FutureMakers Fellowship, the CMU Presidential Fellowship, and the Cylab Presidential Fellowship. This work used the Extreme Science and Engineering Discovery Environment (XSEDE) [46], which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges system [33], which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC).

Review summaries:
The paper analyzes the benefits of the spectral norm on GAN: preventing gradient exploding and gradient vanishing. They prove that SN can control the variance of weights, which can prevent gradient vanishing, following the logic of LeCun initialization. They further propose a bidirectional SN, which is an improvement based on SN, following the logic of Xariver initialization. They conduct experiments on multiple datasets with DCGAN structure.

This submission analyzes spectral normalization (SN), an widely used and effective method for improving GANs, from the viewpoint of its effect on the gradients of the parameters of the networks (as opposed to the gradients with regards to their inputs). Doing so, it shows that SN controls for vanishing and exploding gradients in the discriminator. It also provides a possible explanation as to why the usual implementation of SN is more effective than its "formally correct" counterpart. Finally, drawing links with the popular LeCun initialization, it proposes an improved variant of Spectral Normalization.    

The paper offers a more insightful analysis on Spectral Normalization (SN) and shows that SN controls two important failure modes of GAN: exploding and vanishing gradients. Specifically, the paper provides the theories to prove that SN upper-bounds the gradients and controls the variance of weights closely to LeCun initialization. Furthermore, given the new understanding of the connection between SN and LeCun initialization, the paper proposes a simple modification of SN to control both the variances of internal outputs and the variance of backpropagation gradients, which mimics the way of Xavier initialization. It also introduces new scaling weights, motivated by Kaming initialization, to obtain better performance in practice. The paper provides ablation studies and experiments on multiple benchmark datasets.