Why Spectral Normalization Stabilizes GANs: Analysis and Improvements

1 Introduction . Generative adversarial networks ( GANs ) are state-of-the-art deep generative models , perhaps best known for their ability to produce high-resolution , photorealistic images [ 14 ] . The objective of GANs is to produce random samples from a target data distribution , given only access to an initial set of training samples . This is achieved by learning two functions : a generator G , which maps random input noise to a generated sample , and a discriminator D , which tries to classify input samples as either real ( i.e. , from the training dataset ) or fake ( i.e. , produced by the generator ) . In practice , these functions are implemented by deep neural networks ( DNNs ) , and the competing generator and discriminator are trained in an alternating process known as adversarial training . Theoretically , given enough data and model capacity , GANs converge to the true underlying data distribution [ 14 ] . Although GANs have been very successful in improving the sample quality of data-driven generative models [ 22 , 8 ] , their adversarial training also contributes to instability . That is , small hyper-parameter changes and even randomness in the optimization can cause training to fail . Many approaches have been proposed for improving the stability of GANs , including different architectures [ 38 , 22 , 8 ] , loss functions [ 2 , 3 , 16 , 50 ] , and various types of regularizations/normalizations [ 30 , 9 , 41 ] . One of the most successful proposals to date is called spectral normalization ( SN ) [ 30 , 15 , 12 ] . SN forces each layer of the discriminator to have unit spectral norm during training . This has the effect of controlling the Lipschitz constant of the discriminator , which is empirically observed to improve the stability of GAN training [ 30 ] . Despite the successful applications of SN [ 8 , 28 , 53 , 21 , 52 , 31 , 27 ] , to date , it remains unclear precisely why this specific normalization is so effective . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we show that SN controls two important failure modes of GAN training : exploding gradients and vanishing gradients . These problems are well-known to cause instability in GANs [ 3 , 8 ] , leading either to bad local minima or stalled training prior to convergence . We make three primary contributions : ( 1 ) Analysis of why SN avoids exploding gradients ( § 3 ) . Poorly-chosen architectures and hyperparameters , as well as randomness during training , can amplify the effects of large gradients on training instability , ultimately leading to generalization error in the learned discriminator . We theoretically prove that SN upper bounds gradients during GAN training , mitigating these effects . ( 2 ) Analysis of why SN avoids vanishing gradients ( § 4 ) . Small gradients during training are known to cause GANs ( and other DNNs ) to converge to bad models [ 25 , 3 ] . The well-known LeCun initialization , first proposed over two decades ago , mitigates this effect by carefully choosing the variance of the initial weights [ 25 ] . We prove theoretically that SN controls the variance of weights in a way that closely parallels LeCun initialization . Whereas LeCun initialization only controls the gradient vanishing problem at the beginning of training , we show empirically that SN preserves this property throughout training . Our analysis also explains why a strict implementation of SN [ 12 ] has poor out-of-the-box performance on GANs and requires additional tuning to avoid the vanishing gradient problem , whereas the implementation of SN in [ 30 ] requires no tuning . ( 3 ) Improving SN with the above theoretical insights ( § 5 ) . Given this new understanding of the connections between SN and LeCun initialization , we propose Bidirectional Scaled Spectral Normalization ( BSSN ) , a simple modification of SN that combines two key insights ( Fig . 1 ) : ( a ) It introduces a novel bidirectional spectral normalization inspired by Xavier initialization , which improved on LeCun initialization by controlling not only the variances of internal outputs , but also the variance of backpropagated gradients [ 13 ] . We theoretically prove that BSSN mimics Xavier initialization to give better gradient control than SN . ( b ) BSSN introduces a new scaling of weights inspired by Kaiming initialization , a newer initialization technique that has better performance in practice [ 17 ] . We show that BSSN achieve better sample quality and training stability than SN on several benchmark datasets , including CIFAR10 , STL10 , CelebA , and ImageNet . Note that our goal in this work is not to propose the best normalization or regularization technique for training GANs ; there has been substantial work in this area already with promising results [ 34 , 48 , 20 , 11 ] . Our goal is instead to understand why SN has been so effective , and evaluate whether these insights can inform more effective alternatives . Our theoretical and empirical results suggest that gradient control may play a significant role in this story ( though they need not be the only factor explaining SN ’ s success ) . 2 Background and Preliminaries . The instability of GANs is believed to be predominantly caused by poor discriminator learning [ 2 , 40 ] . We therefore focus in this work on the discriminator , and the effects of SN on discriminator learning . We adopt the same model as [ 30 ] . Consider a discriminator with L internal layers : Dθ ( x ) = aL ◦ lwL ◦ aL−1 ◦ lwL−1 ◦ . . . ◦ a1 ◦ lw1 ( x ) ( 1 ) where x denotes the input to the discriminator and θ = { w1 , w2 , ... , wL } the weights ; ai ( i = 1 , ... , L− 1 ) is the activation function in the i-th layer , which is usually element-wise ReLU or leaky ReLU in GANs [ 14 ] . aL is the activation function for the last layer , which is sigmoid for the vanilla GAN [ 14 ] and identity for WGAN-GP [ 16 ] ; lwi is the linear transformation in i-th layer , which is usually fully-connected or a convolutional neural network [ 14 , 38 ] . Like prior work on the theoretical analysis of ( spectral ) normalization [ 30 , 12 , 42 , 20 ] , we do not model bias terms . Lipschitz regularization and spectral normalization . Prior work has shown that regularizing the Lipschitz constant of the discriminator ‖Dθ‖Lip improves the stability of GANs [ 3 , 16 , 50 ] . For example , WGAN-GP [ 16 ] adds a gradient penalty ( ‖∇Dθ ( x̃ ) ‖ − 1 ) 2 to the loss function , where x̃ = αx + ( 1 − α ) G ( z ) and α ∼ Uniform ( 0 , 1 ) to ensure that the Lipschitz constant of the discriminator is bounded by 1 . Spectral normalization ( SN ) takes a different approach . For fully connected layers ( i.e. , lwi ( x ) = wix ) , it regularizes the weightswi to ensure that spectral norm ‖wi‖sp = 1 for all i ∈ [ 1 , L ] , where the spectral norm ‖wi‖sp is defined as the largest singular value of wi . This bounds the Lipschitz constant of the discriminator since ‖Dθ‖Lip ≤ ∏L i=1 ‖lwi‖Lip · ∏L i=1 ‖ai‖Lip ≤ ∏L i=1 ‖wi‖sp · ∏L i=1 ‖ai‖Lip ≤ 1 , as ‖lwi‖Lip ≤ ‖wi‖sp and ‖ai‖Lip ≤ 1 for networks with ( leaky ) ReLU as activation functions for the internal layers and identity/sigmoid as the activation function for the last layer [ 30 ] . Prior work has theoretically connected the generalization gap of neural networks to the product of the spectral norms of the layers [ 5 , 32 , 20 , 48 ] . These insights led to multiple implementations of spectral normalization [ 12 , 15 , 51 , 30 , 20 , 48 ] , with the implementation of [ 30 ] achieving particular success on GANs . SN can be viewed as a special case of more general techniques for enhancing the stability of neural network training by controlling the entire spectrum of a network ’ s input-output Jacobian [ 34 ] , weight matrices [ 20 ] , or learned embeddings [ 48 ] . In practice , spectral normalization [ 12 , 30 ] is implemented by dividing the weight matrix wi by its spectral norm : wi uTi wivi , where ui and vi are the left/right singular vectors of wi corresponding to its largest singular value . As observed by Gouk et al . [ 15 ] , there are two approaches in the SN literature for instantiating the matrix wi for convolutional neural networks ( CNNs ) . In a CNN , since convolution is a linear operation , convolutional layers can equivalently be written as a multiplication by an expanded weight matrix w̃i that is derived from the raw weights wi . Hence in principle , spectral normalization should normalize each convolutional layer by ‖w̃i‖sp [ 15 , 12 ] . We call this canonical normalization SNConv as it controls the spectral norm of the convolution layer . However , the spectral normalization that is known to outperform other regularization techniques and improves training stability for GANs [ 30 ] , which we call SNw , does not implement SN in a strict sense . Instead , it uses ∥∥wicout× ( cinkwkh ) ∥∥sp ; that is , it first reshapes the convolution kernel wi ∈ Rcoutcinkwkh into a matrix ŵi of shape cout× ( cinkwkh ) , and then normalizes with the spectral norm ‖ŵi‖sp , where cin is the number of input channels , cout is the number of output channels , kw is the kernel width , and kh is the kernel height . Miyato et al . showed that their implementation implicitly penalizes wi from being too sensitive in one specific direction [ 30 ] . However , this does not explain why SNw is more stable than other Lipschitz regularization techniques , and as observed in [ 15 ] , it is unclear how SNw relates to SNConv . Despite this , SNw has empirically been immensely successful in stabilizing the training of GANs [ 8 , 28 , 53 , 21 , 52 , 31 , 27 ] . Even more puzzling , we show in § 4 that the canonical approach SNConv has comparatively poor out-of-the-box performance when training GANs . Hence , two questions arise : ( 1 ) Why is SN so successful at stabilizing the training of GANs ? ( 2 ) Why is SNw proposed by [ 30 ] so much more effective than the canonical SNConv ? In this work , we show that both questions are related to two well-known phenomena : vanishing and exploding gradients . These terms describe a problem in which gradients either grow or shrink rapidly during training [ 6 , 35 , 36 , 7 ] , and they are known to be closely related to the instability of GANs [ 2 , 8 ] . We provide an example to illustrate how vanishing or exploding gradients cause training instability in GANs in App . I . 3 Exploding Gradients . In this section , we show that spectral normalization prevents gradient explosion by bounding the gradients of the discriminator . Moreover , we show that the common choice to normalize all layers equally achieves the tightest upper bound for a restricted class of discriminators . We use θ ∈ Rd to denote a vector containing all elements in { w1 , ... , wL } . In the following analysis , we assume linear transformations are fully-connected layers lwi ( x ) = wix as in [ 30 ] , though the same analysis can be applied to convolutional layers . Following prior work on the theoretical analysis of ( spectral ) normalization [ 30 , 12 , 42 ] , we assume no bias in the network ( i.e. , Eq . ( 1 ) ) for simplicity . To highlight the effects of the spectral norm of each layer on the gradient and simplify the exposition , we will compute gradients with respect to w′i = wi uTi wivi in the following discussion . In reality , gradients are computed with respect to wi ; we defer this discussion to App . C , where we show the relevant extension . How SN controls exploding gradients . The following proposition shows that under this simplifying assumption , spectral normalization controls the magnitudes of the gradients of the discriminator with respect to θ . Notice that simply controlling the Lipschitz constant of the discriminator ( e.g. , as in WGAN [ 2 ] ) does not imply this property ; it instead ensures small ( sub ) gradients with respect to the input , x . Proposition 1 ( Upper bound of gradient ’ s Frobenius norm for spectral normalization ) . If ‖wi‖sp ≤ 1 for all i ∈ [ 1 , L ] , then we have ‖∇wtDθ ( x ) ‖F ≤ ‖x‖ ∏L i=1 ‖ai‖Lip , and the norm of the overall gradient can be bounded by ‖∇θDθ ( x ) ‖F ≤ √ L ‖x‖ ∏L i=1 ‖ai‖Lip . ( Proof in App . A ) . Note that under the assumption that internal activation functions are ReLU or leaky ReLU , if the activation function for the last layer is identity ( e.g. , for WGAN-GP [ 16 ] ) , the above bounds can be simplified to ‖∇wtDθ ( x ) ‖F ≤ ‖x‖ and ‖∇θDθ ( x ) ‖ ≤ √ L ‖x‖ , and if the activation for the last layer is sigmoid ( e.g. , for vanilla GAN [ 14 ] ) , the above bounds become ‖∇wtDθ ( x ) ‖F ≤ 0.25 ‖x‖ and ‖∇θDθ ( x ) ‖ ≤ 0.25 √ L ‖x‖ . A comparable bound can also be found to limit the norm of the Hessian , which we defer to App . D. The bound in Prop . 1 has a significant effect in practice . Fig . 2 shows the norm of the gradient for each layer of a GAN trained on MNIST with and without spectral normalization . Without spectral normalization , some layers have extremely large gradients throughout training , which makes the overall gradient large . With spectral normalization , the gradients of all layers are upper bounded as shown in Prop . 1 . We see similar results in other datasets and network architectures ( App . J ) . Optimal spectral norm allocation . Common implementations of SN advocate setting the spectral norm of each layer to the same value [ 30 , 12 ] . However , the following proposition states that we can set the spectral norms of different layers to different constants , without changing the network ’ s behavior on the input samples , as long as the product of the spectral norm bounds is the same . Proposition 2 . For any discriminator Dθ = aL ◦ lwL ◦ aL−1 ◦ lwL−1 ◦ . . . ◦ a1 ◦ lw1 and D′θ = aL ◦ lcL·wL ◦ aL−1 ◦ lcL−1·wL−1 ◦ . . . ◦ a1 ◦ lc1·w1 where the internal activation functions { ai } L−1 i=1 are ReLU or leaky ReLU , and positive constant scalars c1 , ... , cL satisfy that ∏L i=1 ci = 1 , we have Dθ ( x ) = D ′ θ ( x ) ∀x and ∂nDθ ( x ) ∂xn = ∂nD′θ ( x ) ∂xn ∀x , ∀n ∈ Z+ . ( Proof in App . B ) . Given this observation , it is natural to ask if there is any benefit to setting the spectral norms of each layer equal . It turns out that the answer is yes , under some assumptions that appear to approximately hold in practice . Let D , { Dθ = aL ◦ lwL ◦ . . . ◦ a1 ◦ lw1 : ‖∇wiDθ ( x ) ‖F∥∥∇wjDθ ( x ) ∥∥F = ‖wj‖sp ‖wi‖sp , ai ∈ { ReLU , leaky ReLU } ∀i , j ∈ [ 1 , L ] } . ( 2 ) This intuitively describes the set of all discriminators for which scaling up the weight of one layer proportionally increases the gradient norm of all other layers ; the definition of this set is motivated by our upper bound on the gradient norm ( App . A ) . The following theorem shows that when optimizing over set D , choosing every layer to have the same spectral norm gives the smallest possible gradient norm , for a given set of parameters . Theorem 1 . Consider a given set of discriminator parameters θ = { w1 , ... , wL } . For a vector c = { c1 , . . . , cL } , we denote θc , { ctwt } Lt=1 . Let λθ = ∏L i=1 ‖wi‖ 1/L sp denote the geometric mean of the spectral norms of the weights . Then we have { λθ ‖w1‖sp , . . . , λθ ‖wL‖sp } = arg min c : Dθc∈D , ∏L i=1 ci=1 , ci∈R+ ‖∇θcDθc ( x ) ‖F ( Proof in App . E ) . The key constraint in this theorem is that we optimize only over discriminators in set D in Eq . ( 2 ) . To show that this constraint is realistic ( i.e. , SN GAN discriminator optimization tends to choose models in D ) , we trained a spectrally-normalized GAN with four hidden layers on MNIST , computing the ratios of the gradient norms at each layer and the ratios of the spectral norms , as dictated by Eq . ( 2 ) . We computed these ratios at different epochs during training , as well as for different randomlyselected rescalings of the spectral normalization vector c. Each point in Fig . 3 represents the results averaged over 64 real samples at a specific epoch of training for a given ( random ) c. Vertical series of points are from different epochs of the same run , therefore their ratio of spectral norms is the same . The fact that most of the points are near the diagonal line suggests that training naturally favors discriminators that are in or near D ; we confirm this intuition in other experimental settings in App . K. This observation , combined with Thm . 1 , suggests that it is better to force the spectral norms of every layer to be equal . Hence , existing SN implementations [ 30 , 12 ] chose the correct , uniform normalization across layers to upper bound discriminator ’ s gradients . Implications on other normalization/regularization techniques . Note that the analysis in this section can also be used to show the same results for other normalization/regularization techniques that control the spectral norm of weights , like weight normalization [ 41 ] and orthogonal regularization [ 9 ] . However , these techniques do not necessarily exhibit the more important properties proved in § 4 for SN , and have some known drawbacks ( see more discussion in § 6 ) .

Summary: The paper offers a more insightful analysis on Spectral Normalization (SN) and shows that SN controls two important failure modes of GAN: exploding and vanishing gradients. Specifically, the paper provides the theories to prove that SN upper-bounds the gradients and controls the variance of weights closely to LeCun initialization. Furthermore, given the new understanding of the connection between SN and LeCun initialization, the paper proposes a simple modification of SN to control both the variances of internal outputs and the variance of backpropagation gradients, which mimics the way of Xavier initialization. It also introduces new scaling weights, motivated by Kaming initialization, to obtain better performance in practice. The paper provides ablation studies and experiments on multiple benchmark datasets.