Why Spectral Normalization Stabilizes GANs: Analysis and Improvements

1 Introduction . Generative adversarial networks ( GANs ) are state-of-the-art deep generative models , perhaps best known for their ability to produce high-resolution , photorealistic images [ 14 ] . The objective of GANs is to produce random samples from a target data distribution , given only access to an initial set of training samples . This is achieved by learning two functions : a generator G , which maps random input noise to a generated sample , and a discriminator D , which tries to classify input samples as either real ( i.e. , from the training dataset ) or fake ( i.e. , produced by the generator ) . In practice , these functions are implemented by deep neural networks ( DNNs ) , and the competing generator and discriminator are trained in an alternating process known as adversarial training . Theoretically , given enough data and model capacity , GANs converge to the true underlying data distribution [ 14 ] . Although GANs have been very successful in improving the sample quality of data-driven generative models [ 22 , 8 ] , their adversarial training also contributes to instability . That is , small hyper-parameter changes and even randomness in the optimization can cause training to fail . Many approaches have been proposed for improving the stability of GANs , including different architectures [ 38 , 22 , 8 ] , loss functions [ 2 , 3 , 16 , 50 ] , and various types of regularizations/normalizations [ 30 , 9 , 41 ] . One of the most successful proposals to date is called spectral normalization ( SN ) [ 30 , 15 , 12 ] . SN forces each layer of the discriminator to have unit spectral norm during training . This has the effect of controlling the Lipschitz constant of the discriminator , which is empirically observed to improve the stability of GAN training [ 30 ] . Despite the successful applications of SN [ 8 , 28 , 53 , 21 , 52 , 31 , 27 ] , to date , it remains unclear precisely why this specific normalization is so effective . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we show that SN controls two important failure modes of GAN training : exploding gradients and vanishing gradients . These problems are well-known to cause instability in GANs [ 3 , 8 ] , leading either to bad local minima or stalled training prior to convergence . We make three primary contributions : ( 1 ) Analysis of why SN avoids exploding gradients ( § 3 ) . Poorly-chosen architectures and hyperparameters , as well as randomness during training , can amplify the effects of large gradients on training instability , ultimately leading to generalization error in the learned discriminator . We theoretically prove that SN upper bounds gradients during GAN training , mitigating these effects . ( 2 ) Analysis of why SN avoids vanishing gradients ( § 4 ) . Small gradients during training are known to cause GANs ( and other DNNs ) to converge to bad models [ 25 , 3 ] . The well-known LeCun initialization , first proposed over two decades ago , mitigates this effect by carefully choosing the variance of the initial weights [ 25 ] . We prove theoretically that SN controls the variance of weights in a way that closely parallels LeCun initialization . Whereas LeCun initialization only controls the gradient vanishing problem at the beginning of training , we show empirically that SN preserves this property throughout training . Our analysis also explains why a strict implementation of SN [ 12 ] has poor out-of-the-box performance on GANs and requires additional tuning to avoid the vanishing gradient problem , whereas the implementation of SN in [ 30 ] requires no tuning . ( 3 ) Improving SN with the above theoretical insights ( § 5 ) . Given this new understanding of the connections between SN and LeCun initialization , we propose Bidirectional Scaled Spectral Normalization ( BSSN ) , a simple modification of SN that combines two key insights ( Fig . 1 ) : ( a ) It introduces a novel bidirectional spectral normalization inspired by Xavier initialization , which improved on LeCun initialization by controlling not only the variances of internal outputs , but also the variance of backpropagated gradients [ 13 ] . We theoretically prove that BSSN mimics Xavier initialization to give better gradient control than SN . ( b ) BSSN introduces a new scaling of weights inspired by Kaiming initialization , a newer initialization technique that has better performance in practice [ 17 ] . We show that BSSN achieve better sample quality and training stability than SN on several benchmark datasets , including CIFAR10 , STL10 , CelebA , and ImageNet . Note that our goal in this work is not to propose the best normalization or regularization technique for training GANs ; there has been substantial work in this area already with promising results [ 34 , 48 , 20 , 11 ] . Our goal is instead to understand why SN has been so effective , and evaluate whether these insights can inform more effective alternatives . Our theoretical and empirical results suggest that gradient control may play a significant role in this story ( though they need not be the only factor explaining SN ’ s success ) . 2 Background and Preliminaries . The instability of GANs is believed to be predominantly caused by poor discriminator learning [ 2 , 40 ] . We therefore focus in this work on the discriminator , and the effects of SN on discriminator learning . We adopt the same model as [ 30 ] . Consider a discriminator with L internal layers : Dθ ( x ) = aL ◦ lwL ◦ aL−1 ◦ lwL−1 ◦ . . . ◦ a1 ◦ lw1 ( x ) ( 1 ) where x denotes the input to the discriminator and θ = { w1 , w2 , ... , wL } the weights ; ai ( i = 1 , ... , L− 1 ) is the activation function in the i-th layer , which is usually element-wise ReLU or leaky ReLU in GANs [ 14 ] . aL is the activation function for the last layer , which is sigmoid for the vanilla GAN [ 14 ] and identity for WGAN-GP [ 16 ] ; lwi is the linear transformation in i-th layer , which is usually fully-connected or a convolutional neural network [ 14 , 38 ] . Like prior work on the theoretical analysis of ( spectral ) normalization [ 30 , 12 , 42 , 20 ] , we do not model bias terms . Lipschitz regularization and spectral normalization . Prior work has shown that regularizing the Lipschitz constant of the discriminator ‖Dθ‖Lip improves the stability of GANs [ 3 , 16 , 50 ] . For example , WGAN-GP [ 16 ] adds a gradient penalty ( ‖∇Dθ ( x̃ ) ‖ − 1 ) 2 to the loss function , where x̃ = αx + ( 1 − α ) G ( z ) and α ∼ Uniform ( 0 , 1 ) to ensure that the Lipschitz constant of the discriminator is bounded by 1 . Spectral normalization ( SN ) takes a different approach . For fully connected layers ( i.e. , lwi ( x ) = wix ) , it regularizes the weightswi to ensure that spectral norm ‖wi‖sp = 1 for all i ∈ [ 1 , L ] , where the spectral norm ‖wi‖sp is defined as the largest singular value of wi . This bounds the Lipschitz constant of the discriminator since ‖Dθ‖Lip ≤ ∏L i=1 ‖lwi‖Lip · ∏L i=1 ‖ai‖Lip ≤ ∏L i=1 ‖wi‖sp · ∏L i=1 ‖ai‖Lip ≤ 1 , as ‖lwi‖Lip ≤ ‖wi‖sp and ‖ai‖Lip ≤ 1 for networks with ( leaky ) ReLU as activation functions for the internal layers and identity/sigmoid as the activation function for the last layer [ 30 ] . Prior work has theoretically connected the generalization gap of neural networks to the product of the spectral norms of the layers [ 5 , 32 , 20 , 48 ] . These insights led to multiple implementations of spectral normalization [ 12 , 15 , 51 , 30 , 20 , 48 ] , with the implementation of [ 30 ] achieving particular success on GANs . SN can be viewed as a special case of more general techniques for enhancing the stability of neural network training by controlling the entire spectrum of a network ’ s input-output Jacobian [ 34 ] , weight matrices [ 20 ] , or learned embeddings [ 48 ] . In practice , spectral normalization [ 12 , 30 ] is implemented by dividing the weight matrix wi by its spectral norm : wi uTi wivi , where ui and vi are the left/right singular vectors of wi corresponding to its largest singular value . As observed by Gouk et al . [ 15 ] , there are two approaches in the SN literature for instantiating the matrix wi for convolutional neural networks ( CNNs ) . In a CNN , since convolution is a linear operation , convolutional layers can equivalently be written as a multiplication by an expanded weight matrix w̃i that is derived from the raw weights wi . Hence in principle , spectral normalization should normalize each convolutional layer by ‖w̃i‖sp [ 15 , 12 ] . We call this canonical normalization SNConv as it controls the spectral norm of the convolution layer . However , the spectral normalization that is known to outperform other regularization techniques and improves training stability for GANs [ 30 ] , which we call SNw , does not implement SN in a strict sense . Instead , it uses ∥∥wicout× ( cinkwkh ) ∥∥sp ; that is , it first reshapes the convolution kernel wi ∈ Rcoutcinkwkh into a matrix ŵi of shape cout× ( cinkwkh ) , and then normalizes with the spectral norm ‖ŵi‖sp , where cin is the number of input channels , cout is the number of output channels , kw is the kernel width , and kh is the kernel height . Miyato et al . showed that their implementation implicitly penalizes wi from being too sensitive in one specific direction [ 30 ] . However , this does not explain why SNw is more stable than other Lipschitz regularization techniques , and as observed in [ 15 ] , it is unclear how SNw relates to SNConv . Despite this , SNw has empirically been immensely successful in stabilizing the training of GANs [ 8 , 28 , 53 , 21 , 52 , 31 , 27 ] . Even more puzzling , we show in § 4 that the canonical approach SNConv has comparatively poor out-of-the-box performance when training GANs . Hence , two questions arise : ( 1 ) Why is SN so successful at stabilizing the training of GANs ? ( 2 ) Why is SNw proposed by [ 30 ] so much more effective than the canonical SNConv ? In this work , we show that both questions are related to two well-known phenomena : vanishing and exploding gradients . These terms describe a problem in which gradients either grow or shrink rapidly during training [ 6 , 35 , 36 , 7 ] , and they are known to be closely related to the instability of GANs [ 2 , 8 ] . We provide an example to illustrate how vanishing or exploding gradients cause training instability in GANs in App . I . 3 Exploding Gradients . In this section , we show that spectral normalization prevents gradient explosion by bounding the gradients of the discriminator . Moreover , we show that the common choice to normalize all layers equally achieves the tightest upper bound for a restricted class of discriminators . We use θ ∈ Rd to denote a vector containing all elements in { w1 , ... , wL } . In the following analysis , we assume linear transformations are fully-connected layers lwi ( x ) = wix as in [ 30 ] , though the same analysis can be applied to convolutional layers . Following prior work on the theoretical analysis of ( spectral ) normalization [ 30 , 12 , 42 ] , we assume no bias in the network ( i.e. , Eq . ( 1 ) ) for simplicity . To highlight the effects of the spectral norm of each layer on the gradient and simplify the exposition , we will compute gradients with respect to w′i = wi uTi wivi in the following discussion . In reality , gradients are computed with respect to wi ; we defer this discussion to App . C , where we show the relevant extension . How SN controls exploding gradients . The following proposition shows that under this simplifying assumption , spectral normalization controls the magnitudes of the gradients of the discriminator with respect to θ . Notice that simply controlling the Lipschitz constant of the discriminator ( e.g. , as in WGAN [ 2 ] ) does not imply this property ; it instead ensures small ( sub ) gradients with respect to the input , x . Proposition 1 ( Upper bound of gradient ’ s Frobenius norm for spectral normalization ) . If ‖wi‖sp ≤ 1 for all i ∈ [ 1 , L ] , then we have ‖∇wtDθ ( x ) ‖F ≤ ‖x‖ ∏L i=1 ‖ai‖Lip , and the norm of the overall gradient can be bounded by ‖∇θDθ ( x ) ‖F ≤ √ L ‖x‖ ∏L i=1 ‖ai‖Lip . ( Proof in App . A ) . Note that under the assumption that internal activation functions are ReLU or leaky ReLU , if the activation function for the last layer is identity ( e.g. , for WGAN-GP [ 16 ] ) , the above bounds can be simplified to ‖∇wtDθ ( x ) ‖F ≤ ‖x‖ and ‖∇θDθ ( x ) ‖ ≤ √ L ‖x‖ , and if the activation for the last layer is sigmoid ( e.g. , for vanilla GAN [ 14 ] ) , the above bounds become ‖∇wtDθ ( x ) ‖F ≤ 0.25 ‖x‖ and ‖∇θDθ ( x ) ‖ ≤ 0.25 √ L ‖x‖ . A comparable bound can also be found to limit the norm of the Hessian , which we defer to App . D. The bound in Prop . 1 has a significant effect in practice . Fig . 2 shows the norm of the gradient for each layer of a GAN trained on MNIST with and without spectral normalization . Without spectral normalization , some layers have extremely large gradients throughout training , which makes the overall gradient large . With spectral normalization , the gradients of all layers are upper bounded as shown in Prop . 1 . We see similar results in other datasets and network architectures ( App . J ) . Optimal spectral norm allocation . Common implementations of SN advocate setting the spectral norm of each layer to the same value [ 30 , 12 ] . However , the following proposition states that we can set the spectral norms of different layers to different constants , without changing the network ’ s behavior on the input samples , as long as the product of the spectral norm bounds is the same . Proposition 2 . For any discriminator Dθ = aL ◦ lwL ◦ aL−1 ◦ lwL−1 ◦ . . . ◦ a1 ◦ lw1 and D′θ = aL ◦ lcL·wL ◦ aL−1 ◦ lcL−1·wL−1 ◦ . . . ◦ a1 ◦ lc1·w1 where the internal activation functions { ai } L−1 i=1 are ReLU or leaky ReLU , and positive constant scalars c1 , ... , cL satisfy that ∏L i=1 ci = 1 , we have Dθ ( x ) = D ′ θ ( x ) ∀x and ∂nDθ ( x ) ∂xn = ∂nD′θ ( x ) ∂xn ∀x , ∀n ∈ Z+ . ( Proof in App . B ) . Given this observation , it is natural to ask if there is any benefit to setting the spectral norms of each layer equal . It turns out that the answer is yes , under some assumptions that appear to approximately hold in practice . Let D , { Dθ = aL ◦ lwL ◦ . . . ◦ a1 ◦ lw1 : ‖∇wiDθ ( x ) ‖F∥∥∇wjDθ ( x ) ∥∥F = ‖wj‖sp ‖wi‖sp , ai ∈ { ReLU , leaky ReLU } ∀i , j ∈ [ 1 , L ] } . ( 2 ) This intuitively describes the set of all discriminators for which scaling up the weight of one layer proportionally increases the gradient norm of all other layers ; the definition of this set is motivated by our upper bound on the gradient norm ( App . A ) . The following theorem shows that when optimizing over set D , choosing every layer to have the same spectral norm gives the smallest possible gradient norm , for a given set of parameters . Theorem 1 . Consider a given set of discriminator parameters θ = { w1 , ... , wL } . For a vector c = { c1 , . . . , cL } , we denote θc , { ctwt } Lt=1 . Let λθ = ∏L i=1 ‖wi‖ 1/L sp denote the geometric mean of the spectral norms of the weights . Then we have { λθ ‖w1‖sp , . . . , λθ ‖wL‖sp } = arg min c : Dθc∈D , ∏L i=1 ci=1 , ci∈R+ ‖∇θcDθc ( x ) ‖F ( Proof in App . E ) . The key constraint in this theorem is that we optimize only over discriminators in set D in Eq . ( 2 ) . To show that this constraint is realistic ( i.e. , SN GAN discriminator optimization tends to choose models in D ) , we trained a spectrally-normalized GAN with four hidden layers on MNIST , computing the ratios of the gradient norms at each layer and the ratios of the spectral norms , as dictated by Eq . ( 2 ) . We computed these ratios at different epochs during training , as well as for different randomlyselected rescalings of the spectral normalization vector c. Each point in Fig . 3 represents the results averaged over 64 real samples at a specific epoch of training for a given ( random ) c. Vertical series of points are from different epochs of the same run , therefore their ratio of spectral norms is the same . The fact that most of the points are near the diagonal line suggests that training naturally favors discriminators that are in or near D ; we confirm this intuition in other experimental settings in App . K. This observation , combined with Thm . 1 , suggests that it is better to force the spectral norms of every layer to be equal . Hence , existing SN implementations [ 30 , 12 ] chose the correct , uniform normalization across layers to upper bound discriminator ’ s gradients . Implications on other normalization/regularization techniques . Note that the analysis in this section can also be used to show the same results for other normalization/regularization techniques that control the spectral norm of weights , like weight normalization [ 41 ] and orthogonal regularization [ 9 ] . However , these techniques do not necessarily exhibit the more important properties proved in § 4 for SN , and have some known drawbacks ( see more discussion in § 6 ) .

Summary: The paper analyzes the benefits of the spectral norm on GAN: preventing gradient exploding and gradient vanishing. They prove that SN can control the variance of weights, which can prevent gradient vanishing, following the logic of LeCun initialization. They further propose a bidirectional SN, which is an improvement based on SN, following the logic of Xariver initialization. They conduct experiments on multiple datasets with DCGAN structure.