Precondition Layer and Its Use for GANs

1 INTRODUCTION . Generative Adversarial Nets ( GANs ) ( Goodfellow et al. , 2014 ) successfully transform samples from one distribution to another . Nevertheless , training GANs is known to be challenging , and its performance is often sensitive to hyper-parameters and datasets . Understanding the training difficulties of GAN is thus an important problem . Recent studies in neural network theory ( Pennington et al. , 2017 ; Xiao et al. , 2018 ; 2020 ) suggest that the spectrum of the input-output Jacobian or neural tangent kernel ( NTK ) is an important metric for understanding training performance . While directly manipulating the spectrum of the Jacobian or NTK is not easy , a practical approach is to manipulate the spectrum of weight matrices , such as orthogonal initialization ( Xiao et al. , 2018 ) . For a special neural net , Hu et al . ( 2020 ) showed that orthogonal initialization leads to better convergence result than Gaussian initialization , which provides early theoretical evidence for the importance of manipulating the weight matrix spectrum . Motivated by these studies , we suspect that an ‘ adequate ’ weight matrix spectrum is also important for GAN training . Indeed , one of the most popular techniques for GAN training , spectral normalization ( SN ) ( Miyato et al. , 2018 ) , manipulates the spectrum by scaling all singular values by a constant . This ensures the spectral norm is upper bounded . However , we find that for some hyperparameters and for high-resolution datasets , SN-GAN fails to generate good images . In a study we find the condition numbers of weight matrices to become very large and the majority of the singular values are close to 0 during training . See Fig . 1 ( a ) and Fig . 2 ( a ) . This can happen as SN does not promote a small condition number . This finding motivates to reduce the condition number of weights during GAN training . Recall that controlling the condition number is also a central problem in numerical linear algebra , known as preconditioning ( see Chen ( 2005 ) ) . We hence seek to develop a “ plug-in ” preconditioner for weights . This requires the preconditioner to be differentiable . Out of various preconditioners , we find the polynomial preconditioner to be a suitable choice due to the simple differentiation and strong theoretical support from approximation theory . Further , we suggest to adaptively adjust the strength of the preconditioner during training so as to not overly restrict the expressivity . We show the efficacy of preconditioning on CIFAR10 ( 32 ⇥ 32 ) , STL ( 48 ⇥ 48 ) and LSUN bedroom , tower and living room ( 256 ⇥ 256 ) . Summary of contributions . For a deep linear network studied in ( Hu et al. , 2020 ) , we prove that if all weight matrices have bounded spectrum , then gradient descent converges to global min- imum at a geometric rate . We then introduce a PC-layer ( preconditioning layer ) that consists of a low-degree polynomial preconditioner . We further study adaptive preconditioning ( APC ) which adaptively controls the strength of PC on different layers in different iterations . Applying PC and APC to unconditional GAN training on LSUN data ( 256 ⇥ 256 ) , permits to generate high-quality images when SN-GAN fails . We also show that APC achieves better FID scores on CIFAR10 , STL , and LSUN than a recently proposed method of Jiang et al . ( 2019 ) . 1.1 RELATED WORK . Related to the proposed method is work by Jiang et al . ( 2019 ) , which also controls the spectrum in GAN training . They re-parameterize a weight matrix W via W = USV T , add orthogonal regularization of U , V and certain regularizer of entries of the diagonal matrix S. This approach differs from ours in a few aspects . First , Jiang et al . ( 2019 ) essentially solves a constrained optimization problem with constraints UTU = I , V TV = I using a penalty method ( Bertsekas , 1997 ) . In contrast , our approach solves an unconstrained problem since we add one layer into the neural net , similar to batch normalization ( BN ) ( Ioffe & Szegedy , 2015 ) and SN ( Miyato et al. , 2018 ) . Second , our PC layer is a direct generalization of SN as it includes SN-layer as a special case . In contrast , the method of Jiang et al . ( 2019 ) differs from SN-layer in any case . Our proposed method thus offers a smoother transition for existing users of SN . In a broader context , a number of approaches have been proposed to stabilize and improve GAN training , such as modifying the loss function ( Arjovsky et al. , 2017 ; Arjovsky & Bottou , 2017 ; Mao et al. , 2017 ; Li et al. , 2017b ; Deshpande et al. , 2018 ) , normalization and regularization ( Gulrajani et al. , 2017 ; Miyato et al. , 2018 ) , progressive growing techniques ( Karras et al. , 2018 ; Huang et al. , 2017 ) , changing architecture ( Zhang et al. , 2019 ; Karnewar & Wang , 2019 ) , utilizing side information like class labels ( Mirza & Osindero , 2014 ; Odena et al. , 2017 ; Miyato & Koyama , 2018 ) . Using this taxonomy , our approach fits the “ normalization and regularization ” category ( even though our method is not exactly normalization , the essence of “ embedded control ” is similar ) . Note that these directions are relatively orthogonal , and our approach can be potentially combined with other techniques such as progressive growing . However , due to limited computational resources , we focus on unconditional GANs using classical architectures , the setting studied by Miyato et al . ( 2018 ) . 1.2 NOTATION AND DEFINITION . We use eig ( A ) to denote the multiset ( i.e. , allow repetition ) of all eigenvalues of A . If all eigenvalues of A are non-negative real numbers , we say A is a positive semidefinite ( PSD ) matrix . The singular values of a matrix A 2 Rn⇥m are the square root of the eigenvalues of ATA 2 Rm⇥m . Let max ( A ) and min ( A ) denote the maximum and minimum singular values of A . Let kAk2 denote the spectral norm of A , i.e. , the largest singular value . The condition number of a square matrix A is traditionally defined as ( A ) = kAk2kA 1k2 = max ( A ) min ( A ) . We extend this definition to a rectangular matrix A 2 Rn⇥m where n m via ( A ) = max ( A ) min ( A ) . Let deg ( p ) denote the degree of a polynomial p and let Pk = { p | deg ( p ) k } be the set of polynomials with degree no more than k . 2 WHY CONTROLLING THE SPECTRUM ? . To understand why controlling the spectrum is helpful we leverage recent tools in neural network theory to prove the following result : if weight matrices have small condition numbers , then gradient descent for deep pyramid linear networks converges to the global-min fast . This is inspired by Hu et al . ( 2020 ) analyzing a deep linear network to justify orthogonal initialization . Similar to Hu et al . ( 2020 ) , we consider a linear network that takes an input x 2 Rdx⇥1 and outputs F ( ✓ ; x ) = WLWL 1 . . .W1x 2 Rdy⇥1 , ( 1 ) where ✓ = ( W1 , . . . , WL ) represents the collection of all parameters and Wj is a matrix of dimension dj ⇥ dj 1 , j = 1 , . . . , L. Here we define d0 = dx and dL = dy . Assume there exists r 2 { 1 , . . . , L } , such that dy = dL dL 1 · · · dr , and n d0 d1 · · · dr . This means the network is a pyramid network , which generalizes the equal-width network of Hu et al . ( 2020 ) . Suppose y = ( y1 ; . . . ; yn ) 2 Rndy⇥1 are the labels , and the predictions are F ( ✓ ; X ) = ( F ( ✓ ; x1 ) ; . . . , F ( ✓ ; xn ) ) 2 Rndy⇥1 . We consider a quadratic loss L ( ✓ ) = 12ky F ( ✓ ; X ) k 2 . Starting from ✓ ( 0 ) , we generate ✓ ( k ) = ( W1 ( k ) , . . . , WL ( k ) ) , k = 1 , 2 , . . . via gradient descent : ✓ ( k + 1 ) = ✓ ( k ) ⌘rL ( ✓ ( k ) ) . Denote the residual e ( k ) = F ( ✓ ( k ) ; X ) y . For given ⌧l 1 , µl 0 , l = 1 , . . . , L , define R , { ✓ = ( W1 , . . . , WL ) | ⌧l max ( Wl ) min ( Wl ) µl , 8l } . , LkXk2⌧L . . . ⌧1 ( ke ( 0 ) k+ kXkF ⌧L . . . ⌧1 ) , µ , ( µ1 . . . µL ) 2 min ( X ) 2 . The following result states that if ✓ ( k ) stay within region R ( i.e. , weight matrices have bounded spectrum ) for k = 0 , 1 , . . . , K , then the loss decreases at a geometric rate until iteration K. The rate ( 1 µ ) depends on ( ⌧L ... ⌧1 ) 2 ( µL ... µ1 ) 2 , which is related to the condition numbers of all weights . Theorem 1 Suppose ⌘ = 1 . Assume ✓ ( k ) 2 R , k = 0 , 1 , . . . , K. Then we have ke ( k + 1 ) k2 ( 1 µ ) ke ( k ) k2 , k = 0 , 1 , . . . , K. ( 2 ) See Appendix D.3.1 for the proof and detailed discussions . For proper initial point ✓ ( 0 ) where Wl ( 0 ) ’ s are full-rank , we can always pick ⌧l , l so that ✓ ( 0 ) 2 R. The trajectory { ✓ ( k ) } either stays in R forever ( in which case K = 1 ) , or leaves R at some finite iteration K. In the former case , the loss converges to zero at a geometrical rate ; in the latter case , the loss decreases to below ( 1 µ/ ) Kke ( 0 ) k2 . However , our theorem does not specify how large K is for a given situation . Previous works on convergence ( e.g. , Hu et al. , 2020 ; Du et al. , 2018 ; Allen-Zhu et al. , 2019 ; Zou et al. , 2018 ) bound the movement of the weights with extra assumptions , so that the trajectory stays in a certain nice regime ( related to R ) . We do not attempt to prove the trajectory stays in R. Instead , we use this as a motivation for algorithm design : if we can improve the condition numbers of weights during training , then the trajectory may stay in R for a longer time , and thus lead to smaller loss values . Next , we present the preconditioning layer as such a method .

Summary: This paper mainly solves the instability issue on the spectral normalization for generative adversarial networks (SN-GANs) when training with high dimensional data. To address this, the authors present a preconditioning layer (PC-layer) with two different ways (i.e., FPC and APC) to perform a low-degree polynomial preconditioning. Experiments on LSUN 256x256 training data demonstrate that FPC and APC are able to control the strength of preconditioning. My detailed comments are as follows.