Generalization of GANs and overparameterized models under Lipschitz continuity

1 INTRODUCTION . In Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) , we want to train a discriminator D and a generator G by solving the following problem : min G max D Ex∼Pd log ( D ( x ) ) + Ez∼Pz log ( 1−D ( G ( z ) ) ) ( 1 ) where Pd is a data distribution that generates real data , and Pz is some noise distribution . G is a mapping that maps a noise z to a point in the data space . After training , G can be used to generate novel but realistic data . Since its introduction ( Goodfellow et al. , 2014 ) , a significant progress has been made for developing GANs and for interesting applications ( Hong et al. , 2019 ) . Some recent works ( Brock et al. , 2019 ; Zhang et al. , 2019 ; Karras et al. , 2020b ) can train a generator that produces synthetic images of extremely high quality . To explain those success , one popular way is to analyze generalization of the trained players . There are many existing theories ( Mohri et al. , 2018 ) for analyzing generalization . However , they suffer from various difficulties since the training problem of GANs is unsupervised in nature and contains two players competing against each other . Such a nature is entirely different from traditional learning problems . Neural distance ( Arora et al. , 2017 ) was introduced for analyzing generalization of GANs . One major limitation of existing distance-based bounds ( Arora et al. , 2017 ; Zhang et al. , 2018 ; Jiang et al. , 2019 ; Husain et al. , 2019 ) is the strong dependence on the capacity of the family , which defines the distance between two distributions , sometimes leading to trivial bounds . This limitation prevents us from fullly understanding and identifying the key factors that contribute to the generalization of GANs . For example , it has long been theoretically unclear why Lipschitz constraint empirically can lead to great success in GANs ? The standard learning theories suffer from various difficulties when analyzing overparameterized neural networks ( NNs ) . For example , Radermacher-based bounds ( Bartlett & Mendelson , 2002 ; Golowich et al. , 2020 ) can be trivial ( Zhang et al. , 2021 ) ; algorithmic stability ( Shalev-Shwartz et al. , 2010 ) and robustness ( Xu & Mannor , 2012 ) may not be directly used since instability is the well-known issue when training GANs ( Salimans et al. , 2016 ; Arjovsky & Bottou , 2017 ; Xu et al. , 2020 ) . Those examples are among the reasons for why the theoretical study of modern deep learning is still in its infancy ( Fang et al. , 2021 ) . Although some studies show generalization for shallow networks with at most one hidden layer ( Arora et al. , 2021 ; Mianjy & Arora , 2020 ; Mou et al. , 2018 ; Kuzborskij & Szepesvári , 2021 ; Ji et al. , 2021 ; Hu et al. , 2021 ; Jacot et al. , 2018 ) , it has long been theoretically unknown why deeper NNs can generalize better ? Many great successes of deep learning often need huge datasets , but it has long been theoretically unknown whether or not the generalization of deep NNs suffers from the curse of dimensionality ? This work has the following contributions : B We introduce a Lipschitz theory to analyze generalization of a learned function . This theory is surprizingly simple to analyze various complex models in general settings ( including supervised , unsupervised , and adversarial settings ) . BWe show that Dropout or spectrally-normalized neural networks avoid the curse of dimensionality . The number of layers required to ensure good generalization is logarithmic in sample size . It also suggests that deeper NNs can generalize better . We further show consistency and indentify a sufficient condition to guarantee high performance of DNNs . These results provide a significant step toward answering the open theoretical issues of deep learning ( Zhang et al. , 2021 ; Fang et al. , 2021 ) . B Using Lipschitz theory , we provide a comprehensive analysis on generalization of GANs which resolves the two open challenges in the GAN community : ( i ) Our bounds apply to any particular D or G , and hence overcome the major limitation of existing works ; In particular , for the first time in the literature , we show that Dropout and spectral normalization can help GANs to avoid the curse of dimensionality . ( ii ) Lipschitz constraint is used popularly through various ways including gradient penalty ( Gulrajani et al. , 2017 ) , spectral normalization ( Miyato et al. , 2018 ) , dropout , and data augmentation . Our analysis provides an unified explanation for why imposing a Lipschitz constraint can help GANs to generalize well in practice . Organization : We will review related work in the next section . Section 3 presents the theory connecting Lipschitz continuity with generalization , and some analyses about deep neural networks . In Section 4 , we analyze the generalization of GANs in various aspects . Section 5 concludes the paper . 2 RELATED WORK . Generalization in GANs : There are few efforts to analyze the generalization for GANs using the notion of neural distance , dD ( Pd , Pg ) , which is the distance between two distributions ( Pd , Pg ) , where D is the discriminator family.1 Arora et al . ( 2017 ) analyze generalization by bounding the quantity |dD ( Pd , Pg ) − dD ( P̂d , P̂g ) | , where ( P̂d , P̂g ) are empirical versions of ( Pd , Pg ) . For a suitable choice of the loss V ( Pd , Pz , D , G ) in GANs , we can write |dD ( Pd , Pg ) −dD ( P̂d , P̂g ) | = |maxD∈D V ( Pd , Pz , D , G ) −maxD∈D V ( P̂d , P̂z , D , G ) | , where Pg is the induced distribution by putting samples from Pz through generatorG . Both Arora et al . ( 2017 ) and Husain et al . ( 2019 ) analyze |maxD∈D V ( Pd , Pz , D , Go ) −maxD∈D V ( P̂d , P̂z , D , Go ) | to see generalization of a trained Go , while ( Zhang et al. , 2018 ; Jiang et al. , 2019 ) provide upper bounds for |maxD∈D V ( Pd , P̂z , D , G ) − minG∈G maxD∈D V ( Pd , Pz , D , G ) | . Note that those quantities of interest are non-standard in terms of learning theory . A major limitation of those distance-based bounds ( Arora et al. , 2017 ; Zhang et al. , 2018 ; Jiang et al. , 2019 ; Husain et al. , 2019 ) is the dependence on the notion of distance dD ( · , · ) which relies on the best D ∈ D for measuring proximity between two distributions . The distance between two given distributions ( µ , ν ) may be small even when the two are far away ( Arora et al. , 2017 ) . This is because there exists a perfect discriminator D , whenever µ and ν do not have overlapping supports ( Arjovsky & Bottou , 2017 ) . In those cases , a distance-based bound may be trivial . As a result , existing distance-based bounds are insufficient to understand generalization of GANs . Qi ( 2020 ) shows a generalization bound for their proposed Loss-Sensitive GAN . Nonetheless , it is nontrivial to make their bound to work with other GAN losses . Wu et al . ( 2019 ) show that the discriminator will generalize if the learning algorithm is differentially private . Their concept of differential privacy basically requires that the learned function will change negligibly if the training set slightly changes . Such a requirement is known as algorithmic stability ( Xu et al. , 2010 ) and is nontrivial to assure in practice . Note that this assumption can not be satisfied for GANs since their training is well-known to be unstable in practice . 1In general , D can be replaced by another family of functions to define the neural distance . However , for the ease of comparison with our work , the discriminator family is used . Lipschitz continuity , stability , and generalization : Lipschitz continuity naturally appears in the formulation of Wasserstein GAN ( WGAN ) ( Arjovsky et al. , 2017 ) . It was then quickly recognized as a key to improve various GANs ( Fedus et al. , 2018 ; Lucic et al. , 2018 ; Mescheder et al. , 2018 ; Kurach et al. , 2019 ; Jenni & Favaro , 2019 ; Wu et al. , 2019 ; Zhou et al. , 2019 ; Qi , 2020 ; Chu et al. , 2020 ) . Gradient penalty ( Gulrajani et al. , 2017 ) and spectral normalization ( Miyato et al. , 2018 ) are two popular techniques to constraint the Lipschitz continuity ofD orGw.r.t their inputs . Some other works ( Mescheder et al. , 2017 ; Nagarajan & Kolter , 2017 ; Sanjabi et al. , 2018 ; Nie & Patel , 2019 ) suggest to control the Lipschitz continuity of D or G w.r.t their parameters . Data augmentation is another way to control the Lipschitz constant of the loss , and is really beneficial for training GANs ( Zhao et al. , 2020a ; b ; Zhang et al. , 2020 ; Tran et al. , 2021 ) . Those works empirically found that Lipschitz continuity can help improving stability and generalization of GANs . However , it has long been a mystery of why imposing a Lipschitz constraint can help GANs to generalize well . This work provides an unified explanation . 3 LIPSCHITZ CONTINUITY AND GENERALIZATION . In this section , we will present the main theory that connects Lipschitz continuity and generalization . We then discuss why deep neural networks can avoid the curse of dimensionality , and why deeper networks may generalize better . Notations : Consider a learning problem specified by a function/hypothesis class H , an instance set X with diameter at most B , and a loss function f : H × X → R which is bounded by a constant C. Given a distribution Px defined on X , the quality of a function h ( x ) is measured by its expected loss F ( Px , h ) = Ex∼Px [ f ( h , x ) ] . Since Px is unknown , we need to rely on a finite training sample S = { x1 , ... , xm } ⊂ X and often work with the empirical loss F ( P̂x , h ) = Ex∼P̂x [ f ( h , x ) ] = 1 m ∑ x∈S f ( h , x ) , where P̂x is the empirical distribution defined on S. A learning algorithmA will pick a function hm ∈ H based on input S , i.e. , hm = A ( H , S ) . We first establish the following result whose proof appears in Appendix A. Theorem 1 ( Lipschitz continuity ⇒ Generalization ) . If a loss f ( h , x ) is L-Lipschitz continuous w.r.t input x in a compact set X ⊂ Rnx , for any h ∈ H , and P̂x is the empirical distribution defined from m i.i.d . samples from distribution Px , then sup h∈H |F ( Px , h ) − F ( P̂x , h ) | is upper-bounded by 1 . Lλ + C √ ( dBnxλ−nxe log 4− 2 log δ ) /m with probability at least 1 − δ , for any constants δ ∈ ( 0 , 1 ] and λ ∈ ( 0 , B ] . 2 . ( LB + 2C ) m−α/nx with probability at least 1− 2 exp ( −0.5mα ) , for any α ≤ nx/ ( 2 + nx ) . The assumption about Lipschitzness is natural . When learning a bounded function ( e.g . a classifier ) , the assumption will satisfy if choosing a suitable loss ( e.g . square loss , hinge loss , ramp loss , logistic loss , tangent loss , pinball loss ) and H which has Lipschitz members with bounded ouputs . Crossentropy loss can satisfy if we require every h ∈ H to have outputs belonging to a closed interval in ( 0,1 ) or use label smoothing . Some recent works ( Miyato et al. , 2018 ; Gouk et al. , 2021 ; Pauli et al. , 2021 ) propose to put a penalty on the Lipschitz constant of h only . However , leaving open the Lipschitzness of f w.r.t h may not ensure a small Lipschitz constant of the loss . This theorem tells that Lipschitz continuity is the key to ensure a function to generalize . Its generalization bounds can be better as the Lipschitz constant of the loss decreases . Note that there is a tradeoff between the Lipschitz constant and the expected loss F ( Px , h ) of the learnt function . A smaller L means that both f and h are getting simpler and flatter , and hence may increase F ( Px , h ) . In contrast , a decrease of F ( Px , h ) may require h to be more complex and hence may increase L. Theorem 1 presents generalization bounds , in a general setting , which suffer from the curse of dimensionality . This limitation is common for any other approaches without further assumptions ( Bach , 2017 ) . For some special classes , we can overcome this limitation as discussed next .

Summary: This paper's primary focus is to provide insights into the relation between Lipschitz-continuity and the generalization of DNNs.  Compared to previous work, tighter bounds connecting the Lipschitz-constant of the "loss" and the expected loss of the learned function are provided. The paper also provides insights into why SN and Dropout improve generalizability in DNNs. Some quite insightful theorems connecting the generalizability of GANs to various empirical tricks commonly used to improve their performance are also provided.