Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures

1 INTRODUCTION . The performance of machine learning methods depends strongly on the choice of the data representation ( or features ) on which they are applied . This data representation should ideally contain relevant information about the learning task in order to achieve learning with simple models and small amount of samples . Deep neural networks ( Rumelhart et al. , 1988 ) have particularly shown impressive results by automatically learning representations from raw data ( e.g. , images ) . However , due to the complex structure of deep learning models , the characterization of their hidden representations is still an open problem ( Bengio et al. , 2009 ) . Specifically , quantifying what makes a given deep learning representation better than another is a fundamental question in the field of Representation Learning ( Bengio et al. , 2013 ) . Relying on ( Montavon et al. , 2011 ) a data representation is said to be good when it is possible to build simple models on top of it that are accurate for the given learning problem . Montavon et al . ( 2011 ) have notably quantified the layer-wise evolution of the representation in deep networks by computing the principal components of the Gram matrix G ` = { φ ` ( xi ) ᵀφ ` ( xj ) } ni , j=1 at each layer for n input data x1 , . . . , xn , where φ ` ( x ) is the representation of x at layer ` of the given DL model , and the number of components controls the model simplicity . In their study , the impact of the representation at each layer is quantified through the prediction error of a linear predictor trained on the principal subspace of G ` . Pursuing on this idea , given a certain representation model x 7→ φ ( x ) , we aim in this article at theoretically studying the large dimensional behavior , and in particular the spectral information ( i.e. , eigenvalues and dominant eigenvectors ) , of the corresponding Gram matrix G = { φ ( xi ) ᵀφ ( xj ) } ni , j=1 in order to determine the information encoded ( i.e. , the sufficient statistics ) by the representation model on a set of real data x1 , . . . , xn . Indeed , standard classification and regression algorithms –along with the last layer of a neural network ( Yeh et al. , 2018 ) – retrieve the data information directly from functionals or the eigenspectrum of G1 . To this end , though , one needs a statistical model for the representations given the distribution of the raw data ( e.g. , images ) which is generally unknown . Yet , due to recent advances in generative models since the advent of Generative Adversarial Nets ( Goodfellow et al. , 2014 ) , it is now possible to generate complex data 1For instance , spectral clustering uses the dominant eigenvectors of G , while support vector machines use functionals ( quadratic forms ) involving G. structures by applying successive Lipschitz operations to Gaussian random vectors . In particular , GAN-data are used in practice as substitutes of real data for data augmentation ( Antoniou et al. , 2017 ) . On the other hand , the fundamental concentration of measure phenomenon ( Ledoux , 2005 ) tells us that Lipschitz-ally transformed Gaussian vectors satisfy a concentration property . Precisely , defining the class of concentrated vectors x ∈ E through concentration inequalities of f ( x ) , for any real Lipschitz observation f : E → R , implies that deep learning representations of GAN-data fall within this class of random vectors , since the mapping x 7→ φ ( x ) is Lipschitz . Thus , GAN-data are concentrated random vectors and thus an appropriate statistical model of realistic data . Targeting classification applications by assuming a mixture of concentrated random vectors model , this article studies the spectral behavior of Gram matrices G in the large n , p regime . Precisely , we show that these matrices have asymptotically ( as n , p → ∞ with p/n → c < ∞ ) the same firstorder behavior as for a Gaussian Mixture Model ( GMM ) . As a result , by generating images using the BigGAN model ( Brock et al. , 2018 ) and considering different commonly used deep representation models , we show that the spectral behavior of the Gram matrix computed on these representations is the same as on a GMM model with the same p-dimensional means and covariances . A surprising consequence is that , for GAN data , the aforementioned sufficient statistics to characterize the quality of a given representation network are only the first and second order statistics of the representations . This behavior is shown by simulations to extend beyond random GAN-data to real images from the Imagenet dataset ( Deng et al. , 2009 ) . The rest of the paper is organized as follows . In Section 2 , we introduce the notion of concentrated vectors and their main properties . Our main theoretical results are then provided in Section 3 . In Section 4 we present experimental results . Section 5 concludes the article . Notation : In the following , we use the notation from ( Goodfellow et al. , 2016 ) . [ n ] denotes the set { 1 , . . . , n } . Given a vector x ∈ Rn , the ` 2-norm of x is given as ‖x‖2 = ∑n i=1 x 2 i . Given a p× n matrix M , its Frobenius norm is defined as ‖M‖2F = ∑p i=1 ∑n j=1 M 2 ij and its spectral norm as ‖M‖ = sup‖x‖=1 ‖Mx‖ . for the Hadamard product . An application F : E → F is said to be ‖F‖lip-Lipschitz , if ∀ ( x , y ) ∈ E2 , ‖F ( x ) −F ( y ) ‖F ≤ ‖F‖lip · ‖x− y‖E and ‖F‖lip is finite . 2 BASIC NOTIONS OF CONCENTRATED VECTORS . Being the central tool of our study , we start by introducing the notion of concentrated vectors . While advanced concentration notions have been recently developed in ( Louart & Couillet , 2019 ) in order to specifically analyze the behavior of large dimensional sample covariance matrices , for simplicity , we restrict ourselves here to the sufficient so-called q-exponentially concentrated random vectors . Definition 2.1 ( q-exponential concentration ) . Given a normed space ( E , ‖ · ‖E ) and a real q , a random vector x ∈ E is said to be q-exponentially concentrated if for any 1-Lipschitz real function f : E → R , there exists C ≥ 0 independent of dim ( E ) and σ > 0 such that for all t ≥ 0 P { |f ( x ) − Ef ( x ) | > t } ≤ C e− ( t/σ ) q ( 1 ) which we denote x ∈ Eq ( σ |E , ‖ · ‖E ) . We simply write x ∈ Eq ( 1 |E , ‖ · ‖E ) if the tail parameter σ does not depend on dim ( E ) , and x ∈ Eq ( 1 ) for x a scalar real random variable . Therefore , concentrated vectors are defined through the concentration of any 1-Lipschitz real scalar “ observation ” . One of the most important examples of concentrated vectors are standard Gaussian vectors . Precisely , we have the following proposition . See ( Ledoux , 2005 ) ) for more examples such as uniform and Gamma distribution . Proposition 2.2 ( Gaussian vectors ( Ledoux , 2005 ) ) . Let d ∈ N and x ∼ N ( 0 , Id ) . Then x is a 2-exponentially concentrated vector independently on the dimension d , i.e . x ∈ E2 ( 1 |Rd , ‖ · ‖ ) . Concentrated vectors have the interesting property of being stable by application of Rd → Rp vector-Lipschitz transformations . Indeed , Lipschitz-ally transformed concentrated vectors remain concentrated according to the following proposition . Proposition 2.3 ( Lipschitz stability ( Louart & Couillet , 2019 ) ) . Let x ∈ Eq ( 1 |E , ‖ · ‖E ) and G : E → F a Lipschitz application with Lipschitz constant ‖G‖lip which may depend on dim ( F ) . Then the concentration property on x is transferred to G ( x ) , precisely x ∈ Eq ( 1 |E , ‖ · ‖E ) ⇒ G ( x ) ∈ Eq ( ‖G‖lip |F , ‖ · ‖F ) . ( 2 ) Note importantly for the following that the Lipschitz constant of the transformation G must be controlled , in order to constrain the tail parameter of the obtained concentration . In particular , we have the coming corollary to Proposition 2.3 of central importance in the following . Corollary 2.4 . Let G1 , . . . , Gn : Rd → Rp a set of n Lipschitz applications with Lipschitz constants ‖Gi‖lip . Let G : Rd×n → Rp×n be defined for each X ∈ Rd×n as G ( X ) = [ G1 ( X : ,1 ) , . . . , Gn ( X : ,n ) ] . Then , Z ∈ Eq ( 1 |Rd×n , ‖ · ‖F ) ⇒ G ( Z ) ∈ Eq ( sup i ‖Gi‖lip | Rp×n , ‖ · ‖F ) . ( 3 ) Proof . This is a consequence of Proposition 2.3 since the map G is supi ‖Gi‖lip-Lipschitz with respect to ( w.r.t . ) the Frobenius norm . Indeed , for X , H ∈ Rd×n : ‖G ( X + H ) − G ( X ) ‖2F ≤∑n i=1 ‖Gi‖2lip · ‖H : ,i‖2 ≤ supi ‖Gi‖2lip · ‖H‖2F . 3 MAIN RESULTS . 3.1 GAN DATA : AN EXAMPLE OF CONCENTRATED VECTORS . Concentrated random vectors are particularly interesting from a practical standpoint for real data modeling . In fact , unlike simple Gaussian vectors , the former do not suffer from the constraint of having independent entries which is quite a restrictive assumption when modeling real data such as images or their non-linear features ( e.g. , DL representations ) . The other modeling interest of concentrated vectors lies in their being already present in practice as alternatives to real data . Indeed , adversarial neural networks ( GANs ) have the ability nowadays to generate random realistic data ( for instance realistic images ) by applying successive Lipschitz operations to standard Gaussian vectors ( Goodfellow et al. , 2014 ) . A GAN architecture involves two networks , a generator model which maps random Gaussian noise to new plausible synthetic data and a discriminator model which classifies real data as real ( from the dataset ) or fake ( for the generated data ) . The discriminator is updated directly through a binary classification problem , whereas the generator is updated through the discriminator . As such , the two models are trained alternatively in an adversarial manner , where the generator seeks to better deceive the discriminator and the former seeks to better identify the fake data ( Goodfellow et al. , 2014 ) . In particular , once both models are trained ( when they reach a Nash equilibrium ) , DL representations of GAN-data –and GAN-data themselves– are schematically constructed in practice as follows : Real Data ≈ GAN Data = FN ◦ · · · ◦ F1 ( z ) , where z ∼ N ( 0 , Id ) , ( 4 ) where d stands for the input dimension of the generator model , N the number of layers , and the Fi ’ s either Fully Connected Layers , Convolutional Layers , Pooling Layers , Up-sampling Layers and Activation Functions , Residual Layers or Batch Normalizations . All these operations happen to be Lipschitz applications . Precisely , • Fully Connected Layers and Convolutional Layers : These are affine operations which can be expressed as Fi ( x ) = Wix + bi , for Wi the weight matrix and bi the bias vector . Here the Lipschitz constant is the operator norm ( the largest singular value ) of the weight matrix Wi , that is ‖Fi‖lip = supu6=0 ‖Wiu‖2 ‖u‖2 . • Pooling Layers and Activation Functions : Most commonly used activation functions and pooling operations are ReLU ( x ) = max ( 0 , x ) , MaxPooling ( x ) = [ max ( xS1 ) , . . . , max ( xSq ) ] ᵀ , where Si ’ s are patches ( i.e. , subsets of [ dim ( x ) ] ) . These are at most 1-Lipschitz operations with respect to the Frobenius norm . Specifically , the maximum absolute sub-gradient of the ReLU activation function is 1 , thus the ReLU operation has a Lipschitz constant of 1 . Similarly , we can show that the Lipschitz constant of MaxPooling layers is also 1 . • Residual Connections : Residual layers act the following way Fi ( x ) = x + F ( 1 ) i ◦ · · · ◦ F ( ` ) i ( x ) , where the F ( j ) i ’ s are Fully Connected Layers or Convolutional Layers with Activation Functions , and which are Lipschitz operations . Thus Fi is a Lipschitz operation with Lipschitz constant bounded by 1 + ∏ ` j=1 ‖F ( j ) i ‖lip . • Batch Normalization ( BN ) Layers : They consist in statistically standardizing ( Ioffe & Szegedy , 2015 ) the vectors of a small batch B = { xi } bi=1 ⊂ Rd as follows : for each xk ∈ B Fi ( xk ) = diag ( a√ σ2B + ) ( xk − µB1d ) + b where µB = 1db ∑b k=1 ∑d i=1 [ xk ] i , σ 2 B = 1 db ∑b k=1 ∑d i=1 ( [ xk ] i − µB ) 2 , a , b ∈ Rd are parameters to be learned and diag ( v ) transforms a vector v to a diagonal matrix with its diagonal entries being those of v. Thus BN is a Lipschitz transformation with Lipschitz constant ‖Fi‖lip = supi | ai√σ2B+ | . Therefore , as illustrated in Figure 1 , since standard Gaussian vectors are concentrated vectors as mentioned in Proposition 2.2 and since the notion of concentrated vectors is stable by Lipschitz transformations thanks to Proposition 2.3 , GAN-data ( and their DL representations ) are concentrated vectors by design given the construction in Equation ( 4 ) . Moreover , in order to generate data belonging to a specific class , Conditional GANs have been introduced ( Mirza & Osindero , 2014 ) ; once again data generated by these models are concentrated vectors as a consequence of Corollary 2.4 . Indeed , a generator of a Conditional GAN model can be seen as a set of multiple generators where each generates data of a specific class conditionally on the class label ( e.g. , BigGAN model ( Brock et al. , 2018 ) ) . Yet , in order to ensure that the resulting Lipschitz constant of the combination of the above operations does not scale with the network or data size , so to maintain good concentration behaviors , a careful control of the learned network parameters is needed . This control happens to be already considered in practice in order to ensure the stability of GANs during the learning phase , notably to generate realistic and high-resolution images ( Roth et al. , 2017 ; Brock et al. , 2018 ) . The control of the Lipschitz constant of representation networks is also needed in practice in order to make them robust against adversarial examples ( Szegedy et al. , 2013 ; Gulrajani et al. , 2017 ) . This control is particularly ensured through spectral normalization of the affine layers ( Brock et al. , 2018 ) , such as Fully Connected Layers , Convolutional Layers and Batch Normalization . Indeed , spectral normalization ( Miyato et al. , 2018 ) consists in applying the operation W ← W /σ1 ( W ) to the affine layers at each backward iteration of the back-propagation algorithm , where σ1 ( W ) stands for the largest singular value of the weight matrix W . Brock et al . ( 2018 ) , have notably observed that , without spectral constraints , a subset of the generator layers grow throughout their GAN training and explode at collapse . They thus suggested the following spectral normalization –which happens to be less restrictive than the standard spectral normalization W ← W /σ1 ( W ) ( Miyato et al. , 2018 ) – to the affine layers : W ←W − ( σ1 ( W ) − σ∗ ) u1 ( W ) v1 ( W ) ᵀ ( 5 ) where u1 ( W ) and v1 ( W ) denote respectively the left and right largest singular vectors of W , and σ∗ is an hyper-parameter fixed during training . To get an insight about the influence of this operation and to ensure that it controls the Lipschitz constant of the generator , the following proposition provides the dynamics of a random walk in the space of parameters along with the spectral normalization in Equation ( 5 ) . Indeed , since stochastic gradient descent ( SGD ) consists in estimating the gradient of the loss function on randomly selected batches of data , it can be assimilated to a random walk in the space of parameters ( Antognini & Sohl-Dickstein , 2018 ) . Proposition 3.1 ( Lipschitz constant control ) . Let σ∗ > 0 and G be a neural network composed ofN affine layers , each one of input dimension di−1 and output dimension di for i ∈ [ N ] , with 1-Lipschitz activation functions . Assume that the weights of G at layer i+ 1 are initialized as U ( [ − 1√ di , 1√ di ] ) , and consider the following dynamics with learning rate η : W ←W − ηE , with Ei , j ∼ N ( 0 , 1 ) W ←W −max ( 0 , σ1 ( W ) − σ∗ ) u1 ( W ) v1 ( W ) ᵀ . ( 6 ) Then , ∀ε > 0 , the Lipschitz constant of G is bounded at convergence with high probability as : ‖G‖lip ≤ N∏ i=1 ( ε+ √ σ2∗ + η 2didi−1 ) . ( 7 ) Proof . The proof is provided in Appendix B . Proposition 3.1 shows that the Lipschitz constant of a neural network is controlled when trained with the spectral normalization in Equation ( 5 ) . In particular , recalling the notations in Proposition 3.1 , in the limit where di →∞ with didi−1 → γi ∈ ( 0 , ∞ ) for all i ∈ [ N ] and choosing the learning rate η = O ( d−10 ) , the Lipschitz constant of G is of order O ( 1 ) if it has finitely many layers N and σ∗ is constant . Therefore , with this spectral normalization , it can be assumed that ‖G‖lip = O ( 1 ) when dimensions grow . Figure 2 depicts the behavior of the Lipschitz constant of a linear layer with and without spectral normalization in the setting of Proposition 3.1 , which confirms the obtained bound .

Summary: In this paper, the authors claim to establish a Lipschitz bound on neural networks, initialized randomly, and trained until convergence.  They also claim to establish probabilistic concentration of the resolvent of the Gram matrix from a mixture of k distributions with varying means and covariances.  The authors also present the spectrum and leading eigenspace of the Gram matrix for representations by CNNs of images generated by a GAN.