Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures

1 INTRODUCTION . The performance of machine learning methods depends strongly on the choice of the data representation ( or features ) on which they are applied . This data representation should ideally contain relevant information about the learning task in order to achieve learning with simple models and small amount of samples . Deep neural networks ( Rumelhart et al. , 1988 ) have particularly shown impressive results by automatically learning representations from raw data ( e.g. , images ) . However , due to the complex structure of deep learning models , the characterization of their hidden representations is still an open problem ( Bengio et al. , 2009 ) . Specifically , quantifying what makes a given deep learning representation better than another is a fundamental question in the field of Representation Learning ( Bengio et al. , 2013 ) . Relying on ( Montavon et al. , 2011 ) a data representation is said to be good when it is possible to build simple models on top of it that are accurate for the given learning problem . Montavon et al . ( 2011 ) have notably quantified the layer-wise evolution of the representation in deep networks by computing the principal components of the Gram matrix G ` = { φ ` ( xi ) ᵀφ ` ( xj ) } ni , j=1 at each layer for n input data x1 , . . . , xn , where φ ` ( x ) is the representation of x at layer ` of the given DL model , and the number of components controls the model simplicity . In their study , the impact of the representation at each layer is quantified through the prediction error of a linear predictor trained on the principal subspace of G ` . Pursuing on this idea , given a certain representation model x 7→ φ ( x ) , we aim in this article at theoretically studying the large dimensional behavior , and in particular the spectral information ( i.e. , eigenvalues and dominant eigenvectors ) , of the corresponding Gram matrix G = { φ ( xi ) ᵀφ ( xj ) } ni , j=1 in order to determine the information encoded ( i.e. , the sufficient statistics ) by the representation model on a set of real data x1 , . . . , xn . Indeed , standard classification and regression algorithms –along with the last layer of a neural network ( Yeh et al. , 2018 ) – retrieve the data information directly from functionals or the eigenspectrum of G1 . To this end , though , one needs a statistical model for the representations given the distribution of the raw data ( e.g. , images ) which is generally unknown . Yet , due to recent advances in generative models since the advent of Generative Adversarial Nets ( Goodfellow et al. , 2014 ) , it is now possible to generate complex data 1For instance , spectral clustering uses the dominant eigenvectors of G , while support vector machines use functionals ( quadratic forms ) involving G. structures by applying successive Lipschitz operations to Gaussian random vectors . In particular , GAN-data are used in practice as substitutes of real data for data augmentation ( Antoniou et al. , 2017 ) . On the other hand , the fundamental concentration of measure phenomenon ( Ledoux , 2005 ) tells us that Lipschitz-ally transformed Gaussian vectors satisfy a concentration property . Precisely , defining the class of concentrated vectors x ∈ E through concentration inequalities of f ( x ) , for any real Lipschitz observation f : E → R , implies that deep learning representations of GAN-data fall within this class of random vectors , since the mapping x 7→ φ ( x ) is Lipschitz . Thus , GAN-data are concentrated random vectors and thus an appropriate statistical model of realistic data . Targeting classification applications by assuming a mixture of concentrated random vectors model , this article studies the spectral behavior of Gram matrices G in the large n , p regime . Precisely , we show that these matrices have asymptotically ( as n , p → ∞ with p/n → c < ∞ ) the same firstorder behavior as for a Gaussian Mixture Model ( GMM ) . As a result , by generating images using the BigGAN model ( Brock et al. , 2018 ) and considering different commonly used deep representation models , we show that the spectral behavior of the Gram matrix computed on these representations is the same as on a GMM model with the same p-dimensional means and covariances . A surprising consequence is that , for GAN data , the aforementioned sufficient statistics to characterize the quality of a given representation network are only the first and second order statistics of the representations . This behavior is shown by simulations to extend beyond random GAN-data to real images from the Imagenet dataset ( Deng et al. , 2009 ) . The rest of the paper is organized as follows . In Section 2 , we introduce the notion of concentrated vectors and their main properties . Our main theoretical results are then provided in Section 3 . In Section 4 we present experimental results . Section 5 concludes the article . Notation : In the following , we use the notation from ( Goodfellow et al. , 2016 ) . [ n ] denotes the set { 1 , . . . , n } . Given a vector x ∈ Rn , the ` 2-norm of x is given as ‖x‖2 = ∑n i=1 x 2 i . Given a p× n matrix M , its Frobenius norm is defined as ‖M‖2F = ∑p i=1 ∑n j=1 M 2 ij and its spectral norm as ‖M‖ = sup‖x‖=1 ‖Mx‖ . for the Hadamard product . An application F : E → F is said to be ‖F‖lip-Lipschitz , if ∀ ( x , y ) ∈ E2 , ‖F ( x ) −F ( y ) ‖F ≤ ‖F‖lip · ‖x− y‖E and ‖F‖lip is finite . 2 BASIC NOTIONS OF CONCENTRATED VECTORS . Being the central tool of our study , we start by introducing the notion of concentrated vectors . While advanced concentration notions have been recently developed in ( Louart & Couillet , 2019 ) in order to specifically analyze the behavior of large dimensional sample covariance matrices , for simplicity , we restrict ourselves here to the sufficient so-called q-exponentially concentrated random vectors . Definition 2.1 ( q-exponential concentration ) . Given a normed space ( E , ‖ · ‖E ) and a real q , a random vector x ∈ E is said to be q-exponentially concentrated if for any 1-Lipschitz real function f : E → R , there exists C ≥ 0 independent of dim ( E ) and σ > 0 such that for all t ≥ 0 P { |f ( x ) − Ef ( x ) | > t } ≤ C e− ( t/σ ) q ( 1 ) which we denote x ∈ Eq ( σ |E , ‖ · ‖E ) . We simply write x ∈ Eq ( 1 |E , ‖ · ‖E ) if the tail parameter σ does not depend on dim ( E ) , and x ∈ Eq ( 1 ) for x a scalar real random variable . Therefore , concentrated vectors are defined through the concentration of any 1-Lipschitz real scalar “ observation ” . One of the most important examples of concentrated vectors are standard Gaussian vectors . Precisely , we have the following proposition . See ( Ledoux , 2005 ) ) for more examples such as uniform and Gamma distribution . Proposition 2.2 ( Gaussian vectors ( Ledoux , 2005 ) ) . Let d ∈ N and x ∼ N ( 0 , Id ) . Then x is a 2-exponentially concentrated vector independently on the dimension d , i.e . x ∈ E2 ( 1 |Rd , ‖ · ‖ ) . Concentrated vectors have the interesting property of being stable by application of Rd → Rp vector-Lipschitz transformations . Indeed , Lipschitz-ally transformed concentrated vectors remain concentrated according to the following proposition . Proposition 2.3 ( Lipschitz stability ( Louart & Couillet , 2019 ) ) . Let x ∈ Eq ( 1 |E , ‖ · ‖E ) and G : E → F a Lipschitz application with Lipschitz constant ‖G‖lip which may depend on dim ( F ) . Then the concentration property on x is transferred to G ( x ) , precisely x ∈ Eq ( 1 |E , ‖ · ‖E ) ⇒ G ( x ) ∈ Eq ( ‖G‖lip |F , ‖ · ‖F ) . ( 2 ) Note importantly for the following that the Lipschitz constant of the transformation G must be controlled , in order to constrain the tail parameter of the obtained concentration . In particular , we have the coming corollary to Proposition 2.3 of central importance in the following . Corollary 2.4 . Let G1 , . . . , Gn : Rd → Rp a set of n Lipschitz applications with Lipschitz constants ‖Gi‖lip . Let G : Rd×n → Rp×n be defined for each X ∈ Rd×n as G ( X ) = [ G1 ( X : ,1 ) , . . . , Gn ( X : ,n ) ] . Then , Z ∈ Eq ( 1 |Rd×n , ‖ · ‖F ) ⇒ G ( Z ) ∈ Eq ( sup i ‖Gi‖lip | Rp×n , ‖ · ‖F ) . ( 3 ) Proof . This is a consequence of Proposition 2.3 since the map G is supi ‖Gi‖lip-Lipschitz with respect to ( w.r.t . ) the Frobenius norm . Indeed , for X , H ∈ Rd×n : ‖G ( X + H ) − G ( X ) ‖2F ≤∑n i=1 ‖Gi‖2lip · ‖H : ,i‖2 ≤ supi ‖Gi‖2lip · ‖H‖2F . 3 MAIN RESULTS . 3.1 GAN DATA : AN EXAMPLE OF CONCENTRATED VECTORS . Concentrated random vectors are particularly interesting from a practical standpoint for real data modeling . In fact , unlike simple Gaussian vectors , the former do not suffer from the constraint of having independent entries which is quite a restrictive assumption when modeling real data such as images or their non-linear features ( e.g. , DL representations ) . The other modeling interest of concentrated vectors lies in their being already present in practice as alternatives to real data . Indeed , adversarial neural networks ( GANs ) have the ability nowadays to generate random realistic data ( for instance realistic images ) by applying successive Lipschitz operations to standard Gaussian vectors ( Goodfellow et al. , 2014 ) . A GAN architecture involves two networks , a generator model which maps random Gaussian noise to new plausible synthetic data and a discriminator model which classifies real data as real ( from the dataset ) or fake ( for the generated data ) . The discriminator is updated directly through a binary classification problem , whereas the generator is updated through the discriminator . As such , the two models are trained alternatively in an adversarial manner , where the generator seeks to better deceive the discriminator and the former seeks to better identify the fake data ( Goodfellow et al. , 2014 ) . In particular , once both models are trained ( when they reach a Nash equilibrium ) , DL representations of GAN-data –and GAN-data themselves– are schematically constructed in practice as follows : Real Data ≈ GAN Data = FN ◦ · · · ◦ F1 ( z ) , where z ∼ N ( 0 , Id ) , ( 4 ) where d stands for the input dimension of the generator model , N the number of layers , and the Fi ’ s either Fully Connected Layers , Convolutional Layers , Pooling Layers , Up-sampling Layers and Activation Functions , Residual Layers or Batch Normalizations . All these operations happen to be Lipschitz applications . Precisely , • Fully Connected Layers and Convolutional Layers : These are affine operations which can be expressed as Fi ( x ) = Wix + bi , for Wi the weight matrix and bi the bias vector . Here the Lipschitz constant is the operator norm ( the largest singular value ) of the weight matrix Wi , that is ‖Fi‖lip = supu6=0 ‖Wiu‖2 ‖u‖2 . • Pooling Layers and Activation Functions : Most commonly used activation functions and pooling operations are ReLU ( x ) = max ( 0 , x ) , MaxPooling ( x ) = [ max ( xS1 ) , . . . , max ( xSq ) ] ᵀ , where Si ’ s are patches ( i.e. , subsets of [ dim ( x ) ] ) . These are at most 1-Lipschitz operations with respect to the Frobenius norm . Specifically , the maximum absolute sub-gradient of the ReLU activation function is 1 , thus the ReLU operation has a Lipschitz constant of 1 . Similarly , we can show that the Lipschitz constant of MaxPooling layers is also 1 . • Residual Connections : Residual layers act the following way Fi ( x ) = x + F ( 1 ) i ◦ · · · ◦ F ( ` ) i ( x ) , where the F ( j ) i ’ s are Fully Connected Layers or Convolutional Layers with Activation Functions , and which are Lipschitz operations . Thus Fi is a Lipschitz operation with Lipschitz constant bounded by 1 + ∏ ` j=1 ‖F ( j ) i ‖lip . • Batch Normalization ( BN ) Layers : They consist in statistically standardizing ( Ioffe & Szegedy , 2015 ) the vectors of a small batch B = { xi } bi=1 ⊂ Rd as follows : for each xk ∈ B Fi ( xk ) = diag ( a√ σ2B + ) ( xk − µB1d ) + b where µB = 1db ∑b k=1 ∑d i=1 [ xk ] i , σ 2 B = 1 db ∑b k=1 ∑d i=1 ( [ xk ] i − µB ) 2 , a , b ∈ Rd are parameters to be learned and diag ( v ) transforms a vector v to a diagonal matrix with its diagonal entries being those of v. Thus BN is a Lipschitz transformation with Lipschitz constant ‖Fi‖lip = supi | ai√σ2B+ | . Therefore , as illustrated in Figure 1 , since standard Gaussian vectors are concentrated vectors as mentioned in Proposition 2.2 and since the notion of concentrated vectors is stable by Lipschitz transformations thanks to Proposition 2.3 , GAN-data ( and their DL representations ) are concentrated vectors by design given the construction in Equation ( 4 ) . Moreover , in order to generate data belonging to a specific class , Conditional GANs have been introduced ( Mirza & Osindero , 2014 ) ; once again data generated by these models are concentrated vectors as a consequence of Corollary 2.4 . Indeed , a generator of a Conditional GAN model can be seen as a set of multiple generators where each generates data of a specific class conditionally on the class label ( e.g. , BigGAN model ( Brock et al. , 2018 ) ) . Yet , in order to ensure that the resulting Lipschitz constant of the combination of the above operations does not scale with the network or data size , so to maintain good concentration behaviors , a careful control of the learned network parameters is needed . This control happens to be already considered in practice in order to ensure the stability of GANs during the learning phase , notably to generate realistic and high-resolution images ( Roth et al. , 2017 ; Brock et al. , 2018 ) . The control of the Lipschitz constant of representation networks is also needed in practice in order to make them robust against adversarial examples ( Szegedy et al. , 2013 ; Gulrajani et al. , 2017 ) . This control is particularly ensured through spectral normalization of the affine layers ( Brock et al. , 2018 ) , such as Fully Connected Layers , Convolutional Layers and Batch Normalization . Indeed , spectral normalization ( Miyato et al. , 2018 ) consists in applying the operation W ← W /σ1 ( W ) to the affine layers at each backward iteration of the back-propagation algorithm , where σ1 ( W ) stands for the largest singular value of the weight matrix W . Brock et al . ( 2018 ) , have notably observed that , without spectral constraints , a subset of the generator layers grow throughout their GAN training and explode at collapse . They thus suggested the following spectral normalization –which happens to be less restrictive than the standard spectral normalization W ← W /σ1 ( W ) ( Miyato et al. , 2018 ) – to the affine layers : W ←W − ( σ1 ( W ) − σ∗ ) u1 ( W ) v1 ( W ) ᵀ ( 5 ) where u1 ( W ) and v1 ( W ) denote respectively the left and right largest singular vectors of W , and σ∗ is an hyper-parameter fixed during training . To get an insight about the influence of this operation and to ensure that it controls the Lipschitz constant of the generator , the following proposition provides the dynamics of a random walk in the space of parameters along with the spectral normalization in Equation ( 5 ) . Indeed , since stochastic gradient descent ( SGD ) consists in estimating the gradient of the loss function on randomly selected batches of data , it can be assimilated to a random walk in the space of parameters ( Antognini & Sohl-Dickstein , 2018 ) . Proposition 3.1 ( Lipschitz constant control ) . Let σ∗ > 0 and G be a neural network composed ofN affine layers , each one of input dimension di−1 and output dimension di for i ∈ [ N ] , with 1-Lipschitz activation functions . Assume that the weights of G at layer i+ 1 are initialized as U ( [ − 1√ di , 1√ di ] ) , and consider the following dynamics with learning rate η : W ←W − ηE , with Ei , j ∼ N ( 0 , 1 ) W ←W −max ( 0 , σ1 ( W ) − σ∗ ) u1 ( W ) v1 ( W ) ᵀ . ( 6 ) Then , ∀ε > 0 , the Lipschitz constant of G is bounded at convergence with high probability as : ‖G‖lip ≤ N∏ i=1 ( ε+ √ σ2∗ + η 2didi−1 ) . ( 7 ) Proof . The proof is provided in Appendix B . Proposition 3.1 shows that the Lipschitz constant of a neural network is controlled when trained with the spectral normalization in Equation ( 5 ) . In particular , recalling the notations in Proposition 3.1 , in the limit where di →∞ with didi−1 → γi ∈ ( 0 , ∞ ) for all i ∈ [ N ] and choosing the learning rate η = O ( d−10 ) , the Lipschitz constant of G is of order O ( 1 ) if it has finitely many layers N and σ∗ is constant . Therefore , with this spectral normalization , it can be assumed that ‖G‖lip = O ( 1 ) when dimensions grow . Figure 2 depicts the behavior of the Lipschitz constant of a linear layer with and without spectral normalization in the setting of Proposition 3.1 , which confirms the obtained bound .

Summary: The authors generalize Gaussian random vectors to a broader class of "concentrated" vectors which they use as their primary tool for analysis of the latent representations learned by GANs. They show that the spectral behavior (i.e. spectra and leading eigenspaces) of the Gram matrix computed over GAN representations is the same as those produced by a high dimensional Gaussian Mixture Models (GMMs). Furthermore, they show that the "sufficient statistics" (i.e. measures of information encoded in the latent representations) for GANs depend only on their first and second moments. Thus, for data that follows Gaussian mixture patterns, GANs and GMMs behave identically. The authors also show that common neural network operations (linear/convolutional layers, pooling, batch normalization, ReLU activations) are Lipschitz transformations, where the Lipschitz constants can be bounded using spectral normalization (this is borrowed from previous work). They also provide some empirical analysis to verify their theoretical findings.