Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures
Authors: Mohamed El Amine Seddik
Cosme Louart
Mohamed Tamaazousti
Romain Couillet
Venue:  (2020)
Keywords: Random Matrix Theory
Deep Learning Representations
GANs

Abstract: This paper shows that deep learning (DL) representations of data produced by generative adversarial nets (GANs) are random vectors which fall within the class of so-called concentrated random vectors. Further exploiting the fact that Gram matrices, of the type G = X'X with X = [x_1 , . . . , x_n ] ∈ R p×n and x_i independent concentrated random vectors from a mixture model, behave asymptotically (as n, p → ∞) as if the x_i were drawn from a Gaussian mixture, suggests that DL representations of GAN-data can be fully described by their first two statistical moments for a wide range of standard classifiers. Our theoretical findings are validated by generating images with the BigGAN model and across different popular deep representation networks.

Under review as a conference paper at ICLR 2020
RANDOM MATRIX THEORY PROVES THAT DEEP LEARNING REPRESENTATIONS OF GAN-DATA BEHAVE AS GAUSSIAN MIXTURES
Anonymous authors Paper under double-blind review
ABSTRACT
This paper shows that deep learning (DL) representations of data produced by generative adversarial nets (GANs) are random vectors which fall within the class of so-called concentrated random vectors. Further exploiting the fact that Gram matrices, of the type G = -XXXX X with X [......, 2nd] E TRex and æ; inde- pendent concentrated random vectors from a mixture model, behave asymptoti- cally (as n, oc) as if the T; were drawn from Gaussian mixture, suggests that DL representations of GAN-data can be fully described by their first two sta- tistical moments for wide range of standard classifiers. Our theoretical findings are validated by generating images with the BigGAN model and across different popular deep representation networks.
INTRODUCTION
The performance of machine learning methods depends strongly on the choice of the data repre- sentation (or features) on which they are applied. This data representation should ideally contain relevant information about the learning task in order to achieve learning with simple models and small amount of samples. Deep neural networks (Rumelhart et al. 1988) have particularly shown impressive results by automatically learning representations from raw data (e.g., images). However, due to the complex structure of deep learning models, the characterization of their hidden represen- tations is still an open problem (Bengio et al. 2009).
Specifically, quantifying what makes given deep learning representation better than another is fundamental question in the field of Representation Learning (Bengio et al. 2013). Relying on (Montavon et al. [011] data representation is said to be good when it is possible to build simple models top of it that accurate for the given learning problem. Montavon et al. (2011) have notably quantified the layer-wise evolution of the representation in deep networks by comput- ing the principal components of the Gram matrix G2 = {<b((r,)) (ae.,,,TT(a))))))))))))))))) )=,=== each layer for input data .1,..., Ln, where (e(x)) is the representation of at layer of the given DL model, and
the number of components controls the model simplicity. In their study, the impact of the repre- sentation at each layer is quantified through the prediction error of linear predictor trained on the principal subspace of G..
Pursuing on this idea, given certain representation model x ((x), aim in this arti- ele at theoretically studying the large dimensional behavior, and in particular the spectral infor- mation (i.e,, eigenvalues and dominant eigenvectors), of the corresponding Gram matrix G (,(x;;) in order to determine the information encoded (i.e., the sufficient statistics) by the representation model on a set of real data æ1...., T. Indeed, standard classification and regression algorithms -along with the last layer of a neural network (Yeh et al.) [0018] retrieve the data information directly from functionals or the eigenspectrum of G To this end, though, one needs statistical model for the representations given the distribution of the raw data (e.g. images) which is generally unknown. Yet, due to recent advances in generative models since the advent of Generative Adversarial Nets (Goodfellow al. 2014). it is now possible to generate complex data
For instance, spectral clustering uses the dominant eigenvectors of G. while support vector machines functionals (quadratic forms) involving G.

---

Under review as a conference paper at ICLR 2020
structures by applying successive Lipschitz operations to Gaussian random vectors. In particular, GAN-data are used in practice as substitutes of real data for data augmentation (Antoniou et al. 2017). On the other hand, the fundamental concentration of measure phenomenon (Ledoux 2005) tells us that Lipschitz-ally transformed Gaussian vectors satisfy a concentration property. Precisely, defining the class of concentrated vectors E E. through concentration inequalities of f(() for any real Lipschitz observation f : E' + IR, implies that deep learning representations of GAN-data fall within this class of random vectors, since the mapping æ + ((x)) is Lipschitz. Thus, GAN-data are concentrated random vectors and thus an appropriate statistical model of realistic data.
Targeting classification applications by assuming mixture of concentrated random vectors model, this article studies the spectral behavior of Gram matrices G in the large p. regime. Precisely, we show that these matrices have asymptotically (as n. P - oc with p/n - c < 00) the same first- order behavior as for Gaussian Mixture Model (GMM). As a result, by generating images using the BigGAN model (Brock et al. 2018) and considering different commonly used deep representation models, show that the spectral behavior of the Gram matrix computed on these representations is the same as on a GMM model with the same p-dimensional means and covariances. A surprising consequence is that, for GAN data, the aforementioned sufficient statistics to characterize the quality of given representation network are only the first and second order statistics of the representations. This behavior is shown by simulations to extend beyond random GAN-data to real images from the Imagenet dataset (Deng et al. 2009).
The rest of the paper is organized as follows. In Section we introduce the notion of concentrated vectors and their main properties. Our main theoretical results are then provided in Section 3 In Section 4 we present experimental results. Section 5 concludes the article.
Notation: In the following, we use the notation from (Goodfellow et al. 2016 [x] denotes the set {I..... n2.. Given a vector æ E R", the eg-norm of x is given as x F Di= 2.. Given a x n
matrix M, its Frobenius norm is defined || | E Ej=1 M2 and its spectral norm as
|| = || || Ma|| @ for the Hadamard product. An application F E + F is said to be [1 Lipschitz, if V(x, y) E E., [F(z) - F(y)lle & ||F]l || u|l and ||F]ltpp is finite.
BASIC NOTIONS OF CONCENTRATED VECTORS
Being the central tool of our study, we start by introducing the notion of concentrated vectors. While advanced concentration notions have been recently developed in (Louart & Couillet 2019) in order specifically analyze the behavior of large dimensional sample covariance matrices, for simplicity, we restrict ourselves here to the sufficient so-called g-exponentially concentrated random vectors.
Definition 2.1 (q-exponential concentration). Given a normed space (F Il |[E) and a real q, random vector x E É is said to be q- g-exponentially concentrated for any L-Lipschitz real function f: K, there exists C' independent of dim and >> 0 such that for all t Z 0
P [(f(())= - -Ef(x) >1 s C(t//a))
(I)
which we denote x E Eo @ EI [[... We simply write x E, (IE) if the tail parameter does not depend on dim(E), and E (1) for I scalar real random variable.
Therefore, concentrated vectors are defined through the concentration of any 1-Lipschitz real scalar "observation". One of the most important examples of concentrated vectors are standard Gaussian vectors. Precisely, have the following proposition. See (Ledoux 2005)) for more examples such as uniform and Gamma distribution.
Proposition 2.2 (Gaussian vectors (Ledoux 2005)). Let d E N and 2- 2-exponentially concentrated vector independently on the dimension d, i.e. x E E2 R |l Il).
NOOO Then is
Concentrated vectors have the interesting property of being stable by application of IRd - RP vector-Lipschitz transformations. Indeed, Lipschitz-ally transformed concentrated vectors remain concentrated according to the following proposition.
Proposition 2.3 (Lipschitz stability (Louart & Couillet 2019)). Let x E F, [|x) and G : F + F a Lipschitz application with Lipschitz constant 19 which may depend on dim(F). Then the concentration property on x is transferred to G(z), precisely
(1 E, ([E) = g(x=) & (119 F, [|=).
2

---

Under review as a conference paper at ICLR 2020
Note importantly for the following that the Lipschitz constant of the transformation 9 must be trolled, in order to constrain the tail parameter of the obtained concentration.
In particular, have the coming corollary to Proposition of central importance in the following.
Corollary 2.4. Let 91...., In Rd RP set of Lipschitz applications with Lipschitz
constants [[illipp- Les G RRdx - TRxx be defined for each X E Rdxx as g(X) =
(X.,).)), Gn (x(X.,n) Then,
Z E E [ Rdxn ,11- [111lp)) + G(Z) E Eg sup |i R || 11F) (1F))
(3)
Proof. This is a consequence of Proposition 2.3 since the map g is sup, || ltip Lip.-ihishhtz with respect to (w..t.) the Frobenius norm. Indeed, for X, H E Rdx=; ||g(X + H) g(x) ||
Σ | lipp
| S sup; [Gillipp |H|
3
MAIN RESULTS
3.1 GAN DATA: AN EXAMPLE OF CONCENTRATED VECTORS
Concentrated random vectors are particularly interesting from practical standpoint for real data modeling. In fact, unlike simple Gaussian vectors, the former do suffer from the constraint of having independent entries which is quite restrictive assumption when modeling real data such as images or their non-linear features (e.g., DL representations). The other modeling interest of concentrated vectors lies in their being already present in practice alternatives to real data. Indeed, adversarial neural networks (GANs) have the ability nowadays to generate random realistic data (for instance realistic images) by applying successive Lipschitz operations to standard Gaussian vectors (Goodfellow et al. 2014).
A GAN architecture involves two networks, generator model which maps random Gaussian noise to new plausible synthetic data and discriminator model which classifies real data as real (from the dataset) fake (for the generated data). The discriminator is updated directly through binary classification problem, whereas the generator is updated through the discriminator. As such, the two models are trained alternatively an adversarial manner, where the generator seeks to better deceive the discriminator and the former seeks to better identify the fake data (Goodfellow et al., 2014).
In particular, once both models are trained (when they reach Nash equilibrium), DL representations of GAN-data and GAN-data themselves- are schematically constructed in practice as follows:
Real Data GAN Data o Fi where (O,
where d stands for the input dimension of the generator model, N the number of layers, and the Fi's either Fully Connected Layers, Convolutional Layers, Pooling Layers, Up-sampling Layers and Activation Functions, Residual Layers or Batch Normalizations. All these operations happen to be Lipschitz applications. Precisely,
Ropresomation Network
Desoriirator
Concentrated Vectors
Real Fake
operaion
paddttr
Figure 1: Deep learning representations of GAN-data are constructed by applying successive Lips- chitz operations to Gaussian vectors, therefore they are concentrated vectors by design, since Gaus- sian vectors are concentrated and thanks to the Lipschitz stability Proposition 2.3
3

---

Under review as a conference paper at ICLR 2020
Fully Connected Layers and Convolutional Layers: These are affine operations which can be expressed as
Fi(x) = W, + b,, for W; the weight matrix and b, the bias vector.
Here the Lipschitz constant the operator norm (the largest singular value) of the weight matrix Wi that is [Filltip = su
|Waul2
||a|||2
Pooling Layers and Activation Functions: Most commonly used activation functions and pool- ing operations are
ReI a0),, MaxPooling(z)) = [max max(xs,))
where S,'s patches (i.e., subsets of [dim(z))). These at most 1-Lipschitz operations with respect to the Frobenius norm. Specifically, the maximum absolute sub-gradient of the ReLU acti- vation function is 1, thus the ReLU operation has Lipschitz constant of 1. Similarly, can show that the Lipschitz constant MaxPooling layers is also 1.
Residual Connections: Residual layers act the following way
F((x) = x + F o... F!" (z),
where the F: s are Fully Connected Layers or Convolutional Layers with Activation Functions, and which are Lipschitz operations. Thus F, is Lipschitz operation with Lipschitz constant
bounded by + II 11spp
Batch Normalization (BN) Layers: They consist in statistically standardizing (lffe & Szegedy 2015 the vectors of small batch B (x,)/_ C Rd follows: for each æ E B
Fi(xs)) diag
(*n 1sla) +
Vo +
where μa k= - [x oB to be learned and diag v) transforms vector e to a diagonal matrix with its diagonal entries being those of v. Thus BN is Lipschitz transformation with Lipschitz constant [| [l sup;
di Em=1
(I<x) mB)² 2 b Fd are parameters
Therefore, as illustrated in Figure I since standard Gaussian vectors are concentrated vectors as mentioned in Proposition 2.2 and since the notion of concentrated vectors is stable by Lipschitz transformations thanks to Proposition 2.3 GAN-data (and their DL representations) are concen- trated vectors by design given the construction in Equation (4) Moreover, in order to generate data belonging to specific class, Conditional GANs have been introduced (Mirza Osindero) consequence of
2014); once again data generated by these models concentrated vectors Corollary 2.4 Indeed, a generator of Conditional GAN model can be seen as a set of multiple gen- erators where each generates data of a specific class conditionally the class label (e.g., BigGAN model (Brock et al. 2018)).
Yet, in order to ensure that the resulting Lipschitz constant of the combination of the above oper- ations does not scale with the network or data size, so to maintain good concentration behaviors, careful control of the learned network parameters is needed. This control happens be already considered in practice in order to ensure the stability of GANs during the learning phase, notably to generate realistic and high-resolution images (Roth al. 2017 Brock al. 2018). The control of the Lipschitz constant of representation networks is also needed in practice in order to make them robust against adversarial examples (Szegedy et al. 2013 Guirajani et al. 2017). This control is particularly ensured through spectral normalization of the affine layers (Brock et al.) 2018), such as Fully Connected Layers, Convolutional Layers and Batch Normalization. Indeed, spectral nor- malization (Miyato et al.) 20188) consists in applying the operation W +- W/a (W) ) to the affine layers at each backward iteration of the prok-prroagttion algorithm. where σ1 (1(W) stands for the largest singular value of the weight matrix W. Brock et al. (2018), have notably observed that, without spectral constraints, a subset of the generator layers grow throughout their GAN training and explode at collapse. They thus suggested the following spectral normalization -which happens to be less restrictive than the standard spectral normalization W + W/o,0 17/a,/We (W) (Miyato et al. 2018 the affine layers:
W W
(W) o.) un (W))
(5)
4

---

Under review as a conference paper at ICLR 2020
Largest singular value 01
6
Wilhor SM
5
With
Tacortical hound
4
3
3
2
1 0
200
400
600
800
1,000
Iterations
Figure 2: Behavior of the largest singular value of weight matrix in terms of the iterations of a random walk (see proposition 3.1). without spectral normalization in (blue) and with spectral normalization in (red). The (black) lines correspond to the theoretical bound Vo² + 17 for different a. We took dg d1 100 and rj 1/dg-
where u1 (W) and Un (W) denote respectively the left and right largest singular vectors of W, and a, is an hyper- parameter fixed during training.
To get an insight about the influence of this operation and ensure that it controls the Lipschitz constant of the generator, the following proposition provides the dynamics of random walk in the space of parameters along with the spectral normalization Equation ( Indeed, since stochastic gradient descent (SGD) consists in estimating the gradient of the loss function randomly selected batches data, it can be assimilated to a random walk in the space of parameters (Antognini & Sohl-Dickstein 2018).
Proposition 3.1 (Lipschitz constant control). Let > and G be a neural network composed of N affine layers, each of input dimension di-1 and outpul dimension da for i [N], with 1-1ipschitz activation functions. Assume that the weights of G at layer i + are initialized as U([ 1 VT, va.
and consider the following dynamics with learning rate 1)
W - W E, with Eig 10) W W max(0, i (W)) - (W) (W()),
(6)
Then, VE > 0, the Lipschitz constant of g is bounded at convergence with high probability as:
N
|| < II
(E+
1=1
Proof. The proof is provided in Appendix
B
Proposition shows that the Lipschitz constant of neural network is controlled when trained with the spectral normalization in Equation ( In particular, recalling the notations in Proposition 3.]
in the limit where d, + oo with di + Y (0, oc) for all E [N] and choosing the learning rate d=-1
Od the Lipschitz constant of g is of order ((1) if it has finitely many layers N and a. is constant. Therefore, with this spectral normalization, dimensions grow. Figure 2 depicts the behavior of the Lipschitz constant of a linear layer with and without spectral normalization in the setting of Proposition| 3.1 which confirms the obtained bound.
be assumed that || ((1) when
3.2 MIXTURE OF CONCENTRATER VECTORS
In this section, we assume data to be mixture of concentrated random vectors with controlled O(1) Lipschitz constant (e.g., DI. representations of GAN-data as we discussed the previous section). Precisely, let .1,..., En be a set of mutually independent random vectors in RP. We suppose that these vectors are distributed as one of & classes of distribution laws P1..... Pk with distinct means (mef and "covariances" {Ct} defined receptively
m. Ez [m:], Ce En [a:=]]]
5

---

Under review as a conference paper at ICLR 2020
For some q > O, we consider a q-exponential concentration property on the laws μ4g, in the sense that for any family of independent vectors yi..... y/ sampled from pe: [y/...... yel E (1 RPXs, (|r). Without loss of generality, we arrange the z; in data matrix X [......, Tr,] such that, for each E [k]. E/+ n.?.... " ~ j4g, where ny stands for the number of æ; 's sampled
from pig. In particular, we have the concentration of X as
X E R -1e)).
Such data matrix X can be constructed through Lipschitz-ally transformed Gaussian vectors (q = 2), with controlled Lipschitz constant, thanks to Corollary 2.4 In particular, DL representations
of GAN-data are constructed as such, as shown in Section 3.1 We further introduce the following notations that will be used subsequently.
M mm..... m&] E R J J= I.. jkk E R"** and Z = =zz1..... zn| R
where je E R stands for the canonical vector selecting the æ of distribution jee, defined by (fe): = Lepeu and the z;'s are the centered versions of the x; 's, i.e. 2; 2; my for Iy ~ jac-
3.3 GRAM MATRICES OF CONCENTRATED VECTORS
Now we study the behavior of the Gram matrix G = = X in the large n. p. limit and under the model of the previous section. Indeed, G' appears as a central component in many classification, regression and clustering methods. Precisely, a finer description of the behavior of G' provides access to the internal functioning and performance evaluation of a wide range of machine learning methods such as Least Squares SVMs (AK et al. 2002). Semi-supervised Learning (Chapelle et al., 2009) and Spectral Clustering (Ng et 2002). Indeed, the performance evaluation of these methods has already been studied under GMM models in (Liao & Couillet 2017: Mai & Couillet 2017 Couillet & Benaach-GGeorges] (016)) through RMT. On the other hand, analyzing the spectral hehavior of
G for DL representations quantifies their quality -through its principal subspace (Montavon al.
2011 as we have discussed in the introduction. In particular, the Gram matrix decomposes as
G= G--JMTMJ + p
1 Z - (J Z + MJT)
(10)
p
Intuitively G decomposes as low-rank informative matrix containing the class canonical vectors through J and noise term represented by the other matrices and essentially ZT Given the form of this decomposition, RMT predicts -through an analysis of the spectrum of G and under a GMM model (Benaych- Georges & Couillet| [0166]- the existence of a threshold & function of the ratio p/n and the data statistics for which the dominant eigenvectors of G contain information about the classes only when |MMM] 2 & asymptotically (i.e., only when the means of the different classes are sufficiently distinct).
In order to characterize the spectral behavior (i.e., eigenvalues and leading cigenvectors) of G under the concentration assumption in Equation (9) on X, we will he interested in determining the spectral distribution I. 1 Ei=1 5x, of G, with A Am the eigenvalues of G. where & stands for the Dirac measure at point æ. Essentially, to determine the limiting eigenvalue distribution as n + co and p/n c E (0, oo). conventional approach in RMT consists in determining an estimate of the Stieltjes transform (Silverstein & Choi 1995) m of L, which is defined for some 2 E C (uupp(L
(A 1 x - n
(z) = -
tr ((G - =Im)=1)-
(1I)
Hence, quantifying the behavior of the resolvent of G defined as R(2) = + determines the limiting measure of L through (z). Furthermore, since R(z) and G share the same eigenvec- tors with associated eigenvalues x the projector matrix corresponding to the top m eigenvectors
U = [ui...., um of G can be calculated through Cauchy integral UU 2xd f, R 2222) 2dz where y is an oriented complex contour surrounding the top m eigenvalues of G.
1
To study the behavior of R(z), we look for a so-called deterministic equivalent (Hachem et al, 2007) R(2) for R(2), which is a deterministic matrix that satisfies for all A E Ra** and all u, v E Rn of respectively bounded spectral and Fucildean norms, 1 tr AR(=)) - 1 R tr())(()) + and uT (2)) R(z))u + 0 almost surely n + co. In the following, result which gives such a deterministic equivalent under the concentration assumption on X in Equation 19) and under the following assumptions.
present our main

---

Under review as a conference paper at ICLR 2020
Assumption 3.2. As p oc,
1. p/nn + c (0, 00),
2. The number of classes is bounded, 3. || O(v)).
Theorem 3.3 (Deterministic Equivalent for R(z)). Under the model described in Section 3.2 and Assumptions 3.2
3.2 we have R(2) Ep
1/2 R .11- -11-111)) Furthermore,
BR(=) R(2)) =0 =o(v
(1 log(p)
Ine (1(=)(2)
1 +
R(x) = diag
J (12)
E=1 pz
with S2 M @ diag
( [2)++ t=1 and Q(2) Ei 1+02(0)
+ zIp)
where 8²(2) (8)(2), ..... ......)((2]]] is the unique fixed point of the system of equations
)) for each E E [R:].
5e(2)
1 tr p
Ce
ck Σ
C + 0z(2)
II
Sketch of proof. The first step of the proof is to show the concentration of R(z). This comes from L/pshhipc w.r.t. the Frobenius norm, thus we have by Proposition 2.3 that R(z) E (p 1/2 R || ||r). The second step consists in estimating (2) through deterministic matrix 2(2). Indeed, R(z) can be expressed
the fact that the application X + R(z) is 2=
function of ((2) = (XXT/) /p + 2 2 as R(z) = 2 - XTQ (2)2)) X /p), and exploiting the result of (Luartt & Couillet 2019) which shows that (=) can be estimated through ((2), we obtain the estimator R2(z) for (2).. A more detailed proof is provided in Section A.3 of the Appendix. O
This result allows specifically to (i) describe the limiting eigenvalues distribution of G. (ii) determine the spectral detectability threshold mentioned above, (iii) evaluate the asymptotic "content" of the
leading cigenvectors of G' and, much more fundamentally, (iv) infer the asymptotic performances of machine learning algorithms that are based on simple functionals of G (e.g.. LS-SVM, spectral clustering etc.). Looking carefully at Theorem 3.3 we see that the spectral behavior of the Gram matrix G computed on concentrated vectors only depends on the first and second order statisties of the laws μe (their means m, and "covariances" Cy). This suggests the surprising result that G has the same behavior as when the data follow GMM model with the same means and covariances. The asymptotic spectral behavior of G is therefore universal with respect to the data distribution laws which satisfy the aforemmentooee concentration properties (for instance DL representations of GAN-data). We illustrate this universality result in the next section by considering data as CNN representations of GAN generated images.
GAN Images
Real Images
Figure 3: (Top) GAN generated images using the BigGAN model Brock al. (018). (Bottom) Real images selected from the Imagenet MatasettDeng et al. (2009). We considered n = 1500 images from classes which are Mushroom, Pizza and Hamburger.
7

---

Under review as a conference paper at ICLR 2020
resnet50 (p = 2048)
vgg16 (p = 4096)
2ensenet201 (p = 1920)
CAN magea Countion mobure
CAN image Courtan mocure
CAN imago
Dansity Boop sola)
Coussoon modure
GAN Images
Dansty
16 Figenalure (AP))
Figenollass (4
]
Epennar
5o 2048)
vgg16 (p 4096)
densenet (p 1920)
Real images
Real images
Real miniure
méture
Deraby (hy (ozzs)
micare
Real Images
Deraty Ove
Eigenvales (A
Eigeniiis (499
Eigenealus (201)
Spriidter
Figure 4: (Top) Spectrum and leading eigenspace of the Gram matrix for CNN representations of GAN generated images using the BigGAN model Brock al. (2018). (Bottom) Spectrum and leading eigenspace of the Gram matrix for CNN representations of real images selected from the Imagenet dataset Deng et al. (2009). Columns correspond to the three representation networks (Resnet50, VGG16 and Densener201).
4 APPLICATION TO CNN REPRESENTATIONS OF GAN-GENERATED IMAGES
In this section, we consider n 1500 data æi...., L. E RP as CNN representations -across popular CNN architectures of different sizes p- of GAN-generated images using the generator of the Big-GAN model (Brock et al., 2018). We further use real images from the Imagenet dataset (Deng al. 2009) for comparison. In particular, we empirically compare the spectrum of the Gram matrix of this data with the Gram matrix of a GMM model with the same means and covariances. We also consider the leading 2-dimensional eigenspace of the Gram matrix which contains clustering information as detailed in the previous section. Figure 3 depicts some images generated using the Big-GAN model (Top) and the corresponding real class images from the Imagenet dataset (Bottom). The Big-GAN model is visually able to generate highly realistic images which are by construction concentrated vectors, as discussed Section 3.1
deuretss the spectrum and leading 2D eigenspace of the Gram matrix computed on CNN rep- resentations of GAN generated and real images (in gray), and the corresponding GMM model with same first and second order statistics (in green). The Gram matrix is seen to follow the same spectral behavior for GAN-data as for the GMM model which is a natural consequence of the universality result of Theorem 3.3 with respect to the data distribution. Besides, and perhaps no longer surpris- ingly, we further observe that the spectral properties of for real data (here CNN representations of real images) conclusively matched by their Gaussian counterpart. This both theoretically and empirically confirms that the proposed random matrix framework is fully compliant with the theoretical analysis of real machine learning datasets.
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
Leveraging on random matrix theory (RMT) and the concentration of measure phenomenon, we have shown through this paper that DL representations of GAN-data behave as Gaussian mixtures for linear classifiers, fundamental universal property which is only valid in high-dimension of data. To the best of our knowledge, this result constitutes a new approach towards the theoretical understanding of complex objects such as DL representations, as well as the understanding of the behavior of more elaborate machine learning algorithms for complex data structures. addition, the article explicitly demonstrated our ability, through RMT, to anticipate the behavior of wide range of standard classifiers for data as complex as DI. representations the realistic and surprising images generated by GANs. This opens the way to more systematic analysis and improvement of machine learning algorithms on real datasets by means of large dimensional statistics.
REFERENCES
Suykens Johan AK et al. Least squares support vector machines. World Scientific, 2002.
Joseph Antognini and Jascha Sohl-Dickstein. Pea of high dimensional random walks with com- parison to neural network training. In Advances in Neural Information Processing Systems, pp- 10307-10316, 2018.
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprini 1rXiv: 171..0440, 2017.
Florent Benaych-Georges and Romain Couillet. Spectral analysis of the gram matrix of mixture models. ESAIM: Probability and Statistics, 20:217-237, 2016.
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1888 Aug 2013. ISSN 0162-8828. doi: 10. 10.1009/PAAM..213.5.0.
Yoshua Bengio et al. Learning deep architectures for ai. Foundations and trends in Machine Learning, 2(1):1-127, 2009.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprini arXiv:: 1096, 2018.
Olivier Chapelle. Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2b0o)]book reviews]. IEEE Transactions on Neural Networks, 20(3):542-542, 2009.
Romain Couillet and Florent Ceenaych-Georgee Kernel spectral clustering of large dimensional data. Electronic Journal of Statisties, 10(1):1393-1454 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition,
pp. 248-255. Icee, 2009.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. In Advances in Neural Information Processing Systems 30, pp. 5767-5777 2017.
Walid Hachem, Philippe Loubaton, Jamal Najim, al. Deterministic equivalents for certain func- tionals of large random matrices. The Annals of Applied Probability, 1763):875-930, 2007.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502.03167 2015.

---

Under review as a conference paper at ICLR 2020
Michel Ledoux. The concentration of measure phenomenon. Number 89. American Mathematical Soc., 2005.
Zhenyu Liao and Romain Couillet. Random matrices meet machine learning: large dimensional analysis of Is-svm. In ICASSP, pp. 2397-2401. IEEE, 2017.
Cosme Louart and Romain Couillet. Concentration of measure and large random matrices with an application to sample covariance matrices. submitted, 2019.
Xiaoyi Mai and Romain Couillet. A random matrix analysis and improvement of semi-supervised learning for large dimensional data. arXiv preprini urXiv: 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 14.1. 2014.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: /802.0995, 2018.
GrrSgoire Montavon, Mikio L Braun, and Klaus-Robert MÅller. Kernel analysis of deep networks.
Journal of Machine Learning Research, 12(Sep):2563-2555) 2011.
Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. 2002.
In Advances in neural information processing systems, pp.
Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In Advances in Neural Information Pro- cessing Systems 30, pp. 2018-2028. 2017.
David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, al. Learning representations by back-propagating errors. Cognitive modeling, 5(3):1, 1988.
Jack Silverstein and Sang-Il Choi. Analysis of the limiting spectral distribution of large dimen- sional random matrices. Journal of Multivariate Analysis, 54(2):295-33)) 330, 1995.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Chin-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar, Representer point selection for explaining deep neural networks. In Advances in Neural Information Processing Systems, pp. 9291-9301, 2018
A PROOF OF THEOREM 3.3
A.1 SETTING OF THE PROOF
For simplicity, we will only suppose the case and we consider the following notations that will be used subsequently.
Ex,; C = E. [[],,]]]] X. X E1 Co = =[(xxx]/v)
Let
X_, (ø),.... 0. 0. v. ,
the matrix X with a vector of zeros at its ith column.
Denote the resolvents
X
-1
XXT
-1
XXT
-1
R
+ II
Q
+ =I,
Q-
+ zI,
p
(13)
And let
Q 1 C
+ zI,) -1
(14)
c1 +
10

---

Under review as a conference paper at ICLR 2020
where 8 is the solution to the fixed point equation
8
1 tr
C
(
(
1 c
21p) )
)r
111+8
p
A.2 BASIC TOOLS
Lemma A.1 ((Ledoux 2005 ). Let z E E R", RI 1P.1--11 1)) and M E ( R2x", -l (IF.). Then, for
some numerical constant C
E || Ez| + C vp, E| I E lcc + CVlogp.
E | & JMM| + CVp + n, E M S |EM |r + Cym
Lemma A.2. Denote Qs = (xxT + 2Ip)= we have:
Qæ
and || æQ= O(1).
| 1 +
Moreover, if | 2 vp. Q 0pp-112)
Proof. Since 2Qa = Ip - Qaăx
x = | Qx
and we recover the first identity of the Lemma.
And since the matrix Co is nonnegative symmetric, we have
Co Te
c 8)
Qa
z1p)
1
| + ze(1 + 8)
Therefore, =Qx
<||+)||a||| 1+
((1) and:
2
0(1) if |||||| 1
c(1 + |>
||q0| =
S
- 2 + zc 0)
c(1 + 5) (1)) if [ 2 1.
Proposition A.3. æTE[Q]a= क Q + 0 (V
logg
P
Proof. Let us bound:
x xTQ= - @Q
-- <
E
1 -0; (0)0-.0)))))
b)]
L | a , QCQR
1+
Now let us consider a supplementary random vector # following the same low as the æ; 's and in- dependent of X. We divide the set I = [n + 1| into two sets I and 1lg same cardinality (1 S
#Ig. #11 [++1]] 1). note X (x: i E I2). X (x; i [ig) and introduce the
diagonal matrices A diag ( 1.e.e Q :3; - 5 ll D = diag (1 - x. Qu: l E l

---

Under review as a conference paper at ICLR 2020
We have the bound:
*Q Qu
ar Q-:00
1::0--0 -0]]
1
_ aa; -6) -
1
1 n+1 Q0n+1
Øn+1 Q 2(a+1)0000
E
1 DDDX Q +(++1) X A
p2
[ Qx xi
+(++1) X A ((++1))+++))))++++)))) X D1
E
pd
p
( log
E
x
(x | ||||
xQCQæ
I O
p
thanks to Lemma A.1 and Lemma A.2 (the spectral norm of A and D is just an infinity norm if we see them as random vectors of IR*). We can bound x xTQ_ QCQa the same way to
obtain the result of the proposition.
L
Proposition A.4. | [-
S Q=1 (Vlogg)
148
Proof. Considering E R" such that ||u/|| =
H Q
E| Q_1
1
x +1Qu x)
x Quy
U
Lx Q a
1+0
1=1
0 1 0j
x. 01Q 1Q-10:))
1
Vn
E
E
x -; x [x,
(where + j)
F)
1+0
8
})]
$ Vn
TQa
[
+ v
TTQ - x T Q
where the first term is treated the same way as we did the proof of Proposition A.3 and the second term is bounded thanks to Proposition A.3
A.3 MAIN BODY OF THE PROOF
Proof of Theorem 3.3 Recall the definition of the resolvents R and Q in Equation 13 The first
step of the proof is show the concentration of R. This comes from the fact that the application w.r.f. the Frobenius norm. Indeed, by the matrix identity A B = A(B A A-1) B, we have
X (XT =I, is 22
(X)) ((X + H
HT
X + (X + H + H)
And by the bounds
||AB||
< |A|| F.
X
11/2
and |X(()) S =1 we have
2
| X + H
क X
H
3//
Therefore, given X | ||
R 4 vD) is 2: 3/23 p/tp3/pp-1/-1a L/--sschttt we have by Proposition 2.3 that R E Eg 1/2 "x" ||| llF).
Rxx*
||r) and since the application X
The second step consists estimating ER(²) through deterministic matrix R. Indeed, by the identity (MT M - MT (MMT++1) -1 the resolvent R can be expressed in function
12

---

Under review as a conference paper at ICLR 2020
of Q as follows
R In XQQX
(15)
P
thus deterministic equivalent for R can therefore be obtained through a deterministic equivalent of the matrix XTQX. However, as demonstrated in Louart & Couillet (2019), the matrix Q has as deterministic equivalent the matrix Q defined in equation 14 In the following, we aim at deriving deterministic equivalent for XX TX in function of Q. Let a and be two unitary vectors in R", and let estimate
XTQX XTQX
u XTQQQQ
1 -=XXXXXXIXX
A = E uT
-E
P
p
P
1
P
With the following matrix identities (to explore the independence of the columns of X):
Q_
= Q- p
Qu 1 1 Q-a a
B-
A A
and the decomposition QXXT= En Qu we obtain:
u
QXU uT 1000-,,,,000
1 uu
| QCQX.
A
E
1+0
Q
P
1+0
P
u X Q-xCQ 1
u u 2 Qx_
E
2
1+0
1 1 -
1=1
+ u Q Q_,CQQ
v X_
CQu: u.v
x. uQQ=.QQv;
1+0
1
1
u QQ - QX-
vu X Q0 Qu:
x Q
a x Q
1 L Q
u,v.
1 a
Q 3z
p
u X Q E. QCQXv
1+6
We can show with Holder's inequality and the concentration bounds (mainly the fact that 1 Q-; concentrates around 8) developed in (Louart& & Couillet 2019), that most of the above quantities vanish asymptotically. As a toy example, we consider the following term:
u T Q CQX u Q 10 QX
E
1+8
1 x Q Q-ix;
8 Lo Q
u u 2 QX 1
2
(1 + 8)
Lo Q
1
(uTX) Q 3. (x) QX_
8
x 1
2
p
E
u
x X .Q_ NuxxX_Q-,00
37 )
E
Qx (xx))
x
)
37
E
x Q 100)
Similarly, we can show that:
u.w
CQX
u.u
E
1+0
1+
T.Q
1 uu
luTXQQ
N. QCQX+
=0 vp
1+8
p
8
13

---

Under review as a conference paper at ICLR 2020
Finally, the remaining terms in A can be estimated as follows:
u Q C. Qx
A
E
1 x, Q-1Q-
v.u X Q 0 Qu: x: _ Q
1
u
+0
1 Qr:
a Q 2.
vp
Q
8ur O
logp
1
1+0
Where the last equality is obtained through the following estimation:
v:ui
X Q
10,00
Qæ:
v.ui X
(1 SuuTX_Q-A,0 Q-1
+5) 5) -8 (11 + Qx: ))
E
+ x Q2QQi0
-00
L Q-10) (1 8)
i=1
V:dE(ul X (--00
(1+
1=1
With the following bound:
- (1 11 a Qu:
1
x Q ar
+ -5)-
p
a Qa: (1 +6) -8(1 + 8) + 8(1 + - (611 (1+-x,00 Qx:
p
m Qa: -8 (1+
p
we have again with Holder's inequality and Proposition A.4
1
v.u. X x Qz:
v,ou 1 1 Qa
+ (V
log
2 1=1
1+ x Q
1+0
Now that we estimated A, it remains to estimate E
X Indeed, given two unit norm vectors
u, U.
E R" we have:
1 T X XTXx
u.u El u.u
[uy]]
U.u Q Σ u,o,
P
P iljll
1
x Qau 11 +6 T
Qa
-
Quu
M M
du v O
p
p
P
since we have क Q 0(1) by Lemma
A.2 we introduced the matrix Mi = 11 Therefore we
have the following estimation:
8
(V
log P p
= 1+ I, +
T @QQM, + OHH
P
1 ++
where = 1=- means that its mean, then conclude:
A - B|| = (a(()). Finally, since R concentrates around
1 R
In X
In
6-1
T Q= 0111
(v
log
p
2 1
pz(ö + 1)
14

---

Under review as a conference paper at ICLR 2020
B PROOF OF PROPOSITION 3.1
Proof. Since the Lipschitz constant of a composition Lipschitz functions is bounded by the prod- of their Lipschitz constants, we consider the case N and linear activation function. In this case, the Lipschitz constant corresponds to the largest singular value of the weight matrix. We consider the following notations for the proof
W; W; E with [Ei,, _N(0,) 1)
W1+1 W, max(0, 01. o.) v
where 01, = 01 (Wi,, U1. = 2n (W;) and U1, Vi (.). The effect of spectral normalization observed in the case where d. > 1, otherwise the Lipschitz constant is bounded by o.. We therefore have
W. + & W | + d2
(16)
W W+ P = W 11 + - -0
(17)
If W 2 [Wile, we have by equation
16 and equation
W.l. < [i1l of oî. + 1/ d2do
|Wl σ1, S 0 12 =8
And since |Wi+1 - Will. we have [W
Otherwise, there exits r such that [[+++llee < |W-lrr then for all & > 0 there exists an iteration 7' > such that ||W,, || S 5 + Indeed, otherwise denote &, = W 82
and E > 0 for all t 2 T. And if for all 2 1.W[ll+ S W.llF, we have by equation 16
and equation 17
W W 2
W 8 2 W ++1 1| 8² El+1
Integrating the above expression from to T
T, we end up with
W W ||
Σ E
0
W p < W, /
Σ E
1=T
therefore, when T + co, has to tend to 0 otherwise the right hand-side of the last inequality will tend to oo which absurd.
15


Reviews:
[{"review_id": "rJlB6yg6tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a formal proof that the data produced by a GAN are concentrated vectors. The proof is rather intuitive: a GAN essentially consists of a Lipschitz function that takes as input a latent vector drawn from a Gaussian distribution and therefore one can use Lipschitz concentration bounds to prove that the transformed output satisfies a concentration property. Overall, I think this is a very intuitive idea and I fail to see the added value of such an observation. I provide detailed feedback below, I would really like to get a sensible answer regarding the novelty aspect.\n\nGaussian assumption\nIt is of course very common to sample the latent vectors from a Gaussian distribution, although one could potentially use a different distribution (uniform, gamma, …). To what extent could these results extend to other distributions?\n\nPrior work\nThe paper does not appropriately discuss prior work that impose a Lipschitz constraint while training, see e.g.:\nGulrajani, Ishaan, et al. \"Improved training of wasserstein gans.\" Advances in neural information processing systems. 2017.\nRoth, Kevin, et al. \"Stabilizing training of generative adversarial networks through regularization.\" Advances in neural information processing systems. 2017.\n\nTheorem 3.3\nI would like further clarifications regarding this theorem. This theorem is essentially a concentration bound for the resolvent of the gram matrix G around its mean. The expectation is already computed in (Louart & Couillet, 2019) so the contribution in this theorem seems to be in showing that the result only depends on the first and second order statistics. You claim this is a surprising result, although it does not seem so surprising to me given that this is an asymptotic result. Could you comment on deriving a non-asymptotic result instead?\n\nAdded value\nAs mentioned earlier, I do not see what particular insight is this paper bringing. There are some obvious connections that one could make, e.g. implications such as robustness to adversarial samples depending on the Lipschitz constant, or perhaps improve generalization property, but some of these connections are already made in prior work (see comment above). What particular insight do we get from your analysis?\n\nExperiments\nI think a more detailed study regarding the effect of various regularization schemes would be valuable. One could for instance compare various networks with/without batchnorm, resnet connections, ...\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlAnKJCKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors claim to establish a Lipschitz bound on neural networks, initialized randomly, and trained until convergence.  They also claim to establish probabilistic concentration of the resolvent of the Gram matrix from a mixture of k distributions with varying means and covariances.  The authors also present the spectrum and leading eigenspace of the Gram matrix for representations by CNNs of images generated by a GAN.\n\nIt is not clear to this reviewer what exactly the authors have proved and what significance it has.  For example, in Proposition 3.1, the authors talk about dynamics with a learning rate, where W <- W - \\eta E where E has standard normal entries and \\eta is a learning rate.  It is unclear what the learning problem is, or why the learning rate is modifying a random modification.  I understand the general idea that Lipschitz functions of random-quantities-that-enjoy-concentration-estimates also enjoy concentration estimates.  This is well known from the random matrix theory literature.  The authors need to make a claim about how this informs either the development or understanding of deep learning technologies.  The authors should consider what the implications of their results could be, and then do the research to establish that those implications hold up.\n\nThe authors claim that their results constitute \"a first step towards the theoretical understanding of complex objects such as DL representations.\"  This claim is false.  This very conference is on Learning Representations, and presumably at least one paper in the past 7 years makes progress towards the theoretical understanding of Deep Learning representations.  \n\nBecause of the overly bold claims with insufficient clarity and justification, I recommend that this paper be rejected.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylHxsbqtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors generalize Gaussian random vectors to a broader class of \"concentrated\" vectors which they use as their primary tool for analysis of the latent representations learned by GANs. They show that the spectral behavior (i.e. spectra and leading eigenspaces) of the Gram matrix computed over GAN representations is the same as those produced by a high dimensional Gaussian Mixture Models (GMMs). Furthermore, they show that the \"sufficient statistics\" (i.e. measures of information encoded in the latent representations) for GANs depend only on their first and second moments. Thus, for data that follows Gaussian mixture patterns, GANs and GMMs behave identically. The authors also show that common neural network operations (linear/convolutional layers, pooling, batch normalization, ReLU activations) are Lipschitz transformations, where the Lipschitz constants can be bounded using spectral normalization (this is borrowed from previous work). They also provide some empirical analysis to verify their theoretical findings.\n\nOverall, the paper is well organized and the theoretical results are both compelling and thorough. The experimental results also follow nicely from the theory. Admittedly, this reviewer is not well versed enough in this area of mathematics to provide thorough critical insight about the derivations and proofs. However, the notation is clear and the general arguments appear to be sound. The authors' theoretical results are significant, and provide a much needed step forward in formalizing and understanding deep generative models.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper theoretically shows that the data (embedded by representations learned by GANs) are essentially the same as a high dimensional Gaussian mixture. The result is based on a recent result from random matrix theory on the covariance matrix of data, which the authors extend to a theorem on the Gram matrix of the data. The authors also provide a small experiment comparing the spectrum and principle 2D subspace of BigGAN and Gaussian mixtures, demonstrating that their theorem applies in practice.

Two of the reviews (with confident reviewers) were quite negative about the contributions of the paper, and the reviewers unfortunately did not participate in the discussion period.

Overall, the paper seems solid, but the reviews indicate that improvements are needed in the structure and presentation of the theoretical results. Given the large number of submissions at ICLR this year, the paper in its current form does not pass the quality threshold for acceptance.
Decision: Reject