Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: In this paper, the authors claim to establish a Lipschitz bound on neural networks, initialized randomly, and trained until convergence.  They also claim to establish probabilistic concentration of the resolvent of the Gram matrix from a mixture of k distributions with varying means and covariances.  The authors also present the spectrum and leading eigenspace of the Gram matrix for representations by CNNs of images generated by a GAN.

It is not clear to this reviewer what exactly the authors have proved and what significance it has.  For example, in Proposition 3.1, the authors talk about dynamics with a learning rate, where W <- W - \eta E where E has standard normal entries and \eta is a learning rate.  It is unclear what the learning problem is, or why the learning rate is modifying a random modification.  I understand the general idea that Lipschitz functions of random-quantities-that-enjoy-concentration-estimates also enjoy concentration estimates.  This is well known from the random matrix theory literature.  The authors need to make a claim about how this informs either the development or understanding of deep learning technologies.  The authors should consider what the implications of their results could be, and then do the research to establish that those implications hold up.

The authors claim that their results constitute "a first step towards the theoretical understanding of complex objects such as DL representations."  This claim is false.  This very conference is on Learning Representations, and presumably at least one paper in the past 7 years makes progress towards the theoretical understanding of Deep Learning representations.  

Because of the overly bold claims with insufficient clarity and justification, I recommend that this paper be rejected.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper provides a formal proof that the data produced by a GAN are concentrated vectors. The proof is rather intuitive: a GAN essentially consists of a Lipschitz function that takes as input a latent vector drawn from a Gaussian distribution and therefore one can use Lipschitz concentration bounds to prove that the transformed output satisfies a concentration property. Overall, I think this is a very intuitive idea and I fail to see the added value of such an observation. I provide detailed feedback below, I would really like to get a sensible answer regarding the novelty aspect.

Gaussian assumption
It is of course very common to sample the latent vectors from a Gaussian distribution, although one could potentially use a different distribution (uniform, gamma, …). To what extent could these results extend to other distributions?

Prior work
The paper does not appropriately discuss prior work that impose a Lipschitz constraint while training, see e.g.:
Gulrajani, Ishaan, et al. "Improved training of wasserstein gans." Advances in neural information processing systems. 2017.
Roth, Kevin, et al. "Stabilizing training of generative adversarial networks through regularization." Advances in neural information processing systems. 2017.

Theorem 3.3
I would like further clarifications regarding this theorem. This theorem is essentially a concentration bound for the resolvent of the gram matrix G around its mean. The expectation is already computed in (Louart & Couillet, 2019) so the contribution in this theorem seems to be in showing that the result only depends on the first and second order statistics. You claim this is a surprising result, although it does not seem so surprising to me given that this is an asymptotic result. Could you comment on deriving a non-asymptotic result instead?

Added value
As mentioned earlier, I do not see what particular insight is this paper bringing. There are some obvious connections that one could make, e.g. implications such as robustness to adversarial samples depending on the Lipschitz constant, or perhaps improve generalization property, but some of these connections are already made in prior work (see comment above). What particular insight do we get from your analysis?

Experiments
I think a more detailed study regarding the effect of various regularization schemes would be valuable. One could for instance compare various networks with/without batchnorm, resnet connections, ...

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors generalize Gaussian random vectors to a broader class of "concentrated" vectors which they use as their primary tool for analysis of the latent representations learned by GANs. They show that the spectral behavior (i.e. spectra and leading eigenspaces) of the Gram matrix computed over GAN representations is the same as those produced by a high dimensional Gaussian Mixture Models (GMMs). Furthermore, they show that the "sufficient statistics" (i.e. measures of information encoded in the latent representations) for GANs depend only on their first and second moments. Thus, for data that follows Gaussian mixture patterns, GANs and GMMs behave identically. The authors also show that common neural network operations (linear/convolutional layers, pooling, batch normalization, ReLU activations) are Lipschitz transformations, where the Lipschitz constants can be bounded using spectral normalization (this is borrowed from previous work). They also provide some empirical analysis to verify their theoretical findings.

Overall, the paper is well organized and the theoretical results are both compelling and thorough. The experimental results also follow nicely from the theory. Admittedly, this reviewer is not well versed enough in this area of mathematics to provide thorough critical insight about the derivations and proofs. However, the notation is clear and the general arguments appear to be sound. The authors' theoretical results are significant, and provide a much needed step forward in formalizing and understanding deep generative models.
review_title: Official Blind Review #2

Meta-review: The paper theoretically shows that the data (embedded by representations learned by GANs) are essentially the same as a high dimensional Gaussian mixture. The result is based on a recent result from random matrix theory on the covariance matrix of data, which the authors extend to a theorem on the Gram matrix of the data. The authors also provide a small experiment comparing the spectrum and principle 2D subspace of BigGAN and Gaussian mixtures, demonstrating that their theorem applies in practice.

Two of the reviews (with confident reviewers) were quite negative about the contributions of the paper, and the reviewers unfortunately did not participate in the discussion period.

Overall, the paper seems solid, but the reviews indicate that improvements are needed in the structure and presentation of the theoretical results. Given the large number of submissions at ICLR this year, the paper in its current form does not pass the quality threshold for acceptance.
Decision: Reject