Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

While much work has been devoted to understanding the role of the discriminator in GAN training, comparably little is known about the role of the generator. 
In this work, the author address this important question, more exactly the role that generator overparametrization plays in GAN training.
Empirically, the authors show that when applying DCGAN and RESNET architectures to CIFAR10 and CELEBA, increasing the number of parameters in the hidden layers decreases the Frechet inception distance and improves image quality.
Theoretically, they show that RELU networks with a single hidden layer converge to the global minimum under simultaneous gradient descent with high probabiliy, provided that the dimension of the hidden layer is large enough.

Recommendation:
I think that this work makes a contribution to an important topic that has not been studied very much, so far. My main concern is that I find the presentation somewhat lacking at this point. In particular, as detailed below, I would appreciate a discussion of overparametrization vs model complexity. I would also suggest to the authors to consider beginning by presenting their experiments on real GANs to describe the empirical phenomenon that their theorem is supposed to illuminate.
Finally, I believe that some additional guidance in interpreting the results of the theorem would be helpful. In particular, the comparison of the setting of Theorems 2.1 and 2.1 should be more prominent since the difference between these two result is what really captures the relationship to GANs, as opposed to arbitrary RELU models.

Questions/Suggestions
- How do the authors define "overparametrization"? In particular, how is it different from model complexity. It is not clear to me why Figure 1 should concern "overparametrization" as opposed to model complexity? I understand that in the deep learning setting these concepts might not be easy to differentiate, in either way I believe some more background on this topic would be warranted. Similarly, a key element the lower bound on $k$ in the theorem seems to be that ensures that minimization objective has optimal value of zero, that is the "data" can be reproduced exactly.
This again seems to me to rather be about model complexity than about overparametrization of the model. Looking at the proof it seems that the concentration argument leading to a well-conditioned Jacobian does not depend on model being perfectly able to represent the "training data". This might be worthwhile to explain in more detail in the paper. 

- "The networks are optimized using Gradient Descent/Ascent (GDA) to reach a saddle-point of the min-max optimization problem."
There have been multiple recent works such as [Berard et al.](https://arxiv.org/abs/1906.04848) [Farnia and Ozdaglar et al, 2020](https://arxiv.org/abs/2002.09124), [Schafer et al.](https://arxiv.org/abs/1910.05852) that call the role of nash-equlibria/saddle points into question. I think the authors should engage, at least briefly, with this line of work.

- " One of the key factors that has contributed to the successful training of GANs is model overparameterization. By increasing the complexity of discriminator and generator networks, both in
depth and width, recent papers show that GANs can achieve photo-realistic image and video synthesis (Brock et al., 2019; Clark et al., 2019; Karras et al., 2019)." 
This seems a bit misleading since at least the Karras et al. work suggests much more specific modifications than just increasing the complexity. I don't think any of the references listed support the claims about the importance of overgenearlization as much as the way in which they are cited suggests.

- "In particular, it has been empirically observed (as we also demonstratein this paper) that when the generator/discriminator contain a large number of parameters (i.e. are sufficiently over-parameterized) GDA does indeed find (near) globally optimal solutions. In this section we wish to demystify this phenomenon from a theoretical perspective." 
This statement should be provided with some evidence. it is also not clear what a (near) globally optimal solution means. In the context of GANs, [Arjovsky and Bottou 2017](https://arxiv.org/abs/1701.04862), [Berard et al.](https://arxiv.org/abs/1906.04848), [Schafer et al.](https://arxiv.org/abs/1910.05852) show that GANs do not even converge to (nearly) locally optimal points, despite producing good images, contradicting the statement.

minor points:
- "very simpler" -> much simpler

================================================================================================

After reading the author's response and the other reviews I still lean slightly towards acceptance and have therefore left my rating unchanged.
While not being an expert on the subject, I find the work interesting. In case the paper gets rejected, I recommend to the authors to the feedback provided by the referees to clarify the narrative of the paper. 
review_title: Interesting problem and results, but unclear narrative 

final_score: rating: 7: Good paper, accept
review_content: # Summary
This paper studies the effect of model over-parametrization in GANs. While there is a lot of work on this in the supervised learning setting of classification/regression there is not much in the GAN framework where the minimax objective function complicates such an analysis. This paper considers two types of training of the GAN model, one with the simultaneous gradient descent ascent and one where the discriminator is trained to optimality for every generator update. It provides global convergence results under both algorithms in the case of a generator network with one hidden layer that is large enough and a linear discriminator.

# Contributions: 
Given the importance of over-parametrization of neural networks for better performance and low generalization error this work is a step toward a theoretical understanding of this phenomena in the GAN setting. Even though the global convergence results are provided in a simple setting, the proof techniques using dynamical systems and control theory might be useful for extending these results to a more general framework. 

The comparison between convergence rates of the two training algorithms is also interesting. 

The experimental evidence seems compelling, although it would be nice to consider examples other than simple gaussians for a more comprehensive empirical analysis. 

# Comments: 
1. It would be nice to compare the two types algorithms considered in terms of empirical evidence. For example a combination of the two plots in Figure 2 would be nice in order to get a sense of the effects of the two theorems 2.1 and 2.2 empirically. 
2. Just out of curiosity can we say anything about the training of the discriminator for a finite number of steps and how that affects convergence? Sort of as an interpolation between the two algorithms. 
3. A little more variety in the toy empirical examples considered would be nice but not crucial. While the experiments on the real data are compelling it would be nice to have empirical evaluations where we can compute the true generalization error or know the ground truth which is not simply Gaussian. 
4. It would also be nice if the authors could provide a summary of the proof of the main theorems either in the main paper or at least at the beginning of the appendix jus to provide a rough sketch of what pieces are required to show such a result.

review_title: A good first step toward a theoretical understanding of over-parametrization in GANs.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies how over-parameterization plays a role in GAN training. Theoretically, it shows that a GAN with over-parameterized 1-layer neural network generator and a linear discriminator can converge to global saddle point via stochastic optimisation. Similar results are obtained for nonlinear generators and discriminators under some conditions. It also provides empirical results to support its findings. 

## Pros

- The paper is easy to read.
- The result can be significant as it suggests we simply need larger GANs in practice.

## Cons / Questions

- I'm not sure if the term over-parameterization is well-defined in the paper. For training neural classifiers, the word "over" is clear because of the transition point of the double descending curve. But here we only see a descending curve (for performance) within the regime of parameter space the paper studies. To be more specific, is $k=256$, which is only 2x or 4x of common choices of feature numbers, considered as over-parameterized? Similar for Figure 2, is $k=1024$ for linear layers over-parameterized?
- The paper says, "One of the key factors that has contributed to the successful training of GANs is model over-parameterization". But in fact, a lot of works have been showing that a careful balance of generator and discriminator capacity is in fact important, which is also agreed by the authors in the end of Section 3. So, I'm not sure if this opening sentence is a valid claim.
- The empirical results are based on DCGAN and Resnet that use convolutions. Does similar result hold for MLPs? Testing it on MNIST may be useful.
- Can you explain how step sizes and other parameters are set while varying the capacity of GANs?
- The study on generalization gap seems to be incomplete. In particular, we also need to check the GANs simply memorizes the data while increasing the capacity, right? It's unclear if a dropped FID in both train and test (i.e. Figure 5) can rule out this issue. 
review_title: Many questions

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors proposed to analyze the over-parameterization in GANs optimized by the alternative gradient descent method. Specifically, considering a GAN with an over-parameterized G and a linear D, the authors proposed the theorem 2.1 to provide a theoretical convergence rate in GAN’s training.

However, this paper is problematic. The details are as follows:

1. The gap between the paper’s title and theoretical claims. To my best knowledge, the OVER-PARAMETERIZATION means that the model will overfit to the training data and cannot generalize to test data. Thus, the generalization bound between training error and the test error is the main concern in this topic. However, in this paper, the authors mainly focus on the convergence rate during the training. I think this topic is more correlated to the non-convex optimization problem, rather than the over-parameterization problem.

2. The theoretical claims in Sec 2 are not convincing. First, what is the data distribution? In the whole Sec 2, there is no detailed explanation about the data distribution, which is one of the most important parts of the analysis. Only the `Numerical Validation` part mentioned that the data distribution is a univariate Gaussian. Though we assume that the data is Gaussian, simply minimizing the distance between the mean of data and the ones of G is not enough: the variance in the data is not taken into consideration. I’m not sure that using a linear discriminator is enough for G to model the data distribution; generally, we assume the discriminator has infinity capacity.

3. The theoretical derivation is ambitious. In Theorem 2.1, why $V$ is not optimized in Eqn. (2) & (3)? In Theorem 2.3, why $f$ is a general mapping? In GANs, it should be a parametric mapping from z to x. Besides, Theorem 2.3 claims that it is a general minimax problem. However, it is still restricted to a linear discriminator. Clarity needs to be improved.

4. Finally, the experimental results cannot fully validate the authors’ claims. In Fig. 4, with smaller k, G’s capacity is not sufficient to capture the data distribution. In this case, comparing the convergence rate is unfair. Further, the paper does not provide any bound on the gap between training error and test error. I don’t know the purpose of Fig. 3 and Fig. 5.

Three related work:
[*1] talks about the existence of the equilibrium of GANs’ minimax problem. If the equilibrium does not exist, then the convergence rate cannot validate any claims as I mentioned above.
[*2] also uses control theory to understand GAN’s training and [*3] adopts control theory to improve the training dynamics of deep models. The relationship should be discussed.


[*1] Farnia, Farzan, and Asuman Ozdaglar. "GANs May Have No Nash Equilibria." arXiv preprint arXiv:2002.09124 (2020).

[*2] Xu, Kun, et al. "Understanding and Stabilizing GANs’ Training Dynamics using Control Theory."

[*3] An, Wangpeng, et al. "A PID controller approach for stochastic optimization of deep networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
review_title: More details should be provided

Meta-review: ### Paper summary 
This paper investigates theoretically and empirically the effect of increasing the number of parameters ("overparameterization") in GAN training. By analogy to what happens in supervised learning with neural networks, overparameterization does help to stabilize the training dynamics (and improve performance empirically). This paper provides an explicit threshold for the width of a 1-layer ReLU network generator so that gradient-ascent training with a linear discriminator yields a linear rate of convergence to the global saddle point (which corresponds to the empirical mean of the generator matching the mean of the data). The authors also provides a more general theorem that generalizes this result to deeper networks.

### Evaluation
The reviewers had several questions and concerns which were well addressed in the rebuttal and following discussion, in particular in terms of clarifying the meaning of "overparameterization". After discussing the paper, R1, R2 and R4 recommend acceptance while R3 recommends rejection. The main concern of R3 is that the GAN formulation analyzed in the paper is mainly doing moment matching between the generator  distribution (produced from a *fixed* set of latent variables z_i) and the empirical mean of the data. R3 argues that this is not sufficient to "understanding the training of GANs". At least two aspects are missing: how the distribution induced by the generator converges according to other notion of divergence (like KL, Wasserstein, etc.); and what about the true generator distribution (not just its empirical version from a fixed finite set of samples z_i)? While agreeing these are problematic, the other reviewers judged that the manuscript was useful first step in understanding the role of overparameterization in GANs and thus still recommend acceptance. And importantly, this paper is the first to study this question theoretically.

I also read the paper in more details. I have a feeling that some aspects of this work were already developed in the supervised learning literature; but the gradient descent-ascent dynamic aspect appears novel to me and the important question of the role of overparameterization here is both timely, novel and quite interesting. I side with R1, R2 and R4: this paper is an interesting first step, and thus I recommend acceptance. See below for additional comments to be taken in consideration for the camera ready version.

### Some detailed comments
- Beginning of section 2.3: please be clearer early on that you will keep V fixed to a random initialization rather than learning it. The fact that this is standard in some other papers is not a reason to not be clear about it.
- Theorem 2.2: in the closed form of the objective when $d$ is explicitly optimized, we are back to a more standard supervised learning formulation, for example (5) could look like regression. The authors should be more clear about this, and also mention in the main text that the core technical part used to prove Theorem 2.2 is from Oymak & Soltanolkotabi 2020 (which considers supervised learning). This should also a bit more clear in the introduction -- it seems to me that the main novelty of the work is to look at the gradient-descent dynamic, which is a bit different than the supervised learning setup, even though some parts are quite related (like the full maximization with respect to $d$).
- p.6 equation (8): typo -- the  $-\mu d_t$ term is redundant and should be removed as already included from $\nabla_d h(d,\theta)$.
- p.7 "numerical validations" paragraph: Please describe more clearly what is the meaning of "final MSE". Is this a global saddle point (and thus shows the limit of the generator to match the empirical mean), or is this coming from slowness of convergence of the method (e.g. after a fixed number of iterations, or according to some stopping criterion?). Please clarify.

Decision: Accept (Poster)