Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proves generalization bounds for GANs. I think the paper can be improved significantly in several ways:

1- Writing: The first two sections are relatively well-written. The problem starts at section 3 and continues after that. Some of the things that can be improved:

a) The discussion on the different definitions of generalizations is not really helpful in the current format. You might want to explain how these different definitions relate to each other. For example, if generalization in one of them implies generalization in the other one, etc.

b)Theorem 2.3 is a general statement but it is followed by Corollary 3.3 which is a very specific generalization bound. There is no explanation how one can show the corollary.  Even worse is mixing these two in the proof of the theorem in the appendix.   Please consider improving the use of Theorems, Lemmas and Corollaries.

c) Section 3 and 4 have bunch of theorems and collieries without much explanation. It is not clear that all of these are actually helpful for the main purpose of the paper.

d) I don't completely understand the notation in Corollary 3.3. Eg. what is d_{f,\ell}?

2) Related Work: I think authors need to do a more comprehensive literature review on generalization bounds. Since the generalization bounds presented here are built on the supervised learning bounds, authors discuss the generalization bounds in supervised learning. For example, authors heavily rely on Chen et al. (2019) for their generalization bounds while very similar results where shown before by [1] and [2].
[1] Neyshabur, Behnam, Ryota Tomioka, and Nathan Srebro. "Norm-based capacity control in neural networks." Conference on Learning Theory. 2015.
[2] Golowich, Noah, Alexander Rakhlin, and Ohad Shamir. "Size-independent sample complexity of neural networks." Conference on Learning Theory. 2018.

3) Definition of generalization: I don't think the definition of generalization suggested in this work is much different than Arora et. al. since f really doesn't depend on samples from D_g and hence the empirical and true distributions are not very different. In fact, I think the definition provided by Arora et. al. 2017 is preferred because at the end of the day, we have to estimate the distribution D_g by generating some samples.

4) Generalization bound for fixed g: Unfortunately, the novelty of these generalization bounds are very limited as they are a direct application of known generalization bounds in the supervised settings. Therefore, the authors contributions are very limited here.

5) Generalization bounds for all generators: Again, here the novelty and final result is very limited since the bounds achieved by a union bound arguments and does not really go beyond that.

6) Experiments: Experiments can also be improved significantly. Currently, the correlation is reported for 5 trained networks and it is not clear to me that this result is statistically significant. Moreover, only one hyper-parameter is changed in the experiments which could be problematic. I suggest authors to change multiple hyper-parameters and train more networks to improve the evaluation.


******************************

After author rebuttals:

I have read the authors response and looked at the revision. Unfortunately, many of my concerns are not addressed adequately so my score remains the same.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors study the generalization bound for GANs based on a new definition of generalization error where the distribution corresponding to the generator is assumed to be known for each generator (i.e., there is no empirical distribution for generators). For this generalization error, the authors give both bounds for a fixed generator and a uniform bound for a class of generators.

In my opinion, most of the theoretical results seem follow directly from standard tools in statistical learning theory and existing results on capacity bounds of neural networks. It seems that the authors do not introduce new ideas or techniques in the analysis.

The authors made comparisons with the related results in Arora et al (2017). However, since the generalization bound in Arora et al (2017) is based on a different definition of generalization error where empirical distributions are considered for both discriminators and generators, the comparison seems not fair.

----------------------
After rebuttal:

I have read the authors' response. The contribution seems not novel and enough. I would like to keep my original score.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 
The authors give new generalization bounds for GANs.  The argue for a new definition of generalization for GANs, which isolates the effect of sampling error arising from sampling from
the real distribution.  (Their argument, which I find convincing, may be paraphrased as saying that sampling from the generator should be viewed more as a computational cost than a data-gathering cost, since a procedure may sample from a generator as many times as it wants.)  They give a bound that is uniform both over discriminators and generators.

In my opinion, the mathematical writing is not up to the standard for publication in ICLR.  There are many cases where the paper is unclear, and, in many other cases, I have to guess what they mean, and I have limited confidence in my guess.  Since the theorems of the paper are quite technical, it is very difficult to be confident what their exact statements are.  One example is where they write 
"We define F with weight normalization as ...".  The fact that the c in this definition has a subscript of f led me to think that the bound can depend on f, but this does not make sense.  I assume that the bound is independent of f.   Later, when they write "we define the Lipschitz constant of f", since they write that this Lipschitz constant is with respect to a norm on parameterizations, I take it that they are defining the Lipschitz constant of a mapping from parameters to functions (and not a Lipschitz constant of f).  But they don't indicate what metric is used to define the distance between functions.  

When the compare their bound with previous work, they treat quantities as constants which can be large.  For example, they treat the product of the operator norms of the layers as a constant.  It also is not clear how they get the bound that they attribute to Bartlett, et al from the bound in that paper.  

Most of the technical heavy lifting appears to have been borrowed from the Chen, et al paper.

A more detailed account of how they get their covering bound for their (p,q) norm from the
Dumer paper is needed.  

Since the authors assume that phi is the identity, it seems unnecessary to keep subscript d with it.  On the other hand, subscripting d with calF would make the theorems easier to interpret at a glance.

(Edit on 11/14/19: I have read the response and checked the revision.  Some of the above criticisms have been addressed by the revision.  I have increased my rating.)
review_title: Official Blind Review #1

Meta-review: The authors received reviews from true experts and these experts felt the paper was not up to the standards of ICLR. 

Reviewer 3 and Reviewer 1 disagree as to whether the new notion of generalization error is appropriate. I think both cases can be defended. I think the authors should aim to sharpen their argument in this regard.Several reviewers at one point remark that the results follow from standard techniques: shouldn't this be the case? I believe the actual criticism being made is that the value of these new results do not go above and beyond existing ones. There is also the matter of what value should be attributed to technical developments on their own. On this matter, the reviewers seem to agree that the derivations lean heavily on prior work. 
Decision: Reject