Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes the distribution structure called “forward super-resolution” (forward super-resolution assumes that for image resolution at level $L$ is represented by $G_l*W_l$, where $G_l$ is hidden neuron at layer l of neural network $G$ (with RELU activation) and $W_l$ is certain matrix), which is hierarchically generated, which can be learnt effectively by GAN using stochastic gradient de- scent ascent (SGDA) under certain conditions. In particular, to learn the forward-super resolution, the authors assumes that the true distribution of layers in neural network G follows sparse structure and “not-too-positively correlated” at patch level. With this assumption, the authors provide the theoretical capability and the convergence of the generator and design the learning algorithms for it.  The algorithm breaks the learning into multiple parts (GAN OutputLayer, GAN FirstHidden, GAN FowardSuperResolution) and show how the training of each part is formed and converged. The main idea of the convergence theory showing the generator will learn to match the moments of the true distribution. 
strength_and_weaknesses: Strengths:

1. The theory about the “forward super-resolution” is new to me.

2. The few proofs in the paper looks correct, but most of them cannot be verified due to the limited time. 

3. The new learning algorithm for GAN with the theoretical supports. 

Weaknesses:

1. The paper aims to discuss assumption “forward super-resolution” holds in practice and show the proposed underlying training mechanism simulates the actual learning process of GANs on real-world problems. However, excepts some examples which was developed in some existing GANs, I cannot find anywhere in the paper. Can the author point to where this is discussed? Seems to me if most of them only with discussion rather than theoretical or empirical evidence which is not convincing to me enough. 

2. Moreover, can the authors details how to design the empirical experiments for Figs 2, 3, 4?  In Fig. 3, it is not convincing to me that the learned of network is really sparse (it may be depends on how we design sparsity) but to me it is more on the long-tail distribution since most of columns have the value. Also, the sparsity assumption may depend on the network capacity and how much knowledge of the data needs to be encoded into the networks. For example, can the authors reduce the network at different sizes to see whether it still learn the sparse structure?

3. One suggestion is designing the toy examples to show the learning capability of proposed algorithm as claimed by the authors it is natural in real-world, designing such toy examples may not difficult?
clarity,_quality,_novelty_and_reproducibility: Novel in idea and theoretical analysis. However, it is not easy to reproduces the empirical analysis as not all details are provided. The theory is overwhelming (e.g., in supplementary) but the authors done well by trying to simplify it and with some high-level intuitions in main papers. 
summary_of_the_review: Overall, the high-level idea is interesting and novel to me. Indeed, empirically looks like learning via hierarchical approach in GAN seems more effective as shown in some existing works. However, this is too dense theoretical paper up to 71 pages, challenging for me to verify the correctness of the proofs in limited time. I have a few questions regarding the empirical evidence and main assumptions that hopefully the authors can clarify in the rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a theoretical analysis of GANs and shows how GANs can learn hierarchically generated distributions. Here hierarchical generation means that the distribution comes from an unknown generator that follows a forward super-resolution structure where each successive layer gives a higher resolution image (somewhat similar to progressively growing GANs). It is further assumed the learned generator also follows a similar structure. A learning algorithm is then presented using 3 types of discriminators (for the output layers, for the first hidden layers, and for remaining hidden layers) and it is shown that the generator trained using this algorithm learns the target distribution.
strength_and_weaknesses: [Strengths]
* The paper presents an interesting analysis of a version of GANs.
* The paper is well-written (given the space limit) and the authors have made attempts to give intuitions and visuals to explain the results.

[Weaknesses]
* The analysis is done on a model/network that is not close to practice. Specifically, the discriminator is awkwardly split up into 3 parts which is not seen in practice, as the authors have themselves acknowledged. Some kind of empirical justification for the proposed learning algorithm is also missing.
* Overall, it seems like a conference paper (with a page limit) is not enough to properly explains the ideas presented in this work. The frequent references to the appendix break the reading flow. Therefore, in my subjective opinion, this work is more suitable for a longer format journal. 
clarity,_quality,_novelty_and_reproducibility: * The main text is well-written and authors have provided intuitions for various choices and results. I did not check the proofs/appendix.
* The analysis is novel to the best of my knowledge but, given the disparity from practical GAN implementations, I am not sure how significant/useful the results are for the ICLR community.
* Reproducibility: N/A, theory work.
summary_of_the_review: Overall, this paper presents an interesting theoretical analysis of GANs. However, there is a clear disparity between the model and learning algorithm studied in this paper vs. what's used in practice. Therefore, I am unsure about the significance of the results. It also appears that 9 pages are too few to properly explain the ideas presented in this work and a long-form journal submission may be a better choice. 

Note: I do not have background in learning theory, so I am unable to provide an in-depth review of the analysis and proofs.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper provides a theoretical model of *how* GANs might learn to generate images. The paper defers most of the work to the 71 page appendix, but if I understood it correctly, they model GAN training as iteratively learning "linear" (as in generalised linear model,i.e. including a single nonlinearity ala logistic regression) discriminator/generator layers on objectives at the given resolution, assuming that such a generator can generate the data (with additional sparse basis, non concentration and a couple of other assumptions).

The main theorem is a high probability statement about the trained generator being able to match the true generator (up to a transformation by a column orthonormal basis in the latent state) with high probability.
strength_and_weaknesses: Strengths of the paper:

- while the appendix is quite overwhelming in size, it *is* quite clearly written and legible. Overall, I applaud the authors for attempting to make such an involved work readable within the constraints of ICLR
- I could not verify the proofs in detail, but they seem reasonable to me as far as I could check. If it holds, the work could be the opening gate to a theory of GANs (albeit a very specific type which reminds me a lot of diffusion models in small local changes over multiple layers and relying on supervision in what will be latent stages)


Weaknesses

- the paper is very ambitious with just how much it is trying to present and I'm not sure it is the best way of doing things. A journal submission might have been a better fit, alternatively putting more emphasis on the interpretation and leaning *even more* on the already exhaustive appendix might have been a better move. However, it is difficult to thread this needle, and despite the reviewer guidance I cannot give actionable feedback on *how* to do this rewrite. 
clarity,_quality,_novelty_and_reproducibility: Novelty: if the papers analysis holds, it will be the first theoretical treatment of GANs to my knowledge
Quality: Presentation and arguments are well done as far as I could check

Reproducibility/Clarity: I don't think the tight schedule of ICLR is a good fit for reviewing this work and the clarity suffers by having to be divided between hints and motivation in the main body and the appendix, despite both of them being well done. Overall, both are acceptable in my eyes since most important assumptions as well as notation and results are "seeded" in the main body.
summary_of_the_review: I applaud the authors for what attempting to fit what appears to be a very good 80 page paper into a 13 page conference format, but I am unsure whether it was a good idea. I lean towards a marginal accept because I could not do the required level of verification to stand behind the results stronger and I find a paper which relies almost exclusively on the appendix for verification to be problematic. Again, a journal venue or conference with different constraints might have fit better. However, what I could verify seems correct and I understand the challenge of presenting a work as involved  as this. Hence, marginal accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this work, it is shown that generative adversarial networks (GANs) can efficiently learn certain hierarchically generated distributions that are close to the distribution of real-life images. It is proven that when a distribution has a structure referred to as "forward super-resolution," then training GANs using stochastic gradient descent ascent (SGDA) can learn this distribution efficiently in terms of sample and time complexities. Additionally, empirical evidence is provided to support the assumption of "forward super-resolution" and to demonstrate the underlying mechanisms that allow for the efficient training of GANs in practice. 

The reviewers liked that the paper provided theory for “forward super-resolution” and thought the new learning algorithms for GANs that come with theoretical support are interesting. The reviewers also raised various concerns: (1) more theoretical/empirical evidence is needed for forward super-resolution (2) lack of detail on some of the experimental setup (3) dependence on sparsity level on the architecture. The authors rebuttal seemed to mitigate a lot of these concerns but the reviewers all kept their original marginal accept score. Overall the paper is interesting and merits publication. I therefore recommend acceptance.



justification_for_why_not_higher_score: The paper's setup is clearly somewhat restrictive and thus does not merit a higher score.
justification_for_why_not_lower_score: All reviewers suggested acceptance.
Decision: Accept: poster