Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers the problem of predicting the test performance of neural networks, given only the training set and model hyperparameters. While there have been many methods proposed for this, the paper considers the simple baseline of generating synthetic test samples from a GAN and computing error using these samples. This baseline appears to outperform all or almost all existing methods on a variety of tasks, including a recent competition on predicting generalization.
main_review: Strengths: The paper is well-written, the results are important and convincing, and the related work is covered well.

Weaknesses: The results seem almost too good to be true. The GAN baseline not only outperforms other methods but blows them out of the water (see Table 1). Is there any explanation for how it was possible to improve so much over the previous competition winners? Some brief discussion of this might be helpful.

I want to make it clear that this is a minor weakness--I don't think the papers strong results should be held against it. I just wish there was a bit more discussion on this point.
summary_of_the_review: This paper seems like a clear accept to me, unless there is some serious issue with the experiments that I missed. (I am not an expert in this field but they seemed quite convincing to me and basically speak for themselves.)
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work studies generalization prediction with synthetic data produced by GANs. Specifically the training accuracy gap between synthetic and given data is proposed as a measure for generalization. The authors show strong improvement over other PGDL and DEMOGEN results. A further empirical finding is made on the greater similarity between test and synthetic samples than test and train samples, as measured by FID. A variety of GAN architectures are explored.

main_review: ##########################################################################

Pros: 
 
* Predicting generalization is an timely and relevant problem. 

* Empirical findings are strong.

* Appropriate empirical methodology is used (Jiang et al. 2020)

##########################################################################

Cons: 

* The authors claim to "generalize the notion of FID" by applying it at different layers. It seems unnecessary to claim a generalization here, as its just a small modification to FID.

* No theoretical justifications are made for the observations. This is often the case for new empirical findings, but the paper would be stronger with some theory. I am left wondering if there are possible confounding factors which are leading to the observations.
 

##########################################################################

Questions during rebuttal period: 

Any insight into why exactly GANs produce data more similar to the test set?


##########################################################################


Post-rebuttal:

I found the authors response to my question about theoretical justification interesting and persuasive and decided to raise my score.
summary_of_the_review: Interesting and relevant empirical results on predicting generalization with a simple technique based on synthetic data generated by GANs.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use synthesized data generated by GANs to estimate the generalization of image classification models. The authors argue that the GAN samples are closer to the test set than the training set.
main_review: ### Strengths:
1. The proposed outperforms existing methods on both PGDL and DEMOGEN benchmarks.
2. As far as I know, the proposed method is novel. Also, it is quite interesting and counterintuitive if it works.

### Weaknesses:
1. The most critical weakness of this paper is that it doesn't specify how many synthesized samples are used to compute the synthetic accuracy. It is not clear to the readers how to compute the synthetic accuracy and how many samples are enough. An analysis of the number of samples versus the performance metrics (e.g. Conditional Mutual Information metric) is also missing.
2. The authors do not provide any statistical analysis of the proposed method. For example, each experiment is only conducted once instead of providing the mean and standard deviation of multiple runs. It is not clear how the performance would be affected by the randomness in sampling the synthesized data or training GANs.
3. The reproducibility of the paper is questionable. The authors do not provide any hyperparameters and training details.
4. The proposed method only works for the image classification task. Moreover, the experiments are only conducted on balanced datasets. It is not clear whether it generalizes to unbalanced datasets such as iNaturalist.
5. The GAN models the authors use in the paper are descendants of SN-GAN which all use hinge loss, similar model architecture, conditional BatchNorm, and projection discriminator. It would be better to see a more diverse collection of GAN models being studied.


### Typos:
1. The year of the reference `Dziugaite & Roy` is missing.
2. On the first page, the citation format is inconsistent with the rest of the paper: `(Jiang et al. (2019))` should be `(Jiang et al., 2019)`.
summary_of_the_review: Overall, I think the proposed method is quite interesting but lacks well-conducted experiments and evidence to support it. Based on the current state of the paper, I give a weak reject; however, I am willing to increase the score if the authors can address the weaknesses and update their draft during the rebuttal period.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the task of using GAN’s samples as a substitute for the true test distribution for estimating the generalization. Experiments show that samples from GAN are a powerful substitute for the test set and the predictive power of the synthetic accuracy outperforms all prior methods include the winner of the 2020 PGDL competition, despite all the empirical and theoretical evidence that GANs are not good at approximating the true data distribution. The paper then performs a preliminary investigation on why samples from GANs are a good substitute for the test set and found that using a certain notion of perceptual similarity, samples from GAN’s are much closer to the test distribution than the training distribution despite being trained on the training data.
main_review: This paper proposes a simple method for estimating the generalization gap via training a GAN. The central idea of the paper is straightforward and the presentation of the idea is very clear. As far as the method is concerned, I do not have any concerns. The experimental results are quite compelling. In fact, I believe that the performance of the method might be good enough to be used in some production scenarios where training data is expensive and the main interest is to do model selection or estimate the population risk. The additional analysis on representations learned by GANs is also interesting since it offers another perspective on the common belief that GANs are bad at generating from the full data distribution (although FID does have many problems). A broader implication of the paper is that it suggests there is a subtle difference between generative modeling of the data and discriminative learning such as classification for high dimensional data.

The weaknesses are the following:
- the paper feels hastily put together. While the central idea and results are clear, the entire paper reads very unpolished. For example, some paragraphs do not have line breaks between them (e.g., the first paragraph of section 4) and it is unclear why there is a line break in the abstract. In related works, a lot of the bold points only have one or two sentences following them. There are also some extremely short paragraphs throughout the paper such as 2.1. I believe that many of these can be merged. In equation one, the introduction of uniform convergence could use a bit more formality. Citation styles are also not consistent. For example,  “(Dziugaite & Roy)” does not have a year in the citation. The two tables also do not have consistent styles.
- Maybe I missed this but I cannot find the number of synthetic data generated. Is it the same as the size of the true dataset? Along the same line, how does the estimated change as the function of different samples from the GAN? Specifically, I would be interested to see the variance coming from using different synthetic data from the GAN.
- What is special about GANs? There are many families of deep generative models. Is there a specific reason why GAN was chosen over the other models? If so, it would be nice to see some comparisons. If the other generative models do equally well, it would also be interesting.

If the authors can address my concerns, I will increase my score.

---------------------------

**Update**

I am happy with the authors' response and the changes made to the paper. As promised, I am increasing my score to 8.
summary_of_the_review: The paper offers a simple method for estimating the performance of deep models without test data using GANs. The empirical results are strong and the analysis is interesting. On the other hand, the paper is unpolished and the presentation could be improved.
review_title: null

Meta-review: The paper demonstrates that test error of image classification models can be accurately estimated using samples generated by a GAN. Surprisingly, this relatively simple proposed method outperforms existing approaches including ones from recent competitions. All reviewers agree this is a very interesting finding, even though theoretical analysis is lacking. Given the importance of the problem of predicting generalization, I recommend acceptance.
Decision: Accept (Spotlight)