Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The main idea of the paper is how to use the pre-trained generator network of GAN to improve the downstream task of classification. Specifically, the paper use with VICReg-based method to train the network S (called student) that takes the real inputs and applies different augmentations before putting them as inputs of S, the features output of these augmentations is regularized to be similar (invariant to transformation) while the variance-covariance are maximum (this task is called “span” module). Then, the method applies the pre-trained generator to synthesize images and these images are input into the S to produce the features which are regularized from match with those features from concatenation of feature maps (go through module called “squeeze” module before that) that generated the synthetic image. The losses for synthetic images are similar to on real images except the distilled loss between generator’s features and output of the student' feature. The experiments on low resolution datasets (CIFAR-10, CIFAR-100 and STL10) showing the improvements over two existing works SiamSiam and VICReg trained on the real, fake or mixed data. The paper conducted the ablation study to understand the contributions of different components.  

However, the paper is written by the other way around: starting with “squeeze” and improve with “span” module that I think want to make the idea more interesting (but probably makes reader a bit confusing on what is actual contribution)?  With this order, the paper investigates different ways to exact features from GAN: from discriminator, latent variable and generator. The 2D visualisation and experiments suggest the features from generator are most discriminative. Inspired from the generator’s features, the paper introduces two modules “squeeze” and “span” to extract the generator features for distillation.  
strengths_and_weaknesses: 
**Strength**

I think the main contribution this paper regarding the "squeeze" module and how the generator being used to improve the existing self-supervised models. It is novel enough to me and this brings some improvements (2%) from the simple way of mixing real/fake dataset. The paper is well-written and easy to read. 

**Weakness**

From my perspective, “span” module is considered as one contribution is questionable as it is just similar to VICReg method. Therefore, I am not sure some claims of the span module to address the mode collapse and the gap between real and fake is convincing enough, except if the authors can bring some theoretical / empirical results to show that. Otherwise, it is not highly motivated why we need the generator for distillation instead of the classifier? 

At the moment, I put as borderline accept but the rating can be increased and decreased depends also comments of other reviewers which may point out something I am missing and also the rebuttal. 
questions: See above
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to distill representations from GANs for downstream tasks. Specifically, it emphasizes the importance of generator features and proposes a squeeze and span solution to effectively extract such generator features as image representations. To do so, it learns a feature prediction network that predicts generator features transformed under some kind of semantic-preserving transformations. So that generator features of real images can be directly computed. Moreover, to reduce the gap between real and fake images and avoid the generator representation of a real image suffer from GANs' issues such as mode collapse, the proposed method further applies a spanning operation that trains the feature prediction network also on real images using VICReg. 
strengths_and_weaknesses: Strengths:
+ The task of distilling GAN features for downstream perception tasks is definitely a worthexploring direction

+ The manuscript is easy to follow

+ The visualization included in Figure 2 well motivates the use of generator features over latent variables and discriminator features.

Weaknesses:
- one missing related work:

[a] Generative Hierarchical Features from Synthesizing Images, CVPR 2021.

which also utilizes generator features for various downstream tasks.

- Experiments should be significantly extended. Specifically, authors only conduct experiments on low-resolution images of CIFAR10, CIFAR100 and SHVN. As pointed out by the authors as the third point of the limitations, the proposed method should be validated on more challenging datasets with higher resolution images. I believe this is a must. Since practical value is an important aspect for self-supervised learning methods, CIFAR10, CIFAR100 and SHVN are insufficient to reflect the richness and informativeness of generator features.

- Authors only used StyleGAN2-ADA as the target generator to distill from, which is not sufficient. Studying the generator features of different GAN architectures is important for studying the effectiveness of generator features since different architectures naturally have different inductive bias.

- Authors only used distilled generator features on the test set of the same dataset the gan is trained on. However, one property of self-supervised methods is that their representations are generally useful across datasets. Authors should include such a study.

- Authors only used distilled generator features on image classification. Other downstream tasks are needed to study the pros&cons of generator features as a self-supervised representation. In fact, maybe authors should follow the standard protocol for evaluating self-supervised methods.  

- Just concatenate all layer features as the representation provided by the generator is insufficient to obtain solid observations. Authors should include studies that use different features or combinations from the single generator. 
questions: Overall I think current experiments are significantly insufficient to comprehensively study the effectiveness and different properties of GAN generator features as self-supervised representations. My detailed concerns are covered in weaknesses. I believe studying generator features as self-supervised representations across generator architectures, feature depths, downstream tasks, as well as test datasets will lead to many more insightful observations. 
limitations: Authors discussed the limitations in the main manuscript, which look reasonable to me.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a method to distill representations from a pretrained GAN generator for the downstream classification tasks. The paper further proposes a "squeeze" technique to make the representation compact, and a "span" technique to mitigate the synthetic to real domain gap. Experimental results on small datasets including CIFAR10, CIFAR100, and STL10 show its efficiency in self-supervised representation learning.
strengths_and_weaknesses: __Strengths__

- This paper is well-written and easy to follow.
- The idea of distilling representations from GAN Generators is well-motivated.
- The proposed squeeze and span techniques mitigate the mentioned problems effectively.

__Weaknesses__

- The proposed squeeze and span techniques are heavily inspired by VICReg, which downweights the novelty of the paper.
- From a technical perspective, distillation from "latent variable" and "generator feature" are not that very different. This can be verified by the numbers in Table 1, the vanilla distillation and squeeze accuracies are very similar. I also wonder what are accuracies of the "squeeze and span" for "latent variable" and how would those compare to the proposed full model.
- It is hard to assess the effectiveness of the proposed method without experiments on ImageNet for such an empirical paper. I understand that the limited computational resources might be a concern, but any form of ImageNet like low resolution of 64x64 or TinyImageNet (which has 200 classes) would be helpful.

__Post Rebuttal__

I thank authors for their response. Some of my concerns have been addressed. But my concern on missing results on large-scale datasets still remain. Therefore, I keep my initial rating but lean towards acceptance for this paper.
questions: See "Weaknesses".
limitations: Yes.
review_title: null

Meta-review: metareview: While there were initial concerns about the tasks used for evaluation, the authors did extensive extra experiments and satisfied many of these concerns.
confidence: Certain
Decision: Accept