Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This is an exploratory study of the effects of performing representation learning based on a generative model trained on data, as opposed to the data itself. It is very well written and coherently laid out, and was a pleasure to read. There are no theoretical advances, but the paper provides a useful first foray into an interesting topic, based on well executed empirical studies. 
main_review: This is an exploratory study of the effects of performing representation learning based on a generative model trained on data, as opposed to the data itself. It is very well written and coherently laid out, and was a pleasure to read. There are no theoretical advances, but the paper provides a useful first foray into an interesting topic, based on well executed empirical studies. 

Strengths: the paper provides a well reasoned empirical intro into a fairly novel area. The experiments appear to be well executed and thought through. The results are not earth shattering, but solid work like this should be published and discussed. The concept of using latent space perturbations rather than pixel level perturbations is an interesting (in a good way!) one. One area of further research might be into how one chooses these perturbations in a disciplined way, and how these choices influence the mapping from one latent space to another (the learned latent space). In extremis, would this result in latent space 'cloning'?

Weaknesses: The elephant in the room is that the results depend on the quality of the IGM (implicit generative model) itself - by quality i mean how well has the IGM learnt the data-generating distribution. This of course is a function of both the model's expressivity as well as the complexity of the data-generating distribution (and how well / evenly the training data covers the distribution). The paper makes the claim that the IGM can be considered a compact representation of the data itself - were this true, the performance of models trained on the IGM would be near to indistinguishable from the performance of models trained directly on the data, which isn't the case.
I was less surprised than the authors that Gaussian perturbations in the latent space performed as well as more reasoned perturbations using steering methods - I think because I assume that how we think about data may not be as useful as we think (!); indeed much of modern machine learning bears this out.
summary_of_the_review: No theoretical advances, but some clever thinking and solid results. Happy to recommend acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work investigates **generative models as a data source for self-supervised representation learning**. In particular, the authors propose to form contrastive pairs in the latent space of the generative model and combine it with standard contrastive learning where pairs are formed in image space. 
While performance is **slightly inferior** to training with the full dataset, the proposed approach **only requires storing the model weights** of the generator instead of the full dataset.
main_review: ### Strengths
- The paper is well-written and easy to follow. The proposed method is described and illustrated clearly and appears technically sound.
- Overall, the experiments and ablations systematically support the effectiveness of the proposed method. Both, unconditional and class-conditional representation learning is considered.
- Using generative models as a data source is an interesting idea that is gaining attention recently as it might be beneficial to compress excessively large datasets, address privacy concerns, and enable more diverse data augmentation for training.

### Weaknesses
- A highly similar idea to the proposed latent-space transformations for contrastive learning was already proposed by Yang et al. [1] as “Noise perturbation”. This work is not referred to in the paper. While this work focuses on more stable GAN training and not representation learning, it takes away from the novelty of the proposed method.

> [1] Ceyuan Yang, Yujun Shen, Yinghao Xu, Bolei Zhou. Data-Efficient Instance Generation from Instance Discrimination. arXiv preprint arXiv:2106.04566, 2021.

### Additional questions / comments:
- As the performance of the proposed method probably largely depends on the quality of the generated images, an ablation considering the truncation value would be interesting. I.e., what is more important for representation learning: image diversity or image fidelity? Also, please report how the truncation parameter was determined for the existing experiments.
- What are the details for evaluating the BigBiGAN encoder? In the paper, you mention that it is trained on images of size 256x256 while you use resolution 128x128. Does this mean you resize the images before showing them to the BigBiGAN encoder? Further, please add the result of this experiment to the corresponding table (Table 4) rather than in the text.
- For the ablation in Figure 6: Why does it make sense to keep the number of iterations constant here? Would you not be rather interested in the best possible performance, i.e. training each model for the best validation error with the same number of sufficient epochs for convergence?
- What is your recommendation for tuning the standard deviation? Did you perform a grid search for each dataset, similar to the ablation on ImageNet1000 in Figure 5?  
- Why does steering perform better than Gaussian perturbation in the class-conditional setting (Table 3)?
- ContraD (https://arxiv.org/pdf/2103.09742.pdf) combines contrastive learning with GAN training and finds that training a joint discriminator can be beneficial for both tasks, image synthesis, and representation learning. This should be added and put into context in your related work.
- Another interesting related work where generative models are used as a data source, to generate multiview images and allow for efficient data labeling: https://nv-tlabs.github.io/GANverse3D/. Should be added to related work.

### Misc:
- Table 3: bold number for ImageNet1000 missing
summary_of_the_review: The considered task, i.e. generative models as a data source for representation learning, is **very interesting for the community** and has recently received increasing attention. The experiments in the paper appear sound and provide **interesting insights**. My major concern is about **the contribution regarding the latent space transformations**, as a very similar strategy was already proposed by Yang et al. Nonetheless, the findings of this work are interesting, so currently I am leaning towards accepting the paper.

**Post-rebuttal**:
The authors successfully addressed my main concern about the paper by Yang et al. so I update my rating from 6 to 8.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper investigates if synthetic datasets obtained from implicit generative models can be used for representation learning in place of the original dataset.
To do so, they compare the performance of multiple popular contrastive learning methods trained on real data against the same methods trained on synthetic data generated by GANs.
Notably, for the generated data, in addition to standard pixel transformation, they also evaluate latent space augmentations methods for contrastive learning.
main_review: Strengths:
- The problem of learning with only generated data is very relevant, and the authors achieve reasonably good results.
- The methods are very clearly exposed, the results are comprehensive and the experimental protocol is very detailed.
- It should be appreciated that while the paper is about benchmarking, it makes an effort to play to generative models' strength by also using latent space augmentations for contrastive learning. This approach is novel and the empirical results support its importance.

Weaknesses:
- It might be a little bit misleading that much of the paper is about implicit generative models but only GANs are tested. Especially since while the general idea is universally applicable, the specifics, and in particular latent augmentation, are not necessarily trivial to transferred to other generative models.

summary_of_the_review: The paper is a step forward towards training only using synthetic data, a long-standing goal that has been of interest since the introduction of deep generative models. 
Moreover, it introduces and evaluates a novel way of combining generative data and contrastive learning that is shown to be very promising.
With the rapid advances in representation learning, this submission providing a strong and comprehensive empirical study is likely to be valuable to the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper explores applying a black-box generative model (instead of a fixed training dataset) to learn unsupervised visual representations, mainly using contrastive learning. Standard contrastive learning algorithms generate multiple views of a datapoint using transformations in pixel space. The authors explore using transformations in latent space, either in isolation or in combination with pixel-level transformation methods.

Results are reported for two generative models.
* BigBiGan trained on ImageNet and finetuned on ImageNet100 and ImageNet1000
* StyleGan2 trained on LSUN Car and finetuned on ImageNet

main_review: **Strengths**

* The paper is well motivated. The authors tackle an important and relevant problem, how to use samples from a generative model in a meaningful way for downstream tasks.
* To the best of my knowledge, this is the first paper that attempts to utilize black-box generative models for unsupervised representation learning.
* The entire paper is easy to read and self-contained. The literature review is also very solid.

**Weaknesses**

The major weaknesses of the papers are the empirical results. Here are my concerns:

* It seems that the SimCLR baseline is very weak. The lowest reported number in Chen et al 2020 is 62.8% ImageNet top 1 accuracy using a batch-size of 256 and a number of epochs equal to 100 while the paper reports 43.9%. What are the differences in this paper as compared to the SimCLR baseline? Even though achieving state-of-the-art performance is not the goal of this paper, the authors should strive for a reasonable baseline.
* The authors could try to provide more insight on their negative result which is that the contrastive algorithm trained on generated views underperforms the SimCLR baseline by 1.3%. Is it that the generated samples have poor quality as compared to the ground truth dataset, or do they lack in diversity (for eg, the typical mode collapse issue with GANS)?
* **Steered Latent Views**: It seems that $w_{steer}$ in $z + w_{steer}$ is optimized to match pixel-level transformations targets $T_z(z)$. Then why not use the pixel-level transformations directly? Can the authors clarify the setting where it is preferred to use $T_z(z + \alpha w)$ instead of $T(G(z), \alpha)$?
* The baseline for the real algorithm in Table 2 is missing.
* The ImageNet100 benchmark is not apples-to-apples since both the “real” and “generated” algorithms have access to samples with a varying number of classes. Could the authors clarify the motivation behind using ImageNet100?
* The authors should also plot the "real baseline" bound in Figure 6, to see if the real baseline can be approached simply by increasing the number of samples.

Typos:
* Legends in Figure 6 have the same color.
* Eq 6) should also have T_z?

summary_of_the_review: The direction that the authors pursue in this paper is challenging and novel. However, the results are mainly negative.

For negative results, to be useful to the community, imho either the results should be suprising or should offer some insight. I have given a reject rating mainly because of the empirical results.

Please see the weaknesses section above. If atleast the first two bullets points can be addressed convinvingly by the authors, I can reconsider my rating.

**Update after Rebuttal**:

I thank the authors for their additional experiments and discussion. I've upgraded my rating from 3 (Reject) to 6 (Lean Accept).
The rationale for my rating change:

* The authors confirmed that the SimCLR baseline is lower than the reported results, due to the low resolution (128x128) of the generated model and reduced training time.
* The models (baseline and generative) are trained for 20 epochs using cyclic learning rates so I think the results might be transferrable to full training.
* The authors have shown more promsing results on pretrain LSUN + transfer on Stanford Car classification.
* ImageNet100 results as far as I see are provided only in Section 4.1.2 and 4.3 as a diagnostic tool, so the comparisons are fair.

The authors have taken a first-step towards usage of generative models for unsupervised representation learning. Even though, the results are somewhat underwhelming on ImageNet itself, this work can serve as a promising and strong baseline for future work in this topic.

**For the final version**
If the paper gets accepted, I encourage the authors:
* To report results for full training (100 Epochs), at least with the "Real baseline"  and their best setting for generated data
* To open-source their code.
* To explicitly state that the latent transform is applied only once, as compated to pixel-level transformations, which are applied twice. This might be a trivial detail but can be extremely important to reproduce the authors results.
review_title: null

Meta-review: All of the reviewers appreciate the clarity of exposition and the importance of the problem studied. That said, I agree with Reviewer P9Ys that the results are somewhat underwhelming. The baselines appear weak and are likely not well tuned on the Stanford car dataset. Key question that remains unanswered in my opinion is whether this is the most effective approach to using synthetic data to improve classification accuracy (e.g., in contrast to [Ravuri & Vinyals, 2019](https://arxiv.org/abs/1905.10887) and follow-up work). Nevertheless, I believe the community will benefit from this paper's contributions and this line of work.
Decision: Accept (Poster)