Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces an RL-based exploration strategy to the generative models that have been trained on very large datasets. The proposed approach aims to build a world model by learning a transition function in the latent space of the generative models. It is then used to train an agent interactively in this "environment", which aims at exploration along with unsupervised representation learning. The paper evaluates the learned representations in a number of downstream tasks such as classification, out-of-distribution detection, and representation learning in RL, and presents promising results.
strengths_and_weaknesses: Strengths
1. The paper is well-motivated and well-written. 
2. The idea of leveraging a large generative model as an environment for diverse data sampling is very interesting. 
3. Evaluations are performed on multiple tasks in various domains. 
4. Although it is proposed for unsupervised and task-agnostic representation learning, it could also be augmented with task-based rewards to develop a task-related sampling procedure. 

Weaknesses
1. This feels like a chicken and egg problem in some cases. As it is presented in the Atari experiment, the generative model requires a large number of samples (50M) in the first place. The proposed model requires only 6M samples from it. It seems highly inefficient. What would be the performance if the Atari policy was pre-trained on all the 50M samples and also on a subset with 6M samples, and then fine-tuned?
2. The paper does not present an evaluation of the diversity of the sampled synthetic data points.
3. I think another problem is with the following statement in line #112: "Different from them, our work focuses on leveraging a generative model as an interactive environment and eliminating the need for data augmentation." In the results in Fig. 3, it is clear that the proposed model is not a replacement for augmentation. Maybe the contribution claims regarding the augmentation should be toned down.

questions: 1. For a given representation $h_i$ in the exploration phase, the representations in the temporal vicinity of $h_i$ (i.e., $h_{i-1}$, $h_{i+1}$, etc.) should be the ones in the nearest neighbor set according to the Siamese network’s objective. However, the exploration rewards the discrepancy between them. How do the Siamese network objective and the unsupervised exploration objective work coherently?
 
2. The proposed interaction mechanism is not fully clear to me. What is the purpose of using a random latent sample? Despite the exponential moving average, the updated latent $z_{t+1}$ may produce a significantly different observation $s_{t+1}$ than $s_t$, particularly in the unconditional setting. The predicted action is conditioned on only the state $s_t$ and the policy is unaware of the random latent sample. Could the authors clarify why the proposed approach is used? Couldn’t the policy be conditioned on the random sample?

limitations: Yes.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a method that exploits both the large diverse datasets and exploratory behavior. Authors used the generator that was pre-trained on the static datasets, while introducing the dynamic model in the latent space. Transition dynamics are modeled via mixing an action and a latent variable. Further, temporal persistence is exploited via moving average. Experiments on several down-stream tasks demonstrate the effectiveness of the learned representation.

strengths_and_weaknesses: (+) Authors proposed interesting high-level concepts that try to mix dynamic exploration and the latent variables. The idea itself looks intuitive and intuitive.
(-) The actual implementation of mixing dynamic exploration and the latent variable is quite heuristic. It is the simple weight summation of the two variables. Also, the temporal persistence module is similar. The overall accuracy also varies a lot via the heuristic parameters alpha and beta as shown in Fig. 4. 
(-) The experiments are conducted on several down-stream tasks; while I cannot see what are the benefits when performing the experiments on such domains. I encourage authors to conduct the experiments in more useful and practical situations. 
(-) I cannot understand what trajectories and time axes mean in image classification tasks. It seems that no temporal information exists in such an application; while authors still use the temporal persistency module. Furthermore the accuracy gain is not appealing.

questions: In Fig. 3, I suggest using a finer-grained y axis grid.
I suggest authors conduct experiments in more useful and practical situations.
limitations: None
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper casts the latent space of a generative model (StyleGAN) as an _environment_ for unsupervised RL. The agent moves around in latent space seeking out high novelty states, and simultaneously learns a contrastive representation from temporally adjacent observations (generated images). The paper shows that this procedure can serve as effective pretraining for a vision system.
strengths_and_weaknesses: This is such a cool idea! I think the main strength of the paper is that the _idea_ of active representation learning _in generative latent space_ is highly compelling and pretty original. However, the strength of this idea is not convincingly validated by the experiments and results. I will expand on each point below:

**Originality**
The idea is, in my opinion, highly original. It combines several disparate lines of work -- on representation learning, curiosity, and latent variable generative models -- in a new and interesting way. The idea of treating latent space as an environment is really neat. The nearest works I know are those that train latent world models (but these typically do not use off-the-shelf image generators like StyleGAN) and video generative models that learn temporal dynamics in latent space (e.g., https://arxiv.org/abs/2006.10704; but I have not seen these  used for active representation learning). Given the large body of prior work on latent world models for RL environments (e.g., https://arxiv.org/abs/1912.01603), the Atari experiments are arguably not that novel. More interesting to me is the use of a pretrained StyleGAN as an environment. The present work is also quite related to older work in active learning, where the goal was to optimally sample a dataset such that training on those samples minimizes some form of regret. A good summary of that field is given in [Burr Settles. Active learning literature survey. 2009]. I feel this literature should be mentioned in the related work section, but I don't see it as detracting from the novelty of the current paper.

**Quality**
The quality of the paper can be improved in several ways. I will focus on the experiments, as those are the critical failure in my opinion. I think there are three main issues with the experiments:
1. *Section 5 does not yield insight* I didn't take much from Section 5, which compares PALM to other pretraining methods. The comparisons feel like apples to oranges. What do we learn by seeing that PALM (+ data aug) performs about as well as, e.g., SimCLR on CIFAR? These methods differ in innumerable ways: PALM is trained on synthetic data, SimCLR on real data; PALM uses the SimSiam architecture / loss, SimCLR uses a different architecture / loss; etc. The same can be said about pretty much all the comparisons in this section. I have no idea which difference between the methods was critical for explaining why one or the other does better (and anyway, most do about the same). I would suggest moving this entire section to the appendix, or even omitting it from the paper. I understand that the intent might be to show that PALM is competitive with "SOTA" methods, but if that's the goal then this section is still unconvincing. Comparisons on CIFAR and Atari are simply not the setting where one can make compelling claims about SOTA results, because the field, for the most part, is not competing on these benchmarks anymore. I do think these are perfectly fine datasets on which to do _science_ (carefully controlled experiments that yield understanding), but that's not what Section 5 provides.
2. *Missing baselines / ablations* To me, the really interesting experiments are in Section 6. I think this section should be expanded and strengthened. The core novelty of PALM is the active sampling algorithm, so we need to be convinced that it really makes sense and helps. I think the critical experiment would be to compare PALM's sampling method to other ways of sampling from StyleGAN latent space. For example: the samples in Fig 2 look like random draws from the GAN (conditioned on a fixed class). It may be they are not but that suggests a missing baseline: what if you just randomly sample positives from G(z,c=c') (i.e. fix the class, take two random draws from z). Indeed this baseline has been previously published in ref [26] (appendix table 3) and performs quite well there. How does it perform in PALM's experimental setting? I think Fig 4 gets close to testing this -- it's the setting where alpha and beta both equal 0 -- but only a few selected values for alpha and beta are shown. I would like to see all the possible settings (could plot a heatmap of performance with alpha and beta as the axes). [26] also proposes a variety of other ways to sample from GAN latent space, which do not involve active learning but perform well (e.g., positive pairs are given by z and z+w, where w is Gaussian). How does PALM compare to these? The experiments on static PALM are also close to addressing this but they train on real data rather than synthetic data, so again, it's apples to oranges and unclear if the active learning is what's helping. Passive PALM was hard for me to interpret since the data is still initially collected in an active manner.
3. *Questionable baselines* The performance of static PALM is suspicious to me. If I understand correctly, passive PALM is essentially SupCon but with the SimSiam architecture / loss. Why does it perform so poorly on CIFAR? The SupCon paper reports 96% accuracy on CIFAR10, whereas static PALM gets <55%. That gap is too large for me to believe that static PALM is a compelling baseline. Yes, the SupCon method differs in many ways but it nonetheless demonstrates that static sampling on real data can be very effective. I would like to better understand why the PALM numbers are so low in Fig 5. My concern is that the methods have not been tuned well or trained sufficiently to make meaningful conclusions.


**Clarity**
The paper is mostly clear but 1) the latent dynamics model came across as ad hoc; can it be better motivated? 2) What is a tanh Gaussian (line 206)?

**Significance**
I think the significance is high if the method really works. I'm not yet convinced it does, but I do believe in the framework. As more powerful generative models become available (dall-e, etc), I think the direction explored here might turn out to be increasingly important.


questions: My questions and suggestions are given above, so here I will just list the two things I would most appreciate in a rebuttal:
1. Material that can convince me that the active sampling component of PALM is more effective than passive ways of sampling training data from StyleGAN. 
2. An explanation for the low performance of static PALM.
limitations: The limitations are adequately discussed. I appreciated the discussion about computational efficiency.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose to combine RL and GAN to perform unsupervised pretraining. The RL agent's objective is curiosity-based. The authors perform large-scale experiments and analysis to showcase the proposed approach's effectiveness.
strengths_and_weaknesses: I want to preface my review by saying that I haven't published any paper in unsupervised pretraining, so I'm not at all an expert in this domain.

Strength:
* I find the idea very interesting, and has a lot of potential. Currently, a simple generator is used. But It's easy to imagine that down the road, more advanced generator can be used to cover the long-tail of data. For example, using text-conditional diffusion models like Dalle / Imagen can drastically help an algorithm to cover the long-tail, which is quite exciting. Though this is not in the scope of the current paper.
* The paper is well written. I can easily follow the paper as a non-expert in unsupervised pretraining.
* The experiment section's analysis seems very solid.

Weakness:
* I think what I'm least convinced about is if the effectiveness comes from the RL part or the GAN part. It's easily to imagine that with the 50M images collected from StyleGAN during RL training, we can use these images as a static dataset and evaluate traditional approaches. With the amount of images generated, it's reasonable to expect that traditional approach might benefit from them. Note that this is different from the "Passive PALM" included in the paper.
questions: I don't have any specific technical question.
limitations: The authors adequately discuss the work's limitation.
review_title: null

Meta-review: metareview: This paper formulates a framework that uses the latent space of pretrained generative models, particularly StyleGAN, as the environment for a dynamics model (unsupervised RL). The RL agent explores the latent space decoding actions and random latents to images through the pretrained generator in an unsupervised fashion, learning representations evaluate on downstream tasks.

The reviewers agreed that the paper has clear motivation and is well written, and that the idea has clear novelty and originality.
The only issues raised are to do with a few more informative baselines and comparisons, and on evaluating the diversity of samples in the trajectories, which the authors address with additional experiments and discussion in the updated manuscript.

The paper clearly has merit and explores an original idea with a good set of experiments (post rebuttal) that cover a range of settings, and should be accepted.
Having said that, I would strongly urge the authors to take comments on editing Sec5 into consideration, and also ensure that the manuscript is still within the intended page limits with the edits.

confidence: Certain
Decision: Accept