Palm up: Playing in the Latent Manifold for Unsupervised Pretraining
Authors: Hao Liu
Tom Zahavy
Volodymyr Mnih
Satinder Singh
Venue: NeurIPS 2022 Accept (2022)

Abstract: Large and diverse datasets have been the cornerstones of many impressive advancements in artificial intelligence. Intelligent creatures, however, learn by interacting with the environment, which changes the input sensory signals and the state of the environment. In this work, we aim to bring the best of both worlds and propose an algorithm that exhibits an  exploratory behavior whilst it utilizes large diverse datasets. Our key idea is to leverage deep generative models that are pretrained on static datasets and introduce a dynamic model in the latent space. The transition dynamics simply mixes an action and a random sampled latent. It then applies an exponential moving average for temporal persistency, the resulting latent is decoded to image using pretrained generator. We then employ an unsupervised reinforcement learning algorithm to explore in this environment and perform unsupervised representation learning on the collected data. We further leverage the temporal information of this data to pair data points as a natural supervision for representation learning. Our experiments suggest that the learned representations can be successfully transferred to downstream tasks in both vision and reinforcement learning domains. 

Palm up: Playing in the Latent Manifold for Unsupervised Pretraining
Hao Liu" UC Berkeley
Tom Zahavy Deep.ind
Volodymyr Mnih DeepMind
Satinder Singh DeepMind
Abstract
Large and diverse datasets have been the cornerstones of many impressive advance- ments in artificial intelligence. Intelligent creatures, however, learn by interacting with the environment, which changes the input sensory signals and the state of the environment. In this work, aim to bring the best of both worlds and propose an algorithm that exhibits an exploratory behavior whilst it utilizes large diverse datasets. Our key idea is leverage deep generative models that are pretrained on static datasets and introduce dynamic model in the latent space. The transition dynamics simply mixes action and random sampled latent. It then applies an exponential moving average for temporal persistency, the resulting latent is decoded to image using pretrained generator. We then employ an unsupervised reinforcement learning algorithm to explore in this environment and perform un- supervised representation learning on the collected data. We further leverage the temporal information of this data to pair data points as natural supervision for rep- resentation learning. Our experiments suggest that the learned representations can be successfully transferred to downstream tasks in both vision and reinforcement learning domains.
Introduction
Large and diverse datasets have been the cornerstones of many impressive successes at the frontier of artificial intelligence, such as protein folding [64 [84] image recognition [3)] and understanding natural language 12]. Training machine learning models on diverse datasets that cover breadth of human-wrin written text and natural images often dramatically improves performance and enables These models learned from fixed set of images, videos, or languages, guided by supervision that comes from ground-truth labels [3] [5]]. self-supervised contrastive learning [10, or masked token prediction
impressive generalization capabilities [16 6 55
among other approaches.
In contrast, intelligent creatures learn to perform interactions that actively change the input sensory signals and the state of the environment towards desired configuration. For example, in psychology, has been shown that interaction with the env environment vital for developing flexible and generalize intelligence [23] 74 [77. During the first few months of interactions, infants develop meaningful understandings about objects [67] and prefer to look at exemplars from novel class (e.g., dogs) after observing exemplars from a different class (e.g., cats) (54) We hypothesis that by situating the agent in high semantic complexity environment, the agent can develop interesting cognition abilities.
Given the effectiveness of learning from large and diverse datasets and the significance of interactive learning behaviors in intelligent creatures, it is therefore extremely important to connect them to build better learning algorithms.
To bridge the gap, perhaps one straightforward solution is learning the diverse real world 1 however, teaching a robot to interact in the physical world is both time consuming and resource
"This work was partially done at DeepMind. Correspondence to bao. liucs.. bew.eley.ede
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

intensive, and it is also difficult to scale up. An alternative is learning in simulated environments which have got surged interest recently. There simulators, such as Habitat [6]] 70), RLBench [27]. House3D |77,, and A12THOR [6) enable the agent to interact with its environment. However, the visual complexity of these simulated environments is far from matching the intricate real world. The key limitation is that making hand hand-designed simulation that is close enough to what camera in the real-world would capture is both challenging and tedious.
To remedy the issue, we propose a conceptually simple yet effective method that leverages existing diverse datasets, builds an environment with high semantic complexity from them, and then performs interactive learning in this environment. We do so by leveraging deep generative models that are trained in static datasets and introduce transition dynamics in the latent space of the generative model. Specifically, at each time step, the transition dynamics simply mix action and a random sampled latent. It then applies an exponential moving average for temporal persistency, imitating the prevalent temporal persistency in the real world. Finally, the resulting latent is decoded to an image using trained generator. The generator a conditional generative model which is conditioned on a prompt e.g. a class label sampled at the beginning of an episode for achieving further temporal persistency. For the generative model, we use conditional StyleGAN [32] in this work, chosen for its simplicity, although our method is not restricted to it, and can be also applied to other generative models such as language conditioned model Dalle [56].
We employ unsupervised reinforcement learning (RL) to explore this environment motivated by imitating how intelligent creatures acquire perception and action skills by curiosity 23 35
60]. Specifically, we use the nonparametric entropy maximization method named APT [44] which encourages the agent to actively explore the environment to seek novel and unseen observations. Similar to other bixel-based unsupervised RL methods, APT learns an abstract representation by using off-the-shelf data augmentation and contrastive learning techniques from vision [37 40 39].
While effective, designing these techniques requires domain knowledge. We show that by simply leveraging the temporal nature, representation can be effectively learned. We do so by maximizing the similarity between representations of current and next observations based on siamese network [5] without needing to use domain knowledge or data augmentation. Our method is named as playing in the latent manifold for unsupervised pretraining (PALM).
We conduct experiments in CIFAR classification and -itto--diitribttion detection by transferring our unsupervised exploratory pretrained representations in StyleGAN- environments. Our experiments show that the learned representations achieve competitive results with state-of-the- art methods in image recognition and out-of-distribution detection despite being only trained in synthesized data without data augmentation. We also train StyleGAN observation data collected from Atari and apply our method to it. We found that the learned representation helped in maximizing many Atari game rewards. Our major contributions are summarized below:
We present surprisingly simple yet effective approach leverage generative models as an interactive environment for unsupervised RL. By doing so, we connect vision datasets with RI., and enable learning representation by actively interacting with the environment.
We demonstrate that exploration techniques used in unsupervised RL incentivize RL agent to learn representations from synthetic environment without data augmentations.
We show that PALM matches SOTA self-supervised representation learning methods on CIFAR and out-o--diitribution benchmarks.
We show that PALM outperforms strong model-free and model-based training from scratch RL. It also achieves competitive scores as SOTA exploratory pre-training RL and offline-data pretraining RL methods.
2 Related work
Exploratory pretraining in RL Having an unsupervised pretraining stage before finetuning on the target task has been explored in reinforcement learning to improve downstream task performance. One common approach has been to allow the agent a period of fully-unsupervised interaction with the environment, during which the agent is trained to maximize surrogate exploration-bast based task such as the diversity of the states it encounters 44 objectives to generate intrinsic rewards encouraging agents to visit new states, such as the loss of an
78 Others have proposed to use self-supervised

---

inverse dynamics model [52] SGI 63 combines forward predictive representation learning [62 with inverse dynamics model [5 and demonstrate the power of representation pretraining for down- stream RL tasks. Massive-scale unsupervised pretraining has shown strong results [8] Laskin et al.
41 conducted a comparison different unsupervised pretraining reinforcement learning algorithms. Finally, Chaplot et al. (9. Weihs et al. 75 studied training RI. agent in game simulators and transfer- ring its representation to various vision tasks. Their environments are equipped with carefully chosen domain-specific reward function to guide the learning of RL agent, and the architectures of their RL agents are fairly complicated.
Our work differs in that we do not rely on hand-crafted simulators and renderers which require a huge amount of domain knowledge and effort build, instead we leverage generative models as renders. Unlike many prior work in unsupervised pretraining RL, our work does not focus on improving transfer performance to downstream RL tasks although it can be used for this purpose.
Training with synthetic data Using deep generative models as a source of synthetic data for representation learning has been studied prior work [59 26 33 These generative models are fit to real image datasets and produce realistic-looking images as samples. Baradad et al. (2) studied using data sampled from random initialized generative models to train contrastive representations. Gowal et al. [20] studied combining data sampled from pretrained generative models with real data for adversarial training and demonstrated improved results robustness. The use synthesized data has been explored in reinforcement learning under the heading of domain randomization (72), where 3D synthetic data is rendered under a variety of lighting conditions to transfer to real environments where the lighting may be unknown. Our approach does away with the hand crafted simulation engine entirely by making the training data diverse through unsupervised exploration.
Different from them, our work focus on leveraging a generative model as an interactive environment and learn representation without using data augmentation.
Temporal persistent representation Using temporal persistent information for representation learning has been proposed the past with similar motivations as ours. It has been used in learning
representation from videos [3 [6
48 19 [5] by minimizing different metrics of representation
difference over a temporal segment. Learning persistent representation has been explored reinforce-
ment learning, and has been demonstrated to improve data efficiency (50 65 62 79] and improve downstream task performance 68
In relation to these prior efforts, our work studies visual representation learning from interacted experiences based on real-world data.
3 Preliminary
Unsupervised reinforcement learning Reinforcement learning considers the problem of finding an optimal policy for an agent that interacts with an uncertain environment and collects reward per action [69] The agent maximizes its cumulative reward by interacting with its environment.
Formally, this problem can be viewed as a Markov decision process (MDP) defined by (S, A, T.Pu, 7.7) where S C R", dimensional actions, T S x A x S - [0. 1] is the state transition probability distribution. Po S - [0, 1] is the distribution over initial states, r : x A + IR the reward function, and 7 E [0. 1) is the discount factor. At environment states E S, the agent take actions a E A, in the (unknown) environment dynamics defined by the transition probability T(s'|s. a), and the reward function yields reward immediately following the action a, performed in state st. In value- based reinforcement learning, the agent learns an estimate of the expected discounted return, a.k.a, state-action value function (* a:) Ex441.0-11-- E 7 d. A new policy can be derived from value function by acting c-greedily with respect to the action values (discrete) or by
set of dimdeemniio states, A S R= set of n
using policy gradient to maximize the value function (continuous).
In unsupervised reinforcement learning, the reward function defined as some form of intrinsic reward that is agnostic to standard task-specific reward function r == Tintinsie. The intrinsic reward function usually constructed for a better exploration and is computed using states and actions collected by the agent.
3

---

Generative adversarial model Generative Adversarial Networks (GANs) 18 consider the prob-
lem of generating photo realistic images. The StyleGAN 30 32 BT architecture one of the state-of-the-art in high-resolution image generation for multitude of different natural image cate- gories such as faces, buildings, and animals.
To generate high-quality and high-resolution images, StyleGAN makes use of a specialized generator architecture which consists of mapping network and synthesis network. The mapping network converts latent vector E 2 with Z E In" into an intermediate latent space w E W with W E IR". The mapping network is implemented using multilayer perceptron that typically consists of 8 layers. The resulting vector w in that intermediate latent space is then transformed using learned affine transformations and used as an input synthesis network.
The synthesis network consists of multiple blocks that each takes three inputs. First, they take a feature map that contains the current content information of the image that is to be generated, Second, each block takes a transformed representation the vector w as an input to its style parts, followed by a normalization of the feature map.
Method
Transition Dynamics
Agent
Pretrained Conditional Generative Model
(A) Agent environment
interaction
Label/ Prompt
(B) Transition dynamics
(C) Agent architecture
Reinforcermett Learning
Similarity
H 24 ( + (1 - -))=' t>
t=
Predictor
Stop-et
Policy Value
Projector
Projector
Representation h
Representation h
= caa + (1 a))=
Stop-gred grad-gradient
Encoder
Encoder
Latent 2-1
Action at
Random z
N(N) = Σ log(ll h.
Observation
Observation
Figure I: Overview of the proposed method. (A). A conditional generative model is pretrained in static dataset which is conditioned prompt (e.g. language or class label) sampled at the beginning of an episode, and transition dynamics defined in the latent space of the generator, the agent maximizes the nonparameteric entropy of experience in learned representation space. (B). The transition dynamics consists mixing a randomly sampled latent with action (of the same dimension) followed by exponential moving average for temporal persistency, the resulting latent is decoded to image using pretrained generator. (C). The representation learned without using data augmentation by maximizing the representation similarity hetween consecutive synthesized observations hased on Siamese network. The agent is updated using unsupervised reinforcement learning with representation detached.
Our objective is to leverage pretrained deep generative models G Z latents, C denote prompt or labels and S denote observations to build an interactive environment and train an unsupervised reinforcement learning agent in such environment for representation pretraining.
where denote
4.1 Latent environment dynamics
The transition dynamics are designed in the latent space of StyleGAN. At the beginning of an episode, a class label or prompt is randomly sampled, at each time step, and a latent 2, that depends on the action and previous latent 2, T(a, = are transformed by the synthesis network of StyleGAN into an image which serves as an observation &; G(2,, c). We note that while the environment conditions on the label, the ground-truth information is not directly used by PALM.

---

The transition dynamics model consists of two steps. In the first step, it combines a randomly sampled latent with the agent's 's action obtain new latent z' Toum az) az). Although more advanced combination methods can be used, we use simple weighted summation to combine latent and action Teom a2) oa, + (1 a)z, where a is an action generated by the policy and a E [0, 1] is hyperparameter controls the relative importance of action and latent.
The second step is transformation (=') which designed to ensure temporal persistency the environment. We use a simple exponential moving average (EMA) to implement Tema. Specifically, this module generate new latent by following
{ z'. =
t= 0 B (1- B)z',
t> 0
where the combined latent and action, ž, is the exponential moving average at timestep t, and 3 E [0, 1] is hyperparameter that controls how fast the environment changes in respond to agent. We used 3 = 0.95 in most of our experiments per our initial experiments.
The high 3 is motivated by observations in unsupervised RL and linguistics. It has been shown that a highly random environment hinders the learning of unsupervised RL agents which is also known as the "Noisy-TV" problem [7]. It also has an interesting connection to the uniform information density hypothesis in linguisties which states stat-subject the constraints of the grammar humans prefer sentences that distribute information equally across the linguistic signal, i.e., evenly distributed surprise 42
In addition to the latent 2t, we also condition the generator G(z, c) on a random sample class label sampled at the beginning of episode, resulting in further persistent consistency. Finally, the environment dynamics can be written as T == (TTmm (T (z, a)),
4.2 Temporal persistent representation
Our goal is to pretrain a representation S H which is a mapping that maps observations s to lower dimensional representations h, using the data (experience replay) collected by the agent. Data augmentations are key ingredients of unsupervised representation learning methods [see e.g. [1] However, designing augmentations requires domain knowledge.
Our representation learning is instead based on temporal information. Our key idea is that since each trajectory is generated by interacting with deep generative model conditions on the same label and temporal persistent latent, it provides natural augmentations of visual content under various changing factors, such as occlusion and illumination. Some rollout trajectories during training are presented in Figure 2 In addition, the exploratory learning behavior of the agent increases the diversity of the trajectory.
Specifically, the idea to train the encoder f(s) to produce embeddings that are persistent time in the same trajectory, i.e., two presentations fa2 f(()) and h,+1 f($++) should be similar. To do so, we use the Siamese network (5] and its modern training techniques from SimSiam [T]. Specially, we maximize the similarity between q( (x)) and stop_gra aradi ent (s((()))) where and are multilayer perceptrons. More details can be found in appendix.
4.3 Unsupervised exploration
The objective of unsupervised exploration is maximize the diversity of the data seen by the agent, imitating how intelligent creatures develop recognition.
An observation s; is first encoded using the encoder to get representation h4 f(.). The policy takes representation ht as input and generates action a, of the same dimension. The policy distribution is tanh Gaussian following prior work in unsupervised reinforcement learning [78] 41).
We resort to maximizing the nonparametric entropy of the representation space which has been widely used in unsupervised reinforcement learning [66 visit diverse states, it effectively encourages the agents to explore environments, and thus learn more diverse behaviors. The estimator computes the distance between each particle h, fs() and its k-th
78 By explicitly encouraging the agents to
5

---

nearest neighbor h
H(h) - Σ p(h) log
log l|,
11ll2
where || 112 & norm. We associate each transition from experience replay {(s, 8')} with an intrinsic reward given by
Tintrinsic (s, a, s') log ||
hg 112.
Putting everything together, the diagram our method is shown in Figure
Trajectories
Time
Figure 2: Multiple rollouts collected by interacting with environment in PALM. Each row represents randomly sampled segment of trajectory. From top to bottom, each trajectory conditioned on different class label
5 Results
It is important to stress that this work focuses on studying how combined exploratory pretraining and static datasets contribute representation learning in artificial agents and on developing new, state-of-the-art, methodology for representation learning. Nevertheless, to better situate our results in the context of existing work, we provide strong baselines in our experiments.
100
70
70
Accuracy
PAL M data aug
PALM data aug
Supervised
Supervised
Random
SimCLL
Sandom
SimSlam
SimCL8
PALM
CLIP
PALM
CUP
CIFAR-10
CIFAR-100
Figure 3: Linear classification results from different methods in CIFAR datasets. Model ResNet-18 except for CLIP [55] which is based on ResNet-50. Supervised ResNet 22 is trained from scratch. SimCLR [TO] pretrained paired image-text datascts. We reported scores of each baseline for comparison. All results of method are the average of three runs with different seeds. PALM (our method) actively collect synthesized data from deep generative models based environment. The representation is learned without using data augmentation from explored experience but still achieves competitive results. Our PALM, which is based on SimSiam (. is compatible with data augmentation. adding the same data augmentation used in SimSiam, PALM data aug matches outperforms state-of of-the--tteee art static representation learning algorithms.
and SimSiam [1]] are pretrained on CIFAR dataset. CLIP 55
Image classification We test PALM on image classification tasks using ResNet- 18 as encoder f. The RL agent interacts with the environment 50M steps, and trains representation using procedure described in the method section. The hyperparameters of the representation learning follow the default hyperparameters of SimSiam and more details can be found in the supplemental material.

---

We compare our method with a series of strong methods in static datasets, including state-of-tha of-the--rrtt contrastive representation learning methods SimCLR [T0] and SimSiam, supervised trained model, and CLIP 55
which is pretrained on a large-scale paired image-text datasets.
Figure shows the linear classification performance of each method, PALM is surprisingly effective given the lack of data augmentation and real data. Note that the baselines do not have an option not use data augmentation.
Additionally, adding the same data augmentation techniques used in SimSiam to our method further improves the results, matching outperforming current SOTA static representation learning algo- rithms. Compared with the SOTA method SimSiam, PALM + data augmentation actively explore an environment instead of using static dataset, and the representation learning and data augmentation are applied to synthesized temporal pair instead of two views of the same image.
Table AUROC (%) of classifiers trained on CIFAR-I0. Models are ResNet-18. The reported results of PALM are averaged over five trials. To have a rigorous comparison, the results of baselines are from their papers. + denotes in-distribution datasct A and out-of-distributtin datasct B. Subscripts denote standard deviation, and bold denote the best results.
CIFARIO >
Train method
SVHN
LSUN
CIFAR100
Interp.
CelebA
Standard 124
88. 63.6..0.
90.7
85.8=03
75,4=07
64.1+0.5
SupCLR 34
97.3
92.8
88.6102
75.7=0.1
73.240.3
CSI
7
96.5+.02
96.3
90.5
78.5=02
75.
PALM (ours)
97.9+002
96.1+003
91.5=102
79.9=022
76.8++0.4
While some prior work demonstrated that self-supervised learning approaches significantly improve OOD detection performance (25] [5], their self-supervised pre-train
Out-of-distribution detection
heavily relies on domain-specific data augmentations, we want to study how PALM performs on OOD benchmarks.
We compare PALM with standard supervised model 22 24 and contrastive representation learning based methods including SupCLR (34) and CSI 71 CSI demonstrates state-of-the-art results by combining contrastive representation learning and supervised learning. Table shows the comparison multiple datasets. PALM outperforms CSI in 4 out of 5 out-of-theddistributioo datasets, despite using only synthesized data and no data augmentation. We attribute the effectiveness of PALM on OOD detection to active exploration stimulates the model learn the invariance of data. In doing so, the representation is optimized for distinguishing different observations.
Data efficient representation in RL In this experiment, we are interested in testing PALM pre-trained model for downstream reinforcement learning tasks. To do so, we focus our experimentation on the Atari 100k benchmark introduced by Kaiser et al. [29]. in which agents are allowed only 100k steps of interaction with their environment. There has been extensive usage of Atari for representation learning and exploratory pretraining 8 44 21
Table 2: HNS on Atari100k for PALM and baselines. >H denotes super-human performance and >0 denotes greater than random performance. Mdn and mean denote the median and mean scores. Bold denote the hest results in each category.
Finetuming after exploratory pretrain- ing is highly effective [e.g.. [8) pro-
viding strong baselines to compare to.
Method
Mdn
Mn
>
>0
Data
Unlike experiments in CIFAR datasets, here we train an uncon- ditional StyleGAN model in dataset collected by running a double DQN 7 73 agent. The dataset consists of 50M frames for each game. We use 6M steps for exploratory pretraining in the synthesized environment and we utilize DrQ (37] for online finetuning.
No Pretraining (Finetuning Only)
SimPLe
0.144
0.443
26
DrQ
0.268
0.357
2
24
Exploratory Pretraining + Finetuning
APT
0.475
0.666
26
250M
ATC
0.237
0.462
26
3M
SGI
0.456
0.838
26
6M
Offlinedata Pretraining Finetuning
Table 2 shows the comparison of PALM with basclines. Comparing
ATC
0.219
0.587
26
3M
SGI
0.753
1.598
9
26
6M
Synthesized Exploratory Pretraining + Finetuning
PALM (ours)
0.298
0.411
2
26
50M + 6M

---

with exploratory pretraining baselines (APT 44, ATC 168, and SGI (63]) that are pretrained in Atari games directly, PALM achieves surprisingly good results considering that it is trained in a simple synthesized environment. PALM achieves higher medium and mean score than DrQ in Atari 100k despite the online finetuning of PALM the same as DrQ, demonstrating the benefit of pretrained representation. Compared with offline-data pretraining baselines (ATC and SGI), there is still gap between PALM and the state-of-the-arts. which is probably due to PALM's environment being trained on observations only while ATC and SGI are trained on both observations and actions in model-based style. Worth mentioning that while PALM utilizes 50M offline action-free experience for pretraining, the amount is less than the 250M online action labeled experience used in APT. We would like to emphasize that DrQ is based on data augmentation, but PALM can outperform DrQ without data augmentation given the same amount of online experience.
6 Analysis
Persistent environment and active interaction are important. Next, we perform an ablation study, study the importance of persistent environment. We aim to answer the following question. How important it is have action (controlled by a) and state (controlled by B) in the transition dynamics? When =0, there is effectively no state, i.e., its a bandit. As shown Figure 4 when decreasing the im- portance of states which is denoted as PALM (a 0.5 8 = 0.5). the performance decreases. The extreme case 3 = 0 shows significant drop in accuracy. These results indicate that is important to have temporal persistent transition dynamics. When = O, action ignored in transition dynamics, this ablative baseline performs significantly worse than its default When replacing the agent's s action with random noise which denoted as PALM random action, we also see significant performance drop. These results show having active interaction is important to the results of PALM. We present the results of more combinations of a and 3 in Figure the results show that as decreases from 0.95 to 0, because of the decreasing importance of action, the accuracy continues to drop. a = 0.95 in general performs better than a = 1.0, indicates mixing small amount of random noise with action helps exploration and improves representation learning. When a or 33 is zero, the results are significantly worse. We also see that B E [0.5, 0.75] performs significantly better than other values, showing that persistence of dynamies is crucial.
PALM (a 0.5 B=
PALM (ar 0.5 8 0.5)
PALM (a = 0.5 8= 95, random action)
PALM 0 8-
95)
PALM (a = 0.5 $ 0.95)
10
20
30
CIFAR-100 Accuracy
Figure 4: Ablative analysis PALM on CIFAR- -00 dataset. All results are the average of three runs with different seeds.
0.5.
Exploratory Learning as Curriculum Learning. As we have seen, representations learned from ex- ploratory learning in a synthesized environment can be surprisingly powerful. We are interested in having a perhaps preliminary study of the reasons.
0.0
18.8
18.1
19
18.9
19.2
19.5
19.4
13.2
55
18.3
18.4
22
23.1
23.3
22.3
21.3
16.1
50
45
32.4
alpha
0.6 05
37.4
38.9
40
35
26.8
32.9
42.4
45
49.8
37.9
28.6
25.4
To do so, we compare PALM with its passive version
(passive PALM), static version (static PALM), and
27.5
48.8
52.3
53.9
45.8
34.6
29
25
methods proposed in Jahanian et al. (26. In the random variant, the positives pairs are sampled from G(z,c= c²) (i.e. fix the class, take two random draws from z). We denote this baseline as Random Jahanian et al. In Gaussian Jahanian et al, the positive pairs are given by z and z+w, where w is Gaussian.
56.5
35.9
-00
1.0
50.1
51.3
40.8
29.2
15
0.5 beta
Figure 5: Ablative analysis of PALM on CIFAR- 100 dataset. All results the average of three runs with different seeds.
Statie PALM is not trained in the interactive environ- ment, instead, it is trained with real images. Specifi- cally, we randomly sample two images of the same label from a dataset and use them temporal pair to train the representation learning PALM. We note that static PALM related to SupCon 34 but it differs in that static PALM does not any data augmentation while SupCon leverages
8

---

combination of data augmentations. SupCon also uses large number of negative examples from different categories.
Passive PALM is trained offline on the experience of PALM. We first store all the online experience (50M in total) of PALM, then we use the saved experience train passive PALM without having interact with the environment. We consider two options of passive PALM. The first one is random passive PALM which randomly samples consecutive observations from the stored offline experience. The second one is ordered passive PALM which is trained on the same order of observations as in online experience.
With the number of data increasing, PALM keeps improving with the amount of training data and significantly outperforms static PALM even though static PALM is trained real datasets. We believe the reason is that the diverse synthesized data collected by PALM can greatly help to learn representation. In the low data regime, carefully designed non-active learning based methods [26] outperforms PALM, possibly due to RI. training being sample inefficient. With the increasing number of synthesized samples, PALM continues improve and significantly outperform all baselines. The results also show that PALM scales better than non active learning based methods. PALM nearly always outperforms random passive PALM, demonstrating the effectiveness of exploratory learning. Interestingly but not surprisingly, ordered passive PALM performs nearly the same as PALM, indicating the exploratory learning behavior of PALM might play an important role in curriculum learning. The agent is optimized to search for novel samples in terms of their representation. by doing so, the searched samples should become more difficult distinguish. We believe comprehensive understanding of the effectiveness of PALM is out of the scope of this work, we tend to leave it as future work.
80
Linear Accuracy
70
55
PALM
50
65
passive PALM (random)
55
passive PALM fordered)
Gaussian Jahánian et al
45
Random Jahanian et al
Static PALM
25
15
5
5e4
1M
5M 10M 25M 50M
5e4
1M
SM 10M 25M50M
Sample size on CIFAR-10
Sample size on CIFAR-100
Figure 6: Results of comparing static and passive variants of the proposed method on CIFAR dataset. Sample size denotes the number of images for static PALM and denotes the number of interactions for others. All results the average of three with different seeds.
Diversity analysis. In order to measure the diversity of the sampled synthetic data, we randomly sample 10K images from the CIFAR-I0 train set and generate 10K images from StyleGAN by random sample latents. We also gen- erate 10K images with PALM latent space. For each sample in each set, we find its closest neighbor in Incep- tion feature space (obtained after the pooling layer). The coverage is defined as the proportion of nearest neighbors that are unique in the train set. A better active exploration method would produce samples that are equally likely to be close to any image in the train set. PALM achieves better coverage than random sampling in latent space. Although the Euclidean distance in Inception feature space is inaccurate, the better coverage achieved by PALM indicates that by exploring the latent manifold it gets more diverse data.
Table 3: Analysis of diversity of generated data points on CIFAR-100.
Coverage (%) (higher is better)
Method
Latent space uniform sample
50.16
PALM
58.87
7 Conclusion and discussion
Our work presents a method to connect learning from large, diverse static datasets and learning by interaction, and demonstrates promising results of doing so. We believe it opens up an interesting future direction for advancing both reinforcement learning and sele-super visual representation learning.

---

For limitations, since it is computational expensive to draw samples from large generative model, interacting with such environment is significantly slower than well optimized physical simulations used in RI., e.g., Mujoco. Addressing such limitation would make this work more accessible and facilitate future research.
For future work, it would be interesting to design or learn better transition dynamics. It would be also interesting evaluate our method in RL settings where lot of action-free data are available and won't have to be collected (Minecraft for example). It would be also interesting to use a language so the environment has more conditioned generative model as an alternative e.g. Dalle 57 58 complexity by prompting it with natural language.
Acknowledgements
We would like to thank Guillaume Desjardins for insightful discussion and giving constructive comments. We would also like to thank anonymous reviewers for their helpful feedback.
References
[1] P. Agrawal,, A. Nair, P. Abbeel, J. Malik, and S. Levine. Learning to poke by poking: Experiential learning of intuitive physics. In Advances in Neural Information Processing Systems 29; Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona. Spain, 2016.
[2] M. Baradad, J. Wulff, T. Wang. P. Isola, and A. Torralba. Learning to see by looking at noise. arXiv preprint (00Ki::216.000000 2021.
[3] S. Beeker. Learning temporally persistent hierarchical representations. In Advances in neural information processing systems, pages 824-830. Citeseer, 1997.
[4] Beirlant. Nonparametric entropy estimation: An overview. International Journal of the Mathematical Statistics Sciences, 6:17-39, 1997.
[5] J. Bromley, J. W. Bentz, L. Bottou, I. Guyon, Y. LeCun, C. Moore, E. Säckinger, and R. Shah. Signature verification using a "siamese" time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7(04):669-688) 1993.
[6] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, Sigler, M. Litwin, Gray, B. C.Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[7] Y. Burda, H. Edwards, D. Pathak, A. J. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity-driver learning. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[8] V. Campos, P. Sprechmann, S. S. Hansen, A. Barreto, C. Blundell, A. Vitvitskyi, S. Kapturowski, and A. P. Badia. Coverage as a principle for discovering transferable behavior reinforcement learning, 2021.
[9] S. Chaplot, M. Dalal, S. Gupta, J. Malik, and R. R. Salakhutdinov. Seal: Self-supervised embodied active learning using exploration and 3d consistency. Advances in Neural Information Processing Systems, 34, 2021.
[10] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 2020.
[11] x. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
10

---

[12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre- tre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 4171 Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10. 10.1100
1423.
[13] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
[14] C. Feichtenhofer, H. Fan, B. Xiong. R. Girshick, and K. He. large-scale study on unsuper- vised spatiotemporal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3299-3309, 2021.
[15] S. Fort, J. Ren, and B. Lakshminarayanan. Exploring the limits of out- qut-of-distribution detection. Advances in Neural Information Processing Systems, 34, 2021.
[16] R. Geirhos, K. Narayanappa. B. Mitzkus, T. Thieringer, M. Bethge, F. A. Wichmann, and W. Brendel. Partial success in closing the gap between human and machine vision. arXiv preprint arXiv:2 20021
[17] J. Gibson. The ecological approach to visual perception: classic edition. Psychology Press, 2014.
[18] J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27: Annual Conference Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec. Canada, pages 2672-2683 2014.
[19] R. Goroshin, J. Bruna, J. Tompson, D. Eigen, and Y. LeCun. Unsupervised learning of spatiotemporally coherent metrics. In Proceedings of the IEEE international conference on computer vision, pages 4086-4093, 2015.
[20] S. Gowal, S.-A. Rebuffi, O. Wiles, F. Stimberg, D. A. Calian, and T. A. Mann. Improving robustness using generated data. Advances Neural Information Processing Systems, 34, 2021.
[21] S. Hansen, W. Dabney, A. Barreto, D. Warde-Farley, T. V. de Wiele, and V. Mnih. Fast task infer- ence with variational intrinsic successor features. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR
[23] R. Held and A. Hein. Movement-producee stimulation the development of visually guided behavior. Journal of comparative and physiological psychology, 56(5):872, 1963.
[24] D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and ouito--diitributio examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[25] D. Hendrycks, K. Lee. and M. Mazeika. Using pre- ree-rrining can improve model robustness and uncertainty. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2712-2721. PMLR, 2019.
[26] A. Jahanian, X. Puig, Y. Tian, and P. Isola. Generative models as data source for multiview representation learning. arXiv preprint wwXiv:2106.052555 2021.
[27] S. James, Z. Ma. D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
[28] Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool,
R. Bates, A. Zidek, A. Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588) 2021.
[29] L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, A. Mohiuddin, R. Sepassi, G. Tucker, and H. Michalewski. Model based reinforcement learning for atari. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
11

---

[30] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[31] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.066 06676, 2020.
[32] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110-8119, 2020.
[33] H. Kataoka, K. Okayasu, A. Matsumoto, E. Yamagata. R. Yamada, N. Inoue, A. Nakamura, and Y. Satoh. Pre-training without natural images. In Proceedings of the Asian Conference on Computer Vision, 2020.
[34] P. Khosla, P. Teterwak, C. Wang. A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[35] C. Kidd and B. Y. Hayden. The psychology and neuroscience of curiosity. Neuron, 88(3):
449 460, 2015.
[36] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu,
A. Gupta, and A. Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint 2017.
arXiv:
[37] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 1.0000 2020.
[38] A. Krizhevsky et al. Learning multiple layers of features from tiny images. 2009,
[39] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learming with augmented data. arXiv:2004.1 0et.2004414999, 2020.
[40] M. Laskin, A. Srinivas, and P. Abbeel. CURL: contrastive unsupervised representations for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5639-5650, PMLR, 2020.
[41] M. Laskin, D. Yarats, H. Liu, K. Lee, A. Zhan, K. Lu, C. Cang. L. Pinto, and P. Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:21 2021.
[42] R. Levy and T. F. Jaeger. Speakers optimize information density through syntactic reduction. Advances in neural information processing systems, 19:849, 2007.
[43] H. Liu and Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pages 6733667777 PMLR, 2021.
[44] H. Liu and P. Abbeel. Behavior from the void: Unsupervised active pre-training. arXiv preprint arXiv:2103.04553 2021.
[45] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages
IEEE Computer Society, 2015. doi: 10.1 N0.1109/CCV.2015..22..
[46] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint DrXivv16080 03983, 2016.
[47] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland. G. Ostrovski, et al. Human-level control through deep rein- forcement learning. nature, 518(7540):5295335 2015.
[48] H. Mobahi, R. Collobert, and J. Weston. Deep learning from temporal coherence in video. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 737-744, 2009.
[49] Y. Netzer, T. Wang. A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[50] A. v. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
12

---

[51] A. E. Orhan, V. V. Gupta, and B. M. Lake. Self-supervised learning through the eyes of a child. arXiv preprint arXiv:2007.16189, 2020.
[52] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In Proceedings of the 34th International Conference on Machine Learn- ing, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages
PMLR, 2017.
[53] L. Pinto and A. Gupta. Supersizing self-supervision: Learning grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference on robotics and automation (ICRA), pages 3406-3415 IEEE, 2016.
[54] C. Quinn, P. D. Eimas, and S. L. Rosenkrantz. Evidence for representations of perceptually similar natural categories by 3-month-old and 4-month-old infants. Perception, 22(4):463-475, 1993.
[55] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision.
International Conference on Machine Learning (ICML., volume 139, pages 8748-8763 2021.
[56] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102. 12092, 2021.
[57] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv: Arxiv-2102./2092 2021.
[58] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv: Arxiv-2204.06125 2022.
[59] S. Ravuri and O. Vinyals. Classification accuracy score for conditional generative models. arXiv preprint arXiv: 1905. 10887, 2019.
[60] R. M. Ryan and E. L. Deci. Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary educational psychology, 25(1):54-67) 2000.
[61] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun,
J. Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9339-9347, 2019.
[62] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman. Data-efficient reinforcement learning with eeff-predictive representations. In International Conference on Learning Representations, 2021.
[63] M. Schwarzer, N. Rajkumar, M. Noukhovitch, A. Anand, L. Charlin, D. Hjelm, Bachman, and A. Courville, Pretraining representations for data-efficient reinforcement learning, arXiv preprint orXiv:2/06.0477) 2021.
[64] A. W. Senior, R. Evans, J. Jumper, J. Kirkpatrick, L. Sifre, T. Green, C. Qin, A. Zidek, A. W. Nelson, A. Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792):00-7110 -00, 2020.
[65] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 1134-1141. IEEE, 2018
[66] H. Singh, N. Misra, V. Hnizdo, A. Fedorowicz, and E. Demchuk. Nearest neighbor estimates of entropy. American journal of mathematical and management sciences, 23(3-4):301-321, 2003.
[67] E. S. Spelke and K. D. Kinzler. Core knowledge. Developmental science, 10(1):89-966 2007.
[68] A. Stooke, K. Lee, P. Abbeel, and M. Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pages 9870-9879. PMLR, 2021.
[69] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[70] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information P Processing Systems, 34, 2021.
13

---

[71] J. Tack, S. Mo, J. Jeong, and J. Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. arXiv preprint w:Xiv:207.088//2 2020.
[72] Tobin, W. Zaremba, and P. Abbeel. Domain randomization and generative models for robotic
grasping. CoRR, abs/1710.06425, 2017.
[73] H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2094-2100. AAAI Press, 2016.
[74] R. D. Walk, J. Shepherd, and Miller. Attention and the depth perception of kittens. Bulletin of the Psychonomic Society, 26(3):248-2)
1988.
[75] L. Weihs, A. Kembhavi, K. Ehsani, S. M. Pratt, W. Han, A. Herrasti, E. Kolve, D. Schwenk,
R. Mottaghi, and A. Farhadi. Learning generalizable visual representations via interactive gameplay. arXiv preprint arXiv:1912.081995 2019.
[76] L. Wiskott and T. J. Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-770, 2002.
[77] Y. Wu, Y. Wu, G. Gkioxari, and Y. Tian. Building generalizable agents with realistic and rich 3d environment. arXiv preprint arXiv: 1801, .2209, 2018.
[78] D. Yarats, Fergus, Lazaric, and Pinto. Reinforcement learning with prototypical representations. arXiv preprint arXiv:2102. 11122022.11227 1171, 2021.
[79] W. Ye, Liu. T. Kurutach, Abbeel, and Y. Gao. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 34, 2021.
[80] F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser, and J. Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans the loop. arXiv preprint arXiv: 1506. 03365, 2015.
14


Reviews:
[{"review_id": "ft5sXLaVaKp", "reviewer": "Reviewer_E5Lt", "summary": "This paper casts the latent space of a generative model (StyleGAN) as an _environment_ for unsupervised RL. The agent moves around in latent space seeking out high novelty states, and simultaneously learns a contrastive representation from temporally adjacent observations (generated images). The paper shows that this procedure can serve as effective pretraining for a vision system.", "questions": "My questions and suggestions are given above, so here I will just list the two things I would most appreciate in a rebuttal:\n1. Material that can convince me that the active sampling component of PALM is more effective than passive ways of sampling training data from StyleGAN. \n2. An explanation for the low performance of static PALM.", "limitations": "The limitations are adequately discussed. I appreciated the discussion about computational efficiency.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is such a cool idea! I think the main strength of the paper is that the _idea_ of active representation learning _in generative latent space_ is highly compelling and pretty original. However, the strength of this idea is not convincingly validated by the experiments and results. I will expand on each point below:\n\n**Originality**\nThe idea is, in my opinion, highly original. It combines several disparate lines of work -- on representation learning, curiosity, and latent variable generative models -- in a new and interesting way. The idea of treating latent space as an environment is really neat. The nearest works I know are those that train latent world models (but these typically do not use off-the-shelf image generators like StyleGAN) and video generative models that learn temporal dynamics in latent space (e.g., https://arxiv.org/abs/2006.10704; but I have not seen these  used for active representation learning). Given the large body of prior work on latent world models for RL environments (e.g., https://arxiv.org/abs/1912.01603), the Atari experiments are arguably not that novel. More interesting to me is the use of a pretrained StyleGAN as an environment. The present work is also quite related to older work in active learning, where the goal was to optimally sample a dataset such that training on those samples minimizes some form of regret. A good summary of that field is given in [Burr Settles. Active learning literature survey. 2009]. I feel this literature should be mentioned in the related work section, but I don't see it as detracting from the novelty of the current paper.\n\n**Quality**\nThe quality of the paper can be improved in several ways. I will focus on the experiments, as those are the critical failure in my opinion. I think there are three main issues with the experiments:\n1. *Section 5 does not yield insight* I didn't take much from Section 5, which compares PALM to other pretraining methods. The comparisons feel like apples to oranges. What do we learn by seeing that PALM (+ data aug) performs about as well as, e.g., SimCLR on CIFAR? These methods differ in innumerable ways: PALM is trained on synthetic data, SimCLR on real data; PALM uses the SimSiam architecture / loss, SimCLR uses a different architecture / loss; etc. The same can be said about pretty much all the comparisons in this section. I have no idea which difference between the methods was critical for explaining why one or the other does better (and anyway, most do about the same). I would suggest moving this entire section to the appendix, or even omitting it from the paper. I understand that the intent might be to show that PALM is competitive with \"SOTA\" methods, but if that's the goal then this section is still unconvincing. Comparisons on CIFAR and Atari are simply not the setting where one can make compelling claims about SOTA results, because the field, for the most part, is not competing on these benchmarks anymore. I do think these are perfectly fine datasets on which to do _science_ (carefully controlled experiments that yield understanding), but that's not what Section 5 provides.\n2. *Missing baselines / ablations* To me, the really interesting experiments are in Section 6. I think this section should be expanded and strengthened. The core novelty of PALM is the active sampling algorithm, so we need to be convinced that it really makes sense and helps. I think the critical experiment would be to compare PALM's sampling method to other ways of sampling from StyleGAN latent space. For example: the samples in Fig 2 look like random draws from the GAN (conditioned on a fixed class). It may be they are not but that suggests a missing baseline: what if you just randomly sample positives from G(z,c=c') (i.e. fix the class, take two random draws from z). Indeed this baseline has been previously published in ref [26] (appendix table 3) and performs quite well there. How does it perform in PALM's experimental setting? I think Fig 4 gets close to testing this -- it's the setting where alpha and beta both equal 0 -- but only a few selected values for alpha and beta are shown. I would like to see all the possible settings (could plot a heatmap of performance with alpha and beta as the axes). [26] also proposes a variety of other ways to sample from GAN latent space, which do not involve active learning but perform well (e.g., positive pairs are given by z and z+w, where w is Gaussian). How does PALM compare to these? The experiments on static PALM are also close to addressing this but they train on real data rather than synthetic data, so again, it's apples to oranges and unclear if the active learning is what's helping. Passive PALM was hard for me to interpret since the data is still initially collected in an active manner.\n3. *Questionable baselines* The performance of static PALM is suspicious to me. If I understand correctly, passive PALM is essentially SupCon but with the SimSiam architecture / loss. Why does it perform so poorly on CIFAR? The SupCon paper reports 96% accuracy on CIFAR10, whereas static PALM gets <55%. That gap is too large for me to believe that static PALM is a compelling baseline. Yes, the SupCon method differs in many ways but it nonetheless demonstrates that static sampling on real data can be very effective. I would like to better understand why the PALM numbers are so low in Fig 5. My concern is that the methods have not been tuned well or trained sufficiently to make meaningful conclusions.\n\n\n**Clarity**\nThe paper is mostly clear but 1) the latent dynamics model came across as ad hoc; can it be better motivated? 2) What is a tanh Gaussian (line 206)?\n\n**Significance**\nI think the significance is high if the method really works. I'm not yet convinced it does, but I do believe in the framework. As more powerful generative models become available (dall-e, etc), I think the direction explored here might turn out to be increasingly important.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ejPvF8tb360", "reviewer": "Reviewer_tz3i", "summary": "The authors propose to combine RL and GAN to perform unsupervised pretraining. The RL agent's objective is curiosity-based. The authors perform large-scale experiments and analysis to showcase the proposed approach's effectiveness.", "questions": "I don't have any specific technical question.", "limitations": "The authors adequately discuss the work's limitation.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I want to preface my review by saying that I haven't published any paper in unsupervised pretraining, so I'm not at all an expert in this domain.\n\nStrength:\n* I find the idea very interesting, and has a lot of potential. Currently, a simple generator is used. But It's easy to imagine that down the road, more advanced generator can be used to cover the long-tail of data. For example, using text-conditional diffusion models like Dalle / Imagen can drastically help an algorithm to cover the long-tail, which is quite exciting. Though this is not in the scope of the current paper.\n* The paper is well written. I can easily follow the paper as a non-expert in unsupervised pretraining.\n* The experiment section's analysis seems very solid.\n\nWeakness:\n* I think what I'm least convinced about is if the effectiveness comes from the RL part or the GAN part. It's easily to imagine that with the 50M images collected from StyleGAN during RL training, we can use these images as a static dataset and evaluate traditional approaches. With the amount of images generated, it's reasonable to expect that traditional approach might benefit from them. Note that this is different from the \"Passive PALM\" included in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dAtdYLWvOd", "reviewer": "Reviewer_bFfS", "summary": "This paper introduces an RL-based exploration strategy to the generative models that have been trained on very large datasets. The proposed approach aims to build a world model by learning a transition function in the latent space of the generative models. It is then used to train an agent interactively in this \"environment\", which aims at exploration along with unsupervised representation learning. The paper evaluates the learned representations in a number of downstream tasks such as classification, out-of-distribution detection, and representation learning in RL, and presents promising results.", "questions": "1. For a given representation $h_i$ in the exploration phase, the representations in the temporal vicinity of $h_i$ (i.e., $h_{i-1}$, $h_{i+1}$, etc.) should be the ones in the nearest neighbor set according to the Siamese network’s objective. However, the exploration rewards the discrepancy between them. How do the Siamese network objective and the unsupervised exploration objective work coherently?\n \n2. The proposed interaction mechanism is not fully clear to me. What is the purpose of using a random latent sample? Despite the exponential moving average, the updated latent $z_{t+1}$ may produce a significantly different observation $s_{t+1}$ than $s_t$, particularly in the unconditional setting. The predicted action is conditioned on only the state $s_t$ and the policy is unaware of the random latent sample. Could the authors clarify why the proposed approach is used? Couldn’t the policy be conditioned on the random sample?\n", "limitations": "Yes. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper is well-motivated and well-written. \n2. The idea of leveraging a large generative model as an environment for diverse data sampling is very interesting. \n3. Evaluations are performed on multiple tasks in various domains. \n4. Although it is proposed for unsupervised and task-agnostic representation learning, it could also be augmented with task-based rewards to develop a task-related sampling procedure. \n\nWeaknesses\n1. This feels like a chicken and egg problem in some cases. As it is presented in the Atari experiment, the generative model requires a large number of samples (50M) in the first place. The proposed model requires only 6M samples from it. It seems highly inefficient. What would be the performance if the Atari policy was pre-trained on all the 50M samples and also on a subset with 6M samples, and then fine-tuned?\n2. The paper does not present an evaluation of the diversity of the sampled synthetic data points.\n3. I think another problem is with the following statement in line #112: \"Different from them, our work focuses on leveraging a generative model as an interactive environment and eliminating the need for data augmentation.\" In the results in Fig. 3, it is clear that the proposed model is not a replacement for augmentation. Maybe the contribution claims regarding the augmentation should be toned down.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7VcgAEu7Wj", "reviewer": "Reviewer_ja7N", "summary": "This paper proposes a method that exploits both the large diverse datasets and exploratory behavior. Authors used the generator that was pre-trained on the static datasets, while introducing the dynamic model in the latent space. Transition dynamics are modeled via mixing an action and a latent variable. Further, temporal persistence is exploited via moving average. Experiments on several down-stream tasks demonstrate the effectiveness of the learned representation.\n", "questions": "In Fig. 3, I suggest using a finer-grained y axis grid.\nI suggest authors conduct experiments in more useful and practical situations.", "limitations": "None", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) Authors proposed interesting high-level concepts that try to mix dynamic exploration and the latent variables. The idea itself looks intuitive and intuitive.\n(-) The actual implementation of mixing dynamic exploration and the latent variable is quite heuristic. It is the simple weight summation of the two variables. Also, the temporal persistence module is similar. The overall accuracy also varies a lot via the heuristic parameters alpha and beta as shown in Fig. 4. \n(-) The experiments are conducted on several down-stream tasks; while I cannot see what are the benefits when performing the experiments on such domains. I encourage authors to conduct the experiments in more useful and practical situations. \n(-) I cannot understand what trajectories and time axes mean in image classification tasks. It seems that no temporal information exists in such an application; while authors still use the temporal persistency module. Furthermore the accuracy gain is not appealing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept