Unsupervised Active Pre-Training for Reinforcement Learning
Authors: Hao Liu
Pieter Abbeel
Venue:  (2021)
Keywords: Reinforcement Learning
Unsupervised Learning
Entropy Maximization
Contrastive Learning
Self-supervised Learning
Exploration

Abstract: We introduce a new unsupervised pre-training method for reinforcement learning called $\textbf{APT}$, which stands for $\textbf{A}\text{ctive}\textbf{P}\text{re-}\textbf{T}\text{raining}$. APT learns a representation and a policy initialization by actively searching for novel states in reward-free environments. We use the contrastive learning framework for learning the representation from collected transitions. The key novel idea is to collect data during pre-training by maximizing a particle based entropy computed in the learned latent representation space. By doing particle based entropy maximization, we alleviate the need for challenging density modeling and are thus able to scale our approach to image observations. APT successfully learns meaningful representations as well as policy initializations without using any reward. We empirically evaluate APT on the Atari game suite and DMControl suite by exposing task-specific reward to agent after a long unsupervised pre-training phase. On Atari games, APT achieves human-level performance on $12$ games and obtains highly competitive performance compared to canonical fully supervised RL algorithms. On DMControl suite, APT beats all baselines in terms of asymptotic performance and data efficiency and dramatically improves performance on tasks that are extremely difficult for training from scratch. Importantly, the pre-trained models can be fine-tuned to solve different tasks as long as the environment does not change. Finally, we also pre-train multi-environment encoders on data from multiple environments and show generalization to a broad set of RL tasks.

Under review as a conference paper at ICLR 2021
UNSUPERVISED ACTIVE PRE-TRAINING FOR REIN- FORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce new unsupervised pre-training method for reinforcement learning called APT, which stands for ActivePre- Training. APT learns a representation and policy initialization by actively searching for novel states in reward-free environ- ments. We use the contrastive learning framework for learning the representation from collected transitions. The key novel idea is collect data during pre-training by maximizing a particle based entropy computed in the learned latent representa- tion space. By doing particle based entropy maximization, we alleviate the need for challenging density modeling and are thus able to scale our approach to image observations. APT successfully learns meaningful representations as well as policy initializations without using any reward. We empirically evaluate APT on the Atari game suite and DMControl suite by exposing task specific reward to agent after a long unsupervised pre- pre-training phase. On Atari games, APT achieves human-level performance on 12 games and obtains highly competitive performance compared canonical fully supervised RL algorithms. On DMControl suite, APT beats all baselines in terms of asymptotic performance and data efficiency and dramatically improves performance on tasks that extremely difficult for training from scratch. Importantly, the pre-trained models can be fine-tuned to solve different tasks as long the environment does not change. Finally, also pre-train multi-environment encoders on data from multiple environments and show generalization to a broad set of RL tasks.
INTRODUCTION
Deep reinforcement learning (RL. provides a general framework for solving challenging sequential decision-making problems, it has achieved remarkable success in advancing the frontier of AI technologies thanks scalable and efficient learning algorithms (Mnih al.) 2015 Lillicrap et al.
2015
Schulman et al. 2015 2017). These landmarks include outperforming humans in board (Silver
al. 2016:20188 Schrittwieser et al. 2019) and computer games (Mnih et al. 2015 Berner et al. Schrittwieser et al. 2019 Vinyals et al.) 2019 Badia et al. (002a)). and solving complex al. 2017 Akkaya et al. [2019] Despite these successes, key
2019
robotic control tasks (Andrychowicz challenge with Deep RL is that requires huge amount of interactions with the environment before it learns effective policies, and needs to do so for each encountered task. Environments are required to have carefully designed task-specific reward functions to guide the RL algorithms (Andrychowicz cct al. 2017 Ng et al. 1999). which further limits its wide applications of Deep RL. This is in contrast to how intelligent creatures learn in the absence of external supervisory signals, acquiring abilities in a task-agnostic manner by exploring the environment.
Unsupervised pre-training is framework that trains models without expert supervision has obtained promising results in computer vision (Oord et al. 20188) He et al. 2019 Chen et al. 2020b Caron
al. 2020: Grill al. 2020) and natural language modeling (Vaswani al. 2017 Devlin al.
2018 Peters et al. 2018 Brown et al. 2020). The key insight of unsupervised pre- pre-training techniques
learning good representation or initialization from massive amount of unlabeled data such as
ImageNet (Deng et al. 2009), Instagram image set (He et al. 2019). Wikipedia, and Web (Radford)
al. 2019) which are easier to collect and scales millions trillions of data points. As a result, The learned representation when fine-tuned on the downstream tasks can solve them efficiently
without needing any supervision or few-shot manner.

---

Under review as a conference paper at ICLR 2021
Driven by the significance of the massive abundance of unlabeled data relative to la- beled data, pose the following ques- tion: is enabling efficient unsupervised pre- training for deep RI. as easy as increas- ing the amount of unlabeled data? Un- like the computer vision or language do- mains, in reinforcement learning it's not obvious where to extract large pools of unlabeled data. A natural choice is pre- training on ImageNet and transfer the en- coder to reinforcement learning tasks. We experimented with using ImageNet data for unsupervised representation learning as ini- tialization of the encoder in deep RL agent, specifically, we used the momentum con-
Choetain
Hopper Hop
2000
%
%
050 021
ARE E2N 1.00
125
150
1.7 2.00
535 332 3.9 125 177 2.00
Pre-training Imageiett
Count-based pre-craining
Figure 1: Unsupervised pre-training for deep RL on DM- Control. After pre- wre-training (e.g. on ImageNet or in Chee- tah reward free environment), the agent fine-tunes the pre- trained representation initialization to achieve higher task- specific rewards (e.g. let the Cheetah run faster). ImageNet pre-training denotes training McCo on downsampled Ima- geNet. Count-hased pre-training means training RL, agent with only count-based exploration signal. The training details are in Appendix Section [.I] The results show none of the two methods outperforms training from scratch.
trast (He et al. 2019 Chen et al.) 2020c) method which one of the state-of-the-art methods for representation learning. We
used DrQ (Kostrikov al., 2020) as the RL optimization algorithm. The results on DMControl are shown in Figure LI We can see that using ImageNet pre-trained representations does not lead to any significant improvement over training from scratch. We also experimented with using supervised pre-trained ResNet features as initialization similar to Levine et al.] (2016) (details in Appendix) but the results are no different. This seems in contrast to the preeminent successes of ImageNet pre-trained models in various computer vision downstream tasks (see e.g. Krizhevsky et al., 2012
Zeiler & Fergus, 2014, Hendrycks et al., 2019; Chen et al., 2020a). On the other hand, previous research in robotics also found that ImageNet pre- tre-training did not help (Julian et al.) 2020). We hypothesize that the reason for the discrepancy is that the ImageNet data distribution is far from the induced sample distribution encountered during RL training. It is therefore necessary collect data from the RI. agent induced distribution.
To investigate this hypothesis, we also experimented with training RL agents by 'exhaustively' collecting data during the reward-free interaction. Specifically, during pre-training phase, the only reward signal is defined by the count-based exploration (Bellemare et al. 2016 Ostrovski al. 2017) which is one of the state-of -ftee-of-hhaann methods for exploration (Taiga et al. 2019), and the density estimation model is PixeICNN (Van den Oord et al., 2016). The results of using the resulting pre-trained policy as initialization are shown in Figure I We can see that pre-trained initialization in Cheetah environment does not improve significantly over random initialization on Cheetah Run task. Similarly, pre-trained initialization in Hopper environment only leads to a small improvement over baseline. The reason for this ineffectiveness is that density modeling at the pixel level is difficult especially in the low-data and non-stationary regime. The results on DMControl demonstrate that simply increasing the amount of unlabeled data does not work well, therefore we need a more systematical strategy that caters to RL.
In this paper, we address the issue by proposing to actively collect novel data by exploring unknown areas in the task agnostic environment, Our means is maximizing the entropy of visited state distribu- tion subject to some prior constraints. The entropy maximization principle (Jaynes) 1957) originated in statistical mechanics, where Jaynes showed that entropy in statistical mechanics and information theory were equivalent. Our motivation is that the resulting representation and initialization will encode both prior information while being as agnostic as possible, and can be adapted to various downstream tasks. While the entropy maximization principle seems simple, it is practically difficult to calculate the Shannon entropy (Shannon 2001) as a density model is needed. To remedy this, we resort to the particle-based entropy estimator (Singh et al. 2003 Beirlant 1997) which has wide applications in various machine learning areas (Sricharan et al.) 2013) Pál et al. 2010, Jiao et al., 2018). The particle-based entropy estimator is known to be asymptotically unbiased and consistent. Specificallyy, it computes the average of the Euclidean distance of each sample to its nearest neighbors. We compute the entropy in the latent representation space, for this we adapt the idea of contrastive
learning (Hadsell et al. [0066] Gutmann & Hyvärinen 20100: Mnih & Kavukcuoglu 2013 He et al.)
2019 Chen et al. 2020b) to encode image observations to representation space.
2

---

Under review as a conference paper at ICLR 2021
Our approach alternates between training the encoder via contrastive learning and RL style opti- mization of maximizing expected reward where reward is defined by the particle-based entropy. After the pre-training phase, we can either fine-tune the encoder representation for test tasks that have different action space dimension or fine-tune the policy initialization for tasks with the same action space dimension. Since our method actively collects data during the pre-training phase, the method is named as Active Pre- Fre-Training (APT). We empirically evaluate APT on the Atari game suite and DMControl suite by exposing task-specific reward to the agent after a long unsupervised pre- treetrriiing phase. On the full suite of Atari games, fine-tuning APT pre- pre-trained models achieves human-leva performance on 12 games. On the Atari 100k benchmark (Kaiser et al.) 2019). our fine-tuning achieves 1.3x higher human median scores than state-of-the- training from scratch and 4x higher scores than state-of-the-art pre-training RL. On DMControl suite, fine-tuning APT pre-trained models beating all baselines in terms of asymptotic performance and data efficiency and solving tasks that are extremely difficult for training from scratch.
The contributions of our paper can be summarized as: (i) We propose new approach for pre- training in RL. (ii) We show that our pre-training method significantly improves data efficiency of solving downstream tasks on DMControl and Atari suite. (iii) We demonstrate that pre-training with perticle-bsed entropy maximization in contrastive representation space significantly outperforms prior count-based approaches that rely on density modeling.
2
RELATED WORK
2.1 INTRINSIC MOTIVATION AND EXPLORATION
The learning process of RL agents becomes highly inefficient in sparse supervision tasks when relying on standard exploration techniques. This issue can be alleviated by introducing intrinsic motivation, i.e., denser reward signals that can be automatically computed. These rewards are generally task- agnostic and might come from state visitation count bonus (Bellemare et al., 2016 Tang et al. 2017]] Ostrovski et al. 2017 /haa & Tresp|2019) learning to predict environment dynamics (Meyer &
Wilson 1991] Pathak al. [017]] Burda et al., 2018a Sekar et al. 2020), distilling random neural
networks (Brrda et al. 20118] Choi et al. 2018). hindsight relabeling (Andrychowicz et al.) (017)).
learning options (Sutton et al.) 1999) through mutual information (Jung [ Mohamed &
Rezende 2015), information gain (Lindley] 1956 Sun et al. 2011; Houthoolt et al. (016)) successor
features Kulkarni et al.) 2016 Machado et al., 2018). maximizing mutual information between behaviors and some aspect of the corresponding trajectory (Gregor et al. [016 Florensa et al. 2017]
Warde-Farley et al., 2018 Hausman et al.) 2018; Shyam et al. 2019), using imitation learning to
return to the furthest discovered states (Fcoffer et al.) 2019). self-play curriculum (Schmidhuber)
2013) Sukhbaatar et al. 2017 Liu et al. 2019). exploration in latent space (Vezzani et al.) (019))
injecting noise in parameter space (Fortunato et al.) 2017 Plappert et al. 2017), learning to imitate
self (Oh et al. 2018), predicting improvement measure (Schmidhuber) 1991; Oudeyerr et al. 2007]
Lopes et al. 2012 Achiam & Sastry 2017), and unsupervised auxiliary task (Jaderberg et al. 2016).
The work by Badia et al. (2020b) also considers k-nearest neighbor based intrinsic reward to incentive exploration, and shows improved exploration in sparse reward games. Our work differs in that consider reward-free settings and the objective of our intrinsic reward is based on particle-hased entropy instead of count bonus. The work closest to ours is Hazan et al. (2019) which presents provably efficient exploration algorithms under certain conditions. However, their method directly estimates state visitations through a density model which is difficult to scale. In contrast, our work turns to particle based entropy maximization in contrastive representation space. Concurrent work by Mutti et al., (2020) shows maximizing particle-basee based entropy can improve data efficiency in solving downstream continuous control tasks. However, their method relies importance sampling and on-policy RL which suffers from high variance and is difficult to scale. contrast, our work resorts to biased but lower variance entropy estimator which is scalable for high dimensional observations and suitable for pollyy RL optimization.
2.2 DATA EFFICIENCY IN RL
Deep RL algorithms are sample inefficient compared to intelligent biological creatures, which can quickly learn to complete new tasks. To close this data efficiency gap, various methods have been proposed: Kaiser et al. (2019) introduce a model-based agent (SimPLe) and show that it compares
3

---

Under review as a conference paper at ICLR 2021
favorably to standard RL algorithms when data is limited. Hessel et al. (2018); Kielak 2220); van Hasselt et al. (2019) show combining existing RL algorithms (Rainbow) can boost data efficiency. (2020) proposed to combine contrastive loss with image augmentation while follow-up results from Laskin et al. (2020) suggest that the most of the benefits come from its use of image augmentation. Laskin et al. (2020); Kostrikov et al.] (2020) demonstrate applying modest image augmentation can substantially improve data efficiency in vision-based RL. Our work improves data efficiency of RL in an orthogonal direction by unsupervised pre-training, the above advances can be used inside of APT to obtain better RL optimization in both pre-training and fine-tuning phases.
Srinivas et al.
3 METHOD
Our method shown in Figure consists of contrastive representation learning and particle-based entropy maximization in the learned representation space. Consider agent that sees observation It, takes an action a₂ and transitions to the next state with observation 2141 following unknown envi- ronmental dynamics. We want to incentivize this agent with a reward r, relating to how informative the transition was. The goal of the agent to learn good representation fa(()) of the observation æ, or good initialization of policy π (a(2z) by interacting with the environment in a reward-free manner, such that fine-tuning on downstream tasks achieves higher long-term cumulative task-specific reward than training from scratch.
Batch
Representation
Aug
Contrastive
fo
Loss
Aug(x))
Expected
Reward=log
Reward
K-th nearest neighbor
Figure Diagram of the proposed method Unsupervised Active Pre- Pre-Training: consists of contrastive representation learning on data collected by the agent (equation ( and RL optimization to maximize particle based entropy (equation (5). After pre- pre-training. the task- encoder fe and the RL policy initialization be fine-tur fine-tuned for different downstream tasks maximize task spec-spec reward.
Learning contrastive representations Within each batch of transitions sampled from the replay buffer. We apply data augmentation to each data point and the augmented observations are encoded into small latent space where a contrastive loss is applied. Our contrastive learning based on SimCLR (Chen al. 2020b), chosen for its simplicity.
exp(= 23)
min 0.6
E log
ΣE
2N
[R exp(= ( 2x.)
where æ is ni-dimensional data point in the observation space X' C RR" and 2;, 2j are normalized d2 dg-dimensional vectors of two random augmentations z; and 2, of the data point x followed by deterministic mapping go fe (z.) and fe is representation encoder given by fe R" + Rd and 96 a projection head go : Rdy ddzz, and N is the batch size. This objective tries maximally distinguish an input from alternative inputs xy. The intuition is that by doing so, the representation captures important information between similar data points, and therefore improve performance on downstream tasks.
Particle based entropy maximization Let the distribution of observations be p(x). The entropy of the observations is given by (p) = -Bz=p(() [log ((r). However, in high-dimensional spaces it challenging estimate the density, preventing us from directly maximizing the exact entropy.
To remedy this issue, we resort to the particle-based entropy estimator (Singh et al. 2003 Beirlant 1997) which is based on k-Nearest Neighbors (kNN). The particle based entropy estimate is given by
ik(p))
1 N
log
k + +log v(()
Vot
(2)
NVol
4

---

Under review as a conference paper at ICLR 2021
where V/ is the digamma function, log k 1 (k) is bias correction term, Vol is the volume of the hyper-sphere of radius R; = || 2 I, which is the Euclidean distance between 24 and its k-th
nearest neighbor aNNN The volume is given by:
|| ANN 1 x//2
Vol.
(3)
r +1)
where I is the gamma function, and the dimensions of X. Put equation and equation together, we simplify the entropy:
(p( Σ log | -x,
ANN I
When the target data distribution p2) (as in the case of off ooolcy RL, which we use for sample efficiency) different from the sampling distribution p(z), in principle importance sampling is needed to correct bias (Aigl & Šimandl 2011). However, empirically we find the biased approximation in equation (2 works fine and does not need to estimate the high variance importance ratios, as shown
in Section (Appendix).
Given a batch of transitions sampled from the replay buffer, we associate the particle based entropy estimation with the intrinsic reward, and use off-policy RI. algorithm to maximize the expected reward. With the objective of particle based entropy given in equation 4 we might able to maximize state entropy in continuous control, but it is still not applicable to learn visual RL agents in high- dimensional domains like DMControl and Atari games.
To remedy this issue, we maximize the entropy in our learned lower-dimensional representation space, do this by jointly learning representations by contrastive learning equation (I) and ex- ploring by particle based entropy maximization equation 4 Specifically, for a batch of transitions ((r2, 0z, 31++1) }} sampled from the replay buffer, each Ft+11 treated particle and we associate each transition with a intrinsic reward given by
r(Te, Dts 1++1) log(Ilb441 - 9i-1 In + c)2
where y - (z) is the representation (i.e. we estimate the entropy in the latent space), is constant for numerical stability (fixed to in all our experiments). In order keep the rewards scale, we normalized the intrinsic reward by dividing it by a running estimate of the standard deviation of the intrinsic reward. A detailed computation of the intrinsic reward in Py PyTcc can be found in Algorithm
consistent
With the intrinsic reward defined in equation (5). we can derive the intrinsic reward decreases to O most of the state space visited which is a favorable property for pre-training.
Algorithm 1 Unsupervised Active Pre- The-Trainin
f: encoder network shared between actor & critic projection network for contrastive learning k: hyperparameter of kNN while iterations far < max pre-training iterations do
Lemma 1. Assume we have an episodic MDP setting, and a finite state space X' "CR. R", and buffer of observed states (I1,,..,,37) with total sample size T, a deterministic representation encoder fo : 1 + IRdy, an intrinsic reward defined as equation (5) with l N, and an op- timal policy that maximize the intrinsic rewards. We can derive the intrinsic reward is O in the limit of sample size
Interact with a reward-free environment
I/ sample butch of transitions from replay buffer I! update f and g minimize contrastive loss Update(f.param, param, g-param) I/ compute intrinsic reward for each transition Compute yt+1 for each sampled transition
Set rxx,, de. 1++1) log( log([]+++1 y 12 + 1) I/ update RI. agent to maximize expected reward Update(actor.param critic.param)
a, x') r(z,0,x)) T
lim
Va E X.
(6)
end while
Proof. Since the intrinsic reward r(x,, a, x²) defined in equation depends on the k-th nearest. neighbor in latent space and the encoder fa is deterministic, we just need to prove the visitation count c(x) of is larger than & as T' goes infinity. We know the MDP is episodic, therefore as T + 00, all states communicate and c(x) + oc, thus we have lim c(x) 2 k Vk E N, Vo E X. T-250
While the assumption of finite state space may not be true for large complex environment like Atari games, lemma gives more insights on using this particular intrinsic reward for pre-training. We use
5

---

Under review as a conference paper at ICLR 2021
in implementation because of the structure imposed on the contrastive representations. The kNN in principle should be computed over the entire buffer which means it scales linearly with sample size, we instead compute the intrinsic reward within current batch to trade-off computation. We fixed k: = 3 in all our experiments as we found it works well in initial experiments. APT alternates between minimizing contrastive loss in equation (T) and maximizing expected intrinsic reward in equation (5). The pseudocode of APT is shown in Algorithm | and the full pscudocode of the algorithm summarized in Algorithm 3 (Appendix). The diagram of APT is shown in Figure 2
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
The evaluation benchmarks are DeepMind Control Suite (DMControl; Tassa et al.) 2020) and Atari suite (Bellemare et al. 2013) from OpenAl Gym (Brockman et al. 2016). For DMControl, we use pixel observation instead of state as input. For all of the experiments, we use DrQ as the underlying RL optimization algorithm. Unless stated otherwise. all curves are the average of three runs with different seeds, and the shaded areas standard errors of the mean. The results reported on Atari games suite are averaged over five runs with different seeds. In addition to the existing tasks in DMControl, also design several new sparse reward tasks: (1) /HalfCheetah, Hopper, Walker) Jump Sparse: the agent receives a positive reward for jumping above given height otherwise reward is (2) (HalfCheetah, Hopper, Walker/ Reach Sparse: the agent receives positive reward 1 for reaching given target location otherwise reward is 0. (3) Walker Escape Sparse: the initial position of Walker is turned upside down, and receives reward for successfully turning itself over otherwise 0. In all the considered tasks, the episode ends when the goal is reached.
Table I: Evaluation on Atari games. @N represents the amount of RI interaction utilized. Mdn is the median of coaman--ormmlizedd scores, M is the mean, > 0 is the number of games with better than random performance, and H is the number of games with human-level performance. On cach subset, we mark as bold the highest score. Since different papers report different results supervised RL e-g SimPLe, we choose the best available results and contrast them to APT's results. The results of VISR are cited from Hansen et al. (2020) as the source code is not publicly available. Raw scores of each Atari game given Table 5 (Appendix). Top: data-limited RL. Bottom: RL with unsupervised pre-training.
26 Game Subset
47 Game Subset
Full 57 Games
Algorithm
Mdn
M
>
0
> H Mdn
M
> > H Mdn
M
> 0 > H
// sully-supervised training
SimPLe @100k
14.39
44.30
26
OTRainbow @ @100k
20.40
26.42
26
DrQ @ 100k
28.42
35.70
24
PPO @500k
20.93
43.74
25
7
DQN @10M
27.80
52.95
25
9.91
28.07
41
8.61
27.55
48
Unsupervised pre-training with supervised
fine fine-tuning @100k
DIAYN
0.01
16.94
13
1.31
19.64
28
6
1.55
16.65
33
VISR
9.50
128.07
21
9.42
121.08
35
11
6.81
102.31
40
11
GPI VISR
6.59 111.23
22
11.70
129.76
38
12
8.99
109.16
44
12
Count-based
1.23
21.94
16
APT (ours)
38.23
59.89
26
34.22 61.78 36
12 41.25
87.33
43
12
4.2
SINGLE ENVIRONMENT PRE-TRAINING
We conduct a full evaluation of APT on diverse set of tasks in the single environment setting, the models are pre-traine treitrained on Cheetah, Hopper, and Walker for long period of interacting without reward supervision (5M steps), then fine-tuned for 15 downstream RL tasks such controlling the Walker to turn itself over. Fine-tuning representation is a common practice in deep learning (Krizhevsky ett al. 2012 He et al. 2016). In our RL experiments, APT stands for fine-tuning both representation and RL agent initialization. The main baseline count-based exploration (Bellemare et al.) 2014 2016 Ostrovski et al. 2017), which proposed way estimate counts in high dimensional

---

Under review as a conference paper at ICLR 2021
Cheetah Run
Cheetah Flip Forward
Cheetah Backward
1000
1300
2000
E3100de Nauurn
800
Esleade Reburn
930
600
Esiaade
800
200
230
500 0.25 0..0 1.73
1.81
125 150
175 X/0
550
0.21
0.30
0.35
1-81 125 150 1/5
210
0.75 0.:0 471 1-1 125 15 1/5 210 299
Envronment Steps (4:17)
Envroniment Steps (x127
Emv romimnnttt Steps (x1P)
Cheetah Jump Sparse
Cheetah Reach Sparse
Hopper Hop
0.6
Saccess RoCC
0.5
Saccesss
044
0.0
Envrrommett Staps bx33
Emvronment Staps (-127)
Envrrnmment (x10P)
Hopper Stand
Hoppor Hop Backward
Hopper jump Sparse
1300
900
Epsodde
200
800
044
Stapa
Hopper Roach
Walker
Wallerr Stand
2.0
1000
E.A
Succres Da0k
Eplacde Hatuum
300
6.6
300
600
0.4
600 0.25
0.30
0.36
1.2
1.5
177
2.0
0.2
0.2
Envronment Staps bx:37)
Envronment Staps (x237)
Envrrnmentt Staps (x10P)
Walker Walk
Walker jump Sparse
Walker Escape Sparso
1.00
E244206 1tA.m
900
Success Razo
km
Succes
400
200
6/6
0.8
La
0.2
2.2
0.4
6.6
GUB
0.2
0.4
5.5
Envrnoment Staps (x:33))
Emi Ponmmnnt Stapp Exa2")
Emn ronmmnt Staps Ex11P)
From scratch
Active pre-training (ours)
Count-based pre-training
Figure 3: Evaluation on DeepMind Control suite. Models are pre-trained on Cheetah, Hopper, and Walker, and subsequently fine-tuned on respective downstream tasks. The 'sparse" denotes reward is sparse. The scores of each environment given in (Abpendii
states spaces by estimating density; the agent is then encouraged to visit states with a low visit count. We followed Ostrovski et al. (2017) to use PixelCNN (Van den Oord et al.) (016) as a density model. Count-based exploration with PixeICNN is an efficient exploration method in sparse reward environments (Taïga et al. 2019), we will refer to this baseline as count-based exploration or count-based throughout the rest of the paper.
Results on DMControl shown in Figure Figure demonstrate that APT beats all baselines on all the tasks across different environments, while the sparse reward tasks are extremely difficult for training from scratch. In some cases, APT allows for very rapid fine-tuning, indicating APT learns reward- representation and meaningful RL initialization. The significantly superior performance of APT empirically shows that maximizing particle- particle-based entropy can drive the RL agent to collect diverse samples and learn reward-free initializations that are effective for downstream tasks.
7

---

Under review as a conference paper at ICLR 2021
The evaluation the full suite of 57 Atari games (Bellemare al. 2013) follows the setting of VISR (Hansen et al. 2020). Firstly, we evaluate APT in a two-phase setup. Agents are allowed a long unsupervised pre- pre-traimng phase without access rewards, followed by short test phase(100k steps). DIAYN (Eysenbach et al. 2018) is a skill-discovery method that maximizes the mutual information between latent variable polices and their behavior in terms of state visitation. The main baseline is VISR (and its variant GPI VISR), which combines skill discovery with universal successor approximators (Borsa al. 2018) to enable fast task inference both training and test phases (Barreto et al. 2017 2017/2018). Due to the high computational cost, the count- count-based baseline is not evaluated on the entire 57 games suite but the 26 games subset (Kaiser et al. 2019). Secondly, we contrast APT with canonical RL algorithms in the low-data regime, following the setting of Kaiser jet al. (2019). The compared algorithms include DrQ. SimPLe (Kaiser et al. 2019), proximal policy optimization (PPO) (Schulman et al. 2017). and OTRainbow (Kielak 2020).
Results shown in Table I demonstrate that APT significantly outperforms all baselines and buying performance equivalent hundreds of millions of sampling steps. Note that it is possible we can further improve the performance by directly applying VISR on top of the pre-trained models learned by APT, we leave it for future direction. A further discussion of the connections and differences between APT and VISR can be found in Section i Appendix which gives more intuitions.
4.3
REPRESENTATION AND RL AGENT INITIALIZATION FINE-TUNE
Hopper Hop
Hopper Sparse
Hopper Sta
480
8.
2
Splacst
815
3.50
2.08
2.00
Figure 4: Comparison of fine-tuning representation, fine-tuning both representation and RL agent, and ablated baselines. Models pre-trdined on Hopper and subsequently fine-tuned downstream tasks. The 'sparse' denote reward sparse. Both variants of APT outperform training from scratch and other baselines.
Comparison of fine-tuning representation. fine-tuning both representation and RL agent on Atari games. The results obtained after fine-tuning 100k timesteps and averaged over five random seeds.
Table 2:
Atari Games
Human Random
VISR
APT(representation)
APT (ours)
7/ Hard exploration games (dense reward)
BanHeist
753.1
14.2
200.3 + 14.9
456.7 + 15.6 4571.6 + 234.6
347.7 + 24.6
Hero
30826.4
1027.0
663.5 + 31.8
6789.1 + 171.4
// Hard exploration games (sparse reward)
Freeway
29.6
0.0
-2.1 + 0.5
12.9 0.7
29.9 1.3
MontezumaRevenge 4753.3
0.0
0.0
+ 0.0
43.1 + 15.6
147.0 + 24.3
We investigated the difference between fine-tuning only the representation and fine-tuning both representation and policy initialization. APT denotes fine-tuning both representation and RL policy initialization while APT (representation) stands for fine-tuning encoder from pre-trained models and randomly initializing policy. The notable difference that APT (representation) decouples the action space dimension from pre-trained models. We applied count-based exploration on top of contrastive learning representations to eliminate the potential effect of representation learning difference, this baseline is shown as contrastive count-based pre-training, where we train VAE (Kingma & Welling 2013) on the learned contrastive representations to estimate state visitation.
As we show in Figure 4 APT (representation) beats all of the supervised RL and pre-training RL baselines, fine-tuning both representation and RL initialization further improve performance. In some cases, APT allows for more rapid performance improvement than APT (representation) in small fraction of the total number of samples, indicating APT learns meaningful reward-frr free and task-agnostic behavior. Both APT and APT (representation) significantly outperform the ablated
8

---

Under review as a conference paper at ICLR 2021
contrastive count-based pre- pre-training baseline, confirming the particle-based entropy maximization is a crucial component.
Table shows the results of fine-tuning pre-trained models for 100k timesteps on Atari games. VISR tends to be more data efficient than APT and APT(representation) on easy exploration games, potentially due the explicit reward regression and successor feature in VISR. On hard exploration games, e.g., Freeway, APT has a significant advantage, achieving an order of magnitude higher scores than VISR while maintaining very high score across the remaining games. APT significantly outperforms APT(representation) on Atari games, showing the learned exploratory policy is crucial for learning with very limited number of interactions.
4.4 DISCUSSION OF MULTI-ENVIRONMENT PRE-TRAINING
Cheetah Run
Cheetah Readh Sparse
Hopper Hop
0.0
Epasdd Return
930
Suaroess
030
9.30
200
0.25
8.5
0.75
1.30
175
2.00
84
0.5
1.3
322
655
0.75
1.00
2.55 2000
EfV ronment Steps (x31]
Emv ronmeneenn Stepe (x117)
Envronment Stops (x117)
Hoopper Reach Sparse
Walker Run
Walker Escape Sparse
300
Suacesss Da0
800
Succoss Rote
300
400
8
82
5-4
2.4
000 422
050 0.7
1.01
1.30
1.35
2:00
0.22
8.4
0.0
Emv ronment Staps (x337)
Envreniment Staps (x:37))
Emv mnwimenennn Stapp (x-10))
From scratch
Active pre-training
Active pre-training (multi-env representation)
Figure 5: Comparison between fine-tuning representations learned on single environment and multiple environ- ments. We apply APT to all the three environments.
We completed an initial exploration of APT (representation) in multi-environmenn setting on DMCon- trol suite, results shown in 3 In this setup, APT simultaneously learns pre-trained representation from Hopper, Cheetah, and Walker environments. We evaluate the pre-trained model by using it in separate RL agents learning each downstream task from every environment. As shown in Figure S we found multi colti-enniirommnt pre- pre-training outperforms training from scratch in sparse reward tasks and performs on par better than training from scratch in dense reward tasks, showing multi-en eettrimmnvionnenttt pre-training is efficient. Comparing with single environment APT, entirineviiooenntt pre-training tends have mixed results, indicating there exists intervention between representation learning in different environments. We remark despite the multi-environment variant of APT is outperformed by APT, the multi-environment pre-training a novel research direction and our careful implementation considerations and extensive experimental results allow the method to be widely adopted.
CONCLUSION
A new unsupervised pre-training method for RL is introduced to address reward-free pre-training for visual RL. On DMControl suite and Atari games, our method dramatically improves performance on tasks that are extremely difficult for training from scratch. Our method achieves the results of fully supervised canonical RL algorithms using a small fraction of total samples and outperforms supervised
data-efficient supervised RL methods. Our major contribution is proposing efficient algorithm for maximizing particle-base based entropy in the latent representation space, allowing the same agnostic pre-trained model successfully tackle broad set of RL tasks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-baedd intrinsic motivation for deep reinforcement learning. arXiv preprint /rX3vv:/03.0/111 2017.
Jifí Ajgl and Miroslav Simandl. Differential entropy estimation by particles. IFAC Proceedings Volumes, 44())11191-11196, 2011.
lige Akkaya, Marcin Andrychowicz. Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et Solving rubik's cube with robot hand. arXiv preprint arXiv:
2019.
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay, In Advances neural information processing systems, pp. 1048-5058, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint 160.06660 2016.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agem57: Outperforming the atari human benchmark. arXiv preprint arXiv:2003. 2020a.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.0600 06038, 2020b.
David Barber and Felix V. Agakov. The im algorithm: variational approach information maximization. In Advances in neural information processing systems, 2003.
André Barreto, Will Dabney, Rémi Munos. Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in neural information processing systems, pp. 4055 4065, 2017.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pp. 501- 510. PMLR, 2018.
J Beirlant, Nonparametric entropy estimation: An overview. International Journal of the Mathemati- cal Statistics Sciences, 6:17-39, 1997.
Mare Bellemare, Joel Veness, and Erik Talvitie. Skip context tree switching. International Conference on Machine Learning, pp. 1458-1466, 2014.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems, pp. 1471-1479, 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: arXiv: /91.006800 2019.
Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado van Hasselt, David Silver. and Tom Schaul. Universal successor features approximators. arXiv preprint 1rXiv::812.07626. 2018
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arxiv: 2016.
10

---

Under review as a conference paper at ICLR 2021
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprini 1rXiiv/808.043333 2018a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12894, 2018b.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i Nieto, and Jordi Torres. Explore, discover and lear: Unsupervised discovery of state-covering skills. arXiv preprint arXiv:2002..36647, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33, 2020.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. Proceedings of the 37th International Conference on Machine Leurning, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020c.
Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. Contingency-awar exploration in reinforcement learning. arXiv preprint
arXiv:1811.0148; 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Ieee, 2009.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 10.0005 2018.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint 19XVv:1911.10995 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv: /802.0/561. 2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. arXiv preprint arXiv: /802.0000, 2018.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforce- ment learning. arXiv preprint 17 2017.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprini arXiv: /706. 10295, 2017.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 161 .07507, 2016.
nan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint arXiv:2006. 07733, 2020.
11

---

Under review as a conference paper at ICLR 2021
Michael Gutmann and Aapo Hyvärinen. contraavnt estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. pp. 297-304, 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (C)PR'06), volume 2, pp. 1735-1742. IEEE, 2006.
Steven Hansen, Will Dabney, Andre Barreto, David Marde-Farleyy Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. International Confer- ence on Learning Representations, 2020. URL /ttp:://ppemreieww.ccc net wwt/forunmioi
BJeAHk
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning embedding space for transferable robot skills. In International Conference on Learning Representations, 2018.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pp. 2681-2691, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recagnition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722. 2019.
Dan Hendryeks, Kimin Lee, and Mantas Mazeika. Using pre- pre-training can improve model robustness and uncertainty. arXiv preprint arXiv: 1eeei9991.099 009600, 2019.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):
1735-1788 1780, 1997.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109-1117, 2016.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397 2016.
Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Jiantao Jiao, Weihao Gao, and Yanjun Han. The nearest neighbor information estimator is adaptively near minimax rate-optimal. In Advances in neural information processing systems, pp. 3156-3167, 2018.
R. Julian, B. Swanson, G. Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. 2020.
Tobias Jung, Daniel Polani, and Peter Stone. Empowerment for continuous agent-environmenn systems. Adaptive Behavior, 19(1):16-39) 2011.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.00377 2019.
Kacper Kielak. Do recent advancements in model-based deep reinforcement learning really improve data efficiency? arXiv preprint 2013, 2020.
12

---

Under review as a conference paper at ICLR 2021
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Tejas D Kulkarni, Ardavan Sacedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint (tti::6006002996 2016.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbecl, and Aravind Srinivas. Reinforcement learming with augmented data. arXiv:2004.14990 2020.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):333-1373, 2016.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint (tX.v::509.029970 2015.
Dennis V Lindley. On measure of the information provided by an experiment. The Annals of Mathematical Statistics, pp. 986-1005, 1956.
Hao Liu, Alexander Trott, Richard Socher, and Caiming Xiong. Competitive experience replay. arXiv preprint arXiv: wrXiv:1902.00528 2019.
Manuel Lopes, Tobias Lang, Marc Toussaint, and Pierre- Oudeyer. Exploration in model- based reinforcement learning by empirically estimating learning progress. In Advances in neural information processing systems, pp. 206-214, 2012.
Marlos C Machado, Mare G Bellemare, and Michael Bowling, Count-based exploration with the 11622, 2018.
successor representation. arXiv preprint
Jean-Arcady Meyer and Stewart W Wilson, A possibility for implementing curiosity and boredom in model-building neural controllers. 1991.
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in neural information processing systems, pp. 2265-2273, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533, 2015.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pp. 2125-2133, 2015.
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. A policy gradient method for task-agnostic exploration. arXiv preprint arXiv:2007.046440 2020.
A. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, 1999.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. arXiv preprint 1rXiv:1806.05633 2018
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: htt.:/807.3744 2018.
Georg Ostrovski, Marc G Bellemare, Aaron van den Oord, and Rémi Munos. Count-basce exploration with neural density models. arXiv preprint 17Xiv=1703.01310 2017.
13

---

Under review as a conference paper at ICLR 2021
Pierre- Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary computation, 11(():265-288) 2007.
Dávid Pál, Barnabás Póczos, and Csaba Szepesvári. Estimation of rényi entropy and mutual informa- tion based on generalized nearest-neighbor graphs. In Advances in Neural Information Processing Systems, pp. 1849-1857, 2010.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by superviee prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.088. 1802.05365 2018.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint 1706.
2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Jürgen Schmidhuber. Curious model-building control systems. In Proc. international joint conference on neural networks, pp. 1458-1463, 1991.
Jürgen Schmidhuber. Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers in psychology, 4:313, 2013.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv: (01i::9911.026... 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-18977 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arxiv 2017. arXiv preprint arXiv: mee::/007704433 2017.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. arXiv preprint Ttl::2005505960 2020.
Claude Elwood Shannon. A mathematical theory of communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55, 2001.
Archit Sharma, Shixiang Gu, Sergey Levine. Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: 1oXiv:/907.01657 2019.
Pranav Shyam, Wojciech Jaškowski, and Faustino Gomez. Model-based active exploration. In International Conference on Machine Learning, pp. 5779-5788, 2019.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
David Silver, Thomas Hubert, Julian Schritwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140-1144 1444, 2018.
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Nearest neighbor estimates of entropy. American journal of mathematical and management sciences, 23 (3 (3-4):301-321, 2003.
14

---

Under review as a conference paper at ICLR 2021
Kumar Sricharan, Dennis Wei, and Alfred O Hero. Ensemble estimators for multivariate entropy estimation. IEEE transactions on information theory, 59(7):4374-4388 2013.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004, 0ee::200400416,, 2020.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play arXiv preprint arXiv:1703.05440 2017.
Yi Sun, Faustino Gomez, and Jürgen Schmidhuber. Planning to be surprised: Optimal bayesian ex- ploration in dynamic environments. In International Conference Artificial General Intelligence, pp. 41-51. Springer, 2011.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2)1811211 1999.
Adrien Ali Taīga, William Fedus, Marlos C Machado, Aaron Courville, and Mare Bellemare. Benchmarking bonus-based exploration methods on the arcade learning environment. arXiv preprint arxiv: 02388, 2019.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pp- 2753 2017.
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. co__ottrol: Software and tasks for continuous control. arXiv preprint arXiv:2006. 12983, 2020.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. In Advances in neural information processing systems, pp. 4790 4798, 2016.
Hado P van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? In Advances Neural Information Processing Systems, pp. 14322-14333, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Giulia Vezzani, Abhishek Gupta, Lorenzo Natale, and Pieter Abbeel. Learning latent state representa- tion for speeding up exploration. arXiv preprint arXiv: 9905 /2621, 2019.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354 2019.
David Warde-Farley Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non -orammamtt discriminative rewards. arXiv preprint arXiv:1811.11335 2018
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
Rui Zhao and Volker Tresp. Curiosity- driven experience prioritization via density estimation. arXiv preprint 1902:1902.08800 2019.
15

---

Under review as a conference paper at ICLR 2021
A
REWARD COMPUTATION IN PYTORCH
Algorithm 2 Computation of reward in APT
import torch
device = torch. device ( ("cuda" if torch. cuda. is_available () else "cpu")
class RMS (object):
def
init (self, epsilon=le-4, shape= (1,));
self. M torch, zeros (shape) to (cevice)
self.S torch, ones (shape) to (device)
self.n epsilon
def
call (self, x):
bs x.size (0)
delta torch.mean ix, dim=C) - self.M
new self.M delta bs (self.n bs)
new_s (self.8 self.n torch, var (x, dim=0) by
torch. square (delta) (delta)
* self.n + bs (self.n ba)) / (self.n ba)
self. .M new
self.S S new
self.n +=
return self.M, self.S
rms = RM3 () $ moving statistics of reward source: batch of states sampied from replay buffer (bl, c) target: same as source or all of the states in replay buffer (b2, o) encoder: contrast ivv representat lon enccoer target, encoder, k):
def compute intrinsic rewaderrwwrrdduureeeec with torch ()
source, target encoder (source),
aacoderttarget sim_matrix torch, norm(source None, :l target [None, :l,
dim=-1, p=2)
[bl, 1, e) - (1, b2, c) -> (bl, b2, e) -> (bl, b2) sim_we ght, sim_indice sim_matrix.topt (k, dim=i, largest lse) (bl, k)
reward - torch.oorm norm (source target seiminndceee -1]1, dim-1, p-2, keepdin=True)
(bl, k) k-th based moving_mean mean, noving_sté = rms (reward)
reward reward moving_std reward torch, (reward 1.0)
return reward
16

---

Under review as a conference paper at ICLR 2021
B EVALUATION OF ENTROPY MAXIMIZATION
We conducted experiments evalu- ate APT'S performance in maximizing entropy in state based continuous con- trol tasks from OpenAl Gym (Brock-
Ant-v2
Humanoid-v2
10
entropy index
entropy index
man et al. 2016). We compared
APT with state-of-the-an state entropy
maximization methods, including (1)
MaxEnt (Hazan et al., 2019) which
proposes a provable efficient explo- ration method for maximizing state entropy, and shows their method can improve the efficiency of exploring the state space in continuous control. (2) MEPOL. (Mutti et al. 2020) which
0.6 sample size
167
sample size
167
Random
MaxEnt
MEPOL
APT
Figure 6: Comparison entropy between different state exploration methods. The results are averaged over 5 random trials. Error bar denotes one standard derivation.
is a recent state-of- oote--f-theearr art of exploration in continuous control, their estimation of particle based entropy based on importance sampling and unbiased. (3) APT-like MEPOL which denotes training APT objective with MEPOL's trust region optimization.
Following MaxEnt (Hazan al., 2019), we compute the entropy index by discretizing the state space. The Ant-v2 Jump and Humanoid-v2 Standup tasks are two sparse reward tasks given in MEPOL Muutti al. 2020).
Figure 6 shows the results of entropy achieved by different methods, Table 3 and shows the results of fine-tuning pre-trained models for 0.5. timesteps. MaxEnt performs poorly in high-dimensional continuous control due to state density modeling is difficult. APT-like MEPOL outperforms MEPOL, indicating the objective of APT balances between variance and bias. APT significantly outperforms all baselines in maximizing entropy and also has the highest data efficiency in solving sparse reward tasks, confirming that the off-policy nature of APT is crucial for both pre-training and tine-tuning.
From scratch
APT-like MEPOL
Gym Environments
MEPOI.
MaxEnt
APT (ours)
(state input)
Sparse reward tasks
Success
Ant-v2 Jump Humanoid-v2 Standup
.12 1 .07 10400 .0
.44 1 .29
.52 1 .04 .78 + .06
.18 + .05
.96 1: .10
.72 + 21
.12 +: .04
.94 +: .13
Dense reward tasks
Episode reward
Ant-v2 Humanoid-vy
1467.3 879.4 1635.7 + 876.9 4854.3 1 1012.6
4278.1 + 712.7
1534.6 + 318.6
4857.8
875.9
1427.3 567.8 1987.4 467.8 678.4 267.8
5036.9 1 1367.8
Table 3: Comparison of fine-tuning on continuous control environments in OpenAl Gym. Maximum value for each task is holded. + corresponds to single standard deviation over 3 runs with random seed.
C ANALYSIS VARIATIONAL UNSUPERVISED RL METHODS
We contrast APT with variational unsupervised RL algorithms to give more intuitions besides the empirical evaluation in Table
Utilizing a strong inductive bias that is likely to yield features relevant to rewards of possible downstream tasks has been the central goal of unsupervised RL research. One of the widely used such bias is proposed by /Achiam & Sastry (2017); Gregor et al. (2016) that is to only represent the subset of observation space that the agent can control. This can be accomplished by maximizing the mutual information between policy conditioning variable and the agent's 's behavior. Formally, the goal is to learn latent-conditioned policies Te [a)ss, 2) and define skills as the policies obtained when conditioning parameter maximize the mutual information through various means (see e.g. Eysenbach et al. [2018) Hansen et al., 2020; Warde-Farles al. 2018 Sharma et al. 2019). The quantity can be derived by
fixed value of Z. There exist many algorithms that optimize policy
17

---

Under review as a conference paper at ICLR 2021
expanding the definition and derive variational lower bound (Barber & Agakov 2003
j(0) (2)
(7)
H()) (z|ss) Ex,: log (=|s) + E, [KL (z)
(8)
(9)
+ H()),
(10)
(11)
where g0(2|s a variational approximation. practice, sampling from a fixed distribution yields better and more stable results (Eysenbach al. 2018 Hansen al. 2020), which simplifies the objective to maximizing the conditional entropy,
L(0, Φ) Ex, log qo (z|s)]-
(12)
The optimization of equation
is then accomplished by RL algorithm by defining reward
r(s, a, s') log (z(2(s)
(13)
equation 12) has been shown effective in RL, from learning skills in state based control in DI- AYN (Eysenbach et al. (2018) and EDL (Campos et al. 2020) to combining successor features with skill discovery in VISR (Hansen et al. 2020).
Comparing the variational based intrinsic reward equation
133 with the intrinsic reward used in that the particle-based reward does not need learn parametric
APT( equation ()), we
probabilistic density and thus gracefully scales to high dimensional ussionbbssed RL. see that APT performs considerably better downstream RL tasks as shown in Table 1
D COMPARISON ON DMCONTROL.
From scratch (visual)
From scratch (state)
DMC Environment
Count-based
DIAYN
VISR
APT (ours)
Menss reward tasks
Mean episode return
Cheetah Run
660 + 96
772 60
610 78
256 = 128 640
: 134
671 89
Cheetah Flip Forward
885 = 112
937 73
867 1 98
432.188
897
215
987
97
Cheetah Flip Backward
879 1 87
923 1 145
871
4651899
868
1: 187
856
103
Hopper Hop
314 = 167
287 + 145
346 + 198
389 211
353
189
397 101
Hopper Stand
845 = 103
859 + 97
834 121
821 109
798
+ 123
832 112
Hopper Hop Backward
290 97
381 - 109
361 81
267 65
376
54
387 = 112
Walker Run
713 1.39
787
+ 37
683 + 151
697 63
675
+ 71
765 + 63
Walker Stand
987 + 40
991
29
961
+ 39
974 + 78
996
- 67
979
+ 33
Walker Walk
965 + 35
979 67
958 + 29
945 #1 71
987
37
969 51
//Sparse reward tasks
Mean success rale
Cheetah Jump Sparse
.0 .
.38
.09
: .03
31 .12
.68
+ .13
Cheetah Reach Sparse
.0 = .0
.18
.33 08
.53
133=-15
.15
.93
.16
.12
Hopper Jump Sparse
.0 .0
.53 .14
.08
.51
.33
.87
.12
.91
Hopper Reach Sparse
.0= 0
.37
.16
166 07
.65
.18
.93
+
.94
.04
Walker Jump Sparse
_0 0
.13 113 022
.05
1 01
.14
.08
.68
.12
.87
1
.09
Walker Escape Sparse
.0 .0
.09 .02
.67
4
14
.95
.17
.87
.12
.97
1
.03
Table 4: Comparison of fine-tuning DMControl. Models are pre-trained on Cheetah, Hopper, and Walker. and subsequently fine-tuned respective downstream tasks. The 'sparse' denotes reward is sparse. APT significantly outperforms baselines in most sparse reward tasks.
18

---

Under review as a conference paper at ICLR 2021
E COMPARISON ON ATARI GAMES
Table 5: Comparison of raw scores of each method on Atari games. Results are averaged over five random seeds. denotes dense reward hard exploration games. denotes sparse reward hard exploration games. @.V represents the of RL interaction utilized fine-tuning phase.
Atari game
Human
Random
V 100k
APT@0
APT@ 100k (ours)
Alien
7127.7
227.8
364.4
287.7
2614.8
Amidar
1719.5
5.8
286.0
256.8
1231.4
Assault
742.0
222.4
1209.1
345.1
891.5
Asterix
8503.3
210.0
6216.7
234.8
185.5
Asteroids
47388.7
719.1
4443.3
45.6
678.7
Atlantis
29028.1
12850.0
140542.8
3451.9
40231.0
BankHeist
753.1
14.2
200.3
267.7
456.7
BattleZone
37187.5
2360.0
7072.7
3491.8
7075.1
BeamRider
16826.5
363.9
1741.9
1348.3
3487.2
Berzerk
2630.4
123.7
491.4
120.8
493,4
Bowling
160.7
23.1
21.2
-16.9
-5.55
Boxing
12.1
0.1
43.0
33.7
21.3
Breakout
30.5
1.7
397.9
231.8
480.9
Centipede
12017.1
2090.9
7184.9
3678.1
6233.9
Command
7387.8
881.0
800.8
98.7
317.0
CrazyClimber
35829.4 .4
10780.5
49373.9
1897.9
4128.0
Defender
18688.9
2874.5
15876.1
1154.3
5927.9
DemonAttace
1971.0
152.1
8994.9
945.8
7771.8
DoubleDunk
-16.4
-18.6
-22.6
-10.8
-17.2
Enduro
860.5
0.0
-3.1
-10.9
-0.3
FishingDerby
-38.7
-91.7
993.9
-5.9
-1.6
Freeway
29.6
0.0
-2.1
12.1
29.9
Frostbite
4334.7
65.2
230.9
771.3
2196.1
Gopher
2412.5
257.6
1298.6
1298.6
8190.4
Gravitar
3351.4
173.0
328.1
228.6
542.0
Hero
30826.4
1027.0
663.5
363.9
6789.1
IceHockey
0.9
-11.2
-18.1
-16.4
-30.1
Jamesbond
302,8
29.0
484.4
10.3
356.1
Kangaroo
3035.1
52.0
1761.9
871.9
412.0
Krull
2665.5
1598.0
3142.5
849.8
2312.0
KungFuMaster
22736.3
258.5
16754.9
3871.9
17357.0
MoniezumaRevengg
4753.3
0.0
0,0
1.3
147.0
MsPacman
6951.6
307.3
558.5
403.5
2427.1
NameThisGame
8049.0
2292.3
2605.8
698.5
1387.2
Phoenix
7242.6
761.4
7162.23
2871.9
3874.2
Pitfall
6463.7
229,,4
-370.8
-54.9
0.8
Pong
14.6
-0.77
-26.2
-2188
-8.0
PrivateEye
69571.3
24.9
428.3
-54.9
556.1
Obert
13455.0
163.9
666.3
459.8
17671.2
Riverraid
17118.0
1338.5
5422.2
3871.9
4671.0
RoadRunner
7845.0
11.5
6146.7
987.9
4782.1
Robotank
11.9
2.2
10.0
9.1
13.7
Seaquest
42054.7
68.4
706.6
1891.4
2116.7
Skiing
-4336.9
17098.
4312.4
-29819.4
-3843..1
Solaris
12326.7
1236.3
841.5
367.9
1925.8
SpaceInvaders
1668.7
148.0
9741.0
1389.7
3687.2
StarGunner
10250.0
664.0
25827.5
109.8
8717.0
Surround
6.5
-10.0
-15.5
-5.9
-4.5
Tennis
-8.3
23..8
0.7
-8.7
1.2
TimePilot
5229.2
3568.0
4503.6
968.1
1567.0
Tutankham
167.6
11.4
50.7
78.8
124.6
UpNDown
11693.2
533.4
17037.6
81.1
8289.4
Venture
1187.5
0.0
-1.7
13.8
231.0
VideoPinball
17667.9
0.0
35120.3
989.5
2817.1
WizardOrWor
4756.5
563.5
1453.3
746.9
3465.0
RersRevenge
54576.9
3092.9
5543.5
719.3
1871.5
Zaxxon
9173.3
32.5
1092.5
518.6
5431.0
19

---

Under review as a conference paper at ICLR 2021
F EXPERIMENT DETAILS
F.1 IMAGENET BASED PRE-TRAINING
In Figure ! the ImageNet (Deng al. 2009) pre-trained model is based on MoCo (He et al. 2019
The policies are represented by the Impala convolutional residual network as in (Espeholt et al. 2018 with the LSTM (Hochreiter & Schmidhuber (997) part excluded. Images sampled from ImageNet are downsampled 84 x 84, followed by frame-stacking and data augmentation. We experiment with the data augmentation methods used in He et al. (2019); 99;; Chen et al.. (020c) and the simpler random crop used in RL (Kostrikov et al., 2020). Results on DMControl shown no benefit comes from ImageNet unsupervised pre-trained models. To improve the quality of pre-trained representations, we consider initializing the filters in the first layer with weights from the model of et al. (2016) which trained on ImageNet classification. Similar to fine-tuning MoCo trained models, we observe no difference in performance between fine-tuning supervised pre-trained models observed and training from scratch.
F.2 GENERAL IMPLEMENTATION DETAILS
The encoder network f is a ReLU convolution neural network followed by full-connected layer normalized by LayerNorm [B et al. 2016) and tanh nonlinearity applied to the output of fully- connected layer. The data augmentation a simple random shift which has been shown effective in visual domain RL in DrQ (Kstrikov al. 2020) and RAD (Lsskin et al. 2020). Specifically, the images are padded each side by 4 pixels (by repeating boundary pixels) and then select a random 84 84 crop, yielding the original image. This procedure is repeated every time image is sampled from the replay buffer. The learning rate of contrastive learning is 0.001, the temperature is 0.1. We incorporate the memory mechanism (with a moving average of weights for stabilization) from He et al. 2019;; Chen et al. (2020c). We use DrQ as the RL optimization algorithm at both pre-training phase and fine-tuning phase. The batch size of contrastive learning 1024, the batch size of RL optimization is 512. The pre-training phase consists of 5M environment steps on DMControl and 250M environment steps on Atari games. The replay buffer size is 100K. The projection network two-layer MLP with hidden size of 128 and output size of 64. All hyperparameters are included Table and Table S
F.3 MULTI-ENVIRONMENT PRE-TRAINING DETAILS
For the experiments of multi-environmee separate replay buffer for each of the three environments, and compute environment specific loss using data sampled from its own replay buffer. The contrastive loss and RL loss are then summations of every environment specific loss.
pre-training (results shown in Figure
one
F.4 DMCONTROL HYPERPARAMETERS
Environment name
Action repeat
Cheetah
Walker
Hopper
Table 6: The action repeat hyper-parameter used for each environment.
F.5 ATARI HYPERPARAMETERS
For the experiments in Atari 100k experiments, we largely reuse the hyper-parameters from DrQ (Kottrkkvv et al. 2020). The evaluation is done for 125k environment steps at the end of training for 100k environment steps.
20

---

Under review as a conference paper at ICLR 2021
Algorithm 3 Pseudo-code of Unsupervised Active Pre- Training.
f: encoder network shared between actor & critic g: projection network for contrastive learning
k: hyperparameter of kth-nearest neighbor
learning pre- pre-trained actor-critic while iterations so far < pre- re--trining iterations do
Unsupervised pre-training phase:
Interact with a reward-free environment and collect transitions and save to replay buffer // sample batch of transitions from replay buffer and compute contrastive loss
Compute contrastive loss using equation (I) I/ update and g to minimize contrastive loss
Contrastive loss. baskwaackward() (f param, g-param)
I/ compute intrinsic reward for each transition
Compute representation 3/++1 for each Ut+11 computing &NN.
Compute r dz lt+1) = log(1 W++1 J [12 + 1) (equation 5
// optimize reward via off-policy RL Compute expected reward // update RL agent to maximize expected reward Update param,, critic.param) end while
Supervised fine-tune phase: fine-tuning both actor and critic
// If fine-tune both representation and RL agent, initialize encoder and actor-critic from pre-trained models.
// If fine-tune representation only, initialize encoder from pre- pre-trained models and randomly initialize actor and critic.
while iteration number S max fine-tune iterations do Interact with a task-specific environment and collect transitions and save to replay buffer Sample a batch of transitions
// fine-tune representation or fine-tune both representation and RL agent initialization Update(f. param, actor acrom, critic.paramy end while
21

---

Under review as a conference paper at ICLR 2021
Parameter
Setting
Data augmentation
Random shifts
Frames stacked
3
Action repetitions
Table 6
Replay buffer capacity
100000
Random steps (fine-tuning phase)
1000
RL minibatch size
512
Contrastive learning minibatch size
512
K-Nearest Neighbors K value
3
Discount 7
0.99
/c/oonrrastiee learning optimizer
Adam
RL learning rate
10-3
Contrastive learning rate
10
Learning rate schedule
cosine
Temperature Shared encoder: channels
0.1
32, 32, 32
Shared encoder: filter size
3,3 3,3 x
Shared encoder: stride
2,2,2,
Actor update frequency
2
Actor log stddev bounds
[-10, 2]
Actor: hidden units
1024
Actor: layers
3
Init temperature
0.1
Critic Q-function: hidden units
1024
Critic target update frequency
2
Critic Q-function soft-update rate
0.01
Non-linede Noneriiyy
ReLU
Table 7: Hyper-parameters the DeepMind control suite experiments.
22

---

Under review as a conference paper at ICLR 2021
Parameter
Setting
Data augmentation
Random shifts and Intensity
Grey-scaling
True
Observation down-sampling
84 x 84
Frames stacked
4
Action repetitions
4
Reward clipping
[-1, .]
Terminal on loss of
True
Max frames per episode
108k
Update
Double Q True 1
Dueling
Target network: update period
Discount factor
0.99
Minibatch size
32
RL optimizer
Adam
RL optimizer (pre-training): learning rate
0.0001
RL optimizer (fine-tuning: learning rate
0.001
RL optimizer: 31
0.9
RL optimizer: 83
0.999
RL optimizer:
0.00015
K-Nearest Neighbors K value
3
Contrastive learning optimizer
Adam
Contrastive learning rate
10-3
Learning rate schedule Temperature
inne
0.1
Max gradient norm
10
Training steps Evaluation steps
100k
125k
Min replay size for sampling
1600
Memory size
Unbounded 1 step
Replay period every
Multi-step return length
10
Q network: channels
32, 64, 64
Q network: filter size
x 8, x 4, 3 x
network: stride
4, 2, 1
Q network: hidden units Non-linearity Exploration
512
ReLU e-greedy 2500
c-decay
Table 8: Hyper- parameters in the Atari suite experiments.
23


Reviews:
[{"review_id": "iVLp-mK7bXp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission presents a technique for unsupervised pre-training of representations and policies for RL. Unsupervised representation learning has obtained impressive results in supervised scenarios, and adapting these methods to RL is an important research direction. One of the main challenges in the RL setting is that of defining the distribution of data to learn from, as well as sampling from it. The learned representations are unlikely to be useful for observations that are out of the pre-training distribution, so it is desirable to perform representation learning on data that is representative of the full state space. Previous works (Hazan et al., Lee et al.) proposed strategies to train agents that induce maximally entropic state visitation distributions, but they involve density estimation whose underlying assumptions are not well suited for pixel observations. The authors propose to overcome these limitations by using a particle based entropy estimate in the learned representation space. The pre-trained representations and policies can be used for RL from pixels, obtaining faster convergence and higher end scores than the considered baselines in both DMControl and Atari.\n\nThe method is novel and the experimental results are strong. The paper includes a proper literature review and it is generally easy to follow. However, it would benefit from a more detailed analysis in order to understand where the reported gains come from. The ablation studies suggest that most of the gains are due to the learned representations: fine-tuning the policy as well only provides slightly faster convergence in two Hopper tasks, but the end performance is the same as when transferring representations only. I would like to suggest the following:\n- Reporting the performance of zero-shot transfer for all experiments, i.e. the average return of the pre-trained policy in each task (APT@0). I suggest doing this for both DMControl and Atari experiments.\n- It would be very helpful to see per-game scores in Atari in order to understand what type of environments benefit the most from pre-training. Gains could come from faster convergence on dense reward games thanks to the pre-trained representations, or from higher end performance on hard exploration tasks due to the exploratory behavior of the pre-trained policy. This is important given the fact that APT obtains much higher median human normalized scores than VISR while reaching much lower mean scores.\n- Related to the previous point, comparing full APT and APT (representation) in a selection of Atari games requiring different degrees of exploration would provide insight on the type of challenges APT is addressing.\n- APT (representation) can be understood as pre-training a state representation. How does this version of APT compare to RL from true state on DMControl? A similar analysis was reported by Srinivas et al. (Figure 7, see full reference below).\n\nPlease note that most of the suggestions above do not require running new experiments and can be addressed by providing additional information about the ones that are already reported in the paper.\n\n\nOther comments:\n- To the best of my knowledge, there doesn’t exist a standard procedure for fine-tuning policies, especially for actor-critic architectures such as the one in SAC. It would be helpful to include a description on how this is performed. Is the critic fine-tuned as well, or is it initialized from scratch?\n- Section 3 mentions a robustness analysis showing the impact of varying k in kNN, but I didn’t see it in the paper. Is it referring to initial experiments, or to some ablation study that was not included for some reason?\n- SimCLR fits representations using an objective that is based on cosine similarity, but the reward derived from the particle-based entropy estimate employs L2 distance. Is there a reason for this discrepancy?\n- The subtitle “Supervised training @100k” in Table 1 is not very accurate, as two out of five rows use more than 100k interactions. I suggest using “Fully-supervised training” as the title and appending “@100k” to the name of the first three methods.\n- A single unsupervised pre-training can be leveraged to solve multiple tasks as long as the environment does not change, as shown in the DMControl experiments. I believe this is one of the most appealing properties of the method and might not be highlighted enough in the paper.\n\n\nThe following works are relevant and might be worth citing:\n- Laskin, Michael, Aravind Srinivas, and Pieter Abbeel. \"Curl: Contrastive unsupervised representations for reinforcement learning\". ICML 2020.\n- Lee, Lisa, et al. \"Efficient exploration via state marginal matching\". arXiv preprint arXiv:1906.05274 (2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "IKCjrk4jHC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a pre-training scheme (APT) for RL with two components: contrastive representation learning and particle based entropy maximization. Experiments are done in DeepMind Control Suite (DMControl) and Atari suite to show improved performance and data efficiency. \n\n**Strength**: Pre-training good representations and policy initialization without reward is obviously an important direction in RL. The proposed method is conceptually simple and intuitive, yet achieves some promising results. Particle based entropy estimation (eq. (5)) can be a simple and interesting solution to estimating observation entropy in high dimension space.\n\n**Weakness**: Some concerns about experiment results:\n- For DMControl results, it seems APT only improves significantly over \"From scratch\" and \"Count-based pre-training\" schemes when reward is sparse? I also think the result can be made much stronger if more powerful baselines can be adopted, e.g. DIAYN, VISR, etc.\n- For Atari results, APT versus VISR is interesting. APT achieves better median normalized scores but VISR achieves better average normalized scores. Is it possible to show a game breakdown to understand where these methods work better?\n- In section 4.3, it is awkward that I find APT and APT (representation) similar across different curves in Figure 4, so I'm not convinced the policy initialization is useful. Maybe some more results (e.g. on Atari) can help with this. \nAlso see the **Novelty** part below.\n\n**Novelty**: The two cores of the paper, contrastive representation learning and entropy maximization, are quite established in RL. Particle based entropy estimation is from prior work, but I think it is fairly novel and interesting in the RL domain (despite its usefulness being doubted). \n\n**Clarity**: I think the paper is written clearly in general.\n\n**Question**: Conceptually eq. (5) looks quite noisy (dependent on batch samples a lot), and like some form of contrastive learning objective still (maybe entropy maximization is equivalent to making representation contrastive?). So is it really a stable reward signal? Is it possible to just use eq. (1) as reward also?", "extra_scores": {}, "extra_text": {}}, {"review_id": "2KKrTYFYHIb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*SUMMARY*\n\nThe paper proposes a method to simultaneously learn effective representations and efficient exploration in a reward-free context. The algorithm iterates between minimizing a contrastive loss and maximizing an intrinsic reward derived from a k-NN entropy estimation of the state distribution. Then, authors empirically evaluate the method over a set of visual Mujoco tasks and Atari games.\n\n*STRENGTHS*\n- the paper addresses a very relevant reward-free exploration objective as a preprocessing to RL\n- the paper combines representation learning and state entropy maximization into a promising practical method\n\n*WEAKNESSES*\n- the paper might be too incremental with respect to previous (albeit unpublished) work\n- the paper somewhat fails to empirically illustrate and validate the reward-free phase\n\n*EVALUATION*\n\nUnfortunately, over some concerns regarding the novelty of the presented method and its experimental validation, which I find somewhat weak for an essentially empirical work, I would lean towards rejecting the paper.\n\n*DETAILED COMMENTS*\n\nC1) The exploration component of APT has striking similarities with the method in [1], which also seeks the optimization of a k-NN estimate of the state distribution entropy in a reward-free context. While it would be in general acceptable to overlook unpublished work, I think that the connections with [1] are too many to avoid a deeper discussion over distinctive contributions. The method in [1] does not seem to learn representations, but I am wondering if this contribution alone would be substantial enough.\n\nC2) The paper does not present an explicit empirical evaluation of the reward-free phase, thus I am not sure on how APT is performing in entropy maximization. Especially, what is the impact of the three sources of bias that are introduced in the entropy estimation (avoiding bias correction and constants, avoiding importance weighting, scaling distances with the standard deviation)?\nCan authors present state entropy plots, and possibly compare the performance of APT with other methods seeking a similar objective, such as MaxEnt (with representation learning), SMM [2] or MEPOL [1].\n\nC3) I have some concerns on the theoretical and practical implications of considering a reward function that is actually depending on the current policy. It is sound to fit a value function for a reward of this kind? Maximizing an ever-changing reward might cause instability and prevent convergence?\n\nC4) The benefit of employing a non-parametric method to estimate the entropy of high-dimensional inputs is clear, since density modeling would be quite hard. Could density modeling over a reduced latent space be a viable option instead?\n\nC5) I would argue that the idea of simultaneously learning representations and exploration is the main selling point of the presented method, since maximizing the state entropy might help learning superior representations and viceversa. But from the ablation study in Section 4.3 this conclusion does not arise naturally, as learning representations alone seems almost as good as learning both. May I ask the authors to clarify this point?\n\nC6) The scores on Atari games are reported without confidence intervals, leaving some doubts over the statistical significance of the results. Moreover, it is not completely clear from the aggregate performance where and how APT is helping in these experiments. Can authors provide a deeper explanation on why the original performance of SimPLe and VISR is not reproducible?\n\n[1] Mutti and Restelli. A policy gradient method for task-agnostic exploration. Arxiv, 2020.\n[2] Lee et al. Efficient exploration via state marginal matching. Arxiv, 2019\n\n*QUESTIONS*\n\nMay the authors address the comments listed above in their response?\n\n*ADDITIONAL FEEDBACK*\n- For the Atari experiments I would suggest to focus more on hard-exploration games, such as Montezuma or Pitfalls, instead of providing just an aggregate performance over full sets of games. It would be nice to include some visualizations and interpretations on the behavior that APT learns in the reward-free phase.\n- I believe that the multi-environment pre-training setting is quite interesting and, to the best of my knowledge, completely novel. The results are promising, and I would suggest to include this setting, together with a deeper analysis, in the main paper.\n- Dashing the lines in the reported plots would help visibility, especially without colors.\n- typos and rephrasing:\n\t- when referring to the environment I would use reward-free instead of task agnostic (e.g., page 2, paragraph 3)\n\t- the 26 games subset instead of the 100k subset (page 7, last lines)\n\t- I would rephrase \"The notable difference is that APT (representation) decouples the action space dimension from pre-trained models\" which is not crystal clear (Section 4.3). \n\n\n####################\n\nAFTER RESPONSE\n\nI would like to thank authors for their detailed response and for their effort in improving the paper according to reviewers' suggestions. Unfortunately, after authors clarifications, I still have some doubts on the concerns raised with C1 and C2 (see below). Thus, I am keeping a slightly negative evaluation for this paper. \n\nAuthors claim that the main benefit of APT over a prior work method (MEPOL, Mutti er al., 2020) is a lower variance of the gradient estimation, thanks to the choice of avoiding importance weights corrections. However, in Figure 6 MEPOL does not seem to suffer a particularly high-variance. To me, the most likely reason for the improved performance is that APT guarantees an action-level feedback as opposed to a trajectory feedback (see [1]).\nHowever, I am still skeptical about this action feedback: the reward-to-go becomes non-Markovian and Bellman equations does not hold anymore (see [2]). This casts some doubts on the actor-critic procedure APT employs to optimize the rewards. Authors may have a good point on the notion that the encoder is breaking the dependence between policy and rewards, but I think the topic warrant some additional discussion.\n\nI would suggest the authors to rephrase this work to give a more central role to the scalability to high-dimensional observations, which I believe is the main contribution of the paper, and to include a more thorough discussion of (Mutti et al., 2020) in the main text (beyond the related work section).\n\n[1] Efroni et al. Reinforcement learning with trajectory feedback. Arxiv, 2020.\n\n[2] Zhang et al. Variational policy gradient method for reinforcement learning with general utilities. NeurIPS 2020.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a particle-based entropy estimate for intrinsic motivation for pre-training an RL agent to then perform in an environment with rewards. As the reviewers discussed, and also mentioned in their reviews, this paper bears stark similarity to work of 5 months ago, presented at the ICML 2020 Lifelong ML workshop, namely, "A Policy Gradient Method for Task-Agnostic Exploration", Mutti et al, 2020--MEPOL. What is novel here is the adaptation of this entropy estimate to form an intrinsic reward via a contrastive representation and the subsequent demonstration on standardized RL environments.  The authors have added a comparison to MEPOL, and in these experiments, APT outperforms this method, sometimes by some margin. Unfortunately this work does not meet the bar for acceptance relative to other submissions.

Decision: Reject