Behavior From the Void: Unsupervised Active Pre-Training
Authors: Hao Liu
Pieter Abbeel
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: unsupervised exploration
unsupervised pretraining
reinforcement learning

Abstract: We introduce a new unsupervised pre-training method for reinforcement learning called APT, which stands for Active Pre-Training. APT learns behaviors and representations by actively searching for novel states in reward-free environments. The key novel idea is to explore the environment by maximizing a non-parametric entropy computed in an abstract representation space, which avoids challenging density modeling and consequently allows our approach to scale much better in environments that have high-dimensional observations (e.g., image observations). We empirically evaluate APT by exposing task-specific reward after a long unsupervised pre-training phase. In Atari games, APT achieves human-level performance on 12 games and obtains highly competitive performance compared to canonical fully supervised RL algorithms. On DMControl suite, APT beats all baselines in terms of asymptotic performance and data efficiency and dramatically improves performance on tasks that are extremely difficult to train from scratch. 

and natural Behavior From the Void: Unsupervised Active Pre-Training
Hao Liu UC Berkeley hao. liu@cs. berkeley. edu
Pieter Abbeel
UC Berkeley pabbeel@cs beebeel@cs.berreee berkeley. edu
Abstract
We introduce a new unsupervised pre-training method for reinforcement learning called APT, which stands for Active Pre- Traininn APT learns behaviors and representations by actively searching for novel states reward-free environments. The key novel idea is to explore the environment by maximizing non-parametric entropy computed in an abstract representation space, which avoids challenging density modeling and consequently allows our approach scale much better in environments that have high-dimensional observations (e.g., image observations). We empirically evaluate APT by exposing task-specific reward after long unsuper- vised pre-training phase. In Atari games, APT achieves human-level performance 12 games and obtains highly competitive performance compared canonical fully supervised RL algorithms. On DMControl suite, APT beats all baselines in terms of asymptotic performance and data efficiency and dramatically improves performance on tasks that are extremely difficult to train from scratch.
Introduction
Reinforcement learning (RL) provides a general framework for solving challenging sequential decision-making problems. When combined with function approximation, has achieved remarkable success in advancing the frontier of AI technologies. These landmarks include outperforming humans in computer games 40
5 and solving complex robotic control tasks M. Despite these successes, they have to train from scratch to maximize extrinsic reward for every encountered task. This is in sharp contrast with how intelligent creatures quickly adapt to new tasks by leveraging previously acquired behaviors. Unsupervised pre- pre-training, a framework that trains models without expert supervision, has obtained promising results in computer vision 43 23
language modeling [63] 16, [.. The learned representation, when fine-tuned on the downstream tasks, can solve them efficiently in few-shot manner. With the models and datasets growing, performance continues to improve predictably according to scaling laws.
Driven by the significance of massive unlabeled data, consider an analogy setting of unsupervised pre-training in computer vision where labels are removed during training. The goal of pre-training to have data efficient adaptation for some downstream task defindd the form of rewards. In RL with unsupervised pre-training, the agent is allowed to train for long period without access to environment reward, and then only gets exposed the reward during testing. We first test an array of existing methods for unsupervised pre-training to identity which gaps and challenges exist, we evaluate count-based bonus 100, which encourages the agent to visit novel states. We apply count-based bonus to DrQ K3 which is current state-of-the-art RL for training from pixels. We also evaluate ImageNet pre-trained representations. The results are shown in Figure [ We can that count-based bonus fails to outperform train DrQ from scratch. We hypothesize that the ineffectiveness stems from density modeling at the pixel level being difficult. ImageNet pre-training does not outperform training from scratch either, which has also been shown in previous research
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

in real world robotics [29]. We believe the reason is that neither of existing methods can provide enough diverse data. Count- based exploration faces the difficult of estimating high dimensional data density while ImageNet dataset is outtr---iitribution for DMControl.
To address the issue of obtaining diverse data for RL with unsupervised pre-training, we propose to actively collect novel data by exploring unknown ar- cas in the task-agnostic environment. The underlying intuition is that a general exploration strategy has
Hopper Hop
400
Episode Retur
300
to visit, with high probability, any state where the agent might be rewarded in a subsequent RL task.
200
Algo DrQ from scratch
Concretely, our approach relies on the entropy maxi- mization principle [27 53. Our hope is that by doing so, the learned behavior and representation can be trained on the whole environment while being as task agnostic as possible. Since entropy maximization in high dimensional state space is intractable as an oracle density model is not available, we resort to the particle-bas based entropy estimator [55] 8]. This estimator is nonparametric and asymptotically un- biased. The key idea is computing the average of the Euclidean distance of each particle to its near- est neighbors for set of samples. We consider an abstract representation space in order to make the distance meaningful. To learn such a representation space, we adapt the idea of contrastive representation learning [14] to encode image observations to a lower dimensional space. Building upon this insight, we propose Unsupervised Active Pre-Trainin Training (APT) since the agent is encouraged to actively explore and leverage the experience to learn behavior.
100
ImageNel pre-train
Count based bonus
APT (ours)
0.0
1.5 Supervised Environment Steps (x109)
0.5
1.0
2.0
Comparison of state-of of tatte-of-the-art the-art pixel- based RL with unsupervised pre-training. APT (ours) and count-based bonus (hoth based on DrQ 3 ) are trained for long unsupervised pe- riod (5M environment steps) without access to en- vironment reward, and then gets exposure to the en- vironment reward during testing. APT significantly outperform training DrQ from scratch, count-based bonus, and ImageNet pre-trained model.
Figure 1:
Our approach can be applied to a wide-range of existing RL algorithms. In this paper we consider applying our approach to DrQ [33 which is of visual RL algorithm. On the Atari 26 games subset, APT significantly improves DrQ's 's data- lltaaeffficienc,, achieving 54% relative improvement. On the full suite of Atari 57 games (40, APT significantly outperforms prior state-of-the-arr, achieving a median numan-normalinn RL methods and DQN. On DeepMind control suite, APT beats DrQ and unsupervised RL in terms of asymptotic performance and data efficiency and solving tasks that are extremely difficult to train from scratch. The contributions of our paper can be summarized as: (i) We propose new approach for unsupervised pre-training for visual RL based a nonparametric particle-based entropy maximization. (ii) We show that our pre-training method significantly improves data efficiency of solving downstream tasks on DMControl and Atari suite.
score 3x higher than the highest score achieved by prior unsupervised
2 Problem Setting
Reinforcement Learning (RL) An agent interacts with its uncertain environment over discrete timesteps and collects reward per action, modeled as a Markov Decision Process (MDP) [48], defined by (S, T, Po: 7) where S C R is a of nssdimmnsionnal states, A C I is set of n- dimensional actions, T S x A - [0, 1] is the state transition probability distribution. Po S -> [0, 1] is the distribution over initial states, S x A + R is the reward function, and y [0, is the discount factor. At environment state E S, the agent take actions a E A. in the (unknown) environment dynamics defined by the transition probability T T [ss a,))), and the reward function yields a reward immediately following the action a, performed in state St. We define the discounted return G(st, ax) Σi= r'((+ oc++) as the discounted sum of future rewards collected by the agent. In value-based reinforcement learning, the agent learns an estimate of the expected discounted return, a.k.a, state-action value function Q" (s;,
Ar+1..... D Y'r(S--,, d++:)]. A common way of deriving a new policy from acatonaction value function Is to act e-greedily with respect to the action values (discrete) or to use policy gradient to maximize the value function (continuous).
Unsupervised Pre- Pre-Training RL In pretrained RL, the agent is trained in a reward-free MDP (S, So, A, T, 9) for long period followed by a short testing period with environment rewards IR provided. The goal is to learn pretrained agent that can quickly adapt to testing tasks defined

---

where b(k) is a bias correction term that only depends on the hyperparameter k. and va is the volume (k) the by rewards to maximize the sum of expected future rewards in a zero-shot or few-shot manner. This is also known as the two phases learning in unsupervised pretraining RL (20]. The current state-of-the-ar methods maximize the mutual information (7) between policy-conditioning variable (w) and the behavior induced by the policy terms of state visitation (s).
max (s; w) max H (w) H (w]s),
where w is sampled from a fixed distribution in practice as in DIAYN 17 and VISR 20 The ((((s)))
objective can then be simplified as max H [w]s). Due being intractable to directly maximize this negative conditional entropy, prior work propose to maximize the variational lower bound of the negative conditional entropy instead [7]. The training then amounts to learning posterior of task
variable conditioning on states g(w[s).
- -H (wls) 2 Es [log (ww]s)].
Despite successful results learning meaningful behaviors from reward-free interactions [e.g. 21 17 (2)]. these methods suffer from insufficient exploration because they contain no explicit
exploration.
Another category considers the alternative direction of maximizing the mutual information 12 max (s; w) max (s) (s]w).
This intractable quantity can be similarly lowered bound by variational approximation 7
[s;; w) 2 L. [go(s|w)] - E. [[o]](s)],
where E, p(s) can then be approximated by posterior of state given task variables E, (log((s)) & E.,. [log [s|[w)]. Despite their successes, this category of methods do not ex- plore sufficiently since the agent receives larger rewards for visiting known states than discovering new ones as theoretically and empirically evidenced by Campos al. 12 In addition, they have only been shown to work from explicit state-reprssentations and it remains unclear how to modify to learning from pixels.
In the next section, we introduce a new nonparametric unsupervised pre-training method for RL which addresses these issues and outperforms prior state-of-the-are on challenging risual-domain RL benchmarks.
3 Unsupervised Active Pre-Training for RL
We want to incentivize the agent with a reward r. to maximize entropy in an abstract representation space. Prior work maximizing entropy relies estimating density of states which is challenging and non-trivial, instead, we take a two-step approach. First, we learn a mapping fe : R - R that maps state space an abstract representation space first. Then, we propose particle-based nonparametric approach to maximize the entropy by deploying state-of-the-arl RL algorithms.
We introduce how to maximize entropy via particle-based approximation in Section and describe how to learn representation from states Section 3.2
3.1 Particle-Band -Base Entropy Maximization
Our entropy maximization objective is built upon the nonparametric particle-based entropy estimator proposed by Singh al. [55] and Beirlant (8) and has has been widely studied in statistics (28). Its key idea is to measure the sparsity of the distribution by considering the distance between each sampled data point and its & nearest neighbors. Coneretely, assuming have number of n data points {2, fi=1 from some unknown distribution, the particle- approximation can be written as
k
H ppatticte (2)
log nv
+ b(k))
x log v
72
of the hypersphere of radius ||> - 2 (80 between 2; and k-th nearest neighbor 2 Fuclidean distance.
v |
(k) (1)nn 22/2
(2)
I T(nz/2+1
3

---

Expected Reward
Contrastive Loss
Minibatch representations
busec Entropy
Maximation
Normalization
representation
representation
Projection
Encoder
K ---nrrestt neighhors
dlistance
Reward
observations
representation
Figure Diagram the proposed method APT. On the left shows objective of APT, which to maximize the expected reward and minimize the contrastive loss. The contrastive loss learns an abstract representation from observations induced by the policy. We propose particle- based entropy maximization based reward deploy stateoo--heeart RL methods to maximize entropy in abstraction space of the
function such that induced by the policy, On the right shows the idea of our particle-bat based entropy, which measures the distance between each data point and nearest neighhors.
where I' is the gamma function. Intuitively, vē reflects the sparsity around each particle and equa- tion ① is proportional to the average of the volumes around each particle.
By substituting equation (2) into equation (
we can simplify the particle-based entropy estimation as sum of the log of the distance between each particle and its k-th nearest neighbor.
H Hpaiicte (2) x Σ log = - 11
(3)
Rather than using equation (3) as the entropy estimation, we find averaging the distance over all nearest neighbors leads to more robust and stable result, yielding our estimation of the entropy.
Hputtcct (2) = Σ log
c
Σ
2 - 100
(4)
k
1=1
(N+(
where N& denotes the l nearest neighbors around particle, c a constant for numerical stability (fixed to 1 in all our experiments).
We can view the particle- based entropy in equation 4 as an expected reward with the reward function being r r2+)) log (c + *. E EN,, 2 2 ||=) for each particle 2,. This makes it possible to
deploy RL algorithms to maximize entropy, concretely, for batch of transitions {(s, a, s') sampled from the replay buffer. We consider the representation of each s' as particle in the representation space and the reward function for each transition is given by
(18,,,8) log
c+ T
Σ
110(s)) - -00
(5)
=()ENN(==f(())
In order to keep the rewards on a consistent scale, we normalize the intrinsic reward by dividing it by running estimate of the mean of the intrinsic reward. See Figure for illustration of the formulation.
3.2 Learning Contrastive Representations
Our aforementioned entropy maximization is modular of the representation learning method we choose to use, the representation learning part can be swapped out for different methods if necessary. However, for entropy maximization to work, the representation needs to contain a compressed repre- sentation of the state. Recent work, CURL [35, ATC (56) and SPR (52). show contrastive learning (with data augmentation) helps learn meaningful representations RL. We choose contrastive repre- sentation learning since it maximally distinguishes an observation 8t, from alternative observations 81, according to certain distance metric in representation space, we hypothesize is helpful for learning meaningful representations for nearest neighbors based entropy maximization. Our contrastive

---

learning is based on the contrastive loss from SimCLR [4), chosen for its simplicity. We also use the same set of image augmentations as DrQ 33 consisting of small random shifts and color jitter. Concretely, we randomly sample batch of states (images) from the replay buffer {si )==1- For each state 8;, we apply random data augmentation and obtain two randomly augmented views of the same state, denoted as key s = aug (s;) and query 8 aug(s;). The augmented observations are encoded into a small latent space using the encoder = fo(-) followed by deterministic projection h-(-) where contrastive loss is applied. The goal of contrastive learning to ensure that after the encoder and projection, of is relatively more close to of than any of the data points {s 83 1o
min 8.0
log
2n
E
(lu + tepp(hs(
Following DrQ, the representation encoder fo(-) is implemented by the convolutional residual network followed by a fully-connected layer, LayerNorm and Tanh non-linearity, We decrease the output dimension the fully-coneected layer after the convnet from 50 15. We find it helps to use spectral normalization [39] to normalize the weights and use ELU [15 as the non- li-eaiieartty in between convolutional layers.
Table positions our new approach with respect to existing ones. Figure shows the resulting model. Training proceeds in other algorithms maximizing extrinsic reward: by learning neural encoder and computing intrinsic reward r and then trying to maximize this intrinsic return by training the policy. Algorithm shows the pseudo-code of APT, we highlight the changes from DrQ to APT in color.
Algorithm 1: Training APT
Randomly Initialize f encoder
Randomly Initialize т and Q networks for 1,30 do
for : 1,7 do
Receive observation Se from environment
Take action ~ ot (-188),, receive observation 8 and xg. from environment
DU (st, ae, X5 s()
[(st, ai, Re, +4)
D
// sample mini batch representation learning
Train neural encoder f mini batch
for each i= 1..N do
a ~ x(1s)) Q Qo (0 a()
Compute TAPT with equation
// particle-based entropy reward
y4 FAPT 2
end
lossQ E, (s: as) w:)
Gradient descent step on Q and
// standard actor-critic
end end
Table 1: Methods for pre-training RL in reward-free setting. Exploration: the method explore efficiently. Vi- sual: method works well visual RL. Off-policy: the method is compatible with off-policy RL optimization. means only in state-based RL. o(s) is count-based bonus. 4i(s, a): successor feature, d(s): state representation.
Algorithm
Objective
Visual
Exploration
Off-policy
Pre-Trained model
MaxEnt [22]
max H(s)
न (als)
CBB 110
max E, [c(s)]
n (a]s)
MEPOL 42
max H(s)
X
ा (aass
VISR 20
max - H(2ls)
/
(ss (())
DIAYN 17
max -H(2|s) H(alz,
X
SXX
V
n r(a)s,2)
DADS 55
max H(s) =(s]=)
/
π (a)88,) ( (8)2)
EDL 12
max H(s) - -H(s)2)
/
x (a)5,2)
APT
max H(s)
π (als), Q ((s0)
5

---

4
Related Work
State Space Entropy Maximization. Maximizing entropy of policy has been widely studied in
RL, from inverse RL [69] to optimal control [59 60 49 and actor-critic [19]. State space entropy
maximization has been recently used as exploration method by estimating density of states and maximizing entropy [22]. In Hazan et al. [22] they present provably efficient exploration algorithms
under certain conditions. VAE
32 based entropy estimation has been deployed in lower dimensional observation space [36]. However, due to the difficulty of estimating density in high dimensional
space such as Atari games, such parametric exploration methods struggle to work in more challenging domains. In contrast, our work turns to particle based entropy maximization in a contrastive representation space. Maximizing particle-based entropy has been shown to improved data efficiency in state-based RL as in MEPOL [42. However, MEPOL's entropy estimation depends on importance sampling and the optimization based on-policy RL algorithms, hindering further applications to challenging visual domains. MEPOL also assumes having access to the semantic information of the state, making it infeasible and not obvious how to modify it to work from pixels. In contrast, our method is compatible with deploying state-of-the-art off-policy RL and representation learning algorithms to maximize entropy. Nonparametric entropy maximization has been studied in goal conditioned RL (66]. Pitis et al. [47] proposes maximizing entropy of achieved goals and demonstrates significantly improved success rates in long horizon goal conditioned tasks. The work by Badia et al. [6] also considers k-nearest neighbor based count bonus to encourage exploration, yielding improved performance in Atari games. K-nearest neighbor based exploration is shown to improve exploration and data efficiency in model-based RL. [57]. Concurrently, it has been shown to be an effective unsupervised pre-training objective for transferring learning in RL [3] their large scale experiments further demonstrate the effectiveness of unsupervised pre-training.
Data Efficient RL. To improve upon the sample efficiency of deep RL methods, various methods have been proposed: Kaiser et al. (30 introduce a model-based agent (SimPLe) and show that it compares favorably to standard RL algorithms when data is limited. Hessel et al. 25). Kielak
B1I van Hasselt et al. [6]] show combining existing RL algorithms (Rainbow) can boost data efficiency. Data augmentation has also been shown to be effective for improving data efficiency in vision-based RL has been shown to boost data efficiency [52]. Combining contrastive loss with RL has been shown to improve data efficiency in CPC (24] despite only marginal gains. CURL 351 show substantial data-efficiency gains while follow-up results from Kostrikov et al. B33 suggest that most of the benefits come from its use of image augmentation. Contrastive loss has been shown to learn useful pretrained representations when training on expert demonstration [56), however in our work the agent has to explore the world itself and exploit collect experience.
34 33). Temporal contrastive learning combined with model-based learning
Unsupervised Pre- Training RL. A number of recent works have sought to improve reinforcement learning via the addition of an unsupervised pretraining stage, in which the agent improves its representations prior to beginning learning on the target task. One common approach has been to allow the agent a period of fully- uully-unsupervised interaction with the environment during which the agent trained to learn a set of skills associated with different paths through the environment, as in DIAYN [1]. RL (67. MUSIC [68]. APS (37], and VISR (20. Others have proposed to use self-supervised objectives to generate intrinsic rewards encouraging agents to visit new states, e.g., Pathak et al. (46) use the disagreement between ensemble of latent-space dynamics models. However, our work trained to maximize the entropy of the states induced by the policy. By visiting any state where the agent might be rewarded in subsequent RL task, our work performs better or comparably well as other more complex and specialized state-of-the-an methods.
5 Results
We test APT in DeepMind Control Suite [DMControl::58 and the Atari suite (9. During the the long period of pre-training with environment rewards removed, we use DrQ to maximize the entropy maximization reward defined in equation ( The pre-trained value function Q(s,a) is fine-tuned to maximize task specific reward after being exposing to environment rewards during testing period. For our DeepMind control suite and Atari games experiments, we largely follow DrQ, except we perform two gradient steps per environment step instead of one. Our ablation studies confirm that

---

these changes are not themselves responsible for our performance. Kornia [50 is used for efficient GPU-based data augmentations. Our model implemented in Numpy 21 and PyTorch (45]
APT outperforms prior from scratch SOTA RL on DMControl. We evaluate the performance of different methods by computing the average success rate and episodic return at the end of training.
Run
Cherran Pip Saowwed
Hopper Hop
Hopp Hop Backward
19U8
9-0
000
400
Walcowr Run
Cheddet Sports
Cheaton Raach Spons
000
600
600
200
200
Popper Jump Sparia
Mapper Roadh Sparre
Waltor Jump Spars
Phabe Excaps Spurss
0.5
3.2
Scpenbeed Covioree Eveerned Deps 1X201
Supewieed Divicmant Dees (x30))
Sxpertedd Ditromen Cece (x33))
Supentset Dvirommen Dapa (419))
Arge
(o0s)
Countibeeed borua
Figure 3: Results of different methods in environments from DMControl. All curves are the average of three runs with different seeds, and the shaded areas are standard errors of the mean.
The agent allowed long unsupervised pre-training phase (5M steps), followed by short test phase exposing to downstream reward, during which the pre-trained model is fine-tuned. We follow the evaluation setting of DrQ and test APT on a subset of DMControl suite, which includes training Walker, Cheetah, Hopper for various locomotion tasks. Models are pre-trained on Cheetah, Hopper, and Walker, and subsequently fine-tuned on respective downstream tasks. We additionally design more challenging sparse reward tasks where the robot is required to accomplish tasks guided only by sparse feedback signal. The reason we opted to design new sparse reward tasks to have more diverse downstream tasks. As far as we know, there is only one Cartpole Swingup Sparse that is CartPole based sparse reward task. Due its 2D nature being quite limited, we eventually decided to design distinguishable downstream tasks based on a little bit more complex environment, e.g. Hopper Jump etc. The details of the tasks are included in the supplementary material.
The learning process of RL agents becomes highly inefficient in sparse supervision tasks when relying on standard exploration techniques. This issue can be alleviated by introducing intrinsic motivation, i.e., denser reward signals that can be automatically computed, one approach that works well in high dimensional setting count-based exploration
38
The results are presented in Figure 3 APT significantly outperforms SOTA training from scratch (DrQ from scratch) and SOTA exploration method (count-based bonus) on every task. With only a few number of environment interactions, APT quickly adapt to downstream tasks and achieves higher return much more quicker than prior state-of-the-art RL algorithms. Notably, on the sparse reward tasks that are extremely difficult for training from scratch, APT yields significantly higher data efficiency and asymptotic performance.
APT outperforms from scratch SOTA RL in Atari. We test APT on the sample-efficient Atari setting 30 61 which consists of the 26 easiest games in the Atari suite (as judged by above random performance for their algorithm).
We follow the evaluation setting in VISR, agents are allowed a long unsupervised training phase (250M steps) without access to rewards, followed by short test phase with rewards. The test phase contains 100K environment steps equivalent to 400k frames, just under two hours compared to
7

---

the typical standard of 500M environment steps, or roughly 39 days of experience. We normalize the episodic return with respect to expert human scores to account for different scales of scores in each scores (HNS) of an agent on game is calculated as agent score random score and aggregated across games by mean or median.
game, as done in previous works. The
score random
full list of scores and aggregate metrics on the Atari 26 subset is presented in Table 2 The results the full 57 Atari games suite is presented in supplementary material. For consistency with previous works, we report human and random scores from [25]. In the data-limited setting, APT achieves super-human performance on eight games and achieves scores higher than previous state-of-the-ards In the full suite setting, APT achieves super-human performance on 15 games, compared maximum of 12 for any previous methods and achieves scores significantly higher than any previous methods.
Table 2: Performance of different methods on the 26 Atari games considered by RD after 100K environment steps. The results recorded the end of training and averaged over 10 random seeds for APT. APT outperforms prior methods on aggregate metries, games while using similar amount of experience. Prior work has reported different numbers for some of the baselines, particularly SimPLe and DQN. To be rigorous, we pick the best number for each game across the tables reported in van Hasselt al. 61 and Kielak (31]
exceeds expert human performance on 7 out 26
Game
Random
Human
SimPLe
DER
CURL
DiQ
SPR
VISR
APT (ours)
Alien
227.8 8
7127.7
616,9
739.9
558.2
771.2
801.5
364.4
2614.8
1719.5
88.0
188.6
142.1
176.3
186.0
Assault
222.4
742.0
527.2
431.2
600.6
452.4
571.0
12091.1
891.5
Asterix
210.0
8503.3
1128.3
470.8
734.5
603.5
977.8
6216.7
185.5
Bank Heist
14.2
753.1
34,2
51.0
131.6
168.9
380.9
71.3
416.7
BartleZone
2360.0
37187.5
$184.4
10124.6
14870.0
12954.0
16651.0
7072.7
7065.1
Boxing
0.1
12.1
9.1
0.2
1.2
6.0
35.8
13.4
21.3
Breakout
1.7
30.5
16.4
1.9
4,9
16.1
17.1
17.9
10,9
ChomparCommand
811.0
7387.8
1246.9
861.8
1058.5
780.3
974.8
800.8
317.0
Crazy Climber
10780.5
23829.4
62583.6
42923.6
49373.9
44128.0
Demon Attack
107805
35829.4
62583.6
16185.3
12146.5
20516.5
42923.6
8994,9
5071.8
Freeway
0.0
29.6
20.3
27.9
26.7
9.8
24.4
-12.1
29,9
Frostbite
65.2
4334.7
254.7
866.8
1181.3
331.1
1821.5
230.9
1796.1
Gopher
257.6
2412.5
771.0
349.5
669.3
636.3
715.2
498.6
259M0.
Hero
1027.0
30826.4
2656.6
6857.0
6279.3
3736.3
7019.2
663.5
6789.1
Jamesbond
29.0
302.8
125.3
301.6
471.0
236.0
365.4
484.4
356.1
Kangaroo
52.0
3035.0
323.1
779.3
872.5
940.6
3276.4
1761.9
412.0
Krull
1598.0
2665.5
4539,9
2851.5
4229.6
4018.1
2688.9
3142.5
2312.0
Kung Fu Muster
258.5
22736.3
17257.2
14346.1
14307.8
9111.0
13192.7
16754.9
17357.0
Ms Pacman
307.3
6951.6
1480.0
1204.1
1465.5
960.5
1313.2
558.5
2827.1
Pong
-19.3
-16.5
-5y9
26.2
Private Eye
24.9
69571.3
58.3
97.8
218.4
-13.6
124,0
98.3
96.1
Qhert
163.9
13455.0
1288.8
1152.9
1042.4
854.4
669.1
666.3
17671.2
Road Runner
11.5
7845.0
5640.6
9600.0
5661.0
8895.1
14220.5
6146.7
4782.1
Seaquest
68.4
42054.7
683.3
354.1
384.5
301.2
$83.1
706.6
2116.7
Up N Down
533.4
11693.2
3350.3
2877.4
2955.2
3180.8
28138.5
10037.6
8289.4
Mean HNS
0.000
1.000
44.3
28.5
38.1
35.7
70.4
64.31
69.55
Median HNS
0.000
1,000
14.4
16.1
17.5
26.8
41.5
12,36
47.50
Superruman
N/A
Unsupervised pre-training top of DrQ leads significant increase in performance(a 54% increase in median score, a 73% increase in mean score, and 5 more games with human-level performance), surpassing DQN which trained hundreds of millions of sampling steps.
Compared with SPR 522 which a recent tfateoof-th--ae model-based data-efficient algorithm, APT achieves comparable mean and median scores. The SPR is based on Rainbow which combines more advances than DrQ which is significantly simpler. While the representation of SPR is also learned by contrastive learning, it trains model-based dynamic predict its own latent state representations multiple steps into the future. This temporal representation learning, as illustrated in the SPR paper, contributes to its impressive results compared with standard contrastive representation learning. We believe that it is possible to combine temporal contrastive representation learning of SPR with the effective nonparametric entropy maximization of APT, which is an interesting future direction.
APT outperforms prior unsupervised RL. Despite there being many different proposed unsuper- vised RI. methods, their successes are only demonstrated in simple state based environments. Prior works train the agent for a period of fully-unsupervised interaction with the environment, during which the agent is trained to learn set of skills associated with different paths through the environ-

---

ment, as in DIAYN 17 and VIC [18] or to maximize the diversity of the states it encounters, as in MEPOL 22 and Hazan al. 22 Until recently, VISR [20] achieves improved results in Atari games using pixels as input based using successor feature based approach. In order to compare with prior unsupervised RI. methods, we choose DIAYN due to being based on mutual information maximization and its reported high performance in state-based RI., and MEPOL. due to it being based on entropy maximization. We implement them to take pixels as input in Atari games. Our implementation was checked against publicly available code and we made best effort attempt to tune the algorithms in Atari games. We test two variants of DIAYN and MEPOL, using or not using contrastive representation learning as in APT. In order to ensure a fair comparison, test a variant of APT without contrastive representation learning.
The aggregated results are presented in Table 3 APT significantly outperforms prior state-based unsuper- vised RI. algorithms DIAYN and MEPOL.. Both base- lines benefit from contrastive representation learn- ing, but their scores are still significantly lower than APT's score, confirming that the effectiveness of the off-policy entropy maximization in APT. Compared with the state-of-te of-t--of-the-at method in Atari VISR, APT achieves significantly higher median score despite having& a lower mean score. From the scores break- down presented in supplementary file, APT performs significantly better than VISR in hard exploration games, while VISR achieves higher scores in dense reward games. We attribute this to that maximiz- ing state entropy leads more exploratory behavior while successor features enables quicker adaptation for dense reward feedback. It is possible combine VISR and APT to have the best of both worlds, which we leave as a future work.
Table 3: Evaluation in Atari games. The amount of RL interaction utilized is 100K. Mdn is the dian of human-normalinn normalized scores, M is the mean and > H is the number of games with numan-level performance, CL denotes training representation encoder using contrastive learning and data aug- mentation. On each subset, we mark bold the highest score.
26
Game Subset
Full 57 Games
Algorithm
Mdn
M
>>
Mdn
CBB
1.23
21.94
3
MEPOL
0.34
17.94
2
DIAYN
1.31
25.39
2
2.95
23.90
6
CBB w/ CL
1.78
17.34
2
MEPOL w/ CL
1.05
21.78
3
DIAYN w/ CL
1.76
28.44
2
3.28
25.14
VISR
9.50
128.07
7
6.81
102.31
11
APT w/o CL
21.23
28.12
3
28.65
11.12
9
APT
47.50
69.55
7
33.41
47.73
Ablation study. We conduct several ablation studies to measure the contribution of each component in our method. We test two variants of APT that the same number of gra- dient steps per environment step and use the same activation function as in DrQ. Another variant of APT is based randomly selected neighbors to compute particle-based entropy. We also test a variant of APT that use fixed randomly initialized encoder study the impact of representation learning. Table 4 shows the performance of each vari- ant of APT. Increasing gradient steps of updating value function from 1 to 2 and using ELU activation function yield higher scores. Using k-nearest neighbors cru- cial to high scores, we believe the reason is randomly selected neighbors do not provide necessary incentive to explore. Using randomly initialized convolutional encoder downgrades performance significantly but still achieve higher score than DrQ, indicating our particle- based entropy maximization is robust and powerful.
Table 4: Scores on the 26 Atari games under consideration for variants of APT. Scores averaged over random seeds. All variants listed here use data augmentation.
Variant
Human-Normalinn Normalized Score mean
median
APT
47.50
69.55
APT w/o optim change
41.50
60.10
APT w/o arch chunge
45.71
67.82
APT w/ rand neighbor
20.80
24.97
APT w/ fixed encoder
33.24
41.08
Contrastive learning representation been shown have the "uniformity on the hypersphere" property [65 this leads
has
to the question that whether maximum entropy exploration in state space is impor- tant. To study this question, we have variant of APT "Pos Reward APT" which receives simple positive do not die signal but no particle-based entropy reward. We ran the experiments on MsPaeman, we reduced the pretraining phase to SM steps to reduce computation cost. The evaluation metrics are the number of ram states visited using [2] and the downstream zero shot performance on Atari game, APT visits nearly 27 times more unique ram states than "Pos Reward APT", showing that the entropy intrinsic reward is indispensable for exploration. In downstream task evaluation over 3 random seeds, "Pos Reward APT@0" achieves reward 363.7, "APT@0" achieves reward 687.1, showing that the "do not die" signal is insufficient for exploration or learning pretrained behaviors and representations.

---

We consider a variant of APT that re-initialize the head of pretrained actor-critic. We have run experiments in five differ- ent Atari games, as shown in Table 5 pretrained heads perform better than randomly initialized heads out of games. The experiments demonstrate that finetuning from pretrained actor-critic head accelerates learning. However, believe that which one of the two is better depends on the alignment between downstream reward and intrinsic reward. It would be interesting to study how to better leverage downstream reward to finetune the pretrained model.
Table Scores 5 Atari games under consideration for different variants of fine-tuning. Scores are averaged over random seeds.
Mean Reward (3 seods)
Alien
Freeway
Qbert
Private Eye
MsPacman
APT (pretrained head)
2614.8
29.9
17671.2
96.1 61.3
2827.1
APT (randomm heud)
1755.0
15.2
2138.3
1724.9
Discussion
strategy employed here (when combined with value function) works Limitation: The and extrinsic rewards being of a similar scale. We believe the discrepancy between intrinsic reward scale and downstream reward scale possibly explain the suboptimal perfor- mance of APT in dense reward games. This is an interesting future direction to further improve APT, we hypothesize that reinitializing behaviors part (actor-critic heads) might be useful if the downstream reward scale is very different from pretraining reward scale. One of the principled ways could be adaptive normalization (62) it is an interesting future direction. One challenge of our method is the non-stationarity of the intrinsic reward, being non additive reward poses an interesting challenge for reinforcement learning methods. While our method outperforms training from scratch and prior works, we believe designing better optimization RL methods for maximizing our intrinsic reward can lead to more significant improvement.
best when the intrinsic
Conclusion: A new unsupervised pre-training method for RL is introduced to address reward- -free pre-training for visual RL, allowing the same task-agnostic pre-trained model to successfully tackle a broad set of RL tasks. Our major contribution is introducing practical intrinsic reward derived from particle-basee entropy maximization in abstract representation space. Empirical study on DMControl suite and Atari games show our method dramatically improves performance on tasks that are extremely difficult for training from scratch. Our method achieves the results of fully supervised canonical RI. algorithms using a small fraction of total samples and outperforms data-efficient supervised RI. methods.
For future work, there are a few ways in which our method can be improved. The long pre-training phase in our work is computationally intensive, since the exhaustive search and exploration of high sample complexity. One way to remedy this is by combining our method with successful model-based RL and search approaches reduce sample complexity. Furthermore, fine-tuning the whole pre- trained model can make it prone to catastrophic forgetting. As such, it is worth studying alternative methods to leverage the pre-trained models such as keeping the pretrained model unchanged and combine it with a randomly initialized model.
7
Acknowledgment
This research was supported by DARPA Data-Driven Discovery of Models (D3M) program. We would like to thank Misha Laskin, Olivia Watkins, Qiyang Li, Lerrel Pinto, Kimin Lee and other members at RLI. and BAIR for insightful discussion and giving constructive comments. We would also like to thank anonymous reviewers for their helpful feedback for previous versions of our work.
References
[1] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, al. Solving rubik's cube with robot hand. arXiv preprini arXiv: 19.0077113, 2019.
[2] A. Anand, F. Racah, S. Ozair, Y. Bengio, M. Côté, and R. Hjelm. Unsupervised state representation learning in atari. In Advances in Neural Information Processing Systems 32:
10

---

Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pages 8766-8779, 2019.
[3] M. Andrychowicz. D. Crow, A. Ray, J. Schneider, R. Fong. P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages
2017.
[4] L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: 2016.
[5] A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z. D. Guo, and C. Blundell, Agent57: Outperforning the atari human benchmark. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 507-517. PMLR, 2020.
[6] A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. Arjovsky, A. Pritzel, A. Bolt, and C. Blundell. Never give up: Learning directed exploration strategies, In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[7] D. Barber and F. V. Agakov. The im algorithm: A variational approach to information maxi- mization. In Advances in neural information processing systems, 2003.
[8] J. Beirlant. Nonparametric entropy estimation: An overview. International Journal of the Mathematical Statistics Sciences, 6:17-39, 1997.
[9] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[10] M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems 29; Annual Conference Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1471-1477 2016.
[11] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. B.Chess, Clark, C. Berner, S. McCandlish, A. Radford. I. Sutskever, and D. Amodei. Lan- guage models are few-shot leamers. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[12] V. Campos, A. Trott, C. Xiong, R. Socher, X. Giró-i-Nicto, and J. Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1317-1327. PMLR, 2020.
[13] V. Campos, P. Sprechmann, S. S. Hansen, A. Barreto, C. Blundell, A. Vitvitskyi, S. Kapturowski, and A. P. Badia. Coverage as a principle for discovering transferable behavior in reinforcement learning, 2021.
[14] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 2020.
[15] D. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[16] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: (0.18653/v1/NIB
11

---

[17] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without reward function. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[18] K. Gregor, D. Rezende, and D. Wierstra. Variational intrinsic control. arXiv preprint arXiv: 1,07507 2016.
[19] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsnässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of P Proceedings of Machine Learning Research, pages 1856-1865. PMLR, 2018.
[20] S. Hansen, W. Dabney, A. Barreto, D. Warde-Farley, T. V. de Wiele, and V. Mnih. Fast task infer- ence with variational intrinsic successor features. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[21] C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, Taylor, S. Berg, N. Smith, et al. Array programming with numpy. Nature, 585
(7825):357-362, 2020.
[22] E. Hazan, S. M. Kakade, K. Singh, and A. V. Soest. Provably efficient maximum entropy exploration. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2681-2691. PMLR, 2019.
[23] K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9726-9735. IEEE, 2020. doi: 10.1109//VPR22000 200000095.
[24] O. J. Hénaff. Data-efficient image recognition with contrastive predictive coding. In Proceedings of 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4182-4192. PMLR, 2020.
[25] M. J. J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. G. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 3215-3222 AAAI Press, 2018.
[26] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. D. Turck, and P. Abbeel. VIME: variational information maximizing exploration. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 110-1111,, 117, 2016.
[27] E. T. Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
[28] J. Jiao, W. Gao, and Y. Han. The nearest neighbor information estimator is adaptively near minimax rate-optimal. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 3160-3171, 2018
[29] R. Julian, B. Swanson, G. S. Sukhatme, S. Levine, C. Finn, and K. Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. arXiv e-prints, pages arXiv-2004, 2020.
[30] I. Kaiser, M. Bahaeizadeh, P. Milos, B. Osinski, R. H. Camphell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, A. Mohiuddin, R. Sepassi, G. Tucker, and H. Michalewski. Model based reinforcement learning for atari. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[31] K. Kielak. Do recent ad advancements in model-based deep reinforcement learning really improve data efficiency? arXiv preprint arXiv:2003. 10181, 2020.
[32] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 44, 2014, Conference Track Proceedings, 2014.
12

---

[33] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13499, 2020.
[34] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learming with augmented data, arXiv:2004. 14990,, 2020.
[35] M. Laskin, A. Srinivas, and P. Abbeel. CURL: contrastive unsupervised representations for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages (639-5650. PMLR, 2020.
[36] L. Lee, B. Eysenbach, E. Parisotto, E. Xing. S. Levine, and R. Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint 1906.00900.05550 /906005744 2019.
[37] H. Liu and P. Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pages 9736-6747, PMLR, 2021.
[38] M. C. Machado, M. G. Bellemare, and M. Bowling. Count-based exploration with the successor representation. In The Thirty-Fourte Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 5125 5133. AAAI Press, 2020.
[39] T. Miyato, Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenRe- view.net, 2018.
[40] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. nature, 518(7540):529 553, 2015.
[41] S. Mohamed and J. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2125-2133, 2015.
[42] M. Mutti, L. Pratissoli, and M. Restelli, A policy gradient method for task agnossicc exploration. arXiv preprint arXiv:2007.046640, 2020.
[43] A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arxiv preprint arXiv: we0..0077035555 2018
[44] G. Ostrovski, M. G. Bellemare, A. van den Oord, and R. Munos. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning, ICML. 2017, Sydney, NSW, Australia, 6-1 I August 2017, volume 70 of Proceedings of Machine Learning Research, pages 2721-2730. PMLR, 2017
[45] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, 7. Lin, N. Gimelshein, I.. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. De Vetioo, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, I.. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. Advances Neural Information Processing Sys- tems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024-8035, 2019.
[46] D. Pathak, D. Gandhi, and A. Gupta. Self-supervised exploration via disagreement. In Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5062-50000000 10711 PMLR, 2019.
[47] S. Pitis, H. Chan, S. Zhao, B. Stadie, and J. Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pages 7750-7761. PMLR, 2020.
[48] M. L.. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley &amp; Sons, 2014.
[49] K. Rawlik, M. Toussaint, and S. Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. Proceedings of Robotics: Science and Systems VIII. 2012.
13

---

[50] E. Riba, D. Mishkin, D. Ponsa, E. Rublee, and G. Bradski. Kornia: an open source differentiable computer vision library for pytorch. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3674-3683, 2020.
[51] J. Schrittwisser, I. Antonoglou, T. Hubert, K. Simonyan, L.. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv: arXiv:1911.0820 0802665, 2019.
[52] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman. Data-efficient reinforcement learning with self- ceff--rreictivee representations. In International Conference on Learning Representations, 2021.
[53] C. E. Shamnon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
[54] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-awar aware unsupervised discovery of skills. In 8th International Conference on Learning Representations. ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[55] H. Singh, N. Mísra, V. Hnizdo, A. Fedorowicz, and F. Demchuk. Nearest neighbor estimates of entropy. American journal of mathematical and management sciences, 23(3-4):301-311 2003.
[56] A. Stooke, K. Lee, P. Abbeel, and M. Laskin. Decoupling representation learning from reinforcement learning. arXiv preprint arXiv:2009.08: 08319, 2020.
[57] R. Y. Tao, V. François-Lavet, and J. Pineau. Novelty search in representational space for sample efficient exploration. arXiv preprint arXiv:2009. 13579, 2020.
[58] Y. Tassa, S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez,
T. Lillicrap, and N. Heess. dm_control: Software and tasks for continuous control. arXiv preprint arXiv:2006. /2983, 2020.
[59] E. Todorov. General duality between optimal control and estimation. In 2008 47th IEEE Conference on Decision and Control, pages 4286-4292. IEEE, 2008
[60] M. Toussaint. Robot trajectory optimization using approximate inference. In Proceedings of the 26th Annual International Conference Machine Learning, ICML 2009, Montreal, Quebee, Canada, June 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pages 1049-1056. ACM, 2009. doi: 10. 14/.14//553374153350000
[61] H. Hasselt, M. Hessel, and J. Aslanides. When to use parametric models in reinforcement learning? In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver,
BC, Canada, pages 14322-14333, 2019,
[62] H. P. van Hasselt, A. Guez, M. Hessel, V. Mnih, and D. Silver. Learning values across many orders of magnitude. Advances in Neural Information Processing Systems, 29:4287-4295, 2016.
[63] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. In Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 2017, Long Beach, CA, USA, pages 5998-6009 2017.
[64] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathicu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782) 2019.
[65] T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939. PMLR, 2020.
[66] D. Warde-Farley, T. V. de Wiele, T. D. Kulkarni, C. Ionescu, S. Hansen, and V. Mnih. Unsuper- vised control through non-parametri discriminative rewards. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019.
(67] D. Yarats, R. Fergus, A. Lazaric, and I.. Pinto. Reinforcement learning with prototypical representations. arXiv preprint arXiv:2102. 11271 2021.
14

---

[68] R. Zhao, Y. Gao, P. Abbeel, V. Tresp, and W. Xu. Mutual information state intrinsic control. arXiv preprint arXiv:2103.08100 2021.
[69] B. D. Ziebart, A. L.. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, II, USA, 2008.
15


Reviews:
[{"review_id": "nK5NZh0e-I", "reviewer": "Reviewer_Zmg7", "summary": "This paper proposes an unsupervised (or rather, self-supervised) training method for RL, named active pre-training. This is based on using contrastive learning from pixels, then maximizing state entropy in the abstract space using a k-nearest neighbors density estimate. The authors combine their method with DrQ, demonstrating sample efficiency and asymptotic improvements on DeepMind control suite and Atari, along with several useful ablations.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: The largest limitation of this paper appears to be a relative lack of novelty. The proposed method combines two popular ideas: self-supervised contrastive learning and state entropy maximization. As the authors discuss in section 4, these ideas have been extensively explored in previous works. Indeed, even within state entropy maximization, multiple previous works have used non-parametric density estimators. Note: I would also cite Pitis et al., 2020 as another examples of state entropy maximization via non-parametric estimators. However, in that paper, they perform density estimation in the state space rather than in a latent space. Indeed, the main distinction/contribution of this paper appears to be entropy maximization within a contrastively learned latent space. This differs from previous works that have used contrastive learning purely for pre-trained representations, as well as other works that have used non-contrastively learned latent spaces for exploration. However, given that this is the primary novelty of the paper, it is somewhat unclear and poorly motivated/justified within the paper. For example, as far as I can tell, the main justification for this choice is in sentences like \n\n*“We hypothesize that the ineffectiveness stems from density modeling at the pixel level being difficult.”*\n\n*“We consider an abstract representation space in order to make the distance meaningful.”*\n\nThese statements are not only imprecise, they are also not rigorously analyzed empirically. The main evidence for these claims comes from performance differences between the proposed method and other (state/observation-based) exploration methods. To summarize, this paper combines two useful techniques, contrastive representation learning and state entropy maximization, showing that their combination is even more useful. However, given that this combination is only weakly motivated in the paper, one is left wondering whether this paper provides much in the way of new insights.\n\nPitis et al., 2020, Maximum entropy gain exploration for long horizon multi-goal reinforcement learning.\n\n\n**Quality**: Overall, the paper is fairly high quality, particularly the empirical performance evaluation. The authors compare their proposed method with various visual and exploration RL algorithms on both DeepMind control suite and Atari. The number of reported environments is extensive in both domains, and the results are competitive with SOTA algorithms. I see this evaluation as the strongest point of the paper, especially in a benchmark-focused area like RL. I also found the ablations in Table 4 to be helpful for disentangling the benefits of the various components of the algorithm.\n\n\n**Clarity**: With the exception of the weak motivation issues mentioned above, the paper is clear. For the most part, the background and method are explained well. I found the math fairly straightforward to follow, and I appreciated the diagrams (Figure 2), algorithm box (Algorithm 1), and comparison overview (Table 1). The authors also provide an adequate overview of previous lines of work in the related works section (Section 4).\n\nSome suggestions for improving the clarity of the paper:\n\nIt may help to discuss the benefits and drawbacks of parametric vs. non-parametric density estimators in more detail. Currently, the non-parametric estimator is introduced as the obvious choice, but it’s unclear why this should be any better than, for example, using a VAE or other deep generative model, as in state marginal matching [Lee et al., 2019]. When would one estimator be preferred over the other?\n\nThe intrinsic exploration reward is non-stationary, as it depends on the on-policy state distribution. It would be helpful to discuss any additional challenges associated with optimizing this reward function in comparison with task-specific reward functions.\n\nThe paper discusses entropy maximization in terms of states. Unfamiliar readers are likely more familiar with action/policy entropy maximization. It would be helpful to include a sentence or two describing the distinction between these approaches.\n\n\n**Significance**: I see this paper as fairly significant for the following reasons: 1) the empirical evaluation provides results across an extensive set of environments with reasonable improvements, and 2) the method combines two fairly popular techniques in the literature, and accordingly, is likely easier to reproduce and build upon. For these reasons, I could see this paper as a useful contribution to the RL community. While this may not be the most novel or well-motivated paper, it provides a fairly clear evaluation of a useful technique.\n\n\n**Minor comments**:\n- Line 80: Why is T a distribution and not a density?\n- Line 96: Would be helpful to explain the variable w a bit more.\n- Line 131: Would be helpful to describe some of the design choices around the kernel more explicitly.\n- Line 155: It’s not clear that entropy maximization is entirely modular from representation learning. For entropy maximization to work, the representation needs to contain a compressed representation of the state.\n- Table 1: It’s not clear whether APT necessarily needs to be off-policy. Couldn’t one construct an on-policy version?\n- Line 267: incrase —> increase\n- Line 278: I’m not sure this is necessarily true for SMM.\n- Table 3: Slightly unclear whether APT w/o CL refers to exploration in the pixel space or in a random encoder space.\n- Line 332: phrase —> phase\n\n---\n**Update**: I have read the other reviews and the authors' responses. I had already rated this paper highly, and the authors provided a satisfactory response. The other reviewers also have increased their scores. I will maintain my score at an 8.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors addressed limitations of their work in the conclusion. Specific negative societal impacts from this work are unclear (other than those associated with machine learning generally).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "g5DHoBLXy-8", "reviewer": "Reviewer_n2Z4", "summary": "* This paper introduces APT (Active Pre-Training), a method for unsupervised pre-training in RL environments. \n* During the pertaining phase, APT explores an environment using an intrinsic reward that incentivizes maximizing entropy in a contrastive representation space. \n* APT is evaluated on DM-Control and Atari to test whether pre-training improves data-efficiency. APT outperforms training RL methods from scratch in DMControl and Atari, and the gains are more noticeable in sparse reward environments.  \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This works focuses on an important research problem: How can we equip RL algorithms with strong priors about the environment so that we don’t start tabula-rasa each time we solve a task? APT approaches this by trying to learn a task-agnostic representation of an environment as well as a policy/Q-function initialization during an unsupervised pre-training phase. The representations are learned via a contrastive learning objective that leverages data augmentation, very similar to CURL, DRIML, SPR, ProtoRL and other works in this space: so this isn’t the novel aspect of this paper. To collect the data for training the encoder, APT uses an intrinsic reward that incentivizes maximizing entropy in this contrastive representation space. \n\nThe results on DMControl (particularly the sparse environments) are quite impressive, but the results on Atari are a bit underwhelming: it seems leveraging a massive amount of data (250M steps) buys little in terms of data efficiency (APT is comparable with SPR which uses no pre-training). \n\nI have some questions about the paper, and the impact of the novel exploration method in particular which might be useful for the readers to know: \n\n1. Since the main difference between APT and prior unsupervised RL methods is the introduction of a different intrinsic reward scheme to use for the exploration policy, it would be nice to see the comparison of how well this exploration policy performs against prior methods. In prior works like VISR, this has been well-benchmarked: see Table 1 (first section) of [VISR](https://arxiv.org/abs/1906.05030) [1] which reports Curiosity@0 and VISR@0. Comparing the APT@0 scores would show whether APT performs better exploration than prior methods during the unsupervised phase. \n\n2. I am concerned that contrastive learning is going to lead to maximally spread-out representations that naturally maximize APT’s exploration reward without any actual exploration at all.  A core part of contrastive learning is the \"uniformity on the hypersphere\" [2] property, which means that points in the replay buffer should be encoded as maximally distant from other points in the replay buffer. Hence, it’s not clear that APT’s exploration is much more sophisticated than giving the agent a reward for staying alive.  Note that the VISR paper does report this baseline: \n>To rule this out, an explicit “do not die” baseline was run (Pos Reward NSQ), wherein the terminal signal remains and a small constant reward is given at every time-step.\n\nSo, if we compare with APT@0 with “Pos Reward NSQ @0” (Table 1 in VISR), we should be able to disentangle if this is actually the case. \n\n3. The ablations section is limited and I want to better understand which components of APT provide a strong learning prior, and which don’t. Table 3 and Table 4 already shows that CL is indeed quite helpful, but I am not sure how much do the learned policy/q-networks contribute. Do you have an experiment that ablates  the performance of a randomly initialized policy/q-network vs their pre-trained counterparts? \n\n4. The most interesting experiments in the paper are on the sparse reward environments in DMControl, showing a remarkable improvement in both data efficiency and final performance on sparse environments. I was a bit confused to find out though that these sparse rewards were custom designed, and not evaluated on any of the standard sparse environments already available in DMControl (Cartpole Swingup Sparse, Cartpole Balance Sparse, etc.). Introducing both new envs and new methods in the same paper can often lead to selection bias, and it would be nice to see if the results on already existing DMControl envs so that future works also can benchmark against these results easily. See for example ProtoRL [3], another unsupervised RL method that uses these sparse envs. \n\n5. Can you clarify what \"Value of k\" = \"search in {3, 5, 10}\" in the supplementary refers to? Does this mean the value of k was tuned for each env separately. If so, this is problematic, and deviation from the standard evaluation protocol in Atari where the same algo with the same hyperparams are used for all games. This also makes comparisons to baseline unfair since none of the baselines used per-env hyper-param tuning.\n\nMissing prior work: Though the paper in general does a good job of covering prior work, it misses ProtoRL which is a relatively/concurrent work, and so its okay to not directly compare to it but it should be cited nonetheless.  There should also be a mention of Plan2Exlpore in the related work section. \n\nMinor comments on the writing: \n\n1. In Algo 1: “Train neural encoder f on mini batch” is vague: you can reference the contrastive learning equation here. There should also be one more step that clarifies that data augmentation is performed on the mini-batch. Instead of standard Q-learning, you should say standard actor-critic (since both policy and Q-networks are mentioned).\n2. \"With the models and datasets growing, there is still no sign of saturating performance during pre-training.\" The scaling law curves do start to saturate after reaching irreducible entropy, perhaps a better way to phrase this is “With increasing dataset and mode sizes, performance continues to improve predictably according to scaling laws.”  \n3. The imagenet pertaining experiment in Figure 1 has no details, and in-fact a recent paper ([2107.03380 RRL: Resnet as representation for Reinforcement Learning](https://arxiv.org/abs/2107.03380)) contradicts these results (the paper was released after the NeurIPS deadline). It would be nice to have details of the ImageNet experiment in the supplementary material so that we could compare the differences in experimental setup between these two.  \n\n[1] Hansen, Steven, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. \"Fast task inference with variational intrinsic successor features.\" ICLR 2020\n\n[2] Wang, Tongzhou, and Phillip Isola. \"Understanding contrastive representation learning through alignment and uniformity on the hypersphere.\" ICML 2020\n\n[3] Yarats, Denis, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. \"Reinforcement learning with prototypical representations.\" ICML 2021", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations section should include the fact that APT demands a huge data budget (250M steps) for pretraining in Atari: which would be not be practical for real-world RL. It should also be pointed out APT depends on environment specific data augmentation. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "e025HGd_-By", "reviewer": "Reviewer_zhzo", "summary": "This paper explores the idea of doing unsupervised pre-training in a reinforcement learning context. Before giving task rewards to the agent, it is given many orders of magnitude more experience for unsupervised exploration. \n\nThe key idea is to use a non parametric entropy based reward function to efficiently explore the space and amortize it in a Q function. This Q function is then fine tuned on the task reward as a follow up stage. \n\nCompared to previous unsupervised RL papers, this paper explores a new and interesting particle based formulation instead of count/density based model for intrinsic motivation. The performance metrics are shown on the atari and DMControl suite environments. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The particle based entropy reward is novel as far as I understand. Typically previous work has explored the use of count based, density based and structured representations for directed exploration in a similar learning setting. However the particle based method explored here is simple, cheap to compute and asymptotically accurate. I also think that the experiment validate or answer the hypotheses considered in this paper. I also like that this paper ran strong baselines and also highlighted regimes where this approach is not as good as the baselines (table 3). So overall I think this is an interesting paper with thorough experiments and sound hypotheses. \n\nIn Table 3, VISR is much better in terms of the mean score than the proposed method. The authors explain this as -- \"APT performs significantly better than VISR in hard exploration games, while VISR achieves higher scores in dense reward games\". But if I look at Table 2 and the scores for pong, I would have expected VISR to perform better with this logic as it is fairly a densely rewarded environment. But this is not the case. It would be helpful to analyze and understand where, how and why VISR performs better in more detail. \n\n Also I am concerned about the median vs mean scores reported in the results section. How do all the results change if the number of seeds are increased or decreased?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "seems adequate", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "D8ml2z53iId", "reviewer": "Reviewer_D1C9", "summary": "The authors introduce Active Pretraining (APT), wherein an agent is encouraged to maximization the entropy of the state distribution represented in its replay buffer. The is accomplished by having rewards that are proportional to distance to nearest neighbours in some embedding space. This space is constructed via a separate contrastive learning objective.\n\nThis method is tested in a 2 stage process where the agent initially maximizes this state entropy entropy for a long time, and then finetunes on a down-stream time for a much shorter time, with the final performance on this task being the key metric. By this methodology, APT surpasses both data-efficient methods that train from scratch, as well as other methods that use the same 2 stage training scheme.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty of the method is a bit slight, as using state-entropy as an intrinsic reward has been attempted elsewhere. That said, the authors acknowledge this prior work, and emphasize the scalability and performance of their system. This puts much more pressure on the empirical results to do the heavy lifting, and luckily they largely deliver on this promise.\n\nThe initial negative result showing that naive approaches to pretraining (e.g. ImageNet) are insufficient to accelerate downstream tasks provides some nice motivation. The main results on DMControlSuite and Atari have a large number of baselines whose inclusion is well motivated in the text. The performance of APT is quite convincing, but I would suggest including some notion of spread in all of the table and figures so that the significance is easier to assess. Both ablating contrastive learning from APT and including it in some baseline methods was a nice touch, and the other ablations are also appreciated. That said, Appendix A did give me pause, as the number of architectural flourishes used here feels higher than many other RL papers (e.g. layer norm, spectral norm), but considering you compare favourably to a Rainbow variant (which is probably the fanciest data efficient agent to use as a base) I won't hold that against you.\n\nOne significant concern involve proposition 1. It's correct, if a bit trivial, but it's unclear that it matters in practice, as the number of distinct states is far higher than the number of training steps for the pixel-based domains that are core to this method's evaluation. The applicability is further undercut by the reward normalization scheme -- as the reward goes to zero, the normalized reward goes to infinity! And I'm not even convinced one would even want the intrinsic reward to vanish asymptotically, as that suggests that pretraining for too long would result in a random policy. Luckily I think these two problems cancel out; if you don't want your reward to vanish, then just removing the proposition and the surrounding text would alleviate my concern.\n\nMinor points:\n1) The \"Exploration\" tick marks in Table 1 feel very subjective. Maximizing state or action entropy is exploration, but related mutual information terms isn't? Even if there is an argument for this distinction, since just switching to a base RL method that includes an entropy bonus (e.g. SAC) would resolve it, categorizing methods along this axis feels unhelpful.\n2) The fine-tuning strategy employed here (when combined with a value function) relies on the intrinsic and extrinsic rewards being of a similar scale. This is in contrast to the regression-style approach of VISR, and should thus probably be mentioned as a weakness of APT unless its already being addressed by other means (e.g. van Hasselt et al).\n\nvan Hasselt, Hado P., et al. \"Learning values across many orders of magnitude.\" Advances in Neural Information Processing Systems 29 (2016): 4287-4295.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel unsupervised pre-training method for reinforcement learning, using a particle-based entropy estimator on a contrastive-loss trained feature representation. The method is novel and scalable and the empirical results and rebuttal additional experiments are strong. I trust the authors will incorporate the reviewers' comments and discussion points into the final version of the manuscript.
Decision: Accept (Spotlight)