A Mixture Of Surprises for Unsupervised Reinforcement Learning
Authors: Andrew Zhao
Matthieu Gaetan Lin
Yangguang Li
Yong-jin Liu
Gao Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Unsupervised Reinforcement Learning

Abstract: Unsupervised reinforcement learning aims at learning a generalist policy in a reward-free manner for fast adaptation to downstream tasks. Most of the existing methods propose to provide an intrinsic reward based on surprise. Maximizing or minimizing surprise drives the agent to either explore or gain control over its environment. However, both strategies rely on a strong assumption: the entropy of the environment's dynamics is either high or low. This assumption may not always hold in real-world scenarios, where the entropy of the environment's dynamics may be unknown. Hence, choosing between the two objectives is a dilemma. We propose a novel yet simple mixture of policies to address this concern, allowing us to optimize an objective that simultaneously maximizes and minimizes the surprise. Concretely, we train one mixture component whose objective is to maximize the surprise and another whose objective is to minimize the surprise. Hence, our method does not make assumptions about the entropy of the environment's dynamics. We call our method a $\textbf{M}\text{ixture }\textbf{O}\text{f }\textbf{S}\text{urprise}\textbf{S}$ (MOSS) for unsupervised reinforcement learning. Experimental results show that our simple method achieves state-of-the-art performance on the URLB benchmark, outperforming previous pure surprise maximization-based objectives. Our code is available at: https://github.com/LeapLabTHU/MOSS.

A Mixture of Surprises for Unsupervised Reinforcement Learning
Andrew Zhao Matthieu Gaetan Lin
Yangguang Li
Yong-Jin Liu
Gao Huang 1f
Department of Automation, BÑRist, Tsinghua University 2 Department of Computer Science, BNRist, Tsinghua University
3 SenseTime
{zqc21 mew.h1yymammmlls......eaa tsinghua. edieg edu. en,
Liyangguag@ssneeiime. com gaw.aaaggaaagaagtnnnaaaooooeeeee edu. en,
Abstract
Unsupervised reinforcement learning aims at learning generalist policy in a reward-free manner for fast adaptation to downstream tasks. Most of the existing methods propose to provide an intrinsic reward based on surprise. Maximizing minimizing surprise drives the agent to either explore gain control over its environment. However, both strategies rely on a strong assumption: the entropy the environment's 's dynamics is either high or low. This assumption may not always hold in real-world scenarios, where the entropy of the environment's dynamics may be unknown. Hence, choosing between the two objectives is dilemma. We propose novel yet simple mixture of policies to address this concern, allowing us optimize objective that simultaneously maximizes and minimizes the surprise. Concretely, we train one mixture component whose objective is to maximize the surprise and another whose objective our method does not make assumptions about the entropy of the environment' dynamics. We call our method a Mixture Of SurpriseS (MOSS) for unsupervised reinforcement learning. Experimental results show that our simple method achieves state-of -fethh-rrt art performance on the URLB benchmark. outperforming previous pure surprise maximization-ban based objectives. Our code is available at: https: //github. www com/LeeapaaTTHUU/OSSS
minimize the surprise. Hence,
Introduction
Humans can learn meaningful behaviors without external supervision, ie., in an unsupervised manner, and then adapt those behaviors to new tasks (3. Inspired by this, unsupervised reinforcement learning decomposes the reinforcement learning (RL) problem into pretraining phase and finetune phase [29]. During a pretraining phase, an agent prepares all possible tasks that a user might select. Afterward, the agent tries to figure out the selected task quickly possible during finetuning [22]. Doing so allows solving the RL problem in a meaningful order [3] e.g., a cook first has to look at what in the fridge before deciding what to cook. Unsupervised representation learning has shown great success in computer vision [25] and natural language processing 9 however, one challenge is
that RL includes both behavior learning and representation learning 29 49
Equal contribution. Cooresonnding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Current unsupervised RL methods provide an intrinsic reward to the agent during a pretraining phase Intuitively. this intrinsic reward should incentivize
to tackle the behavior learning problem 29
the agent to understand its environment
13
Current methods formulate the intrinsic reward as
either maximizing or minimizing surprise
52
4
42
12
where
knowledge-bard weaeldgg--aaedd methods quantify surprise as the uncertainty of prediction model TO,
38
39 and
data-based methods measure surprise as an information-theoree
Surprise maximization methods [2 28 33 formulate the problem as an exploration problem. Intuitively, to prepare all possible downstream tasks, an agent has to explore the state space and figure out what is possible in the environment. For instance, instantiation is to maximize the agent's state entropy [33]. In contrast to surprise maximization, another line of work takes inspiration from the free-energy principle 20 19 [8] and proposes to minimize the surprise 66 4T]. In particular, these works argue that external perturbations naturally provide the agent with surprising events. Hence, an agent can learn meaningful behaviors by minimizing its surprise, and minimizing this quantity requires gaining control over these external perturbations 41 6]. For example, SMiRI. 6 proposed to minimize the agent's s state entropy.
quantity [41 6 52
33
S
max
Figure 1: Outline of our Mixture Of SurpriseS (MOSS) strategy. We provide the agent with two paths, each corresponding to maximizing or minimizing surprise. During the pretraining phase, the agent gathers experience from both paths using the skills from the corresponding toolbox. Finally, during the finetuning phase, the agent can choose any skills from any toolbox.
A closer look at these two approaches indicates strong assumption. Substantial external perturba- tions already naturally provide an agent with high state entropy (6) Therefore, surprise-secking approaches assume that the environment does not provide significant external perturbations. On the other hand, in an environment without any external perturbations, the agent already achieves minimum entropy by not acting [41). Therefore, surprise minimization approaches assume that the environment provides external perturbations for the agent to control. However, in real-world scenarios, it is often difficult to quantify the entropy of the environment's dynamics beforehand, or the agent might face both settings. For example, butler robot performs mundane chores daily in household. However, the robot might also encounter surprising events (e.g., putting out a fire). Therefore a fixed assumption on the entropy of the environment dynamics is often not possible. In other words, choosing between surprise maximization or minimization for pretraining is dilemma.
Simultaneously optimizing these two opposite objectives does not make sense. Instead, in this paper, we show that competence-basee methods [29] offer a simple yet effective way to combine the benefits of these two objectives. In addition to conditioning the policy on the state, competence- based methods condition the policy on a latent vector [28] 32
50 21 [ [6]. Conditioning the policy offers an appealing way to formulate the unsupervised RL problem. During pretraining phase, the agent tries to learn set of possible behaviors in the environment and distills them into skills. Then, during finetuning, the agent tries to figure out the selected task as quickly as possible, using the repertoire of skills gathered during pretraining. For example, as illustrated in Fig. [ we view skill distributions as a mixture of policies instead of a single policy. In particular, we train one set of skills whose objective is to maximize the surprise and another set of skills whose objective is to
24
Refer to 29 for detailed definitions of data-based, knowledge-basee and competence-ben based methods.

---

minimize the surprise. Surprisingly, this paper shows that this simple approach works well in practice, which simultaneously optimizes two contradicting objectives. Our primary contribution, presented in Section 4 is simple intrinsic reward called MOSS that does not make assumptions about the
entropy of the environment's dynamics. In Section 5 our experimental results on URLB [29] and
ViZDoom [7 show that, surprisingly, our MOSS methods achieve state-of-the results.
We organize the paper follows. Section briefly analyzes previous unsupervised RL algorithms under the surprise framework. Then in Section 4 we introduce our MOSS method. Next, experimental
results in Section 5 shows that on URLB (29 and ViZDoom [27]. our MOSS method improves
upon previous pure maximization and minimization methods. Finally, we provide discussions and
limitations in Section
2
Preliminaries
Markoy Decision Process. Unsupervised RL methods studied in this paper operate under a Markov Decision Process (MDP) [47.. In particular, we specify an MDP tuple M = (S,A, T pP where S is the state space and A is the action space of the environment. T(S' S, A) represents the state-transition dynamics, p : S - [0, is the initial state distribution, and 7 [0, 1) is the discount factor. At each discrete time step t E 2", the agent receives a state and performs an action, which we denote as S, S and A; AA, respectively. During pretraining, unsupervised RL algorithms compute an intrinsic reward rim! during the finetune phase, the agent receives the extrinsic reward pext given by the environment each interaction.
Skill. Intuitively, a skill is an abstraction of a specific behavior (e.g., walking), and in practice, a skill is latent conditioned policy [15]. Given latent vector z, denote skill (a Sc, z), where Ty is the policy parameterized by 0. For instance, during pretraining, the latent vectors are sampled every n steps such that the latent vector is associated with the behavior executed during the associated n steps.
Mutual Information.
Knowledge- waweeddee-baddd..... lata-basedd and competence- based methods have different measures of surprise. The study in this paper falls into the category of competence-bar based methods. In partieular, data-based and competence- based methods rely information -thooretthe definition of surprise, ie., entropy. Previous competence-basee methods acquire skills by maximizing mutual information [14] between T and skills Z
[(T: Z) H H Z
(1)
= - H
(2)
where T can be the states the joint distribution of state-transitions SS) or the state-transitions (S' S). In particular, these methods differ in how they decompose the mutual information. Theoreti- cally, these different decompositions are equivalent, i.e., they all maximize the mutual information between states and skills. However, the particular choice greatly influences the performance in practice as optimizing this objective relies on approximations.
To motivate the potential of competence-basee methods over data-based or knowledge- based methods, we provide an intuitive understanding Eq. (1). On the one hand, the entropy term says that we want skills in aggregate that explore the state space; we use it as proxy to learn skills that cover the set of possible behaviors. On the other hand, It not enough to learn skills that randomly go to different places. We want to reuse those skills as accurately possible, meaning to discriminate or predict the agent's 's state transitions from skills. To do so, we minimize conditional entropy. In other words, appropriate skills should cover the set of possible behaviors and should be easily distinguishable.
need to be able
Intormation-Thorreee Skill Discovery
Competence basseennce-eeeeddddeee methods employ different intrinsic rewards to maximize mutual information: (1) discrininabillty-bassee and (2) exploratory-based intrinsic rewards. For example, the former rewards the agent for discriminable skills. In contrast, the latter rewards the agent for skills that effectively cover the state space using KNN density estimator 44 33 to approximate the entropy term. Below we analyze both approaches.
3

---

3.1
Discriminabilit--ban Intrinsic Reward
Previous work such as DIAYN 15 decomposition in Eq. and given a variational distribution To. they optimize the following variational lower bound
uses a discriminability- based intrinsic reward. They use the
T(S; 2) (KL[(((s, 2) p(s)p(z))
log 90 s)
E (x
+ (Psp((s) [KL(p(z |)) gø(2 s))]
p(z)
1 (xxxpp(x.s) [log go
s)] Exmp(()(p(())
where ~ p(Z) is discrete random variable. In particular, the discriminator rewards the agent it can guess the skill from the state, i.e., rrm - log 9 s) log p(z). These methods may easily run into chicken and egg problem, where skills learn to be diverse using the discriminator's s output. However, the discriminator cannot learn to discriminate skills if the skills are not diverse. Hence, it discourages the agent from exploring. Previous work [45] has tried to solve this problem by decoupling the aleatoric uncertainty from the epistemic uncertainty.
Furthermore, solving the chicken and egg problem is not enough since a state must map to a single skill in Eq. (2) Accordingly, the methods that use the decomposition in Eq. (2) require the skill space to be smaller than the state space so that the skills are distinguishable. Since previous work showed that high-dimensional continuous skill space empirically performs better, the intrinsic reward should not rely on the decomposition in Eq. 2). Intuitively, a continuous skill space allows (1) to interpolate between skills and (2) to represent more significant set of skills in a more compact representation.
Instead, Eq. (1). in contrast to skills that are predictable by states Hi Z T]. we require that states predietable by skills H[T Z]. Since any state should be predictable from given skill, the skill space must be larger than the state space (i.e, we do not want a skill mapping to multiple states). Therefore, another work [43] relies on variational bound on Eq. ①
log (s' 8,
T(S'; Z S) = (.2...5
+ E (..ppp)z [L s,2) ga |s,)))]
p(s' ()
log (s' s,
E /'
p(s' s)
which uses continuous skill space; however, does not scale high dimensions the intrinsic reward pi''(ss,, a) that p(z s) p(z). Intuitively, given s, we assume each element in the latent vector (z), to be independent of each other, the error of this assumption will be scaled by the dimension of z.
s) A log [a(s'(s,z) relies on an estimation of p(s' s). In particular, they assume
p(s'())
3.2 Exploratory-based Intrinsic Reward
As aforementioned, discriminability-oee bbased intrinsic reward may easily run into a chicken and egg problem. Hence, other work (28 32 uses exploratory- bbssed intrinsic reward to address the chicken and egg problem. In other words, they use the decomposition in Eq. where the intrinsic reward explicitly rewards the agent for exploring through [[[].
Previous work [R] maximizes the state entropy, i.e,, T = However, the authors in 4 argue that often results in the discriminator simply memorizing the last state of each skill. Instead, CIC [28] proposes to maximize the entropy of the joint distribution of state-transitions (which from now on we refer as joint entropy), i.e., [S' S].
Therefore, CIC 28
proposes to estimate the conditional entropy H[S', S Z] using noise contrastive and the joint entropy HI(S', S using a k-nearest neighbor estimator 44 to handle
estimation 28 23
high dimensional skill space.
KNN-density estimation. Previous works 33
28
approximate the joint entropy H[S'' [| using
k-nearest neighbor estimator 44). Given a random sample of size N, {(S()) ) P(S',

---

the estimator computes a Monte Carlo estimate 77 of the entropy defined as
H S
Σ log da (s
( ,s()) + b(k),
N
where Pu (s()) ss()
2 is an estimation of p(s s()), V2 is the volume of the hypersphere NV
of radius R and b(k) is a constant such that the estimator is unbiased. particular, Rg the distance from to its k-nearest neighbor s/. where we assume uniform distribution of points within the hypersphere (local uniformity assumption (34).
In practice, to make the distance R& meaningful, we do not operate in the state space. For instance, CIC 28 trains an MLP f RR -> IR that projects the state into a latent representation such that Rg (s) [(x()) where d is the skill dimension. Moreover, in APT [33], it was found that averaging over all k-nearest neighhors leads to better results. Hence, CIC 28 optimizes a quantity proportional Eq. 3
N Σ log
[ S] A
c+
c
1 H
n
where is constant for numerical stability. We view the joint entropy as an expected reward, and for a transition (s()), s the reward function
(s(() s(0)) & log c+
R(0)
k
Noise Contrastive Estimation. To train parameters t/, CIC (2 minimizes the conditional entropy
H S Z] using noise contrastive estimation (NCE) 23 37 Denote h f (s) and h' fu (s') the projected representations of s and s', respectively. Furthermore, let go. RRd - IR be an MLP that projects the skill into a latent representation and Jo. .R x Rd - Rd be MLP that projects h and h' into a latent representation. We define the NCE loss
Luce (s)
-log
exp c
Σ expo (1)
as an approximation to IH[S" 2), where we define cos (---) as the cosine similarity operator, w ) Io. (h()), serves as representation learning for parameters v.. Intuitively, the NCE loss (3] pushes p(s' s, z) to be a delta-like density function. However, since we define behaviors with n steps, we want wider distribution, so in practice,
as temperature term, and cj cos (go. (z)
As aforementioned, this term
to 0.5 smooth the distribution.
Mixture Of Suprises (MOSS)
Our goal is to provide algorithm that does not build any assumption about the entropy of the environment's s dynamics. A straightforward way to achieve this is to maximize and minimize surprise simultaneously. A benefit of such objective is that promotes exploration to find more effective policy for minimizing the surprise [6]. In particular, in [6], they maximize surprise according to all prior experiences while minimizing surprise according to the current episode. However, they find that it only helps in some cases. The challenge is to find an effective way to optimize these two opposite objectives since if we add the two objectives, would simply yield linear interpolation of the two objectives, which still corresponds to either maximizing minimizing. A previous work [17), dubbed Adversarial Surprise (AS), resolves this challenge using an adversarial game where two policies compete against each other: one maximizes surprise while the other minimizes surprise. AS [I7] requires training two policies that not share parameters or data. Instead, we propose a novel yet simple mixture of policies to maximize and minimize surprise simultaneously. Our mixture of policies does not rely on an adversarial game [17 in which training known to challenging 355 and uses single network for both objectives resulting in higher sample efficiency. Mainly, our method only
5

---

Algorithm 1 MOSS: Unsupervised pretraining
Input:
Environment, number of pretraining steps Nr. Initialize DDPG, and de, 221
fort t = to Nr do
if (t%update skill every) = then
Sample my p(m)
> setting to m = 0 is equivalent to CIC 28
Sample 2, ~ p(z m²)
end if
Take some action a, and observe st+11
Store (sr,8t+11 M,, 2z) into buffer B i 2 4000 then
Sample a batch {(al0, s(), s00' m (6
(0) and compute intrinsic reward اااا using equa-
Update DDPG using intrinsic reward.
Update de. Ps, yo using noise contrastive loss in equation 5
end if' end for
requires minor change CIC (28) that does not involve any additional computational cost during training. In particular, our mixture of policies builds on a competence-basee approach [28), meaning use two disjoint skill sets. We separate the two opposite objectives using different skill set for each objective: one skill set for surprise maximization and another for surprise minimization.
Specifically, we define the mixture of policies using binary random variable M, where we associate each mixture component with continuous uniform distribution 44 b) on an interval (a, b). Each uniform distribution constitutes a skill distribution. Maintaining two different skill distributions allows us to define a different objective for each distribution. In particular, we define two distributions Zmax ~ PZ M 0) and Zmin P(Z M = 1). Since we do not assume any priors on the entropy of the environment's dynamics, we deterministically set M
for the first half of the steps and .M- for the other half of the steps in the episode. Compared to more sophisticated methods (e.g., appendix B) for setting M (i.e., switching the objective), such a heuristic adds no computational cost during training, requires no hyper-parameter tuning, and performs well.
Moreover, motivated by the discussions in Section 3.2 we build our method on top of the CIC framework. This framework is appealing as it bypasses the chicken and egg problem and supports high-dimenssonal skills. In addition. it significantly outperforms previous competence-bed based methods on the URLB benchmark [29]. Hence, following CIC (28), we use the decomposition in Eq. ①
I(S' (: H[S' HS' S Z].
In particular, define surprise the joint entropy, which corresponds to maximizing and minimiz- (28) we use Eq. (4) to approximate the joint entropy. We approximate the conditional entropy H H S Z using a noise contrastive estimation following Eq. ( By minimiz- ing the conditional entropy. we distill behaviors into skills. Doing so also serves as representation learning for computing the joint entropy (28).
ing HH[[S' S). and as in CIC
To ensure fairness, all hyper- parametrssssssrss are kept in CIC 28 and we use the same RL algorithm (i.e., DDPG [1] as implemented in DrQ-V2 V2 (5I).
Implementation.
In practice, Zmax P(Z M = 0) A (0, 1) and Zmin P(Z M= = & (-1,0) (we refer readers to appendix for other variants that we tried), where we set M = for the first half of steps and M- for the remaining half of steps in the episode. Hence, the RI. agent maximizes
R F(0) E F.max (1. Pamms (8' a) [log Pr (8',)) E B E Prmin (8' log Primi (s)s
*The implementation provided CIC version (https://arxiv org 2202 (0161v22 which is an updated version of ICLR version (https // //openreviev net//eruuu -d-iktdkggkkkkkoo KodkkkGAKKAAAXX In particular, in the arxiv version, the intrinsic reward only relies on the KNN estimates and the NCE term is only used to train d., Φx, tb. While, the ICLR version includes the NCE term in the intrinsic reward.
is based the arxiv

---

And the intrinsic reward is a function of the state-transition and the binary random variable M
log
c Ex R
M = 0,
int
( s (
M)
1
(6)
log c+
>
M = 1.
As in CIC 28 we approximate the joint entropy using Eq.
MOSS's s implementation is straightforward; i.e., at each time step, instead of sampling Z P(Z). we sample Z P M) and replace the intrinsic reward with Eq. (6). Pseudocode for MOSS
provided in Alg. In fact, fixing M 0 yields CIC [28
5 Experiments
We present the main results of method by evaluating on the Unsupervised Rein- forcement Learning Benchmark (URLB) [29]. which is standard unsupervised RL benchmark for
Environments.
continuous control. URLB 29 has three different domains listed in order of increasing difficulty:
Walker is biped robot domain constrained to 2D vertical plane. The challenge for this domain is to learn how to balance and maneuver simultaneously.
Quadruped is four-legged robot in 3D space. This domain has higher dimensional state and action space than the Walker domain.
Jaco is a 6-DOF robotic arm domain with three-finger gripper in 3D space that has to learn action constraints and manipulation.
Furthermore, each of the three domains has four different downstream tasks with varying difficulties. We refer readers to Tab. for the complete list of 12 tasks. Finally, in OpenAl gym environments robot falls, the agent will naturally avoid falling the
[] since an episode terminates
ground as it will get a reward. Hence, during pretraining, the OpenAl gym environment { may
leak some task information (e.g., standing) the unsupervised learning agent (28). Therefore, URLB 29 builds on top of the Deepmind Control Suite [88) to avoid leaking some task information.
The domains in URLB 29 use low-dimensional state information, deterministic transitions, and non-evolving environments. In contrast, ViZDoom [27 uses raw pixel observations, stochastic transition, and evolving environments. Furthermore, since the environment randomly spawns enemies that shoot fireballs, the agent faces external perturbations that surprise the agent. We evaluate our method the DefendThelln and TakeCover maps. Please refer Figure 2 for environment renders.
200
80%
(a) URLB based on DMC
(b) The ViZDoom Environment.
Figure 2: Evaluation Environments. We evaluate method in (a) URLB, standard unsupervised reinforce- ment learning benchmark based on continuous control (b) ViZDoom pixel observation based reinforcement learning environment based on Doom game with multiple maps.
Evaluation. follow the benchmark's standard training procedure by pretraining the agent for million steps in each of the three domains and then finetuning the pre-trained agent for 100k steps with downstream task rewards. Since AS trains two network-independent policies, we ensure both policies each perform 2 million environment steps during pretraining. To ensure faimess, we pretrain
7

---

and finetune each method with the same 12 seeds (0-11) using the code provided in URLE 29 We present the results in Fig. for a total of 1584 (= algorithms x12 tasks x12 seeds) runs. Our main evaluation metrics are interquartile mean (IQM) and Optimality Gap (OG) of normalized scores, where we used the score of DDPG [51] agent trained from scratch for million steps as the expert scores (29). On the one hand, IQM is more robust to outliers than the mean, and IQM is significantly less biased than the median [2]. On the other hand, the Optimality Gap measures how far scores are
from the expert scores 2
Baselines. Baselines include methods presented in the URLB benchmark (29] as well as CIC [28] The baseline methods fall into srowledge-basees methods, ie.. ICM [88 Disagreement [39] RND
[0]. data- based methods, i.e., APT
AS 17 and competence-based methods, We refer readers the Appendix of URLB 29 for details on the different baselines. Lastly, since the two policies (Alice and Bob) in AS are
33 Proto-RL
52
i.e. SMM
30]. DIAYN [15), APS 322,
CIC
28
parameterrinndeeedeen, we pretrain for million steps, each policy trained using 2 million steps.
We evaluate our agent in ViZDoom 27 against CIC (28 and a modified version of CIC (28). dubbed NegativeCIC, where the intrinsic reward the negative joint entropy. Since the ViZDoom environment has high natural environment dynamics entropy, NegativeCIC should act as strong baseline. We adopt the same pretraining and finetuning procedure as URBL for ViZDoom.
MOSS. We keep all hyperparameters those in CIC [28]. Specifically, unless specified otherwise, we set M = 0 for the first half of steps and M = 1 for the other half of steps in the episode. We refer readers to the Appendix for more details.
[MC ml pretraan tratur
Med
CM
Hean
Oytimality Eap
1.60
mo05
0.75
Ad
0.50
stag
0:25
Sky?
0:30
0.45
0.00
0.7
6.50
0.50
3.75
0.45
0.00
0.75
0.50
0.5
0.00
0.00
1.
Score
(a) Aggregate Statistics. MOSS obtains the highest IQM and the lowest Optimality Gap.
(b) Performance Profiles. MOSS stochastically dominates all other methods.
Figure 3: Main results. We report aggregate statistics (a) and performance profiles (b) following for 12 downstream tasks URLB with 12 seeds, providing total of 144 seeds.
5.1 Results
Table 1: Numerical results. Following URLB (2) show the mean and standard error over 12 seeds (0-11).
We denote competence-basel methods, knowledge-basee and data-based methods in blue, black, and red, respectively.
Domain
Walker Kun
Quadruped Run
Jaco Bomom Right
Tisk
Plip
Stand
Walk
Jump
Stand
Walk
Botton
'Top Left
Right
38
150
868
568
452115
234118
Disggremmet
313c8
10629
512e14
6864230
358425
12082
112+10
RND 10
412-18
267-18
842-19
604-26
681e11
455+7
875+25
581442
10646
107+5
ICM 33]
590=24
491a18
94923
850g22
508.44
240124
676144
468152
11415
120g3
11624
11428
Proto (22
378-4
225=16
835-24
610s40
426e33
310+22
702459
348+55
130+12
131=11
134+12
146+10
AS Beb
475:16
247:23
917236
675x21
449a21
285123
598137
353139
116121
166:12
143112
139
AS-Alior
491:20
211
868e47
655s36
415a:0
186
590141
337417
109+20
14
140:17
129e15
SMM [D)
428.8
345a3]
924-9
731643
271
222:23
388151
167420
5245
55a2
5362
5745
DIAYN L5
306+11
146-7
304-22
662438
373+19
35+5
35-6
31x5
355218
166215
206a16
379431
192417
561.
CIC
715-40
968=1
7176466
4601436
14748
150m5
130+)
MOSS (Ouns
729
531
962-3
94265
674
48566
911611
635136
15115
150a5
150u5
15016
'Our experiments are based on the code of URLB benchmark and CIChttps://gtthub. http://tthtt..commmlleeerrrrr//a
8

---

To ensure the reliability of our main results, we report results as described in [2] to account for the uncertainty. We present results in Fig. 3 where, surprisingly. our method obtains stae-of-the-aat performance the URLB benchmark [29] on our main evaluation metrics [2] In particular, the performance profile in Fig- h)) reveals that MOSS stochastically dominates other methods. We
also present numerical results in Tab. for detailed breakdown over individual tasks.
For the IQM metric, report increase of 6% compared to CIC, which the second leading method. Furthermore, as for the Optimality Gap metric, we decreased 20% compared to CIC. Furthermore, in Fig. (b) previous methods intersect multiple points (making it hard conclude whether method is better than the other), while MOSS is always above all previous methods. In addition, in Fig. 3)), MOSS maintains all runs greater than a normalized score as high as 0.5.
Table ViZDoom Results. We report maps from ViZDoom to illustrate the robustness of method in different environments.
Defend the Line
Take Cover
NegativeCIC MOSS
462+11 510+22
51793+2936 63497+22884
We also evaluated our method on ViZDoom 27 Since the TakeCover and DefendTheLine maps NegativeCIC the baseline. We present the numerical results in Tab. 2] Since the environments ViZDoom are highly stochastic, we remove the top and bottom 10% of scores and report the mean and standard error to account for any outliers. Like in the main results presented earlier, MOSS consistently outperforms the baseline method under this completely new setting.
in ViZDoom provides natural perturbations,
1000
MESS
MCS5
150
Perfrmace
100
-50
Waleer Tip
Walcer
Waltar Stand
Waller Walc
lask
(a) Ablation on environment. Adding stochasticity in the URLB environment hurts CIC. We notice that MOSS is significantly more robust to the stochas- ticity than CIC.
(b) Ablation on policy. Removing the mixture significantly hurts the performance.
Figure 4: MOSS ablation experiments on the Walker domain (29 In (a), we inject stochasticity by adding
a Gaussian noise on the agent's action according to Bernoulli with p = 0.3 and report performance loss. In (b), we report performance for CIC, MOSS without the mixture of policies, and NegativeCIC represents CIC with the intrinsic reward the negative joint entropy.
and NegativeCIC. Specifically, MOSS SSSSS represents
5.2 Ablation
Since MOSS does not make assumptions about the entropy of the environment's dynamics, we are interested in knowing whether adding stochasticity affects performance. For this stochastic setup. during each interaction with the environment, we sample Y ~ Bernoulli(O33) and if Y = 1. we add a Gaussian Noise (0, 0.2) to the agent's intended action. In Fig. a), we observe that CIC's 28 performance drops while MOSS' performance is almost unaffected. This finding empirically supports previous works which claim that solely maximizing surprise for unsupervised RI. agents sometimes not enough, especially in stochastic environments (6 41)

---

In MOSS, we maintain two different skill distributions, i.e., Zmax ~ (.. 1) and Zmin ~ ~ (-1,0) In partieular, setting min to the same distribution as Zmax ~ (0, 1) is equivalent to directly optimizing CIC [28] with the two objectives simultaneously. We dub MOSS with the same skill distribution, MOSS_SAME SAME. In Fig. bb), we observe that naively doing so can result in poor performance, showing the effectiveness of our method in mitigating the simultancous optimization of two contradicting objectives.
Moreover, as shown in Fig. 4b), NegativeCIC performs poorly on the URLB benchmark 29] since most tasks required structured exploration in a low entropy environment.
6 Discussions & Limitations
We presented MOSS, an unsupervised RL method that does not make assumptions about the envi- ronment dynamics' entropy by simultaneously maximizing and minimizing surprises. Our method falls into competence- methods. Leveraging a mixture of policies for more than surprise maximization and minimization is an interesting future research.
Limitations. One limitation is that our method has a simple and effective deterministic rule for switching between the two objectives independent of state history. We did explore more sophisticated method Appendix B that had a better performance but required tuning and extra computation. Therefore. promising direction is to explore methods that take advantage of state information (e.g., using a meta-controller (26) to output a more optimal switching mechanism that incentivizes deep structured exploration 40 Moreover, we only considered competence based
methods. Extending our method to other unsupervised RL methods would be interesting to investigate in future work. Additionally, when finetuning, we transferred network weights which could lead to unlearning of already learned behaviors [11] It would be meaningful to combine with works that circumvent this issue like [I]] which utilized frozen pretrained policies to improve downstream finetuning further. Lastly, we only considered two objectives, and including more meaningful objectives like empowerment or anxiety (36 46 during pretraining could be an exciting direction.
Potential Negative Impact. Since we operate under the unsupervised reinforcement learning domain, the agent receives rewards that incentives exploration. Therefore, agents under this domain can be more unpredictable than agents that receive task rewards for intended behaviors. Furthermore, there is no current safeguard for the agent to not act maliciously in its environment; the agent may potentially cause harm to properties situated in the environment, including the agent itself. Therefore, unsupervised learning with safety constraints would be an exciting area of future research to mitigate this potential negative impact.
Acknowledgement. This work is supported in part by the National Science and Technology Major Project of the Ministry of Science and Technology of China under Grants 2018 1018AA0101604 the National Natural Science Foundation of China under Grants 62022048 and the State Key Lab of Autonomous Intelligent Unmanned Systems. We also appreciate the generous donation of computing resources by High-Flyer AI.
References
[1] Joshua Achiam, Harrison Edwards, Dario Amodei, and Picter Abbeel. Variational option discovery algorithms, arXiv preprint a8Xi: 10299, 2018
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
[3] Arthur Aubret, Laetitia Matignon, and Salima Hassas. A survey intrinsic motivation in reinforcement learning. arXiv preprint arXiv: 06976, 2019.
[4] Kate Baumli, David Warde-Farley, Steven Hansen, and Volodymyr Mnih. Relative variational intrinsic control. arXiv preprint arXiv:2012.0783) 2020.
[5] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-basee based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
10

---

[6] Glen Berseth, Daniel Geng, Coline Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayaraman. and Sergey Levine. Smirl: Surprise minimizing reinforcement learning in unstable environments. arXiv preprint 1rx.v:191..05500 2019.
[7] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[8] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.0155: 2016.
[9] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leamerrs. Advances in neural information processing systems, 33:1877-1901, 2020.
[10] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: wtt.i18800..9.44. 12894, 2018.
[11] Victor Campos, Pablo Sprechmann, Steven Hansen, André Barreto, Steven Kapturowski, Alex Vitvitskyi, Adrià Puigdomenech Badia, and Charles Blundell. Coverage principle for discovering transferable behavior in reinforcement learning. CoRR, abs/2102.13515, 2021.
[12] Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
[13] Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning. Advances in neural information processing systems, 17, 2004.
[14] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[15] Benjamin Eysenhach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
[16] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of unsupervised reinforcement learning. urXiv preprint arXiv:2110.02719, 2021.
(17] Amaud Fickinger, Natasha Jaques, Samyak Parajuli, Michael Chang, Nicholas Rhinehart, Glen Berseth, Stuart Russell, and Sergey Levine. Explore and control with adversarial surprise. arXiv preprint arXiv:2 w0t.:2007.0394 2021.
[18] Karl Friston. The free-energy principle: a unified hrain theory? Nature reviews neuroscience, 11(2):277
138, 2010.
[19] Karl Friston, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, Giovanni Pezzulo, et al. Active inference and learning. Neuroscience & Biobehavioral Reviews, 68:862-879, 2016.
[20] Karl Friston, James Kilner, and Lee Harrison. A free energy principle for the brain. Journal of physiology- Paris, 100(1-3)70--77, 2006.
[21] Karol Gregor. Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. ArXiv, abs/1611.07507, 2017.
[22] Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-learning for reinforcement learning. arXiv preprint arXiv: 806604 2018
[23] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings. 2010.
[24] Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint arXiv: 05030, 2019.
[25] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint
2021.
[26] Maximilian Igl. Andrew Gambardella, Jinke He, Nantas Nardelli, Siddharth, Wendelin Bochmer, and Shimon Whiteson. Multitask soft option learning. In Jonas Peters and David Sontag, editors, Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), volume 124 of Proceedings of Machine Learning Research, pages 969-978. PMLR, 03-06 Aug 2020.
[27] Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaskowski. Vizdoom:
A doom-based ai research platform for visual reinforcement learning. In 2016 IEEE conference on computational intelligence and gamey (CIG), pages 1-8. IEEE, 2016.
[28] Michael Laskin, Hao Liu, Xue Bin Peng. Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. CIC: Contrastive intrinsic control for unsupervised skill discovery, 2022.
11

---

[29] Michael Laskin. Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021.
[30] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint 19xivv:906.0555
2019.
[31] Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 1509.02971, 2015.
[32] Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference Machine Learning, pages 6736-6747. PMLR, 2021.
[33] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34, 2021.
[34] Damiano Lombardi and Sanjay Pant. Nonparametric k-nearest- neighbo entropy estimator. Physical Review F, 93( 1) ):01 3310, 2016.
[35] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05953 2018.
[36] Shakir Mohamed and Danilo Jimencz Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. Advances in neural information procexsing systems, 28, 2015.
[37] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.0374 037448, 2018.
[38] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driver exploration by self-supervised prediction. In International conference on machine learning, pages 2778-2787. PMLR, 2017.
[39] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supe exploration via disagroement. In International conference on machine learning, pages 5062-5071. PMLR, 2019.
[40] Miruna Pislar, David Szepesvari. Georg Ostrovski, Diana Borsa, and Tom Schaul. When should agents explore? arXiv preprint arXiv:2108 11811, 2021.
[41] Nicholas Rhinehart, Jenny Wang. Glea Berseth, John Co-Reyes, Danijar Hafner, Chelsea Finn, and Sergey Levine. Information is power: Intrinsic control via information capture. Advances in Neural Information Processing Systems, 34, 2021.
[42] Archit Sharma, Michael Ahn, Sergey Levine, Vikash Kumar, Karol Hausman, and Shixiang Gu. Emer- gent real-world robotic skills via unsupervised off-policy reinforcement learning. arXiv preprint arXiv:2004. 12974, 2020.
[43] Archit Sharma, Shixiang Gu, Sergey Levine. Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: 10XX:v:/00.000 1907 2019.
[44] Harshinder Singh, Neeraj Misra, Vladimir Haizdo, Adam Fedorowicz, and Eugene Demchuk. Nearest neighbor estimates of entropy. American journal of mathmatical and management sciences, 23(3-4):301- 321, 2003.
[45] DJ Strouse, Kate Baumli, David Warde-Farley, Vlad Mnih, and Steven Hansen. Learning more skills through optimistic exploration. arXiv preprint arXiv:2107. 14226, 2021.
[46] Chenyu Sun, Hangwei Qian, and Chunyan Miao. From psychological curiosity to artificial curiosity: Curiosity-driven learning in artificial intelligence tasks. arXiv preprint arXiv:2201.08300 2022.
[47] Richard Sutton and Andrew Barto. Reinforcement learning: An introduction. MIT press, 2018
[48] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint 1eXiv:880100069) 2018
[49] Manan Tomar, Utkarsh A Mishra, Amy Zhang, and Matthew E Taylor. Learning representations for pixel-based control: What matters and why? arXiv preprint arXiv:2111.0777,. 2021.
[50] Kelvin Xu. Siddharth Verma, Chelsea Finn. and Sergey Levine. Continual learning of control primitives: Skill discovery via reset-games. Advances in Neural Information Processing Systems, 33:4999-5010. 2020.
[51] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXi::2107.09645. 2021.
[52] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, pages 11920-11931. PMLR, 2021.
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. Please do not modify the questions and only use the provided macros for your answers. Note that Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as a URL)? [Yes] See supplementary material and Section S The code is in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, byperparameters, how they were chosen)? [Yes] See supplementary material and Section
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] See supplementary material.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] Yes code in the supplementary material
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "QhvG92gH0zP", "reviewer": "Reviewer_zGJ4", "summary": "The paper proposes a new method called Mixture of Surprises (MOSS), which tries to maximize and minimize the surprise at the same time. This results in state-of-the-art performance on the URLB benchmark. ", "questions": "Can this method also be used with demonstrations? ", "limitations": "The authors should try to motivate the novelty of the paper a bit more. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe paper is well written and covers the related literature quite well. The experiments are well done on variety of different environments. The proposed method also performs well on the URLB benchmark. \n\n\nWeakness: \nAlready exist methods which try to maximize and minimize surprise at the same time. Thus, it is difficult to find novelty in the proposed method. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DC7do0cQ9_C", "reviewer": "Reviewer_x6bw", "summary": "The paper presents an improved algorithm based on CIC (Laskin et al. 2022), to balance surprise maximization and minimization for RL pertaining. Experiments on a variety of tasks show the effectiveness of the proposed methods.", "questions": "- It would be better to add MOSS results in Figure 4b) for direct comparison. It makes sense that MOSS_SAME performance is in between CIC and negative CIC, the question is why MOSS can do better. One would argue that in expectation MOSS is still just a linear interpolation of CIC and negative CIC (or a scaled CIC). Why is this not the case?\n- How do you sample M? Suppose it's sampled from Bernoulli(p). Does setting p=0 give you CIC and settting p=1 give you negative CIC? What are the optimal p's for different environments? Would that somewhat correspond to environmental stochasticity based on your assumption?", "limitations": "Limitations and the potential negative impact were discussed at the end of the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The motivation of balancing maximizing and minimizing surprises is well explained, but it's unclear how the proposed method resolves the issue and how much it is different from a scaled CIC. I suggest the author make a clear distinction between MOSS objective and a linear interpolation of CIC and negative CIC (or equivalently a scaled CIC), and also show in the experimental section how the prior distribution of M affects MOSS performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AXEBllPVy8X", "reviewer": "Reviewer_GiY2", "summary": "The paper presents a method for unsupervised skill discovery via reinforcement learning. Unlike most prior methods which seek to either maximize or minimize surprise, this paper proposes to learn a mixture policy where one component of the mixture is trained to minimize surprise, and another component is trained to maximize surprise. The paper builds very closely on the work of Laskin et al [1] (CIC), and the proposed change to CIC is very small (just a one-line change in Algorithm 1). Essentially, the proposed method partitions the latent skill space into two parts, where one part is trained to maximize surprise, and the other part if trained to minimize surprise. Whether the agent chooses to sample from the surprise-maximizing part of the skill space or the surprise-minimizing part is somewhat arbitrary: the agent uses surprise maximizing skills for the first half of an episode, and surprise-minimizing skills for the second half of the episode. \n\nThe authors evaluate their method on the URLB benchmark and VizDoom tasks, and show that their method slightly outperforms CIC on both unsupervised learning metrics as well as downstream task performance. The main advantage of the method is that, unlike CIC, we do not need to pre-specify whether the agent should minimize or maximize surprise, and the algorithm is able to do well in environments that require either. \n\n[1] Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. CIC: Contrastive intrinsic control for unsupervised skill discovery, 2022.", "questions": "In the Section 5, you say “but performed well on the standing task, which is inherently a low entropy task.” → did you mean to say high-entropy task instead? \n", "limitations": "I don't think the limitations are particularly well-addressed in the paper. The main limitation mentioned was that the proposed method is only applicable to episode RL settings (and not reset-free settings), but this seems slightly orthogonal to the main issue of the paper. I think a more detailed discussion on how to make the objective switching more adaptive would make the limitations section much stronger. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Simple, easy-to-implement method. \n- A somewhat more general technique for unsupervised reinforcement learning as compared to prior methods. \n- Writing quality is good. \n\nWeaknesses\n- The choice of when to switch between a surprise minimizing and maximizing objectives is somewhat ad-hoc. Authors investigate a slightly more sophisticated procedure in the Appendix, but find it to be very sensitive to hyperparameter choices, and omit it from the main paper. \n- Overall writing quality is good, but the authors spend a lot of time (2.5 pages) going over background material and summarizing past work, while the main method section is about 1 page long, including an algorithm box. The main method section does not start until the second half of page 5! Given that the paper builds very closely on CIC, a more detailed exposition on CIC and limited discussion of other techniques would have sufficed. \n- Some of the related work that also proposes to maximize and minimize surprise was only mentioned in the last section. I think this should be moved up to be in Section 3 instead, where other related works are discussed. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-mPx8SsbOag", "reviewer": "Reviewer_w2pW", "summary": "This paper proposes an unsupervised RL algorithm for pretraining a policy. The algorithm is motivated by learning to both maximize and minimize the mutual information between latent skill variables and the state-transition joint distribution. The method approximates this mutual information following prior work, and proposes a learning algorithm that simply splits up the pretraining episodes into two phases: the first phase with intrinsic rewards corresponding to maximization, and the second phase corresponding to minimization. Experiments on the environments and tasks from the benchmark of [28] illustrate that this method achieves superior finetuning results compared to a variety of other methods. Ablations illustrate that this method has less degradation in performance than a maximizing phase alone (corresponding to prior work) when additional stochasticity is added to the transition dynamics, and an ablation that illustrates the importance of having the latent skill priors be different.", "questions": "- Please address the weaknesses:\n  - Can you fix the terminology about the three types of methods?\n  - Can you fix the characterization of the objective function?\n  - Can you fix the ambiguity in the connection between the objective function and the optimization procedure?\n  - Can you include the two baseline evaluations I requested?\n  - Can you include the ablations I requested?\n  - Can you include an empirical comparison to [18]?\n- L170: \"Intuitively, the NCE loss [ 23 ] pushes p(s′ | s, z) to be a delta-like density function. However, since we define behaviors with n steps, we want a wider distribution\" The second sentence does not follow from the first. Even if p(s'|s,z) is deterministic, it can still be applied for N steps, to yield an N-step behavior. Can the paper clarify these statements?\n- Can you explain the following inconsistency? L260 claims that setting Z_min = Z_max is equivalent to CIC, yet this results in different empircal performances in Fig 4b. By L260, we should expect their results to be equivalent. Why are they not? Or perhaps it's not equivalent to CIC, because it involves flipping the sign of the intrinsic reward function?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The area of investigation is relevant to the community. Pretraining / unsupervised learning for RL agents is an active area of research interest.\n- The proposed method is technically sound.\n- The proposed method is technically straightforward.\n- The proposed method demonstrates compelling performance compared to prior work.\n- The experimental evaluation uses compelling aggregate statistics\n- The experimental evaluation compares to a large variety of prior methods\n\n# Weaknesses\n- There is some undefined terminology. \"Competence\"-, \"Data\"-, and \"Knowledge\"-based methods were never defined. This makes some parts hard to understand, e.g. L99 \"To motivate the potential of competence-based methods over data-based or knowledge-based methods\". I recognize that these terms come from CIC, but this paper should either restate the definitions or refer to CIC.\n- I think the objective function is mischaracterized. L183 \"In particular, we define surprise as the state-transition entropy, which corresponds to maximizing and minimizing $H[S', S]$\". Unless I misunderstand the notation, this is the entropy of the *joint* distribution $p(s',s)$, which is *not* the entropy of the state-transition dynamics distribution $p(s'|s,a)$, nor the entropy of the state-transition dynamics induced by the policy $p(s'|s) = E_\\pi p(s'|s,a)$ (which are the only two distributions that I would refer to as being \"state-transitions\"). Therefore, I would not say that the objective is the state-transition entropy. It is *related* to the state transition entropy, since $H(S',S) = H(S'|S) + H(S)$, i.e. it is the entropy of the induced Markov dynamics plus another entropy. The problem with referring to this as only the \"state-transition\" entropy is that it fails to capture the fact that the value of H(S) affects the resulting value, so I think it is a mischaracterization of the objective. I think it is important to be very precise about this naming, because, in general, optimizing $H(S'|S)$ will lead to very different results than optimizing $H(S'|S)+H(S)$. \n- The connection between the original objective and the resulting optimization procedure is unclear. The objective is $I((S', S) ; Z) = H((S', S)) - H((S', S) | Z)$. The CIC paper, upon which Alg 1 is based, uses both a noise-contrastive estimator (for the latter term) and a KNN estimator (for the former term). Alg 1 states the intrinsic reward is computed using Eq. 9, but this seems to only involve the noise constrastive estimator. Is Alg 1 correct? Where is the KNN estimator?\n- The experiments do not adequately illustrate the effect that finetuning has on the pretrained policy. It is possible that the pretrained policy does not benefit from finetuning in some cases, e.g. taking $z \\sim Z_{min}$ might lead to a high score on environments where the downstream tasks is very related to entropy minimization. Can you add to the evaluations two baselines for non-finetuned policies constructed from each of the $Z$ distributions (i.e. one for $\\pi(a|s, z\\sim p(Z_{max}))$, another for $\\pi(a|s, z\\sim p(Z_{min})$)? These should not be finetuned, instead, simply illustrate how related the learned policy is to the downstream task. I would call these something like $MOSS_{max,frozen}$ and $MOSS_{min,frozen}$ . Note that this doesn't require any additional learning, just an evaluation of these existing policies. Beyond this, you could also include the finetuning learning curves in the appendix, although that's a bit less important.\n- The current ablation does not illustrate the importance of the choice of the structure of the Z priors, which includes both the dimensionality $d$, as well as the event space. One thing that seems most natural is simply to take z \\in {0,1}, where 0 defines 'minimizing' and 1 defines 'maximizing', i.e. use $P_{min}(Z) \\doteq \\delta(z=0)$, and $P_{max}(Z)\\doteq \\delta(z=1)$. It is not clear what the additional dimensionality of $Z$ is useful for (the appendix tells us its 64 dimensional). Some discussion and empirical are needed to justify this choice. It could be the case that very low dimensional Zs are still useful in some (or all) tasks, intuitively, these tasks might be those in which there is only one 'useful' way each to minimize or maximize entropy, rather than environments in which repertoires (>=2 each) of entropy maximizing and minimizing skills is useful. Defining the skill variables to be continuous essentially admits learning an infinite number of skills, but it is not clear that this is the best choice.\n- Although this paper pointed out the relevance, it did not include an empirical comparison to [18], which is also motivated by the task of learning to both maximize and minimize rewards. The inclusion of this comparison would significantly strengthen the empirical results.\n\n# Minor weaknesses\n- Please add the main method's scores to Figure 4(b).\n- By the paper's definition of surprise as the entropy of the joint distribution of state transitions, the method is not optimizing *only* surprise (since it's just one of the two terms of the mutual information). The method name then becomes something of a misnomer...", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept