Learning more skills through optimistic exploration
Authors: DJ Strouse
Kate Baumli
David Warde-Farley
Volodymyr Mnih
Steven Stenberg Hansen
Venue: ICLR 2022 Spotlight (2022)
Keywords: intrinsic control
skill discovery
unsupervised skill learning
uncertainty estimation
optimistic exploration
variational information maximization

Abstract: Unsupervised skill learning objectives (Eysenbach et al., 2019; Gregor et al., 2016) allow agents to learn rich repertoires of behavior in the absence of extrinsic rewards. They work by simultaneously training a policy to produce distinguishable latent-conditioned trajectories, and a discriminator to evaluate distinguishability by trying to infer latents from trajectories. The hope is for the agent to explore and master the environment by encouraging each skill (latent) to reliably reach different states. However, an inherent exploration problem lingers: when a novel state is actually encountered, the discriminator will necessarily not have seen enough training data to produce accurate and confident skill classifications, leading to low intrinsic reward for the agent and effective penalization of the sort of exploration needed to actually maximize the objective. To combat this inherent pessimism towards exploration, we derive an information gain auxiliary objective that involves training an ensemble of discriminators and rewarding the policy for their disagreement. Our objective directly estimates the epistemic uncertainty that comes from the discriminator not having seen enough training examples, thus providing an intrinsic reward more tailored to the true objective compared to pseudocount-based methods (Burda et al., 2019). We call this exploration bonus discriminator disagreement intrinsic reward, or DISDAIN. We demonstrate empirically that DISDAIN improves skill learning both in a tabular grid world (Four Rooms) and the 57 games of the Atari Suite (from pixels). Thus, we encourage researchers to treat pessimism with DISDAIN.

Published as a conference paper at ICLR 2022
LEARNING MORE SKILLS THROUGH OPTIMISTIC EXPLORATION
DJ Strouse; Kate Baumli, David Warde-Farley, Vlad Mnih, Steven Hansen' DeepMind
(strouse, baumli, dwf, vmnih, stevenhansen) @google com
ABSTRACT
Unsupervised skill learning objectives [Eyenbbahh et al. 2019 Gregor et al. 2016) allow agents learn rich repertoires of behavior the absence of extrinsic rewards. They work by simultaneously training policy to produce distinguishable latent-conditioned trajectories, and discriminator evaluate distinguishability by trying to infer latents from trajectories. The hope is for the agent to explore and master the environment by encouraging each skill (latent) to reliably reach different states. However, an inherent exploration problem lingers: when a novel state is actually encountered, the discriminator will necessarily not have seen enough training data to produce accurate and confident skill classifications, leading to low intrinsic reward for the agent and effective penalization of the sort of exploration needed actually maximize the objective. To combat this inherent pessimism towards exploration, we derive an information gain auxiliary objective that involves training an ensemble of discriminators and rewarding the policy for their disagreement, Our objective directly estimates the epistemic uncertainty that comes from the discriminator not having seen enough training examples, thus providing an intrinsic reward more tailored to the true objective compared to pseudocount- based methods (Burda et al.] 2019). We call this exploration bonus discriminator disagreement intrinsic reward, or DISDAIN. We demonstrate empirically that DISDAIN improves skill learning both in tabular grid world (Four Rooms) and the 57 games of the Atari Suite (from pixels). Thus, encourage researchers to treat pessimism with DISDAIN.
INTRODUCTION
Reinforcement learning (RL) has proven itself capable of learning useful skills when clear task- specific rewards are available (OpenAl et al. 2019a| bj Vinyals et al. 2019). However, truly intelligent agents should, like humans, be able to learn even in the absence of supervision in order to acquire repurposable fask-agnostic knowledge, Such unsupervised pre-training has seen recent success in language (Radford et al. 2019 Brown et al.; 2020) and vision (Chen et al. 2020a b), but its potential has yet to be fully realized the learning of behavior.
The most promising class of algorithms for unsupervised skill discovery is based upon maximizing the discriminability of skills represented by latent variables on which a policy is conditioned. Objectives typically derived from variational approximations
the mutual information between latent variables and states visited (Gregor et al. 2016 Eysenbach et al. 2019 Warde- et al., 2019 Hansen et al. 2020 Baumli al., 2222] employing a learned parametric skill discriminator, Both the policy and discriminator have the objective of strong predictive performance by the discriminator, though the discriminator trained with supervised learning rather than RL. The end result a policy capable of producing consistently distinguishable behavioral motifs, or "skills." These skills can be evaluated zero-shot, fine-tuned, or composed in a hierarchical RL setup to maximize task reward when one is introduced (Eysenbach et al. 2019 Hansen et al.) 2020). Unsupervised skill learning objectives can also be maximized in conjunction with task reward, in order to promote robustness of
learned behavior to environment perturbations (Mahajan et al. 2019 Kumar et al. 2020).
"quall contribution

---

Published as a conference paper at ICLR 2022
The degree to which unsupervised skill discovery methods are useful in such downstream applications depends on how many skills they are able to learn. Indeed, [Esenbach et al. (2019) showed that as more skills are learned, the performance obtained using the learned skills in hierarchical reinforcement learning setup improves. In follow up work, Achiam et (2118) showed that methods like DIAYN can struggle to learn large numbers of skills and proposed gradually increasing the number of skills according to a curriculum to make skill discovery easier.
The aim of this work is to improve the ability of skill discovery methods to learn more skills. We highlight an exploration problem intrinsic to the entire class of variational skill discovery algorithms which can inhibit discovery of new skills. During skill learning. the policy will necessarily need to explore new states of the environment. The discriminator must then make latent predictions for states it has never seen before, resulting incorrect and/or low-confidence predictions. The policy will in turn be penalized for this poor discriminator performance, and discouraged from seeking out new states. We refer to this problem as "pessimistic exploration" and describe further in section P
To motivate our solution, we argue that it is important for the policy to distinguish between two kinds of uncertainty in the discriminator aleatoric uncertainty that comes from the policy producing similar trajectories for different skills, and epistemic uncertainty that comes from a lack of training data. The former indicates poor policy performance, but the latter is in fact desirable, and serves as a signal for potential discriminator learning. Unfortunately, skill discovery algorithms treat both types of uncertainty the same and thus ignore this important signal. We propose to capture it.
Our primary contribution, which present in section + is exploration bonus tailored to skill discovery algorithms, designed to overcome pessimistic exploration. We identify states of high epistemic uncertainty in the discriminator by training an ensemble of discriminators and measuring their disagreement. We call this exploration bonus discriminator disagreement intrinsic reward, or DISDAIN. Intuitively, the ensemble members may disagree in novel states, but must come to agree in frequently visited ones. More formally, we derive DISDAIN from Bayesian perspective that 1) represents the posterior over discriminator parameters using an ensemble, and 2) encourages the policy to maximize information gain (i.e. reduce uncertainty) about the parameters of the discriminator. In section 4 we demonstrate empirically that DISDAIN improves skill learning over unbonused skill discovery algorithms in both an illustrative grid world (Four Rooms, |Sutton et al. (1999)) and the Atari 2600 learning environment (A (ALE, [Bllemare et al. (2033) more so than augmenting with popular exploration bonuses not tailored to skill discovery.
2
UNSUPERVISED SKILL LEARNING THROUGH VARIATIONAL INFOMAX
2.1
INTRODUCTION
We now formalize setting of interest: unsupervised skill learning through variational information maximization (Gregor et al. 2016: Eysenbach et all![201) Warde-Farlcy et al. 2019: Hansen et al.)
2020 Baumli et al. (021). J. We consider Markov decision process (MDP), defined by the tuple M= (S, PE, P, r. r), where S and A are state and action spaces, respectively, the environment dynamics (e(s' s. a) specifies the probability of transitioning to state s' E S when taking action a E A in state s E S, p(s) denotes the probability of starting an episode in state and y E [0, 1) is discount factor. Since we focus on unsupervised training, we ignore the environmental reward r.
Our agents seek to learn a repertoire of skills, indexed by the latent variable Z and represented by the policy Te(a| s, 2). which is parameterized by 0 and maps from states and latent variables to distributions over actions. The latent variables are sampled 2 ~ p( Z) at the beginning of each trajectory and then fixed, so each z represents temporally extended behavior. The skill trajectory length T may differ from the episode length, thus new skill might be resampled within an episode.
For conciseness, we will denote trajectories sampled from the policy by T TT2) when conditioning on a particular skill z, and 7 π when collecting trajectories across skills. To simplify our discussion, and because is the most common case practice, we will assume that is categorical with cardinality Nz, although much of the discussion carries over to continuous Z.
large and growing class of objectives for unsupervised skill discovery are derived from maximizing the mutual information between the latent skill Z and some feature of the resulting trajectories O(r):
F(0) = (2, 0) (Z) H(Z O) EE E===p(), plg o(F)) logp(=)). (1)
2

---

Published as a conference paper at ICLR 2022
For example, variational intrinsic control (VIC, Gregor al. (2016) maximizes I[Z Su, Sr) the mutual information between the skill and initial and final states (OT = (Su, $T )) Diversity is all you need (DIAYN, Eysenbach et al. (2019)), the other hand, maximizes 1 (2,5) the mutual information between the skill and each state along the trajectory (or = & for t E 1 :). Intuitively, VIC produces skills that vary in the destination reached (without regard for the path taken), while DIAYN produces skills that vary in the path taken (with less emphasis on the destination reached).
In practice, maximizing equation | is not straightforward, because it requires calculating the condi- tional distribution p(2 [ Q). In general, it is necessary to estimate it with a learned parametric model go(Z OO). We refer to this model as the discrininator, since it is trained to discriminate between skills. Fortunately, replacing with q still yields lower bound on F(0) (Barber and Agakov 2004). and we may instead maximize the proxy objective F(8):
F(0)= F(0) Bggylle), (x))) gs(= ((T)) log
Optimizing F(0) with respect to the policy parameters 0 corresponds to RL on the reward:
Tskill = log go 0) - logp(z).
(3)
Since the intent is for the agent learn a full repertoire of skills, the skill prior p(2) is typically
fixed to be uniform (Eysenbach et al. 2019 Achiam et al. 2018 Baumli et al.] 2021]. in which case - -oggp(( p(z) log Nz. If the discriminator simply ignores the trajectory and guesses skills uniformly as well, then log q (z 0) = log Nz and the reward will be zero. If the discriminator instead guesses perfectly, then log (2 o) = 0, and the reward will be log Nz. More generally, the expected reward is an estimate of the logarithm of the effective number of skills. Thus, measuring the
reward in bits (i.e. using log, in equation we can estimate the number of skills learnt as:
2
Akkillss
We adopt this quantity as our primary performance metric for our experiments.
is as tight as possible, the discriminator qa (Z O) must also To make sure the bound in equation 2
be fit to its target p( / O) through supervised learning (SL.) on the negative log likelihood loss:
L(Ø)= = -B)= [log ( o(r))]
The loss is minimized, and the bound in equation 2 tight, when ga (Z 00 p(7 10).
The joint optimization of F(() by RL and L(ø) by SL forms cooperative communication game p(Z) and generates the "message"
between policy and discriminator. The agent samples a skill T ~ {22) The discriminator receives the message T and attempts to decode the original skill z. When the policy produces trajectories for different skills that do not overlap in the features ((r), the discriminator will easily learn to label trajectories, and when the discriminator makes accurate and confident predictions, the reward in equation 3 will be high. Ideally, the end result is policy exhibiting a maximally diverse set of skills. This coint-traiiing system is depicted in Figure 2a
2.2 PESSIMISTIC EXPLORATION IN UNSUPERVISED SKILL DISCOVERY
A conflict arises between the exploration necessary for skill diversification and rewards supplied by an imperfect discriminator, trained only on past policy-generated experience. Without data (and in the absence of perfect generalization). the discriminator is likely to make poor predictions when presented with trajectories containing previously unseen states, resulting in low reward for the policy. Importantly, this penalization occurs regardless of whether the policy produces distinguishable akill-conditione trajectories if the current discriminator is locally poor approximation to p(Z O) for a region of state space represented in O. We note that this is distinct from issues of pessimism in exploration that arise more generally, including with stationary reward functions (Osband et al., 2019), wherein naive exploration strategies fail to adequately position an agent for further acquisition of information. In the scenario we examine here, the agent's sole source of supervision directly sabotages the learning process when the discriminator extrapolates poorly.
MMor accurately, Gregor et al. 2016) conditioned on initial state and used ((Z, ST So) = H | SS) (1 Sr). However, has subsequently become common not condition the skill sampling distribution on 80 (Eysenbach et al. 2019), in which case HZ | So)) (Z) and T(Z, Sr So) T(Z; Sr)-
3

---

Published as a conference paper at ICLR 2022
T T~7
T~ π (22)
qZ F)
21 VS 22?
< TT = 21 VS 22
(a) Past trajectories.
(b) Aleatoric uncertainty arises when different skills visit similar states.
(c) Epistemic uncertainty arises from exploring novel states.
Figure 1: The pessimistic exploration problem. Because skill discovery objectives do not distin- guish between aleatoric and epistemic uncertainty, they penalize exploration.
We argue that in order to overcome this inherent pessimism, we must distinguish between two kinds of uncertainty in the discriminator: aleatoric uncertainty (figure Lb) that due to the policy producing overlapping skills (i.e. high H (Z O)). and epistemic uncertainty (figure [c] that is due a lack of training data (i.e. poor match between (Z IQ) and p(Z O)). Naively, both kinds of uncertainty contribute to low reward for the policy (equation 3), but while reduction of aleatoric uncertainty requires changes by the policy, epistemic uncertainty can be reduced (and thus reward increased) simply by creating more data (i.e. visiting the same states again). Thus, argue that should "reimburse" the policy for epistemic uncertainty in the discriminator, and in fact encourage the policy visits states of high epistemic uncertainty.
Put another way, when the policy only maximizes the reward of equation 3 it maximizes the lower bound (0) < F(0) without regard for its tightness, despite that a looser bound means a more pessimistic reward. The job of keeping the bound tight left entirely to the discriminator (equation [5). By incentivizing the policy to visit states of high epistemic uncertainty, valuable training data is provided to the discriminator that allows better approximate its target and close the gap between F(0) and F(O). In this way, we encourage the policy to help keep the bound F(0) S F(0) tight.
In the next section, we formalize the intuitions outlined in the last two paragraphs. The result is an exploration bonus measured as the disagreement among an ensemble discriminators.
DISDAIN: DISCRIMINATOR DISAGREEMENT INTRINSIC REWARD
To formalize the notion of discriminator uncertainty, we take Bayesian approach and replace the point estimate of discriminator parameters o with a posterior p(ő). Maintaining a posterior allows us to quantify the information gained about those parameters. Specifically, we will incentivize the policy to produce trajectories in which observing the paired skill label provides maximal information about the discriminator parameters Ø:
I(Z: O) H(Z 0) H(Z 0,
(6)
Rewriting the entropies as expectations over trajectories, we have:
I(Z; 0) = Exax H f p()) as (Z o(T)) do) - p(0) (2 (()))) do
(7)
where we have used that p()) does not depend on the present trajectory and so p(o ((T))== p(o). Note that unlike in equations 2 and the expectation over trajectories is not skill-conditioned. The marginalization over Z happens in the entropy and over the discriminator's posterior go(= s) rather than the agent's 's prior p(z). This is because the discriminator parameters & are now part of the probabilistic model and do not just enter through a variational approximation.
How should we represent the posterior over discriminator parameters p(O)? There is consider- able work in Bayesian deep learning that offers possible answers, but here we take an ensemble approach (Seung et al. 1992 Lakshminarayanan parameters o; for the 2d discriminator. The discriminators are independently initialized and in theory could also be trained on different mini-batches, though in practice, found both sufficient and
al. 2017). We train N discriminators with
4

---

Published as a conference paper at ICLR 2022
IDISDAIN
H avg H H H
Skill Discriminator qZZ (o(s))
z PiZ)
Repeat every steps
r- log o(zto(e)) log piz)
Environment Pat 9.)
Skill-Condd Policy o, mis, 2)
(a) Skill discovery algorithms.
(b) DISDAIN.
Figure 2: Methods. (a) The skill discovery process, where joint optimization of a skill-conditioned policy and skill discriminator ensure reliable and distinct behavior for each skill. (b) DISDAIN: disagreement between an ensemble of skill discriminators informs exploration.
simpler to train them the same mini-batches, others have also found (Osband al. 2016). The posterior p()) is then represented as mixture of point masses at the Φ;:
p( (())
8(6 di).
(8)
N
Substituting the posterior in equation g into equation 7
we have:
1 N
112 Bra
H
go (Z (((T))
Σ
H (2 o(T))]
N
==1
Maximizing equation D with RL corresponds to adding the following auxiliary reward to the policy:
1 N
DDSSDAIN() = H
Σ 4o (Z
0.)
Has (Z oz)]
(10)
In words, this is the entropy of the mean discriminator minus the mean of the entropies of the discriminators. By Jensen's inequality, entropy increases under averaging, and thus rdisDAIn 2
For trajectories on which there has been ample training data for the discriminators, the ensemble members should agree and 4s. (2 () R Lo (z o) for all j. Thus the two terms in equation 10 will be equal and this reward will vanish. For states of high discriminator disagreement, however, this reward will positive, encouraging exploration. Therefore, we call equation th Discriminator Disagreement Intrinsic reward, DISDAIN (figure (b).
DISDAIN is simple to calculate for discrete Z, as is common in the relevant literature (Gregor et al. 2016 Eysenbach et al.) 2019: Achiam et al. 2018 Baumlii al.) 2021). DISDAIN augments any discriminator- based unsupervised skill learning algorithm with two changes. First, an ensemble of discriminators is trained instead of just one, and rskil should be calculated using the ensemble-
averaged prediction (Z O) NV EN ls. (Z O). Second. the DISDAIN reward combined with Fskill, which we do through simple addition with tunable multiplier X. Pseudocode for DISDAIN With N and 0, this standard unbonused skill discovery.
provided in Algorithm [
EXPERIMENTS
We validate DISDAIN by testing its ability to increase skill learning in an illustrative grid world (Four Rooms) as well as a more challenging pixel-based setting requiring function approximation (the 57 Atari games of the Arcade Learning Environment (Bellemare et al.) 2013)). In addition to comparing performance to unbonused skill learning, we also compare to using popular off-the-shelf exploration bonuses that are not tailored to skill discovery. Four Rooms, we compare to using count-based bonuses, which are known to perform well in these settings (Brafman and Tennenholtz! 2002). In Atari, where count-based bonuses become untenable due to the enormous state space, we compare random network distillation (RND: Burda al. (2019)), one of the most commonly used
5

---

Published as a conference paper at ICLR 2022
Algorithm 1:
Skill discovery with DISDAIN
Input:
J-n 1-1 skill features O(7), skill distribution p(2), skill trajectory length T, DISDAIN reward weight À
policy Te. discriminator ensemble {qa. N
while not converged do
Reset environment, sampling initial state 8o
while episode not ended do
Sample skill, ~~ xxil,,=>p(())
Sample trajectory of length T from so, T TT()
Form average discriminator from ensemble, qo = N Lim 96:
Tskill = log (0 O(r)) logp(z)
"DISDAIN H (: [())) E. (
r = Tskill + AIDISDAIN
Update with RL to maximize r
Update ( i=1 with SI. to maximize log qo: (= ((T))
Sp 8T
pseudo-count based methods (Bellemare et al. 2016). In addition, validate that any advantages of DISDAIN are not purely due to using an ensemble of discriminators by evaluating an ablation that
includes the same ensemble but removes the DISDAIN reward (i.e. Algorithm D with À = 0). In all cases, our primary metric for comparison is the effective number of skills learnt, Rssills (equation 4
The effective number of skills is just interpretable transformation of the mutual information objective shared by a large number of unsupervised skill learning algorithms (e.g. Gregor et al. (2016); Achiam et al. 2018); Eysenbach al. (2019); Hansen al. (0200). As such, the fact that DISDAIN helps better maximize this objective should be worthwhile in its own right, considering the various use cases that motivated this objective in the existing literature. That said, we recognize that it is not obvious what utility comes with increasing the number of effective skills. To address this, we also measure three surrogates of skill utility: downstream goal achievement, unsupervised reward attainment, and unsupervised state coverage.
Distributed training We use a distributed actor-leamner setup similar to R2D2 (Kapturowski et al., 2019), except we do not use replay prioritization or burn-in, and Q-value targets are computed with Peng's Q()) (Peng and Williams, 1994) rather than n-step double Q-learning. In all cases, we found it important to train separate Q functions for the skill learning rewards and exploration bonuses (DISDAIN, RND, or count-based). This follows from Burda et al.] (2019), who also found that this setup helped stabilize learning. Unlike that work, we need to specify value target for each Q-function, which we take to be the value of the action that maximizes the composite value function, where the two values are added with a tunable weight A. The discriminator is trained on the same mini-batches sampled from replay to train the Q-functions. For Atari experiments, the Q-networks process batches of state, skill, and action tuples to produce scalar Q-values for each, and the ResNet state embedding network used in Espeholt et al. (2018) is shared by both of the Q-networks and discriminator. For the Four Rooms grid world, we use tabular Q-functions and discriminators. Further implementation details can be found in appendix
Skill discovery Our skill discovery baseline deviates slightly from prior work in order to make it representative of the skill discovery literature as whole. We utilize a simple discriminator that receives only the final state of a trajectory (so O(T) 8T), omitting the state-conditional prior and action entropy bonuses used in specific algorithms (Gregor et al. 2016 Eysenbach et al., 2019).
Hyperparameters Most of the RL hyperparameters were not tuned, but rather taken from standard values known to be reasonable for Peng's Q(A). The skill discovery specific hyperparameters were tuned for the basic algorithm without exploration bonuses, and then reused in all conditions. Our RND implementation was first tuned without skill learning to achieve Atari performance competitive with the original paper. For all exploration bonuses (DISDAIN, RND, count), we tuned their reward weighting (e.g. X in algorithm T). while for DISDAIN additionally swept the ensemble size (N).
Baselines As with the skill discovery reward, we apply exploration bonuses only to the terminal states of each skill trajectory. For the count-based bonus, we track the number of times agent ends

---

Published as a conference paper at ICLR 2022
skill trajectory in each state n(s) and apply the exploration bonus rr 11///(s/) For RND, follow the details of Burda et al. (2019) as closely as possible, For our target and predictor networks, we use the same ResNet architecture as the policy observation embedding described above, and then project to a latent dimension of 128. Rather than normalizing observations based on running statistics, found it more reliable to use the standard 255
normalization of Atari observations.
Our ensemble- based uncertainty estimator required many design choices, including the size of ensembles, and to what extent the ensemble members shared training data and parameters. In all of the domains we tested, we found training ensemble members on different batches to be unnecessary, similar Osband et al. (2016). In the tabular case (Four Rooms), parameter- sharrin is not a concern, and we found an ensemble size of N = 2 to be sufficient. For Atari, the ensemble of discriminators reuse the single ResNet state embedding network which is shared by the value function. The input to the ensemble will inevitably drift, even if the data distribution remains constant, since the ResNet representations evolve according to a combination of the value function and discriminator updates. Expressive ensembles (e.g. with hidden layers) never converged in practice. By contrast, large linear ensembles (N = 40) were reliably convergent, with convergence time increasing with ensemble size. We follow Osband et al. (2016) in scaling the gradients passed backward through the embedding network by 40
DISDAIN
to account for the increase in effective learning rate.
4.1
FOUR ROOMS
First, we evaluate skill learning in the illustrative grid world seen in figure 3 There are 4 rooms and 104 states. The agent begins each episode in the top left corner and at each step chooses an action from the set: left, right, up, down, no- no-op. Episodes are 20 steps and sample one skill per episode (i.e. T = 20). The episodes are long enough to reach all but one state, allowing for a maximum of 103 skills. For each method, we set Nz 128 to make this theoretically possible P
As seen in figure 3 even in this simple task, unbonused agents are unable to exceed 30 skills and barely leave the first room (see figures 18 and 19 for example skill rollouts). With both DISDAIN and a count bonus, agents explore all four rooms and learn approximately triple the number of skills, with the best seeds learning approximately 90 skills. Both bonuses do slow learning and add variance due to the addition of a separate learned Q-function (see 20 individual seeds). The ensemble-only ablation provides a small benefit to skill learning, but far less than DISDAIN. demonstrating that the DISDAIN exploration bonus (FDISDAIN), rather than the ensembling, drives the increased performance. As a simple demonstration of the usefulness of our learnt skills for downstream tasks, we also evaluate each methods" skills on the original Four Rooms reward (i.e. reaching specified goal states) without additional finetuning. Specifically, for each method, we sample each accessible state of the environment as goal, pass it through the trained discriminator qa. choose the highest probability skill x, rollout the policy for one episode conditioned on 2, and track the fraction of goal states successfully reached (similar the imitation learning evaluation of Eysenbach et al. (019)). As seen in Figure 3b) more learnt skills leads to better downstream task performance.
Results
4.2 ATARI
Next consider skill learning on standard suite of 57 Atari games, where prior work has shown that learning discrete skills is quite difficult (Hansen et al. 2020). In addition, it is a non-trivin ooiviaivial test for our ensemble uncertainty based method to work in the function approximation setting, since this depends on how each ensemble member generalizes to unseen data. Here we use Nx = 64 and T = 20. Since Atari episodes vary in length, skills may be resampled within an episode.
The count-based baseline used in the Four Rooms experiments cannot be directly applied to a non- tabular environment like Atari. While pseudo-count methods have been used here (Bellemare et al. 2016). Random Network Distillation (RND) similarly induces long term exploration (Burda et al. 2019) and has been used for this purpose in a state-of-the-an art Atari agent (Badia al. (020). While newer approaches surpass RND in some domains (Raileanu and Rocktäschel] 2020; Seo et al. (2021). is unclear this the case across the full Atari suite. So, at present, we believe RND is the most compelling baseline to compare against DISDAIN.
An open source reimplementation of DISDAIN smaller version of Four Rooms is available http://gthuhh.commdeemm///dbabdd
7

---

Published as a conference paper at ICLR 2022
(c) States reached withou DISDAIN.
(a) Learning curves.
intiaizzziion
Mid-raning
I-40M
100
+
skills learnt
(d) States reached with DISDAIN.
intialiasion alppp)
Mil-saiinn (-200M siupa)
convergince (-800M slspal
2.5e-088 5.0w+-08 7.5e408
environment steps
Unbonused Count Ensemble DISDAIN
(b) Downstream task: goal reaching.
DISDAIN bonuses.
0.75
mntisizatioon (-0 stpps)
Mid-miingg (-200M (-500M
success
0.50
+
0.25
Unbohused Ensemble
Count
DISDAIN
training time
Figure 3: Four Rooms results. (a) Skills learnt for top 10 20 seeds for each method. Mean + std over seeds. (b) Performance on the downstream task of reaching target state, averaged across all possible target states. Mean + std over seeds. (c-d) Example states reached with and without DISDAIN. Plots depict counts of final states reached after one rollout per skill. Columns correspond to different points during training. With DISDAIN, agents learn to reach all states, while without, they barely make it out of the first room. (e) Per-state DISDAIN bonuses for the policy depicted in (d). In the beginning. all exploration encouraged. In the middle, the checkerboard pattern emerges because agents try to space out their skills. By the end of training. DISDAIN gracefully fades away.
Unbonused
RND
Ensemble
DISDAIN
Learning
Skills learnt
Skill boosts Unbonused
1.00
1.00
15.0
Fractionn runs booot
12.5
0.75
0.75
Skills learnt
nurs
7.5
5.0
0.25
0.25
0.00
0.00
200 400
600
800
1000
Number millions)
Figure 4: Number of skills learnt on Atari. Left: Effective skills (see equation 4 over training. measured by interquartile mean (IQM). Center: Distribution of skills learnt across seeds and games. Right: Distribution of skill boosts over Unbonused skill learning across tasks and games. Shaded regions show pointwise 95% confidence bands based on percentile bootstrap with stratified sampling.
Results As pointed out in Agarwal et al. (2021), making statistically sound conclusions can be challenging in the few-seed, many-tase setting of Atari. Thus, we follow their recommendations and focus our results primarily on statistically robust distributional claims here. Individual learning curves and game-by-gat b--gmeee results available in the appendix.
As shown in figure 4 DISDAIN increases the effective number of skills learnt across the Atari suite, with only modest damage sample effiency. Additionally, found that DISDAIN's performance
8

---

Published as a conference paper at ICLR 2022
boosts robust to its key hyperparameters, namely the bonus weight À and ensemble size N (see figure 8). with significant gains over unbonused skill learning maintained over more than an order of magnitude of variation in both.
Notably, our results show that RND fails to significantly aid in skill learning. A sweep over the RND bonus weight is shown in figure but the summary that the RND bonus becomes similar in magnitude to the skill learning reward, it damages skill learning rather than helping, and so the "best" RND bonus weight for skill learning is approximately zero. This is perhaps unsurprising: RND was designed in the context of stationary task rewards (as are most other exploration methods, such as meeud--oounts,, whereas skill learning objectives produce a highly non-stationary reward function. This highlights the importance of using an exploration bonus tailored to the skill discovery setting. Additionally, the failure of the "ensemble-only" baseline to significantly increase skill learning demonstrates that is the "DDIDDIN exploration bonus that is crucial to DISDAIN's success, and not just
the ensembling of the discriminator.
Ensamble
DISDAIN
To further probe the utility of our learnt skills, we measure the unsupervised reward attainment of policy that randomly switches between them. First reported in Hansen et al. (2020), the idea is that while this policy is certainly far from optimal, this metric indicates whether or not the skill space is sufficient perform the various reward collecting behaviors involved in each game. Additionally, we measure lifetime state coverage as an indication of exploration throughout learning (on a subset of games sup- porting this metric; see Appendix B for details). Figure coonfimss that DISDAIN leads to both increased reward attainment and lifetime cov- erage (see figures [4 [5| and I6 for additional lifetime and episodic state coverage results).
Zero- Zero-shot reward Improvement
Lifetime coverage improvement
Frector of nung with bost R.
0.73
0:35
-50%
+50% +100% +150% Liutime coverage boost (T]
06
Figure 5: Qualitative analysis of skills learnt on Atari. Both plots depict improvement over Un- bonused skill learning aggregated across seeds and games. Left: Probability of improvement on zero- shot reward evaluation. Right: Distribution of per- centile improvements on lifetime coverage. Error bars depict 95% stratified boostrap Cls.
5 DISCUSSION & LIMITATIONS
We introduced DISDAIN, an enhancement for unsupervised skill discovery algorithms which in- creases the effective number of skills learnt across a diverse range of tasks, by using ensemble-bast uncertainty estimation to counteract bias towards pessimistic exploration.
The connection between ensemble estimates of uncertainty and infomax exploration dates back to at least (Seung et al. (992).. who use it to select examples in active supervised learning setting. These ideas have more recently found use in the RL literature, with recent work using disagreement
among ensembles of value functions (Chen et al. 2017 Flennerhag al. 2020 Zhang al. 2020) and forward models of the environment (Pathak et al. 2019 Shyam et al. 2019 Sckar et al. (020) to drive exploration. In a closely related line of work, ensembles of O-functions have been used to drive exploration without an explicit bonus based on disagreement (Osband et al. 2016 2018). To our knowledge, DISDAIN represents the first application of these ideas to unsupervised skill discovery.
We focused on discrete skill learning methods due to their relative prevalence (Gregor et al. 2016 Eysenbach et al. 2019; Achiam et al.) 2018: Baumli al.] (021). In some cases, continuous or structured skill spaces might make more sense (Hansen et al. 2020 Warde-Farley et al. 2019)) While the principles behind DISDAIN should still apply, further approximations may be necessary, e.g. for the entropy of the ensemble-averaged discriminator, which may be unavailable in closed form.
Designing agents that explore and master their environment the absence of task reward remains an open problem, for which there exist many different families of approaches (e.g. reward- freee exploration (Jin et al.) 2020; Zhang et al. 2021)). In this paper, we focus on improving one such family- unsupervised skill learning through variational infomax (Section 2 By treating agents with DISDAIN, we empower them to better maximize their objective and learn more skills. Leveraging these skills for rapid task reward maximization remains an important direction for future research.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
The authors would like to thank Stephen Spencer for engineering and technical support, Ian Osband for feedback on an early draft, Rishabh Agarwal for suggestions statistical analysis and plotting, Phil Bachman for correcting the error discussed in Appendix C and David Schwab for pointing us to the original Query by Committee work (Seung al., 1992).
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprini 1807. 2018.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Neural Information Processing Systems (NeuriPS), 2021.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in Atari. In Neural Information Processing Systems (NeurlPS), 2019.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the Atari human benchmark. In International Conference on Machine Learning (ICML), 2020.
David Barber and Felix Agakov. Information maximization in noisy channels: A variational approach. In Neural Information Processing Systems (NIPS), 2004.
Kate Baumli, David Warde-Farley, Steven Hansen, and Volodymyr Mnih. Relative variational intrinsic control. In AAAI Conference on Artificial Intelligence, 2021.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 06 2013.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Neural Information Processing Systems (NIPS), 2016.
Ronen Brafman and Moshe Tennenholtz. R-max general polynomial time algorithm for near- optimal reinforcement learning. Journal of Machine Learning Research (JMLR), 3(Oct):213-25) 2002.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Hosseert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Neural Information Processing Systems (NeurIPS), 2020.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations (ICLR), 2019.
Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. UCB exploration via Q- ensembles. arXiv preprint 17X.v:1706.0/502 2017.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. In Neural Information Processing Systems (NeurIPS), 2020b.
10

---

Published as a conference paper at ICLR 2022
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In International Conference on Machine Learning (ICML), 2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations (ICLR), 2019.
Sebastian Flennerhag, Jane X Wang, Pablo Sprechmann, Francesco Visin, Alexandre Galashov, Steven Kapturowski, Diana Borsa, Nicolas Heess, Andre Barreto, and Razvan Pascanu. Temporal difference uncertainties as a signal for exploration. arXiv preprint arXiv:2010.02255 2020.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 16 1.00507, 2016.
Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features, In International Conference on Learning Representations (ICLR), 2020.
Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward-free exploration for reinforcement learning. In International Conference Machine Learning (ICML), 2020.
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations (ICLR), 2019.
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent RL. In Neural Information Processing Systems (NeurIPS), 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Neural Information Processing Systems (NeurIPS), 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: Multi-agent variational exploration. In Neural Information Processing Systems (NeurIPS), 2019.
OpenAI, llge Akkaya, Marcin Andrychowicz. Maciek Chociej. Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino. Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei 2019a.
Zhang. Solving Rubik's s Cube with a robot hand. arXiv preprint
OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique d. O. Pinto, Jonathan Raiman, Tim Salimans. Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning, arXiv preprint arxiv: /9/00060000 2019b.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Neural Information Processing Systems (NIPS), 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Neural Information Processing Systems (NeurIPS), 2018
Ian Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research (JMLR), 20(124):1 2019.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In International Conference on Machine Learning (ICML), 2019.
Jing Peng and Ronald Williams. Incremental multi-step Q-learning. In International Conference on Machine Learning (ICMI), 1994.
11

---

Published as a conference paper at ICLR 2022
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report. OpenAl, 2019.
Roberta Raileanu and Tim Rocktäschel. RIDE: Rewarding impact-driven exploration for procedurally- generated environments. In International Conference Learning Representations (ICLR). 2020.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning (ICML), 2020.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Picter Abbeel, and Kimin Lec. State entropy maximization with random encoders for efficient exploration. arXiv preprint arXiv:2102.09430, 2021.
H. Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Conference on Learning Theory (COLT), 1992.
Pranav Shyam, Wojciech Jaskowski, and Faustino Gomez. Model-based active exploration. In International Conference Machine Learning (ICML), 2019.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2):181-111 1999.
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Daníhelka, Aja Huang. Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps. and David Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782): 350-354, 2019.
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin lonescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. In International Conference on Learning Representations (ICLR), 2019.
Chuheng Zhang, Yuanying Cai, and Longbo Huang Jian Li. Exploration by maximizing Rényi entropy for reward-free RI framework. AAAI Conference on Artificial Intelligence, 2021.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. In Neural Information Processing Systems (NeurIPS), 2020.
12

---

Published as a conference paper at ICLR 2022
A COMPUTE REQUIREMENTS AND HYPERPARAMETERS
The compute cluster we performed experiments on is rather heterogeneous, and has features such host-sharing, adaptive load-balaneing, etc. is therefore hard give precise details regarding compute resources however, the following is a best-guess estimate.
full experimental training run for Atari lasted days on average. Our distributed reinforcement learning setup (Espeholt et al. 2018) used 100 CPU actors and single V100 GPU learner. Thus, we required approximately 9600 CPU hours and 96 V100 GPU hours per seed, with seeds and conditions per game.
Tuning required approximately 10 different hyper-panee saper-parameters combinations on 6 games, amounting to 864,000 CPU hours and 8,640 V100 GPU hours. The results on the full suite of 57 Atari games required 4, 4,924,800 CPU hours and 49,248 V100 GPU hours. Combining these, we get a total compute budget of 5,788,800 CPU hours and 57 888 V100 GPU hours.
worth remembering that the above is likely quite a loose upper-bound, as this estimate assumes 100 percent up time, which is far from the truth given the host-sharing and load-balancing involved in our setup. Additionally, V100 GPUs were chosen based on what was on hand; our models are small enough to fit on much cheaper cards without much slowdown.
B COVERAGE METRICS
We calculate two related notions of coverage: lifetime and episodic. Lifetime coverage corresponds to the number of unique states encountered during an agent's ' lifetime, whereas episodic coverage corresponds to the number of unique states encountered during each episode. Both rely the notion of a unique state. The subset of games chosen for these metrics were those where a good notion of unique state is simple: they all involve controllable avatar that moves in coordinate system, so unique avatar coordinates are used. This information is exposed in the RAM state the Atari emulator, shown in Anand et al. 2019
C ADDITIONAL IMPLEMENTATION DETAILS
For our skill learning reward, one generally helpful change that we had not previously encountered was to clip negative skill rewards to 0 (raml = max ()). A previous version of this manuscript stated that: "This yields a strictly tighter lower bound on the mutual information (equation 2 since negative rewards imply the discriminator's performance is worse than chance." However, that argument isn't quite correct. Clipping the expected reward (i.e. clipping outside the expectation) would produce tighter bound, but this argument does not hold on a sample-by-sampe basis (i.e.
clipping inside the expectation, as we did). Thus, this should only be viewed as a heuristic.
REFERENCES
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
13

---

Published as a conference paper at ICLR 2022
Hyperparameter
Atari
Four Rooms tabular
Torso
IMPALA Torso Espeholt et al.
2018
Head hidden size
256
Number of actors
100
64
Batch size
128
16
Skill trajectory length (T')
20
same
Unroll length
20
same
Actor update period
100
same
Number of skill latents (N2)
64
128
Replay buffer size
10 unrolls
Optimizer
Adam Kingma and Ba 2015)
SGD
learning rate
* 10-4
10-3
Adam
10
Adam 31
0.0
Adam 32
0.95
RL algorithm
(()) Peng and Williams,
1994
same
À
0.7
same
discount
0.99
same
Target update period
100
DISDAIN ensemble size (N)
40
DISDAIN reward weight (A)
180.0
10.0
RND reward weight
0.3
Count bonus weight
10.0
Table Hyperparameters. Atari hyperparameters were tuned on subset of games (beam_ra ri rider, beeerrbree beakout, pong, qbert, seaquest, and invaadersss.... In both envi- ronments, all RL and skill discovery hyperparameters were tuned for unbonused skill learning and then held fixed when adding exploration bonuses.
14

---

Published as a conference paper at ICLR 2022
25
7.5
10
Se+58
637k best
6806 zire
bea teee
broek
10
10-
Tenstui
2902 padn
choper CARPAS
27.5
15
12.5
10
J0+09
15-
CO026 3.28
andim
Bhii dery
12.5
Treeeay
15
10-
10
75
0e-30
Se-38
10+08
Dee02
See08
1e-0s
0e+30
Seed6
Jee00
Dee03
Seedi
1e-0s
0e-30
Seet8
lee59
Dre02
SeedR
Ieed5
20
hert
20
100 kodayy
amasbond
kargarto
10-
15
20
12
10
skills learnt
Unbonused
00-20
Se-28
1ew09
Dee02
Seed1
1e-0s
0e-30
Se.28
1e+03
0ne03
Se+01
1e-0s
0e-30
Se+28
10+58
Dre02
SeedE
1e+55
RND
Ensemble DISDAIN
CIPS
montenum m2n0ge
RC parman
Tame Th'S 3tme
10.
1s
15
12
06-20
Se-20
10+00
06402
Se+03
1e-02
0e-20
Se.28
10x00
0ee02
Se+01
1e-02
0e.20
50+28
18453
0e402
S06G2
p370
Blas
20
10
15
15
0e-30
Se-28
1e409
0e403
See08
1e-03
0e+30
Se+38
Ie409
06403
Se+08
le-d5
0e-:0
Se458
16459
06405
Seeds
1e+d5
REQUSE
12.5
quarer
SUTOUNS
20
00-30
Se+38
16408
Be+02
Seedl
1e-0s
0e+20
Se+38
Ie+09
Be+03
Se+08
1ewds
Dee
50+58
1ee59
Dee05
SeedE
10+05
Uig
Marribum
9p.n dow!
75
10
15
25.
06
5e-30
16+00
0s+03
Se+01
1e-03
0e+20
Se+28
16+09
0e+03
S6+08
1e-05
0ee
Se438
l8459
0e405
Se-d2
1e+02
20
06-20
Se.28
10:00
0c100
1c-02
0e020
J0
environment steps
Figure 6: Skills learnt per-seed and per-game on all Atari games.
15

---

Published as a conference paper at ICLR 2022
(a) Skills learnt on each game.
skills leamt
Unbonused RND
Ensemble
DISDAIN
you
Sortin Auc
(b) DISDAIN skill boosts cach game.
your
poreccacio posoliu Sor
(c) RND skill boosts on each game.
skill boost
de
paec
(d) Ensemble skill boosts each game.
skill boost
your
Figure 7: Per-game boosts for each method. (b-d) Boosts in skills leartt on each game over Unbonused skill learning. Mean = standard deviation over 3 seeds. All 3 plots use the same y-axis range magnitude so that bar heights are comparable. DISDAIN improves skill learning (91%) of games, with boosts of skills on 18/57 (32%) and ~10 on 6/57 (11 1)) of games. RND and the Ensemblc-only ablation perform closer to chance with improvements (61%). with boosts of 5 skills on 2/57 (4%) and 9/57 (16%) and >10 skills on 1/57 (2%) and 0/57 (0%) of games, respectively. a) Skills learnt on each game for each method for reference. Sorted by magnitude of DISDAIN boost, as in (b).
52/57
28/57 (49%) and 35/57
16

---

Published as a conference paper at ICLR 2022
(a) Robustness to bonus weight (A).
(b) Robustness ensemble size (N).
15.
skills learnt
10.
Unbonused
skills learnt
12-
Unbonused
DISDAIN (ensemble size 20)
Ensemble
DISDAIN (ensemble
DISDAIN (weight = 45)
DISDAIN (ensemble size 80)
DISDAIN (weight 180)
DISDAIN (ensemble size 160)
DISDAIN (weight 720)
DISDAIN (ensemble size 320)
2.5e+08
environment steps
environment steps
Figure 8: DISDAIN robustness to key hyperparameters. (a) Sweeping bonus weight (A) with fixed ensemble size N 160 (see Algorithm []. (b) Sweeping ensemble size (N) with fixed bonus weight À = 180, Curves averaged over 57 games and 3 seeds (for results broken out by game and seed, see figures and I0). For both hyperparameters. DISDAIN's improvements over baselines are robust over more than an order of magnitude of variation. All other experiments use À = 180 and N 40 unless otherwise stated.
17

---

Published as a conference paper at ICLR 2022
4000
at anlls
25-
73-
20
10
255
0c100
S2108
10103
02100
Se108
1e109
0c:00
Sc:08
1or09
02100
Sc:68
10:00
00100
Ser
1e109
0c100
18109
BankChest
batBe 2002
beam Aer
Baraekk
bowinng
boxing
10
30
10-
2,5
0-+00
Se-08
1m-09
0e-00
Se+08
1m-09
De+00
Se+08
1ew09
De+00
Sm-08
10-50
De-d3
SmwD8
Ie+09
0e-00
S0=20
1e+09
Drcardut
pooe
Chopper command
Cray dimber
defander
damon, atbrc
15
5-
10
0--00
Se-08
1e-d5
0+-00
Se+08
1m+09
De+00
5e+08
1e+09
De+00
Se-
10-50
De-03
Spe
Ie+00
0e-00
Se 50
1-
GouBre dunk
andurs
Saiing dariggddr
12.5
Foonay
202har
12-
10-
20
7.5
5
2.5
0--00
Se-08
Im-09
06-00
Se+08
Te-09
De+00
Se+08
1ew0g
De+00
5e-08
1e-50
De-00
Spe08
0e-00
S0-20
1e-09
growkar
holo
hoceyy
amosbond
kangaroo
12-
skills learnt
10
Unbonused
Ensemble
0-+00
Se+08
Te-09
06+00
Se+08
Te-09
De+00
Se+08
1ee0g
De+00
5m-08
1e-50
De-00
Sme
0e-00
50
Im-09
DISDAIN (weight 45)
Kung TJ masier
no Moneeaa revenge
ma B2dman
name PIS game
ploank
DISDAIN (weight 180)
16
DISDAIN (weight 720)
10
10
5-
2.5
0e-00
Se-08
Te-d9
06+00
Se+08
Te-09
De+00
Se+08
1ew0g
Dee00
SP-08
10-50
De-d5
Spe
Je+d0
0e-00
S0-30
Te-09
pong
prhvze Cye
abert
drenad
Dad Junner
mobotane
15
10
15-
5
0e+00
Se-08
1e-05
06+00
Se+08
Te-09
De+00
Se+d8
Tew0s
De+00
Se-08
1e-50
De-03
Spe
Ie+00
0e-00
50-00
1e-09
sling
adars
12.5
space Inaders
Sal round
10
25
10+
D4
0.+00
Se+08
Te+05
0e+00
Se+08
Te-08
De+00
Seed8
lew0g
De+00
Sp-08
10-20
De+03
Spe
Ie+00
0e+00
50
1e-09
Imide
Sime Jlat
venture
video P Foall
15
755
10-
23
255
0e+00
Sa+08
Te+05
0e400
Se+08
Te-09
De+00
Se+08
1ew09
De+00
Se-
10+20
De+03
See
Ie+00
0e+00
50-30
1e408
WlEar of 2007
yars Jevenge
200000
20
10
0c400
S6-08
Jo-03
00:00
56408
1c-09
0s-00
Se.05
16:59
environment steps
Figure 9: Per-game bonus weight sweep for DISDAIN across all 57 Atari games.
18

---

Published as a conference paper at ICLR 2022
10-
20
35.40855
7.5
255
0m+50
Se+58
Inece
Ce-E0
Se-00
1e+39
Dee50
Se+00
10-20
De+50
Se+58
10e50
Se-00
10-08
0e+50
Se+50
10400
20
bzeding
boerg
20
20
20
10
2.5
10
02100
50108
1o119
00-00
10:08
Je199
0e100
Se108
10:00
0e120
Se126
Jont9
De:00
16:08
Je100
0e100
Se108
10119
15
20
20
10
10
0.:00
Se+58
I6400
16:29
S6400
0e.20
S6+28
06:286
10-
20
75
25-
24-
gmstar
140
10
Noakay
Inrestand
15
knll
15
15
10
skills learnt
10
Unbonused
DISDAIN (ensemble size = 20)
Dew20
See58
Imedo
Ce-00
Se-00
1e-39
Dee50
Se450
10-00
Dew30
Se428
J0e50
Ce-C0
Se-00
J0-00
Deec0
Seeto
Jeeco
DISDAIN (ensemble size = 40)
20
King maddee
20
monterura Te0mge
mC. parman
Aame TNC game
12
DISDAIN (ensemble size 80)
DISDAIN (ensemble size 160)
10
10
DISDAIN
(ensemble size 320)
08120
Se128
Je109
0e:00
Se108
Je129
0c100
Se108
10:19
D6120
Se128
J8109
00:00
Se 08
10109
0c100
Se1
10119
p219
100
Furner
19bclakk
15
7.5
10
0e430
Se+56
1es19
te-00
Se108
Jes39
Det00
Sett6
Je-18
D6120
Se456
Jes59
0e:00
Se
1e409
D6400
Setce
40
15
10
30
10
10
36128
2101
20
7.5
10
40
50% mRAKEE
15
10
10
20
Dex20
504
10450
5e-00
Deeco
So
20-c0
environment steps
Figure 10: Per-game ensemble size sweep for DISDAIN across all 57 Atari games.
19

---

Published as a conference paper at ICLR 2022
12.5
10
skills learnt
Unbonused
RND (weight 0.1)
RND (weight = 0.3)
RND (weight 1)
RND (weight 3)
2.5
0.0e+00
2.5e+ 5.0e+08 7.5e+08 1.0c+09
environment steps
RND bonus weight sweep. RND fails significantly improve skill learning the weighting on its contribution to the reward is increased. As soon as the RND bonus becomes of similar order of magniuuee performance begins to decay, suggesting that the kind of exploration encouraged by RND is not conducive to skill learning. Results averaged over 57 games and seeds (see figure 12 for results broken out by game and seed).
Figure 11:
the skill Iearning reward (bonus weight s 1), skill learning
20

---

Published as a conference paper at ICLR 2022
Ellen
amider
15-
soter
5-
5e-20
burk Tebtt
540
bandin
boxhg
12
20
00
20+02
0e-00
36138
56-00
56-28
06420
16.00
15
20
10
0.+00
16+09
0e-00
Se+01
10
Se+38
1++09
36w50
10-10
56-30
Ieeds
06+20
12
23
25
0c-00
Sc+28
16400
16-00
Se+08
10+00
De+00
Sa+28
16400
26+20
Se403
16-29
06400
S6-28
18600
06+00
50.03
16.29
16
hero
kahocky
ammibord
langarco
10
7.5
skills learnt
5-
Unbonused
RND (weight 0.1)
0c100
Sc100
1c103
0c:00
Sc108
16100
02100
Se128
1:100
2e100
Sc108
10:00
02105
Ser08
1e103
00:00
1c 120
RND (weight 0.3)
MORTRES Feannge
mc parman
name th gams
Bhonn
RND (weight 1)
12
RND (weight 3)
1e+09
Se-00
Ser36
1ee09
Sem00
10-19
Se198
10909
pong
20
pRoate
Nanae
reed Riree
Fobebank
7.5
15
5
10
2.5
sking
30
20
10
3x-26
10+02
Se+28
36+20
38450
7.5
5-
2.5
25
0c
1000
1c100
00
Sc105
10100
0c100
Se125
1.100
Sc105
1e:09
00100
Se/08
1:103
06 100
5o 10
16
30
20
0K+00
Scmc6
Je109
Sev00
Se+O8
16:09
0c+00
Seri8
1c+09
environment steps
Figure 12: Per-game bonus weight sweep for RND across all 57 Atari games.
21

---

Published as a conference paper at ICLR 2022
sibrts
450
390
300
15010
230
202
100
200
200
5030
SInK Fell:
3006
Barte Z8ma
1002
Baam T0B
20:201
bowlng
boolig
592-
20
092
70
702
10
200
3e+50
Se-50
Te-0s
0m-02
Se+-5
16-09
0e-03 Se+01 1e408
06400
Se-00
10+50
0e-50
5m-50
Je-29
De-00
5m-38
1m-09
breakous
2406
con comppaze
Esppe commane
C7ay dinber
adeicer
Sanon slack
1.4-
2030
003
6000
320
12-
1640
1236
603
4000
1500
2000
230
Se-50
Se-C0
Ie-0s
0e-02
See
1e-02
0s-03
S0+03
1c+00
06400
Se-00
18+59
De+50
Se-50 50
Je-29
De-00
1e-09
Souble
mnt.m
RoYns dedby
Booher
-03
-92
100
-23.5
-92
420
90
-24
0e+50
50-08
Je-0s
0e-05
Sc+38
10+08
0c+03
Se+01
I0+09
0:400
Se-08
le+69
06450
Se-18
Je-09
6e-00
See 26
Ie-09
graekar
830
-13
60
lanjaos
125
100
600
230
-12
40
2520
Score
20
Unbonused
00120
20/02
00
56
10:00
0s
Sc105
1e100
0:100
Se 00
3e100
0e100
26:00
20/00
0e
3el
25
00
RND
Ensemble
2000
DISDAIN
2000
500
0.025
0.22
602
1200
1000
500
1300
500
-120
-300
-200
50-50
Ie-ds
0e-02
In-03
0e+00
Se+03
10-39
De-d0
1300
-20.6
750
1030
-20.8
500
252
11000
I0-05
0e-02
Se+
1s.02
0c+00
5e+03
0s400
5e+00
20-39
1o-00
15000
dinng
1300
Rumound
200
250
503
100
1000
-8.5
002
33096
302
700
Sulancham
19030
-23
10
10020
1500
22.00
5010
1030
810
Wlare 2f W67
FHR Sevange
303
203
400
102
environment steps
Figure 13: Per-game task reward attainment curves throughout training. We emphasize that agents are trained only to maximize skill learning objective and any associated exploration bonus (i.e. DISDAIN or RND), and not the task reward. Thus, these plots depict zero-shot reward attainment while uniformly randomly switching between skills.
22

---

Published as a conference paper at ICLR 2022
Ser2er1
hee
moneter wtwpccuuunneeeaeeeeee pevenge
20000
50.00
15000
20000
40.00
lifetime coverage
10000
3000
5000
Unbonused
06+00
Sem 08
16+09
06+00
Se+008
10+09
06+00
5c+08
1c+09
RND
prvatoc_pys
11000
Ensemble
1720
R0000
10750
DISDAIN
1700
10590
60000
10250
40000
10000
5ch.00
0c:000
5eh000
10
environment steps
Figure 14: Lifetime coverage for all games, methods, and seeds. All policies quickly achieve same ms ms_pacman and seaquest, those levels removed from the analysis in the main text.
coverage boost
+250%
+200%
+150%
RND Ensemble DISDAIN
+100%
+50%
private eye ms cs_paccmn
seaquest
montezuma, revenge
berzerk
Figure 15: Episodic coverage boosts over unbonused skill learning. DISDAIN provides significant boosts unbonused skill learning hero and perform similarly on the other four levels analyzed. Since state coverage metrics are particularly well suited to mont zuma revenge, the boost there is especially interesting. Results averaged over seeds. For results over training for each seed, see figure 16
while all methods
montazuma reuNriqe
2000
400
1500
500
300
episodic coverage
200
1.0
500
Unbonused
0c-00
Se+08
10+09
00+00
Ser
1c+09
0c-00
Se.08
1c+09
RND
mi pasmtann
privdte aye
seutquest
Ensemble DISDAIN
350
340
4600
1500
.20
4000
300
1000
3600
280
0c100
Sc+08
19+
0c+00
1cr09
0c+00
Sc108
1c+09
environment steps
Figure 16: Episodic coverage for all games, methods, and seeds.
23

---

Published as a conference paper at ICLR 2022
teeskout
cellibetee
delender
dunkk_2unk
1eNogdderyy
Freeway
910ter
prantiar
TAP
CC. Tookdyy
araanbon
bangeroo
Inil
steps
DISDAIN bonus
lung
mutter.ama /RORnge
mS pacmman
name GaTT
phrenic
pllal
Bine
UP.0_SPW
VerTure
J500 36ee_a08al 2764
of
yans, revenge
zBC4cn
Skill learning reward
Figure 17: rskil vs FDISDAIN during learning for all seeds per-game on all 57 Atari games. Each panel includes data from 3 seeds. DISDAIN reward tends to dominate early but fades away skill learning converges.
24

---

Published as a conference paper at ICLR 2022
SUIT 0914
SI 191
Sdi 33 OWW!
148 11 90
SEE 100 540
S: (9189)
Scll
10112510.01
318 11.50)
64 (055)
5L10.721
2M11
SU1225 (2.40
S1B
211 11116.41
(A55)
5910.90
210 90)
24172
SAI
2011000
(2.500
2516.91
90)
13.6
125
12 0V4N
1916 V11
S61 JUM
Sc8 [M
518 12
Sdi 62 0V:9)
638 92
SW8 71348
Sd1 N0
Sal
198 12.300
59:
(35)
3510.01
2M1:2410
SU122
Figure 18: Example DISDAIN skill rollouts Four Rooms. Each panel shows rollout for one skill. Panels are labeled with skill index and the probability the discriminator assigns to the correct skill label. Color indicates time within episode, moving from green (beginning) to yellow (end). Skills are sorted by final state. DISDAIN learns to visit nearly every accessible state.
25

---

Published as a conference paper at ICLR 2022
2210.001
SUEST
S48
4H1
Sell 6E
10:20
0622
2-17
04
Sell
SM 10212.05
29.13110.00
SMI 10.50
34134 13
5e1334
5M 1010.91
3MI137
2H1 23.00
3MI 0U57
36 15
50190 21
201 3.
361 11.001
518 21.33.01
16 90)
24160 25
SAI 35 00.34
S:1 451231
SUERE 29
SU11 (AMS
Sail 12
S88 2310.111
S4 00
SI 291211
50117.1020
(2-00)
Sel
SHIT
3E 1201GA41
1855)
1110.411
SU14 1928:
12L
S01 15.8
S41 0031
(2:0000
7816.301
100 (L.6)
(1
Figure 19: Example Unbonused skill rollouts on Four Rooms. Each panel shows rollout for one skill. Panels are labeled with skill index and the probability the discriminator assigns to the correct skill label. Color indicates time within episode, moving from green (beginning) to yellow (end). Skills are sorted by final state. Unbonused skill learning fails to learn to reach many states beyond the first (upper left) room.
26

---

Published as a conference paper at ICLR 2022
100
75
skills learnt
Unbonused
50-
Ensemble
Count DISDAIN
25-
2.5e408 5,0+408 7.5e408
environment steps
Figure 20: Four Rooms training curves broken out by seed. The exploration bonuses dramatically improve skill learning in most cases, though they also slow learning and add variance due to the training of an additional separate value function (and for DISDAIN, an ensemble of discriminators).
27


Reviews:
[{"review_id": "ieSQFeHJNUd", "reviewer": "Reviewer_rscN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe paper is well-written and I enjoyed reading it. The authors clearly explain the discriminator-based skill discovery via intrinsic rewards given in (3) and why it results in a pessimistic agent. In skill discovery, a low reward is given when the discriminator is poor which happens in the face of unseen states. This is contradictory to the UFO principle for exploration and thus motivates the authors to provide an auxiliary bonus. The discussion on page 4 regarding maximizing the lower bound and ensuring its tightness is interesting. The new exploration method is simple and practical and is shown empirically to improve skill discovery.\n\n**Limitations/Comments**\n\nThe problem considered in this paper is closely related to the reward-free pure exploration setting. For that setting, approaches based on maximum entropy exploration have been proposed, such as\n\nZhang, Chuheng, Yuanying Cai, and Longbo Huang Jian Li. \"Exploration by Maximizing R\\'enyi Entropy for Reward-Free RL Framework.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.\n\nComparison with these methods is not provided in the paper.\n\nAs the authors acknowledge, it is not clear whether the number of learned skills in (4) is a good performance metric. A better metric can be similar to the reward-free paradigm, which is being able to solve any task if the reward function is provided.\n\nIn empirical evaluations, the authors compare with RND which is a common pseudocount-based method. However, several other exploration methods are proposed that significantly improve over RND:\n\nSeo, Younggyo, et al. \"State entropy maximization with random encoders for efficient exploration.\" arXiv preprint arXiv:2102.09430 (2021).\n\nRaileanu, Roberta, and Tim Rocktäschel. \"RIDE: Rewarding impact-driven exploration for procedurally-generated environments.\" arXiv preprint arXiv:2002.12292 (2020).\n\nZhang, Tianjun, et al. \"MADE: Exploration via Maximizing Deviation from Explored Regions.\" arXiv preprint arXiv:2106.10268 (2021).", "summary_of_the_paper": "This paper is concerned with unsupervised RL where an extrinsic reward signal is not available. The objective is for the agent to master the environment by exploring it while learning a diverse set of skills. This is done by simultaneously training a policy conditioned on a latent variable and a discriminator that tries to infer the latent variable from trajectories. The authors identify that the intrinsic rewards used in skill discovery result in the agent being pessimistic towards exploring novel parts of the environment. To alleviate this, the authors propose a new auxiliary objective, which results in a bonus based on the disagreement of an ensemble of discriminators. Empirical results on the grid world and Atari show improvements in skill discovery and solving downstream tasks compared to baselines.", "summary_of_the_review": "The motivation behind the method is well-justified and the approach is interesting and practical. The discussion on pessimism in skill discovery is insightful. The paper can be improved by further comparison with methods that are not focused on skill discovery.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YhfklLzLxuP", "reviewer": "Reviewer_vjRd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n1. The paper is well motivated. The shortcomings of existing unsupervised skill discovery algorithms are clearly explained along with how the proposed method overcomes these shortcomings.\n\n1. The derivation of the proposed objective is intuitively appealing and theoretically sound, the final form of the reward bonus is a simple one (a desirable trait) -- entropy of the mean discriminator minus the mean entropy of the ensemble of discriminators.\n\n1. The paper does a very good job in explaining all the hyperparameter and modeling choices made in their experiments and reproduction of baselines.\n\n1. The paper does a good job at thoroughly analysing the pedagogical 4-room setting as well as all 57 Atari game environments.\n\n1. The results in the 4-room and Atari environments are both positive in terms of the three metrics evaluated -- number of skills learned, transfer of skills to downstream tasks and lifetime state space coverage. Appropriate ablations are compared to in all cases, specifically, \n\n## Weaknesses\n\n1. Unlinke prior work (DIAYN), the paper restricts the choice of environments to the 4-room and Atari environments. Given that DIAYN and subsequent works (e.g. DADS [1]) have shown skill learning in continuous control tasks such as in the MuJoCo locomotion environments, it would be good to see results of the presented experiments in such domains.\n\n## Other comments and feedback\n\n1. Figure 2 does not do a good job in conveying meaningful information about the methodology. For 2(a), the notation is dense and hard to parse, the different colored boxes for each component don’t help. I don’t see the need for Figure 2(b) just to explain the disagreement objective from the ensemble, this was better conveyed in text. Dropping this figure may be a good idea.\n\n### References:\n\n[1] Sharma, A., Gu, S., Levine, S., Kumar, V., & Hausman, K. (2019). Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657.\n\n\n================\n## Post-rebuttal review update\n\nThe authors have addressed the concerns raised by all reviewers and I will maintain my score of 8 (Accept).\n", "summary_of_the_paper": "The paper proposes a novel unsupervised skill discovery algorithm. Beginning with the family of such methods (DIAYN, VIC, etc) that learn a discriminator to distinguish skills given some observations of the trajectory and a policy that executes a skill conditioned on the (discrete) skill random variable Z, the premise of the paper is that such methods on their own will fail when new states are encountered during the skill learning process as the discriminator would not have had sufficient data to learn to distinguish novel states. The paper proposes a novel reward bonus that works in addition to a base method such as DIAYN (DIversity is All You Need), such that this bonus “reimburses” the policy for visiting states where the discriminator uncertainty is high (measured using a form of disagreement across an ensemble of discriminators). Experiments on the pedagogical 4-room environment and the Atari suite of environments demonstrates the benefit of the proposed reward bonus in not only learning more skills (or “empowerment” in the VIC nomenclature), but the learnt skills are also superior for downstream tasks (external reward) and lifetime state coverage.", "summary_of_the_review": "The paper is well motivated, provides a simple and theoretically sound novel reward bonus for overcoming the stated “pessimistic exploration” problem and demonstrates positive results across the board for the 4-room and the 57 Atari game environments. No experiments are presented on continuous control tasks such as MuJoCo locomotion environments, which would have strengthened the paper further.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TchJlbz2xFR", "reviewer": "Reviewer_7dHZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths:\n- An important problem in unsupervised skill discovery through info-max methods (such as VIC and DIAYN) is identified. \n- A simple and intuitive approach is proposed to fix the issue. \n- The problem and the solution method are both well-motivated.\n- Experiments (featuring a standard toy problem --- Four Rooms domain --- as well as pixel-based Atari domains) show strong support for the effectiveness of the solution method. \n- Justifiable sets of baselines are used in each environment. \n- Presentation is clear and provides a good summary of prior work. \n\n## Weaknesses:\n- Discussion of limitations is only included in the supplementary material and should be brought to the main text and discussed earlier on.\n- Given the higher variability per seed, why do none of the plots shows variance across seeds?\n- Also, not sure why only 3 seeds are run in Atari? Considering this higher variability, shouldn't a larger number of seeds be used? For DQN, the standard is 5 seeds and DQN already shows very low variability across seeds.  \n- It is unclear how the authors have realized the sufficiency of not using bootstrapping for training the ensemble (i.e. they use the same mini-batches across the ensemble) --- see my Question 1. Also, not clear how much this choice is responsible for higher variability across seeds?  \n- It would have been nice to see a qualitative analysis of learned skills in Atari. Is that not something that can be done quite easily?\n \n\n## Additional questions: \n1. On page 4, it is stated that the same mini-batches are used to train the discriminators within the ensemble and that this was found to be \"*both sufficient and simpler*\". Regarding sufficiency: (i) how did you experiment to find out bootstrapping vs. same mini-batches yield similar performances? (ii) wouldn't this, in the long run, be problematic/limiting (in a general sense)?\n\n2. In Sec. 5 it is mentioned that this work is the first to apply ensemble learning and epistemic uncertainty capturing to unsupervised skill discovery. However, curiosity has been combined with ensemble models before (Pathak et al., 2019). Also, curiosity has been used to acquire skills by snapshotting (Ref. [1]). While I agree that in some senses the statement in the paper is reasonable, I think this general statement confuses more than it informs. Wouldn't it be more clear if this was stated in the specific context of *diversity-seeking* unsupervised skill discovery?   \n\n## Minor:\n- Fig. 4: Legend seems out of place (it's included in part (b), while it's least useful in the context of part (b)).\n- Colors in Fig. 4 are too close (RND and Ensemble are difficult to distinguish in part (a)). \n\n### References: \n[1] Groth, O. et al. (2021). Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration. *arXiv 2109.08603*.", "summary_of_the_paper": "This paper identifies a source of pessimism in DIAYN-style methods for exploring new parts of the state space. They argue that this issue is due to using a single point estimator as a discriminator and that capturing the epistemic uncertainty of the discriminator could serve as an additional signal to guide exploration. They achieve this by using an ensemble of discriminators and incorporating the epistemic uncertainty across the ensemble into an additional intrinsic reward to the diversity of skills reward (through a mixing parameter $\\lambda$). They examine this method against DIAYN-style methods and count-based methods and show that this new approach broadly outperforms both these classes of methods.    ", "summary_of_the_review": "The paper introduces and solves an important problem, proposes a technically sound method, has a very clear presentation of related works, ideas, and results, and offers good experimental evidence for the claims.\n\nNevertheless, I believe certain clarifications/additions could improve the paper.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EbQCa_hz1K", "reviewer": "Reviewer_Upj9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: \nThe strengths are summarized below:\n\n1. The writing flow of the paper is very nice, including comprehensive background, easy-to-follow methodology. It would be better to pair with some nice figures.\n3. The experiments are pretty solid including various settings environments including tabular form and image-based tasks (Atari). Experiments in Atari are very solid.\n\nWeakness:\nI would like to raise several questions to the paper: \n\n1. Can the author provide some experimental supports for the argument that the reward provided by the discriminator is pessimistic in face of uncertain states? I am curious how the reward looks like in Eq. 3 when using functional approximation. \n\n2. As also noted in the paper, the idea of using the ensemble method to encourage exploration has been widely employed in the general RL setting. The proposed intrinsic reward is very similar to those adopted previously, despite the setting being unsupervised skill learning. Thus, I think the algorithm itself is not very novel.\n\n3. In terms of experiments, I think the author should add more comparison between the proposed method and more baselines: for example, a direct comparison would be RND + ensemble. \n\n4. I didn't see the learning curves for downstream tasks (maximize reward) of Atari in the appendix. The author can provide the learning curve for (d) in figure 4. \n\nClarity: The paper is well-written and clear in the flow.\n\nFeedbacks & Questions: Please see details in the weakness.", "summary_of_the_paper": "This paper adopts the framework of unsupervised skill learning. To solve the problem that the discriminator will have low confidence in the unseen data thus providing a low intrinsic reward, the paper derives an information gain auxiliary objective that involves training an ensemble of discriminators and rewarding the policy for their disagreement. The paper conducts extensive experiments on tabular grid world and 57 games of the Atari Suite. ", "summary_of_the_review": "Overall, I think the paper is well written and has extensive experiments. My main concern is about the novelty of the method. These concerns are pretty important and thus I encourage the author to engage in the discussion period and clarify these if there is any misunderstanding. I am happy to re-evaluate if the author convinces me. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the problem of exploration using intrinsic rewards in RL in states that have never been encountered before. The authors derive an information gain auxiliary objective that involves training an ensemble of discriminators and rewarding the policy for their disagreement, which estimates the epistemic uncertainty that comes from the discriminator not having seen enough training examples. The intrinsic reward resulting from the so-called DISDAIN (discriminator disagreement intrinsic reward) exploration bonus is more tailored to the true objective compared to pseudocount-based methods.

Reviewers agree that the paper is well-motivated and well-written, that the proposed DISDAIN exploration method is simple and practical, and that experiments are convincing. Experiments on continuous control tasks such as MuJoCo locomotion environments could have strengthened the paper further.
Decision: Accept (Spotlight)