DISCO-DANCE: Learning to Discover Skills with Guidance
Authors: Hyunseung Kim
Byungkun Lee
Sejik Park
Hojoon Lee
Dongyoon Hwang
Kyushik Min
Jaegul Choo
Venue: Submitted to ICLR 2023 (2023)
Keywords: Unsupervised skill discovery
Reinforcement Learning

Abstract: Unsupervised skill discovery (USD) allows agents to learn diverse and discriminable skills without access to pre-defined rewards,
by maximizing the mutual information (MI) between skills and states reached by each skill.
The most common problem of MI-based skill discovery is insufficient exploration, because each skill is heavily penalized when it deviates from its initial settlement. Recent works introduced an auxiliary reward to encourage the exploration of the agent via maximizing the state's epistemic uncertainty or entropy. 
However, we have discovered that the performance of these auxiliary rewards decreases as the environment becomes more challenging. Therefore, we introduce a new unsupervised skill discovery algorithm, skill discovery with guidance (DISCO-DANCE), which (1) selects the guide skill which  has the highest potential to reach the unexplored states, (2) guide other skills to follow the guide skill, then (3) the guided skills are diffused to maximize their discriminability in the unexplored states. Empirically, DISCO-DANCE substantially outperforms other USD baselines on challenging environments including two navigation benchmarks and a continuous control benchmark.

Under review as a conference paper at ICLR 2023
DISCO-DANCE: LEARNING TO DISCOVER SKILLS WITH GUIDANCE
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised skill discovery (USD) allows agents to learn diverse and discrim- inable skills without access to pre-defined rewards, by maximizing the mutual information (MI) between skills and states reached by each skill. The most com- mon problem of MI-based skill discovery is insufficient exploration, because each skill is heavily penalized when it deviates from its initial settlement. Recent works introduced an auxiliary reward to encourage the exploration of the agent via max- entropy. However, we have discov-
imizing the state's epistemic uncertainty ered that the performance of these auxiliary rewards decreases as the environment becomes more challenging. Therefore, we introduce new unsupervised skill dis- covery algorithm, skill discovery with guidance (DISCO-DANC) NCE),, which (1) se- lects the guide skill which has the highest potential to reach the unexplored states, (2) guide other skills to follow the guide skill, then (3) the guided skills are diffused to maximize their discriminability in the unexplored states. Empirically, DISCO- DANCE substantially outperforms other USD baselines on challenging environ- ments including two navigation benchmarks and continuous control benchmark.
INTRODUCTION
In recent years, Deep Reinforcement Learning (DRL) has shown great success in various complex tasks, ranging from playing video games (Mnih et al. 2015 Silver et al. (016) to complex robotic manipulation (Andrychowicz
al., 20177; Gu 2017). Despite their remarkable success, most DRI. models focus on training from scratch for every single task, which results in significant ineffi- ciency. In addition, the reward functions adopted for training the agents are generally handcrafted, acting impediment that prevents DRL to scale for various real- world tasks. For these reasons, there has been an increasing interest in training tank-agn policies without access to a pre-defined reward function (Campos et al.] 2020 Eysenbach et al., 2018 Gregor et al.) 2016 Hansen et al. 2019 Laskin et al. 2021 Liu & Abbeel [021 Sharma et al. [019]] Strouse et al. [0222] Park et al.
2022
Laskin et al. 2022 Shafiullah & Pinto, 2022). This training paradigm falls in the category
of Unsupervised Skill Discovery (USD) where the goal of the USD is to acquire diverse and dis- criminable behaviors, known skills. These pre-trained skills can be utilized as useful primitives or directly employed to solve various downstream tasks.
Most of the previous studies in USD (Achiam et al. 2018 Eysenbach et al. 2018 Gregor et al. 2016 Hansen al. 2019 Sharma al.) (019) discover of diverse and discriminable skills by maximizing the self-supervised, intrinsic motivation as form of reward. Commonly. mutual information (MI) between the skill's latent variables and the states reached by each skill is utilized as this sulf-super reward. However, it has been shown in recent research that solely maximizing the sum of MI rewards is insufficient to explore the state space because the agent receives larger rewards for visiting known states rather than for exploring the novel states asymptotically (Campos) ctt al. 2020 Liu & Abbeel 2021 Strouse et al. 2022).
To ameliorate this issue, recent studies designed auxiliary exploration reward that incentivizes the agent when it succeeds in visiting novel states (Strouse et al. 2022 Lee et al., 2019 Liu & Abbeel 2021). However, albeit provided these auxiliary rewards, previous approaches do not often work efficiently in complex environment. Fig. [eonceptuull illustrates how previous methods work ineffectively in simple environment. Suppose that the upper region in Fig. Th is hard to reach

---

Under review as a conference paper at ICLR 2023
(b) DISDAIN
Skill trajectories
reward
Exploration reward
Skill trajectories
06
(c) APS, SMM
06
Force
reward
Exploration reward
Total reward
Skill trajectories
(d) DISCO-DANCE
-02
Reward
Select guide
reward
Guide reward
Total reward
trajectories
Figure I: Conceptual illustration of previous methods and DISCO-DANCE, Each skill is shown with a grey-colored trajectory. Blue skill z' indicates an unconverged skill. Here, (b,c) illustrates reward landscape of previous methods, DISDAIN, APS, and SMM. (b) DISDAIN fails to reach upper region due to the absence of pathway to the unexplored states. (c) APS and SMM fail since they do not provide exact direction to the unexplored states. On the other hand, (d), DISCO-DANCE directly guides zf towards selected guide skill which has the highest potential to reach the unex- plored states. Detailed explanation of the limitations of each baselines described Section 2.2
with MI rewards, resulting in obtaining skills which are stuck the lower-left region. To make these skills explore the upper region, previous methods provide auxiliary exploration reward using intrinsic motivation (e.g., disagreement, curiosity based bonus). However, since they do not indicate exactly which direction explore, becomes more inefficient in challenging environments. We detail the limitations of previous approaches in Section 2.2
In response, we design new exploration objective that aims provide direct guidance to the unexplored states. To encourage skills to explore the unvisited states, we first pick guide skill which has the highest potential to reach the unexplored states (Fig. d--1). Next, we select the skills which will move towards a guide skill (e.g., relatively unconverged skills; receiving low MI rewards). Then they are incentivized to follow the guide skill, aiming to e over the region with low MI reward (Fig. [c-4)). Finally, they are diffused to maximize their distinguisability (Fig. Tc- 5)), resulting obtaining a set of skills with high state-coverage. We call this algorithm as skill discovery with guidance (DISCO-DANCE) and is further presented in Section 3 DISCO-DANCE can be thought as filling the pathway to the unexplored region with positive dense reward.
In Section 4 demonstrate that MISCO-DANCE outperforms previous approaches with auxiliary exploration reward in terms of state space coverage and downstream task performances in two nav- igation environments (2D mazes and Ant mazes), which have been commonly used to validate the performance of the USD agent (Campos et al. 2020 Kamienny et al. [202]). Furthermore, we also experiment in DMC (Tunyasuvunakool al. 2020 and show that the learned of skills from DISCO-DANCE provides better primitives for learning general behavior (e.g., run, jump and flip) compared to previous baselines.
PRELIMINARIES
In Section 2.1 we formalize USD and explain the inherent pessimism that arise in USD. Section describes existing exploration objectives for USD and the pitfalls of these exploration objectives.
2

---

Under review as a conference paper at ICLR 2023
2.1 UNSUPERVISED SKILL DISCOVERY AND INHERENT EXPLORATION PROBLEM
Unsupervised Skill Discovery (USD) aims to learn skills that can be further utilized as useful primi- tives directly used to solve various downstream tasks. We cast USD problem as discounted, finite horizon Markov decision process M with states s E S, action a E A. transition dynamics p E T, and discount factor ? Since USD trains the RL agents learn diverse skills in an unsupervised manner, we assume that the reward given from the environment is fixed to O. The skill is commonly formulated by introducing a skill latent 2 2 to a policy resulting in catent-conditooned policy T (als, 2). Here, the skill's s latent variable can be represented as an one-hot (i.e., discrete skill) or a continuous vector (i.e., continuous skill). In order to discover a set of diverse and discriminable skills, standard practice is to maximize the mutual information (MI) between state and skill's ' latent
variable [(: Z) (Achiam et al. 2018 Eysenbach et al. 2018 Gregor et al. 2016 Hansen et al,
2019 Sharma al. 2019).
I(2,S) (Z)) + H(Z) = (amp(s)) [og p(2[s) - p(z)
(1)
Since directly computing the posterior p(z[s) is intractable, learned parametric model (o(2)s), which we call discriminator, is introduced to derive the lower-bound of the MI instead.
S) 2 i(2,5) = [==p(())ss=ss (log go (=|ss logp(=)]
Then, the lower bound is maximized by optimizing the skill policy x (a]s, 2) via any RL algorithm with reward log qo (z|s) log p(z) (referred to rsan). Note that each skill-conditioedd policy gets different reward for visiting the same state (i.e., (2;, 8) + ram (2),s))).. It results in learning skills which visit different states, making them discriminable.
s)).
However, maximizing the MI objective is insufficient to fully explore the environment due to an inherent pessimism of its objective (Campos al.) 2020; Liu & Abbeel 2021 Strouse al. [0022). When the discriminator (=(=)s) succeeds to distinguish the skills, the agent receives larger rewards for visiting known states rather than for exploring the novel states. This lowers the state coverage of given environment, suggesting that there are limitations in discovering what skills are available (i.e., achieving a set of skills that only reach in a limited state space).
Therefore, recent studies in USD provide auxiliary rewards to overcome pessimistic exploration.
2.2 PREVIOUS EXPLORATION BONUS AND ITS LIMITATIONS
DISDAIN (Struse et al. 2022) trains an ensemble of N discriminators and rewards the agent for their disagreement, represented as H(; E Lo. (Z)s)) N Σ Hgo, (Z|s)). Since states which have not been visited frequently will have high disagreement among discriminators, DIS- DAIN implicitly encourages the agent move novel states. However, since such exploration bonus is a consumable resource that diminishes as training progresses, most skills will not benefit from this if other skills reach these new states first and carn the bonus reward.
We illustrate this problem in Fig. Пp. Suppose that all skills remain in the lower left states, which casy to reach with MI rewards. Since the states in lower left region is accumulated in the replay buffer, disagreement between discriminators remains low (e.g., low exploration reward in lower left region). Therefore, there will be exploration reward left in these states. This impedes 2; from escaping its current state to unexplored states, as shown in Fig. Db-4.
On the other hand, SMM encourages the agent to visit the states where it has not been before using learned density model de (Lee et al. 2201). APS incentivizes the agent to maximize the marginal state entropy via maximizing the distance of the encoded states (e()s) between its k nearest neighbor ff (st) (Liu & Abbeel 2021
SMM X - (o))
exploration
APS X log [fa(s) f3 (3)
exploration
These rewards push each skill out of its converged states (in Fig. [p]. However, they still do not provide a specific direction on where to move in order to reach unexplored states. Therefore, in difficult environment with larger state space, it is known that these exploration rewards can
3

---

Under review as a conference paper at ICLR 2023
operate inefficiently (Campos et al. 2020 Ecoffet al. 2019). In the next section, we introduce new exploration objective for USD which addresses these limitations and outperforms prior methods on challenging environments.
METHOD
Unlike previous approaches, we design a new exploration objective where the guide skill z" directly influences other skills to reach explored regions. DISCO-DANCE consists of two stages: (i) selecting guide skill zt and the apprentice skills which will be guided by z", and (ii) providing guidance to apprentice skills via guide reward, which will be described in Section 3.1 and respectively.
3.1 SELECTING GUIDE SKILL AND APPRENTICE SKILLS
injedories
Injjectoiios
dursity
guide
Figure 2: Guide skill selection, An illustration of a random walk process which finds the skill with highest potential that can reach an unexplored states.
Guide skill. We recall that our main objective is to obtain set of skills that provides high state space coverage. To make other skills reach unexplored state, we define the guide skill z" as the skill which is most adjacent to the unexplored states. One naive approach in selecting the guide skill is to choose the skill whose terminal state is most distant from the other skills' terminal state (e.g., blue skill in Fig. 2 I). However, such selection process does not take into account whether the guide skill's terminal state is neighboring promising unexplored states. In order to approximate whether skill's terminal state is near potentially unexplored states, we utilize a random walk process (Fig. 2 To be specific, when there are P skills, (i) first perform R random walks from each skill's terminal state and collect total PR number of random walk arrival states. Repeat (i) process for M times, collect PRM number of states in total. (ii) Then pinpoint the state in the lowest density region among the collected random walk arrival states and select the skill which that state originated from the guide skill 2*, For (ii), one could use any algorithm measure the density of the random walk state distribution. For our experiments, we utilize simple k nearest neighbor algorithm,
2" : arg max max PE (E......P ref rEft.....MMM
Σ
||spr
(epp))
where Spr r-th random walk arrival state of skill p
(- = k-nearest neighbors
In practice, in environment with long-horizon such as DMC, our random walk process may cause sample inefficiencies. Therefore, we present an efficient random walk process which we describe detail in Appendix Further ablation experiments on how to select guide skills are available in Section 4.4.1
Apprentice skills. We select apprentice skills as the skills with low discriminability (i.e., skill 2 with low (o(2"(s); which fails to converge) and move them towards the guide skill. If most of the skills are already well discriminable (i.c., high MI rewards), we simply add new skills and make them as apprentice skills. This will leave converged skills intact and send new skills to unexplored states. Since the total number of skills is gradually increasing as the new skills are added, this would bring the side benefit of not relying pre-defined number of total skills hyperparameter.
We note that adding new skills during training is generally difficult to apply to previous algorithms, because the skills will also face the pessimistic exploration problem. The new skills simply

---

Under review as a conference paper at ICLR 2023
exisling skills
guide skill
apprentice skill
Skill trajectories
Select guide skill (a) bottleneck maze
Skill trajeclories
guktanos
guidento
before gulsance
Afte guidance
trajectories
guide
Insjetories
(b) cheetah
Figure 3: Qualitative results of the guiding procedure in (a) navigation and (b) continuous control.
converge by overlapping with existing skills (e.g., left lower room Fig which exacerbates the situation (i.e., reducing discriminator accuracy without increasing the state coverage). In Ap- pendix D we empirically show that this curriculum procedure brings benefit to our methods, while the improvement over the baselines marginal.
3.2 PROVIDING DIRECT GUIDANCE VIA AUXILIARY REWARD
After selecting the guide skill = and apprentice skills, now formalize the exploration objective, considering two different aspects: (i) objective of the guidance, and (ii) the degree of the guidance. crucial account for these desiderata since strong guidance will lead apprentice skills simply imitate the guide skill z whereas the weak guidance will not be enough for skills to overcome pessimistic exploration.
In response, we propose a exploration objective that enables our agent to learn with guidance. We integrate these considerations into a single soft constraint as
maximize =p [rsku [guide] where raill = log (= (2'1s)
- logp(=²),
(5)
Iguide = a (29((=))s)) S <) (1 (=(())))) Dxu (1) (a =²)).
As we describe in Section 3.1 we select skills with low discriminator accuracy (i.c., I(go (z*ls) e))
apprentice skills. For (1), minimize the KI-ddveergence between the apprentice skill and the guide skill policy (i.e., DKL (To (a]ss, (T)pss =))). For (ii), the extent to which the apprentice skills are encouraged follow z", can represented as the weight the KL-divergence. We the weight as 1 - (=*),s)) to make the skills with low discriminator accuracy to be guided more.
Fig. 3 shows that this guidance boosts skill learning for both navigation environment and continuous control environment. In navigation task (Fig. Bh), the skill in the upper left is selected as guide through the random walk process and the apprentice skills stuck the lower left are directly dragged into the unexplored states by guide skill. In addition, in continuous control environment (Fig. 3p), the skill that learned run is selected as the guide skill, leading the apprentice skills that are barely moving. Then the apprentice skills learns to run, in an instant. This salient observation suggests that the concept of guidance can be utilized, even for non-navigation tasks. Section 4.4.2] we show that all three component of guude necessary through additional experiments.
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Environments We evaluate DISCO-DANCE on three different types environments: 2D naviga- tion, Ant Maze, and Deepmind Control Sutie (DMC). Fig (a) illustrates five different layouts in

---

Under review as a conference paper at ICLR 2023
Empty
W
Bottleneck
Ant Empty
Ant
Ant
Chestah
Quadruped
(a) 2D Maze
(b) Ant Maze
(c) DMC
Figure 4: Three environments to evaluate skill discovery algorithms. (a) Fully-continuous 2D mazes with various layouts, (b) High dimensional ant navigation tasks and (c) Continuous control environment with diverse downstream tasks.
2D navigation environment. The layout becomes more challenging from empty maze to bottleneck maze (i.e., Easy-Emptey Normal-Y. Y, W. S. Hard-Bottee BotteeBkttleeeck.. These environment has been commonly
used for testing the exploration ability of skill-learning agents (Campos et al. 2020 Kamienny et al, 2021]. The agent determines where it should move given the current x, y coordinates (2-dimensional state, action space). Fig (b) shows three different layouts of AntMaze environment from (Chane- Sane al., [ Nasiriany et 2019) which aims evaluate the effectiveness of algorithm with high-dimensional input. In Antmaze, the state space consists of joint angles, joint velocity and the center of mass and the action space consists of torques of each joint. Following previous work (Ey- senbach et al., 2018: (Sharma et al.) 2019), we restrain the input of the discriminator as x.y coordi- nates since the main goal in navigation environments is to learn set of skills which are capable to cover the majority of the state space. (i) displays two different environment in DMC, Cheetah and Quadruped. DMC is selected to evaluate the algorithm's capability of learning general skills. Following URLB (Laskin et al., 2021), we consider eight different tasks (e.g., walk, run, jump, flip).
Evaluation We use the state space coverage and downstream task performances as our main metrics. To measure the state coverage, we discretize the x and y axes of the environment into 10 buckets (i.e.. total 100 buckets) and count the number of buckets reached by learned skills (Fig. A...) For 2D mazes, we trained each algorithm with 2M training steps for easy and medium levels, 5M training steps for hard level (Fig. 4a). For Ant mazes, we trained for 5M training steps for the high-dimensional Ant navigation tasks (Fig. 4D). For DMC, after 2M steps pretraining with USD algorithms, we first select the skill with maximum downstream task reward, and then finetune 100k steps for each task. (Fig. 4c). For detailed explanation, see Appendix B. Bold scores in Table indicate the best model performance and underlined scores indicate the second best.
Baselines We compare DISCO-DANCE with DIAYN (Eysenbach et al. 2018), which is the most widely used baseline in USD. We also compare DISCO-DANCE with SMM (Lee al., 2019), APS (Liu Abbeel 2021) and DISDAIN (Strouse et al. 2022). We note that all baselimes and DISCO-DANCE utilize discrete skills, except APS which utilizes continuous skills. In downstream tasks, we compare baselines with random inil baseline that initialized from scratch. We use SAC as the backbone RL algorithm (Haarnoja et al. (2018). We train DIAYN, APS and SMM using the code and hyperparameters provided by URLB (Laskin et al. 2021). In addition, we re-implement DIS- DAIN by strictly following the details of the paper. Additional details are available in Appendix B
4.2
NAVIGATION ENVIRONMENTS
4.2.1
2D MAZES
Table 1: State space coverages of DISCO-DANCE and baselines on 2D maze. The results are averaged over 5 random seeds accompanied with a standard deviation.
Models DIAYN
Fmppyy
Y
W
Bottleneck
100.00-44.00
71.20 +:500
71.40
+4.62
53.80+7.01
53.00-4.36
SMM
100.00-10.00
89.60++.50
74.20:16.22
57.20=7.46
57.20
APS
95.80:1..55
90.20
+:5022
74.20:1 10.62
77.60 17.50
61.80 15.06
DISDAIN
100.00 +-0.00
82.40
+3,91
84.80+ 11.30
61.80+7.33
60.60
DISCO-DANCE
100.00:10.00
99.00-1141
94.80.1.49
83.00=1.22
87.40-111.59
Table summarizes the performance of skill learning algorithms on the 2D maze environments. Through empirical results, observe the following results: methods that utilize auxiliary

---

Under review as a conference paper at ICLR 2023
reward (i.e., SMM, APS, and DISDAIN) outperform DIAYN for every maze, showing that the aux- iliary exploration rewards encourage the agent to visit unseen states. However, we find out that previous studies provide less benefit the performance as the layout becomes complicated. As mentioned in section 2.2, we empirically discover that auxiliary exploration reward approaches are less useful when the environments getting more complicated. In contrast, for each 21) maze environ- ment, DISCO-DANCE has negligible performance reduction with increased layout complexity. In addition, DISCO-DANCE outperforms previous methods in every 2D maze environment, exhibiting the largest performance gap against baselines in the most complex environment, bottleneck maze.
DIAYN
(b) DISDAIN
(c) SMM
(PI
DISCO-DANCE
Figure 5: Visualization of the learned skills on bottleneck maze. Multiple rollouts by each skill discovery algorithm.
Fig. illustrates multiple rollouts of various skills learned in the 2D bottleneck maze. For the 2D bottleneck maze, the upper left room is the most difficult region to reach since the agents need to pass multiple narrow pathways. While other baselines methods are not able to effectively explore the upper left region, only DISCO-DANCE is able to fully cover the environment.
4.2.2 ANT MAZES
Table reports the state coverage of DISCO- DANCE and baselines. We find that among baselines, only SMM gets better performance than DIAYN. We speculate that additional re- ward signal may interrupt the agent to converge in high-dimensional environment. Empirically, DISCO-DANEE shows superior performance the baselines where achieves the best state- coverage performance in high-dimensional en- vironments which contains obstacles (i.e., Ant U-maze and IT-maze) and gets competitive results against SMM with marginal performance gap in the environment without any obstacle (i.e., Ant Empty-maze).
Table 2: State space coverages of DISCO-
DANCE and baselines on Ant Maze. The results averaged over random seeds.
Models
Ant Empty-maze
Ant U-maze
Ant TT-maze
DIAYN
74.80:15577
50.40=10.50
22.80::.35
SMM APS
99,80-:0.45
63.00-7.58
27.20+5503
73.40::3..4
61.20=396
27.448.79
DISDAIN
$4,401133.0 06
35.60-28.50
19.40-11.19
DISCO-DANCE
95.40:0089
72.00-8.12
34,60:5008
In addition, we conduct experiment to eval- uate whether the learned skills could be a good starting point for downstream task which aims to reach goals (Fig. 6). We set the farthest re- gion from the initial state the goal state (the
U-maze
Ant N-maze
SUccesS rate
success rate
most difficult configurations) following Chane- Sane al. (2021) and measure the success rate of reaching the established goal state. For methods with discrete skills, we select the skill
A2MAA
64
25
65
environment steps
environment steps
Figure 6: Accelerating learning with policy ini- tialization. Curves averaged seeds with standard deviation.
with the maximum return (i.e., skill whose state space closest the goal state) and initialize the policy and value network with the selected skill. We then fine-tune the initialized agent to maximize the reward (i.e., minimize distance to goal state) with IM training steps. For methods with continuous skills, follow the fine-tuning proce- dure in APS (Liu Abbeel (2221).
random
In Ant U-maze, DISCO-DANCE outperforms prior methods including random init in terms of sam- ple efficiency, and successfully reaches the goal-state with only 100k interactions. In the most chal- lenging environment, Ant II ml-ma only DISCO-DANCE
able succeed in reaching the goal state within 1M environment interactions. We also observe that baselines underperform in com- parison with random init. As shown in URLB continuous control tasks (Laskin al. 2021

---

Under review as a conference paper at ICLR 2023
conjecture that task-specific training might be more efficient than transferring general behavior. Additional details about the finetuning tasks are available in Appendix B.1
4.3
DEEPMIND CONTROL SUITE
DISRAIN
SMV
RANDOO INIT
APS
DISED-OANCE
VEDWN
IOM
MEAN
Optimallty
APS
DEANN
SYM
DISDAIN
0.45
0.60
0.75
0.45
0.60
0.75
0.4
0.5
0.6
0.7
0.4
0.5
0.6
Expert Nermalzzdd Score
22 3.1 04
Figure 7: Performance comparisons on Deepmind Control Suite. Aggregate statistics and perfor- mance profiles with 95% bootstrap confidence interval are provided (Agarwal et al. 2021). which are calculated across 72 seeds (3 pretrained models x 3 seeds x 8 tasks) for all algorithms.
To demonstrate that DISCO-DANCE can also effectively learn the general skills in environments other than navigation related environments, we conduct additional experiments on Deepmind Con- trol Suite (DMC) (Tunyasuvunakool et al. 2020). Fig. shows the performance comparisons on DMC across 8 tasks with seeds. As shown Fig. 3.. find that the concept of guidance boosts the skill learning process in the continuous control environment, achieving comparably good results in comparison with other baselines. Note that the performance of baselines with discrete skills are equal or less than the randomly initialized SAC, similarly to Ant mazes (Fig. 6 We suspect that the current finetuning strategy widely used in USD research for discrete skills (only finetune one selected skill among all skills) is not able to properly evaluate the performance of the obtained set of skills on downstream tasks. We believe that new finetuning methods that can fully leverage the char- acteristics of skills learned through USD will be an exciting direction for future work. We provide the full performance table at Section F
4.4 ABLATION STUDIES
To further understand the role of each model components we perform two ablation studies: ablation of (1) the guide skill selection and (2) the importance of the KL coefficient Eq.
Random walk
Terminal states
w! KL coefficient
wio KL coefficient
state coveraga (N6
bettleneek mare
stato coverage (96)
buctlerreck mene
state coverage (96
Ana mane
35
54 3p
environment
environment
environmen: steps
(a) Guide skill selection
(b) KL coefficient
Figure 8: (a) Comparing state coverage with random walk guide skill selection and terminal state
guide skill selection process. (b) Importance of KL coefficient in Eq
5
4.4.1 GUIDE SKILL SELECTION PROCESS
We empirically compare our random walk guide skill se- lection process with the most distant terminal state guide skill selection process. in 2D bottleneck maze and AntU maze. Results in Fig- Eg shows that the random walk guide selection process is superior to the terminal state guide se- lection process in terms of state coverage. We also provide qualitative results where the terminal state guide selection process fails to select the skill which is closest to the un- explored region the guide skill while the random walk guide selection process successfully finds the appropriate guide skill in Fig. 9
Figure 9: Qualitative results for differ- ent guide skill selection processes.
8

---

Under review as a conference paper at ICLR 2023
4.4.2 KL COEFFICIENT
We fürther conduct an ablation study to show the importance of the KL coefficient in the DISCO- DANCE objective. Specifically, we report the state coverage performance of DISCO-DANCE when (s) in Equation (i.e. KL coefficient as 1). to figure out the influence of guiding only apprentice skills (i.c..g make all other skills as apprentice skills). Fig B shows that guiding too many skills could degrade performance in high- dimension environment and that it is important to only select unconverged skills as apprentice skills for effective learning.
trained without the KL coefficient, [[go(2"ls) < <c-
5 RELATED WORK
Previous research in Unsupervised Skill Discovery. Several approaches have been made to learn a set of useful skills in fully-unsupervised manner. Since MI based method suffers from inherent pes- simism, EDL (Campos et al. 2020) decomposes training procedure, first explores the environment with SMM (Lee et al. 2019). and then utilize VAE to discover skills. Since they completely sepa- rate between the exploration phase and skill learning phase, EDL relies heavily on the performance of exploration module. CIC (Laskin al., 2022) uses noise contrastive estimation (NCE) (Gutmann & Hyvärinen| 2010) to enable learning high-dimensional skills for diverse behaviors. L.DD) (Park regularization term to learn dynamic behaviors, and has
2022) utilizes Lipschitz-constrainn shown that it works well in downstream task in a zero-shot manner. DISK (Shafiullah & Pinto 2022) aims to learn diverse skills which can quickly adapt a non-stationary environment. To achieve this, Disk learns skills in an incremental fashion, which encourages new skill to be consistent within itself and to be diverse from the previous skills. UPSIDE (Kamienny et al., (02T) utilizes a hierarchical tree-structure which execute multiple skills sequentially to reach distant states. Further discussion of the differences between DISCO-DANCE and UPSIDE are available in Appendix G
Guidance based exploration in RL. Go- (Ecoffet et al., 2021 2019) stores the visited states in the buffer and starts re-exploration from the samples states in the buffer to boost the exploration process. DISCO-DANCE and Go-Explore share similar motivation that the agent guides itself:
DISCO-DANCE learns from other skills and Go-Explore learns from previous experiments that have reached novel states. Another line of guided exploration is to utilize KL-regularrizaion between the policy and the demonstration (i.e., guide) (Chane- -ane et al. [022]] et al.) al.] (2020).
SPIRL (Pertsch et al., 2020) which is an offline supervised skill discovery method, denotes action sequences as (z) and Tearns the skill prior p(z[s) using offline data. For finetuning, it reduces the KL divergence between the agent's policy and the skill prior as an auxiliary reward accelerate exploration for the new task. RIS (Chane-Sane et al.) 2021 I has been proposed to stimulate efficient exploration for gonl-conditioned RI, where reaching distant goal is challenging due to the sparse rewards. RIS efficiently learns to reach distant goals by imitating the policy that aims to reach a subgoal generated between its position and distant goal, which makes the rewards more accessible.
CONCLUSION
In this paper, we introduce DISCO-DANCE, a novel, efficient exploration strategy for USD. It di- rectly guides the skills towards the unexplored states, by forcing them to follow the guide skill. We provide quantitative and qualitative experimental results which demonstrates that DISCO-DANCE outperforms existing methods in two navigation benchmarks and continuous control benchmark.
Although not thoroughly studied in this paper, we think that there is still enough room for improve- ment in finetuning strategies (Laskin et al. 2021). Since USD learns many different task-agnostic behaviors, a new fine-tuning strategy that can take advantage of these points would make the down- stream task performances more powerful. Also, while this paper focuses on learning discrete skills, the concept of our guide and explore can be extended to continuous skill spaces. DISCO-DANCE can easily be extended to jointly work with other algorithms which aim learn diverse behavioral patterns (Achiam et al. 2018). Since DISCO-DA -ANCE accelerates the exploration process, such al- gorithms can benefit by learning skills from a wider reachable state-space. Finally, discussion of limitation and future directions are available in Appendix

---

Under review as a conference paper at ICLR 2023
Reproducibility Statement To ensure reproducibilityy we provide (i) pseudo-code and implemen- tation detail in Appendix [A] (ii) all hyperparameters Table (iii) thorough explanation of three environments in Appendix BBI and (iv) implementation details of other baselines in Appendix B.2 We will share the self-contained code when the discussion period opens.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint 1807. 2018.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 34:29304-29320, 2021.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. NIPS, 30, 2017.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Interna- lional Conference on Machine Learning, pp. 1317-1327. PMLR, 2020.
Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning. pp. 1430-1440. PMLR, 2021.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a approach for hard-exploration problems. arXiv preprint arXiv:/901. 10995, 2019.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 59007847):500-586, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. Proc. the International Conference on Learning Rep- resentations (ICLR), 2018.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07500 2016.
Shixiang Gu, Ethan Holly. Timothy Lillicrap. and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer- ence on robotics and automation (ICRA), pp. 3389-3336. IEEE, 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304. PMLR, 2010.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proc. the International Conference on Machine Learning (ICML), pp. 1861-1870, 2018
Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. Proc. the International Conference on Learning Representations (ICLR), 2019.
Pierre-Alexande Kamienny, Jean Tarbouriech, Alessandro Lazaric, and Ludovic Denoyer. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. Proc. the International Conference on Learning Representations (ICLR), 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:141 1412 69800 2014.
10

---

Under review as a conference paper at ICLR 2023
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Ler- rel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2021.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:2202.00166 2022.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274, 2019,
Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In Proc. the Interna- tional Conference on Machine Learning (ICML), 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 5 518(7540):52955333 2015.
Soroush Nasiriany, Vitchyr H. Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. Proc. the Advances Neural Information Processing Systems (NeurIPS), 2019.
Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. Proc. the International Conference on Learning Repre- sentations (ICLR), 2022.
Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelerating reinforcement learning with learned skill priors. CoRL, 2020.
Nur Muhammad Shafiullah and Lerrel Pinto. One after another: Learning incremental skills for changing world. Proc. the International Conference on Learning Representations (ICLR), 2022.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. Proc. the International Conference on Learning Representations (ICLR), 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
DJ Strouse, Kate Baumli, David Warde-Farley, Vlad Mnih, and Steven Hansen. Learning more skills through optimistic exploration. Proc. the International Conference Learning Representations (ICLR), 2022.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm.ontrol Software Impacts, 2020.

---

Under review as a conference paper at ICLR 2023
A DETAILS OF DISCO-DANCE
Algorithm 1: Skill Discovery through Guidance
Initialize replay buffer D, skills 2 2....,.."" KL coefficient a, discriminator accuracy cutoff' Initialize policy The re(a)s,=), Skill discriminator (=(8))
for k 2. do
Collect experience in D using in the environment
Sample batch (st, at, $1+1,2) D
if Most existing skills are discriminable enough then
L Select guide skill x² t-
skill( x'....,"" 2*)
rsail = log 90 (2((5))))) log p(2)
Yguide - (=)((=>)))) < (1 [o(z|s)) (To =4)
10
r Taail + a ''guide
Update Te to maximize sum of
12
Update ds to maximize log 4o (z|s²)
The full DISCO-DANCE algorithm is shown in Algorithm We list the implementation details of below.
Entropy term in SAC Haarnoja et al. (2018) can be seen as KL divergence between the policy and uniform distribution with constant Pertsch et al.] (2020) (H(TT- (ss 2))) x - (T (T((-1,,, 2t (.)))((.)) Instead of replacing the entropy term with guide term in DISCO- DANCE, we found that using both term simultaneously helped stabilize learning by enjoy- ing the advantage of maximum entropy RL.
We utilize KL-regularization between the overlapping skill and the guide skill policy. However, the guide skill policy changes during training. To improve the stability of our method,
a separate target policy network for the guide skill. More precisely, when we select the guide skill, we clone the policy network to obtain a target policy network and use the target network for generating the fixed guide skill policy outputs.
For adding new skills, we need to define hyperparameters: when to extend skills and how many to increase. We gradually extend skills as the discriminator converges. In detail, count the number skills that have low discriminator accuracy (i.e., less than e) and the number is less than the threshold (i.e., p), extend skills.
As the number of skills is gradually increasing, the discriminator struggles to learn due to the sparse update for each skill by the decreased probability of sampling each skill. We mitigate this issue by resorting skill sampling scheme where each skill sampling weight is proportionate to its discriminator error (e.g., skills with high discriminator error are more likely to be sampled
When extending new skills, initialize their policy weights with the guide skill policy weights rather than random weights as in SPIRL Pertsch et al.] (2020). This accelerates exploration by encouraging the newly added skills to directly reach the unexplored regions.
After following the guide skill, the overlapped skills need to get diffused to maximize their distinguishabiiity When getting diffused, KL divergence between the overlapped skills and the guide skill becomes larger. To allow this, the skills get the same KL divergence when the KL divergence is lower than threshold (i.e., 17).
We include the number of timesteps used in guide skill selection in the total pretraining timesteps. We measure the state coverage with the checkpoints of each pretrained model with earlier timesteps. For example, in 2D bottleneck maze, 135, 000 samples are used for selecting guide skill, therefore we report the state coverage with model checkpoints at t == 4, 500, 000 (<5M- 135,000).
For Antmaze, we select 3 pretrained models among 5 random seeds and fine-tuned each model for random seeds. In Fig b All curves are averaged over 9 random seeds (3 pre- trained models x 3 seeds for all algorithms.
12

---

Under review as a conference paper at ICLR 2023
B FURTHER DETAILS ON ENVIRONMENTS AND HYPERPARAMETERS
B. ENVIRONMENTS AND EVALUATION
2D maze The width and height of easy and medium level mazes are both 5, and for hard level maze Campos al. (2020); Kamienny et al. (202T). There is explicit terminal state, and episode ends only when the maximum timestep (30 for easy and medium levels and 50 for hard level) is reached.
Ant maze The width and height of maze are (7.7) and Empty, II-shaped maze are (8,8) re- spectively Chane-Sane al. (2021); Nasiriany al. (2019). Similar 2d-maze, episode ends only when the maximum timestep (400 for every mazes) is reached. For goal-reaching down- stream tasks, we design distance between current state and goal state as a penalizing reward (i.e., -1x distance = reward). In detail, directly computing distance with x.y coordinates might be inac- curate because there can be obstacles (i.e., walls) between two states. Therefore, we design a reward function that returns the actual distance that goes around when there is an obstacle between two states. Then, we normalize the reward to set the lowest value as -.
Deepmind Control Suite We conduct our environment in total of eight downstream tasks pro- vided by URLB (Laskin et al. 202Tp. In detail, we utilize four tasks each two continuous control domains: Run, Run backward, Flip, and Flip backward for Cheetah and Jump, Run, Stand, and Walk for Quadruped. Following the convention of Laskin et al. 202T), we define expert score as the per- formance of randomly initialized SAC for 2M timestep, whereas only 100k timesteps for finetuning are allowed. For DIAYN, SMM, DISDAIN and DISCO-DANCE (discrete skills), we finetune the skill with maximum downstream task reward (same as Ant mazes). For APS (continuous skill), we first randomly select an arbitrary skill and rollout episodes, then solve thde linear regression problem select the task-specific skill (following same protocol in APS paper). After the skill is selected for finetuning, for all algorithms, selected skill is fine-tuned for 100K steps. Note that this different from URLB in that the skill selection process included in 100K, but this is still fair comparison as all algorithms use the same number of interactions (samples) to select skills.
B.2 HYPERPARAMETERS
Detailed hyperparameters of our method DISCO-DANCE are listed in the table
Table 3: Hyperparameters used in DISCO-DANCE.
Hyperparameter
2D maze
Ant maze
DMC
Hidden dim
128
1024
1024
Batch size
64
512
1024
easy, normal: 30
Skill trajectory length
400
1000
hard: 50
Initial number of skill Replay buffer size
30
10
10
106
106
106
Optimizer
Adam
Adam
Adam
Discriminator learning rate
3 x 10 44
3 x 10-4
3 x 10
Critic learning rate
10-4
3 x 10-4
x 10-4
Actor learning rate
3 10-4
3 x 10-4
x 10
RL algorithm
Soft Actor Critic 0.99
Soft Actor Critic
Soft Actor Critic 0.99
discount
0.99
Entropy coef (B)
0.2
0.6
0.2
Select guide
K nearest neighbor
K nearest neighbor
K nearest neighbor
KL coef (a)
1
10-4
10-4
Number of skills to extend Extending threshold
5
5
5
5
Accuracy threshold
0.5
0.5
0.5
KI. threshlod (7)
10
10
10
13

---

Under review as a conference paper at ICLR 2023
We construct two-layer MLP with 128 (2D maze) or 1024 (Ant maze, DMC) hidden units, and use adam Kingma & Ba (2014) optimizer with 3 x 10 learning rate. For each skill, we sam- ple 50 (bottleneck maze), 400 (Ant maze), 1000 (DMC) length trajectory and train the agent with SAC Haarnoja et al. (2018) using 64 (2D maze), 512 (Ant maze), 1024 (DMC) batch size. We set the discount factor y 0.99 and the SAC entropy coefficient / as 0.2 for 2D maze and DMC, 0.6
for Ant maze.
For DISDAIN, we search the number of ensembles from [5, 10, 20, 40J and disdain reward coeffi- cient from [10, 50, 100]. We train DISDAIN using 40 size of ensembles and set DISDAIN reward weight as 100, for all environments. We also train separate Q-funcetions for rskill and ddssdain and add two Q-values with weight 0.7 (reward = Tskill + 0.7 rdisdain) following (Strouse et al. 2022).
For APS, we search the value of (i.e., number of neighbors in k nearest neighbor algorithm) from [5, 10] and the dimension of skills from [5, 10]. For 2d-mazes, we utilize 10 for k and 5 for skill dimensions. For Ant mazes, we utilize for and for skill dimensions. For DMC, we utilize 12 for k and 10 for skill dimensions.
For SMM, we strictly follow the implementation detail from the paper. We search the entropy coef- ficient from 10.0005, 0.005, 0.01, 0.1, 0.25, 0.5, 1J. We utilize entropy coefficient as 0.1 for 2D maze and Antmaze, 0.01 for DMC.
For the methods using the curriculum approach, we use the initial number of skills 30 for the 2D maze, and 10 for the Antmaze and DMC. During training, count the number of skills in which discriminator accuracy is less than the threshold 0.5, and if it is less than the extending threshold 5, extend 5 number skills.
C ADDITIONAL ABLATION STUDIES
w/ KL confficient
wlo KL coefficient
0.5
= 0.7
= 0.3
bottieneck maze
Antu maze
bottlenack maze
800
322
200
state stale covergg
60-
22
envfronment stops
enwiron steps
environment steps
(a) Guided initialization
(b) Threshold sensitivity
Figure 10: Ablation studies. (a) Guided initialization. (b) Sensitivity to the accuracy threshold.
During the curriculum learning procedure, we initialize the weights of newly added skills using the guide skill directly reach the unexplored states. To analyze the importance of initialization, compare this approach to an ablation that training from scratch. In (0 a), initializing the weights by using the guide skill makes the performance better. We speculate that guide initialization is di- rectly minimizing the kl divergence with guide skill, it makes the agent start with the maximum guide reward. In order further analyze the sensitivity of the accuracy threshold e, conduct ablation study. We vary the accuracy threshold from {0.3, 0.5, 0.7} and fix the remaining hyperpa- rameters. In Fig 10 b), we observe that the performance of DISCO-DANCE is robust the accuracy threshold c.
D COMPARING CURRICULUM APPROACH EFFECTS ON USD METHODS
To demonstrate DISCO-DANCE is highly compatible with increasing the number or skills during training, also combine the curriculum approach to the baselines which utilize discrete skill spaces
14

---

Under review as a conference paper at ICLR 2023
DISCO-DANCE DIAYN' DISDAIN
existing skills new skills
state coverage (96)
environment steps
Soreeae (%6
state Sooe
DIAYN
DISDAIN! (a) Skill trajectories
DISCO DANCE
environment steps
(b) State space coverage
Figure 11: Gradually increasing the number of skills for DISCO-DANCE, DIAYN, and DIS- DAIN. (a) Qualitative results on bottleneck maze and Ant ml-maze. Colored skills indicate new skills and grey skills are previously converged skills. (b) Training curves on each environment.
for comparison. In order to empirically verify whether previous approaches can also benefit from curriculum procedure, we combined the curriculum approach mentioned in Section 3.1 to DIAYN and DISDAIN As shown in Fig. 1 we discover that DIAYN and DISDAIN are not able to benefit from curriculum learning and that the newly added skills are mainly rooted near the initial starting point. This results in making the covered state space more crowded, so the curriculum approach not only does not help to cover more state spaces, but exacerbate the situation by reducing discriminator accuracy without increasing the state coverage. However for DISCO-DANCE, the newly added skills are able to directly reach unexplored regions with the aid the guide skill.
E EFFICIENT RANDOM WALK PROCESS
Our random walk process in Section 3.1 is as follows
1. After performing rollout for all P learned skills till the terminal state, we perform R random walks and collect P x R number of random arrival states (R hyper-parameter and we set R as 0.2 time horizon T for all environments). Repeat this process a total of M times.
2. Pinpoint the state the lowest density region among the P * R M number of states. We measure the density of the states using the k nearest neighbors.
3. Select the skill which that state originated from.
The number of environment steps in the random walk process is P (T + 0.27) * M. Therefore, the total number of environment steps performed during the random walk process pretraining stage is,
(Ninitial + (Niiitial + 5) + (NVintial + +
* +.. (NViinal - 6)) (T + 0.22") x M
where Nuuual the number of initial skills, is the number of skills to extend when most of the
existing skills converged (e.g., number of skills are 10 15 - 20 ... - 50 55 when 11 8=5, N5mm=555) Note that Nunal is the number of skills when the pretraining is ended (not predefined as a hyperparameter).
However, in long-horizon environment such as DMC (1000 timesteps), our random walk process can cause non-negligible sample inefficiencies. Thus, we further present an efficient random walk process that approximates an original random walk process with a simple approach. The detailed process is as follows:
1. During pretraining, if the skill satisfies go (o())s)) e in the terminal state sr (i.e., which is not an apprentice skill), additionally perform R random walk steps and store the random arrival states at the temporary buffer (queue).
15

---

Under review as a conference paper at ICLR 2023
2. When selecting guide skill, instead of rolling out all skills, select the state in the lowest density region among the states at the temporary buffer.
3. Select the skill which that state originated from.
Since there are no additional environment steps when selecting the guide skill, the efficient random walk process can significantly reduce the number of environment steps compared to the original ver- sion (added environment steps 0.27 per trajectory (only if its skill satisfies accuracy threshold), whereas original random walk process requires P x (T + 0.27) x M). In Fig and Table 4 we empirically show that the efficient random walk process still achieves comparably good scores on DMC.
F FULL RESULTS ON DEEPMIND CONTROL SUITE TASKS
Table 4: Performance comparison of DISCO-DANCE and baselines on DMC. Bold scores indicate the best model performance and underlined scores indicate the second best. The results are averaged over 9 random seeds with a standard deviation.
Models
Cheetah
Quatruped
Avg
Rus
Run Rockward
Flip
Flip Backwasd
Jump
Run
Staad
Wilk
Ranslm Init
8.55
384 1744
708.02: 17.87
715.07:9:26
383 41:14112
150
450.59 180.10
147.74:200.66
416.4%
DIAYN
144.66
534.23:12.11
422.67
SMM
360
106.8319000
682.73:-3..8
11:01
176 12:41777777
DISDAIN
22.15=9.89
18.7426.73
00:0882552
626.36-99133
13:11:17
$30.57:151.74
479.03-10..3
350.60
DISCO-DANCE
5011294085T5
418/259816007
671 071/46699130
510 38089199
200399-291.46
330 72:30473
334.40
Table summarizes the full results for each task in DMC. We can observe that DISCO-DANCE gets first or second performance in seven out of eight tasks. In contrast, other baselines show leading per- formance only for a few tasks (i.e., DISDAIN underperforms in cheetah domain tasks). We can find out DISCO-DANCE outperforms other baselines and random init in terms of average performance, and this indicates that DISCO-DANCE, learned more general behaviors through guidance.
G COMPARISON WITH UPSIDE
DISCO-DANCE and UPSIDE Kamienny al. 2021 have similar motivation, alleviating the in- herent pessimism problem. Both of their strategies can be summarized as "agent guides itself" (i.e., existing skills help new/unconverged skills to explore). In DISCO-DANCE, guide skill encourages apprentice skills (unconverged skills) to explore unexplored states by minimizing the KL divergence between guide skill and apprentice skills. While DISCO-DANCE learns a set of single skills, UP- SIDE learns tree-structured policy which is composed of multiple skill segments.
Fig illustrates an example where a total of skills are Iearned. When learning the policy in the unsupervised pretraining stage, UPSIDE (1) selects the skill with the largest discriminator accuracy among the leaf node skills (e.g., 2 = 4), (2) adds new skills as its children nodes (e.g., 2 = 7, 8), (3) freeze the parent skills (e.g., 2 1, are not trained) and only train newly added skills, and (4) iter- atively repeat these procedures to construct the tree-structured policies (Fig 12 b)). This makes the fundamental difference between DISCO-DANCE and UPSIDE, that UPSIDE requires sequential execution from ancestors' skills to child skill in top-down manner due to its tree-structured
skill policies. For example, if we want to run skill x 8, UPSIDE needs to execute skills sequen- tially (z = 1 - 4 + 8) for T timesteps, where T is hyperparameter that decides how dense the generated tree will be.
One of the key objectives of unsupervised skill discovery is to utilize the learned skills as a useful primitive at the fine-tuning stage. However, these sequential executions of UPSIDE bring signif- icant inefficiency at the finetuning stage, due to the following reason. Suppose the given down- stream task is to reach the goal state as fast as possible where the distance from the goal state and total execution time is given penalty (Fig 12 Since
8 is the skill with minimum distance selects skill 2 = 8 and finetunes ((a) (s, & = 8) to optimize given re- ward functions (Fig 12 a)). However, for UPSIDE to finetune 8, we need to finetune 1,4,8
from the goal,
16

---

Under review as a conference paper at ICLR 2023
Optimal trajectory
Fine-tuned trajectory
goal
goa
goal
2-1
2-3
2+1
2
2-5
2-5
2
2-2
Z-2
2-2
Tree structure broken
(a) DISCO-DANCE: Finetune skill x =
(b) UPSIDE:
(c) UPSIDE:
Finetune only leaf skill
Finetune = 1,4,8 simultaneoualy
Figure 12: Conceptual illustration of (1) learned skills of DISCO-DANCE and UPSIDE, and
(2) goal-reaching downstream task. Black edge represents skill and black circle represents the terminal states of its skill. (a) After unsupervised pretraining, DISCO-DANCE obtains set of skills that can be used individually without the need for each skill to be used together. In finetuning stage, DISCO-DANCE selects skill z = 8 (i.e., skill with highest downstream task reward) and finetunes the skill to reach goal fastly. (b) However, to execute z 8, UPSIDE requires sequential execution of its all ancestors skills (i.e., executing = 1,4, sequentially). In finetuning stage, UPSIDE is not able to learn optimal policy since the ancestors skills are kept fixed during finetuning. (c) finetune ancestors skills simultancously, the tree structure learned in pretraining phase will be broken, which makes finetuning unstable and difficult.
simultaneously (Fig 12 c)). In that case, the tree-structured skill policies learned in the pretraining stage is broken during the finetuning stage. That is, the dictionary {tz = 8' [1, 4, 8]} cannot be used anymore because executing = 1 for T timesteps does not move the agent to the original 2 terminal nodes (red in Fig 12 ()). This means that the skill tree learned in pretraining can
no longer be used, leading ineffecient finetuning.
To avoid this limitation, UPSIDE only finetuned the leaf skill in the original paper (i.e., freeze ancestors 1,4 and only finetunes 2 = 8 in Fig 12 b)). However, this would be still ineffective because the fixed ancestors 2 = 1, 4 are not the optimal solution to solve the given downstream task (green lines from <start> to z 4 nodes in Fig b)). The problem becomes more serious in long-horizon environment such as DMC (where horizon is 1000) if we freeze all the pretrained ancestors skills and finetune only the last leaf skill. In contrast, DISCO-DANCE selects a single skill to finetune which can be executed independently, so it does not suffer from the above problem.
H LIMITATIONS AND FUTURE DIRECTIONS
may cause cost (sample) inefficiency when measuring the density of the state dis- tribution in environment with high dimensional input, such as control from pixel observations. In this case, we could utilize the downscaling technique (e.g., cell representation in Go-Explore) to reduce computational costs.
For the guide skill selection process to work stably, the termination state should be reliably reached. Therefore, when the stochasticity the environment is too large, would not be easy to select guide skill only with the simple random walk process we proposed. We leave it as future work for alleviating such difficulties.
17

---

Under review as a conference paper at ICLR 2023
1 VISUALIZATION OF LEARNED SKILLS
diayn
disdain
Smm
aps
DISCO-DANCE (a) Empty maze
SHee aps
diayn disdain
DISCO-DANCE (b) maze
Figure 13: Qualitative visualization of the learned skills Emtpy maze and Y maze. Visual- ization multiple rollouts of learned skills by baseline models. For DISCO-DANCE, we visualize curriculum precedure during training. Bold red lines indicate the guide skill.
18

---

Under review as a conference paper at ICLR 2023
diayn
disdain
smm
aps
LI
DISCO-DANCE
(a) W maze
diayn
disdain
Smm
aps
U
DISCO-DANCE (b) maze
Figure 14: Qualitative visualization of the learned skills on W maze and S maze. Visualization of multiple rollouts of learned skills by baseline models. For DISCO-DANCE, we visualize our curriculum precedure during training. Bold red lines indicate the guide skill.
19

---

Under review as a conference paper at ICLR 2023
diayn
disdain
Smm
aps
DISCO-DANCE
(a) Ant Empty-maze
diayn
disdain
smm
aps
DISCO-DANCE
(b) Ant U-maze
diayn
disdain
aps
T
DISCO-DANCE
(c) Ant Il-aze
Figure 15: Qualitative visualization of the learned skills on Ant Empty-maze, Ant U-maze, and Ant II-maze. Visualization multiple rollouts of learned skills by baseline models. For DISCO- DANCE, we visualize curriculum precedure during training. Bold red lines indicate the guide skill.
20

---

Under review as a conference paper at ICLR 2023
Figure 16: Qualitative visualization of the learned skills in cheetah domain in DMC. DISCO- DANCE learned 100 skills without reward function. To effectively show how diverse skills DISCO- DANCE learned, we select 17 skills based on x coordinate of each skill's terminal state.
21


Reviews:
[{"review_id": "bcPZc1Fy1VS", "reviewer": "Reviewer_DqoP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper suggests an unsupervised skill discovery method named DISCO-DANCE. The authors focus on the issue that the commonly used MI reward ($I(S; Z)$) does not encourage exploration, and propose a novel approach to resolve this problem. Their method (DISCO-DANCE) first finds a \"guide\" skill that likely leads to unexplored states and then learns several \"apprentice\" skills that explore near the terminal state of the guide skill. They evaluate DISCO-DANCE on 2D maze and continuous control environments, showing that their proposed method outperforms previous skill discovery methods, such as DIAYN, SMM, APS, and DISDAIN.", "summary_of_the_review": "While the method tackles a relevant problem of skill discovery and shows improved performance, due to insufficient evaluations and comparisons, I am not able to recommend acceptance.\n\n(11/18 Update): While I appreciate the response as well as the effort to make comparisons with UPSIDE, I believe a fair comparison between DISCO-DANCE and its closest baseline (UPSIDE) is necessary for assessing this work. I acknowledge that it could be difficult to reproduce the baseline's result without its code, but I feel it is a bit premature to conclude that UPSIDE is not reproducible within this short period of the discussion phase. Therefore, I cannot recommend acceptance in its current form.", "strength_and_weaknesses": "### Strengths\n- The authors tackle an important problem of skill discovery with a sensible approach.\n- The paper is clearly written and the figures are helpful for understanding the concept of the method.\n\n### Weaknesses\n- There is a missing baseline, UPSIDE. Since UPSIDE uses a very similar strategy (they also have a \"directed\" part and \"random-walk\" part as in this work) and shows significantly improved coverage in the same 2D maze domain, I believe the authors should include comparisons with UPSIDE and report the results.\n- The proposed method seems to leverage a fairly strong assumption that the environment is resettable to any arbitrary state (for performing multiple random walks from the terminal state of each skill). Is there any efficient way to avoid using this assumption, and how well does the method perform without it? For example, just running each skill $R$ times and then doing a single random walk after each of the rollouts could be one naive way to resolve this, but this strategy may cause significant inefficiency in terms of the number of environment steps. Also, how do the authors take this (the number of samples required for random walks) into account when computing the number of total training steps (e.g., 5M steps for hard mazes)?\n- It seems that the method is tailored toward synthetic 2D maze environments. What kind of skills does DISCO-DANCE learn in the DMC HalfCheetah and Quadruped environments? Videos and figures of learned skills in such environments would be very helpful to further understand the method.\n- The result in Table 2 is only computed from three random seeds. Given its marginal performance improvement and the overlapping confidence intervals, it would be difficult to say DISCO-DANCE shows superior performance to the baselines in this environment.\n\n### Additional questions\n- Why is the performance of DISDAIN in Table 4 significantly worse than DIAYN in HalfCheetah? According to the DISDAIN paper, DISDAIN with $N = 1$ and $\\lambda = 0$ (in their notations) exactly recovers the standard DIAYN, so I believe its performance should be at least equal to DIAYN's.\n- How do the authors decide to add new skills versus reuse existing ones?\n- How do the authors fine-tune their agents (and other baselines) on the DMC benchmark?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "UlgMxE4GLea", "reviewer": "Reviewer_RjRU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper at hand proposes a new inductive bias (in the form of a proxy reward) for MI-based unsupervised skill discovery. The reward is designed so that exploration is improved, in particular in environments with bottleneck states or non-trivial dynamics. This is turn motivated by the observation that MI-based methods generally exhibit poor exploration capabilities as it is not strictly required to satisfy the main training criterion (the ability to discriminate skills by they states they visit).", "summary_of_the_review": "Overall, my main concern with the paper is the supposedly minor amount of novelty with respect to UPSIDE, which it is not considered as a baseline and the pros/cons of both approaches are not discussed. I would be happy to see the authors addressing this concern.", "strength_and_weaknesses": "Strengths:\n- The paper is well-written with a clear presentation and motivation\n- The exploration problem is quite pressing in unsupervised skill discovery (USD), and failure modes of previous works are nicely summarized in Figure 1.\n- Although the approach is primarily motivated in navigation environments, it also achieves comparably good scores on DMC-100k.\n\nWeaknesses:\n- UPSIDE would be a very relevant baseline here since its main motivation (improving exploration by extending skills) is very related. I'm not 100% sure whether the environments are matching, but what about adding the results from Table 2 in Kamienny et al. (2021) to your Table 1? UPSIDE achieves a coverage of 85.67 in the bottleneck maze, which (assuming the environments are the same), almost matches your result.\n- Another potentially USD baseline would be the training of a goal-based policy, e.g., with hindsight experience replay. This should be particularly effective in navigation environments, assuming the extents of the maze are known.\n- The goal skill selection seems to be mainly based on heuristics, assuming that (1) skill termination states can be reliably reached and (2) random walks. It would good to (briefly) discuss possible failure modes here, e.g., in large state spaces (I assume that in the Ant mazes, you consider X/Y only for the random walk end states?), for stochastic environments, or if dynamics differ more significantly across the state region as in some Atari games.\n- In a similar vein, the proposed algorithm introduces several hyper-parameters. Apart from an ablation of major components (which is useful), the sensitivity to, e.g., the accuracy threshold for which skills should be guided is not discussed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3wyC7ULmSt", "reviewer": "Reviewer_xuUS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Disco-Dance is a method for improving skill learning by expanding the diversity of the states that skills can reach.  The method first selects guide skills by finding skills most likely to lead to new states, then trains new skills (or old skills that have low discriminability) to reach novel states using mutual-information maximization.  ", "summary_of_the_review": "An interesting method for dealing with a difficult problem.  My concerns mostly involve similarity to previous work without comparison to those approaches.  If the differences can be clarified, both through explanation and experiment, I would be willing to raise my score.", "strength_and_weaknesses": "The paper addresses an important problem in skill learning, which is the diversity of states reached by current unsupervised rewards.  The Disco-Dance method is very well described, and experiments are clear as well.\n\nMy major concerns:\n\n- Similarity to previous methods is difficult to assess.  Disco-Dance seems very similar to both Go-Explore and Direct-then-Diffuse (UPSIDE) both of which are cited in the current paper.  However, despite the citations, there is little discussion of the differences between Disco-Dance and these prior approaches, and disappointingly there is no experimental comparison to those previous approaches.  This makes the impact of the current paper hard to assess.\n\n- In high-dimensional state spaces, like in the ant-maze experiments, how is the distance measured?  Does the approach require knowledge of which subspace represents the x-y plane?  If not, I'd imagine the number of skills required to get the coverage numbers reported in table 2 would be very large.  If so, does that not defeat the purpose of having high-dimensional state spaces, at least for the concerns of the current approach?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject