Learning to Discover Skills through Guidance
Authors: Hyunseung Kim
Byungkun Lee
Hojoon Lee
Dongyoon Hwang
Sejik Park
Kyushik Min
Jaegul Choo
Venue: NeurIPS 2023 poster (2023)
Keywords: Unsupervised skill discovery
Reinforcement Learning
TL;DR: We introduce an unsupervised skill learning algorithm, designed to provide direct guidance to encourage exploration.

Abstract: In the field of unsupervised skill discovery (USD), a major challenge is limited exploration, primarily due to substantial penalties when skills deviate from their initial trajectories. To enhance exploration, recent methodologies employ auxiliary rewards to maximize the epistemic uncertainty or entropy of states. However, we have identified that the effectiveness of these rewards declines as the environmental complexity rises. Therefore, we present a novel USD algorithm, skill **disco**very with gui**dance** (**DISCO-DANCE**), which (1) selects the guide skill that possesses the highest potential to reach unexplored states, (2) guides other skills to follow guide skill, then (3) the guided skills are dispersed to maximize their discriminability in unexplored states. Empirical evaluation demonstrates that DISCO-DANCE outperforms other USD baselines in challenging environments, including two navigation benchmarks and a continuous control benchmark. Qualitative visualizations and code of DISCO-DANCE are available at https://mynsng.github.io/discodance/.

Learning to Discover Skills through Guidance
Hyunseung Kim Byungkun Lee Hojoon Lee'
Dongyoon Hwang' Sejik Park' Kyushik Min Jaegul Choo'
Kim Jacchul Graduate School of AI, KAIST. 2KAKAO Corp.
[mynsng, byungkun, lee, joonleesky, godnpeter, sejik.pake jchoo))kaist... queue wit.eemmoiaaaooorpppp. com
Abstract
In the field of unsupervised skill discovery (USD), major challenge limited exploration, primarily due to substantial penalties when skills deviate from their initial trajectories. To enhance exploration, recent methodologies employ auxiliary rewards to maximize the epistemic uncertainty or entropy of states. However, we have identified that the effectiveness of these rewards declines as the envi- ronmental complexity rises. Therefore, we present novel USD algorithm, skill discovery with guidance (DISCO-DANCE), which (1) selects the guide skill that possesses the highest potential to reach unexplored states, (2) guides other skills to follow guide skill, then (3) the guided skills are dispersed maximize their discriminability in unexplored states. Empirical evaluation demonstrates that DISCO-DANCE outperforms other USD baselines challenging environ- ments, including two navigation benchmarks and continuous control bench- mark. Qualitative visualizations and code of DISCO-DANCE are available at httpp://myyun github. io/ /sddcodocaacc//////
Introduction
Deep Reinforcement Learning (DRL) has shown remarkable success wide range of complex
tasks, from playing video games 28 49 to complex robotic manipulation 4 [4]. However, the majority of DRL models are designed to train from scratch for each different task, resulting in significant inefficiencies. Furthermore, reward functions adopted for training the agents are generally handcrafted, acting as an impediment that prevents DRL from scaling for real-world tasks. For these reasons, there has been increasing interest in training task aask-agnostic policies without access to pre- defined reward function. One approach that has been widely studied for achieving this is unsupervised skill discovery (USD), which is a training paradigm that aims acquire diverse and discriminable behaviors, referred to as skills I 17 22 25 47 52 32 20, 46). These pre-trained skills can be utilized as useful primitives or directly employed solve various downstream tasks.
Most of the previous studies in USD discover set of diverse and discriminable skills by maximizing the self-supervised, intrinsic motivation form of reward [ 17 47]. Commonly, mutual information (MI) between the skill's latent variables and the states reached by each skill utilized as the self-supervised reward. However, it has been shown in recent research that solefy maximizing the sum of MI rewards insufficient in exploring the state space because, asymptotically, the agent receives larger rewards for visiting known states rather than for exploring novel states due to the high
MI reward it receives at fully discriminable state 7 25 52 20
To ameliorate this issue, recent studies designed an auxiliary exploration reward that incentivizes the However, albeit provided these auxiliary
agent when it succeeds in visiting novel states 52
Equual contributions.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(b) Uncertainty exploration
Skoll trajectories
reward
Exploration reward
Total reward
trajectories
(c)) Entropy based exploration
Force
MI reward
Exploration reward
Total reward
Skill trajectories
DISCO-DANCE
Reward
Select guide skill
rewarr
Guide
Total reward
Skill trajectories
Figure 1: Conceptual illustration of previous methods and DISCO-DANCE, Each skill is shown with grey-colored trajectory. Blue skill 2' indicates an unconverged skill, and the reward landscape of 2 is represented in green. Here, (b.c) illustrates reward landscape of previous methods, DISDAIN, APS, and SMM. (b) DISDAIN fails to reach the upper region due to the absence of a pathway to the unexplored states. (c) APS and SMM fail since they do not provide exact directions to the unexplored states. On the other hand, (d), DISCO-DANCE directly guides 2" towards selected guide skill 2* which has the highest potential reach the unexplored states. detailed explanation of the limitations of each baseline is described in Section 2.2
rewards, previous approaches may exhibit decreased effectiveness in challenging environments. Fig. conceptually illustrates the ineffectiveness of previous methods. Suppose that the upper region of Fig. a is difficult reach with MI rewards, resulting in obtaining skills that stuck the lower-left region. To make these skills explore the upper region, previous methods provide auxiliary exploration rewards using intrinsic motivation (e.g., disagreement, curiosity-based bonus). However, since they do not indicate exactly which direction to explore, it becomes more inefficient in challenging environments. We detail the limitations of previous approaches in Section 2.2
In response, we propose new exploration objective that aims to provide direct guidance to the unexplored states. To encourage skills explore unvisited states, first identify guide skill x which possesses the highest potential for reaching unexplored states (Fig. [T-)). Next, we select skills that relatively unconverged, and incentivize them to follow the guide skill =" in effort to leap over the state regions with low MI rewards (Fig. (Fig.Td-2:4] Finally, they are dispersed to maximize their distinguishability (Fig. [15), resulting in obtaining call this algorithm as skill discovery with guidance (DISCO-DANCE) and is further presented in Section DISCO-DANCE can be thought of filling the pathway the unexplored region with positive dense reward.
of skills with high state coverage. We
Through empirical experiments, demonstrate that DISCO-DANCE outperforms previous ap- proaches terms state space coverage and downstream task performances in two navigation environments (2D mazes and Ant mazes), which have been commonly used to validate the perfor- mance of the USD agent 7 19]. Furthermore, we also experiment in Deepmind Control Suite (3], and show that the learned set of skills from DISCO-DANCE provides better primitives for learning general behavior (e.g., run, jump, and flip) compared to previous baselines.
Preliminaries
we formalize USD and explain the inherent pessimism that arises in USD. Section 2.2 describes existing cxploration objectives for USD and the pitfalls of these exploration objectives.
In Section 2.1

---

2.1 Unsupervised Skill Discovery and Inherent Exploration Problem
Unsupervised Skill Discovery (USD) aims learn skills that can be further utilized as useful primitives or directly used to solve various downstream tasks. We cast the USD problem as discounted, finite horizon Markov decision process M with states s E S, action a E A, transition dynamics p E TT and discount factor y. Since USD trains the ag.nts to learn diverse skills in an unsupervised manner, we assume that the reward given from the environment is fixed to The skill is commonly formulated by introducing a skill's s latent variable = E 2 to policy resulting in a latent-conditioned policy x {a]s, 2). Here, the skill's latent variable 2 can be represented as one-hot vector (i.e., discrete skill) or continuous vector (i.e., continuous skill). In order to discover set of diverse and discriminable skills, a standard practice is to maximize the mutual information (MI) between state and skill's s latent variable (S; 2) I 13 17 47).
5( (Z + H(Z) =
(age)))))ee))))lp((pp - log p(z)
(1)
Since directly computing the posterior p(z[s) is intractable, Iearned parametric model (z)ss)) which we call discriminator, is introduced to derive lower-bound of the MI instead.
((2 > i(2,S) Be=-(=)))==(s)) [log (=(s)) log [o]](=
Then, the lower bound is maximized by optimizing the skill policy π [a]ss 2) via any RL algorithm with reward log (z(2)s) log p(=) (referred to raiil). Note that each skill-conditioned policy gets a different reward for visiting the same state (i.e., raill (zi, 8) o raan( (zj, 8)). It results in learning skills that visit different states, making them discriminable.
However, maximizing the MI objective is insufficient to fully explore the environment due to inherent pessimism of its objective 77 25 52). When the discriminator go succeeds in distinguishing the skills, the agent receives larger rewards for visiting known states rather than for exploring novel states. This lowers the state coverage of given environment, suggesting that there are limitations in the set of skills learned (i.e., achieving a set of skills that only reach a limited state space).
2.2 Previous exploration bonus and Its limitations
In order to overcome the inherenl pessimistic exploration of USD, recent studies have attempted to provide auxiliary rewards. DISDAIN [52) trains ensemble of N discriminators and rewards the >i=
agent for their disagreement, represented as H = 9s. (Z]s)) + Σ H(g,, (Z]s)). Since states that have not been visited frequently will have high disagreement among discriminators, DISDAIN implicitly encourages the agent to move to novel states. However, since such exploration bonus is a consumable resource that diminishes as training progresses, most skills will not benefit from this if other skills reach these new states first and exhaust the bonus reward.
We illustrate this problem in Fig. Lb. Suppose that all skills remain in the lower left states, which are easy to reach with MI rewards. Since the states in the lower left region are accumulated in the replay buffer, disagreement between discriminators remains low (e.g., low exploration reward in the lower left region). Therefore, there will be no exploration reward left in these states. This impedes 2; from escaping its current state to unexplored states, as shown in Fig. D
On the other hand, SMM encourages the agent to visit states where it has not been before using learned density model de (24). APS incentivizes the agent to maximize the marginal state entropy via maximizing the distance of the encoded states fo(st) between its k nearest neighbor f/ (s:) 25
.MMM exploration
X log (s)))
APS
"xxploration x log [fo(s) - f (s)
These rewards push each skill out of its converged states (in Fig. (p. However, they still do not provide a specific direction on where to move in order to reach unexplored states. Therefore, in difficult environment with a larger state space, it known that these exploration rewards can operate inefficiently 7 [9. In the next section, we introduce new exploration objective for USD which addresses these limitations and outperforms prior methods on challenging environments.
3 Method
Unlike previous approaches, we design a new exploration objective where the guide skill z" directly influences other skills to reach explored regions. DISCO-DANCE consists of two stages: (i) selecting
3

---

guide skill 2' and the apprentice skills which will be guided by 2", and (ii) providing guidance to apprentice skills via guide reward. which will be described in Section and 3.2 respectively.
3.1 Selecting Guide skill and Apprentice Skills
Guide skill. We recall that our main objective is to obtain set of skills that provides high state space coverage. To make other skills reach the unexplored state, we define the guide skill 2* the skill which is most adjacent the unex- plored states. One naive approach in selecting the guide skill is choose the skill whose ter- minal state is most distant from the other skills' terminal state (e.g., blue skill in Fig. (D). How- ever, such selection process does not take into account whether the guide skill's terminal state is neighboring promising unexplored states, In order approximate whether skill's ' terminal
rajestores
(b) most
Sebeett shll with randdom walk proceoo
Figure 2: Guide skill selection. (a) A set of skill trajectories. (b) Naively selecting the skill whose terminal state is most distant from others. (c) Se- lecting the skill through the random walk process.
state is near potentially unexplored states, we utilize a simple random walk process (Fig. 2 To elaborate, given a set of P skills, (i) rollout skill trajectory (I' timesteps) and perform R random walks from the terminal state of each skill, repeated M times (i.c., a total of P(T + R)M steps, collecting PRM random walk arrival states as Fig. [2). Then (ii) we pinpoint the state in the lowest density region among collected random walk arrival states and select the skill which that state originated from as the guide skill 2 (red skill Fig. ( For (ii), one could use any algorithm to measure the density of the random walk state distribution. For our experiments, we utilize simple k-nearest neighbor algorithm,
2" := argmax max PE(L....PP) rE(1.....RA RMY k S).-EN((epr)
Σ
||spr
spr ll2
(4)
where Spr = r-th random walk arrival state of skill Na(-) k-nearest neighbors.
In practice, in environment with long horizon (i.e., high 7), such DMC, random walk process may cause sample inefficiencies. Therefore, we also present an alternative, efficient random walk process, which is thoroughly detailed in Appendix B Ablation experiment
guide skill
selection methods (Fig. 2h.b) is provided in Section 4.4
Apprentice skills. We select apprentice skills as the skills with low discriminability (i.e., skill 2² with low 4o (=) which fails to converge) and move them towards the guide skill. most of the skills are already well discriminable (i.c., high MI rewards), apprentice skills. This will leave converged skills intact and send new skills to unexplored states. Since the total number of skills is gradually increasing the new skills are added, this would bring the side benefit of not relying on a pre-defined number of total skills as a hyperparameter.
simply add new skills and make them
In Appendix E, we empirically show that adding new skills during training is generally difficult to apply to previous algorithms because the new skills will face pessimistic exploration problems. The new skills simply converge by overlapping with existing skills (e.g., left lower room in 3.. which exacerbates the situation (i.e., reducing discriminator accuracy without increasing state coverage).
3.2 Providing direct guidance via auxiliary reward
After selecting the guide skill 2 and apprentice skills, now formalize the exploration objective, considering two different aspects: (i) the objective of the guidance, and (ii) the degree of the guidance. It is crucial to account for these desiderata since strong guidance will lead apprentice skills to simply imitate guide skill 2", whereas weak guidance will not be enough for skills to overcome pessimistic exploration problem.
In response, we propose an exploration objective that enables our agent to learn with guidance. We integrate these considerations into single soft constraint

---

maximize (=) (ram + rguide]
where risklll - log 2o logp(=)),
(5)
Igguid -a (2 8) 1 (1 (o())ls)) ( (a
As describe Section 3.1 we select skills with low discriminator accuracy (i.e., (2')s) c)) as apprentice skills. For (i), we minimize the KL-divergence between the apprentice skill and the guide skill policy (i.e., Dxi (T [(] (a))s==(()) For (ii), the extent to which the apprentice skills are encouraged to follow z", can be represented as the weight of the -l.-divereence. We set the weight as - qs (=")s) [s) to make the skills with low discriminator accuracy guided more.
existing skills
guide skill
apprentice skills
Skill tra ectories
Select guide skill with randam walk process
Sell trajectories
Skill trajectories
Select guide skill with randam walk process
Skil trajectorios
(a) bottlenack maze
(b) cheetah
Figure 3: Overall procedure of DISCO-DANCE, in (a) navigation and (b) continuous control tasks.
Fig. shows that the guidance strategy significantly enhances the learning of skills in both navigation and continuous control environments. In the navigation task (Fi 33). the skill located in the upper left corner is selected as a guide through the random walk and utilized navigate the apprentice skills, which were previously stuck in the lower left corner, into unexplored states. Similarly, in continuous control environment (Fig. BD), the skill that has been learned run is selected as the guide, leading the apprentice skills that were barely moving. This guidance allows the apprentice skills to quickly learn run. This salient observation suggests that the concept of guidance can be utilized, even for non-navigation tasks. In Section 4.4 we show that all of these components of Tguide are necessary through additional experiments. We provide pscudocode for DISCO-DANCE in Algorithm!]
Algorithm 1: Skill Discovery through Guidance
Initialize skills 21 2n RL policy [a 2). skill discriminator go (z(ss.
2 Initialize guide skill x- = None, Tguide = 0
Hyperparameters guide coef a, apprentice cutoff e
for k = 1, do
Sample batch (st, at, $1+1, z²) from replay buffer D
if most skills discriminable enough then
L guide skill =x + finddgguddl paiddddsiiillllT____ ....,,,²")
rsam = log uo (=((111444) log ((=)(==)
if 2" is not None then
10
Iguide (1(o0(211s)
D (To (1=')())
TT)
11
= Tsai
guuid
12
Update ne to maximize sum of
Update 4s to maximize log 4s (=|s;)
4 Experiments
4.1 Experimental Setup
Evaluation. We evaluate DISCO-DANCE across three distinet types of environments commonly used in USD: 2D Maze, Ant Maze, and Deepmind Control Suite (DMC). We utilize state space

---

Empty
W
Bottleneck
Ant Eirply
Ant (b) Ant Maze
Ant
Cheatah
Quadruped
(a) 2D Maze
(c) DMC
Figure 4: Three environments to evaluate skill discovery algorithms. (a) Continuous 2D mazes with various layouts, (b) high dimensional ant navigation tasks, and (c) continuous control environ- ments with diverse downstream tasks.
coverage and downstream task performance our primary evaluation metrics. To measure state coverage, we discretize the x and y axes the environment into 10 intervals (i.e., total 10x 10 buckets) and count the number of buckets reached by learned skills (Fig. a..). For downstream task performance, we finetune the agent pretrained with baselines and DISCO-DANCE (Fig. 4 (b,c)). Further details regarding the environments and evaluation metries can be found Appendix D
For Ant mazes, we provide the (x.)) coordinates as the input for the discriminator for all algorithms following previous work 47), since the main objective of navigation environments is learn set of skills that are capable of successfully navigating throughout the given environment. For DMC, utilize all observable dimensions (e.g., joint angles, velocities) the input of the discriminator to learn more general behaviors (c.g., running, jumping, and flipping) that can be used as useful primitives for unknown continuous control tasks.
Baselines. We compare DISCO-DANCE with various skill-based algorithms, focusing on addressing the pessimistic exploration of USD agents. We compare DISCO-DANCE with DIAYN [II, which trains an agent using only Y'ail without any exploration rewards. We also evaluate the performance of
SMM [24. APS (25). and DISDAIN [52], which incorporate auxiliary exploration rewards. We note that all basclines and DISCO-DANCE utilize discrete skills, except APS, which utilizes continuous skills. For downstream tasks, also compare USD baselines with SCRATCH, which represents learning from scratch (i.c., no pretraining). Additional details are available in Appendix [
In addition, we include UPSIDE 19 as a baseline. UPSIDE learns tree-structured policy composed of multiple skill nodes. Although UPSIDE does not utilize auxiliary reward, DISCO-DANCE and UPSIDE both aim to enhance exploration by leveraging previously discovered skills. However, UPSIDE's sequential execution of skills from ancestors children nodes top-down manner leads to significant inefficiency during finetuning, leading to lower performance on downstream tasks. Furthermore, UPSIDE selects the skill with highest discriminator accuracy (i.e., corresponding to Fig. 2)) for expanding tree policies, resulting in reduced state coverage the pretraining stage. A detailed comparison between DISCO-DANCE and UPSIDE can found Appendix
4.2
Navigation Environments
4.2.1
2D Mazes
First, we evaluate DISCO-DANCE in 2D mazes (Fig. 4h), which has been commonly used for testing the exploration ability of skill learning agents 7 19). The layout becomes more challenging, from an empty maze to bottleneck maze. The agent determines where to move given the current and coordinates (2-dimensional state and action space).
Table 1: State space coverages of DISCO-DANCE and baselines two navigation benchmarks. The results are averaged over random seeds accompanied by a standard deviation. Scores bold indicate the best perforerfon model and underlined scores indicate the second-best.
Models
Empty
Y
W
Bottlencck
Ant Empty-maze
Ant U-maze
Ant I[-maze
DIAYN
100.00_0.00
69.80
5.39
71.50
15.10
52.50015.13
52.40.13.77
72.70..10.95
16.8018.41
22.50
SMM
100.00-0.00
89.20
=5.00
73.40
=6.15
55. 55.00%s5.59
54,50 $4.80
99.50=80.4
58. 90.8.233
25. 35.40:4497
APS
97.90+3375
90:00
-5.51
81.50
+11.31
20-20-+5.88
61.90+ 10
33.10-24.88
6.860--8885
26.3-7.68
DISDAIN
100.00_0.00
87.20 6681
85.00
18.69
61.3001..44
61.70.14.85
70.1014.97
80.44.8
22.90
UPSIDE
99.00a:9.25
85.20 19.39
90.20
16.15
69.90
78.90+13.09
74.50.6.22
63.60=11.55
29 39.90-2242
88.50)=66.45
98.90=1.85
68.30+44.47

---

Table Ta) summarizes the performance of skill learning algorithms on 2D mazes. Our empirical analysis indicates that baseline methods with auxiliary reward (i.e., SMM, APS, and DISDAIN) exhibit superior performance compared to DIAYN. However, we find out that previous studies provide less benefit on performance as the layout becomes more complex, as we mentioned in section 2.2] In contrast, the performance of DISCO-DANCE remains relatively stable regardless of layout complexity. Additionally, DISCO-DANCE outperforms UPSIDE in 2D mazes, which we attribute to the inefficiency of adding new skills to leaf nodes with high discriminator accuracy (Fig.
(a) DIAYN
(b) DISDAIN
(c) SMM
(d) APS
(e) UPSIDE
(f) DISCO DANCE
Figure 5: Visualization of the skills in bottleneck maze. Multiple rollouts by each algorithm.
Fig. 5 illustrates multiple rollouts of various skills learned in the 2D bottleneck maze. For the 2D bottleneck maze, the upper left room the most difficult region to reach since the agents need to pass multiple narrow pathways. Fig. shows that only DISCO-DANCE and UPSIDE effectively explore the upper left region, with DISCO-DANCE displaying a relatively denser coverage on the upper left state space. More qualitative results available in Appendix
4.2.2 Ant mazes
To evaluate the effectiveness of DISCO-DANCE in the enviromment with high dimensional state and action space, we train DISCO-DANCE in three Ant mazes where the state space consists of joint angles, velocities, and the center of mass, and the action space consists of torques of each joint X 30
Table [b) reports the state coverage of DISCO-DANCE and baselines. DISCO-DANCE shows superior performance to the baselines where it achieves the best state-coverage performance in high- dimensional environments that contain obstacles (i.e., Ant U-maze and 11-maze) and gets competitive results against SMM with a marginal performance gap in the environment without any obstacle (i.e., Ant Empty-maze). These results demonstrate the effectiveness of our proposed direct guidance in navigating an environment with obstacles. Additional qualitative results of each algorithm can be found in the Appendix[]
To further assess whether the learned skills could be a good starting point for downstream tasks, we conduct goal- reaching navigation experiments where the goal state is
UPSID
DEDANN
SONTCII
to the region farthest from the initial state for the more challenging navigation environment, Ant 11-maze. We
of Successful Seeds
20-
20-
26
measure the number of seeds that successfully reach the
3.0
established goal state (out of a total 20 seeds). We select
the skill with the maximum return (i.e., skill whose state space is closest to the goal state) from the set of pretrained skills and finetune the policy (a selected skill 2). Ad- ditional experiments in Ant U-maze and details are avail- able in Appendix D
8.
0.5
0.8
0.9
Environment Steps (x10f)
Figure Finetuning results on the Ant II-maze. We plot the number of success- ful seeds of total 20 seeds.
Fig. shows that DISCO-DANCE outperforms prior base- line methods terms of sample efficiency. Specifically, DISCO-DANCE is the only algorithm that successfully reaches the goal state for all seeds (20 seeds) in Ant 11-
maze. In addition, UPSIDE fails to reach the goal for all seeds, which indicates that the tree-structured policies learned during the pretraining stage can impede the agent from learning optimal policies for fine-tuning.

---

MEDIAN
MEAN
Optimality Gap
SCRATCH
DIAYN
SMM
APS
DISDAIN
UPSIDE
DISCO-DANCE
0.45 0.60 0.75
0.60
0.75
0.90
0.6
0.7
0.8
0.2
0.3
0.4
0.5
Expert Normalized Score
Figure 7: Performance comparisons on Deepmind Control Suite. performance profiles with 95% bootstrap confidence interval are provided across 120 seeds (15 seeds x 8 tasks).
Aggregate statistics and which are calculated
2
4.3 Deepmind Control Suite
While the experiments in 2D and Ant mazes demonstrate that the skills DISCO-DANCE learns can be utilized as useful primitives for navigation tasks, to demonstrate the versatility and effectiveness of DISCO-DANCE in acquiring general skills (e.g.. run, jump, flip), we conduct additional experiments on the DMC benchmark [53]. Specifically, we consider two different environments in DMC, Cheetah, and Quadruped, and evaluate DISCO-DANCE across four tasks (e.g., run, flip, and jump) within each environment (total tasks). The DMC benchmark evaluates the diversity the learned skills, meaning that agents must learn suitable set of skills for all downstream tasks in order to achieve consistently high scores across all tasks. We first perform pretraining for 2M training steps and select the skill with the highest return for each task (same the Ant maze experiment). Then finetune USD algorithms for 100k steps. More experimental details are available at Appendix D
Fig. 7 shows the performance comparisons DMC across tasks with 15 seeds. First, all USD methods perform better than training from scratch (i.e., SCRATCH), indicating that pretraining with USD is also beneficial for continuous control tasks. Furthermore, we described in Fig-Br we find that our proposed guidance method boosts the skill learning process for the continuous control environment, resulting in DISCO-DANCE outperforming all other baselines. As shown in Ant mazes finetuning experiments, UPSIDE significantly underperforms compared to other auxiliary reward-based USD baselines. Since UPSIDE necessitates the sequential execution of all its ancestor skills, its finetuning process becomes less efficient than other methods.
In addition, baselines with auxiliary rewards, which enforce exploration, are ineffective in DMC. We attribute these results to their training inefficiency, that the current experimental protocol of pretraining for 2M frames is insufficient for them to converge in a meaningful manner. For example, DISDAIN requires billions of environment steps the grid world environment in the original paper. In contrast, DISCO-DANCE successfully converges within 2M frames and outperforms all other baselines. We provide the full performance table and visualizations of skills in Appendix [
4.4 Ablation Studies
4.4.1
Model Components
We further conduct an ablation analysis to show that each of model components is critical for achieving strong performance. Specifically, we experiment with two variants of DISCÓ- DANCE, each of which is trained (i) without guide coefficient (i.e., guide coefficient as 1) and (ii) without the random walk based guide selection process in Table[2]
Table 2: Ablation study on model components.
Models
Bottleneck maze
DISCO-DANCE
86.30=17.01
w/o guide cocf
76.75::
DISCO-DANCE w/o random walk
70.30= 10,72
DIAYN (no exploration reward)
52,40:13.77
First, without guide coefficient (x")s= < (1 (1-a((((1))))) DISCO-DANCE sets all other skills as apprentice skills, causing all other skills to move towards the guide skill. As shown in Table 2 this method of guiding too many skills degrades performance, and it is important to only select unconverged skills apprentice skills for effective learning.

---

Second, Table 2 also shows that random-walk based guide skill selection (I.e., selecting the skill with the lowest den- sity region among PRM terminal states in Eq. 4 is superior to naively selecting the most distant skill without random walk process (i.e., selecting the skill with the lowest den- sity region among terminal states in Eq. ( We also provide qualitative results (Fig. () where selecting the most distant skill as the guide skillfails fails to select the skill closest to the unexplored state region since the blue skill the most distant from the terminal states of the other skills. On the other hand, random walk guide selection successfully finds the appropriate guide skill.
Randam
Most distant
Figure Qualitative results for differ- ent guide skill selection processes.
4.4.2 Compatibility with other skill discovery method
In this paper, our primary goal is to compare DISCO-DANCE with algorithms that focus on constructing an effective auxiliary exploration reward, denoted as l'epploaaionn However, several research studies exist that seek to refine raain enhance explo-
Table Ablation study on "skill.
Taiil
Texploration
Ant II-maze
22.50 ::3.34
ration R2 20 27 (detailed explanation in Section 5) To eval- uate how effectively DISCO-DANCE can enhance exploration when aligned with different skkill we carried out comparative analysis with LSD [32]. The recently introduced skill discovery algorithm, LSD, has demonstrated commendable outcomes, especially in Ant Mazes.
DIAYN
I'uide
39.00 114.85
38.80 #3334
LSD
"guide
45.80 +33.4
Table 3 iillustrates the performance comparison with LSD. Our findings reveal that LSD, even in the absence of Fexplocations exhibits superior performance compared to DIAYN in Ant-II maze. Such an observation possibly highlights LSD's proficiency in mitigating the inherent pessimism linked with the previous mutual information objective. An observation worth highlighting is the substantial enhancement in performance for both algorithms when integrated with our T'uide- We propose that raill and guudde have distinct roles, and when combined, they can collectively boost their overall performance.
Related Work
Pretraining methods RL Pretraining methods for RL primarily fall into two categories (55). The first is online pretraining, where agent interact with the environment in the absence of any reward signals. This method generally operates under the presumption that agents can engage with the environment at minimal cost. The primary objective of online pretraining is to acquire essential prior knowledge through unsupervised interactions with the cuvironment. Recently, there has been a lot of research in this area, often referred to "Unsupervised RL". Such methodologies typically employ intrinsic rewards, to learn representations or skills which will be useful for downstream tasks. The acquired pretrained weights, such as encoder, actor, and critic, subsequently reused for downstream tasks.
Depending how the intrinsic reward is modeled, online pretraining can be further categorized. Curiosity-driven approaches focus on probing states of interest, often characterized by high predictive emror, in order to obtain more environmental knowledge and reduce prediction errors across states 6 35 43 34]. Data coverage maximization methods aim to maximizes the diversity of the data the policy has collected, for instance, by amplifying the entropy of state visitation distributions 44 Meanwhile, Unsupervised Skill Discovery involves learning policies, denoted as (a s, guided by distinct skill vector This vector then be adjusted or combined to address subsequent tasks
18 26 57
60 39 33
The second category is offline pretraining. This method leverages an unlabeled offline dataset accumulated through various policies, to learn the representation. The learning objective in offline pretraining involves pixel or latent space reconstruction
[58]
59
61
54 future state prediction A1
42 45 23), learning useful skills from data 37
50
36
16
56
and contrastive learning focusing
on either instance [21 [2] temporal discrimination
31
51

---

In this paper, our primary emphasis is on Unsupervised Skill Discovery within the domain of online pretraining, For a comprehensive overview of pretraining methods for RL, we direct the reader to 22 55.
Unsupervised skill discovery. There have been numerous attempts learn of useful skills for RL agents in a fully unsupervised manner. These approaches can be broadly classified into two categories: (i) one studies how learn the skill representations in an unsupervised manner (i.e., how to make Takill in Eq. 5) and (ii) the other focuses on devising an effective auxiliary reward which encourages the exploration of skills address the inherent pessimistic exploration problem (i.e., how to make T Texploration in Eq. 5
Research that falls under the first category focuses on how to train skills that will be useful for unseen tasks. DIAYN [II,, VISR (1), EDI. (7) and DISK [46] maximizes the mutual information (MI) between skill and state reached using that skills. DADS (47 utilizes model-based RL approach to maximize conditional MI between skill and state given the previous state. LSD [32] which is
non-MI-based algorithm, provides I-Lipschitz constraint on the representation of skills learn dynamic skills. CIC [20] replace Takill with a contrastive loss to learn high-dimensional latent skill representations, and additionally utilizes APP
'exploration (Eq. 3 for exploration. Choreographer (27 learns
world model to produce imaginary trajectories (which are used for learning skill representations) and additionally utilizes APS esploraaton
for exploration.
DISCO-DANCE belongs to the second category that focuses on devising exploration strategy for USD. This category includes DISDAIN [52]. UPSIDE [19], APS [5 and SMM 244 which are included as our baseline methods. Detailed explanation and limitations of these methods are described
in Section 2.2 4.
Fig.
and Appendix
Guidance based exploration in RL. Go-Explore O 10 stores visited states in buffer and starts re-exploration by sampling state trajectory, which is the most novel (i.e., the least visited) in the buffer. DISCO-DANCE and Go-Explore share a similar motivation that the agent guides itself: DISCO- DANCE learns from other skills and Go-Explore learns from previous trajectories. Another line of guided exploration is utilize a KL KL-regularization between the policy and the demonstration or
SPIRL [37], an offline skill discovery method, minimizes the KL. divergence between the skill policy and the learned prior from offline data. RIS K has been
sub-policy (i.e., guide) 37
38
29
48
proposed for efficient exploration in goal-conditions RL, by minimizing KL divergence between its policy and generated subgoals.
6 Conclusion
We introduce DISCO-DANCE, a novel, efficient exploration strategy for USD. It directly guides the skills towards the unexplored states by forcing them to follow the guide skill. We provide quantitative and qualitative experimental results that demonstrate that DISCO-DANCE outperforms existing methods in two navigation and a continuous control benchmark.
In this paper, in order to evaluate solely the effectiveness of Texploration methods, we fix the backbone T Tskill as DIAYN. Combining DISCO-DE -DNCE with other recent Tskil methods (e.g.. CIC) is left as an interesting future work. Also, we think that there is still enough room for improvement in finetuning strategies (22). Since USD learns many different aask-agnostic behaviors, a new fine-tuning strategy that can take advantage of these points would make the downstream task performances more powerful. discussion of limitations is available in Appendix G
Acknowledgments and Disclosure of Funding
This work was supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) NNo.2019-0-00073 Artificial Intelligence Graduate School Program(KAIST), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2022RI A A2B5B02001913 and the National Supercomputing Center with supercomputing resources including technical support (KSC-2023- CRE-0074).
10

---

References
[1] Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 2018.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. Proc. the Advances in Neural Information Processing Systems (NeurIPS),
2021.
[3] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Of- fline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. 13611, 2020.
[4] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. NIPS, 30, 2017.
[5] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[6] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. Proc. the International Conference on Learning Representations (ICLR), 2018.
[7] Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills, In International Conference Machine Learning, pages 1317-1327. PMLR, 2020.
[8] Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning, pages 1430-1440.
PMLR, 2021.
[9] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arxivv/9011 10995, 2019.
[10] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590078477:580-558, 2021.
[11] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. Proc. the International Conference Learning Representations (ICLR), 2018.
[12] Jiameng Fan and Wenchao Li. Dribo: Robust deep reinforcement learning via multi-view information bottleneck. In Proc. the International Conference Machine Learning (ICML), pages 6074-6102. PMLR, 2022.
[13] Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv: 161 1.07507, 2016.
[14] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3389-3396. IEFE, 2017.
[15] Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proc. the International Conference on Machine Learning (ICML), pages 1861-1870, 2018
[16] Kourosh Hakhamaneshi, Ruihan Zhao, Albert Zhan, Pieter Abbeel, and Michael Laskin. Hi- erarchical few-shot imitation with skill transition models. arXiv preprint arXiv:2107.08988 079818 2021.
[17] Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. Proc. the International Conference on Learning Representations (ICLR), 2019.
[18] Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. International Conference on Machine Learning, pages 2691-2691. PMLR, 2019.
11

---

[19] Pierre-Alexande Kamienny, Jean Tarbouriech, Alessandro Lazaric, and Ludovic Denoyer. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. Proc. the International Conference on Learning Representations (ICLR), 2021.
[20] Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:220..00161 2022.
[21] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In Proc. the International Conference on Machine Learning (ICML), 2020.
[22] Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang. Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. Proc. the Advances Neural Information Processing Systems (NeurIPS), 2021.
[23] Hojoon Lee, Koanho Lee, Dongyoon Hwang. Hyunho Lee, Byungkun Lee, and Jaegul Choo. On the importance of feature decorrelation for unsupervised representation learning in reinforcement learning. arXiv preprint arXiv:2306.06337, 2023.
[24] Lisa Lec, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Rus- lan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint 196.v:1900.005244, 2019.
[25] Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In Proc. the International Conference on Machine Learning (ICML), 2021.
[26] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2021.
[27] Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, and Sai Rajeswar. Chore- ographer: Learning and adapting skills in imagination. Proc. the International Conference on Learning Representations (ICLR). 2023.
(28) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540):529-533 2015.
[29] Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, and Joseph J Lim. Skill-based meta-reinforcemeen learning. Conference on Robot Learning (CoRL), 2022.
[30] Soroush Nasiriany, Vitchyr H. Pong, Steven Lin, and Sergey Levine. Planning with goal- conditioned policies. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2019.
[31] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXiv:1807.03748. 2018
[32] Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. Proc. the International Conference on Learning Representations (ICLR), 2022.
[33] Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllabilty--wware unsuper- vised skill discovery. arXiv preprint arXiv:2302.05 2023.
[34] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning, pages 2778- 2787. PMLR, 2017
[35] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagree- ment. In International conference machine learning, pages 5062-5071. PMLR, 2019.
[36] Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pages 188-204. PMLR, 2021.
[37] Karl Pertsch, Youngwoon Lee, and Joseph J Lim. Accelcrating reinforcement learning with learned skill priors. CORL. 2020.
[38] Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph Lim. Demonstration-gin serintdrratiogggideeddded guided reinforcement learning with leamned skills. Conference Robot Learning (CoRL), 2021.
12

---

[39] Sai Rajeswar, Pietro Mazzaglia, Tim Verbelen, Alexandre Piché. Bart Dhoedt, Aaron Courville, and Alexandre Lacoste. Unsupervised model-based pre-training for data-efficient control from pixels. arXiv preprint arXiv:2209. 12016, 2022.
[40] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. Proc. the International Conference on Learning Representations (ICLR), 2016.
[41] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self- pref-prreie representations. In Proc. the International Conference on Learning Representations (ICLR), 2020.
[42] Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R Devon Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 2021.
[43] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pages 8583-8592. PMLR, 2020.
[44] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State en- tropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, pages 9443-9454. PMLR, 2021.
[45] Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In Proc. the International Conference on Machine Learning (ICML), pages 19561-19579, PMLR, 2022.
[46] Nur Muhammad Shafiullah and Lerrel Pinto. One after another: Learning incremental skills for changing world. Proc, the International Conference on Learning Representations (ICLR), 2022.
[47] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. Proc. the International Conference on Learning Representations (ICLR), 2019.
[48] Lucy Xiaoyang Shi, Joseph J Lim, and Youngwoon Lee. Skill- model-based reinforcement learning. Conference on Robot Learning (CoRL), 2022.
[49] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[50] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arX1v:2011.10024, 2020.
[51] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Proc. the International Conference on Machine Learning (ICML), 2021.
[52] DJ Strouse, Kate Baumli, David Warde-Farley, Vlad Mnih, and Steven Hansen. Learning more skills through optimistic exploration. Proc. the International Conference on Learning Representations (ICLR), 2022.
[53] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 2020.
[54] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.0611
2022.
[55] Zhihui Xie, Zichuan Lin, Junyou Li, Shuai Li, and Deheng Ye. Pretraining in deep reinforcement learning: A survey, arXiv preprint wwXiv:2211.03955 2022.
[56] Mengjiao Yang, Sergey Levine, and Ofir Nachum. Trail: Near-optimal imitation learning with suboptimal data. arXiv preprint 1rX1v:2110.14770, 2021.
[57] Denis Yarats, Rob Fergus, Alessandro Lazaric and Lerrel Pinto. Reinforcement learning
with prototypical representations. In International Conference on Machine Learning, pages 11930 PMLR, 2021.
13

---

[58] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proc. the AAAI Conference on Artificial Intelligence (AAAI), volume 35, pages 10674-10681, 2021.
[59] Tao Yu, Zhizheng Zhang, Cuiling Lan, Yan Lu, and Zhibo Chen. Mask-based latent reconstruc-- tion for reinforcement learning. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 35:25117-25131, 2022.
[60] Jesse Zhang, Haonan Yu. and Wei Xu. Hicrarchical reinforcement learning by discovering intrinsic options. arXiv preprint arXiv:2101.0652/ 2021.
[61] Jinhua Zhu, Yingce Xia, Lijun Wu, Jiajun Deng, Wengang Zhou, Tao Qin, Yan Liu, and Houqiang Li. Masked contrastive representation learning for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3421-344 --3333,, 2022.
14

---

Appendix
Implementation Details of DISCO-DANCE
15
Efficient Random Walk Process
16
Random Walk Process in High-dimensional State Space
17
D Experimental Details
18
D.I Environments
18
D.2 Evaluation
18
D.3 Hyperparameters
19
D.4
Resource Requirements
20
D.5 License
20
Additional Experimental Results
20
E.1
Additional fine-ture fine-tuning results the Ant U-maze
20
Full Results on Deepmind Control Suite Tasks
Additional Ablation studies
21
E.4
4
Comparing curriculum approach effects other USD methods
22
Comparison with UPSIDE
22
Limitations and Future Directions
24
Broader Impact
24
Visualization of learned skills
25
2D mazes
25
1.2 Ant Mazes
27
I.3 Deepmind Control Suite
28
A
Implementation Details of DISCO-DANCE
For reproducibility, we (i) provide pseudo code in Algorithm I and (ii) open-source code at https://mynsng, github...... hodiiscodanecce/////////cooo// Below, We describe further details regarding the implementation of DISCO-DANCE.
(line in Algorithm[] 1 When extending new skills, we initialize their policy weights with the guide skill policy weights rather than random weights as SPIRL [37]. This accelerates exploration by encouraging the newly added skills directly reach the unexplored regions. Appendix E.3 we conduct ablation studies of this component.
(line Algorithm I As the number of skills gradually increasing, the discriminator struggles to learn due to the sparse update for each skill by the decreased probability of sampling each skill. We mitigate this issue by utilizing a skill sampling scheme where each skill sampling weight is proportionate to its discriminator enror (e.g., skills with high discriminator error are more likely to be sampled). This approach is analogous to the methodology adopted in Prioritized Experience Replay [40].
15

---

(line 5-6 in Algorithm| 1 For adding new skills, we need to define hyperparameters: (1) when extend skills and (2) how many to increase. We gradually extend skills as the discriminator converges. In detail, count the number of skills that have low discriminator accuracy (i.e., less than <), and if the number less than the threshold (i.e., P), extend p skills.
(line 8-9 in Algorithm D We utilize KL divergence between the apprentice skills and the guide skill policy as rguide- However, as the guide skill policy changes during training, can lead to instability in the training of the apprentice skills. To address this issue, use a separate target policy network for the guide skill. More precisely, when we select the guide skill, we copy and fix the policy network (denoted as target policy network f) and calculate the KL divergence reward utilizing this target policy network (i.e., calculating (To 8t, = (T (aelsss (1, )))).
Iguiiee as -I -[(a((= 8) < <1
(line 13-14 in Algorithm T Entropy term in SAC can be seen as KL divergence between the policy and uniform distribution with constant (37 (H(TT(-)s,, =)) x (1,
Instead of replacing the entropy term with guide term in DISCO-DANCE, we found that using both term simultaneously helped stabilize learn- ing by enjoying the advantage of maximum entropy RL (i.e., RL agent maximizes Fx 1E 7 (raam +
guide 3 H (T(()))))))
B Efficient Random Walk Process
Our random walk process in Section 3 is as follows:
1. After performing rollout for all P learned skills till the terminal state, we perform R random walks and collect P R number of random arrival states (R a hyper-parametts and we set I as 0.2 time horizon T for all environments). Repeat this process total of M times.
2. Pinpoint the state in the lowest density region among the P * R M number of states. We measure the density of the states using the k nearest neighbors.
3. Select the skill which that state originated from.
The number of environment steps in the random walk process is P * (7 + 0.27) * M. Therefore, the total number of environment steps performed during the random walk process in pretraining stage is,
(Niittial + (Ninitial + 5) + (Ninitial + 2 5) +.. + (Nimal - 8)) (T 0.2T) M
where Nuniial is the number of initial skills, is the number of skills to extend when most of the existing skills are converged (e.g., number of skills are 10 - 15 - 20 .... + 55 when 110 5=5, N5m==55), Note that Nimal is the number of skills when the pretraining is ended (not predefined as a hyperparameter).
However, in long-horizon environment such as DMC (1000 timesteps), our random walk process can cause non-negligible sample inefficiencies. Thus, further present an efficient random walk process that approximates an original random walk process with a simple approach. The detailed process is as follows:
1. During pretraining, if the skill satisties (2(15)))) < in the terminal state sT (i.e., which is not an apprentice skill), additionally perform R random walk steps and store the random arrival states at the temporary buffer (queue).
2. When selecting guide skill, instead of rolling out all skills, select the state in the lowest density region among the states at the temporary buffer.
3. Select the skill which that state originated from.
Since there are no additional environment steps when selecting the guide skill, the efficient random walk process can significantly reduce the number environment steps compared to the original version (added environment steps are 0.2T per trajectory (only if its skill satisfies the accuracy threshold), whereas original random walk process requires +T+ 0.2T) x M). Our experimental results on DMC demonstrate that the efficient random walk process outperforms other basclines.
16

---

C Random Walk Process in High-dimensional State Space
To evaluate the scalability of our random walk process within high-dimensional state spaces, we' identified two essential questions to consider:
1. When the agent starts to explore from arbitrary terminal state, can the agent visit a diverse range states through the random walk process?
2. Given the diverse range of visited states, possible to identify the terminal state (guide skill) within the least explored region?
In response, we conducted synthetic experiment on Montezuma's Revenge, which is a high- dimensional pixel-based environment characterized by an 84x84x3 imput and is well-known for its exploration challenges.
Terminal State
Random Walk Steps
The agent can visit diverse states by Random
Walk Process in high dimensional environment
State
Skill
State
Skill
Cell
The lowest density (i.e,, The lowest visitation]. Select Skill as Guide Skill
State
Skill
Cell
Skill
Cell
Figure 9: Random walk process in high dimensional environment
Figure 2 provides a visual illustration of our random walk approach within Montezuma's Revenge. In this experiment, we:
1. Randomly reset the agent in the initial room of Montezuma's Revenge.
17

---

2. Execute a random walk for 100 steps from the randomly initialized state.
3. Iterate the step 1,2 for N cycles.
This experimental design mirrors the algorithmic design of DISCO-DANCE where each reinitialized point indicates the terminal state of different skill. Correspondingly, each skill undergoes random walk for 100 steps, aligning with the parameters used for our paper (i.e., P=N, R=100, M=1 in Equation 4
Regarding the first question, as illustrated by Figure 9 even in such a high-dimensional state space, the agent was able to visit a variety of states within just 100 random steps. For instance, for Skill ',, the agent was able to move to another room. And with for Skill 'J', the agent successfully picked up the key. This experimental result supports the versatility of the random walk process, even when the environment is high-dimensional.
Regarding the second question, we employed the cell-centric technique from Go-Explore [1] for density estimation: (i) segmenting the aggregated states into discrete cells, (ii) count the number of each cell's visitation, and (iii) select the least visited cell. As emphasized in Figure / the cell marked in red is selected as guide skill, indicating that skill 'I' is the prime candidate for guide skill in DISCO-DANCE.
This combined approach of exploration through random walk then identifying unique states (which is used in DISCO-DANCE), parallels the approach adopted by Go-Explore. This technique has consistently demonstrated its efficacy across varied domains, including Atari games and robotic settings. In summary, we believe our synthetic experiment affirms the scalability of our random walk process in DISCO-DANCE, even within a nigh-dimensional pixel-based environment.
D Experimental Details
D.1 Environments
2D maze The width and height of easy and medium level mazes are both 5, and for hard level maze (7). There is no explicit terminal state, and the episode ends only when the maximum timestep (30 for easy and medium levels and 50 for hard level) is reached. The state space and action space are both 2-dimensional (state: x, y coordinates, action space: distance to move in x, y coordinates).
Ant maze The width and height of U-maze are (7.7) and Empty, I1-shaped maze are (8.8) respec- tively g 30 Similar to 2d-maze, episode ends only when the maximum timestep (400 for every mazes) reached. The state space 31 dimeensional,, and the action space is 8- 8-dimensional.
Deepmind Control Suite We conduct our environment in total of eight downstream tasks provided by URLB (22). In detail, we utilize four tasks each in two continuous control domains: Run, Run backward, Flip, and Flip backward for Cheetah and Jump, Run, Stand, and Walk for Quadruped. For cheetah, the state space is 17-dimensional, and the action space is 6-dimensional. For Quadruped, the state space is 78-dimensional, and the action space is 12-dimensional.
D.2 Evaluation
State Coverage (2D, Ant mazes) We discretize the and y of the environment into 10 intervals (i.e., total 10x 10 buckets) and count the number of buckets reached by learned skills.
Downstream Task Performance For Ant mazes, we conduct goal-reaching downstream tasks. We design a distance between the current state and the goal state as a penalizing reward (i.c., -1 distance = reward). In detail, directly computing distance with and coordinates might be inaccurate because there can be obstacles (i.c., walls) between two states. Therefore, we design a reward function that returns the actual distance that goes around when there is an obstacle between two states. Then, we normalize the reward to set the lowest value as 11. For Ant mazes and DMC, first perform pretraining for 5M(Ant mazes) 2M(DMC) training steps and select the skill with the highest return for each task. Then we finetune for IM(Ant mazes) 100k(DMC) steps.
For DIAYN, SMM, DISDAIN, and DISCO-DANCE (discrete skills), we finetune the skill with maximum downstream task reward. For APS (continuous skill), we first randomly select an arbitrary skill and rollout episodes, then solve the linear regression problem to select the task-sp skill
18

---

(following the same protocol in the APS paper). After the skill is selected for finetuning, we initialize the agent with the pre-trained network parameters. Utilizing pretrained parameters for the entire parameters of the critic can impede the rapid adaptation process, the pretrained network is already optimized for the intrinsic reward. To mitigate this issue, we initialize only the encoder of the critic network with pretrained parameters (i.e., excluding the last fully connected layer). In addition, we initialize the actor network with entire pretrained parameters (including the last fully connected layer).
Note that our evaluation on the downstream task is different from URLB in that the skill selection process is not included in 100K, but this is still a fair comparison as all algorithms use the same number of interactions (samples) select skills. For UPSIDE, (i) first we select the leaf node skill with maximum downstream task rewards, (ii) then freeze all the pretrained ancestors of that selected leaf node skills and (iii) only finetune the leaf node skill, same as the original paper. detailed explanation on the fine-tuning procedure of UPSIDE is described in Appendix
D.3 Hyperparameters
Detailed hyperparameters of our method DISCO-DANCE are listed in the table 4 Note that the additional steps used to perform the random walk process have been included in the overall count of environment steps for training.
Table 4: A common set of hyperparameters used in DISCO-DANCE.
Hyperparameter
Value
Replay buffer size
10t
Optimizer
Adam
Learning rate
3 x 10-4
RL algorithm
Soft Actor Critic 0.99
discount
Select guide
nearest neighbor 10-4
Guide coef (ox)
Extending threshold p
Accuracy threshold
0.5
Table 5: Per environment sets hyperparameters used in DISCO-DANCE.
Hyperparameter
2D maze
Ant maze
DMC
Hidden dim
128
1024
1024
Batch size
64
512
1024
casy, normal: 30
Skill trajectory length
400
1000
hard: 50
Initial number of skill Entropy coef (3)
30
30
10
0.2
0.6
0.2
easy, normal: 106
Number pre pre-training frames
5 x 106
2 106
hard: 5 x 106
For DISDAIN, we search the number ensembles from [2, 10, 20, 40] and disdain reward coefficient from [10, 50, 100]. For 2D maze, We train DISDAIN using 40 sizes of ensembles and set DISDAIN reward weight as 100. For Antmaze, we set 20 for the number of ensembles and 100 for the reward coefficient. For DMC, utilize 2 for the number of ensembles and 10 for the reward coefficient.
For APS, we search the value of (i.e., the number of neighbors from [5, 10, 12] and the dimension of skills from [5, 10). For 2D maze, we utilize 10 for k and 5 for skill dimensions. For Antmaze, we utilize 5 for k and for skill dimensions. For DMC, we utilize 12 for k and 10 for skill dimensions.
nearest neighbor algorithm)
19

---

For SMM, we search the entropy coefficient from [0.0005, 0.005, 0.01, 0.1, 0.25, 0.5, 1]. We utilize the entropy coefficient as 0.1 for 2D maze and Antmaze, and 0.01 for DMC.
For UPSIDE, there are five kinds of hyperparameters: maximum timestep T, diffusing timestep H, discriminator threshold 17. initial extend skill number staart and maximum extend skill number Nmaa- We set 7' = H = 5 for 2D maze, T' = H = 50 for Antmaze and ''' H = 125 for DMC. In addition, we utilize 1л = 0.5, NN sttar 2, and Nmaz 8 for all environments.
We use SAC as the backbone RL algorithm [5]. We train DIAYN, APS, and SMM using the code re-imllement DISDAIN and UPSIDE by strictly following
provided by URLB (22). In addition, the details of the paper.
We use a fixed number of skills for the baselines (i.e., DIAYN, DISDAIN, and SMM), which utilize discrete skill spaces. We set 50 for easy and normal level 2D mazes, 70 for the hard level 2D maze, and 100 for Antmaze and DMC.
D.4 Resource Requirements
Full experimental training runs 12 hours for the 2D maze and 24 hours for Antmaze and DMC. We conducted all experiments using a single RTX 3090 GPU, and each experiment requires approximately up 2GB of memory.
D.5 License
Environment
2D mazes: We used the publicly available environment code in 7
Ant mazes: We used the publicly available environment code in
g
Deepmind Control suite: We used the publicly available environment code in
53
Baseline algorithm
For DIAYN, APS and SMM, used the code provided in 22
E
Additional Experimental Results
E.1
Additional fine-tuning results on the Ant U-maze
To further evaluate whether the learned skills can be useful for downstream tasks, we conducted goal-reaching navi- gation experiment in Ant U-maze, wherein the goal state was designated as the region positioned furthest from the initial state. The quantification of successful goal attain- ment was measured by determining the number of seeds, out of total of 20, in which the agents reached the goal state. We select the skill with the maximum return (i.e.,
0.SCC-ANCC
APS
SHN SONATCH
CMNN
DE
D:52ANN
Su Seeds
skill whose state space closest to the goal state) from the
set of pretrained skills, and finetune the policy. Notably,
DISCO-DANCE reached the goal in the Ant U maze faster than the other algorithms (as indicated by the earliest point where the y-axis reaches 20).
Environment Steps
Figure 10: Finetuning results on the Ant U-maze. We plot the number of successful seeds out of total 20 seeds.
Fig. 10 shows that DISCO-DANCE consistently reaches the goal state (for all 20 seeds) with the fewest number of environmental steps. As shown in the Ant II-maze finetuning experiment (Section 4), the performance of UPSIDE is significantly inferior to the performance of DISCO-DANCE, with only 5 seeds out of 20 seeds reaching the goal state. These results suggest that the tree-structured policies during the pretraining stage hinders the agent's ability to learn optimal policies during the fine-tuning phase.
20

---

E.2 Full Results on Deepmind Control Suite Tasks
Table Performance comparison of the best model performance and underlined scores indicate the second best. The results averaged over 15 random seeds with a standard deviation.
and baselines on DMC. Bold scores indicate
Models
Cheeiah
Quadiuped
Avg
Run
Run Blackward
Flip
Flip Backward
Jump
Run
Stand
Walk
SCRATCH
411 111.25:55.24
386.20 10
704
16.34
714.1018.22
345.05:1:99. 17
140.64:50.11
552.074:244.5 25
119,00111168
421.71
DIAYN
512,63-145.12
450,35 98.9
571.
16 26
717 01
628 26:-3172000
414 59-23447
743, 533.53635533 31
567 67-666444.11
553.40
SMM
580 79 77.--4400 97
436 33.0
668. 120 16
733.79: 16.86
490 38--388577
542.5 2114419
434 175-766
5114015300
520.44
APS
574.00 56-47
437 16.96
640.51:19.51
684 384-86115 146.9-888
572, 942.82
237.20:219.22
640.51::79.77
06641198555
$31.69
DISDAIN
386 00.-48:-5222
465 6541557
673.54 02.544.95544
731 .9..9.24444
476, 164:200191
246.45::33432
725. 225.42:20000
M7.24
529.13
UPSIDE
309.811121.50
259, 175:91744
607, 967.976117
622 84-38 64
695. 294
246.32:1196.2
035.24::12211
359 339.62.4337..3
442.29
DISCE-DANCE
705/231/560
Table summarizes the full results for each task in DMC. We can observe that DISCO-DANCE gets first or second performance in seven out of eight tasks. The performance of the USD baseline was superior to the scratch, demonstrating that previously learned behavior can be effectively utilized when solving downstream tasks. DISCO-DANCE outperforms other baselines in terms of average performance, and this indicates that DISCO-DANCE learned more general behaviors through guidance.
E.3 Additional Ablation studies
w/ Init
wio init
5
K
K 10
bottlenec maze
AntU ma20
AntU Maze
ensiromment Steps
anvirurmmnt steps
environment stops
(a) Apprentice skill initialization
(b) Number of
KNN
KDE
0.27
0.17
0.5T
0.5
0.7
0.3
60
Antui Mare
AntU Mare
bottlenect maze
5000
(c) Density estimation
(d) Random walk steps
(e) Threshold sensitivity
Figure 11: Additional ablation studies. (a) Apprentice skills initialization. (b) A number of in K-nearest neighbors. (c) Density estimation methods. (d) A number of random walk steps. (e) Sensitivity to the accuracy threshold.
First, incrementally introduce new skills, we employ the guide skill to initialize the weights of the apprentice skills, allowing for direct access to unexplored states. To assess the significance of this initialization method, we compare it to control condition in which the weights from scratch. As shown in Fig a). utilizing the guide skill for initialization leads to improved performance. We speculate that guide initialization directly minimizes the KL divergence with guide skill, thereby maximizing the guide reward at the onset training.
trained
Nest, we examined the sensitivity of the results regarding the value of k in k-nearest neighbors. As observed in our experiments (Fig. [Tp), there were no significant differences in state coverage. This is because the most exploratory skill already has its terminal state far enough from the terminal states
21

---

of other skills. Therefore, adjusting the value of k higher or lower does not significantly impact the selection of guide skills. This demonstrates the robustmess of DISCO-DANCE with respect the hyperparameter k.
Additionally; analyze how the DISCO-DANCE performs with different measure of dis- tance/density, we employed kernel density estimation (KDE) for selecting the guide skill. Specifically, (i) performed random walks (as our random walk guide selection process), (ii) trained KDE model on the random walk arrival states, (iii) selected the state in the lowest density region among the collected random walk arrival states, and (iv) chose the skill from which that state originated as the guide skill. As shown in Fig. [T, we found that the performance of both methods was nearly indistinguishable. This is likely due to the terminal state of the most exploratory skill being sufficiently distant from the terminal states of other skills, as was observed in the knn ablation study.
In addition, conducted additional experiments to investigate the effect of varying the length of the random walk steps. Since the environment steps used during the random walk process are included in our total environment steps (total environment steps general trajectory collection steps random walk process steps), increasing the number of random walk steps leads to a decrease in the number of states utilized for model updates. As result, we observed slightly lower performance in the initial environment steps when comparing the result of IT and 0.5T. However, we found that the models converge after 3M steps, with smaller performance differences between them.
To further examine the sensitivity of the accuracy threshold e, we conduct an ablation study by varying from [0.3, 0.5, 0.7]. As illustrated in Fig. TTe, the performance of our proposed method is found to be robust to changes in the accuracy threshold c.
E.4
Comparing curriculum approach effects on other USD methods
existing skils
skills
DISCO-DANCE DIAYN!
DISDAIN
State coverage 196)
2200 coverage
60
environment steps
environment
DWYN'
(a) Skill trajectories
(b) State space coverage
Figure 12: Gradually increasing the number of skills for DISCO-DANCE, DIAYN, and DIS-
DAIN. (a) Qualitative results bottleneck maze and Ant 11-maze. Colored skills indicate new skills and grey skills are previously converged skills. (b) Training curves on each environment.
To demonstrate the compatibility of our proposed method DISCO-DANCE with increasing the number of skills during training, we also incorporate a curriculum approach into the baselines which utilize discrete skill spaces. As depicted in Fig. h2h, we observe that DIAYN and DISDAIN do not benefit from curriculum learning, the newly added skills primarily remain in proximity the initial starting point. This results in a more congested covered state space, thus the curriculum approach fails to increase state coverage and instead decreases discriminator accuracy. In contrast, for our proposed method DISCO-DANCE, the newly added skills are able to access unexplored regions with the assistance of the guide skill.
F Comparison with UPSIDE
DISCO-DANCE and UPSIDE [19] have similar motivation, alleviating the inherent pessimism problem. Both of their strategies can be summarized as "agent guides itself" (i.e., existing skills
22

---

help new/unccnverged skills to explore). In DISCO-DANCE, guide skill encourages apprentice skills (unconverged skills) explore unexplored states by minimizing the KL divergence between guide skill and apprentice skills. While DISCO-DANCE learns set of single skills, UPSIDE learns tree-structured policy which composed of multiple skill segments.
Optimal trajectory
Fine-tuned trajectory
goal
goal
gool
2+3
2-3
2-3
2-5
2-5
2-5
2-1
2ml
2+2
A-2
2-2
cntro
Tree structure broken
(a) DISCO-DANCE: Finetune skill =
(b) UPSIDE:
(c) UPSIDE:
Finetune only leaf skill z Finetune 2 = 1,4,8 4,8 simultaneously
Figure 13: Conceptual illustration of (1) learned skills of DISCO-DANCE and UPSIDE, and (2) goal-reaching downstream task. The black edge represents skill, and the black circle represents the terminal states of its skill. (a) After unsupervised pretraining, DISCO-DANCE obtains a set of skills that can be used individually without the need for each skill to be used together. In the finetuning stage, DISCO-DANCE selects skill 8 (i.e., skill with the highest downstream task reward) and finetunes the skill to reach the goal quickly. (b) However, to execute 1, 8 sequentially). In
8, UPSIDE
requires sequential execution of all its ancestors' skills (i.e., executing the finetuning stage, UPSIDE is not able to learn optimal policy since the ancestors' skills are kept fixed during finetuning. (c) If finetune ancestors' skills simultaneously, the tree structure learned in the pretraining phase will be broken, which makes finetuning unstable and difficult.
Fig B illustrates an example where a total of 8 skills are learned. When learning the policy in the unsupervised pretraining stage, UPSIDE (1) selects the skill with the largest discriminator accuracy among the leaf node skills (e.g.,
4), (2) adds new skills as its children nodes (e.g.. 7, 8). (3) freeze the parent skills (e.g.. 1, 4 are not trained) and only train newly added skills, and (4) iteratively repeat these procedures to construct the tree-structured policies (Fig[1][b). This makes the fundamental difference between DISCO-DANCE and UPSIDE, that UPSIDE requires sequential execution from ancestors' skills to child skill in a top-down manner due to its tree-structured skill policies. For example, if we want to run skill 8, UPSIDE needs to execute skills sequentially (z = + 4 - 8) for I" timesteps, where T is a hyperparameter that decides how dense the generated tree will be.
One of the key objectives of unsupervised skill discovery is to utilize the learned skills as a use- ful primitive at the fine-tuning stage. However, these sequential executions of UPSIDE bring significant inefficiency at the finetuning stage due to the following reasons. Suppose the given downstream task is to reach the goal state as fast as possible, where the distance from the goal state 8 is the skill with minimum distance from the goal, DISCO-DANCE selects skill z 8 and finetunes Ta |s, = 8) to optimize given reward functions (Fig a3)))). However, for UPSIDE to finetune = = 8, we need to finetune
and total execution time are given as penalty (Fig (3). Since
2 = 1, 4, 8 simultancously (Fig [3.c)). In that case, the tree-structured skill policies learned in the pretraining stage are broken during the finetuning stage. That is, the dictionary {²2 8" : [1, 4, 8]} cannot be used anymore because executing 2 1 for timesteps does not move the agent to the original = 1 terminal nodes (red in Fig (e)). This means that the skill tree learned in pretraining can no longer be used, leading to inefficient finetuning.
To avoid this limitation, UPSIDE only finetuned the leaf skill in the original paper (i.e., freeze ancestors 2 = 1, 4 and only finetunes =
Fig 13). However, this would still be ineffective because the fixed ancestors = 1,4 are not the optimal solution to solve the given downstream task (green lines from <start> to z = 4 nodes in Fig (3(b)). The problem becomes more serious in a long- long-horizon environment such DMC (where the horizon is 1000) if we freeze all the pretrained
23

---

ancestors' skills and finetune only the last leaf skill. In contrast, DISCO-DANCE selects a single skill to fine-tune, which can be executed independently, so does not suffer from the above problem.
There are further differences between DISCO-DANCE and UPSIDE the pretraining phase in addition to the differences in the finetuning stage. In the pretraining stage, both algorithms leverage previously discovered skills for enhanced exploration. st, UPSIDE chooses one of the leaf nodes the parent node, which additional new skill nodes will be added as children that node (e.g.. select = 4 as parent and add 2 = 7,8 as children nodes in Fig. D3p. Then 2 = 7, 8 explores the state region over the state space occupied with previous skills (e.g., 2 = 4). On the other hand, DISCO-DANCE selects guide skill from among the existing skills, and the KL divergence between the guide skill and the apprentice skill is used reward signal. The main difference that UPSIDE selects the parent skill with the highest discriminator accuracy (which corresponds to Fbg.2b in main paper) and DISCO-DANCE utilizes proposed random walk guide selection process (Fig.2c in main paper). As the most distant skill is typically the one with the highest discriminator accuracy (because the discriminator can easily classify the distant skill), the selection strategy utilized by UPSIDE is less effective in exploration, as we demonstrate in Table 2 and Fig.8 (main paper).
Lastly, both DISCO-DANCE and UPSIDE employ random walks (i.e., the diffusing component in UPSIDE) during the pretraining phase. However, the utilization of random walks serves completely different role in each approach. walk + finding lowest density region via knn) is only utilized to select the guide skill, which is orthogonal the policy To learning. contrast, UPSIDE produces rewards from the random walks. Specifically, UPSIDE (1) performs random walks at each skill's terminal nodes and (2) updates the policy No and discriminator ds with random walked states (i.e., gets more reward if their random walked states are discriminable from other skill's 's random walked states). This process induces the local coverage of each skill (i.e., each skill occupies the random walked states, where other skills cannot come in).
DISCO-DANCE, the random walk process (random
G Limitations and Future Directions
DISCO-DANCE may cause cost (sample) inefficiency when measuring the density of the state distribution in environment with high dimensional input, such control from pixel observations. In this case, we could utilize the downscaling technique (e.g., cell representation in Go-Explore) to reduce computational costs.
For the guide skill selection process to work stably, the termination state should be reliably reached. Therefore, when the stochasticity of the environment is too large, it would not be easy to select guide skill only with the simple random walk process we proposed. We leave it as future work for alleviating such difficulties.
H Broader Impact
Unsupervised skill discovery (USD) promotes efficient learning in complex environments, allowing agents learn without the need for explicitly labeled data. This technique has real-world applications ranging from robotic manipulation to autonomous vehicles and can also improve the Al's ability to generalize its skills new situations. However, one of the most significant problems of USD is its sample inefficiency due to the necessity of running extensive simulations. Research like DISCO-DA RANCE can substantially enhance the sample efficiency of USD.
24

---

Visualization of learned skills
I.1 2D mazes
diayn
disdain
smm
aps
upside
DISCO-DANCE (a) Empty maze
diayn
disdain
upside
DISCO-DANCE
(b) maze
Figure 14: Qualitative visualization of the learned skills on Emtpy maze and Y maze. Visualiza- tion of multiple rollouts of learned skills by baseline models. For DISCO-DANCE, we visualize our guiding procedure during training. Bold red lines indicate the guide skill.
25

---

diaym
disdain
Smm
aps
upside
DISCO-DANCE
(a) W maze
diaym disdain
smm
aps
upside
DISCO-DANCE
(b) S maze
Figure 15: Qualitative visualization the learned skills on W maze and S maze. Visualization of multiple rollouts of learned skills by baseline models. For DISCO-DANCE, we visualize our guiding procedure during training. Bold red lines indicate the guide skill.
26

---

1.2 Ant Mazes
diayn
disdain
Smm
aps
upside
DISCO-DANCE
(a) Ant Emply-maze
diaym
disdain
smm
aps
upside
DISCO-DANCE
(b) Ant U-maze
diayn
disdain
smm
aps
upside
DISCO-DANCE
(c) Ant N-maze
Figure 16: Qualitative visualization of the learned skills on Ant Empty-maze, Ant U-maze, and Ant Il-maze. Visualization of multiple rollouts of learned skills by baseline models. For DISCO-DANCE, we visualize our guiding procedure during training. Bold red lines indicate the guide skill.
27

---

1.3 Deepmind Control Suite
Figure 17: Qualitative visualization the learned skills in the cheetah in DMC. DISCO-DANCE learned 100 skills without reward function. In order to facilitate visualization, the skills selected based on their corresponding terminal states.
28

---

Z10
Z1
29
12
27
23
24
Figure 18: Qualitative visualization of the learned skills in the quadruped in DMC. DISCO- DANCE learned 100 skills without reward function. In order to facilitate visualization, the skills are selected based on their corresponding terminal states.
29


Reviews:
[{"review_id": "toj5vG2hK9", "reviewer": "Reviewer_gakY", "summary": "The paper introduce a reward shaping approach called DISCO-DANCE to enhance exploration in unexplored states in the context of latent variable skill learning in self-supervised RL context. The approach consist in defining a guide skill with the highest potential for reaching unexplored states for a given environment, then select unconverged skills and incentivize them to follow the guide skill expressed at terminal state, helping them bypass state regions with low rewards. Finally, the approach disperse the skills to maximize their distinctiveness, resulting in a set of skills that cover a wide range of states. DISCO-DANCE fills the pathway to unexplored regions with positive rewards. The approach is compared in navigation and locomotion scenarios surpassing some previous methods in terms of exploring the state space and performing well in navigation tasks in 2D mazes and Ant mazes.", "questions": "Why not having compare with DADS, MUSIC and LSD?\nIs the random walk used at the last step of the guide skill discovery is scalable to larger state environments ?", "limitations": "The experiments are rather limited to justify the benefit of the approach.\nThe random walk as last step of the definition of the guide skill definition isn't shown to be scalable to larger state environments.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper is well written and explain rather well his approach.\nThe problem of skill learning with self-supervised learning is important and actual.\nThe justification is reasonably clear, maybe it would have been interesting to discuss more the difference between environment were most degrees of freedom are part of the action space, like in locomotion and the cases of manipulation that poses the most issues.\nThe experiment use classic but simple 2D navigation and simulated locomotion scenarios to illustrates the benefit.", "weaknesses": "The comparison is rather limited, we woud have like to see DADS, MUSIC and LSD for example.\nWe would also have like to see experiments in more known challenging environment like manipulation where MI is the most in trouble.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback. Please let us know if you have any further comments or feedback. We will do our best to address them.\n\n> Q1. Why not having compare with DADS, MUSIC and LSD? We would also have like to see experiments in more known challenging environment like manipulation where MI is the most in trouble.\n\nA1. We agree that including more baseline algorithms such as DADS, MUSIC, and LSD [1,2,3] would further increase the impact of DISCO-DANCE, and it's indeed a valuable direction for subsequent work. Our primary focus in this work, however, is to tackle an issue of pessimistic exploration inherent in MI-based skill discovery algorithms. As highlighted in Section 5 (line 297:302, 312:315), our chosen baselines are algorithms which focus on devising an effective auxiliary reward  ($r_\\text{exploration}$) to help exploration. On the other hand, DADS, MUSIC, and LSD mainly focus on modeling the representation of skills ($r_\\text{skill}$). This is why these algorithms are not included in our paper as baselines.\n\nFurthermore, our experimental environments are chosen to measure the effectiveness of the devised $r_\\text{exploration}$. To evaluate how well $r_\\text{exploration}$ helps overcome pessimistic exploration, we utilized environments such as 2D and Ant mazes. In such settings, our approach outperformed baseline algorithms, validating the effectiveness of $r_\\text{exploration}^\\text{DISCO-DANCE}$ in enhancing the exploration. Moreover, to demonstrate DISCO-DANCE aids in not only navigation tasks but also in learning useful skills in more diverse settings, we conducted experiments on a widely-used continuous control benchmark, URLB [4], and were also able to surpass all regarding baselines.\n\nTo clarify the significance of devising a well-designed $r_\\text{exploration}$, we conduct additional experiments on LSD (with discrete skills) within a 2D bottleneck maze (please refer to Figure \\(c\\) in the attached pdf). Utilizing their open-source code, we optimized three important hyperparameters (learning rate, entropy coefficient of SAC ($\\alpha$), dimension of skills). As the main objective of LSD is to obtain 'far-reaching' skills, far-reaching skills are well learned in an obstacle-free empty environment (similar to Figure 4 in the LSD paper).\n\nHowever, Figure \\(c\\) shows that LSD struggles to efficiently navigate in bottleneck maze, underscoring that replacing $r_\\text{skill}^\\text{DIAYN}$ with $r_\\text{skill}^\\text{LSD}$ is not enough for solving hard-exploration environments. This emphasizes the importance of well-designed $r_\\text{exploration}$, regardless of what $r_\\text{skill}$ is used. \n\n[1] Dynamics-Aware Unsupervised Discovery of Skills. Sharma et al., ICLR 2020\n\n[2] Mutual Information State Intrinsic Control. Zhao et al., ICLR 2021\n\n[3] Lipchitz-constrained Unsupervised Skill Discovery. Part et al., ICLR 2022\n\n[4] URLB: Unsupervised Reinforcement Learning Benchmark. Laskin et al., NeurIPS 2021\n\n\n> Q2. Is the random walk used at the last step of the guide skill discovery is scalable to larger state environments?\n\nA2. Please refer to General Response #1, titled **General Response: Random walk process in high-dimensional state space**."}, {"review_id": "qmKQCIS7ej", "reviewer": "Reviewer_adLT", "summary": "This paper addresses the challenge of learning diverse skills in unsupervised reinforcement learning by introducing a method to selectively guide candidate skills to areas of the state space with low coverage from the current set of skills.  Rather than relying directly on mutual information maximization like many skill-learning approaches do, this paper proposes an algorithm for selecting guidance skills and optimizing apprentice policies through a combination of mutual information rewards and a guidance reward.  The guidance reward is high when the apprentice skill reaches similar areas of the state space as the guidance skill, which can help direct the apprentice skills to previously unexplored areas.  The paper shows that the proposed method, DISCO-DANCE, can cover more of the state space than alternate approaches by evaluating in maze navigation environments.  The paper also provides ablation studies to help understand what each part of the method contributes to the overall performance.  ", "questions": "\n1.  Appendix F describes the additional costs of the random walk process, and proposes an efficient method.  Are the additional steps used to perform random walks, even for the temporary buffer, included as environment steps in the overall training budget?  If not, this seems an unfair advantage given exclusively to DISCO-DANCE when comparing to other methods.  \n\n2.  The hyperparameters used for UPSIDE (from Appendix B) seem to limit the number of learned skills to 8 while DISCO-DANCE seems to start with more skills and has the possibility to expand.  Can you comment on this choice?  Wouldn't a higher number of skills for UPSIDE reduce the described fine-tuning difficulty by allowing more coverage?\n\n3.  DISCO-DANCE seems to require skills to reliably end in a terminal state.  How are those states tracked and determined when the skills are still being trained?", "limitations": "Limitations are explicitly described and claims are not overly grand.  ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written, with each aspect of the DISCO-DANCE algorithm explained and compared with other methods from the literature.  The contribution, both in terms of empirical results and novelty of the proposed method, is well defined.  Diagrams were effectively used to make the approach intuitive.  \n\nIn addition to describing the method well, the paper evaluates the experimental performance thoroughly, comparing it to several similar methods on several domains.  The experiments use both state coverage and fine-tuning performance, and show that DISCO-DANCE performs well.", "weaknesses": "The main weakness of the paper mostly involves my uncertainty around the questions in the next section.  I hope that the rebuttal can help clear up confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful questions and positive support. Please let us know if you have any further comments or feedback. We will do our best to address them.\n\n> Q1. Appendix F describes the additional costs of the random walk process, and proposes an efficient method. Are the additional steps used to perform random walks, even for the temporary buffer, included as environment steps in the overall training budget? If not, this seems an unfair advantage given exclusively to DISCO-DANCE when comparing to other methods.\n\nA1. Yes, the additional steps used to perform random walks have been included in the overall count of environment steps for training. \n\n> Q2. The hyperparameters used for UPSIDE (from Appendix B) seem to limit the number of learned skills to 8 while DISCO-DANCE seems to start with more skills and has the possibility to expand. Can you comment on this choice? Wouldn't a higher number of skills for UPSIDE reduce the described fine-tuning difficulty by allowing more coverage?\n\nA2. UPSIDE sets the initial state as the parent node, from which it generates $N_\\text{start}=2$ skills (i.e., leaf nodes) that move a short distance from parent node. UPSIDE then incrementally adds new leaf node to sufficiently cover the state space around the parent node, up to a maximum of $N_\\text{max}=8$ nodes. Subsequently, among these leaf nodes, the most discriminable skill is chosen and set as a new parent node, which then generates its leaf nodes. This procedure of skill tree expansion is iteratively executed. Hence, if there is remaining state space in the environment, UPSIDE can continue to increase the number of skills through the aforementioned process until the end of training.\n\nIn our experiments, we performed hyper-parameter search for $N_\\text{max}$ from 6 to 10, and found that there was minimal difference in performance. Therefore, we selected 8 for $N_\\text{max}$ (which was used in orignal paper).\n\n\n> Q3. DISCO-DANCE seems to require skills to reliably end in a terminal state. How are those states tracked and determined when the skills are still being trained?\n\nA3. As outlined in Algorithm 1 on page 5, line 5, the guide skill is chosen once the majority of skills are discriminable enough (i.e., high discriminator accuracy). This indicates that the terminal states achieved by each skill are consistent. This ensures that, when selecting the guide skill, most skills will consistently end in the (almost) same terminal state across multiple rollouts.\n"}, {"review_id": "GDerGXPaZO", "reviewer": "Reviewer_b566", "summary": "In this submission, the authors propose an unsupervised skill discovery method called DISCO-DANCE. It samples *guide skills* with random walk processes that start from the terminal states of a set of skills and use them to guide less discriminable or new skills toward those guide skills so that they can reach unexplored areas more easily. They test their method in two navigation environments (2d maze and Ant maze) and Deepmind Control Suite and compare the state space coverages and performances with the baselines.", "questions": "- Do you think there could be an alternative exploration strategy other than the random walk process?", "limitations": "The authors state some limitations of the proposed method (difficulty in high-dimensional state spaces and stochastic environments), but I encourage the authors to consider taking the points I listed in the Weaknesses section into account.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- In terms of orignality, the main idea of this work to find a reachable state that is close to the unexplored region and to expand the skill set based on it is novel to some degree.\n- The manuscript is mostly clear and easy to follow. Also, the concept figure effectively provides the intuition behind the method.\n- The state space coverage problem is an important aspect of unsupervised skill discovery.", "weaknesses": "- The exploration issue with the mutual information (MI) objective could be more than what is described in this work. In theory, the MI objective is not supposed to contribute to the exploration meaningfully, especially in continuous control environments (Park et al. [21]), which can make this method mostly rely on the random walk processes for its exploration.\n- I believe one important weakness of this submission is the random walk process. The manuscript mentions that the rise of the environmental complexity makes existing skill discovery methods less effective and motivates this work, but ironically, in complex environments (e.g., with high-dimensional state spaces), random walk would be one of the main bottlenecks in encouraging exploration. In such environments, this algorithm could require a large number of iterations.\n- In terms of writing, I think it is not very fair to call the state spaces of the environments used for the benchmark *high-dimensional*. They are higher-dimensional compared to the 2D maze environment, but labeling them high-dimensional in general may not be a good standard for the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time and insightful comments. Please let us know if you have any further comments or feedback. We will do our best to address them.\n\n> Q1. The exploration issue with the mutual information (MI) objective could be more than what is described in this work. In theory, the MI objective is not supposed to contribute to the exploration meaningfully, especially in continuous control environments (Park et al. [21]), which can make this method mostly rely on the random walk processes for its exploration.\n\nA1. According to the findings presented in Park [21], they suggest \"MI objective can be fully maximized even with small differences in states as long as different z's correspond to even marginally different ${s_{T}}'s$, not necessarily encouraging more 'interesting' skills\". I believe this aligns with our discussion in lines 29-32. However, if there are any differences between the two that I might have overlooked, I would really appreciate it if the reviewer could inform us. We will reflect on it in the main paper.\n\n\n> Q2. I believe one important weakness of this submission is the random walk process. The manuscript mentions that the rise of environmental complexity makes existing skill discovery methods less effective and motivates this work, but ironically, in complex environments (e.g., with high-dimensional state spaces), random walk would be one of the main bottlenecks in encouraging exploration. In such environments, this algorithm could require a large number of iterations.\n\nA2. Please refer to general response #1, titled **General Response: Random walk process in high-dimensional state space**.\n\n> Q3. In terms of writing, I think it is unfair to call the state spaces of the environments used for the benchmark high-dimensional. They are higher-dimensional compared to the 2D maze environment, but labeling them high-dimensional in general may not be a good standard for the field.\n\nA3. We agree. We will revise it in the final manuscript.\n\n> Q4. Do you think there could be an alternative exploration strategy other than the random walk process?\n\nA4. Yes, there are alternative strategies, such as RND[1] and ICM[2], in order to choose the skill that visits the state closest to the least dense states. However, such an approach requires additional network training. Since the purpose of the guide skill selection process is merely to identify the skill with the highest potential to access unexplored states, we find that a simple random walk process is sufficient. In General response #1, we show that even in Montezuma's Revenge - which features a high dimensional pixel-based state space and is notorious for exploration challenges - 100 random walks are enough to select the guide skill.\n\n We have shown that in our benchmarks (e.g., Ant mazes, DMC, and pixel-based Atari in General response #1), a simple random walk process successfully identifies the guide skill without additional training. We believe that in more challenging environments (e.g., real-world robotics), employing a well-designed exploration strategy will be beneficial, which we leave for future work.\n\n[1] Exploration by Random Network Distillation. Burda et al., Arxiv 2018\n\n[2] Curiosity-driven Exploration by Self-supervised Prediction. Pathak et al., ICML 2017\n\n> Q5. The authors state some limitations of the proposed method (difficulty in high-dimensional state spaces and stochastic environments), but I encourage the authors to consider taking the points I listed in the Weaknesses section into account.\n\nA5. We will add (i) an in-depth analysis of the advantage and limitations of the random walk process (including General Response #1) in Section \"Limitations and Future Directions\" and (ii) a more detailed explanation of the exploration issue with mutual information in Section 2."}, {"review_id": "douVWOWI8b", "reviewer": "Reviewer_EWQN", "summary": "This paper proposes a new unsupervised skill discovery method based on the guidance of exploration. A policy is conditioned on a latent skill variable like in prior work. The method first starts by identifying a \"guide\" skill variable that is likely near unexplored states, the novelty of unexplored states is measured by the density of random walk arrival states started from the terminal states of each skill. It then trains other skills that explore the vicinity of terminal states region from guide skill.\n", "questions": "1. To select guide skills for exploration, the proposed method assumes the terminal state is resettable and needs to measure the density of the state by generating about $O(PRM)$ random walks arrival states from terminal states as section 3.1 mentioned. I wonder whether is there any more sample-efficient and easy ways to do that.  e.g. Is it equivalent to picking a guide skill by directly estimating the density of terminal states sampled from history episodes or fixed-length horizon? if the skill is discriminable (MI reward is well optimized), the terminal states from the same skill may locate in a subregion of (unexplored) states.\n\n2. What does the y-axis mean in Figure 3（b?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper analyzes the limitations of state coverage in previous skill discovery methods and provides nice motivation for the proposed method. \n\n2. The empirical result in mazes and control tasks are promising.\n", "weaknesses": "1. Measuring the density of the state distribution via generating random walk arrival states from terminal states is not sample efficiency. \n\n2. The method needs to select a guide skill that is most adjacent to the unexplored states. In the bottleneck maze tasks in Figure 3, what if all skills including the guide skill cannot pass the first room? Will this method also encourages effective exploration? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thorough review and thoughtful comments. Please let us know if you have any further comments or feedback. We will do our best to address them.\n\n> Q1. Measuring the density of the state distribution via generating random walk arrival states from terminal states is not sample efficiency.\n\n> To select guide skills for exploration, the proposed method assumes the terminal state is resettable and needs to measure the density of the state by generating about O(PRM)  random walks arrival states from terminal states as section 3.1 mentioned. I wonder whether is there any more sample-efficient and easy ways to do that. e.g. Is it equivalent to picking a guide skill by directly estimating the density of terminal states sampled from history episodes or fixed-length horizon? if the skill is discriminable (MI reward is well optimized), the terminal states from the same skill may locate in a subregion of (unexplored) states.\n\nA1. \n\nWe really appreciate your feedback. First, we wish to clarify that our approach does not strictly assume the terminal state is resettable. While leveraging the simulation environment may allow for hard resets to a terminal state of each skill, we consider such an assumption to be unrealistic. Instead, we just rollout each skill to reach the terminal state. Specifically, in the context of the Efficient Random Walk Process (detailed in Appendix F), during training, once the selected skill $z_i$ is clearly distinguishable (i.e., $q_\\phi(z^i|S_T) > \\epsilon$), we just perform an additional 0.2$T$ random walks from the terminal state and perform the density estimation based on these 0.2$T$ number of collected states.\n\n\nAs the reviewer pointed out, there is an alternative approach where we could directly estimate density using the states from replay buffer without a random walk process.  However, such a strategy could result in a scenario as in Figure 2(b). As corroborated by the findings presented in Table 2 and Figure 8, direct estimation from the replay buffer (i.e., no random walk) during guide skill selection can potentially give rise to situations where the nearest skill to unexplored states remains unidentified. Instead, a skill distanced farthest from other skills might be chosen, consequently leading to diminished performance. This underscores that Random Walk Process is necessary for DISCO-DANCE.\n\nIn addition, Random Walk Process is triggered only when the skills are sufficiently distinctive. Therefore, during training, the random walk process is not frequently executed (e.g., 8 times in 2D maze, 12 times in AntMaze, and 18 times in DMC). Moreover, for each Efficient Random Walk Process, it requires just an additional $0.2T$ steps for each skill. Considering its frequency and amount of required steps for each skill, the total steps taken by the Efficient Random Walk Process are not extensive. In summary, the Efficient Random Walk Process consumes a relatively minor portion compared to the overall training steps.\n\nIt's important to note that while the random walk process does introduce some extra steps, exploration signals from the random walk process make DISCO-DANCE a more efficient explorer compared to the other baselines. Consistently, DISCO-DANCE surpasses the performance of the other baselines when measured against the same number of environment steps.\n\n> Q2. The method needs to select a guide skill that is most adjacent to the unexplored states. In the bottleneck maze tasks in Figure 3, what if all skills including the guide skill cannot pass the first room? Will this method also encourage effective exploration?\n\nA2. Yes. To demonstrate that DISCO-DANCE is beneficial in such scenarios, we provide snapshots of our 2D bottleneck maze experiments (please refer to the qualitative figure(b) in the attached pdf). As shown in the figure, even if all skills cannot pass the first room, the skill with the highest potential to approach the unexplored states (i.e., the next room) is chosen as the guide skill through the random walk process. This, in turn, motivates apprentice skills to move towards more explorable states.\n\n\n\n> Q3. What does the y-axis mean in Figure 3(b)?\n\nA4. The y-axis represents each individual skill policy that the agent has (in no particular order). We will add more explanation in the main text.\n"}, {"review_id": "N1SQk72rh7", "reviewer": "Reviewer_oWju", "summary": "This paper proposes an algorithm called DISCO-DANCE for unsupervised skill discovery in RL. The algorithm augments the mutual information reward of DIAYN with a gudiance reward. The guidance reward encourages indistinguishable or unconverged skills to follow skills that can potentially visit under-explored states, such that the over all skill collection can have broader state coverage. The authors conduct experiments on 2d navigation, Ant maze, and DMControl suites by evaluating the state coverage and the fine-tuned downstream task performance, showing that DISCO-DANCE outperforms baselines in most of these benchmarks.", "questions": "+ The authors mentioned that new skills are added if existing skills are mostly converged. In my opinion, the skill-latent-conditioned policy is an MLP with states and latents concatented as the input. If a new skill is added, the input dimension is also changed. How do you add a new skill? Do you perform a neural network surgery or specify a sufficiently large number of skills at the beginning of training?\n\n+ Correlated with the above question, since DISCO-DANCE adds new skills during training, are the total number of skills trained by DISCO-DANCE and baselines the same in experiments?\n\n+ In the first sentence of bstract, I think diminished bonuses (e.g. DIAYN) are not equivalent to \"penalties\".\n\n+ What is the x/y-axis of fig.3(b)?\n\n+ Fig.6 is wierd. Why not present, e.g., the success rate over 100 trials averaged over 20 seeds with error bars?\n\n+ In Algorithm 1, the guide skill z* is not defined if the \"most skills are discriminable enough\" condition is not satisfied.\n\n+ Many meaningful results are presented in the Appendix (the base RL algorithm, additional ablation studies). I think they should be linked in the main body such that readers can have a sense.", "limitations": "+ The authors have addressed several main limitations in appendix I.\n+ The selection of the guide skill depends on the final state visited existing skills. It does not seem to be a general solution to select guide skills even for state-based environments.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Illustration of the main idea (i.e., find and follow a guide policy) is clear (in particular fig.1) and well-motivated.\n\n+ The authors provide open-source code and extensive experiment details such as hyperparameters and resource requirements, which make the results presented in this paper reproducible.\n\n+ The appendix provides extensive discussions about the algorithm's limitation and comparison with other baselines, which can help reader understand the proposed algorithm deeper.\n\n+ Experiment results are promising.", "weaknesses": "+ The authors tend to address the problem unsupervised discovery in complex environments where existing methods are no longer effective, but experiments are mainly conducted on common benchmarks. I acknowledge that several maps in the navigation task are challenging, but the locomotion tasks (AntMaze and DMC) are not. Actually, baseline algorithms can outperform the proposed algorithm in terms of downstream task performance (see Appendix G).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful and constructive feedback. \n\n> Q1. Why use common benchmarks (AntMaze, DMC) to evaluate the exploration ability? \n\nA1. As the reviewer pointed out, 2D mazes (especially bottleneck mazes) have more complex layouts than the other two benchmarks. However, despite this, the other two environments still require effective exploration.\n\nFor Antmaze, while its environmental layout may seem simpler than a 2D maze, it is noteworthy that the dimensionality of the state space and action space is considerably higher. This intricacy makes it the agent more challenging to optimize RL policy. Consequently, Antmaze presents as a complex environment for Unsupervised Skill Discovery agents, necessitating exploration strategies. In practice, as shown in Figure 7, we can find out the agent cannot go far from the initial state as the maze layout becomes more complex (Empty-maze, U-maze, $\\Pi$-maze). This empirical trend underscores the challenging nature of the Antmaze environment and affirms its suitability to measure the agent's exploration ability.\n\nWe agree that DMC is not typically used as a hard exploration environment. However, for unsupervised skill discovery (USD) agents, DMC presents a non-trivial exploration challenge. For example, in Cheetah, a USD agent may easily stay near the initial location due to the pessimistic exploration problem. This occurs because the discriminator can easily distinguish skills by observing ‘slight movements’ (e.g., marginally lifting joints).  As a result, the agent may not learn to move further (i.e., run) since skills that involve staying near the starting point are already easily distinguishable. Therefore, without additional exploration signals (e.g., exploration reward), learning running skills is not easy.\n\nURLB involves various tasks within a single environment (e.g., Cheetah-run, flip). Therefore, merely excelling in one task (e.g., run) does not guarantee high performance in others (e.g., flip). The DMC benchmark evaluates the diversity of the learned skills, meaning that agents must learn a suitable set of skills for all downstream tasks in order to achieve consistently high scores across all tasks. The results indicate that DISCO-DANCE outperforms the other baseline methods, as evidenced by the aggregated IQM value (i.e., DISCO-DANCE learns diverse skills to quickly adapt to diverse tasks)\n\n> Q2. How do you add a new skill? Do you perform a neural network surgery or specify a sufficiently large number of skills at the beginning of training?\n\nA2. We set a maximum number of skills that DISCO-DANCE can acquire (e.g., 100) and initialized the network accordingly (e.g., first FC layer of policy as nn.Linear(observation_dimension + 100, hidden_dimension)). \n\n> Q3. Are the total number of skills trained by DISCO-DANCE and baselines the same in experiments?\n\nA3. Yes. We first conducted experiments with DISCO-DANCE for each environment. Based on the maximum number of skills DISCO-DANCE acquired for each environment, we set the total number of skills for other algorithms accordingly.\n\n> Q4. In the first sentence of abstract, I think diminished bonuses (e.g. DIAYN) are not equivalent to \"penalties\".\n\nA4. We agree. We will revise this to \"a significant reduction in reward acquisition\".\n\n> Q5. What is the x/y-axis of fig.3(b)?\n\nA5. The x-axis shows how much the cheetah has moved horizontally. The y-axis represents each individual skill policy that the agent has (in no particular order). We will add more explanation in the main text.\n\n> Q6. Fig.6 is weird. Why not present, e.g., the success rate over 100 trials averaged over 20 seeds with error bars?\n\nA6. Unlike typical experiments in goal-conditioned RL, where trials are performed 100 times per seed with varying goals and success rates are averaged across all seeds, our approach focused on a single, fixed goal - the most distant point from the initial state. This design choice evaluated the effectiveness of previously acquired skills in reaching challenging states (i.e., farthest state from the initial state). As a result, if a particular seed learned a policy that reaches this fixed goal, it would succeed in nearly all of its 100 trials. On the other hand, if it hasn't learned the policy of the goal yet, it would barely succeed in any of the 100 trials (i.e., the success rate for each seed is almost always either 1 or 0). Therefore, in Fig.6, instead of average success rates, we plotted the number of seeds that succeeded in reaching the goal at each timestep for simplicity.\n\n> Q7. In Algorithm 1, the guide skill z* is not defined if the \"most skills are discriminable enough\" condition is not satisfied.\n\nA7. Thank you for the correction. We will add \"$\\text{Initialize guide skill } z^* = \\text{ None}$\" in line 1 and \n\"\"$\\text{If guide skill } z^* \\text{ is not None:}$\"\" in line between 7 and 8.\n\n> Q8. Link the main paper and the Appendix.\n\nA8. Thank you for recognizing the depth of our results in the Appendix. We will add explicit references to the Appendix content in the main paper.\n\n> Q9. The selection of the guide skill depends on the final state visited existing skills. It is not a general solution to select guide skills, even for state-based environments.\n\nA9.  As outlined in Algorithm 1, line 5, the guide skill is chosen once most skills are discriminable enough (i.e., high discriminator accuracy). This indicates the skills reliably end in the same terminal state across multiple rollouts. However, as we mentioned in Appendix I, in a highly stochastic environment, it would not be straightforward to select the guide skill just by employing a random walk process. However, in a highly stochastic environment, the learned skills will visit different states for each rollout, which is a problem not only for DISCO-DANCE but for all current skill discovery algorithms. We believe this remains an important direction for future work.\n\n[1] First return, then explore. Ecoffet et al., Nature 2021."}]

Author rebuttal:
**Response to all reviewers**

 We deeply appreciate all five reviewers for their thoughtful feedback and valuable suggestions. R1, R2, R3, R4 and R5 indicate reviewer gakY, reviewer adLT, reviewer b566, reviewer EWQN, reviewer oWju respectively. 
 
 Reviewers identified the following strength in our submission:

- The main idea is intuitive and well-motivated (R2, R3, R4, R5).
- The proposed method was evaluated against several methods in various domains, and the empirical results are promising (R2, R4, R5).

 At the same time, reviewers identified the following weaknesses in our submission:

- Scalability of Random Walk Process to high dimensional state spaces (R1, R3).
- Missing baselines and missing experiments in manipulation environments (R1).
- Sample inefficiency of Random Walk Process (R4).


We hope our responses address all reviewer’s concerns, and we welcome any additional comments and clarifications.


**General Response: Random walk process in high-dimensional state space**

Reviewer R1 and R3 raised questions regarding the potential scalability of our random walk process within high-dimensional state spaces. To address this, we've identified two essential questions to consider:

1. When the agent starts to explore from an arbitrary terminal state, can the agent visit a diverse range of states through the random walk process?
2. Given the diverse range of visited states, is it possible to identify the terminal state (guide skill) within the least explored region? 


In response, we conducted a synthetic experiment on Montezuma's Revenge, which is a high-dimensional pixel-based environment characterized by an 84×84×3 input and is well-known for its exploration challenges.

Figure (a) in the attached PDF provides a visual illustration of our random walk approach within Montezuma's Revenge. In this experiment, we:

1. Randomly reset the agent in the inital room of Montezuma's Revenge.
2. Execute a random walk for 100 steps from the randomly initialized state.
3. Iterate the step 1,2 for N cycles.


This experimental design mirrors the algorithmic design of DISCO-DANCE where each reinitialized point indicates the terminal state of a different skill. Correspondingly, each skill undergoes a random walk for 100 steps, aligning with the parameters used for our paper (i.e., P=N, R=100, M=1 in Equation 4).

> 1. When the agent starts to explore from an arbitrary terminal state, can the agent visit a diverse range of states through the random walk process?

As illustrated by Figure (a), even in such a high-dimensional state space, the agent was able to visit a variety of states within just 100 random steps. For instance, for Skill 'I', the agent was able to move to another room. And with for Skill 'J', the agent successfully picked up the key. This experimental result supports the versatility of the random walk process, even when the environment is high-dimensional.


> 2. Given the diverse range of visited states, is it possible to identify the terminal state (guide skill) within the least explored region? 


After random walk, our aim shifts to identifying the guide skill within the least explored area. For density estimation, we employed the cell-centric technique from Go-Explore [1]: (i) segmenting the aggregated states into discrete cells, (ii) count the number of each cell's visitation, and (iii) select the least visited cell. As emphasized in Figure (a), the cell marked in red is selected as guide skill, indicating that skill 'I' is the prime candidate for guide skill in DISCO-DANCE.


This combined approach of exploration through random walk then identifying unique states (which is used in DISCO-DANCE), parallels the approach adopted by Go-Explore. This technique has consistently demonstrated its efficacy across varied domains, including Atari games and robotic settings.

In summary, we believe our synthetic experiment affirms the scalability of our random walk process in DISCO-DANCE, even within a high-dimensional pixel-based environment. In the revised manuscript, we'll be integrating these insights into the 'Limitations and Future Directions' section.

[1] First return, then explore. Ecoffet et al., Nature 2021.

Meta-review: The paper describes a method for unsupervised skill discovery based on a guidance principle, where a random walk process guides the discovery process towards new unexplored areas in the state space. Initial ratings were mixed (5,4,6,3,7) and evolved to (6,4,6,3,7). Reviewers appreciated an interesting idea (random walks to discover unexplored areas), a well written paper, 

Weaknesses raised were missing baselines, links to mutual information, sample efficiency, difficulty of the tasks (ab application to manipulation was requested).

The AC's own reading confirmed the strengths of the paper and some of weaknesses raised by the reviewers, but the AC did not side with all of them. In particular, the AC considered that the novelty and interestingness of the paper (guidance through random walk) outweights the issues, which are not important enough to prevent the paper from having a positive impact on the community. The AC recommends acceptance.

This paper was discussed between the AC and SAC.
Decision: Accept (poster)