SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This combined approach of exploration through random walk then identifying unique states (which is used in DISCO-DANCE), parallels the approach adopted by Go-Explore. This technique has consistently demonstrated its efficacy across varied domains, including Atari games and robotic settings. In summary, we believe our synthetic experiment affirms the scalability of our random walk process in DISCO-DANCE, even within a high-dimensional pixel-based environment.
2D maze The width and height of easy and medium level mazes are both 5, and 10 for hard level maze [7]. There is no explicit terminal state, and the episode ends only when the maximum timestep (30 for easy and medium levels and 50 for hard level) is reached. The state space and action space are both 2-dimensional (state: x, y coordinates, action space: distance to move in x, y coordinates).
Ant maze The width and height of U-maze are (7,7) and Empty,  $\Pi$ -shaped maze are (8,8) respectively [8, 30]. Similar to 2d-maze, episode ends only when the maximum timestep (400 for every mazes) is reached. The state space is 31-dimensional, and the action space is 8-dimensional.
Deepmind Control Suite We conduct our environment in a total of eight downstream tasks provided by URLs [22]. In detail, we utilize four tasks each in two continuous control domains: Run, Run backward, Flip, and Flip backward for Cheetah and Jump, Run, Stand, and Walk for Quadruped. For cheetah, the state space is 17-dimensional, and the action space is 6-dimensional. For Quadruped, the state space is 78-dimensional, and the action space is 12-dimensional.
D.2 Evaluation
State Coverage (2D, Ant mazes) We discretize the  $x$  and  $y$  axes of the environment into 10 intervals (i.e., total  $10 \times 10$  buckets) and count the number of buckets reached by learned skills.
Downstream Task Performance For Ant mazes, we conduct goal-reaching downstream tasks. We design a distance between the current state and the goal state as a penalizing reward (i.e.,  $-1 \times$  distance = reward). In detail, directly computing distance with x and y coordinates might be inaccurate because there can be obstacles (i.e., walls) between two states. Therefore, we design a reward function that returns the actual distance that goes around when there is an obstacle between two states. Then, we normalize the reward to set the lowest value as  $-1$ . For Ant mazes and DMC, we first perform pretraining for 5M(Ant mazes) / 2M(DMC) training steps and select the skill with the highest return for each task. Then we finetune for 1M(Ant mazes) / 100k(DMC) steps.
For DIAYN, SMM, DISDAIN, and DISCO-DANCE (discrete skills), we finetune the skill with maximum downstream task reward. For APS (continuous skill), we first randomly select an arbitrary skill  $z$  and rollout episodes, then solve the linear regression problem to select the task-specific skill  $z$ .
(following the same protocol in the APS paper). After the skill is selected for finetuning, we initialize the agent with the pre-trained network parameters. Utilizing pretrained parameters for the entire parameters of the critic can impede the rapid adaptation process, as the pretrained network is already optimized for the intrinsic reward. To mitigate this issue, we initialize only the encoder of the critic network with pretrained parameters (i.e., excluding the last fully connected layer). In addition, we initialize the actor network with entire pretrained parameters (including the last fully connected layer).
Note that our evaluation on the downstream task is different from URLB in that the skill selection process is not included in 100K, but this is still a fair comparison as all algorithms use the same number of interactions (samples) to select skills. For UPSIDE, (i) first we select the leaf node skill with maximum downstream task rewards, (ii) then freeze all the pretrained ancestors of that selected leaf node skills and (iii) only finetune the leaf node skill, same as the original paper. A detailed explanation on the fine-tuning procedure of UPSIDE is described in Appendix F
Detailed hyperparameters of our method DISCO-DANCE are listed in the table 4. Note that the additional steps used to perform the random walk process have been included in the overall count of environment steps for training.
Table 4: A common set of hyperparameters used in DISCO-DANCE.
Hyperparameter	Value
Replay buffer size	10^6
Optimizer	Adam
Learning rate	3 × 10^-4
RL algorithm	Soft Actor Critic
discount γ	0.99
Select guide	K nearest neighbor
Guide coef (α)	10^-4
Extending threshold ρ	5
Accuracy threshold ε	0.5
Table 5: Per environment sets of hyperparameters used in DISCO-DANCE.
Hyperparameter	2D maze	Ant maze	DMC
Hidden dim	128	1024	1024
Batch size	64	512	1024
Skill trajectory length	easy, normal: 30 hard: 50	400	1000
Initial number of skill	30	30	10
Entropy coef (β)	0.2	0.6	0.2
Number pre-training frames	easy, normal: 2 × 106hard: 5 × 106	5 × 106	2 × 106
For DISDAIN, we search the number of ensembles from [2, 10, 20, 40] and disdain reward coefficient from [10, 50, 100]. For 2D maze, We train DISDAIN using 40 sizes of ensembles and set DISDAIN reward weight as 100. For Antmaze, we set 20 for the number of ensembles and 100 for the reward coefficient. For DMC, we utilize 2 for the number of ensembles and 10 for the reward coefficient.
For APS, we search the value of  $\mathrm{k}$  (i.e., the number of neighbors in  $\mathrm{k}$  nearest neighbor algorithm) from [5, 10, 12] and the dimension of skills from [5, 10]. For 2D maze, we utilize 10 for  $\mathrm{k}$  and 5 for skill dimensions. For Antmaze, we utilize 5 for  $\mathrm{k}$  and 5 for skill dimensions. For DMC, we utilize 12 for  $\mathrm{k}$  and 10 for skill dimensions.
For SMM, we search the entropy coefficient from [0.0005, 0.005, 0.01, 0.1, 0.25, 0.5, 1]. We utilize the entropy coefficient as 0.1 for 2D maze and Antmaze, and 0.01 for DMC.
For UPSIDE, there are five kinds of hyperparameters: maximum timestep  $T$ , diffusing timestep  $H$ , discriminator threshold  $\eta$ , initial extend skill number  $N_{start}$ , and maximum extend skill number  $N_{max}$ . We set  $T = H = 5$  for 2D maze,  $T = H = 50$  for Antmaze and  $T = H = 125$  for DMC. In addition, we utilize  $\eta = 0.5$ ,  $N_{start} = 2$ , and  $N_{max} = 8$  for all environments.
We use SAC as the backbone RL algorithm [15]. We train DIAYN, APS, and SMM using the code provided by URLB [22]. In addition, we re-implement DISDAIN and UPSIDE by strictly following the details of the paper.
We use a fixed number of skills for the baselines (i.e., DIAYN, DISDAIN, and SMM), which utilize discrete skill spaces. We set 50 for easy and normal level 2D mazes, 70 for the hard level 2D maze, and 100 for Antmaze and DMC.
D.4 Resource Requirements
Full experimental training runs 12 hours for the 2D maze and 24 hours for Antmaze and DMC. We conducted all experiments using a single RTX 3090 GPU, and each experiment requires approximately up to 2GB of memory.
D.5 License
- Environment
- 2D mazes: We used the publicly available environment code in [7]².
- Ant mazes: We used the publicly available environment code in  $[8]^3$ .
- Deepmind Control suite: We used the publicly available environment code in [22, 53] $^4$ .
- Baseline algorithm
- For DIAYN, APS and SMM, we used the code provided in [22].
E.1 Additional fine-tuning results on the Ant U-maze
To further evaluate whether the learned skills can be useful for downstream tasks, we conducted goal-reaching navigation experiment in Ant U-maze, wherein the goal state was designated as the region positioned furthest from the initial state. The quantification of successful goal attainment was measured by determining the number of seeds, out of a total of 20, in which the agents reached the goal state. We select the skill with the maximum return (i.e., skill whose state space is closest to the goal state) from the set of pretrained skills, and finetune the policy. Notably, DISCO-DANCE reached the goal in the Ant U maze faster than the other algorithms (as indicated by the earliest point where the y-axis reaches 20).
Fig. 10 shows that DISCO-DANCE consistently reaches the goal state (for all 20 seeds) with the fewest number of environmental steps. As shown in the Ant II-maze finetuning experiment (Section 4), the performance of
UPSIDE is significantly inferior to the performance of DISCO-DANCE, with only 5 seeds out of 20 seeds reaching the goal state. These results suggest that the tree-structured policies during the pretraining stage hinders the agent's ability to learn optimal policies during the fine-tuning phase.
Figure 10: Finetuning results on the Ant U-maze. We plot the number of successful seeds out of total 20 seeds.
$^{2}$ https://github.com/victorcampos7/edl
<sup>3</sup>https://github.com/elliotchanesane31/RIS
<sup>4</sup>https://github.com/rl1-research/url_benchmark
E.2 Full Results on Deepmind Control Suite Tasks
Table 6: Performance comparison of DISCO-DANCE and baselines on DMC. Bold scores indicate the best model performance and underlined scores indicate the second best. The results are averaged over 15 random seeds with a standard deviation.
Models	Cheetah	Quadruped	Avg
Run	Run Backward	Flip	Flip Backward	Jump	Run	Stand	Walk
SCRATCH	411.25±58.24	386.20±16.35	704.54±16.84	714.10±8.62	345.05±196.17	140.64±80.13	552.07±224.75	119.90±171.63	421.71
DIAYN	532.63±143.12	450.35±8.90	571.16±164.78	717.72±20.01	629.26±217.02	414.89±230.47	743.53±257.33	367.65±344.14	553.40
SMM	580.79±54.92	436.35±13.04	668.12±169.16	733.79±16.86	460.38±245.73	342.51±214.29	434.17±266.74	511.40±330.69	520.94
APS	574.00±56.47	437.66±16.96	640.51±80.81	684.86±48.82	572.32±247.61	337.20±217.22	640.51±279.73	366.45±305.55	531.69
DISDAIN	586.44±59.24	445.85±15.87	673.54±89.86	731.95±8.24	476.16±200.91	246.45±124.32	725.42±200.08	347.24±179.32	529.13
UPSIDE	309.81±121.52	259.17±94.74	607.96±117.82	622.84±78.48	495.39±269.78	248.32±198.72	635.24±312.12	359.62±307.36	442.29
DISCO-DANCE	602.74±36.64	448.50±12.93	703.80±60.44	705.23±56.96	693.23±249.01	477.41±239.69	941.94±125.25	664.66±346.16	654.69
Table 6 summarizes the full results for each task in DMC. We can observe that DISCO-DANCE gets first or second performance in seven out of eight tasks. The performance of the USD baseline was superior to the scratch, demonstrating that previously learned behavior can be effectively utilized when solving downstream tasks. DISCO-DANCE outperforms other baselines in terms of average performance, and this indicates that DISCO-DANCE learned more general behaviors through guidance.
E.3 Additional Ablation studies
(a) Apprentice skill initialization
(c) Density estimation
(d) Random walk steps
(e) Threshold sensitivity
Figure 11: Additional ablation studies. (a) Apprentice skills initialization. (b) A number of K in K-nearest neighbors. (c) Density estimation methods. (d) A number of random walk steps. (e) Sensitivity to the accuracy threshold.
First, as we incrementally introduce new skills, we employ the guide skill to initialize the weights of the apprentice skills, allowing for direct access to unexplored states. To assess the significance of this initialization method, we compare it to a control condition in which the weights are trained from scratch. As shown in Fig 11(a), utilizing the guide skill for initialization leads to improved performance. We speculate that guide initialization directly minimizes the KL divergence with guide skill, thereby maximizing the guide reward at the onset of training.
Nest, we examined the sensitivity of the results regarding the value of  $\mathrm{k}$  in  $\mathrm{k}$ -nearest neighbors. As observed in our experiments (Fig. 11b), there were no significant differences in state coverage. This is because the most exploratory skill already has its terminal state far enough from the terminal states
of other skills. Therefore, adjusting the value of  $\mathrm{k}$  higher or lower does not significantly impact the selection of guide skills. This demonstrates the robustness of DISCO-DANCE with respect to the hyperparameter  $\mathrm{k}$ .
Additionally, to analyze how the DISCO-DANCE performs with a different measure of distance/density, we employed kernel density estimation (KDE) for selecting the guide skill. Specifically, we (i) performed random walks (as in our random walk guide selection process), (ii) trained a KDE model on the random walk arrival states, (iii) selected the state in the lowest density region among the collected random walk arrival states, and (iv) chose the skill from which that state originated as the guide skill. As shown in Fig. 11c, we found that the performance of both methods was nearly indistinguishable. This is likely due to the terminal state of the most exploratory skill being sufficiently distant from the terminal states of other skills, as was observed in the knn ablation study.
In addition, we conducted additional experiments to investigate the effect of varying the length of the random walk steps. Since the environment steps used during the random walk process are included in our total environment steps (total environment steps = general trajectory collection steps + random walk process steps), increasing the number of random walk steps leads to a decrease in the number of states utilized for model updates. As a result, we observed a slightly lower performance in the initial environment steps when comparing the result of 0.1T and 0.5T. However, we found that the models converge after 3M steps, with smaller performance differences between them.
To further examine the sensitivity of the accuracy threshold  $\epsilon$ , we conduct an ablation study by varying  $\epsilon$  from [0.3, 0.5, 0.7]. As illustrated in Fig. 11e, the performance of our proposed method is found to be robust to changes in the accuracy threshold  $\epsilon$ .
E.4 Comparing curriculum approach effects on other USD methods
Figure 12: Gradually increasing the number of skills for DISCO-DANCE, DIAYN, and DIS-DAIN. (a) Qualitative results on bottleneck maze and Ant II-maze. Colored skills indicate new skills and grey skills are previously converged skills. (b) Training curves on each environment.
To demonstrate the compatibility of our proposed method DISCO-DANCE with increasing the number of skills during training, we also incorporate a curriculum approach into the baselines which utilize discrete skill spaces. As depicted in Fig. 12a, we observe that DIAYN and DISDAIN do not benefit from curriculum learning, as the newly added skills primarily remain in proximity to the initial starting point. This results in a more congested covered state space, thus the curriculum approach fails to increase state coverage and instead decreases discriminator accuracy. In contrast, for our proposed method DISCO-DANCE, the newly added skills are able to access unexplored regions with the assistance of the guide skill.
F Comparison with UPSIDE
DISCO-DANCE and UPSIDE [19] have similar motivation, alleviating the inherent pessimism problem. Both of their strategies can be summarized as "agent guides itself" (i.e., existing skills
help new/unconverged skills to explore). In DISCO-DANCE, guide skill encourages apprentice skills (unconverged skills) to explore unexplored states by minimizing the KL divergence between guide skill and apprentice skills. While DISCO-DANCE learns a set of single skills, UPSIDE learns tree-structured policy which is composed of multiple skill segments.
Optimal trajectory Fine-tuned trajectory
(a) DISCO-DANCE:
Finetune skill  $z = 8$
Figure 13: Conceptual illustration of (1) learned skills of DISCO-DANCE and UPSIDE, and (2) goal-reaching downstream task. The black edge represents skill, and the black circle represents the terminal states of its skill. (a) After unsupervised pretraining, DISCO-DANCE obtains a set of skills that can be used individually without the need for each skill to be used together. In the finetuning stage, DISCO-DANCE selects skill  $z = 8$  (i.e., skill with the highest downstream task reward) and finetunes the skill to reach the goal quickly. (b) However, to execute  $z = 8$ , UPSIDE requires sequential execution of all its ancestors' skills (i.e., executing  $z = 1, 4, 8$  sequentially). In the finetuning stage, UPSIDE is not able to learn optimal policy since the ancestors' skills are kept fixed during finetuning. (c) If we finetune ancestors' skills simultaneously, the tree structure learned in the pretraining phase will be broken, which makes finetuning unstable and difficult.
(b)UPSIDE:
Finetune only leaf skill  $z = 8$
(c)UPSIDE:
Finetune  $z = 1,4,8$  simultaneously
Fig 13 illustrates an example where a total of 8 skills are learned. When learning the policy in the unsupervised pretraining stage, UPSIDE (1) selects the skill with the largest discriminator accuracy among the leaf node skills (e.g.,  $z = 4$ ), (2) adds new skills as its children nodes (e.g.,  $z = 7, 8$ ), (3) freeze the parent skills (e.g.,  $z = 1, 4$  are not trained) and only train newly added skills, and (4) iteratively repeat these procedures to construct the tree-structured policies (Fig 13(b)). This makes the fundamental difference between DISCO-DANCE and UPSIDE, that UPSIDE requires sequential execution from ancestors' skills to child skill in a top-down manner due to its tree-structured skill policies. For example, if we want to run skill  $z = 8$ , UPSIDE needs to execute skills sequentially ( $z = 1 \rightarrow 4 \rightarrow 8$ ) for  $T$  timesteps, where  $T$  is a hyperparameter that decides how dense the generated tree will be.
One of the key objectives of unsupervised skill discovery is to utilize the learned skills as a useful primitive at the fine-tuning stage. However, these sequential executions of UPSIDE bring significant inefficiency at the finetuning stage due to the following reasons. Suppose the given downstream task is to reach the goal state as fast as possible, where the distance from the goal state and total execution time are given as a penalty (Fig 13). Since  $z = 8$  is the skill with minimum distance from the goal, DISCO-DANCE selects skill  $z = 8$  and finetunes  $\pi(a|s, z = 8)$  to optimize given reward functions (Fig 13(a)). However, for UPSIDE to finetune  $z = 8$ , we need to finetune  $z = 1, 4, 8$  simultaneously (Fig 13(c)). In that case, the tree-structured skill policies learned in the pretraining stage are broken during the finetuning stage. That is, the dictionary  $\{ "z = 8" : [1, 4, 8] \}$  cannot be used anymore because executing  $z = 1$  for  $T$  timesteps does not move the agent to the original  $z = 1$  terminal nodes (red × in Fig 13(c)). This means that the skill tree learned in pretraining can no longer be used, leading to inefficient finetuning.
To avoid this limitation, UPSIDE only finetuned the leaf skill in the original paper (i.e., freeze ancestors  $z = 1,4$  and only finetunes  $z = 8$  in Fig 13(b)). However, this would still be ineffective because the fixed ancestors  $z = 1,4$  are not the optimal solution to solve the given downstream task (green lines from <start> to  $z = 4$  nodes in Fig 13(b)). The problem becomes more serious in a long-horizon environment such as DMC (where the horizon is 1000) if we freeze all the pretrained
ancestors' skills and finetune only the last leaf skill. In contrast, DISCO-DANCE selects a single skill to fine-tune, which can be executed independently, so it does not suffer from the above problem.
There are further differences between DISCO-DANCE and UPSIDE in the pretraining phase in addition to the differences in the finetuning stage. In the pretraining stage, both algorithms leverage previously discovered skills for enhanced exploration. st, UPSIDE chooses one of the leaf nodes as the parent node, which additional new skill nodes will be added as children to that node (e.g., select  $z = 4$  as a parent and add  $z = 7,8$  as children nodes in Fig. 13). Then  $z = 7,8$  explores the state region over the state space occupied with previous skills (e.g.,  $z = 1,4$ ). On the other hand, DISCO-DANCE selects a guide skill from among the existing skills, and the KL divergence between the guide skill and the apprentice skill is used as a reward signal. The main difference is that UPSIDE selects the parent skill with the highest discriminator accuracy (which corresponds to Fig.2b in main paper) and DISCO-DANCE utilizes a proposed random walk guide selection process (Fig.2c in main paper). As the most distant skill is typically the one with the highest discriminator accuracy (because the discriminator can easily classify the distant skill), the selection strategy utilized by UPSIDE is less effective in exploration, as we demonstrate in Table 2 and Fig.8 (main paper).
Lastly, both DISCO-DANCE and UPSIDE employ random walks (i.e., the diffusing component in UPSIDE) during the pretraining phase. However, the utilization of random walks serves a completely different role in each approach. In DISCO-DANCE, the random walk process (random walk + finding lowest density region via knn) is only utilized to select the guide skill, which is orthogonal to the policy  $\pi_{\theta}$  learning. In contrast, UPSIDE produces rewards from the random walks. Specifically, UPSIDE (1) performs random walks at each skill's terminal nodes and (2) updates the policy  $\pi_{\theta}$  and discriminator  $q_{\phi}$  with random walked states (i.e., gets more reward if their random walked states are discriminable from other skill's random walked states). This process induces the local coverage of each skill (i.e., each skill occupies the random walked states, where other skills cannot come in).
G Limitations and Future Directions
DISCO-DANCE may cause cost (sample) inefficiency when measuring the density of the state distribution in environment with high dimensional input, such as control from pixel observations. In this case, we could utilize the downscaling technique (e.g., cell representation in Go-Explore) to reduce computational costs.
For the guide skill selection process to work stably, the termination state should be reliably reached. Therefore, when the stochasticity of the environment is too large, it would not be easy to select a guide skill only with the simple random walk process we proposed. We leave it as future work for alleviating such difficulties.
Unsupervised skill discovery (USD) promotes efficient learning in complex environments, allowing agents to learn without the need for explicitly labeled data. This technique has real-world applications ranging from robotic manipulation to autonomous vehicles and can also improve the AI's ability to generalize its skills to new situations. However, one of the most significant problems of USD is its sample inefficiency due to the necessity of running extensive simulations. Research like DISCO-DANCE can substantially enhance the sample efficiency of USD.
I Visualization of learned skills
I.1 2D mazes
diayn
upside
DISCO-DANCE
…→
(a) Empty maze
smm
Figure 14: Qualitative visualization of the learned skills on Emtpy maze and Y maze. Visualization of multiple rollouts of learned skills by baseline models. For DISCO-DANCE, we visualize our guiding procedure during training. Bold red lines indicate the guide skill.
DISCO-DANCE
(b) Y maze
(a) W maze
(b) S maze
Figure 15: Qualitative visualization of the learned skills on W maze and S maze. Visualization of multiple rollouts of learned skills by baseline models. For DISCO-DANCE, we visualize our guiding procedure during training. Bold red lines indicate the guide skill.
I.2 Ant Mazes
(a) Ant Empty-maze
(b) Ant U-maze
(c) Ant  $\Pi$  -maze
Figure 16: Qualitative visualization of the learned skills on Ant Empty-maze, Ant U-maze, and Ant II-maze. Visualization of multiple rollouts of learned skills by baseline models. For DISCO-DANCE, we visualize our guiding procedure during training. Bold red lines indicate the guide skill.
I.3 Deepmind Control Suite
Figure 17: Qualitative visualization of the learned skills in the cheetah in DMC. DISCO-DANCE learned 100 skills without a reward function. In order to facilitate visualization, the skills are selected based on their corresponding terminal states.