SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fig. 5 illustrates multiple rollouts of various skills learned in the 2D bottleneck maze. For the 2D bottleneck maze, the upper left room is the most difficult region to reach since the agents need to pass multiple narrow pathways. Fig. 5 shows that only DISCO-DANCE and UPSIDE effectively explore the upper left region, with DISCO-DANCE displaying a relatively denser coverage on the upper left state space. More qualitative results are available in Appendix I.
4.2.2 Ant mazes
To evaluate the effectiveness of DISCO-DANCE in the environment with high dimensional state and action space, we train DISCO-DANCE in three Ant mazes where the state space consists of joint angles, velocities, and the center of mass, and the action space consists of torques of each joint [8, 30].
Table 1(b) reports the state coverage of DISCO-DANCE and baselines. DISCO-DANCE shows superior performance to the baselines where it achieves the best state-coverage performance in high-dimensional environments that contain obstacles (i.e., Ant U-maze and II-maze) and gets competitive results against SMM with a marginal performance gap in the environment without any obstacle (i.e., Ant Empty-maze). These results demonstrate the effectiveness of our proposed direct guidance in navigating an environment with obstacles. Additional qualitative results of each algorithm can be found in the Appendix I.
To further assess whether the learned skills could be a good starting point for downstream tasks, we conduct goal-reaching navigation experiments where the goal state is set to the region farthest from the initial state for the more challenging navigation environment, Ant II-maze. We measure the number of seeds that successfully reach the established goal state (out of a total 20 seeds). We select the skill with the maximum return (i.e., a skill whose state space is closest to the goal state) from the set of pretrained skills and finetune the policy  $\pi_{\theta}(a|s,$  selected skill  $z)$ . Additional experiments in Ant U-maze and details are available in Appendix D, E.
Fig. 6 shows that DISCO-DANCE outperforms prior baseline methods in terms of sample efficiency. Specifically, DISCO-DANCE is the only algorithm that successfully reaches the goal state for all seeds (20 seeds) in Ant II-
Figure 6: Finetuning results on the Ant II-maze. We plot the number of successful seeds out of total 20 seeds.
maze. In addition, UPSIDE fails to reach the goal for all seeds, which indicates that the tree-structured policies learned during the pretraining stage can impede the agent from learning optimal policies for fine-tuning.
Figure 7: Performance comparisons on Deepmind Control Suite. Aggregate statistics and performance profiles with  $95\%$  bootstrap confidence interval are provided [2], which are calculated across 120 seeds (15 seeds  $\times 8$  tasks).
Expert Normalized Score
4.3 Deepmind Control Suite
While the experiments in 2D and Ant mazes demonstrate that the skills DISCO-DANCE learns can be utilized as useful primitives for navigation tasks, to demonstrate the versatility and effectiveness of DISCO-DANCE in acquiring general skills (e.g., run, jump, flip), we conduct additional experiments on the DMC benchmark [53]. Specifically, we consider two different environments in DMC, Cheetah, and Quadruped, and evaluate DISCO-DANCE across four tasks (e.g., run, flip, and jump) within each environment (total of 8 tasks). The DMC benchmark evaluates the diversity of the learned skills, meaning that agents must learn a suitable set of skills for all downstream tasks in order to achieve consistently high scores across all tasks. We first perform pretraining for 2M training steps and select the skill with the highest return for each task (same as the Ant maze experiment). Then we finetune USD algorithms for 100k steps. More experimental details are available at Appendix D.
Fig. 7 shows the performance comparisons on DMC across 8 tasks with 15 seeds. First, all USD methods perform better than training from scratch (i.e., SCRATCH), indicating that pretraining with USD is also beneficial for continuous control tasks. Furthermore, as we described in Fig. 3b, we find that our proposed guidance method boosts the skill learning process for the continuous control environment, resulting in DISCO-DANCE outperforming all other baselines. As shown in Ant mazes finetuning experiments, UPSIDE significantly underperforms compared to other auxiliary reward-based USD baselines. Since UPSIDE necessitates the sequential execution of all its ancestor skills, its finetuning process becomes less efficient than other methods.
In addition, baselines with auxiliary rewards, which enforce exploration, are ineffective in DMC. We attribute these results to their training inefficiency, that the current experimental protocol of pretraining for 2M frames is insufficient for them to converge in a meaningful manner. For example, DISDAIN requires billions of environment steps in the grid world environment in the original paper. In contrast, DISCO-DANCE successfully converges within 2M frames and outperforms all other baselines. We provide the full performance table and visualizations of skills in Appendix E, I.
4.4.1 Model Components
We further conduct an ablation analysis to show that each of our model components is critical for achieving strong performance. Specifically, we experiment with two variants of DISCO-DANCE, each of which is trained (i) without guide coefficient (i.e., guide coefficient as 1) and (ii) without the random walk based guide selection process in Table 2.
Table 2: Ablation study on model components.
Models	Bottleneck maze
DISCO-DANCE	86.30±17.01
DISCO-DANCE w/o guide coef	76.75±13.05
DISCO-DANCE w/o random walk	70.30±10.72
DIAYN (no exploration reward)	52.40±3.77
First, without guide coefficient  $\mathbb{I}\big(q_{\phi}(z^i |s)\leq \epsilon \big)(1 - q_{\phi}(z^i |s))$  , DISCO-DANCE sets all other skills as apprentice skills, causing all other skills to move towards the guide skill. As shown in Table 2, this method of guiding too many skills degrades performance, and it is important to only select unconverged skills as apprentice skills for effective learning.
Second, Table 2 also shows that random-walk based guide skill selection (i.e., selecting the skill with the lowest density region among PRM terminal states in Eq. 4) is superior to naively selecting the most distant skill without random walk process (i.e., selecting the skill with the lowest density region among P terminal states in Eq. 4). We also provide qualitative results (Fig. 8) where selecting the most distant skill as the guide skill fails to select the skill closest to the unexplored state region since the blue skill is the most distant from the terminal states of the other skills. On the other hand, random walk guide selection successfully finds the appropriate guide skill.
Figure 8: Qualitative results for different guide skill selection processes.
4.4.2 Compatibility with other skill discovery method
In this paper, our primary goal is to compare DISCO-DANCE with algorithms that focus on constructing an effective auxiliary exploration reward, denoted as  $r_{\text{exploration}}$ . However, several research studies exist that seek to refine  $r_{\text{skill}}$  to enhance exploration [32, 20, 27] (detailed explanation in Section 5). To evaluate how effectively DISCO-DANCE can enhance exploration when aligned with different  $r_{\text{skill}}$ , we carried out a comparative analysis with LSD [32]. The recently introduced skill discovery algorithm, LSD, has demonstrated commendable outcomes, especially in Ant Mazes.
Table 3: Ablation study on  $r_{\mathrm{skill}}$
\(r_{skill}\)	\(r_{exploration}\)	Ant II-maze
DIAYN	- \(r_{guide}\)	22.50 ±3.34 39.00 ±4.85
LSD	- \(r_{guide}\)	38.80 ±3.34 45.80 ±3.34
Table 3 illustrates the performance comparison with LSD. Our findings reveal that LSD, even in the absence of  $r_{\text{exploration}}$ , exhibits superior performance compared to DIAYN in Ant-Ⅱ maze. Such an observation possibly highlights LSD's proficiency in mitigating the inherent pessimism linked with the previous mutual information objective. An observation worth highlighting is the substantial enhancement in performance for both algorithms when integrated with our  $r_{\text{guide}}$ . We propose that  $r_{\text{skill}}$  and  $r_{\text{guide}}$  have distinct roles, and when combined, they can collectively boost their overall performance.
Pretraining methods in RL Pretraining methods for RL primarily fall into two categories [55]. The first is online pretraining, where agent interact with the environment in the absence of any reward signals. This method generally operates under the presumption that agents can engage with the environment at a minimal cost. The primary objective of online pretraining is to acquire essential prior knowledge through unsupervised interactions with the environment. Recently, there has been a lot of research in this area, often referred to as "Unsupervised RL". Such methodologies typically employ intrinsic rewards, to learn representations or skills which will be useful for downstream tasks. The acquired pretrained weights, such as encoder, actor, and critic, are subsequently reused for downstream tasks.
Depending on how the intrinsic reward is modeled, online pretraining can be further categorized. Curiosity-driven approaches focus on probing states of interest, often characterized by high predictive errors, in order to obtain more environmental knowledge and reduce prediction errors across states [6, 35, 43, 34]. Data coverage maximization methods aim to maximizes the diversity of the data the policy has collected, for instance, by amplifying the entropy of state visitation distributions [5, 18, 24, 26, 57, 44]. Meanwhile, Unsupervised Skill Discovery involves learning policies, denoted as  $\pi(a|s,z)$ , guided by a distinct skill vector  $z$ . This vector can then be adjusted or combined to address subsequent tasks [13, 11, 17, 47, 7, 25, 19, 32, 1, 60, 39, 33].
The second category is offline pretraining. This method leverages an unlabeled offline dataset accumulated through various policies, to learn the representation. The learning objective in offline pretraining involves pixel or latent space reconstruction [58, 59, 61, 54], future state prediction [41, 42, 45, 23], learning useful skills from data [37, 3, 50, 36, 16, 56] and contrastive learning focusing on either instance [21, 12] or temporal discrimination [31, 51].
In this paper, our primary emphasis is on Unsupervised Skill Discovery within the domain of online pretraining. For a comprehensive overview of pretraining methods for RL, we direct the reader to [22, 55].
Unsupervised skill discovery. There have been numerous attempts to learn a set of useful skills for RL agents in a fully unsupervised manner. These approaches can be broadly classified into two categories: (i) one studies how to learn the skill representations in an unsupervised manner (i.e., how to make  $r_{\mathrm{skill}}$  in Eq. 5) and (ii) the other focuses on devising an effective auxiliary reward which encourages the exploration of skills to address the inherent pessimistic exploration problem (i.e., how to make  $r_{\mathrm{exploration}}$  in Eq. 5).
Research that falls under the first category focuses on how to train skills that will be useful for unseen tasks. DIAYN [11], VISR [17], EDL [7] and DISk [46] maximizes the mutual information (MI) between skill and state reached using that skills. DADS [47] utilizes a model-based RL approach to maximize conditional MI between skill and state given the previous state. LSD [32], which is non-MI-based algorithm, provides a 1-Lipschitz constraint on the representation of skills to learn dynamic skills. CIC [20] replace  $r_{\mathrm{skill}}$  with a contrastive loss to learn high-dimensional latent skill representations, and additionally utilizes  $r_{\mathrm{exploration}}^{\mathrm{APS}}$  (Eq. 3) for exploration. Choreographer [27] learns a world model to produce imaginary trajectories (which are used for learning skill representations) and additionally utilizes  $r_{\mathrm{exploration}}^{\mathrm{APS}}$  for exploration.
DISCO-DANCE belongs to the second category that focuses on devising exploration strategy for USD. This category includes DISDAIN [52], UPSIDE [19], APS [25] and SMM [24], which are included as our baseline methods. Detailed explanation and limitations of these methods are described in Section 2.2, 4.1, Fig. 1, and Appendix F.
Guidance based exploration in RL. Go-Explore [9, 10] stores visited states in a buffer and starts re-exploration by sampling state trajectory, which is the most novel (i.e., the least visited) in the buffer. DISCO-DANCE and Go-Explore share a similar motivation that the agent guides itself: DISCO-DANCE learns from other skills and Go-Explore learns from previous trajectories. Another line of guided exploration is to utilize a KL-regularization between the policy and the demonstration or sub-policy (i.e., guide) [37, 38, 8, 29, 48]. SPIRL [37], an offline skill discovery method, minimizes the KL divergence between the skill policy and the learned prior from offline data. RIS [8] has been proposed for efficient exploration in goal-conditioned RL, by minimizing KL divergence between its policy and generated subgoals.
We introduce DISCO-DANCE, a novel, efficient exploration strategy for USD. It directly guides the skills towards the unexplored states by forcing them to follow the guide skill. We provide quantitative and qualitative experimental results that demonstrate that DISCO-DANCE outperforms existing methods in two navigation and a continuous control benchmark.
In this paper, in order to evaluate solely the effectiveness of  $r_{\text{exploration}}$  methods, we fix the backbone  $r_{\text{skill}}$  as DIAYN. Combining DISCO-DANCE with other recent  $r_{\text{skill}}$  methods (e.g., CIC) is left as an interesting future work. Also, we think that there is still enough room for improvement in finetuning strategies [22]. Since USD learns many different task-agnostic behaviors, a new fine-tuning strategy that can take advantage of these points would make the downstream task performances more powerful. A discussion of limitations is available in Appendix G.
This work was supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2022R1A2B5B02001913), and the National Supercomputing Center with supercomputing resources including technical support (KSC-2023-CRE-0074).
[58] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proc. the AAAI Conference on Artificial Intelligence (AAAI), volume 35, pages 10674–10681, 2021.
[59] Tao Yu, Zhizheng Zhang, Cuiling Lan, Yan Lu, and Zhibo Chen. Mask-based latent reconstruction for reinforcement learning. Proc. the Advances in Neural Information Processing Systems (NeurIPS), 35:25117-25131, 2022.
[60] Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options. arXiv preprint arXiv:2101.06521, 2021.
[61] Jinhua Zhu, Yingce Xia, Lijun Wu, Jiajun Deng, Wengang Zhou, Tao Qin, Tie-Yan Liu, and Houqiang Li. Masked contrastive representation learning for reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3421-3433, 2022.
A Implementation Details of DISCO-DANCE 15
B Efficient Random Walk Process 16
C Random Walk Process in High-dimensional State Space 17
D Experimental Details 18
D.1 Environments 18
D.2 Evaluation 18
D.3 Hyperparameters 19
D.4 Resource Requirements 20
D.5 License 20
E Additional Experimental Results 20
E.1 Additional fine-tuning results on the Ant U-maze 20
E.2 Full Results on Deepmind Control Suite Tasks 21
E.3 Additional Ablation studies 21
E.4 Comparing curriculum approach effects on other USD methods 22
F Comparison with UPSIDE 22
G Limitations and Future Directions 24
H Broader Impact 24
I Visualization of learned skills 25
I.1 2D mazes 25
I.2 Ant Mazes 27
I.3 Deepmind Control Suite 28
A Implementation Details of DISCO-DANCE
For reproducibility, we (i) provide pseudo code in Algorithm 1 and (ii) open-source code at https://mynsng.github.io/discodance/. Below, We describe further details regarding the implementation of DISCO-DANCE.
- (line 1 in Algorithm 1) When extending new skills, we initialize their policy weights with the guide skill policy weights rather than random weights as in SPIRL [37]. This accelerates exploration by encouraging the newly added skills to directly reach the unexplored regions. In Appendix E.3, we conduct ablation studies of this component.
- (line 4 in Algorithm 1) As the number of skills is gradually increasing, the discriminator struggles to learn due to the sparse update for each skill by the decreased probability of sampling each skill. We mitigate this issue by utilizing a skill sampling scheme where each skill sampling weight is proportionate to its discriminator error (e.g., skills with high discriminator error are more likely to be sampled). This approach is analogous to the methodology adopted in Prioritized Experience Replay [40].
- (line 5-6 in Algorithm 1) For adding new skills, we need to define hyperparameters: (1) when to extend skills and (2) how many to increase. We gradually extend skills as the discriminator converges. In detail, count the number of skills that have low discriminator accuracy (i.e., less than  $\epsilon$ ), and if the number is less than the threshold (i.e.,  $\rho$ ), extend  $\rho$  skills.
- (line 8-9 in Algorithm 1) We utilize a KL divergence between the apprentice skills and the guide skill policy as  $r_{\mathrm{guide}}$ . However, as the guide skill policy changes during training, it can lead to instability in the training of the apprentice skills. To address this issue, we use a separate target policy network for the guide skill. More precisely, when we select the guide skill, we copy and fix the policy network (denoted as target policy network  $\hat{\theta}$ ) and calculate the KL divergence reward utilizing this target policy network (i.e., calculating  $r_{\mathrm{guide}}$  as  $-\mathbb{I}\big(q_{\phi}(z^i |s) < \epsilon \big)(1 - q_{\phi}(z^i |s))D_{\mathrm{KL}}(\pi_{\theta}(a_t|s_t,z^i)||\pi_{\hat{\theta}}(a_t|s_t,z^*))$ ).
- (line 13-14 in Algorithm 1) Entropy term in SAC can be seen as KL divergence between the policy and uniform distribution with a constant [37]  $(\mathcal{H}(\pi (\cdot |s_t,z_t))\propto -D_{KL}(\pi (\cdot |s_t,z_t)||U(\cdot)))$  . Instead of replacing the entropy term with guide term in DISCO-DANCE, we found that using both term simultaneously helped stabilize learning by enjoying the advantage of maximum entropy RL (i.e., RL agent maximizes  $E_{\pi}[\sum_{t = 1}^{T}\gamma^{t}(r_{\mathrm{skill}} + \alpha r_{\mathrm{guide}} + \beta H(\pi (\cdot |s_t))))$
B Efficient Random Walk Process
Our random walk process in Section 3 is as follows:
1. After performing rollout for all  $P$  learned skills till the terminal state, we perform  $R$  random walks and collect  $P*R$  number of random arrival states ( $R$  is a hyper-parameter and we set  $R$  as  $0.2 *$  time horizon  $T$  for all environments). Repeat this process a total of  $M$  times.
2. Pinpoint the state in the lowest density region among the  $P*R*M$  number of states. We measure the density of the states using the k nearest neighbors.
3. Select the skill which that state originated from.
The number of environment steps in the random walk process is  $P * (T + 0.2T) * M$ . Therefore, the total number of environment steps performed during the random walk process in pretraining stage is,
$$
\left(N _ {\text {i n i t i a l}} + \left(N _ {\text {i n i t i a l}} + \delta\right) + \left(N _ {\text {i n i t i a l}} + 2 * \delta\right) +.. + \left(N _ {\text {f i n a l}} - \delta\right)\right) * (T + 0. 2 T) * M
$$
where  $N_{\mathrm{initial}}$  is the number of initial skills,  $\delta$  is the number of skills to extend when most of the existing skills are converged (e.g., number of skills are  $10 \to 15 \to 20 \to \ldots \to 50 \to 55$  when  $N_{\mathrm{initial}} = 10$ ,  $\delta = 5$ ,  $N_{\mathrm{final}} = 55$ ). Note that  $N_{\mathrm{final}}$  is the number of skills when the pretraining is ended (not predefined as a hyperparameter).
However, in long-horizon environment such as DMC (1000 timesteps), our random walk process can cause non-negligible sample inefficiencies. Thus, we further present an efficient random walk process that approximates an original random walk process with a simple approach. The detailed process is as follows:
1. During pretraining, if the skill satisfies  $q_{\phi}(z^i | s_T) < \epsilon$  in the terminal state  $s_T$  (i.e., which is not an apprentice skill), additionally perform  $R$  random walk steps and store the random arrival states at the temporary buffer (queue).
2. When selecting guide skill, instead of rolling out all skills, select the state in the lowest density region among the states at the temporary buffer.
3. Select the skill which that state originated from.
Since there are no additional environment steps when selecting the guide skill, the efficient random walk process can significantly reduce the number of environment steps compared to the original version (added environment steps are  $0.2T$  per trajectory (only if its skill satisfies the accuracy threshold), whereas original random walk process requires  $P \times (T + 0.2T) \times M$ ). Our experimental results on DMC demonstrate that the efficient random walk process outperforms other baselines.
C Random Walk Process in High-dimensional State Space
To evaluate the scalability of our random walk process within high-dimensional state spaces, we've identified two essential questions to consider:
1. When the agent starts to explore from an arbitrary terminal state, can the agent visit a diverse range of states through the random walk process?
2. Given the diverse range of visited states, is it possible to identify the terminal state (guide skill) within the least explored region?
In response, we conducted a synthetic experiment on Montezuma's Revenge, which is a high-dimensional pixel-based environment characterized by an  $84 \times 84 \times 3$  input and is well-known for its exploration challenges.
Figure 9: Random walk process in high dimensional environment
Figure 9 provides a visual illustration of our random walk approach within Montezuma's Revenge. In this experiment, we:
1. Randomly reset the agent in the initial room of Montezuma's Revenge.
2. Execute a random walk for 100 steps from the randomly initialized state.
3. Iterate the step 1,2 for  $N$  cycles.
This experimental design mirrors the algorithmic design of DISCO-DANCE where each reinitialized point indicates the terminal state of a different skill. Correspondingly, each skill undergoes a random walk for 100 steps, aligning with the parameters used for our paper (i.e.,  $\mathrm{P} = \mathrm{N}$ ,  $\mathrm{R} = 100$ ,  $\mathrm{M} = 1$  in Equation 4).
Regarding the first question, as illustrated by Figure 9, even in such a high-dimensional state space, the agent was able to visit a variety of states within just 100 random steps. For instance, for Skill 'I', the agent was able to move to another room. And with for Skill 'J', the agent successfully picked up the key. This experimental result supports the versatility of the random walk process, even when the environment is high-dimensional.
Regarding the second question, we employed the cell-centric technique from Go-Explore [1] for density estimation: (i) segmenting the aggregated states into discrete cells, (ii) count the number of each cell's visitation, and (iii) select the least visited cell. As emphasized in Figure 9, the cell marked in red is selected as guide skill, indicating that skill 'I' is the prime candidate for guide skill in DISCO-DANCE.