SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hyunseung Kim\*,1 Byungkun Lee\*,1 Hojoon Lee $^{1}$  Dongyoon Hwang $^{1}$  Sejik Park $^{1}$  Kyushik Min $^{2}$  Jaegul Choo $^{1}$ $^{1}$ Kim Jaechul Graduate School of AI, KAIST.  $^{2}$ KAKAO Corp. {mynsng, byungkun.lee, joonleesky, godnpeter, sejik.park, jchoo}@kaist.ac.kr queue.min@kakaocorp.com
In the field of unsupervised skill discovery (USD), a major challenge is limited exploration, primarily due to substantial penalties when skills deviate from their initial trajectories. To enhance exploration, recent methodologies employ auxiliary rewards to maximize the epistemic uncertainty or entropy of states. However, we have identified that the effectiveness of these rewards declines as the environmental complexity rises. Therefore, we present a novel USD algorithm, skill discovery with guidance (DISCO-DANCE), which (1) selects the guide skill that possesses the highest potential to reach unexplored states, (2) guides other skills to follow guide skill, then (3) the guided skills are dispersed to maximize their discriminability in unexplored states. Empirical evaluation demonstrates that DISCO-DANCE outperforms other USD baselines in challenging environments, including two navigation benchmarks and a continuous control benchmark. Qualitative visualizations and code of DISCO-DANCE are available at https://mynsng.github.io/discodance/.
Deep Reinforcement Learning (DRL) has shown remarkable success in a wide range of complex tasks, from playing video games [28, 49] to complex robotic manipulation [4, 14]. However, the majority of DRL models are designed to train from scratch for each different task, resulting in significant inefficiencies. Furthermore, reward functions adopted for training the agents are generally handcrafted, acting as an impediment that prevents DRL from scaling for real-world tasks. For these reasons, there has been an increasing interest in training task-agnostic policies without access to a predefined reward function. One approach that has been widely studied for achieving this is unsupervised skill discovery (USD), which is a training paradigm that aims to acquire diverse and discriminable behaviors, referred to as skills [7, 11, 13, 17, 22, 25, 47, 52, 32, 20, 46]. These pre-trained skills can be utilized as useful primitives or directly employed to solve various downstream tasks.
Most of the previous studies in USD discover a set of diverse and discriminable skills by maximizing the self-supervised, intrinsic motivation as a form of reward [1, 11, 13, 17, 47]. Commonly, mutual information (MI) between the skill's latent variables and the states reached by each skill is utilized as the self-supervised reward. However, it has been shown in recent research that solely maximizing the sum of MI rewards is insufficient in exploring the state space because, asymptotically, the agent receives larger rewards for visiting known states rather than for exploring novel states due to the high MI reward it receives at a fully discriminable state [7, 25, 52, 20].
To ameliorate this issue, recent studies designed an auxiliary exploration reward that incentivizes the agent when it succeeds in visiting novel states [52, 24, 25]. However, albeit provided these auxiliary
Figure 1: Conceptual illustration of previous methods and DISCO-DANCE. Each skill is shown with a grey-colored trajectory. Blue skill  $z^i$  indicates an unconverged skill, and the reward landscape of  $z^i$  is represented in green. Here, (b,c) illustrates a reward landscape of previous methods, DISDAIN, APS, and SMM. (b) DISDAIN fails to reach the upper region due to the absence of a pathway to the unexplored states. (c) APS and SMM fail since they do not provide exact directions to the unexplored states. On the other hand, (d), DISCO-DANCE directly guides  $z^i$  towards selected guide skill  $z^*$  which has the highest potential to reach the unexplored states. A detailed explanation of the limitations of each baseline is described in Section 2.2.
rewards, previous approaches may exhibit decreased effectiveness in challenging environments. Fig. 1 conceptually illustrates the ineffectiveness of previous methods. Suppose that the upper region of Fig. 1a is difficult to reach with MI rewards, resulting in obtaining skills that are stuck in the lower-left region. To make these skills explore the upper region, previous methods provide auxiliary exploration rewards using intrinsic motivation (e.g., disagreement, curiosity-based bonus). However, since they do not indicate exactly which direction to explore, it becomes more inefficient in challenging environments. We detail the limitations of previous approaches in Section 2.2.
In response, we propose a new exploration objective that aims to provide direct guidance to the unexplored states. To encourage skills to explore unvisited states, we first identify a guide skill  $z^{*}$  which possesses the highest potential for reaching unexplored states (Fig. 1d-1). Next, we select skills that are relatively unconverged, and incentivize them to follow the guide skill  $z^{*}$  in an effort to leap over the state regions with low MI rewards (Fig. 1d-2:4). Finally, they are dispersed to maximize their distinguishability (Fig. 1d-5), resulting in obtaining a set of skills with high state coverage. We call this algorithm as skill discovery with guidance (DISCO-DANCE) and is further presented in Section 3. DISCO-DANCE can be thought of as filling the pathway to the unexplored region with a positive dense reward.
Through empirical experiments, we demonstrate that DISCO-DANCE outperforms previous approaches in terms of state space coverage and downstream task performances in two navigation environments (2D mazes and Ant mazes), which have been commonly used to validate the performance of the USD agent [7, 19]. Furthermore, we also experiment in Deepmind Control Suite [53], and show that the learned set of skills from DISCO-DANCE provides better primitives for learning general behavior (e.g., run, jump, and flip) compared to previous baselines.
In Section 2.1, we formalize USD and explain the inherent pessimism that arises in USD. Section 2.2 describes existing exploration objectives for USD and the pitfalls of these exploration objectives.
2.1 Unsupervised Skill Discovery and Inherent Exploration Problem
Unsupervised Skill Discovery (USD) aims to learn skills that can be further utilized as useful primitives or directly used to solve various downstream tasks. We cast the USD problem as discounted, finite horizon Markov decision process  $\mathcal{M}$  with states  $s\in S$ , action  $a\in \mathcal{A}$ , transition dynamics  $p\in \mathcal{T}$ , and discount factor  $\gamma$ . Since USD trains the RL agents to learn diverse skills in an unsupervised manner, we assume that the reward given from the environment is fixed to 0. The skill is commonly formulated by introducing a skill's latent variable  $z\in \mathcal{Z}$  to a policy  $\pi$  resulting in a latent-conditioned policy  $\pi(a|s,z)$ . Here, the skill's latent variable  $z$  can be represented as a one-hot vector (i.e., discrete skill) or a continuous vector (i.e., continuous skill). In order to discover a set of diverse and discriminable skills, a standard practice is to maximize the mutual information (MI) between state and skill's latent variable  $I(S;Z)$  [1, 11, 13, 17, 47].
$$
I (Z, S) = - H (Z | S) + H (Z) = \mathbb {E} _ {z \sim p (z), s \sim \pi (z)} [ \log p (z | s) - \log p (z) ] \tag {1}
$$
Since directly computing the posterior  $p(z|s)$  is intractable, a learned parametric model  $q_{\phi}(z|s)$ , which we call discriminator, is introduced to derive lower-bound of the MI instead.
$$
I (Z, S) \geq \tilde {I} (Z, S) = \mathbb {E} _ {z \sim p (z), s \sim \pi (z)} [ \log q _ {\phi} (z | s) - \log p (z) ] \tag {2}
$$
Then, the lower bound is maximized by optimizing the skill policy  $\pi (a|s,z)$  via any RL algorithm with reward  $\log q_{\phi}(z|s) - \log p(z)$  (referred to as  $r_{\mathrm{skill}}$ ). Note that each skill-conditioned policy gets a different reward for visiting the same state (i.e.,  $r_{\mathrm{skill}}(z_i,s)\neq r_{\mathrm{skill}}(z_j,s)$ ). It results in learning skills that visit different states, making them discriminable.
However, maximizing the MI objective is insufficient to fully explore the environment due to inherent pessimism of its objective [7, 25, 52]. When the discriminator  $q_{\phi}(z|s)$  succeeds in distinguishing the skills, the agent receives larger rewards for visiting known states rather than for exploring novel states. This lowers the state coverage of a given environment, suggesting that there are limitations in the set of skills learned (i.e., achieving a set of skills that only reach a limited state space).
2.2 Previous exploration bonus and Its limitations
In order to overcome the inherent pessimistic exploration of USD, recent studies have attempted to provide auxiliary rewards. DISDAIN [52] trains an ensemble of  $N$  discriminators and rewards the agent for their disagreement, represented as  $H\left(\frac{1}{N}\sum_{i=1}^{N}q_{\phi_i}(Z|s)\right) - \frac{1}{N}\sum_{i=1}^{N}H(q_{\phi_i}(Z|s))$ . Since states that have not been visited frequently will have high disagreement among discriminators, DISDAIN implicitly encourages the agent to move to novel states. However, since such exploration bonus is a consumable resource that diminishes as training progresses, most skills will not benefit from this if other skills reach these new states first and exhaust the bonus reward.
We illustrate this problem in Fig. 1b. Suppose that all skills remain in the lower left states, which are easy to reach with MI rewards. Since the states in the lower left region are accumulated in the replay buffer, disagreement between discriminators remains low (e.g., low exploration reward in the lower left region). Therefore, there will be no exploration reward left in these states. This impedes  $z_{i}$  from escaping its current state to unexplored states, as shown in Fig. 1b-4.
On the other hand, SMM encourages the agent to visit states where it has not been before using a learned density model  $d_{\theta}$  [24]. APS incentivizes the agent to maximize the marginal state entropy via maximizing the distance of the encoded states  $f_{\theta}(s_t)$  between its  $k$  nearest neighbor  $f_{\theta}^{k}(s_{t})$  [25].
$$
r _ {\text {e x p l o r a t i o n}} ^ {\text {S M M}} \propto - \log d _ {\theta} (s)
$$
$$
r _ {\text {e x p l o r a t i o n}} ^ {\mathrm {A P S}} \propto \log | | f _ {\theta} (s) - f _ {\theta} ^ {k} (s) | | \tag {3}
$$
These rewards push each skill out of its converged states (in Fig. 1c). However, they still do not provide a specific direction on where to move in order to reach unexplored states. Therefore, in a difficult environment with a larger state space, it is known that these exploration rewards can operate inefficiently [7, 9]. In the next section, we introduce a new exploration objective for USD which addresses these limitations and outperforms prior methods on challenging environments.
Unlike previous approaches, we design a new exploration objective where the guide skill  $z^{*}$  directly influences other skills to reach explored regions. DISCO-DANCE consists of two stages: (i) selecting
guide skill  $z^{*}$  and the apprentice skills which will be guided by  $z^{*}$ , and (ii) providing guidance to apprentice skills via guide reward, which will be described in Section 3.1 and 3.2, respectively.
3.1 Selecting Guide skill and Apprentice Skills
Guide skill. We recall that our main objective is to obtain a set of skills that provides high state space coverage. To make other skills reach the unexplored state, we define the guide skill  $z^{*}$  as the skill which is most adjacent to the unexplored states. One naive approach in selecting the guide skill is to choose the skill whose terminal state is most distant from the other skills' terminal state (e.g., blue skill in Fig. 2b). However, such a selection process does not take into account whether the guide skill's terminal state is neighboring promising unexplored states. In order to approximate whether a skill's terminal
(a) Skill trajectories
Figure 2: Guide skill selection. (a) A set of skill trajectories. (b) Naively selecting the skill whose terminal state is most distant from others. (c) Selecting the skill through the random walk process.
(b) Select the most distant skill
(c) Select skill with random walk process
state is near potentially unexplored states, we utilize a simple random walk process (Fig. 2c). To elaborate, given a set of  $P$  skills, (i) rollout skill trajectory ( $T$  timesteps) and perform  $R$  random walks from the terminal state of each skill, repeated  $M$  times (i.e., a total of  $P(T + R)M$  steps, collecting  $PRM$  random walk arrival states as in Fig. 2c). Then (ii) we pinpoint the state in the lowest density region among collected random walk arrival states and select the skill which that state originated from as the guide skill  $z^{*}$  (red skill in Fig. 2c). For (ii), one could use any algorithm to measure the density of the random walk state distribution. For our experiments, we utilize a simple k-nearest neighbor algorithm,
$$
z ^ {*} := \operatorname {a r g m a x} _ {p \in \{1, \dots , P \}} \max  _ {r \in \{1, \dots , R M \}} \frac {1}{k} \sum_ {s _ {p r} ^ {j} \in N _ {k} \left(s _ {p r}\right)} \left\| s _ {p r} - s _ {p r} ^ {j} \right\| _ {2} \tag {4}
$$
where  $s_{pr} = r$ -th random walk arrival state of skill  $p$
$N_{k}(\cdot) = \mathbf{k}$  -nearest neighbors.
In practice, in an environment with a long horizon (i.e., high  $T$ ), such as DMC, our random walk process may cause sample inefficiencies. Therefore, we also present an alternative, efficient random walk process, which is thoroughly detailed in Appendix B. Ablation experiment on two guide skill selection methods (Fig. 2a,b) is provided in Section 4.4.
Apprentice skills. We select apprentice skills as the skills with low discriminability (i.e., skill  $z^i$  with low  $q_{\phi}(z^i|s)$ ; which fails to converge) and move them towards the guide skill. If most of the skills are already well discriminable (i.e., high MI rewards), we simply add new skills and make them apprentice skills. This will leave converged skills intact and send new skills to unexplored states. Since the total number of skills is gradually increasing as the new skills are added, this would bring the side benefit of not relying on a pre-defined number of total skills as a hyperparameter.
In Appendix E, we empirically show that adding new skills during training is generally difficult to apply to previous algorithms because the new skills will face pessimistic exploration problems. The new skills simply converge by overlapping with existing skills (e.g., left lower room in Fig 3), which exacerbates the situation (i.e., reducing discriminator accuracy without increasing state coverage).
3.2 Providing direct guidance via auxiliary reward
After selecting the guide skill  $z^{*}$  and apprentice skills, we now formalize the exploration objective, considering two different aspects: (i) the objective of the guidance, and (ii) the degree of the guidance. It is crucial to account for these desiderata since strong guidance will lead apprentice skills to simply imitate guide skill  $z^{*}$ , whereas weak guidance will not be enough for skills to overcome pessimistic exploration problem.
In response, we propose an exploration objective that enables our agent to learn with guidance. We integrate these considerations into a single soft constraint as
$$
\underset {\theta} {\text {m a x i m i z e}} E _ {z ^ {i} \sim p (z), s \sim \pi_ {\theta} (z ^ {i})} \left[ r _ {\text {s k i l l}} + r _ {\text {g u i d e}} \right]
$$
$$
\text {w h e r e} r _ {\text {s k i l l}} = \log q _ {\phi} \left(z ^ {i} | s\right) - \log p \left(z ^ {i}\right), \tag {5}
$$
$$
r _ {\text {g u i d e}} = - \alpha \mathbb {I} \left(q _ {\phi} \left(z ^ {i} | s\right) \leq \epsilon\right) \left(1 - q _ {\phi} \left(z ^ {i} | s\right)\right) D _ {\mathrm {K L}} \left(\pi_ {\theta} \left(a | s, z ^ {i}\right) \mid \pi_ {\theta} \left(a | s, z ^ {*}\right)\right).
$$
As we describe in Section 3.1, we select skills with low discriminator accuracy (i.e.,  $\mathbb{I}(q_{\phi}(z^i |s)\leq \epsilon))$  as apprentice skills. For (i), we minimize the KL-divergence between the apprentice skill and the guide skill policy (i.e.,  $D_{\mathrm{KL}}(\pi_{\theta}(a|s,z^{i})||\pi_{\theta}(a|s,z^{*}))$ ). For (ii), the extent to which the apprentice skills are encouraged to follow  $z^{*}$ , can be represented as the weight of the KL-divergence. We set the weight as  $1 - q_{\phi}(z^{i}|s)$  to make the skills with low discriminator accuracy be guided more.
Figure 3: Overall procedure of DISCO-DANCE in (a) navigation and (b) continuous control tasks.
Fig. 3 shows that the guidance strategy significantly enhances the learning of skills in both navigation and continuous control environments. In the navigation task (Fig. 3a), the skill located in the upper left corner is selected as a guide through the random walk and utilized to navigate the apprentice skills, which were previously stuck in the lower left corner, into unexplored states. Similarly, in a continuous control environment (Fig. 3b), the skill that has been learned to run is selected as the guide, leading the apprentice skills that were barely moving. This guidance allows the apprentice skills to quickly learn to run. This salient observation suggests that the concept of guidance can be utilized, even for non-navigation tasks. In Section 4.4, we show that all of these components of  $r_{\mathrm{guide}}$  are necessary through additional experiments. We provide pseudocode for DISCO-DANCE in Algorithm 1.
Algorithm 1: Skill Discovery through Guidance
1 Initialize skills  $z^1,\ldots ,z^n$  RL policy  $\pi_{\theta}(a|s,z)$  skill discriminator  $q_{\phi}(z|s)$    
2 Initialize guide skill  $z^{*} =$  None,  $r_{\mathrm{guide}} = 0$    
3 Hyperparameters guide coef  $\alpha$  , apprentice cutoff  $\epsilon$    
4 for  $k = 1,2,\dots$  do   
5 Sample batch  $(s_t,a_t,s_{t + 1},z^i)$  from replay buffer  $D$    
6 if most skills are discriminable enough then   
7 guide skill  $z^{*}\gets$  find_guide_skill(  $\pi_{\theta},z^{1},\dots,z^{n})$    
8  $r_{\mathrm{skill}} = \log q_{\phi}(z^{i}|s_{t + 1}) - \log p(z^{i})$    
9 if  $z^{*}$  is not None then   
10  $\begin{array}{r}\big{\lfloor}r_{\mathrm{guide}} = -\mathbb{I}\big(q_{\phi}(z^{i}|s) <   \epsilon \big)(1 - q_{\phi}(z^{i}|s))D_{\mathrm{KL}}(\pi_{\theta}(a_t|s_t,z^i)||\pi_{\theta}(a_t|s_t,z^*))\\ r = r_{\mathrm{skill}} + \alpha \cdot r_{\mathrm{guide}} \end{array}$    
12 Update  $\pi_{\theta}$  to maximize sum of  $r$    
13 Update  $q_{\phi}$  to maximize log  $q_{\phi}(z|s_t)$
Evaluation. We evaluate DISCO-DANCE across three distinct types of environments commonly used in USD: 2D Maze, Ant Maze, and Deepmind Control Suite (DMC). We utilize state space
Figure 4: Three environments to evaluate skill discovery algorithms. (a) Continuous 2D mazes with various layouts, (b) high dimensional ant navigation tasks, and (c) continuous control environments with diverse downstream tasks.
coverage and downstream task performance as our primary evaluation metrics. To measure state coverage, we discretize the x and y axes of the environment into 10 intervals (i.e., total  $10 \times 10$  buckets) and count the number of buckets reached by learned skills (Fig. 4.a,b). For downstream task performance, we finetune the agent pretrained with baselines and DISCO-DANCE (Fig. 4. (b,c)). Further details regarding the environments and evaluation metrics can be found in Appendix D.
For Ant mazes, we provide the  $(\mathrm{x},\mathrm{y})$  coordinates as the input for the discriminator for all algorithms following previous work [11, 47], since the main objective of navigation environments is to learn a set of skills that are capable of successfully navigating throughout the given environment. For DMC, we utilize all observable dimensions (e.g., joint angles, velocities) to the input of the discriminator to learn more general behaviors (e.g., running, jumping, and flipping) that can be used as useful primitives for unknown continuous control tasks.
Baselines. We compare DISCO-DANCE with various skill-based algorithms, focusing on addressing the pessimistic exploration of USD agents. We compare DISCO-DANCE with DIAYN [11], which trains an agent using only  $r_{\mathrm{skill}}$  without any exploration rewards. We also evaluate the performance of SMM [24], APS [25], and DISDAIN [52], which incorporate auxiliary exploration rewards. We note that all baselines and DISCO-DANCE utilize discrete skills, except APS, which utilizes continuous skills. For downstream tasks, we also compare USD baselines with SCRATCH, which represents learning from scratch (i.e., no pretraining). Additional details are available in Appendix D.
In addition, we include UPSIDE [19] as a baseline. UPSIDE learns a tree-structured policy composed of multiple skill nodes. Although UPSIDE does not utilize auxiliary reward, DISCO-DANCE and UPSIDE both aim to enhance exploration by leveraging previously discovered skills. However, UPSIDE's sequential execution of skills from ancestors to children nodes in a top-down manner leads to significant inefficiency during finetuning, leading to lower performance on downstream tasks. Furthermore, UPSIDE selects the skill with highest discriminator accuracy (i.e., corresponding to Fig. 2b) for expanding tree policies, resulting in reduced state coverage at the pretraining stage. A detailed comparison between DISCO-DANCE and UPSIDE can be found in Appendix F.
4.2 Navigation Environments
4.2.1 2D Mazes
First, we evaluate DISCO-DANCE in 2D mazes (Fig. 4a), which has been commonly used for testing the exploration ability of skill learning agents [7, 19]. The layout becomes more challenging, from an empty maze to a bottleneck maze. The agent determines where to move given the current x and y coordinates (2-dimensional state and action space).
Table 1: State space coverages of DISCO-DANCE and baselines on two navigation benchmarks. The results are averaged over 10 random seeds accompanied by a standard deviation. Scores in bold indicate the best-performing model and underlined scores indicate the second-best.
Models	(a) 2D mazes	(b) Ant mazes
Empty	Y	W	S	Bottleneck	Ant Empty-maze	Ant U-maze	Ant II-maze
DIAYN	100.00±0.00	69.80 ±5.39	71.50 ±5.10	52.80±5.13	52.40±3.77	72.70±10.95	46.80±8.41	22.50±3.34
SMM	100.00±0.00	89.20 ±5.80	73.40 ±6.15	55.00±5.59	54.80±5.11	99.50±0.84	58.90±8.23	25.40±4.97
APS	97.90±3.75	90.00 ±5.51	81.50 ±11.31	80.20±5.88	61.90±10.55	83.10±24.80	59.60±5.85	26.3±7.68
DISDAIN	100.00±0.00	87.20 ±6.81	85.00 ±8.69	61.30±7.04	61.70±4.85	70.10±4.97	45.80±4.98	22.90±2.88
UPSIDE	99.00±9.25	88.20 ±19.39	90.20 ±6.15	69.90±7.63	78.90±13.69	74.50±6.22	63.60±11.58	29.90±2.42
DISCO-DANCE	100.00±0.00	99.10 ±1.66	98.30 ±2.62	88.50±6.45	86.30±17.01	98.90±1.85	68.30±4.47	39.00±4.85
Table 1(a) summarizes the performance of skill learning algorithms on 2D mazes. Our empirical analysis indicates that baseline methods with auxiliary reward (i.e., SMM, APS, and DISDAIN) exhibit superior performance compared to DIAYN. However, we find out that previous studies provide less benefit on performance as the layout becomes more complex, as we mentioned in section 2.2. In contrast, the performance of DISCO-DANCE remains relatively stable regardless of layout complexity. Additionally, DISCO-DANCE outperforms UPSIDE in all 2D mazes, which we attribute to the inefficiency of adding new skills to leaf nodes with high discriminator accuracy (Fig. 2b).
(a)DIAYN
(b) DISDAIN
(c) SMM
Figure 5: Visualization of the skills in bottleneck maze. Multiple rollouts by each algorithm.
(d) APS
(e)UPSIDE
(f) DISCO-DANCE