Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning
Authors: Jiaheng Hu
Zizhao Wang
Peter Stone
Roberto Martín-Martín
Venue: NeurIPS 2024 poster (2024)
Keywords: Unsupervised Skill Discovery
Hierarchical Reinforcement Learning
Disentanglement
Primary area: reinforcement_learning

Abstract: A hallmark of intelligent agents is the ability to learn reusable skills purely from unsupervised interaction with the environment. However, existing unsupervised skill discovery methods often learn entangled skills where one skill variable simultaneously influences many entities in the environment, making downstream skill chaining extremely challenging. We propose Disentangled Unsupervised Skill Discovery (DUSDi), a method for learning disentangled skills that can be efficiently reused to solve downstream tasks. DUSDi decomposes skills into disentangled components, where each skill component only affects one factor of the state space. Importantly, these skill components can be concurrently composed to generate low-level actions, and efficiently chained to tackle downstream tasks through hierarchical Reinforcement Learning. DUSDi defines a novel mutual-information-based objective to enforce disentanglement between the influences of different skill components, and utilizes value factorization to optimize this objective efficiently. Evaluated in a set of challenging environments, DUSDi successfully learns disentangled skills, and significantly outperforms previous skill discovery methods when it comes to applying the learned skills to solve downstream tasks.

Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning
Jiaheng Hu University of Texas at Austin jiahengh@utexas.edu
Zizhao Wang University of Texas at Austin zizhao. wangoutexas edu
Peter Stone
Roberto Martin-Martín' University of Texas at Austin robertomn8cs. utexas. edu
University of Texas at Austin, Sony Al pstoneEcs. utexas. edu
Abstract
A hallmark of intelligent agents the ability to learn reusable skills purely from unsupervised interaction with the environment. However, existing unsupervised skill discovery methods often learn entangled skills where one skill variable si- multancously influences many entities in the environment, making downstream skill chaining extremely challenging. We propose Disentangled Unsupervised Skill Discovery (DUSDi), method for learning disentangled skills that be efficiently reused to solve downstream tasks. DUSDi decomposes skills into dis- entangled components, where each skill component only affects one factor of the state space. Importantly, these skill components can be concurrently composed generate low-level actions, and efficiently chained to tackle downstream tasks through hierarchical Reinforcement Learning. DUSDi defines a novel mutual- information-based objective to enforce disentanglement between the influences of different skill components, and utilizes value factorization to optimize this objective efficiently. Evaluated in a of challenging environments, DUSDi successfully learns disentangled skills, and significantly outperforms previous skill discovery methods when comes to applying the learned skills solve downstream tasks. Code and skills visualization at j jiahenghu. github.ao. io/DUSDi-site/
Introduction
Reinforcement learning (RL) algorithms have achieved many successes in challenging tasks, including magnetic plasma control (II) automobile racing (54), and robotics [47]. However, applying existing RL algorithms to every new task in tabula rasa manner often results in low sample efficiency that limits RL's broader applicability [18]. Unsupervised skill discovery holds the promise of improving the sample efficiency of Reinforcement Learning, by learning free interaction with the environment that can later recombined to tackle multiple downstream tasks more efficiently. In practice, prior unsupervised RL skills represented on a skill variable to generate diverse behaviors, and have led to successful and efficient learning of downstream tasks when combined with skill fine-tuning hierarchical RL skill selection [3] 24 58
of reusable skills through reward-
policy that conditions
Despite prior successes, a common limitation of the skills learned by existing unsupervised RL methods is that they are entangled: any change in the skill variable causes the agent to induce changes in multiple dimensions of the state space simultaneously. Learning to use and recombine these entangled skills can be extremely hard for agent trying solve downstream tasks, especially in
Equal supervision.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prior Works
DUSDi (ours) (2=),,,,, lz', (22)), 20 20,2,,2))
z
[2 2 20]
2
Figure 1: Consider an agent practicing driving skills by learning to control a car's speed (length orange arrow), steering (curvature of orange arrow), and headlights (blue symbol). (Left) previous unsupervised skill discovery methods learn entangled skills, where change in the skill variable can cause all three environment factors to change (Right) DUSDi Iearns disentangled skills with concurrent components, where each skill component only affects one factor of the state space, enabling efficient downstrcam task learning with hicrarchical RL.
complex domains like multi-agent systems or household humanoid robots, where the agent needs to concurrently change multiple independent dimensions the state to complete the task. For example, consider an agent learning operate car: if single skill variable simultaneously changes the speed, steering, and headlights of the car, it will be extremely challenging for the agent learn how to turn on/off the headlights while keeping the car at the right speed and direction. In contrast, humans naturally have the ability to concurrently and independently adjust the car's acceleration, steering, and headlights based on the car's current speed, surroundings, and lighting conditions. In other words, humans naturally obtain disentangled skill components where each component only affects one or few state variables, and can easily recombine these skill components into compositional skills 2 to control multiple factors simultancously.
In this work, we aim to create such a mechanism for artificial agents to learn disentangled skills that facilitate solving downstream tasks. We introduce Disentangled Unsupervised Skill Discovery (DUSDi), novel method for unsupervised discovery of disentangled skills. A key insight of DUSDi to take advantage of state factorization that naturally available in unsupervised RL environments [33 35 17 (e.g. speed, direction, and lighting conditions of the car in the driving example; the state of different objects in a household environment). These factored state spaces provide a natural inductive bias we leverage for disentanglementt DUSDi decomposes skills into disentangled components, and encourages each skill component to affect only one state factor while discouraging from affecting any other factors. To that end, DUSDi designs novel intrinsic reward for unsupervised skill learning based on mutual information (MI) between disentangled skills and state factors: the learning agent receives high rewards for 1) increasing the MI between state factor and the skill component assigned to change it, and 2) for decreasing the MI between that skill component and all other state factors.
DUSDi introduces set of technical innovations tractably and efficiently optimize the proposed mutual information objective. Once the DUSDi skills are learned, they can be used as the low-level policy in a hierarchical reinforcement learning (HRL) setting to tackle downstream tasks. Compared to using entangled skills, a key benefit of using the disentangled DUSDi skills is that they guarantee more efficient exploration during downstream task learning and therefore often lead to significantly better performance. Furthermore, the structured skill space of DUSDi opens up additional possibilities to inject domain knowledge into the learning process to further improve the efficiency of both skill learning and downstream task learning.
DUSDi is easy to implement and can be integrated into any MI-based unsupervised skill discovery approach. In our experiments, we integrate DUSDi with DIAYN [ and evaluate the performance on four domains: 2D agent navigation domain, a DMC walker domain, a large-scale multi-agent particle domain, and a 3D realistic simulated robotics domain. Our experiments indicate that DUSDi can indeed learn disentangled skills, and significantly outperforms other Unsupervised Reinforcement Learning methods on solving complex downstream tasks with HRL.

---

(8)
(b)
disentangled Q prediction for state factor
(s,
Thigh (2/s)
21 tadk
..... reth.
(s, 0,
RL
training:
81-11-.. Strd
env
ITaaa 8,
Intrinsic reward
(=())ss))
To(a|a, 2)
XN (# state factors)
Figure 2: Two learning stages of DUSDi: (a) in disentangled skill learning stage, DUSDi creates one-to-one mapping between state factors and skill components each disentangled skill component 2' only influences state factor s'. DUSDi designs a novel
intrinsic reward to enforce disentanglement and utilize Q-value decomposition to learn the skill policy He efficiently. (b) in the task learning stage, the skill policy is used frozen low-level policy and high-level policy Thigh is learned to select skill for every L steps, by maximizing the task reward tosk
2 Preliminaries
Factored Markov Decision Process (f-MDP) In this work, we consider unsupervised skill discov- ery in reward-free Factored Markov Decision Process. Following Osband and Van Roy [33], Mohan et al. [32], we define a Factored Markov Decision Process by the tuple M = (S, A, P). where = S X.... x S.N is a factored state space with N factors such that each state E S consists of E St. .A the action space, and P is an unknown Markovian transition model, S x A S. Notice that a factored state space is often naturally available in 35 17 49 as it can naturally represent environments with separate elements (e.g., objects) that be changed independently. DUSDi leverages the property that factors often have sparse dynamics dependencies, which opens up the possibility of learning disentangled skills control the state of each factor. Moreover, many downstream tasks are defined by specific changes in one or a few factors (e.g.. changing the state of single object and not others), which are easier to learn with disentangled skills. domains with only image-based (unfactored) observations. factored state space can be extracted using disentangled representation learning or object-centric representation learning methods 31 20,, which we empirically evaluated in Sec. 4.5
N state factors: = (s' ......... ),), s²
domains used by prior works [3
24
Based Skill Discovery Mutuallinforrmatio--boo based skill discovery methods, such the paradigmatic DIAYN 13 specify the skills with a latent variable Z, and learns skill-conditioned policy π (a|s, 2). The optimization objective these methods use to learn the skills to maximize the mutual information (MI) between the state, s, and the skill latent variable, [(; 2), which incentivizes the agent to reach diverse and distinguishable states. One popular way to determine the MI, I(S; 2), is to decompose it as I(S: Z) (2)
[Z S), where H denotes entropy. Since the skill variable is typically sampled from a fixed distribution, H(Z) can be assumed constant: maximizing T(S; 2) is thus equivalent to minimizing H(2 (). Following the definition
of conditional entropy. - -H(2 2 S) = I.,, (log p( [s)]. DIAYN proposes to approximate p(z[s) with a learned discriminator g(2 s) that predicts the skill latent, z, given the state, s.
After discovering the skills, swward-uperrviedd tasks. Many methods (e.g., DIAYN) adopt hierarchical RL structure for this second phase, where the skill policy is used as low-level "frozen" element, and high-level policy Thigh (2 (s) learns to sequentially activate skill based on observations. The high- policy is trained to maximize the provided task reward, R, with Z as the action space.
wwwualliiformation--ssseee methods apply them to learn downstream
3 Learning Disentangled Skills with DUSDi
Similar to prior works unsupervised skill discovery, DUSDi implements a two-stage learning procedure for the agents: in the first phase, DUSDi develops library of skills without external reward (Sec. B.T). The key to DUSDI's success is to encourage disentanglement between different skill components through novel learning objective that restricts the effect of each disentangled skill

---

Algorithm DUSDi Skill Learning
Initialize skill policy zo, discriminators 1 Th and value function Q' for each state factor S¹.
2: for each skill training episode do
3: Sample skill 2
p(=). >>p()).
4: Collect state transitions with actions from ( 2).
5: Sample batch of (s, a, 2) from the replay buffer.
6: for 1,......
7:
Update d/ (=")s" and g ()) with discrimination losses.
8: Calculate r² based Eq.
9: Update (s, a, z) with reward r' using SAC.
end for
11: Update Te with Q E Q' using SAC.
end for
component to independent factors. In the second phase, DUSDi leverages the learned skills to solve downstream tasks through Hierarchical Reinforcement Learning, achieving higher returns Σ Rusk than methods with entangled skills (Sec.3.3 In practice, learning disentangled skills in environments with many factors can be challenging. To address this challenge, we introduce improvements to DUSDI's ş first phase based Q-function decomposition (Sec. 3.2 pipeline in Fig. 2 and the pseudo-code Alg.
We present the entire DUSDi
3.1 Disentangled Skill Spaces and Learning Objective
DUSDi aims create disentangled skill components that can be easily recombined to solve down- stream tasks. To that end, DUSDi proposes novel factorization of the latent skill conditioning variable, 2, into N independent disentangled components such that the latent space becomes 2 z² x... z We equate N the number of state factors and consider z Z² the disentan- gled skill component that affects state factor i. The skill policy x (a]s,, 2) takes in E 2, which is composition of the skill components.
While the factored latent space Z could be discrete or continuous, we consider discrete skill space in this paper, and discuss how DUSDi can be applied to continuous skills in Appendix A We can then assume that each disentangled component 2" takes the form of an integer, 2' E [1, k]. resulting in a compositional skill, 2, with the form of N-dimensional multi-categorical vector with NN possible values. During skill learning, we independently sample each disentangled component z² from fixed uniform distribution p(="). similar previous works [13 35).
Given this factored skill space, our goal to learn a skill policy network, To S + A, such that each disentangled component 2 affects and only affects the value of a state factor, St. For each disentangled component and state factor pair (Z',St), encourage diverse and distinguishable behaviors by maximizing their mutual information [(S Z'). While this objective enables disentanglement skill component to affect the corresponding factor, it does not restrict the component from affecting other factors. This undesirable since the resulting skill components would still be entangled in their effects. To prevent that, we propose to ensure that each skill component, Z'. minimally affects the rest of the state factors, S where S denotes the subspace formed by all other state factor spaces except S" S X Si-1 x Si+1 x x.. x SN Specifically, we incorporate an entanglement penalty to minimize, I(S 2"), which corresponds to the mutual information between skill component and all other state factors that should not affect.
Formally, the skill policy aims to maximize the following objective:
N I(0) Σ - x
where À < 1 is hyperparameter that controls the importance of the entanglement penalty relative to the skill-factor association. We restrict À to be smaller than one the following reason: some environments, due to intrinsic dynamical dependencies between state factors themselves, controlling

---

a state factor, S has to introduce some association between Z' and other factors in S e.g., when controlling an object whose manipulation requires the agent to use other objects tools. In these cases, the policy learns to maximize the MI between a skill and factor, 1(8", zz), the MI with other factors, I(S 2" may also increase. For these cases, the use of entanglement penalty does not overpower the association reward, and the policy is still incentivized learn disentangled skill components that change St distinguishably while introducing minimal changes on other factors. In practice, we simply set À 0.1 in all our experiments.
will ensure that the
Optimizing DUSDi's Objective: Directly maximizing the objective in Eq. 1 intractable. Alter- natively, we propose to approximate the objective using a variational lower bound of the mutual information [I]
((()))) - 2 C Ez,s log ('
where C' represents the constant value of HZ2)), the entropy of the prior distribution over the skill latent variable, which does not change during training, and %% is variational distribution.
Similarly, we can approximate the MI in the entanglement penalty by:
[(S ',Z') 2 C' Bx,. log (= ")
where d another variational distribution Importantly, when these g approximations perfectly recover the posterior distribution of 2", we obtain equality in Eq. 2 and Eq. 3 In DUSDi, we implement the variational distributions, 9á and lo. as neural network discriminators mapping input state factor(s) to the predicted disentangled component values, 21.
To optimize [0),, we alternate between two steps: 1) performing variational inference to train the discriminators d' and g, through gradient ascent, and 2) using d' and d' to learn disentangled skill policy through RL by maximizing the following intrinsic reward approximating Eq.
(s(s, a) A
(s (2)((2))))))
(4)
Interestingly, the decomposed nature of our intrinsic reward allows a convenient avenue for shaping skill behaviors based on domain knowledge. In particular, we can restrict state factor s only take certain values by constraining g (z*|s") accordingly. While not the main focus of this work, we briefly explore this further optimization enabled by DUSDi in Appendix H
3.2 Accelerating Skill Learning through Q Decomposition
When using reinforcement learning (RL) to optimize the intrinsic reward function defined in Eq. standard RI. algorithms treat the reward function as black box and learn single value function from the mixture of intrinsic reward terms. While this approach may be sufficient for environments with few state factors, doing so for complex environments with many state factors (large .N often leads to suboptimal solutions. A key reason is that the mixture of 2N reward terms leads inevitably high variance in the reward, making the value of the Q function oscillate. Furthermore, the sum of reward terms obscures information about each term's 's value, which hinders credit assignment.
4
DUSDi overcomes this issue by leveraging the fact that the intrinsic reward function in Eq. is a linear sum over terms associated with each disentangled component. Thanks to the linearity of
'Wiile lower-boundigg J (0) requires upper-bounding I(S 4; (z)),, we stick with the variational lower bound for this term because of the complexity in upper bounding MIs 41
5

---

expectation, we can decompose the Q function into N disentangled Q functions as follows:
Qx = Es
rrl
1=U
N
Ee
cx(="ls") 22b
1=0
Σ
E.
('ss,a,2)
(5)
where Q' represents each disentangled Q function, one for each disentangled component. The disentangled Q functions can be then updated only with their corresponding intrinsic reward terms, r A (=*) - X (=*Is (t). During policy learning, we sum all disentangled functions together to recover the global critic, QT. as shown in Fig. (a), top. Compared to learning Qx directly from all 2N reward terms, learning disentangled Q functions significantly reduces reward variance, allowing Q. to converge faster and more stably.
3.3 Downstream Task Learning
Similar to Eysenbach et al. [13], in DUSDi we utilize hierarchical RL solve reward-supervised downstream tasks with the discovered skills, as depicted in (b). The skill policy, S x Z + A, acts as the low-level policy and is kept constant while a high-level policy, Thigh -> 2, learns to select which skill to execute for I, steps using the skill latent variable, 2. Thus, the skill latent conditioning space, 2, acts as the action space of the high-level policy, Thigh- As extensively evaluated in our experiments, without any additional "ingredient", performing downstream task learning in the action space formed by DUSDi skills often results in significantly superior performance compared to an action space formed by entangled skills. We show that the superior performance of DUSDi can be explained by more efficient exploration when using the DUSDi skills for hierarchical RL, which we elaborate in Appendix B through analyzing the benefits and search complexity of DUSDi's skill space over DIAYN's.
Depending on the nature of the downstream tasks, we can often take further advantage of the disentangled skills learned by DUSDi through leveraging its structure. One such scenario when the downstream task has composite reward function consisting of multiple terms. Previous works 16 46) have shown that when the causal dependencies from action dimensions to reward terms are available (e.g., the reward for speed only depends on actions that affect speed), one can use Causal Policy Gradient (CPG) to decompose the policy update (e.g., only the "speed actions" get updated by the speed reward) and greatly improve sample efficiency, especially when the dependencies are sparse. In downstream task learning, with an action space (of the high-level policy) consisting of the skills learned by DUSDi, we have convenient way of applying causal policy gradient, where the causal dependencies between the action dimensions (i.c., skill components) and reward terms often sparse and can be easily obtained by examining the state factor that skill component is associated with, which we evaluate empirically in Sec. 4.6
4 Experimental Evaluation
In the evaluation of DUSDi, we aim to answer the following questions: Q1: Are skills learned by DUSDi truly disentangled (Sec. y Q2: Can Q-decomposition improve skill learning efficiency (Sec. 4.3 ? Q3: Do our disentangled skills perform better when solving downstream tasks compared to other unsupervised reinforcement learning methods (Sec. 4.4 Q4: Can DUSDi be extended to
image observation environments (.5c Q5: Can we leverage the structured skill space of DUSDi
to further improve downstream task learning efficiency (Sec 4

---

Table 1:
Evaluation of skill disentanglement based on the DCI metric, shown as mean and standard deviation across skill policies trained with random seeds.
2D GUNNER
MULTTIPARTICLA
1GIBSON
DUSDi (ours)
DIAYN-MC
DUSDI (ours)
DIAYN-MC
DUSDi (ours)
DIAYN-MC
Disentanglement (1)
0.864 0.018
0.016 0.002
0.705 + 0.037
0.002 0.000
0.833 + 0.022
0.017 0.006
Completeness (f)
0.864 + 0.017
0.024 = 0.004
0.750 = 0.041
0.003 + 0.000
0.834 = 0.021
0.019 = 0.005
Informativeness (1)
0.897 0.012
0.821 = 0.010
0.849 = 0.052
0.791 0.032
0.854 = 0.006
0.752 = 0.015
4.1 Evaluation Environments
Previous works 13
34
35
23 extensively rely on standard RL environments such as DMC 48
and OpenAI Fetch evaluate unsupervised RL methods. However, unlike previous unsupervised skill discovery methods, DUSDi focuses on learning a set of disentangled skill components that can be concurrently executed and re-combined to complete downstream tasks. As such, it only makes sense to examine the performance of DUSDi in challenging tasks that require concurrent control of many environment entities (e.g. multi auget- systems, complex household robots). Previous environments lack this property: in DMC for example, while the state and action space can be very complex, the predominant downstream tasks are just to move the center-of-mass of the agent to different places. In such cases, there is no need for concurrent skill components, and therefore we do not expect large gains from using DUSDi's disentangled skills. Nevertheless, we include an evaluation on the DMC-Walker 48) environment to demonstrate that our method is also applicable to those environments, but focus the majority of our evaluation on environments that DUSDi is designed for, including 2D Gunner, Multi-Particle [30], and iGibson (26).
The 2D gunner is a relatively simple domain, where point agent can navigate inside a continuous 2D plane, collecting ammo and shooting at targets. Multi-Partion is multi- domain modified based on [30]. this domain, a centralized controller simultaneously controls 10 heterogenous point-mass agents to interact with 10 stations, where each agent can only interact with a specific station. We evaluate in this domain to test the scalability of our methods
large number of state factors. iGibson [26] is a challenging simulated robotics domain with the same action space and complexity as real-world robots, where a mobile manipulator can navigate in a room, inspect the room using its head camera, and interact with electric appliances in the room by pointing remote control to them and switching them on/off. We evaluate in this domain to examine whether our method can handle home-like like environments with complex dynamics. We provide visualizations and additional information about each of the environments in Appendix C
4.2 Evaluating Skill Disentanglement
First, we examine whether the skills learned by DUSDi are truly disentangled (Q1) using the DCI metric proposed by Eastwood and Williams (2]. The DCI metric consists of three terms, namely disentanglement, completeness, and informativeness, explained detail in Appendix F In the original work, measuring DCI requires knowing the ground truth generative factors. In case, the generative factors are simply the state factors, and we only need to discretize the value of each state factor to make it compatible for evaluation. For each method on each domain, collect 100K rollout steps using the learned skill policy, π (s, z). where the skill is (re)sampled from the uniform prior distribution, p(=), every 50 steps. These (state, skill) pairs are then used to calculate DCI.
We compare against DIAYN-MC (Multi-channe) DIAYN) that uses the same skill representation as DUSDi but optimizes the DIAYN objective of I(S; Z), and show results in Table I Unsurprisingly, DUSDi significantly outperforms DIAYN-MC, especially on Disentanglement and Completeness, across all three environments. These results indicate that DUSDi learns truly disentangled skills, enabling efficient downstream task learning, as we will show in Sec. [.44. We encourage the readers to visit our project website for qualitative visualization of the learned skills.
4.3 Evaluating Skill Learning Efficiency with Q-decomposition
To examine the importance of Q-decomposition (Q2), we measure the performance of optimizing the DUSDi objective during skill learning with and without a decomposed Q network. We compare the classification accuracy of the skill discriminators g <²'ss)), averaged over all skill channels, which
7

---

1.0-
1.0
1.0
medetton Accurary
0-Deoampeskan
Prediction Accursey
Decomposttion
0.6
6.
Docomessiton
Decompps Bon No 0-Decmposkk
DDDeomppp
0.2
0
2N
Number Stees
Number
(a) 2D Gunner
(b) Multi-Particle
(c) iGibson
Figure 3: Evaluation of the effect of Q- Q-decomposition in skill learning. The plots depict the mean and standard deviation of accuracy (1) when predicting the skill component z' based on the state factor s'. computed across 3 training processes. The higher prediction accuracy indicates that the policy learns control more state factors in more distinguishable ways, leading to more efficient downstream task learning.
indicates progress towards discovering diverse and distinguishable skills, with higher accuracy being better. We depict our results in Fig. We observe that Q-decomposition has similar performance to the regular Q network in the simplest 2D gunner domain, but significantly outperforms the regular Q network in domains with more state factors MMulti-Particlo and more complex dynamics (iGibson), suggesting that Q-decomposition is necessary for scaling towards complex domains.
4.4 Evaluating Downstream Task Learning
The promise of DUSDi is to incorporate disentanglement into skills so that the skills can be effectively used in downstream task learning. Therefore, the most critical evaluation of our work focuses comparing the performance of different unsupervised RL methods on task learning (Q3). We compare against existing state-of-the-an unsupervised reinforcment learning algorithms, including
DIAYN [3]. CIC (24), CSD [5J. METRA [36], ICM (37]. RND [5), ELDEN [5T], and Vanilla
RL 14, where these baselines are further explained in Appendix E
Similar to the evaluation setting the URLB benchmark 23 we allow each method to train for 4 million steps without access reward (i.e., pretraining phase) before the reward is revealed to the agent and the downstream learning takes place. During the pre- pre-training phase, all methods use soft actor-critic (SAC) [14] to optimize the intrinsic reward. For all skill discovery methods (i.e., DUSDi, DIAYN, CIC, CSD, METRA), a skill-conditioned policy, To ro(a|s (, 2), is Iearned during the pretraining phase. During downstream learning. the skill network is fixed, whereas upper policy, (=)s), is trained using proximal policy optimization (PPO)
44 to optimize the task reward. we omit proprioceptive states from the MI optimization for all skill discovery methods to facilitate more meaningful explorations. For exploration methods (i.e., RND, ICM, ELDEN), a policy ma(als) is learned during the pretraining phase on intrinsic reward and fine-tuned using the task reward during the downstream learning phase. The hyperparameters are specified in Appendix G
Similar to previous works [ 44
We evaluate all methods in four environments and 13 downstream tasks, detailed in Appendix D The results are depicted in Fig. As expected, DUSDi performs similarly to previous unsupervised RL methods in the DMC walker environment due to the simplicity in terms its downstream objectives (all related to center-of-mass locomotion), but significantly outperforms all previous methods domains where downstream tasks require coordinative control of multiple state factors. The most crucial comparison is between DUSDi and DIAYN. DIAYN is special case of DUSDi where there is only one state factor (consisting of the entire state) and one skill component. Therefore comparing against DIAYN offers straightforward examination of the effect of disentangled skills for downstream task learning. DUSDi significantly outperforms DIAYN in all downstream tasks, demonstrating the effectiveness of using disentangled skills. In general, we found exploration-baser methods to be less capable than skill discovery methods, possibly due to their lack of temporal abstraction. CIC performs very poorly, likely because the CIC objective does not explicitly encourage distinguishable skills and instead generates the intrinsic reward solely based on state entropy, making it very hard for the upper policy to select the right skill. This result again shows the importance of having a proper skill representation. DUSDi also outperforms CSD and METRA on most downstream tasks, especially on the more complex and high-dimensional domains, like Multi-Particle. This superiority is perhaps surprising considering that in our experiments, DUSDi only relies on the simple DIAY intrinsic reward for skill discovery, but further demonstrates the importance of learning a disentangled skill space. It is important notice that many techniques proposed to improve skill discovery quality
8

---

DUSDI (ours)
CSD
METRA
DIAYN
CIC
ICM
RND
ELDEN
SAC
60
2<
24
(a) Walker-run
(b) Walker-goal
(c) 2DG-unlim
(d) 2DG-lim
100
Sumeer cades
Surber of
Rumber Spisece
surbe sod
(c) Masseq-casy
(f) MP-seq-medium
gg) MP-seq-hard
(h) MP-fp-casy
MP-fp-medium
(i) MP-fp-hard
(k) IG-look
(l) IG-housekeep
(m) IG IG-inspect
Figure 4: Training curves DUSDi and baselines on multiple downstream tasks (reward supervised second phase). The plots depict the mean and standard deviation of the return of each method over 3 random seeds. DUSDi outperforms all baselines that learn entangled skills, converging faster and to higher returns.
(e.g., Baumli et al. B3, Zhao et al. (57). can be seamlessly incorporated into DUSDi. Therefore, we expect our method perform even better new advances are made in unsupervised skill discovery.
4.5 Extending DUSDi to Image Space
Although this paper primarily focuses on applying DUSDi factored state space, we can straightfor- wardly extend it to image space through existing works in factored object-centric representation learning 29 20
55 (Q4). We empirically illustrate this capability in the Multi-Particle envi- ronment, where we replace the low-dimensional state observation with 64 x 64 image observations. Specifically, we first pretrain an object-centric encoder following Yang et al. (55). and then use our method on top of the extracted representation to learn disentangled skills. Hence, essentially, the skill policy uses images as observation. As shown in Fig. 5 when learning from image observation, DUSDi achieves similar performance to learning from state space, whereas the baseline methods are unable to learn these two tasks even when learning from the low-dimensional state space as in Fig.
53 28
4.6 Leveraging Structure of DUSDI Skills
While DUSDi can already learn downstream tasks quite efficiently, it possible to further improve the sample efficiency of downstream task learning through leveraging the structured skill space of DUSDi (Q5), as described in the second paragraph of Sec Specifically, we apply Causal Policy domain, where the causal dependencies between state factors and
Gradient [16] to the Multi-Particn reward terms are easy to identify. We present our results Fig- 6 where the sample efficiency of downstream task learning is greatly improved thanks to the structured skill space of DUSDi.
Related Work
Unsupervised Skill Discovery In unsupervised skill discovery, the goal of an agent is learn task-agnostic skills without external rewards. To learn such skills, previous methods propose various forms of intrinsic reward: (1) maximizing the mutual information between visited states and the skill
variables [3] 44
6 24]. (2) maximizing the traveled distance along the direction specified by the
skill variables 34. 36), (3) learning to reach diverse set of goals [52
38 These skills can be
used to boost the sample efficiency of downstream task learning, for example, (1) using hierarchical
RI. where high-level policy learns to select which skill execute [3] or (2) using the skill policy
to initialize the task solving policy and then fine-tuning it (24].

---

100
122
Retum
20
DUST wi 9300 we wilmage
DUST w/ 5000 imags
DUSB DUED
Dush DUDD LOG
4c
Narioer of baiadddd
Sumber episodes
number of ipiooees
Namoer saboods
(a) MP-ff-medium
(b) MP-fp-hard
(a) MP-fp-medium
(b) MP-fp-hard
Figure 5: Performance of DUSDI with image ob- servations on two multi-particle downstream tasks over three random seeds. With the help of dis- entangled representation learning. DUSDi effec- tively learns skills based only on image observa- tions and leverages the skills to solve challenging downstream tasks where baseline methods fail.
Figure 6: Performance of DUSDI in two multi- particle downstream tasks when combined with Causal Policy Gradient (CPG, orange). The disen- tangled skills of DUSDi provide opportunities for leverage structure and speed up downstream task learning, greatly improving the sample efficiency when learning downstream tasks.
State Space Factorization in RL RL, there is a long history of leveraging state factorization,
including learning world model between state factors for planning 22 55] augmenting data [39].
and providing intrinsic rewards 42
17 9 Relevant to our work are skill discovery methods that
learn to either reach a goal for each controllable object [9 [ or achieve interactions between
pair of specified objects [8]. Though these methods achieve disentanglement by influencing one or pair of objects during skill, they do not apply to tasks that require controlling multiple objects simultaneously. like driving where we need to control the car's ' speed and heading directions at the same time. In contrast, method can combine disentangled skill components into concurrent skills [0 to solve a wide range of tasks.
Disentanglement in Skill Learning Inspired by the benefits of compositionality, disentanglement has been extensively studied, mainly in learning image representations (8 There are few works investigating disentanglement in unsupervised skill discovery. Lee et al. 25 consider a special case of disentangled skills - for a multi-arm robot, learning independent skills for each arm. However, they rely manually factored action spaces which is an assumption that often limits the behavior of the agent. Kim et al. (2 encourage the disentanglement between different dimensions of the skill variable by regularizing it with V-E objective [15), but Locatello et al. (27) point out that such regularization is impossible to achieve disentanglement. To learn disentangled skills, Song et al. 45 learns a decoder from skill variables to state trajectories and their generation factors, which is then used to train the skill policy through imitation learning. However, their training of the decoder requires pre-collected trajectories and corresponding generation factors, whereas method is fully unsupervised with no expert data.
6 Conclusion
We present DUSDi, an unsupervised skill discovery method for learning disentangled skills by lever- aging the factorization of the state space. DUSDi designs a skill space that exploits the factorization of the state space and learns a skill-conditioned policy where each sub-skill affects only one state factor. DUSDi enforces disentanglement through an intrinsic reward based on mutual information, and shows superior performance on a set of downstream tasks with naturally factored state spaces compared to baselines and state-of-thh-arr unsupervised RI. methods.
One limitation of DUSDI is the assumption of access to factored state space. While a factored state space is naturally available in many existing RI. environments, and can be extracted from images as we have shown in our experiment (Sec. 4.5). we believe that future advances in disentangled representation learning will greatly broaden the applicability of DUSDi towards partially observable, pixel-based environments. Secondly, DUSDi primarily focuses on learning structured skill space for more efficient downstream learning, and its exploration capability during skill learning is largely determined by the specific algorithm used to optimize for our mutual information objective. While we used DIAYN [13 in this work due to its simplicity. it would be interesting to examine extending the idea of learning disentangled skills to other skill discovery methods, e.g., Zhao et al. Laskin
et al. 24 including those that are not based on mutual information 335 56
10

---

Acknowledgements This work took place at the Learning Agents Research Group (LARG) and the Robot Interactive Intelligence Lab (RobIn) at UT Austin. RobIn supported in part by DARPA TIAMAT program (HR0011-244 1-24-99---4) LARG research is supported in part by NSF (FAIN- 2019844, NRT-2125858), ONR (N00014-18-2443) ARO (W91INF-23-2-00N) and UT Austin's Good Systems grand challenge. Peter Stone serves as the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.
Lockheed Martin,
References
[1] David Barber and Felix Agakov. Information maximization in noisy channels: A variational approach. Advances Neural Information Processing Systems, 16, 2003.
[2] André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Shibl Mourad, David Silver, Doina Precup, The option keyboard: Combining skills in reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[3] Kate Baumli, David Warde-Farley, Steven Hansen, and Volodymyr Mnih. Relative variational intrinsic control. Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 6732-6740, 2021.
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1606v:/606.0/55555 2016.
[5] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation, 2018
[6] Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
[7] Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu. Varia- tional empowerment representation learning for goal-based reinforcement learning. arXiv preprint 100.0:2006.01444, 2021.
[8] Jongwook Choi, Sungtae Lee, Xinyu Wang, Sungryull Sohn, and Honglak Lee. Unsupervised object interaction learning with counterfactual dynamics models. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023.
[9] Caleb Chuck, Kevin Black, Aditya Arjun, Yuke Zhu, and Scott Niekum. Granger-causal hierarchical skill discovery. arXiv preprint 2rXiv:2306.09509 2023.
[10] Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Y Oudeyer. Autotelic agents with intrinsically motivated conliconditioned reinforcement learning: short survey, 2022.
[11] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897): 414-419, 2022.
[12] Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International conference on learning representations, 2018.
[13] Benjamin Eysenbach. Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: 8802006070 2018.
[14] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-
policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
11

---

[15] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2016.
[16] Jiaheng Hu, Peter Stone, and Roberto Martin-Martinn Causal policy gradient for whole-body mobile manipulation. arXiv preprint arXiv:2335..044: 0.4666,, 2023.
[17] Jiaheng Hu, Zizhao Wang, Peter Stone, and Roberto Martin-Martin Elden: Exploration via local dependencies. arXiv preprint wrXiv22300008702, 2023.
[18] Jiaheng Hu, Rose Hendrix, Ali Farhadi, Aniruddha Kembhavi, Roberto Martin-Martin Peter Stone, Kuo-Hao Zeng, and Kiana Ehsan. Flare: Achieving masterful and adaptive robot policies with large-scale reinforcement learning fine-tuning, arXiv preprint arXiv:2409. 16578, 2024.
[19] Xing Hu, Rui Zhang, Ke Tang, Jiaming Guo. Qi Yi, Ruizhi Chen, Zidong Du, Ling Li, Qi Guo, Yunji Chen, et al. Causality- hierarchical structure discovery for reinforcement learning. Advances in Neural Information Processing Systems, 35:20064-20076, 2022.
[20] Jindong Jiang, Fei Deng. Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. arXiv preprint Xiv: 10834, 2023.
[21] Jackyeom Kim, Scohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. arXiv preprint arXiv:2106. 14305, 2021.
[22] Thomas Kipf, Elise Van der Pol, and Max Welling, Contrastive learning of structured world models. arXiv preprint arXiv: 1. /2247, 2019.
[23] Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark, 2021.
[24] Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint 2rXiv:2202.00/61, 2022.
[25] Youngwoon Lee, Jingyun Yang, and Joseph J Lim. Learning to coordinate manipulation skills via skill behavior diversification. In International conference on learning representations, 2019.
[26] Chengshu Li, Fei Xia, Roberto Martin,Martin, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, C. Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks, 2021.
[27] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 41 14- 4124. PMLR, 2019.
[28) Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael T'channen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pages 1348-6359, PMLR, 2020.
[29] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention, 2020.
[30] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compelee environments. Neural Information Processing Systems (NIPS), 2017.
[31] Sindy Löwe, Phillip Lippe, Francesco Locatello, and Max Welling, Rotating features for object discovery. arXiv preprint arXiv:2306.00666 2023.
[32] Aditya Mohan, Amy Zhang, and Marius Lindauer. Structure in reinforcement learning: survey and open problems. arXiv preprint arXiv:2306.16022 2023.
12

---

[33] Ian Osband and Benjamin Van Roy. Near-optimal reinforcement learning in factored mdps. Advances in Neural Information Processing Systems, 27, 2014.
[34] Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. arXiv preprint arXiv:2202.00914 2022.
[35] Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllabillty unsuper- vised skill discovery. arXiv preprint 1005 2023.
[36] Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric- aware abstraction. arXiv preprint urXiv:2310.08888 2023.
[37] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity- explo- ration by self-supervised prediction, 2017.
[38] Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi- reinforcement learning. In International Conference on Machine Learning, pages 7750-7761. PMLR, 2020.
[39] Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. Advances in Neural Information Processing Systems, 33:3976-3990, 2020.
[40] Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering supervupe reinforcement learning. arXiv preprint 1903,0099, 2019.
[41] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180. PMLR, 2019.
[42] Cansu Sancaktar, Sebastian Blaes, and Georg Martius. Curious exploration via structured world models yields zero-shot object manipulation. Advances in Neural Information Processing Systems, 35:24170-224 70-24183, 2022.
[43] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[44] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. arXiv preprint 1907.0:990../// 2019.
[45] Wonil Song, Sangryul Jeon, Hyesong Choi, Kwanghoon Sohn, and Dongbo Min. Learning disentangled skills for hierarchical reinforcement learning through trajectory autoencoder with weak labels. Expert Systems with Applications, page 120625, 2023.
[46] Thomas Spooner, Nelson Vadori, and Sumitra Ganesh. Factored policy gradients: Leveraging structure for efficient learning in momdps, 2021.
[47] Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martin-Martin and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes. arXiv preprint arXiv:2408. 035339 2024.
[48] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite, 2018.
[49] Zizhao Wang, Jiaheng Hu, Caleb Chuck, Stephen Chen, Roberto Martin,Martín, Amy Zhang, Scott Niekum, and Peter Stone. Skild: Unsupervised skill discovery guided by local depen- dencies. In Workshop on Reinforcement Learning Beyond Rewards@ Reinforcement Learning Conference 2024.
[50] Zizhao Wang, Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal dynamics learning for maskinndependent state abstraction. arXiv preprint arXiv:2206.1344 2022.
[51] Zizhao Wang, Jiaheng Hu, Peter Stone, and Roberto Martin-Martin, Elden: exploration via local dependencies. Advances in Neural Information Processing Systems, 36, 2024.
13

---

[52] David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non- con-mammetrii discriminative rewards. arXiv preprint arXiv: 18/ 1359, 2018.
[53] Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, and Animesh Garg, Slotformer: Unsuper- vised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05811, 2022.
[54] Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896): 223-228, 2022.
[55] Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning via neural structural causal models. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages
2021.
[56] Rushuai Yang, Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, and Xuelong Li. Behavior contrastive learning for unsupervised skill discovery, 2023.
[57] Rui Zhao, Yang Gao, Picter Abbeel, Volker Tresp, and Wei Xu. Mutual information state intrinsic control. arXiv preprint arXiv:2103.088/11 2021.
[58] Yifeng Zhu, Peter Stone, and Yuke Zhu. Bottom-up skill discovery from unsegmented demon- strations for long-horizon robot manipulation. JEEE Robotics and Automation Letters, 7()): 4126 1126-4133, 2022.
14

---

A Extension to Continuous Skills
For continuous skills, we can simply define each skill component 2$ as a m-dimensional continuous vector (and therefore the length of the skill would be m x N, where N is the total number of skill components). Now we can define the prior distribution for each skill component distribution p(z") as an m-dimensional continuous uniform distribution (e.g. 2[|1,, 1). Notice that our objective remains unchanged, as the MI is well-defined no matter whether the skill is discrete or continuous. Lastly, need to change the output head of our skill prediction networks q, such that instead of outputting a categorical distribution, it will output a continuous probability distribution (e.g. a multivariate Gaussian distribution with diagonal covariance).
B Entangled vs. Disentangled Skill Components for Policy Learning
Compared to entangled skills, the advantages of using disentangled components mainly reside in an casier exploration in the skill space. For skill spaces of equivalent capacity, the DIAYN latent skill variable is single integer between I and k>V and the DUSDi skill variable is N-ddimensional vector with each dimension representing a disentangled component with & possible values. this section, we analyze the benefits and search complexity of DUSDI's space over DIAYN's for two main cases: when there are no dynamical dependencies between state factors (optimal case for disentangled components) and where there are intrinsic dependencies between state factors.
State Factors without Dynamical Dependencies: In this case, for DIAYN to find the correct skill execute at the current time step, in the worst case, needs iterate through all skills, resulting in I-step exploration sample-efficiency of O(k*), ). In contrast, for DUSDi, as disentangled components are independent of each other, with one skill trial, the agent can simultaneously observe the effects of setting each disentangled component as 2 z². Hence, for intelligent agent, to understand the effects of each disentangled component at the current state, it only needs to sweep through each disentangled component space with & trials (e.g., setting all disentangled components z 1,.... k). After that, as the effects of each disentangled component are independent, by compositing disentangled components in novel ways, the agent has the ability to imagine the effects of all skills, leading to Okk) exploration efficiency.
State Factors with Dynamical Dependencies: When there are dynamical dependencies, we denote PA as parent indices of state factors that S depends on, e.g., when moving a mouse (S'). SPA denotes the hand. In such cases, the effect of Z is conditioned on the value of PAA and we need to iterate through all (Z', 2Pxx') pairs to observe all possible influences on S As a result, the exploration is constrained by the state factor with the largest number of parents. Denoting |PA'l as the number of parent factors for St the exploration sample-efficiency is O / [PA'I). We can see that the O(k) efficiency when there no dynamical dependencies is a special case of max, |PA'| = Despite lower efficiency than Okk), in many environments, the dynamics of each state factor only depend on small number of other factors, i.e., max; [PA'| < N. Hence, exploration with disentangled components is still more sample-efficient than using entangled skills.
C Environment Details
We test DUSDi on four environments, where a visualization of each of the environments is presented in Fig. 7
2D Gunner: Shown in Fig. 7 (a), the blue star marks the position of the agent, the blue line marks its shooting direction, the red diamond marks ammo location, and the orange cross marks the target position. The agent has a 7-dimensional observation space, consisting of 3 state factors: [Agent Position, Ammo State, Target State). The action is 5-dimensional, 2 for agent movement, 2 for ammo pickup, and 1 for shooting direction.
DMC- Walker: Shown in Fig. (b). a Segrree-fffrredom robot can locomote on a 2D plane through joint motions. The agent has a 26-dimensional observation space consisting of 3 state factors: (Body Position, Body Velocity, Robot Proprioception).
15

---

(a) 2D Gunner
(b) DMC Walker
(c) Multi- Multi-Particle
(d) iGibson
Figure 7: Environments Visualization
Shown in Fig. 7 (c), the agents are marked by small circles, while the stations are marked by large circles. Only stations and agents of the same color can interact with each other. The Multi-Particle environment has 70-dimensional observation space, consisting of 20 state factors. The state factors include states for each landmark and states for each agent. The action space is 50-dimensional, with 5 dimensions per agent that control their motions and interactions with the landmarks.
Multi-Particle:
Shown in Fig. 7 (d). iGibson has 42- 42-dimensional observation space consisting of 4 state factors, including [Agent Location, Electric Appliances State, Object(s) in View, Robot Propriocep- tion]. The action space is I--iimnsionall, consisting of base velocity (2D), head motion (2D), arm motion (6D), and gripper motion (1D).
iGibson:
D Downstream Tasks
DMC- Walker (Walker):
Run: In this task, the walker agent is rewarded for moving forward at particular velocity.
Goal Reaching: In this downstream task, agent has to reach randomly generated goal positions.
2D Gunner (2DG):
Unlimited Ammo (unlim): In this downstream task, set of targets will randomly appear, where the agent needs to navigate to a position close to the target and shoot them order to score. The ammo is unlimited the agent does not need to worry about picking up ammo.
Limited Ammo (lim): This downstream task is different from the "unlimited ammo" in that the agent starts with no ammo and needs pick up ammo in order shoot. Everything else identical.
Multi- PartilPartiele (MP):
Sequential interaction (seq) (easy, medium, hard): In this task, agents need to sequentially interact with their corresponding station following an instruction sequence given at the start of each episode. Interacting with stations in the wrong order will be penalized. The easy version of this task has sequence length of 2, while medium and hard have a sequence length
and 8 respectively.
Food-poison (fp) (easy, medium, hard): In this downstream task, each station will offer either food or poison to the corresponding agent. Fach agent needs to decide whether to interact with its corresponding station based on a sequence of binary indicators provided the agents. The difficulty level has the same meaning as in the sequential interaction task.
iGibson (IG):
Look around: In this task, the robot needs look at objects in the room sequentially.
Appliances inspection: In this task, the robot needs to navigate different electric appliances, and test whether each of them working correctly by pointing remote control towards it.
Housekeeping: In this task, the robot needs to manage the electric appliances intelligently. Specifi- cally, the robot needs to first look at screen to receive instructions. Depending on the instruction, the robot needs to off certain electric appliances using the remote control.
16

---

E Baseline Methods
During downstream task evaluation, we compared against the following state-of-the-ane unsupervised
RL methods:
DIAYN 13 represents skill variable as an integer between I to k- and learns skills by maximiz- ing I(S: 2). the MI between 2 and all state factors S.
CIC 24 learns state representation with contrastive learning and learns skills by maximizing transition entropy in the representation space.
CSD 5 learns skills maximizing distance traveled along the direction of the state space, where distance is measured in controllabilit--waan manner.
METRA (6) learn set of behaviors that collectively cover as much the state space as possible through optimizing Wasserstein variant of the state-skill Mutual Information.
ICM 37]: encourages visiting novel states by using prediction errors of action consequences as intrinsic rewards.
RND 5 encourages visiting novel states by using prediction errors of features computed from randomly initialized network as intrinsic rewards.
ELDEN 17 operates in factored state space similar our approach, and encourages visiting states that induce novel factor dependencies.
SAC [14] where pretraining used, and vanilla RI is directly applied to tackle the downstream tasks.
F
Evaluating Skill Disentanglement Details
The DCI metric consists of three terms, namely disentanglement, completeness, and informative- ness. In the context of this work, disentanglement (1) measures, on average, to what extent each skill component only affects single state factor. Completeness score (1) measures, on average, to what extent each state factor is only influenced by single skill component. Informativeness score (f) measures the repeatability of learned skills: given the skill 2, how accurately we can predict which states will be visited. We refer the reader to the work by Eastwood and Williams [2] for detailed discussion of these metrics and how they are calculated.
G Hyperparameters
Skill Dimensions: For all skill learning methods with discrete skills (i.e. DUSDi, DIAYN), we make sure that they have equivalent capacity. Specifically, for igibson and 2D gunner, each DUSDi skill consists of skill components, each component with 5 possible values. As result, DIAYN skill is an integer between to 125 in these two domains. The only exception Multi-Particle, where DUSDi has ten sub-skills, each with 5 possible values. Since skill an integer between I and 5 9765625 obviously challenging for DIAYN to converge, we set the number of discrete skills to be 4096 for DIAYN. For continuous skills (L.e. CSD, CIC, METRA), follow the skill dimensions specified in the original papers (64D for CIC, 3D for CSD and METRA), which were shown to be effective for the respective methods.
Skill Learning Parameters: All skill learning methods in our baselines use SAC to optimize for the intrinsic reward, with the same policy and value network architecture. DUSDi applies additional decomposition and masking to the value networks, as described Section. 3.2 which is not applicable to the baseline methods. Due to Q-decomposition. when using the same value network architecture, DUSDI's value network capacity is N times of the capacity of other methods' value networks (including when comparing the variations of DUSDi, i.c., no decomposition). For fair comparison, we also tried to increase value network capacity for other methods to match the capacity for DUSDi, but found that their skill/task learning performances do not improve significantly. This suggests (1) that, for skill learning, reward variance, rather than network capacity, the key reason for no Q-composition variation of DUSDi converge slowly, and (2) that, for task learning, disentangled skills, rather than network capacity, is what make DUSDi significantly outperform baselines.
We present the hyperparameters for SAC in Table. 2
All methods use low-level step size of L 50.
17

---

Table 2: Hyperparameters of Skill Learning.
Name
Value
optimizer
Adam
activation functions
ReLu
learning rate batch size critic target r MLP size steps per update # of environments Temperature log std bounds
10
1024
SAC
[1024, 1024]
2
0.02
[-10, 21
Downstream Hierarhical Learning: For all skill discovery methods, downstream learning of the skill selection policy is implemented with PPO. We used the same hyperparameters for all methods across all tasks, specified in Table. g
Table 3: Hyperparameters of Downstream Learning.
Name
Value
optimizer
Adam
activation functions
Tanh
learning rate
1 10
batch size
32
clip ratio
PPO
MLP size
[128, 128)
GAE À target steps n steps of environments
0.98
250
20
of low-level steps I.
50
Downstream Finetuning: For all non-skill discovery methods, downstream learning is done using the same hyperparameters as pretraining (table. [ replacing the intrinsic reward with the task reward.
H Behavior Restriction of Skills via Domain Knowledge
Due to the decomposable nature of the intrinsic reward of DUSDi, we can conveniently restrict the behavior of skills by constraining the skill predictor q (=*(s*) for particular state factor i. For example, if we want s to stay within certain range, can set o (=*ls") to be uniform distribution for all not within this range, effectively discouraging the agent from going out of range. In the extreme case, we can fully specify the mapping between 2 and s", essentially resulting in performing goal-conditioned RL for state (as also pointed out in while performing DUSDi for the rest of the state factors.
We qualitatively examine this idea the iGibson domain. By restricting mobile manipulator to only locomote in regions that are close to whiteboard, our robot successfully learns diverse board-wiping behaviors which are otherwise extremely hard to learn. Visualizations of the learned skills can be seen on our project website.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
This paper's main contribution is on unsupervised reinforcement learning for
efficient downstream task learning, which is also the focus of the abstract and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are clearly discussed the last paragraph of conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
The main results are empirical, while the theoretical results are proofed in the paper.
Justification:
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provided all hyperparameters in Appendix G as well as our codebase.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provided our codebase with detailed instructions. This paper does not use any existing dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
oolicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwppppguuuiiieeeee Sttp:////uiiees//eeSuuubbiisi
(CodeSubiisss wwtauubmmsssooni ssiiiissiicccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All hyperparameters specified in the Appendix G Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We include standard deviation for all quantitative our results (Sec. 4p.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: provide computer resources in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is advance the field of Machine Learning, There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We explicitly cites all environments that we used in this paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "3uACH9MI42", "reviewer": "Reviewer_bkGy", "summary": "This paper proposed DUSDi, which learned disentangled skills in an unsupervised manner. Specifically, DUSDi learns each skill component only affects one factor of the state space. DUSDi is evaluated in a specialized tasks comparing with skill discovery methods.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed concept of disentanglement is novel\n- the paper is easy to follow.", "weaknesses": "- It is not convincing how the disentanglement is beneficial in real-world scenarios.\n- The empirical evaluation is limited; standard benchmarks (e.g., D4RL) is not used, and it is questionable whether it can be used in more complex environments like pixel-based environments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reading of our paper and constructive suggestions.\n\n> It is not convincing how the disentanglement is beneficial in real-world scenarios.\n\nFirst, we would like to point out the realism of our experiments. We used iGibson, one of the benchmarks that best match the complexity of real-world robotics tasks. The action space in iGibson is exactly the same as in a real-world mobile manipulator with manipulation, navigation, and active sensing capabilities, and previous works have demonstrated that policies learned in iGibson can transfer zero-shot to the real world [1, 2]. The performance of DUSDi in iGibson and the other experiments is a strong support of the benefits of our method in real-world scenarios.\nSecond, as discussed in the introduction, disentanglement is particularly helpful when dealing with complex state spaces consisting of many factors. This situation often appears in real-world applications such as complex robot systems (e.g., mobile manipulators), multi-object environments, or multi-agent systems, rendering DUSDi a critical solution for common scenarios.\n\n[1] Li, Chengshu, et al. \"igibson 2.0: Object-centric simulation for robot learning of everyday household tasks.\" CoRL 2021.\n\n[2] Hu, Jiaheng, et al. \"Causal policy gradient for whole-body mobile manipulation.\" RSS 2023.\n\n> standard benchmarks (e.g., D4RL) is not used\n\nWe would like to point out that D4RL is a dataset for offline reinforcement learning, while our method, as well as all the baselines we compared against, are fully online. We do compare on standard online RL benchmarks as in DMC-walker (which is similar to the mujoco environment in D4RL), and discuss why we do not further test on more of those domains in section 4.1.\n\n> it is questionable whether it can be used in more complex environments like pixel-based environments\n\nWe would like to respectfully point out that we have conducted experiments in pixel-based environments, and the entire section 4.5 is devoted to that set of experiments, where our results indicate that DUSDi can be used in pixel-based environments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fbNael0UPI", "reviewer": "Reviewer_QWMX", "summary": "Grounding on previous work on mutual information for skill learning, DUSDi proposed an algorithm for disentangled skill learning, which introduces several advances. First, DUSDi proposes to map factorized state components to factorized skill components. Then, it proposes the adoption of Q decomposition and Causal Policy Gradient to further enhance stability and efficiency.", "questions": "* the authors explicitly state that their objective is not optimizing a lower bound (line 150). Why not trying to actually keep a bound? e.g. following [1]\n* since the authors discuss the unsupervised RL benchmark, and they use a similar setup, why not presenting results on the popular benchmark? I understand their point about the fact that these environments mostly require manipulating a single variable, but being a more standard benchmark, it would have been useful to see complete results on it (not just in the Walker env)\n* what happens if you don't omit proprioceptive states from the MI optimization? (line 264-265)\n\n\n[1] On Variational Bounds of Mutual Information, B. Poole et al,", "limitations": "The limitations of the approach are addressed in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The work presents a stable and efficient approach for unsupervised skill discovery, which presents the following strengths:\n* **Simplicity**: the idea is straightforward and doesn't introduce complex components. Actually, through Q decomposition, DUSDi simplifies the learning of multiple skills.\n* **Empirical study**: the empirical study is extensive, it includes modern baselines, such as CSD and METRA and it highlights well the strengths of the approach.\n* **Presentation**: the presentation of the work is clear and of good quality. The additional visualizations on the website are interesting and further demonstrate the approach works as expected.", "weaknesses": "* **State space factorization**: the approach works well thanks to the assumption that the state space is well-factorized. However, for POMDPs or high-dimensional environments, the algorithm may not work as expected. The authors show that object-centric approaches can obviate such weakness. This works well for simple environments where colours can easily help distinguish objects, e.g. the multiparticle env, but it may not work as well in more complex environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reading of our paper and very constructive suggestions!\n\n> for POMDPs or high-dimensional environments, the algorithm may not work as expected.\n\nWe agree with the reviewer that extending our method to partially observable complex pixel environments is non-trivial, and thus a great direction for future work that is opened up by this paper. For such complex domains, we may have to extend DUSDi with some form of explicit representation or memory, or with some implicit stateful architecture, combined with more powerful visual representation techniques such as SAM2 [1]. We will add this direction into the future work section in the next version of this paper. \n\n[1] Ravi, N., Gabeur, V., Hu, Y.-T., et al. (2024). SAM 2: Segment Anything in Images and Videos. arXiv preprint.\n\n\n\n> Why not trying to actually keep a bound?\n\nWe thank the reviewer for pointing out this excellent paper. Since the objective we are trying to optimize contains negative signs in front of some MI terms, if we want to derive a lower bound for our objective, we need to derive upper bounds for the “negative MI terms”, As pointed out in [2], upper bounding MIs is more challenging than lower bounding it, and often involves more complicated expressions, which is why we ended up deciding to use lower bounds for all the MI terms. That being said, it is certainly interesting for future work to examine whether we can actually lower bound our objective in a computationally tractable way, and how that will affect the quality of learned skills. We will add this direction into the future work section in the next version of this paper. \n\n[2] On Variational Bounds of Mutual Information, B. Poole et al\n\n\n\n> I understand their point about the fact that (the standard) environments mostly require manipulating a single variable, but being a more standard benchmark, it would have been useful to see complete results on it (not just in the Walker env)\n\nWe thank the reviewer for acknowledging our reasons for devoting our computing resources to more informative domains. Below, we provide some preliminary results for running on the quadruped domain  in URLB, where we follow exactly the setup of URLB, with 2×10^6 frames of pretraining and 1×10^5 frames of fine-tuning:\n\nTask| DDPG   | ICM         | RND         | DIAYN         | DUSDi  |\n----------|--------|-------------|-------------|-------------|--------|\nJump    | 236±48 | 205±33 | 590±33 |578±46 | 704±42 |\nRun    | 157±31 | 133±20 |  462±23 |415±28 | 450±13 |\nStand  | 392±73 | 329±58 |  804±50 |706±48 | 844±26 |\nWalk | 229±57 | 143±31 |  826±19 |406±64 | 706±57 |\n\nNotice that baseline results are directly obtained from the URLB paper [3]. The table includes the returns obtained by each method. Despite not being the type of problem DUSDi is optimized for, we observe that DUSDi obtains best performance in 2 tasks, and performance comparable to the best solution in the others.\n\n[3] Laskin, Michael, et al. \"Urlb: Unsupervised reinforcement learning benchmark.\" (2021).\n\n> what happens if you don't omit proprioceptive states from the MI optimization?\n\nThe main reason that previous works like DIAYN and DADS omit proprioceptive states from the MI optimization is to prevent the agent from focusing too much on “less meaningful” states such as body postures, and we simply follow this setup since it is an easy heuristic to focus the learning on more meaningful explorations.\nCompared to methods like DIAYN, we anticipate our method to be less affected by adding the proprioceptive states into the MI optimization, since while one component of our objective will now be focusing on diversifying the less meaningful body posturing, the other components of our objective can still facilitate the mastering of more meaningful skills.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nE1ZuSbMhB", "reviewer": "Reviewer_JxTV", "summary": "The paper presents a method (DUSDi) for learning reusable skills through unsupervised interactions. Unlike existing methods that produce entangled skills, DUSDi focuses on disentangling skills into components that each affect only one factor of the state space. This enables efficient chaining of skills via hierarchical reinforcement learning. A mutual-information-based objective ensures skill disentanglement, and value factorization optimizes the process.", "questions": "Please address my concerns in the Weakness section", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written, and the empirical results show consistent advantages over baselines\n2. The idea is simple and effective on the tested environments.", "weaknesses": "1. The paper assumes that the skill space is discrete; there is little information on how the formulations generalize to continuous skill space. \n2. There is a lack of visualization of the learned skill embeddings, and how different skills affect specific factors of the state space, highlighting the disentangled nature, which makes it hard to determine the contribution of the proposed method to the overall performance gain over various baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reading of our paper and very constructive suggestions!\n\n> there is little information on how the formulations generalize to continuous skill space\n\nFor continuous skills, we can simply define each skill component z^i as a m-dimensional continuous vector (and therefore the length of the skill would be m*N, where N is the total number of skill components). \nNow we can define the prior distribution for each skill component distribution p(z^i) as an m-dimensional continuous uniform distribution (e.g. U[-1, 1]).\nNotice that our objective remains unchanged, as the MI is well-defined no matter whether the skill is discrete or continuous.\nLastly, we need to change the output head of our skill prediction network, such that instead of outputting a categorical distribution, it will output a continuous probability distribution (e.g. a multivariate gaussian distribution with diagonal covariance).\nAs a proof-of-concept, we have implemented a continuous version of our method with the settings described above, and examined it on 2DG domain, where the results are shown below:\n\nTask     | DUSDi-discrete | DUSDi-continuous | \n------------|----------------|------------------|\n2DG-unlim           | 39.7±0.4       | 39.7±0.6         |\n2DG-lim         | 30.4±1.7       | 29.9±2.2         | \n\nAs we can see, the continuous version of our method has similar performances as the discrete one on the 2DG domain, showcasing that we can indeed extend DUSDi to continuous skill space.\nWe thank the reviewer for pointing this out, and we will add this content to the appendix for the next version of this paper.\n\n> There is a lack of visualization of the learned skill embeddings, and how different skills affect specific factors of the state space, highlighting the disentangled nature\n\nWe agree with the reviewer that understanding (and visualizing) the disentangled nature of our learned skills is important to understanding the contribution of our work. However, we consider our manuscript already contains elements for that:\nwe would like to respectfully point out that we provided visualization of the skills on our (anonymized) project website, with a special emphasis on the disentanglement of skills; we refer to the website multiple times in the paper (lines. 240 & footnote on page 1). We understand that reviewers are not forced to check these additional materials but due to space constraints, we are not able to include these visualizations in the main paper. We kindly refer the reviewer to our website for visualizations of the effect of each skill.\nIn section 4.2 of our paper, we also provide quantitative evaluations of the level of disentanglement of our learned skills, as shown by Table. 1.\nWith respect to the “visualizations of the skill embeddings” we would like to remark that in our work the skill variables themselves are discrete vectors. Beyond our visualizations of the (disentangled) effect of each skill, we are not sure what kind of visualization would show the “skill embeddings”, but we would gladly consider any suggestion the reviewer may have in mind.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oy9c8fzI9s", "reviewer": "Reviewer_hssy", "summary": "This paper proposes a method for learning disentangled skills that can be efficiently reused to solve downstream tasks. The mutual information objective design is simple yet effective. Intensive empirical results show the superiority of the proposed method.", "questions": "Please see the weakness part.", "limitations": "Provided.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The algorithm design, based on factored MDP, is well motivated. The empirical study shows that DUSDi consistently outperform other unsupervised skill discovery and unsupervised RL methods.", "weaknesses": "(a) The algorithmic contribution is not significant compared with other MI-based unsupervised skill discovery methods.\n\n(b) The key objective design lacks theoretical support. As the author mentioned, Eq. (4) does not constitute a lower bound of the real objective.\n\n(c) A qualitative study to show if the learned skill embedding is semantically detangled would be beneficial.\n\n(d) The submitted code folder should include a readme file on how to reproduce the paper results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reading of our paper and very constructive suggestions!\n\n>  The algorithmic contribution is not significant compared with other MI-based unsupervised skill discovery methods.\n\nWe believe simple ideas that work well are very valuable for the community. As we demonstrate in our experiments, the algorithmic contribution of DUSDi, while may be considered simple, is highly effective and novel.\nMoreover, our contribution is orthogonal to most of the previous unsupervised skill discovery works and can be combined (as we discussed in the conclusion section).\n\n> The key objective design lacks theoretical support. As the author mentioned, Eq. (4) does not constitute a lower bound of the real objective.\n\nFirst, even though it is not a lower bound, our objective is still a plausible approximation of the true objective, and will be accurate when q is equal to p (line 142-143). More importantly, our empirical evaluations show that this objective works very well in practice; we explained and discussed in the paper the possible reasons (line 151-152). Lastly, we would like to point out that similar approximations have also been used in previous works [1].\n\n[1] Baumli, Kate, et al. \"Relative variational intrinsic control.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 35. No. 8. 2021.\n\n> A qualitative study to show if the learned skill embedding is semantically detangled would be beneficial.\n\nWe agree with the reviewer that understanding (and visualizing) the disentangled nature of our learned skills is important to understanding the contribution of our work. However, we consider our manuscript already contains elements for that:\nWe would like to respectfully point out that we provided visualization of the skills on our (anonymized) project website, with a special emphasis on the disentanglement of skills; we refer to the website multiple times in the paper (lines. 240 & footnote on page 1). We understand that reviewers are not forced to check these additional materials but due to space constraints, we are not able to include these visualizations in the main paper. We kindly refer the reviewer to our website for visualizations of the effect of each skill.\nIn section 4.2 of our paper, we also provide quantitative evaluations of the level of disentanglement of our learned skills, as shown by Table. 1.\n\n> The submitted code folder should include a readme file on how to reproduce the paper results.\n\nWe would like to respectfully point out that we have provided a README file with instructions on how to run our code, as in “/neurIPS-DUSDi/README.md” in the supplementary material. Please let us know if you have further suggestion about the content of the README file, and we will be working on polishing it for the next version of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces DUSDi, a method for learning disentangled skills in unsupervised environments that can be efficiently reused and chained to solve downstream tasks. The idea presented in the paper is significant, well-executed, and introduces novelties around the disentanglement of skills which could potentially be scaled to more complex settings. The primary concern is that the key novelty (Equation 4) should be more strongly supported from a theoretical standpoint, although the authors provide a reasonable justification surrounding this point, including previous works which have incorporated similar approximations. Apart from this, the discrete nature of skill spaces is fairly common, and is as such, not viewed as a major limitation/flaw, although means to extend the idea to continuous skill spaces is welcome. The proposed method also demonstrates that it can be applied in pixel-based environments, which demonstrates its general applicability. Although D4RL was suggested, it is a baseline commonly employed for offline methods and is as such, not applicable to the method in question.
Decision: Accept (poster)