Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning

Authors: jiaheng  hu
zizhao  wang
peter  stone
roberto  martín-martín
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 07:44:00

Abstract: A hallmark of intelligent agents is the ability to learn reusable skills purely from unsupervised interaction with the environment. However, existing unsupervised skill discovery methods often learn entangled skills where one skill variable simultaneously influences many entities in the environment, making downstream skill chaining extremely challenging. We propose Disentangled Unsupervised Skill Discovery (DUSDi), a method for learning disentangled skills that can be efficiently reused to solve downstream tasks. DUSDi decomposes skills into disentangled components, where each skill component only affects one factor of the state space. Importantly, these skill components can be concurrently composed to generate low-level actions, and efficiently chained to tackle downstream tasks through hierarchical Reinforcement Learning. DUSDi defines a novel mutual-information-based objective to enforce disentanglement between the influences of different skill components, and utilizes value factorization to optimize this objective efficiently. Evaluated in a set of challenging environments, DUSDi successfully learns disentangled skills, and significantly outperforms previous skill discovery methods when it comes to applying the learned skills to solve downstream tasks.

Main paper: 
 introduction: 
 Reinforcement learning (RL) algorithms have achieved many successes in challenging tasks, including magnetic plasma control [11] , automobile racing [54] , and robotics [47] . However, applying existing RL algorithms to every new task in a tabula rasa manner often results in low sample efficiency that limits RL's broader applicability [18] . Unsupervised skill discovery holds the promise of improving the sample efficiency of Reinforcement Learning, by learning a set of reusable skills through rewardfree interaction with the environment that can be later recombined to tackle multiple downstream tasks more efficiently. In practice, prior unsupervised RL skills are represented as a policy that conditions on a skill variable to generate diverse behaviors, and have led to successful and efficient learning of downstream tasks when combined with skill fine-tuning or hierarchical RL skill selection [13, 24, 58] .Despite prior successes, a common limitation of the skills learned by existing unsupervised RL methods is that they are entangled: any change in the skill variable causes the agent to induce changes in multiple dimensions of the state space simultaneously. Learning to use and recombine these entangled skills can be extremely hard for an agent trying to solve downstream tasks, especially in Prior Works DUSDi (ours)Figure 1 : Consider an agent practicing driving skills by learning to control a car's speed (length of orange arrow), steering (curvature of orange arrow), and headlights (blue symbol), (Left) previous unsupervised skill discovery methods learn entangled skills, where a change in the skill variable can cause all three environment factors to change (Right) DUSDi learns disentangled skills with concurrent components, where each skill component only affects one factor of the state space, enabling efficient downstream task learning with hierarchical RL. complex domains like multi-agent systems or household humanoid robots, where the agent needs to concurrently change multiple independent dimensions of the state to complete the task. For example, consider an agent learning to operate a car: if a single skill variable simultaneously changes the speed, steering, and headlights of the car, it will be extremely challenging for the agent to learn how to turn on/off the headlights while keeping the car at the right speed and direction. In contrast, humans naturally have the ability to concurrently and independently adjust the car's acceleration, steering, and headlights based on the car's current speed, surroundings, and lighting conditions. In other words, humans naturally obtain disentangled skill components where each component only affects one or few state variables, and can easily recombine these skill components into compositional skills [2] to control multiple factors simultaneously.In this work, we aim to create such a mechanism for artificial agents to learn disentangled skills that facilitate solving downstream tasks. We introduce Disentangled Unsupervised Skill Discovery (DUSDi), a novel method for unsupervised discovery of disentangled skills. A key insight of DUSDi is to take advantage of state factorization that is naturally available in unsupervised RL environments [13, 35, 17] (e.g. speed, direction, and lighting conditions of the car in the driving example; the state of different objects in a household environment). These factored state spaces provide a natural inductive bias we leverage for disentanglement: DUSDi decomposes skills into disentangled components, and encourages each skill component to affect only one state factor while discouraging it from affecting any other factors. To that end, DUSDi designs a novel intrinsic reward for unsupervised skill learning based on mutual information (MI) between disentangled skills and state factors: the learning agent receives high rewards for 1) increasing the MI between a state factor and the skill component assigned to change it, and 2) for decreasing the MI between that skill component and all other state factors.DUSDi introduces a set of technical innovations to tractably and efficiently optimize the proposed mutual information objective. Once the DUSDi skills are learned, they can be used as the low-level policy in a hierarchical reinforcement learning (HRL) setting to tackle downstream tasks. Compared to using entangled skills, a key benefit of using the disentangled DUSDi skills is that they guarantee more efficient exploration during downstream task learning and therefore often lead to significantly better performance. Furthermore, the structured skill space of DUSDi opens up additional possibilities to inject domain knowledge into the learning process to further improve the efficiency of both skill learning and downstream task learning.DUSDi is easy to implement and can be integrated into any MI-based unsupervised skill discovery approach. In our experiments, we integrate DUSDi with DIAYN [13] and evaluate the performance on four domains: a 2D agent navigation domain, a DMC walker domain, a large-scale multi-agent particle domain, and a 3D realistic simulated robotics domain. Our experiments indicate that DUSDi can indeed learn disentangled skills, and significantly outperforms other Unsupervised Reinforcement Learning methods on solving complex downstream tasks with HRL. = (s 1 , . . . , s N ), s i ∈ S i .A is the action space, and P is an unknown Markovian transition model, S × A → S. Notice that a factored state space is often naturally available in domains used by prior works [13, 24, 35, 17, 9, 49] as it can naturally represent environments with separate elements (e.g., objects) that can be changed independently. DUSDi leverages the property that factors often have sparse dynamics dependencies, which opens up the possibility of learning disentangled skills to control the state of each factor. Moreover, many downstream tasks are defined by specific changes in one or a few factors (e.g., changing the state of a single object and not others), which are easier to learn with disentangled skills. In domains with only image-based (unfactored) observations, a factored state space can be extracted using disentangled representation learning or object-centric representation learning methods [31, 20] , which we empirically evaluated in Sec. 4.5.Mutual-Information-Based Skill Discovery Mutual-information-based skill discovery methods, such as the paradigmatic DIAYN [13] , specify the skills with a latent variable z ∈ Z, and learns a skill-conditioned policy π(a|s, z). The optimization objective these methods use to learn the skills is to maximize the mutual information (MI) between the state, s, and the skill latent variable, z: I(S; Z), which incentivizes the agent to reach diverse and distinguishable states. One popular way to determine the MI, I(S; Z), is to decompose it asI(S; Z) = H(Z) -H(Z | S), where H denotes entropy. Since the skill variable is typically sampled from a fixed distribution, H(Z) can be assumed constant: maximizing I(S; Z) is thus equivalent to minimizing H(Z | S). Following the definition of conditional entropy, -H(Z | S) = E s,z [log p(z|s)], DIAYN proposes to approximate p(z|s) with a learned discriminator q(z|s) that predicts the skill latent, z, given the state, s.After discovering the skills, mutual-information-based methods apply them to learn downstream reward-supervised tasks. Many methods (e.g., DIAYN) adopt a hierarchical RL structure for this second phase, where the skill policy is used as a low-level "frozen" element, and a high-level policy π high (z|s) learns to sequentially activate skill z based on observations. The high-level policy is trained to maximize the provided task reward, R, with Z as the action space.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: This paper's main contribution is on unsupervised reinforcement learning for efficient downstream task learning, which is also the focus of the abstract and introduction sections.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations are clearly discussed in the last paragraph of conclusion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
learning disentangled skills with dusdi: 
 Similar to prior works in unsupervised skill discovery, DUSDi implements a two-stage learning procedure for the agents: in the first phase, DUSDi develops a library of skills without external reward (Sec. 3.1). The key to DUSDi's success is to encourage disentanglement between different skill components through a novel learning objective that restricts the effect of each disentangled skill Algorithm 1 DUSDi Skill Learning 1: Initialize skill policy π θ , discriminators q i ϕ , q i ψ and value function Q i for each state factor S i . 2: for each skill training episode do 3:Sample skill z ∼ p(z).Collect state transitions with actions from π θ (a|s, z).Sample a batch of (s, a, z) from the replay buffer.for i = 1, . . . , N do 7:Update q i ϕ (z i |s i ) and q i ψ (z i |s ¬i ) with discrimination losses.Calculate r i based on Eq. 49:Update Q i (s, a, z) with reward r i using SAC. Update π θ with Q = N i=1 Q i using SAC. 12: end for component to independent factors. In the second phase, DUSDi leverages the learned skills to solve downstream tasks through Hierarchical Reinforcement Learning, achieving higher returns R task than methods with entangled skills (Sec. 3.3). In practice, learning disentangled skills in environments with many factors can be challenging. To address this challenge, we introduce improvements to DUSDi's first phase based on Q-function decomposition (Sec. 3.2). We present the entire DUSDi pipeline in Fig. 2 , and the pseudo-code in Alg. 1.DUSDi aims to create disentangled skill components that can be easily recombined to solve downstream tasks. To that end, DUSDi proposes a novel factorization of the latent skill conditioning variable, z, into N independent disentangled components such that the latent space Z becomes Z = Z 1 × • • • × Z N . We equate N to the number of state factors and consider z i ∈ Z i the disentangled skill component that affects state factor i. The skill policy π(a|s, z) takes in z ∈ Z, which is a composition of the skill components.While the factored latent space Z could be discrete or continuous, we consider discrete skill space in this paper, and discuss how DUSDi can be applied to continuous skills in Appendix A. We can then assume that each disentangled component z i takes the form of an integer, z i ∈ [1, k], resulting in a compositional skill, z, with the form of a N -dimensional multi-categorical vector with k N possible values. During skill learning, we independently sample each disentangled component z i from a fixed uniform distribution p(z i ), similar to previous works [13, 35] .Given this factored skill space, our goal is to learn a skill policy network, π θ : S × Z → A, such that each disentangled component Z i affects and only affects the value of a state factor, S i . For each disentangled component and state factor pair (Z i , S i ), we encourage diverse and distinguishable behaviors by maximizing their mutual information I(S i ; Z i ). While this objective enables a disentanglement skill component to affect the corresponding factor, it does not restrict the component from affecting other factors. This is undesirable since the resulting skill components would still be entangled in their effects. To prevent that, we propose to ensure that each skill component, Z i , minimally affects the rest of the state factors, S ¬i , where S ¬i denotes the subspace formed by all other state factor spaces except S i : S 1 × . . . , S i-1 × S i+1 × • • • × S N . Specifically, we incorporate an entanglement penalty to minimize, I(S ¬i ; Z i ), which corresponds to the mutual information between a skill component and all other state factors that it should not affect.Formally, the skill policy aims to maximize the following objective:EQUATIONwhere λ < 1 is a hyperparameter that controls the importance of the entanglement penalty relative to the skill-factor association. We restrict λ to be smaller than one for the following reason: in some environments, due to intrinsic dynamical dependencies between state factors themselves, controlling a state factor, S i , has to introduce some association between Z i and other factors in S ¬i , e.g., when controlling an object whose manipulation requires the agent to use other objects as tools. In these cases, as the policy learns to maximize the MI between a skill and a factor, I(S i , Z i ), the MI with other factors, I(S ¬i ; Z i ), may also increase. For these cases, the use of λ < 1 will ensure that the entanglement penalty does not overpower the association reward, and the policy is still incentivized to learn disentangled skill components that change S i distinguishably while introducing minimal changes on other factors. In practice, we simply set λ = 0.1 in all our experiments.Optimizing DUSDi's Objective: Directly maximizing the objective in Eq. 1 is intractable. Alternatively, we propose to approximate the objective using a variational lower bound of the mutual information [1] :EQUATIONwhere C represents the constant value of H(Z i ), the entropy of the prior distribution over the skill latent variable, which does not change during training, and q i ϕ is a variational distribution. Similarly, we can approximate the MI in the entanglement penalty by:EQUATIONwhere q i ψ is another variational distribution 3 . Importantly, when these q approximations perfectly recover the posterior distribution of z i , we obtain equality in Eq. 2 and Eq. 3. In DUSDi, we implement the variational distributions, q ϕ and q ψ , as neural network discriminators mapping input state factor(s) to the predicted disentangled component values, z i .To optimize J (θ), we alternate between two steps: 1) performing variational inference to train the discriminators q i ϕ and q i ψ through gradient ascent, and 2) using q i ϕ and q i ψ to learn a disentangled skill policy π θ through RL by maximizing the following intrinsic reward approximating Eq. 1:EQUATIONInterestingly, the decomposed nature of our intrinsic reward allows a convenient avenue for shaping skill behaviors based on domain knowledge. In particular, we can restrict a state factor s i to only take certain values by constraining q i ϕ (z i |s i ) accordingly. While not the main focus of this work, we briefly explore this further optimization enabled by DUSDi in Appendix H.When using reinforcement learning (RL) to optimize the intrinsic reward function defined in Eq. 4, standard RL algorithms treat the reward function as a black box and learn a single value function from the mixture of intrinsic reward terms. While this approach may be sufficient for environments with few state factors, doing so for complex environments with many state factors (large N ) often leads to suboptimal solutions. A key reason is that the mixture of 2N reward terms leads inevitably to high variance in the reward, making the value of the Q function oscillate. Furthermore, the sum of reward terms obscures information about each term's value, which hinders credit assignment.DUSDi overcomes this issue by leveraging the fact that the intrinsic reward function in Eq. 4 is a linear sum over terms associated with each disentangled component. Thanks to the linearity of expectation, we can decompose the Q function into N disentangled Q functions as follows:EQUATIONwhere Q i represents each disentangled Q function, one for each disentangled component. The disentangled Q functions can be then updated only with their corresponding intrinsic reward terms,r i ≜ q i ϕ (z i |s i ) -λq i ψ (z i |s ¬i ).During policy learning, we sum all disentangled Q functions together to recover the global critic, Q π , as shown in Fig. 2 (a), top. Compared to learning Q π directly from all 2N reward terms, learning disentangled Q functions significantly reduces reward variance, allowing Q π to converge faster and more stably.Similar to Eysenbach et al. [13] , in DUSDi we utilize hierarchical RL to solve reward-supervised downstream tasks with the discovered skills, as depicted in Fig. 2 (b ). The skill policy, π θ : S × Z → A, acts as the low-level policy and is kept constant while a high-level policy, π high : S → Z, learns to select which skill to execute for L steps using the skill latent variable, z. Thus, the skill latent conditioning space, Z, acts as the action space of the high-level policy, π high . As extensively evaluated in our experiments, without any additional "ingredient", performing downstream task learning in the action space formed by DUSDi skills often results in significantly superior performance compared to an action space formed by entangled skills. We show that the superior performance of DUSDi can be explained by more efficient exploration when using the DUSDi skills for hierarchical RL, which we elaborate on in Appendix B, through analyzing the benefits and search complexity of DUSDi's skill space over DIAYN's.Depending on the nature of the downstream tasks, we can often take further advantage of the disentangled skills learned by DUSDi through leveraging its structure. One such scenario is when the downstream task has a composite reward function consisting of multiple terms. Previous works [16, 46] have shown that when the causal dependencies from action dimensions to reward terms are available (e.g., the reward for speed only depends on actions that affect speed), one can use Causal Policy Gradient (CPG) to decompose the policy update (e.g., only the "speed actions" get updated by the speed reward) and greatly improve sample efficiency, especially when the dependencies are sparse. In downstream task learning, with an action space (of the high-level policy) consisting of the skills learned by DUSDi, we have a convenient way of applying causal policy gradient, where the causal dependencies between the action dimensions (i.e., skill components) and reward terms are often sparse and can be easily obtained by examining the state factor that a skill component is associated with, which we evaluate empirically in Sec. 4.6.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We provided our codebase with detailed instructions. This paper does not use any existing dataset.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental evaluation: 
 In the evaluation of DUSDi, we aim to answer the following questions: Q1: Are skills learned by DUSDi truly disentangled (Sec. 4.2)? Q2: Can Q-decomposition improve skill learning efficiency (Sec. 4.3)? Q3: Do our disentangled skills perform better when solving downstream tasks compared to other unsupervised reinforcement learning methods (Sec. 4.4)? Q4: Can DUSDi be extended to image observation environments (Sec.4.5)? Q5: Can we leverage the structured skill space of DUSDi to further improve downstream task learning efficiency (Sec.4.6)? Previous works [13, 34, 35, 44, 23] extensively rely on standard RL environments such as DMC [48] and OpenAI Fetch [4] to evaluate unsupervised RL methods. However, unlike previous unsupervised skill discovery methods, DUSDi focuses on learning a set of disentangled skill components that can be concurrently executed and re-combined to complete downstream tasks. As such, it only makes sense to examine the performance of DUSDi in challenging tasks that require concurrent control of many environment entities (e.g. multi-agent systems, complex household robots). Previous environments lack this property: in DMC for example, while the state and action space can be very complex, the predominant downstream tasks are just to move the center-of-mass of the agent to different places. In such cases, there is no need for concurrent skill components, and therefore we do not expect large gains from using DUSDi's disentangled skills. Nevertheless, we include an evaluation on the DMC-Walker [48] environment to demonstrate that our method is also applicable to those environments, but focus the majority of our evaluation on environments that DUSDi is designed for, including 2D Gunner, Multi-Particle [30] , and iGibson [26] .The 2D gunner is a relatively simple domain, where a point agent can navigate inside a continuous 2D plane, collecting ammo and shooting at targets. Multi-Particle is a multi-agent domain modified based on [30] . In this domain, a centralized controller simultaneously controls 10 heterogenous point-mass agents to interact with 10 stations, where each agent can only interact with a specific station. We evaluate in this domain to test the scalability of our methods to a large number of state factors. iGibson [26] is a challenging simulated robotics domain with the same action space and complexity as real-world robots, where a mobile manipulator can navigate in a room, inspect the room using its head camera, and interact with electric appliances in the room by pointing a remote control to them and switching them on/off. We evaluate in this domain to examine whether our method can handle home-like environments with complex dynamics. We provide visualizations and additional information about each of the environments in Appendix C.First, we examine whether the skills learned by DUSDi are truly disentangled (Q1) using the DCI metric proposed by Eastwood and Williams [12] . The DCI metric consists of three terms, namely disentanglement, completeness, and informativeness, explained in detail in Appendix F. In the original work, measuring DCI requires knowing the ground truth generative factors. In our case, the generative factors are simply the state factors, and we only need to discretize the value of each state factor to make it compatible for evaluation. For each method on each domain, we collect 100K rollout steps using the learned skill policy, π(s, z), where the skill is (re)sampled from the uniform prior distribution, p(z), every 50 steps. These (state, skill) pairs are then used to calculate DCI.We compare against DIAYN-MC (Multi-channel DIAYN) that uses the same skill representation as DUSDi but optimizes the DIAYN objective of I(S; Z), and show results in Table 1 . Unsurprisingly, DUSDi significantly outperforms DIAYN-MC, especially on Disentanglement and Completeness, across all three environments. These results indicate that DUSDi learns truly disentangled skills, enabling efficient downstream task learning, as we will show in Sec. 4.4. We encourage the readers to visit our project website for a qualitative visualization of the learned skills.To examine the importance of Q-decomposition (Q2), we measure the performance of optimizing the DUSDi objective during skill learning with and without a decomposed Q network. We compare the classification accuracy of the skill discriminators q i ϕ (z i |s i ), averaged over all skill channels, which indicates progress towards discovering diverse and distinguishable skills, with higher accuracy being better. We depict our results in Fig. 3 . We observe that Q-decomposition has a similar performance to the regular Q network in the simplest 2D gunner domain, but significantly outperforms the regular Q network in domains with more state factors (Multi-Particle) and more complex dynamics (iGibson), suggesting that Q-decomposition is necessary for scaling towards complex domains.The promise of DUSDi is to incorporate disentanglement into skills so that the skills can be effectively used in downstream task learning. Therefore, the most critical evaluation of our work focuses on comparing the performance of different unsupervised RL methods on task learning (Q3). We compare against existing state-of-the-art unsupervised reinforcment learning algorithms, including DIAYN [13] , CIC [24] , CSD [35] , METRA [36] , ICM [37] , RND [5] , ELDEN [51] , and Vanilla RL [14] , where these baselines are further explained in Appendix E.Similar to the evaluation setting in the URLB benchmark [23] , we allow each method to train for 4 million steps without access to reward (i.e., pretraining phase) before the reward is revealed to the agent and the downstream learning takes place. During the pre-training phase, all methods use soft actor-critic (SAC) [14] to optimize the intrinsic reward. For all skill discovery methods (i.e., DUSDi, DIAYN, CIC, CSD, METRA), a skill-conditioned policy, π θ (a|s, z), is learned during the pretraining phase. During downstream learning, the skill network is fixed, whereas an upper policy, π high (z|s), is trained using proximal policy optimization (PPO) [43] to optimize the task reward. Similar to previous works [13, 44] , we omit proprioceptive states from the MI optimization for all skill discovery methods to facilitate more meaningful explorations. For exploration methods (i.e., RND, ICM, ELDEN), a policy π θ (a|s) is learned during the pretraining phase on intrinsic reward and fine-tuned using the task reward during the downstream learning phase. The hyperparameters are specified in Appendix G.We evaluate all methods in four environments and 13 downstream tasks, detailed in Appendix D. The results are depicted in Fig. 4 . As expected, DUSDi performs similarly to previous unsupervised RL methods in the DMC walker environment due to the simplicity in terms of its downstream objectives (all related to center-of-mass locomotion), but significantly outperforms all previous methods on domains where downstream tasks require coordinative control of multiple state factors. The most crucial comparison is between DUSDi and DIAYN. DIAYN is a special case of DUSDi where there is only one state factor (consisting of the entire state) and one skill component. Therefore comparing against DIAYN offers a straightforward examination of the effect of disentangled skills for downstream task learning. DUSDi significantly outperforms DIAYN in all downstream tasks, demonstrating the effectiveness of using disentangled skills. In general, we found exploration-based methods to be less capable than skill discovery methods, possibly due to their lack of temporal abstraction. CIC performs very poorly, likely because the CIC objective does not explicitly encourage distinguishable skills and instead generates the intrinsic reward solely based on state entropy, making it very hard for the upper policy to select the right skill. This result again shows the importance of having a proper skill representation. DUSDi also outperforms CSD and METRA on most downstream tasks, especially on the more complex and high-dimensional domains, like Multi-Particle. This superiority is perhaps surprising considering that in our experiments, DUSDi only relies on the simple DIAYN-style intrinsic reward for skill discovery, but further demonstrates the importance of learning a disentangled skill space. It is important to notice that many techniques proposed to improve skill discovery quality DUSDi outperforms all baselines that learn entangled skills, converging faster and to higher returns.(e.g., Baumli et al. [3] , Zhao et al. [57] ), can be seamlessly incorporated into DUSDi. Therefore, we expect our method to perform even better as new advances are made in unsupervised skill discovery.Although this paper primarily focuses on applying DUSDi to factored state space, we can straightforwardly extend it to image space through existing works in factored / object-centric representation learning [29, 20, 53, 28, 55 ] (Q4). We empirically illustrate this capability in the Multi-Particle environment, where we replace the low-dimensional state observation with 64 × 64 image observations. Specifically, we first pretrain an object-centric encoder following Yang et al. [55] , and then use our method on top of the extracted representation to learn disentangled skills. Hence, essentially, the skill policy uses images as observation. As shown in Fig. 5 , when learning from image observation, DUSDi achieves similar performance to learning from state space, whereas the baseline methods are unable to learn these two tasks even when learning from the low-dimensional state space as in Fig. 4 .While DUSDi can already learn downstream tasks quite efficiently, it is possible to further improve the sample efficiency of downstream task learning through leveraging the structured skill space of DUSDi (Q5), as described in the second paragraph of Sec.3.3. Specifically, we apply Causal Policy Gradient [16] to the Multi-Particle domain, where the causal dependencies between state factors and reward terms are easy to identify. We present our results in Fig. 6 , where the sample efficiency of downstream task learning is greatly improved thanks to the structured skill space of DUSDi. 
related work: 
 Unsupervised Skill Discovery In unsupervised skill discovery, the goal of an agent is to learn task-agnostic skills without external rewards. To learn such skills, previous methods propose various forms of intrinsic reward: (1) maximizing the mutual information between visited states and the skill variables [13, 44, 6, 24] , (2) maximizing the traveled distance along the direction specified by the skill variables [34] [35] [36] , (3) learning to reach a diverse set of goals [52, 40, 38] . These skills can be used to boost the sample efficiency of downstream task learning, for example, (1) using hierarchical RL where a high-level policy learns to select which skill to execute [13] , or (2) using the skill policy to initialize the task solving policy and then fine-tuning it [24] . State Space Factorization in RL In RL, there is a long history of leveraging state factorization, including learning a world model between state factors for planning [22, 50] , augmenting data [39] , and providing intrinsic rewards [42, 17, 9] . Relevant to our work are skill discovery methods that learn to either reach a goal for each controllable object [19, 9] or achieve interactions between a pair of specified objects [8] . Though these methods achieve disentanglement by influencing one or a pair of objects during a skill, they do not apply to tasks that require controlling multiple objects simultaneously, like driving where we need to control the car's speed and heading directions at the same time. In contrast, our method can combine disentangled skill components into concurrent skills [10] to solve a wide range of tasks.Inspired by the benefits of compositionality, disentanglement has been extensively studied, mainly in learning image representations [28] . There are a few works investigating disentanglement in unsupervised skill discovery. Lee et al. [25] consider a special case of disentangled skills -for a multi-arm robot, learning independent skills for each arm. However, they rely on manually factored action spaces which is an assumption that often limits the behavior of the agent. Kim et al. [21] encourage the disentanglement between different dimensions of the skill variable by regularizing it with β-VAE objective [15] , but Locatello et al. [27] point out that such regularization is impossible to achieve disentanglement. To learn disentangled skills, Song et al.[45] learns a decoder from skill variables to state trajectories and their generation factors, which is then used to train the skill policy through imitation learning. However, their training of the decoder requires pre-collected trajectories and corresponding generation factors, whereas our method is fully unsupervised with no expert data. 
conclusion: 
 We present DUSDi, an unsupervised skill discovery method for learning disentangled skills by leveraging the factorization of the state space. DUSDi designs a skill space that exploits the factorization of the state space and learns a skill-conditioned policy where each sub-skill affects only one state factor. DUSDi enforces disentanglement through an intrinsic reward based on mutual information, and shows superior performance on a set of downstream tasks with naturally factored state spaces compared to baselines and state-of-the-art unsupervised RL methods.One limitation of DUSDi is the assumption of access to a factored state space. While a factored state space is naturally available in many existing RL environments, and can be extracted from images as we have shown in our experiment (Sec. 4.5), we believe that future advances in disentangled representation learning will greatly broaden the applicability of DUSDi towards partially observable, pixel-based environments. Secondly, DUSDi primarily focuses on learning a structured skill space for more efficient downstream learning, and its exploration capability during skill learning is largely determined by the specific algorithm used to optimize for our mutual information objective. While we used DIAYN [13] in this work due to its simplicity, it would be interesting to examine extending the idea of learning disentangled skills to other skill discovery methods, e.g., Zhao et al. [57] , Laskin et al. [24] , including those that are not based on mutual information [35, 56] .Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All hyperparameters are specified in the Appendix G.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We include standard deviation for all quantitative our results (Sec. 4).Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide computer resources in the Appendix G.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We explicitly cites all environments that we used in this paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: For continuous skills, we can simply define each skill component z i as a m-dimensional continuous vector (and therefore the length of the skill would be m × N , where N is the total number of skill components). Now we can define the prior distribution for each skill component distribution p(z i ) as an m-dimensional continuous uniform distribution (e.g. U[-1, 1]). Notice that our objective remains unchanged, as the MI is well-defined no matter whether the skill is discrete or continuous. Lastly, we need to change the output head of our skill prediction networks q, such that instead of outputting a categorical distribution, it will output a continuous probability distribution (e.g. a multivariate Gaussian distribution with diagonal covariance).Compared to entangled skills, the advantages of using disentangled components mainly reside in an easier exploration in the skill space. For skill spaces of equivalent capacity, the DIAYN latent skill variable is a single integer between 1 and k N , and the DUSDi skill variable is a N -dimensional vector with each dimension representing a disentangled component with k possible values. In this section, we analyze the benefits and search complexity of DUSDi's space over DIAYN's for two main cases: when there are no dynamical dependencies between state factors (optimal case for disentangled components) and where there are intrinsic dependencies between state factors.State Factors without Dynamical Dependencies: In this case, for DIAYN to find the correct skill to execute at the current time step, in the worst case, it needs to iterate through all skills, resulting in 1-step exploration sample-efficiency of O(k N ). In contrast, for DUSDi, as disentangled components are independent of each other, with one skill trial, the agent can simultaneously observe the effects of setting each disentangled component as Z i = z i . Hence, for an intelligent agent, to understand the effects of each disentangled component at the current state, it only needs to sweep through each disentangled component space with k trials (e.g., setting all disentangled components Z i = 1, . . . , k). After that, as the effects of each disentangled component are independent, by compositing disentangled components in novel ways, the agent has the ability to imagine the effects of all skills, leading to O(k) exploration efficiency.State Factors with Dynamical Dependencies: When there are dynamical dependencies, we denote PA i as parent indices of state factors that S i depends on, e.g., when moving a mouse (S i ), S PA i denotes the hand. In such cases, the effect of Z i is conditioned on the value of Z PA i , and we need to iterate through all (Z i , Z PA i ) pairs to observe all possible influences on S i . As a result, the exploration is constrained by the state factor with the largest number of parents. Denoting |PA i | as the number of parent factors for S i , the exploration sample-efficiency is O(k 1+maxi |PA i | ). We can see that the O(k) efficiency when there is no dynamical dependencies is a special case of max i |PA i | = 0. Despite lower efficiency than O(k), in many environments, the dynamics of each state factor only depend on a small number of other factors, i.e., max i |PA i | ≪ N . Hence, exploration with disentangled components is still more sample-efficient than using entangled skills.We test DUSDi on four environments, where a visualization of each of the environments is presented in Fig. 7 .2D Gunner: Shown in Fig. 7 (a), the blue star marks the position of the agent, the blue line marks its shooting direction, the red diamond marks ammo location, and the orange cross marks the target position. The agent has a 7-dimensional observation space, consisting of 3 state factors: [Agent Position, Ammo State, Target State]. The action is 5-dimensional, 2 for agent movement, 2 for ammo pickup, and 1 for shooting direction. DMC-Walker (Walker):• Run: In this task, the walker agent is rewarded for moving forward at a particular velocity.• Goal Reaching: In this downstream task, the agent has to reach randomly generated goal positions.• Unlimited Ammo (unlim): In this downstream task, a set of targets will randomly appear, where the agent needs to navigate to a position close to the target and shoot them in order to score. The ammo is unlimited so the agent does not need to worry about picking up ammo.• Limited Ammo (lim): This downstream task is different from the "unlimited ammo" in that the agent starts with no ammo and needs to pick up ammo in order to shoot. Everything else is identical.• Sequential interaction (seq) (easy, medium, hard): In this task, agents need to sequentially interact with their corresponding station following an instruction sequence given at the start of each episode. Interacting with stations in the wrong order will be penalized. The easy version of this task has a sequence length of 2, while medium and hard have a sequence length of 5 and 8 respectively.• Food-poison (fp) (easy, medium, hard): In this downstream task, each station will offer either food or poison to the corresponding agent. Each agent needs to decide whether to interact with its corresponding station based on a sequence of binary indicators provided to the agents. The difficulty level has the same meaning as in the sequential interaction task.• Look around: In this task, the robot needs to look at objects in the room sequentially.• Appliances inspection: In this task, the robot needs to navigate to different electric appliances, and test whether each of them is working correctly by pointing a remote control towards it.• Housekeeping: In this task, the robot needs to manage the electric appliances intelligently. Specifically, the robot needs to first look at a screen to receive instructions. Depending on the instruction, the robot needs to turn on / off certain electric appliances using the remote control.During downstream task evaluation, we compared against the following state-of-the-art unsupervised RL methods:• DIAYN [13] represents skill variable z as an integer between 1 to k N and learns skills by maximizing I(S; Z), the MI between Z and all state factors S. • CIC [24] learns a state representation with contrastive learning and learns skills by maximizing transition entropy in the representation space. • CSD [35] learns skills maximizing distance traveled along the direction of z in the state space, where distance is measured in a controllability-aware manner. • METRA [36] learn a set of behaviors that collectively cover as much of the state space as possible through optimizing a Wasserstein variant of the state-skill Mutual Information. • ICM [37] : encourages visiting novel states by using prediction errors of action consequences as intrinsic rewards. • RND [5] encourages visiting novel states by using prediction errors of features computed from a randomly initialized network as intrinsic rewards.• ELDEN [17] operates in a factored state space similar to our approach, and encourages visiting states that induce novel factor dependencies. • SAC [14] where no pretraining is used, and vanilla RL is directly applied to tackle the downstream tasks.The DCI metric consists of three terms, namely disentanglement, completeness, and informativeness. In the context of this work, disentanglement (↑) measures, on average, to what extent each skill component only affects a single state factor. Completeness score (↑) measures, on average, to what extent each state factor is only influenced by a single skill component. Informativeness score (↑) measures the repeatability of learned skills: given the skill z, how accurately we can predict which states will be visited. We refer the reader to the work by Eastwood and Williams [12] for a detailed discussion of these metrics and how they are calculated.Skill Dimensions: For all skill learning methods with discrete skills (i.e. DUSDi, DIAYN), we make sure that they have equivalent capacity. Specifically, for igibson and 2D gunner, each DUSDi skill consists of 3 skill components, each component with 5 possible values. As a result, DIAYN skill is an integer between 1 to 125 in these two domains. The only exception is Multi-Particle, where DUSDi has ten sub-skills, each with 5 possible values. Since skill as an integer between 1 and 5 10 = 9765625 is obviously challenging for DIAYN to converge, we set the number of discrete skills to be 4096 for DIAYN. For continuous skills (i.e. CSD, CIC, METRA), we follow the skill dimensions specified in the original papers (64D for CIC, 3D for CSD and METRA), which were shown to be effective for the respective methods.Skill Learning Parameters: All skill learning methods in our baselines use SAC to optimize for the intrinsic reward, with the same policy and value network architecture. DUSDi applies additional decomposition and masking to the value networks, as described in Section. 3.2, which is not applicable to the baseline methods. Due to Q-decomposition, when using the same value network architecture, DUSDi's value network capacity is N times of the capacity of other methods' value networks (including when comparing the variations of DUSDi, i.e., no decomposition). For a fair comparison, we also tried to increase value network capacity for other methods to match the capacity for DUSDi, but found that their skill/task learning performances do not improve significantly. This suggests (1) that, for skill learning, reward variance, rather than network capacity, is the key reason for no Q-composition variation of DUSDi to converge slowly, and (2) that, for task learning, disentangled skills, rather than network capacity, is what make DUSDi significantly outperform baselines.We present the hyperparameters for SAC in Table . 2. All methods use a low-level step size of L = 50. Downstream Hierarhical Learning: For all skill discovery methods, downstream learning of the skill selection policy is implemented with PPO. We used the same hyperparameters for all methods across all tasks, as specified in Table . 3. Downstream Finetuning: For all non-skill discovery methods, downstream learning is done using the same hyperparameters as pretraining (table . 2), replacing the intrinsic reward with the task reward.Due to the decomposable nature of the intrinsic reward of DUSDi, we can conveniently restrict the behavior of skills by constraining the skill predictor q i ϕ (z i |s i ) for a particular state factor i. For example, if we want s i to stay within a certain range, we can set q i ϕ (z i |s i ) to be a uniform distribution for all s i not within this range, effectively discouraging the agent from going out of range. In the extreme case, we can fully specify the mapping between z i and s i , essentially resulting in performing goal-conditioned RL for state i (as also pointed out in [7] ) while performing DUSDi for the rest of the state factors.We qualitatively examine this idea in the iGibson domain. By restricting a mobile manipulator to only locomote in regions that are close to a whiteboard, our robot successfully learns diverse board-wiping behaviors which are otherwise extremely hard to learn. Visualizations of the learned skills can be seen on our project website.† Equal supervision.While lower-bounding J (θ) requires upper-bounding I(S ¬i ; Z i ), we stick with the variational lower bound for this term because of the complexity in upper bounding MIs[41]. 


Decision: The paper introduces DUSDi, a method for learning disentangled skills in unsupervised environments that can be efficiently reused and chained to solve downstream tasks. The idea presented in the paper is significant, well-executed, and introduces novelties around the disentanglement of skills which could potentially be scaled to more complex settings. The primary concern is that the key novelty (Equation 4) should be more strongly supported from a theoretical standpoint, although the authors provide a reasonable justification surrounding this point, including previous works which have incorporated similar approximations. Apart from this, the discrete nature of skill spaces is fairly common, and is as such, not viewed as a major limitation/flaw, although means to extend the idea to continuous skill spaces is welcome. The proposed method also demonstrates that it can be applied in pixel-based environments, which demonstrates its general applicability. Although D4RL was suggested, it is a baseline commonly employed for offline methods and is as such, not applicable to the method in question.

Reviews: review: paper_summary: paper_summary: This paper proposed DUSDi, which learned disentangled skills in an unsupervised manner. Specifically, DUSDi learns each skill component only affects one factor of the state space. DUSDi is evaluated in a specialized tasks comparing with skill discovery methods.
strength_weakness: strength_weakness: - It is not convincing how the disentanglement is beneficial in real-world scenarios.
- The empirical evaluation is limited; standard benchmarks (e.g., D4RL) is not used, and it is questionable whether it can be used in more complex environments like pixel-based environments
questions: questions: See weaknesses.
limitations: limitations: See weaknesses.
paper_summary: paper_summary: Grounding on previous work on mutual information for skill learning, DUSDi proposed an algorithm for disentangled skill learning, which introduces several advances. First, DUSDi proposes to map factorized state components to factorized skill components. Then, it proposes the adoption of Q decomposition and Causal Policy Gradient to further enhance stability and efficiency.
strength_weakness: strength_weakness: * **State space factorization**: the approach works well thanks to the assumption that the state space is well-factorized. However, for POMDPs or high-dimensional environments, the algorithm may not work as expected. The authors show that object-centric approaches can obviate such weakness. This works well for simple environments where colours can easily help distinguish objects, e.g. the multiparticle env, but it may not work as well in more complex environments.
questions: questions: * the authors explicitly state that their objective is not optimizing a lower bound (line 150). Why not trying to actually keep a bound? e.g. following [1]
* since the authors discuss the unsupervised RL benchmark, and they use a similar setup, why not presenting results on the popular benchmark? I understand their point about the fact that these environments mostly require manipulating a single variable, but being a more standard benchmark, it would have been useful to see complete results on it (not just in the Walker env)
* what happens if you don't omit proprioceptive states from the MI optimization? (line 264-265)


[1] On Variational Bounds of Mutual Information, B. Poole et al,
limitations: limitations: The limitations of the approach are addressed in the paper.
paper_summary: paper_summary: The paper presents a method (DUSDi) for learning reusable skills through unsupervised interactions. Unlike existing methods that produce entangled skills, DUSDi focuses on disentangling skills into components that each affect only one factor of the state space. This enables efficient chaining of skills via hierarchical reinforcement learning. A mutual-information-based objective ensures skill disentanglement, and value factorization optimizes the process.
strength_weakness: strength_weakness: 1. The paper assumes that the skill space is discrete; there is little information on how the formulations generalize to continuous skill space. 
2. There is a lack of visualization of the learned skill embeddings, and how different skills affect specific factors of the state space, highlighting the disentangled nature, which makes it hard to determine the contribution of the proposed method to the overall performance gain over various baselines.
questions: questions: Please address my concerns in the Weakness section
limitations: limitations: NA
paper_summary: paper_summary: This paper proposes a method for learning disentangled skills that can be efficiently reused to solve downstream tasks. The mutual information objective design is simple yet effective. Intensive empirical results show the superiority of the proposed method.
strength_weakness: strength_weakness: (a) The algorithmic contribution is not significant compared with other MI-based unsupervised skill discovery methods.

(b) The key objective design lacks theoretical support. As the author mentioned, Eq. (4) does not constitute a lower bound of the real objective.

(c) A qualitative study to show if the learned skill embedding is semantically detangled would be beneficial.

(d) The submitted code folder should include a readme file on how to reproduce the paper results.
questions: questions: Please see the weakness part.
limitations: limitations: Provided.

Comments: comment: Thank you addressing my concerns. I will maintain my rating.
We would like to express our gratitude for carefully reading our paper and providing valuable feedback. These suggestions are very constructive and have definitely helped us improve the quality of our paper in meaningful ways. Thank you!
Thank you for your detailed feedback. I will maintain my rating, as only minor revisions have been made. The technical contribution and novelty of this paper are still borderline. Also, I suggest providing codes and instructions to reproduce ALL results in the paper.
I am happy with the author's rebuttal.

I recommend they add some of the insights/comments provided here to the paper, as these may be useful for future readers, to have better insights into their work and its extendability.

I am also glad they are adding experiments on the Quadruped domain of URLB, where I see their method obtains satisfactory performance (though not very outstanding).

I will keep my score and recommend acceptance of the work.
We would like to friendly remind the reviewers that our rebuttal contents and additional results can be seen in replies to each individual reviewer. We hope our rebuttal addresses your concern, and please kindly let us know if you have any further questions.