SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions

Authors: zizhao  wang
jiaheng  hu
caleb  chuck
stephen  chen
roberto  martín-martín
amy  zhang
scott  niekum
peter  stone
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 03:09:02

Abstract: Unsupervised skill discovery carries the promise that an intelligent agent can learn reusable skills through autonomous, reward-free interactions with environments. Existing unsupervised skill discovery methods learn skills by encouraging distinguishable behaviors that cover diverse states. However, in complex environments with many state factors (e.g., household environments with many objects), learning skills that cover all possible states is impossible, and naively encouraging state diversity often leads to simple skills that are not ideal for solving downstream tasks. This work introduces Skill Discovery from Local Dependencies (SkiLD), which leverages state factorization as a natural inductive bias to guide the skill learning process. The key intuition guiding SkiLD is that skills that induce \textbf{diverse interactions} between state factors are often more valuable for solving downstream tasks. To this end, SkiLD develops a novel skill learning objective that explicitly encourages the mastering of skills that effectively induce different interactions within an environment. We evaluate SkiLD in several domains with challenging, long-horizon sparse reward tasks including a realistic simulated household robot domain, where SkiLD successfully learns skills with clear semantic meaning and shows superior performance compared to existing unsupervised reinforcement learning methods that only maximize state coverage.

Main paper: 
 introduction: 
 Reinforcement learning (RL) achieves impressive successes when solving decision-making problems with well-defined reward functions [65, 20, 33] . However, designing this reward function is often not trivial [7, 61] . In contrast, humans and other intelligent creatures can learn, without external reward supervision, behaviors that produce repeatable and predictable changes in the environment [18] . These behaviors, which we call skills, can be later repurposed to solve downstream tasks efficiently. One of the promises of this form of unsupervised RL is to endow artificial agents with similar capabilities to discover reusable skills without explicit rewards.One predominant strategy of prior skill discovery methods focuses on training skills to reach diverse states while being distinguishable [19, 59, 50] . However, in complex environments that contain many state factors-distinct elements such as individual objects in a household (a formal description in Sec. 2.1), the exponential number of distinct states makes it impossible to learn skills that cover every state. Consequently, these methods typically result in simple skills that only change the easy-to-control factors (e.g., in a manipulation task moving the agent itself to diverse positions or Figure 1 : Skill Discovery from Local Dependencies (SkiLD) describes skills that encode interactions (i.e., local dependencies) between state factors. In contrast to prior diversity-based methods that can easily get stuck by moving the robot to diverse, but non-interactive states, and factor-based methods that are trained to manipulate the hammer and nail, but not their interactions, SkiLD not only manipulate each object (left, middle) but also induce interactions between them (right), by specifying different local dependencies. These skills are often more useful than the "easy" skill learned by previous methods for downstream task-solving.manipulating each factor independently), and fail to cover other desirable but challenging behaviors. Meanwhile, in a factored state space, many downstream tasks require inducing interactions between state factors, e.g., cooking requires using a knife to cut the ingredients and cooking them in a pan, etc. Unsurprisingly, these simple skills often struggle to solve such tasks, resulting in poor downstream performance.Our key insight is to utilize interactions between state factors as a powerful inductive bias for learning useful skills. In factored state spaces and their downstream tasks, there usually exist bottleneck states that an agent must pass through to explore different regions of the environment, and many of them can be characterized by interactions between state factors. For example, in a household environment, a robot must first grasp the knife before moving it to different locations, with the bottleneck being the interaction between the robot and the knife. In environments that have a large state space due to many state factors, rather than inefficiently relying on randomly visiting different states to reach such bottlenecks, we propose to train the agent to actively induce these critical interactions.To this end, we introduce Skill Discovery from Local Dependencies (SkiLD), a novel skill discovery method that explicitly learns skills that induce diverse interactions. Specifically, SkiLD models the interactions between state factors using the framework of local dependencies (where local refers to state-specific, see details in Sec. 2.2) and proposes a novel intrinsic reward that 1) encourages the agent to induce specified interactions, and 2) encourages the agent to discover diverse ways of inducing specified interaction, as visualized in Figure 1 . During skill learning, SkiLD gradually discovers new interactions and learns to induce them, based on the skills that it already mastered, resulting in a diverse set of interaction-inducing behaviors that can be readily repurposed for downstream tasks. During task learning, the skill policy is reused, and a task-specific policy is learned to select (a sequence of) skills to maximize task rewards efficiently.We evaluate the performance of SkiLD on factor-rich environments with 10 downstream tasks against existing unsupervised reinforcement learning methods. Our experiments indicate that SkiLD learns to induce diverse interactions and outperforms other methods on most of the examined tasks. 
background: 
 In this paper, our unsupervised skill discovery method is set up in a factored Markov decision process and builds off previous diversity-based methods, as described in Sec. 2.1. To enhance the expressivity of skills, our method further augments the skill representation with interactions between state factors, which we formalize as local dependencies as described in Sec. 2.2.We consider unsupervised skill discovery in a reward-free Factored Markov Decision Process [8] defined by the tuple M = (S, A, p). S = S 1 × • • • × S N is a factored state space with N subspaces, where each subspace S i is a multi-dimensional continuous or discrete random variable. Then, correspondingly, each state s ∈ S consists of N state factors, i.e., s = (s 1 , . . . , s N ), s i ∈ S i . In this paper, we use uppercase letters to denote random variables and lowercase for their specific values (e.g., S denotes the random variable for states s). A is the action space, and p is an unknown Markovian transition model that captures the probability distribution over the next state S ′ ∼ p(•|S, A).The factorization in S inherently exists in many environments, and is a common assumption in prior unsupervised skill discovery works [22, 29, 27] . For example, in robotics, an environment typically consists of a robot and several objects to manipulate, and, for each object, S i would represent its attributes of interest, like pose. In this work, we explore how we can utilize a given state factorization to improve unsupervised skill discovery. In practice, the factorization can either be directly provided by the environment or obtained from image observations with existing disentangled representation learning methods [47, 31] .Following prior work, our method consists of two stages-skill learning and task learning. During the skill learning phase, we seek to learn a skill policy π ω (•|s, z), which defines a conditional distribution over actions given the current state s and some skill representation z, where skills indicate the desired behaviors of the agent. Once the skills are learned, they can be chained together to solve downstream tasks during the task learning phase through an extrinsic reward-optimizing policy. During task learning, a downstream task reward function r : S × A → R is provided by the environment. A high-level policy π(z|s) is then trained to optimize the expected return through outputting correct skills z given state s.A key insight of SkiLD is to utilize interactions between state factors (or, formally, local dependencies) as part of the skill representation. In later sections, these local dependencies are compiled into a binary matrix G(s, a, s ′ ) = {0, 1} N ×(N +1) representing the local dependencies between all factors. In this section, we first formally define local dependencies, introduce their identification, and finally discuss their application to factored MDPs.SkiLD takes a causality-inspired approach for defining and detecting local dependencies [6, 58] , where we use local to refer to a particular assignment of values for a random variable, as opposed to global which applies to all values. Formally, for an event of interest Y = y and its potential causes X = (X 1 , . . . , X N ), given the value of X = x, local dependencies focus on which X i s are the state-specific cause of the outcome event Y = y (for simplicity of presentation, in this section we overload N as the number of potential causes rather than number of variables and p as the transition function according to a subset of the variables). Formally, we denote the general data generation process of Y as p : X → Y and the data generation process when Y is only influenced by a subset of X as p X : X → Y , where X ⊆ X. Then, given the value of all variables, X 1 = x 1 , • • • , X N = x N and Y = y, we say Y locally depends on X, if X is the minimal subset of X such that knowing their values is necessary and sufficient to generate the result of Y = y, i.e.,EQUATIONwhere | X| is the number of variables in X. For example, suppose that a robot opens a refrigerator door in a particular transition. The event of interest Y = y is the refrigerator door becoming open, and it locally depends on two factors: the robot and the refrigerator door, while other state factors such as objects inside the refrigerator do not locally influence Y .To identify local dependencies, one can conduct a conditional independence test y ⊥ ⊥ x i |{x/x i } to examine whether a variable X i is necessary for predicting Y = y. In prior works, one form of this test is to examine whether the pointwise conditional mutual information (pCMI) is greater than 0, pCMI(y;EQUATIONIf so, then it suggests that knowing X i = x provides additional information about Y that is not present in {X/X i }, and Y locally depends on X i . As the data generation processes are generally unknown, one has to approximate them with learned models. Recent work in RL has utilized various approximations such as attention weights [53] , Granger causality [14] , and input gradients [63] .In this work, for a transition (S = s, A = a, S ′ = s ′ ), the event of interest is each next state factor being (S i ) ′ = (s i ) ′ , and we infer whether it locally depends on each state factor S j and the action A (i.e., whether there is an interaction between state factors i and j, where factor j influences i). Then we aggregate all local dependencies into a state-specific dependency graph (abbreviated in this work to dependency graph). This overall dependency graph is represented with G(s, a, s ′ ) = {0, 1} N ×(N +1) , and an edge G ij (s, a, s ′ ) denotes, during the transition (s, a, s ′ ), that state factor (s i ) ′ (the "Y = y") locally depends on s j (one of the X j ):EQUATIONThis graph is used to enhance skill representation, as explained in detail in Section 3.3 Skill Discovery from Local Dependencies (SkiLD)In this section, we describe SkiLD, which enhances skills using local dependencies. the skill policy π skill (a|s, z) and its corresponding reward function R skill , presented in Sec. 3.3.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We explicitly discuss the limitations of this work in the second paragraph of the Conclusion section. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
skill representation z: 
 Prior unsupervised skill discovery methods usually focus skill learning on changing the state or each factor diversely, which is inefficient when there exist bottleneck states for explorations. Consequently, they are can be limited to learning simple skills, for example, only changing the easiest-to-control factor in the state (i.e., the agent itself). To address this problem, SkiLD not only focuses on changing the state but also considers the interactions between state factors.Skill Representation. SkiLD represents the skill space as the combination of two components: Z = G × B,if i % L == 0 then 5:Sample the target dependency graph g ∼ π G (s). Collect state transitions (s, z, a, s ′ ) with actions from π skill (a|s, z).Infer the induced dependency graph g induced (s, a, s ′ ) using the dynamics model f (Sec. 3.1).Update the history of the seen graphs with g induced (s, a, s ′ ). Sample a batch of (s, z, a, s ′ ) from the replay buffer. Update the high-level policy with reward R G (Eq. 5) with the history of seen graphs.Update the discriminator q(b|s, g) with the discrimination (cross-entropy) loss.Infer the induced dependency graph g induced (s, a, s ′ ) using the dynamics model f (Sec. 3.1).Infer the diversity reward R diversity = log q(b|s, g).Update π skill with R skill = 1[g induced (s, a, s ′ ) = g] • (1 + λR diversity ) (Eq. 4). 20: end for are induced. Before we describe π skill training in Sec. 3.3, we first discuss how to select the skill z for π skill to follow during the skill learning stage.To acquire skills that are useful for downstream tasks, the skill policy π skill needs to learn to induce a wide range of local dependencies sample-efficiently. To this end, we propose to learn a graph-selection policy π G : S → G to guide the training of π skill . Specifically, training π skill requires a wise selection of graphs -as graph space G increases super-exponentially in the number of state factors N , many graphs are not inducible. To this end, we only select target graphs for the skill policy from a history of all seen graphs. As the agent learns to induce existing graphs in diverse ways, new graphs may be encountered, gradually expanding the set of seen graphs. However, though this history guarantees graph inducibility, two challenges still remain: (1) How to efficiently explore novel local dependencies, especially hard-to-visit ones? (2) For all seen graphs, which one should π skill learn next to maximize training efficiency? We address these challenges based on the following insight -compared to well-learned skills, π skill should focus its training on underdeveloped skills. Meanwhile, learning new skills opens up the possibility of visiting novel local dependencies, e.g., learning to grasp the hammer makes it possible for the robot to hammer the nail.According to this idea, we learn a graph-selection policy π G that guides the exploration and training of the skill policy π skill . Specifically, π G : S → G selects a new dependency graph the skill policy should induce for the next L time steps. To increase the likelihood of visiting hard graphs, π G is trained to maximize the following graph novelty rewardEQUATIONwhere C(g visited ) is the number of times that we have seen the graph in the collected transition.Given the skill parameter z from the graph-selection policy, SkiLD learns skills as a skill-conditioned policy π skill : S × Z → A, where π skill learns to reach diverse states while ensuring that the local dependencies specified by g are induced. During skill learning, we select actions by iteratively calling the skill policy π skill , and we denote g induced (s, a, s ′ ) as the graph that describes the local dependencies induced in a transition (s, a, s ′ ) when executing a selected action a. We design the reward function of the skill policy as:EQUATIONwhere 1[g induced (s, a, s ′ ) = g] measures whether the induced dependency graph matches the desired graph, R diversity is the weighted diversity reward that further encourages visiting diverse states when the desired graph is induced, and λ is the coefficient of diversity reward. In the following paragraphs, we describe how we infer g induced (s, a, s ′ ) and estimate R diversity for each transition.Inferring Induced Graphs. To infer the induced graph for a transition (S = s, A = a, S ′ = s ′ ), we need to determine, for each (S ′ ) i , whether it locally depends on each factor S j and the action A. Following Sec. 2.2, we evaluate the conditional dependency (s i ) ′ ̸⊥ ⊥ s j |{s/s j , a} by examining whether their pointwise conditional mutual information (pCMI) is greater than a predefined threshold ϵ. If pCMI ij = p((s i ) ′ |s,a) p((s i ) ′ |{s/s j ,a}) ≥ ϵ, it suggests that s j is necessary to predict (s i ) ′ and thus the local dependency exists. Meanwhile, as the transition probability p is unknown, we approximate it with a learned dynamics model that is trained to minimize prediction error.Finally, after obtaining the induced dependency graph, we evaluate 1[g induced (s, a, s ′ ) = g] by examining whether each edge g ij induced matches the corresponding edge in the desired graph g ij . As R skill only provides sparse rewards to the skill policy when the desired graph is induced, we use hindsight experience replay [2] to enrich learning signals, by relabelling induced graphs as desired graphs in some episodes.Diversity Rewards. When the skill policy induces the desired graph, R diversity further encourages it to visit different distinguishable states under different diversity indicators b, e.g., hammering the nail to different locations. This diversity enhances the applicability of learned skills. To this end, we design the diversity reward R diversity as the forward mutual information between visited states and the diversity indicator I(s; b), following DIAYN. To estimate the mutual information, we approximate it with a variational lower bound I(s; b) ≥ E b,s log q(b|s), where q(b|s) is a neural network discriminator trained to predict the diversity indicator b from the visited state.In practice, rather than learning a single low-level skill to handle all graphs, SkiLD utilizes a factorized lower-level policy. When the target dependency graph is specified, SkiLD identifies which state factor should be influenced and uses its corresponding policy to sample primitive actions. More details about this subdivision can be found in Appendix A.In SkiLD, after the skill learning stage, we utilize hierarchical RL to solve reward-supervised downstream tasks with the discovered skills. The skill policy, π skill acts as the low-level policy while a task policy, π task : S → Z, is learned to select which skill z = (g, b) to execute for L steps. Compared to diversity-based skills that are limited to simple behaviors, our local-dependency-based skills enable a wide range of interactions between state factors, leading to more efficient exploration and superior performance of downstream task learning.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiments: 
 In this section we aim to provide empirical evidence towards the following questions: Q1) Do the skills learned by SkiLD induce a diverse set of interactions among state factors? Q2) Do the skills learned by SkiLD enable more efficient downstream task learning compared to other unsupervised reinforcement learning methods? Our learned skills are visualized at https://sites.google. com/view/skild/.In this work, we focus on addressing the challenge of vast state space brought by the number of state factors. Hence, we evaluate our method on two challenging object-rich embodied AI benchmarks: Mini-behavior [32] and Interactive Gibson [42] . The Mini-behavior (Mini-BH) domain [32] (Figure 3a ) contains a set of gridworld environments where an agent can move around and interact with a variety of objects to accomplish certain household tasks. While conceptually simple, due to highly sequentially interdependent state factors (see details in the Appendix), this domain has been shown to be extremely challenging for the agent's exploration ability, especially under sparse reward [32] . Each Mini-BH environment contains different objects and different success criteria. We tested on three particular environments in Mini-behavior, including:• Installing Printer: A relatively simple environment with three state factors: the agent, a table, and a printer that can be installed. • Cleaning Car: An environment where the objects have rich and complex interactions. The state factors include the agent, a toggleable sink, a piece of rag that can be soaked in the sink, a car that the rag can clean, a soap and a bucket which can together be used to clean the rag. • Thawing: An environment with lots of movable objects. The state factors include the agent, a sink, a fridge that can be opened, and three objects that can be thawed in the sink: fish, olive, and a date.The Interactive Gibson (iGibson) domain [43] (Figure 3b ) contains a realistic simulated Fetch Robot that operates in a kitchen environment with a refrigerator, sink, knife, and peach. The peach can be washed or cut. This domain is very difficult especially when using low-level motor commands because much of the domain is free space, meaning that only a minute fraction of action sequences will manipulate the objects meaningfully.Both Mini-BH and iGibson require learning long-horizon policies spanning many low-level actions from sparse reward, making these challenging domains (see details in Appendix).Before evaluating the empirical questions, we provide a brief description of the baselines. These baselines include unsupervised skill learning, and causal and hierarchical methods.Diversity is all you need (DIAYN [19] ): This method learns unsupervised state-covering skills using a mutual information objective. SkiLD utilizes a version of this for state-diversity skills modulated by a desired dependency graph. This baseline determines how incorporating graph information affects the algorithm.Controllability-Aware Skill Discovery (CSD [50] ): Extends DIAYN with a factorization based on controllability. This baseline is a comparable skill learning method that leverages state factorization but does not encode local dependencies.Exploration via Local Dependencies (ELDEN [63] ): This method utilizes gradient-based techniques to infer local dependencies for exploration. However, without a skill learning component, it can struggle to chain together complex behavior.Chain of Interaction Skills (COInS [14] ): This is a hierarchical algorithm that constructs a chain of skills using Granger-causality to identify local dependencies. Because it is restricted to pairwise interactions, it struggles to represent the rich policies necessary for these tasks.Vanilla RL: This baseline uses PPO [57] to directly train an agent with the extrinsic reward. Unlike other baselines, this method does not have a pertaining phase. Since all the task rewards are sparse and the tasks are often long horizon, vanilla RL often struggles. We first evaluate whether SkiLD is indeed capable of achieving complex interaction graphs (Q1), comparing against two strong skill discovery baselines introduced earlier: DIAYN and CSD.Each of these methods is trained for 10 Million steps without having access to any reward. Then to evaluate their learned skills, we unroll each of them for 500 episodes with randomly sampled skills z and examine the diversity of the interaction graphs they can induce. Figure 4 illustrates the percentages of episodes where some hard local dependencies have been induced at least once, in Mini-BH Cleaning Car (for simplicity of presentation, see Appendix for results on all inducible local dependency graphs and their meanings). We find that DIAYN and CSD are limited to skills that only manipulate one object individually, for example, picking up the rag (agent, rag, action → rag) or the soap (agent, soap, action → soap). By contrast, SkiLD learns to induce more complicated causal interactions, such as soaking the rag in the sink (sink, rag → rag) and cleaning the car with the soaked mug (car, rag → car).Next, we evaluate whether the local dependency coverage provided by SkiLD leads to a performance boost in downstream task learning under the same number of environment interactions (Q2). We follow the evaluation setup in the unsupervised reinforcement learning benchmark [38] , where for a given environment, an agent is first pre-trained without access to task reward for K pt steps, and then finetuned for K ft steps. Importantly, the same pre-trained skills are reused on multiple distinct downstream tasks within the same environment, so that only the upper-level skill-selection policy is task-specific. We have K pt = 2M , K ft = 1M for installing printer, K pt = 10M , K ft = 5M for thawing and cleaning car, and K pt = 4M , K ft = 2M for iGibson, and evaluate each method for each task across 5 random seeds. Hyperparameter details can be found in Appendix D. Specifically, we evaluate on the following downstream tasks:• Installing Printer: We have a single downstream task in this environment, where the agent needs to pick up the printer, put it on the table, and turn it on. • Thawing: We have three downstream tasks: thawing the fish or the olive or the date.• Cleaning Car: We consider three downstream tasks, where each task is a pre-requisite of the following one. The tasks are: soak the rag in the sink; clean the car with the rag; and clean the dirty rag using the soap in the bucket. • IGibson: The tasks for this domain are: grasping the peach, washing the peach in the sink, and cutting the peach with a knife.After skill learning, we train a new upper-level policy that uses z as actions and is trained with extrinsic reward, as described in Section 3.4. Figure 5 illustrates the improvement of SkiLD as compared to other methods. Without combining dependency graphs with skill learning, other methods struggle with any but the simpler tasks. COInS performs poorly because of its chain structure, which restricts the agent controlling policy from picking up objects. ELDEN's exploration reaches graphs, but without skills struggles to utilize that information in downstream tasks. DIAYN learns skills, but few manipulate the objects, so a downstream model struggles to utilize those skills to achieve meaningful rewards. By comparison, SkiLD achieves superior performance on 9 of the 10 downstream tasks evaluated. In the two hardest tasks which require a very long sequence of precise controls, Clean Rag and Cut Peach, SkiLD is the only method that can achieve a non-zero success rate (although still far from fully mastering the tasks), showcasing the potential of local dependencies for skill learning.We also explore the functionality of the graph and diversity components of the skill parameter z by assessing the downstream performance of SkiLD without these components. This produces two ablative versions of SkiLD: SkiLD without diversity and SkiLD without dependency graphs. To isolate learning from the effect of learned local dependencies, we use ground truth dependency graphs for ablative evaluations where relevant. In Figure 6 , learning without graphs results in zero performance, consistent with DIAYN results. In addition, removing diversity produces a notable decline in performance, especially on more challenging tasks like clearning the rag. These evaluations demonstrate that SkiLD benefits from both the incorporation of dependency graphs and diversity. 
related work: 
 This work lies in the unsupervised skill learning framework [37] , where the agent must discover a set of useful skills which are reward independent. It then extends these skills to construct a 2-layer hierarchical structure [60] , where the upper policy receives reward both for achieving novel skills, and can then be tuned to utilize the learned skills to accomplish an end task. Finally, the skills are identified using token causality, a specific problem identified in causal literature.This work describes a framework for utilizing local dependency graphs and diversity to discover unsupervised skills. Diversity-based state coverage skills have been explored in literature [19] utilizing forward and backward mutual information techniques to learn a goal space Z, and a skill encoder q(z|•) [11] . This unsupervised paradigm has been extended with Lipschitz constraints [49] , contrastive objectives [39] , information bottleneck [35] , population based methods such as particle estimation [45] , quality diversity [44] and mixture of experts [12] . These skills can then be used for hierarchical policies or planners [56, 67, 23] , which mirrors the same structure as SkiLD. Unlike these methods, SkiLD adds additional subdivision through dependency graphs, which mitigates the combinatorial explosion of skills that can result from trying to cover a large factored space.The hierarchical policy structure in SkiLD where a higher level policy passes a parameter to be interpreted by low-level planners has been formalized in [60] , and learned using deep networks utilizing Each curve depicts the mean and standard deviation of the success rate over 5 random seeds. Without graphs, the method collapses completely, while removing diversity results in a noticeable reduction in downstream performance.extrinsic reward [3, 62] , attention mechanisms [16] , initiation critera [34, 4] and deliberation cost [26] . Hierarchies of goal-based policies [40] has been extended with object-centric representations [66] , offline data [48] , empowerment [41] and goal counts [51] . In practice, SkiLD uses graph and diversity parameters similar to goal-based methods. However, the space of goals can often be intractable large, and methods to address this use graph laplacians [36] causal chains [13, 14] or general causal relationships [29] . SkiLD is similar to these causal methods but utilizes local dependence along with general two-layer architectures, thus showing increased generalizability.This work investigates the application of local dependency to hierarchical reinforcement learning. This kind of reasoning has been described as "local causality" or "interactions" in prior RL work for data augmentation [53, 54] , learning skill chains [13, 14] and exploration [63] . This work is the first synthesis of unsupervised skill learning and local dependencies applied to general 2-layer hierarchical reinforcement learning. Other general causality work investigates action-influence detection [58, 28] , affordance learning [10] , model learning [30, 21] , critical state identification [46] , and disentanglement [17] . In the context of relating local dependency and causal inference, we provide a discussion in Appendix C. SkiLD incorporates causality-inspired local dependence to skill learning, resulting in a set of diverse skills.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: In https://github.com/wangzizhao/SkiLD, we provide all code for the environments, as well as install dependencies for training SkiLD in the Readme of the interac-tionHRL folder. With the environments, any data for this environment is generated. We also provide a description of the compute requirements for running these experiments in Appendix D.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 Unsupervised skill discovery is a powerful tool for learning useful skills in long-horizon sparse reward tasks. However, many unsupervised skill-learning methods do not take advantage of factored environments, resulting in poor performance in complex environments with several objects. Skill Discovery from Local Dependencies utilizes state-specific dependency graphs, identified using learned pointwise conditional mutual information models, to guide skill discovery. The framework of defining skills according to a dependency graph and diversity goal, combined with a learned sampling scheme, achieves difficult downstream tasks. In domains where hand-coded primitive skills are typically given to the agent, like Mini-behavior and Interactive Gibson, SkiLD can achieve high performance without requiring explicit domain knowledge. These impressive results arise intuitively from incorporating local dependencies as skill targets, illuminating a meaningful direction for unsupervised skill learning to be applied to a wider array of environments.Limitations and Future Work An important assumption of SkiLD is its access to factored state space. While factored state space can often be naturally obtained from existing RL benchmarks and many real-world environments, developments in disentangled representation learning [47, 31] will help with extending SkiLD to unfactored image domains. Secondly, SkiLD requires accurate detection of local dependencies. While off-the-shelf methods [63, 58] work well for detecting local dependencies in our experiments, future works that can more accurately detect local dependencies will be beneficial to the performance of SkiLD.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The core experimental setting is presented in Sec. 4. The detailed settings are included in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
acknowledgement: 
 This work has taken place in the Learning Agents Research Group (LARG) at the Artificial Intelligence Laboratory, The University of Texas at Austin. LARG research is supported in part by the National Science Foundation (FAIN-2019844, NRT-2125858), the Office of Naval Research (N00014-18-2243), Army Research Office (W911NF-23-2-0004, W911NF-17-2-0181), Lockheed Martin, and Good Systems, a research grand challenge at the University of Texas at Austin. The views and conclusions contained in this document are those of the authors alone. Peter Stone serves as the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We provided error bars in all quantitative results, which are standard deviations computed from 5 random seeds.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We specify the computation resources used in our experiments in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Learning to reach both a desired graph g and a diversity parameter b through primitive actions is challenging. First, different graphs often have substantially different characteristics, with some graphs that are easy to achieve (eg. action→agent), and others that are quite challenging and rare (eg. agent, knife, fruit→fruit). Not only would it be challenging for a single policy to encode all of these behaviors, the diversity parameter notwithstanding, but over-training the frequency at which certain graphs are called might vary significantly. Rather than trying to learn a single monolithic policy, then, we instead structure the skill parameterized policy π skill as a collection of factored skills: π skill,i , for each factor i ∈ {1, . . . , N }.This modification to the policy structure results in three changes: 1) The upper-level action space passes a single row of the graph G, denoted with g i , and the desired factor i. 2) Instead of achieving an entire graph use the achieved row 1[g achieved,i = g i ].3) The history of seen graphs H is replaced with a history of factored graph rows H f . Define the history of graph rows as H f := {unique (i, g achieved,i ∀i ∈ 1, . . . , N ∀g achieved ∈ D)}. This takes the unique graph rows from all those seen in previous data. Then the upper policy uses the same historical sampling procedure as with unfactorized graphs: the policy samples discretely from the new history, which will by default return i, g i , a graph row, and the desired factor. This resolves points 1,3. Point 2 is addressed by replacing Equation 5with 1[g achieved,i = g i ].Empirically, we found that without this change, the lower policy rarely learns anything, even simple control of the agent.In this section, we provide a detailed description of the environment, including its semantic stages representing internal progress toward task completion, state space, and action space. We also highlight that while each task consists of multiple semantic stages, agents do not have access to this information. Installing Printer As shown in Fig. 7 (a), the Installing Printer environment is relatively simple, consisting of 3 factors: the agent, a printer, and a table. The task requires the agent to complete the following stages: (1) pick up the printer, (2) bring the printer to and place it on the table, and (3) turn on the printer. The discrete state space consists of (i) the agent's position and direction, (ii) the positions of the printer and whether it is on or off, and (iii) the position of the table. The discrete action space consists of (i) moving forward, turning left or right, (ii) picking up / placing down the printer, and (iii) turning on / off the printer.Thawing As shown in Fig. 7 (b) and Fig. 10 (a), the Thawing environment consists of 6 factors: the agent, a sink, a refrigerator, and three frozen objects: fish, olive, and date. Thawing each object requires the agent to complete the following stages: (1) move to and open the refrigerator, (2) take the frozen fish out of the refrigerator, (3) put the fish into the sink, and (4) turn on the sink to thaw it. The discrete state space consists of (i) the agent's position and direction, (ii) the positions of all environment entities, (iii) whether the sink door is turned on, (iv) whether the refrigerator door is opened, and (v) the thawing status of three objects. The discrete action space consists of (i) moving forward, turning left or right, (ii) opening / closing the refrigerator, (iii) turning on / off the sink, and (iv) picking up / placing down each object.Cleaning Car As shown in Fig. 7 (c), the Cleaning Car environment consists of 7 factors: the agent, a car, a sink, a bucket, a shelf, a rag, and a piece of soap. Cleaning both the car and the rag requires the agent to complete the following stages: (1) take the rag off the shelf, (2) put it in the sink, (3) toggle the sink to soak the rag up, (4) clean the car with the soaked rag, (5) take the soap off the self, and ( 6) clean the rag with the soap inside the bucket. The discrete state space consists of (i) the agent's position and direction, (ii) the positions of all environment entities, (iii) whether the sink is turned on, (iv) the soak status of the rag, (v) the cleanness of the rag, and (vi) the cleanness of the car. The discrete action space consists of (i) moving forward, turning left or right, (ii) turning on / off the sink, and (iii) picking up / placing down the rag / soap.iGibson As shown in Fig. 7 (d), the iGibson environment consists of 4 factors: the robot, a knife, a peach, and a sink. The robot can do the following things: (1) grasp peach: move close to the peach and grasp it, (3) wash peach: grasp the peach and place it into the sink, (3) grasp knife: move close to the knife and grasp it, (4) cut peach: grasp the knife and use it to cut the peach. The continuous state space consists of (i) the robot's proprioception, (ii) the poses of all environment entities, and (iii) whether the peach is cut. The continuous action space consists of (i) end-effector position change, (ii) base linear and angular velocity, and (iii) gripper torque (to open/close the gripper).Though looking conceptually simple, we emphasize that these environments are challenging because of the following factors:• The state factors are highly sequentially interdependent, making skill learning and task learning challenging: for example, in cleaning car environments, the agent can't clean the car until it picks up the rag, turns on the sink, and soaks the rag. These interdependencies between state factors pose great challenges to the agent's exploration ability. • During the skill-learning stage, we would like the agents to learn all possible skills (e.g., manipulating all objects) rather than learning to manipulate a single object. • During task learning, we use sparse rewards, and thus it is further challenging for agents to explore. • In addition, we use primitive actions, and many actions have no effect if it's not applicable in the current state. So the task is especially challenging for exploration.In this work, we define local dependencies according to the state factors X = (X 1 , . . . , X N ) and event of interest Y , which in the context of an MDP is a subset of the next state factors X ′ = (X ′ 1 , . . . , X ′ N ). In the factored MDP formulation [8] , we assume that p, the transition dynamics, are represented by a dynamic Bayesian network (DBN) which is a time-directed bipartite graph, with edges only from factors in X to factors in X ′ . In this work, we assume that the underlying ground truth DBN, that is the transition function p, can be decomposed according to subsets of state factors X, such there exists a p X (Y = y| X = x) for every state.The factored transition dynamics analogizes with causal inference in the following way: If the state factors and next state factors are each assigned a causal variable by adding the assumption that they can be independently intervened on, and each next state variable carries an associated unobserved noise variable U i , which we assume is independent of any X k not connected to X ′ j and any other next state variable X ′ j , then we can represent the transition dynamics p with a structural causal model (SCM) [52], a graph connecting the causal variables in X to the causal variables in X ′ .For a particular outcome variable Y that is one of the next state causal variables X ′ , we can describe local dependence in the RL context according to assumptions about the structural causal model. Represent the non-noise parents of Y as pa(Y ), and the noise parents as pa U (Y ). Under normal causal assumptions, the structural causal model for Y is a function f Y (pa(Y ), pa U (Y )) = Y . Define X as a subset of the endogenous parents of Y and Ū as an equivalent subset of the noise variables. Further define the values that pa(Y ), pa U (Y ), X, Ū can take on as pa(y), pa U (y), x, ū respectively, and (pa(Y)), X , Ū as the set of states the parents of Y , the variables in X and variables in Ū can take on respectively.To formalize local invariance, we add the assumption that f Y can be decomposed into a series of functions (f Y 1 ( X1 = x1 , Ū1 = ū1 ), . . . , f Y k ( Xk = x, Ūk = ūk )) and g Y (pa(Y ) = pa(y), pa U (Y ) = pa U (y)), where each f Y i : X × Ū → Y and g : pa(Y) → {1, . . . , k}, a function mapping the parents of Y to one of the functions. Then if f is represented as:EQUATIONThe local dependence of Y = y in a particular state (x, x ′ ) is then the set of variables in Xi for the particular i where 1(g Y (pa(y), pa U (y)) = i) = 1, and the pCMI test is a way of uncovering these local dependencies from observational data.Local dependence has been investigated in the field of context-specific independence [55, 9] , which seeks to find particular assignments of a subset of the causal variables under which an outcome is independent of some subset of the inputs. In particular, context-set specific independence [9] determines if a variable is independent of other variables on a particular subset of states, described as the partial context set. While our work uses the pCMI test described in Equation 2, context-specific independence focuses on complete independence using knowledge of the structural model.Alternatively, interactions can be viewed as the causes ( X) of particular effects (Y ), which have also been investigated under the description of token or actual cause [25] (as opposed to general cause). Actual cause utilizes a series of counterfactual tests to determine if a cause is necessary, sufficient, and minimal for an outcome. Actual cause has primarily been applied in simple, discrete examples [5, 24] , making it difficult to directly apply to RL. However, recent work has incorporated the notion of context-specific independence and extended actual cause to more complex domains [15] .The hyperparameters of skill learning and task learning can be found in Table 1 . As it is challenging to identify local dependencies using learned dynamics models in Thawing and iGibson environments, we use ground truth local dependencies from simulator. The codebase is built on tianshou [64] for backend RL, though with significant modifications. The 5 seeds selected are 0 -4. The experiments were conducted on machines of the following configurations:• Nvidia A40 GPU; Intel(R) Xeon(R) Gold 6342 CPU @2.80GHz • Nvidia A100 GPU; Intel(R) Xeon(R) Gold 6342 CPU @2.80GHzFigure 8 illustrates the percentages of episodes where all local dependencies have been induced at least once, in Mini-BH Cleaning Car. Again, SkiLD (ours) induces all inducible dependency graphs, while baselines fail to induce hard graphs with challenging pre-conditions.The meaning of the graphs are:• agent, action → agent: agent moving. • agent, rag, action → rag: agent picking up the rag or moving it.• agent, soap, action → soap: agent picking up the soap or moving it.• agent, X → agent: X blocking the agent's motion.• agent, sink, action → sink: agent turning on or off the sink. • agent, sink, rag, action → rag: agent soaking the rag in the sink (which requires that the sink is turned on). • sink, rag → rag: the same as above.• car, rag → car: the rag cleaning the car and getting dirty (which requires that the rag is soaked).• car, rag → rag: the same as above. • collecting a unit of wood to craft a wood stick, • collecting another unit of wood and combining it with the stick to craft a wood axe that is required for collecting the stone and for removing the rock, • collecting a unit of wood and a unit of stone to craft a stick and then a stone axe that is required for collecting the gold, remove the rock surrounding the gold and collect the gold with the stone axe.As shown in Fig. 9 , SkiLD still outperforms all baselines in this complex task, demonstrating the usefulness of its learned skills for downstream task solving.In Figure 10 we visualize three challenging long-horizon skills learned by SkiLD: thawing the olive, cleaning the car, and cutting the peach. All of these skills require a sequence of interactions that is difficult to recover without directed behavior. Thus, comparable baselines do not learn skills of similar complexity. More skill visualizations can be found at: https://wangzizhao.github.io/SkiLD/. (a) Existing methods that reach diverse, distinguishable states can struggle in complex environments with state factors because they focus on simple state factors: This result is empirically verified in the performance results in experimental domains in Figure 5 , where DIAYN and CSD struggle to capture any of the skill requiring the manipulation of multiple state factors, such as cleaning the car or cutting the peach. In Figure 4 we also demonstrate how CSD and DIAYN fail to capture graphs that cover multi-factor interactions. (b) Guiding skill learning with interactions and exploring state coverage within that space can uncover desirable interactions: This result is empirically verified in the performance results of SkiLD illustrated in Figure 5 , where SkiLD is able to handle complex, multi-object interactions such as cleaning the rag after cleaning the car or cutting the peach. Figure 4 also illustrates how desirable interactions like putting soap in the bucket or cleaning with the rag are uncovered. (c) The skills learned directed by local dependencies can be reused to efficiently maximize task rewards: Again, Figure 5 illustrates that compared to vanilla PPO learning, SkiLD, which uses local dependencies, is able to reuse skills with greater sample efficiency, and often even higher downstream performance. In these results, the pretrained skills are reused across tasks in the same domain, such as thawing the fish, thawing the olive and thawing the date. We provide details about teh training procedure in Section 4.4. (d) Combining local dependencies with state diversity is important to induce diverse interactions: This claim is supported in Section 4.5 and Figure 6 , where performance noticeably declines after removing the local dependencies and state diversity individually, especially for more long horizon multi-interaction tasks such as cleaning the car or cleaning the rag. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper introduces SkiLD, a method for unsupervised skill discovery that uses state factorization to learn skills inducing diverse interactions between state factors. Although the reviewers found the paper interesting with good results, some of them expressed concerns about clarity in the writing and in several technical aspects related to the method (refer to comments by reviewers c2kz and WEDj). Although some of these were addressed in the authors’ responses, the reviewers expressed that the authors failed to address exactly how they would revise the paper to address all the mentioned issues. Apart from this, the reviewers pointed out inadequately justified claims pertaining to a "robust set of transferable skills" without sufficient experimental justification, and mention "large number of state factors" although only 3 to 6 state factors were used. Although the authors have promised to use more precise language, it is unclear how this would affect the overall readability of the paper. Although incomplete, the inclusion of results on the more complex 2d minecraft experiments looks promising. The reviewers also felt their concern regarding the usefulness of skills was not sufficiently addressed. Overall, the paper is relevant and interesting in the context of unsupervised skill discovery. However, it could benefit from clearer presentation, more accurate representations of the scope of the work, with the technical aspects described fully and precisely with appropriate context.

Reviews: review: paper_summary: paper_summary: The paper introduces SkiLD, a novel method leveraging state factorization to guide skill learning in unsupervised reinforcement learning. SkiLD emphasizes learning skills that induce diverse interactions between state factors, which are crucial for solving downstream tasks. The authors demonstrate that SkiLD outperforms existing unsupervised RL methods through empirical validation.
strength_weakness: strength_weakness: 1. **Assumption of Factored State Space**: SkiLD assumes access to a factored state space, which may not always be available or easy to obtain in real-world applications.
2. **Evaluation on Limited Domains**: The evaluation domains are somewhat restricted. Broader evaluation across more commonly used environments could emphasize the method's performance. Evaluating environments like Crafter, 2D Minecraft, or manipulation environments with multiple objects would better illustrate the benefits of focusing on local dependencies.
3. **Scalability**: There is a need to explore how SkiLD scales with an increasing number of state factors, which is crucial for practical applications.
questions: questions: Please refer to Weakness
limitations: limitations: Please refer to Weakness
paper_summary: paper_summary: The paper introduces SkiLD (Skill Discovery from Local Dependencies), an unsupervised skill discovery method. Unlike existing methods that focus on state diversity, SkiLD leverages state factorization to guide skill learning by inducing diverse interactions between state factors.

This method is designed to be more effective in complex environments, such as household settings with numerous objects. SkiLD uses local dependencies to model interactions and introduces a novel intrinsic reward mechanism.

The method is evaluated in several domains, including a realistic household robot simulation, demonstrating superior performance compared to existing methods.
strength_weakness: strength_weakness: Minor weaknesses:
- The method's reliance on accurately detecting and modeling local dependencies adds a layer of complexity that may limit its applicability.
- The effectiveness of SkiLD hinges on the availability of a factored state space, which may not always be available or easily obtainable.
questions: questions: The two points raised within the weaknesses limit the applicability and thus the impact of the paper. Do you have insights on this? Especially, how the method can be made more general?

Nitpicking:
> L88: Formally, for an event of interest $Y$ and its potential causes $X = (X^1,  ..., X^N )$, given the value of $X = x$, local dependencies focus on which $X$ is are the state-specific cause of the outcome event $Y = y$.

I think the vocabulary used in this sentence is wrong.
- $Y$ is not an event but a random variable.
- $Y = y$ is an event.

> L108: In this work, for a transition $(\mathcal{S} = s, \mathcal{A} = a, \mathcal{S}^\prime = s^\prime)$...

If I understand correctly the notations, this shouldn't be $\mathcal{S}$ and $\mathcal{A}$ but $S$ and $A$, because $\mathcal{S}$ and $\mathcal{A}$ are the state space and action space and are not random variables. Plus, you said in the Background section that "In this paper, we use uppercase letters to denote random variables and lowercase for their specific values...".

Same remark for L149-150.

> In this section, we describe SkiLD, which enhances the expressivity of skills using local dependencies. SkiLD represents local dependencies as state-specific dependency graphs, defined in Sec. 2.2.

State-specific dependency graphs are actually not defined in Section 2.2, there is just a sentence at the end of the section. It would be useful to better explain what are these graphs.
limitations: limitations: Yes, the authors adequately addressed the limitations.
paper_summary: paper_summary: In the presented paper the authors propose a novel skill discovery method called Skill Discovery from Local Dependencies (SkiLD). The method utilizes the concept of local dependencies to incorporate the interaction of factors in a factorized state space. A novel intrinsic reward signal is introduced to guide the skill discovery. In the skill learning phases, different skill-conditioned policies are learned for the skills and in the task learning phase these (frozen) skills are used to learn a task policy solving more complex downstream tasks. The method is evaluated in two simulation environments and compared to several baseline methods.

---
**Post-Rebuttal/Discussion**: I appreciate the discussion with the authors and their effort in clarifying open questions, my main criticism wrt presentation and evaluation are not solved though, and I remain my score of 5.
strength_weakness: strength_weakness: My main concern is related to the use of overexaggerations and mismatch in some claims and with respect to the experimental results. For example, ‘…resulting in a robust set of transferable skills’, there is no evaluation or similar regarding robustness (in fact that word only occurs in that claim). Or more prominent, the domains are introduced with ‘…a LARGE number of state factors.’, while the state factors range from only 3 to 6 (Section 4.1).

While mentioning the research questions is good, the chosen evaluations do not fit these questions very well. Q1 is about the diversity of the interactions, but Figure 4 then only shows a comparison of some ‘particular’ (how are they chosen?) interactions. For a diversity comparison a general metric comparing all found/used interactions should be compared. The number of skills is also not shown/discussed or evaluated.

Similar, Q2 is about more efficient downstream learning. Efficiency in terms of what exactly? The according evaluation in Figure 5 (and Section 4.4) measures the performance. Is performance used as a proxy for efficiency here?

As there are many details, modules and methods, it is not completely clear what is assumed to be given/known, or how certain quantities are inferred/used. For example, the events of interest (every factor in just the next_state of the s,a,s’ tuple?) or the target dependency graph (a crucial parameter, and lines 157ff mentions HER for that, but not for all cases?). What exactly does the diversity parameter do? How is the exact flow of the skill learning phase (getting the target graph, the intrinsic reward, learning the policies, and multiple policies can be learned for each skill?)? Some additional effort in clarifying such details would greatly benefit the paper.  

Some critical details, like that for some setups the ground truth dependencies were used instead of the learned, are only provided and thus somehow hidden in the Appendix.

Additional comments are given in the Questions.

While computational demanding and, hence, understandable, 5 seeds/runs are quite low for comparing RL based algorithms (known problem in the community, although often neglected/ignored). Recent frameworks and metrics have been proposed tackled at these problems related to low number of runs and comparing overall algorithms performance [1]. It would be beneficial to add such metrics (e.g., performance profiles and IQM/IQR) to the paper for a better comparison of the proposed algorithm.

----------- 

[1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021)
questions: questions: How can the approach deal with varying number of objects? The policies depend on the states and the state-specific graph?

Does the chosen representation allow for ‘infinite’ number of skills? How does the approach scale with increasing skill number? How does it effect the learned policies (skill and task policy)?

How are the two learning phases (skill and task) orchestrated? Is each phase done once? Or do they take turns, if so, how?

In the factored MDP, is the action space A defined over the full S, or does it take the factorization into account?

L168: ‘…we utilize a factorized lower-level policy, where there is a separate policy for each factor.’ So each skill is realized with multiple policies? How is this modelled and especially used? How are skills then chosen, does this affect the action set of the task policy?

Notation in Figure 4 (and same in text) is unclear, what does the ‘x,y -> x’ notation represent?

Figure 1 seems to be never referenced?

How are the skill policies modelled? Table 1 only give learning details, is it a neural network? What are its details? Same for other methods, the presented parameters are not complete.

What does ‘state-specific’ dependency graph mean? 

Figure 6 misses the plot description (is it mean and std as well?).
limitations: limitations: Limitations are discussed, focusing on the assumption of known state factorization and accurate detection of local dependencies.

An additional (potential) limitation is related to the scaling of the approach wrt. the training (data, time, ..), the number of objects and skills (rather low currently), or the transfer of skills to novel objects.
paper_summary: paper_summary: The authors present SkiLD, an unsupervised RL method that learns skills by augmenting DIAYN reward with a graph-state dependency reward to induce meaningful changes in object interactions.
strength_weakness: strength_weakness: **Clarity: Quite a few clarity issues**

- In the main paper, the method could be presented more clearly at a higher level. For example, introducing the Graph-Selection policy first (in 3.1 instead of 3.2) could help with understanding, at a higher level, what the method is trying to do. Currently the paper first talks about the skill selection policy but, by the time the reader gets to 3.1, it’s not fully clear yet (it’s mentioned at the top of section 3 but then gets into details a bit too suddenly in 3.1) that there is an initial unsupervised skill learning phase and what that phase is trying to do. Going from high level idea → high level policy → low level policy would make this more clear.
- More explicit details are needed. How is the dynamics model trained? What’s the high-level overview of the algorithm? When and how are diversity rewards applied? What exactly are the input/outputs features of the graph selection policy? All of this could be more clear if the authors provide some pseudocode linked to from the main paper and beef up the appendix.
- High-level details about environment assumptions (learned/given dynamics, what is given in the graph factorization, etc.) in section 4.1 should be given, even if details are already in the appendix.

**Experiments:**

- Are 10M timesteps really needed even in Mini-behavior domain? This seems extremely sample-inefficient. Same with requiring up to 5M timesteps to learn downstream tasks reasonably in these environments. Perhaps some details about why all methods need so many timesteps would be helpful.
- How are hparams selected and tuned? And for the baselines? How do we know this is a fair comparison?

**Minor issues:**

- L163: missing end of sentence period
- Fig 4: CaSk?
questions: questions: One limitation of all unsupervised skill learning methods is that a significant portion of the learned skills are meaningless. Qualitatively, what proportion of the skills ended up being meaningful? Was it more with SKilD than baselines like DIAYN? This perhaps could be visualized by randomly sampling ~50 skills and visualizing them to see which ones are actually meaningful.
limitations: limitations: Adequately addressed.

Comments: title: scalability experiments on 2d minecraft
comment: I deeply appreciate the effort put into conducting the additional experiments. Regarding the 2D Minecraft task, my concerns about limited domains and scalability issues have been resolved. I would appreciate it if the authors could incorporate these experimental results into the final version.

Given these improvements, I will increase my score to 5.
We agree with the reviewer that further evaluating our method in larger-scale settings (with more state factors) is important. To do so, we evaluate our method in **2D Minecraft with 15 state factors** following Andreas et al [1], and we hope this result addresses the reviewer's concern about the evaluation and scalability of our method. 

Specifically, we use the **Mine Gold** task described below, measure the task success rate after **3M** time steps, and show the **IQM score** (the higher the better) as suggested by Reviewer3 c2kz across **5 random seeds**. Again, our method SkiLD outperforms the following baselines during task learning.

| **Task**         | **SkiLD**         | **Elden**         | **DIAYN**         | **Vanilla**       |
|-------------------|-------------------|-------------------|-------------------|-------------------|
| Mine Gold     | **0.613$\pm$ 0.065**  | 0.000 $\pm$ 0.000  | 0.000 $\pm$ 0.000  | 0.000 $\pm$ 0.000  

[1] Andreas, Jacob, Dan Klein, and Sergey Levine. "Modular multitask reinforcement learning with policy sketches." International conference on machine learning. PMLR, 2017.

We will attach the figure of task training curves in the next version of the paper (unfortunately, we can't upload figures during the discussion period). Also, due to the limit of time and computation, we will include the results of COInS and CSD in the next version of the paper.

In case you are interested, the **environment details** are listed below. We make sure they will be added to the appendix.

* state space (15 state factors): the agent (location and direction), 10 environment entities (the positions of 3 wood, 1 grass, 1 stone, 1 gold, and 4 rocks surrounding the gold), and 4 inventory cells (i.e., the number of stick, rope, wood axe, and stone axe that the agent has).
* action space (9 discrete actions):
   * 4 navigation: going up, down, left, right
   * pick up the environment entity in front, no effect if the agent does not have the necessary tool for collecting it
   * 4 crafting: craft a stick/rope/wood axe/stone axe, no effect if the agent does not have enough ingredients
* Mine Gold task: the agent will receive a **sparse reward** after finishing **all** the following steps
   * collecting a unit of wood to craft a wood stick
   * collecting another unit of wood and combining it with the stick to craft a wood axe that
is required for collecting the stone and for removing the rock,
   * collecting a unit of wood and a unit of stone to craft a stick and then a stone axe that is
required for collecting the gold,
   * remove the rock surrounding the gold and collect the gold with the stone axe.
Again, we thank the reviewer for the constructive feedback, including the new suggestions of a more detailed description of the mini-behavior environment and the visualization of state distribution! We will ensure that they are incorporated in the next version of the paper.
Thanks for the detailed response.

Due to the neurips rebuttal format, I can't verify the suggested changes will be made, so I can only partially consider those in whether to increase the score. However, the plan makes sense. I can reconsider if the authors explicitly list out quotes for what will be changed.

> Though mini-behavior looks simple, the environments are challenging because of the following factors:

This is a reasonable description, thank you. This information however is not presented in the actual paper, with the only hint at this being: "While conceptually simple, this domain has been shown to be extremely challenging for Vanilla RL."

> Fig 4 in the paper (full version: Fig 1 of the global response) shows the semantic meaningfulness of skills acquired by different methods (in terms of which interactions they can induce).

This doesn't fully answer my question (though it is a helpful visualization) of "what proportion of skills end up being meaningful" as it's easier to induce these interactions in a discrete environment over a long horizon of 500 steps. Randomly sampling skills and visualizing their behavior, truly at random, in the Gibson environment (cont. control) would be a better comparison, or perhaps plotting a graph of the state distribution of the skills (again sampled at random) against x-y positions in the minigrid environments to get more information about what the policy is doing with each skill.

I have currently raised my score in response to the other parts of the rebuttal.
Thank you for your rebuttal.

While I appreciate the clarifications and explanations provided, I believe the current version of your paper still requires additional experimental validation to convincingly demonstrate the effectiveness of the proposed framework. I will maintain my score.
We appreciate your quick response! Please find below our responses to address your concerns regarding our work:

>Regarding incorporating necessary information

We thank the reviewer for the heads-up. The main space-taking thing we are planning to incorporate into the main text is the pseudocode. Since there will be one more page for the camera-ready version were this paper to be accepted. We believe we will have enough space to incorporate all feedback into the next version of our paper.

>Regarding diversity metric

Fig 1 in rebuttal shows all the induced interactions AND the percentage of times they appear at least once within an episode. Obviously, some interactions will be induced all the time while some hard interactions can rarely be induced, which is what the numbers reflect. The error bar shows the standard deviation of this percentage across random seeds (the number shows the mean).

Now, the new metric that we are reporting in the rebuttal (following the suggestions of the reviewer) is WHETHER an interaction will appear at all through randomly sampled skills, which can be obtained by counting the non-zero entries in the chart – 15 for SkiLD, 12 for CSD, and 9 for DIAYN, which is why our method has a skill coverage of 100%.
Why an interaction should count even if it is induced rarely – since we **uniformly sample skills**, it is very unlikely to fulfill the preconditions of a hard interaction and induce it. However, during task learning, one can **optimally select the sequence of skills** to induce it using planning/a learned task policy. Therefore, even if an interaction is induced infrequently in Fig 1, it shows the skills are useful for solving tasks relevant to this interaction.
Thanks for detailed response that clarified most points and the provided pseudo-algorithm helps to understand the flow better. However, it will be a challenge to incorporate all this necessary information in an updated paper. 

One follow up question:

*"The diversity metric would be (# induced interactions) / (# induciable interactions). As shown in global response Fig 1, SkiLD induces all inducible interactions, while CSD only induces 80% and DIAYN only induces 60% of all inducible interactions."*

I do not understand how the mentioned 100% and 80% and the global response Fig 1 are connected? Fig 1 shows in how many episodes each graph was induced, except the first one which all methods finds, no other has 100%. Moroever, one graph was only induced in 0.5% according to the Fig 1. How does this fit together with the mentioned numbers? (and with 5 random seeds, how can the Figure numbers occur, shouldn't it be in steps of 20% then? the connection between seeds and episodes is unclear here)
Thank you for your detailed rebuttal. I will maintain my score.