SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions
Authors: Zizhao Wang
Jiaheng Hu
Caleb Chuck
Stephen Chen
Roberto Martín-Martín
Amy Zhang
Scott Niekum
Peter Stone
Venue: NeurIPS 2024 poster (2024)
Keywords: unsupervised skill discovery
Primary area: reinforcement_learning

Abstract: Unsupervised skill discovery carries the promise that an intelligent agent can learn reusable skills through autonomous, reward-free interactions with environments. Existing unsupervised skill discovery methods learn skills by encouraging distinguishable behaviors that cover diverse states. However, in complex environments with many state factors (e.g., household environments with many objects), learning skills that cover all possible states is impossible, and naively encouraging state diversity often leads to simple skills that are not ideal for solving downstream tasks. This work introduces Skill Discovery from Local Dependencies (SkiLD), which leverages state factorization as a natural inductive bias to guide the skill learning process. The key intuition guiding SkiLD is that skills that induce \textbf{diverse interactions} between state factors are often more valuable for solving downstream tasks. To this end, SkiLD develops a novel skill learning objective that explicitly encourages the mastering of skills that effectively induce different interactions within an environment. We evaluate SkiLD in several domains with challenging, long-horizon sparse reward tasks including a realistic simulated household robot domain, where SkiLD successfully learns skills with clear semantic meaning and shows superior performance compared to existing unsupervised reinforcement learning methods that only maximize state coverage.

SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions
Zizhao Wang!* Jiaheng Hul-
Caleb Chuck Stephen Chen
Roberto Martín-Martín' Amy Zhang' Scott Nickum" Peter Stone 2.3
University of Texas at Austin University of Massachusettess, Amherst SSony Al [zizhao. wang, steephen,steeennn.... chen, robertomm @utexas edu amy. www.agaaanann.nnnmmma utexas. edu, sniekum@umass edu, {calebc, pstone (cs. utexas edu
Abstract
Unsupervised skill discovery carries the promise that intelligent agent can learn reusable skills through autonomous, reward-free environment interaction. Existing unsupervised skill discovery methods learn skills by encouraging distinguishable behaviors that cover diverse states. However, in complex environments with many state factors (e.g.. household environments with many objects), learning skills that cover all possible states is impossible, and naively encouraging state diversity often leads to simple skills that are not ideal for solving downstream tasks. This work introduces Skill Discovery from Local Dependencies (SkiLD), which leverages state factorization as natural inductive bias guide the skill learning process. The key intuition guiding SkiLD that skills that induce diverse interactions between state factors are often
valuable for solving downstream tasks. To this end, SkiLD develops a novel skill learning objective that explicitly encourages the mastering of skills that effectively induce different interactions within an environment. We evaluate SkiL.D in several domains with challenging, Long-horizon sparse reward tasks including realistic simulated household robot domain, where SkiLD successfully learns skills with clear semantic meaning and shows superior performance compared to existing unsupervised reinforcement learning methods that only maximize state coverage. Code and visualizations are at h/app:::///aggzzhaa github. io/SkiLD/
Introduction
Reinforcement learning (RL) achieves impressive successes when solving decision-maiee makinn problems with well-defined reward functions 65 trivial
However, designing this reward function is often not contrast, humans and other intelligent creatures can learn, without external reward supervision, behaviors that produce repeatable and predictable changes in the environment [18). These behaviors, which we call skills, can be later repurposed to solve downstream tasks efficiently. One of the promises of this form of unsupervised RL is to endow artificial agents with similar capabilities to discover reusable skills without explicit rewards.
20 33
One predominant strategy of prior skill discovery methods focuses on training skills to reach diverse states while being distinguishable 19 59 50]. However, complex environments that contain many state distinct elements such as individual objects in household (a formal description in Sec. 2. the exponential number of distinct states makes impossible learn skills that cover every state. Consequently, these methods typically result in simple skills that only change the easy-to-contee factors (e.g., manipulation task moving the agent itself diverse positions or
Indicates equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

b+1
1+1
Figure 1: Skill Discovery from Local Dependencies (SkiLD) describes skills that encode interactions (i.e., local dependencies) between state factors. In contrast to prior diversity- -rased methods that can easily get stuck by moving the robot to diverse, but non-interactive states, and factor-based methods that are trained to manipulate the hammer and nail, but not their interactions, SkiLD not only manipulate each object (left, middle) but also induce interactions between them (right), by specifying different local dependencies. These skills are often more useful than the "easy" skill learned by previous methods for downstream task-solving.
manipulating each factor independently), and fail to cover other desirable but challenging behaviors. Meanwhile, in factored state space, many downstream tasks require inducing interactions between state factors, e.g., cooking requires using knife to cut the ingredients and cooking them in a pan, etc. Unsurprisingly, these simple skills often struggle to solve such tasks, resulting in poor downstream performance.
Our key insight is to utilize interactions between state factors as a powerful inductive bias for learning useful skills. In factored state spaces and their downstream tasks, there usually exist bottleneck states that an agent must pass through explore different regions of the environment, and many of them can be characterized by interactions between state factors. For example, in a household environment, a robot must first grasp the knife before moving it to different locations, with the bottleneck being the interaction between the robot and the knife. In environments that have a large state space due to many state factors, rather than inefficiently relying on randomly visiting different states to reach such bottlenecks, we propose to train the agent to actively induce these critical interactions.
To this end, we introduce Skill Discovery from Local Dependencies (SkiLD), novel skill discovery method that explicitly learns skills that induce diverse interactions. Specifically, SkiLD models the interactions between state factors using the framework of local dependencies (where local refers to state-specific, see details in Sec. 2.2 and proposes novel intrinsic reward that 1) encourages the agent to induce specified interactions, and 2) encourages the agent to discover diverse ways of inducing specified interaction, as visualized in Figure I During skill learning, SkiLD gradually discovers new interactions and learns to induce them, based on the skills that it already mastered, resulting in diverse set of interactioniinduiigg behaviors that can be readily repurposed for downstream tasks. During task learning, the skill policy is reused, and a task-specific policy is learned to select (a sequence of) skills to maximize task rewards efficiently.
We evaluate the performance of SkiLD on factor-rich environments with 10 downstream tasks against existing unsupervised reinforcement learning methods. Our experiments indicate that SkiLD learns to induce diverse interactions and outperforms other methods on most of the examined tasks.
2 Background
In this paper, our unsupervised skill discovery method is set up in a factored Markov decision process and builds off previous diversity-based methods, as described in Scc. To enhance the expressivity of skills, our method further augments the skill representation with interactions between state factors, which we formalize as local dependencies as described in Sec. 2.2
2.1 Factored Markoy Decision Process (Factored MDP)
We consider unsupervised skill discovery in a reward-free Factored Markov Decision Process g defined by the tuple M = (S, A, p). S s x ... S is a factored state space with N subspaces, where each subspace S is a lineti-dimensional continuous or discrete random variable. Then, correspondingly, each state 8 consists of N state factors, i.e., a = (s',..., sN\ ), sf E S". this paper, use uppercase letters to denote random variables and lowercase for their specific values (c.g..

---

S denotes the random variable for states s). A is the action space, and p is an unknown Markovian transition model that captures the probability distribution over the next state S' ~ p(-1S,
The factorization in S inherently exists in many environments, and is a common assumption in prior unsupervised skill discovery works 22 29 27 For example, in robotics, an environment typically consists of a robot and several objects to manipulate, and, for each object, St would represent its attributes of interest, like pose. In this work, we explore how we can utilize a given state factorization to improve unsupervised skill discovery. In practice, the factorization can either be directly provided by the environment or obtained from image observations with existing disentangled representation learning methods 47 31.
Following prior work, our method consists of two stages skill learning and task learning. During the skill learning phase, we seek to learn a skill policy Tm. [8, 2), which defines a conditional distribution over actions given the current state and some skill representation 2, where skills indicate the desired behaviors of the agent. Once the skills are learned, they can be chained together to solve downstream tasks during the task learning phase through an extrinsic reward-optimizing policy. During task learning, a downstream task reward function r S x A + R is provided by the environment. A high-level policy (=]ss) is then trained optimize the expected return through outputting correct skills z given state 8.
2.2 Identifying Local Dependencies between State Factors
A key insight of SkiLD is to utilize interactions between state factors (or, formally, local dependencies) as part of the skill representation. In later sections, these local dependencies are compiled into a binary matrix [(s, a, s') = (0, 1) NNN+()+1) representing the local dependencies between all factors. In this section, we first formally define local dependencies, introduce their identification, and finally discuss their application to factored MDPs.
SkiLD takes cuusality-inspire approach for defining and detecting local dependencies 66 58). where we use local to refer to a particular assignment of values for a random variable, as opposed to global which applies to all values. Formally, for an event of interest Y y and its potential causes XNN,, given the value of X = x. local dependencies focus on which X's are the state-specific cause of the outcome event Y = (for simplicity of presentation, in this section we overload N as the number of potential causes rather than number of variables and p as the transition function according to subset of the variables). Formally, we denote the general data generation process of as p X + and the data generation process when Y is only influenced by subset of X as xx : + Y. where X C X. Then, given the value of all variables, X x², ....... ,.','''''''''' = z and Y y, we say Y locally depends on X, if X is the minimal subset of X such that knowing their values is necessary and sufficient to generate the result of Y = y. i.e.,
X = (X
arg migmin/ [X| XCX
s.t.
px = ②) = p(Y = x
(I)
where |X| is the number of variables in X. For example, suppose that robot opens a refrigerator door in particular transition. The event of interest Y = y is the refrigerator door becoming open, and it locally depends on two factors: the robot and the refrigerator door, while other state factors such objects inside the refrigerator do not locally influence Y.
To identify local dependencies, one can conduct conditional independence test y x (z/=")) to examine whether variable X is necessary for predicting Y = y. In prior works, one form of this test is to examine whether the pointwise conditional mutual information (pCMI) is greater than 0,
p(y2z)
PCMI Hx/x*)) = log
p (xx/X') (vl (x/-a}) 0.
If so, then it suggests that knowing X provides additional information about Y that is not present in {X/X), and Y locally depends on X As the data generation processes are generally unknown, one has approximate them with learned models. Recent work in RL has utilized various approximations such as attention weights [53], Granger causality [4] and input gradients [63].
In this work, for transition (S 8, A a, S' = s'), the event of interest is each next state factor being (Si)² = Ys'),, and we infer whether locally depends on each state factor S and the action A (i.c., whether there is an interaction between state factors i and j, where factor influences i). Then
3

---

Skill Learning Phase
Task Learning Phase
graph-salection policy
task policy
(o(+la)
R2
dynamics model
Kraak (- (+]4)
Eq.
desired local dependencies
diversty indicator
b
(s, a, +
Env
z
R Ex. (3)
Fakil 4,
Env
[ +)
skill policy
frozen skill policy
Figure 2: During skill learning of SkiLD, the graph-selection policy specifies desired local depen- dencies for the skill policy to induce, and the induced dependency graph identified by the dynamics model and used to update both policies. During task learning (right), the skill policy is kept frozen and task policy is trained to select skills to maximize task reward.
we aggregate all local dependencies into a state-specine dependency graph (abbreviated in this work to dependency graph). This overall dependency graph is represented with [(s, a, s') = {0, 1 (x(++1) and an edge G (s, a, s') denotes, during the transition (s, a, s'), that state factor (s*y (the "Y- = y²" locally depends on s (one of the X
g PCMI((=')) (x//=>1)))) >
(3)
This graph is used to enhance skill representation, explained in detail in Section B
3 Skill Discovery from Local Dependencies (SkiLD)
In this section, we describe SkiLD, which enhances skills using local dependencies. SkiLD represents local dependencies as state-specific dependency graphs, defined in Sec. 2.2 and learns to induce different dependency graphs in the environment for different skills. To intelligently generate target dependency graphs during training, SkiLD frames unsupervised skill discovery as a hierarchical RL problem described in Fig. 2ig.2pm and Alg. [ where a high-level graph selection policy chooses target local dependencies to guide exploration and skill learning, and a graph conditoond skill policy learns to induce the specified local dependencies using primitive actions.
This framerwork requires formalizing three components: (1) the skill representation 2, presented in Sec. 3.1 (2) the graph selection policy (z|s)) and its reward function Rg, presented in Sec. 3.2 and (3) the skill policy J.ain (alss 2) and its corresponding reward function Raaill, presented in Sec. 3.3
3.1 Skill Representation 2
Prior unsupervised skill discovery methods usually focus skill learning on changing the state each factor diversely, which is inefficient when there exist bottleneck states for explorations. Consequently, they are can be limited learning simple skills, for example, only changing the easiest-to- cosiest-to-control factor in the state (i.e., the agent itself). To address this problem, SkiLD not only focuses on changing the state but also considers the interactions between state factors.
Skill Representation. SkiLD represents the skill space as the combination of two components: Z G x B, where g G is state-specific dependency graph that specifies the desired local dependencies between state factors (e.g.. hammering the nail), and E B is a diversity indicator the same that used in Eysenbach al. [19]. While the agent inducing particular local dependencies we use to further encourage to visit distinguishable states (e.g., under different b values, training the agent to hammer the nail into different locations). Specifically, the dependency graph is represented as a binary matrix G {0, (() As described in Sec. 2.2 each edge G denotes, during the transition (s, a, s'), whether the state factor (s*y locally depends on s². The diversity indicator B can be discrete continuous. In this work, without loss of generality, we follow the procedure of Eysenbach et al. 19 and use a discrete b sampled uniformly from {1...., K}, where K is a predefined number.
Given this skill space, SkiL.D learns the skills as skill-conditioned policy Tamn SSx 2 - A, where Takil is trained to reach diverse states while ensuring that local dependencies specified by the graph

---

Algorithm SkiLD Skill Discovery
1: Initialize the high-level graph-selection policy иş S + G, the low-level skill policy waail
S 2 - A, the diversity indicator discriminator q : 8 - p(B), and the dynamics model
f:Sx x x + S, graph selection interval L.
2: for each skill training timestep do
3:
// data collection
4:
i % == 0 then
5:
Sample the target dependency graph g (T)ss)
6:
Sample the diversity indicator from uniform distribution Uniform(B),
7:
Compose the skill variable = (g. b).
8:
end if
9:
Collect state transitions (s, 2, a, 8') with actions from waml (a|s,
Infer the induced dependency graph (s, a, 8') using the dynamics model f' (Sec. 3.1).
11:
Update the history of the seen graphs with (nnduced s')..
12:
// training
13:
Sample batch of (s, s') from the replay buffer.
14:
Update the dynamics model f(s,, a) by minimize the prediction error W.r.t. s'.
15:
Update the high-level policy with reward Rg (Eq. 5) with the history of seen graphs.
16:
Update the discriminator (b)ss, g) with the discrimination (cross-entropy) loss.
17:
Infer the induced dependency graph Ginduced
s') using the dynamics model f (Sec. 3.1).
18:
Infer the diversity reward Rdiversity = log q(b(8,g)
19:
Update Takill with Raai
[Yindwedd (8, 8)) (1 + didversity) (Eq. 4).
20:
end for
are induced. Before we describe Takin training in Sec. Takill to follow during the skill learning stage.
3.3
we first discuss how to select the skill for
3.2 Hive-Level Graph-Selection Policy mg
To acquire skills that are useful for downstream tasks, the skill policy aaii needs to learn to induce wide range of local dependencies policy Tg S - G to guide the training of Takil. Specifically, training Takill requires a wise selection of graphs - as graph space G increases super-exponentially in the number state factors N, many graphs not inducible. To this end, we only select target graphs for the skill policy from history of all seen graphs. As the agent learns to induce existing graphs in diverse ways, new graphs may be encountered, gradually expanding the of seen graphs.
cmmple-efficiently. To this end, we propose to learn graph-selection
However, though this history guarantees graph inducibility, two challenges still remain: (I) How to efficiently explore novel local dependencies, especially hard-to-visit ones? (2) For all seen graphs, which one should Taall learn next to maximize training efficiency? We address these challenges based the following insight compared to well-learned skills, Tsiil should focus its training underdeveloped skills. Meanwhile, learning new skills opens up the possibility of visiting novel local dependencies, e.g., learning to grasp the hammer makes it possible for the robot to hammer the nail.
According to this idea, we learn a graph-selection policy Th that guides the exploration and training of the skill policy Tal-- Specifically, Tg S - G selects new dependency graph the skill policy should induce for the next time steps. increase the likelihood of visiting hard graphs, mg trained to maximize the following graph novelty reward
Rg
(4)
V C(vvisiedd)
where C(Ivistee) the number of times that we have seen the graph in the collected transition.
3.3 Low-Level Skill Policy Takill
Given the skill parameter z from the graph-selection policy, SkiLD learns skills as policy Tainl S - A, where Taill learns to reach diverse states while ensuring that the local dependencies specified by g induced. During skill learning, select actions by iteratively calling
5

---

the skill policy Taill, and we denote ginduced a, s') ) as the graph that describes the local dependencies induced in transition (s, a, s') when executing a selected action a. We design the reward function of the skill policy as:
Rkll [finduced (8, 8') .) (1 + X
where 1 [Sindweed (8, 01 8 g) measures whether the induced dependency graph matches the desired graph, Rdiversity is the weighted diversity reward that further encourages visiting diverse states when the desired graph is induced, and À is the coefficient of diversity reward. In the following paragraphs, we describe how we infer finduced (s, a, 8') and estimate Rdiversity for each transition.
Inferring Induced Graphs. To infer the induced graph for transition (S = =s a, = we need to determine, for each (S")², whether it locally depends on each factor S and the action
A. Following Sec. 2.2 we evaluate the conditional dependency (s* - s s/ss//i s a} by examining whether their pointwise conditional mutual information (pCMI) is greater than a predefined threshold p(s.a)
6. If PCMI p((a')(<//r 2 e, suggests that is necessary to predict (s*) and thus the local dependency exists. Meanwhile, as the transition probability is unknown, we approximate it with learned dynamics model that trained minimize prediction error.
Finally, after obtaining the induced dependency graph, we evaluate 1 [9indweed a, s') = gl by examining whether each edge Pinduced matches the corresponding edge in the desired graph g², As Rua only provides sparse rewards to the skill policy when the desired graph is induced, we use hindsight experience replay [2] to enrich learning signals, by relabelling induced graphs as desired graphs in some episodes.
Diversity Rewards. When the skill policy induces the desired graph, Rdiversity further encourages it to visit different distinguishable states under different diversity indicators b, e.g-, hammering the nail to different locations. This diversity enhances the applicability of learned skills. To this end, we design the diversity reward Reiversity as the forward mutual information between visited states and the diversity indicator I (s; following DIAYN. To estimate the mutual information, we approximate it with variational lower bound (s; b) 2 Ba, log q(b(s), where q(b|s) neural network discriminator trained to predict the diversity indicator b from the visited state.
In practice, rather than learning single low-level skill to handle all graphs SkiLD utilizes factorized lever-level policy. When the target dependency graph is specified, SkiLD identifies which state factor should be influenced and uses its corresponding policy to sample primitive actions. More details about this subdivision can be found in Appendix
3.4 Downstream Task Learning
In SkiLD, after the skill learning stage, we utilize hierarchical RL to solve reward-supervised downstream tasks with the discovered skills. The skill policy, Taail acts as the low-level policy while a task policy, Fiss : S - 2, is learned to select which skill = (g, b) to execute for steps. Compared to svverstt-bbssed skills that are limited to simple behaviors, our local-dependeney-baee skills enable a wide range of interactions between state factors, leading to more efficient exploration and superior performance of downstream task learning.
4 Experiments
In this section aim to provide empirical evidence towards the following questions: Q1) Do the skills learned by SkiLD induce diverse set of interactions among state factors? Q2) Do the skills learned by SkiLD enable more efficient downstream task learning compared other unsupervised reinforcement learning methods? Our learned skills are visualized at hhttps:/ //sstss////e.... //tgeesoosssseeeeeeee com/ com/view/ske skild/
4.1 Domains
In this work, we focus on addressing the challenge of vast state space brought by the number of state factors. Hence, we evaluate our method on two challenging object-rich embodied Al benchmarks: Mini-behavior 32] and Interactive Gibson 42

---

(a) Thawing
(b) Cleaning Car
(c) Interactive Gibson
Figure 3: Evaluation domains: iGibson.
Mini-behavior: Installing Printer, Thawing and Cleaning Car, and
The Mini-behavior (Mini-BH) domain [32 (Figure
3a) contains set of gridworld environments where an agent can move around and interact with variety of objects to accomplish certain household tasks. While conceptually simple, due to highly sequentially interdependent state factors (see details in the Appendix), this domain has been shown to be extremely challenging for the agent's exploration ability, especially under sparse reward [32]. Each Mini-BH environment contains different objects and different success criteria. We tested on three particular environments in Mini-behavior, including:
Installing Printer: relatively simple environment with three state factors: the agent, table, and a printer that can be installed.
Cleaning Car: An environment where the objects have rich and complex interactions. The state factors include the agent, a toggleable sink, a piece of rag that can be soaked in the sink, car that the rag can clean, a soap and bucket which can together be used clean the rag.
Thawing: An environment with lots of movable objects. The state factors include the agent, sink, a fridge that can be opened, and three objects that can be thawed the sink: fish, olive, date.
The Interactive Gibson (iGibson) domain
43 (Figure 3 3D) contains a realistic simulated Ferch Robot that operates in kitchen environment with a refrigerator, sink, knife, and peach. The peach can be washed or cut. This domain is very difficult especially when using low-level motor commands because much of the domain is free space, meaning that only a minute fraction of action sequences will manipulate the objects meaningfully.
Both Mini-BH and iGibson require learning aag-horizon policies spanning many low-level actions from sparse reward, making these challenging domains (see details Appendix).
4.2 Baselines
Before evaluating the empirical questions, we provide brief description of the baselines. These baselines include unsupervised skill learning, and causal and hicrarchical methods.
Diversity is all you need (DIAYN 19): a mutual information objective. SkiLD utilizes a version this for state-diversity skills modulated by desired dependency graph. This baseline determines how incorporating graph information affects the algorithm.
This method learns unsupervised state-covering skills using
Extends DIAYN with factorization based on controllability. This baseline is a comparable skill learning method that leverages state factorization but does not encode local dependencies.
Cantrolability-AAweee Skill Discovery (CSD 550):
Exploration via Local Dependencies (ELDEN 633): This method utilizes gradient-based techniques to infer local dependencies for exploration. However, without skill learning component, it can struggle to chain together complex behavior.
Chain of Interaction Skills (COInS 14)): This is a hierarchical algorithm that constructs a chain identify local dependencies. Because it is restricted to pairwise
of skills using Granger-causalin interactions. it struggles to represent the rich policies necessary for these tasks.
Vanilla RL: This baseline uses PPO 57 to directly train an agent with the extrinsic reward. Unlike other baselines, this method does not have a pertaining phase. Since all the task rewards are sparse and the tasks are often long horizon, vanilla RL often struggles.
7

---

4.3 Interaction Graph Diversity
We first evaluate whether SkiLD is indeed ca- pable of achieving complex interaction graphs (Q1), comparing against two strong skill discov- ery baselines introduced earlier: DIAYN and CSD.
106.0
agant action
100.0
92.5
act
agent, soap, action soap
Each of these methods is trained for 10 Million steps without having access to any reward. Then evaluate their learned skills, we unroll each of them for 500 episodes with randomly sampled skills 2 and examine the diversity of the interac- tion graphs they can induce. Figure S fillustrates the percentages of episodes where some hard local dependencies have been induced at least once, in Mini-BH Cleaning Car (for simplic- ity of presentation, see Appendix for results all inducible local dependency graphs and their meanings). We find that DIAYN and CSD are limited to skills that only manipulate one ob- ject individually, for example, picking up the rag (agent, rag, action - rag) or the soap (agent, soap, action -> soap). By contrast, SkiLD learns to induce more complicated causal interactions, such soaking the rag in the sink (sink, rag rag) and cleaning the car with the soaked mug (car, rag + car).
agent, sink, action > Sink
33.9
agant, sink, reg, action rag
0.0
-2555
rag reg
119.0
-16.6
bucket, soap > bucket
car, rag - rag
SKILD lours)
bucket, rag -> rag
visitation lkkthood
Figure 4: The percentage episodes where a de- pendency graph is induced through random skill sampling. Standard deviation is calculated across five random seeds.
4.4 Sample Efficiency and Performance
Next, we evaluate whether the local dependency coverage provided by SkiLD leads to a performance boost in downstream task learning under the same number of environment interactions (Q2). We follow the evaluation setup in the unsupervised reinforcement learning benchmark [38], where for a given environment, an agent is first pre-trained without access to task reward for Kμt steps, and then finetuned for KA steps. Importantly, the same pre-trained skills are reused on multiple distinct downstream tasks within the same environment, so that only the upper-level skill-selection policy is task-specific We have Kpt 2M, K₁ 1M for installing printer, Kpt 10M, K₂ 5.M for thawing and cleaning car, and Kpt = 4M, Kn = 2M for iGibson, and evaluate each method for each task across 5 random seeds. Hy Hyperparameter details can be found in Appendix D Specifically, evaluate on the following downstream tasks:
We have a single downstream task in this environment, where the agent needs to pick up the printer, put on the table, and turn it on.
Installing Printer:
Thawing: We have three downstream tasks: thawing the fish or the olive or the date.
Cleaning Car: We consider three downstream tasks, where each task a pre-requisite of the following one. The tasks are: soak the rag in the sink; clean the with the rag; and clean the dirty rag using the soap in the bucket.
IGibson: The tasks for this domain are: grasping the peach, washing the peach in the sink, and cutting the peach with a knife.
After skill learning, we train a new upper-level policy that uses as actions and is trained with extrinsic reward, as described in Section 3.4 FigureBjillussee other methods. Without combining dependency graphs with skill learning, other methods struggle with any but the simpler tasks. COInS performs poorly because of its chain structure, which restricts the agent controlling policy from picking up objects. ELDEN's exploration reaches graphs, but without skills struggles to utilize that information in downstream tasks. DIAYN learns skills, but few manipulate the objects, so downstream model struggles to utilize those skills to achieve meaningful rewards. By comparison, SkiLD achieves superior performance on of the 10 downstream tasks evaluated. In the two hardest tasks which require very long sequence of precise controls, Clean Rag
the improvement of SkiLD as compared to
8

---

(a) Install Printer
(b) Thaw Olive
(c) Thaw Fish
(d) Thaw Date
(c) Soak Rag
10%g
(f) Clean Car
(g) Clean Rag
(h) Grasp Peach
Wash Peach
Cut Peach
SkiLD (ours)
DIAYN
CSD
ELDEN
COInS
Vanilla
Figure 5: Training curves SkiLD and baselines on multiple downstream tasks (reward supervised second phase). Each curve depicts the mean and standard deviation of the success rate over 5 random seeds. SkiLD outperforms all baselines for most tasks, converging faster and to higher returns.
and Cut Peach, SkiLD is the only method that can achieve non-zero success rate (although still far from fully mastering the tasks), showcasing the potential of local dependencies for skill learning.
4.5 Graph and Diversity A blations
We also explore the functionality of the graph and diversity components of the skill parameter : by assessing the downstream performance of SkiLD without these components. This produces two ablative versions of SkiLD: SkiLD without diversity and SkiLD without dependency graphs. To isolate learning from the effect of learned local dependencies, graphs for ablative evaluations where relevant. In Figure 6 learning without graphs results in zero performance, consistent with DIAYN results. In addition, removing diversity produces a notable decline in performance, especially on more challenging tasks like clearning the rag. These evaluations demonstrate that SkiLD benefits from both the incorporation of dependency graphs and diversity.
use ground truth dependency
Related Work
This work lies in the unsupervised skill learning framework (37] where the agent must discover set of useful skills which are reward independent. It then extends these skills to construct a 2-layer hierarchical structure [60), where the upper policy receives reward both for achieving novel skills, and can then be tuned to utilize the learned skills to accomplish an end task. Finally, the skills are identified using token causality, specific problem identified in causal literature.
5.1 Unsupervised Skill Learning
This work describes a framework for utilizing local dependency graphs and diversity to discover unsupervised skills. Diversity-based state coverage skills have been explored literature [19] utilizing forward and backward mutual information techniques to learn goal space Z, and a skill encoder q(2l-) [TT]. This unsupervised paradigm has been extended with Lipschitz constraints 449. contrastive objectives [39), information bottleneck (35), population based methods such particle estimation 45). quality diversity [44 and mixture of experts [12]. These skills can then used for
hierarchical policies planners [56 67 23), which mirrors the same structure as SkiL.D, Unlike
these methods, SkiLD adds additional subdivision through dependency graphs, which mitigates the combinatorial explosion of skills that can result from trying to cover large factored space.
5.2 Hierarchical Reinforcement Learning
The hierarchical policy structure in SkiLD where higher level policy passes a parameter to be inter- preted by low-level planners has been formalized in (60), and learned using deep networks utilizing

---

1.0
1.0
0.10-
0.5
0.5
0.05
0.0
2.5M
5.0M
0.0
2.5M
5.
0.00
2.5M
5.0M
(a) Soak Rag
(b) Clean Car
(c) Clean Rag
SkiLD (base)
SkiLD w/o Graphs
SkiLD w/o Diversity
Figure A figure illustrating the ablative performance of SkiLD without diversity or without graphs. Each curve depicts the mean and standard deviation of the success rate over random seeds. Without graphs, the method collapses completely, while removing diversity results in a noticeable reduction in downstream performance.
extrinsic reward 3 (22). attention mechanisms [16]. initiation critera 33 and deliberation cost [26
Hierarchies goal-baseds policies [40] has been extended with object-centric representations [66]. offline data [48], empowerment [1] and goal counts [5I]. In practice, SkiLD uses graph and diversity parameters similar to goal-based methods. However, the space of goals can often be intractable
large, and methods to address this use graph laplacians (36] causal chains [13 [4] general causal
relationships [29]. SkiLD is similar to these causal methods but utilizes local dependence along with general two-layer architectures, thus showing increased generalizability.
5.3 Causality in Reinforcement Learning
This work investigates the application local dependency to hierarchical reinforcement learning. This kind of reasoning has been described as "local causality" or "interactions" in prior RL work
for data augmentation [53 54], learning skill chains [1 [4] and exploration [63] This work is the first synthesis of unsupervised skill learning and local dependencies applied general 2-layer hierarchical reinforcement learning. Other general causality work investigates action-influence detection (88 28) affordance learning [10), model learning (30 21), critical state identification [46), and disentanglement [17. In the context of relating local dependency and causal inference, we provide discussion in Appendix C SkiLD incorporates causality-inspired local dependence to skill learning, resulting in a set of diverse skills.
6 Conclusion
Unsupervised skill discovery is a powerful tool for learning useful skills in long-horizon sparse reward tasks. However, many unsupervised skill-learning methods do not take advantage of factored environments, resulting poor performance in complex environments with several objects. Skill Discovery from Local Dependencies utilizes state-specife dependency graphs, identified using learned pointwise conditional mutual information models, to guide skill discovery. The framework of defining skills according sampling scheme, achieves difficult downstream tasks. domains where hand-coded primitive skills are typically given the agent, like Mini-behavior and Interactive Gibson, SkiLD achieve high performance without requiring explicit domain knowledge. These impressive results arise intuitively from incorporating local dependencies as skill targets, illuminating meaningful direction for unsupervised skill learning to be applied to a wider array of environments.
dependency graph and diversity goal, combined with a learned
Limitations and Future Work An important assumption of SkiLD is its access factored state space. While factored state space can often be naturally obtained from existing RL benchmarks and many real-wor world environments, developments in disentangled representation learning [47 31] will help with extending SkiLD to unfactored image domains. Secondly, SkiLD requires accurate detection of local dependencies. While off-the-shed methods dependencies in our experiments, future works that can more accurately detect local dependencies will be beneficial the performance of SkiLD.
663 58 work well for detecting local
10

---

7 Acknowledgement
This work has taken place in the Learning Agents Research Group (LARG) at the Artificial In- telligence Laboratory. The University of Texas at Austin. LARG research is supported in part by the National Science Foundation (FAIN-2019844, NRT-2125858), the Office of Naval Research 1000014-18-22433 Army Research Office (W91 W911NFF23-2-0004 191
Lockheed Martin, and Good Systems, research grand challenge at the University of Texas at Austin. The views and conclusions contained in this document are those of the authors alone. Peter Stone serves as the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas Austin in accordance with its policy on objectivity in research.
References
[1] Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International conference machine learning, pages 166-175, PMLR. 2017.
[2] Marcin Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[3] Pierre-Lur Bacon, Jean Harb, and Doina Procup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence. volume 31, 2017.
[4] Akhil Bagaria, Ben Abbatematteo, Omer Gottesman, Matt Corsaro, Sreehari Rammohan, and George Konidaris. Effectively learning initiation sets hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[5] Sander Beckers, Hana Chockler, and Joseph Halpern. A causal analysis of harm. Advances in Neural Information Processing Systems, 35:2365-2377 2022.
[6] Gianluca Bontempi and Maxime Flauder. From dependency to causality: a machine learning approach. J. Mach. Learn. Res., 16(1):2437-2457, 2015.
[7] Serena Booth, W Bradley Knox, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifications. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5920-5929, 2023.
[8] Craig Boutilier, Thomas Dean, and Steve Hanks. Decision-theoretie planning: Structural assumptions and computational leverage. Journal of Artificial Intelligence Research, 11:1-94, 1999.
[9] Craig Boutilier, Nir Friedman, Moises Goldszmidt, and Daphne Koller. Context-specific independence in bayesian networks. arXiv preprint arXiv:1302.3566 2013.
[10] Jake Brawer, Meiying Qin, and Brian Scassellati. A causal approach to tool affordance learning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8394-8399. IEEE, 2020.
[11] Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
[12] Onur Celik, Dongzhuoran Zhou, Ge Li, Philipp Becker, and Gerhard Neumann. Specializing versatile skill libraries using local mixture of experts. In Conference on Robot Learning, pages 1423-1433, PMLR, 2022.
[13] Caleb Chuck, Supawit Chockchowwat, and Scott Niekum. Hypothesis-driven skill discovery for hierarchical deep reinforcement learning. In 2020 IEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5572-5579. IEEE, 2020.

---

[14] Caleb Chuck, Kevin Black, Aditya Arjun. Yuke Zhu, and Scott Niekum. Granger- hierarchical skill discovery. arXiv preprint arXiv:2306.09500 2023.
[15] Caleb Chuck, Sankaran Vaidyanathan, Stephen Giguere, Amy Zhang, David Jensen, and Scott Niekum. Automated discovery of functional actual causes in complex environments. arXiv preprint arXiv:2404. 10883, 2024.
[16] Raviteja Chunduru and Doina Precup. Attention option-critic. arXiv preprint arXiv:220.002225 2022.
[17] Oriol Corcoll and Raul Vicente. Disentangling controlled effects for hierarchical reinforcement learning. In Bernhard Schölkopf, Caroline Uhler, and Kun Zhang, editors, Proceedings of the First Conference Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research. pages 178- 200.. PMLR, 1 11-13 Apr 2022. URL BttL.tttp://proceediig
mlr. -ppess//177/ccorro1222 html
[18] Yuqing Du, Eliza Kosoy, Alyssa Dayan, Maria Rufova, Pieter Abbeel, and Alison Gopnik. What can ai learn from human exploration? autrinsically-motiee humans and agents in open-world exploration. In NeurlPS 2023 workshop: CntomatiionTThoreeie Principles in Cognitive Systems, 2023.
[19] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function, arXiv preprint arXiv: arXiv: /802.66000 2018.
[20] Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino
Paredes, Mohammadanin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszez, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-53, 2022.
[21] Fan Feng and Sara Magliacane. Learning dynamic sutribute-fattored world models for efficient multi-object reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[22] Pierre Fournier, Cédric Colas, Mohamed Chetouani, and Olivier Sigaud. Clic: Curriculum learning and imitation for object control in nonrewarding environments. IEEE Transactions on Cognitive and Developmental Systems, 13(2):239-248, 2019.
[23] Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning. pages 1851-1860. PMLR, 2018
[24] Joseph Y Halpern, Actual causality. MIT Press, 2016.
[25] Joseph Y Halpern and Judea Pearl. Causes and explanations: structural-model approach. part i: Causes. The British journal for the philosophy of science, 2005.
[26] Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option:
Learning options with a deliberation cost. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[27] Jiaheng Hu, Zizhao Wang, Peter Stone, and Roberto Martín-Martin Disentangled unsupervised skill discovery for efficient hierarchical reinforcement learning. In Workshop on Reinforcement Learning Beyond Rewards@ Reinforcement Learning Conference 2024.
[28] Jiaheng Hu, Peter Stone, and Roberto Martin-Martin. Causal policy gradient for whole-body mobile manipulation. arXiv preprint arXiv:2305.0444 044666,..0...... 2023.
[29] Xing Hu, Rui Zhang, Ke Tang, Jiaming Guo, Qi Yi, Ruizhi Chen, Zidong Du, Ling Li, Qi Guo, Yunji Chen, et al. causality-driven hierarchical structure discovery for reinforcement learning. Advances in Neural Information Processing Systems, 35:20064-2000 2022.
[30] Yixuan Huang, Adam Conkey, and Tucker Hermans. Planning for multi-object manipulation with graph neural network relational classifiers. In 2023 IEFE International Conference on Robotics and Automation (ICRA), pages 1822-1829. IEEE, 2023.
12

---

[31] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. arXiv preprint arXiv:2303. 10834, 2023.
[32] Emily Jin, Jiaheng Hu, Zhuoyi Huang, Ruohan Zhang, Jiajun Wu, Li Fei-Fei, and Roberto Martin-Martin, Mini-behavior: A procedurally generated benchmark for long-horizon decision- making in embodied ai. arXiv preprint arXiv:2310.01824, 2023.
[33] Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müllrr, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982 987, 2023.
[34] Khimya Khetarpal, Martin Klissarov, Maxime Chevalier-Boisvert,, Pierre-Luc Bacon, and Doina Precup. Options of interest: Temporal abstraction with interest functions. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4444-44511 2020.
[35] Jaekyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5572-5582. PMLR, 18-24 Jul 2021.
mlr. www.mmlrrreeer pressprsss/vv
kim21j html
[36] Martin Klissarov and Marlos C. Machado. Deep Laplacian-based options for temporally- extended exploration. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En- gelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research. pages 17198-17217. PMLR, 23-29 Jul 2023. URL /tttps://procedeee v202/klissarov23a. html
mlr press//
[37] Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep re- inforcement learning: A survey. Information Fusion, 85:1-22, 2022. ISSN 1566-2535. doi: https://dbi.org/10.011/// inw.:/...nnffus.222.0...... ccieece//rricce///////6666220000
URL htthh:tt//wwwt http://wwwwccieceerreett.tooc com/
[38] Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark, 2021.
[39] Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:2202.00161, 2022.
[40] Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical reinforcement learning with hindsight.
In International Conference on Learning Representations, 2019. URL httpps://operreviie
net/forum?1d-ryaEoon
[41] Andrew Levy, Sreehari Rammohan, Alessandro Allievi, Scott Niekum, and George Konidaris. Hierarchical empowerment: Towards tractable empowerment- skill-leariing arXiv preprint arXiv:2307.02722 2023.
[42] Chengshu Li, Fei Xia, Roberto Martin-Marn Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, C. Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks, 2021.
[43] Chengshu Li, Fei Xia, Roberto Martín-Martín, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Elliott Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, Karen Liu. Hyowon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. In Alcksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 455 455-465. PMLR, 08-11 Nov 2022. html
URL httph:tt// http:tppp///rrcceiggn..... .ml. press/v164/11225 html
[44] Bryan Lim, Luca Grillotti, Lorenzo Bernasconi, and Antoine Cully. Dynamics-aware quality- diversity for efficient learning of skill repertoires. In 2022 International Conference on Robotics and Automation (ICRA), pages 5360-5366. IEEE, 2022.
13

---

[45] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 05118499-84773 2021.
[46] Haozhe Liu, Mingchen Zhuge, Bing Li, Yuhui Wang, Francesco Faccio, Bernard Ghanem, and Jürgen Schmidhuber. Learning to identify critical states for reinforcement learning from videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1955-1965, 2023.
[47] Francesco Locatello. Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pages
PMLR, 2020.
[48] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Pata-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
[49] Seohong Park, Jongwook Choi, Jackyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. In International Conference on Learning Representa- tions, 2021.
[50] Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability-aware unsuper- vised skill discovery. arXiv preprint arXiv:2302.05103 2023.
[51] Shubham Pateria, Budhitama Subagdja, Ah-Hwee Tan, and Chai Quek. End-to-end hierarchical reinforcement learning with integrated subgoal discovery. IEEE Transactions on Neural Networks and Learning Systems, 33(12):7778-7790, 2021.
[52] Judea Pearl. Causality, Cambridge university press, 2009.
[53] Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. Advances in Neural Information Processing Systems, 33:3976-3998 2020.
[54] Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. Mocoda: Model-based counterfactual data augmentation. Advances in Neural Information Processing Systems, 35: 18143-18156, 2022.
[55] David Poole and Nevin Lianwen Zhang. Exploiting contextual independence in probabilistic inference. Journal of Artificial Intelligence Research, 18:263-313, 2003.
[56] Rafael Rodriguez-Sanchez and George Konidaris. Learning abstract world models for value- preserving planning with options. In NeurlPS 2023 Workshop on Generalization in Planning, 2023.
[57] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[58] Maximilian Seitzer, Bernhard Schölkopf, and Georg Martius. Causal influence detection for improving efficiency in reinforcement learning. Advances in Neural Information Processing Systems, 34:22905- 229182291112 2021.
[59] Wonil Song, Sangryul Jeon, Hyesong Choi, Kwanghoon Sohn, and Dongbo Min. Learning disentangled skills for hierarchical reinforcement learning through trajectory autoencoder with weak labels. Expert Systems with Applications, page 120625, 2023.
[60] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2): 181-211, 1999.
[61] Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martín-Martin and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes. arXiv preprint 2035022408.03339, 2024.
[62] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukeuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pages 3540-3549. PMLR, 2017.
14

---

[63] Zizhao Wang, Jiaheng Hu, Peter Stone, and Roberto Martín-Martin Elden: Exploration via local dependencies. Advances in Neural Information Processing Systems, 36, 2024.
[64] Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1-6, 2022. URL httt:://jllr.org/ppperr..
v23/21-1122 html
[65] Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896): 223-228, 2022.
[66] Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Self-supervised visual reinforce- ment learning with object-centric representations. In International Conference on Learning Representations, 2021. URL https:t/oppeeeeeiiieee
[67] Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options. arXiv preprint arXiv:2101.0652), 2021.
15

---

A Factored Skills
Learning to reach both desired graph and diversity parameter through primitive actions is challenging. First, different graphs often have substantially different characteristics, with some graphs that are easy to achieve (eg. action aetion+tageni) and others that are quite challenging and rare (eg. agent, knife, fruit->fruit). Not only would it be challenging for single policy to encode all of these behaviors, the diversity parameter notwithstanding, but over-training the frequency at which certain graphs are called might vary significantly. Rather than trying to learn a single monolithic policy, then, instead structure the skill parameterized policy Tkkill as collection of factored skills: A for each factor i { N.
This modification to the policy structure results in three changes: 1) The upper-level action space passes a single row of the graph G, denoted with Is. and the desired factor i. 2) Instead of achieving an entire graph use the achieved row [gachieved, gil. 3) The history of seen graphs H. is replaced with history of factored graph rows H.p.
Define the history of graph rows as Hy == (unique (i, Iachieved, Vi E EI...., N gachieved E )D)- This takes the unique graph rows from all those seen in previous data. Then the upper policy uses the same historical sampling procedure as with unfactorized graphs: the policy samples discretely from the new history, which will by default return 9i. a graph row, and the desired factor. This resolves points 1,3. Point is addressed by replacing Equation with 1 [fachieved. = 9i].
Empirically, found that without this change, the lower policy rarely learns anything, even simple control of the agent.
B Environment Details
In this section, we provide detailed description of the environment, including its semantic stages representing internal progress toward task completion, state space, and action space. We also highlight that while each task consists of multiple semantic stages, agents do not have access to this information.
(a) Installing Printer
(b) Thawing
(c) Cleaning Car
(d) iGibson
Figure 7: Environments.
Installing Printer As shown in Fig. aa). the Installing Printer enviromment is relatively simple, consisting of 3 factors: the agent, printer, and a table. The task requires the agent to complete the following stages: (1) pick up the printer, (2) bring the printer and place it on the table, and (3) turn on the printer. The discrete state space consists of (i) the agent's position and direction, (ii) the positions the printer and whether
on or off, and (iii) the position of the table. The discrete action space consists of (i) moving forward, turning left or right, (ii) picking up / placing down the printer, and (iii) turning on / off the printer.
Thawing As shown in Fig. 7 bb) and Fig. 10 a), the Thawing environment consists of 6 factors: the agent, sink, refrigerator, and three frozeen objects: fish, olive, and date. Thawing each object requires the agent to complete the following stages: (1) move to and open the refrigerator, (2) take the frozen fish out of the refrigerator, (3) put the fish into the sink, and (4) turn on the sink to thaw it. The discrete state space consists of (i) the agent's 's position and direction, (ii) the positions of all environment entities, (iii) whether the sink door turned on, (iv) whether the refrigerator door opened, and (v) the thawing status of three objects. The discrete action space consists of (i) moving forward, turning left or right, (ii) opening closiig the refrigerator, (iii) turning off the sink, and (iv) picking up/ placing down each object.
16

---

Cleaning Car As shown in Fig. cc) the Cleaning Car environment consists of 7 factors: the agent, a car, a sink, a bucket, shelf, rag, and a piece of soap. Cleaning both the car and the rag requires the agent to complete the following stages: (1) take the rag off the shelf, (2) put it in the sink, (3) toggle the sink to soak the rag up, (4) clean the car with the soaked rag. (5) take the soap off the self, and (6) clean the rag with the soap inside the bucket. The discrete state space consists of (i) the agent's position and direction, (ii) the positions of all environment entities, (ii) whether the sink is turned on, (iv) the soak status of the rag, (V) the cleanness of the rag, and (vi) the cleanness of the car. The discrete action space consists
(i) moving forward, turning left or right, (ii) turning on / off the sink, and (iii) picking up / placing down the rag / soap.
iGibson As shown in Fig. 7 d). the iGibson environment consists of factors: the robot, knife, a peach, and a sink. The robot can do the following things: (1) grasp peach: move close to the peach and grasp it, (3) wash peach: grasp the peach and place it into the sink, (3) grasp knife: move close to the knife and grasp it, (4) cut peach: grasp the knife and use it to cut the peach. The continuous state space consists of (i) the robot's proprioception, (ii) the poses of all environment entities, and (iii) whether the peach is cut. The continuous action space consists of (i) end-effector position change, (ii) base linear and angular velocity, and (iii) gripper torque (to open/close the gripper).
Though looking conceptually simple, we emphasize that these environments are challenging because of the following factors:
The state factors are highly sequentially interdependent. making skill learning and task learning challenging: for example, in cleaning car environments, the agent can't 't clean the car until it picks up the rag, turns on the sink, and soaks the rag. These interdependencies between state factors pose great challenges to the agent's 's exploration ability.
During the skill-learning stage, we would like the agents to learn all possible skills (e.g., manipu- lating all objects) rather than learning manipulate single object.
During task learning, we use sparse rewards, and thus it is further challenging for agents to explore.
In addition, we use primitive actions, and many actions have no effect if it's not applicable in the current state. So the task especially challenging for exploration.
C Local Dependencies and Causal Inference
In this work, we define local dependencies according to the state factors X (x!,... N and event of interest Y. which in the context of an MDP is subset of the next state factors (X'), .....,'N), M). In the factored MDP formulation [8) we assume that p. the transition dynamics, represented by dynamic Bayesian network (DBN) which is time-directed bipartite graph, with edges only from factors in X to factors in X'. In this work, we assume that the underlying ground truth DBN, that is the transition function p. can be decomposed according to subsets of state factors X, such there exists p. (Y = y|X = x) for every state.
The factored transition dynamics analogizes with causal inference in the following way: If the state factors and next state factors are each assigned a causal variable by adding the assumption that they can be independently intervened on, and each next state variable carries an associated unobserved noise variable U., which we assume is independent of any X not connected to X' and any other next state variable X"), then we can represent the transition dynamics with a structural causal model (SCM) [52]_ graph connecting the causal variables in X the causal variables in X'.
For a particular outcome variable Y that is one of the next state causal variables X', we can describe local dependence in the RL context according to assumptions about the structural causal model. Represent the non-noise parents of Y as pa( (Y), and the noise parents as pa,r(()) Under normal causal assumptions, the structural causal model for Y is function fy (pa(Y,, (Y)) Y. Define X as a subset of the endogenous parents of Y and Ü as an equivalent subset of the noise variables. Further define the values that pa(Y), Pay (Y), X, Ü can take on as pa(y).payr()) (v), T. ä respectively, and (,), X, 27 as the set of states the parents of Y, the variables in X and variables in U can take on respectively.
(Y),
To formalize local invariance, we add the assumption that fr can be decomposed into series of func- tions (X, T1, Uj ūz),... ..., fy. = I, Un (x)) and gy (pa(Y) pa(y), pay (Y)
17

---

pag(y)), where each fy;: : & xu - ) and g: pa())) + fl..... k). function mapping the parents of Y one the functions. Then f is represented as:
f(pa(x), pap (v)) = (pa(y) pay (a) = =i (2,, u.)
(6)
The local dependence of Y = in particular state (x, x') ) is then the set of variables in X; for the particular where (gy (pa(y), (B))
and the pCMI test is way of uncovering these
local dependencies from observational data.
Local dependence has been investigated in the field of context-specific independence 55 9 which seeks to find particular assignments independent of some subset of the inputs. In particular, context-set specific independence [ determines if variable is independent of other variables on particular subset of states, described as
subset of the causal variables under which an outcome
the partial context set. While our work uses the PCMI test described in Equation 2 context-specific independence focuses on complete independence using knowledge of the structural model.
Alternatively, interactions be viewed as the causes (X) of particular effects (Y), which have also been investigated under the description of token or actual cause [25] (as opposed to general cause). Actual cause utilizes series of counterfactual tests to determine if cause is necessary, sufficient, and minimal for an outcome. Actual cause has primarily been applied in simple, discrete examples [5] 24] making it difficult directly apply RL. However, recent work has incorporated the notion of context-specific independence and extended actual cause to more complex domains [15
D Implementation Details
The hyperparameters of skill learning and task learning can be found in Table I As it challenging to identify local dependencies using learned dynamics models in Thawing and iGibson environments, we use ground truth local dependencies from simulator. The codebase built on tianshou [64] for backend RL, though with significant modifications.
The 5 seeds selected are 0 4. The experiments were conducted machines of the following configurations:
Nvidia A40 GPU; Intel(R) Xeon(R) Gold 6342 CPU @2.80GHz
Nvidia A100 GPU; Intel(R) Xeon(R) Gold 6342 CPU @2.80GHz
E
Additional Results
E.1
Interaction Graph Diversity
Figure iillustrates the percentages of episodes where all local dependencies have been induced at least once, in Mini-BH Cleaning Car. Again, SkiLD (ours) induces all inducible dependency graphs, while baselines fail to induce hard graphs with challenging pre-conditions.
The meaning of the graphs are:
agent, action + agent: agent moving.
agent, rag, action - rag: agent picking up the rag moving it.
agent, soap, action > soap: agent picking up the soap or moving it.
agent, X > agent: X blocking the agent's 's motion.
agent, sink, action - sink: agent turning on or off the sink.
agent, sink, rag, action rag: agent soaking the rag in the sink (which requires that the sink is turned on).
sink, rag - rag: the same above.
car, rag car: the rag cleaning the and getting dirty (which requires that the rag is soaked).
rag + rag: the same as above.
18

---

Table 1: Parameters of Skill Learning and Task Learning. Parameters shared if not specified.
Name
Environments
Printer
Thawing
Cleaning Car
iGibson
algorithm
Rainbow
TD3
step
3
5
skill horizon
30
100
Skill Policy
exploration noise
0.4
0.2
MLP size optimizer
[512, 512]
Adam
learning rate
x 10-*
batch size
64
algorithm
PPO
optimizer
Adam
learning rate
10
Graph Selection
batch size
1024
Policy
clip ratio
0.1
MLP size
[512, 512]
GAE A
0.95
entropy coefficient
optimizer
Adam
learning rate hatch size number of attention layers attention emhedding size number of heads
x 10
Learned Dynamics Model
128
128
algorithm
PPO
MLP size optimizer
[512, 512]
Adam
Task Skill Selection Policy
learning rate batch size
10
1024
clip ratio GAE A entropy coefficient
0.1
0.95
0.02
# of random seeds diversity reward coefficient 8
Training
0.5
bucket, soap - bucket: the water in the bucket getting soap in it.
bucket, rag - rag: the rag getting cleaned the bucket (which requires that the rag is dirty and the water in the bucket gets soap in it).
E.2 2D Minecraft Results
In addtion the environments shown in the paper, we further evaluating our method in larger-scale settings in 2D Minecraft with 15 state factors following Andreas et al. [1]
The state space (15 state factors) consits of: the agent (location and direction), 10 environment entities (the positions of wood, grass, stone, gold, and 4 rocks surrounding the gold), and 4 inventory cells (i.e., the number of stick, rope, wood axe, and stone axe that the agent has).
The action space (9 discrete actions) consists of:
4 navigation actions: moving up, down, left, right,
picking up the environment entity in front, which has no effect if the agent does not have necessary tool for collecting it,
4 crafting actions: crafting a stick/rope/wood axe/stone axe, no effect if the agent does not have enough ingredients.
In the downstream Mine Gold task, the agent will receive sparse reward after finishing all the following steps
19

---

agent, action > agent
100.0
agent, rag, rag
9.0
176.1
agent, soap, soap
+13.7
0.9
72..7
sink agent
651.0
H4.2
164.8
agent sink, action >> sink
H47.9
H+
2.5
61.6
agent, rag agent
14466
agent, car agent
52.4
54.7
agent, agent
42.6
1499
50.5
agent, agent
12.5
133.9
agent, sink, action rag
0.1
0.0
25.5
sink, rag >> rag
10.4
0.0
car, rag >> car
soap
0.0
13.7
car, rag rag
0.0
0.0
(ours)
10.5
CSD
bucket, rag
-0.0
0.0
DIAYN
visitation likelihood (%)
Figure 8: Among all inducible dependency graphs, the percentage of episodes where each graph is induced through random skill sampling. Standard deviation is calculated across five random seeds.
collecting unit of wood craft wood stick,
collecting another unit of wood and combining it with the stick to craft a wood axe that is required for collecting the stone and for removing the rock,
collecting unit of wood and unit of stone to craft stick and then collecting the gold, remove the rock surrounding the gold and collect the gold with the stone axe.
axe that required for
As shown in Fig. 9 SkiLD still outperforms all baselines in this complex task, demonstrating the usefulness of its learned skills for downstream task solving.
Skill Visualizations
In Figure I0 we visualize three challenging long-horizon skills learned by SkiLD; thawing the olive, cleaning the car, and cutting the peach. All of these skills require sequence of interactions that is difficult recover without directed behavior. Thus, comparable basclines do not learn skills of similar complexity. More skill visualizations can be found at: https: htttps://wagzztt /awsss////gzzzzaoooo.o ..ooo.giii github. io/SkiLD/
20

---

1.0
0.5
0.0
1M
2M
3M
4M
5M
(a) Mine Gold
SkiLD (ours)
DIAYN
CSD
ELDEN
COInS
Vanilla
Figure 9: Training curves of SkiLD and baselines on the 2D Minecraft downstream task (reward supervised second phase). Each curve depicts the mean and standard deviation of the success rate over 5 random seeds. SkiLD outperforms all baselines, converging faster and to higher returns.
21

---

Thaw Olive Skill
Clean Car Skill
(c) Cut Fruit Skill
Figure 10: Policy rollouts for learned policies that achieve long horizon tasks (a) Mini-BH thaw olive,
(b) Mini- BH clean car, (b) iGibson cut peach.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We make the following claims in the abstract and introduction.
(a) Existing methods that reach diverse, distinguishable states can struggle in complex environments with state factors because they focus on simple state factors: This result is empirically verified in the performance results in experimental domains in Figure 5 where DIAYN and CSD struggle to capture any of the skill requiring the manipulation of multiple state factors, such cleaning the also demonstrate how CSD and DIAYN fail to capture graphs that cover multi-factor interactions.
cutting the peach. In Figure 4 we
(b) Guiding skill learning with interactions and exploring state coverage within that space can uncover desirable interactions: This result is empirically verified in the performance results of SkiLD illustrated in Fi where SkiLD is able handle complex, multi-object interactions such as cleaning the rag after cleaning the car or cutting the peach. Figure illustrates how desirable interactions like putting soap in the bucket or cleaning with the rag are uncovered.
(c) The skills learned divected by local dependencies can be reused to efficiently maximize task rewards: Again, Figure iillustraee that compared to vanilla PPO learning, SkiLD, which uses local dependencies, is able to reuse skills with greater sample efficiency, and often even higher downstream performance. In these results, the pretrained skills are reused across tasks in the same domain, such as thawing the fish, thawing the olive and thawing the date. We provide details about tch training procedure in Section 444 (d) Combining local dependencies with state diversity important induce diverse in- teractions: This claim is supported in Section 4.5 and Figure 6 where performance noticeably declines after removing the local dependencies and state diversity individu- ally, especially for more long horizon inlliiinteeaction tasks such as cleaning the car or cleaning the rag.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We explicitly discuss the limitations of this work in the second paragraph of the Conclusion section.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
23

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: We do not make any theoretical claims, since this work primarily focuses on incorporating an inductive bias into unsupervised skill learning and empirical assessment of that claim.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Section 3 provides a complete description of the method, with a diagram illustrating the dataflow in Figure 2 The environments are described in Section 4. with additional details Appendix B The number of seeds and amounts data used for the skill pretraining and task learning is provided in Section 4.4 We provide all hyperparameters in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
we provide all code for the envi- ronments, as well as install dependencies for training SkiL in the Readme of the interac- tionHRL. folder. With the environments, any data for this environment is generated. We also provide a description of the compute requirements for running these experiments in Appendix D
Justification: In https://github.ccom https///ghhhbb.ooo/wwaihhh/////i
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ptaiiss.cc/uulli/guuieess /ooeeees//ooeeemmneeo ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
25

---

6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The core experimental setting is presented in Sec. The detailed settings are included in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We provided error bars in all quantitative results, which are standard deviations computed from 5 random seeds.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify the computation resources used in experiments in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
26

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp:///aarrggg..c/ppaala//aaaa
Answer: [Yes]
Justification:
We did violate the NeurIPS Code Ethics in this research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: To the best of our knowledge, the work performed does not have notable societal impacts that need be explicitly addressed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is such risk.
Guidelines:
27

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cited the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswith thoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
28

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "Uv13Xao8Io", "reviewer": "Reviewer_dKfG", "summary": "The paper introduces SkiLD, a novel method leveraging state factorization to guide skill learning in unsupervised reinforcement learning. SkiLD emphasizes learning skills that induce diverse interactions between state factors, which are crucial for solving downstream tasks. The authors demonstrate that SkiLD outperforms existing unsupervised RL methods through empirical validation.", "questions": "Please refer to Weakness", "limitations": "Please refer to Weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Novel Approach**: SkiLD introduces a unique approach to skill discovery by focusing on local dependencies and interactions between state factors, addressing the limitations of state diversity methods.\n2. **Empirical Validation**: The effectiveness of SkiLD is demonstrated through experiments in various environments, showcasing its superior performance compared to baseline methods.", "weaknesses": "1. **Assumption of Factored State Space**: SkiLD assumes access to a factored state space, which may not always be available or easy to obtain in real-world applications.\n2. **Evaluation on Limited Domains**: The evaluation domains are somewhat restricted. Broader evaluation across more commonly used environments could emphasize the method's performance. Evaluating environments like Crafter, 2D Minecraft, or manipulation environments with multiple objects would better illustrate the benefits of focusing on local dependencies.\n3. **Scalability**: There is a need to explore how SkiLD scales with an increasing number of state factors, which is crucial for practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the valuable and constructive feedback. We are particularly excited that the reviewer finds our idea of learning skills to induce diverse interactions between state factors novel and our evaluation sound. \n\nPlease find below our responses to address your concerns regarding our work:\n\n>Assumption of Factored State Space: SkiLD assumes access to a factored state space, which may not always be available or easy to obtain in real-world applications.\n\nWith the recent advances in object segmentation and object-centric representation techniques, such as SAM [1] and SOLV [2], one potential way to construct the factored state space is to use these models to extract factored representation from real-world image observations.\n\nWe agree that testing the effectiveness of SkiLD with learned factored representations would be an interesting next step, and will add it to the future work section of the next version of this paper.\n\n[1] Ravi, Nikhila, et al. \"SAM 2: Segment Anything in Images and Videos.\" arXiv preprint arXiv:2408.00714 (2024).\n[2] Aydemir, Görkay, Weidi Xie, and Fatma Guney. \"Self-supervised object-centric learning for videos.\" Advances in Neural Information Processing Systems 36 (2024).\n\n>Evaluation on Limited Domains: The evaluation domains are somewhat restricted. Broader evaluation across more commonly used environments could emphasize the method's performance. Evaluating environments like Crafter, 2D Minecraft, or manipulation environments with multiple objects would better illustrate the benefits of focusing on local dependencies.\n>Scalability: There is a need to explore how SkiLD scales with an increasing number of state factors, which is crucial for practical applications.\n\nWe appreciate the reviewer’s suggestions and agree that further empirical testing in other commonly used or larger-scale settings is an important direction for future work.\n\nMeanwhile, we emphasize that, while exploring scaling further is valuable, we believe that it will not fundamentally change the scientific insights we introduced – compared to prior works in unsupervised skill discovery, by focusing on local dependencies between state factors, SkiLD learns skills that induce more diverse interactions and enable more sample-efficient downstream task learning, for the following two reasons:\n\n1. The environments used in the paper are also challenging and baselines fail to learn diverse skills and solve downstream tasks. \n   * Specifically, the mini-behavior environments have the same **inter-dependency between state factors** as 2D Minecraft (e.g., in Cleaning Car, the agent cannot clean the car until it soaks the rag in the sink).\n   * For the iGibson environment, despite the four task-relevant factors, it also contains many intractable objects such as jars, microwaves, garbage cans, etc.\n\n   In contrast, **most prior works in unsupervised discovery are only evaluated in environments with one or two state factors (where skills are limited to moving the agent to different locations)**. As a result, as shown in Fig 4 and 5, when evaluated on Mini-behavior and iGibson,  baselines fail to learn diverse skills and solve downstream tasks. Compared to them, SkiLD learns to induce those complex inter-dependency graphs and solves tasks successfully.\n2. In principle, nothing in our method prevents it from scaling to more state factors.\n   * For methods that focus on reaching diverse states (like DIAYN), due to the **exponentially growing state space**, it would be more challenging to learn to induce meaningful interactions.\n   * In contrast, **the number of inducible interactions typically increases much more slowly**. As a result, by focusing on inducing diverse interactions, our method in principle has larger advantages than existing methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AAa7FxqaWu", "reviewer": "Reviewer_jfpH", "summary": "The paper introduces SkiLD (Skill Discovery from Local Dependencies), an unsupervised skill discovery method. Unlike existing methods that focus on state diversity, SkiLD leverages state factorization to guide skill learning by inducing diverse interactions between state factors.\n\nThis method is designed to be more effective in complex environments, such as household settings with numerous objects. SkiLD uses local dependencies to model interactions and introduces a novel intrinsic reward mechanism.\n\nThe method is evaluated in several domains, including a realistic household robot simulation, demonstrating superior performance compared to existing methods.", "questions": "The two points raised within the weaknesses limit the applicability and thus the impact of the paper. Do you have insights on this? Especially, how the method can be made more general?\n\nNitpicking:\n> L88: Formally, for an event of interest $Y$ and its potential causes $X = (X^1,  ..., X^N )$, given the value of $X = x$, local dependencies focus on which $X$ is are the state-specific cause of the outcome event $Y = y$.\n\nI think the vocabulary used in this sentence is wrong.\n- $Y$ is not an event but a random variable.\n- $Y = y$ is an event.\n\n> L108: In this work, for a transition $(\\mathcal{S} = s, \\mathcal{A} = a, \\mathcal{S}^\\prime = s^\\prime)$...\n\nIf I understand correctly the notations, this shouldn't be $\\mathcal{S}$ and $\\mathcal{A}$ but $S$ and $A$, because $\\mathcal{S}$ and $\\mathcal{A}$ are the state space and action space and are not random variables. Plus, you said in the Background section that \"In this paper, we use uppercase letters to denote random variables and lowercase for their specific values...\".\n\nSame remark for L149-150.\n\n> In this section, we describe SkiLD, which enhances the expressivity of skills using local dependencies. SkiLD represents local dependencies as state-specific dependency graphs, defined in Sec. 2.2.\n\nState-specific dependency graphs are actually not defined in Section 2.2, there is just a sentence at the end of the section. It would be useful to better explain what are these graphs.", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The clarity, presentation and writing of the paper are great.\n- The problem of unsupervised skill discovery is an important one.\n- The paper presents strong empirical results, demonstrating that SkiLD outperforms other unsupervised reinforcement learning methods in various challenging tasks.\n- The experimental setup is properly designed (i.e., right choice of baselines and domains).", "weaknesses": "Minor weaknesses:\n- The method's reliance on accurately detecting and modeling local dependencies adds a layer of complexity that may limit its applicability.\n- The effectiveness of SkiLD hinges on the availability of a factored state space, which may not always be available or easily obtainable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the valuable and constructive feedback. We are particularly excited that the reviewer finds our presentation clear and our evaluation sound. \n\nPlease find below our responses to address your concerns regarding our work:\n\n>The method's reliance on accurately detecting and modeling local dependencies adds a layer of complexity that may limit its applicability.\n\nWith respect to detecting and modeling local dependencies, two possible avenues offer promising directions:\n* Relaxing the dependence of precise local-dependency identification by using a window of states rather than a per-state reward when incorporating dependency information.\n* Since many state-specific dependencies fall into broad categories (contact, co-occurrence, etc.), employing a meta-learning strategy for identifying local dependencies in a lifelong setting can improve the effectiveness of dependency identification.\n\n>The effectiveness of SkiLD hinges on the availability of a factored state space, which may not always be available or easily obtainable.\n\nWith respect to factored states, we agree that SkiLD requires a factored state space. In the meantime, we believe that advances in vision, especially related work such as SAM [1] and other object-centric modeling [2] offer a path forward for extracting a factored state space from a dense state such as pixels, thereby extending the application scope of SkiLD.\n\n[1] Ravi, Nikhila, et al. \"SAM 2: Segment Anything in Images and Videos.\" arXiv preprint arXiv:2408.00714 (2024).\n\n[2] Aydemir, Görkay, Weidi Xie, and Fatma Guney. \"Self-supervised object-centric learning for videos.\" Advances in Neural Information Processing Systems 36 (2024).\n\n>Regarding the usage of symbols\n\nWe appreciate the detailed feedback and will make sure we use the correct symbol in the paper!\n\n>Regarding the definition of state-specific dependency graphs\n\nWe appreciate this insightful comment and will add definitions of state-specific dependency graphs to section 2.2 to help it better integrate into section 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S1wfAJczTF", "reviewer": "Reviewer_c2kz", "summary": "In the presented paper the authors propose a novel skill discovery method called Skill Discovery from Local Dependencies (SkiLD). The method utilizes the concept of local dependencies to incorporate the interaction of factors in a factorized state space. A novel intrinsic reward signal is introduced to guide the skill discovery. In the skill learning phases, different skill-conditioned policies are learned for the skills and in the task learning phase these (frozen) skills are used to learn a task policy solving more complex downstream tasks. The method is evaluated in two simulation environments and compared to several baseline methods.\n\n---\n**Post-Rebuttal/Discussion**: I appreciate the discussion with the authors and their effort in clarifying open questions, my main criticism wrt presentation and evaluation are not solved though, and I remain my score of 5.", "questions": "How can the approach deal with varying number of objects? The policies depend on the states and the state-specific graph?\n\nDoes the chosen representation allow for ‘infinite’ number of skills? How does the approach scale with increasing skill number? How does it effect the learned policies (skill and task policy)?\n\nHow are the two learning phases (skill and task) orchestrated? Is each phase done once? Or do they take turns, if so, how?\n\nIn the factored MDP, is the action space A defined over the full S, or does it take the factorization into account?\n\nL168: ‘…we utilize a factorized lower-level policy, where there is a separate policy for each factor.’ So each skill is realized with multiple policies? How is this modelled and especially used? How are skills then chosen, does this affect the action set of the task policy?\n\nNotation in Figure 4 (and same in text) is unclear, what does the ‘x,y -> x’ notation represent?\n\nFigure 1 seems to be never referenced?\n\nHow are the skill policies modelled? Table 1 only give learning details, is it a neural network? What are its details? Same for other methods, the presented parameters are not complete.\n\nWhat does ‘state-specific’ dependency graph mean? \n\nFigure 6 misses the plot description (is it mean and std as well?).", "limitations": "Limitations are discussed, focusing on the assumption of known state factorization and accurate detection of local dependencies.\n\nAn additional (potential) limitation is related to the scaling of the approach wrt. the training (data, time, ..), the number of objects and skills (rather low currently), or the transfer of skills to novel objects.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper tackles an interesting, challenging and important research direction: equipping agents to acquire skills autonomously and in the absence of a concrete task that can then be used in given tasks.\n\nThe main idea behind the method – utilizing local dependencies/interactions (realized with dependency graphs) and diversity – is well motivated and introduced.\n\nOverall the paper is written well and understandable, notations are solid, and the figures are of good quality (especially Figure 2 gives a good overview of the method).\n\nThe explicit formulation of the two main research questions is appreciated and gives a good structure and focus for the evaluation.\n\nSimilar, the explanations of baselines and especially what the respective comparison highlight is a very nice. (Explanations of differences to the different related work sections as well). \n\nAdditional ablations show the effect of the main idea.", "weaknesses": "My main concern is related to the use of overexaggerations and mismatch in some claims and with respect to the experimental results. For example, ‘…resulting in a robust set of transferable skills’, there is no evaluation or similar regarding robustness (in fact that word only occurs in that claim). Or more prominent, the domains are introduced with ‘…a LARGE number of state factors.’, while the state factors range from only 3 to 6 (Section 4.1).\n\nWhile mentioning the research questions is good, the chosen evaluations do not fit these questions very well. Q1 is about the diversity of the interactions, but Figure 4 then only shows a comparison of some ‘particular’ (how are they chosen?) interactions. For a diversity comparison a general metric comparing all found/used interactions should be compared. The number of skills is also not shown/discussed or evaluated.\n\nSimilar, Q2 is about more efficient downstream learning. Efficiency in terms of what exactly? The according evaluation in Figure 5 (and Section 4.4) measures the performance. Is performance used as a proxy for efficiency here?\n\nAs there are many details, modules and methods, it is not completely clear what is assumed to be given/known, or how certain quantities are inferred/used. For example, the events of interest (every factor in just the next_state of the s,a,s’ tuple?) or the target dependency graph (a crucial parameter, and lines 157ff mentions HER for that, but not for all cases?). What exactly does the diversity parameter do? How is the exact flow of the skill learning phase (getting the target graph, the intrinsic reward, learning the policies, and multiple policies can be learned for each skill?)? Some additional effort in clarifying such details would greatly benefit the paper.  \n\nSome critical details, like that for some setups the ground truth dependencies were used instead of the learned, are only provided and thus somehow hidden in the Appendix.\n\nAdditional comments are given in the Questions.\n\nWhile computational demanding and, hence, understandable, 5 seeds/runs are quite low for comparing RL based algorithms (known problem in the community, although often neglected/ignored). Recent frameworks and metrics have been proposed tackled at these problems related to low number of runs and comparing overall algorithms performance [1]. It would be beneficial to add such metrics (e.g., performance profiles and IQM/IQR) to the paper for a better comparison of the proposed algorithm.\n\n----------- \n\n[1] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the valuable and constructive feedback. We are particularly excited that the reviewer finds our method well-motivated and our presentation clear. \n\nPlease find below our responses to address your concerns regarding our work:\n\n>Regarding the use of overexaggerations and mismatch in some claims and with respect to the experimental results\n\nWe appreciate the reviewer’s observation and will make sure to eliminate all overexaggerations. We have made adjustments to the language of the paper to better match the empirical results. In particular, we have removed any mention of “large” number of state factors or “robustness”, as well as other possible exaggerations such as “object-rich” or “realistic”, and replace with more precise language such as object-factored and 3D real-world-based simulation.\n\n>Regarding Q1 evaluation\n\nWe appreciate this insightful comment and will incorporate the following clarifications into the paper.\n\nDue to space limitations, we deliberately only show task-relevant interactions in Figure 4, since these interactions are more informative. That being said, we agree with the reviewer that a general metric comparing all found interactions against all inducible interactions will better showcase the diversity of interactions for different methods. To this end, **we additionally show *all inducible interactions* and whether each interaction is induced, in the global response Fig 1**. \n\n**The diversity metric would be (# induced interactions) / (# induciable interactions)**. As shown in global response Fig 1, SkiLD induces all inducible interactions, while CSD only induces 80% and DIAYN only induces 60% of all inducible interactions.\n\nRegarding the number of skills, we will add the following clarification:\n* SkiLD (ours): in the Cleaning Car environment, we uniformly sample from the history of 15 seen graphs and 4 discrete diversity variables: leading to 60 discrete skills\n* DIAYN: we uniformly sample from 64 discrete skills\n* CSD: CSD uses continuous skill space, so there are an infinite number of skills.\n\n>Regarding efficiency in Q2\n\nWe measure sample efficiency in terms of performance after a certain number of environment transitions.\nWe appreciate this comment and will add this description to the paper.\n\n>Regarding details, modules, and methods\n\nWe appreciate this comment and will add the following details to the manuscript. We also attach the pseudocode in the global response Alg. 1 to improve the clarity of the workflow.\n\n* Regarding the events of interest, it is known from the collected transitions.\n* Regarding the target dependency graph, it is sampled from the high-level policy and known.\n* Regarding HER, following the original paper, we apply it to 87.5% of sampled trajectories, and we will add this hyperparameter to the paper.\n* Regarding the diversity variable,\n   * Fig 1 mentions that the skill has two goals: (1) inducing the target dependency graph, and (2) inducing the graph in diverse ways (i.e., reaching different states).\n   * Lines 173 - 175 mention that the diversity variable fulfills the second goal, in the same way as DIAYN – the agent will be rewarded to visit states if the diversity variable can be distinguished from the states. Hence, the agent is trained to visit different distinguishable states under different diversity variables.\n* Regarding the flow of the skill learning phase: we describe it in the pseudocode in the global response and will include it in the paper.\n\n>Regarding other metrics\n\nWe appreciate this insightful comment and have added IQM scores to the global response.\n\n>Regarding varying numbers of objects\n\nPlease note that for the scope of this paper, we only consider MDPs, which are fully observable by definition, with a known state space (and therefore a known number of objects)).\n\nTo handle varying numbers of objects, one possible extension is to pre-assign a large number of state factors and use placeholders for objects that are not present. The assumption is that objects that are not present do not involve interactions, and their states do not change.\n\n>Regarding allowing for ‘infinite’ number of skills and scaling to it\n\nA simple way to extend to an ‘infinite’ number of skills is to have the diversity variable be continuous instead of discrete. As the number of skills increases, the capacity of the agent’s behavior grows proportionally, allowing for more diverse skills. \n\nIn the meantime, it will also be harder for the skill policy and the discriminator to process these skills, potentially hampering skill learning. Finding the “right” number of skills is often a domain-specific problem\n\n>Regarding two learning phases\n\nEach phase is done once.\n\n>Regarding the action space\n\nIt is defined over the full state space.\n\n> Regarding the factorized lower-level policy\n\nEach skill is realized by one policy network.\n* As described in Appendix Sec A, there are N (N: # of state factors) parameterized low-level networks, one per state factor.\n* When the target dependency graph is specified, we will identify which state factor should be influenced and use its corresponding policy to sample primitive actions\n   * As a result, only one of the N networks is activated at every time step.\n* The task policy still selects the target graph from the history of seen graphs and thus is unaffected by this specific design of low-level policy.\n\n>Regarding Fig 4 notations,\n\n$x,y \\\\rightarrow x$ represents that $x_{t+1}$ locally depends on $x_t$ and $y_t$.\n\n>Regarding ‘state-specific’ dependency graph\n\nA general dependency does not necessarily happen in every state. For example, though a knife can cut fruit, when they are far away, the cutting will not happen. The state-specific dependency graph encodes this state-specific dependency between factors, the edge represents the dependency happening in a given state.\n\n>Regarding Fig 6\n\nIt is also mean and standard deviation, and we will add that clarification to the final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QWmolM7hZQ", "reviewer": "Reviewer_WEDj", "summary": "The authors present SkiLD, an unsupervised RL method that learns skills by augmenting DIAYN reward with a graph-state dependency reward to induce meaningful changes in object interactions.", "questions": "One limitation of all unsupervised skill learning methods is that a significant portion of the learned skills are meaningless. Qualitatively, what proportion of the skills ended up being meaningful? Was it more with SKilD than baselines like DIAYN? This perhaps could be visualized by randomly sampling ~50 skills and visualizing them to see which ones are actually meaningful.", "limitations": "Adequately addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Experiments:** Experiments are performed on a reasonably comprehensive set of 10 downstream tasks in 2 domains, and demonstrate that SkiLD is significantly better than all baselines.\n\n**Novelty and Idea:** The idea of using explicit factors as skill learning reward is interesting and novel. At a high-level, it’s also pretty intuitive.", "weaknesses": "**Clarity: Quite a few clarity issues**\n\n- In the main paper, the method could be presented more clearly at a higher level. For example, introducing the Graph-Selection policy first (in 3.1 instead of 3.2) could help with understanding, at a higher level, what the method is trying to do. Currently the paper first talks about the skill selection policy but, by the time the reader gets to 3.1, it’s not fully clear yet (it’s mentioned at the top of section 3 but then gets into details a bit too suddenly in 3.1) that there is an initial unsupervised skill learning phase and what that phase is trying to do. Going from high level idea → high level policy → low level policy would make this more clear.\n- More explicit details are needed. How is the dynamics model trained? What’s the high-level overview of the algorithm? When and how are diversity rewards applied? What exactly are the input/outputs features of the graph selection policy? All of this could be more clear if the authors provide some pseudocode linked to from the main paper and beef up the appendix.\n- High-level details about environment assumptions (learned/given dynamics, what is given in the graph factorization, etc.) in section 4.1 should be given, even if details are already in the appendix.\n\n**Experiments:**\n\n- Are 10M timesteps really needed even in Mini-behavior domain? This seems extremely sample-inefficient. Same with requiring up to 5M timesteps to learn downstream tasks reasonably in these environments. Perhaps some details about why all methods need so many timesteps would be helpful.\n- How are hparams selected and tuned? And for the baselines? How do we know this is a fair comparison?\n\n**Minor issues:**\n\n- L163: missing end of sentence period\n- Fig 4: CaSk?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the valuable and constructive feedback. We are particularly excited that the reviewer finds our idea of learning skills to induce diverse interactions between state factors novel and our evaluation sound.\n \nPlease find below our responses to address your concerns regarding our work:\n\n>Regarding reorganizing Sec 3\n\nWe appreciate the reviewer's suggestions, and we will rearrange these sections for greater clarity. \n\nWe will also add more detail to the high-level description at the beginning of the section to make it clear what the high-level and low-level inputs are:\n* the high-level policy outputs graphs and diversity parameters.\n* the low-level skills take in both and output primitive actions.\n\n>Regarding details on dynamics model training, when and how are diversity rewards applied, what exactly are the input/output features of the graph selection policy?\n\nWe appreciate this comment and will add the pseudocode to the main paper (it is also attached to the **global response**). We will also add the appendix section addressing the following components,\n* Loss functions for dynamics model: line 167 mentions that the loss is the prediction error.\n\n   Specifically, we measure it as cross-entropy loss for discrete state space and mean squared error for continuous state space.\n* When and how are diversity rewards applied: as shown in Eq. 3, when the induced graph matches the desired graph, the diversity is applied as **an additional reward** to the graph reward.\n* The input/output features of the graph selection policy:\n   * As shown in Fig 2 and line 186, the input is the current state\n   * The output feature is a categorical distribution over N, where N represents the number of unique graphs in the history of seen graphs.\n\n>High-level details about environment assumptions (learned/given dynamics, what is given in the graph factorization, etc.) in section 4.1 should be given, even if details are already in the appendix.\n\nWe agree with this constructive feedback and have added this information to Sec 4.1.\n\n>Why do all methods need so many timesteps?\n\nThough mini-behavior looks simple, the environments are challenging because of the following factors:\n* The state factors are **highly sequentially interdependent**, making skill learning and task learning challenging: for example, in cleaning car environments, the agent can’t clean the car until it picks up the rag, turns on the sink, and soaks the rag. These interdependencies between state factors pose great challenges to the agent’s exploration ability.\n* During the skill-learning stage, we would like the agents to learn **all possible skills** (e.g., manipulating all objects) rather than learning to manipulate a single object. Hence, it requires more samples. Furthermore, Fig. 4 shows that, with 10M timesteps, baselines still fail to learn to manipulate all objects.\n* During task learning, we use **sparse rewards**, and thus it is further challenging for agents to explore. Our results match the results in the mini-behavior paper, which shows that it takes millions of timesteps to learn the task with sparse rewards.\n* In addition, we use primitive actions, and **many actions have no effect if it’s not applicable in the current state**. So the task is especially challenging for exploration.\n\n>Regarding hyperparameters tuning.\n\nWe apply grid search on the following hyperparameters for the following methods to ensure their best performance. We will add these details to the paper Appendix. Due to space limits, we leave the values we searched in the Appendix.\n* SkiLD (ours):\n   * skill-learning:\n      * low-level policy: exploration noise, skill horizon\n      * high-level policy: entropy coefficient\n   * task learning: skill selection policy’s entropy coefficient\n* DIAYN:\n   * skill-learning: low-level policy exploration noise\n   * task learning: skill selection policy’s entropy coefficient\n* CSD:\n   * skill-learning: low-level policy exploration noise, intrinsic reward coefficient, learning rate\n   * task learning: skill selection policy’s entropy coefficient, skill horizon\n* ELDEN:\n   * dynamics model: regularization lambda and its annealing, gradient cutoff threshold\n   * task policy: entropy coefficient\n* COInS:\n   * Dynamics model: \n      * Passive and active model cutoff thresholds\n      * Interaction state reweighting ratio\n   * Granger-causal score threshold\n   * Skill learning:\n      * Goal sampling and relative action sampling distance\n      * Hindsight reweighting ratio\n* Vanilla RL (PPO): entropy coefficient, learning rate, batch size\n\n>Regarding meaningfulness of learned skills.\n\nFig 4 in the paper (full version: Fig 1 of the global response) shows the semantic meaningfulness of skills acquired by different methods (in terms of which interactions they can induce).\n\nSpecifically, for each method, we randomly sample its learned skills (see details of # of skills below) and we record which interactions they induce in each episode. From the global response Fig 1, we can see that\n* SkiLD induces all inducible graphs, suggesting that it learns skills that cover all possible interactions with objects (i.e., at least 15 skills are semantically meaningful), though there may also exist redundant or less meaningful skills.\n* In contrast, for DIAYN, with a similar number of skills, most interactions it induces are agent moving (i.e., agent, action -> agent) and agent getting blocked by other objects (e.g., agent, car -> agent). It suggests that a large portion of skills are navigation skills, and they are less meaningful when solving tasks requiring inducing object interactions.\n\nthe number of skills:\n* SkiLD (ours): in the Cleaning Car environment, we uniformly sample from the history of 15 seen graphs and 4 discrete diversity variables: leading to 60 discrete skills\n* DIAYN: we uniformly sample from 64 discrete skills.\n\n>Regarding typos\n\nWe appreciate the detailed feedback and have corrected these typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback and suggestions. We are particularly excited that the reviewers find our idea of learning skills to induce diverse interactions between state factors well motivated (R3), novel (R1, R2, and R4), effective (R1, R2), and sound (R1, R2, R4). We also appreciate the reviewers for commending the clarity of the technical details (R2 and R3). In this thread, we summarize our responses to the common concerns shared by the reviews. Please refer to the rebuttal attached to each reviewer's comments for a reviewer-specific response.

1. To improve the clarity of the paper, in Alg 1 of the attached pdf, we show a pseudocode of how SkiLD learns diverse skills during the skill-learning stage, and we will add it to the paper.
2. To further evaluate the diversity of each method’s learned skills, as suggested by Reviewer3 c2kz, we show **ALL inducible interactions** and whether each interaction is induced by different methods, in Fig 1 of the attached pdf. Again, SkiLD (ours) induces all inducible dependency graphs, while baselines fail to induce hard graphs with challenging pre-conditions.
3. For a better comparison between different methods on their performance during task learning, as suggested by Reviewer3 c2kz, we compute and report the IQM scores as follows:

| **Task**          | **SkiLD**         | **Elden**         | **COInS**         | **DIAYN**         | **CSD**           | **Vanilla**       |
|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|
| Install Printer   | $0.996\pm 0.005$  | $0.981\pm 0.035$  | $0.000\pm 0.000$  | $0.000\pm 0.000$  | $0.000\pm 0.000$  | $0.886\pm 0.000$  |

| **Task**          | **SkiLD**         | **Elden**         | **COInS**         | **DIAYN**         | **CSD**           | **Vanilla**       |
|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|
| Clean Rag         | **0.016** $ \pm $ 0.031 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 |
| Clean Car         | **0.177** $ \pm $ 0.286 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.177 $ \pm $ 0.286 | 0.000 $ \pm $ 0.000 |
| Soak Rag          | **0.960** $ \pm $ 0.031 | 0.098 $ \pm $ 0.196 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.031 | 0.000 $ \pm $ 0.000 |

| **Task**          | **SkiLD**         | **Elden**         | **COInS**         | **DIAYN**         | **CSD**           | **Vanilla**       |
|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|
| Thaw Olive        | **0.223** $ \pm $ 0.157 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 |
| Thaw Date         | **0.646** $ \pm $ 0.432 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.254 $ \pm $ 0.000 |
| Thaw Fish         | **0.486** $ \pm $ 0.819 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 |
| Thaw any two      | **0.101** $ \pm $ 0.093 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 |

| **Task**          | **SkiLD**         | **Elden**         | **COInS**         | **DIAYN**         | **CSD**           | **Vanilla**       |
|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|
| Peach             | **0.999** $ \pm $ 0.002 | 0.153 $ \pm $ 0.307 | 0.097 $ \pm $ 0.153 | 0.402 $ \pm $ 0.176 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 |
| Wash Peach        | **0.990** $ \pm $ 0.013 | 0.000 $ \pm $ 0.000 | 0.005 $ \pm $ 0.010 | 0.001 $ \pm $ 0.002 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 |
| Cut Peach         | **0.119** $ \pm $ 0.051 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 | 0.000 $ \pm $ 0.000 |

Meta-review: The paper introduces SkiLD, a method for unsupervised skill discovery that uses state factorization to learn skills inducing diverse interactions between state factors. Although the reviewers found the paper interesting with good results, some of them expressed concerns about clarity in the writing and in several technical aspects related to the method (refer to comments by reviewers c2kz and WEDj). Although some of these were addressed in the authors’ responses, the reviewers expressed that the authors failed to address exactly how they would revise the paper to address all the mentioned issues. Apart from this, the reviewers pointed out inadequately justified claims pertaining to a "robust set of transferable skills" without sufficient experimental justification, and mention "large number of state factors" although only 3 to 6 state factors were used. Although the authors have promised to use more precise language, it is unclear how this would affect the overall readability of the paper. Although incomplete, the inclusion of results on the more complex 2d minecraft experiments looks promising. The reviewers also felt their concern regarding the usefulness of skills was not sufficiently addressed. Overall, the paper is relevant and interesting in the context of unsupervised skill discovery. However, it could benefit from clearer presentation, more accurate representations of the scope of the work, with the technical aspects described fully and precisely with appropriate context.
Decision: Accept (poster)