SUSD: Structured Unsupervised Skill Discovery through State Factorization
Authors: Seyed Mohammad Hadi Hosseini
Mahdieh Soleymani Baghshah
Venue: ICLR 2026 Poster (2026)
Keywords: Unsupervised Skill Discovery
Hierarchical RL
Primary area: reinforcement learning
TL;DR: We present a novel factorized method that exploits the compositional structure of environments to acquire diverse and dynamic behaviors.

Abstract: Unsupervised Skill Discovery (USD) aims to autonomously learn a diverse set of skills without relying on extrinsic rewards. One of the most common USD approaches is to maximize the Mutual Information (MI) between skill latent variables and states. However, MI-based methods tend to favor simple, static skills due to their invariance properties, limiting the discovery of dynamic, task-relevant behaviors. Distance-Maximizing Skill Discovery (DSD) promotes more dynamic skills by leveraging state-space distances, yet still fall short in encouraging comprehensive skill sets that engage all controllable factors or entities in the environment. 
In this work, we introduce SUSD, a novel framework that harnesses the compositional structure of environments by factorizing the state space into independent components (e.g., objects or controllable entities). SUSD allocates distinct skill variables to different factors, enabling more fine-grained control on the skill discovery process. A dynamic model also tracks learning across factors, adaptively steering the agent’s focus toward underexplored factors.
This structured approach not only promotes the discovery of richer and more diverse skills, but also yields a factorized skill representation that enables fine-grained and disentangled control over individual entities which facilitates efficient training of compositional downstream tasks via Hierarchical Reinforcement Learning (HRL).
Our experimental results across three environments, with factors ranging from 1 to 10, demonstrate that our method can discover diverse and complex skills without supervision, significantly outperforming existing unsupervised skill discovery methods in factorized and complex environments. Code is available at the anonymous repository: [https://anonymous.4open.science/r/SUSD](https://anonymous.4open.science/r/SUSD).

Published as a conference paper at ICLR 2026
SUSD: STRUCTURED UNSUPERVISED SKILL DISCOV- ERY THROUGH STATE FACTORIZATION
Seyed Mohammad Hadi Hosseini & Mahdieh Soleymani Baghshah Department of Computer Engineering
Sharif University of Technology (hadi. wwssooeeenne ni17, soslllllene soleymani @esharif.ede
ABSTRACT
Unsupervised Skill Discovery (USD) aims to autonomously learn a diverse set of skills without relying on extrinsic rewards. One of the most common USD approaches is maximize the Mutual Information (MI) between skill latent vari- ables and states. However, MI-based methods tend to favor simple, static skills due to their invariance properties, limiting the discovery of dynamic, task-relevant behaviors. Sattance-Mxxiiizingg Skill Discovery (DSD) promotes more dynamic skills by leveraging state-space distances, yet still fall short in encouraging com- prehensive skill sets that engage all controllable factors entities in the envi- ronment. In this work, we introduce SUSD, a novel framework that harnesses the compositional structure of environments by factorizing the state space into independent components (e.g., objects or controllable entities), SUSD allocates distinct skill variables different factors, enabling more fine-grained control on the skill discovery process. A dynamic model also tracks learning across fac- tors, adaptively steering the agent's focus toward underexplored factors. This structured approach not only promotes the discovery of richer and more diverse skills, but also yields a factorized skill representation that enables fine- tieeggraneed and disentangled control over individual entities which facilitates efficient training of compositional downstream tasks via Hierarchical Reinforcement Learning (HRL). Our experimental results across three environments, with factors ranging from 1 to 10, demonstrate that our method can discover diverse and complex skills with- out supervision, significantly outperforming existing unsupervised skill discovery methods in factorized and complex environments. Code is publicly available at: https://githubs www.cc////aaaa. hosseini/SUSD
INTRODUCTION
Reinforcement Learning (RL) (Sutton et al.] 1998) has made significant strides in solving complex
tasks, such strategic games (Schrittwieser et al. 2020 Badia et al. 2020 Mnih al., 2013)
and robotic manipulation (Gu et al. [016]] Andrychowicz et al.) 2020), especially ly when supported
by well-shaped, dense reward functions (Booth et al. [023 Tang et 2025 Silver et al. 2018
However, in many real-world settings, rewards are sparse and tasks are long- long-horizon, making learn- ing much harder.
Therefore, key limitation is the need to mamually design and tune reward functions, which is diffi- cult scale across multiple tasks (Park et al. 2022)) To address this, Unsupervised Skill Discovery (USD) has become widely attended in (Sharma et al. Eysenbach et al. 22019) Park et al.) 2022) 2023 2024 Campos et al. 2020 Achiam et al. 2018)) as an unsupervised RL. approach that enables an agent to autonomously explore its environment and acquire diverse set of useful behaviors. Such pre-trained agents can serve as general-purpose foundation for efficiently learning a wide range of downstream tasks without relying task-specific supervision. Precisely. in USD approach, an agent is placed in an environment without any predefined reward function and learns to acquire as many diverse and distinct skills possible solely through interaction with the environment. After- wards, these learned skills are leveraged by a high-level policy or zero-shot goal- reaching methods to solve a variety of downstream tasks.

---

Published as a conference paper at ICLR 2026
Ss+1
#t
Factorized Enbedding
(())
+())
+t(+)
do(())
ex(+(")
+x(+)
(6)
(o(tin)) o(ei))"
bosk
Taigh (s)
st s, 2)
E 1-
Lugelloin s)r
Ag- Ao
aftis a))
aftii (e)
auriosfty Based Factor weighing
a/a (4)
To(a s,
(e. Get---1
8t
(a)
(c)
Figure I: Illustration of the SUSD Method. (a) In the skill learning stage, factorized embedding of the current and next state is passed through the corresponding mapping function ((-) to obtain a skill latent embedding. Additional details about factorized embedding are shown in (b), where factor s', i [i..... J. together with the skill factor inputs, are used to compute the intrinsic reward r' of factor i. In the curiosity-based factor weighting module, a density model takes the full current state as input and estimates the probability of each next-state factor given the current state - log 1 These probabilities are then used as weights to scale the factor-wise intrinsic rewards, which are summed to form the final intrinsic reward for training the skill policy. (c) In the task learning stage, the learned skill policy is frozen as a low-level policy, while a high-level policy Thigh is trained to select a skill z every steps by maximizing the task reward tack
N.. is mapped its embedding through the function Φ.. These embeddings,
Two main branches of USD have gained popularity. The first focuses on MI methods (Eysenbach et al. 2019. (haarma et al.) 2019 Kamienny et al.] [022 Kim et al. (023), that aim to maximize the mutual information between latent skills and states. The second branch centers on DSD meth- ods (Park et al., 2024 2023 2022 Kim et al.) 2024 Rho et al., 2025), that seek to maximize state changes along skill-specific directions. While MI-based methods often suffer from issues related to invariance transformations,
aspect discussed in prior work (Park al.) 2024 2023). DSD methods attempt to encourage more dynamic behaviors by maximizing state-space distances. How- ever, the latent space of DSD methods mostly emphasize only easily controllable factors. Therefore, although state-of-the-ar DSD methods perform well simple environments such as Ant, HalfChee- tah, and Walker (Todorov al. 2012 Brockman al. 2016 Towers et 2024), their performance degrades in more complex environments consisting several controllable elements. This bias arises because DSD lacks explicit mechanisms to ensure skill diversity across all controllable components. As result, the learned skill latent spaces tend to underrepresent hurder-to-contee or less immedi- ately responsive elements, limiting the overall expressiveness and coverage of the discovered skills.
To address these limitations, we propose factorized embedding approach that explicitly leverages the inherent factorization the environment's state space, an inductive bias commonly present in structured domains such as multi-object environments. Although factorization has been recently explored for skill discovery in MI methods (Hu et al., 20244] Wang et al. 2024), has not been utilized within DSD methods. In our framework, the state space is decomposed into factors, each corresponding to a distinct subset of controllable features. We associate dedicated subset of skill variables with each factor, enabling the agent to develop localized skills within each subspace. This decomposed skill representation also naturally supports composability of skills through chaining which makes the learned behaviors more reusable for downstream tasks. However, unlike (Hu al.
2

---

Published as a conference paper at ICLR 2026
2024 do not impose the more restrictive independence assumption. Furthermore, in contrast to existing DSD methods that treat the state space holistically and fail to account for the varying difficulty of controlling different factors, introduce suriosity-driver factor weighting mecha- nism. This component dynamically adjusts the learning emphasis across factors based on the agent's progress, encouraging exploration of underdeveloped harder-to-conttol components others mastered.
The main contributions of our work are as follows: (I) We propose novel method for skill learning in environments where the state space can be meaningfully decomposed into subspaces, leveraging this inductive bias learn a repertoire of skills that collectively cover the entire state space. (2) We design an adaptive weighting mechanism that assigns greater emphasis to less explored factors, encouraging balanced skill acquisition. (3) We evaluate our approach across multiple environments and experiments, demonstrating that our method outperforms leading DSD methods as well as the MI-based simple and factorized approaches.
2
RELATED WORKS
2.1 UNSUPERVISED SKILL. DISCOVERY
USD methods aim enable an agent to learn diverse repertoire of skills without relying on pre- defined tasks. These learned skills can later be used to solve downstream tasks either in a zero-shot manner or as components of high-level policy strategies. Research this area can be broadly catego- rized into two main directions: (1) Approaches that maximize the MI between skill latent variables and the states. (2) DSD methods, which introduce various distance metrics to explicitly encourage distinguishable behaviors across skills. Our method is built upon the DSD framework.
2.1.1 MUTUAL INFORMATION- BAASDD SKILL DISCOVER
These methods aim to maximize the MI between states and latent skills, [[S; Z), encouraging dif- ferent skills to induce distinct and diverse behaviors. This objective is generally intractable, and previous studies approached it using two main strategies: (1) reverse mutual information (Reverse- MI), and (2) forward mutual information (Forward-MI)). As regard to Reverse-MI (Mazzaglia al.,
2023 Kamienny et al., 2022; Wang et al. 2024 Kim et al. 2023 Hu et al. 2024 Eysenbach et al,, 2019. optimization is in the form of / (S; () H(Z)
H[ (). where H[2) constant due to assuming the skill prior distribution p(z) is fixed. In this approach, a variational lower bound of (S; Z) is obtained as I(S; Z) E.,. llg s)] - = [log p(2)] 2 E..s (o |s]] + const in which neural network go (z is typically used to model p(z[s). Regarding Molward-MI 2022 Sharma et al. 2211). optimization can expressed as
(o
(Liu & Abbeel 2021 Laskin et al.
(S; H(S) (S)Z)). To approximate Z(S neural network is typically used to model a variational distribution. Additionally, maximizing (S) can be achieved through entropy estima- tion techniques. A key limitation of these methods is their invariance scaling any invertible transformation of input variables. As result, they tend to learn simple behaviors such as opening
the arm while failing to capture more dynamic and complex skills (Park et al.) 2022 2023
2.1.2 DISTANCE-MAXIMIAAA SKILL DISCOVERY
Several recent works have adopted DSD to discover diverse and meaningful skills (Park et al. 2022
2023 2024 Rho al. 2025 Kim al. 2024). DSD utilize the Wasserstein dependency measure as learning objective for USD. defined as:
Tw (8; =
(1)
where W(:)) denotes the Wasserstein distance. In contrast to I[S; Z) (KL(p(s, 2 p))sps))) in which -elbbeck-Leibler divergence, DKI., is insensitive to the underlying geometry of the state space, the Wasserstein-based objective explicitly accounts for state-space distances.
3

---

Published as a conference paper at ICLR 2026
DSD methods can intuitively be simplified and expressed as aiming maximize the state change along the direction specified by the skill z with the following objective (Park et al. 2024):
sup E pp(),))
Σ (0(s++1) - o(ss)) 1=0
5.0
s.t. 11((() < S d(d,,) ,8), v(8,8') E Sadj
where, Sadj denotes the set of adjacent state pairs in the Markov Decision Process (MDP), and L is a norm (L2 norm is common in prior works).
The main difference between our work and others is that they typically consider relatively simple environments, such as Ant and HalfCheetah (Todorov et al. 2012; Brockman et al. [016 Towers) et al. 2024), where the state can be easily mapped to Now-dimensional latent space via ((-). Such simple environments with well-defined mapping from state space to latent space can allow them to easily cover the state space. In contrast, they struggle in more complex, object-centric environments where multiple objects and agents are present and even a simple action can simultancously affect several factors.
2.2 STATE SPACE FACTORIZATION IN RL
Many works leverage knowledge of state factorization for various purposes, including planning (Wang et al.] 2022), data augmentation (Pitis et al. 2020), providing intrinsic rewards (Wang et al.
|023| al. 2024). and goal-condin conditiood learning (Chuck et al. 2025). Recently, some methods et 2024 Wang et al. 2024 Chuck et al.) 2023) have been proposed that consider interactions among objects the environment and leverage this inductive bias with mutual information objec- tives to discover diverse skills. For example, DUSDi (Hu et al.] (024)) learns disentangled skills where each skill component influences a specific factor, while SkiLD (Wang et al. (024) induces different interaction graphs to capture object relationships. Although, utilizing the compositional structure of environments help to discover more diverse skills, the existing DSD methods have not yet exploited the state factorization and miss an opportunity to better structure the skill learning process and uncover more comprehensive and composable behaviors.
3 PRELIMINARIES AND PROBLEM SETTING
An MDP is defined by the tuple (S, A, p). where E S denotes a state in the state space, a E A an action in the action space, p(-|s, a) the transition probability function, and S A R+ the reward function. In this work, we focus on a specialized class of MDPs designed for unsupervised skill discovery in reward-fres free Factored Markov Decision Processes (FMDPs) (Karns & Koller
1999
Osband & Van Roy| 2014; Mohan et al. 2024). There is long line of prior work (Wang Hu et al.J 2024] Wang et al. 2023; Eysenbach et al.) 2019 Pitis et al.) 2020, Choi 2024 has adopted Factored Markov Decision Processes (FMDPs) as their environmental
et al.
2024.
et al.
such settings, the state space structured as a Cartesian product of N subspaces: S : S x S.N where each St corresponds to distinct factor of the environment. Each skill is represented by a latent vector Z E Z, where the skill space is factored as Z := Z' x ... x Z with each factor Z E R.P- As a result, the overall skill space lies in IR E D; A shared skill-conditioned policy <(als, 2) maps states and skill vectors to an action distribution. While the skill space Z can be either discrete or continuous, we focus on the continuous setting in this work. Nonetheless, our method is readily applicable to discrete skill spaces too as mentioned Appendix FF] To collect a skill trajectory, we sample skill from a predefined skill prior distribution p(2) the beginning of an episode. We then roll out the skill policy π ( (ss 2) with the sampled z for the entire episode. For the skill prior, use standard normal distribution.
framework.
In settings where the true state vector is available, the underlying factors can typically be derived directly. When only pixel-based observations accessible, encoder can be employed alongside representation learning techniques to extract disentangled factors from the visual input. However, assume direct access to the underlying state vector that available in many environments.
4

---

Published as a conference paper at ICLR 2026
$/2
Contours g(s'
[41
$
a(x' 8)
01
W2
02 $/2
9 (+)
a(
Figure 2: Left: The state is passed to the density model, which estimates the and variance of (ss(ss). These statistics are then partitioned by factors obtain ((s{[s)))=-- Right: Point shows high probability in factor I but low probability in factor 2-a distinction that cannot be leveraged by the CSD method, which assigns a single weight the entire state transition rather than individual state factors.
METHOD
Existing DSD methods tend to focus on easily controllable state factors and lack mechanisms to en- courage diversity across all controllable aspects. To address this, we propose factorized embedding framework for DSD methods that decomposes the state space into distinct controllable factors, en- abling localized and composable skill learning. Additionally, we introduce a curiosity-driven factor weighting mechanism that adaptively emphasizes harder-to-contee factors during training.
In this section, we first describe how the observation space, skill latent space, and mapping function are factorized (Section[4.) 4.1). Then, we explain how the weight of each factor is computed, which re- flects the extent of the agent's struggle with that factor and, consequently, its curiosity to acquire the corresponding skill (Section 4.2). Finally, we present the SUSD method in its entirety, detailing its intrinsic reward, the loss functions used for network updates, and the overall algorithm (Section 4.3).
4.1 FACTORIZED DSD
We factorize the observation space into factors [ each corresponding to a meaningful com- ponent the environment. In essence, leverage the compositional structure of the environment as an inductive bias to guide this partitioning of the observation space. More precisely, different controllable elements in the environment (e.g., the target, agent, and ammo in the 2D-Gunne (Lowe et al. 2017)) can be considered as different state factors. Moreover, each skill vector is composed of N factors, each of dimension D. Specifically. for every state factor s², we associate latent skill factor 2² within the skill vector. This design enables the agent to learn localized skills specialized to each subspace, improving both disentanglement and control. More precisely, the function ş), which maps the state space to the skill latent space, is structured
N scparate networks where each net- work takes the as input and outputs (s*). Therefore, the original optimization problem defined in Eq. 2 is converted to the following factorized version:
T-1 ΣΣ
Ep(T.2) 1
sup
-
1=1 1=
N
subject to
- (st
1,
V(s,s') Sadj
(3)
4.2 CURIOSITY-BASER FACTOR WEIGHTING
Inspired by Park al. 2023) which prioritizes state transitions that difficult to achieve under the current skill policy, we aim to encourage the discovery of hard-to-learn behaviors. To this end, train density model qo (s) (s)((s), Eo(s) on 8') tuples collected by the skill policy
5

---

Published as a conference paper at ICLR 2026
and negative log-likelihood of a transition from the current skill policy, log (st+111s))) is used as a controllability aware distance function. It assigns high values for rare transitions while assigns small values for frequently visited transitions.
As discussed in Section 4.1 we factorize the state space into N distinct factors. This raises the question of which factors the agent should attend at each timestep and how much importance each factor should receive. In this section, we begin with a lemma that allows us to reformulate Eq. so that the distance term is incorporated directly into the objective, rather than appearing in constraint. Using this lemma, we then derive our final objective function.
Lemma 4.1, In the DSD optimization problem (Eq. 2). we can include the distance term as a coefficient alongside the intrinsic reward. More generally. Eq. can be reformulated as follows:
Σ ((((s)) <(s))) =
sup x.
(4)
t=
[[o(s) o(s')li 11, v(s, s') Sulj
The proof of this lemma provided in Appendix A and is adapted from (Kim et al. 2024).
To move beyond the coarse-graid garseegraiied weighting used in (Park et al., 2023). which assigns a single curiosity score per transition, we propose a fine-grained factor-wise weighting mechanism that dy- namically adjusts the contribution of each state factor during training. Specifically, to compute the factor-wise curiosity signals - log qo (s)+1 [st). we feed & into the network to obtain μo ( and di- agonal Es(s;) of multivariate Gaussian distribution. Since the marginals of Gaussian distribution are themselves Gaussian, we can easily extract the marginal mean and variance for each state factor by partitioning (*t) and Eg(st) according to the predefined factorization of the observation space. The resulting factor-wise parameters are used to calculate the curiosity weight for each factor as:
- lo (1)) x (si+1 (s))) (s::) - pb((s))
(5)
The square root of - -log (0(4)) | st) can be interpreted as valid distance metric and thus in- corporated into the objective defined in Eq. + according to Lemma [. The siriosity-bassed fac- tor weighting module is shown on the bottom of Figure [ Furhermore, Figure shows how this mechanism provides curiosity-based factor weighting. In this figure, the point x may have a high probability mass under the first factor while it has low probability mas under the second one. This indicates that more weight should be assigned to the second factor transition than the first one while coarse-grained view of states, used in CSD (Park al. 2023), overlook such factor-level controllability.
Accordingly, we our final optimization problem as follows:
T-1
N
sup
E Ep(r,e)
log (0)()_ -(())))
x, {
N
subject to
- $1, V(s,8') E Sadj
(6)
4.3 SUSD TRAINING
We optimize our objective using dual gradient descent (Park et al. 2024; 2023 2022). That is, with Lagrange multiplier À 2 O. use the following dual objectives train SUSD:
SUSD (0)()()1) (s(()))TTe 2t, ri
JSUSD, = SUSD + X- min (c, 1 - (s)Ii)]
(8)
JSUSD.A - E min (s. 1 - ( - ((s))))),
(9)
R = log qo vo(s) 15))
SUSD
(10)

---

Published as a conference paper at ICLR 2026
where R is the intrinsic reward for the skill policy, and J and JSUSD, are the objectives for 6; and X, respectively. We update for each factor independently. The variables st-1 and Sy are sampled from state pair distribution p($++1, 8t) that imposes the constraint in Eq. 6 The slack variable > O prevents the gradient of from always being nonnegative. Using these objectives, train SUSI by optimizing the policy using Eq. 10 as the intrinsic reward, while updating the other components using objectives in Eqs. and
The skill policy <a 8, trained with Soft Actor-Critic (SAC) (Haamoja et al. 2018) according to the obtained reward in Eq. [0 as an intrinsic reward. We train the other components with stochas- tic gradient descent. We summarize the training procedure of SUSD Algorithm Implementation details are provided in Appendix
Algorithm SUSD: Structured Unsupervised Skill Discovery
1: Initialize skill policy x(a 2, function {di(s*)) conditional density model go(s" | s),
Lagrange multiplier
2: for i <- to #epochs do
3:
for j + 1 to #episodes per epoch do
4:
Sample skill p(z) Sample trajectory t with r(a 8,
5:
6: end for
7: Fit conditional density model (s(s(ss) using current trajectory samples
8: Update (s*(*) with gradient ascent JSUSD,o, Eq.
9: Update À with gradient ascent on JSUSD,>, > Eq.
Update (a s, 2) using SAC with intrinsic reward RD Eq. 10
11: end for
5 EXPERIMENTS
evaluating SUSD, we begin by describing the experimental setup, including details of the en- vironments and baselines (Section [5.1] and then address three key questions: Q1: In factorized environments, do our discovered skills outperform other unsupervised reinforcement learning meth- ods downstream tasks? (Section 5.2) Q2: In unfactorized environments, does method remain competitive with alternative baselines? (Section 5.3 Q3: Does our method truly account for all
factors during the skill-learnigg phase? (Section 5.4
5.1 EXPERIMENTAL SETUP
We compare method against five wate-of-theeart unsupervised skill discovery approaches and evaluate these methods in five environments including factorized and unfactorized ones.
Baselines: We evaluate approach against three DSD based methods, namely LSD (Park al. 2022]. CSD (Park et al., (023), and METRA (Park et al.) 2024), as well as DIAYN (Eysenbach et al,
2019 which serve representative MI-based method and DUSDi (Hu al. 2024) as a factorized
bI-based method. Additional information about these methods have been presented in Appendix
More details about the hyperparameters are provided in Appendix
We evaluate our method on the HalfCheetah and Ant environment Tooorovv et al., Brockman et al., 2016) to demonstrate its applicability unfactorized settings. The 2D- Gunner is relatively simple domain, where a point agent can navigate inside a continuous 2D plane, collecting ammo and shooting targets. Multi-Particle based on (Lowe et al.] (017) and this modified version has been introduced in (Hu et al.) 2024). In this domain, a centralized controller simultaneously controls heterogenous point-mass agents to interact with 10 stations, where each agent can only interact with specific station. The Kitchen environment (Zhu et al. 2020; Wang et al. 2023) features robotic arm and multiple objects, in- cluding butter, meatball, stove, and button. In this environment, the agent must leam cookery skills, such as placing the butter or the meatball in the pot. An overview of all environments is shown in Figure 5) The majority of our evaluation focuses more complex environments, namely
Environments:
2012
multi-agent domain modified

---

Published as a conference paper at ICLR 2026
Method
CSD
DAYN
DUSDI
LSD
METRA
SUS0
Multiparticle Poidon Difficult
Multiparticle FoodéPoison Hard
Multiparticle Food&Poison Medium
25
20
2000
6000
0000
8000
10000
2000
0000
6000
2000
10000
2000
2000
0000
8000
10000
Faltdde
Eprcddc
Fpitades
Multiparticle Poisnn Easy
Multiparticle Sequential Easy
Multiparticle Sequential
20.
173
40-
3.-
10-
25-
0.2
Eplodds
Epitades
Eptecdes
Multiparticle Sequential Hard
Kitchen Butter in Pot
Kitchen Meatball Pot
2.5
20-
7.5
3.0
1.-
0.0
0.0
Figure 3: Training curves of SUSD and baseline methods on multiple downstream tasks in the Multi- Particle and Kitchen environments. Each plot shows the mean and standard deviation of returns over 3 random seeds.
Multi-Particle and Kitchen. Further details on the environments and downstream tasks are provided in Appendices B and C. respectively.
5.2 EVALUATING FACTORIZED ENVIRONMENTS
In this section, we evaluate the impact of SUSD on factorized environments (Q1) by assessing the performance of our method and the baselines across all downstream tasks in the MP and Kitchen en- vironments. We apply a simple factorization strategy where each entity is treated as distinct factor. Additionally, we can incorporate the agent's state into the entity factors to more accurately capture the causal influence of the agent on the entities in the latent space. Additional details on factoriza- tion and further experimental results are provided in Appendix H As shown Figure our method generally outperforms all compared methods by good margin, demonstrating its effectiveness and its ability to leverage the structure of the environment. Moreover, for the Kitchen as a more complex environment (with high-dimensional observation space), larger gap is observed. A detailed anal- ysis of the effects of distorted inductive biases in factorization is provided Appendix M For an ablation study demonstrating the impact of the curiosity- based weighting mechanism, please refer to Appendix G
5.3 UNFACTORIZED ENVIRONMENTS
To answer (Q2). we evaluate the performance of our method in the Ant and HalfCheetah envi- ronment (Toorov al. 2012 Brockman al., 2016). These environments ple, consisting single agent (c.g. the ant) that can move freely, without any explicit structure. We compare our method to other USD baselines zero-shot goal-reaching, described in ME-
relatively sim-
8

---

Published as a conference paper at ICLR 2026
TRA (Par et al. 2024) (this technique be for DSD-based methods). It evaluates the agent's ability to achieve goals without additional training. this task, the agent is allowed 20K steps to accumulate as much reward as possible. To reduce the impact of randomness, the experiment across eight different seeds. The results are shown in Figure 6
5.4 RICHNESS OF FACTORIZED LATENT SKILLS
We conduct two experiments to measure the richness of latent skill embedding (Q3), the first ex- periment, analyze the Multi-Particle environment and show that SUSD achieves relatively higher state coverage across different factors compared to other baselines. In the second experiment, we demonstrate that our method learns a richer latent skill embedding, capturing more comprehensive representation of factors and outperforming other prior DSD-based approaches.
5.4.1 STATE COVERAGE ACROSS FACTORS
We evaluate SUSD by randomly selecting a skill every 200 steps and collecting 20K rollout steps in the Multi-Particle environment. For each factor (agent), compute the number of unique states visits, rounding the agent's positions to determine distinct states. Specifically, following prior works (Park et al. 2022 2024 2023), use the agents' state coverage using their and y positions. To enable counting. we round each coordinate to two decimal places (e.g.. rounding (0.27392337, 00.40446577) to
In Multiple-Particle environment that contains 10 agents, we report the minimum state coverage across agent factors as Worst Agent State Cover- age, and the mean state coverage as Average Agent State Coverage. As shown in Figure SUSD achieves substantially better coverage than the baselines, particularly DUSDi. This trend is also observed in 2D-Gunner. This result highlights that SUSD not only learns diverse skills but also en- sures balanced exploration across all factors, including the weakest one. We further evaluate binning coverage in Appendix Additional comparisons with DUSDi, as another factorized method, are provided in Appendix H
londd Ageet Ttate Couerae (W.HH-Perrtee
Averaee Agent State Coveragee (Mui5 Dotttaer
Agant State Coverage 170 3D-Suteed
9350
3.50
00.00
1/504
50000
5.50
50/04
4000
D653
30000
2030
4.54
10205
20303
+200
0000
00000
15300
20203
0000
00000
00.03
(a) Worst Agent State Cov. MP
(b) Avg. Agent State Cov. MP
(c) Agent State Cov. in 2D-Gunner
Figure 4: Comparison of state/factor coverage across different factorized environments.
5.4.2 FACTOR DECODING
The objective of this experiment is to demonstrate that our method can learn latent skill embed- ding that captures comprehensive, compositional information about all factors. Specifically, map- ping observations into the latent skill space via the function Φ(.) should produce embeddings that encode meaningful information for each factor. By training a decoder on top of these embeddings to reconstruct the observations, we can evaluate reconstruction quality using factor-wise MSE. A high- quality latent skill embedding will yield low MSE for each factor, outperforming other baselines and indicating that the embedding effeetively captures all relevant information necessary to reconstruct the observations. The factor wise reconstruction errors from this experiment are reported in Table with details of the training process provided in Appendix

---

Published as a conference paper at ICLR 2026
Table I: Factor decoding errors across different factorized environments.
Method
SUSD
METRA
CSD
LSD
Env
Murti-Particll
0.060
0.147
0.313
0.308
Kitchen
0.014
0.028
0.049
0.038
2D-Gunner 0.080
0.186
0.404
0.224
5.5 EPFECTIVENESS OF LEARNED SKILLS DURING SKILL LEARNING
To evaluate the effectiveness of the learned skills during the skill-learning phase, we designed an experiment that highlights the strength of our method compared to baseline approaches. In the Kitchen environment, execute 10K rollout steps using the learned skill policy. Every 50 steps, we switch to a randomly selected skill, allowing us to test a wide range of skills. We repeat this experiment times obtain stable averages. During the rollouts, we measure how often the agent accidentally completes a task (i.e., receives the corresponding task reward) purely by executing these learned skills. The results are summarized Table 2
For example, on BiP task, our method achieves average reward of 39.875 across 8 runs, whereas none of the bascline methods managed to complete this task even once arbitrarily.
Table 2: Average of task rewards obtained through accidental task completions using only the learned skills during the skill-learning phase. Averages computed over independent runs.
SUSD
CSD
METRA
LSD
DUSDi
BiP (Butter in Pot)
39.875 + 18.452
0.0 0.0
0.0 0.0
0.0 0.0
0.0 0.0
MiP (Meatball in Pot)
58.875 1 784
0.0
0.0
0.0
2.5 1.14
PoS (Pot Stove)
20.5 1 17.965
0.0
0.0
0.0
1,275 0.954
PoT (Pot on Target)
13.75 1 6.923
0.0
0.0
0.0
0.0
CONCLUSION
Although excellent prior works in unsupervised skill discovery have successfully learned diverse behaviors without supervision, these methods often struggle in complex devironments-setiieee with multiple objects. To address this limitation, we introduced DSD-based method designed to handle factorized environments by leveraging the environment's structure as an inductive bias to learn diverse skills. We propose a factorized embedding architecture and allocate subset of skill variables to each controllable factor avoid underrepresention of harder-to-conter elements of the environment the skill latent space. Moreover, we reformulate the DSD optimization problem and integrate the concept of curiosity-base requiring more attention and adjusts the reward weights for each factor accordingly. We empirically demonstrate that SUSD enables agents to acquire diverse and dynamic skills in factorized environ- ments.
factor weighting, which dynamically identifies the factors
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: /0299, 2018.
Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint
arXiv:1803.08375. 2018.
OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Adrià Puigdomênech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International conference on machine learning, pp. 507-517. PMLR, 2020.
Serena Booth, W Bradley Knox, Julie Shah, Scott Nickum, Peter Stone, and Alessandro Allievi. The perils of trial-and-eroor reward design: misdesign through overfitting and invalid task spec- ifications. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp- 5920-5929, 2023.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint #rXiv:1606.01500 2016.
Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills, Interna- tional conference machine learning, pp. 1317-1327. PMLR, 2020.
Jongwook Choi, Sungtae Lee, Xinyu Wang, Sungryull Sohn, and Honglak Lee. Unsupervised object interaction learning with counterfactual dynamics models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11570-11575 2024.
Caleb Chuck, Kevin Black, Aditya Arjun, Yuke Zhu, and Scott Niekum. Granger-causal hierarchical skill discovery. arXiv e-prints, pp. arXiv-2306, 2023.
Caleb Chuck, Fan Feng, Carl Qi, Chang Shi, Siddhant Agarwal, Amy Zhang, and Scott Niekum. Null counterfactual factor interactions for coal-conditioned reinforcement learning. In Interna- fional Conference on Learning Representations, 2025.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. In International Conference on Machine Learning, 2019.
Shixiang Gu, Ethan Holly, Timothy P Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation. arXiv preprint 16 1(1), 2016.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. In International conference on machine learning. 2018.
Jiaheng Hu, Zizhao Wang, Peter Stone, and Roberto Martin-Martin. Disentangled unsupervised skill discovery for efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 37:76529-7653 2024.
Plerre-Alexandre Kamienny, Jean Tarbouriech, Sylvain Lamprier, Alessandro Lazaric, and Ludovic Denoyer. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. In International Conference on Machine Learning, 2022.
Michael Kearns and Daphne Koller. Efficient reinforcement learning in factored mdps. In IJCAI, volume 16, pp. 740-747, 1999.
Hyunseung Kim, Byung Kun Lee, Hojoon Lee, Dongyoon Hwang, Sejik Park, Kyushik Min, and Jaegul Choo. Learning to discover skills through guidance. Advances Neural Information Processing Systems, 36:28226-2877 2023.

---

Published as a conference paper at ICLR 2026
Hyunseung Kim, BYUNG KUN LEE, Hojoon Lee, Dongyoon Hwang, Donghu Kim, and Jaegul Choo. Do's and don'ts Learning desirable skills with instruction videos. Advances in Neural Information Processing Systems, 37:47741-47766 2024.
Diederik Kinga, Jimmy Ba Adam, et al. A method for stochastic optimization. In International conference on learning representations (ICLR), volume 5. California;, 2015.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, MA NYU, Aravind Rajeswaran, and Pieter Abbeel. Contrastive intrinsic control for unsupervised reinforcement learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 34478-34491, 2022.
Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pp. 6736667677700 6747. PMLR, 2021.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- environments. Neural Information Processing Systems
critic for mixed (NIPS), 2017.
Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, and Sai Rajeswar. Choreog- rapher: Learning and adapting skills in imagination. In International Conference on Learning Representations, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Aditya Mohan, Amy Zhang, and Marius Lindauer. Structure in deep reinforcement learning: A survey and open problems. Journal of Artificial Intelligence Research, 79:1 19:316,,, 2024.
Ian Osband and Benjamin Van Roy. Near-optimal reinforcement learning in factored mdps. Ad- vances in Neural Information Processing Systems, 27, 2014.
Scohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. In International Conference on Learning Represen- tations, 2022.
Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllabiliy--aware unsuper- vised skill discovery. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 27225-27245. PMLR, 23-29 Jul 2023.
Scohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric-aware abstraction. In International Conference on Learning Representations, 2024.
Silviu Pitis, Elliot Creager, and Animesh Garg. Counterfactual data augmentation using locally factored dynamics. Advances Neural Information Processing Systems, 33:3976-3990, 2020.
Seungeun Rho, Laura Smith, Tianyu Li, Sergey Levine, Xue Bin Peng, and Sehoon Ha. Language guided skill discovery. In International Conference Learning Representations, 2025.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-699 2020.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-awee unsupervised discovery of skills. In International Conference on Machine Learning. 2019.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self- self-play. Science, 362(6419): 140- 1144, 2018.
12

---

Published as a conference paper at ICLR 2026
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martín-Martín, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 28694-28698, 2025.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-500 5033. IEEE, 2012.
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: standard interface for reinforcement learning environments. arXiv preprint ar 2077 2024.
Zizhao Wang. Xuesu Xiao, Zifan Xu, Yuke Zhu, and Peter Stone. Causal dynamics learning for task- independent state abstraction. In Proceedings of the 40th International Conference on Machine Learning. 2022.
Zizhao Wang, Jiaheng Hu, Peter Stone, and Roberto Martin-Martín. Elden: Exploration via local dependencies. Advances in Neural Information Processing Systems, 111155566-54744, 2023.
Zizhao Wang, Jiaheng Hu, Caleb Chuck, Stephen Chen, Roberto Martin-Martin, Amy Zhang, Scott Niekum, and Peter Stone. Skild: Unsupervised skill discovery guided by factor interactions. Advances in Neural Information Processing Systems, 37:77748-77776 2024.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martín, Abhishek Joshi, Soroush Nasiri- any, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiy:2009. 2293, 2020.
13

---

Published as a conference paper at ICLR 2026
A
PROOF OF LEMMA 4.1
We first start with Eq.
T-1 [ o(ss)) s.t. [lo(s) - 2(s') I d(s, V(s,s') Sag.. (11)
sup Bp(T.))
t=
Let the scaled state functiin be defined as é(s) == 2(8,8))
d(s)
Then, we can transform the constraint
term in Eq. 11 as follows (since d(s, s') 2 0):
sup E (p(,a))
Σ (0(s)+1) -(((4))))((s)))
s.t. [[((s) o(5')l12 S 1, v(s,8') Sadj.
(12)
5.0
By replacing o(s) with ((s) -d(s, in Eq. 12 we obtain:
T-1
sup E p(T,2) 8
Σ
d(sr. - b(s)))
s.t. 11ō(s) - ~(s'))la s V(s,s') Sadj
(13)
B ENVIRONMENT DETAILS
B. ANT
As shown in Figure a), the Ant (Todorov et al. 2012 Brockman et al. 2016) environment has an episode length of 200 steps. The observation space consists of single factor representing the state of the Ant that 29-dimensional. The action space is continuous, corresponding to the control of the Ant's joints, and has 8 dimensions.
B.2 HALFCHEETAH
As shown in Figure bb), the HalfCheetah (Todorov et al. 2012; Brockman et al. 2016) environment has episode length of 200 steps. The observation space consists of single factor representing the state of the cheetah and is 18-dimensional. The action space continuous, corresponding to the control of the cheetah's joints, and has dimensions.
B.3 GUNNER
As shown in Figure Cic), in the 2D-Gunner (Hu et al. 2024 environment, the blue star marks the position of the agent, the blue line marks its shooting direction, the red diamond marks ammo location, and the orange cross marks the target position. The agent has 18-dimensional observation space, consisting of state factors: Agent Position, Ammo State, Target State. The action is 6- dimensional, for agent movement, for ammo pickup, and for shooting direction.
B.4 KITCHEN
As shown in Figure d), the Kitchen (Hu et al. 2024 Zhu al. 2020 environment contains robot arm, a piece of butter, a meatball, a pot, a stove with its switch, and a target location marked in red. The agent operates in 4-dimensional action space, while the observation space is 142- 42-dimensional. This observation can be decomposed into seven components: 33 dimensions for the arm, 22 for the pot, 18 for the meatball, 19 for the button, 22 for the stove, and 14 for the target.
14

---

Published as a conference paper at ICLR 2026
Ant
b) HalfCheetah
c) 2D Gunner
Kitchen
e) Multi-Particle
Figure 5: Benchmark Environments
B.5 MULTI-PARTICLE
As shown in Figure 5(e), agents represented by small circles, while stations represented by large circles. Agents can only interact with stations of the same color. 'he Multi-Partie (Lowe et al. 2017) environment has 70-dimensional observation space, composed of 10 state factors that capture the states of each agent and its corresponding landmark. The action space is 50-dimensional, with dimensions per agent controlling their movements and interactions with the landmarks.
C DOWNSTREAM TASKS
C.I ANT
The task requires the agent to reach four target goals, each within a radius of 3. Each goal is randomly selected from the region [s- - 7.5, 82 + 7.5] x [s, 7.5, Sy 7.5), where (S2, sy) denotes the agent's current position in the z-y plane. The agent is awarded 2.5 upon reaching a goal. A new goal is generated either when the current goal is reached or if the agent fails reach within 50 steps.
Multi-goal Ant (Park et al. 2024):
C.2 HALFCHERTAH
HallCheetahGoal Park et al. 2024 The task is to reach a target goal (within radius of 3) that is randomly sampled from 10, I0J. The agent receives reward of 10 upon reaching the goal.
C.3 GUNNER
Unlimited Ammo (unlim) (Hu et al. 2024): In this downstream task, targets appear at random lo- cations, and the agent must approach and shoot each target score. Since ammunition unlimited, the agent does not need to collect any.
Limited Ammo (lim) (Hu al. 2024): This downstream task differs from the "Unlimited Ammo" task in that the agent begins without ammunition and must collect it before shooting, while all other aspects remain unchanged.
C.4 KITCHEN
Put Butter in the Pot (BiP): In this downstream task, the agent's goal is to place the butter in the pot and keep it there. receives a reward for each step during which the task successfully maintained.
Put Meatball the Pot (MiP): In this downstream task, the agent's 's goal is to place the meatball in the pot and keep there. It receives reward of for each step the task is successfully maintained.
Put Pot on the Stove (PoS): In this downstream task, the agent's goal is to place the pot on the stove and keep it there. It receives reward of for each step the pot remains the stove.
Put Pot on the Target (PoT): In this downstream task, the agent's 'ss goal is to place the pot on a target position and keep it there. It receives reward of I for each step the pot remains the target.
15

---

Published as a conference paper at ICLR 2026
(a) Zero-shot Goal Reaching for Ant
(b) Zero-shot Goal Reaching for HalfCheetah
Figure 6: Zero-shot goal reaching performance of policies learned by skill discovery methods across 8 random seeds. SUSD achieves competitive results in both unfactorized environments.
C.5 PARTICLE (MP)
Sequential interaction (seq) (easy, medium, hard) (Hu et al., 2024): In this task, agents are re- quired to interact with their assigned stations following the sequence given by an instruction the start of each episode. Interacting with stations out of sequence incurs penalty. In the easy version, the sequence has length 2; in the medium version, length 3; and in the hard version, length
Food-poison (fp) (easy, medium, hard, difficult) (Hu al. 2024): In this downstream task, each station delivers either food or poison its corresponding agent. Agents must decide whether to interact with their station based on sequence of binary indicators provided at the beginning of each episode. The casy version uses a sequence of length 2, the medium version length 5, the hard version length 8, and the difficult version length 10.
D COMPARED METHODS
LSD (Park al. 2022). CSD Park et al. 2023), and METRA (Park et al., 2024) are the compared DSD-based methods. LSD measures the distance between states using the & norm, defined as d(s', ||s s||- CSD defines distance in terms of transition probabilities, d(s', p(s'ls). METRA employs temporal distance, given by d(s', s) = 1 for the adjacent states the trajectories. In contrast, DIAYN trains discriminator predict the corresponding latent variable from given state. Among recent skill learning methods, DUSDi (Hu et al.) 2024) leverages the compositional structure of environments by incorporating mutual information (MI) between skill components and state factors into its objective. Specifically. the agent is rewarded for maximizing the MI between each state factor and its corresponding skill component, while minimizing the MI between that component and all other state factors. Although SkiLD (Wang et al.) (024)) also adopts environment factorization, is constrained to environments with small state space and limited number of actions. As result, we were unable to evaluate it the environments introduced in this work.
E IMPLEMENTATION DETAILS
Dimension of the latent space. For unfactorized environments (i.e., Ant and HalfCheetah), we set the latent skill dimension D 2 for all evaluated methods. factorized environments, we use the hyperparameter settings for DUSDi as reported in its paper (Hu et al. 2024). For SUSD, the number of factors set equal to the number of entities, and we also use skill dimension of 2 for each factor in all environments. Therefore,
set N 3 in 2D-Gunner (for agent, amno, and target), N = Kitchen (for arm, butter, meatball, button, stove, pot and target entities), and Multi-partch (for 10 agents and 10 stations). When grouping agent and station in this environment, we consider = 10 factors. Appendix show that higher dimensions of latent space generally do Multi-Particle environment.
20 for
not improve the results of baseline methods (i.e., CSD, METRA)
High-level controllers for downstream tasks. In Figure I we evaluate the learned skills on down- stream tasks by training high-level controller n (z s, Smoo), which selects skill every K
16

---

Published as a conference paper at ICLR 2026
Table Hyperparameters for unsupervised skill discovery methods.
Hyperparameter
Value
Learning rate
0.0001
Optimizer
Adam
episodes per epoch # gradient steps per epoch
8
50
Minibatch size
256
Discount factor 7
0.99
Replay buffer size
106
# hidden layers
2
# hidden units per layer
1024
Target network smoothing coefficient
0.995
Entropy coefficient
0.1 (Adaptive)
SUSD C
10-6
SUSD initial À for each factor
3000
# Number of factors N
10 (MP), (Kitchen), 3 (Gunner), (Ant, HalfCheetah)
# Dimensions of each factor in Z
2
Table 4: Hyperparameters for downstream policies.
Hyperparameter
Value
# training epochs
104
# episodes per epoch
1
gradient steps per epoch
50
# skill sample frequency R
10
Replay buffer size
106
Target network smoothing coefficient
0.995
Entropy coefficient
0.1 (Adaptive)
Skill range
[.5. 1.5
environment steps for both MP and Kithchen. At each selection step, the high-level policy chooses a skill z, and the pre-trained low-level skill policy n' (a s. 2) executes this skill for the next K steps. High-level controllers trained using SAC (Haarnoja et al., 2018) for continuous skills, with hyperparameters identical to those used in the unsupervised skill discovery methods (Table
Zero-shot goal-conditioned RL. In Figure we evaluate the zero-shot performance of our method against other DSD baselines on goal-conditioned downstream tasks in the Ant and HalfCheetah environments. The skill vector is recomputed at every step.
We present the full list of hyperparameters used for skill discovery methods in Table
F EXTENSION TO DISCRETE SKILL SPACE
For discrete skills, we construct the skill space as == := 21 x of factors and each 2 is D-dimensional one-hot vector, 2 [0, Although we concatenate this skill vector with the observation feed into the skill policy, using one-hot representation for each factor can lead skill learning collapse. Details on this collapse are discussed in prior DSD- based methods (Park et al.) 2022 2024). To prevent this collapse, we compute the intrinsic reward such that the sum over each factor in the skill vector is mero-mean. This is done using the formula in Eq. 14. Assume that the k-th dimension of factor Z is 1:
x N where N the number
(o(s
((s))
1
Σ JEIL.2....DDVE
(p(s).)) ((s))s
(14)
N
To evaluate the effectiveness of our discrete approach, we implement the discrete skill space in the 2D>Gunnner environment and compare its performance side by side with the continuous skill space,
17

---

Published as a conference paper at ICLR 2026
SUSO Continuous
5L.50 Disorete
Gunner with Limitation
Gunner without Limitation
03
60-
40
0
1000
4333
9000
0000
10000
2000
4302
0003
5000
1000
Figure 7: Comparison of discrete and continuous skill spaces in the 2D-Gunner environment.
as shown in Figure 7
As shown, the two methods achieve very similar performance on downstream tasks, indicating that our discrete skill formulation is competitive with the continuous one.
G ABLATION STUDY
In this study, we evaluate the individual contributions of the curiosity-based weighting and factor- ization modules by selectively removing them. This allows us to quantify the performance drop when only factorization is applied and when both factorization and curiosity- weighting are ablated. Specifically, SUSD-w refers to our method without the curiosity-based weighting module, while SUSD>-f refers to our method without both the curiosity-basee weighting and factorized em- bedding. As shown in Figure 8) the performance of these variants decreases compared to our full method which incorporates both modules. This demonstrates that both factorization and curiosity- based weighting are essential for achieving strong results.
H FACTORIZATION AND FURTHER EXPERIMENTS
DUSDi has carefully engineered factors of the state mentioned in (Hu al. 2024). Furthermore, by leveraging knowledge of downstream tasks, the observation space is filtered to focus exclusively on discovering skills that are directly relevant for solving these tasks through HRL. contrast, adopt a simpler approach by assigning attributes related to different environmental entities to distinct factors, with the agent factor that can be concatenated other factors facilitate the discovery of interactions more easily. In the Multi-Particle environment with 20 objects (agents and stations), we group each agent with its corresponding station as single factor leads better performance compared to a factorization treating each object (i.e., agent station) separate factor. Figure 9 compares these two factorizations, showing that incorporating prior knowledge (of requiring each agent interact with its own station) to align factorization with downstream task structure improves results.
I IMPACT OF INCREASED SKILL DIMENSIONALITY ON BASELINE PERFORMANCE
Previous USD methods do not exhibit significant performance gains merely by increasing the di- mensionality of the skill space. In Figure 10. we show that even when the skill dimensionality of METRA and CSD is increased from 2 to 20, their performance still does not match ours. This highlights a sigmificant gap and demonstrates that leveraging the environment's 's factorized inductive bias, together with suriosity-weighte approach, can achieve better performance than merely in- creasing skill dimensionality. It is worth noting that we use D 2 for all factors across different environments.
J TRAINING DETAILS FOR FACTOR DECODING
For decoding, we use an MLP having one-hidden layer (with ReLU (Agarap) 2018 activation function) optimized with Adam (Kinga al. 2015) and squared (MSE) loss for 100
18

---

Published as a conference paper at ICLR 2026
5:50
Multiparticle Food&Polson Difficult
Multiparticle Food&Polson Hard
Multiparticle Food&Polson Medium
2000
1002
2002
4000
12300
2000
+000
6.00
10690
Multiparticle Food&Poison Easy
Multiparticle Sequential Easy
Multiparticle Sequential Medium
15.0
2.5
50-
Batorto
Egrudho
Multiparticle Sequential Hard
Gunner with Limitation
without Limitation
330
20.0
2%
5.-
2.5
60-
3.5
Ephocee
Lpitades
Splodds
2.5
2.5
1.5
1200
Figure 8: Effect of the curiosity-based weighting module
epochs with learning rate 0.0001 and batch size 1024. Training data is collected from 100K roll- out steps. Every 200 steps. we sample a random skill and store the corresponding (state, skill) pair at each step. We use 80% of the collected data for training and 10% for evaluation and 10% for test. The decoder consists of a hidden layer. We use cross-validation to determine the opti- mal hidden size for each method. For the Multi-Particle environment, candidate hidden sizes (30, 35, 40, 45, 50, 55, 60, 65). For 2D-Gunner, the candidate values are {10, 12, 14, 16} and for the Kitchen environment are {20, 30, 40, 50, 60, 70, 80, 90).
K BINNING COVERAGE
In this experiment, we aim to evaluate how well our learned skills can cover the state space. To do this, we first divide the r and axes into b bins each, forming a b x b grid. We then perform rollouts for a specified number of steps and compute the percentage of grid cells visited by the policy. We conduct this experiment in both the 2D-Gunner and Multi-Particle environments. In the 2D-Gunner environment, we measure bin coverage by discretizing the agent's positions into a 50 x 50 grid, resulting in 2500 cells. In the Multi-Particle environment, use the same setting as outlined above. For each agent, we compute the bin coverage and report both the minimum and average coverage across all agents. The results for this experiment presented in Figure 11
19

---

Published as a conference paper at ICLR 2026
SUSD IN-10)
SUSD IN-20)
Difficult
Food&Poison
Food&Poison Easy
760
31.5
15.0
12.5
10.0
2000 4000
6000
30000
2000
4000
6000
8000
11900
4000
8000
32800
2000
4000
0000
2000
30000
Epiodds
5s5odss
59300e
Sequential Easy
Sequential
Sequential
15.0
30
10.0
1.5
20
2.5
10-
00
2000
6000
20002
2000
4000
6000
8000
12500
4000
8000
32000
Figure 9: Factorization in Multi-Particle: grouping each agent with its station outperforms treating objects independently.
Multiparticle Sequential (D=2)
(0-20)
METRA
METRA
Muitiparticie Sequential Medium
Multiparticle Sequential Hard
15.0
3.5
10-
0.0
-2.5
-5
Splocee
Eshodds
oodes
Figure 10: Impact of increasing skill dimensionality on baseline USD methods. Even when METRA and CSD increase their skill dimensionality from 2 to 20, their performance remains below SUSD.
Canrage Bin Coverage Pwrtie 2300 Sims.
it Coveraon (Mu,t1 Perti
Coverage 20 Guroer Rin
5U00
CSC
X.20
6.4
20000
40009
400.0
0000
20000
2030
40.30
5000
8000
10000
20300
40/.0
0000
60000
13000
(a) Average Bin Coverage in MP
(b) Minimum Bin Coverage in MP
(c) Bin coverage in 2D-Gunner
Figure 11: Bin coverage comparison. (a) Average bin coverage across factors in MP. (b) Minimum bin coverage across factors in MP. (c) Bin coverage in Gunner environment for 2500 bins.
L ADDITIONAL EXPERIMENTS IN THE KITCHEN ENVIRONMENT
We introduce two additional tasks, PoS and PoT, whose definitions are provided in Appendix C These tasks are added to the Kitchen environment to further evaluate the effectiveness of our method in 3D setting. The performance of our method compared to the baselines on these tasks is shown in Figure 12
20

---

Published as a conference paper at ICLR 2026
Kitchen
Meatball
1.2
00
Figure 12: Comparison of SUSD and baseline methods on the PoS and PoT downstream tasks.
M ANALYZING OVERFACTORIIZATION AND UNDERFACTORIZATION EFFECTS
In environments containing multiple objects, our approach naturally enables the model to encode object-specific information in separate components. fnder-factorizztton reduces the model to conventional DSD-based formulation that lacks compositional inductive bias and instead learns a single embedding over the entire state vector. Conversely, vvrrffcctorization (e.g., decomposing the state into excessively fine-grained components) can introduce misaligned inductive biases and consequently degrade model performance. This occurs because the reward can only incorporate linear combinations of the individual factor embeddings, as Σ; (0, 10---al ())(s))²²z= 2", whereas embedding dimensions that capture interaction of factors (e.g., ó(s s²)) may be essential for ef- fectively learning skill policy. To assess the impact of factorization performance, we conducted some experiments on the 2D-Gunner environment.
the Gunner environment, the true factors are Agent, Ammo, and Target. Over-factorizing the environment into four factors, by artificially splitting the Agent factor into two random sub-factors, leads to a decline in performance. We also examine under- fattoizaatoi by reducing the model two factors, merging the Agent and Ammo factors into a single one. As shown in Figure [3 performance highest when the true factorization structure used, and degrades more less factors are imposed.
SUS0 Foctors)
SUS0 Nactors)
SUSO Pactors) Gunner Limitation
20
2005
02:00
10000
20:0
6.0
6003
120
198C
Figure 13: Effect of overfluderrfacctorizaiiin on performance in the 2D-Gunner environment.
N THE USE OF LARGE LANGUAGE MODELS (LLMs)
We used Large Language Models (LLMs) only in a limited way, specifically for minor writing polish and phrasing suggestions.
21


Reviews:
[{"review_id": "ISjcDPVbmI", "reviewer": "Reviewer_8o7W", "summary": "This paper focuses on unsupervised skill discovery and aims to improve distance-maximizing skill discovery performance in environments in multiple controllable objects. To this end, it assumes the state factorization is given and incoroporates the factorization into METRA objective. The proposed method further weights each factorization's contribution to the intrinsic reward based on its prediction accuracy.", "questions": "Different state factors can have different scales and thus different $-\\log(s^i_{t+1}|s^i_t)$ scales, how do you account for this scale differences when weighting the intrinisc reward?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and easy to follow, especially in the method description and the experiment results.\n\nThe proposed method of improving METRA's controllbility over multiple objects, to the best of my knowledge, is novel in the field of unsupervised skill discovery.\n\nThe ablations and extra experiment results in the appendix are throughout and provides good insight of each component's importance.", "weaknesses": "My major question is the empirical evaluation for the motivation of the paper -- learning skills that engage all controllable factors in the environment.\nSpecifically, in Sec 5.4.1, since the state space is continuous, the # of unique states is infinite, so the policy may still visit many unique states while only covering a small portion of the state space. It's more appropriate to discretize the state space into bins and see the % of bins that are covered by the policy. I notice the authors mentioned that they \"round\" the state, kindly correct me if you are already doing that (would be helpful to explain how exactly the rounding is conducted). In addition, visualizing the each factor's coverage is another valid way to show better controllability.\n\nMinor:\nmissing space on Line 478.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x3frMzjchx", "reviewer": "Reviewer_ThUn", "summary": "This paper introduces SUSD, a new framework for unsupervised skill discovery (USD) that aims to exploit the compositional structure of environments. It builds upon Distance-Maximizing Skill Discovery (DSD) methods like METRA, introducing (1) state space factorization to structure the skill space (similar to DUSDi) and (2) a curiosity-based factor weighting mechanism that prioritizes underexplored factors (similar to CSD). The paper evaluates SUSD across five environments (Ant, HalfCheetah, 2D-Gunner, Kitchen, and Multi-Particle), showing improvements over recent USD baselines, including DIAYN, LSD, CSD, METRA, and DUSDi.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposed a method with strong empirical results, combining the strengths of skill disentanglement (e.g. DUSDi) and Distance-Maximizing Skill Discovery (e.g. METRA). Ablation studies and factorization sensitivity provide useful insight into component contributions.\n\nThe factorized embedding formulation is clean and intuitively appealing; it aligns well with factored MDP structure.\n\nThe curiosity-based factor weighting is a natural and well-motivated extension to encourage balanced skill learning.\n\nThe combination of DUSDi, CSD and METRA is new.\n\nThe paper is well-written, where the presentation is clear and easy to understand", "weaknesses": "My main concern with this paper is its novelty and conceptual contribution: while the idea of combining state factorization with distance-based skill discovery is sensible, it is not clear how much SUSD goes beyond a straightforward integration of DUSDi, CSD and METRA: \n- The factorized skill structure is conceptually almost identical to DUSDi, except that it is applied within a DSD objective rather than a mutual-information one.\n- The curiosity-based weighting resembles the controllability weighting from CSD and METRA, but applied per-factor.\n\nAdditionally, while the proposed factorization makes intuitive sense in multi-entity environments, the paper doesn’t clearly characterize when this inductive bias is beneficial.\nFor instance, is the method sensitive to over- or under-factorization? Does performance degrade if the factor decomposition is slightly misaligned with the true controllable entities? A brief sensitivity analysis or discussion would make the claims more general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYMFZPprpl", "reviewer": "Reviewer_jqsH", "summary": "Introduces factorization to distance maximizing skill discovery from mutual-information skill discovery by giving each factor a different skill factor, and then introducing a balancing strategy based on the learning level of the skills for that factor. This is done by working in the FMDP framework and adding a curiosity bonus. Illustrates results on a few factored environments, including 2d modified gridworld and 2d object manipulation.", "questions": "Skill discovery, even in its modern form, is not introduced by the MISL framework but has been part of hierarchical reinforcement learning for a long time, so the choice of citations is quite narrow for simply calling it \"Unsupervised Skill Discovery.\" The references focus on mutual information based skills.\n\nAnother important limitation of using state-space distances is that these distances can often be either hard to learn, such as in the case of temporal distances, or not useful, such as in the case when the state spaces is images. This seems like a more significant issue than simple controllable factors, which is a challenge for both DSD and MISD methods. Thus the motivation does not entirely make sense.\n\nIt is probably worth highlighting early in the work that the primary contribution of this work is to apply concepts from existing MISL methods, such as DUSDi or SkiLD, to DSD methods, as factorization and load balancing have been previously addressed.\n\nBy assigning each factor a skill parameter, even those that are not controllable, if there is a significant amount of background noise, this could result in poor performance, since the changes from these uncontrollable factors would wash out the reward from the controllable ones.\nIt seems like curiosity-based factor weighting could result in poor performance in cases where there are factors that change constantly, but are not easy to control. Thus, this relies on a kind of quasistatic assumption where hard-to-control factors are generally stationary, so that getting reward from them will generally be constant until the agent learns to manipulate that factor. This is a challenge with all curiosity-based methods, but moreso in a case where the agent needs to manipulate multiple factors at once. \n\nBy giving each factor one skill, this seems like it could fail to load balance factors that require more fine-grained control, and those which require less fine grained control. \n\nThis method assumes that all factors are along a flat hierarchy of control: skills learned on one factor are not useful for downstream factors, but this is often not the case (the easy-to-control factors are often useful for downstream factors), and it is not clear if there is a solution for this. \n\nIt is not clear why SUSD should outperform any of the methods in the Ant or Halfcheetah environments, yet it appears to do so in Ant. This suggest perhaps a lack of variation or failure to sufficiently optimize baselines. \n\nFor factor decoding, doesn't SUSD learn a separate feature map for each factor using ground truth information? Especially since in the FMDP setting the factors are already given. So isn't SUSD given privileged information? A fairer comparison might be a comparison against representations from DUSDi, but that would require significant changes to that algorithm. Thus, it seems unneessary to have this section.\n\nConsidering factored MISD methods like SkiLD have been applied to 3D environments like iGibson, it would be good to have a more comprehensive evaluation in a more complex domain to demonstrate that DSD improves performance even in 3D.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Adds a novel combination of techniques (factored learning to distance-based skill discovery).\n\nProvides adequate empirical evidence to support the method\n\nhandles the load-balancing question for factors in a sufficient way.", "weaknesses": "Provides marginal change from existing methods in the unsupervised skill discovery space, since both factorization and skill learning have been tried before.\n\nDoes not provide clear ablations for which change produces the gain in improvement.\n\nProvides a somewhat limited view of skill learning, focused only on unsupervised skill discovery.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qsaeDtkfeq", "reviewer": "Reviewer_1HYa", "summary": "This paper has proposed a new unsupervised skill discovery method. Different from the previous mutual information-based and distance-based skill discovery methods, the proposed approach learn skills through state factorization. The proposed approach is evaluated in the Mujoco environment and the MPE environment.", "questions": "Please see the weaknesses part.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThis paper is well written, which clearly conveys the importance of the unsupervised skill learning problem.\n\n2.\tThe proposed approach is extensively evaluated in various domains, such as Mujoco and MPE.", "weaknesses": "1.\tIt is confusing what aspect the state factorization based on, which is related to the definition of skills. As stated in Section 4.2, the skills are learned with the curiosity-based rewards. However, curiosity can vanish with learning, so will the skills collapse to single action? How to prevent the collapse of the skills?\n\n2.\tThe experiment results demonstrate returns in the downstream tasks. However, as this paper aims at unsupervised skill learning, it is more important to show the comparison of the learned skills among the baselines. Can the skills learned in the Kitchen environment manipulate the task-related objects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)