Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learning
Authors: Patrik Reizinger
Bálint Mucsányi
Siyuan Guo
Benjamin Eysenbach
Bernhard Schölkopf
Wieland Brendel
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
representation learning
identifiability
ICA
exploration
unsupervised skill discovery
Primary area: reinforcement learning
TL;DR: Mutual Information Skill Learning (MISL) methods learn identifiable features in reinforcement learning via diverse policies and inner product feature parametrizations

Abstract: Self-supervised feature learning and pretraining methods in reinforcement learning (RL) often rely on information-theoretic principles, termed mutual information skill learning (MISL). These methods aim to learn a representation of the environment while also incentivizing exploration thereof. However, the role of the representation and mutual information parametrization in MISL is not yet well understood theoretically. Our work investigates MISL through the lens of identifiable representation learning by focusing on the Contrastive Successor Features (CSF) method. We prove that CSF can provably recover the environment's ground-truth features up to a linear transformation due to the inner product parametrization of the features and skill diversity in a discriminative sense. This first identifiability guarantee for representation learning in RL also helps explain the implications of different mutual information objectives and the downsides of entropy regularizers. We empirically validate our claims in MuJoCo and DeepMind Control, and show that CSF provably recovers the ground-truth features from both states and pixels. Our code is available at https://github.com/bmucsanyi/identifiable-misl.

Published as a conference paper at ICLR 2026
SKILL LEARNING VIA POLICY DIVERSITY YIELDS IDENTIFIABLE REPRESENTATIONS FOR REINFORCE- MENT LEARNING
Patrik Reizinger" Bálinn Mucsányi**4 Siyuan Guo Benjamin Eysenbach Bernhard Schölkopf" 1.2 and Wieland Brendel 1.2
Max Planck Institute for Intelligent Systems, Tübingen, Germany
ELLIS Institute Tübingen, Tübingen, Germany University of Tübingen, Tübingen, Germany 'Tübingen AI Center, Tübingen, Germany University of Cambridge, Cambridge, United Kingdom Princeton University, Department of Computer Science, Princeton, United States
ABSTRACT
Self-supervised feature learning and pretraining methods in reinforcement learning (RL) often rely on information-thhorrttic principles, termed mutual information skill learning (MISL). These methods aim to learn a representation of the environment while also incentivizing exploration thereof. However, the role of the representation and mutual information parametrization in MISI is not yet well understood theoret- ically. Our work investigates MISL through the lens of identifiable representation learning by focusing on the Contrastive Successor Features (CSF) method. We prove that CSF can provably recover the environment's ' ground-trut trutt features up to a linear transformation due the inner product parametrization of the features and skill diversity in a discriminative sense. This first identifiability guarantee for representation learning in RL also helps explain the implications of different mutual information objectives and the downsides of entropy regularizers. We empirically validate claims in MuloCo and DeepMind Control, and show that CSF provably recovers the ground-truth -trtt features from both states and pixels, Our code is available at httttps://gii hub. com/ httm//uussany/iideeo ifiable misl
INTRODUCTION
The field of Reinforcement Learning (RL) faces several challenges, such as learning under sparse rewards, exploring the environment, and designing an appropriate reward function. Many solutions use different self-supervised approaches including curiosity, intrinsic motivation, or unsupervised skill discovery (USD) (Eysenbach et al. (2019; Sharma et al. 2020 Pathak al. 2017 Pathak et al.) Sancaktar et al. 2023 Ha & Schmidhuber 2018). mutual information skill learning (MISL) methods a subclass of USD that use Mutual Information (MI) objectives. Despite this common design factor, they exhibit wildly varying performance (Park et al.) 2024b ZZeng et al., (025). which is not yet well understood theoretically. Empirical evidence suggests guidelines on, e.g., parametrizing the action-value function (Zheng et al., 2025 Liu et al. [002)) but it does not explain how similar principles lead such large differences performance.
We build upon recent theoretical advancements in self-supervised learning (SSL), particularly nonlin- ear Independent Component Analysis (ICA) theory (Zimmermann al., 2021 Hyvarinen et al. 2019
Reizinger et al. [2024a] Roeder et al. (020) and Causal Representation Learning (CRL) (Schölkopf et al. [021]] Wendong et al.] Reizinger et al., 2024b; Rajendran al. 2023 Guo et al. 2023). Identifiability results derive guarantees on learning the latent factors from data, i.e., recovering the underlying data generating process (DGP) from high-level observations such as pixels. Identifiability results are relevant for RL in the partially observable Markov Decision Processes (POMDPs), as the states are not always observed. Thus, the goal is to provably infer the ground-truth states from
"quual contribution. Correspondence to wepp::ik...iiiinggrttuee.....e..
I Equal supervision.

---

Published as a conference paper at ICLR 2026
reward: max rz (2((()),
(o(o), o(o')) 7
skills p(=)
policy T(a (()), 2]
environment p(a s,
enender Φ(o), o(o')
critic
(< (()))
The building blocks of mutual information skill learning (MISL) Zheng et al. 2025): The MISL method Contrastive Successor Features (CSF) (Zheng et al. 2025) uses uniformly drawn skills on the hypersphere to learn skill-conditioned policy by maximizing skill diversity via Eq. 33. CSF learns an encoder & to map observations o to features ((0), and uses an inner product parametrization in the critic (o(o,, ((o')), where q is trained via a contrastive loss (I) to infer the skill 2, from features of consecutive states. Intuitively, this is only possible if each skill representative of only a subset of states. The skill distribution and the environment dynamics are fixed, whereas the policy and the encoder are learned, for details, refer to $ $ Main result: We formalize the underlying assumptions of CSF with a probabilistic DGP on the hypersphere, then, building prior works from identifiability, prove that the building blocks of CSF are such that after training, the learned features identify the ground-trutt states up to a linear transformation, which explains practical differences between some objectives and parametrizations 4.2)
Figure 1:
observations. Intuitively, to achieve identifiability, the training samples should cover all aspects of the underlying world model. That is, they require diversity (also called sufficient variability), mirroring the goal of USD) to explore and learn diverse skills. In the same way that identifiability theory helps explain the success of self-supervised pretraining in computer vision (Zimmermann et al. 2021 Rusak et al. 2025 Ibrahim et al., 2024 Reizinger et al. (024a), we use recent insights
from identifiability study self superviee RL. methods. Using Zheng al. (2025) as prototypical
MISL method, we provide theory to elucidate why and when MISL works by proving that Contrastive
Successor Features (CSF) identifies the ground-truth states of the underlying Markov Decision
Process (MDP) ($ 4. Our analysis also helps reason about failure cases and formulate practical
recommendations 4..2). 2)
Thus, focus on identifying the components of successful MISL methods, following Reizinger ct al. (2024a); Park et al. (2023a). We show that
The good performance of MISI. hinges on learning representations via mutual information estimation under diverse policies and inner product parametrization of the model.
Identifiability also sheds light on the limitations of some methods and design choices ($ 4.2) e.g..
why the maximum-entrope policy is suboptimal in skill learning, or why feature parametrization matters. By drawing on diversity and variability assumptions from the ICA and CRL literatures, we formalize what constitutes a diverse policy and analyze the roles feature dimensionality and skill space coverage, yielding practical insights. Our contributions (Fig. I
We explain the success of mutual information skill learning (MISL) as the interplay of MI estimation
under diverse policies and inner product model parametrization, leading the first identifiability
results in RI. for Contrastive Successor Features (CSF) (6 (.T);
Our theoretical results enable quantify what diverse policy means and to pinpoint limitations of previous methods, leading to practical recommendations ($ 4.2);
We validate our theoretical claims of feature identifiability in CSF in both state- and pixel-based
MuloCo and Deep Control environments ($ 5).
2
RELATED WORK
2.1
UNSUPERVISED SKILL DISCOVERY
While reinforcement learning (RL) is typically cast as problem of learning single policy to maximize a scalar reward function (Sutton al. 1998), recent research focused on self-supervised objectives, inspired by information theory, to incentivize exploration while side-stepping the problem
of reward specification (Park et al. 2024b Eysenbach et al. 2019; Sharma et al., 2020 Choi et al.
2021). Such methods aim to learn "universal" representations that can be used to solve diverse down- stream tasks. Paradigms include curiosity (Pathak et al., 2017 Burda et al.). regularity (Sancaktar et al. 2023), model uncertainty (Treven al. (023). and disagreement (Pathak et al. Sekar et al.
2

---

Published as a conference paper at ICLR 2026
2020 Mendonca et al. 2023). with works showing connections to Contrastive Learning (CL)
(Eysen
bach et al. [022] 2023 Zheng et al. [2023] Hansen et al., 2019; Park & Pardalos 2021 Laskin et al.
2022). Other works in wnsupervised skill discovery (USD) aim to learn set of policies (1.e.. "skills"
that span the space of behaviors that an agent might perform in an MDP (Eysenbach et al. 2019
Achiam et al.] 2018| Zheng et al. 2025 (Sharma et al.) 2020 Park et al. 2022 Suttonn et al. 1998).
Mutual information skill learning (MISL) is a subset of USD methods that relies on information-
theoretic principles (Zeng et al., 2025; Park et al., 2024b. Sharma et al. 2020; Eysenbach al.
Mohamed & Jimenez Rezende, 2015] Gregor et al., 2017; Achiam et al.) 2018). Yang et al. (2025) investigate MISL from skill learning perspective, though they define the skill to include the parameters of the policy network.
2.2 SELF-SUPERVISED LEARNING
Self-supervised methods aim to use pretraining task to learn "universal" representations that facilitate solving downstream tasks, such as classification, object detection, or segmentation (Chen
et al. 2020 Oquab et al., (2024) Radford et al. 2021, [Bardes et al.] [022] [Baessriiero et al. 2023
Balestriero & LeCun 2022; Siméoni et al. 2025). Many SSL methods are related to information
theoretic principles (Zimermann et al. 2021 Bizeul et al. 20244 Liu et al. 2022 (Shwartz-Ziy
et al. 2022). These methods often aim to leamn representations such that similar samples have similar representations and dissimilar samples dissimilar representations (Wang & Isola 2020).
Recent advancements in the nonlinear ICA literature explained the success of many contrastive SSL
algorithms by proving their identifiability (Zimmermann et al. 2021 Rusak et al. 2025
Reizinger
cet al. 2024a).
2.3 CAUSAL REPRESENTATION LEARNING
Recent work has shown fundamental connections between RL, SSL, and causality. For example, there is long line of work using self-supervised learning to drive RL, particularly in skill discov- cry (Eysenbach et al.
2019 2022 (Sharma et al., 2020; Pathak et al. 2017 Pathak et al. Hansen Park et al. 2024b; 2023b). Indeed, the RL problem is fundamentally about agents taking interventions (1.e., actions), pinpointing the natural connection to the field of causality (Pearl 2009 Spirtes et al. which to answer causal queries about the effect of interventions. That is, to predict what happens when an agent takes series of actions. Thus, naturally, many CRL methods are often tested on tasks involving an agent such as a robotic arm or a control system (Schölkopf
al. |
eet al. 2021 Liu et al. 2023 Lippe et al. 2022a b Yang al. 2023)
3 BACKGROUND
Notation. We consider a partially observable Markov Decision Process (POMDP) without a reward function. We denote successive states as and s' with initial state distribution p(s). actions as the state transition distribution as p(s' s, a), observed variables as o = g(s), g: Rd + R p2d being a deterministic generator function, representations or features as @0)), and skills as z;, which are all random variables (RVs) sampled from a prior skill distribution p(z). The skill-conditioned policy is (a 0, 2), and the variational model for the critic is q(2 ((0), o(o'))
3.1 IDENTIHIABILITY IN SELF-SUPERVISED LEARNING.
Identifiability means that, assuming an underlying DGP for the data, the corresponding "ground-truth" latent factors can be recovered up to simple transformations (such as linear maps). That is, for ground-truth states s and features Φ(o), it holds for a particular linear map A that s()) As. In the nonlinear case, identifiability is only possible with further assumptions (Darmois) 1951 Hy Hyvärinen & Pajunen| 1999: Locatello et al., 2019). which restrict either the model class (c.g.. to have a specific Jacobian structure (Gresele et al., (021)) and/or the latent distribution (c.g.. nonstationary time series (Hyvarinen & Morioka (016)). Intuitively, these assumptions are required to "break the symmetries" e.g., the rotational symmetry of the Gaussian distribution or the rotational symmetry of the inner product parametrization. These symmetries emerge from the likelihood be thought of as maximizing the data likelihood a related quantity (such as cross-entropy))- where we plug in the model parametrization and the learned latent distribution via a change of variables. ICA theory shows that estimating MI is generally insufficient to learn a "useful" representation without further assumptions (Tschannen et al. 2020 Roeder et al. 2020 Hyvärinen & Pajunen| 1999
methods can
Locatello al. 2019 Reizinger et al. 2024a)
3

---

Published as a conference paper at ICLR 2026
A prominent family is that of auxiliary variable methods (where the latents conditionally indepen- dent given the auxiliary variable) (Hyvarinen et al.
Gresele et al.) |0200] Khemakhem et al. 2020a Hälvä et al.] 2021;; Hyvarinen & Morioka Khemakaee et al. 2020b; Locatello et al., 2020; Morioka & Hyvarinen 2023] Morioka et al. 2021 as we will show, skills in MISI, can also be interpreted auxiliary variables. Intuitively, diverse skills representative of set of distinct states. To prove identifiability, ICA usually assumes specific DGPs such as energy-based models or inner product parametrization. Recently, Reizinger al. (0244)) showed that the cross-entropy loss is key to explaining why many (self- >supervised deep learning models learn useful (identifiable) representations. We use this insight to prove the identifiability of CSF.
2019)
3.2 MUTUAL INFORMATION SKILL LEARNING (MISL)
Representation learning. MISL is based on representation learning, providing a form of infor- mation bottleneck, which is crucial for success (Zheng et al. 2025 Park et al. (021). The learned representation captures the relationship between (s. s') and 2, including state transition information. The representation is usually constrained to the (unit) hypersphere, following the practice of CL (Chen) et al. 2020). To reflect the dynamics of the environment, i.e., the relationship between consecutive states (s, s'), ), it is common to use the feature differences in the objective function, i.e., ø(o') Φ(o) with encoder 6. For a detailed review on representation learning in RL, cf. [Echchahed & Castro (2025).
Diverse skills: mixture policies. MISL aims to learn distinguishable, i.e., diverse that solving a variety of tasks requires versatile policy. This implemented via a skill-conditioned policy, i.e., x (alo,, z) which is trained to maximize diversity. Intuitively, the marginal policy T(ass) T(aao, =) =)p(=) can be thought of as a mixture policy, where each mixture component is a different
tool in the agent's toolbox (cf. Ex. D.. Diversity means that a discriminative model can uniquely
infer the skill 2; from consecutive states s'). This notion relates to sufficient variability conditions
in ICA (Hyvarinen & Morioka 2016 Hyvarinen
al. 2019 Khemakhem al. 2020b) and
interventional discrepancy from CRL (Wendong et al. 2023). We refer the reader to Reizinger et al. (2024b) for a discussion on different ways of measuring diversity. Formally:
Definition 1 (Diverse skill-conditioned policies). For set of skills 2; E E|S form affine generator system of Idd (cf. Defn. 2). we call a skill-conditioned policy aaalo, 2) diverse if an ideal discrininative model can uniquely infer the skill from consecutive states (s, given a set of skills. Alternatively, T <alo,, z) is diverse, if for given state transitions p(& a). the integral ((ss))s,)<((e) zi zx)p(s] (x) ds is not equal almost surely for any two Zi; Zjzi.
inner product parametrization. MISL objectives approximate MI via an evidence lower bound (ELBO), for which they require a variational approximation. The variational posterior (i.e., the critic or the Q-vauue function) is often parametrized as an inner product, which is critical for achieving great performance (Zheng et al.) (202)), though an explanation is yet to be provided. This parametrization is prevalent in SSL, and was theoretically shown be crucial for identifiability guarantees (Roeder et al., 2020;; Zimmermann et al.] 2021 Hyvarinen et al. 2019 Hyvarinen &
Architecture:
Morioka 2016
Khemakhem al. 2020bb Reizinger al. 2024a).
Our identifiability proof in $ $ is for which is representative of prior work Park eet al.] 2024bb Gregor al. [017] [ Warde-Farleyy et al. 2018). CSF learns both feature representation via an encoder and a skill- soiill-oondiiooeee policy; the skills are represented by vectors 2 drawn uniformly from the hypersphere (cf. Fig. 1
Contrastive Successor Features (CSF) (Zheng et al.
2025
a prototypical MISL method, CSF (Zheng et al., 2025
State representation. CSF learns a probabilistic critic g(=(((o), ((o')) with encoder & to dis- criminate the skills based on consecutive observations (o, o') corresponding to the state transition (s, s'). The encoder can be trained either from direct state observations or from pixels. The loss is a contrastive lower bound on the mutual information 1(s,
(zz1o(o).((o')) p(z:) exp (0(0') - ((o)) 2
Ep(z) exp (00(0') - o(o)) T
which is equivalent to cross-entropp loss, as it was shown for different parametrizations (Hyvarinen) & Morioka 2016 Hyvarinen et al. 2019 Zimmermann et al. 2021 Rusak et al., (025). Intuitively, contrastive losses are used to learn probabilistic model over latent vectors that prescribes the
4

---

Published as a conference paper at ICLR 2026
relationship between similar and dissimilar samples. To fit this distribution, they minimize a statistical distance (the BeulbackkLeibler divergence), which relates the loss to cross entropy. The loss in (I) can be equivalently seen as a parametric instance discrimination objective (Wu et al. 2018 Oord
eet al., 2019 He et al. 2020) on the feature difference (øfo') d(o)), akin to formulations in prior
work (Ibrahim al., 2024 Reizinger al.] 2024a). Parametrizing the critic as a log-lineat model
with an inner product parametrization is crucial for identifiability (Hyvarinen et al. 2019 Roeder
al. 2020, Zimmermann et al., 202T; Reizinger et al., 2024a).
The policy. The ckill-conditioned policy zi Φ(0). 2) is learned by optimizing the reward function rz (o((), o(d')) (ø(o') d(o)) 2, where Φ the same encoder as above:
= arg max Ep(z) Ba(
Σ 1=0
C'rrs6(0), o(0')) = Bas) Ba()
Σ
((((0)))(0))
(2)
1=
where s p(s' 8, s, a a~ x), p(z)= _ Uniform
(3)
3.3 BUILDING THE BRIDGE: MODELING THE MISL PROBLEM AS A DGP.
Real-world RL problems often only provide high-dimensional observations but access to the ground-truth states s. MISL methods learn representation ofo) of these observations and use those downstream to determine the next action. We investigate whether and to what extent the learned representation ((o) captures important information about the states s. We use tools from identifiability theory to answer this question. For this, need to formally define data generating process (DGP), which connects the POMDP of the RL problem to a latent variable model. In essence, the DGP posits probabilistic model of states, actions, transitions, and skills. Identifiability guarantees require assumptions on the DGP, which requires us to transform the design choices of MISL methods into formal assumptions. To match the ICA literature, we model the skills as a set, with each skill having corresponding high-ding -igh-dimensional unit vector 2; E auxiliary variables in the ICA literature. This is slightly different from how MISL methods handle the skills by sampling them for each rollout from, e.g. a uniform p(z). To reconcile this modeling difference, we experimentally compare fixed set of skills (of varying number) and skill sampling in Fig. 4 However, as identifiability guarantees only require that the skills span Rdd, these modeling choices are compatible, as with enough samples, the skill vectors span Id almost surely. The POMDP includes the model of the state transitions Ms'|s, a), from which we observe a, but might not directly observe 8, only a function thereof via the generator o = g(s). The goal of ICA is to invert g. i.e., to extract the state from ICA further requires making assumptions about the probabilistic model, typically about conditional distribution. As MISL methods aim to infer the skill from state pairs, we will assume specific form for the conditional p(s' s|z), where the feature difference s' -s defined in S
-his way, the skills can be treated as the
4 IDENTIFIABILITY INSIGHTS IN MUTUAL INFORMATION SKILL LEARNING (MISL)
Intuition. RL and representation learning methods are often pretrained via self-supervised tasks to leam a "universal" representation that can solve many downstream tasks (.22)
Our insight is that learning diverse skills is equivalent learning distinguish data under different distribution shifts or interventions, and this leads to RL agents identifying the ground-truth states of the underlying POMDP up a linear transformation.
Before analyzing the relevant technical assumptions, we provide an illustrative example of how distinguishing skills can be useful to learn the states of the underlying POMDP:
Example 1. Assume that robot moves around in maze to create a map of it. However, it does not have access to other sensors but a camera. To create the map, i.e., to learn the underlying state information such the position of the robot, the walls, other objects, it needs to move around to collect representative images. The ICA setting assumes that we already have such images and aims reconstruct the state. Skill- -ksle- RL solves harder problem, as it also needs to learn policy to explore while learning the underlying state representations. The question we answer in this work is: do those representations identify important quantities such as position and orientation?
5

---

Published as a conference paper at ICLR 2026
4.1 THE IDENTIFIABILITY OF CONTRASTIVE SUCCESSOR FEATURES (CSF)
Our key insight is to analyze recent advances in MISL EEsenbachh et al. 2019 Hansen al. 2019
Sharma et al., 2020, Park et al. 2021 Eysenbach et al. 2022| Park et al., 2023b 2024b Zheng et al.
2025) through the lens of identifiability theory: The success of MISL methods, including the role of diversity and importance of a linearly parametrized critic, can be explained by identifiability theory. To show the identifiability of the CSF features, for this section, we assume that we have access to diverse skill-conditioned policy and recall how we related the POMDP to a DGP. We proceed in the following steps:
(i) We show that given a diverse policy, the collected data (state trajectories or observations thereof) satisfy the assumptions required for identifiability in ICA. iii) Then we investigate what this identifiability result implies for CSF.
Matching the assumptions of CSF to ICA. We investigate whether the assumptions of nonlinear ICA theory match those in MISL. The assumptions below are sufficient to recover the ground-truth states up to a linear transformation, i.e., one can fit linear map between the features learned by the model and the true ones (e.g., if one has access to them, such as in a simulator). We state our assumptions informally (the formal statement is in Appx. B.I
Assumption (Informal). For consecutive states 8, 8' E TR. (5' E E and skills 2, E 8-1
(i) There is a finite set of skills on the unit hypersphere, which are diverse in the sense of Defn. 7
(i) Conditioning on the skill, the consecutive states s, s' are close to each other.
(ii) Each state difference (s' -s) is marginally equiprobable.
(iv) The observations o are generated by passing the latent state s through a continuous and injective generator, where dim o 2 dim s.
(v) We train an encoder with an inner product parametrization to map observations to features b s.I. dim Φb dim 8, and assume that it can globally optimize the contrastive objective in Eq. 1
The above assumptions are sufficient for the identifiability result; however, this does not imply they are all necessary. Namely, many identifiability results, despite their assumptions, can be successfully applied in practical scenarios such as ecology, climate science, robotics, functional medicine, dynamical systems, and neuroscience (Zhu et al.) 2025| Wismüller et al.) 2022 Locatello et al. 2020 Lippe et al., 2022b Zhou & Wei 2020; Yao et al. (0.4b/a) a). Many works show that identifiaiility is robust to assumption Violations in practice (Zimmermann et al., 202T Sliwa et al.
2022, von Kügelgen et al., 2023 Montagna et al. 2023 Reizinger | [2024a). Thus, the real question is whether they are realistic in practical RL scenarios? Based on empirical evidence from CSF, our answer is affirmative. As in practice, one might not have access to the true states, formulate these statements in terms of the learned features (60).
(i) In CSF skills are drawn uniformly from the hypersphere, and if we have sufficiently many of them, then they are "diverse" enough to almost surely form an affine generator system R4. We demonstrate that this setup is sufficient, but not necessary: a set of discrete skills also leads to high identifiability scores (Fig. 4. Also, empirical observations show that the skill- lkill-conditioned policy cannot be optimal if it does not depend on the skill, i.e., if has maximum entropy.
(ii) Empirical observations show that the features of consecutive observations Φ(o), Ø(o') are close
to each other on the hypersphere (Zheng et al., 2025 Fig. 2(a-b))
(ili) Empirical observations show the learned feature differences ((o')= Φ(o) are uniformly dis-
tributed on the hypersphere (Zheng et al. 2025 Fig. 2(c))
(iv) In practice, MISL methods are either trained from directly observing the state or from pixels.
Both cases reasonably fulfill the assumption.
(v) The neural networks used for training use an inner product parametrization, and empirical evidence by Zheng et al. (2025) showed that this works the best in practice.
Identifiability of the underlying features. Under Assum. I the feature differences are identified up to a linear map (we defer the formal statement to Appx. B.2):
Proposition (Identifiability of CSF feature differences (informal)). Under Assum. 7 when a
continuous encoder and a linear classifier Z globally minimize the cross-entropy objective ① then
the state differences s' - 8 are identified up to a linear map A E Rdxd, i.e. o(o')-p(o) A -s.. s].
Zheng et al. 2025 evaluate the representations learned with METRA. We also verify with CSF in all environments evaluate in Appx. D.4
6

---

Published as a conference paper at ICLR 2026
1(,,51=) (o(0) <(d')
((s;2) : <(s)=
((0') o(0)
o(o).o(o))
o(o') d(n)
((o), o(a') =|
Figure Illustrating the advantage of parametrizing mutual information as T(s, z;) vs.
T(s; =:): the reward function and the feature learning objective impose different inductive biases on the structure of the latent space. Left: using feature Φ(o') - Φ(o) to parametrize
ensures that the embeddings of consecutive states are close but distinct. Right: optimizing [(s; a) incentivizes the embeddings of consecutive states to both be parallel to skill 2,, which can lead to collapsed features (i.e., Φ(0) = =o(o')).
The proof follows by the straightforward application of the proof technique of Reizinger et al. (2024a) Thm. 1)-in Appx. B.2 we provide high-level overview thereof. Since the linear map in Prop. is the same for all nomtl-oomallzzdd state differences and the model has an inner-product parametrization, linear identifiability also holds for the states. We formalize this statement in Prop. and defer it to Appx.
These results mean that the features learned by CSF (Zheng et al. 2025) will correspond to the ground-truth states of the underlying POMDP up to linear transformation. Having identifiability both for and (s' s) might suggest that does not matter whether the objective optimizes lower bound on [s, s': or [(s; 2). As we show in $ 4.2 the difference lies the additional geometric constraints the latent space. Namely, there exist spurious solutions of the InfoNCE objective that do not preserve the structure of the latent space (Wang et al. 2022).
4.2 INSIGHTS FROM ICA THEORY
Mutual information formulation matters for the geometry of feature space. In the liter- ature, there many choices for optimizing MI, namely, I(s, s'; 2) versus [ss,, s; s; versus [s;;))<t cf. Fig.
2 for a comparison. By looking into the policy and analyzing what maximizing E.... (Ø(o') {(o)] z means, we hope to shed light on the advantages of [(s, s'; over the latter versions. Maximizing the inner product (o(o') d(o)] z means that the difference o(d') - d(o) needs to be parallel to This implies that neither (b()) nor ((o') can collapse to the same vector, as they need to be distinct such that their difference parallel to z. On the other hand, optimizing [(s; z) would mean two separate conditions for ((o) and @(o'). But if both parallel to then they must be parallel, i.e., the representation collapses. This violates the assumption that consecutive states should be close each other in embedding space (but neither the same, nor very far apart, ef. Assum. ()). The same argument holds for / (30, z) with the difference of offsetting the whole space by the initial state. Instead, I(8, =) prescribes the "closeness" of consecutive states.
A practical implication of diversity rewards. A perhaps interesting interpretation of rewards such as (3) is that it quantifies notion of data diversity-for other means measure diversity, refer to (Reizinger et al. 2024b). From theoretical perspective, diversity is a binary question, as is required to make matrix invertible. But this matrix can also be ill-conditioned without being rank-deficient,, leading to performance deterioration (Rajendran et al. 2023). Understanding when the reward good predictor of learning useful representations from a given data set (e.g., in offline RL) is an interesting avenue for future work.
Maximum entropy policies lead to worse performance. Although several RL methods use an entropy regularizer (Eysenbach et al., 2019 Sharma et al.) 2020 Park et al. (2024b). too strong regularization can lead to worse performance. Entropy regularization, in this extreme, is also not suitable for skill diversity, as maximum entropy policy breaks the dependence on the skill of the skill-conditioned policy. We state this observation from the literature formally in Appx. [B.] and, thus, the state transitions do not depend on the skill, then from given state pair (s, s') it is impossible to infer the skill with the discriminative model q(2|s, s and the reward cannot be optimal.
Intuitively, if the actions- -ndon,
7

---

Published as a conference paper at ICLR 2026
Half Chectah
Quadruped State
Quadruped Pixel
Kitchen
Robobin
CDR
0.8
0.8
0.8
0.8
0.4
0.4
0.4
0.4
0.4
din)
0.0
0.0
0.0
vico
-22
Oracle Ret.
300
2000
180
10
40
1000
250
150
15
200
Env. steps
x X10*
Env. steps
x10"
Env. steps x 100
Env. steps X10*
Env. steps
x10"
Figure 3: CSF explores the state space and identifies the underlying states in MuJoCo and DMC, up to a linear transformation. Top: coverage-dependent Rt score, measuring both state exploration and linear identifiability for both features 6(0) and feature differences ((o') Φ(o) (higher is better); Bottom: oracle return indicating zero-shot task transfer. Error bars represent one standard deviation.
EXPERIMENTS
Setup. We use the codebase of Zheng et al. (2025) and run experiments in the MuJoCo and Deep- Mind Control (DMC) environments with the CSF algorithm. During self-supervised pretraining, we monitor the learned (successor) features. To evaluate identifiability. we set the feature dimensionality to match the number of ground-truie features. We use either states (Half Cheetah, Ant, Quadruped State) or pixels (Quadruped Pixel, Kitchen, Robobin) as observations to learn the features 6(o). In the state-based environments, the encoder is, in principle, able to represent the identity map; however, is unclear whether such features would optimize the loss. Unless otherwise noted, error bars represent one standard deviation. Refer to Appx. C for details.
Metrics. The benefit of using simulated MuJoCo and DMC environments is having access to the ground truth states, enabling one to evaluate the relationship between the learned features Φ(0) and the ground-truth latent factors s. To measure exploration, we report the state coverage, i.e., the unique states observed across all evaluation trajectories using different skills. To assess downstream performance, we also report the oracle return from the best rollout among the sampled skills, i.e., the extrinsic reward defined by the environment. The oracle return evaluates zero-shot skill transfer: a diverse skill set should naturally contain one that performs well on the downstream task. As the identifiability guarantees (Prop. (T] hold up to a linear transformation. we fit a linear map A between the features (0)) and the ground truth states s by minimizing ||s A and report the coefficient of determination R² (Wright 1921) of the linear fit, which the standard metric in the ICA literature (Hyvarinen & Morioka [016)] Hyvarinen et al.) 20199] 2023 Hyvärinen et al. (023)) However, the 1t2 score can be misleading in RL: it measures performance only on the visited states and not all states; thus, it cannot distinguish between a collapsed scenario (i.e., no exploration) and well-explored one. Thus, we introduce a relative roverage-dependens R² score (CDR²), ), defined as the harmonic mean of the normalized coverage and the R score. This yields metric which is near zero if an agent has either a low R² score or poor coverage. For details, refer to Appx. C.3
Results. We investigate whether CSF identifies the ground-truth latent states up to linear transfor- mation and how this relates the oracle return. We report scores in Fig. 3 across two state-based MDPs (Cols. 1-2; the Ant results are in Appx. D and three pixel-based environments (Cols. 3-5). CSF both explores the state space and identifies the ground -ruth states up to a linear transformation (Fig. 3 top; separate R² and coverage results are in Figs. D.1 and D.2). aligning with Prop. | In the state-based environments, it is by no means obvious that the encoder will keep all information about the underlying states, e.g., there could have existed shortcut solution that optimizes the CSF objective while discarding some information about the states. In btate- environments, the oracle return and identifiability performance (CDR²) are strongly correlated. In pixel-based environments, the relationship is less clear, though the agents both explore and learn to extract the ground -truth states, even though the oracle return is noisier. We provide correlation analysis in Tab. D.I
To test the role of the diversity conditions nonlinear ICA (Hyvarinen et al. 2019 Rajendran et al. 2023 Wendong et al. 2023 Reizinger et al. 2024b), investigate how skill diversity affects state
8

---

Published as a conference paper at ICLR 2026
1.0
Coverage
1300
2000 R!a
Coverage
Oruele Return
0.8
0.7 3/
0.6
1150
0.4
0.5
1100
dun
500
1050
R
sampling
0.0
Number skills
15
40
Latent space dimensionality
Figure 4: Diverse skills are necessary for iden- tifiability and coverage in the Ant environ- ment, validating Assum. Ti). Skills are sam- pled from p(z) at the start of pretraining and kept fixed throughout, except in the 'sampling" case where skills are redrawn from p(2) dur- ing training, emulating infinite set of skills (dim s = 29).
Figure 5: Sufficient latent space dimensional- ity is necessary for identifiability, validating Assum. [iv. Linear identifiability requires that the feature space has at least many dimensions as the true state. However, a smaller latent space is more beneficial for task transfer.
identifiability and coverage. Namely, Prop. only requires that skills span R.. suggesting that limited set of skills might be sufficient to achieve identifiability. We vary the number of skills z; and use fixed of z; vectors. This scheme is compared to the original version of sampling each time from the uniform p()). Using a small fixed set of pre-sampled skills is insufficient to cover the state space during pretraining or yield identifiable representations, as shown by the peak coverage and corresponding R² score in Fig. We also investigate how latent space dimensionality affects both identifiability and zero-shot task transfer-if if the feature space is lower dimensional than the state space in the POMDP, then all state information might be present but not linearly decodable. Fig. 5 shows n2 scores and zero-shot task transfer performance for varying latent space dimensionality. A low-dimensional latent space yields an information bottleneck that prevents the feature differences from encoding the ground truth states linearly; however, CSF still performs well on task transfe Notably, this result does not imply that CSF is not able to extract the ground-truth states, only means that it is not able to do linearly indeed, works studied representations with different geometries (Csordás et al.) 2024); however, both theoretical understanding and corresponding metrics are missing. We defer how latent dimensionality affects state coverage to Appx. C
Appx. C shows further results for zero-shot task transfer.
DISCUSSION
Limitations. We connected identifiability insights from nonlinear Independent Component Analysis (ICA) mutual information skill learning (MISL) and formulated practical insights. Our result relies on the observations of Zheng et al. (2025) and holds across multiple state- and pixel-based environments (for verification, refer Appx. D.4]. However, it requires further research to determine whether the technical assumptions can be relaxed to incorporate broader range of environments.
Extension to related works. Our work is of an explanatory nature, advancing our understanding of mutual information skill learning (MISL) methods Reinforcement Learning (RL). To the best of our knowledge, we are the first to prove identifiability of the learned features in RL (Prop. D particularly, for Contrastive Successor Features (CSF) (Zheng al., (025). Nonetheless, as the key components for state identifiability are shared across many MISI. methods, we expect our results to hold more generally. This insight helped explain why particular design choices in MISL are successful, including the inner product parametrization of the critic and also shed light on how different mutual information formulations affect the learned representation (8 4.2).
Conclusion. Our work theoretically proves that learning diverse skills a meaningful surrogate objective in Reinforcement Learning (RI.) for learning the ground-truth states of the environment up to a linear transformation. We show this by connecting the mutual information skill learning
CCSF is designed for lower feature space dimensionality, and empirically found it to be sensitive to increasing this hyperparameter

---

Published as a conference paper at ICLR 2026
(MISL) family to nonlinear Independent Component Analysis (ICA) methods, and proving linear identifiability for the features learned by Contrastive Successor Features (CSF) (Zheng et al. (025). Our identifiability guarantees not only provide possible explanation of why MISL works, but also identify the key components of successful MISI. methods. Furthermore, our theoretical insights help elucidate some failure modes of previous methods. We validated our theoretical claims empirically in DMC and MuJoCo environments, showing that CSF simultancously explores the state space and learns features that identify the ground-truth states up to a linear transformation. As exploration and state identification showed positive correlation with the extrinsic oracle return, defined by each environment, this suggests that identifiability can be helpful for zero-shot task transfer. However, as it is well known in the identifiability literature, in some cases recovering more latent factors (i.e., better identifiability score) can be odds with downstream performance on a particular task (Rusak) et al. 2025). We hope that our insights will open up new research possibilities and also help practical algorithm design.
REPRODUCIBILITY
We provide shell scripts to reproduce our results the scripts folder of our GitHub repository To obtain error bars, we additionally varied the seed parameter in the scripts to obtain three five independent runs.
ACKNOWLEDGEMENTS
The authors thank Jens Tuyls for his help regarding the MISL experiments. Patrik Reizinger and Bálint Muesányi acknowledge their membership in the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program and thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for its support. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S 18039A. Wieland Brendel acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 1 and via the Open Philanthropy Foundation funded by the Good Ventures Foundation. Wieland Brendel is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: /807. 10299, 2018. 3
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. Advances in Neural Information Processing
Systems, 35:26671-26668 2022. 3
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gor- don Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. A Cookbook of Self-Supervised Learning, April 2023. URL http:://arxi> /rg/ abs/ 2304 12210
arXiv:2304. 12210 [es]. 3
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: hatpeee--iinaiinnc--ovvaaaaaa regularization for self-supervised learning. In International Conference Learning Representattons, 2022. URL
net. wwwptne//ouuuuue forum?iouoo http::mmmmmmmmmmm20000x
Alice Bizeul, Bernhard Schölkopf, and Carl Allen. A Probabilistic Model to explain Self-Supervisee Representation Learning, February 2024. URL http:/pp///rxx..ooo///////2222/0 arXiv:2402.01399 [cs, stat].
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-Scale Study of Curiosity-Drivar Learning. pp. 15.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.
1597-11077 PmLR, 2020. 4
10

---

Published as a conference paper at ICLR 2026
Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu. Variational Empowerment as Representation Learning for Coal-Conditioned Reinforcement Learning. In Proceedings of the 38th International Conference Machine Learning, pp. 1953-1963. PMLR, July 2021. URL 2640-3498.
httpee///poceeeddiggo m wtt.ggs................. pprass//139/cho//
html ISSN:
Róbert Csordás, Christopher Potts, Christopher Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 248 262, Miami, Florida, US, November 2024. Association for Computational Linguistics. doi: 10. 10.1863//1//024.11 (16555///022.baaccooooo11111 blackboxnll-l.11..
URL htt.itts///aaaaaallllleee org/
1.111 9
George Darmois. Analyse des liaisons de probabilité. In Proc. Int. Stat. Conferences 1947, pp. 231, 1951.
Ayoub Echchahed and Pablo Samuel Castro. A Survey of State Representation Learning for Deep Reinforcement Learning. Transactions on Machine Learning Research, March 2025. ISSN 2835-8856. URL httpp:://openree//// net wtwp://wwwwweeetom.ecoeo...m 3
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference on Learning Representations, 2019. URL
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R. Salakhutdinov. Contrastive Learning Conditondt Reinforcement Learning. Advances in
Neural Information Processing Systems, 35:35603 35620, December 2022. URL
http:::/proceedinng..nnu cc http:///ppppppp.p.eeaeeee///////
3
Benjamin Eysenbach, Vivek Myers, Sergey Levine, and Ruslan Salakhutdinov. Contrastive Represen-
tations Make Planning Easy. December 2023. URL httpssss//opeeeee..iiet wttp:.ee/forrr.oo oor.e//rrrreeeeeeeefooononn
id=WObhHvgKK0
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational Intrinsic Control. February 2017. URL htt.ap/se//ppeeeeee..eeeeeeoooos
Fo4gg
Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf.
The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. In Uncertainty in Artificial Intelligence, pp. 217-227. PMLR, 2020. 4
Luigi Gresele, Julius Von Kügelgen, Vincent Stimper, Bernhard Schölkopf. and Michel Besserve. Independent mechanism analysis, a new concept? Advances in neural information processing systems, 34:28233-2822 -22248, 2021.
Siyuan Guo, Viktor T6th, Bernhard Schölkopf, and Ferenc Huszár. Causal de finetti: On the identification of invariant causal structure in exchangeable data. In Thirly-seventh Conference Neural Information Processing Systems, 2023. [
David Ha and Jürgen Schmidhuber. World Models. March 2018, doi: 10.528 wtw.eooooo.122776... [cs, stat].
URL http: /ttp///avvv//.////////eee/eaaae/ org/ www.////...... 10122
Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast Task Inference with Variational Intrinsic Successor Features. September 2019. URL
https: opw.revieww.w.e net wwwp/eeeet/ennnnooooooooeooo wow.mmiddBBeeaaeaa
3
Kaiming He, Haoqi Fan, Yuxin Wu. Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 9729-9738. 2020.
11

---

Published as a conference paper at ICLR 2026
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastiive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016. 3
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd international conference artificial intelligence and statistics, pp. 859-868. PMLR, 2019.
5
Aapo Hyvarinen, Ilyes Khemakhem, and Hiroshi Morioka. Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learning, March 2023. URL
http:://axxv..ogg/bss.tsse
16535 arXiv:2303. 16535 [cs, stat] version: 1. S
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence
and uniqueness results. Neural Networks, 12(3):429 439, April 1999. ISSN 0893-6080. doi:
1011016/509922266000990000000000 URL https:ttp//. www. www.encieiei....... / rect. com/
com com.scienccmmmmieccceccccm scce/ce///ceeeee//
articl atticcle//11//00330333333333 11000140333 403
Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural-equuation models: from linear nonlinear, February 2023. URL http:///rxiv
org/abs/230z 02672 arXiv:2302.02672 [cs, stat]. 8
Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA. arXiv:2 09620 [cs, stat], June 2021.
09620
arXiv: 2106.09620. 4
Mark Ibrahim, David Klindt, and Randall Balestriero. Occam's 's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?, June 2024. URL /ttt:://aaxiv.ort org/
abs/2406. 10743 arXiv:2406. 10743 [cs]. 5
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational Autoen- coders and Nonlinear ICA: A Unifying Framework. In International Conference on Artificial Intelligence and Statistics, pp. 2207-2217. PMLR, June 2020a. URL http:::/procceednng.. ISSN: 2640-3498.
mlr. wttp:p/es//llleeeeeaaaeeeeeeeeee
html
Ilyes Khemakhem, Ricardo Pio Monti, Diederik P. Kingma, and Aapo Hyvärinen. ICE-BeeM: Identi- 1553 [cs, fiable Conditional Energy-Based Deep Models Based Nonlinear ICA. arXiv:2002. 1537
stat], October
WRRLattpp//axxx///
abaaabs/2000 /202222222coo 11537
arXiv: 2002.11537.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Unsupervised Reinforcement Learning with Contrastive Intrinsic Con- Advances in Neural Information Processing Systems, 35:34478-34491, December
trol.
2022.
URL https::/procceeeeeeee wtwpeeedddiinnn...e.eeee wwwrins..cccceooo..... wtt wwtpcc/ppppppppeeteeeeeeeeeee..e fawrrffiiess//e http:ppper....leepapeee/////////
hash hhth///6662221110000000022222332
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M. Asano, Taco Cohen, and Efstratios Gavves. CITRIS: Causal Identifiability from Temporal Intervened Sequences, June 2022a. URL http:
//arxiv.
03169
Number: arXiv:2202.03169 arXiv:2202.03166 [cs, stat].
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M. Asano, Taco Cohen, and Efstratios Gavves. iCITRIS: Causal Representation Learning for Instantancous Temporal Effects. July 2022b. URL
net -oroumiidxxxDDX222772 3
Grace Liu, Michael Tang, and Benjamin Eysenbach. A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals, August 2024. URLhttpp /tat:://axxiv...
abs 2408 05804 MxXiv:2408.08800 [cs].
Xin Liu, Zhongdao Wang. Yali Li, and Shengjin Wang. Self-SSperrviedd Learning via Maximum Entropy Coding, 2022. doi: (0.4550/ARXIV.221...1
htlllttps::/aaxxx. wtwpii.orrr//..........
221 0. 11464 Publisher: arXiv Version Number: 1.
12

---

Published as a conference paper at ICLR 2026
Yuejiang Liu, Alexandre Alahi, Chris Russell, Max Hor,, Dominik Zietlow, Bernhard Schölkopf, and Francesco Locatello. Causal Triplet: An Open Challenge for Intervention-centricc Causal Representation Learning, January 2023. URL
arXiv:2301.055 .05169 [cs]. [
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learning of Disen- tangled Representations. In International Conference Machine Learning, pp. 4114-4124 PMLR, May 2019.
(ppzess/vv//oocteelllll.
html
ISSN: 2640-3498.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael T'channen. Weaely-Supeervieed Disentanglement Withoout Compromises. arXiv:2002.02888 [cs, star), October 2020. URL UtL.:ppe///aarx..or/////////2220 02886
arXiv: 2002.02886,
Russell Mendonca, Shikhar Bahl, and Deepak Pathak. ALAN: Autonomously Explornn Robotic Agents in the Real World, February 2023. URL BtLL///c///aaz2..or///////220200 06X46 [cs, eess].
Shakir Mohamed and Danilo Jimenez Rezende. Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
wew..... CC/ Abst httml
Francesco Montagna, Atalanti A. Mastakouri, Elias Fulig, Nicoletta Noceti, Lorenzo Rosasco, Dominik Janzing. Bryon Aragam, and Francesco Locatello. Assumption violations in causal discovery and the robustness of score matching, October 2023. URL
abs/2310.1332 13387 12X3322310.13387 [cs, stat].
Hiroshi Morioka and Aapo Hyvarinen. commectivity-conrastiio learning: Combining causal discov- ery and representation learning for multimodal data. In Proceedings of The 26th International Conference on Artificial Intelligence and Statisties, pp. 3399 3426. PMLR, April 2023. URL ISSN: 2640-3498. 4
ml
htm_
Hiroshi Morioka, Hermanni Hälvä, and Aapo Hyvärinen. Independent Innovation Analysis for Nonlinear Vector Autoregressive Process. UrXiv:2006.10994 [cs, stat]. February 2021. URL
https: /rrviv. owg/ http://bb/2000... 10944
arXiv: 2006.10944.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. /107.03/033374 fcs, stat), January 2019. URL http:://aaxxxv.org
abs/1807 03748
arXiv: 1807.03748. [
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features with- out supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL htttps://openreviewt net
6zFt. Featured Certification. 3
Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models, November 2023a. URL htt.:tt:://axxi..org/////// 2311. 03658 arXiv:2311 Stti:23111.33633 (cs, stat].
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The Geometry of Categorical and Hierarchical Concepts in Large Language Models, June 2024a. URL http: wtt.i..oggooo/.............. abs/ 0055/2406..o 01506 ArXiv:2406.01555 [cs, stat].
Scohong Park, Jongwook Choi. Jackyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitee Unsupervised Skill Discovery. October 2021. URL https: http:///pppeeeeeeeeeeeeee/e/////
Nd=BGvttghNNg
13

---

Published as a conference paper at ICLR 2026
Scohong Park, Jongwook Choi, Jackyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz-constrainee unsupervised skill discovery. In International Conference on Learning Representations, 2022.
Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability-AAware Un- supervised Skill Discovery, June 2023b. URL http:://aaxxv..rrr//////22000000 arXiv:2302.05103,
Seohong Park, Oleh Rybkin, and Sergey Levine. METRA: Scalable unsupervised RL with metric- aware abstraction. The Twelfth International Conference on Learning Representations, 2024b. URL
net/ forum? (irrrmmmiddccc5..00000
Sconho Park and Panos M. Pardalos. Deep Data Density Estimation through Donsker- Varadhan Representation. arXiv:2104, 06612 [cs, math), April 2021. URL 2104. 06612 arXiv: 2104.06612, g
Donsker-Varadhan orgaabs/
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-Supervised Exploration via Disagreement. pp. 10.
3
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-Driven Exploration by Self-Supervised Prediction. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 488-489, Honolulu, HI, USA, July 2017. IEEE. ISBN 978-1-5386-0733- 6. doi: 10. 109/CVPRW.2017.10 URL httt:::/ieeepp..ooo///// ieee org/ document
801 /
Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, edition, 2009. ISBN 978-0-511-8031661.1 doi: 10.1017/CB09780511 1803161. URL http:
//ebooks..... www..aaaari..... dge. wwtp://..rrg.e..eeeeeeeeeeeeee/e https//aaaabbbeeeeeeeeeeeeeea//a
8031 61 3
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, pp. 8748-8763. PMLR, July 2021. URL
httpps://proceddini mwr www.sp.esssvvvv. wtt.p/ppppeeeee1///eeeeeeee http:////rraoodd21eoa/ html
ISSN: 2640-3498. 3
Goutham Rajendran, Patrik Reizinger, Wieland Brendel, and Pradeep Ravikumar. An Interventional Perspective Identifiability in Gaussian LTI Systems with Independent Component Analysis, November 2023. URL/tttp://axxxv.co http///wwwwaaoaaaaaaaooo///o ort. abtr:/bb//2333 /211.. 18048 arXiv:2311.18049 [cs, eess,
stat]. 7
Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E. Vogt, Randall Balestriero, Wieland Brendel, and David Klindt. Cross-Entropy Is All You Need To Invert the Data Generating Process. October
2024a. URL bttps://openrevicc net www.:oooonnoonooaoaaa
17 18
Patrik Reizinger, Siyuan Guo, Ferenc Huszár, Bernhard Schölkopf, and Wieland Brendel. Identifiable Exchangeable Mechanisms for Causal Structure and Representation Learning. October 2024b.
URL
net forum km
Geoffrey Roeder, Luke Metz, and Diederik P. Kingma. On Linear Identifiability of Learned Repre- sentations. arXiv:2007 2000000070000000 fcs, stat), July 2020. URL /tLL:ttt:://arrxw.or////b/
00810 arXiv: 2007.000000 10.
Evgenia Rusak, Patrik Reizinger, Attila Juhos, Oliver Bringmann, Roland Zimmermann, and Wieland Brendel. Infonce: Identifying the gap between theory and practice. In International Conference on Artificial Intelligence and Statistics, pp. 4159 4167. PMLR, 2025. 10
Cansu Sancaktar, Justus Piater, and Georg Martius. Regularity as Intrinsic Reward for Free Play. Advances in Neural Information Processing Systems, 36. December
2023. URL hhttps: wtt.:iiinnnnnnnuuuuere neurips ceppprpeccppppppp.pppp.. f1les//aperr/223/ hat5//2220hccaa1ee66666711353363 org.a//aee...aaaaaeeee.ee.eearer
14

---

Published as a conference paper at ICLR 2026
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-6344 634444, 2021.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to Explore via Supf-Suuee World Models. arXiv:2005.08 0056000000600.. [cs, stat], June 2020. URL http:://aax.vooo//////aa.//
05960
arXiv: 2005.05960.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
URL haattps://ppenrevieeee net forumoormmm
Ravid Shwartz-Ziv, Randall Balestriero, and Yann LeCun. What Do We Maximize in Self-Supervised Learning?, July 2022. URL http::/aaxiii// wwwp://rggg.b/.brt.e attpaabb/222...2 10081 arXiv:2207.10881
[cs]. 3
Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie, Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. DINOv3, August 2025. 10104 arXiv:2508. 10104 [cs].
URL http:
Joanna Sliwa, Shubhangi Ghosh, Vincent Stimper, Luigi Gresele, and Bernhard Schölkopf. Probing the Robustness of Independent Mechanism Analysis for Representation Learning, July 2022. URL
http:// /ttE:://arxi..om org/ abs
arXiv:2207.066 0061777777777 [cs, stat]. 6
Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.
Richard Sutton, Andrew Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Lenart Treven, Cansu Sancaktar, Sehastian Blaes, Stelian Coros, and Andreas Krause. Optimistic Ac- tive Exploration of Dynamical Systems. Advances in Neural Information Processing Systems, 36. December 2023. URL https: /ttp0///////////proeeeeeeeeeee..
2023/ html
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On Mutual Information Maximization for Representation Learning, January 2020. URL http:
//arxiv.orr abtp///bb///./..///./ 1t11//9771.11.... 13625 arXiv:1907.1366 [cs].
Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Barein- boim, David M. Blei, and Bernhard Schölkopf. Nouparametric Identifiability of Causal Represen- tations from Unknown Interventions, October 2023, htt::ttt://aat /tEtp://axxiv. /atpiv..vvv.. org/ abs 2306
00542 arXiv:2306.0055 00542 [cs, stat].
Tongzhou Wang and Phillip Isola. Understanding Contrastive Representation Learning through Alignment and Uniformity the Hypersphere. 1rXiv:2005.10242 [cs, stat], November 2020.
URL http:::/txxii.oooot//
abs/ 10242
arXiv: 2005.10242, 3
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a Ladder: New Theoretical Understanding of Contrastive Learning via Augmentation Overlap. May 2022. URL
http://arxiv.cm http://wwxww.....aaaaooo wwtp:////2.....3..3....// 13457 arXiv:2203. 13457 [cs, stat].
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv: /./ 1359, 2018.
Liang Wendong, Armin Kekić, Julius Kügelgen, Simon Buchholz, Michel Besserve, Luigi Gresele, and Bernhard Schölkopf. Causal component analysis. Advances in Neural Information Processing Systems,
2023.
g
15

---

Published as a conference paper at ICLR 2026
Axel Wismüller, M Ali Vosoughi, Adora DSouza, and Anas Z Abidin. Investigation of large-scale extended granger causality (Isxge) on synthetic functional mri data. In Medical Imaging 2022: Biomedical Applications in Molecular, Structural, and Functional Imaging. volume 12036, pp. 42-51. SPIE, 2022. 6
Sewall Wright. Correlation and causation. Journal of Agricultural Research, (7), 1921.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised Fea-
ture Learning via Non-Parametric Instance Discrimination. pp. 3733 3342, 2018. URL https:: thecvf
cvpr 2018/htm //t
html
S
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. CausalVAE:
Structured Causal Disentanglement in Variational Autoencoder, December 2023. |
//arxiv. http://o//abb//......//e/o
arXiv:2004.08697 [cs].
Yucheng Yang, Tianyi Zhou, Qiang He, Lei Han, Mykola Pechenizkiy, and Meng Fang. Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsuper- vised Reinforcement Learning, June 2025. UtLLtttt.///aa.x..c///////////// 10629 arXiv:2506. 10629 [es].
Dingling Yao, Caroline Muller, and Francesco Locatello. Marrying Causal Representation Leanee Learning with Dynamical Systems for Science, May 2024a. URL http://aaxx...or//aabb//////
13888 arXiv:2405. [cs, stat]. 6
Dingling Yao. Dario Rancati, Riccardo Cadei, Marco Fumero, and Francesco Locatello. Unifying Causal Representation Learning with the Invariance Principle, September 2024b. URL http:
//axxiv.org/aass///.
02772 arXiv:2409.02772 [cs, stat].
Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive Difference Predictive arXiv:2310:20141
Coding, October 2023. URL hattp::/arxiv.or//aba////22
[es]. 3
Chongyi Zheng, Jens Tuyls, Joanne Peng, and Benjamin Eysenbach. Can MISI fly? analysis and ingredients for mutual information skill learning. The Thirteenth International Confer-
ence on Learning Representations, 2025. URL https: /ttp///pepewweeeeeeeeeeeeeeeeeee net/forum?i
>xoIevdF07U
10
17
19
23
27 29
Ding Zhou and Xue-Xin Xin Wei.
Learning identifiable and interpretable latent models of high- dimensional neural activity using pi-vae. Advances neural information processing systems, 33: 7234-7247, 2020. 6
Yongjie Zhu, Lauri Parkkonen, and Aapo Hyvärinen. Second-order instantancous causal analysis of spontaneous meg. Imaging Neuroscience, 3:imag_a a a_05553, 2025. 6
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. International conference on machine learning, pp. 1 12990. PMLR, 2021.
16

---

Published as a conference paper at ICLR 2026
A
IMPACT STATEMENT
This paper presents work whose goal is advance the fields of Reinforcement Learning and Identifi- able Representation Learning. Our focus on representation identifiability promotes transparency and interpretability, which are important safeguards against unintended use.
B PROOFS
B.1 FORMAL ASSUMPTIONS AND FEASIBILITY
Definition 2 (Affine Generator System Reizziner et al. 2024a) (Defn. 1) ). A system of vectors {2j E RRa) is called affine generator system if any vector in R. is affine linear combination of
the vectors in the system. Put into symbols: for any 2, IR there exist coefficients as E R, such that
Σ 04z and Ea=
Lemma (Properties of affine generator systems (Reizinger al. 2024a (Lem. 1)). The following hold for any affine generator system {2, E ]:
1. for any i # the system {z+ 2j} is now generator system of R4; the invertible linear image of an affine generator system also an affine generator system.
Assumption 2
(artiionn(AApppppa (Adapted from (Reizinger et al. 2024a) (Assm. IC)), For consecutive states 8, 8 E IR:
(8' -8) E and skills z e E we assume:
(i) The finite set of skills is unit nnti-normalized and forms an affine generator system of Rd (Defn. 2 Each pair of features corresponds to one skill. That is. an ideal discriminator can uniquely
map () 2, yielding diverse skills (cf. Defn. [
(i) The skill-conditioned features p(s' - follow a von Mises-Fisher (vMF) distribution on the hypersphere with mean expressing that consecutive states close to each other:
(s' s) p(s' s|2) cdlz.s'-o).
(5)
(ii) The marginal of the features (s' s) ) is uniform on the hypersphere.
(iv) The critic g(2)@(o), o(o')) uses an encoder ó R2D - IR to learn the features with an inner product parametrization (Ø(o') - <(o)) (M(o)-"(o)]] 2, it optimizes contrastive objective ① and is expressive enough to reach the global optimum of the objective.
(v) The observations o are generated by passing the latent state through continuous and injective generator function g 8 D, i.e. = g(s), where D 2 d.
Assumption feasibility. As we are modeling a practical scenario, we need to investigate whether these assumptions are realistic:
(i) In CSFFskills are drawn uniformly from the hypersphere, and if we have sufficiently many of them, they span Rdd almost surely, thus they almost surely form an affine generator system. We demonstrate that this setup is sufficient, but not necessary: set of discrete skills also leads to z;. As show in Prop. maximum entropy policy is not diverse. Applying our argument to pairs of skills, one can always improve diversity if the state transitions depend on any one of those (as opposed to no skill dependence, which is implied by a maximum entropy policy), which implies that each state transition can be mapped to a single skill. Refer Prop- 4 for details.
high identifiability scores (Fig. 4). The policy optimizes [d(o') <(o)]
skills
(ii) Empirical observations show that the learned features á(o') - -(() follow vMF for
CSF ZZeng et al.) 2025 Fig. 2(a-b)). Our empirical validation also verifies this assump-
tion for Half Cheetah, Kitchen, Quadruped, and Robobin in Appx. D.4]
(iii)
Empirical observations show the learned features ((o') d(o) follow a uniform marginal on
the hypersphere for CSF (Zheng et al. 2025 Fig. 2(c))
(iv) The neural networks used for training use an inner product parametrization, and empirical
evidence by (Zheng et al. 2025) showed that this works the best in practice.
(v) In practice, MISL methods are either trained from directly observing the state (i.e., is the identity) from pixels. Both cases reasonably fulfill the assumption.
Different pairs of consecutive states can map the skill
17

---

Published as a conference paper at ICLR 2026
B.2 IDENTIFIABILITY OF FEATURE DIFFERENCES
Proposition 2 (Identifiability of CST feature differences). When Assum. 2 and continuous
encoder Φ: RD + E and linear classifier Z globally minimize the cross-entropy objective
then the composition h Φ o is a linear map from Sa-1 to R. i.e. o(d') o = A |s where 1dd is linear map.
Proof. The proof follows by the straightforward application of the proof technique (Reizinger et al. 2024a
Thm. 1).
To provide a high-level overview of how we have arrived this result, we state our thought process:
1. We establish that the assumptions from Reizinger et al. (2024a) hold for CSF;
2. Our insight to connect the contrastive objective to the state differences that the CSF loss has the same parametrization as the one in (Reizinger et al. (2024a) with the substitution
z = (s s') (z is the notation of RReizinger et al. (2024a), and does not refer to skills);
3. Then apply the identifiability result of (Reizinger et al. (2024a) ). which proceeds by arguing about the Bayes optimum of cross entropy minimization. For this, the important substeps are:
(a) Exploiting the symmetries of the inner product parametrization;
(b) Using skill diversity (required to construct invertible matrix solve linear equation system to express the learned features in terms of the ground-truth states;
4. This gives us linear identifiability of (s - 5') yielding the formula (ss-')) s') The linear map in Prop. 2 is the same for all (unil-normalized) states. Combined with the linear parametrization, this implies that the feature differences are also identified up to linear transforma-
tion:
Proposition 3 (Feature identifiability in CSF Prop. 2 implies by the inner product parametrization of 4(2(s, s') that s(o) = As and ((o') As' up to linear transformation.
with the same A, thus, the features are also identified
Proof. The proof follows from the linear parametrization of the model. By linear identifiability of the feature differences (as they lie S
SK-1 an offset not possible), we have A
((0')
[As As)
B.3 SUBOPTIMALITY OF THE MAXIMUM ENTROPY POLICY
Proposition 4. [A maximum-entropp policy in CSF is not diversel A maximum entropy skill- conditioned policy 2) Uniform is not diverse and cannot maximize the reward Es,z,a (Ø(0') <(0)]
Indirect. Fix the initial state and assume that the skill-conditioned policy has maximum entropy. i.c., it follows a uniform distribution and maximizes the reward-given that the policy network is sufficiently flexible to express such policy. This implies that the expectation over the actions does not depend on 2, yielding in expectation the same [á(o') xo)) for each skill.
Ta (0(0), o(o')) Es,,,a (Ø(o') o(o) 2
(6)
(o(o') o(o)] 2p(s']s, (0lo
(7)
In this case, the skill-conditioned policy becomes independent of z;, the uniform distribution over the action space has maximum entropy. Substituting <(alo, 2) =aalo) yields
[Ø(o') - p(o)] pps (ddddd
(8)
8,2.0
and by reordering the terms, we get
/
[á(o') [(o] хπ =T(a)))p(s') 9,2.0
(9)
Note that p(s'ls,a) (alo), and p(s) are independent of 2;, thus, (o(o') x(o) are also independent of 2, As a skill vector parallel to [Ø(o') - x(o) maximizes the inner product, and the skills are on the unit hypersphere, this yields unique solution. However, the skills are drawn uniformly from the unit hypersphere, they are distinct. That is, the inner products will differ for 2; # 2zzi- Thus, both skills cannot maximize the reward, leading a contradiction.
18

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL DETAILS
C.1 COMPUTE RESOURCES
All experiments were run in a compute cluster using an Intel Xeon Gold CPU (16 cores, 2.9 GHz) and NVIDIA RTX 2080 Ti GPUs and used most 48 GB of RAM. No experiment required more than 3 days of runtime. In total, our experiments took 0.4 GPU years.
C.2 HYPERPARAMETER SEARCH
To train the CSF method in the Ant, Half Cheetah, Quadruped State, and Quadruped Pixel envi- ronments, we used the hyperparameters included the GitHub repository of [Zheng al. (2025) as a starting point and modified (i) the encoder ø's backbone architecture to aid identifiabili lity by introducing skip-connections, (ii) the latent space dimensionality dimensionality, (iii) the trade-off factor between the two factors of the contrastive loss (Zheng et al., 2025] Eq. (10)& & paragraph below), and (iv) the number of negative samples the contrastive loss. We found the CSF method to be sensitive to the latent space dimensionality. but appropriate choice of & mitigated this sensitivity and encouraged learning. Increasing the number negative samples in the contrastive loss led to performance gains in some environments (w.r.t. state space coverage and oracle return), which is intuitively explained by requiring more samples to cover the latent space well. In experiments that varied the number of skills in the discrete skill set, we kept all other hyperparameters fixed to the continuous sampling case. In the experiments varying the latent space dimensionality, the other hyperparameters were fixed to those of dim 29.
match the ground truth state's
For the exact hyperparameter configurations, see the code in the supplementary material.
C.3 EVALUATION METRICS DETAILS
Coverage. We define our coverage indicators following Zheng et al. (2025). In the Ant, Half Chectah, Quadruped, and Robobin environments, we define coverage as the number of unique integer-discretized coordinate vectors observed across all evaluation trajectories (using different skills). In the Kitchen environment, we define coverage as the sum of binary success indicators across all six kitchen tasks (BottomBurner, LightSwitch, SlideCabinet, HingeCabinet, Microwave, Kettle). Intuitively, the agent requires a diverse skillset to achieve broad coverage.
Relative coverage. To make coverage comparable across environments, we define relative coverage the coverage normalized by (an estimate of) the best attainable coverage. We use empirical estimate by choosing the maximum coverage from either our experiments or from coverage results in the literature (when available).
Oracle return. We define the oracle return the highest cumulative discounted return attained across all evaluation trajectories. Intuitively, this is the return the skill-conditioned agent would achieve if oracle shared the best skill for the task before the rollout. In a meaningfully diverse skillset, it is likely that there exists skill that solves the task defined via the (PO)MDP.
depeedeedd R score CDR A limitation of the R2 score is that can obtain a high value even if the agent has not explored the environment well. Namely, if the ground-truth latents from all visited states are recovered up to a linear transformation but the visited states only form small subset of all states, then we cannot expect good downstream performance from our agent. To make the Rt_ score distinguish between collapsed (no exploration) and well-explored scenarios, peveerageddeendeentt R² score, which defined as the harmonic mean of the relative we introduce C where C is the coverage score and Cmax is the highest attainable coverage the coverage (C := Cass
environment) and the
Rd
score. That is,
2CR²
CDR
Ö + R2
We choose the harmonic mean and not the arithmetic or geometric means, as we want to assign low score to an agent that has either a low R2 score or poor coverage, and out of the three means, the harmonic mean punishes low values most severely. If either component is near zero, the harmonic mean will be near zero regardless of how high the other component is.
Our plots showing the CDR score also serve comparison against a random baseline, demon- strating the shortcoming of not explored the state space but CDR is close zero
Namely, at the start of training, R² might be high as the agent has
19

---

Published as a conference paper at ICLR 2026
Half Chectah
Ant
Quadruped State
1.00
1.000
6(
R
0.98
0.992
d(o)
0.981
0.96
o(o)
o
((a)
0.0
0.5
1.0
6
x10
x 10
X10
Coverage
1600
800
100
800
400
0
1
2
0.0
0.5
1.0
4
X10
x X10
1
CDR2
0.8
0.4
0.4
0.1
d(o)
olo)
0.0
8(0)
0.0
0.0
x10
x 10
Oracle Rot.
300
2000
800
250
400
200
Env. steps
x 1
Env. steps
x 10
Env. steps
x x
Figure D.1: CSF identifies the underlying states in MuJoCo and DMC up to a linear transfor-
mation. First: Identifiability of both features ((o) and feature differences é(o') - (()), measured by the R score (higher is better); Second: state coverage, indicating exploratory behavior; Third:
coverage-dependent It" score, measuring both state exploration and state identifiability; Fourth: ora-
cle return indicating zero-shot task transfer performance. Error bars represent standard deviation.
C.3.1 NUMERICAL INSTABILITY IN CALCULATING
R SCORES
We investigated the discrepancy between the raw Rtd score of embeddings 6(o) and single-step embedding differences (ø(o') (())], and found that the reason numerical instability, which caused by two factors:
1. some dimensions in the ground-trute truth states having no or extremely low variance during a rollout, leading to extreme sensitivity in the R score even in double precision, and
2. the neighboring states having extremely small differences some environments, leading to small numerical imprecisions inflicting significant relative errors.
We have resolved (i) by filtering constant or extremely low-variance (1 10-$) state dimensions, and (ii) by calculating the differences over multiple time steps in some environments for stable and meaningful results. Fig. 3 applies these fixes to the R² scores.
D FURTHER RESULTS
D.1 CORRELATION ANALYSIS FOR PERFORMANCE METRICS
Tab. D.I shows the shortcoming of R² distinguishing collapsed and well-explored scenarios R vs
Coverage); furthermore. is also not indicative of the oracle return R vs Oracle). However,
CDR"
more indicative of the oracle return, especially in pixel-based environments CDR
Oracle).
20

---

Published as a conference paper at ICLR 2026
Quadruped Pixel
Kitchen
Robobin
o(o)
0.96
o(o')
s(o)
0.98
R²
0.80
*+ plo)
p(o)
0.80
olo
2
3
2
10
210
107
4000
Coverage
200
3.0
2000
100
1.5
2
10
x10
X
0.8
0.8
CDR
0.4
0.4
((v))
o(
4(c)
o(o
0.0
x
x10*
x
Oracle Ret.
180
10
150
-48
2
Env. steps
x
Env. steps
x10
Env. steps
x10
Figure D.2: CSF identifies the underlying states MuJoCo and DMC up to a linear transfor- mation. First: Identifiability of both features @(o) and feature differences Φ(o') 6(0). measured by the R score (higher is better); Second: state coverage, indicating exploratory behavior; Third: coverage-dependentt/ Rt score, measuring both state exploration and state identifiability: Fourth: ora- cle return indicating zero-shot task transfer performance. Error bars represent one standard deviation.
21

---

Published as a conference paper at ICLR 2026
Environment
CDR 48 Oracle
R
vs Oracle
R vs Coverage
Half Cheetah
0.9261
0.7095
00.7888
Ant
0.9441
0.3465
0.1331
Quadruped State
0,9673
-0.3623
-0.2395
Quadruped Pixel
0.2068
0.5348
-0.7623
Kitchen
0.4125
0.2092
-0.5005
Robobin
-0.006
-0.0708
-0.336
Table D.1: Correlation analysis of the pairs of metries from Figs. D.1 and D.2 the results pinpoint the insufficiency of [ta to predict good zero-shot performance on the oracle return, as it does not consider the coverage of the state space. CDR on the other hand, more indicative of the oracle return, especially in state-based environments.
D.2 THE EFFECT OF SKILL DIVERSITY ON HERO-SHOT PERFORMANCE
Fig. D3 shows the counterpart of Fig. where the zero-shot skill transfer performance is evaluated instead of state space coverage. The result indicates that Assum. ii) i.e., having a sufficiently diverse of skills, seems be necessary for good downstream performance.
1200
1.0
Oracle Return
R
1000
Oracle Return
0.8
800
0.7 22
600
0.5
200
0.4
5
29
500
sampling
Number of skills
Figure D.3: The effect of skill diversity on state identifiability and zero-shot skill transfer in the Ant environment. Skills are sampled from p(z) at the start of pretraining and kept fixed throughout, except 'sampling' case where skills are redrawn from p(z) during training, emulating an infinite set of skills. An insufficient number of skills violates Assum. i), leading to both weaker zero-shot skill transfer and lower R score.
D.3 THE EFFECT OF LATENT SPACE DIMENSIONALITY ON COVERAGE
Fig. sh4ss a variant of Fig. with state space coverage reported instead of zero-shot skill transfer performance. For linear identifiability, dimensions as the ground-truth states-as required by Assum. v This does not necessarily mean that the features do not capture (all) information about the ground-truth states; it only means that the information cannot be decoded linearly to reconstruct all ground ground-truth states. With increasing latent dimensionality, coverage tends decrease, primarily because the CSF algorithm was designed for lower latent dimensionality.
required that the feature space has at least as many
D.4 MARGINAL G-TESTS AND HISTOGRAMS FOR VERIFYING FEATURE CONCENTRATION ON THE HYPERSPHERE
We have verified Assum. ii) for all environments we evaluate on using a series of G-tests on the marginal histograms (i.e., the 1D histograms corresponding to individual dimensions in the latent space). We tested for Gaussianity, as high-dimensional Gaussian random variables concentrate on corresponding hypersphere. We then investigated whether the number of nominal 5% rejections was larger than chance would produce all marginals were truly Gaussian (via a one-sided binomial test with rate 0.05). In all environments except Half Cheetah, this excess-rejection test (all p > 0.05), meaning we found no environment with a violation of Gaussianity. In Half Cheetah, observed some latent dimensions with bimodal marginals next to Gaussian-like ones.
significant
22

---

Published as a conference paper at ICLR 2026
1.0
2800
0.8
Coverage
2600
dim
2400
Coverage
0.2
R
0.0
2.
29
Latent space dimensionality
Figure D.4: The effect of latent space dimensionality on state identifiability and coverage in the Ant environment. Linear identifiability requires that the feature space has at least as many dimensions as the true state (cf. Assum, v))).
We include 4-4 histograms of randomly sampled feature dimensions for the Half Cheetah (Fig. D.7 Kitchen (Fig. D.5), Quadruped (Fig. D.6). and Robobin (Fig. D.8) environments These figures, excluding the single exception of feature dimension in the Half Cheetah environment (Fig. indicate that Assum. holds across multiple environments.
320
972
803
203
#60 602
2100
060 [k
(a) Feature dimension
(b) Feature dimension
2202
022
600
420
405
202
(c) Feature dimension 14
(d) Feature dimension 21
Figure D.5: Marginal histograms for four randomly sampled feature dimensions in the Kitchen environment: the histograms clearly show that Assum. ti) holds across multiple environments.
D.5 IDENTIFIABILITY OF OBJECTS WITHIN STATES
To measure whether the representation captures the task-relevant objects (instead of only the agent's s proprioception, which is easier to reconstruct), we also report an "object R² score", computed on the
*Refer /heng et al. 2025 Fig. for histograms in the Ant environment.
23

---

Published as a conference paper at ICLR 2026
2000
1500
2250
1000
200
2.0
-25
-0.5
0.5
3.5
4677- 400 -11e
A61
(a) Feature dimension
(b) Feature dimension
2303
1400
2253
320
303
200
253
-25
3.0
2210m33
(c) Feature dimension 14
(d) Feature dimension 35
Figure D.6: Marginal histograms for four randomly sampled feature dimensions in the Quadruped environment: the histograms clearly show that Assum. 2 holds across multiple environments.
24

---

Published as a conference paper at ICLR 2026
600
432-
800
332
200
0.00
-2.50
0.25
0.50
807
-210mm
(a) Feature dimension 3
(b) Feature dimension
1202
650
Qutt
300
802
100
202
-3.0
0.0
1.5
(xC)
-00-- -2idbe
-210m
(c) Feature dimension 12
(d) Feature dimension 27
Figure D.7: Marginal histograms for four randomly sampled feature dimensions in the Half Cheetah environment: the histograms, excluding the single exception of feature dimension 3, show that Assum. 2 holds across multiple environments.
25

---

Published as a conference paper at ICLR 2026
3900
120
2500
Couk
500
200
-25
-50
-2.5
0.0
2.5
20.0
125
(a) Feature dimension
Feature dimension
2503
1502
2303
303
46-2- 2014-1 pin ot
#x1-w0-220 [8ma)
(c) Feature dimension 2
(d) Feature dimension 8
Figure D.8: Marginal histograms for four randomly sampled feature dimensions in the Robobin environment: the histograms clearly show that Assum. 2 ii) holds across multiple environments.
26

---

Published as a conference paper at ICLR 2026
Kitchen
Robobin
0.88
0.96
R
0.80
0.88
((0)
O(o)
0.80
p(e (60
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3
x 100
x 107
1.0
0.90
Object R
0.9
0.75
o(o) ((0" (te)
p(n o(o)
0.0
0.5
x 100
x10
Env. steps
Env. steps
Figure D.9: CSF identifies both the underlying object states and the full states in the Kitchen and Robobin environments up to linear transformation. Object states refer to the object-relatee subspace the full ground-truth state. First row: Identifiability of both features Φ(o) and feature differences (oo') ((0), measured by the R score (higher is better); Second row: The same metrics computed between the full features and the ground -ouun object states. For details, refer to Appx. D.5
object coordinates only. Robobin, the object slice (state dimensions 3-8) contains the 3D positions of the two cubes being pushed or picked, excluding the robot hand. In the Kitchen environment, the object slice (dimensions 11-29) covers all manipulated fixtures: the two stove burner knobs, the light switch, the slide and hinge cabinet doors, the microwave door hinge, and the kettle's ' 7-DoF pose. The strong object Rt values in Fig. D.9 shows that the encoder identifies the manipuland, not just the arm state, up to linear transformation.
D.6 GENERALIZATION OF THE ENCODER TO UNSEEN TRAJECTORIES
We evaluate the generalization of the learned encoder representations trajectories collected by independently trained SAC agents in two state-based and two pixel-based environments: Ant and Half Cheetah resp. Kitchen and Robobin. For each saved SAC snapshot, we feed its ground-truth states into the CSF encoder and report the R² score between the encoded latents and the ground-truth states. To separate interpolation from extrapolation, we split the SAC states into (i) covered states that fall into the same discretized bins as the states visited by our own method, and (ii) uncovered states that the evaluated CSF agent has covered its rollouts. The high R² scores on the subset covered in Fig. D. show the encoder faithful on already visited states, while performance on the uncovered subset directly measures generalization to expert trajectories that lie outside the representation' training distribution. For the Kitchen environment, there is no existing definition of state coverage; therefore, we only report the R22 score on the unpartitioned rollouts collected by an independently trained SAC agent.
D.7 DETAILED DEFINITION OF COVERAGE METRIC
We follow Zheng et al. 2025 in defining the coverage metric. Let rollouts be indexed by
1...., R, each as a list of latent states 81 (r) 8 For environment e, a projection P. selects specific state components and discretizer D. either rounds each selected component to two decimal
places or applies [.]. Define
R T. D. (P)(s
C. = U r=li= 1=1
27

---

Published as a conference paper at ICLR 2026
Ant
Half Cheetah
Kitchen
Robobin
d(o)
0.99
0.90
Rollout R
A(a)
1.00
0.96
0.98
0.96
0.81
0.90
0.96
* d(o? o( ((0)
0.78
0.84
0.0
0.5
1.0
0.0
2.5
5.0
X10
x10
X10
X10
N
SAC R
0.88
0.9
0.8
0.8
0.81
0.84
0.7
o((o) o(o)
6(6)
4(n)
p(n 6
0.80
1
x x100
x 10
SAC Covered R
0.96
V
0.90
0.90
0.88
N/A
0.84
0.84
d(o)
0.80
o(o)
0.78
+ S(o)
0.0
1.0
10
x10
10
SAC Uncovered R
1.0
+ 0(o)
M
o(o')
0.8
WY
N/A
* 0(a)
0.85
0.8
M.oio
0.80
oto
o(o)
0.4
0.0
0.5
1.0
Env. steps 10
Env. steps x10
Env. steps x107
Figure D.10:
CSF shows strong generalization to SAC trajectories that are uncovered by the current CSF agent and remains faithful on the covered trajectories. First row: The R² metric
between (0)) resp. <(o') (()) and the ground-truth state over rollouts collected by the CSF agent. Second row: The R² metric between the same quantities as the first row, but over observations that collected by independently trained SAC agents. Third row: Same the second row, but over only the observations that the CSF CSFJagent agent itself also visited during its evaluation rollouts, measuring in-distribution identifiability. Fourth row: Same the second row, but over the complement states of the third row; i.e., over states collected by the SAC agents that have not been visited by the CSF agent during the rollouts, measuring the generalizability of the encoder's identifiability. Note that the reported 1 scores on the y axes have different scales, For details, refer to Appx. D.6
28

---

Published as a conference paper at ICLR 2026
Half Cheetah
Ant
Quadruped State
MSE
0.06
o(o
6(n)
4(03
0.12
0.03
0.02
0.06
0.00
0.00
0.00
0.0
0.5
1.0
0.0
0.5
1.0
x10
x10*
x10
Quadruped Pixel
Kitchen
Robobin
0.16
MSE
0.02
A
0.012
0.08
0.01
0.006
ott
0.00
0.00
0.000
0.0
0.5
1.0
2
0.0
0.5
1.0
1.5
Env. steps
X10
Env. steps
x10*
Env. steps
x10
Figure D.11: CSF accurately reconstructs the ground-truth state from the CSF representation when measured by the MSE metric, not only w.r.t. the R² score. For details, refer to Appx. D.8.
State coverage is defined Coverage, = |Cel-
Half Cheetah. Pe keeps the forward state coordinate of the root body of the half-cheetah. De applies [-]. Coverage counts unique floored scalars Ce.
Ant. P. keeps the planar (z. y) state coordinates of the ant's body. De applies L-J. Coverage counts unique floored 2-dimensional state vectors in Ce.
Quadruped. Pe keeps the planar (I,, y) state coordinates of the quadruped's ' body. D. applies [.]- Coverage counts unique floored 2-dimensional state vectors in Ce-
Robobin. Pe keeps nine state components: the robotic hand and the two objects. D. rounds each kept component decimal places. The coverage counts unique 9-dimensional rounded state vectors in Ce.
Kitchen. the Kitchen environment, we define coverage differently, following Zheng et al. (2025).
The Kitchen environment contains six tasks: Bottom Burner, Light Switch, Slide Cabinet, Hinge Cabinet, Microwave, and Kettle. The environment logs binary success flags for each task per rollout. The Kitchen task coverage the number of tasks solved in at least one rollout.
D.8 MSE RESULTS
In this section, investigate whether the approximate states recovered from the encoder & by linear probe W are close to the ground-trute -ruth states according to the MSE metric. Concretely, we freeze the encoder and fit linear probe W that predicts the ground-truth simulator state from the learned representation (()) via & = W ((0), just like for computing the R²2 score. We then compute the MSE between s and using the formula
N
MSE-
8ij
Nd
where N is the number of test samples and d is the state dimensionality. The low errors in Fig. D.I1 show that a simple linear decoder can accurately reconstruct the ground-truth state from the CSF representation when measured by the MSE metric, not only w.r.t. the R² score.
29

---

Published as a conference paper at ICLR 2026
E ACRONYMS
ELBO evidence lower bound
POMDP
partially observable Markov Decision Process
CL Contrastive Learning
CRL Causal Representation Learning
RL Reinforcement Learning
CSF Contrastive Successor Features
RV random variable
DGP data generating process
SSL self-supervised learning
ICA Independent Component Analysis
USD unsupervised skill discovery
MDP Markov Decision Process
MI Mutual Information
MISL mutual information skill learning
MMF von Mises-Fisher
F
NOMENCLATURE
CDR coverage-dependent coefficient of deter-
R coefficient of determination
mination
S hypersphere
30


Reviews:
[{"review_id": "0miH42obEL", "reviewer": "Reviewer_UPeg", "summary": "Despite the popularity of mutual information skill learning (MISL) methods in RL, they are not theoretically well understood. This paper provides an explanation of why MISL methods work by drawing from the theory of non-linear independent component analysis (ICA). Specifically, the paper draws a connection between POMDPs and data generating processes in ICA, analyzes the feasibility of assumptions required for ICA results, and proves that the features learned by one particular MISL method, contrastive successor features, can be used to recover the underlying states of the (PO)MDP up to linear transformation. The paper also explains the utility of some algorithmic design choices from the ICA perspective. The paper further verifies the theoretical results by experiments in MuJoCo and Deep-Mind Control environments.", "questions": "- Do you foresee that some of the assumptions that rely on empirical observations (e.g., assumptions ii, iii, and iv) will hold for other MISL methods?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper makes a novel connection between MISL methods and non-linear ICA.\n- The paper proves identifiability results for CSF, providing insight into why the method succeeds and why certain design choices are better.\n- The paper provides empirical results to back up the theoretical insights in a number of environments.\n- The paper opens up a new direction of research for understanding self-supervised RL methods, and can be of wide interest to the RL community.", "weaknesses": "- The paper is dense and can sometimes be hard to follow.\n\t- For example, in Section 2.3, where the paper draws a connection between MISL and DGP, it was initially unclear to me how skills fit into the picture. It was mentioned in Section 2.1 that skills can be viewed as auxiliary variables, which can be brought up here again to aid explanation.\n\t- Perhaps due to a limitation in space, there's almost no spacing between some paragraphs.\n\t- There can be more discussion on the technical details of ICA in the background. This can make the paper more self-contained.\n- The feasibility of some of the assumptions made by the authors to prove identifiability for CSF relies on empirical observations in a previous paper (Figure 2 of Zheng et al., 2024). However, it seems that Figure 2 of Zheng et al. (2024) presents results for METRA, not CSF.\n- Line 478-480: \"As exploration and state identification showed a positive correlation with the extrinsic oracle return, defined by each environment, this suggests that identifiability is helpful for zero-shot task transfer.\" But in Figure 3, there does not appear to be a correlation for the columns 3-5.\n- References can be better polished. There exists papers without venue names, and some papers have the arXiv version cited when they were in fact published in conferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yoNZ1CD4Gi", "reviewer": "Reviewer_o5Lj", "summary": "This work aims demonstrate that the causal features of an environment can be identified using contrastive successor features from up to a linear transformation. This forms an identifiability guarantee for unsupervised RL in the POMDP setting, so that the true states are derived from observations. Then, the work demonstrates empirically that the ground-truth features in several domains are recovered using CSF.", "questions": "It might be easier to read a slightly less aggressive citation style (the green and red boxes inserted by the citation links make the make it hard to parse the introduction). \n\nThe second paragraph of the introduction is unclear: How does the natural connection to causality blur the borders between these fields? Interconnection does not produce a distinction between decisionmaking with scalar rewards (RL) and cause-effect models. \n\nTraditionally, unsupervised skill discovery is in the setting of fully observed MDPs rather than partially observed, so the focus on identificability results is somewhat jarring. This focus could be introduced a bit earlier rather that in the description of ICA.\n\nDoes the definition of diversity (definition 1) really capture the meaning? It seems like diversity should also cover some description of the breadth of skills, but in this case any pair of skill parameters is sufficient for diversity. Maybe a term such as distinguishable is more appropriate.\n\nWhile CSF is a meaningful algorithm, is it really \"prototypical\" (l183) of MISL methods? In many cases it is closer to a representation learning algorithmm with a skill parameter added to it, rather than a MISL algorithm which often has the representation learning component built into the skill learning.\n\nIt makes sense that the assumptions are required to guarantee identifiability, but in order to make the claim that identificability is robust to assumption violations in practice it does not seem sufficient to simply claim success in other fields, since RL is a significant different setting, especially since the agent is part of the data generating process. Is there some stronger evidnece to suggest robustness to assumption violations, especially theoretically.\n\nTHe fact that Empirical observations show that the fatures of consecutive observations are close is not entirely convincing universally, since in many cases with locomotion tasks that these algorithms are tested on, the observations are often visually close together as well (an ant moving around will have similar appearances), so this closeness could just be a consequence of this. \n\nWhile it is certainly the case that I(s,s';z) is more stable to optimize, it seems like this state difference may not always be the best choice for representing a variety of different policies, since identifying policies from their state difference means that it can be hard to distinguish when policies should start changing their direction.\n\nThe experimental domains are sufficeint for demonstrating that the underlying state is discovered, though they are certainly not complete. In particular these environments focus on motion planning with various morphologies, not on the manipulation of different eleemtns in the domain, of whose underlying state might be much harder to describe, especially when randomly initialized.\n\nIs the true underlying state really the state of the joints in mujoco, since to some extent this also is a layer of abstraction abouve the length of the joints or the shape of the agent.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work applies an elegant description of identificabiltiy in a novel context.\n\nThe theoretical framework is well articulated and provides clear reasons advantages.\n\nThe empirical results are sufficient to support the theoretical claims", "weaknesses": "The empirical results are somewhat limited in scope, considering the extension to the POMDP setting\n\nThe work is not particularly self contained, in that many of the claims are fully described in the appendix.\n\nCSF is not the most representative MISL algorithm because it detaches teh representation learning from the policy learning more than most methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F1DSlPmrr4", "reviewer": "Reviewer_vUpn", "summary": "This paper investigates identifiability in mutual information skill learning and contrastive successor features (CSF). The authors provide some evidence for the ability of CSF representations to recover the ground truth state used to produce the CSF because of the inner product parameterization of the loss. Further, the paper investigates the theoretical merit of different instantiations of the MISL objective, whether to use $\\phi(s)^Tz$, $[\\phi(s) - \\phi(s')]^T$, or $[\\phi(s_0) -\\phi(s)]^Tz$. Finally, the authors provide some identifiability results on common RL tasks in the Deep Mind control environments.", "questions": "Please refer to the weaknesses section for my questions about the work.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is well motivated and well written. Its objectives are clear and, to my knowledge, provide the first analysis of ground truth identifiability using mutual information skill learning (MISL) losses. The paper introduces the notion that a set of diverse skills and an inner product parameterization are necessary for learning a robust representation that provably recovers the ground truth state.", "weaknesses": "There are several weaknesses that exist are present in the paper that must be addressed. \n\n## Major Weaknesses\n1. **Reality of assumptions**: It is not clear that the assumptions made in the paper are representative of reality. Namely, is it common that \"each state difference is equiprobable\"? What is the support for this claim?\n2. **Transitions are typically not skills**: The authors also assume that \"each pair of consecutive states is a skill\". I believe that this is not a typical definition of skills. Skills are often defined on a longer time horizon and so single-step transitions are not wholly descriptive of what is traditionally called a skill. Further, there is literature that demonstrates that the controllable state (i.e., features that are affected by actions) are not identifiable using single-step transitions (see [1]). With that in mind, i believe it is not reasonable to define skills as (s,s') pairs.\n3. **Lacking explanation for necessity of inner product parameterization**: The authors repeatedly claim that the inner product parameterization is important for identifiability, but do not share any evidence towards this end. Is there a specific result that they found that would indicate this? Or is this notion sourced from previous literature?\n4. It is not clear to me why $I(s; z)$ or $I(s_0, s; z)$ are reasonable alternatives to $I(s, s'; z)$ as the former two information quantities would imply things about the visitation distributions alone. Is there a valid reason to consider these as alternatives to $I(s, s'; z)$?\n\n### Minor Weaknesses\n\n1. The authors claim that  $\\phi(s) - \\phi(s')$ is typically a learning target (line 161). Can the authors point to a method that uses such a construction? Are the authors are referring to metric-based methods like value implicit pre-training where $\\|phi(s) - \\phi(g) \\| $ is used?\n2. It believe it would be good to see actual MSE on extracted states as opposed to the correlation coefficients. In my experience, it is very possible to extract the ground truth state from encoded images in DM Control.\n### Nitpicks\nLine 267 has colon at end of line", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sHmzmAi0di", "reviewer": "Reviewer_YV3A", "summary": "This paper offers an explanation for the effectiveness of skill-learning methods based on mutual information (MISL). The authors provide both theoretical and experimental analyses of how the representations learned by a representative approach (CRL) of MISL relate to the true latent factors of the environment. Their results show that, under certain assumptions, CRL implicitly learns representations that are linearly related to the environment’s underlying latent state.\n\nI believe the paper touches an important topic and provides an interesting and novel analysis on the representations leaned by skill learning methods. However, the writing could be improved and the analysis could be strengthened to support the claims.\n\nWriting:\n- The introduction could be made easier to follow. Some parts would fit better in the related work section, and long citation lists should be avoided. The connection to causality is not entirely clear, and lines 81–89 do not seem directly related to the paper’s main motivation.\n- The explicit definition of identifiability should appear earlier in the manuscript to help readers grasp the theoretical framework. The discussion in the introduction is not sufficiently explicit.\n- I recommend separating the related work from the background section to improve readability and structure.\n- Line 156: This paragraph lacks clarity and could benefit from rephrasing or additional context.\n- Definition 1 (line 171): The proposed definition seems closer to “distinguishability” than “diversity.” Two skills may produce very similar trajectories yet still represent different behaviors. The relationship between this definition and the notion of diversity used in Assumption 1 should be clarified.\n- The statement “Each pair of consecutive states corresponds to one skill” introduces a strong and limiting assumption. This is unlikely to hold in many environments. For instance, an agent might need to traverse a corridor before exploring an area, or operate in an MDP with discrete actions and deterministic transitions. A more thorough discussion of such edge cases would strengthen the paper. Furthermore, the connection to the statement line 293 is unclear.\n- In Figure 2 (right), the direction of the arrows is ambiguous and should be clarified.\n- First paragraph of Section 3.2. I do not catch why z and feature representations could be antipodal by maximizing their dot product.\n- In Section 3, the example at line 241 is particularly helpful and could be emphasized earlier in the section. However, the statement \"to distinguish data under different distribution shifts or interventions\" is not \"intuitive\".\n\nExperiments:\n- In Figure 3, using features or feature differences do not induce meaningful difference in identifiability. This should be discussed. Overall, it lacks a baseline that gives indications of what is a high identifiability.\n- The authors should formally define state coverage in the manuscript. \n- The authors rightly note that the R2 score is influenced by state coverage, but the CDR² curve remains difficult to interpret and, in practice, seems to primarily reflect coverage rather than the main contribution of this work.To clarify this point, the authors should compute the R2 score over a set of presampled agent positions, either manually defined or extracted from trajectories of trained DRL agents. Optionally, they could separately evaluate the R2 for covered and uncovered states to analyze potential generalization properties of the learned representations.\n- From Figure D1, it seems relatively easy to achieve high identifiability in these experiments. The metric starts very high, drops sharply when learning begins, and then rises again. This pattern raises concerns that learning state-related representations might be too simple in these environments, and therefore not representative of more complex scenarios. Some environments include objects, i.e. parts of the true state that are relatively difficult to manipulate and represent. To strengthen the paper’s validity, the authors should include a correlation coefficient that relates to only these parts of the state space and discuss the result with respect to the ability of the agent to reach these states.", "questions": "Please, see above.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper makes an original contribution, supported by both theoretical and experiment results.", "weaknesses": "The writing should be improved and the analysis should be strengthened.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)