Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning
Authors: Chongyi Zheng
Jens Tuyls
Joanne Peng
Benjamin Eysenbach
Venue: ICLR 2025 Oral (2025)
Keywords: unsupervised learning
reinforcement learning
mutual information
successor feature
Primary area: reinforcement learning
TL;DR: Through careful analysis of a prior method, we develop a new method called Contrastive Successor Features (CSF) that illustrates mutual information skill learning can be made highly effective.

Abstract: Self-supervised learning has the potential of lifting several of the key challenges in reinforcement learning today, such as exploration, representation learning, and reward design. Recent work (METRA) has effectively argued that moving away from mutual information and instead optimizing a certain Wasserstein distance is important for good performance. In this paper, we argue that the benefits seen in that paper can largely be explained within the existing framework of mutual information skill learning (MISL).
Our analysis suggests a new MISL method (contrastive successor features) that retains the excellent performance of METRA with fewer moving parts, and highlights connections between skill learning, contrastive representation learning, and successor features. Finally, through careful ablation studies, we provide further insight into some of the key ingredients for both our method and METRA.

Published as a conference paper at ICLR 2025
CAN A MISL FLY? ANALYSIS AND INGREDIENTS FOR MUTUAL INFORMATION SKILL LEARNING
Chongyi Zheng, Jens Tuyls", Joanne Peng & Benjamin Eysenbach Department of Computer Science Princeton University {chongyiz, jtuyls s (css www.ss.rinneoo.noo edu
ABSTRACT
Self-supervised learning has the potential of lifting several of the key challenges in reinforcement learning today, such as exploration, representation learning, and reward design. Recent work (METRA (Park et 2024)) has effectively argued that moving away from mutual information and instead optimizing a certain Wasser- stein distance important for good performance. In this paper, we argue that the benefits seen in that paper can largely be explained within the existing framework of mutual information skill learning (MISL). Our analysis suggests new MISL method (contrastive successor features) that retains the excellent performance of METRA with fewer moving parts, and highlights connections between skill learn- ing, contrastive representation learning, and successor features. Finally, through careful ablation studies, we provide further insight into some of the key ingredients for both our method and METRA
INTRODUCTION
Self-supervised learning has had a large impact on areas of machine learning ranging from audio
processing (Oord et al. 2016, 2018) computer vision (Radford et al. 202T
Chen et al. (2220) to
natural language processing (Devlin et al. 2019 Radford & Narasimhan 2018 Radford et al. 2019
Brown 2020). In the reinforcement learning (RL) domain, the "right" recipe apply self-supervised learning is not yet clear. Several self-supervised methods for RL directly apply off-the-shelf methods 2022). but have achieved limited
from other domains such masked autoencoding (Liu et al.
success so far. Other methods design self-supervised routines more specifically built for the RL setting (Burda et al. 2019 Pathak al. 2017 Eysenbach et al. 2019 Sharma al., 2020 Pong ct al. 2020). We will focus on the skill learning methods, which aim to learn a set of diverse and distinguishable behaviors (skills) without an external reward function. This objective is typically formulated as maximizing the mutual information between skills and states (Gregor et al. 2016
Eysenbach et al. 2019), namely mutual information skill learning (MISL.. However, some promising recent advances in skill learning methods build on other intuitions such as Lipschitz constraints (Park ctt al. 2022) or transition distances (Park et al. (023). This paper focuses on determining whether the good performance of those recent methods can still be explained within the well-studied framework of mutual information maximization.
METRA PPar al.) (024). of the strongest prior skill learning methods, proposes maximizing the Wasserstein dependency measure between states and skills as an alternative to the idea mutual information maximization. The success of this method calls into question the viability of the MISL framework. However, mutual information has a long history dating back to Shannon (9988) and gracefully handles stochasticity and continuous states (Myers et al.) 2024). These appealing properties of mutual information raises the question: Can we build effective skill learning algorithms within the framewrka or MISL fundamnttlll flawed?
We start by carefully studying the components of METRA both theoretically and empirically. For representation learning, METRA maximizes a lower bound on the mutual information, resembling
"These authors contributed equally.
Website and code: httpt:tt///......... / wwtporriintttttrrreee itttttaatttooottattttatttiit ive ve- successor features

---

Published as a conference paper at ICLR 2025
contrastive learning. For policy learning, METRA optimizes mutual information term plus extra exploration term. These findings provide an interpretation of METRA that does not appeal to Wassertein distances and motivate simpler algorithm (Fig. D
Building upon our new interpretations of METRA, we propose a simpler and competitive MISL algo- rithm called Contrastive Successor Features (CSF). First, CSF learns state representations by directly optimizing a contrastive lower bound on mutual information, preventing the dual gradient descent procedure adopted by METRA. Second, while any off-the-shelf RL algorithm (e.g. SAC (Haarnoja et al. 2018) is applicable, CSF instead learns a policy by leveraging successor features of linear rewards defined by the learned representations. Experiments on six continuous control tasks show that CSF is comparable with METRA, as evaluated on exploration performance and on downstream tasks. Furthermore, ablation studies suggest that rewards derived from the information bottleneck as well as a specific parameterization of representations are key for good performance.
Key Takeaways
1. METRA can be explained within the MISL framework: learning representations through maximizing MI (Sec. 4.1). and learning policies through maximizing MI plus an exploration bonus (Sec. 4.2)
2. Based on our understanding of METRA, we propose CSF (Sec. 5) a simple MISL algorithm that retains SOTA performance (Sec. 6.4).
3. We find several ingredients that are key to boost MISL performance (Sec. 6.3
RELATED WORK
METRA
Contrastive Successor
Through careful theoretical and experimental analysis, we develop a new mutual informa- tion skill learning method that builds upon con- trastive learning and successor features.
Features (CSF)
max max (2)2
max RS, S:
(Mutual Information)
max 4,6222
max KS, Z) HSSSS ((5) (KS)
(eformatio Bottleneci)
Unsupervised skill discovery. Our work builds upon prior methods that perform unsupervised skill discovery. Prior work has achieved this aim by maximizing lower bounds (Tschannen al. 2020) Poole et al. 2019) of different mu- tual information formulations including diverse and distinguishable skill-conditioned trajecto- ries (Li et al.
Optimize
Contrastive leamiing
Wasserstein
connects (S) and 7
distance with temporal
Successor Features optimizo r= (4K5) #55)
Figure From METRA to MISL. (Left) METRA argues optimizing a Wasserstein distance is superior to using mutual information. (Right) Through careful anal- ysis, show METRA still bears striking similarities MISI. algorithms, which allows us to develop a new MISL algorithm (CSF) that matches the performance of METRA while retaining the theoretical properties associated with MI maximization.
2023 Eysenbach et al., 2019
Hansen et al.
2020 Laskin et al., 2022) Strouse
al. 2022). intrinsic empowerment (Mohamed
& Jimenez Rezende 2015; Choi et al.) (2021). distinguishable termination states (Gregor et al.
2016
Warde-Farley et al. (201)) Baumli et al.
2021 entropy bonus (Florensa et al. [2016] Lee
ct al. 2019 Shafiullah & Pinto 2022), predictable transitions (Sharma et al. 2020 Campos et al.
2020),
etc. Among those prior methods, perhaps the most related works are CIC (Laskin al. 2022)
and VISR (Hansen et al. 2020). We will discuss the difference between them and our method in Sec. 5 Another line of unsupervised skill learning methods utilize ideas other than mutual informa-
tion maximization, such as Lipschitz constraints (Park et al. (022), MDP abstraction (Park & Levine
2023), model learning (Park et al.) 2023), and Wasserstein distance (Park et al. 2024 He al.) 2022). Our work will analyze the state-of of-te-of-the-ar method named METRA (Park et al. (0244) that builds on
the Wasserstein dependency measure (Ozair et al.) 2019), provide an alternative explanation under the well-studied MISL framework, and ultimately develop simpler method.
Contrastive learning. Contrastive learning has achieved great success for representation learning in
natural language processing and computer vision (Radford al. [021 Chen et al. 2020 Gao et al.
2021 Sohn 2016 Chopra et al., 2005] Oord et al. 22188) Gutmann & Hyvärinen 2010 Ma & Collins
2018
Tschannen et al. 2020). These methods aim push together the representations of positive
2

---

Published as a conference paper at ICLR 2025
pairs drawn from the joint distribution, while pushing away the representations of negative pairs drawn from the marginals (Ma & Collins 2018 Oord et al. 2018). In the domain of RL, contrastive
learning has been used to define auxiliary representation learning objective for control (Laskin et al.
2020 Yarats et al. (021), I. solve goal-conditioned RI. problems (Zheng et al. 2024 Eysenbach et al.
2022 2020 Ma et al.) 2023), and derive representations for skill discovery (Laskin al. [2022).
Prior work has also provided theoretical analysis for these methods from the perspective of mutual information maximization (Poole et al. 2019: Tschannen et al. 2020) and the geometry of learned representations (Wang & Isola (020). Our work will combine insights from both angles to analyze METRA and show its relationship contrastive learning, resulting in a new skill learning method.
Successor features. Our work builds on successor representations (Dayan 1993). which encode the discounted state occupancy measure of policies. Prior work has shown these representations can be learned on high-dimensional tasks (Kulkarni et al., 2016: ZZhng et al. 2017) and help with transfer learning (Barreto et al. 2017). When combined with universal value function approximators (Schaul al. 2015), these representations generalize to universal successor features, which estimates a value function for any reward under any policy (Borsa et al., 2019). While prior methods have combined successor feature learning with mutual information skill discovery for fast task inference (Hansen (et al., 2020) Liu & Abbeel, (202]). we instead use successor features to replace Q estimation after learning state representations (Sec.
3 PRELIMINARIES
Mutual information skill learning. The MISL problem typically involves two steps: (1) unsupervised pretraining and (2) downstream control. For the first step, we consider a Markov decision process (MDP) without reward function defined by states S, actions a E A, initial state distribution Po E A(S), discount factor 7 E (0, 1). and dynamics p S x A - (SS), where A(-) denotes the probability simplex. The goal of unsupervised pretraining is to learn a skill-conditioned policy π : S x 2 ~ (AA) that conducts diverse and discriminable behaviors, where 2 is a latent skill space. We use 8 : S Z + (AA) to denote the behavioral policy. We select the prior distribution of skills as a uniform distribution over the d-dimensional unit hypersphere p(z) = UNIF(Sd-1) (a uniform von Mises-Fise Fisher distribution (Wikipedia, 2024 and will use this prior throughout our discussions to unify the theoretical analysis.
Given a latent skill space Z. prior methods (Eysenbach et al.) 2019 Sharma et al. 2020; Laskin et al. 2022 Gregor et al. 2016, Hansen et al. (020) maximizes the MI between skills and states ("; Z) or the MI between skills and transitions ("(S, S¹; Z) under the target policy. We will focus on (S, S" Z) but our discussion generalizes to I (S; 2). Specifically, maximizing the MI between skills and transitions can be written as
max 1st ())
const.
max E
pmp(()) (o. [oggp" (z s')l smpp" (s (s/)).2)
where n (s+, 8 2) is the discounted state occupancy measure (Ho& & Ermon 2016 Nachum et al. 2019 Eysenbach et al. 2022 Zheng et al. 2024) of policy x conditioned on skill z, and p" 8, the state transition probability given policy æ and skill z. This optimization problem can be casted into an iterative min-max optimization problem by first choosing a variational distribution q(2 s') to fit the historical posterior (= s, 8'), which is an approximation of p" (z 8, 8'), and then choosing policy ı to maximize discounted return defined by the intrinsic reward log q(> 8, 8'):
1k+1 <- arg max E (s.'', Dog ((= 8, (.,8)),
(2)
Mk+1 < arg max (a.0' xx) (
( ss, ss,'
(3)
where & indicates the number of updates. See Appendix A.l for detailed discussion.
For the second step, given a regular MDP (with reward function), we reuse the skill-conditioned policy x to solve a downstream task. Prior methods achieved this aim by (1) reaching goals in a
zero-shot manner (Park et al. 2022 2023 2024), (2) learning hierarchical policy Tr S ト A
that outputs skills instead of actions (Eysenbach et al. 2019:: Laskin et al.] [002 Gregor et al. 2016).
or (3) planning in the latent space with learned dynamics model (Sharma al. 2020).
3

---

Published as a conference paper at ICLR 2025
METRA. Maximizing the mutual information between states and latent skills [(S; Z) only encour- ages an agent to find discriminable skills, while the algorithm might fail to prioritize state space coverage (Park et al., 2024: 2022). A prior state-of- the-art method METRA (Park al. 2024) proposes to solve this problem by learning representations of states Φ S - IR via maximizing the Wasserstein dependency measure (WDM) (Ozair et al. 2019) between states and skills (w 7). Specifically, METRA chooses to enforce the I-Lipschitz continuity of & under the temporal distance metric, resulting constrained optimization problem for Φi:
max (a.")) (1)(s))))) d(s)) s.t. (1(((')) - ( S V(s, 8') E S uij
where (s, =) denotes the probability of first sampling from the discounted state occupancy measure (s+ = s 2) and then transiting to s' by following the behavioral policy 3. and S denotes the set of all the adjacent state pairs visited by B. In practice, METRA uses dual gradient descent to solve Eq. resulting in an iterative optimization problem
minmax max L(ó, X) X20
I(ó, X) 2 Eeterp
(())) + X (1 Epp(s, [llx(s') - -))))
(5)
Importantly, L(d, A) not the Lagrangian of Eq. because L(d, A) does not contain dual variable for every (s, 8') E S.nj We will discuss the actual METRA representation objective and the behavior of convergent representations in Sec. 4.1
After learning the state represennation 6, METRA finds its skill-conditioned policy æ via maximizing the RI. objective with intrinsic reward (o(*') (())) 2:
max (T),, ((T) A E (xppp()
[[0(6] - [+))
In the following sections, we will provide another way to understand this SOTA method, draw connections with contrastive learning (Oord et al. (018) and the information bottleneck (Alemi et al. 2017), and then derive simpler MISI. algorithm.
4 UNDERSTANDING THE PRIOR METHOD
In this section we reinterpret METRA through the lens of MISL, showing that:
1. The METRA representation objective is nearly identical to contrastive loss (which maxi- mizes a lower bound on mutual information). See S.1.4.1
2. The METRA actor objective is equivalent to a mutual information lower bound plus extra temm. This extra term is related to an information bottleneck (Tishby et al., 2000
Alemi
et al. 2017) and our experiments will show it is important for exploration. See Sec. 4.2
Sec. 5 will then introduce a new mutual information algorithm that combines these insights to match the performance of METRA while (!) retaining the theoretical grounding of mutual information and (2) being simpler to implement.
4.1 CONNECTING METRA'S 's REPRESENTATION OBJECTIVE AND CONTRASTIVE LEARNING
Our understanding of METRA starts by interpreting the representation objective of METRA contrastive loss. This interpretation proceeds by two steps. First, we focus on understanding the actual representation objective of METRA, aiming to predict the convergent behavior of the learned representations. Second, based on the actual representation objective, we draw a connection between METRA and contrastive learning. In Sec. 6 conduct experiments verify that METRA learns optimal representations in practice and that they bear resemblance to contrastive representations.
Sec. mentioned that the Lagrangian [(Ø, A) used the METRA representation objective does not correspond to the constrained optimization problem in Eq. 4 raising the following question: What is the actual METRA representation objective? To answer this question, we note that, rather than using distinct dual variables for each pair of (s, E S T(cb X) employs a single dual variable, imposing
We ignore the slack variable in Park et al. (024 hecause takes fairly small value 10-3 &
4

---

Published as a conference paper at ICLR 2025
expected temporal distance constraint over all pairs of ss) under the historical transition distribution (s,, s'). This observation suggests that METRA's representations are optimized with the following objective
max LE (s
[z))) s.t. (s,a)) - S 1.
Applying KKT conditions to L(Ø, A), claim that
Proposition 1. The optimal state representation o² of the actual METRA representation objective (Eq. 7 satisfies
(pp(s [1lØ*(s') - (s) =1
The proof is in Appendix A.2 Constraining the representation of consecutive states in expectation not only clarifies the actual METRA representation objective, but also means that we can predict the value of this expectation for optimal Sec. [6.1 includes experiments studying whether the optimal representation satisfies this proposition in practice. Importantly, identifying the actual METRA representation objective allows to draw a connection with the rank-hased contrastive loss (InfoNCE (Oord et al. 2018 Ma & Collins) 2018). which we discuss next.
We relate the actual METRA representation objective to contrastive loss, which will specify first and then provide some intuitions for what it is optimizing. This loss is a lower bound on the mutual information 1 S" Z) and variant of the InfoNCE objective (Henaft) 2020 Ma & Collins 2018
Zheng et al. 2024 Starting from the standard variational lower bound (Barber & Agakov 2004
Poole et al. 2019). prior work derived an unnormalized variational lower bound on (S:
(TUBa in Poole et al. 2019)),
(', 2 Ep (s (s, s), (as() log Ept=') [ (z))
where f:S x S x 2 - the critic function (Ma & Collins 2018 Poole et al. 2019 Eysenbach
et al. 2022 Zheng et al. 2024). Since the critic function takes arbitrary functional form, one can the inner product between the difference of transition representations
choose to parametenize
and the latent skill, i.e. f( 8 (Ø(s') o(s)) This yields specific lower bound:
1 (S,S': 2 Eps (x) ((((s') - o(s)) [x] -E (pp() log Bp(e')
((()))-0)) #
LB (p). (8)
LB (4)
Intuitively, (o) pushes together the difference of transition representations, d(s') - <(s) and the latent skill sampled from the same trajectory (positive pairs), while LB pushes away o(s') (s) and sampled from different trajectories (negative pairs). This intuition is similar to the effects of the contrastive loss, and we note that Eq. S only differs from the standard InfoNCE loss in excluding the positive pair in LB (d). We will call this lower bound on the mutual information the contrastive lower bound.
We now connect the contrastive lower bound I.B (() (Eq. 8) to the actual METRA representation loss L( A) (Eq. 5). While both of these optimization problems share the positive pair term (LB (6)). they vary the way they handle randomly sampled (s, 8 82,2) pairs (negatives): METRA constrains the expected L2 representation distances
((s)l while the contrastive lower bound minimizes the log-expected-exp score (LB" (o)). However, we bridge this difference by viewing the expected 1.2 distance as a quadratic approximation of the tog-expected-exp score:
(e.(e.")) [llo(s')
Proposition 2. There exists Au(d) depending on the dimension of the state representation & such that the following second-orde order Taylor approximation holds
Ao - -E,," [l])(s))
LB (6).
See Appendix A.3 for proof. This approximation shows that the constraint in the actual METRA representation loss has effects similar to LB (ø). namely pushing d(s') Ø(s) away from randomly sampled skills. Furthermore, this proposition allows us to spell out the (approximate) equivalence between representation learning in METRA and the contrastive lower bound on (S, S'; Z):
5

---

Published as a conference paper at ICLR 2025
Corollary 1. The METRA representation objective is equivalent to a second-order Taylor approxi- mation of S'; 2), Le., (o(]) x +IB8(()
The METRA representation objective can be interpreted as a contrastive loss, allowing us to predict that the optimal state representations o² (Prop. [ have properties similar to those learned via contrastive learning. In Appendix C.I. we include experiments studying whether the approximation in Prop. is reasonable in practice. In Sec. 6.22 we empirically compare METRA's representations to those learned by the contrastive loss. Sec. 6.33will will study whether replacing the METRA representation objective with a contrastive objective retains similar performance.
4.2 CONNECTING METRA'S S ACTOR OBJECTIVE WITH AN INFORMATION BOTTLENECK
This section discusses the actor objective used in METRA. We first clarify the distinction between the actor objective of METRA and those used in prior methods, helping to identify term that discourages exploration. Removing this anti-exploration term results in covering a larger proposition of the state space while learning distinguishable skills. We then relate this anti-exploration term to estimating another mutual information, drawing a connection between the entire METRA actor objective and a variant of the information bottleneck (Tishby et al.
2000 Alemi al. 2017).
While prior work (Eysenbuch et al. 2019 Gregor et al. 2016 Sharma et al. 2020 Hansen et al.
2020, Campos et al., 2020) usually uses the same functional form of the lower bound on (Eq. 2
& 3) different variants of the mutual information to learn both representations and skill-conditioned policies (see Appendix for details), METRA uses different objectives for the representation and the actor. Specifically, the actor objective of METRA J(xx) (Eq. 6) only encourages the similarity between the difference of transition representations ((s') - o(s) and their skill z (positive pairs),
while ignoring the dissimilarity between ((s') ((s) and random skill (negative pairs):
[(T) LB (Φ) I.B
(+) (Ø),
where LB (o). LB (6), and LB (6) are under the target policy æ instead of the behavioral policy 3. The SOTA performance of METRA and the divergence between the functional form of the actor objective (positive term) and the representation objective (positive and negative terms) suggests that LB (Ø) may be term discouraging exploration. Intuitively, removing this anti-exploration term boosts the learning of diverse skills. We will empirically study the effect of the anti-exploration term in Sec. 6.3 and provide theoretical interpretations next.
Our understanding of the anti- exti-exploration term LBT (Ø) relates it to a resubstituion estimation of the differential entropy ATd (Ø(S") - o(S)) in the representation space (see Appendix A.4 for details), i.e.,
LB = he (d(S') - ((S)). Note that this entropy is different from the entropy of states (S), indicating that want to minimize the entropy of difference of representations (0s') (08) to encourage exploration. There are two underlying reasons for this (seemly counterintuitive) purpose: METRA aims (1) constrain the expected L2 distance of difference of representations o(s') (0s)
(Eq. 5) and (2) push difference of representations ((s') - o(s) towards skills 2 sampled from UNIF d Nonetheless, this relationship allows us further rewrite the anti-exploration term LB (d) as estimation of the mutual information 1 (d"(S)) - ((S"); S"),, connecting the METRA actor objective to an information bottleneck:
Proposition 3. The METRA actor objective is a lower bound on the information bottleneck ("(S,S': Z) (S S" ((S)) - o(S)). i.e. ((T) S
-(())
See Appendix A.4 for a proof and further discussions. Maximizing the information bottleneck (" S'; /) 7 (S, S¹ ((S)) o(S)) compresses the information in transitions (s, into differ- ence in representations o(s') ((s) while relating these representations to the latent skills 2 (Alemi
al. 2017 Tishby et al. 2000). This result implies that simply maximizing the mutual information
("(S, (SS Z may be insufficient for deriving a diverse skill-conditioned policy म, and removing the anti-exploration LB (Φ) may be a key ingredient for the actor objective. In Appendix A.7 we
propose a general MISL framework based on Prop. 3
A SIMPLIFIED ALGORITHM FOR MISL VIA CONTRASTIVE LEARNING
In this section, we derive a simpler unsupervised skill learning method building upon our understand- ing of METRA (Sec. 4). This method maximizes MI (unlike METRA), while retaining the good

---

Published as a conference paper at ICLR 2025
performance of METRA (see discussion in Sec. We will first the contrastive lower bound to optimize the state representation & and estimate intrinsic rewards, and then we will learn the policy и using successor features. We use contrastive successor features (CSF) to refer to method.
5.1 LEARNING REPRESENTATION THROUGH CONTRASTIVE LEARNING
Based on our analysis in Sec. 4.1 we use the contrastive lower bound on 1 (S, ;); 2) to optimize the state representation directly. Unlike METRA, we obtain this contrastive lower bound within the MISL framework (Eq. 2 & by employing parameterization of the variational distribution g( 8,)") mentioned in prior work (Poole et al., 2019 Song Kingma, 2021). Specifically, using scaled energy-based model conditioned representations of transition pairs (s, s'), we define the variational distribution as
p(2)e(0(6))-(0))
<(= 8, 8,85)
(9)
E (000))))000))00000 [']
Plugging this parameterization into Eq. 2 produces
0k+1 arg max (p [(o(s') (())s')((s))) (e,("))))
log Ep(z)) [c(((0)) Tx (10)
which is exactly the contrastive lower bound on (S, S" Z). This contrastive lower bound allows us to learn the state representation while getting rid the dual gradient descent procedure (Eq, (5) adopted by METRA. In practice, find that adding fixed coefficient to the second term of Eq. 10 helps boost performance. We include further discussions g in Appendix A.6 and ablation studies Appendix C.3
In the same way that the METRA actor objective excluded the anti-exploration term (Sec. 4.2), we propose construct the intrinsic reward by removing the negative term from representation objective (Eq. 10). resulting in the same RL objective as (TT) (Eq, 6
7k arg max E (e,e" (8, =)] (s. (0)(s') cb)()))
(11)
a
We use this RL objective the update rule for the skill-conditioned policy i in our algorithm.
5.2 LEARNING A POLICY WITH SUCCESSOR FEATURES
To optimize the policy (Eq. 11 we will
actor-critic method. Most skill learning methods an off-the-shelf RL. algorithm (e.g.. TD3 (Fujimoto et al.) 2018). SAC (Haarnoja et al.) (018)) to fit the critic. However, by noting that the intrinsic reward function r(s, s', is linear combination between basis ((s') ((s) E Rd and weights = E Z c IRd we can borrow ideas from successor representations learn a vector-valued critic. We learn the successor features 3htt Sxxxx Rd.
d (8, a, 2) (Fapp"(e+=ell ( (Ø(s') ((s)),
with the corresponding skill-conditioned policy a in actor-critic style:
(8,1(s,0,2) + arg min EE (s.0,))<p (s.a, of [ (v(s, (s,
where (x a' (x(()) (x) + (s', ('', [Wh (8, 2) T²]
TE+1 <- arg max E, (s, (x..))) (((as)))
where &a is an estimation _. practice, we optimize v and π for one gradient step iteratively.
Algorithm Summary. In Alg. I we summarize CSF, our new algorithm Starting from an existing MISI algorithm (e.g., DIAYN (Fysenbach et al., 2019) and METRA (Park et al., (024)), implementing our algorithm requires making three simple changes: (I) learning state representations do by minimizing an InfoNCE loss (excluding positive pairs in the denominator) between pairs of
WWe ignore the iteration & for notation simplicity.
*Code: https//ggihuh.coo/PPnnccoo httpe//wwwo..oooooaaaaeeeooooooo

---

Published as a conference paper at ICLR 2025
Algorithm I Contrastive Successor Features
1: Input state representations Φs, successor features ths, skill-conditioned policy Ty. and target successor feature Maa-
2: for each iteration do
3: Collect trajectory with z ~ p(z) and a a Ty(a I 8,2), and then add 7 to the replay buffer.
4: Sample {(s, as,,,, s', ~ replay buffer. [a'} ~ Tzpaa s', z), and [z'] ~ p(=').
L(0) - (a)))))) [(do(s') @b(s)) 2] + ((,,')) log En eloald'))
5:
6:
L(w) + (( (1, 0, (60(5') - Φo(s) + TV " a)))))) [s, a, 2)
7:
C(n) + -B -B(s,)),aw,,
8:
Update 0, w, and r by laking gradients of (()), ((w). and C(n)-
9:
Update using exponential moving averages.
10:
Return do; thus and TT.-
(s, 8') and 2, (2) using a critic the with d-dimensional outputs and replacing the scalar reward with the vector dh (s") (s) (3) sampling the action from the policy Ory to maximize the inner product Pia (8, a, 2) Tz.
Unlike CIC (Laskin al. 2022), our method does not use the standard InfoNCE loss and instead employs variant of it. Unlike VISR (Hansen et al.) (020), our method does not train the state representation & using a skill discriminator. Unlike METRA, our method learns representations using the contrastive lower bound directly, avoids the Wasserstein distance and dual gradient descent optimization, and results in a simpler algorithm (see Appendix B.I for further discussions).
6 EXPERIMENTS
The aims of our experiments are (I) verifying the theoretical analysis in Sec. expprrmentallyy. (2) identifying several ingredients that are key to making MISL algorithms work well more broadly. and (3) comparing our simplified algorithm CSF to prior work. Our experiments will use standard benchmarks introduced by prior work on skill learning. All experiments show means and standard deviations across ten random seeds.
6.1
METRA CONSTRAINS REPRESENTATIONS EXPECTATION
Sec. 4.1 predicts that the optimal METRA representation satisfies its constraint (())
strictly (Prop. 1. We study whether this condition holds after training the algorithm for long time. To answer this question, we conduct didactie experiments with the state-based Ant from METRA (Park et al. 2024) navigating in open space. We set the dimension of á to d = 2 such that visualizing the learned representations becomes casier. After training the METRA algorithm for 20M environment steps (50K gradient steps), we analyze the norm of the difference in representations [[o(s') - (13
(s, s') [[]s, [|(((s))
We plot the histogram of (|p(s)) - < over 10K transitions randomly sampled from the replay buffer (Fig. 2a). The observation that the empirical average of lé(s')= <(s)]l converges to 0.9884 suggests that the learned representations are feasible. Stochastic gradient descent methods typically find globally optimal solutions on pver-parameterized neural networks (Du et al.) 2019), making us conjecture that the learned representations are nearly optimal (Prop. Furthermore, the spreading of the value of lo(s')= +(s)) & implies that maximizing the METRA representation objective will nor
learn state representations & that satisfy [lx((') <bs) 1| < 1 for every (s, E S These results help to explain what objective METRA'S representations are optimizing.
6.2 METRA LEARNS CONTRASTIVE REPRESENTATIONS
We next study connections between representations learned by METRA and those learned by con- trastive learning empirically. Our analysis in Sec. 4.1 reveals that the representation objective of METRA corresponds to the contrastive lower bound on 9 (S. S" Z). This analysis raises the question whether representations learned by METRA share similar structures to representations learned by contrastive losses (Gutmann & Hyvärinen] 2010 Ma & Collins 2018
Wang & Isola 2020).
To answer this question, we reuse the trained algorithm in Sec. 6.1 and visualize two important statistics: (I) the conditional differences in representations o(s') o(s) - 2 and (2) the normalized
marginal differences in representations (ø(s') - ((s)) ///(///''''/////// lls) The resulting histograms (Fig. 2b & [c) indicate that the conditional differences in representations Ø(s') - ((s) z converges to isotropic Gaussian in distribution while the normalized marginal differences in representations
8

---

Published as a conference paper at ICLR 2025
100
800
300
0.5
400
20
-0.5
0.3
Cou15
502
1800--001 P008
#07
ccuna
anglle
(a) Expected constraint
(b) Gaussianity
(c) Uniformity
Figure 2: Histograms of METRA representations. (a) The expected distance of representations converges
1.0, helping explain what objective METRA's representations are optimizing. (b) Given a latent skill, the
conditional difference representations (ø(s') - Φ(s) x) converges to an isotropic Gaussian distribution. (c)
Taking the marginal over latent skills, the normalized difference in representations
[ofts)) +
(d(*") (((")=(()))
converges to
a UNIF(S" These ohservations are consistent with our theoretical analysis (Cor. suggesting that METRA
is performing form of contrastive learning.
Contrastive loss
important exploration.
Parametrization key.
State Devengee
2000
2000
2000
CSE NLF
2000
1000
CSF
1000
-Jiles
NETRA-C
0.5
1.0
3.0
Steps
Steps
Steps
Figure 3: Ablation studies. (Left) Replacing the METRA representation loss with contrastive loss retains performance. (Center) Using an information bottleneck to define the intrinsic reward is important for MISL. (Right) Choosing the right parameterization is crucial for good performance. Shaded areas indicate std. dev. (ó(s') (())) sphere St in distribution. Prior work (Wang & Isola 2020) has shown that representations derived from contrastive learning preserves properties similar to these observations. We conjecture that maximizing the contrastive lower bound (S, S": Z) directly has the same effect as maximizing the METRA representation objective. See Appendix A.8 for formal claims and connections.
/l'(('')) ((s))/((((())))))
(ss) converges to a uniform distribution on the d-dimensional unit hyper-
6.3 ABLATION STUDIES
We now study various design decisions of both METRA and CSF, aiming to identify some key factors that boost these MISL algorithms. We will conduct ablation studies Ant again, comparing coverage of (x, y) coordinates different variants.
(1) Contrastive learning recovers METRA's representation objective. Our analysis (Sec. and experiments (Sec. 6.2 have shown that METRA learns contrastive representations. We now test whether retain the performance of METRA by simply replacing its representation objective with the contrastive lower bound (Eq. {8. Results in Fig. 3 (Left) suggest that using the contrastive loss (METRA-C) fully recovers the original performance, circumventing the Wasserstein dependency measure.
(2) Maximizing the information bottleneck is important. In Sec. 4.2 we interpret the intrinsic reward in METRA as lower bound on an information bottleneck. We conduct ablation experiments study the effect of maximizing this information bottleneck over maximizing the mutual information directly, strategy typically used by prior methods (Eysenbach et al., 2019 Mendonca et al.) 2 Hansen et al., (220)). Results in Fig. (Center) show that CSF failed to discover skills when only maximizing the mutual information (i.e. including the anti-exploration term). These results indicate that using the information bottleneck as the intrinsic reward may be important for MISL algorithms.
(3) Parameterization is key for CSF When optimizing lower bound on the mutual information (S, St: Z) using a variational distribution, there are many ways to parametrize the critic f(s, s', 2). In Eq. 9 we chose the parameterization (ø(s') d(s)) z, but there are many other choices. Testing the sensitivity of this choice of parameterization allows us to determine whether a specific form of the lower bound is important. In Fig. we study several variants of CSF that use (1) monolithic network MLP(s, s/) 2 (2) a Gaussian kernel (- ((s') (l)) or (3) a Laplacian kernel (-llo(s')= d(s)]l)) the critic parameterization. We find the alternative parameterizations are catastrophic for performance, suggesting that the inner product parameterization is key to CSE We
provide some insights for this parameterization in Appendix A.9

---

Published as a conference paper at ICLR 2025
HalfCheetah (State Coverage)
HalfCheetah (Zero-shot Goal)
HalfCheetahHurdle (Hierarchical)
Coverage
200
Staying Tima
Return (past
1.00
ENw Steps
Env Stops
Erv Stops
Humanold (State Coverage)
Humanold (Zero-shot Goal)
HumanoldGoal (Hlerarchical)
Coverage
Raturn tpast 10)
Stawing Time
1.00
20
0.1
0.0
8.0
Env Steps
Loc
Env Steps
Enw Steps
140
(ours)
METRA
DIAYN
DADS
Figure 4: CSF performs on par with METRA. We compare with baselines on state coverage (left), zero-shot goal reaching (middle), and hicrarchical control (right). CSF performs roughly on par with METRA and outperform all other basclines in most settings. Shaded areas indicate one standard deviation. Appendix Fig. 5 6 7 show the learning curves for all tasks.
6.4
CSF MATCHES SOTA FOR BOTH EXPLORATION AND DOWNSTREAM PERFORMANCE
Our final set of experiments compare CSF to prior MISL algorithms, measuring performance on both unsupervised exploration and solving downstream tasks.
Experimental Setup. We evaluate on the same five tasks as those used in Park ct al. (2024) plus Robobin from LEXA (Mendonca et al. 202I). For basclines, we also use subset from Park ct al.
(2024) (METRA (Park et al. 2024), CIC (Laskin et al. 2022). DIAYN (Eysenbach et al. 2019 2019) 9. and
DADS (Sharma al. 2020 along with VISR (Hansen al. 2020 See Appendix B. for details.
Exploration performance. To measure the unsupervised exploration capabilities of each method, compute the state coverage by counting the unique number of (ı, y) coordinates visited by the agent. Fig. 4 (left) shows CSF matches METRA on both Ha l fCheetah and Humanoid d. See Appendix B.3 for full results on exploration.
Zero-shot goal reaching. In this setting, the agent infers the right skill given a goal without further training on the environment, We evaluate on the same of six tasks and defer both the goal sampling and skill inference strategies to Appendix B.4 We report the staying time fraction, which is the number of time steps that the agent stays at the goal divided by the horizon length. In Fig. (middle), we find all methods to perform similarly on Ha fCheetah, while METRA and CSF perform best on Humano d, with METRA performing slightly better on the latter. See Appendix B.4 for full results on zero-shot goal reaching.
Hierarchical control. We train hierarchical controller n 8) that outputs latent skills 2 as actions for every fixed number of time steps to maximize the discounted return in two downstream tasks from Park et al. (2024), one of which requires to reach a specified goal (Humano iGoan hurdles Se11FCheetaaHurddee The results in Fig. 4 (right) show CSF and METRA are the best performing methods, showing mostly similar performance. See
and one requires jumping
Appendix B.5 for full results and details.
Taken together, CSF is a competitive MISL algorithm that matches the current SOTA. On the full set of results (Appendices B.3 B.4 andB..) find that CSF continues to perform roughly par with METRA on most tasks, though there are some tasks where CSF performs better and vice versa.
7 CONCLUSION
In this paper, we show our understanding of a current SOTA unsupervised skill discovery algorithms through the lens of MISL. Our analysis allowed the development of simpler method CSF, which performs on par with METRA in most settings. More broadly, we provide evidence that mutual information maximization can still be effective to build high performing skill discovery algorithms.
Limitations. While CSF performs relatively well on the standard benchmarks, it is unclear how to scale the performance to increasingly complex environments such as Craftax (Matthews et al., 2024) or VIMA (Jiang et al. 2022), which present increased number of objects, partial observability, stochasticity, and discrete action spaces. Another open question is how to perform scalable pre- training large datasets, e.g., BridgeData V2 (Walke al., 2023) YouCook2 (Zhou al.) 2018), using MISL algorithms such as CSF to get both transterable state representations and diverse skill-conditione policies. We leave investigating these empirical scaling limits future work.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We have made our code publicly available here:
successor-features] In addition, we have dedicated several sections in the appendix further ensure reproducibility of our results. Appendix B provides a detailed account of all experimental details, including GPU types, training times, hyperparameters and architectural details (Tables 1 and 3). and detailed descriptions of the parameters of the various settings we use to compare with prior work (Appendices [.3 B.4 B.5 Finally, we have also included full proofs for the theoretical results stated in Proposition and which can be found in Appendices A.2 A.3 and A.4 respectively.
ACKNOWLEDGEMENTS
We thank the National Science Foundation (Grant No. 2239363) for providing funding for this work. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We thank Seohong Park for providing code for several of the baselines the paper. addition, we thank Qinghua Liu for providing feedback early drafts of this paper. We also thank Princeton Research Computing.
REFERENCES
Milton Abramowitz and Irene A Stegun. Handbook of mathematical functions with formulas, graphs, and mathematical lables, volume 55. US Government printing office, 1968.
Ibrahim Ahmad and Pi-Erh Lin. A nonparametric estimation of the entropy for absolutely continuous distributions (corresp.). IEEE Transactions on Information Theory, 22(3):372-3755 1976.
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. In International Conference on Learning Representations, 2017. URL https
//openreview. n forum? / dd-yyozBceeee
David Barber and Felix Agakov. The im algorithm: variational approach information maximiza- tion. Advances in neural information processing systems, 16(320):201, 2004.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
Kate Baumli, David Warde-Farley, Steven Hansen, and Volodymyr Mnih. Relative variational intrinsic control. Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 6732-6740, 2021.
Diana Borsa, Andre Barreto, John Quan, Daniel J Mankowitz, Hado van Hasselt, Remi Munos, David Silver, and Tom Schaul. Universal successor features approximators. In International Conference on Learning Representations, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 160iv:1606.015:0 2016.
Tom B Brown. Language models are few-shot learners. arXiv preprint ArXiv:2005. /4165, 2020.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In Seventh International Conference on Learning Representations, pp. 1-17, 2019.
Víctor Campos, Alexander Trotr, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Tor- res. Explore, discover and learn: Unsupervised discovery of state-covering skills. International Conference on Machine Learning, pp. 1317-1327. PMLR, 2020.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR,, 2020.
11

---

Published as a conference paper at ICLR 2025
Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu. Variational empowerment as representation learning for goal-based reinforcement learning. arXiv preprint arXiv:2/06.01404, 2021.
S. Chopra, R. Hadsell, and Y. LeCun. Leamning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 1, pp. 539-546 vol. 2005. doi: 10.1 10.1
Peter Dayan. Improving generalization for temporal difference learning: The successor representation.
Neural computation, 5(4):613-62) 624, 1993.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert:
Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associa- fion for Computational Linguistics, 2019. URL pattps://aag www.nt.
org/
CorpusID:5296739:
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pp. 1675-1685
PMLR, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019. /RLLsttss//ppeeeviieee wtwpeeeeiiiiieeeeeeeeeeeeeee wtt.:e/eoooommm21-55133333333333
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning achieve goals via recursive classification. arXiv preprint arXiv:201 1. 1.08909, 2020.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35:35603-35620, 2022.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforce- ment learning. In International Conference on Learning Representations, 2016.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894 6910, 2021.
Karol Gregor. Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.00 100707, 2016
Michael Gutmann and Aapo Hyvärinen. contasconeeeetive estimation: new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings. 2010.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastivn estimation: A new estimation principle for unnormalized statistical models. Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL http::tttps/sc... pttttp:://procceaingg. mlr wwwps//wee/eeeeeeeeeeeeeeeeeeeee http:////gttaaannnaaa html
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning. pp. 1861-1870. PMLR, 2018.
Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In International Confer- ence on Learning Representations, 2020. URL https:://peezevieeet wtttfooummm forum?id
B.JAAHrYDDS)
12

---

Published as a conference paper at ICLR 2025
Shuncheng He, Yuhang Jiang. Hongchang Zhang, Jianzhun Shao, and Xiangyang Ji. Wasserstein unsupervised reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 6884-6892, 2022.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International conference on machine learning, pp. 4182-4192. PMLR, 2020.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances neural information processing systems, 29, 2016.
Wolfram Research, Inc. Mathematica, Version 14.0, 2024. URL htLhttp:::/mmm www wol wolfram. com/ Mathhmmaiica Champaign, IL, 2024.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022.
Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep successor
reinforcement learning. arXiv preprint arXiv:160..0233: 2016.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International conference on machine learning, pp. 5639-5650. PMLR, 2020.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:2202.00161, 2022.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv: (556://990005220 2019.
Mengdi Li, Xufeng Zhao, Jae Hee Lee, Cornelius Weber, and Stefan Wermter. Internally rewarded reinforcement learning. In International Conference on Machine Learning, pp. 20556-20574.
PMLR, 2023.
Fangchen Liu, Hao Liu, Aditya Grover, and Pieter Abbeel. Masked autoencoding for scalable and generalizable decision making. Advances Neural Information Processing Systems, 35: 12608-12618, 8, 2022.
Hao Liu and Pieter Abbeel. Aps:
Active pretraining with successor features. In International Conference on Machine Learning, pp. 6747. PMLR, 2021.
Yecheng Jason Ma. Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang, Vip: Towards universal visual reward and representation via value-implicit pre-training, In The Eleventh International Conference on Learning Representations, 2023.
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2698-3707, 2018
Michael Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Thomas Jackson, Samuel Coward, and Jakob Nicolaus Foerster. Craftax: A lightning-fast benchmark for open-ended reinforeement learning. In Forty-first International Conference on Machine Learning. 2024.
Russell Mendonca, Oleh Rybkin, Kostas Danilidiis, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models. Advances in Neural Information Processing Systems, 34: 24379-244 24391, 2021.
Shakir Mohamed and Danilo Jimenez. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. Advances in neural information processing systems, 28, 2015.
Vivek Myers, Chongyi Zheng, Anea Dragan, Sergey Levine, and Benjamin Eysenbach. Learning temporal distances: Contrastive successor features can provide a metric structure for decision- making. arXiv preprint a40iv:2406. /7098, 2024.
13

---

Published as a conference paper at ICLR 2025
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior- agnostic estimation of discounted stationary distribution corrections. Advances in neural information processing systems, 32, 2019.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In Speech Synthesis Workshop, 2016. URL http:: /api semanticschol.
org/CorpusTD: 6254678
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini 1807:088003344 2018.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. Advances in Neural Information Processing Systems, 32, 2019.
Seohong Park and Sergey Levine. Predictable mdp abstraction for unsupervised model-based rl. In International Conference on Machine Learning, pp. 27246-27268. PMLR, 2023.
Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- unsupervised skill discovery. In International Conference on Learning Representations, 2022.
Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability unsupervised skill discovery. In International Conference on Machine Learning, pp. 27225-27245. PMLR, 2023.
Seohong Park, Oleh Rybkin, and Sergey Levine. METRA: Scalable unsupervised RL with metric- aware abstraction. In The Tweifth International Conference on Learning Representations, 2024.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787 PMLR, 2017.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: state-covering self-supervised reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, pp. 7783-7792, 2020.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180.
PMLR, 2019.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- training. unpublished, 2018. URL URL Itttps://app. www.aaniiisccooaaooooooooooo
49313245
Alec Radford. Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. unpublished, 2019. URL https://api semant. i cscho warcholar..r ort.or/g/ooooo Corpus ID: 60025533]
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Kuttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1312 1320. Lille, France, 07 Jul 2015, PMLR. URL https://proceedingg mlr. www.s.press/vv.
sschal1 15.
14

---

Published as a conference paper at ICLR 2025
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Nur Muhammad Mahi Shafiullah and Lerrel Pinto. for changing world. In International Conference on Learning Representations, 2022. URL
One after another: Learning incremental skills
https://openrevall
net w 9moSRq1o
Claude Elwood Shannon. A mathematical theory of communication. cal Joumna,, 27:379 -23, 1948. URL http::/plans www.en.eellllaaaaa com/ hom/cm// ma/ whatt/
The Bell System Techni-
Thammnnday/shanoon1153
pdf
Archit Sharma, Shixiang Gu Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
URL /oRL:ttep://penrevvieee/ wwwp:eeeeeetoooridiiiiiieie 1d-HHgLZ2RKK
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objec- tive. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL httpps:/ 7tle://f.800777bbb00000090222323
nwwrips. wttper///llss//pe////2110
Yang Song and Diederik P Kingma. How to train your energy- based models. arXiv preprint arXiv:2101.03288, 2021.
DJ Strouse, Kate Baumli, David Warde-Farley, Volodymyr Mnih, and Steven Stenberg Hansen. Learning more skills through optimistic exploration. In International Conference on Learning Representations, 2022. URL /RR/https:///opeeevieee
net/
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control.
In 2012 IEEF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10.1109/RR0S 201.:0000..2...3660999
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations, 2020.
Homer Rich Walke, Kevin Black, Tony Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. Conference on Robot Learning, pp. 1723-1736. PMLR, 2023.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International conference on machine learning, pp. 9929-9939. PMLR, 2020.
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through parammetit discriminative rewards. In International Conference Learning Representations, 2019.
Wikipedia. Von Mises-Fisher distribution Wikipedia, the free encyclopedia.
http:://w....iiipeiaaaaoo//oonnp http:://www.iiee..p..tptt... htw.::ttteeeeoo22MMsssssssses s00ns22222s SSIadeeescciiiraaaotoooooeeee222 2024].
2024. [Online; accessed 10-July-
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. URL htt.:htttt://peeeee
www.:eewwwwwwwttteeeeeeaaaaaeeae d= d 6aTvGaf
15

---

Published as a conference paper at ICLR 2025
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep rein- forcement learning with successor features for navigation across similar environments. 2017 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 2371-2378. IEEE, 2017.
Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding. In The Twelfth International Conference on Learning Representations, 2024. URL
apenreview https://openreieew
https://
net net
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
16

---

Published as a conference paper at ICLR 2025
A THEORETICAL ANALYSIS
A.1 MUTUAL INFORMATION MAXIMIZATION AS MIN-MAX OPTIMIZATION PROBLEM
Maximizing the mutual information (" S" 2) (Eq. is more challenging than standard RL because the reward function logp" s') depends on the policy itself. To break this cyclic dependency, we introduce variational distribution q(= 8, E Q _ {g(= s, )') to approximate the posterior p" (z s, s'). where we assume that the variational family is expressive enough to cover the ground true distribution under any π.
Assumption 1, For any killl-onndtioned policy л ::S x 2 ~ A(A), there exists q" (z s, 5') E Q such that (= 8, s') (= s').
This assumption allows us rewrite Eq. as
max (s,s' (ax) log (= s, s')
-min (s,ss) [DEL (-1 |l q(( 1,, ss,s SEQ
where DKL (p" (- 8,) q(. 8,
8)) ) is the KL divergence between distributions p" and g and satisfies DxL (p" 8, |l q(- s, s. = 0 = p= 8, s') q(> s, s'). The new max-min optimization problem can be solved iteratively by first choosing variational distribution g(2 s,s²) to fit the ground truth p" (= s, s') and then choosing policy maximize discounted return defined
by the intrinsic reward q(2 s'):
4c+1 arg max (s,sts, [log (= s, s,,')l,
7k+1 <- arg max (e [lg qi((2 | 8'),
where indicates the number of updates. In practice, the data used to update are uniformly sampled from replay buffer typically containing trajectories from historical policies. Thus, the behavioral policy is exactly the average of historical policies becomes
- Σt mi(a s, 2) and the update rule for
4h++1 arg max (e,er, (yo |s,s')]
gEQ
A.2 PROOF OF PROPOSITION
Proposition 1. The optimal state representation o² of the actual METRA representation objective (Eg. satisfies
(aso) (z)) =1
Proof.
Suppose that the optimal ø* satisfies
E (e [1lØ 6- (>)))) a² <1,
(12)
where 0 S a < 1. Then, there exists a 1/a > 1 that scales the expectation in Eq. 12 to exactly 1:
a2 E (p.)) [110* (s') o (s) =
Note that, when Ep(a) a larger number
[(0*(s') (s)) 2 O. the 6* /a will also scale the objective to
E a (P(e)) [(0 (s)) E p(t))p [(0*(5) -s')-<(sss) 2]
which contradicts the assumption that 6* optimal. When E (a ((''('') (s') x² (s)) 2] < taking -Ø" /a gives us the same result. Therefore, we conclude that the optimal Φ" must satisfy (ps((.s))) (8')*(')) 6² [s) 1.
17

---

Published as a conference paper at ICLR 2025
A.3 PROOF OF PROPOSITION
Proposition 2. There exists An(d) depending on the dimension d of the state representation e such that the following second-order Taylor approximation holds
(o(d)( - Epp [llx((') -16((s))+(<s113)) (6).
Proof. We first compute log (z) (c(((6))(10))
analytically,
log (2) (el((s)) = log Ca(C)))))) (60())))())) =dz
Co(0)
log Ca(ll((s)) 6())
+ log / -(())(()))())))))ddo)>)))))))) -(0)))))llll (000 (
ocilla de
Ca(0)
log
Ca(llo(s)) ((8)122
T (d/2) (2 / I//2) ( (())11)
log
2n 2/ ( 6(s)
IIa/2-I (1(0(s)) (())12))
log
(13)
1 (1)) d/2-1
where (-)) is the Gamma function, (-) denotes the modified Bessel function of the first kind at order
v, Ca(-) denotes the normalization constant for d-dimensional von Mises-Fishhe distribution, and in (a)
we use the definition of the density of vMF distributions. Applying Taylor expansion (Abramowitz &
Stegun 1968) to Eq. 13 around [[o(s') <(s)]12 by using Mathematica (Inc. 2024) gives us
polynomial approximation
(1d/))29/2-11ap2-(1l1 ((s))a) = l0((s') -(s))))))))+))((())))) 2(s)
1
log
llo(s') 6()) 4/2-1
Now can simply set (d) 24 to get
- (1
- d(s)) = - log (x) [(((()))-(0)) + const..
Hence, conclude that (d) - (p,( [1]Ø(s') - ((s) is second-order Taylor approxi-
mation LB (Ø) E (.,,)) log Ep()) (((())))6())) around [[g(s') <bs) 0 up to
constant factor of (d)
A.4 PROOF OF PROPOSITION
Proposition 3. The METRA actor objective is lower bound the information bottleneck
(S, - ( (( o(S). ie., < (S,, ( o(S)..
Proof. We consider the mutual information between transition pairs and skills under the target policy
I Si: Z). The standard variational lower bound (Barber Agakov 2004 Poole al. 2019) of
(S, Z) can we written as:
(S, 2 h(2) + Epr(s. (lg q(z | s,
where q(2 s,s is arbitrary variational approximation of (z 8, s')). We can set log q(s ss s') to be
log q(2 s.s' f(s.s',2) logp(2) log(p(e) [c(6)(
18

---

Published as a conference paper at ICLR 2025
resulting in lower bound:
(S, 2 (s) ((s))
(pr log Ep(s) ((0(51))000)
LB7 (0)
LB (+)
where LB$ (6) exactly the same as the RL objective [(x) (Eq. Eq. [8 but it is under the target policy ı instead.
6 This lower bound is similar to
Equivalently, we can write the RL objective as
( = Bp(=)p= (
(d((5)) - o(s)) 2- B-log ((((('))-(e))
+ log Eapcr) (c(((=))(())) [1] = LB (Ø) - LBB (6)
where the two log-expected-exps cancel with each other. We next focus the additional LB (6) - (p(s)p = log Ep(x') ((()))))<()))"
which can be interpreted as
resubstitution entropy estimator of (6s') ((5)
(Wang & Isola 2020 Ahmad & Lin 1976):
LB (Ø) -Bp
log Bp(x)
((0(=')-0(0)
=1]
Calllo(s)) ((s))l2)e [0(0))) o(ss)lla (o(e)) ((=)))) Ta
(a)
log
1oco))
N
log
Ca(llo(s() ((2))
log KDE( - ((:))) + log Calll((s)) - p(s;))l2
= (o(S") x(S)) (s logg Calllo(s)) (8:))12)
1 @ ("((,'': - o(S) (ppe))))) log Calllo(s() - ((s))11))
@ (=. o(S)) - Eo(d) (s.a)) [llo(s') ((s) + const.
2 ('''s'' ((S) + const.,
where Ca(-) denotes the normalization constant for d-dimensional von Mises-Fisher distribution, in (a) we use Monte Carlo estimator with N transitions and skills {(s,, sy = to rewrite the expectation, in (b) we replace the entropy estimator het with the mutual information estimator f" since <(s') ((s) is deterministic function of (s, s'), in (c) we apply the same approximation in Prop. 2 and in (d) the expected squared norm is replaced by 1.0, assuming that Prop. hoolds. Taken together, we conclude that maximizing the RL objective J J(T) is approximately equivalent to maximizing
lower bound on the information bottleneck (S S" Z) (S, ((8))) -(S)).
A.5 MUTUAL INFORMATION OBJECTIVES USED IN PRIOR METHODS
Prior MISL methods (Eyenbaahh al. 2019: Gregor et al. 2016, Sharma et al. 2020 Hansen et al., 2020
Campos et al.] 2020) adopt the min-max optimization procedure (Eq. 2 & 3 and use the same functional form of the lower bound on different variants of the mutual information as their objectives. We elaborate which mutual information each prior method optimizes next.
For DIAYN (Eysenbach et al., 2019) and VISR (Hansen et al. 2020), both representation learn- ing and policy learning objectives
(up to some constants) lower bounds on I(S; Z) 2 Epp(s) [log p()) + "Bpee.a)loog ((2 s)], where 2 denotes > up to constant scaling or shifting. See Eq. 2 & 3 of (Eysenbach et al. 2019 and Eq. of (Hansen al. 2020 for details.
For VIC (Gregor et al. 2016), both representattion learning and policy learning objectives are lower bounds 7(ST; Z
5o) 2 (ogg so)] + Epp(rr.))))) [log((2 q(= Sn, 8T)], where So
19

---

Published as a conference paper at ICLR 2025
Algorithm 2 A General Mutual Information Skill Learning Framework
1: Input state representations Φ + IR latent skill distribution p(z), and skill-conditioned policy : S 2 + (AA).
2: for each iteration do
3: Collect trajectory with ~ p(2) and a x(a s, and then add r to the replay buffer.
4: Sample = s's,s',)) from the replay buffer.
5: Update & by maximizing lower bound (S, S"; Z) constructed using B.
6: Relabel the intrinsic reward as a lower bound on (S, ; (S, x(5)).
7: Update æ using an off-policy RL algorithm with B U (r(8,8',)))
8:
Return o and n.
denotes the random variable for initial states and Sr denotes the random variable for terminal states. See Eq. 3 of (Gregor al. 2016 for details.
For DADS (Sharma et al.
2020) both representation learning and policy learning objectives are
(approximate) lower bounds T("; Z () - (peell))loggg {)] + Eppee (al))) [log 7(s s =)]-
See Eq. 5 & 6 of (Sharma et al. 2020 for details.
A.6 THE EFFECT OF THE SCALING CORFFICIENT
In Sec. 5.1 we introduce a coefficient & to scale the negative term LB (Ø) in the contrastive lower bound and find that proper choice of & improves the performance of CSF empirically (Appendix (C.3). This coefficient has an effect similar to the tradeoff coefficient used in information bottleneck optimization (3 in (Alemi et al. 2017)). We also note that when setting & > 1, the new representation objective LB (6) + {LB (6) is still (scaled) contrastive lower bound the mutual information SS, Z). The reason is that LB (Ø) is always non-positive:
LB (0)
E (x.or)
log Epcer) (((0(5))019)) #1]
(a)
el('' Becesla(]
(s.n")
(b) - Epp((ss (s,at) [loge
0,
where in (a) we apply Jensen's inequality and in (b) we use the symmetry of UNIF(Sd-1), Therefore,
for any & 2 1, we have LB (d) + ELB (Ø) < LB + LB (ω).
A.7 A GENERAL MUTUAL INFORMATION SKILL LEARNING FRAMEWORK
The general mutual information skill learning algorithm alternates between (1) collecting data, (2) learning state representation & by maximizing lower bound on the mutual information (5, S" Z) under the behavioral policy 3, (3) relabeling the intrinsic reward as lower bound the information bottleneck (S, S" Z) (S, S¹; o(S") - o(S)), and finally (4) using an off-the-shelf off policy RL algorithm to learning the skill kklll-condiiooned policy We show the pseudo-code of this algorithm in Alg. 2
A.8 CONNECTION BETWEEN REPRESENTATION LEARNED BY METRA AND CONTRASTIVE REPRESENTATIONS
In our experiments (Sec. 6.2), we sample 10K pairs of (s, s', =) from the replay buffer and use them to visualize the histograms of conditional differences in representations d(s') o(s) and normalized marginal differences in representations (ø(s') - (()))/ Φ(s) 2-2 The resulting histograms
(Fig. (Center) & (Right)) indicate two intriguing properties of representations learned by METRA. First, given a set of skills {z}. the differences in representations subtracting the corresponding skills
di Φ(s) z converges isotropic Gaussian distribution:
20

---

Published as a conference paper at ICLR 2025
Claim The state representations & learned by METRA satisfies that ((6') S NN or, equivalently, 6(s') (b()) S oZ,,,0))) where 4n denotes convergence in distribution and as is the standard deviation of the isotropic Gaussian.
Second, taking the marginal over all possible skills, the normalized difference in representations (ó(s') ((x)) / (())ll2 converges to uniform distribution on the d-dimensional unit hypersphere S
Claim The state representations & learned by METRA also satisfy o(d')-((s) [(=())))())]=
4 UNIF(S
We next propose Lemma that relates isotropic Gaussian distribution von Mises-Fisher distribution (Wikipedia) (2024) and then draw the connection between Claim aad Claim
Lemma 1. Given an n-dimensional isotropic Guussian distribution N N(, ( with ||al22 rp von Mises-Fisher distribution VMF (n/Tus
///a²) (p//uoTp./0 can be obtained by restricting the support to be hypersphere with radius Tp- i.e., {x: ||2||2 rpt-
Proof. The probability density function of (H, I) be written as
p(x)
1
20 (x 11) (x (1) 2
exp
(2TTT)
1
2x (1|=112 2x T 1 + 11101(3)
exp
(2TTO)
3
When conditioning on ||ا|||2 Tuu- we have
20 (1
2x T a + [1xa12)
(2r 2r 11)
exp
exp
(2TT)
(2
2
11
r
exp
(2TT))
02 T4
Fu
11
x cxp
Pp
After recomputing the normalizing constant, we recover the probability density function of the von Mises-Fisher distribution vMF (2/Tu:T,//0")
Since the didactic experiments in Sec. 6.2 have shown that ((s') ((s) 2 converges to a Gaussian
distribution (V=, T) (Claim]) 1 and note that ||=||2 1, by applying Lemma A.8, we conjecture
that restricting ((s)) d(s) within {1lo('') <(s)]12 1} produces Mises-Fisher distribu-
tion, i.e., ((8))o(s)
derive the marginal density of 1 4, 100(). Furthermore,
d(e') ((+)) To(x') -(
((s') - d(s)) -(0))
= =f)===/(a
-sbete p(>)p ( d(s') o(s)
p [o(ss)) 0(s)112)
(T(((s))
Φ(s) z)
))dz dz
(e) Cz((0) / Ca
(Ø(s') - dss))
exp
dz
0²
[[o(s') o(s)]12
Ca(0),
where in (a) the symmetric property of the density function of von Mises-Fisher distribu- tions. Crucially, the marginal density indicates that To(s') follows a uniform distribution UNIF()d-1), which is exactly the observation in experiments (Claim 2
o(s))-((s)
21

---

Published as a conference paper at ICLR 2025
A.9 INSIGHTS FOR THE INNER PRODUCTION CRITIC PARAMETERIZATION
Our ablation studies in Sec. 6.3 shows that the inner product parameterization of the critic f( x',,)',z) (Ø(5') - ((s)) is important for CSF (Fig. 3 (Right)). There are two explanations for these observations. First, the inner product parameterization tries to push together the difference of the representation of transitions o(s') o(s) and the corresponding skill 2, instead of focusing on representations of individual states (ss) or ó(s'). This intuition is inline with the observation from prior work that mutual information ((; Z) is invariant to bijective mappings on (See Fig. of (Park et al. 2024), ) e.g.. translation and scaling, indicating that maximizing the mutual information between change of states and skills (S, S" Z)) encourages better state space coverage. Second, the inner product parameterization allows us to analytically compute the second-order Taylor approxi- mation Proposition 2, drawing the connection between the METRA representation objective and contrastive learning.
A.10 INTUITION FOR ZERO SHOT GOAL INFERENCE
In zero-shot goal reaching setting, we want to figure out the corresponding latent skill z when given goal state or image g; problem which can be cast as inferring the Z that maximizes the posterior p" (z s, 9). Since the ground truth posterior p² s.g) is unknown, typical workaround first estimating a variational approximation of p" (z 8.g) and then maximizing the variational posterior g(= 8, g). We provide an intuition for zero-shot goal inference by specifying the variational
posterior as
p(z)e((la)- -(+)))
q(= s,
Ep(=) (c(((a))(())) T
and solving the optimization problem in the latent skill space 2
arg max log q(2 s,g),
=E2
or equivalently,
arg max (ó(g) - d(s)) s.t. || 1.
Taking derivative of the Lagrangian and setting it to zero, the analytical solution is exactly z" o(g) (())
To - ((e) suggesting that the heuristic used by prior methods and our algorithm can be understood
as a maximum a posteriori (MAP) estimation.
B EXPERIMENTAL DETAILS
All experiments were run on a combination of GPUs consisting of NVIDIA GeForce RTX 2080 Ti, NVIDIA RTX A5000, NVIDIA RTX A6000, and NVIDIA A100. All experiments took at most 1 day to run to completion.
B. SIMPLICITY OF CSF COMPARED TO METRA
The main differences between our method and METRA (1) directly using the contrastive lower bound on the mutual information (S, S' Z) as the representation objective, and (2) learning policy by estimating the successor features. which a vector-valued critic, instead of scalar Q in an actor-critic style. Our method can be implemented based on METRA by making three changes (see algorithm summary Sec. 5.2). Since the policy learning step only requires changing the output dimension of neural networks, CSF reduces the number of hyperparameters in the representation learning step compared to METRA. Next, we provide a hyperparameter comparison between CSF and METRA.
On the one hand, since our algorithm prevents the dual gradient descent optimization in the METRA representation objective, do not have the slack variable do not have to specify the initial value of the dual variable X, and remove the dual variable optimizer with its learning rate, resulting in three less hyperparameters. On the other hand, we introduce one coefficient scale the negative term
22

---

Published as a conference paper at ICLR 2025
Table 1: CSF hyperparameters for unsupervised pretraining.
Hyperparameter
Value
Learning rate
0.0001
Horizon
200, except for 50 in Ki tchen
Parallel workers
8, except for 10 in Robobin
State normalizer
used in state-based environments only
Replay buffer batch size
256
Gradient updates per trajectory collection round
50 (Ant, Cheetah) 200 (Humanoid,
Quadruped), 100 (Kitchen) chen, Robobin) 3 for image-based, n/a for state-based
Frame stack
Trajectories per data collection round Automatic entropy tuning
8, except for 10 in Robobin yes
& (scales second term in Eq. 10
5
Number of negative zs compute LB_ (0)
256 (in-batch negatives)
EMA 7 (target network)
5e-3
Ø, n, ş network hidden dimension
1024
Ø, n, network number of layers
I input, hidden, 1 output
Φ, n, & network nonlinearity
relu (ø), tanh (T), relu (+))
LB
LB2()) (Ø) in the contrastive lower bound, which has an effect similar to the tradeoff coefficient used information bottleneck optimization (3 in (Alemi al.) (017)). Taken together, CSF uses three less hyperparameters than METRA and introduces one hyperparameter to balance the positive term (LB (ø)) and the negative term (LB (ø)) in the representation objective.
B.2 EXPERIMENTAL SETUP
Environments. We choose to evaluate on the following six tasks: Ant and Hal ffafetheea from Gym Todorov et al. 2012 Brockman et al. 2016), Quadruped and Humano iiciiddd d from DeepMind Control (DMC) Suite (Tassa et al. 2018), and Kitchen and Robobin from LEXA (Mendoncal et al., 202T). We choose these six tasks to be consistent with the original METRA work (Park al., 2024). In addition, we added Robobin n as another manipulation task since the original five tasks are all navigation tasks except for Kit chen. The observations are state-based in Ant and HalfCheetah and 64 x 64 RGB images of the scene in all other tasks.
Baselines. We consider five baselines. (1) METRA (Park et al., 2024) the state-of-the-ar approach which provides the motivation for deriving CSF. (2) CIC (Laskin et al. 2022) uses rank-based contrastive loss (InfoNCE) to learn representations of transitions and then maximizes state entropy estimate constructed using these representations.
(3) DIAYN (Eysenbach et al. 2019) represents a broad category of methods that first learn parametric discriminator q(z 8, (or q(> s)) to predict latent skills from transitions and then construct the reverse variational lower bound on mutual information (Campos et al. 2020) as an intrinsic reward. (4) DADS (Sharma et al. 2020) builds upon the forward variational lower bound on mutual information (Campos al. (2020) which requires maximizing the state entropy h(S) to encourage state coverage while minimizing the conditional state entropy h(S Z) to distinguish different skills. There is a family of methods studying variational approximations of h(S) and h(S Z) (Campos et al. 2020 Liu & Abbeel
2021 Laskin et al., 2022 Lee et al. 2019 Sharma et al. 2020) which DADS is representative. (5) VISR (Hansen et al. 2020) is similar DIAYN in that it also trains the representations ø by learning discriminator to maximize the likelihood of skill given a state, though the discriminator is parametrized as vMF distribution. In addition, VISR learns successor features that allow it to perform GPI as well as fast task adaptation after unsupervised pretraining. Note that our version of VISR does not include GPI since evaluate continuous control environments.
23

---

Published as a conference paper at ICLR 2025
Table 2: Skill dimensions per method and environment. We list the skill dimension for all methods and environments reported in the paper.
Ant
HalfCheetah
Quadruped
Humanoid
Kitchen
Robobin
CSF
2
4
8
4
9
METRA
2
16
4
2
24
9
DIAYN
50
50
50
50
50
50
DADS
3
CIC
64
64
64
64
64
64
VISR
5
5
5
5
(States)
HalfCheetah (States)
Qua Guadruped (Pixels)
State Coverage
200
300
2000
1000
1.0
100
le7
Humanold (Pixels)
Kitchen (Pixels)
Robobin (Pixels)
100
4000
2
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.2
0.4
0.5
0.3
1.0
Steps
Staps
Steps
CSF (ours)
METRA
DIAYN
+ DADS
CIC
VISR
Figure 5: State space coverage. We plot the unique number of coordinates visited by the agent, except for Kitchen where we plot the task coverage. We find CSF matches the prior wate-of-the-ar MISL algorithms on 4/6 tasks, and strongly outperforms METRA in Robobin. Shaded areas indicate one standard deviation.
B.3 EXPLORATION PERFORMANCE
Please see Fig, for the full of exploration results. We can see that CSF continues to perform on par with METRA, while sometimes outperforming METRA (Rebobin) and sometimes underperforming METRA (Quadruped).
For CSF, all tasks were trained with continuous 2 sampled from a uniform vMF distribution and À 5. METRA also uses continuous 2 sampled from uniform vMF distribution for all environments except for Hal fCheethh and Kit chen, where we used one-hot discrete z, consistent with the original work (Park al. 2024). CIC uses a continuous z sampled from a standard Gaussian for all environments. DIAYN uses one-hot discrete for all environments. DADS uses a continuous sampled from a uniform distribution on [-1, 1] for all environments. Finally, VISR uses a continuous sampled from uniform vMF distribution for all environments. Please refer to Table 2 for full overview of skill dimensions per method and environment. A table with all relevant hyperparameters for the unsupervised training phase can be found in Table [
B.4 ZERO-SHOT GOAL REACHING
Please see Fig. 6 for the full set goal reaching results. We find CSF to generally perform closely to METRA, though slightly underperforming in Quadruped, Humanoid, d, and Kitchen. In Ant however, CSF outperforms METRA.
Goal sampling. We closely follow the setup in Park et al. (224). For all baselines, 50 goals are randomly sampled from [-50, 50] in Ant, [100, 100 in Hal aCheetah, [-15, 15] in Quadruped,
24

---

Published as a conference paper at ICLR 2025
Ant (States)
HalfChectah (States)
Quadruped (Pixels)
Staying Time Fraction
0.25
8.10
0.1
0.00
Env Steps
107
Steps
107
Env Steps
Humanoid (Pixels)
Kitchen (Pixels)
Robobin (Pixels)
0.3
0.4
0.2
0.2
Staingg
6.1
6.
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Steps
Steps
Env Steps
CSF (ours)
METRA
DIAYN
VISR
Figure 6: Goal reaching. We compare CSF with baselines on goal-reaching tasks. We find that CSF achieves strong performance on Ant and mostly outperforms DIAYN and VISR. However, CSF lags bit behind METRA on Quadruped, Ki Kitchen, and Humano d. All means and standard deviations are computed across ten random seeds. Shaded areas indicate one standard deviation.
and [-10, 10] in Humanoid, In Kitchen, we sample 50 times at random from the following built-in tasks: BottomBurner, LightSwitch, SlideCabinet, HingeCabinet, Microwave, and Kettle. In Rebobin, we sample 50 times at random from the following built-in tasks: ReachLeft, ReachRight, PushFront, and PushBack.
Evaluation Unlike prior methods PPak et al. 2022 2024 Sharma et al., 2020 Mendonca et al.) 2021]. we choose the staying time fraction instead of the success rate as our evaluation metric. The staying time indicates the number of time steps that the agent stays at the goal divided by the horizon length, while the success rate simply indicates whether the agent reaches the goal at any time step. Importantly, high success rate does necessarily imply high staying time fraction (e.g., the agent might overshoot the goal after success).
Skill inference. Prior work (Park et al., 2022 2024 2023)
has proposed simple inference method ((g)=
d(a) + where g indicates the goal. We choose to use the same approach for CSF and METRA and provide some theoretical intuition
by setting the skill to the difference in representations
this strategy in Appendix A0. For DIAYN, we follow prior work (Park al., (024) and set one hot (arg max; g(2]g),)
B.5 HIERARCHICAL CONTROL
Please see Fig. 7 for the full set of hierarchical control results. We find CSF to perform closely to ME- TRA most environments, though it outperforms METRA on AntMu 1 Goal and underperforms METRA QuadrupedGoal. CSF outperforms all other baselines on all environments.
We use SAC (Haarnoja et al. 2018) for AntMult Goal, HumanoidGoal, and QuadrupedGoal. We use PPO (Schulman et al. (017) for CheetahGoal and CheetahHurdle. For all state-based environments, we initialize (and freeze) the child policy with a checkpoint trained with 64M environment steps. For image-based based environments, checkpoints trained with 4.8M environments. A table with all relevant hyperparameters for training the hierarchical control policy can be found in Table 3
25

---

Published as a conference paper at ICLR 2025
Table 3: CSF hyperparameters for hierarchical control.
Hyperparameter
Value
Learning rate
0.0001
Option timesteps length
25
Total horizon length
200
Parallel workers
8
Trajectories per data collection round
8, except for Cheetah where we use 64
Algorithm
SAC, except for Cheetah where PPO
State normalizer
used in state- based environments only 256
Replay buffer batch size
Gradient updates per trajectory collection round
50, except for Cheetah where we use 10
Frame stack
3 for image-based, n/a for state-based
(parent, child) networks hidden dimension
1024
(parent, child) networks number of layers (parent, child) networks nonlinearity
input, hidden, output tanh
Child policy frozen?
yes
AntMultiGoal
HalfCheetahGoal
HalfCheetahHurdle
75
50
Betum (past
40
20
3.0
0.2
0.4
0.6
3.8
1.0
Env Steps
Env Staps
Enw Steps
QuadrupedGoal
HumanoldGoal
CSF (ours)
(past
80
DIAYN
20
DADS
8.00
0.0
VISR
Env
Ent Steps
165
Figure Hierarchical control. We compare CSF with baselines on hierarchical control tasks using returns averaged over the 10 past evaluations. We find CSF to perform mostly competi- tively compared to METRA, outperforming METRA AntMultiGoal, but underperforming in QuadrupedGoal (and to small extent in Hal FCheetah@oal and HumanoidGoaa All means and standard deviations are computed across ten random seeds. Shaded areas indicate one standard deviation.
C ADDITIONAL EXPERIMENTS
C.I QUADRATIC APPROXIMATION OF (o)
We conduct experiments to study the accuracy of the quadratic approximation in Prop. 2 in prac- tice. To answer this question, we reuse the METRA algorithm trained on the didactic Ant environment and compare log (e((e) #] against [[ø(x') ((s) We can compute log (e0))))
in our experiments. Results in Fig. shows clear linear relationship between log (p(z) (e(0"))) =] and ||ő(s') Φ(s) sug- gesting that the slope of the least squares linear regression is near the theoretical prediction, i.e., (d) 0.25 0.2309. We conjecture that this linear relationship still exists for higher dimensional and, therefore, the recond-order Taylor approximation proposed by Prop. is practical.
analytically because
C.2 METRA AND CSF ARE SENSITIVE TO THE SKILL DIMENSION
METRA leverages different skill dimensions for different environments. This caused us to investigate what the impact of the skill dimension exploration performance is. In Fig. g we find that both
26

---

Published as a conference paper at ICLR 2025
044
3000
log Epgie win w000
As(d) - 0.2309
Coverage
2500
2000
0.2
500
CSF (2)
CSF (32)
METRA (B)
0
10t5) 41511
Env Steps
Figure 9: Ablation of the skill dimension. CSF second order Taylor approximation of and METRA lesser extent) are sensitive to the dimension of skill (indicated in parentheses).
Figure {o(d)] (()) is
log pp)) ( (((((())(()))))))))) *]. where (o 2d
2500
3020
2530
2003
2030
Stat:
Am
2
1000
0.0
3.0
(a) À (dual variable initial value)
(b) (slack variable)
(c) Learning of the dual variable
Figure 11: METRA is robust to the initial value of the dual variable X, the slack variable c, and the learning rate of the dual variable.
METRA (to lesser extent) and CSF are quite sensitive to the skill dimension. conclude that skill dimension is key parameter to tune for practitioners when training their MISL algorithm.
C.3 SENSITIVITY OF CSF TO THE SCALING COEFFICIENT E
We conduct ablation experiments to study the effect of the scaling coefficient on the negative
3000
2500
term of the contrastive lower bound (LB (Φ)
State Coverage
as well as investigating our theoretical predic-
tion for selecting compare the state coverage of different variants
in Appendix A.6 We
1500
of CSF with & choosing from (0.5, 1, 2, 10),
1000
plotting the mean and standard deviation over
(6= (F=
(E=5) CSF (E=10)
random seeds. Results in Fig. [0 suggest that
500
CSF 21
increasing & to a higher values (2, 5, 10) boosts
the state coverage of CSF, while 5 < 1 hurts
Env Steps
367
the performance. We choose use E = 5 in our benchmark experiments.
Figure 10: The effect of & in CSE Increasing to slightly higher values (2, 5, 10) boosts perfor- mance of CSE, while a & < hurts performance substantially.
C.4 SENSITIVITY OF METRA TO X, AND THE DUAL LEARNING RATE
In Fig we study the impact on the state coverage of METRA in Ant for various hyperparameters. Specifically, vary the initial value of the dual variable X, the slack variable and the learning rate of the dual variable. We find that METRA is robust to all three hyperparameters. All experiments plot the mean and standard deviation over 2 random seeds.
27

---

Published as a conference paper at ICLR 2025
20
50
-100
-40-
-20
-300
-50
100
(a) Ant
(b) Cheetah
15
15
10-
-10
-10-
-15
-10
15
-15
-30
(c) Humanoia d
(d) Quadruped
Figure 12: Trajectory visualization. We visualize (ar, trajectories of different skills (colors) learned by CSF on (a) Ant, (b) Cheetah, (c) Humanoid, (d) Quadruped, showing that CSF learns diverse locomotion behaviors.
C.5 SKILL VISUALIZATIONS
In Fig. 12 we visualize (z, y) trajectories of different skills learned by CSF with different colors. We find that CSF learns diverse locomotion behaviors. Videos of learned skills different tasks can be found on
C.6 FAILED EXPERIMENTS
We experimented with both CSF and METRA on Min Hack (Samvelyan et al. 202T). This environment has discrete action space and requires the agent to navigate between 5 different rooms using various doors and corridors. None these methods got consistently stable performance from (1) the discrete action space, (2) randomized initial states and dynamics (layout of the rooms), and (3) partial observability.
from Mini-
this environment. We conjecture that the difficulties could come
28


Reviews:
[{"review_id": "qx7JZxnxaa", "reviewer": "Reviewer_8yQh", "summary": "The paper critiques a recent method (METRA), which optimizes a Wasserstein distance for skill learning, and argues that its benefits can be explained within the existing framework of mutual information skill learning (MISL). The authors propose a new MISL method called Contrastive Successor Features (CSF), which retains METRA's performance with fewer complexities (namely fewer hyperparameters but same performance).\nThe paper highlights connections between skill learning, contrastive representation learning, and successor features, and provides insights through ablation studies.", "questions": "- Can the authors elaborate on the assumptions made in the theoretical analysis, and how they might affect the generalizability of the results? It is not clear from the main text if these assumptions are sound and/or restrictive in any significant way.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper provides a thorough theoretical analysis of METRA, reinterpreting it within the mutual information skill learning (MISL) framework. This helps demystify the method and connects it to well-established concepts like contrastive learning and information bottlenecks.\n\nThe presentation is clear and to the point. The writing is excellent, I did not find typos or mistakes.\n\nThe paper includes extensive empirical evaluations, comparing CSF with existing methods across various tasks. This robust experimental setup strengthens the validity of the proposed method.", "weaknesses": "While I appreciate that the paper is presented as an improvement on METRA, I'd have enjoyed more a reading that was presenting a new method that is then shown to be equivalent to METRA under certain conditions.\n\nGiven that the presented method performs are par with METRA, it would also be nice to show where (if anywhere) one fails when the other succeeds. Perhaps partially observed MPDs, more interactive objects or discrete actions spaces would be key in identifying where exactly both methods stand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I1T9F6HK7W", "reviewer": "Reviewer_PYdS", "summary": "This work makes two major contributions. First, it establishes an approximate equivalence that bridges the representation objectives of the state-of-the-art method METRA with contrastive loss, specifically similar to InfoNCE. It shows that the actor objective in METRA is equivalent to the information bottleneck of $I(S, S'; Z) - I(S, S'; \\phi(S') - \\phi(S))$ (lower bounded). Essentially, it uses mutual-information-based skill discovery to elucidate METRA. Building on the analytical framework, the authors propose contrastive successor features to simplify METRA, employing contrastive objectives for representation learning and successor features for policy learning. Results indicate that the proposed method is empirically competitive with METRA. \n\nOverall, I like the analytical framework that unifies METRA with the mutual-information-based skill discovery method, and the theoretical foundation appears solid. Additionally, the results support the hypotheses and propositions, suggesting that the proposed method is even more flexible than METRA. Given these strengths, I would recommend an accept in this initial review.", "questions": "- Are there any theoretical insights supporting the assertion that parameterization is key for contrastive successor features? Did you experiment with different kernel functions?\n\n- Any insights or rough ideas on how to further scale the framework to more complicated cases, perhaps through large-scale pre-training or using foundation models to replace the representation learning component, would be beneficial to include in the main paper", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **[Technical soundness and novelty]** The technical soundness is robust; this work provides a thorough in-depth analysis of the METRA method, finding approximate equivalences with contrastive objectives and the information bottleneck. The analysis leads to a novel method that simplifies METRA, and I found no technical flaws; the method is both novel and solid.\n\n\n- **[Evaluation]** The empirical evaluation effectively validates the hypotheses and theoretical analysis, enhancing the overall persuasiveness of the work.\n\n- **[Presentation]** The presentation is clear and easy to follow.", "weaknesses": "- **[About performances]** A significant question arises especially in the Quadruped experiments, where performance still shows room for improvement compared to METRA. Given that the proposed framework has a similar objective function, fewer hyperparameters, and avoids complex min-max optimization, why does the empirical performance (or at least the rate of convergence) not exceed that of METRA? Any discussion on this would be beneficial.\n- **[About demonstrations]** It would be advantageous to include demonstrations or other forms of visualization for the skills learned, as I did not find this in the appendix code.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NjkXtSS5hP", "reviewer": "Reviewer_8Sfa", "summary": "This work introduces a new approach for learning different locomotor skills without supervision. The paper explains relation between SOTA method (METRA) with mutual information maximization between skills and state transitions, an objective shared by most related works. Following the analysis, it proposes modifications to METRA to make it explicitly maximize such a mutual information. Experiments show that the new approach matches METRA.\n\nOverall, the paper is easy to follow and well-written, the analysis is sound and the experiments are relevant. The main limitation is the novelty of the final approach, which is, in the end, very close to METRA and CIC. It also does not show strong improvements.\n\nI'm balanced but a lean towards borderline reject, I did not find sufficiently strong arguments justifying the proposed approach, compared to METRA (see Questions).", "questions": "The statement line 278 does not hold for all cited methods (like DIAYN), as the \"omitted\" is sometimes a useless constant. This is actually illustrated by Eq 2 and 3.\n\nI also find the argument that the method removes 5 hyper-parameters compared to METRA to be bad faith: \"(1) the ϵ slack variable, (2) the norm constraint value in Eq. 4, (3) the dual gradient descent learning rate, (4) the dual gradient descent optimizer, and (5) the choice of discrete or continuous skills z.\" The choice between discrete/continuous skills is actually a positive thing, the norm constraint value is not a hyper-parameter, and the Lagrange multiplier seems (as far as I understand) to use the same optimizer/learning rate as the rest of the parameters in the original paper. Could you clarify which hyper-parameter actually require tuning and how many hyper-parameters you introduce ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper is easy to follow and well-written, the analysis is sound and the experiments are relevant.", "weaknesses": "The final model is very close to previous work and do not present substantial improvements on the different environment compared to METRA. This is overall acknowledged by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4K3y10eQRA", "reviewer": "Reviewer_eNEc", "summary": "The authors introduce a novel self-supervised skill learning algorithm in the RL setting. Their work is motivated by recent work (METRA) that suggests moving away from the typical MI setting, which they analyse, and show that it could be reinterpreted in the familiar MI setting. In doing so, they create a simplified version of METRA, CSF, which achieves the same performance as METRA. The authors combine both theoretical and empirical evidence to support their claims.", "questions": "- Where did the \"fixed coefficient ξ = 5\" come from/does it have any interpretation? How sensitive is CSR to this hyperparameter?\n- There is a note that there are many possible parameterisations for the optimisation of a lower bound on I(S, S′; Z), but CSR (and METRA) use (ϕ(s′)−ϕ(s))⊤z; an ablation study shows that this is a crucial choice. Can the authors provide a further comment on the importance of the temporal distance metric?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper is incredibly well-written. The message/purpose is clear. There is an extensive literature survey, and the authors' discussion of the relevant material and methodology is informative. Unlike some papers that resort to mathematics unnecessarily, all components seem necessary, and are meaningfully explained in text. The authors perform a relatively large set of experiments to both show the performance of their algorithm, but also to back up other claims (such as the properties of representations learned). I also commend the authors for the extended information in the appendices, and also for providing code (I checked these briefly but not extensively).", "weaknesses": "I do not believe there are any substantive weaknesses of this work, but there are some questions the authors could address. As acknowledged by the authors in their own Limitations section, it is unclear how well these algorithms scale beyond the relatively simple MuJoCo benchmarks, but the authors have performed a significant amount of experiments on these domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)