Exploration by Learning Diverse Skills through Successor State Representations
Authors: Paul-Antoine LE TOLGUENEC
Yann BESSE
Florent Teichteil-Königsbuch
Dennis George Wilson
Emmanuel Rachelson
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Exploration
Deep Learning
Primary area: reinforcement_learning

Abstract: The ability to perform different skills can encourage agents to explore. In this work, we aim to construct a set of diverse skills that uniformly cover the state space. We propose a formalization of this search for diverse skills, building on a previous definition based on the mutual information between states and skills. We consider the distribution of states reached by a policy conditioned on each skill and leverage the successor state representation to maximize the difference between these skill distributions. We call this approach LEADS: Learning Diverse Skills through Successor State Representations. We demonstrate our approach on a set of maze navigation and robotic control tasks which show that our method is capable of constructing a diverse set of skills which exhaustively cover the state space without relying on reward or exploration bonuses. Our findings demonstrate that this new formalization promotes more robust and efficient exploration by combining mutual information maximization and exploration bonuses.

Exploration by Learning Diverse Skills through Successor State Representations
Paul-Antoine Le Tolguenec
Yann Besse
ISAE-Supaero Airbus
Airbus yann. www.beeee@aibbusccom
paul nall-antoine.. 1e- p hew.ooogooeeeeerrrruuucoooo
Florent Teichtel--Konigsbuch Airbus
Dennis G. Wilson
ISAE-Supacro, Université de Toulouse
florent. wew..eechhel--ooiigaucaaaaaaaaaa com
dennis. wwlllonnisse-ssppeeeeeeee
Emmanuel Rachelson
ISAB-Supaero, Université de Toulouse emmanuel raacieelsoneiiia-sssaerro...
Abstract
The ability to perform different skills encourage agents to explore. In this work, we aim to construct a set of diverse skills that uniformly cover the state space. We propose formalization of this search for diverse skills, building previous definition based on the mutual information between states and skills. We consider the distribution of states reached by a policy conditioned each skill and leverage the successor state representation to maximize the difference between these skill distributions. We call this approach LEADS: Learning Diverse Skills through Successor State Representations. We demonstrate our approach on a set of maze navigation and robotic control tasks which show that our method is capable of constructing a diverse set of skills which exhaustively cover the state space without relying on reward or exploration bonuses. Our findings demonstrate that this new formalization promotes more robust and efficient exploration by combining mutual information maximization and exploration bonuses.
Introduction
Humans demonstrate outstanding ability to elaborate repertoire of varied skills and behaviors, without extrinsic motivation and supervision. This ability is currently captured through the problem of unsupervised skill discovery in reinforcement learning [41, where one seeks to learn finite set of policies in given environment whose behaviors are notably different from each other. Implicitly, secking behavioral diversity is also related to the question of efficient exploration,
set of skills that better covers the state space is often preferable. Secking diversity has recently been successfully studied through the prism of mutual information maximization 17 13 g. In this article, argue maximizing mutual information might be ambiguous when seeking exploratory behaviors and propose an alternative, motivated variant. The key intuition underpinning this formulation is that good set of exploratory skills should maximize state coverage, while preserving the ability to distinguish skill from another. Then, propose new algorithm which implements this generalized objective, leveraging neural networks as estimators the state occupancy measure. This algorithm demonstrates better exploration properties than state of the methods, both those designed for reward-free exploration and those seeking skill diversity.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

We motivate our developments with first illustrative toy example. Let consider reward- Markov decision process [38] MDP] (S, A, y, 66) where S is the state space, A the action space, P the transition function, the discount factor and do the initial state distribution. A behavior policy, parameterized by 0, maps state to distribution over actions. We define skill encoding E TR as an abstract set of skill descriptors that enhances the policy description and conditions the action taken a ~ (5, (,,5,)). 2), making the policy a function т : θ x S x R + ((A). Unsupervised skill discovery generally considers finite collection Z { heg 1.m] of skills, and distribution p(=) on skills within 2 (generally uniform). For a given 8, each skill 2 induces state distribution p(s[2). The state distribution under the full set of skills hence p(s)
Σ p ())) When maximizing diversity,
we aim to find 0* such that, for any pair of skills 21, 22 2, the states visited by each skill are as
separable as possible. Additionally, we would like the state coverage of the full set of skills to cover as much of the state space as possible.
Maximizing diversity has been formalized in the literature as the maximization of the mu- tual information I(S, Z) between the state ran- dom variable S and the skill descriptor Z. This mutual information (39 MI] quantifies how
X
predictable S is, when Z is known (and con- versely):
I(S, 2) & (PPS, ())(()))(() =H(S) - H(S)Z),
2
Figure 1: State distributions of two sets 21 (left) and Z2 (right) of four skills each on grid maze. Each skill's visited states represented by dif- ferent symbol and distributed uniformly. The gray boxes unreachable.
where D is the Kullback- Leibler divergence and H. is the (conditional) entropy.
Figure Iillustrates why I(S, Z) an imperfect measure of diversity. In this figure, two sets 21 and Z2 of four skills each are represented by their respective state coverage in a grid maze MDP (one symbol per skill). In each set of skills, each skill is picked with probability p(=) = 1/4. On the leftmost image (Z1), each skill covers single state, hence p(s|2) 1 in this state and otherwise. On the rightmost image (Z2). p(s|2) = 1/2 on each covered state and otherwise. Consequently, In (S, Z) (8) (S) Z) = log 4 and Iz(S, (S, Z) H2(S) (S) log 4. The mutual information is hence ambiguous as it ranks the two sets of skills equally, while 2₂ enables better exploration.
Our contribution structured around maximizing a variant of I(S, Z). using the successor state representation [4] to enable its estimation. Section puts this idea in perspective of the relevant literature, highlighting similarities and contrasts with previous works. Then Section introduces and discusses our LEADS algorithm. Section S reports empirical evidence of the good exploratory properties LEADS, demonstrating its ability to outperform state-of-the-art methods on span of benchmarks.
2 Related work
This section endeavors to provide the reader with a comprehensive overview of approaches which aim exploration in RL and skill diversity.
Exploration bonuses. Driving exploration by defining reward bonuses has been a major field of research for over two decades [25] 6 36 2 19). These methods generally involve defining reward bonuses across the state space, which decrease over time in well-explored states. large state spaces where count-based methods are challenging, these approaches require function approximation to generalize across the state space. However, as explained by Jarrett et al. 23] identifying these exploration bonus approximators is an adversarial optimization problem making these approaches unstable in practice [[2].
Quality Diversity. Exploration can also be achieved as by-product of behavior diversity search. Evolutionary algorithms have been used search for new behaviors, such as in Novelty Search 277. and for diversity of behaviors which encourage high performance through the growing domain

---

of Quality Diversity algorithms, exemplified by MAP-Eliter [T]. usually characterized through hand-crafted behavior descriptors, although there are methods which learn such descriptors through variational inference [T0 9 Diversity is expressed as the coverage over such descriptors, and policies are modified through gradient-free [1]] or gradient-based [32] updates in order to incrementally populate an archive of diverse skills. While these algorithms have demonstrated competitive results with exploration algorithms in RL [9] in general, they require expertise to define behavior descriptor and are less sample-efficient than RL methods (28).
In these algorithms, skills are
Information theory approaches diversity. The field of information theory offers valuable insights into designing algorithms that generically promote diversity. Gregor et al. [17]. Eysenbach et al. [13 and Sharma al. [40] were pioneers in proposing methods to maximize either the reverse or forward form of [SS Z) for a fixed set of skills Z. To address the issue of unbalanced state distributions across skills throughout the entire environment, Campos et [ suggest sequencing the problem into three stages: pure exploration in S to promote sampled states diversity, state clustering via variational inference, and skill learning by maximisation of the forward form as proposed by Sharma al. [40]. In the exploration phase, Campos et al. [ employs the State Marginal Matching algorithm proposed by Lee et al. [26). which resembles MI-based algorithms as combines mutual information maximization with an exploration bonus efficiently explore the state space.
Focusing even more on exploration, Liu and Abbeel 29 automate learning Novelty Search behavior descriptors by training an encoder with contrastive learning in order to promote exploration. Focusing also on exploration, Kamienny et al. 22 employ an incremental approach combined with the maxi- mization of the reverse form of MI to eliminate skills that are not discernable enough, subsequently generating new skills from the most discernable ones.
Finally, recent studies including those by Park et al. 133 Park et al. [34], and Park et al. [35], integrate mutual information (MI) with craecttoyy-base metrics between states to enhance exploration. Specifically, these studies aim to maximize the Wasserstein distance, also known as the Earth mover's distance, between IP(S, Z) and P(S)P(Z)) (Z) while the definition of MI uses the KL divergence. This
distance requires specifying a metric for the state space. Park et al. (33) and Park et al. [34 uses the Euclidean distance, while Park et (35 employs the temporal distance.
is important to note that, although these methods induce an interesting approach to exploration (as a by-product of diversity), they do not explicitly aim encouraging coverage of the MDP's state space.
Successor features for MI maximization. Few attempts have been made to exploit universal successor features (5) in MI maximization methods. Machado et al. (30] decompose the learning process into two stages: first, they automatically acquire specific options, which are then used to develop a hierarchical policy that integrates these options. In contrast, Hansen et al. 21 first learn general task-based policy conditioned on a task 2, and subsequently optimize the reverse form of MI by utilizing universal successor features to identify range of diverse skills efficiently. More recently, Grillotti et al. [18] used successor features to optimize the quality of predefined behaviors based on the Quality Diversity formulation, which requires hand-crafted behavior descriptors.
The method we propose explicitly aims at promoting diversity with the goal of achieving good exploration. For this purpose, we turn the MI objective function into new objective which promotes exploration, separating this work from previous contributions in MI maximization.
3 Promoting exploration with successor state representations
We now derive an algorithm that explicitly encourages exploration in diversity search. We start by casting mutual information maximization in terms of state occupancy measures and derive lower bound which can be maximized with stochastic gradient ascent on the policy's parameters (Section 3.1). Then, as maximizing MI does not encourage exploration, we transform this lower bound so that its maximization fits this goal, leading to new target quantity designed to promote both diversity and exploration (Section 3.2). This enables defining and implementing a general algorithm for Learning Diverse Skills through successor state representations (LEADS, Section 3.3
3

---

3.1 MI lower bound with successor state representations
The mutual information I(S, Z) is naturally expressed through the (conditional) probability densities of S and Z. Successor state representation estimators [4] SSRJ provide a direct way of estimating these densities. More precisely, we identify the Successor State Representation as the density of the measure defined the set of states. The SSR (( 0 2) of skill (0, 2) is the state occupancy measure of policy To (s, 2), when starting from 81, that is
01,2|s1,0,21
81 82
ae To(8e,
80 81,
(I)
t=0
For the sake of readability, we make implicit in our notations going forward. This measure can be understood as the probability density of reaching the state starting from another state when the skill is used [22]. It encodes all the paths of the Markov chain that can be taken from 81 to 82 when the skill is used 4 Here (81, 82) E 8 can be any two states of the Markov chain.
The SSR is the fixed point of Bellman operator and one can employ temporal difference (TD) learning to guide function approximator towards this fixed point. There are numerous methods
for estimating the SSR density 22 [5). In this paper. we use C-Learning [14] for estimating
the SSR for all experiments. C-Learning proposes to cast the regression problem of learning as a classification problem. Specifically, for a given pair (s1, a, 2), a classifier (f 0, 82, z)), where a is the sigmoid function, is trained to predict whether $2 was sampled from an outcome trajectory from $1 (s2 p(-1s1, a, 2)) or from the marginal density of all possible skills (82 ~ p(s), the distribution over S). The optimal classifier holds the following relation with the SSR's density: efo(e....2..)) p[==2(11 (81, a, =)/p(s2). We denote m a. 82, =) ele(e1.0,22.3)) as the SSR estimated via C-Learning taking a (81. 2). For the sake of brevity, we will use the notation mo(81, 82, (=1.e))
E [mo(11, a, 82, =]]. We note that any method which reliably estimates
mp $2, $2, 2) can be used instead of C-Learning in LEADS.
As described in Section the mutual information between the state random variable S and the skill descriptor 7 is I(S, Z)= = D (PPS, ()))P())P(Z)). This can be expressed as
I(S, Z) = ((s [log (p(s2 <)/p(s>))]
(2)
State 52 can be any state within the set S. yet we refer to it 82 ensure that when incorporate the definition of the SSR, aligns directly with the notation of Equation By definition of the SSR under given policy (s, 2), the state distribution p(s|2) obeys
p(52|z) E,, <p(sle) [p(s2] [$1, =]]
By sampling p(s|2), we can therefore estimate p(s2lz), allowing for the injection of m(s1, 82, 2) p(22|11.
(1, 2) (s22) from Eysenbach et al. 14 into Equation
I(S, 2) ((22,)))p(s.s) [log (Eer=p(()=)) (m(s1, $2, =)))]
Monte Carlo estimator of (S, Z) can be obtained by sampling (s, x) from a replay buffer. However, for every state sampled this way, the estimate of p(s[2) provided by Equation 3 requires sampling a batch of $1 ~ p(s[z). This is possible through keeping separate replay buffers for each skill, but would be computationally intractable for accurate estimation. Hence, in similar methods, turn rather to a lower bound on I(S, Z). which admits a Monte Carlo estimator, and which will be maximized with respect to 0 using stochastic gradient ascent.
Applying Jensen's inequality to the inner expectation in the previous expression yields
I(S, 2) 2 E
[log(m(81,82) 82, =))]
s>p((>) =))=(()(=
$1 ()(())
Note that 0 participates in this lower bound through mn(1,, 82, 2) Eawzo(fle (m a, 82, 2). This quantity is essentially what an algorithm like DIAYN [3] maximizes: given separate experience buffers for each skill, one can compute a Monte Carlo estimate of this lower bound on the gradient of the mutual information I(S, Z) with respect to 0.

---

3.2 Promoting exploration with different skills
Looking closely at Equation 4 one can note that the interplay between each skill's state distribution is lost when deriving the lower bound. Hence, this bound can be maximized without actually pushing skills towards distinct states, hence without skill diversity. To regain the incentive to promote skill diversity, we propose to encourage exploration in each skill to put probability mass on states which receive little coverage by the full set of skills. other words, for given transition (s, =())). we want to augment the probability of occurrence for one skill while decreasing it for all others.
Since mm(1 8, Z 0, we can introduce the following bound:
m((), 52, 2)
I(S,2)
E
log
(5)
())p(=)
1 +
Σ
m m(81, 82, 21)
agmpps (2)())2
x'E2
While Equation is a looser lower bound than Equation 4 goes towards the goal of diversity search originally captured by MI maximization: to have distinct state distributions for each skill.
However, as illustrated in the introductory example, maximizing MI does not promote large state coverage, nor exploratory behaviors. Equation promotes skill diversity, but it does not explicitly encourage state coverage. We therefore argue that exploratory behaviors can be obtained by focusing the state distribution of each skill towards specific states within the support of p(s[z). Instead of sampling s according p(s[2). we encourage exploration by attributing more probability mass to states that trigger exploration. We do so by creating sampling distribution 6(s|2) based on an uncertainty measure (s, 2). as explained next.
Biasing a skill towards promising states for exploration often achieved through exploratory bonuses based on uncertainty measures (36] 7 [2], or repulsion mechanisms [[6]. These heuristic measures encourage exploration by pushing policies towards states of high uncertainty or away from previously covered transitions. The novelty of our approach is the use of an exploration bonus in the framing of mutual information maximization. However, by replacing $2 ~ p(s[2) by another sampling distribution 8(s[2), we lose the theoretic guarantee of maximizing the lower bound of Equation 5 Rather, we propose new objective. expressing an incentive to explore within the search for skill diversity, but with greater focus on state coverage through distinct skills than previous lower bounds on mutual information. The quantity we maximize is then:
m(51, ag 82, 2)
9(0)
E 2mpp(2)
log 1
(6)
Σ m(81, 02', 82,
(tmp(((a)
Z'EZ
0x momo (81,5)
(a(())
The uncertainty measure in which defines the distribution 6(s[2) is designed to explore under-visited areas and to create repulsion between different skills. We define three desired properties for states to prioritize using this measure. To describe these properties, consider sequence of policies тe. each defining a set of skills (m.( =))ee,, and the corresponding sequence of SSRs my; the three properties are then as follows. (1) A good target state sē for skill at time step is one that has high probability of being visited by (e =), but was relatively infrequently visited by previous policies' state occupancy measures {m
for any skill. (2) It is also state which has both high probability of being reached by the current Te(.,2) and low probability of being reached by any other current skill, starting from sp- (3) Given a previous target state s² good new target state is one that has high (resp. low) probability to be reached by xe(., =) (resp. any other skill), starting from * While the first property explicitly encourages visiting under-visited states, the two others do not encourage exploration per se, and rather strengthen skill diversity by pushing their state distributions apart. This leads to the idea of ranking target states according to:
m4(30, 8,
(5 m2 (s (,,""",,,8"" 8,
mi (S0, 8, 2) my m=(30, 8,
ue(s, 2) = log
+
log
+ log
1== Eg (30. 8, 2))
of #
Explore under-visited areas
Repulsion between skills
5

---

The (s|22) distribution of Equation therefore allocates more probability to states with high (s, 2). Empirically, we found that making deterministic as the Dirac distribution on a state that maximizes wr(s,z) z) enabled efficient exploration. Appendix A provides a more formal perspective on the derivation of the v, uncertainty measure.
3.3 The LEADS algorithm
define the LEADS (Learning Diverse Skills With the objective of maximizing ((0) (Equation 6) through successor state representations) algorithm, presented in Algorithm
In order to maximize (0), we must first sample from p(s 2) for each skill 2, then compute the SSR for the cur- rent skills Ty (:) order to define (()), then finally update / to maxi- mize G()). As such, an iteration of LEADS features three phases in order to learn a new set of skills: sampling, learning the SSR, and optimizing the policy parameters
Algorithm LLADD
Initialize On
for E [0, N] do # Collect samples
D2 = 0, Vz
for E Mepl do
Sample skill 2 ~ p(z)
{s ut Te, s()} episode with To, (1)2) from so
First, the sampling phase populates separate replay buffers D: for each skill = p(=). This is achieved by rolling out Mep episodes with given policy T)( Then, C-learning is used to compute the parameters Φt of joint SSR m
D. = D. [( a, rı, {))}
end for
Learn the SSR
Learn mø, for πo, using on-policy C-learning Sample 8 ~ 8(s[=)
Improve
z) for ne, and store We note that the on- on-policy ver- sion of C-learning can be used for this step as the data in the replay buffers has been collected with the current policy πe. This SSR, and all previous ones, are then used to define us(s,)) 2). which in turn defines the distribution 6(s, 2). As this can be approximated for static policy, sampling can be performed before updating 0. Sampling ~ (ss)2) (2) is performed by running all states in D3 through (,,s, although, to limit computational cost, we can restrict this evaluation and selection to only a uniformly drawn subset of each D..
for i E [1, AsGD] do p(z), $1 p(s]z) 0 - 0 + a Vo + A H(0)
Sample
Update d using off-policy C-learning
end for
end for
We therefore have the necessary components optimize 0 according (()): means of sampling states and the SSR. At each gradient ascent step, we sample a mini- batch of states $1 from D, for given skill p(z) to estimate p(s|2). This permits calculating the objective ((0) for the current using ma and performing a gradient step. In the gradient calculation, we also include an action entropy maximization term, done in other works 31 201
H(()
E [- log(mo(a|s1,
(8)
2><([=]
0 is therefore updated to maximize G(0) + >₁ H()), although A, is intentionally kept small (0.05) to focus on the principal LEADS objective (()).
Finally, after each gradient step, the SSR is updated using the off aff-policy formulation of C-learning so that my remains representative of the state distribution under тo, This done without sampling new transitions and is justified by the fact that the target state 5f and the states along a trajectory to of are already within the replay buffer D.. RsGD steps of gradient ascent on 0 are performed in this way, before new iteration of LEADS is started.

---

Experiments & Results
We demonstrate LEADS on set maze navigation and robotic control tasks and compare its behavior state-of-the-art exploration algorithms. We provide all code for LEADS and the bascline algorithms, as well as the scripts to reproduce the experiments (repository). All hyperparameters are summarized in Appendix C
4.1 Evaluation benchmarks
Mazes. Maze navigation has been frequently used in the exploration literature, as 2D environments allows for clear visualization of the behaviors induced by an algorithm. The assessment of state states. We design
coverage is also easier to understand than in environments with high-dimension three different maze navigation tasks, named Easy, U, and Hard (depicted in Figure P of increasing difficulty in reaching all parts of the state space. In each of the mazes, the state space is defined by the agent's Fuclidean coordinates = [_, and the action space corresponds the agent's velocity A= [-1, 1j². Hitting wall terminates episode, making exploration difficult.
Robotic control. We further assess the capabilities of LEADS in complex robotic control tasks. These tasks allow evaluating the ability of LEADS to explore in diverse and high-dimensional state spaces. We evaluate LEADS on variety of MuJoCo 42 environments from different benchmark suites. Fetch-Reach (3 is 7-DoF (degrees of freedom) robotic arm equipped with two- two-finoge fingered parallel gripper: its observation space 10-dimensional. Fetch-Slide extends the former with a puck placed platform in front of the arm, increasing the observation space dimension to 25. Hand 24-DoF anthropomorphic robotic hand, with 63-dimensional observation space. Finger 3-DoF, 12-dimensional observation space, manipulation environment where planar finger is required to rotate an object on an unactuated hinge. Appendix (epeendixDddiccuseee discusses additional experiments and limitations on other MuJoCo tasks.
37
LEADS
LSD
DIAYN
Skill
Skill
2
Skill
Skill
Figure 2: Skill visualisation for each algorithm. Per algorithm, the tasks are the mazes Easy (top left), U (top right), Hard (bottom left), and the control task Fetch-Reach (bottom right).
4.2 Skill Visualization
The first goal of this work, as exposed in the introductory example, is to obtain a set of skills that are as distinct from each other as possible in their visited states, while simultaneously covering as much of the state space as possible. We first assess this property in LEADS, DIAYN 13] and LSD B3 through visual analysis of their state space coverage in Figure 2 For all experiments, the number of skills is 7.aall 6. To ensure fairness, for each algorithm, we report the skill visualization from the experiment that achieves maximum coverage (as defined in the following section) out of five runs. For each algorithm, Figure 2 presents four figures: the three mazes and the Fetch-Reach environment. In all the mazes, we visualize the 2D coordinates of the states reached over training. Given that visualization is not suitable for the
environment due its higher state space dimension.
7

---

we project the state onto the two first cigenvectors of a Principal Component Analysis (PCA) of the states encountered by all skills in this environment.
We note that LEADS clearly defines distinct skills in the state space. Furthermore, it leads to a more extensive exploration of the environment than LSD and DIAYN. The Hard maze (bottom left) is noteworthy. as some parts of the environment are difficult to access due to bottleneck passages in the maze. LEADS is the only algorithm that manages reach all sections of the Hard maze.
20
303
68
48 284
A85
Figure 3: LEADS exploration of the Hand environment state space, using Rekill = 12 skills and a PCA over all explored states.
(a) Occupancy measure
(b) Uncertainty
Figure 4: (a): The SSR m(50, 8, at given epoch during training on Hard maze, per skill, normalized in [o. 1]. (b): The uncertainty measure u(s, 2) at the final epoch on Hard maze, per skill, with the maximum state highlighted.
Figure 3 further demonstrates the ability of LEADS to create distinct skills in the high-dimensional state space of the Hand environment. reports the state distribution obtained by A'aill 12 skills found by LEADS, projected onto the two first components of PCA on all visited states. As in the maze navigation tasks, LEADS explores states distinct clusters, organized by skill. These skills correspond to visual variety of finger positions, illustrating the ability discover varied behaviors.
Section 3.2 claimed that the uncertainty measure u. (s, z) of Equation 7 and its associated distribution 5 triggered relevant exploratory behaviors. Figures 4(a) and ((b) display the final SSR and uncertainty measure, respectively, for the Hard maze task. The same visual representation of these densities for the other environments can be found in Appendix B Figure 4(a) demonstrates that the SSR clearly separates states based on skill. The SSR zero for all states outside of the distribution of given skill, and all states in skill have a non-zero probability of being reached by that skill. This confirms that this SSR can reliably used within the objective function of Equation 6
Similarly, Figure 4(b) illustrates how the uncertainty measure concentrates on states that promote both exploration and diversity. High values for this measure are found in very different states for each skill (hence promoting diversity), and within a skill, the highest value is found far from the starting state So (hence promoting exploration). The state maximizing u, is represented by colored dot for each skill. This confirms that the uncertainty measure achieves the joint goal of exploring visited areas and creating repulsion between skills. By defining 8(s[2) as the state which maximizes de (s, 2) for each skill 2, LEADS ensures continuing exploration of the state space.
4.3 Quantitative Evaluation
We now turn to a quantitative evaluation of the ability to explore using diversified skills. For the sake of completeness, we compare LEADS to seven seminal algorithms. Five of these are skill-based, namely DIAYN [3], SMM (26), LSD 33 CSD [34, and METRA 35). We use n*sill for each baild-based algorithm and for all environments. The two last algorithms are pure-exploration

---

Easy
Umaze
Hard
120
LEADS
NGU
AND
DAYN
X3D
0.0
0.6
X12
02
0.5
X10
1.0
Fotch Reach
Finger
Fatch Slide
SHH
30
80
60
20
0.5
3.0
2.0
0.8
02
0.6
Figure 5: Relative coverage evolution across six tasks. The x-axis represents the number of samples collected since the algorithm began.
Method
Easy (%)
Umaze (%)
Hard (%)
Fetch Reach (%)
Finger (%)
Fetch Slide (%)
RND
76.6
39.6 + 5,4
30.8 + 4.3
17.6 2.7
1.6
21.6
DIAYN
8.6
43.8 8.2
4.5
$5.6
53.8 12.3
8.5
SMM
0.0
53.4 7.76
22.3 5.5
1..2
5.4
NGU
8.4
7.04
8.3
53.4
5.6
5.5
LSD
9.4
7.5
7..0
7.2
6.8
6.25
CSD
97.4 3.4
7.8
6.20
83.2 0
96.2 9.1
63.0
METRA 92.8 + 4.2
78.0
9.5
82.5 + 1.5
$3.4
50.7
2.2
LEADS
100.0 0.0 96.0 + 4.3 90.8 5.3
89.7 + 8.8
87.4 + 4.6
85.4
7.2
Table 1: Final coverage percentages for each method each environments. Bold indicates when single method is statistically superior to all other methods (p < 0.05). Full T-test results are presented in Appendix
ones, which do not rely on skill diversity and rather rely on exploration bonuses: Random Network Distillation (RND) 7 and Never Give Up (NGU) (2].
A metric of state space coverage should characterize how widespread the state distribution is, in particular across behavior descriptors that are meaningful for the task at hand. Following the practice of previous work, for high- dinensiinnii environments, we use a featurization of the state, retaining a diweensnnional representation of meaningful variables: the (z. y, y. 2) coordinates of the gripper in Fetch-Reach. the (x, y) coordinates of the puck in Fetch-Slide, and the angles of the finger's s two joints and the hinge's angle Finger. This projection is then discretized and the coverage of trajectory is defined as the number of visited cells. Figure 5 depicts the evolution of coverage, normalized by the maximum coverage achieved by any run in the current environment. This figure illustrates how coverage progresses relative to the number of samples taken in the environment since the beginning of the algorithm. Shaded areas represent the standard error across five seeds for each algorithm.
One can note that LEADS outperforms other methods across almost all tasks. This is especially notable in the Fetch-Slide environment, where LEADS exceeds the coverage of all other methods by over 20%. Furthermore, besides LEADS, no algorithm consistently good across all tasks. In the Fetch-Reach task, DIAYN, METRA, and CSD each achieved a coverage between 80% and 85%. LEADS excelled with 90% coverage, significantly outperforming NGU 50% and SMM and RND, both at 20%. CSD surpasses LEADS and all other methods on the Finger environment, achicving

---

relative mean coverage of 96% where LEADS is still competitive with 87%. On this specific benchmark, NGU. METRA and LSD are competitive with LEADS, whereas DIAYN, SMM and RND exhibit low coverage.
5 Conclusion
In this work, we consider the problem of exploration in reinforcement learning, via the search for behavioral diversity in a finite set of skills. We take inspiration from the classical framework of maximization of the mutual information between visited states and skill descriptors, and illustrate why it might be insufficient to promote exploration. This motivates the introduction of new objective function for skill diversity and exploration. This objective exploits an uncertainty measure to encourage extensive state space coverage. We leverage off-the-shelf estimators of the successor state representation to estimate this non-stationary uncertainty measure, and introduce the LEADS algorithm.
Specifically, LEADS estimates the successor state representation for each skill, then specializes each skill towards under-visited states while keeping skills in distinct state distributions. This results in an efficient method for state space coverage via skill diversity search. Our experiments highlight the efficacy of this approach, achieving superior coverage in nearly all tested environments compared to fate-oof-the-ar baselines.
LEADS intrinsically relies good SSR estimators. In this work, used C-Learning [14] for all experiments, which proved efficient but might reach generalization limits in some environments. As consequence, advances the question of reliable SSR estimation will directly benefit LEADS, opening new perspectives for research.
Acknowledgments
This work was performed using HPC resources from CALMIP (Grant 2016-1p21001J.
References
[1] Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems, 21, 2008
[2] Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi. Daniel Guo, Bilal Piot, Steven Kapturowski. Olivier Ticleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.0603: 2020.
[3] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[4] Léonard Blier, Corentin Tallec, and Yann Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. arXiv preprint arXiv:2 2021.
[5] Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado Van Hasselt, David Silver. and Tom Schaul. Universal successor features approximators. arXiv preprint 1rXiv:181..07626. 2018.
[6] Ronen Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, ((0et):213-33,, 2002.
[7] Yuri Burda. Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12890 2018.
[8] Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference Machine Learning, pages 1317-1327. PMLR, 2020.
10

---

[9] Felix Chalumeau, Raphael Boige, Bryan Lim, Valentin Macé, Maxime Allard, Arthur Flajolet, Antoine Cully, and Thomas Pierrot. Neuroevolution is a competitive alternative to reinforcement learning for skill discovery. arXiv preprint arXiv:2210.0:556 2022.
[10] Antoine Cully. Autonomous skill discovery with quality-diversity and unsupervised descriptors. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 81-89, 2019.
[11] Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Bape con-Battiste Mouret. Robots that can adapt like animals. Nature, 521(7553):0335077, 2015.
[12] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore:
a new approach for hard-exploration problems. arXiv preprint arXiv:1901.1099) 2019.
[13] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function, arXiv preprint arXiv: /00.::/0000000 2018.
[14] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine C-learning: Learning to achieve goals via recursive classification. arXiv preprint arXiv:2011.08909, 2020.
[15] Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning, Advances in Neural Information Process- ing Systems, 35:35603-35620 2022.
[16] Yannis Flet- Bet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adver- sarially guided actor-critic. In ICLR 202 In2l-nnteraational Conference on Learning Representa- tions, 2021.
[17] Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07507, 2016.
[18] Luca Grillotti, Maxence Faldor, Borja G León, and Antoine Cully. Skill-conditioned policy optimization with successor features representations. In Second Agent Learning in Open- Endedness Workshop, 2023.
[19] Zhaohan Guo, Shantanu Thakoor, Miruna Pislar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Exploration by bootstrapped prediction. Advances neural information processing systems, 855-31870 2022.
Grill. Yunhao Tang, et al. BYOL-explore:
[20] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018
[21] Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint 1rXiv:1966005030. 2019.
[22] Michael Janner, Igor Mordatch, and Sergey Levine. gamma-models: Generative temporal difference learning for infinite- horizonn prediction. Advances in Neural Information Processing Systems, 33:1724-1777 1734, 2020.
[23] Daniel Jarrett, Corentin Tallec, Florent Altché, Thomas Mesnard, Rémi Munos, and Michal Valko. Curiosity in hindsight. arXiv preprint arXiv:2211.1053 2022.
[24] Pierre-Alexandre Kamienny, Jean Tarbouriech, Sylvain Lamprier, Alessandro Lazaric, and Ludovic Denoyer. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. arXiv preprint arXiv:2/10.144577 2021.
[25] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49:209-232, 2002.
[26] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Rus- lan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint 10X1v:1906.05274.00
2019. 2019.

---

[27] Joel Lehman and Kenneth O Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pages 211-218, 2011.
[28] Bryan Lim, Manon Flageat, and Antoine Cully. Understanding the synergies between quality- diversity and deep reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1212-1220, 2023.
[29] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems,
2021.
[30] Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv: 1711v11710.11089, 2017.
[31] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
[32] Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Proceedings of the Genetic and Evolutionary Computution Conference, pages 866-875, 2021.
[33] Seohong Park, Jongwook Choi, Jackyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz- constrained unsupervised skill discovery. In International Conference on Learning Representa- tions, 2022.
[34] Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllabilly--wware unsuper- vised skill discovery. arXiv preprint 2wt.:020220000,,, 2023.
[35] Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric- aware abstraction. In The Twelfth International Conference on Learning Representations, 2023.
[36] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778- 2787. PMLR, 2017.
[37] Matthias Plappert, Marcin Andrychowicz. Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv: 1802.09444 2018
[38] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[39] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
[40] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. arXiv preprint arXiv:1907.0166 190600077.77.... 2019.
[41] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press. 2018.
[42] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[43] Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL https: / /eoooo hng/recoord/812700
12

---

[44] Saran Tunyasuvunakool. Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. dm dil control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020.
[45] Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding, arXiv preprint 1r11v:2310.20141, 2023.
13

---

A A formal motivation for the uncertainty measure of Equation 7
In Section 3.2 we introduced a new optimization target, based on the uncertainty measure de (s, 2), used in LEADS to strategically distribute probability mass across states for each skill to foster effective exploration. 41(s, 2) introduced in Equation which is reproduced here for clarity:
(1, log
ms( 81
Σ z/
log
11 + log m
m4 (s0 8,
E Jom E m& (30, 8, 2')
(N0 8,
Explore under visited
Repuision between skills
This section delves into the theoretical foundations of this target. We begin by discussing the left term of Equation 7 which corresponds the exploration component of the measure. Subsequently, we detail the right term, which promotes greater repulsion between the skills.
A.1 Exploration term (left side of Equation 7
The Successor State Representation (SSR) extends the concept of the occupancy measure. Unlike the traditional occupancy measure, which begins with the initial state distribution, the SSR is an estimation of the occupancy measure starting from any state. In practice, work with the expected sum of discounted presences state over time because we know this measure exists regardless of the ergodicity of the Markov process.
LEADS is an epoch-based algorithm with each epoch consisting of four phases, as presented in Section 2 and Algorithm
Sample transitions from the environment for each skill,
2. Learn the SSR of each skill (for the given policy),
3. Define state which to put probability mass for each skill,
4. Use the objective of Equation 6 to update the policy so that it implements this probability mass transport plan.
Since the policy changes in each epoch, the discounted occupancy measure of the policy also evolves over time. For given skill and epoch n, can be defined as:
E
sprudo a P(-((Seraa))
Σ 1 =s)
This occupancy measure has direct estimator, using the SSR in the initial state sot
Tz (s) = m, (So. 8, 2)
At epoch the state occupancy measure of all skills is the mixture of all skills' state occupancy measures:
(.)(s)
(()))= z
Then, the left side term the definition of the uncertainty measure u188, 2) corresponds to the contribution of a state to the following Kullback-Leibler divergence:
P (s)
KL
P
= E
log
(9)
Σ (8)
==1
P (s)
S Pn (s) log
ds
k -1 PI (8)
[ 17 (8) log
min Mm( 8,
ds
11 Ee my (Su, 8, 8,2²)
14

---

Note in particular that this divergence accounts for the discrepancy between P ie. the states visited by skill at epoch n, and _k=1 1 Pi: ie. the states previously explored by all skills in all past epochs. Thus, the KL divergence of Equation D serves as a measure of novelty for the current distribution. Intuitively, the newer state visited by the current distribution p (s) is, the larger its contribution to this KL. divergence will be. Recall that at this stage, we do not modify the policy (it is the role of the objective function of Equation [6), but rather aim identifying which states within the support of p% should be those skill z is encouraged to explore next. By defining the state-wise novelty
score log Σ= En (s..s,)))) the distribution 8(s|=) unfolds as the Dirac distribution which maximizes the expected value of this score. Overall, LEADS aims identify and prioritize states with the highest novelty score for each skill, thereby enhancing exploration by shifting each skill's s induced distribution towards these novel states.
(20.80.8.)
A.2 Repulsion term (right side of Equation
Building on the previous discussion about the formal motivation of the exploration term on the left side in 7 we now explore the right side term, which is intended to enhance repulsion between skills.
The right side term of Equation 7 multiple Kullback-Leibe divergences:
can be interpreted as the contribution of a state s within a sum of
Σ <m || mn (S + (L (s) (s))
(10)
(a)
(b)
In this equation, term (a) represents the KL divergence of the occupancy measure for the current skill 2, starting from its last target state, compared to the occupancy measure of each other skill starting from their respective last target states. Maximizing this KL divergence aims to make the distribution of each skill (starting from their last target state) as distinct as possible. Similarly, term (b) is the KL divergence between the occupancy measures of skill and that of any other skill 2/. Maximizing these two KL divergences leads to disjoint distributions for each skill. As in the previous subsection, we do not change the policy at this stage (this would lead to ill-defined optimization problem the KL can become unbounded), but rather aim at identifying which are the states that contribute most to these KL divergences, so that can promote exploration towards them. Overall, LEADS combines all these terms into the uncertainty measure of Equation (ariiionnttto to determine which state contributes the most these KL divergences for each skill. In turn, these states define the distribution 5 designed to promote exploration, by increasing the weight of the most likely states of 5 in Equation 6
A.3 Numerical advantage of Lower Bound 4
This section offers more detailed explanation of the passage from the definition of Mu- tual Information 2 to the Lower Bound 4 The mutual information can be defined as: ((s,,))<p(s.e) [log (Es) [mn(s1; 82, 2)]},, with 8 ~ p(s1|z).
Suppose we wish to maximize this quantity using SGD. For each sample (82, =). we need to compute the expectation E,, (m(s1, 82, =)) (because the log not linear). In practice, this is compu- tationally expensive because it requires sampling mini-batch of 81 states for every single 82 state. Instead, we use Jensen's 's inequality, leveraging the concavity of the log function derive the lower bound:
Ezwp((s) s2n=p(a(z) simp(el=) [log(m(s), 82, =))]
Maximizing this new quantity can be done with SGD by sampling three independent mini- batches at each iteration.
B Supplementary Analyses and Visualizations
LEADS is based on the estimation of the successor state representation for each skill. Figure and
Figure 7 show the final SSR and uncertainty measure, respectively, for the maze and Fetch tasks.
These correspond to the same runs shown in Figure 2 for LEADS.
15

---

XXXXX
Figure 6: The SSR m( s, 2) at a given epoch during training on each task, per skill, normalized in [0, 1].
211
Figure 7: The uncertainty measure u(s, 2) at given epoch during training on each task, per skill, with the maximum state highlighted.
Figure iillustrates the temporal evolution of coverage for all methods across various environments, evaluated over five different seeds. We extend our analysis by presenting the p-values from the t-tests conducted on the distributions the first method to both the second and third methods. We conducted t-test for each environment in which no method outperformed the second method by a margin greater than 10%. The results are visible in the table 2
the final (highest) coverage values for these five seeds, comparing
Environment: Method 1/ Method 2
p-value
Umaze: LEADS/CSD
0.01522
Umaze: LEADS/LSD
0.01152
Hard: LEADS/CSD
0.00246
Hard: LEADS/L.SD
0.00977
Fetch Reach: LEADS/CSD
0.06354
Fetch Reach: LEADS/DIAYN
0.08307
Finger: CSD/LEADS
0.07072
Finger: CSD/LSD
0.00753
Fetch Slide: LEADS/CSD
0.00070
Fetch Slide: LEADS/LSD
0.00055
Table 2: Comparison of p-values for paired t-tests across methods
16

---

B.1 Key Differences with Baseline Methods
The LEADS algorithm learns diverse skills that evolve over time to maintain exploration of the state space. Unlike the baselines. LEADS does not rely on any reward function to be maximized. Instead, uses the same tool, the SSR, plan exploration and adjust the set of skills ensure effective exploration. Unlike static skills, the skills learned by LEADS are dynamic, continuously adapting throughout the algorithm's 's execution to move away from previously explored areas, as explained in uncertainty measure, which includes elements from the
A LEADS achieves this by maximizing Kullback-Leibler divergence between the current occupancy measure of each skill and the history of occupancy measures from the set of skills.
This mechanism differs from algorithms such DIAYN 13 LSD
3 and METRA 351, which aim to learn static set of skills. These methods evolve the skills until reaching an optimum or sub-optimum. DIAYN maximizes mutual information, represented as the Kullback- Leibler divergence between ((S, Z) and (Z)) while LSD, CSD, and METRA maximize the Wasserstein distance between these distributions using different metrics.
CSD
In contrast, NGU 2 and RND 7 use uncertainty methods that evolve only a single policy over time and do not seek to learn set of diverse skills. The method closest LEADS is SMM [26). which also learns a set of diverse skills that explore using specific uncertainty measure. SMM uses the loss from VAE 17 the uncertainty measure, framing the optimization two-player adversarial game.
LEADS uses novelty- based uncertainty measure, which appears to lead to more stable optimization process leading to a more continous and efficient exploration.
C Hyperparameters
The following table (Table summarizes the hyperparameters used in experimental setup.
Hyperparameter
Value
Tikill
6
Zdim
20
A
0.05
7
0.95
Àc learning
0.5
Ogg
5 x 10
Ore-learning
5 x 10-4
Tlepisode
16
nsGD, c-learning
256
NsGD, actor
16
Ranchive
batch siZCe-leamning
1024
batch size
1024
Table 3: Hyperparameters used for LEADS
Layer
Type
Input Dimensions
Output Dimensions
Activation
Classifier Network
Dense
(State Dim)x2 + Action dim + +Z
256
ReLU
2
Dense
256
128
ReLU
3
Dense
128
Sigmoid
Actor Network
1
Dense
State Dim Action Dim
256
ReLU
2
Dense
256
256
ReLU
3
Dense
256
Action Dim
Linear
4
Dense
256
Action Dim
Tanh
Table 4: Structure of the Classifier and Actor Networks
17

---

C.1 The number of skills
As with other skill-based algorithms, LEADS's coverage changes with the number of skills. However, the skills learned by LEADS evolve throughout the entire training procedure to visit unexplored areas. Hence, one can expect LEADS require fewer skills to achieve the same final coverage performance than algorithms that learn static skills, like DIAYN 133. Choosing the number of skills involves a tradeoff between the acceptable computational runtime of the algorithm (which increases with the number of skills) and the desired efficiency in state space coverage. This choice is also highly problem-dependent Dynamic adaptation of the number of skills is challenging and open topic, covered by works such as the one proposed by Kamienny et al. 124). Although this remains open question, for LEADS we worked with a hand-chosen, predetermined number of skills and did not focus on this aspect.
D Failure cases of the successor state representation estimation
LEADS relies intrinsically on the quality of the SSR estimator. When this estimator is poor, LEADS might not yield diverse and exploratory skills. For instance, in a number of standard MuJoCo environments from the Gymnasium suite (43. LEADS performs poorly as a consequence of C- Learning's 's inability obtain reliable estimation of the successor state representation. Figures and 9 iillustrate this phenomenon on the HalfCheetah benchmark after 25 epochs of LEADS. We evaluate the discrepancy between the estimated state occupancy measure and the actual distribution of states for set of skills, To enable visualization of these quantities, we project these densities on the (z, y) plane where and y are the coordinates of the center of mass of the cheetah's torso. Figures reports the estimated state occupancy measure from the initial state m (Su, [sss), 2). where II(s) (I,, y). using the SSR. In contrast, Figure | displays the scatter plot of visited states for each skill in the (z. y) plane. Despite experimenting with an extensive range of hyperparameters for C-Learning, satisfactory estimation of the measure could not obtained and the probability density Figure could not match the (x, y) coordinates of the visited states of Figure 11
As comparison, Figures 10 and IT iilustrate the identical experiment conducted in the Fetch-Reach environment reported in the main body of the paper. The state occupancy estimated with the SSR of Figure I0 matches closely the states encountered by each skill in Figure TT This corroborates the quantitative results reported in Figure when reliable estimation of the SSR is available, LEADS performs effectively.
Investigating why C-Learning performs poorly in such MuJoCo environments is beyond the scope of this paper. One could conjecture, following the conclusions of Blier al. [4] that learning successor state representation that generalizes efficiently through high-dimensional state spaces is a challenging task. Although we could not obtain conclusive results with C-Learning in these specific cases, recent methodologics [15] 45 demonstrate improved generalization capabilities. These new methods appear be more successful in MuJoCo environments. We reserve the extension of LEADS to such SSR estimators for future work.
18

---

4.0
130
101)
Figure 9: (x.y) coordinates of the states sampled by each skill for 16 rollouts each in HalfCheetah environment
Figure 8: The SSR m(s. (s), 2) at epoch 25 for each skill in HalfCheetah environment
2.50
18114
149
169
800
220
1.6
1a
60
200
100
134
03
19
50 60 3K 00 en um 10 150
Figure 11: (x.y) coordinates of the states sampled by each skill for 16 rollouts each in FetchReach environment
Figure 10: The SSR m(s0, II(s), 2) at epoch 25 for each skill in FetchReach environment
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our study illustrates the limitations of using mutual information (MI) as an objective for exploration. propose a refined objective, offer a theoretical perspective on it, and demonstrate experimentally that it surpasses state-of-the-art approaches.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
19

---

The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss in a section in the appendix. See section D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: See A
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
20

---

Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes].
Justification: The code was provided, and the description of the method, along with the details provided in in[C ensures full reproducibility of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- ppen-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification: Everything is provided in this repository
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new spun-s benchmark).
21

---

reproducible, they The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
sesbbbbmmssiiooo wwwiissiiiiiiiiiiic for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: The paper is self-contained. The method section 3 provide the pseudo-code of the algorithm with all the hyper-panee parameters used and C provide the values used in this study.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: For every quantitative experiment, the specific standard deviation is made explicit. Additionally. providde the p-value when the final distributions of coverage are closely matched between methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
22

---

8. Experiments Compute Resourees
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [NA].
Justification: All experiments conducted in this study can be performed on any minimal system architecture, without the need for GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes].
Justification:
We preserved anonimity and conformed with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification:
There is no societal impact of work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
23

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- usal-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA].
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "3LF2HG5NeC", "reviewer": "Reviewer_QVxs", "summary": "Having an agent learn a set of diverse skills potentially affords the agent better environment exploration. A lack of diversity among the learned skills may reduce the agent's ability to discover newer informative states in the environment. This paper formalizes a method for an agent to learn a set skills by enhancing previous defined mutual information between the states and skills so that it explicitly encourages diversity in exploration. They then evaluate their approach (LEADS) against other algorithms in a number of domains to demonstrate its ability to more effectively explore the state space.", "questions": "* Notation is imprecise. E.g. what is the domain of policy? $\\pi_\\theta: \\mathcal{S} \\times \\mathbb{R}^d \\rightarrow \\mathcal{A}$ or  $\\pi_\\theta: \\mathcal{S}  \\rightarrow \\mathcal{A}$​​?\n\n* The problem setting of Fig 1 is confusing. Is each state reachable? If so the mutual information calculation is wrong. If we assume only the states denoted by the skills are reachable then the calculation is correct but this assumption should be explicitly stated. \nTreatment of how hyperparameters were handled was not specified. Were baselines tuned? How was tuning done? (E.g. grid search? random search? etc.)\n\n* The first bound was introduced to remove the need to sample a minibatch $s_1$ from $z$, but this is still the case after the introduction of the bound. What was gained out of the replacement?", "limitations": "N/A.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**Originality**\nWhile the paper does not introduce any new concept or problem, I consider that the paper does propose an interesting variation to the use of mutual information between states and skills to learn exploratory options that maximize coverage of the state space at the same time that the options are diverse. Both successor features and using mutual information for exploration existed in the literature before this work. The combination done in this way seems interesting.\n\n**Significance**\nThe algorithm represents progress towards the solution of a very relevant problem in deep reinforcement learning: continual exploration based on online observations of the state space.\n\n**Clarity**\nThe explanation of the problem and the general idea of the solution and the paper are easy to follow. However there are some issues explained later.\n\n**Quality**\nThe intuitive concept behind combining successor state measures and mutual information is sound. While the resultant algorithm makes sense, I do not think it works necessarily how the authors explain. The experiments performed are reasonable for empirically backing their claims but the results were not analyzed properly (more later)", "weaknesses": "* While the explanation of the problem and the general idea of the solution are easy to follow. There is some mathematical imprecision in Section 3:\n   1. A measure is a function from a $\\sigma$-algebra to non-negative reals. Here the measure is defined for states rather than sets of states. This is more accurately the successor representation.\n   2. The letter $p$ is used to denote multiple different probability density functions. This is confusing since, $s_1$ and $s_2$ correspond to different random variables: $p(s_1|z)$, the effective state distribution observed in the replay buffer is different from $p(s_2|z)$, which is the state distribution resulting from the discounted visitation.\n\n* Besides the lack of mathematical precision, there is a statement that is not backed up by any argument: line 163 states that the lower bound results in eliminating the natural interplay between the skills that is embedded in the mutual information. There is no explanation for this assertion.\n\n* The related works does not reference work on options or Eigenoptions which are very closely related do successor representations.\n\n* The t-test seems an inappropriate statistical test --- Paired t-tests are better than t-test for comparing algorithms. (See https://arxiv.org/pdf/2304.01315#page=29)\n\n* Sample standard deviation (as used in Fig. 5) is a measure of the variability of a distribution NOT a confidence interval over a mean. Statements such as \"One can note that LEADS outperforms other methods across almost all tasks\"  are unsubstantiated by the provided evidence. One should instead use standard error (as is commonly done in the literature) to obtain a confidence interval. However, a more appropriate choice would either by student-t confidence interval or a bootstrap confidence interval. Please see: https://arxiv.org/pdf/2304.01315#page=13", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Answer: Reviewer QVxs\n\nWe thank Reviewer QVxs for their comments. In our general response, we address several of the concerns raised by the reviewer. Below, we specifically address their additional questions and concerns about our study.\n\n### Statement in line 164:\n\nThe sentence \"natural interplays between skills\" in the text refers to the interaction or coordination required among the different skills during the maximization of Mutual Information. Specifically, the reversed form of Mutual Information (MI) is:\n\n$$\n\\mathcal{I}(S,Z) = \\mathbb{E}_{\\substack{z \\sim p(z) \\\\ s \\sim p(s|z)}} \\left[\\log \\left(\\frac{p(z|s)}{p(z)}\\right)\\right].\n$$\n\nTo maximize this quantity, the distribution $p(z|s)$ in the log term must be as concentrated as possible on the skill $z$ that was used to sample this state. Therefore, the probability of any other skill on that state must be minimal. The term \"natural interplays\" thus informally describes the necessary coordination between skills to ensure that each skill is uniquely associated with specific parts of the environment, leading to maximal MI.\n\nIn contrast, maximizing the lower bound given in Equation 4:\n\n$$\n\\mathbb{E}_{\\substack{z \\sim p(z) \\\\ s_2 \\sim p(s|z) \\\\ s_1 \\sim p(s|z)}} \\left[\\log(m(s_1, s_2, z))\\right]\n$$\n\ncorresponds to increasing the probability mass on state $s_2$ (sampled in $p(s|z)$) starting from $s_1$ (also sampled in $p(s|z)$) for the skill $z$, independently of the probability mass of other skills on these two states. The maximization of this quantity does not account for the interactions between skills.\n\nIn Equation 5, we derive a new lower bound that reintroduces these interplays:\n\n$$\n\\underset{\\substack{z \\sim p(z) \\\\ s_2 \\sim p(s|z) \\\\ s_1 \\sim p(s|z)}}{\\mathbb{E}} \\left[\\log \\left(\\frac{m(s_1, s_2, z)}{1 + \\sum_{z' \\in \\mathcal{Z}} m(s_1, s_2, z')}\\right)\\right]\n$$\n\nIn this formulation, the interplays between skills are captured in the denominator of the log term. To maximize this log expression, the denominator must be minimal. This occurs when the probability of transitioning to state $s_2$ from state $s_1$ (which were sampled using skill $z$) is minimal for all other skills $z'$.\n\nTo conclude, the term \"natural interplays\" refers to whether the relationships and interactions between skills are preserved in the maximization process of these expectations. We propose to include the above reasoning as an explanation of the phrase in a new short appendix section.\n\n### Eigenoptions in related work:\n\nIn the study of Eigenoptions, the authors derive a way to decompose a given task into different options for which they propose automatic learning. We agree that the study could be relevant in the related work subsection \"Successor features for MI maximization\". We will include the following reference [1] in the final version of the paper.\n\n### Statistical Analysis of the Results:\n\nWe thank the reviewer for their suggestions and will modify the results in the final section. Specifically, we will use Standard Error and a Paired t-test in our final results. Below, we include p values using the paired t-test, which we will add to Appendix B. We note that the significance indication in Table 1 does not change when using the paired t-test; LEADS is significantly superior to all other methods on the Umaze, Hard, and Fetch Slide environments, and no other method significantly outperforms all other methods.\n\n| Method                     |   pvalue       | \n|----------------------------|----------------|\n| Umaze :  LEADS/CSD         |     0.01522    |\n| Umaze :  LEADS/LSD         |     0.01152    |\n| Hard :  LEADS/CSD          |     0.00246    |\n| Hard :  LEADS/LSD          |     0.00977    |\n| Fetch Reach :  LEADS/CSD   |     0.06354    |\n| Fetch Reach :  LEADS/DIAYN |     0.08307    |\n| Finger :  CSD/LEADS        |     0.07072    |\n| Finger :  CSD/LSD          |     0.00753    |\n| Fetch Slide :  LEADS/CSD   |     0.00070    |\n| Fetch Slide :  LEADS/LSD   |     0.00055    |\n\n### Hyperparameter tuning:\n\nWe propose the inclusion of the following statement in the article: \"Hyperparameters were determined through manual testing on the Hard-Maze environment. Resulting hyperparameters are displayed in Table 3. Default values from the literature were used as hyperparameters for other methods.\" We include hyperparameters of all methods in the included code (https://anonymous.4open.science/r/LDS-BE6B).\n\n### Figure 1 clarification:\n\nIn the Figure 1, only the states denoted by the skills are reachable.\nWe propose making the unreachable states gray and indicating this in the figure explanation.\n\n### Clarification on the Introduction of the First Bound and its Benefits:\n\nWe believe the reviewer refers to the use of Jensen's inequality to derive the first lower bound in Equation 4. The mutual information can be defined as: $\\mathbb{E}_{(s_2, z) \\sim p(s, z)} \\left[ \\log \\left( \\mathbb{E}s_1 \\left[ m(s_1, s_2, z) \\right] \\right) \\right]$, with $s_1 \\sim p(s_1|z)$.\n\nSuppose we wish to maximize this quantity using SGD. For each sample $(s_2,z)$, we need to compute the expectation $\\mathbb{E}_{s_1 \\sim p(s|z)} \\left( m(s_1,s_2,z) \\right)$ (because the log is not linear). In practice, this is computationally expensive because it requires sampling a mini-batch of $s_1$ states for every single $s_2$ state. Instead, we use Jensen's inequality, leveraging the concavity of the log function to derive the lower bound:\n\n$$\n\\mathbb{E}_{\\substack{z \\sim p(z) \\\\ s_2 \\sim p(s|z) \\\\ s_1 \\sim p(s|z)}}\\left[\\log(m(s_1,s_2,z))\\right]\n$$\n\nMaximizing this new quantity can be done with SGD by sampling three independent mini-batches at each iteration. We propose including the above explanation in a new appendix section that expands on section 3.1\n\n[1] Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G., & Campbell, M. (2018, February). Eigenoption Discovery through the Deep Successor Representation. In International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2e5VoUSte3", "reviewer": "Reviewer_98FT", "summary": "The authors aim to solve the problem of exploring to learn a diverse set of skills. To do this, they modify a commonly used mutual information objective in two ways: first, they apply it to the successor measure, and second, they change the sampling distribution to focus on states with high uncertainty. They test this method on a number of challenging goal-conditioned environments, and show that it performs better than existing methonds on several environments, as well as being the only method which achieves good state coverage on all environments.", "questions": "What hand environments are being used? I assume this is HandReach?", "limitations": "The authors adequately address the limitations of their work", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: \n\t\tThere are two main original contributions to this method: the application of mutual information to state successor measures, and the substitution of the uncertain measure as the sampling distribution. In my opinion, this is sufficient originality. \n\n\tQuality: \n\t\tThe derivation of the objective is correct. The authors test on a varied set of environments in challenging high dimensional spaces. \n\n\tClarity: \n\t\tThe work is presented well and is easy to follow. \n\n\tSignificance:\n\t\tThe work is a decent improvement over SOTA the authors compare to.", "weaknesses": "Using the uncertainty measure instead of the actual distribution breaks most of the theoretical properties this algorithm may have had, but I can see why it was done. It would be nice if there were a more solid mathematical interpretation of this objective, such as a tradeoff between an explicit exploration objective and the mutual information objective, but I understand this is not always possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Answer: Reviewer 98FT\nWe thank Reviewer 98FT for their comments. In the following, we address their specific questions and concerns about our study.\n\n### Hand environment:\nIndeed, the environment used in our tests is HandReach. We will make this clear in the experimental section.\n\n### Theoretical study of LEADS\n\nWe agree with the reviewer's comment about the benefits of a deeper theoretical understanding of the LEADS algorithm and specifically the interplay between the mutual information objective and an explicit exploration objective. Through this study, we aim to demonstrate that mutual information maximization can lead to exploration when coupled with an appropriate objective, such as the maximization of the uncertainty measure we propose, along with a sufficient successor state measure. A theoretical understanding of this interplay motivated the writing of Appendix A, which we hope serves as a starting point for this discussion. More specifically, the analysis of the exploration term in Equation 7, detailed in Appendix A.1, draws a link between the uncertainty measure we use and a specific Kullback-Leibler divergence, providing a practical understanding of its maximization. \n\nIn future work, we hope to build upon this theoretical insight to derive more formally motivated objectives based on this study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i7lTcrr08E", "reviewer": "Reviewer_axfE", "summary": "The paper introduces LEADS, an algorithm to learn diverse skills that additionally encourages exploration. LEADS is motivated by the observation that common mutual information-based diversity-seeking algorithms cannot effectively encourage exploration. LEADS instead proposes a new objective based on successor state measures that explicitly encourage the coverage of state spaces. Experiments conducted on state-based control problems validate the effectiveness of LEADS in learning diverse skills that better cover the whole state space than baselines.", "questions": "See the weaknesses part.", "limitations": "LEADS relies on the state occupancy measure estimator to learn diverse skills and encourage exploration. For behaviors that cannot be easily distinguished from the state occupancy measure, the algorithm might not be able to handle them properly.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The work is well motivated with both illustrative examples and theoretical analysis.\n2. This work provides extensive comparisons with baselines. The qualitative visualization is particularly helpful for readers to understand the learned exploratory behavior of the proposed algorithm.", "weaknesses": "1. In the experiment section, it would be better to briefly introduce the compared baseline methods and point out the main difference between LEADS and these methods before delving into the detailed discussion of results.\n2. Besides better coverage of states, are there other advantages of learning more exploratory behaviors? For example, can LEADS achieve better success rates on hard-exploration problems?\n3. How does LEADS perform with a different number of skills? Is there a best choice of the number of skills or is LEADS insensitive to this hyper-parameter?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Answer: Reviewer axfE\n\nWe thank Reviewer axfE for their comments. In our general response, we address their specific concern regarding a more comprehensive comparison of LEADS with other baselines. Below, we will address the remaining questions and concerns about our study.\n\n### Beyond the maximum coverage objective:\nWe appreciate the reviewer raising this insightful question, which opens up important avenues for discussion.\nAs supported by multiple studies [1][2][3][4], exploration represents a critical step in deriving effective policies, especially in hard exploration problems. We conjecture that initializing, for example, a goal-reaching algorithm using the replay buffer obtained via LEADS would yield better success rates, particularly in hard exploration problems. For instance, in Figure 2, in the Hard-maze environment, the transitions sampled by skill 4 are essential for training any goal-reaching policy over goals in that part of the maze. We also conjecture that another expected advantage could be the design of reusable/composable skills in a hierarchical setting, but we consider it out of the scope of this study and reserve it for future work. We propose mentioning this direction in the conclusion section.\n\n### Different number of skills:\nAs with other skill-based algorithms, LEADS's coverage changes with the number of skills. However, the skills learned by LEADS evolve throughout the entire training procedure to visit unexplored areas. Hence, one can expect LEADS to require fewer skills to achieve the same final coverage performance than algorithms that learn static skills, like DIAYN. Choosing the number of skills involves a tradeoff between the acceptable computational runtime of the algorithm (which increases with the number of skills) and the desired efficiency in state space coverage. This choice is also highly problem-dependent. Dynamic adaptation of the number of skills is a challenging and open topic, covered by works such as [5]. Although this remains an open question, for LEADS we worked with a hand-chosen, predetermined number of skills and did not focus on this aspect, as is common in most works in the literature. We propose including this discussion in Appendix C on Hyperparameters.\n\n\nWe hope these answers address the concerns of the reviewer and lead to further exchanges.\n\n[1] Deepak, Pulkit et al. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778–2787. PMLR, 2017.\n\n[2] Yuri, Harrison et al. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018.\n\n[3] Adrià, Pablo et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038, 2020.\n\n[4] Zhaohan, Shantanu, Miruna et al. BYOL-explore: Exploration by bootstrapped prediction. Advances in neural information processing systems, 35:31855–31870, 2022.\n\n[5] Kamienny, Tarbouriech et al.(2022, April). Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching. In ICLR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hC7rL3VPsp", "reviewer": "Reviewer_8kZQ", "summary": "This paper proposes a new skill-based exploration method that leverages successor states. The authors start by arguing the inadequacy of maximizing mutual information as objective for learning diverse skills that also encourage exploration. Then the authors define the mutual information in terms of successor state measures to derive a novel uncertainty measure that ranks states highly that encourage visitation within the current skill but that other skills haven’t visited.  The authors provide comprehensive experiments showing that they outperform common intrinsic motivation methods and other well known skill diversity exploration methods.", "questions": "- have you tested your methods on any rgb observations? It would be interesting to see if C-Learning is still robust in this case. \n- Could you think of any way to perform C-Learning in high dimensional environments? Is the bottleneck related more to the task complexity or observational complexity?", "limitations": "The authors have addressed the limiting components of their method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The presentation of the paper is excellent. It is easy to read and follow. \n- The authors motivate the paper very well and start from existing work to derive a novel method. \n- The experiments seem fitting although somewhat simple. \n- The authors perform thorough ablations and also analyse failure cases.", "weaknesses": "- The first 2-D environment seems a bit simple. Although I think it is adequate to demonstrate exploration behaviour, it would be interesting to see how LEADS performs with more complex observations or tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Answer: Reviewer 8kZQ\nWe thank the reviewer 8kZQ for their very positive comments on the paper presentation and for their insightful comments on C-learning. We hope to address their questions in our response.  \n\nWe did find that the application of C-learning is limited based on the dimensionality of the problem. The application of C-learning to the high-dimensional state spaces of the MuJoCo environments was a challenging aspect of this work, as detailed in Appendix D. We therefore did not try RGB environments, as this would require further fine-tuning of C-learning on a convolutional architecture. However, recent contributions [1][2] building upon C-Learning illustrate that learning this measure in such environments is feasible, making this application an interesting direction for future work. We will include these references in the discussion on the limitations of C-learning in Appendix D.\n\n[1] Eysenbach, Benjamin, et al. \"Contrastive learning as goal-conditioned reinforcement learning.\" Advances in Neural Information Processing Systems 35 (2022): 35603-35620.\n[2] Zheng, Chongyi, Ruslan Salakhutdinov, and Benjamin Eysenbach. \"Contrastive Difference Predictive Coding.\" The Twelfth International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful feedback, which has been crucial in refining our paper and pinpointing areas that require further clarification. In this general response, we address the concerns raised by reviewers regarding the paper's clarity. We believe these concerns will lead to the main changes in the text, and we include them in the general response to encourage further discussion if needed. Additionally, we address all remaining concerns raised by each reviewer in the respective rebuttal sections.


* **The use of the word measure in "Successor State Measure":**


(reviewer QVxs)
We agree that the term "Successor State Measure" can be misleading and that the Successor State (Representation in the more generic case) of a policy does not adhere to the strict definition of a measure. Considering a measurable space $(\mathcal{S}, \Sigma)$, where $\mathcal{S}$ is a set and $\Sigma$ is a $\sigma$-algebra on $\mathcal{S}$, our definition of Successor State Measure indeed refers to the density of a measure defined on that measurable space. Depending on the reviewer’s viewpoint, we propose to either clarify this in the definition of the Successor State Measure or to replace "Successor State Measure" with "Successor Representation" throughout the text.

* **The use of $p$ to denote distributions:**


(reviewer QVxs)
The reviewer’s remark is correct. In Equation 3, we use the law of total probability to obtain: $p(s_2|z) = \mathbb{E}_{s_1 \sim p(s|z)}\left[ p(s_2|s_1,z) \right]$. We then use our estimation of the Successor State as an approximation of $p(s_2|s_1,z)$. More specifically, just like the density of the random variable $s_1$ (the replay buffer) is an empirical approximation of the density $p(s|z)$ obtained via a Monte Carlo process, the density of the random variable $s_2$ is an approximation of $p(s|z)$ obtained by approximating our Successor State as $p(s_2|s_1, z)$ in the expectation. We could indeed propose new notations in the text, but we believe it would be more appropriate to explicitly clarify these approximations in the text to avoid overly complex notations.

* **Policy mapping:**


(reviewer QVxs)
The policy's domain is $\mathcal{S} \times \mathbb{R}^d$ and its image by the policy $\pi$ is the set of distributions over the action space $\Delta(\mathcal{A})$. We will make this more explicit in the text.

* **Providing a better distinction with other baselines:**


(reviewer axfE) 
Due to lack of space, including such details and comparison was not possible in the main text of the paper. But we agree it would make the paper easier to read. We have added an Appendix to better cover the baseline methods and their differences with LEADS, and referred to it at the beginning of the experimental section in the main text.

Meta-review: This paper received four positive reviews with minor points that were all clarified in the author's rebuttal.

All reviewers agreed the idea proposed in this work (combining successor representations in the mutual information objective typically used in skill learning) is original and significant. There is also consensus that the proposed algorithm is correct. The experimental results are convincing and compare with seven related methods in many scenarios. The paper is also clearly presented.

While intuitively well motivated, the algorithm lacks a detailed theoretical analysis, and it is only motivated formally in Appendix A. However, I believe that the empirical part is strong enough to recommend acceptance.

The authors are encouraged to incorporate the following suggestions in the revised version.

- Incorporate all the points addressed in the rebuttal, including Figure 1 clarification, and the sentences mentioned in the response to reviewer QVxs.

- The authors should fix the mathematical and notational imprecisions mentioned in the rebuttal phase.

- It is recommended to replace the term "measure" with "representation".

- Incorporate relevant missing related work, such as [1].

- Include the discussion regarding the choice of the number of skills and the existing trade-offs.


[1] Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G., & Campbell, M. Eigenoption Discovery through the Deep Successor Representation. ICLR 2018
Decision: Accept (poster)