Title: Exploration by Learning Diverse Skills through Successor State Representations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning

Exploration

Deep Learning
Abstract: The ability to perform different skills can encourage agents to explore. In this work, we aim to construct a set of diverse skills that uniformly cover the state space. We propose a formalization of this search for diverse skills, building on a previous definition based on the mutual information between states and skills. We consider the distribution of states reached by a policy conditioned on each skill and leverage the successor state representation to maximize the difference between these skill distributions. We call this approach LEADS: Learning Diverse Skills through Successor State Representations. We demonstrate our approach on a set of maze navigation and robotic control tasks which show that our method is capable of constructing a diverse set of skills which exhaustively cover the state space without relying on reward or exploration bonuses. Our findings demonstrate that this new formalization promotes more robust and efficient exploration by combining mutual information maximization and exploration bonuses.

Exploration by Learning Diverse Skills through Successor State Representations

Abstract

The ability to perform different skills can encourage agents to explore. In this work, we aim to construct a set of diverse skills that uniformly cover the state space. We propose a formalization of this search for diverse skills, building on a previous definition based on the mutual information between states and skills. We consider the distribution of states reached by a policy conditioned on each skill and leverage the successor state representation to maximize the difference between these skill distributions. We call this approach LEADS: Learning Diverse Skills through Successor State Representations. We demonstrate our approach on a set of maze navigation and robotic control tasks which show that our method is capable of constructing a diverse set of skills which exhaustively cover the state space without relying on reward or exploration bonuses. Our findings demonstrate that this new formalization promotes more robust and efficient exploration by combining mutual information maximization and exploration bonuses.

1 Introduction

Humans demonstrate an outstanding ability to elaborate a repertoire of varied skills and behaviors, without extrinsic motivation and supervision. This ability is currently captured through the problem of unsupervised skill discovery in reinforcement learning [41], where one seeks to learn a finite set of policies in a given environment whose behaviors are notably different from each other. Implicitly, seeking behavioral diversity is also related to the question of efficient exploration, as a set of skills that better covers the state space is often preferable. Seeking diversity has recently been successfully studied through the prism of mutual information maximization [17, 13, 8]. In this article, we argue maximizing mutual information might be ambiguous when seeking exploratory behaviors and propose an alternative, motivated variant. The key intuition underpinning this formulation is that a good set of exploratory skills should maximize state coverage, while preserving the ability to distinguish a skill from another. Then, we propose a new algorithm which implements this generalized objective, leveraging neural networks as estimators of the state occupancy measure. This algorithm demonstrates better exploration properties than state of the art methods, both those designed for reward-free exploration and those seeking skill diversity.

We motivate our developments with a first illustrative toy example. Let us consider a reward-free Markov decision process [38, MDP] ( S , A , P, γ, δ 0 ) where S is the state space, A the action space, P the transition function, γ the discount factor and δ 0 the initial state distribution. A behavior policy, parameterized by θ , maps state to distribution over actions. We define a skill encoding z ∈ R d as an abstract set of skill descriptors that enhances the policy description and conditions the action taken a ∼ π θ ( s, z ) , making the policy a function π : Θ ×S × R d ↦→ ∆( A ) . Unsupervised skill discovery generally considers a finite collection Z = { z i } i ∈ [1 ,n ] of n skills, and a distribution p ( z ) on skills within Z (generally uniform). For a given θ , each skill z induces a state distribution p ( s | z ) . The state distribution under the full set of skills is hence p ( s ) = ∑ Z p ( s | z ) p ( z ) . When maximizing diversity, we aim to find θ ∗ such that, for any pair of skills z 1 , z 2 ∈ Z , the states visited by each skill are as separable as possible. Additionally, we would like the state coverage of the full set of skills to cover as much of the state space as possible.

Maximizing diversity has been formalized in the literature as the maximization of the mutual information I ( S, Z ) between the state random variable S and the skill descriptor Z . This mutual information [39, MI] quantifies how predictable S is, when Z is known (and conversely):

where D KL is the Kullback-Leibler divergence and H is the (conditional) entropy.

Figure 1 illustrates why I ( S, Z ) is an imperfect measure of diversity. In this figure, two sets Z 1 and Z 2 of four skills each are represented by their respective state coverage in a grid maze

Z

1

Z

2

MDP (one symbol per skill). In each set of skills, each skill is picked with probability p ( z ) = 1 / 4 . On the leftmost image ( Z 1 ), each skill covers a single state, hence p ( s | z ) = 1 in this state and 0 otherwise. On the rightmost image ( Z 2 ), p ( s | z ) = 1 / 2 on each covered state and 0 otherwise. Consequently, I 1 ( S, Z ) = H 1 ( S ) -H 1 ( S | Z ) = log 4 and I 2 ( S, Z ) = H 2 ( S ) -H 2 ( S | Z ) = log 4 . The mutual information is hence ambiguous as it ranks the two sets of skills equally, while Z 2 enables better exploration.

Our contribution is structured around maximizing a variant of I ( S, Z ) , using the successor state representation [4] to enable its estimation. Section 2 puts this idea in perspective of the relevant literature, highlighting similarities and contrasts with previous works. Then Section 3 introduces and discusses our LEADS algorithm. Section 4 reports empirical evidence of the good exploratory properties of LEADS, demonstrating its ability to outperform state-of-the-art methods on a span of benchmarks.

2 Related work

This section endeavors to provide the reader with a comprehensive overview of approaches which aim at exploration in RL and skill diversity.

Exploration bonuses. Driving exploration by defining reward bonuses has been a major field of research for over two decades [25, 6, 1, 3, 36, 7, 2, 19]. These methods generally involve defining reward bonuses across the state space, which decrease over time in well-explored states. In large state spaces where count-based methods are challenging, these approaches require function approximation to generalize across the state space. However, as explained by Jarrett et al. [23], identifying these exploration bonus approximators is an adversarial optimization problem making these approaches unstable in practice [12].

Quality Diversity. Exploration can also be achieved as a by-product of behavior diversity search. Evolutionary algorithms have been used to search for new behaviors, such as in Novelty Search [27], and for a diversity of behaviors which encourage high performance through the growing domain

of Quality Diversity algorithms, exemplified by MAP-Elites [11]. In these algorithms, skills are usually characterized through hand-crafted behavior descriptors , although there are methods which learn such descriptors through variational inference [10, 9]. Diversity is expressed as the coverage over such descriptors, and policies are modified through gradient-free [11] or gradient-based [32] updates in order to incrementally populate an archive of diverse skills. While these algorithms have demonstrated competitive results with exploration algorithms in RL [9], in general, they require expertise to define a behavior descriptor and are less sample-efficient than RL methods [28].

Information theory approaches to diversity. The field of information theory offers valuable insights into designing algorithms that generically promote diversity. Gregor et al. [17], Eysenbach et al. [13] and Sharma et al. [40] were pioneers in proposing methods to maximize either the reverse or forward form of I ( S, Z ) for a fixed set of skills Z . To address the issue of unbalanced state distributions across skills throughout the entire environment, Campos et al. [8] suggest sequencing the problem into three stages: pure exploration in S to promote sampled states diversity, state clustering via variational inference, and skill learning by maximisation of the forward form as proposed by Sharma et al. [40]. In the exploration phase, Campos et al. [8] employs the State Marginal Matching algorithm proposed by Lee et al. [26], which resembles MI-based algorithms as it combines mutual information maximization with an exploration bonus to efficiently explore the state space.

Focusing even more on exploration, Liu and Abbeel [29] automate learning Novelty Search behavior descriptors by training an encoder with contrastive learning in order to promote exploration. Focusing also on exploration, Kamienny et al. [24] employ an incremental approach combined with the maximization of the reverse form of MI to eliminate skills that are not discernable enough, subsequently generating new skills from the most discernable ones.

Finally, recent studies including those by Park et al. [33], Park et al. [34], and Park et al. [35], integrate mutual information (MI) with trajectory-based metrics between states to enhance exploration. Specifically, these studies aim to maximize the Wasserstein distance, also known as the Earth mover's distance, between P ( S, Z ) and P ( S ) P ( Z ) while the definition of MI uses the KL divergence. This distance requires specifying a metric for the state space. Park et al. [33] and Park et al. [34] uses the Euclidean distance, while Park et al. [35] employs the temporal distance.

It is important to note that, although these methods induce an interesting approach to exploration (as a by-product of diversity), they do not explicitly aim at encouraging coverage of the MDP's state space.

Successor features for MI maximization. Few attempts have been made to exploit universal successor features [5] in MI maximization methods. Machado et al. [30] decompose the learning process into two stages: first, they automatically acquire specific options, which are then used to develop a hierarchical policy that integrates these options. In contrast, Hansen et al. [21] first learn a general task-based policy conditioned on a task z , and subsequently optimize the reverse form of MI by utilizing universal successor features to identify a range of diverse skills efficiently. More recently, Grillotti et al. [18] used successor features to optimize the quality of predefined behaviors based on the Quality Diversity formulation, which requires hand-crafted behavior descriptors.

The method we propose explicitly aims at promoting diversity with the goal of achieving good exploration. For this purpose, we turn the MI objective function into a new objective which promotes exploration, separating this work from previous contributions in MI maximization.

3 Promoting exploration with successor state representations

We now derive an algorithm that explicitly encourages exploration in diversity search. We start by casting mutual information maximization in terms of state occupancy measures and derive a lower bound which can be maximized with stochastic gradient ascent on the policy's parameters θ (Section 3.1). Then, as maximizing MI does not encourage exploration, we transform this lower bound so that its maximization fits this goal, leading to a new target quantity designed to promote both diversity and exploration (Section 3.2). This enables defining and implementing a general algorithm for Learning Diverse Skills through successor state representations (LEADS, Section 3.3).

3.1 MI lower bound with successor state representations

The mutual information I ( S, Z ) is naturally expressed through the (conditional) probability densities of S and Z . Successor state representation estimators [4, SSR] provide a direct way of estimating these densities. More precisely, we identify the Successor State Representation as the density of the measure defined on the set of states. The SSR p ( s | s 1 , θ, z ) of a skill ( θ, z ) is the state occupancy measure of policy π θ ( s, z ) , when starting from s 1 , that is

For the sake of readability, we make θ implicit in our notations going forward. This measure can be understood as the probability density of reaching the state s 2 starting from another state s 1 when the skill z is used [22]. It encodes all the paths of the Markov chain that can be taken from s 1 to s 2 when the skill z is used [4]. Here ( s 1 , s 2 ) ∈ S 2 can be any two states of the Markov chain.

The SSR is the fixed point of a Bellman operator and one can employ temporal difference (TD) learning to guide a function approximator towards this fixed point. There are numerous methods for estimating the SSR density [4, 22, 14, 15]. In this paper, we use C-Learning [14] for estimating the SSR for all experiments. C-Learning proposes to cast the regression problem of learning p as a classification problem. Specifically, for a given pair ( s 1 , a, z ) , a classifier σ ( f ϕ ( s 1 , a, s 2 , z )) , where σ is the sigmoid function, is trained to predict whether s 2 was sampled from an outcome trajectory from s 1 ( s 2 ∼ p ( ·| s 1 , a, z ) ) or from the marginal density of all possible skills ( s 2 ∼ p ( s ) , the distribution over S ). The optimal classifier holds the following relation with the SSR's density: e f ϕ ( s 1 ,a,s 2 ,z ) = p ( s 2 | s 1 , a, z ) /p ( s 2 ) . We denote m ϕ ( s 1 , a, s 2 , z ) = e f ϕ ( s 1 ,a,s 2 ,z ) , as the SSR estimated via C-Learning, taking a ∼ π θ ( s 1 , z ) . For the sake of brevity, we will use the notation m ϕ ( s 1 , s 2 , z ) = E a ∼ π θ ( s 1 ,z ) [ m ϕ ( s 1 , a, s 2 , z )] . We note that any method which reliably estimates m ϕ ( s 1 , s 2 , z ) can be used instead of C-Learning in LEADS.

As described in Section 1, the mutual information between the state random variable S and the skill descriptor Z is I ( S, Z ) = D KL ( P ( S, Z ) || P ( S ) P ( Z )) . This can be expressed as

State s 2 can be any state s within the set S , yet we refer to it as s 2 to ensure that when we incorporate the definition of the SSR, it aligns directly with the notation of Equation 1. By definition of the SSR under a given policy π ( s, z ) , the state distribution p ( s | z ) obeys

By sampling s 1 ∼ p ( s | z ) , we can therefore estimate p ( s 2 | z ) , allowing for the injection of m ( s 1 , s 2 , z ) = p ( s 2 | s 1 , z ) /p ( s 2 ) from Eysenbach et al. [14] into Equation 2:

AMonte Carlo estimator of I ( S, Z ) can be obtained by sampling ( s, z ) from a replay buffer. However, for every state s sampled this way, the estimate of p ( s | z ) provided by Equation 3 requires sampling a batch of s 1 ∼ p ( s | z ) . This is possible through keeping separate replay buffers for each skill, but would be computationally intractable for an accurate estimation. Hence, as in similar methods, we turn rather to a lower bound on I ( S, Z ) , which admits a Monte Carlo estimator, and which will be maximized with respect to θ using stochastic gradient ascent.

Applying Jensen's inequality to the inner expectation in the previous expression yields

Note that θ participates in this lower bound through m ( s 1 , s 2 , z ) = E a ∼ π θ ( ·| s 1 ) m ( s 1 , a, s 2 , z ) . This quantity is essentially what an algorithm like DIAYN [13] maximizes: given separate experience buffers for each skill, one can compute a Monte Carlo estimate of this lower bound on the gradient of the mutual information I ( S, Z ) with respect to θ .

3.2 Promoting exploration with different skills

Looking closely at Equation 4, one can note that the interplay between each skill's state distribution is lost when deriving the lower bound. Hence, this bound can be maximized without actually pushing skills towards distinct states, hence without skill diversity. To regain the incentive to promote skill diversity, we propose to encourage exploration in each skill to put probability mass on states which receive little coverage by the full set of skills. In other words, for a given transition ( s, π ( s )) , we want to augment the probability of occurrence for one skill while decreasing it for all others.

Since m ( s 1 , s, z ) ≥ 0 , we can introduce the following bound:

While Equation 5 is a looser lower bound than Equation 4, it goes towards the goal of diversity search originally captured by MI maximization: to have distinct state distributions for each skill.

However, as illustrated in the introductory example, maximizing MI does not promote large state coverage, nor exploratory behaviors. Equation 5 promotes skill diversity, but it does not explicitly encourage state coverage. We therefore argue that exploratory behaviors can be obtained by focusing the state distribution of each skill towards specific states within the support of p ( s | z ) . Instead of sampling s according to p ( s | z ) , we encourage exploration by attributing more probability mass to states that trigger exploration. We do so by creating a sampling distribution δ ( s | z ) based on an uncertainty measure u t ( s, z ) , as explained next.

Biasing a skill towards promising states for exploration is often achieved through exploratory bonuses based on uncertainty measures [36, 7, 2], or repulsion mechanisms [16]. These heuristic measures encourage exploration by pushing policies towards states of high uncertainty or away from previously covered transitions. The novelty of our approach is the use of an exploration bonus in the framing of mutual information maximization. However, by replacing s 2 ∼ p ( s | z ) by another sampling distribution δ ( s | z ) , we lose the theoretic guarantee of maximizing the lower bound of Equation 5. Rather, we propose a new objective, expressing an incentive to explore within the search for skill diversity, but with a greater focus on state coverage through distinct skills than previous lower bounds on mutual information. The quantity we maximize is then:

The uncertainty measure u t which defines the distribution δ ( s | z ) is designed to explore under-visited areas and to create repulsion between different skills. We define three desired properties for states to prioritize using this measure. To describe these properties, consider a sequence of policies π t , each defining a set of skills { π t ( · , z ) } z ∈Z , and the corresponding sequence of SSRs m t ; the three properties are then as follows. (1) A good target state s z t for skill z at time step t is one that has high probability of being visited by π t ( · , z ) , but was relatively infrequently visited by previous policies' state occupancy measures { m k } k ∈ [1 ,t -1] for any skill. (2) It is also a state which has both high probability of being reached by the current π t ( · , z ) and low probability of being reached by any other current skill, starting from s 0 . (3) Given a previous target state s z t -1 , a good new target state is one that has high (resp. low) probability to be reached by π t ( · , z ) (resp. any other skill), starting from s z t -1 . While the first property explicitly encourages visiting under-visited states, the two others do not encourage exploration per se, and rather strengthen skill diversity by pushing their state distributions apart. This leads to the idea of ranking target states according to:

̸

The δ ( s | z ) distribution of Equation 6 therefore allocates more probability to states with high u t ( s, z ) . Empirically, we found that making δ deterministic as the Dirac distribution on a state that maximizes u t ( s, z ) enabled efficient exploration. Appendix A provides a more formal perspective on the derivation of the u t uncertainty measure.

3.3 The LEADS algorithm

With the objective of maximizing G ( θ ) (Equation 6), we define the LEADS (Learning Diverse Skills through successor state representations) algorithm, presented in Algorithm 1.

In order to maximize G ( θ ) , we must first sample from p ( s | z ) for each skill z , then compute the SSR for the current skills π t ( · , z ) in order to define G ( θ ) , then finally update θ to maximize G ( θ ) . As such, an iteration of LEADS features three phases in order to learn a new set of skills: sampling, learning the SSR, and optimizing the policy parameters θ .

First, the sampling phase populates separate replay buffers D z for each skill z ∼ p ( z ) . This is achieved by rolling out n ep episodes with a given policy π θ ( · , z ) . Then, C-learning is used to compute the parameters ϕ t of a joint SSR m t ( s 1 , a, s, z ) for π t , and store it. We note that the on-policy version of C-learning can be used for this step as the data in the replay buffers has been collected with the current policy π t . This SSR, and all previous ones, are then used to define u t ( s, z ) , which in turn defines the distribution

δ ( s, z ) . As this can be approximated for a static policy, sampling can be performed before updating θ . Sampling s ∼ δ ( s | z ) is performed by running all states in D z through u t ( s, z ) , although, to limit computational cost, we can restrict this evaluation and selection to only a uniformly drawn subset of each D z .

We therefore have the necessary components to optimize θ according to G ( θ ) : a means of sampling states and the SSR. At each gradient ascent step, we sample a mini-batch of states s 1 from D z for a given skill z ∼ p ( z ) to estimate p ( s | z ) . This permits calculating the objective G ( θ ) for the current θ using m ϕ t and performing a gradient step. In the gradient calculation, we also include an action entropy maximization term, as done in other works [31, 20]:

θ is therefore updated to maximize G ( θ ) + λ h H ( θ ) , although λ h is intentionally kept small (0.05) to focus on the principal LEADS objective G ( θ ) .

Finally, after each gradient step, the SSR is updated using the off-policy formulation of C-learning so that m ϕ t remains representative of the state distribution under π θ t . This is done without sampling new transitions and is justified by the fact that the target state s z t and the states along a trajectory to s z t are already within the replay buffer D z . n SGD steps of gradient ascent on θ are performed in this way, before a new iteration of LEADS is started.

4 Experiments & Results

We demonstrate LEADS on a set of maze navigation and robotic control tasks and compare its behavior to state-of-the-art exploration algorithms. We provide all code for LEADS and the baseline algorithms, as well as the scripts to reproduce the experiments (repository). All hyperparameters are summarized in Appendix C.

4.1 Evaluation benchmarks

Mazes. Maze navigation has been frequently used in the exploration literature, as 2D environments allows for a clear visualization of the behaviors induced by an algorithm. The assessment of state coverage is also easier to understand than in environments with high-dimensional states. We design three different maze navigation tasks, named Easy , U , and Hard (depicted in Figure 2), of increasing difficulty in reaching all parts of the state space. In each of the mazes, the state space is defined by the agent's Euclidean coordinates S = [ -1 , 1] 2 and the action space corresponds to the agent's velocity A = [ -1 , 1] 2 . Hitting a wall terminates an episode, making exploration difficult.

Robotic control. We further assess the capabilities of LEADS in complex robotic control tasks. These tasks allow evaluating the ability of LEADS to explore in diverse and high-dimensional state spaces. We evaluate LEADS on a variety of MuJoCo [42] environments from different benchmark suites. Fetch-Reach [37] is a 7-DoF (degrees of freedom) robotic arm equipped with a two-fingered parallel gripper; its observation space is 10-dimensional. Fetch-Slide extends the former with a puck placed on a platform in front of the arm, increasing the observation space dimension to 25. Hand [37] is a 24-DoF anthropomorphic robotic hand, with a 63-dimensional observation space. Finger [44] a 3-DoF, 12-dimensional observation space, manipulation environment where a planar finger is required to rotate an object on an unactuated hinge. Appendix D discusses additional experiments and limitations on other MuJoCo tasks.

LEADS

LSD

DIAYN

Skill 0

Skill 1

Skill 2

Skill 3

Skill 4

Skill 5

4.2 Skill Visualization

The first goal of this work, as exposed in the introductory example, is to obtain a set of skills that are as distinct from each other as possible in their visited states, while simultaneously covering as much of the state space as possible. We first assess this property in LEADS, DIAYN [13] and LSD [33] through a visual analysis of their state space coverage in Figure 2. For all experiments, the number of skills is n skill = 6 . To ensure fairness, for each algorithm, we report the skill visualization from the experiment that achieves maximum coverage (as defined in the following section) out of five runs. For each algorithm, Figure 2 presents four figures: the three mazes and the Fetch-Reach environment. In all the mazes, we visualize the 2D coordinates of the states reached over training. Given that 2D visualization is not suitable for the Fetch-Reach environment due to its higher state space dimension,

we project the state onto the two first eigenvectors of a Principal Component Analysis (PCA) of the states encountered by all skills in this environment.

We note that LEADS clearly defines distinct skills in the state space. Furthermore, it leads to a more extensive exploration of the environment than LSD and DIAYN. The Hard maze (bottom left) is noteworthy, as some parts of the environment are difficult to access due to bottleneck passages in the maze. LEADS is the only algorithm that manages to reach all sections of the Hard maze.

(a) Occupancy measure

(b) Uncertainty

skill 0

skill 1

skill 2

skill 3

skill 4

skill 5

skill 0

[ IIDS

skill 2

skill 3

skill 4

skill 5

Figure 3 further demonstrates the ability of LEADS to create distinct skills in the high-dimensional state space of the Hand environment. It reports the state distribution obtained by n skill = 12 skills found by LEADS, projected onto the two first components of a PCA on all visited states. As in the maze navigation tasks, LEADS explores states in distinct clusters, organized by skill. These skills correspond to a visual variety of finger positions, illustrating the ability to discover varied behaviors.

Section 3.2 claimed that the uncertainty measure u t ( s, z ) of Equation 7, and its associated distribution δ triggered relevant exploratory behaviors. Figures 4(a) and 4(b), display the final SSR and uncertainty measure, respectively, for the Hard maze task. The same visual representation of these densities for the other environments can be found in Appendix B. Figure 4(a) demonstrates that the SSR clearly separates states based on skill. The SSR is zero for all states outside of the distribution of a given skill, and all states in a skill have a non-zero probability of being reached by that skill. This confirms that this SSR can reliably be used within the objective function of Equation 6.

Similarly, Figure 4(b) illustrates how the uncertainty measure concentrates on states that promote both exploration and diversity. High values for this measure are found in very different states for each skill (hence promoting diversity), and within a skill, the highest value is found far from the starting state s 0 (hence promoting exploration). The state maximizing u t is represented by a colored dot for each skill. This confirms that the uncertainty measure achieves the joint goal of exploring under-visited areas and creating repulsion between skills. By defining δ ( s | z ) as the state which maximizes u t ( s, z ) for each skill z , LEADS ensures a continuing exploration of the state space.

4.3 Quantitative Evaluation

We now turn to a quantitative evaluation of the ability to explore using diversified skills. For the sake of completeness, we compare LEADS to seven seminal algorithms. Five of these are skill-based, namely DIAYN [13], SMM [26], LSD [33], CSD [34], and METRA [35]. We use n skill = 6 for each skill-based algorithm and for all environments. The two last algorithms are pure-exploration

LEADS

NGU

RND

DIAYN

LSD

CSD

METRA

SMM

0.0

0.2

0.4

0.6

0.8

1.0

×10 6

0

20

40

60

80

100

Easy

0.0

0.2

0.4

0.6

0.8

1.0

×10 6

0

20

40

60

80

100

Umaze

0.0

0.2

0.4

0.6

0.8

1.0

×10 6

0

20

40

60

80

100

Hard

0.0

0.5

1.0

1.5

2.0

×10 5

0

20

40

60

80

100

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

0

20

40

60

80

100

Finger

0.0

0.2

0.4

0.6

0.8

1.0

×10 5

0

20

40

60

80

100

ones, which do not rely on skill diversity and rather rely on exploration bonuses: Random Network Distillation (RND) [7] and Never Give Up (NGU) [2].

A metric of state space coverage should characterize how widespread the state distribution is, in particular across behavior descriptors that are meaningful for the task at hand. Following the practice of previous work, for high-dimensional environments, we use a featurization of the state, retaining a low-dimensional representation of meaningful variables: the ( x, y, z ) coordinates of the gripper in Fetch-Reach, the ( x, y ) coordinates of the puck in Fetch-Slide, and the angles of the finger's two joints and the hinge's angle in Finger. This projection is then discretized and the coverage of a trajectory is defined as the number of visited cells. Figure 5 depicts the evolution of coverage, normalized by the maximum coverage achieved by any run in the current environment. This figure illustrates how coverage progresses relative to the number of samples taken in the environment since the beginning of the algorithm. Shaded areas represent the standard error across five seeds for each algorithm.

One can note that LEADS outperforms other methods across almost all tasks. This is especially notable in the Fetch-Slide environment, where LEADS exceeds the coverage of all other methods by over 20%. Furthermore, besides LEADS, no algorithm is consistently good across all tasks. In the Fetch-Reach task, DIAYN, METRA, and CSD each achieved a coverage between 80% and 85%. LEADS excelled with 90% coverage, significantly outperforming NGU at 50% and SMM and RND, both at 20%. CSD surpasses LEADS and all other methods on the Finger environment, achieving

a relative mean coverage of 96% where LEADS is still competitive with 87%. On this specific benchmark, NGU, METRA and LSD are competitive with LEADS, whereas DIAYN, SMM and RND exhibit low coverage.

5 Conclusion

In this work, we consider the problem of exploration in reinforcement learning, via the search for behavioral diversity in a finite set of skills. We take inspiration from the classical framework of maximization of the mutual information between visited states and skill descriptors, and illustrate why it might be insufficient to promote exploration. This motivates the introduction of a new objective function for skill diversity and exploration. This objective exploits an uncertainty measure to encourage extensive state space coverage. We leverage off-the-shelf estimators of the successor state representation to estimate this non-stationary uncertainty measure, and introduce the LEADS algorithm.

Specifically, LEADS estimates the successor state representation for each skill, then specializes each skill towards under-visited states while keeping skills in distinct state distributions. This results in an efficient method for state space coverage via skill diversity search. Our experiments highlight the efficacy of this approach, achieving superior coverage in nearly all tested environments compared to state-of-the-art baselines.

LEADS intrinsically relies on good SSR estimators. In this work, we used C-Learning [14] for all experiments, which proved efficient but might reach generalization limits in some environments. As a consequence, advances on the question of reliable SSR estimation will directly benefit LEADS, opening new perspectives for research.

Acknowledgments

This work was performed using HPC resources from CALMIP (Grant 2016-[p21001]).

References

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems , 21, 2008.

Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038 , 2020.

Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems , 29, 2016.

Léonard Blier, Corentin Tallec, and Yann Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. arXiv preprint arXiv:2101.07123 , 2021.

Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado Van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators. arXiv preprint arXiv:1812.07626 , 2018.

Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research , 3(Oct):213-231, 2002.

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 , 2018.

Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning , pages 1317-1327. PMLR, 2020.

Felix Chalumeau, Raphael Boige, Bryan Lim, Valentin Macé, Maxime Allard, Arthur Flajolet, Antoine Cully, and Thomas Pierrot. Neuroevolution is a competitive alternative to reinforcement learning for skill discovery. arXiv preprint arXiv:2210.03516 , 2022.

Antoine Cully. Autonomous skill discovery with quality-diversity and unsupervised descriptors. In Proceedings of the Genetic and Evolutionary Computation Conference , pages 81-89, 2019.

Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature , 521(7553):503-507, 2015.

Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995 , 2019.

Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 , 2018.

Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. arXiv preprint arXiv:2011.08909 , 2020.

Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems , 35:35603-35620, 2022.

Yannis Flet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adversarially guided actor-critic. In ICLR 2021-International Conference on Learning Representations , 2021.

Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07507 , 2016.

Luca Grillotti, Maxence Faldor, Borja G León, and Antoine Cully. Skill-conditioned policy optimization with successor features representations. In Second Agent Learning in OpenEndedness Workshop , 2023.

Zhaohan Guo, Shantanu Thakoor, Miruna Pîslar, Bernardo Avila Pires, Florent Altché, Corentin Tallec, Alaa Saade, Daniele Calandriello, Jean-Bastien Grill, Yunhao Tang, et al. BYOL-explore: Exploration by bootstrapped prediction. Advances in neural information processing systems , 35:31855-31870, 2022.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint arXiv:1906.05030 , 2019.

Michael Janner, Igor Mordatch, and Sergey Levine. gamma-models: Generative temporal difference learning for infinite-horizon prediction. Advances in Neural Information Processing Systems , 33:1724-1735, 2020.

Daniel Jarrett, Corentin Tallec, Florent Altché, Thomas Mesnard, Rémi Munos, and Michal Valko. Curiosity in hindsight. arXiv preprint arXiv:2211.10515 , 2022.

Pierre-Alexandre Kamienny, Jean Tarbouriech, Sylvain Lamprier, Alessandro Lazaric, and Ludovic Denoyer. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. arXiv preprint arXiv:2110.14457 , 2021.

Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning , 49:209-232, 2002.

Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274 , 2019.

Joel Lehman and Kenneth O Stanley. Evolving a diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation , pages 211-218, 2011.

Bryan Lim, Manon Flageat, and Antoine Cully. Understanding the synergies between qualitydiversity and deep reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference , pages 1212-1220, 2023.

Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems , 34:18459-18473, 2021.

Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv:1710.11089 , 2017.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PMLR, 2016.

Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Proceedings of the Genetic and Evolutionary Computation Conference , pages 866-875, 2021.

Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitzconstrained unsupervised skill discovery. In International Conference on Learning Representations , 2022.

Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability-aware unsupervised skill discovery. arXiv preprint arXiv:2302.05103 , 2023.

Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metricaware abstraction. In The Twelfth International Conference on Learning Representations , 2023.

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning , pages 27782787. PMLR, 2017.

Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464 , 2018.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal , 27(3):379-423, 1948.

Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamicsaware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657 , 2019.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo PerezVicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL https://zenodo.org/record/8127025 .

Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts , 6:100022, 2020.

Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding. arXiv preprint arXiv:2310.20141 , 2023.

A A formal motivation for the uncertainty measure of Equation 7

In Section 3.2, we introduced a new optimization target, based on the uncertainty measure u t ( s, z ) , used in LEADS to strategically distribute probability mass across states for each skill to foster effective exploration. u t ( s, z ) is introduced in Equation 7, which is reproduced here for clarity:

̸

This section delves into the theoretical foundations of this target. We begin by discussing the left term of Equation 7, which corresponds to the exploration component of the measure. Subsequently, we detail the right term, which promotes greater repulsion between the skills.

A.1 Exploration term (left side of Equation 7)

The Successor State Representation (SSR) extends the concept of the occupancy measure. Unlike the traditional occupancy measure, which begins with the initial state distribution, the SSR is an estimation of the occupancy measure starting from any state. In practice, we work with the expected sum of discounted presences of a state over time because we know this measure exists regardless of the ergodicity of the Markov process.

LEADS is an epoch-based algorithm with each epoch consisting of four phases, as presented in Section 3.2 and Algorithm 1:

Sample transitions from the environment for each skill,

Learn the SSR of each skill (for the given policy),

Define a state on which to put more probability mass for each skill,

Use the objective of Equation 6 to update the policy so that it implements this probability mass transport plan.

Since the policy changes in each epoch, the discounted occupancy measure of the policy also evolves over time. For a given skill and epoch n , it can be defined as:

This occupancy measure has a direct estimator, using the SSR in the initial state s 0 :

At epoch n , the state occupancy measure of all skills is the mixture of all skills' state occupancy measures:

Then, the left side term in the definition of the uncertainty measure u t ( s, z ) corresponds to the contribution of a state s to the following Kullback-Leibler divergence:

Note in particular that this divergence accounts for the discrepancy between ρ π z n , ie. the states visited by skill z at epoch n , and ∑ n -1 k =1 ρ π k , ie. the states previously explored by all skills in all past epochs. Thus, the KL divergence of Equation 9 serves as a measure of novelty for the current distribution. Intuitively, the newer a state visited by the current distribution ρ π z n ( s ) is, the larger its contribution to this KL divergence will be. Recall that at this stage, we do not modify the policy (it is the role of the objective function of Equation 6), but rather aim at identifying which states within the support of ρ π z n should be those skill z is encouraged to explore next. By defining the state-wise novelty score log ( m n ( s 0 ,s,z ) ∑ n -1 k =1 ∑ z ′ m k ( s 0 ,s,z ′ ) ) , the distribution δ ( s | z ) unfolds as the Dirac distribution which maximizes the expected value of this score. Overall, LEADS aims to identify and prioritize states with the highest novelty score for each skill, thereby enhancing exploration by shifting each skill's induced distribution towards these novel states.

A.2 Repulsion term (right side of Equation 7)

Building on the previous discussion about the formal motivation of the exploration term on the left side in 7, we now explore the right side term, which is intended to enhance repulsion between skills.

The right side term of Equation 7 can be interpreted as the contribution of a state s within a sum of multiple Kullback-Leibler divergences:

̸

In this equation, term (a) represents the KL divergence of the occupancy measure for the current skill z , starting from its last target state, compared to the occupancy measure of each other skill starting from their respective last target states. Maximizing this KL divergence aims to make the distribution of each skill (starting from their last target state) as distinct as possible. Similarly, term (b) is the KL divergence between the occupancy measures of a skill z and that of any other skill z ′ . Maximizing these two KL divergences leads to disjoint distributions for each skill. As in the previous subsection, we do not change the policy at this stage (this would lead to an ill-defined optimization problem as the KL can become unbounded), but rather aim at identifying which are the states that contribute most to these KL divergences, so that we can promote exploration towards them. Overall, LEADS combines all these terms into the uncertainty measure of Equation 7 to determine which state contributes the most to these KL divergences for each skill. In turn, these states define the distribution δ designed to promote exploration, by increasing the weight of the most likely states of δ in Equation 6

A.3 Numerical advantage of Lower Bound 4

This section offers a more detailed explanation of the passage from the definition of Mutual Information 2 to the Lower Bound 4. The mutual information can be defined as: E ( s 2 ,z ) ∼ p ( s,z ) [log ( E s 1 [ m ( s 1 , s 2 , z )])] , with s 1 ∼ p ( s 1 | z ) .

Suppose we wish to maximize this quantity using SGD. For each sample ( s 2 , z ) , we need to compute the expectation E s 1 ∼ p ( s | z ) ( m ( s 1 , s 2 , z )) (because the log is not linear). In practice, this is computationally expensive because it requires sampling a mini-batch of s 1 states for every single s 2 state. Instead, we use Jensen's inequality, leveraging the concavity of the log function to derive the lower bound:

Maximizing this new quantity can be done with SGD by sampling three independent mini-batches at each iteration.

B Supplementary Analyses and Visualizations

LEADS is based on the estimation of the successor state representation for each skill. Figure 6 and Figure 7 show the final SSR and uncertainty measure, respectively, for the maze and Fetch tasks. These correspond to the same runs shown in Figure 2 for LEADS.

skill 0

skill 1

skill 0

skill 1

skill O

skil1

skill 0

skill 1

skill 2

skill 3

ski 2

skil3

skill 2

skil3

skill 2

skill 3

KXXX

skill 4

skill 5

skill 4

S IIPS

skl14

skill5

skill 4

skill 5

skill 0

skill1

skil 0

skill 1

skil

skill 1

skil 0

skil1

skill 2

skill 3

skill 2

skill 3

skill 2

skll3

skil 2

skil13

ski15

skil 4

skill 5

skill 4

skill5

skil 4

skill 5

Figure 5 illustrates the temporal evolution of coverage for all methods across various environments, evaluated over five different seeds. We extend our analysis by presenting the p-values from the t-tests conducted on the distributions of the final (highest) coverage values for these five seeds, comparing the first method to both the second and third methods. We conducted a t-test for each environment in which no method outperformed the second method by a margin greater than 10%. The results are visible in the table 2.

B.1 Key Differences with Baseline Methods

The LEADS algorithm learns diverse skills that evolve over time to maintain exploration of the state space. Unlike the baselines, LEADS does not rely on any reward function to be maximized. Instead, it uses the same tool, the SSR, to plan exploration and adjust the set of skills to ensure effective exploration. Unlike static skills, the skills learned by LEADS are dynamic, continuously adapting throughout the algorithm's execution to move away from previously explored areas, as explained in A. LEADS achieves this by maximizing an uncertainty measure, which includes elements from the Kullback-Leibler divergence between the current occupancy measure of each skill and the history of occupancy measures from the set of skills.

This mechanism differs from algorithms such as DIAYN [13], LSD [33], CSD [34], and METRA [35], which aim to learn a static set of skills. These methods evolve the skills until reaching an optimum or sub-optimum. DIAYN maximizes mutual information, represented as the KullbackLeibler divergence between P ( S, Z ) and P ( S ) P ( Z ) , while LSD, CSD, and METRA maximize the Wasserstein distance between these distributions using different metrics.

In contrast, NGU [2] and RND [7] use uncertainty methods that evolve only a single policy over time and do not seek to learn a set of diverse skills. The method closest to LEADS is SMM [26], which also learns a set of diverse skills that explore using a specific uncertainty measure. SMM uses the loss from a VAE [17] as the uncertainty measure, framing the optimization as a two-player adversarial game.

LEADSuses a novelty-based uncertainty measure, which appears to lead to a more stable optimization process leading to a more continous and efficient exploration.

C Hyperparameters

The following table (Table 3) summarizes the hyperparameters used in our experimental setup.

C.1 The number of skills

As with other skill-based algorithms, LEADS's coverage changes with the number of skills. However, the skills learned by LEADS evolve throughout the entire training procedure to visit unexplored areas. Hence, one can expect LEADS to require fewer skills to achieve the same final coverage performance than algorithms that learn static skills, like DIAYN [13]. Choosing the number of skills involves a tradeoff between the acceptable computational runtime of the algorithm (which increases with the number of skills) and the desired efficiency in state space coverage. This choice is also highly problem-dependent. Dynamic adaptation of the number of skills is a challenging and open topic, covered by works such as the one proposed by Kamienny et al. [24]. Although this remains an open question, for LEADS we worked with a hand-chosen, predetermined number of skills and did not focus on this aspect.

D Failure cases of the successor state representation estimation

LEADS relies intrinsically on the quality of the SSR estimator. When this estimator is poor, LEADS might not yield diverse and exploratory skills. For instance, in a number of standard MuJoCo environments from the Gymnasium suite [43], LEADS performs poorly as a consequence of CLearning's inability to obtain a reliable estimation of the successor state representation. Figures 8 and 9 illustrate this phenomenon on the HalfCheetah benchmark after 25 epochs of LEADS. We evaluate the discrepancy between the estimated state occupancy measure and the actual distribution of states for a set of 6 skills. To enable visualization of these quantities, we project these densities on the ( x, y ) plane where x and y are the coordinates of the center of mass of the cheetah's torso. Figures 8 reports the estimated state occupancy measure from the initial state m ( s 0 , Π( s ) , z ) , where Π( s ) = ( x, y ) , using the SSR. In contrast, Figure 9 displays the scatter plot of visited states for each skill in the ( x, y ) plane. Despite experimenting with an extensive range of hyperparameters for C-Learning, a satisfactory estimation of the measure could not be obtained and the probability density of Figure 8 could not match the ( x, y ) coordinates of the visited states of Figure 11.

As a comparison, Figures 10 and 11 illustrate the identical experiment conducted in the Fetch-Reach environment reported in the main body of the paper. The state occupancy estimated with the SSR of Figure 10 matches closely the states encountered by each skill in Figure 11. This corroborates the quantitative results reported in Figure 5: when a reliable estimation of the SSR is available, LEADS performs effectively.

Investigating why C-Learning performs poorly in such MuJoCo environments is beyond the scope of this paper. One could conjecture, following the conclusions of Blier et al. [4], that learning a successor state representation that generalizes efficiently through high-dimensional state spaces is a challenging task. Although we could not obtain conclusive results with C-Learning in these specific cases, recent methodologies [15, 45] demonstrate improved generalization capabilities. These new methods appear to be more successful in MuJoCo environments. We reserve the extension of LEADS to such SSR estimators for future work.

0.75

1.00

skill o

0.75

1.00

skill 1

0.50

0S'0

0.25

0.25

00°0

00°0

0.25

0.50

0.25

05'0

0.8

0.75

1.00

1.00

10.0 7.5 5.0 2.50.0

skill 2

2.5

5.07.510.0

0.75

1.00

1.00

10.0 7.5 5.0 2.50.0 2.5

skili 3

5.0

7.510.0

0.75

0.75

0.6

0s'0

0S'0

0.25

0.25

00°0

00°0

0.25

0.25

0.50

0.75

1.00

1.00

10.0 7.5 5.0 2.5  0.0  2.5

skill 4

5.07.510.0

05'0

0.75

1.00

1.00

10.0 7.5 5.0 2.5 0.0  2.5

skill 5

5.0

7.510.0

0.4

0.75

0.75

0.25

05'0

0.25

05'0

0.2

000

00'0

0.25

0.25

0.50

05'0-

0.75

1.00

1.00

0.75

0.0

10.0 7.5 5.0 2.5 0.02.55.07.510.0

10.0 7.5 5.02.50.02.55.07.510.0

1.75

2.00

skill o

2.00

1.75

skill 1

1.0

1.50

1.50

1.25

1.25

1.00

1.00

0.75

0.50

0.75

05'0

0.8

0.25

0.00

2.00

0.000.25 0.500.75 1.00_ 1.251.501.752.00

skill 2

0.00

0.25

2.00

0.00 0.25 0.50 0.75 1.00_ 1.25 1.50 1.75 2.00

1.75

1.75

0.6

1.50

1.50

1.25

1.25

1.00

1.00

0.75

S/0

0S'0

2.00

0.25

0.00

0.00 0.25 0.50 0.75_1.00 1.25 1.50 1.75 2.00

skill 4

0S°0

0.25

0.00

2.00

0.00 0.25 0.50 0.75 1.00_ 1.25 1.50 1.75 2.00

skill 5

0.4

1.75

1.75

1.50

1.50

0.2

1.25

1.25

1.00

1.00

0.75

0.75

05'0

0S°0

0.25

0.00

0.25

0.00

0°0

0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our study illustrates the limitations of using mutual information (MI) as an objective for exploration. We propose a refined objective, offer a theoretical perspective on it, and demonstrate experimentally that it surpasses state-of-the-art approaches.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

1.00

0S'0

0.75

skill 0

1.00

0.75

skill 1

Sz'0

0.00

0.00

0.25

0.25

05'0-

0.75

0.50

0.75

1.00

1.00

0.75

10.0 7.5 5.0

S'-

skill 2

5.07.5

10.0

1.00

0.75

1.00

10.0 7.55.0

2.5

skili 3

2.55.07.510.0

0.50

0.50

0.25

0.25

0.00 -

0.00

0.25

0.25

0.50

0.50

0.75

1.00

1.00

10.0 7.5 5.0

2.5

skill 4

.0.0

2.5

5.07.5

10.0

1.00

0.75

1.00

10.0 7.5 5.0 2.50.0

skill 5

2.55.07.510.0

05'0

oS'O

SZ'0

0.00

0.25

0.00

0.25

0.25

- 0s'0

0.50

s'0

0.75

1.00

10.0 -7.5 5.0 2.5 0.02.55.07.510.0

1.00

-10.0 7.5 5.0 2.5 0.02.55.07.510.0

1.50

2.00

1.75

skill

2.00

1.75

1.50

skill 1

1.25 -

1.25

1.00

1.00

0.75

0.75

05'0

0.25

0s'0

0.25

0.00

2.00

1.75 -

0.000.250.500.75

ski02

1.251.501.75 2.00

2.00

1.75

0.00

1.50

1.50

1.25

1.25

1.00

1.00

0.75

0.75

os'0

0.00

2.00

1.75 -

F Sz'0

0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00

skill 4

2.00

0.00

1.75

0.000.250.500.75

1.00 1.251.50 1.75 2.00

skill 5

1.50

1.50

1.25

1.25

1.00 -

1.00

0.75

0.75

0S'0

F os'0

SZ'0

0.25

0.00

0.00

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss it in a section in the appendix. See section D

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] .

Justification: The code was provided, and the description of the method, along with the details provided in C, ensures full reproducibility of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] .

Justification: Everything is provided in this repository.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes] .

Justification: The paper is self-contained. The method section 3 provide the pseudo-code of the algorithm with all the hyper-parameters used and C provide the values used in this study.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes] .

Justification: For every quantitative experiment, the specific standard deviation is made explicit. Additionally, 2 provides the p-value when the final distributions of coverage are closely matched between methods.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA] .

Justification: All experiments conducted in this study can be performed on any minimal system architecture, without the need for a GPU.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: We preserved anonimity and conformed with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA] .

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes a new skill-based exploration method that leverages successor states. The authors start by arguing the inadequacy of maximizing mutual information as objective for learning diverse skills that also encourage exploration. Then the authors define the mutual information in terms of successor state measures to derive a novel uncertainty measure that ranks states highly that encourage visitation within the current skill but that other skills haven’t visited.  The authors provide comprehensive experiments showing that they outperform common intrinsic motivation methods and other well known skill diversity exploration methods.

STRENGTHS:
- The presentation of the paper is excellent. It is easy to read and follow. 
- The authors motivate the paper very well and start from existing work to derive a novel method. 
- The experiments seem fitting although somewhat simple. 
- The authors perform thorough ablations and also analyse failure cases.

WEAKNESSES:
- The first 2-D environment seems a bit simple. Although I think it is adequate to demonstrate exploration behaviour, it would be interesting to see how LEADS performs with more complex observations or tasks.

QUESTIONS:
- have you tested your methods on any rgb observations? It would be interesting to see if C-Learning is still robust in this case. 
- Could you think of any way to perform C-Learning in high dimensional environments? Is the bottleneck related more to the task complexity or observational complexity?

LIMITATIONS:
The authors have addressed the limiting components of their method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

comments: SUMMARY:
The paper introduces LEADS, an algorithm to learn diverse skills that additionally encourages exploration. LEADS is motivated by the observation that common mutual information-based diversity-seeking algorithms cannot effectively encourage exploration. LEADS instead proposes a new objective based on successor state measures that explicitly encourage the coverage of state spaces. Experiments conducted on state-based control problems validate the effectiveness of LEADS in learning diverse skills that better cover the whole state space than baselines.

STRENGTHS:
1. The work is well motivated with both illustrative examples and theoretical analysis.
2. This work provides extensive comparisons with baselines. The qualitative visualization is particularly helpful for readers to understand the learned exploratory behavior of the proposed algorithm.

WEAKNESSES:
1. In the experiment section, it would be better to briefly introduce the compared baseline methods and point out the main difference between LEADS and these methods before delving into the detailed discussion of results.
2. Besides better coverage of states, are there other advantages of learning more exploratory behaviors? For example, can LEADS achieve better success rates on hard-exploration problems?
3. How does LEADS perform with a different number of skills? Is there a best choice of the number of skills or is LEADS insensitive to this hyper-parameter?

QUESTIONS:
See the weaknesses part.

LIMITATIONS:
LEADS relies on the state occupancy measure estimator to learn diverse skills and encourage exploration. For behaviors that cannot be easily distinguished from the state occupancy measure, the algorithm might not be able to handle them properly.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
The authors aim to solve the problem of exploring to learn a diverse set of skills. To do this, they modify a commonly used mutual information objective in two ways: first, they apply it to the successor measure, and second, they change the sampling distribution to focus on states with high uncertainty. They test this method on a number of challenging goal-conditioned environments, and show that it performs better than existing methonds on several environments, as well as being the only method which achieves good state coverage on all environments.

STRENGTHS:
Originality: 
		There are two main original contributions to this method: the application of mutual information to state successor measures, and the substitution of the uncertain measure as the sampling distribution. In my opinion, this is sufficient originality. 

	Quality: 
		The derivation of the objective is correct. The authors test on a varied set of environments in challenging high dimensional spaces. 

	Clarity: 
		The work is presented well and is easy to follow. 

	Significance:
		The work is a decent improvement over SOTA the authors compare to.

WEAKNESSES:
Using the uncertainty measure instead of the actual distribution breaks most of the theoretical properties this algorithm may have had, but I can see why it was done. It would be nice if there were a more solid mathematical interpretation of this objective, such as a tradeoff between an explicit exploration objective and the mutual information objective, but I understand this is not always possible.

QUESTIONS:
What hand environments are being used? I assume this is HandReach?

LIMITATIONS:
The authors adequately address the limitations of their work
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nHaving an agent learn a set of diverse skills potentially affords the agent better environment exploration. A lack of diversity among the learned skills may reduce the agent's ability to discover newer informative states in the environment. This paper formalizes a method for an agent to learn a set skills by enhancing previous defined mutual information between the states and skills so that it explicitly encourages diversity in exploration. They then evaluate their approach (LEADS) against other algorithms in a number of domains to demonstrate its ability to more effectively explore the state space.\n\nSTRENGTHS:\n**Originality**\nWhile the paper does not introduce any new concept or problem, I consider that the paper does propose an interesting variation to the use of mutual information between states and skills to learn exploratory options that maximize coverage of the state space at the same time that the options are diverse. Both successor features and using mutual information for exploration existed in the literature before this work. The combination done in this way seems interesting.\n\n**Significance**\nThe algorithm represents progress towards the solution of a very relevant problem in deep reinforcement learning: continual exploration based on online observations of the state space.\n\n**Clarity**\nThe explanation of the problem and the general idea of the solution and the paper are easy to follow. However there are some issues explained later.\n\n**Quality**\nThe intuitive concept behind combining successor state measures and mutual information is sound. While the resultant algorithm makes sense, I do not think it works necessarily how the authors explain. The experiments performed are reasonable for empirically backing their claims but the results were not analyzed properly (more later)\n\nWEAKNESSES:\n* While the explanation of the problem and the general idea of the solution are easy to follow. There is some mathematical imprecision in Section 3:\n   1. A measure is a function from a $\\sigma$-algebra to non-negative reals. Here the measure is defined for states rather than sets of states. This is more accurately the successor representation.\n   2. The letter $p$ is used to denote multiple different probability density functions. This is confusing since, $s_1$ and $s_2$ correspond to different random variables: $p(s_1|z)$, the effective state distribution observed in the replay buffer is different from $p(s_2|z)$, which is the state distribution resulting from the discounted visitation.\n\n* Besides the lack of mathematical precision, there is a statement that is not backed up by any argument: line 163 states that the lower bound results in eliminating the natural interplay between the skills that is embedded in the mutual information. There is no explanation for this assertion.\n\n* The related works does not reference work on options or Eigenoptions which are very closely related do successor representations.\n\n* The t-test seems an inappropriate statistical test --- Paired t-tests are better than t-test for comparing algorithms. (See https://arxiv.org/pdf/2304.01315#page=29)\n\n* Sample standard deviation (as used in Fig. 5) is a measure of the variability of a distribution NOT a confidence interval over a mean. Statements such as \"One can note that LEADS outperforms other methods across almost all tasks\"  are unsubstantiated by the provided evidence. One should instead use standard error (as is commonly done in the literature) to obtain a confidence interval. However, a more appropriate choice would either by student-t confidence interval or a bootstrap confidence interval. Please see: https://arxiv.org/pdf/2304.01315#page=13\n\nQUESTIONS:\n* Notation is imprecise. E.g. what is the domain of policy? $\\pi_\\theta: \\mathcal{S} \\times \\mathbb{R}^d \\rightarrow \\mathcal{A}$ or  $\\pi_\\theta: \\mathcal{S}  \\rightarrow \\mathcal{A}$​​?\n\n* The problem setting of Fig 1 is confusing. Is each state reachable? If so the mutual information calculation is wrong. If we assume only the states denoted by the skills are reachable then the calculation is correct but this assumption should be explicitly stated. \nTreatment of how hyperparameters were handled was not specified. Were baselines tuned? How was tuning done? (E.g. grid search? random search? etc.)\n\n* The first bound was introduced to remove the need to sample a minibatch $s_1$ from $z$, but this is still the case after the introduction of the bound. What was gained out of the replacement?\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}