Exploration by Learning Diverse Skills through Successor State Representations

Authors: paul-antoine  le tolguenec
yann  besse
florent  teichteil-konigsbuch
dennis g wilson
emmanuel  rachelson
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 21:07:25

Abstract: The ability to perform different skills can encourage agents to explore. In this work, we aim to construct a set of diverse skills that uniformly cover the state space. We propose a formalization of this search for diverse skills, building on a previous definition based on the mutual information between states and skills. We consider the distribution of states reached by a policy conditioned on each skill and leverage the successor state representation to maximize the difference between these skill distributions. We call this approach LEADS: Learning Diverse Skills through Successor State Representations. We demonstrate our approach on a set of maze navigation and robotic control tasks which show that our method is capable of constructing a diverse set of skills which exhaustively cover the state space without relying on reward or exploration bonuses. Our findings demonstrate that this new formalization promotes more robust and efficient exploration by combining mutual information maximization and exploration bonuses.

Main paper: 
 introduction: 
 Humans demonstrate an outstanding ability to elaborate a repertoire of varied skills and behaviors, without extrinsic motivation and supervision. This ability is currently captured through the problem of unsupervised skill discovery in reinforcement learning [41] , where one seeks to learn a finite set of policies in a given environment whose behaviors are notably different from each other. Implicitly, seeking behavioral diversity is also related to the question of efficient exploration, as a set of skills that better covers the state space is often preferable. Seeking diversity has recently been successfully studied through the prism of mutual information maximization [17, 13, 8] . In this article, we argue maximizing mutual information might be ambiguous when seeking exploratory behaviors and propose an alternative, motivated variant. The key intuition underpinning this formulation is that a good set of exploratory skills should maximize state coverage, while preserving the ability to distinguish a skill from another. Then, we propose a new algorithm which implements this generalized objective, leveraging neural networks as estimators of the state occupancy measure. This algorithm demonstrates better exploration properties than state of the art methods, both those designed for reward-free exploration and those seeking skill diversity.We motivate our developments with a first illustrative toy example. Let us consider a reward-free Markov decision process [38, MDP] (S, A, P, γ, δ 0 ) where S is the state space, A the action space, P the transition function, γ the discount factor and δ 0 the initial state distribution. A behavior policy, parameterized by θ, maps state to distribution over actions. We define a skill encoding z ∈ R d as an abstract set of skill descriptors that enhances the policy description and conditions the action taken a ∼ π θ (s, z), making the policy a function π : Θ × S × R d → ∆(A). Unsupervised skill discovery generally considers a finite collection Z = {z i } i∈ [1,n] of n skills, and a distribution p(z) on skills within Z (generally uniform). For a given θ, each skill z induces a state distribution p(s|z). The state distribution under the full set of skills is hence p(s) = Z p(s|z)p(z). When maximizing diversity, we aim to find θ * such that, for any pair of skills z 1 , z 2 ∈ Z, the states visited by each skill are as separable as possible. Additionally, we would like the state coverage of the full set of skills to cover as much of the state space as possible.Figure 1 : State distributions of two sets Z 1 (left) and Z 2 (right) of four skills each on a grid maze. Each skill's visited states are represented by a different symbol and distributed uniformly. The gray boxes are unreachable.Maximizing diversity has been formalized in the literature as the maximization of the mutual information I(S, Z) between the state random variable S and the skill descriptor Z. This mutual information [39, MI] quantifies how predictable S is, when Z is known (and conversely):I(S, Z) ≜ D KL (P(S, Z)||P(S)P(Z)), = H(S) -H(S|Z),where D KL is the Kullback-Leibler divergence and H is the (conditional) entropy.Figure 1 illustrates why I(S, Z) is an imperfect measure of diversity. In this figure, two sets Z 1 and Z 2 of four skills each are represented by their respective state coverage in a grid maze MDP (one symbol per skill). In each set of skills, each skill is picked with probability p(z) = 1/4. On the leftmost image (Z 1 ), each skill covers a single state, hence p(s|z) = 1 in this state and 0 otherwise. On the rightmost image (Z 2 ), p(s|z) = 1/2 on each covered state and 0 otherwise. Consequently, I 1 (S, Z) = H 1 (S) -H 1 (S|Z) = log 4 and I 2 (S, Z) = H 2 (S) -H 2 (S|Z) = log 4.The mutual information is hence ambiguous as it ranks the two sets of skills equally, while Z 2 enables better exploration.Our contribution is structured around maximizing a variant of I(S, Z), using the successor state representation [4] to enable its estimation. Section 2 puts this idea in perspective of the relevant literature, highlighting similarities and contrasts with previous works. Then Section 3 introduces and discusses our LEADS algorithm. Section 4 reports empirical evidence of the good exploratory properties of LEADS, demonstrating its ability to outperform state-of-the-art methods on a span of benchmarks. 
related work: 
 This section endeavors to provide the reader with a comprehensive overview of approaches which aim at exploration in RL and skill diversity.Exploration bonuses. Driving exploration by defining reward bonuses has been a major field of research for over two decades [25, 6, 1, 3, 36, 7, 2, 19] . These methods generally involve defining reward bonuses across the state space, which decrease over time in well-explored states. In large state spaces where count-based methods are challenging, these approaches require function approximation to generalize across the state space. However, as explained by Jarrett et al. [23] , identifying these exploration bonus approximators is an adversarial optimization problem making these approaches unstable in practice [12] .Quality Diversity. Exploration can also be achieved as a by-product of behavior diversity search.Evolutionary algorithms have been used to search for new behaviors, such as in Novelty Search [27] , and for a diversity of behaviors which encourage high performance through the growing domain of Quality Diversity algorithms, exemplified by MAP-Elites [11] . In these algorithms, skills are usually characterized through hand-crafted behavior descriptors, although there are methods which learn such descriptors through variational inference [10, 9] . Diversity is expressed as the coverage over such descriptors, and policies are modified through gradient-free [11] or gradient-based [32] updates in order to incrementally populate an archive of diverse skills. While these algorithms have demonstrated competitive results with exploration algorithms in RL [9] , in general, they require expertise to define a behavior descriptor and are less sample-efficient than RL methods [28] .Information theory approaches to diversity. The field of information theory offers valuable insights into designing algorithms that generically promote diversity. Gregor et al. [17] , Eysenbach et al. [13] and Sharma et al. [40] were pioneers in proposing methods to maximize either the reverse or forward form of I(S, Z) for a fixed set of skills Z. To address the issue of unbalanced state distributions across skills throughout the entire environment, Campos et al. [8] suggest sequencing the problem into three stages: pure exploration in S to promote sampled states diversity, state clustering via variational inference, and skill learning by maximisation of the forward form as proposed by Sharma et al. [40] . In the exploration phase, Campos et al. [8] employs the State Marginal Matching algorithm proposed by Lee et al. [26] , which resembles MI-based algorithms as it combines mutual information maximization with an exploration bonus to efficiently explore the state space.Focusing even more on exploration, Liu and Abbeel [29] automate learning Novelty Search behavior descriptors by training an encoder with contrastive learning in order to promote exploration. Focusing also on exploration, Kamienny et al. [24] employ an incremental approach combined with the maximization of the reverse form of MI to eliminate skills that are not discernable enough, subsequently generating new skills from the most discernable ones.Finally, recent studies including those by Park et al. [33] , Park et al. [34] , and Park et al. [35] , integrate mutual information (MI) with trajectory-based metrics between states to enhance exploration. Specifically, these studies aim to maximize the Wasserstein distance, also known as the Earth mover's distance, between P(S, Z) and P(S)P(Z) while the definition of MI uses the KL divergence. This distance requires specifying a metric for the state space. Park et al. [33] and Park et al. [34] uses the Euclidean distance, while Park et al. [35] employs the temporal distance.It is important to note that, although these methods induce an interesting approach to exploration (as a by-product of diversity), they do not explicitly aim at encouraging coverage of the MDP's state space.Successor features for MI maximization. Few attempts have been made to exploit universal successor features [5] in MI maximization methods. Machado et al. [30] decompose the learning process into two stages: first, they automatically acquire specific options, which are then used to develop a hierarchical policy that integrates these options. In contrast, Hansen et al. [21] first learn a general task-based policy conditioned on a task z, and subsequently optimize the reverse form of MI by utilizing universal successor features to identify a range of diverse skills efficiently. More recently, Grillotti et al. [18] used successor features to optimize the quality of predefined behaviors based on the Quality Diversity formulation, which requires hand-crafted behavior descriptors.The method we propose explicitly aims at promoting diversity with the goal of achieving good exploration. For this purpose, we turn the MI objective function into a new objective which promotes exploration, separating this work from previous contributions in MI maximization.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss it in a section in the appendix. See section D Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
promoting exploration with successor state representations: 
 We now derive an algorithm that explicitly encourages exploration in diversity search. We start by casting mutual information maximization in terms of state occupancy measures and derive a lower bound which can be maximized with stochastic gradient ascent on the policy's parameters θ (Section 3.1). Then, as maximizing MI does not encourage exploration, we transform this lower bound so that its maximization fits this goal, leading to a new target quantity designed to promote both diversity and exploration (Section 3.2). This enables defining and implementing a general algorithm for Learning Diverse Skills through successor state representations (LEADS, Section 3.3).The mutual information I(S, Z) is naturally expressed through the (conditional) probability densities of S and Z. Successor state representation estimators [4, SSR] provide a direct way of estimating these densities. More precisely, we identify the Successor State Representation as the density of the measure defined on the set of states. The SSR p(s|s 1 , θ, z) of a skill (θ, z) is the state occupancy measure of policy π θ (s, z), when starting from s 1 , that isEQUATIONFor the sake of readability, we make θ implicit in our notations going forward. This measure can be understood as the probability density of reaching the state s 2 starting from another state s 1 when the skill z is used [22] . It encodes all the paths of the Markov chain that can be taken from s 1 to s 2 when the skill z is used [4] . Here (s 1 , s 2 ) ∈ S 2 can be any two states of the Markov chain.The SSR is the fixed point of a Bellman operator and one can employ temporal difference (TD) learning to guide a function approximator towards this fixed point. There are numerous methods for estimating the SSR density [4, 22, 14, 15] . In this paper, we use C-Learning [14] for estimating the SSR for all experiments. C-Learning proposes to cast the regression problem of learning p as a classification problem. Specifically, for a given pair (s 1 , a, z), a classifier σ(f ϕ (s 1 , a, s 2 , z)), where σ is the sigmoid function, is trained to predict whether s 2 was sampled from an outcome trajectory from s 1 (s 2 ∼ p(•|s 1 , a, z)) or from the marginal density of all possible skills (s 2 ∼ p(s), the distribution over S). The optimal classifier holds the following relation with the SSR's density: e f ϕ (s1,a,s2,z) = p(s 2 |s 1 , a, z)/p(s 2 ). We denote m ϕ (s 1 , a, s 2 , z) = e f ϕ (s1,a,s2,z) , as the SSR estimated via C-Learning, taking a ∼ π θ (s 1 , z). For the sake of brevity, we will use the notationm ϕ (s 1 , s 2 , z) = E a∼π θ (s1,z)[m ϕ (s 1 , a, s 2 , z)]. We note that any method which reliably estimates m ϕ (s 1 , s 2 , z) can be used instead of C-Learning in LEADS.As described in Section 1, the mutual information between the state random variable S and the skill descriptor Z is I(S, Z) = D KL (P(S, Z)||P(S)P(Z)). This can be expressed asEQUATIONState s 2 can be any state s within the set S, yet we refer to it as s 2 to ensure that when we incorporate the definition of the SSR, it aligns directly with the notation of Equation 1. By definition of the SSR under a given policy π(s, z), the state distribution p(s|z) obeysEQUATIONBy sampling s 1 ∼ p(s|z), we can therefore estimate p(s 2 |z), allowing for the injection of m(s 1 , s 2 , z) = p(s 2 |s 1 , z)/p(s 2 ) from Eysenbach et al. [14] into Equation 2:I(S, Z) = E (s2,z)∼p(s,z) log E s1∼p(s|z) (m(s 1 , s 2 , z)) .A Monte Carlo estimator of I(S, Z) can be obtained by sampling (s, z) from a replay buffer. However, for every state s sampled this way, the estimate of p(s|z) provided by Equation 3 requires sampling a batch of s 1 ∼ p(s|z). This is possible through keeping separate replay buffers for each skill, but would be computationally intractable for an accurate estimation. Hence, as in similar methods, we turn rather to a lower bound on I(S, Z), which admits a Monte Carlo estimator, and which will be maximized with respect to θ using stochastic gradient ascent.Applying Jensen's inequality to the inner expectation in the previous expression yieldsEQUATIONNote that θ participates in this lower bound through m(s 1 , s 2 , z) = E a∼π θ (•|s1) m(s 1 , a, s 2 , z). This quantity is essentially what an algorithm like DIAYN [13] maximizes: given separate experience buffers for each skill, one can compute a Monte Carlo estimate of this lower bound on the gradient of the mutual information I(S, Z) with respect to θ.Looking closely at Equation 4, one can note that the interplay between each skill's state distribution is lost when deriving the lower bound. Hence, this bound can be maximized without actually pushing skills towards distinct states, hence without skill diversity. To regain the incentive to promote skill diversity, we propose to encourage exploration in each skill to put probability mass on states which receive little coverage by the full set of skills. In other words, for a given transition (s, π(s)), we want to augment the probability of occurrence for one skill while decreasing it for all others.Since m(s 1 , s, z) ≥ 0, we can introduce the following bound:EQUATIONWhile Equation 5 is a looser lower bound than Equation 4, it goes towards the goal of diversity search originally captured by MI maximization: to have distinct state distributions for each skill.However, as illustrated in the introductory example, maximizing MI does not promote large state coverage, nor exploratory behaviors. Equation 5 promotes skill diversity, but it does not explicitly encourage state coverage. We therefore argue that exploratory behaviors can be obtained by focusing the state distribution of each skill towards specific states within the support of p(s|z). Instead of sampling s according to p(s|z), we encourage exploration by attributing more probability mass to states that trigger exploration. We do so by creating a sampling distribution δ(s|z) based on an uncertainty measure u t (s, z), as explained next.Biasing a skill towards promising states for exploration is often achieved through exploratory bonuses based on uncertainty measures [36, 7, 2] , or repulsion mechanisms [16] . These heuristic measures encourage exploration by pushing policies towards states of high uncertainty or away from previously covered transitions. The novelty of our approach is the use of an exploration bonus in the framing of mutual information maximization. However, by replacing s 2 ∼ p(s|z) by another sampling distribution δ(s|z), we lose the theoretic guarantee of maximizing the lower bound of Equation 5.Rather, we propose a new objective, expressing an incentive to explore within the search for skill diversity, but with a greater focus on state coverage through distinct skills than previous lower bounds on mutual information. The quantity we maximize is then:EQUATIONThe uncertainty measure u t which defines the distribution δ(s|z) is designed to explore under-visited areas and to create repulsion between different skills. We define three desired properties for states to prioritize using this measure. To describe these properties, consider a sequence of policies π t , each defining a set of skills {π t (•, z)} z∈Z , and the corresponding sequence of SSRs m t ; the three properties are then as follows. (1) A good target state s z t for skill z at time step t is one that has high probability of being visited by π t (•, z), but was relatively infrequently visited by previous policies' state occupancy measures {m k } k∈[1,t-1] for any skill. (2) It is also a state which has both high probability of being reached by the current π t (•, z) and low probability of being reached by any other current skill, starting from s 0 . (3) Given a previous target state s z t-1 , a good new target state is one that has high (resp. low) probability to be reached by π t (•, z) (resp. any other skill), starting from s z t-1 . While the first property explicitly encourages visiting under-visited states, the two others do not encourage exploration per se, and rather strengthen skill diversity by pushing their state distributions apart. This leads to the idea of ranking target states according to:u t (s, z) = log m t (s 0 , s, z i ) t-1 k=1 z ′ m k (s 0 , s, z ′ )Explore under-visited areas+ z ′ ̸ =z log m t (s z t-1 , s, z) m t (s z ′ t-1 , s, z ′ ) + log m t (s 0 , s, z) m t (s 0 , s, z ′ )Repulsion between skills (7) The δ(s|z) distribution of Equation 6 therefore allocates more probability to states with high u t (s, z). Empirically, we found that making δ deterministic as the Dirac distribution on a state that maximizes u t (s, z) enabled efficient exploration. Appendix A provides a more formal perspective on the derivation of the u t uncertainty measure.With the objective of maximizing G(θ) (Equation 6), we define the LEADS (Learning Diverse Skills through successor state representations) algorithm, presented in Algorithm 1.Algorithm 1 LEADS Initialize θ 0 for t ∈ [0, N ] do # Collect samples D z = ∅, ∀z ∈ Z for e ∈ [1, n ep ] do Sample skill z ∼ p(z) {(s t , a t , r t , s ′ t )} = episode with π θt (•, z) from s 0 D z = D z ∪ {(s t , a t , r t , s ′ t )} end for # Learn the SSR Learn m ϕt for π θt using on-policy C-learning Sample s ∼ δ(s|z) # Improve θ for i ∈ [1, n SGD ] do Sample z ∼ p(z), s 1 ∼ p(s|z) θ ← θ + α∇ θ [G(θ) + λ h H(θ)]Update ϕ t using off-policy C-learning end for end forIn order to maximize G(θ), we must first sample from p(s|z) for each skill z, then compute the SSR for the current skills π t (•, z) in order to define G(θ), then finally update θ to maximize G(θ). As such, an iteration of LEADS features three phases in order to learn a new set of skills: sampling, learning the SSR, and optimizing the policy parameters θ.First, the sampling phase populates separate replay buffers D z for each skill z ∼ p(z). This is achieved by rolling out n ep episodes with a given policy π θ (•, z). Then, C-learning is used to compute the parameters ϕ t of a joint SSR m t (s 1 , a, s, z) for π t , and store it. We note that the on-policy version of C-learning can be used for this step as the data in the replay buffers has been collected with the current policy π t . This SSR, and all previous ones, are then used to define u t (s, z), which in turn defines the distribution δ(s, z). As this can be approximated for a static policy, sampling can be performed before updating θ. Sampling s ∼ δ(s|z) is performed by running all states in D z through u t (s, z), although, to limit computational cost, we can restrict this evaluation and selection to only a uniformly drawn subset of each D z .We therefore have the necessary components to optimize θ according to G(θ): a means of sampling states and the SSR. At each gradient ascent step, we sample a mini-batch of states s 1 from D z for a given skill z ∼ p(z) to estimate p(s|z). This permits calculating the objective G(θ) for the current θ using m ϕt and performing a gradient step. In the gradient calculation, we also include an action entropy maximization term, as done in other works [31, 20] :EQUATIONθ is therefore updated to maximize G(θ) + λ h H(θ), although λ h is intentionally kept small (0.05) to focus on the principal LEADS objective G(θ).Finally, after each gradient step, the SSR is updated using the off-policy formulation of C-learning so that m ϕt remains representative of the state distribution under π θt . This is done without sampling new transitions and is justified by the fact that the target state s z t and the states along a trajectory to s z t are already within the replay buffer D z . n SGD steps of gradient ascent on θ are performed in this way, before a new iteration of LEADS is started.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: See A. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments & results: 
 We demonstrate LEADS on a set of maze navigation and robotic control tasks and compare its behavior to state-of-the-art exploration algorithms. We provide all code for LEADS and the baseline algorithms, as well as the scripts to reproduce the experiments (repository). All hyperparameters are summarized in Appendix C.Mazes. Maze navigation has been frequently used in the exploration literature, as 2D environments allows for a clear visualization of the behaviors induced by an algorithm. The assessment of state coverage is also easier to understand than in environments with high-dimensional states. We design three different maze navigation tasks, named Easy, U, and Hard (depicted in Figure 2 ), of increasing difficulty in reaching all parts of the state space. In each of the mazes, the state space is defined by the agent's Euclidean coordinates S = [-1, 1] 2 and the action space corresponds to the agent's velocity A = [-1, 1] 2 . Hitting a wall terminates an episode, making exploration difficult.Robotic control. We further assess the capabilities of LEADS in complex robotic control tasks. These tasks allow evaluating the ability of LEADS to explore in diverse and high-dimensional state spaces. We evaluate LEADS on a variety of MuJoCo [42] environments from different benchmark suites. Fetch-Reach [37] is a 7-DoF (degrees of freedom) robotic arm equipped with a two-fingered parallel gripper; its observation space is 10-dimensional. Fetch-Slide extends the former with a puck placed on a platform in front of the arm, increasing the observation space dimension to 25. Hand [37] is a 24-DoF anthropomorphic robotic hand, with a 63-dimensional observation space. Finger The first goal of this work, as exposed in the introductory example, is to obtain a set of skills that are as distinct from each other as possible in their visited states, while simultaneously covering as much of the state space as possible. We first assess this property in LEADS, DIAYN [13] and LSD [33] through a visual analysis of their state space coverage in Figure 2 . For all experiments, the number of skills is n skill = 6. To ensure fairness, for each algorithm, we report the skill visualization from the experiment that achieves maximum coverage (as defined in the following section) out of five runs.For each algorithm, Figure 2 presents four figures: the three mazes and the Fetch-Reach environment. In all the mazes, we visualize the 2D coordinates of the states reached over training. Given that 2D visualization is not suitable for the Fetch-Reach environment due to its higher state space dimension, we project the state onto the two first eigenvectors of a Principal Component Analysis (PCA) of the states encountered by all skills in this environment.We note that LEADS clearly defines distinct skills in the state space. Furthermore, it leads to a more extensive exploration of the environment than LSD and DIAYN. The Hard maze (bottom left) is noteworthy, as some parts of the environment are difficult to access due to bottleneck passages in the maze. LEADS is the only algorithm that manages to reach all sections of the Hard maze. Section 3.2 claimed that the uncertainty measure u t (s, z) of Equation 7, and its associated distribution δ triggered relevant exploratory behaviors. Figures 4(a ) and 4(b), display the final SSR and uncertainty measure, respectively, for the Hard maze task. The same visual representation of these densities for the other environments can be found in Appendix B. Figure 4 (a) demonstrates that the SSR clearly separates states based on skill. The SSR is zero for all states outside of the distribution of a given skill, and all states in a skill have a non-zero probability of being reached by that skill. This confirms that this SSR can reliably be used within the objective function of Equation 6.Similarly, Figure 4 (b) illustrates how the uncertainty measure concentrates on states that promote both exploration and diversity. High values for this measure are found in very different states for each skill (hence promoting diversity), and within a skill, the highest value is found far from the starting state s 0 (hence promoting exploration). The state maximizing u t is represented by a colored dot for each skill. This confirms that the uncertainty measure achieves the joint goal of exploring under-visited areas and creating repulsion between skills. By defining δ(s|z) as the state which maximizes u t (s, z) for each skill z, LEADS ensures a continuing exploration of the state space.We now turn to a quantitative evaluation of the ability to explore using diversified skills. For the sake of completeness, we compare LEADS to seven seminal algorithms. Five of these are skill-based, namely DIAYN [13] , SMM [26] , LSD [33] , CSD [34] , and METRA [35] . We use n skill = 6 for each skill-based algorithm and for all environments. The two last algorithms are pure-exploration ones, which do not rely on skill diversity and rather rely on exploration bonuses: Random Network Distillation (RND) [7] and Never Give Up (NGU) [2] .A metric of state space coverage should characterize how widespread the state distribution is, in particular across behavior descriptors that are meaningful for the task at hand. Following the practice of previous work, for high-dimensional environments, we use a featurization of the state, retaining a low-dimensional representation of meaningful variables: the (x, y, z) coordinates of the gripper in Fetch-Reach, the (x, y) coordinates of the puck in Fetch-Slide, and the angles of the finger's two joints and the hinge's angle in Finger. This projection is then discretized and the coverage of a trajectory is defined as the number of visited cells. Figure 5 depicts the evolution of coverage, normalized by the maximum coverage achieved by any run in the current environment. This figure illustrates how coverage progresses relative to the number of samples taken in the environment since the beginning of the algorithm. Shaded areas represent the standard error across five seeds for each algorithm.One can note that LEADS outperforms other methods across almost all tasks. This is especially notable in the Fetch-Slide environment, where LEADS exceeds the coverage of all other methods by over 20%. Furthermore, besides LEADS, no algorithm is consistently good across all tasks. In the Fetch-Reach task, DIAYN, METRA, and CSD each achieved a coverage between 80% and 85%. LEADS excelled with 90% coverage, significantly outperforming NGU at 50% and SMM and RND, both at 20%. CSD surpasses LEADS and all other methods on the Finger environment, achieving a relative mean coverage of 96% where LEADS is still competitive with 87%. On this specific benchmark, NGU, METRA and LSD are competitive with LEADS, whereas DIAYN, SMM and RND exhibit low coverage.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] . Justification: The code was provided, and the description of the method, along with the details provided in C, ensures full reproducibility of the experiments. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification: All experiments conducted in this study can be performed on any minimal system architecture, without the need for a GPU.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
conclusion: 
 In this work, we consider the problem of exploration in reinforcement learning, via the search for behavioral diversity in a finite set of skills. We take inspiration from the classical framework of maximization of the mutual information between visited states and skill descriptors, and illustrate why it might be insufficient to promote exploration. This motivates the introduction of a new objective function for skill diversity and exploration. This objective exploits an uncertainty measure to encourage extensive state space coverage. We leverage off-the-shelf estimators of the successor state representation to estimate this non-stationary uncertainty measure, and introduce the LEADS algorithm.Specifically, LEADS estimates the successor state representation for each skill, then specializes each skill towards under-visited states while keeping skills in distinct state distributions. This results in an efficient method for state space coverage via skill diversity search. Our experiments highlight the efficacy of this approach, achieving superior coverage in nearly all tested environments compared to state-of-the-art baselines.LEADS intrinsically relies on good SSR estimators. In this work, we used C-Learning [14] for all experiments, which proved efficient but might reach generalization limits in some environments. As a consequence, advances on the question of reliable SSR estimation will directly benefit LEADS, opening new perspectives for research.A A formal motivation for the uncertainty measure of Equation 7In Section 3.2, we introduced a new optimization target, based on the uncertainty measure u t (s, z), used in LEADS to strategically distribute probability mass across states for each skill to foster effective exploration. u t (s, z) is introduced in Equation 7, which is reproduced here for clarity:u t (s, z) = log m t (s 0 , s, z i ) t-1 k=1 z ′ m k (s 0 , s, z ′ )Explore under-visited areas+ z ′ ̸ =z log m t (s z t-1 , s, z) m t (s z ′ t-1 , s, z ′ ) + log m t (s 0 , s, z) m t (s 0 , s, z ′ ) 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] .Justification: We preserved anonimity and conformed with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] .Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This section delves into the theoretical foundations of this target. We begin by discussing the left term of Equation 7, which corresponds to the exploration component of the measure. Subsequently, we detail the right term, which promotes greater repulsion between the skills.A.1 Exploration term (left side of Equation 7)The Successor State Representation (SSR) extends the concept of the occupancy measure. Unlike the traditional occupancy measure, which begins with the initial state distribution, the SSR is an estimation of the occupancy measure starting from any state. In practice, we work with the expected sum of discounted presences of a state over time because we know this measure exists regardless of the ergodicity of the Markov process.LEADS is an epoch-based algorithm with each epoch consisting of four phases, as presented in Section 3.2 and Algorithm 1:1. Sample transitions from the environment for each skill, 2. Learn the SSR of each skill (for the given policy), 3. Define a state on which to put more probability mass for each skill, 4. Use the objective of Equation 6to update the policy so that it implements this probability mass transport plan.Since the policy changes in each epoch, the discounted occupancy measure of the policy also evolves over time. For a given skill and epoch n, it can be defined as:ρ πz n (s) = E s0∼δ0 a∼π(•|st,z) st+1∼P (•|(st,at)) ∞ t=0 γ t 1(s t = s)This occupancy measure has a direct estimator, using the SSR in the initial state s 0 :ρ πz n (s) = m n (s 0 , s, z)At epoch n, the state occupancy measure of all skills is the mixture of all skills' state occupancy measures:ρ π n (s) = Z p(z)ρ πz n (s)Then, the left side term in the definition of the uncertainty measure u t (s, z) corresponds to the contribution of a state s to the following Kullback-Leibler divergence:EQUATION= S ρ πz n (s) logρ πz n (s) n-1 k=1 ρ π k (s) ds = S ρ πz n (s) log m n (s 0 , s, z) n-1 k=1 z ′ m k (s 0 , s, z ′ )ds Note in particular that this divergence accounts for the discrepancy between ρ πz n , ie. the states visited by skill z at epoch n, and n-1 k=1 ρ π k , ie. the states previously explored by all skills in all past epochs. Thus, the KL divergence of Equation 9 serves as a measure of novelty for the current distribution. Intuitively, the newer a state visited by the current distribution ρ πz n (s) is, the larger its contribution to this KL divergence will be. Recall that at this stage, we do not modify the policy (it is the role of the objective function of Equation 6), but rather aim at identifying which states within the support of ρ πz n should be those skill z is encouraged to explore next. By defining the state-wise novelty score log mn(s0,s,z) n-1 k=1 z ′ m k (s0,s,z ′ ) , the distribution δ(s|z) unfolds as the Dirac distribution which maximizes the expected value of this score. Overall, LEADS aims to identify and prioritize states with the highest novelty score for each skill, thereby enhancing exploration by shifting each skill's induced distribution towards these novel states.A.2 Repulsion term (right side of Equation 7)Building on the previous discussion about the formal motivation of the exploration term on the left side in 7, we now explore the right side term, which is intended to enhance repulsion between skills.The right side term of Equation 7 can be interpreted as the contribution of a state s within a sum of multiple Kullback-Leibler divergences:EQUATIONIn this equation, term (a) represents the KL divergence of the occupancy measure for the current skill z, starting from its last target state, compared to the occupancy measure of each other skill starting from their respective last target states. Maximizing this KL divergence aims to make the distribution of each skill (starting from their last target state) as distinct as possible. Similarly, term (b) is the KL divergence between the occupancy measures of a skill z and that of any other skill z ′ . Maximizing these two KL divergences leads to disjoint distributions for each skill. As in the previous subsection, we do not change the policy at this stage (this would lead to an ill-defined optimization problem as the KL can become unbounded), but rather aim at identifying which are the states that contribute most to these KL divergences, so that we can promote exploration towards them. Overall, LEADS combines all these terms into the uncertainty measure of Equation 7to determine which state contributes the most to these KL divergences for each skill. In turn, these states define the distribution δ designed to promote exploration, by increasing the weight of the most likely states of δ in Equation 6A.3 Numerical advantage of Lower Bound 4This section offers a more detailed explanation of the passage from the definition of Mutual Information 2 to the Lower Bound 4. The mutual information can be defined as:E (s2,z)∼p(s,z) [log (Es 1 [m(s 1 , s 2 , z)])], with s 1 ∼ p(s 1 |z).Suppose we wish to maximize this quantity using SGD. For each sample (s 2 , z), we need to compute the expectation E s1∼p(s|z) (m(s 1 , s 2 , z)) (because the log is not linear). In practice, this is computationally expensive because it requires sampling a mini-batch of s 1 states for every single s 2 state. Instead, we use Jensen's inequality, leveraging the concavity of the log function to derive the lower bound:E z∼p(z) s2∼p(s|z) s1∼p(s|z) [log(m(s 1 , s 2 , z))]Maximizing this new quantity can be done with SGD by sampling three independent mini-batches at each iteration.LEADS is based on the estimation of the successor state representation for each skill. Figure 6 and Figure 7 show the final SSR and uncertainty measure, respectively, for the maze and Fetch tasks. These correspond to the same runs shown in Figure 2 for LEADS. Figure 5 illustrates the temporal evolution of coverage for all methods across various environments, evaluated over five different seeds. We extend our analysis by presenting the p-values from the t-tests conducted on the distributions of the final (highest) coverage values for these five seeds, comparing the first method to both the second and third methods. We conducted a t-test for each environment in which no method outperformed the second method by a margin greater than 10%. The LEADS algorithm learns diverse skills that evolve over time to maintain exploration of the state space. Unlike the baselines, LEADS does not rely on any reward function to be maximized. Instead, it uses the same tool, the SSR, to plan exploration and adjust the set of skills to ensure effective exploration. Unlike static skills, the skills learned by LEADS are dynamic, continuously adapting throughout the algorithm's execution to move away from previously explored areas, as explained in A. LEADS achieves this by maximizing an uncertainty measure, which includes elements from the Kullback-Leibler divergence between the current occupancy measure of each skill and the history of occupancy measures from the set of skills. This mechanism differs from algorithms such as DIAYN [13] , LSD [33] , CSD [34] , and METRA [35] , which aim to learn a static set of skills. These methods evolve the skills until reaching an optimum or sub-optimum. DIAYN maximizes mutual information, represented as the Kullback-Leibler divergence between P(S, Z) and P(S)P(Z), while LSD, CSD, and METRA maximize the Wasserstein distance between these distributions using different metrics.In contrast, NGU [2] and RND [7] use uncertainty methods that evolve only a single policy over time and do not seek to learn a set of diverse skills. The method closest to LEADS is SMM [26] , which also learns a set of diverse skills that explore using a specific uncertainty measure. SMM uses the loss from a VAE [17] as the uncertainty measure, framing the optimization as a two-player adversarial game.LEADS uses a novelty-based uncertainty measure, which appears to lead to a more stable optimization process leading to a more continous and efficient exploration.The following table ( As with other skill-based algorithms, LEADS's coverage changes with the number of skills. However, the skills learned by LEADS evolve throughout the entire training procedure to visit unexplored areas. Hence, one can expect LEADS to require fewer skills to achieve the same final coverage performance than algorithms that learn static skills, like DIAYN [13] . Choosing the number of skills involves a tradeoff between the acceptable computational runtime of the algorithm (which increases with the number of skills) and the desired efficiency in state space coverage. This choice is also highly problem-dependent. Dynamic adaptation of the number of skills is a challenging and open topic, covered by works such as the one proposed by Kamienny et al. [24] . Although this remains an open question, for LEADS we worked with a hand-chosen, predetermined number of skills and did not focus on this aspect.LEADS relies intrinsically on the quality of the SSR estimator. When this estimator is poor, LEADS might not yield diverse and exploratory skills. For instance, in a number of standard MuJoCo environments from the Gymnasium suite [43] , LEADS performs poorly as a consequence of C-Learning's inability to obtain a reliable estimation of the successor state representation. Figures 8 and 9 illustrate this phenomenon on the HalfCheetah benchmark after 25 epochs of LEADS. We evaluate the discrepancy between the estimated state occupancy measure and the actual distribution of states for a set of 6 skills. To enable visualization of these quantities, we project these densities on the (x, y) plane where x and y are the coordinates of the center of mass of the cheetah's torso. Figures 8 reports the estimated state occupancy measure from the initial state m(s 0 , Π(s), z), where Π(s) = (x, y), using the SSR. In contrast, Figure 9 displays the scatter plot of visited states for each skill in the (x, y) plane. Despite experimenting with an extensive range of hyperparameters for C-Learning, a satisfactory estimation of the measure could not be obtained and the probability density of Figure 8 could not match the (x, y) coordinates of the visited states of Figure 11 .As a comparison, Figures 10 and 11 illustrate the identical experiment conducted in the Fetch-Reach environment reported in the main body of the paper. The state occupancy estimated with the SSR of Figure 10 matches closely the states encountered by each skill in Figure 11 . This corroborates the quantitative results reported in Figure 5 : when a reliable estimation of the SSR is available, LEADS performs effectively.Investigating why C-Learning performs poorly in such MuJoCo environments is beyond the scope of this paper. One could conjecture, following the conclusions of Blier et al. [4] , that learning a successor state representation that generalizes efficiently through high-dimensional state spaces is a challenging task. Although we could not obtain conclusive results with C-Learning in these specific cases, recent methodologies [15, 45] demonstrate improved generalization capabilities. These new methods appear to be more successful in MuJoCo environments. We reserve the extension of LEADS to such SSR estimators for future work. Justification: Our study illustrates the limitations of using mutual information (MI) as an objective for exploration. We propose a refined objective, offer a theoretical perspective on it, and demonstrate experimentally that it surpasses state-of-the-art approaches.• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper received four positive reviews with minor points that were all clarified in the author's rebuttal.

All reviewers agreed the idea proposed in this work (combining successor representations in the mutual information objective typically used in skill learning) is original and significant. There is also consensus that the proposed algorithm is correct. The experimental results are convincing and compare with seven related methods in many scenarios. The paper is also clearly presented.

While intuitively well motivated, the algorithm lacks a detailed theoretical analysis, and it is only motivated formally in Appendix A. However, I believe that the empirical part is strong enough to recommend acceptance.

The authors are encouraged to incorporate the following suggestions in the revised version.

- Incorporate all the points addressed in the rebuttal, including Figure 1 clarification, and the sentences mentioned in the response to reviewer QVxs.

- The authors should fix the mathematical and notational imprecisions mentioned in the rebuttal phase.

- It is recommended to replace the term "measure" with "representation".

- Incorporate relevant missing related work, such as [1].

- Include the discussion regarding the choice of the number of skills and the existing trade-offs.


[1] Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G., & Campbell, M. Eigenoption Discovery through the Deep Successor Representation. ICLR 2018

Reviews: review: paper_summary: paper_summary: Having an agent learn a set of diverse skills potentially affords the agent better environment exploration. A lack of diversity among the learned skills may reduce the agent's ability to discover newer informative states in the environment. This paper formalizes a method for an agent to learn a set skills by enhancing previous defined mutual information between the states and skills so that it explicitly encourages diversity in exploration. They then evaluate their approach (LEADS) against other algorithms in a number of domains to demonstrate its ability to more effectively explore the state space.
strength_weakness: strength_weakness: * While the explanation of the problem and the general idea of the solution are easy to follow. There is some mathematical imprecision in Section 3:
   1. A measure is a function from a $\sigma$-algebra to non-negative reals. Here the measure is defined for states rather than sets of states. This is more accurately the successor representation.
   2. The letter $p$ is used to denote multiple different probability density functions. This is confusing since, $s_1$ and $s_2$ correspond to different random variables: $p(s_1|z)$, the effective state distribution observed in the replay buffer is different from $p(s_2|z)$, which is the state distribution resulting from the discounted visitation.

* Besides the lack of mathematical precision, there is a statement that is not backed up by any argument: line 163 states that the lower bound results in eliminating the natural interplay between the skills that is embedded in the mutual information. There is no explanation for this assertion.

* The related works does not reference work on options or Eigenoptions which are very closely related do successor representations.

* The t-test seems an inappropriate statistical test --- Paired t-tests are better than t-test for comparing algorithms. (See https://arxiv.org/pdf/2304.01315#page=29)

* Sample standard deviation (as used in Fig. 5) is a measure of the variability of a distribution NOT a confidence interval over a mean. Statements such as "One can note that LEADS outperforms other methods across almost all tasks"  are unsubstantiated by the provided evidence. One should instead use standard error (as is commonly done in the literature) to obtain a confidence interval. However, a more appropriate choice would either by student-t confidence interval or a bootstrap confidence interval. Please see: https://arxiv.org/pdf/2304.01315#page=13
questions: questions: * Notation is imprecise. E.g. what is the domain of policy? $\pi_\theta: \mathcal{S} \times \mathbb{R}^d \rightarrow \mathcal{A}$ or  $\pi_\theta: \mathcal{S}  \rightarrow \mathcal{A}$​​?

* The problem setting of Fig 1 is confusing. Is each state reachable? If so the mutual information calculation is wrong. If we assume only the states denoted by the skills are reachable then the calculation is correct but this assumption should be explicitly stated. 
Treatment of how hyperparameters were handled was not specified. Were baselines tuned? How was tuning done? (E.g. grid search? random search? etc.)

* The first bound was introduced to remove the need to sample a minibatch $s_1$ from $z$, but this is still the case after the introduction of the bound. What was gained out of the replacement?
limitations: limitations: N/A.
paper_summary: paper_summary: The authors aim to solve the problem of exploring to learn a diverse set of skills. To do this, they modify a commonly used mutual information objective in two ways: first, they apply it to the successor measure, and second, they change the sampling distribution to focus on states with high uncertainty. They test this method on a number of challenging goal-conditioned environments, and show that it performs better than existing methonds on several environments, as well as being the only method which achieves good state coverage on all environments.
strength_weakness: strength_weakness: Using the uncertainty measure instead of the actual distribution breaks most of the theoretical properties this algorithm may have had, but I can see why it was done. It would be nice if there were a more solid mathematical interpretation of this objective, such as a tradeoff between an explicit exploration objective and the mutual information objective, but I understand this is not always possible.
questions: questions: What hand environments are being used? I assume this is HandReach?
limitations: limitations: The authors adequately address the limitations of their work
paper_summary: paper_summary: The paper introduces LEADS, an algorithm to learn diverse skills that additionally encourages exploration. LEADS is motivated by the observation that common mutual information-based diversity-seeking algorithms cannot effectively encourage exploration. LEADS instead proposes a new objective based on successor state measures that explicitly encourage the coverage of state spaces. Experiments conducted on state-based control problems validate the effectiveness of LEADS in learning diverse skills that better cover the whole state space than baselines.
strength_weakness: strength_weakness: 1. In the experiment section, it would be better to briefly introduce the compared baseline methods and point out the main difference between LEADS and these methods before delving into the detailed discussion of results.
2. Besides better coverage of states, are there other advantages of learning more exploratory behaviors? For example, can LEADS achieve better success rates on hard-exploration problems?
3. How does LEADS perform with a different number of skills? Is there a best choice of the number of skills or is LEADS insensitive to this hyper-parameter?
questions: questions: See the weaknesses part.
limitations: limitations: LEADS relies on the state occupancy measure estimator to learn diverse skills and encourage exploration. For behaviors that cannot be easily distinguished from the state occupancy measure, the algorithm might not be able to handle them properly.
paper_summary: paper_summary: This paper proposes a new skill-based exploration method that leverages successor states. The authors start by arguing the inadequacy of maximizing mutual information as objective for learning diverse skills that also encourage exploration. Then the authors define the mutual information in terms of successor state measures to derive a novel uncertainty measure that ranks states highly that encourage visitation within the current skill but that other skills haven’t visited.  The authors provide comprehensive experiments showing that they outperform common intrinsic motivation methods and other well known skill diversity exploration methods.
strength_weakness: strength_weakness: - The first 2-D environment seems a bit simple. Although I think it is adequate to demonstrate exploration behaviour, it would be interesting to see how LEADS performs with more complex observations or tasks.
questions: questions: - have you tested your methods on any rgb observations? It would be interesting to see if C-Learning is still robust in this case. 
- Could you think of any way to perform C-Learning in high dimensional environments? Is the bottleneck related more to the task complexity or observational complexity?
limitations: limitations: The authors have addressed the limiting components of their method.

Comments: title: Official Comment to Reviewer 98FT
Official Comment to Reviewer QVxs
Response to rebuttal
Official Comment to Reviewer axfE
Official Comment to Reviewer 8kZQ
comment: We thank you for your feedback and for the time you invested in reviewing our paper.
Thank you for your response. We appreciate your acknowledgment of the changes made and are glad to hear that they address your major concerns. Your feedback has been instrumental in improving the paper. Thank you for your thorough review and valuable suggestions.
I would like to thank the authors for their response. The changes made to paper will definitely improve it and they address my major concerns. I will be increasing my score.
Thank you for your response.

The connection to the KL divergence does help motivate the method a bit better. I have increased my score to a 7
Thank you for your response and for raising the score. We appreciate your feedback, which has been instrumental in improving the readability of our paper. Thank you for your time and consideration.
Thank you for your positive feedback and insightful suggestion. We will certainly take this direction into account for future research. We appreciate your valuable input and thank you for taking the time to review our paper.
Thank you for your answer. My questions about further advantages beyond maximum coverage and the number of skills are mostly addressed. I would like to raise the score to 6.
Thank you for the answer!

I think it would be interesting to try these paradigms on problems that do not require immediate skill learning such as in MuJoCo, but where skill also goes through many hierarchies. For example, the agent needs to learn how to pick up an object on a more detailed level, but also needs to understand how to use that object in a broader context. I do realise this is out of the scope for the paper.

Overall, I think this is a nice paper, with a simple and clear idea.