Efficient Skill Discovery via Regret-Aware Optimization
Authors: He Zhang
Ming Zhou
Shaopeng Zhai
Ying Sun
Hui Xiong
Venue: ICML 2025 poster (2025)
Keywords: reinforcement learning
unsupervised skill discovery
Primary area: reinforcement_learning->deep_rl

Abstract: Unsupervised skill discovery aims to learn diverse and distinguishable behaviors in open-ended reinforcement learning.
For the existing methods, they focus on improving the diversity via pure exploration, mutual information optimization and learning temporal representation. 
Despite they perform well on exploration, they remain limited in terms of efficiency, especially for the high-dimensional situations.
In this work, we frame the skill discovery as a min-max game of skill generation and policy learning, proposing a regret-aware method on top of temporal representation learning that expands the discovered skill space along the direction of upgradable policy strength.
The key insight behind the proposed method is that the skill discovery is adversarial to the policy learning, i.e., skills with weak strength should be further explored while less exploration for the skills with converged strength.
As an implementation, we score the degree of strength convergence with regret, and guide the skill discovery with a learnable skill generator. To avoid degeneration, the skill generation comes from an upgradable population of skill generators.
We conduct experiments on environments with varying complexities and dimension sizes.
Empirical results show that our method outperforms baselines on both efficiency and diversity.
Moreover, our method achieves 15\% zero-shot improvement on high-dimensional environments, compared to existing methods.

Efficient Skill Discovery via Regret-Aware Optimization
He Zhang 12
Ming Zhou?
Shaopeng Zhai?
Ying Sun
Hui Xiong 13
Abstract
1. Introduction
Deep Reinforcement Learning (RL) has demonstrated its potential to surpass human decision-making abilities in both simulated environments and real-world applications, such as gaming (Silver et al., 2018; Vinyals et al., 2019), recommendation systems (Xu et al., 2019; Zheng et al., 2023), urban computing (Sun et al., 2023), and robotic control (Black et al., 2024). However, RL is still unable to self-improve autonomously by accumulating new skills without end, preventing the achievement of artificial gen- eral intelligence (Hughes al., 2024; Haber et al., 2018; Colas al., 2022). The principle of Unsupervised Skills Discovery (USD) aligns with this challenge, which focuses autonomously summarizing diverse and useful skills for downstream tasks without explicit rewards.
Unsupervised skill discovery aims to learn diverse and distinguishable behaviors in open-ended re- inforcement learning. For existing methods, they focus on improving diversity through pure ex- ploration, mutual information optimization, and learning temporal representation. Despite that they perform well on exploration, they remain limited in terms of efficiency, especially for the high-dimensional situations. In this work, we frame skill discovery as a min-max game of skill generation and policy learning, proposing regret- aware method on top of temporal representation learning that expands the discovered skill space along the direction of upgradable policy strength. The key insight behind the proposed method is that the skill discovery is adversarial to the pol- icy learning, i.e., skills with weak strength should be further explored while less exploration for the skills with converged strength. As an implementa- tion, we score the degree of strength convergence with regret, and guide the skill discovery with a learnable skill generator. To avoid degeneration, skill generation comes from up up-gradable popu- lation of skill generators. We conduct experiments on environments with varying complexities and dimension sizes. Empirical results show that our method outperforms basclines in both efficiency and diversity. Moreover, our method achieves a 15% zero shot improvement in high-dimensional environments, compared to existing methods.
USD has emerged from the field of unsupervised RI. (Hao et al., 2023; Eysenbach et al., 2022), exploring environ- ments without external rewards based on information theory. Pioneering work (Gregor et al., 2016; Eysenbach al., 2019; Laskin et al., 2022) in this area introduced the idea of using diverse and distinguishable skills by calculating mu- tual information. Building on this, LSD (Park et al., 2022) achieved a novel discovery of temporal dynamic skills by learning temporal representations of states. Subsequently, METRA (Park et al., 2024c) combined temporal represen- tation learning with mutual information and surprisingly demonstrated that their method is scalable, as it closely re- sembles PCA clustering. However, these approaches assume that the skills are uniformly distributed, which simplifies the optimization objective. Other works (Lee & Seo, 2023; Wang et al., 2024a) have explored skill selection policies within hicrarchical frameworks address changing envi- ronments for downstream tasks. However, they did not fully investigate the exploration of diverse skills within a single environment. In general, previous work has overlooked the critical role of skill selection in improving learning effi- ciency and enhancing the diversity of skills.
'Thrutt of Artificial Intelligence, The Hong Kong Univer- sity of Science and Technology (Guangzhou) "Shanghai AI Lab ? "Department of Computer Science and Engineering. The Hong Kong University of Science and Technology Hong Kong SAR. Correspondence to: Ming Zhou <zhouming @ ppw.b.orrg.ccoo en'>, Ying Sun <yings@ @ www.ttgz..du......o gz.ddu.n>>, Hui Xiong <xionghui @ hs..
To address these challenges, our key insights are as follows: (I) skill discovery should be grounded by the strength of the policy, sharing similar idea in auto-curriculum leamn ing (Jiang et al., 2021b); and (2) skills may exhibit sequen- tial dependencies, requiring some to be acquired before oth- ers. For instance, in a manipulation task, the agent should
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Efficient Skill Discovery via Regret-Aware Optimization
generators, which ensures both the skill diversity and their learnability by the agent policy.
converged policy strength
We evaluate our approach across environments with varying state dimensions and complex map structures. The results show that our method significantly improves learning effi- ciency and increases skill diversity in complex environments. In addition, our method achieves up to 16% improvement in zero-shot evaluation compared to baselines.
uniferm skill discovery
skill
unconverged policy strength
rugred-wwree skill discovery
2. Preliminary
Figure 1. Comparison between uniform skill discovery and regret- aware skill discovery. Blue lines indicate skills with converged strength (low regret). Red lines indicate skills need further explore (high regret). By re-balancing exploration based on regret signals, the regret-aware method exhibits improved efficiency.
Conditioned MDP. Following prior work (Park et al., 2024c), we formulate the unsupervised skill discovery problem within a conditioned Markov Decision Process (MDP) framework. Conditioned MDP is formulated as M (S, A, P, R. No, Z). The state space is S C IR". with E S denoting
individual state with dimensions. The action space is A C RMM where E A represents an individual action with dimensions m. The transition dynam- ics is given by the transition function P(s'|s, a) E [o, 1]. po(s) denotes the initial state distribution over S. And Z {z|2 E Rd denotes the latent space of the skill. Given skill z, the skill-conditioned policy (Kim al., 2021) can be defined π 2, [0, 1] that satisfies Enen (a 8, 2) = 1. With the time horizon denoted as T, the trajectory derived from it can be formulated as:
learn a basic skill 'picking up' before the advanced skill "opening the grip' (Liu et al., 2024). Intuitively, for the skills that have been, or no longer be further mastered, the policy strength for these skills towards converging; whereas the policy strength for the unmastered skills can be further improved. Therefore, to quantify this principle, we consider using regret to measure the convergence of policy strength.
In the context of USD, regret quantifies the discrepancy be- tween the actual policy strength and the maximum strength that the agent could have achieved. Lower regret indicates faster convergence to skills, whereas higher regret suggests slower convergence. As illustrated in Figure 1, incorpo- rating regret awareness into skill discovery can enhance learning efficiency compared to naive methods. By leverag- ing regret signals to rebalance exploration, policy learning prioritizes exploration of underconverged skills, rather than indiscriminate exploration of the entire skill space.
{(SL, at, st+1))
If p(2) denotes the distribution of skills, we have
T-1
p" (T (2) =oo(22) II 1
T(a st, P(:441 I (t. ar),
which indicates the probability of r derived from ı and z. In the context of conditioned MDP, the learning objective of the unsupervised skill discovery problem is to find a policy π that maximizes the expectation of the cumulative return over the whole skill space Z, which can be formulated as
On top of these observations, we propose an efficient Regret- aware Skill Discovery (RSD) framework for USD, which is framed as a min-max adversarial optimization algorithm. For implementation, RSD comprises two interleaved pro- cesses: (1) learning an agent policy within a bounded repre- sentation space minimize regrets for generated skills; and (2) learning population of Regret-aware Skill Generator (RSG) for skill generation towards maximizing regrets. The skill learning process builds on METRA (Park et al., 2024c), yet it struggles to distinguish interdependent or similar skills at the representation level due to ignoring the magnitude of skill vectors. As an improvement, we propose a bounded representation learning approach for agent policy, which significantly enhances the diversity of skills. Moreover, we found that using a single learnable skill generator at time leads to skill forgetting, limiting skill diversity, hindering ef- ficiency, and resulting in degeneration (Wang et al., 2024b), which is a common issue in previous work. To address this, we propose maintaining an updatable population of skill
V=(s) EE asmx(-1ee. xnp(())
Σ " =
9441 ((-15,())))))
(1)
where r(st. S141, R and R R: 1--1,11111
Temporal Representation for Mutual Information. Pre- vious works (Gregor et al., 2016; Eysenbach et al., 2019; Laskin et al., 2022) achieve skill discovery by maximizing the mutual information (MI) between states and skills:
p(s, 2) p(s)p(z)
[(S 2) = Ep(x.)) log
Based on the MI objective, METRA (Park et al., 2024c) leverages Wasserstein Dependency Measure (WDM) (Ozair
2

---

Efficient Skill Discovery via Regret-Aware Optimization
Projection
+D
Agent Policy No
Regret-aware Skill Generator We,
Agent Policy sea
%
N
Ni-a
Spacet
dbu
P
Ha-
Ne-1
NG
Skill Distiitttion X Sumpled Vaeke
Na-2
Nix
Ni-ix
Boundal Tempoed Represertation Space
Part 1: Ra, learns skills accarding to skill distribrations
Part 2: learne new 26 arcording
Part Mo, learns skillss according to updated siill distriburions
regret. Upditte
Representation Learning
Stage
Stage
Figure 2. Overview of RSD. The leftmost illustrates how the state space is projected onto bounded temporal representation space. The right side shows learning process of the skill generator: cincles denotes skill distributions, dashed regions indicate coverage. the larger the diverse, solid lines represent trajectories, and flags correspond skills sampled from these distributions. Collectively, these cireles form the population of skill generators for the agent's 's policy. In Part 1, the agent policy expands its coverage by mastering the skills (as seen by the red region outgrowing the blue region). In Part the RSG trains a new Tay to generate high-regre skills, depicted by circle replacing a blue one. Here, Mà represents the skill distribution of the current Tez.
al., 2019) to actively maximize the representation distance between different skill trajectories. The objective be expressed using Kantorovich-Rubenstee duality as follows:
viewed min- max optimization between these two pro- cesses: (I) training agent policy me, to minimize the regret incurred under a set of skills and (2) optimizing a skill gen- erator policy Te, to maximize this regret by proposing new challenging skills. We summarize the overall framework in Figure and present the complete procedure in Algorithm 1, The functionalities of the two components are described:
Iw(S:2)= sup (f(s,))]((ss22) 2)] (p(ss)(s)f((ss2)) 11ll
where f scores the similarity between each s and 2, and is formulated inner production of the state representation d(s) and i... f(s, <(s)) With further derivation, Eq. (2) is equivalent to maximize the following objective:
Agent Policy TØ, This policy maps a state and a skill vector z to an action. At each iteration, we sample a skill from a population of skill generators, and condition the agent policy on 2 to perform reinforcement learning. The objective is improve policy competence and reduce regret in the current skill space.
Iw(St;2) 2) sup Ep(r.) Hó 11
Σ
-(())) 2)
Skill Generator Policy ma, In contrast, this policy op- erates entirely in the latent skill space and generates new skill vectors. Given fixed agent policy T&, we opti- mize næ, to generate skills that induce high regret for the agent. After optimization, To, is added to the population of the skill generator Ps. while outdated generator is removed to maintain population diversity.
A 1 s.t. (0(8+1) o(8))l 1.
(3)
Thus, maximization of Iw can be achieved by learning a temporal representation o(-). With that, METRA further gives unified reward function for policy learning, i.e.,
In the following sections, we first formalize the min-man max optimization framework to claim the global optimization objective, then describe the training of To and present the regret-maximizing optimization of the skill generator тe
r(st, S141, 2) (0(s ((s.))
(4)
The RL process for the agent policy of our method builds upon METRA, while incorporating additional enhance- ments both representation learning and reward function design. These improvements ensure skill differentiation and promote diversity, thus enhancing learning efficiency.
3.1. Min-max Optimization for USD
In this section, formulate our framework min-max adversarial optimization for USD. We begin by separately analyzing the optimization objectives of the two consecutive processes both from the perspective of regret, and then unify them to represent the overall optimization objective.
3. Methodology
Our approach learns diverse and competent behaviors via regret-aware skill discovery. At a high level, RSD can be
3

---

Efficient Skill Discovery via Regret-Aware Optimization
Algorithm 1 RSD
SAC (Haarmoja al., 2018) framework. The value function conditioned be formulated as following equation:
1: Initialize A0 P.. 9, A
2: for k= 1,,,, .,...... Epoch_max do
(= [2) Eaw- (s, a, a log (alss,
(6)
3: Part 1: Updating the TØ;
4: Sample a batch of = from Ps
where (s, 2) is the Q function, and denotes the tropy regularization coefficient defined in SAC. The value function at stage k is defined as Vzn (5o | z). After each stage, the agent policy moves closer to the optimal solu- tion. Thus, the improvement of Vadg (so 22 means that Te, learns a better skill behavior starting from su- The regret of z at stage h can be estimated using the following equation:
5: Collect trajectories using TT, (-1z)
6: j 2..................... Agent, Agent_Poliey Training Steps do
7: Sample transitions from buffer: (st, 88+1, a, 2)
8: Update Φb to maximize (8( 21+1,2) (St, 8t+1)
9:
Update À to minimize (Co
10: Update TH, using SAC with reward (8t 86+1,2)
11:
end for
Regb (2) = Vag (su 2) V 2):
12:
Part 2: Updating the Tea
for ..2..... RSG Sraiiinngn Steps do
where ny represents the agent policy at learning stage k.
14:
Sample from initialized #0,
15:
Update Tdz to maximize Jø, (z) according to Tg.17 "01
Considering these two policies as min-max adversarial op- timization RL problem (Durugkar et al., 202 I), we describe our algorithm in following mathematical form:
k-1
16:
end for
17:
Update skill population Pa using T0,
18:
end for
min max Ex 01
(Vay (so 12)
V m0z
(50
| [2) (8)
02
The RL process for no, aims to maximize the cumulative intrinsic return related to skills, according to Eq. 4. Accord- ing to ideas from no-regret algorithms (Sutton, 2018), the value of regret reflects the policy's learning strength. Thus, in the context of USD, regret conditioned on serves as the discrepancy between the actual strength of the skill policy and the maximum strength that the agent could potentially have achieved. However, calculating regret based on its original definition assumes that the optimal cumulative re- turn from the optimal policy can be determined, which is challenging under our circumstances. Inspired by previous work (Rutherford et al., 2024), we use the TD error in n steps to estimate the value of regret. We assume that the regret at learning stage can be calculated follows:
3.2. Agent Policy Learning in RSD
Following the formulation in Section 2, the agent's pol- icy learning consists of representation optimization and policy optimization. According to Eq. 7, searching & in an unbounded space for the maximum of regret is infeasi- ble. To address this limitation, we introduce representation optimization within bounded representation space. Our bounded representation function is defined é(-) that sat- isfies || || S 1 by employing the function tanh(-).
In order to represent more information, we assume that the skill & can be any arbitrary vector in a bounded space. Due to this transformation, the scale of the inner product output changes, which undermines the learning stability of o. To address the problem, we design Zupdated to replace the unit z, which is formulated as:
Regx = V/= - V/-1,
(5)
where V is the value function of the RL policy as defined in Eq. 1. Consequently, the objective of the agent's policy can be regarded as minimizing regret after each iteration.
o(s.) || ((+))
uppaated
In addition, the population Ps of Te, aims to identify the most promising skills with a significant lack of convergence to learn. The regret, estimated using TD-error, also serves as a feasible measure for this purpose. A positive regret for a skill indicates that the latest learning policy surpasses the previous one, and the larger the regret, the more under- converged strength the skill possesses. Thus, the objective of no, can be regarded as finding distribution that corre- sponds to the maximum regret at stage k.
In this way, the information from the magnitude of can be used and the scale of the inner product keeps stable the same time. The objective of & can be formulated as:
La(s,, S141, = Ep(ra) (
Zupdated
(10)
Due to the bounded nature of (()), it naturally satisfies the Lipschitz constraint. However, an additional constraint arises from the maximum number of timesteps, dictated by
To explicitly estimate the value of regret, we first calculate the value function of no,, which is optimized using the

---

Efficient Skill Discovery via Regret-Aware Optimization
the temporal relationship. A well-learned skill trajectory {18t, a, 8t+1, =) in the bounded space must hold:
unseen 2, the bias can mislead RSG, which would sample skills far from the frontier of the agent's is capability. This issue is analogous to the challenge of maintaining stabil- ity of policy updates in RL. For instance, Proximal Policy Optimization (PPO) (Schulman et al., 2017) mitigates ex- cessive policy deviation by employing a clipped advantage technique. Inspired by PPO, we propose a skill proximity regularizer to measure the distributional distance between the newly learned skill distribution and the previous ones.
Σ (1((s(1)) - -(s)) 1
(11)
A direct way to satisfy this constraint is to set an upper bound on the distance at each timestep. Thus, the constraint on the representation function o can be formulated as:
1 T - (1(1((()+)) (++)) 2 0, vl E (0....,T-1}, 1}.
To ensure diversity for skill learning, we first introduce a regularization to ensure that x2_ is different from Pz. This term measures the distributional distance between the newly learned distribution Tेg and the current Pz. to avoid adding redundant skills. We use KI. divergence to estimate the diversity of the skill distributions follows:
Using the dual objective to optimize the constrained opti- mization problem, the final objective of the representation function o can be formulated as:
min max To(Ae, (1+1,) + XC &L))))
(12)
d₂ DRL [p(2 P!) || T (=)]
(16)
where Cp(se, min(e, - T l|0($(+1) (ss))l).
Furthermore, we introduce another regularization to ensure that zे, remains close to the frontier of the agent's capa- bility. If _ is too far from the distribution of seen states, learning the new skill becomes too difficult. To mitigate this uncertainty, we maintain buffer By that only stores the representation of states observed during each stage k. Since the distribution of B, is generally not a standard distribu- tion, we cannot directly compute the KL divergence as in de (z). Instead, we focus on the nearest point in B₁, relative to n... If the probability of the nearest point to Tř, is high, the distance between the point and सž. is small, indicating that the uncertainty of the distribution is low and vice versa. Therefore, the regularization do can be defined follows:
After optimizing o, the skill x is aligned with the repre- sentation of the states. To utilize the magnitude of 2, we assume that agent Te (- 2) eventually arrives the vector in the representation space. Therefore, the intrinsic reward is designed based on the relative distance as follows:
ro( S1+1, 2) = ||2 ((s)) - || 1(++ (13)
Finally, no, (a 2, is optimized using the objective of SAC with the reward r r8(81, 2)) at each learning stage.
3.3. Regret-aware Skill Generator for Skill Generation
According to Eq. 8, the objective of RSG is to find to maximize the regret. Using the policy gradient (Sutton et al., 1999; Silver et al., 2014) framework, we consider the distribution of 2 as the action, and the regret serves as the value of the policy. The is initialized at each stage before optimization. The objective of ma, at stage ls is written as:
d max log Tu2 (dseen))
(17)
where éxeen represents the state representations from the buffer at stage k. We are going to increase dø ensure that the new distribution is close to the distribution of seen states.
max J(z) = B B [R) 02
(14)
Finally, we define the positive constants a1 and 02 as weights for d, and dg. respectively. Therefore, the objective function to optimize Tæ₂ can be summarized as follows:
We assume that a skill distribution may not be fully mastered in a single stage. To stabilize the learning process, we maintain a population of skill generators, denoted as P., which can be formulated as follows:
max Joz -EZnat [Rega(z)) + and; + ardg- 0,
(18)
pk
{ T | 0, ...... min(1,)) k)},
(15)
4. Experiments
where I denotes the maximum size of Ps, and m/.- is the historical me, stage h i. The probability of 2 given ph is given by: p(2 P!) 1 [ (2)] where ( denotes that Té. is uniformly sampled from P...
4.1. Environments and Baselines
We compare our method with baselines on ant environ- ment from dm control (Tunyasuvunakool et al., 2020), Maze2d-large, Antmaze-medium, and Antmaze- large from
Although regret could reflect the converge strength, the re- gret estimate itself inevitably contains bias. Especially for
D4RL (Fu et al., 2020). The Ant environment is used in LSD and METRA to demonstrate that the agent can learn
5

---

Efficient Skill Discovery via Regret-Aware Optimization
CoverCoond
CovorCaena
PTM
S0eps
Stops
Stepa
Stepe
(a) Ant.
(b) Maze2d-large.
(c) Antmaze-medium
(d) Antmaze-
Figure Comparison between RSD baselines, represented as red curves in different environments. The x-axis shows timesteps
interaction, while the y-axis represents Unique Coordinates, which measure state coverage achieved through sufficiently sampled skills.
State Space
Repr. Space
State Space
Repr. Space
Mape Lunge
030
-100
Aztmaze large
State
Repe Space
Stote
Hepe Stace
(a) RSD.
(b) METRA.
Figure 4. Visualization of Dynamical skills learned in Ant envi-
ronment. Trajectories in State Space is shown on the left, and
(a) RSD.
METRA.
trajectories in Representation Space (Repr. Space) on the right.
Figure 5. The visualization of state and representation coverage in Maze2d-large and Antmaze- large. The results of RSD shown on the left, and the results of METRA are on the right.
temporal dynamic skills. The Ant environment contains an ant robot moving in an open map without obstacles, where exploration any direction is nearly equivalent. We refer to this type of environment as skill-symmetric en- vironment. In contrast, exploration in environments with obstacles should vary in directions, which we refer to as skill-asymmetric environments. Maze2d-large has a lower state dimension compared to the other environments. The nutmaze-mediune map is smaller and less complex than the Maze2d-large and Antmaze-lan large maps. We use these envi- ronments demonstrate the performance of our method in high-state dimension and more complex map.
can improve efficiency simply by adaptively sampling skills, strategy that has been overlooked in previous work.
When evaluating, we thoroughly sample the in the latent skill space Z to ensure that they cover the full range of skills. We then assess state coverage by recording the unique coor- dinates visited on the map, referred to as CoverCoords, to evaluate the diversity of skills. A higher value of CoverCo- ords indicates higher state coverage across all skills, which means that the skill set is more diverse. particular, our RSG learns Gaussian distribution each stage, making Pa Gaussian Mixture Model (GMM). Additionally, design greedy adaptive sampling technique during the agent's s learning process. A more detailed experimental setup and techniques of our method can be found in the appendix C.
We primarily compare our method with the baselines ME- TRA, LSD, DIAYN, and DADS. DIAYN one of the first methods to highlight the potential of diverse skill discov- ery to solve general tasks based on mutual information re- formulated with entropy. DADS prioritizes discovering predictable behaviors while simultaneously learning the dy- namics of the environment. LSD further introduces novel Lipschitz representation space to discover temporal dynamic skills. METRA builds on LSD and mathematically estab- lishes the relationship between its objective and PCA-based clustering, demonstrating RSD with the baselines demonstrate that ours retains the advantages of METRA while improving learning efficiency and state coverage in skill-asymmetric environments. Other USD methods, such as (Gregor et al., 2016; Laskin et al., 2022; Bac et al., 2024), also use maze environments to test their diversity. However, they either fail to learn dynamic skills in high-dimensional environments or rely on addi- tional exploration intrinsic rewards. In contrast, our method
4.2. Experiment Analysis
Figure 3 shows the curves of CoverCoords over timesteps. Our method demonstrates competitive results in skill- asymmetric environments. According to Figure 3a, we demonstrate that RSD has the ability to learn dynamic tem- poral skills in different directions, compared DIAYN and DADS. Also, the curves show that the state coverage of RSD is lower than LSD and METRA. This outcome in skill aslmmmsyymttric environments can be attributed to two key factors: First, in baseline methods, the skill is represented as a unit vector, lacking magnitude information. In contrast, our approach samples 2 from the entire space Z, requiring the learning broader range of skills compared to previ-
scalability. compare our
6

---

Efficient Skill Discovery via Regret-Aware Optimization
Repr. Space
Repr. Space
1.5
440
Rogne
Emesteps
Emessops
1a1 6.5
1
210
800
Figure 7. The curves of Regret and Entropy over timesteps in Maze2d-large. Regret is calculated using Eq. 7 after updating the Ta. And the Entropy denotes the entropy of the P. after updating the Taz estimated by Monte Carlo method.
(a) 2c7 Steps
(b) 4c7 Steps
Repr. Space
Repr. Space
of similar but different skills. Figure 5 visualizes why our algorithm enhances state coverage in these environments. Our method is capable of discovering skills to more corners in maze maps, and its learned representations exhibit more discriminative details compared to those of METRA.
1.25
-05
+833
(c) 7c7 Steps
(d) 9c7 Steps
To further demonstrate this, we present contour maps of the skill distributions Pa at different learning stages. In order to further highlight the learning process related to the skill pop- ulation Pz, we also plot the trajectories in the representation space on the map, as shown in Figure 6. A direct observa- tion that P. differs between different learning stages. This shows that Ps is capable of sampling in various directions following the objective of Eq. 18. As shown in Figure 6b, a single Ps may exhibit multiple peaks, indicating the pres- ence of valuable skills to learn at the same time. These adjacent peaks can serve as form of contrastive learning for similar behaviors, which explains why RSD learns more detailed skills. Further results in Figure 6c confirms that the agent actually learns the corresponding skills with a high probability Figure 6b after training subsequently. Specifically, this region corresponds the branching area in the upper-right corner of the lower-right corner on the Antmaze-large map.
Figure 6. Filled contour plot of the population of skill generators in representation space across different training stages. Color-coded lines depict distinct skill trajectories sampled according to P..
ous methods within the same number of timesteps. Second, in skill-symmetric enviromments like Ant, every direction equally potential, making uniform sampling the most effective approach. However, our method can still learn sufficiently diverse directions, producing results similar to those of METRA, as shown in both spaces in Figure 4.
As shown in Figures 3b, 3c, and 3d, our approach achieves greater efficiency and greater state coverage in all three envi- ronments. these -symmerimetiiccc environments, DIAYN and DADS struggle to explore beyond the first corner of the map, resulting low coverage. The reason is that, although these previous methods effectively differentiate and cluster skills in the latent space, they struggle with long-range ex- ploration and fail to learn temporally dynamic skills. Both METRA and LSD perform reasonably well in Maze2d-large environment, where the state space has relatively low di- mension. However, as the state dimension increases in the following environments, the efficiency of the LSD methods decreases. In comparison, our method in these environments not only has greater learning efficiency, but also achieves higher state coverage without incorporating additional in- trinsic exploration rewards, such as RND (Bae et al., 2024).
Figure 7 depicts the trends of the Regret calculated based on Eq. 7 and the Entropy of Pa in Maze2d-large environ- ments. The sharp rise of Regret demonstrates that Pa first identifies the skills that maximize regret, presumed to be the agent's capability boundary. Meanwhile, the entropy increases, indicating that the skill population becomes more diverse. Subsequently, the Regret begins to decrease, while the entropy remains stable. This shows that the agent has mastered a wider set of skills. Finally, the regret diminishes to zero, suggesting that further learning yields no additional benefits. This trend also serves as evidence that our algo- rithm is capable of achieving convergence.
We observe that the baseline method tends to explore pri- mary directions, consistent with the PCA-like theoretical ex- planations provided in their paper (Park al., 2024c). How- ever, in kkill-ssymmetry environments, this feature leads to homogeneous skills, resulting in a less diverse skill set. In contrast, our method employs non-unit vector z and non- uniform sampling strategies. The non-unit vector helps to distinguish the points behind barriers, and the non-uniforn sampling strategies allow for more focused exploration
To better understand the contribution of each module, conduct ablation studies, with detailed results presented in Appendix E. Additionally, we evaluate our method in the kitchen environment, utilizing a discrete skill settings alongside our proposed skill generation policy. Further details can be found in Appendix
7

---

Efficient Skill Discovery via Regret-Aware Optimization
4.3. Zero-shot Performance
Table I. Performance evaluation under the goal-condition tasks, The bolded data represents the best performance for the correspond- ing metric. Metra-f (Metra-fixed) represents the setting where 2 remains fixed after initialization. Metra-d (Metra-dynamic) repre- sents the setting where z is recalculated at each timestep.
We use navigation tasks as downstream tasks to evaluate the zero-shot capability of our method. During the evaluation process, the agent is provided with a specific goal state Sy. without any further training process or any external signals. We directly measure the agent's final distance (FD) to the goal point and the success rate (AR) under the correspond- ing zg. For baselines, 2g is defined as the unit vector of the difference between the representation space vectors of Sy and 8o- For our algorithm, 2g corresponds to the vec- tor representation of sg. We selected Maze2d-large and Antmaze- environments for comparison, as they share similarly shaped maps but differ in state dimensionality. We uniformly sampled goal points from each map to serve as Sg. with detailed visualizations provided in Appendix D.
Mazeed-
Antmaze-large
FDJ
ART
FD]
AR1
DIAYN (Eysenbach et al., 2019)
4.249
0.130
23.7
0.043
DADS (Sharma et al., 2020)
4.972
0.073
23.8
0.043
LSD (Park et al., 2022)
2.843
0.304
8.12
0.362
METRA-F (Park et al., 2024c)
3.335
0.333
9.90
0.450
METRA-d (Park et al., 2024c)
0.658
0.782
6.76
0.435
RSD(Ours)
0.535
0.811
6.42
0.507
glia et al. (Mazzaalll et al., 2022) achieves skill discovery through model-based RI., which is more computationally expensive and requires additional exploration strategies.
As shown in Table I, RSD achieves the highest success rates (AR) and the smallest final distances (FD) in both environments. These results demonstrate that our algo- rithm has zero-shot capability after unsupervised explo- ration and skill learning, similar to previous work. Further- more, the improvement in AR is more pronounced in the higher-dimensional Antmaze-large environment, whereas the reduction in FD less significant compared to the lower- dimensional Maze2d-large environment. This discrepancy arises because the baseline can already explore most rela- tively reachable points in Maze2d-large. The improvement in FD is less sigmificant in digh-dimensional environments because Sy is more likely to fall out of distribution, leading it to misinterpret the goal and select suboptimal or incorrect directions at the beginning. This mismatch explains why the reduction in FD is less pronounced in high-dimeesional environments compared to low-dimensional counterparts.
However, previous works like (Eysenbach et al., 2019; Sharma et al., 2020; Gregor et al., 2016) does not account for temporal dynamics in skill learning in high-dimension environments, as pointed by Park, S et al. (Park et al., 2022).
To address this limitation, they propose LSD to address this limitation by introducing temporal representation learning in Lipschitz space. METRA (Park et al., 2024c) further focuses on mutual information between the last states and skill variables, and utilizes Wasserstein dependency to de- compose the objective. They demonstrate that METRA is approach analogous to PCA clustering, and is reliable in nigh-dimensional spaces. However, these methods typically sample skills uniformly, overlooking the relation between skills and the learning policy in different environments.
5.2. Automatic curriculum learning
5. Related Work
Automatic curriculum learning (ACL) is mechanism that leverages the learning status of the agent's capabilities to adaptively change the distribution of training data (Portelas et al., 2020). Generally, ACL have multiple explicit tasks or goals to leam, which is widely used in robotic manipula- tion (Fournier et al., 2018; Fang et al., 2019), or multi-goal navigation (Sukhbaatar et al., 2018; 8; Florensa et al., 2018; Colas et al., 2020). Another purpose of ACL is to facilitate the open-ended exploration (Pong et al., 2020; Jabri et al., 2019; Colas et al., 2020), where the distribution of achiev- able goals is initially unknown. Among these, PLR (Jiang et al., 202 is most related to our topic, identifying the lim- itation of random sampling of training levels independently of the agent policy. They introduces scoring functions based on GAE (Schaul et al., 2016) to indicate the learning sta- tus across varying environments. Subsequent work (Jiang et al., 2021a) has expanded on this foundation. They pro- pose teacher-student dual-policy architectures to handle with more difficult maps. PParker-Holder et al., 2022) also uti-
5.1. Unsupervised skill discovery
Unsupervised skill discovery (USD) is an important subfield of RI. exploration based on information theory (Hao et al., 2023; Ladosz et al., 2022). USD addresses unsupervised ex- ploration and learning useful behaviors (Gregor et al., 2016), enabling rapid adaptation to downstream tasks. As one of pioneering works, DIAYN (Eysenbach et al., 2019) demon- strates the feasibility that the discovering diverse skill im- proves downstream task performance. They employ mutual information conditioned on skill vectors and optimize a vari- ational lower bound as their objective. Later, CIC (Laskin al., 2022) maximize the entropy by employing contrastive learning between states and skills, enhancing the diversity of learned skills. While Yang et al. (Yang et al., 2023) also successfully leverage the advantage of contrastive learning, their method fails in high-dimensional settings and suffers from unstable training due to conflicting objectives. Mazza-
8

---

Efficient Skill Discovery via Regret-Aware Optimization
lizes min-max regret policy design environment for improving the robustness of the RI. agent. In other appli- cation domains, adversarial learning frameworks (Li et al., 2023; 2024) have also demonstrated effectiveness in en- hancing model robustness against aatto--distribution data scenarios. However, these ACL approaches rely on explicit rewards, making them unsuitable for unsupervised discov- ery scenarios which lack stable reward signals. Our method distills insights from them and leverage them for efficient unsupervised skill discovery.
foresee any ethical concerns associated with this work.
Software
Our code is open-source at ZhHe11/RSD.
com/
References
Afsar, M. M., Crump, T., and Far, B. Reinforcement learn- ing based recommender systems: survey, ACM Com- puting Surveys, 55(7):1-38, 2022.
6. Conclusion
In this paper, we argue that the skill discovery process is influenced by policy converged strength. Based on this in- sight, we proposed min-max adversarial algorithm (named RSD) which comprises agent policy and RSG to hance learning efficiency. By further incorporating skill proximity regularizer and an updatable skill population, our method effectively balances performance and diversity within a bounded temporal representation space. Experi- mental evaluations demonstrated the effectiveness of our approach, highlighting improvements in sample efficiency, skill diversity, and zero-shot abilities, particularly in skill- asymmetrical scenarios. These findings underscore the po- tential of our method to address challenges in efficient ex- ploration without relying on additional intrinsic exploration rewards. For future work, we aim to integrate our frame- work with hierarchical RL architectures, enabling efficient applications in the scenarios that involve asymmetric skills.
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. CoRR, abs/1707.014999 2017.
Bae, J., Park, K., and Lee, Y. Tldr: Unsupervised goal- conditioned rl via temporal distance-aware representa- tions. arXiv preprint arXiv:2407.08464 2024.
Bagaria, A., Senthil, J. K., and Konidaris, G. Skill discovery for exploration and planning using deep skill graphs. In International Conference on Machine Learning, pp. 521- 531. PMIR, 2021.
Bai, C.,, Yang. R., Zhang, Q., Xu, K., Chen, Y., Xiao, T., and Li, X. Constrained ensemble exploration for unsupervised skill discovery. In Forty-first International Conference on Machine Learning, 2024.
Acknowledgements
Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K.,, Ichter, B., Jakubczak, S., Jones, T., Ke, L., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Shi, L. X., Tanner, J., Vuong, Q. Walling, A., Wang, H., and Zhilinsky, U. mp: A lision-lagguage-actio flow model for general robot control, 2024.
This work was supported in part by the National Key R&D Program of China (Grant No. 2023YFF0725001, in part by Shanghai Artificial Intelligence Laboratory and National Key R&D Program of China (20227D01 19320),, in part by the National Natural Science Foundation of China (Grant No. 92370204, 62306255), in part by the Guang- dong Basic and Applied Basic Research Foundation (Grant No. 2023B 1515 1550520557, 2024A151501 Guangghhu--HKUSTCCCC Joint Funding Program (Grant No. 2023A03J0008), the Fundamental Research Project of Guangzhou (No. 2024A0434233), and Education Bureau of Guangzhou Municipality.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Explo- ration by random network distillation. In International Conference on Learning Representations, 2019.
in part by
Campos, V., Trott, A., Xiong, C., Socher, R., Giró-i Nieto, X., and Torres, J. Explore, discover and leam: Unsu- pervised discovery of state-covering skills. In Interna- tional Conference on Machine Learning, pp. 1317-1327. PMLR, 2020.
Impact Statement
This paper advances unsupervised skill discovery in rein- forcement learning by improving both efficiency and di- versity in skill acquisition. Potential applications include real-world manipulation tasks involving sparse rewards and open-ended challenges, offering benefits such reduced computational costs and improved adaptability, We do not
Colas, C., Karch, T., Lair, N., Dussoux, J.-M., Moulin- Frier, C., Dominey, P., and Oudeyer, P.-Y. Language as cognitive tool to imagine goals curiosity driven exploration. Advances in Neural Information Processing Systems, 13:3761-37744 2020.

---

Efficient Skill Discovery via Regret-Aware Optimization
Colas, C., Karch, T., Sigaud, O., and Oudeyer, P.- Y. Autotelic agents with intrinsically motivated goal- conditioned reinforcement learning: short survey. Jour- nal of Artificial Intelligence Research, 74:1159-1199, 2022.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-pooliyy maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018
Ding. Y., Florensa, C.,, Abbeel, and Phielipp. M. Goal- conditioned imitation learning. Advances in neural infor- mation processing systems, 32, 2019.
Haber, N., Mrowca, D., Fei-Fei, L., and Yamins, D. Emergence of structured behaviors from curiosity- -based intrinsic motivation. arXiv preprint 1rXiv:1802.077 2018.
Durugkar, I., Tec, M., Nickum, S., and Stone, Adversar- ial intrinsic motivation for reinforcement learning. Ad- vances in Neural Information Processing Systems, 34: 8622-8636, 2021.
Hao, J., Yang, T., Tang, H., Bai, C., Liu, J., Meng, Z., Liu, P., and Wang, Z. Exploration deep reinforcement learning: From single-agent multiagent domain. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Representations 2019.
Hughes, E., Dennis, M., Parker-Holder, J., Behbahani, F, Mavalankar, A., Shi, Y., Schaul, T., and Rocktaschel, T. Open-endedness is essential for artificial superhuman intelligence. arXiv preprint M..iv:2406.0426600 2024.
Eysenbach, B., Salakhutdinov, R., and Levine, S. C- learning: Learning to achieve goals via recursive clas- sification. arXiv preprint arXiv:2011.08909 2020.
Jabri, A., Hsu, K., Gupta, A., Eysenbach, B., Levine, S., and Finn, Unsupervised curricula for visual meta- reinforcement learning. Advances Neural Information Processing Systems, 32, 2019.
Eysenbach, B., Zhang, T., Levine, S., and Salakhutdinov, R. R. Contrastive learning as goal-conditioned reinforce- ment learning. Advances in Neural Information Process- ing Systems, 35:35603-35620, 2022.
Ji, Y., Sun, Y., Zhang, Y., Wang, Z., Zhuang, Y., Gong, Z., Shen, D., Qin, C., Zhu, H., and Xiong, H. A comprehen- sive survey on seff-interpretable neural networks. arXiv preprint arXiv:2501.15632 2025.
Fang, M., Zhou, T., Du, Y., Han, L., and Zhang, Z. Curriculum-guit -uideed hindsight experience replay. Ad- vances in neural information processing systems, 32, 2019.
Jiang, M., Dennis, M., Parker-Holder, J., Foerster, J. N., Grefenstette, E.,, and Rocktäschel, T. Replay- ad- versarial environment design. CoRR, abs/2110.02439, 2021a.
Florensa, C., Held, D., Geng, X., and Abbeel, Automatic goal generation for reinforcement learning agents. In International conference machine learning, pp. 1515- 1528. PMLR, 2018
Jiang, M., Grefenstette, E., and Rocktäschel, T. Prioritized level replay. In International Conference on Machine Learning, pp. 4950-4950. PMIR, 2021b.
Fournier, P., Sigaud, O., Chetouani, M., and Oudeyer, P.-Y. Accuracy-baut based curriculum learning in deep reinforce- ment learning. arXiv preprint arXiv:1806.099 09688666066444..01 2018.
Kaelbling, L. P. Learning to achieve goals. In IJCAI, vol- ume 2, pp. 1094-8. Citeseer, 1993.
Fu. J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Kaplan, and Oudeyer, P.-Y. In search of the neural circuits of intrinsic motivation. Frontiers in neuroscience, 1:9, 2007.
Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C., Eysenbach, B., and Levine, S. Learning to reach goals via iterated supervised learning. arXiv preprint /rXiv: 2019.
Kim, H., Lee, B. K., Lee, H., Hwang, D., Park, S., Min, K., and Choo, J. Learning to discover skills through guidance. Advances in Neural Information Processing Systems, 36, 2024.
Gottlieb, J. and Oudeyer, P.-Y. Towards a neuroscience of active sampling and curiosity. Nature Reviews Neuro- science, 19(12))758-777 2018.
Kim, J., Park, S., and Kim, G. Unsupervised skill discovery with bottleneck option learning, 2021.
Gregor, K., Rezende, D. J., and Wierstra, D. Variational intrinsic control, 2016.
Kim, K., Sano, M., De Freitas, J., Haber, N., and Yamins, D. Active world model learning with progress curiosity. In
10

---

Efficient Skill Discovery via Regret-Aware Optimization
International conference on machine learning, pp. 5306- 5315. PMLR, 2020.
Park, S., Rybkin, O., and Levine, S. METRA: Scalable unsupervised RL with metric-aware abstraction. In The Twelfth International Conference on Learning Represen- tations, 2024c.
Ladosz, P., Weng, L.,, Kim, M., and Oh, H. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22 2022. ISSN 1566-2535.
Parker-Holder, J., Jiang, M., Dennis, M., Samvelyan, M., Fo- erster, J., Grefenstette, E., and Rocktäschel, T. Evolving curricula with regret-based environment design. In Inter national Conference on Machine Learning, pp. 17473- 17498. PMLR, 2022.
Laskin, M., Liu, H., Peng, X. B., Yarats, D., Rajeswaran, A., and Abbeel, P. Cie: Contrastive intrinsic control for unsupervised skill discovery, 2022.
Lee, S.-H. and Sco, S.-W. Unsupervised skill discovery for learning shared structures across changing environments. In International Conference Machine Learning, pp. 19185-19199. PMLR, 2023.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity driven exploration by self-supervised predic- tion. In Precup. D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2778-2787. PMLR, 06-11 Aug 2017.
Li, K., Liu, Y., Ao, X., and He, Q Revisiting graph adversar- ial attack and defense from a data distribution perspective. In The Eleventh International Conference Learning Representations, 2023.
Pong, V. H., Dalal, M., Lin, S., Nair, A., Bahl, S., and Levine, S. Skew-fit: state-covering self self-supervised rein- forcement learning. In Proceedings of the 37th Interna- tional Conference on Machine Learning, pp. 7783-7792, 2020.
Li, K., Chen, Y., Liu, Y., Wang. J., He, Q. Cheng, M., and Ao, X. Boosting the adversarial robustness of graph neural networks: An ood perspective. In The Twelfth International Conference Learning Representations, 2024.
Portelas, R., Colas, C., Weng, L., Hofmann, K., and
Oudeyer, P.-Y. Automatic curriculum learning for deep rl: A short survey. arXiv preprint arXiv:2003. 046664 2020.
Liu, G., Tang, M., and Eysenbach, B. single goal is all you need: Skills and exploration emerge from contrastive rl without rewards, demonstrations, or subgoals. arXiv preprint arXiv:2408.085 055044 2024.
Rutherford, A., Beukman, M., Willi, T., Lacerda, B., Hawes, N., and Foerster, J. N. No regrets: Investigating and improving regret approximations for curriculum discov- ery. In The eiirth- Annual Conference on Neural Information Processing Systems, 2024.
Mazzaglia, P., Verbelen, T., Dhoedt, B., Lacoste, A., and Rajeswar, S. Choreographer: Learning and adapting skills in imagination. In 3rd Offtine RL Workshop: Offline RL "Launchpad", 2022.
Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prior- itized experience replay. In Bengio, Y. and LeCun, Y. (eds.), 4th International Conference on Learning Repre- sentations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016
Nasiriany, S.. Pong, V.. Lin, S., and Levine, S. Planning with goal-conditioned policies. Advances in neural infor- mation processing systems, 32, 2019.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms.
Ozair, S., Lynch, C., Bengio, Y., Van den Oord, A., Levine, S., and Sermanet, P. Wasserstein dependency measure for representation learning. Advances in Neural Information Processing Systems, 32, 2019.
arXiv preprint 1707 2017.
Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
Park. S., Choi, J., Kim, J., Lee, H., and Kim, G. Lipschitz- constrained unsupervised skill discovery. In International Conference on Leurning Representations, 2022.
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. In International conference on machine learning, pp. 387- 395. Pmlr, 2014.
Park, S., Ghosh, D., Eysenbach, B., and Levine, S. Hiql: Offline goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36, 2024a.
Silver, D.,, Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Grae- pel, T., Lillicrap, T., Simonyan, K., and Hassabis, D. A general reinforcement learning algorithm that masters
Park, S., Kreiman, T., and Levine, S. Foundation policies with hilbert representations. In Forty-first International Conference on Machine Learning, 2024b.

---

Efficient Skill Discovery via Regret-Aware Optimization
chess, shogi, and go through self-play. Science, 362 (4419):1440-1144 2018
Xu, Y., Dai, L., Singh, U., Zhang, K., and Tu, Z. Neu- ral program synthesis by self-learning. arXiv preprint arXiv: arXiv:1910.060 1910.05865, 2019.
Sukhbaatar, S., Lin, Z., Kostrikov, I., Synnaeve, G., Szlam, A., and Fergus, R. Intrinsic motivation and automatic curricula via asymmetric self-play. In 6th International Conference on Learning Representations, ICLR 2018, 2018.
Yang, R., Bai, C., Guo, H., Li, S., Zhao, B., Wang, Z., Liu, P., and Li, X. Behavior contrastive learning for unsupervised skill discovery. In International conference machine learning, pp. 39183-39208 PMLR, 2023.
Sun, H., Li, Z., Liu, X., Zhou, B., and Lin, D. Policy continuation with hindsight inverse dynamics. Advances in Neural Information Processing Systems, 32, 2019.
Zhang, H.,, Sun, Y., Guo, W., Liu, Y., Lu, H., Lin, X., and Xiong, H. Interactive interior design recommendation via coarse-to-fine multimodal reinforcement learning. In Proceedings of the st ACM International Conference on Multimedia, pp. 6472-666666 6480, 2023.
Sun, Q.. Zhang, L., Yu, H., Zhang. W., Mei, Y., and Xiong, H. Hierarchical reinforcement learning for dynamic au- tonomous vehicle navigation intelligent intersections. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 4852-4861, 2023.
Zheng, Z., Sun, Y., Song, X., Zhu, H., and Xiong. H. Gen- erative learning plan recommendation for employees: A performance-awart reinforcement learning approach. In Proceedings of the 17th ACM Conference on Recom- mender Systems, pp. 443-454, 2023.
Sun, Y., Ji, Y., Zhu, H., Zhuang, F., He, Q., and Xiong, H. Market- long-term, job skill recommendation with explainable deep reinforcement learning, ACM Transac- tions on Information Systems, 43(2):1-35, 2025.
Sutton, R. S. Reinforcement learning: An introduction. A Bradford Book, 2018
Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, Erez, T., Lillicrap, T., Heess, N., and Tassa, Y. dm-control: Software and tasks for continuous control. Software Impacts, 6: 00002, 2020. ISSN 2665-9638.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575 (7782)3350-354, 2019.
Wang, T., Torralba, A., Isola, P., and Zhang, A. Opti- mal goal-reaching reinforcement learning via quasimetric learning. In International Conference on Machine Learn- ing, pp. 36411-6311 1-34400. PMLR, 2023.
Wang, Z., Hu, J., Chuck, C., Chen, S., Martin-Martin, R., Zhang, A., Niekum, S., and Stone, P. SkiLD: Unsuper- vised skill discovery guided by factor interactions. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Wang, Z., Yang, E., Shen, L., and Huang, H. A comprehen- sive survey of forgetting in deep learning beyond contin- ual learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024b.
12

---

Efficient Skill Discovery via Regret-Aware Optimization
A. Additional Related Work
A.1. Other technique lines for unsupervised skill discovery
Recent works (Bac et al., 2024; Bai et al., 2024; Kim et al., 2024) attempt to improve the skill diversity by introducing additional intrinsic rewards for exploration such as prediction error models (Burda et al., 2019; Pathak et al., 2017) or K-nearest neighbors, facing limitations in high-dimensional spaces. In contrast, we argue that efficiency can be improved simply by adaptively sampling strategies according the policy regret. Recent works also explored learning skills using successor features in dynamic environments (Lee & Seo, 2023) or proposed triple-hierarchical decision frameworks for downstream tasks (Wang et al., 2024a). However, they did not fully explore the skills within an environment and overlooked the importance of skill policies for efficiency improvement.
A.2. Goal-conditional RL with temporal representation
coaal-condiional reinforcement learning (GCRL) has flourished in recent years due to its potential for multitask general- ization with sparse rewards (Andrychowicz et al., 2017; Kaelbling, 1993; Ding et al., 2019; Ghosh et al., 2019; Nasiriany et al., 2019; Sun et al., 2019; Eysenbach et al., 2020; Liu et al., 2024). This area is also important because it can enhance the interpretability of learned models (Ji et al., 2025). An approach to providing continuous and dense intrinsic rewards to leverage the temporal corelation between goals and states. For instance, QRL (Wang et al.. 2023) introduces the use of quasimetric reconstruct the distance in temporal representation space. HIQI. (Park et al., 2024a) acquires feasible subgoals by utilizing the vectors between goals and states in the temporal representation space, which benefits long-horizon tasks. Later, HILPS (Park et al., 2024b) proposed an unsupervised offline training method, based on learning the temporal representation in a Hilbert space first. Although these approaches perform well in offline GCRL settings, their effectiveness in online unsupervised RL scenarios is limited. Instead, we build upon problem formulation in GCRL and incorporate the concept of temporal representation learning for the discovery of unsupervised skills, as suggested in previous work (Park al., 2024c).
ji025coopreehenivee
B. Environment Description
E
Figure 8. Examples of the environments
antmaze- hetnmaz---mmdiummdddee-------- and antmaze- large-diverse-vO, as illustrated in Figure 13. Specifically, the state dimension of Ant is 29, Maze2d is 4, and Antmaze also 29. In particular, the Antmaze-Large map resembles that Maze2d- Large in map shape but differs scale, which explains the varying FD values in Table 1. Many studies (Bagaria al., 2021; Campos al., 2020; Kim al., 2024; Bac et al., 2024) on USD utilize the maze environment, to demonstrate the diversity of skills and their effectiveness in solving downstream tasks. Our experiments are similarly designed to validate the efficacy of our method based on these two criteria. While these methods have achieved diversity in the maze environment, previously noted, they exhibit certain limitations. Some methods succeed only in low-dimensional spaces, such as Maze2d, but fail in environments like Ant or Antmaze due to the lack of temporal representation learning. Others rely on additional unsupervised exploration techniques, such as RND, which introduce instability and expose weaknesses associated with model prediction errors, such as the inability to handle the white noise scenarios, which contains diverse but meaningless states.
Our research encompasses four environments: Ant, Sen--ddumaze--en-e
13

---

Efficient Skill Discovery via Regret-Aware Optimization
C. Experimental Setup
C.I. Parameter Setting
HYPERPARAMETER
VALUE
MAX PATH LENGTH
300
TRAJECTORY BATCH SIZE
16
MAX BUFFER SIZE
3000000
OPTION DIM
LEARNING RATE (common)
0.0001
LEARNING RATE (6)
0.001
MODEL LAYER
MODEL DIM
1024
DISCOUNT FACTOR
0.99
BATCH SIZE
1024
DUAL SLACK
0.001
01 (To2)
5
02 (TO,) )
MAX SIZE (P₂)
15
STEPS IN EACH STAGE
50
This shows the parameters in Maze2d-large environment. To ensure faireess, the parameters shared between the baseline and our method were kept consistent. All experiments were conducted single NVIDIA A100 GPU with PyTorch 2.0.1.
C.2. Explanation for Learning Process
Our method offers aligns closely with the foundational theories of the Learning Process. These theories suggest that the Learning Process (Kaplan & Oudeyer, 2007) can serve an essential guide for exploration, effectively replacing random exploration strategies. We base our approach on the following two assumptions: 1) The difficulty of skills varies depending on the environment. 2) Skills are interrelated, with basic skills learned before advanced ones. By analyzing successive Learning Progress (Gottlieb & Oudeyer, 2018; Kim et al., 2020) through the lens of converged proficiency, our method addresses these two assumptions: 1) sample fewer easy skills that have already shown no further progress; and 2) sample fewer advanced skills, which also show no progress until the agent masters the basic ones".
C.3. Experiment Techniques
For the skill population, we employ three techniques in practical implementation:
C.3.1. CENTERING REPRESENTATIONS
To ensure that all directions in the representation space fully utilized, introduce constraint center the representation of the initial state so. This is achieved by enforcing Φo Ö. Based on Eq. 3.2, add additional constraint as follows:
Co(so) -1)((s))11-
C.3.2. HEURISTIC POPULATION UPDATE
We adopt a heuristic method to update the skill population efficiently. Specifically, if the current population size |P reaches its maximum limit I, we remove the component with the lowest regret value. This approach allows for more efficient utilization of the population and reduces the probability of learning redundant skills.
14

---

Efficient Skill Discovery via Regret-Aware Optimization
C.3.3. GREEDY ADJUSTMENT OF POPULATION DISTRIBUTION
During each learning stage Part adjust the distribution of the skill population Ps using greedy method. In our implementation, P. is modeled as Gaussian Mixture Model (GMM), where each component corresponds to Gaussian distribution TT. Adjusting the sampling probability of cach component is feasible. We leverage the Osen values, which is formulated during the computation of the distance regularizer dæ 17. We consider the final state $ in new collected trajectories. Then, the representation of 8y is denoted as Φs,- If d(8y), compared to Øseen. is closer to a specific component in Ps, it indicates that the corresponding distribution is being effectively learned. Consequently, we increase the sampling probability of that component. The GMM's component probabilities are recalibrated using a softmax function:
max
(log( Te, (da))))
max Batmm
(Tog( (0x,))
P(i) exp(w:;
E, exp(ux;)
where wi is the weight associated with the i-th component, Buew.t inj and Butaaj represent the trajectories collected at two successive steps. To ensure fairness and diversity, we enforce a minimum probability threshold for each component, guaranteeing that every component has a chance of being sampled.
D. Visualization of Zero-shot Evaluation
Figure 9. Goals for zero-shot evaluation.
Figure 10. Visualization of our zero-shot performance.
Figure shows the goals we used for zhro evaluation Maze2d-large environment, which are evenly distributed across the reachable areas of the map. Figure 10 shows the trajectories of our method's 's performance on the map. We consider the agent have reached the goal if the L.2 distance less than
15

---

Efficient Skill Discovery via Regret-Aware Optimization
E. Hyperparameter Tuning
70
60
CoverCoords
40
20
10
Figure 11. Hyperparameter Tuning of on. 02 and maximum length I.
We aim to analyze the impact of the hyperparameters an, 022 and the maximum length on the performance of our model. In Figure 11, each cluster of results corresponds to the adjustment of a single parameter, while keeping the other parameters fixed.
The parameter a,, which controls the diversity of Pa. When a too large, the policy Ta, tends to ignore the regret values and prioritize diversity, leading behavior similar to uniform sampling. In this case, the value of regret are not utilized effectively. Conversely, when Oj is too small, the policy may repeatedly sample skills from similar regions, resulting in redundant skill exploration.
The parameter which prevents P= from venturing too far from the seen regions. When is set to O. we observe that To frequently selects unseen z values for learning. This is due to the regret estimation function inaccurately assigning high regret values to unseen 2, which the agent is not yet equipped to handle effectively. On the other hand, if a2 is too large, the policy tends to sample already explored points excessively, reducing the probability of exploration, and thereby hindering the model's efficiency.
Lastly, the maximum length of P.. Increasing I provides more steps for skill learning and helps prevent forgetting, thereby improving performance. However, further increasing yields diminishing returns. Based on our observations, we choose 1 = 15 as a balanced value that ensures effective learning.
F. Kitchen Environment Evaluation
F.1. Motivation for Kitchen Selection
To further validate our method's effectiveness, we conducted additional experiments in the Kitchen environment, known to present significant challenge in skill discovery. Figure 5 illustrates the performance comparison between METRA and proposed method in this challenging scenario. Notably, METRA struggles in this environment due skill asymmetry, an issue specifically targeted by our approach. To test our mehtod's 's ability on pixel-based environment, we extended our experimental assessment to the Kitchen environment, characterized by pixel-based observations and complex robotic manipulation tasks, as previously explored by METRA.
F.2. Experimental Setup
Consistent with METRA's established methodology, performance evaluation was based on the completion of a set of 7 predefined tasks. We report the mean and standard deviation (mean + std) across three random seeds 0, 2, 4. Experiments were conducted at varying training stages (100k, 200k, 300k, and 400k steps) and with two different skill dimensions (24 and 32).
16

---

Efficient Skill Discovery via Regret-Aware Optimization
Figure 12, Example of Kitchen Environment.
F.3. Results at different Skill Dimensions
Results at Skill Dimension = 24: Our method significantly outperformed METRA, particularly at later training stages. At 400k steps, our method achieved 5.08+0.45 completed tasks compared to METRA's 3.9440.36, showing clear improvement of +1.114 tasks.
Model
100k
200k
300k
400k
METRA
2.72+4.19
3.20+0.27
3.9320.67
3.94+0.36
Ours
2.41#0.02
3.61
4.29+0.11
5.08+0.45
Table 2. Kitchen Environment Performance (Skill Dimension=24)
Results at Skill Dimension 32: At higher skill dimension, approach continued to surpass METRA. At 400k steps, we obtained 4.55+0.35 completed tasks against METRA's 3.99+0.94, marking improvement of +0.56 tasks.
Model
100k
200k
300k
400k
METRA
3.21+0.71
3.66+0.69
3.83#0.90
3.99+0.94
Ours
3.51+0.23
3.9140.43
4.34+0.39
4.55+0.35
Table 3. Kitchen Environment Performance (Skill Dimension=32)
Kitchen Environment Perforancc (Skill Dimension=22)
Kitchen Environment Performance (Skill Dimension-32)
HETRA
5,0
METRA
4--
1204
2004
900x
100k 200k
300k
400K
Trining Steps
Stops
Figure 13. Examples of the environments
Overall, our method consistently demonstrated superior sample efficiency and final performance compared to METRA. Particularly at a skill dimension of 32, the performance gains were increasingly pronounced as training progressed (e.g., at 300k steps was +0.51 compared to +0.36 at dimension 24). These improvements are clearly visible in performance curve visualizations, underscoring our method's robustness and scalability.
17


Reviews:
[{"review_id": "rez5UYVldK", "reviewer": "Reviewer_uCJY", "summary": "This paper is on unsupervised skill discovery within the context of Markov decision processes. It builds on a collection of earlier papers that aim to learn diverse and distinguishable behaviours, for example, DIAYN by Eysenbach et al., (ICLR 2019). The contribution of this paper is a new algorithm, which the authors call Regret-Aware Skill Discovery (RSD). Conceptually, their main idea is to prioritise the exploration of skills the authors call \"under-converged\" rather than indiscriminately exploring the whole skill space. The authors present an experimental evaluation of their approach.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Motivation of \"Policy Strength\"\n1. **Definition:**\n\n    The term *policy strength*, derived from \"the strength of the policy\", refers to the **capability** of a policy to accomplish its designated objective—corresponding to **a specific skill** in our context.\n2. **Why we use \"policy strength\":**\n\n    In this work, we adopt the term *policy strength* to concisely describe the **capability of the policy under skill-conditioned training**. More specifically, we are interested in how well the policy can be optimized given a particular skill, and whether it has **converged**, or **how far it is from convergence**. This perspective helps us assess whether a skill has been sufficiently explored or still requires further training attention.\n3. **Motivation and necessity:**\n\n    Initially, we considered using phrases like *\"the performance of the policy\"*, but we were concerned that *performance* might be misinterpreted as referring to the cumulative reward. This can be misleading in unsupervised skill discovery, where intrinsic rewards are not always directly comparable between different skills. In contrast, *policy strength* focuses on the **optimization dynamics** and **convergence behavior** of the policy, thus avoiding ambiguity tied to reward metrics.\n    \nWe acknowledge the term may be unconventional and will revise the introduction to clearly define it.\n\n---\n### Q2: More Experimental Evaluation\nThank you for encouraging broader validation.\n1. **Choice of Maze Environment**\n\n    We extend METRA’s evaluation by including **Maze**, a more challenging setup for skill discovery [1]. As seen in *Figure 5*, METRA underperforms here due to the **skill-asymmetry issue**, which our method specifically targets.Additionally, we assess **zero-shot performance** on downstream tasks, following standard protocol.\n2. **Evaluation on the Kitchen Environment**\n\n    In response to your concern, we additionally include experimental results on the **Kitchen**:\n    - **Kitchen Environment**:\n        Kitchen involves **pixel-based observations** and more complex robot-**manipulation tasks**, which are evaluated in METRA.\n    - **Why Kitchen**:\n        As noted in [this issue](https://github.com/seohongpark/METRA/issues/9#issuecomment-2565876012), pixel-based tasks are indeed **computationally demanding**. We chose Kitchen for its practical challenge and feasibility under limited time.\n    - **Experimental Setup**:\n        Following METRA’s setup, we measure the **number of completed tasks** (7 total) using **mean ± std** over three seeds `{0, 2, 4}`. We evaluate across different training steps and skill dimensions (`dim=24, 32`).\n        #### Performance @ Skill Dim = 24\n        |**Model**| **Step = 100k**| **Step=200k** | **Step = 300k** | **Step = 400k** | **Δ@400k** |\n        |:-:|:-:|:-:|:-:|:-:|:-:|\n        |METRA|**2.72±0.19**|3.20±0.27|3.93±0.67|3.94±0.36||\n        |Ours|2.41±0.02|**3.61±0.02**|**4.29±0.11**|**5.08±0.45**|+1.14|\n        #### Performance @ Skill Dim = 32\n        |**Model** | **Step = 100k** | **Step = 200k** | **Step = 300k** | **Step = 400k** | **Δ@400k** |\n        |:-:|:-:|:-:|:-:|:-:|:-:|\n        |METRA|3.21±0.71|3.66±0.69|3.83±0.90|3.99±0.94||\n        |Ours|**3.51±0.23**|**3.91±0.43**|**4.34±0.39**|**4.55±0.35**|+0.56|\n    - **Results Summary**:\n        Our method outperforms METRA in both **sample efficiency** and **final performance**. At **skill dim = 32**, our method exhibits greater gains with training progress (e.g., Δ@300k = +0.51 vs. +0.36 at dim=24). This trend is even clearer when visualized in the curves.\n---\n### Q3: Baseline Selection\n1. **Baseline Selection**\n\n    Our method is built upon METRA’s framework. Our primary contribution lies in **enhancing the sampling efficiency** by incorporating ideas from **Prioritized Level Replay (PLR)**, enabling more effective skill discovery within the METRA setup.\n2. **Why we do not compare against intrinsic reward-based methods**\n\n    As discussed in **Appendix A.1**, we chose not to include other intrinsic reward methods for the following reasons:\n    - **RND-based** methods improve exploration but are **not tailored for skill discovery**.\n    - **KNN-based** methods are limited in high-dimensional representation space.\n\n    Crucially, our focus is different: we optimize **skill distribution** during training using a regret-driven min-max formulation, unlike the uniform sampling in METRA and related work.\n    \nWe hope this explains our baseline decisions.\n\n---\n### Q4: Define $\\phi$ on Line 131\nWe will add the following definition to improve clarity:\n> A representation function $ \\phi: \\mathcal{S} \\rightarrow \\mathbb{R}^d $, which maps states into a skill-aligned latent space that shares the same dimensionality as the skill variable $ z \\in Z $.\n\n---\n[1] Campos, Víctor, et al. \"Explore, discover and learn: Unsupervised discovery of state-covering skills.\" *International conference on machine learning*. PMLR, 2020.", "claims_and_evidence": "I find that this paper is not clearly written. It is difficult to pinpoint exactly what the claims are. One claim seems to be that \"skill discovery should be grounded by the policy strength\" (page 1). I am not fully clear on what is meant by policy strength. In addition, given such a broad claim about skill discovery, the empirical evaluation should present evidence from a broad collection of skill discovery algorithms, in a broad range of problem domains; this is not the case in the paper. \n\nA second claim is that the proposed method \"significantly improves the learning efficiency and increases skill diversity in complex environments\" and \"achieves up to a 16% improvement in zero-shot evaluation compared to baselines\". The support for these claims comes from a rather narrow experimental evaluation in terms of baseline algorithms considered and domains tested (details below) compared to the general practice in the literature (see, for example, METRA by Park et al, ICLR 2024).", "methods_and_evaluation_criteria": "The experiments are quite narrow in their scope. The authors can provide stronger support for their claims by evaluating their approach in a border range of domains and by comparing to a broader set of baseline algorithms. Currently, all experiments in the paper have been performed in the ant environment. In terms of algorithms, the baselines tested in the paper are only those that are very closely related to the proposed approach. If the claim is a new approach that \"significantly improves the learning efficiency\", a broader set of algorithms should be tested as baselines, for example, those that use intrinsic rewards.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The primary issue with the experimental design is its narrow scope.", "relation_to_broader_scientific_literature": "The paper introduces a new approach within a particular line of research on unsupervised skill discovery. I am not able to judge its significance based on the relatively narrow analysis in the paper.", "essential_references_not_discussed": "Nothing particular.", "other_strengths_and_weaknesses": "The paper may have a useful contribution to make but requires a more clear statement of its claims and a more rigorous evaluation of those claims.", "other_comments_or_suggestions": "Define Phi on line 131.", "questions_for_authors": "Nothing particular.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "OOvT7icPQQ", "reviewer": "Reviewer_42wC", "summary": "The paper presents Regret-aware Skill Discovery (RSD), a novel approach to unsupervised skill discovery in reinforcement learning. The authors conceptualize skill discovery as a min-max adversarial game between skill generation and policy learning. Their key insight is that skill discovery should be guided by policy strength convergence - focusing exploration on skills with weak, unconverged strength while reducing exploration for skills with already converged strength. To implement this, they use a regret measure to quantify policy strength convergence and employ a learnable skill generator within a population-based framework.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. Below we address each point in detail.\n\n### **Q1. Statistical Reporting**\n\nWe appreciate the reviewer’s emphasis on statistical rigor.  \nAll experiments were conducted using **five independent random seeds**: `{0, 2, 4, 8, 16}`.\n\n- In **Figure 3**, the **bold lines** indicate the **mean**, and the **shaded areas** represent the **standard deviation** across these runs.\n- In **Table 1**, we originally reported the **mean performance** only. In the revised version, we will **add uncertainty measures** to each entry. For example, in the *Maze2d-large* environment:\n\n```\nRSD (ours):     FD↓ = 0.535 ± 0.152, AR↑ = 0.811 ± 0.046\nMETRA-d:        FD↓ = 0.658 ± 0.154, AR↑ = 0.782 ± 0.043\n```\nThese results demonstrate that RSD achieves competitive performance with **comparable variance**, supporting the robustness of our method.\n\n### **Q2. Applicability Beyond METRA**\n\nThank you for the insightful question regarding generality. We certainlly plan to explore this framework with other models (for example in VLA) in future work.\n\nOur claim that RSD is a **general framework** refers to the **min-max optimization formulation in Equation 8**, which is **algorithm-agnostic** and, in principle, applicable to a wide range of **unsupervised skill discovery (USD)** methods.\n\nIn this work, we chose to **implement RSD on top of METRA** rather than LSD. This is because **METRA**, an improved version of LSD, provides **key theoretical properties** (Equation 3) that we leverage. In Section 3.2 (Equations 2 and 13), we introduce **targeted modifications** to enhance performance in skill-asymmetric environments.\n\n### **Q3. Epochs in Figure 6**\n\nWe appreciate the reviewer’s comment regarding Figure 6.\n\nThe relationship between **epochs and timesteps** is reflected in **Algorithm 1 (lines 2 and 5)**:\n\n```\ntimesteps = epoch × num_trajectories × max_length\n```\n\nAs described in **Appendix C.1**, we use:\n\n- `num_trajectories = 16`\n- `max_length = 300`\n\nSo the approximate mapping is:\n\n```\n4000 epochs   → 2e7 timesteps  \n8000 epochs   → 4e7 timesteps  \n14000 epochs  → 7e7 timesteps  \n18000 epochs  → 9e7 timesteps\n```\nIn the revised version, we will **redraw Figure 6** with **explicit timestep annotations** and **improve color contrast**, as suggested, to improve readability.\n\n### **Q4. Timestep Scales in Figures 3 and 7**\n\nThank you for noting the difference in timestep scales.\n\nThis design choice is **intentional**, as the two figures serve **different purposes**:\n\n- **Figure 3** highlights **early-phase learning efficiency** (shown up to **5e6** timesteps), with focus on the **0–2e6** range.\n- **Figure 7** illustrates **long-term convergence and stability**, plotted up to **1e7** timesteps.\n\nTo avoid confusion, we will **explicitly clarify this** in the caption of Figure 7 in the final version.\n\n### **Additional Comments**\n\n#### 1. Ablation Study Referencing\n\nWe agree that the ablation study contains important insights. In the revised version, we will:\n\n- **Explicitly reference Appendix E in the main text**\n- **Summarize key findings** to highlight contributions of individual components\n\n#### 2. Missing AntMaze2D in Appendix D\n\nThank you for catching this. **Appendix D contains visualized results** for experiments Table 1. We will:\n\n- **Clarify this reference in the main text**\n- Add **explanatory notes in Appendix D** to improve readability\n\n#### 3. Zero-shot Evaluation Concern\n\nWe appreciate your thoughtful concern. The goal of our zero-shot experiment is to evaluate **practical utility**.  \nIndeed, **RSD achieves superior zero-shot performance partly due to higher coverage** — which we see as a **desired property**.  \nThe improved zero-shot success rate is a **direct consequence of more effective skill discovery**, which we believe substantiates our central claim.\n\n### **Typos and Presentation Issues**\n\nWe thank the reviewer for the attention to detail. All identified **typos and presentation issues have been carefully corrected** in the revised version to improve clarity and presentation quality. \n\nOnce again, we sincerely appreciate the reviewer’s time and valuable, constructive feedback. We believe it has significantly enhanced the clarity, rigor, and overall quality of our paper.", "claims_and_evidence": "The main claims about RSD outperforming baselines in terms of efficiency and diversity are generally supported by the experimental results presented in the paper. \nThe authors show performance comparisons across multiple environments (Ant, Maze2d-large, Antmaze-medium, Antmaze-large), demonstrating improved coverage metrics and zero-shot performance. \n\nHowever, the evidence would be stronger with clearer reporting of statistical significance - the paper lacks information about the number of independent seeds for experiments and doesn't provide error bars or uncertainty measurements for the reported results in Table 1.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are reasonable for the problem of unsupervised skill discovery. The authors use appropriate metrics like state coverage (CoverCoords) to measure skill diversity and zero-shot performance to assess skill utility. \n\nHowever, the CoverCoords metric could be more interpretable if reported as percentages of achievable coordinates on the map rather than absolute values. \n\nThe zero-shot experiment look a bit artifical to me: it shows improved results for RSD, but given that RSD achieves higher coverage (CoverCoords), it naturally has better chances of reaching goals.", "theoretical_claims": "Not applicable (there are no theoretical claims)", "experimental_designs_or_analyses": "I checked the experimental design for comparing RSD against baseline methods. \nWhile the experimental setup appears sound overall, there are several omissions:\n\n1. The number of independent seeds run for each experiment is not specified\n2. Statistical significance of results is not reported (no error bars in Figure 3 or uncertainty measures in Table 1)\n3. There are inconsistencies in the reported timescales (Figure 3 shows results for 5e6 timesteps for Maze2d-large, while Figure 7 shows results for 1e7 timesteps for the same environment)\n4. The AntMaze2D results mentioned in Appendix D appear to be missing.", "relation_to_broader_scientific_literature": "The paper positions RSD appropriately within the unsupervised skill discovery literature, building upon temporal representation learning approaches like METRA. \nThe authors clearly acknowledge and compare to previous work on mutual information-based skill discovery (DIAYN, DADS, LSD, METRA). \nThey also explain how their approach differs by focusing on the relationship between skills and policy learning strength. \nThe regret-aware optimization bears some resemblance to concepts from Learning Process, which the authors acknowledge.", "essential_references_not_discussed": "To the best of my knowledge, all the relevant related works are cited/discussed in the paper.", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper is well-organized and clearly written.\n- The algorithm is well-illustrated with helpful figures that explain the approach.\n- The experiments demonstrate meaningful improvements over existing methods.\n- The bounded representation learning approach appears to be a useful contribution.\n\nWeaknesses:\n\n- The authors claim RSD is a framework, but it's only implemented on top of METRA. To substantiate this claim, they should demonstrate applicability to other base algorithms like LSD.\n- Ablation studies are relegated to the appendix (Appendix E) without being properly referenced in the main text.", "other_comments_or_suggestions": "Typos and presentation issues:\n\n- Algorithm 1, line 10: should be $\\pi_{\\theta_1}$ instead of $\\pi_{\\theta_2}$\n- Line 216: extra \"the\"\n- Line 236, 2nd column: missing space before \"Therefore\"\n- Figure 6: \"Sapce\" instead of \"Space\"\n- Line 328: \"DIYAN\" instead of \"DIAYN\"\n- Line 312, 2nd column: \"the both\" is awkward phrasing\n- Line 380: \"RSDlearns\" should have a space\n- Line 345, 2nd column: \"Maeze2d-large\" instead of \"Maze2d-large\"\n- The colors in Figure 6 make it difficult to read", "questions_for_authors": "1. How many independent seeds were run for each experiment? What do the bold lines and shaded areas represent on the plots in Figure 3? What do the numbers in Table 1 represent (averages, medians)? What are the uncertainties in these measurements?\n2. You claim RSD is a framework applicable beyond METRA. Have you tested applying it to other skill discovery algorithms such as LSD? If so, what were the results?\n3. In Figure 6, how many timesteps does an \"epoch\" represent?\n4. Why are the results in Figure 7 presented for 1e7 timesteps, whereas for the same environment in Figure 3, the results are presented for 5e6 timesteps?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O5Oj3eVdCY", "reviewer": "Reviewer_3FHL", "summary": "This paper propose a new unsupervised skill discovery method, which use regret to guide the skill sampling and skill policy learning. The regret is computed by the estimation error of value function in learning. Based on that, sampling strategies is not a parameters-constant distributions as previous methods. the paper claims the skill policy can prioritizes the exploration of under-converged skills during policy learning. This method improves the learning efficiency and increases skill diversity in simulation environment.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback.\n### Q1: Unique Coordinates Metric\n1. **Origin of the Metric**: \nThe metric is inspired by the *Policy State Coverage* measure introduced in the METRA paper (Section 5.3), where it was used to evaluate the **spatial coverage** of skill policies—referred to as x-y Coverage in the Maze environment. In our work, we rename it to *Unique Coordinates* to improve specificity, as noted in Section 4.1 (line 312).\n2. **Definition**: \nThe metric essentially counts the number of unique 2D coordinates visited by the agent during skill executions. We extract the x and y coordinates from states (hence the name Coordinates), then count the number of distinct positions (hence using Unique).\n3. **Computation**:\nFor each algorithm:\n   - Extract trajectory x-y states for each skill;\n   - Discretize the x-y values; (e.g., rounded to integers);\n   - Count the number of unique coordinate pairs across all rollouts.\n4. **Significance**: \nThis metric reflects how diversely the skills explore the state space. As supported in prior work, reaching distinct regions of the environment implies higher skill diversity(Figure 4). \n\nWe will include this clarification in the revised version.\n\n---\n### Q2: Additional Experiments\n1. **Kitchen**: It uses **pixel-based observations** and involves **manipulation** tasks, making it different from Ant-based settings.\n2. **Motivation**: Following ([issue](https://github.com/seohongpark/METRA/issues/9#issuecomment-2565876012)), pixel-based tasks are indeed computationally demanding. We prioritized the Kitchen environment as it is both challenging and practically meaningful under time constraints.\n3. **Experimental Design**: Following METRA’s setup, we report **the number of completed** tasks (7 in total) using  **mean ± std** over three seeds {0,2,4}. \n#### Performance @ Skill Dim = 24\n|**Model**| **Step = 100k**| **Step=200k** | **Step = 300k** | **Step = 400k** | **Δ@400k** |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|METRA|**2.72±0.19**|3.20±0.27|3.93±0.67|3.94±0.36||\n|Ours|2.41±0.02|**3.61±0.02**|**4.29±0.11**|**5.08±0.45**|+1.14|\n#### Performance @ Skill Dim = 32\n|**Model** | **Step = 100k** | **Step = 200k** | **Step = 300k** | **Step = 400k** | **Δ@400k** |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|METRA|3.21±0.71|3.66±0.69|3.83±0.90|3.99±0.94||\n|Ours|**3.51±0.23**|**3.91±0.43**|**4.34±0.39**|**4.55±0.35**|+0.56|\n\nOur method shows both **higher sample efficiency** and **stronger final performance**. At **dim = 32**, our method exhibits **faster** efficiency gains (e.g., Δ@300k = **+0.51**) compared to **dim = 24** (Δ@300k = **+0.36**), suggesting that our efficient approach benefits more from a larger latent space. This trend is even clearer when visualized in the learning curves.\n\n---\n### Q3: Hyperparameter Sensitivity and Stability\n1. **Ablation**: We include ablations (Appendix E) for critical hyperparameters, such as the regret window size (`window`) and regularization weights (`alpha_1`, `alpha_2`). \n2. **Tuning**: We argue that hyperparameter tuning in our method is **guided rather than purely search-based**, as they primarily influence skill diversity. For instance, the KL divergence term in Eq. (16) provides a direct and interpretable signal for adjusting `alpha_1`.\n3. **Empirical Settings**:\n   - **Ant**/**Maze**: Used a fixed setting (`alpha_1=5`, `alpha_2=1`, `window=15`) across experiments.\n   - **Kitchen**: Used lighter settings (`alpha_1=1`, `alpha_2=0`, `window=8`) due to inherent diversity.\n\nWhile we also considered adaptive schemes (e.g., Lagrange multipliers), we avoided them to maintain a lightweight design.\n\n---\n### Q4: Related Work\n1. **Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills**  \n   This paper is cited in our Appendix B. It shares similar experimental setups with ours. However, we did not include it as a primary baseline because it focuses on simpler 2D navigation tasks and does not scale easily to high-dimensional settings.\n2. **Behavior Contrastive Learning for Unsupervised Skill Discovery**  \n   Thank you for the recommendation. Despite conceptual similarity, it lacks demonstration in high-dimensional environments. We also tried contrastive losses but observed unstable learning, potentially due to conflicting gradients in METRA. We acknowledge this as a promising direction.\n3. **Choreographer: Learning and Adapting Skills in Imagination**  \n   An insightful suggestion. The key differences are:\n   - It's a **model-based** RL approach, which increases sample efficiency but at a higher computational cost;\n   - It's **exploration-agnostic**, relying on separate exploration strategies (as noted in their Appendix A), whereas our method integrates exploration directly into the skill learning process.\n\nAlthough these works tackle skill discovery, we address different challenges. We will add these references to our revised Related Work section.\n\nWe hope these resolve your concern!", "claims_and_evidence": "The claim of \"incorporating regret awareness into skill discovery can enhance the learning efficiency\" is supported by the experiment in Figure 3, with more higher Unique Coordinate measure during training.", "methods_and_evaluation_criteria": "The regret-awared methods is well fit in unsupervised skill discovery problems.", "theoretical_claims": "This paper mainly focus on emperical results and no major issues of theroerm detected.", "experimental_designs_or_analyses": "The experiments conducted in the DM Control and D4RL environments are well-designed, with statistical significance reported, providing confidence in the results. The visualization experiment is also clear.  \n\nHowever, the experimental settings primarily focus on state-based environments, while the baselines compared like RSD and METRA, include more complex environments with pixel-based observations as input. This discrepancy makes it difficult to definitively conclude that the proposed method outperforms these baselines across a broader range of scenarios. Particularly, the reported sensitivity in skill generation (from Section 3.3) to hyperparameters raises concerns about the robustness of the method. This instability could limit its reliability and generalizability across a broader range of scenarios beyond the state-based environments tested.", "relation_to_broader_scientific_literature": "The paper builds on skill discovery RL methods like compared baselines LSD, RSD, METRA. The \"regret bonus\" is a well-known concept in exploration RL setting.", "essential_references_not_discussed": "Some reference I know are related to unsupervised skill discovery setting but seem not included in this paper: \n\n\n[1] Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills. \nVíctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-i-Nieto, Jordi Torres\n\n[2] Behavior Contrastive Learning for Unsupervised Skill Discovery\nRushuai Yang, Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, Xuelong Li\n\n[3] Choreographer: Learning and Adapting Skills in Imagination\nPietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, Sai Rajeswar", "other_strengths_and_weaknesses": "Strengths: \n1. This paper is well-written and easy to read.\n2. The regret-aware idea for skill discovery is novel.\n\nWeaknesses:  \n1. Please see Experimental Designs Or Analyses above.", "other_comments_or_suggestions": "Consider adding more experiments to strengthen applicability, e.g. pixel-based version of Kitchen as suggested by METRA^[1], or Unsupervised Reinforcement Learning Benchmark^[2].\n\n\n[1] METRA: Scalable Unsupervised RL with Metric-Aware Abstraction. Seohong Park, Oleh Rybkin, Sergey Levine.\n\n[2] URLB: Unsupervised Reinforcement Learning Benchmark. Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, Pieter Abbeel", "questions_for_authors": "1. How to caculate the Unique Coordinates in y-axis of Figure 3? it’s unclear whether the score (presumably represented on the y-axis) shows marginal improvement over the baselines or iterations. Could you clarify the calculation process and whether the trend indicates meaningful improvement? A detailed response would help me assess the significance of the reported results and could influence my evaluation of the method’s effectiveness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nKr0eBav2H", "reviewer": "Reviewer_aHNd", "summary": "This paper proposes regret-aware skill discovery (RSD) for unsupervised skill discovery. RSD is built upon METRA, a previous temporal distance-based skill discovery method. The key idea behind RSD is to use a separate, learned skill sampler policy to sample $z$'s for better exploration (unlike the uniform distribution in METRA). Specifically, they train two policies in an adversarial manner: an action policy $\\pi_{\\theta_1}$ and a skill sampler policy $\\pi_{\\theta_2}$. $\\pi_{\\theta_1}$ maximizes an objective similar to METRA (but with slight modifications to make it compatible with bounded $\\phi$). $\\pi_{\\theta_2}$ maximizes the regret (defined as $V_k - V_{k-1}$) of the action policy to guide the agent into a region where there's room for improvement in skill learning. The authors experimentally show that RSD leads to better state coverage and downstream performance in Ant, Maze2d, and AntMaze domains.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1: Collapsing Concern**\nThank you for raising this insightful question. Our method is specifically designed to prevent skill collapse. As shown in Eq. (15), the skills are maintained within a population $P_z$. We address the diversity concern (i.e., avoiding convergence to a single point) from three perspectives:\n1. **KL divergence constraint (Eq. 16):**  \n   We enforce a divergence between the newly generated skill distribution $\\pi_{\\theta_2}$ and existing skills in $P_z$ using KL divergence. This ensures new skills remain distinct.\n2. **Minimum variance and overlap check:**  \n   The sampler $\\pi_{\\theta_2}$ (Gaussian) is constrained to maintain a minimum std (1e-1). We also check for distributional overlap before adding new skills, preserving diversity.\n3. **Empirical validation (Figure 7):**  \n   Figure 7 shows a consistent decrease in average regret, mainly driven by SAC. As SAC improves, $\\pi_{\\theta_2}$ keeps adapting, avoiding collapse to a single mode.\n\nPlease let us know if further clarification is needed!\n## **Q2: State-dependent Skill $z_{\\text{updated}}$**\nThank you for your careful observation.\n- When $z$ is fixed across time, Eq. (4) tends to force straight-line trajectories. In skill-asymmetric environments, skills often share key states, making this problematic (Figure 5).\n- Allowing $z_{\\text{updated}}$ to vary with $\\phi$ and time $t$ enables distinct **trajectory curves**, even with shared segments, better capturing behavioral nuances.\n\n**Advantages:**  \nImproves **skill diversity** (Figures 5 & 6), better covers complex regions (e.g., maze corners), and outperforms METRA in skill expressiveness.\n\n**Disadvantages:**  \nIncreases **learning complexity**, as seen in Figure 3(a), where RSD is slightly less efficient in simpler environments.\n## **Discussion on Related Work**\nWe appreciate the reviewer highlighting TLDR[Bae et al., 2024], which is cited in Lines 303, 329, and Appendix A.1 (L608). Our method focuses on skill sampling scheme, whereas TLDR relies on RND and KNN-based exploration strategies.\nWe will clarify this in the revised related work section.\n\n## **Weakness Response**\n### 1. Clarity of Policies $\\pi_{\\theta_1}$ and $\\pi_{\\theta_2}$\nWe revised the start of Section 3 using a bullet-point format to clearly distinguish. We believe this change helps readers immediately grasp the roles of each policy.\n### 2. Organization of Section 3\nOur intention was to follow a “global-to-local” structure for clarity. \n-  Base reward is defined earlier in **Preliminaries (Eq. 4)**.\n- We were concerned that Keeping Section 3.1 ahead preserves motivation for why $\\pi_{\\theta_1}$ minimizes and $\\pi_{\\theta_2}$ maximizes regret.\n### 3. Why $\\pi_{\\theta_1}$ Minimizes Regret (Eq. 8)\nGreat point — we clarify:\n- Regret is estimated via Eq. 7.\n- If $\\pi_{\\theta_1}$ has converged for skill $z$, regret (value improvement) is near zero.\n- Thus, minimizing regret implies convergence under that skill.\n\nNote: $\\pi_{\\theta_1}$ is still trained with standard SAC loss (Eq. 13, Alg. 1 line 10).\n### 4. Intuition Behind Eq. (11)\nFor example, if $\\||\\phi(s_T)\\|| \\leq 1$ and we set $\\phi(s_0) = 0$, then by telescoping:\n$ \\|| \\sum_t^{T-1} \\phi(s_{t+1}) - \\phi(s_t) \\|| \\leq 1$. \nTo simplify, we enforce Eq. (11), and one solution is:  $\\||  \\phi(s_{t+1}) - \\phi(s_t) \\|| \\leq 1/T$.\n### 5. Additional Results\nWe added experiments in the **Kitchen environment** (pixel observations, robotic control), following **METRA**’s setup and reporting success out of 7 tasks.\n\nAs shown in the tables below, our method shows both **higher sample efficiency** and **stronger final performance**.\nInterestingly, at **dim = 32**, our method exhibits **faster efficiency gains** (e.g., Δ@300k = **+0.51**) compared to **dim = 24** (Δ@300k = **+0.36**), suggesting that our efficient approach benefits more from a larger latent space. This trend is even clearer when visualized in the learning curves.\n#### Performance @ Skill Dim = 24\n| **Model** | **Step = 100k** | **Step = 200k** | **Step = 300k** | **Step = 400k** | **Δ@400k** |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| *METRA*   | **2.72 ± 0.19**     | 3.20 ± 0.27     | 3.93 ± 0.67     | 3.94 ± 0.36  |  |\n| *Ours*    | 2.41 ± 0.02 | **3.61 ± 0.02** | **4.29 ± 0.11** | **5.08 ± 0.45** | **+1.14**  |\n#### Performance @ Skill Dim = 32\n| **Model** | **Step = 100k** | **Step = 200k** | **Step = 300k** | **Step = 400k** | **Δ@400k** |\n|:-:|:-:|:-:|:-:|:-:|:-:|\n| *METRA*| 3.21 ± 0.71| 3.66 ± 0.69| 3.83 ± 0.90| 3.99 ± 0.94     |    |\n| *Ours*| **3.51 ± 0.23** | **3.91 ± 0.43** | **4.34 ± 0.39** | **4.55 ± 0.35** | **+0.56**  |\n### 6. Grammatical Errors\nWe have addressed the mentioned issues. For the Lipschitz constraint, METRA uses L2 in the paper, though their code supports L1. Line 193 has also been revised for rigor.\n\nThank you for your time and for recognizing our work! Due to space constraints, some explanations have been shortened — please feel free to reach out if any clarification is needed.", "claims_and_evidence": "Their claims made in the abstract and introduction are generally well-supported by empirical evidence.", "methods_and_evaluation_criteria": "Their approach is sensible and the evaluation criteria look reasonable to me.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "They use standard tasks in unsupervised RL (Ant, PointMaze, and AntMaze), which look reasonable to me.", "relation_to_broader_scientific_literature": "This paper tackles unsupervised skill discovery, and the main takeaway (to me) is that a non-uniform skill sampling distribution can lead to better exploration. This is often overlooked in the previous literature, as prior works mostly employ a symmetric, uniform distribution (Gaussian, uniform over a box, etc.). I believe this is a nice insight to the community.", "essential_references_not_discussed": "One particularly related missing work is \"TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations\" by Bae et al. (2024). Their method is also based on METRA and is evaluated mainly on AntMaze. While I don't believe the comparison with this method is necessary in assessing this paper, it'd have been better if the authors had (at least) discussed this work in the related work section.", "other_strengths_and_weaknesses": "### Strengths\n* The concept of using a learned skill sampling distribution seems (relatively) novel to me (at least it has not been well-studied in the literature). The use of regrets seems quite sensible to me as well.\n* Figure 5 is particularly convincing to me. The authors convincingly demonstrate that RSD leads to better diversity and state coverage than METRA by having a non-uniform skill sampler.\n* The paper provides diverse analyses studying different aspects of RSD training.\n\n\n### Weaknesses\n* In my opinion, the main weakness of this paper is writing.\n  * Section 3 does not flow very well. It directly starts with the detailed description of $\\pi_{\\theta_1}$ and $\\pi_{\\theta_2}$ and presents the full algorithm box without motivating or defining the notations. In particular, the \"types\" of the policies are never defined -- I later realized that $\\pi_{\\theta_1}$ outputs actions and $\\pi_{\\theta_2}$ outputs $z$s, which confused me a bit.\n  * Equation (8) is also quite confusing. It is unclear how $\\theta_2$ affects the objective at this point -- it turns out only in Section 3.3 that $P_z$ depends on $\\pi_{\\theta_2}$. Also, why does $\\pi_{\\theta_1}$ minimize this objective? Isn't it also supposed to maximize $V_{\\pi_{\\theta_1}}^k$ (treating $V_{\\pi_{\\theta_1}}^{k-1}$ as a constant)?\n  * I believe Section 3.2 and Section 3.3 should come before Section 3.1 (or at least the paper should be heavily restructured in general) as $Q$ and $V$ depend on the reward function defined in Section 3.2.\n  * Equation (11) needs a better explanation. While I understood this, it might be unclear why \"a well-learned skill trajectory in the bounded space must hold\" the equation below to the general reader.\n  * L193: \"Due to the bounded nature of $\\phi( \\cdot )$, it naturally satisfies the Lipschitz constraint.\": Boundedness and Lipschitzness are separate concepts. For example, the Heaviside step function is bounded but not Lipschitz (and not even continuous).\n* Regarding experiments, the domains are limited to PointMass and Ant. In its current form, it is unclear how RSD scales to pixel-based observations or other types of environments (e.g., whole-body humanoid control, robotic manipulation, etc.).", "other_comments_or_suggestions": "I spotted quite a lot of grammatical errors and inconsistencies throughout the paper. I'd recommend running a spell/grammar checker. To list only a few:\n* L145: \"for agent policy\" -> \"for the agent policy\"\n* L148: \"These improvements ensures\" -> \"These improvements ensure\"\n* L245: \"denotes maximum size of\" -> \"denotes the maximum size of\"\n* L7 of Algorithm 1: Why is $\\langle, \\rangle$ used instead of $(, )$ to denote tuples as in L72?\n* Equation (2) uses $L$ but Equation (3) uses $\\mathcal{L}$. Moreover, this notation is not defined (and it's Lipschitz w.r.t. which metric?).", "questions_for_authors": "* Is there a mechanism to prevent a collapse of $\\pi_{\\theta_2}$? For example, it can collapse to a single point (e.g., $(1, 1, \\cdots, 1)$) to maximize $d_z$, and keeping this collapsed skill sampler policy for a while would maximize $d_\\theta$ as well. If this specific skill leads to high regrets, then (hypothetically) the agent might end up learning only a single skill. Is there a reason why this doesn't happen in general?\n* Unlike METRA, $z_\\mathrm{updated}$ in Equation (10) depends on the current latent state $\\phi(s_t)$. How does this affect skill learning? Can the authors elaborate on the pros and cons of this choice?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper frames unsupervised exploration as a min-max game, and find skills which maximize Regret, as this suggests that the policy has not yet fully converged -- building on the success of regret in other domains such as UED.

The use of these ideas in the unsupervised skill discovery is novel and a strong contribution, and was demonstrated effective in a number of experimental domains against competitive baselines. We would encourage the authors to follow through on the improvements to the related work, writing clarity and exposition of the statistical significance of their results as detailed throughout the rebuttal period discussion.
Decision: Accept (poster)