AMPED: Adaptive Multi-objective Projection for balancing Exploration and skill Diversification
Authors: Geonwoo Cho
Jaemoon Lee
Jaegyun Im
Subi Lee
Jihwan Lee
Sundong Kim
Venue: ICLR 2026 Poster (2026)
Keywords: unsupervised reinforcement learning
skill discovery
self-supervised learning
multi-joint robot locomotion
Primary area: reinforcement learning
TL;DR: AMPED proposes a novel framework for skill‐based reinforcement learning that simultaneously maximizes state coverage and skill diversity through several carefully designed components.

Abstract: Skill-based reinforcement learning (SBRL) enables rapid adaptation in environments with sparse rewards by pretraining a skill-conditioned policy. Effective skill learning requires jointly maximizing both exploration and skill diversity. However, existing methods often face challenges in simultaneously optimizing for these two conflicting objectives. In this work, we propose a new method, Adaptive Multi-objective Projection for balancing Exploration and skill Diversification (AMPED), which explicitly addresses both: during pre-training, a gradient-surgery projection balances the exploration and diversity gradients, and during fine-tuning, a skill selector exploits the learned diversity by choosing skills suited to downstream tasks. Our approach achieves performance that surpasses SBRL baselines across various benchmarks. Through an extensive ablation study, we identify the role of each component and demonstrate that each element in AMPED is contributing to performance. We further provide theoretical evidence that, with a greedy skill selector, greater skill diversity reduces fine-tuning sample complexity. These results highlight the importance of explicitly harmonizing exploration and diversity and demonstrate the effectiveness of AMPED in enabling robust and generalizable skill learning. https://geonwoo.me/amped/

Published as a conference paper at ICLR 2026
AMPED: ADAPTIVE MULTI-OBIECTIVE PROJECTION FOR BALANCING EXPLORATION AND SKILL DIVERSIFI-
CATION
Geonwoo Cho Jaemoon Leea. Jacgyun Im Subi Leet Jihwan Leet Sundong Kim!
'Gwangju Institute of Science and Technology SSouul National University (gwcho.public, dlwoans0001, jaegyun jaegyun. public, soobi1234, ghdd12650, sdkim0211 1 |@ @om
ABSTRACT
Skill-based reinforcement learning (SBRL) enables rapid adaptation in environ- ments with sparse rewards by pretraining -kili-conditioned policy. Effective skill learning requires jointly maximizing both exploration and skill diversity. However, existing methods often face challenges in simultaneously optimizing for these conflicting objectives. In this work, we propose a new method, Adaptive Multi- objective Projection for balancing Exploration and skill Diversification (AMPED), which explicitly addresses both: during pre-training, gradient-surgery projection balances the exploration and diversity gradients, and during fine-tuning, skill selector exploits the learned diversity by choosing skills suited to downstream tasks. Our approach achieves performance that surpasses SBRL baselines across various benchmarks. Through an extensive ablation study, we identify the role of each component and demonstrate that each element AMPED is contributing to performance. We further provide theoretical and empirical evidence that, with a greedy skill selector, greater skill diversity reduces fine-tuning sample complexity. These results highlight the importance of explicitly harmonizing exploration and diversity and demonstrate the effectiveness of AMPED in enabling robust and generalizable skill learning. https: geonwoo me amped/
INTRODUCTION
Efficient exploration remains major challenge in reinforcement learning (RL), particularly in environments with sparse or delayed rewards (Sutton & Barto, 2018; Schmidhuber, 2010; Vinyals al., 2017; Litman, 2005). While biological agents naturally discover rewarding behaviors, artificial agents often rely on handcrafted reward functions, which demand extensive domain knowledge and limit scalability (Kwon et al., 2023). Skill-Based Reinforcement Learning (SBRL) addresses this by pretraining a skill-conditioned policy through unsupervised skill discovery (Gregor et al., 2016; Shi al., 2022), enabling efficient adaptation to downstream tasks.
A common approach in SBRL is to use Unsupervised Reinforcement Learning (URL) objectives during pretraining to discover diverse and useful skills (Eysenbach et al., 2019; Gregor et al., 2016). Two widely used URL objectives are: (1) maximizing mutual information (MI) between skills and promote skill diversity, and (2) maximizing state entropy encourage exploration (Laskin et al., 2022; Liu & Abbeel, 2021b) (Appendix F). However, MI-driven objectives often induce premature specialization by curtailing exploration (Campos et al., 2020; Jiang et al., 2022; Strouse et al., 2022), while entropy-basee exploration sacrifices skill distinguishability. limiting downstream utility. The core problem is how to balance these competing objectives, without resorting to ad-hoc heuristics.
their state trajectories
In this work, we bridge two URL paradigms in the theoretical framework of multi-objective reinforce- ment learning, proposing Adaptive Multi-objective Projection for balancing Exploration and skill Diversification (AMPED). Few previous studies, such as CeSD (Bai et al., 2024) and ComSD (Liu al., 2025), have explored similar integrations but either lack solid theoretical foundation exhibit significant limitations (Appendix G).
"Equal contribution.

---

Published as a conference paper at ICLR 2026
Skill Pretraining
Fine-tuning
Stand
Stand
Skill
Skill
a
Skill
SkIl 3
Skill 3
(a) Initialization
(b) Maximization
(c) Skill Selection
(d) Adaptation
Figure I: Graphical scheme explaining our method, AMPED, (a) At initialization, the skills exhibit small coverage that are close to each other in the task space. (b) During skill pretraining, exploration and diversity objectives encourage skills to widen and repel each regions. (c) fine-tuning, the skill selector identifies the skill best aligned with the target task at each step. (d) The selected skill is further adapted via extrinsic rewards to maximize performance on the target task.
Our key insight is that gradient conflicts naturally arise between diversity objectives and exploration objectives, leading inefficient updates that hinder learning (Yu al., 2020). To address this issue, we adopt gradient surgery method inspired by multi-objective RI, ensuring that conflicting gradient components are removed before applying updates (Yu et al., 2020). We use particle-based entropy and Random Network Distillation (RND) (Burda et al., 2019) to drive exploration, and adopt the AnInfoNCE objective for skill diversity. Furthermore, rather than selecting skills uniformly at random during fine-tuning, as is common in prior SBRL approaches (Bai et al., 2024; Eysenbach et al., 2019; Yang et al., 2023), we introduce a Soft Actor-Critic (SAC) based skill selector that learns to select the best matching pretrained skill (Haarnoja al., 2018). This adaptive selection manner maximally leverages the inherent diversity of the skill repertoire. graphical summary of these contributions provided in Figure! 1.
AMPED improves performance over strong baselines, DIAYN, COMSD, RND, CeSD, BeCL, CIC, and APT, across benchmarks. We evaluate AMPED on Maze environments (Campos, 2020) and the Unsupervised Reinforcement Learning Benchmark (URLB) (Laskin et al., 2021). In the Maze suite, AMPED learns well-separated skills while simultaneously achieving high state coverage, whereas competing methods fail to maximize both. On URLB, AMPED delivers statistically significant improvements in return the baselines. Taken together, these results show that explicitly resolving siiloration-diversity gradient conflicts yields substantial gains in SBRL.
Ablation studies further confirm that each component of our framework, entropy bonuses, RND, Anisotropic InfoNCE, gradient surgery, and the skill selector, contributes meaningfully to overall performance. Moreover, we show, theoretically and empirically, that greater skill diversity reduces fine-tuning sample complexity when paired with greedy skill selector. This clarifies the respective roles of diversity and selection and motivates further work on principled skill selector design.
We provide the full implementation of AMPED to facilitate the reproduction of our main results at https: /ttpss:/ggthii tttpb/ttttuuu..gg www.cooooocoooeeecoo Geonwoo wwtpe///.//........o/ amped.
PRELIMINARIES
2.1 MARKOV DECISION PROCESS (MDP) AND CONDITIONAL MDP (CMDP)
MDP is tuple M = AS, R, μi 7), where S is state space; A is an action space; P S x A (SS) is a transition model where we denote the probability of transitioning from to s with action a by P(s" a, s); R S + is reward function: μ E A is the initial state distribution; and YE (0, 1] is discount factor. A trajectory is sequence of states and actions, for example: T ($1, 01, $2, 02, -
11 $ ).) We will only consider finite horizon MDP, i.. H co. policy ((A) maps states action probabilities, denoted (als).
2

---

Published as a conference paper at ICLR 2026
Skill Pretraining
Fine-tuning
Skill Selector
Environment
Downstream task
p #t)
St
Agent
8t, Tt-1
-6- U(A)
2/
at
at
f
Agent m(a.(s
8t+1
GradientSurgery' Lerph, Vaav))
I Texploration
Udiversity
Figure 2: Overview of the training process of AMPED. During the skill pretraining phase, the agent conditioned on randomly sampled skills and optimized using intrinsic rewards for exploration and diversity. These gradients In fine-tuning phase, skill selector adaptively selects skills on each step, based on -use-specific feedback, and the agent is further optimized using extrinsic rewards from the downstream target task.
not directly used, but are balanced via gradient surgery mechanism.
CMDP extends MDP by introducing latent variable E Z, often representing skill context. The policy becomes (a]s, =), additionally conditioned on z. CMDP is used in skill discovery, where the goal learn diverse, distinct behaviors parameterized by
Both MDP and CMDP aim to maximize the expected cumulative discounted reward: max, (T EE, (()rr)] where trajectory is generated from policy
2.2 ENTROPY AND MUTUAL INFORMATION
For random variables X, Y, Shannon entropy and MI are defined as HX),, I(X;Y), respectively:
(X) ((]), I(XY)) = DKL (px,y || Px pv)
(1)
where DKL. is Kullback-Leible divergence, and Px,y is joint distribution. Higher entropy corresponds to higher unpredictability, the state distribution becomes more uniform in the state space, thereby facilitating broader exploration. In contrast, higher MI indicates stronger statistical dependence between two random variables. MI commonly used between skills and trajectories, so that each skill reliably produces its characteristic behavior. Morcover, by using contrastive learning to estimate MI, each skill repel the others, thereby achieving skill diversity.
These ieformationnthhoreeee terms widely used an intrinsic objective in URL. For example, CIC (Laskin et al., 2022): I(T;2), DIAYN (Eysenbach et al., 2019): I(S; Z) + H(ASS) - [AA; Z(S), BeCL (Yang et al., 2023): I(S()) (((s()))s))))) Details on the objectives are provided Appendix H.
2.3 GRADIENT CONFLICT
In multi-objective RL, optimizing multiple objectives simultaneously with the same network can lead to conflicts between the gradients of each objective. A naive implementation computes the gradients for each objective independently and performs gradient descent using their sum. However, this can result in gradient conflict, where the update direction that benefits one task negatively impacts another (Yu al., 2020).
To address this issue, Yu et al. (2020) proposed PCGrad, a gradient surgery method designed to mitigate such conflicts by removing interfering gradient components. Given a set of objectives Lx (0) for k = 1..... the corresponding gradients are first computed 9I: V Co The gradients are then processed sequentially
random order. For each pair of gradients gi and gv. a conflict is detected, i.e., if g, 9s 0, then the projection of gi onto Gb. is subtracted from g,- Thus, the modified gradient is guaranteed not to interfere with the descent directions of other tasks. Moreover, PCGrad paper showed that under appropriate conditions, projected gradient step can outperform standard stochastic gradient descent (SGD). Finally, the adjusted gradients are aggregated and applied using conventional SGD.
3

---

Published as a conference paper at ICLR 2026
3 ADAPTIVE MULTI-OBIECTIV PROJECTION FOR EXPLORATION AND DIVERSIFICATION (AMPED)
Our goal is maximize both skill diversity and exploration, supported by prior works (Eysenbach et al., 2019; Gregor et al., 2016; Yang et al., 2023; 2024). Previous methods in URL, such as CeSD and ComSD (Bai et al., 2024; Liu et al., 2025), have combined these objectives. Following this approach, we optimize the discounted cumulative return E |>, 7 (Tesploraiion + Fdiversity) )] using a DDPG agent (Lillicrap et al., 2015). Here, Texploration incorporates entropy and RND-bbseed objec- tives (Burda et al., 2019), while Tdiversity includes the AnInfoNCE term (Rusak et al., 2024). The specific formulations and the rationale for their use are detailed in Section 3.2. We illustrate our overall method in Figure 2.
Maximizing state entropy is essential because it induces a uniform visitation distribution, minimizing worst-case regret as shown by Gupta et al. (2018). And this principle has been empirically validated in prior works (Jain et al., 2023; Liu Abbeel, 2021b). We now briefly motivate the importance of skill diversity for downstream tasks via the following theoretical analysis.
3.1 THEORETICAL ANALYSIS OF SKILL DIVERSITY
Assume a finite state space S with cardinality S, and finite horizon H. Suppose we are given skill-conditioned policies (a 8, 2) with finite number of skills and downstream task. Let be the target policy and p E (S)) be a corresponding discounted state occupancy measure. Also set v. as a best policy in the sense that 2, probability distributions by d(p1, p2) = -lppi p2ll1.
Segminnd(pp Pz). We denoted the total variation of two
Theorem 1. Define 5 = miniyy d( Pz,, diversified, so that A = 2$ >
dp..p..). Assume that the skills are sufficiently
Draw i.i.d. trajectories from target policy ((1), s()) т and form the empirical state distribution p. Consider the greedy skill selector == arg min, d(o, pz). Then
n
Pr + 2.] 2 H exp
(2)
terms of confidence level 17 E (0, ifn 2 2 (5 log H - -lo 1)) we have Pr(# * 2.] S n.
Thus, greater diversity between skills (6) increases the margin A and reduces the number of required samples to skill selector be optimal. This formalizes the intuition that diverse skills ease identifying the skill whose induced policy is closest to the target policy æ. Proof of this theorem is provided in Appendix A.
3.2 EXPLORATION & DIVERSITY INTRINSIC REWARDS
Exploration Reward. Our exploration reward consists of two components: an entropy-based term and a RND term (Burda et al., 2019). The entropy component, widely used in SBRL (Gregor et al., 2016; Laskin et al., 2022; Liu & Abbeel, 2021a), enhances exploration when maximized and is defined as H (Stoe), where (s) = (1 7) Σ ''p(s, s) is the discounted state occupancy measure. Since the exact discounted state occupancy measure is unknown,
approximate it using a particle-based method by Laskin et al. (2022). Each particle is an embedded state pair z; = Jo. (T:) where ;; (St, 11++1) and Iva denotes the embedding function. Then the distribution is estimated using distances to its kth nearest neighbor, Ri The intrinsic reward is then computed as
Tentropy (s) = log O R.,,n),, which captures the entropy contribution of each particle.
To construct a meaningful latent space, we train an encoder with the contrastive loss:
Oen (T,) Jrp (2:) (T) llsv. (7))lll (T))[TT log
1
N
exp ( Ivr (T₂) Jwh (2,)
(3)
N
(1ge, Ill (z)
here Iwh encodes skills, and T > 0 is temperature hyperparameter from CIC (Laskin et al., 2022).
Despite its benefits, entropy- exploration alone is insufficient in fiigh-iimnnsional spaces. The particle-based entropy estimator has time complexity of On log n)), where denotes the number
4

---

Published as a conference paper at ICLR 2026
Skill
Sexpt
Sutio
t: 8 fiiual
T: projoction
(a) CeSD
(b) BeCL
Figure 3: Exploration trajectories in the Square Maze with six skills. CeSD yields more contiguous cover- age, while enforces stronger sep- aration, leaving noticeable gaps.
Figure 4: Graphical illustration of gradient surgery. When diversity gradient (red) and explo- ration gradient (blue) conflict, one gradient is ran- domly projected onto the orthogonal complement of the other to balance updates. Added gradient (purple) is used for update of parameters.
of states in the replay buffer, because it requires computing the k-nearest neighbors for every state pair. This makes it impractical for large replay buffers. In practice, following Liu & Abbeel (2021a), we truncate the buffer and compute entropy using only the most recent 5,000 states, which reduces computational overhead but inevitably degrades the fidelity of the entropy estimate.
To address this limitation, we combine entropy bonuses with RND. RND trains a predictor network fe to match the output of fixed, randomly initialized target network fuugget. producing an intrinsic
reward (s) = (s) - furger (s)ll where larger prediction error indicates novel states and thus encourages exploration. Importantly, RND provides a model-based intrinsic reward with linear complexity in (i.e., (n))). However, when data scarce in early training, RND can be noisy and unstable.
By leveraging entropy-basee exploration to supply reliable signal when the replay buffer is small, and using RND to compensate for the computational and statistical limitations of the particle-based estimator the buffer grows, the two methods complement each other and together provide more robust exploration mechanism.
We define the exploration reward a linear combination of the RND and entropy terms. Specifically, l'epploatton (s) or entropy (s) ms)m)() where and B is the positive scaling coefficients that modulate the relative influence of the entropy-basee and RND rewards. Ablation studies Appendix confirm that combining entropy and RND significantly improves exploration efficiency in high-dimensional environments.
Diversity Reward. To motivate our diversity reward formulation, we first revisit CeSD (Bai et al., 2024). CeSD optimizes (S + Ldiveritty where Ldiversity encourages trajectories generated by different skills to avoid overlapping.
However, this term treats all non-overlapping trajectories equally and therefore fails to distinguish between skills that are only mildly separated and those that are strongly separated (see Appendix G.1 for details). To address this limitation, we adopt the mutual information objective ((S(2), ((2) from BeCL. (Yang et al., 2023), where {(1) and ((2) denote states sampled from trajectories induced by the same skill 2. Maximizing I ( 5(2)) encourages states generated by the same skill to cluster together while pushing apart those generated by different skills.
Unlike CeSD's 's heuristic penalty, the MI-based formulation explicitly pushes apart skill distributions, resulting in substantially stronger skill separation. Furthermore, BeCL shows that sufficiently maxi- mizing this objective also increases state entropy, thereby jointly promoting exploration and diversity. Our 2D maze experiment (Figure 3) empirically confirms that the MI objective produces clearer skill differentiation than CcSD. A detailed discussion of skill diversification under the BeCL objective is provided in Appendix B.
For MI estimation, we employ AnInfoNCE (Rusak et al., 2024), an anisotropic variant of In- foNCE (Oord et al., 2018) designed to capture asymmetries in latent factors. To the best of our knowledge, AnInfoNCE has not previously been used for skill diversification. However, empirical results (Section 4.3) show that it provides a more effective MI estimate than standard InfoNCE in this setting, motivating its use in AMPED.
5

---

Published as a conference paper at ICLR 2026
04
05
02
T0x mlaninng Stoot
Texeiig
Walker
Quadruped
Jaco
Figure 5: Evolution of gradient conflict ratio in URLB environment.
15
0.84
4.5
0.50
0.00
0.20
3.54
3.54
1.8
1.20
1.5
Walker
Quadruped
Jaco
Figure 6: Evolution gradient similarity in URLB environment.
The AnInfoNCE. loss is defined
-1((st))((()))
CAINCE (f,
-E s,st (17) In - -l- - EM -1f(( f-)
where s, s are positive samples (from the same skill), and {s are negative samples (from different skills). The matrix À is learnable diagonal matrix, and the weighted norm is defined as ||^=|] x" Åx. The state encoder f and Å are updated via loss minimization. Accordingly, we define contrastive reward ldiversity the term inside the bracket.
3.3 BALANCING EXPLORATION AND DIVERSITY OBJECTIVES
Differentiable skills improve adaptability in dynamic skill selection but often compromise exploration ability (Yang et al., 2023), leading to suboptimal performance in environments. By treating diversity and exploration as distinct objectives, our problem can be framed as a multi-objective RL setting, in which two objectives are optimized concurrently. In this perspective, observed gradient interference can be interpreted as a form of gradient conflict, a well-documented challenge in multi-objective RL.
Figure 5 reports the gradient conflict ratio throughout training, averaged over three seeds. The conflict ratio at each step denotes the fraction minibatch samples whose exploration and diversity gradients exhibit negative inner products: (geeph, 9div)) < 0. Across all URLB domains, observe substantial gradient conflict (Walker: 0.9997 0.0004, Quadruped: 0.9997 + 0.0006, Jaco: Jaco: 0.7777 + 0. 02166, indicating persistent disagreement between objectives. To further characterize the nature these interactions, Figure 6 visualizes the gradient similarity (raw inner product). These results underscore the importance of explicitly resolving gradient conflicts enable stable optimization when combining exploration and diversity objectives.
To mitigate this issue, we integrate gradient projection method, known as gradient surgery or projecting conflicting gradients (PCGrad), proposed by Yu et al. (2020). The key idea is to remove gradient interference by projecting one objective's s gradient onto the orthogonal complement of the other (Figure 4). At each update, we randomly choose which gradient to adjust: with probability p project Ezzpl to gai.. and with probability obtained by summing two gradients, one projected. The procedure is detailed in Algorithm 1. Although more advanced methods exist (e.g., Liu et al. (2021); Navon et al. (2022)), we opted for the original gradient surgery approach due to its simplicity and ease of integration. Despite its straightforward design, this method proved sufficiently effective in mitigating gradient conflicts for application.
vice versa. Then, the final update gradient g ginall

---

Published as a conference paper at ICLR 2026
3.4 ADAPTIVE SKILL SELECTION
To fully exploit the diversity of the learned skill set, we adopt a skill-selection mechanism during downstream fine-tuning. Specifically, we train a skill selector p(z s) jointly with the policy. At cach time step, the selector samples a skill according to p( 8). and the policy conditioned on the selected skill is updated using the downstream task reward. The skill selector is optimized to maximize the same downstream reward signal as the policy. Both the policy and the skill selector are updated at every training step. To balance exploration of new skills with exploitation of high-performing ones, we employ an e-greedy strategy with e decaying over the course of training.
Prior methods often impose constraints to stabilize skill learning. For example, DIAYN (Eysenbach ct al., 2019) freezes the prior distribution, VIC (Gregor et al., 2016) fixes the skill at initialization, and other approaches rely labeled demonstrations. In contrast, our method jointly trains the policy and skill distribution, which turns out to be stable and effective.
During evaluation, the skill selector becomes deterministic, employing a greedy strategy to maximize task performance. This hierarchical framework facilitates efficient skill transfer and adaptation while maintaining decision stability. Detailed descriptions of the implementation of the skill selector provided in Appendix I.4.
(a) DIAYN
(b) BeCL.
(c) CIC
(d) CeSD
(e) ComSD
(f) AMPED
Figure 7: Agents exploring on Tree Maze after pretrained from different skill discovery objec- tives. From (a) to (f) each are trained with six skills. Visually, our approach exhibits the most distinct skills while ensuring full coverage of the state space.
4 EXPERIMENTS
In this section, we report results on the URLB benchmark, which we use as our main quantitative evaluation of downstream performance and sample efficiency. Before that, we additionally present visualizations on the Tree Maze environment to provide qualitative insight into how our method explores the state space and separates skills. We then run a series of targeted ablations to study the impact of each algorithmic component (RND, AnInfoNCE, gradient surgery, and the skill selector), and conclude by visualizing representative skills learned during URLB pretraining.
4.1 SKILL DISCOVERY IN TREE MAZE
The experiment demonstrating the skill discovery capability is conducted in a Tree Maze environ- ment (Campos et al., 2020). For details on the environment, implementations, and hyperparameters. refer to the Appendix I. The Tree Maze serves as a toy environment for preliminary analy and insight; accordingly, we evaluate a reduced set of baselines, DIAYN, BeCL, CIC, CeSD, and ComSD, compared to those used in URLB. Refer to the Appendix H for comprehensive details on the baselines.
Figure 7 illustrates the visualization of each baseline's performance after pretraining with six skills. In terms of skill distinguishability. our observations indicate that DIAYN, BeCL are capable of learning distinct skills, enabling clear differentiation among the states covered by each skill. Conversely, the skills learned by CIC are less distinguishable, likely due to the absence of a skill-differentiatiin term in its reward function. Regarding state coverage, CIC, CeSD and ComSD nearly reach the state coverage limit, whereas DIAYN and BeCL exhibit inferior performance in this regard. Notably, our proposed method, AMPED, demonstrates superior performance in both maximizing skill discriminability and state coverage, achieving the state coverage limit while each skill clearly separated. Additional experiments on the effect of varying the number of skills, results in other maze layout, and the evolution of skills over training steps are provided in Appendix C.
7

---

Published as a conference paper at ICLR 2026
Median
IQM
Mean
Optimality Gap
AMPED (Ours)
CIC
BeCL
CeSD
RND
CoMSD
DIAYN
0.45 0.60 0.75
0.45 0.60
0.75
0.45
0,60 0.75
0.30
0.45
0.60
Expert Normalized Score
Figure 8: Aggregated expert-normalized performance 12 URLB downstream tasks. Four metrics, median, IQM, arithmetic mean, and optimality gap, are plotted using the evaluation protocol introduced by Agarwal al. (2021). Our method (gray) achieves the highest median, IQM, and mean scores and the smallest optimality gap, outperforming the previous state-of-the-art APT (pink) and other baselines.
4.2 EVALUATION ON URLB
To evaluate the performance our method on downstream tasks, we utilize 12 tasks from the URLB. The benchmark comprises three domains: Walker, Quadruped, and Jaco. Detailed descriptions of the URLB are provided in Appendix I.3. Each method first pretrained for 2M steps using an intrinsic reward, followed by fine-tuning for 100K steps the downstream tasks.
For comparative evaluation, we selected strong baseline methods from the URLB, including DIAYN, APT, BeCL, CIC, RND, and the recently proposed CeSD and ComSD. Furthermore, methods such as LSD (Park et al., 2022). CSD (Park et al., 2023), and Metra (Park et al., 2024) were omitted because they do not exhibit performance improvements 2024). Our implementation adheres to the official URLB code (Laskin & Yarats, 2025). Additional information hyperparameters and network architectures can be found in Appendix For more details on reproducing these baselines, see Appendix 1.8.
the URLB relative CeSD (Bai et al.,
To ensure fair comparison in Figure 8, we fine-tuned all methods under identical conditions and did not use the skill selector. Instead, following the downstream fine-tuning protocol of Yang et al. (2023), the skill was uniformly sampled from the full skill set and switched every 50 steps.
Unless otherwise stated, all experiments on URLB were pre-trained across 10 random seeds, and each fine-tuning run reused its corresponding pre-training seed. We aggregate statistics using the rliable RL. framework (Agarwal et al., 2021), reporting median, mean, interquartile mean (IQM), and optimality gap. Here, the IQM the average over the middle 50% of returns (between the 25th and 75th percentiles), and the optimality gap is the normalized difference between an expert reference score and the agent's score, as defined (Agarwal al., 2021). The expert score derived from an expert DDPG agent following (Agarwal et al., 2021).
As shown in Figure 8, our method achieves the best results on the URLB. As recommended by Agarwal et al. (2021), use the IQM our primary performance measure. In particular, it surpasses the Skilldiiferenniaaiing methods BeCL by 17.96%, the matropy-mxiiiization method CIC and APT by 15.02% 9.73%, well the recent diversity-exploraaion hybrids CeSD and ComSD by 20.91 %, 35.01% %. These results suggest that considering both diversity and exploration is critical for downstream task performance, and more importantly, appropriately balaneing these objectives is essential. All per-task scores for each method are reported in Appendix N. In addition, we include the evolution of the four evaluation metries over training time for the Quadruped environment.
4.3 ABLATION STUDIES
In this section, present the main analyses of AMPED, Additional results, including URLB reward-scaling ablations (Appendix D) and an extended evaluation of the skill-selection mechanism (Appendix E), further support our findings.
Q1. Do all components of AMPED contribute to its performance?
8

---

Published as a conference paper at ICLR 2026
Table 1: Episode returns under component ablation. Ablating any single component, RND, AnInfoNCE loss, gradient surgery, or the skill selector, occasionally improves performance on individual tasks, yet yields degraded overall returns. AMPED (Ours) indicate the procedure including all of them; RND reward, AnInfoNCE, gradient surgery, and skill selector. The best result is shown bold, and the second-best it is underlined.
W.O.
W.O.
Domain
Task
AMPED (Ours)
Gradient Surgery
Skill
RND
AnInfoNCE
Selector
Flip
674 + 105
487 47
536 + 75
625 + 48
686 = 133
Run
467 + 103
341 67
440 +
427 + 57
517 49
Walker
Stand
951 + 38
917 + 67
950 + 25
939 + 26
947 + 19
Walk
929 +
638 60
923 + 18
899 45
886 63
Sum
3021
2383
2849
2890
3036
Jump
720 + 32
597 + 154
705 + 22
641 + 64
699 + 68
Run
494 + 53
410 : 84
496 37
453 + 13
493 54
Quadruped Stand
906 + 67
905 + 10
867 + 70
890 + 34
816 = 150
Walk
890 + 59
611 + 228
870 + 26
747 = 114
816 = 116
Sum
3010
2523
2938
2731
2824
Re. bottom left
143 +
147 14
105 33
111 27
139 34
Re. bottom right
144 + 25
132 + 40
148 + 14
114 + 35
140
- 21
Jaco
Re. top left
140 39
163
36
140 23
96 + 23
130
38
Re. top right
154 + 46
144 47
92 + 24
106 + 49
146 49
Sum
581
586
485
427
555
Ablation of AMPED shows that each component, RND, contrastive diversity, gradient surgery, and skill selection, makes non-redundant and substantial contribution the overall efficacy of AMPED. To quantify each component's s impact, we individually ablated it within AMPED and reporting the resulting relative change in total returns with standard deviation (Table 1). the Walker domain, removing RND incurs a 21. % drop; in Quadruped it costs 16.2%; in Jaco, 0.9% increased but negligible, confirming RND's crucial exploration role. Dropping the AnInfoNCE diversity term reduces Walker by 5.7%, Quadruped by 2.4%, and Jaco by 16.5% underscoring the need for skill separation. Disabling gradient surgery degrades returns by 4.3% (Walker), 9.3% (Quadruped), and 26.5% (Jaco), highlighting the value of conflict resolution. Finally, omitting the skill selector yields 0.5% gain in Walker but decreases Quadruped by 6.2% and Jaco by 4.5%, demonstrating the importance of the skill selector.
Q2. How does the gradient projection ratio affect downstream task performance?
AMPED (Ours)
0.0
1.0
1000
1000
1000
800
800
800
Return
400
400
400
200
200
Flip
Run
Stand
Walk
jump
Run
Stand
Walk
Re.bl
Re.br
Re.ti
Re.tr
(a) Walker
(b) Quadruped
(c) Jaco
Figure Task-level return comparison under different projection ratios. In the Jaco domain, task labels "Re.bl", "Re.br", "Re.1l", and "Re.tr" refer to reaching the bottom-left, bottom-right, top-left, and top-right targets, respectively.
balanced projection ratio effectively mitigates gradient conflicts and consistently improves skill learning across diverse environments (Figure 9). We compare three projection strategies: projecting

---

Published as a conference paper at ICLR 2026
the exploration gradient onto the diversity gradient (p 0.0), projecting the diversity gradient onto the exploration gradient (p = 1.0), and our default AMPED configuration, which uses a fixed, -neiroi projection ratio. The exact values of the projection ratios provided in Appendix 1.5. Results are reported as mean + standard deviation; the = 0.0 and p = 1.0 variants averaged over three random seeds. AMPED achieves the highest aggregate returns in all three domains. Full numerical results for the projection ratio ablations are provided in Appendix N.2.
Q3. How does the number of skills affect downstream task performance?
AMPED (Ours)
skill-8 8
skill-32 -32
1000
1000
1000
800
800
Rotum
600
600
400
400
400
200
Flip
Run
Stand
Walk
Jump
Run
Stand
Walk
Re.bl
Re.br
Re.tt
Re.tr
(a) Walker
(b) Quadruped
(c) Jaco
Figure 10: Task-level return comparison under different numbers of skills.
Increasing the number of skills does not necessarily increase diversity. As illustrated in Figure 12, small skill set may already achieve complete coverage of the environment, leaving limited room for additional skills to expand diversity. Moreover, even in larger environments. diversity is opti- mized through an approximate mutual-information objective rather than an exact estimator, meaning that increasing the number of skills does not guarantee proportional increases in diversity. Conse- quently, choosing an appropriate skill dimension is critical for maximizing diversity and improving downstream performance. To examine this effect, we conduct an ablation study on the skill dimension.
We vary the skill dimension while keeping all other components of AMPED fixed and report the downstream performance in Figure 10. Our default configuration (16 skills) achieves the strongest results in Quadruped and Jaco, while performing competitively with the 32-skill variant in Walker. These findings indicate that an appropriately chosen skill dimension is essential. In experiments, we follow prior work (Yang et al., 2023) and adopt 16 skills, but developing principled methods for selecting the optimal number of skills remains an interesting direction for future research. Full numerical results for the projection ratio ablations are provided in Appendix N.3.
5 CONCLUSION
In this work, we introduce AMPED to jointly address exploration and skill diversity SBRL. Our framework unifies entropy-based exploration with contrastive skill separation, explicitly mitigates their gradient conflicts via PCGrad, and employs skill selector to adaptively deploy skills during fine-tuning. Empirically, we find that (i) reducing exploration-divens to improved performance, (ii) combining AnInfoNCE AnInfoNCEEinspprre diversity losses with RND-driven entropy bonuses balances the objectives. Taken together, our theoretical analysis highlights the crucial role of skill diversity in reducing sample complexity and enabling more effective skill selection.
gradient interference leads
While AMPED was developed for skill-based RI. its core insight, treating exploration and diversity as competing objectives and resolving their gradient conflicts via projection, extends to other settings with multiple learning signals, motivating broader use of gradient- prooiettt-rjectionn methods. Future research could adopt more advanced conffict-resolution techniques and remove remaining heuristics, or identify alternative objective functions that better reconcile exploration and diversity. Further details are available in Appendix K. By tackling these challenges, the SBRL community can progress toward creating richer, more capable agents.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by IITP (RS-2024000445087; 10%, RS-2025-25410841 10%, No. 2019-0- 01842; 10%), NRF TRS-2024-00451 162; 20%, RSS-2225-0263335 20%, RKS-2025-2541992 20%), GIST (KH0870; 10%) funded by the Ministry of Science and ICT, Korea. Computing resource were supported by GIST SCENT-AI, AICA, KISTI, and NIPA.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34:29304-29332 2021.
Ziqiao Ao and Jinglai Li. Entropy estimation via normalizing flow. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 9990-9998, 2022.
Akhil Bagaria, Jason K Senthil, and George Konidaris. Skill discovery for exploration and plan- ning using deep skill graphs. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 521-531. PMLR, 18-24 Jul 2021.
Chenjia Bai and Rushuai Yang. Constrained ensemble exploration for unsupervised skill discovery, 2024. URL https::///......... / / g thub. com Ba chen j j CeSD.
Chenjia Bai, Rushuai Yang, Qiaosheng Zhang, Kang Xu, Yi Chen, Ting Xiao, and Xuelong Li. Constrained ensemble exploration for unsupervised skill discovery. In Proceedings of the s International Conference Machine Learning, ICML'24. JMLR.org. 2024.
Kate Baumli, David Marde-Farley, Steven Hansen, and Volodymyr Mnih. Relative variational intrinsic control. Proceedings of the AAAI Conference Artificial Intelligence, (118))6633-67000 May 2021. doi: a0.1669/aaai.v3555 16832. URL httpttttt///c....t. http:::://ojs.aaeee aaai www..//....a.aa...a........ index. php/ php/AAAI/ /AAI// article/viewwaeeeeee......cttttt 16632..
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, ICML' 18. JMLR.org, 2018.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.
Victor Campos. Explore, discover and learn: Unsupervised discovery of state-covering skills, 2020.
URL https: tit. com/ vi victorcampos /edl.
Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-I-Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of statecoveriing skills. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 1 119 Proceedings of Machine Learning Research, pp. 1317-1327. PMLR, 13-18 Jul 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924. Curran Associates, Inc., 2020.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Representa- fions, 2019.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of unsupervised reinforcement learning. In International Conference on Learning Representations, 2022. URL https: /ttp::/:///////e///eeee//ee net fww.meiimmioooooe wtw.rrmmmmmmiw22000000wr..

---

Published as a conference paper at ICLR 2026
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforce- ment learning. In International Conference on Learning Representations. 2017.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. ArXiv preprint ar xxxv:1611.00500) 2016.
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta- learning for reinforcement learning. ArXiv, abs/1806.00444) 2018. URL https://api.
semanticscholas wwtpooroosooooooooooooooo
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pp. 1861-1870 PMLR, 2018.
Arna Kumar Jain, Lucas Lehnert, Irina Rish, and Glen Berseth. Maximum state entropy explo- ration using predecessor and successor representations. In Information Processing Systems, 2023. URL https: http:s:/oppeereiieeeeeee /oppnrevieww net forum? ? t tFsttqGmkn.
Conference on Neural
Zheyuan Jiang, Jingyue Gao, and Jianyu Chen. Unsupervised skill discovery via recurrent skill training. In Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 39034-39046. Curran Associates, Inc., 2022.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In The Eleventh International Conference on Learning Representations, 2023. URL
https: wpwpreviewweeeeeo///ee net/ / dateruumiid11ooNNggggggg
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85:1-22, 2022. ISSN 1566-2535, doi: https://doi. 11gp:.00166o..nffuu.222222..00 aaticieepp11/1555653322200
URL https: http:/wwwwccc.. wttp://wwwweeedeeeeettetceeeeeee wwt.:eeettcttceeeeeeeeeeeeeeeeee com/science/
Michael Laskin and Hao Liu. Cic: Contrastive intrinsic control for unsupervised skill discovery, 2022.
URL https:ttp:://tc. //t:tt//ttttt/bb com/ feom/rll-ree http://llrreerrccci...ee
Michael Laskin and Denis Yarats. Urlb: Unsupervised reinforcement learning benchmark, 2025.
URL http::tpp/:::m //g /gg thub. com/ / wwl--rrseecc/urrree wwtprrcchhhl__eehhhheeee rk.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Unsupervised reinforcement learning with contrastive intrinsic control. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advunces in Neural Information Processing Systems, volume 35, pp.
Curran Associates, Inc., 2022.
Misha Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Timothy Lillicrapp, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ArXiv preprint a5Xiv:1509. 02971, 2015.
Jordan Litman. Curiosity and the pleasures of learning: Wanting and liking new information. Cognition and Emotion, 1966):93-8144 2005. doi: 10. 00.008000269993334100000000 URL https: ///oi.oor////
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and qiang liu. Conffict-averse gradient descent for multi-task learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6736-6747. PMLR, 18-24 Jul 2021a.
12

---

Published as a conference paper at ICLR 2026
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 18459-18473. Curran Associates, Inc., 2021b.
Xin Liu. Comsd: Balancing behavioral quality and diversity in unsupervised skill discovery, 2025.
URL htttps://githiii
com/ 1 Liuxiinn 824 ComSD.
Xin Liu, Yaran Chen, Guixing Chen, Haoran Li, and Dongbin Zhao. Balancing state exploration and skill diversity in unsupervised skill discovery. IEEE Transactions Cybernetics, 2025.
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16428-16446. PMLR, 17-23 Jul 2022.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. In International Conference on Machine Learning, pp. 26228-26244, PMLR, 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03745 2018.
Seohong Park, Jongwook Choi, Jackyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz-constrained unsupervised skill discovery. In International Conference on Learning Representations, 2022.
URL wwtptttt//// /ppnnevieneeeeeeeeeeeeeeeeeereee net.eeeweeeeeeeeeeeeeeeee..eeeee /ateee/frrrrttt ? Bd= Bd OghNgA.
Seohong Park, Kimin Lee,, Youngwoon Lee, and Pieter Abbeel. Controllability-aarr unsupervised skill discovery. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 17725-277244 PMLR, 23 29-29 Jul 2023.
Seohong Park, Oleh Rybkin, and Sergey Levine. METRA: Scalable unsupervised RL with metric- aware abstraction. The Twelfth International Conference on Learning Representations, 2024.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell, Curiosity- driven exploration by self-supervised prediction. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2778-2787. PMLR, 06-11 Aug 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5062-5071, PMLR, 09-15 Jun 2019.
Roben Delos Reyes, Kyunghwan Son, Jinhwan Jung, Wan Ju Kang, and Yung Yi. Curiosity-driven multi-agent exploration with mixed objectives. arXiv preprint 12X.v:2210.164:4 2022.
Evgenia Rusak, Patrik Reizinger, Attila Juhos, Oliver Bringmann, Roland S Zimmermann, and
Wieland Brendel. InfoNCE: Identifying the gap between theory and practice. Highh-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024. URL https:
//openreview.r net htw.://wwwneeeecommmiieedooooeee for.m?id=-uufuu12222
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 2(3):2002444 2010. doi: 10.1109TTAMD. 2010.2056368.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv preprint 1707.00:707..33330 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-awan unsupervised discovery of skills. In International Conference on Learning Representations, 2020. URLhtttp:://ppeeeerieee http:////ppeeeeeeeeee net hwt.:neuttrrtneeee forum? ZHH0g,ZRKK
URL https:
13

---

Published as a conference paper at ICLR 2026
Lucy Xiaoyang Shi, Joseph J. Lim, and Youngwoon Lee. Skill-based model-based reinforcement learning. Conference on Robot Learning, 2022.
DJ Strouse, Kate Baumli, David Warde-Farley, Volodymyr Mnih, and Steven Stenberg Hansen. Learning more skills through optimistic exploration. In International Conference on Learning Representations, 2022. URL https: opwnreview....ee/ooooe nnt forum? ? BeldddUUUkuuuuue
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning, second edition: An Introduction. Bradford Books, 2018
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. al. Deepmind control suite. ArXiv preprint arXiv: 2018.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John P. Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, H. V. Hasselt, David Silver, Timothy P. Lillicrap. Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp. and Rodney Tsing. Starcraft ii: A new challenge for reinforcement learning. ArXiv, abs/1708. 04782, 2017.
Rooshy Yang. Beel: Behavior contrastive learning for unsupervised skill discovery, 2023, URL https: / com/ wwt./oom/////eeeeee mooshhy-yange BecL.
Rushuai Yang. Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, and Xuelong
Li. Behavior contrastive learning for unsupervised skill discovery. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org. 2023.
Yucheng Yang, Tianyi Zhou, Qiang He, Lei Han, Mykola Pechenizkiy, and Meng Fang. Task adaptation from skills: Information geometry, disentanglement, and new objectives for unsupervised reinforcement learning. In The Twelfth International Conference on Learming Representations. 2024.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11900-11931. PMLR, 18-24 Jul 2021.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN
Rui Zhao, Yang Gao, P. Abbeel, Volker Tresp. and W. Xu. Mutual information state intrinsic control. In International Conference Learning Representations, 2021.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A PROOF OF THE THEOREM
Assume finite state space S with cardinality S, and finite horizon H. Suppose given skill-conditioned policies (a 8,2) with a finite number of skills and downstream task. Let æ be the target policy and p E (S)) be corresponding discounted state occupancy measure. Also set 2, as a best policy in the sense that 2, = argmin, _dpp, Pz). We denoted the total variation two probability distributions by d(p1, P2) = 2lpp= - p2111.
Theorem Define 8 minez dp Ps,), d(p, pz.). Assume that skills are diversified enough. so that A 25 > 0.
Draw i.i.d. trajectories from the target policy ((1), ...., ((m) n, and form the empirical state distribution P. Consider the greedy skill selector 2 == arg min; d(p. Pz). Then
n1a
Pr x.] S 2 H
Hep exp
(5)
2
In terms of confidence level E if
2 A
(S log 2 + log H log 17),
(6)
we have + 2.] S n
Proof.
Step 1. A sufficient condition for correct selection.
Define d= d(p, p). Triangular inequality gives dl Pz,) s d(p.pz.) + d((.p) = =
A
d <
2 2 c, i.e. 8> 2(e + d),
then 2 2, because for every & # 2., by triangular inequality,
d(p.p:)= 2 d(p:.. d p..) 2 8 (e d) > d(p.ps.).
Hence
Pr I Pr
Step 2. Convergence of ő By Bretagnolle- Huber-Crrb (BHC) incquality, for a discrete random variable X X2,....... X&),
Pr
X; npdl 2 2\v s z -22 X ~ Mult(n, p).
1=1
Let the state space S = (s1,...88) If we denote ((k) = (St
(St....s
St,) E S", by definition,
h
1 ss..
nHl
k=
Define S-dimension random vectors X, .x.) where j 1,..., H as X.r #li: 8m }. Then Xj is n--muliinomial distribution with
P (w),...p)) (p(S) st),-..,p(S, =ss))-
15

---

Published as a conference paper at ICLR 2026
Thus
ā
1 2
M|l1
Σ
(p()) p())
Σ
n
Σ 1=
1 (s)
1
Σ p(s; =
nH
H
SES
,=1
#li: S.. = - -
=
2nH
j=1 #
Xy - x n
2nH
H
Pr i 2
Pr
m npp > 2nHE
/=1 m=1
< Pr
Ej s.t.
X nppl 2 2nd
m=1
Σ Pr
XJ mppl
2 2n
$ 2 He 2nE2
Now taking & = and combining with equation 7 gives
nd2
Pr #x] < 2 H exp
2
In terms of confidence level 11 E (0, 1),
n N 12 (S 2 10---1 loog H - -loogg 1)
2+) Sn.
B ANALYSIS ON ANINFONCE
Contrastive
-5
-6-
Distance of Vectors
Figure 11: Effeet of mean vector's distance on Loss. AnInfoNCE objective has a positive correlation with distance between two distributions. Loss was calculated by sampling 1000 points from two 5-dimensional Gaussian distributions.
16

---

DIAYN Published as a conference paper at ICLR 2026
An analysis conducted to evaluate the behavior of the AnInfoNCE objective function of dicr-distribution separation. Two mean vectors were first drawn independently from the standard normal distribution and used to parameterize two Gaussian distributions with identity covariance. AnInfoNCR was then estimated by Monte Carlo sampling from each distribution. Empirically, we observe that the AnInfoNCE loss increases monotonically with the Euclidean distance between the two mean vectors, indicating that larger separations yield higher objective values (Figure 11). This monotonic relationship highlights the ability of AnInfoNCE to promote diversity between learned skills, in contrast to the CeSD objective, which collapses to zero whenever the support sets of the two distributions do overlap.
C ADDITIONAL EXPERIMENTS IN MAZE
C.I ANALYSIS OF THE EFFECT OF SKILL COUNT
BeCL
CIC
ComSD
AMPED (Ours)
skills
10 skills
15 skills
20 skills
25 skills
Figure 12: Skills learned in the Tree Maze under varying skill counts. This visualization shows the skill allocations of each method as the number of skills increases from 5 to 25. AMPED consistently fills the maze with well-separated regions, whereas DIAYN and BeCL leave gaps, and CIC and ComSD exhibit increasing overlap as the skill count grows.
Figure 12 illustrates how different methods partition the Tree Maze as the number of skills increases from to 25. Unlike DIAYN and BeCL, which tend to leave large regions unexplored or produce overlapping trajectories, our method fills the entire maze while maintaining clear separation between skills. When using 10 or 15 skills, both CIC and ComSD exhibit substantial mixing between skill regions, whereas AMPED preserves distinct, non-overlapping coverage for each skill. At higher skill counts (20 and 25), all methods begin to overlap simply due to capacity limits, making AMPED's ' advantage ComSD less visually pronounced. Nonetheless, it still outperforms CIC in maintaining
17

---

Published as a conference paper at ICLR 2026
cleaner skill boundaries. These results confirm that AMPED effectively balances exploration and diversity even as the dimensionality of the skill space grows.
C.2 COMPARISON ON THE SQUARE MAZE
DIAYN
BeCL
CIC
ComSD
AMPED
Figure 13: Skills learned in the Square Maze. Each method's trajectories shown: AMPED explores every corridor with separated regions, whereas DIAYN and BeCL leave gaps and CIC and ComSD exhibit overlapping trajectories.
In the Square Maze, AMPED achieves full coverage with largely distinct skill regions and only minor overlap (Figure 13). All visualizations use 15 skills per method. By contrast, DIAYN and BeCL leave large areas under-explored learn only few broad behaviors, sacrificing either coverage or separation. CIC covers most of the state-space but generates highly entangled trajectories, indicating poor skill disentanglement. ComSD attains coverage similar AMPED but exhibits more pronounced region mixing. Taken together, these results show that AMPED not only generalize beyond the Tree Maze but both maximizes exploration and enforces strong skill diversity in the Square Maze as well.
C.3
COMPARISON OF MI AND ENTROPY
2.901
23700
Information
3.7-
2.0
3.2-
0.5
Condd
ADFED (Cus)
2000
4000
M000
8502
1000C
BAW
(Dun)
(a) Entropy Estimation
(b) MT Estimation
Figure 14: Entropy and MI estimates in the Square Maze. (a) Particle- based entropy estimates (Liu & Abbeel, 2021b) show that AMPED achieves significantly higher state entropy (exploration) than diversity-focused methods (BeCL., DIAYN), while matching CIC and ComSD. (b) MI estimated via MINE (Belghazi et al., 2018) indicates that AMPED attains diversity comparable to BeCL. and DIAYN and substantially exceeds CIC and ComSD.
In the Square Maze, AMPED maximizes both mutual information (MI) and state entropy (Fig- ure 14). To assess whether multi-objective optimization degrades any single objective relative to single-objective training,
compare AMPED's MI and entropy losses against mono-objective base- lines. Following BeCL, we use a particle-based entropy estimator and MINE-based MI estimator, evaluating 10 skills in the Square Maze (Belghazi et al., 2018). Taken together, these results indicate that AMPED simultaneously maximizes exploration and diversity.
C.4 EVOLUTION OF SKILLS ACROSS TIME STEPS
As illustrated in Figure 15, early in training (epoch 1), the policy aggressively explores new branches, rapidly expanding its state coverage. By epoch and 5, the trajectory has spanned nearly the entire
18

---

Published as a conference paper at ICLR 2026
(a) Epoch
(b) Epoch!
(c) Epoch
(d) Epoch
(e) Epoch
Figure 15: Skill trajectory evolution in the Tree Maze over training epochs. A representative skill's path is shown at 2-epoch intervals, illustrating initial rapid expansion of coverage followed by progressive refinement into distinct, well-separated skill regions.
maze, maximizing exploration. later stages (epoch and 9), the skill's s path refined: it begins to adjust which corridors it traverses carve out distinct regions and increase diversity. Although such clear visualizations are not possible in high-dimensions spaces, this simple sequence provides intuition for how method first drives broad exploration and then sculpts well-separated skill behaviors.
D ABLATION STUDY ON REWARD SCALING FACTOR
Table Performance comparison under extreme a and / settings. and / control the relative weight of entropy-based and RND rewards. AMPED (Ours) result are computed as return (mean = standard deviation) over random seeds, while each a or 3 configuration is evaluated using three random seeds. The best result is shown in bold, and the second-best is underlined.
Domain
Task
AMPED (Ours)
100
8=
8 1000
Flip
674 + 105
609 + 44
587 74
524 + 38
586 + 88
Run
467 + 103
505 + 10
420 = 99
382 + 29
505 + 39
Walker
Stand
951 +:
942 26
956
948 1
923 1:
Walk
929 1
43
863 1:
878 1
Sum
3021
2969
2871
2717
2892
Jump
720 + 34
677 + 50
710 59
623 + 100
648 + 61
Run
494 1: 56
493 23
459 1 122
371 127
613 1 92
Quadruped
Stand
906 + 71
55
837 + 117
904 +
875 +
Walk
890 +
58
805 73
118
852 +
Sum
3010
2879
2811
2618
2998
Rc. bottom left
143 + 34
94 + 31
123 = 22
133 + 35
134 + 16
Re. bottom right
144 27 27
111 15
96 15
114 35
118 + 37
Jaco
Re. top left
140 1
11 18
24
145
121 1
Re. top right
154 1
112 22
18
-
106 1
Sum
581
432
521
504
479
In this ablation, we keep every setting in Appendix I.5 fixed except for one of the reward-scaling factors. As shown in Table 2, deviating from the defaults on either or 3 degrades the sum of episode returns in most domains. Setting = 0 (no entropy reward) / (no RND reward) leads to substantial drops, while excessively large values (a 100 or 3 1000) improve some individual tasks but hurt overall performance. The default AMPED weights achieve the best aggregate performance, underscoring the need for balanced scaling between exploration and novelty signals.
In Table 3, we present hyperparameter search results for the entropy reward weight a and the RND reward weight 3. In the Walker domain, the average sum return across five (a. 8) configurations 2989.4, 4. which still ranks second among the baselines Table 13. Notably, the (0.01, 8) setting achieves return of 3323, outperforming all baselines. Similarly, in the Quadruped domain, the
19

---

Published as a conference paper at ICLR 2026
Table 3: Hyperparameter search results for and 3. Results reflect single-run returns for each modified configuration, except for AMPED. The best result is shown in bold, and the second-best is underlined.
Domain
Task
AMPED (0.01, 10)
(0.02, 10)
(0.005, 10)
(0.01, 12)
(0.01, 8)
Flip
674 + 105
667
535
599
886
Run
467 + 103
451
354
550
565
Walker
Stand
951 = 38
960
942
964
949
Walk
929 19
893
891
797
923
Sum
3021
2971
2722
2910
3323
Task
AMPED (0.002, 8) (0.004, 8)
(0.001, 8)
(0.002, 10)
(0.002, 6)
Jump
720 + 34
597
702
710
717
Run
494 = 56
499
276
512
488
Quadruped
Stand
906 = 71
956
792
918
918
Walk
890 62
833
853
891
869
Sum
3010
2885
2623
3031
2992
average across the five configurations is 2908.2, exceeding the best baseline performance. Moreover, the (0.002, 10) configuration yields even better performance than our default hyperparameter choice. These results suggest that careful tuning of and / can yield further improvements for AMPED.
E ANALYSIS ON SKILL SELECTION
One of the central motivations for our approach is that the skill selector, responsible for choosing among diverse, pretrained set of skills, should substantially enhance overall performance. However, an ablation the Walker domain (3021 with the selector vs. 3036 without it) revealed that adopting the skill selector does not lead consistent improvements.
To investigate this discrepancy, we designed a complementary experiment isolating each pretrained skill. For each task, fix single skill and condition the policy exclusively on that skill during fine-tuning, both for training and evaluation. We conducted three-seed evaluations on Walker-flip, run, where the selector had previously degraded performance, and on Quadruped-stanne it had been beneficial (3,010 with the selector vs. 2,824 without).
Table 4: Fine-tuning returns under different skill-selection regimes. "Single-Skill" reports the average return across fine-tuning each pretrained skill individually; "Oracle Best Skill" denotes the highest return achieved by the single best skill. Results are computed over three random seeds and reported as mean = standard deviation. The best result is shown in bold, and the second- best is underlined.
Domain
Task
Skill Selector
Random Skill
Single-Skill
Oracle Best Skill
Flip
674 + 105
686 + 133
719 121
913 +
Walker
Run
103
517 + 49
503 = 74
603 +
Stand
906 + 71
816 150
911 + 43
959 +
Quadruped
Walk
+ 62
816+ 116
837
+
912 + 17
The fixed-skill results in Table show that the skill selector underperforms single-skill fine-tuning across three tasks. Crucially, this deficit persists whether the selector had previously appeared beneficial (as in the Quadruped environments) or not (as in Walker), indicating that the selector itself, rather than domain-specific factors, is the primary bottleneck. Although dedicated fine-tuning benefits from sustained gradient updates targeted at a single skill, the observed performance gap suggests that the selector's 's learning hampered, likely by sparse rewards. Consequently, its value estimates
20

---

Published as a conference paper at ICLR 2026
Table 5: Number of unique skills used per task. Results are computed over three random seeds and reported as mean + standard deviation.
Domain
Task
Unique Skills Used
Flip Run
4.00 1: 1.73
Walker
3.67 + 1.53
Stand
2.67 0.58
Quadruped
Walk
4.33 1: 0.58
remain unstable, leading to suboptimal choices. These findings underscore the need for more robust training strategies for the skill selector.
To assess the stability and consistency of the skill selector, we measure the number of distinct skills invoked during the last 5K fine-tuning episodes for each downstream task. As summarized in Table the selector consistently concentrates on a small subset of available skills, particularly toward the end of training. This pattern suggests convergence stable, task-specific mapping from observations to high-level skills, underscoring the utility and reusability of the learned skill space.
F RELATED WORKS
F.1 UNSUPERVISED REINFORCEMENT LEARNING
URL aims to train general-purpose policies capable of rapid adaptation to diverse downstream tasks. This is achieved through the design of intrinsic objectives rewards that guide exploration without relying on explicit external feedback. URL typically involves two stages: (1) pretraining, where agents develop foundational behaviors driven by intrinsic motivation, and (2) fine-tuning, where these behaviors are adapted to task-specife objectives.
URLB (Laskin et al., 2021) categorizes existing URL algorithms into three primary groups:
1. Data-based approaches encourage agents to explore novel states by maximizing state entropy, fostering diverse experiences during pretraining. Notable methods include APT (Liu & Abbeel, 2021b), which utilizes particle-based entropy estimators to maximize the distance between k-nearest. neighbors (kNN) in observation embeddings. ProtoRI. (Yarats et al., 2021) builds on this idea by incorporating prototypical representation learning, inspired by SWaV (Caron et al., 2020), to enhance exploration efficiency. CIC (Laskin et al., 2022) extends ProtoRL by introducing skills, positioning CIC as both data-based and competence-bast baseed method.
Knowledge-base approaches aim to improve an agent's understanding environmental dynamics by maximizing prediction errors, thus incentivizing the exploration of novel poorly understood states. The Intrinsic Curiosity Module (ICM) (Pathak et al., 2017) encourages exploration by reward- ing agents based the error in predicting future state transitions. Reyes et al. (2022) extended this idea by incorporating the prediction of joint observations. On the other hand, disagreement-hh -aased methods (Pathak al., 2019) quantify uncertainty through an ensemble of predictive models, reward- ing states where model predictions diverge significantly. Random Network Distillation (RND) (Burda al., 2019) measures novelty via the prediction error of random, fixed target network, where higher errors indicate unfamiliar states. Nikulin et al. (2023) enhanced this idea by applying Feature-wist Linear Modulation.
3. Competence-basee based approaches. often referred to as unsupervised skill discovery, seek to develop diverse repertoire of skills without relying on external rewards. These methods are grounded in iifommation-theoree or trajectories to ensure meaningful and diverse behaviors. For instance, VIC (Gregor et al., 2016) maximizes controllability of skills by setting MI between skills and final state, given the initial state as an objective. DIAYN (Eysenbach et al., 2019) encourages diversity by maximizing MI between skills and states while ensuring skills are distinguishable. BeCL (Yang et al., 2023) leverages contrastive learning to enhance skill discriminability by maximizing MI between trajectories generated from the same skill; this also has side effect that maximizes the entropy in the limiting case.
principles, typically maximizing MI between skill embeddings and state,
21

---

Published as a conference paper at ICLR 2026
Our approach synthesizes principles from data-based, knowledge- bawle, and competence-bast based methods, drawing on models such as CIC (Laskin et al., 2022), RND (Burda et al., 2019), CeSD (Bai et al., 2024), and BeCL (Yang et al., 2023). Specifically, address the limitations of these models in balancing exploration and skill diversity by introducing novel methods for integrating them.
F.2 UNSUPERVISED SKILL DISCOVERY
Competence- based approaches, commonly referred as unsupervised skill discovery, have garnered significant attention in recent years due to their potential to enable agents to acquire diverse, dis- criminative behaviors without external supervision. focuses on enabling agents to learn distinct, discriminating behaviors without external supervision. Skill diversity has been shown to be critical for downstream task performance, both empirically and theoretically (Eysenbach et al., 2019; Laskin al., 2022; Yang et al., 2024). This is often achieved by maximizing the MI between states trajec- tories with skills, encouraging agents to develop diverse and meaningful behaviors. Key contributions in this area include works by (Gregor et al., 2016; Florensa et al., 2017; Eysenbach et al., 2019; Sharma et al., 2020; Baumli et al., 2021).
However, these studies (Campos et al., 2020; Strouse et al., 2022; Park al., 2022) have highlighted limitations in traditional MI-based methods, noting that maximizing MI between states and skills can lead to suboptimal exploration. It is also theoretically shown that such approach can not construct an optimal policy for some downstream tasks (Eysenbach et al., 2022; Yang et al., 2024). There are some methods to address this when the observation space Cartesian coordinate space (Park et al., 2022; Zhao et al., 2021); while effective in specific navigation tasks, these approaches impose strong assumptions and are less adaptable to general situation. To address these limitations, alternative approaches introduce auxiliary exploration mechanisms and refined training techniques aimed at enhancing exploration. While many methods focus on modifying the objective functions, such as DIAYN, BeCL, CeSD, ComSD, and CSD, others explore architectural innovations and dynamic exploration strategies, as seen in DSG (Bagaria et al., 2021), EDL (Campos et al., 2020), and ReST (Jiang et al., 2022). These techniques aim to promote diverse exploration without relying solely on objective modifications.
G DIFFERENCE WITH FORMER STUDIES
Prior to our work, two representative methods for jointly considering exploration and diversity are CeSD and ComSD. However, method departs from these approaches in several important ways, which will be explained in detail. Note that on URLB, our approach achieves 20.91 % and 35.01% % higher returns than CeSD and ComSD, respectively.
G.1 DIFFERENCE WITH CESD
Instead of diversifying skills using MI, CeSD maximizes exploration using the entropy, while adding a regularization term for diversifying skills. This approach mitigates the paucity of exploration while simultaneously accounting for diverse array of skills. Unfortunately, the algorithm is time-consuming because it includes clustering states. The paper avoids this bottleneck by choosing a subset of states for clustering, which would lead to inaccurate estimation of clustering and, therefore, instability of training. Also, their diversity regularization becomes ineffective once the state distributions of different skills no longer intersect. This can be seen from their auxiliary reward:
+
Here, S; denotes the state coverage set of skill i, S. is the expanded coverage obtained during the exploratory phase, and sçlu is the clustered coverage set after applying clustering to the union of all collected S/" across skills. Because clustering is applied to this union, once the supports of different skills are already disjoint, the diversity term effectively saturates: the clustering no longer distinguishes between "slightly separated" and "'very far apart" supports. In this regime, the regularizer ceases to provide a meaningful gradient that encourages further separation or broader coverage, and skills can end up concentrating on a limited portion of the state space even when other regions remain under-explored.
22

---

Published as a conference paper at ICLR 2026
Such an effect can be seen in the 2D maze experiment; other methods like BeCL DIAYN separate skills effectively, while CeSD does not. This may be advantageous in low dimension environment like 2D maze because such as URLB, insufficient separation of skills degrades downstream task performance, as established by our Theorem 1.
fully cover the whole space. However, in high- nigh-dimensional domains
G.2 DIFFERENCE WITH COMSD
Similar to ComSD, our approach aims to balance the diversity and exploration objectives. ComSD) uses the entropy of trajectory (T) as a exploration objective, and negative entropy of trajectory conditioned to skill (7
diversity objective. H (T) is estimated using particle-based approach and (T is estimated using a variational approach. To balance exploration and diversity, ComSD employs specialized weighting mechanism called Skill-based Multi-objective Weighting (SMW), which assigns different optimization objectives to different skills; some skills emphasizing diversity while others prioritize exploration. However, this selective assignment does necessarily lead to optimal overall performance. ComSD's s method merely differentiates each skill's repulsiveness from others, which does not guarantee an ideal trade-off between exploration and diversity, Moreover, it lacks a solid theoretical foundation to justify the weighting strategy.
In contrast, our method explicitly aims maximize both exploration and diversity, grounded in the concept of gradient conflict, which has been extensively studied in prior research (Yu et al., 2020; Liu al., 2021; Navon et al., 2022). By directly addressing the conflicts between exploration and diversity gradients, our method achieves more stable and theoretically justified optimization process.
H OBJECTIVES AND REWARDS
Table 6: Comparison of algorithms. Intrinsic Objectives and Rewards of each methods are shown.
Algorithm
Intrinsic Objective
Intrinsic Reward (rt)
APT
(ó(())
ppurl
ICM
E, [x. rrrl
(1 (1(411)l
RND
E, E, 7'r
[f(x)) - f(x,)112
CIC
I[(T) (T(22)
partt log (T(=)) log (2(s)) 1) log p(z)
DIAYN
H(z) H(zss) 2a)ss
DADS
H(s'(s) - H
log 18,2) - log > ( 2s))
BcCL
(150)s(01)
peantr
CeSD
H(s) a ..E dd ds)(s))
ppant
[a(1Spp0 S 1 a. prenee
ComSD
H(T) H(Tl2)
part
AMPED (Ours) a (s)) + 3 LAND + 1(s(2) s(2))
a pport + 3 (1f(xz) - f(x,)11 Antnfo
denote skill, d(s) denote a encoded state, and NN; denote A nearest neighbor. Neglected the loss for training state encoders. prart a particle-based entropy estimation, and ponnr is a contrastive- based MI estimation: the specific reward varies slightly depending the method. The canonical MI objective by InfoNCE defined as:
n
exp(f(s()) (1) f(s(2))/k)) Σ Es- exp(s(s()) -((;))k))
port Σ
log | - NNx(2))l.
contr
(8)
I IMPLEMENTATION DETAILS
1.1 MAZE ENVIRONMENTS
The maze environments are adapted from the open-source EDL implementation by (Campos, 2020). In this setup, the observation is given S TR2, which represents the current position, while the
23

---

Published as a conference paper at ICLR 2026
Table Environment detail of square-maze used for evaluation.
Parameter
Value
State space
S E R2
Action space
A [-.955, 0.95]²
Episode length
Size: Square maze (Figure 13)
5 5
Size: Tree maze (Figure and Figure 12)
7
action is given as A E RR2, corresponding to the velocity and direction. The agent observes only its current position and does not have access to the locations of walls, which must be inferred through interaction with the environment.
the start of each episode, the agent's initial state uniformly sampled within a x I tile. Table 7 summarizes the details and topological characteristics of each maze used in the experiments.
1.2 NETWORK ARCHITECTURES FOR MAZE EXPERIMENTS
All code was based on the open- EDL implementation by Campos (2020). We used PPO (Schul- man et al., 2017) as our on on-policy algorithm, with both policy and value functions parameterized by three hidden layers of size 128 and ReL.U activations. The policy network takes the concatenated state and goal vectors, passes them through three 128-unit MIP layers, then applies a tanh output scaled by the action range. The critic shares the same three-layer backbone but outputs single scalar Q-value given a).
For intrinsic rewards, we employed three specialized networks: CIC encoder comprising a state network that maps the state vector through two 128-unit hidden layers to an n-dimensional embedding and a predictor network that takes the concatenated pair of these embeddings (size 2n), processes it through two 128-umit hidden layers, and outputs an n-dimensional prediction; an RND network comprising predictor and frozen target MIPs (each with two 128-unit hidden layers) mapping observations n-dimensional feature space, where the mean squared prediction error defines Trndi and a BeCL encoder, a three-layer 128-unit MLP that maps observations to an n--imensional skill embedding for the AnInfoNCE loss, encouraging non-overlapping skill distributions.
1.3 URLB ENVIRONMENTS
The Walker domain focuses on training a hiped constrained to a 2D vertical plane to acquire balancing and locomotion skills (Laskin al., 2021). It includes four downstream tasks: Stand, Walk, Flip, and Run. The observation space is defined as S E 24, and the action space as A E R2
The Quadruped domain involves training four-legged agent for balance and locomotion within a 3D environment. This domain includes four tasks (Figure 16): Stand, Walk, run, and flip. The observation space 78 and the action space is A R1
The Jaco domain is designed for manipulation tasks using 6-DoF robotic arm equipped with three-finger gripper. It includes four tasks: Reach Top Left. Reach Top Right, Reach Bottom Left, and Reach Bottom Right. The observation space is S R555 and the action space is A E RR.
1.4 NETWORK ARCHITECTURES FOR URLB EXPERIMENTS
This section describes the network architecture of our method. At the beginning each episode, skill vector z is sampled, where the default setting uses one-hot encoding with skill dim 16. This skill vector is concatenated with the processed observation features and used as input to the policy, value, and intrinsic reward modules.
The raw observation is first processed by four-layer convolutional encoder, where each layer has 32 channels and uses Rectified Linear Unit (RcLU) activations. The encoder's output feature map is then flattened into a latent feature vector of dimensions (repr_ dim = 32 x 35 x 35). The resulting feature vector is combined with the skill vector before being forwarded the downstream networks.
24

---

Published as a conference paper at ICLR 2026
Walker
Quadruped
Jaco
Stand
Walk
Stand
Reach Bottom Left
Right
Flip
Jump
Reach Top Left
Reach Right
Figure 16: Visualization of the downstream tasks: Walker, Quadruped, and Jaco domains.
For decision making, the actor network processes this combined observation and skill vector through trunk consisting of linear layer, layer normalization, and hyperbolic tangent (Tanh) activations. The resulting feature is passed through two fully connected layers with 1024 hidden dimensions and ReLU activations, and finally projected to the action space produce the action distribution. The critic network applies the same trunk structure to the combined representation. It then concatenates the resulting features with the action, and processes them through two additional hidden layers with 1024 dimensions each to estimate the Q-values.
The RND module constructs predictor-target architecture by copying the observation encoder. Both the predictor and the frozen target network share the same initial encoder and are extended with multilayer perceptron (MLP) composed two hidden layers of 1024 dimensions. The predictor is trained to minimize the mean-squared error relative to the target's output.
The CIC module integrates three coordinated components, which consist of state encoder that transforms both the current and next observation features into embeddings within the skill space, skill projection network that embeds the sampled skill vector, and predictor network that takes the concatenated state and next-state embeddings and transforms them into representation aligned with the skill embedding. All components use MIP with hidden layers of 1024 dimensions, and the module is optimized using contrastive predictive coding (CPC) objective to encourage alignment between state transitions and the correct skill representation. The state encoder's outputs are used to calculate the Tentropy: based kNN distances.
The BeCL module takes observation features (exeluding the skill vector) and processes them through embedding network with two hidden layers of 1024 dimensions to produce compact representa- tion. This representation is then passed through a projection head, which includes another hidden layer with 1024 dimensions and output layer, producing outputs that match the skill dimension. This layered architecture enables the module to generate embeddings that are optimized for contrastive learning, effectively encouraging skill discrimination the learned space.
During fine-tuning, we employ a Soft Actor-Critic (SAC)-based skill selector to adaptively choose skill vector given the current observation. SAC offers off-policy sample efficiency and entropy- regularized stability, which help balance exploration and exploitation. The skill selector consists of policy network and value network. The policy network maps observations discrete distribution over skills. It consists of a linear layer follwed by layer normalization and Tanh activation, then two fully connected layers with 256 hidden dimensions and ReI.U activations, producing logits over the skill space. A skill is sampled from this distribution using epsilon- -geeddy strategy. The critic network use the same input processing as the policy network. It then maps the resulting feature through two fully connected layers with 256 hidden dimensions produce Q-values for each skill. During training, the critic is updated using temporal difference learning, while the actor is optimized via cntropy-regularizze policy gradients (Haamoja al., 2018).
In the e-greedy skill selection strategy, the exploration probability e decays exponentially over time, starting from 1.0 and gradually decreasing to 0.01 with decay factor of 20000
25

---

Published as a conference paper at ICLR 2026
steps. This encourages etrly-stage exploration of diverse skills and gradually shifts toward selecting high-performing skills.
1.5 HYPERPARAMETERS
Table and Table 9 contains the hyperparameters we use. Hyperparameter values for the Maze environment were adopted directly from the EDL repository (Campos, 2020). while those for the URLB environment follow the defaults provided by the URLB codebase (Laskin & Yarats, 2025). We perform hyperparameter tuning in URLB, focusing on three key components, intrinsic reward weights (a, (), the projection probability p. We explored values in the ranges [10-5, 0.1... 3 E [10-8, 10).
p E [0.5, 1.0].
Table 8: Hyperparameter settings for URLB experiments.
Intrinsic reward hyperparameter
Walker
Quadruped
Jaco
skill dimension
16
16
16
contrastive update rate
3
3
3
temperature
0.5
0.5
0.5
alpha (a)
0.01
0.002
0.03
beta (()
10
8
0.005
projection probability (p)
0.6
0.65
0.8
Number of nearest neighbors (k)
16
16
16
Skill selector hyperparameter
Value
epsilon start
1.0
epsilon end
0.01
epsilon step
20000
learning rate
10
DDPG hyperparameter
Value
replay buffer capacity
10"
warmup frames
4000
n-step returns
mini-batch size
1024
discount (?)
0.99
learning
10
agent update frequency
critic target EMA rate (TQ) exploration stddev clip exploration stddev value number pre-training frames
0.01
0.3
0.2
10
number of fine-tuning frames
1 x 10
Table 9: Hyperparameter settings for Tree-maze experiments.
HyperParameter
Value
learning rate (r)
3e-4
discount (-)
0.99
GAE lambda
0.98
entropy lambda
0.025
hidden dim
128
temperature
0.5
alpha (a:)
0.01
beta (.9)
le-4
projection probability
knn k
16
knn clip
5e-4
epoch
50
26

---

Published as a conference paper at ICLR 2026
1.6 URLB TRAINING PIPELINE
Algorithm I Gradient Surgery
1: Given: V Liivensity V eexplooration: probability parameter of critic eriiie and a learning rate 1).
2:
if (V Ldiverityy V expplrration) < then
3:
With probability pi
4:
Ldiversity + Proju
(V Ediversity)
eeptorition
5:
Otherwise:
6:
Lexplooration Projy diverity
(V Exxploration)
7:
end if
8: Areiiic = VVVVVVV Ldiversity + Lexplorratio))
9: Beiiti f- Driii Aferitic
Algorithm 2 Unsupervised Pretraining with Intrinsic Rewards and Gradient Surgery
1: Given: number skills n, pretraining frames NeT, seed frames T, batch size N, update interval Nupdate. policy me, critic Q
2: Initialize: replay buffer < 0, timestep t +-
3: while Npr do
4: if mod Nupdate = 0 then
5: Sample skill 2, ~ Uniform(L,
6: end if
7: Collect transition (st, 04; 81+1) no(. 80, 21); p(8++1
8t, 8t)04)
8: store (st, de, S1+1, 24) in B
9:
if 2 T then
begin intrinsic-reward pretraining
10:
Sample batch {(s, a, s', 2)) B
11:
Update encoders: Minimize contrastive loss (Eq. 3), RND prediction loss, and AnInfoNCE loss (Eq. 4)
12:
13:
Compute intrinsic rewards:
Calculate Fexploration "diversity defined in Sec.
15:
Update critic & actor:
Compute gradients for diversity and exploration losses
17:
Apply Gradient Surgery (Alg. 1)
Update policy To and critic Q
19:
end if
20:
t 1+1
21:
end while
Algorithm 3 Fine-Tuning with Extrinsic Rewards and Joint Skill Selector Training
1: Given: number of finetuning frames Ner, batch size N, update interval U, critic Q pretrained policy ma, skill selector (=(= [=]]] (s
2: Initialize: replay buffer D 2- 0, timestep L+-0
3: while < NFT do
4: Observe state st
5: Select skill ,, (z(z (z) st))
6: Select action 0t ~ Ta(a 8t, 2t)
7: Execute o to obtain (81+1, rt)
8:
Store (St, az, Tt, S1+1, 2x) in if t mod U = 0 then
9:
Update both selector and agent
10:
Sample batch {(s, a, r,s', N D Update 0, z using extrinsic reward
11:
12:
end
13:
1-- (
14:
end while
27

---

Published as a conference paper at ICLR 2026
1.7 TRAINING TIME COMPARISON
Table 10 compares the wall-clock training time of AMPED against a range of baselines on the Walker, Quadruped, and Jaco domains. Notably, AMPED incurs only modest increase in runtime relative to competitive methods like CeSD and BeCL despite exceeding their downstream performance (see Figure 8). Although AMPED requires more computation than CIC (an overhead of 3-6 hours), this extra cost yields substantial performance gains over CIC's purely entropy-based exploration. Overall, these findings demonstrate that AMPED strikes a favorable balance between computational cost and empirical performance.
Table 11 reports fine-tuning times. Because AMPED (Ours) includes the SAC-based skill selector, its fine-tuning incurs modest overhead of approximately 0.06-0.11 h (4-7 min) compared to baselines such as CIC and BeCL.. future work, we aim to further optimize runtime efficiency, perhaps via more strcamlined encoder updates or low-precision training, while preserving the joint handling of exploration and diversity.
Table 10: Pretraining time (hours with decimal minutes) comparison across baselines. Results are computed over 10 random seeds and reported as mean + standard deviation.
Domain
AMPED (Ours)
CcSD
CIC
BeCL
APT
RND
DIAYN
DDPG
ComSD
Walker
13.47 0.06
22.28 1. 0.08 7.34 1 0.18 18.13 4.65
11.02 1
5.19 0.14 7.34 0.14 4.34 0.03 7.5 0.03
Quadruped
13.62 1. 0.1
23.01 1 0.0 7.6 1 0.21 13.42 2.72
11.18 1 0.01 5.45 0.02 6,96 1 1.37 4.49 1 0.05 7.74 1 0.03
Jaco
13.72 0.03
22.76 + 0.07 7.61 + 0.03 14.88 + 3,11 11.39 0.03 6.4 + 1.09 8.11 0.02 4.78 0.1 7.91 0.02
Table 11: Fine-tuning time (hours with decimal minutes) comparison across baselines. Results computed over 10 random seeds and reported as mean + standard deviation.
Domain
AMPED (Ours)
CeSD
CIC
BeCL
APT
RND
DIAYN
ComSD
Walker
0.35 1:
0.7 1: 0
0.26 d: 0
0.24 1:
0.23 1
0.36 1: 0
0.37 1. 0.01
0.25 1: 0
Quadruped 0.35 0.01
0.73 +: 0.01 0.29 +: 0.01
0.29 1: 0.01
0.26 4: 0.01
0.44 +: 0.01
0.4 = 0.01
0.28 +: 0.01
0.03
0.72 = 0.02 0.27 0.01
+
0.28
0.06
+
0.26 +
I.8 REPRODUCING BASELINES
All baseline methods were integrated from their respective open-source implementations and evaluated under our unified settings. In the Maze environment, DIAYN, CIC, and BeCL. were reproduced using the EDL repository (Campos, 2020), and ComSD was imported from its official codebase (Liu, 2025)). Since no public implementation of CcSD exists for the Maze tasks, we followed the visualizations described in the original CeSD paper.
For the URLB, we leveraged the official URLB code (Laskin & Yarats, 2025) to reproduce DIAYN, RND, and APT. CIC, BeCL, and CeSD were run using their respective public implementations (Laskin & Liu,, 2022; Yang, 2023; Bai & Yang, 2024). As ComSD) lacks an official URLB release, we reimplemented from its 2D-Maze variant, strictly adhering to the hyperparameters reported in its original publication.
Using the unmodified hyperparameters provided the official CeSD codebase and paper, we were unable to reproduce the authors' reported performance. Our analysis indicates that CeSD exhibits substantially higher variance across random seeds than competing methods. Although minor implementation evaluation differences cannot be entirely ruled out, the magnitude of the variance suggests that the observed gap is primarily attributable to CeSD's inherent instability rather than to specific experimental deviations (Table 13).
1.9 EXPERIMENTAL SETUP AND REPRODUCIBILIIT
All experiments were conducted on a Windows 11 workstation equipped with an AMD Ryzen 7700 8 8-core processor (3.80 GHz), 64GB DDR5 RAM, and NVIDIA RTX 3060 GPU (12GB GDDR6).
28

---

Published as a conference paper at ICLR 2026
Each experiment are summarized in Table 10 and Table 11.
single GPU. The detailed wall-clock time for training and fine-tuning
We implemented all experiments in Python 3.8.10, using PyTorch (v1.9.0ccul1I) as the primary deep learning framework. The DeepMind Control suite (Tassa et al., 2018) (dm-control v1.0.8) was used for environment simulation, and agent-environment communication was handled through the dm env interface (v1.6).
dm_env
J COMPARISON WITH METRA
We compare AMPED with METRA (Park et al., 2024), a metric-aware skill-based reinforcement learning method designed primarily for high-dimensional environments. Table 12 reports results URLB. Because METRA is not available URLB, we re-implemented it following the original paper, evaluating two variants: (i) Discrete METRA, which uses categorical skill variable, and (ii) Continuous METRA, which samples skills from Gaussian distribution.
In our experiments, both METRA variants underperform AMPED across all evaluated tasks, despite METRA outperforming CIC in the high-dimensional pixel-based settings reported in Park et al. (2024). This discrepancy highlights that the comparison is not entirely symmetric: METRA was developed for high-dimensional environments, where exploration is particularly challenging, whereas our evaluation focuses on the URLB control suite to isolate and validate the effect of balaneing diversity and exploration downstream task performance. Extending our framework to pixel-based domains a natural next step, and investigating how METRA's metric-aware abstraction interacts with explicit -xploration-diversttyy balancing in high-dimensional state spaces remains interesting direction for future work.
Table 12:
Performance comparison between AMPED and METRA. All results computed using 10 seeds. Across all three environments, both discrete and continuous METRA variants underperform compared to AMPED.
Domain
Task
AMPED (Ours)
Discrete Metra
Continuous Metra
Flip
674 + 105
329 + 50
373 + 64
Run
467 + 103
174 + 36
274 + 48
Walker
Stand
951 + 38
764 = 138
781 + 154
Walk
929 + 19
406 + 95
502 + 206
Sum
3021
1673
1930
Jump
720 + 32
218 + 194
183 + 148
Run
494 + 53
144 + 46
127 + 86
Quadruped Stand
906 1 67
333 + 122
240 + 127
Walk
890 + 59
153 + 65
160 + 117
Sum
3010
849
710
Re. bottom left
143 + 32
14 12
26 + 10
Re. bottom right
144 + 25
11 12
21 + 29
Jaco
Re. top left
140 + 39
9
21 - 1 14
Re. top right
154 + 46
5
24
Sum
581
39
93
K LIMITATIONS
As with any research, our approach presents several limitations that highlight opportunities for future investigation:
Better gradient conflict resolver. Although PCGrad easy to implement and powerful, it has a few limitations. First, Liu et al. (2021) demonstrates that PCGrad does not preserve the original
29

---

Published as a conference paper at ICLR 2026
objectives: instead, it merely guarantees convergence to the Pareto set. More advanced gradient conflict-resolution techniques have since been developed; future work can select the method best suited for SBRL.
Removing heuristics. Although we have introduced theory- gradient surgery to balance explo- ration and diversity of skills, still use the rule of thumb such positive hyperparameters a, 8 for + eotropy + 3r Future work should eliminate such empirical rule of thumb.
Tioral ddiversity
Inaceuracy and inefficiency of Estimators. AnInfoNCE lacks precision, future research should consider approaches that tighten the MI bound. And for entropy which has a high computational overhead, one should explore methods that are computationally efficient and capable of functioning effectively in high-dimensional state spaces. Model-based approaches, such as those using normalized flows (Ao & Li, 2022), could be potential solutions.
Better objectives. The diversity term adopted from the BeCL paper influences entropy, leading to gradient conflicts. Future research could focus on developing better objectives that maintain diversity without compromising entropy. In addition to entropy and RND based exploration, there has been lot of research going on (Ladosz et al., 2022). One may find a better way to explore more efficiently and effectively.
Balancing Other Factors Beyond Diversity and Exploration. While our work primarily focuses on diversity and exploration, other aspects are also being actively studied to improve performance. Exploring how to harmonize our method with these additional aspects could be a valuable direction for future research. For instance, recent studies, such as (Park et al., 2023), rewards states that are difficult to reach.
Fixed Number of Skills. The current model treats the number of skills fixed hyperparameters. which of course not ideal across different environments; see Figure C. While too few skills limit overall state coverage, once exploration saturates, adding more skills offers further benefit. Developing mechanisms to dynamically adjust the number of skills according to the environment's requirements could enhance flexibility and performance.
Relation Between Gradient Conflict and Environment. Our experiments indicate that the degree of gradient conflict is strongly task-dependent. For example, Figure 5 shows that the percentage of conflicting gradients is noticeably lower in the Jaco domain than in Walker or Quadruped. This suggests that in some environments, behaviors that improve exploration (c.g., moving in varied directions) might naturally also promote diversity, leading to more aligned gradients; whereas in others exploratory behavior may cause different skills to visit overlapping regions, increasing conflict with the diversity objective. A more systematic characterization of how environment structure influences exploration-diversity gradient interactions, and how to adapt AMPED to these regimes, is an interesting direction for future work.
L LLM USAGE
We used large language model (LLM) solely for language editing. Concretely, the LLM assisted with grammar and style polishing, LaTeX phrasing (c.g., equation and caption wording), and improving clarity and concision of author- written text. The LLM was not used to generate ideas, design algorithms, select hyperparameters, run experiments, analyze data, create figures/tables, write code, produce mathematical results.
30

---

Published as a conference paper at ICLR 2026
M VISUALIZATION OF SKILLS
Figure illustrates the skills acquired during the pretraining stage for each environment.
Walker: Getting up from the ground
Walker: Stepping forward
Walker: Backward somersault
Quadruped: Recovers from an upside-down position to stand upright
Quadruped: Backward somersault
Quadruped: Clockwise rotational movement
Jaco: Reaches left grasp the target
Jaco: Reaches toward the right area and grasps the object
Jaco: Upward lifting motion while attempting to grasp the target
Figure 17: Representative skills learned by our method. Walker skills include rising from supine position, stepping forward, and performing backward somersault. Quadruped skills demonstrate self-righting, acrobatic flips, and rotational maneuvers. Jaco skills capture precise arm motions such as leftward reaching, rightward grasping, and upward lifting toward a target.
31

---

Published as a conference paper at ICLR 2026
N
NUMERICAL RESULTS
N.1 PER-TASK EPISODE RETURNS ON URLB DOMAINS
Table 13: Performance comparison with baselines. Numerical results corresponding to Figure Results are computed over 10 random seeds and reported as mean + standard deviation. The best result shown bold, and the second-best is underlined.
DIAYN
Flip
686 133
623
637
108
625 +
729
129
483
329
39
531
488 57
Run
49
454
435
542
183
327
115
Walker
sum
3036
2720
2904
2831
3112
2538
1662
2499
2592
Jump
529
720
337
+
390
Quadruped
+
+
Walk
816 +: 116
562 630
183
635
75% 1: 192
571
1: 404
1: 200
209 :: 396 + 182
sum
2824
2334
2345
2337
2767
2488
2001
1308
2023
left
1:
1:
Rc. bottom right
140 1:00::20
134 7
152 23
140
126
25
115 -
+
115 -
111 41
Jaco
Re. top
130 1300::35 38
175
137 21
+
124
22
97
30
22 +
101
-
126
Re. top right
146 49
97 1: 29
149
1: 19
116 1:
113
4: 25
122 1 30
1: 12
87 53
121 15
sum
555
542
573
527
483
435
76
436
484
Total
Sumn
6415
5596
5822
5705
6362
5461
3747
4243
5099
Median
IQM
Mean
Optimality Gap
Jump
0k
25k
50k Steps
75k
100k
0K
25k
50kk
75k
100k
0k
23k
5.0k Steps
75k
100k
Steps
Steps
0.8
0.6
Run
0.0
100k
75c
100k
75k
50x
100k
Steps
Steps
Sleps
Steps
1.0
0.6
Stand
0.4
0k
25k
3.k Steps
75k
100k
0k
25k
50k
75k
100k
06
25k
50k
75k
100k
0k
23k
50k Steps
75k
100k
Steps
Steps
1.0
0.8
Walk
0.2
0k
25k
30k Steps
75k
100k
30k Steps
75k
100k
25k
50k Steps
75k 100k
0k
23k
5.0k 100k
Stegs
AMPED
CIC
BeCL
Figure 18: Performance on Walker tasks normalized by experts' scores. Results show Median, IQM, Mean, and Optimality Gap computed over 10 random seeds. Shaded regions represent 95% confidence intervals.
32

---

Published as a conference paper at ICLR 2026
Median
IQM
Mean
Optimality Gap
Jump
100k
100k
75k
Steps
Steps
0.8
0.0
0.0
0.6
0.0
Ran
0.4
0.0
0.0
0.0
0.0
0K
25K
75k
100k
00
30k
100k
25k
50k
100k
25K
75k
100k
Steps
Steps
Steps
Steps
1.0
1.0
1.0
1.0
0.8
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0k
25k
5.0 Steps
75k
100k
05
256
50%
75k
100k
25k
50%
75k
100k
0%
25k
50c
75k
100K
Steps
Steps
Steps
1.0
1.0
1.0
0.8
0.8
0.8
0.9
0.6
Walk
0.2
0.2
0.2
0.0
0.0
0.0
0.0
Steps
AMPED
CIC
BeCL
Figure 19: Performance Quadruped tasks normalized by experts' scores. Results show Median, IQM, Mcan, and Optimality Gap computed over 10 random seeds. Shaded regions represent 95% % confidence intervals.
Median
IQM
Mcan
Optimality Gap
Rc.
0.6
0.6
0.6
0.0
bottom
0.4
0.4
0.4
0.4
left
0.2
0.0
0k
25k
30k Steps
75k
100k
0.2
02
25k
30k Steps
75k
100k
0.2
0.0
0k
25
50k Steps
75k
100k
0.2
0.0
0K
23k
30k Steps
75k
100k
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
bottoen
0.4
0.4
0.4
0.4
right
0.0
0.0
Steps
Steps
1.0
0.8
0.8
0.8
Re.
0.6
0.6
0.6
0.6
kn
0.2
0.2
0.2
0.2
0k
25k
50k Steps
75k
100k
02
50k
75k
100k
25k
50k Steps
75k
100k
0.0
25k
50k
75k
100k
Steps
Steps
1.0
1.0
1.0
1.0
0.8
0.8
top
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
right
0.2
0.0
ok
25k
50k
75k
100k
0.0
0k
25k
50k Steps
75k.
100k
0.0
25k
50k Steps
75k
100k
0.0
25k
50k
75k.
100k
Steps
Steps
AMPED
CIC
BeCL
Figure 20: Performance on Jaco tasks normalized by experts' scores. Results show Median, IQM, Mean, and Optimality Gap computed over 10 random seeds. Shaded regions represent 95% confidence intervals.
We report per-task episode returns (mean + standard devition over 10 seeds) following the evaluation protocol of Agarwal et al. (2021). All methods pretrained for 2M steps with only intrinsic rewards,
33

---

Published as a conference paper at ICLR 2026
then finetuned for 100k steps each downstream task by adding the extrinsic reward. Table 13 presents results on the Walker, Quadruped, and Jaco domains.
AMPED achieves at least second best performance in almost every discipline, and has the highest total sum, which confirms that our method consistently perfoms well on different tasks (Table 13). More concretely, AMPED achieves cumulative total sum of 6415, which is the highest among all methods. The next best is APT with 6362, trailing AMPED by 53 points, and CIC comes third at 5822, well behind by 593 points. These per-task breakdowns confirm that AMPED's s joint handling of entropy, RND, and diversity objectives delivers consistently strong performance across a diverse set of URLB challenges. Although CeSD and ComSD also aim balance diversity and exploration. AMPED outperforms both on all but one task (Re., top left), demonstrating that our unified objective formulation more effective.
We additionally include time-series plots of downstream fine-tuning returns for AMPED, CIC, and BeCL (Figure 18, Figure 19, Figure 20). In both Walker and Quadruped, AMPED consistently surpasses CIC and BeCL throughout training, whereas Jaco its performance is comparable to two methods, which focus exclusively on exploration or diversity. These results underscore the importance of jointly balancing both objectives.
N.2 ABLATION STUDY ON PROJECTION RATIO
As detailed in Section 4.3, AMPED's s balanced projection ratio mitigates gradient conflicts and improves skill learning across diverse environments. The full numerical results corresponding to Figure 9 are provided in Table 14. The projection ratios used for AMPED (Ours) are listed in Appendix I.5.
Table 14: Performance under different projection-ratio settings (p). Numerical results corre- sponding to Figure 9. AMPED (Ours) uses the default projection ratio, chosen separately for each environment. Results are computed over three random seeds and reported as mean standard devia- tion, except AMPED, which uses 10 seeds. The best result is shown in bold, and the second-best is underlined.
Domain
Task
AMPED (Ours)
= 0.0
p= 1.0
Flip
674 105
628 + 55
606 37
Run
467 103
427 + 44
533 71
Walker
Stand
951 = 38
949 + 3
931 + 12
Walk
929 1: 19
939 1 13
828 1: 73
Sum
3021
2943
2898
Jump
720
706 +
661 34
Run
494 1 56
483 1
501 1: 29
Quadruped
Stand
906 + 71
859 + 109
905 + 4
Walk
890 + 62
613 + 183
626 + 129
Sum
3010
2661
2693
Re. bottom left
133 1
10
Re. bottom right
144 + 27
108 + 53
140 17
Jaco
Re. top left
140 - 41
84 + 37
126 + 29
Re. top right
154 1: 49
103 1: 32
139 1: 34
Sum
581
428
531
34

---

Published as a conference paper at ICLR 2026
N.3 ABLATION STUDY ON NUMBER OF SKILLS
As discussed in Section 4.3, the number of skills plays an important role in downstream performance. The full numerical results corresponding Figure 10 summarized in Table 15. AMPED (Ours) uses 16 skills in all domains.
Table 15:
Performance under different skill dimension setting. Results computed over three random seeds and reported as mean + standard deviation, except AMPED, which uses 10 seeds.
Domain
Task
AMPED (Ours)
Skill dimension 8
Skill dimension 32
Flip
674 + 105
552 + 86
680 + 196
Run
467 103
468 84
511 + 32
Walker
Stand
951 + 38
942 13
907 = 83
Walk
929 + 19
929 + 25
940 + 22
Sum
3021
2891
3038
Jump
720 + 32
493 + 182
386 + 116
Run
494 + 53
380 + 73
247 = 49
Quadruped Stand
906 + 67
705 202
602 + 203
Walk
890 + 59
488 + 201
388 + 190
Sum
3010
2066
1623
Rc. bottom left
143 + 32
137 38
142 +
Re. bottom right
144 25
153 + 22
124 + 15
Jaco
Re. top left
140 + 39
117 49
108 +
Re. top right
154 + 46
148 + 15
132 + 44
Sum
581
555
506
35


Reviews:
[{"review_id": "4bs8s5vxj5", "reviewer": "Reviewer_1a1c", "summary": "The authors propose AMPED, a skill-based reinforcement learning framework that balances exploration and skill diversity objectives during pretraining. The method treats these as competing objectives in a multi-objective RL setting and applies gradient surgery (PCGrad) to resolve conflicts between diversity gradients (from AnInfoNCE) and explroation gradients (from particle-based entropy and RND). Those skills are exploited in the following fine-tuning phase, in which a SAC-based skill selector adaptively chooses skills for downstream tasks.", "questions": "Table 10 shows skill dimension is fixed at 16 for all URLB experiments. How do diversity and performance vary with different numbers of skills (e.g., 4, 8, 32)? This would help validate whether increased diversity consistently improves results.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is very clear and easy to follow, and the proposed method is well-motivated.\n- Strong results on several tasks from the URLB benchmark with comparisons against 7 baselines, with consistent improvements.\n- Table 2 effectively demonstrates that each component (RND, AnInfoNCE, gradient surgery, skill selector) contributes meaningfully to performance.\n- Figure 5 (Tree Maze) provides intuitive evidence that AMPED achieves both skill separation and state coverage where baselines fail at one or both.", "weaknesses": "Section 4.4 claims are problematic in multiple ways:\n- Line 200 claims \"empirical validation appears in Section 4.4,\" but Theorem 1 assumes sampling trajectories from the **optimal** policy (requiring access to it), while Section 4.4 measures convergence speed to the optimal policy during fine-tuning. Theorem 1 establishes that diversity reduces sample complexity for skill identification given optimal trajectories—not that diversity accelerates convergence during learning.\n- The statement \"Combined with higher final returns, this indicates faster convergence of AMPED\" is inaccurate, as other variants finish converging before (e.g. BeCL actually converges 90% faster on Flip, and many other comparisons are not statistically significant). The paper should also report the **rate of improvement** (reward gained per step) rather than steps-to-threshold, especially when final returns differ across methods.\n- The logic \"AMPED has higher diversity than CIC/BeCL\" + \"AMPED improves faster\" $\\Rightarrow$ \"diversity causes faster improvement\" seems flawed. Other algorithmic differences (objective functions, gradient surgery, skill selector) could explain the improvement rate. To isolate diversity's effect, ablations should vary diversity within AMPED itself (e.g., different numbers of skills or projection ratios $p$), not compare across different methods.\n- Section 4.4 only evaluates Walker tasks, making it difficult to establish a general relationship between diversity and rate of improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "odaWpocfVX", "reviewer": "Reviewer_3YZa", "summary": "This paper introduces a new framework for skill-based reinforcement learning called AMPED. During the pre-training phase, the method uses a \"gradient-surgery projection\" to mitigate conflicting gradient components. During the fine-tuning phase, it employs an \"adaptive skill selector\" to leverage the learned skills for downstream tasks. Experiments demonstrate that this method surpasses baselines on the URLB and Maze benchmarks, and proves that greater diversity.", "questions": "1. The authors state that METRA was omitted because it does not exhibit performance improvements on the URLB relative to CeSD. However, I noted in the paper[1] that METRA outperforms CIC on some tasks. Given that CIC is a relatively strong performer in the authors' own experiments, could the authors please add supplementary experiments comparing against METRA?\n\n2. Could you provide more comparisons and analysis of the algorithm's efficiency? For instance, by plotting learning curves that show the evaluation metrics as a function of training steps?\n\n[1] METRA: SCALABLE UNSUPERVISED RL WITH METRIC-AWARE ABSTRACTION", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The experimental analysis is thorough. In particular, extensive ablation studies are provided to demonstrate the effectiveness of each component in different scenarios.\n\n2. The paper is exceptionally clear and logically fluent. The progression from the problem definition in the Introduction, to the Methods section, and then to the Experimental Analysis is easy to follow. Furthermore, the Appendix provides extremely detailed further analysis, enhancing comprehensibility.\n\n3. The paper models this problem as a \"gradient conflict\" in multi-objective optimization and introduces \"gradient-surgery projection\" to resolve it. This is a very insightful and novel approach.", "weaknesses": "1. I find that the method presented in the paper largely consists of combining existing techniques, and it lacks significant methodological innovation.\n\n2. Directly using SAC to learn a policy over the skill repertoire (i.e., changing the prior z) seems feasible. However, the paper does not couple the two RL training processes well, which may lead to unstable convergence and, as noted in Table 6, instances of ineffectiveness.\n\n3. The experimental environments are relatively simple, with experiments primarily compared on the Maze and 12 URLB downstream tasks. I believe the contribution to practical applications (e.g., pixel-based environments) is minimal.\n\n4. The algorithmic complexity is high (integrating CIC, RND, SAC, etc.), making the training process relatively heavy. Moreover, the efficiency analysis provided is not intuitive enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H5gWXIQmkU", "reviewer": "Reviewer_8RFz", "summary": "This work proposes an adaptive training method to balance exploration and diversity in the application of unsupervised reinforcement learning (URL) for skill discovery. The authors argue that optimizing exploration and diversity in skill discovery tasks can be contradictory. Therefore, they design the algorithm from a multi-objective optimization perspective, defining \"enhancing skill exploration\" and \"improving skill diversity\" as two distinct objectives for the URL algorithm. They then propose using a gradient clipping approach to balance these two optimization goals. The proposed method demonstrates superior performance compared to existing baselines across multiple benchmarks. Furthermore, the paper includes comprehensive ablation studies that experimentally validate the necessity and correctness of the various components within the proposed algorithm.", "questions": "- In the proof of Theorem 1, the authors state that a larger $\\delta$ increases $\\Delta$ and thus reduces the required sample size. However, Inequality 3 only provides a lower bound for $n$ and does not explicitly demonstrate that increased diversity reduces the necessary sample size. Could the authors clarify this point?\n- What are the exact calculation of median, IQM (is it Interquartile Mean?), and mean scores and optimality gap in Figure6?\n- Table 1 shows the percentage of conflicts. What does the corresponding learning curve look like for these conflicts?\n- How does the dimension of $z$ affect the final policy performance?\n- The presentation of the conflicts during the skill learning process is not intuitive. Could the authors make it clearer, perhaps by reporting the curve of gradient similarity changes?\n- Is the gradient similarity related to the type of task? For instance, in some scenarios, the conflict between exploration and diversity might be small, while in others it could be large.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is easy to follow.\n- The results of the ablation study are comprehensive.", "weaknesses": "- Some content is unclearly explained, such as the specific form of $\\rho$ and the meaning of the several metrics in Figure 6.\n- The proof of Theorem 1 is currently difficult to ascertain as valid (see Questions for details).\n- The experimental settings are too simple, primarily consisting of basic maze environments.\n- The presentation of the conflicts during the skill learning process is not intuitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eLRYOzeK8k", "reviewer": "Reviewer_B5oH", "summary": "This paper studies unsupervised skill learning and proposed AMPED, a novel method that aims to learn skills that are both diverse and explore the state space well. To do this, AMPED relies on an intrinsic reward based on an entropy bonus combined with RND for exploration, and an intrinsic reward based on contrastive learning for incentivizing the skills to be diverse. Finally, AMPED uses a gradient-surgery projection procedure to balance the exploration and diversity gradients and avoid gradient conflict. On the URLB benchmark, they find their method to outperform all baselines in terms of mean, IQM, and optimality gap. The authors also provide ablation studies to test the importance of the entropy bonuses, the RND bonus, the Anisotropic InfoNCE reward, the gradient surgery procedure, and the skill selector.", "questions": "1. Page 7: “In order to ensure a fair comparison, we fine-tuned all methods under identical conditions without using the skill selector.” → how does that work? Every skill gets fine-tuned towards the same task reward? Does this preserve the skill behavior? Do the skills change a lot during fine-tuning?\n2. Section 3.1: \"and ρ⋆ ∈∆(S^H) be a corresponding state distribution.\" -> If this is a state distribution, shouldn't it just be ∆(S) instead of ∆(S^H)? The latter seems like a distribution over trajectories. This also confuses me later where it says \"Draw n i.i.d. trajectories from optimal policy S(1),...,S(n) ∼ρ⋆\". This seems like drawing n states instead of n trajectories?\n3. Could the authors expand a bit on exactly why it's necessary to have both RND and entropy exploration? There's some discussion about it on page 5 but it wasn't super clear to me.\n4. Page 5: \"However, if the supports of different skill coverages become disjoint, the diversity loss Ldiversity no longer enforces inter-skill distributional separation, which can lead to skill clustering rather than promoting broad coverage of the state space.\" -> Could the authors provide some clarification on this statement? I couldn't really follow what exactly is meant here.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Evaluation was done over 10 random seeds and reports results according to recommendations from the Rliable framework.\n2. Strong results: outperforms all baselines in terms of mean, IQM, and optimality gap.\n3. The motivation for having explicit exploration and diversity rewards is clear.\n4. Figure 4 provides a very helpful visualization of the gradient surgery procedure.\n5. Figure 5 provides a good qualitative visualization of the exploration performance and the distinguishabiltiy of the learned skills for all considered methods in a toy maze.", "weaknesses": "1. Part 2 of Figure 2 is nothing new: this seems to be a standard HRL setup for skill discovery methods? In general, the paper emphasizes the benefits of the skill selector, while to me this just seems like HRL over the learned skills which has been done in prior work already (see METRA or CSF). Could the authors comment on this? Is there any differences or contributions that I’m missing?\n2. Page 4: “is the discounted total state distribution” → do the authors mean the discounted state occupancy measure here? If so, I’d recommend replacing it with that since it’s the canonical terminology for this.\n3. Section 4.4: “… this indicates faster convergence of AMPED” → based on Table 3, I disagree with the authors on this claim. It appears to me all the numbers reported fall within each other’s error bars, questioning the statistical significance. Could the authors comment on this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Uq8mOVNdV", "reviewer": "Reviewer_mrH4", "summary": "This paper introduces \"AMPED\", a method for unsupervised skill learning and efficient adaptation to downstream tasks.\n\n**Main contribution:** During pretraining, the method explicitly handles two potentially conflicting objectives:  \n1. Achieving wide exploration of the state space  \n2. Achieving a diverse distribution of skills  \n\nThe pretraining method relies on gradient surgery to ensure that conflicts between the gradients for the two objectives are resolved, thereby avoiding that the competing objectives negatively affect the pretraining process.\n\nAfter pretraining, the downstream task can efficiently be solved by exploiting the skills discovered during pretraining.  \nFor this, AMPED learns a selector over the skills to maximize the downstream reward, which allows for more efficient adaptation compared to randomly sampling or fine-tuning skills.\n\nThe method is evaluated against a large number of unsupervised skill discovery baselines, on both didactic maze environments and more challenging continuous control tasks, all showing strong performance gains of AMPED compared to prior works.\nAn ablation over AMPED's components helps understanding the importance of each.", "questions": "As part of the ablation study, lines 425–431 discuss using different projection ratios. Can you elaborate a bit more on what is actually meant by *AMPED (ours)* here? Is it the procedure outlined in lines 287–295?  \n  If yes, I think it would improve clarity to make that connection more explicit in the main text.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear: conflicting objectives can deteriorate performance during pretraining.  \n- The paper is well written and easy to follow. Key mathematical objects and issues are presented and explained clearly.  \n- The paper formally connects the diversity of the pretrained skills with the number of samples needed to discover the optimal skill during the downstream adaptation step.  \n- The experiments include a wide range of representative baselines, and the evaluation environments feature both a didactic 2D maze example and challenging URLB environments.  \n- The results are strong, convincing, and consistent with the central claims of the paper.", "weaknesses": "- The key insights are already well understood, and the novelty is somewhat limited. The paper’s key insight, line 053, that conflicts can arise between gradients for different objectives, is well understood in the multi-objective RL and optimization literature, and thus not a strong novel finding.  \n\n- To the best of my knowledge, the paper does not introduce fundamentally new approaches to solving the challenges in multi-objective optimization, but rather combines existing techniques and applies them to unsupervised skill discovery and downstream adaptation. The contribution is still valuable, but the paper is relatively applied in nature, IMO.  \n- Without digging into the related works, it is unclear whether estimating the mutual information with AInfoNCE is a contribution of the authors. If so, I acknowledge it as a strength, slightly weakening my limited-novelty critique.  \n- The learned hierarchical selector during fine-tuning is not conceptually different from prior works such as DIAYN, which also features a hierarchical selector over discovered skills.  \n\n**Clarity:** Lines 300–304 do not sufficiently explain how the skill selector is learned exactly, and what skill fine-tuning entails. Please elaborate and explain this more clearly.  \n  - Is “fine-tuning” used interchangeably with learning the skill selector?  \n  - Are the skills themselves being fine-tuned to maximize the downstream reward?  \n  This needs to be made more explicit. Consider describing clearly which parameters of your architecture are updated during the adaptive skill selection step, and how.  \n\n**Nitpicks:**\n- Lines 183–184: $z$ is defined as a skill, but $z^\\ast$ is defined as a best policy, which is confusing. Does $z^\\ast$ always induce the optimal state visitation distribution? What if that \\$z^\\ast$ is given to a suboptimal policy that cannot execute the optimal skill well?  \n- Lines 251–252: I assume (1) and (2) are indices of different skills? The wording can be understood like they belong to the same skill.  \n- Line 86 mentions “statistically significant results,” but no formal hypothesis test ($p < 0.05$) is performed. Consider changing the wording to “consistent improvements” or similar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)