Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment
Authors: Jinwoo Choi
Seung-Woo Seo
Venue: ICLR 2025 Poster (2025)
Keywords: Skill Learning
Hierarchical Reinforcement Learning
Primary area: reinforcement learning

Abstract: Reinforcement learning (RL) has made significant progress in various domains, but scaling it to long-horizon tasks with complex decision-making remains challenging. Skill learning attempts to address this by abstracting actions into higher-level behaviors. However, current approaches often fail to recognize semantically similar behaviors as the same skill and use fixed skill lengths, limiting flexibility and generalization. To address this, we propose Dynamic Contrastive Skill Learning (DCSL), a novel framework that redefines skill representation and learning. DCSL introduces three key ideas: state-transition based skill definition, skill similarity function learning, and dynamic skill length adjustment. By focusing on state transitions and leveraging contrastive learning, DCSL effectively captures the semantic context of behaviors and adapts skill lengths to match the appropriate temporal extent of behaviors. Our approach enables more flexible and adaptive skill extraction, particularly in complex or noisy datasets, and demonstrates competitive performance compared to existing methods in task completion and efficiency.

Published as a conference paper at ICLR 2025
DYNAMIC CONTRASTIVE SKILL LEARNING WITH STATE-TRANSITION BASED SKILL CLUSTERING AND DYNAMIC LENGTH AD- JUSTMENT
Jinwoo Choi, Seung- Woo Seo" Department of Electrical and Computer Engineering Scoul National University
Scoul, South Korea
[w1 sdn sd39500 sseo snu. ac.kr
ABSTRACT
Reinforcement learning (RL) has made significant progress in various domains, but scaling it to long-horizon tasks with complex decision making remains chal- lenging. Skill learning attempts to address this by abstracting actions into higher- level behaviors. However, current approaches often fail to recognize semantically similar behaviors as the same skill and use fixed skill lengths, limiting flexibil- ity and generalization. To address this, we propose Dynamic Contrastive Skill Learning (DCSL), a novel framework that redefines skill representation and learn- ing. DCSL introduces three key ideas: state-trane tratettranitionn based skill representation, skill similarity function learning, and dynamic skill length adjustment. By fo- cusing on state transitions and leveraging contrastive learning, DCSL effectively captures the semantic context of behaviors and adapts skill lengths to match the appropriate temporal extent of behaviors. Our approach enables more flexible and adaptive skill extraction, particularly in complex or noisy datasets, and demon- strates competitive performance compared existing methods in task completion and efficiency.
INTRODUCTION
Reinforcement learning (RL) has advanced significantly in various domains (Vinyals et al. (2019). Silver et al. (2016)) yet scaling it to long-horizon tasks with complex decision-maken making remains challenging Aulac-Armld et al. (2019) ). Skill learning from datasets ((ynchh al. (200 Sharma jet al. (201)). Freed et al. (2023), Shi et al. (2022), Hao et al. (2024)) addresses these challenges by abstracting action sequences into higher-level behaviors, simplitying decision-making and improv-
ing efficiency in long-horizon tasks.
Despite its potential, prior supervised skill-leaming methods (Freed et al. (0233 Shi al. (022) face significant limitations. First, action sequence encoding methods often struggle to identify se- mantically similar behaviors as the same skill when there are variations in the action sequences. As illustrated in Fig. [ the actions required for 'grabbing the object' differ based on the object's position, leading to the learning of these behaviors distinct skills. This limitation requires large skill dimension space to accommodate various skills (Pertsch et al. (202I)). which in turn creates inefficiency when searching for the optimal combination of skills for new tasks. Since skills are stored without considering their semantic context, if a small skill dimension space is given, a di- mensional collapse (Ting et al.] (202T) may occur where different behaviors are mapped to similar representations in the skill space.
Second, fixed skill lengths fail to reflect the varying durations real-world behaviors. For instance, in pick-drop and pice data, if the length of the 'pick' behavior length, the model fails to learn pick, drop, and move as individual skills, as humans would. While
is shorter than the fixed skill
"Corresooniigg author

---

Published as a conference paper at ICLR 2025
Skill Similarity
state
skill
state
Dalasel: Grab abject located Pick up
Dataset: Grab object located Pick
Cluster Similar Behaviors: 'Grab Object'
Adjust Skill Length: 'Pick up', 'Drop',
Figure 1: Illustration of DCSL's key ideas. Previous methods recognized 'grab object' as different skills based on object position, while DCSL clusters these into unified skill using learned skill similarity. DCSL overcomes the limitation of fixed-length skills by dynamically adjusting skill lengths, enabling the recognition of shorter actions like 'Pick up" as independent skills.
fixed skill lengths have limitations, prior research using datasets of task-agnostic but meaningful be- haviors has shown that learned skills still retain semantic content. However, this approach struggles with more complex or noisy data, where irrelevant or meaningless actions are included. In such scenarios, the ability adapt skill lengths becomes crucial.
To address these challenges, we propose Dynamic Contrastive Skill Learning (DCSL), a novel framework that offers new perspective on skill representation. DCSL introduces three key ideas: state-transition based skill representation, skill funntion learning. and dynamic skill length adjustment. Unlike traditional methods that define skills based on action sequences, DCSI. focuses on state transitions, enabling more effective capture of the semantic context of behaviors. Through contrastive learning (Oord et al. (2018), DCSL learns a skill similarity function that clusters se- mantically similar behaviors into the same skill, promoting more meaningful skill representations. DCSL leverages this function to dynamically adjust skill lengths based on observed state transitions, allowing for more flexible and adaptive skill extraction, particularly in noisy datasets. By combin- ing these approaches, DCSL overcomes the limitations of existing methods, enabling effective skill learning that adapts diverse environments and task structures.
In summary, our framework redefines skill representation and learning through three key contribu- tions:
We propose a state-transition based skill representation that captures the semantic context of behaviors more effectively than action sequence encoding. This approach enables the identification of similar skills even when the underlying action sequences differ.
We introduce skill similarity function learned through contrastive learning, which clusters semantically similar behaviors into the similar skill embeddings. This method effectively differentiates between states belonging to the same skill and those that do not.
We develop dynamic skill length adjustment method based observed state transitions. Using the learned skill similarity function, our approach identifies high-similarity state tran- sitions within the dataset and relabels skill lengths accordingly, allowing for more flexible and adaptive skill extraction in complex noisy datasets.
2 RELATED WORK
Skill Learning from Dataset Skill learning from datasets has emerged as a promising approach to improve efficiency and generalization in Long-horizon reinforcement learning tasks. Foundational works like Pertsch et al. (202I) learned latent skill spaces from offline datasets, showing signifi- cant improvements in guiding exploration and task completion in novel environments. Lynch et al.
2

---

Published as a conference paper at ICLR 2025
2020) leveraged unlabeled play data to learn goal-conditioned policies in latent plan space, while Pertsch et al. (2022) explored cross cross-domain skill transfer. In multi-task learning, Yoo et al. (2022) applied skill regularization and data augmentation to handle diverse offline datasets. Model-basee approaches (Shi et al. 2022), Freed et al. (2023)) integrated skill dynamics with planning to im- prove long-term task planning without additional interaction. Existing skill learning methods lack flexibility due to their use of fixed-length action sequences, and struggle to recognize semantically similar behaviors. To address these limitations, our framework leverages contrastive learning to define skills based on state transitions and introduces technique for dynamically adjusting skill lengths, enabling more flexible and adaptive skill extraction.
Contrastive Learning in Reinforcement Learning Contrastive learning has gained significant traction in RL for enhancing state representations and improving sample efficiency. These methods typically employ contrastive losses to prioritize task-relevant features while disregarding irrelevant details. In the context of general state representation learning. Laskin et al. (2020) incorporated con- trastive learning with off-policy RL algorithms, leveraging data augmentation and contrastive losses to extract meaningful representations from raw pixel inputs. For goal-conditioned RL, several works (Eysenbach al. (2020), |Eysenbach et al. (2022), Zheng et al. (2023) ) have reframed goal achieve- ment as a density estimation problem, using classifiers to predict future state distributions without explicit reward functions. In the domain of unsupervised skill discovery, Yang al. (2023) applied multi-view contrastive learning to foster greater similarity within the same skill while maintaining diversity across different skills. Our DCSL framework builds upon these ideas by applying con- trastive learning specifically to skill extraction from datasets. Unlike previous methods, we uniquely integrate dynamic skill length adjustment using learned contrastive representations. addressing the limitations of fixed-length approaches and enhancing adaptability across diverse tasks.
Dynamic Length Adjustment in Temporal Abstractions Dynamic length adjustment in tem- poral abstractions enhances RL by enabling more flexible and hierarchical decision decision-making across variable time scales. The Options framework (Sutton et al. ( (1999)) laid the groundwork for tem- poral abstraction in reinforcement learning. Building on this, |aaco et al. (2017) proposed the Option-Crite CiitonCCiiti architecture, which learns options end-to-end with differentiable option policies and termination functions. (Shankar & Gupta (2020) introduced method using temporal variational inference to discover skills from demonstrations in an unsupervised manner. Further contributions include the Harutyunyan et al. (2019). which learns option termination conditions, and the Diang et al. (2022). J, which addresses the underspecification problem in skill learning by combining maxi- mum likelihood objectives with a penalty skill description length. While these approaches have significantly advanced temporal abstraction in RL, DCSL introduces a novel perspective. Instead of relying on learned termination functions, DCSL employs a skill similarity measure to dynamically adjust skill lengths. This approach allows DCSL to capture more meaningful and adaptable skill representations.
3 PROBLEM FORMULATION AND PRELIMINARIES
3.1 PROBLEM FORMULATION
In this work, we consider skill learning from unlabeled offline data, where the agent has access to large, task-agnostic dataset D {TI, 7,,,,,,..,,,N J. Each trajectory 9; {(*t,, (t)) consists of sequences of states 8t E S and actions 0t E A, without any task-speeific reward labels. We define skill as latent vector E Z that encodes temporally extended behavior, capable of generating a sequence of actions given an initial state. Our aim is to learn skills that encapsulate diverse behaviors, which can later be adapted to solve new tasks efficiently. Each task is modeled as a Markov decision process (MDP) defined by the tuple A, P,r, P. 7), representing states, actions, transition probabilities, rewards, initial state distribution, and discount factor.
3.2 SKILL-BASED RL
Skill-based RL aims to improve sample efficiency and generalization in long- tasks by ab- stracting low-level actions into higher-level skills. SPiRL (Pertsch et al. (202)) h) is framework that leverages prior data to learn a skill embedding space and a skill prior. SPiRL consists of three main components: skill encoder g(2aa), la), a skill decoder n rraas, (s, 2), and skill prior pa (=]s). Here, bolded
3

---

Published as a conference paper at ICLR 2025
denotes sequence of actions ( where H is skill length. The skill encoder and skill decoder are trained through a Variational Autoencoder (VAE), while the skill prior is learned by modeling the distribution of from the data. During downstream task learning, the skill prior guides exploration by regularizing the policy towards skills that are likely under the prior with a weighting coefficient
T Σ
"t (2|8) arg max B
This approach enables efficient learning of complex behaviors by leveraging the structure of previ- ously seen tasks encoded in the skill prior.
3.3 CONTRASTIVE LEARNING FOR REPRESENTATION LEARNING
Contrastive learning is self supervised technique that learns discriminative representations by con- trasting positive and negative sample pairs. Initially developed for computer vision (Chen et al. 22020). et al. (0200) and natural language processing (Radford et al. (202T). ), has gained trac- tion in RL for its ability to extract useful features from unlabeled data (Eysenbach et al. (2022) )
Zheng et al. (2023)).
). The objective is to Iearn representations that bring positive (similar) samples closer together and push negative (dissimilar) samples apart.
In this framework, given two inputs, u (anchor) and v (positive or negative), the model maximizes the similarity between u and positive samples wt while minimizing the similarity with negative samples e The contrastive loss function (Ma & Collins 2018)) is:
Leontrastive = -E(u,t) [log (f )]] + (u,)) [log(1 - (f " [))]]
(2)
where fu, v) is similarity function, and (.) is the sigmoid function. Positive pairs (u, wt) are drawn from the joint distribution, while negative pairs (u, are sampled from the product of marginals.
PROPOSED METHOD
This section introduces the Dynamic Contrastive Skill Learning (DCSL) framework, explaining its architecture and core components. DCSL aims to cluster semantically similar behaviors as the same skill and adjust skill lengths dynamically. The framework consists of three main components: skill learning through contrastive learning, skill length relabeling, and downstream task learning. An overview of the entire DCSL. framework is presented Fig. The following subsections explore each component in depth.
4.1 EXTRACTING SKILLS WITH CONTRASTIVE LEARNING
This section explains the process of skill learning using contrastive learning. Our goal is to learn skills such that similar behaviors
clustered into single skill. First, we label initial skill length H, for each {(s, az) ) in the given dataset D. This is defined as the length of the skill when the initial state of the skill is 8t. The initial skill lengths are
the same value.
Skill Embedding We select four key states from each skill duration: the initial state 8t, the terminal state S1+H1-1, and two randomly sampled intermediate states St+a and 8146 (where t < 8 ++ a < 1 1++ < t + H, 1). This approach provides snapshot of the skill's progres- sion in unified representation, enabling more effective representation of the overall trajectory and behavioral context. The selected states & = {si, Stta, $146, -11 are encoded into skill embedding ž using an LSTM-based encoder qe, (2(5). The loss function for skill embedding is:
Lembesding = (( (LEg,, (2|3) log no_ (1,, - B- Paa( (x13)llz))
+ Asp DRL ( (=)))) (=13)) |pa, (z]s))]
4

---

Published as a conference paper at ICLR 2025
Extracting Skill
Skill Relabeling
Downstream RL
Environment
Skill Encoder
Seil Similarity
(x13)
2s)
Relabel Skill Length
x (z|s)
Skill Target Predictorl
Task Policy
T(5p(5,2)
(a)ss)
Skill Prior
[a]s, 2)
Still Policy
p(2a))
Skill Policy
Figure 2: The overall architecture of our framework, including skill extraction, skill length relabel- ing, and the application of skills to downstream tasks. Our method samples four states to encode skills, with red-bordered states representing negative samples. In the relabeling process, green bor- ders indicate the skill initial state, and yellow borders denote the skill terminate state. The learned skills are then utilized in downstream RL tasks.
Here, ā represents the actions corresponding cients, and 0g 0z. and Op are trainable parameters. Additionally, sg denotes the stop gradient oper- ator. The prior p(z) is defined as tanh transformed standard Gaussian distribution, represented as tanh(N((., 1)). The skill decoder (ā) 2) reconstructs these actions using behavior cloning loss. The skill prior p(z(s,) captures the distribution of likely skills for a given state.
while ÁBC ÀSp. and are weighting coeffi-
Contrastive Learning for Skill Discrimination Our main goal is to cluster similar behaviors into the same skill. To achieve this, we introduce a skill similarity function foy (8, 2, (0o, (s, 2), voo (s')). Here, do, (s, 2) skill-conditioed state representation, and via (s') is state representation, both of which are trainable networks (Eysenbach et al. (2022)). Our core as- sumption that semantically similar behaviors will display similar state change patterns after ap- plying skill. To train the skill similarity function, we need both positive and negative samples. Positive samples pair the initial state 8, with the skill Σ, and an intermediate state 814b, chosen to capture long-term skill effects. Negative samples states s that are unreachable from 8, given the skill 24, sampled from different skill trajectories. This can be expressed as:
Uniform(1s E D|3e' + 2, E skill (=kil(()))))
(4)
where ekkill (2) {St,..., S represents the set of states belonging the trajectory of skill 2. For simplicity, we denote this distribution as p(s + =). We use the Noise Contrastive Estimation (NCE) binary loss function (Ma & Collins 2018) train the skill similarity function. Here, a-)) denotes the sigmoid function, while dcL represents the weighting coefficient, and 0, denotes the trainable parameter. The contrastive loss is follows detailed analysis can found in A.pl)
Lcontrastive dCL E(,,a)= Egg, (z|5) log (fe, (8, sb))
E Ba-op(lle'
[log(1 ofor [|]]]
(5)
Skill Terminate State Predictor For downstream task learning, determining the point to start the next skill is crucial due to the variable lengths skills our method. Inspired by SkiMo (Shi et al. 2002)), ) repurposed their dynamics module to function for use downstream tasks. This predictor is capable of handling these variable skill lengths. Our approach predicts the skill target state $1+H, which is the state after the skill has been executed from the initial state s,. When the skill target state reached during downstream task execution, the skill is considered complete, allowing the next skill proceed.
skill target state predictor specifically
5

---

Published as a conference paper at ICLR 2025
We state encoder Egg (st) and observation decoder O&o (Eer (st)) to map observations to a latent state space and ensure accurate state representations by reconstructing the original observa- tions. The loss function for the skill target state predictor To, (-) is as follows:
Ltarget Σ BEnD
[ [e (2)13) ARELlS OBo (Eeg (s:))[1
AST [oor (st); 21) - Eon [s]+]]
(6)
Here, ARF and ÄST represent the weighting coefficients, and 00,0E, Or are the trainable parameters. This loss includes the observation reconstruction term | Oeo (ag (s+)) and the skill target
state prediction term |T Eg (fe), =x) Fig (st-H.) )11-
Objective Function The total objective function for the model is combination of the skill em- bedding, contrastive learning, and skill target state predictor losses:
Litall = Lembedding + Coontrastive + Lerrgot.
4.2 SKILL LENGTH RELABELING
We introduce a dynamic skill length relabeling process based on the learned skill similarity function fo, s'). Starting from each skill's initial state 8t in the dataset, obtain using the pre- relabeling skill length H, We then compute similarity values for subsequent states 8t+a (a > 0) and define the new skill length as the maximum number of consecutive time steps where the similarity remains above threshold
H = + max a fo, (st, 24, s(ta) > 2
(8)
This approach allows for dynamic adjustment of skill lengths based on observed state transitions, ensuring that each skill's duration accurately reflects the temporal extent of its effect. To ensure stable learning, relabeling is performed every Trelaee training steps. Additionally, constraints on
skill length (6min I He S 8max) are applied prevent abrupt changes. A detailed convergence analysis and theoretical justification are provided in Appendix A.2
4.3 DOWNSTREAM TASKS LEARNING
The learned skills can be applied to downstream tasks using both model-free and model-based ap- proaches. As our research focuses on skill extraction and representation, we utilized previous ap- proaches such as SPiRI (Pertsch et al. (20T)) and SkiMo (Shi et al.] (2022) ) for downstream learn- ing. the model-free setting, Soft Actor-Critic (SAC) (Haarnoja et al. (2018)) is used to train a high-level policy that selects skills based on the current state. Meanwhile, in the model-based set- ting, skill target state prediction combined with the Cross-Entropy Method (CEM) (Rubinstein (1997) to enable long-horizon planning through skill simulations.
A key feature of our framework in downstream learning the dynamic adjustment of skill durations during execution. Unlike previous methods with fixed-length skills, approach uses skill target state predictor to adapt skill execution. Skills are continued to execute until the predicted target state reached until certain number of timesteps, at which point the high-level policy selects the next skill. Detailed explanations of the model-free and model-based implementations are provided in Appendix B.2
EXPERIMENTS
We designed our experiments to answer the following three key questions: (1) Can the skills learned through DCSL capture meaningful behaviors that enable efficient performance new tasks in both

---

Published as a conference paper at ICLR 2025
navigation and manipulation domains? (2) Is DCSL able extract meaningful and common be- haviors through skill length adjustment even in more complex or noisy data that include irrelevant meaningless actions? (3) Are similar behaviors effectively clustered into similar skills, resulting in an efficiently learned skill representation space? Our experimental results provide compelling evidence to address these questions, demonstrating the effectiveness of DCSL across various envi- ronments and data conditions.
5.1 EXPERIMENTAL SETUP
We conducted experiments in the antmaze, kitchen, and pick-and-place environments 6 Antmaze is long-horizon navigation task, where the goal for an Ant agent reach a designated goal. We employed two variants, Antmaze-Medium and Antmaze-Large. For each environment, we used the maamaaze-ediummdiive at. (. In the Kitchen environment, the agent is tasked with completing four independent sub- tasks. We utilized the kitchen-mixet dataset from D4RL for this environment. In the Meta-world Pick-and-Place environment (Yu et al. (2020)), used datasets provided by Yoo et al. (2022)
and 'antmaze- "antmaze-large-diverse datasets provided by D4RL (Fu
taining varying levels of noise. These datasets are ordered from least to most noisy: Axeeuu-Bxperrt (ME), Medium-Replay (MR), and Replay (RP).
Baselines To assess the effectiveness of our framework, we compared with several well- established baselines. First, evaluate the utility of the skills learned by DCSL, we used two skill learning comparison groups: SPiRL (Pertsch et al. (2021)) and SkiMo (Shi et al. )2022)) SPiRL employs a learned skill prior guide exploration, while SkiMo integrates skill dynamics with model-based planning. Both methods defined skills as fixed-length action sequences. We eval- uated SkiMo two variants: SkiMo-SAC and SkiMo-CEM, using the SAC algorithm and CEM respectively. Our research focuses primarily on the skill extraction process. To evaluate our ex- tracted skills, we applied existing methods for downstream task learning, testing two versions of DCSL framework: Ours-SAC and Ours-CEM, In all experimental environments, the prede- fined skill length for the baseline methods and the initial skill length for DCSL were set to 10. To constrain the skill space, set the skill dimension to 5 for Antmaze and Kitchen environments, and 2 for Pick-and-Place. To further evaluate DCSL's effectiveness in utilizing offline datasets for task performance, we included additional baselines. We compared against Behavioral Cloning (BC) (Pomerleau] (1988)) which learns policies directly from demonstration data, and Conservative Q- Learning (CQL) (Kumar al. (2020)), an offline RL method that ensures policy actions remain within the data distribution. We also included CQL+OfI-DADS (Sharma et al. ), which learns skills by maximizing mutual information between skills and trajectories, and CQL+OPAL (Ajay et al. (2020)), which uses unsupervised learning to discover skills that reduce temporal abstraction. While CQL+Off-DADS and CQL+OPAL also learn skills, they differ from DCSL in their use of CQL for high-level policy learning.
5.2 EXPERIMENTAL RESULTS
Fig. 3 and Table T present the results for each environment through learning graphs and success rates, while Table 2 shows the number of timesteps required to complete the tasks. These results demonstrate the performance and efficiency of our DCSL framework across various environments and task complexities.
The Antmaze environment is sparse reward task where the agent only re- ceives a reward upon reaching the goal. As a result, it requires extensive maze exploration, with skill learning efficiency dependent on the clustering of similar behaviors. Effective clustering of- fers useful skill representation. redundancy reduction, and improved generalization. Results from Antmaze- Mediium and Antmaze-Large demonstrate DCSL's effective skill clustering and represen- tation capabilities. In Antmaze-Medium, Ours-SAC outperformed SkiMo-SAC, suggesting DCSL generated more accurate skill representations, leading to improved exploration efficiency. SkiMo- CEM showed high success rates in this environment, while Ours-CEM struggled to achieve sig- nificant performance. This discrepancy can be attributed to DCSL's variable skill lengths, which destabilize CEM's planning horizon. The inconsistent time scales hinder CEM's effectiveness in Long-horizon tasks, exacerbating cumulative errors. Comparing with other basclines utilizing of- fline datasets, we observe that BC and CQL., which do not employ skill learning, struggle to reach
Antmaze-medium

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of success rates. Boxes containing '-' denote results that could not be obtained due to the unavailability of public implementation code.
Method
Environment
BC
CQL
CQL-Off-DADS
CQL+OPAL
Ours-SAC
Ant-Medium
0.0
53.7 1
59.6 = 29
81.1 #31
68.0 + 36.9
Ant-Large
0.0
14.9 32
70.3 2.9
73.7 +: 5.9
Kitchen
47.5
52.4 2.5
69.3 #
94.7
the goal. While DCSL, outperforms CQL+Off-DADS, CQL+OPAL achieves the highest success rate. This performance difference can be attributed to the varying approaches in learning high-level policies, similar to what we observed with SkiMo-CEM.
Antmaze-large In Antmaze-Large, Ours-SAC significantly outperformed all other methods, demonstrating DCSL's generalization ability in more complex environments. The performance gap with SkiMo-SAC implies DCSL better addresses the sparse reward problem. CEM-based methods' performance degradation highlights DCSL's superior skill representation space learning in chal- lenging environments. These results prove DCSL. enhanced exploration efficiency and generaliza- tion ability across various environmental scales through effective skill clustering and representation. In comparison to other offline dataset baselines, BC and CQL face learning difficulties similar to those in Antmaze-medium Although CQL+OPAL demonstrates high performance, DCSL achieves a higher success rate due to its superior exploration efficiency. This highlights DCSL's ability to effectively navigate more complex environments.
Kitchen Unlike the Antmaze environment, the Kitchen environment demands precise object ma- nipulation skills. DCSL's performance here demonstrates its flexibility and adaptability. Ours-SAC successfully completed almost all four subtasks, indicating DCSL's effectiveness in tasks requiring delicate action sequences, despite using state-based skill embeddings. Our approach performed sim- ilarly or better than action sequence- based methods, suggesting DCSL better captures task essences by focusing on state transitions. While Ours-SAC and SkiMo-CEM completed similar number of subtasks, our method required fewer timesteps to accomplish the tasks, as shown in Table 2 This efficiency highlights the effectiveness of the skills that DCSL has learne.. Although Ours-CEM showed slightly lower performance, the overall results of DCSL are promising, particularly consid- ering its ability to generalize across different task types. This underscores DCSL's strength in learn- ing transferable skills that are effective in both exploration-centric tasks and those requiring precise manipulation, demonstrating robust performance across diverse environments. When compared to other offline dataset baselines, DCSL significantly outperforms its counterparts. The superior per- formance this manipulation task, which requires precise control, demonstrates the effectiveness of state-trand transittanitinn based skills in capturing intricate action sequences. This result underscores DCSL's adaptability to tasks demanding detailed object manipulation.
Pick-and-Place The Pick-and-Placs environment evaluates the impact of data quality on skill learning, showcasing the importance of DCSL's skill length adjustment. In the high-quality ME dataset, all methods achieved similar success rates. However, performance differences became more pronounced in the lower-quality MR dataset, with DCSL showing improved performance. In the noisy RP dataset, Ours-SAC outperformed all other variants, demonstrating DCSL's effectiveness in handling data with high noise and irrelevant actions. These results show that DCSL maintains strong performance across datasets of varying quality, with its importance becoming more evident as data quality decreases. In the MR and RP datasets, our method not only achieved high success rates but also required the least number of timesteps complete the tasks.
5.3 ABLATION STUDY
The ablation study (Fig. 4 conducted in the Pick-and-Place environment utilized three datasets (ME, MR, and RP) to evaluate the impact of the similarity function and skill length readjustment process in the DCSL framework. In the ME dataset, all methods achieved similarly high success rates, suggesting that skill length readjustment has relatively little impact high-quality data. However, without a similarity function, efficient skills cannot be effectively learned, resulting in the high-level policy requiring training steps to converge. The benefits of skill length readjustment became
8

---

Published as a conference paper at ICLR 2025
SPIRL
SKIMo-CEM
SKIMo-SAC
Ours-CEM
Ours-SAC
Moder
Moddeellc weweellaamm...mmmm dan
[8] Antmaze- -Mdium
(b) Antmazz-Laage
[6) Kicchn-Mixed
206 256
055 036
206
326
256
336
055 adc
(d) Pick-and-Pllae (ME)
Pick-and-Place (MR)
(1) Piok-enndPPaee (RP)
Figure 3: Performance comparison across Antmaze-Mediumg Antmaze-Large. Kitchen, and Pick- and-Place environments (with 5 different random seeds). Results demonstrate the adaptability and effectiveness of DCSL in handling diverse and long-horin fong-horizon tasks. Dark lines represent the average returns, and shaded areas represent standard deviations. A single training step was conducted after rolling out episode.
Table 2: Comparison of the number of timesteps required to complete tasks across different envi- ronments and algorithms.
Method SkiMo-SAC
Environment
SPiRL
SkiMo-CEM
Ours CEM
Ours-SAC
Ant-Medium
988.5 19.8
311.2 + 95.7
833.7 +: 288.0
1000.0 0.0
453.6 + 143.7
Ant-Large
990.2 = 19.5
993.5 = 13.9
881.5 164.9
1000.0
672.2 = 72.9
Kitchen
276.6 5,9
205.8 = 29.0
251.3 23.7
262.0=2011
165.1 = 4.4
PP (ME)
87.8 +
54.1 21.3
58.0 =
76,0
80.1 13.7
PP (MR)
138.0 63.2
184.8 +: 24.0
87.3 1: 57.6
62.9 5.88
56.1 5.1
PP (RP)
130.6 69.4
193.2 + 13.5
200.0 0.0
85.1 22.3
64.4 16.3
more apparent the MR dataset. Ours-CEM outperformed "w/o relabel (CEM)', indicating that skill length readjustment enhances the noise handling capability of CEM-based methods. The importance of skill length readjustment was most pronounced in the RP dataset. Ours-SAC achieved the highest success rate, while 'w/o relabel (CEM)' and "w/o sim' showed significantly lower success rates. These results demonstrate that the similarity function and skill length readjustment process in DCSL plays crucial role in maintaining consistently strong performance across datasets varying quality.
5.4 QUALITATIVE EVALUATION SKILL CLUSTERING
We conducted qualitative evaluation (Fig. 5 to assess whether DCSL effectively clustered similar behaviors into the same skill, thus efficiently learning the skill representation space. For this purpose, we modified the Antmaze environment by removing all obstacles except the borders and placing the Ant agent at the center of the maze. We sampled 50 random skills and visualized the Ant's movements during the exccution each skill.
The results show that for SPiRL and SkiMo, similar behavior patterns repeatedly appeared across multiple random skills. This suggests that these methods are experiencing a dimensional collapse, where similar behaviors learned different skills. In contrast, DCSL exhibited wide range of behavior patterns, including movements in various directions. This demonstrates that DCSL effectively clustered similar behaviors, leading efficient learning of the skill representation

---

Published as a conference paper at ICLR 2025
wio (SAC)
relabel (SAC)
wio relabel (CEM)
Ours- CEM
Outs-SAG
66
04
07
00
-22
-02
Mambar ofteroooreeeeooo
Hamber wwwaarro sto
Hamber aloutnnattto
(a) Pick-and-Place (ME)
(b) Pick-and-Place (MR)
(c) Pick-and-Place (RP)
Figure 4: Ablation study results in the Pick-and-Placc environment across the ME, MR, and RP datasets, comparing the impact of the similarity function and skill length relabeling in our frame- work. These results highlight the significance of these techniques in maintaining performance ro- bustness in suboptimal and noisy environments.
(a) SPiRL
(b) SkiMo
(c) DCSL (Ours)
Figure 5: Comparison of execution results for 50 randomly sampled skills in the Antmaze environ- ment. Each colored line represents the trajectory of the Ant agent during the execution of a single skill. While SPiRL and SkiMo show similar behavior patterns repeating across multiple skills, DCSL demonstrates movements in various directions and patterns. This suggests that DCSL has learned more efficient skill representation space by effectively clustering similar behaviors.
space. These findings indicate that DCSL can learn more diverse and meaningful set of skills compared to existing methods.
CONCLUSION
In this paper, we introduced Dynamic Contrastive Skill Learning (DCSL), a novel framework that addresses challenges in long-horizon reinforcement learning tasks by focusing rather than fixed action sequences. DCSL makes two key contributions: it uses contrastive learning cluster semantically similar behaviors, enabling more effective skill extraction, and it dynami- cally adjusts skill lengths based on a learned similarity function. Our experiments navigation and manipulation tasks demonstrated DCSL's ability extract useful skills from diverse datasets, including those containing a mix of relevant behaviors and irrelevant or random actions, enhancing generalization and adaptability across various environments and showing competitive performance compared to existing methods in task completion and efficiency.
stteetransition
Looking ahead, we envision several promising directions for future work. First. we aim extend the use of the learned similarity function beyond skill learning and length adjustment to downstream task learning, potentially improving task performance by leveraging the semantic understanding of skills during execution. Second, we plan to explore the integration of language instructions with skill learning using contrastive methods, enabling agents to follow natural language commands and generate relevant skills, thus expanding the framework's potential in interactive environments. Lastly, we intend to investigate the application of DCSL in multi-agent systems and dynamic task structures, further testing its adaptability and scalability. These extensions would not only enhance the capabilities DCSL but also bridge the gap between skill learning, language understanding, and task execution reinforcement learning.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We implemented all learning models using PyTorch. The experiments were conducted in environ- ments provided by D4RL (Fu (2020) and Metaworld (Yu et (2020). detailed explanation of our framework's 's overall algorithm, model implementation detais, and environment configura- tions are provided Appendix and Appendix
ACKNOWLEDGEMENTS
This research was supported by the Challengeable Future Defense Technology Research and De- velopment Program through the Agency For Defense Development(ADID) funded by the Defense Acquisition Program Administration(DAAA)) in 2024(No.115088201)
REFERENCES
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline prim- itive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010. 2020.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAl conference artificial intelligence, volume 31, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Gabriel Dulac-Amold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv: 12901, 2019.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning achieve goals via recursive classification. arXiv preprint arXiv:2011.08900 2020.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learn- ing as goal-conditioned reinforcement learning. Advances in Neural Information Processing Sys- fems, 35:35603-35620 2022.
Benjamin Freed, Siddarth Venkatraman, Guillaume Adrien Sartoretti, Jeff Schneider, and Howie Choset. Learning temporally abstractworld models without online experimentation. In Intern- tional Conference on Machine Learning, pp. 10338-10356. PMLR, 2023.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: wt..iv:/8/2.000 18/2.09955 2018.
Ce Hao, Catherine Weaver, Chen Tang, Kenta Kawamoto, Masayoshi Tomizuka, and Wei Zhan. Skill-critic: Refining learned skills for hierarchical reinforcement learning. IEEE Robotics and Automation Letters, 2024.
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos, and Doina Precup. The termination critic. arXiv preprint arXiv: 2019.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition, pp. 9729-9738. 2020.
Yiding Jiang, Evan Liu, Benjamin Eysenbach, Zico Kolter, and Chelsea Finn. Learning options via compression. Advances in Neural Information Processing Systems, 35:21 184-21 199, 2022.

---

Published as a conference paper at ICLR 2025
Li Jing, Pascal Vincent. Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive superviuer learning. arXiv preprint arXiv:2110.09348. 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179- 19:1199-119, 2020.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International conference machine learning, pp. 5639- 5650. PMLR, 2020.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp. 1113- 1132. PMLR, 2020.
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint arXiv: 180.008//, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 187.v:1807.03348. 2018
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188- 204. PMLR, 2021.
Karl Pertsch, Ruta Desai, Vikash Kumar, Franziska Meier, Joseph J Lim, Dhruv Batra, and Akshara Rai. Cross-domain transfer via semantic skill imitation. arXiv preprint arXiv:2212.07407, 2022.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- PMLR, 2021.
Reuven Y Rubinstein.
Optimization of computer simulation models with rare events. European Journal of Operational Research, 99(1)899-12, 1997.
Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. In International Conference on Machine Learning, pp. 8624-8633, PMLR, 2020.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint 1rXiv:1907.0000 2019.
Archit Sharma, Michael Ahn, Sergey Levine, Vikash Kumar, Karol Hausman, and Shixiang Gu. Emergent real-world robotic skills via unsupervised off-po reinforcement learning. arXiv preprint arXiv:2004.1299 2020.
Lucy Xiaoyang Shi, Joseph J Lim, and Youngwoon Lee. Skill-based model- reinforcement learning. arXiv preprint arXiv:2207.07556 2022.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):4844488) 2016.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782)3550-354 2019.
12

---

Published as a conference paper at ICLR 2025
Rushuai Yang. Chenjia Bai, Hongyi Guo, Siyuan Li, Bin Zhao, Zhen Wang, Peng Liu, and Xuelong Li. Behavior contrastive learning for unsupervised skill discovery. International Conference Machine Learning, pp. 39183-39204. PMLR, 2023.
Minjong Yoo, Sangwoo Cho, and Honguk Woo. Skills regularized task decomposition for multi- task offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 2022.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100, PMLR, 2020.
Chongyi Zheng, Benjamin Eysenbach, Homer Walke, Patrick Yin, Kuan Fang, Ruslan Salakhutdi- nov, and Sergey Levine. Stabilizing contrastive rl: Techniques for robotic goal reaching from offline data. arXiv preprint arXiv:2306.033444 2023.
A THEORETICAL ANALYSIS
A.1 THEORETICAL ANALYSIS CONTRASTIVE LEARNING FOR SKILL DISCRIMINATION
This section provides a rigorous theoretical foundation for understanding how contrastive learning enables effective skill clustering in DCSL. We present two key theorems that demonstrate the rela- tionship between our contrastive learning objective and mutual information maximization, as well as connection to true skill discrimination.
A.1.1 CONTRASTIVE LEARNING OBJECTIVE AND MUTUAL INFORMATION MAXIMIZATION
The contrastive learning objective used in DCSL is defined as:
Leuntrattvve = >CL. E(E,,)D Egg, (=|²) log a (Jo, (s, 8b))
E, ~p(ss=+4)) [log(1 o (s, s
MI]
(9)
where Sa, (8, 2, 82) (0o,, (8,2)) 2), (/0))))))) is the skill similarity function.
Theorem 1. Assuming an optimal discriminator, the contrastive learning objective maximizes the mutual information between skills and state transitions.
Proof. The optimal solution for the inner expectation of the contrastive loss is achieved when:
p())(s,)) p(s' + p(s))
offo (8,
Substituting this into the contrastive loss and simplifying:
Coontrustive -" log p(s)|s,2) -I(;
p(s')
Where ((2:51(5) is the conditional mutual information between skills Z and future states S' given the current state S.
This result demonstrates that minimizing the contrastive loss is equivalent to maximizing the mutual information between skills and state transitions, leading to more discriminative skill representations. As corollary, maximizing this mutual information promotes improved skill clustering. Intuitively, this can be explained by three key points: First, higher mutual information indicates that skills are more predictive of future states. Second, similar state transitions will be associated with similar skill representations to maximize this predictiveness. Finally, this similarity in skill representations for similar behaviors naturally leads clustering in the skill space.
13

---

Published as a conference paper at ICLR 2025
A.1.2 LOWER BOUND ON SKILL DISCRIMINATION
Theorem 2. The contrastive learning objective provides a lower bound on the true skill discrimi- nation task.
Proof. Let C(s, 2, 8') be binary random variable indicating whether s' is reachable from using skill z. The true skill discrimination task can be formulated as maximizing:
E (s,,,,,") (opla.z")))pi Dog 1|8, 2, 2,) + ((..)))-p(e.))))""ppt) (logg (C p(C 0 0,
can be shown that our contrastive loss provides a lower bound on this objective:
Lcontrstive I ((sss P(C 1|s, 2, s')l ((s.))=((sa)) - p(C = 0,,,
This theorem demonstrates that by optimizing contrastive loss, effectively learning to discriminate between reachable and unreachable states given a skill, which is crucial for meaningful skill clustering.
A.1.3 CONCLUSION
These theoretical results provide rigorous foundation for understanding why contrastive learning leads to effective skill clustering in DCSL. By maximizing the mutual information between skills and state transitions, and providing lower bound the true skill discrimination task, our method naturally groups similar behaviors into coherent skill representations.
A.2 THEORETICAL ANALYSIS OF SKILL. LENGTH RELABELING
This section provides an intuitive explanation and brief analysis illustrating how constraining skill lengths and performing periodic relabeling contribute to stabilizing the training process.
A.2.1 REDUCING TRAINING INSTABILITY BY LIMITING SKILL LENGTH VARIATIONS
By enforcing constraints on skill lengths (6min < H2 < 6max), we prevent the skill duration He from changing abruptly during the relabeling process. Skills group the data into distinct behavioral categories, allowing the model to handle each category with specialized representations. Sudden large changes skill lengths can cause abrupt shifts in the data distribution, potentially leading to unstable training dynamics. By limiting the variation in H,, we ensure that changes in the data distribution occur gradually, allowing model to adapt more smoothly.
Additionally, small changes in H, lead to gradual transitions in the loss landscape, resulting in more consistent gradient updates during backpropagation. This consistency helps stabilize the learning process by preventing drastic fluctuations in gradients.
ANALYSIS: BOUNDING CHANGES IN LOSS AND GRADIENTS
When the change in skill length AH, Hi H, is limited by AH, 6max - 8min- the variations in the loss function Lioaal and its gradients with respect to model parameters are expected to be bounded. While exact mathematical bounds depend on the specific properties of the loss function and model architecture, constraining skill lengths reduces the potential for large fluctuations in loss and gradients, thereby contributing training stability.
A.2.2 MODEL ADAPTATION THROOGHI PERIODIC RELABELING
Performing skill length relabeling every T' training steps allows the model sufficient time to adapt to the current skill assignments before they are updated again. Reducing the frequency relabeling prevents the model from continuously adjusting to skill assignments, minimizing potential dis- ruptions during training. This periodic approach enables the model parameters to be updated based consistent set of skills over multiple iterations, facilitating stable convergence.
14

---

Published as a conference paper at ICLR 2025
ANALYSIS: REDUCING GRADIENT VARIANCE
By maintaining consistent skill assignments over 7 training steps, the variability in parameter up- dates be reduced. While do not assume that gradients independent identically dis- tributed across steps, the consistent skill assignments help in smoothing out fluctuations in the train- ing process. Averaging updates over multiple steps with stable skill assignments contributes to more consistent gradient directions and magnitudes, aiding in stable learning.
A.2.3 ENSURING STABILITY THRROUGI THE COMBINATION OF CONSTRAINTS AND PERIODICITY
Combining constraints on skill lengths with periodic updates ensures that changes to the model's input distribution and internal representations occur gradually. This gradual evolution minimizes abrupt shifts that could destabilize training. Stable skill lengths and periodic updates enable the model to effectively learn skill embeddings that capture semantically similar state transitions, en- hancing representation learning.
ANALYSIS: SMOOTHING THE OPTIMIZATION LANDSCAPE
Constraining skill lengths and performing periodic relabeling help smooth the optimization land- scape by preventing sudden changes in the loss function with respect model parameters. This smoothing effect reduces the likelihood of encountering sharp gradients or irregularities that can hinder optimization. While we do make specific assumptions about the second derivatives (Hes- sian) of the loss function, the general effect of these techniques is to promote a smoother loss surface, facilitating more reliable convergence of optimization algorithms.
B IMPLEMENTATION DETAILS
This section describes the implementation details of our framework.
B.1 ALGORITHM
Algorithms D and describe, respectively, the method for extracting skills from datasets and the process of relabeling skill lengths in our framework.
B.2 DOWNSTREAM RL
Here, we outline the algorithm used to train downstream tasks. Rather than developing novel approach for efficiently searching through skill combinations, focus these learned skills, we used the same method for
refining the skills
themselves. To showcase the effectiveness combining skills in previous work, drawing on established approaches such as SPiRL (Pertsch) jet al. (202T) ) and SkiMo (Shi et al. (2022)). A brief overview each method provided below, with further details available in the corresponding papers.
Model-Free Approach For the model-free setting, we employ the SPiRLeertsch et al. (2021) framework, which introduces skill prior to guide exploration in the skill space. This approach leverages a learned skill prior Pa (2]*),, which captures which skills are most relevant given the current state, reducing exploration inefficiencies.
The high-level policy (2)=)))) selects skill embeddings , that are decoded into action sequences. To encourage the exploration of meaningful skills, we regularize the policy with the learned skill prior using the following objective:
T/H Σ t=
LSPIRL E,
(r: aDKI (To(2(ss)))))
(10)
15

---

Published as a conference paper at ICLR 2025
Algorithm Skill Extraction with Contrastive Learning
Require: Unlabeled offline dataset D {T1, T2,..., TNY, where each 72 {(5 a.)]= Relabel-
in interval Telabel
Ensure: Learned skill embeddings = 2
1: Initialize skill encoder f0,- skill decoder n0,, skill prior po,
2: Initialize skill similarity function fa,
3: Initialize state encoder Egg. observation decoder Ogo skill termination predictor Tor
4: Initialize skill length H,, for each (5t, at) D
5: for each training iteration do
6: Sample batch of skill trajectories ( 1 from
7: for each s2iill batch do
8:
Select key states & {st, Stta, S14b. S1+H,-1} from r.cilll (. -1+H Encode skill: q0, (2(
9:
10:
Decode actions: à = TT, (a F),
11:
Compute prior: Pe, (=)=t)
12:
Compute Cembedding using Eq 3
13:
Sample negative state 8 from different skill trajectory 73 zokill # T.kill
14:
Compute similarity: Ja, (8,, 8148) and Ju, (,,,
Compute Coontastive using Eq
16:
Encode states: h, = Eeg (5t), MetH. = Eeg (8L-H.)
17:
Predict target state: f44H1 Top (h),2)
18:
Decode observation: & O00 (h4)
19:
Compute Curgat using Eq
20:
end for
21:
Compute total loss: Lioial Lremhedddng + Leontrastive + Liarget
22:
Update model parameters 0g 6n. 0p. Os. 2g Go. and 0r to minimize Liotal
if mod Tellabel then
24:
Perform skill length relabeling (Algorithm) 2
end if
26:
end for
Algorithm 2 Skill Length Relabeling
Require: Dataset D, Similarity function f. Threshold Min/max skill length 8min Swax Sample sasmpl states from
2: for each sampled state 8, do
3: Retrieve corresponding episode Tepisode
4: Encode skill: 2, <- 98, (81, 814a: St+b: 86+H5-1)
5: for a = 1 to len (episode) - do
6: Compute similarity: sim fa, (St, Z4, S140)
7:
sim S c then break
8:
9: end
end for
11: Relabel the skill duration: H + a 11
Constrain skill length: Hi max(@min min(H' 8max))
13:
end for
16

---

Published as a conference paper at ICLR 2025
Here, rt is the cumulative reward over H -step action sequences, and the KL divergence ensures that the policy samples skills aligned with the learned prior. This improves exploration efficiency by focusing on promising skills, rather than exploring uniformly.
The Soft Actor-Critic (SAC) algorithm is extended to handle skill embeddings instead of primitive actions. The critic updates the Q-values for the selected skills, incorporating the learned skill prior to regularize the policy:
(st 2x) = + +Q (e ( m())++)))
(11)
The learned skill prior Pa (z(s:) plays crucial role in guiding the policy toward effective skills
during both training and execution, significantly accelerating downstream task learning.
Model-Based Approach In the model-based reinforcement learning setting, we leverage the Skill Dynamies Model from SKiMo (Shi al. (2022)) to perform efficient long-horizon planning in the skill space. Given initial state 8, and a skill embedding 2t, the skill dynamics model Do predicts the latent state after the skill has been exccuted for H steps, as h To(hs, 2t), where he (s:) is the latent state produced by the state encoder.
In addition to skill dynamics, the model includes reward prediction and value estimation for each skill. The reward model (g) predicts the cumulative reward r, over the skill's duration, while the Q-value model (he, 21) estimates the expected return from executing the skill. These predictions used both for planning and for policy optimization, with the Q-value computed as:
Qo(1, 21) r + (Q (ooteeHHHTT(CHHHHH (b++))))
(12)
Temporal Difference Model Predictive Control (TD-MPC) is used for planning. This method lever- ages skill dynamics and reward models to simulate long-term trajectories, optimizing skill sequences over a defined planning horizon. The Cross- Entso-y Method (CEM) employed to search for the optimal sequence of skills, and the first skill in the sequence is then executed in the environment. The planning step selects skills by maximizing predicted Q-values over the horizon:
K Σ
Plan st) arg max
24+HH]
(13)
Re:c++K
where K represents the planning horizon. This method allows for efficient skill-based planning by minimizing the number of predictions required, reducing cumulative prediction errors.
The model trained using a combination of losses, including latent state consistency, reward pre- diction, and value prediction:
Lmmodel-hnsed E [Ar.l 2x) h44 +\ Rg (e)1,
21) Quarger |2]]
(14)
where Qtarget is the target Q-value computed from future predictions. By using TD-MPC with skill dynamics, reward, and value predictions, this approach enables efficient long-horizon planning in the skill space, improving sample efficiency for reinforcement learning in complex tasks.
B.3 MODEL IMPLEMENTATION DETAILS
The key models we need to train include the skill encoder 9a,, skill decoder no+, skill prior Po, skill-conditioned state representation do,, state representation vhy state encoder Føg observation decoder Oo and skill termination predictor To,. Each of these models is implemented as neural network. The skill encoder qe, skill decoder Te.,, skill prior pa,, observation decoder Oag and skill termination predictor Te, are all 4-layer MLPs with hidden dimension of 256, using the ELU activation function. The skill-conditioned state representation Φo, and the state representation the 2-layer MLPs with hidden dimension of 128 and employ the ReLU activation function. de, and tia output the input values to a representation dimension of 16. Additionally, the state encoder Fog g is an LST with hidden dimension of 128. The learning rate for all models was set to 0.0003.
17

---

Published as a conference paper at ICLR 2025
Figure 6: Environments from left-to-right: Antmaze, long-horizon navigation task with two vari- ants Anttmaze-Mediue and Antmaze-Large) Kitchen, where the agent tasked to complete four independent subtasks, and Meta-world Pick-and-Plase where the goal is to pick up an object and place it at target location.
C EXPERIMENTAL DETAILS
C.I ENVIRONMENTS
In this section, we provide a more detailed description of the experimental environments.
The goal of the antmaze task is to guide the ant agent to the goal position within the maze. The observation space has 29 dimensions, capturing the agent's position and joint information. The action space consists of 8 dimensions, corresponding to the control of the agent's 8 joints. In this environment, we set the skill dimension needs to determine whether the pblueicted skill termination state has been reached. This is assessed using the ant robot's and position, the z-coordinate of the torso, its orientation (quaternion x, y, w), and the hip joint angles of the 4 legs (I per leg), resulting in a total of 14 dimensions. The achievement of the skill target state is determined by the 1.2 distance between the pblueicted state and the current state. The success threshold is set at distance of 0.5.
for all baselines. As mentioned earlier, our framework
In the Kitchen environment, the objective to manipulate various objects achieve desiblue states. We used the 'mixed-00' dataset, where subtasks are presented in disjoint manner, requiring the agent to generalize across non-sequential task executions. Similar to antmaze, we set the skill mension to 5 for all baselines. The observation space is 30-dimensional, representing the robot's s joint states, while the action space contains dimensions for controlling its joints. Skill termination determined based on the robot's 9 joint positions and velocities (18 dimensions), with success threshold distance of 0.1.
For Meta- BoradWsord'ss pick-and-place task, the goal is place object target location. We used datasets that include both successful and failed task executions, as provided by Yoo et al. (2022). The datasets consist of three types: Medium-Replat (MR) with 150 trajectories from intermediate policy stages, Replay (RP) with 100 trajectories spanning the entire learning process (including relevant and irrelevant behaviors), and Medium-Expert (ME) with 50 expert-level trajectories. The observation space is 18-dimensional, representing the robot's 's joint positions, velocities, and the object's position, while the action space has dimensions. In this environment, the skill dimension is set to 2 for all baselines. The skill termination is evaluated based on the robot's 3D end-effector position and the open/closed state of the gripper dimensions). with success threshold distance of 0.02.
C.2 HYPERPARAMETERS
D ADDITIONAL EXPERIMENTS
D.I RESULTS ADDITIONAL ENVIRONMENTS
We conducted additional experiments in an environment that, like Pick-and- allows to com- pare results based on data quality. Peg-Insert-Sin Yu al. (2020) uses the same robot as Pick-and- Place, but the task involves inserting peg into hole in box. While both tasks involve grasping and moving objects, there are two significant differences between these environments.
18

---

Published as a conference paper at ICLR 2025
Table 3: DCSL hyperparameters.
Hyperparameter
AntMaze
Kitchen
PickPlace
Batch Size (skill extraction)
256
256
128
Batch Size (Downstream learning)
256
256
128
Max Global Step (skill extraction)
100,000
Max Global Step (Downstream learning)
4,000
Weighting coefficient ABC
2
Weighting coefficient Asp
Weighting coefficient ÅcL
Weighting coefficient ARE
Weighting coefficient ÄST
Skill Dimension
5
5
2
Initial Skill Length H
10
Relabeling Interval T
20,000
Relabeling Threshold e
Length Constraint 8min
4
Length Constraint maz
Distance Threshold
0.5
0.1
0.02
SKIMo-CEM
SkiMo-SAC
Ours-CEM
12
1.0
10
08
Santru Rax
0.6
Sacess
00
23s. wwwraaiiiinni creat.
206.
25x, maa affraiiing theas (b) Peg-Insert-Siid (MR)
25k abx
Manree stnorwigg thres
(9) Peg-Insert-Side (ME)
(c) Peg-Insert-Side (RP)
Figure 8: Performance comparison of various methods in the Peg-Insert-Se task across different dataset qualities (ME, MR, RP). All methods demonstrate high success rates, highlighting the task's relative ease compared to Pick-and-Place.
First, there's a difference in the size of the object the robot needs to grasp. In Bick-and-Place the object is very small, so even a slight misalignment of the robot's grip- per result in failed grasp. In contrast, the peg in Peg-Insert-Sice is longer, making it casier for the robot to grasp.
The second difference is that in Peg-Insert-Sie the robot can use the box guide when positioning the object, By moving the peg along the box's surface, the robot can more easily find the hole. Pick-and-Plas doesn't 't offer such guidance, making it more challenging environment for the robot to complete its task.
These differences are reflected in the experimental re- sults. Regardless of the dataset quality and the method used for finding downstream tasks, all methods showed high success rates in Ingertnset-Side This confirms that one of our framework's 's major strengths skill length ad- justment becomes more pronounced as the task diffi- culty increases.
Figure 7: Peg-Insert-Side task: The robot must grasp long object and in- sert it into hole in the box.
19

---

Published as a conference paper at ICLR 2025
46
34
43
03
0.5 06
222
005
-225
030
122
055
675
140
18
(a) SPiRL
DCSL (Ours)
Figure 9: Visualization of skill usage in the pick-and-place task. The x-axis represents the skills reduced through PCA analysis, while the y-axis indicates the timesteps which the high-level policy operates.
30
25
Skill Length
20
5
Ant-Medium
Ant-Large
Kitchen
PP(ME)
PP(MR)
PP(RP)
Environment
Figure 10: Distribution relabeled skill lengths across various environments using our framework. These results emphasize the adaptability of DCSL in tailoring skill lengths to the specific demands of each task, improving flexibility and generalization horizoonizon RL tasks.
D.2 VISUALIZATION SKILL SELECTION
We visualized how skills are utilized in the pick-and-place task to verify whether DCSL effectively clusters similar behaviors into similar skills. Using PCA analysis, reduced the dimensionality of the skills selected by the high-level policy to one dimension. In each episode, the initial and target positions of the object varied. For more precise analysis, configured DCSL's high-level policy to sustain a selected skill for the duration of 10 low-level actions, matching the behavior of SPiRL. The results are presented in Fig. These figures illustrate that, despite variations in the object's initial and target positions, the patterns of skill usage in DCSL remain consistent across different scenarios. This consistency suggests that our method captures some level semantic context in skill utilization, even though further analysis is required for explicit interpretability. In contrast, SPiRL also exhibits instances of selecting similar skills in certain segments. For example, at timestep 2, the skills selected are closely clustered, likely corresponding to the robot arm's gripper performing the action of grasping the object. As previously mentioned, SPiRL embeds action sequences as skills, which explains why similar skills are selected in segments where the gripper's movements are identical. However, outside of these segments, SPIRL demonstrates less consistent patterns of skill usage compared to DCSL.
20

---

Published as a conference paper at ICLR 2025
Average Subtask
2-
key states (min length 5)
key states (min length 10)
key states 10 (min length 10)
0.5K
1.0K
2.5K 3.0K 3.5K Number training steps
1.5K
2.0K
Figure 11: The ablation study the number of key states for the Kitchen task.
D.3 SKILL LENGTH ANALYSIS
We analyzed the distribution of relabeled skill lengths produced by the DCSL framework across dif- ferent environments using violin plots (Fig. T0). These plots reveal that skill durations were dynam- ically adjusted according to state transitions and task complexity, rather than relying the initial lengths. the antmaze-mediue environment, skill lengths primarily ranged from 10 to 15 steps, reflecting medium-length behaviors suitable for the task. In contrast, skills in the antmaze-large environment were generally shorter, supporting the need for more frequent and precise decision- making. For more complex environments, such the kitchen and pick-and-place tasks (ME, MR, RP datasets), the skill length distribution was more diverse, indicating that DCSL effectively cap- tured both short and long action sequences necessary handle varied behaviors. This adaptability highlights the framework's 's ability to tailor skill durations to the specific demands of each task, en- hancing its flexibility and generalization in long-horizon horizon reinforcement learning scenarios.
D.4 ADDITIONAL ABLATION STUDIES
D.4.1 NUMBER OF KEY STATES
To address the impact of the number of key states DCSL's performance, conducted an ablation study the Kitchen environment. This study explores the relationship between the number of key states and the minimum skill length, which
interconnected parameters in our framework. We compared three configurations: (1) 4 key states with a minimum skill length of 5, (2) 4 key states with a minimum skill length of 10, and (3) 10 key states with minimum skill length of 10. The results, presented in Fig. [ show that using 4 key states with minimum skill length of achieved the highest average return. The superior performance of the 4 key states (min length 5) configu- ration suggests that allowing for shorter skills can capture more fine-grad fine-grained behaviors, which is particularly beneficial in manipulation tasks like those in the Kitchen environment. Interestingly, increasing the number of key states from 4 to 10 while maintaining minimum skill length of 10 performance. The minimal difference performance between
showed only a slight improvement the configurations with key states (min length 10) and 10 key states (min length 10) be at- tributed to the effectiveness of the similarity function in DCSL. This similarity function, learned through contrastive learning, allows the model capture the semantic context of behaviors regard- less the number of key states used. By focusing on state transitions, DCSL can effectively cluster similar behaviors into coherent skills, even with fewer key states.
D.4.2 ABLATION ON SIMILARITY THRESHOLD
To evaluate the sensitivity of our DCSL framework to the similarity threshold e, we conducted a comprehensive analysis in the pick-and-place environment using the Replay (RP) dataset (Fig. T2]
21

---

Published as a conference paper at ICLR 2025
1.2
-30
-
1.0
0
+5
0.8
Success Rate
+30
0.6
-0.2
0.5K
1.0K
1.5K
2.0K
2.5K
3.0K
3.5K
4.0K
Number training steps
Figure 12: The ablation study the similarity threshold for the pick-and-plase task.
Table 4: Comparison of the number of timesteps required to complete pick-and
Task
w/o Relabeling
with Relabeling
PP(ME)
75.1 + 16.9
80.1 + 13.7
PP(MR)
77.8 + 8.1 98.9 10.8
56.1 = 5.1
PP(RP)
64.4 + 16.3
We tested five different e values: -30, -5, O, 5, and 30. Our results demonstrated that the framework achieved high success rates across all threshold settings, indicating robustness this parameter. However, we observed that setting to 30 led to a slight increase in the number of training steps required for convergence compared to other values. This be attributed the stricter criterion imposed by a higher value for considering state transitions as part of the same skill during the length relabeling process. Consequently, this stricter threshold tends to generate shorter skills, which in turn necessitates more training steps for the high-level policy to learn effectively. This analysis provides valuable insights into the trade-off between skill granularity and learning efficiency in our framework, demonstrating its adaptability across different threshold values while highlighting the impact of skill length on downstream task learning.
D.4.3 COMPARISON OF TASK LEARNING EFFICIENCY WITH AND WITHOUT RELABELING
We conducted a comparison of the timesteps required for task completion across different datasets with skill length adjustment (Table 4 Interestingly, the results revealed that skills extracted from suboptimal datasets enabled tasks to be completed more efficiently compared to those extracted from the high-quality PP(ME) dataset. This finding suggests that, despite containing more irrele- vant actions, suboptimal datasets allow DCSL to capture greater diversity of behaviors as skills, ultimately contributing to more efficient task completion.
E LIMITATIONS
While DCSL demonstrates promising results across various environments, it also has several limi- tations that warrant further investigation. First, DCSL shows limited performance with -ased planning. Our experiments reveal that the variable skill lengths in DCSL can destabilize the plan- ning horizon in Cross-Entropy Method (CEM)-based approaches. This inconsistency in time scales reduces CEM's effectiveness, particularly in long-horizer tasks, leading to suboptimal performance compared to SAC-based methods. Second, DCSL's generalization to novel tasks remains limited. Although it performs well within the distribution of the training dataset, its ability to transfer learned skills to entirely new tasks or significantly different environments is constrained. This highlights the need for further research into more robust mechanisms for skill transfer. Additionally, combining
22

---

Published as a conference paper at ICLR 2025
approach with intrinsic rewards that aid exploration could be viable method for learning new skills through online interaction. Third, DCSL heavily depends on the diversity of the training dataset. Its skill learning is restricted to the behaviors present in the offline dataset, making it incapable of learn- ing skills that are not represented in the training data. This limitation suggests potential future work in incorporating online interaction through exploration rewards to enable the learning of new skills beyond the initial dataset. Fourth, the computational complexity of DCSL is another challenge. The processes of dynamic skill length adjustment and contrastive learning introduce additional computa- tional overhead compared to fixed-length skill learning methods, which could limit its applicabilitty resource-consee constrained environments real-time applications. Finally, there is an issue with the interpretability of the learned skills. While DCSL effectively clusters similar behaviors, the seman- tie meaning of these clustered skills is not always clear or interpretable. This lack of interpretability could pose challenges in applications where explainability is crucial. Addressing these limitations offers exciting opportunities for future research. Potential directions include developing more robust planning methods compatible with variable-length skills, enhancing skill transfer capabilities, and exploring hybrid offline-online learning approaches to broaden the range of learnable skills.
23


Reviews:
[{"review_id": "AFC6ZFta8Y", "reviewer": "Reviewer_fLcu", "summary": "Use discriminability to identify skills, then use a skill similarity function trained using contrastive learning with a state-skill embedding and a state embedding. The representation is then used as an input to the termination condition, which compares the current autoencoded state to the skill embedding. The model is then trained as a joint objective. Compute skill duration as the maximum number of steps where the skill similarity is greater than a threshold value.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Investigates some of the core questions in skill learning, including variable-length skills and skill differentiability.\n\nOffline skill learning is an increasingly appplicable field, especially as large robotics datasets become more prevalent\n\nProvides a mostly clear and understandable description of a complex method and motivation for each algorithmic choice.", "weaknesses": "Of the original claims made in the introduction, there is a disconnect between those and the empricial results. In particular, while it is clear that the skills appear to be more effective for downstream tasks the original claims do not appear to be verified. It is not as obvious that the skills capture ``more semantic context'', as the only experiment in this context is antmaze (Figure 6), and while this appears to show some separation, it is not particularly semantic. It would be more useful to provide some clearly semantic task transfer, such as opening multiple different drawers in franka kitchen with the same skill. Similarly, it is not clear the how the skill lengths are used for performance, in particular the effect of skill length relabeling appears to be marginally significant. It would be valuable to have some analysis indicating that for some tasks shorter/longer skills are assigned appropriately. Finally, it is not actually that obvious why this method takes advantage of the offline RL setting and could not be directly applied to unsupervised skill learning.\n\nThe method itself is straightforward, but there are a substantial number of models that must be trained together (two encoders in the similarity measure, and encoder-decoder in the state encoder, target state model, skill prior, policy). Training these together appears to have been done using a few objectives without any hyperparameters. In practice, combining these kinds of models often requires some kind of tuning, which seems like it would be a limitation. How is this done in practice for this method?\n\nIn the formulation of the skill embeddings, two randomly sampled intermediate states are used to identify the skill embedding. This seems like it could introduce a significant amount of variance. Also, it is not clear how that number of intermediate states is decided. A more careful analysis of this design choice seems appropriate (is this something used by related work? How does a different number of intermediate states compare? What are the theoretical ramifications of such a choice?) and is missing from the paper. \nThe level of comparison is a little bit limited. In particular, this method seems at least peripherially related to unsupervised skill learning work, so it seems appropriate to make at least some comparison to that work. I expect that without access to the advantage of offline data those methods would not perform particularly well, but in a domain like antmaze, they may exihibit better skill differentiation. \n\nAs mentioned before, a more in-depth analysis of skill length would be valuable in this work. Even in the domains where additional skill length showed performance benefit, it is not obvious why it would, since these are pick and place tasks. Some kind of analysis which indicated how much time was wasted without skill length relabeling would probably be demonstrative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5TooUSwVur", "reviewer": "Reviewer_pKgb", "summary": "The paper proposes DCSL - a method for learning skills based on state sequences from offline data, using a contrastive loss. A similarity function is parameterized by a neural network and learned, which is then used to cluster similar skills together. In addition, the approach also allows for flexible learning skills of various durations by relabelling the duration of each skill based on the same similarity function. The results show better performance on downstream tasks when these skills are used with a model-based or model-free high-level controller.", "questions": "## Questions\n1. The second contribution mentions that the similarity function “clusters semantically similar behaviors into the same skill category.” Just to clarify, is “skill category” used loosely here? As I understand the skills are continuous N dimensional vectors and similar behaviors (in terms of state sequences) will be encoded to similar skills, but there are no discrete groups or categories of skills?\n\n\n2. Are semantically similar skills close to each other in Z space? For example in Fig. 6c), would behaviors moving in more or less the same direction have similar skill value?\n\n\n3. The skill prior is used for both the embedding loss and to guide exploration for downstream tasks. How is this skill prior learned (as mentioned in line 687)?\n\n\n4. Have you done any ablation on the number of key states - is there a specific reason for choosing 4?\n\n\n5. It is a bit unclear to me how key states are sampled and associated with skills. Each trajectory tau_I contains T state-action pairs. \nDoes each trajectory have one or several skills associated with it? From line 4 on Algorithm 1 it seems that each trajectory has one skill associated with it and the skill duration H for each trajectory then changes based on the relabelling (Algorithm 2)? What if a single trajectory contains multiple behaviors? \n\n6. How does this method compare to works like OPAL [1] or to using unsupervised skill discovery with offline RL to learn skills from a dataset  [2], [3]? If possible, a comparison could strengthen the contributions.\n\n## Review summary:\nOverall I think this is a high quality work and the proposed method is promising. The paper is well-written and easy to follow (with a couple of caveats mentioned above). The main concern is that I think more comparisons should be made with other relevant methods which learn skills from offline datasets with or without behavior cloning.\n\nReferences:\n\n[1]\tA. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum, “OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning,” May 04, 2021, arXiv: arXiv:2010.13611.\n\n[2]\tS. Park, T. Kreiman, and S. Levine, “Foundation Policies with Hilbert Representations,” presented at the Forty-first International Conference on Machine Learning, Jun. 2024.\n\n[3]\tJ. Kim, S. Park, and S. Levine, “Unsupervised-to-Online Reinforcement Learning,” Aug. 27, 2024, arXiv: arXiv:2408.14785. doi: 10.48550/arXiv.2408.14785.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Skills are more general and less overfitting to specific types of behaviors due to the similarity function.\n- Dynamic relabelling allows for skills for different lengths and the paper shows cases where that's beneficial.\n- Significantly better performance than baselines on downstream tasks.\n- Seems more robust than baselines to lower quality demonstrations in the dataset as shown by the PP ME dataset.", "weaknesses": "- Comparisons are only done with variations of two methods (SPiRL and SkiMo), which makes it harder to judge the strength of the method.\n- Since the method uses Behavior cloning to learn the skills, I’m not sure how well it will perform in datasets with large amounts of sub-optimal data. While the results show that it outperforms BC-based baselines, there’s no comparison with non-BC methods like offline RL.\n- Qualitative comparisons (videos) of the behaviors would be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2v0DeSr2MN", "reviewer": "Reviewer_aqwT", "summary": "This paper investigates skill learning in reinforcement learning (RL) and proposes Dynamic Contrastive Skill Learning (DCSL), a method to learn skill embeddings based on state transitions. The approach uses contrastive learning to cluster semantically similar skills and dynamically adjusts skill length during the learning process. Experiments are conducted on Ant-Maze, D4RL Kitchen, and Meta-World Pick-and-Place environments, demonstrating that DCSL, combined with the downstream RL algorithm SAC, outperforms previous skill learning methods.", "questions": "1. **Performance of SPiRL in Figure 4(c)**: The performance of the SPiRL baseline appears significantly worse than reported in the original SPiRL paper on D4RL Kitchen tasks. Could this discrepancy be due to differences in the datasets used in D4RL Kitchen environments?  \n2. **Definition of training steps (Figure 4\\)**: In Figure 4, the x-axis refers to the number of training steps, but RL research typically focuses on environment steps to measure sample efficiency. Could you clarify the meaning of \"training steps\" in this context? Additionally, would it be possible to provide a comparison of different methods based on sample efficiency?  \n3. **Additional datasets in D4RL Kitchen**: The experiments in the D4RL Kitchen environment are conducted using only the mixed dataset, but this benchmark includes two additional datasets. Could you extend the comparisons to these datasets? Will DCSL maintain its advantage across different datasets?  \n4. **Meta-World experiments**: While DCSL performs well on the pick-and-place task in Meta-World, could you test it on more challenging tasks within Meta-World to fully validate its robustness and generalization?  \n5. **Ablation without similarity function**: In the ablation study, could you include a variant of DCSL that omits the similarity function? This would result in a method that relies purely on state-transition-based skill learning. How would this variant compare to SPiRL, which is action-based skill learning?  \n6. **Skill target state in continuous space (Line 262\\)**: The sentence mentions that a skill target state is considered \"reached\" during downstream execution, but how do you determine whether a target state is reached in continuous state spaces? Is there a threshold used to make this determination?  \n7. **Definition of skill trajectory (Line 248\\)**: The sentence refers to the \"trajectory of skill z.\" Could you provide a formal definition of what constitutes the trajectory of a given skill?  \n8. **Negative sampling in contrastive learning (Equation 5\\)**: In Equation 5, how is z′≠z determined, given that z lies in a continuous space? For any trajectory other than the anchor trajectory, the skill embedding z’ will always differ from z, so does this mean any state from other trajectories would serve as a negative sample?  \n9. **Sensitivity to similarity threshold ϵ\\\\epsilonϵ (Equation 9\\)**: How sensitive is DCSL’s performance to the value of the hyperparameter \\\\epsilon in Equation 9? A sensitivity analysis would be useful to understand the robustness of the method.  \n10. **Sparse reward problem (Line 369\\)**: The paper claims that DCSL addresses the sparse reward problem better, but there is no detailed explanation of the sparse reward issue in the Ant-Maze environments. Moreover, conducting experiments on more tasks with sparse rewards would strengthen this claim.  \n11. **Generalization to unseen tasks**: Can the skills learned during training be transferred to unseen tasks that are relevant to the training tasks? For example, SPiRL demonstrated generalization by applying learned skills to more complex mazes. Does DCSL exhibit similar generalization capabilities?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The approach is well-motivated, addressing important challenges in capturing semantically similar skills and adapting skill lengths across different tasks.  \n* The experiments across multiple benchmarks and tasks provide strong empirical support for the benefits of DCSL over baseline methods.", "weaknesses": "* The experimental and ablative studies could be more comprehensive, covering a broader range of tasks and datasets within D4RL and Meta-World.  \n* The paper’s writing could be clearer and more accessible (see clarification questions below).  \n* While the paper offers some theoretical insights in Appendix A, DCSL lacks a rigorous theoretical foundation with formal mathematical derivations and analysis.  \n* The paper does not discuss its limitations, which is essential for understanding the boundaries of its applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)