Learning Parameterized Skills from Demonstrations
Authors: Vedant Gupta
Haotian Fu
Calvin Luo
Yiding Jiang
George Konidaris
Venue: NeurIPS 2025 poster ()
Keywords: Skill Learning
Hierarchical Reinforcement Learning
Imitation Learning
Primary area: reinforcement_learning
TL;DR: We present an algorithm to learn parameterized skills from demonstration data and show that our approach results in improved adaptation to unseen tasks.

Abstract: We present DEPS, an end-to-end algorithm for discovering parameterized skills from expert demonstrations. Our method learns parameterized skill policies jointly with a meta-policy that selects the appropriate discrete skill and continuous parameters at each timestep. Using a combination of temporal variational inference and information-theoretic regularization methods, we address the challenge of degeneracy common in latent variable models, ensuring that the learned skills are temporally extended, semantically meaningful, and adaptable. We empirically show that learning parameterized skills from multitask expert demonstrations significantly improves generalization to unseen tasks. Our method outperforms multitask as well as skill learning baselines on both LIBERO and MetaWorld benchmarks. We also demonstrate that DEPS discovers interpretable parameterized skills, such as an object grasping skill whose continuous arguments define the grasp location.

Learning Parameterized Skills from Demonstrations
Vedant Gupta'
Haotian Fu'
Calvin Luo'
Yiding Jiang? *Carnegie Mellon University
George Konidaris
Brown University
Abstract
We present DEPS, an end-to-end algorithm for discovering parameterized skills from expert demonstrations. Our method learns parameterized skill policies jointly with meta-policy that selects the appropriate discrete skill and continuous pa- rameters at each timestep. Using a combination of temporal variational inference and information-the -heorrett regularization methods, we address the challenge of degeneracy common in latent variable models, ensuring that the learned skills are temporally extended, semantically meaningful, and adaptable. We empirically show that learning parameterized skills from multitask expert demonstrations signif- icantly improves generalization to unseen tasks. Our method outperforms multitask well skill learning baselines both LIBERO and Meta World benchmarks. We also demonstrate that DEPS discovers interpretable parameterized skills, such an object grasping skill whose continuous arguments define the grasp location.
Introduction
The standard application of reinforcement learning to long-horizon sequential decision-making problems often fails to leverage inherent behavioral patterns, leading to sample inefficiency. In contrast, humans exhibit a remarkable ability to extract and reuse strong priors from past experiences. For instance, training single task-specific policy to master an Atari game can demand over 10 million samples [T8] 9), whereas humans can achieve effective gameplay after merely episodes. A promising avenue for explicitly learning such priors from experience lies within the options framework (27). which aims to discover modular and temporally extended skills. These skills can then be flexibly reused and composed, facilitating generalization novel tasks and reducing the effective planning horizon.
While prior research has predominantly focused on learning either purely discrete continuous skills, these approaches possess inherent limitations. Discrete skills alone may lack the flexibility required for broad generalization, while continuous skills can be less structured and challenging to interpret. propose that learning parumeterized skills 4 5 which are discrete in nature but can be modulated by continuous arguments, offers synergistic combination of the benefits from both independent approaches. Parameterized skills retain the structured and interpretable nature discrete skills, while enabling flexible reuse in novel settings through continuous conditioning.
Consider the task of learning a skill to slice fruit. Generalization is crucial here, as no two fruits are identical, and the skill might need to be applied across diverse kitchen environments with varying tools. Learning distinct discrete skill for every possible scenario not only computationally intractable but also generalizes poorly to unseen situations. Conversely, learning a parameterized skill like slice_ fruit (x,y, ,2,, where the continuous parameters specify the slicing action based on particular instance, compactly represents a family of policies for fruit slicing. This can facilitate robust generalization to previously unseen slicing angles and accommodate different parameterizations for various fruits, tools, and kitchen conditions.
Website: bittssgooogg..cooo//////annanans Code; giihubbcom/gguubooDDERS Correspondence: vedaantguptta@@mail.ccomm
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we introduce Discovery of GEneralizable Parameterized Skills (DEPS), an algorithm for discovering parameterized skills from expert demonstrations in an end- to-end manner. DEPS trains the three-level hierarchy illustrated in Figure T (i) a dis- crete skill selector, (ii) continuous param- eter selector conditioned on the discrete skill, and (iii) subpolicy conditioned on both. A standard implementation of this approach is under-specifee and often susceptible to de- generate solutions that minimize the behav- ior cloning loss without acquiring meaningful skill abstractions [1]. To mitigate these de- generate solutions, DEPS incorporates several coormatiinntteoreele constraints and architec- tural choices, detailed in Section 4 These in- clude compressing the observation embedding before feeding it to the subpolicy networks limit information flow and predicting contin- uous parameters conditioned on each discrete skill rather than at every timestep. These design choices compel the model to rely on the latent variables to solve the task, resulting more robust, interpretable, and generalizable skills.
Shilk
Discrete Skill Policy
Invironment State
Continuous Parameter Policy
Low-lovel Action Policy
Compressed
One-dimensional State
Figure 1: Three-level hierarchy of DEPS. The dis- crete skill policy selects skill from the library given the full environment observation. Conditioned on that choice, the parrmmmeeeerrreteer policy outputs continuous parameters that modulate the chosen skill, tracing trajectory manifold (illustrated on the left). Finally, the low-level action policy, which sees only compressed one-dimensional robot state, produces the primitive action.
We evaluate the efficacy of DEPS in learning parameterized skills across two challenging multitask environments: LIBERO [6] and MetaWorld-v2 [30]. Our primary focus is on the rapid generalization capabilities of the learned skills, assessing their ability to adapt to novel tasks through finetuning with limited data. We demonstrate significant quantitative performance improvements over prior work in low-data regimes and provide qualitative visualizations of learned skills corresponding to fundamental actions like grasping, moving, and releasing objects. We find that DEPS consistently achieves the highest average success rate across various pretraining settings compared to existing methods, underscoring its ability learn flexible and gerh-performin skills.
2 Related Work
A straightforward approach to learning from multitask demonstration data is to train monolithic policy that maps state and task label to single action. However, previous works suggest that the difficulty of sequential decision-making problem scales with the problem horizon [24 10 21). To improve sample efficiency and generalization, many methods hence focus on learning temporally- extended action abstractions called options/skills [27]
There is a large body of work focusing on learning skills from demonstrations [4 19
15
13
26 25
II| 33 6). Notably, Shankar and Gupta [25] introduce temporal variational inference to learn skills from demonstrations. However, their method is restricted to low-dimensional state spaces. Prior work also found multi-level hierarchy be useful for skill learning [3] 7 20 g 22 All the methods above are restricted to learning a fixed number of discrete skills or continuous skills.
Early work on parameterized skills, including work by da Silva et al. (4), proposed the construction of parameterized skills by analyzing the structure of policy manifolds. However, this required labeled parameters of tasks for training. More recent methods, such as LOTUS 28) and EXTRACT [I] learn coal-conditioned discrete skills by first clustering demonstration trajectories into different discrete skills using pretrained models such as VLMs, and then learning parameterized policies corresponding to each cluster. However, this approach implicitly makes the assumption that the same discrete skill takes place in visually similar environments, which may not hold in practice. Fu et al. S propose to learn parameterized through online meta-learning. However, their approach requires several separate stages of training online and needs to manually design a set of tasks for each skill.
Using parameterized skills as the new action space, large body of previous work focuses on Parameterized-action Markov Decision Process (PAMDP) [17 a predefined library parameterized skills, whereas our three-level hierarchy learns every policy layer end end-to-end directly from trajectory demonstrations.
2 29 32 23 These works presume

---

Low-level Policy
(at Sygs O1g-
Variational Fncoder 4: k,),)
Legend:
Observed variables
Unobserved variables
Discrete and Continucus Policy xf1R, Bign (1n-1-
OOO Inferred variables
(2 Fis &,,-1
K Discrete skills
2p Continuous parameters
The underlying probabilistic graphical model of DEPS. The variational encoder has access all the information of the trajectory from history to future. The discrete and continuous policy works as the high level policy that infers the parameterized skills based on information from previous timesteps. The low-level subpolicy infers actions based on the parameterized skills as well as the current state. Variables observable by each model are shaded in gray.
Figure 2:
3 Background
We begin with deriving training objective for learning continuously parameterized discrete skills from provided demonstration data. Consider of tasks T, which assumed to share consistent state space, action space, and transition function. In the Learning from Demonstration setting, we assume access training dataset of task demonstrations D {2: each demonstration is denoted as T; (T) where т, {st, a, description associated with the trajectory. Our goal is to learn policy that can not only perform well for tasks covered within the training dataset Teeen U(4) but also generalize novel tasks Tunasen = T Teeen-
M; at the trajectory and I E is the task goal
For complex, long-horizon tasks T, we assume natural decomposition into a sequence of discrete skills x = (ki J where E K represents set of distinct skills. These discrete skills are further conditioned on continuous-valued parameters E Rd. Our hierarchical approach comprises three main components:
1. A discrete policy N (kr 91.r, 01:t-1,() that predicts the discrete skill.
2. A continuous policy π 22; 81:1; 01:-0111,, 1) that predicts the continuous parameters for the chosen discrete skill.
3. A subpolicy T~ (az $1st, 01:1-11111... kr, =x) that generates an executable action based on the current discrete skill and its continuous parameterization.
During interaction, the agent first samples a discrete skill k, ~ K then a continuous parameter 2t 2 and finally an action a; A
4 Discovery of GEneralizable Parameterized Skills (DEPS)
We introduce DEPS, framework for learning continuously parameterized skills from demonstrations. DEPS jointly trains a hierarchical policy by maximizing a variational lower bound on the likelihood of demonstrated trajectories. This section details the derivation of our training objective using temporal variational inference and then describes our approach to modeling skills as parameterized trajectory manifolds, key component for generalization.

---

4.1 Variational Inference for Parameterized Skill Discovery
Consider sampled demonstration T {r. E D, where {st, at JM and some corresponding
sequence of discrete skills ベ {&; j and continuous parameters ( = {4 the joint likelihood of these sequences p(r, n, (,1) as follows:
M Then, we can write
((T,,,,S,1) p(s1, ()
II
(T K (kals12, a1 hi 2 1) (ze 81:1 014-1, kidt 214-1,
π (aylsaa, 01--1, () ( at)).
To train policies x 2², and T that are autoregressively usable at inference time, each policy can only be conditioned on prior observations, actions, discrete skills, and continuous parameters. line with prior literature, aim to maximize the objective (rr,))= p(T, ()]. However, calculating log pgr(( 1) exactly involves an intractable marginalization over all possible sequences of discrete and continuous skills. We use temporal variational inference [25] to estimate ( [logp(T, !)] while maintaining the autoregressive structure of the policies learned. We introduce variational distribution q(k,, <|T, which is meant to approximate p(k, CIT, (). Due to the non-negativity of KL divergence:
(rr [log p(T, 2 ((+0)~ log
Dlpp,, S r,D)) = L.
Note that the bound above is tight when g(x, <lT, 1) = p(k, 11T,1). Now,
M
(r,)) (1..)))ox(((1..) logp(s1, 1) +
Σ log π (Ne (H + log 7² (21 H()
+ log (at H(,a) + log p(gp( 88, at) - log ((1 < T,2)
(2)
where H ($1 d1æ-1: ki--1, 11. H ($1.2, disss kiets -1 (Bias d1--11 kr, 2),, and where the last step uses the joint distribution derived in Equation |
1). and HPd
Keeping computational efficiency during training in mind, to enable the discrete skills and con- tinuous parameters for each timestep to be sampled in parallel, we assume that conditional on previous states and actions, (ke, 24) is independent of (he, 24) for i < This allows us to rewrite Tt (ka] Q1:-1, k14-1, 21.--1.)) as T (kg|s1:t. d1æ-1,.) and n (2, 01:-11,k1.t æ1:-1,1) as π (2e(81:: 41:-11, 1, k 1). Since the subpolicy a only conditioned on the current continuous and discrete parameters, one can show that for any fixed ΛA the joint likelihood in Equation can be maximized by the simplified expressions for Λ R and я _ above. Our empirical results in Section also validate that this assumption does not prevent useful skills from being learned.
Since (Kx, () is meant to approximate p(K,, {/T), one can now rewrite q(F,, <\7, as follows:
M II ((())),)((2) t=1
(I.,))
Using Equation and the observation that the dynamics p(*++ |s,, az) and the joint distribution of the task and initial state, p(#1, 1), do not affect the gradient of our loss, we can rewrite Equation 2 the following, cleaner, form:
in
Σ log n (as (a. H - B Σ Det (a(ke 1, (H(
C = E(r.)) (x.0))((x.)).))
-Brene(helt.)) DKL (g(za T, Ke, 1 || (- H4 [(
(4)
where H/AAA = (St.r: -1, 2,,, th (812) -1a-1, 1), H²? ($1.r) a1.--1, k.
We represent the discrete skills using categorical distributions and the continuous parameters using Gaussian distributions. Hence, the KL.-divergence terms above can be calculated exactly. We therefore
minimize this loss to learn the discrete and continuous policies well as the low-level policy.

---

4.2 Skills as Parameterized Trajectory Manifolds
common failure mode in learning skills from demonstrations for the low-level policy minimize behavior cloning loss without discovering meaningful, generalizable skill abstractions. This often occurs when state spaces for different tasks have minimal overlap, allowing trighccppcity policy to memorize task-specific behaviors in different state space subsets. To address this, DEPS introduces an information bottleneck in the subpolicy and models skills as manifolds of parameterized trajectories.
Information Bottleneck via State Compres- sion. Our core strategy is to provide the subpol- icy (aaals ke, ze) with a compressed, lower- dimensional version of the current state, 5, in- stead of the full observation St. This aims to:
Low Low-Level Policy
Skill Library
Diu
Robot State
1. Enhance State-Space Overlap and Gen- eralization: Mapping raw states into shared, compressed space increases input distribution overlap across tasks, resulting in policies that generalize novel tasks within or near this compressed manifold. 2. Force Reliance on Latent Variables: A highly compressed state s/ alone is often insufficient to determine the correct action at. The subpolicy must rely on the discrete skill & and continuous parameter 2t to re- solve ambiguity, compelling these latents to encode crucial skill-related information.
One-dimensional
Action
Compression
Figure 3: Skills as Parameterized Trajectory Man- ifolds. We hypothesize that single skill corre- sponds to a family of parameterized trajectories. A one-dimensional state representation indexes into this generalizable manifold to predict actions.
Conceptualizing Skills. What distinguishes a versatile "skill" from a collection of disparate be- haviors? We posit that a versatile skill exhibits lower-dimensional structure. Trajectories from executing a coherent skill (e.g.. "pick up toy" from various positions) likely lie on or near a common low-dimensional manifold, with variations corresponding to different paths or modulations. In contrast, trajectories from disparate behaviors (e.g., "clean dishes" and "pick up toys") belong to different manifolds. parameterized skill (ks 2) thus selects and refines a family of trajectories sharing common structure. Movement along such a trajectory can be indexed by dimeennniional variable (e.g.. progress), motivating aggressive state compression.
Projective State Compression to One Dimension. Given the discrete skill k, continuous parameter 2,, and a feature vector sproj from the raw observation 8t (e.g., robot end-effector coordinates), we compress sproj into a scalar s//:
s/ = tanh (W(ke,e)) proj + b.p...))
Here, w(k..))) (a vector) and 2()). (a scalar bias) are outputs of a small MLP, Seompress (kr, z(). The vector W(k,,z)) defines a skill-specifie projection axis. The tanh activation normalizes s to [-1, 1]. ensuring bounded index. This ID compressed state s/, along with k, and 2t, fed to a feedforward network to output action distribution parameters. This forces kn and 2 to encode task-relevant information not captured by the highly compressed s/c
4.3 Overall Framework and Flow
The DEPS framework consists of four interconnected neural network components: a variational network (q), discrete policy network (IT), continuous policy network (T²), and a subpolicy network A). These are trained jointly using the objective in Equation
The variational network qts,, Çlr, !) is implemented as bidirectional GRU. It processes entire demonstration trajectories infer posterior distributions over skills k; and parameters 2,, To encourage temporally extended skills, continuous parameters are predicted per discrete skill k rather than per timestep within a trajectory. This prevents the continuous parameters from rapidly changing, and potentially embedding the action to take at each timestep. Additionally, to prevent the continuous parameters from overfitting to specific trajectories, we introduce a Skill Purameter Norm Penalty to discourage magiitud continuous parameters.
5

---

The discrete policy x (kg (k:,, 01:t-1.() and continuous policy x² (2g($12 (1,-1,k,,1) form the high-level policy. practice, the entire conditioning context is often unnecessary; our experiments, we find that removing the history of actions from x and z still produces high empirical performance while simplifying the implementation. Both x and π² are implemented as unidirectional GRUs that process the history of states. Their outputs, combined with the current task (and k, for x r are passed through MLPs to predict the current skill Ke and its parameters 2t autoregressively, The low-level subpolicy n (a. s/,, Ke, z4) executes the chosen skill.
Unlike the variational network's 's strategy of predicting one set of continuous parameters per skill instance for an entire trajectory. the continuous policy network π predicts continuous parameters for every timestep. This allows for refinement of the continuous parameter 2 at each step based on the most recent observation, enabling more reactive behavior during inference.
Crucially, DEPS incorporates an information asymmetry: the high-level policies (variational net- work, discrete policy, and continuous policy) have access to both rich image observations available and the robot's proprioceptive state. In contrast, the low-level subpolicy π only observes the robot's proprioceptive state (which is then compressed into s)). This restriction forces the subpolicy to rely on (k,, 2)) for skill-specific guidance and prevents it from overfitting to visual details that might hinder generalization. By learning to operate from a more abstract, compressed state representation conditioned by the latent skill variables, the subpolicy encouraged to learn more generalizable be- haviors. Appendix A contains additional details, along with the pretraining and finetuning procedures.
5 Experiments
We empirically evaluate the ability of DEPS to generalize to novel tasks with minimal finetuning across two challenging multitask environments, LIBERO [66] and MetaWorld-v? [30). Within each benchmark, we first pretrain on a series of tasks to learn parameterized skills. The pretrained model is then finetuned an unseen task for 500 gradient steps, and its performance is evaluated with 20 rollouts after every 50 steps of finetuning (for total of 10 evaluations). Each rollout is assigned a binary reward based on whether the task was successfully completed. We evaluate the performance of our approach on series of novel and previously used evaluation sets. For each set of evaluation tasks, we report two metries:
Mean Success:
The average success rates across all unseen tasks, averaged across all 10 evaluations. This measures the algorithm's 's consistency.
Mean Highest Success:
The highest success rate across all 10 evaluations of a given task, averaged across all unseen tasks. This measures the performance of the algorithm when using the optimal number of gradient steps during finetuning for each specific task. Previous work [66 33
uses this metric evaluate performance.
We report both metrics along with their standard deviation across evaluated seeds. We find that DEPS displays consistently superior rapid generalization to unseen tasks across diverse evaluation settings.
5.1 Baselines
We compare our approach against the following baselines (1) A multitask behavior cloning network (BC), (2) the same multitask BC architecture, but without any pretraining (BC-Untrained)) and (3) PRISE B3), the state-of-the-ar baseline that learns action "tokens" and then applies byte pair encoding to learn common action sequences (PRISE). Information on the specific hyperparameters used and steps taken to ensure a fair comparison across baselines can be found in Appendix C
5.2 Experiment Setting
LIBERO LIBERO is a multitask benchmark involving various object manipulation tasks with robot arm (visualizations of example tasks are provided in Figure 4 across visually diverse environments. To pre-train model architectures, we use 80 tasks from LIBERO-90 using the offline dataset provided by Liu et al. [16]. For each of the 80 pretraining tasks, the dataset provides 50 demonstrations collected using human tele-operation. The state-space for each task includes images from two different viewpoints, a 9-dimensional vector representing the robot's state, and a language description of the task at hand. The action space consists of a 6-dimensional real-valued vector representing desired arm movements, along with a binary variable to open/close the gripper. We perform 20 passes over the pretraining data for each method.

---

Table 1: Average success rate across evaluation settings LIBERO and MetaWorld-v2. All results are averaged across 5 seeds.
Evaluation Set
Algorithm
Mean Success
Mean Highest Success
DEPS PRISE
0.34 = 0.08
0.66 +: 0.12
0.10 = 0.09
0.27 = 0.23
LIBERO-OOD
BC
0.15 + 0.04
0.36 + 0.08
BC-Untrained
0.01 = 0.00
0.08 # 0.02
Figure 4: Images of example
DEPS
0.08 + 0.04
0.24 +: 0.09
LIBERO-10
PRISE
0.02 0.02
0.07 + 0.06
tasks from LIBERO
BC
0.07 = 0.02
0,18 + 0.03
BC-Untrained
0.00 : 0.00
0.01 # 0.00
DEPS
0.26 +: 0.03
0.49 1: 0.03
LIBERO-3-shot
PRISE
0.07 + 0.07
0.19 + 0.14
BC
0.11 0.05
0.22 0.08
BC- BC-Untrained
0.01 = 0.00
0.03 + 0.01
DEPS
0.45 1: 0.03
0.65 0.03
Vanilanlla
PRISE
0.21 0.07
0.33 4
BC
0.35 0.02
0.51 1: 0.01
BC- BC-Untrained
0.25 + 0.02
0.45 + 0.05
DEPS
0.32 1: 0.03
0.53 1. 0.03
MW-PRISE
PRISE
0.06 0.02
0.17 + 0.05
Figure 5: Images of example
BC
0.25 0.02
0.41 +: 0.03
tasks from Meta W2
BC-Untrained
0.12 1: 0.01
0.29 1: 0.01
Pretrained checkpoints are then evaluated the following three test settings:
LIBERO-OOD:
10 unseen tasks from LIBERO-90 involving previously unseen environments and objects, making it strong test of generalization to distr-iittiibttion tasks. Descriptions of the tasks in this set can be found in Appendix D Each task comes with 50 expert demonstrations. LIBERO-10: The standard LIBERO evaluation dataset, consisting of long-horizon tasks, which are mostly concatenations of tasks seen in the pretraining set, measuring the ability of an algorithm accurately complete in-distribution but long-horizon horizon tasks. Each task comes with 50 expert demonstrations
LIBERO-3-shot: This dataset consists of the tasks in LIBERO- OOD but with only 3 demon- strations per task, testing the ability to successfully learn new tasks with minimal data.
Meta World-v2 MetaWorld-v2 multitask benchmark involving various object manipulation tasks with a robot arm (visualizations of example tasks are provided in Figure5). To evaluate perfor- mance on MetaWorld-x we utilize the provided expert scripted policies provided in Meta World, collecting 50 demonstration trajectories for each task. The state-space for each task includes an image and an 8-dimensional vector representing the robot's state, while the action space is 4-dimensional real-valued vector. We pretrain each method on set of 10 tasks, performing 40 passes over the training data. We then evaluate the performance of pretrained checkpoints on two different evaluation sets described below (information on the specific tasks used for pretraining and finetuning can be found in Appendix D). Due to the shorter average time horizon and lower task diversity compared to LIBERO, we focus on the 3-shot fine-tuning performance in both evaluation sets.
MW-Vanilla: A standard set of 5 unseen tasks proposed in MetaWorid. MW-PRISE: 5 unseen tasks as used in PRISE, consisting of longer horizon tasks on average.
5.3 Primary Experimental Results
Table [ the mean success and mean highest success rate with minimal finetuning on each of the evaluation datasets outlined in Section 5.2) DEPS achieves a sigmificantly higher mean success and mean highest success rate across evaluation regimes, showing the effectiveness of parameterized skills and observation-spaa which tests rapid generalization to out-of df-triiiibbtiion environments and objects, DEPS achieves a mean success rate of 0.34, which is more than double that of the standard BC approach (0.15) and more than triple the performance of PRISE 10). Even in the extremely data-scarce LIBERO-3-shot setting, DEPS maintains robust performance (0.26 mean success), outperforming both PRISE (0.07) and standard BC (0.11) by substantial margins. DEPS maintains its advantage across both evaluation sets in Meta HerldWrld-y2 v2, where tasks have generally shorter horizon.
compression in learning generalizable abstractions. Notably, in LIBERO-OOD.
7

---

LINGGROOBUO
MR-Abnilh
Table 2: Robustness of parameterized skills different total pretraining epochs (PT). Results 50-shot finetuning over LIBERO-OOD and are averaged across 3 seeds.
EI2'5 EDkn))
DC
Algorithm
Mean Success
Mean Highest Success
DEPS
+ 0.09
PAISE
0.02 juplo 0.05] 0.07 +: 0.03
0.11 1: 0.13
240 Trasning Staps
130
350
200
BC
0.30 + 0.08
Iminiin
Figure 6: Average success rates as a function of the number of finetuning gradient steps taken for LIBERO-OOD (left) and MW-Vanille (right).
DEPS
0.39 1: 0.02
0.75 1: 0.01
PAISE
0.12 + 0.15
0.27 = 0.33
BC
0,10 0.06
0.30 +: 0.12
DEPS
0.39
0.04
15
PAISE
0.12 1:12::00 0.14
0.33 + 0.26
BC
0.12 0.06
0.32 = 0.07
The relatively poor performance of PRISE, despite its strong results in Zheng et al. [33] highlights the challenges of rapid generalization compared the extensive fine-tuning procedures used in prior work. Similarly, the near-zero performance of the untrained BC baseline on LIBERO tasks (0.01 mean success LIBERO LIBERO-OOD) underscores the importance of pretraining for successful adaptation in complex environments.
Figure Glsows the mean success rates on LIBERO-OOD and MW-Vanilla function of the number of fine-tuning gradient steps taken (averaged across seeds). DEPS outperforms other baselines irrespective of the number of gradient steps taken. These results collectively demonstrate that DEPS provides a robust foundation for rapid generalization across diverse robotic manipulation tasks.
5.4 Robustness to Pretraining Amounts
While the results in Section 5.3 use 20 epochs of pretraining for LIBERO and 40 epochs of pretraining for MetaWorld-v2. we also evaluate the performance our approach on smaller pretraining amounts to evaluate its data efficiency and robustness to different pretraining amounts. Table 2 shows the performance of DEPS and evaluated baselines on smaller pretraining amounts. DEPS consistently outperforms evaluated baselines across pretraining amounts. This same observation also holds across Meta
MetaWorld World (results can be found in Appendix E). Notably, with limited pretraining (e.g. epochs), the margin between our method and evaluated baselines increases. This suggests that, in addition to improving generalization unseen tasks, learning parameterized skills with compression might also increase the data efficiency of pretraining.
5.5 Additional Quantitative Results
We provide additional experimental results in the Appendix that suggest that (i) compression 1D state is essential to DEPS' performance (Appendix [.. (ii) changing the limit on the maximum number of discrete skills considerably improves the performance of DEPS. suggesting improvements over the presented results may occur with a hyperparameter sweep (Appendix G). (iii) learning only discrete or only continuous skills does not replicate DEPS' performance (Appendix HH). and DEPS maintains its performance advantage on increasing the number of finetuning gradient steps (Appendix IT.
5.6 Qualitative analysis of Learned Parameterized Skills
In addition to its strong quantitative performance, DEPS also learns interpretable parameterized skills, with discrete skills and continuous parameters encoding skill-relevant information. To this end, we provide visualizations showing the (i) segmentation of tasks into intuitive discrete skills, (ii) smooth variations in the skill policy on varying the continuous parameter (iii) high overlap in the continuous parameters used across tasks and (iv) monotonicity in the learned compressed state embeddings.
Analysis of Learned Trajectory Segmentations. We find that DEPS discovers intuitive param- eterized skills. In LIBERO, the learned skills correspond to primitive behaviors such as grasping, moving, and releasing objects. The same discrete skills are used consistently across environments and object types, with changing continuous parameters to encode task-specific details. We provide a representative visualization of the decomposition of trajectories into skills Figure 7 and more visualizations can be found on project website

---

grasp abject
move abj_cbeeettttte
release object
close drawer
Task: put the butter at the back in the top draver of the cabinet and close it
grasp object
objeet
release _bbject
close door
Task: (zomove the obstacle and) close the microwave dooz
Figure 7: Visualization of the discovered discrete skills for two sample tasks. Images represent the points at which the sampled discrete skill changes, with the discrete skill taking place between two images indicated on top (DEPS labels discrete skills as integers, which are visible in the figure. The natural language skills were selected to be consistent with the discovered segmentations of the demonstration trajectories). We find that general skills corresponding to grasping, moving, and releasing objects learned and robustly applied across visually diverse tasks. Furthermore, for less frequent subtasks such as closing drawers and microwaves, independent discrete skills are discovered.
Analysis of Learned Continuous Parameterizations We find that changing the continuous pa- rameters provided to a single discrete skill results in smooth variations in the resulting policy. For example, changing the continuous parameters the discovered skill to grasp objects smoothly varies the resulting grasp location. Additionally, for a given skill, we find that there is overlap in the continuous parameters used for different tasks, suggesting that the continuous parameters encode skill-relevant information, not task or environment relevant information. Visualizations of this can be found on our project website.
Figure 8: Plot of the learned 1-dimensional compressed state indices (y-axis) with respect the time-step (x-axis). Each plot represents a single trajectory from a different task in LIBERO-90, and each colour corresponds to different discrete skill. We find that the compressed state changes monotonically within discrete skill, suggesting its use as an index.
Analysis of Learned Observation Compressions We verify whether the learned 1D compressed states function "indices" into trajectory outlined Section 4.2) Figure 8 shows the learned compressed state for single trajectories as function of the timestep. We find that the state embeddings for a single skill monotonically increase/decrease at roughly uniform rates as the timestep increases, as would be expected from an index into a trajectory. Furthermore, we find sudden changes in the learned compressed state when the selected discrete skill and continuous parameter switches, which consistent with expectation that the index should "reset" changing the trajectory represented by the current parameterized skill.
In Appendix we provide an ablation of the performance of DEPS with and without compression dimenninnion state. We find that our compression method significantly improves the task- generalization ability of the learned parameterized skills.

---

6 Conclusion
We present DEPS, an end-to-end method to learn parameterized skills from demonstration data. To do this, initially derive a loss function using temporal variational inference; however, simultaneously, we note that directly utilizing this loss function admits degenerate solutions. To avoid this, we utilize series of information-theoretiic methods. Notably, we present novel view of skills families of similar trajectories, motivating the aggressive compression of observations to one-dimensional "indices." " We evaluate the resulting algorithm across diverse and rigorous set of evaluation using LIBERO and Meta World-v2, measuring its performance on unseen tasks under (i) different pre- training/finetuiing task splits, (ii) different pretraining budgets (iii) different finetuning budgets (iv) different amounts of finctuning data and (v) different measures of success (Mean Success and Mcan Highest Success). We find that DEPS consistently outperforms existing baselines across evaluation settings. The performance improvement offered by DEPS is especially notable when evaluated on outto--distribution tasks (LIBERO-OOD, Table [ low-data regimes (LIBERO-3-shep Table and limited compute availability (Table 2 We discuss limitations of our approach in Appendix []] We believe our approach shows promise towards better data-efficient task generalization in roboties.
Acknowledgments and Disclosure of Funding
This project was supported by ONR REPRISM MURI N00014-24-1-2003 and ONR grant N00014- 22-1-2592. YJ was supported by the Google PhD fellowship. This work was conducted using computational resources and services at the Center for Computation and Visualization, Brown University.
We thank the anonymous reviewers of this work for their helpful comments, which helped us strengthen and clarify this work, We would also like thank the authors of LIBERO and MetaWorid for developing and open-soureing the benchmarks used in our experiments.
References
[1] André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Jonathan J. Hunt, Shibl Mourad, David Silver, and Doina Precup. The option keyboard: Combining skills in reinforcement learning. In NeurIPS, 2019.
[2] Craig J. Bester, Steven James, and George Dimitri Konidaris. Multi pass genetworks for deep reinforcement learning with parameterised action spaces. ArXiv, abs/ 1905.04355. 2019.
[3] John D. Co-Reyes, Yuxuan Liu, Abhishek Gupta, Benjamin Eysenbach, Abbeel, and Sergey Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings. In ICML, 2018
[4] Bruno Castro da Silva, George Dimitri Konidaris, and Andrew G. Barto. Learning parameterized skills. In Proceedings of the 29th International Conference Machine Learning. ICML 2012, Edinburgh, Scotland, UK, June 26- July 1, 2012. icml.ec / Omnipress, 2012.
[5] Haotian Fu, Shangqun Yu, Saket Tiwari, Michael Littman, and George Konidaris. Meta-learning parameterized skills. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 10461-10481, PMLR, 2023.
[6] Haotian Fu, Pratyusha Sharma, Elias Stengel-Eskin, George Konidaris, Nicolas Le Roux, Marc- Alexandre Côté, and Xingdi Yuan. iaoguage-guided skill learning with temporal variational inference. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[7] Carles Gelada. Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G. Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97
of Proceedings of Machine Learning Research, pages 2170-2179, PMLR, 2019.
10

---

[8] Anirudh Goyal, Shagun Sodhani, Jonathan Binas, Xue Bin Peng, Sergey Levine, and Yoshua Bengio. Reinforcement learning with competitive ensembles of information-coonsrainee primi- tives. ArXiv, abs/1906. 10667, 2020.
[9] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[10] Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 international conference on autonomous agents and multiagent systems, pages 1181-1189 2015.
[11]
Yiding Jiang, Evan Liu, Benjamin Eysenbach, Zico Kolter, and Chelsea Finn. Learning options via compression. Advances in Neural Information Processing Systems, 35:21 184 199, 2022.
[12] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14- 1-1 2014, Conference Track Proceedings. 2014.
[13] Thomas Kipf. Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzabee Edward Grefenstette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, pages 3418-3425 PMLR,, 2019.
[14] George Dimitri Konidaris, Scott Kuindersma, Roderic A. Grupen, and Andrew G. Barto. Robot learning from demonstration by constructing skill trees. Int. J. Roboties Res., 31(3):300-375, 2012.
[15] Sanjay Krishnan, Roy Fox, lon Stoica, and Ken Goldberg. Ddco: Discovery of deep continuous options for robot learning from demonstrations. In Conference on robot learning, pages 418 -477. PMLR, 2017.
[16] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776-44791, 2023.
[17] Warwick Masson, Pravesh Ranchod, and George Dimitri Konidaris. Reinforcement learning with parameterized actions. In Dale Schuurmans and Michael P. Wellman, editors, AAAI, 2016, pages 1934-1940, 2016.
[18] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[19] Scott Niekum, Sachin Chitta, Andrew G. Barto, Bhaskara Marthi, and Sarah Osentoski. Incre- mental semantically grounded learning from demonstration. In Paul Newman, Dieter Fox, and David Hsu, editors, Robotics: Science and Systems IX, Technische Universität Berlin, Berlin, Germany, June 24- June 28, 2013, 2013.
[20] A. H. Qureshi, Jacob J. Johnson, Yuzhe Qin, Taylor Henderson, Byron Boots, and Michael C. Yip. Composing task-agnostic policies with deep reinforcement learning. In ICLR, 2020.
[21] Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. Advances in Neural Information Processing Systems, 33:2914-2924, 2020.
[22] Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Manfred Otto Heess, and Raia Hadsell. Learning transferable motor skills with hierarchical latent mixture policies. ArXiv, abs/21122 12,006,,, 2021.

---

[23] Martin A. Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Manfred Otto Heess, and Jost Tobias Springen- berg. Learning by playing solving sparse reward tasks from scratch. ArXiv, abs/ 10211802.15566 2018.
[24] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635 JMLR Workshop and Conference Proceedings, 2011.
[25] Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. In International Conference on Machine Learning, pages 0624-8633. PMLR, 2020.
[26] Mohit Sharma, Arjun Sharma, Nicholas Rhinehart, and Kris M. Kitani. Directed-ing info GAIL: learning hierarchical policies from unsegmented demonstrations using directed information. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[27] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2): 181-211, 1999.
[28] Weikang Wan, Yifeng Zhu, Rutav Shah, and Yuke Zhu. Lotus: Continual imitation learning for robot manipulation through unsupervised skill discovery. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 537-544. IEEE, 2024.
[29] Jiechao Xiong, Qing Wang, Zhuoran Yang, Peng Sun, Lei Han, Yang Zheng, Haobo Fu, Zhang, Ji Liu, and Han Liu. Parametrized deep q-networks learning: Reinforcement learning with discrete-come costtiou- hybrid action space. ArXiv, abs/1 18 11.066394, 2018.
[30] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learming, pages 1094-1100, PMLR, 2020.
[31] Jesse Zhang, Minho Heo, Zuxin Liu, Erdem Biyik, Joseph J. Lim, Yao Liu, and Rasool Fakoor. EXTRACT: efficient policy learning by extracting transferable robot skills from offline data. In Pulkit Agrawal, Oliver Kroemer, and Wolfram Burgard, editors, Conference Robot Learning, 6-9 November 2024, Munich, Germany, volume 270 of Proceedings of Machine Learning Research, pages 1148 1172. PMLR, 2024.
[32] Renhao Zhang, Haotian Fu, Yilin Miao, and George Konidaris. Model-based reinforcement learning for parameterized action spaces. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[33] Ruijie Zheng, Ching-An Cheng, Hal Daumé III, Furong Huang, and Andrey Kolobov. PRISE: I1m-style sequence compression for learning temporal action abstractions in control. In Forty- first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27. 2024. OpenReview.net, 2024.
12

---

A Implementation Details
A.1 Variational Network
The variational network learns the distribution q(k, SlT, l). We implement this as a bidirectional GRU that takes as input the encoded states and actions corresponding a sampled demonstration trajectory T as well as a task ID (or the provided task embedding in the case of LIBERO), denoted by I. At each timestep i. the outputs of the GRU are passed through prediction head predict a categorical distribution of the discrete skill at that particular timestep t. A separate output head is used to predict the continuous parameters. However, to prevent the continuous parameters from changing at each timestep, we predict the continuous parameters for each discrete skill instead of each timestep. This "restricts" " the amount information conveyable through the discrete skills and continuous parameters that encode a trajectory. forcing the subpolicy networks to learn temporally extended policies. We model the continuous parameters as Gaussian distributions using the reparameterization trick 12
Once the discrete skills are sampled, they can be used as indices into the set of predicted continuous parameters choose the associated continuous parameter for each timestep.
As training progresses, we observe that the average distance between continuous parameters increases, possibly indicating overfitting. To encourage compact and generalizable representations, we introduce a Skill Parameter Norm Penalty that discourages urge-magnitude continuous parameters. Formally,
>M we add the following regularization term to the loss in Equation 4 Lnorn = Apor Et=1 || where 2, denotes the continuous parameters active at timestep (corresponding to the skill ki)) as predicted by the variational network for that skill instance, and A is regularization coefficient.
A.2 Discrete and Continuous Networks
DEPS implements the skill distribution x. (kals1:; (1:-1;11) and parameterization distribution x² (z (81:, 01:1-1, with a discrete and continuous neural network, respectively. Both mod- els have the same architecture, implemented as a one-directional GRU, which takes in as input the history of states. The output of the GRU at each timestep is then concatenated with the task ID and passed through an MLP: for the discrete skill network, this MLP predicts a categorical distribution with K possible values, from which k2 is sampled at each timestep. For the continuous skill network this MLP predicts set of parameters for each of the | skills. In practice, we find that is sufficient to only predict the mean for each dimension of the continuous parameters, i.e. |K] x d values for a d-dimensional continuous parameterization. We then index into the parameters corresponding to the
sampled discrete skill k; at each timestep.
Unlike the variational network's strategy of predicting set of continuous parameters per skill instance for an entire trajectory, the continuous policy network n² predicts continuous parameters for every timestep. This allows for refinement of the continuous parameter 2 at each step based on the most recent observation, enabling more reactive behavior during inference.
A.3 Low-Level Policy Details
The subpolicy network Tá learns the distribution π (acasss ke, 2x).. However, find that empirical performance improves by removing he as a direct input to the subpolicy network. Including k, can lead to optimization pitfalls such collapse
single discrete skill or training instability.
We note that explicitly removing k from the set of conditioning variables does not reduce the information available to TH. as the continuous parameters corresponding to different discrete skills occupy disjoint subspaces in practice. Under this formulation, the discrete skill acts as a "selector" that indexes into one of | subspaces of continuous parameters, where each subspace represents
While this restriction of one continuous parameter value per discrete skill per trajectory works in experiments, it is possible that longer-horizon trajectories will require the same discrete skill to be used with multiple continuous parameterizations. However, the same principle easily extends to this setting, as one can predict n sets of continuous parameters per discrete skill per trajectory, where is chosen hyperparameter. Another key difference between our architecture and that used by Shankar and Gupta (25] is that we not predict binary variables representing skill termination at each timestep. We find that including this variable makes it significantly harder to get training to stabilize by adding dependencies between different time steps. which also slows the implementation down.
13

---

a fundamentally different skill type. Conditioning on the continuous parameter alone can thus still expose information about the disjoint space it inhabits and therefore the associated discrete skill being utilized.
Hence, F takes the 1D compressed state s/ and the continuous parameter 2t- This combined vector is processed by a feedforward neural network (the "Markov policy network" head) that outputs the parameters of the action distribution (e.g. mean and variance for a Gaussian action space, or mixture parameters for a Gaussian Mixture Model).
A.4 Pretraining (Skill Discovery)
Pretraining involves optimizing the objective in Equation 4 on the provided expert demonstration dataset D. During this phase, the sequences of discrete skills and continuous parameters are sampled from the variational network q(k, CIT, I). The bidirectional nature of q, which considers the entire trajectory (both past and future), is crucial for discovering meaningful skill abstractions. The first term in Equation 4 (the reconstruction term for actions) encourages the subpolicy ' A and the variational posteriors q to jointly explain the actions in the demonstrations. The KL-divergence terms ensure that the autoregressive policies (T 2²) learn to predict skill sequences similar to those inferred by y. while also regularizing g towards distributions that can be effectively modeled by these autoregressive policies at inference time.
A.5 Finetuning (Task Generalization)
For generalizing to new, potentially unseen tasks, we finetune the learned policies (T x², x AA) For task id I, in MetaWorld we use one-hot vector for all the 50 tasks, while in LIBERO we use the provided latent language embeddings as done previous work. The finetuning process also optimizes the objective in Equation However, key difference from pretraining is that the discrete skills and continuous parameters Ç are now sampled from the autoregressive policies T (Kels a1 I) and n (2g) 1 01:t-1: 1, ke, I), respectively, instead of from the variational network q. This aligns the training conditions during finetuning more closely with the actual inference procedure, where q is not used. While sampling from x" and N for training the subpolicy can be biased, empirically find that this strategy improves performance on unseen tasks at inference time.
B Task Prediction from Observations
To motivate the need to provide a compressed version of the observation space to subpolicy networks, we study the overlap in the observation space across different tasks in LIBERO-90. Specifically, for each task in LIBERO-90, we split the 50 provided demonstration trajectories into 45 training trajectories and test trajectories. Using the 45 90 4050 trajectories in the training set, we train simple classifier to predict which of the 90 tasks a single observation belongs to. Our model consists of the same Resnet-18 image encoder used in our experiments, with two-layer MLP on top (with hidden size of 1024). After two epochs through the training data, our model achieves an accuracy of 84% % on the held-out test data.
Note that our model has access to a single observation as input, and therefore can not analyze temporal information about the demonstration trajectory. Furthermore, the training and test datasets come from separate demonstration trajectories, so it is also not possible for the model to reason based on trajectory-spechi attributes. Hence, for the model to correctly map observations to input, it must find task-specifi attributes that generalize to unseen trajectories in the test set. For this to be possible, trajectories from the same task must occupy a common subspace. We conclude that there must be little overlap in the observation space covered by demonstration trajectories corresponding to different tasks.
If one implements a subpolicy network that has access to previous observations in addition to the current observation, this would possibly increase the disjointedness between observation encodings across different tasks. This explains why we observe no significant difference in the cloning loss achieved by single model mapping observation action compared to a naive implementation of
4 While the observations used in LIBERO [16] include an embedding of the current task, we remove this embedding for the purposes of this experiment
14

---

temporal variational inference. As explained in Section we mitigate this issue by passing compressed version of the observation space to the subpolicy network. forcing meaningful discrete skills and continuous parameters to be learned in order to minimize behavior cloning loss.
C Model Architectures and Hyper-parameters
For our experiments using LIBERO, we use the Resnet-18 image encoder provided in LIBERO [6] for each of the baselines. For Meta MeraWorld, we use the same CNN image encoder used by PRISE. For a fair comparison across methods, we ensure that each algorithm uses the same image encoders, sees the same amount of training data per batch, and is trained/finnttneed for the same number of gradient steps. We highlight the architectural details of each baseline below:
C.1 DEPS
The variational network contains a two-layer bidirectional GRU with a hidden size of 1024. The individual heads for the discrete and continuous parameters both have two layers each, with hidden size of 1024. For every timestep, the output head for continuous parameters returns an array of dimensions (IK), d), where K is the number of discrete skills (set 10) and d is the dimensionality of the continuous parameters (set to 4). We then average the outputs of the continuous parameters output heads to get one set of continuous parameters per discrete skill, as highlighted Section 4.3
The discrete and continuous networks both consist of one-directional GRU with a hidden size of 1024, followed by two layers of hidden size 1024.
To calculate the one-dimensional compressed state s (see Section 4.2 for notation), we use the provided robot proprioceptive states (8 (8-dimensional in MetaWorld, 9-dimensional in LIBERO) as the feature vector sprojj The compression MLP, fcompress (kr, 2,), consists of two layers with hidden size of 128. The compressed state is then concatenated with the discrete skill and continuous parameter before being passed through 2-layer MLP with hidden size 1024 and Gaussian mixture model head with two intermediate layers and hidden size of 1024. We use the same Gaussian mixture model implementation LIBERO [16]. For MetaWorld we replace the GMM policy head with the deterministic MLP, as has been done by previous work [33].
To calculate the loss during pretraining, in LIBERO we weigh the KL divergence terms corresponding to the discrete skills and continuous variables by 0.5 and 0.01, respectively. The skill parameter norm penalty is weighted by 0.1. For MetaWorld, we scale these hyperparameter down to 0.1, 0.03, and 0.03 respectively, to account for the lower loss magnitudes (due to using deterministic policy head instead of GMM).
Batch sizes are chosen to fit a single GPU during pretraining and finetuning. This results in batch size of 3 during pretraining and a batch size of 2 during finetuning for LIBERO, and batch sizes of 8 during pretraining and during finetuning (i.e. all of the data as do 3-shot finetuning) for Meta World.
We learning rate of 3e - 4 and the random seeds 95, 96, 97, 98, and 99.
C.2 Behavior Cloning
We use the same general setup as described for DEPS. The main changes are that the variational, discrete, and continuous networks are no longer used, and the subpolicy network has access to the complete observation, including images and proprioceptive state. The observation is passed through a I-layer LSTM with hidden size 1024 before being passed to the same GMM head used above.
C.3 PRISE
We use the original implementation and hyperparameters provided in Zheng et al. [33, which comes with implementations tuned for LIBERO and MetaWorld. For fair comparison to other algorithms, we make two changes to the provided implementation: (i) we reduce the batch size to fit our computational resources, using batch sizes to have the same amount of data per batch as the other algorithms (ii) reduce the total number of gradient steps taken to match the other algorithms.
15

---

D Descriptions of Tasks in the Test Set
We perform pretraining on the first 80 tasks in LIBERO-90 [6] The LIBERO-OOD dataset, which tests generalization to unseen tasks consisting of novel objects and environments, corresponds to the last 10 tasks in LIBERO-90. Task IDs and descriptions for this dataset are provided We also test performance on the LIBERO-10, 10, details for which can be found in LIBERO.
LIBERO
in Table 3
Meta World-v2 We perform pretraining on the first tasks in MetaWorld. Finetuning then performed on the MW- Vanilla and MW-PRISE datasets, explained in Section 5.2 Information on the specific tasks used in each dataset provided in Table
Table 3: Tasks in LIBERO-OOD
Task Index
Task Description
80
STUDY SCENE2 pick up the book and place it in the right compartment of the caddy
81
STUDY SCENE3 pick up the book and place it in the front compartment of the caddy
82
STUDY SCENE3 pick up the book and place it in the left compartment of the caddy
83
STUDY SCENE3 pick up the book and place it in the right compartment of the caddy
84
STUDY SCENE3 pick up the red mug and place it to the right of the caddy
85
STUDY SCENE3 pick up the white mug and place it to the right of the caddy
86
STUDY SCENE4 pick up the book in the middle and place it on the cabinet shelf
87
STUDY SCENE4 pick up the book on the left and place on top of the shelf
88
STUDY SCENE4 pick up the book on the right and place it on the cabinet shelf
89
STUDY SCENE4 pick up the book on the right and place under the cabinet shelf
Table 4: Pretraining (left) and finetuning (right) task splits for Meta World
Task Index
Pretraining Task Description
Finetune Dataset
Task Index
Finetune Task Deseription
assembly-v2
bin-picking-v2
baskethall x2
46
box-close-12
MW Vanilla
47
hand insert y2
button lopdown wall-v2
donr-lock-v2
button-press-v2
49
door-unlock-v2
button-press wall-v2
disassemble-v2
coffee-button-22
pick placc--alll--eeeee wall-v2
coflee pull-vé v22
MW-PRISF
stick-pull-v2
8
coffee-pushh push-v2
46
box-close-v2
dial-rum--2
hand-insert-ve
E Performance on Meta M with Varying Pretraining Amounts
In Section 5 we perform 20 epochs of pretraining on LIBERO and 40 epochs of pretraining on MetaWorld, showing that DEPS outperforms existing baselines its rapid generalization to diverse datasets of unseen tasks. We also show that DEPS retains its performance advantage on reducing the number of pretraining epochs (using LIBERO-OOD) and, in the case of low pretraining amounts, the performance gap between DEPS and other baselines increases. In Table [ we present results on MetaWorld analogous to those presented Table but with half as much pretraining (i.e. 20 epochs), As can be seen DEPS retains in its performance advantage, suggesting that it is robust to different pretraining compute budgets.
F
DEPS with Different Amounts of State Compression
One may ask whether compressing the observation to ID is overly aggressive.
As outlined Section 4.2 we believe this not the case, as parameterized skill can be viewed as a family of related parameterized trajectories. Each trajectory is defined by choice of continuous parameterization, and given a fixed trajectory, a ID state is sufficient as one only needs an "index" into the trajectory (Figure 3h. As the dimensionality of the state embedding increases, we expect
16

---

Table Evaluation results on MetaWorld with half much pretraining (20 epochs). All results are
averaged across 5 seeds
Evaluation Set
Algorithm
Mean Success
Mean Highest Success
DEPS
0,40 + 0.02
0.58 = 0.04
MW-Vanilla
PRISE
0.19 0.06
0.31 = 0.08
BC
0.35 + 0.02
0.50 = 0.03
DEPS
0.30 0.04
0.51 = 0.05
MW-PRISE
PRISE
0.07 - 0.03
0.18 + 0.06
BC 0.24 +
0.42 + 0.03
more information to be encoded in the state embedding, which implies (i) lower overlap in the embedded state-space across tasks and (ii) less reliance on the skill parameters encode meaningful information, resulting in poorer performance in unseen tasks/environments.
To validate this intuition, ablate the performance of DEPS against versions of DEPS using 2D and 3D state compression instead. Additionally, we compare against DEPS (Uncompressed), which uses a subpolicy implementation that takes the history of robot proprioceptive states, which then passed through an LSTM (without any compression) before being concatenated with the current discrete skill and continuous parameter and passed through an output head.
As can seen Table DEPS significantly outperforms each of the baselines.
Table 6: Performance of parameterized skills with different compression dimensions. Results are on LIBERO-OOD. ID state compression results taken from Table T. Results for 2D and 3D compression and DEPS (Uncompressed) are averaged over seeds.
Compression Dimension
Mean Success Rate
Mean Highest Success Rate
DRPS
0.34 + 0.08
0.66 + 0.12
DEPS (2D state)
0.13 0.12
0.30 0.18
DEPS (3D state)
0.05 + 0.04
0.17 + 0.11
DEPS (Uncompressed)
0.19 + 0.05
0.48 + 0.03
G DEPS with Different Limits on the Maximum Number of Discrete Skills, K
In all our experiments in Section we set the maximum number of discrete skills K to 10. This is simply meant to be an upper bound on the number of discrete skills learnable by DEPS. To demonstrate the robustness of DEPS to different values of this hyperparameter, we ablate its performance with a different value of this same hyperparameter (K 20) in Table 7 As can be seen, setting K 20 in fact significantly improves the performance of DEPS. It is possible that the performance of DEPS might increase even more significantly with thorough hyperparameter sweep.
Table 7: Performance of DEPS with different values of K. Results for K = 10 taken from Table 1 Results for K 20 averaged across 3 seeds.
K
Mean Success Rate
Mean Highest Success Rate
10 (from paper)
0.34 + 0.08
0.66 + 0.12
20
0.47 + 0.01
0.74 + 0.05
H On the Importance of Parameterized Skills
As discussed in Section I we believe that learning purely discrete continuous skills possesses inherent limitations. DEPS' two tiered design combines the benefits of these two approaches, providing the representation power of continuous parameters with the structured and interpretable
17

---

nature of discrete skills. In Table 8 we we compare the empirical performance of DEPS against approaches using only discrete or continuous parameters, showing that DEPS outperforms both settings.
Table 8: Performance comparison between DEPS and ablated versions with only discrete skills or only continuous parameters. Results are on LIBERO-OOD, Results for DEPS are taken from Table Results for discrete-only and continuous-only are averaged across 3 seeds.
Method
Mean Success
Mean Max Success
DEPS (results from paper)
0.34 + 0.08
0.66 + 0.12
Discrete-only
0.06 + 0.02
0.16 + 0.01
Continuous-only
0.15 = 0.16
0.32 + 0.30
I On the Impact of Evaluation Horizon
The choice to evaluate DEPS after a limited number of finetuning gradient steps (500) is intentional and reflects the core motivation of our work: to learn transferable representations that enable rapid adaptation to new tasks. We believe this evaluation protocol serves stronger test of generalization ability as it demonstrates that DEPS learns genuinely transferable skills rather than simply learning the downstream tasks from scratch.
For completeness, we provide additional results comparing the performance of DEPS and the BC baselines after 5x as much finetuning (i.e. 2500 gradient steps). We perform rollouts after every 250 gradient steps of finetuning and report the Mean Success Rate and the Mean Highest Success Rates. As can be seen. DEPS maintains its performance advantage, with the gap between DEPS and BC actually increasing
Table 9: Performance comparison between DEPS and BC on LIBERO-OOD with 5x many finetuning gradient steps. Results are averaged across 3 seeds.
Method
Mean Success
Mean Max Success
DEPS
0.49 + 0.05
0.75 0.02
BC
0.20 + 0.04
0.35 + 0.04
J Limitations
DEPS shows significant performance improvements in rapid generalization abilities in diverse and challenging environments. However, there are some important limitations to consider:
Generalization and accuracy tradeoffs As can be seen in Table I the performance of DEPS is not as strong on LIBERO-10, 10, which consists largely of in-distribution but longer-horizon tasks, compared to other held-out task sets. Given the in-distribution nature of LIBERO-10, it makes sense that parameterized skills will offer less of a performance improvement as there is need to generalize to new tasks or environments. However, compress observations to one dimension before passing them to the subpolicy network, which enables generalization, potentially comes at the cost of perfect reconstruction of in-distribution long- long-horizon demonstration trajectories. The observed tradeoff may be acceptable in many practical robotics scenarios, where learned skills from demonstrations typically serve as a starting point for further refinement using RL. In such cases, the initial generalization capability is more valuable than perfect execution on specific, lengthy sequences.
also possible that our choice to aggressively
We envision several promising approaches to address this limitation while preserving the general- ization benefits of our method. For instance, one may train a residual policy that has access to the full-observation space in addition to the current discrete skill and continuous parameter, correcting the output of the original subpolicy network. We leave more rrigorous evaluation of this potential shortcoming and its remedies to future work.
18

---

Reliance on proprioceptive robot states To implement the compression strategy outlined in Sec- tion 4.2 we the robot's proprioceptive state, which is provided both LIBERO and MetaWorld. as the feature vector st srroo This provides a compact, low-dimensional representation state, while excluding information about the environment and goal, which must then be encoded in the discrete skills and continuous parameters. While we expect our method to generalize to environments where proprioceptive state is not provided, we leave evaluation of this to future work.
the robot's
Method complexity Compared to naive behavioral cloning, DEPS is more complex to implement and has more hyperparameters, resulting in more hyperparameter tuning. While we show that DEPS has clear performance advantages, may not be appropriate for simpler environments or narrow task distributions requiring less generalization.
K
Societal Impacts and Potential for Misuse
We do not perform studies involving human subjects, and properly attribute datasets and software used their authors. The research presented, which focused enabling robots learn parame- terized skills from demonstrations, has the potential for significant positive societal impact. These include enhanced automation industrial and domestic settings, improved assistive technologies for individuals needing support, and safer execution of tasks in hazardous environments. On the other hand, our method inherits the common potential for misuse of robotics research, such as autonomous weapons, labor displacement, and economic disruption, but these potentials are not unique to this paper.
Further Visualization of Learned Skills
We provide visualizations of the discovered parameterized skills bico.ooogg..oom/e/w//aaammmeeecs
website:
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Every claim made in the abstract clearly corresponds to the content of a section in the paper. For example, the abstract claims improved generalization unseen tasks in LIBERO and MetaWorld, which is clearly conveyed in our Experiments section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide limitations section in the appendix, and clearly reference it in main body.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification: The main theoretical result in our paper is the derivation of an evidence-based lower bound learn parameterized skills. We provide a derivation for this objective and clearly state the assumptions made for computational tractability.
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide our code in the supplemental meterial as well as information on the model sizes and hyperparameters for each baseline and information about the spe- cific datasets used for evaluation. The provided information is sufficiently granular for reproduction of our results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
21

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide our implementation along with instructions to reproduce our results with our supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/guidess httpai///uiids//ooeuuuibiissssss onloiicyo)yy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. //nips.c/pput httpl//ic/guidee//oeeeeeeeeeeeee
sssiiissiiniii ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Enough detail is provided in our main body for a reader to faithfully understand our experimental settings and results. We have comprehensive appendix sections on train/test splits and hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report error bars clearly next to every reported result.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information on the computational resources used in our appendix, including number/type GPUs and the time taken.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ftttp:://neurips http::/:///ooupp...c/cb/ttt////
Answer: [Yes]
Justification: We do not perform studies involving human subjects, and properly attribute all datasets and software used to their authors. We also include section our appendix to discuss societal impact and potential harmful consequences.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
23

---

Justification:
We discuss potential positive and negative impacts of out work in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymennt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not release any data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite all owners of code repositories used in the text of our paper as well as in our code files.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
24

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palasets.papersstt paperssi thcode. com con/dataset package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not perform any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not perform any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We do not use LLMMs in any way that is important or non-standard.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
26


Reviews:
[{"review_id": "0Ao9UB4oDt", "reviewer": "Reviewer_7pDZ", "summary": "The authors introduce an approach called deps which learns a skill parameterized policy from demonstrations.  The approach learns a discrete latent that encodes a set of skills and a continuous latent that modulates the skill. A key insight is compressing the state into a 1d vector to select along the manifold of skills. This forces the skill latents to encode useful information. The authors compare their approach to prior work and show that it outperforms these baselines in terms of task success rate.", "questions": "Please include an ablation analysis showing the benefit of compression and tradeoffs with different compression sizes for st. I will increase my score if this is provided.\n\nPlease explain if and why zt changes at each timestep", "limitations": "None", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths - the is paper presents a sound and interesting approach that enables semantically skill learning. The authors provide convincing results that are both qualitative and quantitative. Their idea to compress observation embedding into a 1d vector to ensure that the skill space is forced to capture useful skill-related information. The separation of skill parameterization into a discrete and continuous latent make the learned policy more interpretable and successful in terms of generalization. \n\nFig 2. is confusing. I am not sure what the gray areas represent and the components could be explained better. \n\nThe paper could benefit from an ablation analysis showing that a 1d scalar is the best choice (why not 2d)?\n\nA highly compressed st means that kt and zt are forced to encode all task-critical information - perhaps even information that isnt relevant to skill. Could the authors provide more compelling results showing that kt and zt encode only skill relevant information?\n\nIt is confusing that zt is sampled per kt but can be redefined at each timestamp. it would be helpful if the authors could further elaborate on what this means. If it can adapt at each timestamp, are there any constraints imposed on how it can adapt?", "was_revised": true, "final_justification": "I appreciate the authors response and I think this paper should be accepted at Neurips. Therefore, i will keep my score of 5", "rebuttal": "We thank you for your thorough review and positive comments. We address your concerns below.\n\n**Q1:** Fig. 2 is confusing. I am not sure what the gray areas represent, and the components could be explained better.\n\n**A1:** As denoted in Figure 2’s legend, the gray area represents the variables that are observable to each policy/encoder. For example, the variational encoder can see the whole trajectory history and future, while the low-level policy can only see the previous transitions and the skills being selected at the current time step. We will clarify this further in the revision.\n\n**Q2:** Please include an ablation analysis showing the benefit of compression and tradeoffs with different compression sizes for $s_t$.\n\n**A2:** We thank the reviewer for the helpful suggestion. We have added a set of new experiments, comparing the mean success rates and the mean highest success rates on the LIBERO-OOD test setting for DEPS with 1D state compression (results from the paper) against DEPS with 3D state compression. The results for 3D state compression are averaged across 3 seeds. \n\n| Compression Dimension | Mean Success Rate | Mean Highest Success Rate |\n|----------------------|-------------------|---------------------------|\n| 1D (from paper)      | 0.34 ± 0.08      | 0.66 ± 0.12              |\n| 3D                   | 0.05 ± 0.04      | 0.17 ± 0.11              |\n\nThe dramatic performance drop from 1D to 3D compression (0.34 → 0.05 mean success rate) suggests that aggressive compression to a single dimension is crucial for the method's effectiveness.  \n\n**Q3:** A highly compressed st means that $k_t$ and $z_t$ are forced to encode all task-critical information - perhaps even information that isn’t relevant to skill. Could the authors provide more compelling results showing that $k_t$ and $z_t$ encode only skill-relevant information?\n\n**A3:** We thank the reviewer for the insightful comment; it is indeed important to ensure that $k_t$ and $z_t$ encode skill-relevant information rather than just general task information. One of DEPS’ main goals is to ensure this precise behavior. As outlined in Appendix A1, during pretraining, the variational network predicts continuous parameters per discrete skill for each forward pass over a given trajectory (instead of predicting a new continuous parameterization at each time step, which could lead to overfitting).  Additionally, to prevent the continuous parameters from overfitting to specific tasks, we introduce the $\\mathcal{L}_{\\text{norm}}$ (Appendix A1) regularization term, which bounds the range of the learned continuous parameters. This limits the amount of information encoded in the discrete skill and continuous parameters, forcing the 1D state to encode meaningful information as an “index” into parameterized skill trajectories.  This is validated in Figure 8, where the learned latent 1D states vary monotonically within discrete skills, with discontinuities when the discrete skill switches. The strong performance of DEPS on out-of-distribution downstream tasks (Table 1, LIBERO-OOD) further supports our hypothesis that DEPS learns meaningful abstractions instead of overfitting to $k$ or $z$ values. This is particularly evident in low-data regimes (Table 1, LIBERO-3-shot) and low pretraining settings (Table 2).\n\nWe provide several lines of evidence that $k_t$ and $z_t$ do encode skill-relevant information:\n\n1. **Consistent application of discrete skills (Section 5.5 and Figure 7):** We find that DEPS discovers semantically meaningful discrete skills corresponding to primitive behaviors such as grasping objects, opening doors, etc. Importantly, we find that the discovered skills are applied *consistently* across tasks. An example can be seen in Figure 7, where the same ``grasp_object’’ skill is used to pick butter in a cabinet environment (top) and pick a mug (bottom) in a kitchen environment (bottom). This is not a cherry-picked example; we can release an annotated version of the entire LIBERO dataset across multiple tasks, showing that learned discrete skills consistently hold across distinct tasks and environments. \n\n2. **Smooth variation in policy on varying continuous parameters (Section 5.5):** We find that for a given discrete skill, slight modifications in the continuous parameter result in smooth variations in the resulting policy. For example, modifying the continuous parameters for a discrete skill used to turn a knob smoothly varies the location at which the knob turning action is executed. Visualizations of this are provided on our website (which was already included at the time of submission; the URL is in Section 5.5). \n\n3. **Overlap of skill-specific continuous parameterizations across tasks:** We have visualizations that show that for a given discrete skill, there is significant overlap in the continuous parameters chosen for different tasks, which suggests that $z$ only encodes skill-specific, and not task-specific, information. We are happy to update the project website with these visualizations for the reviewers' interest, provided the AC gives approval or clarification on the website update policy; regardless, we will include such visualizations in the final camera-ready.\n\n**Q4:** Please explain if and why $z_t$ changes at each timestep.\n\n**A4:** *During pretraining*, $z_t$ is not changed at every timestep. Instead, $z_t$ can only change when the sampled discrete skill changes. This is essential to ensure that the learned parameterized skills are temporally extended, and helps prevent the model from taking degenerate shortcuts (such as encoding the action at each timestep directly in the continuous parameter) that would prevent the model from learning meaningful abstractions. Details on this can be found in Appendix A1.\n\nOn the other hand, *during finetuning*, we allow the model to choose to change its choice of continuous parameter at each timestep if necessary. Our goal during the finetuning phase is to tune the abstractions learned during pretraining to maximize performance on downstream tasks. By modifying its continuous parameter at each timestep, the robot can “close the planning loop” and therefore more flexibly react to potential stochasticity (or unexpected outcomes) at inference time. Details on this can be found in Appendix A2.\n\nIn summary, during pretraining, our goal is to learn reusable abstractions/compressions, and therefore, we do not change $z_t$ at each timestep to encourage compression. On the other hand, during inference, we allow $z_t$ to change at each timestep as our goal is to use learned specialized abstractions to maximize performance on a task.\n\nWe appreciate the reviewer’s thoughtful comments and suggestions on strengthening our work, and welcome further opportunities to clarify any additional questions that may remain."}, {"review_id": "XALNdKTIWU", "reviewer": "Reviewer_pzRg", "summary": "This paper presents an algorithm for the Discovery of GEneralizable Parameterized Skills (Deps). The proposed method learns skills from demonstrations in an end-to-end way, by training a 3-level architecture with a discrete skill selector, a continuous parameter selector conditioned on the discrete skill, and a subpolicy conditioned on both. The proposed method is evaluated on two multitask environments: LIBERO and MetaWorld, and compared against BC untrained and pretrained, and Prise.", "questions": "As mentioned, it seems that in some cases the additional complexity of the proposed method doesn't pay off in terms of improved performance, especially in longer horizon tasks. Can you point out which are the case where instead there is a clear benefit and the improved performance clearly justify the additional complexity?\n\nAre discrete skills transferable from tasks to tasks? \n\nAre discrete skills interpretable (eg grasp, lift, ...)?\n\nWhat are low-level actions a_t (joint positions, velocities, cartesian, ...)? Is one choice of the other preferable in your formulation?\n\nHow do the representation of skills learned by your method compare to those from Prise's action abstractions?\n\n[Figure 7] \"We find that general skills corresponding to grasping, moving, and releasing objects are learned and consistently applied across visually diverse tasks.\" -- are these skills learned every time or are they \"transferred\" through the learned intermediate level of the architecture?", "limitations": "Limitations are only discussed in the appendix. The limitations of performance on long horizon tasks is not actually addressed, but authors seem to hint at the fact that adopting a higher dimensional state for compression might help this issue. Are there any additional experiments pointing towards this intuition?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "The paper is presented in a clear and well structured way. It nicely motivates the problem at hand and scopes it within relevant literature. The problem tackled is important and can have impact across various areas where decision making is used.\nThe method and it's mathematical formulation are presented clearly and looks correct to the best of my understanding.\nThe proposed algorithm is also evaluated and analyzed though several experiments, and compared to other methods. This helps understanding the method strengths in comparison to other approaches.\nHowever, in my opinion, the originality of the method introduced is marginal, and the choice of a one-dimensional state compression is arguable: while the paper initially mentions manifolds as a geometry to describe this step, a one-dimensional state seems very limitative. And as authors mention in the limitation section in appendix, this in fact hinders performance in longer horizon tasks. This make me doubt the actual value of adopting the additional complexity of the method if gains are only observed in relatively short tasks.", "was_revised": true, "final_justification": "I believe my questions have been addressed, I confirm my score, and I have completed the mandatory acknowledgement.", "rebuttal": "Thank you for the positive evaluation and thorough comments. We seek to address your questions below.\n\n**Q1: [paraphrased]**  The choice of a one-dimensional state compression is arguable\n\n**A1:** This is a natural question. To answer this question, we ran additional experiments to compare the mean success rates and the mean highest success rates on the LIBERO-OOD test setting for DEPS with 1D state compression (results from the paper) against DEPS with 3D state compression. The results for 3D state compression are averaged across 3 seeds. \n\n| Compression Dimension | Mean Success Rate | Mean Highest Success Rate |\n|----------------------|-------------------|---------------------------|\n| 1D (from paper)      | 0.34 ± 0.08      | 0.66 ± 0.12              |\n| 3D                   | 0.05 ± 0.04      | 0.17 ± 0.11              |\n\nThe dramatic performance drop from 1D to 3D compression (0.34 → 0.05 mean success rate) suggests that aggressive compression to a single dimension is indeed crucial for DEPS effectiveness, likely due to reduced overfitting to training environments. \n\nIn response to the reviewer’s comment that “authors seem to hint at the fact that adopting a higher dimensional state for compression might help…”, we would like to clarify this potential misunderstanding.  As can be seen in the empirical result above, in the case of LIBERO-OOD, adopting a higher dimension for state compression does not increase performance. In the Limitations Section, we suggest that training a residual policy [1] which has access to the full observation space and augments the output of the original subpolicy (which will still utilise 1D state compression), may be a promising approach - we leave an evaluation of this to future work.\n\n[1] Silver et. al, Residual Policy Learning. 2018\n\n**Q2:** As mentioned, it seems that in some cases, the additional complexity of the proposed method doesn't pay off in terms of improved performance, especially in longer horizon tasks. Can you point out which are the cases where, instead, there is a clear benefit and the improved performance clearly justifies the additional complexity?\n\n**A2:** You raise an important point about the relatively weaker performance of DEPS on  LIBERO-10. We would like to stress that the tasks in LIBERO-10 mostly consist of concatenations of tasks in the pretraining dataset, and therefore, there is a *minimal distribution shift between the pretraining and finetuning tasks in this setting.*  This is precisely the scenario where we'd expect less benefit from parameterized skills, as there is no need to generalize to new tasks or environments. We would like to note that despite the in-distribution nature of LIBERO-10, DEPS still performs better than PRISE and BC.\n\nWe expect DEPS to perform strongly on long-horizon and *out-of-distribution* tasks, and leave an evaluation of this to future work. \n\nWe believe that there are several scenarios where there is a clear benefit to using DEPS:\n\n1. **Out-of-Distribution Generalization (LIBERO-OOD, page 7 Table 1):** DEPS shows substantial improvements when generalizing to entirely unseen environments and objects:\n\n - Mean success: 0.34 ± 0.08 vs BC: 0.15 ± 0.04 (127% improvement)\n - Mean highest success: 0.66 ± 0.12 vs BC: 0.36 ± 0.08 (83% improvement)\n\n   This represents the core scenario DEPS is designed for - adapting learned behavioral primitives to novel contexts. We believe this scenario is highly relevant for robotics applications, where robots have to deal with limited training datasets and extremely diverse tasks and environments..\n\n2. **Low-Data Regimes (LIBERO-3-shot, Table 1):** For robotics applications, there often is very little demonstration data available for downstream tasks. When reducing the amount of training data available for downstream tasks, the gap between DEPS and other baselines increases. \n\n - Mean success: 0.26 ± 0.03 vs BC: 0.11 ± 0.05 (136% improvement)\n - Mean highest success: 0.49 ± 0.03 vs BC: 0.22 ± 0.08 (123% improvement)\n\n3. **Limited Pretraining Compute (Table 2):** When reducing the amount of pretraining performed, DEPS significantly outperforms other baselines \n\n - At 5 epochs: DEPS 0.24 ± 0.08 vs BC 0.07 ± 0.03 (243% improvement)\n - At 15 epochs: DEPS 0.39 ± 0.05 vs BC 0.12 ± 0.06 (225% improvement)\n\n   This suggests that, in addition to improving downstream generalization to out-of-distribution tasks, DEPS’ learned abstractions improve the efficiency of pretraining (page 8, section 5.4).\n\n We will add a discussion on the specific regimes where DEPS is useful in the revised version of the paper.\n\n**Q3:** Are discrete skills transferable from tasks to tasks?\n\n**A3:** Yes! As shown in Figure 7 (page 9), DEPS discovers semantically meaningful skills corresponding to action primitives such as grasping objects, closing doors etc. Importantly, we find that the discovered skills are applied *consistently* across tasks. An example can be seen in Figure 7 where the same grasp_object skill is used to pick butter in a cabinet environment  (top) and pick a mug (bottom) in a kitchen environment (bottom). This is not a cherry-picked example; we can release an annotated version of the entire LIBERO dataset across multiple tasks, showing that learned discrete skills consistently hold across distinct tasks and environments. The transferability of the learned discrete skills is supported by the observation that there is significant overlap in the distribution of continuous parameters used for a given discrete skill across different tasks. It is precisely this non-overfitted continuous parameterization that enables the discrete skills to easily generalize and transfer to unseen tasks. \n\n**Q4:** Are discrete skills interpretable (eg grasp, lift, ...)?\n\n**A4:** The learned skills are interpretable and correspond to action primitives such as object grasping/releasing, door closing, knob turning, etc. An example of this is provided in Figure 7. Furthermore, the learned continuous parameters represent smooth modulations in the behaviour of the discrete skill (visualizations are on our website, which was already included at the time of submission; the URL is in section 5.5).\n\n**Q5:** What are low-level actions $a_t$ (joint positions, velocities, cartesian, ...)? Is one choice of the other preferable in your formulation?\n\n**A5:** The action space in LIBERO consists of a 6-dimensional continuous-values vector representing OSC_POSE, which consists of the desired position and orientation of the end effector. In addition, there is a binary value representing the gripper state (opened/closed). In line with previous work, we train our policies to output a Mixture of Gaussians representing a desired distribution over actions. \n\nIn Metaworld, the action space consists of a 3D vector representing the desired *change* in the end effector position as well as a binary variable representing the gripper (opened/closed). In line with previous work, we use a deterministic policy that outputs the desired action.\n\nWe believe that DEPS is compatible with diverse low-level action types. Our empirical results on LIBERO and Metaworld use their standard respective environment action spaces, as also used in previous works, with minimal differences in the hyperparameters used.\n\n**Q6:** How do the representation of skills learned by your method compare to those from Prise's action abstractions?\n\n**A6:** Unfortunately, to the best of our knowledge, PRISE does not provide visualizations of its learned action abstractions. \n\nPRISE learns discrete action abstractions, each of which represents a fixed sequence of “action tokens”. We believe the parameterized skills learned by DEPS, which can be varied smoothly by changing the continuous parameter and do not have a fixed length (unlike in PRISE), present a more flexible and interpretable representation. \n\n**Q7:** [Figure 7] \"We find that general skills corresponding to grasping, moving, and releasing objects are learned and consistently applied across visually diverse tasks.\" -- are these skills learned every time or are they \"transferred\" through the learned intermediate level of the architecture?\n\n**A7:** The learned skills are transferred across the intermediate level of the architecture. This is supported by several aspects of our experiments, including:\n\n1. The same discrete skill is consistently applied in all relevant contexts. For example, discrete skill 9 (“grasp_object”) (as shown in Figure 7) is applied every time an object needs to be grasped, irrespective of the task.\n\n2. Table 2 (page 8) shows that the gap between DEPS and other baselines increases when the number of pretraining epochs is decreased. This supports the argument that the learned skills are being used across tasks, therefore increasing the efficiency of pretraining through cross-task generalization.\n\nThank you again for your constructive feedback, which has helped improve the clarity of several key aspects of our paper. Please let us know if you have any additional questions."}, {"review_id": "7zypfCMu8Z", "reviewer": "Reviewer_9spN", "summary": "This paper introduces **DEPS**, an end-to-end framework that jointly discovers discrete skills and continuous parameters from demonstrations, trained via temporal variational inference augmented with information-bottleneck tricks. Empirically, DEPS outperforms BC and PRISE on LIBERO and MetaWorld benchmarks under few-shot fine-tuning.", "questions": "1. **How is *K* chosen?** Please describe any heuristics, cross-validation or sensitivity analysis; current text implies it is fixed but gives no guidance.\n2. **Single-latent ablation** – add experiments with (i) only discrete skills and (ii) only continuous parameters to prove the necessity of the two-tier latent design.\n3. **End-to-end baselines** – include a diffusion-policy or transformer-BC baseline that maximises πᵃ directly without latent regularisation to test whether πᵃ+{πᴷ,πᶻ} is truly superior.\n4. **Risk of latent shortcuts** – with 1-D state input the sub-policy may over-fit to *k* or *z* values rather than learning robust motor primitives. Can you visualise latent distributions to rule this out or try less aggressive compression?\n5. **Evaluation horizon** – report performance after full convergence on the new task. Including both “rapid-adapt” and “fully-finetuned” results would clarify absolute capability and asymptotic ceiling.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n* **Principled objective** – derives a tractable ELBO with exact KL terms for both categorical and Gaussian latents.\n* **Degeneracy mitigation** – the 1-D state compression and latent-conditioned sub-policy convincingly force reliance on the latents, and ablations show sizable gains.\n* **Interpretability** – visual segmentations reveal intuitive “grasp / move / release / close” skills reused across tasks; continuous parameters smoothly modulate grasp poses.\n\n**Weaknesses**\n\n* **Hyper-parameter sensitivity** – the number of discrete skills *K* is fixed a-priori; no heuristic or validation study is given.\n* **Two-level latent justification weak** – authors assert both *k* and *z* are needed but provide no ablation vs. single-latent variants.\n* **Baselines limited** – modern end-to-end policies (e.g., Diffusion Policy) are absent; PRISE is the only hierarchical competitor.\n* **Evaluation setup peculiar** – success is measured after ≤ 500 gradient steps instead of on a fully converged policy, making absolute numbers hard to interpret.\n* **Compression extremity** – projecting proprioception to a single scalar risks under-representation; possible shortcut utilisation by latents is not checked.", "was_revised": true, "final_justification": "The paper's experimental setting is uncommon and does not include more mainstream experimental settings, and lacks some more common baseline comparisons.", "rebuttal": "Thank you for your thoughtful and thorough review. We address your questions below.\n\n**Q1: How is K chosen?** Please describe any heuristics, cross-validation, or sensitivity analysis; the current text implies it is fixed but gives no guidance.\n\n**A1:** You raise an important question about the hyperparameter sensitivity of DEPS. In our experiments, we set the maximum number of skills (K) to the same value (10) across all datasets and environments. The value of K is meant to simply be a large number that sets an upper bound on the number of discrete skills that can be learned by DEPS. This is a common technique used in previous skill learning papers (e.g. [1] and [2]). In practice, we find that the number of discrete skills learned by DEPS varies across training datasets, but is below K on average. Because of this, we believe DEPS is robust to higher values of this hyperparameter, and therefore did not prioritize sweeping different values of K.\n\nTo validate the robustness of DEPS to different values of K, we present some new results of DEPS on LIBERO-OOD setting K to be twice as large (20). Results for K=20 are averaged across 3 seeds. \n\n| K | Mean Success Rate | Mean Highest Success Rate |\n|----------------------|-------------------|---------------------------|\n| 10 (from paper)      | 0.34 ± 0.08      | 0.66 ± 0.12              |\n| 20                   | 0.47 ± 0.01      | 0.74 ± 0.05              |\n\nAs can be seen, setting K=20 in fact improves the performance of DEPS. In general, we find that DEPS is highly resilient to different hyperparameter settings. Due to limited compute availability, we did not do much hyperparameter tuning on DEPS, and mostly used the same hyperparameter settings across datasets/environments. \n\n[1] Learning Options via Compression. Jiang et. al. NeurIPS 2022.\n[2] Language Guided Skill Learning with Temporal Variational Inference. Fu et al. ICML 2024\n\n**Q2: Single-latent ablation** – add experiments with (i) only discrete skills and (ii) only continuous parameters to prove the necessity of the two-tier latent design.\n\n**A2:** Thank you for the suggestion. We have added the new experiments, and the following table compares the performance on LIBERO-OOD between DEPS (results from the paper), (ii) only discrete skills (iii) only continuous parameters. The results for (ii) and (iii) are averaged across 3 seeds\n\n| Method | Mean Success | Mean Max Success |\n|--------|--------------|------------------|\n| DEPS (results from paper) | 0.34 ± 0.08 | 0.66 ± 0.12 |\n| Discrete only | 0.06 ± 0.02 | 0.16 ± 0.01 |\n| Continuous only | 0.15 ± 0.16 | 0.32 ± 0.30 |\n\nDEPS outperforms both settings. We find these results to be consistent with our presented motivation for a two-tiered design in Section 1. Learning only discrete skills can prevent generalization, while learning purely continuous parameters can lead to less structured and interpretable representations. On the other hand, DEPS learns semantically interpretable discrete skills whose policies vary smoothly with the continuous parameter value. Visualizations can be found at our website (which was already included at the time of submission; the URL is in section 5.5).\n\n**Q3: End-to-end baselines** – include a diffusion-policy or transformer-BC baseline that maximises πᵃ directly without latent regularisation to test whether πᵃ+{πᴷ,πᶻ} is truly superior.\n\n**A3:** We would like to clarify that the BC baseline included in all our experiments (e.g., in Tables 1 and 2)  is end-to-end. To ensure a fair comparison between DEPS and the BC baseline, we use the same network architecture (LSTM) for both. Therefore, we don't compare diffusion-policy or transformer-BC as the difference in the fundamental network structure makes direct comparison difficult. We focus on DEPS as a general *framework* for learning parameterized skills from demonstrations; it can be easily extended to diffusion/transformer based policies, and we leave such architecture sweeps as future work.\n\n**Q4: Risk of latent shortcuts** – with 1-D state input the sub-policy may over-fit to $k$ or $z$ values rather than learning robust motor primitives. Can you visualise latent distributions to rule this out or try less aggressive compression?\n\n**A4:** **(On latent shortcuts)** You raise an important point about the potential to overfit to $k$ or $z$ values. One of DEPS’ main goals is to prevent this from occurring. As outlined in Appendix A1, during pretraining, the variational network predicts a fixed continuous parameter per discrete skill for each forward pass over a given trajectory (instead of predicting a new continuous parameterization at each time step, which could lead to overfitting). Furthermore, to prevent the continuous parameters from overfitting to specific tasks, we introduce the $\\mathcal{L}_{\\text{norm}}$ (Appendix A1) regularization term, which bounds the range of the learned continuous parameters. This limits the amount of information encoded in the discrete skill and continuous parameters, forcing the 1D state to encode meaningful information as an “index” into parameterized skill trajectories.  This is validated in Figure 8, where the learned latent 1D states vary monotonically within discrete skills, with discontinuities when the discrete skill switches. The strong performance of DEPS on out-of-distribution downstream tasks (Table 1, LIBERO-OOD) further supports our hypothesis that DEPS learns meaningful abstractions instead of overfitting to $k$ or $z$ values. This is particularly evident in low-data regimes (Table 1, LIBERO-3-shot) and low pretraining settings (Table 2).\n\n**(On visualizing latent distributions)** In Section 5.5, we provide visualizations showing that discovered discrete skills correspond to primitive behaviors such as grasping objects, opening doors, etc. Importantly, we find that the discovered skills are applied consistently across tasks. An example can be seen in Figure 7, where the same ``grasp_object’’ skill is used to pick butter in a cabinet environment  (top) and pick a mug (bottom) in a kitchen environment (bottom). This is not a cherry-picked example; we can release an annotated version of the entire LIBERO dataset across multiple tasks, showing that learned discrete skills consistently hold across distinct tasks and environments. Additionally, we have visualizations that show that for a given discrete skill, there is significant overlap in the continuous parameters chosen for different tasks, ruling out the possibility of overfitting to $z$. This further validates that DEPS learns robust motor primitives. We are happy to update the project website with such visualizations for the reviewers' interest during the discussion period, provided the AC gives approval or clarification on the website update policy; regardless, we will include such visualizations in the final camera-ready.\n\n\n**(On compression aggression)** To validate the importance of 1D state compression to DEPS’ strong performance, we compare the mean success rates and the mean highest success rates on the LIBERO-OOD test setting for DEPS with 1D state compression (results from the paper) against DEPS with 3D state compression. The results for 3D state compression are averaged across 3 seeds. \n\n\n| Compression Dimension | Mean Success Rate | Mean Highest Success Rate |\n|----------------------|-------------------|---------------------------|\n| 1D (from paper)      | 0.34 ± 0.08      | 0.66 ± 0.12              |\n| 3D                   | 0.05 ± 0.04      | 0.17 ± 0.11              |\n\nThe dramatic performance drop from 1D to 3D compression (0.34 → 0.05 mean success rate) suggests that aggressive compression to a single dimension is crucial to DEPS’ performance.\n\n\n**Q5: Evaluation horizon** – report performance after full convergence on the new task. Including both “rapid-adapt” and “fully-finetuned” results would clarify absolute capability and asymptotic ceiling.\n\n**A5:** Our choice to evaluate after limited gradient steps (500 steps) is intentional and reflects the core motivation of our work: to learn transferable representations that enable rapid adaptation to new tasks. We believe this evaluation protocol serves as a stronger test of generalization ability as it demonstrates that DEPS learns genuinely transferable skills rather than simply learning the downstream tasks from scratch. \n\nTo evaluate DEPS’ asymptotic ceiling, we’ve performed additional experiments comparing the performance of DEPS and BC on the LIBERO-OOD evaluation setting with 5x as much finetuning (i.e. 2500 gradient steps). We perform rollouts after every 250 gradient steps of finetuning and report the Mean Success Rate and the Mean Highest Success Rates. The following results are averaged across 3 seeds: \n\n| Method | Mean Success Rate | Mean Highest Success Rate |\n|--------|-------------------|---------------------------|\n| DEPS | 0.49 ± 0.05 | 0.75 ± 0.02 |\n| BC | 0.20 ± 0.04 | 0.35 ± 0.04 |\n\nWe find that our method is still better than the baseline, and the gap between DEPS and BC actually increases.\n\nWe appreciate the constructive feedback, which has helped improve the experimental results in our paper. Please let us know if you have any further questions, and we hope you will consider raising your score in light of our response and additional results."}, {"review_id": "kc3LC4hPxw", "reviewer": "Reviewer_KY1m", "summary": "The authors propose a novel skill discovery and skill-based imitation learning algorithm that learn parameterized skills from demonstrations.  The parameterized skills are a policy conditioned on both a discrete input that acts as a skill selector and a continuous input that acts as the parameter to that skill.  The discrete skill selector, continuous parameter selector, and low level policy are learned end to end, VAE-style using BC loss, with a variational distribution on the discrete and continuous skill variable given a task trajectory.  In order to handle degenerate solutions and ensure the skill variables actually encode useful information, the authors propose to compress the state input to the low-level policy into a single dimension.   They evaluate their model's ability to generalize to novel tasks with a fixed finetuning budget after being pretrained on demonstrations from the same domain in LIBERO and MetaWorld task suites and compare favorably to BC and an LLM-style skill discovery method.", "questions": "1. How is your. method finetuned?\n2. How do you ensure both $k$ and $z$ encode useful and distinct information?", "limitations": "Yes, although I would prefer if they were summarized in the main paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Novel and well-motivated method idea, particularly in using state compression in the low level policy.\n2. Strong experimental results\n3. Clear and well-written paper\n\nWeaknesses:\n1. Why are there no comparisons to other variational skill discovery methods like [1], [2]?\n2. This work is missing some ablations, especially ablating having both discrete and continuous skill latent variables instead of just one or the other.  I am curious how important it is to actually have both or could the skill policy be successful parameterized with just a continuous variable.  It would also be helpful to ablate the state compression mechanism to see how much compression is necessary to learn meaningful skills.\n\n[1] Joshua Achiam, Harrison Edwards, Dario Amodei, Pieter Abbeel. Variational Option Discovery Algorithms\n[2] Karl Pertsch, Youngwoon Lee, Joseph J. Lim. Accelerating Reinforcement Learning with Learned Skill Priors.", "was_revised": true, "final_justification": "I am recommending a score of 5.  This is a well-written paper with clear contributions and novelty.  I appreciate the authors' ablations on the state compression from the rebuttal which provided additional insight into their method.", "rebuttal": "Thank you for the positive and thoughtful comments. We address your questions below.\n\n**Q1:** Why are there no comparisons to other variational skill discovery methods like [1], [2]?\n\n**A1:** We center our comparisons against PRISE, as it is a recent and performant work; we thus utilize the same environments (LIBERO and Metaworld) for fair, standardized evaluations.  To the best of our knowledge, there are no available, performant implementations or existing evaluations for [1] and [2] that have been tuned for LIBERO and Metaworld.\n\n**Q2:** This work is missing some ablations, especially ablating having both discrete and continuous skill latent variables instead of just one or the other. I am curious how important it is to actually have both or could the skill policy be successful parameterized with just a continuous variable. It would also be helpful to ablate the state compression mechanism to see how much compression is necessary to learn meaningful skills.\n\n**A2:** Thank you for raising this important suggestion. We have run the ablations you have recommended. We present the results here:\n\n1. **Only discrete or continuous variables:** the following table compares the performance on LIBERO-OOD between DEPS (results from the paper), (ii) only discrete skills (iii) only continuous parameters. The results for (ii) and (iii) are averaged across 3 seeds\n\n   | Method | Mean Success | Mean Max Success |\n   |--------|--------------|------------------|\n   | DEPS (results from paper) | 0.34 ± 0.08 | 0.66 ± 0.12 |\n   | Discrete only | 0.06 ± 0.02 | 0.16 ± 0.01 |\n   | Continuous only | 0.15 ± 0.16 | 0.32 ± 0.30 |\n\n   DEPS outperforms both settings. We find these results to be consistent with our presented motivation for a two-tiered design in Section 1. Learning only discrete skills can prevent generalization over a smoothly varying distribution of tasks. Similarly, learning purely continuous parameters leads to less structured and interpretable representations. \n\n2. **Amount of state compression:** To validate the importance of 1D state compression to DEPS’ strong performance, we compare the mean success rates and the mean highest success rates on the LIBERO-OOD test setting for DEPS with 1D state compression (results from the paper) against DEPS with 3D state compression. The results for 3D state compression are averaged across 3 seeds. \n\n\n   | Compression Dimension | Mean Success Rate | Mean Highest Success Rate |\n   |----------------------|-------------------|---------------------------|\n   | 1D (from paper)      | 0.34 ± 0.08      | 0.66 ± 0.12              |\n   | 3D                   | 0.05 ± 0.04      | 0.17 ± 0.11              |\n\n   The dramatic performance drop from 1D to 3D compression (0.34 → 0.05 mean success rate) suggests that aggressive compression to a single dimension is crucial to DEPS’ performance.\n\n**Q3:** How is your method finetuned?\n\n**A3:** We outline the finetuning procedure in Appendix A5. While we use the same general approach as in the pretraining phase, we no longer utilize the variational network. Instead, we now sample the discrete skill, $k$, and the continuous parameter, $z$, from the autoregressive policies $\\pi^K$ and $\\pi^Z$. This is done to make the finetuning process align with the process used to sample actions during inference. \n\n**Q4:** How do you ensure both $k$ and $z$ encode useful and distinct information?\n\n**A4:** Thank you for the insightful question! Empirically, the first ablation in A2 validates the need for both $k$ and $z$. \nOne of DEPS’ main goals is to ensure that $k$ and $z$ encode useful information. As outlined in Appendix A1, during pretraining, the variational network predicts continuous parameters per discrete skill for each forward pass over a given trajectory (instead of predicting a new continuous parameterization at each time step, which could lead to overfitting).  Additionally, to prevent the continuous parameters from overfitting to specific tasks, we introduce the $\\mathcal{L}_{\\text{norm}}$ regularization term (Appendix A1), which bounds the range of the learned continuous parameters. This limits the amount of information encoded in the discrete skill and continuous parameters, forcing the 1D state to encode meaningful information as an “index” into parameterized skill trajectories.  This is validated in Figure 8, where the learned latent 1D states vary monotonically within discrete skills, with discontinuities when the discrete skill switches. The strong performance of DEPS on out-of-distribution downstream tasks (Table 1, LIBERO-OOD) further supports our hypothesis that DEPS learns meaningful abstractions instead of overfitting to $k$ or $z$ values. This is particularly evident in low-data regimes (Table 1, LIBERO-3-shot) and low pretraining settings (Table 2).\n\nWe provide several lines of evidence that $k_t$ and $z_t$ do encode skill-relevant and distinct information:\n\n1. **Consistent application of discrete skills (Section 5.5 and Figure 7):** We find that DEPS discovers semantically meaningful discrete skills corresponding to primitive behaviors such as grasping objects, opening doors, etc. Importantly, we find that the discovered skills are applied *consistently* across tasks. An example can be seen in Figure 7, where the same ``grasp_object’’ skill is used to pick butter in a cabinet environment (top) and pick a mug in a kitchen environment (bottom). This is not a cherry-picked example; we can release an annotated version of the entire LIBERO dataset across multiple tasks, showing that learned discrete skills consistently hold across distinct tasks and environments.\n\n2. **Smooth variation in policy on varying continuous parameters (Section 5.5):** We find that for a given discrete skill, slight modifications in the continuous parameter result in smooth variations in the resulting policy. For example, modifying the continuous parameters for a discrete skill used to turn a knob smoothly varies the location at which the knob turning action is executed. Visualizations of this are provided on our website (which was already included at the time of submission; the URL is in section 5.5). \n\nThank you again for your constructive feedback, which has helped improve the clarity and position of our paper. Please let us know if you have any additional questions."}]

Author rebuttal:

Meta-review: This paper introduces DEPS, an end-to-end algorithm for learning parameterized skills from expert demonstrations, which is a non-trivial and valuable contribution. Experiments on LIBERO and MetaWorld show improved generalization and performance over baseline methods.

The main concerns include (1) limited analysis of the 1D state compression technique, (2) reliance on non-standard evaluation metrics, (3) the use of relatively weak baselines for LIBERO and MetaWorld, and (4) modest performance partly due to weak base models. The paper could be strengthened by providing deeper interpretation and validation of the compression technique, reporting results with standard imitation learning metrics (e.g., success rates at convergence or fixed training steps), and demonstrating scalability with more advanced methods (e.g., action chunking) and stronger baselines (e.g., ACT, Diffusion Policy).

Overall, I lean toward accept, as the paper presents a non-trivial and well-executed approach to parameterized skill discovery from demonstrations, with potential for meaningful impact despite the noted limitations.
Decision: Accept (poster)