EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data
Authors: Jesse Zhang
Minho Heo
Zuxin Liu
Erdem Biyik
Joseph J Lim
Yao Liu
Rasool Fakoor
Venue: CoRL 2024 (2024)
Keywords: reinforcement learning
skill-based reinformement learning
skill learning
transfer learning
foundation models for robotics
robot learning
TL;DR: We extract discrete, meaningfully-aligned skills from offline data for efficient reinforcement learning of new tasks.

Abstract: Most reinforcement learning (RL) methods focus on learning optimal policies over low-level action spaces. While these methods can perform well in their training environments, they lack the flexibility to transfer to new tasks. 
Instead, RL agents that can act over useful, temporally extended skills rather than low-level actions can learn new tasks more easily.
Prior work in skill-based RL either requires expert supervision to define useful skills, which is hard to scale, or learns a skill-space from offline data with heuristics that limit the adaptability of the skills, making them difficult to transfer during downstream RL.
Our approach, EXTRACT, instead utilizes pre-trained vision language models to extract a discrete set of semantically meaningful skills from offline data, each of which is parameterized by continuous arguments, without human supervision. 
This skill parameterization allows robots to learn new tasks by only needing to learn when to select a specific skill and how to modify its arguments for the specific task.
We demonstrate through experiments in sparse-reward, image-based, robot manipulation environments that EXTRACT can more quickly learn new tasks than prior works, with major gains in sample efficiency and performance over prior skill-based RL.

EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data
Jesse Zhang', Minho Heo², Zuxin Liu', Erdem BıyIk', Joseph J. Lim?, Yao Liu", Rasool Fakoor
University of Southern California, PKAIST, ²MUU, 4 "Amazon Web Services JessezQusc.edu
Most reinforcement learning (RL) methods focus on learning optimal policies over low-level action spaces. While these methods can perform well their training environments, they struggle with learning new tasks. Instead, RL agents acting over useful, temporally extended skills rather than low-level actions can learn new tasks more easily. Prior work in skill-based RL either requires expert supervision to define useful skills, which is hard scale, or learns skill- space from offline data with heuristics that limit the adaptability of the skills, making them difficult to transfer. Our approach, EXTRACT, instead utilizes pre- trained vision language models to extract discrete set of semantically meaningful skills from offline data, each of which is parameterized by continuous arguments, without human supervision. This skill parameterization allows robots learn new tasks by only needing to learn when to select specific skill and how to modify its arguments for the specific task. We demonstrate through experiments in sparse- reward, image-based. robot manipulation environments, both in simulation and in the real world, that EXTRACT can more quickly learn new tasks than prior works, with major gains in sample efficiency and performance over prior skill-based RL. Our website https: // jessezhang... net/ wwt.:nettteeeeeeeeeeee http//ooooctt//ttaacccss
Abstract:
Introduction
Imagine learning to play racquetball as complete novice. Without prior experience in racket sports, this poses a daunting task that requires learning not only the (1) complex, high-level strategies to control when to serve, smash, and return the ball but also (2) how to actualize these moves in terms of fne-grained motor control. However, a squash player should have considerably easier time adjusting racquetball as they already know how to serve, take shots, and return; they simply need to learn when to use these skills and how to adjust them for larger racquetball balls. Our paper aims to make use of this intuition to enable efficient learning of new robotics tasks.
In general, humans can learn new tasks quickly- given prior experience-by adjusting existing skills for the new task [1, 2]. Skill-based reinforcement learning (RL) aims to emulate this trans- fer [3, 4, 5, 7, 8, 10, 11, 12] in learned agents by equipping them with a wide range of skills (i.e., temporally-extended action sequences) that they call upon for efficient downstream learn- ing. Transferring to new tasks in standard RL, based low-level environment actions, is chal- lenging because the learned policy becomes more task-specific it learns to solve its training tasks [13, 14, 15, 16, 17]. In contrast, skill-based RL leverages temporally extended skills that can be both transferred across tasks and yield more informed exploration [6. 18, 19], thereby lead- ing to more effective transfer and learning. However, existing skill-based RL approaches rely on costly human supervision [20, 21, 9, 10] or restrictive skill definitions [22, 6, 8] that limit the ex- pressiveness and adaptability of the skills. Therefore, we ask: how can robots discover adaptable skills for efficient transfer learning without costly human supervision?
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

1. Offline
Skill Learning
3. Efficient Transfer to New Tasks
Dataset Skill
Extraction
Skill Trajectories
Policy
SMIEID
Cantinana ARR
Turn the stove knoo
2
Decoder
Skill
Pretraimed VLM
Skill Decoder
Open the diiweee
Figure 1: EXTRACT unsupervisedly extracts a discrete of skills from offline data that can be used for efficient learning of new tasks. (1) EXTRACT first uses VLMs to extract a discrete set of aligned skills from image-action data. (2) EXTRACT then trains skill decoder output low-Ievel
actions given discrete skill IDs and learned continuous arguments. (3) This decoder helps a skill- based policy efficiently learn new tasks with a simplified action space over skill IDs and arguments.
Calling back to the squash racquetball transfer example, we humans categorize different racket movements into discrete skills-for example, "forehand swing" is distinct from "backhand re- turn." These discrete skills be directly transferred by making minor modifications for racquet- ball's larger balls and different rackets. This process is akin to that of calling programmatic API, e.g., def forehand y) where learning to transfer reduces to learning when to call discrete functions (e.g., forehand vs backhand ) and how to execute them (i.e., what their arguments should be). In this paper, we propose a method to accelerate transfer learning by enabling robots to learn, without expert supervision, discrete set of skills parameterized by input arguments that are useful for downstream tasks (see Figure 1). We assume access to an offline dataset of image-action pairs of trajectories from tasks that different from the downstream target tasks. Our key insight is aligning skills by extracting high-level behaviors, i.e., discrete skills like "forehand swing," from images in the dataset. However, two challenges preclude realizing this insight: (1) how to extract these imput-parameterizee skills, and (2) how to guide online learning of new tasks with these skills.
To this end, we propose EXTRACT (Extraction of Transferable Robot Action Skills), a framework for extracting discrete, parameterized skills from offline data to guide online learning of new tasks. We first pre-trained vision-language models (VLMs), trained to align images with language de- scriptions [23] so that images of similar high-level behaviors are embedded to similar latent embed- dings [24]. to extract- from our offline data image embedding differences representing changes in high-level behaviors. Next, we cluster the embeddings in an unsupervised manner to form discrete skill clusters that represent high-level skills. To parameterize these skills, we train a skill decoder on these clusters, conditioned on the skill (e.g., representing a "backhand return") and learned argument (c.g., indicating velocity), to produce a skill consisting of a temporally extended, variable- length action sequence. Finally, to train robot for tasks, train skiel-based RL policy act over this skill-space while being guided by skill prior networks, learned from our offline skill data, guiding the policy for (1) when to select skills and (2) what their arguments should be.
In summary, EXTRACT enables sample-efficient transfer learning for robotic tasks by extracting a meaningful set of skills from offline data for an agent use for learning new tasks. We first validate that EXTRACT learns well-clustered set of skills. We then perform experiments across challeng- ing, long-horizon, sparse-reward, image-based robotic manipulation tasks, both in simulation and in the real world on a Panda Franka arm, demonstrating that EXTRACT agents can more quickly transfer skills to new tasks than prior work.
2 Related Work
Defining Skills Manually. Many works require manual definition of skills, e.g., as pre-defined primitives [4, 25, 26], subskill policies [27, 20, 28), or task sketches [29, 21], making them chal- lenging to scale to arbitrary environments. Closest to ours, Dalal et al. [9] and Nasiriany et al. [10]

---

hand-define a set of skills parameterized by continuous arguments. But this fand-definttion requires expensive human supervision and task-specific, environment-speciffe, or robot-specific fine-tuning. In contrast, EXTRACT automatically learns skills from offline data, which is much more scalable to enable learning multiple downstream tasks. We demonstrate in Section 5 that, given sufficient data coverage. skills extracted from data can transfer as effectively as hand-defined skills.
Unsupervised Skill Learning. A large body of prior work discovers skills in an unsupervised manner to accelerate learning new tasks. Some approaches use heuristics to extract skills from offline data, like defining skills as randomly sampled trajectories [30, 31, 32, 33, 34, 6, 8, 7, 35]. While these approaches have demonstrated that randomly sampled skill sequences can accelerate downstream learning, EXTRACT instead uses visual embeddings from VLMs to combine sequences performing similar behaviors into the same skill while allowing for intra-skill variation through their arguments. We show in Section that our skill parameterization allows for more efficient online learning than randomly assigned skills. Moreover, Wan et al. [36] also learns skills via clustering visual features; however. in addition to major differences in methodology, they focus on imitation lequiing-rqqirring significant algorithmic changes to facilitate learning new tasks online [37, 38, 39]. Instead, we directly focus on online reinforcement learning of new tasks.
Another line of work aims to discover skills for tasks without offline data. Some learn skills while simultaneously attempting to solve the task [3, 40, 5, 41, 19, 11]. However, learning the skills and using them simultaneously is challenging, especially without dense reward supervision. Finally, some prior works construct unsupervised objectives, typically based on entropy maximization, to learn task-agnostic behaviors [42, 43, 44, 45, 46]. However, these entropy maximization objectives lead to learning a large set of skills, most of which form random behaviors unsuitable for any mean- ingful downstream task. Thus, using them to learn long-horizon sparse- tasks difficult. We focus on first extracting skills from demonstration data, assumed have meaningful behaviors to learn from, for online learning of unseen, sparse-reward tasks.
3 Preliminaries
Problem Formulation. We assume access to an offline dataset of trajectories D {TI, .2.... where each trajectory consists of ordered image observation and action tuples, T₃ (s1,, (82,a2),-..) The downstream transfer learning problem is formulated sion Process in which want to learn a policy the offline dataset 'D does not contain trajectories from downstream task(s); we assume that the state space S has the same dimensions and that actions in D can be used to solve downstream tasks.
Markov Deci- maximize downstream rewards. We note that
SPiRL. In order to extract skills from offline data and use these skills for a new policy, we build on top of a previous skill-based RL method, namely SPiRL [6]. SPiRL focused on learning skills defined by randomly sampled, fixed-length action sequences. We briefly summarize SPiRL here: Given H-length sequences of consecutive actions from D: à ai,..., dH, SPiRL learns (I) generative skill decoder model. (ā 2). which decodes learned, latent skills encoded by skill encoder q(= @.) into environment action sequences ā, and (2) a state-conditioned skill prior [=(2] s) that predicts which latent skills 2 are likely to be useful at state s. To learn a new task, SPiRL trains skill-based policy <(> 8), whose outputs z are skills decoded by pa | 2) into low- level environment actions. The objective of policy learning is to maximize returns under T(2 s)
with KL divergence constraint to regularize against the prior (=(2 |8).
4 Method
EXTRACT aims to discover a discrete skill library from an offline dataset that can be modulated through input arguments for learning new tasks efficiently. EXTRACT operates in three stages: (1) an offline skill extraction stage, (2) an offline skill learning phase in which we train a decoder model to reproduce action sequences given a skill choice and its arguments, and finally (3) the online RL stage for training an agent to utilize these skills for new tasks. See Figure 2 for detailed overview.
3

---

Offline Skill Extraction
Offline Skill Learning
3. Online Skill-based RL
Image
Skill Trajectories
Pretrained VLM
Policy
Oftline
Actions &
State 55
Skill
Stale
Visual Difirense Embeldings
Skill Selection Policy kg(dla)
Clustering
Skill Ang Erunder 4(4(8. d) Skill Angument
Skill Sclection Price
(d(ss))
Skill of
Skill Clusters
Skill Visualization
Skill Argument Policy (2(1)
Skill
Skill Ang Pricr (.
Env
Augument
Frozen
Skill Decoderr
Skill Decoder Pa(O)
Pu(8)2,4)
Figure 2: EXTRACT consists of three phases. (1) Skill Extraction: We extract discrete set of skills from offline data by clustering together visual VLM difference embeddings representing high-level behaviors. (2) Skill Learning: We train skill decoder model, Pa | 2, d). to output variable-length action sequences conditioned on a skill ID d and a learned continuous argument z. The argument z is learned by training (a 2, d) with a VAE reconstruction objective from action sequences encoded by a skill encoder, g(= a, a.d). We additionally train a skill selection prior and skill argument prior pa(d s), P=(z] 8, d) to predict which skills d and their arguments are useful for a given state s. Colorful arrows indicate gradients from reconstruction, argument prior, selection prior, and VAE losses. (3) Online RL: To learn new task, we train skill selection and skill argument policy with RL while regularizing them with the skill selection and skill argument priors.
4.1 Offline Skill Extraction
Feature extraction. We leverage vision-language models (VLMs), trained align large corpora of images with natural language descriptions [23, 47, 48, 49], extract high-level features used to label skills. Although our approach does not require the use of language, utilize VLMs because, as VLMS were trained to align images with language, VLM image embeddings represent a semanti- cally aligned embedding space. However, one main issue precludes the naïve application of VLMs in robotics. In particular, VLMs do not inherently account for object variations or robot arm starting positions across images [50, 24, 51, 52]. But in robot manipulation, high-level behaviors should be characterized by changes in arm and object positions across a trajectory picking up a cup should be considered the same skill regardless of if the cup is to the robot's left or right. Our initial experi- ments of using the embeddings directly resulted in skills specific to type of environment layout or object. Therefore, to capture high-level behaviors, we use trajectory-level embedding differences by taking the difference of each VLM image embedding with the first one in the trajectory:'
VLM(ss)) VLM(s1).
Skill label assignment. After creating embeddings e, for each image st, we assign skill labels in an insupervised manner based on these features. Inspired by classical al- gorithms from speuker diarization, a long-studied prob- lem in speech processing where the objective is to assign a "speaker label" each speech timestep [53], we first perform unsupervised clustering with K-means on the en- tire dataset of embedding differences e; to assign per- timestep skill labels (the label is the cluster ID), then we smooth out the label assignments with a simple median filter run along the trajectory sequence to reduce the fre- quency of single or few-timestep label assignments. See Figure for visual demonstration of this process.
Offline Dataset Observations
VLM & K-Maane Chatviin
Each color regresents a different skill
Sugge
Median Filter Smoothing
New skill traj when Isbel changre
unds
Figure 3: Skill label assignment con- sists of (1) using the VLM embedding differences for clustering, then (2) ap- plying a median filter over the labels to smooth out noisy assignments.
"o ensure that each timestep has embedding, assign embedding er to be identical to ez.

---

In summary, we first extract observation embedding difference features with VLM and then per- form unsupervised K-means clustering to obtain skill labels for each trajectory timestep. This forms the skill-labeled dataset Da = ...... where each Ta is a trajectory of sequential (s, a) tuples that all belong to one skill d. Next, we perform skill learning on Da.
4.2 Offline Skill Learning
We aim to learn a discrete set of skills, parameterized by continuous arguments, similar to functional API over skills (see Figure 2 middle). Therefore, we train generative skill decoder Pa(a 2, d) to convert a discrete skill choice d and continuous argument for that skill, 2, into an action sequence. As alluded to in Section 3, we build upon SPiRL by Pertsch et al. [6]. However, they train their decoder to decode fixed-length action sequences from a single continuous latent 2. In contrast, we automatically extract a set of variable-length skill trajectories with labels denoted d and parameterize each skill by a learned, continuous latent argument 2.²
We train an autoregressive VAE [54] consisting of the following learned neural network components: a skill argument encoder q(z a, d) mapping to continuous latent conditioned on discrete skill choice and an action sequence a, and an autoregressive skill decoder Pa (ā 2, d) conditioned on the latent 2 and the discrete skill choice d.² Because the action sequence ä can of various lengths, the decoder also learns to produce a continuous value at each autoregressive timestep representing the proportion of the skill completed the current action. This variable is used during online RL to stop the execution of the skill when equals (see Appendix B.1 for further details).
Recall that SPiRL also trains a skill prior network (2(= 8) that predicts which = is useful for an observation 8; this prior is used to guide a high-level policy toward selecting reasonable = while performing RL. In contrast with SPiRL where z uniquely represents a skill, we train two prior networks, one to guide the selection of the skill d. Pa(d s), and one to guide the selection of its trained with the observation from the first timestep of argument given d, p=(= {z 8, s,d). These
the sampled trajectory, $1, to be able to guide skill-based policy during online RL in choosing d and z. Our full objective for training this VAE is to maximize the following:
E R.d.a soug(-ln,d))
log 2,
3 KL (9(2 d.d) || (). + log log 81) + log log s1.d)
(2)
action rec. prugress pood.
regularization
discrete skill prior
continuos arg. prior
where the stop-gradient s-(() prevents prior losses from influencing the encoder and & is sampled from the encoder g(= | a, d). The first two terms are the 3-VAE objective [55]; the last two train priors to predict the correct skill and continuous argument given $1-
Additional fine-tuning. On extremely challenging transfer scenarios, demonstrations may still be needed to warm-start reinforcement learning [56]. EXTRACT can also flexibly be applied to this setting by using the same K-means clustering model from Section 4.1, which was trained to cluster Da to assign skill labels to an additional, smaller demonstration dataset. After pre-training on Da, we then fine-tune the entire model on that labeled demonstration dataset before performing RL.
4.3 Online Skill-Based Reinforcement Learning
Finally, describe how perform RL for new tasks by training skill-based policy to select skills and their arguments to solve new tasks. See Figure 2, right, for an overview of online RL.
Policy parameterization. After pre-training the decoder (ā 2, d), we treat it as frozen lower- level policy that a learned skill-based policy can use to interact with a new task. Specifically, we train a baied-based policy <dd s) to output a (d, 2) tuple representing discrete skill choice and its continuous argument. We parameterize this policy as product of two policies: (dd, s) Ta(d (2 d) so that each component of (dd, s) can be regularized with our trained
TTo simplify notation, we use for both our method and SPIRL. However, is important note that uniquely determines the skill in SPiRI, while z denotes continuous latent argument in our method.
pa (a 2, d) can also be condiionn..o We opt not to for better transfer to new tasks with unseen states.
5

---

priors pa(d s) and (2 s, d). Intuitively, this parameterization separates decision decision-making into what skill to use and how to use it. The complete factorization of the skill-based policy follows:
xa 8) Pala 2, (d, 11) (a 2, 2, Ta(d 8) 7= s.d).
skill decoder
skill decoder
learned skill-based policy
Policy learning. We train the skill-based policy with online data collection using any entropy- regularized RI. algorithm, such as SAC [57] or RLPD [58], where we regularize against the skill priors instead of against a max-entropy uniform prior. Because we have factorized T(d, 2 {)) into two separate policies, we can easily regularize each with the priors trained in Section 4.2. The training objective for the policy with SAC maximize over Tdy Tg²
E a. (-1.(-)s.d)
Q 2, as (=(T=(> s,d) || P s, Ord KL(Ta(d 8) pa (-1 8))
(4)
skill argument guidance
skill choice guidance
where as and ag control the prior regularization weights. The critic objective is also correspond- ingly modified (see Appendix Algorithm 4). Despite the hierarchical architecture, this objective is stable to train the lower-level skill decoder is frozen and the priors regularize the high-level policy.
In summary, EXTRACT first extracts set of discrete skills from offinn image-action data (Sec- tion 4.1), then trains an action decoder to take low-level actions in the environment conditioned on discrete skill and continuous latent (Section 4.2), and finally performs prior-guides reinforce- ment learning over these skills online in the target environment to learn new tasks (Section 4.3). See Algorithm (appendix) for the pseudocode and Appendix B. for additional implementation details.
Experiments
Our experiments investigate the following questions: (1) Does EXTRACT discover meaningful, well-aligned skills from offline data? (2) Do EXTRACT-acqured skills help robots learn new tasks?
(3) What components of EXTRACT are important in enabling transfer?
5.1 Experimental Setup
We evaluate EXTRACT two long-horizon, continuous-control. robotic manipulation domains: Franka Kitchen [59] and LIBERO [60]. and the weal-world FurnitureBench [61]. All environments use image observations and sparse rewards. For both Franka Kitchen and LIBERO, our method EX- TRACT uses the R3M VLM [47] and K-means with K for offline skill extraction (Section 4.1). In FurnitureBench, K =
6. We list specific details below; see Appendix B.3 for more.
This environment, originally from Gupta et al. [22], Fu et al. [59] contains Franka Panda arm operating in kitchen environment. Simi- larly to Pertsch et al. [6]. we test transfer learning of sequence of 4 subtasks never performed in sequence in the dataset. Agents are given a reward of for completing each subtask.
Franka Kitchen:
LIBERO: LIBERO [60] consists of Franka Panda arm interacting with many objects and drawers. We test transfer to four task suites, HIBERO--Objeec Spatial, Goal, 10} consisting of 10 unseen environments/tasks each, span- ning various transfer scenarios (40 total tasks). LIBERO tasks are language conditioned (e.g., "turn on the stove and put the moka pot on it"); for pre- training and RL, we condition all methods on the language instruction. Due to LIBERO's difficulty [62], for pre-trained methods, we first fine-tune to provided additional tar- get task dataset with 50 demos per task before performing RL. During RL, we fine-tune all tasks within each suite simultancously. To the best of our knowledge, are the first report successful RL results on LIBERO tasks.

---

Claster Opening
PCA Cluster Embeddings
Grabbing Hlpping
Iraj
Cluster
Traj
Traj
Chipter
clúster
Clustepo
Chuster
Cluster TwistingGGrabeinng
Claster 3: Moving Towards Kottle
Fraj
Traj
Clu
Clusser
Figure 4: 100 randomly sampled trajectories from the Franka Kitchen dataset after being clustered into skills and visualized in 2D (originally 2048) with PCA. Even in 2 dimensions. clusters can be clearly distinguished. We visualize 2 randomly sampled skills in each cluster, demonstrating that our skill assignment mechanism successfully aligns trajectories performing similar high-level behaviors.
FurnitureBench: FurnitureBench [61] tests an agent's ability to assemble real- world furniture with Franka Panda arm. We pre-train on one-leg assembly data without initial object placement randomness and test real-world RL transfer to the same task with + 5cm of initial object and end-effector position randomness, plus + 15 degrees of end-effector angle randomization. We use RLPD [58], sample-efficiens actor- critic RL algorithm, for more efficient real-world training. RL is run for 100 training trajectories after pre-training on 500 demonstration trajectories.
Baselines and Comparisons. We compare: (1) oracle (RAPS [9]), which is given ground truth discrete skills, with continuous input arguments, designed by humans specifically for Franka Kitchen; (2) methods that pre-train with the same data-named SPiRL [6] which extracts sequences of fixed-length random action trajectories as skills, EXTRACT-UVD which replaces our discrete skill extraction with UVD's "LM-based mechanism [63], and BC, behavior cloning using the same offline data but no temporally extended skills; and (3) SAC [57], i.e., RL without any offline data. See Appendix for implementation details. Sim results include standard deviations over seeds.
5.2 Offline Skill Extraction
ability to discover meaningful, well-aligned skills during skill extraction. In Figure 4, we plot K-means (K = 8) skill assignments in Franka Kitchen. We project VL.M em- bedding differences down to 2-D with PCA for visualization. These skill assignments demonstrate that unsupervised clustering of VLM embedding differences can create distinctly separable cluster- ing assignments. For example, skill 4 (Figure 4. top left) demonstrates cabinet opening behavior. See additional visualizations for all environments in Appendix C.I. We also analyze quantitative clustering statistics in Appendix C.2. Next, let's see how these skills help with learning new tasks.
We first test EXTRACT's
5.3 Online Reinforcement Learning of New Tasks
Simulated Envs. We investigate the ability of all methods to transfer to new tasks in simulation in Figure 5. In Kitchen, EXTRACT matches the oracle performance while being 10x more sample- efficient than SPiRL, with SPiRL needing 3M timesteps to reach EXTRACT's performance at 300k. In all LIBERO suites, EXTRACT performs best in either sample efficiency or final performance due to its discrete-continuuus skill separation enabling easier downstream RL; it outperforms SPiRL and EXTRACT-UVD the most in LIBERO-10, the suite with the longest-horizon tasks. EXTRACT- UVD is unstable Franka Kitchen (see Appendix C.4 for analysis) and generally performs worse as UVD's s skill extraction mechanism does not perform our discrete skill clustering. Meanwhile, SAC and BC perform poorly, indicating our tasks are difficult to solve with standard RI or without skills.
7

---

Franka Kitchen
LIBERO-Object
LIBERO-Spatial
LIBERO-GOAl
LIBERO-10
2.0
1.0
1.0
1.0
Average Retum
Average Return
Average Rettrrn
Average Return
Average Koturn
1.0
1.5
0.8
0.8
2.0
0.5
0.5
0.5
0.5
02
02
82
0.0
0.0
1.0
au
Environment Steps (IM)
Environment Stepa (IM)
Envirooment Steps (IM)
Environment
Steps (1M)
Environment Steps (IM)
Oradle
EXTRACT
EXTRACT-OVD
SPIRL
SAC
BC
Figure 5: EXTRACT outperforms SPiRL and EXTRACT-UVD in RL across all comparisons,
demonstrating the advantages of our clustered skill-space. SAC and BC struggle, demonstrating the need for skill-based RL. In LIBERO-(Objee (Object, Spatial, Goal}, return is success rate.
Our method outperforms others due to its semantically aligned, discrete skill-space. For example, to open drawers, EXTRACT's policy only needs to learn a single discrete drawer-opening skill when the gripper is near any drawer. In contrast, SPiRL requires memorizing and distinguishing continuous skills for each specific drawer-opening behavior. Additionally, EXTRACT allows easier exploration later in the task, enabling the policy to reuse the same skill for other drawers. For more details, Appendix E. Next, conduct ablation study on EXTRACT's components.
Real world. Finally, we assess EXTRACT's real-world performance on FurnitureBench for one-leg assembly in Table 1. We report the average
Table 1 Furniture RI.
Method
Start
End
completed subtask (20 trials) out of a maximum of EXTRACT outper-
SPIRI
1.35
1.55
forms SPiRL both before and after 100 episodes of real-world RL fine-
EXTRACT
1.90
2.50
tuning, showing effective skill transfer. Overall, EXTRACT excels across 42 tasks and 3 domains, outperforming other skill-based RL, BC, and online RL methods, both in simulation and on robots.
5.4 EXTRACT RL Ablation Studies
VLMs. We first ablate the use of VLMs from selecting features for clus- tering. Therefore, we compare against Action, where skill labels are gener- ated by clustering robot action differences. We also compare against State where skills are labeled by clustering ground truth state differences (c.g.. robot joints, states of all objects). State represents an oracle scenario as ground truth states of all relevant objects are difficult to obtain in the real world. We plot results in Franka Kitchen in Figure 6. EXTRACT with VL.M- extracted skills performs best, as both ground truth state and raw environment action differences can be difficult to directly obtain high-level, semantically meaningful skills from. For ablations against pure proprioception and CLIIP [23], see Appendix C.3.
Outs
State
0.0
0.5
1.0
Env Steps (1M)
Figure 6: Embed- ding ablations.
Number of Clusters. Finally, we ablate the number of K-means clusters. Too few too many clusters can affect RL by balancing the ease of selecting the correct discrete skill against the complexity of choosing the right continuous argument. In Figure 7, we show average returns at 1M timesteps for EXTRACT in Kitchen with K = 3, 5, 8, 15. Performance remains stable, with a drop only at K = 15, indicating that EXTRACT robust to variations in the number of discovered discrete skills.
2.
Average Retumoo
1.5
K-15
Figure 7: Kitchen K ablations.
Discussion
We presented EXTRACT, method for enabling efficient agent transfer learning by extracting a discrete set of input-are papuuatr parameterized skills from offline data for a robot use in new tasks. Compared to standard RI., our method operates over temporally extended skills rather than low-level environment actions, providing greater flexibility and transferability to tasks, demonstrated by our comprehensive experiments. Our experiments demonstrated that EXTRACT performs well across 41 total tasks across 2 robot manipulation domains. We discuss limitations in Appendix F.
8

---

Acknowledgments
Most of this work was performed while Jesse Zhang and Zuxin Liu were interns at Amazon Web Services. After the internships, this work supported by USC Viterbi Fellowship, compute infrastructure from AWS, Institute of Information & Communications Technology Planning & Evaluation (ITP) grants -No.RS-2019-11190075, Artificial Intelligence Gradu- ate School Program, KAIST; No. R8S-0022-11229944 Development of Artificial Intelligence Technology for Personalized Plug-and-Playy Explanation and Verification of Explanation), a Brain
National Research Foundation of Korea (NRF) grant SOAF-2021HID3AAA0011000 Pool Research Program) funded by the Korean government (MSIT), Electronics and Telecom- munications Research Institute (ETRI) grant funded by the Korean government foundation [24ZB 1200, Research of Human-centered autonomous intelligence system original technol- ogyl, and Samsung Electronics Co., Ltd (0020086-02015520 Smith and Sidhant Kaushik for their valuable feedback on early versions of the paper draft.
Finally, we thank Laura
References
[1] Fitts and M. Posner. Human Performance. Basic concepts in psychology series. Brooks/Cole Publishing Company, 1967. ISBN 9780134452470.
[2] R. Anderson. Acquisition of cognitive skill. Psychological Review, 89:369 406, 1982.
[3] R. S. Sutton, D. Precup, and Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence. 11(()))11-211, 1999. ISSN 0004-3702. 1,3
[4] Schaal. Dynamic movement primitives-a framework for motor control in humans and hu- manoid robotics. Adaptive Motion of Animals and Machines, 01 2006. 1,2
[5] K. Hausman, J. T. Springenherg, 7.. Wang, N. Heess, and M. Riedmiller. Learning embed- ding space for transferable robot skills. In ICLR, 2018. 1,
[6] K. Pertsch, Y. Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL), 2020. 1, 5, 6, 7, 16, 17, 18, 25
[7] J. Zhang. K. Pertsch, J. Yang, and J. J. Lim. Minimum description length skills for accelerated reinforcement learning. In Self-Supervision for Reinforcement Learning Workshop ICLR 2021, 2021. 3,
[8] A. Ajay. A. Kumar, P. Agrawal, S. Levine, and O. Nachum. (OPAL): Offline primitive discov- ery for accelerating offline reinforcement learning. In International Conference on Learning Representations, 2021. 1,3
[9] M. Dalal, D. Pathak, and R. Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. NewrIPS, 2021. 2, 17
[10] S. Nasiriany, H. Liu, and Y. Zhu. Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In IEEE International Conference on Robotics and Automation (ICRA), 2022. 1.2
[11] G. Zhang, A. Jain, Hwang, S.-H. Sun, and Lim. Every policy has something share: Efficient mult reinforcement learning via selective behavior sharing, 2023. 1,
[12] Zhang, Pertsch, J. Zhang, and J. Lim. Sprint: Scalable policy pre-training via language instruction relabeling. In International Conference on Robotics and Automation, 2024.
[13] Schmidhuber, Zhao, and M. Wiering. Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-improvement, Machine Learning, 28(1):105-130,
Jul 1997. ISSN 1573-0565. 1

---

[14] S. Thrun. Is learning the n-th thing any easier than learning the first? In Advances in neural information processing systems, pages 640-646, 1996.
[15] M. Taylor and P. Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
[16] R. Fakoor, P. Chaudhari, S. Soatto, and A. J. Smola. Meta-q-learning. In International Con- ference on Learning Representations, 2020.
[17] M. Caccia, Mueller, Kim, Charlin, and Fakoor. Task-agnostic continual reinforce- ment learning: Gaining insights and overcoming challenges. In Conference on Lifelong Learn- ing Agents, 2023. 1
[18] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine. Parrot: Data-driven behavioral priors for reinforcement learning. ICLR, 2021.
[19] Zhang, H. Yu, and W. Xu. Hierarchical reinforcement learning by discovering intrinsic options. In International Conference Learning Representations, 2021. 1,
[20] Y. Lee, S.-H. Sun, S. Somasundaram, E. S. Hu, and J. Lim. Composing complex skills by learning transition policies. In International Conference on Learning Representations, 2018. 1,2
[21] K. Shiarlis, M. Wulfmeier, S. Salter, Whiteson, and Posner. Taco: Learning task decom- position via temporal alignment for control. ICML, 2018. 1,2
[22] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. CoRI., 2019. 1, 6, 18
[23] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, Askell, Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021. 2, 4, 8, 19,22
[24] S. A. Sontakke, J. Zhang, S. Amold, K. Pertsch, E. Biyik, D. Sadigh, C. Finn, and L. Itti. RoboCLIP: One demonstration is enough to learn robot policies. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 2, 4
[25] P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal. Learning and generalization of motor skills by learning from demonstration. In 2009 IEEE International Conference Robotics and Automation, pages 763-768. IEFF, 2009. 2
[26] H. Lin, R. Corcodel, and D. Zhao. Generalize by touching: Tactile ensemble skill transfer for robotic furniture assembly, 2024. 2
[27] J. Oh, S. Singh, H. Lee, and Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference Machine Learning, 2017. 2
[28] D. Xu, S. Nair, Zhu, J. Gao, A. Garg, L. Fei-Fei, and Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In International Conference on Robotics and Automation, 2018. 2
[29] J. Andreas, D. Klein, and S. Levine. Modular multitask reinforcement learning with policy sketches. In International Conference on Machine Learning, pages 166-175 175.175.77 PMLR, 2017. 2
[30] T. Kipf, Y. Li, H. Dai, V. Zambaldi, E. Grefenstette, P. Kohli, and P. Battaglia. Compositional imitation learning: Explaining and executing one task a time. ICML, 2019. 3
[31] T. Shankar, S. Tulsiani, L. Pinto, and A. Gupta. Discovering motor programs by recomposing demonstrations. In ICLR, 2019. 3
10

---

[32] J. Merel, S. Tunyasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, Pham, T. Erez, G. Wayne, and N. Heess. Catch & carry: Reusable neural controllers for vision-guided whole-body tasks. ACM. Trans. Graph., 2020.
[33] T. Shankar and A. Gupta. Learning robot skills with temporal variational inference. In Inter- national Conference on Machine Learning, pages 8624-8633. PMLR, 2020. 3
[34] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on Robot Learning, pages 1113 1132, 2020.
[35] L. X. Shi, Lim, and Y. Lee. Skill- model-based reinforcement learning. In Conference on Robot Learning, 2022.
[36] W. Wan, Y. Zhu, R. Shah, and Y. Zhu. Lotus: Continual imitation learning for robot manipula- tion through unsupervised skill discovery. In 202/ IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[37] A. Nair, A. Gupta, M. Dalal, and S. Levine, Awac: Accelerating online reinforcement learning with offline datasets, 2021. 3
[38] A. Kumar, J. Hong, A. Singh, and S. Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.056/1 2022.
[39] Q. Zheng, A. Zhang, and A. Grover. Online decision transformer, 2022. 3
[40] P.-L. Bacon, J. Harb, and D. Precup. The option-critic architecture. In Association for the Advancement of Artificial Intelligence, 3017.
[41] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. In Neural Information Processing Systems, 2018. 3
[42] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. In ICLR, 2019.
[43] D. Warde-Farley, T. V. de Wiele, T. Kulkarni, C. Ionescu, S. Hansen, and V. Mnih. Unsuper- vised control through non-parametric discriminative rewards. In ICLR, 2019. 3
[44] K. Gregor, G. Papamakarios, F. Besse, L. Buesing, and T. Weber. Temporal difference varia- tional auto-encoder. In International Conference Learning Representations, 2019. 3
[45] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics- aware unsupervised discovery of skills. In ICLR, 2020. 3
[46] M. Laskin, H. Liu, X. B. Peng, D. Yarats. A. Rajeswaran, and P. Abbeel. Cie: Contrastive intrinsic control for unsupervised skill discovery, 2022. 3
[47] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and Gupta. R3m: A universal visual represen- tation for robot manipulation. In CoRL, 2022, 4, 6, 19,21
[48] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06177 2022. 4
[49] Y. J. Ma, W. Liang V. Som, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman. Liv: Language-image representations and rewards for robotic control, 2023. 4
[50] Y. Cui, S. Nickum, A. Gupta, V. Kumar, and A. Rajeswaran. Can foundation models per- form zero-shot task specification for robot manipulation? In R. Firoozi, Mehr, E. Yel, R. Antonova, J. Bohg, M. Schwager, and M. Kochenderfer, editors, Proceedings of The 4th Annual Learning for Dynamics and Control Conference, volume 168 of Proceedings of Ma- chine Learning Research, pages 893-905. PMLR, 23-24 Jun 2022. 4
11

---

[51] J. Rocamonde, V. Montesinos, E. Nava, E. Perez, and D. Lindner. Vision-language models are zero-shot reward models for reinforcement learning. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023. 4
[52] Y. Wang, Z. Sun, J. Zhang, Z. Xian, E. Biyik, D. Held, and Z. Erickson. RI-vim-f: Reinforce- ment learning from vision language foundation model feedback. In International conference on machine learning, 2024. 4
[53] X. Anguera, S. Bozonnet, N. Evans, Fredouille, G. Friedland, and Vinyals. Speaker diarization: review of recent research. IEEE Transactions audio, speech, and language processing, 20(2)355-370, 2012. 4, 16
[54] D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
[55] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2016. 5
[56] I. Uchendu, T. Xiao, Y. Lu, B. Zhu, M. Yan, J. Simon, M. Bennice, C. Fu, C. Ma, J. Jiao, S. Levine, and K. Hausman. Jump-start reinforcement learning, 2023. 5, 16
[57] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML, 2018. 6, 7, 15, 17
[58] P. J. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data, 2023. 6, 7, 17, 19
[59] Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint Xiix:200440022199 2020. 6, 18
[60] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning. 2023. 16, 18
[61] M. Heo, Lee, D. Lee, and J. Lim. Furniturebenc:: Reproducible real-world benchmark for long-horizon complex manipulation. Robotics: Science and Systems, 2023. 6, 7, 16, 19
[62] Z. Liu, Zhang, K. Asadi, Liu, D. Zhao, S. Sabach, and Fakoor. TAIL: Task-spectis adapters for imitation learning with large pretrained models. In ICLR, 2024. 6, 16
[63] Z. Zhang, Y. Li, O. Bastani, A. Gupta, D. Jayaraman, Y. J. Ma, and Weihs. Universal visual decomposer: Long- horizon manipulation made easy, 2023, URL https: pattps://ariiv. /awxiv.vvvvv org/ org/abs/ abs/ 2310. 08581. 7, 17, 19
[64] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski,
P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, i. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, H. Ribeiro, F. Pedregosa, van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing Python, Nature Methods, 17:261-272 2020. 16
[65] Christodoulou. Soft actor-critic for discrete action settings, 2019. 17
[66] Y. Zhu, Wong, A. Mandlekar, R. Martin-Martin, Joshi, S. Nasiriany, and Y. Zhu. ro- bosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.1220 12293, 2020. 18
12

---

[67] N. Reimers and I. Gurevych. Sentence-berr bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. 18
[68] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks, 2020. URL https: / arxiv. org/ 0ttb///1...000.3.. 07035. 19
[69] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu. C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do. M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, A. Gebreselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, S. Kottur, A. Kumar, Landini, C. Li, Y. Li, 7. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Y. Sug- ano, Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Z. Zhao, Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, C. Fuegen, B. Ghanem, K. Ithapu, C. V. Jawahar, H. Joo, K. Ki- tani, H. Li, R. Newcombe, A. Oliva, H. S. Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, L. Torresani, M. Yan, and J. Malik. Ego4d: Around the world in 3,000 hours of egocentric video. 2022. 22
[70] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018. 25
[71] G. Zhang, L. Zhong, Y. Lee, and J. J. Lim. Policy transfer across visual and dynamics domain gaps via iterative grounding. Robotics: Science and Systems, 2021. 25
[72] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo-
ration. In International Conference on Machine Learning, pages 2052-2062, 2019. 25
[73] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off off-policy reinforcement learning, 2019. 25
[74] S. Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01664, 2020. 25
[75] A. Singh, A. Yu, J. Yang, J. Zhang, A. Kumar, and S. Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. CoRL, 2020. 25
[76] R. Fakoor, J. W. Mueller, K. Asadi, P. Chaudhari, and A. J. Smola. Continuous doubly strained batch reinforcement learning. Advances in Neural Information Processing Systems, 34:1 1260-11273, 2021. 25
[77] Y. Liu, Chaudhari, and R. Fakoor. Budgeting counterfactual for offline rl. In Advances in
Neural Information Processing Systems, volume 36, pages 5729-5751, 2023. 25
13

---

A Full Algorithm
Algorithm EXTRACT Algorithm, Section 4.
Require: Dataset D. VLM, Target MDP M, Optional target task fine-tuming tinettunigg dataset DM
1: Da. CM <- OFFLLESSILLEXTTAATTOON model, Algorithm
VLM) Get discrete skill labels and clustering
2: Init q(2 a. ( z, 2, s2, d)
Skill argument encoder, skill decoder,
discrete skill prior, continuous argument prior
9, Pur Pa.Pad Ps -FFLLNSSKILLLARNNNNGG 9, Pas Pa. P&)
Learn skills offline,
Algorithm 3
4: if DM exists then
5:
DM,d t Assign skills to DM with existing clustering model CM
6: q. Pa: Pa- OFLLINSSKILLAANNNNDD q: Pai Pds Ps) Optionally fine-tune on target task M
7: end if
8: BATLL..BSSDDONINERR..COM Pa: Pa;
RL on target task M, Algorithm
Algorithm Offline Skill Extraction, Section 4.1.
1:
procedure OFL.IESSKI.EXXYTAATIIOOOC VLM)
2: EMBEDS + 0
> Init VLM embedding differences
3:
for trajectory [($1, a1), ... (sT, ar)] do for (si, a;) in do
4:
5:
e, VLM(s;; VLM(s1)
b Embedding differences, Equation (1)
6:
7:
end for end for
8:
9:
CM < Init (K-Means) clustering model
10: LABELS <- CCM(EBBEDS)
D Run unsupervised clustering to get cluster labels
Da - 0
& Init skill labeled dataset
12:
for trajectory
[[s1, (1),... (ST, ar) in D do di...., dr t- Get labels from LABELS
13:
14:
dp M.. dr MEDIANFILTER(dI; ..., dr) Du + Da [(81, d2), (sT, 01 drr) end for
Smooth out labels, see Appendix B.1
15:
16:
17: return Dá, CM
18:
end procedure
Algorithm Offline Skill Learning, Section 4.2.
procedure OFFLNEESKILLEARNINNCO. q, Pas Pas Pz)
2: while not converged do
3: Sample Ta from Da
4: Train q, Po. Pa. Ps with Equation (2)
5: end while
6:
return q. Pa.
7:
end procedure
We present the full EXTRACT pseudocode Algorithm 1. Algorithm 2 details offline skill extrac- tion using a VLM, Algorithm details the offline skill learning procedure, and Algorithm 4 details how to perform online skill-based RI. on downstream tasks using Soft Actor-Critie (SAC). Note that any entropy-regularizzed algorithm can be used here with similar modifications, not just SAC. Differences from SAC during online RL are highlighted in red. For further implementation details and hyperparameters of EXTRACT, see Appendix B.I.
14

---

Algorithm Skill-Based Online RL (with SAC [57]), Section 4.3, Red marks policy and critic loss differences against SAC.
procedure SKILLBASFDDOL SXI.BB.STTON..NRLMMM...... 2,d), P=(p
8,d)) Section 4.3
2: Freeze Pā z. d). Pa.Ps weights
3: Ta(d 5) - Pa(d s)
Init iT as discrete skill prior Pa Init IT as cont. argument prior Pa D Init buffer
4: (z 8, - P= s,d)
5:
B+0
B
6:
for each rollout do
7:
+
8:
Sample discrete skill > Sample continuous argument for skill Sample action sequence (11 ... 0r and progress predictions 1....,, up to max sequence length I, see Appendix 1.1.2.
di Ta(d 8t)
9:
21 ~ (= 8, de) ,,,,.., 0T,, 11...
10:
Pa(a z.)d)
11:
for in 11--.-., dz until / 1 do
12:
Execute actions in M, accumulating reward sum řzz end for B B+BuB B Zt, Fr, St
14:
Add sample to buffer Sample from B
(s, 2,
16:
Tz < max Q(s, 2,
a (L s. ll p. (s,
18:
a (lall s) p s))
> Update policies, Equation (4)
Q + minQ Q(s, -(,,,,d) +0Q(s' "''..'d')
20:
a, ( s.d) s,)))
DaD pa
b Update critic
22:
end for end procedure
B Experiment and Implementation Details
In this section, we list implementation details for EXTRACT (Appendix B.1), the specific environ- ment setups (Appendix B.3), and details for how we implemented baselines (Appendix B.2).
B.1 EXTRACT Implementation Details
EXTRACT implementation details follow in the same order as each method subsection was pre- sented in the main paper in Section 4.
B.1.1 Offline Skill Extraction
We first extract skills from dataset D using VLM by clustering VLM embedding differences of image observations in D (see pseudocode in Algorithm 2).
Clustering. We use K-means for the clustering algorithm as it is performant, time-efficient, and can be easily utilized in batched manner if all of the embeddings are too large to fit in memory at once.* When extracting skills from the offline dataset DD we utilize K-means clustering on VLM embedding differences with K = Franka Kitchen and LIBERO, as we found duce the most visually pleasing clustering assignments in Franka Kitchen and we directly adapted the Franka Kitchen hyperparameters LIBERO to avoid too much environment-specffcc tuning. In FurnitureBench, found K 6 to produce the most visually distinguishable clustering assign- ments.
to pro-
*We did perform preliminary experiments early on with DBSCAN, which doesn't require presetting the number of clusters. However. DBSCAN requires parameter which found to greatly affect the skill clustering results datasets, with some values of resulting very poorly clustered skills.
15

---

Median Filtering, After performing K-means, we utilize standard median filter, as is commonly performed in classical speaker diarization [53], to smooth out any possibly noisy assignments (see Figure 3). Specifically, we use the Scipy scipy.signal... medfilt (kernel_size-7) [64] filter for all environments. This corresponds to a median filter with window size 7 that slides over each trajectory's 's labels and assigns the median label within that window to all 7 elements. Empirically, found that this increased the average length of skills it reduced the occurrence of short, noisy assignments.
B.1.2 Offline Skill Learning
Here, we train a VAE consisting of skill argument encoder q(> a, d), skill decoder pa | d). discrete skill prior pa(d s), and continuous skill argument prior p=(z 8, d) (see pseudocode in Algorithm 3).
Model architectures. We closely follow SPiRL's model architecture implementations [6] as we build upon SPiRI.. The encoder g(= d, and decoder Pa @ 2, d) are implemented with recurrent neural networks. The skill priors are both standard multi-layer perceptrons. The skill argument space 2 has 5 dimensions. In Kitchen and LIBERO, our 3 for the 3-VAE KL regularization term in Equation (2) is 0.001.
Skill progress predictor. During training, for GPU memory reasons, we sample skill trajectories with a maximum length as is common when training autoregressive models. In Franka Kitchen, this is heuristically set to 30 based on reconstruction losses and in LIBERO, this is set to 40. In FurnitureBench, this is set to 30. a skill trajectory is longer than this maximum length, we simply sample a random contiguous sequence the maximum length within the trajectory. To ensure that predicted action sequences stay in-distribution with what was seen during training, we also use these maximum lengths as maximum skill lengths during online RL; e.g., if a skill runs for 30 timesteps in Franka Kitchen without stopping, we simply resample the next skill (see Line 10 of Algorithm 4).
As discussed in Section 4.2, given the variable lengths of action sequences ā the decoder Pa (ā 2, d) trained to generate a continuous skill progress prediction value at each timestep. This value represents the proportion of the skill completed the current time. During online policy rollouts, the execution of the skill is halted when I reaches 1. To learn this progress prediction value, formulate it as follows: when creating labels for such a sequence, step, denoted as yt. based on its position in the sequence. Specifically, yi is set to N for each time step t, where N represents the sequence length. To train the model for this function, we use the standard mean-squared error loss. This ensures that the model learns to predict the end of an action sequence while also ensuring that receives dense, per-timestep supervision while training function.
assign a label to each time
Additional target task fine-tuning, Optionally, for very difficult tasks, some target-task demon- strations may be needed [56, 62, 60]. We perform additional target task fine-tuning in LIBERO [60] and BunnhtureBench [61]. We use the learned clustering model that was trained cluster the original dataset D to directly assign labels to the task-specific dataset DM without up- dating the clustering algorithm parameters (see Algorithm Line 5). Then, we fine-tune the entire model, q, Pa. Pa, Ps. with the same objective in Equation (2) on the labeled target-task dataset D D,.d-
B.1.3 Skill-Based Online RL
For online RL, we utilize the pre-trained skill decoder Pa | 2, d), and the skill priors pa(d | s),p<(1 8, d) for skill-based policy learning (see Algorithm 4).
Policy learning. Our policy skill-based policy <dd, | s) is parameterized as a product of discrete skill selection policy Tz(d s) and continuous argument selection policy Tz(2 s, (see Equation (4)). To train with actor-critic RL, sum over the policy losses in each discrete skill dimension weighted by the probability of that skill, similar to discrete SAC loss proposed by
16

---

Christodoulou [65]:
Σ Tald s) - (L,
(s (.pp(-ls,d) - asKL ( s) pal: s))). (5)
-
Meanwhile, critic losses are computed with the skill d that the policy actually took. Our critic networks ((s, 2, take the image and argument input and have d-headed output for each of the skills.
We do not use automatic KL tuning (standard in SAC implementations [57]) as we found it to be unstable; instead, we manually set entropy coefficients az and az for the policy (Equation (4)) and critic losses. In Kitchen, Qa = 0.1, a= = 0.01; in LIBERO aa 0.1, 0₂ 0.1. These values are obtained by performing search over odt {0.1, 0.01] and a {001,
In FurnitureBench, we set Qt 0.5 and = 2.0 to prevent the policy losses from diverging significantly as we use RLPD [58] with a high critic update ratio of 2 per environment step and a higher policy update ratio of per environment step.
B.2 Baseline Implementation Details
Oracle. Our oracle baseline is RAPS [9]. We run RAPS to convergence and report final perfor- mance numbers because its expert-desinned skills operate different control frequency; it takes hundreds of times more low-level actions per environment rollout. We only evaluated this method on Franka Kitchen the authors did not evaluate our other environments, and we found the implementation and tuning of their hand-designed primitives to work well on other environments to be non-trivial and difficult to make work.
We adapt SPiRL, implemented on top of SAC [57], to our image- based settings and envi- ronments using their existing code to ensure the best performance. For each environment, we tuned SPiRL parameters (entropy coefficient, automatic entropy tuning, network architecture, etc.) first and then built method upon the final SPiRL network architecture ensure the fairest compar- ison. SPiRL uses the exact same datasets ours but without skill labels. We also experimented with changing the length of SPiRI. action sequences, and similar to what was reported in Pertsch et al. [6]. we found that a fixed length of 10 worked best. We also found fixed prior coefficients KL divergence to perform better with SPiRL for our environments than automatic KL tuning.
SPiRL.
EXTRACT-UVD. Universal Value Decomposer (UVD) segments trajectories into trajecccessss using VL.M features for image goal-conditioned behavior cloning [63]. was originally made for goal-conditioned imitation learning: we combine it with EXTRACT and adapt for our setting of online reward-based reinforeement learning by using to segment subtrajectories with the same VLM as EXTRACT, then treating each subtrajectory as separate skill trajectory to condition EX- TRACT's model on (without discrete skill extraction process). Essentially, this model acts as EX- TRACT but with skill trajectories determined by UVD's trajectory segmentation method instead of that of EXTRACT. This also makes the comparison against our method more fair as it receives temporally extended skills, just like SPiRL or EXTRACT.
BC. We implement behavior cloning with network architectures similar to ours and using the same datasets. Our BC baseline learns an image-conditioned policy 7 (a s) that directly imitates single- step environment actions. We fine-tune pre-trained BC models for online RL with SAC [57].
SAC. We implement Soft-Actor Critic [57] directly operating low-level environment actions with identical architecture to the BC bascline. It does not pre-train on any data.
17

---

B.3 Environment Implementation Details
(a) Franka Kitchen
(b) LIBERO
(c) FurnitureBench
Figure 8: Our two image-based, continuous control robotic manipulation evaluation domains. (a) Franka Kitchen: The robot must learn to execute an unseen sequence of sub-tasks in a row. (b) LIBERO: We evaluate 4 task suites of 10 tasks, each consisting of long-horizon, unseen tasks with new object, spatial, and goal transfer scenarios. (c) FurnitureBench: We evaluate online RL adaptation to unseen object and gripper placement randomizations.
Franka Kitchen. We use the Franka Kitchen environment from the D4RL benchmark [59] origi- nally published by Gupta et al. [22] (see Figure 8a). The pre-training dataset comes from the "mixed" dataset D4RL consisting of 601 human teleoperation trajectories each performing subtasks in sequence in the environment (e.g., open the microwave). Our evaluation task comes from Pertsch et al. [6]. where the agent has to perform unseen sequence of 4 subtasks. The original dataset contains ground truth environment states and actions; create an image-action dataset by resetting to ground truth states in the dataset and rendering the corresponding images. For all methods, we perform pre-training and RL with 64x64x3 RGB images and framestack 4. Sparse reward of given for each subtask, for a maximum return of 4, The agent outputs 7-dimensional joint veloc- ity actions along with a 2-dimensional continuous gripper opening/closing action. Episodes have maximum length of 280 timesteps.
LIBERO. LIBERO [60] is a continual learning benchmark built upon Robosuite [66] (see Fig- 8b). For skill extraction and policy learning, use the agentview_rgh 3rd-person cam- view images provided by the LIBERO datasets and environment. For pre-training, we the LIBERO-90 900 pre-training dataset consisting of 4500 demonstrations collected from 90 different environment-tase combinations each with 50 demonstrations. We condition all methods on 84x84x3 RGB images with a framestack 2 along with language instructions provided by LIBERO. We condition methods on language by embedding instructions with a pre-trained, frozen sentence em- bedding model [67], all 311--iniLM--66-2 to a single 384-dimensional embedding and then feeding it to the policy. For EXTRACT, we condition on language by conditioning all networks on language; 4. Pz, Pa: Pa all additionally conditioned the language embedding and thus the skill-based pol- icy is also conditioned language. We also condition all networks in all baselines on this language embedding in addition to their original inputs.
When performing additional fine-tuning to LIBERO-(10, Goal, Spatial, Object), for all methods (except SAC) we use the given task-specific datasets each containing 50 demonstrations per task before then performing online RL. In LIBERO-(Gone provided upon successfully completing the task, so the maximum return is 1.0. In LIBERO- 10, tasks are longer-horizon and consist of two subtasks, so we provide rewards at the end of each subtask for maximum return of 2.0. Episodes have max length of 300 timesteps.
Spatial, Object] ), sparse reward is
18

---

FurnitureBench. FurnitureBench [61] is a real-worid furniture assembly benchmark, where the task is to assemble 3D printed furniture pieces with a single Franka Arm (see Figure 8c). We closely reproduced the environment setup presented the original paper through manual camera calibra- tion of RealSense D435 cameras. For skill extraction and policy learning, we use two cameras, wrist-mounted camera and front mounted camera facing the arm workspace. To cluster skills, we embed both the wrist camera and front camera images with R3M and concatenate the embedding before clustering with K-Means (K 6). Following the baselines implemented in the paper, we encode all RGB images with the frozen R3M video encoder to a 2048 dimensional vector first for all components of skill/policy learning. Also following the paper, we don't use any framestacking. Additionally, despite the presence of AprilTTags, we do not use April'Tag-bar Tageeased state estimation for any part of our experiments; we perform purely image-based continuous control.
The pre-training dataset consists of 500 demonstrations from the one-leg assembly task collected by Heo et al. [61]. See Figure for an image of the pieces used. The environment action space is absolute 3D position control plus a 6-dimensional rotation representation [68]. We run real-world online RL with RLPD [58], sample-efficiett actor- critic algorithm that uses a high critic update ratio, layer norms in the Q functions, large number of Q functions, and samples from offline data and new online-collected data at 50/50 ratio. We did not have mod- ify the policy/training objectives in Algorithm 4 from SAC for RLPD. To train EXTRACT's high-level policy with offline data in RLPD, we use the skill d in the offline trajectories as high-level skill selection ac- tions for (d s) and encode offline sampled trajectory actions with the pre-trained, frozen encoder 7(= a, d) to obtain continuous arguments for T(2 s). The offline data also comes with sub-task completion timestamps (picked up table top, placed into the corner, picked up leg, inserted leg, screwed in) that we convert into +1 rewards for RLPD training for maximum return of 5.
Figure FurnitureBench table used for our one leg assembly task.
3D-printed
In our real-world RL setup, we run all methods for 100 trajectories with variations of 5em for the initial object positions and end effector positions, along with 1:1 degrees initial end effector rotation. Meanwhile, we use a dataset of 500 demonstrations from the "low randomness" dataset split for one-leg assembly from [61], which contains no intentional randomness for both the object and end effector poses. This is challenging the robot arm, camera positioning, etc., and now initial object and end effector locations, are all different from those in the dataset as collected by the Burniture authors and the policy must transfer its knowledge to this new setting to solve the task. We provide two rewards when training online RL: +1 for completing an assembly sub-task successfully and 0 for all other timesteps. The max return also
Episodes have maximum length of 500 timesteps. Each trajectory takes approximately 50s of robot interaction time when run completion and 30s reset, resulting in 11.5 minutes of real-world time per trajectory.
C Additional Experiments and Qualitative Visualizations
In this section, we perform additional experiments and ablation studies. In Appendix C.I, we visu- alize 2D PCA plots of clusters generated by EXTRACT in all environments. Appendix C.2, we analyze statistics of the skill distributions generated by EXTRACT. In Appendix C.3 for more abla- tion studies comparing using CLIP [23] or proprioceptive states instead of R3M [47] for clustering feature extraction. Finally, in Appendix C.4 analyze skills extracted through UVD [63] against ours.
19

---

C.1 Additional PCA Cluster Visualizations
PCA Cluster Embeddings
Embeddings
Cluster
Clustir 6
cli Cluster wapg
cluster
Caletee
Clusreo
%
Cluster
g
Guitte
Clustor
Cluster
Quteter
(a) Franka Kitchen.
(b) LIBERO.
Figure 10: 100 randomly sampled trajectories from environment pre-training datasets after being clustered into skills and visualized in 2D with PCA. Clusters are well-separated, even in just 2- dimensions with a linear transfromation.
Here we display PCA skill cluster visualizations in Figure 10. Franka Kitchen clusterings are very distinguishable, even in dimensions. (this is the same embedding plot as Figure 4 in the main paper). LIBERO-90 clusters still demonstrate clear separation, but are not as separable after being projected down to 2 dimensions (from 2048 original dimensions). However, in Figure 15 we clearly see distinguishable behaviors among different skills in LIBERO.
20

---

C.2
Visualizing Cluster Statisties
Histogramn Lengths
Histogrann Lengths
0.02
0.01
0.00
125
0.00
Box Mot Skill Langths
Gewrall Langth: 33.5
Box Met of Skill Langths Aug Overall Length: 30.9
of
Lengt: of Seil
220
Lungtht per Claster
Skill Lengths per Cluster
60
40
20
100
120
140
lungth
length
(a) Franka Kitchen, K = 8.
(b) LIBERO, K 8.
H Hittogram of Skill Lengths
0.03
0.02
0.01
Avg Skill Lengths: 24.3
200
300
250
200
150
100
Court
40
30
length
(c) FurnitureRench, K =
Figure 11: Skill/clustering statistics in all environments. We use the R3M VLM [47] and K for K-means. In FurniturBeench, K = 6. The top plots are skill length histograms for all skill trajectories combined, middle plots correspond to box-and-whisker plots with skill ID on the x-axis and lengths on the y-axis, and the bottom plots represent distributions of skill lengths separated by color for each skill ID.
We visualize skill clustering statisties in all pre- pre-training environments in Figure 11. The plots demonstrate that average skill lengths about 30 timesteps for all environments and that there
21

---

is clear separation among the different skills just in terms of the distributions of skill lengths that they cover. For a qualitative look at the skills, see Appendix D.
C.3 Additional Ablation Studies
Here, we augment Section 5.4 with additional ablation studies. We compare against CLIP [23] embeddings differences instead of R3M and against Proprio, i.c., robot joint and gripper state dif- ferences, Franka Kitchen in Figure 12,
We originally chose R3M as the base VLM because, in contrast with R3M, CLIP is trained on images, not videos, on a general dataset of image- language pairs instead of a dataset of humans performing real-world tasks from an egocentric viewpoint that more closely mimies roboties environ- ments (Ego4D, [69]). However, performs on par with EXTRACT with R3M. This demonstrates that EX- TRACT is robust to the choice of VLM for clustering; it's likely that CLIP was pre-trained on sufficient data to extract useful embedding differences for clustering. However, proprioceptive state differences are not as effec- tive as proprioception can be difficult directly obtain high-level, seman-
Average Return -
can see that EXTRACT with CLIP
CLIP
Proprio
0.0
1.0
Env Steps (1M)
tically meaningful skills from.
Figure 12: EXTRACT with R3M vs with CLIP or propriocep- tive states.
C.4 Visualizing UVD's Skill Extraction Ours
In Section 5.3, we found EXTRACT with UVD's skill extraction method to have un- stable RL performance in Franka Kitchen and overall performed worse than EXTRACT with our skill extraction method.
EXTRACT PCA Skill Embeddings
UVD Embeddings
a
Cubbee
cúmera
To analyze why, we plot R3M skill embed- dings of skill trajectories extracted by EX- TRACT against those of UVD, projected to dimensions, in Franka Kitchen in Figure 13. We can see that UVD-generated trajectory embeddings much harder to distinguish from each other than EXTRACT's, as evi- denced by the distinct separation seen in the 4 corners the plot compared to UVD.
Figure 13: 100 randomly sampled trajectories from Franka Kitchen after being projected to 2D with PCA. EXTRACT embeddings are identical those in Figure 10.
This difference in trajectory separability, combined with EXTRACT's skill clustering approach that forms discrete-continuous skill space for RL policies to learn new tasks with, helps explain the instability of RL with UVD in Franka Kitchen.
D Visualizing skill trajectories
Here, we visualize skill trajectories in our environments. In Figure 14, we visualize purely randomly sampled clusters (i.e., without any cherry-picking) in Franka Kitchen, where we see skills are gener- ally semantically aligned. For example, skill 3 trajectories correspond to manipulating knobs, skill trajectories reach for the microwave door, and skill 7 trajectories are reaching for the cabinet handle.
We visualize LIBERO skills in Figure 15, where we can also see that skills are generally aligned.
22

---

CLUSTER O
CLUSTER
CLUSTER 2
CLUSTER 3
"LUSTER 4
CLUSTER 5
CLUSTER 6
FLUSTER 7
Figure 14: Kitchen skill visualizations. We randomly sample labeled skill trajectories (no cherry- picking) and visualize the trajectory's 's images in sequence after labeling with EXTRACT's skill extraction phase. Clusters are generally semantically aligned.
23

---

CLUSTER O
CLUSTER
CLUSTER 2
CLUSTER 3
UUSTER 4
CLUSTER
LLUSTER CLUSTER 6
Figure 15: LIBERO. We randomly sample labeled skill trajectories (no cherry-picking! and vi- sualize the trajectory's 's images in sequence after labeling with EXTRACT's skill extraction phase. Clusters are generally semantically aligned.
E EXTRACT RL Performance Analysis
Our method's performance improvement SPiRL is likely due two reasons: longer average skills and semantically structured skill-space instead of the random latent skills that SPiRL learns. In Section 5.3 we analyze the semantically structured skill-space. Here, we additionally analyze the longer average skills.
As plotted in Appendix Figure 11, EXTRACT extracts skills of various lengths, many of which are quite long. This translates into longer-executed skills: we plot histogram of the lengths of the skills the skill-based policy actually leams use convergence in Franka Kitchen in Figure 16. CETRAACT-execute skills average 25 timesteps in length as compared to 10 for SPiRL. We experi-
24

---

0.4
0.3
0.2
0.1
0.0
0
5
10
15
20
25
30
Avg Skill Lengths: 24.8
Figure 16: Skill lengths histogram of actually used EXTRACT skills in Franka Kitchen at train- ing convergence. As explained in Appendix B.1, we limit skill execution lengths to 30 in Franka Kitchen.
mented with longer skill lengths for SPiRL, but online RL performance suffered, finding consistent with results presented in their paper [6].
Longer skills shorten the effective time horizon of the task by a factor the average skill length for the based agent because the skill-based agent operates on an MDP where transitions are defined by the end of execution of a skill which can be comprised of many low-level environment actions. By shortening the task time horizon, the learning efficiency RL algorithms [70] can be improved by, for example, reducing value function bootstrapping error accumulation as there are less timesteps between sparse reward signal and the starting state.
remporal-differeme learning
F
Limitations
While EXTRACT enables efficient transfer learning, we still need the initial dataset from envi- ronments similar to the target environments for learning skills from. It would be useful to extend EXTRACT to data from other robols or other environments significantly different from the target environment to case the data collection burden-possibly wth sim to real techniques [71]. Further- more, in future work, we plan combine our method with offline RL [72, 73, 74, 75, 76, 77] to learn skills from suboptimal data without the need interact with environment, targeting even greater sample efficiency. Furthermore, while EXTRACT is sample-efficient than all other comparisons, it still requires many online samples to learn to execute new tasks with RL. We plan to investigate future directions that will allow to combine offline learning approaches such as offline reinforcement learning with skill learning on the offline dataset to allow even more efficient transfer. Finally, EXTRACT requires image observations for the VLMs; skill learning from more input modalities would be interesting future work. Processing more input modalities to learn skills would be interesting future work.
25


Reviews:
[{"review_id": "5DAO0cuLhO", "reviewer": "Reviewer_moSg", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper shows that discrete sets of skills can be extracted with embedding differences from a trajectory of images using a VLM (R3M). With these discrete skill-sets, after smoothing and clustering, a skill decoder network can be trained to output actions given a skill ID and vector of arguments. Policy learning then becomes identifying what skill and arguments for a given task. This method is demonstrated on the Franka Kitchen and LIbero experimental setups in simulation. Results show improvement over previous method SPiRL.", "summary_of_recommendation": "This is a strong paper. A pathway to hardware experiments should be considered. -> Recommendation changed after rebuttal and inclusion of hardware experiments.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "See above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "82QXZ5lltO", "reviewer": "Reviewer_2VAd", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a unified approach towards skill discovery from an offline dataset and skill-guided hierarchical reinforcement learning. A pre-trained visual language model is used to extract robust embeddings from images. K-Means clustering is then employed to categorize these visual embeddings into a fixed number of skills. Following this, a stochastic action decoder is learned. This decoder samples variable-length feasible trajectories from discrete skill labels using a conditional variational autoencoder in an autoregressive manner. Finally, the approach is transferred to new tasks online. Two maximum entropy reinforcement policies are learned for selecting optimal skills and optimal actions given the skills. The effectiveness of this approach is demonstrated by its ability to transfer to new tasks on two datasets. This approach provides a comprehensive method for offline skill discovery and online task adaptation, showcasing the potential of combining unsupervised learning with reinforcement learning in complex environments.", "summary_of_recommendation": "Overall, this paper is presents a new formulation of employing VLM features for skill driven RL, the approach is well motivated and clearly presented. However, the evaluation is bit weak, the paper could benefit from comparison with stronger baselines and real robot experiments.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "1.\tAblation study suggests that performance stays similar for 3-12 clusters, and only 3 skills are enough to solve most of the franka kitchen tasks. Can authors give insight on this issue?\n2.\tTraining hierarchical policies could have stability issues, what additional measures were used to fine tune policies?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h8Qihgccyn", "reviewer": "Reviewer_Jq78", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes EXTRACT, an approach to using pretrained visual-language models to segment and cluster trajectories into a discrete set of skills, and a novel framework for learning parameterized policies in an offline manner, as well as learning high-level skill selection policy in an online manner.", "summary_of_recommendation": "I lean towards accepting this paper, but the experimental validation is lacking (both in terms of comparison to baselines for skill extraction and hardware experiments)", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "- Why were there no experiments done comparing the offline skill extraction method to other skill extraction methods like standard segmentation approaches based on proprioception / gripper state change, or other SOTA methods like UVD [1]?\n- How would the offline skill segmentation approach deal with a skill like opening and closing, where each individual image can appear in both trajectories, and is only discernable when using multiple frames? Would it just make one cluster for opening/closing?\n- In figure 2, should there be a reconstruction arrow pointing from [a1,a2,a3,...] to the skill decoder?\n- Just to clarify, In equation 2, does z come from the skill arg encoder? \n- Why was k-means chosen, which requires the number of clusters to be chosen ahead of time, instead of a clustering method that doesn’t require a hand-specified number of clusters like DBSCAN?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In summary, the initial reviews highlight the following strengths and weaknesses of this submission.

Strengths:
- The approach is novel and interesting.
- The paper is well written and clear.

Weaknesses:
- It is unclear how the method can distinguish skills which are defined by dynamic states.
- Generalization to variations of the environment unseen during training is unclear.
- Additional ablations, for instance, regarding the design choice for the VLM or conditioning on language are missing.
- Missing experimental comparison with baselines (e.g., xskills/UVD).
- The paper should evaluate on real robots or discuss how the approach could be scaled to real robots.

The author response and subsequent discussion addressed several of the reviewers' concerns including real robot experiments. Subsequently, all reviewers voted for accepting the paper. One reviewer asked for evaluation of alternative hierarchical RL methods on the extracted skill segments, which has not been addressed properly in the rebuttal and is recommended to be included in the final version of the manuscript.
Decision: Accept