Semantic Temporal Abstraction via Vision-Language Model Guidance for Efficient Reinforcement Learning
Authors: Tian-Shuo Liu
Xu-Hui Liu
Ruifeng Chen
Lixuan Jin
Pengyuan Wang
Zhilong Zhang
Yang Yu
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning; Vision-Language Models; Temporal Abstraction
Primary area: reinforcement learning

Abstract: Extracting temporally extended skills can significantly improve the efficiency of reinforcement learning (RL) by breaking down complex decision-making problems with sparse rewards into simpler subtasks and enabling more effective credit assignment. However, existing abstraction methods either discover skills in an unsupervised manner, which often lacks semantic information and leads to erroneous or scattered skill extraction results, or require substantial human intervention. In this work, we propose to leverage the extensive knowledge in pretrained Vision-Language Models (VLMs) to progressively guide the latent space after vector quantization to be more semantically meaningful through relabeling each skill. This approach, termed **V**ision-l**an**guage model guided **T**emporal **A**bstraction (**VanTA**), facilitates the discovery of more interpretable and task-relevant temporal segmentations from offline data without the need for extensive manual intervention or heuristics. By leveraging the rich information in VLMs, our method can significantly outperform existing offline RL approaches that depend only on limited training data. From a theory perspective, we demonstrate that stronger internal sequential correlations within each sub-task, induced by VanTA, effectively reduces suboptimality in policy learning. We validate the effectiveness of our approach through extensive experiments on diverse environments, including Franka Kitchen, Minigrid, and Crafter. These experiments show that our method surpasses existing approaches in long-horizon offline reinforcement learning scenarios with both proprioceptive and visual observations.

Published as a conference paper at ICLR 2025
SEMANTIC TEMPORAL ABSTRACTION VIA VISION- LANGUAGE MODEL GUIDANCE FOR EFFICIENT REIN- FORCEMENT LEARNING
Tian-Shuo Liu Li Xu-Hui Liu'; Ruifeng 2 Lixuan Jin', Pengyuan Wang* Zhilong Zhang" Yang Y2
NNational Key Laboratory for Novel Software Technology, Nanjing University, China &School of Artificial Intelligence, Nanjing University, China
"Polixir Technologies
edu. cn, jinlxesmail nju. edu. cn, [liuts, liuxh, chenrf) chenrf)@lamdd @lamda nju.edu.
(wangpy, zhangzl] [aanzzl@llmdde @lamda. nju. edu. cn, yuyênju. edu. cn
ABSTRACT
Extracting temporally extended skills can significantly improve the efficiency of reinforcement learning (RL) by breaking down complex Decision-making prob- lems with sparse rewards into simpler subtasks and enabling more effective credit assignment. However, existing abstraction methods either discover skills in an unsupervised manner, which often lacks semantic information and leads to er- roneous scattered skill extraction results, require substantial human inter- vention. In this work, we propose to leverage the extensive knowledge in pre- trained Vision-Language Models (VLMs) to progressively guide the latent space after vector quantization to be more semantically meaningful through relabel- ing each skill. This approach, termed Vision-language model guided Temporal Abstraction (VanTA), facilitates the discovery of more interpretable and task- relevant temporal segmentations from offline data without the need for extensive manual intervention or heuristics. By leveraging the rich information in VLMs, our method can significantly outperform existing offline RL approaches that de- pend only on limited training data. From a theory perspective, we demonstrate that stronger internal sequential correlations within each sub-task, induced by VanTA, effectively reduces suboptimality in policy learning. We validate the effectiveness of our approach through extensive experiments on diverse environments, including Franka Kitchen, Minigrid, and Crafter. These experiments show that our method surpasses existing approaches Long-horizon offline reinforcement learning sce- narios with both proprioceptive and visual observations.
INTRODUCTION
Reinforcement learning (RI.) (Sutton & Barto 2018 has demonstrated remarkable success across variety of domains, including robotics (Peng et al. 2020 Haarnoja et al. 2018), 1 games (Silver et al.) 2016 Zha et al., al. 2021), and combinatorial optimization (Geng et al. [024 Wang et al. 2024). How- in complex environments with sparse rewards, learning efficiently over long horizons remains challenging. Temporal abstraction (Sutton et al. 1999 Villecroze et al. 2022 Park et al. 2023 Fu et al. 2024) provides solution by breaking down complex, long-temm term problems into simpler sub- tasks, allowing agents to learn hierarchically, where credit assignment can be more easily managed the segment level. These methods seek to extract temporally extended primitive structures from tasks into skills for further learning, while in real-world scenarios with logged data, this falls under the scope of offline RL (Ajay al. 2021] Venkatraman et al. 2024). The agent learns to behave within the constraints of the skill context, which reduces the action search space.
"Equal contribution.
Corresponning author.

---

Published as a conference paper at ICLR 2025
Despite previous efforts in hierarchical goal-conditioned RL focusing skill extraction for downstream policy learning, creating useful and effective abstractions remains challenging. Some methods apply unsupervised methods to discover skills (Ajay et al.) 2021] Venkatraman et 2024), which often result in fragmented segments. While heuristic approaches, such as using fixed horizons to define subgoals (Emmons et al.) 2022, Ghosh et al. 202T Yang et al. (022), can lead to unmean- ingful divisions. In long-horizon tasks, particularly in offline RL, misleading extraction of primitive options can result in poor value function learning and suboptimal policy learning. To address this issue, another line of research incorporates human annotations or language descriptions into skill extraction (Lee et al., 2019 Fu et al. 2024 PPng et al. (024). but these methods are heavily reliant on human supervision, which contradicts the pursuit of fully automated intelligence.
Humans inherently excel at efficient learning by instinctively decomposing complex tasks into smaller, manageable components. Drawing inspiration from this phenomenon, we aim to extract interpretable segments that emulate this task decomposition process, focusing on identifying mean- ingful contexts that denote subtasks. These contexts serve as temporally extended signals, represent- ing the current subtask or the next subgoal, and condition the learning of skills (low-level contextual policies) for effective task execution. How can we discover more reasonable context and build skills without relying on cumbersome human labor? One promising approach to addressing this challenge the integration of reinforcement learning agents with the prior knowledge and reasoning capabil- ities of pretrained foundation models, such as Vision-Language Models (VLMs). These models, trained on large-scale Internet data, offer rich interpretations of the environment and semantic un- derstanding, allowing agents to reason in a more meaningful and effective manner (Du et al. 2024 Mees al. 2023).
Inspired by the success of VLM, we propose a novel approach that leverages the prior knowledge of VLMs decompose trajectories into reasonable subsequences, termed Vision-Language Model Guided Temporal Abstraction (VanTA). Our method progressively shapes the latent space after vec- tor quantization to be more semantically meaningful by relabeling each segment. This process facil- itates the discovery of more interpretable, task-relevant temporal segmentations, all while reducing the reliance on extensive manual intervention. Specifically, the latent space undergoes alternating updates in two stages: first, it commits to the vector-quantized results; then, after VLM assigns indices to the segments, the latent space is further refined. We employ a two-level policy extrac- tion framework, where the high-level policy selects from a compact, discrete primitive skill space, and Implicit Q-Learning (IQL) (Kostrikov et al.] (022) is used for low-level policy training. From theoretical standpoint, demonstrate that our method, in hierarchical manner, compresses the raw MDP action space and reduces the horizon. Compared to the primitive skill space with- out VLM guidance, VanTA reduces the complexity of low-level policy space, allowing the agent to address subtasks more efficiently in expectation. We empirically validate our approach through extensive experiments on environments including Franka Kitchen in D4RL (Fu et al., 2020), Mini- grid (Chevalier- Boisvert et al. 2023). and Crafter (Hafner (022I). These experiments show that our method outperforms previous approaches long-horizon offline reinforcement learning scenarios for both proprioceptive and visual observations.
2 RELATED WORK
2.1 TEMPORAL ABSTRACTION IN HIERARCIICAL REINFORCEMENT LEARNING
Temporal abstraction is key topic in hierarchical reinforcement learning. Many previous methods rely on unsupervised objectives to discover skills (Pertsch et al. 2020 Eysenbach et al. 2019 Singh ct al. 2021). which can lead to scattered and unmeaningful skill extraction. To achieve more co- herent skill discovery, some approaches incorporate human intervention (Lin et al. 2024 Oh et al.) 2017 Xu et al. 2018). Then these skills are used for planning (Sharma et al.) 2020). fho-shot imita- tion learning (Nam et al. 2022), or online RL (Nachum et al. 2019). In contrast, our work leverages VLM-guided skill extraction to enhance the limited available data, reducing the need for extensive human intervention. While temporally extended and recurring structures have been effective in solv- ing complex tasks in online RL, integrating them into offline RL is also beneficial (Dietterich) [998] Sutton et al.. 1999 Kulkarni et al. 2016). The main challenge lies in how to hierarchically decom- pose trajectories (Ajay et al. 2021] 122 Jiang et al.) 20033 Pertsch et al. 2020). Our method focuses on extracting more meaningful skill context from offline data via VLM guidance.
2

---

Published as a conference paper at ICLR 2025
2.2 KNOWLEDGE DISTILLATION FROM VLM FOR REINFORCEMENT LEARNING
Vision-language models (VLMs) possess extensive world knowledge. The efficiency of reinforce- ment learning is crucial (Chen et al. 2024| Wang et al. 2022), and leveraging knowledge in VLMs enhance it. VLMs used to decompose high-level long-see Long-sequence tasks into multiple low-level
exccutable step-by-step plans (Huang et al. 2022 Wang et al. 2023 Brohan et al. 2023). These plans then serve as conditions for downstream reinforcement learning tasks to guide RL learning (Huang et al. 2023). However, due to the limited reasoning abilities of VLMs, they struggle to accu- rately recognize dynamic environments and reason precisely. On the other hand, VLMs are also used
2023: Myers et al.) 2023). feedback (Bucker et al. 2023) and data for world modeling (Lin al.
to embed useful information such as instructions (Liu et al. [022 Mees et al.
(023). These ap-
proaches leverage (V)LMs to encode the semantic information of input text and images. Compared
to traditional knowledge distillation methods, our approach uses VLMs as discriminative model to
assist in segmentation, which is more reliable than relying on their generation abilities (Pang et al.)
2024).
PRELIMINARIES
Let M = PS,A,, po) represent a Markov Decision Process (MDP), where is the state space, is the action space, P. S x A - A(S) is the transition function (A(-) is the probability simplex), r : S x A - [0, Rmax] is the reward function, ? E [0. 1) is the discount faction and Po is the initial distribution
S + A(A) describes a distribution actions for each state. The goal of RL is to learn the best policy x that maximizes cumulative discounted reward, i.e., L, Eai "rrs, 0z). The value function and Q function of policy are (T((s = Ee B(a (arisihhhrrsss (1" ae)[so = s), Q* = E, Ea
states. A policy
a [s s, 4o = a]. V and Q" be the shorthand for V* and Q" respectively. We assume access to experience dataset D = (Ti == (st, dz, St+ 1
Vector Quantized Variational Autoencoder (VQ-VAB) (yan den Oord al. 2017) is neural net- work architecture designed for unsupervised learning of latent representations. In VQ-VAF, the input data is mapped to continuous latent space by the encoder 9s- which is then quantized to the nearest codeword z in the discrete codebook embedding space Z. This codeword, cj. is indexed in the codebook C, where the size of the codebook set denoted [C'. The decoder P& maps the discrete code back to the output space, generating new samples. VQ-VAE updates the encoder, decoder, and codebook parameters with the loss function Eq. [1 respectively reconstruction loss, codebook loss and commitment loss:
L EEND || a1l2 [|sg(h) 21/ + B1 sg (z) 11
where & ~ 22) h (- and sg(.) represents the stop-gradient operation. Additionally, we follow the technique from Gumbsch et al. (2024). which models sequential transitions in latent space by introducing L Z--regularization context changes Ah, |l -hel1, the context code is intended to change relatively sparsely.
4 METHOD
In this section, we introduce VanTA, novel framework for extracting skills from offline trajectory datasets D, grounded in pretrained vision-language models (VLMs). We incorporate VLM into the skill context codebook 2 update process, which distills states into a discrete space no (=)=))). Then conditioned on the extracted skill, low-level policy further learned by offline RI.
Our framework, as illustrated in Fig. consists of two stages. In the first stage, state sequences are initially segmented using vector quantization (VQ) techniques to generate the initial set skills. These skills are then passed to the VLM, which relabels them based on its internal knowledge. The relabeled skills are subsequently used to update the VQ codebook following the method described in van den Oord al. (017)). This process is repeated iteratively until convergence. In the second stage, the low-level policy is refined using the skills derived from the first stage. The following sections provide detailed insights into the implementation of our approach.
3

---

Published as a conference paper at ICLR 2025
Stage
Primitive Skill Extraction
Offline Data
Tnitial
Skill Label
de (M)) Emurder
Every I epoch
Pretrained VLM
Codebook
00
Skill Relabel
Policy Decnder
Pal (a(a(6.2)
Skill
Skill More
Skill Slide the
Turm
Stage 2 Policy Extraction-
Observation
Low-level Policy
Action
$1, S2. S3
High-level Policy (z|s)
T (als,))
a2 0g
Vision- laiiga- model guided Temporal Abstraction (VanTA) Overview: First stage for primitive skill extraction, second stage for policy extraction, conditioned on the extracted skill.
Figure I:
4.1 SKILL CONTEXT LEARNING
To better distill the skill context, we leverage the knowledge in the foundation model VLM. VQ provides a standard codebook 2, but directly querying each image may lead to hallucinations due to the lack clear meaning in a single observation. Additionally, this approach inefficient when han- dling large offline datasets. Therefore, we start with the VQ method to initialize trajectory segments. The input for VQ is single observation. By projecting consecutive observations into a single skill context, segments are formed, which serve proper initialization for querying the VLM. We pro- pose a modification to the original VQ- VAE architecture, to learn text-aligned temporal abstractions of trajectories for decision-making policies. The original decoder is replaced with policy decoder Po (a|8, 2). as temporal abstractions focus on behavior information for policy learning rather than sharing the reconstruction objective.
As the encoder qu (h]s) embeds the state into the continuous latent space, it is then projected into discrete code vector zy. where the index j corresponds codebook Gj in C. Consecutive states signed to the same code index naturally segment the trajectory into sub- sebb-rraeectriiss where the initial and terminal states of each sub-trajectory define the boundaries of a primitive skill. To ground the codebook embedding space with semantic meaning, we query the VLM as follows: j = VLM(3), where ş represents the initial and terminal states of the primitive skill. The returned j is the iden- tified index. We adopt the update rule from |van den Oord et al. (2017): [oy et al. (2018); Razavi] iet al. (2019) to update the ej embedding žj. applying an exponential moving average with smaller decay rate after querying the VLM:
- A
Σ
h1s1 -),,
where Eny represents all projected embeddings assigned to index by the VLM. We denote the EMA target of (En, by h1 (1- A). En, h2(1 - (1-A)....) as e, where j ranges from to (C] in Eq.
Scattered or unmeaningful skills are common during unsupervised learning. Leveraging knowledge in the VLM, primitive skills process gradually makes the segmentation more semantically meaningful. Exception handling for VLM crucial, as the output of VLM can be stochastie, often providing misleading answers when
be merged, and misleading indices corrected. This alternation
4

---

Published as a conference paper at ICLR 2025
an image only partially depicts the subtask. We thereby add option for VLM to choose as no suitable skill. During optimization, we increase the smoothness loss to improve consistency with surrounding states, incorporating scattered observations into neighboring skills in subsequent iterations. The final loss for training encoder Tι, decoder P is defined as:
E [lla all2 + 3 B|| sg(())B + ||= sg (h) + || - el2 - A]]]]]] 6,00
(3)
where sg(.) represents the stop-gradient operation, and Ah denotes the smoothness loss, which ensures representation consistency between consecutive states, following the approach in Gumbsch et al. (2024). And Z is the latent variable discretized by vector quantization in this loss function.
4.2 LOW-LEVEL POLICY EXTRACTION
We collect replay buffer B from the dataset D, consisting of segmented sub uub-trajectories, and store transition tuples {a, = (St, at, Zz. Tt:ttKKKi -1,, $t+ CC ] from Ø; ~ B. Here, st and ae repre- sent the state-action pairs from D, 2t the latent variable labeled by our primitive skill extraction method, T-+K-1 denotes the 7-discounted sum of rewards accumulated over K - time steps start- ing from time step in og and S represents the initial state of the next skill. We first update the high-level Q-function, which is conservatively learned using temporal-difference error at the skill level. A batch primitive transitions is sampled from B for maximizing the cumulative return of the primitive skills. Since the latent space in the codebook is quantized, apply discrete Deep Q-learning under offline setting in conservative manner (Kumar et al. (020). The Bellman error for the Q-update in high-level policy learning can be expressed as:
( 21) & K-14K-1 + 7 Q( arg max Q((+kK,2)), z=mo(=lse+k)
Once the high-level policy is applied to generate candidate primitive skills during evaluation, the final policy for VanTA generates low-level actions conditioned on both the primitive skill and the observation. The low-level policy is learned using a behavior cloning-style algorithm, where the policy network approximates the action logits from the offline dataset:
min J(w; ETTD
log no (arlst, 24)
(5)
where = is the latent space label assigned the state. Additionally, apply importance weighting during the update, using exp(Q(s, V(s)) (Kostrikov et al. 2022) with a certain coefficient.
5 ANALYSIS
Intuitively, more semantically meaningful skills can lead to better policies. To theoretically validate this intuition, we analyze the relationship between algorithm performance and the learned skills. We compare VanTA with reinforcement learning in the original space or hierarchical reinforcement learning methods that lack external semantic information guidance in this section. Our algorithm is divided into two parts: offline reinforcement learning for the high-level policy and weighted behavior cloning for the low-level policy. First, we focus on the offline reinforcement learning part. In the offline setting, the performance of the policy is highly related to the dataset. Therefore, we introduce an assumption on the quality of the dataset as previous works did (Chen & Jiang 2019 Liu et al. 2024). We also give the definition of admissible distributions.
Definition 5.1 (Admissible distributions). We say a distribution E A(S x A) is admissible in MDP M, if there exists t 2 and policy æ such that v(s. a) Pr(s; s, ag also Po. T).
In other words, admissible distributions distributions that can be generated by some policy under the given MDP.
Assumption 5.2 (Concentratabiliity coefficient). Given the dataset D, the distribution of the dataset be u(s, a). We assume that there exists C < 0o s.t. for any admissible v, p(s.a) S C,
v(s.a)
((s,a) S A.
5

---

Published as a conference paper at ICLR 2025
The assumption means the distribution of the offline dataset should be able to include all the state- action pairs. Chen & Jiang (2019) gives the sample complexity of performing fitted Q iteration under this assumption.
Theorem 5.3 (Theorem 2 of Chen & Jiang (2019)). Given a dataset D with sumple size |D] =
and F satisfies completeness, i.e., V/ E F. TI E F, w.p. 2 5, the output policy of Fitted Q
Colog 1z
Iteration after k iterations, Th, satisfies o* T I <Vnux when k- - oc and = =0 (((1-))
We assume the size of the function class [F] is influenced by the dimensions of the state space and action space, i.e, [F] = [A))., This assumption is reasonable because the complexity of function approximation is closely tied to the number of possible state-action pairs. Specifically, in a tabular setting, the size of the function class can be represented as: |F] ([S]| where [S] the cardinality of the state space, |AA] is the cardinality of the action space, and R represents the range of possible values that the Q-function can take. If we do Q iteration without establishing the hierarchical structure, the sample complexity of the learning process is indicated by Thm. 5.3
In our setting, the high-level policy outputs not the actions but the skills, the low-level policy learns with weighted BC conditioned on the skill output by the high-level policy. To facilitate our analysis, we assume the length of skills is the same, denoted as K. The learning process of the high-level policy similar to fitted Q iteration except the action space replaced with the skill space and the horizon changes correspondingly. Therefore, we get the sample complexity of high-level policy directly from Thm. 5.3
Corollary 5.4. Given a dataset D with sample size |D| = and satisfies completeness, w.p. 2 1 8, the output policy of FQI after k iterations, The satisfies 8 - S cVmax when k - co and n = O (
( Clog 2111)121
2
where 6* is the optimal policy conditioned on the tow-level policy.
Note that or in Thm. 5.3 is replaced with 8" this is because the quality of the skills is also an influence factor of the performance. Then we need to determine the difference between 6* and e².
Note that the low-level policy is trained by weighted behavior cloning, the optimal policy of weighted behavior cloning is Kostrikov et al. (022)): #* (als) μ(als) exp (S(((s,a)
(s))) ((()))
We denote the value induced by the optimal policy as i" Therefore, o² - 0* can be decomposed as e² 8- 8- 8* Let A(3, Q, V) denote v² - U-" this is because this term is only related to 3, Q and V. If = Q. V- = V*, and B + then ō" + v". The last term we need to determine is 0" 0* We give the result here, and the proof deferred to Appx. A
Lemma 5.5. For any # II, weighted BC algorithm ensures that with probability at least 1 8,
8- - S 0(1) V Q (3)
H.
0 log (1((1118-1) + +R(log(()) (O(Rllo)))))))) (1 H log (1)8
K
K
where G is the value function induced by the learnt policy â. n is the number of samples in the dataset, var = Var exp(B(Q(s.a) V((M)) 7(s) is the normalization factor, and
2(a)
Σ E [(oT (xt,) QF' (se, (o)...)))]
K
Note that semantic skill implies temporal correlations of action sequences, which means the policy space of a semantic skill is not the same as the original policy space. For case of discussion, we can make the following assumptions:
1. Original policy space (no temporal correlation assumptions): In the original policy space, there is no temporal correlation between action sequences. The agent can freely choose each action without considering the relationships between actions over time. This space is denoted as Io
2. Semantic skill policy space (with temporal correlation): Introducing temporal correlation means that actions at the current time step are influenced not only by the current state but also by previous actions. This will reduce the size of the strategy space, not all action sequences valid.

---

Published as a conference paper at ICLR 2025
Suppose the autocorrelation coefficient of action sequences generated by the policy belongs to the new policy space is larger than a, then the new space can be denoted as IIa = { Ion (o()) > a), where p(tt) is the autocorrelation coefficient of the action sequence generated by policy
Combine the results together, we have
Theorem 5.6. Given dataset with sample size n, under the hierarchical learning process, with probability as least 1 1 6,
Clog (1(1))1211)
H
log( II 1
0¹ - 1T 50
Vmax + +0(1) V (v)
)4
K
71
H [og((1a|80] AB,
+ *((log(())
(3) K
Compare Thm.
5.6 with Thm. 5.3 the hierarchical structure reduces the size of Q function class
from g(SS), to ((S) 2), and the scale induced by horizon is reduced from (1 10--) 7)* 9 to (1
yK However, it also introduces three other terms. Generally, A (5, Q V is not large, as the weighted BC algorithm has already achieved good performance (Nair al. 2020 Zhao al. 2022]
Kostrikov et al. 2022) and (53, Q, V) is even smaller because there no BC loss and the high-level policy is assumed as optimal. If the hierarchical structure of the problem is good, for example, the action sequence of skill has high temporal correlation and the middle two terms are also small, then the performance bound of hierarchical methods is better than the original one.
Furthermore, Thm. 5.6 provides us with an approach to determine the performance of different hierarchical methods. If the action sequence of the learned skill has high temporal correlation, [Ta can much smaller, then the performance bound is much better. Intuitively, the skills with semantic information than previous methods. We test the temporal correlation in the experiment Sec. 6.5
temporal correlated, which explains why method can be better
Table 1: Performance on Franka Kitchen, MiniGrid and Crafter environments, averaged over random seeds. The best results are bolded and the second- best results are underlined.
Task Name
BC
CQL
IQL
RyS
GCSL
WGCSL
LDCQ
VanTA (Ours)
kitchen-complete-vi)
65.0
43.8
62.5
50.2
58.6
57.7
52.8
69.2:18.5
kitchen parttn---aiillv
38.0
50.1
46.3
60.3
55.0
59.4
67.8
71.2::5.7
kitchen-mixed-vo
51.5
52.4
51.0
51.4
56.2
49.6
62.3
68.5:14.44
MiniGrid- MmiirridDDorKee--oooe DoorKey-6x6-comppeee
0.92
0.67
0.91
0.89
0.82
0.85
0.86
0.92:10.0
MiniGrid- KennKey---66ooo 6x6-mixed-vV
0.67
0.80
MiniGrid-Doote DoorKey-Sx8-ono cx8-complete-v0
0.88
0.43
0.89
0.94
0.87
0.76
0.70
0.92
MiniGrid- MiniGrid-DoorKa DoorKey-8aa 8x8 mixed-v0
0.43
0.30
0.47
0.32
0.39
0.44
0.38
0.51::0..15
MiniGrid-Ke KeyCorridoSSRRR-oopeeee..
0.47
0.62
0.72
0.64
0.57
0.69
0.81:10.06
MiniGrid- Coy
0.09
0.23
0.51
0.42
0.23
0.38
0.61:10.12
0.80
0.51
0.90
0.78
0.73
0.61
0.64
0,85 11
MiniGrid- Ceerrrdd-ReeBllnn BeddBHlueDDoor--66--meeeeee
0.64
0.47
0.69
0.42
0.72
0.51
0.44
0.220
Crafter-partee
2.69
1.73
2.75
2.11
0.96
5.46
EXPERIMENTS
In this section, we empirically validate the effectiveness of VanTA in providing a hierarchical structure that improves agent learning performance. Sec. evaluates the performance of offline RL methods against hierarchical goal-conditioned approaches on tasks with hierarchical sub-tasks. Sec. 6.2 explains VanTA's superior performance by showcasing how it segments trajectories into interpretable skills, outperforming traditional VQ methods. Sec. reiifforess this with an ablation study, comparing codebooks with and without VLM guidance, showing that VLM guidance en- hances learning, while its absence hinders performance. Sec. 66.4 demonstrates VanTA's strength in data-scarce scenarios, leveraging pretrained VLM knowledge to augment datasets and improve re- sults. Finally, Sec. 6.5. verifies For experimental benchmarks, we conduct our experiments on three long-horize horizon domains: Franka Kitchen (Gupta et al. 2019) for manipulation tasks, MiniGrid
theory and demonstrate the higher temporal correlation of Van
-Boisver al. 2023)

---

Published as a conference paper at ICLR 2025
Figure 2: Skill extraction results without VLM guidance; no clear semantic meaning in each skill.
Figure 3: Skill extraction results using VanTA: sequentially 1) open the mierowave, 2) move the kettle, 3) rotate the knob, and 4) open the cabinet.
for navigation, and Crafter (Hafner 2021). an open-wordd survival game. Further implementation details, including model architectures and hyperparameter settings, provided in Appx. B.5.4 We first introduce the environment setup.
This environment is initially introduced in Gupta et al. 2019 In exper- iments, we use offline data from the Fuu et al.] (2020) benchmark, which includes three types of datasets collected from human demonstrations, with visual observations rendered using OpenAl's Gym framework rendering functions (Brockman et al. 2016) as input for VLMs.
Franka Kitchen:
al. (2023) is lightweight grid-world environment suite with Ope- nAI Gym interfaces. All tasks fall within the grid navigation domain, where the agent starts from an initial position, picks up an object, and navigates the goal destination with sparse rewards.
MiniGrid: Snevalier-Boisveed
Crafter: (Hafner 2021 is a 2D open-world survival game benchmark with visual inputs that evalu- ates wide range general abilities within a single environment. We collect data using intermediate model training checkpoints from online reinforcement learning with PPO (Schulman et al. 2017).
6.1 EVALUATION ON BENCHMARKS
We empirically evaluate our two-level policy extraction method on diverse tasks, with all of them providing sparse reward feedback. Prior approaches include typical offline reinforcement learn- ing or imitation learning methods including Behavior Cloning (BC), Conservative Q-Learning (CQL) (Kumar et al.) 2020), and Implicit Q-Learning (IQL) (Kostrikov et al. (022), along with goal-conditioned variants RvS (Emmons et al.) (022)) GCSL (Ghosh et al. 2021 I) WGCSL (Yang) et al. 2022) and LDCQ /Venkatraman et al. (2024), which use goal-setting methods like hindsight goal relabeling or prior-shaped latent space. As shown in Fig. [ VanTA outperforms the baselines in most tasks, which demonstrates that the extracted primitive skills aid in policy learning. Since Crafter is open-world, iision-based environment, manually setting desirable observation the
8

---

Published as a conference paper at ICLR 2025
goal in GCSL and WGCSL is infeasible. Investigating the Kitchen tasks, when performing the final subtask of opening the cabinet, the agent selects actions from higher-level, discrete and internally- compact skill space, where the robot arm has already been raised, rather than searching through the full original action space. In Crafter, which involves more subtasks, the extracted skills motivate the agent explore wider range of tasks, demonstrated in Tab. A similar conclusion can be drawn in the KeyCorridor environment, which is comparatively more complex in MiniGrid.
6.2 VISUALIZATION OF SEGMENTATION RESULT
In this section, we compare the segmentation results with the vanilla VQ method demonstrate VanTA's superiority in interpretable skill extraction. In the kitchen-comee tichen-commplee-vi task, the agent sequentially opens the microwave door, moves the kettle, turns on the light, and finally opens the cabinet. The skill assignment results, shown in Fig. 3 display the first two observations and the terminal observation for each skill, highlighting how VanTA reduces the incoherence primitive skills. In contrast, the vanilla VQ method may abruptly shift to new skill in successive time steps, as seen during the microwave door opening in Fig. 2 Intuitively, although VQ occasionally provides imperfect initialization. VLM intervention quickly corrects these erroneous assignments and merges meaningful skills. Additional segmentation visualizations can be found in Appx. B.5.1
6.3 ABLATION
VanTA v.s. VanTA w/o VLM guidance We next evaluate whether VLM guidance sigmifi- cantly improves learning in hierarchical mod-
Ablation Study on Franka Kitchen
els. To test this, we replace the indices as- signed to primitive skills with those from the
Normalized Retur
60
VanTA without VLM guidance method, As
shown in Fig. 4 in the Franka Kitchen en-
vironment, VLM-guided skills are conducive to policy learning, demonstrating superior- ity across all environments. When compared with the performance of offline reinforcement learning baselines shown in Tab. I the perfor- mance of using non-V non-VI.M-pee M-guided extracted skill contexts supplement state information for policy learning even shows drop. This suggests that an erroneous broken code- book introduces challenges in two-level pol- icy learning, especially making it more diffi- cult to learn primitive skills. Besides, using irrational primitive skills to augment the state also may not be beneficial. Additional ablation results available in Appx B.5.3
30
kachen-maaddo
Environment
Figure 4: Comparison of normalized return between VanTA and VanTA without VLM guidance in the Franka Kitchen environment.
6.4 EXPERIMENTS IN LOW-DATA REGIME
Leveraging the rich knowledge pretrained VLMs enhances the efficiency of reinforcement learn- ing, is supplement the limited available data. We further investigate whether VanTA enables more efficient learning with limited data using varying data ratios: 100%, 50%, and 10%. As shown in Tab. 2 VAnTA maintains its performance advantage and degrades relatively more slowly as the available data decreases. Notably, on the sitchen-complede smallest dataset in our experiment, VanTA achieves 28 6% improvement, highlighting its ability to boost the efficiency of offline reinforcement learning.
task with only 10% of the data, the
6.5 REDUCTION IN COMPLEXITY
As discussed in Sec. | the performance of the algorithm is influenced by the size of the policy space it identifies. This size is largely determined by temporal correlations in the data. Therefore, in this section, we calculate the temporal correlation to assess the identified policy space. We assume that the original policy space is derived from the original dataset. To analyze this, randomly sample

---

the autocor- Published as a conference paper at ICLR 2025
Table 2: Performance with reduced offline data usage in Kitchen, averaged over random seeds.
Task Name
BC
CQL
IQL
RvS
GCSL
WGCSL
LDCQ
VanTA (Ours)
kitchen-com complee-compeee--O (100%)
65.0
43.8
62.5
50.2
58.6
57.7
52.8
69.2:
Kitchen-complete-vv complete-v (50%)
23.6
18.6
35,4
36,0
28.3
29.5
32.5
44.2:16.2
Miichen-complete-OO/ (10%)
7.6
2.7
13.3
5.0
7.3
9.1
12.5
17.117.77
Sitchen-partiale (100%)
38.0
50.1
46.3
60.3
55.0
59.4
67.8
71,2++5.7
kitchen-partie partona--ttillOOO (50%)
27.1
12.8
39.7
15.6
12.9
25.8
17.7
42.8+
kitchen-paree Saahhen-prrtil--O0 (10%)
19.1
9.2
33.2
19.5
14.2
36.9
21.3
37,5++3.
kitchen-mixed--O (100%)
51.5
52.4
51.0
51.4
56,2
49.6
62.3
68.5
kitchen-mixed-v0 (50%)
30.4
14.4
25.5
20.8
16.9
31.4
21.2
4,2
kiichen-mixed--O (10%)
21.2
8.7
23.4
20.0
15.2
27.8
11.4
29.9+3.0
fractions of the trajectories from the dataset and calculate their temporal correlation. This allows obtain the distribution of temporal correlations within the original policy space. For various hierarchical algorithms, including VanTTA and VanTA without VLM guidance, we similarly calculate the temporal correlations of the identified skills. By comparing the range of temporal correlations from these algorithms with the original distribution, we can estimate the ratio between the size of the policy space identified by each algorithm and that of the original policy space.
To approximate the distribution of the temporal correlation of the original policy space, we sample 1000 fractions of trajectories and use Kernel Density Estimation (KDE). We also use 100 samples to approximate the range of temporal correlation of low-level policies. The temporal correlation is measured by the autocorrelation coefficient. For an action sequence {a1, ..y...,,.k] a2,...,
rk (E a)(a -())===aa)) -)) E (at a)2. In the experiment, we set k 1, with the results presented in Tab. VanTA achieves a higher autocor- relation coefficient than the non-V non-VLM-a VLM-guidee version across all three environments. On average, the policy space is reduced to 64% without VLM guidance and further to 54% with VL.M guidance. This reduction in policy space complexity explains VanTA''s superior performance and supports the theory discussed in Sec.
relation coefficient is defined
Table 3: The comparison of the policy spaces of the low-level policy learned using VanTA without VLM guidance alone and those learned using VanTA.
VanTA w/o VLM
VanTA
Task Name
Ratio
F1
Ratio
kitchen-complete-wi
0.42
60%
0.69
41%
kitchen-partial-v0
0.44
80%
0.59
69%
biccen-miixd-vee
0.67
53%
0.68
52%
CONCLUSION
Discussion. We introduce VanTA, method for extracting discrete, task-relevant and semantic skills from offline data with the guidance of pretrained Vision-Language Models (VLMs), which improves learning efficiency. This grounding of the codebook in semantic knowledge facilitates subsequent offline RL. In future work, we primarily aim to explore temporal abstraction with VLM for offline preference learning (Zhang et al. 2024) and cooperative multi-agent systems (Yuan et al. 2023). Our initialization technique has significantly reduced query complexity, which aim to reduce further in future work. However, VanTT has certain limitation. In our experiments, proper initialization is essential for effective VLM guidance; without it, significant query complexity can arise, making it difficult to extract interpretable meaning from single observation. Additionally, if mode collapse occurs, common issue in unsupervised learning (Srivastava al. 2017). VLM unable to rectify these errors. Therefore, in future work, we aim to explore simpler yet more effective initialization methods.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
In this study, to ensure the reproducibility of our approach, we provide key information from our submission as follows.
Training Algorithm. We provide the architecture Sec. and pseudo-code of our ap- proach in Appx.
2. Source Code. We have submitted the source code of our approach in the supplementary materials. For BC, IQL (Kostrikov et al., 2022), CQL (Kumar al., 2020) in Tab. [
apply the code from httpps//githu..com/yiiii
124/OfflineL-KKii For the other base-
lines in Sec. 6 we follow the official code and build upon them. We use the code from Emmons et al. (2022) for RvS. available at
For GCSL (Ghosh et 2021) and WGCSL (Yang al., 2022), we modify upon https::/gitthbb.c htt.gg.ccoom///oo// YungRui2015/A /0000/////WSSLLLS For LDCQ (Venkatra- man al. 2024), we build upon the official code, available at httpss//gihuu..oo//////////
learning- Bouning-via-supervve wwwppeeeeee-eaamiiiii
3. Experimental Details. We list the detailed experiment settings, computational resources and hyperparameters in Appx.
4. Theoretical Proofs. We provide the missing proof's in Appx.
AUTHOR CONTRIBUTIONS
Y. Y: led the direction, oversaw the project;
T.-S. L., X.-H. L., R. C.: designed the method;
X.-H. L.: proved the theory;
T.-S. L., P. W.: implemented the codes;
T.S.. L.: designed the experiments;
L. J.: conducted experiments;
T.S.. L.: analyzed the results;
T.SS. L., R. C.: wrote the paper;
P. W., Z. Z.: revised the paper;
T.-S. L.:: discussed and wrote the response:
ACKNOWLEDGEMENTE
This work was supported by the NSFC (62495093) and Jiangsu SF (BK20243039).
REFERENCES
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: offline primitive discovery for accelerating offline reinforcement learning. In Proceedings of the 9th International Conference on Learning Representations (ICLR'21), Virtual Event, 2021.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, 155/1666.015444 2016.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can. not as i say: Grounding language in robotic affordances. In Proceedings of the 7th Annual Conference on Robot Learning (CoRL'233 Atlanta, USA, 2023.
Arthur Bucker, Luis Figueredo, Sami Haddadin, Ashish Kapoor, Shuang Ma, Sai Vemprala, and Rogerio Bonatti. Latte: Language trajectory transformer. In 2023 IEEE International Conference Robotics and Automation (ICRA '23), London, UK, 2023.
11

---

Published as a conference paper at ICLR 2025
Jinglin Chen and Nan Jiang. Information-theone considerations in batch reinforcement learn- ing. In Proceedings of the 36th International Conference on Machine Learning (ICML' (CMML Long Beach, USA, 2019.
Ruifeng Chen, Xu-Hui Liu, Tian-Shuo Liu, Shengyi Jiang, Feng Xu, and Yang Yu. Foresight distri- bution adjustment for off- off-cy reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS' 24), pp. 317-325, Auck- land, New Zealand, 2024.
Maxime Chevalier-Boisver, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan K. Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Ad- vances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023 (NeurlPS'23), New Orleans, USA, 2023.
Thomas G. Dietterich. The MAXQ method for hierarchical reinforcement learning. In Jude W. Shavlik (ed.), Proceedings of the 15th International Conference on Machine Learning (ICML'98), Madison, Wisconsin, USA, 1998.
Yilun Du, Sherry Yang. Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tomp- son. Video language planning. In Proceedings of the 12th International Conference on Learning Representations (ICLR 24), Vienna, Austria, 2024.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline RL via supervised learning? In The Tenth International Conference on Learning Representations (ICLR'22) 22). Virtual Event, 2022.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. In Proceedings of the 7th International Conference Learning Representations (ICLR' 19), New Orleans, USA, 2019.
Dylan J. Foster, Adam Block, and Dipendra Misra. Is behavior cloning all you need? understanding horizon in imitation learning. CoRR, abs/2407. 2024.
Haotian Fu, Pratyusha Sharma, Elias Stengel-Eskin, George Konidaris, Nicolas Le Roux, Marc- Alexandre Côté, and Xingdi Yuan. Language-guided skill learning with temporal variational inference. In Proceedings of the 4/st International Conference on Machine Learning (ICML'24), Vienna, Austria, 2024.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219. 2020.
Zijie Geng, Jie Wang, Ziyan Liu, Siyuan Xu, Zhentao Tang, Mingxuan Yuan, Jianye Hao, Yongdong Zhang, and Feng Wu. Reinforcement learning within tree search for fast macro placement. In Forty-first International Conference on Machine Learning (ICML'24) Vienna, Austria, 2024.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysen- bach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In Proceedings of the 9th International Conference on Learning Representations (ICLR'21, Virtual Event, 2021.
Christian Gumbsch, Noor Sajid, Georg Martius, and Martin V. Butz. Learning hierarchical world models with adaptive temporal abstractions from discrete latent dynamics. In The Twelfth Inter- national Conference Learning Representations (ICIR'224) Vienna, Austria, 2024.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving
tasks via imitation and reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the 3rd Annual Conference on Robot Learning (CoRI.'19) Osaka, Japan, 2019.
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning (ICMI) 18), Stockholmsmässan Sweden, 2018.
12

---

Published as a conference paper at ICLR 2025
Danijar Hafner. Benchmarking the spectrum of agent capabilities. CoRR, abs/2109.06780, 2021.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. In Proceedings of the 39th International Conference on Machine Learning (CCML222) Baltimore, USA, 2022.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In Proceedings of the 7th Annual Conference on Robot Learning (CoRL'23), Atlanta,USA, 2023.
Zhengyao Jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefen- stette, and Yuandong Tian. Efficient planning in compact latent action space. In Proceedings of the 11th International Conference on Learning Representations (ICLR 23), Kigali, Rwanda, 2023.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In Proceedings of the 10-th International Conference on Learning Representations (ICLR 22). Virtual Event, 2022.
Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Anmual Conference on Neural Information Processing Systems (NeurIPS' 16), Barcelona, Spain, 2016.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances Neural Information Processing Systems 33 (NeurIPS Virtual Event, 2020.
Youngwoon Lee, Shao-Hua Sun, Sriram Somasundaram, Edward S. Hu, and Joseph Lim. Com- posing complex skills by learning transition policies. In Proceedings of the 7th International Conference on Learning Representations (ICLR') 19), New Orleans, USA, 2019.
Zhixuan Liang, Yao Mu, Hengbo Ma, Masayoshi Tomizuka, Mingyu Ding, and Ping Luo. Skilldif- fuser: Interpretable hierarchical planning via skill abstractions in diffusion based task execution. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'24), Seattle, USA, 2024.
Haohong Lin, Radu Corcodel, and Ding Zhao. Generalize by touching: Tactile ensemble skill trans- fer for robotic furniture assembly. In IEEE International Conference Robotics and Automation (ICRA 224,, 2024.
Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to model the world with language, 2023.
Hao Liu, Lisa Lee, Kimin Lee, and Pieter Abbeel. fostlwctionnfollooiing agents with multimodal transformer. CoRR, aab//221..1431, 2022.
Xu-Hui Liu, Tian- Shuo Liu, Shengyi Jiang, Ruifeng Chen, Zhilong Zhang, Xinwei Chen, and Yang Yu. Energy-guides diffusion sampling for offline-to-online reinforcement learning. In Proceed- ings of the 4/st International Conference on Machine Learning (ICML'24), Vienna, Austria, 2024.
Oier Mees, Jessica Borja-Diaz, and Wolfram Burgard. Grounding language with visual affordances over unstructured data. In 2023 IEEE International Conference on Robotics and Automation (ICRA 223, London, UK, 2023.
Vivek Myers, Andre Wang He, Kuan Fang, Homer Rich Walke, Philippe Hansenn-sttruch, Ching-An Cheng, Mihai Jalobeanu, Andrey Kolobov, Anca Dragan, and Sergey Levine. Goal representa- tions for instruction following: A semi-supervised language interface to control. In Proceedings of the 7th Annual Conference on Robot Learning (CoRI'23), Atlanta, 2023.
13

---

Published as a conference paper at ICLR 2025
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. In Proceedings of the 7th International Conference on Learning Representations (ICLR') 19), New Orleans, USA, 2019.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. CoRR, abs/2006. 09359, 2020.
Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, and Joseph J. Lim. Skill-based meta- reinforcement learning. In Proceedings of the 10th International Conference on Learning Repre- sentations (ICLR'22) Virtual Event, 2022.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning (ICML'17), Sydney, Australia, 2017.
Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hur Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self- sefr-immprvement by reinforcement learning contemplation. In Proceedings of the 12th International Conference on Learning Representations (ICLR'24), Vi- enna, Austria, 2024.
Seohong Park. Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. HIQL: offline goal- conditioned RL with latent states as actions. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 (NeurIPS'23), New Orleans, USA, 2023.
Andi Peng, Belinda Li, Ilia Sucholutsky, Nishanth Kumar, Julie Shah, Jacob Andreas, and An- dreea Bobu. Adaptive ranguage-guider abstraction from contrastive explanations. In Proceedings of the 8th Annual Conference on Robot Learning (CoRL'24), Munich, Germany, 2024.
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wai Wei Edward Lec, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. In Proceedings of the Science and Systems (RSS'20), Virtual Event, 2020.
14th Robotics:
Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors. In Jens Kober, Fabio Ramos, and Claire Tomlin (eds.), Proceedings of the 4th Annual Conference on Robot Learning (CoRL'20)), Virtual Event, 2020.
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019 19url Vancouver, Canada, 2019.
Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments on vector quantized autoencoders. CoRR, abs/1805.11066 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CORR, abs/ 07700000637,220 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-awer aware unsupervised discovery of skills. In Proceedings of the 8th International Conference Learning Representations (ICLR 20), Addis Ababa, Ethiopia, 2020.
David Silver. Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nat., 529, 2016.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In Proceedings of the 9th International Conference Learning Representations (ICLR'2/), Virtual Event, 2021.
14

---

Published as a conference paper at ICLR 2025
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, and Charles Sutton. VEE- GAN: reducing mode collapse in gans using implicit variational learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Confer- ence on Neural Information Processing (NeurIPS Long Beach, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 112, 1999.
Aäron van den Oord, Oriol Vinyals. and Koray Kavukcuoglu. Neural discrete representation learn ing. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems (NeurIPS 17), Long Beach, USA, 2017.
Siddarth Venkatraman, Shivesh Khaitan, Ravi Tej Akella, John Dolan, Jeff Schneider, and Glen Berseth. Reasoning with latent diffusion in offline reinforcement learning. In Proceedings of the 12th International Conference on Learning Representations (ICLR'S 24), Vienna, Austria, 2024.
Valentin Villecroze, Harry J. Braviner, Panteha Naderian, Chris Maddison, and Gabriel Loaiza- Ganem. Bayesian nonparametries for offline skill discovery. In Proceedings of the 39th Interna- tional Conference on Machine Learning (ICML'22) Baltimore, USA, 2022.
Zhihai Wang, Jie Wang, Qi Zhou, Bin Li, and Houqiang Li. Sample- reinforcement leamn ing via conservative model-baat based actor-critic. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI'22), 2022.
Zhihai Wang, Jie Wang, Dongsheng Zuo, Yunjie Ji, Xinli Xia, Yuzhe Ma, Jianye Hao, Mingxuan Yuan, Yongdong Zhang, and Feng Wu. A hierarchical adaptive multi-task reinforcement learning framework for multiplier circuit design. In Proceedings of the 41st International Conference on Machine Learning (ICML 224), Vienna, Austria, 2024.
Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma. Yitao Liang, and Team Craft- Jarvis. Deseribe, explain, plan and select: interactive planning with large language models enables open- open-world multi-task agents. In Proceedings of the 37th International Conference on Neural In- formation Processing Systems (NeurIPS:33) 23), New Orleans, USA, 2023.
Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, and Silvio Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA'I8), Brisbane, Australia, 2018
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjic Zhang. Rethinking coadiiontt supervised learning and its connection to offline RL. In The 10th International Conference on Learning Representations (ICLR") 22), Virtual Event, 2022.
Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. A survey of progress on cooperative multi-agent reinforcement learning in open environment. CoRR, abs/2312.01058, 2023.
Daochen Zha, Jingru Xie, Wenye Ma, Sheng Zhang, Xiangru Lian, Xia Hu, and Ji Liu. Douzero: Mastering doudizhu with self-play deep reinforcement learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning (ICML'21), Virtual Event, 2021.
Zhilong Zhang, Yihao Sun, Junyin Ye, Tian-Shuo Liu, Jiaji Zhang, and Yang Yu. Flow to better: Of- fline preference-bant based reinforcement learning via preferred trajectory generation. In The Twelfth International Conference on Learning Representations (ICLR'24) Vienna, Austria. 2024.
Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behavior cloning regularization for stable offineeto-online reinforcement learning. CoRR, abs/2210. 13846, 2022.
15

---

Published as a conference paper at ICLR 2025
A
MISSING PROOFS
We state the proof of Lemma 5.5 here. Foster et al. (0244) provided the performance bound of BC algorithms:
Theorem A.1 (Corollary 3.1 of (Foster et al. 2024
For any xt E IT, BC algorithm ensures that
with probability at least
K log(
O(Rlog(n)) 1og((11(011)
where G is the value function induced by the learnt policy ā, n is the number of samples in the dataset, and
Σ B h=1
[1oT (ot, (f,, QT (sc, ae))
KK
Note that weighted BC does not share the same performance bound as BC. Given the same dataset, weighted BC has a higher variance, which increases the bound Theorem A. Intuitively, ap- plying weighted BC in a given dataset can be seen applying BC in another data distribution at the expense of excluding apply Theorem A. to our setting, we introduce the concept of effective sample size, under which the estimator of the unweighted distribution has the same variance as that of the weighted estimator under the original sample size.
data that does not conform to the new data distribution. Therefore, to
Lemma A.2. Suppose the distribution before adding weight is p. and after adding weight is the effective sample size
Thef
1 Var
p
where is the size of the original dataset.
Proof. 1. Consider an unbiased estimator ji obtained through resampling:
1 N
Here, X; are the observed values, and w; are the weights reflecting the resampling. If all weights wi are equal (e.g., 1), this corresponds simple random sampling. However, with weighted sampling, unequal weights can reduce the "effective" information in the sample.
2. To derive effective sample size, we introduce the variance of the estimator:
1 n
Var Var(())eiighne
Σ w Var(
n
where 2² is the variance of single sample X,.
In the absence of weighting (i.e., we = 1), the variance becomes:
Var unweighted
7
Replace n in unwweighed as Tieff we make the two variances to the same by tuning Nefr i.e.,
1
0²
n
Neff
n
Roff
Σ 1=
w
16

---

Published as a conference paper at ICLR 2025
In other words, effective sample size is the total sample size squared divided by the sum of the squared weights.
olm.) 4. Given that w; p(a:)"
we have:
N
N
Σ
w1=1
Σ
wi
N
We need evaluate the sum of the squared weights:
N
This can related to the variance of the weights. Recall that the variance of set values is defined
Var(w)
N wp
w
where is the mean the weights. Given that the weights are normalized to have a mean of (w = 1), the variance simplifies to:
N
Var(w)
wy
1)²
N
1=1
Expanding this:
(ar(w)=
W
Wy
+
Σ
1
Σ W
2N + +N
Σ 1 N
1=1
Rearranging solve for w
N
/ N(1 + Var(w))
1=1
5. Recall the original formula:
n²
Reff
Σm= w
Substitute m = n11 + Var(w)):
n²
n
nat n11 + Var Var(w))
+ Var(w)
Thus, we obtain:
Nell
g(x) Var
p(z]
17

---

Published as a conference paper at ICLR 2025
Let v(s,a) denote the state-action distribution of the dataset. In setting, p(s.a) u(s,a), (8,,)) v(s, a) exp(B(Q(ss, a) V(s))). Then
Var(p/q) Var (ep((((((ss)) (+)))) (3),
((s)
where Z(s) is the normalization factor. (3) related Q V and B. Use Bett replace n in
Theorem A. we conclude our proof.
B EXPERIMENTAL DETAILS
All experiments are conducted on AMD EPYC 9654 CPUs and NVIDIA RTX 4090 GPUs. GPT- 4o is used as VLMs, while prompt designs are listed in Sec. [B.] The input to the VQ model can be a state representation ((s) or a high-dimensional pixel- input, but for the VLM, rendered observations required.
B. ENVIRONMENT DESCRIPTION
Franka Kitchen
Franka Kitchen (Gupta et al. 2019) offers variety of tasks in simulated kitchen environment, such as turning on the stove, turning on the light, opening microwave doors, and sliding drawers. Each collected trajectory sequentially lists these subtasks, making it well-suited for continuous control studies. The environment simulates realistic physical interactions, allowing the robot arm to perform complex actions like grasping, rotating, and sliding. With tasks naturally broken down into subtasks, it provides an ideal platform for exploring hierarchical reinforcement learning. Additionally, the sparse reward mechanism, where rewards are granted only upon completing specific goals, requires precise credit assignment. Minigrid
MiniGrid (Chevalier- Boisvert et al. 2023) lightweight, fast grid-world environment suite with OpenAl Gym interfaces, specifically designed for reinforcement learning tasks. Each environment consists of 2D grid, where cells may empty or occupied by objects such as walls, keys, doors, goals. In our experiments, we fix both the initial and terminal states, using rendered images to represent the environment state, providing a controlled setting for reinforcement learning studies. Agents required to perform actions like turning left right, moving forward backward, and interacting with objects such as picking up or dropping items to complete tasks efficiently. The reward function is sparse, with non-zero rewards typically given only upon task completion, and penalties applied based on the number of time steps taken.
Crafter
Crafter (Hafner 2021 designs a survival game benchmark to evaluate reinforcement learning agents. Inspired by Minecraft, the game features procedurally generated 2D environments-forests lakes, mountains, and caves-where agents perform survival tasks such as foraging, gathering re- sources, crafting tools, and defending against enemies. These tasks are organized into 22 semanti- cally meaningful achievements, making Crafter an ideal environment for researching semantic skill extraction. Each new achievement unlocked during an episode rewards the agent with +1, making achievements the primary source of rewards in the game.
B.2 MODEL ARCHITECTURE AND HYPERPARAMETERS
For the visual encoder, we adopt the same architecture as described Liang et al. (2024). For the state encoder, we use a linear layer, and for the policy decoder, we employ an MLP network that maps the observation and skill context to the corresponding action. We list the hyperparameters used in VanTA for Kitchen, MiniGrid, and Crafter, respectively.
B.3 PROMPT FOR VLMS
We use GPT-4o as the VLM to extract semantic skills. into two parts: first, provide the initial state, followed by the terminal state with designed
The query and response are separated
18

---

Published as a conference paper at ICLR 2025
g
9
1
Franka Kitchen
Minigrid
Crafter
Figure 5: Rendering of the procedurally-generatee environments in our experiments.
Table 4: Hyperparameters and their values in VanTA.
Hyperparameter
Kitchen
MiniGrid
Crafter
Batch size
256
256
256
Learning rate
le-4
le4
le-4
Optimizer
Adam
Adam
Adam
Iteration interval (/)
50
20
20
VQ option dim
16
16
16
VQ option num
20
6
20
B
3
5
Y
0.1
0.2
0.2
X
0.99
0.99
0.99
Gradient step
3M
1M
2M
prompt. The query format follows user-assistant-user structure. This simulated dialogue signifi- cantly reduces VLM hallucinations in skill identification. However, the environment context differs slightly. For MiniGrid, the listed prompt is tailored to KeyCorridorS3R3 and MiniGrid-D Doorrky.-Dorrky, the task descriptions require minor modifications.
while for MiniGrid-
Prompt Design and Q&A procedure for kitchen
19

---

Published as a conference paper at ICLR 2025
User: Are you familiar with the Franka Kitchen environment used for reinforcement learning? In this environment, various skills represent different subtasks. You can deduce which skill executed identify the agent's subprocess based on the initial and terminal observations. will now show you an image of the initial state. Here are examples of possible subtasks already summarized: 1. No obvious skill 2. Red flame circles appear the stove it turns the gas stove 3. Rotate the knob to turn on the white light above the stove, which is the left one among those knobs 4. Open the cabinet Move the kettle Open the microwave. From my perspective, you can reason what the skill is according to the initial observation and terminal observation. Now will show you the image, it is the initial state. If you think the task happened between two states in the summrrized list, point the index out. Remember maybe many changes happen between two images, choose the most suitable skill name. If not in them, and you still regard it as a new skill, try your best to give only one suitable new name. Each time, you necessary and carefully ensure that it is truly different from the names in the original set. In total, you can describe up to 20 skills.
provide only
name if
Assistant: Got it! Please provide the terminal observation image, and help identify the skill subtask executed between the initial and terminal states.
User: Now will show you the image of the terminal state, which means it is the last frame of this segment. You should first judge what is the most obvious one if many changes happen. Check the difference and choose the most probable one. Please determine whether to choose from the given subtask list or name yourself. Write down your choice format: the index selected by you; all the skills index and content. For example 1; content of 1 2.content of etc. Do not need to display your reasoning process.
the following
Prompt Design and Q&A procedure for MiniGrid
User: Are you familiar with the Minigrid environment used for reinforcement learning? In this environment, Door Key task have several subtasks. This environment has key that the agent must pick up in order to unlock door and then get to the green goal square. You can deduce which skill was executed or identify the agent's subprocess based on the initial and terminal observations.
will now show you image of the initial state. the middle
Here are all the possible subtasks already summarized: 1. part, move to the blue door 2. Try to get out from the left part 3. In the middle part, move to the red door 4. Reach the Goal. From my perspective you can reason what the skill is according to the initial observation and terminal observation. Now i will show you the image, it is the initial state.
Assistant: Got it, let's proceed with identifying the skills based on the initial and terminal observations. Please go ahead and show the terminal observation so we can determine which subtask was executed or describe a new possible skill if necessary.
User: Now I will show you the image of the terminal state, which means this segment. You should first judge what the most obvious one if many changes happen. Please determine which one to choose from the given subtask list or name yourself. Write down your choice in the following format: the index selected by you; all the skills index and content provided. For example 1;1. No obvious skill 2. Initial Positioning 3. Move to Key 4. Pass the door 5. Reach the goal position. Do not need to display your reasoning process.
it is the last frame
20

---

Published as a conference paper at ICLR 2025
Prompt Design for Crafter
User: Are you familiar with the Crafter environment used for reinforcement learning? Crafter features randomly generated 2D worlds where the player needs to forage for food and water, find shelter to sleep, defend against monsters, collect materials, and build tools. It is the open world survival game for evaluating a wide range of agent abilities. The agent's view evolves the timestep increases, like from wood to water. You can deduce which skill was executed or identify the agent's subprocess based on the initial and terminal observations. I will initial state. Here are all the possible subtasks already summarized: 1. No obvious skill collect sapling 3. collect stone. pay attention that the legend at the botton of the image shows new tools the agent made if it changes. From my perspective, you can reason what the skill is according to the initial observation and terminal observation. Now will show you the image, it is the initial state. The task happened between two states is summarized in the list, just point out the index. Try your best to suspect the skill.
show you an image of the
Assistant: Thank you for sharing the initial state image. To proceed, please provide the terminal state image as well. With both the initial and terminal observations, agent's subprocess according to the summarized list of possible subtasks. User: Now will show you the image of the terminal state, which means it is the last frame of this segment. You should first judge what is the most obvious one if many changes happen. Please determine which one choose from the given subtask list. Please only choose no obvious skill when indeed necessary, otherwise try your best to analyze according like water, stone or wood! Write down your choice with the following format: the index selected by you; all the 16 skills index and content provided. This is an example: 3; 1. No obvious skill 2. collect sapling 3. collect stone etc. not need to display your reasoning process.
can help deduce the executed skill identify the
the iconic item,
B.4 PSEUDO-CODE FOR VANTA
We provide the pseudo-code for VanTA further clarify the process. During the offline initialization phase, we iterate at times the VLM frequency to prepare the initial segmentation and reduce query complexity in the early stages. The absence of smooth loss may give rise misleading initialization which is not conducive for VLM to reidentify the sub-task. Therefore, we add the smooth loss with the appropriate coefficient, if meeting the no obvious skill option, the coefficient will increase to 10 times.
Algorithm Vision- languae model guided Temporal Abstraction (VanTAA) for offline RL
Require: offline data D; pretrained VLM; high-level policy πe; low-level policy π&;
query VLM frequency 1; temporal abstraction epoch N; policy update epoch M. 1: // Primitive Skill Extraction
2: for iter 1 to N do
3: Sample a minibatch {(s, a, s',r)} from D
4: Update codebook with Eq. (3)
5: if step meets query VLM frequency / then
6:
Query VLM with initial segmentation (51 $2,
7: Update codebook with VLM indice using Eq. 2
8: end if
9: end for
Relabel offline data with primitive skill D* (s s(,r,,
11: // Policy Extraction
12: Tw, Re = Policy Extraction(D::
Output: policy nw. Ne.
21

---

Published as a conference paper at ICLR 2025
B.5 ADDITIONAL RESULTS
B.5.1
MISSING SKILL EXTRACTION RESULTS
Figure 6: Skill extraction results without VLM guidance in -itchen-mixed; no clear semantic mean- ing each skill.
Figure 7: Skill extraction results using VanTA in buchen-mixedd sequentially 1) open the mi- crowave, 2) red flame circles appear on stove as it turns on the gas stove, 3) rotate the knob to turn on the white light above the stove, which is the left one among those knobs, and 4) open the cabinet.
Figure 8: Skill extraction results without VLM guidance in kitchen-partial; no clear semantic mean- ing each skill.
22

---

Published as a conference paper at ICLR 2025
Figure 9: Skill extraction results using VanTA in (ichenn-artiial sequentially 1) Move the kettle, 2) red flame circles appear on the stove as it turns on the gas stove, 3) rotate the knob to turn the white light above the stove, which is the left among those knobs, and 4) open the cabinet.
Figure 10: Skill extraction results without VLM guidance in MiniGrid-Doorkey-6x6: the baseline method mistakenly treats wandering as a valid action.
Figure 11:
Skill extraction results using VanTA MiniGrid- SeeiGrri-DDooKe--xx--- sequentially move to key, 2) pass the door, and 3) reach the goal position. Our method accurately segments the dataset into these three essential skills without any redundant parts.
23

---

Published as a conference paper at ICLR 2025
Figure 12: Skill extraction results without VLM guidance in MiniGrid-DDorrey-88868 no clear semantic meaning in each skill.
sequentially 1) move to key, 2) pass the door, and 3) reach the goal position. Unlike the baseline method, our approach can effectively separate the semantic information for 'move to the key" and 'pass the door'.
Figure 13: Skill extraction results using VanTA in ManiGrid-DoorKey--a
Figure 14: Skill extraction results without VLM guidance in MiniGrid- SonrGidddKKCCrrrdoSSSRR.. although it can be roughly divided into four semantic parts: move to the sky blue door at the top right corner, move to the purplish red door at the top right corner, reach the red key at the top left corner, and move to the door at the bottom right corner, there are still some unnecessary skills in between.
24

---

Published as a conference paper at ICLR 2025
Skill extraction results using VanTA in MiniGrid- MiniGrid-KeyCorricnn KeyCCoriddoSSRR3 sequentially 1) move to the sky blue door the top right corner, move the purplish red door the top right corner, 3) reach the red key at the top left corner, and 4) move to the red door at the bottom right corner. Compared to the baseline, our method merges segments that do not carry meaningful semantics.
Figure 15:
Figure 16: Skill extraction results without VLM guidance BiniGrid-RedBlueDe no clear semantic meaning in each skill.
Figure 17: Skill extraction results using VanTA in MiniGrid-RedBBleDDons:: sequentially 1) in the middle part move to the blue doors, and 2) in the middle part move the red door.
25

---

Published as a conference paper at ICLR 2025
Figure 18: Skill extraction results without VLM guidance in Crafter-partial; no clear semantic mean- ing each skill.
9-39 9-19'
3-399
$.39'
4.39'
3445
Skill extraction results using VanTA in Crafter-partial: sequentially 1) collect sapling, 2) sleep to refresh, 3) colleet water drink, and 4) collect wood. The extracted segments largely correspond to the achievements in the Crafter environment, and there significant scene differ- ences between distinct segments.
Figure 19:
B.5.2 VISUALIZATION OF THE REDUCED DATA EXPERIMENTS
We have included graphical visualization that more clearly demonstrates the slower degradation of VanTA compared other methods, such as CQL and GCSL. We believe this visualization high- lights the comparison more effectively and strengthens the clarity of our claims. The graphical representation provided in Fig. 20
Mean formom tca Digredation Across Kitenen rommerta
Ry
Dela Aviliilityy PII
Figure 20: Visualization reduced offline data experiment.
26

---

Published as a conference paper at ICLR 2025
B.5.3 FULL ABLATION STUDY
We conduct an ablation study across all environments, comparing the performance of VanTA with its variant that lacks VLM guidance. The results show that conditioning training on irrational primitive skills negatively impacts the final policy performance.
Table 5: Ablation study VanTA and conditioning only non-VLM guidance skills, averaged over random seeds.
Task Name
w/o VL.M Guidance
w/ VLM Guidance
MiniGrid-DD MiniGrid-DDooKey--ttoeee Deo-Key--xx--oompleett
0.91=0.02
0.92++0.03
MiniGrid DoorKey-6x6-mixee
0.72=0.14
0,80++0.17
MiniGrid-DD SauiGrid-DoorKae Key-8xx-compllee--
0.90=0.033
0.92++0.07
MiniGrid DoorKey-8x8-1o 8x8-mixed-v0
0.43
0.51
MiniGrid- KeeriidKeyCCooo KcyyCorridoSSSRR-ooppteeee
0.77110.12
0,81110,06
MiniGrid- Key CorridorS3R3-mixedd
0.47-10.09
0.61
MiniGrid-Redee RedBlueDoors 6x6-complete-vi
0.82=0.07
0.85
MiniGrid RedBlueDoors SoweeeDoooo----miioooeoo- 6x6-mixed-v0
0.67=0.12
0.73+1..20
Crafter-partial
2.8
5.46
B.5.4 DETAILS OF CRAFTER PERFORMANCE
observe that the policy induced by VanTA achieves the highest success rate almost half of the achievements. The overall score is calculated the geometric mean of success rates across all achievements.
Table 6: Performance listing of all achievements on Crafter benchmark for VanTA and 5 baselines, averaged over 5 random seeds.
Success Rate (%)
Achievement
BC
CQL
IQL
RvS
LDCQ
VanTA(Ours)
Collect Coal
0.2
0.0
0.0
0.0
0.0
0.6 1:: 0,2
Collect Diamond
0.0
0.0
0.0
0.0
0.0
0.0. 0.0
Collect Drink
15.6
33.4
4.2
20.0
0.0
17.91 5.5
Collect Iron
0.0
0.0
0.0
0.0
+ 0.0
Collect Sapling
42.8
63.4
88.2
90.6
99.0
9.4
Collect Stone
2.4
0.0
0.0
0.0
0.0
4.4+2.2
Collect Wood
73.6
53.8
60.2
73.8
0.0
74.1+-8.5
Defeat Skeleton
0.2
0.4
1.8
0.0
0.0
2.2 + 0.7
Defeat Zombie
11.8
20.8
28.2
15.0
21.8++6.3
Fat Cow
21.6
7.4
27.0
20.0
15.6
27.2+-2.4
Eat Plant
0.1
0.0
0.2
0.0
0.0
0.0 + 0.0
Make Iron Pickaxe
0.0
0.0
0.0
0.0
0.0
0.0 0.0
Make Iron Sword
0.0
0.0
0.0
0.0
Make Stone Pickaxe
0.0
0.0
0.0
0.0
0.0
0.0
Make Stone Sword
0.0
0.0
0.0
0.0
0.0
0.0
Make Wood Pickaxe
31.0
0.0
0.4
0.4
0.0
30.5 6.7
Make Wood Sword
17.2
0.0
15.0
0.0
0.0
21.7.18.9
Furnace
0.0
0.0
0.0
0.0-10.0
Place Plant
0.0
7.2
86.8
70.6
98.0
64.41 5.6
Place Stone
0.0
0.0
0.0
0.0
0.0
4,4
Place Table
53.8
0.0
36.8
7.0
0.0
73.1 + 2.4
Wake Up
0.2
47.6
0.0
0.0
16.748..1
Score
2.69
1.73
2.75
2.11
0.96
5.46
*The score is computed as S = exp
DN
In(1 st)) where [0, 100) is the success rate of the
N
i-th achievement and N 22 denotes the total number of achievements.
27

---

Published as a conference paper at ICLR 2025
Table 7: Performance listing of all achievements for VanTA ablation study on Crafter benchmark.
Success Rate (%)
Achievement
w/o VLM Guidance
VLM Guidance
Collect Coal
0.0% 0.0
0.6 0.2
Collect Diamond
0.0.1
0.0.L 0.0
Collect Drink
10.816.00
17.91 5.5
Collect Iron
0.0.1.00
0.0 +: 0.0
Collect Sapling
94.2 2.2
78.7 9.4
Collect Stone
0.0+0.0
4.4::2.2
Collect Wood
41.3#16.3
7..1
Defeat Skeleton
1.1+0.7
2,2 0.7
Defeat Zombie
20.3 2.3
6.3
Fat Cow
20.5 2.0
27.2++2.4
Eat Plant
0,0 0.0
0.0 0.0
Make Iron Pickaxe
0.0
0.0 0.0
Make Iron Sword
0.0 0.0
0.0
Make Stone Pickaxe
0.0
0.0
Make Stone Sword
0.0
0.0
Make Wood Pickaxe
3.2
30,5 6.7
Make Wood Sword
2 1.6
21.7.1899
Place Furnace
0.0+0.00
0,0+0..0
Place Plant
81.5
64.4 5.6
Place Stone
1,6 1.2
4.4
Place Table
18.211.8
73.1 + 2.4
Wake Up
1.8
16.7-8.1
Score
2.85
5.46
*The score is computed as S = exp i-th achievement and N 22 denotes the total number of achieverents.
1 N
Σn In(1 + s:)) - 1. where 8; E [0, 100) is the success rate the
28


Reviews:
[{"review_id": "mHvWDAzEfQ", "reviewer": "Reviewer_rUvx", "summary": "The paper proposes using vision-language models (VLMs) to annotate temporally extended skills from offline datasets. Specifically, the annotation is on the latent space after vector quantization and is improved iteratively. It saves the extensive manual labeling process. The proposed method shows robust improvements over the existing state-of-the-art techniques and the authors conduct ablations on the effectiveness of VLM guidance.", "questions": "Is it possible to finetune the VLMs to achieve even higher performance since the tasks are quite different from what VLMs are typically trained on (Internet data)? And is it cost-effective to fine-tune VLMs just to label the skills?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed method utilizes the knowledge of VLMs as accurate skill annotations to iteratively identify helpful low-level skills for various tasks;\n\n+ The annotation from VLMs is done in discrete latent space from the codebook which improves both the learning of the codebook, and the representation of each latent;\n\n+ The paper provides theoretical analysis to guide the algorithm designs;\n\n+ The empirical experiments show decent and robust improvements over existing methods.", "weaknesses": "Is it possible to finetune the VLMs to achieve even higher performance since the tasks are quite different from what VLMs are typically trained on (Internet data)? And is it cost-effective to fine-tune VLMs just to label the skills?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NWUqmQrO6C", "reviewer": "Reviewer_QKoU", "summary": "The method proposes a hierarchical policy, with a high-level skill selection policy trained via offline RL. A second low-level policy trained by behavior cloning outputs the actions given the current state and selected skill. For the skill extraction, a vector quantizied VAE is trained with labels provided by a VLM.", "questions": "- Out of interest: How many skills were identified with VLM guidance compared to those without VLM guidance?\n- From my understanding, the codebook is learned self-supervised with labels provided by the VLM. Do you have any experiments in a purely IL setting? Or could you provide further motivation for the policy extraction via Q-Learning dependent on a reward function?\n- For 6.4: could you provide some graphical visualization of the table? I believe this would make the claimed slower degradation more apparent.\n- Does the VQ get a single frame or multiple frames to segment? How does the segmentation work? How are the initial and terminal state of a primitive skill $\\bar{s}$ determined?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed hierarchical policy is an interesting method, with promising results.\n- The discretization offered by VQ-VAE enables the use of Q-Learning for the high-level skill policy while retaining the continuous output space of a low-level action policy.", "weaknesses": "- The clarity of the text should be improved. It often is hard to parse, and the text leaves it unclear, what the intentions are. For Example:\n    - The introduction leaves it unclear what the authors define as \"skill\"\n    - The Preliminary section's formatting makes it hard to parse. The Markov Decision Process (MDP) abbreviation is never actually defined. For VQ-VAE sg() is not defined. Same for Eq. 3\n    - Section 5 suddenly appears without any stated goal or context. What is it supposed to show?\n    The experiment section introduction is hard to read. In the last sentence of Section Five, the text describes the following order:\n    6.5 -> 6.2 -> 6.1 -> 6.4 -> 6.3. Then, it references Fig. 5, which is late in the Appendix.\n    - Line 210 states Eq. 3 but actually references Eq. 1\n- Figure 1 suggests the policy extraction process requires the codebook model. But the High-level policy, in my understanding, outputs a skill selection, i.e., a codebook entry. Why does it require the codebook?\n- Please clarify: Is the VLM only used during training or during rollout? Follow-up Question for the Ablation 6.3: For the method without VLM, in which stages has it been left out?\n- The mentioned main limitation is \"proper initialization\". Please explain what initialization means, as this is barely mentioned in the rest of the main text.\n- Ablations regarding the VAE policy decoder are missing. In general, the architecture of the policy decoder is not clear from the paper.\n\nThe proposed method and use of VQ-VAE + VLM is interesting. However, in its current state, the paper is hard to parse. Without improvements to the text, I tend towards reject.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lflcKYAXUe", "reviewer": "Reviewer_nVvm", "summary": "This paper proposes a novel method called VanTA (Vision-language model guided Temporal Abstraction) that utilizes Vision-Language Models (VLMs) to extract meaningful skills from offline reinforcement learning. VanTA extracts skills through an iterative process that involves initial segmentation based on VQ-VAE, followed by using VLM to assign meaning to the skills and update the codebook.", "questions": "* The experiment section appears to lack baselines related to skill learning among the comparisons. It seems necessary to include comparison groups associated with skill learning in the baselines.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper proposes a novel method called VanTA (Vision-language model guided Temporal Abstraction) that utilizes Vision-Language Models (VLMs) to extract meaningful skills from offline reinforcement learning. It claims to overcome the limitations of existing unsupervised learning approaches or methods that require human intervention.", "weaknesses": "The experiment section lacks baselines related to skill learning among the comparisons. It is difficult to determine whether VanTA's performance is due to the proposed algorithm or simply because it uses a skill learning framework. Given that skills are learned in the form of a codebook, a comparison with [1] seems necessary, and a comparison with [2], which utilizes LLM, also appears to be needed.\n    \n[1] Mazzaglia, Pietro, et al. \"Choreographer: Learning and adapting skills in imagination.\" *arXiv preprint arXiv:2211.13350* (2022).  \n[2] Fu, Haotian, et al. \"Language-guided Skill Learning with Temporal Variational Inference.\" *arXiv preprint arXiv:2402.16354* (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tdk8NlV4z3", "reviewer": "Reviewer_R8wN", "summary": "This paper introduces VanTA, an approach that integrates vision-language models (VLMs) into offline reinforcement learning (RL) for sparse reward settings. The author presents a hierarchical offline RL framework, where high-level skill policy training is augmented by the use of VLMs. Specifically, VLM is queried with image pairs to identify performed skills (e.g., pulling, pushing, picking, …), and this result is incorporated into the learning process of vector-quantized variational autoencoder (VQVAE), aiming to guide the discrete latent skill space to be more semantically meaningful. Conditioned on the skill, low-level control policy is separately learned via behavior cloning. Experiments on multiple environments show the superiority of VanTA.", "questions": "1. The implementation of behavior cloning (BC) for low-level policy learning (Line 229) appears to require expert demonstrations in the offline dataset. Then, how does BC-based learning achieve good performance with mixed-quality datasets in the Minigrid experiments?\n\n2. When reading through the middle sections of the paper, I  expect the VQVAE's encoder and decoder to serve as the high-level and low-level policies, respectively. Given that the training already utilizes expert datasets and Line 159 indicates that the VQVAE's decoder is replaced with a policy decoder, the authors should justify their architectural decision to train separate high-level and low-level policies instead of leveraging the VQVAE's existing structure.\n\n3. Given the large model size of the VLM used in this paper, I am not surprised that it outperforms other baselines. Did other baselines employ models of comparable size to the VLM?\n\n4. Was there any specific fine-tuning or prompt engineering applied to the VLM? While Line 104 points out the limited reasoning abilities of VLMs, it appears that this paper uses them as-is without any specialized adaptation techniques.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research direction of grounding VLMs to manipulation and beyond represents a significant and promising area of investigation in the field.\n2. The paper is well-organized and demonstrates consistent terminology usage, making it clear to readers.\n3. The experimental validation is comprehensive, spanning multiple environments.", "weaknesses": "1. The primary concern is the novelty of the contribution. The concept of extracting semantic skills via VLMs and grounding them to action policies has been previously explored, as demonstrated in [1]. The authors should clearly differentiate their approach from existing work, particularly regarding their VQVAE implementation compared to [1].\n\n2. A fundamental aspect of skill-based RL approaches is their ability to reuse extracted skills while reducing action space exploration, enabling rapid [2, 3] or even zero-shot [1] adaptation to new tasks. The paper should elaborate on the method's generalization capabilities beyond the tasks represented in the offline dataset.\n\n3. The application of VLMs in this work may not fully leverage their potential. Foundation models' primary advantage lies in their domain-agnostic knowledge, which should facilitate rapid policy adaptation across diverse domains. However, the proposed method's reliance on domain-specific reward structures potentially limits its generalization capabilities. The authors should address whether policies trained on specific tasks can generalize to different tasks.\n\n[1] One-shot Imitation in a Non-Stationary Environment via Multi-Modal Skill, ICML 2023\n\n[2] Skill-Based Meta-Reinforcement Learning, ICLR 2022\n\n[3] Accelerating reinforcement learning with learned skill priors, CoRL 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)