Discrete Latent Plans via Semantic Skill Abstractions
Authors: Haobin Jiang
Jiangxing Wang
Zongqing Lu
Venue: ICLR 2025 Poster (2025)
Keywords: Hierarchical Learning
Skill Learning
Imitation Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose LADS,  a novel hierarchical approach for learning skill abstractions from language.

Abstract: Skill learning from language instructions is a critical challenge in developing intelligent agents that can generalize across diverse tasks and follow complex human instructions. Hierarchical methods address this by decomposing the learning problem into multiple levels, where the high-level and low-level policies are mediated through a latent plan space. Effective modeling and learning of this latent plan space are key to enabling robust and interpretable skill learning. In this paper, we introduce LADS, a hierarchical approach that learns language-conditioned discrete latent plans through semantic skill abstractions. Our method decouples the learning of the latent plan space from the language-conditioned high-level policy to improve training stability. First, we incorporate a trajectory encoder to learn a discrete latent space with the low-level policy, regularized by language instructions. Next, we model the high-level policy as a categorical distribution over these discrete latent plans to capture the multi-modality of the dataset. Through experiments in simulated control environments, we demonstrate that LADS outperforms state-of-the-art methods in both skill learning and compositional generalization.

Published as a conference paper at ICLR 2025
DISCRETE LATENT PLANS VIA SEMANTIC SKILL AB- STRACTIONS
Haobin Jiang', Jiangxing Wang', Zongqing Lu Lu 'School of Computer Science, Peking University Beijing Academy of Artificial Intelligence
ABSTRACT
Skill learning from language instructions is a critical challenge in developing in- telligent agents that can generalize across diverse tasks and follow complex hu- man instructions. Hierarchical methods address this by decomposing the learning problem into multiple levels, where the high-level and low-level policies are me- diated through latent plan space. Effective modeling and learning of this latent plan space are key to enabling robust and interpretable skill learning. In this paper, we introduce LADS, a hierarchical approach that learns sunguage-condttion crete latent plans through semantic skill abstractions. Our method decouples the learning of the latent plan space from the language-conditioned high-level policy to improve training stability. First, we incorporate a trajectory encoder to learn discrete latent space with the low-level policy, regularized by language instruc- tions. Next, we model the high-level policy as a categorical distribution over these discrete latent plans to capture the multi-modality of the dataset. Through experi- ments in simulated control environments, we demonstrate that LADS outperforms state-of-the-arr methods in both skill learning and compositional generalization. The code is available at
dis-
INTRODUCTION
Creating an agent capable of understanding and executing natural language instructions has been long-standing goal in both reinforcement learning (RL) and imitation learning (IL.) (Luketina et al., 2019; Nair al., 2022). This capability is essential for developing a generalist artificial intelligence (AI) that can follow human commands to perform a wide range of control tasks, such as playing vir- tual video games (Lifshitz al., 2023) or performing real robotic manipulation (Brohan et al., 2022). Other generalist policies often condition on goal images (Nair et al., 2018) or states (Andrychowicz et al., 2017), where the goals are naturally grounded in the observation space. In contrast, language- conditioned policies face the challenge of grounding language into the observation space (Nair et al., 2022). With the development of vision-language models (VLMs), recent work has explored using pretrained models to achieve language grounding (Shridhar et al., 2022; Jiang & Lu, 2024). How- ever, these methods typically focus on visual inputs and object grounding, with limited effectiveness in understanding numerical states, such as robotic proprioception, and motion information.
Learning a hierarchical language-conditionee policy is an effective approach to addressing the chal- lenge of grounding language without being constrained by data modality. Hierarchical policy learn ing provides an intermediate representation that aligns language instructions and low-level control in shared latent space. This approach significantly simplifies the complexity of language ground- ing by only requiring the language-conditione high-level policy to map language instructions into temporally and semantically abstract latent plan space (Lynch et al., 2020), rather than directly controlling actions. This process is often described as decomposing a task into multiple smaller sub- tasks (Rosete- Beas et al., 2023). The low-level policy then responsible for generating the precise actions to interact with the environment. Specifically, it conditions on latent plan vector and acts as a skill for completing the sub-task indicated by this latent plan. In addition, hierarchical policies offer the benefits of sample efficiency in learning complex, long-horizon tasks and improve gener- alization to unseen scenarios through task decomposition (Garg et al., 2022; Mees et al., 2022a).
"Correspondence Zongqing Lu <rongging.lu@ppuu.cdu

---

Published as a conference paper at ICLR 2025
Recent work has advanced in this direction by implementing hierarchical policy, where the low- level policy is learned from an offline dataset annotated with language instructions (Garg et al., 2022; Ju et al., 2024; Liang et al., 2024; Fu et al., 2024). The challenge in acquiring skill abstractions, including the low-level policy and the latent plan space, from language instructions lies in learning them in unsupervised manner while ensuring the skills are both composable for complex, long- horizon tasks and interpretable for humans (Garg et al., 2022). To address this, these works opt for a discrete latent plan space for skill learning, as offers better controllability and interpretability compared to continuous representations. Furthermore, discrete latent representations have also been proven effective in various fields such as world models (Hafner et al., 2020; 2023), image generation (Esser et al., 2021; Rombach et al., 2022), and audio codecs (Zeghidour et al., 2021).
While promise has been shown, there a limitation in these methods where the high-level policy and low-level policy are trained jointly in an end-to-end manner (Garg et al., 2022; Ju et al., 2024; Liang et al., 2024). This can lead to potential training instability and difficulty, as the learning of the latent plan space and the language-conditionee high-level policy are entangled. The two components may affect each other's 's learning progress, resulting in index collapse in the codebook and thus requiring additional techniques to refine and stabilize the codebook (Ju et al., 2024). Inspired by task-agnosti skill learning methods (Pertsch et al., 2021; Rosete- Beas et al., 2023), we argue that incorporating an additional posterior distribution encode the low-level action sequences and learning the latent plan space in a variational way (Kingma, 2013; Van Den Oord et al., 2017) would be beneficial. At the same time, using language instructions as a regularizer might help construct a latent plan space with semanties and interpretability. By decoupling the learning of the high-level policy and low- level policy, we can model the high-level policy as a categorical distribution over the discrete latent plan space, allowing it to capture the multi mudti-mo of the dataset more effectively. For example, given one instruction, there may be multiple potential sub-tasks to choose from next.
In this work, we present LAnguage-con contiiigieod Discrete latent plans via semantic Skill abstractions (LADS) to address the limitation of joint end-to-end training discussed above. Our method consists of three main modules: a high-level policy, a low-level policy, and trajectory encoder. We use VQ- VAE (Van Den Oord et al., 2017) to jointly learn the low-level policy and the trajectory encoder. This results in discrete latent plan space, i.e., the VQ-VAE codebook. The high-level policy, conditioned on the language instruction, learns to make predictions in this discrete latent plan space for the next skill to execute. Specifically, it outputs categorical distribution over the discrete space and is supervised by the latent plan provided by the trajectory encoder over the future trajectory. Therefore, the learning of the high-level policy does interfere with the latent space. Furthermore, we align the latent plan sequence of each trajectory with its corresponding language instruction to regularize the latent space learned by VQ-VAE. We evaluate LADS in two simulated robotic control environments, LOReL (Nair et al., 2022) and Kitchen (Gupta et al., 2019), both of which have condaiiioood datasets. Our results demonstrate that LADS outperforms state-of-the- art baselines in skill learning and compositional generalization across instructions. Additionally, the ablation study confirms the significance of the proposed modules.
To summarize, our contributions are as follows: (1) We present LADS, novel hierarchical policy learning framework for skill abstraction from language, decoupling the learning of the language- conditioned high-level policy and the latent plan space. (2) We introduce trajectory encoder and utilize VA-VAE to learn a discrete latent space with semantic regularization to guarantee controlla- bility and interpretability. (3) We propose modeling the high-level policy as a categorical distribu- tion to effectively capture the dataset's 's multi-modality. (4) We demonstrate the superiority of LADS through quantitative comparisons and qualitative latent plan visualizations.
2 RELATED WORK
Hierarchical Policy Learning. Hierarchical policy learning is a widely explored approach to im- prove the efficiency and generalization of policies in both RL and IL. Typically, a high-level policy can generate goals explicit future states (Nair & Finn, 2019; Du et al., 2024; Black et al., 2024), implicit latent plans (Lynch et al., 2020; Pertsch et al., 2021; Rosete- Beas et al., 2023), or language (Hu et al., 2019; Jiang et al., 2019; Chen et al., 2021b). A low-level policy then takes action based on these assigned goals. In RL, the lowelevel policy is usually learned using information based ob- jectives (Eysenbach et al., 2018; Laskin et al., 2022; Park et al., 2023) or through joint training with
2

---

Published as a conference paper at ICLR 2025
the high-level policy maximize environment rewards (Kulkarni et al., 2016; Bacon et al., 2017; Vecriah et al., 2021). In IL, the low-level policy can be trained from an offline dataset using goal- conditioned IL (Kujanpää et al., 2023; Du et al., 2024; Black et al., 2024) latent variable modeling (Lynch et al., 2020; Pertsch et al., 2021; Rosete-Bean et al., 2023). In this work, we adopt the hi- erarchical policy learning framework and learn the low-level policy using discrete latent variable modeling (Van Den Oord et al., 2017) from an offline dataset with language instructions.
Language-Conditee Conditioodd Policy Learning. Enabling policy to follow natural language instruc- tion is a crucial step toward achieving generalist AI. Language can directly serve as a form of task representation for the policy (Hermann et al., 2017: Lynch & Sermanet, 2020; Jang et al., 2022). However, this requires the network to learn the structure of the language space and grounding into the environment from scratch, which presents significant challenges. Recent research uses pretrained large language models (LLMs) or VLMs to provide priors, simplifying anguage-conditione
train- ing through language grounding (Shridhar et al., 2022; Stone et al., 2023; Gao et al., 2024; Jiang & Lu, 2024) or task decomposition (Huang et al., 2022; Du et al., 2023; Singh et al., 2023). In this work, we train the low-level policy with semantic regularization to ground language with latent high-level policy.
plans and thus facilitate the learning of the Ianguage-conditionn
Conditioned Skill Abstractions. Recent work explores learning semantic and inter- pretable skills from ainguage-conditioned offline datasets. LISA (Garg et al., 2022) uses an end- to-end hierarchical policy to jointly learn the high-level and low-level policies from the dataset. SkillDiffuser (Liang al., 2024) improves this by using Diffuser (Ajay et al., 2022) as the low- level policy. LCSD (Ju et al.. 2024) adopts a one-step hierarchical policy with an auxiliary mutual information objective and diffusion policy (Ho al., 2020). LAST (Fu et al., 2024) applies vari- ational temporal inference (Kim et al., 2019) to learn skills but relies on an LLM for segmentation priors, limiting its applicability in environments without language- based action spaces. Our method builds on LISA and improves it by decoupling the learning of latent plans and skills from the learn- ing of the high-level policy, thereby enhancing robustness. Additionally, we introduce a categorical prediction head over the discrete latent plan space for the high-level policy, improving its ability to model the dataset's 's modaaity.......
3 PROBLEM SETUP
We consider a multi-task learning environment modeled as a task-augmented Markov Decision Pro- cess (MDP) (Garg et al., 2022). The set of tasks is T. where each task in T consists of one more sub-tasks e E. That is, T is a subset of the powerset of the sub-task set E, i.e., T C P(E). Each task is described by natural language instruction I E L. which specifies the sub-tasks included in the task. As shown in Figure 1, language instruction can contain two sub-tasks, e.g., open drawer and furn faucet right. We assume access to dataset consisting of N language-conditionel trajecto- ries D = [['', [[',,f s1. oi...., ST. azr, Fi=1 collected by a sub-optimal policy in the environment, where 8, E S denotes the state, ae E A denotes the action, and 7; is the length of the trajectory i.
We consider the problem of learning language-conditioned policy π <<ae (s,, 1) that outputs an action a, given the current state 8, and language instruction 1, under the dynamics p S A + S defined by the tusk-augee MDP. The decomposable structure of the task space TT makes this learning problem different from the standard multi-task imitation learning, where each trajectory is independently considered as single task in monolithic fashion (Jang et al., 2022; Mees et al., 2022b;a; Black et al., 2024). To improve sample efficiency and generalization, the policy must leverage the shared structure across trajectories, ie., common sub-tasks, to reduce the task space. However, trajectories are not annotated with the sub-tasks executed at each step. Therefore, the core challenge in this problem setup is to learn and reuse skills for sub-tasks in an unsupervised manner.
4 METHOD
In this section, we present the details of our method. We begin by defining objective for learning the skill-based hierarchical policy and optimizing its lower bound (Section 4.1). We decompose the objective into three components. First, we focus on learning skill abstractions from trajectory via VQ-VAE (Van Den Oord et al., 2017), which provides a discrete latent plan space (Section 4.2). Next, we build the high-level policy as a categorical distribution to predict the discrete latent plan
3

---

Published as a conference paper at ICLR 2025
Instruction: open drower and tem fourst right
Alignment
Whole Traj.:
2g
Fkx
History Trajectory
Cletr bution of lanent pians
Loos
Data
High-Lavel Policy
Categorical Prediction
Latent plans coces)
Not
my (z)s,
Trajectory Segment Tuit (1:(k) 1)8f
Cross Entropy
Behavior Cloning
Traj. Encoder g( s; a)
Low-Level Policy Tii (u|ss, =)
VQ
Z
Figure 1: Overview of Lannguage-oondttion Discrete latent plans via semantic Skill abstractions (LADS). For each trajectory segment 1: we use a trajectory encoder to map it into a discrete latent plan 2g through vector quantization (VQ). The low-level policy reconstructs actions from 2n- Meanwhile, the high-level policy predicts the index of 2 in the discrete latent plan space, based on the history trajectory T:kH and language instruction Lastly, we regularize the latent plan space by aligning the sequence {zo, 21,.... 1 of one trajectory to its language instruction.
for the next few steps (Section 4.3). Finally, we impose semantic regularization on the discrete plan space by aligning the sequence of latent plans with language instructions (Section 4.4). We train all modules jointly by combining the proposed losses (Section 4.5).
4.1 SKILL-BASED HIERARCHICAL LEARNING
We implements hierarchical framework consisting of high-level policy ny (2/TTa, () and a low- level policy
8t, 2). Specifically, the high-level policy takes as input the history trajectory T. {s1, .1....,......................... St, ag s(+1) } and the language instruction and selects latent plan from the latent plan space Z. Once a z is assigned, the low-level policy acts as a skill that executes the latent plan z. Following previous work (Garg et al., 2022; Liang et al., 2024), we assume that each skill lasts for H timesteps. We propose the following objective for learning this hierarchical policy,
max log pe (T++1:++ Hii-iiiHHTTT.. TT.. (),
which aims to maximize the likelihood of the future trajectory over the next timesteps T141:
{s--1. 0441..... St+H. a++H } given the history trajectory and the language instruction. 0 denotes the learnable parameters of the hierarchical policy. As for LISA (Garg et al., 2022), its learning objective can be viewed as a lower bound of Equation (1), as detailed in Appendix A. However, LISA learns the high-level policy and low-level policy in an end-to-end manner, causing the learning of the language-conditiones policy (high-level policy) to be entangled with the learning of the latent plan space (low-level policy). As a result, LISA has been found to show poor training stability and prone to cause index collapse in latent space (Ju et al., 2024).
To decouple the learning of the high-level and low-level policies, we introduce a trajectory encoder ((217(+1=++)) which encodes the ground-trun -ruth future trajectory over the next H timesteps into the latent plan space. This allows us to learn the latent plan space in a variational manner, along with the levelevel policy. We begin by bounding the learning objective in Equation (1) follows,
(e(())ru4...) log 2
log logp(T=+1++1rs
((21Ta+H,1)
where q an approximated posterior. We replace this posterior distribution with our trajectory encoder q(2|744144)) based on the intuition that the single latent plan should represent the low- level action sequences, relying solely on the future trajectory data. Then we can rewrite the RHS of Equation (2) and get the following learning objective to maximize,
2(7 2/7a, ((2(Te+11441)
(O) Eg (g(= log
Eg
log logp(a 2) DKL (e4 ()), (3)
4

---

Published as a conference paper at ICLR 2025
where constant terms related to environment dynamics are already removed. The detailed derivation process is available in Appendix A.2. By substituting the high-level policy (2(2 1) and low- level policy (r(aelse)2) 2) into p(2(T=;:) and p(ar+h Se+h,; 2) in (L)Ds((), respectively, obtain the objective for optimizing our skill-based hierarchical framework.
4.2 SKILL ABSTRACTIONS
To optimize the first term in FLADS we implement VQ- VAE (Van Den Oord et al., 2017) to learn the trajectory encoder and low-level policy, resulting in a discrete latent plan space Z. Skills, by design, are often distinct and categorical in nature, such as open drawer, move mug right, or pick up kettle The discrete latent space provided by VQ-VAE aligns well with this requirement because it forces the model to group similar low-level action sequences into the same cluster. In addition, the discrete latent plan behavior (Garg et al., 2022; Liang et al., 2024).
enhance the interpretability and controllability of the low-level policy's 's
Given input trajectory segment T++1++ the trajectory encoder ( maps the segment into a latent vector 2, which is then quantized to the nearest point in a set of discrete latent codes Z ( ,....,, A5) from the latent codehook of size M. This process be expressed as,
arg min [[9(T+11++)) 2 ''E2
The decoder, ie., the low-level policy mr(ae 2), then takes as input this quantized latent vector and reconstructs the future trajectory T++1:+H, which is used for both the skill execution by the low-level policy and training via the behavior cloning loss,
H
LBC = Σ
log mE (()
(5)
The VQ-VAE optimization objective includes two parts: the behavior cloning loss for reconstruction and a codebook loss to ensure the discrete latent vectors 2 are effectively learned,
LvQ LBC + [|gg[(((+14++))
z112 + Scommit (1q
sg[z]]]
(6)
where sg/-] denotes the stop-gradd goo-ggraieet operation, and Scommit is a hyperparameter controlling the commitment loss that encourages the encoder to produce that close to the quantized vectors.
4.3 DISCRETE LATENT PLANS
In our learning objective (O), the second term is KL divergence between the trajectory encoder and the high-level policy. This KL loss trains the high-level policy to predict the next latent plan and regularizes the latent plan space. We can treat the two learning processes separately by using a stop- gradient operation (Hafner et al., 2020),
DKL DDLL(ggat (elt ())
a www..ussguua/ - - DxI. (
(0)
(0).
where a controls the balance between the two KL terms. In this section, we describe the design of the high-level policy and its training with the objective (0). The second objective (0) regularizes the latent space and is detailed in Section 4.4.
Given the discrete latent plan space, two approaches can be used to build the high-level policy, The first approach predicts latent vector ? and then quantizes it to the nearest in the latent codebook, similar previous methods (Garg al., 2022; Ju et al., 2024; Liang et al., 2024). This approach trains the high-level policy in regressive manner, using the latent plan provided by the trajectory encoder as the target for The second approach to predict the index of the latent plan z in the codebook by formulating the high-level policy categorical distribution. This allows to optimize (0) via a cross-yntropy loss,
LCE log Ta (id()))
(8)
For VQ-VAE, deterministic encoder q(Fe+1a+11) replace the distribution q q=
5

---

Published as a conference paper at ICLR 2025
where 2 is selected according to Equation (4) and id(2) denotes the index of in the codebook.
We build our high-level policy following the second approach. By predicting discrete latent plans in categorical way, the high-level policy can more effectively model the dataset's 's inherent multi- modality. In datasets such as LOReL (Nair et al., 2022), the execution order of sub-tasks in a trajectory does not always follow the annotated language instruction. This means that for given language instruction, multiple latent plans & could be possible for the following steps. For instance, consider the instruction "open drawer and turn faucet right"; Some trajectories might execute sub- task open drawer first, while others might execute sub-task turn faucet right first. As result, the skills corresponding to these two sub-tasks may both have high probabilities at the start of the trajectory for this instruction. Furthermore, even trajectories corresponding to the same sub-task can exhibit multi-modality, because the collected data is sub-optimal and might contain multiple modes of behavior, such as performing the sub-task open drawer quickly or slowly (Rosete-Beas et al., 2023). Categorical prediction can naturally capture such multi-modal distributions, whereas regressing the value of may result in sub-optimal predictions between multiple potential options.
4.4 SEMANTIC REGULARIZZTION
In Equation (7), the objective (0) regularizes the latent plans towards the high-level policy. How- ever, in method, the high-level policy is modeled categorical distribution over the index of the latent plan 2, while the trajectory encoder directly provides a latent plan z. This makes intractable to compute the KL divergence between them on each trajectory segment. Therefore, we redefine (0) by summing it over the entire trajectory and get,
(0) log p(|]z),
where z = {20, 21..... ........... 3 represents the sequence of latent plans 2k. each encoded from trajectory segment +1)++)( in trajectory annotated with language instruction 1. The deriva- tion is detailed in Appendix A.3. (O) regularizes the latent plan space by encouraging the latent plan sequence z of the trajectory to predict its annotated language instruction This objective does not involve the high-level policy na. so optimizing it naturally stops gradient from backpropagating to the high-level policy and only regularizes the trajectory encoder.
Given the high dimensionality of the language space, it is extremely challenging to train network that directly predicts the language instruction. Therefore, we implement a surrogate loss function by aligning the sequence of latent plans with the language instruction I, following a contrastive loss approach similar to CLIP (Radford et al., 2021),
Lalign = (xL() (()LLI())),
(10)
where ((-) and (b) are two learnable projectors that map z and into the same dimensional space, respectively, and fCIIP(() denotes CLIP text embedding of the instruction 1. Specifically, d(z)
and (f(CT.I (4)) corresponding to the same trajectory are viewed as positive pairs, while other combinations of (zz) and vi ((SCLIP() (1)) are treated as negative pairs. The contrastive loss maximizes the cosine similarity between positive pairs and minimizes the cosine similarity between negative pairs using InfoNCE (Oord et al., 2018). The specific definition of LCL is provided in Appendix B.
By using this surrogate loss function Caligns we avoid the complexity of directly predicting language instructions. Instead, we impose semantic regularization on the discrete latent space by improving the alignment between latent plans and instructions.
4.5 LEARNING LADS
The total loss for LADS is given by combining the losses above,
LLADS = Lvo + LCE + >atign Catign
(1I)
where Aalign controls the weight of the semantic regularization. We optimize the trajectory encoder, high-level policy, and low-level policy jointly. As illustrated in Figure 1, each trajectory is divided into segments 1:(++ Among these losses, Lvq and LCE are calculated on each trajectory segment, while Latign is calculated over the entire trajectory.
Following previous work (Nair et al., 2022; Garg et al., 2022; Ju et al., 2024; Liang et al., 2024), we pretrained DistilBERT (Sanh, 2019) as the language encoder and a causal transformer (Chen

---

Published as a conference paper at ICLR 2025
et al., 2021a) as the high-level policy. During the evaluation, the high-level policy provides an index every H timesteps, and the low-level policy takes as input the latent plan 2 corresponding to this index in the codebook execute over the next H timesteps. More details on network architecture and training hyperparameters and the pseudocode for training LADS are listed Appendix B.3.
5 EXPERIMENTS
5.1 ENVIRONMENTS AND DATASETS
The two environments that we use to evaluate LADS and the corresponding datasets are as follows.
LOReL (Nair et al., 2022) is simulated domain developed on top of Meta-World (Yu et al., 2020), which contains Sawyer robot interacting with a drawer, faucet, Figure 2(a). and two mugs, as shown The robotic arm is controlled using delta end-effector control, and the observation space supports both fully observable state observation and partially observable im- age observation. The dataset consists of 50K trajectories, each with 20 timesteps. collected by random RI. policy in the en- vironment and labeled with procedurally generated post-hoc language instructions.
(a) L.ReI.
(b) Kitchen
Figure 2: Visualization of simulated environments that
we evaluate our method on: (a) LOReL, (b) Kitchen.
Following previous work (Nair et al., 2022; Garg et al., 2022; Liang et al., 2024), the evaluation in this environment consists of three parts: (1) atomic seen instructions, including 6 tasks, open drawer, close drawer, turn faucel left, turn faucel right, move black mag right, and move white mug down; (2) atomic rephrased instructions, containing rephrasings of the atomic seen instructions by replacing the noun or verb, or rewriting the entire instructions, (3) composite instructions, con- sisting of 12 instructions combining two or more atomic instructions, e.g., open drawer and turn faucer right. The performance evaluation metric is the success rate. Further details on the LOReL. environment settings and evaluation protocols are discussed Appendix C.I.
Kitchen (Gupta al., 2019) is a simulated domain developed on top of MuJoCo (Todorov et al., 2012), which contains Franka robot interacting with a kitchen scene including a microwave, kettle, four oven burners, a light switch, two hinged cabinets, and sliding cabinet, shown in Fig- ure 2(b). The robotic arm is controlled by joint position control, and the environment also supports both state and image observation. The dataset consists of 566 demonstrations collected by humans using VR controllers, with each demonstration manipulating four different elements of the scene in sequence. Since the original dataset lacks language instructions, we procedurally labeled each demonstration based on the four sub-tasks completed in it. For testing, we selected 3 instructions, resulting training dataset of 22 instructions and 509 demonstrations, Although the 3 instructions do not appear in the training dataset, the sub-tasks they contain are all present in the training dataset.
The evaluation in this environment includes: (1) seen instructions, referring the 22 instructions in the training dataset, and (2) unseen instructions, which are the 3 test instructions not present in the training dataset. We use two performance evaluation metrics: (1) N-rate, which counts the number of successfully completed sub-tasks out of four (Gupta et al., 2019; Pertsch et al., 2021); and (2) K- rate, which measures the rate which at least K out of four sub-tasks are completed (Mees et al., 2022b; Shin et al., 2024). More details on dataset construction are provided in Appendix C.2.
5.2 BASELINES
We compare LADS with the following baselines for learning language-conditioned skill abstrac- tions. (1) Lang DT (Garg et al., 2022): A language-conditioned Decision Transformer (Chen et al., 2021a) as baseline without hierarchical structure. (2) LISA (Garg al., 2022): A language- conditioned hierarchical learning method that trains the high-level and low-level policies end-to- end, using VQ (Van Den Qord et al., 2017) to discretize the latent space. (3) LCSD (Ju al., 2024):
7

---

Published as a conference paper at ICLR 2025
Table I: Success rates (%) of different methods on evaluation instructions in LOReL. (upper) with state observation space and (lower) with image observation space. LADS outperforms other meth- ods across most instructions. The best-performing methods are highlighted in bold.
Instructions
Lang DT
LISA
LCSD2 SkillDiffuser
VAE
Cluster
LADS
atomic seen atomic rephrased
27.8
38 6:3.9
60.2
18.7
47.7
32,4+2.2
68.3+4.5
21.426.9
38.4232
35.6
16.5
40.7a4.8
26,822.6
56.9
composite
12.5+2.4
13.424.2
11.8
21.0=3.3
11.3:3.1
30.0:11,2
atomic seen
15,0
40.0
50.8
39.3
17.4+24.1
18.3+10.1
52.5+2.1
atomic rephrased
24.2
27.5
32,9
36.6
15.7a22.8
11.8s48
36.2a2.1
composite
13.3
20.9
20.8
8.4+14.2
3.7+1.8
24.0=0.6
Similar to LISA, but adopts a diffusion-based (Ho et al., 2020) low-level policy and high-level policy that predicts the latent plan at every step, and introduces an additional mutual information objective. (4) SkillDifuueer (Liang et al., 2024): Based on LISA, replaces the low-level policy with Diffuser (Ajay et al., 2022). LCSD and SkillDiffuser are only evaluated in LOReL. (5) VAE: In- spired by Pertsch et al. (2021) and Cosete-Beas et al. (2023), this baseline uses VAE (Kingma, 2013) to learn continuous latent plan space from trajectory segments without semantic regularization. (6) Cluster: Inspired by Yuan et al. (2024), this baseline method first uses t-SNE (Van der Maaten & Hinton, 2008) and K-Means (Lloyd, 1982) acquire discrete latent plan space from trajectories and then learn the high-level and low-level policies. More detailed descriptions of these methods and their implementations can be found Appendix D.
5.3 EVALUATION RESULTS
Firstly, by evaluating these methods in LOReL, we aim to answer the question: How does the performance of LADS in skill learning and composition compare with other baselines? Our results in LOReL with state observation and with image observation are shown in Table 1. Success rates either cited from the original papers calculated by testing the trained model over 50 episodes. Due to space limitations, a detailed discussion is provided in Appendix D. The results containing standard deviations are calculated from three models trained with different random seeds.
The evaluation results show that LADS outperforms other methods on both atomic seen and com- posite instructions, demonstrating its superior ability to learn skill abstractions and sequentially combine learned skills. By comparing LADS with LISA, LCSD, and SkillDiffuser, we demonstrate the importance of decoupling the learning of the language- conditioned high-level policy and the latent plan space. Compared to VAE, we confirm the effectiveness of using a discrete latent plan space with semantic regularization. For Cluster in LOReL (image). we use the pretrained ResNet18 (He et al., 2016) to first embed the image observations and then perform clustering and hierarchical learning. However, this approach fails to achieve high success rates. We suspect the reason may be that this method places high demands on good image representation (Yuan et al., 2024), strong alignment with language in the evaluation environment, such as the use MineCLIP Minecraft (Fan al., 2022). Furthermore, distance might not be able to construct an optimal latent space for skill learning. Lastly, the results on atomic rephrased instructions suggest that LADS robust variations in language instructions.
discussed by Garg al. (2022), K-Means based on Euclidean
Secondly, by evaluating these methods in Kitchen, we aim to answer the question: How does the performance of LADS in generalization over unseen combinations of sub-tasks compare with other baselines? Results in Kitchen, measured by N-rate, are shown in Table 2. K-rate are shown in Figure 3 and Appendix E.I. Both metrics are calculated by testing each trained model over 50 episodes, with standard deviations calculated from three models trained with different random seeds.
The evaluation results show that LADS completes the most sub-tasks both seen and unseen instructions. demonstrating its superior ability to handle long-horizon tasks and generalize to unseen combinations of learned sub-tasks. LISA performs worse than the non-hion hon-hierarchical baseline, Lang
2LCSD has neither provided the evaluation result compasite instructions nor open-sourced the code.
8

---

Published as a conference paper at ICLR 2025
LADS
VAE USA
Lang
Table 2: N-rates of different methods on and unseen in- structions in Kitchen (upper) with state observation space and (lower) with image observation spapce. LADS outperforms other methods across all instructions. The best-performing
100
Rate
methods are highlighted in bold.
50
25
Instructions Lang DT
LISA
VAE
Cluster
LADS
seen 1.3920.12
0.71a0044
1.9320.03
0.6620.09
2.25
Sub-tasks
unseen
1.27+0.17
0.71+0.53
1.79+0.02
0.50+0.26
2.23=0.16
(a) seen instructions
seen
1.32a0.41
0.93+0.12
1.00=0.40
0.33s0.08
1.61
unseen
1.07
1.2920.20
1.27.0.60
0.6320.19
1.44.0.22
75
% Rate
50
Table Success rates (%) of LADS and its ablation meth- ods on evaluation instructions in LOReL (image). The best- performing methods are highlighted in bold.
25
2 Sub-Lasks
Instructions
LADS
w/ VAE
w/o CP
w/o SR
(b) unseen instructions
atomic seen
52.5+2.1
44.1+7.2
43.7+78
6.245.2
Figure 3: K-rates on (a) seen and (b) unseen instructions in Kitchen (state).
atomic rephrased
36.2a2.1
36,6:11.5
31.425.2
5.724.6
composite
24.0a0.6
18.2+1.0
20.6+1.3
1.1
DT, suggesting that the long context increases the learning difficulty for the high-level policy, further complicating the latent space learning. LADS avoids this issue by having the trajectory encoder map much shorter trajectory segments of length H into the latent space, reducing the complexity of learning. Additionally, LADS continues to outperform methods that use continuous latent space or direct clustering. By K-rates shown in Figure 3, we can observe that LADS achieves higher completion rate for each number of sub-tasks compared to other methods.
5.4 ABLATION STUDY
We compare LADS with the following ablation methods verify the effectiveness of the three components introduced in Section (1) LADS w/ VAE: Replaces VQ-VAB with a VAE, resulting in continuous latent space. Unlike the baseline VAE, this method includes semantic regularization.
(2) LADS w/o CP: The high-level policy directly predicts the latent plan & instead of predicting categorical distribution. In this method, the high-level policy is trained using an MSE loss. (3) LADS w/o SR: Our method without semantic regularization by setting Aulign = 0 in LLADS-
The ablation results in LOReL (image) shown in Table 3. Replacing VQ-VAE with VAE and using regressive prediction instead of categorical prediction both mildly impair our method, empha- sizing the importance of discrete latent plan space and the ability model the dataset's multi- modality. Removing semantic regularization from our method significantly affects its performance LOReL (image). In this environment, aligning the latent plan sequences with instructions helps prevent the discrete latent plan space from overfitting to sub-optimal trajectories by ensuring it en- codes semantic information relevant the sub-tasks. The ablation results in Kitchen can found in Appendix F.2. We also conduct a hyperparameter study on horizon H and the size of the latent codebook Z in Appendix E.3.
5.5 LATENT PLAN ANALYSIS
We visualize the distributions of latent plans predicted by the high-level policy learned in LADS to intuitively demonstrate: How well does LADS capture multi-modality? In LOReL (image), we use composite instructions and their atomic instructions for visualization. For example, one composite instruction "open drawer and move black mug right" and the atomic instructions "open drawer" and "move black mag right" contained in it. We then record the categorical distribution of the discrete

---

Published as a conference paper at ICLR 2025
0.6
0.8
Composite
atomicc
aismic
aiomiCA
atomic
8 0.3
S 0.4
latand plans
latent
(a) open drawer and move black mug right
(b) close drawer and turn faucet right
Figure 4: The distributions of latent plans predicted by the high-level policy. The line represents the distribution for the composite instruction, while the bars in two colors represent the distributions for two atomic instructions included the composite instruction, denoted as atomic A and atomic B.
latent plans predicted by the high-level policy at the start of an episode for each instruction. As shown in Figure 4, the latent plans with high probability for the composite instruction also show high probability for one of its atomic instructions. This means that our high-level policy learns to predict latent plans for composite instruction by approximately combining possible latent plans for every atomic instruction. Details of visualization process and additional visualization results are available in Appendix E.4.
We demonstrate the interpretability of our discrete latent plans by sualizing the predicted distributions above. We then illustrate the con- trollable behavior generated by our method. In LOReL (image), we visu- alize the gripper's trajectory over the first steps of episodes for the composite instruction "open drawer and move black mug right" and its atomic instructions. As shown in Figure 5, by comparing the trajecto- ries generated by LADS w/ VAE and LADS, we can observe the advantage of discrete latent plans: it enables the low-level policy to follow one of the atomic instructions at the beginning, thereby allowing the completion of sub-tasks sequence. In contrast, a continuous latent plan space causes the low-level policy to generate trajectory that falls between the two atomic instructions. This occurs because the high-level policy predicts latent plan that is between those of the atomic instructions, affecting the model's ability to complete either sub-task. A similar phenomenon also observed in LADS w/o CP, as discussed in Appendix E.4.
asomic
SEomiC
atamic
3.5
ac 42
(a) LADS w/ VAE
(b) LADS
Figure 5: Visualization of gripper trajectories in 3D space generated by (a) LADS w/ VAE and (b) LADS for the com- posite instruction "open drawer and move black mug right" and the atomic instructions included in it.
6 CONCLUSION
We propose LADS, a novel hierarchical approach for learning skill abstractions from language. Our method employs a discrete latent plan space to learn the low-level policy with semantic regular- ization and language-conditioned high-level policy to predict the distribution over discrete plans. Through experiments in two simulated control environments, we demonstrate the effectiveness of LADS and its superiority over kunguage-conditioned hierarchical methods which are trained end- to-end (Garg et al., 2022; Ju et al., 2024; Liang et al., 2024), as well as task-agnostic skill learning approaches (Pertsch al., 2021; Kosete-Beas et al., 2023; Yuan al., 2024).
Limitations and future work: A limitation of our method is its reliance on a preset horizon H, which assumes each skill lasts for fixed duration. requires prior knowledge of the dataset and environment to determine a suitable H. An alternative approach would be to learn additional termi- nation detection for skills. However, empirical results have shown that this method can easily fall into local optima, treating either single step the entire trajectory approach also requires prior knowledge, such as a minimum horizon for one skill (Kim et al., 2019). Therefore, key challenge in skill learning domain is how to design a low-level policy that adapt flexibly to different skill horizons without significantly increasing the learning difficulty at the same time, which is an issue left for future research.
skill (Fu et al., 2024). This
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by NSFC under Grant 62450001 and 62476008. The authors would like to thank the anonymous reviewers for their valuable comments and advice.
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making making? arXiv preprint arXiv:2211.156577 2022.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Picter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference artificial intelligence, volume 31, 2017.
Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Rich Walke, Chelsea Finn, Aviral Ku- mar, and Sergey Levine. Zero-shot robotic manipulation with pre-trained image-editing diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for wellwwrlld control at scale. arXiv preprint arXiv:2212.068/ 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 04:15084-15097, 2021a.
Valerie Chen, Abhinav Gupta, and Kenneth Marino. Ask your humans: Using human instructions to improve generalization in reinforcement learning. In International Conference on Learning Representations, 2021b.
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining reinforcement learning with large language models. In International Conference Machine Learning, pp. 8657-8677. PMLR, 2023.
Patrick Esser, Robin Rombach, and Bjor Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recogni- tion, pp. 12873-12883, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: 10XX::8802.0077, 2018.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35: 19343-18362. 2022.
Haotian Fu, Pratyusha Sharma, Elias Stengel-Eskin, George Konidaris, Nicolas Le Roux, Marc- Alexandre Côté, and Xingdi Yuan. Language-guided skill learning with temporal variational inference. arXiv preprint axXiv:2402.16354 2024.
Jialu Gao, Kaizhe Hu, Guowei Xu, and Huazhe Xu. Can pre-trained text-to-image -iaate--mmgeeao models generate visual goals for reinforcement learning? Advances Neural Information Processing Systems, 36, 2024.
11

---

Published as a conference paper at ICLR 2025
Divyansh Garg, Skanda Vaidyanath, Kuno Kim, Jiaming Song, and Stefano Ermon. Lisa: Learn- ing interpretable skill abstractions from language. Advances Neural Information Processing Systems,
2022.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956. 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02193: 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.041000 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer. David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded lan- guage learning in simulated 3d world. arXiv preprint arXiv: 706.7066.5555 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. Hierarchical de- cision making by generating and following natural language instructions. Advances in neural information processing systems, 32, 2019.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learming, pp. 911899111111111 9147. PMLR, 2022.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Confer- ence on Robot Learning, pp. 991-1002. PMLR, 2022.
Haobin Jiang and Zongqing Lu. Visual grounding for object-level generalization in reinforcement learning. arXiv preprint arXiv:2408.0/94 2024.
Yiding Jiang, Shixiang Shane Gu, Kevin Murphy, and Chelsea Finn. Language as an abstrac- tion for hierarchical deep reinforcement learning. Advances Neural Information Processing Systems, 32, 2019.
Zhaoxun Ju, Chao Yang, Fuchun Sun, Hongbo Wang, and Yu Qiao. Rethinking mutual information for language conditioned skill discovery on imitation learning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 34, pp. 301-309, 2024.
Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. Advances in
Neural Information Processing Systems, 32, 2019.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Kalle Kujanpää, Joni Pajarinen, and Alexander Ilin. Hierarchical imitation learning with vector quantized models. In International Conference on Machine Learning, pp. 17919, PMLR, 2023.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Michael Laskin, Hao Liu, Xue Bin Peng. Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Unsupervised reinforcement learning with contrastive intrinsic control. Advances Neural In- formation Processing Systems, 35:34478-34491, 2022.
12

---

Published as a conference paper at ICLR 2025
Zhixuan Liang, Yao Mu, Hengbo Ma, Masayoshi Tomizuka, Mingyu Ding, and Ping Luo. Skilldif- fuser: Interpretable hierarchical planning via skill abstractions in diffusion based task execution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 16467-16476, 2024.
Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila Mcllraith. Steve-l: A generative model for text-to-behavior in minecraft. Advances in Neural Information Processing Systems, 36, 2023.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2): 137, 1982.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. In Proceedings of the Twenty-Eighth International Joint Conference on Artifi- cial Intelligence. International Joint Conferences on Artificial Intelligence Organization, 2019.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.0 0704:20005.7648,, 2020.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp. 1113- 1132. PMLR, 2020.
Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): 1205- 11212, 2022a.
Oier Mees, Lukas Hermann, Erick Cosete-Beas, and Wolfram Burgard. Calvin: A benchmark for Lunguage-conditinn policy learning for long-horizon robot manipulation tasks. IEEE Roboties and Automation Letters, 7(3):7327-7333) 2022b.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018
Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. arXiv preprint 1909.099 2019.
Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. In Conference Robot Learning, pp. 1303-1315. PMLR, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprinl arXiv: 2018.
Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllabllty-awware unsupervised skill discovery. arXiv preprini arXiv:2302.057003 2023.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelcrating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188-204 PMLR, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
resolution image syuthesis with latent diffusion models. In Proceedings the
Erick Rosete-Bei Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for aask-agnottic offline reinforcement learning. In Conference on Robot Learning, pp- 1838-1849. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2025
V Sanh. Distilbert, distilled version of bert: Smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
Sangwoo Shin, Minjong Yoo, Jeongwoo Lee, and Honguk Woo. Semtra: A semantic skill translator for cross-domain zero-shot policy adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15000-15008, 2024.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on robot learning. pp. 894-98 906. PMLR, 2022.
Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. Progprompt: Generating situated robot task plans using large language models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 11523-11201 11530. IEEE, 2023.
Austin Stone, Ted Xiao, Yao Lu, Keerthana Gopalakrishnan, Kuang- Huei Lee, Quan Vuong, Paul Wohlhart, Sean Kirmani, Brianna Zitkovich, Fei Xia, et al. Open- world object manipulation using pre-trained vision- Aasoonlanguaae models. arXiv preprint arXiv:2200. 000300 2023.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pp. 5026-5033 IEEE, 2012.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 9(1)), 2008.
Vivek Veeriah, Tom Zahavy, Matteo Hessel, Zhongwen Xu, Junhyuk Oh, lurüi Kemaev, Hado P van Hasselt, David Silver, and Satinder Singh. Discovery of options via meta-learned subgoals. Advances in Neural Information Processing Systems, 34:29861 20066119973,, 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Haoqi Yuan, Zhancun Mu, Feiyang Xie, and Zongqing Lu. Pre-training goal-based models for sample-efficient reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:495-500 2021.
14

---

Published as a conference paper at ICLR 2025
A LEARNING OBJECTIVE DERIVATION
A.1 LISA LEARNING OBJECTIVE
LISA (Garg et al., 2022) consists of two modules: high-level policy nh < 1) and low-level policy (a 8t; z). LISA optimizes these two modules jointly in an end-to-end manner using be- havior cloning loss and VQ loss. The behavior cloning loss expressed as,
H
LBC = -R log (8eth,
(12)
h=1
From the learning objective in Equation (1), we have,
pog Tux I) log D Tas
2 d) log (( Ta,))
H
= () log
p o acth)
h=1
= ()
log 2o C.
(13)
h=1
where C' denotes the constant term related to environment dynamies. By replacing p(2(Ta, 1) with the high-level policy and p(ac+t|2 22, T=+h-111) with the low-level policy under the assumption that given the latent actions taken over the following H timesteps are independent of the past trajectory and language instruction, we can conclude that minimizing LBc is equivalent to maximizing a lower bound of the proposed learning objective in Equation (1).
A.2 LADS LEARNING ORJECTIVE
We begin by bounding the learning objective in Equation (1) as follows,
dogp(Tenn+ E (e log p(T 2(Ta,)
p(z|Ta+H,1 !)
Eg(elraam..) log
(14)
a( ((31T=+H,1)
introduce heuristic prior into the model that the single should represent the low-level action sequences, based solely on the future trajectory data. Then can rewrite the lower bound and get the following learning objective,
By replacing q(2|T=4+H) with the trajectory encoder q=
I(0) Egtelren a) log 21141)4
g(2(Tc+10-H)
where 2 I) can be factored as,
P(Te4144 p(2|T., 0) II
a(tn).
(15)
We apply the same assumption introduced Appendix A.I that given the latent z, actions taken over the following H timesteps are independent of the past trajectory and language instruction by replacing p(actalz, 11) with p(a++ 18th, 2). Then,
I(H)- B
logp(a+4lesthn2 2) D (lp(2/Ta, + (16)
15

---

Published as a conference paper at ICLR 2025
A.3 LADS SEMANTIC REGULARIZATION
We sum the learning objective in Equation (1) over all segments of length H in one trajectory 7 and obtain equivalent objective as follows,
log p((() Σ log po
(17)
where K == [[]] -1 - the number of segments. We can then bound this by,
logp(r|() 2 By((/r.)) log p(T,
(2
(2/t/)) log 2(T12,0)((111
(18)
g(21T,1)
Under the same independence assumptions Section 4.2, Appendix A. and Appendix A.2, we replace (x//T, I) with (=(rr) and p(r)z,() with p(r(z). This allows to rewrite the lower bound and define the learning objective as,
[(0) Ex(((r) log p(r(z)p(z11)
(((TT)
[x/r)) (og(((1()) Dkr,((
(19)
We factor q(2/T) and p(r[z) for each trajectory segment (8+11) follows,
<(=|T) = II
(((((Te--1(444)))
(20)
k==
p()(2)
IIII II
C.
(21)
Jmmn 0 Aml
Then the first term in J(0) can be factored as,
K
(x(r)) log (((l()))))
E Eg(ea (e
Σ logp(atzral 24) C,
(22)
k=
which is equivalent to the summing of first term in (O) over the entire trajectory. The second term in ((0) regularizes the trajectory encoder the level of the entire trajectory. Using Bayes' theorem, p(=l()= = p=))()())))p(( we can rewrite this term as follows,
(DkL_((<))7]lp(/ll) = Bg(()r) log p(1(=)p(2)
((2(T)p(1)
=Eg(+)r) log logp(1|z) Dxa(((=))))lp())) + C.
(23)
particular, the term -D. (g(2))) (p(()) regularizes the latent plans = generated by the trajectory encoder toward prior distribution p(z). Since we implement the trajectory encoder q( (=/r) as VQ, this KL term effectively reduces to the commitment loss that ensures each latent plan 2 generated by
the trajectory encoder is close to the nearest discrete latent vector in the codebook Z. This process
encourages the latent plan space to remain well-organized and regularized around a set of discrete latent codes.
Therefore, we can convert the original (,( in Equation (7) from being based on each trajectory segment to being based on the entire trajectory,
(0) Byga))) log p(]]z),
(24)
which imposes semantic regularization on the discrete latent space by encouraging the sequence of latent plans over trajectory to predict the corresponding language instruction. This objective does not involve the high-level policy тr so optimizing it naturally stops gradient from backpropagating to the high-level policy and only regularizes the trajectory encoder.
16

---

Published as a conference paper at ICLR 2025
B IMPLEMENTATION DETAILS
B.1 NETWORK ARCHITECTURE
Language Encoder. We use pretrained DistilBERT (Sanh, 2019) the language encoder. follow- ing previous work (Garg et al., 2022; Liang et al., 2024). We freeze its parameters to stabilize the language understanding process.
Observation Encoder. For state observations, we use a linear layer to encode the state. For image observations in LOReL (image), use the convolution layers adopted in previous work (Nair et al., 2022; Garg et al., 2022) to encode the images. In Kitchen (image), we use a pretrained ResNet18 (He et al., 2016) encode the images, with the ResNet18 parameters kept frozen during training.
High-Level Policy. Following LISA (Garg et al., 2022), we use a causal transformer (Chen et al., 2021a) as the language-conditioned high-level policy. The causal transformer takes as input a se- quence in the form {I1, 22, LL $1, Q1, $2, .g...., at). word embeddings from the language encoder. The output at each state token is passed through a linear layer to produce the logits for the latent plans. The transformer applies causal mask to hide future trajectory information.
where ø1, .2....,, JL represent the
Low-Level Policy, For LOReL and Kitchen (state), we use an MLP our low-level policy. The low-level policy takes the observation and latent plan as input and outputs the predicted action for one step. For LOReL,, we implement a 4-layer MLP with Leaky ReL.U. For Kitchen (state), we follow Pertsch et al. (2021) and use a 7-layer MLP with Leaky ReLU and ID batch normalization.
For Kitchen (image), our preliminary experiment results showed that policies without sufficient history information perform poorly in this environment. Therefore, we implement a causal trans- former with 2 layers and heads the low-level policy. It takes as input sequence in the form {z,, +1: 1o-H-W+1:.... SkH GkH. SkH+1, OkH+1.--. ((4+1) 1(k+1) which in- cludes the k-th trajectory segment, its latent plan Zk, and a history window of size W. The out- puts at state tokens JRHH+1,-... I( are passed through a linear layer to predict the actions agH+1,.... a(k++))H The transformer uses causal mask hide future trajectory information. Un- like the MLP-based low-level policy, this history-aware low-level policy leverages more information than the low-level policy used in LISA. So we also conduct an additional experiment to augment the original LISA with our low-level policy in Kitchen (image), and the results are presented in Appendix E.5.
Trajectory Encoder. For LOReL and Kitchen (state), we simply implement an LSTM to encode the trajectory segment. It takes as input the concatenation of the encoded state and action at each timestep, where the state is encoded by the observation encoder and the action by a linear layer. The output at the final timestep is passed through 3-layer MLP with Leaky ReLU to obtain the latent plan z, which is then quantized as described in Equation (4). For Kitchen (image), to handle the high dimensionality of ResNet embeddings, we use 2-layer, 4-head transformer to encode the sequence +1H+1; 0gH++. 8(k+ H): O(K+1) and then take the average
the sequence.
Semantic Regularization. We use CLIP VIT-B/32 to encode the instruction into a 112-dimessional embedding. The projector (--) is 3-layer MLP with ReLU that maps the 512-dimensional lan- guage embedding into a 64-dimensional space. Similarly, the projector d(.) is 3-layer MI. with ReLU that maps the sequence of trajectory into 64-dimensional space. For LOReL., where all trajectories have the same length, we concatenate each trajectory's 's sequence as the input to d(-). For Kitchen, where trajectories vary in length, introduce an additional transformer encode the 2 sequence into an embedding before passing it to ((-).
The contrastive loss we implement is as follows.
LCL. =
log N) ((fcrur())) + log NCF((((ccur(()) (2,1)~B
Σ
(25)
where B represents batch. (CE(6()), is given by.
exp(o(z) ((fc)r((+))/+))
NCE(o()).
(26)
EiE exp(o(z) (()c/1())/7))
17

---

Published as a conference paper at ICLR 2025
where is learnable temperature parameter, I+ is the positive instruction paired with the tra- jectory and } are negative instructions paired with other trajectories in the training batch. NCE()(fcLrr()) o(z)) in similar form by swapping and in Equation (26).
B.2 TRAINING HYPERPARAMETERS
The hyperparameters for the network architecture not covered in Appendix B.1, well as those related to training, are listed in Table 4. In LOReL, we train each model for 500 epochs. In Kitchen, train each model for 3000 epochs.
Table 4: Hyperparameters of our experiments.
LOReL (state)
LOReL (image)
Kitchen (state)
Kitchen (image)
High-Level Policy
hidden size
128
128
128
128
num layer
2
4
4
nur head
4
4
4
4
dropout
0.1
0.1
0.1
0.1
Low-Level Policy
hidden size
128
128
128
window size W
10
Trajectory Encoder
hidden size
128
128
128
128
codebook size
20
20
20
20
codebook dim
16
16
16
16
Scommis
0.25
0.25
0.25
0,25
VQ EMA update
0.99
0.99
0.99
0.99
Semantic Regularization
hidden size
256
256
128
128
num layer
2
2
num head
4
4
dropout
0.1
Aalign
0.0
0.4
1.0
0.01
Training
horizon Hf
5
10
20
20
batch size
256
256
16
16
learning rate
le-4
le-4
le44
le-4
optimizer
Adam
Adam
Adam
Adam
B.3 ALGORITHM
We provide the pseudocode of LADS as shown in Algorithm 1.
C ENVIRONMENT AND DATASETS
C.I LOREL
Observation Spaces. The state observation space for LOReL consists of 15 dimensions, including the poses robotic joints and objects in the environment. The image observation space for LOReL 64x64 RGB image. The dataset contains both state and image observations for each timestep.
18

---

Published as a conference paper at ICLR 2025
Algorithm Training LADS
Require: Dataset D consisting of language-paired trajectories, skill horizon high-level policy
The low-level policy Ti, trajectory encoder q, learnable projectors é and vb.
while not converged do
Sample (1, {s1. ......, ST, ar)) from dataset D
for 0 /
do
Set history trajectory T.KH = (51, a1.............................. SKH, OkH, SkH-1]
Set trajectory segment +1 = { Op#+1...... 1(++))H a(+1))#}
Acquire a latent code 2Ą from the trajectory segment +1H++1;(*+1)1 by Equation (4)
Compute behavior cloning loss this segment Lnc by Equation (5)
Compute VQ-VAE loss on this segment Liq by Equation (6)
Compute cross-entropy loss for the high-level policy on this segment L by Equation (8)
end for
Obtain Lvo by averaging { } and obtain LCv by averaging { {
Concatenate the latent codes of all segments = 20, 21.
1
Compute semantic regularization Lalign by Equation (10)
Update The Ti, 9. & and to using LLADS defined in Equation (11)
end while
Atomic Rephrased Instructions. We use the same rephrased instructions as in previous work (Garg et al., 2022; Liang al., 2024), which includes unseen noun, unseen verb, unseen noun + verb, and human provided categories. The atomic rephrased instructions consist of 71 distinct variations across all six tasks. We evaluate the performance of each method on these instructions by averaging the results across all individual rephrased instructions, rather than first calculating the average success rate for each category and then averaging across all categories.
Composite Instructions. We use the same composite instructions as in previous work (Garg al., 2022; Liang et al., 2024), which are listed in Table
Table 5: Composite instructions adopted in our evaluation.
Composite Instructions
open drawer and move black mug right pull the handle and move black mug down move white mug right move black mug down close drawer and turn faucet right close drawer and turn faucet left turn faucet left and move white mug down turn faucet right and close drawer
move white mug down and turn faucet left close the drawer, turn the faucet left and move black mug right open drawer and turn faucet counterclockwise slide the drawer closed and then shift white mug down
C.2 KITCHEN
Observation Spaces. The state observation space for Kitchen consists of 30 dimensions, includ- ing the poses robotic joints and objects in the environment. The image observation space for Kitchen is x RGB image, which we resize 224 fit the input requirements
19

---

Published as a conference paper at ICLR 2025
of ResNet18. Since the original dataset only contains state observations, reconstruct each state using env. sim. set_state and render images to obtain the image observations. In our experi- ments, the image observation space is augmented with a 9-dimensional joint state of the robotic arm. The policy network needs to learn how to extract the poses of task-related objects from the image.
Language Labeling. Since the original dataset lacks language instructions, procedurally labeled each demonstration based on the four sub-tasks completed it. The atomic instruction correspond- ing to each sub-task is listed in Table 6. We then generate the instruction for each trajectory by concatenating the four atomic instructions with "and" in order.
Table Atomic instructions labeled sub-tasks.
Sub-tasks
Atomic Instructions
bottom burner
activate bottom burner
top burner
activate top burner turn on light switch open sliding cabinet open left hinge cabinet open microwave door move kettle to top left burner
light switch slide cabinet
hinge cabinet
microwave
kettle
Seen and Unseen Instructions. We choose the following three instructions for compositional gen- eralization test:
Open microwave door and move kettle to top left burner and activate bottom burner and turn on light switch.
Move kettle to top left burner and activate bottom burner and open sliding cabinet and open left hinge cabinet.
Move kettle to top left burner and activate top burner and turn on light switch and open sliding cabinet.
The difficulty of these instructions increases progressively. For the first instruction, the training dataset includes instructions with the same first three sub-tasks. For the second instruction, the dataset has instructions with the same first two sub-tasks. However, for the third instruction, the training dataset does not include any instruction with the same first two sub-tasks.
In summary, during evaluation, the policy will encounter an unseen sub-task combination after com- pleting three sub-tasks in the first instruction, two sub-tasks in the second instruction, and one sub- task in the third instruction.
D BASELINES
Lang DT. We directly cite the success rates of Lang DT in LORel. (image) from Garg et al. (2022). Since Garg et al. (2022) does not provide the performance of Lang DT on atomic rephrased instruc- tions and composite instructions in LORel (state), we rerun the code again and report the average performance over three models trained with different random seeds in Table For experiments in Kitchen, we also enlarge the model by double the number of layers in the transformer.
LISA. We directly cite the success rates of LISA in LOReL (image) from Garg et al. (2022). Since Garg et al. (2022) does not provide the performance of LISA on atomic rephrased instructions and composite instructions in LORel (state), we renun the code again and report the average perfor- mance over three models trained with different random seeds in Table 1. For experiments in Kitchen, we also enlarge the model by double the number of layers in the transformer.
During reproduction of LISA in LOReL (state), we encountered the same issue discussed by Ju et al. (2024), where the released code could not achieve normal performance due to index collapse.
Phttps:/ comob..com/iiiii
20

---

Published as a conference paper at ICLR 2025
We optimized the code by updating its vector quantization algorithm to the latest version. As shown in Table I, this partially resolved the index collapse problem, resulting in meaningful success rates. However, were still unable to reproduce the 66.7% success rate reported in the original paper.
LCSD. We directly cite the success rates of LCSD in LOReL (state) and LOReL (image) from Ju et al. (2024).
SkillDiffuser. For LOReL (image), we test the trained checkpoint over 50 episodes to obtain a more found the results be quite unstable. For LOReL (state), modify
accurate evaluation, the code* to support state observations. This modification does not affect the diffusion model's capability, it diffuses the embedding of the state image rather than the raw observation.
VAF. We replace the VQ-VAE module with a VAE while keeping all other settings the same as in our method, except for the semantic regularization. We set the 3 of the KL regularization in the VAE objective to le-3 for LOReL and 5e-4 for Kitchen.
Cluster. In LOReL (state) and Kitchen (state), directly apply t-SNE the states and then use K- K-Means to obtain 20 clusters, matching the number of codes in our VQ-VAE codebook. In LOReL (image) and Kitchen (image), we apply the same clustering techniques of the images. The low-level policy (T(gg((t.g) is trained by setting the last observation in each trajectory segment as the goal. The high-level policy ny [T= 1) is trained by predicting the index of the cluster center nearest to the last observation in the next trajectory segment.
the & embeddings
E EXTENDED EXPERIMENT RESULTS
E K-RATE IN KITCHEN (IMAGE)
Results in Kitchen (image) measured by K-rate are shown in Table 7.
Table 7: K-rates of different methods on seen and unseen instructions Kitchen (image).
seen
unseen
Methods
2
3
4
2
3
4
Lang DT
73.9+20.4
42.7+14.5
12.9+5.1
2.0+0.9
78.0=12.7
25.6=18.0
2.9+2.1
0.0=0.0
LISA
62.3+8.6
20.8+3.2
3.0an.9
0.040.0
88.9+7.4
34.4+11.7
5.8+2.1
0.2+0.4
VAE
69.1
26.14
4.1.3.6
0.3a0.3
76.9=24.3
42.9a28.0
6.729.8
0.2+0.4
Cluster
32.928.6
0.0g0.0
0.020.0
0.0a0.0
63.1
0.020.0
0.0g0.0
0.0g0.0
LADS
90.6+10.3
55.5+17.5
14.3+4.9
4. 1.4.
91.627.8
47.3+18.3
4,9+1.5
0.4+0.4
E.2 ABLATION STUDY IN KITCHEN
We also conduct the abation study in Kitchen (state) and the result is shown in Table 8. LADS outperforms ablation methods on seen and unseen instructions.
Table 8: N-rates of LADS and its ablation methods on evaluation instructions in Kitchen (state).
The best-performing methods are highlighted in bold.
Instructions
LADS
LADS w/ VAE
LADS w/o CP
LADS w/o SR
seen
2.25
2.01
1.08=0.12
2.15.0.03
unseen
2.23
2.05
1.87=0.1
2.04
hhtpp::///giubbb.cco///aaaa//// ffuser
hhttps://gi
21

---

Published as a conference paper at ICLR 2025
E.3 HYPERPARAMETER STUDY
We analyze the influence of the horizon H and the size of the codebook Z on the performance of methods in LOReL (image) and Kitchen (state). As shown in Figure 6(a), overly short horizons can cause the low-level policy to focus on learning primitive action patterns, which is highly noisy. reducing the overall controllability of the hicrarchical framework. Conversely, overly long horizons place an excessive burden on the low-level policy by making it learn long-horizon skills.
50
LOReL
2.4
55
2.4
Success rate 195)
Success (96)
40
2.2
N-rate
30
2.0
25
1.6
10
Kitchen
1.2
20
5
40
Horizon
Codebook size
(a) Horizon
(b) Codebook size
Figure 6: Influence of (a) horizon H and (b) codebook size on success rates in LOReL. (image) and N-rates in Kitchen (state).
We observe that 20 codes are an appropriate choice for both LOReL and Kitchen for the codebook size, shown in Figure 6(b). A smaller codebook size is insufficient for the latent space and low- level policy to model the diverse sub-tasks and multi-modality present in the dataset. While larger codebook size theoretically offers better modeling capacity, it may potentially affect the learning efficiency of the latent space, as observed in Kitchen. In contrast, this influence appears to be minor LOReL.
E.4 VISUALIZATION
Latent plan distribution. As shown in Figure 4, we demonstrate the distributions of latent plans predicted by the high-level policy. Taking the instruction "open drawer and move black mug right" as example, we illustrate how to obtain the latent plan distributions for both the composite in- struction and its corresponding atomic instructions. First, we initialize the LOReL environment and input the composite instruction "open drawer and move black mug right" along with the current observation into the high-level policy. The high-level policy outputs the predicted distribution of latent plans, which visualize the green lines in Figure 4.
Next, we input the atomic instruction "open drawer" and the current observation into the high-level policy to obtain its corresponding latent plan distribution, visualized the blue bars in Figure 4. Similarly, we input the atomic instruction "move black mug right" to obtain its latent plan distri- bution, which is visualized the orange bars in Figure 4. We observe that the latent plans with high probabilities for the composite instruction also show high probabilities for one of its atomic instructions.
Trajectory visualization. In LOReL, the policy controls a robotic arm. To illustrate the skills learned by LADS, we visualize the trajectory of the end-effector, i.e., the gripper of the robotic arm. As shown in Figure 5, the three axes represent the y, and coordinates of the end-effector. All trajectories start from the same initial position, corresponding the end-effector's starting point at the beginning of each episode.
Latent plan visualization. We visualize the continuous latent plan = predicted by the high-level policy at the start of each trajectory, given the composite instruction: "open drawer and move black mug right" and the atomic instructions contained in it. For each instruction, we sample 50 times. We use multidimensional scaling (MDS) to reduce the raw latent plan to 2 dimensions, as this method preserves the distances between data points. As shown in Figure 7(a), the latent plan & output by the high-level policy of LADS w/ VAE for the composite instruction tends fall between those for the atomic instructions. Consequently, the gripper trajectories generated by LADS w/ VAE exhibit the
same pattern, seen in Figure 5(a). A similar pattern is observed in the latent plan output by the
22

---

Published as a conference paper at ICLR 2025
Dim
atomic composite
atomic compasite
quanti zed
Dim
Dim
(a) LADS w/ VAE
LADS w/o CP
Figure 7:
Continuous latent plan output by the high-level policy of (a) LADS w/ VAE and (b)
LADS w/o CP before quantization, reduced to dimensions.
handle open rotate turn
move right
dark
drawer rotate hancle
openturn faucet
move darkmug
open
dr awer
f anclore
faucet
faucet
$11de
rotate
blac Kahite
Gabanet
bl acck
shut
right
ggth cup push
e container
pplll
pushmug
aresser
push
shut
Slide
(a) code
(b) code
(c)
code
(d) code
(e) code 4
push
handle e
tap
right
faucet
right right&lgckiise
clockwise
bomove
mug
White
feft
pulll
faucet
cup
rotate
left
sefta
turn
move
mu
away
Tull
white
Tauce
left
rotate
rotate dark D blak
cup
d
rotate
awer
turntap
white
mug
d
open
b
black
awwe
faucet
lefr
lefe
turn
tap
counter coock coochelse
(I) code
(g) code
(h)
code
(i) code
ii
shut
closer
Cup
left
white
push
move
Fransiare
cup black
turn
cup
mug
white
dr awwerr
white
Lighten
mug
rum
ght
mu
go
Shifto
Lud
faucet
cup push
push tuueet
otate
open
mug rotate
move Lligh shift
dark
black
silde
push
right
retete
gttt.
handle
clockwise
(k) code 10
code
11
(m) code 12
(n) code
code 14
light
white
drawer
right rotate
right tap white
back
mug
mug cup
rotate
white
push
Fotate
gg mu
move
move
elocausze
white eeteennaines left faucet
bblack move
push
nove left
cup
shift
e
push
Lugete
mace
away
lefrott
te
cabine dr aaer
cup
unclose
(p) code 15
(q) code 16
(r) code 17
code 18
code 19
Figure 8: Word clouds in LOReL (image) for each latent plan
high-level policy LADS w/o CP. Although the quantization operation ensures that the provided to the low-level policy is meaningful within the codebook. there is still potential issue: quantizing z for the composite instruction, which falls between the atomic instruction plans, may result sub-optimal for completing sub-tasks.
Word clouds. To enhance the interpretability of our learned latent plan, we present word clouds corresponding to each latent code, ranging from 0 to 19, in LOReL (image). We record the latent plans selected by the high-level policy during the execution each atomic instruction and then reorganize the data associate each latent code with the words in its corresponding instructions.
23

---

Published as a conference paper at ICLR 2025
This transformed data is then used to generate word clouds. For each instruction, execute learned policy for 50 episodes. As illustrated in Figure each latent code approximately represents distinct atomic instruction. For example, codes 0, 7, 9, and 11 correspond to "turn faucet leff", while codes 1, 4, 13, and 15 correspond to "move black mug right".
Latent plan heat map. Based the statistical results described above, also generate heat maps illustrating the relationships between latent plans and text. As shown Figure 9, the heat maps include: (a) a correlation matrix between each latent plan and individual words; (b) column- wise normalization of this matrix to represent the frequency of each word associated with given latent plan; (c) row-wise normalization of this matrix to represent the frequency of each latent plan associated with given word; and (d) a correlation matrix between each latent plan and atomic instructions. The results demonstrate that most of the latent plans learned by LADS are selected during evaluation, and no index collapse, depicted in Figure (upper) of Ju et al. (2024), occurs.
(a) Correlation between latent plans and words
(b) Word frequency for each latent plan (column normalized)
(c) Latent plan frequency for each word (row normalized)
sun dicht
blick right
(d) Correlation between latent plans and instructions
Figure 9: Latent plan heat maps.
Multimodality under the same skill. As shown Figure and Figure 9(),, single atomic instruction skill can activate multiple latent plans. For instance, in Figure 9(d), the high-level
24

---

Published as a conference paper at ICLR 2025
policy tends to select latent plans 2, 3, 6, and 10 for the instruction "open drawer" and latent plans and for "turn faucet left". This indicates that multiple latent plans can represent the same skill. To explore this further, visualize the trajectories generated by manually setting the latent plan for the low-level policy to address the question: What distinguishes the behavior patterns of latent plans corresponding to the same skill? Similar to Figure 5, plot the end-effector trajectories over the first 10 steps of 10 episodes for two cases: (I) the low-level policy following latent plans 2, 6, and 10, associated with open drawer, and (2) the low-level policy following latent plans 0 and 9, associated with turn faucet left, as shown in Figure 10.
code
code 6
code
code 10
code
021
0.0
64
66
332
Y
Y BAR 400
(a) open drawer
(b) turn faucet left
Figure 10: Visualization of the end-effector's trajectories in 3D space generated by the low-level policy (a) following latent plans 3, 6, and 10, and (b) following latent plans 0 and 9.
As shown in Figure 10(a), the trajectories of latent plans 2, 6, and 10 move in the positive x-axis direction, indicating that the robotic arm is being controlled by the low-level policy move toward the drawer. In the LOReL environment, the drawer is located along the positive x-axis relative to the robotic arm's starting position. The trajectories of the three latent plans do not fully overlap, suggesting that different latent plans represent distinct behavioral patterns for accomplishing the same skill, as discussed in Section 4.3. In this case. they correspond to different spatial paths toward the drawer. In contrast, the trajectory of latent plan moves in the opposite direction, which we speculate represents the sequence of actions involved in pulling the drawer open after the robotic arm has gripped the handle. Similarly, we observe that latent plans 0 and 9 also show different behavior patterns for the same skill turn faucet left.
E.5 LISA WITH OUR LOW-LEVEL POLICY
Table 9: N-rates of methods LADS, LISA, and LISA with our low-level policy on evaluation in- structions in Kitchen (image).
Instructions
LADS
LISA
LISA w/ our low-level policy
seen
1.61
0.93:0.12
0.8420.27
unscen
1. 1,44+0.32
1.29+0.20
1.05=0.14
In Kitchen (image), we implement a history- low-level policy that leverages more information than the low-level policy used LISA. To ensure a fair comparison, we also augment the original LISA with our history-awart low-level policy. As shown in Table 9, LISA with our low-level policy performs at the same level as the original LISA. Therefore, we can conclude that LISA's failure in Kitchen (image) is due to its end-to-end training approach.
25


Reviews:
[{"review_id": "cDKa1uBTUX", "reviewer": "Reviewer_Rvu8", "summary": "This paper introduces a method for for language-guided. Hierarchical RL. The paper learns a high-level and low-level policy connected through a latent plan space. Unlike previous approaches, these policies are not learned end-to-end but are decoupled by learning a discrete latent space for the high-level policy to sample from. The paper shows results on two language-instructed control environments.", "questions": "Line 362ish\nWhat does \"procedurally label[ing]\" the demonstrations mean? Like, use the simulator to manually label what each of the trajectories are? This isn't really explained.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is extremely well presented and written. The methodology is quite complex, but the good use of figures (Fig. 1) and a clearly written methodology section (that explains the motivation behind each step of deriving the losses rather than just throwing math at you) makes the method easy to follow.\n\nThe paper is clear on what it sets out to do: learn a better way of jointly learning HRL policies in a latent planning space while avoiding some of the mode collapse and other issues of previous works and this goal is validated through experiments.\n\nThe experimental evidence of the claims of the paper appear to be well backed up in two environments. There are a few possibly concerns in the experiments, or ways to maybe make this stronger (see weaknesses), but holistically the experiments do show the methodology is effective, which is good.", "weaknesses": "The related work on language-guided HRL is a little bit sparse and skews very recent. Some early important works here such as [1], [2], [3] should be included and also possibly serve as inspiration for more simple baselines (see below) for language-based HRL.\n\nI do wonder if there are some simpler, language based HRL baselines that could have been compared to. Language DT is a good baseline, so appreciate that, but I wonder if there is a simple hierarchical baselines that could be done here as well. From the above works, people had much simpler methods of avoiding the collapse between the hierarchical policies. For instance, in [2], the high-level policy was learned through IL and then held fixed while the low-level policies were fine-tuned, with the intermediate space being a hidden state, rather than in language space. \n\nThere are quite a few loss functions that go into this method, L_VQ, the VQ-VQA objective, L_BC the behavioral cloning objective, L_CE, the categorical objective and L_align, the alignment objective. I was really hoping to see a more thorough ablation of all of these losses, but did not see them. Some of these, (w/o VAE or CP) sort of get at these, but not entirely. I would have in particular been curious on how important the BC loss was to these results (it's kind of a footnote in the description of the method but I suspect it's rather important to getting it to work), and how important some of these other losses are, the alignment loss in particular. Especially because this work is quite complicated and incorporates so many losses, it seems important to know how vital each of them actually is (even if for some it will quite obviously fail without, that's good to know).\n\n[1] Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. In Advances in Neural Information Processing\nSystems, pp. 9414–9426, 2019.\n[2] Chen, Valerie, Abhinav Gupta, and Kenneth Marino. \"Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning.\" International Conference on Learning Representations 2021.\n[3] Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. Hierarchical decision making by generating and following natural language instructions. In Advances in neural\ninformation processing systems, pp. 10025–10034, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bkmkwsQuBW", "reviewer": "Reviewer_3qsh", "summary": "The paper introduces a hierarchical skill learning framework called LADS (LAnguage-conditioned Discrete latent plans via semantic Skill abstractions) aimed at learning skill abstractions from language instructions to enhance skill generalization across diverse tasks. This method is designed to help AI agents interpret and follow human instructions by modeling a discrete latent plan space, which decouples high-level language-based policy from low-level control.\nLADS uses a VQ-VAE model to encode sequence of actions into a discrete latent skill space, which allows the high-level policy to predict categorical distributions over potential skills. The hierarchical structure includes three components: a high-level language-conditioned policy, a low-level skill policy, and a trajectory encoder, all of which contribute to stable training. The method demonstrates its effectiveness in robotic control environments like LOReL and Kitchen, outperforming baselines in skill learning and generalization to new task compositions. The key contributions include a decoupled hierarchical policy, improved discrete skill space interpretability, and enhanced multi-modal skill selection.", "questions": "Could you respond to the following two concerns I raised in the weaknesses section?\n- *Concern about Scalability*\n- *Concern about Discreteness*", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- **Clear Presentation:**\n    - The motivation is clearly presented.\n    - The related work section is very well-organized, outlining the relationship between this work and others across three different aspects, which helps readers—especially those unfamiliar with the field—understand the context more clearly.\n    - The methods are well-decomposed.\n\n- **Significant Performance Improvement on the Benchmark**", "weaknesses": "- **Concern about Scalability**\n\n    According to Appendix B.2, the codebook size for the trajectory encoder is 20, which is relatively small. With a larger dataset that includes more diverse atomic instructions, the method would likely require a larger codebook size, increasing the output dimension of the high-level policy. In such cases, I wonder if:\n    \n    - *LADS without CP* might outperform, as the high-level policy dimension would not necessarily need to increase.\n    - *LADS with VAE* might be advantageous due to its continuous latent space, which doesn’t require an expanded codebook size.\n    - A combination of the two, *LADS without CP and with VAE*, might also offer a performance benefit.\n\n- **Concern about Discreteness**\n\n    The set of atomic instructions used in each task appears discrete, meaning each instruction is mutually exclusive. However, if we were to encounter instructions with overlapping meanings but different magnitudes, such as:\n    \n    - *open the drawer 5 cm, open the drawer 10 cm, open the drawer 15 cm, ...*\n    - *move white mug right 5 cm, move white mug right 10 cm, move white mug right 15 cm, ...*\n\n    In such cases, I’m uncertain that having discrete latent plans would be beneficial for task-solving. Perhaps in these scenarios, *LADS with VAE* (as discussed in Section 5.4), with its continuous latent space, could be more effective. This raises the possibility that the proposed method may be overly tailored to tasks with discrete instructions.\n\n- **Need for Pseudocode**\n\n    Given that the proposed framework involves training multiple components, with each component requiring several neural networks, including pseudocode would provide readers with a clearer, holistic understanding of the process.\n\n- **Minor Correction**\n\n    - *Line 192:* The objective function does not include the parameter θ. It may be clearer to adjust the notation to explicitly include θ in the objective function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Cx0SWrbkj", "reviewer": "Reviewer_aYjy", "summary": "The authors introduce a two stage method which starts by learning a discrete latent space for compressing trajectories and then learns a high-level categorical policy in this latent space. They also add a semantic alignment loss to align latent plans with language instructions. The authors show good results in some language-conditioned BC tasks.", "questions": "* I find it very surprising that the version of this method without semantic regularization does so poorly. Do you have any insights as to why that might be? Why do other baselines, like VAE, not have a similar problem? Why do algorithms like VQ-BeT, which has a more expressive high-level policy head, not suffer from the same overfitting.\n * Can you please confirm the difference between your work and similar works like VQ-BeT? As far as I can tell, aside from some architectural differences the main difference is the semantic alignment.\n * Line 348 you mention that your evaluation scheme follows previous work, but none of the papers you subsequently cite have the same 'atomic seen / atomic rephrased / composite' division. Can you please explain the relationship between your scheme and the scheme used by LOREL and SkillDiffuser (seen / unseen noun / unseen verb / unseen noun / human provided)? This will help me more directly compare with the results in those works.\n\n\nSmall comments / questions:\n * Equation 10: I don't believe you define $\\mathcal{L}_\\textrm{CL}$ anywhere", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n * The semantic alignment objective seems to be novel\n\nQuality:\n * The authors ablate the main parts of their method and see that they are helpful\n * The authors achieve strong performance compared to the baselines they include\n * The authors provide a nice experiment showing the multimodal capabilities of their work.\n\nClarity\n * Overall the paper and its conclusions are easy to follow\n\nSignificance\n * Language conditioned imitation learning is an important problem and this paper does a good job trying to address it. I find the insight about the semantic alignment interesting.", "weaknesses": "Important issues:\n* A key weakness of the work is that it is very similar to several recent works that it does not compare to. In particular, [VQ-BeT](https://sjlee.cc/vq-bet/), [QueST](https://arxiv.org/abs/2407.15840) have very similar setups with a discrete skill learning step followed by a high-level policy learning step. QueST is relatively newer but VQ-Bet is a pretty big oversight, and there should be a comparison to one of or both of these works. \n * VQ-BeT also evaluates in the kitchen environment and it would seem that their results, as well as the results from several baselines, are far better than yours. In particular, you report an average of 2.25 goals achieved while they report 3.78, and their worst-performing non-naive baseline reports 3.09. It's important to add this comparison and explain why that is the case to justify the publication of this paper.\n\nMinor issues:\n * As I mention in the following section, it's difficult to compare the LOREL results to other works because your evaluation schema is different.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9V0MUcB0MI", "reviewer": "Reviewer_N4Ju", "summary": "Learning skills from language instructions poses a significant challenge for current algorithms, particularly when dealing with complex tasks that require sequential execution of multiple skills. Previous research (Garg et al. 2022, Liang et al. 2024, Ju et al. 2024) has addressed this issue by unsupervised learning of constituent skills from language-labeled trajectories. Notably, LISA (Garg et al. 2024) has developed a hierarchical approach to learn interpretable skill codes for each task; however, it encounters stability issues. This work improves upon LISA by decoupling the learning of high-level and low-level policies and introducing additional loss terms to enhance stability. Additionally, it addresses dataset multimodality by modeling the high-level policy as a categorical policy. Their approach, **LADS**, demonstrates more stable training, leading to improved generalization and compositionality performance, with ablation studies analyzing the impact of each modification made.", "questions": "In addition to the questions mentioned in the Weaknesses section, I have a few specific inquiries about the paper:\n\n* L252: What does it mean when you say the high-level policy regularizes the latent space?\n\n* I find Figure 4 quite confusing. Could you provide a clearer explanation? Specifically, how is the composite distribution calculated? What does the x-axis represent? Why does the probability of the composite distribution approach 0?\n\n* Figure 5 is also somewhat unclear. What are the labels for the axes, and what do the curves represent? Why are we only considering the first 10 steps?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The authors have pinpointed a specific and relevant issue in the prior algorithm, LISA (Garg et al. 2022), and have methodically sought to address it. They hypothesize that the end-to-end training of high-level and low-level policies in LISA leads to instability, and they have developed a new algorithm to overcome this challenge.\n\n* The paper is well-written, with each loss term clearly explained and supported by thorough proofs and mathematical details.\n\n* The authors have conducted extensive experiments, comparing their method to various baselines from previous research, which bolsters the credibility of their approach and hypotheses. The metrics and evaluations align with prior work, facilitating straightforward comparisons.\n\n* They have achieved impressive results in both compositionality and generalization tasks during evaluation.\n\n* The comprehensive ablation studies provide insights into the sources of performance improvements, allowing readers to gain a deeper understanding of the proposed algorithm.", "weaknesses": "* The paper argues that instability issues in LISA stem from the end-to-end training of high-level and low-level policies. While LADS decouples these policies, it still trains the trajectory encoder and low-level policy together in an end-to-end manner. How does this approach enhance stability compared to LISA's method? Additionally, the trajectory encoder in LADS has the benefit of looking ahead by $H$ steps before predicting the next skill code $z$. Could this be LADS's primary advantage over LISA?\n\n* The interpretability results presented in Section 5.5 are somewhat unclear and could be better articulated. I have specific questions in the Questions section below, but I am curious about how the skill codes in LADS compare to those in LISA. Are they interpretable? LISA demonstrated a mapping from skill codes to word clouds and behaviors. Do we observe similar behaviors in LADS? Are the skill codes collapsing like in LISA, or do they exhibit more stability?\n\n* The argument for multi-modality is currently presented in a somewhat unconvincing manner. For example, in Section 4.3, the authors state that the dataset may perform the same skill, such as \"open drawer,\" either quickly or slowly, leading to multimodality. How does this impact the categorical representation of skills, considering both variations would still be captured under the same skill? Furthermore, how does the distribution of skills matter if the low-level policy acts on only one skill at a time? Although the ablation results indicate that the categorical distribution performs slightly better, could this be due to the inherent difficulty of predicting the latent code $z$ directly for the high-level policy?\n\n* Semantic regularization appears to play a crucial role in this algorithm, but a compelling rationale for its necessity is lacking. Why do we require both $J_p$ and $J_q$? Why not use just one? What is the significance of the stop gradients? The explanation in lines 474-476 is confusing; it seems that even $J_p$ alone could suffice for language alignment. Why is $J_q$ also needed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)