LISA: Learning Interpretable Skill Abstractions from Language
Authors: Divyansh Garg
Skanda Vaidyanath
Kuno Kim
Jiaming Song
Stefano Ermon
Venue: NeurIPS 2022 Accept (2022)
Keywords: Imitation Learning
Natural language processing
compositional representation learning

Abstract: Learning policies that effectively utilize language instructions in complex, multi-task environments is an important problem in imitation learning. While it is possible to condition on the entire language instruction directly, such an approach could suffer from generalization issues. To encode complex instructions into skills that can generalize to unseen instructions, we propose Learning Interpretable Skill Abstractions (LISA), a hierarchical imitation learning framework that can learn diverse, interpretable skills from language-conditioned demonstrations. LISA uses vector quantization to learn discrete skill codes that are highly correlated with language instructions and the behavior of the learned policy. In navigation and robotic manipulation environments, LISA is able to outperform a strong non-hierarchical baseline in the low data regime and compose learned skills to solve tasks containing unseen long-range instructions. Our method demonstrates a more natural way to condition on language in sequential decision-making problems and achieve interpretable and controllable behavior with the learned skills.

LISA: Learning Interpretable Skill Abstractions from Language
Divyansh Garg" Skanda Vaidyanath"
Kuno Kim
Jiaming Song Stefano Ermon
Stanford University
{divgarg, svaidyan, khkim, tsong, ermon) @stanford. edu
Abstract
Learning policies that effectively utilize language instructions complex, multi- task environments is an important problem in sequential decision-making, While it is possible to condition on the entire language instruction directly, such an approach could suffer from generalization issues. In our work, we propose Learning Interpretable Skill Abstractions (LISA), a hierarchical imitation learning framework that can learn diverse, interpretable primitive behaviors or skills from language- conditioned demonstrations to better generalize to unseen instructions. LISA uses vector quantization to learn discrete skill codes that are highly correlated with language instructions and the behavior of the learned policy. In navigation and robotic manipulation tasks, LISA outperforms strong non-hierarchical Decision Transformer baseline the low data regime and is able to compose learned skills to solve tasks containing unseen long-range instructions. Our method demonstrates a more natural way to condition on language in sequential decision-making problems and achieve interpretable and controllable behavior with the learned skills.
Introduction
Intelligent machines should be able to solve variety of complex, long-horizon tasks an environment and generalize to novel scenarios. In the sequential decision-making paradigm, provided expert demon- strations, an agent can learn to perform these tasks via multi-task imitation learning (IL). As humans, it is desirable to specify tasks to an agent using con- venient, yet expressive modality and the agent should solve the task by taking actions in the environment. There are several ways for humans to specify tasks agent, such task IDs, goal images, and goal demonstrations. However, these specifications tend to be ambiguous, require significant human effort, and can be cumbersome to curate and provide at test time. One of the most natural and versatile ways for humans to specify tasks is via natural language.
"pull the handla and
Figure 1: Overview of LISA. Given language instruction, our method learns discrete skill ab- stractions 2, picked from a vector codebook C. The policy conditioned on a skill executes distinct be- haviors and solves different sub-tasks. See GIF.
The goal of tinguage-conditionen IL is to solve tasks trajec-
in an environment given language-conditione tories training time and a natural language instruc- tion at test time. This becomes challenging when the task involves completing several sub-tasks sequen-
Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

tially, like the example shown in Figure 1, A crucial step towards solving this problem is exploiting the inherent hierarchical structure of natural language. For example, given the task specification "pull the handle and move black mug right", we can split it into learning two independent primitive behaviors or skills, i.e. "pull the handle" and "move black mug right". If we are able to decompose the problem of solving these complex tasks into learning skills, we can re-use and compose these learned skills to generalize to unseen tasks in the future. This is especially useful in the low-data regime, since we may not see all possible tasks given the limited dataset, but may see all the constituent sub-tasks. Using such hierarchical learning, we can utilize language effectively and learn skills as the building blocks of complex behaviors.
Utilizing language effectively to learn skills is non-trivial problem and raises several challenges. (i) The process of learning skills from language-conditng conditioned trajectories is unsupervised as we may not have knowledge about which parts of the trajectory corresponds to each skill. (ii) We need to ensure that the learned skills are useful, i.e. encode behavior that can be composed to solve new tasks. (iii) We would like the learned skills to be interpretable by humans, both in terms of the language and the behaviours they encode. There are several benefits of interpretability. For example, it allows us to understand which skills our model is good at and which skills struggles with. In safety critical settings such as robotic surgery or autonomous driving, knowing what each skill does allows us to pick and choose which skills we want to run at test time, It also provides a visual window into neural network policy which is extremely desirable [54]. There have been prior works such as [38, 47, 13] that have failed to address these challenges and condition on language in a monolithic fashion without learning skills. As result, they tend to perform poorly on long- hong-horizon composition
tasks such as the one in Figure 1.
To this end, we propose Learning Interpretable Skill Abstractions from language (LISA), a hierarchi- cal imitation learning framework that can learn interpretable skills from language-conditioned offline demonstrations. LISA uses a two-level architecture a skill predictor that predicts quantized skills from a learnt vector codebook and policy that uses these skill vector codes to predict actions. The discrete skills learned from language are interpretable (see Figure 2 and 4) and can be composed to solve long-rang tasks. Using quantization maximizes skill reuse and enforces a bottleneck to pass information from the language to the policy, enabling unsupervised learning of interpretable skills. We perform experiments on grid world navigation and robotic manipulation tasks and show that our hierarchical method can outperform a strong non-hierarchical baseline based Decision Transformer [11] in the low-data regime. We analyse these skills qualitatively and quantitatively and find them to be highly correlated to language and behaviour. Finally, using these skills to perform long-range composition tasks on a robotic manipulation environment results in performance that is nearly 2x better than the non-hierarchical version.
Concretely, our contributions are as follows:
We introduce LISA, a novel hierarchical imitation framework to solve complex tasks specified via language by learning re-usable skills.
We demonstrate the effectiveness of our approach in the low-data regime where its crucial to break down complex tasks generalize well.
We show our method performs well in long-range composition tasks where we may need to apply multiple skills sequentially.
We also show that the learned skills are highly correlated to language and behaviour and can easily be interpreted by humans.
2 Related Work
2.1 Imitation Learning
Imitation learning (IL) has a long history, with early works using behavioral cloning [41-43] to learn policies via supervised learning on expert demonstration data. Recent methods have shown significant improvements via learning reward functions [21] or Q-functions [18] from expert data to mimic expert behavior. Nevertheless, these works typically consider a single task. An important problem here is multi-task IL, where the imitator is trained to mimic behavior on a variety of training tasks with the goal of generalizing the learned behaviors to test tasks. A crucial variable in the multi-task set-up is how the task is specified, egg vectorized representations of goal states [37], task IDs [24], and single demonstrations [56, 14, 16, 57]. In contrast, we focus on a multi-task IL setup

---

Figure 2: LISA Skill Heat map. We show the corresponding word frequencies for 100 different learned
skills on BabyAl BossLeevel task (column normalized). The x-axis is the skill index and the y-axis is the task vocabulary. LISA's learned skills interpretable, encode diverse behavior, and are distinctly activated for different words. (zoom in for the best view)
with task-specification through language, one of the most natural and versatile ways for humans to communicate desired goals and intents.
2.2 Language Grounding
Several prior works have attempted ground language with tasks or language source of
instructions for learning tasks with varying degrees of success [32, 55, 4, 39, 5J. [27] is a good reference for works combining language with sequential-decision making.
But apart from a few exceptions, most algorithms in this area use the language instruction monolithic fashion and are designed to work for simple goals that requires the agent to demonstrate single skill [40, 9, 20, 6] or tasks where each constituent sub-goal has to be explicitly specified [10, 46, 34, 3, 52, 50, 17, 35, 31J. Some recent works have shown success using play data [28] or pseudo-expert data such as LOReL [38] and CLIPORT [47]. LOReL and CLIPORT are not hierarchical techniques. [28] can be interpreted
hierarchical technique that generates latent sub-goals as function of goal images, language instructions and task IDs but the skills learned by LISA are purely function of language and states alone and do not require goal images or task IDs. [23, 22] and [49] are some examples of works that use a two-level architecture for language conditioned tasks but neither of these methods learn skills that interpretable.
2.3
Latent-models and Hierarchical Learning
Past works have attempted to learn policies conditioned on latent variables and some of them can be interpreted as hierarchical techniques. For example, [15] learns skills using latent variables that visit different parts of the environment's state space. [45] improved on this by learning skills that were more easily predictable using a dynamics model. But these fall more under the category of skill discovery than hierarchical techniques since the skill code is fixed for the entire trajectory, is the case with [15]. [29] and [26] are other works that use a latent- variable approach to IL. But these approaches don't necessarily learn latent variable with the intention of breaking down complex tasks into skills. With LISA, we sample several skills per trajectory with the clear intention of each skill corresponding to completing sub-task for the whole trajectory. Also, none of the methods mentioned here condition on language.
There has been some work on hierarchical frameworks for RL to learn high-level action abstractions, called options [51], such as [25, 58, 36] but these works are not goal-conditioned. Unlike LISA, these works don't it use language and the options might lack diversity and not correspond to any concrete or interpretable skills. Furthermore, none have used the VQ technique to learn options and often suffer from training instabilities.
3

---

3 Approach
The key idea of LISA is to learn quantized skill representations that are informative of both language and behaviors, which allows us to break down high-level instructions, specified via language, into discrete, interpretable and composable codes (see Fig. 2, Fig. 6 and Fig. 8 for visualizations). These codes enable learning explainable and controllable behaviour, as shown in Fig. and Fig. 4.
Section 3.1 describes the problem formulation, an overview of our framework, and presents our Lunuuage-oonittoong model. Section 3.2 provides details on the training approach.
3.1 Language-conditioned Skill Learning
3.1.1 Problem Sctup
We consider general multi-task environments, represented as a task-augmented Markov decision process (MDP) with family of different tasks T. A task Ti may be composed of other tasks in T and encode multiple sub-goals. For example, in navigation environment, a task could be composed of two more sub-tasks "pick up ball", "open door". in any hierarchical order. S,A represent state and action spaces. We assume that each full task has a single natural language description E L, where I. represents the space of language instructions. Any sub-goals for the task encoded within this single language instruction.
We assume access to an offline dataset 'D of trajectories obtained from an optimal policy for a variety of tasks in environment with only their language description available. Each trajectory r' = (r,[((s, af), (s2, ad),..., (07, ) consists of the language description and the observations 8t E S, actions a; E A taken over T timesteps. The trajectories are not labeled with any rewards.
Our aim is to predict the expert actions or, given a language instruction and past observations.
Note that each trajectory in the training dataset can comprise of any number of sub-tasks. For example, we could have trajectory to "open door" and another to "pick up ball and close the door" in the training data. With LISA we aim to solve the task "open a door and pick up the ball" at test time even though we haven't seen this task at training time. In a trajectory with multiple sub-tasks, the training dataset does not give us information about where one sub-task ends and where another one begins.
LISA must learn how to identify and stitch together these tasks learned during training, in order to solve a new language instruction such the one shown in Fig, 1 at test time.
3.1.2 Hierarchical Skill Abstractions
We visualize the working of LISA in Figure 3. Our framework consists of two modules: a skill predictor T. S C and policy æ
Instruction
Observations
x C - A. Here, C = {=,,..., , learnable codebook of K quantized skill latent codes. D is the dimension of the latent space of skills.
is a
VQ Codebook C
Lang
Skill Predictor
3 E R
Our key idea is to break learning behavior from
Vector Quartization 1
language in two stages: I) Learn discrete la-
tent codes z, representing skills, from the full- language instruction to decompose the task into smaller sub-goals
Skills E C
Learn a policy æ condi- tioned only on these discrete codes. In LISA, both stages are trained end-to-end.
Policy
Actions
(st, at) the skill predictor f predicts a skill code ă E RD at timestep v = fl, (st, st.)...)) These codes are discretized using vector quantization operation q(-) that maps latent its closest codebook entry = q()). The quantization operation q(.) helps in learning discrete skill codes and acts as a bottleneck passing language information. We detail its operation in Sec. 3.2.
Given an input
Figure 3: LISA Architecture: The skill predictor f gets the language instruction and sequence of observa- tions as the input, processed through individual encoders. It predicts quantized skill codes z using a learnable cook- hook C, that encodes different sub-goals, and passes them to the policy æ. LISA is trained end-to-end.

---

The chosen skill code 2, is persisted for H timesteps where H is called the horizon. More details on how we chose the horizon and ablations studies on the choice of can be found in appendix sections D and F.1. After H timesteps, the skill predictor is invoked again to predict new skill. This enforces the skill to act as a temporal abstraction on actions, i.e. options [51]. The policy и predicts the action a, at each timestep conditioned on the state and single skill code that is active at that timestep. For x to correctly predict the original actions, it needs to use the language information encoded the skill codes.
LISA learns quantized skill codes in a vector codebook instead of continuous embeddings as this encourages reusing and composing these codes together to pass information from the language input to the actual behavior. Our learnt discrete skill codes adds interpretability and controllability to the policy's 's behavior.
3.2 Training LISA
Learning Discrete Skills. LISA uses Vector Quantization (VQ), inspired from [53]. a natural and widely-used method to map an input signal to low-dimensional discrete learnt representation. VQ Iearns a codebook {='...., x of K embedding vectors. Given an embedding ž from the skill predictor it maps the embedding to the closest vector in the codebook:
=: arg min
2= 1
EEEc
with the codebook vectors updated to be the moving average of the embeddings closest to them. This can be classically seen as learning K cluster centers via k-means [19].
Backpropagation through the nonfdifferentiable quantization operation is achieved by straight- through gradient estimator, which simply copies the gradients from the decoder to the encoder, such that the model and codebook can be trained end-to-end.
VQ enforces each learnt skill to lie in C, which can be thought as learning K prototypes or cluster centers for the language embeddings using the seen states. This acts as a bottleneck that efficiently decomposes a language instruction into sub-parts encoded as discrete skills.
LISA Objective. LISA is trained end-to-end using an objective LLISA LBC + AL where Lae is the behavior-cloning loss on the policy mo. À is the VQ loss weight and Liv. is the vector quantization loss on the skill predictor fo given as:
LvQ(f) E, || [q(3] -112
with = (l (Se: s..),
sg [-] denotes the noo-gradient operation. LvQ is also called commitment loss. It minimizes the conditional entropy of the skill predictor embeddings given the codcbook vectors, making the embeddings stick to single codebook vector.
The codebook vectors are learnt using an exponential moving average update, same as [53].
Avoiding language reconstruction. LISA avoids auxiliary losses for language reconstruction from the skills latent codes and it's not obvious why the skill codes are properly encoding language, and we expand it here.
For given a signal X and code Z, reconstructing the signal from the code as X f(Z) using cross-entropy loss amounts to maximizing the Mutual Information (MI) [(X, Z) between X and 7 [1,7] 7]. In our case, we can write the MI between the skill codes and language using entropies as: T(z,1) H(=) - (= 1), whereas methods that attempt to reconstruct language apply the following decomposition: T(2, 1) (1) - (1 2). Here, (I), the entropy of language instructions, constant, and this gives the cross-entropy loss.
Thus we can avoid language reconstruction via cross-entropy loss by maximizing [(2, 1) directly. In LISA, Lvg (2 l), and we find there is need place constraint on Hx2) as the learned skill codes diverse, needing encode enough information to correctly predict the correct actions.
In experiments, we tried enforcing a constraint on H (z) by using extra InfoNCE loss term but don't 't observe any gains.
5

---

Algorithm Training LISA
Input: Dataset D of language-pant parrrdee-paded trajectories
Input: Num skills K and horizon H
I: Initialize skill predictor fø. policy Te
2: Vector Quantization op q(-)
3: while not converged do
4: Sample - = (1, {8n, 81, 8T) (op, an, (2....7))
5: Initialize 5 (8))
List of states Sample skill every H steps
6: for k 0.. T do
7:
- q(f((1,))
8:
for step t = 1..H do
> Predict actions using a fixed skill and context length H
9:
OkH+1 (2, H]) S SU
10:
Append seen state
11:
end for 12: Train fø, to using objective LLISA
13:
end for
14:
end while
14
container dr rwer
push close
Figure Behavior with fixed LISA options. We show the word clouds and the behavior of the policy obtained by using a fixed skill code 2 14 for an entire episode. We find that this code encodes the skill "closing the drawer", as indicated by the word cloud. The policy executes this skill with high degree of success when conditioned this code for the entire trajectory, across multiple environment initializations and seeds.
As result, LISA can maximize the MI between the learnt skills and languages without auxiliary reconstruction losses and enforcing only Lvg the skill codes. We empirically estimate the MI between the language and skill codes and find that our experiments confirm this in Section 4.6.
3.2.1 LISA Implementation
LISA can be be implemented using different network architectures, such as Transformers MLPs.
In our experiments, we use Transformer architectures with LISA, but we find that out method is effective with simple architectures choices such MLPs, as shown in the appendix section F.5. Even when using Transformers for both the skill predictor and the policy network, our compute requirement is comparable to the non-hierarchical Flat Transformer policy using fewer layers in each module.
get away with
Language Encoder. We use a pre-trained DistilBERT [44] encoder generate language embeddings from the text instruction. We fine-tune the language encoder end-to-end and use the full language embedding for each word token, and not a pooled representation the whole text.
Observation Encoder. For image observations, we use convolution layers to generate embeddings. For simple state representations, we use MLPs.
Skill Predictor. The skill predictor network is implemented small Causal Transformer network that takes in the language embeddings and the observation embeddings at each time step. The language embeddings are concatenated at the beginning of the observation embeddings before being fed into the skill predictor. The network applies a causal mask hiding the future observations.
Policy Network. Our policy network ग, also implemented Decison Transformer (DT) [11]. However, unlike DT, our policy is not conditioned on any reward signal, but on the skill code. The sequence length of the horizon H of the skills which is much smaller compared to the length of the full trajectory.
small Causal Transformer inspired by
Flat Decision Transformer Baseline. Our flat baseline is based on DT and implementation-vee similar to LISA, but without skill predictor network. The policy here is a Causal Transformer, where

---

we modify DT to condition on the language instruction embedding from pre-trained DistillBERT text encoder instead of the future sum of returns. We found this baseline be inefficient at handling long-range language instructions, needing sequence lengths of 1000 on complex environments such as Aloosss--Bssseeell in our experiments.
Since LISA has two transformers opposed to just one in the flat baseline we ensured that the baseline and our method had a similar number of total parameters. To this end, the flat baseline uses Transformer network with 2 self-attention layers, and LISA's skill predictor and policy use Transformer network with a single self-attention layer each. We also ensured that the embedding dimension and the number of heads in each layer were exactly the same in both LISA and the flat baseline. Details of this are provided in appendix sections D. and D.2 respectively. In fact, one could argue that LISA has less representation power because the policy transformer can only attend to the last H steps while the flat baseline can attend to the entire trajectory which is what makes it an extremely strong baseline. The flat baseline also uses the same pre-trained DittilIBERT text encoder model as LISA for dealing with natural language input.
Experiments
In this section, we evaluate LISA on grid-world navigation and robotic manipulation tasks. We compare the performance of LISA with a strong non-hierarchical baseline in the low-data regime. We then analyse our learnt skill abstractions in detail - what they represent, how we can interpret them and how they improve performance on downstream composition tasks.
For the sake of brevity, we present additional ablations in the Appendix F, on doing manual planning with LISA skills (Section F.2), transferring learned skills to different environments (Section E.3) and learning continuous skills (Section F.6).
4.1 Datasets
Several language-condditonee datasets have been curated as of late such as [46, 48, 13, 38, 33, 2, 10, 12J. Nevertheless, a lot of these datasets focus on complex-state representations and navigation in 3D environments, making them challenging train on and qualitatively analyze our skills shown in Fig. 4. We found BabyAl, a grid-worle navigation environment and LOReL, a robotic manipulation environment as two diverse test beds that were very different from each other and conducive for hierarchical skill learning as well as detailed qualitative and quantitative analysis of our learned skills and we use them for our experiments.
BabyAl Dataset. The BabyAI dataset [13] contains 19 levels of increasing difficulty where each level is set in grid world and an agent sees a partially observed ego-centric view square of size 7x7. The agent must learn to perform various tasks of arbitrary difficulty such as moving objects between rooms, opening or closing doors, etc. all with a partially observed state and a language instruction. The language instructions for easy levels are quite simple but get exponentially more challenging for harder levels and contain several skills that the agent must complete in sequence (examples in appendix section C. The dataset provides million expert trajectories for each of the 19 levels, but we use 0.1 - 10% of these trajectorics to train our models. We evaluate our policy on set of 100 different instructions from the gym environment for each level, which contain high percentage of unseen environments layouts and language instructions given the limited data we use for training. More details about this dataset can be found in Appendix C.I and in the Baby paper.
LOReL Sawyer Dataset. This dataset [38] consists of pseudo-experl trajectories or play data collected from replay buffer of random RL policy and has been labeled with post-hoc crowd- sourced language instructions. Hence, the trajectories complete the language instruction provided but may not necessarily be optimal. Play data inexpensive to collect [30] in the real world and it is important for algorithms to be robust to such datasets as well. However, due to the randomness in the trajectories, this makes the dataset extremely difficult use in a behavior cloning (BC) setting. Despite this, we are able to achieve good performance on this benchmark and are able to learn some very useful skills. The LOReL Sawyer dataset contains 50k trajectories of length 20 on simulated environment with Sawyer robot. We evaluate on the same set of 6 tasks that the original paper does for our results in Table I: close drawer, open drawer, lurn faucel right, turn faucet left, move black mug right, move white mug down. We use two different settings with robot state space observations and aartially-observer image observations. More details can be found in the Appendixd C.2 and in the LOReL paper.
7

---

Table 1: Imitation Results: We show our success (in %) compared to the original method and flat non hicrarchical Decision Transformer baseline on each dataset over 3 seeds. LISA outperforms all other methods in the low-data regime, and reaches similar performance as the number of demonstrations increases. Best method shown in bold.
Task
Num Demos
Original
Lang DT
LISA
BabyAl Gooooeq
1k
33.3 = 1.3
49.3 0.7
59.4 : 0.9
BabyAl Go ToSq
10k
40.4 = 1.2
62.1 1.2
65.4 1.6
BabyAl Goooo
100k
47.1 = 1.1
74.1 2.3
77.2 - 1.7
Ba SynthSeq
1k:
12.9 = 1.2
42.3 = 1.3
46.3 + 1.2
AT SynthSeq
10k:
32.6 - 2.5
52.1 = 0.5
53.3 + 0.7
BabyAl SynthSeq
100k
40.4 3.3
64.2
61.2 + 0.6
Ba BossLeevel
1k:
20.7 + 4.6
44.5
= 3.3
49.1 + 2.4
BabyAl Bessl
10k
28.9 + 1.3
60.1
+ 5.5
58 + 4.1
Ba IT Bess
100k
45.3 - 0.9
72.0 4.2
69.8 3.1
LOReL States (fully obs.)
50k
6 1.2
33.3 5.6
66.7 5.2
LOReL Images (partial obs.)
50k
29.5
0.07
15 -
3.4
40 2.0
We optimized a language conditioned BC model following the LORel. puper to the best of our abilities but could not get better performance.
4.2 Baselines
Original. These refer to the baselines from the original paper for each dataset. For BabyAl, we trained their non-hien -on-hierarchiaa RNN based method on different number of trajectories. Similarly, BC. The original LOReL method learned reward function to get around the sub-optimal nature of the
LOReL we compare with the performance of language-conditee uses planning algorithm trajectories. We found the BC baseline as more fair comparison, as LISA is trained using BC as well. Nonetheless,
compare with the original LOReL planner in Section 4.7 for composition tasks. LOReL results in Table refer to the performance on the 6 seen instructions in the LOReL. evaluation dataset, same as ones reported in the original paper.
Flat Baseline, We implement a non- non-hierarchical baseline using language-connitioned Decision Transformer denoted as Lang DT, the details of which are in section 3.2.1.
4.3
How does performance of LISA compare with non-hierarchical baselines in low-data regime?
We consider three levels from the BabyAl environment and the LORel. Sawyer environment. For Baby AI, we consider the GoToSeq, SynthSeq and BossL tasks since they are challenging and require performing several sub-tasks one after the other. Since these levels contain instructions that are compositional in nature, when we train on limited data the algorithm must learn skills which form complex instructions generalize well to unseen instructions at test time.
Our experimental results are shown in Table 1. We train the models on randomly sampled 1k, 10k and 100k trajectories from the full Baby Al dataset and 50k trajectories on the LORel. dataset. We use more data from the LORel. dataset because of the sub-optimal nature of the trajectories. On all the environments, our method is competitive or outperforms the strong monhhierarc Decision Transformer baseline.
The gap grows larger as we reduce the number of trajectories trained on, indicating that our method is able to leverage the common sub-task structures better and glean more information from limited data. As expected, with larger amounts of training data it becomes hard to beat the flat baseline since the model sees mentioned above, we evaluate on the same 6 seen instructions the original LORel. paper did. We also evaluate the performance on varying language instructions on LOReL., similar to the original paper, with additional results in Appendix E.
compositions during training and can generalize better at test time [8]. As
We were pleasantly surprised that LISA is 2x better than the flat Lang-DT baseline on LOReL tasks, reaching 40% success rate using partial image observations despite the sub-optimal nature of the data. One explanation for this that the discrete skill codes are able to capture different ways of doing the same lask, thereby allowing LISA to learn implicit mulli-modal policy. This is not possible with the flat version as it has no way to compartmentalize these noisy trajectories, and perbaps tends to overfit on this noisy data, leading to performance degradation.
8

---

Figure 5: LISA Skill Heat map on LOReL. We show the corresponding word frequency for 20 learned skills on LOReL. (column normalized). The sparsity and the bright spots show that specific skills correspond to specific language tokens. (zoom in for best view)
4.4 What skills does LISA learn? Are they diverse?
To answer this question, we analyse the skills produced by LISA and the language tokens corresponding to each skilI. We plot a heat map in Figure 5 corresponding to the correlation between the language tokens and skill codes, Here, we plot the map corresponding to the LOReL dataset. From the figure, we can see that certain skill codes cor- respond very strongly to certain language tokens and by extension, tasks. We also see the sparse nature of the heat maps which indicates that each skill corresponds to distinct language tokens. We also plot word clouds corresponding to four different options in the LOReI. environment in Figure and we notice that different options are triggered by different language tokens. From the figure, it is clear that the skill on the top left corner corresponds to close the drawer and the skill on the top right corresponds to turn faucet left. Similar word clouds and heat maps for the BabyAl environments in the appendix section B.3.
container drawe awer push pushclose
rot
contain
turn
turnleft
faucet
drawer
pull open
black cup
mu
bo
te
push
container shut
right
handl
dr awer c109ed
down
s move
creesser
kig
Figure 6: Word clouds LOReL: We
show the most correlated words for 4 differ- ent learnt skill codes on LOReL. We can see that the codes represent interpretable and dis- tinguishable skills. For e.g. the code the top left corresponds closing the drawer. (note that container a synonym for drawer in the LOReL datasct)
4.5
Do the skills learned by LISA correspond interpretable behavior?
We have seen that the different skills correspond different language tokens, but do the policies conditioned on these skills behave according to the language tokens? To understand this, we fix the skill code for the entire trajectory and run the policy i.e. we are shutting off the skill predictor and always predicting the same skill for the entire trajectory. As we can see from the word cloud and the corresponding trajectory Figure 4, the behaviour for skill code is exactly what we can infer from the language tokens in the word cloud close the drawer. More such images and trajectories can be found in the appendix section B.5.
4.6 Why do LISA learned skills show such a strong correlation to language?
As mentioned in section 3.2, the com- mitment loss from VQ acts way
Mutual Information
to increase the MI between the lan-
guage and the skill codes during train-
Oettoeeette Hortzoer1
ing. This allows the codes to be highly correlated with language without any
Outterset, Hortzsecte
DoL-ors-1e0, Horazoewte
Oatiora=d0, Horissn-10
reconstruction losses. To analyze this,
oettersmte, Horizons10
we plot the MI between the options and the language during training on
Batiorsede, Horazoned
Oatiorando, Horbzone10
Baby BossLevel with trajec- tories and the plot can be seen in Fig- ure The plots show the MI increas- ing over training for wide range of
Truit (errttorss
Figure 7: MI between language and skill codes: We show the Mutual Information over training iterations for various settings of LISA on the BabyAl BossLevel environment

---

100k trajs settings as we vary the number of skills and the horizon. In the ablation studies below, we report the success rate corresponding to each of these curves and we notice that there's 's almost a direct comelaatin with increasing MI and task performance. This is very encouraging since it clearly shows that the skills are encoding language and that directly impacts the performance of the behavior cloning policy.
4.7 Can we use the learned skills to perform new composition tasks?
To test our composition performance, we evaluate on LOReL composition tasks using images in Table 2. To this end, we handcraft 15 unseen composi- tion instructions. We have listed these instructions in the Appendix Table with one such example "pull the han- dle and move black mug down". We
Table 2: LISA Composition Results: We show our performance on the LOReL Sawyer environment on 15 unseen instructions com- pared to baselines
Method
Success Rate (in %)
Flat
13.33 + 1.25
LOReL Planner
18.18 1 1.8
LISA (Ours)
20.89 + 0.63
ran 10 different runs of each instruction across 3 different seeds. As nearly 2x that of the non- non-hierarchical baseline. We also compare with the original LOReL planner on these composition tasks and notice that we perform slightly better despite them having access to reward function and dynamics model pre-trained on 1M frames while LISA is trained from scratch. We set the max number of episode steps to 40 from the usual 20 for all the methods while performing these experiments because of the compositional nature of the tasks.
see, our performance is
Note that results in Table show compositionality performance on the Al dataset as we train with 0.1%. 10% of the data. When we evalu- ate on the gym environment generat- ing any possible language instruction from the Baby Al grammar, we may come across several unseen instructions at test time. To give a sense of the % of unseen instructions for BabyAl Al when we evaluate the gym environment, we take the different BabyAl environments and report the % of unseen language instructions seen at test time for different training data regimes in Table 3. For each statistic, we sample 10,000 random instructions from the environment and check how many are unseen in the training dataset used, repeated over 3 different seeds.
Table 3: BabyAI: % of unseen instructions at test time for different
training regimes. Environment Ik trajs
10k trajs
GoToSeq 76%
63.8% 66%
48%
SynthSeq 76.3%
50.7%
BossLevel 77.3%
66.9%
51.3%
4.8 How does LISA compare to simply doing K-Means clustering on the language and state embeddings?
In LISA, the VQ approach can be seen as taking the concatenated language-state inputs and projecting it into learned embedding space. VQ here simply learns K embedding vectors that act as K cluster centers for the projected input vectors in this embedding space, and allows for differentiability, enabling learning through backpropagation. This is also similar to propotypical methods used for few-shot learning and allows for deep differentiation clustering, giving an intuition of why works.
To compare against k-means, we construct a simple unsupervised learning baseline that clusters trajectories in the training dataset using k-means. Specifically, in the Baby AI BossLeve environment using 1 training trajectories, we take all concatenated language-state vectors for all trajectories in the dataset and cluster them using k-means and use the assigned cluster centers as the skill codes. We then learn a policy using these skill codes to measure their efficacy and found that LISA has a performance of 49.1 + 2.4% and k-means has performance of 20.2 + 5.2% over 3 seeds.
Thus, we see that using the simple k-means skills is insufficient to learn a good policy to solve the BabyAl BossLevel task, the skills are not representative enough of the language instructions. A reason for this is that language and state vectors lie in different embedding spaces, and K-means based on euclidean distance is not optimal on the concatenated vectors.
Limitations and Future Work
We present LISA, a hierarchical imitation learning framework that can be used to learn interpretable skill abstractions from language-conditioned expert demonstrations. We showed that the skills are diverse and can be used to solve long-range language tasks and that our method outperforms a strong non-hierarchical bascline in the low-data regime.
10

---

However, there are several limitations to LISA and plenty of scope for future work. One limitation of LISA that there are several hyperparameters tune that may affect performance like the number of options and the horizon for each option. It certainly helps to have a good idea of the task to decide these hyperparameters even though the ablations show that the method fairly robust to these choices. Its also useful to learn the horizon for each skill by learning a termination condition and we leave this for future work.
Although our method has been evaluated on the language coniiiggn-odiiioeeedd imitation learning setting, its difficult modify this method to make it work for image goals or demos, and in the RL setting as well. Its interesting see if the vector quantization trick can be used to learn goal-conditioned skills in a more general framework.
A Acknowledgments and Disclosure of Funding
We are thankful to John Schulman, Chelsea Finn, Karol Hausman and Dilip Arumugam for initial discussions regarding our method, and to Suraj Nair for providing help with the LOReL baseline.
This research was supported in part by NSF (#1651565, #1522054, #1733686), ONR (N00014-19-1- 2145), AFOSR 1195550-19-1-0224 FLI and Samsung.
References
[1] Felix Agakov and David Barber. Variational information maximization for neural coding. In Nikhil Ranjan Pal, Nik Kasabov, Rajani K. Mudi, Srimanta Pal, and Swapan Kumar Parui, editors, Neural Information Processing, pages 543-548, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg. ISBN 978-3-540-30499. 9. 5
[2] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-ane aadion-and-lameee and-language navigation: Interpreting visually- visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
grounded
[3] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton van den Hengel. Vision-and aadoon-nn--aaaaann language navigation: Interpreting visually-grounded navigation instructions in real environments, 2018.
[4] Dilip Arumugam, Siddharth Karamcheti, Nakul Gopalan, Lawson Wong. and Stefanie Tellex. Accurately and efficiently interpreting human-robot instructions of varying gramularities. Robotics: Science and Systems XIII, Jul 2017. doi: 10.15607/rss 2017:///....17.iii.i.i URL http: http:://da. //tx//dxxx///oo// doi. hrgg/0..5607.. /RSS. 2017 XIII 056.006... 3
[5] Dilip Arumugam, Siddharth Karamcheti, Nakul Gopalan, Edward C. Williams, Mina Rhee, Lawson I.. Wong, and Stefanie Tellex. Grounding natural language instructions to semantic goal representations for abstraction and generalization. Auton. Robots, 43(2):449 468, feb 2019, ISSN 0929-5593. doi: 10.1000/s10511 10077//1511--08.. 14-018-9792-88 8. URL https: / //do:i.//////// http///i..og////////// 10. 1007 s1001/////11-//11 1118-979-88.
[6] Valts Blukis, Nataly Brukhim, Andrew Bennett, Ross A. Knepper, and Yoav Artzi. Following high-level navigation instructions on simulated quadcopter with imitation learning, 2018.
[7] Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In European Conference Computer Vision, pages 548-564. Springer, 2020. 5
[8] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, 00./2005.14666 2020. URL https: aarxiv org/ abs/2005. 14165. 8
[9] Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding, 2018. 3
[10] Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments, 2020. 3,7
11

---

[11] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement leamnng via sequence modeling, 2021. 6
[12] Valerie Chen, Abhinav Cupta, and Kenneth Marino. Ask your humans: Using human instructions improve generalization reinforcement learning, 2021.
[13] Maxime Chevalier- Boisserr, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency grounded language learning. 2019. 2, 7, 19
[14] Yan Duan, Marein Andrychowicz, Bradly Stadie, Openai Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbecl, and Wojeiech Zaremba. One-Shot imitation lcarning. In Guyon, V Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan, and R Garnett, editors, Advances in Neural Information Processing Systems 30, pages 1087-1098. Curran Associates, Inc., 2017. 2
[15] Benjamin Eysenhach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: learning skills without a reward function, 2018.
[16] Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning, 2017. 2
[17] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkparrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision-and-language navigation, 2018. 3
[18] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song. and Stefano Ermon. Iq-learn: Inverse soft-q leaming for imitation. In Thirty-Fifth Conference Neural Information Processing Systems, 2021. URL https:/ http//ppeeeeeeeeeeeeeeeeeeeeeeee net/
wwwpe////oouueee ht///oormmmm///// derddiidddoooooeeeee -ex--xtbbb....... 2
[19] R.M. Gray and D.L. Neuhoff. Quantization. IEEE Transactions Information Theory, 44(6):2325-2335) 1998. doi: 10. 10.404
[20] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang. Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris Apps, Demis Hassabis, and Phil Blunsom. Grounded language learning in a simulated 3d world, 2017.
[21] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. 2016.
[22] Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. Hierarchical decision making by generating and following natural language instructions, 2019. 3
[23] Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as abstraction for hierarchical deep reinforcement lcarning, 2019.
[24] Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale, 2021. 2
[25] Alexander Li, Carlos Florensa, Ignasi Clavera, and Picter Abbecl. Sub-policy adaptation for hicrarchical reinforcement learning, 2020. 3
[26] Yunzhu Li, Jiaming Song, and Stefano Ermon. demonstrations, 2017.
Infogail: Interpretable imitation learning from visual
[27] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language, 2019. 3
[28] Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems, 2021. URL attpp:://aat /ttp/////ivv//v///////////a/.a peexiv.orrge org/ brg/abs//a
3
[29] Corey Lynch, Mohi Khansari. Ted Xiao, Vikash Kumar. Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. Conference on Robot Learning (CoRL), 2019. URL https:
//arxiv. //arxiv.or//bbl abs///////9 01973.
[30] Corey Lynch, Mohi Khansari. Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. Conference on Robot Learning (CoRL),201 2019. URL https:
// /arxivv wtwaaiiior//aababaeeaaea
abs//903..11.. 01973.
12

---

[31] Chih- Yao Ma. Jiasen Lu, Zuxuan Wu, Ghassan AlRegib. Zsolt Kira, Richard Socher, and Caiming Xiong. Self-monitoring navigation agent via auxiliary progress estimation, 2019. 3
[32] Matt MacMah Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language. knowledge, and action in route instructions. In Proceedings of the s1 National Conference on Artificial Intelligence- Volume 2. AAA''6, page 1475-1482. AAAI Press, 2006. ISBN 9781577352815. 3
[33] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks, arXiv preprint arXiv:21
2021. 7
[34] Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning, 2017. 3
[35] Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Mapping instructions to actions in 3d environments with visual goal prediction, 2019. 3
[36] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforvement learning, 2018. 3
[37] Ashvin Nair, Vitchyr Pong. Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforce- ment learning with imagined goals, 2018. 2
[38] Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, and Chelsea Finn. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation, 2021. 2, 3, 7, 20
[39] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning, 2017. 3
[40] Chris Paxton, Yonatan Bisk, Jesse Thomason, Arunkumar Byravan, and Dieter Fox. Prospection: Inter- pretable plans from language by predicting the future, 2019. 3
[41] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neurul computation, 3(1))88-97, 97, 1991. ISSN 0899-76677 2
[42] Stéphane Ross and Drew Bagnell, Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 661-66666666 1668, 2010.
[43] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings the fourteenth international conference on artificial intelligence and statistics, pages 627-635, 2011.
[44] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of hert: smaller, faster, cheaper and lighter. 2020.
[45] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-awar unsupervised discovery of skills, 2020.
[46] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbch Mottaghi, Luke Zettlemoyer, and Dicter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks, 2020. 3,7
[47] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipula- tion, 2021. 2,3
[48] Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté. Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning, 2021. 7
[49] Simon Stepputtis. Joseph Campbell, Mariano Phielipp. Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-condittong imitation learning for robot manipulation tasks, 2020.
[50] Alane Suhr and Yoav Artzi, Situated mapping of sequential instructions to actions with single-step reward observation, 2018. 3
[51] Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. 1999. 3,5
[52] Hao Tan, Licheng Yu, and Mohit Bansal, Learning to navigate unseen environments: Back translation with environmental dropout, 2019. 3
13

---

[53] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. 2018. 5
[54] Jesse Vig. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting the Association for Computational Linguistics: System Demonstrations, pages 37-42. Florence, Italy. July 2019. Association for Computational Linguistics. doi: 10.18653/1//119-3007 URL
https://www.r aclweb. org/ anthology/P19 3007. 2
[55] Sida Wang, Percy Liang, and Christopher Manning. Learning language games through interaction, 2016. 3
[56] Danfei Xu, Suraj Nair. Yuke Zhu, Julian Gao, Animesh Garg, Fei-Fei, and Silvio Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 JEEE International Conference on Robotics and Automation (ICRA), pages 3795-3802. IEEE, 2018.
[57] Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via Nomain-adaptive meta-learning, 2018. 2
[58] Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options, 2021. 3
14


Reviews:
[{"review_id": "qwGW-RBgyhq", "reviewer": "Reviewer_k4fJ", "summary": "***Post rebuttal update: Raising score from 5 to 6***\n\n\nThis work proposes a method to learn a hierarchical instruction following agent policy from demonstrations.The policy is composed of a skill predictor which plays the role of a high-level controller, and a low level controller predicts actions conditioned on the skill. The skills are trained from scratch with a vector quantization approach; the skills are represented by codes in a codebook and the skill predictor chooses the skill to execute based on a language instruction and a history of observations. The proposed method is shown to perform well on BabyAI and Lorel Sawyer benchmarks and outperforms baseline non-hierarchical policies on long horizon tasks, especially when supervision is limited. In addition, the authors show that the learned skill vectors are interpretable and correspond to meaningful primitive skills. ", "questions": "* How is the MI computed in Fig 6?\n* Have you considered alternative approaches where you simply segment the trajectories and cluster them to identify salient skills? I believe this could be a strong baseline. Can you provide some intuition to why the vector quantization approach would be preferred over this approach?  \n\nPresentation\n* 132: “task can be a union of other tasks” - this is not clear to me\n* 136: “sub-goals for the task are encoded within this single instruction” - didn’t understand this\n* 217: “finetune the language encoder to the vocabulary” - what does this mean? \n* There are mentions of ‘GIF’ in the paper. Good to rephrase. \n", "limitations": "Seems adequate. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Learning reusable skills from data is an important problem with potential for better generalization, interpretability and controllability. \n* Proposed approach outperforms baseline approaches when supervision is limited. \n* Authors show that the learned skills are interpretable, meaningful and lead to better generalization on long-horizon tasks. \n\nWeaknesses\n* The precise source of performance is hard to pinpoint. Does LISA work better than the flat baseline because it has more parameters? Were hyperparameters of baseline approaches carefully tuned? The authors can try to better isolate the source of performance. \n* The proposed approach has many hyperparameters to be tuned such as codebook size, horizon, etc. I could not find discussion about how these parameters were chosen. \n* The fixed horizon assumption is a limitation. Have the authors examined how the choice of this parameter affects the results? \n\nThe proposed approach is interesting and shows positive results on two benchmarks. But the approach has key limitations which needs to be better addressed in the paper. This includes discussion about hyperparameter choices and how they affect the performance and more clarity on how the skills are interpreted. Based on the quantitative results presented on the paper alone it is hard to gauge the proposed approach due to confounding factors such as hyperparameter optimization and parameter count. Addressing these issues can significantly improve the paper. \n\nI found the discussion in sec 4.7 interesting. The proposed approach can potentially work very well on compositional generalization. Expanding the scope of the analysis (Eg. defining a clear evaluation setup, experimenting on a large number of tasks, etc.) in sec 4.7 can significantly strengthen the paper and become a core contribution.\n\nOther\n* I couldn’t find enough details in the paper to understand how the correlation between skills and words is computed and how it is visualized. \n* Algorithm 1: line 12 needs to be described better\n* Plots are hard to visualize due to small font sizes (fig 4, fig 6). Also, there are layout issues that need to be fixed (Unclear why Fig 6 appears in p8)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cwAYcZfUx0X", "reviewer": "Reviewer_txnz", "summary": "The submission presents a method, LISA, a hierarchical imitation learning framework that train a language-conditioned agent with an end-to-end fashion. LISA learns a skill abstraction through a skill predictor module and vector quantization conditioned on language input and observations, a policy module is learned conditioned on the skill abstraction. LISA is trained with behavior-cloning and vector quantization objectives. LISA's performance is evaluated on two tasks, BabyAI, LOReL; some baselines are created by authors and from original task paper. Comparing with baselines, the results demonstrate good sample efficiency, possibility for perform unseen composition task and good interpretability from learned skill code.", "questions": "1. line 222: I am confused why use a causal mask for hiding future observation since skill predictor only takes previous states and language as input according to line 164.\n2.  line 257 and line 267 describing generalization and compositionally related claims, but how likely the evaluation set is unseen is not quantified.\n3.  The composition task dataset appears to be small, is a larger version available at this time?", "limitations": "The hyperparameter limitations are included in the paper. The interpretation ability of skill code is limited and relies on problem setting.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\nIt is a recombination of existing techniques, where vector quantization is used in representation learning and behavior cloning is also not new. Nonetheless, it is the first work that use a combination that learn a quantized abstract from language and state, and develop a policy based on the produced skill abstract. \n### Quality\nThe technical details are provided in the writing or references are given. Most experiment result and analysis are well supported. I believe the claims made in the contribution part are mostly fulfilled. The *weakness* of the paper are: the number of skill code is task dependent, the horizon of each skill has to be fixed, and chosen ahead; (the weakness aforementioned are mentioned in the writing). The conclusion related to compositionally is less convincing for me as I posted in question section below. The skill codes interpretation make more sense for LOReL than BabyAI due to larger of vocabulary size, thus, there are limitations on how we could use this interpretation.\n### Clarity\nThe writing is mostly clear, but there are errors in table name (eg Table 10 referred multiple times but it should be table 1). Also table and figure are often too far from the description (Fig 3, Tab 1); line 173, does the policy module take input of both state and skill code or only skill code? The two options for LOReL task in table 1 is not explained in the paper.\n### Significance\nLISA presents a new hierarchical imitation learning framework that utilize vector quantization to learn skill abstraction and further build a policy module on top of skill abstractions. LISA is sample efficient and is useful for low-data regime. The learned skill codes are interpretable on limited conditions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Py4pWQv3Lm1", "reviewer": "Reviewer_f8PZ", "summary": "This paper proposes a hierarchical model for instruction-conditioned imitation learning (instruction following). A skill predictor network conditions on instructions and observations to predict discrete skill codes, and a policy network conditions on these skill codes to carry out actions. Discrete skill codes are learned using vector quantization. The paper evaluates on BabyAI and LOReL imitation learning tasks, finding that (1) the method outperforms a non-hierarchical imitation learning baseline model in low-resource training settings and when carrying out complex compositional tasks and (2) discrete skills induced are correlated with words in the language.", "questions": "Q1) Could more details be clarified on the difference between instructions seen in training and in evaluation for both BabyAI and LOReL, especially in the \"new composition\" settings?\n\nQ2) Is 4.7, is the time horizon increased at inference time only or just at training time? It's also unclear to me why H would need to be increased, since if I understand correctly it's the length of time each skill is invoked for and it seems that while the task contains more skills in this setting, skills themselves need not be longer.\n", "limitations": "I appreciated the discussion of the option-related limitations: needing to fix the horizon, and number of options. It would help to more clearly outline how the values for these were chosen in this work.\n\n*Suggestions*\n\n- I think it would really strengthen the paper to either \n\n(1) extend the \"manual planning\" experiments given in the Appendix to show that control codes can be intervened on to make the models carry out behavior in novel contexts where the behaviors weren't seen in training or \n\n(2) show that the models can carry out unseen compositions of sub-tasks (i.e. similar to the compositional generalization setting explored in Corona et al.)\n\n- The claims in 294 and 302-305 about sub-task reuse, and skill codes capturing task variation were pretty abstract and it would really strengthen the paper if some concrete evidence for them could be presented.\n\n*Minor comments*\n\n- line 105: Hu et al. does learn interpretable skills - language descriptions of skills are produced by their instructor model.\n- 145: the description here could be a bit more rigorous. Is there mean to be a latent partitioning of each trajectory into steps for each sub-task, or can steps be associated with multiple sub-tasks in some soft way?\n- 171: it would help to give more details on how H is set, perhaps with references to the values & ablations in the appendix.\n- 181: this equation above doesn't describe the learning process for the cluster centers.\n- 198: this is not obvious to me; a citation would help.\n- 198: fragmented sentences\n- 242: \"off\" -> \"of\"\n\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\nS1) The approach in the paper is well-motivated and is likely to be of interest both to the sequential decision making and language grounding communities. Most past work on hierarchical language-conditioned imitation learning has relied on supervising the intermediate layer of the hierarchy, so an end-to-end approach is exciting.\n\nS2) The method is generally well-described, although some details were only clear from the appendix.\n\nS3) I appreciate that the paper evaluated on two different datasets, and presented pretty thorough ablations and qualitative analysis (in the appendix). \n\n*Weaknesses*\n\nW1) The main novelty of the paper is the learning of discrete skills from language & demonstrations in an end-to-end fashion -- without the discrete skill bottleneck, it seems that the LISA approach would be an end-to-end neural instruction following model of the type very common in work on language grounding (e.g. Mei et al 2016, Anderson et al. 2018), and presented as an ablation in appendix F.2. ~~But the paper doesn't currently have enough evidence to convince me that these discrete skills are useful:~~\n\nW1a) ~~While LISA does outperform a baseline approach (the \"flat\" model), this baseline has a pretty different architecture and initialization (in particular, LISA uses a pre-trained DistillBERT text encoder while the baseline model is trained from scratch) and so I think is much less suitable than as a baseline than the ablation in appendix F.2, which performs comparably to LISA. So I don't think it's reasonable to claim performance improvements from the LISA approach based on the current results.~~\n\n_Update after response:_ Thanks to the authors for the baseline clarifications; the baseline is much more comparable to LISA than I'd thought. \n\nW1b) It's unclear to me that the use of discrete codes adds interpretability. The main interpretation method (word clouds and heatmaps) maps these skills to words in the existing instructions. Similar methods can also be applied to continuous representations (e.g. Andreas et al., Translating Neuralese). In the skill code fixing example (4.5), it seems plausible that the method is just carrying out \"close the drawer\" given the affordance of the environment: the arm is located next to an open drawer. See suggestions, in the limitations section, below for suggestions on this. \n\n_Update after response:_ I appreciate the author's arguments about the easy of interpretability, and the additional experiments showing that skill codes are being used and it's not just environment affordances.\n\nW2) Although not a crucial weakness, since this paper does contribute beyond them, some relevant works are\n\nSharma et al. Skill Induction and Planning with Latent Language ACL 2022\n\nCorona et al. Modular Networks for Compositional Instruction Following NAACL 2021\n\nW3) Some of the experimental descriptions could be clearer, in particular whether the paper is interested in evaluating generalization to instructions unseen at training time; see questions below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4BRvIbQ-sSW", "reviewer": "Reviewer_qGvo", "summary": "The paper proposes a hierarchical imitation learning where language instruction is used as input and converted to a latent code. This latent code is then used to condition the policy and guide the policy to the target.", "questions": "The approach seems to be a simple extension to various works in unsupervised skill discovery methods such as DIAYN, RVIC, etc. The difference being here the latent codes are not randomly initialized but instead are learned through VQVAE. I'm curious though how would LISA perform if all the instructions and the resulting codes are given at once and not sampled every H steps. Is it the sampling every H steps that's important to the resulting behavior?\n\nIs the choice of using VQVAE to convert to discrete codes arbitrary? How about using approximate backpropagation techniques such as Gumbel-Softmax? Would this change the result?\n\nDid the authors look into the effective number of skills used? This helps in highlighting that all codes in the codebook are indeed useful and has some relationship with some linguistic object.", "limitations": "The authors discuss the limitations of the work in the last section. Besides tuning hyperparameters, is there any limitation on the length of the language instruction used? How does the approach scale when we increase the number of skills in the codebook for the same task?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is nicely written and the main arguments are clearly explained. The figures although are a bit harder to discern due to the smaller size. The proposed approach, LISA, is one of the first imitation learning methods that employs learning low-level skills from a set of instructions.\n\nThe results show that the agents are able to outperform the baselines and the word clouds depict the different skills associated with different latent variables.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept