Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a hierarchical imitation learning where language instruction is used as input and converted to a latent code. This latent code is then used to condition the policy and guide the policy to the target.
strengths_and_weaknesses: The paper is nicely written and the main arguments are clearly explained. The figures although are a bit harder to discern due to the smaller size. The proposed approach, LISA, is one of the first imitation learning methods that employs learning low-level skills from a set of instructions.

The results show that the agents are able to outperform the baselines and the word clouds depict the different skills associated with different latent variables.
questions: The approach seems to be a simple extension to various works in unsupervised skill discovery methods such as DIAYN, RVIC, etc. The difference being here the latent codes are not randomly initialized but instead are learned through VQVAE. I'm curious though how would LISA perform if all the instructions and the resulting codes are given at once and not sampled every H steps. Is it the sampling every H steps that's important to the resulting behavior?

Is the choice of using VQVAE to convert to discrete codes arbitrary? How about using approximate backpropagation techniques such as Gumbel-Softmax? Would this change the result?

Did the authors look into the effective number of skills used? This helps in highlighting that all codes in the codebook are indeed useful and has some relationship with some linguistic object.
limitations: The authors discuss the limitations of the work in the last section. Besides tuning hyperparameters, is there any limitation on the length of the language instruction used? How does the approach scale when we increase the number of skills in the codebook for the same task?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The submission presents a method, LISA, a hierarchical imitation learning framework that train a language-conditioned agent with an end-to-end fashion. LISA learns a skill abstraction through a skill predictor module and vector quantization conditioned on language input and observations, a policy module is learned conditioned on the skill abstraction. LISA is trained with behavior-cloning and vector quantization objectives. LISA's performance is evaluated on two tasks, BabyAI, LOReL; some baselines are created by authors and from original task paper. Comparing with baselines, the results demonstrate good sample efficiency, possibility for perform unseen composition task and good interpretability from learned skill code.
strengths_and_weaknesses: ### Originality
It is a recombination of existing techniques, where vector quantization is used in representation learning and behavior cloning is also not new. Nonetheless, it is the first work that use a combination that learn a quantized abstract from language and state, and develop a policy based on the produced skill abstract. 
### Quality
The technical details are provided in the writing or references are given. Most experiment result and analysis are well supported. I believe the claims made in the contribution part are mostly fulfilled. The *weakness* of the paper are: the number of skill code is task dependent, the horizon of each skill has to be fixed, and chosen ahead; (the weakness aforementioned are mentioned in the writing). The conclusion related to compositionally is less convincing for me as I posted in question section below. The skill codes interpretation make more sense for LOReL than BabyAI due to larger of vocabulary size, thus, there are limitations on how we could use this interpretation.
### Clarity
The writing is mostly clear, but there are errors in table name (eg Table 10 referred multiple times but it should be table 1). Also table and figure are often too far from the description (Fig 3, Tab 1); line 173, does the policy module take input of both state and skill code or only skill code? The two options for LOReL task in table 1 is not explained in the paper.
### Significance
LISA presents a new hierarchical imitation learning framework that utilize vector quantization to learn skill abstraction and further build a policy module on top of skill abstractions. LISA is sample efficient and is useful for low-data regime. The learned skill codes are interpretable on limited conditions.
questions: 1. line 222: I am confused why use a causal mask for hiding future observation since skill predictor only takes previous states and language as input according to line 164.
2.  line 257 and line 267 describing generalization and compositionally related claims, but how likely the evaluation set is unseen is not quantified.
3.  The composition task dataset appears to be small, is a larger version available at this time?
limitations: The hyperparameter limitations are included in the paper. The interpretation ability of skill code is limited and relies on problem setting.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: ***Post rebuttal update: Raising score from 5 to 6***


This work proposes a method to learn a hierarchical instruction following agent policy from demonstrations.The policy is composed of a skill predictor which plays the role of a high-level controller, and a low level controller predicts actions conditioned on the skill. The skills are trained from scratch with a vector quantization approach; the skills are represented by codes in a codebook and the skill predictor chooses the skill to execute based on a language instruction and a history of observations. The proposed method is shown to perform well on BabyAI and Lorel Sawyer benchmarks and outperforms baseline non-hierarchical policies on long horizon tasks, especially when supervision is limited. In addition, the authors show that the learned skill vectors are interpretable and correspond to meaningful primitive skills. 
strengths_and_weaknesses: Strengths
* Learning reusable skills from data is an important problem with potential for better generalization, interpretability and controllability. 
* Proposed approach outperforms baseline approaches when supervision is limited. 
* Authors show that the learned skills are interpretable, meaningful and lead to better generalization on long-horizon tasks. 

Weaknesses
* The precise source of performance is hard to pinpoint. Does LISA work better than the flat baseline because it has more parameters? Were hyperparameters of baseline approaches carefully tuned? The authors can try to better isolate the source of performance. 
* The proposed approach has many hyperparameters to be tuned such as codebook size, horizon, etc. I could not find discussion about how these parameters were chosen. 
* The fixed horizon assumption is a limitation. Have the authors examined how the choice of this parameter affects the results? 

The proposed approach is interesting and shows positive results on two benchmarks. But the approach has key limitations which needs to be better addressed in the paper. This includes discussion about hyperparameter choices and how they affect the performance and more clarity on how the skills are interpreted. Based on the quantitative results presented on the paper alone it is hard to gauge the proposed approach due to confounding factors such as hyperparameter optimization and parameter count. Addressing these issues can significantly improve the paper. 

I found the discussion in sec 4.7 interesting. The proposed approach can potentially work very well on compositional generalization. Expanding the scope of the analysis (Eg. defining a clear evaluation setup, experimenting on a large number of tasks, etc.) in sec 4.7 can significantly strengthen the paper and become a core contribution.

Other
* I couldn’t find enough details in the paper to understand how the correlation between skills and words is computed and how it is visualized. 
* Algorithm 1: line 12 needs to be described better
* Plots are hard to visualize due to small font sizes (fig 4, fig 6). Also, there are layout issues that need to be fixed (Unclear why Fig 6 appears in p8)
questions: * How is the MI computed in Fig 6?
* Have you considered alternative approaches where you simply segment the trajectories and cluster them to identify salient skills? I believe this could be a strong baseline. Can you provide some intuition to why the vector quantization approach would be preferred over this approach?  

Presentation
* 132: “task can be a union of other tasks” - this is not clear to me
* 136: “sub-goals for the task are encoded within this single instruction” - didn’t understand this
* 217: “finetune the language encoder to the vocabulary” - what does this mean? 
* There are mentions of ‘GIF’ in the paper. Good to rephrase. 

limitations: Seems adequate.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a hierarchical model for instruction-conditioned imitation learning (instruction following). A skill predictor network conditions on instructions and observations to predict discrete skill codes, and a policy network conditions on these skill codes to carry out actions. Discrete skill codes are learned using vector quantization. The paper evaluates on BabyAI and LOReL imitation learning tasks, finding that (1) the method outperforms a non-hierarchical imitation learning baseline model in low-resource training settings and when carrying out complex compositional tasks and (2) discrete skills induced are correlated with words in the language.
strengths_and_weaknesses: *Strengths*

S1) The approach in the paper is well-motivated and is likely to be of interest both to the sequential decision making and language grounding communities. Most past work on hierarchical language-conditioned imitation learning has relied on supervising the intermediate layer of the hierarchy, so an end-to-end approach is exciting.

S2) The method is generally well-described, although some details were only clear from the appendix.

S3) I appreciate that the paper evaluated on two different datasets, and presented pretty thorough ablations and qualitative analysis (in the appendix). 

*Weaknesses*

W1) The main novelty of the paper is the learning of discrete skills from language & demonstrations in an end-to-end fashion -- without the discrete skill bottleneck, it seems that the LISA approach would be an end-to-end neural instruction following model of the type very common in work on language grounding (e.g. Mei et al 2016, Anderson et al. 2018), and presented as an ablation in appendix F.2. ~~But the paper doesn't currently have enough evidence to convince me that these discrete skills are useful:~~

W1a) ~~While LISA does outperform a baseline approach (the "flat" model), this baseline has a pretty different architecture and initialization (in particular, LISA uses a pre-trained DistillBERT text encoder while the baseline model is trained from scratch) and so I think is much less suitable than as a baseline than the ablation in appendix F.2, which performs comparably to LISA. So I don't think it's reasonable to claim performance improvements from the LISA approach based on the current results.~~

_Update after response:_ Thanks to the authors for the baseline clarifications; the baseline is much more comparable to LISA than I'd thought. 

W1b) It's unclear to me that the use of discrete codes adds interpretability. The main interpretation method (word clouds and heatmaps) maps these skills to words in the existing instructions. Similar methods can also be applied to continuous representations (e.g. Andreas et al., Translating Neuralese). In the skill code fixing example (4.5), it seems plausible that the method is just carrying out "close the drawer" given the affordance of the environment: the arm is located next to an open drawer. See suggestions, in the limitations section, below for suggestions on this. 

_Update after response:_ I appreciate the author's arguments about the easy of interpretability, and the additional experiments showing that skill codes are being used and it's not just environment affordances.

W2) Although not a crucial weakness, since this paper does contribute beyond them, some relevant works are

Sharma et al. Skill Induction and Planning with Latent Language ACL 2022

Corona et al. Modular Networks for Compositional Instruction Following NAACL 2021

W3) Some of the experimental descriptions could be clearer, in particular whether the paper is interested in evaluating generalization to instructions unseen at training time; see questions below.

questions: Q1) Could more details be clarified on the difference between instructions seen in training and in evaluation for both BabyAI and LOReL, especially in the "new composition" settings?

Q2) Is 4.7, is the time horizon increased at inference time only or just at training time? It's also unclear to me why H would need to be increased, since if I understand correctly it's the length of time each skill is invoked for and it seems that while the task contains more skills in this setting, skills themselves need not be longer.

limitations: I appreciated the discussion of the option-related limitations: needing to fix the horizon, and number of options. It would help to more clearly outline how the values for these were chosen in this work.

*Suggestions*

- I think it would really strengthen the paper to either 

(1) extend the "manual planning" experiments given in the Appendix to show that control codes can be intervened on to make the models carry out behavior in novel contexts where the behaviors weren't seen in training or 

(2) show that the models can carry out unseen compositions of sub-tasks (i.e. similar to the compositional generalization setting explored in Corona et al.)

- The claims in 294 and 302-305 about sub-task reuse, and skill codes capturing task variation were pretty abstract and it would really strengthen the paper if some concrete evidence for them could be presented.

*Minor comments*

- line 105: Hu et al. does learn interpretable skills - language descriptions of skills are produced by their instructor model.
- 145: the description here could be a bit more rigorous. Is there mean to be a latent partitioning of each trajectory into steps for each sub-task, or can steps be associated with multiple sub-tasks in some soft way?
- 171: it would help to give more details on how H is set, perhaps with references to the values & ablations in the appendix.
- 181: this equation above doesn't describe the learning process for the cluster centers.
- 198: this is not obvious to me; a citation would help.
- 198: fragmented sentences
- 242: "off" -> "of"
review_title: null

Meta-review: metareview: This paper provides a hierarchical approach to imitation learning guided by language, which the reviewers unanimously found empirically compelling, as well as of potential interest to the broader NeurIPS community. I am happy to go with the consensus and recommend acceptance.
confidence: Certain
Decision: Accept