Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper takes advantage of the pre-trained large language models (LLMs) and assesses whether they can be helpful for embodied agents. The authors choose the VirtualHome environment as the testbed and prompt the LLMs to decompose a high-level instruction to detailed, actionable instructions. Human evaluation is conducted to evaluate the correctness and executability of the generated instructions. 
main_review: Strengths:
- LLMs are powerful tools and learn knowledge that we may even not know. So prompting them to utilize the learned knowledge is a promising research direction. Prompting LLMs for actionable knowledge extraction is new and interesting. 
- Different LLMs including GPT-3 and Codex are used and evaluated here.  
- The prompt and translation method is reasonable on the VirtualHome environment.

Weaknesses:
- The paper's claim is to extract actionable knowledge for embodied agents, but in fact, it seems the translation process is particularly tailored for VirtualHome. I couldn't see a clear way to generalize to other embodied environments. It would be great if the authors can show its generalization to other environments such as Room2Room and ALFRED. I am not sure if the translation part is really necessary in other environments.
- Moreover, it seems there is no embodied agent to execute those instructions. All experiments are conducted on the text level only, but we all know that there is a huge gap between text instructions and actual actions taken by the agent. So it would be much more convincing if the authors can show an embodied agent can actually follow the generated instructions and complete tasks in the environment. 
- There seems to be a tradeoff between executability and correctness for the current approach. The correctness is significantly dropped from 56% to 34% for Translated Codex 12B. Why are some correctness numbers missing in Table 1? What is the correctness of Translated GPT-3 175B? The Correctness metric is also missing in Table 2 and Table 3. Why? 
- It is actually pretty easy to improve the executability by finding the most similar actions from the action set of the environment. The real challenge is to make the actions more executable while preserving their correctness, which, however, isn't achieved in this work and it quite a pity. Without considering the correctness, executability does not really matter much, for an extreme example,  a random agent may achieve  high executability but low correctness. 
- Many technical details are missing in the paper, for example, 
  - It is unclear how Executability is calculated. There is only high-level language description in the paper. Can you elaborate it? 
  - It is unclear how the sampling for LMs is done. Do you sample multiple ones and then rely on human evaluation to select the best one? This doesn't sound right and scalable. 
  - Section 2.1: Appendix ?? points to nowhere. Similar issues exist in many places of the paper. 
  - "" in the paper is in the wrong format. 

summary_of_the_review: I like the idea of prompting LLMs to decompose high-level instructions and extract actionable knowledge, which is a novel perspective for embodied agents. However, the execution of the idea and the experiments are not right there to support the claim. It would be great if the authors can show its effectiveness on "actual" embodied agents in different environments instead of text games.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes using pretrained language models for planning actions. They achieve this by providing an example task sequence as a prompt to generate an action description for each step autoregressively. At each step, they translate the action description to the environment action by computing the similarity between the description and the available environment actions. The results show that there is a trade-off between executability and correctness. 
main_review: This paper shows that the pretrained language model contains knowledge that is useful for action planning. While this is a promising direction to use pretrained language model, I have several concerns and questions for this paper. 

1. Grounding & planning vs. knowledge extraction. When we do grounding or planning, the agent’s actions may change given the environment they are put in. The proposed approach is closer to procedure knowledge extraction using language models rather than planning or grounding. While the available actions in VirtualHome might be different from other virtual environments (therefore, this paper proposed a translation step), the generated action plans don’t change if we change the environment from one home to another. So, the generated plans are not really grounded in the domain or environment the agent is planning for. The authors may consider repositioning this paper to knowledge extraction papers that fit the contribution better.

2. Choice of VirutalHome task for planning. The tasks are mainly around the daily home tasks like setting up a dinner table. To perform such tasks, most of the time, the agent may just apply predefined procedure knowledge. However, procedure knowledge and the translation approach proposed by this paper cannot plan for the task like “stack two plates on the right of a cup” because the language model doesn’t ground its knowledge to the environment so the agent cannot tell the difference between putting down on the left or right of the cup. This is the main limitation to using the proposed approach as a planner.

3. Evaluating correctness and executability separately. The goal of task planning is to find the plan that is executable and correct, i.e. success rate. However, most of the tables in the evaluation only show one or another. Only Table 4 shows “# of C and E”. If we compute the success rate for Translated Codex 12B, it is 15/88=0.17 and would be lower for other models. This reflects the performance of the proposed method better and the overall success rate is quite low. 

4. Missing related work. This paper missed some related work on knowledge extraction/mining using pertained language models. Just to list a few below. The authors will need to discuss how the proposed approach is different from other knowledge mining approaches compared to prior work.
 - Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A.H. and Riedel, S. “Language models as knowledge bases?” In EMNLP 2019.
 - Davison, Joe, Joshua Feldman, and Alexander M. Rush. "Commonsense knowledge mining from pretrained models." In EMNLP 2019. 
 - Jiang, Z., Xu, F.F., Araki, J. and Neubig, G. “How can we know what language models know?”. Transactions of the Association for Computational Linguistics, 8, pp.423-438.

summary_of_the_review: This paper shows promising results on extracting executable actions using pretrained language model. However, there are major concerns about how this relates and contributes to planning and grounding.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates whether large language models (LLMs) are capable, without additional training, of decomposing high-level tasks into a sequence of instructions (i.e., a plan) and grounding them in an embodied environment. Specifically, they relied on prompt engineering to provide enough context to the LLMs for generating sensible instructions. The authors also suggest using another language model to translate generated instructions into actions that are parsable by the embodied environment.

Through a series of experiments, the authors empirically show that larger LLMs are able to produce sensible plans (according to human annotators) that can also be mapped to executable actions within an embodied environment. In their experiments, the authors evaluate GPT-2, GPT-3, and Codex (including different sizes for the GPT-* family). Part of the paper is also dedicated to answering several hypotheses related to the investigation.
main_review: ## What I like about this paper

- Rather than trying to solve a particular task using large language models, this paper focuses on analyzing what is the capability of pre-trained language models in terms of decomposing high-level tasks into a sequence of instructions. In other words, what kind of actionable knowledge, if any, is stored in those large language models?

- Using two axes of evaluation - executability, and correctness - better characterize the strengths and limitations of the language models studied in this paper. Looking at the discrepancy between the correctness, as reported by the human annotators, and the executability of a plan motivates the need for a translation LM. It can overcome some of the restrictions interactive environments have, i.e. their action space contains instructions with a specific syntax.

- I like the suggestion of using another language to translate generated instructions to grounded actions to boost executability. Also, it makes sense to me to use an autoregressive trajectory correction approach to improve the executability of a plan.

## Concerns

- My main concern lies with the type of language models used in this analysis. Other than being really popular, it is not clear why the authors decided to only investigate models in the GPT-* family? Is it that other large language models (e.g., T5) are less amenable to prompt-engineering? To me, the main message/claims of the paper seem to be about large language models in general. It would be interesting to know if actionable knowledge is only present in GPT-* models or not.


-----
### Typos
- p.7: "We find that despite [being] more similar ..." - Missing word.
- p.7: "...; we find that for [a] many tasks we ..." - Out-of-place word.
- p.7: "... by [our -> the] human annotators" - Suggested change.
summary_of_the_review: In addition to their impressive performance on several standard NLP tasks, large language models have been found useful for a wide variety of tasks when prompted appropriately. I believe understanding, via investigative work, what is the extent of the knowledge contained in those models to be important as it can better guide the research community towards what to explore next. The proposed paper is exactly about doing such an investigation. I found the paper well-motivated, and it includes the experiments to support the authors' analysis and discussion. For those reasons, I recommend this paper be accepted at ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores the ability of pre-trained language models to generate plans or action sequences from a text instruction. The in-context learning ability of language models is used where the model is prompted with an example instruction and corresponding action sequence and the query instruction. Since text sequences generated by the language model may not be directly usable in the agent environment, the closest valid text actions are identified using a retrieval approach. Experiments compare performance of different language models on tasks from the VirtualHome benchmark. 
main_review: After rebuttal: I have read the author response. Some of my comments regarding the dataset and evaluation were partially addressed, so I am raising my score. However, I feel ambivalent about the paper due to concerns about the evaluation (See detailed comments in discussion thread). 

====

Pros
* Using language models to infer actionable plans from text instructions is interesting
* Paper is generally easy to follow

Cons
* The paper doesn’t provide sufficient details about the data, the train and test splits, and the challenging aspects of the task.
* The pipelined approach considered here seems fairly limited
* Limited technical novelty
* Weak experiments

The paper should provide more details about the data. For instance, how similar are the train and test tasks? I would encourage the authors to provide a summary of the tasks. How was the 204/88 task split decided?

The metrics considered in the paper are less ideal and do not provide a holistic view of model performance. For instance, 
* Executability doesn’t consider whether predicted actions are relevant to the task
* LCS penalizes actions that don’t appear in ground truth
* Correctness: Why is the correctness of GT plans only 55%? This makes it very hard to interpret these results and I am not sure how meaningful the results are. 
Furthermore, it’s hard to say if the Translated variants in Table 1 are better than the plain language models. While these models are better in terms of executability and LCS, they are inferior in terms of correctness.

How were the human annotators instructed to label model generated plans? 

Is it possible to define a straightforward metric that identifies whether a given task was successfully completed or not?

The proposed approach should be compared against other baselines. For instance, a fine-tuning baseline can be considered where the model translates instructions to plans. 

I would encourage the authors to analyze and report model prediction errors.

Links to appendix are broken.

summary_of_the_review: While the authors explore an interesting approach to planning based on language models, the paper is weak in terms of technical novelty and experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies left-to-right language models for on the VirtualHome planning task. In this task, a model is given a high level goal like "get glass of milk" and it must generate a sequence of lower-level steps to execute that goal ('walk to kitchen', 'open fridge', etc.). These steps correspond to the annotated scratch programs built by crowd workers as part of VirtualHome, so as such they are executable plans to achieve the given goal.

This paper uses a left-to-right LM to generate those lower-level steps, given one prompted example. The 1.5B-sized LMs do not seem to do very well at generating actions given this prompt. The larger 12B, etc. LMs do better, but frequently deviate from the set of valid actions, so the authors introduce a Translation LM that measures the similarity between any "action" that the LM generates and the set of valid actions. At each step, this translation LM is used in the loop to constrain the chosen action, and this improves executability significantly (e.g. 7% -> 73% for GPT-3 and 18% -> 78% for Codex). Helpful analysis breaks down this performance.
main_review: Strengths:
* The paper studies an idea that seems relatively novel to this reviewer -- how well can models generate action plans in an embodied environment (not just code, or language)?
* The paper presents a simple idea -- using a translation LM to constrain the generated actions -- that greatly improves performance, by ensuring that the actions generated by the LM are valid within the environment.
* The paper has results that consider LMs of a variety of different sizes, from 1.5B to 175B, including both GPT-3 and codex families. The results seem to suggest that size is important, at least in this setting, insofar as the smaller models just copy from the input prompt.
* This paper presents analysis about the kinds of programs generated by LMs which I think might help future work build off of this direction.

Weaknesses:
* The experimental setup proposed by this paper for measuring grounding might not be ideal. In the VirtualHome setup, the action space is composed of mid-level actions (like "walk to home office") versus low-level actions ("walk forward 1m" ,etc), so the promise of LMs for embodied understanding might not generalize to more difficult environents (like Alfred; Shridhar et al 2020a). That said, I think this limitation is discussed well in the limitations section (Sec7).
* Though I appreciate the authors' work on making the evaluation robust, it is not clear to this reviewer that it is a good measure of correctness. To the best of this reviewer's understanding (of Section 2.2; Correctness), in the VirtualHome environment that the authors study, it is nontrivial to compare whether the final environment state of a generated program is similar to the "ground truth" environment state. The details of the human evaluation are a bit unclear, but it seems like humans are just given the list of actions rather than the world state after executing those actions. In this setup, it seems like there is a very strong bias towards language that 'looks like' real action plans, rather than language that is truly actionable. The result is that models outperform humans at the task. To this reviewer, the paper could be improved if the correctness measured something grounded, rather than just something that looks grounded.


Clarification:
* I'd appreciate more details on the human evaluation somewhere (like the interface being shown to the turkers)
summary_of_the_review: Overall, my opinions on this paper are fairly positive, because it tackles an interesting new direction with new simple methodology for generating actions from LMs. However, there are some issues with evaluation, that could greatly improve the paper if addressed (or at the least, discussed a bit more in the limitations section). 

--

Update post review: I am lowering my score from 8 -> 6. I still think this is a good paper and it should be accepted, but I am not very convinced by the evaluation. I think the evaluation issues I (and KhzN) have issues with could be improved between acceptance and publication though.
review_title: null

Meta-review: This manuscript presents a method to refine high-level task descriptions into mid-level executable steps. The idea of using language models to generate steps for a robot to follow is very interesting. Reviewer concerns focused on the general applicability of the approach and the evaluation.

Reviewers pointed out that the method is tied to VirtualHome which has various properties that are in general not true: the action space is small, the action space is very sparse, and objects tend to be unique.

First, the method enumerates a sentence for every possible action and object combination in the environment. The fact that VirtualHome has few verbs and few objects and that neither of these has complex additional structure (adjectives, adverbs, etc.) means that this is practical. But in any other practical setting this will be impossible. The manuscript mentions this limitation and hints at possible ways to resolve it.

Second, the method requires that the action space must be incredibly sparse. Moreover, a set of common sense rules are needed which are environment specific and must be hand curated. VirtualHome disallows microwaving a cup for example. It also disallows opening the TV. Both of these are valid actions that happen all the time.

Third, the method requires that objects be unique. If multiple plates, vacuum cleaners, lotions, etc. existed and had to be manipulated, e.g., there is no mechanism to refer to any one plate consistently. The model could generate something like "the first plate" but how to actually execute such an action is far from clear.

This third issue is related to the problem of grounding. Normally, grounding means connecting an abstract concept to something concrete in the environment. All of the grounding that is performed here is by virtue of VirtualHome having unique objects in its environments and the actions not requiring multiple instances of the same object. This is not addressing the problem of grounding. Reviewers requested that grounding be removed from the manuscript. This would significantly enhance it, as the model is inherently incapable of grounding as the authors say: "Indeed, one limitation of our approach is that we do not condition on environment state"

Reviewers took issue with details of the evaluation, which are largely a consequence of the choice of VirtualHome. Sometimes this manifested as strange results like models outperforming humans in terms of correctness. As reviewers pointed out, this is worrisome.

Reviewers were also concerned about the title. It implies that language models are zero-shot planners, but this is not the case. They are instead able to decompose actions into mid-level steps. Reviewers suggested that it would be better to focus the title and tone of the manuscript on extracting task/subtask structures from language models.

The idea presented here, that language models can break tasks into subtasks is interesting. But the manuscript goes a step further and discusses embodied agents which to reviewers appeared to be a reach: there is no grounding and in no sense is the output of the language model any different if the agent is embodied. Even the most positive reviewers felt that discussing embodied agents is unhelpful: it would be better to focus on task/subtask structures. And indeed, this would be more general. All of the concerns that reviewers had around the evaluation would be alleviated by focusing on a language task instead. And the effect of a narrow space of actions, constraints on those actions, and multiple objects of the same class, could be evaluated and reported. Even if the authors had to collect such a corpus, given the difficulties they describe in evaluating on VirtualHome, this would be less of a burden. This could be a strong submission in the future.
Decision: Reject