Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
Authors: Wenlong Huang
Pieter Abbeel
Deepak Pathak
Igor Mordatch
Venue: ICLR 2022 Submitted (2022)
Keywords: GPT-3
Codex
LLMs
Language Models
Knowledge Extraction
Embodied Agents
Action Planning

Abstract: Can world knowledge learned by large language models (LLMs) be used to act in interactive environments? In this paper, we investigate the possibility of grounding high-level tasks, expressed in natural language (i.e. "make breakfast"), to a fixed set of actionable steps (i.e. "open fridge"). While prior work focused on learning from explicit step-by-step examples of how to act, we surprisingly find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into low-level plans without any further training. However, the plans produced naively by LLMs often cannot map precisely to admissible actions. We propose a procedure that conditions on existing demonstrations and semantically translates the plans to admissible actions. Our evaluation in the recent VirtualHome environment shows that the resulting method substantially improves executability over the LLM baseline. The conducted human evaluation reveals a trade-off between executability and correctness but shows a promising sign towards extracting actionable knowledge from language models. Videos at https://sites.google.com/view/language-model-as-planner

Under review as a conference paper at ICLR 2022
LANGUAGE MODELS AS ZERO-SHOT PLANNERS: EXTRACTING ACTIONABLE KNOWLEDGE FOR EM- BODIED AGENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Can world knowledge learned by large language models (LLMs) be used interactive environments? In this paper, we investigate the possibility of ground- ing high-level tasks, expressed in natural language (i.e. "make breakfast"), to a chosen set of actionable steps (i.e. "open fridge"). While prior work focused on learning from explicit step-by-step examples of how to act, we surprisingly find that if pre-trained LMs are large enough and prompted appropriately, they can effectively decompose high-level tasks into low-level plans without any fur- ther training. However, the plans produced naively by LLMs often cannot map precisely to admissible actions. We propose a procedure that conditions on ex- isting demonstrations and semantically translates the plans to admissible actions. Our evaluation in the recent VirtualHome environment shows that the resulting method substantially improves executability over the LLM baseline. The con- ducted human evaluation reveals a trade-off between executability and correct- ness but shows promising sign towards extracting actionable knowledge from language model!"
INTRODUCTION
Large language models (LLMs) have made impressive advances language generation and under- standing in recent years (Devlin et al. 2018 Radford et al. 2019 Raffel et al. 2019 Brown et al.) 2020). See [Bommasani et al. (2021) for a recent summary their capabilities and impacts. Being trained on large corpora of human-produced language, these models are thought contain lot of
information about the world (Roberts al. 2020 Li et al. 202T BIG-bench collaboration, 202T albeit in linguistic form.
We ask whether we can use such knowledge contained LLMs not just for linguistic tasks, but to make goal-driven decisions that can be enacted in interactive, embodied environments. But are not simply interested in whether we can train models on a dataset of demonstrations collected for some specific environment
are instead interested in whether LLMs already contain information necessary to accomplish goals without any additional training.
More specifically, we ask whether world knowledge about how perform high-level tasks (such as "make breakfast") can be expanded to a series of groundable actions (such as "open fridge", "grab milk", "close fridge". etc) that can be executed in the environment. For our investigation, we use recently proposed VirtualHome environment (Puig et al. 2018). It can simulate large variety of realistic human activities in a household environment and supports ability perform them via embodied actions defined with a verb-ob ject syntax. However, due to open-ended nature of the tasks, is difficult to autonomously evaluate their success. We rely on human evaluation (conducted on Mechanical Turk) decide whether sequences of actions meaningfully accomplish posed tasks.
We find that large GPT-3 Brown al. 2020) and Codex (Chen et al. 2021) models, when prompted with single fixed example of task description and its associated sequence of actions, can produce very plausible action plans for the task we'r re interested in. Such completions reflect the information already stored in the model no model fine-tuning is involved. Additionally, only observe this
Results and videos https://ai googgee.ooooooooooooooooooeo 1e con/ wtt../////vwewwwa/ga.eeeeee model -as---asppppp -aspllanne

---

Under review as a conference paper at ICLR 2022
100
GPT2-1.5B
Codex-12B
Translated Codex- 12B
Teik: Brush teeth Step Go bothroom
Tast: Brush twwth
Tosk: Brush teeth
Walk bathroom
&Crrrectness
GPES-175R
Step
Step Open door
Codes-122
Executabie Ground
Step 3: Find toothbruch
Step 3: Walk sink
Step 4: Pick up toothorush
Step Put pot stnkk
Step Put tocthirush in mouth
Step Put brush on
GPT3:126
Step Sp tocthpaste and
toothipaste Step toothpaste on toothorush
Step
Translated
Step
water and rinse brush
Codex-121B
one minute
GPT2-1.58
(Ours)
Step Turn off water sink and return brush
Step 8: Put tewtth on toochorush
cupboard
01
%/ Fxecutability
Samples Generated by LLMs
Executability v.s. semantic correctness of generated action plans (left) and sample action plans gen- erated by different models (right). Large models can produce action plans indistinguishable from plans created by humans, but frequently are executable in the environment. Using our techniques, we can significantly improve exccutability, albeit at the cost correctness.
Figure| 1:
effect in the larger models. Unfortunately, despite their semantic correctness, the produced action plans often not executable in the environment. Produced actions may not map precisely to admissible actions, or may contain various linguistic ambiguities.
We propose several tools improve executability of the model's outputs. First, we enumerate all admissible action phrases and map the model's output action to the most semantically-siminn missible action (we use similarity measure between sentence embeddings produced by a RoBERTa model Liu et al. (2019) in this work, but other choices are possible). Second, we use the model to autoregressively generate actions in a plan by conditioning past actions that have been made admis- sible via the technique above. Such on the fly correction can keep generation anchored to admissible actions. Third, we provide weak supervision
ad-
the model by prompting the model with a known task example similar to the query task. This is somewhat reminiscent of prompt tuning approaches, but does not require access to gradients or internals of the model.
Using above tools to bias model generation, we find that we improve executability of instructions from 18% to 79% (see Figure IT without any invasive modifications to model parameters or any extra gradient internal information beyond what is returned from the model's forward pass. This advantageous because it does not require any modifications to model training procedure and can fit within existing model serving pipelines. However, we do find there to be significant drop in cor- rectness of the instruction sequences generated with above tools (as judged by humans), indicating promising step, but requiring more research the topic.
To summarize, our paper's contributions are as follows:
We show that without any training, large language models can be prompted to generate plausible goal-driven action plans, but such plans are frequently not executable interac- tive environments.
We propose several tools to improve executability of the model generation without invasive probing or modifications to the model.
We conduct human evaluation of multiple techniques and models and report on the trade- offs between executabiltiy and semantic correctness.
EVALUATION FRAMEWORK
Simulating open-ended tasks that resemble naturalistic human activities requires environment to support a rich set of diverse interactions, rendering most existing embodied environments unsuitable for our investigation. One exception is VirtualHome (Puig et al. 2018), which models human activ- ities in a typical household. Therefore, we only provide evaluation in this environment. To further measure correctness given open-ended tasks, conduct a human evaluation. We note that since further training is involved throughout our investigations, the observations and findings presented in this paper should also translate to similar embodied environments.
2

---

Under review as a conference paper at ICLR 2022
2.1 EVALUATED ENVIRONMENT: VIRTUALHOME
Preliminaries In VirtualHome, actívities are expressed programs. Each program consists of a sequence of steps, where each step is written as: [action] (argi)(dd) ... (arga)idd,)- Each action refers to atomic actions such as "walk". "open", and "put". A total of 45 atomic actions are supported by VirtualHome. Different actions take in different numbers of arg necessary for specifying interaction. Associated with each arg is a unique id specifying the corresponding node in the environment graph, in case of multiple instances of the same object class are present in the graph. For the sake of simplicity, we omit the id in the remaining discussions of this paper and allow automatic assignment by the environment. An example program is shown in Appendix
Evaluated Tasks We the knowledge base collected by VirtualHome for evaluation. The knowledge base contains household activities crowd-sourced from Amazon Mechanical Turk (MTurk). The MTurk workers were asked to provide natural language descriptions of daily house- hold actívities and all actionable steps necessary for completing the activities. The descriptions are both given high-level task descriptions and step-by-step instructions. We omit the use of step- by-step instructions in this work as we desire direct extraction of executable programs from only task descriptions. For evaluations, we randomly sample subset of 88 high-level tasks, each having one or more annotated ground-truth programs. The remaining 204 tasks are used as demonstration set, from which we are allowed to select as example(s) for prompting language models. Note that no training or fine-tuning is performed using these tasks and their annotations. More details of the evaluated tasks can found in Appendix 8.6
2.2 METRICS
A program that commands the agent to wander around in household environment is highly ecutable but may not complete the desired task. On the other hand, a program composed of step instructions from knowledge bases can likely complete the task but cannot be executed. The reason is that free-form instructions can be ambiguous and may lack necessary common-sense actions. To this end, we consider two axes for evaluation: executability and correctness.
Executability Executability measures whether an action plan can be correctly parsed and satisfies the common-sense constraints of the environment. To be correctly parsed, action plan must be syntatically correct and contain only allowed actions and recognizable objects. To satisfy the common-sense constraints, each action step must not violate the set of its pre-conditions (e.g. the agent cannot grab milk from the fridge before opening it) and post-conditions (e.g. the state of the fridge changes from "closed" to "open" after the agent opens it). We report the average executability across all 88 tasks and across all 7 VirtualHome scenes.
Correctness Unlike most embodied environments where the completion of task be easily judged, the ambiguous and multimodal nature of natural language task specification makes it im- practical to obtain gold-standard measurement of correctness. One approach could be measuring similarity of the final environment state produced by executing predicted and ground-truth programs, but VirtualHome initializes an environment differently based on the co-be-executed program, mak- ing comparisons difficult if measured in such way. Therefore, we conduct human evaluation for the highlighted methods. More details of the human evaluations can be found in Appendix [8.]] For the remaining methods and ablation studies, we rely on match-based metric that measures how similar a generated program to human annotations. Specifically, we follow Puig et al. (2018) and calculate the longest common subsequence (LCS) between two programs, normalized by the maxi- mum length of the two. In the presence of multiple ground-truth programs for a single task, we take the maximum LCS across the ground- truth programs. However, note that the majority of the tasks only have one ground- truth annotation, but there are often many plausible ways to complete a certain task, making this metric imperfect at evaluation program correctness] Although correlation between the two is shown by Puig et al. (018), we consider it only as a proxy metric in replacement of unscalable human evaluation.
Atthoug L.C has mathematical range of [o, 1). we measure the LCC between different ground-truth programs for the same task and find an empirical maximum of (1.899.
3

---

Under review as a conference paper at ICLR 2022
Supp 1: nasir
Task: Shave
Prompt
Step 1: Squeeze out gllo lotion
Task: Slaret
Step Grab
Prompt
Switch
Step 2: Wash roor
Step
Frowen
Step 3: Switch on razor
Tark Apply lobion
Pre-Trained Masked LLM
Task:Apply lation
Step 1:Pour lation right hand
Frozen
Pre-Trained Causal LLM
Frozen
Pre-Trained Causal LLM
Step 1: Squeeze out a glob lotion
Step 1: Pour lotion into right hand
Step-By-Step
Zero-Shot Planning via Causal LLM
Translation Admissible Action
Autoregressive Generation
Figure 2: We investigate the possibility of extracting actionable knowledge from pre-trained language mod- els without any additional training. We first show surprssing finding that large language models (LLMs) can decompose high-level tasks into sensible low-level action plans (left). To make the action plans exccutable, we propose to translate each step into admissiblc action via another LLM (middle). The translated action is appended to the original prompt used for generating the remaining steps (right).
METHOD
In this section, investigate the possibility of extracting actionable knowledge from pre- pre-trained language models without further training. We first give an overview of the common approach to query large language models (LLMs) and how it may be used for embodied agents. Then de- scribe an inference-time procedure that addresses several deficiencies of the LLM baseline and offers better executability in embodied environments. We break down the proposed procedure into three
individual components, each discussed in Sec 3.2 3.3 and 3.4
Since LMs excel at dealing with natural language text instead of the specific format required by VirtualHome as described in Section 2.1 we only expose natural language text to LMs. To do this, we define a mapping for each atomic action that parses natural language phrase the required format. For instance, "Walk to living room" is converted to "[Walk] (living room) (1)". When an LM output cannot parsed to any of the allowed action, the entire program considered syntactically incorrect and thus not executable.
3.1 PROMPTING
Previous works have shown that large language models pre-trained colossal amount of data contain useful world knowledge that can be probed to perform various down-stream tasks (Radford et al. 2019: Brown et al. 2020). Notably, autoregressive LLMs can even perform in-context learn- ing, an approach to solve tasks using only contextual information without gradient updates (Brown) al. 2020). Contextual information is given part of the input prompt and LMs asked to complete the remaining text. often consists natural language instructions and/or a number of examples containing the desired input/output pairs.
We adopt the same approach to query LLMs to generate action plans for high-level tasks. Specif- ically, we prepend one example task description sentence and its annotated action plan from the demonstration sei to the query task description, as shown in Fig 2 To obtain text completion results, we sample from autoregressive LLM using temperature sampling and nucleus sampling (Holtzman et al.] 2211).. We refer to this LM as Planning LM and the approach using this LM for plan genera- tion as Vanilla /L.M, where [L.]] replaced by specific language model such as GPT-3 or Codex.
To further improve the quality of the generated output, we follow Chen et al. (2021) that uses LMs for program synthesis to sample multiple output for each task. However, unlike prior works in program synthesis that choose the sample with highest unit test pass rate, we only consider the setting where one sample allowed to be evaluated for each task. This is because repetitive trial- and-errors can be dangerous in the real world, and executing many action plans is equivalent to probing the environment for privileged information, which is often considered not viable.
3.2 ROBUST PARSING BY SEMANTIC TRANSLATION
One issue arises when naively following the above approach to generate action plans for high-level tasks: the action plan is often not executable hecause LMs are allowed generate free-form text.
4

---

Under review as a conference paper at ICLR 2022
Therefore, most of the time the output cannot be mapped to unambiguous actionable step. And many reasons can cause such failures: 1) the output does not follow pre-defined mappings of any atomic action (i.e. "I first walk to the bedroom" does not follow "Walk to (PLACE)"), 2) the output may refer to atomic action and objects using words unrecognizable by the environment (i.e. "Clean the dirty dishes in the sink" where "clean" and "dirty dishes in the sink" cannot be mapped to precise action and object), 3) the oulput contains lexical ambiguous words (i.e. "Open TV" should instead be "Switch on TV"), or 4) the output may use disallowed action (i.e. "Microwave the cup").
Instead of developing a set of rules to transform the free-form text into admissible action steps, we propose to again leverage world knowledge learned by large language models to semantically translate the action. For each step in the action plan å, we aim to find the most similar admissible environment action de as measured by cosine similarity:
argmax f(ā)- -f(e) where / is an embedding function. [If(a))
dg
To embed the output text and environment actions, we use a BERT-style LM (Devlin et al., 2018
Liu al. 2019) trained with Sentence-BERT (Reimers & Gurevych 2019) objective because of its suitability for sentence modeling. The sentence embedding is obtained by mean-pooling the last layer hidden states across all tokens. We refer this LM as Translation LM. Note that this is a different LM than the GPT-style Planning LM discussed in the text so far. Using single LM for both purposes could as well be possible and likely more efficient, but we leave such investigation to future works. While the set of actions in our environment is discrete and possible to exhaustively enumerate, sampling or projection can be employed in larger discrete or continuous action spaces.
Since Translation LM can guarantee the parsed action is allowed by the enviromment, we can trade- off semantic soundness of an LMM step by how likely it can be mapped to an admissible action the environment. This can be achieved by simple modification to the scheme that we use to choose the best sample from the LM output. Instead of only using mean token log probability as a ranking metric, we choose the sample with the highest score calculated the cosine similarity to the closest allowed action and 3 is a weighting parameter.
C' + B logprob, where C is
3.3 AUTOREGRESSIV TRAJECTORY CORRECTION
Translating each step of the program after the entire program has been synthesized is analogous to open-loop planning and is subject compounding errors. In practice, LLMs might output com- pounded instructions for single step. even though it cannot be completed using one admissible action in the environment. To this end, we can instead interleave plan generation and action trans- lation to allow for automatic trajectory correction. At each step, we first query Planning LM to gen- erate samples for a single action. Then we calculate score s for each sample using Translation LM and append the translated action to the unfinished text completion. This way all subsequent steps will be conditioned on admissible actions instead of free-form text output generated by Planning LM. Furthermore, we can use Translation LM to detect out-of-distribution actions, those outside the capabilities of a robot, and terminate a program early instead of mapping to a faulty action. This can be easily implemented by setting threshold such that if C at step the program is terminated early. We empirically show this leads to better executability while maintaining similar correctness of the generated action plans.
3.4 DYNAMIC EXAMPLE SELECTION FOR IMPROVED KNOWLEDGE EXTRACTION
So far in the text, we always give the same example in the prompt for all evaluated high-level tasks. However, consider the task of "ordering pizza". Prompting LLMs with this task may give the assumption that the agent initialized in front of computer, and the LLMs may guide the agent to search for pizza store and click "checkout my cart". Although these are reasonable and feasible in the real world, such assumption cannot always be made as these interactions may not be supported in simulated environments like VirtualHome. In fact, the closest series of actions that human experts give may be "walking to a computer", "switching on the computer", and "typing the keyboard".. Without being finetuned on these data, LLMs would often fail at these tasks. To provide weak supervision at inference time, we propose to use Translation LM to select the most similar task from the demonstration set to be used the example in the prompt. Specifically, we choose the task
5

---

Under review as a conference paper at ICLR 2022
Hommo Office
Sir Coalr
Lumpute
Louk Comppatee
Sec Ceccaaa Mie
Welk Kitchea
Ojec Fridge
Grub MIL
Close Fridge
Figure 3: Visualization of Virtualloome programs generated by our approach. The row shows the execu- tion of the task "Complete Amazon Turk Surveys", and the bottom shows the task "Get Glass of Milk". show LLMs not only can generate sensible action plans given only high-level lasks but also contains actionable knowledge that can be extracted for grounding in embodied environments.
whose high-level description matches closely the query task as measured by cosine similarity. This allows Planning LM to reason how to perform a similar task given a evaluations, make sure the demonstration set is not overlapping with our test queries.
example. In
Combining the various improvement discussed above, we refer to the final approach as Translated [LMJ, where [LM] is replaced by specific language model used such as GPT-3 and Codex.
RESULTS
In this section, first show that language models generate sensible action plans for many high- level tasks, even without any additional training. Then we highlight its inadequacy when naively applied to embodied environments and demonstrate how this be improved by again leveraging world knowledge learned by LLMs. Visualization of generated programs are shown in Fig
Sampling from IMM Pre-trained LMs are sensitive to sampling parameters and the specific ex- ample given in the prompt. For all evaluated methods, we perform hyper-parameter search over various sampling parameters, and for methods using fixed prompt example, report metrics aver- aged across three randomly chosen examples. To select the best run for each method, we rank the runs by LCS executability, each normalized by human-expert scores Further details can be found in Appendix 8.3
Model Choices We find empirically the combination of Codex- 12B and Sentence-RoBERTA 355M work well in our setting. Codex and GPT-3 are accessed using OpenAI API. The remaining models are accessed through open-source packages, Hugging Face Transformers (Wolf et al., 2019 and SentenceTrane Transformers (Reimers Gurevych, 2019), without additional modifications.
4.1 Do LLMS CONTAIN ACTIONABLE KNOWLEDGE FOR HIGH-LEVEL. TASKS?
We first investigate whether LLMs can generate sensible action plans expressed in free-form lan- guage. We use the approach described in Section 3.1 to query pre-trained LLMs. To evaluate the correctness of generated action plans, we conduct human evaluations. For each model, we ask 10 human annotators to determine by answering "Yes" or "No" whether each task can be completed using provided action steps. To provide a reference of how humans might rate the action plans pro- vided by other humans, also ask annotators to rate the ground-truth action plans provided in the VirtualHome dataset for the same set of tasks. In contrast to the free-form text output by LLMs, the ground-truth action plans from VirtualHome that enforces strict syntax, although annotators were allowed to compose necessary actions.
generated via graphical programming interface
We show the human evaluation results in Fig I where y-axis shows correctness averaged across all tasks and all annotators. Surprisingly, when L.MSs are large enough and without imposed syn- tactic constraints, they generate highly realistic action plans whose correctness as deemed by human annotators even surpasses human-labele ground-truth. Yet another interesting finding is
See footmote 2.

---

Under review as a conference paper at ICLR 2022
Language Model
Executability
LCS
Correctness
Vanilla GPT-2 117M
18.66%
3.19%
14.27%
Vanilla GPT-2 1.5B
39,40%
7.78%
19.51%
Vanilla Codex 2.5B
17.62%
15.57%
53.44%
Vanilla GPT-Neo 2.7B
29.92%
11.52%
50.74%
Vanilla Codex 12B
18.07%
16.97%
56.06%
Vanilla GPT-3 12B
25.87%
13.40%
30.98%
Vanilla GPT-3 175B
7.79%
17.82%
65.19%
Annotated GT Fine-tuned GPT-3 12B
100.00%
N/A
54.80%
66.07%
34.08%
50.56%
Our Final Methods
Translated Codex 12B
78.57%
24.72%
34.97%
Translated GPT-3 175B
73.05%
24.09%
51.73%
Table Human-evaluated correctness and evaluation results in VirtualHome. Although action plans generated by GPT-3 and Codex can even surpass the annotated GT in correctness measure, they are rarely executable. By translating the naive action plans, we show an important step towards grounding LLMs in embodied environ- ments, but observe room to achieve this without trading exccutability for correctness. We also observe failure mode among smaller models that lead high executability.
that Codex outperforms GPT-3 significantly under the same number of model parameters. One hy- pothesis could be that by fine-tuning on structured data (docstrings and code), Codex specializes at decomposing high-level objective into number of basic operations, deseribed in natural language. We also observe some level correctness for smaller models such GPT-2. However, inspection of its produced output indicates that it often generates significantly shorter plans by ignoring common-sense actions or by simply rephrasing the given task (e.g. the task "Go to sleep" produces only single step "Go to bed"). These failure modes sometimes mislead human annotators to mark them correct as the annotators may ignore common-senss actions in their judgment as well, resulting in a higher correctness rate than the quality of the output shows.
with those operations
4.2 How EXECUTABLE ARE THE LLM ACTION PLANS?
We analyze the executability of LLM plans by evaluating them in all 7 household scenes in Virtual- Home. As shown in Table we find action plans generated naively by LLMS are generally not very executable. Although smaller models seem to have higher executability, we find that the majority of these executable plans are produced by ignoring the queried task and repeating the given GT example of different task. This is validated by the fact that smaller models have lower LCS than larger models despite having higher executability, showing that this failure mode is prevalent among smaller models. In contrast, larger models do not suffer severely from this failure mode. Yet result of being more expressive, their generated programs are substantially less executable.
4.3
CAN LLM ACTION PLANS BE MADE EXECUTABLE BY ACTION TRANSLATION?
In this section, we evaluate the effectiveness of our proposed procedure of action translation. We first create bank of all allowed 47522 action steps in the environment, including all possible com- binations of atomic actions and allowed arguments/objects. Then we use off-the-shelf Sentence- RoBERTa (Liu et al. 2019 Reimers & Gurevych, (2019) as Translation LM to create embeddings for actions and output text. For better computational efficiency, we prmpco the embeddings for all allowed actions, leaving minor computation overhead for our procedure over the baseline methods at inference time. As shown Table executability of generated programs is significantly improved. Furthermore, we also observe improved LCS because the translated action steps precisely follow the program syntax and thus are more similar to ground-trui annotations. One sample output is shown in Fig I and larger random subset of generated samples can be found in Appendix 8.7
To validate their correctness, we again perform human studies using the same procedure in Sec Results are shown in Table [ We find that despite being more similar GT, the programs deemed less correct by humans. By examining the generated output, we observe two main sources of errors. First, find Translation LM is poor mapping compounded instructions succinet

---

Under review as a conference paper at ICLR 2022
admissible action. This is partly due to that Translation LM is trained much smaller dataset and contains much a smaller number of parameters, so we expect further improvement by using a larger pre-trained model for translation. The second source of error comes from imperfect expressivity of the environment; we find that for many tasks we evaluate, certain necessary actions or objects are not implemented. This is also reflected by out human evaluation results of the GT programs, only half of the programs are considered complete by the human annotators.
5 ANALYSIS AND DISCUSSIONS
Methods
Executability
L.S
5.1 ABLATION OF DESIGN DECISIONS
Translated Codex 12B
78.57%
24.72% %
w/o Action Translation
31,49%
22.53%
We perform ablation studies to show the ef-
w/o Dynamic Example
50.86%
22.84%
fectiveness and necessity of three components
w/ Iterative
55.19%
24.43%
of our proposed procedure, each described
3.c 3.3 3.4 As shown in Table leav- ing out any of the three components would all lead to decreased performance in both executability and LCS. Notably, not doing action transla- tion leads to the most significant executability drop, showing the importance of action translation in extracting executable action plans from LLMs.
Table 2: Ablation of three proposed techniques.
5.2 CAN LLMS GENERATE ACTIONABLE PROGRAMS BY FOLLOWING DETAILED INSTRUCTIONS?
Prior works often focus translating stepp instructions into executable programs. We eval- uate LLMs under this setting using a prompt format shown in Appendix 8.2 Although this setting is easier as it does not require rich actionable knowledge, detailed instructions can help resolve much ambiguity of exactly how to perform a high-level task when multiple solutions are possible. Therefore, Translated Codex 12B achieves exccutability of 78.57% and LCS of 32.87%. where LCS sees a considerable bump from the setting without detailed instructions. Surprisingly, the LCS result very close to that of a supervised LSTM (Hochreiter Schmidhuber 1997) baseline from VirtualHome trained on human-annotated data, which is at 34.00%. Note that since code to train the baseline and the specific train/test split is not publicly released, we only show results reported in Puig et al. (2018) as a reference. We also cannot compare executability as it is not reported.
5.3 IS ACTION TRANSLATION NECESSARY FOR ACTIONABLE KNOWLEDGE GROUNDING?
The investigations of this paper are two-fold: 1) Is actionable knowledge present in LLMs? 2) Can we ground this actionable knowledge in in- teractive environment? In this section, we focus attention the second question by condi- tioning on the assumption that first question is true. To do this, since successful execution of correct action plans directly measures ground- ing, we select only the correct plans generated by LLMs and measure how executable they are. We deem an action plan to be correct if 70% or more human annotators decide it is correct.
Methods
# of C
#o C and E
E/C
GPT-2 1.5B
8
0
0.00%
GPT-3 12B
24
2
8.33%
GPT-3 175B
68
5
7.35%
Codex 12B
45
17.78%
Translated Codex 12B
18
15
83.33%
Table 3: Count of correcteexecutable programs and percentage of executable among correct. C indicates correct and E indicates executable.
As shown by Table 3 when an LM is not large enough (e.g. GPT-2), not only contains little actionable knowledge, but this knowledge cannot be grounded at all. GPT-3 and Codex, on the other hand, can generate highly correct action plans in free-form language. However, they do not have the capability to ground their actionable knowledge in interactive environments. What's more interesting, by comparing GPT-3 of both 12B parameters and 175B parameters, ratio of executable plans does not improve with the parameter count. This shows that simply training larger models does not necessarily lead to better knowledge grounding. In the meantime, action translation offers promising way towards grounding actionable knowledge by producing highly executable plans. However, we again note that it comes at a trade-off of producing less correct plans as compared to its vanilla counterpart, and hope future endeavors for bridging the gap.
8

---

Under review as a conference paper at ICLR 2022
6 RELATED WORKS
Large-scale natural language modeling has witnessed rapid advances since the inception of the Transformer architecture (Vaswani et al. 2017). It has been shown by recent works that large lan- guage models (LLMs) pre-trained on large unstructured text corpus not only can perform strongly on various down-stream NLP tasks (Devlin et al. 2018 Radford et al.) [019 Raffel et al. 2019)
Brown et af. 2020) but also can internalize an implicit knowledge base containing rich information
about the world (Petroni et al. 2019 Jiang et al. 2020 Davison et al.] 2019] Talmor et al. 2020 Roberts et al. 2020). Furthermore, the learned representations can be used to model relations of entities (Li et al. 2021), retrieve matching visual features (lhhacco al. 2020). and even valuable priors when applied to diverse tasks from different modalities (Lu et al. 2021] Tsimpoukelli et al. 2021]. Compared to prior works in knowledge extraction that extract single-step factual answers memorized by the models (e.g. "Dante was born in [PLACE]"), we aim to extract sequential action plans complete an open-ended human activity (e.g- "make breakfast"). We further require these plans to only contain allowed actions and satisfy the pre/post-conditions of actions in order to be executed by an embodied agent.
At the same time, there has also been growing interest and development in grounding language in embodied environment. A series of prior works have investigated the possibility of parsing language instructions into formal logic to resolve various linguistic ambiguities for embodied agents (Arizi) & Zettlemoyer 2013 Misra et al., 2013: Tenorth al.] 2010). However, they often scale poorly to complex tasks and environments. Recently, more research efforts have been put into creating better and more realistic environments with the goal to further advances in this area (Puig et al,) 2018 Shridhar et al. [2020a] b) Kolve et al., 2017] [Savva et al.) 2019). At the same time, by lever- aging the better representation power of neural architectures, a number of works have looked into creating instruction-followee agents that can perform manipulation (Lynch & Sermanet 2020
navigation (Majumdar et al.) 2020), or both (Suglia et al. 2021 Hill et al. 2020).
Notably, most of these prior works do not leverage full- blown pre-trained LLMs (Suglia et al. 2021 or do not scale to complex human activities (Hill et al. 2020 Lynch & Sermanet 2020). Perhaps more importantly, few works have evaluated LLMs in embodiment setting that realizes the full potential of the world knowledge these models contain: the tasks evaluated are often "pick", "pick",""grb"""" "grab", "open", and etc, which do not resemble the highly diverse activities that humans perform in daily lives. The development of VirtualHome environment Puig et al. (2018) enables such possibility. However, relevant works (Puig al., 2020 Liao al. 2019) rely on human-annotated data and perform supervised training from scratch. Due to the lack of rich world knowledge, these models can only generate action plans given step-by-ster instructions of how to act or video demonstrations. In this work, we take step further by conditioning only on the high-level descriptions and by extracting executable action plans from LLMS without any additional training.
LIMITATIONS AND CONCLUSION
There are several notable limitations of this work. First, although our approach presents a viable way to ground world knowledge in embodied environments, is still a trade-off rather than one best solution since we observe considerable drop in correctness. Second, we focus on high-level to mid- level grounding, assuming there is a controller that can execute mid-level tasks (such as "grab cup"). Our work does not investigate usefulness of LLMs for low-level sensorimotor behavior grounding. The third limitation is that we do not incorporate observation context or feedback into our models. To some extent, we approach LLMs in the same way as how VirtualHome asks human annotators to give action plans for a given huamn activity by imagination, in which case the human human-generan generated action plans also do not incorporate observation context. However, we do see incorporating observation context for complex activities as an exciting future direction.
REFERENCES
Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics, 1:49-62, 2013.

---

Under review as a conference paper at ICLR 2022
BIG-bench collaboration. Beyond the imitation game: Measuring and extrapolating the capabil- ities of language models. In preparation, 2021. URL https://githbil com/ google/
BIG-bench//
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Ste- fano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto. Pe- ter Henderson, John Hewitt, Daniel E. Ho. Jenny Hong. Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Kohd, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma. Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair. Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadim- itriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models, 2021.
Tom В Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Ed- wards, Yura Burda, Nicholas Joseph, Greg Brockman. et al. Evaluating large language models trained on code. arXiv preprint a1Xiv:2107.03374, 2021.
Joe Davison, Joshua Feldman, and Alexander M Rush. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joinl Conference on Natural Language Processing (EMNLP-IJCNIP)) pp. 1173-1178 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instruction-following with deep reinforcement learning via transfer-leamingg from text. arXiv preprint arXiv:2005.00938) 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv: /ttp//944409/5// 2019.
Gabriel Ilharco, Rowan Zellers, Ali Farhadi, and Hannaneh Hajishirzi. Probing text models for common ground with visual representations. arXiv e-prints, pp. arXiv-2005, 2020.
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Lingusstics, 8:423-438, 2020.
Eric Kolve, Roozbeh Mottaghi. Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv: 1712.05474. 2017.
Belinda Z Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint arXiv:2106.00737, 2021.
10

---

Under review as a conference paper at ICLR 2022
Yuan-Hong Liao, Xavier Puig, Marko Boben, Antonio Torralba, and Sanja Fidler. Synthesizing environment-await activities via activity sketches. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 6291-6299, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zer,lemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Kevin Lu, Aditya Grover, Picter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.05247, 2021.
Corey Lynch and Pierre Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648 2020.
Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Improving iiioo-andd-lnguagg navigation with image-text pairs from the web. In European Conference Computer Vision, pp. 259-274. Springer, 2020.
Dipendra Misra, Kejia Tao, Percy Liang, and Ashutosh Saxena. Enviromment-drive lexicon induc- tion for high-level instructions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 992-1002, 2015.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sehastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: 2019.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Tor- ralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8494-8502, 2018.
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: A challenge for social perception and human-ai collaboration. arXiv preprint htt::200000999000 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1066, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: 1908. 10084, 2019.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08911 2020.
Manolis Savva, Abhishck Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp-
0339-9347.7 2019.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020a.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandr Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint w00::20000003768 2020b.
11

---

Under review as a conference paper at ICLR 2022
Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, and Gaurav Sukhatme. Embodied bert: A transformer model for embodied, language-guine guieedgudddddddd visual task completion. arXiv preprint arXiv:2108.04922 049277 2021.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. olmpics-on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743-758, 2020.
Moritz Tenorth, Daniel Nyga, and Michael Beetz. Understanding and executing instructions for everyday manipulation tasks from the world wide web. In 2010 ieee international conference on robotics and automation, pp.
IEEE, 2010.
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multi- modal few-shot learning with frozen language models. arXiv preprint arXiv:2106. 13884, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's ' transformers: State-of-the -tr-artt natural language processing. arXiv preprint drXiv:1910.03771, 2019.
8 APPENDIX
8.1 EXAMPLE PROGRAM IN VIRTUALHOME
[Walk] |living_rooe room ) (1)
[Walk] television ) (1)
[Find] televisiion (1)
[SwitchOn] (television) (1)
[Find] sofa (1)
[Sit] { sofa ) (1)
[TurnTo] ( televisioon ) (1)
[Watch] teleevision (1)
Table 4: An example program for the activity "Relax on sofa".
8.2 EXAMPLE PROMPT CONTAINING STEP-BY-STEP INSTRUCTIONS
Task:
Read book
Step 8: Sit on chair
Description: Walk to home office, turn on light, grab a book, sit in book.
Step 9: Read novel
chair, start to read
Task:
Find dictionary
Step 1: Walk home office
Description: Move towards the bookshelf, scan the bookshelf for the dictionary, when the dictionary is found, pick up the dictionary.
Step 2: Walk to light
Step 3: Find light
Step 4: Switch on light
Step 5:
Find novel
Step 6: Grab novel
Step 7: Find chair
Figure 4: An example prompt containing step-by-step instructions.
8.3 HYPERPARAMETER SEARCH
For each evaluated method, perform grid search over the following hyperparameters:
12

---

Under review as a conference paper at ICLR 2022
Name
Description
Search Values
epsilon (e)
OOD cutoff threshold used in iterative action translation
{0, 0.4, 0.8}
temperature
sampling parameter adjusting relative probabili- {0.1, 0.3, 0.6) ties across tokens
k
number of samples generated when querying LMs each time
{1, 10)
frequence.penalty
OpenAl API specific; penalize new tokens based their existing frequency in the text so far
{0.1, 0.3, 0.6, 0.9)
presence-penalty
OpenAl API specifie; penalize new tokens based on whether they appear in the text so far
{0.3, 0.5, 0.83
repetition_penaly
Hugging Face Transformers specifie, penalize new tokens based on whether they are repeating existing text
(1.0, 1.2, 1.5, 1.8)
For methods that use fixed example across evaluated tasks, we search over the following three ran- domly chosen examples:
Example
Example 2
Example
Task: Use computer
Task: Relax on sofa
Task: Read book
Step 1: Walk to home office
Step 1: Walk to home office
Step 1: Walk home office
Step 2: Walk to chair
Step 2: Walk to couch
Step 2: Walk to novel
Step 3: Find chair
Step 3: Find couch
Step Find novel
Step 4: Sit on chair
Step 4: Sit on couch
Step 4: Grab novel
Step 5: Find computer
Step 5: Find pillow
Step 5: Find chair
Step 6: Switch on computer
Step 6: Lie on couch
Step 6: Sit on chair
Step 7: Turn to computer
Step Read novel
Step 8: Look at computer
Step 9: Find keyboard
Step 10: Type on keyboard
8.4 ANALYSIS OF PROGRAM LENGTII
Shorter programs have a natural advantage of being more executable. Consider a task "wash hands" and corresponding program that only commands agent "go to bathroom" without additional steps. The program is obviously incorrect yet trivially executable. To validate our approach does not simply generate very short program, we calculate the average program length across the 88 evaluated tasks. Results are shown in Table 6 In addition to the failure mode discussed in Section 4.2 that leads incorrect yet executable programs, smaller LMs such as GPT-2 also generate programs significantly shorter than larger models, making them more executable. In contrast, larger models like Codex- generate more expressive program of high correctness, but they often suffer from executability. We show action translation can lead to benefits of both worlds, generating programs that are highly executable while maintaining similar expressiveness in terms of program length.
Methods
Executability
Average Length
Vanilla GPT-2 1.5B
39.40%
4.24
Vanilla Codex 12B
18.07%
7.22
Translated Codex 12B
78.57%
7.13
Ground-Truth
100.00%
9.66
Table Average executability & program length of different methods and human annotations.
13

---

Under review as a conference paper at ICLR 2022
8.5 DETAILS OF HUMAN EVALUATIONS
Human evaluations are conducted on Amazon Mechanical Turk. For each method, we generate action plans for all 88 high-level tasks. To account for expressivity of the VirtualHome environ- ment (Puig et al. 2018), we further include ground-truth action plans from the VirtualHome dataset in our human evaluations. The human evaluations are conducted in the form of questionnaires containing all action plans with unknown corresponding methods. The questionnaire contains the following instructions at the top:
For every question below, determine whether the task can completed in any reasonable scenario using the provided steps. In other words, can the task be decomposed into these steps? Note that simply re-stating the task does not mean completing it.
Human annotators required to answer all the questions in the questionnaire, where each question an action plan generated by a method unknown to the annotator. The order of the questions is randomly permuted before presented to each annotator. For each question, the annotators need to answer either "Yes" or "No" indicating if they believe the action plan completes the task. For each method, we report correctesss percentage averaged across 10 participated human annotators.
8.6 DETAILS OF VIRTCALHOME TASKS
VirtualHome ActivityPrograms is a knowledge base collected on Mechanical Turk by Puig et al. 2018). The knowledge base features real household activities of humans and correspondingly quences of basic actions for robots to perform these activities. Each entry contains a high-level task name (i.e. "Watch TV"), mid-level detailed description (i.e. "Sit on my couch directly opposite my TV, switch on my TV with the remote control and watch"), "),ch"),,, and sequence of allowed actions ka.k.a. a program) to perform the task in VirtualHome. There are 292 distinct high-level tasks and a total of 2821 different entries in the knowledge base, where each high-level task may correspond one or several programs. The knowledge base covers diverse set real human activities and a wide range of different actions and objects. Puig et al. (2018) also conduct additional human evaluation on 100 randomly sampled programs and show that 64% of them are complete (i.c. contain all necessary steps). Readers are encouraged to refer to Puig al. 20188 for more comprehensive analysis of the dataset.
Train/Test Split From all 292 distinct high-level tasks from the ActivityPrograms knowledge base, randomly sample 88 high-level tasks for evaluation. The remaining 204 tasks are used demon- stration set from which we are allowed to select as example(s) for prompting language models, or in the case of supervised fine-tuning, they are used to fine-tune pre-trained language models.
8.7 RANDOM SAMPLES OF GT, VANILLA CODEX, AND TRANSLATED CODEX
Annotated Ground-Trun
Codex
Translated Codex (Ours)
Task: Go to sleep
Task: Go to sleep
Task: Go to sleep
Step 1: Walk to bedroom
Step 1: Walk to bedroom
Step 1: Walk to bedroom
Step 2: Walk to bed
Step 2: Walk to bed
Step 2: Walk to bed
Step 3: Find bed
Step 3: Find bed
Step 3: Lie bed
Step 4: Turn to bed
Step 4: Sit on bed
Step 4: Sleep
Step 5: Look at bed
Step 5: Find pillow
Step 6: Sit on bed
Step 6: Sleep on pillow
Step 7: Sleep
14

---

Under review as a conference paper at ICLR 2022
Task: Browse internet
Task: Browse internet
Task: Browse internet
Step 1: Walk to home office
Step 1: Walk to home office
Step 1: Walk to home office
Step 2: Walk to chair
Step 2: Walk to chair
Step 2: Walk to desk
Step 3: Find chair
Step 3: Find chair
Step 3: Find chair
Step 4: Sit chair
Step 4: Sit chair
Step 4: Sit chair
Step 5: Find computer
Step 5: Find computer
Step 5: Find computer
Step 6: Switch on computer
Step 6: Switch on computer
Step Switch on computer
Step 7: Turn to computer
Step 7: Turn to computer
Step 7: Find mouse
Step 8: Look at computer
Step 8: Look at computer
Step 8: Grab mouse
Step 9: Find keyboard
Step 9: Find keyboard
Step 9: Turn to computer
Step 10: Type on keyboard
Step 10: Type on keyboard
Step 10: Point at computer
Task: Listen to music
Task: Listen to music
Task: Listen to music
Step 1: Walk to bedroom
Step 1: Walk home office
Step 1: Walk home office
Step 2: Walk to computer
Step 2: Walk to chair
Step 2: Walk to computer
Step 3: Find computer
Step 3: Find chair
Step Sit chair
Step 4: Turn to computer
Step 4: Sit on chair
Step 4: Switch on computer
Step 5: Look at computer
Step 5: Find computer
Step Find music stand
Step 6: Walk to computer
Step 6: Switch on computer
Step 7: Find chair
Step 7: Turn computer
Step 8: Pull chair
Step 8: Look at computer
Step 9: Sit chair
Step 9:
Find keyboard
Step 10: Find headset
Step 10: Type on keyboard
Step 11: Grab headset
Step 12: Put headset
Step 13: Switch on computer
Step 14: Find keyboard
Step 15: Type on keyboard
Task: Brush teeth
Task: Brush teeth
Task: Brush teeth
Step Walk to bathroom
Step Walk to bathroom
Step Walk to bathroom
Step 2: Walk to tooth paste
Step 2: Walk to sink
Step 2: Open door
Step 3: Find tooth paste
Step 3: Find toothbrush
Step 3: Walk to sink
Step 4: Grab tooth paste
Step 4: Pick up toothbrush
Step 4: Put pot sink
Step 5: Find toothbrush
Step 5: Put toothbrush mouth
Step 5: Put brush on toothbrush
Step 6:
Grab toothbrush
Step 6: Move brush around
Step 6: Turn to toothpaste
mouth for two minutes
Step Put toothpaste on tooth-
Step 7: Spit
toothpaste and
brush
brush into sink
Step 8: Put teeth on toothbrush
Step 8: Turn on water in sink and rinse brush for one minute
Step 9: Turn off water in sink and return brush to cupboard
Task: Dry soap bottles
Task: Dry soap bottles
Task: Dry soap bottles
Step 1: Find towel
Step 1: Walk to kitchen
Step 1: Walk to bathroom
Step 2: Find soap
Step 2: Walk to sink
Step 2: Walk to shower
Step 3: Grab soap
Step 3: Find sink
Step 3: Find soap
Step 4: Wipe soap
Step 4: Turn on water tap
Step 5: Put back soap
Step 5: Wait for water to be hot
Step 6: Find shampoo
enough
Step 7: Grab shampoo
Step 6:
Fill container with hot
Step 8: Wipe shampoo
water
Step 9: Put back shampoo
Step 7: Find soap bottles
Step 10: Find shampoo
Step Pour soap into container
Step 11: Grab shampoo
with hot water
Step 12: Wipe shampoo
Step 9: Wait for soap to dissolve
Step 13: Put back shampoo
in water
Step 10: Turn off tap and empty
container into sink
15

---

Under review as a conference paper at ICLR 2022
Task: Vacuum
Task: Vacuum
Task: Vacuum
Step 1: Walk to home office
Step 1: Walk to hallway
Step 1: Walk to home office
Step 2: Walk to computer
Step 2: Walk to closet
Step 2: Walk to dresser
Step 3:
Find vacuum cleaner
Step 3: Find closet
Step 3: Open dresser
Step 4:
Switch
vacuum
Step 4: Open closet
Step 4: Find vacuum cleaner
cleaner
Step 5: Find vacuum cleaner
Step 5: Grab vacuum cleaner
Step 5: Switch off vacuum
Step 6: Turn on vacuum cleaner
Step Pull vacuum cleaner
cleaner
Step 7: Walk to living room
Step 7: Plug in vacuum cleaner
Step 8:
Vacuum floor in living
Step 8: Switch on vacuum
room
cleaner
Step 9: Pull vacuum cleaner
Step 10: Push vacuum cleaner
Task: Find dictionary
Task: Find dictionary
Task: Find dictionary
Step 1: Walk to home office
Step 1: Walk to home office
Step 1: Walk to home office
Step 2: Walk to bookshelf
Step 2: Walk chair
Step 2: Walk book
Step 3: Turn to bookshelf
Step 3: Find chair
Step 3: Find textbook
Step 4: Look at bookshelf
Step 4: Sit on chair
Step 5: Turn to bookshelf
Step 5: Find dictionary
Step 6: Look bookshelf
Step 6: Open dictionary
Step 7: Turn to bookshelf
Step 7: Look at dictionary
Step 8: Look at bookshelf
Step 9: Turn to bookshelf
Step 10: Look at bookshelf
Step 11: Turn to bookshelf
Step 12: Look at bookshelf
Step 13: Turn to bookshelf
Step 14: Look at bookshelf
Step 15: Find novel
Step 16: Grab novel
Task: Wash monitor
Task: Wash monitor
Task: Wash monitor
Step 1: Walk to home office
Step 1: Walk to home office
Step 1: Walk to check
Step 2: Walk to computer
Step 2: Walk to chair
Step 2: Walk to desk
Step 3: Find computer
Step 3: Find chair
Step 4: Turn computer
Step 4: Sit chair
Step 5: Look at computer
Step 5: Find computer
Step 6: Switch off computer
Step 6: Switch
computer
Step 7: Find brush
Step 7: Turn to computer
Step 8: Turn brush
Step 8:
Look at computer
Step 9: Look at brush
Step 9:
Find keyboard
Step 10: Grab brush
Step 10: Type on keyboard
Step 11: Wipe computer
Step 12: Drop brush
Step 13: Find napkin
Step 14: Grab napkin
Step 15: Wash napkin
Step 16: Wipe computer
Task: Read newspaper
Task: Read newspaper
Task: Read newspaper
Step 1: Walk to home office
Step Walk home office
Step Walk home office
Step 2: Walk to chair
Step 2: Walk to chair
Step 2:
Walk to chair
Step 3: Find chair
Step 3: Find chair
Step Find chair
Step 4: Sit on chair
Step 4: Sit on chair
Step 4: Sit on chair
Step 5: Find newspaper
Step 5: Find newspaper
Step 5: Find newspaper
Step 6: Grab newspaper
Step 6: Open newspaper
Step 6: Grab newspaper
Step 7: Open newspaper
Step 7: Read newspaper
Step 7: Read newspaper
Step 8: Read newspaper
16

---

Under review as a conference paper at ICLR 2022
Task: Apply lotion
Task: Apply lotion
Task: Apply lotion
Step 1: Find bed
Step 1: Walk to bathroom
Step 1: Walk to bathroom
Step 2: Turn to bed
Step 2: Walk to cabinet
Step 2: Turn to lotionbottle
Step 3: Point at bed
Step 3: Open cabinet
Step 3: Grab lotionbottle
Step 4: Wash bed
Step 4: Find lotion
Step 5: Turn to bed
Step 5: Apply lotion on hands
Step 6: Look at bed
Step 6: Rub hands together for
Step 7: Push bed
10 seconds
Step 8: Sit on bed
Step 7: Apply lotion on face
Task: Draw picture
Task: Draw picture
Task: Draw picture
Step 1: Walk to home office
Step 1: Walk to home office
Step 1: Walk to home office
Step 2: Walk to chair
Step 2: Walk to chair
Step 2: Walk to wall
Step 3: Find chair
Step 3: Find chair
Step 3: Walk to drawing
Step 4: Sit on chair
Step 4: Sit on chair
Step 4: Find drawing
Step 5: Find desk
Step 5: Find computer
Step 5: Turn drawing
Step 6: Open desk
Step 6: Switch on computer
Step 6: Look at drawing
Step 7: Find check
Step 7: Turn computer
Step 7: Touch drawing
Step 8: Grab check
Step 8: Look at computer
Step 8: Pull drawing
Step 9: Put check on desk
Step 9:
Find keyboard
Step 9:
Push drawing
Step 10: Find pencil
Step 10: Type on keyboard
Step 11: Grab pencil
Step 12: Close desk
Task: Prepare pot of boiling wa-
Task: Prepare pot of boiling wa-
Task: Prepare pot of boiling
ter
ter
ter
Step 1: Walk to dining room
Step 1: Walk kitchen
Step Walk kitchen
Step 2: Walk to pot
Step 2: Walk to sink
Step 2: Walk to cabinet
Step 3: Find pot
Step 3: Find sink
Step Find pot
Step 4: Grab pot
Step 4: Turn on tap
Step 4: Grab pot
Step 5: Walk to sink
Step 5: Fill kettle with water
Step 5:
Walk to stove
Step 6: Find faucet
Step 6: Put kettle on stove
Step 6: Find stove
Step 7: Switch on faucet
Step 7: Turn on stove
Step 7: Switch on stove
Step 8: Put pot on sink
Step 8: Wait for water to boil
Step 9: Grab pot
Step 10: Switch off faucet
Step 11: Grab pot
Step 12: Walk to oven
Step 13: Put pot on oven
Step 14: Switch on oven
Task: Look at mirror
Task: Look at mirror
Task: Look at mirror
Step 1: Walk to bathroom
Step 1: Walk to bathroom
Step 1: Walk to bedroom
Step 2: Walk to mirror
Step 2: Walk to mirror
Step 2: Walk to mirror
Step 3: Find mirror
Step 3: Look at mirror
Step 3: Find mirror
Step 4: Turn to mirror
Step 4: Turn to mirror
Step 5: Look at mirror
Step 5: Look at mirror
Task: Wipe down sink
Task: Wipe down sink
Task: Wipe down sink
Step 1: Walk to dining room
Step 1: Walk to kitchen
Step 1: Walk to bathroom
Step 2: Walk to rag
Step 2: Walk to sink
Step 2: Walk to cleaning solution
Step 3: Find rag
Step 3: Find sink
Step 3: Find cleaning solution
Step 4: Grab rag
Step 4: Wipe down sink
Step 4: Grab cleaning solution
Step 5: Walk to sink
Step 5: Return to kitchen
Step 5: Find sink
Step 6: Wipe sink
Step 6: Pour cleaning solution
Step 7: Put back rag
into glasses
Step 7: Find faucet
Step 8: Switch on faucet
Step 9: Scrub sink
Step 10: Wash sink
17

---

Under review as a conference paper at ICLR 2022
Task: Complete surveys on ama-
Task: Complete surveys on ama-
Task: Complete surveys on
zon turk
zon turk
zon turk
Step I: Walk to chair
Step 1: Walk to home office
Step I: Walk to home office
Step 2: Sit on chair
Step 2: Walk to chair
Step 2: Walk to desk
Step 3: Find computer
Step 3: Find chair
Step 3: Find chair
Step 4: Switch on computer
Step 4: Sit on chair
Step 4: Sit on chair
Step 5: Turn computer
Step 5: Find computer
Step 5: Find computer
Step 6: Look at computer
Step 6: Switch on computer
Step 6: Switch on computer
Step 7: Find mouse
Step 7: Turn to computer
Step 8: Grab mouse
Step 8:
Look at computer
Step 9: Turn to computer
Step 9:
Find keyboard
Step 10: Point at computer
Step 10: Type on keyboard
Step 11: Find check
Step 12: Grab check
Step 13: Read check
Step 14: Put back mouse
Step 15: Find keyboard
Step 16: Type on keyboard
Task: Write book
Task: Write book
Task: Write book
Step 1: Walk to home office
Step 1: Walk to home office
Step I: Walk to home office
Step 2: Walk to laptop
Step 2: Walk to chair
Step 2: Walk to light
Step 3: Find laptop
Step 3: Find chair
Step 3: Find light
Step 4: Grab laptop
Step 4: Sit on chair
Step 4: Switch on light
Step 5: Find electrical outlet
Step 5: Find computer
Step 5: Find pen
Step 6: Plug in laptop
Step 6: Switch on computer
Step 6: Grab pen
Step 7: Walk to love seat
Step 7: Turn to computer
Step 7: Find chair
Step 8: Sit on love seat
Step 8: Look at computer
Step 8: Sit on chair
Step 9: Switch on laptop
Step 9:
Find keyboard
Step 9: Turn to novel
Step 10: Find keyboard
Step 10: Type on keyboard
Step 11: Type on keyboard
Task: Paint ceiling
Task: Paint ceiling
Task: Paint ceiling
Step 1: Find drawing
Step 1: Walk to home office
Step 1: Walk to home office
Step 2: Turn to drawing
Step 2: Walk to chair
Step 2: Walk to ceiling
Step 3: Point at drawing
Step 3: Find chair
Step 3: Walk to painting
Step 4: Watch drawing
Step 4: Sit on chair
Step 4: Find painting
Step 5: Touch drawing
Step 5: Find paint can
Step 5: Turn to painting
Step 6: Walk to bedroom
Step 6: Open paint can
Step 6: Look at painting
Step 7: Pour paint into paint can
Step 7: Touch painting
Step 8: Close paint can
Step 8: Pull painting
Step 9: Find roller
Step 9:
Push painting
Step 10: Roll paint onto ceiling
Task: Change clothes
Task: Change clothes
Task: Change clothes
Step 1: Walk to bedroom
Step 1: Walk to closet
Step 1: Walk to bedroom
Step 2: Walk to dresser
Step 2: Open closet
Step 2: Walk to dresser
Step 3: Open dresser
Step 3: Find clothes
Step 3: Find dresser
Step 4: Find dress
Step 4: Put on clothes
Step 4: Open dresser
Step 5: Grab dress
Step 5: Close closet
Step 5: Find pants
Step 6: Put on dress
Step 6: Grab pants
Step? 7: Close dresser
Step 7: Put on pants
Step 8: Find shirt
Step 9: Grab shirt
Step 10: Put on shirt
Step 11: Close dresser
18

---

Under review as a conference paper at ICLR 2022
Task: Make bed
Task: Make bed
Task: Make bed
Step 1: Walk to bedroom
Step 1: Walk to bedroom
Step 1: Stand up
Step 2: Walk to bed
Step 2: Walk to bed
Step 2: Walk to bedroom
Step 3:
Find sheets
Step 3: Find bed
Step 3: Walk to hed
Step 4: Grab sheets
Step 4: Sit bed
Step 4: Turn to bed
Step 5: Pull sheets
Step 5: Find sheet
Step 5; Put sheets on bed
Step 6: Pick up sheet
Step 7: Find pillow
Step 8: Pick up pillow
Step 9: Find blanket
Step 10: Pick up blanket
Task: Read yourself to sleep
Task: Read yourself to sleep
Task: Read yourself to sleep
Step 1: Walk to bedroom
Step 1: Walk to home office
Step 1: Walk to bedroom
Step 2: Walk to novel
Step 2: Walk to chair
Step 2: Walk to book
Step 3: Find novel
Step 3: Find chair
Step Sit chair
Step 4: Grab novel
Step 4: Sit on chair
Step 4: Read check
Step 5: Find bed
Step 5: Find computer
Step Sleep
Step 6: Sit on bed
Step 6: Switch on computer
Step 7: Find bookmark
Step 7: Turn computer
Step 8: Grab bookmark
Step 8: Look at computer
Step 9: Read novel
Step 9:
Find keyboard
Step 10: Put bookmark novel
Step 10: Type on keyboard
Step 11: Sleep
Task: Hang keys
Task: Hang keys
Task: Hang keys
Step 1: Walk to bedroom
Step 1: Walk home office
Step Walk home office
Step 2: Walk to dresser
Step 2: Walk to chair
Step 2: Walk to table
Step 3: Find purse
Step 3: Find chair
Step Find keys
Step 4: Grab purse
Step 4: Sit on chair
Step 4: Grab keys
Step 5: Open purse
Step 5: Find computer
Step 5: Walk to wall
Step 6: Find keys
Step 6: Switch on computer
Step 6: Find hanger
Step 7: Grab keys
Step 7: Turn to computer
Step 7: Put keys on hanger
Step 8: Find bowl
Step 8:
Look at computer
Step 9: Put keys on bowl
Step 9:
Find keyboard
Step 10: Close purse
Step 10: Type keyboard
Step 11: Put back purse
Task: Eat cheese
Task: Eat cheese
Task: Eat cheese
Step 1: Walk to dining room
Step 1: Walk fridge
Step Walk dining room
Step 2: Walk to freezer
Step 2: Open fridge
Step 2: Walk to table
Step 3: Open freezer
Step 3: Find cheese
Step Find table
Step 4: Find cheese
Step 4: Pick up cheese
Step 4: Turn to table
Step 5: Grab cheese
Step 5: Walk to table
Step 5: Find chair
Step 6: Eat cheese
Step 6: Place cheese on table
Step 6: Sit chair
Step 7: Close fridge
Step 7: Find food
Step 8: Walk to chair
Step 8: Grab food
Step 9: Sit on chair
Step 9:
Find plate
Step 10: Eat cheese
Step 10: Put food plate
19


Reviews:
[{"review_id": "zKHB8ZZA5Kj", "reviewer": "Reviewer_mJtc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "After rebuttal: I have read the author response. Some of my comments regarding the dataset and evaluation were partially addressed, so I am raising my score. However, I feel ambivalent about the paper due to concerns about the evaluation (See detailed comments in discussion thread). \n\n====\n\nPros\n* Using language models to infer actionable plans from text instructions is interesting\n* Paper is generally easy to follow\n\nCons\n* The paper doesn’t provide sufficient details about the data, the train and test splits, and the challenging aspects of the task.\n* The pipelined approach considered here seems fairly limited\n* Limited technical novelty\n* Weak experiments\n\nThe paper should provide more details about the data. For instance, how similar are the train and test tasks? I would encourage the authors to provide a summary of the tasks. How was the 204/88 task split decided?\n\nThe metrics considered in the paper are less ideal and do not provide a holistic view of model performance. For instance, \n* Executability doesn’t consider whether predicted actions are relevant to the task\n* LCS penalizes actions that don’t appear in ground truth\n* Correctness: Why is the correctness of GT plans only 55%? This makes it very hard to interpret these results and I am not sure how meaningful the results are. \nFurthermore, it’s hard to say if the Translated variants in Table 1 are better than the plain language models. While these models are better in terms of executability and LCS, they are inferior in terms of correctness.\n\nHow were the human annotators instructed to label model generated plans? \n\nIs it possible to define a straightforward metric that identifies whether a given task was successfully completed or not?\n\nThe proposed approach should be compared against other baselines. For instance, a fine-tuning baseline can be considered where the model translates instructions to plans. \n\nI would encourage the authors to analyze and report model prediction errors.\n\nLinks to appendix are broken.\n", "summary_of_the_paper": "This paper explores the ability of pre-trained language models to generate plans or action sequences from a text instruction. The in-context learning ability of language models is used where the model is prompted with an example instruction and corresponding action sequence and the query instruction. Since text sequences generated by the language model may not be directly usable in the agent environment, the closest valid text actions are identified using a retrieval approach. Experiments compare performance of different language models on tasks from the VirtualHome benchmark. ", "summary_of_the_review": "While the authors explore an interesting approach to planning based on language models, the paper is weak in terms of technical novelty and experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XTeq2BxDIU", "reviewer": "Reviewer_28Tw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## What I like about this paper\n\n- Rather than trying to solve a particular task using large language models, this paper focuses on analyzing what is the capability of pre-trained language models in terms of decomposing high-level tasks into a sequence of instructions. In other words, what kind of actionable knowledge, if any, is stored in those large language models?\n\n- Using two axes of evaluation - executability, and correctness - better characterize the strengths and limitations of the language models studied in this paper. Looking at the discrepancy between the correctness, as reported by the human annotators, and the executability of a plan motivates the need for a translation LM. It can overcome some of the restrictions interactive environments have, i.e. their action space contains instructions with a specific syntax.\n\n- I like the suggestion of using another language to translate generated instructions to grounded actions to boost executability. Also, it makes sense to me to use an autoregressive trajectory correction approach to improve the executability of a plan.\n\n## Concerns\n\n- My main concern lies with the type of language models used in this analysis. Other than being really popular, it is not clear why the authors decided to only investigate models in the GPT-* family? Is it that other large language models (e.g., T5) are less amenable to prompt-engineering? To me, the main message/claims of the paper seem to be about large language models in general. It would be interesting to know if actionable knowledge is only present in GPT-* models or not.\n\n\n-----\n### Typos\n- p.7: \"We find that despite [being] more similar ...\" - Missing word.\n- p.7: \"...; we find that for [a] many tasks we ...\" - Out-of-place word.\n- p.7: \"... by [our -> the] human annotators\" - Suggested change.", "summary_of_the_paper": "This paper investigates whether large language models (LLMs) are capable, without additional training, of decomposing high-level tasks into a sequence of instructions (i.e., a plan) and grounding them in an embodied environment. Specifically, they relied on prompt engineering to provide enough context to the LLMs for generating sensible instructions. The authors also suggest using another language model to translate generated instructions into actions that are parsable by the embodied environment.\n\nThrough a series of experiments, the authors empirically show that larger LLMs are able to produce sensible plans (according to human annotators) that can also be mapped to executable actions within an embodied environment. In their experiments, the authors evaluate GPT-2, GPT-3, and Codex (including different sizes for the GPT-* family). Part of the paper is also dedicated to answering several hypotheses related to the investigation.", "summary_of_the_review": "In addition to their impressive performance on several standard NLP tasks, large language models have been found useful for a wide variety of tasks when prompted appropriately. I believe understanding, via investigative work, what is the extent of the knowledge contained in those models to be important as it can better guide the research community towards what to explore next. The proposed paper is exactly about doing such an investigation. I found the paper well-motivated, and it includes the experiments to support the authors' analysis and discussion. For those reasons, I recommend this paper be accepted at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "N3yxAVW7p6_", "reviewer": "Reviewer_gxEx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The paper studies an idea that seems relatively novel to this reviewer -- how well can models generate action plans in an embodied environment (not just code, or language)?\n* The paper presents a simple idea -- using a translation LM to constrain the generated actions -- that greatly improves performance, by ensuring that the actions generated by the LM are valid within the environment.\n* The paper has results that consider LMs of a variety of different sizes, from 1.5B to 175B, including both GPT-3 and codex families. The results seem to suggest that size is important, at least in this setting, insofar as the smaller models just copy from the input prompt.\n* This paper presents analysis about the kinds of programs generated by LMs which I think might help future work build off of this direction.\n\nWeaknesses:\n* The experimental setup proposed by this paper for measuring grounding might not be ideal. In the VirtualHome setup, the action space is composed of mid-level actions (like \"walk to home office\") versus low-level actions (\"walk forward 1m\" ,etc), so the promise of LMs for embodied understanding might not generalize to more difficult environents (like Alfred; Shridhar et al 2020a). That said, I think this limitation is discussed well in the limitations section (Sec7).\n* Though I appreciate the authors' work on making the evaluation robust, it is not clear to this reviewer that it is a good measure of correctness. To the best of this reviewer's understanding (of Section 2.2; Correctness), in the VirtualHome environment that the authors study, it is nontrivial to compare whether the final environment state of a generated program is similar to the \"ground truth\" environment state. The details of the human evaluation are a bit unclear, but it seems like humans are just given the list of actions rather than the world state after executing those actions. In this setup, it seems like there is a very strong bias towards language that 'looks like' real action plans, rather than language that is truly actionable. The result is that models outperform humans at the task. To this reviewer, the paper could be improved if the correctness measured something grounded, rather than just something that looks grounded.\n\n\nClarification:\n* I'd appreciate more details on the human evaluation somewhere (like the interface being shown to the turkers)", "summary_of_the_paper": "This paper studies left-to-right language models for on the VirtualHome planning task. In this task, a model is given a high level goal like \"get glass of milk\" and it must generate a sequence of lower-level steps to execute that goal ('walk to kitchen', 'open fridge', etc.). These steps correspond to the annotated scratch programs built by crowd workers as part of VirtualHome, so as such they are executable plans to achieve the given goal.\n\nThis paper uses a left-to-right LM to generate those lower-level steps, given one prompted example. The 1.5B-sized LMs do not seem to do very well at generating actions given this prompt. The larger 12B, etc. LMs do better, but frequently deviate from the set of valid actions, so the authors introduce a Translation LM that measures the similarity between any \"action\" that the LM generates and the set of valid actions. At each step, this translation LM is used in the loop to constrain the chosen action, and this improves executability significantly (e.g. 7% -> 73% for GPT-3 and 18% -> 78% for Codex). Helpful analysis breaks down this performance.", "summary_of_the_review": "Overall, my opinions on this paper are fairly positive, because it tackles an interesting new direction with new simple methodology for generating actions from LMs. However, there are some issues with evaluation, that could greatly improve the paper if addressed (or at the least, discussed a bit more in the limitations section). \n\n--\n\nUpdate post review: I am lowering my score from 8 -> 6. I still think this is a good paper and it should be accepted, but I am not very convinced by the evaluation. I think the evaluation issues I (and KhzN) have issues with could be improved between acceptance and publication though.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IXBMuHgTwd", "reviewer": "Reviewer_KhzN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper shows that the pretrained language model contains knowledge that is useful for action planning. While this is a promising direction to use pretrained language model, I have several concerns and questions for this paper. \n\n1. Grounding & planning vs. knowledge extraction. When we do grounding or planning, the agent’s actions may change given the environment they are put in. The proposed approach is closer to procedure knowledge extraction using language models rather than planning or grounding. While the available actions in VirtualHome might be different from other virtual environments (therefore, this paper proposed a translation step), the generated action plans don’t change if we change the environment from one home to another. So, the generated plans are not really grounded in the domain or environment the agent is planning for. The authors may consider repositioning this paper to knowledge extraction papers that fit the contribution better.\n\n2. Choice of VirutalHome task for planning. The tasks are mainly around the daily home tasks like setting up a dinner table. To perform such tasks, most of the time, the agent may just apply predefined procedure knowledge. However, procedure knowledge and the translation approach proposed by this paper cannot plan for the task like “stack two plates on the right of a cup” because the language model doesn’t ground its knowledge to the environment so the agent cannot tell the difference between putting down on the left or right of the cup. This is the main limitation to using the proposed approach as a planner.\n\n3. Evaluating correctness and executability separately. The goal of task planning is to find the plan that is executable and correct, i.e. success rate. However, most of the tables in the evaluation only show one or another. Only Table 4 shows “# of C and E”. If we compute the success rate for Translated Codex 12B, it is 15/88=0.17 and would be lower for other models. This reflects the performance of the proposed method better and the overall success rate is quite low. \n\n4. Missing related work. This paper missed some related work on knowledge extraction/mining using pertained language models. Just to list a few below. The authors will need to discuss how the proposed approach is different from other knowledge mining approaches compared to prior work.\n - Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A.H. and Riedel, S. “Language models as knowledge bases?” In EMNLP 2019.\n - Davison, Joe, Joshua Feldman, and Alexander M. Rush. \"Commonsense knowledge mining from pretrained models.\" In EMNLP 2019. \n - Jiang, Z., Xu, F.F., Araki, J. and Neubig, G. “How can we know what language models know?”. Transactions of the Association for Computational Linguistics, 8, pp.423-438.\n", "summary_of_the_paper": "This paper proposes using pretrained language models for planning actions. They achieve this by providing an example task sequence as a prompt to generate an action description for each step autoregressively. At each step, they translate the action description to the environment action by computing the similarity between the description and the available environment actions. The results show that there is a trade-off between executability and correctness. ", "summary_of_the_review": "This paper shows promising results on extracting executable actions using pretrained language model. However, there are major concerns about how this relates and contributes to planning and grounding.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9drtVJNKb3X", "reviewer": "Reviewer_VpdM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- LLMs are powerful tools and learn knowledge that we may even not know. So prompting them to utilize the learned knowledge is a promising research direction. Prompting LLMs for actionable knowledge extraction is new and interesting. \n- Different LLMs including GPT-3 and Codex are used and evaluated here.  \n- The prompt and translation method is reasonable on the VirtualHome environment.\n\nWeaknesses:\n- The paper's claim is to extract actionable knowledge for embodied agents, but in fact, it seems the translation process is particularly tailored for VirtualHome. I couldn't see a clear way to generalize to other embodied environments. It would be great if the authors can show its generalization to other environments such as Room2Room and ALFRED. I am not sure if the translation part is really necessary in other environments.\n- Moreover, it seems there is no embodied agent to execute those instructions. All experiments are conducted on the text level only, but we all know that there is a huge gap between text instructions and actual actions taken by the agent. So it would be much more convincing if the authors can show an embodied agent can actually follow the generated instructions and complete tasks in the environment. \n- There seems to be a tradeoff between executability and correctness for the current approach. The correctness is significantly dropped from 56% to 34% for Translated Codex 12B. Why are some correctness numbers missing in Table 1? What is the correctness of Translated GPT-3 175B? The Correctness metric is also missing in Table 2 and Table 3. Why? \n- It is actually pretty easy to improve the executability by finding the most similar actions from the action set of the environment. The real challenge is to make the actions more executable while preserving their correctness, which, however, isn't achieved in this work and it quite a pity. Without considering the correctness, executability does not really matter much, for an extreme example,  a random agent may achieve  high executability but low correctness. \n- Many technical details are missing in the paper, for example, \n  - It is unclear how Executability is calculated. There is only high-level language description in the paper. Can you elaborate it? \n  - It is unclear how the sampling for LMs is done. Do you sample multiple ones and then rely on human evaluation to select the best one? This doesn't sound right and scalable. \n  - Section 2.1: Appendix ?? points to nowhere. Similar issues exist in many places of the paper. \n  - \"\" in the paper is in the wrong format. \n", "summary_of_the_paper": "This paper takes advantage of the pre-trained large language models (LLMs) and assesses whether they can be helpful for embodied agents. The authors choose the VirtualHome environment as the testbed and prompt the LLMs to decompose a high-level instruction to detailed, actionable instructions. Human evaluation is conducted to evaluate the correctness and executability of the generated instructions. ", "summary_of_the_review": "I like the idea of prompting LLMs to decompose high-level instructions and extract actionable knowledge, which is a novel perspective for embodied agents. However, the execution of the idea and the experiments are not right there to support the claim. It would be great if the authors can show its effectiveness on \"actual\" embodied agents in different environments instead of text games. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript presents a method to refine high-level task descriptions into mid-level executable steps. The idea of using language models to generate steps for a robot to follow is very interesting. Reviewer concerns focused on the general applicability of the approach and the evaluation.

Reviewers pointed out that the method is tied to VirtualHome which has various properties that are in general not true: the action space is small, the action space is very sparse, and objects tend to be unique.

First, the method enumerates a sentence for every possible action and object combination in the environment. The fact that VirtualHome has few verbs and few objects and that neither of these has complex additional structure (adjectives, adverbs, etc.) means that this is practical. But in any other practical setting this will be impossible. The manuscript mentions this limitation and hints at possible ways to resolve it.

Second, the method requires that the action space must be incredibly sparse. Moreover, a set of common sense rules are needed which are environment specific and must be hand curated. VirtualHome disallows microwaving a cup for example. It also disallows opening the TV. Both of these are valid actions that happen all the time.

Third, the method requires that objects be unique. If multiple plates, vacuum cleaners, lotions, etc. existed and had to be manipulated, e.g., there is no mechanism to refer to any one plate consistently. The model could generate something like "the first plate" but how to actually execute such an action is far from clear.

This third issue is related to the problem of grounding. Normally, grounding means connecting an abstract concept to something concrete in the environment. All of the grounding that is performed here is by virtue of VirtualHome having unique objects in its environments and the actions not requiring multiple instances of the same object. This is not addressing the problem of grounding. Reviewers requested that grounding be removed from the manuscript. This would significantly enhance it, as the model is inherently incapable of grounding as the authors say: "Indeed, one limitation of our approach is that we do not condition on environment state"

Reviewers took issue with details of the evaluation, which are largely a consequence of the choice of VirtualHome. Sometimes this manifested as strange results like models outperforming humans in terms of correctness. As reviewers pointed out, this is worrisome.

Reviewers were also concerned about the title. It implies that language models are zero-shot planners, but this is not the case. They are instead able to decompose actions into mid-level steps. Reviewers suggested that it would be better to focus the title and tone of the manuscript on extracting task/subtask structures from language models.

The idea presented here, that language models can break tasks into subtasks is interesting. But the manuscript goes a step further and discusses embodied agents which to reviewers appeared to be a reach: there is no grounding and in no sense is the output of the language model any different if the agent is embodied. Even the most positive reviewers felt that discussing embodied agents is unhelpful: it would be better to focus on task/subtask structures. And indeed, this would be more general. All of the concerns that reviewers had around the evaluation would be alleviated by focusing on a language task instead. And the effect of a narrow space of actions, constraints on those actions, and multiple objects of the same class, could be evaluated and reported. Even if the authors had to collect such a corpus, given the difficulties they describe in evaluating on VirtualHome, this would be less of a burden. This could be a strong submission in the future.
Decision: Reject