Inner Monologue: Embodied Reasoning through Planning with Language Models
Authors: Wenlong Huang
Fei Xia
Ted Xiao
Harris Chan
Jacky Liang
Pete Florence
Andy Zeng
Jonathan Tompson
Igor Mordatch
Yevgen Chebotar
Pierre Sermanet
Tomas Jackson
Noah Brown
Linda Luu
Sergey Levine
Karol Hausman
brian ichter
Venue: CoRL 2022 Poster (2022)
Keywords: Large Language Models
Grounding Models
Multi-modal Control
Task and Motion Planning
Mobile Manipulation

Abstract: Recent works have shown how the reasoning capabilities of Large Language Models (LLMs) can be applied to domains beyond natural language processing, such as planning and interaction for robots. These embodied problems require an agent to understand many semantic aspects of the world: the repertoire of skills available, how these skills influence the world, and how changes to the world map back to the language. LLMs planning in embodied environments need to consider not just what skills to do, but also how and when to do them - answers that change over time in response to the agent's own choices. In this work, we investigate to what extent LLMs used in such embodied contexts can reason over sources of feedback provided through natural language, without any additional training. We propose that by leveraging environment feedback, LLMs are able to form an inner monologue that allows them to more richly process and plan in robotic control scenarios. We investigate a variety of sources of feedback, such as success detection, scene description, and human interaction. We find that closed-loop language feedback significantly improves high level instruction completion on three domains, including simulated and real table top rearrangement tasks and long-horizon mobile manipulation tasks in a kitchen environment in the real world.

Inner Monologue: Embodied Reasoning through Planning with Language Models
Wenlong Huang', Fei Xia Ted Xiao", Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, Brian Ichter Robotics at Google, equal contribution and alphabetically listed Project website: https://mmemmoollguuggihbbbb
Abstract: Recent works have shown how the reasoning capabilities of Large Language Models (LLMs) can be applied domains beyond natural language processing, such as planning and interaction for robots. These embodied problems require an agent to understand many semantic aspects of the world: the repertoire of skills available, how these skills influence the world, and how changes to the world map back to the language. L.MS planning in embodied environments need to consider not just what skills to do, but also how and when to do them answers that change over time in response to the agent's own choices. In this work, we investigate to what extent L.LMs used such embodied contexts can reason over sources of feedback provided through natural language, without any additional training. We propose that by leveraging environment feedback, LLMs are able to form an inner monologue that allows them to more richly process and plan in robotic control scenarios. We investigate a variety of sources feedback, such as success detection, scene description, and human interaction. We find that closed-loop language feedback significantly improves high-level instruction completion three domains, including simulated and real table top rearrangement tasks and long-horizon mobile manipulation tasks a kitchen environment in the real world.
1
Introduction
Intelligent and flexible embodied interaction requires robots to be able to deploy large repertoires of basic behaviors in appropriate ways, sequence these behaviors as needed for long horizon tasks, and also recognize when to switch to different approach if particular behavior plan is unsuccessful. High-level planning, perceptual feedback, and low-level control are just a few of the sub-tasks that would need to be seamlessly combined together to perform the sort of reasoning required for embodied agent, such as a robot, to intelligently act in the world. While conventionally these challenges have been approached from the perspective of planning (e.g.. TAMP [1]) or hierarchical learning (e.g.. HRL [2]), effective high-level reasoning about complex tasks also requires semantic knowledge and understanding of the world.
One of the remarkable observations in recent machine learning research is that large language models (LLMs) can not only generate fluent textual descriptions, but also appear to have rich internalized knowledge about the world [3, 4, 5, 6, 7J. When appropriately conditioned (e.g., prompted), they can even carry out some degree of deduction and respond to questions that appear to require reasoning and inference [8, 9, 10, 11, 12, 13). This raises intriguing possibility: beyond their ability interpret natural language instructions, can language models further serve as reasoning models that combine multiple sources of feedback and become interactive problem solvers for embodied tasks, such as robotic manipulation?
Prior studies show that language helps humans internalize our knowledge and perform complex relational reasoning through thinking in language [14, 15, 16, 17, 18). Imagine the "inner monologue" that happens when a person tries to solve some task: "I have to unlock the door; let me try to pick up the key and put in the lock... no, wait, it doesn't fit, I'll try another one... that one worked, now can turn the key." The thought process in this case involves choices about the best immediate action to solve the high-level task ("pick up the key"), observations about the outcomes of attempted actions ('it doesn't 't fit"), and corrective actions that are taken in response to these observations ('T'll try another one"). Inspired by the human thought process, we propose that such inner monologue is natural framework for incorporating feedback for LLMs.
6th Conference on Robot Lemingg (CoRI. 2022), Auckland, New Zealand.

---

Robot Planning & Interaction
Grounded Closed-Loop Feedback
Robot Environments
Bobut
Can you bring me the drink from the table?
M
Subot
Action: 'go table"
Desoriptio
sse: coke, water. chocolate bar.
Do you want water or coke?
Coke please.
Sobet
(b)
Action: "iick up the coke'
Guoczess Dentcttor
Action was not successful.
Action: "pick up the coke'
Success Denector
dh
Action was successful
Action: "bring it to you"
(c)
Figure 1: Inner Monologue enables grounded closed-loop feedback for robot planning with large language models by leveraging a collection of perception models (e.g., scene descriptors and success detectors) in landem with pretrained iangunge-conditionne
robot skills. Experiments show system can reason and replan to accomplish complex long horizon tasks for (a) mobile manipulation and (b,c) tabletop manipulation both simulated and real settings. Our work studies these questions by combining LLMs with various sources of textual feedback, only utilizing few-shot prompting without any additional training. We observe that similarly to recent work [19], natural language provides a universal and interpretable interface for such grounding of model communication and allows them to incorporate their conclusions in an overarching inner monologue driven by language model. While prior work has investigated using language models as planners [20, 21] or incorporating multimodal-informen
perception through language [19], to the best of our knowledge no work has studied the critical link of not only planning with language, but also informing embodied feedback with language, which we investigate in this work.
Specifically, we study methods and sources of feedback for closing the agent-envirrnment loop via an inner monologue and their impact on downstream execution success and new capabilities arising from such interaction. In particular, language-condinn conditionnd semantic classification or manguage-baseer scene description, together with feedback provided by a human user that the robot is cooperating with. To execute the commands given by user, the actions are chosen from a set of pre-trained robotic manipulation skills together with their textual descriptions that can be invoked by language model. Our proposed system Inner Monologue chains together these various components (perception models, robotic skills, and human feedback) in a shared language prompt, enabling successfully perform user instructions.
combine multiple perception models that perform various tasks such as
Finally, we show that Inner Monologue, without requiring additional training beyond a frozen language model and pre-trained robotic skills, can accomplish complex, long-horizon, and unseen tasks in simulation as well as on two real-world robotic platforms. Notably, we show that it can efficiently retry under observed stochastic failure, replan under systematic infcasibility, or request human feedback for ambiguous queries, resulting in significantly improved performance in dynamical environments. As demonstration of the versatility of LLMs and grounded closed-loop feedback, we additionally show several surprising capabilities emerging from the inner monologue formulation, including continued adaptation to new instructions, self- -ropopeo goals, interactive scene understanding, multilingual interactions, and more.
2 Related Work
Task and Motion Planning. Task and motion planning [22, 23] requires simultancously solving a high-level, discrete task planning problem [24, 25, 26], and a low-level, continuous motion planning problem [27]. Traditionally, this problem has been solved through optimization [28, 29] or symbolic reasoning [24, 26]. but more recently machine learning has been applied to aspects of the problem via leamed representations, leamed task- primitiiee, and more [30, 31, 32, 33, 34, 35, 36, 37, 38). Some works utilize language for planning and grounding [39, 40, 41, 42, 43, 44]. Others have approached the problem through hierarchical learning [45, 46, 34, 47, 48, 49, 50]. In this work, we leverage pre-trained LLMs and their semantic knowledge, along with trained low-level skills, to find feasible plans.
Task Planning with Language Models. Various prior works have explored using language as a space for planning [51, 52, 53, 20, 54, 21]. Some methods use prompt structure, self-talk, or discussion between

---

Success Detection In contoxt
Passive Scone Description context
Active Scana Description In context:
Robot Action: up
Robot Action:
Robot Action:
Success:
Scane: soda, coke, energy
Robot the drawer open?
Robot Action Bring it to user
Robol Action: pick up energy ber
Human: The drawer is closed.
Robot Action: Open the drawer
Figure 2: Various types of textual feedhack. Success Detection gives task-specific task completion information, Passive Scene Description gives structured semantic scene information at every planning step, and Active Scene Description gives unstructured semantic information only when queried by the LLM planner.
experts to reason about plans or semantic concepts [55, 19, 10, 11]. Similar to ours are recent task planning approaches that leverage pre-trained autoregressive LLMs decompose abstract, high-level instructions into a sequence of low-level steps executable by an agent [20, 21] in a zero-shot manner. Specifically, Huang et al. [20] prompt GPT-3 [9] and Codex [56] to generate action plans for embodied agents, where each action step is semantically translated to an admissible action with a Sentence-RoBERTa model [57, 58). SayCan [21] instead grounds the actions by multiplying each candidate action's 's probability under FLAN [59] with the action's 's value function, which serves as a proxy for affordance [34]. However, both approaches effectively produce the plan while assuming that each proposed step is executed successfully by the agent. As a result, these approuches may not be robust in handling intermediate failures in dynamic environments or with poor lower level policies. We explore in Inner Monologue ways to incorporate grounded feedback from the environment into the LLM as we produce each step in the plan.
Fusing Vision, Language, and Control in Roboties. Various works have investigated strategies for the challenging problem of fusing vision, language, and control [60, 61, 62, 63, 64, 65, 66]. Some works have been trained directly for language-basee interaction in robotic tasks [67, 68, 69, 70]. Recent large visual- language models (e.g., CLIP (71J) have been trained on joint image(s) and corresponding text captions via variants of masked language modeling objective [72, 73, 74, 75], contrastive loss [76, 77, 71] or other supervised objectives[78 [78, 79j. CL.IP has been employed in several robotics and embodied settings in zero-shot manner [80], or combined with Transporter networks [81] as in CLIPort [82]. Finally, Socratic Models [19] proposes the combination of different foundation models (c.g., GPT-3 [9]. ViLD [83]) and language-conditiones conditioned policies, using language as the common interface. While Socratic Models has been demonstrated on tabletop object manipulation task, Inner Monologue examines additional challenges for robots operating dynamic environments, which require closed-loop feedback to the planner.
3 Leveraging Embodied Language Feedback with Inner Monologue
We consider the setting where embodied robotic agent attempts to perform high-level natural language instruction i. This robotic agent is only capable of executing short-horizon skills from a library of previously trained policies na E TI with short language descriptions &a. which may be trained with reinforcement learning or behavioral cloning. The "planner," which is pretrained L.M [20, 21], attempts to find sequence of skills to accomplish the instruction. To observe the environment, the planner has access to textual feedback from the environment that can be appended to the instruction or requested by the planner. Our work studies to what extent the LLM planner is able to reason over and utilize such feedback to "close the loop" with the environment and improve planning.
3.1 Inner Monologue
We formulate an "inner monologue" by continually injecting information from the various sources of feed- back into the LLM planning language prompts as the robot interacts with the environment. While LLMs have demonstrated exceptional planning capabilities for embodied control tasks [20], prior works have found crucial to ground LLM predictions with external components such as affordance functions [21] in order to produce useful plans that are executable by robots. However, L.MMs used in this context have thus far remained one-directional providing list of skills, without making corrections or leveraging opportunities to replan accordingly. In contrast, Inner Monologue studies settings where grounded environment feedback
3

---

al blocks mismatching bowls. Scene: There yollow block, yallow bowl, blue
Rearangement (Sim)
the yellow block on bowl
block, blue bow, rod block. red bowl.
Tablctop
Roboot My goal l'ylllw block blue bowf, 'red
Initial
block yallow bowf, biuu block red bowif].
AcTiieed
Robott Pick up yellow block and place in blue bowl.
Sceene: You achieved (yellow block in blue bowf"
Actroved
Robott: ined red block yellow bowl. blue block in
True
Roboott Plck up red block and place yellow bowl.
Object Recognition
Tesk-Progrest Scane Description
Success Detection
Rearrangmmen (Reel)
Human: Sort fruits and condiments in different pletes
Action: put the egg on the blue plate
Scene: objects (apple. egg. plum, pizza, coolde, selt. mustard, ketchup, lemonn)
Tabletop
Scene: occluded objects D
Roboot: robott pickplaceettrreeeeeeeooooo bereewbbbyyyyyy green plete)
Robot: pisppppcccklccaaaaii shaker, blue plate)
Success Detector: False.
Robott robot, peootp.cckpaccssaai shaker, blue plate)
True
False
Robott: roloot.topp
Object Recognition (wul Potential Occlusion)
Detection
Human: Bring me drink from the table
Action:
Kitchen Mobile Manipulatiin (Real)
Robott: go to the table
pick up the coke
Scone: |soo a coko and lime sodo
drink
you
Jnittal
up coke
Human Something with caffeiine
Robolt: Pick up coke
Success Deleccor: False
Lin
Robott: Pick up
Robot: Bring to you
False
Robott: Put down the ooke
Robott: Done
Object Recognition
Success Detection
Affordance Grounding
Figure 3: Different instantiations of Inner Monologue in three distinct domains - simulated tabletop rearrangement (top), real-world tabletop rearrangement (middle). and real-world kitchen mobile manipulation (bottom). Each domain uses different prompts and different feedhack models. Sharing across the domains is the same Inner Monologue formu- lation that uses a pre-trained langauge model to take in human instruction and decompose into sequence ac- tionable steps by the agent, while accounting for injected embodied feedback from different models, such as object rec- ognizers and success detectors. In real-world world kitchen mobile manipulation domain (buttem), we additionally ground the actions using pre-trained affordance functions built in [21], which do not communicate back to the language model.
is provided directly to the LLM in closed-loop fashion. This promotes improved LLM reasoning in complex long-horizon settings, even before any external affordance- grounding methods are applied.
Our analysis assumes textual feedback is provided to the planner, but does not assume a single specific method of fusing LLM planning with low-level robotic control or specifie method of extracting environment feedback into language. Rather than focusing on particular algorithmic implementation, our aim to provide a case study on the value of incorporating different types of feedback into closed-loop LLM- -aseed planning. Thus, Inner Monologue in See 4 utilizes language feedback within separate systems that incorporate different L.L,s, different methods of fusing planning with control, different environments and tasks, and different methods of acquiring control policies. We note that in our specific implementations of Inner Monologue, we use pre-trained L.MMs for planning that are not finetuned, but rather evaluated solely with few-shot prompting; the full prompts can be found in the Appendix.
3.2 Sources of Feedback
In theory any type of environment feedback can inform the LLM planner, as long as it can be expressed through language. We focus on the specific forms of feedback shown in Fig 2: (1) task-specific feedback, such as success detection, and (2) scene-specific feedback (either "passive" or "active"), which describes the scene. Specific instantiations and implementation details of each type of feedback can be found in Sec 1. See 4.2, and Sec 4.3 respectively for each domain.
Success Detection. The Success feedback gives binary "yes" or "no" response in language form, specifying whether the low-level skill Ty has succeeded. Engineered success detectors can operate on ground -tuth state in simulation, while learned success detectors can be trained real examples of successes and failures in the real world [84, 85, 86, 87, 88].
Passive Scene Description. We refer broadly to any sources of scene feedback that are consistently and automatically injected into the L.LM prompt as Passive Scene Description, which also typically follow some structure. One common type of such feedback is object recognition [89, 90, 91, 92] that returns list of

---

present objects, to which we refer as Object feedback. We also demonstrate the use of a task progress scene description in the simulated tabletop rearrangement environment, which we refer as Scene feedback.
Active Scene Description. As the proactive counterpart, Active Scene Description encompasses sources of feedback that are provided directly in response active queries by the LLM planner, which answered either by person, or by another pretrained model, such as Visual Question Answering (VQA) model [93, 94, 95, 96]. Unlike the passive counterpart which are strictly structured and narrow in their scope, this feed- back allows the planner to actively gather information relevant to the scene, the task, or even preferences of the user. The combined output we send the LLM planner includes both the LLM LLM- generated question along with the response. As we aim to investigate whether and how LLM planner can incorporate such feedback and wish study both structured VQA-style human feedback as well as unstructured human preferences feedback, we only consider human- provided response in this work, which we refer to as Human feedback.
4 Experimental Results
In order to study how different sources of environment feedback can support rich inner monologue that enables complex robotic control, we study different Inner Monologue implementations in three environments, each with different LLM and different sources of feedback from the environment: 1) simulated tabletop manipulation (Sec 4.1). 2) real-world tabletop manipulation (Sec 4.2), and 3) real-world mobile manipulation in office kitchen (Sec 4.3). For more details about the experiment setup and results, please refer to the Appendix.
4.1 Simulated Tabletop Rearrangement
We experiment with Ravens-based [81] environment, where robotic arm with a gripper is tasked with rearranging blocks and bowls in some desired configuration, specified by natural language. We evaluate each method on four seen tasks and four unseen tasks, where seen tasks may be used for training (in the case of supervised baseline) or used as few-shot prompting.
This instantiation of Inner Monologue uses (i) InstructGPT [9, 97] for planning [20, 211, (ii) scripted modules to provide language feedback in the form of object recognition (Object), success detection (Success), and task-progress scene description (Scene), and (iii) a pre-trained language-conddtioned pick-and-place primitive (similar to CL.IPort [82] and Transporter Nets [81]. Object feedback informs the list of present objects and Success feedhack informs the success/failure of the most recent action. However, consider the task of stacking multiple blocks, because the unfinished tower of blocks may be knocked over by the robot, it is also critical reason about overall task progress. Therefore, task-progress scene description (Scene) describes the semantic sub-goals inferred by the LLM towards completing the high-level instruction that are achieved by the agent so far.
We additionally compare to multi-task CLIPort directly trained on bong-hoiizon task instructions. Because CLIPort is single-step policy and does not terminate spontaneously during policy rollout, we report CLI- Port evaluations with oracle termination (i.e., repeat until oracle indicates task completion) and fixed-step ter- mination (i.e., repeat for 15 steps). To simulate real-world disturbances and evaluate the system's 's robustness to disturbances, we add Gaussian noise to multiple levels of the system test time: A ((,33) for pixel ob- servation, NV (0,2.5) for policy primitive (i.e., pick-place pixel heatmaps), N (0.0.02m for place locations.
+LLM
+nner Monologue
Tasks
CLIPort
+oracle
Object
Object Success
Object Scene
"Pick and place"
24.0%
74.0%
80.0%
90.0%
94.0%
"Stack all the blocks"
2.0%
32.0%
4.0%
10.0%
26.0%
Seen Tasks
"Put all the blocks on the [x] comer/side"
2.0%
32.0%
30.0%
28.0%
30.0%
"Put all the blocks in the [x] bowl"
32.0%
94.0%
52.0%
46.0%
56.0%
"Put all the blocks in different comers"
0.0%
0.0%
20.0%
20.0%
26.0%
"Put the blocks in their matching bowls"
0.0%
0.0%
56.0%
70.0%
82.0%
Unseen Tasks
"Put the blocks on mismatched howls"
0.0%6
0.0%
62.0%
76.0%
86.0 %
"Stack all the blocks on the [x] corner/side"
0.0%
0.0%
0.0%
4.0%
6.0%
Table 1: Success rates averaged across 50 episodes in simulated pick-and-place. CLIPort + oracle indicates that CLIPort was provided "termination" oracle. LLM-infomed foodback effectively enable rerriin/rcplanning in the presence of test-time disturbances, while enjoying the generalization benefits LLMs to unseen tasks.
5

---

Analysis. As shown in Table I, Inner Monologue effectively enables retrying and replanning in the face of test-time disturbances, where Object Scene performs the best because of its ability to keep track of sub-goal conditions. Furthermore, this performance directly translates to unseen tasks by leveraging rich semantic knowledge of I.LM. Finally, we observe that non-hierarchical and solitary systems such as CI.IPort (i) struggle at generalizing to unseen long-horizon tasks under test-time disturbances, and (ii) on training tasks, an oracle is also often required to indicate task completion for good performance.
4.2 Real-World Tabletop Rearrangement
We evaluate Inner Monologue on a real-world robot platform designed to resemble the simulation experiments. This instantiation uses (i) InstructGPT [9, 97] for planning, (ii) MDETR [98] for open-vocab object recognition (Ohject) (iii) heuristics on the object bounding box predictions from MDETR for Success Detection (Success), and (iv) suction-based pick-and-place motion primitive that uses LLM to parse target objects from a language command (e.g.. given by the planner).
We investigate two tasks: (i) 3-block stacking task where 2 blocks are already pre-stacked, and (ii) long-horizon sorting task to place food in one plate and condiments in another (where categorizing food versus condiments is autonomously done by the LLM planner). In additional to additional challenges of real-world perception and clutter, we artificially inject Gaussian noise into the policy actions (i.e., add standard deviation 0-4mm clipped at 2x) to stress test recovery from failures via replanning with grounded closed-loop feedback. Results are presented in Table 2.
LLM
Inner Monologue
Task Family
Object
Object
Success
Object + Success
Finish 3-block stacking Sort fruits from bottles Total
20%
40%
40%
100%
20%
50%
40%
80 %
20%
45%
40%
90%
Table 2: Success rates averaged across 10 runs in real-world pick-and-place. We observe significant improvement in Inner Monologue with Object and Success feedback, with the two feedback being complementary to each other.
Analysis. We compare to variants with only Object or Success feedhack, as well as an open-loop variant ("LLM Object") that only runs object recognition once the beginning of the task (similar to the system demonstrated in [19]). The partial 3-block stacking task highlights an immediate failure mode of the open-loop bascline, where the initial scene description struggles to capture complete representation of the scene (due to clutter and occlusion) to provide as input to the multi-step planner. As a result, the system only executes one mick-and-place action and cannot recover from mistakes. To address these shortcomings, Inner Monologue (Object + Success) leverages closed-loop scene description and success detection after each step, which allows it to successfully replan and recover from policy mistakes.
4.3 Real-World Mobile Manipulator in a Kitchen Setting
We implement Inner Monologue a robotic system using the kitchen environment and task definitions de- scribed in SayCan [21]. The Everyday Robots robot, a mobile manipulator with RGB observations, is placed in an office kitchen to interact with common objects using concurrent [99] continuous closed-loop control.
The baseline, SayCan [21], is a method that plans and acts in diverse real world scenarios by combining an LLM with value functions of control policies. While SayCan creates plans that are grounded by the affordances of value functions, the LLM predictions in isolation are never given any closed-loop feedback.
We use instantiation of Inner Monologue that uses (i) PALM [8] for planning. (ii) value functions from pre-trained control policies for affordance grounding [21], (ii)) a learned visual classification model for Success feedback, (iv) provad-provided Object feedback, and (v) pre- pre-trained control policies for relevant skills in the scene. We also perform a case study where we allow the agent to ask questions and source Human feedback directly; results are shown in Fig 5a and the Appendix.
We evaluate on 120 runs over three task families: (1) four manipulation tasks, (2) two dexterous manipulation tasks utilizing drawers, and (3) two long-horizon combined manipulation and navigation tasks. We consider both cases with and without manually-adde policy executions that cause skill policy rollouts to fail. While these failures occur naturally even without perturbances, the adversarial disturbances creates a consistent comparison between methods that requires retrying or replanning to accomplish the original instruction.
adversarial disturbances during control

---

+Inner Monologue
Taltar
Contel Fatlam
Fallamm (locen)
Task Family
SayCan
Success
Object + Success
SeyCan
No Disturbances
IM wf Buxcesa
Manipulation
50.0%
62.5%
75.0% %
Mobile Manipulation
50.0%
50.0%
75.0% %
IM w Success Object
Drawers
83.3%
83.3%
100.0%
With Disturbances
IM SeyCan
Manipulation
12.5%
25.0%
33.3%
Mobile Manipulation
0.0%
25.0%
75.0%
Drawers
0.0%
44.4%
44.4% %
Total
30.8%
48.7%
60.4%
Table 3: Averaged success rate across 120 evaluations on several task families in our real-world mobile manipulation environment. We consider standard setting and adversarial setting with exter- nal human disturbances. all cases, IJ M--informed embodied feedback is shown effective improving robustness the system, especially when low-level policies are prone to failures.
Disturbance
0%
25%
50%
75%
100%
120 evaluations. When disturbances are added (red), only the Inner Monologue variants consistently complete the instructions.
Figure 4: Failure
Analysis. Without adversarial disturbances, the baseline SayCan performs reasonably all tasks, yet incorporating LLL--iffomed feedback in Inner Monologue allows further improvement by effectively retrying or replanning under natural failures. The most notable difference is in the cases with adversarial disturbances. Without any LLM--iformed feedback SayCan has success rate close to 0% since LLM always assume successful execution of previous skills. Inner Monologue significantly outperforms SayCan because of its ability to invoke appropriate recovery modes depending on the environment feedback. Analysis on the failure causes indicates that Success and Object feedback can reduce LLM planning failures and thus overall failure rate, albeit at the cost of introducing new failure modes to the system.
4.4 Plan Generalization Capabilities
Although LLMs can generate fluent continuation from the prompted examples, we surprisingly find that, Inner Monologue demonstrates many impressive reasoning and replanning behaviors beyond the examples given in the prompt. Using pre-trained LLM as the backbone, the method also inherits many of the appealing properties from versatility and general-purpose language understanding. In this section, we demonstrate few of these capabilities; additional capabilities are shown in Appendix (Fig ?? and Fig ??).
Continued Adaptation to New Instructions. Although explicitly prompted, the I L.M planner can react to human interaction that changes the high-level goal mid-task. Fig 5a demonstrates a challenging case, where Human feedback changes the goal during the plan execution, and then changes the goal yet again by saying "finish the previous task". In another instance, despite not being explicitly prompted to terminate after human says "please stop", the LLM planner generalizes to this scenario and predicts "done" action.
Self-Proposing Goals under Infeasibility, Instead of mindlessly following buman-gives instructions, In- Monologue also propose alternative goals achieve when the previous goal becomes infeasible. In Fig 5b, to solve the task "put any two blocks inside the purple bowl", while the first attempted block intentionally made heavy for the robot, Inner Monologue proposes to "find lighter block" and successfully solves the task.
Multilingual Interaction. Pre-trained LLMs are known to be able to translate from language to another, without any finetuning. We observe that such multilingual understanding also transfers to the embodied settings. Fig 5c shows case when an instruction is in Chinese, the LLM planner can still correctly interpret it, re-narrate it as concrete goal to execute in English, and accordingly replan its future actions. Occasionally, find that this capability even extends symbols and emojis.
Retrospective Scene Understanding. We also observe that Inner Monologue demonstrates retrospective scene understanding based on past actions and environment feedback, which requires temporal and embodied reasoning. In Fig 5d, after series actions, we can turn ask questions about the resulting scene, again structure that has not appeared in the prompt.
7

---

Humant throw the snack counterr
plock boon, Mlok ikmmon Pat any hwo nlokle in purple howe
Rotoot the close cour and makk:
Humant actually changeed my mind Tiwant
Vihu se soacks are the courter
nu sarple howe 1]
lidbos actions the
place the purple
you to theow mony someth on the babll
gio to the tabll and usi: What snorkko
Sconus Completed Blud block is purple bowel gie
HHama neveer mind mwint you to frish your
previous tack
on the 2
dhice coutee
zurpleoood
(entort/mummmm mee grugh 7i-tt tt-rp att thithe"
go the close counter and asic Whet
Scane: The purple block noo heaey bo plcked up.
anackss on the counter??
a your maiy plan?
Humann apple and chip bag
plck up apple and continue
REbiT atkort MoTThiiaa Bl and Black ITatttalTT
6. puf the apple
go
and continue
Scaner Completed ('wwo bloda busidle purple
cowl
(a) Comtinued Adaettation to New Instructions
Self Proposing Goals under Infensibility
Scons: The boock orange borl grng block. Humani Put blocia insies the parple boow
parple
Scaoe There is yellow brow yelloww block blue block, blve
Akymars the blocks bowis with mitomnnshing colion.
Rabooe thought Finll cosee in ""helnen blnckii in the blun boutt" TRi, biock ln yeltzo howel bosst
Robot actions Pick the rod block and place it insidd purple bow..
pellowo
placeilt
Robot actions Pick the purpple troos and place insicle the purple brsw..
Scanee Completed "he brlnce in the tn coouk))
pedaot tesurtt H. bebce to not
toows Gurmant 4 8 The 75
purple
haaky pickted up
your peorplen??
thought
Roboot actionn Pick the baock and place traide purple
Robot thoughts rnsed to find ai light block,
plobot
ITTT WhwFF ofjectis ane the pirpll
Scano Completed (he block in blue powd "Buu Wir""aa da (BWV)) Robot
parple
bium block.
turmnn Whene le the purp e block?
Boobot thougghe purple block
"Trandaatonn "I changed my plan. Please too put the blue black inside the blue bowl""
2obot thougge The prange brow la
(c) Multilinuual Interaction
(d) Interactive Scone Understanding
Figure S: Informing LLM with embodied feedhack enables many generalization capabilities, all of which are achieved without similar prompted examples. For instance, Inner Monologue can continually adapt to new instructions given by humans, propose new goals to achieve when faced with infeasibility for the previous plan, interact with humans in different natural languages, and answer questions about the current scene given past actions and feedback.
Despite the appealing findings about these generalization capabilities, we observe that they are of varying levels of consistency when no similar examples have been provided in the prompt, likely limited by the current capabilities of the language models. However, we believe that further investigations into these behaviors and addressing their limitations would each lead to exciting future directions.
5 Conclusions, Limitations & Future Works
In this work, we investigated the role that environment feedback plays for LLMs reasoning in tasks involving embodied robotic planning and interaction. We presented a general formulation Inner Monologue that combines different sources of environment feedback with methods fusing LLM planning with robotic control policies and studied its instantiations in three distinct domains. We found that environment feedback significantly improves high-level instruction completion, especially in challenging scenarios with adversarial disturbances. Finally, we analyze generalization capabilities of Inner Monologue that highlight how closed-loop language feedback enables replanning even in complex unseen settings.
Limitations. In Sec 4.1 and Sec 4.3, we assume access to oracle scene descriptors in the form of human observers or scripted systems to provide textual description back to the LLM planner. We study the viability of learned systems scene description and object recognition in Appendix Table ??. As for failure modes, Inner Monologue may fail due to several sources of errors: (1) success detections, (2) LLM planning errors, and (3) control errors. False negative predictions from the success detector lead to additional retry attempts, while false positive predictions add adversarial partial observability to the environment. In some instances, we found that the I.LM planners ignored the environment feedback and still proposed policy skills involving objects not present in the scene. Additionally, the performance of low-level control policies limits not only overall high-level instruction completion performance, but also limits the scope of tasks that the LLM is able to plan actions for.
Future Works Several fronts can be improved by future works. First, with advances in image/video captioning and visual-question answering. a fully automated system of Inner Monologue can be implemented without human in the loop as an oracle. Second, improvements can be made on how to aggregate potentially inaccurate sources of information, such as using text to describe the uncertainty of the feedback modules, or including additional feedback modules for safety and ethics for the proposed plans. Finally, enabling low-level control policies to take as input the textual feedback by L.I also leads to exciting future directions.
8

---

Acknowledgments
The authors would like to thank Kanishka Rao and Vincent Vanhoucke for valuable feedback and discussions. In addition, the authors would like to acknowledge the large team who built [21], upon which we construct our Kitchen Mobile Manipulation experiments.
References
[1] L. P. Kaelbling and T. Lozano-Pérez. Integrated task and motion planning in belief space. The International Journal of Robotics Research, (209-10)111441111 1227, 2013.
[2] A. G. Barto and S. Mahadevan. Recent advances in hicrarchical reinforcement learning. Discrete event dynamic systems, 13(1):41-77, 2003.
[3] F. Petroni, T. Rocktüschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel. Language models as knowledge bases? arXiv preprint (1X.::19990000666 2019.
[4] Z. Jiang, F. F. Xu, J. Araki, and G. Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
[5] J. Davison, J. Feldman, and A. M. Rush. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 conference empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 1173-1178, 2019.
[6] A. Talmor, Y. Elazar, Y. Goldberg, and J. Berant. olmpics-on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743-758, 2020.
[7] Roberts, C. Raffel, and Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:200..08910. 2020.
[8] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint wwXiv:2204022311 2022.
[9] T. Brown, B. Mann, N. Ryder, M. Subbiah. J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, Sastry, A. Askell, et al. Language models are few-shot learners. Advances neural information proessing systems, 33:1877-1901, 2020.
[10] J. Wei, X. Wang. D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting clicits reasoning in large language models. arXiv preprint 2000 2022.
[11] T. Kojima, S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205. /9916, 2022.
[12] A. K. Lampinen, I. Dasgupta, S. C. Chan, K. Matthewson, M. H. Tessler, A. Creswell, J. L. McClelland, X. Wang, and F. Hill. Can language models leam from explanations in context? arXiv preprint arXiv:2204.02329 2022.
[13] M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowyez, M. Bosma, D. Luan, al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint 1..01221 2021.
[14] S. Vygotsky. Thought and language. MIT press, 2012.
[15] P. Carruthers. Thinking in language?: evolution and a modularist possibility. Cambridge University Press, 1998.
[16] L. Vygotsky. Tool and symbol in child development. The vygotsky reader, 1994.
[17] L. S. Vygotsky, Play and its role in the mental development of the child. Soviet psychology. 5(3): 6-18, 1967.
[18] C. Colas, Karch, C. Moulin-Frier, and P.- Oudeyer. Vygotskian autotelic artificial intelligence: Language and culture internalization for human-like ai. arXiv preprint arXiv:2206.00 1344, 2022.

---

[19] A. Zeng, A. Wong. S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.
[20] W. Huang, P. Abbeel, D. Pathak, and Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning. PMLR, 2022.
[21] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, I.. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan,
A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, and M. Yan. Do as can and not as say: Grounding language in robotic affordances. In arXiv preprint vrXiv:2204.0169) 2022.
[22] L. P. Kaclbling and T. Lozano-Pérne Pérez. Hicrarchical planning in the now. In Workshops at the Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010.
[23] Srivastava, E. Fang, I.. Riano, R. Chitnis, S. Russell, and P. Abbeel. Combined task and motion planning through an extensible plamner-independent interface layer. In 2014 IEEE international conference on robotics and automatioo (ICRA), 2014.
[24] R. E. Fikes and N. J. Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, 1971.
[25] E. D. Sacerdoti. A structure for plans and behavior. Technical report, SRI International, Menlo Park Californa Artificial Intelligence Center, 1975.
[26] Nau, Y. Cao, A. Lotem, and H. Munoz-Avila. Shop: Simple hierarchical ordered planner. In Proceedings of the 16th international joint conference on Artificial intelligence, 1999.
[27] S. M. LaValle. Planning algorithms. Cambridge university press, 2006.
[28] M. Toussaint. motion planning. In Fhenty-FFuurth International Joint Conference Artificial Intelligence, 2015.
programming: An optimization based approach to combined task and
[29] M. A. Toussaint, K. Allen, A. Smith, and B. Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. Robotics; Science and Systems Foundation, 2018.
[30] Eysenbach, R. Salakhutdinov. and S. Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances in Neural Information Processing Systems, 2019,
[31] D. Xu, S. Nair, Y. Zhu, Gao, A. Garg, L. Fei-Fei, and S. Savarese. Neural task programming: Learning to generalize across hierarchical tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018
[32] D. Xu, R. Martín-Mrtúnn D.-A. Huang. Y. Zhu, S. Savarese, and L. F. Fei-Fei. Regression planning networks. Advances in Neural Information Processing Systems, 32, 2019.
[33] T. Silver, R. Chitnis, N. Kumar, W. McClinton, T. Lozano-Perez, P. Kaelbling, and J. Tenenbaum. Inventing relational state and action abstractions for effective and efficient bilevel planning. arXiv preprint 2xXi:2203.09634 2022.
[34] D. Shah, P. Xu, Y. Lu, T. Xiao, A. Toshey, S. Levine, and B. Ichter. Value function spaces: Skill-centric state abstractions for long-horizon reasoning. ICLR, 2022. URL
https:///penrevview
[35] A. Srinivas, A. Jabri, P. Abbeel, S. Levine, and C. Finn. Universal planning networks: Learning generalizable representations for visuomotor control. In International Conference on Machine Learning, pages 4741-4741. PMLR, 2018.
[36] T. Kunutach, A. Tamar, G. Yang, S. J. Russell, and P. Abbeel. Learning plannable representations with causal infogan. Advances in Neural Information Processing Systems, 31, 2018.
10

---

[37] A. Akakzia, C. Colas, P.-Y. Oudeyer, M. Chetouani, and O. Sigaud. Grounding language to autonomously-acqqured skills via goal generation. In International Conference on Learning Representations, 2021. URL hotos////ppppoeeeeeeeeeee/ooooot ISKMHG.
www..net/formmmd=cche..
[38] S. Pirk, K. Hausman, A. Toshev, and M. Khansari. Modeling long-horizon tasks sequential interaction landscapes. arXiv preprint arXiv:2006.04843 2020.
[39] T. Kollar, S. Tellex, D. Roy, and N. Roy. Toward understanding natural language directions. In 2010 5th ACM/IEEE International Conference on Human-Ron Roboot Interaction (HRI), pages 259-266, IEEE, 2010.
[40] S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 1507-1514, 2011.
[41] M. Bollini, S. Tellex Thompson, N. Roy, and D. Rus. Interpreting and executing recipes with a cooking robot. In Experimental Robotics, pages 481 481-495. Springer, 2013.
[42] S. Tellex, R. Knepper, A. Li, D. Rus, and N. Roy. Asking for help using inverse semantics. 2014.
[43] T. Kollar, S. Tellex, D. Roy, and N. Roy. Grounding verbs of motion in natural language commands to robots. In Experimental robotics, pages 31-47. Springer, 2014.
[44] V. Blukis, Y. Terme, E. Niklasson, R. A. Knepper, and Y. Artzi. Learning to map natural language instructions to physical quadcopter control using simulated flight. arXiv preprint arXiv:1910.08 10.09664, 2019.
[45] S. Nair and C. Finn. Hierarchical foresight: Self-supervised learning of Long-horizon tasks via visual subgoal generation. ArXiv, awss/109.05555. 2020.
[46] F. Xia, C. Li, R. Martín-Martin, O. Litany, A. Toshev, and S. Savarese. Relmogen: Integrating motion generation in reinforcement learning for mobile manipulation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021.
[47] Li, F. Xia, R. Martin-Martin, and S. Savarese. Hrl4in: Hierarchical reinforcement leamning for interactive navigation with mobile manipulators. In Conference on Robot Learning, 2020.
[48] Y. Jiang, S. Gu, K. Murphy, and C. Finn. Language as an abstraction for hierarchical deep reinforcement learning. In NeurlPS, 2019.
[49] D. Hafner, K.-H. Lee, I. Fischer, and P. Abbeel. Deep hierarchical planning from pixels. arXiv preprint arXiv:2206.044 04114, 2022.
[50] S. Mirchandani, S. Karamcheti, and D. Sadigh. Ella: Exploration through learned language abstraction. Advances in Neural Information Processing Systems, 34:29529-2999 29540, 2021.
[51] P. A. Jansen. Visually-grounded planning without vision: Language models infer detailed plans from high-level instructions. arXiv preprint arXiv:2009.14255 14259, 2020.
[52] P. Sharma, A. Torralba, and J. Andreas. Skill induction and planning with latent language. arXiv preprint arXiv:2110.01517.222 2021.
[53] V. Micheli and F. Fleuret. Language models are few-shot butlers. arXiv preprint arXiv:2104.07972. 2021.
[54] S. Li, X. Puig, Y. Du, C. Wang, E. Akyurek, A. Torralba, J. Andreas, and I. Mordatch. Pre-trained language models for interactive decision-making arXiv preprint 2rXiv:2202.01771, 2022.
[55] V. Shwartz, P. West, R. L. Bras, C. Bhagavatula, and Y. Choi. Unsupervised commonsense question answering with self-talk. arXiv preprint 204:2204.005433. 2020.
[56] M. Chen, J. Tworek, H. Jun, Q Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2007033744 2021.

---

[57] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoy- anov. Roberta: robustly optimized bert pretraining approach. arXiv preprint 1XXv:1907.11192 2019.
[58] N. Reimers and L. Gurevych. Sentence- bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908. 10084, 2019.
[59] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. arXiv preprint arXi::2 2021.
[60] C. Paxton, Y. Bisk, Thomason, A. Byravan, and D. Foxl. Prospection: Interpretable plans from language by predicting the future. In 2019 International Conference on Robotics and Automation (ICRA), pages 6942-6948, IEEE, 2019.
[61] S. Stepputtis, J. Campbell, M. Phielipp. S. Lee, C. Baral, and H. Ben Amor. Language-conditioned imitation learning for robot manipulation tasks. Advances in Neural Information Processing Systems, 33:13139-13154 2020.
[62] V. Blukis, R. A. Knepper, and Y. Artzi. Few-shot object grounding and mapping for natural language robot instruction following. arXiv preprint arXiv:2011.073388 2020.
[63] C. Lynch and P. Sermanet. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems, 2021. URL https://arxia: https/////arrio.ooaaaa/////s
[64] Y. Chen, R. Xu, Y. Lin, and P. A. Vela. A joint network for grasp detection conditioned natural language commands. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4576 45722. IEEE, 2021.
[65] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning arXiv preprint arXiv:2204.06252, 2022.
[66] C. Yan, F. Camevale, P. Georgiev, A. Santoro, A. Guy, A. Muldal, C.-C. Hung, J. Abramson, T. Lillicrap, and G. Wayne. Intra-agent speech permits zero-shot task acquisition. arXiv preprint arXiv:2206.03139, 2022.
[67] Kuhlmann, P. Stone, R. Mooney, and J. Shavlik. Guiding a reinforcement learner with natural language advice: Initial results in robocup soccer. In The AAAI-2004 workshop on supervisory control of learning and adaptive systems. San Jose, CA, 2004.
[68] A. Najar, O. Sigaud, and M. Chetouani. Interactively shaping robot behaviour with unlabeled human instructions. Autonomous Agents and Multi-Agent Systems, 34(2):1-35 2020.
[69] P. Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, Andreas, and D. Fox. Correcting robot plans with natural language feedback. arXiv preprint arXiv:2204.05188 2022.
[70] R. Loftin, B. Peng. J. MacGlashan, M. L. Littman, M. E. Taylor, J. Huang, and D. L. Roberts. Learning behaviors via human-delivered discrete feedback: modeling implicit feedback strategies to speed up learning, Autonomous agents and multi- agent systems, 30(1):30-59, 2016.
[71] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8766 PMLR, 2021.
[72] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint urXiv:1810.04800 2018.
[73] J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and- and-languagee tasks. Advances in neural information processing systems, 32, 2019.
[74] Z. Wang. J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 10904, 2021.
[75] A. Suglia, Q. Gao, J. Thomason, G. Thattai, and G. Sukhatme, Embodied bert: A transformer model for embodied, language-guide guided visual task completion. arXiv preprint arXiv:2108.0492. 04927, 2021.
12

---

[76] T. Chen, S. Korbblith, K. Swersky, M. Norouzi, and G. E. Hinton. Big self-supervised models are strong semi-supervised learners. Advances neural information processing systems, 33: 22243-22255 2020.
[77] A. Jain, M. Guo, K. Srinivasan, T. Chen, S. Kudugunta, C. Jia, Y. Yang, and J. Baldridge. Mural: multimodal, multitask retrieval across languages. arXiv preprint arXiv:2009005125, 2021.
[78] J. Sun, D.-A. Huang, B. Lu, Y.H. Liu, B. Zhou, and A. Garg. Plate: Visually gruuuly planning with transformers in procedural tasks. IEEE Robotics and Automation Letters, 7(2):492444930, 2022.
[79] F. Sener and A. Yao. Zero-shot anticipation for instructional activities. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 862-871, 2019.
[80] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi. Simple but effective: Clip embeddings for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14829-14838, 2022.
[81] Zeng, P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020.
[82] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pages 894-906. PMLR, 2022.
[83] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint (IXh::21441132)., 2021.
[84] I. Lenz, H. Lee, and A. Saxena. Deep learning for detecting robotic grasps. The International Journal of Robotics Research, 34(4-5):705-72,, 2015.
[85] F.-J Chu, R. Xu, and P. A. Vela. Real-world multiobject, multigrasp detection. IEEE Robotics and Automation Letters, 3(4):3355-334 -3662,,333,22-- 2018.
[86] D. Kalashnikov, Varley, Y. Chebotar, Swanson, Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint 104iv:2144.08222 2021.
[87] T. Migimatsu and J. Bohg. Grounding predicates through actions. arXiv preprint arXiv:2109.147 2021.
[88] Y. Cui, S. Nickum, A. Gupta, V. Kumar, and A. Rajeswaran. Can foundation models perform zero-shot task specification for robot manipulation? In Learning for Dynamics and Control Conference, pages 893-905. PMLR, 2022.
[89] M. Liang and X. Hu. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3367-3375, 2015.
[90] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances neural information processing systems, 28, 2015.
[91] Z. Zou, Z. Shi, Y. Guo, and J. Ye. Object detection in 20 years: A survey. arXiv preprint 10500590,.00005, 2019.
[92] A. Bachkovskiy, C.-Y. Wang, and H.-Y. M. Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020.
[93] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision, pages 2425-2433, 2015.
[94] L. Zhou, H. Palangi. I.. Zhang, H. Hu, J. Corso, and J. Gao. Unified vasion-language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13041-13049 2020.
13

---

[95] H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020. URL https://arxivv www.:/i..orrg/oooooooo org. pdf/1908 09791 .dff.
[96] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds. et al. Flamingo: a visual language model for few-shot learning. arXiv preprint wXIv:2204.14198, 2022.
[97] L. Ouyannggg.g................... Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203002/55, 2022.
[98] Kamath, M. Singh, Y. LeCun, Synnaeve, I. Misra, and N. Carion. Moetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1799 1790, 2021.
[99] T. Xiao, E. Jang, D. Kalashnikov, S. Levine, J. Ibarz, K. Hausman, and A. Herzog- Thinking while moving: Deep reinforcement learning with concurrent control. arXiv preprint arXiv22004.06089, 2020.
14


Reviews:
[{"review_id": "Hfa9_yMUL8U", "reviewer": "Reviewer_YS48", "summary": "This work extends prior algorithms for robotic planning with LLMs to incorporate feedback, which increases tasks success rate in presence of imperfect low-level control and a dynamic environment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The approach is simple and effective.\n- The types of feedback considered are diverse.\n- The various implementations of the idea using different LLMs, feedback mechanisms, primitive descriptions/language encoding, low-level control primitives in different domains demonstrates the generality of the idea.\n- Video does a great job at motivating and describing the approach, as well as demonstrating the system's capabilities.\n- The figures are informative.\n\nWeakness:\n-  The key weakness in their goal of studying how environment feedback helps planning is in their lack of investigation into feedback handling mechanisms (such as high level replanning). Their experiments only demonstrate retrying until actions are successful. More details in issues section.\n- The key weakness in making the system a fully autonomous language-conditioned system is its reliance on oracle (human or scripted systems) feedback. However, this weakness does not affect their goal in studying how environment feedback helps planning.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Inner Monologue demonstrates a diverse set of systems implementing their key idea of environment feedback to LLM planner. All components of their systems are comprehensively studied except for feedback handling behaviour, which in my opinion is a crucial aspect in studying environment feedback and its impact on downstream action success.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": " - A key limitation of the approach in my opinion is in action failure handling, which only retries actions. However, feedback from the environment could also require major replanning. For instance, failure to throw an object into the trash because it was full, or failure to open a drawer because it is locked. In my opinion, studying \"methods and sources of feedback ... and their impact on downstream execution success\" should go in tandem with studying methods for handling such feedback. While the rest of the system demonstrated different implementations by swapping out the LLM, hardware, low-level action policies, the prompts in the appendix implies that Inner Monologue only retries actions until it is successful."}}, {"review_id": "8owcXoxzCQm", "reviewer": "Reviewer_JWy9", "summary": "This paper proposes using language as a form of embodied feedback for LLM-based robot planning. Three types of environment feedback, i.e. success detection, passive and active scene descriptions, are expressed in language to show how they improve the success rate for planning in simulated and real-world environments.  ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This is the first study that incorporate environment feedback into LLM-based planner. This indicates it is possible to use LLM to represent the knowledge of the robot and reason over the it with feedback represented in language which opens ways to incorporate LLMs as reasoning models for task planning.\n- This paper evaluates different types of feedback in both simulated and real world domains to show the feasibility of the proposed ideas to how it can improve reasoning in long-horizon tasks.\n\nWeakness\n- It is unclear which type of feedback should be active at each time step. It seems that the selection of feedback is hand-crafted for different types of planning steps. But sometimes the hand-crafted feedback may not be sufficient. For example, if the robot fails to put the blue block in the blue tray, the success detector says false, but the block is dropped to a red try, therefore, another scene detector needs to recognize this and inform the planner that the blue block is in the red tray. So, when to instantiate a type of feedback and how to combine multiple feedback are important to utilize feedback.\n- Each experiment only includes a subset of feedback. For example, scene feedback is not used in real world experiments. It is unclear why those types of feedback are omitted. \n- The planner is LLM-based. So it is natural to incorporate feedback as part of the language prompt. It will be very interesting if this language feedback can be incorporated into other types of planners too. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents an interesting case study to incorporating environment feedback into LLM-based planners and shows that these feedback can help task success rate. However, the current implementation still requires heuristics to design which feedback to apply for each planning step which may not utilize the available feedbacks.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "It would be helpful if the authors can address the following issues.\n- Discuss how to extend the language feedback beyond LLM-based planners.\n- Clarify how to determine which feedback to be active at each time step.\n- Minor:\n    - On line 218, this paper mentioned an open-loop variant, which one is it in table 2?\n    - What is an LLM error in fig 6? Does it mean that the LLM planner generates the wrong steps? If so, I’m surprised that SayCan has such significant increase of LLM failures when there is disturbances.\n    - The line hight in this paper is considerably smaller than other papers."}}, {"review_id": "5JVDv4QMvMb", "reviewer": "Reviewer_Ywcr", "summary": "The paper proposes a system for using natural language feedback generated by large language models(LLMs) to improve task planning in embodied environments. The natural language feedback is generated by using auxiliary models for success detection and scene description and prompting the LLM with information from their outputs. The paper demonstrates the effectiveness of the system on a good set of real-world and simulated environments of a robot performing different tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n**Strengths:**\n- The system does not require training a specialized model for the task and uses existing pre-trained models prompted with information from the environment. \n- The benefit of updating the plans using specialized feedback from the auxiliary models is clear. \n\n**Weakness**\n- The idea of using multiple large pre-trained models (of vision and language) in a zero-shot manner via multimodal-informed prompting for successful task completion has been studied by Zeng et al. 2022. However, Zeng et al. do not demonstrate its use for robotics tasks.  What aspect of the system required a change when applied to a new domain?\n\n- The importance of feedback for long horizon tasks is emphasized throughout the paper. However, no baseline verifies this need for increased feedback for longer tasks. A plot comparing models with and without feedback versus the length of the trajectory is useful. How many times in the course of a trajectory was the model able to benefit from feedback?\n\n- Missing Prior Work:\n   - There is no mention of prior work on self-talk [1] or of papers that follow up on the same idea.\n   - There is no mention of work on the use of natural language feedback in robotic systems. \n\n- No comparison against better decoding methods. An important baseline to look at in the case of fixing robot task plans is to 1) decode multiple possible top-scoring plans and 2) execute the next highest scoring plan in the case of a failure. However, there is no comparison with such a baseline.\n\n- Failure modes of the large language model planner: The limitation section describes how the model ignores feedback from the environment several times. What do some of these cases look like? What are recurrent patterns of failures?\n\n- \"Emergent\" human feedback processing:\nThere is nothing emergent about the model's ability to use human feedback. The feedback is still in the form of natural language, very much in the domain of prompting the LLM. The property of the model to be able to process human feedback to modify its suggested plan is hence not an emergent property but one to expect given the experiments on using specialized cues from the environment.\n\n[1] Unsupervised Commonsense Question Answering with Self-Talk(https://aclanthology.org/2020.emnlp-main.373) (Shwartz et al., EMNLP 2020)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While using natural language feedback generated by an LLM prompted with specialized environment information is interesting. The technical novelty of the paper is limited. \n1. The idea of using multiple large pre-trained models (of vision and language) in a zero-shot manner via multimodal-informed prompting for successful task completion has been studied by Zeng et al. 2022. \n2. The idea of using \"inner-monologue\" is not new. Prior work in NLP has proposed the use of \"self-talk\" as a system for corrections. \n3. The paper is missing important baselines and analysis.\n\n\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Issues are listed in the weaknesses.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)