ThinkBot: Embodied Instruction Following with Thought Chain Reasoning

## Abstract

Abstract Embodied Instruction Following (EIF) requires agents to complete human instruction by interacting objects in complicated surrounding environments. Conventional methods directly consider the sparse human instruction to generate action plans for agents, which usually fail to achieve human goals because of the instruction incoherence in action descriptions. On the contrary, we propose ThinkBot that reasons the thought chain in human instruction to recover the missing action descriptions, so that the agent can successfully complete human goals by following the coherent instruction. Specifically, we first design an instruction completer based on large language models to recover the missing actions with interacted objects between consecutive human instruction, where the perceived surrounding environments and the completed sub-goals are considered for instruction completion. Based on the partially observed scene semantic maps, we present an object localizer to infer the position of interacted objects for agents to achieve complex human goals. Extensive experiments in the simulated environment show that our ThinkBot outperforms the state-of-the-art EIF methods by a sizable margin in both success rate and execution efficiency.

## 1 Introduction

Figure: Figure 1: Comparison between conventional EIF methods (Prompter []) and our ThinkBot. Existing methods directly leverage sparse human instruction to generate action sequence, which usually get stuck due to the incoherence of instruction. Our ThinkBot recovers missing action descriptions by reasoning the thought chain in sparse human instruction, and can successfully complete challenging tasks.
Refer to caption: x1.png

Designing autonomous agents for diverse household tasks has been highly desired in research of artificial intelligence for a long time.
Recent advances in computer vision [, ] and natural language processing [, ] enable autonomous agents to complete complex human requirements, because the appeared large pre-trained models can comprehend human instruction and perceive the world accurately.
Embodied instruction following (EIF) [, , , ] requires the agent to ground human instruction to consecutive task plans with feasible execution, which requires high success rate and completion efficiency.

Figure: Figure 2: The overall pipeline of ThinkBot, which consists of an instruction completer and an object localizer. The instruction completer generates the coherent instruction with interacted objects based on sparse human instruction and the current visual perception results, and the object localizer predicts the position of the interacted object for manipulation and navigation.
Refer to caption: x2.png

To accomplish the challenging EIF task, imitation learning [, , , , , , , , ] is widely adopted to generate low-level actions from historical observation and given instruction, while they fail to adapt to new scenarios due to the insufficient pair-wise data between human instruction and low-level actions.
To this end, modular methods [, , , , , , ] decompose complex tasks to high-level planning conditioned on instruction and low-level execution with predefined controllers.
However, human instruction is usually sparse with incoherence for action plan generation of agents.
For example, in [Fig. 1](#S1.F1), humans may give the instruction *‘Prepare a spoon, Take a mug’* for making cold brew coffee.
In realistic scenarios, the mug may be stored in a fridge, and the agent may need to open the fridge first to get the mug. Therefore, the dense instruction should be *‘Go to the fridge, open the fridge, and take a mug’*.
Lacking coherent instruction usually disables the agent to acquire feasible action sequences in execution, and the success rate across diverse tasks in complicated indoor environments still remains low.

In this paper, we propose a ThinkBot agent to accurately complete diverse EIF tasks in interactive environments.
Unlike existing methods that directly employ the sparse human instruction for agent action sequence generation, we recover the missing action descriptions for agent execution by reasoning the thought chain in sparse human instruction. Therefore, coherent instruction can be leveraged to generate more feasible agent actions in complex EIF tasks especially with long sequences.
More specifically, we first propose an instruction completer based on large language models to predict the missing actions with interacted objects in the sparse human instruction, where we carefully design prompts to consider the perceived objects in the surrounding environments and the completed sub-goals in the task execution sequences.
We then present a multimodal object localizer that predicts the position of the recovered missing objects based on the scene semantic maps, where the mined object correlation is also leveraged to enhance the localization ability.
Extensive experiments on ALFRED [] show that our ThinkBot outperforms the state-of-the-art EIF methods by a sizable margin in both success rate and execution efficiency. Our main contributions can be summarized as follows:

- •
We propose a ThinkBot agent that reasons the thought chain in sparse human instruction for coherence mining to successfully complete complex EIF goals.
- •
We present an instruction completer based on large language models to generate the missing actions with interacted objects, and propose an object localizer to predict the position of objects for interaction.
- •
We conduct extensive experiments of diverse EIF tasks on ALFRED benchmark, and the results demonstrate that our method achieves higher success rate and path-length-weighted success rate than the state-of-the-art methods on unseen environments.

## 2 Related Work

Embodied Instruction Following: Developing generalist agents that can follow human instruction to complete diverse tasks in interactive environments is a long-standing goal.
In the pursuit of this goal, EIF has been widely studied in recent years for high generalizability and practicality.
Prior arts can be divided into two categories: end-to-end methods and modular methods.
End-to-end methods directly generate low-level actions conditioned on the current state of the environment and human instruction [, , , , , , , , ].
For instance, Pashevich *et al*. [] developed an episodic transformer to encode language inputs and the episode history of visual observation and actions, which was decoded for action sequence generation with auto-regression.
However, end-to-end methods often struggle to generalize to unseen scenes due to insufficient pair-wise data between instruction and low-level action sequences.
To address this, modular methods [, , , , , , ] plan high-level action sequences and execute them with pre-defined local policies guided by online semantic maps, which are free of the pair-wise data between instruction and low-level actions.
In modular methods, selecting correct targets and actions for navigation and interaction is very important for searching efficiency and task success. Min *et al*. [] directly employed convolutional networks to predict the target position from current semantic map, and Murray *et al*. [] proposed a landmark classification model based on BERT [] for target selection only based on original human instruction. Inoue *et al*. [] predicted landmark objects according to offline co-occurrence probabilities of objects evaluated by pre-trained language models, and Kim *et al*. [] yield the detailed plan by incorporating the contextual information of natural language instructions.
Nevertheless, directly considering the sparse human instruction for agent action generation usually fails to achieve human goals due to the instruction incoherence with missing action descriptions.

LLMs for Embodied Agents: Large language models (LLMs) [, ] have demonstrated their capability in embodied AI tasks such as visual-language navigation [, , , , , ], object navigation [, ], open-world exploration [, ], and embodied planning [, , , , , , , , ], where the high generalizability across deployment scenes and downstream tasks of LLMs enables embodied agents to achieve diverse human goals in complex environments.
With the rich commonsense embedded in LLMs, fine-grained actions regarding human instruction can be directly generated.
Zhou *et al*. [] directly prompted LLMs to perform zero-shot sequential action prediction by taking the textual descriptions of historical visual observations as inputs.
To facilitate efficient exploration, ESC [] generated frontier candidates on the observed semantic map, and employed LLM to determine the next frontier by considering hand-crafted soft constraints jointly.
While direct generation of fine-grained actions is challenging for LLMs due to the extremely large search space, other methods decompose the overall solution into high-level plan generation and low-level action controlling [, , , , , ].
Wu *et al*. [] crafted a large-scale embodied planning dataset and finetuned different plan generators on the dataset for task plan grounding. Yao *et al*. [] generated consecutive plans by prompting LLMs to synergize reasoning and acting, and achieved impressive performance on text-based benchmarks [].
However, despite the high reasoning ability of LLMs, the spatial localization ability for interacted objects is weak in LLMs. We present a multimodal transformer-based object localizer to provide spatial guidance for agents in object interaction accurately.

## 3 Approach

In this section, we first briefly introduce the problem in EIF and our overall pipeline of the ThinkBot agent ([Sec. 3.1](#S3.SS1)). Then we detail the instruction completer that recovers the missing actions with interacted objects for coherence mining ([Sec. 3.2](#S3.SS2)), and demonstrate the multimodal object localizer which provides spatial guidance for agents in interaction ([Sec. 3.3](#S3.SS3)).

### 3.1 Problem Statement and Overall Pipeline

The embodied instruction following task requires an agent in the interactive environment to finish the human goals physically by generating action sequences, where the human goals and the step-by-step instruction are given to the agent for guidance. Following the modular methods, we first generate high-level subgoal sequences and then execute them with a pre-defined controller guided by online semantic maps. In the $t_{th}$ step, the agent needs to generate a high-level subgoal $A_{t}$ based on the current instruction $I_{t}$ and the object state $S_{t-1}$ after implementing the last subgoal by the controller. A high-level plan $A_{t}$ is a tuple $(a_{t},o_{t},p_{t})$, where $a_{t}$ is the primitive action and $o_{t}$ means the interacted object with the position $p_{t}$. In EIF tasks, human instruction is usually sparse with significant incoherence between consecutive steps. Therefore, it is very challenging to generate feasible action $A_{t}$ based on the current step-wise instruction $I_{t}$ and the object state $S_{t-1}$ after implementing the last action.
For example, humans may provide the instruction *‘Prepare a spoon, take a mug’* for coffee making.
However, the mug may be stored in the fridge in realistic scenes. Consequently, it is difficult for the agent to take the mug with spoon without other instruction after preparing the bread. The coherent instruction should be *‘Prepare a spoon, go to the fridge, open the fridge, take a mug’*.
Therefore, our goal is to recover the missing action descriptions in the sparse human instruction.

Since the thought chain reveals a series of intermediate steps from the initial problem to the final solution [], it can be leveraged to decompose the original complex problems and enhance the feasibility of the solution. In EIF tasks, reasoning the thought chain can predict the missing action descriptions in the sparse human instruction to successfully achieve the goal. The overall pipeline of our ThinkBot agent is shown in [Figure 2](#S1.F2), which consists of an instruction completer recovering the missing actions with interacted objects and an object localizer predicting the object location for agent interaction. For the instruction completer, we design prompts for the pre-trained large language model including the descriptions of scene information and the task completion process, which is expected to reason the thought chain in sparse human instruction to provide coherent instruction. For the object localizer, the generated missing actions with interacted objects and the perceived semantic map of the scene are utilized to predict the object location for the agent to interact with, where multimodal transformers are employed for the alignment between language instruction and visual clues in the environment. Finally, the agent can easily complete the human goals to achieve significantly higher success rate with the coherent instruction and the explicit interaction location.

### 3.2 Instruction Completer

Figure: Figure 3: Input and output of the instruction completer based on LLMs. The input contains system message describing the world properties and agent message demonstrating perceived environment information. The output includes the thought chain in sparse human instruction and missing subgoals with interacted objects.
Refer to caption: x3.png

To recover the missing action descriptions in the sparse human instruction, we employ LLMs with rich commonsense to reason the thought chain in the instruction.
While LLMs have demonstrated remarkable abilities in various tasks, unlocking their complete potential requires prompt engineering. To enable the pre-trained LLMs to predict the missing actions with interacted objects accurately, we carefully design the prompt by organizing the system message and agent message that respectively describe the innate world properties and the perceived information. The input and output of the instruction completer are shown in [Figure 3](#S3.F3) with an example.

System Message: The system message describes the innate world properties in the simulator including the role explanation, message definition, primitive actions and response format, which remains unchanged during the whole EIF process for the given task and environment. The role explanation specifies the simulation scenario of household AI assistant for LLM response, and the message definition explains the meaning of each input message. Primitive actions constrain the available selection for agent in manipulation and navigation, while response format limits the response structure for LLMs to generate the missing actions with interacted objects.

Agent Message: The agent message demonstrates the perceived information including the human instruction, task completion process, and observed objects in the scene, which is updated along with the EIF process. The human instruction means the given sparse step-by-step instruction sentences and the final goal. The task completion depicts completed subgoals, the current subgoal and all subgoals, which reflects the execution process of the final goal. The information on observed objects in the scene is represented by the object category and the coordination of the instances that have been seen by the agent, which demonstrates the affordable candidates for interaction during the EIF process. Since providing a large number of candidate objects causes severe hallucination, we only leverage objects that are possible to appear in the current room type as candidate objects. Finally, the system message and the agent message are concatenated to form the prompt for recovering missing actions with interacted objects in the sparse human instruction.

Thought Chain Reasoning: To accurately recover the missing actions in sparse human instruction, we enforce LLMs to reason the thought chain in human instruction that indicates the detailed process from the initial state to the final goal. However, the recovered actions can only be executed if the agent knows the exact position of the interacted objects. Therefore, we also require LLMs to generate the missing subgoals in a structured format for subsequent location prediction. The recovered subgoals are leveraged in subsequent object localizer for position prediction, so that the agent can manipulate the object to successfully achieve the subgoal for task completion.

### 3.3 Multimodal Object Localizer

Although LLMs can accurately reason the thought chain in sparse human instruction to recover the missing actions with interacted objects, the inferring of object position for agent interaction remains challenging because of the weak spatial localization ability of the language description. To deal with the challenge, we present an object localizer based on multimodal transformers to predict the position of interacted objects based on the recovered instruction and the partially observed semantic map. [Figure 4](#S3.F4) depicts the framework of the object localizer. The recovered instruction acquired from the instruction completer provides the goal for the agent to interact with, and the partially observed semantic map demonstrates the positions of seen objects. The representation of semantic maps and the instruction are leveraged by the decoder to generate the probability heatmap for location prediction of the interacted object. Since the semantic correlation among objects can provide beneficial priors to precisely locate the interacted objects based on the visual perception results, we mine the object correlation graph to strengthen the semantic map representation. In this section, we first detail the encoder design for all inputs in representation extraction, and then illustrate the alignment strategy among different inputs with the multimodal transformer. Finally, we introduce the learning objective and the training pipelines of the object localizer.

For the instruction encoder, we leverage a pre-trained BERT [] to extract the instruction features $\mathbf{X}_{s}$ for the human instruction and the goal object, which are acquired from the prediction results of instruction completer. All layers except the last one are frozen during the object localizer finetuning for high generalizability in understanding instruction for complex tasks. For the map encoder, we employ convolutional neural networks to extract the initial map features $\mathbf{X}^{\prime}_{t}$ that take the partially observed semantic maps as the input.
Since semantic correlation among objects provides beneficial priors for accurate localization, we mine the object correlation graph to embed the priors into semantic map features for further enhancement of object position prediction. The graph is defined as $\mathcal{G}=(\mathbf{V},\mathbf{E})$, where $\mathbf{V}$ is the set of nodes and $\mathbf{E}$ are the edges between different nodes. Each node represents a possible object category in the simulator, and the element in the $i_{th}$ row and $j_{th}$ column of $\mathbf{E}$ indicates the correlation of objects between the $i_{th}$ and the $j_{th}$ classes.
While conventional graph convolutional networks use external priors to handcraft a predefined graph structure, we construct the object correlation graph by learning from the semantic map features $\mathbf{X}^{\prime}_{t}$ to adapt to different scenes.
The object correlation graph is generated from the semantic map features $\mathbf{X}^{\prime}_{t}$ by $\mathbf{E}_{t}=f(\mathbf{X}^{\prime}_{t}\mathbf{W}_{e})$, where $f(\cdot)$ denotes the activation function and $\mathbf{W}_{e}$ is a learnable matrix for graph generation.
The process of learning the object correlation graph can be regarded as encoding the object correlation priors among the categories.
We then employ graph convolutional layers to enhance the semantic map representation with the object correlation encoding, and acquire the map features for multimodal alignment according to the following:

$$ $\mathbf{X}_{t}=\mathbf{X}^{\prime}_{t}+\mathbf{E}\mathbf{X}^{\prime}_{t} \mathbf{W}_{a},$ (1) $$

where $\mathbf{W}_{a}$ is a learnable weight matrix for message passing to embed the object correlation priors into the map features.

Figure: Figure 4: The overall pipeline of the multimodal object localizer, which uses recovered instruction and observed semantic map to predict object positions for interaction. The object correlation graph is also learned to strengthen the map features.
Refer to caption: x4.png

After the representation extraction for the instruction and the semantic map, we leverage a map-instruction aligner to align the instruction features and the map features for object position prediction.
Since the semantic map is updated in an online manner with high frequency during agent navigation, we take the features of the semantic maps as the query. The instruction remains unchanged in a subtask, which is utilized as the key and the value features. Specifically, they can be acquired via the following equations:

$$ $\mathbf{Q}_{s}=\mathbf{X}_{s}\mathbf{W}_{q},\quad\mathbf{K}_{t}=\mathbf{X}_{t} \mathbf{W}_{k},\quad\mathbf{V}_{t}=\mathbf{X}_{t}\mathbf{W}_{v},$ (2) $$

where $\mathbf{Q}_{s}$, $\mathbf{K}_{t}$ and $\mathbf{V}_{t}$ respectively mean the query features from the semantic maps, the key and value features from the instruction. The matrices $\mathbf{W}_{q}$, $\mathbf{W}_{k}$, $\mathbf{W}_{v}$ are leveraged to obtain the query, key and value features respectively. The representation $\mathbf{H}_{t}^{s}$ that aligns the visual information provided by semantic maps and the language information in the instruction can be acquired as follows:

$$ $\mathbf{H}_{t}^{s}={\rm Softmax}\left(\frac{\mathbf{Q}_{s}\mathbf{K}_{t}^{T}}{ \sqrt{d}}\right)\mathbf{V}_{t},$ (3) $$

where $d$ means the dimension of the representation $\mathbf{H}_{t}^{s}$.
The representation is then leveraged to decode the predicted location of the interacted objects.

In order to train the encoder, the map-instruction aligner, and the decoder in the object localizer, we need to acquire the groundtruth instance position for the interacted objects that are predicted by the instruction completer. We replay expert demonstrations in the training data, and record the masks of interacted objects when a subtask is successfully completed. With the projection from egocentric mask to the top-down view by depth estimation, the groundtruth positions are used to supervise the object localizer to predict the position of interacted objects. The model is trained with a pixel-wise binary cross-entropy loss, where the labels of pixels covered by the interacted objects are set to one and otherwise to zero.

**Table 1: Comparison with the state-of-the-art methods in SR, GC, PLWSR, PLWGC on the test seen and test unseen splits.**
| Method | Test Seen | Test Unseen |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PLWGC | GC | PLWSR | SR | PLWGC | GC | PLWSR | SR |  |
| Seq2seq | 6.27 | 9.42 | 2.02 | 3.98 | 4.26 | 7.03 | 0.08 | 3.9 |
| MOCA | 22.05 | 28.29 | 15.10 | 22.05 | 9.99 | 14.28 | 2.72 | 5.30 |
| E.T. | 34.93 | 45.44 | 27.78 | 38.42 | 11.46 | 18.56 | 4.10 | 8.57 |
| LWIT | 23.10 | 40.53 | 43.10 | 30.92 | 16.34 | 20.91 | 5.60 | 9.42 |
| HITUT | 17.41 | 29.97 | 11.10 | 21.27 | 11.51 | 20.31 | 5.86 | 13.87 |
| ABP | 4.92 | 51.13 | 3.88 | 44.55 | 2.22 | 24.76 | 1.08 | 15.43 |
| LLM-Planner | - | 26.77 | - | 18.20 | - | 23.37 | - | 16.42 |
| FILM | 15.59 | 39.55 | 11.27 | 28.83 | 15.13 | 38.52 | 11.32 | 27.80 |
| LGS-RPA | 28.97 | 48.66 | 21.28 | 40.05 | 22.76 | 45.24 | 22.76 | 35.41 |
| Prompter | 30.72 | 63.43 | 25.81 | 53.23 | 26.22 | 58.76 | 20.76 | 45.72 |
| CPEM | 27.49 | 59.40 | 22.61 | 50.62 | 27.00 | 61.10 | 22.61 | 49.84 |
| Prompter+ | 36.35 | 70.20 | 31.12 | 60.86 | 30.09 | 65.71 | 26.22 | 55.46 |
| ThinkBot (Ours) | 37.01 | 71.64 | 32.02 | 62.69 | 30.73 | 67.75 | 26.93 | 57.82 |

## 4 Experiments

In this section, we first introduce the experiment setup including datasets, baseline methods, evaluation metrics and implementation details. Then we compare our method with the state-of-the-art EIF approaches to show the superiority in success rate and efficiency, and conduct an ablation study to verify the effectiveness of the instruction completer and the object localizer. Finally, we also demonstrate the visualization results of our method to depict our intuition.
Additional results and case studies are provided in the supplementary material.

### 4.1 Experimental Setup

Dataset: For the simulation of EIF tasks, we utilize the well-recognized ALFRED benchmark [] within the AI2-THOR [] virtual environment. The ALFRED benchmark includes $25$,743 trajectory-instruction pairs, covering $7$ different task types with varying levels of complexity. The benchmark is divided into five splits including *train*, *test seen*, *test unseen*, *valid seen* and *valid unseen*. The ALFRED benchmark poses significant challenges for EIF agents, as it requires them to ground incoherent natural instruction of different granularity into various household tasks that involve long-horizon reasoning plans.

Baselines: We compare our agent, ThinkBot, with previously published state-of-the-art EIF models. The counterparts include end-to-end methods Seq2seq [], MOCA [], E.T. [], LWIT [], HITUT [], ABP [], and modular methods LLM-Planner [], FILM [], LGS-RPA [], Prompter [], CPEM []. We also construct a strong baseline in our experiments, which is a modified version of Prompter [] denoted as Prompter+. The strong baseline combines environment-aware memory [] and a re-trained object detector [] with the vanilla Prompter.

Evaluation Metrics: We follow the evaluation protocol outlined in the ALFRED benchmark.
The primary metric is the success rate (SR) that measures the percentage of tasks completed, and we also report the goal-condition success rate (GC), which evaluates the percentage of satisfied goal conditions for all subgoals in step-by-step instruction.
To account for efficiency in task completion, both SR and GC are penalized by the length of the execution sequence to compute a path-length-weighted (PLW) score for each metric, which are termed PLWSR and PLWGC respectively.

Implementation Details: The instruction completer adopts the publicly released GPT-3.5 API GPT-3.5-turbo as the backbone architecture, where we set the generation temperature to 0 0 for stability enhancement. For prompt design, we leverage emotion prompt [] and prompt optimization [] in the system message template to further boost the performance of LLMs. For the multimodal object localizer, we employ a truncated ResNet18 [] for the map encoder and follow [] to mine the inter-modal representation interaction with the multimodal transformer. The object correlation graph is learned via graph convolutional layers []. Moreover, we collect a dataset of partially-observed semantic maps and target location map pairs by replaying expert trajectories in the simulator for object localizer training, and AdamW optimizer [] with the initial learning rate $5\text{\times}{10}^{-4}$ and step decay is employed for parameter update.

**Table 2: Comparison of methods combining different proposed techniques, where valid unseen and the selected hard valid unseen splits are used for evaluation.**
| Method | Valid Unseen | Hard Valid Unseen |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PLWGC | GC | PLWSR | SR | PLWGC | GC | PLWSR | SR |  |
| Random | 26.18 | 67.64 | 23.80 | 59.68 | 0.32 | 5.41 | 0 | 0 |
| Prompter+ | 29.36 | 72.00 | 26.82 | 64.43 | 0.48 | 5.41 | 0 | 0 |
| Groundtruth Location | 39.71 | 72.75 | 37.01 | 67.97 | 0.79 | 5.41 | 0 | 0 |
| w/o Instruction Completer | 29.09 | 72.38 | 26.43 | 64.92 | 0.48 | 5.41 | 0 | 0 |
| w/o Object Localizer | 30.24 | 74.37 | 27.87 | 66.99 | 9.29 | 22.41 | 8.11 | 16.22 |
| w/o Object Correlation Graph | 30.41 | 73.89 | 28.14 | 67.36 | 11.31 | 29.46 | 9.74 | 21.62 |
| ThinkBot (Ours) | 31.11 | 75.30 | 28.73 | 67.72 | 11.95 | 30.86 | 10.26 | 22.97 |

Figure: Figure 5: Visualization of the agent action sequence acquired by Prompter+ (top) and our ThinkBot (bottom), where our method can recover the missing actions with interacted instances ‘Open Fridge’ and ‘Open Cabinet’ to successfully achieve the human goal.
Refer to caption: x5.png

### 4.2 Comparison with the State-of-the-Arts

In this section, we compare the proposed ThinkBot with the state-of-the-art methods on the ALFRED benchmark. [Table 1](#S3.T1) illustrates the comparison of SR, GC, PLWGC, and PLWSR on the test splits for both seen and unseen scenarios. Our ThinkBot achieves the best performance on all four metrics in both test seen and test unseen split, outperforming the previous arts including both end-to-end and modular methods by a sizable margin.
Compared with CPEM [], ThinkBot surpasses by 7.98% and 12.07% SR on the test unseen and test seen split, respectively.
Besides, Prompter leverages the large language models to infer object co-occurrence probability for semantic search in EIF, and is further enhanced with better visual perception modules to acquire our Prompter+. However, they still suffer from the sparse human instruction with incoherence that usually causes execution failure.
On the contrary, our ThinkBot reasons the thought chain in the sparse human instruction to recover the missing action descriptions, which can provide coherent instruction for the agent to successfully complete the human goal with high efficiency.
As a result, our method outperforms the second-best Prompter+ method by 1.83% (62.69% vs. 60.86%) and 1.44% (71.64% vs. 70.20%) in SR and GC respectively in the test seen split.
Moreover, the advantages of our method are more obvious in the test unseen split, which are 2.36% (57.82% vs. 55.46%) in SR and 2.04% (67.75% vs. 65.71%) in GC.
The results indicate the high generalization ability of our ThinkBot even in novel scenarios that are never seen in training data.
In terms of the efficiency in task completion, our method achieves $32.02$% PLWSR and $37.01$% PLWGC in the test seen split, and $26.93$% PLWSR and $30.73$% PLWGC in the test unseen split, which demonstrates the efficiency of our method. In conclusion, our ThinkBot agent is more practical than stat-of-the-art methods for EIF tasks in scenarios with complex environments and long sequences.

Figure: Figure 6: The visualization of the predicted and groundtruth positions of interacted objects, where the partially observed semantic maps are also depicted.
Refer to caption: x6.png

### 4.3 Ablation Study

Our instruction completer recovers the missing actions with interacted objects by reasoning thought chain in sparse human instruction, and the object localizer predicts the position of the interacted objects.
In [Table 2](#S4.T2), we conduct an ablation study to verify the effectiveness of each presented technique. Since recovering the incoherent instruction is especially beneficial in the hard cases where target objects are located inside closed containers (e.g. The task *‘to get a chilled mug for coffee’* where the mug is in a closed fridge), we also evaluate different methods in the hard cases extracted from valid unseen split, referred to as *hard valid unseen*. In this subset, the agent must open the receptacles to locate the target objects by recovering the missing interaction with the receptacles from sparse instruction, and then the challenging tasks in the hard cases can be successfully completed.

Instruction Completer: We first implement our ThinkBot without the instruction completer, where we directly predict the location of the target in the original sparse human instruction with the object localizer for interaction.
The results in success rate drops $2.24$% compared with our vanilla ThinkBot in the valid unseen split.
Directly predicting the location of interacted objects in incoherent instruction usually causes large deviation, because the semantic correlation is weak between the target object and the observed semantic map.
In hard cases, the agent is more likely to fail to complete the task due to the missing interaction with the containers, and the invisible target object is hard to discover by the agent.
As a result, the performance in SR without instruction completer drops to zero in hard unseen split.
On the contrary, our instruction completer can reason intermediate action descriptions to mine the semantic correlation, which provides fine-grained instruction for the agent to achieve human goals.

Object Localizer: We also evaluate the performance of our ThinkBot without the object localizer, where the agent leverages the baseline Prompter method to predict interaction location with the coherent human instruction recovered by our instruction completer. We observe notable performance drops in SR and PLWSR.
In the valid unseen split, the performance drops $0.75$% in SR and $0.86$% in PLWSR compared with our vanilla ThinkBot.
Meanwhile, we also implement the object localizer without the object correlation graph, the performance is also degraded in both valid unseen and hard valid unseen split, because the object localizer is required to predict the position of both the target objects and the corresponding containers. The correlation mining between target objects and the containers can significantly enhance localization accuracy.
Besides, we present the results of Prompter+ with random and Prompter’s search policy, which drop significantly in SR and PLWSR.
Hence, the agent without the object localizer fails to locate those target objects to achieve the generated intermediate subgoals, which leads to the failure of task completion.
We also provide the results for the settings where the groundtruth location of the interacted objects is given. Our ThinkBot achieves a similar success rate, which indicates the precise prediction of the object location.
In conclusion, the results verify the necessity of the object localizer for agent interaction in order to achieve human goals.

### 4.4 Qualitative Analysis

Action Sequence Visualization:
We present two qualitative examples of the generated action sequence in [Figure 5](#S4.F5) from Prompter+ and our ThinkBot.
In the left case, the agent is instructed to *‘Take a knife. Cut the lettuce in the fridge’*. The results show that the previous agent struggles to complete the task due to the missing ‘open’ action and interacted object ‘fridge’.
In contrast, our ThinkBot first reasons the thought chain of human instruction, and then recovers the missing ‘open’ action and interacted object ‘fridge’ from the instruction, thus successfully completes the task.
In the right case, ThinkBot not only recovers the missing ‘open’ action and interacted object ‘cabinet’, but also interacts with the right cabinet that contains the tissue box.
The case study demonstrates the effectiveness of ThinkBot in recovering the missing actions and interacted objects from sparse human instruction.

Visualization of the Object Localizer:
[Figure 6](#S4.F6) shows the groundtruth and the predicted location of the interacted objects from the object localizer, where the object category is generated from the upstream instruction completer.
In the top case, the interacted object ‘pencil’ is located on the table, and the object localizer predicts the position of the interacted object with negligible deviation. In the bottom case where multiple instances exist for the recovered object, the object localizer is able to locate all objects in the same category. Meanwhile, our ThinkBot can correctly assign the largest probability to the instance that is described by the instruction, and the human goals can be successfully achieved following the step-by-step human instruction.

## 5 Conclusion

In this paper, we have presented a ThinkBot agent that reasons the thought chain for missing instruction recovery in EIF tasks. We design an instruction completer to predict the intermediate actions with interacted objects between incoherent human instruction, and then leverage a multimodal transformer to infer the interaction location for the agent. Extensive experiments in a wide variety of EIF tasks demonstrate the superiority of our method regarding the success rate and execution efficiency.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - How does the performance of ThinkBot vary when using different LLMs with potentially different training datasets? Have you tested the robustness of the prompt with LLMs such as Llama 3 or other open-sourced LLMs?
- How does the ThinkBot approach handle the increased complexity and variability of real-world environments compared to simulated ones?
- What is the inference time for the LLM-based reasoning process, and how does this delay impact the feasibility of ThinkBot for real-world applications? How does the computational resource requirement of the ThinkBot system impact its scalability, especially when considering the constraints of embodied systems in real-world scenarios?
- Can the agent learn from its mistakes to improve its performance over time, and is there a mechanism for self-improvement during interactions?
- How to address the generalization of the reasoning process to new scenarios or tasks, ensuring that the agent can adapt and learn from a wide range of situations?
soundness: 3
strengths: - The paper presents a novel approach to Embodied Instruction Following (EIF) by introducing the concept of a "thought chain" to recover missing action descriptions from sparse human instructions. Integrating an instruction completer and an object localizer enhances the agent's ability to follow instructions in complex environments. This is an effective way to address the coherence issue in action plans.
- The experiments in the paper demonstrate the effectiveness of ThinkBot through extensive testing on the ALFRED benchmark, showing significant improvements in success rates and execution efficiency over state-of-the-art methods.
- The paper is well-structured. The figures and diagrams, such as the pipeline of ThinkBot and the comparison with conventional methods, are clear and effectively illustrate the proposed methods and results. The writing is coherent, and the instructions for reproducibility, including the full system prompt for the instruction completer, contribute to the transparency and clarity of the research.
summary: The paper introduces ThinkBot, an agent designed for Embodied Instruction Following (EIF) that effectively executes human instructions in complex environments by reconstructing missing action descriptions through the chain of thought. The authors introduce an instruction completer using large language models to predict missing actions and an object localizer to determine object positions for agent interaction. Extensive experiments show that ThinkBot surpasses existing state-of-the-art EIF methods in both success rate and execution efficiency. Key contributions include the development of the ThinkBot agent, the instruction completer for coherent instruction generation, and the object localizer for precise spatial guidance, all enhancing success rates in EIF tasks.
weaknesses: - The introduced approach heavily relies on the inherent reasoning capabilities of LLMs. If the LLMs are not trained on diverse or comprehensive datasets, it could lead to biases or limitations in the agent's performance. The authors need to demonstrate the robustness of the prompt to different LLMs, e.g., Llama 3, and other open-sourced LLMs (gemma2, qwen2.5). 
- While the paper demonstrates strong performance in simulated environments, there might be concerns about how well ThinkBot's approach translates to real-world applications, where the complexity and variability can be significantly higher. Besides, the time cost of the reasoning process should be considered for real-world applications. However, LLM-based reasoning will lead to a high delay in the inference time. The use of LLMs and complex reasoning processes may require significant computational resources, which could limit the scalability of the ThinkBot system, especially in resource-constrained embodied systems.
- The paper does not extensively discuss how ThinkBot handles errors or recovers from incorrect actions once taken. This is an important aspect of robust autonomous agent design. It is important to make the agent self-improve during the interactions, instead of manually adjusting the prompt. This is also important to generalize the reasoning process to other scenarios or tasks. It is necessary to conduct experiments demonstrating the agent's ability to learn from mistakes (adding noise to actions) and improve performance over time without manual prompt adjustments or discuss how the reasoning process could be made more generalizable.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: What if you take input the instruction with current egocentric image to gpt-4o? I believe it might improve the overall reasoning performance.
soundness: 3
strengths: The proposed pipeline is technically sound. Using LLM to understand free-form instructions is an effective way to reason about actions in unstructured environments. And the proposed object localizer complements the LLM by addressing its spatial understanding limitations.
summary: This paper proposes a novel agent for Embodied Instruction Following (EIF) tasks. The method uses thought chain reasoning to complete missing action descriptions and employs a multimodal object localizer for enhanced object interaction. Extensive experiments demonstrate better performance in both success rate and execution efficiency compared to previous methods.
weaknesses: 1. While effective, this work appears to be a combination of LLM with a localizer, which might lack of novelty.

Existing multimodal LLMs (MLLMs) can achieve better reasoning performance than the LLM used here. The authors employed GPT-3.5, but using a state-of-the-art MLLM like GPT-4o could improve performance.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: See weaknesses above.
soundness: 3
strengths: - ThinkBot tries to figure out missing actions from instructions, which is an important and challenging problem of embodied instruction following.
- ThinkBot shows a strong improvement over the previous methods (yet the baseline, Prompter+, on which ThinkBot is built already beats SoTA, see weaknesses).
- The paper is well structured and shows clear presentation.
summary: The paper presents ThinkBot, a system designed to infer missing actions from incomplete human instructions using LLMs and to predict target object locations for these inferred actions. ThinkBot employs a structured CoT prompt to obtain the missing actions using LLMs. With the actions and target objects identified, the multimodal object localizer creates a positional heatmap the given instruction, and the inferred action. ThinkBot achieves improvements over previous methods in the ALFRED benchmark.
weaknesses: - It looks like the instruction completer finds where is a target object by addressing missing information in the goal statement, but the step-by-step instructions usually contain the missing information. Why not just directly parse the missing information in the step-by-step instructions? Why do we still need the proposed instruction completer?

- Previous work (Prompter) also similarly finds the next receptacle by using language models (BERT) by guessing where a target object is likely to be. Instead, the instruction completer uses LLM with CoT. We may use better language models with prompting techniques for better results, but this is expected.

- Why not just using semantic information in the spatial semantic map, not predicting where is a target by the multimodal object localizer?

- The ablated performance of the multimodal object localizer and the correlation graph seems marginal in valid unseen but not in Hard Valid Unseen. It seems like the model is designed for closed-receptacle cases. And why are they marginal?

- I think that the authors can plug the proposed module in the other baselines like FILM and CPEM. Improvements over these models may indicate good generalizability of the proposed approach.

- The proposed model is evaluated in a single benchmark. I'm not sure if this approach works well in other tasks.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - What makes the difference between ThinkBot and LLM-Planner in high-level planning (Table 3)?
soundness: 3
strengths: - The paper introduces a new modular approach for EIF tasks, a ThinkBot agent. The idea of recovering sparse human instruction is a novel direction. 
- The paper conducts extensive analysis, including ablation studies and qualitative analysis, which helps to understand the proposed method.
- The paper is well-written and easy to understand.
summary: This paper presents a ThinkBot agent that recovers underspecified instructions through thought chain reasoning and performs embodied instruction following tasks. The proposed method consists of three components: the instruction completer, the object localizer, and the object correlation graph. The authors employ large language models for the instruction completer and a multimodal transformer for the object localizer. Experimental results show that a ThinkBot agent outperforms prior arts on the ALFRED benchmark, and ablation studies demonstrate the effectiveness of each component.
weaknesses: - While the authors demonstrated the effectiveness of the instruction completer in ablation studies (Table 2), the improvements are quite marginal in the valid unseen split. It indicates that recovering sparse human instruction is sometimes helpful, but at the same time, this tells us that it is not a fundamental problem in EIF tasks.

Meta-review: The paper proposes ThinkBot, a framework for embodied instruction following (EIF) that uses chain-of-thought-like reasoning to follow instructions in which some of the actions may not be explicitly conveyed. The "instruction completer" uses a large language model (LLM) to infer missing actions and object localizer helps to identify the location of the target objects. In experiments using the Alfred simulator, ThinkBot outperforms EIF baselines in terms of success rate and task efficiency.

The paper is highly topical---the use of large language models to facilitate instruction-following in embodied settings continues to attract significant attention. ThinkBot differs from some of this work in its focus on explicitly inferring actions that are important to the success of a task, but may not be mentioned explicitly, something that regularly happens in practice. The reviewers acknowledged the importance of this problem and at least two reviewers appreciated the contributions of the thought chain approach to recovering latent action descriptions. The reviewers noted the paper's extensive experimental evaluation, which demonstrates the effectiveness of ThinkBot relative to contemporary baselines. Meanwhile, at least two reviewers commented that the paper is well structured and easy to read. However, the reviewers raised some concerns about the paper as originally submitted. Among them is the method's sensitivity to the choice of LLMs and whether using an improved LLM would match the gains provided by ThinkBot. During the rebuttal, the authors provided experimental results with different language models, including some with GPT-4o, which suggest that ThinkBot's performance gains are robust, at least for now. Another concern was whether the method could generalize to real-world settings, due to their higher variability and complexity, as well as be capable of inference rates that are sufficiently fast for real-time use. The authors made an effort to address these concerns, included by reporting new experimental results, which the reviewer found satisfying. However, the AC disagrees with the claim that an operating frequency of 1.5 Hz is sufficient for practical use. Additionally, there were questions about how ThinkBot recovers after executing incorrect actions, which were addressed with results that involve deploying ThinkBot with an agent whose actions exhibit some randomness and, in turn, are more prone to error.

Overall, the paper addresses an important problem in embodied instruction following. The method itself is interesting, though the extent to which it is novel is not entirely clear. That said, the paper reveals respectable performance gains over existing methods. Most of the reviewers primary concerns were addressed during the rebuttal period.

The AC acknowledges that Reviewer ZBDZ's criticisms were not fully substantiated and that they did not respond to the authors' rebuttal or their attempts to engage the reviewer in discussion. As a result, the AC placed less weight on their review when making the recommendation.

Decision: Accept (Poster)