ThinkBot: Embodied Instruction Following with Thought Chain Reasoning

Authors: Guanxing Lu, Ziwei Wang, Changliu Liu, Jiwen Lu, Yansong Tang
Primary area: applications to robotics, autonomy, planning
Keywords: Embodied Instruction Following (EIF), Large Language Model, Chain-of-thought Reasoning

Abstract:
Embodied Instruction Following (EIF) requires agents to complete human instruction by interacting objects in complicated surrounding environments. Conventional methods directly consider the sparse human instruction to generate action plans for agents, which usually fail to achieve human goals because of the instruction incoherence in action descriptions. On the contrary, we propose ThinkBot that reasons the thought chain in human instruction to recover the missing action descriptions, so that the agent can successfully complete human goals by following the coherent instruction. Specifically, we first design an instruction completer based on large language models to recover the missing actions with interacted objects between consecutive human instruction, where the perceived surrounding environments and the completed sub-goals are considered for instruction completion. Based on the partially observed scene semantic maps, we present an object localizer to infer the position of interacted objects and the related Bayesian uncertainty for close-loop planning. Extensive experiments in the simulated environment show that our ThinkBot outperforms the state-of-the-art EIF methods by a sizable margin in both success rate and execution efficiency. Project page: https://guanxinglu.github.io/thinkbot/.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730099254559

thread_type:
official_review

<thread_object>
creator: reviewer_thpp

<content>
summary:
This paper presents a ThinkBot agent that recovers underspecified instructions through thought chain reasoning and performs embodied instruction following tasks. The proposed method consists of three components: the instruction completer, the object localizer, and the object correlation graph. The authors employ large language models for the instruction completer and a multimodal transformer for the object localizer. Experimental results show that a ThinkBot agent outperforms prior arts on the ALFRED benchmark, and ablation studies demonstrate the effectiveness of each component.

strengths:
- The paper introduces a new modular approach for EIF tasks, a ThinkBot agent. The idea of recovering sparse human instruction is a novel direction. 
- The paper conducts extensive analysis, including ablation studies and qualitative analysis, which helps to understand the proposed method.
- The paper is well-written and easy to understand.

weaknesses:
- While the authors demonstrated the effectiveness of the instruction completer in ablation studies (Table 2), the improvements are quite marginal in the valid unseen split. It indicates that recovering sparse human instruction is sometimes helpful, but at the same time, this tells us that it is not a fundamental problem in EIF tasks.

questions:
- What makes the difference between ThinkBot and LLM-Planner in high-level planning (Table 3)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730454589914

thread_type:
official_review

<thread_object>
creator: reviewer_owwc

<content>
summary:
The paper presents ThinkBot, a system designed to infer missing actions from incomplete human instructions using LLMs and to predict target object locations for these inferred actions. ThinkBot employs a structured CoT prompt to obtain the missing actions using LLMs. With the actions and target objects identified, the multimodal object localizer creates a positional heatmap the given instruction, and the inferred action. ThinkBot achieves improvements over previous methods in the ALFRED benchmark.

strengths:
- ThinkBot tries to figure out missing actions from instructions, which is an important and challenging problem of embodied instruction following.
- ThinkBot shows a strong improvement over the previous methods (yet the baseline, Prompter+, on which ThinkBot is built already beats SoTA, see weaknesses).
- The paper is well structured and shows clear presentation.

weaknesses:
- It looks like the instruction completer finds where is a target object by addressing missing information in the goal statement, but the step-by-step instructions usually contain the missing information. Why not just directly parse the missing information in the step-by-step instructions? Why do we still need the proposed instruction completer?

- Previous work (Prompter) also similarly finds the next receptacle by using language models (BERT) by guessing where a target object is likely to be. Instead, the instruction completer uses LLM with CoT. We may use better language models with prompting techniques for better results, but this is expected.

- Why not just using semantic information in the spatial semantic map, not predicting where is a target by the multimodal object localizer?

- The ablated performance of the multimodal object localizer and the correlation graph seems marginal in valid unseen but not in Hard Valid Unseen. It seems like the model is designed for closed-receptacle cases. And why are they marginal?

- I think that the authors can plug the proposed module in the other baselines like FILM and CPEM. Improvements over these models may indicate good generalizability of the proposed approach.

- The proposed model is evaluated in a single benchmark. I'm not sure if this approach works well in other tasks.

questions:
See weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624175806

thread_type:
official_review

<thread_object>
creator: reviewer_zbdz

<content>
summary:
This paper proposes a novel agent for Embodied Instruction Following (EIF) tasks. The method uses thought chain reasoning to complete missing action descriptions and employs a multimodal object localizer for enhanced object interaction. Extensive experiments demonstrate better performance in both success rate and execution efficiency compared to previous methods.

strengths:
The proposed pipeline is technically sound. Using LLM to understand free-form instructions is an effective way to reason about actions in unstructured environments. And the proposed object localizer complements the LLM by addressing its spatial understanding limitations.

weaknesses:
1. While effective, this work appears to be a combination of LLM with a localizer, which might lack of novelty.

Existing multimodal LLMs (MLLMs) can achieve better reasoning performance than the LLM used here. The authors employed GPT-3.5, but using a state-of-the-art MLLM like GPT-4o could improve performance.

questions:
What if you take input the instruction with current egocentric image to gpt-4o? I believe it might improve the overall reasoning performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646233400

thread_type:
official_review

<thread_object>
creator: reviewer_pz3k

<content>
summary:
The paper introduces ThinkBot, an agent designed for Embodied Instruction Following (EIF) that effectively executes human instructions in complex environments by reconstructing missing action descriptions through the chain of thought. The authors introduce an instruction completer using large language models to predict missing actions and an object localizer to determine object positions for agent interaction. Extensive experiments show that ThinkBot surpasses existing state-of-the-art EIF methods in both success rate and execution efficiency. Key contributions include the development of the ThinkBot agent, the instruction completer for coherent instruction generation, and the object localizer for precise spatial guidance, all enhancing success rates in EIF tasks.

strengths:
- The paper presents a novel approach to Embodied Instruction Following (EIF) by introducing the concept of a "thought chain" to recover missing action descriptions from sparse human instructions. Integrating an instruction completer and an object localizer enhances the agent's ability to follow instructions in complex environments. This is an effective way to address the coherence issue in action plans.
- The experiments in the paper demonstrate the effectiveness of ThinkBot through extensive testing on the ALFRED benchmark, showing significant improvements in success rates and execution efficiency over state-of-the-art methods.
- The paper is well-structured. The figures and diagrams, such as the pipeline of ThinkBot and the comparison with conventional methods, are clear and effectively illustrate the proposed methods and results. The writing is coherent, and the instructions for reproducibility, including the full system prompt for the instruction completer, contribute to the transparency and clarity of the research.

weaknesses:
- The introduced approach heavily relies on the inherent reasoning capabilities of LLMs. If the LLMs are not trained on diverse or comprehensive datasets, it could lead to biases or limitations in the agent's performance. The authors need to demonstrate the robustness of the prompt to different LLMs, e.g., Llama 3, and other open-sourced LLMs (gemma2, qwen2.5). 
- While the paper demonstrates strong performance in simulated environments, there might be concerns about how well ThinkBot's approach translates to real-world applications, where the complexity and variability can be significantly higher. Besides, the time cost of the reasoning process should be considered for real-world applications. However, LLM-based reasoning will lead to a high delay in the inference time. The use of LLMs and complex reasoning processes may require significant computational resources, which could limit the scalability of the ThinkBot system, especially in resource-constrained embodied systems.
- The paper does not extensively discuss how ThinkBot handles errors or recovers from incorrect actions once taken. This is an important aspect of robust autonomous agent design. It is important to make the agent self-improve during the interactions, instead of manually adjusting the prompt. This is also important to generalize the reasoning process to other scenarios or tasks. It is necessary to conduct experiments demonstrating the agent's ability to learn from mistakes (adding noise to actions) and improve performance over time without manual prompt adjustments or discuss how the reasoning process could be made more generalizable.

questions:
- How does the performance of ThinkBot vary when using different LLMs with potentially different training datasets? Have you tested the robustness of the prompt with LLMs such as Llama 3 or other open-sourced LLMs?
- How does the ThinkBot approach handle the increased complexity and variability of real-world environments compared to simulated ones?
- What is the inference time for the LLM-based reasoning process, and how does this delay impact the feasibility of ThinkBot for real-world applications? How does the computational resource requirement of the ThinkBot system impact its scalability, especially when considering the constraints of embodied systems in real-world scenarios?
- Can the agent learn from its mistakes to improve its performance over time, and is there a mechanism for self-improvement during interactions?
- How to address the generalization of the reasoning process to new scenarios or tasks, ensuring that the agent can adapt and learn from a wide range of situations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731916497312

thread_type:
public_comment

<thread_object>
creator: ~zhiwei_jia1

<content>
title:
An interesting approach for embodied instruction following

comment:
Hi authors,

Thanks for the great work! Please consider citing a relevant work [1] that takes an approach similar to the proposed multimodal object localizer by taking egocentric visual features and instance segmentation results to produce 2D semantic maps for downstream tasks in ALFRED.

[1] Learning to Act with Affordance-Aware Multimodal Neural SLAM, IROS 2022
</content>
</thread_object>

</thread>
<thread>
creation date:
1732591462541

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Response Summary

comment:
We express our gratitude to the four reviewers for their valuable feedback and recognition of our strengths, including 'showing significant improvements in success rates and execution efficiency over state-of-the-art methods' (Reviewer pZ3K), 'the proposed pipeline is technically sound' (Reviewer ZBDZ), 'an important and challenging problem of embodied instruction following' (Reviewer oWwC), and 'the paper is well-written and easy to understand' (Reviewer tHpP). 

We conduct and report additional experiments in the manuscript, and briefly conclude the major results here:
- In response to Reviwer pZ3K's question about inference speed, we report the average time per step of different LLMs and other components in the pipeline. The system executes at 1.37Hz on average, which is feasible in real-world settings.
- In response to Reviwer pZ3K and oWwC's question about generalizability, we implement Thinkbot in HM3D ObjectNav benchmark and a real-world language-conditioned manipulation task. We observe that Thinkbot enhances representative baselines with consistent improvements.
- In response to Reviewr pZ3K, ZBDZ and oWwC's question about the usage of different LLMs, we experiment with open-sourced LLMs, and include baselines using MLLMs and Prompter+ with advanced LLMs. The results confirm the proposed reasoning and feedback framework is robust and independent of specific LLM choices.

The answers are as follows. Please let us know if you have any additional questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes ThinkBot, a framework for embodied instruction following (EIF) that uses chain-of-thought-like reasoning to follow instructions in which some of the actions may not be explicitly conveyed. The "instruction completer" uses a large language model (LLM) to infer missing actions and object localizer helps to identify the location of the target objects. In experiments using the Alfred simulator, ThinkBot outperforms EIF baselines in terms of success rate and task efficiency.

The paper is highly topical---the use of large language models to facilitate instruction-following in embodied settings continues to attract significant attention. ThinkBot differs from some of this work in its focus on explicitly inferring actions that are important to the success of a task, but may not be mentioned explicitly, something that regularly happens in practice. The reviewers acknowledged the importance of this problem and at least two reviewers appreciated the contributions of the thought chain approach to recovering latent action descriptions. The reviewers noted the paper's extensive experimental evaluation, which demonstrates the effectiveness of ThinkBot relative to contemporary baselines. Meanwhile, at least two reviewers commented that the paper is well structured and easy to read. However, the reviewers raised some concerns about the paper as originally submitted. Among them is the method's sensitivity to the choice of LLMs and whether using an improved LLM would match the gains provided by ThinkBot. During the rebuttal, the authors provided experimental results with different language models, including some with GPT-4o, which suggest that ThinkBot's performance gains are robust, at least for now. Another concern was whether the method could generalize to real-world settings, due to their higher variability and complexity, as well as be capable of inference rates that are sufficiently fast for real-time use. The authors made an effort to address these concerns, included by reporting new experimental results, which the reviewer found satisfying. However, the AC disagrees with the claim that an operating frequency of 1.5 Hz is sufficient for practical use. Additionally, there were questions about how ThinkBot recovers after executing incorrect actions, which were addressed with results that involve deploying ThinkBot with an agent whose actions exhibit some randomness and, in turn, are more prone to error.

Overall, the paper addresses an important problem in embodied instruction following. The method itself is interesting, though the extent to which it is novel is not entirely clear. That said, the paper reveals respectable performance gains over existing methods. Most of the reviewers primary concerns were addressed during the rebuttal period.

The AC acknowledges that Reviewer ZBDZ's criticisms were not fully substantiated and that they did not respond to the authors' rebuttal or their attempts to engage the reviewer in discussion. As a result, the AC placed less weight on their review when making the recommendation.