Multi-Step Reasoning for Embodied Question Answering via Tool Augmentation

## Abstract

Abstract Embodied Question Answering (EQA) requires agents to explore 3D environments to obtain observations and answer questions related to the scene.
Existing methods leverage VLMs to directly explore the environment and answer questions without explicit thinking or planning, which limits their reasoning ability and results in excessive or inefficient exploration as well as ineffective responses.
In this paper, we introduce ToolEQA , an agent that integrates external tools with multi-step reasoning, where external tools can provide more useful information for completing the task, helping the model derive better exploration directions in the next step of reasoning and thus obtaining additional effective information.
This enables ToolEQA to generate more accurate responses with a shorter exploration distance.
To enhance the model’s ability for tool-usage and multi-step reasoning, we further design a novel EQA data generation pipeline that automatically constructs large-scale EQA tasks with reasoning trajectories and corresponding answers.
Based on the pipeline, we collect the EQA-RT dataset that contains about 18K tasks, divided into a training set EQA-RT-Train, and two test sets EQA-RT-Seen (scenes overlapping with the training set) and EQA-RT-Unseen (novel scenes).
Experiments on EQA-RT-Seen and EQA-RT-Unseen show that ToolEQA improves the success rate by 9.2 ∼ \sim 20.2% over state-of-the-art baselines, while outperforming the zero-shot ToolEQA by 10% in success rate.
In addition, ToolEQA also achieves state-of-the-art performance on the HM-EQA, OpenEQA, and EXPRESS-Bench datasets, demonstrating its generality.
Our homepage see https://tooleqa.github.io .

## 1 Introduction

Embodied Question Answering (EQA), a challenging task in computer vision and robotics, requires agents to navigate in a 3D environment, actively gather visual information through exploration, and answer questions about the scene (Das et al., 2018a).
Existing methods (Ziliotto et al., 2025; Ren et al., 2024a; Zhai et al., 2025; Cheng et al., 2024; Jiang et al., 2025) leverage VLMs to understand environment for guiding exploration and answering questions, but they generally lack explicit intermediate reasoning and planning.
For example, as shown in Figure [1](https://arxiv.org/html/2510.20310v2#S0.F1),
(1) The agent often answers the question before fully identifying all relevant objects, resulting in incorrect final answers due to insufficient information gathering capabilities.
(2) The agent makes suboptimal route plans, prolonging the exploration process and reduce efficiency due to limited reasoning abilities.
This motivates us to leverage tools to enhance the information-gathering capabilities of the agent, and use multi-step explicit reasoning to improve its reasoning ability during the exploration process, enabling it to complete EQA tasks with more efficient exploration distances.

In this paper, we propose ToolEQA, an agent that leverages tool augmentation to perform multi-step reasoning for EQA tasks. ToolEQA reasons over both current observations and historical information, selects appropriate tools to invoke, and integrates the additional information they provide (e.g., 3D bounding boxes) into the reasoning process. To ground reasoning in the environment, we abstract the action space into tool sets and execute them as actions. The agent iteratively reasons and applies tools, acquiring new observations until the final answer is derived. By effectively integrating collected information and identifying shorter exploration paths, ToolEQA improves both exploration efficiency and accuracy in solving EQA tasks.

To enhance the reasoning capability of the ToolEQA agent, we introduce a novel EQA data generation pipeline that automatically generates large-scale EQA tasks with reasoning trajectories via three steps: EQA task generation, reasoning trajectory generation, and validation.
Specifically, we first employ a 3D detection model to identify all objects in the current scene and extract their attributes, such as size and spatial coordinates.
Based on this object-level information, we then leverage GPT-4o (OpenAI, 2024a) to automatically generate diverse questions and their corresponding answers.
Subsequently, to generate optimal reasoning trajectories, we extract all relevant objects mentioned in the question and determine the shortest path by combining their positions with an A-star algorithm.
On top of this path, we incorporate reasoning steps and tool usage into the path by employing GPT-4o to generate complete trajectories.
To ensure the correctness of questions, we design question-type-specific prompt templates that guide the generation process, thereby ensuring both path optimality and consistency in task solving.
Finally, to preserve data quality, the generated EQA tasks and trajectories are passed through an EQA task verifier and trajectory verifier to discard low-quality data and rectify incorrect trajectories.

With the data generation pipeline, we construct EQA-RT, a dataset of 18K EQA question–answer pairs with reasoning trajectories.
We further split it into a training set (EQA-RT-Train) and two test sets, where two test sets contain EQA-RT-Seen (in-domain scenes overlapping with the training set) and EQA-RT-Unseen (out-of-domain scenes for evaluating generalization).
We train the proposed ToolEQA agent on EQA-RT-Train using supervised fine-tuning.
We comprehensively evaluate the tuned ToolEQA agent and the zero-shot ToolEQA agent on HM-EQA (Ren et al., 2024a), Open-EQA (Majumdar et al., 2024), ExpressBench (Jiang et al., 2025), EQA-RT-Seen and EQA-RT-Unseen.
The ToolEQA agent consistently achieves improvements on untrained VLMs and outperforms them by 11%.
This indicates that our method enables agents to have powerful capability for practical EQA tasks with complex and diverse trajectories.
In summary, our contributions are three-fold.

- •
We propose the ToolEQA agent which performs multi-step reasoning for environment exploration and question answering, achieving improved effectiveness and efficiency in solving EQA tasks.
- •
We introduce an EQA data generation pipeline that automatically generates large-scale EQA tasks with reasoning trajectories.
- •
We introduce EQA-RT, a dataset containing 18K question–answer pairs for EQA, covering diverse and complex question types with high-quality reasoning trajectories.

## 2 Realted Works

### 2.1 Embodied question answering

Embodied question answering (Das et al., 2018a; Gordon et al., 2018; Yu et al., 2019; Cangea et al., 2019; Das et al., 2018b) has become a challenging paradigm for testing a robot’s ability to autonomously plan tasks and establish semantic understanding of the environment in order to correctly answer natural language questions.
Yu et al. (2019) constructed a multi-target question answering dataset in a virtual environment and introduced a multi-target EQA method.
Ren et al. (2024a) first applied VLMs to EQA and built the HM-EQA dataset with more open-ended questions for realistic and diverse evaluation. Subsequent studies extended VLMs for EQA. Majumdar et al. (2024) used video memory for implicit questions with long-context VLMs. Saxena et al. (2024) embedded a planner into compact scene representations to bridge semantic memory and planning.
Jiang et al. (2025) added exploration-trajectory annotations to EQA datasets and incorporated exploration into evaluation metrics.
However, those methods demonstrate limited reasoning capacity, as the lack of explicit thinking and planning, which often leads to redundant or inefficient exploration.
To solve this problem, we introduce a multi-step reasoning process for solving EQA task via tool augmentation, enhancing effectiveness and efficiency.

### 2.2 Multi-step Reasoning

Multi-step reasoning can significantly enhance a model’s ability to solve complex tasks while improving interpretability.
In recent years, research on multi-step reasoning in large language models (LLMs) (Ranaldi et al., 2024; OpenAI, 2024b; Chen et al., 2024; Yao et al., 2025) and multi-modal systems has made notable progress.
Li et al. (2024) proposed the VoCoT framework, which integrates vision-guided and object-centric chain-of-thought reasoning to improve the reasoning performance of large-scale multimodal models on complex tasks.
The ReAct (Yao et al., 2023) framework is a multi-step reasoning paradigm that decomposes tasks through iterative Reason–Act–Observe cycles, which greatly benefits to solving challenging problems.
In the domain of Embodied Question Answering (EQA), Fine-EQA (Jiang et al., 2025) introduced a new benchmark that emphasizes dynamic exploration and multi-step reasoning in 3D environments, aiming to improve both exploration efficiency and evaluation metrics.
However, the exploration of multi-step reasoning in embodied question answering remains at a relatively early stage.
We propose ToolEQA, which integrates explicit multi-step reasoning into the multi-step exploration process in embodied scenarios. This step-by-step thinking strategy not only shortens exploration paths but also enhances the accuracy of question answering.

### 2.3 Tool Usage Agent

Recent work have equipped VLMs with tool-usage capabilities.
Frameworks such as ReAct (Yao et al., 2023) and Toolformer (Schick et al., 2023) demonstrated the effectiveness of coupling reasoning traces with tool execution, while embodied agents like SayCan (Ahn et al., 2022) showed how language-guided tool usage can translate high-level instructions into low-level actions.
T3-Agent (Gao et al., 2024) leveraged automatically generated multimodal tool-usage data and fine-tunes vision-language models (VLMs) as controllers to enable strong tool-based reasoning.
Li et al. (2025) proposed the MeCo framework, which captures the model’s “cognitive signals” to assess its capability boundaries and thereby decide whether to invoke external tools.
These works enhance multi-modal reasoning by calling predefined tools to acquire additional information.
However, such methods perform reasoning only within static cyberspace. In contrast, we define the physical environment itself as a tool, thereby situating reasoning steps within embodied interactions and enabling more autonomous embodied agents.

Figure: Figure 2: Overview of the ToolEQA agent workflow.
Refer to caption: x2.png

## 3 ToolEQA Agent

To enable the agent to reason and act in complex environments, we propose a ToolEQA agent that integrates tool-usage strategies for the reasoning process.
ToolEQA conducts step-by-step reasoning on past observations and, at each step, generates corresponding thoughts and code to execute tools.
Code offers greater flexibility than formats such as JSON for handling diverse inputs and outputs.
As shown in Figure [2](https://arxiv.org/html/2510.20310v2#S2.F2), ToolEQA comprises three components: a planner for generating overall task plan $p$, a controller for generating thought $t$ and code $c$, and an executor for executing code in environment.
Given a query $Q$ and a scene $S$, the $i$-step of the agent is formulated as

$$ $\displaystyle t_{i}^{*},c_{i}^{*}=\arg\max P(t_{i},c_{i}|Q,S,h_{i},p),$ (1) $$

where $t_{i}^{*}$ and $c_{i}^{*}$ are generated thought and code for the $i$-th step, and $h_{i}=\{t_{1},c_{1},o_{1},...,t_{i-1},c_{i-1},o_{i-1}\}$ is the history (thought, code, and observation of previous steps).

#### Planner.

Given an EQA task, the planner, modeled as an LLM, takes the query as input, interprets the task objectives, and outputs an overall plan that decomposes the task into sub-goals.
The structured sub-goals are provided to the controller
to prevent blind exploration, enhancing the efficiency and accuracy of task execution.

#### Executor.

We deploy real-executable tools for the agent. Our tools include GoNextPoint, ObjectLocation2D, ObjectLocation3D, ObjectCrop, VisualQA, FinalAnswer, the details of tools see Appendix [A.3](https://arxiv.org/html/2510.20310v2#A1.SS3).
With the generated code, the executor calls executable tools in the environment to obtain new observations for further exploration, thus solving the EQA task.

#### Controller.

The controller performs dynamic reasoning for deciding which tool to use and executing it, on the basis of the question and the previous observations, with the guidance of plans. By invoking the executor to gather new observations for further reasoning until the answer is derived.
The reasoning process can be divided into three situations.

- •
The collected information is insufficient and the current scene lacks required objects. ToolEQA infers missing objects from previous observations and the query, and estimates their likely positions. Then, ToolEQA combines these estimates with its current location to decide a walking direction, and uses the ‘GoNextPoint’ (for example, ‘GoNextPoint(“turn left”)’) to gather the needed information.
- •
The collected information is insufficient and the current scene contains involved objects. ToolEQA reasons over the question and invokes suitable tools to obtain relevant information. For example, as shown in Figure [1](https://arxiv.org/html/2510.20310v2#S0.F1), ToolEQA utilizes ‘ObjectLocation3D’ to extract the size of objects, and then continues exploration following the above step.
- •
The collected information is sufficient. ToolEQA needs to reason over the question, use appropriate tools on the current image, and integrate information for the final answer. For example, it processes the detected object, then writes ‘Python’ codes for comparing sizes, and employs ‘FinalAnswer’ to produce the ultimate output.

Figure: Figure 3: EQA Data Generation Pipeline.
Refer to caption: x3.png

## 4 EQA Data Generation Pipeline

### 4.1 Formulation

#### Data Format.

We format the EQA tool-usage data as $\{S,pos,Q,p,T,C,O,A\}$, where $S$ denotes the scene, $pos$ denotes the initial position of agents, $Q$ denotes the question, $p$ denotes the overall plan for solving the task, $T$ denotes the generated thought, $C$ denotes the generated code, $O$ denotes observation (outputs of using tools), and $A$ means the ground truth answer.
Considering that solving one real-world EQA task may require multiple steps involving multiple tools, $T$, $C$, and $O$ can be represented by the integration of thought, code, and observation in multiple steps, and the data format is reformulated as $\{S,pos,Q,p,\{t_{1},\cdots,t_{n}\},\{c_{1},\cdots,c_{n}\},\{o_{1},\cdots,o_{n}\},A\}$, where $t_{i},c_{i}$, and $o_{i}$ indicate the thought, code, and observation in the $i$-th step respectively, and there are $n$ steps in total. The thought, code, and observation are composed of a trajectory $\{t_{1},c_{1},o_{1},\cdots,t_{n},c_{n},o_{n}\}$ of $n$ steps to solve the task.

#### Scene Source.

HM3D (Ramakrishnan et al., 2021) is a comprehensive dataset comprising 3D reconstructions of 1,000 large-scale buildings collected from diverse real-world locations.
We select 713 high-quality scenes from HM3D as our data source, sample object images from them, and generate questions and answers.

The proposed data generation pipeline is shown in Figure [3](https://arxiv.org/html/2510.20310v2#S3.F3), including three steps: EQA task generation, reasoning trajectory generation, and data verification.

### 4.2 EQA task generation

Our goal is to generate a large set of diverse, practical, and complex EQA tasks.
We first apply a 3D detection model to obtain each object’s bounding box, position, and category, and sample the object image from detected objects.
The object attributes and corresponding visual information are then fed into GPT-4o along with example question-answer pairs designed from brainstorming to simulate natural home conversations.
Guided by the prompt, GPT-4o generates questions and answers across six types: relationship, status, distance, location, counting, and attribute, where location is divided into two subcategories ‘location-location’ and ‘location-special’, and attribute is divided into three subcategories ‘color’, ‘special’, and ‘size’.
The answers are open-ended or multiple-choice, enabling the evaluating different capabilities of agents.

### 4.3 Reasoning Trajectory Generation

Given an EQA task, we construct an exploration trajectory that records reasoning steps, tool selections, and observations. The trajectory is constrained to follow the shortest path and ensure consistency between reasoning and tool usage. We extract objects mentioned in the question using their locations and the agent’s position, and compute the shortest path using the A^∗ algorithm, generating intermediate waypoints and navigation directions.

Based on these trajectories, GPT-4o enriches each step with reasoning and tool selections.
Steps are categorized as key, where the target object is found, and non-key, where it is not.
For non-key steps, GPT-4o receives the current image and exploration direction to generate reasoning.
For the key steps, we select possible tools required to solve the task from the toolset, and then prompt GPT-4o to output which specific tool should be invoked under the current observation and the corresponding rationale.
To ensure consistency and rationality, we design question-type–specific prompts containing task-specific considerations, reasoning strategies, tool-usage guidelines, and examples, allowing GPT-4o to produce thought and code across different question types.

### 4.4 Data Verification

To preserve the quality of generated data, we design an EQA task verifier and a trajectory verifier to filter out low-quality data.
Using LLMs to verify generated tasks and trajectories has proven effective (Gao et al., 2024; Liu et al., 2024). Inspired by this, we use LLMs to verify generated tasks and trajectories.

EQA Task Verifier. Since object descriptions in generated questions or options may not always match the scene, we use two complementary strategies: confidence-based matching and LLM-based structured scoring to evaluate quality and filter out low-quality samples.
For confidence, we first extract object descriptions from the question and options, locate the corresponding objects in the scene to obtain their images, and then use Grounded-SAM (Ren et al., 2024b) to compute a score reflecting how well each image matches its description.
For LLM-based scoring, we feed the question and object images into GPT-4o, which outputs a similarity score.
We set thresholds for the two strategies respectively, and samples below thresholds on either score are filtered out.

Trajectory Verifier.
To verify the rationality of tool usage and reasoning in the generated trajectories, we adopt two strategies: rule-based and LLM-based validation.
For the rule-based validation, we design several checks:
(1) the key tools should exist and be invoked at the correct step (e.g., GoNextPoint should be called at every step before reaching the target);
(2) the invoked tool should belong to the predefined tool set, and its parameters should be passed correctly.
For the LLM-based validation, we prompt GPT-4o to consider the following factors:
(1) the predicted answer should be semantically consistent with the ground-truth answer;
(2) the reasoning in non-key steps should avoid hallucinations;
(3) the final answer should be reasonably derived from the reasoning and observations.
We set a threshold for three checks, and samples below the threshold are filtered out.

### 4.5 EQA-RT

Figure: (a) EQA-RT-Train

By utilizing the developed EQA task generation pipeline, we construct EQA-RT, encompassing about 18K EQA tasks. We further split it into a training set (EQA-RT-Train), a seen test set (EQA-RT-Seen) and a unseen test set (EQA-RT-Unseen), where the test set contains both in-domain scenes overlapping with the training set and out-of-domain scenes for evaluating generalization.
It is worth noting that the tasks in EQA-RT-Seen are more challenging than those in EQA-RT-Unseen, with detailed evidence provided in Appendix [A.1](https://arxiv.org/html/2510.20310v2#A1.SS1).
As shown in Figure [4](https://arxiv.org/html/2510.20310v2#S4.F4), we show the question types of the generated EQA tasks in EQA-RT-train, EQA-RT-Seen and EQA-RT-Unseen. More statistical data can be found in Appendix [A.1](https://arxiv.org/html/2510.20310v2#A1.SS1).

### 4.6 Training

Given a data point $\{S,pos,Q,p\{t_{1},...,t_{n}\},\{c_{1},...,c_{n}\},\{o_{1},...,o_{n}\},A\}$, we train the VLM controller using the cross-entropy loss,

$$ $\displaystyle\min\mathbb{E}_{(Q,S,pos,T,C,O,A)\sim\mathbb{D}}\left[-\sum_{i=1}^{n}P(t_{i},c_{i}|Q,S,pos,h_{i})\right],$ (2) $$

where $\mathbb{D}$ is the EQA-RT-Train dataset and we sum the loss values of the n steps in the trajectory.
Note that, in training VLMs, we do not fit the final answer $A$, as we encourage the controller to leverage tools in solving given tasks, instead of directly producing an answer based on biases in VLMs.
The average length of exploration and reasoning trajectories reaches 12.69 steps (as shown in the statistics in Appendix Table [6](https://arxiv.org/html/2510.20310v2#A1.T6)). A longer number of steps results in an extended trajectory history $h$, which in turn enlarges the model input and ultimately causes substantial time and memory consumption during training.
To address this issue, we propose a trajectory sampling strategy. Specifically, we retain the key steps and randomly sample an equal number of non-key steps to reduce resource overhead. This design is motivated by the fact that non-key steps dominate the exploration process and are highly redundant, and they mostly consist of repeated direction predictions and frequent use of the GoNextPoint tool. In contrast, key steps involve diverse tool usage and reasoning changes.
After training, ToolEQA agent can present powerful ability of reasoning and tool-usage, further enhance the effectiveness and efficiency of solving EQA tasks.

**Table 1: Baseline evaluation on EQA-RT-Seen.**
| Setting | Model | $recall$ $\uparrow$ | $e_{path}$ $\uparrow$ | succ. (%) $\uparrow$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $@5$ | $@10$ | $@15$ | $@5$ | $@10$ | $@15$ |  |  |  |
| Multi<br>Choices | Explore-EQA | 0.06 | 0.11 | 0.14 | 0.04 | 0.07 | 0.09 | 44.7 |
| Memory-EQA | 0.06 | 0.12 | 0.13 | 0.04 | 0.07 | 0.11 | 48.2 |  |
| ToolEQA (gpt-4o) | 0.06 | 0.14 | 0.19 | 0.08 | 0.2 | 0.27 | 55.37 |  |
| ToolEQA (qwen2.5vl) | 0.04 | 0.09 | 0.11 | 0.06 | 0.13 | 0.17 | 53.1 |  |
| ToolEQA (qwen2.5vl ft) | 0.06 | 0.15 | 0.21 | 0.07 | 0.23 | 0.3 | 57.31 |  |
| Open<br>Vocabulary | Explore-EQA | 0.04 | 0.10 | 0.13 | 0.04 | 0.06 | 0.09 | 30.6 |
| Memory-EQA | 0.05 | 0.10 | 0.13 | 0.04 | 0.09 | 0.11 | 35.1 |  |
| ToolEQA (gpt-4o) | 0.06 | 0.15 | 0.21 | 0.07 | 0.2 | 0.27 | 49.2 |  |
| ToolEQA (qwen2.5vl) | 0.05 | 0.12 | 0.16 | 0.03 | 0.10 | 0.14 | 44.9 |  |
| ToolEQA (qwen2.5vl ft) | 0.06 | 0.15 | 0.20 | 0.08 | 0.22 | 0.3 | 53.6 |  |

**Table 2: Baseline evaluation on EQA-RT-Unseen.**
| Setting | Model | $recall$ $\uparrow$ | $e_{path}$ $\uparrow$ | succ. (%) $\uparrow$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $@5$ | $@10$ | $@15$ | $@5$ | $@10$ | $@15$ |  |  |  |
| Multi<br>Choices | Explore-EQA | 0.06 | 0.12 | 0.15 | 0.05 | 0.08 | 0.10 | 47.0 |
| Memory-EQA | 0.06 | 0.13 | 0.14 | 0.06 | 0.09 | 0.11 | 48.9 |  |
| Too l (gpt-4o) | 0.07 | 0.16 | 0.21 | 0.08 | 0.2 | 0.28 | 57.9 |  |
| ToolEQA (qwen2.5vl) | 0.04 | 0.11 | 0.13 | 0.07 | 0.16 | 0.21 | 55.3 |  |
| ToolEQA (qwen2.5vl ft) | 0.07 | 0.14 | 0.19 | 0.08 | 0.24 | 0.3 | 59.5 |  |
| Open<br>Vocabulary | Explore-EQA | 0.05 | 0.09 | 0.15 | 0.05 | 0.10 | 0.13 | 31.4 |
| Memory-EQA | 0.05 | 0.09 | 0.15 | 0.06 | 0.15 | 0.18 | 35.9 |  |
| ToolEQA (gpt-4o) | 0.06 | 0.16 | 0.21 | 0.06 | 0.21 | 0.27 | 49.3 |  |
| ToolEQA (qwen2.5vl) | 0.06 | 0.13 | 0.17 | 0.05 | 0.16 | 0.2 | 45.1 |  |
| ToolEQA (qwen2.5vl ft) | 0.08 | 0.17 | 0.24 | 0.09 | 0.24 | 0.32 | 56.1 |  |

## 5 Experiments

### 5.1 Setting

we tested ToolEQA on the EQA-RT and HM-EQA (Ren et al., 2024a) datasets and compared it with existing open-source methods, Explore-EQA (Ren et al., 2024a) and Memory-EQA (Zhai et al., 2025). We also examined the impact of different models (GPT-4o (OpenAI, 2024a), Qwen2.5-VL-7B (Wang et al., 2024), and fine-tuned Qwen2.5-VL-7B (Wang et al., 2024)) as controllers on performance. In addition, we conducted a qualitative analysis of ToolEQA, investigating how reasoning and tool invocation affect the efficiency and success rate of completing EQA tasks.

#### Training

We trained the controller using the EQA-RT training set.
During the training of the VLM-based controller, we froze the vision encoder and the visual token compressor, and fine-tuned the language model with LoRA (Hu et al., 2022).
We adopted the AdamW optimizer with a cosine annealing scheduler, using a learning rate of 1e-6 and a batch size of 1. We used 4 Nvidia Tesla H100 GPUs to train for 2 days.

#### Metrics

We use three metrics for evaluating ToolEQA and existing EQA methods.
The success rate is divided into two parts, for multi-choices tasks, we calculate average accuracy between the output of model and ground truth answer; for open vocabulary tasks, we prompt LLM to obtain the semantic similarity between the output of model and ground truth answer.
$recall@D$ is used to evaluate whether objects related to the problem were found during the
exploration process.
$e_{path}@D$ is an indicator that combines success rate, recall, and exploration path length.
The details of the metrics can be found in Appendix [A.2](https://arxiv.org/html/2510.20310v2#A1.SS2).

**Table 3: Comparison between the original model and the finetuned model in terms of the number of key steps, thought length, correct tool usage rate, and success rate on the EQA-RT-Unseen dataset.**
| Model | Step | Thought | Tool | Succ. |
| --- | --- | --- | --- | --- |
| ToolEQA (0-shot) | 1.24 | 90.26 | 58 | 45.1 |
| ToolEQA (ft) | 1.98 | 116.15 | 69 | 56.1 |

### 5.2 Main Results

As shown in Table [1](https://arxiv.org/html/2510.20310v2#S4.T1) and Table [2](https://arxiv.org/html/2510.20310v2#S4.T2), we report the performance of different methods on EQA-RT-Seen and EQA-RT-Unseen. Our ToolEQA consistently outperforms reasoning-inefficient methods Explore-EQA (Ren et al., 2024a) and Memory-EQA (Zhai et al., 2025) across all metrics, demonstrating its effectiveness in tackling complex tasks.
The comparison between agents equipped with fine-tuned and non-fine-tuned VLMs further validates the effectiveness of our data generation pipeline.
The success rate of fine-tuned Qwen2.5VL-7B compared to the original Qwen2.5VL-7B on EQA-RT-Unseen improved from 45.1 to 56.1, the recall rate increased from 0.17 to 0.24, and $e_{path}$ improved from 0.2 to 0.32.
Compared with the non-fine-tuned Qwen2.5-VL-7B, ToolEQA with GPT-4o achieves better performance, indicating that the controller’s capability directly determines the performance of ToolEQA.
However, the fine-tuned Qwen2.5-VL-7B surpasses GPT-4o in $e_{path}$ and success rate, while achieving comparable recall.
This indicates that our training has enabled the VLM to learn how to think and solve problems more effectively in indoor scenarios.

We typically consider that the length of thoughts is positively correlated with reasoning ability (Jin et al., 2024). Therefore, we evaluate the impact of fine-tuning the VLM on EQA-RT-Unseen with respect to thought length and the accuracy of tool calls (i.e., the proportion of calls that successfully acquire the information required to answer the question).
As shown in Table [5](https://arxiv.org/html/2510.20310v2#S5.T5), after fine-tuning, thought length increases from 90.26 to 116.15, tool call accuracy improves from 58 to 69, and the success rate rises from 45.1 to 56.1. This indicates that reasoning ability is crucial for accomplishing the EQA task.

As shown in Table [5](https://arxiv.org/html/2510.20310v2#S5.T5), we compare our method with Fine-EQA (Jiang et al., 2025) on EXPRESS-Bench (Jiang et al., 2025), and with Explore-EQA (Ren et al., 2024a), Efficient-EQA (Cheng et al., 2024), Memory-EQA (Zhai et al., 2025), and Graph-EQA (Saxena et al., 2024) on HM-EQA (Ren et al., 2024a) and OpenEQA (Majumdar et al., 2024). Our method achieves comparable success rates with shorter exploration distances across all these datasets.

Figure: Figure 5: Illustration of how explicit reasoning guides efficient exploration, enabling ToolEQA to answer questions faster and more accurately.
Refer to caption: x7.png

Figure: Figure 6: Demonstration that the visual tools outperforms direct VLM inference without tools by accurately localizing, and comparing object volume.
Refer to caption: x8.png

### 5.3 Qualitative results

As shown in Figure [5](https://arxiv.org/html/2510.20310v2#S5.F5), the decision-making process integrates spatial layout, functional space, and environmental cues to guide navigation toward the target object (towels). Each step is supported by clear reasoning, such as moving forward to approach potential bathroom space, turning right to explore a promising corridor, or turning left after excluding non-target rooms. This information-driven and reasoning-based decision paradigm ensures that the generated exploration trajectory maintains a high degree of proximity to the ground-truth trajectory, effectively validating the rationality and effectiveness of the decision-making framework in target-oriented spatial exploration tasks.

Figure [6](https://arxiv.org/html/2510.20310v2#S5.F6) highlights the clear advantage of tool-driven reasoning over direct Visual-Language Model (VLM) inference. Without tools, VLM often fails to localize objects precisely or distinguish fine-grained attributes such as size. In contrast, by integrating these specialized tools, our method obtained critical, fine-grained information (precise object localization, clutter-free cropping, and accurate size identification) that cannot be reliably captured by direct VLM inference on unprocessed images. This structured tool usage ensured the final comparison concluding that “The chair has smaller volume” was grounded in objective data, ultimately achieving a more accuracy response than would be possible with VLM alone.

## 6 Conclusion

In this work, we introduced ToolEQA, an embodied question answering agent that integrates explicit multi-step reasoning with tool usage to guide exploration and information acquisition. By coupling dynamic reasoning with executable tools, ToolEQA enables more efficient exploration paths and more reliable utilization of gathered observations. To support training, we proposed a scalable EQA data generation pipeline and constructed EQA-RT, a dataset of 18K automatically generated tasks with validated reasoning trajectories. Comprehensive experiments on HM-EQA, OpenEQA, ExpressBench and EQA-RT demonstrate that ToolEQA achieves significant improvements in both accuracy and efficiency over prior methods.
These results highlight the importance of explicit multi-step reasoning and tool-usage in EQA agents, and suggest promising directions for developing more generalizable and interpretable frameworks for complex embodied AI tasks.

## 7 Ethics Statement

This research adheres to the ICLR Code of Ethics. We confirm that all aspects of the study were conducted with the highest ethical standards. The work does not involve human subjects, and no personally identifiable or sensitive data were used. All datasets (HM3D, HM-EQA, OpenEQA, ExpressBench and EQA-RT) utilized were publicly available and comply with privacy regulations. We have taken steps to ensure that the models and algorithms developed are fair and free from bias. No conflicts of interest or external sponsorship influenced this work. Additionally, the findings are presented honestly, with all methodologies thoroughly validated for accuracy and reproducibility. We believe this research aligns with the ethical guidelines set forth by ICLR and contributes to the integrity and transparency of the scientific community.

## 8 Reproducibility Statement

To ensure the reproducibility of our results, we provide an anonymous code repository [https://anonymous.4open.science/r/ReactEQA-DDE3](https://anonymous.4open.science/r/ReactEQA-DDE3) that includes the complete implementation of data generation, data validation, and the ToolEQA framework. The repository also contains a comprehensive README.md file that details the installation requirements, dataset preparation, and step-by-step execution instructions. By following these guidelines, all experiments and results reported in this paper can be reproduced.

## References

- Ahn et al. (2022)
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al.
Do as i can, not as i say: Grounding language in robotic affordances.
*arXiv preprint arXiv:2204.01691*, 2022.
- Cangea et al. (2019)
Cătălina Cangea, Eugene Belilovsky, Pietro Liò, and Aaron Courville.
Videonavqa: Bridging the gap between visual and embodied question answering.
*arXiv preprint arXiv:1908.04950*, 2019.
- Chen et al. (2024)
Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che.
M3cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought.
*arXiv preprint arXiv:2405.16473*, 2024.
- Cheng et al. (2024)
Kai Cheng, Zhengyuan Li, Xingpeng Sun, Byung-Cheol Min, Amrit Singh Bedi, and Aniket Bera.
Efficienteqa: An efficient approach for open vocabulary embodied question answering.
*arXiv preprint arXiv:2410.20263*, 2024.
- Das et al. (2018a)
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra.
Embodied question answering.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 1–10, 2018a.
- Das et al. (2018b)
Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra.
Neural modular control for embodied question answering.
In *Conference on robot learning*, pp. 53–62. PMLR, 2018b.
- Gao et al. (2024)
Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaojian Ma, Tao Yuan, Yue Fan, Yuwei Wu, Yunde Jia, Song-Chun Zhu, and Qing Li.
Multi-modal agent tuning: Building a vlm-driven agent for efficient tool usage.
*arXiv preprint arXiv:2412.15606*, 2024.
- Gordon et al. (2018)
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi.
Iqa: Visual question answering in interactive environments.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 4089–4098, 2018.
- Hu et al. (2022)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.
Lora: Low-rank adaptation of large language models.
*ICLR*, 1(2):3, 2022.
- Jiang et al. (2025)
Kaixuan Jiang, Yang Liu, Weixing Chen, Jingzhou Luo, Ziliang Chen, Ling Pan, Guanbin Li, and Liang Lin.
Beyond the destination: A novel benchmark for exploration-aware embodied question answering.
*arXiv preprint arXiv:2503.11117*, 2025.
- Jin et al. (2024)
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du.
The impact of reasoning step length on large language models.
*arXiv preprint arXiv:2401.04925*, 2024.
- Li et al. (2025)
Wenjun Li, Dexun Li, Kuicai Dong, Cong Zhang, Hao Zhang, Weiwen Liu, Yasheng Wang, Ruiming Tang, and Yong Liu.
Adaptive tool use in large language models with meta-cognition trigger.
*arXiv preprint arXiv:2502.12961*, 2025.
- Li et al. (2024)
Zejun Li, Ruipu Luo, Jiwen Zhang, Minghui Qiu, Xuanjing Huang, and Zhongyu Wei.
Vocot: Unleashing visually grounded multi-step reasoning in large multi-modal models.
*arXiv preprint arXiv:2405.16919*, 2024.
- Liu et al. (2024)
Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh RN, et al.
Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets.
*Advances in Neural Information Processing Systems*, 37:54463–54482, 2024.
- Majumdar et al. (2024)
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al.
Openeqa: Embodied question answering in the era of foundation models.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 16488–16498, 2024.
- OpenAI (2024a)
OpenAI.
Hello gpt-4o.
2024a.
URL [https://openai.com/index/hello-gpt-4o⁠](https://openai.com/index/hello-gpt-4o%E2%81%A0).
- OpenAI (2024b)
OpenAI.
Introducing openai o1-preview, 2024b.
URL [https://openai.com/index/introducing-openai-o1-preview/](https://openai.com/index/introducing-openai-o1-preview/).
- Ramakrishnan et al. (2021)
Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al.
Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.
*arXiv preprint arXiv:2109.08238*, 2021.
- Ranaldi et al. (2024)
Leonardo Ranaldi, Giulia Pucci, Barry Haddow, and Alexandra Birch.
Empowering multi-step reasoning across languages via program-aided language models.
In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 12171–12187, 2024.
- Ren et al. (2024a)
Allen Z Ren, Jaden Clark, Anushri Dixit, Masha Itkina, Anirudha Majumdar, and Dorsa Sadigh.
Explore until confident: Efficient exploration for embodied question answering.
*arXiv preprint arXiv:2403.15941*, 2024a.
- Ren et al. (2024b)
Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, et al.
Grounding dino 1.5: Advance the” edge” of open-set object detection.
*arXiv preprint arXiv:2405.10300*, 2024b.
- Saxena et al. (2024)
Saumya Saxena, Blake Buchanan, Chris Paxton, Bingqing Chen, Narunas Vaskevicius, Luigi Palmieri, Jonathan Francis, and Oliver Kroemer.
Grapheqa: Using 3d semantic scene graphs for real-time embodied question answering.
*arXiv preprint arXiv:2412.14480*, 2024.
- Schick et al. (2023)
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
Toolformer: Language models can teach themselves to use tools.
*Advances in Neural Information Processing Systems*, 36:68539–68551, 2023.
- Wang et al. (2024)
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al.
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.
*arXiv preprint arXiv:2409.12191*, 2024.
- Yao et al. (2025)
Huanjin Yao, Jiaxing Huang, Yawen Qiu, Michael K Chen, Wenzheng Liu, Wei Zhang, Wenjie Zeng, Xikun Zhang, Jingyi Zhang, Yuxin Song, et al.
Mmreason: An open-ended multi-modal multi-step reasoning benchmark for mllms toward agi.
*arXiv preprint arXiv:2506.23563*, 2025.
- Yao et al. (2023)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
In *International Conference on Learning Representations (ICLR)*, 2023.
- Yu et al. (2019)
Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L Berg, and Dhruv Batra.
Multi-target embodied question answering.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 6309–6318, 2019.
- Zhai et al. (2025)
Mingliang Zhai, Zhi Gao, Yuwei Wu, and Yunde Jia.
Memory-centric embodied question answer.
*arXiv preprint arXiv:2505.13948*, 2025.
- Ziliotto et al. (2025)
Filippo Ziliotto, Tommaso Campari, Luciano Serafini, and Lamberto Ballan.
Tango: training-free embodied ai agents for open-world tasks.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pp. 24603–24613, 2025.

## Appendix A Appendix

### A.1 Data Statistic

Figure: (a) Statistics on the number of questions with different steps counts.
Refer to caption: x9.png

**Table 6: Statistic about average exploration steps, the average number of tools used per task, and the average exploration length.**
|  | Step | Tool | Length (m) |
| --- | --- | --- | --- |
| EQA-RT-Train | 12.74 | 12.39 | 13.13 |
| EQA-RT-Seen | 12.56 | 12.20 | 12.71 |
| EQA-RT-Unseen | 12.13 | 11.76 | 12.38 |
| EQA-RT | 12.69 | 12.35 | 13.07 |

As shown in Figure [7](https://arxiv.org/html/2510.20310v2#A1.F7), the dataset exhibits a pronounced long-tail distribution in both the average exploration steps and the number of related objects per question. Most questions require around ten exploration steps; among them, 10,224 involve a single target, 7,098 involve two targets, and 940 involve three or more objects. Figure [7](https://arxiv.org/html/2510.20310v2#A1.F7)(c) further compares EQA-RT-Seen and EQA-RT-Unseen in terms of the average number of objects per question, revealing that EQA-RT-Seen involves more objects. Since object count reflects task difficulty, this suggests that tasks in EQA-RT-Seen are more challenging.

In addition, as shown in Table [6](https://arxiv.org/html/2510.20310v2#A1.T6), we report the exact values of the average exploration steps, the average number of tools used per task, and the average exploration length across different sets. By comparing the statistics of EQA-RT-Seen and EQA-RT-Unseen, it can also be inferred that the tasks in EQA-RT-Seen are more challenging.

### A.2 Metric Details

To comprehensively evaluate the effectiveness of our approach, we use $recall@D$, $e_{path}@D$ and success rate as metrics.

The success rate is divided into two parts, for multi choices tasks, we calculate average accuracy between the output of model and ground truth answer; for open vocabulary, we prompt LLM to obtain the semantic similarity $\sigma_{i}\in\{0,1,2,3,4,5\}$ between the output of model and ground truth answer, and then calculate $\textit{LLM-Match Score}=\frac{1}{N}\sum_{i=1}^{N}\frac{\sigma_{i}}{5}\times 100\%$.

The $recall@D$ is used to evaluate whether objects related to the problem were found during the exploration process.
So we first define $n$ be the number of objects and $T$ the number of camera steps.
At step $t$, the camera position is $\mathbf{p}_{t}\in\mathbb{R}^{3}$ and its yaw angle (around the $y$-axis) is $\theta_{t}$.
The forward unit vector of the camera is $\mathbf{f}_{t}=(\sin\theta_{t},\,0,\,-\cos\theta_{t})$.
The position of object $j$ is $\mathbf{o}_{j}\in\mathbb{R}^{3}$, and the distance from the camera to the object is $d_{j,t}=\|\mathbf{o}_{j}-\mathbf{p}_{t}\|$.
The $recall@D$ can be formalized as

$$ $recall@D=\frac{1}{n}\sum_{j=1}^{n}\;\max_{t}\left\{\Big(1-\tfrac{d_{j,t}}{D}\Big)\;\mathbf{1}\!\left[d_{j,t}\leq D,\;\frac{\mathbf{f}_{t}\cdot(\mathbf{o}_{j}-\mathbf{p}_{t})}{\|\,\mathbf{o}_{j}-\mathbf{p}_{t}\,\|}\;\geq\;\cos\!\left(\tfrac{\mathrm{FOV}}{2}\right)\right]\right\},$ $$

where $d_{j,t}=||\mathbf{o}_{j}-\mathbf{p}_{t}||$, $\mathbf{f}_{t}=(\sin{{\theta_{t}},0,-\cos{\theta_{t}}})$, $\mathbf{1[\cdot]}$ is indicator function.

The $e_{path}@D$ is an indicator that combines success rate, recall, and exploration path length.
The specific calculation process is as follows

$$ $e_{path}@D=\frac{1}{N}\sum_{j=1}^{N}(\text{success rate})\times recall@D\times\exp(\frac{l_{i}}{\max(p_{i},l_{i})}),$ $$

where $l_{i}$ is length of the shortest path, $p_{i}$ is length of the exploring path.

### A.3 Tools Description

**Table 7: The description of tools.**
| Tool | Description |
| --- | --- |
| GoNextPoint | The agent conitnue explore next point and obtain next observation (rgb image). |
| ObjectLocation2D | A tool that can localize objects in given images, outputing the bounding boxes of the objects. |
| ObjectLocation3D | Localize 3D objects in the scene and return their 3D bounding boxes and center coordinates. |
| ObjectCrop | Given the bounding boxes of objects, crop and save the relevant objects from the image. |
| SegmentInstance | A tool that can do instance segmentation on the given image. |
| VisualQA | A tool that can answer questions about attached images. |
| FinalAnswer | Provides a final answer to the given problem. |

As shown in Table [7](https://arxiv.org/html/2510.20310v2#A1.T7), we present all the tools used and their corresponding descriptions.

### A.4 Efficiency Analysis

**Table 8: Efficiency Analysis about time consumption (Time), LLM Token usage (Token) and GPU Memory usage (Memory).**
|  | Time (s) | Token | Memory (G) |
| --- | --- | --- | --- |
| Planner | 0.52 | 118 | - |
| Controller | 25.4 | 9512.7 | 40.5 |
| Executor | 5.54 | - | - |

We conducted an efficiency analysis of ToolEQA, and Table [8](https://arxiv.org/html/2510.20310v2#A1.T8) reports the time consumption, LLM token consumption, and memory usage of each module. It is worth noting that the planner is executed only once before the exploration begins, whereas the controller and executor run continuously throughout the entire exploration process.

### A.5 The Use of Large Language Models

A large language model (ChatGPT, Deepseek-R1, Doubao) was employed during manuscript preparation. The model was used for grammar checking, sentence refinement, and improving the readability of the text. In addition, the model was consulted for assistance in drafting segments of project code, primarily for debugging and improving code efficiency. Within the ToolEQA framework, large language models were also utilized as the Controller, including GPT-4o and Qwen2.5VL-7B, to generate control decisions during exploration. All scientific ideas, analyses, and final implementations were designed and verified by the authors.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1.The ToolEQA framework closely mirrors existing tool-augmented LLM paradigms. Could you elaborate on the unique challenges that embodiment introduces and how ToolEQA's architecture was specifically adapted to address these? For instance, how does the system handle or recover from incorrect predictions from its perception tools? What is the core innovation beyond applying a known method to a new domain?
2.A significant portion of the paper's contribution is the EQA-RT dataset, which is generated by a pipeline that naturally creates tasks solvable by a tool-based agent. How can we be confident that the agent is learning a generalizable reasoning skill rather than simply overfitting to the specific patterns of the EQA-RT generation process? Have you considered evaluating its performance on tasks that are explicitly designed to challenge the limits of its predefined toolset?
3.The current work relies on a small, fixed set of tools. How do you envision the framework scaling to more open-ended real-world scenarios? How does the agent react when faced with a task where its tools are insufficient? Can the fine-tuned controller generalize to use new tools effectively without requiring SFT? Does the framework support the autonomous discovery or learning of new tool functionalities?
4.Regarding the reasoning process itself, is a higher frequency of tool calls always beneficial? Have you analyzed the potential for redundant tool calls, where the agent re-requests information it already possesses or could have inferred? Could you provide an analysis of the correlation between the number of tool invocations per task and the final success rate? This could help clarify whether the agent's reasoning is efficient or simply exhaustive.
soundness: 3
strengths: 1.The ToolEQA framework uses explicit reasoning to guide its actions, resulting in shorter navigation paths and more efficient task completion compared to previous methods.
2.The paper contributes EQA-RT, a large-scale dataset with detailed reasoning trajectories. It ingeniously integrates 3D object detection, GPT-4o for question generation, and A* for optimal path planning, all validated by a multi-level verifier, to produce high-quality training data. The automated pipeline used to generate this data is a valuable resource for the community, enabling the creation of complex, structured EQA tasks.
summary: This paper proposes ToolEQA, a framework for Embodied Question Answering (EQA) where an agent uses external tools to navigate and gather information within 3D environments. By breaking down tasks into explicit reasoning steps and generating code to call these tools, the agent achieves more efficient exploration. To train their model, the authors created EQA-RT, a large-scale dataset with automatically generated reasoning trajectories. Experiments show ToolEQA achieves strong performance, particularly in reducing the exploration path length required to answer questions.
weaknesses: 1.The core ToolEQA framework is conceptually indistinct from established tool-augmented LLM paradigms like ReAct and Toolformer. Its "thought-code-observation" loop is a direct application of this existing work, making the contribution feel more like a conceptual repackaging for the EQA domain rather than a fundamental innovation. The authors should more clearly articulate what unique, non-trivial architectural adaptations were made to the framework specifically for the challenges of embodiment, beyond simply creating a new set of tools.
2.The paper's most significant and original contribution appears to be the EQA-RT dataset and its automated generation pipeline. While this is an impressive engineering system, it does not in itself constitute a novel methodology. The work's core value lies in the resource it provides to the community, which raises questions about its fit as a methods paper, as the novelty in the learning algorithm or agent architecture is minimal.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: see above
soundness: 2
strengths: The paper focuses on an important problem of using VLM agents to solve embodied QA. Overall, the paper is well written. The main agentic contribution also is novel and seems to provide some potential benefit over baselines.
summary: This paper focuses on the embodied QA which requires a robot to navigate a home environment to find answer for the provided question. This paper focuses on an agentic setup to achieve this. This agentic setup is also the main contribution of the paper. Overall, the agent (robot) has a set of tools at it’s disposal. These tools include GoNextzPoint, ObjectLocation2D, etc. — some of these tools are used for navigation planning, while others are used for visual information extraction. Prior approaches often end up training models directly to solve the task end-to-end (often with VLMs). By contrast, the current paper aims to use an agentic setup to learn how to do the task using the provided tools. To achieve this, the paper focuses on creating a dataset with the appropriate tool calls. For this, the paper curates the appropriate questions given a scene, from these questions important objects are extracted and paths to reach close to these objects are planned using A*. Once this trajectory has been created, GPT-4O is used to annotate each step path with a reasoning/thought trace ($t_i$) and a code output $o_i$. This data is then used to FT a VLM (Qwen-7B). Experiments show that the finetuned models perform better than zero-shot models with access to the agent API.
weaknesses: Questions

4.3 details: I think the most important part of the paper is the dataset curation strategy. While there are sufficient details around how the questions and paths are generated, there is insufficient details around how the reasoning / thought traces were curated. For example, reasoning trace is also generated when the model/agent should output “Move Left” — how does the thought trace look for this kind of action? When generating the GT data do you provide all the history (with all the observations) to the model? The prompts used for GT data curation should be in the appendix. 

Also, when we use the trajectory from A* we do have the GT for some actions (e.g. moving actions such as GoNextPoint), but how do you generate GT for the model that it should use other tools (e.g. ObjectCrop), why should GPT-4o use this tool? It is quite unclear how the model is choosing the right tool. This is an extremely important detail which is completely missing from the paper. How do you ensure that these tool calls are optimal? Also, some of these tool calls e.g. ObjectCrop can conflict with going closer to the target object (and zooming into the right object) so there is some weird design choice here. 

*Ablation on training on thoughts*: Currently, the paper finetunes the model on the code output as well as the intermediate thoughts. What happens if we only finetune on the code (which is I assume the desired output) instead of the thought and maybe do some small RL to let the model curate it’s own thought. It would be interesting to see how the model performs then.

How do the tool calls look like for the zero-shot model (GPT-40 and QWEN-2.5 VL)

*Results:* There is very little performance difference (using e_path @5) metric between GPT-4o and the FT-VLM for MCQ questions. Are most of the GPT failures due to navigation actions being incorrect and the agent not exploring enough or due to some other scenarios. I am wondering if the agent has an access to a navigation tool which can plan for longer and the agent doesn’t have to output every point to go to next, how would the performance change.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. The descriptions of tools are missing. For example, it is not clear how VisualQA or SegmentInstance are implemented and used.
2. It would be useful to include the prompts used for data generation and the VLM-based controller.
3. Have the authors considered using the Exploration-Answer Consistency (EAC) metric from FineEQA? Why do authors not use a consistent set of metrics cross all benchmarks?
4. L175: Details of the planner are missing. What structure do the generated plans follow ($p$ in L203)? The planner also seems to be absent in the presented example.
5. Can authors include more details and examples for the different categories of tasks (location-location, location-special)? 
6. It would be useful to include prompts for the controller and the data generation pipeline.
7. For EQA task verification, why is it important to use both object detector and LLM? How do these compare?
8. How efficient is the data generation procedure? What fraction of trajectories get filtered out for each of the different reasons (L283-L286)?
9. L320: It is not clear how finetuning to predict the final answer ($c_t = \texttt{FinalAnswer}$) would hurt the model? Can the authors elaborate more on how this encourages the model to use its biases by ignoring the context?
10. The length of trajectories is lower than the trajectories -- an average of 12.69 (L321). Based on the distances in Table 6, does this mean the agent moves by more than 1m in each step? How does this not prevent from taking finer steps for moving closer to objects or navigating cluttered scenes?
11. Some typos and questions in writing:
- L119: ~REALTED~ RELATED
- L131: ~the lack of~ they lack
- L156: ~finetunes~ finetuned
- L183: ~executing~ executes
- L184: By invoking the executor to: Typo or incomplete
- L171: The agent should not have access to an additional scene representation $S$ beyond the observations $o_i$?
- L361: ~we~ We
soundness: 2
strengths: 1. Authors explore the relatively understudied extension of tool usage to the application of embodied question answering.
2. Usefulness of generated data: finetuning on the collected dataset improves tool usage and EQA success rates.
3. The paper's ToolEQA agent outperforms prior method (Fine-EQA) on Fine-EQA's benchmark (EXPRESSBench).
summary: The paper contributes a framework, ToolEQA that augments VLM-based EQA agents with tools (eg. bounding box detection, navigation primitives) for eliciting multi-step reasoning. The work further develops a data generation pipeline involving task creation, trajectory generation, reasoning trace generation and data verification, collecting a EQA-RT dataset. The results demonstrate that ToolEQA agents outperform prior works on the proposed EQA-RT benchmark and EXPRESS-Bench framework over most prior works, and that finetuning base VLM models on the collected dataset further improves performances.
weaknesses: 1. The paper has critical gaps in evaluation and analysis that do not fully establish the benefits of tool usage as well as improvements over prior works:

    a. The authors do not compare to a comparable baseline that follows the same pipeline but replaces the tools with VLM (eg. use the same VLM to compare sizes of objects in two frames). It is not clear if the gains are coming from breaking down reasoning into tools (multi-step reasoning) or the use of external tools. 

    b. The evaluation metric ($e_{\text{path}}$) combines recall, and hence it is not clear if ToolEQA, and its finetuning, result in efficiency improvements. Can the authors report the efficiency metric from OpenEQA?

    c. In tables 4 and 5, it is not clear if the ToolEQA used in evaluations is the finetuned version, leaving open the following questions:  Does finetuning on EQA-RT improve performance on other benchmarks? Are the performance improvements over prior works (eg.  Fine-EQA) coming from additional finetuning?
    
    d. GraphEQA, which also incorporates multi-step reasoning, appears to be better than HM-EQA on the only evaluation in Table 5. In the absence of additional comparisons and ablations, it is not clear if the components introduced in this work are essential.

    e. The method does not maintain and have access to a global scene representation (eg. scene graph, semantic map), and thus makes locally optimal decisions (eg. the doorway that looks promising without consideration of the expected scene layout). It is not clear how this choice affects the results and the comparisons to prior works (eg. Fine-EQA, GraphEQA) that maintain such representations.

    f. The work invokes a VLM in every step for low-level navigation commands ("turn left"), compared to prior works (eg. GraphEQA) that predict navigation targets offloading navigation to low-level planner. The effect of this choice on results is also unclear.

    g. While the work includes and additional test set with unseen scenes, it is not clear if the reported generalization (similar performance on seen and unseen scenes) is due to the unseen scenes having fewer target objects (Fig. 7c). It might be useful to break down the performance on the number of target objects to see if it has an effect on the performance of different splits (as hypothesized in the paper in L310).

2. Multi-step reasoning with tool usage has been explored to great extent in non-embodied settings (ViperGPT, VisProg, T3-Agent) and multi-step reasoning has been explored for embodied question answering (Fine-EQA). As a result the technical contributions of the work are limited, i.e., applying T3-Agent to embodied QA.
3. Details of critical components of the method (eg. planner, tool implementations, prompts used) are missing in the main paper and the supplementary. These are described under "Questions".

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: - How are the set of tools defined? Among the code blocks, are there any interesting action other than `print`?
soundness: 3
strengths: - The paper proposes a simple yet effective method for EQA. 
- The data generation pipeline and the dataset is very useful. There is clear performance gain from finetuning. 
- The paper conducted comprehensive experiments.
summary: The paper proposes ToolEQA, a novel method for embodied question answer (EQA). The agent itself utilizes multistep reasoning and uses tools and code execution as the means to interaction with the virtual environment. In addition to that, the paper synthetically generated a EQA dataset (EQA-RT), by 1) asking GPT-4o to generate question-answer pairs, 2) generating optimal trajectory with A* and enriching each step with reasoning and tools, and 3) verifying each task. The ToolEQA demonstrates improved performance on EQA benchmarks and in particular finetuning clearly boosts performance.
weaknesses: - There could be more ablation of the method to better understands the contribution of each component.


Decision: Reject