Embodied Instruction Following in Unknown Environments

## Abstract

Abstract Enabling embodied agents to complete complex human instructions from natural language is crucial to autonomous systems in household services. Conventional methods can only accomplish human instructions in the known environment where all interactive objects are provided to the embodied agent, and directly deploying the existing approaches for the unknown environment usually generates infeasible plans that manipulate non-existing objects. On the contrary, we propose an embodied instruction following (EIF) method for complex tasks in the unknown environment, where the agent efficiently explores the unknown environment to generate feasible plans with existing objects to accomplish abstract instructions. Specifically, we build a hierarchical embodied instruction following framework including the high-level task planner and the low-level exploration controller with multimodal large language models. We then construct a semantic representation map of the scene with dynamic region attention to demonstrate the known visual clues, where the goal of task planning and scene exploration is aligned for human instruction. For the task planner, we generate the feasible step-by-step plans for human goal accomplishment according to the task completion process and the known visual clues. For the exploration controller, the optimal navigation or object interaction policy is predicted based on the generated step-wise plans and the known visual clues.
The experimental results demonstrate that our method can achieve 45.09% success rate in 204 complex human instructions such as making breakfast and tidying rooms in large house-level scenes.

## 1 Introduction

Building intelligent autonomous systems [, , , ] to complete household tasks such as making breakfast and tidying rooms is highly demanded to reduce the laborer cost in our daily life. The agent is required to understand the visual clues of the surrounding scene and the language instructions, and feasible action plans are then generated for object interaction with the goal of high success rate and low action cost to accomplish human demands.

To achieve this, end-to-end methods [, , ] directly generate the low-level actions from raw image input and natural language with the supervision of expert trajectories. To reduce the learning difficulties in the complex task, modular methods [, , , ], sequentially learn the instruction comprehension, state perception, spatial memory construction, high-level planning and low-level control to complete human goals. Since embodied agents are expected to complete more diverse and complex instructions, large language models (LLMs) are widely employed in EIF [, , , , ] due to their strong reasoning power and high generalization ability. However, existing methods can only generate plans in known environments where categories of all interactable objects in the scene are given to LLMs. Since the agent does not know the objects in the unknown environment, the generated plans are usually infeasible because of interacting with non-existing objects.
Figure 1 (a) demonstrates an example of existing methods, where the agent is unaware that no bottles exist in the unknown environment. Interacting with the non-existent bottles based on the infeasible plan fails to accomplish the human goals of water serving.

In realistic deployment scenarios, household agents usually work in unknown environments without stored scene maps. Building scene maps in advance cannot accurately represent the scene, where object properties such as location and existence change frequently due to human activity in daily life. For example, the mug may be on the dining table and the coffee table respectively when humans are having dinner and watching TV. Meanwhile, potatoes might have been consumed and tomatoes are then purchased for the next breakfast. Therefore, failing to generate feasible plans in unknown environments strictly limits the practicality of the embodied agents. The agent working in realistic deployment scenarios is required to build real-time scene maps, where feasible plans are generated with minimal exploration cost.

In this paper, we propose an EIF method for complex tasks in the unknown environment. Different from conventional methods that assumes knowing interactable objects in advance, our method navigates the unknown environment to efficiently discover objects that are relevant to the complex human requirements. Therefore, the embodied agent can generate feasible task plans in realistic indoor scenes where the locations and existence of objects are frequently changing.
Figure 1 (b) also demonstrates the same example of water serving implemented by our method, and our agent efficiently discovers the mug and uses it as the receptacle of water because no bottles exist in the scene.
We first construct a hierarchical EIF framework including the high-level task planner and the low-level exploration controller with multi-modal LLMs, which are finetuned by the large-scale generated trajectories of the complex EIF tasks. We then design a scene-level semantic representation map to depict the visual clues in the known area, through which the goals of the task planner and the exploration controller can be aligned to feasibly complete human instructions.

More specifically, the goal of the task planner is to generate feasible plans for human instruction including navigation and manipulation in natural language. The task planner predicts the next step based on the semantic representation map and the task completion process. The exploration controller aims at discovering task-related objects with low action cost, which selects the optimal navigation policy from all navigable borders or object interaction policy according to the semantic representation map and the generated step-wise plans. For the scene-level semantic feature map, we project the CLIP features of collected RGB images during exploration to the top-down map with dynamic region attention, which preserves the task-relevant visual information in the map without redundancy.
The experimental results in ProcTHOR and AI2THOR simulation environments show that our method can achieve 45.09% success rate in 204 complex human instructions in large house-level scenes.

Figure: Figure 1: Comparison between conventional EIF methods and our approach in unknown environments. Existing methods fail to complete the instruction even with long exploration cost, while our method efficiently achieves the goal with efficient navigation and object interaction.
Refer to caption: x1.png

## 2 Related Works

Embodied Instruction Following:
The EIF task requires the robot to follow human instructions represented by natural language in the interactive environment.
Recent research can be divided into two categories: end-to-end approaches and modular methods.
For the former, agents utilize RGB-D image sequences to represent the scene and generate low-level interactive actions directly through image sequences and language embedding.
Singh *et al.* [] mitigated semantic confusion by providing global and pixel-level semantic cues for action prediction and object localization, and Nguyen *et al.* [] exploited the relationship between instructions and visual clues.
However, the end-to-end approaches require large-scale images and annotated low-level action data, resulting in low performance on unseen scenes.
The modular approaches contain high-level planning, low-level control and spatial memory construction modules with enhanced unreliability.
Min *et al.* [] significantly improved task execution success via employing explicit spatial memory and semantic search strategies for agent guidance, and Kim *et al.* [] enhanced the efficiency of the agent by focusing on task-relevant objects with the usage of contextual planning and memory frameworks.
However, existing EIF methods are designed for known environments where visual clues of the whole scene can be easily acquired by looking around, and fail to discover required visual clues in unknown environments for feasible action generation.

Scene Representation for Visual-language Navigation:
Visual-language navigation requires agents to explore unknown environments to locate target objects and follow natural language instructions. The primary challenge lies in efficiently representing expansive unknown scenes for generating navigation policies.
Existing scene representations consist of three categories: 2D semantic maps, 3D geometric maps and scene graphs. Early works [, ] constructed the 2D semantic maps by projecting visual clues in the top-down view, which are leveraged for navigation frontier selection for target finding. PONI [] proposed a scoring network for all potential frontiers of unseen regions 2D semantic maps, and L3MVN [] determined the semantic relevance of the objects around each frontier to the target by BERT [].
To embed the geometric information, 3D geometric maps are investigated by fusing the structure and semantic information.
LERF [] and ConceptFusion [] integrated fine-grained alignment of semantic features with 3D maps in SLAM, multi-view fusion, and NeRF [] for multiple downstream tasks.
To reduce the storage overhead, scene graphs [, ] are proposed to represent objects or concepts as nodes and spatial relations as edges to represent the scene topology efficiently. SayPlan [] enabled agents to focus on task-relevant nodes by integrating subgraph folding and replanning mechanisms.
Inspired by the above approaches, we construct semantic feature maps to empower embodied agents to explore unknown environments, where task-relevant information can be acquired for action generation with low exploration cost.

## 3 Problem Statement

Given the human instruction $I$ in natural language, the robot should generate a sequence of action primitives including (PickUp, Place, Open, Close, ToggleOn, ToggleOff, Slice) to complete the instruction.
The agent can only acquire the scene information for instruction following via an RGB-D camera mounted on the agent, through which the agents build a semantic map $S$ to generate the feasible interaction.
In realistic deployment, the embodied agent usually work in unknown environments, where the location and existence of objects in the house-level scene are not known. Therefore, we add an additional action primitive (Navigation) to enable the agent to explore the scene for visual information collection.

The agent consists of a high-level planner that reasons step-by-step plans $P=\{p_{i}\}_{i=1}^{T}$ from human instructions and a low-level controller that predicts the specific actions $A=\{a_{j}^{i}\}_{j=1}^{\tau_{i}}$ for each step for scene navigation or object interaction. $T$ means the number of steps to achieve the human goal, and $\tau_{i}$ is the number of special actions to achieve the $i_{th}$ step in the high-level plan. The high-level planner is represented by natural language (e.g. Step 2. Heat the potato) given the human instruction (e.g. Can you make breakfast for me?), and the low-level controller transfers the step-by-step plans into executable actions with action primitives, location and target objects (e.g. Place, potato, (10, 8) or Navigate, frontier, (2, 3)). Finally, the agent only manipulates the existing relevant objects to achieve human goals.

Figure: Figure 2: Overview of our approach. The scene feature map is constructed based on real-time RGB-D images, which is leveraged as visual clues for the high-level planner and the low-level controller. The planner generates the step-wise plans, which are leveraged to predict the specific actions in the controller. The optimal border between unknown and known regions is selected for scene exploration, and the scene feature map is updated with the visual clues seen in during the exploration.
Refer to caption: x2.png

## 4 Approach

In this section, we first introduce the overall pipeline of our EIF method designed for unknown environments, and then we describe the details of the high-level planner and the low-level controller. Moreover, we elaborate the construction of the online semantic feature maps that ground the planner and the controller to the physical scene. Finally, we demonstrate the model training and the inference of our framework in practical deployment.

### 4.1 Overall Pipeline

In realistic deployment scenarios of household robots, the physical world is usually unknown for the agent because the existence and locations frequently change due to human activity.
Therefore, the agents are required to construct the online scene feature map according to the real-time visual perception during the robot navigation, through which the agent generates feasible step-by-step plans to achieve the human goal and the efficient exploration trajectories for the unknown scene including navigation and object interaction to complete each step in the plan.
Figure 2 demonstrates the overall pipeline of our agent. The scene feature map represents the visual clues of the scene in the top-down view based on the collected RGB-D images during exploration, where the pre-trained features of regions with higher relevance to the instruction are assigned with higher importance for feature map construction.
The high-level planner generates the plans for the next step with natural language based on the task completion process and the semantic feature map, and the low-level controller predicts the templated action primitives, location and target objects for executable navigation or manipulation based on the scene feature map and the plan for the next step.

### 4.2 Hierarchical Embodied Agents for EIF in Unknown Environments

We decompose EIF in unknown environments into two sub-tasks including the high-level planning and the low-level exploration. The generated high-level plans are leveraged as guidance for the agent to select the most relevant regions for exploration, and the predicted low-level actions update the semantic feature maps to provide visual clues for feasible plan generation.
Both the planner and the explorer are implemented by a finetuned LLaVA model.

High-level planner:
The planner generates the plan for the next step in natural language, which considers the textual information including the human instruction and the completed steps and the visual clues represented by the semantic feature maps. The forward pass of the high-level planner $HP$ can be represented as follows:

$$ $\displaystyle p_{i}=HP(I,\{p_{k}\}_{k=1}^{i-1};S_{i-1})$ (1) $$

where $S_{i}$ means the semantic feature maps updated in the $i_{th}$ step and we leverage a LLaVA model whose visual encoder is the ViT-L/14 architecture for the high-level planner.

Low-level controller:
The low-level controller predicts the specific actions including the action primitives, locations, and target objects according to the generated high-level plans and the semantic feature maps, which explores the unknown scene and completes the step-wise plan. The forward pass of the low-level controller $LC$ can be represented as follows:

$$ $\displaystyle\{a_{j}^{i},l_{j}^{i},o_{j}^{i}\}=LC(p_{i},\{f_{m}^{i}\}_{m};\{s_ {m}^{i}\}_{m})$ (2) $$

where $l_{j}^{i}$ and $o_{j}^{i}$ are the predicted location and target objects for the $j_{th}$ actions in the $i_{th}$ step of the high-level plan. Meanwhile, $f_{m}^{i}$ means the textual features of the $m_{th}$ segment of the frontier between known and unknown regions for $S_{i}$, where $m$ represents the number of frontier segments in the entire $S_{i}$.
The textual features are demonstrated by the coordinate of the middle point for the frontier segment. $s_{m}^{i}$ denotes the semantic features of the $m_{th}$ frontier segments, which is demonstrated by the semantic feature map patches containing the corresponding frontiers.
The low-level controller not only explores the unknown scene with navigation and object interaction but also completes the step-wise plans by manipulating the target object (e.g. pick up the tomato).
For action primitives except for navigate, the predicted actions are implemented on the target objects. For navigate, the robot just moves to the predicted locations without object interaction.

### 4.3 Online Semantic Feature Maps

The high-level planner and the low-level controller should be aligned so that they can generate feasible plans and exploratory actions to achieve human instructions in the unknown environment. The semantic feature maps can be leveraged for alignment since they provide visual clues of the scene for both the high-level planner and the low-level controller. In realistic deployment scenarios of household robots, the existence and locations frequently change due to human activity. Therefore, we propose an online semantic feature map that is dynamically updated during the exploration of the unknown scene for each human instruction.

Semantic feature maps mean the visual representation for each pixel in the top-down view of the entire house. Compared with conventional semantic maps, our semantic feature maps can represent implicit relationships between objects in the scene instead of only categories, which provides crucial information for effective exploration policy generation. For EIF in unknown environments, the visual information collected in the $i_{th}$ timestep contains the RGB image $C_{i}$ and the depth image $D_{i}$.
To enable the semantic feature maps to acquire high generalization ability in diverse human instructions, we leverage CLIP to extract the pixel-wise visual features $\mathbf{f}_{xy}^{i}$ at time $i$ for the pixel in $x_{th}$ row and $y_{th}$ column of $C_{i}$ via following ConceptFusion [], and fuse the feature of the entire image and that of the instance mask for the pixel as the pixel-wise visual features $\mathbf{f}_{xy}^{i}$ in the same way.
The visual features contribute to the projected location in the scene feature map in the top-down view, which can be depicted as follows:

$$ $\displaystyle\mathbf{F}_{uv}^{i}=\sum_{x,y}\mathbf{f}_{xy}^{i}\cdot\mathbb{I}( \mathcal{P}_{D_{i}}(x,y)\in\mathcal{S}(u,v))$ (3) $$

where $\mathbf{F}_{uv}^{i}$ means the contribution to the element in the $u_{th}$ row and $v_{th}$ column of the semantic feature map from the visual information collected in time $i$, and $\mathcal{P}_{D_{i}}(x,y)$ demonstrates the projected coordinates in the top-down view for of the pixel $(x,y)$ based on the depth image $D_{i}$. $\mathcal{S}(u,v)$ means the pixel in the $u_{th}$ row and $v_{th}$ column in the semantic feature map, and The indicator function $\mathbb{I}(\cdot)$ equals one for true and zero otherwise.

The semantic feature map is updated at each time step during the exploration process, where the agent sees new visual information for recording. Since the house for embodied instruction following in realistic world is usually very large, regarding all images with equal importance in semantic feature map construction leads to significant information redundancy. Meanwhile, different visual clues usually make various contribution to the given human instruction. Therefore, we should assign large importance to relevant visual clues when updating the semantic feature maps, so that sufficient visual information can be represented without redundancy for high-level planning and low-level exploration. The task relevance can be acquired as follows. The high-level planner is also required to generate the demanded objects $\{O_{k}\}_{k}$ for the predicted corresponding step-wise plan, which are leveraged to construct three prompts including (a) the image contains $\{O_{k}\}_{k}$, (b) the image does not contain $\{O_{k}\}_{k}$ and (c) the image contains nothing. We then leverage a pre-trained LongCLIP [] to predict the similarity score between the image and all prompts. Finally, the online semantic feature map is updated with dynamic region attention:

$$ $\displaystyle\mathbf{S}_{uv}^{i}=(1-w_{i})\mathbf{S}_{uv}^{i-1}+w_{i}\mathbf{F }_{uv}^{i}$ (4) $$

where $\mathbf{S}_{uv}^{i}$ means the features in the $i_{th}$ row and $j_{th}$ column of the semantic feature maps at time $i$.
The normalized weight $w_{i}=\hat{w_{i}}/\frac{1}{i}\sum_{k=1}^{i}\hat{w}_{k}$ represents the importance of the current semantic features compared with known visual clues, where $w_{k}$ is the original similarity score between the image and the prompt in the $k_{th}$ time step. The online semantic feature maps contain rich visual information, and the most relevant regions can be explored via navigating the optimal border and interacting with related objects to achieve human goals with minimized action cost.

### 4.4 Training and Inference

Training: The training samples for the high-level planner consist of human instruction, current completed plans, current semantic feature maps and the groundtruth plan for the next step, and those for the low-level controller include plan for the next step, textual and semantic features for current border segments and the groundtruth action sequences representing primitives, location and targets. The details of input and output are provided in Appendix E.
We leverage GPT-4 and the ProcTHOR simulator to generate the large-scale dataset to train the LLaVA-based high-level planner. We annotate several seed instructions and leverage GPT-4 to generate more instructions and corresponding plans based on the object list for each scene in the ProcTHOR, where samples with logical errors are filtered with PDDL parameters [].
We then implement the generated plans in ProcTHOR and collect the navigation trajectories, RGB-D images, object locations and robot poses as the training data. Finally, the generated samples are parsed into high-level planning samples and low-level action data.
We follow the supervised fine-tuning paradigm in LLM for training the LLaVA model in high-level planner and low-level controller, where we mask out $p_{i}$ and $\{a_{j}^{i}\}_{j=1}^{\tau_{i}}$ in the $i_{th}$ step. The CE loss leveraged in the training process is represented by:

$$ $\displaystyle\mathcal{L}_{\text{}}=-\mathbb{E}_{(\boldsymbol{X}_{T}, \boldsymbol{R})\sim\mathcal{D}}\Big{[}\sum_{m=1}^{M}\log p_{\boldsymbol{\theta }}(R_{m}|\boldsymbol{R}_{<m},\boldsymbol{X}_{V},\boldsymbol{X}_{T})\Big{]}$ (5) $$

where $\boldsymbol{X}_{V}$ denotes scene feature maps and $\boldsymbol{X}_{T}$ means input text prompt tokens. $\boldsymbol{R}_{<m}$ represents the output text tokens before the $m_{th}$ token $R_{m}$ and $M$ arenumber of output tokens.
In this way, the pre-trained multimodal LLMs can be grounded to high-level planning and low-level control tasks in realistic scenes, where executable plans and actions are generated based on the scene representation.

Inference: The high-level planner generates the planning for the next step based on the current RGB-D image and scene information represented by the semantic map, and the low-level controller predicts the action primitives, target object and interaction position based on the generated step-wise plan. The semantic feature maps are updated when implementing the low-level action sequences. The high-level planner will generate the plans for the next step only when the current low-level action is successfully achieved. The detailed process is illustrated in Appendix B.

**Table 1: Comparison with different EIF methods across different instructions in the ProcTHOR simulator, where LLM-P^∗ represents the LLM-P without performing re-planning.**
| Method | Small-scale | Large-scale |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SR | PLWSR | GC | PLWGC | Path | SR | PLWSR | GC | PLWGC | Path |  |
| Target-specific Short |  |  |  |  |  |  |  |  |  |  |
| LLM-P^∗ | 27.86 | 23.49 | 41.50 | 35.35 | 25.27 | 17.16 | 11.70 | 33.25 | 22.87 | 65.75 |
| LLM-P | 28.36 | 23.62 | 42.33 | 35.57 | 27.47 | 18.63 | 12.64 | 35.21 | 24.63 | 63.47 |
| FILM | 5.97 | 5.97 | 11.17 | 11.17 | 16.55 | 0.49 | 0.49 | 4.84 | 4.84 | 33.68 |
| Ours | 45.77 | 40.75 | 57.88 | 51.14 | 23.29 | 45.09 | 34.41 | 58.21 | 43.13 | 59.11 |
| Target-specific Long |  |  |  |  |  |  |  |  |  |  |
| LLM-P^∗ | 5.97 | 5.14 | 18.91 | 17.26 | 60.56 | 1.52 | 0.82 | 15.28 | 13.05 | 78.03 |
| LLM-P | 5.97 | 4.80 | 19.65 | 17.30 | 64.89 | 1.52 | 1.01 | 16.04 | 14.17 | 64.14 |
| FILM | 0.00 | 0.00 | 4.14 | 4.14 | 79.17 | 0.00 | 0.00 | 6.26 | 6.26 | 70.14 |
| Ours | 13.43 | 12.44 | 27.11 | 24.67 | 62.21 | 19.70 | 17.34 | 35.61 | 31.08 | 78.99 |
| Abstract |  |  |  |  |  |  |  |  |  |  |
| LLM-P^∗ | 1.32 | 0.92 | 15.68 | 12.57 | 38.69 | 6.16 | 2.83 | 16.92 | 11.21 | 70.92 |
| LLM-P | 3.95 | 2.33 | 16.78 | 12.45 | 36.27 | 6.16 | 3.58 | 18.15 | 12.42 | 67.20 |
| FILM | 0.00 | 0.00 | 4.87 | 4.87 | 33.23 | 0.00 | 0.00 | 8.02 | 8.02 | 49.45 |
| Ours | 10.53 | 8.09 | 24.23 | 19.68 | 35.90 | 9.59 | 5.74 | 21.30 | 15.01 | 61.54 |

**Table 2: Comparison with different methods on the ALFRED benchmark with the AI2THOR simulator. We conduct the experiments with different numbers of samples for model adaptation.**
| Method | Shot Num. | Test |  | Validation |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Seen |  | Unseen |  | Seen |  | Unseen |  |  |  |  |  |  |
| SR | GC |  | SR | GC |  | SR | GC |  | SR | GC |  |  |
| HLSM | 100 | 0.82 | 6.88 |  | 0.61 | 3.72 |  | 0.13 | 2.82 |  | 0.00 | 1.86 |
| FILM | 100 | 0.20 | 6.71 |  | 0.00 | 4.23 |  | 0.00 | 13.19 |  | 0.00 | 9.65 |
| LLM-P^∗ | 100 | 13.05 | 20.58 |  | 11.58 | 18.47 |  | 11.82 | 23.54 |  | 11.10 | 22.44 |
| LLM-P | 100 | 15.33 | 24.57 |  | 13.41 | 22.89 |  | 13.53 | 28.28 |  | 12.92 | 25.35 |
| Ours | 0 | 16.11 | 28.51 |  | 14.32 | 27.83 |  | 13.17 | 27.54 |  | 13.40 | 31.39 |

## 5 Experiments

### 5.1 Implementation Details

Training configurations:
We employed the LLaVA-7B architecture with the Vincua-1.3-7B pre-training weights for the high-level planner and the low-level controller, which is finetuned with our generated data by the LoRA strategy. For the visual encoder, we sampled 32 visual embeddings from each frontier in the semantic feature maps up to 256 tokens as scene information representation.
We generated 2k instructions with three subparts (1386 target-specific short, 333 target-specific long and 332 abstract instructions) for 2509 scenes in ProcTHOR, which results in 30k groundtruth plans for training the high-level planner.
We implemented the plans in ProcTHOR with $A^{*}$ algorithm to collect the expert trajectory as the groundtruth for training low-level controller.
Target-specific short and long instructions mean those containing objects to be interacted (e.g. Place the egg in the bowl) for task achievements, whose number of step plan is respectively lower than 15 and not.
Abstract instructions do not contain the interacted objects in the instructions (e.g. Make a simple lunch for me).
We also generate 201, 67 and 152 data for each subpart as the test set.
We utilized 8 NVIDIA 3090 GPUs to finetune the high-level planner and the low-level controller for an hour in the training stage. More details are provided in Appendix A.

Figure: Figure 3: Example visualization of dynamic region attention weights.
Refer to caption: x3.png

Metrics:
Following the ALFRED benchmark [], we use success rates (SR), goal condition success (GC), path length and their path-length-weighted (PLW) counterparts for evaluation.
SR means the ratio of the cases where the agent completely achieve the human instructions, and GC measures the ratio of objects in the state of goal achievements.
PLWSR and PLWGC calculate SR and GC weighted by the expert trajectory planning step number divided by the actual execution step number, which measures the trade-off between performance and efficiency.

Simulated environments:
We perform extensive experiments in the ProcTHOR and AI2THOR simulators, where the step size of translation and rotation for the agent is $0.25$m and $90^{\circ}$ respectively. ProcTHOR contains 10k house-level scenes with objects from 93 categories, where the agent receives $600\times 600$ RGB-D images in the egocentric view. We divide the scenes into small-scale ([0, 10]) and large-scale ([10, 16]) ones based on the side length of the room. AI2THOR has 120 room-level scenes with 105 categories of objects, and we follow the ALFRED setting for a fair comparison with other methods [, , ]. The agent receives $300\times 300$ RGB egocentric views and employs models provided by FILM for depth estimation.

### 5.2 Comparison with Baselines

Table 1 demonstrates the results on ProcTHOR for LLM-Planner [], FILM [] and our method, where our approach significantly outperforms the state-of-the-art-method LLM-Planner.
Although LLM-Planner utilizes the rich commonsense embedded in LLMs to generate plans for the agent, it fails to align the pre-trained LLM with the scene information. The generated plans are usually infeasible due to the non-existence of the objects for interaction, and the re-planning module suffers from low success rate and low efficiency. On the contrary, our method construct the semantic feature maps which grounds the pre-trained multimodal LLMs to the realistic physical scene, and the unknown environment can be efficiently explored by understanding the visual clues for executable plan generation.
Our approach remains leading in performance in more challenging abstract task settings and loses less than 1% of performance on large-scale scenes compared to small ones. The semantic feature map allows our approach to efficiently find the suitable target to satisfy the instruction during active exploration.
Meanwhile, the leading PLWSR and PLWGC metrics verify that our low-level controller can find the target object at a lower navigation cost.
Moreover, the success rate of conventional methods in the large-scale scenes is near zero, while our approach can achieve 9.59% success rate. Since the service robot is usually deployed in house-level scenes, our method is proven to be more practical.

Figure: Figure 4: All failure cases on ProcTHOR simulator.
Refer to caption: x4.png

In order to evaluate the generalization ability of our method, we compare our method with the state-of-the-art EIF approaches on the ALFRED benchmark with few or no samples for finetuning as illustrated in Table 2, where our method is evaluated in zero-shot settings without finetuning the pre-trained agent. Despite leveraging the commonsense in the pre-trained multimodal LLMs, our method provides rich visual information of the scene to ground the multimodal LLM for feasible plan generation.
As a result, we achieve higher success rate with shorter path length than LLM-Planner, even though we do not leverage any samples on ALFRED for finetuning. The performance improvement is less than that on ProcTHOR because ALFRED only contains room-level scenes, and simply looking around can collect information of most objects without the requirement of semantic map construction and scene exploration.

We demonstrate the quantitative results in Figure 5, where we show the step-wise plan, the exploration process and the robot implementation during a whole sequence for EIF.
In the beginning, the agent is initialized in the bedroom area and selects the navigation borders outside the room for exploration, as the instruction ’making breakfast’ is irrelevant to bedrooms.
During the navigation, the agent gradually knows to explore the kitchen area by observing the dining table and the counter, and it is even aware that opening the fridge may find food for breakfast due to the rich commonsense in our finetuned low-level controller.
As a result, abstract instruction is achieved by serving diverse food for breakfast, where only related regions are navigated with high exploration efficiency in the unknown environment. Figure 4 illustrates the statistics of failure cases caused by different reasons. The failure mostly comes from unsuccessful navigation because of the large house-level scene, and the top reasons including ’too close to targets’ and ’fail to see closed space’ indicate that navigation algorithms should be designed with high compatibility of the subsequent manipulation.

Figure: Figure 5: An example of EIF in unknown environments. The agent only navigates the task-related regions for visual clue collection with high efficiency, and generates feasible plans to complete the abstract instructions.
Refer to caption: x5.png

### 5.3 Ablation Studies

Effectiveness of the high-level planner and the low-level controller: We evaluated the variants of our method where the planner and the controller are respectively replaced with the groundtruth step-wise plans and groundtruth action sequences. Table LABEL:table:ab_gt demonstrates the results where the performance of our methods is close to that of the groundtruth, which indicates the effectiveness of our LLaVA-based planner and controller. Moreover, the performance of planners mainly influences the success rate because it is important for the agent to interact with the right objects in the correct orders, while the controller significantly impacts the path length since directly exploring the related regions enables the agent to accomplish the instruction faster.

Effectiveness of the online semantic feature map: The semantic feature map provides visual information of explored regions for the planner and the controller to generate feasible plans and efficient actions, and we report the performance of different semantic maps to validate the effectiveness of our method.
Table LABEL:table:ab_map demonstrates the results for the settings of no semantic maps, semantic maps with only category information, semantic feature maps without dynamic attention and our semantic feature maps. The results demonstrate that the implicit rich semantic features are necessary for effective exploration of unknown environments, and the dynamic attention also enhances the performance of the semantic feature map as it removes the information redundancy for the large house-level scenes.
We also visualize the dynamic region attention when the agent builds the semantic feature map in the unknown environment as illustrated in Figure 3. For the instruction Slice the tomato for salad, the features of the kitchen area especially the tomato and the sink are considered with high attention(The green color represents greater weight), which indicates that the dynamic region attention learns relevant visual clues for feasible action generation.

## 6 Conclusion

In this paper, we have proposed an EIF approach for unknown environments, where the agent is required to explore the environment efficiently to generate feasible action plans with existing objects to achieve human instructions. We first build a hierarchical EIF framework including a high-level planner and a low-level controller, and then build a semantic feature map with dynamic region attention to provide visual information for the planner and the controller. Extensive experiments demonstrate the effectiveness and efficiency of our framework in the house-level unknown environment.

Limitations: This work lacks real manipulation implementation and the designed navigation policy ignores the compatibility with manipulation. We will design mobile manipulation strategies for general tasks and implement the closed-loop system on real robots in the future.

## Appendix A Train Details

High-level planning and low-level controller:
In the supervised instruction fine-tuning stage, we reduce memory usage via DeepSpeed ZeRO-2. The learning rate for the feature mapping layer and the LLM backbone network is set to $2\times 10^{-}5$, and the batch size is set to 8. Fine-tuning is performed for only one epoch.
Since the semantic feature maps have been constructed through CLIP, the scene visual tokens are directly fed into the mapping layer without the visual coder during the training stage.

Visual perception:
We selected 100k images from the captured expert trajectories as the training set for instance segmentation model Detic[], fine-tuning the pre-trained model with the learning rate of $1\times 10^{-4}$ and performing 180k iterations.
The batch size is set to 16 and the Adam optimiser is applied.

## Appendix B Inference Details

Overview:
At the $i_{th}$ time, the agent surrounds to perceive the scene information and constructs semantic feature map $S_{i}$ and frontiers mask.
Then, the agent will generate the $i_{th}$ high-level planning $p_{i}$ based on the scene information, user instruction $I$ and finished step-by-step planning $\{p_{k}\}_{k=1}^{i-1}$.
The low-level controller generates specific interaction actions $a_{j}^{i}$, target objects $o_{j}^{i}$ and positions $l_{j}^{i}$ based on $p_{i}$, semantic feature $\{s_{m}^{i}\}$ and textual feature $f_{m}^{i}$: 1) If the $o_{j}^{i}$ is observed by the agent, $l_{j}^{i}$ will be the location recorded on the map; 2) If not be observed, $l_{j}^{i}$ will be the frontier position.
During the inference process, each instruction $I$ performs up to 30 steps of high-level planning.

Figure: Algorithm 1 Inference Process

Frontier representation:
We follow [] to generate frontier masks that distinguish between known and unknown regions based on the occupancy map.
Through connected component analysis, we obtain the mask of each frontier instance.
We further remove frontiers with areas smaller than the threshold (150 pixels) to reduce redundancy exploration.
We sample 32 visual embeddings as frontier tokens according to the frontier instance mask on the corresponding region of the feature map, while utilizing the coordinates of their centroids for the frontier text description. The specific representation is illustrated in Figure 6 (a).

Region Attention:
Since the importance of the observed image for the completed instruction is different for each frame, it is desirable to assign higher weights to task-relevant visual embeddings to enable $LC$ to generate more efficient navigation exploration planning.
$HP$ generates target objects that might be required to interact to complete instruction $I$ while generating $p_{i}$ and converts them into a sentence describing $L_{dec}$.
However, measuring the relevance of an image to the instruction with only a single description is not discriminative enough to highlight task-relevant regions on the feature map.
To this end, we add additional variants of descriptions to calibrate the relevance of images to their corresponding descriptions for exploiting the prior knowledge of the pre-trained models extensively.
Specifically, we further expand $L_{dec}$ into $\overline{L_{dec}}$ and $L_{none}$ to match the input requirements of image and text alignment models such as CLIP. $\overline{L_{dec}}$ and $L_{none}$ describe the image as not containing the target objects and not containing the objects, respectively.
We adopt LongCLIP [] to retrieve the similarity between $\smash{\left\{L_{dec},\overline{L_{dec}},L_{none}\right\}}$ and RGB images as illustrated in Figure 6 (b), and consider the score of $L_{dec}$ as the attention weight.

Figure: Figure 6: Details of frontier representation and region attention weights.
Refer to caption: x6.png

## Appendix C More Result

Influence w.r.t. navigation frontier construction:
Navigation frontier means the border between the known and unknown regions, which are represented by multiple segments.

We only select the frontiers that are longer than a threshold as the candidates for agent navigation, because extremely short frontiers usually indicate corner regions that reveals uninformative information. Therefore, we can enhance the exploration efficiency significantly. Table D illustrates the success rate and path length for different thresholds. The results demonstrate that low thresholds result is redundant navigation with high path length, while high thresholds degrade the success rate because of important scene information. We set the frontier threshold to 150 pixels to achieve higher performance and navigation cost trade-off.

Qualitative results:
We demonstrate more unknown environment EIF execution sequences to reflect the superiority of our approach.

Figure: Figure 7: Our approach active search for lettuce in the fridge to complete the instruction.
Refer to caption: x7.png

Figure: Figure 8: Our approach consistently completes complex instructions for preparing breakfast in large-scale unknown scenes.
Refer to caption: x8.png

## Appendix D Data

Figure: Table 5: Results regarding frontier thresholds.

Training Data:
Existing EIF datasets are still limited in instruction diversity and scene scale.
We design a dataset synthesis framework to minimize the generation cost and increase the scale of EIF datasets, enabling agents to adapt to large-scale unknown scenes and complex tasks.
Therefore, the dataset synthesis framework consists of two main stages.
The first stage is to employ GPT-4 to generate extensive high-level planning with corresponding low-level actions based on prompt and scene information, then filter logical error samples with TextWorld [].
The second stage is to execute the interactions specifically with the oracle in the simulator, grounding the generated plans and actions into the physical scene and collecting expert trajectories.

TextWorld data generation:
We collect object lists contained in each scene as scene information in ProcTHOR, consisting of the location and size of each object.
GPT-4 will generate task plans based on the object information and prompts. Specifically, we annotated 22 seed tasks manually to inspire GPT-4 to generate confirmed responses.
Each response contains instructions, step-by-step high-level actions, and corresponding low-level actions.
We further employ self-instruction [] to ensure the diversity of instructions(The similarity filtering threshold is set to 0.9).
Meanwhile, GPT-4 will generate PDDL parameters that satisfy the ALFRED benchmarks to verify the feasibility of the planning.
The generated candidate samples are sent to TextWorld and check whether the task can be executed based on the PDDL parameters to ensure the quality of the training dataset.

Grounding the generated plans:
The synthetic dataset that passes the PDDL check is fed into the ProcTHOR simulator for specific interactions.
We collect navigation trajectories in ProcTHOR based on the planning generated in the first stage under oracle settings, which contain RGB images, depth maps, segmentation masks, and robot poses.
According to the semantic feature map building approach presented in Section 4.3,
we obtain real-time semantic feature maps $S_{i}$, frontier text features $\{f_{m}^{i}\}_{m}$, and semantic embedding $\{s_{m}^{i}\}_{m}$ sequences as the agents perform interactions.
Based on the step-by-step planning generated by GPT-4, we split the above sequences into step-by-step instruction-following samples.
As for $HP$, we feed instruction $I$, scene information $S_{i}$ and completed steps $\{p_{k}\}_{k}^{i-1}$ as prompts, expecting to generate the next step $p_{i+1}$ to be done.
Meanwhile, in order to ensure the consistency of the high-level task planning, we require $HP$ to give the planning of all subsequent actions as illustrated in Figure 9 (a).
For $LC$, we take the current $p_{i}$, frontier text features $\{f_{m}^{i}\}_{m}$ and semantic features $\{s_{m}^{i}\}_{m}$ as prompt, and expect $LC$ to output the action primitives executed by the agent under the oracle setting.
Specifically, if the agent observes the target object, LC generates the specific target and action primitive based on the input $p_{i}$. If it does not observe, we require $LC$ to generate the closest frontier to the oracle path as the next exploration region.
$LC$ training samples are as illustrated in Figure 9 (b).

Figure: Figure 9: Visualization of training samples for high-level planner and low-level controller.
Refer to caption: x9.png

## Appendix E Prompt

Figure 10 briefly demonstrates the prompt words employed to inspire GPT-4 in generating the EIF dataset, which consists of the following four main parts:

System Prompt:
Primarily designed to set up the GPT-4 contextual environment for generating task planning based on a virtual robot.

Action Primitive:
It is applied to constrain the scope of the interaction action generated by the GPT to ensure that it is executable.

Response Format:
Ensure consistency in response format to parse specific instructions, planning and actions.

PDDL Params:
Record the requirements for the completion of the instruction, including the target object and its state. An interaction is only successful if the state of the target object in the scene matches the PDDL parameter record.

Figure: Figure 10: Prompt words for GPT-4 synthetic EIF dataset.
Refer to caption: x10.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 2
questions: See weaknesses above.
soundness: 2
strengths: - The paper is generally written well and easy to follow.
- The paper aims to address a challenging problem of EIF in environments that may change during the agent's task completion.
- The proposed approach achieves strong performance over the baselines.
summary: The paper proposes a hierarchical approach for embodied instruction following in unknown environments where room configurations may change due to human activities. The proposed approach consists of a high-level planner, a semantic feature map, and a low-level controller. Given an egocentric RGBD image, the agent updates its semantic feature map by weight-summing the newly obtained pixel-wise pretrained LongCLIP features and the old ones. The high-level planner receives as input a human instruction and the updated semantic feature map to generate high-level plans. For each high-level plan, the low-level controller predicts the target next location based on the semantic feature, text feature, and the high-level plan. The paper builds training and evaluation datasets based on ProcTHOR with GPT-4 and shows that the proposed approach achieves strong performance over the baselines.
weaknesses: - Unknown environments are not well defined. An environment is comprised of many elements, such as layouts, textures, object classes and instances, *etc.* It is not clear what is unseen to the agent. In addition, existing EIF benchmarks, like ALFRED, also evaluate agents in "unseen" setups. What is the difference between the unseen environments used here and the existing EIF benchmarks?

- Unlike prior EIF literature, the paper uses depth as additional input, but such depth information is unreliable in many cases. It is unclear if the proposed approach can be applied to RGB-only scenarios like prior EIF benchmarks and noisy-depth scenarios.

- The proposed approach maintains a semantic "feature" map as scene representation. Why not directly use the projected depth map and semantics (*e.g.*, object classes) associated with the corresponding pixels? The features in the map are updated in a weight-sum manner, but we can simply replace the part of the semantic map of what the agent is currently seeing with the old one.

- The low-level policy is learned to predict the target navigational location in a supervised manner. While simulated environments can provide such labels, obtaining them in real-world scenarios may not be available. Can this proposed approach applied to these scenarios as well?

- In Table 3, Ours vs. Random Attention and No Attention show marginal drops. I'm not sure if the proposed component is indeed effective.

- It looks like the proposed task setup closely resembles the ALFRED benchmark. Incorporating the proposed approach to more recent state-of-the-art methods here may imply the generalizability of the proposed setup.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 2
questions: see weaknesses.
soundness: 3
strengths: 1. The motivation of the paper is very good, taking into account situations in real environments where items may not be present at the time of planning. 
2. The proposed framework is effective and also well-motivated. Combinations between high-level planners and low-level controllers are common, but the feature maps used in this work are interesting.
3. The paper is written in a fluid and well-organized manner.
summary: This paper introduces a hierarchical embodied instruction following (EIF) framework for autonomous systems to complete complex tasks in unknown environments. It addresses the limitation of conventional methods that fail to generate feasible plans when deployed in environments with unknown objects. The proposed solution includes a high-level task planner and a low-level exploration controller, informed by multimodal large language models (LLMs). A semantic representation map with dynamic region attention aligns task planning and scene exploration with human instructions. The framework efficiently explores unknown environments to discover relevant objects and generate feasible task plans. The authors report a 45.09% success rate in executing 204 complex instructions in large house-level scenes, demonstrating the effectiveness of their approach over existing methods.
weaknesses: 1. In the main experiment the authors only compared LLM-P and FILM, the lack of other baselines weakens the superiority of the proposed approach. Here are three different types of baselines:

   (1) I would suggest adding some Object Navigation Method with LLM as Planner, such as MOPA[1], LGX[2].  

   (2) Another very similar work, Demand-driven Navigation [3] also uses human instructions as input, has similar tasks, e.g., "I am thirsty" with "I need to drink water" and similar task motivation. I would suggest for baseline comparison or discussion at related work. 

   (3) Multimodal LLM like GPT-4o, Gemini, LLaMA-3.2, Qwen2-VL can be directly used as baselines.

2. Need more details on task setting: 

   (1) What is the step limit for each task? 

   (2) How to determine task success and failure? If the agent completes the task with an undocumented solution (i.e., it was successful in the human viewpoint, but was not judged successful in simulator), how should the result be determined, or is there a high probability of this occurring?

   (3) If the task is partially successful, e.g. I want drink a cup of water, find the cup, but not the water, how should it be determined? Or conversely, would it make a difference if the water source is found but not the cup? 

3. How much time and computational resources are needed for inference per task execution and how does it compare to baselines? Since the method uses a multimodal LLM like LLAVA, the speed of inference and computational resources are very important if it is to run in a real environment.

4. I believe the superiority of the methodology could have been enhanced if the authors could supplement experiments in real environments.

5. Need more details on the action space:

   (1) Does the agent need to have a knife in its hand to perform the Slice action? What happens if the agent does not have a knife in its hand but outputs a Slice? For other action, is there any precondition?

   (2) Can an agent only PickUp one item in hand at a time? Will it fail if it first PickUp item A and then item B?

   (3) How are Navigation low-level actions generated?

[1] MOPA: Modular Object Navigation with PointGoal Agents

[2] Can an Embodied Agent Find Your “Cat-shaped Mug”? LLM-Based Zero-Shot Object Navigation

[3] Find what you want: learning demand-conditioned object attribute space for demand-driven navigation

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: When will the code be open-sourced? Can the 7B version of LLaVA really handle such complex and diverse tasks that require precise outputs? Additionally, as far as I know, the 7B model of LLaVA has limitations in understanding depth maps. I’m surprised by the performance described in your paper!
soundness: 1
strengths: The use of scene feature maps to facilitate exploration in unknown environments, serving as visual input for VLMs while integrating textual instructions, presents an intriguing, well-founded, and promising approach. This framework combines planning and action seamlessly through the high-level task planner and low-level exploration controller, resulting in a cohesive and efficient system for embodied instruction following.
summary: This paper proposes a novel embodied instruction following (EIF) method focused on scenarios that require executing complex tasks in unknown environments. The approach leverages a hierarchical EIF framework with a high-level task planner and a low-level exploration controller based on multimodal large language models. Utilizing semantic feature maps and dynamic region attention mechanisms, the method aligns task planning with the exploration process to fulfill complex human instructions. In extensive experiments within a large simulated environment, the approach demonstrated significantly improved success rates for complex tasks, such as preparing breakfast and organizing rooms, showing clear advancements over existing methods.
weaknesses: Experimental Environment Limitations: This method has been validated on a single simulator. It would strengthen the evaluation to include additional simulators, such as Habitat or iGibson, or to test the method on a real-world robotic platform to further demonstrate its robustness.
Limited Scalability and Computational Efficiency: The approach currently shows limited scalability and relatively low computational efficiency, which could impact its practical deployment in larger environments.
Lack of Comparative Models: The paper does not compare against certain state-of-the-art general models. For example, it would be interesting to see how a zero-shot baseline like GPT-4o performs across the various evaluation metrics.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - In the paper, the train and test data share the same data distribution. Why do the authors use the word unknown environments?
- How the performance will be if the LLaVA model is trained on limited amounts of data (i.e., few-shot or zero-shot)?
soundness: 1
strengths: Strengths
- The paper presents a new modular approach for embodied instruction following tasks, consisting of a high-level planner, low-level controller, and online semantic feature map. 
- The paper conducts diverse analysis, including ablation studies and qualitative analysis, which helps to understand the proposed method.
summary: This paper introduces a modular approach for embodied instruction following tasks, which consist of three components: 1) high-level planning, 2) low-level control, and 3) building semantic feature maps. The proposed approach employs the LLaVA models and fine-tunes them on data generated by GPT-4 and the ProcTHOR simulator. Experiments show the comparison with baselines, ablation studies, and qualitative analysis to verify the effectiveness of the proposed approach.
weaknesses: Weaknesses
- I find no clear distinction between the “unknown environments” mentioned in this paper and existing test environments. Existing studies have also validated their proposed approach in unseen environments where embodied agents do not observe during training. Furthermore, prior work like LLM-Planner leverages the off-the-shelf object detector to feed the detected objects into LLMs, rather than providing all object categories with LLMs. How unknown environments are qualitatively or quantitatively different from unseen environments? Can authors verify the difference?

- In my current understanding, the paper conducts unfair comparisons. The proposed approach based on LLaVA is fine-tuned on 2k instructions generated by GPT-4, but the baseline model (LLM-Planner [1]) is a few-shot learning method that just uses 9 samples for in-context learning. The authors should compare their proposed method with more powerful models like EPO [2] which is trained on the in-domain data.  


References

[1] Llm-planner: Few-shot grounded planning for embodied agents with large language models. Song et al. In Proceedings of ICCV, 2023.

[2] EPO: Hierarchical LLM Agents with Environment Preference Optimization. Zhao et al. In Proceedings of EMNLP, 2024.

Meta-review: The paper received ratings that were all below the acceptance thresholds (5,5,3,3). The reviewers raised several concerns such as lack of a proper definition for unknown environments, lack of certain baselines, high cost of inference preventing real-world deployment, and unfair comparisons. The authors provided responses to the reviewers. While the rebuttal addressed some of the concerns, the reviewers still had concerns (details below). Therefore, the AC follows the recommendation of the reviewers and recommends rejection.

Decision: Reject