# FOM-Nav: Frontier-Object Maps for Object Goal Navigation

Thomas Chabal, Shizhe Chen, Jean Ponce, Cordelia Schmid

<figure><img src="image_1.png" /></figure>

To cite this version:

Thomas Chabal, Shizhe Chen, Jean Ponce, Cordelia Schmid. FOM-Nav: Frontier-Object Maps for Object Goal Navigation. 2025. hal-05392088

HAL Id: hal-05392088

https://hal.science/hal-05392088v1

Preprint submitted on 2 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# FOM-Nav: Frontier-Object Maps for Object Goal Navigation

Thomas Chabal†‡

Shizhe Chen†‡

Jean Ponce‡*

Cordelia Schmid†‡‡

**Abstract**— This paper addresses the Object Goal Navigation problem, where a robot must efficiently find a target object in an unknown environment. Existing implicit memory-based methods struggle with long-term memory retention and planning, while explicit map-based approaches lack rich semantic information. To address these challenges, we propose FOM-Nav, a modular framework that enhances exploration efficiency through Frontier-Object Maps and vision-language models. Our Frontier-Object Maps are built online and jointly encode spatial frontiers and fine-grained object information. Using this representation, a vision-language model performs multimodal scene understanding and high-level goal prediction, which is executed by a low-level planner for efficient trajectory generation. To train FOM-Nav, we automatically construct large-scale navigation datasets from real-world scanned environments. Extensive experiments validate the effectiveness of our model design and constructed dataset. FOM-Nav achieves state-of-the-art performance on the MP3D and HM3D benchmarks, particularly in navigation efficiency metric SPL, and yields promising results on a real robot.

## I. INTRODUCTION

Autonomous navigation has been a long-standing challenge in robotics [1], dating back to the pioneering work on the robot Shakey [2] in the 1960s. While early work focused on navigating to specific points [3], [4] with a preconstructed map [5], [6], recent research has progressively shifted towards navigation in unknown environments using textual [7], [8] or visual [9] goals, which is an essential capability for enabling mobile manipulation systems [10], [11] to perform diverse real-world tasks.

In this work, we focus on the object goal navigation task (ObjectNav) [8], where an agent must navigate to a target object category in an unknown environment using RGB-D observations. This task requires long-horizon multimodal scene understanding and efficient exploration. The robot should not only recognize objects within its current field of view but also use previous observations to develop more accurate scene understanding. Furthermore, effective object goal exploration demands maintaining long-term memory of visited areas and learning implicit environment priors, e.g., understanding that pillows are more likely to be found in a bedroom or a living room than in the kitchen.

Prior ObjectNav methods fall into two categories based on their memory representations. The first approach employs implicit models, typically encoded within hidden states of recurrent neural networks [12]–[14] or transformers [15]–[18]. While effective for multimodal understanding [16],

<figure><img src="image_3.png" /><figcaption>Fig. 1: The proposed frontier-object map is a rich representation of objects and frontiers (boundaries of the explored scene), displayed here as colored point clouds and red lines. It encodes geometric, distance and visual/textual information for frontiers and objects.</figcaption></figure>

[17], [19], [20] with large language models (LLMs) [21], [22] or vision-language models (VLMs) [23], [24], they require large-scale training and struggle with long-term spatial memory [13], limiting both long-horizon planning and interpretability. The second approach constructs explicit representations of the environment [25]–[30]. Most construct 2D semantic maps [25]–[28] and learn CNN-based policies to predict a long-term navigation waypoint, but these maps cover few object categories, lack semantic relationships across categories (e.g., “chair” and “armchair” are similar), and fail to exploit rich visual cues. VLMaps [30] uses semantic embeddings instead of fixed object labels, yet still loses fine-grained 3D information. VLFM [29] employs a VLM [31] to construct a value map based on image-text similarity, achieving strong zero-shot performance. However, its maps are tailored to a single target object, cannot be reused, and cannot be further improved from robot data.

In this work, we introduce **FOM-Nav**, a modular framework with **Frontier-Object Maps** for object goal Navigation (Figure 2). We propose *online frontier-object mapping* that explicitly maintains navigation memory, including obstacle and exploration maps alongside 3D object point clouds (Figure 1). Frontiers [29], [32] which represent boundary lines between explored and unexplored regions, are extracted from the maps to guide efficient exploration, while object point clouds augmented with visual and text features capture fine-grained semantic representations of the scene. Then, the *high-level goal prediction* module leverages a VLM to predict the next navigation goal given the encoded frontiers, objects and past trajectories. Finally, the *low-level path planning* module converts the high-level goal into an executable trajectory via A\* [33] and fast marching algorithms [34]. To support model training, we automatically collect a large-scale

† Inria. ‡ Department of Computer Science, École normale supérieure (ENS-PSL, CNRS, Inria). * Courant Institute of Mathematical Sciences and Center for Data Science, New York University. {firstname.lastname}@inria.fr
<figure><img src="image_4.png" /><figcaption>Fig. 2: FOM-Nav framework. At each step, the agent receives a posed RGB-D image. First, we back-project depth into a point cloud to maintain a 3D obstacle map, from which we derive 2D obstacle and exploration maps and frontiers. Simultaneously, we segment objects, and extract and store their geometric, visual and textual features in a 3D object map. Then, a VLM processes frontiers, objects, and history path to predict a high-level navigation goal. Finally, a low-level path planner (A* and fast marching algorithm) plans a path to the goal.</figcaption></figure>

robot navigation dataset using real-world scanned environments [35]. FOM-Nav achieves state-of-the-art performance on the MP3D and HM3D v1, v2 benchmarks in single-floor settings. Ablations confirm the contributions of our frontier-object mapping, feature design, and training data.

Our work makes the following key contributions:

* We introduce online frontier-object mapping to explicitly capture fine-grained memory with rich spatial-semantic information for object-goal navigation.

* We leverage a VLM architecture to predict high-level navigation goals by encoding frontier, object and history path features, and construct an automatic robot navigation dataset to support model learning.

* FOM-Nav sets a new state-of-the-art in ObjectNav benchmarks, surpassing prior methods on HM3D v2 with improvements of **11.8** in success rate and **14.9** in SPL, and shows promising results in the real world.

Our model, code and dataset are publicly available at
https://github.com/thomaschabal/fom-nav.

## II. RELATED WORK

**Semantic Mapping for Navigation.** Semantic maps enhance traditional geometric maps [6] by incorporating semantic information such as region types and object names, which are crucial for long-horizon multimodal navigation problems [36]. Building semantic maps requires robot localization, 3D projection, feature extraction and map accumulation, where many methods assume perfect localization for simplicity. Existing semantic mapping approaches can be categorized by their structural representations, including spatial grids [25], [30], topological graphs [37], [38], dense point clouds [39], [40] and hybrid maps [41]. The 2D semantic map [25], [26], [42] is one of the most widely used structures in spatial grids for navigation, but suffers from predefined categories and the lack of rich visual information. Dense points address these limitations but incur large memory cost. The hybrid approaches balance these trade-offs, yet their potential for online exploration remains unexplored. Our

work introduces a hybrid representation frontier-object maps to support efficient online object goal navigation.

**Object Goal Navigation.** After decades of focusing on point-based navigation [2], [3], [6], research [43] has shifted towards navigation with goals specified as object categories [8], images [9] or text instructions [7]. This requires multimodal understanding for goal identification and common sense reasoning for efficient navigation. Current approaches for ObjectNav differ in map representations, exploration strategies and path planning. One line of work uses implicit representations [12]–[15], [44], [45] learned from neural networks, which are trained via behavior cloning (BC) or reinforcement learning (RL). Another approach employs a modular pipeline [25]–[30], [40], [42]: semantic mapping, long-term goal prediction, and path planning. The goal prediction strategies range from RL with extrinsic rewards [25], [40], to BC with heuristic scores obtained from privileged information [42], [46], scene imagination [27], [28], and language-grounded value extraction [29]. For path planning, either deterministic planners [34] or PointNav policies [47] are utilized. Our work follows the modular pipeline but introduces an exploration policy powered by vision-language models and proposes frontier-object maps.

**Large Language and Vision Models.** Recently, large language models (LLMs) [21], [22] have achieved significant success trained with internet-scale data. They have been extended to vision-language models (VLMs) with inputs of images [23], videos [24] or 3D [48], showing promise for robotic applications [49], [50]. In navigation specifically, early efforts have incorporated LLMs through prompt engineering and post-processed the models’ text outputs [19], [20]. While achieving promising results, they fail to fully leverage rich visual information. To address these limitations, subsequent research uses VLMs for navigation by processing the entire image history during navigation [16]–[18]. These remain computationally intensive due to the large number of visual tokens. In this work, we integrate a compact, structured memory representation into VLMs to perform efficient exploration and navigation.
### III. METHOD

#### A. Overview

**Task definition.** The ObjectNav task [8] requires an agent to navigate in an unknown indoor environment to find a specified object category such as a “chair”. At each step, the agent receives an RGB-D observation and its camera pose, assumed to be provided by an external method, e.g. visual SLAM [6]. It can take one of four actions: move forward by 25cm, turn left or right by 30°, or stop. An episode is considered successful if the agent stops within 1 meter of an instance of the target object category, with the object visible from the stopping location, and within a 500-step limit.

**Overall framework.** The proposed FOM-Nav model consists of three modules as illustrated in Figure 2: online frontier-object mapping, high-level goal prediction, and low-level path planning. The online mapping module employs a segmentation model to process the RGB image at each step, projects the segmentation results into 3D space, and integrates them with the previously constructed map. The process generates an updated frontier-object map, containing both segmented objects and newly identified frontiers which represent the boundary between explored and unexplored regions. The goal prediction module then selects either a frontier or an object as the navigation goal. Finally, the path planning module generates an executable trajectory using the agent’s four actions to reach the predicted high-level goal.

#### B. Online Frontier-Object Mapping

We construct sparse 3D point clouds for obstacles and objects on-the-fly during navigation. At each step *t*, the scene point cloud $P_s^t$ is used to build a 2D obstacle map and extract frontiers, while a separate point cloud $P_{o_i}^t$ for each object $o_i$ allows dynamically merging seen objects and appending newly discovered ones.

**Obstacle mapping.** Given the depth image and the camera pose, we back-project the depth image into a point cloud $\hat{P}_s^t$ expressed in the world coordinate frame, without considering semantic information. We then add $\hat{P}_s^t$ to the previous scene point cloud $P_s^{t-1}$, and downsample it to a 5cm voxel resolution to obtain the current scene point cloud $P_s^t$. From $P_s^t$, we generate a 2D obstacle and exploration map $M_s^t$ by projecting $P_s^t$ onto the horizontal plane, centering at the robot’s current position, and inflating the obstacles by the robot’s radius. Using $M_s^t$, we follow prior work [29], [32] to track the explored regions and extract frontiers $\{F_j^t\}_{j=1}^{n_f}$. Additionally, we extract a visual feature $v_{f_j}^t$ for each newly detected frontier *j* at step *t*, as detailed in Section III-C.

**Object mapping.** For each RGB image at step *t*, we use an instance segmentation model Mask2Former [51] to automatically obtain segmentation masks with the corresponding probability distribution $\hat{c}_{o_i}^t$ over a set of predefined categories. For each segmented object $o_i$, we extract its visual feature $\hat{v}_{o_i}^t$ as detailed later in Section III-C, and back-project its pixels using the depth image and camera pose to create a 3D point cloud. We further filter isolated points in this point cloud using DBSCAN [52] and store it at a low resolution of 5cm as $\hat{P}_{o_i}^t$. At the first step, we convert all the segmented

objects $\{\hat{P}_{o_i}^1\}$ into the initial object point clouds $\{P_{o_i}^1\}$. In subsequent steps, for each newly segmented object, we first identify the closest existing object $P_{o_k}^{t-1}$ in $\{P_{o_i}^{t-1}\}$. If the proportion of matched points within a small distance exceeds a defined threshold, we merge the new object $\hat{P}_{o_j}^t$ with $P_{o_k}^{t-1}$; otherwise, we add the object $\hat{P}_{o_j}^t$ as a new entity. When merging objects, we combine their respective point clouds, compute a weighted average of their predicted category distributions and visual features, with weights proportional to the number of points in each point cloud. If two objects are merged, we continue merging the resulting one with other objects in $\{P_{o_i}^{t-1}\}$ until no further merges are found. This is because merged objects may connect two original disconnected point clouds. In this way, we obtain object point clouds $\{P_{o_i}^t\}_{i=1}^{n_o}$ from the entire observation history, where each object contains an accumulated visual feature $v_{o_i}^t$ and category distribution $c_{o_i}^t$.

#### C. High-level Goal Prediction

Our high-level goal prediction model adopts a similar architecture to recent VLMs including LLaVA [23]. As illustrated in Figure 3, the model processes the frontier-object map to determine the next navigation goal.

**Input Encoding.** We encode frontiers, objects and the robot’s history path separately. All the spatial coordinates below are expressed in the agent’s egocentric frame.

**Frontier embedding:** It combines three complementary features: geometric, visual and distance features.

The geometric feature encodes the 2D spatial configuration of the frontier in the obstacle map $M_s^t$. Let $F_j^t$ denote a frontier at step *t*, represented as a boundary line from $(x_j^{e_1}, y_j^{e_1})$ to $(x_j^{e_2}, y_j^{e_2})$, with center point $(x_j^c, y_j^c)$. We use two multi-layer perceptrons (MLP) $f_{ends}^F$ and $f_{center}^F$ to encode the endpoints and the center point respectively.

The visual feature captures the surrounding environment of the frontier. Let $I^t$ represent the RGB image at which the frontier $F_j^t$ is first detected. We apply DINO-v2 [53], [54] to extract patch-level image features from $I^t$. Instead of using a global image feature, we extract localized visual features around the frontier. Given all 2D points along the frontier line, we extend them vertically to 3D points with heights ranging from ground level to 50cm above the robot. These 3D points are projected onto the camera frame of $I^t$, generating a rectangular mask for the frontier region in the image. The frontier mask is resized to match the patch-level feature size, and we compute the mean feature within the masked region to obtain the frontier’s visual feature $v_{f_j}^t$, which is then processed by another MLP, $f_{vis}^F$.

The distance feature quantifies the cost of reaching a frontier. We compute the geodesic distance $d_j$ from the robot’s current position to the frontier center on the obstacle map $M_s^t$ using the fast marching method (FMM) [34]. This distance is encoded using an MLP, $f_{dist}^F$.

The final embedding of the frontier $F_j^t$ is computed as follows, where $T_F$ a learnable vector representing the type
<figure><img src="image_5.png" /><figcaption>Fig. 3: Our high-level goal prediction module employs a transformer-based LLM to processes a rich semantic representation of the scene, including objects, frontiers, path history and texts. Objects and frontiers consist of visual, geometric, distance, and textual features. The prediction head operates in two phases: 1) the transformer's last output token $E_N$ is classified as either a frontier or object destination type; 2) $E_N$ is matched against output tokens for objects or frontiers using a learnable matrix A to identify the best navigation goal.</figcaption></figure>

of frontier tokens:

$$ \mathcal{E}_j^F = f_{center}^F(x_j^c, y_j^c) + f_{ends}^F(x_j^{e1}, y_j^{e1}, x_j^{e2}, y_j^{e2}) + f_{vis}^F(v_{fj}) + f_{dist}^F(d_j) + T_F. \quad (1) $$

**Object embedding:** Similar to frontier embedding, object embedding integrates geometric, distance, visual and additional text category features.

The geometric feature encodes the object's spatial information, including the coordinates of its point cloud center $(x_{o_i}^t, y_{o_i}^t, z_{o_i}^t)$ and bounding box corner coordinates $B_{o_i}^t$. The distance feature encodes the geodesic distance $d_{o_i}$ from the robot's current position to the object.

The visual feature is accumulated over all frames in which the object is detected according to the merged object point cloud $P_{o_i}^t$. At each frame, we extract patch-level image features using Dino-v2. The object's segmentation mask is used to average patch-level features within the masked region as $\tilde{v}_{o_i}^t$. The final visual feature $v_{o_i}^t$ is a weighted average of $\{v_{o_i}^{t', t'}\}_{t'=1}^t$, providing robustness to partial occlusions and varying viewpoints.

The text category feature is derived from the object's accumulated category distribution $c_{o_i}^t$. We select the category name with the highest probability in $c_{o_i}^t$ and encode it using the CLIP [55] text encoder $f_{cat}^O$.

The final embedding of the object $o_i$ is as follows:

$$ \mathcal{E}_t^O = f_{center}^O(x_{o_i}^t, y_{o_i}^t, z_{o_i}^t) + f_{corners}^O(B_{o_i}^t) + f_{dist}^O(d_{o_i}) + f_{vis}^O(v_{o_i}^t) + f_{cat}^O(c_{o_i}^t) + T_O, \quad (2) $$

where $T_O$ is a learnable vector for denoting the type of the object token.

**Path embedding:** The history path can provide contextual information to maintain decision consistency for navigation. We encode 2D locations $(x_{t'}, y_{t'})^t_{t'=1}$ expressed in the current robot frame along the robot's history path via an MLP $f_{loc}^P$ as follows, with $T_P$ being a learnable vector to represent the

## Next Goal Prediction

### Phase 1: frontier or object classification

<figure><img src="image_6.png" /></figure>

### Phase 2: best match selection

<figure><img src="image_7.png" /></figure>

type of path tokens:

$$ \mathcal{E}_{t'}^P = f_{loc}^P(x_{t'}, y_{t'}) + T_P. \quad (3) $$

**LLM.** We construct a prompt for the LLM with the following template: "Given seen objects `<objects>`, frontiers of explored areas `<frontiers>` and past locations `<path history>`, where should I go to find a `<target category>`?" The `<target category>` is replaced with the specific category name to search for. The `<objects>`, `<frontiers>` and `<path history>` are substituted with their corresponding token embeddings. Early experiments with other prompts showed similar performance.

Given $n_f$ frontier tokens and $n_o$ object tokens, the LLM generates corresponding output embeddings for frontiers and objects, denoted as $E_N \in \mathbb{R}^{n_f \times D}$ and $E_O \in \mathbb{R}^{n_o \times D}$ respectively, where D is the feature dimensionality. Additionally, the model produces an embedding for the next token prediction, represented as $E_N \in \mathbb{R}^{1 \times D}$.

**Prediction Head.** We predict the next navigation goal in two phases. First, we apply a binary classification to the next token embedding $E_N$ using a linear layer to determine whether the next goal should be a frontier or an object. Based on this decision, we compute the similarity between $E_N$ and the frontier or object embeddings as:

$$ S = E_N^T(A^T A)E, \quad (4) $$

where $A \in \mathbb{R}^{D \times D'}$ is a learnable matrix and $E = E_F$ if $E_N$ is classified as a frontier and $E_O$ otherwise. The next goal is selected as the frontier or object with the highest similarity.

**Training.** We train the navigation model on a dataset described in Section IV. The training objective consists of two components: a binary cross-entropy (BCE) loss $\mathcal{L}_{BCE}$ for frontier-object classification and a cross-entropy (CE) loss $\mathcal{L}_{CE}$ for selecting the specific frontier or object.

$$ \mathcal{L} = \mathcal{L}_{BCE} + \lambda \mathcal{L}_{CE}. \quad (5) $$
### D. Low-level Path Planning

Given the predicted goal from the high-level model, we first determine a precise target location or area. We then use a path planner to navigate towards this target.

**Determining the target location.** The robot follows different strategies based on the goal type. If the next goal is a frontier, we consider the center point of the frontier as the target. If the goal is an object, we either compute a valid stopping area next to it if the object is in the explored area, or we move to the closest frontier to the target.

**Path planner.** Once the target location/area is determined, we compute a distance map from that location to the explored area using the Fast Marching algorithm (FMM) [34], which serves as a heuristic cost for $A^*$ [33]. Unlike prior work [25], [26] using FMM alone, incorporating $A^*$ on top of FMM ensures consistent long-term trajectory planning. Following [25], [26], we maintain a collision map that stores locations where the agent collides with the environment, defined as no position change after a *move forward* action, or gets stuck in holes of the scanned environment meshes. This collision map is combined with the obstacle map to extract frontiers and plan a path. As in prior work [29], we start each episode with a 360° rotation to observe the surroundings before taking any action.

Compared to point-goal navigation policies [47] used in prior works [29], our path planner is not restricted to discrete action spaces, making it directly applicable to navigation with continuous actions. In that case, $A^*$ becomes unnecessary and collision maps are less important, only restricted to scene parts invisible to a fixed camera.

## IV. OBJECTNAV DATASET CONSTRUCTION

Existing ObjectNav datasets either use shortest paths [35], which limit exploration, or rely on costly human demonstrations [12]. To overcome this, we automatically generate navigation trajectories using our online frontier-object maps, enabling scalable training for FOM-Nav.

**Dataset construction.** Our dataset is built upon scanned real-world environments with semantic annotations such as HM3D [35] and MP3D [56].

We first pre-process each scene to obtain ground-truth stopping locations for each object. These are viewpoints where the target object is visible and within 1 meter, consistent with the evaluation metric. We then collect navigation trajectories using the Habitat simulator [57], randomly selecting initial robot locations and target objects on the same floor, as crossing stairs with wheeled robots is impractical.

At the start of each navigation episode, the robot performs a 360° rotation to observe its surroundings. It constructs a frontier-object map on-the-fly as described in III-B. At each step, in the case a target object is determined to be visible, it is selected as the ground-truth next sub-goal. Otherwise, for each frontier, we compute its distance to the robot and to the closest ground-truth stopping locations. The best frontier is chosen as the one minimizing the average distance to the agent and the target.

We collect datasets using either ground-truth segmentations or automatic ones from an instance segmentation model for online mapping. In the latter case, we still rely on ground-truth segmentations to identify the target object in the automatically constructed object maps, which is the object with highest point cloud overlap with the ground-truth object. We store the built object maps, frontiers, path history and target labels at each step.

**Dataset statistics.** We collect navigation trajectories with target categories in existing benchmarks on training environments. For the HM3D v1/v2 benchmark [35], we gather a total of 490k and 270k steps with either ground-truth or automatic segmentations for mapping from 145 scenes, corresponding to 12.6k and 7k episodes, respectively. For the MP3D benchmark [56], we use 50 scenes and collect 215k and 80k steps with ground-truth or automatic segmentation in a total of 5k and 2k episodes, respectively.

## V. EXPERIMENTS

### A. Experimental Setup

**ObjectNav Benchmarks.** We evaluate our model on three ObjectNav benchmarks. The MP3D benchmark [8] evaluates 21 object categories, and has 56 training and 11 validation scenes. The HM3D benchmark [35] focuses on 6 object categories, including 145 scenes for training and 36 for validation. It contains two validation versions: v1 from Habitat 2022 challenge [58] and v2 from Habitat 2023 challenge [59]. The main difference is that v2 has better ground-truth annotations and does not require navigation across floors. The HM3D-OVON benchmark [60] operates on the same HM3D environments but includes hundreds of target categories, split between those seen, unseen or synonyms to objects in training scenes.

**Evaluation metrics.** We use two primary metrics following [8] for online navigation evaluation: success rate (SR) and success rate weighted by path length (SPL). The SR metric measures the ratio of successful episodes where the robot stops within 1 meter to the target object with the object visible, and within 500 steps. The SPL metric considers the robot's path length $d_T$ and the geodesic distance $d_G$ from the start to the closest target object, defined as: $SPL = \text{Success} \times \frac{d_G}{\max(d_G, d_T)}$. This metric emphasizes exploration efficiency besides accuracy.

**Implementation details.** We train FOM-Nav using the constructed dataset described in Section IV for each benchmark. For online mapping, we train Mask2Former [51] on images from training scenes of the respective benchmarks as the instance segmentation model. For the high-level goal prediction model, the encoding MLPs all consist of 2 hidden layers with 2048 neurons per layer. The LLM is initialized from LLaVA-v1.6 7B model [61] and remains frozen. The decoding projection in the prediction head has $D' = 2048$. The goal prediction model is trained for 2 epochs on 4 H100 GPUs, which takes around 2 hours for MP3D and 4 hours for HM3D, using a learning rate of $2 \times 10^{-4}$ and batch size of 32 per GPU. We use $\lambda = 0.5$ in the loss.
<table><caption>TABLE V: Results on the validation splits of MP3D and HM3D v1 benchmarks including episodes that require crossing floors. VLFM* is our reproduced results and VLFM† is VLFM with our segmentation model.</caption><thead><tr><th rowspan="2">Memory</th><th rowspan="2">Method</th><th colspan="2">MP3D</th><th colspan="2">HM3D v1</th></tr><tr><th>SR</th><th>SPL</th><th>SR</th><th>SPL</th></tr></thead><tbody><tr><td rowspan="5">Implicit</td><td>Habitat-Web [12]</td><td>35.4</td><td>10.2</td><td>57.6</td><td>23.8</td></tr><tr><td>RIM [15]</td><td><strong>50.3</strong></td><td><strong>17.0</strong></td><td>57.8</td><td>27.2</td></tr><tr><td>OVRL-v2 [14]</td><td>—</td><td>—</td><td>64.7</td><td>28.1</td></tr><tr><td>PIRLNav [13]</td><td>—</td><td>—</td><td>70.4</td><td>34.1</td></tr><tr><td>XGX [62]</td><td>—</td><td>—</td><td><strong>72.9</strong></td><td><strong>35.7</strong></td></tr><tr><td rowspan="5">Explicit</td><td>SGM [28]</td><td>37.7</td><td>14.7</td><td><strong>60.2</strong></td><td>30.8</td></tr><tr><td>VLFM [29]</td><td><strong>36.4</strong></td><td>17.5</td><td>52.5</td><td>30.4</td></tr><tr><td>VLFM* [29]</td><td>34.4</td><td>16.7</td><td>52.0</td><td>30.2</td></tr><tr><td>VLFM†</td><td>36.3</td><td>17.1</td><td>53.5</td><td>32.0</td></tr><tr><td>FOM-Nav (Ours)</td><td>35.0</td><td><strong>18.7</strong></td><td>57.3</td><td><strong>40.8</strong></td></tr></tbody></table>

a target object was actually reached. On HM3D v1/v2 benchmarks with higher-quality annotations, the performance improvement is larger. For example, FOM-Nav achieves an increase of up to 11.8 and 14.3 points in SR and SPL respectively compared to the best VLFM on HM3D v2.

We include for completeness in Table V results obtained on the full MP3D and HM3D v1 benchmarks including those cross-floor navigation episodes. Implicit memory-based approaches are able to navigate through stairs given cross-floor training data, while explicit ones currently consider them as obstacles since wheeled robots cannot climb stairs. In this setting, implicit methods do, as expected, better. Identifying stairs and considering them as traversible areas connecting two floors would address this limitation. Compared to the state of the art, our method has a SR comparable to the best explicit approaches and to implicit ones such as Habitat-Web [12] on both benchmarks. Notably, it obtains the highest SPL metric among all the methods, showcasing again the efficiency of our exploration.

**OVON benchmark.** To assess open-vocabulary ObjectNav, we further evaluate on the OVON benchmark in Table VI. For efficiency, we use Llama 3.2 1B [63] as the base LLM. Because FOM-Nav introduces its own input encoders and output heads, it cannot directly inherit the generalization capabilities of pretrained VLMs without training these components on Internet-scale data. We therefore evaluate two variants of our model. The first variant is trained on all annotated categories in the HM3D training houses (row HM-all), including synonyms and unseen categories within those houses. This model achieves a large SPL improvement over prior methods, demonstrating its efficient open-vocabulary navigation ability. Its SR is comparable to VLFM and DAgRL+OD but lower than Uni-NaVid and MTU3D, suggesting that Internet-scale pretraining is important for object recognition. However, this setup is somewhat favorable to us since unseen categories appear in the training environments, whereas prior methods rely solely on Internet data for generalization to unseen categories. To create a fairer comparison, we keep episodes targeting synonyms and unseen categories but remove their ground-truth labels,

<table><caption>TABLE VI: Results on OVON validation splits. VLFM [29] and DAgRL+OD [60] use OWLv2 [64] open-vocabulary detector. Uni-Navid [18] and MTU3D [65] train on large-scale vision-language data and navigation data. ‘HM-all’ uses all HM3D categories and ‘auto-label’ replaces GT labels for synonyms and unseen categories with automatic annotations.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">Training categories</th><th colspan="2">Val seen</th><th colspan="2">Val synonyms</th><th colspan="2">Val unseen</th></tr><tr><th>SR</th><th>SPL</th><th>SR</th><th>SPL</th><th>SR</th><th>SPL</th></tr></thead><tbody><tr><td>VLFM</td><td>open-vocab</td><td>35.2</td><td>18.6</td><td>32.4</td><td>17.3</td><td>35.2</td><td>19.6</td></tr><tr><td>DAgRL+OD</td><td>open-vocab</td><td>38.5</td><td>21.1</td><td>39.0</td><td>21.4</td><td>37.1</td><td>19.9</td></tr><tr><td>Uni-Navid</td><td>open-vocab</td><td>41.3</td><td>21.1</td><td>43.9</td><td>21.8</td><td>39.5</td><td>19.8</td></tr><tr><td>MTU3D</td><td>open-vocab</td><td><strong>55.0</strong></td><td>23.6</td><td><strong>45.0</strong></td><td>14.7</td><td><strong>40.8</strong></td><td>12.1</td></tr><tr><td>FOM-Nav<br>(Ours)</td><td>HM-all</td><td>42.5</td><td><strong>27.8</strong></td><td>36.3</td><td><strong>23.4</strong></td><td>38.1</td><td><strong>25.7</strong></td></tr><tr><td></td><td>HM-all auto</td><td>37.6</td><td>24.6</td><td>29.4</td><td>18.5</td><td>30.9</td><td>20.6</td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>Fig. 4: Visualization of a navigation episode to a bed in an HM3D environment [35]. Left: RGB view from the agent. Center: Online frontier-object maps: grey = inflated obstacles, green = explored area, purple dots = objects, red circles = frontiers, and filled orange/cyan circles = selected object/frontier goals. If the chosen object lies in unexplored space, the agent first moves toward the corresponding frontier. Planned and executed paths are shown in pink and navy. Right: Agent trajectory (navy), ground-truth object (white box), and stopping area (pink) on the GT map.</figcaption></figure>

automatically annotating targets using a VLM (row HM-all auto-label). Although performance decreases relative to HM-all, this variant still surpasses or achieves comparable performance to previous methods in SPL, using far fewer resources than Uni-Navid and MTU3D. Future work may improve generalization further by training FOM-Nav on larger-scale vision-language data and more environments.

**Qualitative results.** We show in Figure 4 an example of an episode with the target category bed. After an initial self-rotation (a), the agent decides to go to the corridor instead of searching in the kitchen (b). Once the bed becomes visible, FOM-Nav selects it as the next target instead of exploring another frontier and room (c). It then computes a valid stopping area and stops inside (d).
## D. Real-world deployment

We deploy our model on a wheeled TIAGo++ robot in our laboratory. We mount an Orbbec Femto Mega RGB-D camera on the robot's head. All the models are run on an external desktop equipped with an RTX 4060 GPU. The localization is done with the RGB-D version of ORB-SLAM3 [66] and the image segmentation uses an off-the-shelf Mask2Former trained on the ADE20K dataset. Our supplementary video demonstrates the successful transfer of our method to the real world.

## VI. CONCLUSION

This paper introduces FOM-Nav for object goal navigation, leveraging frontier-object maps for rich semantic scene representation, a vision-language model for goal prediction, and low-level path planner for action execution. Trained on automatically constructed navigation datasets, FOM-Nav achieves state-of-the-art performance on two ObjectNav benchmarks, particularly in exploration efficiency. Future work includes extending the approach to multi-floor and more complex semantic navigation tasks.

## ACKNOWLEDGMENT

This work was granted access to the HPC resources of IDRIS under the allocation 2021-AD011012725 made by GENCI. It was supported in part by the French government under management of Agence Nationale de la Recherche as part of the "France 2030" program, PR[AI]RIE-PSAI projet, reference ANR-23-IACL-0008. JP was supported in part by the Louis Vuitton/ENS chair in artificial intelligence and a Global Distinguished Professorship at the Courant Institute of Mathematical Sciences and the Center for Data Science at New York University.

## REFERENCES

[1] W. Burgard et al., "The interactive museum tour-guide robot," in AAAI/IAAI, 1998.
[2] N. NILLSON, "Shakey the robot," *SRI International, Technical Note*, vol. 323, 1984.
[3] G. N. DeSouza and A. C. Kak, "Vision for mobile robot navigation: A survey," *IEEE TPAMI*, 2002.
[4] F. Bonin-Font et al., "Visual navigation for mobile robots: A survey," *J. Intell. Robotic Syst.*, vol. 53, no. 3, pp. 263-296, 2008.
[5] J. J. Leonard and H. F. Durrant-Whyte, "Simultaneous map building and localization for an autonomous mobile robot," in *IROS*, 1991.
[6] S. Thrun and J. J. Leonard, "Simultaneous localization and mapping," in *Springer Handbook of Robotics*. Springer, 2008, pp. 871-889.
[7] P. Anderson et al., "Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments," in *CVPR*, 2018.
[8] D. Batra et al., "Objectnav revisited: On evaluation of embodied agents navigating to objects," *CoRR*, vol. abs/2006.13171, 2020.
[9] Y. Zhu et al., "Target-driven visual navigation in indoor scenes using deep reinforcement learning," in *ICRA*, 2017.
[10] D. Watkins-Valls et al., "Mobile manipulation leveraging multiple views," in *IROS*, 2022.
[11] P. Liu et al., "Ok-robot: What really matters in integrating open-knowledge models for robotics," *CoRR*, vol. abs/2401.12202, 2024.
[12] R. Ramrakhya et al., "Habitat-web: Learning embodied object-search strategies from human demonstrations at scale," in *CVPR*, 2022.
[13] ——, "Pirlnav: Pretraining with imitation and RL finetuning for OBJECTNAV," in *CVPR*, 2023.
[14] K. Yadav et al., "OVRL-V2: A simple state-of-art baseline for image NAV and object NAV," *CoRR*, vol. abs/2303.07798, 2023.

[15] S. Chen et al., "Object goal navigation with recursive implicit maps," in *IROS*, 2023.

[16] J. Zhang et al., "Navid: Video-based VLM plans the next step for vision-and-language navigation," in *RSS*, 2024.

[17] Z. Xu et al., "Mobility VLA: multimodal instruction navigation with long-context vlms and topological graphs," in *CoRL*, 2024.

[18] J. Zhang et al., "Uni-navid: A video-based vision-language-action model for unifying embodied navigation tasks," in *RSS*, 2025.

[19] G. Zhou et al., "Navgpt: Explicit reasoning in vision-and-language navigation with large language models," in *AAAI*, 2024.

[20] ——, "Navgpt-2: Unleashing navigational reasoning capability for large vision-language models," in *ECCV*, 2024.

[21] H. Touvron et al., "Llama: Open and efficient foundation language models," *CoRR*, vol. abs/2302.13971, 2023.

[22] OpenAI, "GPT-4 technical report," *CoRR*, vol. abs/2303.08774, 2023.

[23] H. Liu et al., "Visual instruction tuning," in *NeurIPS*, 2023.

[24] G. Team, "Gemini: a family of highly capable multimodal models," *arXiv preprint arXiv:2312.11805*, 2023.

[25] D. S. Chaplot et al., "Object goal navigation using goal-oriented semantic exploration," in *NeurIPS*, 2020.

[26] H. Luo et al., "Stubborn: A strong baseline for indoor object navigation," in *IROS*, 2022.

[27] A. J. Zhai and S. Wang, "PEANUT: predicting and navigating to unseen targets," in *ICCV*, 2023.

[28] S. Zhang et al., "Imagine before go: Self-supervised generative map for object goal navigation," in *CVPR*, 2024.

[29] N. Yokoyama et al., "VLFM: vision-language frontier maps for zero-shot semantic navigation," in *ICRA*, 2024.

[30] C. Huang et al., "Visual language maps for robot navigation," in *ICRA*, 2023.

[31] J. Li et al., "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models," in *ICML*, 2023.

[32] B. Yamauchi, "A frontier-based approach for autonomous exploration," in *CIRA*, 1997.

[33] P. E. Hart et al., "A formal basis for the heuristic determination of minimum cost paths," *IEEE Trans. Syst. Sci. Cybern.*, 1968.

[34] J. A. Sethian, "A fast marching level set method for monotonically advancing fronts." *Proceedings of the National Academy of Sciences of the United States of America*, 1996.

[35] S. K. Ramakrishnan et al., "Habitat-matterport 3d dataset (HM3D): 1000 large-scale 3d environments for embodied AI," in *NeurIPS Datasets and Benchmarks*, 2021.

[36] S. Raychaudhuri and A. X. Chang, "Semantic mapping in indoor embodied AI - A survey on advances, challenges, and future directions," *Trans. Mach. Learn. Res.*, 2025.

[37] S. Garg et al., "Robohop: Segment-based topological map representation for open-world visual navigation," in *ICRA*, 2024.

[38] S. Chen et al., "Think global, act local: Dual-scale graph transformer for vision-and-language navigation," in *CVPR*, 2022.

[39] S. Peng et al., "Openscene: 3d scene understanding with open vocabularies," in *CVPR*, 2023.

[40] J. Zhang et al., "3d-aware object goal navigation via simultaneous exploration and identification," in *CVPR*, 2023.

[41] Q. Gu et al., "Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning," in *ICRA*, 2024.

[42] S. K. Ramakrishnan et al., "PONI: potential functions for object goal navigation with interaction-free learning," in *CVPR*, 2022.

[43] P. Anderson et al., "On evaluation of embodied navigation agents," *CoRR*, vol. abs/1807.06757, 2018.

[44] J. Ye et al., "Auxiliary tasks and exploration enable object goal navigation," in *ICCV*, 2021.

[45] A. Majumdar et al., "ZSON: zero-shot object-goal navigation using multimodal goal embeddings," in *NeurIPS*, 2022.

[46] M. Zhu et al., "Navigating to objects in unseen environments by distance prediction," in *IROS*, 2022.

[47] E. Wijmans et al., "DD-PPO: learning near-perfect pointgoal navigators from 2.5 billion frames," in *ICLR*, 2020.

[48] C. Zhu et al., "Llava-3d: A simple yet effective pathway to empowering lmms with 3d-awareness," in *ICCV*, 2025.

[49] D. Driess et al., "Palm-e: An embodied multimodal language model," in *ICML*, 2023.

[50] K. Black et al., "$\pi_Q$: A vision-language-action flow model for general robot control," *CoRR*, vol. abs/2410.24164, 2024.

[51] B. Cheng et al., "Masked-attention mask transformer for universal image segmentation," in *CVPR*, 2022.
[52] M. Ester et al., "A density-based algorithm for discovering clusters in large spatial databases with noise," in *KDD*, 1996.
[53] M. Oquab et al., "Dinov2: Learning robust visual features without supervision," *Trans. Mach. Learn. Res.*, 2024.
[54] T. Darcet et al., "Vision transformers need registers," in *ICLR*, 2024.
[55] A. Radford et al., "Learning transferable visual models from natural language supervision," in *ICML*, 2021.
[56] A. X. Chang et al., "Matterport3d: Learning from RGB-D data in indoor environments," in *3DV*, 2017.
[57] M. Savva et al., "Habitat: A platform for embodied AI research," in *ICCV*, 2019.
[58] K. Yadav et al., "Habitat challenge 2022," https://aihabitat.org/challenge/2022/.
[59] ——, "Habitat challenge 2023," https://aihabitat.org/challenge/2023/.
[60] N. Yokoyama et al., "HM3D-OVON: A dataset and benchmark for open-vocabulary object goal navigation," in *IROS*, 2024.
[61] H. Liu et al., "Improved baselines with visual instruction tuning," in *CVPR*, 2024.
[62] J. Wasserman et al., "Exploitation-guided exploration for semantic embodied navigation," in *ICRA*, 2024.
[63] A. Dubey et al., "The llama 3 herd of models," *CoRR*, vol. abs/2407.21783, 2024.
[64] M. Minderer et al., "Scaling open-vocabulary object detection," in *NeurIPS*, 2023.
[65] Z. Zhu et al., "Move to understand a 3d scene: Bridging visual grounding and exploration for efficient and versatile embodied navigation," *CoRR*, vol. abs/2507.04047, 2025.
[66] C. Campos et al., "ORB-SLAM3: an accurate open-source library for visual, visual-inertial, and multimap SLAM," *IEEE Trans. Robotics*, 2021.
[67] K. Yadav et al., "Habitat-matterport 3d semantics dataset," in *CVPR*, 2023.
[68] T. Wolf et al., "Transformers: State-of-the-art natural language processing," in *EMNLP - Demos*, 2020.
[69] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization," in *ICLR*, 2019.
[70] T. Dao, "Flashattention-2: Faster attention with better parallelism and work partitioning," in *ICLR*, 2024.
[71] B. Zhou et al., "Semantic understanding of scenes through the ADE20K dataset," *IJCV*, 2019.
[72] A. Gupta et al., "LVIS: A dataset for large vocabulary instance segmentation," in *CVPR*, 2019.
# S1. ADDITIONAL IMPLEMENTATION DETAILS

## A. Instance segmentation model

We fine-tune Mask2Former [51] pretrained on Ade20k dataset as our instance segmentation model. To this end, we automatically collect 2000 RGB images taken at random view points in each training house of the MP3D [56] and HM3D [35] benchmarks. Ground-truth segmentation masks are rendered for each image using the Habitat simulator [57] and the annotated meshes from MP3D and HM3D [67]. For MP3D, annotations contain around 35 categories. For HM3D, we remove labels corresponding to walls, floors or ceilings and use the 1.4k remaining categories for segmentation. The large diversity of labels in HM3D may provide more fine-grained scene understanding to learn richer environment priors and enable efficient navigation than the limited set of labels from MP3D. Tiny objects occupying less than 100 pixels are excluded from training. We then train one segmentation model per benchmark. The model is fine-tuned for one epoch on the collected dataset, using a learning rate of $5 \times 10^{-5}$ with Adam optimizer and a batch size of 16. During online evaluations, we keep masks with a class probability larger than a threshold of 0.65 for HM3D and 0.8 for MP3D. We filter masks that are too close to the borders of an image, i.e., at less than 30 pixels from the top or bottom or 20 pixels from the left or right sides, and discard masks covering less than 100 pixels. We use the same models and parameters for the VLFM† baseline.

## B. Frontier and object mapping

**Object merging.** During the object point cloud merging phase, we match two objects A and B, with respective point clouds $P_A$ and $P_B$, by computing the ratio of their points at a distance smaller than $\delta = 3\text{cm}$ from the other point cloud. We merge A and B if at least one of these two ratios is above 50%.

**Frontier mask for visual features.** We show examples of RGB images and the masks applied to these images to extract frontier visual features in Figure S1. These localized masks are computed following the procedure described in Section 3.3 of the main paper and focus on specific regions of the image while the Dino-v2 model extracting their visual features processes the whole image, providing more context about the surroundings of each frontier. The global feature computed in the ablation of Table 1 from the main paper is instead obtained from the whole RGB image, and thus different frontiers may contain the same visual feature.

**Dealing with holes in Habitat simulator.** During online navigation in the MP3D and HM3D environments, we set the camera's minimal depth value to 0 meter. This allows us to separate pixels corresponding to holes in the scene meshes, for which the Habitat simulator [57] renders a depth value of 0, and not back-project them in point cloud creation. If in a frame we observe both pixels with a valid depth and ones corresponding to mesh holes, we set the depth for these hole pixels to the maximum depth of their neighbouring valid pixels. This often creates obstacles in our maps at the

location of holes and avoids considering nonexistent frontiers there. As several scenes also contain holes in the ground or the ceiling, from which the robot sees objects located at other floors, we additionally filter all the objects located at other floors in our object map.

## C. High-level goal prediction model

**Inputs.** The VLM used as the base of our navigation model, i.e., LLaVA v1.6 [23], [61], processes embeddings of size $D = 4096$. Our added tokens for frontiers, objects and paths are each of size $D$. Visual features for frontiers and objects are encoded using the large version of Dino-v2 with registers [53], [54]. We use the text encoder from ViT-L-patch14-336 version of CLIP [55], available through the transformers library [68] to encode categories. All the distances and spatial coordinates are divided by a factor $\tau = 10$ before being fed to the model.

**Training.** As in LLaVA [23], we train our model using an AdamW optimizer [69] and a cosine learning rate scheduler. During training, we shuffle input frontier and object embeddings separately before feeding them to the transformer. Early experiments showed no additional benefit of using RGB images at the current step in our goal prediction model. As the frontier and object representations already contain visual features, RGB images may bring redundant information while significantly increasing the computational cost of both training and inference. The current image may also not be informative about the long-term path to follow or the exploration to perform, e.g., seeing a textureless wall or a corridor may not tell anything about where the target is located.

**Online navigation** We evaluate the last checkpoint saved in training. The navigation model is queried after each *move forward* action to reduce the risk of alternating between two frontiers and getting stuck.

## D. Navigation stopping area

After an object has been selected as the next goal and if it is already in the explored part of the scene, we compute a stopping area in which the agent is closer than some threshold to the target. We obtain a target-free navigability map by subtracting three elements to the scene navigability one:

* The target object 2D map, obtained by projecting its point cloud onto a horizontal plane,

* The 2D map of obstacles placed above or below the target in the scene,

* All the obstacles placed on lines between the object's center and the 2D locations from which the agent observed the object during its navigation.

The distance map to the target is then computed using this target-free obstacle map. We consider as the valid stopping area the locations closer than 80cm from the target in HM3D and 70cm in MP3D.
<figure><img src="image_9.png" /><figcaption>Fig. S1: Visualizations of RGB images (top left of each block) and the masks used to compute visual features for each frontier. The masks are the highlighted parts of the darker images, localized around each frontier. Some frontiers are located on image borders and their mask is computed as a vertical stripe, see the last image in the top blocks.</figcaption></figure>

## S2. EVALUATIONS

### A. Evaluation details

**Metrics.** We notice that computing success for the HM3D v2 dataset using the 3D distance to target view points results in a high number of failed episodes in which the agent correctly reaches the target. Replacing the 3D distance to the target view points with the 2D one in the horizontal plane solves this issue, as several ground-truth target view points appear to have incorrect heights. We use it for evaluations on the HM3D v2 benchmark exclusively.

**ObjectNav benchmarks.** We use the same robot configuration as previous work [15], [29] for these benchmarks, i.e., images of size 640 × 480 acquired at 88cm from the ground with a field of view of 79°. The OVON benchmark evolves in the same houses as HM3D but considers a larger set of categories, split into 280 categories seen during training, and 50 synonyms and 49 unseen categories for validation. We adopt its robot configuration for these evaluations, i.e., images of size 360 × 640 acquired at 1.31m from the ground with a horizontal field of view of 42°.

**Incorrect annotations.** The MP3D [56] and HM3D v1 [35], [58] datasets contain incomplete ground-truth annotations, i.e., some correct objects are not annotated as goals in several environments or are wrongly considered as part of another category. We show such examples in Section S2-B. This impacts the performance of all the methods in the

<table><caption>TABLE S1: Ablation of the base language model used in our method on the HM3D v2 validation split.</caption><thead><tr><th></th><th>SR</th><th>SPL</th></tr></thead><tbody><tr><td>Llama 3.2 1B Instruct [63]</td><td>74.8</td><td>47.3</td></tr><tr><td>LLaVA v1.6 7B [61]</td><td><strong>75.8</strong></td><td><strong>47.9</strong></td></tr></tbody></table>

MP3D benchmark, partially explaining the performance gap compared to HM3D. Annotations in the HM3D v2 [59] are of a higher quality, resulting in higher SR and SPL on the v2 split than the v1.

**Computation and memory costs.** FOM-Nav uses 20Gb of GPU memory: 15 for LLaVA and 5 for Mask2Former, Dino-v2 and the map storage. Our maps are updated in 130 ms, with most time spent on point cloud cleaning using DBSCAN. A forward pass through LLaVA takes 100 ms, using FlashAttention-2 [70]. Its typical inputs contain 5 frontiers, 100 objects, 50 path tokens and the text ones, resulting in around 200 tokens, which is 2 times smaller than, e.g., Uni-Navid [18].

### B. Additional results

**Navigation visualizations.** We show additional examples of successful navigation episodes on the HM3D benchmark in Figures S3 and S4 and on MP3D in Figure S5, where the agent must find the categories *chair*, *toilet* or *counter*.

**Ablation of the LLM base model.** In Table S1, we evaluate,
<figure><img src="image_10.png" /><figcaption>Fig. S2: Examples of wrong ground-truth annotations in the MP3D dataset. The agent decides to stop in these frames with the target category indicated under each image, but the episode is considered a failure as these objects are not part of the GT.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>Fig. S3: Visualization of an episode with navigation to a *chair* in an HM3D scene. (Left) RGB image seen by the agent. (Right) Visualization of the agent's trajectory and the groundtruth stopping area on HM3D GT maps. (Center) Our navigation maps built online. Grey pixels correspond to inflated obstacles, the green area is the explored surface, purple dots and red circles are objects and frontiers, respectively, and filled orange and cyan circles are selected next object or frontier. The planned and actual paths are shown as pink and navy curves, respectively. (Right) Visualization, over the GT map, of the agent's trajectory (navy curve), ground-truth object (white box), and ground-truth stopping area (pink area).</figcaption></figure>
<figure><img src="image_12.png" /><figcaption>Fig. S4: Visualization of an episode with navigation to a *toilet* in an HM3D scene. (Left) RGB image seen by the agent. (Center) Our navigation maps built online. Grey pixels correspond to inflated obstacles, the green area is the explored surface, purple dots and red circles are objects and frontiers, respectively, and filled orange and cyan circles are selected next object or frontier. The planned and actual paths are shown as pink and navy curves, respectively. (Right) Visualization over the GT map of the agent's trajectory (navy curve), ground-truth object (white box), and ground-truth stopping area (pink area).</figcaption></figure>
<figure><img src="image_13.png" /><figcaption>Fig. S5: Visualization of an episode with navigation to a *counter* in an MP3D scene. (Left) RGB image seen by the agent. (Center) Our navigation maps built online. Grey pixels correspond to inflated obstacles, the green area is the explored surface, purple dots and red circles are objects and frontiers, respectively, and filled orange and cyan circles are selected next object or frontier. The planned and actual paths are shown as pink and navy curves, respectively. (Right) Visualization, over the GT map, of the agent's trajectory (navy curve), ground-truth object (green box), and ground-truth stopping area (pink area).</figcaption></figure>
<figure><img src="image_14.png" /><figcaption>Fig. S6: Real-world navigation episode, going to a *box*. (Left) Visualization of the agent's trajectory at different time steps. The agent starts by doing a 360° rotation to observe its surroundings and start building a map. It then navigates to the best frontier or object as selected by the model, avoiding obstacles on its path, until reaching the target object and stopping there. (Right) The constructed top-down 2D map at the end of the trajectory. Red circles correspond to frontiers while purple dots are objects. The green area represents the explored parts of the scene. Obstacles are in white, inflated in grey to indicate the non-navigable area.</figcaption></figure>