MLFM: Multi-Layered Feature Maps for Richer Language Understanding in Zero-Shot Semantic Navigation

## Abstract

Abstract Recent progress in large vision-language models has driven improvements in language-based semantic navigation, where an embodied agent must reach a target object described in natural language. Yet we still lack a clear, language-focused evaluation framework to test how well agents ground the words in their instructions.
We address this gap by proposing LangNav , an open-vocabulary multi-object navigation dataset with natural language goal descriptions (e.g. ‘ go to the red short candle on the table ’) and corresponding fine-grained linguistic annotations (e.g., attributes: color=red, size=short; relations: support=on).
These labels enable systematic evaluation of language understanding. To evaluate on this setting, we extend multi-object navigation task setting to Language-guided Multi-Object Navigation ( LaMoN ), where the agent must find a sequence of goals specified using language.
Furthermore, we propose Multi-Layered Feature Map ( MLFM ), a novel method that builds a queryable, multi-layered semantic map from pretrained vision-language features and proves effective for reasoning over fine-grained attributes and spatial relations in goal descriptions.
Experiments on LangNav show that MLFM outperforms state-of-the-art zero-shot mapping-based navigation baselines.

## 1 Introduction

Semantic navigation is a rapidly growing sub-field in embodied AI [11] where an agent is tasked with navigating to a target object described using either an object category [1, 3, 44], image [21] or natural language descriptions [25, 34]. Each of these target specifications poses unique challenges. Among them, natural language is the most natural way for humans to express goals, yet it is also inherently ambiguous and context-dependent.
In this paper, we focus on natural language guided semantic navigation, where goals are specified using language descriptions, for example, ‘*Go to the white table lamp on the wooden corner table*’ ([Figure 1](https://arxiv.org/html/2507.07299v2#S1.F1)).

Prior efforts to build datasets with language descriptions [20] often rely on large vision-language models (VLMs) to extract attributes from images and large language models (LLMs) to merge them into sentences.
However, VLMs frequently hallucinate attributes [26], especially in cluttered or out-of-distribution scenes, introducing noise that biases both agent learning and evaluation.
To mitigate this, we propose the *LangNav* dataset derived from the Habitat Synthetic Scenes Dataset (HSSD) [19]. LangNav uses ground-truth attributes for objects, with all descriptions manually validated to avoid VLM errors. Furthermore, it provides fine-grained linguistic annotations–capturing object attributes (e.g. color, size, material) and spatial relations (e.g. ‘on’, ‘near’, ‘left’)–to enable systematic evaluation of language understanding.

Vision-and-language navigation (VLN) [2, 22, 33] differs from our setting in that it provides full route instructions (e.g., “*leave the bedroom, turn left, pass the painting, and stop at the sofa*”), which are mostly crowdsourced and rarely contain hallucinated attributes. By contrast, object navigation specifies only the goal object, coupling grounding with exploration. In real-world interaction, people rarely provide step-by-step paths; instead, they say, “*bring me the blue mug on the bedside table*”.
Goal-only language is therefore both *natural* and *challenging*: the agent must explore unseen space, decide when it has found the correct object, and remember what it has already observed.
Sequential tasks such as Multi-Object Navigation (MultiON) [37, 32] and GOAT-Bench [7, 20] capture this difficulty by revealing each target only after the previous one is reached.
We extend such task settings by proposing *Language-guided Multi-Object Navigation* (*LaMoN*) where each of three sequential goals is specified through a natural language description with different levels of specificity.
For instance, in [Figure 1](https://arxiv.org/html/2507.07299v2#S1.F1), depending on the level of specificity, the description can match just one or multiple target objects.
Success in LaMoN requires two key abilities: ($i$) building a semantic map memory while exploring, and ($ii$) querying that memory to reason about objects whose descriptions vary in granularity. To this end, we propose a novel method *Multi-Layered Feature Map* (*MLFM*) that constructs a multi-layered semantic map, leverages it to identify the goal object, semantically explores the environment and navigates via a path planner in an unseen environment.

Figure: Figure 1: Language-guided Multi-Object Navigation(LaMoN) requires an agent to navigate to multiple goals, described using descriptions (left). We evaluate fine-grained language understanding by tagging each description with attributes (*color*) and spatial relations (*support*) (right). There may be multiple positive matches (objects matching all attributes and relations in the instruction) and the agent is scored correct if it stops at any (check marks a match and cross marks a non-match).
Refer to caption: x1.png

In summary, our contributions are threefold:
(i) *LangNav*: an open-vocabulary multi-object navigation dataset with natural language goal descriptions and fine-grained linguistic annotations, enabling systematic evaluation of language understanding;
(ii) *LaMoN*: an extension of MultiON, where goals are specified using language descriptions at varying levels of specificity;
(iii) *MLFM*: a multi-layer semantic-mapping approach that preserves rich object features
and achieves improved performance over state-of-the-art zero-shot map-based semantic navigation baselines on the LangNav dataset.

## 2 Related Work

Semantic navigation.
In embodied AI, one of the basic semantic navigation tasks is the Object Navigation (ObjectNav), where an agent is required to navigate to an object specified by its category. While earlier benchmarks [1, 3] have used a closed set of object categories, HM3D-OVON [44] introduces a benchmark containing open-vocabulary categories to describe the goal.
Multi-Object Navigation (MultiON) [37] extends this single object navigation to a multi-object setting where multiple objects are made available to the agent in sequence. MultiON requires the agent to remember objects observed in the past so as to come back if needed, for efficiency. Another type of semantic navigation is Instance-Nav [21], where the agent is given an image of the goal object. While this task has its own challenges with respect to finding a specific object instance observed from a specific view angle, it does not reflect real-world scenarios where it is more likely for someone to describe an object of interest in words.
A more natural semantic navigation is where the goal object is described using natural language, a task called language-based InstanceNav [25, 34].
GOAT-Bench [20] was introduced to provide a benchmark on multi-object multi-modal navigation, combining different modalities (object category, image and language) to specify the goals.
Compared to GOAT-Bench, where the language descriptions are generated via a VLM and contain significant errors, our LangNav dataset contains manually validated language descriptions at varying levels of specificity as well as annotations for fine-grained linguistic tags.

Semantic maps.
A widely used representation in semantic navigation is a semantic map that serves as a memory of observations that the agent has made throughout its navigation. It has been shown to improve efficiency in the longer-horizon MultiON task [20] by remembering objects of importance. While some works [8, 14, 46] store explicit information such as object category in the map, more recent methods store implicit features in the map [17, 13, 9], enabling open-vocabulary memory representation. These features are often extracted from a large VLM such as CLIP [31].
VLMaps [17] uses pixel-level embeddings from LSeg [23], while VLFM [43] uses BLIP-2 [24] image features to build a 2D top-down map and compute semantic frontiers to explore the environment while performing navigation. Others build 3D maps [45, 15] which preserve geometric details of the objects and allow better reasoning, but are very costly to build and maintain, especially in long-horizon tasks.
In this work, we propose a *multi-layer map* that serves as a middle-ground between a 2D and a 3D grid map and contains stacked layers of 2D maps, providing more granularity and flexibility than a 2D map but avoids the cost of a full 3D grid map.

Evaluation benchmarks and datasets.
Current semantic navigation methods focus on task success, i.e. whether the agent is able to find a goal, but lack an evaluation framework to test language understanding. At the same time, in other fields such as 3D scene understanding and grounding, evaluating AI models against natural language understanding has gained a lot of interest. OpenLex3D [18] proposes a benchmark that evaluates open-vocabulary scene representation methods by introducing new label annotations, such as synonyms, depictions, visual similarity, and clutter, in existing scene datasets.
Eval3D [12] proposes an evaluation tool to evaluate 3D generative models to assess their geometric and semantic consistencies.
Agentbench [29] introduces a testbed to systematically evaluate LLMs acting as agents on their reasoning capabilities.
ViGiL3D [36] introduces a dataset for evaluating visual grounding methods with respect to a diverse
set of language patterns, such as object attributes, relationships between objects, target references, etc.
Motivated by these works, we introduce *LangNav*, a dataset containing fine-grained linguistic annotations to evaluate semantic navigation methods.

## 3 Task and Dataset

Task.
Our *Language-guided Multi-Object Navigation* (*LaMoN*) task is an extension of the MultiON task [37, 32] and the GOAT-Bench task [7, 20], where an agent is spawned in an unseen indoor environment and tasked with sequentially navigating to multiple goal objects.
While MultiON uses a closed-vocabulary category name to describe each goal, GOAT-Bench uses multiple modalities–image, category and open-vocabulary language descriptions–to describe each goal. In contrast, we set up our task in an open-vocabulary setting, specifying each goal using *language descriptions with different levels of specificity*.
Each episode contains *three* goals, disclosed to the agent one at a time.
Language descriptions in *LaMoN* describe objects with varying levels of specificity, e.g. ‘go to the couch’ vs. ‘go to the black couch’ vs. ‘go to the black three piece L-shaped sectional couch’.
This allows us to evaluate the agent’s capability to understand coarse vs fine-grained linguistic cues in the descriptions.
At every step, the agent takes as inputs egocentric RGB-D images, GPS and compass readings relative to the agent’s starting pose in the episode, and the language instruction for the current goal $g_{i}$, The agent takes one of four actions: move forward by 25 cm, turn left by 30° or turn right by 30°, and found. A goal is successful if the agent generates found within 1.5 m of the goal object and under 500 steps. The episode continues even when the agent fails to navigate to one goal.

### 3.1 Dataset

In this section, we introduce the *LangNav* dataset that implements the LaMoN task.
Among the prior datasets to evaluate various semantic navigation tasks, GOAT-Bench [20] is the closest to ours ([Table 1](https://arxiv.org/html/2507.07299v2#S3.T1)). However, due to the use of a VLM during creation without manual verification, their language goals contain significant errors (see [Section A.1](https://arxiv.org/html/2507.07299v2#A1.SS1)).

**Table 1: Comparison with prior datasets. LangNav contains open-vocabulary object+instance goals, annotated with fine-grained linguistic tags and language descriptions are manually verified and exist at different levels of specificity (*Specificity*), depending on which it can match with one or multiple correct target objects.**
| Datasets | Task | ObjectNav<br>goals | InstanceNav<br>language goals | Open-<br>vocab | Linguistic<br>tags | No significant<br>description<br>errors | Specif-<br>icity |
| --- | --- | --- | --- | --- | --- | --- | --- |
| OVMM[42] | Manipulation | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | - | - |
| ObjectNav[3, 41] | Object nav | $\times$ | $\times$ | $\times$ | $\times$ | - | - |
| HM3D-OVON[44] | Object nav | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | - | - |
| MultiON[37, 32] | MultiON | $\checkmark$ | $\times$ | $\times$ | $\times$ | - | - |
| OneMap[5] | Object nav, MultiON | $\checkmark$ | $\times$ | $\times$ | $\times$ | - | - |
| Goat-Bench[20] | Multi-(object+instance) nav | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ |
| *LangNav* | Multi-(object+instance) nav | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ |

Episode generation.
We use high-quality synthetic 3D scenes from HSSD [19].
For each episode we draw a random navigable start pose for the agent and choose three goals from the objects in the scene.
For each goal, we store the *language description*, *linguistic tags* along with the *viewpoints* or navigable positions around the goal (see [Section A.1](https://arxiv.org/html/2507.07299v2#A1.SS1) for detail).

Attribute descriptions.
We convert the ground-truth attributes available in HSSD into fluent object descriptions with GPT-4 [30].
The model is prompted with in-context examples [4, 27, 39] to form a coherent natural language description.
Next, we use GPT-4 to also tag every instruction with the linguistic cues [36] and release the annotations along with the dataset:
• Color—a color adjective in the description, e.g. ‘the brown chair’.
• Size—an explicit size cue, e.g. ‘the small candle’.
• Texture—a surface pattern or finish, e.g. ‘knitted bath mat’.
• State—a transient state that could change, e.g. ‘illuminated makeup mirror’.
• Number—a numeral in the description, e.g. ‘the cabinet with two doors’.
• Material—the substance an object is made of, e.g. ‘wooden shoe rack’.
• Modifier—any other adjectival qualifier, e.g. ‘the easy chair’.

Spatial relation descriptions.
We programmatically extract spatial relationships between a pair of objects from the ground-truth 3D bounding boxes in HSSD, following methodologies similar to prior works in text-conditioned 3D scene generation [6, 35] (see [Section A.1](https://arxiv.org/html/2507.07299v2#A1.SS1)).
We extract the following spatial relations:
• Egocentric directional–a cardinal direction relative to the agent’s orientation, e.g. ‘the plant to *your left*’.
• Allocentric–a spatial relation relative to other objects independent to the agent’s current pose and further sub-divided into:
(a) Support–an explicit support relation, e.g. ‘the potted plant on the console table’.
(b) Directional–a direction-dependent spatial arrangement which is not a support relation, e.g. ‘the mirror *above* the sink’.
(c) Proximity–a relation encoding the notion of metric or qualitative closeness, e.g. ‘the floor lamp *near* the bed’ or ‘the plant *next to* the couch’.
(d) Containment–an inclusion relationship, e.g. ‘the plush toy *inside* the crib’.

Statistics.
We generate two splits for our dataset: validation and test, with distinct set of target objects and distinct scenes. The validation split contains 932 episodes, each with three goals, totaling 2796 goal descriptions spanning 20 scenes. The test split contains 855 episodes, each with three goals, totaling 2565 goal descriptions spanning 15 scenes (see [Section A.1](https://arxiv.org/html/2507.07299v2#A1.SS1)).

## 4 Method

We propose *Multi-Layered Feature Map* (*MLFM*), a zero-shot semantic navigation method that incrementally builds a semantic map by storing visual features from a pretrained vision-language model [31]. At each step, MLFM queries the map by comparing stored features with the goal’s language embedding, generating a similarity heatmap over cells. If a likely target cell is found, the agent navigates to it via a path planner; otherwise, it continues exploring using a semantic exploration method VLFM [43].

Two phase navigation (EAE-E).
For navigation, we introduce a two-phase method called *EAE-E*.
At the beginning of an episode the map is still sparse and because it stores *features* rather than explicit class labels, its cosine scores can be noisy.
An open-vocabulary detector, e.g. YOLO-World [10], on the other hand, yields high-precision category predictions whenever the object is in view.
Hence, early on the detector is the more reliable signal while the map mainly proposes candidate locations that must later be verified by the detector. As exploration continues and the map densifies, trust gradually shifts toward the map.
Following this idea we introduce two-phase navigation:
($i$) Explore-And-Exploit (EAE). For the first $\mathcal{N}_{e}$ steps a location is accepted as the goal only if *both* the similarity map $\mathcal{S}$
and the detector agree, similar to consensus filtering [5]. Upon agreement the agent moves to that cell via the path planner; otherwise it resumes exploration.
($ii$) Exploit-only (E). For the remaining ($\mathcal{N}-\mathcal{N}_{e}$) steps it relies solely on the map, selecting the cell with highest similarity to thes goal description and navigates to it.

### 4.1 Mapping

The motivation behind using a multi-layered map is that it allows us to capture the vertical context without incurring the cubic memory cost of a full 3D voxel grid.
Though flattening everything into a single 2D plane is a commonly used approach, doing so merges features from stacked objects—small items vanish beneath larger ones—and discards the height cues needed for reasoning about spatial relations between objects.
We therefore adopt a *multi-layer* top-down map that stores only a handful of horizontal slices: granular enough to separate objects at different heights, yet linear in memory and update cost.

Figure: Figure 2: Method. (a) The agent takes as input the RGB image from which the map building (b) extracts learned visual embeddings and projects onto layers using the depth and camera pose inputs. The map is then queried based on the input instruction by employing one of three techniques (c)–*vanilla*, *VLM* or *RGraph*. Once the agent identifies a possible goal location, it navigates to it using a path planner (d). The agent activates the object detector as an additional signal during the initial phase (EAE) of the navigation.
Refer to caption: x2.png

To this end, we build a multi-layer feature map, $\mathcal{M}\in\mathbb{R}^{L\times h\times w\times f_{d}}$ where $L$ is the number of layers, each containing a 2D top-down map $m\in\mathbb{R}^{h\times w\times f_{d}}$ and $f_{d}$ is the feature dimension stored at each map cell. $h$ and $w$ correspond to the ($X$ x $Y$) space of the physical environment, while $L$ corresponds to its discretized height dimension $Z$.
[Figure 2](https://arxiv.org/html/2507.07299v2#S4.F2) presents a schematic of the map building process.
Given the RGB observation $I_{t}\in\mathbb{R}^{H\times W\times 3}$ at each step $t$, we extract patch-wise features $Fp_{t}\in\mathbb{R}^{H\times W\times n_{p}\times f_{d}}$ using CLIP [31] image encoder from SED [40]. We rely on a CLIP-based vision–language model to store features embedded in a joint image–text space, so that the map can be queried with natural language descriptions.
Here $n_{p}$ is the number of patches.
Using the depth observation $D_{t}\in\mathbb{R}^{H\times W}$ and the camera intrinsics, we then project the features $Fp_{t}$ onto a 3D point-cloud.
Next we project the 3D points onto the 2D plane. For each point $p_{i}$ in the point-cloud at the 3D location ($x_{i},y_{i},z_{i}$), we find the appropriate layer using the height $z_{i}$ of the point in the point-cloud, such that: $l_{i}=\left\lfloor\frac{h_{i}}{\Delta h}\right\rfloor$, where $l_{i}$ is the layer index and $\Delta h$ denotes the height range of each layer.

### 4.2 Querying

We implement three different querying techniques for MLFM to query the built map with the input goal description. Below we describe each technique.

MLFM-vanilla. This variant relies on the most basic querying technique to find the target on the multi-layer map by identifying candidates on the map where the cosine similarity between stored visual features and the query text is above a threshold and then selects the highest score as the target.

MLFM+VLM. In this variant, we use a large vision-language model (VLM) to query the relationship between objects. We first find the most likely locations of the target object and the reference object by computing the highest similarity scores for each. We then color the target object with ‘red’ and the reference object with ‘blue’. We additionally mark the agent location on the map with a ‘green’ square. We generate an image with image tiles corresponding to each layer of the map and indicate that tile 1 is at a lower height than tile 2, which is at a lower height than tile 3 and so on. We then prompt GPT-4 and ask it to infer the spatial relation between the ‘red’ object A and the ‘blue’ object B, from the agent’s viewpoint:
*“Infer the spatial relation between red A and blue B. If a green box is present, it indicates agent location. Decide relation from the agent’s viewpoint.”*.
We compare the inferred relation to the query relation by encoding them with a CLIP text encoder and computing the cosine similarity. If the score is above a threshold, we consider the target as found.

MLFM+RGraph.
In this variant we construct a *relation graph* (*RGraph*), with nodes representing objects and directed edges encoding spatial relations between object pairs (e.g., edge from A to B with label ‘above’ indicates ‘A is above B’). Each node corresponds to a candidate location of either the target or the reference object, identified from the multi-layered map if its similarity score with the text query exceeds a threshold. Along with the object identity, we associate each node with the visual features extracted from its corresponding map region.
Edges are introduced between two nodes if the objects are in close spatial proximity, determined by the Euclidean distance on the map. Each edge is labeled with one or more relations inferred from the object map locations, and stores the corresponding CLIP text features. Note that this supports multiple relations between A and B, since A can be near B and at the same time to the left of B.
In this way, the RGraph explicitly captures not only objects but also pairwise relations grounded in spatial structure and allows the agent to disambiguate which specific object pair is referenced (e.g. ‘the picture next to the cabinet’ vs. ‘the picture above the cabinet’). See [Section A.2.3](https://arxiv.org/html/2507.07299v2#A1.SS2.SSS3) for more details on the graph construction.

## 5 Experiments

In this section, we compare MLFM with state-of-the-art zero-shot semantic navigation methods on our LangNav dataset.

Implementation.
For all the baselines, we use an A* [16, 5] planner.
We use SED [40] patch features to build the multi-layer map and Yolo-World as an object detector in MLFM. For experiments reported in this section, our map has three layers and each 2D grid cell side maps to 6 cm in the physical space. Using 1$\times$A40 GPU, it takes $\sim$8 hours for evaluating our method on the full test split.
For our experiments, we report the mean performance over five runs with random seeds and observe a standard deviation of $\pm$2.5.

Evaluation metrics.
We use standard navigation metrics, the success rate (SR) and the success weighted by inverse Path Length (SPL) [1] to measure whether the agent can reach a target object successfully and how well the agent trajectory matches the shortest path to the target respectively. More specifically, we report SR against each of the linguistic features–attributes (*color, size*, etc.) and spatial relations (*support, egocentric directional*, etc.).

Baselines.
We focus on understanding how an explicit semantic map helps agents solve the LaMoNtask on the LangNav dataset, especially when the goal descriptions vary in linguistic attributes.
Accordingly, we compare MLFM with state-of-the-art *mapping-based* zero-shot navigation methods: VLMaps [17], VLFM [43], OneMap [5] and MOPA [32] (see [Section A.3](https://arxiv.org/html/2507.07299v2#A1.SS3)).
We have two additional baselines, VLFM-v2 and OneMap-v2, where we adapt the two phase-navigation EAE-E ([Section 4](https://arxiv.org/html/2507.07299v2#S4)), explore-and-exploit (EAE) followed by exploit-only (E) phases, for the two baselines, VLFM and OneMap.
Furthermore, we compare with a baseline EgoImageMap+VLM, where we store egocentric RGB images on the map, similar to Modular GOAT [7, 20]. We then use the two phase-navigation (EAE-E) and query GPT-5 with the stored images to select the goal.

### 5.1 Results on LangNav

Below we discuss agent performance on various aspects of language understanding. [Table 2](https://arxiv.org/html/2507.07299v2#S5.T2) shows that MLFM variants (rows 8-10) outperform the 2D map based baselines (rows 1-7) on overall SR and SPL, highlighting the effectiveness of the granular multi-layered representation over 2D top-down maps. This applies to fine-grained attribute understanding (with few exceptions), spatial understanding, as well as descriptions containing only goal category (+3.9% gain in row 8 over 6).

Multi-layered maps capture fine-grained linguistic cues better than 2D maps.
[Table 2](https://arxiv.org/html/2507.07299v2#S5.T2) shows that MLFM significantly outperforms the baselines (rows 8-10 vs. 1-7) in attribute understanding, especially on attributes–color, number, material and modifier–by +3.8%, +7.2%, +51.6% and +2.8% respectively (row 8 vs. 6). This indicates that the layered map represents visual features better than a 2D map, where the projected features get aggregated on a single layer, making it hard to distinguish intricate details about objects. Moreover, MLFM+RGraph outperforms MLFM-vanilla and MLFM+VLM on color, material and modifier attributes.
However, MLFM fails to identify texture of objects. This is mainly the limitation of the feature extractor (see ablation in [Section 5.3](https://arxiv.org/html/2507.07299v2#S5.SS3)). MLFM also fell short of OneMap-v2 on descriptions with the state attribute (see analysis is [Section 5.2](https://arxiv.org/html/2507.07299v2#S5.SS2)).

Two-phase navigation is effective in long-horizon tasks.
Poor performance in VLFM can be attributed to the reliance on the built map, even during the initial stages of navigation when the map may not contain the target object. This is evident from the improved performance of VLFM-v2 (row 5),
where we adapt the two-phase navigation, over the vanilla VLFM (row 2), even when both use 2D maps. We observe a similar trend in OneMap (rows 6 vs. 4).
This shows that our two-phase navigation (EAE-E) approach is extremely effective when dealing with longer-horizon tasks.

**Table 2: Performance. MLFM+RGraph outperforms others on overall metrics, most attributes and spatial relations. The MLFM variants do best on ‘cat’ and on ‘color’, ‘num’, ‘mat’ and ‘mod’ attributes. Note-*cat*:category, *tex*:texture, *num*:number, *mat*:material, *mod*:modifier, *ego-dir*:egocentric directional, *allo-dir*:allocentric directional, *supp*:support, *prox*:proximity, *cont*:containment.**
| Methods | Overall |  |  | Attribute understanding |  | Spatial relations understanding |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SR$\uparrow$ | SPL$\uparrow$ | cat |  | color | size | tex | num | mat | state | mod |  | ego-<br>dir | allo-<br>dir | supp | prox | cont |
| 1) VLMaps | 5.0 | 1.2 | 3.1 |  | 1.4 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 3.0 |  | 0.0 | 1.0 | 0.6 | 3.7 | 0.0 |
| 2) VLFM | 7.1 | 2.6 | 2.4 |  | 1.6 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 3.3 |  | 0.4 | 1.6 | 1.3 | 6.9 | 2.1 |
| 3) MOPA | 6.3 | 4.4 | 2.8 |  | 2.6 | 3.6 | 0.0 | 0.0 | 0.0 | 0.0 | 4.2 |  | 0.0 | 2.1 | 1.4 | 4.1 | 2.0 |
| 4) OneMap | 15.3 | 7.1 | 3.7 |  | 4.7 | 3.9 | 3.0 | 2.3 | 3.1 | 0.0 | 6.7 |  | 2.1 | 4.1 | 10.7 | 5.5 | 5.7 |
| 5) VLFM-v2 | 24.8 | 9.7 | 28.2 |  | 11.5 | 33.3 | 0.0 | 7.1 | 43.1 | 28.6 | 17.7 |  | 0.0 | 2.8 | 12.9 | 4.2 | 3.3 |
| 6) OneMap-v2 | 26.7 | 10.1 | 29.1 |  | 26.9 | 41.7 | 0.0 | 7.1 | 0.0 | 57.1 | 21.1 |  | 2.1 | 4.7 | 22.1 | 5.9 | 6.5 |
| 7) EgoImageMap+VLM | 35.1 | 10.9 | 33.7 |  | 34.8 | 39.5 | 33.3 | 11.7 | 53.1 | 17.8 | 35.0 |  | 34.6 | 20.2 | 33.3 | 22.2 | 17.5 |
| 8) MLFM-vanilla | 28.8 | 10.3 | 33.0 |  | 30.7 | 41.7 | 0.0 | 14.3 | 51.6 | 14.3 | 23.9 |  | 16.7 | 12.5 | 20.4 | 14.1 | 14.3 |
| 9) MLFM+VLM | 33.3 | 10.7 | 33.3 |  | 34.5 | 41.7 | 0.0 | 14.3 | 51.9 | 14.3 | 31.3 |  | 17.1 | 19.3 | 21.2 | 16.9 | 14.7 |
| 10) MLFM+RGraph | 39.5 | 14.9 | 33.6 |  | 34.9 | 41.7 | 0.0 | 14.3 | 53.8 | 14.3 | 37.6 |  | 37.7 | 20.7 | 34.2 | 32.5 | 21.1 |

Multi-layered maps capture spatial relations better than 2D maps.
[Table 2](https://arxiv.org/html/2507.07299v2#S5.T2) shows that *MLFM-vanilla* with its basic querying mechanism outperforms VLFM-v2 and OneMap-v2 (rows 8 vs. 5-6) across all spatial relation types. This reaffirms that the multi-layer map not only helps in attribute understanding, but the granular representation also helps in spatial understanding.

Spatial understanding benefits from enhanced representation and querying.
Both *MLFM+VLM* (row 9) and *MLFM+RGraph* (row 10), equipped with enhanced querying mechanisms, outperform *MLFM-vanilla* across all relation types, with *MLFM+RGraph* achieving the best performance. This underscores that spatial relationships can be effectively modeled as graphs, allowing for better reasoning capabilities.

VLMs reason better on raw egocentric images than top-down abstract projections.
*EgoImageMap+VLM* (row 7) achieves comparable performance as our best performing model, and outperforms *MLFM+VLM*. This indicates that the VLM is able to reason over egocentric RGB images better than top-down projections of image features (see [Section 5.2](https://arxiv.org/html/2507.07299v2#S5.SS2)). Moreover, it achieves the best results on descriptions with only goal category and with texture attribute.

### 5.2 Failure analysis

MLFM vs. OneMap-v2. To understand why OneMap-v2 performs better than MLFM-vanilla for descriptions with *state* attribute,
we compute the percentage of times the agent makes a wrong detection or identifies the wrong goal on the map or runs out of time.
First, we observe that there are more wrong detections (71.4% in MLFM-vanilla vs. 28.6% in OneMap-v2) than wrong goals on the map (14.3% in both). This indicates that performance could be improved by using a better object detector (confirmed by ablation in [Table 3](https://arxiv.org/html/2507.07299v2#S5.T3)).
Second, MLFM-vanilla and OneMap-v2 have the same wrong goal on map percentage (14.3%), indicating that this error is not due to using multi-layer map instead of 2D map. Moreover, an ablation in [Section 5.3](https://arxiv.org/html/2507.07299v2#S5.SS3) indicates that the performance in *state* attribute improves by using image-level features instead of patch-level features.

Overall MLFM failure.
Performing a similar failure analysis on the overall performance of MLFM-vanilla, we observe that 31.7% of the error cases are due to wrong detection and 17.0% are due to wrong goal identification on the map. The remaining 48.3% is due to the agent running out of time budget while exploring and 3% is where the agent identifies the goal but fails to reach it on time.

MLFM+VLM vs. MLFM+RGraph.
We find that the primary source of failure for *MLFM+VLM* arises from the bias of VLMs, which are trained on web-scale photo and video corpora dominated by egocentric perspectives. As a result, they infer spatial relations more reliably from egocentric images than from top-down maps. For instance, when presented with a top-down projection where A’s red blob appears above B’s blue blob, the VLM tends to predict ‘A is above B’, mirroring how it would interpret an egocentric image, rather than the intended relation A is ‘near’ or ‘next to’ B (see [Figure 3](https://arxiv.org/html/2507.07299v2#S5.F3)).
*MLFM+RGraph*, on the other hand, might fail to differentiate between ‘inside’, ‘above’, ‘below’ and ‘on top of’ if both the objects are projected on the same map layer, depending on the number of layers in the multi-layered map.

Figure: Figure 3: Comparisons showing that *VLM* struggles to reason on projected abstract features, often interpreting them as egocentric views (example 1). *RGraph* struggles distinguishing ‘inside’ from ‘above’ when both objects are projected onto the same map layer (example 3).
Refer to caption: x3.png

### 5.3 Ablations

To gain more insights into why MLFM fails to understand attributes like *texture* and *state*, we perform ablations ([Table 3](https://arxiv.org/html/2507.07299v2#S5.T3)) with open-vocabulary feature extractors and object detectors.

Patch vs. image vs. pixel features. When compared with their image-level counterparts (rows 1–2 vs. 3–4), SED-CLIP and BLIP-2 patches generally perform better on attribute understanding (with a few exceptions).
This can be attributed to the fact that when we store the same global feature vector in every layer (CLIP or BLIP image-level), the map gains no additional information.
This is evident for
*color*, *size*, *material*, and *modifier* cues ($+26.9\%,+25.0\%,+16.3\%,+3.1\%$, respective gains in row 2 over 4).
Image features, on the other hand, edge out patches on number and state attributes (rows 2 vs. 4), suggesting that a holistic view occasionally helps when counting instances or detecting object states.
Pixel features from LSeg perform worst overall (row 5), indicating that overly fine-grained features hinder object-level reasoning and dilute broader context.

BLIP-2 vs. CLIP.
BLIP-2 patches outperforms SED-CLIP patches on the *color* (+3.9%) and *modifier* (+1.0%) attributes (row 2 vs. 1). Additionally, BLIP-2 image outperforms CLIP on *texture* (+20.0%) and *number* (+21.4%) comprehension (row 4 vs. 3). These findings confirm BLIP-2’s stronger vision-language alignment for such properties.

**Table 3: Ablations. Patch outperforms image followed by pixel features. However, the choice of feature extractor and object detector might influence understanding of certain attribute types.**
| Object detector | Features | Type |  | Attribute understanding |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | color | size | texture | number | material | state | modifier |  |  |  |
| 1) YOLO-World [10] | SED [40] | patch |  | 30.7 | 41.7 | 0.0 | 14.3 | 51.6 | 14.3 | 23.9 |
| 2) | BLIP-2 [24] | patch |  | 34.6 | 41.7 | 20.0 | 14.3 | 16.3 | 14.3 | 24.9 |
| 3) | CLIP [31] | image |  | 11.5 | 8.3 | 0.0 | 0.0 | 0.0 | 28.6 | 17.1 |
| 4) | BLIP-2 [24] | image |  | 7.7 | 16.7 | 20.0 | 21.4 | 0.0 | 28.6 | 21.8 |
| 5) | LSeg [23] | pixel |  | 3.8 | 0.0 | 0.0 | 7.1 | 0.0 | 14.3 | 7.1 |
| 6) Grounding-DINO [28] | SED [40] | patch |  | 26.9 | 49.7 | 0.0 | 0.0 | 53.7 | 42.8 | 25.7 |

Grounding-DINO vs. YOLO-World.
To assess the influence of the open-set detector, we swap YOLO-World for Grounding-DINO inside MLFM (rows 1 vs. 6).
Grounding-DINO ends up stronger for specific attribute types, such as +8.0% for size, +28.5% for state and +1.8% for *modifier*.

Overall, we find that patch features are often better at capturing granular attribute cues than pixel or image features. However the choice of specific feature extractors and object detectors might heavily influence understanding of certain attribute types.

### 5.4 Results on GOAT-Bench

To demonstrate the effectiveness of our *MLFM+RGraph* method on real-world scans, we perform an experiment on the language goals from GOAT-Bench [20] and compare to three baselines from the GOAT-Bench paper – the zero-shot *Modular GOAT* [7] and two trained methods: *RL Skill Chain* and *RL Monolithic*.
MLFM+RGraph, being a zero-shot method is however only comparable to the Modular GOAT method, that builds a semantic map memory (containing object categories) as well as an instance-specific map memory (containing egocentric images and CLIP image features).
[Table 4](https://arxiv.org/html/2507.07299v2#S5.T4) shows that MLFM+RGraph outperforms the baselines
on the language goals of GOAT-Bench, with +2.7% in SR and +5.3% in SPL.

**Table 4: On GOAT-Bench, MLFM+RGraph outperforms the baselines on language goals.**
| Methods | trained | zero-shot | SR | SPL |
| --- | --- | --- | --- | --- |
| RL Skill Chain | $\checkmark$ | $\times$ | 16.3 | 7.4 |
| RL Monolithic | $\checkmark$ | $\times$ | 12.6 | 6.5 |
| Modular GOAT | $\times$ | $\checkmark$ | 21.5 | 16.2 |
| MLFM+RGraph | $\times$ | $\checkmark$ | 24.2 | 21.5 |

## 6 Conclusion

We extend the semantic navigation task to *LaMoN*, to evaluate attribute-aware and spatially-aware language understanding.
We propose *LangNav* dataset to implement this task, where episodes have sequences of goals, with manually verified descriptions and per-instruction linguistic tags.
We also propose a novel method, *Multi-Layered Feature Map* (*MLFM*), that preserves fine-grained visual detail in a multi-layer semantic map representation and we demonstrate that it improves fine-grained and spatial language grounding.
Experiments show that language understanding is still far from being solved and offer ample scope for new ideas to improve perception, memory, and language grounding.
We hope our findings provide useful insights to drive future research in this direction.

Limitations.
The LangNav dataset has several limitations including lacking complex linguistic structures such as coreference (“place *it* on the table”), negation (“a chair that is *not* black”), interaction-based action directives (“pick up”, “bring back”) and multi-object relations (“chairs around the dining table”) (see [Appendix B](https://arxiv.org/html/2507.07299v2#A2)). However, we leave this for future work as possible extensions to LangNav.

LLM usage. An LLM was used in the dataset generation process (see [Section 3.1](https://arxiv.org/html/2507.07299v2#S3.SS1) and [Section A.1.1](https://arxiv.org/html/2507.07299v2#A1.SS1.SSS1) for detail), in the EgoImageMap+VLM baseline (see [Section 4](https://arxiv.org/html/2507.07299v2#S4) and [Section A.2.1](https://arxiv.org/html/2507.07299v2#A1.SS2.SSS1)) and in the MLFM+VLM method (see [Section 4](https://arxiv.org/html/2507.07299v2#S4) and [Section A.2.2](https://arxiv.org/html/2507.07299v2#A1.SS2.SSS2)). We also sparingly used an LLM [30] for rewording some text in the paper.

##### Acknowledgement.

This work was funded in part by a CIFAR AI Chair, a Canada Research Chair, and NSERC Discovery grants. Infrastructure was supported by CFI JELF grants. TC was supported by the PNRR project Future AI Research (FAIR - PE00000013), under the NRRP MUR program funded by the NextGenerationEU.
We thank Jelin Raphael Akkara for sharing his viewpoint generation code to be used in this project.
We also thank Francesco Taioli and Austin T. Wang for helpful discussions and feedback.

Contributions.

Sonia Raychaudhuri created the LangNav dataset, implemented the MLFM codebase, re-implemented and integrated baseline methods, conducted all experiments reported in the paper, and contributed extensively to writing the paper.

Enrico Cancelli developed the initial codebase and performed preliminary experiments.

Tommaso Campari contributed to the refinement of ideas, coordinated the team, and was extensively involved in paper writing.

Lamberto Ballan, Manolis Savva and Angel X. Chang provided supervision, guidance throughout the project, and critical feedback on the manuscript.

## References

- Anderson et al. [2018a]
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al.
On evaluation of embodied navigation agents.
*arXiv preprint arXiv:1807.06757*, 2018a.
- Anderson et al. [2018b]
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel.
Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.
In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 3674–3683, 2018b.
- Batra et al. [2020]
Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans.
ObjectNav revisited: On evaluation of embodied agents navigating to objects.
*arXiv preprint arXiv:2006.13171*, 2020.
- Brown et al. [2020]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
*Advances in neural information processing systems*, 33:1877–1901, 2020.
- Busch et al. [2025]
Finn Lukas Busch, Timon Homberger, Jesús Ortega-Peimbert, Quantao Yang, and Olov Andersson.
One map to find them all: Real-time open-vocabulary mapping for zero-shot multi-object navigation.
In *IEEE International Conference on Robotics and Automation (ICRA)*, 2025.
URL [https://arxiv.org/abs/2409.11764](https://arxiv.org/abs/2409.11764).
- Chang et al. [2014]
Angel Chang, Manolis Savva, and Christopher D Manning.
Learning spatial knowledge for text to 3D scene generation.
In *Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 2028–2038, 2014.
- Chang et al. [2023]
Matthew Chang, Theophile Gervet, Mukul Khanna, Sriram Yenamandra, Dhruv Shah, So Yeon Min, Kavit Shah, Chris Paxton, Saurabh Gupta, Dhruv Batra, et al.
GOAT: Go to any thing.
*arXiv preprint arXiv:2311.06430*, 2023.
- Chaplot et al. [2020]
Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov.
Object goal navigation using goal-oriented semantic exploration.
*Advances in Neural Information Processing Systems*, 33:4247–4258, 2020.
- Chen et al. [2023]
Shizhe Chen, Thomas Chabal, Ivan Laptev, and Cordelia Schmid.
Object goal navigation with recursive implicit maps.
In *Proceedings of the International Conference on Intelligent Robots and Systems (IROS)*, 2023.
doi: 10.48550/arXiv.2308.05602.
- Cheng et al. [2024]
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan.
Yolo-world: Real-time open-vocabulary object detection.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 16901–16911, 2024.
- Deitke et al. [2022]
Matt Deitke, Dhruv Batra, Yonatan Bisk, Tommaso Campari, Angel X Chang, Devendra Singh Chaplot, Changan Chen, Claudia Pérez D’Arpino, Kiana Ehsani, Ali Farhadi, et al.
Retrospectives on the embodied AI workshop.
*arXiv preprint arXiv:2210.06849*, 2022.
- Duggal et al. [2025]
Shivam Duggal, Yushi Hu, Oscar Michel, Aniruddha Kembhavi, William T Freeman, Noah A Smith, Ranjay Krishna, Antonio Torralba, Ali Farhadi, and Wei-Chiu Ma.
Eval3D: Interpretable and Fine-grained Evaluation for 3D Generation.
In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2025.
doi: 10.48550/arXiv.2504.18509.
- Gadre et al. [2023]
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song.
CoWs on pasture: Baselines and benchmarks for language-driven zero-shot object navigation.
In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 23171–23181, 2023.
- Gervet et al. [2023]
Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot.
Navigating to objects in the real world.
*Science Robotics*, 8(79):eadf6991, 2023.
- Gu et al. [2024]
Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, Chuang Gan, Celso Miguel de Melo, Joshua B. Tenenbaum, Antonio Torralba, Florian Shkurti, and Liam Paull.
ConceptGraphs: Open-vocabulary 3D scene graphs for perception and planning.
In *IEEE International Conference on Robotics and Automation (ICRA)*, 2024.
- Hart et al. [1968]
Peter E Hart, Nils J Nilsson, and Bertram Raphael.
A formal basis for the heuristic determination of minimum cost paths.
*IEEE transactions on Systems Science and Cybernetics*, 4(2):100–107, 1968.
- Huang et al. [2023]
Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard.
Visual language maps for robot navigation.
In *IEEE International Conference on Robotics and Automation (ICRA)*, pages 10608–10615. IEEE, 2023.
- Kassab et al. [2025]
Christina Kassab, Sacha Morin, Martin Büchner, Matías Mattamala, Kumaraditya Gupta, Abhinav Valada, Liam Paull, and Maurice Fallon.
OpenLex3D: A New Evaluation Benchmark for Open-Vocabulary 3D Scene Representations.
*arXiv preprint arXiv:2503.19764*, 2025.
- Khanna et al. [2024a]
Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva.
Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation.
In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2024a.
- Khanna et al. [2024b]
Mukul Khanna, Ram Ramrakhya, Gunjan Chhablani, Sriram Yenamandra, Theophile Gervet, Matthew Chang, Zsolt Kira, Devendra Singh Chaplot, Dhruv Batra, and Roozbeh Mottaghi.
GOAT-bench: A benchmark for multi-modal lifelong navigation.
In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 16373–16383, 2024b.
- Krantz et al. [2022]
Jacob Krantz, Stefan Lee, Jitendra Malik, Dhruv Batra, and Devendra Singh Chaplot.
Instance-specific image goal navigation: Training embodied agents to find object instances.
*arXiv preprint arXiv:2211.15876*, 2022.
- Ku et al. [2020]
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge.
Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding.
In *Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 4392–4412, 2020.
- Li et al. [2022]
Boyi Li, Kilian Q. Weinberger, Serge J. Belongie, Vladlen Koltun, and René Ranftl.
Language-driven semantic segmentation.
In *Proceedings of the International Conference on Learning Representations (ICLR)*, volume abs/2201.03546, 2022.
- Li et al. [2023a]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
In *Proceedings of the International Conference on Machine Learning (ICML)*, pages 19730–19742. PMLR, 2023a.
- Li et al. [2021]
Weijie Li, Xinhang Song, Yubing Bai, Sixian Zhang, and Shuqiang Jiang.
ION: Instance-level object navigation.
In *Proceedings of the ACM International Conference on Multimedia*, pages 4343–4352, 2021.
- Li et al. [2023b]
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.
Evaluating object hallucination in large vision-language models.
In *Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)*, 2023b.
URL [https://arxiv.org/abs/2305.10355](https://arxiv.org/abs/2305.10355).
- Liu et al. [2021]
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen.
What Makes Good In-Context Examples for GPT-$3$?
*arXiv preprint arXiv:2101.06804*, 2021.
- Liu et al. [2024]
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al.
Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection.
In *Proceedings of the European Conference on Computer Vision (ECCV)*, pages 38–55. Springer, 2024.
- Liu et al. [2023]
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al.
Agentbench: Evaluating llms as agents.
*arXiv preprint arXiv:2308.03688*, 2023.
- OpenAI [2023]
OpenAI.
GPT-4 technical report, 2023.
- Radford et al. [2021]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *Proceedings of the International Conference on Machine Learning (ICML)*, pages 8748–8763. PMLR, 2021.
- Raychaudhuri et al. [2024]
Sonia Raychaudhuri, Tommaso Campari, Unnat Jain, Manolis Savva, and Angel X. Chang.
MOPA: Modular Object Navigation with Pointgoal Agents.
In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, pages 5763–5773, January 2024.
- Raychaudhuri et al. [2025]
Sonia Raychaudhuri, Duy Ta, Katrina Ashton, Angel X. Chang, Jiuguang Wang, and Bernadette Bucher.
Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation, 2025.
URL [https://arxiv.org/abs/2411.07848](https://arxiv.org/abs/2411.07848).
- Taioli et al. [2025]
Francesco Taioli, Edoardo Zorzi, Gianni Franchi, Alberto Castellini, Alessandro Farinelli, Marco Cristani, and Yiming Wang.
Collaborative Instance Navigation: Leveraging Agent Self-Dialogue to Minimize User Input.
In *Proceedings of the IEEE International Conference on Computer Vision (ICCV)*, 2025.
URL [https://arxiv.org/abs/2412.01250](https://arxiv.org/abs/2412.01250).
- Tam et al. [2025]
Hou In Ivan Tam, Hou In Derek Pun, Austin T. Wang, Angel X. Chang, and Manolis Savva.
SceneEval:Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesis.
*arXiv preprint arXiv:2503.14756*, 2025.
- Wang et al. [2024]
Austin T. Wang, ZeMing Gong, and Angel X. Chang.
ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Grounding.
In *Proceedings of the Conference of the Association for Computational Linguistics (ACL)*, 2024.
doi: 10.48550/arxiv.2501.01366.
- Wani et al. [2020]
Saim Wani, Shivansh Patel, Unnat Jain, Angel Chang, and Manolis Savva.
MultiON: Benchmarking semantic map memory using multi-object navigation.
*Advances in Neural Information Processing Systems*, 33:9700–9712, 2020.
- Wijmans et al. [2019]
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra.
DD-PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames.
In *Proceedings of the International Conference on Learning Representations (ICLR)*, 2019.
- Wu et al. [2023]
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong.
Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering.
In *Proceedings of the Conference of the Association for Computational Linguistics (ACL)*, 2023.
- Xie et al. [2024]
Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang.
SED: A simple encoder-decoder for open-vocabulary semantic segmentation.
In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 3426–3436, 2024.
- Yadav et al. [2022]
Karmesh Yadav, Santhosh Kumar Ramakrishnan, John Turner, Aaron Gokaslan, Oleksandr Maksymets, Rishabh Jain, Ram Ramrakhya, Angel X Chang, Alexander Clegg, Manolis Savva, Eric Undersander, Devendra Singh Chaplot, and Dhruv Batra.
Habitat challenge 2022.
[https://aihabitat.org/challenge/2022/](https://aihabitat.org/challenge/2022/), 2022.
- Yenamandra et al. [2023]
Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin S Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alexander Clegg, John M Turner, et al.
HomeRobot: Open-vocabulary mobile manipulation.
In *Proceedings of the Conference on Robot Learning (CoRL)*, pages 1975–2011. PMLR, 2023.
- Yokoyama et al. [2023]
Naoki Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, and Bernadette Bucher.
VLFM: Vision-language frontier maps for zero-shot semantic navigation.
Workshop on Language and Robot Learning, CoRL 2023, 2023.
Atlanta, GA, USA.
- Yokoyama et al. [2024]
Naoki Yokoyama, Ram Ramrakhya, Abhishek Das, Dhruv Batra, and Sehoon Ha.
HM3D-OVON: A dataset and benchmark for open-vocabulary object goal navigation.
In *Proceedings of the International Conference on Intelligent Robots and Systems (IROS)*, 2024.
doi: 10.48550/arXiv.2409.14296.
- Zhang et al. [2023]
Jiazhao Zhang, Liu Dai, Fanpeng Meng, Qingnan Fan, Xuelin Chen, Kai Xu, and He Wang.
3D-aware object goal navigation via simultaneous exploration and identification.
In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 6672–6682, 2023.
- Zhang et al. [2025]
Lingfeng Zhang, Xiaoshuai Hao, Qinwen Xu, Qiang Zhang, Xinyao Zhang, Pengwei Wang, Jing Zhang, Zhongyuan Wang, Shanghang Zhang, and Renjing Xu.
MapNav: A novel memory representation via annotated semantic maps for VLM-based vision-and-language navigation.
*arXiv preprint arXiv:2502.13451*, 2025.

## Appendix A Supplementary Material

In this supplementary document we
(i) present additional details of the LangNav corpus (Sec. [A.1](https://arxiv.org/html/2507.07299v2#A1.SS1));
(ii) provide more detail on the method (Sec. [A.2](https://arxiv.org/html/2507.07299v2#A1.SS2));
(iii) briefly describe the baselines used in the main paper (Sec. [A.3](https://arxiv.org/html/2507.07299v2#A1.SS3));
and
(iv) report further ablations that quantify the contribution of each component in our method (Sec. [A.4](https://arxiv.org/html/2507.07299v2#A1.SS4)).

### A.1 LangNav dataset

In this section we present more details about our LangNav dataset, including details on dataset statistics ([Table 5](https://arxiv.org/html/2507.07299v2#A1.T5)).

#### A.1.1 LLM for dataset generation

Forming coherent sentences from annotations.
We prompt GPT-4 [30] to form a sentence in natural language that would make sense to a human:

> “You will be provided with a list of attributes, and your task is to convert them to a free-flowing natural language sentence in English as spoken by native speakers. Drop brand names and all capital lettered words from the text.”

We also provided a list of five examples to improve GPT’s performance via in-context learning.

Extracting attributes from sentences for annotations.
We use GPT-4 to also extract attributes from a language sentence which we then store as the fine-grained linguistic annotations, so that we can evaluate agent’s performance on language understanding:

> You are given a sentence with the description of an object that someone is supposed to find in a scene. Your goal is to extract the attributes used to describe the object: <input_sentence>. Return the output in a JSON format according to the following format:

#### A.1.2 Episode generation

For each episode in the LangNav dataset, we first draw a random navigable start pose for the agent, then choose an object category that satisfies the following constraints:
(i) at least one navigable viewpoint lies within 1.5 m of the object;
(ii) the start pose and all three goals are on the same floor;
(iii) every goal is reachable from the start pose;
(iv) each goal is at least 2 m from the start pose;
(v) the geodesic-to-Euclidean distance ratio from the start pose to each goal exceeds 1, ensuring non-trivial navigation;
(vi) the geodesic distance from the start pose to the first goal is at least 2.5 m.
For each goal, we also store ‘viewpoints’ or navigable positions around the goal object, such that navigating near to any of those will be considered successful (see [Section A.1.5](https://arxiv.org/html/2507.07299v2#A1.SS1.SSS5)).

**Table 5: Dataset statistics. LangNav contains val and test splits, each with distinct sets of scenes and goal object categories. The goal descriptions might contain one or more linguistic features.**
|  | Validation | Test | Overall |
| --- | --- | --- | --- |
| scenes | 20 | 15 | 35 |
| episodes | 932 | 875 | 1807 |
| distinct object categories | 19 | 12 | 31 |
| distinct object instances | 160 | 154 | 314 |
| goal descriptions | 2796 | 2625 | 5421 |
| descriptions with attributes/relationships | 1659 | 2181 | 3840 |
| total linguistic features | 2534 | 3524 | 6058 |
| unique linguistic feature values | 136 | 99 | 235 |

#### A.1.3 Spatial relation descriptions

Here we describe in detail the procedure used to extract spatial relationships between a pair of objects from the ground-truth 3D object bounding boxes in HSSD.
For *egocentric directional relations*, we take two objects with non-overlapping bounding boxes and compute the centroids of each bounding box. We transform them into the camera coordinate frame using the provided camera extrinsics. We then project these centroids into the image plane using the camera intrinsics and compare their horizontal pixel coordinates to assign “left” or “right” relations. This ensures that directional relations are defined relative to the agent’s pose and viewpoint.
For *allocentric relations*, we directly compare the relative positions of object bounding boxes in the world coordinate system. For allocentric *directional relations*, we cast rays from the object’s bounding box in the direction of the relation (vertical direction for “above” and “below”) to retrieve the object with hits(^1^11Code adapted from Habitat-Lab [https://github.com/facebookresearch/habitat-lab/blob/5e0d63838cf3f6c7008c9eed00610d556c46c1e3/habitat-lab/habitat/sims/habitat_simulator/sim_utilities.py#L724](https://github.com/facebookresearch/habitat-lab/blob/5e0d63838cf3f6c7008c9eed00610d556c46c1e3/habitat-lab/habitat/sims/habitat_simulator/sim_utilities.py#L724)). For instance, object $A$ is “above” $B$, if rays from $A$’s bounding box when shot downwards hit $B$’s bounding box.
For allocentric *support relations*, we utilize the ground-truth support data available for the objects in HSSD.
Allocentric *proximity relations* are derived by measuring the minimum Euclidean distance between the surfaces of two bounding boxes. We adopt a tiered thresholding scheme - objects within 0.0–0.2 meter are labeled as “next to” each other, while objects within 0.2–1.0 meter are labeled as “near” each other. This design captures both immediate adjacency and a more relaxed proximity. Finally, *containment relations* are identified when the bounding box of one object is fully enclosed within that of another across all three spatial dimensions.
Our procedure benefits from the availability of ground-truth 3D bounding boxes in synthetic scenes, allowing us to generate accurate and consistent spatial relation annotations. The use of ray-casting provides a principled way to capture directional relations and the surface-to-surface distances enable a faithful encoding of adjacency to capture proximity relations. The pipeline is fully automatic and scalable, making it possible to construct a large dataset of spatial relations without requiring manual annotation.

#### A.1.4 Understanding the dataset.

Why the need for a new dataset?
The closest to our dataset is the GOAT-Bench [20] which introduced a multi-object navigation dataset that combines ObjectNav and InstanceNav goal descriptions. In contrast to GOAT-Bench, our dataset contains language descriptions at varying amounts of specificity, such that one or multiple target locations could be a match. We also include annotations for linguistic tags for each description which allows for fine-grained language evaluation. Moreover, GOAT-Bench contains several errors propagated from the VLM, which we mitigate by using ground-truth object annotations and also manually validating them.

The dataset creation pipeline in GOAT-Bench derives object attributes by prompting a pretrained BLIP-2 model, and this reliance on automatic extraction introduces noticeable noise in the captions. Typical issues include ([Figure 4](https://arxiv.org/html/2507.07299v2#A1.F4)) ($i$) *partial matches*, where the instruction fits the target only in part; ($ii$) *hallucinations*, where attributes mentioned in the text are absent from the scene; and ($iii$) *mesh artefacts*, where incomplete geometry in HM3D-Sem hides the object, rendering the description misleading; ($iv$) *reference to object bounding box*, where the goal descriptions contain references to object bounding box, which the agent might not have access to during navigation (e.g. ‘…region defined by the bathtub bounding box’); ($v$) *spatial errors*, where the spatial relationship is wrong between objects (e.g. ‘blanket near the bed on the left side’ when it is clearly towards the foot of the bed in the figure).
To obtain a statistic on how often these five error types occur, we randomly sample 100 goals from GOAT-Bench and manually inspect the descriptions along with the corresponding goal images. We found that only 33% of them were accurate whereas the rest had errors - 22% were hallucination errors, 7% were mesh artefact errors, 3% were partial match errors, 23% were spatial errors and 12% were due to reference to object bounding box or the image frame, totaling to 67% error. The most frequent error types were spatial errors and hallucination errors.
Moreover, in GOAT-Bench the language goals are generated to admit a single matching target, and they were not systematically validated to rule out additional scene objects that satisfy the same description.

Figure: Figure 4: Language descriptions in Goat-Bench contain errors propagated from the BLIP-2 model. This figure shows examples for *partial match*, *hallucination*, *mesh artifact*, *spatial error* and *reference to object bounding box* errors.
Refer to caption: x4.png

We mitigated these errors by first using scenes and objects from HSSD dataset. The scenes are synthetic and hence free of mesh-related issues. The dataset also provides ground-truth categories and attributes for every object; we pass this structured metadata to GPT to obtain fluent natural-language instructions ([Section A.1.1](https://arxiv.org/html/2507.07299v2#A1.SS1.SSS1)) that remain anchored to the ground truth. All generated sentences are then manually reviewed and, where necessary, corrected to eliminate any residual LLM errors (see [Figure 5](https://arxiv.org/html/2507.07299v2#A1.F5)). Finally, we tag each instruction with the linguistic cues used in our benchmark (colour, size, material, state, and so on) ([Section A.1.1](https://arxiv.org/html/2507.07299v2#A1.SS1.SSS1)) and include these annotations with the dataset—information that no previous semantic-navigation corpus offers.
Moreover, all possible correct goal locations corresponding to a single description are added in the episode to account for multiple matches.

Figure: Figure 5: In LangNav, we use objects and attributes available in HSSD synthetic scenes, thus producing error free language descriptions.
Refer to caption: x5.png

#### A.1.5 Viewpoint generation

Similar to the ObjectNav task [3], we store *viewpoints* or navigable positions around the goal object, such that navigating near to any of those will be considered successful. To generate candidate viewpoints around an object, we use depth observations together with the object’s 3-D position and dimensions to obtain viewpoints with a clear line of sight to the target object. First, we trace the object’s non-navigable boundary by sweeping a full $360^{o}$ and recording every transition point from free space to obstacle.
Around each boundary point we then place radial samples at uniform intervals out to 1.5 m.
Each sample is accepted as a valid viewpoint only if it meets two criteria: the location is navigable, and—after orienting the agent toward the object—no intervening obstacle blocks the view. We describe the full algorithm below.

Figure: Algorithm 1 generate_view_points

### A.2 MLFM method

#### A.2.1 EgoImageMap+VLM baseline

For the EgoImageMap+VLM baseline method, we prompt a GPT-5 model with the following prompt to infer spatial relations between two objects in an egocentric RGB image:

> “You are a vision-language assistant. You are given a query and a group of images. Each group contains several images. For every image in each group, return whether the object in the query is clearly visible. Respond strictly in JSON. Group numbers are integers (0, 1, 2, …). Use lowercase booleans (‘true‘ / ‘false‘). Do not include explanations, only the JSON.”

#### A.2.2 MLFM+VLM

In our MLFM+VLM variant, we prompt GPT-5 to infer the most likely relation between two objects from a list of images:

> “Your task is to infer the most likely relation between object A (colored in red) and object B (colored in blue) from the list of images. Note that the images are top-down map projections and image 1 is physically at a lower height than image 2 which is lower than image 3 and so on. In these images, you will find a green box, denoting the agent location. Return a json with "relation" and "rationale" fields. Follow these rules. Use Euclidean proximity for near/next to (next to = very close, non-overlapping; near = close but not touching). If green box for the agent is present, decide LEFT/RIGHT relative to the ray from agent toward the midpoint between A and B. Consider the nearest blobs for A and B to the agent. Be concise and avoid extra prose.”

#### A.2.3 MLFM+RGraph

In this section, we provide more detail into how we construct the MLFM+RGraph.
The agent incrementally builds and updates the relation graph during navigation. As it encounters new views, nodes may be added or merged, and edges updated if new relational evidence is observed. Importantly, since objects may extend over multiple adjacent grid cells in the map, we collapse these into a single graph node and record the set of grid cells spanned by the object ($map_span$). This aggregation serves two purposes: (1) it yields a segmentation-like mask of the object’s spatial extent, and (2) it allows more accurate reasoning about its relation to nearby objects, which can now be computed relative to the object’s full footprint rather than a single cell.

To infer relations between two nodes, we apply geometric rules on their $map_span$, which records the grid cells occupied by each object as $[x,y,z]$ coordinates ($(x,y)$ for 2D grid location and $z$ for map layer). Vertical relations (above, below) are determined by comparing $z$ values, while proximity is inferred if objects are within $\sim$20 grid cells (grid resolution is 6 cm). Using the agent’s pose, we transform object locations into the egocentric frame and their horizontal displacement then yields left or right relations. Containment is inferred by checking whether A’s boundaries lie mostly within B’s boundaries. We record all relations satisfying these rules for each node pair and store their corresponding text features in the edge. These geometric rules serve as a deterministic first step for relation inference; these can be replaced with a learned model trained on collected episodes to resolve ambiguous cases and support unseen relation types.

During the exploitation phase, the agent uses the RGraph to answer relational queries. Given a text query, we encode it using the CLIP text encoder and compute its cosine similarity with the stored graph features. Specifically, for each edge, we get (a) visual features for the source node, (b) visual features for the sink node and (c) the edge text features. The agent selects the candidate target node whose triplet score (target-relation-reference) is highest and exceeds a confidence threshold.

This formulation allows the agent to reason jointly over object semantics and spatial relations captured in the RGraph. Moreover, comparing CLIP text embeddings of the relational word allows the model to generalize to unseen but semantically similar relations (e.g., ‘under’ and ‘below’, ‘within’ and ‘inside’, ‘beside’ and ‘next to’). In effect, RGraph provides a structured interpretable memory that grows with exploration and enables reasoning over spatial relations during exploitation.

### A.3 Semantic navigation baselines

In this section, we briefly describe the baselines against which we compare our method in the main paper.

VLMaps [17] store LSeg [23] pixel-level embeddings in a 2D top-down grid map, along with their 3D location. They average the pixel embeddings when multiple pixels are projected onto the same grid and also across multiple views.

VLFM [43] builds a 2D value map that stores BLIP-2 [24] similarity scores of the observed images and object category, and uses it to semantically explore the environment. It achieves SOTA performance in the ObjectNav task.

MOPA [32] iteratively builds a 2D category map after detecting objects in the frame, randomly explores the environment and navigates to the goal once found. We replace their PointNav [38] with the A* planner for fair comparison.

OneMap [5] builds a 2D feature map with SED [40] patch features by using a confidence-based fusion mechanism.
They employ an open-set object detector YOLO-World [10] along with the map to identify the target and use an A* planner to navigate.

### A.4 Ablations

In this section, we report ablations to determine how the different components contribute to MLFM’s performance. We ablate on grid resolution in [Section A.4.1](https://arxiv.org/html/2507.07299v2#A1.SS4.SSS1), followed by EAE percentage in [Section A.4.2](https://arxiv.org/html/2507.07299v2#A1.SS4.SSS2), and the contribution of map and object detector in [Section A.4.3](https://arxiv.org/html/2507.07299v2#A1.SS4.SSS3).

#### A.4.1 Effect of grid resolution and number of layers

**Table 6: Varying map resolution and number of layers in our multi-layered map contributes towards MLFM’s performance.**
|  | cm per 2D grid cell | Number of layers | SR$\uparrow$ | SPL$\uparrow$ |
| --- | --- | --- | --- | --- |
| 1) | 6 | 1 | 29.7 | 9.3 |
| 2) |  | 2 | 37.0 | 14.1 |
| 3) |  | 3 | 39.5 | 14.9 |
| 4) |  | 4 | 38.5 | 13.8 |
| 5) |  | 5 | 27.6 | 10.1 |
| 6) | 10 | 3 | 25.1 | 9.6 |

[Table 6](https://arxiv.org/html/2507.07299v2#A1.T6) reports an ablation on the two key map hyper-parameters: the number of height layers $L$ and the $(x,y)$ resolution of each slice. Success rises as $L$ grows from one to three layers, but degrades when a fourth layer is added. Beyond three slices large objects become fragmented, making them harder to localise; we therefore set $L=3$ in all main experiments.
Keeping $L$ fixed, performance improves when we refine the $(x,y)$-grid—i.e. when the centimetres-per-cell value decreases (rows 1 vs. 6).
A finer resolution allows the map to preserve small objects’ geometry and thus boosts the agent’s success rate.

#### A.4.2 Effect of explore-and-exploit (EAE) percentage

[Table 7](https://arxiv.org/html/2507.07299v2#A1.T7) varies the share of the episode spent to Explore-and-Exploit (EAE) versus Exploit-only (E). Overall performance peaks when the agent spends the first $40\%$ of its time budget in EAE and the remaining $60\%$ in E: both *Success Rate* ($SR$) and *Success weighted by Path Length* ($SPL$) achieve their highest values under this $40\!:\!60$ split.
The table also reports the error composition. As the EAE portion grows, the *wrong-goal-on-map* rate declines, indicating that extended exploration yields a more accurate map. However, longer exploration leaves less budget for exploitation, so *out-of-time* failures increase once the EAE share exceeds $40\%$.

**Table 7: The agent achieves the best performance when it spends 40% in Explore-and-Exploit (EAE) and the remaining time in Exploit-only phase.**
| EAE<br>(%) | SR$\uparrow$ | SPL$\uparrow$ | Error$\downarrow$ |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Wrong detection | Wrong goal on map | OOT |  |  |  |
| 20 | 36.7 | 11.9 |  | 25.1 | 26.0 | 48.9 |
| 40 | 39.5 | 14.9 |  | 31.7 | 17.1 | 51.2 |
| 60 | 36.6 | 9.8 |  | 28.7 | 15.5 | 55.8 |
| 80 | 33.3 | 9.6 |  | 28.5 | 8.2 | 63.3 |

#### A.4.3 Role of map and object detector in MLFM

**Table 8: MLFM achieves the best result when both the map and the object detector are used.**
| Map | Object detector | SR$\uparrow$ | SPL$\uparrow$ |
| --- | --- | --- | --- |
| $\checkmark$ | $\times$ | 32.8 | 14.1 |
| $\times$ | $\checkmark$ | 29.0 | 8.6 |
| $\checkmark$ | $\checkmark$ | 39.5 | 14.9 |

We next assess the individual contributions of the object detector and the multi-layer map. [Table 8](https://arxiv.org/html/2507.07299v2#A1.T8) compares three variants: map only, detector only, and the full system that combines both.
Using both achieves the best performance followed by the *map-only* version followed by *detector-only* version.

## Appendix B Possible future work

The language in LangNav can be improved to include complex linguistic structures—coreference (e.g., “place *it* on the table”) and negation (“a chair that is *not* black”) and action directives beyond navigation verbs (e.g. “pick up”, “bring back”).
Our dataset can be further augmented with additional kinds of spatial relations such as egocentric proximity (e.g. “the chair near you”), allocentric relations based on object placement (e.g. “the chair in the corner of the room”) or relational with respect other instances of the same objects (e.g. “the taller plant”). We leave this for future work.
Our dataset generation approach makes a few assumptions. Bounding boxes only approximate true object geometry, which can lead to mislabels for irregularly shaped objects (e.g., a table placed in front of an L-shaped sofa may be incorrectly labeled as “inside” the sofa because its bounding box lies within that of the sofa). Fixed proximity thresholds (0.2 meter) may not generalize across scales (e.g., “near” may imply a few centimeters for tabletop objects but several meters in an office setting when describing “the water cooler near the meeting room”). Future work could incorporate more perceptually grounded measures, such as learned priors for spatial relations or scene-normalized distance metrics (e.g., defining “near” as within 10% of a room’s longest dimension).

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. What is the performance of recent end-to-end navigation foundation models on the LaMoN benchmark?

2. The paper presents the LangNav dataset—does this dataset benefit the training of navigation models on other object navigation benchmarks?

3. Given that obstacle sizes vary across different object types, how to define adaptive horizontal slices and construct a multi-layer feature map suitable for different scenes?

4. What is the decision frequency of the proposed MLFM approach?
soundness: 3
strengths: 1. The proposed LaMoN benchmark is well-motivated with clearly presented details. Since most zero-shot object navigation (ZSON) approaches are still evaluated and compared on limited object categories and coarse-grained descriptions, this benchmark offers a superior alternative.
2. The paper presents a novel modular approach, MLFM, which is generalizable to different object navigation benchmarks including LaMoN and GOAT, and achieves competitive results on both.
3. The paper conducts detailed failure analysis and in-depth ablation studies on the impact of different types of image features in constructing the Multi-Layer Feature Map (MLFM) and the map query scheme.
summary: This paper introduces LaMoN, a novel object navigation benchmark designed to address the limitations of existing ObjectNav benchmarks. It achieves this improvement primarily through two innovations: (1) It enriches the diversity of object descriptions by incorporating linguistic tags, allowing for more nuanced object definitions. (2) It integrates spatial relation descriptions for target objects, making the task more aligned with real-world navigation scenarios. These two features effectively narrow the gap between simulated benchmarks and practical applications, while also providing a more comprehensive evaluation platform for object navigation methods. Additionally, the paper proposes MLFM, a modular-based approach tailored for the LaMoN benchmark. Experimental results show that MLFM outperforms multiple baseline methods on ObjectNav tasks.
weaknesses: 1. The paper lacks essential comparisons with state-of-the-art navigation foundation models (e.g., NaViD[1], NaViLA[2], StreamVLN[3]) that are capable of handling general navigation tasks.

2. The contributions and differences between the proposed LaMoN benchmark and recent object navigation benchmarks (e.g., DOZE[4]) are not clearly justified.

3. The performance of MLFM in real-world scenarios remains unclear.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: See weakness.
soundness: 3
strengths: 1. Manual validation eliminates VLM hallucinations (a major flaw of GOAT-Bench), and fine-grained tags enable dimensional evaluation of language understanding.

2. By requiring sequential language-specified goals without path instructions, LaMoN’s task better simulates real human-robot interaction than step-by-step VLN tasks.

3. The multi-layer map avoids 3D’s high memory cost and 2D’s height information loss.
summary: This paper addresses the lack of language-centric evaluation in embodied semantic navigation by proposing three core contributions: (1) LangNav, an open-vocabulary multi-object navigation dataset with manually validated natural language goal descriptions and fine-grained linguistic annotations (attributes like color/size, spatial relations like "on/near"); (2) LaMoN, a task extending MultiON to require sequential navigation to language-specified goals with varying specificity; (3) MLFM, a multi-layered feature map method that balances 2D map efficiency and 3D height information, with three query variants (vanilla/VLM/RGraph) for fine-grained reasoning. Experiments on LangNav show MLFM+RGraph outperforms SOTA zero-shot baselines, and it generalizes to GOAT-Bench.
weaknesses: 1. Dataset limitations undermine generalizability: (1) Scenes are restricted to synthetic HSSD and semi-real GOAT-Bench. This paper did not test on physical environments (e.g., real apartments), where lighting/occlusions differ sharply from synthetic data. (2) Language is overly simple: no complex structures (coreference, negation) or action directives (e.g., "pick up"), making it irrelevant to real-world tasks like home service.

2. MLFM’s attribute understanding is incomplete: (1) Texture attribute success rate remains 0% (Table 2/3), but the paper only attributes it to feature extractors without proposing solutions or deeper analysis (e.g., fine-tuning CLIP on texture datasets). Moreover, state attribute (e.g., "illuminated mirror") underperforms OneMap-v2 (42.8% vs. 57.1%), but the failure analysis only simply mentions "detector issues" without detailed debugging. Better provide some experiments to discuss the issues mentioned above.

3. RGraph lacks adaptability: (1) Spatial relations are inferred via hand-crafted rules (e.g., Euclidean distance <0.2m = "next to"), not learned from data. This fails in scenes where "near" has different scales (e.g., 0.5m is "near" for a shelf but "far" for a table). (2) Cannot distinguish overlapping relations on the same layer (e.g., "inside" vs. "above" when two objects project to the same layer), with no workaround proposed.

4. Experiment scope is narrow: No comparison to state-of-the-art LLM-driven navigation methods (e.g., GPT-4V + embodied agents, 2024) – it’s unclear if MLFM is competitive with recent approaches. Besides, MLFM’s memory/compute cost vs. 2D/3D maps is unreported, which is critical for real-time robot deployment.

5. Some related and important works are missing citations: [1] NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigation [2] Learning Active Camera for Multi-Object Navigation [3] Mo-ddn: A coarse-to-fine attribute-based exploration agent for multi-object demand-driven navigation

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. I am wondering whether the paper positions itself as an evaluation benchmark for open-vocabulary navigation, given that it provides only validation and test datasets.
2. In Section 4.2, the paper explains that MLFM+VLM is used to query relationships between objects, while MLFM+RGraph constructs a relational graph. Why isn’t the relationship queried directly from MLFM+RGraph? This seems like a chicken-and-egg problem—does the RGraph get constructed dynamically during the process?
soundness: 3
strengths: Overall, the problem setting is both realistic and practical, as it aligns closely with how users naturally specify sequential goals through language instructions.
summary: This paper introduces an open-vocabulary multi-object navigation dataset featuring natural language goal descriptions. It extends the traditional multi-object navigation task to a new setting, where an agent must locate a sequence of goals described in language. The authors further propose a novel approach that constructs a queryable, multi-layered semantic map using pretrained vision-language features, demonstrating its effectiveness in reasoning about fine-grained attributes and spatial relationships expressed in goal descriptions.
weaknesses: In this setting, the task involves finding three sequential goals. I am curious about how this number was determined—have you experimented with different numbers of goals or tested the model’s performance under varying sequence lengths? What is the intuition behind choosing three? Additionally, it might be beneficial to provide a training dataset to further support the research community and enable more comprehensive exploration of this task.


Decision: Reject