Title: CAPNAV: TOWARDS ROBUST INDOOR NAVIGATION WITH DESCRIPTION-FIRST MAPS
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Vision-Language Navigation, Multi-View Captioning, Attribute-Guided Retrieval, LLM, Clustering

Abstract: While many navigation instructions mention object categories, real-world commands often hinge on fine-grained attributes such as color, shape, and texture. Most Vision-and-Language Navigation (VLN) approaches are not designed for such descriptions: they loosely combine visual and textual features and rarely verify if the perceived object actually matches the stated attributes. We introduce CapNav, a description-first navigation framework that unifies attribute-based language grounding with visual perception to improve object localization. CapNav constructs a 3D instance-level map in which every object is annotated with both semantic embeddings and explicit attribute fields. These are generated through multiview captioning and viewpoint-consistent feature clustering using foundation models (DINOv2, CLIP). The resulting attribute-enriched descriptors are indexed for retrieval and queried through a multimodal large-language model to infer the location of the requested object. A subsequent navigation stage verifies attribute consistency before selecting a goal location. By combining visual grounding with semantic reasoning, CapNav can follow fine-grained, description-based instructions. Experiments in simulated indoor environments demonstrate that CapNav surpasses category-only baselines on multi-object navigation tasks under attribute ambiguity and produces interpretable navigation decisions grounded in structured captions.

TL;DR: CapNav unifies attribute-aware grounding and perception to navigate to precisely described objects, outperforming category-only methods under ambiguous, multi-object scenes.