Semantic Mapping in Indoor Embodied AI - A Survey on Advances, Challenges, and Future Directions
Authors: Sonia Raychaudhuri
Angel X Chang
Venue: Accepted by TMLR ()

Abstract: Intelligent embodied agents (e.g. robots) need to perform complex semantic tasks in unfamiliar environments. Among many skills that the agents need to possess, building and maintaining a semantic map of the environment is most crucial in long-horizon tasks. A semantic map captures information about the environment in a structured way, allowing the agent to reference it for advanced reasoning throughout the task. While existing surveys in embodied AI focus on general advancements or specific tasks like navigation and manipulation, this paper provides a comprehensive review of semantic map-building approaches in embodied AI, specifically for indoor navigation. We categorize these approaches based on their structural representation (spatial grids, topological graphs, dense point-clouds or hybrid maps) and the type of information they encode (implicit features or explicit environmental data). We also explore the strengths and limitations of the map building techniques, highlight current challenges, and propose future research directions. We identify that the field is moving towards developing open-vocabulary, queryable, task-agnostic map representations, while high memory demands and computational inefficiency still remaining to be open challenges. This survey aims to guide current and future researchers in advancing semantic mapping techniques for embodied AI systems.

Published in Transactions on Machine Learning Research (August/2025)
Semantic Mapping in Indoor Embodied AI A Survey on Advances, Challenges, and Future Directions
Sonia Raychaudhuri Simon Fraser University
sraychau@sfu.ca
Angel X. Chang
angeIn@ssfu.cu
Simon Fraser University
Alberta Machine Intelligence Institute (Amiis)
Reviewed on OpenReview: https: / net/ forum? id= US Q38RG6G
Abstract
Intelligent embodied agents (e.g. robots) need to perform complex semantic tasks in unfamiliar environments. Among many skills that the agents need to possess, building and maintaining a semantic map of the environment is most crucial in long-horizon tasks. semantic map captures information about the environment in a structured way, allowing the agent to reference it for advanced reasoning throughout the task. While existing surveys in embodied AI focus on general advancements or specific tasks like navigation and manipulation, this paper provides comprehensive review of semantic map-building approaches in embodied AT, specifically for indoor navigation. We categorize these approaches based on their structural representation (spatial grids, lopological graphs, dense geomelric or hybrid maps) and the type of information they encode (implicit features or explicit environmental data). We also explore the strengths and limitations the map building tochniques, highlight current challenges, and propose future research directions. We identify that the field moving towards developing open-vocabulary, queryable, task-agnostic map representations, while high memory demands and computational inefficiency still remaining to be open challenges. This survey aims to guide current and future researchers in advancing semantic mapping techniques for embodied AT systems.
1 Introduction
Intelligent agents, whether physical robots virtual embodied systems, must operate in complex, unstructured environments where effective behavior demands more than just low-level sensing and actuation. To act meaningfully, agents must form structured internal representations that link perception to reasoning and decision-making, Semantic maps serve this role by encoding not only spatial geometry but also high-level semanntcs of the environment (object categories, affordances, etc.). Semantic mapping is thus foundational in both robotics and embodied AI, particularly in open-world settings like autonomous driving (Bao et al., 2023; Li et al., 2024b); search-and-rescue operations (Gautham al., 2023), automated cleaning robots (Singh et al., 2023) among others. While traditional mapping techniques prioritize geometric accuracy for localization and obstacle avoidance, recent. progress in deep learning, computer vision and multi-modal perception has shifted focus toward semantically rich maps. Despite extensive literature, semantic mapping remains open and rapidly evolving research area.
Existing surveys review semantic mapping within the context of its usage in downstream applications, thus largely centered on task advancement (Pfeifer & lida, 2004; Kostavelis Gasteratos, 2015; Cadena et al., 2017; Xia al., 2020; Duan al., 2022; Deitke et al., 2022; Zhu et al., 2021; Zhang et al., 2022b; Wu et al,, 2024; Lin al., 2024; Garg et al., 2020; Achour et al., 2022; Zheng et al., 2024a). Song al. (2025) has recently reviewed semantic mapping in indoor robotics applications, but focused the type of

---

Published in Transactions on Machine Learning Research Alugust/2025)
Sofa
Coffee
Table
Explicit Information
Implicit Features
Occupancy Explored
Closed-Vocabulary Open-Vocabulary
Encoding
Bed
Dining
Object Type
Table
Room Type
Table
Audio Intensity
Topological Map
Grid Map
Geometric Map
Hybrid
Structure
Sofa
Coffee Table
Bed
Dining Table
Bed
Sofa
Coffee Table
Bed
Table
Dining Table
Sparse
Dense
Figure 1: Semantic maps. The survey categorizes semantic map building methods in embodied agents based on their structure and the encoding it stores. Structure: a map of a physical environment can be structured a topological map (with nodes and edges), a spatial grid, a dense geomelric map or a hybrid map combining two or more the others. Encoding: the structured maps can store either explicit (occupancy, object type, etc.) or implicit information (learned neural features) corresponding to the observation made that location.
semantics acquired and the techniques of such extraction. In contrast, our survey offers a comprehensive review of semantic map-building methods, organized around the underlying map representations themselves, independent of specific downstream tasks. This perspective is both timely and necessary, given the recent. advances in foundation modcls and the growing demand for gencral-purpose, open-vocabulary, task-agnostic representations.
To provide a principled understanding of semantic map-building methods, this survey categorizes the literature along two fundamental axes (sec Fig. 1): map structure (c.g., topological graphs, spatial grids, dense gcometric, and hybrid representations) and semantie encoding (explicit annolations vs. learned implicit features). This taxonomy reflects core design choices that influence map scalability, interpretability, generalization, multi- modal fusion and querying. By organizing approaches in this manner, aim to unify diverse strands of research (see Tab. for summary of the reviewed papers), highlight the trade-offs between different representations and present key challenges and future opportunities in semantic mapping. This survey centers semantic mapping approaches in the context of indoor mobile robots, a domain that offers a well-defined, practically relevant and technically rich environment for research. We focus on semantic mapping within embodied AI, that enables controlled environments capable of isolating high-level reasoning abilities of semantic maps from the complexities of noisy sensors and real-world observations. At the same time the survey establishes connections to SLAM-based approaches underscore their methodological overlap and to bridge foundational techniques in robotics with emerging paradigms in embodied AI.
This survey is organized as follows. We begin the survey in Sec. 2 with background context to ground our discussion on semantic mapping. Sec. introduces semantic maps and lays out the foundational components. We then explore different map structures in Sec. 4, followed by encoding strategies in Sec. 5. Sec. 6 reviews existing evaluation methodologics. In Sec. 7, we examine the key challenges in building semantic maps, Icading into Sec. 8, which outlines promising directions for future research. We conclude in Sec. by synthesizing the main insights and takeaways from the survey.
2

---

Published in Transactions on Machine Learning Research (August/2025)
Table 1: Summary table. We characterize works that use maps for embodied Al as well as robotics by the type of structure they use (Grid, Topological, Dense geometric maps) and how information is encoded in the map (Explicit vs Implicit). Explicit encodings are pre-selected information such as occupancy visitation time i and others. Implicit encodings are learned representations such as visual (V) or visual-and-language (VL) features. The use of VL features (typically from large pretrained models) enable building open vocabulary maps. Works that aggregate implicit features onto grid map, but finally decode into explicit encodings are marked as 'Implicit to Explicit' in this table.
explored-area , object category
Structure
Encoding
Grid
Topological
Dense geometric
Occupancy Grdds [Llles, 1080]
Topo SLAM (Choset & Nogatam, 2001) Topomap (Blochiiger et al., 2018)
ANS (Chappor es al., 2019)
6D SLAM (Nüchter al., 2007)
Explicit scusaaties)
UPEN (Googgkis
Integrating (1 Trnn et aL. 10050)
2022.)
Foot (Chen
Droid-S AMM (Thee & Ding, 2021)
mapping (May et al., 2009))
Comblaing (Tocatts et ol., 2001
Voldor+ SLAM (Miz. Dunn, 2321)
Curtora Geonge (Meger
Elficient (Bu al. 2013)
SemlExp (Chapplst et al. 20120)
SLAM++ (Sakss Moreno es al., 2013)
1.2M (Georgakis
lunitation (Davallet ol., 2013]
Explicit (semautios)
(SEAL (Chaplot nl., 2021 -
Compact (Patki et 2010)
Towards semantle mape (Nüchter & Hertaberg, 2008)
CMP (Georgakis es al, 2022c)
Multimodal (Arkin al, 2020s)
3D object-clase map (Stäckler al., 2013)
MOPA (Raychonchuri
2025
LIFGIF (Rayvanndhur al, 2025)
Parsing ('riihbel Street Somia ( (Floras Leibe, 2013)
2012)
GOAT-Beuch (Klanna
(Chang et al., 20125)
May-Ne (Zhaug et al., 2025)
Dense 3D (Hermnne all, 2014)
Instruction-giile (Wang al, 27251
SamentiePusioa (MeCCrraa
Pollowing
(Matusekk et al, 2010)
Concapzaal (Zeuks et al., 2008)
SemuntioGraph (Hemschrrdra et al., 2014)
Leuningg
maye (Walter
Lecring models (Henschandr al., 2015)
Inferting mape (Duvalles ot al., 2016)
BEV Bett (An al., 2023)
LaneSigNes (IA et al., 2020b)
Contextually guided [Amand et al., 2013;
aDSG (Armeni 2019)
3D-DSG (Rosino) 2020s
Hydda (llughes et al., 2022]
S-graphs (Shen ol., 2019)
graphss (Barle al. 2023
CURB-SG (Crove of sl,, 2024)
Implacit to Expliett
SemanticMapNNt (Cartillier et ol., 2021)
Seeso (Wu sl., 2021)
MapNet (Heeriiqus & Vedaldi, 2018)
CMP [Gupta et ai, (i, 2017)
SPTM (Savinov et al. 2018)
NTS (Chaplot al., 2020b)
XPCour (I (Thomas al., 2019)
Implici (V)
MulnON (Want al., 2021)
CMTP (Chen 2021)
RointRosNet (Dessi al., 2023)
RNR-Map (Koon et al. 2023)
VGM (Kwon es al., 2001)
Noural Fusion (Morur et al., 2025)
CoW [Uldte 2.23)
VI.Map (Heang al. 2023k)
Le RNN-MMp Map (Thioli et al., 2023)
OpenScene (Peng es al,, 2023))
LM-NNw (Shoh et al, 2023)
CLIPPalds (Stadiullah: sl,, 2003)
NLMap (Chen et al., 20236)
Rohoffop (Garg 2024)
ConceptRusion (Jatovillabiila et 2023)
Implicit (VL)
VLFM (Yokoyama al., 2025;
VouPaser (Hunng et al., 20256))
Cllo (Maggio et al., 20144)
3D aware ObiNaw (Zhang al,, 2023s)
CLIP2Scene (Chon al., 2025))
InstreetNa (Long al., 2024)
OVL MAP (Wen 2025)
CouceptCraphs (Ga et al. 2124)
HOV-SG (Werhy al. 2024)
StructNav (Chen et al., 2025b)
2 Background Reading
In this section, provide background readings that help to contextualize the survey semantic mapping. We first introduce key tasks (Sec. 2.1) in both robotics (Sec. 2.1.1) and embodied Al (Sec. 2.1.2), since semantic mapping is often studied in conjunction with such tasks. In Sec. 2.2, we discuss the basics (Scc. 2.2.1) and techniques of SLAM (Sec. 2.2.2) as well as the historic evolution of Sernantic SLAM (Sec. 2.2.3), which has an overlap with the map building techniques that we later discuss throughout the survey. Next we briefly discuss two popular system design choices in Sec. 2.3, end-to-end learning (Sec. 2.3.1) and modular pipeline (Sec. 2.3.2), since it helps the readers understand how semantic map construction varies between end-to-end and modular approaches.
3

---

Published in Transactions on Machine Learning Research (Auuust/202))
Language Understanding
Perception
Navigation Let go the nursery...
"Find red and blue striped zebra toy in the nursery"
What does the text mean?
How does the object look like?
Have seen it before?
O
Semantic Map Memory
How to structure the map?
What to store in map?
How to build the map?
Figure 2: Motivation. To perform complex (ask in indoor enviromment, the robotic agent must possess multiple skills of language understanding, visual perception, navigation, etc. Among these the most crucial is building and maintaining a semantic map of the environment so that it can come back to while performing the task.
2.1 Embodied tasks
Embodied tasks involve an intelligent agent, either physical robot or a simulated body, perceiving and interacting with the enviromment through its embodiment (sensors, actuators etc.). These tasks require the agent to not only understand the world (via vision, language, etc.) but also take meaningful actions within it (such as navigation or object manipulation). In this section, we provide a brief overview of the embodied tasks explored in both robotics and embodied AI, highlighting their evolution a growing research direction. This serves contextualize our focus semantic mapping, which is often studied in conjunction with such embodied tasks.
2.1.1 Robotics tasks
The era of modern robotics began with Unimate (Detesan & Moholca, 2024), the first industrial manipulator arm and Shakey (Center, 1984), the first mobile robot. Since then the field has evolved to have increased autonomy and complex reasoning skills. This has been enabled by research in various areas spanning basic navigation and obstacle avoidance to complex real-world capabilitics involving perception, mapping, and manipulation.. Early research focus on collision avoidance (Fox et al., 1997), Monte Carlo Localization (Thrun et al., 2001) and SLAM frameworks enabling robots localize while mapping unknown environments (Thrun, 2002; Thrun & Montemerlo, 2006; Taheri & Xia, 2021). As sensors have improved over the years, semantic mapping gained traction, integrating object recognition and scene understanding into spatial maps (Salas- Moreno et al., 2013). Concurrently, robotic manipulation has become central focus, with tasks such pick- and-place, insertion, and rearrangement. Classical approaches like force-control -based manipulation (Raibert & Craig, 1981; Yoshikawa, 1985) have been complemented by modern reinforcement bearning-based systems like QT-Opt (Kalashnikov et al., 2018) and Transporter Networks (Zeng et al., 2020). The field has also expanded to multi-task and long-horizon planning, where agents are required to complete sequences of interconnected tasks rather than isolated actions. A prominent framework in this area is Task and Motion Planning (TAMP) (Garrett et al., 2020), which integrates high-level symbolic reasoning (e.g., planning to "pick up the cup, open the door, and place the cup the table") with low-level continuous motion planning (e.g., calculating joint angles and trajectorics to exccute those actions). This enables agents to operate in complex and dynamic environments. Recent trends include uncertainty-awart planning (Stachniss et al., 2005; Blanco et al., 2008a; Georgakis et al., 2022a; Carlone et al., 2013; Pan et al., 2019), semantic mapping (Rosinol et al., 2020b) and task planning dynamic environments (Rosinol al., 2020a). Another fast growing field in robotics is autonomous driving (Guan al., 2024: Zhao et al., 2024), where recent trends increasingly rely on Bird's Eye View (BEV) representations, which transform multi-view sensor inputs into unified top-down map
4

---

Published in Transactions on Machine Learning Research (Aguust/2025)
using end-to-end learned models. BEV is common intermediate representation used to simplify reasoning about spatial layouts, obstacles, lanes and other semantic elements. Transformer-based architectures like BEVFormer (Li et al., 2024c) and BEVFusion (Liu et al., 2023b) have emerged as state-of-the-art, enabling spatially consistent, semantic-rich BEV maps that support tasks such as detection, planning and trajectory prediction. For example, LaneSegNet (Li et al., 2024b) trains an end-lo-end system to predict lane segments from multi-view surrounding images. Unlike semantic maps that are discussed throughout this survey, BEV maps are typically short-lived local (per-frame) representations and specialized for driving scenarios. Overall, the advances in robotics collectively demonstrate a shift from reactive, single-task robots toward robust, general-purpose systems capable of operating autonomously in dynamic real-world environments.
2.1.2 Embodied Al tasks
The field of embodicd AI began to emerge prominently around 2017-2018, catalyzed by the availability of simulation environments, such as A12-THOR (Kolve et al., 2017), Habitat (Savva et al., 2019), etc., and benchmarks such as Aision-and-Languag Navigation (VLN) (Anderson et al., 2018b), Embodied Question Answering (EQA) (Wijmans et al., 2019a), etc. Historically, embodied AT diverged from robotics by focusing more heavily on learning-basse agents interacting with simulated environments using vision, language, and action, often without relying on physical hardware. The divergence was motivated by the scalability and reproducibility challenges in real-world robotics, and the need to study cognition, planning, and multi-modal learning at scale. Broadly speaking, embodied AI positions itsclf at the intersection of computer vision, natural language processing and reinforcement learning, while classical robotics focuses on control, perception and physical interaction with the real world.
Embodied AI tasks vary depending on the type interaction of an agent with its environment. Broadly, we can group embodied tasks into three groups Eaploration task (Chaplot el al., 2019) requires agent to efficiently explore its environments; Navigation task (Wijmans et al., 2019b; Batra et al., 2020b) requires the agent to take actions in order to move around the environment; Manipulation task (Szot et al., 2021; Weihs et al., 2021) requires the agent to perform interactive actions to change the state of other objects in the environment. The taxonomy of tasks can be further broken down by the target specification provided to the agent, which impacts the information that need to be retained. For instance, for navigation, the following are commonly studied. In Poinl- Navigalion (Wijmans et al., 2019b) (PointNav), the agent is given a target coordinate relative to its starting position, whereas in Image-Goal Navigation (ImageNav) it is given a target image (Chaplot et al., 2020b). In the Object- Goal Navigation (ObjectNav) task, the agent needs to navigate to any instance of an object category (Yadav et al., 2022). An exlension to the ObjectNav task is the Multi- Object Navigation (MultiON) (Wani et al., 2020) task where the agent is required to navigate to multiple objects in a particular sequence. Vision-and Languugge Navigation (VLN) (Anderson et al., 2018b) requires the agent to find the target as specified by a natural language instruction. In Audio- Visual Navigation task, the agent needs to navigate to an object emitting particular sound in an indoor environment (Chen et al., 2020a; Gan et al., 2019). Depending on the type of task, it may be sufficient to store just the object category (e.g. for ObjectNav) in the map, or it is may be necessarily to retain more finer-grained information (e.g. VLN). In this survey, we will mainly focus on recent work on room-scale map building for navigation as these methods can be extended for maps for manipulation and used for exploration.
2.2 Simultaneous Localization and Mapping (SLAM)
Although we briefty discuss robotics applications in Sec. 2.1, we address Simultaneous Localization and Mapping (SLAM) (Thrun et al., 1998a; 2000; Ferris et al., 2007; Huang et al., 2011; Carlone et al., 2024) scparately here due to its close connection with semantic mapping, particularly the Semantic SLAM literature. While both aim to enrich spatial representations with semantic information, they differ in scope and purpose. Rooted in robotics, Semantic SLAM focuses on building globally consistent, pose-aware maps augmented with semantics to improve localization, mapping accuracy and long-term robustness. In contrast, semantic mapping in embodied AI emphasizes representations that support high-level reasoning and decision-makiin making, often abstracting away precise localization and low-level sensor noise in favor of adaptability.
5

---

Published in Transactions on Machine Learning Research (lugust/202))
2.2.1 Basics of SLAM
SLAM (Simultaneous Localization and Mapping) enables a robot to leverage data from multiple sensors (cameras, LiDAR, IMU etc.) to perceive the environment and build a map while simultaneously localizing itself on the map. Mathematically speaking, SLAM aims to estimate the robot's trajectory and a map of the environment simultancously, using noisy sensor data and imperfect motion estimates. A robot's motion is modeled by a state transition function that predicts its next pose based on the previous pose and control inputs:
Tt = f(I=11, ue) +
(1)
where, de is the robot's state al time L, ur is the control input (such as wheel odometry or IMU), and wy is process noise, Simultaneously, the robot makes observations of the landmarks in the environment, which is modeled by the measurement equation:
2 = h( m))) + Uz
where 2/ is the observation of landmark i at time t, m, is the position of the landmark and oy is the measurement noise. The difference between the predicted and the actual observations is the residual (error):
cx(x) = --)
SLAM formulates the estimation problem as a nonlinear least-squares optimization, which aims at finding the optimal trajectory x* and landmark positions m that best explain all measurements:
m arg min ΣΣ (x) Qee(())
(4)
2,
where Q4 is the information matrix (inverse covariance) that weights the measurements according to their uncertainty, i.c. more weight is given to mcasurement that is very certain. (x) is the transpose of the error (x()) and the term ef (x) Q,e((z) represents weighted squared error. In graph-based SLAM, this optimization corresponds to optimizing a factor graph (Loeliger, 2004; Dellaert et al., 2017), where nodes represent robot poses and edges represent spatial constraints, such that Equation (4) becomes:
x= arg min Σ || Liy(s, =,),,,)))),, 1,), (L.))E
(5)
where, (i,j) E represent edges in the pose graph, Zij is the relative pose measurement between nodes i and j. 2ij is the predicted relative measurement from current estimates, and || Il3,, is the Mahalanobis distance (weighted squared error). This global optimization ensures that the estimated map and trajectory are as consistent as possible with the entire history of noisy sensor data.
2.2.2 SLAM Techniques
Active SLAM (Ahmed et al., 2023) enables an autonomous agent to actively choose its actions to improve its map and localization, rather than passively mapping the environment as it moves (Fig. 3-1eft). The SLAM system (Cadena et al., 2016; Pu et al., 2023) primarily consists of a front-end module and a back-end module (Fig. 3-right). The front-end module computes feature extraction, data association and feature classification on the sensor observations. Moreover applics algorithms such as Iterative Closest Point (ICP) (He et al., 2017b) that aligns sensor observations from subsequent frames into consistent 3D geometry as well loop closure (Tsintotas et al., 2022) that recognizes whether the current observation matches with previously visited area. The backend module, the other hand, is responsible for pose optimization and map estimation based on the data from the front-end. The back-end also provides feedback to the front-end for loop closure and verification. While some SLAM methods use LiDARs (Khan et. al., 2021), others use camera as the primary sensor (visual SLAM or vSLAM), due to the availability of less expensive cameras and the advances in computer vision. Visual SLAM methods often extract and match geometric features (points, lines, or planes) (Mur-Artal al., 2015; Yang & Scherer, 2017; Kaess, 2015; Cai et al., 2021) from the image, with the help of feature detection algorithms such as SIFT (Lowe, 2004), SURF (Bay et al., 2008), and ORB (Rublee
6

---

Published in Transactions on Machine Learning Research (August/2255)
et al., 2011). Others operate directly on the image pixel intensities (Newcombe et al., 2011b; Engel al., 2014) and thus retain all information about the image. While these methods use RGB camera as the sensor, RGB-D SLAM (Newcombe et al., 2011a; Kaess et al., 2012; Endres et al., 2013; Whelan et al., 2016) uses RGB-D cameras to simultancously collect color images as well as depth images. In contrast, Visual-Inertial SLAM (Mur-Artal & Tardós, 2017; Cheng et al., 2021) uses an additional IMU sensor to mitigate the effects of image blur and poor illumination from using camera sensor alone.
Mapping
SLAM
Kaypor plannng
Sensors
Frent-end
SLAM
Exploration
Robok
Map=Pose
Tajesay planzing
Active SLAM
Back-end
Controller
Localization
Planning
Active Localization
Figure 3: SLAM. (left) At the core of classical mobile robotics lic three core tasks mapping, localization, and planning. These are oflen interdependent on each other and overlap to form other tasks such as SLAM (localization and mapping), exploration (mapping and planning), active localization (planning and localization) and active SLAM (mapping, localization, and planning) [figure reproduced from Fairfield (2009)). (right) Architecture of SLAM includes the front-end to handle perception tasks and the back-end to estimate pose and the map. Active ST.AM (A-SLAM) additionally includes planning and controller modules [Figure reproduced from Ahmed et al. (2023)).
2.2.3 Semantic SLAM
While geometric maps in SLAM systems prove to be effective for simple navigation and control (Shan et al., 2020b; Campos et al,, 2021), they provide limited understanding of the environment's semantic content. As robotic systems grew more complex and began interacting with unseen unstructured dynamic environments, the need for higher-level understanding led to the emergence of Semantic SLAM. enriches spatial giometric maps with meaningful concepts like objects, rooms, and affordances, thus bridging the gap between perception and task-level reasoning in modern embodied agents. Early approaches do so by matching image features to object model databascs (Civera et al., 2011; Salas-Moreno et al., 2013) or by incorporating semantic and geometric cues into structure- wwwoorroooooeeeeeee motion (Bao & Savarese, 2011; Bao et al., 2012) and SLAM optimization processes (Fioraio & Di Stefano, 2013). With advances in deep learning, recent. works on Semantic SLAM (McCormac et al., 2017; Yin et al., 2020) have improved semantic representation, real-time performances and dynamic environment modeling. Some works show that using semantic maps improves the performance of the various modules in a SLAM system (Xiang Fox, 2017; Tateno al., 2017, 7; Qin al., 2021; Qian et al., 2021), while others show that using SLAM to impose consistency constraints improves semantic segmentation (Mozos et al., 2007; Lai et al., 2014; Pronobis & Jensfelt, 2012; Pillai & Leonard, 2015; Cadena et al., 2015). Many also show that combining SLAM and semantic segmentation into joint frameworks enhances both mapping and object recognition (Flint et al., 2011; Bao et al., 2012; Hane et al., 2013; Kundu et al., 2014; Sengupta & Sturgess, 2015; Vineet et al., 2015).
Moreover, semantic SLAM works can be categorized based on the structure and encoding of the semantic maps, the same way as semantic mapping in embodied AL For example, some works use sparse repre- sentations (Bowman al., 2017; Yang & Scherer, 2019; Nicholson al., 2018; Chen et al., 2020d; Shan et al., 2020a; Atanasov al., 2018; Feng et al., 2019; Moren-Moreno et al., 2013), while others dense
7

---

Published in Transactions on Machine Learning Research (lngust/2025),
M
Sensor Inputs
Actions
f2
a
End-to-End Trained
Sensor
M
Inputs
Actions
Visual
Mapper
Exploration
Planner
Encoder
a
Figure 4: End-to-end vs Modular. (Top) End-to-end model is trained as single pipeline which generates actions directly from sensory imputs. (Bottom) Modular pipeline consists of various sub-modules, each with a specific function so that they can be trained independently of the others.
representations (Rosinol et al., 2021; Grinvald et al., 2019; McCormac et al., 2018; Chen et al., 2019b; Maturana et al., 2018; Miller et al., 2021; 2022). Some works store explicit semantic concepts such rooms (Pronobis & Jensfelt, 2012) or objects (Pillai & Leonard, 2015) in the map, while others store implicit neural features (Xiang & Fox, 2017; Tateno et al., 2017; Qin et al., 2021; Shah et al., 2023; Maggio et al., 2024; Werby et al., 2024). Throughout this survey, we will revisit these papers to categorize them based on the categories we use for semantic mapping embodied AT.
For a more comprehensive reading on SLAM and semantic SLAM, we refer the readers the following survey papers - Thrun (2003); Kostavelis & Gasteratos (2015); Cadena et al. (2017); Younes et al. (2017); Taketomi et al. (2017); Landsiedel et al. (2017); Saputra et al. (2018); Sualeh & Kim (2019); Chen et al. (2020c); Lluvia et al. (2021); Taheri & Xia (2021); Tourani et al. (2022); Pu et al. (2023); Racinskis et al. (2023); Sousa et al. (2023); Wang et al. (2024); Chen et al. (2025).
2.3 System design strategies
Designing systems for embodied agents involves fundamental architectural choice between end-to-end learning and modular pipelines. While end-to-end approaches map raw sensory input directly to actions using single neural network, modular systems decompose the task into interpretable components. Understanding this distinction is essential for situating semantic mapping within the broader system design, as it influences how maps are constructed, represented and utilized. This section briefly outlines these paradigms (Sec. 2.3.1 and Sec. 2.3.2) and discuss trade-offs between them in Sec. 2.3.3, to provide a background for the mapping methods discussed in the remainder of this survey.
2.3.1 End-to-end approaches
The embodied AI and robotics community has seen lot of progress in training task-specific end-to-end models with reinforvement learning (RI.) (Tai et al., 2017) that directly learns to predict discrete (Wani et al., 2020) or continuous actions (Kalapos et al., 2020) from visual observations (see Fig. 4). These methods may consist of an unstructured memory such as LSTM (Mei et al., 2016) (Dobrevski & Skočaj, 2021). However, such representation lacks reasoning about 3D space and geometry and fail to perform well in long- ong-horizon path planning. This has lead the development of approaches that build an intermediate map representation. Such a map can be implemented using differentiable operations so as to facilitate end-to-end training. Gupta et al. (2017) shows that an egocentric map built this way is beneficial for both PointNav and ObjectNav tasks, whereas Henriques & Vedaldi (2018) learns a global map for the task of localization. While these methods are trained using supervisod lcarning, Wani et al. (2020) use RI to Icarn to predict actions based on an intermediate global map of the environment to address the complex MultiON task. In aulonomous driving and planning, end-to-end approaches aim to learn a direct mapping from raw sensor inputs (e.g., images, LiDAR) to control outputs or motion trajectories. These methods often leverage transformer-basse architectures (Lin et al., 2022) jointly model perception, spatial reasoning and decision-making within single framework. By optimizing the entire pipeline holistically, they promise improved robustness and
8

---

Published in Transactions on Machine Learning Research (Aguust/2025)
efficiency, particularly in complex, dynamic environments. Recent work demonstrates that such end-to-end models can learn to implicitly reason about obstacles, goals, and traffic patterns. However challenges remain in interpretability, generalization and safety-critical validation. However, irrespective of the map representation or the mode of training, these approaches need to be retrained every time the task definition changes and even the basic skills required lo perform a lask need to be learned from scratch.
2.3.2 Modular pipeline
Another line of work explores how to breakdown a complex navigation task into a set of basic skills that the agent needs to acquire. Such skills can then be learned independently of each other so that they can be leveraged across various tasks without the need to be retrained from scratch. While some use two modules (Bansal et al., 2020), one for high-level decision making and another for low-level planner, others (Chaplot et al., 2019; 2020a; Gervet et al., 2023; Raychaudhuri et al., 2023) further breakdown into sub-modules, such as visual encoder, mapper and exploration in addition to the low-level planner. Fig. 4 shows the difference between the end-to-end approach and the module approach, and we elaborate on the modules and popular design choices below.
Visual Encoder. This module encodes agent observations to produce semantic visual features and predictions at every time step. Prior works have used visual features from pretrained backbones such as ResNet He et al (2016) or ViT (Dosovitskiy et al., 2020), and often leveraging object detectors MaskRCNN He et al. (2017a) or FasterRCNN Ren et al. (2015). As we will see in Sec. 5.2.2, with the development of large pretrained vision-language models and open-vocabulary detectors, pretrained models such as CLIP (Radford et al., 2021), LSeg (Li et al., 2022), DINO (Caron et al., 2021; Oquab et al., 2023), and others are increasingly popular as the basis for building large to open-vocabulary maps. The visual encoder used will determine the information captured in the features, and whether there are detected object instance bounding boxes or segmentations for integration into the mapping modules.
Mapper. The mapper is responsible for building a semantic map of the enviromment from the encoded image features and agent pose. To build a global map over time, the mapper lypically aggregates the current, map with the map from previous step (see Sec. 3.4 for details). In this paper, we survey how recent methods structure the map (Sec. 4) and what information can be encoded in it (Sec. 5). Tab. 2 summarizes the type of information stored in the map by various methods. This can be occupancy information, explored area or semantic labels of the detected objects.
Exploration. This module enables the agent to explore its environment efficiently to either ensure the map is complete (by maximizing the covered area) or selecting unvisited areas where the target is likely to be. Typically, the exploration module selects a point or region to explore given the obstacle map built by the mapper and the current agent pose. Agents can use simple cuuristics-based methods such as sampling a point at uniform (Zhang el. al., 2021; Raychaudhuri et al., 2023), systematically sampling four corners of a grid centered at the agent (Luo et al., 2022) or selecting a point from the unexplored frontier (Yamauchi, 1997). To docide which frontier point the agent should explore, various strategies are employed, such selecting the nearest point to the agent (Gervet et al., 2023) or the most promising point based on semantic reasoning. In the semantic reasoning based exploration methods, the agents may select the highest text-image relevance score (Gadre et al., 2023; Yokoyama et al., 2023) from a pretrained large vision-language model such as BLIP-2 (Li et al., 2023), select the highest probabilistic output of the VLM directly (Ren et al., 2024), or leverage a LLM to extract common-sense knowledge (Zhou et al., 2023). Researchers have also used learned policies (Chaplot et al., 2019; 2020a), where the agents are generally trained with RL using rewards, such as coverage (Chen et al., 2019a) or curiosity (Pathak et al., 2017; Mazzaglia et al., 2022). Although there is less hand-crafted rules in learning- basedd methods, they need millions of training steps and careful reward engineering.
Planner. Once a map is built, a low level path-planning module is used to plan a path to the goal location from the agent's current location. The path consists of low-level actions that can be executed by the agent to move to the goal. While this is implemented as a heuristics- baatd Fast Marching Method (Sothian, 1996) in most of the prior works, a recent approach MOPA by (Raychaudhuri et al., 2023) has used learned PaintNav policy trained offline with DD-PPO (Wijmans et al., 2019b).
9

---

Published in Transactions on Machine Learning Research (August/2025)
In Tab. 2, compare common modular approaches and look at how they leverage various heuristics- based or learned approach for each of the modules. The advantages of modular pipeline include its ability to leverage pretrained models from other tasks (Gervet et al., 2023; Raychaudhuri et al., 2023) and its ability to transfer from simulation to real-world robots better (Gervet et al., 2023).
Table 2: We show how prior works build map either as part of an end-to-end architecture or as modular architecture consisting of four basic modules. For the modular approaches, we summarize the different module choices as well as what they store in their map.
Thokk
End-tto-For
Vieml Encoder
Map
Modular
Exploration
Planner
ANS (Chaplot et al,, 2019)
Exploration
ResNestS (Ile et al., 2016)
occupancy + ex- plerd
loarned poley
Fost Marching
NIS (Choploc et al., 21206)
ImnageNax ObjectNav
BesNes18 (Ile et al., 2016)
topelogical map occuponcy + semantic
Joazued polcy
A*
Sem (Chapled 33206)
Maskil CNN (Ile ol,, 2017s)
loarned poiry
Post Marching
Moellaom (Geret al,, 2023)
Obijrct Nawe
MokER-CNN (He al, 20174)
cempaniy + pland ammanti lehele wounttic labiis
leamna SomEap
Foot Marching
NIOPA (Bayehanlhuri al,, 2023)
MoliON
PNN (Rom et ol,, 2015)
Uniform
Sun- Explo
Point Nao(NNjjoonnn prooo 20106)
(Cupta 2017) MappNat (Hlariqus Vodalii, 2018)
PountNor ObgectNor
Localization MaliON
MultON (Waul et al, 2020)
2.3.3 Trade-offs
Both the design choices have their own merits and limitations. End-to-end systems are simple to train and deploy, and they directly optimize task performance. However, they often lack interpretability, generalize poorly to new scenarios, and make component reuse difficult. In contrast, modular pipelines offer greater transparency, reusabilily, and flexibility lo combine learned and classical methods, making them well-suited for complex tasks involving morioorizon planning and semantic reasoning, despite the potential for error propagation and suboptimal integration of modules.
3
Semantic map
In this section, we introduce semantic maps (Sec. 3.1), and provide an overview of their structural representa- tions (Sec. 3.2), semantic encodings (Sec. 3.3), and the process of building them (Sec. 3.4).
3.1 What are semantic maps?
While maps, capturing the geometry of the space, can help an agent avoid obstacles, they are not sufficient for the demands of more complex reasoning tasks. An enhanced map that goes beyond geometry to capture meaning and context in its environment aligns with how humans perceive and navigate its surroundings. These are semanlic maps, which provide a richer and more nuanced understanding about the objects and places in the environment and are indispensable for performing complex tasks such as navigating to specific room (kitchen) (Narasimhan et al., 2020), rearranging objects (Trabucco et al., 2022) performing an action specific object (silting environment through sensors (camera, LiDAR, etc.), uses cognition to classify the objects and regions it perceives (Ren et al., 2015; Нe et al., 2017a; Liu et al., 2024; Kirillov et al., 2023; Zhang et al., 2023b), stores them in a structured semantic map which can be queried for efficient reasoning.
couch) (Peng et. al., 2023). A physical or virtual agent perceives the
3.2 What is the structure of this map?
A semantic map can be ured as a spalial grid map, lopological map, dense geometric or a hybrid map (Fig. 1). A spatial grid map is a top-down grid where each grid cell represents an area in the physical environment. So if an object is a certain location (X, Y, Z) in a. 3D scene, the semantic map will contain information about that object at the corresponding grid cell (æ, v), where x and y are the row and column numbers respectively such that there is direct mapping from (X, Y, to (z, For navigation, most spatial
10

---

Published in Transactions on Machine Learning Research AAugust/2025
Current Observation
Map Update
Feature Extraction
4
Localization
Figure 5: Map building involves localization (where the agent is on the map), feature extraction (extracting useful semantic information from the observations), and map update (building the map by aggregating the semantic information over time).
maps are 2D, such that grid cell ignores the Z-axis (up direction) by aggregating the semantic information across the up axis. However they can also be 3D where the Z-axis is divided into discrete bins. On the other hand, a topological map is a graph-like structure where nodes represent objects or important landmarks in the scene and edges represent relationship (distance, spatial relation, etc.) between them. It is also possible to store semantic information on dense geometric, which can be viewed as a 3D map with varying density. a dense geometrie, information is associated with each point (a, y. corresponding in the physical space. Unlike the voxcl-grid, which is regularly spaced, points can be sampled at varying densities. Some works combine two more of the above structures to form a hybrid map since each structure has its own ad vanataee and limitations. We discuss each of these in detail in Sec. 4.
3D location (x, Z)
3.3 What encoding is stored in this map?
The semantic map stores information about a particular 3D location (X, Y, 2) in the physical enviromment. This information can either be explicit or implicit. Explicit encodings have clear specific meanings assigned each value. For instance, each cell (X, Y, Z) can store information about whether there are any obstacles that position, whether that location has been explored by the agent, the category of the object present there and on. On the other hand, implicit encoding is feature encoding capturing information derived from the sensory input (c.g. images) that the agent observes at that particular location (X, Y. Z). The fcatures are typically extracted from pre-trained encoders. Depending on whether the feature encoder was pre-trained on a set of images from limited categories or a large internet-scale dataset of image and language data, the implicit encoding can be either closed-vocalle vocabulayy or open-vocabulary. The term closed- vocabulary used to indicate only a limited set of object categories recognized, while in a open- vocabulary setting, the features extractors theoretically identify 'any' object.
3.4 How is the map built?
Creating accurate and detailed semantic maps requires integrating data from various sources and sensors such as camera, LiDAR and depth sensors. More specifically, map building consists of having an agent navigate about a space, and accumulating observations O₂ at time step into the appropriate map structure my- To build an accurate map, the agent first needs have an estimate of where is (localization). Next it extracts semantic information from an obscrvation F(O,) (feature extraction). and combines the features into common map over time (accumulation) (refer to Fig. 5). While building spatial grid map, an additional step is to project the features onto the map (projection). It is common to group the last three steps into mop building and study it jointly with localization in Simultancous Localization and Mapping (SLAM). We discuss SLAM methods briefly in Sec. 2.2.
Localization. Localization can be challenging due to noisy sensors and actuators. To simplify the problem, it common the embodied AI community either assume perfect localization is given at each time step (Cartillier et al., 2021) or localize the agent with respect to its starting position in an episode (Henriques & Vedaldi, 2018) assuming perfect actuation. The latter is more easily adapted to the real-world setting since
11

---

Published in Transactions on Machine Learning Research (August/2225)
it doesn't require the exact knowledge about the agent's pose. Instead the relative displacement of the agent with respect to its starting pose is enough to build the map eventually.
Feature extraction. Feature extraction is a crucial part of building a semantic map. Ideally these features should be representative the objects present in the map. We discuss this topic at length in section Sec. 5.
Projection. An important step building a spatial grid map taking the 2D observations and project them into 3D. Typically, this relies on having depth information and known camera parameters in order to convert 2D pixel coordinates to 3D world coordinates. To project a particular pixel in the camera frame, first ray is shot from the camera center through the image pixel (i:j) to the depth di, to get 3D point in the camera coordinate frame. Next the camera coordinates are converted to the world coordinates (X, Y, Z). For a 2D spatial map, the 3D coordinate( X, 2) is mapped to the grid cell indices æ and in the spatial map. The transformation for the standard pinhole camera with known camera pose (3D rotation and 3D translation t) and intrinsics (K) can be written as:
X
i
di R
K
-
(6)
Z
and the orthographic projection can be written as,
X
Y
y/
P
(7)
0
Z
where where P. is known orthographic projection matrix to convert 3D world coordinates into 2D grid cell indices. If more than one point are projected to the same grid cell in the spatial map, they are accumulated into the cell using an function to aggregate the features or predictions.
Accumulation. During map update there are many ways to aggregate features or prediction into the map including 1) overwriting the map with the latest observations (m, m--1), 2) performing mathematical operations such as max (m, max(m;, m)-1) mean (m, mean(m,, m)-1)), and 3) using learned neural network. For learned aggregation functions, it is common to use recurrent network (LSTM, GRU) (m, = GRU(m,,
During the process of building the map, there are several other important aspects to consider.
Egocentric vs allocentric. There are also choices in the reference frame used for map-building, to either maintain a map with an egocentric coordinate frame that is relative to the agent (c.g. +y coordinate to the front of the agent) or allocentric (e.g. world) coordinate frame.
Tracking visited areas. For the map be complete, it is important for the agent be able to determine whether it has already visited location not, and whether there are unexplored locations. For specific embodied task, it may not always be necessary for the agent to built a complete map if the task can be accomplished.
View point selection. In the case of the embodied setting, the agent is also limited in the possible viewpoints it can observe, and must accumulate into the map, observations in a sequential manner. This contrast to the non-embodied setting, where there can be more freedom in selection of viewpoints, and observations can be first collected and then analyzed logether.
Online offline map building. It is possible for the agent to build map by exploring environment first. After the map has been built, the agent can then start to perform the specific task. In this scenario, the agent builds the map and performs the task in two separate phases, a process known as offine method of map building. Although this method saves compute time during the actual task, there is the overhead of an extra exploration phase for the agent to familiarize itself with a new environment. This approach can be appropriate when the agent is expected to be reused in the same environment repeatedly, However, since the map is a static snapshot and if it is not updated during the task, there be mismatch between the actual
12

---

Published in Transactions on Machine Learning Research (Auuust/2025)
state of the environment what precomputed. For instance, it might happen that the agent ends up at location which has not been captured in the map. This might lead to the task failure. Moreover, in real-life applications where a robot is expected to perform a task in an unseen environment, such as search-and-rescut operations, it's not idcal for it to spend extra time exploring the environment first and then performing the task. In contrast a better way to build or update the map during the task or online so as to keep updated at all times.
Map building in real world. Maps built in simulation in embodied AI tasks are often noisy due to unrealistic assumptions that limit its usage in the real-world. Map building has been mostly studied in the community as sub-module in conjunction to solving more complex high-level reasoning tasks. Rescarchers have thus tried to investigate what lype of maps are useful for which tasks and decoupling the issue of noisy sensors from map building enables them to do exactly that. The most prominent of the assumptions is that of noiseless sensors. For example, sometimes the community assumes perfect localization (agent's ' current. location and orientation) at all times during navigation, which is unrcalistic in real-world. This is mainly because GPS and Compass sensors are generally noisy, whenever available. However, in most indoor spaces, GPS might not even be available. SLAM methods which work really well in real- weal-world robots operate under the assumption that GPS is not available, and relies on the onboard sensors to estimate its location on the map. Another example of the noiseless sensor assumption is that of a perfect actuation, which means that when an agent initiates an action to move forward by 25cm, it will end up exactly at a location 25cm ahead of its current position. But real-world actuators are noisy and affected by varying friction on different surfaces, which results in significant drifts over time. SLAM systems are inherently capable of addressing such issues by operating under uncertainty in the robot's 's pose estimation. Loop Closure is a sub-algorithm of SLAM which identifies previously visited locations and then uses them to correct accumulated crrors in pose estimation. In general SLAM systems build more consistent and accurate map of the euvironment in real-world setting than the current mapping techniques in embodied AI.
Spatial (54.1)
Ocoupancy Grid (Fitee)
Curious Geange (Meger et al.) Empien (Hu et al.)
ANS (Chappiat MaphNtt [Herriquee Vedoldii
SemEEg (Chaplot all
L2M (Georgakis nily
UPEN (Georgaite
VeaFoser (Huang et
InstructNav GOAT-Bench (Khamnn
Map /hang ni.)
MuttiN (Wenni
SEAL (Chapiot al.)
CH (Grorgnitee
Dol (Gedrre
OVL-MAP (Win
ANR RNR-Map (Kwon et all)
Topological (54.2)
Topo SLAM [Choset Nagetani)
SLAM++ (Selas- Morino al.)
Topomao (Bkochiiger ger os all
NTS (Chapkot et al)
VGM ('wonn et UMPP (Chen ala)
Fast [Chon et al.)
LM Nav (Shah et
Clio (Haggio at.)
ASHITA (Chang et al.) LIRGIF (Raychau dhuri et
Dense Geometric (84.3)
000-5 AM (Niichter al.]
Seemanticff uniio (MoCormmw al.]
Decid-S0 AM (Teed Deng)
PointReesee (Dassi wt al.)
(Shediulish
Hybrid (84.4)
Integrating [Thrun at)
SermanticoZreph (-lomachandia al.)
Inferring maps (Dunallet ot
3D-D8G (Rosinol all
Hytto (Huughs alu)
Concapttiwaps (Gu et ol.)
HDV-SG [Werby etal.)
BEVBert (An wil wl.)
2001
2001-2015
2015-2019
2020
2021
2022
2023
2024
2025
Explicit (85.1)
Closed-vocab Implicit (85.2.1)
Open-vocab Implicit (95.2.2)
Figure 6: This timeline highlights how semantic mapping methods have progressed over the years, with increasing diversity in map structure and encoding. While various map structures have long been explored, the last few years have seen shift toward open-vocabulary semantic maps, driven by advances in large language and vasion-language models.
4 Map structure
In this section we will look at various map structures that have been used in prior works (sce Tab. 1) in more depth. A semantic map can be structured in various ways: spatial grid map, topological map, dense geometric hybrid map. Spatial grid maps are metric maps of the environment. such that its dimensions align to that.
13

---

Published in Transactions on Machine Learning Research Alugust/2025)
Grid Map Building
eg
Ground Projection
Segment
Note:
Localization
Map Update
Ground Segment Projection
Denoising
Online
Sometimes replaced by Learned
stlo
Planning
Spatial
Features
Topological Map Building
Map Update
Control
(Gupta ol., 2017)
Action
Data Association
node update node
less noisy than 2,3
Offline
G = (v. e)
Encode
Ground Projection
Segment
Planning
Localization
Segment+
Ground Projection
Control
Action
Ground
Segment
Projection
Dense Geometric Map Building
(Cartillier et al., 2021)
Task Task-specific Planning and Control
Map Update
Topological Map Contents
Segment
ego map
Image
Nodes:
Image features
(M.).
Detected objects
Point-cloud construction
Detected regions
allo aup
connectivity
Edges:
Localization
(e....e.)
spatial
relationships
Figure 7: Grid map building. A spatial grid map has dimensions (M N K) where M and N are spatial dimensions and K is the number of semantic channels. The common pipeline to build the map is to segment the input image, then ground project into an egocentric map egomap. which is then registered to the allocentric map allomap via map updale using the localized agent. pose. A denoising step generally follows and the map built online along with the task planning and control. While Gupta et al. (2017) learns a spatial representation without segmenting and ground projecting, Cartillier al. (2021) obscrves that encoding followed by ground projecting and then segmenting reduces noise in the produced map.
Topological map building. A topological map is graph-like structure, G (v, with nodes (r (UI,..., )) and edges (e (E,,,.,e))) that can be built Online Offline with before task-specific planning and control respectively. Based on current observation, the agent performs localization on the graph and then performs matching ( (Dala Association) with the current node, based on which either a new node is created or an existing node is updated. The nodes and edges may contain varions types semantic information thus enabling decision making in a task.
Dense geometric map building. A dense geometric map accumulates semantic information directly onto the 3D geometry of scene, most popularly using a point-cloud.
of the environment and they can be structured as either 2D or 3D grids. Topological maps, on the other hand, represent the environment through a set of landmarks represented as nodes and relation between adjacent landmarks represented as edges the form of graph. Dense geometric maps are the densest form of 3D maps whose all three dimensions align the 3D space such that each 3D point in the scene captured in the map. Two more of these types of maps sometimes combined together to form Hybrid maps.
14

---

Published in Transactions on Machine Learning Research (August/2025)
4.1 Spatial grid map
A spatial grid map my is a (Mx x N x K) matrix where M and N are the spatial dimensions of the map and K denotes the number of channels to store semantic information at that location. It is a grid like structure where each cell has a width and a height which correspond to a certain area in the physical environment. Current research on indoor embodied AT make use of environments from datasets such as Matterport3D (MP3D) (Chang et al., 2017) and Sabieat-Matterporttt (HM3D) (Ramakrishnan et al., 2021) which are 3D reconstructions of real-world spaces. Compared to MP3D, HM3D contains around 10x more scenes with high visual fidelity and lesser reconstruction artifacts. These environments are typically for houses or office spaces with total arca of < 1000m². A grid map with each cell representing an arca of 400 - 9000mm is found to be good enough to represent such spaces (Wani et al., 2020; Raychaudhuri et al., 2023). At the start of each episode, the spatial map is initialized with a tensor of size (M x N x K); and gradually built as the agent moves around the environment. These are often 2D top-down maps (Gupta et al., 2017; Henriques & Vedaldi, 2018; Narasimhan et al., 2020; Cartillier et al., 2021) with the first two dimensions corresponding to the spatial dimensions of the environment. However, some build 3D spatial maps (Chaplot et al., 2021) to capture the vertical dimension, in which case the map my is a 4D tensor (M N x P x K).
A spatial map may be built in mumber of ways depending on whether raw features are directly projected onto the map, whether semantic segmentation is used (see Fig. 7). One way is to learn an egocentric projection of the image fcatures as in CMP (Gupta et al., 2017) which forms the egocentric map. In CMP, the egocentric observations are first encoded with a learned image encoder network such as ResNet (He el al., 2016)), and then the network learns to predict an egocentric projection of the image features, without explicit supervision on the map. Instead, the mapper trained end-to-end along with the planner to predict actions. Egocentric maps, however, not only fail to capture the global structure of the environment, but 'forget' most of the past observations. Thus in long-horizon planning tasks, where the agent needs 'remember' its past. observations for efficiency, egocentric maps fall short.
To maintain an allocentric map, it is necessary to take the egocentric information at each time step and aggregate it into a global map. One way to achieve this is to first obtain egocentric projection of image features and then aggregate to a global allocentric map of the environment via a process known as registration. Registration allows the map to incorporate new observations on to specific grid cells. In case the grid cells are already occupied, the new observations are accumulated with the existing ones by employing an aggregation function. This aggregation function can be as simple as taking the latest or the average, but can also be learned network (see Sec. 3.4). Map.Net (Henriques & Vedaldi, 2018) builds allocentric map by projecting the egocentric image features using depth observations and known camera intrinsics on a 2D top-down grid. This ground projection results in an egocentric projection on a spatial neighborhood around the camera. Next it performs registration by first obtaining a stack of egocentric maps rotated r times and then performing a dense matching with the allocentric map from the previous step to obtain the agent 's current pose on the map. The dense matching efficiently implemented with convolution operators. A LSTM then performs the aggregation of the current observations rotated by the current posc with the allocentric map from the previous step. While an LSTM is used in this work for aggregation, other functions and neural architectures can be used for aggregation as well (see Tab. 3 for a summary of aggregation methods used in different works).
Another way to build an allocentric spatial map is to first convert the image pixels to 3D coordinates in the camera space with known camera intrinsics. The camera coordinates are then converted to world coordinates with known camera pose. Finally the 3D world coordinates are voxelized and projected on a top-down 2D grid with a known projection matrix by summing over the height dimension. This approach is followed in Semantic MapNet (Cartillier et al., 2021) and MOPA (Raychaudhuri et al., 2023). Spatial maps built this way might be noisy due to noisy sensors and need additional denoising step. Semantic MapNet uses a Icarned denoising network while MOPA employs hcuristic approach by selecting the centroid of a noisy cluster to obtain a clean map. Following on the last technique, Semantic MapNet (Cartillier et al., 2021) shows that first encoding the image, followed by projecting on to the ground plane and finally performing segmentation on the 2D map reduces noise in the spatial map thereby climinating the need for an additional denoising step. Irrespective of the approach, may happen that multiple image features are projected on to the same grid cell. In such cases, it is important to have a scheme for aggregating the features. Some common
15

---

Published in Transactions on Machine Learning Research (Auuust/2025
approaches to aggregation is to take the maximum (Henriques & Vedaldi, 2018; Wani et al., 2020; Cartillier et al., 2021), mean (Huang et al., 2023a), the sum of the feature values (Chaplot et al., 2020a).
Table 3: Spatial grid maps. Prior works build spatial grid maps (2D or 3D) for various embodied tasks.
Information aggregated onto the map over time many different ways such as learned recurrent nelworks and replacing with most recent information among others.
Method
Task
Environment
Dataset
Aggregation
Efficient (2013)
3D scone analysis
robot
VMR-
most recent
Oakland (2011),
Freiburg (2012)
CMP (2017)
Point.av, Object.Nav
Custom simulator
S3DIS (2016)
weighted mean
MapNet (2018)
Mapping
Doom (2016)
Active Vision
LSTM
Dataset (2017)
ANS (2019)
Exploration
Habitat (2019)
Gibson (2018).
channel-wise max-pool
MP3D (2017)
MultiON (2020) SemExp (2020s)
MultiON
Habitat (2019)
MP3D (2017)
clement-wise max-pool
ObjectNav
Habitat (2019)
Gibson (2018),
channel-wise max-pool
MP3D (2017)
SemanticMapNet (2021)
ObjectNav, Visual QA ObjectNav
Habitat (2019) Habitat (2019)
MP3D (2017)
GRU most recent
L2M (2021)
MP3D (2017),
Object-
Nav (2020b)
SEAL (2021)
ObjectNav
Habitat (2019)
custom with channel-wise max-pool
Gibson (2018)
scenes
CM2 (2022c)
Vision and Language Nav
Habitat (2019)
VLN-CE (2020)
most recent
RNR-Map (2023)
Visual Nav
Habitat (2019)
Gibson (2018)
menn mean
Le-NNR-Map (2023)
Visual Nav
Habitat (2019)
Gibson (2018)
VoxPoser (2023b)
Table-top manipulation
SAPIEN (2020), robot
custom
MOPA (2023)
MultiON
Habitat (2019)
HM3D) (2021)
most recent
GOAT-Bench (2024)
Multimodal Object.Nav Nav
Habitat (2019)
HM3D
most recent
Sem (2023)
Instruction-guidedd (2025)
Vision-lagguage navigation
Habitat
VLN-CE (2020) max-pool
Summary. Spatial grid maps capture dense information about the environment. Such representations are useful for the agent to better reason about the spatial structure of the environment. However, the spatial maps need to be initialized with a certain width and height and as such is hard to scale if the environment size changes. Moreover, it consumes a lot of memory which might affect agent performance in the task.
4.2 Topological map
Compared the high-precision grid maps, topological maps are graph-like structures with nodes connected to each other by edges. This essentially abstracts large space into significant areas (nodes) where the agent can take decisions and connections paths between them (edges) (Johnson, 2018). This enables parsing the environment into local and a global structure such that the agent can plan locally in the small space represented as nodes while navigating the large space through graph search following the edges. This way of planning and navigating is inspired from how humans navigate in an unseen environment in that they identify and memorize significant landmarks and find paths reach those landmarks (Janzen & Van Turennout, 2004; Foo et al., 2005; Chan et al., 2012; Epstein & Vass, 2014). Moreover, in instructioon-fllooinng tasks, where robot follows instructions (e.g., 'go to the yellow chair in front of you'), the language instruction is often represented it uses probabilistic model to update this graph, leading to improved task performance (Tucker et al., 2019; Patki et al., 2019; Arkin et al., 2020a; Raychaudhuri et al., 2025). Thus topological maps have been a popular choice in traditional robotics research such as SLAM (Thrun Montemerlo, 2006; Rosinol et al., 2020b; Campos et al., 2021), manipulation (Arkin et al., 2020b; Patki al., 2020), object search (Aydemir 2011; 2013; Lorbach et al., 2014), language grounding (MacMahon et al., 2006; Chen & Mooney, 2011; Tellex et al., 201 la; Matuszek et al., 2013; Duvallet et al., 2013; Gong & Zhang, 2018; Paul et al., 2018) as well
graph with high initial uncertainty. As the robot moves and observes its environment,
al.,
16

---

Published in Transactions on Machine Learning Research (Auuust/2025)
in embodied AI research (Savinov et al., 2018; Chen et al., 2021; Chaplot et al., 2020b; Kwon et al. 2021; Gu et al., 2024; Mehan et al., 2024; Garg et al., 2024; An et al., 2024; Yang et al., 2024; Tang et al., 2025).
Table 4: Topological maps. Various works in indoor embodied AT build topological map either in an exploration phase or online while performing the task. The nodes often store explicit, learned features about the observation temporal information (visitation timestep), while edges may store relative poses between pair of nodes types of edges.
Method
Task
Map Building Node values
Node Feature Encoder
Edge values
Phase
Topo SLAM (2001)
SLAM
Online
Pose
X
SLAM++ (2013)
SLAM
Online
Pose, object type
x
Imitation (2013)
Instruction following
Online
Pose
x
TopoMap (2018)
SLAM
Online
free space
X
SPTM (2018)
ImageNav
EecEExploration
Image features
ResNet18 (2016)
Compact (2019)
Instruction following
Online
Semantic attributes
Multimodal (2020s)
Instruction following
Online
Semantic attributes
NTS (2020b)
ImageNav
Online
Image features
ResNet18
Relative pose po- coordinatee Relative pose in dis- crete polar coordi- nates (8 directions, 3 distanccs (0-2m.2. 5m,5-5m)
CMTP (2021)
Instruction following
Tre-Exploration
Imnage features
ResNet152
VGM (2021)
ImageNay
Ouline
Imagee feutures, visitation ResNet18 timestepp
Fast (2022)
Path planning
Pre-Exploration Online
free space ImageNode stores im- age features: ObjectNode and object en- stores feutures for de- coders
x pretrained image
TSGM (2023)
TmageNav
tected objects
LM-Nav (2023)
Instruction following
Online
Image features Image features for each
CLIP (2021)
RoboHop (2024)
Language querying
Online
CLIP,
DI-
edge types denot-
image segment
NOv2 (2023)
ing inter- and intra- image connectivity
Clio (2024)
Mobile manipulation
Ouline
object semantics
CLIP
LIFGIF (2025)
Instruction following
Online
Semantic attributes
ASHITA (2025)
Task planning
Online
Task, sub-task, objects
MobileCLIP (2024)
A key design decision during topological map building is what should be represented as nodes and what should be edges. Generally speaking, the nodes encode semantic information about locations in the environment such that the agent can make a decision whereas the edges store relat ionship or connection between the nodes. For indoor navigation, the landmarks for the nodes are typically objects in the environments. They can also be openings intersections (Fredriksson al., 2023), locations the agents has visited (Chaplot et al., 2020b), and other regions of interest (Kim et al., 2023; Shah et al., 2023; Garg et al., 2024). For navigation, two nodes are connected with an edge if it is possible to navigate from one node to another. Some methods also store spatial relationships between the nodes (Gu et al., 2024) in the edges to enable better reasoning.
One way to construct a topological map (see Fig. 7) is during an exploration phase previous to the actual task and then use the graph to plan a path the node most similar to the target, for example, in Semi-Parametric Topological Memory (SPTM) (Savinov et al., 2018). During exploration the agent follows multiple random trajectories for each environment to form node for every visited location and add edge between the current node and the previous one to encode connectivity or reachability between them. A common prot-processing step includes trimming out redundant nodes and edges to form a sparse graph (Chen et al., 2021). When the graph of one environment is collected from multiple random trajectories, it is also common to merge these graphs into one. However, a topological map generated this way in a pre-exploration phase still sparse meaning that some observations the environment might not have boen captured by the graph. This affects the agent performance in the downstream task. Moreover, they need pre-exploration phase which makes them unsuitable for unseen environments.
17

---

Published in Transactions on Machine Learning Research (Auuust/2225)
To mitigate this issue some works construct the topological map online while the agent is navigating during performing the task as is the case with Neural Topological SL.A (NTS) (Chaplot et al., 2020b). NTS consists of several modules- "Graph Update' to update the topological map from observations, 'Global Policy' to sample subgoals on the map and 'Local Policy' which outputs discrete navigation actions to reach the subgoal. The *Graph Update' method gradually updates the nodes and edges in the graph from the current. observations and agent poses. It first attempts to localize the agent on the graph from the previous timestep. If the agent gets localized in an existing node, it adds an edge between that node and the node from the last timestep. It also stores the relative pose between the two nodes represented as (r, 0) where r is the relative distance between the nodes and f is the relative direction. If the agent is unable to be localized, a new node is added to the graph.
Another important aspect in the topological map creation is how to determine if two observations are similar to each other, in which case the two are mapped to the same node. If they are not similar, two different nodes exist for the two observations. This requires the map building methods to compare RGB images. The goal here is to classify two images as similar if (1) they are exactly the same or (2) there is slight change in direction or distance between the two. Traditionally this is the problem of data association in SLAM-based systems, where an incoming obscrvation could be matched to multiple landmarks (nodes) and either the best match is selected or a new node is created to mark a new landmark (Bowman et al., 2017; Dellaert et al., 2000). In embodied AT some works use a pretrained classifier network to implement this. The network is trained to classify whether two images are from the same area. NTS uses MLP trained with a cross-entropy loss in supervised manner to predict whether are similar. This however needs annotated pairs of training data. Cross-Modal Transformer Planner (CMTP) (Chen et al., 2021); on the other hand, uses oracle 'Reachability Estimator' to first obtaín the geodesic distance between the two underlying locations based on the traversibility of the 3D mesh. If the distance is below threshold, it maps them to the same node. Visual Graph Memory (VGM) (Kwon et al., 2021) also uses a pretrained network to determine if two images are similar. But they learn an unsupervised representation of the observations which are then projected onto an embedding space. The idea is to have the embeddings of observations coming from nearby areas clustered together because they are likely to have similar a appearances. The training data in this case consist. of randomly sampled observations from the training environments, thus eliminating the need for manual annotations. Kim et al. (2023) on the other hand use semantic similarity score obtained from pretrained network (Li et al., 2021a) ween two images to determine whether they are the same nodes. A similar approach is taken in LM-Nav (Shah et al., 2023) where they use CLIP to calculate the cosine similarity between image features. Tab. 4 compares different methods that build topological maps.
Summary. In summarry topological maps are convenient to build and maintain due to their concise and condensed representation when compared to spatial maps. They are memory-efficient and can easily be scaled as the environment size increases by simply adding more nodes to the graph. However, they capture only certain landmarks in the environment and as such lack dense global information. This might lead to overlooking visual cues in a cluttered indoor scene that could be helpful for the agent to carry out spatial reasoning.
4.3 Dense geometric map
Semantic information can also be directly accumulated onto the 3D geometry of a scene using triangle meshes (Valentin et al., 2013), surfels (Stückler & Behnke, 2014) or point clouds (Jatavallabhula et al., 2023), with point cloud maps being the most widely used due to simplicity and case of use. While these representations are not always traditionally considered "maps", they effectively function as semantic maps by coupling geometric structure with semantic content. Dense geometric map representations, particularly point clouds, have long been widely used in robotics for tasks such as mapping, localization and navigation, as well as in broader 3D scene understanding tasks (Peng et al., 2023; Xu et al., 2024). Recently, there has been a surge of interest in applying these representations to embodied AI for downstream reasoning and recision-making (Gu et al., 2024). We further categorize dense geometric map representations into point cloud maps (Sec. 4. and neural fields (Soc. 4.3.2). Point cloud maps store 3D points with associated semantic labels whereas neural fields represent scenes as continuous functions using neural networks (see Tab. for a summary of prior works).
18

---

Published in Transactions on Machine Learning Research (August/202)
Table Dense geometric maps allow storing semantics the 3D geometry of scene. It can be further categorized into point-cloud maps and neural fields. Here, VL is disual-ndd-laguaaee Str is Structure, PC is point-cloud, SF is surfel, GS is gaussian splat and NF is neural fields.
Method
Task
Env robot
Str
Encoding
Train
6D-SLAM (2007)
SLAM
PC
geometry
Towards semantic maps (2008)
SLAM
robot
PC
3D location, category
label, rohot pose
Robust 3D mapping (2009)
SLAM
robot
PC
geometry
3D object-class map (2012)
SLAM
robot
SF
category label
Parsing (2012)
Semantic segmentation
robot
PC
3D location, category
label
Street Scenes (2012)
Semantic segmentation
simulator
PC
3D location, category
label
Dense 3D (2014)
3D reconstruction
simulator
PC
3D location, category
label
SementicFusion (2017)
SLAM
robot
SF
3D location, normal, category label
Proid-SLAM (2021)
SLAM
robot
PC
geometry
Voldor+ SLAM (2021)
SLAM
robot
PC
geometry
Semantic-NeRF (2021)
Scene understanding
Habitat, Replica
NF
category label
NeSF (2021)
Semantic sugmentation
Kubric
NF
category label
3D aware ObjNav (2022a)
Object navigation
Habitat
PC
3D location, semantic
category, consistency information
ConceptFusion (2023)
Rearrangement, Replica, robot tonomous driving, semantic segmentation Open-vocab scene un- derstanding
PC
3D) location, normal, confidence, color, pixel aligned VL features
OpenScene (2023)
ScanNet, MP3D, muscenes robot
PC
CLIP
LERF (2023)
Open-vocab derstanding
NF
learned neural VL fea- tures
LGM (2023)
Open-vocas mobile robot
GS
CLIP dense patch fea- tures
nipulation
CLIP-Fields (2023)
Object navigation Habitat, robot
NF
learned neural VL fea-
tures
LangSplat (2024)
Open-vocab semantic segmentation
LERF
GS
VL features from CLIP, SAM
SemanticGaussians (2024)
Open-vocab scene un- derstanding
ScanNet, LERF
GS
VI. features from CLIP, SAM, LSeg
GaussNav (2025) GeF' (2024)
Visual navigation
Habitat
GS
category label
Open-vocab mobile nipulation
robot
GS
MaskCLIP (2022)
SGS-SLAM (2024a)
SLAM
robot
GS
category label
GaussianGrasper (2024b)
Open-vocab tabletop manipulation
robot
GS
CLIP
GeomGS (2025)
Localization
robot
GS
geometric confidence
4.3.1 Point cloud maps
Point clouds have become increasingly popular the years due to their ease of use. Given 3D point cloud, semantic labels can be associated with each point and thus by coupling gcometric structure with semantic content, they serve as a dense semantic map. It also possible to spatially extend each point to a 3D Gaussian with a covariance matrix. Such representations are called Gaussian splats, and allow for differentiable rendering. Point clouds (and Gaussian splats more recently) have been adopted in a variety of robotics tasks as well as 3D scene understanding tasks, gaining popularity in the embodied AI communtty in recent. years.
19

---

Published in Transactions on Machine Learning Research (August/2055)
In robotics, point cloud-based mapping has undergone significant evolution, driven by advances in sensing, computational power and SLAM algorithms. Early efforts (Nüchter et al., 2007; May et al., 2009) use 3D sensor scans to build geometrically consistent representations of large-scale environments by emphasizing accuracy and loop-closure. As real-time visual SLAM has matured, dense point cloud generation from RGB-D sensors became leasible, enabling detailed indoor reconstructions (Whelan el al., 2015; Dai et al., 2017; Schops et al., 2019; Min & Dunn, 2021). More recently, learning- SLAM systems like Droid-SLAM (Teod & Deng, 2021) integrate deep feature extraction and end-to-end optimization to improve SLAM. CoFusion (Rünz & Agapito, 2017) introduce systems for dense SLAM with semantic fusion, pushing toward maps that are not only geometrically accurate but also semantically meaningful. This trend has been acceeratee recently with Concept Fusion (Jatavallabhula et al., 2023), which aligns ppen-oocabulary visual features (e.g., CLIP) with 3D reconstructions to create lask-agnostic and queryable maps. These efforts represent a shift from purely geometric maps toward dense, semantically-rich downstream tasks. Moreover, Gaussian splatting has recently emerged as a powerful alternative to traditional point cloud representations in robotics, enabling high-fidelity scene reconstruction and differentiable 3D representations useful for tasks such as visual localization, SLAM, and manipulation. Recent works such SGS-SLAM (Li et al., 2024a) demonstrate how to include semantic labels onto Gaussian splats to effectively perform SLAM, and GaussianGrasper (Zheng et al., 2024b) demonstrates how these representations can be fused with semantics to support open-vocabulary grasping-
open-vocabulary representations that support a wide range of
In embodied AI, maintaining point cloud maps typically prove to be memory-intensive and not ideal for real-time use. However some efforts have been made to improve efficiency. For instance, Zhang et al. (2023a) builds 3D semantic scene representation based on an online point cloud-based construction algorithm (Zhang et al., 2020), made efficient by using a trce-based dynamic data structure. This method is very memory efficient and lot faster when applied to the ObjectNav task. Recent efforts also use Gaussian splatting in navigation tasks. For example, Lei et al. (2025), based on Gaussian splatting, achieves state-of-the-arr performance on the ImageNav task.
In 3D scene understanding, point cloud representations have become a cornerstone due to their flexibility and fine spatial resolution. Early learning- based approaches such as PointNet (Qi et al., 2017a) and PointNet-+ (Qi et al., 2017b) enable direct processing of unordered point sets for tasks like semantic segmentation and object classification. These were followed by more expressive models like KPConv (Thomas et al., 2019), which introduce learnable convolutions, and Point Transformer (Zhao et al., 2021), which leverage attention mechanisms to capture long-range dependencies across points. More recent work integrates multi-modal and language supervision into point cloud understanding(Pout et al., 2023; Xu et al., 2024), thus enabling fine- grained in infernnce such as semantic segmentation, affordance prediction and 3D object search. OpenScene (Peng et al., 2023), for example, prodicts dense 3D features so that they are co-embedded with the corresponding text. and the image in the CLIP embedding space. This allows the association of each 3D point in the scene with semantic information such that the scene can be queried using text to infer physical properties, affordances, etc. CL.IP (Chen et al., 2023c) also uses CL.IP to perform a 3D point cloud segmentation on outdoor scenes for application in autonomous driving. However, because the optimization happens per scene in many of these methods, they are lypically expensive and not suitable for real-time use in embodied applications.
Summary. Point cloud maps remain a widely used representation in the research community due to their simplicity, high fidelity and direct correspondence with sensor data. However, challenges persist in handling noise, achieving scalability in large environments and maintaining consistent semantics over time. Future research should aim to integrate multi-modal features and improve robustness to dynamic scenes in point cloud maps, all the while maintaining efficiency.
4.3.2 Neural fields
Neural fields are continuous functions that map spatial coordinates (and somctimes viewing directions) to signals such as color, occupancy or semantic features. Unlike point clouds that discretely store information, neural fields encode entire 3D scenes as compact, continuous functions. Often, a neural network (typically MLP) is used to obtain the features associated with each point given the position (x.yzz as input (Mildenhall et al., 2021; Mescheder et al., 2019). There are typically two strategies to train neural fields 1) use distillation (Kerr et al., 2023; Qiu et al., 2024) to provide features that are similar to pretrained 2D backbone, such
20

---

Published in Transactions on Machine Learning Research (August/2025)
CLIP (Radford et al,, 2021), and 2) use of differentiable renderer to match the rendered semantics in addition to color (Zhi et al., 2021; Vora et al., 2021).
In robotics, recent works (Sucar et al., 2021; Zhu et al., 2022; Rosinol et al., 2023) have integrated neural fields into SLAM systems for real-time mapping. While these enable compact, high-fidelity 3D reconstruction and view synthesis, they are more computationally intensive and less interpretable than traditional methods. Thus challenges remain in training speed, memory demands and dynamic scene handling. In contrast, neural Signed Distance Fields (SDFs) are specific type of neural field that encode the signed distance to the closest surface, enabling precise geometric reasoning. Neural SDFs have been shown to be particularly suited for tasks like mapping, collision checking, and planning duc to their ability to represent detailed geometry with differentiable distance metrics (Ortiz et al., 2022; Camps et al., 2022; Talha Bukhari et al., 2025). In embodied AI, a representative work using neural field maps inchude CLIP-Fields (Shafiullah et al., 2023), which learns a continuous, neural implicit representation of 3D environments by aligning 2D image features from CLIP with 3D points in space using volumetric rendering. This enables the agent to build an open-vocabulary, queryable 3D map where each spatial point can be semantically queried using natural language, without needing task-specific supervision.
In 3D scene understanding, neural fields have emerged as a powerful representation from early works on NeRF (Mildenhall et al., 2021) to incorporating semantics into neural fields, enabling tasks such as semantic segmentation, object discovery and scene parsing in 3D (Fu et al., 2022). More recent advances (Wang et al., 2022; Kerr et al., 2023) introduce language-supervieed neural fields by integrating CLIP features, enabling open-vocabulary semantic reasoning and text-driven queries over 3D scenes.
Summary. Neural fields are gaining popularity in robotics and embodicd AT for building continuous 3D scene representations that capture both geometry and semantics. However, challenges remain in real-time training and deployment in dynamic environments. An important future direction would be to focus on making neural fields more efficient, adaptable, and compatible with foundation models for open-world semantic reasoning.
4.4 Hybrid maps
So far we have seen how prior works structure maps as either a spatial grid or a landmark based scene graph or a more dense geometric map. However there is a more recent effort on combining these different structures into single map representation. This helps to capture information at various granularity and perform different types of reasoning on the environment.
The combination of metric information from based maps together with a topological map is also known as topometric map (Thrun et al., 1998b; Tomatis et al., 2001; Blanco et al., 2008b; Konolige et al., 2011; Ko et al., 2013; An et al., 2023; Mozos et al., 2007; Vasudevan & Siegwart, 2008; Zender et al., 2008; Pronobis et al., 2010; Hemachandra et al., 2011; Walter et al., 2013; 2014; Hemachandra et al., 2015; Duvallet et al., 2016; Walter al., 2022). Thrun et al. (1998b) proposes a single statistical mapping algorithm that first constructs coarse topological map and uses it to construct a fino-grained grid map. They show that the topological map solves global alignment problem by correcting large odometry errors, while the grid map solves a local alignment problem by producing high-resolution maps. Tomatis et al. (2001), on the other hand, proposes compact environmental model where corners and hallways are represented by a topological map and rooms are represented by a grid map, both of which are connected in a single representation. When the robot is moving in hallways, it creates and updates the global topological map, and as soon as enters a room, it creates a new local metric map'. They argue that the robot will only need to be procise inside rooms (e.g. manipulating objects, etc.) which justifies the need for the fine-grained precise metric maps, while the topological map is used to simply maintain global consistency in indistinguishable spaces such long hallways and transitioning between significant places. BEVBert (An et al., 2023) is more recent method that constructs hybrid maps offline and then learn a multimodal map representation to perform better spatial reasoning in the complex task of language-guided navigation.
Thhey differentiate hallways and rooms by using laser sensor such that thin long open spaces are considered hallways whereas other open spaces are considered as rooms.
21

---

Published in Transactions on Machine Learning Research (August/2025)
There are also hybrid maps that combines grids, dense geometric, and topological maps. StructNav (Chen et al., 2023b) builds such a hybrid map where the spatial grid stores occupancy information, scene graph stores landmarks with their connectivity, and a 3D semantic point cloud where each 3D point in the environment has a semantic label associated with it. Thus the spatial grid allows for obstacle avoidance and low-lovel path planning, the scene graph allows high-level reasoning about the relationship among the landmarks, and the 3D point cloud allows for a more dense semantic and spatial matching.
It is also possible to build maps that capture the semantic hierarchy of scenes at various levels of abstrac- tion (Galindo et al., 2005; Pronobis & Jensfelt, 2012; Hemachandra et al., 2014; Werby et al., 2024) that allows for various levels of reasoning. Armeni et al. (2019) represent a static scene at multiple levels of hierarchy (buildings, rooms and objects), where entities are represented as nodes in a hierarchical graph and connected by edges representing coordinate frame transformations. Tang et al. (2025) (OpenIn) build a similar hierarchical scene graph to track objects in dynamically changing indoor environments. Rosinol et al. (2020a) (Dynamic Scene Graphs) too build layered scene graphs to track moving agents and objects in addition to building a dense 3D metric spatial map by modeling spatio-temporal relations between objects and agents. Hughes et al. (2024) demonstrate that hierarchical scene representations scale better than flat representations in large environments and thereafter introduces a system called Hydra that incrementally builds 3D scene graphs from sensor data in real-time. Fischer et al. (2024) introduces a multi-level scene graph representation of large-scale dynamic urban environments from a set of images captured from moving vehicles and proposes a new view synthesis benchmark for urban driving scenarios.
Summary. Different types of map structures have their own strengths and limitations. While the coarse topological map is useful to represent significant landmarks in an environment, fine-grained metric maps are useful to represent its precise geometry and dense geometric map is useful to represent an even more dense 3D geometry of objects in the scene, Therefore it's crucial to combine two or more of these structures in order to create better representations of the environment, preferably at varying levels of semantic a abstraction. While hybrid maps have been explored in robotics, they still remain undor-cxplored in embodied AI. However, weaknesses of these maps need to be considered carefully before combining them. For example, topological and dense geometric maps scale better than a grid map with larger environments, and dense geometric map needs the most and topological map needs the least memory to be stored.
5 Map encoding
In this section we will discuss different ways information is encoded and stored into semantic maps. Irrespective of how the map is structured, the map encoding, i.e. the values stored in the map can be either explicit or implicit. An explicit map encoding is one where the type of information stored is clearly known. An implicit encoding, on the other hand, uses feature embedding that may not be directly interpretable. We now summarize various works that explore these two types of encodings (Fig. 8).
5.1 Explicit encoding
Many prior works store explicit information in the map, such occupancy, which has proven beneficial in obstacle avoidance (Elfes, 1989; May et al., 2009; Chaplot et al., 2019; Georgakis et al., 2022a). In such cases, the spatial map stores a binary occupancy value of 1 or 0 depending on whether the corresponding location in the physical environment is 'occupied' by objects or not.
In an exploration task, however, the agent needs to maximize the explored area in an environment while being efficient, in which cases, the information of whether location has already been explored encourages the agent to explore unexplored areas more. Active Neural SLAM (Chaplot et al., 2019) stores the explored (binary) information in addition to the occupancy information. Active Neural SLAM consists of several modules connected logether to perform the task of Exploration to maximize the explored area. The 'Neural SLAM' module takes as inputs the visual observations and agent pose and outputs a top-down egocentric spatial map by learning to predict occupancy and explored information using a binary cross-cntropy loss. All the modules are jointly trained with different losses. VLMNav (Goetting et al., 2024) similarly stores explored information in a top-down voxel map to demonstrate its generalizability to downstream navigation tasks.
22

---

Published in Transactions on Machine Learning Research (August/2025)
Explicit Encoding
Implicit Encoding
Occupancy
Binary
[0,1]
Closed Vocabulary
Explored
Binary
[o,
Task-level
Image
RGB pixel values
Pretrained
Jui
Encoder
Task-level Pretrained Encoder
Objects
RGB pixel values
One-hot for predefined object types
[0,1....
Open Vocabulary
LVLM
Text labels object types
"bed"
Rooms
One-hot for predefined types
[0.1....m
Open-vocab Detection
Text labels room types
"bedroom"
LVLM
Audio Intensity
Single value in the range
[0-1]
Figure 8: Map Encoding refers to the values stored in the map and can be either explicit or implicit, depending whether the information is hand-sclected or learned feature representation of the observation.
However, in more complex tasks such as ObjectNav, which requires the agent to navigate to a particular semantic category in its environment, storing occupancy and explored information alone might not be sufficient, In such cases, important for the agent identify the semantic category of the object (Meger et al., 2008; Georgakis et al., 2021; Chaplot et al., 2021; Raychaudhuri et al., 2023; Khanna et al., 2024; Zhang et al., 2025). Goal-oriented Semantic Exploration (SemExp) by Chaplot et al. (2020a) additionally stores the seraattic class labels of the objects that the agent identifies through its visual observations. SemExp uses a MaskRCNN (He et al., 2017a) on the RGB observations to predict the semantic categories of the objects and then project these on the map using depth observations. aggregates the occupancy and explored information using element-wise max pooling whereas the semantic categories are overwritten by the latest prediction. A denoising network then used to get the final map. This work demonstrates that using this semantic map predict a long-term goal helps the agent find the goal object category more efficiently. The mapping module is trained using supervised learning with cross-entropy loss the predicted semantic map. Similarly, semantic calegories map also helps more complex longer-horizon task of MultiON. MOPA (Raychaudhuri et. al., 2023) shows that maintaining a memory with objects that the agent observes while moving around is crucial to perform this task efficiently. However, a map built by segmenting the image first and then projecting may result 'label splattering' i.e. noisy category labels splattered across multiple grid cells in a spatial map. This arises mostly due to noisy depth observations and might negatively affect agent performance in a task. Semantic MapNet (Cartillier et al., 2021) finds that projecting encoded features on map and then segmenting produces tasks effectívely. However this method requires explores the environment to build the map first and then use that map in the downstream task. A recent work GOAT (Chang et al., 2023; Khanna et al., 2024) shows that having an object instance map helps to navigate to a goal specified by either language, image or a category label. They achieve this by storing raw images and later using CLIP (Khandelwal et al., 2022) for image-to-language matching and SuperGluc (Sarlin et al., 2020) for imago-to-image matching. MapNav (Zhang et al., 2025) builds an annotated semantic map by storing text labels of the segmented objects and shows that this helps VLM to ground objects better.
noise-free map. They show that this map can be then be applied to two different
additional exploration phase where the agent effectively
23

---

Published in Transactions on Machine Learning Research (August/2025)
While a map with semantic categories help in object navigation, acoustic map storing audio intensity is found to be useful in the Audio-Visus Navigation task (Chen et al., 2020b). Here the map is aggregated by averaging the intensity. In the more complex Interactive Question Answering task, Gordon et al. (2018) find that storing object detection probabilities in a spatial map helps agent performance. They use a GRU recurrent memory lo aggregate the current map with the previous one.
While the above works build explicit spatial maps, some works also build explicit topological maps (Choset & Nagatani, 2001; Blochliger et al., 2018; Chen et al., 2022; Salas-Moreno et al., 2013; Duvallet et al., 2013; Patki et al., 2019; Arkin et al., 2020a; Raychaudhuri et al., 2025; Chang et al., 2025). Kwon et al. (2021) store the visitation timestep in the graph node in order to encode temporal relation between visited locations. This information is then replaced by the latest visitation timestep while aggregating the map.
Summary. The advantage of explicit map encoding is its interpretability and the fact that it allows investigating the type of information that is beneficial for various downstream tasks. However, the type of semantic information to be stored is a design choice based on the task. Moreover, the above approaches require predefined set of categories to be mentioned beforehand to the mapper. This restricts the map to store only limited number of object categories.
5.2 Implicit encoding
Implicit maps store latent features in a semantic map. While most of the prior works use extracted features from a vision model pre-trained on flosod-vocabulary set of object categories, recent methods use features extracted from a pre-trained large vision-language model thus producing flexible open-vocabulary queryable maps. It is also possible to store features that are not necessarily queryable with language, but captures the visual information at that location. One example is RNR-Map (Kwon et al., 2023) which uses a grid map with latent codes that corresponds to a neural field that can be used to render possible views that location.
5.2.1 Closed-vocabulary encoding
These features can be learned from scratch during training. For example, (Wani et al., 2020) learn image features using CNN blocks and use them to build global spatial map of the environment. This is trained end-to-end to predict actions in the MultiON task. On the other hand, the features may also be extracted from a vision model such as ResNet, pre-trained on the ImageNet (Deng et al., 2009) data to encode RGB images. For example, (Gupta et al., 2017) introduces Cognitive Mapper and Planner (CMP) that uses a pretrained ResNet-50 model to encode the egocentric RGB images and then projecting on the map using a differentiable mapper module. In CMP, the learned map encoding is not explicitly supervised but learned conjunction with a differentiable planner. This enables the mapper to learn to store information that is most useful for the planner to perform the task efficiently. Also the map is accumulated over time meaning that the map from one navigation step is integrated into the next using a differentiable warp. Similarly MapNet (Henriques & Vedaldi, 2018) also uses a pretrained ResNet-50 model to extract image features but they build an allocentric global map of the environment instead of an egocentric map. They do so by first ground projecting the features and then registering these into a global allocentric map, updating the map at every navigation step. This model learned end-to-end on the task of localization and trained with series of RGB-D observations and the corresponding ground-truth positions and orientations.
While the above methods build spatial map, other methods similar approach to store implicit features in the nodes of a topological map. Each node stores encoded features from the observations at particular location in the environment. Here too, using a pretrained ResNet encoder to extract RGB image features is a popular choice among prior works (Chaplot et al., 2020b; Chen et al., 2021).
Summary. The advantage of using pretrained ResNet model over learning from scratch is that it has already been trained to encode useful features and thus sample efficient. However, using a pre-trained ResNet is limited by the number of object categories it was trained on.
24

---

Published in Transactions on Machine Learning Research (August/2025)
Table 6: Open-vocabulary maps. Various works build open-vocabulary semantic map using trained as well as off-the-shelf pretrained models in both simulation and real-world robots. These methods use either heuristics- LLM-based planner to perform the downstream taskss.
Method
Environment
Training
VL Encoder
Task ner
Plan-
Aggregation
CoW (2023)
Habitat, RoboTHOR
CLIP
A*
highest
similarity
score
VLMap (2023a)
Habitat, robot robot
LSeg
A*
mean
NLMap (2023a)
X
CLIP, ViLD
LLM-based GPT-4
mulli-view fusion
CraceptGraplus (2024)
A12Thor, robot
CLIP, DINO
highest
similarity
score
VoxPoser (2023b)
Sapien, robot Habitat, robot
OWL-VIT
GPT-4
VLFM (2023)
BLIP-2
pretrained
highest score
similarity
PointNav
CLIP-Fields (2023)
Habitat, robot robot
CLIP
SLAM
weighted mean
LM-Nav (2023)
CLIP
modified Dijk-
CLIP similarity
stra
ConceptFusion (2023)
A12Thor, robot Habitat
CLIP
LLM-based
weighted mean latest
Le-RNR-Map (2023)
CLIP
Random walk
exploration
HOV-SG (2024)
Habitat, robot
CLIP
A
weighted mean mean mean
RoboHop (2024)
Habitat, robot robot
CLIP
Dijkstra
Clio (2024)
CLIP
Dijkstra
OneMap (2025)
Habitat, robot
X
SED
A*
weighted sum with uncertainty- based weights
OVT.-Map (2025)
Habitat
/
LSeg
DD-
weighted
PPO (2019b)
5.2.2
Open-vocabulary encoding
The limitation of the closed-vocabularyy encoding can be mitigated by extracting features from Large Vision-Tanguage Model (LVLM) that was jointly trained vast amount of internet data of images and their text captions, such CLIP (Radford et al., 2021). This allows the map to store information about 'any' object in the environment and eventually be queried via an open-vocabulary text query (Wen et al., 2025) not limited to a predefined set of object categories. For example, an agent may be asked to 'find a red and blue striped zebra toy in the children's room'. It is likely that the agent has not scen a 'red and blue striped zebra toy' during training but can leverage a LVLM to reason about its prior knowledge about zebras, colors and rooms in general. Moreover, recently large language models (LLMs), such as GPT-4 (OpenAI, 2023), have been shown to be able to perform complex task planning. Thus open-vocabulant maps built using LVLM along with LLM-based planners (Taioli et al., 2023; Huang et al., 2023b; Long et al., 2024) have led to recent line of works in embodied agents (Tab. 6, Fig. 9).
A pretrained CLIP model can be used to compute similarity scores between an input image and a natural language description with the highest score corresponding to the most likely image-text match. CLIP has been successfully used in map building where the map stores the similarity scores between each image that the agent observes and the language instruction describing an object. Popularly, these maps are called volue maps. CoW (Gadre et al., 2023) shows that such 2D value map be successfully applied in the downstream task of language-driven ObjectNav in zero-shot manner without any re-training. The planner in this method plans a path to the object when the stored similarity score exceeds a certain threshold. VLFM (Yokoyama et al., 2023) follows a similar stralegy to perform ObjectNav task by using the BLIP-2 (Li et al., 2023) 2D value map to semantically explore the environment. InstructNav (Long et al., 2024) extends this idea to enhance semantic value maps with multi-sourced value maps encoding actions, landmarks, and navigation history, thus improving generic instruction following. VoxPoser (Huang al., 2023b), on the other hand, builds similar 3D value map to efficiently perform table-top robot manipulation. Such map encoding is quite
25

---

Published in Transactions on Machine Learning Research (August/2025)
powerful compared to the previous encodings that stores the semantic labels for predefined set of objects. However, this particular map still lacks the ability to perform spatial reasoning because it performs similarity matching to the entire input image ignoring the semantic information about individual objects in the scene.
Map Building
Inference: Querying the map
Tap-dowe Projection
USeg Text bmscatle
"chell "lacter,
"foor" "wall
VLMap
Depth
Label Set (Stentries
Open--ercatalery
DD
Ruga
Payz
Camera Pose
Ling Visual Encoder (Ironen)
VLMap
Mlle ()
Siilirity
Pee-Phoat Embbedding
Argmax
Imput image
Segmentation Masks
Sren Faptomaioo
Proposal+ Mutthiew Fuston
Open-wochhleryy Queryable Fepresentation
Cipat Preemeel
Toll can' Trecoke oor"
NLMap
Caeyy
"oide faued
1e to
Optirt
Raad raliaad
Nering
pictupooee
graan
ConceptGraphs
Figure 9: Vpea-locab map building. There has been a growing interest to build flexible open- vocabulary maps which can be built once and then used in various downstream tasks during inference. VLMap (Huang et al., 2023a) and NLMap (Chen et al., 2023a) structure their maps as spatial maps while ConceptGraphs (Gu et al., 2024) build a topological map. (Figures reproduced from paper.)
To mitigate this issue, researchers need to develop methods that identify where objects are located in the image, and then extract features for those objects. One way to achieve this is to store feature embeddings for each pixel in the input image. This can be done using LSeg (Li et al., 2022) which outputs pixel-level embeddings given an image as is done in VLMaps (Huang et al., 2023a). The embeddings are then projected into 2D spatial map using depth observations. When multiple points are projected onto the same grid cell on the map, VLMaps aggregates by averaging the features. During inference, they extract object names from the language query and calculate pixel-text similarity scores the map to retrieve the objects of interest. However, these pixel-level embeddings are very dense and can be redundant since not all pixels in an image contain objects. Second, some information might be lost. while averaging the pixel-level embeddings. A third issue is that it does not encode object-level semantics, thus ignoring information about spatial relationships. OneMap (Busch al., 2025) stores patch-level features extracted from SED (Xic et al., 2024) with hicrarchical encoder- based backbone which has shown to capture spatial information better than the transformer-basee architecture. This alleviates the issues of using pixel-based features.
26

---

Published in Transactions on Machine Learning Research (August/2025)
However, a better way is to first identify all objects present in an image, which can be done by using class-agnostic region proposal network to propose regions of interest (objects) in an image. This approach is used in NLMap (Chen et al., 2023a) which uses ViLD (Gu et al., 2021) as its class agnostic region proposal network. For each region of interest, they extract image embeddings using an ensemble of CLIP and ViLD. These features are then stored in a 3D spatial map along with the 3D location and estimated size of the object. They show that this spatial map representation can be applied to any downstream task by performing natural language based query. This is achieved by extracting object names from the query and using them to select the object from the map with the highest similarity scores.
While NLMap builds 3D spatial map, many (Shah et al., 2023; Garg et al., 2024) build topological map following a similar approach. For instance, Grapeps (Gu et al., 2024) is a recent approach that retrieves the objects of interest from the imput image by using the class-agnostic 2D segmentation model, Segment-Anything (SAM) (Kirillov et al., 2023) to obtain candidate masks. These objects then form the nodes of the topologiaal map. The image features for each object can then be extracted by CLIP and DINO (Oquab al., 2023) to be stored in the corresponding node. Additional information about the objects can also be stored in each node. For example, ConceptGraphs store the point cloud of the proposed masks and a caption for cach object as obtained by using LLaVA (Liu et al., 2023a) and GPT-4 (OpenAT, 2023) along with a point cloud obtained by projecting the object mask proposed by SAM into 3D space. After the nodes are formed, edges between the nodes can be constructed depending on whether the objects are spatially related. In ConceptGraphs,, spatial relation between two objects is determined by whether the point clouds of the respective objects have a geometric similarity or overlap. In other words, when certain proportion of points in point cloud of one object lie within distance threshold of that of the second object, the objects can be said to be spatially related to each other and an edge is constructed between the corresponding nodes. The edges additionally store spatial relationship description obtained from LLM. When a newly detected object is found to be similar to a node, it is updated with averaged object features, union of point clouds and the latest object caption. In case it is not similar, a new node is added to the graph. ConceptGraphs show that a single map representation built in this manner can be successfully and effectively used in object grounding, robot navigation and robot manipulation lasks.
Summary. The advantage of open-vocabulary map encoding is that it can be built once and then transferred to several different downstream tasks. It can be queried using an open-vocabulary text effectively and is highly interpretable. However one current limitation is that the computational costs of using large foundation models can be significant.
6 Evaluation
In the embodied AI and robotics literature, very little focus has been given to evaluating the built map (intrinsic evaluation). Instead the focus has been to evaluate the agent performance on the downstream tasks (extrinsic evaluation) through various metrics. This is because these works build semantic maps as an intermediate step while performing downstream tasks, such as localization, navigation, exploration, manipulation and so on. Moreover, most of these works focus on a single task and hence it suffices for them to evaluate the task performance directly without caring about how well the map representation is. However, we think that it is crucial to do intrinsic evaluation based on the accuracy, completeness, consistency and robustness of the built map, in addition to evaluating the task performance. We next discuss the evaluation metrics (both extrinsic and intrinsic) in detail.
6.1 Extrinsic evaluation
This section discusses extrinsic (or task-level) evaluation metrics commonly used to assess agent performance across embodied tasks. In navigation, success is not simply reaching the goal but signaling it intentionally, often via a dedicated action like 'Stop' or 'Found' (Anderson et al., 2018a). Accordingly, Success Rate (SR) is used to evaluate whether the agent correctly signals goal completion, while Success weighted by inverse Path Length (SPL) accounts for trajectory efficiency. These metrics are standard in navigation benchmarks (Beeching et al., 2020: Batra et al., 2020b; Chaplot et al., 2020a;b; Wijmans et al., 2019b; Li et al., 2021c; Krantz et al., 2022; Georgakis et al., 2022a; Taioli et al., 2024; Yokoyama et al., 2024). Additional metrics include
27

---

Published in Transactions on Machine Learning Research (August/2025)
Oracle SR. (OSR), which accounts for when the agent reaches the goal but fails to signal, and Navigation Error (NE); which measures the final distance to the goal. In multi-step tasks or multi-goal navigation, metrics like Progress (Wani et al., 2020; Khanna et al., 2024) or Goal- Conl-Coniition Success (Shridhar et al., 2020; Padmakumar et al., 2022) capture the proportion of successfully completed subgoals. For instruction following, normalized Dynamic Time Warping (nDTW) is used to evaluate how well the agent's 's trajectory aligns with the reference path (Mharco et al., 2019; Krantz et al., 2020; Raychaudhuri et al., 2021; 2025). Tasks involving classification or detection often use precision (how often predictions are correct), recall (how well all relevant targets are found), F1 score (combines and balances precision and recall) and recetver-operating characteristic curve or ROC curve (measures the tradeoff between sensitivity or true positive rate and specificity or false positive rate) (Chen & Mooney, 2011; Magnusson et al., 2009; Matuszek et al., 2012; Tellex et al., 2011b; Wellhausen et. al., 2020) to evaluate prediction quality. In exploration lasks, coverage is used to measure the proportion of the environment observed (Chaplot et al., 2019). In manipulation tasks such as object rearrangement or tabletop operations, task success rate, measuring the percentage of trials where the agent achieves the desired manipulation (e-g., grasping, placing), is the dominant metric (Kalashnikov et al., 2018; Batra et al., 2020a; Lin et al., 2020; Zeng et al., 2020; Huang et al., 2023b; Gu et al., 2023). Some works also evaluate policy outputs using precision (Zeng et al., 2017), or employ pose error metrics to assess deviations between the final and target object positions (Liu et al., 2012).
Summary. Extrinsic evaluation metrics have been extensively studied across the literature (Deitke et al., 2022) and are contimually evolving in response to the growing complexity and diversity of embodied tasks. As new challenges emerge in navigation, manipulation, and instruction following, the need for muanced and task-specific performance metrics has driven the development of more robust and comprehensive evaluation frameworks. Although extrinsic metrics have dominated evaluation in the field, we believe it's time for the communily to place greater emphasis on intrinsic metrics (we discuss this in next section). This shift is especially important given the recent progress in building open- open-vocab,lary, flexible, and general- maps, which require more muanced assessment of map quality beyond task success alone.
6.2 Intrinsic evaluation
In this section, we discuss intrinsic evaluation or map-level evaluation which evaluates the map directly. While most robotic and embodied AI systems are evaluated based on downstream task success, a few focus on directly evaluating the quality of the maps built during these tasks. However, there is widely agreed-upon metric suite for evaluating the map representation itself. We next list the various dimensions across which we believe the map can be evaluated and discuss the gaps and future scope for each.
Accuracy. Map accuracy refers to how accurately the map captures geometric or semantic information when compared against the ground truth. However obtaining the ground truth map can be challenging in most cases. Some works (Georgakis et al., 2022a; Ramakrishnan et al., 2020) evaluate the 2D occupancy maps by using map accuracy to measure how much of that area matches with the ground-truth, and loU to measure intersection over union between the built and the ground-truth map. While others (Cartillier et al., 2021; Georgakis et al., 2022b) evaluate the accuracy of the built 2D semantic map using semantic segmentation metrics such as pizel-wise labeling accuracy, pirel-based F1 score, IoU score and a contour- based average boundary F1 score. However, there is a lack of standardized metrics that can be used across various map types. For example, IoU is not directly applicable to topological maps. Moreover, while evaluating accuracy for semantic maps that store a fixed set of object categories is straightforward, is tricky to evaluate semantic maps that store open-vocabulary features. As an alternative, OpenScene (Peng et al., 2023) performs semantic segmentation evaluate it using accuracy and loU, ConceptGraphs (Gu et al., 2024), the other hand, evaluate their open-vocabulary topological map by employing human evaluators on Amazon Mechanical Turk (AMT) to report the scene graph accuracy. OpenLex3D (Kassab et al., 2025) has been recently introduced to evaluate vocabulca 3D scene representations, by providing object labels at various gramularity or precision. Similar evaluation techniques could benefit intrinsic evaluation embodied sctting as well. The challenge to evaluate modern-day apen-vocabulary maps lies in collecting precise ground-trtth maps for large real-world environments, which is labor-intensive infeasible. This limits the use of direct comparisons in order to
their open-vocabulary map for a fixed set of categories and thereafter
28

---

Published in Transactions on Machine Learning Research (Auuust/2025)
calculate map accuracy. Moreover, accuracy in dynamic or cluttered environments is also rarely assessed. Both these provide possible future area of research.
Completeness. Map completeness measures how completely a generated map represents an environment, encompassing both geometric and semantic coverage. Geometric coverage refers to the fraction of environment that has been mapped whereas semantic coverage refers to the completeness of the semantic information captured in the map. Completeness is particularly critical for tasks like search and rescue, where a thorough understanding of the surroundings is essential. Morcover, complete geometric and semantic mapping support better decision decision-making by reducing dependence on incomplete or inaccurate data. The extent of map completeness depends on how thoroughly the robot explores the environment during the downstream task and is closely tied to the 'stopping criteria' method that determines when to end the exploration process. In embodied AI, exploration typically ends when the task is deemed complete or when a predefined time budget is reached. However, reaching this time budget does not always guarantee that the map is fully complete, making the development of reliable stopping critoria an ongoing challenge in roboties as well as embodied AI research (Placed & Castellanos, 2022; Luperto et al., 2024). Geometric coverage metric has been reported in prior works that performs the task of exploration (Chaplot et al., 2019) in terms of the fraction of the environment explored by the robot. However, a lot of works that tackle other embodicd AI tasks fail to report this metric even if exploration is a crucial part of their task (Gervet et al., 2023; Raychaudhuri et al., 2023; Yokoyama et al., 2023). Semantic coverage on the other hand relies on the presence of a detailed ground-truth map with semantic information and could be hard to obtain, as discussed in the previous paragraph Accurrcyy. Due to the lack of detailed ground-trut truth semantic maps, semantic completeness is rarely quantified.
Geometric consistency of a map refers to how accurately the spatial structure (distances, angles, and relative positions of prructuresoobectts of the map represents the physical layout of the enviromment. Accurate geometry ensures safe and efficient path planning and obstacle avoidance. This is crucial in classical robotics for identifying loop closure, a popular technique to recognize previously visited locations in SLAM, which refines the map and reduces drift. Popular loop closure metrics (Li ct al., 2021b; Qian et al., 2022) include Absolute Trajectory Error (ATE) that evaluates the cumulative error, in addition to precision and recall. However, in embodied AI systems, there is no drift due to absence of sensory and actuator noise and hence the generated maps are mostly consistent with the environment. Hence prior embodied AI works do not report geometric consistency metries for the built map. That said, ensuring geometric consistency will be crucial in future where dynamic moving objects may disrupt the map's structural fidelity. It can be measured by reporting the Root Mean Square Error (RMSE) from the ground-truth geometry. On the other hand, the semantic consistency of a map refers to the alignment between the semantic information of /oruecures//bjectss and their physical locations in the enviromment. Semantic consistency is particularly crucial to remain consistent over time despite changes in perspective, lighting, or environment dynamics when the robot moves around the environment. This could be measured using temporal accuracy metric that measures how accuracy of the semantic information changes over time. However, none of the prior works measure semantic consistency and may be considered in future research.
Consistency.
Robustness. Evaluating robustness in semantic maps is mostly crucial for assessing their reliability in unpredictable or dynamic environments. A robust map exhibits low uncertainty and high confidence in its semantic information, allowing the robot to adapt to errors, sensor noise, and environmental changes. In SLAM, some study system robustness to mcasure Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) under noise perturbations (Prokhorov el al., 2019: 99; Yang et al., 2025). On the other hand, since recent. approaches use pretrained models, measuring the confidence of the model's predictions could be beneficial to assess map robustnoss, with higher confidence indicating a more robust map. Alternatively, model uncertainty can be measured by assessing the variance in model predictions, reflecting epistemic uncertainty in the model, where low uncertainty will indicate a robust map. While some studies in embodied AT (Georgakis et al., 2022a; Raychaudhuri et al., 2025) incorporate uncertainty into task planning, it has not been used as a formal metric in previous research, presenting an area for future exploration. We think that uncertainty- based reasoning in the SLAM and active robotic exploration can provide valuable insights into designing uncertainty-based map robustness metrics. More specifically, joint entropy, expected map information, and uncertainty-auunt mayping all relate to reasoning under uncertainty in the robotics literature. Joint entropy quantifies the total uncertainty over both the robot's trajectory and the map, while expected map information (often expressed
29

---

Published in Transactions on Machine Learning Research (August/2025)
mutual information) captures how much that uncertainty in the map is expected to decrease after future observation. These are widely used as internal utilities for planning (Stachniss et al., 2005; Blanco et al., 2008a; Georgakis et al., 2022a; Carlone et al., 2013), guiding action selection in active SLAM and exploration by prioritizing trajectories that maximize expected information gain. Uncertainty-aware mapping refers to the practice of explicitly representing and propagating uncertainly within the map itself, often using probabilistic models like occupancy grids with confidence estimates, Gaussian processes, or learned uncertainty models. This enables more robust perception and safer navigation, especially in dynamic or partially observed environments. While joint entropy and expected map information are used to decide where to go next, uncertainty-awan mapping focuses on how we represent what we' ve already seen. However, these are rarely reported as evaluation metrics, as they are task-dependent, abstract, and computationally intensive to calculate precisely.
Map evaluation. While extrinsic or task-level evaluation has been extensively studied over the years, there has been little progress in intrinsic evaluation to assess the quality of the built map across various dimensions such as accuracy, completeness, consistency and robustness. In this study, we emphasize the need for standardized suite of intrinsic metrics to be applied across different map structures and encodings.
Table 7:
Evaluation
Dimensions
Metrics
Task
Extrinsic
Utility
Success rate, SPL,, NE, OSR nDTW
Navigation (2020; 2020b; 2019b; 2021:; 2024; 2024) Instruction following (2020; 2025)
QA accuracy, Nav accuracy
Embodied QA (20 2024)
Precision, Recall, F1, ROC
Language grounding (2011; 2012; 2011b)
RPE (translation, rotation)
Localization (2019)
Intrinsic
Accuracy
Map accuracy, IoU, Pixel F1, Boundary F1 Geometric coverage
Semantic segmentation (2021; 2021)
Completeness Consistency Robustness
Exploration (2019) Loop closure (2021b; 2022)
Precision, Recall, True Pasitives, ATE ATE and RPE under noise perturbations
Visual SLAM (2019)
6.3 Summary
In this section we summarize the key challenges and the future scope of evaluation (Tab. 7). Extrinsic evaluation metries have been widely explored in the literature and contimue to evolve with the emergence of increasingly complex tasks. However, evaluating the quality of maps (intrinsic) in semantic SLAM and embodied AI continues to be understudied and presents two key challenges. First, collecting accurate, fine- grained, veen-vocabulrry ground-truth annotations for large, real-world environments is inherently difficult. This limitation makes it particularly challenging to assess important properties of the built maps such semantic accuracy, consistency, and completeness of the maps. Without reliable ground truth, these evaluations often remain coarse or dataset-specific. Second, there is currently no standardized suite of evaluation metrics that can be consistently applied across different types of map structures (topological, spatial grid, dense geometric, etc.) or across different encoding schemes (explicit and implicit fcatures). Together, these challenges highlight critical gap in the field and point toward the need for developing scalable, generalizable map evaluation frameworks, offering an important direction for future research.
7 Challenges
Despite significant progress in semantic map building, numerous challenges remain that warrant further atlention and improvement. This section outlines key open problems in the field across multiple dimensions.
Efficiency. As semantic maps grow richer by incorporating fine-grained semantic information, they become increasingly data-heavy, posing significant challenges for memory efficiency and compact storage, particularly on robots with constrained hardware. The underlying map structure plays a crucial role in determining storage and computational efficiency. Different representations offer trade-offs among semantic richness, spatial fidelity, and scalability. For example, spatial grids uniformly capture space but are memory-intensiv and difficult to scale in large, high-resolution environments. Dense geometric maps provide high spatial fidelity and support per-point semantic features (e.g., color, normals), yet are often redundant and storage-heavy. Tn
30

---

Published in Transactions on Machine Learning Research (August/2025)
contrast, topological maps are lightweight and scalable but lack the geometric detail needed for tasks like manipulation precise localization. Hybrid topometric maps attempt to balance these aspects by combining the spatial accuracy of grids with the efficiency of topological graphs, though managing and updating both layers increases complexity and computational load. Map encoding methods also impact efficiency. Learned high-dimensional representations (e.g,, neural leatures) offer compressed yet expressive encodings, but they are often harder to interpret or update, and computationally expensive to query. Overall, achieving efficient. storage and memory usage in semantic mapping remains an open and active research challenge.
Scalability. As semantic mapping systems are deployed in increasingly large, dynamic, and diverse envi- romments, scalability emerges as a critical challenge. It refers to a system's 's ability to accommodate growth in spatial extent (e.g,, large physical spaces), semantic complexity (e.g., diverse object categories and scene types), and temporal evolution (e.g., long-term and lifelong operation), while maintaining efficiency, accuracy, and robustness. High-resolution, globally consistent maps require significant computational and memory resources, which grow rapidly with the size and richness of the environment. For example, a home robot navigating multiple cluttered rooms over extended periods must. relain detailed spatial layouts and object- level information without exhausting onboard storage or processing capacity. The challenge intensifies in dynamic settings, such as when furniture is rearranged or objects are moved, requiring the map to adapt continuously. Moreover, as maps expand, core operations like loop closure, re-localization and map querying become increasingly expensive in both simulation and real robots. This underscores the need for lightweight, adaptable semantic mapping methods that can scale with environmental complexity while staying efficient on resource-cnnsrranee hardware.
Real-time processing. Real-time processing is critical challenge in semantic map building, especially in applications such as autonomous driving and real-time human-robot interaction, where decisions must be made within strict latency bounds. Semantic mapping in these domains involves constructing maps with semantic and spatial integrity at frame rates fast enough to support safe and responsive behavior. However, this is extromely challenging. For instance, an autonomous vehicle navigating an urban environment must simultaneously detect traffic signals, update the map with moving vehicles or pedestrians, and plan a safe trajectory in milliseconds. Any lag in perception or map update could lead to disastrous decisions. Balancing the tradc-off between semantic richness and processing speed, often memory-constrainee hardware, remains an open research challenge. Advances in model compression, edge computing, and efficient data representations are being explored, but ensuring high-fidelity semantic understanding under tight latency requirements continues to be a major hurdle for real-world deployment.
Noise and uncertainty. Handling noise and uncertainty is a fundamental challenge in semantic map building, particularly when operating in real-world environments characterized by sensor imperfections, dynamic changes and ambiguous semantics. Noise from cameras, LiDARs, or depth sensors can lead to inaccurate observations, while errors in object detection, segmentation, or classification models may introduce semantic inconsistencies in the map. For example, a robot navigating a cluttered indoor environment might repeatedly misclassify similar-looking objects (mistaking a stool for chair) or fail to detect partially occluded objects or objects in poor lighting. Over time, these errors can compound, leading to semantic drift or contradictions in the map. Furthermore, the uncertainty in sensor data and learned representations is often not explicitly modeled or propagated, which limits the system's ability to reason about the confidence of its predictions. This becomes especially critical in safety applications where overconfident yet incorrect map entries can mislead planning and decision-making. While some approaches use Bayesian filters probabilistic representations to quantify uncertainty in robotics (Blanco et al., 2008a; Georgakis et al., 2022b), these are computationally expensive and may not scale well. Morcover, semantic mapping methods in embodicd AI often assume noiseless sensors in simulated environments, and thus do not transfer well to real robots. Thus, building robust semantic maps that explicitly model and manage uncertainty remains a key open problem in both robotics and embodied AI.
Multi-modal fusion. Recent semantic map building approaches increasingly rely on fusing diverse sensory inputs, including vision, depth, audio, natural language, speech etc., to construct rich interpretable represen- tations of the environment. However, robust multi-modal fusion remains core challenge, with alignment and integration between the modalities a non-trivial problem. For instance, visual inputs provide spatial and appearance cues, while natural language and speech often provide high-level semantic cues that may
31

---

Published in Transactions on Machine Learning Research (Auuust/2025)
be ambignous, indirect or context-dependent (e.g., 'go to the room where the baby is sleeping'). Moreover such grounding is often compounded by noisy or incomplete observations. Audio cues such as footsteps, conversations or object sounds offer valuable environmental context but are transient and hard to spatially localize or persistently encode. Morcover, multi-modal fusion needs to be done in a way that supports downstream reasoning and generalization in unseen environments. For instance, enabling agents to query the semantic map for complex spatial reasoning (e.g. 'the cup behind the fruit bowl', 'the vase between the kettle and the dish') and affordances (e.g. 'where can I sit', 'which objects are fragile"). The emergence of multi-modal foundation models (Alayrac et al., 2022; Driess et al., 2023; OpenAI, 2023) offers new promise in addressing some of these challenges. These models are capable of aligning and reasoning across modalities, enabling semantic parsing of complex instructions, visual understanding and basic spatial reasoning. However, they are not trained explicitly for map construction or long-term memory integration, limiting their direct. applicability in real-time embodied settings. Moreover, deploying such large models on robot hardware poses computational and energy constraints, particularly in latency-sensitiee domains like autonomous driving. Thus the challenge lies in building efficient multi-modal fusion strategies that also support reliable and flexible querying, making this an open research problem.
Lifelong learning. Lifelong learning poses significant challenges for semantic mapping, particularly in real-world applications where robots are expected to operate continuously over extended periods in dynamic environments. Unlike static maps that assume a fixed world, real-world settings are inherently non-stationary (furmituure is rearranged, objects are moved, and scenes evolve with time and seasons). To support lifelong learning and long-term autonomy, semantic maps must be capable of continuous adaptation, i.c. updating outdated information, integrating new observations and distinguishing between transient and persistent. changes. This requires temporal reasoning, mechanisms to prevent catastrophic forgetting and the ability to resolve conflicting data collected across different times or viewpoints. The importance of long-term autonomy lies in its potential for practical deployment, i.e. robots that can adapt to changing homes, warchouses or outdoor environments without manual reconfiguration. It is a foundational requirement for truly intelligent agents that not only act within their world but grow and evolve alongside it, building richer and more useful semantic representations over time. However, most current systems are not yet equipped with robust mechanisms for continuous semantic learning and remain limited in handling evolving, open-world environments.
Standardized evaluation framework. A significant challenge in semantic map building lies in the absence of standardized evaluation frameworks that can consistently benchmark performance across diverse tasks, environments and map representations, as we discuss at length in Sec. 6.2. While the community has focused more on lask-specific extrinsic evaluation, limited progress have been made towards developing intrinsic measures to evaluate the map quality. This lack of standardized intrinsic evaluation hinders progress, as it becomes unclear whether improvements in downstream task success stem from better semantic understanding other factors like better control policies. Developing general, interpretable and task-agnostic evaluation metrics is crucial, not only to track progress but also to facilitate cross-domain generalization. As open- vocabulary multi-modal queryable cencral-purposs. semantic maps become incrcasingly central to reasoning in embodied AI and real-world robots, creating a unified benchmark suite that reflects both structural and semantic fidelity is essential for advancing the field.
Summary. Several critical challenges remain unaddressed in semantic mapping approaches across various dimensions. Key open problems include building efficient, scalable and real-time maps, which are robust. to noise and uncertainty. Building multi-modal maps, that are reliable and flexible, too remains open challenge. Maps that are able to handle lifelong learning also remain under-explored. Morcover, lack of a standardized evaluation framework require further research and innovation.
8 Future research direction
In this section we highlight the current challenges in semantic map building and outline potential directions for future rescarch. Although semantic map building has advanced significantly over the past decades, several challenges and opportunities for improvement remain. The field is evolving toward creating maps that are flexible, general-purpose, open-vocabulary and queryable, enabling the same map representation to support.
32

---

Published in Transactions on Machine Learning Research (August/2025)
wide range of downstream tasks. This shift aims to make maps more versatile and suitable for complex, multi-task robotic systems. Moreover, to enable efficient reasoning about the spatial and semantic structure of the environment, the focus is also on developing dense, scalable, and memory-effccient maps. Such maps should maintain high resolution and detailed spatial information while being computationally efficient and consistent across dynamic and large-scale environments. Achieving this balance is critical for applications that require real-time processing or operate in resource-conttrained settings. Furthermore, the emphasis has largely been on evaluating agent performance in downstream tasks using various metrics, with limited attention given to assessing the quality of the built maps. It is however essential to evaluate semantic maps beyond their utility for specific tasks, focusing on metrics such accuracy, completeness, consistency, and robustness to ensure they are reliable and effective for broader applications. Next, we provide an in-depth discussion of potential future directions that we believe are most critical for advancing research in semantic mapping.
8.1 General-purpose maps
Creating general-purpose semantic maps in robotics and embodied AI is crucial for enabling robots to perform a wide variety of tasks in diverse environments with minimal reconfiguration. The idea is to design a general-purpose semantic map that serves as a single comprehensive representation of the environment, combining spatial geometry and semantic information. This eliminates the need for task-specific maps, making it easier to reuse the same map for different tasks such as navigation, object manipulation, and scene understanding. To enable this, the maps need to be vocabvocab that allow the robots to understand and integrate previously unseen objects using natural language descriptions. This capability broadens the scope of the downstream tasks the robot can perform, especially in unstructured or novel environments. Such semantic maps provide an opportunity to thoroughly evaluate their ability to handle textual queries involving complex spatial and semantic reasoning. Despite the recent progress towards achieving this, it still remains an open research problem. Open-vocabulary maps are currently limited by the pretrained alassagnootic object detectors used in building such maps. For example, these detectors often struggle with detecting small, thin obscure objects, thus limiting the semantic maps that rely on them. Moreover, open- vveebvocyy object detectors that incorporate unseen classes through textual descriptions are still not perfect and lack robustness. Thus improving open-vocabulary object detectors that can recognize new objects without extensive retraining could improve the quality of semantic maps that rely on them and hence present a future research avenue. Another challenge that general-purpose map building face is that they are computationally expensive and memory-intensive due to the rich semantic and geometric data stored in them. Balancing map detail with resource efficiency is challenging task and impacts the ability of robots to process large areas continuously update the map in real-time without overwhelming computational resources. This presents future research direction worth pursuing.
8.2 Dense yet efficient maps
Following our discussion on various map structures, we find it crucial that the semantic map be able to capture dense visual cues to allow for complex spatial reasoning among objects. For example, beyond addressing straightforward queries like 'Where is the table?', the semantic maps can be assessed on more intricate spatial reasoning queries such as 'Can you retrieve my phone from my desk beside my laptop?: To perform such reasoning, the semantic map needs to capture fine-grained detail about the spatial arrangement of the objects (phone and laptop on the table) in the scene. While at one end of the spectrum topological maps are too sparse failing to capture the dense semantics of a scene, at the other end the dense geometric maps are too dense capturing rodundant empty space information. Although spatial top-down 2D maps cxist somewhere in the middle, they fail to capture the semantic information in the third dimension (height). This is crucial where the navigation is in 3D space, for example in drones. Hence there is still a need of a dense-enough map to capture the 3D space in its entircty at the same time intelligently ignoring empty space information. Additionally, a dense map will consume more memory and will be difficult to update. So a dense map representation which is still scalable, memory and computation efficient is research direction worthwhile to pursue.
33

---

Published in Transactions on Machine Learning Research (August/2025)
8.3 Dynamic maps
Current mapping techniques in indoor environments assume that the objects present in the environment are static and only the agent is moving- Although this assumption is reasonable in an indoor environment, is unrealistic in an ouldoor setting in the presence of moving vehicles and people. This entails investigating how well current map building approaches capture moving objects effectively in dynamic environment and focus building efficient dynamic maps. Building such maps involves continuous tracking and updating of objects in real-time, as they may move unpredictably. Sensor fusion, which integrates data from multiple sensors like LiDAR R. and cameras, is often used to detect and track these objects. However, real-time updates can be computationally intensive, particularly in high-traffic arcas. Thus, efficiently storing and representing dynamic data in a way that is both memory and computationally scalable remains a key challenge and an ongoing area of research. Moreover, the dynamic nature of these maps complicates their use downstream tasks. For instance, robot may need to navigate around moving pedestrian vehicle, which requires understanding the object's 's trajectory and predicting its future movements to avoid collisions. Efficiently integrating this dynamic data into decision-making processes is a significant challenge in autonomous navigation.
8.4 Hybrid map structure
A spatial map is able to capture the geometry of a 3D space, which helps to reason about complex spatial relations among objects and areas. A topological map, on the other hand, lacks such geometric spatial understanding but is able to explicitly capture semantic relationships (edges) among objects (nodes). Since both structures have different merits, there have been research to explore a 'hybrid' map structure that leverages the geometric accuracy of spatial maps with the semantic and relational power of topological maps, providing a more comprehensive and elficient tool for complex reasoning lasks. For example, a large-scale outdoor environment, a robot could use the topological map for long-range navigation to reach one building from another, while switching to spatial map for rlose-range navigation to avoid obstacles or interact with objects in room. A hybrid approach are less resource-intensive and can provide high-level guidance, while spatial maps can be used for precise actions in local regions of interest, reducing the need for continuous, high-cost processing across the entire environment. Moreover, maintaining a separate level of hicrarchy to track dynamic objects can also reduce computation load of frequent real-time updates. Although several approaches to hybrid mapping have been proposed in recent years, one of the main challenges in creating hybrid maps is effectively integrating both spatial and topological representations without sacrificing the quality of either. Moreover, combining the two representations requires the robot to make intelligent decisions about when to transition from one to another. Hence significant research is still needed in optimizing hybrid map building, ensuring scalability for large-scalc, rcal-time applications and intelligent algorithms to transition between the different maps.
also help balance the computational load. Topological maps
8.5 Devising evaluation metrics
As we discuss in Sec. 6, the evaluating semantic maps in embodied AI research has received limited attention compared to assessing agent performance in downstream tasks. However, believe that advancing the field requires stronger emphasis on map evaluation using metrics such as accuracy, completeness, consistency, and robust ness. Regardless of the rowmstream lask, maps should be assessed on how well they capture semnantic information (accuracy), their geometric and semantic coverage (completeness), their spatial and semantic reliability in dynamic environments (consistency), and their confidence and ability to handle uncortainty and noise (robustness). Establishing standardized evaluation metries and frameworks for semantic maps remains critical challenge with substantial opportunities for future research.
8.6 Summary
This section outlines promising future directions, as derived from the current challenges semantic mapping in Sec. 7. As the field moves toward creating flexible, general-purpont and queryable maps, researchers face open problems in balancing semantic richness with efficiency, ensuring scalability and robustness in dynamic
34

---

Published in Transactions on Machine Learning Research (August/202))
environments, and developing better evaluation metrics that go beyond task performance to assess map quality directly. These provide multitude of future research directions, which will help advance the field.
9 Conclusion
In this survey, we review a wide range of semantic map-building approaches and categorize them based on their underlying map structure, such as spatial grids, topological graphs, dense goometric and hybrid representations, as well as the nature of semantic encoding, such as explicit or implicit. This perspective is timely the light of recent. advances foundation models and the increasing need for general-purpose, multi-modal, open-vocabulary and queryable map representations. This survey helps identify key challenges in current semantic mapping paradigms and present promising directions for future research in semantic mapping.
Existing works primarily employ spatial maps to capture geometric layouts or topological maps to model landmark-based relationships. While many robotics studies have explored hybrid maps that combine spatial and landmark information, this approach remains under-explored in embodied AI research. It presents a promising avenue for future work, particularly in leveraging such maps to enhance performance on complex spatial reasoning tasks. We also discuss how dense geometric maps, created by associating semantie information with point clouds, triangle meshes or surfels, offer potential for embodied AI tasks but remain under-explored. While promising for spatial reasoning, their high memory demands, computational inefficiency, and the presence of mostly empty 3D space in indoor environments pose significant challenges. Morcover, most current mapping approaches assume static environments, limiting their applicability in dynamic settings, for example in scenes where furniture is rearranged or humans move around.
We hope that the insights presented in this survey serve guide and inspire further advancements in the field by the research community.
The authors were supported by a Canada CIFAR AI Chair grant and an NSERC Discovery Grant. We also thank Manolis Savva, Yasutaka Furukawa, Tommaso Campari, Austin T. Wang, Xingguang Yan, Bernadette Bucher, Duy Ta and Sachini Herath, and the anonymous reviewers for their valuable foedback on our paper.
Acknowledgements
References
Abdessalem Achour, Hiba Al-Assaad, Yohan Dupuis, and Madeleine El Zaher. Collaborative mobile robotics for semantic mapping: survey. Applied Sciences, 12(20):10316, 2022. URL https: httppp:///www.ddee
2076-3411/12/2//10811 1
Muhammad Farhan Ahmed, Khayyam Masood, Vincent Fremont, and Isabelle Fantoni. Active slam: A review on last decade. Sensors, 23919)88097, 2023. URL https: //wp: http://www.mpp....//////. com/ com/1424- 1424-422002222 10/20/23/19/8000 6, 7
Jean-Baptiste Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: vi- sual language model for few-shot learning. Advances in Neural Information Processing Sys- lems, 35:23716 23736, 2022. URL httphhttpp/pprcceenggg.. neurips..... cc/paper seaaer_ffles_.e http./ss///apee/22////llee Pawpaaaaaaaoooaa-a-a-oeee-eeee
pdf. 32
Phil Ammirato, Patrick Poirson, Eunbyung Park, Jana Košecká, and Alexander C Berg. A dataset for developing and benchmarking active vision. In IEEE International Conference on Robotics and Automation, 16
pp. 1378-1385, IEEE, 2017. URL https: https:// //ieee/p/reeeeeeeeeeee/ee ieee. org/ http///aett/bcuuue/teeeaaee
Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. BEVBert:
Multimodal Map Pre-training for Language-guided Navigation. Proceedings of the IEEE/CV International Conference Computer Vision, 2023. URL https: htttps://githin /ggggttth....... com/MarSaKi SaaaaaSaai//Li//C WINN-BEVerr.. 3, 21
Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, and Liang Wang. ETPNaY: Evolving topological planning for vision-language navigation in continuous environments. IEFE Transactions
35

---

Published in Transactions on Machine Learning Research (August/2025)
Pattern Analysis and Machine Intelligence, 2024. URL httpp///ieeepplorr.eeeeoorr/r/ee document/10495141 17
Abhishek Anand, Hema Swetha Koppula, Thorsten Joachims, and Ashutosh Saxena. Contextually guided semantic labeling and search for three-dimeniional point clouds. The International Journal of Robotics Re- search, 32(1):19 34, 2013. URL pettps:/jjurnnne .aggluu..bebbb.o...... sagepub.com//oi/ftunni /tta///////a11111.117///33399999
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra. Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. Xixiv preprint Xitx:180.066777 2018a. URL https://arxiv http:////////...o///// 27
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Tan Reid, Stephen Gould, and Anton van den Hengel. Vision-and-languag navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pattern Recognition, pp. 3674-3683, 2018b. URL attp://openaccees. thecvf. com/ homp/contntt_ee wwwpcvp011010 html Andd/Andesoon__oo Banderson_Visionne Vision-ana LangLagee Navigation
cVppeeeeiin________eeee 2018 1arp1__ppprr____ html. 5
Jacob Arkin, Daehyung Park, Subhro Roy, Matthew R. Walter, Nicholas Roy, Thomas M Howard, and Rohan Paul. Multimodal estimation and communication of latent semantic knowledge for robust execution of robot instructions. The International Journal of Robolics Research, 39(10-11): 1249--1344, 2020a. URL https://journas wwgeep....essseuu...... com/ doi/
/tt//////a//////aa/o aaa/a//sa000//////////////////// 1177 1177700778364920917755 3, 16, 17, 24
Jacob Arkin, Rohan Paul, Dachyung Park, Subhro Roy, Nicholas Roy, and Thomas M Howard. Real-time human-robot communication for manipulation tasks in partially observed environments. In Proceedings Springer, 2020b. URL https:
of the International Symposium on Experimental Robotics, pp.
//link. //link.springer. http:/pppaaa/.o.o/77//-----00--- 0-0---3-0000-350---- 0_99. 16
Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, loannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference on computer vision and pallern recognilion, pp. 1534-1543, 2016. URL. htttp://openceee www.ctttccccccooooeeoeeeo com/content. cvpr_2016/ wypp_2016/hcmm/Ammmnn 30t_Semaati__arssiieaeeeee CVPR__CV______ 1efe__211___________ hapeerppprr... html. 16
Tro Armeni, Thi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3D scene graph: A structure for unified semanties, 3D space, and camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1664-5673, 2019. URL http: http:///ppeacceess thecvf. www content wwwtoonneentoooooooooooooooeoeee _tt___CCCV_______ 20t/2200199/m/11 wwwll///meen_eeeee Sermeni_3D_Sccee 3D_Scene_______eeeee____eee_____ Loo_Sceee_Grah___ Graph_A_ Structure_for.2 Uww.uiiiddddeemtiiiaaneeeeeeee
Sacpcee_ccv____1___e paperr html. 3,
Nikolay Atanasov, Scan L Bowman, Kostas Danillidis, and George J Pappas. A Unifying View of Geometry, Semantics, and Data Association in SLAM. In Proceedings of the International Joint Conference Artificial Intelligence, pp. 5204-5208, 2018. URL http:///rrl ucdd. edu/ref/ Aauuref/Atanasoo /atp//tttaaaaateaaee Sww.stttaaanaa......... IJCAI18. pff.
Alper Aydemir, Kristoffer Sjöö, John Folkesson, Andrzej Pronobis, and Patric Jensfelt. Search in the real world: Active visual object search based on spatial relations. Tn IEFE International Conference on Robolies and Automation, pp. 2818 2824. IEEE, 2011. URL attps://xeeexplore ieeeeereeeeeeeeeeeeeeeeeeeeeeeee org/ abatrract//oocmmmnt/ 5980495. 16
Alper Aydemir, Andrzej Pronobis, Moritz Göbelbecker, and Patric Jensfelt. Active visual object search in unknown enviromments using uncertain semantics. IEEE Transactions on Robotics, 29(4):986-10) 10006, 2013. 16
URL https::///// /tt::///eeeeeeeeeeee leee. .ow. http://ataattt/ccceee///oo//
Somil Bansal, Varun Tolani, Saurabh Gupta, Jitendra Malik, and Claire Tomlin. Combining optimal control and learning for visual navigation in novel environments. In Conference on Robot Learning, pp. 420-429. PMLR, 2020. URL http://proceedings mlr. prwss/ wttpa/////oo//baa/a0/ae0
36

---

Published in Transactions on Machine Learning Research (August/2255)
Sid Yingze Bao and Silvio Savarese. Semantic structure from motion. In Proceedings of the IEEE/C/ /VFF Conference on Computer Vision and Poltern Recognition, pp. 2032.2032. TEEE, 2011. URL https:
// //ieeexplrre.iess www.ooreeeiee..o... htt..////.aaaaccc/ccae/e/e//aaae
Sid Yingze Bao, Mohit Bagra, Yu- Wei Chao, and Silvio Savarese. Semantic structure from motion with points, regions, and objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2703 2710. IEEE, 2012. URL https: /ttes///eeeeeepeeeeeeo ieee. .ew...rgeeeeee.ee...........o /t//aabatt///ocmme////////////// 6247992/.
Zhibin Bao, Sabir Hossain, Haoxiang Lang, and Xianke Lin. review of high-definition map creation methods for autonomous driving. Engineering Applications of Artificial Intelligence, 122:106125, 2023.
URL htt.://titt////ra.aaoaaaaaaeerre http::/wwweeeeeeee..c....aaaaaaa
Dhruv Batra, Angel X Chang, Sonia Chernova, Andrew J Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. Rearrangement: A challenge for embodied ai. arXiv preprint orXic:2011.01975 2020a. URL https:/ http://aaxivvoooaa///////ae/ 01975. 28
Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. ObjectNav revisited: On evaluation of embodied agents navigating to objects. arviv preprint 10111100..1111,, 2020b. URL
13171.
5, 16, 27, 30
Hriday Bavle, Jose Luis Sanchez-Lopez, Muhammad Shahoer, Javier Civera, and Holger Voos. S-graphs+: Real-time localization and mapping leveraging hierarchical representations. IEEE Robolics and Aulomation Letters, 49944449744 2023. URL https://orbili unni wwtpaaannee/099///// 10993/5439999
Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Lue Van Gool. Speeded-up robust features (surf). Compuler vision and image understanding, 110(3):346 359, 2008. URL https: httpss//www..eeeeedieee... wwt.e/iicccccccneceeeeeeeeeeeeee (rrzccle/p1//11777337700000111
6
Edward Beeching, Jilles Dibangoye, Olivier Simonin, and Christian Wolf. Learning to plan with uncertain topological maps. In European Conference Computer Vision, pp. 473 490. Springer, 2020. URL
27, 30
Jens Behley, Volker Steinhage, and Armin В Cremers. Performance of histogram descriptors for the classification of 3D laser range data in urban environments. In JEEE International Conference on Robotics and Aulomation, pp. 4391-4398. IEEE, 2012. URL 6225003/. 16
ieee. org/ atggaaa/cccctt/mmeet/////
José-Luis Blanco, Juan-Antonio Fermández-Madrigal, and Javier González. A Novel Measure of Uncertainty for Mobile Robot SLAM with Rao Blackwellized Particle Filters. The International Journal of Robotics Research, 27:73 89, 2008a. URL
:55.rps 5143649. 4, 30, 31
Jose-Taiis Blanco, Juan-Antonio Fernández-Madrigal, and Javier Gonzalez. Toward unified bayesian approach to hybrid metric topological slam. ΣEEE Transactions on Robotics, 24(2):259 270, 2008b. URL https://leeexplore. ieee.ee..... www..ergeeeeeeeeeeee......ooo..e htt..//...../aaaatr////oooatttta /ttpa/act/occcmeet///a////eaeeae 44422211/ 21
Fabian Blochliger, Marius Fehr, Marcin Dymezyk, Thomas Schneider, and Rol Siegwart. Topomap: Topological mapping and navigation based on visual slam maps. In IEEE International Conference on Robotics and Aulomalion, pp. 3825.. IEEE, 2018. URL /ttees///ieeepploree ieee. org/ wwgg/bastract/tocuueeee 8460641/. 3, 17, 24
Sean L Bowman, Nikolay Atanasov, Kostas Danillidis, and George J Pappas. Probabilistic data association for semanntcc SLAM. In JEEE International Conference on Robotics and Automation, pp. 1722-1729. IEEE, 2017. URL https: h/tp:://eeeexeoree... leee. ott....eeeeeee..e..eee...eeeeeee wttp://asstrrrt/oooou/e/////////
7, 18
Finn L Busch, Timon Homberger, Jesús Ortega-Peimbert, Quantao Yang, and Olov Andersson. One Map to Find Them All: Real-time Open-Vocan Vocabulary Mapping for Zero-shot Multi-Object Navigation. In IEEE International Conference on Robotics and Automation, 2025. URL
11764.
25, 26
37

---

Published in Transactions on Machine Learning Research (August/2225
Cesar Cadena, Anthony Dick, and lan D Reid. A fast, modular scene understanding system using context- aware object detection. In TEFE International Conference on Robolics and Aulomation, pp- 4859 4866. LEEE, 2015. URL
leee. org/ httpr////sstaacccooooete////////
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, Ian D Reid, and John J Leonard. Simultaneous localization and mapping: Present, future, and the robust-perceplion age. IEFE Transactions on Robolics, 32, 2016. URL https://core... wtt.///......../oooooooooooooo wttp:///wwwwooopp/////////o///// /107999911199999991/ pdf. 6
Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzzaa, José Neira, lan Reid, and John J Leonard. Past, present, and future of simultancous localization and mapping: Toward the robust-perception age. IEEE Transactions on robolics, 32(6):1309 1332, 2017. URL https://com ac ww/.w//wwwwodddl / 130797919. -df.. 1,
Lecai Cai, Yuling Ye, Xiang Gao, Zhong Li, and Chaoyang Zhang. An improved visual SLAM based on affine transformation for ORB feature extraction. Optik, 227:165421, 2021. URL
Carlos Campos, Richard Elvira, Juan J Gómez Rodriguez, José MM Montiel, and Juan D Tardós. ORB-SLAMB. An accurate open-source library for visual, visual-inertial, and multimap SLAM. IEEE Transactions Robotics, 37()))1874 1890, 2021. URL 7, 16
wetpppoorreeeeeee...eeeeeeeeeeee org/ attst/ccc///cmmm///4400/2/
Gadiel Sznaier Camps, Robert Dyro, Marco Pavone, and Mac Schwager. Learning Deep SDF Maps Online for Robot Navigation and Exploration. Xi preprint arXiv:2207. 10782, 2022. URL httth:t/aaxxii..oo 1ttp22277.17780
Luca Carlone, Jingjing Du, Miguel Efrain Kaouk Ng, Basilio Bona, and Marina Indri. Active SLAM and Exploration with Particle Filters Using Kullback-Leibler Divergence. Journal of Intelligent & Robotic Systems, 75:291 - 311, 2013. URL https: stt.apppissmmannscchhhllrarra :ttp://cc.........oooooooooooaaa org/CorpusID:2271733503
4, 30
Luca Carlone, Ayoung Kim, Frank Dellaert, Timothy Barfoot, and Daniel Cremers. SLAM Handbook, chapter 1. Cambridge University Press, 2024. URL https://github hlam-handbook-pae ppublic-releaa 5
github....b.... www.ttu......ett.e com/SLLM-Bandbook -ondi contributors/
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021. URL htttps://peeaacess.oo in Ln_Seeff-Supervised Visiid wtt.eaiiiiiiiiiiiiieo Transformers I_____ICV________ _CC__2221__pprrrr
wterscccessssteeteeeeeee thectf wwtpc///ooonet///////2.22 http:/22///ttm/a//araeee Eerrroin__ernnnnn..e Emergging.PPoperties
Vincent Cartillier, Zhile Ren, Neha Jain, Stefan Leo, Irfan Essa, and Dhruv Batra. Semantic MapNet: Building allocentric semantic maps and representations from egocentric views. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021. URL attps://oje aaw.////.aaaa........a/.aaaaaa.. wttpaa..///..ee.pppp///aa//////a php/AAAt//artticc////ee/////11 3, 11, 14, 15, 16, 23, 28, 30
Artificial Intellgence Center. Shakey the robot. wikipedia, 1984. URL http:///wwiiklpdda..rrrr////// org/wikk// Shakey_the thbotttttt..oeeeeee
robot.
Edgar Chan, Oliver Baumann, Mark Bellgrove, and Jason Mattingley. From objects to landmarks: The function of visual location information in spatial navigation. Frontiers in Psychology, 3:304, 08 2012. doi: 10.3389 (w..3389ffpsy.22111 201syg20122003044 URL 00304/ful1. 16
htt.c//ese/....////ppssse.e 2012.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGB-D data in indoor environ- ments. In International Conference on 3D Vision, 2017. URL https://collaborate,, princcton..nnn edu/ httpppulaatoo///tttaapptttaaaaaa htgp::/eeeeeettt.oooooooraaattte
15, 16
38

---

Published in Transactions on Machine Learning Research (August/2025)
Matthew Chang, Theophile Gervet, Mukul Khanna, Sriram Yenamandra, Dhruv Shah, So Yeon Min, Kavit Shah, Chris Paxton, Saurabh Gupta, Dhruv Batra, et al. GOAT: Go to any thing. or Xiv preprinl arXiv:231.006430, 2023. URL https:/ http//awwv.oogga///////// 06430. 23
Yun Chang, Leonor Fermoselle, Duy Ta, Bernadette Bucher, Luca Carlone, and Jiuguang Wang. ASHITA: Automatic Scene-grounded Hlerarchical Task Analysis. In Proceedings of the Com- puter Vision and Pattern Recognition Conference, PP. 29458 29558, 2025. URL https: //openaccess thecvf. com/ theeraaccsss.eeccf.....
/Caann. ASHiTA COTrTAAAttmaticcccm Automatic__cene-eeeeee ctene-grounded. Hlerarchical_ Laaiicallaas_____ Aaaaskknaalasss...aaaaaaaaaaaaea _ctrss__________ SUPPR_2025_PPP0 2atp__222__ppper..___ html. 3, 17, 24
Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov.
Learning to explore using active neural SLAM. In International Conference on Learning Represenlations, 2019. URL https://oppnreviii http:////pppaeeeeeeeeeeeoerrrree
3, 5, 9, 10, 16, 22, 28, 29, 30
Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In Advances in Neural Information Processing Systems, volume 33, pp. 4247-4258, 2020a. URL https: https://pprcceeinng www.cceeinngn.nnuuaae www.neuurriccoooomo ctw.ppp.c/ppppp/2220000 /
c Abstttttt--abatttt..aaaa.aaaaa.e html. 3, 9, 10, 16, 23, 27
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupla, and Saurabh Gupla. Neural Topological SL.MM for Visual Navigation. In Proceedings of the IEFE/CVP Conference on Computer Vision and Pattern Recognition, 2020b. URL http://openacceess thecvf.e.. wwcoo//ootett_teeeeee _t____C2RRR_________ 22200002mmm http//htll/cCht /Chaplot_ Tooraal_ooooggcc__SLAAA Lariicaal_sAM_____e Sat__for__iss_l_aeeeeeee Wis__iislll__aa Narilllaiviggatoo_vVPP carpppo____crr_ee___ppeeee... paper..ppppp.. html. 3, 5, 10, 17, 18, 24, 27
Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta, Jitendra Malik, and Russ R Salakhutdinov. SEAL: Self-supervised embodied active learning using exploration and 3D consistency. Advances in Newral Information Processing Systems, 34:13086-13098, 2021. URL https: / hetess//pcceedigss.. srrrceedins..euurpss.coo www..ne..ccppppppeeeoeee
-tasttttttttttt-ttrttttt.... Abstract ctmaraacttt...............m... 3, 15, 16, 23
Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S Ryoo, Austin Stone, and Daniel Kappler. Open-vocabulary queryable scene representations for real world planning. In IEEE International Conference on Robotics and Automation, pp.
IEEE, 2023a. URL
https://1seexpl.oo ieee. .rgo arg//occuent/101615554. 3, 25, 26, 27
Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. SoundSpaces: Audio-visual navigation in 3D environments. In European Conference Computer Vision, pp. 17-36. Springer, 2020a. URL. https: //lin///nnn/////// springer. com/ con/chapter/1D 00t/////000000//77000000008-8880
5
Changan Chen, Sagnik Majumder, Ziad Al-Halah, Ruohan Gao, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Learning to waypoints for audio-visual navigation. In International Conference on Learning Representations, 2020b. URL https://openreviii www/ffeuutiiiiccccm ctt.dd=c99F9FFF... 24
Changhao Chen, Bing Wang, Chris Xiaoxuan Lu, Niki Trigoni, and Andrew Markham. A survey on deep learning for localization and mapping: Towards the age of spatial machine intelligence. ar preprint arXiv:2006.122: 12567, 2020c. URL https://arrat /ttesv///xivv.v/ cetriivoorr//aa argg/bss/000 12567.
David Chen and Raymond Mooncy. Learning to Interpret Natural Language Navigation Instructions from Observations. Proceedings of the AAAI Conference on Artificial Intelligence, 25(1) 2551):859 865, Aug. 2011. doi: 16, 28, 30
10.1609/aaaie wttp:/aaai............. 79744.7777 URL https: //ojs. aaai. wrr.aaaiiaaaaaaaaaaaaaaeeaeeeeee
orgginoex. index.nexxx.e http://nox..pp//A//ataiiiaee /ttp:///A///aattteee/e//////////
Junting Chen, Guohao Li, Suryansh Kumar, Bernard Ghanem, and Fisher Yu. How To Not Train Your Dragon: Training-fre free Embodied Object Navigation with Semantic Frontiers. In Proceedings of Roboties: Science and System, 2023b. URL https: http://wwwreeeerccccocoeecc
ethz. ch/ ch/handle/20..m. 500. 118.....110000277731
3, 22
Kaiqi Chen, Junhao Xiao, Jialing Liu, Qiyi Tong, Heng Zhang, Ruyu Liu, Jianhua Zhang, Arash Ajoudani, and Shengyong Chen. Semantic visual simultaneous localization and mapping: A survey. IEEF Transactions
39

---

Published in Transactions on Machine Learning Research (August/2225
Intelligent Transportation Systems, 2025. URL https://ieeexplore..ees ieee. http://.......oo.atttttt//ooooa.
11005698. 8
Kevin Chen, Junshen K Chen, Jo Chuang, Marynel Vázquez, and Silvio Savarese. Topo- logical planning with transformers for vision-and-languugee navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11276 11286, 2021. thecvf. www.coo///ooooc Transformers afreerrrrrrrrroeeeeeeeeereeeeeee aassion-add-Laguaae wwteeeeeaaaiannaoneeee Bateaas__________ 2021 _aaplpppppr.___ peaper.html.. html. 3, 17, 18, 24
URL attp://openaccess. theppaac
Topooooioll Ballooiica_laaanngggge With
Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. CLIP2Scene: Towards label-efficient 3D scene understanding by CLIP. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 7020 7030, /ctp////ootat////2/////////
2023c. URL http://openaccess.. thecvf. ctm Label Label-Efficient_5 3D_ Beceeeettt___eeee_e__eeee SrrrricciennsssSSScoeeneetrrrrr Uwwersstandingg.y_yoooo..o CLIP Ctt_y_c_P___________EC____ Bee___CCR____222 _a____2222_____________________e html. 3, 20
http//0022/tmmmhhenn _LIP2Scene_Tovards.. Towards_
Steven W Chen, Guilherme V Nardari, Elijah S Lee, Chao Qu, Xu Liu, Roseli Ap Francelin Romero, and Vijay Kumar. Sloam: Semantic lidar odometry and mapping for forest inventory. JEEE Robotics and Automation Letters, 5(2):612-619, 2020d. URL https: / htt////eeeepporre www.oppooreeeee................m leee. http//absrraac/ddcuaat////e/
7
Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. In International Conference on Learning Represenlations, 2019a. URL
net/ wtw.://eww...t...oooooo//////a/a
9
Xicyuanli Chen, Andres Milioto, Emanuele Palazzolo, Philippe Giguere, Jens Bchley, and Cyrill Stachniss. Suna++: Elficient. lidar-based semantic slam. In IBEE/RSJ International Conference Intelligent Robols and Systems (IROS), pp. 1530-4537, IEEE, 2019b. URL https://ieeen/: //ieee////eee ieee. org/ /bgtractt document/8967704. 8
Xinyi Chen, Boyu Zhou, Jiarong Lin, Yichen Zhang. Fu Zhang, and Shaojie Shen. Fast 3D sparse topological skeleton graph generation for mobile robot global planning. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 10283-10289. IEEE, 2022. URL https://ieeexp.on ieee. /rgg abstract//bccume//99988833
3, 17, 24
Jun Cheng, Liyan Zhang, and Qihong Chen. An improved initialization method for monocular visual-inertial SLAM. Electronics, 10(24)33063, 2021. URL https://www, mdpi
7
Howie Choset and Keiji Nagatani. Topological simultaneous localization and mapping (SLAM)) toward exact. localization without explicit localization. IEEE Transactions on robotics and automation, 17(2):125 137, 2001. URL https://iseexpporee ieee.
3, 17, 24
Javier Civera, Dorian Gálvez- Luis Riazuelo, Juan D Tardós, and Jose Maria Martinez Montiel. Towards semantic SLAM using monocular camera. In IEEE/RS. international conference intelligent robots and systems, pp. 1277-1284 IEEE, 2011. URL https://ieeexpporre ieee. wwwg/bbstract/docmeee 6094648/.
Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Tzadi, and Christian Theobalt. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (ToG), 36(4):1, 2017. URL https://dl. http//lllccmmm////c/////////.. org. doi aoooi//ab/11... htt/a//a....111171///1/// 1145/30725999 000.17999999934799.... 20
Matt Deitke, Dhruv Batra, Yonatan Bisk, Tommaso Campari, Angel X Chang, Devendra Singh Chaplot, Changan Chen, Claudia Pérez D' D'Ario,,, Kiana Ehsani, Ali Farhadi, et al. Retrospectives on the embodied Al workshop. anXiv preprint arXiv:2210.0688 2022. URL https:
1, 28
Frank Dellaert, Steven Seitz, Sebastian Thrun, and Charles Thorpe Feature correspondence: A markov chain monte carlo approach. Advances in Neural Information Processing Systems, 13, 2000. URL https: // //proceedings... wwwpeeedinnn...uuiigge http:///wwpppoooo......to..ttoa.
Abstract html.
18
Frank Dellaert, Michael Kaess, et al. Factor graphs for robot perception. Foundations and Trends@ in Robotics, 6(1-2):1-139, 2017. URL httpp::///et http://wwwnouuuuiissheeheee httpa///eeeeeeeeeeaaaa///aaaaaaa
40

---

Published in Transactions on Machine Learning Research (August/2255
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEER/CVE Conference Computer Vision and Poltern Recognilion, ieee.
pp. 248 255, 2009. URL
24
Aadesh Desai, Saagar Parikh, Seema Kumari, and Shanmuganathan Raman. PointResNet: residual network for 3D point cloud segmentation and classification. arXiv preprint arXix:2211.10440 2022. URL https: wwwp:/v..rr/bbb//21/12 11040. 3
Ovidiu-Aurelian Detesan and Iuliana Fabiola Moholca. Unimate and Beyond: Exploring the Genesis of Industrial Robotics. In International Conference on Robolics in Alpe- Adria Danube Region, pp. 263 271. Springer, 2024. URL https::/ httpe///nnnnssssinre.oomm www.c/////cccc......1
Matej Dobrevski and Danijel Skočaj Deep reinforcement learning for map-less goal-driven robot nav- igation. International Journal of Advanced Robolic Systems, (111::1729881122199222 2021. URL
https://journall htg.://.ourrnasss..eppppp sagepub. com/ cot./oo/aabs//..177///11111 11111/1111771111111119999222 8
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint or Xix: 11929, 2020. URL https: // //arxiv. owtpdd//o//////////////// 22100. 11929/1000..
Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Peter R. Florence. PaLM-E: An Embodicd Multimodal Language Model. In International Conference on Machine Learning, 2023. URL https::/apt semanticscholes org/ org/CorpusII:255/0 2255334442. 32
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied Al: From sinulators to research tasks. IEEE Transactions on Emerying Topics in Compulational Intelligence, 6(2): 230-20 244, 2022. URL
ieee.
Felix Duvallet, Thomas Kollar, and Anthony Stentz. Imitation learning for natural language direction following through unknown environments. In IEEE International Conference on Robotics and Automation, pp. 1053-1053. IEEF, 2013. URL pettps://eeexplleee ieee. leee.eeeeegee
3, 16,
17, 24
Felix Duvallet, Matthew R Walter, Thomas Howard, Sachithra Hemachandra, Jean Oh, Seth Teller, Nicholas Roy, and Anthony Stentz. Inferring maps and behaviors from natural language instructions. In Experimental Robotics: The International Symposium on Experimental Robotics, pp. 373 388. Springer, 2016. URL springer. https://link: https://iik.spprineer com/ c 3-1-788--3311--77---7778 3, 21
Elfes. Using occupancy grids for mobile robot perception and navigation. Computer, 22(6):46-57 1989.
doi: 10.1109/2.30720. URL https: https:///eeet:: //te::::///eeeee//////// ore. ieee, wtg/abbstrract/acceeet///tt
3,
Felix Endres, Jürgen Hess, Jürgen Sturm, Daniel Cremers, and Wolfram Burgard. 3-D mapping with an RGB-D camera. IEEE transactions on robolics, 30(1):177-1877 187, 2013. URL httes://ieeexpllrre ieee.
ot//ABstrac//bocmee///5499////// 7
Jakob Engel, Thomas Schöps, and Daniel Cremers. LSD-SLAM: Large-scale direct monocular SLAM. European conference on computer vision, pp. 834-844 849, Springer, 2014. URL https://limt springer. com/
Russell Epstcin and Lindsay K Vass. Neural systems for landmark based wayfinding humans. Philosophical Transactions of the Royal Society B: Biological Sciences, 369 20335)) 2014. URL https:// royallsocletyuulliiiinnggg wtt.iing..rrr/////////a/ooee 109th/////.0099/ttbbbb 2011 2012. 0533. 16
41

---

Published in Transactions on Machine Learning Research (August/2023,
Nathaniel Fairfield. Localization, mapping, and planning in 3D environments. Ph. D. dis- sertalion, 2009. URL https://citeseerrs pss. wtt.:///wwcma....p...ee.eepeeppa 7
Qiaojun Feng, Yue Meng, Mo Shan, and Nikolay Atanasov. Localization and mapping using instance-specifie mesh models. In JEEE/RS RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4985 4991. IEEE, 2019. URL https://ieeexplln ieeee
Brian Ferris, Dieter Fox, and Neil D Lawrence. Wifi-slam using gaussian process latent variable models. In Proceedings of the International Joint Conference on Artificial Intelligence, volume 7, pp. 2480-2485, 2007. URL httph:tt//////tc //dp: acm. wot.oogg//ooooo doi/ abaaabaaaaa 1aobs//0..am ww5.555112227555 1625675.
Nicola Fioraio and Luigi Di Stefano. Joint detection, tracking and mapping by semantic bundle ad- justment. In Proceedings of the JEEE Conference on Computer Vision and Pattern Recognition, pp. 1538-1545, 2013. URL
thecvf.ccom htnn/content_cvppl wttpeeetttcoeoeeeeeeeeee 2013//ttm///aa/1 httss//tmm/iorroooooioiooo Detection_Trent Teant_rraaannnnn1neeee Bet___cV_________ html. 7
Tobias Fischer, Lorenzo Porzi, Samuel Rota Bulo, Mare Pollefeys, and Peter Kontschieder. Multi- level neural scene graphs for dynamic urban environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp- 21125 21125-21135, 2024. URL teeressssteeeiiteee theccheeeeeee Graphs for Saarr_DDnmm_c__ue Uuban Stweurrrraanuneneeeeeee Bevirrneentssannne CV_R__CRRR______ _02___aa22_________________ _aaprr aemeerrhtmm..... httl. 22
http://openaccess.
Multi-Level Neerrll_ueent
Alex Flint, David Murray, and Tan Reid. Manhattan scene understanding using monocular, stereo, and 3d features. In International Conference on Computer Vision, pp. 2228-222222222 IEEE, 2011. URL https: //ieeexplore. ieee. org/ abstrrac//bccumet///25600
Georgios Floros and Bastian Leibe. Joint 2d-3d temporally consistent semantic segmentation of street scenes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2823 2830. IEEE, 2012. URL https://teeexpl.oo www.looreeeeeee.e.ee... org/ /ttstraact/bocuuee/////
3, 19
Patrick Foo, William H Warren, Andrew Duchon, and Michael J Tarr. Do humans integrate routes into a cognitive map? map-versus madmaar--based navigation of novel shortcuts, Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(2):195, 2005. URL https:/ htsp:::/ppscceet.... apa. org/record/ 2005-02160-002. 16
Dieter Fox, Wolfram Burgard, and Sebastian Thrun. The dynamic window approach to collision avoid- ance. JEEE Robotics Autom. Mag., 4:23-30 33, 1997. URL https: h/ap::///ppii semanticscholar, org/ Darg/CrpusID: 1718553. 4
Scott Fredriksson, Akshit Saradagi, and George Nikolakopoulos. Semantic and topological mapping using in- tersection identification. LFAC-PapersOnLiin 56(2):9251 9256, 2023. URL http///www..ieeneeeeeettt. con/sciiecc//rtiicl///pppo http://c/e//e////////9/03330///0
17
Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and
Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoplic urban scene segmentation. In International Conference on 3D Vision (3DV), pp. 1-11. IEEE, 2022. URL https: //te:expereeeeeeeeeeeoee ieee. arg/abstract/ document/10044395/ 21
Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Iharro,, Ludwig Schmidt, and Shuran Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23171-23181, 2023. URL http:://penaaccess.. theceeffsssseeceet..eeeeeeeeeeee thecvf. com/ http://cnneaet/////20////////aaa
CoWs_on Baatanaasuueeeeae Car Bataaaaasttaaaallliiiaaaaa Baselines
and L____Bnnhmrrrss_f____ Laaguggee_ggg wwwrreenerr--httttte orw.eeacenn......aaoooooooeoeeee
CVcp___CPP_____ 2023__223________2__ wwwp:o22ppppeeepeeee html. 3, 9, 25
Cipriano Galindo, Alessandro Saffiotti, Silvia Coradeschi, Pär Buschka, Anto-no Fernandez- Fernandez-Madrigaal and Javier González. Multi-hierarchical semantic maps for mobile robotics. In JEEE/RS international conference on intelligent robots and systems, pp. 2278-2288 IEEE, 2005. URL https://10eexplore ieee.
org//bbttaact/ooomnn///6511111
42

---

Published in Transactions on Machine Learning Research [August/202))
Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum. Look, Listen, and Act: Towards Audio Visual Embodieed Navigation. arXiv preprint ar Xrv: 11684, 2019. URL https: 5
//ieeexplore meteexpporeeieeee.... wtt.:/reeeeeeeeeeeaeeeeeeeeeeeee http:////bb.aatc//ooo/a/////////
Sourav Garg, Niko Sünderhauf, Feras Dayoub, Douglas Morrison, Akansel Cosgun, Gustavo Carneiro, Qi Wu, Tat-Jun Chin, lan Reid, Stephen Gould, et al. Semantics for robotic mapping, perception and interaction: A survey. Foundations and Trends@ in Robotics, 8(1-2):1 224, 2020. URL https:
//www. www.nnooibbsser..... homm/arriiccle/eeals/////
Sourav Garg, Krishan Rana, Mehdi Hosseinzadeh, Lachlan Mares, Niko Sünderhauf, Feras Dayoub, and Ian Reid. Robohop: Segment-based topological map representation for open-world visual navigation. In IEEE International Conference on Robolics and Aulomalion, 2024. URL attps://ieeexplee iee.eorrreeeeerrr............
abstraact/documen///uee 106100234. 3, 17, 25, 27
Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomas Lozano-Peree Integrated Task and Motion Planning. ArXiv, abs/2010.01083,, 2020. URL
https://api. swmantccscholar. oeeaatttecccoola...oo//rroppssss
:221124222222224
JS Gautham, Akash Sharma, Samiappan Dhanalakshmi, and Kumar Ramamoorthy. 3D seene reconstruction and mapping with real time human detection for search and rescue robolics. In AIP Conference Proceedings, volume 2427. ATP Publishing, 2023. URL attps:// /pubs. ppubs.aapp aip. hatp.org/aap/acppppp http:///ppp/peraaaaaaeeeeeataaaa 020002/2866461. 1
Georgios Georgakis, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, and Kostas Danillidis. Learning to Map for Active Semantic Goal Navigation. In International Conference on Learning Representations, 2021. URL attps://par.aat .sf gow./serrleet/ppm///33000000
3, 16, 23, 30
Georgios Georgakis, Bernadette Bucher, Anton Arapin, Karl Schmeckpeper, Nikolai Matni, and Kostas Daniilidis. Uncertainty-driven planner for exploration and navigation. In International Conference Roboties and Automotion (ICRA), pp. 11295 11302. IEEE, 2022a. URL abstract/bocument/8888222
ieee ieee..rg/
3, 4, 22, 27, 28, 29, 30
Georgios Georgakis, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, and Kostas Daniilidis. Learning map for active semantic goal navigation. In International Conference on Learning Representations, 2022b. URL https://par..... http:://aaaa.......eeee http:///eeeeeeeeeae///////33//./ http:////////teee/o//oo////o////
28, 31
Georgios Georgakis, Karl Schmeckpeper, Karan Wanchoo, Soham Dan, Eleni Miltsakaki, Dan Roth, and Kostas Daniilidis. Cross-modal map learning for vision and language navigation. In Procced- ings of the IEEE/CVE Conference on Computer Vision and Pallern Recognition, pp. 2022c. URL http:://opnccccsss.o thecvf.eee... cwm/ /ot////////////222222.. httc/////22///mmm/eeeeeeee Satmm//GergaisseCe. Cross-rosssiii Cross-Modall _ap____app____ Learning_foo for Vision and Language_aavriaatoooo httpiiio__cv___2222____2 2022_paper______ peaper.hmml.. 3, 16
Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot. Navigating to objects in the real world. Science Robotics, 8(79) :eadff991,, 2023. URL attps://wwwe http:///e/wwcciceeoo/e.o.mm science. org/
doi/abs/ 10. wtt111///ccrii htt////ciroooiicccco adf wwt.:ob.tt..c.......... 9, 10, 29
Dylan Goetting, Himanshu Gaurav Singh, and Antonio Loquercio. End-to-Ent Navigation with Vision Lan- guage Models: Transforming Spatial Reasoning into Question-Answering. arXiv preprint 2Xxi:2441.0075: 2024. URL
org/ httgg//bs/2411...ee
22
Ze Gong and Yu Zhang. Temporal spatial inverse semantics for robots communicating with humans. In IEEE International Conference on Robolics and Aulomation (ICRA), pp. 4451-4458. IEEE, 2018. URL https://ieeexplore ieee.
16
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. IQA: Visual Question Answering interactive environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. cwtpae___________ 2018 _aat__ppper___ html. html. 24
2018. URL http://openaccess. thecvf. cctpp_/018/ttmm//ooooo wttp//tttmm//ooonnne/ea IQA, fisual_Questiin
con/content_cvpr
43

---

Published in Transactions on Machine Learning Research (August/2255)
Elias Greve, Martin Büchher, Niclas Vödisch, Wolfram Burgard, and Abhinav Valada. Collaborative dynamic 3d scene graphs for automated driving. In IEEE International Conference on Robolics and Automation, pp- 11118 11124. IEEE, 2024, URL https://ieeexplore .eee..rg/ee...e............... htt.r///w...aaccccoooooo/////110
3
Margarita Grinvald, Fadri Furrer, Tonci Novkovic, Jen Jen Chung, Cesar Cadena, Roland Siogwart, and Juan Nieto. Volumetric instance-aware semantic mapping and 3d object discovery. IEEE Robolics and Aulomation Letters, 4(3):3037-3344) 2019. URL https:/ /ttes///eeeeeeereeeee ieee.
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills. In International Conference on Leurning Representalions, 2023. URL https://openreevier net. waddffddbb_caDSSDDD.. 28
Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In IEEE International Conference on Robolics and Aulomation (ICRA), pp. 5021 5028. IEEE, 2024. URL 18, 25, 26, 27, 28
ieee, , org/ /ttps//aatc/ooooueee//eeoeoe
3, 17,
Xiuye Gu, Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Represenlations, 2021. URL https://iclr http:://Accr.cc//eeia///-------
paf. f 27
Yanchen Guan, Haicheng Liao, Zhenning Li, Jia Hu, Runze Yuan, Guohui Zhang, and Chengzhong Xu. World models for autonomous driving: An initial survey. IEEE Transactions on Intelligent Vehicles, 2024. URL
https://ieeexplore. ieee. oeg.e.rr/eeeeeeeeeeeeeeeeeeooeee wtrg/bbstacc////////e///e
/occmmet/10522295. /b////cc///ccmmo/////055533 4
Jun Guo, Xiaojian Ma, Yue Fan, Huaping Liu, and Qing Li. Semantic Gaussians: Npen-Vocabulary Scene Understanding with 3D Gaussian Splatting. arXiv preprint 0035522403.15024, 2024. URL https:
19
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and thecvf. cww cooteent_cvp__000
Pattern Recognition, pp. 2616 2625, 2017. URL Cttpuuptt_oooiiiieeeeeeeaeeeeeee Maapiiig_aaaan.n..__m CVPR_2011111111 2017_2017__aa 20177_pppprrh__t. html. 3, 8, 10, 14, 15, 16, 24
Christian Hane, Christopher Zach, Andrea Cohen, Roland Angst, and Marc Pollefeys. Joint 3D scene reconstruction and class segmentation. Tn Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 97 104, 2013. URL http:// http:///ppncccss..m www.se.thhee.e..ee htt.e//cccccccottttc.c.... _aitaan__oit_____________ Biiis_D_SSceee Se__Seee_____eee _0___0133__v___ @0033_cVP__pape_ html. 7
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 770-778, thecvf. . CVPR_2016 2awee_e11eeee-_111111 html. 9, 10, 15, 17
2016. CRL
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEFE/CVF International Conference Computer Vision, pp. 2961-2969, 20. URL http: htttt:////pecccsss... thecvf. com/ content content_iccc__eeee_ 2eccv_2017//0 20t.200t1////eeeeeee Baa__Mas_______ CCCCN_ICCV____CCC _aperr html. 9, 10, 23
Ying Hc, Bin Liang, Jun Yang, Shunzhi Li, and Jin Hc. An iterative closest points algorithm for registration of 3D laser scanner point clouds with geometric features. Sensors, 1868) 2017b. URL https:
//www. h/wws.mdpi.co http://////2---22///////////
Sachithra Hemachandra, Thomas Kollar, Nicholas Roy, and Seth Teller. Following and interpreting narrated guided tours. In IEEE International Conference on Robotics and Automation, pp. 2574-2579 IEEE, 2011. URL https://ieeexploree ieee. www..e.eee...eeeeeeeeeeeeeeeeeee htt.://rw....eee.ee/eaaaaaaa///a otr//batratt//ccmm////00/0///// 21
44

---

Published in Transactions on Machine Learning Research (August/2025)
Sachithra Hemachandra, Matthew R Walter, Stefanie Tellex, and Seth Teller. Learning spatial-semantic representations from natural language descriptions and scene classifications. In TEEE International Conference on Robotics and Automation, pp. 2623 2630. IEEE, 2014. URL 3, 22
ieee.
Sachithra Hemachandra, Felix Duvallet, Thomas M Howard, Nicholas Roy, Authony Stentz, and Matthew R Walter. Learning models for following natural language directions in unknown environments. In IEEE International Conference on Robotics and Automation (ICRA), pp. 5608 5615. IEEE, 2015. URL https: 3, 21
//ieeexplore. ieee. httpaaaaraaacttccceee//ataao
Joao F Henriques and Andrea Vedaldi. MapNet: An allocentric spatial memory for mapping environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 8476 8484, 2018. URL http://openaccess. thecvf. , com/ www./oootettoooooooooccco content_cvpp___eee_ooooo___ hwta//ttmm/eeerriie Batlsss_p_ee_aaaa An_
Allocentric Dilooentric__CPP_20125 SBPPR_2018___aaa
3, 8, 10, 11, 15, 16, 24
Alexander Hermans, Georgios Floros, and Bastian Leibe. Dense 3d semantic mapping of indoor scenes from rgb-d images. In TEEE International Conference on Robotics and Automation, pp. 2631-2633 TEFE, 2014.
URL http::///eeexploorr leee. org/
3, 19
Hanzhang Hu, Daniel Munoz, J Andrew Bagnell, and Martial Hebert. Efficient 3-d scene analysis from streaming data. In IEEE international conference on robotics and automation, pp. 2297 2304. IEEE, 2013.
URL attpp://ieeexpllae
ieee. .rg/ aataattrccc///cmmmm///3/8000
3, 16
Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In JEEE International Conference on Robotics and Automation, pp. 10608-10615, IEEE, 2023a. URL
https://ieeexplort ieee. org/ abbtrract/bocuuet///11000
3, 16, 25, 26
Joseph Huang, David Millman, Morgan Quigley, David Stavens, Sebastian Thrun, and Alok Aggarwal. Efficient, generalized indoor wifi graphslam. In JEEE international conference on robotics and automation, 5
pp. 1038-1043. IEEE, 2011. URL https://ieeexppors ieee. oot//ogg///a.ooooooo
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. VoxPoser: Composable 3D value maps for robotic manipulation with language models. In Conference on Robol Learning, 2023b. URL
.ithhg............ 1o/. 3, 16, 25, 28
Nathan Hughes, Yun Chang, and Luca Carlone. Hydra: A real-time spatial perception system for 3D scene graph construction and optimization. ar preprint arXiv:2201.13360, 2022. URL https://arxiv. org/
0ttp/22011133660 3
Nathan Hughes, Yun Chang, Siyi Hu, Rajat Talak, Rumaia Abdulhai, Jared Strader, and Luca Carlone. Foundations of spatial perception for robotics: Hicrarchical representations and real-time systems. The International Journal of Robotics Research, pp. 02783619244122972 2024. URL https://journals..
22
Gabriel Ihaarc,, Vihan Jain, Alexander Ku, Eugene Te, and Jason Baldridge. General Exaluation for Instruction Conditioned Navigation using Dynamic Time Warping, 2019. URL htttp:://axxxv.or/abb/////// 05446.
28
Gabriele Janzen and Miranda Van Turennout. Selective neural representation of objects relevant for navigation. Nalure newroscience, 7(6):673 677, 2004. URL https: http:::////paaaaaapppaaapapp apa hrrpa///rrrrr/204441111-0010 16
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omana, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. ConceptFFusion: Open-set Multimodal 3D Mapping. Proceedings of Robotics: Science and System, 2023. URL ieee. aheee.org/abee
3, 18, 19, 20, 25
Collin Eugene Johnson. Topological Mapping and Navigation in Real- World Environments. PhD thesis, University of Michigan, 2018. URL https://deepblun lib. umich edu/handle/2027 000000/722/44144.44 16
45

---

Published in Transactions on Machine Learning Research (August/2025)
M Kaess, M Fallon, H. Johannsson, and JJ Leonard. Kintinuous: Spatially extended kinectfusion. In Proceedings of the RSS Workshop on RGB-D: Advanced Reasoning with Depth Cameras, Sydney, Australia, pp. 9 10, 2012. URL https://dapacee mitttaappaeeeaaaaaaaeeaam.ammaae mit hetp/hanndle//Ta..7111
7
Michael Kaess. Simultaneous localization and mapping with infinite planes. In IEEE International Conference on Robotics and Automation (ICRA), pp. 4605 4611. IEEE, 2015. URL https::/iseexxplore. ieee. org/ abstraatt4bocmme//133333
András Kalapos, Csaba Gór, Róbert Moni, and István Harmati. Sim-to-real reinforcement learning applied to end-to-end vehicle control. In International Symposium on Measurement and Control in Robotics (ISMCR), pp. 1-6. IEEE, 2020. URL https://teeeepplore 1eee. ottp/abbttacct/cccuee//t/t/t
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. ArXiv, abs/1806.10293, 2018. URL
attpp://apil sew.anicccchcccchar....... o 49470584. 4, 28
Christina Kassab, Sacha Morin, Martin Büchner, Matías Mattamala, Kumaraditya Gupta, Abhinav Valada, Liam Paull, and Maurice Fallon. OpenLex3D: A New Evaluation Benchmark for Open- Vocabbucyyyy 3D Scene Representations. orXic preprinl XrXiv:2503. 19764, 2025. URL https://aarat https///axwvvww//aaabeee wwwg/aa///aaae http://5.....1... 28
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. LeRF: Language embedded radiance fields. In Proceedings of the IEFF/CVF International Conference on Computer Vision, thecvf. hwaa///onet////eeeeee /ttp///eee///////////a/aa// wtt.///2///tmm///rr/////
pp. 19729 19739, 2023. URL https://openaccee Language_ Emwedded__eaaaeeeeeeeeeee Raciidccaaaaeeeeeeeeeeeeeeeaeeee F_ieel_Felll_eleee Baclasss_vv____ee 2022_2223_____222222 papeer html. 19, 20, 21
Misha Urooj Khan, Syed Azhar Ali Zaidi, Arslan Ishtiaq, Syeda Ume Rubab Bukhari, Sana Samer, and Ayesha Farman. comparative survey of lidar-slam and lidar based sensor technologies. In Mohammad Ali Jinnah University International Conference Computing (MAJICC), pp. 1-8. IEEE, 2021. URL https::/ieeeexploree iee... org/ centtrracc//ccuuent..oo 99566666/
Apoory Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but effective: CLIP embeddings for embodied AI. Proceedings of the IEEE/CVE Conference on Computer Vision and Pul-
(at//2222/ntmm//aaddlles Simple_but Sout___ffettie____ CLIP Embeddings_foo for Liloorr_mmmeedddeeeeeeeeeeeeeeee AI SI__VPR_2000 2DIRR_222__pp___ papee_ppprr.. html. 23
tern Recognition, 2022. URL attp://openaccess thecvf. com/ content/CVP22022/////////////// /Khandelwal
Mukul Khanna, Ram Ramrakhya, Gunjan Chhablani, Sriram Yenamandra, Theophile Gervet, Matthow Chang, Zsolt Kira, Devendra Singh Chaplot, Dhruv Batra, and Roozbeh Mot- taghi. Goal-bench: A benchmark for multi-modal lifelong navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16373 (aanteet//VP2202//ttm/l/// Navigation cwwpioonn_v__eee_ee
GOAT-Bench_A 16383, 2024. URL http://openaccess. thecvf hetp:ion_ccccc22___pep_e html. 3, 16, 23, 28 Benchmark_ fallffooooiiiiieeeee Multi- MM oig.adaa_iilloogggg.aagaatoooooo
Nuri Kim, Obin Kwon, Hwiyeon Yoo, Yunho Choi, Jeongho Park, and Songhwai Oh. Topological semantic graph memory for image-goal navigation. In Conference on Robot Learning, pp. 393 402.. PML.R, 2023.
URL attps://proceeednng. wwa.iingss............ htt.ppress/v25//// kam00/a.//m.m.m..mmmm......aa1.m html. 17, 18
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment Anything. ar preprint arXiv:2304.0200 2023. URL attp://openaccest thecvf. www.coocccoooocc /tta///ooteett////////////a htt////tmm/////t//e Seaegeilov_SSeeentttt Segment_Anythingg. Anything. ICCV_2023_par_ 21123_ppper.o html. 10, 27
Dong Wook Ko, Chuho Yi, and Il Hong Suh. Semantic mapping and navigation: A Bayesian approach. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2630-2636. IEEE, 2013. URL
https://iseexplore.r ieee. .rg/ www..e/..././aaaatt/bbboooooeoee
21
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, Aniruddha Kembhavi, Abhinay Kumar Gupta, and Ali Farhadi.
46

---

Published in Transactions on Machine Learning Research (August/2255
AI2-THOR: An Interactive 3D Environment for Visual AI. ArXin, abs/1712.05474, 2017. URL https: satppissemanticcccooe. wtt.accc......ooooooooooaa1/2212 28328611. 5
Kurt Konolige, Eitan Marder-Eppstein, and Bhaskara Marthi. Navigation in hybrid metric-topological
maps. In IEEE International Conference on Robotics and Automation, pp. 3041 3047. IEEE, 2011. URL
ieee.
21
Ioannis Kostavelis and Antonios Gasteratos. Semantic mapping for mobile robotics tasks: survey, Robotics and Autonomous Systems, 66:86-103, 2015. URL sttps://sww.sieeneedaa. htt.///ew.cccc/cccccc///rtttllla
pii/$0921889014003033 1,
Jacob Krantz, Erik Wijmans, Arjun Majundar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: aantn---adllanguagee navigation in continuous environments. In European Conference Computer Vision (ECCV), pp. 104-120. Springer, 2020. URL
github. 10/ lo/vlnce/..... 16, 28, 30
Jacob Krantz, Stefan Lec, Jitendra Malik, Dhruv Batra, and Devendra Singh Chaplot. Instancc-specifie image goal navigation: Training embodied agents to find object instances. arXiv preprinl vrXiu:2211.15877 2022. URL https: //ar/ii///xvvv/a org/ abs/ ab//a2ss/.1...////////////////// 2211. 15876. 27
Abhijit Kundu, Yin Li, Frank Dellaert, Fuxin Li, and James M Rehg. Joint semantic segmentation and 3d reconstruction from monocular video. In European Conference on Computer Vision, pp. 703-718. Springer, 2014. URL https://links spriinke..rnngrr...r..rr com/ wwwpphhattt///ee 100. 1007 0000000007300 1/1///9888-3--1100- 3--31910599---1- 45.
Obin Kwon, Nuri Kim, Yunho Choi, Hwiyeon Yoo, Jeongho Park, and Songhwai Oh. sual graph memory with unsupervised representation for visual navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15890-15899, 2021. URL
Vi-
http://openaccess www.cf.coo//oooeeeeeteoooooooeee http:///.//.ooaaat////////////// http:///ttmn//soooossaaall @eraph_Memory_ __ith_ htw.deeeeeeeeeettttooooooooo for Visual Naw.iaatiinnnooyyy..ooooooao 2021 paper_21___pppprrrrrrae.. pappr.html.... html. 3, 17, 18, 24
Obin Kwon, Jeongho Park, and Songhwai Oh. Renderable neural radianco map for visual navigation. In
Proceedings of the IEEE/CVP Conference on Computer Vision and Pallern Recognition, pp. 9099 9108, 2023. URL http:// hwtpp:/cssppacccss...p thecvf. com/ content/CVPR2023 hhtm//mm////////// Lewoo__enderrale_eeeee Leaee_Neeaaaaaaa _Raiiance____ 2aa___22222_________ paper. html. 3, 16, 24
hap_for_Vissull watlisaallaviiaaiooonnooo CVPR_ 2023
Kevin Lai, Licfeng Bo, and Dieter Fox. Unsupervised feature learning for 3d scene labeling. In IEEE International Conference on Robolics and Aulomation (ICRA), pp. 3050 3057. IEEE, 2014. URL https: // //ieeexplore. ieee, waaaaaaac/toccmeenttttt 6907298//
Christian Landsiedel, Verena Rieser, Matthew Walter, and Dirk Wollherr. A review of spatial reasoning and interaction for real-world robotics. Advanced Robotics, 31(5):222-242 2017. URL https: //www. baandfoomliee.cmm///////aaaaa 10tp:/0110000000111111666144 2016 1277554.
Jaewon Lee, Mangyu Kong, Minseong Park, and Fantai Kim. GeomGS: LiDdR-Guided Geometry-Aware Gaussian Splatting for Robot Localization. arXio preprint drXiv:2501.13417, 2025. URL https://arxiv.
org/abs/2501 13417. 19
Xiaohan Lei, Min Wang, Wengang Zhou, and Houqiang Li. Gaussnav: Gaussian splatting for visual navigation. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2025. URL https://arxii: abg/abs/ 2403. 11625. 19, 20
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations, 2022. URL httt:::///operrvieer net/forumiid-Rriijddaoe 9, 26
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsu-
pervised representations. International Conference on Learning Representations, 2021a. URL http:///pppaoiieeeeeeeeeooaaaaaa https://openrevieu net/forum/idd-Kyypponn
47

---

Published in Transactions on Machine Learning Research (August/2025
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprinl arXin:2301.1257 125997, 2023. URL
https://proceedings mlr. press/v202/1i23q 9, 25
Lin Li, Xin Kong, Xiangrui Zhao, Wanlong Li, Feng Wen, Hongbo Zhang, and Yong Liu. SA-LOAM: Semantic- aided LiDAR SLAM with Loop Closure. IEEE International Conference on Robotics and Automation
(ICRA), pp. 7627-7634, 2021b. URL https: www.ntiiscoollle.......o org/CorpusID 235592935. 29, 30
Mingrui Li, Shuhong Liu, Heng Zhou, Guohao Zhu, Na Cheng, Tianchen Deng, and Hongyu Wang. SGS-SLAM:
Semantic Gaussian splatting for neural dense SLAM. In European Conference Computer Vision, pp. 163 179. Springer, 2024a. URL https: //ti////nnn//// http:s// 20
19,
Tianyu Li, Peijin Jia, Bangjun Wang, Li Chen, Kun Jiang, Junchi Yan, and Hongyang Li. LaneSegNet: Map Learning with Lane Segment Perception for Autonomous Driving. In International Conference on Learning Representations, 2024b. URL https://hub. hku. htt.aandle/1722/33133333
1, 3,
Weijie Li, Xinhang Song, Yubing Bai, Sixian Zhang, and Shuqiang Jiang. Ion: Instance-level object navigation. In Proceedings of the ACM international conference multimedia, pp. 4343 4352, 2021c. URL https: //al. acm. orgoo//ggooo /tt/d/////b////////////// 11..1...11111411000 3475575. 27, 30
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: learning bird's-cye-view represcntation from lidar-camera via spatiotemporal transformers. IEEE Transactions on Pullern Analysis and Machine Intelligence, 2024c. URL httpss//ieeexpporre
ieee. org/ seeetraat//ccmmmm///7999990
Ming- Yi Lin, Ou-Wen Lee, and Chih-Ying Lu. Embodied AI with Large Language Models: A Survey and New HRI Framework. In International Conference Advanced Robotics and Mechatronics (ICARM), pp.
983. IEEE, 2024. URL
leee. wwt.::eeeeeeeeeeeeeeeeeeeee
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. Al open, 3:111-133 1112, 2022. URL
8
Yen-Chen Lin, Andy Zeng, Shuran Song, Phillip Isola, and Tsung-Yr Yi Lin. Learning to see before learning to act: Visual pre-training for manipulation. In JEEE International Conference on Robotics and Automation, 2020. URL htttps:/ieeexxpoost leee.org/ /btpa/racct///ccme/////9/////
28
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural
information processing systems, 36:34802-3400 2023a. URL https::/prooceedngs.. neurips hetp://ggs...rrpppppppppaprrr Abstract- -bsttraatt-toofeeenceeeee Conference html. 27
Ming-Yu Yu Liu, Oncel Tuzel, Ashok Veeraraghavan, Yuichi Taguchi, Tim K. Marks, and Rama Chellappa. Fast object localization and pose estimation in heavy clutter for robotic bin picking. The International Journal of Robotics Research, 31:951 973, 2012. URL https://apl swmanticscholss.... org/CorpusID:7197444 28
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pp. 38 55. Springer, 2024. URL https://link. springer. com/ chtpcchept....1// 100p://100000000-1111110 0001007797777--01--777-------000 3. 10
Yu Liu, Yvan R. Pétillot, David M. Lane, and Sen Wang. Global Localization with Object-Level Semantics and Topology. International Conference on Robotics and Automation (ICRA), pp. 1909-4915 2019. URL
attps://api... www.ssecnnnncclan.nnoooo http:s///ppi.eemnncccchooooo wtt.aaacccooaat.ooooooooppaaaaaa
30
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's ard's-cye view representation. In IEEE international conference on robotics and automation (ICRA), pp. 2774-2781. IEEE, 2023b. URL https://1oeexploee
ieee. Leee..rg/absteeeee or//abttart//ccmm/////66666/
48

---

Published in Transactions on Machine Learning Research (August/222)
Iker Lluvia, Elena Lazkano, and Ander Ansuategi. Active mapping and robot exploration: A survey. Sensors, 21 (7)::444,, 2021. URL htttps:/www.... httpppp..oom4444222222/7///1/111
8
H-A Locliger. An introduction to factor graphs. IEEE Signal Processing Magazine, 21(1):28 41, 2004. URL https::/ieeexxploree wew.loorreieeeror.... wtw..//...../bbbaaaattoooooooom 1267047// 6
Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment. In Conference on Robot Learning, 2024.
URL
3,
Malte Lorbach, Sebastian Höfer, and Oliver Brock. Prior-assisted propagation of spatial information for object search. In IEEE/RSSJ International Conference on Intelligent Robots and Systems, pp. 2904-2909.
IEEE, 2014. URL https: https://1eeexxpoor leee. habsstrreet/ocmmme//2225555 16
David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of com- puter vision, 60:91-110, 2004. https://:: bk. tudelft. twwelff.nnlcouusee/eeeeeeeoooe matching pdf.
Haokuan Luo, Albert Yue, Zhang-Wei Hong, and Pulkit Agrawal. Stubborn: A strong baseline for indoor object navigation. arXiv preprint 0:X0::2003.0755,, 2022. URL https://ieeexpl.e ieee. arg/abstract/ document/9981646/ 9
Matteo Luperto, Marco Maria Ferrara, Giacomo Boracchi, and Francesco Amigoni. Estimating Map Com- pleteness in Robot Exploration. Xiv preprint urXiv:2406. 13482, 2024. URL attps://arxivv ortp:/o.rr//bs////// 2406. 13482.
13482.
29
Matt MacMahon, Brian Stankiewicz, and Benjamin Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. Def, 2(6):4, 2006. URL attps://cdhn aaai pdf. 16
aaai.org//a.....a.......a.....aa org/ AAAT/22006/AAATO6-----
Dominic Maggio, Yun Chang, Nathan Hughes, Matthew Trang, Dan Griffith, Carlyn Dougherty, Eric Cristofalo, Lukas Schmid, and Luca Carlone, Clio: Real-time task-driven open-set 3d scene graphs. IEEE Robotics and Automation Letters, 2024. URL https::/leeexplore ieee. org orgpaastracc//ccmme////66600/) 3, 8, 17, 25
Martin Magnusson, Henrik Andreasson, Andreas Nuchter, and Achim J. Lilienthal. Appearance-based loop detection from 3D laser data using the normal distributions transform. In IEEE International Conference on Robotics and Automation, pp. 23 28, 2009. doi: 10.109/ROBB0T:200555517777 URL https://leeexppore ieee
28
Aravindh Mahendran, Hakan Bilen, João F Henriques, and Andrea Vedaldi. Researchdoom and cocodoom: Learning computer vision with games. arXiu preprint vrXxx:16100024: 2016. URL https://arxiv.orr 0bb/1610.02433 16
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael HenafT, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, Karmesh Yadav, Qiyang Li, Ben Newman, Mohit Sharma, Vincent-Pierre Berges, Shiqi Zhang, Pulkit Agrawal, Yonatan Bisk, Dhruv Batra, Mrinal Kalakrishnan, Franziska Meier, Chris Paxton, Alexander Sax, and Aravind Rajeswaran. OpenEQA: Embodied Question Answering in the Era of Foundation Models. IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16488 16498, 2024. URL https: pettps://ai... semanticscholar org/CorpusIle 268066655. 30
Daniel Maturana, Po- Wei Chou, Masashi Uenoyama, and Sebastian Scherer. Real-time semantic mapping for autonomous off-road navigation. In Field and Service Robotics: Results of the International Conference, pp.
335-350. Springer, 2018. URL https://link: swringerrrrerrr...rree waaannerr.oom.. www.///aww//....oooooooooo 1007:////10.10///9788 91111/777---11-77-------1- 8
49

---

Published in Transactions on Machine Learning Research (August/225)
Cynthia Matuszek, Dieter Fox, and Karl Koscher. Following directions using statistical machine translation. In ACM/TERR International Conference on Human-Rohe Interaction (HRT), pp. 251-252 TEEE, 2010. URL https://ieeexplae
leee. http://bbaaattccccm/////3//3/51/
Cynthia Matuszck, Evan V. Herbst, Luke Zettlemoyer, and Dicter Fox. Learning to Parse Natural Language Commands to a Robot Control System. In International Symposium on Experimental Robolics, 2012. URL
http:///pp..smaattiieeeeee wttpseaacccaaarrr/rrrorooe http://pp..aaaaaaaaaaooaoorrrrra 1658890. 28, 30
Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. Learning parse natural language commands to robot control system. In Experimental robotics: the international symposium on exper- imental robolics, pp. 403 415. Springer, 2013. URL https: //w::://nnnn/e./ aarikk..rrngerr....r com/ com/chaateer110.100 300-3-339900005-77 16
Stefan May, David Dröschel, Stefan Fuchs, Dirk Holz, and Andreas Nüchter. Robust. 3D-mapping with timc-of-flight cameras. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1673 1678. IEEE, 2009. URL
ieee. .eeee.org//eeeeee.....eee http:///strrtc/ommmm/////555 3, 19, 20,
22
Kirill Mazur, Edgar Sucar, and Andrew J Davison. Feature-realistic neural fusion for real-time, open set scene understanding. In IEEE International Conference Robotics and Automation, pp. 8201 8207. IEEE, 2023. URL https://iseexpllaee
ieee. htrp///sstraattcccmm//////t00000
3
Pietro Mazzaglia, Ozan Catal, Tim Verbelen, and Bart Dhoedt. Guriosity-driver exploration via latent bayesian surprise. In Proceedings of the AAAT conference on artificial intelligence, volume 36, pp. 7752-7760, 2022. URL https://ojs aaai. index.
9
John McCormac, Ankur Handa, Andrew Davison, and Stefan Leutenegger. Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. In IEEE International Conference on Robotics and Automation, pp. 4628-4635. IEEE, 2017. URL https://1seexploree leee. org htt.://astrrtc/ccccc/////955555 19
7,
John McCormac, Ronald Clark, Michael Bloesch, Andrew Davison, and Stefan Leutenegger. Fusion+ Volumetric object-level slam. In International Conference on 3D Vision (3DV), pp. 32-41. IEFF, 2018.
URL https: //ti://eeeeeeeeeeee ieee. http//aarrrcccccccceee/////e
David Meger, Per-Erik Forssén, Kevin Lai, Scott Helmer, Sancho McCann, Tristram Southey, Matthew Baumann, James J Little, and David G Lowe. Curious george: An attentive semantic robot. Robotics and Autonomous Systems, 56(6):503-511 511, 2008. URL https: http://wwwww.ieneeeeeeeee.e. htt.://dtttttttcccc/ccc/////ataa
pis/S0921888900800033 3, 23
Yash Mchan, Kumaraditya Gupta, Rohit Jayanti, Anirudh Govil, Sourav Garg, and Madhava Krishna. QueSTMaps: Queryable Semantic Topological Maps for 3D Scene Understanding. arXiv preprint 17
arXiv:240.004442 2024. URL https://1eeexploree ieee. htt..o///.b.a...cc//ctetee//ee/e
Hongyuan Mei, Mohit Bansal, and Matthew Walter. Listen, attend, and walk: Neural mapping of navigational instructions action sequences. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016. URL httpp:///// aaai. wow.ooode....ooeeeoeoooooooooeee ///i/indxxx// php/ http://AA/aaateeeeeee//////////e 10364. 8
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4460-4470, 2019. URL http://openaccent thecvf. ctteffccomcoooo content Caaeeeetttcc_ee__eeeeee http//ttt//MMscceeeeee 3D_ Leerniing33DRRcccttutttiiii in httprrrruccccoiii_nmiiiioooooepp Sirrace_cPP___
Sooupancc_Metoorks.. _wwpe_c1e_________ 201-22219______ _alppppprr.pp paperrhtun html. 20
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1): 99 106, 2021. URL https::/ /dll. acm http://l...ccooo/aa////aaaaa/e/a
1041/11011335 20, 21
50

---

Published in Transactions on Machine Learning Research (August/225))
Ian D Miller, Anthony Cowley, Ravi Konkimalla, Shreyas S Shivakumar, Ty Nguyen, Trey Smith, Camillo Jose Taylor, and Vijay Kumar. Any way you look at it: Semantic crossview localization and mapping with lidar. IEEE Robotics and Automation Letters, 6(2))2397 2404, 2021. URL https://ieeexplore.tt
(bstracc//dccumnnt888111001 8
lan D Miller, Fernando Cladera, Trey Smith, Camillo Jose Taylor, and Vijay Kumar. Stronger together: Air-ground robotic collaboration using semantics. JEFE Robolies and Automation Letters, 7(4):9643-0))) 99550, 2022. URL attps://eeeexplore. ieee. orgeee.or//aee httg///ssttacc/ccccut///////000
8
Zhixiang Min and Enrique Dunn. Voldor+ slam: For the times when feature-based or direct methods are not good enough. In IEEE International Conference Robotics and Automation, pp. 13813 191399 IEEE, 2021. URL. attps://sseexpporee leee.
3, 19, 20
Oscar Martinez Mozos, Rudolph Triebel, Patric Jensfelt, Axel Rottmann, and Wolfram Burgard. Supervised se- mantic labeling of places using information extracted from sensor data. Robolics and Aulonomous Systems, 55 (5):391-400 402, 2007. URL https: //wp:///wwwww///./ www.:wsscieee... sci aciencedirect. cww/cc/oc/ccccccee sci attpa//aarrraaeaaa///////0000000 7, 21
Raúl Mur-Artal and Juan D Tardós Visual-inertial monocular SLAM with map reuse. IEEE Robotics and Automation Letters, 2(2):796-80)) 2017. URL attps://i0eexplore leee. org/abstrrat///oommntt 7817784/.
Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE transactions on robotics, 31(5):1147 1163, 2015. URL
https://1eeexplore.Bee ieee. www..rgg/gaao htggabbsraact///
wtt..///.../bbtttt/////aooaaaaa. //sstracc/cccmmttt//444///////
Medhini Narasimhan, Erik Wijmans, Xinlei Chen, Trevor Darrell, Dhruv Batra, Devi Parikh, and Amanpreet Singh. Seeing the un-scene: Learning amodal semantic maps for room navigation. In ECCV, pp. Springer, 2020. URL https://tttt sirine...ingrr........rr..rmr... wwaigge..co/cchppeeee /ttp://///0...000/777-----------
30. 10, 15
Richard Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In IEEE international symposium on mired and augmented reality, pp- 127 136. leee, 2011a. URL https: https////eeeeeppe..ee// ieee.
7
Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison. DTAM: Dense tracking and mapping in real-time. In international conference computer vision, pp. / //ieeexplore. ieee. htrr//aasrractaccceeeett//t
IEEE, 2011b. URL https:
Lachlan Nicholson, Michael Milford, and Niko Sünderhauf. Quadricslam: Dual quadrics from object detections as landmarks in object-oriented slam. IEEE Robolies and Aulomation Lelfers, 4(1):1 8, 2018. URL
https://lseexploree leee.or.. wtt....g//asstatee///eaaaae/eeee Brtrractt/occuuee///40000000
Andreas Nüchter and Joachim Hertzberg. Towards semantic maps for mobile robots. Robotics and Au- lonomous Systems, 56(11):915 926, 2008. URL https://www.et. www.:eiceeeeddeeeeccccceeeeeeeee
pi1/90922188900001777 3, 19
Andreas Nüchter, Kai Lingemann, Joachim Hertzberg, and Hartmut Surmann. 6D SLAM- 3D mapping outdoor environments. Journal of Field Robotics, 24(8-9):699-722, 2007. URL https://onninellirrrr
wwley.com/ salley.com/doi/abb////e http/////abbbb/o//////a http://...00////// wtw.10002bbb.00000 3, 19, 20
OpenAL. Gpt-4 technical report, 2023. URL https: / http://axxvvvvo/ hwwl:/oor/aaaee
25, 27, 32
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacliin El-Nouby, al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0719. 2023. URL https //tp///xxvvvv//////e/// orgg www.ooogg////./.ooooo/..o/... aas///bbs//2 07193. 9, 17, 27
51

---

Published in Transactions on Machine Learning Research (August/2255
Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, and Mustafa Mukadam. ISDF: Real-time neural signed distance fields for robot perception. ar preprinl arXiv:2304.02296, 2022. URL https://arxiv.orr www.:o/gggggg....../o 02296. 21
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Marayan-Chen Spandana Gella, Robinson Piramuthu, and Dilek Hakkani-Tuu Gokhan Tur and. TEACh: Task-driven Embodied Agents that Chat. In Conference on Artificial Intelligence (AAAT), 2022. URL https: http:/:///r/vvvvv//// /bs//bs/
28
Xinlei Pan, Daniel Seita, Yang Gao, and John F. Canny. Risk Averse Robust Adversarial Reinforcement Learning, International Conference on Robotics and Automation (ICRA), pp. 8522 8528, 2019. URL http:://app.semaatissccoooo
attpss//app semanticscholar wwt.r//ooo....oooouussuiiiiia org/CorpusID:90222211 4
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self- 2017. URL. https: httpse///proceeingg. mlr, prtss./r//s///////
supervised prediction. In ICML, pp.
pathak17a.htmn 9
Siddharth Patki, Andrea F Danicle, Matthew Walter, and Thomas M Howard. Inforring compact representations for efficient natural language understanding of robot instructions. In IEEE International Conference Robotics and Automation, pp. org/ brggabbsract/doooo /ttprractt//cccmme//////6///// 3, 16, 17, 24
IEEE, 2019. URL
ieee.
Siddharth Patki, Ethan Fahnestock, Thomas Howard, and Matthew R Walter. Language-guided semantic mapping and mobile manipulation in partially observable environments. In Conference on Robot Learning, pp. 1201-1210. PMLR, 2020. URL http://procceddnnss mlr, https//v100//attt120..
16
Rohan Paul, Jacob Arkin, Derya Aksaray, Nicholas Roy, and Thomas M Howard. Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms. The International Journal of Robotics Research, 37(10):1269 1299, 2018. URL petts::/dspaat http//daaaeeiiee..co www.mmit.meetie. www.eeddhhaan/11//222
16
Songyou Peng, Kyle Genova, Chiyu 'Max' Jiang, Andrea Tagliasacchi, Mare Pollefeys, and Thomas Funkhouser. OpenScene: 3D scene understanding with open vocabularies. In Computer Vision and Pattern Recognition,
Proceedings of the IEEE/CVF Conference 2023. URL http:// thecvf. cwm/ co/teett////22222 hatia///ttmm//eee httmm//ang.ppenscee.u. 3D_Scene_ iiderstanding.oo With Open_ Vocabularies_cvVel 2023 _apper_pppp____ pmaper.hmml html. 3, 10, 18, 19, 20, 28
Rolf Pfeifer and Fumiya lida. Embodied artificial intelligence: Trends and challenges. Lecture notes in computer science, pp. 1 1-26, 2004. URL httts:://iniie /ppiess................... http:///ww.....c//////ae///t//ta 1ttp:///11110000999770700 /b99/79997../ .df..
Sudeep Pillai and John Leonard. Monocular slam supported object recognition. arXiv:1506.00 01732,132,2 2015. URL htttps///axiv..oo hwtgg/bb/1500.001.1 7, 8
arXiv preprint
Julio A Placed and José Castellanos. Enough enough: Towards autonomous uncertainty-driven stopping
criteria. IFAC-PapersOnlline, 55(14):126 132, 2022. URL
arricie/pp//22005962200000
29
David Prokhorov, Dmitry Zhukov, Olga Barinova, Anton Komushin, and Anna Vorontsova. Measuring robustness of Visual SLAM. International Conference on Machine Vision Applications (MVA), pp. 1-6, 2019. URL https:tt/// httpp::/app...emmtttceeeeeeeeeee semanticscholarttttttttaccaccaaa wwwpaaantiiccolaaroo//rrrrrerree 1918842788 29, 30
Andrzej Pronobis and Patric Jensfelt. Large-scale semantic mapping and reasoning with heterogeneous modalities. Tn IEFE international conference on robotics and aulomation, pp. 3522- IEEF, 2012. URL attps://ieeexplie
ieee. otg.abattracc//ccceett///ee
7, 22
Andrzej Pronobis, Oscar Martinez Mozos, Barbara Caputo, and Patric Jensfelt. Multi-modal semantic place classification. The International Journal of Robotics Research, 29(2-3):298-322 2010. URL https: 21
//journals. aatgppa...oooaa////////../77700.
52

---

http:////1..0000/----0000----000 Published in Transactions on Machine Learning Research (Auuust/2055)
Huayan Pu, Jun Luo, Gang Wang, Tao Huang, and Hongliang Liu. Visual SLAM integration with semantic segmentation and deep learning: A review. TEEE Sensors Journal, 23(11):22119-2211 22138, 2023. URL https://ieeexpporee ieee. org/ abstract/document
htt.g///r.......o.oaaaaaart//oa. www.eentt/22229///.om 6, 8
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652 660, 2017a. URL http:///penaccoss... wtw.oppaccssseeeee.tteeeeeeeeeee cww/content.cccmmmooooooooooooc. ceconteet_cvp_____0eee_ 201_0017//0
_vppr_o11/mttmmmmmmmm 20007/html/0t
PointNet_ Deneblet_Deep_LLarninn CV_S__CPR____ 2012_0001___ html. 20
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++:: Deep hier- archical feature learning on point sets in a metric space. Advances in neural informa- lion processing systems, 30, 2017b. URL https://proceedings.. betpeeeddnnnn.uuripp... www.npe...//pppppeeeeee httpripp.ccppappee/1/1////////// /hahh/ (BBf44bb380601774450eee888333333 -bstrrcct.bttttt.....om.o html. 20
Zhentian Qian, Kartik Patath, Jie Fu, and Jing Xiao. Semantic slam with autonomous object-level data association. In JEEE international conference on robotics and automation (ICRA), pp. 11203 11203-11299, IEEE, 2021. URL attps://1seeexplore wewe..er/eaeaeeee.e..e...eeee... htagg//bbrraattboooutttttee//6
7
Zhentian Qian, Jie Fu, and Jing Xiao. Towards Accurate Loop Closure Detection in Semantic SLAM With 3D Semantic Covisibility Graphs. IEEE Robolics and Aulomation Letlers, 7:2455 2462, 2022. URL
www.sticccholllaaa...... wwwproorg//...a........o. httppss/appp..eennttccccoooooo
246357678. 225. 29, 30
Cao Qin, Yunzhou Zhang, Yingda Liu, and Guanghao Lvv Semantic loop closure detection based on graph matching in multi-objects scenes. Journal of Visual Communication and Image Representation, 76:103072, 2021. URL //ww://wwwwwwww
7,
Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. LangSplat: 3D language Gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20051-20000 2024. URL http://openaccees. thecvf htt///oonent/////22///// Langsppaat_____anugeeeee Qin_ Lin_SantSplat__ _e_ttt_________ Language htwpa//aagggggeeaassaaaaaannnaa Beluusian_Splattingg Batpinng_vc_________ pwt.pp.ppe..l..l.... html. 19
Ri-Zhao Qiu, Yafci Hu, Ge Yang, Yuchen Song, Yang Fu, Jianglong Ye, Jiteng Mu, Ruihan Yang, Nikolay Atanasov, Sebastian Scherer, et al. Learning generalizable feature fields for mobile manipulation. arXic preprint arXit:2403.0756: 2024. URL attps://arse org/ abs http///b/2223.... 07563. 19, 20
Peteris Racinskis, Janis Arents, and Modris Greitans. Constructing maps for autonomous robotics: An intro- ductory conceptual overview. Electronics, 12(13):2925, 2023. URL https: //tp:///wwwww/ http://wwwwwpppp.. www.mmpp..com/2299992222000 com/2099-929// 12/13/2925. 8
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, el al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8763. PMLR, 2021. URL http://proceeddings mar.mmrr...... mlr.press/ v139/radford21a. 9, 17, 21, 25
M. H. Raibert and J. J. Craig. Hybrid position/force control of manipulators. Journal of Dynamic Systems, Measurement, and Control, 103(2):126-13) 133, 06 1981. ISSN 0022-0434. doi: 10.1115/1.3139652. URL
https://doi.ors org. 1...1111/1.313955 4
Santhosh K Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Occupancy anticipation for efficient exploration and navigation. In ECCV, pp. 400-418. Springer, 2020. URL https: http:///nnnnssssiingrr..e com/
24. 28
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexan- der Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterport 3D dataset (HM3d): 1000 large-scale 3D cnvironments for embodicd AI. In NeurIPS Datasets and Benchmarks Track (Round 2), 2021. URL 9447
beniksar-bbcmmaakkee bewccaaa-----poocdedeeeeee pdf. 15, 16
httpllpp.ccpppppppppeeepp /0/pape//202///11e
53

---

Published in Transactions on Machine Learning Research (August/2025)
Sonia Raychaudhuri, Saim Wani, Shivansh Patel, Unnat Jain, and Angel Chang. Language-aligned waypoint (law) supervision for vision-anddllnguuage navigation in continuous environments. In EMNLP, pp. 4018 4018-4028, wwnlemmaaeeeeeeeeeeee 328/. 28
2021. URL http:://acclatttologg org/2021.
Sonia Raychaudhuri, Tommaso Campari, Unnat Jain, Manolis Savva, and Angel X. Chang. MOPA: Modular Object Navigation with Point Goal Agents. arXiv preprint arXiv:2304.0m 03696, 2023. URL https:// 3dlg-hove.gae gitrcc.gituubb.... waabb/i./////.../o// 3, 9, 10, 15, 16, 23, 29
Sonia Raychaudhuri, Duy Ta, Katrina Ashton, Angel X. Chang, Jiuguang Wang, and Bernadette Bucher. Zero- shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation, 2411.07848 3, 16, 17, 24, 28, 29, 30
2025. URL
Allen Z Ren, Jaden Clark, Anushri Dixit, Masha Itkina, Anirudha Majumdar, and Dorsa Sadigh. Explore until Confident: Efficient. Exploration for Embodied Question Answering. arxiv preprind ar (rx::22403.115 15941, 2024. URL attps:// htpp:///arxivv org/obs aaaaabs//// /203. 15941. 9
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Nearal Information Processing Systems, 28, 2015. URL https: // proceedings neurips httppp///...../.ee/////eeeeeeeea
Abstract html.. 9,
10
Rosinol, A. Gupta, M. Abate, J. Shi, and L. Carlone. 3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans. In RSS, 2020a. URL 3, 4, 22
06289.
Antoni Rosinol, Marcus Abate, Yun Chang, and Luca Carlone. Kimera: an open-source library for real- time metric-semantic localization and mapping. In IEEE International Conference on Robotics and Automation (ICRA), pp. 1689-1696. TEEE, 2020b. URL https://teeexplone feee. org/ absttact/ document/91966885/ 4, 16
Antoni Rosinol, Andrew Violette, Marcus Abate, Nathan Hughes, Yun Chang, Jingnan Shi, Arjun Gupla, and Luca Carlone. Kimera: From SLAM to spatial perception with 3D dynamic scene graphs. The International Journal of Robotics Research, (0(12-14):150)) 1546, 2021. URL https://journatt wag.lub..aa..b....... www.cooooooooooo 11711.11770/077554421115555
8
Antoni Rosinol, John J Leonard, and Luca Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields. In FRE//SS] International Conference on Intelligent Robots and Systems (IROS), pp. 3437 3444. IEEE, 2023. URL
21
Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2564 2571. IEEE, 2011. URL https: // http://eeeeepleeeeeee ieee. weweeeereee...eeeeeeeeee..eeeeee abwstractt/dccueet//eeee
6
Martin Rünz and Lourdes Agapito. Co-fusion: Real-time segmentation, tracking and fusion of multiple objects. In IEEE International Conference on Robotics and Automation (ICRA), pp. 4471 4478. IEEE 2017. URL attps://eeeexploxx leee. htag///atrrrtttttoee//eeeeee
20
Renato F Moreno, Richard A Newcombe, Hauke Strasdat, Paul HJ Kelly, and Andrew J Davi- son. Slam+ Simultaneous localisation and mapping at the level of objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1352 1359, 2013. URL http://openaccess. thecvf. cww/ hete/contett_eee conntet__vpr__000 htpp__211/htmm/// Mate//aaaaa-MMeooo....o Simultaneous.
Localisation Baapa__vVR__paee__ html. 3, 4, 17, 24
Muhamad Risqi Saputra, Andrew Markham, and Niki Trigoni. Visual SLAAM and structure from motion in dynamic enviromments; survey. ACM Computing Surveys (CSUR), 51(2):1-36, 2018. URL https:
//dl. acm. wtw.c..rr//////a////.... 11t////10114/337773... 8
54

---

Published in Transactions on Machine Learning Research (August/2025)
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the BEEE/CV Conference on Computer Vision and Pottern Recognition, pp. 4938 4947, 2020. URL http:://ppenaccess. thecvf. com/ Matching Baacchh_rappp__e Graph_Neural
www/content.ttcooomoemommmmcec.. content Sete___VR______ 2020/htm1/Sarlnn Seciiinnuueerueeee Laciiee_eerannnenaaaeeeee
CVpR__ 2020 _at___ppppr_____ html.
Nikolay Savinov, Alexey Dosovilskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. In International Conference on Learning Representations, 2018. URL nttps://arxiv: org/abs/////.oooooo..oo..oo.. htt///bs///000 1803. 00653. 3, 17
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Ju- lian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied Al research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9339-9347, 2019. URL attp:// http:////ppcccccs.eee thecvf. com/ connetttttt///./n crttteetttccC______ L_____CCVV_________ 2019/htmm/Savee Habitat Lacaat____la_la Platform for Eer__Embdeed__. AI Reweeeeeeeeeeeee _01___001______ pwppp.html.....p.pppppppp.p...pm html. 5, 16
Thomas Schops, Torsten Sattler, and Mare Pollefeys. Bad slam: Bundle adjusted direct rgb-d slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 134-144, 2019. URL http://apenaccess. thecvf. com/ content, cettiieet__cR____ html. 20
CVPR_ (PPR_2119httmm/Shho/
_LAM
Bundle Adjustedd Direct__eoe SLAM_ CWwR__219________
Sunando Sengupta and Paul Sturgess. Semantic octree: Unifying recognition, reconstruction and representation via an octree constrained higher order mrf. In IEEE international conference on robotics and automa- tion (ICRA), pp. 1874-1879. IEEE, 2015. URL https://iocexppore. wwtp:ppoore..ee.eeo.eeeeeeeeeeee wtt.:///bbaaat///occeeeeeoo 7139442/.
James A Sethian. Fast-marching level-set methods for three-dimensional photolithography development. In Optical Microlithography IX, volume 2726, pp. 262-272. International Society for Optics and Photonics, 1996. URL
httpb//aw..ogggcoooooooeeeeeeeee for- coor-three-diinnee dimeneiddlmmmininnnaaaa - hrw.reeeeeeiiinnoonoooooooooooor
1117/12.240962. short. 9
Nur Muhammad Mahi Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur Sxlam. CLIP- Fields: Weakly supervised semantic fields for robotic memory. In IEEE International Conference on Robolics and Automation, 2023. URL
2ttg///0b/22100666.... 3, 19, 21, 25
Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. LM-Nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning, pp. 502-504. PMLR, 2023. URL.
3, 8, 17, 18, 25, 27
Mo Shan, Qiaojun Feng, and Nikolay Atanasov. OreVIO: Object residual constrained visual-inertial odometry. In TEEE/RS) International Conference Intelligent Robots and Systems (IROS), pp. 5104-5111. IEEE, 2020a. URL
ieee. org/ /ttpb//rree//tcccoo///////////// http:///gaaa.aaoeeeeeeeaaaaaaaaa
Tixiao Shan, Brendan Englot, Drew Meyers, Wei Wang, Carlo Ratti, and Daniela Rus. Lio-sam: Tightly- coupled lidar inertial odometry via smoothing and mapping. In IEEE/RSA international conference on intelligent robots and systems (IROS), pp. 5135 5142. IEEE, 2020b. URL nttps://ieeexplore ieee.
William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola. Distilled feature fields enable few-shot language-guided manipulation. arXiv preprint orXio:2308.07991 2023. URL https://arzis httpa////arivooooa/aa///e/ /ttps////230000000 19
William B Shen, Danfei Xu, Yuke Zhu, Leonidas J Guibas, Li Fei-Fei, and Silvio Savarese. Situational fusion of visual representation for visual navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2881-2890, 2019. URL http:///penaccess. thecvf. wwwhwwf.co/cooneececeeeeeeeoocce wwt.onttetttcc_211/////e1e ottoo/otoooaaaa////esststttttttt Representation for Visual Maaiillavvgattinn Ciceetttiinnic____eeeeee ICCV_2019 papperhtml. html. wttiooonnnasssionooiee Fusion_off of_illl
Situational
55

---

Published in Transactions on Machine Learning Research (August/225))
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. URL
attps://arx:s www.iv.org/aas///ooooo wttg/b///////////////e 111101111.11744 28
Ishneet Sukhvinder Singh, ID Wijegunawardane, SM Bhagya P Samarakoon, MA Viraj Muthugala, and Mohan Rajesh Elara. Vision-based dirt distribution mapping using deep learning. Scientific Reports, 13(1):
12741, 2023. URL https:/ /tttps://www... www.wr..te.... www.are..ooeeee.eeeaaaa
Xu Song, Xuan Liang, and Zhou Huaidong. Semantic mapping techniques for indoor mobile robots: Review and prospect. Measurement and Control, 58(3):377-393, 2025. URL http:////ooonaassssepplll
http//bb/110.1177002223333
1
Ricardo B Sousa, Héber M Sobreira, and António Paulo Moreira. A systematic literature review on long-term localization and mapping for mobile robots. Journal of Field Robotics, 40(5):1245-1322) 2023. URL
https://onliieeliraane wiley. coom/oo/abs//0..../ 101...0///bbb./111
C. Stachniss, Giorgio Grisetti, and Wolfram Burgard. Information Gain-based Exploration Using
Rao- Bao-Blackwellized Particle Filters. In Robotícs: Science and Systems, 2005. URL https://api.
semanticscholar, .rg. CorpusID: 8255559. 4, 30
Jörg Stückler and Sven Behnke. Multi-resolution surfel maps for efficient dense 3D modeling and tracking. Journal of Visual Communication and Image Represenlation, 25(1):137-147, 2014. URL https: //www. sciencedirect. cwm/ comm/scienc///reeee src.cbece/aattteeeeepp//////1110
18
Jörg Stückler, Nenad Biresev, and Sven Behnke. Semantic mapping using object-class segmentation RGB-D images. In IEEE/RS, International Conference on Intelligent Robots and Systems, pp. 3005 3010. IEEE, 2012. URL
ieee. .ee..org//eeeeeeeeeeee..e...eeoe wwwataatt/ootuuaeaaaaaaaaaaeeeee w330mm90....... 3, 19
Muhammad Sualeh and Gon- Woo Kim. Simultaneous localization and mapping in the epoch of semantics: a survey. International Journal of Control, Aulomation and Systems, 17(3):729-7444 2019. URL https: //link. springer....
//1ink.springer. http//aarccce/1..10...../1 10010000//s11 018-0130-x. 8
Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew Davison. imap: Implicit mapping and positioning in real-
time. In Proceedings of the IEEE/CVF international conference compuler vision, pp. 6229 6238, 2021.
URL http: http::///oppacccsssso thecvf. com httm///ooneenttcccc/ooe http://cccoo2//mmm/s/uuaaar iMAP Implicit Beapliici_Maaping..._m MMa__anap__n__n__ee _nd Positioning. Batteg_____eeaaaaaa
paper html. 21
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Newral Information Processing Systems, 2021. URL https:// proceedings neurips. 2
Abstract. .html. 5
Hamid Taheri and Zhao Chun Xia. SLAMM; definition and evolution. Engineering Applications of Ar- tificial Intelligence, 97:104032, 2021. URL
$0952197620303092. 4,
Lei Tai, Giuseppe Paolo, and Ming Liu. Virtual-to-real deep reinforcement learning: Continuous control mobile robots for mapless navigation. In JEEE/RSJ international conference on intelligent robots and systems (TROS), pp. 31-36. IEEE, 2017. URL https://seeexplore leee. www.:eeeeeee...e..eee arg/abstraat//ooumen// 8202134/.
Francesco Taioli, Federico Cunico, Federico Girella, Riccardo Bologna, Alessandro Farinelli, and Marco Cristani. Languag--enhanced RNR-map: Querying renderable neural radiance field maps with nal- ural language. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4669-4674, 2023. URL
thecvf. /ttp////oattttt////////////Laaa i Bac
Sanguage-Enhanee Bacaccdd_NRRRMM Bataoaaaa----uuuueuiiieeeeeeeee ICCVW 20wp22223____20 _ae3_____r___ html. 3, 16, 25
Neural Ltt_aaaadaaaaellllaaa Baaeaieeeeeddddapa____eee--_ with Natural_
56

---

Published in Transactions on Machine Learning Research (August/2025
Francesco Taioli, Edoardo Zorzi, Gianni Franchi, Alberto Castellini, Alessandro Farinelli, Marco Cristani, and Yiming Wang. Collaborative Instance Navigation: Leveraging Agent Self-Dialogue to Minimize User Input. arXiv preprint arXte:2412.01120, 2024. URL https://uu /tt..uiaaaabbbse/ hww.ard. edu/abs/ 2024arXlw2422122077/ 27, 30
Takafumi Taketomi, Hideaki Uchiyama, and Sei Ikeda. Visual SLAM algorithms: A survey from 2010 to 2016. IPSJ Iransactions on computer vision and applications, 9(1):16, 2017. URL https: www.ikkssringgrr.oomm com/
article/10. 10. 1000111///1144-00000077-----
S Talha Bukhari, Daniel Lawson, and Ahmed H Qureshi. Differentiable Composite Neural Signed Distance Fields for Robot Navigation in Dynamic Indoor Environments. arXiv e-prints, pp. XxXiv22522, 2025. URL
https://arxiv/
httpa://araav.orr/aa///555 wtg. 02664. 21
Yujie Tang, Meiling Wang, Yinan Deng, Zibo Zheng, Jingchuan Deng, and Yuleng Yue. OpenIN: Open- Vocabulary Instance- Instance-Oriented Navigation in Dynamic Domestic Environments. arXiv preprint ar NKi::2511.04270 2025. URL nttps://leeexplore leee. /tt.:///////88333/////////111111 17, 22
Keisuke Tateno, Federico Tombari, Tro Laina, and Nassir Navab. Cun-slam: Real-time dense monocular slam with learned depth prediction. In Proceedings of the IEEE conference on computer vision and pallern recognilion, pp. 6243 6252, 2017. URL http://openaccest thecvf wttpc//oneettccrrr_200000 html/Tateno_ http:/atteoocNN-SSAAAAA SLAM Real-Raalnnnlaaaa Beealliie_eeee______________ CVPR_ 10TRa_211____00 _aperp html. 7,
Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 1411558-16669,, 2021. URL https / wew.eeeiinnnnnuriii..e Abstract...tt.............a html. 3, 19, 20
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI conference artificial intelligence, volume 25, pp. 1507-1514 1514, 2011a. URL 16
https://ojs. aaai. orgo index.ndexee//ee Saadex.pppAAA.... ppppAAAI/aaaea TAAAA/article/rie////99
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth J. Teller, and Nicholas Roy. Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation. In AAAT Conference on Artificial Intelligence, 2011b. URL, https:/ //api. semanticacholar.
org/CorpusID: 220828823. 28, 30
Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6411 6420, 2019. URL http://openaccess. thecvf. cwm content and www.:eeeeoeeeeeeeeme wtw.aaaleeeoooolutoonoooooo for -ottiiitooooooooooooo_ot Choint_Cloudee Iicllssi_vv____ee Bacecv_2119_pppppe__ html. 3, Xtt_ak__Feoeoil______
Sebastian Thrun. Probabilistic robotics. Communications of the ACM, 45(3):52-55) 2002. URL https:
a/t..aa...og//oo...... wttp:////o/...aaaa/ooooo..1.0 111.a/////.1111010221111 504754..
Sebastian Thrun. Robotie mapping: a survey. Robolic mapping: a survey, 2003. URL https://api.
semanticscholar.
:t: 14633188.
Sebastian Thrun and Michael Montemerlo.
The graph SLAM algorithm with applications to large-scale mapping of urban structures. The International Journal of Robotics Research, 25(5-6):403-45 -299, 2006. URL htggppp..co//aall/////..........
https://journals sagepub. com/ doi /tt/////0..117/00000040000000000
4,
Scbastian Thrun, Wolfram Burgard, and Dicter Fox. A probabilistic approach to concurrent mapping and localization for mobile robots. Autonomous Robots, 5:253-271, 1998a. URL https: htt///iiin.iiiigge...
:tttccee/1..1023A::1000 100000020001000000555544444
57

---

Published in Transactions on Machine Learning Research (August/2025)
Sebastian Thrun, Jens-Steffen Gutmann, Dieter Fox, Wolfram Burgard, Benjamin Kuipers, et al. Integrating topological and metric maps for mobile robot navigation: A statistical approach. AAAT/TAAT, 9:989-999 -995, 1998b. URL https://cdn. www..aaii.o..m Orgg/AAA//1999/AAAT0--- 140.:88811.......... 3, 21
Sebastian Thrun, Wolfram Burgard, and Dieter Fox. A real-time algorithm for mobile robot mapping with applications to multi-robot and 3D mapping. In Proceedings ICRA. Millennium Conference. IEEE International Conference Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 1, pp. 321-325 IEFE, 2000. URL https: http://eeeeeppeeeeeeeee ieee. 5
Scbastian Thrun, Dicter Fox, Wolfram Burgard, and Frank Dellacrt. Robust Monte Carlo localization for mobile robots. Artif. Intell, 128:99 141, 2001. URL
128871. 4
Nicola Tomatis, Illah Nourbakhsh, and Roland Siegwart. Combining topological and metric: A natural integration for simultaneous localization and map building. In Proceedings of the Fourth European Workshop on Advanced Mobile Robots (Eurobot). ETH-Zürich, 2001. URL https: httpp://wwwrr www..reeearch-coleetttiie ethz. corisstream/handeee 200 500. 11850/ 11850/82590 82590/ /eth- wtte/ethhh-85-7 8357- påf. 3, 21
Ali Tourani, Hriday Bavle, Jose Luis Sanchez-Lopez, and Holger Voos. Visual slam: What are the current trends and what to expect? Sensors, 22(23):9297, 2022. URL https::t///// htttps:/wwww.mmi.... http:/cc////12222///////////
8
Brandon Trabucco, Gunnar A Sigurdsson, Robinson Piramuthu, Gaurav Sukhatme, and Ruslan Salakhut- dinov. A simple approach for visual room rearrangement: 3D mapping and semantic search. In The International Conference on Learning Representations, 2022. URL https: http://pppnneeeeneeeeeeeeee net/forum?id 1C6nCCaRe6p. 10
Rudolph Triebel, Rohan Paul, Daniela Rus, and Paul Newman, Parsing outdoor scenes from streamed 3d laser data using online clustering and incremental belief updates. In AAAT, volume 26, pp. 2012. URL https://oje aaai orgi.org///a.aaaaa....... org/iidex////// /inex.dxxx http:/inndpppddaaaee/e
3, 19
Konstantinos A Tsintotas, Loukas Bampis, and Antonios Gasteratos. The revisiting problem in simultancous localization and mapping: survey on visual loop closure detection. IEEE Trunsactions on Intelligent Transportation Systems, 23(11):19929-199 19953, 2022. URL
www..eeeee.eeeeeeeeeeeeeeeeeeeee org/abstract.
document/9780121/. 6
Mycal Tucker, Derya Aksaray, Rohan Paul, Gregory J Stein, and Nicholas Roy. Learning unknown groundings for natural language interaction with mobile robots. In Robotics Research: The Interna- tional Symposium ISRR, pp. 317-333. Springer, 2019. URL 16
bhink.springer wtt.cc////////cceeee ham/chapter/10...
00000/978-3-00-2811199
Julien PC Valentin, Sunando Sengupta, Jonathan Warrell, Ali Shahrokni, and Philip HS Torr. Mesh based semantic modelling for indoor and outdoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2067-2074, 2013. URL /opta///ppeaccees. thecvf.com/ Mesh Based Seteeateeeaeaaaeeeee 2t__2221111___pp______ CWPR__apper.eeeeeeeeppp html. 18 Maeh_Baaseeeee_
content_cvpr._ 1ipp__2013/htmm/aaeess
Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobileclip: Fast image-text models through multi-modal reinforced training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 15963 15974, 2024. URL http://openaccest thecvf. wwt. /ww.ttttenn//////////eetttaoaaaa http:///mmmm///mmaaeeeeaaaa Models, Baciss_thrrugh_uua
/Vaas_MMoilaeLLLTTTT MobileclillPPp Fast_Imagee Image- Textt_
Reinforced Swwaaforree_Traiinnnnn_tom Biteaiiinn__cR____1 CWWCC_VVR___p_eeeee 20w___2ae2_________2__ html. 17
Shrihari Vasudevan and Roland Siegwart. Bayesian space conceptuallzation and place classification for semantic maps in mobile robotics. Robotics and Autonomous Systems, 56(6):522-537, 2008. URL https:
// //wwwsccienceirrectt httpa///iinneaaarraea///////////
21
Vibhav Vincet, Ondrej Miksik, Morten Lidegaard, Matthias Nicßner, Stuart Golodetz, Victor A Prisacariu, Olaf Kähler, David W Murray, Shahram Izadi, Patrick Pérez, et al. Incremental dense semantic stereo fusion
58

---

Published in Transactions on Machine Learning Research (Auuust/2225
for large-scale semantic scene reconstruction. In JEEE international conference robotics and automation (TCRA), pp. 75 75-82. 82. TEEE, 2015. URL https:///eeexplloe www.eeeeerrr.
Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mchdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. NeSF: Neural semantic fields for generalizable semantic seginentation of 3D scenes. arXiv preprint 0rXiv:2111.1300 2021. URL http:///ww...e.eeeeeaaaaaaaaaaea htt///bbttrttt/ooeeee///////S/e
wwwpe.rggggaa...... att////s//1111 13260. 19, 21
Matthew R Walter, Sachithra Hemachandra, Bianca Homberg, Stefanie Tellex, and Seth J Teller. Learning Semantic Maps from Natural Language Descriptions. In Robotics: science and systems, volume 2, 2013.
URL https://dspace wttp:eeeetteeeeeeeeeeemeeemme edu/handle/1721. /tt.//11........1... 1/87051. 21
Matthew R Walter, Sachithra Hemachandra, Bianca. Homberg, Stefanie Tellex, and Seth Teller. A framework for learning semantic maps from grounded natural language descriptions. The International Journal of Robotics Research, (399)11667-1990) 2014. URL 0278364914537359 3, 21
Matthew R. Walter, Siddharth Patki, Andrea F. Daniele, Ethan Fahnestock, Felix Duvallet, Sachithra Hemachandra, Jean Oh, Anthony Stentz, Nicholas Roy, and Thomas M. Howard. Language Understanding for Field and Service Robots in a Priori Unknown Environments. Field Robotics, 2:1191-1231, 2022. doi: 00.54477/f:2022044 URL attps://ieeexplore org/ abstract//oouuunt/ooo////
21
Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text- and-image driven manipulation of neural radiance fields. In Proceedings of the TREE/CVE conference on computer vision and pattern recognition, pp. 3835 3844, 2022. URL http: //openaccess thecvf. cooteett/UVP22222/ttma Manipulation of Neural Radiance Fields Lateie_eeellleeeeeeeeeeeeeeeeeee 2UPPP_2022_____11111___
CLIP-NeRF Text-ande aentt-ann-mmgaee....
html. 21
Yanan Wang, Yaobin Tian, Jiawei Chen, Kun Xu, and Xilun Ding. A survey of visual SLAM in dynamic environment: The evolution from geometric to semantic approaches. IEEE Transactions Instrumentation 8
and Measurement, 2024. CRL https://ieeexppore ieee. wtg//bbstraactbbocun////////7
Zehao Wang, Mingxiao Li, Minye Wu, Faanc-Francine Moens, and Tinne Thytelaars. Instruction-guidee path planning with 3D semantic maps for vision-language navigation. Newrocomputingg pp. 129457, 2025. URL https:/ //ww. sciencedirect.
3, 16
Saim Wani, Shivansh Patel, Unnat Jain, Angel Chang, and Manolis Savva. MultiON: Benchmark- ing semantic map memory using multi-object navigation. Advances in Neural Information Pro- cessing Systems, 33:9700 9712, 2020. URL attps://proceeding prroceedings.neuripa. cc/paper/202///aaaeeeeeeeeeeeee 6601883f00a177651111154477744844 Aattaa--bttaat......e html. 3, 5, 8, 10, 15, 16, 24, 28
Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In Proceedings of the IEEE/CVE Conference on Computer Vision and Patlern Recognilion, 2021. /tte//eiihs__ssaaalaaeaaaaaaaall Visual Room Reoo_neeeraagementt
URL attp://openaccess. thecvf. com/ http://ootaaa////20200////////te
html.
Lorenz Wellhauson, René Ranftl, and Marco Hutter. Safe Robot Navigation Via Multi-Modal Anomaly De- Lection. IEEE Robolies and Automation Lellers, 5:1326 1333, 2020. URL ltlo.
semanticscholl...
thg/CorpusDD:21085544 28
Shuhuan Wen, Ziyuan Zhang, Yuxiang Sun, and Zhiwen Wang. OVL-MAP: An Online Visual Language Map Approach for Aision-nddLangguge Navigation in Continuous Environments. IEEE Robotics and Automation Letters, 2025. URL https: /ttee://ieeexploree ieee. orgg///scaat///cocmm////01111/// 3, 25
Abdelrhman Werby, Chenguang Huang, Martin Büchner, Abhinav Valada, and Wolfram Burgard. Hicrarchical Open- Vocabulayy 3D Scene Graphs for Language-Grounded http://pwweeveeee.eeeee/r//rr/ap
Grounded Robot Navigation. In RSS, 2024. URL
22, 25
59

---

Published in Transactions on Machine Learning Research (Auuust/2255
Thomas Whelan, Stefan Leutenegger, Renato F Salas-Moreno, Ben Glocker, and Andrew J Davison. Elastic- Fusion: Dense SLAM without pose graph. In Robolics: science and systems, volume 11, pp. 3. Rome, 2015. URL http:///bbbiicpppcceiiggggggge httph//g....rrrrr//.........1 pdf. 20
Thomas Whelan, Renato F Salas-Moreno, Ben Glocker, Andrew J Davison, and Stefan Leutenegger. Flastic- Fusion: Real-time dense SLAM and light source estimation. The International Journal of Robotics Research, URL
35(14):1697-17 1716, 2016.
pdf. 7
Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied Question Answering in Photorealistic Environments with Point Cloud Perception. In Proceedings of the IEEE Conference Computer Vision and Pattern Recog- nition (CVPR), 2019a. URL http://openaccess twwctf.com/coteeeeeeeeee contentotnttt..m/e _______V________ Embodied _esttdiueeeinnnne Answering Becsin__hhotonn Phootooioooo sooooorrallsiiiifaavoooneesssss Environments_Mive Saatth_Poitt__eee Point_Coonttdddoottt _ec_ooddeeeeeptoorrrrceoeo papprrppppr.._ html. 5, 30
CVPR_2019/html/// jmammass__
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Trfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO: Learning near- perfect pointgoal navigators from 2.5 billion frames. In International Conference on Learning Representations, 2019b. URL https://arxiv wtgg/ab//911..00/ 00357. 5, 9, 10, 25,
27, 30
Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scene- graphfusion: Isncremental 3d scene graph prediction from rgb-d sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7515-7525, 2021. URL
thecvf. Baaee-DD_Suuuces______ __P_____________________ 2021 20wp__aper_______ htm.p..per.............. html.
Cttpa///0///tta/aaoooooorrrrraar SceneGraphFus Incremental
3D_ Sceneeen___ Crtph_rapppppa Prredetieeddiinnn From From_RCB-D R_B___GBDD____
3D_
Yuchen Wu, Pengcheng Zhang, Meiying Gu, Jin Zheng, and Xiao Bai. Embodied navigation with multi- modal information: A survey from tasks to methodology. Information Fusion, pp. 102532, 2024. URL
Fei Xia, Amir R. Zamir, Zhi- Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson Env: real-world perception for embodied agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. URL http: thecvf. comp/content_ccpp Sepepatt__________ cvpp__208//taaaaam h/html/
Saa__eel--ee Bhal-Worldd_rVe SV___VVR__0000 _0___2211________ html. 16
Linlin Xia, Jiashuo Cui, Ran Shen, Xun Xu, Yiping Gao, and Xinying Li. A survey of image semantics- based visual simultancous localization and mapping: Application-oriented solutions to autonomous navigation of mobile robots. International Journal of Advanced Robotic Systems, 17(3))112988122099998 2020. URL htttps:/journalll sagepub. wwt.coocc/cccccccccccctcoc...ttt /as//bbs/0000 0ttp//...117/1720100 1TT.//17228814209999982
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangehen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. In Pro- Computer Vision and Pattern Recognition, pp. 11097-11107, ceedings of the IEEE/CVF Conference ctm/ cont//tntett...m __VP___CP______________ 2020/ http//httm/////e/ SAPIEN A SimulAted 2020. URL http://openaccess twwcwf.....cc Bart-Based__m Interactive Wwwieveeeeeeeeeeeee_eo Seepttt__vR_____22 _aw.020_ppppr.._.. ootaaaaetteecccceeeeeeeeetoooooo
SAPIEN
html. 16
Yu Xiang and Dieter Fox. DA-RNN: Semantic mapping with data associated recurrent neural networks.
arXiv preprint arxiv: 170.: 11111703..00000 03098, 2017. URL https://arxiv http://aaxxvo.rgab////////s
org/abs//////oooooooo...oo... 1703. watpp///7...00...... 7,
Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. SED: A simple encoder- decoder for open-vocabulary semantic sogmentation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 3426 1126-3436, 2024. URL httpp///peeaacees.. thecvf www.cooccccooo SearrrsSSmmmiiiiie http://rw....antteeeeeeeeerraarr
cttp///oattatt////22//////////// SED A L Encoder-
html. 26
Xuchan Xiong, Daniel Munoz, J Andrew Bagnell, and Martial Hebert. 3-D scene analysis via sequenced predictions over points and regions. In IEEE International Conference on Robotics and Automation, pp. IEFE, 2011. URL http:///weeeeoee....eeerr/e//e/s
16
60

---

Published in Transactions on Machine Learning Research (August/2255)
Xhuwei Xu, Huangxing Chen, Linqing Zhao, Ziwei Wang, Jie Zhou, and Jiwen Lu. EmbodiedSAM: Online Segment Any 3D Thing in Real Time. Xiv preprint Xrxv:22488. 1 11811, 2024. URL https://arxiv.orrs abs/2408. 11811. 18, 20
Karmesh Yadav, Santhosh Kumar Ramakrishnan, John Turner, Aaron Gokaslan, Oleksandr Maksymets, Rishabh Jain, Ram Ramrakhya, Angel X Chang, Alexander Clegg, Manolis Savva, Eric Undersander, Devendra Singh Chaplot, and Dhruv Batra. Habitat challenge 2022. attps://aihabita org/ challenge/ 2022/, 2022. 5
Karmesh Yadav, Ram Ramrakhya, Santhosh Kumar Ramakrishnan, Theo Gervet, John Turner, Aaron Gokaslan, Noah Maestre, Angel Xuan Chang, Dhruv Batra, Manolis Savva, et al. Habitat-Matterpon 3D Semantics dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 4927-4936, 2023. URL
thecvf com/ contenttCCVPR2223/mmma /Yadav__
Rabitat-Matterport.u Serrrport_t_Semmattii Se__emmaaticcsss Datasaattttttae CVPR SttpP__222_pppprr._.__ . html. 16
Brian Yamauchi. A frontier-basee approach for autonomous exploration. In IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA). 'Towards New Computational Principles for Robotics and Automation', pp. 146-151, 1997. URL
leee. org/ atg/abbbrrat//dccmennt
613851/. 9
Bo Yang, Tri Minh Triet Pham, and Jinqiu Yang. Robustness of LiDAR-Based Pose Estimation: Evaluating and Improving Odometry and Localization Under Common Point Cloud Corruptions, 2025. URL https:
//arxiv. org/abs https//29....888 29
Shichao Yang and Sebastian Scherer. Direct monocular odometry using points and lines. In JEEE International Conference on Robotics and Automation (ICRA), pp. 3871 3877. EEEE, 2017. URL https://ieeexploor
ieee. Leee.org/ee http///ttract/cmmeeee/e///eeeeee
Shichao Yang and Sebastian Scherer. Cubeslam: Monocular 3-d object slam. IEEE Transactions on Robotics, 35(4):925-938, 2019. URL https://eeeexplore ieee. org/ /ttp///atcet//omoo//////////////
Yuncong Yang, Han Yang, Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, and Chuang Gan. 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning, 2024. URL 17735. 17
2411.
Ruochen Yin, Yong Cheng, Huapeng Wu, Yuntao Song, Biao Yu, and Rumxin Niu. Fusionlane: Multi- sensor fusion for lane marking semantic segmentation using deep neural networks. IEEE Transactions on Intelligent Transportation Systems, 23(2):1543 1553, 2020. URL https://ieeexpporeeeeeee abssract/2ocmmet/92271
Naoki Yokoyama, Schoon Ha, Dhruv Batra, Jiuguang Wang, and Bernadette Bucher. VLFM: Vision-language frontier maps for zero-shot semantic navigation. Workshop Language and Robot Learning, CoRL 2023, 2023. URL https: h/1eexp//eee leee. 25, 29
Atlanta, GA, USA. 3, 9,
Naoki Yokoyama, Ram Ramrakhya, Abhishek Das, Dhruv Batra, and Schoon Ha. HM3D-OVON. A Datasct and Benchmark for Open-Vocabuler Object Goal Navigation. aiv preprint ar Xrv:224409. 14296, 2024. ieee.
URL https://ieeexplae
27, 30
Tsuneo Yoshikawa. Manipulability of Robotic Mechanisms. The International Journal of Robolics Research, 4:3 - 9, 1985. URL
htt.aapp.smmaaniccchlitttte http:/oaa.....t///ppppp1111111/1 4
Georges Younes, Daniel Asmar, Elie Shammas, and John Zelek. Keyframe-based monocular SLAM: design, survey, and future directions. Robotics and Autonomous Systems, 98:67 88, 2017. URL https: //www sciencedirect. sccianccdieeet..oossssccc//rrrre http://canneerrreeee//e/////////
8
Hendrik Zender, Martínez Mozos, Patric Jensfelt, G-JM Kruijff, and Wolfram Burgard. Conceptual spatial representations for indoor mobile robots. Robotics and Autonomous Systems, 56(6):493-500, 2008. URL rett http///wwwaaaateeeoo///oocaaaaat 3, 21
61

---

Published in Transactions on Machine Learning Research (August/2255)
Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois Robert Hogan, Maria Bauzá, Daolin Ma, Orion Taylor, Melody Liu, Fudald Romo, Nima Fazeli, Ferran Alet, Nikhil Chavan Dafle, Rachel Holladay, Isabella Morona, Prem Qu Nair, Druck Green, lan H. Taylor, Weber Liu, Thomas A. Funkhouser, and Alberto Rodriguez. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. The International Journal of Robolics Research, 41:690 705, 2017. URL https://api... semanticscholar org/ hetg/CCrpusDD... 45488377. 28
Andy Zeng, Peter R. Florence, Jonathan Tompson, Stefan Welker, Jonathan M. Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee. Transporter Networks: Rearranging the Visual World for Robotic Manipulation. Tn Conference on Robot Learning, 2020. URL
http:///app.ssmmaicccccllllaa wew.ataccccoooaa...ooooooppppppa ottp/// 4, 28
Jiazhao Zhang, Chenyang Zhu, Lintao Zheng, and Kai Xu. Fusion-aware point convolution for online semantic 3d scene segmentation. In Proceedings of the TEEE/CVF Conference Compuler Vision and Pattern Recognition, pp. 4534-4543, 2020. URL http://openacceet thecvf. cwom/contettteeee Secpat____P______0 Baaal/hhang_Fusiine Swaiiioonwwwae...... Latare_Poiitt,. Cotttttoooooooioonnoooo for for_Online i _D LUPR_2020_papeet html. 20
Srriiic___eeeeeeeeeee Scene Segmentation.
Jiazhao Zhang, Liu Dai, Fanpeng Meng, Qingnan Fan, Xuelin Chen, Kai Xu, and He Wang. 3D-aware object goal navigation via simultaneous exploration and identification. arXiv preprint 2r1 00338, 2022a. URL attp://openaccess thecvf. com/ /ttp///cee////Pr20////a/////aaaa Goal Www.aaaaiinnvviie _iaaa_iiii____ Simultaneous SaamulaaaeesssEEplllorrtooooaan and Idw.aiiiittiiiiaiiionnn.ooo http:/riiiaaiiooc_c_PO_0 papeppaperr..ee paper.html. html. 19
//aammm/////aaaaaa 3D- Aware _bbjcct__
Jiazhao Zhang, Liu Dai, Fanpeng Meng, Qingnan Fan, Xuclin Chen, Kai Xu, and He Wang. 3D-aware object goal navigation via simultaneous exploration and identification. In Proceed- tngs of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. Awaee _
2023a. URL http:://penaccent thecvf. cwm/comm//////cocce /ttp///en/////2////mmm/a//////// Goal Navigation via Simultaneous Exploration and Identificatio wetpaatiionccvR______ 2023 paper. html. 3, 20
Lingfeng Zhang, Xiaoshuai Hao, Qinwen Xu, Qiang Zhang, Xinyao Zhang, Pengwei Wang, Jing Zhang, Zhongyuan Wang, Shanghang Zhang, and Renjing Xu. MapNav: A Novel Memory Representation via An- notated Semantic Maps for VLM-based Vision-and- Lan-uaugg- Navigation. arXie preprinl trXiv:2502.13551 13451, 2025. URL bctps://arzis wwwrrivv.ogg/aae ab0// 13451. 3, 23
Tianyao Zhang, Xiaoguang Hu, Jin Xiao, and Guofeng Zhang. A survey of visual navigation: From geometry to embodied AI. Engineering Applications of Artificial Intelligence, 114:105036, 2022b. URL https://ieexxplore.iee org/
Xuetao Zhang, Yubin Chu, Yisha Liu, Xuebo Zhang, and Yan Zhuang. A novel informative autonomous exploration strategy with uniform sampling for quadrotors. IEEE transactions industrial electronics, 69 (12):13131-1311) 2021. URL https: //mp::////eeex/ore/ ieee. iew....eeeeeeee..........eeeeee. http:///asssaatttt/oee/////eeeee
9
Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. arXiv preprint ar arXiv:2306. 03514, 2023b. URL https://openaccess thecvf com cot.een//CVRR2224//// httm///hang_Recee Recoggggeeeeeeeeeeeeeeeeeeeeeeee Eeccognize_Anythinn.g. Anything. A __Strongg.. Image BaaigiinnnnoMee_____ 2222 _014__apprr.._ pamper.hmmm... html. 10
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer.
Proceedings of the IEEE/CVF international conference on computer vision, pp. 16259-16268, 2021. URL //ppss///ppncccss... seeeraaccess.theee.eeeee thecvf. tttecvf.com//onteeeeet hwtp////ttee///////////////////a Point Transformer. ICCV
@BcV_2021_papeee www.ppappp...t. hhmpl. html. 20
Junhui Zhao, Jingyue Shi, and Li Zhuo. BEV percoption for autonomous driving: State of the and future perspectives. Expert Systems with Applications, 258:125103, 2024. URL https://www.sciencdne
62

---

Published in Transactions on Machine Learning Research (August/225))
Ying Zheng, Lei Yao, Yuejiao Su, Yi Zhang, Yi Wang, Sicheng Zhao, Yiyi Zhang, and Lap-Pui Chan. A survey of embodied learning for object-centric robotic manipnlation. arXie preprint arXiv:2408. 1 11537, 2024a. URL https://inn.sppinger.com
springgerr http:////....///.../////1/111-00
Yuhang Zheng, Xiangyu Chen, Yupeng Zheng, Songen Gu, Runyi Yang, Bu Jin, Pengfei Li, Chengliang Zhong, Zengmao Wang, Lina Liu, et al. GaussianGrasper: 3D language gaussian splatting for open-vocabulary robolic grasping. IEEE Robolics and Aulomation Lellers, 2024b. URL https: / //eee/xpeeeeeeeeeeeeee ieee org/ interract/bccumnt//0007777
19, 20
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15838 15847, 2021. URL https://openacceen thecvf. com/ /ontoat///o//2/////mmm////// In-IIn-PPPP Place_Scene Scene Labellingeelaaeee and_ Understanding_Ni hetptttannnnn_iihhhhhhe Baaichh_mmliiitSeeeeeeeee S_cetttS_eeee_ee Representation Bicpeaatttooioooo__ee 1aal___2221______111 ppaper.hre html. 19, 21
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from CLIP. In European Conference Computer Vision, pp. 696 712. Springer, 2022. URL https: https://link... http//iinikk.iingr.....mm www.aape...oo/cecieieeeeeeeeeeee ccmm/haapter110100077 978- 1178--031--0 1180001111111111 1_40. 19
Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, and Xin Eric Wang. ESC: Exploration with soft commonsense constraints for zero-shot object navigation. In ICML, pp. 42829 42842. PMLR, 2023. URL https::/proceediii wwtp./......ee//////ooo//o/
html. 9
Fengda Zhu, Yi Zhu, Vincent Lee, Xiaodan Liang, and Xiaojun Chang. Deep learning for embodied vision navigation: survey, orXio preprint arXiv:2108.04097 2021. URL https://aaxxv..rrgabb///// 04097
Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin Os- wald, and Mare Pollefeys. NICE-SLAM: Neural implicit scalable encoding for SLAM. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12786 12796, 2022. URL http::/openaccceeett whwccceessss...e.eeeeeee thecvf. com/ wwwp///oteee//////////e/ttteee 2tt/222222/hm//huuu Implicit_Scall Scalable Bacooole_Ecodinn_tee. for SLAM CVAM__CVPRRREA 2022 paperp html. 21
Neural
63


Reviews:
[{"review_id": "D12ocnHSAQ", "reviewer": "Reviewer_qqo5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The paper presents a categorization of semantic mapping papers based on two criteria. \n* The paper is fluently written and easy to follow. \n\n\nWeaknesses:\n\n* This paper limits its scope to \"semantic mapping in indoor embodiment AI\". It assumes perfect localization performance and explicitly excludes papers related to \"SLAM\" and \"semantic mapping in robotics (real-world robots)\". While I can appreciate the authors' intent to focus specifically on the mapping component (as illustrated in Fig 3), I still feel the SLAM literature captures a huge portion of semantic mapping and the survey should include how the mapping process has evolved in the semantic SLAM literature. Also, most semantic mapping works that are designed for *real-world robots* can be applied or adapted to embodied AI tasks, and I don't think they should be excluded based on this criterion. \n* Some categories are missing and not reviewed by the paper, which mostly corresponds to empty entries in the upper triangle of Table 2. I'm sure there are many works that store explicit semantic encodings on point clouds and topological maps (e.g. [R1]). Why are they not included in this survey? \n* While Section 4.3 is a \"point-cloud map\", it mostly reviews the semantic mapping based on neural fields but not the works that actually use point clouds as 3D representations (e.g. [R2]). I suggest the authors separate the implicit 3D representations (SDF, Neural Field, NeRF, 3DGS; 3DGS-based maps are entirely missing from this survey) as an extra subsection in \"Map Structure\". Review on point cloud-based maps should be redone. ([R1-R3] are all important works in this subfield but they are not mentioned at all in this survey.) \n* The Map Evaluation section is very superficial. Discussion of existing benchmarks, evaluation metrics, and quantitative comparisons should included. A table listing and comparing popular benchmarks and evaluation metrics from different embodied AI tasks is helpful. The quantitative comparisons can come from some existing popular datasets and benchmarks (e.g. 3D semantic segmentation on ScanNet, navigation performance on HM3D dataset). Without these, I hardly learn any takeaways from this paper on how different techniques compare to each other. \n* Section 2 is loosely written and I don't see how it provides value to a survey on semantic mapping. Section 3 reads similarly loose. Some less relevant discussions can be omitted. \n* Overall the writing of the paper needs to be improved. Specifically, I want the authors to highlight how they \"draw new connections, highlight trends, and suggest new problems in an area\" (according to the TMLR review guidance). \n\n\n[R1] J. McCormac, A. Handa, A. Davison and S. Leutenegger, \"SemanticFusion: Dense 3D semantic mapping with convolutional neural networks,\" 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 2017, pp. 4628-4635, doi: 10.1109/ICRA.2017.7989538.\n\n[R2] Jatavallabhula, Krishna Murthy, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li et al. \"Conceptfusion: Open-set multimodal 3d mapping.\" RSS 2023.\n\n[R3] Werby, Abdelrhman, Chenguang Huang, Martin Büchner, Abhinav Valada, and Wolfram Burgard. \"Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation.\" RSS 2024.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a survey on semantic mapping in the research field of indoor embodied AI. \nThe paper first introduces the background of indoor embodied AI and discusses the basic aspect of semantic mapping. \nThen it extensively reviews and comments on the relevant papers in the mentioned area. \nThese papers are further categorized based on two criteria. \nBased on map structure, this paper categorizes them into spatial grid maps, topological maps, and point-cloud maps. \nBased on semantic encodings stored on the map, this paper categorizes the papers into explicit encodings and implicit encodings. \nThen the paper briefly introduces the types of evaluation performed on the built semantic map. \nFinally, the paper discusses several potential future directions for the research community to pursue.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the weaknesses I mentioned in the previous section. \n\nAlso, I think this survey probably misses many important works and entire subfields in the literature ([R1-R3] as mentioned above and those mentioned by the Action Editor). Please conduct more extensive research and include them.", "broader_impact_concerns": "I didn't find any significant broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2yHsHf4xur", "reviewer": "Reviewer_oQ6x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ I liked the conceptual presentation of the work, explaining the process in terms of building blocks. It more or less walks the reader through the the feature extraction process, lifting to 3D and generation of map in world coordinates or BEV, together with SLAM. \n\n- As I work in autonomous driving and BEV fusion, and mostly using transformers as machinery, I feel that the work does not cover this aspect adequately. DETR based approaches should be covered in much more detail. Likewise, NERF based approaches also warrant similar comment.\n\n- The above also goes into covering end to end approaches for planning (i.e. going all the way from perception to trajectories and planning). \n\n- SLAM also needs more coverage with equations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a review paper on semantic mapping in indoor environments for robotics applications. The paper goes over the general setting for  mapping and localization and its motivation in robotics applications (e.g. end to end approaches), goes over some of the methods and points to future directions. The main focus seems to be on the semantic mapping ideas, which is explained phenomenonologically, and broken down into commonly used mapping types (e.g. topological map).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please cover transformer based approaches in more detail, together with NeRF and Gaussian splatting. I think it at least needs a passing mention of how transformers can be used for the task - an encoder maps the data to a latent representation, and on the decoder side, we could use it for map/lane estimation tasks e.g. [1]\n\n[1]: https://arxiv.org/abs/2312.16108", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Enqi85jnHP", "reviewer": "Reviewer_d4MD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n+ The survey covers a wide range of works in semantic mapping and navigation, which are timely topics in embodied-AI. Quite a lot of learning-based methods struggle to solve long-horizon and partially-observable instruction-following tasks. Semantic mapping provides a structured alternative to end-to-end learning, and this survey provides a nice summary of such methods. \n+ The background material on end-to-end vs. modular methods, SLAM, and semantic mapping provide a good intro to keep the survey self-sufficient.\n+ The categorization of methods seem appropriate, i.e. map structure into: grid, topological, point-cloud hybrid maps, and map encoding into: explicit and implicit.\n\nWeaknesses  \n- While the survey is informative, the general structure and writing can be greatly improved. More specifically:  \n(1) The introduction seems disconnected from the rest of the survey. The first few paragraphs motivate a generic problem in object navigation (i.e., finding “a red and blue striped zebra toy in the nursery”). The same problem is also highlighted in Figure 1. The “zebra toy” example gives the impression that the survey is about vision-language grounding or vision-language navigation, which are potential “applications of semantic mapping” but not about “semantic mapping” itself.  Understandably, object-navigation is closely linked to semantic mapping, but the introduction should be clearly on what is the key problem.   \n(2) The introduction also doesn’t motivate the key categorizations in the survey, e.g. map structure into: grid, topological etc. It would be good to motivate the categorizations early on, since they are the core contributions of a survey paper.   \n(3) The survey would greatly benefit from a unifying table or figure to summarize all method subsections and their categories. Right now, it’s difficult to quickly grasp the structure without looking through 26 pages of content.  \n  \n\n  \n- The evaluation metrics are mentioned in passing without sufficient detail. For example, SPL is mentioned, without explaining what SPL stands for, i.e. success-weighted path length, or why SPL is necessary. It is important to keep such details self-contained. Besides just summarizing techniques, a survey should also motivate the reason behind techniques, and expound on their strengths and limitations.\n\n- The title mentions “advances, challenges, and future directions”, but “challenges” are not explicitly discussed for all methods. Some challenges are discussed alongside specific methods, but there are no overarching challenges discussed for a group/category of methods. \n\n- The citations in Table 2, 4, and 5 are too sparse and biased towards recent works. The AC has provided a broad set of relevant papers in semantic and topological mapping. It would be good to incorporate these works and sort them into the existing categorization. \n\n\nSmall nits:\n- Table 2: the split between Explicit (no semantics), Explicit (semantics), and Implicit to Explicit rows is not clear. Perhaps adding additional lines or separate colors could make it easier to read.\n\n- Some sub-sections end with a summary (e.g. Section 4), but not others. Why not include summaries for all sections?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a survey of semantic mapping and navigation methods in indoor (simulated) environments. The survey is split into 6 sections: (1) background material on tasks, approaches, and SLAM, (2) the what and how of semantic mapping, (3) an overview of structural representations for maps like grids, topological, and hybrid maps, (4) a summary on map encoding methods, (5) evaluation metrics, and (6) future directions for the field. The survey covers a broad range of works from the embodied-AI community, with a strong emphasis on indoor environments set in simulated household scenes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Summarizing from above:\n- Improve the connection between introduction and subsections.\n- Choose a better example for Figure 1 that is more aligned with “semantic mapping”, rather than \n“language grounding for object-navigation”. \n- Consider an overview figure/table that summarizes all the categorizations.\n- Explicitly distinguish between “semantic mapping” and “object-navigation”. Or separate “object-navigation” into an independent Applications section.\n- Explain evaluation metrics in more detail so that the section is self-sufficient.\n- Discuss challenges explicitly in a separate section, or make the challenges in the existing sections clear.\n- Add additional references provided by AC, and sort them with the existing categories.", "broader_impact_concerns": "No major concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)