Topo-Field: Topometric mapping with Brain-inspired Hierarchical Layout-Object-Position Fields

## Abstract

Abstract Spatial cognition empowers animals with remarkably efficient navigation abilities, largely depending on the scene-level understanding of spatial environments. Recently, it has been found that a neural population in the postrhinal cortex of rat brains is more strongly tuned to the spatial layout rather than objects in a scene. Inspired by the representations of spatial layout in local scenes to encode different regions separately, we proposed LOP-Field that realizes the Layout-Object-Position(LOP) association to model the hierarchical representations for robotic scene understanding. Powered by foundation models and implicit scene representation, a neural field is implemented as a scene memory for robots, storing a queryable representation of scenes with position-wise, object-wise, and layout-wise information. To validate the built LOP association, the model is tested to infer region information from 3D positions with quantitative metrics, achieving an average accuracy of more than 88%. It is also shown that the proposed method using region information can achieve improved object and view localization results with text and RGB input compared to state-of-the-art localization methods.

## 1 Introduction

Spatial cognition is a fundamental function that enables humans and animals to achieve long-term autonomy in their environment. A cognitive map is considered a mental representation of spatial information about the relative locations and attributes of phenomena in our everyday spatial environment . Place cells encode the specific locations of rodents in the environment depending on both scene content and spatial layout .
Spatial view cells in the hippocampus become active when scene contents of the environment are in the animal’s field of view . Various boundary cells ; ; ; encode the allocentric scene border whatever the scene contents are.
A particular population of neurons in the postrhinal cortex (POR) is more sensitive to the spatial layout of a local scene than the spatial contents . A theory of geometry representations is proposed to describe various boundary-related cells and representations of POR in a unified framework. The predicted geometry cells by the theory are able to encode spatial layouts with different geometric structures, which helps to quickly form a high-level cognitive map representation . The spatial layout, connected by regions, may play a vital role in spatial cognition, reasoning, and navigation, integrating with the purpose of the scene and object content semantics.

Inspired by neural representations of spatial layout, scene contents, and locations, spatial information can be categorized into (1) layout-level information, which includes the layout, region, and connectivity of spaces, (2) object-level information, which includes the attributes, appearance, and positions of various objects, and (3) position-level information, which includes the relative positions, associations, and modes of interaction among object parts. For example, people can distinguish different regions within their homes, recognizing the differences between the living room, bedroom, and kitchen. They can build knowledge and memories about the relationship between target objects (e.g., bed) and their corresponding regions (e.g., bedroom), and they can distinguish similar objects within different regions (e.g., a cup in the living room versus a cup in the kitchen). Similarly, if a robot could understand the relationships between spatial regions as humans do, it would be able to perform tasks such as spatial reasoning and layout-object associations. Fig. 1 shows that with layout-object-position association, robots could have enhanced spatial cognition and understanding capabilities.

Figure: Figure 1: Dividing the scene information into layout, object, and position, and modeling them explicitly, layout-object-position association enables robots to address relative problems and realize a more comprehensive spatial cognition.
Refer to caption: extracted/5659575/images/idea.png

In robotics research on spatial scene understanding, current efforts have yielded impressive results in tasks such as 3D environment reconstruction; ; ; ; ; , object detection; ; ; ; , and object segmentation; ; ; ; ; . However, most of these works have focused on producing lifelike scene reconstructions and precise geometric and semantic information about objects, with relatively few studies addressing the modeling and recognition of spatial layouts, such as scene regions, and the association with spatial contents. The lack of layout information and scalable association in scenes hinders a robot’s comprehensive understanding and makes it difficult to interpret related commands.

How to enable robots to learn about spatial regions and association with contents remains a challenging problem, however, recent advances in large foundation models offer potential solutions. Large foundation models trained on massive datasets across various scenes, such as vision-language models(VLMs) like CLIP and large language models(LLMs) like Sentence-BERT, are believed to have the ability to reason with general knowledge and perform zero-shot inference on multiple tasks. Numerous studies leverage these models to process visual-textual features of scenes, establishing links between spatial coordinates and these features. For instance, works like CLIP-Fields and VLMaps establish mappings between spatial positions and object visual-language features, while GARField proposes hierarchical segmentation and grouping, dividing scenes into different physical scales. These efforts facilitate linking object features to 3D positions, but most existing research does not establish region recognition and lacks the integration of layout-object-position information.

To effectively integrate spatial layouts, scene objects, and position information, we introduce the LOP-Field, which realizes the Layout-Object-Position (LOP) association to model the hierarchical representations for robotic scene understanding. It integrates the spatial layout connected by regions and object-level semantics with context on 3D positions. It is equipped with the ability to reason about the relationship between the regions of the scene and its content, thus enhancing the object-level 3D reasoning capabilities of the previous work. Such a neural field can serve as a scene memory for robots, storing a queryable representation of scenes with hierarchical LOP information.
By inputting RGB-D sequences, the LOP-Field is optimized using a contrast loss between its predicted features and features from the VLM and LLM, resulting in little need for annotation. To validate the established LOP relationship, we conducted experiments on several multi-room apartment scenes. We evaluated the model’s ability to infer region information from 3D positions, providing quantitative metrics. We also demonstrated improved object and view localization results using object-region relations with text and RGB inputs. These experiments conclusively prove that LOP-Field effectively associates information of layout and scene contents from different scales.

Our contributions can be listed as follows:

- •
Inspired by the recent significant findings in neuroscience, we propose a neural scene representation named LOP-Field that integrates spatial layouts, scene objects, and 3D positions for robotic scene understanding.
- •
By fusing the object information from detected objects and layout region information from background contexts, LOP-Field builds layout-object-position association in a neural scene representation to match the vision-language and semantic feature space of large foundation models with little need for annotation.
- •
Various experiments are conducted to validate the layout-object-position association. LOP-Field achieves an accuracy of more than 95% on region inference using 3D positions and we demonstrate the help of scalable information association in downstream object and image localization tasks.

## 2 Related Works

Figure: Figure 2: Pipeline of the target embedding processing and neural implicit rendering during training. Above is the ground truth generation of layout-object-position vision-language and semantic embeddings for weakly-supervising. Below is the neural implicit network mapping 3D positions to target feature space. A contrastive loss is optimized against each other.
Refer to caption: extracted/5659575/images/method.png

Figure: Figure 3: The application examples pipeline of the LOP-Field. The region inference using position input is shown in (a). The LOP association helped localization of text and image query is shown in (b).
Refer to caption: extracted/5659575/images/infer.png

### 2.1 Spatial Understanding with Layout Information

Understanding the mechanisms of spatial cognition in humans has been a challenging and active areas of cognitive science, which also serves as an important reference for enabling robots with scene understanding.
During decades of research, scientists have made great efforts to understand the mechanism of spatial cognition. A mental representation of spatial information is proposed to describe the relative locations and attributes of phenomena in our everyday spatial environment, called a cognitive map. Place cells, as the embodiment of the cognitive map, encode the specific locations of rodents in the environment depending on both scene content and spatial layout . Scene content of the environment is represented by spatial view cells in the hippocampus, while Various boundary cells ; ; ; encode the allocentric scene boundary regardless of the scene content. Recently, Patrick et al. showed that a population code in the POR is more strongly tuned to the spatial layout than to the content in a scene. The firing activities remain consistent even when the environmental content and lighting conditions change. This suggests that there are specialized cells and signaling mechanisms to process layout in the process of scene understanding, which captures the spatial layout of complex environments to rapidly form a high-level cognitive map representation .
We propose that the spatial layout connected by regions, as a high-level abstract semantic feature, is closely related to the object contents and purposes of the scene, and therefore it can establish connections with object semantics more easily than other layout information such as area, volume, and boundary.
However, the layout regions of the scene and their association with scene content have received little attention in current robotics research on scene understanding.

### 2.2 Neural Scene Representation

Traditional robotic scene recognition methods, such as multi-view synthesis; ; ; and grid-based scene representation; , aim to reconstruct realistic new views and predict complete geometry and appearance information. However, approaches based on reprojection losses struggle to obtain sufficient constraints for optimization, and voxel-based representations face challenges when scaling to large-scale or high-resolution scenes.
To address these issues, NeRF (Neural Radiance Fields) introduced a novel approach that uses implicit neural fields to represent scene information. Subsequently, numerous efforts have been made to improve the training and inference speed of neural rendering fields; ; ; ; , to adapt them to larger scenes; ; , and to explore extended application scenarios; ; . A popular research direction is to integrate high-dimensional information, such as semantics, with NeRF to achieve a more comprehensive understanding of scenes and to address a wider range of downstream tasks; ; . However, training accurate NeRF models that incorporate semantic information requires costly manual annotation and presents challenges in adapting and applying them to different scenes.

### 2.3 Large Foundation Model Powered Scene Understanding

Recently, several robotics works have utilized large foundation models trained on web-image data to assist in the understanding of semantic information in scenes. These works have demonstrated that models trained on web-image data can be used for self-supervised learning.
Seal employed a detection model trained on web-image data to establish the connection between semantics and 3D voxels. Cliport and achieved scene understanding using weakly supervised models trained on web-image data, leveraging techniques such as CLIP. Huy Ha et al. utilized CLIP features to annotate 3D points in space. CLIP-Fields and VLMaps directly train an implicit representation of a scene using visual-linguistic features, establishing correspondence between 3D spatial points and semantics.
However, the semantic feature field learned in the above methods represents object semantics and does not include scene-level features. In contrast, in our work, CLIP and Sentence-BERT are used to generate vision-linguistic and semantic features for objects, spatial regions, and contexts, respectively. In addition to using object semantics generated by back-projection for 3D points in the scene, we annotate the belonging regions of 3D points based on spatial layout and regional division of scenes. Such annotations incur minimal cost but establish connections between the position of 3D points, object semantics, and scene regions.

## 3 Method

In this section, we first elaborate on the problem formulation of how to associate layout regions, object semantics, and 3D positions. We provide examples of the usage of LOP association, like region inference from positions and downstream localization tasks. Next, we present the process of generating the target features for training. Furthermore, we explain the structure of the employed implicit scene representation. Lastly, we describe the training procedures.

### 3.1 Problem Formulation

#### 3.1.1 Foundation-Model-Based Neural Implicit Representation

Our goal is to learn an implicit representation of a scene by establishing associations between 3D positions and their corresponding layout regions and object features. Therefore, we need to design a scene-dependent implicit function, denoted as

$$ $F:\mathbb{R}^{3}\to\mathbb{R}^{n},$ $$

where for any point $P$ in space, $F(P)$ represents the layout-object-position associated features of that point.
CLIP is introduced as the VLM in this work to encode the object and region information, integrating the vision and language feature space. Besides, the Sentece-BERT feature is also introduced in this work. Because intuitively, unlike objects that can have similar appearances within a certain category, region information often lacks specific visual appearances and is closely related to semantic representations like the integration purpose of the scene and object semantics. Models trained on large-scale question-answering datasets can aid in understanding the semantic relationships between regions and objects.
Consequently, $\mathbb{R}^{n}$ stands for embeddings:

$$ $\mathcal{E}=\{(e_{v},e_{s})\}$ $$

including vision-language embedding $e_{v}$ and semantic embedding $e_{s}$ in our approach. These predicted implicit representation outputs are targeted to match the features from the pre-trained CLIP $C$ and Sentence-BERT $S$ separately.

#### 3.1.2 Target Feature Processing

To get the target layout-object-position features, RGB-D image sequences with poses are accepted as input, what’s more, for pure RGB image sequences, depth point clouds and camera poses estimated through methods like COLMAP or simultaneous localization and mapping(SLAM) can also be used. For each image $I$, we employ Detic $D$ as the detection model to generate bounding-box-constrained object patches $B=\{b_{1},b_{2},\dots,b_{i}\}$ and labels $L=\{l_{1},l_{2},\dots,l_{i}\}$, followed by CLIP and Sentence-BERT to process the vision-language and semantic features. Given the related region $r_{P}$ and object instance $o_{P}$ of point $P$, $P$ can be labeled with $\{C(b_{P}),S(o_{P},r_{P})\}$, where the text prompt is formed as $o_{P}$ in $r_{P}$. What’s more, the background appearance is also considered which we proposed to include context information for region layout.
For background pixel $Q$ out of the object masks, its related region $r_{Q}\in R=\{r_{1},r_{2},\dots,r_{m}\}$ is regarded as the text label and its label can be calculated as $\{C(Q),S(r_{Q})\}$.
To obtain region labels of image pixels, we back-project them to 3D space based on depth and simply consider the top-down view of the 3D point cloud. The space can be partitioned into different regions using walls as dividers. Consequently, the target feature space processed by foundation models can be denoted as

$$ $\mathcal{F}=\{(f_{v},f_{s})\},$ $$

where $f_{v}$ is the visual-language feature from CLIP and $f_{s}$ is the semantic feature from Sentence-BERT. The processing pipeline is shown in Fig. 2.

#### 3.1.3 Layout-Object-Position Association

With the function and feature representation mentioned above, we can infer the region information and utilize it for various downstream tasks.

Region Inference. Using spatial 3D point $P_{i}$ as input, assuming a collection of space regions $R$, we compute the vision-language features $\mathcal{C}_{R}=\{C(r_{1}),C(r_{2}),\dots,C(r_{m})\}$ and semantic features $\mathcal{S}_{R}=\{S(r_{1}),S(r_{2}),\dots,S(r_{m})\}$.
Then the similarity between $\mathcal{E}_{P_{i}}=\{(e_{v},e_{s})\}$ and $\{\mathcal{C}_{R},\mathcal{S}_{R}\}$ is calculated to find the most likely region to which $P_{i}$ belongs. The inference process is shown in Fig. 3.

LOP Guided Object Localization. For text input $t$, such as "cup in the bedroom," most existing robotic scene representations struggle to locate specific objects of interest (differentiating between cups in the living room and the bedroom, for example). However, with our proposed LOP-Field that includes scene region information, we can calculate the similarity between $\{\mathcal{C}_{t},\mathcal{S}_{t}\}$ and the embeddings $\mathcal{E}_{P^{*}}$ of the sampled point $P^{*}$ set from the scene.
Compared with previous object localization methods, $\mathcal{E}_{P^{*}}=\{(e_{v},e_{s})\}$ includes contexts between region layout and objects by considering the object information of detected objects and region information of the background appearance.

LOP Guided View Localization. Another common robotic application is to localize a captured image of the scene. Unlike previous methods that only encode the object semantics to find matches, LOP-Field introduces region features to constrain the prediction. For image input $I$, the similarity of $\{\mathcal{C}_{I},\mathcal{S}_{I}\}$ with $\mathcal{E}_{P^{*}}=\{(e_{v},e_{s})\}$ is calculated. Compared to previous methods that only encode objects, the text label of object point $P$ is formed as $o_{P}$ in $r_{P}$(e.g., cup in the kitchen), and the background appearance with region label is also encoded. These all contribute to a more accurate localization of a specific image view. The localization of both text query and image query is shown in Fig. 3.

### 3.2 Model Architecture

Our proposed LOP-Field involves an implicit mapping function to encode the 3D positions and separate head processing encodings to match the target feature space.
To select an appropriate implicit function, considering that the target feature space includes object-level local features and layout-level region feature representation, we employ the Multi-scale Hierarchical Encoding (MHE) introduced in Instant-NGP. The feature pyramid structure used in MHE allows for considering structural features ranging from coarse to fine in the spatial domain. Additionally, MHE has a faster training speed compared to traditional NeRF network structures. For mapping the position encodings to the target feature space, we employ a unified and simple Multi-Layer Perceptron(MLP) network structure. It includes heads $head_{v}$ for obtaining vision-language features and $head_{s}$ for semantic features. The model for training is shown in Fig. 2.

### 3.3 Training

The pipeline of ground truth data generation is described in Section 3.1.2. To fit the multiple embeddings generated by the implicit representation introduced in Section 3.1.1 to the target feature space, we design the loss function through a contrastive approach.
For the vision-language feature optimization, the tempered similarity matrix on point $P$ is denoted as

$$ $\text{Sim}_{v}=\tau e_{v}C(P),$ $$

where $\tau$ is the temperature term. Using cross-entropy loss, the vision-language loss can be calculated as

$$ $\mathcal{L}_{v}=-e^{-\text{dist}_{P}}(H(\text{Sim}_{v})+H({\text{Sim}_{v}}^{T} )),$ $$

where $\text{dist}_{P}$ is the distance from $P$ to camera, and $H$ is the cross-entropy function. For the semantic loss, similarity on object points $P_{o}$ and background points $P_{b}$ can be calculated as

$$ $\text{Sim}_{s}^{P_{o}}=\tau e_{s}S(o_{P_{o}},r_{P_{o}}),\ \text{Sim}_{s}^{P_{b }}=\tau e_{s}S(r_{P_{b}}).$ $$

Similarly, semantic loss can be denoted as

$$ $\mathcal{L}_{s}=-\text{conf}(H(\text{Sim}_{s})+H({\text{Sim}_{s}}^{T})),$ $$

where $conf$ is the prediction confidence from the detection model.
The total loss is computed by:

$$ $\mathcal{L}=\mathcal{L}_{v}+\mathcal{L}_{s}.$ $$

In our experiments, an NVIDIA RTX3090 GPU is utilized and the batch size is set to $12544$ to maximize the capability of our VRAM. The MHE has $18$ levels of grids and the dimension of each grid is $8$. We train the neural implicit network for $100$ epochs with a decayed learning rate of $1e-4$. Each epoch contains $3e6$ samples.

## 4 Experimental Results

Figure: Figure 4: The object localization results among state-of-the-art methods and our method with text input in the form of object in the region. Red stars show the position of the found results of input texts.
Refer to caption: extracted/5659575/images/exp3.png

Figure: Figure 5: The image localization heatmaps among state-of-the-art methods and our method with text input in the form of object in the region. Red lines and the sector represent the field of view region.
Refer to caption: extracted/5659575/images/exp4.png

To validate the established layout-object-position relationship of LOP-Field, we designed the following experiments related to region information in scenes. Our experimental data consists of multi-room environment from Matterport3D as well as apartment environment, which allows us to demonstrate that our approach can be generalized in diverse scenarios. The data environment is of single-floor residential buildings which is the common working scenario of household robots widely studied in this field.

### 4.1 Region Inference

**Table 1: Region prediction results on the test set of different scenes from the Matterport3D dataset. Accuracy, precision, and F1 score are used as metrics.**
| Regions | Scene1 | Scene2 | Scene3 | Scene4 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc. | Pre. | F1 | Acc. | Pre. | F1 | Acc. | Pre. | F1 | Acc. | Pre. | F1 |  |
| Living Room | 0.948 | 0.970 | 0.959 | 0.870 | 0.881 | 0.875 | 0.778 | 0.810 | 0.793 | 0.902 | 0.949 | 0.925 |
| Bedroom | 0.943 | 0.825 | 0.880 | 0.925 | 0.923 | 0.924 | 0.687 | 0.767 | 0.725 | 0.920 | 0.870 | 0.894 |
| Bathroom | 0.466 | 0.680 | 0.554 | 0.903 | 0.898 | 0.901 | 0.875 | 0.463 | 0.605 | 0.797 | 0.831 | 0.814 |
| Dining Room | - | - | - | 0.961 | 0.794 | 0.870 | 0.774 | 0.732 | 0.752 | 0.933 | 0.887 | 0.910 |
| Lobby | 0.681 | 0.941 | 0.790 | 0.853 | 0.951 | 0.899 | 0.978 | 0.510 | 0.671 | 0.855 | 0.698 | 0.769 |
| Family Room | - | - | - | - | - | - | 0.903 | 0.571 | 0.700 | 0.926 | 0.936 | 0.931 |
| Kitchen | 0.994 | 0.654 | 0.789 | 0.788 | 0.836 | 0.811 | 0.833 | 0.833 | 0.833 | 0.758 | 0.854 | 0.803 |
| Office | - | - | - | 0.969 | 0.848 | 0.905 | - | - | - | 0.953 | 0.883 | 0.917 |
| Toilet | - | - | - | - | - | - | 0.900 | 0.711 | 0.795 | - | - | - |
| Avg. Acc./Samples | 0.886 / 169k | 0.900 / 185k | 0.884 / 111k | 0.894 / 112k |  |  |  |  |  |  |  |  |

To demonstrate the built LOP association integrates positions with layout, we designed experiments that accept 3D positions as input to infer
the region information. For quantitative evaluation, we divided the RGB-D sequences of data into training and testing sets. The LOP-Field is trained according to Section 3.3 on the training set and tested with data from the test set. As the region inference task can be treated as a multi-class classification task for each input, the accuracy, precision, and F1-score are used as metrics. Tab. 1 shows the region inference results. It can be seen that in multi-region environments with different scales and layouts, the average accuracy exceeds 88%.
This experiment demonstrates that the implicit representation of the scene can successfully establish the connection between 3D points and their corresponding region features.

### 4.2 LOP Guided Localization

Text Input Object Localization: For objects of the same category existing in multiple regions, we input the textual description of the target object in the form of "object in the region" and infer the specific location of the target, comparing the results with the predictions of current state-of-the-art visual-language algorithms.
Fig. 4 demonstrates the advancements of LOP-Field in object localization tasks involving region information, which allows for the localization of specific target objects based on the description and features of the region, while other methods confuse objects from different regions. We tested over 160 text queries on 4 scenes of Apartment and Matterport3D dataset. The accuracy of LOP-Field to localize the specific objects in the target regions exceeds 90%, while other methods have a significant fluctuation in accuracy. More results can be seen in the appendix.

Image Input View Localization: To validate the help of region information in the image view localization task. We localize the images from the test set in the trained LOP-Field. The localization results are shown in Fig. 5 in the form of heatmaps. VLMaps* is a self-implemented version, because origin VLMaps does not implement the image localization task. To align with CLIP-Field and our work, the LSeg used in VLMap is replaced by CLIP. The results show that LOP-Field constrains the localization results to a smaller range in the exact region. We sampled more than 40 images on each of the 4 scenes from Apartment and Matterport3D dataset. By drawing the predicted camera view on the top-down view, we estimated the localization precision and found that almost all views can be ranged into a specific view on the target field of view, while other methods struggle to get precise results.

## 5 Conclusion and Limitations

Inspired by neural representations of spatial layout, scene contents, and locations, this paper proposed the LOP field, an implicit scene representation field that associates layout-object-position information, powered by foundation models for robotic scene understanding. Our experiments show that with the help of LOP association, region inference ability and better results in several downstream tasks are achieved.
However, due to time and page constraints, this study explored only a very limited application of scalable associative information. In addition, the accuracy of the model decreases when distinguishing between regions with similar functions (such as living room, family room, and TV room). Furthermore, we currently lack a good method to model the confidence of the predicted results when presented with images that do not contain representative objects (with sufficient information to infer the region) or with degraded visual features.

We will investigate how the layout-object-position association information can be effectively used to perform challenging tasks that previously relied solely on the semantic information of the objects.
Examples of such tasks include complex environment relocalization problems, reasoning about logical relationships between regions and objects, and efficient navigation between different regions. We believe that the integration of layout-object-position can significantly enhance a robot’s spatial perception capabilities and encourage researchers to pay attention to the encoding of scene-specific information.

## Appendix A Appendix / supplemental material

### A.1 Scene Partation Example

The scene can be partitioned into different regions using walls as dividers and lines can be aligned to these walls. This is similar in most scenarios, making the annotation of scene regions a straightforward task as shown in Fig. A1.

Figure: Figure A1: Using walls as dividers to associate lines with them, the scene can be divided into various regions and 3D points can be labeled with related regions easily.
Refer to caption: extracted/5659575/images/region_divide.png

### A.2 Vision-language Embeddings Similarity of Region and Objects

To demonstrate that the relationship of the vision-language and semantic embeddings for different regions is related to our intuition, we compare the similarity in region-region and object-region form and show the results in Fig. A2. It can be seen that based on general knowledge, cognitively related regions(e.g., the dining room and kitchen) and object-region pairs(e.g., sink and kitchen) are also more correlated in the vision-language and semantic feature spaces.

Figure: Figure A2: The similarity of a set of region embeddings(as shown in a) and object-region embeddings(as shown in b). The left graph shows the vision-language embedding similarity and the right one shows the semantic embedding similarity.
Refer to caption: extracted/5659575/images/embed.png

### A.3 Ablation Study

To explicitly encode the region information, we apply the LVM to process the background pixels out of the object bounding box and LLM to encode the region label text. What’s more, for object pixels, object label text is combined with the region text in the form of ’object in the region’ before being encoded by LLM.

Source of Region Information. In our very initial version, we assume that objects with region text include enough information to encode region layouts rather than encoding the background appearance. The region embeddings completely come from the region text label, and object embeddings are learned separately. Fig. A3 shows the difference in embedding processing between the initial version and the current method. Ablation results in Fig. A4 show that context and layout information in background pixels is necessary for layout-object-position association.

Figure: Figure A3: The different source of region information. The initial version which encodes regions from text description is shown in (a), and the current method which encodes background context is shown in (b).
Refer to caption: extracted/5659575/images/diff_method_1.jpg

Figure: Figure A4: Ablation results on the accuracy of region prediction on Matterport3D with 3D positions input. The w/o BG stands for not encoding background pixels to get region embeddings, and v-s weight ablates the weight of vision-language and semantic embeddings in the embeddings similarity contribution. Error bars show the results among samples from different scenes in Matterport3D.
Refer to caption: extracted/5659575/images/ablation.png

Vision-language and Semantic Embeddings Weight. To ablate the contribution of vision-language embeddings from CLIP and semantic embeddings from Sentence-BERT in encoding region features, we compare different weight settings between the v-s embeddings when inferring the regions with 3D position inputs. Results are shown in Fig. A4. It can be seen that both vision-language embeddings and semantic embeddings are indispensable, and weight settings with the greatest results are used for LOP-Field.

### A.4 Additional Experiment Results

Additional experiments results of object localization using text query inputs and view localization using image query inputs.

Figure: Figure A5: Text query localization on scene 2t7WUuJeko7.
Refer to caption: extracted/5659575/images/scene_1.png

Figure: Figure A6: Text query localization on scene 17DRP5sb8fy.
Refer to caption: extracted/5659575/images/scene_2.png

Figure: Figure A7: Text query localization on scene Apartment.
Refer to caption: extracted/5659575/images/scene_3.png

Figure: Figure A8: Text query localization on scene HxpKQynjfin.
Refer to caption: extracted/5659575/images/scene_4.png

Figure: Figure A9: Image query localization on scene 2t7WUuJeko7.
Refer to caption: extracted/5659575/images/a_scene1.png

Figure: Figure A10: Image query localization on scene 17DRP5sb8fy.
Refer to caption: extracted/5659575/images/a_scene2.png

Figure: Figure A11: Image query localization on scene HxpKQynjfin.
Refer to caption: extracted/5659575/images/a_scene4.png

## NeurIPS Paper Checklist

- 1.
Claims
- Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
- Answer: [Yes]
- Justification: This paper claims that with the help of region layout information, robotics can reach better spatial cognition inspired by neuroscience. The claim is explained in detail in the abstract and introduction, and the main contributions are listed in the introduction in Sec. 1.
- Guidelines:
•
The answer NA means that the abstract and introduction do not include the claims made in the paper.
•
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
•
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
•
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
- 2.
Limitations
- Question: Does the paper discuss the limitations of the work performed by the authors?
- Answer: [Yes]
- Justification: The limitations of this work lie in the incomplete usage of region layout and the lack of confidence evaluation for challenging inputs, which is declared in Sec. 5.
- Guidelines:
•
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
•
The authors are encouraged to create a separate "Limitations" section in their paper.
•
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
•
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
•
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
•
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
•
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
•
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
- 3.
Theory Assumptions and Proofs
- Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
- Answer: [N/A]
- Justification: This paper does not include theoretical results. We use experiments to prove that considering region layout for robotics leads to better results, and this idea comes from neuroscience.
- Guidelines:
•
The answer NA means that the paper does not include theoretical results.
•
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
•
All assumptions should be clearly stated or referenced in the statement of any theorems.
•
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
•
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
•
Theorems and Lemmas that the proof relies upon should be properly referenced.
- 4.
Experimental Result Reproducibility
- Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
- Answer: [Yes]
- Justification: The data processing pipeline and method architecture of the proposed LOP-Field which builds layout-object-position association is shown in Fig. 3.2. The details of problem formulation, target feature processing, model architecture, and training strategy are introduced in Sec 3.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
•
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
•
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
•
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a)
If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b)
If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
- 5.
Open access to data and code
- Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
- Answer: [Yes]
- Justification: We use open-sourced dataset in experiments which is declared in Sec. 4. Codes with model architecture, training, and testing scripts are provided.
- Guidelines:
•
The answer NA means that paper does not include experiments requiring code.
•
Please see the NeurIPS code and data submission guidelines ([https://nips.cc/public/guides/CodeSubmissionPolicy](https://nips.cc/public/guides/CodeSubmissionPolicy)) for more details.
•
While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
•
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ([https://nips.cc/public/guides/CodeSubmissionPolicy](https://nips.cc/public/guides/CodeSubmissionPolicy)) for more details.
•
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
•
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
•
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
•
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
- 6.
Experimental Setting/Details
- Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
- Answer: [Yes]
- Justification: The training details and model setups are declared in Sec. 3 and also in codes.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
•
The full details can be provided either with the code, in appendix, or as supplemental material.
- 7.
Experiment Statistical Significance
- Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
- Answer: [Yes]
- Justification: The error bars are available for experiments on the encoding of background appearance which we claim to include region information and on the usage of vision-language and semantic embeddings from large foundation models as shown in Fig. A4.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
•
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
•
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
•
The assumptions made should be given (e.g., Normally distributed errors).
•
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
•
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
•
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
•
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
- 8.
Experiments Compute Resources
- Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
- Answer: [Yes]
- Justification: The computation setup is declared in Sec. 3.3 and it is the same for training and testing.
- Guidelines:
•
The answer NA means that the paper does not include experiments.
•
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
•
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
•
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
- 9.
Code Of Ethics
- Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics [https://neurips.cc/public/EthicsGuidelines](https://neurips.cc/public/EthicsGuidelines)?
- Answer: [Yes]
- Justification: This research conform with the NeurIPS Code of Ethics.
- Guidelines:
•
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
•
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
•
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
- 10.
Broader Impacts
- Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
- Answer: [N/A]
- Justification: This research focuses on improving the spatial cognition ability of robotics, which is currently in the stage of pre-research of algorithms and has no social impact.
- Guidelines:
•
The answer NA means that there is no societal impact of the work performed.
•
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
•
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
•
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
•
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
•
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
- 11.
Safeguards
- Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
- Answer: [N/A]
- Justification: This research utilizes open-sourced datasets and the model does not have a risk for misuse.
- Guidelines:
•
The answer NA means that the paper poses no such risks.
•
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
•
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
•
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
- 12.
Licenses for existing assets
- Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
- Answer: [Yes]
- Justification: The using of existing datasets and foundation models is correctly cited. We obey the license and usage instructions.
- Guidelines:
•
The answer NA means that the paper does not use existing assets.
•
The authors should cite the original paper that produced the code package or dataset.
•
The authors should state which version of the asset is used and, if possible, include a URL.
•
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
•
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
•
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, [paperswithcode.com/datasets](paperswithcode.com/datasets) has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
•
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
•
If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
- 13.
New Assets
- Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
- Answer: [Yes]
- Justification: We provide instructions to utilize our codes and models for researching usage.
- Guidelines:
•
The answer NA means that the paper does not release new assets.
•
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
•
The paper should discuss whether and how consent was obtained from people whose asset is used.
•
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
- 14.
Crowdsourcing and Research with Human Subjects
- Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
- Answer: [N/A]
- Justification: This research does not include crowdsourcing experiments or human subjects.
- Guidelines:
•
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
•
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
•
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
- 15.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
- Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
- Answer: [N/A]
- Justification: This paper does not involve crowdsourcing nor research with human subjects. It is a robotic research on the existing open-sourced dataset.
- Guidelines:
•
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
•
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
•
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
•
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 1
questions: It would be very important if you can justify those points in the weakness part.
soundness: 2
strengths: 1. The authors tackle the problem of hierarchical robotic scene understanding, which is an interesting and important topic
2. The proposed LOP is bio-inspired, to me this concept seems interesting.
summary: The paper introduces Topo-Field, a framework designed to enhance mobile robot navigation by integrating detailed semantic information about layouts, objects, and their positions (LOP) into a neural field representation. Interestly, such structure is inspired by the role of postrhinal cortex neurons on spatial layout. By querying a learned NeRF, Topo-Field constructs a semantically rich yet computationally efficient topometric map for hierarchical robotic scene understanding. Experimental results demonstrate its effectiveness in tasks like position inference, localization, and planning, bridging the gap between detailed scene understanding and efficient robotic navigation.
weaknesses: **Unclear descriptions of target feature processing in Sec 4.1**
1. How do you know if a 3D point belongs to the object, or the background? Do you use the GT annotations from the dataset? (the Matterport3D you show has that information I believe?)
2. For the background features, you will get only a single feature for each image. How do you fuse those features from different views? 
3. Also, isn’t it making more sense to take per-pixel CLIP features from models like LSeg/OpenSeg, and fuse that information?

**Unclear descriptions of neural scene encoding in Sec 4.2**
1. Related to the questions above. In this section you mention that there are object-level local features and layout-level region features and MHE seems to be a good representation for learning such hierarchy. However, how exactly do you learn these two set of features respectively under MHE? No details there
2. To learn MHE or NeRF in general, you need to actually shoot a ray for each pixel and sample along the ray. The final features are the weighted sum of all values along the ray, with volume rendering. How do you make sure your features on the 3D surface point are exactly the feature you render? 

**Unclear Topometric Mapping in Sec 4.3**
1. Line 309, what is ${C_t, S_t}$? What are the differences to ${C_R, S_R}$ (I know this is the embeddings for region) in Line 304, and ${C_I, S_I}$ in Line 314? You did not specify them before. It is confusing and making it hard to understand
2. Figure 3 (b) does not really match with what you write in “localization with text/image query” between Line 306-318. In the figure, all you get are the per-point features, and try to match with query features, omitting many important details in your description. 
3. “Matching” in Figure 3 is never really discussed. What kind of matching? Do you mean calculating the cosine similarity among the features, and take the one with the highest score?

**Text query localization in experiments**
1. How do you decide the similarity threshold for the bounding box? Do you need to choose a different threshold for each text query? My own experience is that, it is not really possible to get a single threshold for every query.
2. One more thing: once you have the right threshold, how exactly do you get the bounding boxes out from thresholding? 
3. What are the “samples” in Table 1?
4. How many queries are you considering for each scene, and how do you obtain the GT? Same question applies to Table 3 as well.

**Image query localization in experiments**  
if I understand correctly, you show the heatmap of the query. You claim that “Topo-Field constrains the localization results to a smaller range in the exact region”. However, that is not really true to me. If you look at the washbasin in the bathroom, you also have many points highlighted in other regions, like kitchen, and even some points in the bedroom. In such a case, how can you get such good numbers in Table 3? 

**Ablation study**
How come your ablation in Table 4 is only evaluating the region prediction accuracy, which does not even require most parts of your methods (objects, the graph you build, etc). Why not evaluate on other things as well? And even that, your default strategy seems not outperform much over any of baselines, even the very simple baseline 1 in some scenes. 

**Writing**  
- Overall I think the writing is not good since many things are not justified well. 
- There are so many cases when the author writs (), a space is not added before, e.g. L214 …mapping(SLAM), L259 Multi-layer Perceptron(MLP), etc.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - In line 235: what is C and S? I assume it is the output of CLIP and Sentence-BERT? How are the regions r_p defined? 
- In line 239: what is m in this equation? 
- In page 5, line 241: It would seem that the partitioning of the space requires human labeling? If that is the case, it is a significant limitation of the approach. 
- In line 242: Could you clarify the sentence "the predicted implicit representation outputs are targeted to match the features from the pre-trained models separately", what it means in practice or how this is achieved. I assume this is what is described in 4.2, but it would be good to make it unambiguous if that is the case, as F is not referenced in that section. 
- Could you make the description in Section 4.2 more specific and formal? The only description of inputs/outputs and process we are provided is via the diagram in figure 1, it would be good to have a proper formal description of the process, description of the architecture, and format/dimensionality of inputs and outputs for each component, as well as a formal algorithm
- In line 254: Could you provide a more in-depth argument for using MHE? The computational cost of standard NeRFs is well known, but is MHE the only possible solution? How does it compare with other fast approaches discussed in the literature, like, for example, Gaussian Splatting? 
- In line 258: Could you describe the mapping in more formal terms? Fig. 2 only provides a schematic description of the process. 
- In line 268: How is the similarity between E_pi and {C_R, S_R} calculated? It would be good to have a formal equation for this operation.
soundness: 2
strengths: - The combination of structural and semantic information in a way that is efficient for robotic system to query and plan on is a critical problem for robotics. 
- The approach seems to perform very well on the evaluation, clearly outperforming the presented baselines on those tasks. 
- The proposed approach is also reasonable in computational terms, as all experiments were performed on a single GPU (no information is given on training time though).
summary: This article proposes a novel approach for encoding scene information into a topometric map, for improving localisation and planning. The proposed approach is based on a Layout-Object-Position (LOP) approach. Layout information from knowledge of the environment's rooms. Object information from semantic segmentation (Detic) and a joint encoding of the segmented object patch using clip and of the object-region labels using Sentence-BERT. Finally, position information is produced by a 3D reconstruction of the scene using Multi-scale Hashing Encoding (MHE). This information is combined into a single Topometric map coined Topo-Field. 
The proposed method is evaluated for the inference of position attributes and localisation and appears to clearly outperform the presented baselines.
weaknesses: - The description of the approach is lacking specifics, and the reader has to infer the architecture and information flow from the provided diagrams rather than formal description in mathematical and algorithmic terms. 
- It is not fully clear how the partitioning of the environment (location) into rooms is performed and how well it would generalise to new environments. 
- The motivation of the work from neuroscience is interesting, but remains very vague. Little discussion is provided on how well the proposed approach may model the neural structures it claims to be inspired by. 
- The performance is very good compared to the discussed baselines, but it would seem that the proposed appraoch also benefits from significantly more task-specific information for those tasks (ie, the room information is provided directly). This is not a critical issue in my view, but it would be good to discuss the limitations of the presented baselines and issue of fairness of comparison some more. 
- I note that the reference for Reimers & Gurewych should probably cite the published version of the article rather than the pre-print.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The experimental results are impressive when compared to baseline performances; however, it is unclear whether the benchmarks used are new proposed by the authors or following existing ones, which raises concerns about the fairness of the evaluation. What are the primary factors driving the significant improvements?

Computation: the paper mentions a large batch size of 12,544. It would be helpful to clarify what specific data is contained within this batch size.
soundness: 2
strengths: The paper addresses a compelling problem in semantic mapping and its applications for enabling robots to navigate real-world environments. The experimental results demonstrate impressive improvements in performance. Additionally, the supplementary materials, such as the code snippets and prompts, enhance the understanding of the proposed method's details and implementation.
summary: This paper presents a method for training a neural implicit field that utilizes supervisory signals from pre-trained foundation models to capture semantic features. The proposed model is applicable to several critical downstream tasks in robotics, including text/image query localization, semantic navigation, and path planning. Experimental results demonstrate significant improvements in performance metrics, supported by qualitative evidence.
weaknesses: Despite its potential, the system heavily relies on various input types, such as annotated room maps and camera poses, as well as off-the-shelf object detection methods for generating bounding boxes and masks. This dependence poses challenges in real-world applications, where inaccuracies in these inputs can lead to errors. Additionally, the system's reliance on ChatGPT complicates debugging and explanation when errors occur in complex real-world environments.

Encoding semantic information and supervising it with pre-trained features alleviates some annotation burdens; however, this approach is already a common practice in the field of implicit representation for semantic mapping [1][2]. The overall system resembles a large engineering project, making it challenging to distill its theoretical contributions.

[1] V. Tschernezki, I. Laina, D. Larlus and A. Vedaldi, "Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations," 2022 International Conference on 3D Vision (3DV), Prague, Czech Republic, 2022, pp. 443-453, doi: 10.1109/3DV57658.2022.00056.
[2] Zhu, Siting, et al. "Sni-slam: Semantic neural implicit slam." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Table 4 could benefit from clearer labeling, where Baselines 1-4 are not explicitly defined. A reference to Figure 7 could help.

The authors frequently reference the postrhinal cortex from the biological literature, but the connection to the proposed method is not clearly articulated. Topological mapping is indeed a common computer vision task relevant to navigation.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: With the issues addressed above, the authors should revise the paper accordingly.
soundness: 1
strengths: + The idea of constructing a topometric map using the implicit neural field is interesting
summary: This paper targets an interesting problem of topometric mapping but is not ready for publishing. The quality is poor regarding writing, organization, annotations, and experimental setups.
weaknesses: The writing is far from satisfactory. The corresponding authors should revise the manuscripts besides the abstract and introduction.
+ Though the paper proposes a Topo-field to integrate layout-object-position, this representation is not clearly presented in Sec. 3. The definition of the topometric map (or the graph structure in Eq. 3) is vague and hard to follow, and the generation of the graph from dense field F (L199) is unclear. Note that the implicit neural field F is similar to previous methods with a distilled feature field, the novelty and the contribution of the proposed method are unclear.
+ The hierarchical structure of point-object-room is common in scene graph generation. However, no relevant work (e.g., CLIO, HOV-SG) is referred to in the related work section or the experiments section.
+ Multiple annotations are not formally defined in the paper (e.g., the functions $C_t, S_t$). The training stage in Sec. 4.4 should be carefully revised to make it clear.
+ The experimental setups lack clear demonstration, and comparisons against recent methods are missing.