Topo-Field: Topometric mapping with Brain-inspired Hierarchical Layout-Object-Position Fields

Authors: Jiawei Hou, Wenhao Guan, Longfei Liang, Xiangyang Xue, Taiping Zeng
Primary area: applications to robotics, autonomy, planning
Keywords: Robotic scene understanding, Neural scene representation, Hierarchical representation, Topometric map

Abstract:
Mobile robots require comprehensive scene understanding to operate effectively in diverse environments, enriched with contextual information such as layouts, objects, and their relationships. While advancements like Neural Radiance Fields (NeRF) offer high-fidelity 3D reconstructions, they are computationally intensive and often lack efficient representations of traversable spaces essential for planning and navigation. In contrast, topological maps generated by LiDAR or visual SLAM methods are computationally efficient but lack the semantic richness necessary for a more complete understanding of the environment.
Inspired by neuroscientific studies on spatial cognition, particularly the role of postrhinal cortex (POR) neurons that are strongly tuned to spatial layouts over scene content, this work introduces Topo-Field, a framework that integrates Layout-Object-Position (LOP) associations into a neural field and constructs a topometric map from this learned representation. LOP associations are modeled by explicitly encoding object and layout information, while a Large Foundation Model (LFM) technique allows for efficient training without extensive annotations. The topometric map is then constructed by querying the learned NeRF, offering both semantic richness and computational efficiency.
Empirical evaluations in multi-room apartment environments demonstrate the effectiveness of Topo-Field in tasks such as position attribute inference, query localization, and topometric planning, successfully bridging the gap between high-fidelity scene understanding and efficient robotic navigation.

Decision: reject
Reviewer scores: 5, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730662448296

thread_type:
official_review

<thread_object>
creator: reviewer_gndj

<content>
summary:
This paper targets an interesting problem of topometric mapping but is not ready for publishing. The quality is poor regarding writing, organization, annotations, and experimental setups.

strengths:
+ The idea of constructing a topometric map using the implicit neural field is interesting

weaknesses:
The writing is far from satisfactory. The corresponding authors should revise the manuscripts besides the abstract and introduction.
+ Though the paper proposes a Topo-field to integrate layout-object-position, this representation is not clearly presented in Sec. 3. The definition of the topometric map (or the graph structure in Eq. 3) is vague and hard to follow, and the generation of the graph from dense field F (L199) is unclear. Note that the implicit neural field F is similar to previous methods with a distilled feature field, the novelty and the contribution of the proposed method are unclear.
+ The hierarchical structure of point-object-room is common in scene graph generation. However, no relevant work (e.g., CLIO, HOV-SG) is referred to in the related work section or the experiments section.
+ Multiple annotations are not formally defined in the paper (e.g., the functions $C_t, S_t$). The training stage in Sec. 4.4 should be carefully revised to make it clear.
+ The experimental setups lack clear demonstration, and comparisons against recent methods are missing.

questions:
With the issues addressed above, the authors should revise the paper accordingly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730669626009

thread_type:
official_review

<thread_object>
creator: reviewer_wzd7

<content>
summary:
This paper presents a method for training a neural implicit field that utilizes supervisory signals from pre-trained foundation models to capture semantic features. The proposed model is applicable to several critical downstream tasks in robotics, including text/image query localization, semantic navigation, and path planning. Experimental results demonstrate significant improvements in performance metrics, supported by qualitative evidence.

strengths:
The paper addresses a compelling problem in semantic mapping and its applications for enabling robots to navigate real-world environments. The experimental results demonstrate impressive improvements in performance. Additionally, the supplementary materials, such as the code snippets and prompts, enhance the understanding of the proposed method's details and implementation.

weaknesses:
Despite its potential, the system heavily relies on various input types, such as annotated room maps and camera poses, as well as off-the-shelf object detection methods for generating bounding boxes and masks. This dependence poses challenges in real-world applications, where inaccuracies in these inputs can lead to errors. Additionally, the system's reliance on ChatGPT complicates debugging and explanation when errors occur in complex real-world environments.

Encoding semantic information and supervising it with pre-trained features alleviates some annotation burdens; however, this approach is already a common practice in the field of implicit representation for semantic mapping [1][2]. The overall system resembles a large engineering project, making it challenging to distill its theoretical contributions.

[1] V. Tschernezki, I. Laina, D. Larlus and A. Vedaldi, "Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations," 2022 International Conference on 3D Vision (3DV), Prague, Czech Republic, 2022, pp. 443-453, doi: 10.1109/3DV57658.2022.00056.
[2] Zhu, Siting, et al. "Sni-slam: Semantic neural implicit slam." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Table 4 could benefit from clearer labeling, where Baselines 1-4 are not explicitly defined. A reference to Figure 7 could help.

The authors frequently reference the postrhinal cortex from the biological literature, but the connection to the proposed method is not clearly articulated. Topological mapping is indeed a common computer vision task relevant to navigation.

questions:
The experimental results are impressive when compared to baseline performances; however, it is unclear whether the benchmarks used are new proposed by the authors or following existing ones, which raises concerns about the fairness of the evaluation. What are the primary factors driving the significant improvements?

Computation: the paper mentions a large batch size of 12,544. It would be helpful to clarify what specific data is contained within this batch size.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730812592108

thread_type:
official_review

<thread_object>
creator: reviewer_djx7

<content>
summary:
This article proposes a novel approach for encoding scene information into a topometric map, for improving localisation and planning. The proposed approach is based on a Layout-Object-Position (LOP) approach. Layout information from knowledge of the environment's rooms. Object information from semantic segmentation (Detic) and a joint encoding of the segmented object patch using clip and of the object-region labels using Sentence-BERT. Finally, position information is produced by a 3D reconstruction of the scene using Multi-scale Hashing Encoding (MHE). This information is combined into a single Topometric map coined Topo-Field. 
The proposed method is evaluated for the inference of position attributes and localisation and appears to clearly outperform the presented baselines.

strengths:
- The combination of structural and semantic information in a way that is efficient for robotic system to query and plan on is a critical problem for robotics. 
- The approach seems to perform very well on the evaluation, clearly outperforming the presented baselines on those tasks. 
- The proposed approach is also reasonable in computational terms, as all experiments were performed on a single GPU (no information is given on training time though).

weaknesses:
- The description of the approach is lacking specifics, and the reader has to infer the architecture and information flow from the provided diagrams rather than formal description in mathematical and algorithmic terms. 
- It is not fully clear how the partitioning of the environment (location) into rooms is performed and how well it would generalise to new environments. 
- The motivation of the work from neuroscience is interesting, but remains very vague. Little discussion is provided on how well the proposed approach may model the neural structures it claims to be inspired by. 
- The performance is very good compared to the discussed baselines, but it would seem that the proposed appraoch also benefits from significantly more task-specific information for those tasks (ie, the room information is provided directly). This is not a critical issue in my view, but it would be good to discuss the limitations of the presented baselines and issue of fairness of comparison some more. 
- I note that the reference for Reimers & Gurewych should probably cite the published version of the article rather than the pre-print.

questions:
- In line 235: what is C and S? I assume it is the output of CLIP and Sentence-BERT? How are the regions r_p defined? 
- In line 239: what is m in this equation? 
- In page 5, line 241: It would seem that the partitioning of the space requires human labeling? If that is the case, it is a significant limitation of the approach. 
- In line 242: Could you clarify the sentence "the predicted implicit representation outputs are targeted to match the features from the pre-trained models separately", what it means in practice or how this is achieved. I assume this is what is described in 4.2, but it would be good to make it unambiguous if that is the case, as F is not referenced in that section. 
- Could you make the description in Section 4.2 more specific and formal? The only description of inputs/outputs and process we are provided is via the diagram in figure 1, it would be good to have a proper formal description of the process, description of the architecture, and format/dimensionality of inputs and outputs for each component, as well as a formal algorithm
- In line 254: Could you provide a more in-depth argument for using MHE? The computational cost of standard NeRFs is well known, but is MHE the only possible solution? How does it compare with other fast approaches discussed in the literature, like, for example, Gaussian Splatting? 
- In line 258: Could you describe the mapping in more formal terms? Fig. 2 only provides a schematic description of the process. 
- In line 268: How is the similarity between E_pi and {C_R, S_R} calculated? It would be good to have a formal equation for this operation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730935871747

thread_type:
official_review

<thread_object>
creator: reviewer_ms6u

<content>
summary:
The paper introduces Topo-Field, a framework designed to enhance mobile robot navigation by integrating detailed semantic information about layouts, objects, and their positions (LOP) into a neural field representation. Interestly, such structure is inspired by the role of postrhinal cortex neurons on spatial layout. By querying a learned NeRF, Topo-Field constructs a semantically rich yet computationally efficient topometric map for hierarchical robotic scene understanding. Experimental results demonstrate its effectiveness in tasks like position inference, localization, and planning, bridging the gap between detailed scene understanding and efficient robotic navigation.

strengths:
1. The authors tackle the problem of hierarchical robotic scene understanding, which is an interesting and important topic
2. The proposed LOP is bio-inspired, to me this concept seems interesting.

weaknesses:
**Unclear descriptions of target feature processing in Sec 4.1**
1. How do you know if a 3D point belongs to the object, or the background? Do you use the GT annotations from the dataset? (the Matterport3D you show has that information I believe?)
2. For the background features, you will get only a single feature for each image. How do you fuse those features from different views? 
3. Also, isn’t it making more sense to take per-pixel CLIP features from models like LSeg/OpenSeg, and fuse that information?

**Unclear descriptions of neural scene encoding in Sec 4.2**
1. Related to the questions above. In this section you mention that there are object-level local features and layout-level region features and MHE seems to be a good representation for learning such hierarchy. However, how exactly do you learn these two set of features respectively under MHE? No details there
2. To learn MHE or NeRF in general, you need to actually shoot a ray for each pixel and sample along the ray. The final features are the weighted sum of all values along the ray, with volume rendering. How do you make sure your features on the 3D surface point are exactly the feature you render? 

**Unclear Topometric Mapping in Sec 4.3**
1. Line 309, what is ${C_t, S_t}$? What are the differences to ${C_R, S_R}$ (I know this is the embeddings for region) in Line 304, and ${C_I, S_I}$ in Line 314? You did not specify them before. It is confusing and making it hard to understand
2. Figure 3 (b) does not really match with what you write in “localization with text/image query” between Line 306-318. In the figure, all you get are the per-point features, and try to match with query features, omitting many important details in your description. 
3. “Matching” in Figure 3 is never really discussed. What kind of matching? Do you mean calculating the cosine similarity among the features, and take the one with the highest score?

**Text query localization in experiments**
1. How do you decide the similarity threshold for the bounding box? Do you need to choose a different threshold for each text query? My own experience is that, it is not really possible to get a single threshold for every query.
2. One more thing: once you have the right threshold, how exactly do you get the bounding boxes out from thresholding? 
3. What are the “samples” in Table 1?
4. How many queries are you considering for each scene, and how do you obtain the GT? Same question applies to Table 3 as well.

**Image query localization in experiments**  
if I understand correctly, you show the heatmap of the query. You claim that “Topo-Field constrains the localization results to a smaller range in the exact region”. However, that is not really true to me. If you look at the washbasin in the bathroom, you also have many points highlighted in other regions, like kitchen, and even some points in the bedroom. In such a case, how can you get such good numbers in Table 3? 

**Ablation study**
How come your ablation in Table 4 is only evaluating the region prediction accuracy, which does not even require most parts of your methods (objects, the graph you build, etc). Why not evaluate on other things as well? And even that, your default strategy seems not outperform much over any of baselines, even the very simple baseline 1 in some scenes. 

**Writing**  
- Overall I think the writing is not good since many things are not justified well. 
- There are so many cases when the author writs (), a space is not added before, e.g. L214 …mapping(SLAM), L259 Multi-layer Perceptron(MLP), etc.

questions:
It would be very important if you can justify those points in the weakness part.
</content>
</thread_object>

</thread>


Metareview: