Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigation

Authors: zihan  wang
xiangyang  li
jiahao  yang
yeqi  liu
shuqiang  jiang
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-03 07:06:27

Abstract: Vision-and-language navigation (VLN) enables the agent to navigate to a remote location in 3D environments following the natural language instruction. In this field, the agent is usually trained and evaluated in the navigation simulators, lacking effective approaches for sim-to-real transfer. The VLN agents with only a monocular camera exhibit extremely limited performance, while the mainstream VLN models trained with panoramic observation, perform better but are difficult to deploy on most monocular robots. For this case, we propose a sim-to-real transfer approach to endow the monocular robots with panoramic traversability perception and panoramic semantic understanding, thus smoothly transferring the high-performance panoramic VLN models to the common monocular robots. In this work, the semantic traversable map is proposed to predict agent-centric navigable waypoints, and the novel view representations of these navigable waypoints are predicted through the 3D feature fields. These methods broaden the limited field of view of the monocular robots and significantly improve navigation performance in the real world. Our VLN system outperforms previous SOTA monocular VLN methods in R2R-CE and RxR-CE benchmarks within the simulation environments and is also validated in real-world environments, providing a practical and high-performance solution for real-world VLN.

Main paper: 
 introduction: 
 Vision-and-Language Navigation (VLN) tasks [1, 2, 3, 4] require an agent to understand natural language instructions and move to the destination. In the continuous environment setting (i.e., VLN-CE) [4] , the navigation agent is free to traverse any unobstructed location with low-level actions (e.g., turn left 15 degrees, turn right 15 degrees, or move forward 0.25 meters). These VLN agents are typically trained and evaluated within simulators [5, 6] , equipped with panoramic RGB-D cameras or just a forward-facing RGB-D camera.Upon receiving instructions, the VLN agent begins exploring, moving gradually to the destination to find the target, as shown in Figure 1 . However, agents equipped with monocular cameras [7, 8] have a limited field of view (yellow sector in Figure 1 ), restricting their environmental perception. This limitation often leads to navigation failures, such as missing the target, exemplified by the basketball in Figure 1 (b). To enhance performance on benchmarks, most VLN agents [9, 10, 11, 12] use panoramic cameras (blue circular area in Figure 1 ), achieving nearly a 20% increase in navigation success rate. However, this trick has significantly hindered the deployment of VLN models on physical robots, primarily because most robots don't come equipped with panoramic RGB-D cameras due to their high cost and large size. Our goal is to design a VLN system that enables robots with only a monocular camera to achieve near-panoramic perception, ensuring both practicality and high performance. In this way, two challenges must be addressed: 1) identifying traversable areas around the robot, and 2) representing semantic and spatial relationships within the panorama.Instruction: Go past the table with a laptop, find the basketball. How to develop a VLN system combining:Step 1Step 2Figure 1 : The VLN models [7, 8, 13] equipped with a monocular camera have limited navigation success rates of less than 39% on the R2R-CE Val Unseen split. Most VLN models [9, 10] are trained and evaluated in the simulator [6] with the panoramic observation, achieving navigation success rates of over 57%, but hard to deploy on real-world robots.For the first challenge, we propose a method to construct the semantic traversable map to identify traversable areas and navigable waypoints around the VLN robot, simplifying the action space and achieving obstacle avoidance in real-world navigation. Previous works [14, 9] use panoramic RGB-D images and multi-layer transformers to model spatial relationships and predict candidate waypoints. However, due to strict camera hardware requirements, these VLN models have not been successfully deployed and validated in real-world environments. In contrast, more practical VLN robots [15] use 2D laser scanners to create radial occupancy maps around the robot and employ a UNet [16] to predict navigable waypoints. This approach has two main drawbacks: 1) The 2D laser scanner can only detect obstacles on a fixed-height plane, which significantly reduces real-world VLN performance [15] .2) The 2D laser scanner cannot identify obstacles with traversability such as stairs and doors. Using depth cameras for occupancy mapping, our solution can detect obstacles at varying heights. Furthermore, leveraging environmental layout knowledge acquired through pre-training, our semantic map predictor can forecast the obstacle layout beyond the current field of view. Additionally, the semantic map in our approach can also significantly identify traversable landmarks (e.g., stairs and doors). All these advantages of the semantic traversable map enhance the navigation abilities of real-world robots.For the second challenge, the 3D Feature Fields model [12] is utilized to represent the visual semantics of each waypoint to select the optimal one to move. During navigation, the 3D Feature Fields model [12] encodes 2D visual observations and projects them into 3D feature space via the depth map. Using volume rendering [17] , the model can decode semantic representation of novel views from the feature fields and align them with the CLIP embeddings [18] . The 3D Feature Fields can generalize to unseen scenes, allowing real-time construction and dynamic updates. In our work, the feature fields constructed from historical observations can predict the novel view representations within the robot's panorama, significantly broadening the limited field of view of the monocular robot (69°HFOV and 42°VFOV), and improving navigation performance in the real world.In this work, our main contributions include:• We design a novel approach using a monocular RGB-D camera to construct the Semantic Traversable Map for predicting navigable waypoints and introduce the 3D Feature Fields for predicting novel view representations, endowing monocular VLN robots with panoramic perception capabilities.• Our work provides a practical and high-performance solution for VLN sim-to-real transfer.Extensive experiments in the simulator and the real-world environments demonstrate the effectiveness of our approach. 
related works: 
 Vision-and-Language Navigation. Vision and Language Navigation (VLN) [1, 2, 4, 19, 20, 21, 10, 22, 23] has gained significant attention in recent years. Unlike object goal navigation [24, 25, 26, 27, 28, 29] , VLN agents must understand more complex natural language instructions and navigate to the described destination. Early VLN research focused on discrete environments due to the computational demands of exploring large action spaces in continuous environments. In discrete environments like the Matterport3D simulator [5] , a predefined navigation connectivity graph is used. Agents observe panoramic RGB and depth images and can teleport to nearby nodes in the graph, navigating from a starting node to a target node following natural language instructions. Despite efficient training and high navigation performance, these settings are impractical for real-world applications. To address this, the continuous environment setting (VLN-CE [4] ) was introduced. In VLN-CE, agents freely navigate any unobstructed location in 3D environments using low-level actions (e.g., turn left 15 degrees, turn right 15 degrees, or move forward 0.25 meters) in the Habitat simulator [6] . To overcome inefficiencies and poor performance in atomic action prediction, many works have designed subgoal or waypoint predictors to generate navigable waypoints, making VLN-CE more similar to discrete environments. By predicting several candidate waypoints, the agent selects the optimal one and executes atomic actions to move. This strategy significantly improves VLN-CE performance. However, these methods often rely on panoramic RGB-D images [15, 30, 14] , limiting their applicability to real-world robots. In contrast, our work utilizes only a monocular RGB-D camera to construct a dynamic global semantic and occupancy map and then predicts the candidate waypoints, with better practicality.Sim-to-Real Transfer for VLN. Anderson [15] makes the first attempt at VLN sim-to-real transfer with panoramic cameras, which propose a subgoal model based on a 2D laser scanner to identify nearby waypoints. He finds that sim-to-real transfer to an unseen environment is challenging with no prior navigation graph. The core issue is that the navigable candidate waypoints predicted by the subgoal model exhibit significant deviations compared to the pre-defined navigation graph. More recently, to leverage the powerful reasoning and generalization capabilities of large language models or multimodal models to assist in real-world VLN, VLMaps [31] and DiscussNav [32] achieve VLN by combining multiple large models or foundation models. However, the navigation performance of these zero-shot methods is very limited. Therefore, NaVid [7] attempts to fine-tune the large model [33] using a large amount of video-based VLN data to achieve superior performance for real-world VLN with only a monocular camera. The large models for step-by-step atomic action prediction incur significant computational costs and response delays. Our work aims to design a universal VLN framework for sim-to-real transfer with practicality, high performance, and low cost.Generalizable 3D Feature Fields. The neural radiance field (NeRF) [17, 34] has gained significant popularity in various AI tasks, which predicts an image from an arbitrary viewpoint in a scene. However, the traditional NeRF methods with implicit MLP networks can only synthesize novel view images in seen scenes, which makes it difficult to generalize to unseen scenes and adapt to many embodied AI tasks. To avoid the difficulties of pixel-wise RGB reconstruction in unseen environments, GeFF [35] and HNR [12] attempt to encode 2D visual observations into 3D representations (called 3D feature fields) via the depth map. Using volume rendering [17] , these models decode novel view representations from the 3D feature field and align them with open-world features (e.g., CLIP embeddings [18] ). The 3D feature fields can generalize to unseen scenes, enabling real-time construction and dynamic updates, which assist with various embodied tasks. In this work, we utilize the HNR model [12] to provide panoramic perception capabilities for monocular robots.3 Methods 
overview: 
 The baseline framework. As shown in Figure 2 , for most VLN-CE works [9, 10, 11] , at each time step t, the agent observes panoramic RGB images R t = {r t,i } 12 surrounding its current location (i.e., 12 view images with 30 degrees separation). Subsequently, a pre-trained waypoint predictor [14] takes the panoramic observation R t and D t as input and predicts the navigable waypoints around the agent. The VLN model typically encodes the visual observation of these waypoints with their location information (relative direction and distance) and then constructs a topological map. The Cross-Modal Graph Transformer [9, 20] is used to encode the topological map with the instruction and selects the optimal waypoint as the next goal to move. However, the pipeline above is challenging to implement for monocular robots due to its severely restricted field of view.The sim-to-real transfer framework. As shown in Figure 3 and Figure 4 , we design a new waypoint predictor based on the semantic traversable map, capable of predicting waypoints with only a monocular RGB-D camera, achieving nearly 360°coverage. Meanwhile, the 3D Feature Fields model [12] is used to predict visual representations of these waypoints even outside the field of view. Using Semantic Traversable Map and 3D Feature Fields, we successfully transfer several VLN-CE models with panoramic observation (ETPNav [9] and GridMM [11] ) to real-world monocular robots, achieving superior performance. Ground-projecting the global map. As shown in Figure 4 , we design a waypoint predictor that generates the egocentric traversable map based on the global semantic map and occupancy map updated by monocular RGB-D observations. During navigation, following [36, 13] , the model groundprojects depth observations to a global occupancy map O t ∈ R h×w×3 with classes void, f ree, and occupancy, where h = w = 512 with each pixel corresponding to 5cm × 5cm region. Similarly, the semantic segmentation is extracted by a pre-trained UNet model [13] and ground-projected to a global semantic map S t ∈ R h×w×27 with 27 classes. These global maps are accumulated and updated continuously during navigation. (1) The ground-truth semantic maps and occupancy maps are obtained following CM 2 [13] from the 3D semantic information in Matterport3D dataset [5] .Predicting the traversable map. Finally, the refined semantic map S ′′ t , occupancy map O ′′ t and the learnable positional embedding are concatenated as the input ∈ R 192×192×(27+3+16) for another UNet model, which outputs a probability distribution map of the traversable areas T t ∈ R 192×192 . The ground-truth traversable map is derived from the annotations of CWP [14] . CWP takes nodes from the navigable connectivity graph of the discrete R2R dataset [1] as the ground-truth waypoints. All these waypoints near the agent are used to construct 2D Gaussian distribution maps {G k ∈ R 192×192 } K k=1 centered at these waypoints with σ = 5. The ground-truth traversable map is obtained by combining these 2D Gaussian distributions into a single 2D mixture Gaussian distribution:EQUATIONThe mean squared error (i.e., MSE loss) is used to optimize the prediction of the traversable map:EQUATIONPredicting the candidate waypoints. To obtain navigable waypoints from the traversable map, the map is segmented into 12 sections, each spanning 30 degrees from the center. The waypoint with the highest value in each section is selected, but waypoints in occupied regions of the occupancy map O ′′ t are excluded. Ultimately, the Top-K waypoints with the highest probability values are chosen as the candidate waypoints for navigation.After identifying candidate waypoints, a hierarchical neural radiance representation (HNR) model [12] is utilized to encode 3D feature fields and decode novel view representations within the panorama. As shown in Figure 3 , during navigation, the monocular RGB images are processed using a pre-trained CLIP-ViT-B/16 [18] model to extract fine-grained visual features and then mapped to 3D world positions using depth map and camera parameters.Volume rendering. For decoding novel view representations, the HNR model generates a feature map R ∈ R 8×8×512 (which contains 8 × 8 subregions) for novel view by predicting subregion features through volume rendering in 3D feature fields. Specifically, the HNR model uniformly samples N points along the ray from the camera position to the predicted subregion's center, search for k-nearest features of these points, and predicts the volume density σ n and latent vector r n using an MLP network for each sampled point. These latent vectors are composited into a subregion feature R (u,v) through volume rendering, as shown in Figure 3 and follows:EQUATIONτ n represents volume transmittance, and ∆ n is the distance between adjacent sampled points. R (u,v) represent the region feature at the u-th row and v-th column of the predicted feature map R.Novel view representation. Finally, a transformer-based decoder is used to aggregate the feature map R ∈ R 8×8×512 and generate the novel view representation V ∈ R 1×512 , which is aligned with CLIP embedding of the novel view. At each step of navigation, via 3D feature fields and the HNR model, the VLN agent can perceive 12 views within the panorama. The forward-facing view feature is extracted from the monocular RGB image using the CLIP model, and the other 11 novel view features are predicted from the 3D feature fields. All these representations are input into the baseline VLN models in Section 3.1 for navigation. 
experiment setup: 
 Simulated environments. We evaluate our approach on the R2R-CE [4] and RxR-CE [2] datasets in simulated environments [6] . R2R-CE [4] is collected based on the Matterport3D scenes [5] with the Habitat simulator [6] . The R2R-CE dataset includes 5,611 trajectories divided into the train, validation seen, validation unseen, and test unseen splits. Each trajectory has three English instructions, with an average path length of 9.89 meters and an average instruction length of 32 words. The camera's HFOV for R2R-CE is 90 Real-world environments. We conduct all experiments using an Interbotix LoCoBot WX250 equipped with an Intel RealSense D435 camera to capture both depth and RGB images. The experiments are carried out using ROS Noetic in an unstructured lab environment unseen to all VLN models, these VLN models are run on a laptop with the GTX 1060 GPU. The lab environment is approximately 100 m 2 , featuring movable walls and various pieces of furniture. We build five different house layouts, and each layout contains four types of rooms: living room, kitchen, bathroom, and bedroom. We annotate 20 data samples (trajectories and instructions) for each layout, resulting in 100 samples for real-world VLN. Details of these examples can be found in the appendix.Metrics. There are several standard metrics [1] for evaluating the agent's performance in VLN-CE, including Navigation Error (NE), Success Rate (SR), SR given the Oracle stop policy (OSR), Success Rate weighted by normalized inverse Path Length (SPL).Training Details. The predictor of the semantic traversable map is trained for 20K episodes on two RTX 3090 GPUs. Each episode consisted of a 15-step trajectory randomly sampled in the Habitat simulator. This model is optimized by training losses in Section 3.2. Our approach uses the pretrained 3D Feature Fields model [12] to predict novel view representations within the panorama, without further training. The VLN models are trained using 4 RTX 3090 GPUs for 10K episodes on the R2R-CE dataset and 20K episodes on the RxR-CE dataset. All parameters are initialized from the trained baselines (i.e., ETPNav [9] and GridMM [11] ).Table 1 and Table 2 represent the performance of our proposed approach compared with existing VLN models on the R2R-CE and RxR-CE datasets respectively. Overall, compared with other monocular VLN methods, our approach "ETPNav w/ 3D Feature Fields" achieves state-of-the-art results in most metrics, demonstrating the effectiveness of the proposed approach. As illustrated in Table 1 , for both the val unseen split and test unseen split of the R2R-CE dataset, our model outperforms previous works (e.g., WS-MGMap [8] and NaVid [7] ) over 7% on navigation success rate (SR). It can be noted that our method does not show a significant advantage in the SPL metric. This is because the better SPL metric requires a shorter navigation path as well as a higher navigation success rate. However, constructing 3D Feature Fields requires the agent to conduct more exploration and observation, resulting in extra movements. Nevertheless, the excellent navigation success rate (SR) is still sufficient to demonstrate the superiority of our approach. We conduct real-world experiments on four different types of scenes: living room, bedroom, kitchen, and bathroom. As shown in Table 3 , the methods using 3D Feature Fields demonstrate more significant advantages in real-world environments than in simulators, outperforming previous methods. The Intel RealSense D435 camera equipped on our robot has an HFOV of 69°and a VFOV of 42°, which is much smaller than the 90°HFOV and VFOV used in the simulator. The minimal field of view severely restricts the performance of VLN models in real-world environments. Fortunately, the VLN methods using 3D Feature Fields overcome this difficulty well with their stronger panoramic perception capabilities. 4 , we analyzed the crucial role of 3D feature fields in our method. "Panorama w/o Depth feature" is our baseline method ETPNav [9] without the depth features, using only the CLIP features extracted from the panoramic RGB images. In the "w/o Feature Fields" setting, only the CLIP features from the forward-facing view are retained, with all other directions' features set to zero. This leads to a performance drop of over 20% on OSR, SR, and SPL metrics compared to the baseline, which shows that directly transferring the panoramic VLN model to a monocular camera setting is completely unfeasible. By constructing the feature fields (i.e., "w/ Feature Fields"), the predicted novel view representations around the agent enhance the perception capability, leading to a 12% improvement in SR metrics compared to "w/o Feature Fields". Furthermore, to validate the capability of 3D Feature Fields to memorize and adapt to new environments, we draw on lifelong strategies from previous navigation works [38, 39] . Specifically, we use a group of 10 navigation examples within the same scene as a whole episode. During navigation in an episode, the feature fields constructed in the previous examples are preserved (but the traversable map is not preserved, as a 2D map cannot handle cross-floor scenarios across different examples), in this way, the previously constructed feature fields can support subsequent navigation and gain performance improvements on "w/ Feature Fields + Memory" setting.Traversable Map. As shown in Table 5 , we analyze the critical roles of various components in the Semantic Traversable Map. The performance of both "w/o Semantic map" and "w/o Occupancy map" is lower than that of the "Full pipeline", indicating that the semantic map and occupancy map are both crucial for predicting traversable areas. Especially when the semantic map is not available, the lack of identifiable landmarks with traversability (e.g., doors and stairs) significantly compromises navigation performance. Additionally, the importance of learnable positional embeddings is also confirmed for predicting traversable maps using UNet. Removing these positional embeddings (i.e., "w/o Positional embeddings") significantly reduces the UNet's spatial understanding from an agent-centric view, thereby impairing navigation performance.  
conclusion: 
 In this work, based on the proposed semantic traversable map and 3D feature fields, we enable VLN robots with monocular cameras to have panoramic perception capabilities, which help various VLN models achieve high-performance sim-to-real transfer. We verify the significant role of panoramic perception for vision-and-language navigation, as well as the feasibility of traversable map and 3D feature fields to enhance perceptual capabilities, which may help a broader range of embodied tasks.Limitations. Since both the semantic traversable map and the 3D feature fields rely on information provided by past observations, the agent tends to engage in more exploration to gather sufficient environmental observations before making navigation decisions, which results in longer path length and a lower SPL metric. In the future, we are looking to improve the prediction ability of the traversable map and 3D feature fields for unobserved areas, thereby reducing extensive exploration. 
 
 Appendix: A Presentation of the Real-world VLN Instruction: Go past the green plant into the bathroom, and stop in front of the toilet.Instruction: Pass by the table and stop in front of the green plant next to the basketball.Step 1Step 2Step 3Step 4Step 5Step 6Step 7Step 8Step 1Step 2Step 3Step 5Step 6Step 7Step 8Step 4 Figure 7 and Figure 8 illustrate the construction and updating of the semantic map and traversable map during navigation. Both maps are agent-centered to better predict candidate waypoints. In the traversable map, warmer color indicates higher traversability probabilities, while the square black spots in the semantic map represent predicted navigable waypoints. In step 1 of navigation, the VLN model executes a 360-degree rotation to construct a more comprehensive semantic map and 3D feature fields. In the subsequent steps, the agent can only observe the forward-facing view. These examples illustrate that our approach can predict high-quality semantic maps of the environment, the traversable map can identify candidate waypoints well and achieve obstacle avoidance.Instruction: Turn around, walk out the door on the right, and walk across the hall into the bedroom on the left.Step 1Step 2Step 4Step 7Step 8Step 3Step 9Step 10Step 5Step 11Step 6 Instruction: Exit the room. Turn left and walk straight toward the pool. Wait near the pool.Step 1Step 2Step 4Step 5Step 6Step 3Step 7Step 8 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. 


Decision: # Strengths
1. The idea of using a 3D generalizable feature field for sim-to-real transfer is interesting.
1. Experiments are conducted across multiple benchmarks.
1. The real-world demonstrations showcase the effectiveness of the approach.
1. Most of the paper is well-written and easily comprehensible.

# Weaknesses
1. Comparison experiments are insufficient. For example, the proposed method should be compared with a simple baseline where the robot rotates at each waypoint.
1. The paper focuses on improving some of the standard metrics, sacrificing other important metrics.
1. The references are not comprehensive.

### Post-rebuttal comment 
The reviewers agree in their recommendation to accept the paper. I agree with their consensus.

Reviews: review: title: title: Official Review of Paper160 by Reviewer U8bM
paper_summary: paper_summary: This study proposes a Vision-and-Language Navigation method based on images captured by a monocular camera. The proposed method utilizes NeRF, enabling consideration of objects outside the view of the camera. The method is evaluated on two simulator-based benchmarks and real-world experiments. Results indicate that the proposed approach achieves a higher task success rate compared to existing monocular camera-based models.
main_review: main_review: ## Strength
- There is a novelty of extracting panoramic visual features using NeRF under narrow field-of-view conditions.
- The proposed method is well-explained. In particular, Figures 3 and 4 are clear and greatly aid in understanding the approach.
- Experiments are conducted across multiple benchmarks and environments, with appropriate baseline methods used for comparison.
- In sim2real experiments, the proposed method significantly outperforms baseline approaches, achieving scores comparable to simulation results.
- The ablation study appropriately analyzes contributions related to the method.
- The videos included in the supplementary materials are very effective in understanding the real-world experimental setup.
- The limitations section appropriately discusses the drawbacks of the proposed method, including potential future research directions.

## Weakness
- In the related work section,  in particular, the discussion of Vision-and-Language Navigation is insufficient. For example, highly relevant papers (e.g., [1], [2]) to this research are not cited.
- While the proposed method focuses on improving success rates, one concern is the apparent disregard for SPL (Success weighted by Path Length), which is an important metric in VLN tasks. If SPL can be ignored, the authors should discuss how success rates would be affected by using heuristic methods (e.g. an exhaustive search method). Furthermore, given that path length is sacrificed, the modest improvements - only 6 points on the R2R-CE dataset and only 1.7 points on the RxR-CE dataset - warrant further discussion.

[1] Hong, Y., Zhou, Y., Zhang, R., Dernoncourt, F., Bui, T., Gould, S., & Tan, H. Learning Navigational Visual Representations with Semantic Map Supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3055-3067, 2023.  
[2] Wang, H., Liang, W., Van Gool, L., & Wang, W. Dreamwalker: Mental Planning for Continuous Vision-Language Navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10873-10883, 2023.

## Minor Comments
- Is the absence of bold formatting for the highest values in Tables 3, 4, and 5 intentional?
- Is the line to the right of NE only in Table 4 intentional? If it's because of improvement with memory, this should be explicitly stated.
- Are the two arrows next to SPL in Table 2 a typo?
- Lines 265-266 are questionable, as the scores for the main metrics have decreased.
- In equation (4), aren't $u$ and $v$ undefined?
questions: questions: Please review the above weaknesses. Below is an additional question.

- If constructing the 3D feature field results in longer paths, wouldn't it be just as effective to capture multiple viewpoints with the monocular camera and create a panoramic image? The authors should verify this with experimental results.
limitations: limitations: 3
review_summary: review_summary: This paper, as mentioned above, shows some concerns regarding the assumptions of the proposed method and contains several notation errors. However, the novelty of using NeRF to compensate for the limitations of a monocular camera is valuable, and this is supported by the experimental results. Moreover, the proposed method's effectiveness is reported across multiple benchmarks, which lends high credibility to the validity of its novelty. Therefore, my decision is a "weak accept."
title: title: The usage of 3D neural fields to increase perception is technically sound, and the results support it as a SOTA level performance work. However, I believe it uses many off-the-shelf models designed directly for the simulator[9, 20, 12], which could lead to overfitting to the simulators. Nevertheless, the real-world experiments support that it can work in the real world, although the trajectories are relatively short. Overall, I am led to a positive conclusion.
paper_summary: paper_summary: This paper proposes leveraging a traversable map and a 3D feature map for waypoint prediction and selection, respectively. By building the temporal map, the agent gains enhanced perception of indoor scenes. These improvements enable the agent to more successfully navigate and search indoor environments.
main_review: main_review: Most of the paper is well-written with good illustrations, and the topic of panoramic and monocular observation is interesting. The benchmark experiments demonstrate the good performance of this paper. 
But I have some concerns:                                                                                                                                  
1.While I understand that the authors use a Cross-Modal Graph Transformer for waypoint selection, I would like to see a discussion or experiment about using feature neural fields with these methods.                                                                 
2.I believe a very important experiment is to compare with a method where the robot rotates at each waypoint. This baseline can directly reflect the real performance of this method.                                               
3.Many techniques rely on noiseless depth maps and odometry. I would like to know the authors’ comments on using these techniques in the real world.
questions: questions: See review part
limitations: limitations: 2
review_summary: review_summary: This paper is technically sound, and the experiments support most of the claims. I have some concerns about the experiments and real-world implementations, which are listed above. Overall, I am inclined to be positive if the authors can address my concerns.
title: title: seems like a good work but not in my expertise
paper_summary: paper_summary: The work proposes several techniques to deploy navigation robots with monocular RGBD camera and algorithmatically given panoramic traversability perception and semantic understanding.
main_review: main_review: The work proposes several techniques to deploy navigation robots with monocular RGBD camera and algorithmatically given panoramic traversability perception and semantic understanding. The techniques include predicting semantic traversable map for waypoint prediction and using 3D feature fields for panoramic perception. Training neural networks to do these allows real robots with only monocular limited-view RGBD camera to gain panoramic map and semantics and also allow sim-to-real.

Strengths:
- the proposed problem is a challenging yet important problem to address. The idea to build panoramic map and semantic understanding given only RGBD cameras is very important to the field.
- the proposed method has decent technical difficulty and novelty to the best of my judgement. The idea of using 3D generalizable feature field for potential sim-to-real transfer seems valid, interesting, and important to the field.
- experiments show clearly better performance than the state-of-the-art methods.
- ablation studies and analysis are also presented in a comprehensive and informative manner.

Weaknesses;
- mostly leveraging proposed technical components from prior work.
- longer navigation required
questions: questions: - how to address the longer navigation steps?
limitations: limitations: 3
review_summary: review_summary: not in my field of expertise, but looks like a good paper with decent story, method, and sufficient experiments to support