Context-Aware Replanning with Pre-Explored Semantic Map for Object Navigation

Authors: po-chen  ko
hung-ting  su
ching-yuan  chen
jia-fong  yeh
min  sun
winston h hsu
equal  contribution
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:31:57

Abstract: Pre-explored Semantic Map, constructed through prior exploration using visual language models (VLMs), has proven effective as a foundational element for training-free robotic applications. However, existing approaches assume the map's accuracy and do not provide effective mechanisms for revising decisions based on incorrect maps. This work introduces Context-Aware Replanning (CARe),, which estimates map uncertainty through confidence scores and multi-view consistency, enabling the agent to revise erroneous decisions stemming from inaccurate maps without additional labels. We demonstrate the effectiveness of our proposed method using two modern map backbones, VLMaps and OpenMask3D, and show significant improvements in performance on object navigation tasks.

Main paper: 
 introduction: 
 Navigating in indoor environments to locate and reach target objects is a fundamental capability for robotic applications. Conventional training-based object navigation methods necessitate extensive annotations, meticulous model design, and prolonged training periods to effectively align the control actions with visual perception. Advancements in visual language models (VLMs) have led to the development of modular approaches that separate perception from actions, utilizing pre-trained knowledge. Under this framework, visual perception can be independently learned without direct control, making exploration prior to task execution an effective strategy. Pre-explored Semantic Map [1, 2, 3, 4, 5] , constructed through prior exploration and using visual language models (VLMs), has become a fundamental backbone for robotics tasks. By constructing the map during environmental exploration, Pre-explored Semantic Map equips the agent with pre-existing knowledge of the environment, facilitating training-free robotic tasks such as manipulation motion planning [6] , interactive exploration [7] , and zero-shot object navigation [5] . However, current approaches presume that Pre-explored Semantic Map is always accurate-unbiased and noise-free-and they lack effective mechanisms for revising decisions based on incorrect maps. This assumption is flawed for real-world applications, where visual perception must contend with diverse environments, including varying lighting conditions, textures, weather, and dynamic elements. Consequently, visual perception cannot be assumed to be perfect. In addition, evaluating the quality of the Pre-explored Semantic Map is also challenging, as there are no existing labels for comparison.Intuitively, the map-based agent plans its path by retrieving the goal from the map, where retrieval is facilitated by sorting confidence scores based on map-query matching. Therefore, should the initial attempt fail, the agent can proceed to another unvisited location with the highest confidence score. Nevertheless, this approach assumes the map is accurate. However, failure to retrieve the map 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. with the highest confidence score may indicate a mismatch or error in the map-query alignment.Given the observation that the visual observation upon which the map is based fails to satisfy the query, we hypothesize that integrating Uncertainty measurement into the decision-making process can enhance the agent's ability to cope with inaccuracies in the map. By prioritizing areas with the greatest uncertainty, the agent targets high-uncertainty points for maximum information gain. This improves performance because initial prediction failures often result from biases or errors in the map data. Replanning based solely on confidence scores can perpetuate these biases. Highuncertainty areas, however, are less influenced by existing biases. Therefore, integrating uncertainty measurement reduces bias impact and increases the likelihood of successfully finding the goal. In addition, the same region can be observed from multiple viewpoints. If the VLM predictions from these multiple viewpoints are not aligned, the prediction is likely incorrect. Therefore, Multi-view Consistency can also serve as an unsupervised metric for agent replanning.To this end, we propose a novel Context-Aware Replanning (CARe) to revise the plan when the initial planning fails. Specifically, we first select k candidate regions as a strategy to mitigate the limitations of imperfect perception systems. While the top-ranked region might not always be reliable due to potential inaccuracies in perception, selecting a broader set of top candidates ensures a more robust assessment. Afterward, we measure uncertainty and multi-view consistency for candidate regions and select the best candidate based on the measured values. Our evaluation in two popular Pre-explored Semantic Map backbones, VLMaps [5] and OpenMask3D [3] , showcased that our method consistently outperforms the strategy of selecting the highest-scored unvisited region.The contribution of this work is summarized as follows:• We propose a novel Context-Aware Replanning (CARe), which replans based on the context of an incorrect Pre-explored Semantic Map when the initial task fails.• We design two variants of Context-Aware Replanning, based on uncertainty and multi-view inconsistency, without requiring additional annotations.• Our method consistently outperforms the strategy of selecting the highest-scored unvisited region using two different backbones [5, 3] , demonstrating its effectiveness and robustness. 
related work: 
 Visual Language Models (VLMs) serve as the backbone of modern modular training-free robotic applications by aligning visual perception with language, enabling robots to understand and follow natural language instructions. CLIP [8] achieves this alignment through contrastive learning, forming the foundation for various training-free robotic applications. For instance, OpenMask3D [3] and CLIP-Fields [2] utilize CLIP to construct maps. Additionally, several open-vocabulary visual perception tools, such as GLIP [9] for object detection and LSeg [10] for semantic segmentation, are crucial for robotic applications. For instance, VLMaps [5] uses LSeg to build its maps. While VLMs achieve significant performance on various computer vision tasks, robotic applications must handle diverse environments with varying lighting conditions, textures, weather, and object categories. Therefore, accounting for and replanning with errors in maps is crucial when leveraging maps constructed with VLMs. 
method: 
 In this section, we first introduce the background of our method in 3.1. Next, we illustrate the motivating concept behind our method to address a significant issue in current approaches in Section 3.2.We then detail our approach, which primarily focuses on leveraging uncertainty measures to select from a set of high-confidence candidates, as described in Section 3.3. Within this section, we further explain the single and multi-view uncertainty measures that our method utilizes. Subsequently, we outline the strategies we tested to generate the set of high-confidence candidates in Section 3.4. Finally, we describe the maps used in our experiments and discuss how they were adapted to integrate with our method in Section 3.5. Figure 2 illustrates a brief overview of our method.3.1 Map-Based Navigation.3.1.1 Goal-Oriented Object Navigation as Object Retrieval from Map.Navigating in a pre-explored environment typically involves an agent's internal representation of the environment, referred to as the map. Recent methods [3, 5] have reframed the goal-oriented object navigation task into an object retrieval task on the map by employing a path planner that devises a route from the agent's current position to the target, coupled with a policy that actuates the agent to follow this planned path.In practical scenarios, retrieval from the map is not always successful. In such cases, the capability to automatically replan becomes crucial for an autonomous agent, as it circumvents the need for costly human intervention. An intuitive approach to replanning would involve selecting the unvisited candidate location with the highest retrieval score.3.2 Uncertainty aware object retrieval.The conventional way to use navigation maps typically involves using a text query to retrieve the candidate with the highest matching score. Such a score is calculated with pretrained visual-language grounding models such as CLIP [8] . These grounding models have demonstrated impressive generalization abilities due to the large dataset used in the pertaining phase, which is probably why most of the works navigating with maps assume that the map is perfect. Nevertheless, we argue that despite the size of the training dataset, there might still be bias introduced by the model architecture or the dataset, which causes the user expectation of a query to diverge from the model's belief. Furthermore, during the construction phase of the map, the model prediction might be interfered with by surrounding objects or some view-dependent bias, causing the extracted feature to be noisy.To address these issues, our framework integrates uncertainty measures to enhance retrieval accuracy. These measures can potentially reflect biases or noise associated with a candidate, thereby improving retrieval performance.3.3 Uncertainty Measures.We hypothesize that when the grounding model retrieves the wrong object with high confidence, the model might have shown some bias in the object category. This is a common failure mode in conventional map-based navigation because the algorithm always chooses the candidate with the highest score/confidence. When the model bias occurs, solely looking at the confidence/score might give misleading retrieval results. On the other hand, completely ignoring the confidence/score might also be suboptimal, as it disregards valuable information from the strong grounding model. In this case, encouraging the retrieval of high confidence and high classification uncertainty candidates might be helpful as it both leverages the confidence information and mitigates the impact of potential biases by encouraging the exploration of uncertain candidates. Specifically, we first identify a list of high-confidence candidates and choose the candidate in the list with the highest entropy.3.3.1.1 Entropy: Let C = {c 1 , c 2 , . . . , c n } be the set of high-confidence candidates. For each c i , we compute the entropy H(c i ) of its classification distribution P i = {p 1 , p 2 , . . . , p k } as:H(c i ) = - k j=1 p j log p jwhere p j is the probability of candidate c i belonging to class j. We select the candidate c * with the highest entropy:c * = arg max ci∈Chigh H(c i )With this definition, a high entropy indicates that the classification distribution resembles a spreadout probability distribution, suggesting that the model considers several classes as plausible alternatives, thereby reflecting uncertainty. This approach ensures that we are not only considering candidates with high confidence but also those with high uncertainty, thereby reducing the likelihood of biased retrievals. For the specific method of selecting high-confidence candidates, please refer to Section 3.4.When the feature of a candidate is extracted by a small number of views, the resulting feature can easily be interfered by nearby objects or view-dependent model bias. To mitigate this issue, we propose to encourage multi-view consistency when retrieving the candidate. Similar to the single-view case, we first identify a list of high-confidence candidates and then encourage the retrieval of lowmulti-view inconsistency (uncertainty) candidates. We experimented with two kinds of multi-view uncertainty measures: channel-average feature standard error and mean pairwise KL divergence on multi-view classification probabilities.Let C = {c 1 , c 2 , . . . , c n } be the set of candidates, and V i = {v 1 , v 2 , . . . , v mi } be the set of views for candidate c i . For a candidate c i with only a single view feature, we set the uncertainty score U (c i ) to infinity.For each candidate c i with multiple views, the standard error SE(c i ) is computed as:SE(c i ) = 1 d d k=1 σ k √ m iwhere σ k is the standard deviation of the k-th feature across all views, d is the dimensionality of the feature vector, and m i is the number of views for candidate c i .Finally, we choose the candidate c * with the lowest standard error. Since most maps utilize the mean feature from multiple views for retrieval, a low standard error on these multi-view features indicates that the mean feature is statistically more reliable. Therefore encouraging low standard error could potentially reduce noisy retrievals.Pairwise KL Divergence: For each candidate c i , let P ij = {p ij1 , p ij2 , . . . , p ijk } be the classification probability distribution for view v j . The mean pairwise KL divergence D KL (c i ) is computed as:D KL (c i ) = 2 m i (m i -1) mi-1 j=1 mi l=j+1 1 2 D KL (P ij ∥ P il ) + 1 2 D KL (P il ∥ P ij )where D KL (P ∥ Q) is the KL divergence between two probability distributions P and Q.We then choose the candidate c * with the lowest mean pairwise KL divergence. A low mean pairwise KL divergence indicates that the classification distribution is consistent across multiple views. Similar to the standard error, promoting consistency in the classification distribution could help reduce noisy retrievals.In perception process, the highest-ranked class may not always be accurate because of potential errors or limitations in the system's capabilities. By selecting the top k candidate instead of relying solely on the top-ranked one, we mitigate the risk of inaccuracies and enhance the robustness of the assessment. This approach allows us to account for possible errors and improve the overall reliability by considering multiple promising candidates.Following the conventional and intuitive method, we simply choose the point with the highest confidence:arg max i conf cls iwhere conf cls i is the confidence, which is the probability or LSeg [10] score in the case of OpenMask3D [11] and VLMaps [5] respectively, of a single point i with the specified goal class cls in the task.Similar to the concept of top-k accuracy (Acc@K), we filter the point by whether its top-k confident predicted class contains the specified goal class or not, which can be formulated as follows:Filter(cls, conf i , k) = True, if cls ∈ (argsort cls conf i ) [0 : k] False, otherwisewhere cls is the target class and (argsort cls conf i ) [0 : k] is the highest k class prediction for the point i.We experimented with the proposed uncertainty-aware navigation method on two popular maps for object navigation: OpenMask3D [11] and VLMaps [5] . As described in Section 3.1.1, we focus on retrieval from the map by assuming successful path planning and following in OpenMask3D.In VLMaps, we employed the built-in path planner and follower from the HabitatSim simulator [12, 13, 14] .OpenMask3D [11] takes in the scene point cloud and posed RGB images, generates class-agnostic 3D masks on the point cloud, and uses the posed RGB images where the object is visible to provide semantic features. The features are calculated with CLIP with cropped images of the corresponding object and thus enable retrieval later on. If the object is visible in multiple views, OpenMask3D takes the average of multi-view features. The implementation details are further described in A.2.VLMaps takes the RGB-D image and its corresponding pose as input. It first generate the local point cloud which is then projected to the world coordinate frame and the map position with the information of camera poses. After that, the RGB image is fed into Visual Encoder of LSeg [10] to get its image feature, which will then be projected to the corresponding map position. Though we do not modify the process of building the map, we store some more data such as the standard error and KL divergence of each point on the map to decrease the size of the map and avoid repetitive computation in the evaluation process. When calculating the Channel-average Feature Standard Error, we incorporate the distance weighting [1, 5] information of each feature which is utilized when building the map. For more details, please refer to A.3. 
experiments: 
 4.1 OpenMask3D Object Retrieval Benchmark.To validate our method, we conducted an experiment in the Matterport3D environment [13, 14, 12] using OpenMask3D on a two-shot object retrieval task. In this task, a second retrieval attempt is allowed if the first one fails. Each room is benchmarked separately, and the results are presented in Table 1 . Note that in all baselines, the first candidate is always retrieved based on maximum confidence in the query class.We utilized the Matterport raw category, which contains 1658 classes, as the vocabulary required by OpenMask3D. In total, 10 scans (houses), 214 rooms, and 5370 object instances were evaluated in this experiment.• No replan: This is the top-1 retrieval accuracy without a second retrieval attempt.• Oracle: This is the upper-bound baseline where we count it success if any of the predicted 3D masks, regardless of its class scores, matches the GT one. This can also be seen as a top-infinity accuracy baseline. • Max confidence (top2 acc): This is a naive baseline where the candidate with the secondhighest confidence is selected when the first one fails, which matches the typical usage of navigation maps. • Random replan: This baseline randomly selects a candidate from all unvisited candidates as the second attempt. • Random from top-k: This baseline randomly selects a candidate from the high-confidence candidates set with the corresponding selection criteria as described in 3.4. Given an object mask retrieved with the text query, we calculate its point cloud IoU with the GT mask corresponding to the query. We count it a success if the IoU between GT mask and the retrieved mask is greater than 0.25.The best baseline entry from Table 1 is the "Random from top-2 confidence" method, which surpasses the naive maximum confidence baseline. This suggests that the map begins to provide biased confidence after the first failed retrieval.Our experiments demonstrate that both single-view and multi-view methods can enhance replanning performance. Specifically, 20 out of 25 experiment entries outperformed the best baseline entry, highlighting the efficacy and robustness of replanning with uncertainty and multi-view consistency.To show the effectiveness of our method on the downstream task brought by the improvement of the object retrieval performance, we further conducted some experiments on object navigation tasks using VLMaps. Following the setup of [5] , we randomly generate 10 scenes as well as some random poses for building the maps, and 10 object navigation tasks with some subtasks and subgoals for each scene. We use the Matterport3D dataset with HabitatSim simulator [13, 14, 12] for the agent to perceive and navigate. If the navigation fails, we then propose a new position of the specified class and check whether it is near the correct object. Due to the computational budget, we only select those methods with competitive results in the previous section.Given a proposed point for replanning, we follow the settings of VLMaps which checks whether the distance from the point to the nearest specified object is less than 1 meter.Table 2 showcases the effectiveness of various replanning strategies compared to the "Max confidence (highest score)" baseline. The "Max entropy" strategies demonstrate significant improvements over the baseline, reducing bias by considering entropy. The "Min stderr" strategy consistently outperforms all other methods, including the baseline, by focusing on minimizing the standard error in confidence predictions. This approach proves to be the most robust and effective, highlighting the importance of incorporating uncertainty and multi-view consistency in replanning.  
limitation: 
 While our proposed CARe effectively cooperates with existing pre-explored semantic maps and navigation models in a training-free manner to achieve better performance, it may be limited by one major assumption: CARe assumes that the navigation model has consistent decision biases. This assumption holds true when working with a frozen model. However, if the navigation model is updated, these decision biases may be eliminated or changed, resulting in less significant performance improvements from CARe. While our proposed uncertainty measures are theoretically modalityagnostic and applicable to various 2D or 3D modalities, practical challenges such as sensitivity to channel scale in feature-based methods and the need for classifiers or alternative approaches in distribution-based methods should be considered to ensure their effectiveness. Additionally, because the structure of the semantic map may vary, CARe only uses the fixed semantic map for re-planning and does not further update the map with new information during the process. This research direction has the potential to continuously improve performance but is beyond the scope of this study. We will discuss and verify this direction in our future work. 
conclusion: 
 Summary: We propose a novel method, Context-Aware Replanning (CARe), which accounts for unavoidable errors in Pre-explored Semantic Map and revises the plan. By leveraging uncertainty and multi-view consistency in Pre-explored Semantic Map, we replan the agent without additional human effort. We demonstrate the effectiveness and robustness of CARe by integrating it with two Pre-explored Semantic Map backbones, VLMaps [5] and OpenMask3D [3] . This integration consistently outperforms all baselines with various hyperparameters, achieving a peak success rate of 18.75% and a subgoal success rate of 82.7%.Future Work: Given the portability of CARe, we are optimistic about its potential benefits for future robotic research and applications. By leveraging CARe, future research areas such as visualand-language navigation (VLN) and applications like healthcare robots could facilitate replanning with reduced labeling costs. For example, a VLN system can leverage Pre-explored Semantic Map and enhance its success rate and Success weighted by Path Length (SPL) by replanning. The concept of CARe can also be adopted in the pre-exploring stage to enhance the quality of the Pre-explored Semantic Map by measuring with uncertainty and re-exploring. C = score of O from high to low, shape (N,O ′ = points with top k highest scores, shape (N ′ , f )else if S == TopPrediction then if U == Entropy then ▷ Choose a point with uncertainty measure 17:EQUATIONO * = argmax(E), shape(1, f ) else if U == KL then 23:EQUATIONO * = argmin(KL), shape(end if 
 
 Appendix: return O * 27: end if View Selection: Given a 3d object mask, the 3d points in the mask are projected back to 2d for all posed RGB images in the scene. We then validate whether the object is visible in a view by checking if any of the points projected to 2d lies within the image. If there are more than 5 views where the object is visible, we rank the images by the number of object pixels and choose the top 5 views. This not only helps us manage the computation cost but also encourages the selection of views that are closer to the object, which might be helpful in filtering out far-away views that might not capture the object clearly.Feature Extraction: Following the original OpenMask3D implementation, we used CLIP-ViT-L/14 for encoding images and texts. The 3d to 2d projection operation mentioned in the last paragraph has also allowed us to calculate the bounding-box of the object which we refer to as "object crops". For feature extraction, we encode the object crops with the CLIP visual encoder and save them all instead of taking the average of them. In the original OpenMask3D implementation, they also used multi-scale cropping for each object crop as a data augmentation. Since data augmentation is orthogonal to the direction of this work, we omitted this part for simplicity.Method Overview: Following the original VLMaps [5] , we first select 10 scenes and randomly generate several poses, which includes position and rotation, with their corresponding RGBD observations. Then, the image features generated by LSeg [10] model are projected to the global frame.Channel-average Feature Standard Error for VLMaps: Let C = {c 1 , c 2 , . . . , c n } be the set of candidates, and V i = {v 1 , v 2 , . . . , v mi } be the set of views for candidate c i . For a candidate c i with only a single view feature, we set the uncertainty score U (c i ) to infinity.Recall that for each candidate c i with multiple views, we compute the standard error SE(c i ):SE(c i ) = 1 d d k=1 σ k √ m iwhere σ k is the standard deviation of the k-th feature across all views, d is the dimensionality of the feature vector, and m i is the number of views for candidate c i .And in the case of VLMaps, we incorporate the distance weighting [1, 5] information of each feature which is utilized when building the map, that is we replace the standard deviation and the sample size with the weighted version:SE w (c i ) = 1 d d k=1 σ w k m ef f iwhere σ w k is the weighted standard deviation and m ef f i is the effective sample size, calculated as follow:m ef f i = ( n i=1 w i ) 2 n i=1 w 2 iFinally, we choose the candidate c * with the lowest standard error.Map Generation: As previously mentioned, we build the map by the method that is identical to VLMaps. However, we further save some metrics for each grid which are utilized in our work, such as entropy, standard error, and KL divergence. Additionally, to align with the method of feature fusion [1] adopted in VLMaps, we calculate the above metrics in their weighted version.Navigation and Planning: In the navigation stage, VLMaps first generate a mask indicating the presence of a specific object class, and it then plans a path to the boundary of the nearest object. With the provided path, it further calculate the angle and distance between two subsequent halfway point, generating the low-level actions that is used in the HabitatSim [13, 14, 12] .Evaluation and Re-proposing: After all the actions are executed, we calculate the distance between the agent and the approximate boundary of the nearest object with the ground truth data provided by the simulator. Following the settings in VLMaps, we count it success when the distance is less than or equal to 1 meter. If it fails, we then generate a new proposal of where the object may be by our method CARe. Similarly, we calculate the distance between the new point and its nearest object, checking whether the distance is less than or equal to 1 meter.Quantitative Results: In the setting of VLMaps, we also conduct the KL divergence method as our uncertainty measure as shown in Table 3 . However, some scenes with larger space or more data, may cause the calculation of pairwise KL divergence become quite computational intensive. Thus, we have to skip two larger scenes due to the limitation of memory size, which makes the result not comparable to others.Qualitative Results: We provide more qualitative results on the anonymized project page C Supplementary Experiments and Analysis.C.1 Computational Complexity Analysis.Our method incorporates a replanning phase when the initial retrieval attempt is unsuccessful. Such replanning phase includes the calculation of uncertainty measures, which introduces additional computational time compared to the baseline methods. To evaluate the computational overhead introduced by our approach, we performed a supplementary experiment and analysis. Table 4 presents a theoretical analysis for each replanning strategy. The computational complexity for the entropy and stderr measures increases linearly, whereas, for the pwKL measure, the complexity grows quadratically due to the need for pairwise computations between candidates. While the pairwise KL divergence method provides superior retrieval performance in the main experiments, this theoretical analysis suggests that it might suffer from larger computational costs when the candidate count n is too large. In this case, users could consider using the stderr metric with linear complexity to measure multi-view consistency. N WLPHV WRS WRSHQWURS\ WRSVWGHUU WRSNO To further demonstrate the real-time applicability of our method, we measured the time requirements in the OpenMask3d experiment setting. We assumed that object features are precomputed and stored in the maps, with uncertainty measures such as entropy and KL divergence computed on the fly. Figures 5 and 6 present the results under different top-k strategies. For all uncertainty measures, both top-k selection criteria and all tested values of k, a retrieval attempt typically takes less than 15 milliseconds. Considering that real-world physical navigation involves seconds or minutes of path-following interactions after determining a destination, we believe that the millisecond-level computational overhead introduced by our method is negligible.Note that in practice, the uncertainty measures can also be precomputed like the object features. In this case, the latency for our method could be further decreased. In latency-critical scenarios, users of our method can trade space for time by precomputing and storing the uncertainty measures.D Limitations and Future Works.While our proposed CARe effectively cooperates with existing pre-explored semantic maps and navigation models in a training-free manner to achieve better performance, it may be limited by one major assumption: CARe assumes that the navigation model has consistent decision biases.This assumption holds true when working with a frozen model. However, if the navigation model is updated, these decision biases may be eliminated or changed, resulting in less significant performance improvements from CARe. Additionally, because the structure of the semantic map may vary, CARe only uses the fixed semantic map for re-planning and does not further update the map with new information during the process. This research direction has the potential to continuously improve performance but is beyond the scope of this study. We will discuss and verify this direction in our future work.https://carmaps.github.io/supplements/ 


Decision: The paper addresses an interesting area of research. The authors have reasonably addressed the reviewers' concerns improving the quality of the paper. However, there is still room for improvement in terms of real-work experiments.

Reviews: review: title: title: Context-Aware Replanning (CARe)
paper_summary: paper_summary: The paper presents a novel approach, Context-Aware Replanning (CARe), to address the challenge of inaccurate maps in robotic navigation tasks that utilise pre-explored semantic maps. The core idea is to incorporate uncertainty estimation and multi-view consistency into the decision-making process, allowing the robot to revise its plan when the initial plan fails due to map inaccuracies. The method is evaluated on object navigation tasks using two different map backbones, demonstrating improved performance compared to traditional approaches.
main_review: main_review: The work is intuitive and the authors clearly explain their methodology and the evaluation metrics used. The ablation studies further strengthen the validity of their approach by demonstrating the impact of different uncertainty measures. The results presented in the paper are convincing and support the claims made by the authors (the reviewer appreciates the supplementary materials provided).

Clarity: The paper is well-written and organized, making it easy to follow the authors' thought process. The introduction effectively motivates the problem and highlights the limitations of existing approaches. The related work section provides a good overview of the relevant literature. The methodology is described in detail, with clear explanations of the uncertainty measures and the map backbones used. The experimental setup and results are presented clearly, and the ablation studies provide further insights into the method's effectiveness.

Originality: The use of uncertainty estimation and multi-view consistency for replanning is a unique approach that addresses the limitations of existing methods that assume map accuracy. The modularity and portability of the proposed method to different map backbones further enhance its originality.

Significance: The research addresses a significant challenge in robotic navigation, where map inaccuracies can lead to task failures. The proposed method has the potential to improve the robustness and reliability of robotic systems in real-world environments where perception is imperfect. The modularity and portability of the method make it potentially applicable to a wide range of robotic applications beyond navigation.
questions: questions: 1. The evaluation is limited to object navigation tasks in a simulated and relatively constrained environment. Has there been any attempts at a real-world implementation of a real robotic system?
2. The paper does not discuss the proposed method's computational complexity. It would be beneficial to analyse the computational overhead and its impact on real-time performance.
3. The paper could benefit from a more detailed discussion of the limitations of the proposed method and potential future directions for research.
limitations: limitations: 1
review_summary: review_summary: The work is interesting and addresses a relevant question within object navigation. It is a relatively small incremental work with a modest implementation in simulation.
title: title: Context-Aware Replanning with Pre-Explored Semantic Map for Object Navigation
paper_summary: paper_summary: This paper introduces Context-Aware Replanning (CARe) for object navigation using pre-explored semantic maps constructed through visual language models (VLMs). Unlike traditional approaches that assume the accuracy of the map, CARe estimates map uncertainty through confidence scores and multi-view consistency, allowing for the revision of erroneous decisions without additional labels. The effectiveness of CARe is demonstrated using two modern map backbones, VLMaps and OpenMask3D, showing significant improvements in object navigation tasks.
main_review: main_review: This paper presents a novel approach to object navigation by integrating uncertainty estimation into the decision-making process. The main contributions of this work are:

    Originality: The approach of using confidence scores and multi-view consistency to evaluate map uncertainty is innovative and addresses a critical gap in current methodologies.
    Technical Quality: The proposed method is rigorously tested using state-of-the-art map backbones, VLMaps, and OpenMask3D. The experimental results demonstrate substantial improvements over existing methods.
    Clarity: The paper is well-structured, providing clear explanations of the theoretical background, methodology, and experimental results.
    Impact: This research has the potential to significantly enhance the accuracy and reliability of robotic object navigation in real-world applications.

However, several areas need improvement:

    Keywords: The keywords are not appropriately set, which indicates a potential oversight in ensuring the quality of the paper. The authors should carefully review the details of the manuscript.
    Abstract: It is unusual to include citation numbers in the abstract. The authors should remove these references.
    Novelty: While the use of VLMs for semantic mapping has been extensively studied recently, the approach of using confidence scores and uncertainty measurements predates VLM-based semantic mapping. Therefore, the novelty of this work appears limited.
    Experimental Details: Some experimental values are missing, such as those in Tables 1 and 2. A more comprehensive evaluation is necessary.
    Limitations: The paper does not discuss the limitations of the proposed method. It would benefit the readers to understand the constraints and potential issues. Specifically, the applicability of this method beyond the VLMaps and OpenMask3D backbones should be addressed. Semantic maps come in various forms (2D, 3D, object-centered, RGB-D maps, neural network embeddings), and it is unclear which types are compatible with the proposed method.
    Planning Discussion: Although the focus of this research is on replanning, the section on the proposed method primarily discusses uncertainty quantification, confidence selection, and map representation, with insufficient detail on the actual planning process.
questions: questions: Could you provide more details on the computational cost of the proposed method, particularly regarding its applicability in real-time applications?
Do you have plans to conduct additional experiments to validate the effectiveness of the proposed method under different environments and conditions? If so, please provide details.
limitations: limitations: 1
review_summary: review_summary: This paper introduces a novel approach to object navigation by incorporating map uncertainty estimation. While the technical quality and potential impact are high, there are concerns regarding the novelty, completeness of experimental results, and the discussion of limitations. The authors should address these issues to strengthen the paper.
title: title: The idea is great and meaningful but the experiments need to be improved
paper_summary: paper_summary: The authors proposed several metrics to evaluate the uncertainty in current open-vocabulary mapping methods and use those metrics to help with open-vocabulary object navigation in explored environment
main_review: main_review: Strength:
The initiative of computing uncertainty in current open-vocabulary mapping is important and meaningful. The authors proposed to utilize the top-k candidates of the labels, and compute their probabilistic distribution, entropy, multi-view feature standard deviation, and average pairwise view KL-divergence of the distribution prediction. The metrics proposed are intuitive and make sense. In the overview, the results show valid improvement of performance when the metrics are applied to evaluating the meaningful goals to navigate to.

Weakness:
The major drawback of the paper is the clarity of the writing (method and experiment). 
1. The metrics in the paper are well defined but further explanation of such definition would be appreciated. For example, SE(c_i), why do the authors think this metric can reflect the uncertainty? Same for D_{KL}(c_i) etc. It would be great to add one to two sentences to explain the meaning of the metrics.
2. At the beginning of the method part, it would be great to have an overall introduction of how the proposed metrics can be integrated into the pipeline of open-vocabulary mapping. Or cover what will be introduced in the subsections. Currently, each subsection of the method part is meaningful to some extent but there are no clear connections among them.
3. The experiments have a lot of rows but is not clear how they matched with the introduced methods (which row means what metric). It would be great if the authors can define those rows and the metrics they mean. At the same time, there are a lot of redundant descriptions like "top k confidence"/"top k category". The author can introduce a new column with the title "selection criterium" or "select with top-k", and in each row write either "confidence" or "category". Same for "std"/"stderr"/"pwKL"
4. The k seems to be a hyperparameters and the authors should bold the best performing entry for each column.
5. The conclusion is not clear.
6. The limitation of the methods are not described.
questions: questions: Address the bullet points in the review.
limitations: limitations: 1
review_summary: review_summary: The idea and initiative of the paper are good but more clarity is required. If the authors can improve it according to some of the reviews, I would raise the rating to one level higher.