GAMap: Zero-Shot Object Goal Navigation with Multi-Scale Geometric-Affordance Guidance

Authors: shuaihang  yuan
hao  huang
yu  hao
congcong  wen
anthony  tzes
yi  fang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:41:23

Abstract: Zero-Shot Object Goal Navigation (ZS-OGN) enables robots to navigate toward objects of unseen categories without prior training. Traditional approaches often leverage categorical semantic information for navigation guidance, which struggles when only partial objects are observed or detailed and functional representations of the environment are lacking. To resolve the above two issues, we propose \textit{Geometric-part and Affordance Maps} (GAMap), a novel method that integrates object parts and affordance attributes for navigation guidance. Our method includes a multi-scale scoring approach to capture geometric-part and affordance attributes of objects at different scales. Comprehensive experiments conducted on the HM3D and Gibson benchmark datasets demonstrate improvements in Success Rates and Success weighted by Path Length, underscoring the efficacy of our geometric-part and affordance-guided navigation approach in enhancing robot autonomy and versatility, without any additional task-specific training or fine-tuning with the semantics of unseen objects and/or the locomotions of the robot.

Main paper: 
 introduction: 
 Zero-Shot Object Goal Navigation (ZS-OGN) is a pivotal research domain in embodied AI and robotics, enabling robots to navigate towards the objects of unseen categories without training or fine-tuning on these objects [1, 2, 3, 4, 5] . This capability is crucial for real-world robots, such as home service robots and blind guiding robots, allowing them to interact with diverse objects in real-world scenarios, thereby enhancing their autonomy and versatility.Prior works on ZS-OGN either leverage deep neural networks to directly map RGB-D observations to actions learned from paired training data [6, 7, 8, 9, 10, 11] or utilize map-based navigation methods [12, 13, 14, 15, 16, 17, 3] . However, deep neural network approaches often struggle due to their dependence on extensive annotated data, resulting in poor generalization to unseen environments [18, 16] , while map-based navigation methods instead offer an alternative. Map-based navigation methods track categorical semantics and topological information observed by the agent to select promising exploration locations [17] . With the advent of foundation models, the studies [3, 17, 19] have exploited the reasoning capabilities of Large Language Models (LLMs) to strategically select Figure 1 : The leftmost RGB image shows the same observation for both methods. Our method (top row) effectively identifies the geometric part of the chair back, which is missed by the traditional method (bottom row). Consequently, GAMap successfully guides the agent to the target object, while the traditional method fails. The red circles highlight the areas where the chair is located, and the GA score is high, indicating the effectiveness of our approach in localizing relevant regions.waypoints by analyzing commonsense, such as object co-occurrence relationships, to navigate robots towards the target. However, LLM-based approaches require converting visual and semantic information into categorical descriptions, which leads to a loss of spatial and visual information [9] . Vision Language Models (VLMs) enhance semantic reasoning capabilities, but still rely on maps that encompass only categorical information [16] . The primary limitation of exclusively relying on categorical information is that such maps treat objects as monolithic entities, disregarding local geometric features. This becomes particularly problematic when only the target object is partially observed, leading to incorrect categorical information and potential errors in waypoint selection.We argue that using a categorical map for robot exploration is suboptimal, as it discards intricate geometric details and functional representations of the environment, as illustrated in Figure 1 . Drawing inspiration from human cognitive processes -where distinctive geometric parts are often identified first when locating an object in an unfamiliar environment [20, 21] -we propose Geometric-part and Affordance Maps (GAMap), a zero-shot approach, for the geometric parts and affordance attribute driven semantic navigation to explore and find the target object in an unseen environment. Specifically, given a target object, our proposed method starts by using an LLM to infer the object's geometric parts and potential affordance attributes, aiming at providing a detailed understanding of both the object's physical structure and its functional properties. Given depth observations, GAMap maintains a 2D projection of obstacles and explored areas. Instead of relying on object detection and prompt engineering of LLMs to select the next area to explore, our approach employs a pre-trained CLIP [22] to score observations based on their similarity to the reasoned geometric parts and affordance attributes, guiding the exploration process. To construct the proposed GAMap, which requires obtaining scores for geometric parts at different scales, we further propose a Multi-scale Geometric and Affordance score (GA score). Such an integration addresses a notable limitation in existing approaches that compute similarity at a single scale, which usually results in the oversight of fine-grained details of an object, as such details are potentially essential for an accurate identification of geometric parts or affordance attributes of objects with different sizes.Our proposed method is evaluated on HM3D [23] and Gibson [24] benchmarks and achieves significant improvements in Success Rate (SR) of 26.4% on HM3D [23] and 23.7% on Gibson [24] compared to previous approaches. Additionally, we achieve substantial gains in Success weighted by Path Length (SPL) of 37% on Gibson, highlighting the efficiency and effectiveness of our proposed geometric parts and affordance guided navigation. The contributions of our method are mainly summarized as follows:1. We propose a novel Geometric-part and Affordance Map (GAMap) for ZS-OGN using object part and affordance attributes as guidance. To the best of our knowledge, this is the first work to study the integration of these attributes in ZS-OGN.2. Recognizing that geometric parts and affordance attributes often relate to multiple scales of an object, we propose a Multi-scale Geometric and Affordance score, which allows GAMap to be constructed in real-time, better capturing these attributes at different scales.3. We achieve state-of-the-art performance on two navigation benchmark datasets without any training or fine-tuning with the semantics of unseen objects and/or the locomotions of the robot, which demonstrates the effectiveness of our method in unseen environments. 
related work: 
 Semantic Mapping. In the context of object goal navigation, it is crucial to transform observations into structured information for effective decision-making. Frontier-based methods [3, 16, 19] utilize categorical semantic information near frontiers to select exploration areas. Additionally, graph-based mapping methods [17, 25, 18] predict waypoints from RGB-D images or simplified maps to create topological representations of the environment. Most of the aforementioned works rely on semantic segmentation or object detection to build semantic maps, which are constrained by the pre-defined semantic classes and thus fail to capture the full semantic richness of environments [18, 26] . To overcome these limitations, recent approaches like VLMaps [18] have introduced open-vocabulary semantic maps, enabling natural language indexing and expanding the scope of semantic mapping.In addition, previous works [27, 28] attempt to utilize attributes and long descriptions for object perception. While these methods have advanced the navigation field, they often overlook object parts, treating objects as monolithic entities and leading to errors when these objects are partially observed.Inspired by human cognitive processes [21, 20] , where distinctive geometric parts are identified first in unseen environments, we propose Geometric-part and Affordance Maps (GAMap). Unlike previous methods focusing solely on categorical information, GAMap integrates geometric parts and affordance attributes, providing a richer and more functional representation of the environment.Zero-shot Object Goal Navigation. In the context of object goal navigation, the aim is to efficiently explore a new environment while searching for a target object that is not directly visible. Previous research relies heavily on visual context via imitation [29, 6] or reinforcement learning [7, 8, 9] to guide robots. These approaches often require extensive data collection and annotation for training, which limits their practical application in real-world environments. Thus, the focus in object goal navigation has been shifting towards zero-shot object navigation, which aims to equip robots with the ability to adapt to unseen objects and environments without the need for training [12, 30, 13, 14] . Clip-Nav [31] utilizes CLIP [22] to execute vision-and-language navigation in a zero-shot scheme, whilst CoW [32] employs CLIP for object goal navigation. Recently, Frontier-based Exploration (FbE) [33] is widely adopted in navigation by moving the robot to the frontier between known and unknown spaces [32, 34, 35, 36, 37] , leading to promising performance compared to learning-based exploration methods [38, 39] . More recently, ESC [3] leverages the reasoning ability of LLMs to select frontiers using pre-defined rules. Chen et al. [15] explore frontier selection by jointly considering the shortest path to frontiers and the relevance scoring between objects for exploration.To enable more robust and reliable exploration and waypoint selection, Wu et al. [17] propose a Voronoi-based scene graph for waypoint selection. Unlike the above methods that use the reasoning ability of LLMs to select frontiers, VLFM [16] introduces a value map to score frontiers based on the categorical similarity between the observation and the target object. In contrast to prior work, for the first time, we explore robot navigation using geometric parts and affordance attributes as guidance. This approach integrates detailed geometric parts and functional properties of objects, offering a more comprehensive strategy for navigation. 
method: 
 We first formalize the ZS-OGN problem in Section 3.1. Then, we detail our method, as shown in Figure 2 , from four phases: attribute generation in Section 3.2, multi-scale attribute scoring in Section 3.3, GAMap generation in Section 3.4, and exploration policy in Section 3.5. Initially, our method generates geometric parts and affordance attributes for the target object. During the exploration, the method computes a multi-scale attribute score from the RGB observations collected by the agent. These scores are then mapped onto a 2D geometric parts and affordance map, which is pivotal in guiding the exploration process. Subsequently, the agent selects the location with the highest score for further exploration. The RGB observation is partitioned into multiple scales, and a CLIP visual encoder generates multi-scale visual embeddings. GA scores are computed using cosine similarity between attribute text embeddings from a CLIP text encoder and the multi-scale visual embeddings. These scores are averaged and projected onto a 2D grid to form the GAMap.In ZS-OGN, the robot must navigate to a target object g i that has never been encountered before in an unseen environment s i , without any training on g i and s i . A navigation episode can be defined as E i = {g i , s i , p 0 }, where p 0 denotes the robot's initial location, the subscript i refer to the i th episode. The robot receives a color image I t , a depth image D t , and its pose, i.e., position (x t , y t ) and orientation θ t , at each exploration step t. We denote these readings as an observation O t = {I t , D t , x t , y t , θ t }. The agent accumulates pose readings over time to determine its relative position p t . Based on the readings at each step, the robot needs to select an action a t from the action space. A navigation episode is marked as successful if the robot executes the STOP action within a pre-defined distance to the target object. In this work, we approach the navigation task as a sequence of decisions made by the robot. The process starts at the initial time step t = 0 and ends at the final time step T . This final step is either when the robot executes the STOP action or when a pre-defined maximum number of exploration steps is reached.We focus on two types of attributes essential for object recognition: Affordance and Geometric-part attributes. Affordance attributes refer to the potential actions that an object facilitates [40] , which are crucial for understanding how an agent might interact with different objects within an environment. Geometric-part attributes, on the other hand, describe the shape and spatial characteristics of an object, aiding in its visual identification and differentiation from other objects.To extract these attributes, we employ an LLM to reason about the target object's characteristics. Specifically, we utilize GPT-4V [41] for the attributes generation. We initiate this process by setting the system prompt as: "I am a highly intelligent question-answering bot, and I answer questions from a human perspective." Subsequently, we employ two tailored prompts to extract the desired attributes. For affordance attributes, we design prompt as: "For the target object <target object g i >, please provide <N a > affordance attributes that to the most reflect its characteristics." to query N a number of affordance attributes. For geometric parts, the prompt is: "Summarize <N g > geometric part visual features of the <target object g i > which are typically used to identify why it is a <target object g i >." to get N g number of geometric attributes. Once the set of affordances attributes {A n } Na n=1 and geometric attributes {G n } Ng n=1 have been identified, the next stage involves coupling these attributes with the agent's observations by computing Geometric-part and Affordance scores (GA scores) through multi-scale visual features, as detailed in the following.The key idea of Multi-scale Attribute Scoring is to quantify the relevance of observed areas' geometric and affordance characteristics in locating the target object. These scores determine the agent's subsequent exploration decisions, guiding it to the areas most likely to contain the target object.To correlate the scoring with both the global frame and localized patches, ensuring that finer details and smaller components are effectively scored, we partition the observed image into patches of equal size across different scales. Specifically, given an RGB observation image I t , with height H and width W , the partitioning process is as follows. At level k, the image is partitioned into 4 (k-1) equal parts with each patch at level k of size H2 (k-1) × W 2 (k-1). The patches at level k can be represented as: 1) where P h,w denotes the patch located at the h th row and w th column of the partitioned image at level k. All patches from all levels are then resized to the same dimensions for further processing. We utilize CLIP [32] to calculate the visual embeddings for all patches from all levels. For each patch P k h,w at level k, its visual embedding v k h,w is computed using the image encoder of CLIP. Simultaneously, each attribute embedding e is computed using the text encoder of CLIP, given the attribute descriptions, {A n } Na n=1 and {G n } Ng n=1 , generated from an LLM. For each patch P k h,w , we calculate the cosine similarity between its visual embedding v k h,w and the attribute embedding e to obtain a similarity score S k,e h,w as follows:I k t = P h,w | 1 ≤ h ≤ 2 (k-1) , 1 ≤ w ≤ 2 (k-EQUATION)Due to the hierarchical partitioning of the image, we accumulate the scores across all scales for each pixel location and then take the average to obtain the final score for the image. Specifically, let L be the number of levels, and the accumulated score for the pixel at position (p, q) in the image I t is calculated by summing the scores from all levels and then averaging:EQUATIONwhere h(p, q, k) and w(p, q, k) map the pixel position (p, q) in the image to the corresponding patch indices at level k. The scores are then used to generate a Geographic and Affordance Map (GAMap) for the explored area. The process for generating the GAMap is detailed in the next section.At the core of our approach is the Geographic and Affordance Map. This map assigns a GA score to each pixel within the explored area to quantify the relevance of different regions in locating the target object, associating the areas with the highest values as the most promising for further exploration.We define the GAMap at time step t as M t ∈ R Ĥ× Ŵ ×C , where Ĥ and Ŵ are the dimensions of the 2D projection grid map, and C is the number of attributes, i.e., C = N a + N g with N a and N g representing the number of attributes and parts, respectively. To construct the GAMap from the RGB-D observation I t and the depth data D t , we back-project every pixel from D t to reconstruct the point cloud following [18] :EQUATION)where K is the intrinsic matrix of the depth camera, and D t (p, q) is the depth value of the pixel at the coordinate (p, q). To transform the point cloud into the world coordinate frame, we use X world = T W • X, where T W is the transformation matrix from the camera coordinate to the world coordinate. The 3D points are then projected onto the ground plane to determine the corresponding positions on the grid map. We assume perfect calibration between the depth and RGB cameras, allowing us to project each image pixel's score to its corresponding grid cell in the map. Given that multiple 3D points may project to the same grid location, we retain the maximum value for each channel that falls into the same grid cell as the score of this cell:M t ( ĥ, ŵ, e) = max S(p, q, e) | (p, q) ∈ cell M t ( ĥ, ŵ) ,where M t ( ĥ, ŵ, e) represent the score of attribute e in the 2D grid at position ( ĥ, ŵ), S(p, q, e) is the score for the pixel at (p, q) for the attribute e, and cell M t ( ĥ, ŵ) denotes the grid cell on the GAMap. When the robot moves to a new position, resulting in overlapping observations with the previously explored regions, the GA scores for each pixel in the overlapping region are updated. The updated GAMap is computed by taking the maximum of the current attribute score and the previous attribute score for each cell:EQUATIONwhere M ( ĥo , ŵo , e) is the GAMap constructed from the previous step, and the subscript o represents the overlapped grid cell.To determine the next area to explore, the robot selects the region with the highest GA score, calculated as the average of all attribute channels. To enable efficient exploration, only areas near the frontier with the highest scores are selected. Once the area with the highest GA score is identified, a heuristic search algorithm, i.e., the Fast Marching Method (FMM) [42] , is employed to find the shortest path from the robot's current location to the selected area. The robot then generates the appropriate actions to navigate along this path. At each step, the GAMap is updated based on new observations. We repeat this process until the robot either identifies and reaches the target object or the episode ends.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: We do not have assumptions Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiment: 
 Datasets. HM3D [23] is a dataset consisting of 3D data from real-world indoor spaces, along with semantic annotations, serving as a foundational resource for the Habitat 2022 ObjectNav Challenge [23] . This comprehensive dataset includes 142,646 object instance annotations, organized into 40 distinct classes across 216 environments, covering a total of 3,100 individual rooms. We follow the validation settings from [3, 32] to evaluate our proposed method. Gibson [24] was developed by Al-Halah et al. [43] . The dataset comprises 5 validation scenes across 6 object categories, and we adhere to the standard evaluation protocol [44, 19, 30, 2] to use all 5 validation scenes for evaluation.Metrics. Success Rate (SR, %) [45] focuses on the agent's accuracy in reaching the designated target, where a higher value indicates better performance. SR is computed based on whether the robot successfully stops within 0.1m of the target object g i :EQUATIONwhere K is the number of episodes, d(g i , p Ti ) is the distance between the target object g i and the robot's final position p Ti in the i th episode, and 1 {•} is an indicator function. Success weighted by Path Length (SPL, %) [45] evaluates success relative to the shortest possible path, normalized by the actual path taken by the agent, measuring the efficiency of the agent's success in reaching a goal, defined as:EQUATIONwhere L i is the actual path length traveled by the robot in the i th episode and L * i is the shortest possible path length to the target in the same environment.We compare our method against several ZS-OGN approaches, including the state-of-the-art methods: Random Exploration: takes random actions to explore the environment. Nearest FbE [33] : explores the environment by selecting the nearest frontier. SemExp [2] : utilizes a category semantic map and trains a local navigation policy for exploration. PixNav [9] : trains models for navigation by selecting pixels as intermediate goals. PONI [44] : uses potential functions to select frontiers for exploration. ZSON [30] : employs categorical information to train a model for object-based navigation tasks. CoW [32] : uses CLIP for categorical information extraction and explores using the nearest frontier-based exploration. ESC [3] : utilizes a categorical semantic map and commonsense reasoning for target object exploration. L3MVN [19] : uses an LLM to reason about the next exploration area based on a trained detection head for semantic map construction. VLFM [16] : employs BLIP-2 [46] and categorical information of the target object to evaluate and select frontiers for exploration. VoroNav [17] : uses a Voronoi-based decomposition strategy for navigation. SemUtil [15] : considers the shortest path distance to frontiers and the relevance scoring between objects for exploration. We compare our method with existing approaches across four categories: those utilizing both locomotion and semantic training [2, 30] , those employing only locomotion training [9, 16] , those using only semantic training [19, 44] , and those that do not incorporate any training [3, 32, 17, 15] . Locomotion training involves learning-based methods for navigation, while semantic training requires training or finetuning a perception module to construct a semantic map. The results and comparisons are shown in Table 1 .On the HM3D dataset, our method achieved a SR of 53.1% and a SPL of 26.0%. This represents a significant improvement over the best method [17] that does not use locomotion and semantic training, with a 26.4% increase in SR.Although the SPL (26.0%) of our method is lower than that of VLFM (30.4%), this discrepancy can be attributed to the fact that VLFM's local policy planning is trained. Considering the different path planning methods adopted in our approach and VLFM, we construct a comparative experiment with VLFM. Specifically, we kept the VLFM value map generation process unchanged and replaced its path planning method with FMM instead of the trained policy. The detailed results are shown in the second last row in Table 1 . The comparison reveals that our model outperforms VLFM-based mapping method by 4.32% and 10.17% in SR and SPL on HM3D. On the Gibson dataset, GAMap attained an SR of 85.7% and an SPL of 55.5%, which marks a substantial improvement over methods that do not utilize locomotion and semantic training [17] , with a 23.7% increase in SR and a 37.0% increase in SPL.5 Ablative StudyQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We include it in the appendix Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
effectiveness of affordance and geometric-part attributes: 
 We analyzed the effectiveness of the number of affordances and geometric parts in our proposed method. Ablation studies were conducted by varying N a and N g to assess performance gain versus time loss, as well as the contribution of each attribute type. We evaluated on the official mini-validation split of the HM3D dataset.The heatmap, as shown in Figure 3 , illustrates the SR with different combinations of N a and N g . The color changes from dark purple to yellow indicate the increased percentage in SR. Red sold and dashed lines with labels indicate the time cost associated with each combination of N a and N g . Increasing the number of geometric parts (N g ) from 0 to 3 results in a significant improvement in SR across all levels of N a . For example, when N a = 0, increasing N g from 0 to 3 raises the SR more than increasing N a from 0 to 3 when N g = 0. This demonstrates the substantial contribution of geometric parts to navigation performance. To more clearly demonstrate the specific impacts of N a and N g on various performance metrics, we have converted Figure 3 to Table 6 in the Appendix.Similarly, increasing the number of affordance attributes (N a ) also improves SR, though the effect is slightly less pronounced than that of geometric parts. The best performance is achieved when both N a and N g are maximized, with both set to 3. This suggests a synergistic effect, where the combination of both attributes leads to optimal navigation performance. However, it requires an 18.2% increase in time cost. We analyzed the effectiveness of different scaling levels (L). Figure 4 presents changes in SR and SPL when using different numbers of scales and the time required for processing on the mini-validation split of HM3D.Increasing the number of scaling levels from 1 to 4 leads to notable changes in both SR and SPL. At the highest scale level of 4, the SR improves by approximately 10%, and the SPL increases by about 20% than L = 1. However, there is a trade-off between performance improvement and time cost. The time required increases with the number of scales, as indicated by the red line in the figure. The time cost starts at 0.66 seconds for a single scale and rises progressively, reaching 0.78 seconds at the fourth scale level. This increase in time cost suggests that while higher scaling levels improve both SR and SPL, they also demand more computational resources and time. This emphasizes the need to balance performance and computational efficiency when determining the optimal scaling level for practical applications.We analyzed the proposed patch-based method by comparing it with the gradient-based method. The visualization of the GA score for different methods is shown in Figure 5 , which illustrates the GA score of the armrest, backrest, and seat attributes of a target object chair. Observations indicate that the gradient-based method often attends to irrelevant areas. For example, the ceiling of the room has a higher GA score, which is incorrect. In contrast, the patch-based method more accurately focuses on relevant areas, such as the armrest, backrest, and seat of the chair, validating its effectiveness over the gradient-based method. Moreover, as observed from Table 2 , the gradient-based method is also slower than the patch-based method. One reason for this is that the gradient-based method requires back-propagation of the gradient. Based on different methods, we further explored the effectiveness of various pretrained encoders, as shown in Table 2 . We compared SR and computation time required by three types of encoders: CLIP (ours), BLIP, and BLIP-2. Although using more powerful encoders such as BLIP and BLIP-2 leads to better performance, they require significantly more time than CLIP, while the performance gain is limited. This trade-off makes it less valuable for us to use a BLIP-based encoder.Backrest Armrest Seat Gradient Based Patch BasedMoreover, we analyzed different ways of aggregating GA scores using patch-based methods. Given the multiple levels of patches, we aimed to find the best method to aggregate the final GA score from multilevel GA scores.We compared "Max Value" and "Average Value" (ours). In the "Max Value" method, the maximum GA score across different scale levels is used. The "Average Value" method, which we propose, calculates the average GA score across levels. Note that the gradient-based method directly gives the score, so it is not analyzed. As shown in Table 2 , using the average to aggregate the score gives us the best performance. Different methods of updating GA scores and their impact on navigation performance are shown in Table 3 . In the "Replacement" method, the previous value is disregarded and overwritten with the new one. The "Average" method calculates the new value as the average of the previous and current values. Our approach, "Max", retains the maximum scores between the previous and new values, which memorizes the most salient score in a specific direction, as the agent could observe an object from different perspectives during exploration, thus potentially finding the optimal perspective. Our findings indicate that the "Max" method consistently enhances performance compared to the other two methods across all three datasets. In Table 3 , the "Max" method achieves SR of 50.25% and SPL of 25.3%. The "Average" method results in a 3% decrease in SR and a 10% decrease in SPL, indicating a moderate impact on performance. The "Replacement" method shows the most significant performance drop, with a 4% decrease in SR and a 17% decrease in SPL. These results highlight that the "Max" method is the most effective in maintaining and enhancing navigation performance, as it better captures and retains the most relevant object attributes from different perspectives.We evaluated the effectiveness of the proposed part and affordance guidance navigation by analyzing three types of errors: detection error, planning error, and exploration error. 1) Detection error happens when the agent either misses the goal or incorrectly believes it has detected the goal. 2) Planning error arises when the agent either recognizes the target but cannot reach it or gets stuck without spotting the goal, reflecting the path-planning ability of the system. 3) Exploration error occurs when the agent fails to see the goal object due to issues other than planning or detection, assessing its ability to approach the goal. Table 4 shows the comparison of these errors using our proposed method versus using categorical semantic information as the guidance for navigation. Note that our proposed method significantly decreases the errors in all three categories. Furthermore, we conduct an experiment to verify whether VLM [47] models can directly capture enough multi-scale information. We randomly selected a scene and compared the ability of GPT-4V and our proposed CLIP with multi-scale scoring method to identify the target object. As shown in Figure 6 , we input an image with a sofa located in a distant corner as the target object and compared the subsequent movement trajectories of the two algorithms. As illustrated in Figure 6 , our method successfully captures the small sofa back in the far corner, leveraging geometric parts and affordance attributes to guide the exploration process. In contrast, GPT-4V failed to identify the object.Figure 6 : The top row of images shows our proposed method, where the multi-scale approach effectively captures objects at all scales, such as the sofa back in the background. The bottom row of images shows the results of GPT-4V.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We will release our code upon the decision of the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this work, we introduced the Geometric-part and Affordance Maps (GAMap) for zero-shot object goal navigation, leveraging geometric parts and affordance attributes to guide exploration in unseen environments. Our method employs LLMs for detailed attribute inference and VLMs for multi-scale scoring, capturing object intricacies at various scales. Comprehensive experiments on HM3D and Gibson datasets exhibit significant improvements in SR and SPL over previous methods. These results highlight the effectiveness of our approach in enhancing navigation efficiency without any task-specific training or fine-tuning.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We follow the standard testing details which are stated in the paper.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The standard evaluation not include the error bar.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: This is included in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper has no such risk. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All assets are referenced.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: As shown in Table 5 , the key difference between our method and other methods is that we leverage geometric parts and affordance information to represent the environment, in addition to using objectlevel category information as previous methods do. Furthermore, we utilize multi-scale feature representation to capture local features, enhancing the overall accuracy and robustness.Table 5 : The differences between our work and existing methods.Method Mapping Multi-Scale Zero-shot Training Locomotion Semantic SemExp [2] Categorical × × ✓ ✓ ZSON [30] Categorical × × ✓ ✓ PixNav [9] Language-Grounded × ✓ ✓ × VLFM [16] Language-Grounded × ✓ ✓ × PONI [44] Categorical × ✓ × ✓ L3MVN [19] Categorical × ✓ × ✓ CoW [32] Categorical × ✓ × × ESC [3] Categorical × ✓ × × VoroNav [17] Categorical × ✓ × × GAMap Affordance+Gemoetric ✓ ✓ × ×In our experiment, we adopt GPT-4V as the LLM to generate the geometric and affordance attributes. We set N a to 1 and N g to 3 for the experiments on the HM3D and Gibson datasets. For the partition process, we use three scaling levels in all our experiments: the first level is the original image, the second level has 4 equal-sized patches, and the third level has 16 equal-sized patches. We use CLIP as the pre-trained visual and text encoder. Following the standard evaluation protocol [19] , we use 2000 episodes on the validation split of HM3D to report the results. Similarly, we follow this method [19] to produce the results on the Gibson dataset. We use a Titan XP GPU for the experiment evaluation, and the entire evaluation process takes around 44 hours.We provide the quantitative result for Figure 3 in the main paper. Time is measured in seconds. In this section, we visualize the navigation paths on both the Gibson and HM3D datasets, as shown in Figures 7 and 8 . Part of the visualization code is adapted from L3MVN [19] .To validate the efficiency of our method, we compare the FPS of our method to SemExp [2] , L3MVN [16] , and VLFM [19] in navigation tasks on the HP3D dataset. The experimental results are shown in Table 7 . It can be observed that SemExp has the highest FPS, indicating the fastest processing speed. This is because it uses a detection head and does not employ a foundation model. However, SemExp has the lowest SR and SPL, indicating that despite its fast processing speed, it performs poorly in navigation accuracy and path efficiency. In contrast, L3MVN has the second-highest FPS as it uses a lightweight foundation model. Although its processing speed is not as fast as SemExp, it shows improved navigation accuracy and path efficiency, achieving an SR of 76.1% and an SPL of 37.7%. On the other hand, VLFM has a lower FPS of only 2, but it significantly improves SR and SPL, reaching 84.0% and 52.2%, respectively. This indicates that although VLFM has a slower processing speed, it has considerable advantages in navigation accuracy and path efficiency. Our model has the same FPS as VLFM, both at 2, but further improves SR and SPL, reaching 85.7% and 55.5%, respectively. This demonstrates that our method maintains high navigation accuracy and path efficiency while providing comparable processing speed to VLFM. These experimental results verify that our proposed method achieves a good balance between time and accuracy. In our real-world experiment, we will evaluate the performance of four zero-shot object goal navigation algorithms, including L3MVN [19] , COW [32] , ESC [3] , and VLFM [16] , within a standard indoor apartment environment consisting of two bedrooms, two bathrooms, one kitchen, and one living room. The experiment adopts a four-wheeled robot. Specifically, we use a JetAuto-Pro from Hiwonder equipped with an Intel Realsense D435i camera as our robot agent to navigate the environment and locate specific target objects, including a bed, toilet, table, sofa, and chair, without any prior knowledge of their locations. To ensure a fair comparison, the starting position of the robots was kept consistent across all trials for each algorithm. The video demo can be found on our project page: https://shalexyuan.github.io/GAMap/. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: This paper received divergent initial opinions with two reviewers in support of acceptance and two negative reviewers.  The main concerns by the negative reviewers revolved around narrowness and reproducibility of the contribution, missing time cost comparisons for the method and baselines, missing comparisons to prior work using VLMs for navigation, and missing experiments with real-world robot navigation.

The rebuttal responded to reviewer concerns, and one initially negative reviewer raised their opinion to accept.  Another negative reviewer acknowledged that some of their concerns were alleviated but remained opposed to acceptance citing the lack of convincing real-world robot experiments as the rationale.  That concern was not shared by any of the other reviewers, and in fact the authors do describe a real-world experiment in their rebuttal (albeit at a small scale).  Thus, the AC does not find a basis to overrule the reviewer consensus in favor of acceptance.  The AC recommends acceptance and strongly encourages the authors to incorporate clarifications and revisions to improve the final manuscript.

Reviews: review: paper_summary: paper_summary: The paper proposes Geometric and Affordance Maps to tackle zero-shot object-goal navigation. It focuses on how to leverage VLMs to navigate toward objects of unfamiliar categories without prior training. The authors use GPT-4 to generate multiple attributes that describe affordance and geometry, and acquire attribute embeddings by CLIP text encoder. Besides, they propose a multi-scale scoring approach which divides each RGB image into multiple patches processed by CLIP visual encoder, to acquire multi-scale visual embeddings. Cosine similarity can be computed between those embeddings, resulting in a map used for navigation (e.g., applying Fast Marching Method based on the map). Experiments on HM3D and Gibson demonstrate improvements in success rates and success weighted by path length (SPL). The authors also provide an analysis on error pattern, which shows that detection error (missing the target object) is significantly reduced.
strength_weakness: strength_weakness: 1. The overall approach is limited by LLMs and VLMs. The analysis in Sec. 5.9 (Table 4) implies that the improvement mostly results from the reduced detection error. Since off-the-shelf LLMs and VLMs are used, the technical contribution of the paper seems to be limited by this fact. Although geometric and affordance attributes are proved effective in Figure 3, the conclusion might be limited to the specific VLM used in this paper, namely CLIP.
2. It is better to replace Figure 3 with a table. It is not easy to tell the exact effect of $N_a$ and $N_g$.
3. Some sentences in Sec. 5.1 are with typos (perhaps in a rush). For example, "Gibson is generated by Al-Halah et al. [36] for agents with configuration A". And the citations in L214 are messy.
questions: questions: 1. Can the authors compare the proposed method with PIVOT[1], which can also leverage VLMs to tackle robotic navigation? PIVOT focuses more on VQA, while this paper focuses on the similarity between the text and visual embeddings generated by CLIP. CLIP may not capture enough details, and thus the authors may need to introduce a multi-scale scoring approach. It is interesting to see whether VLM models can directly capture enough multi-scale information.
2. Can the authors also compare the time cost with baselines, like SemExp and VLFM?


[1] PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
limitations: limitations: The authors have addressed the limitations in the appendix.
paper_summary: paper_summary: The authors propose an algorithm for the object-goal navigation/exploration problem by using a 2D navigation map containing geometric and affordance scores for the target object. Geometric and affordance features for the target object are extracted by asking an LLM to list them for the object, and these words are then used with a multi-scale CLIP setup to get the resulting "GA" scores that are projected into a 2D map and accumulated over time.
strength_weakness: strength_weakness: * Narrowness of contribution: Projecting CLIP-like open-vocab scores into a 2D navigation map has been done in earlier work like VLFM [17]. The novelty here appears to be that you ask an LLM for multiple (geometric, affordance) properties of the object to query CLIP with. 

* Motivation of baselines: The results only seem about 1% better than the recent VLFM paper, which you dismiss because it contains "locomotion" training. However, no mention is made how your robot does motion planning. Since your results measure path length, you have to use something to get to a high-scoring frontier without colliding. The digitized environments are also just referred to as "data sets", but there has to be a simulator running on top of them to simulate some type of robot moving through the environment. Some details on this may help motivate why you dismiss VLFM. 

* Reproducibility: The results are all with (and contributions hinges on) GPT4, but the back-end of GPT4 changes substantially over time and there is afaik no way to run a specific version of GPT4 (only major versions). This means the numbers in the paper may not be reproducible at the time of publication. It would be helpful to also include results with a specific version of an open-source LLM to at least have something reproducible.

--------
After rebuttal: The authors ran more experiments which addressed weaknesses #2 and #3. The quantitative improvements weren't as good as originally claimed but good enough. They even threw in a real robot experiment (video would also have been nice!)..The contribution is still a bit narrow but given the SOTA results on ObjNav it merits publication..
questions: questions: * Please clarify what kind of path planning and robot model you use, and in more detail why VLFM is not comparable.

* Even assuming the VLFM "locomotion" is not comparable, could you not just use their 2D map component with your path planning? Both are 2D maps with scores.
limitations: limitations: Adequate except for the points above.
paper_summary: paper_summary: The paper presents an approach for the task of zero-shot object goal navigation. The approach first generates a geometric and affordance description of the goal object. This description is then matched with the visual embedding of the RGB frame (at multiple scales) in the CLIP embedding space to compute a similarity score. They project this score on an orthographic 2D map. During navigation, the agent explores the frontiers/unexplored regions with the highest score. They demonstrate the efficacy of their method on HM3D and Gibson object-goal benchmark.
strength_weakness: strength_weakness: - The idea of using attributes and longer descriptions for objects instead of just object label has been well studied [A, B] and I would have expected the authors to make connections to this popular line of work on effectively utilising CLIP representations. 
- I also thought that the 23.7% increase in SPL, and 37.0% in SPL claim is quite misleading specially because the only difference between them and VLFM is that VLFM uses a learned policy for navigation. It has been well established that using FMM for navigation works well enough for indoor navigation

[A] Visual Classification via Description from Large Language Models, ICLR 2023. 
[B] LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions, 2023
[C] Navigating to Objects in the Real World, 2022.
questions: questions: - The authors argue that comparison to VLFM is unfair because they use a trained policy compared to FMM to achieve high SPL on HM3D. Even then, they obtain better SPL on Gibson. Can the authors comment on why a trained policy will help HM3D, but not Gibson? 
- I didn't fully understand Figure 3. What does decrease in success rate mean? Is it relative or absolute decrease? Is it percentage or absolute points?
limitations: limitations: The authors don't discuss any limitation of their work in the manuscript. 
- Maybe the authors should add a small section discussing potential limitations such as lack of real-robot experiments in the paper. 
- assumptions about perfect odometry / localization to build a map. 

Not a limitation, but a suggestion to improve the manuscript: 
- I thought that the writing could be clearer. Some of the sentences were overly complicated. For example Line 42-43 - "proposed method initiates with the LLM inferring the object’s geometric part attributes and potential
affordances attributes, delving into a detailed understanding of the object’s physical structure and
functional properties". The paper will benefit from more succinct and clearer writing.
paper_summary: paper_summary: The paper is studying the problem of zero-shot object goal navigation. To solve the problems in traditional approaches, the authors propose a novel method, named Geometric and Affordance Maps (GAMap), which incorporates both geometric and affordance attributes as navigation guidance. A multi-scale scoring approach is proposed to cope with objects at different scales. Experiments on HM3D and Gibson datasets show the effectiveness of the proposed method for the metrics of success rates and success weighted by path length.
strength_weakness: strength_weakness: 1. Although the authors have demonstrated the difference of their work compared with previous ones, I am still not convinced about the key novelty of the work. In terms of organisation of the paper, it would be better if the authors could provide a table to better compare the difference of their work with existing works. 

2. There seems no real world experiments. I am not fully convinced about the effectiveness of the proposed method unless results of real robot experiments are provided.

3. The authors have not removed the checklist instruction blocks.

4. In Line 214, there is an issue with the citations. The paper needs further proofread.
questions: questions: It would be good if the authors could address my concerns in the weakness part. I will make my final decision after reading the authors' rebuttal and other reviewers' comments.
limitations: limitations: The authors have discussed about limitation in appendix.

Comments: title: Thanks!
In response to fGDQ’s further feedback
In response to KUEn’s further feedback
Response to Author Rebuttal
comment: Thank the authors for the response. My concerns are resolved. I will keep my rating or raise the score based on discussion with other reviewers and AC.
Thanks for providing the clarifications. I am happy with your responses. Since most of my questions and weaknesses were clarifications, I will stick to my original rating.
Thank you for your further feedback and suggestions. We understand your concerns regarding the real-world experiments and the importance of validating our proposed method across more diverse scenarios and through comparisons with existing methods. In fact, since we first received your suggestion about real-world experiments, we have been working diligently to enhance our experiments. Due to the one-week rebuttal period, we only presented a demo in one real-world scenario during this stage. Before the camera-ready deadline, we will definitely include the completed real-world experiments.

In our real-world experiment, we will evaluate the performance of four zero-shot object goal navigation algorithms, including L3MVN, COW, ESC, and VLFM, within a standard indoor apartment environment consisting of two bedrooms, two bathrooms, one kitchen, and one living room. The experiment involved both a Unitree B1 quadruped and a small, four-wheeled robot, each equipped with an RGB-D sensor, to navigate the environment and locate specific target objects, including a bed, toilet, plant, table, sofa, chair, trash can, TV, and remote control, without any prior knowledge of their locations. To ensure a fair comparison, the starting position of the robots was kept consistent across all trials for each algorithm. The experiment was conducted ten times, with performance measured based on success rate to determine the most effective navigation method.

If you have any additional suggestions regarding the experimental setup, we would be glad to incorporate them. We will make sure all these results will be included in the camera-ready version of the paper.

We hope these steps will address your concerns and provide a more comprehensive validation of our method's effectiveness, particularly in light of your feedback on the importance of robust real-world testing across multiple scenarios and comparisons with existing methods. We appreciate your thoughtful review and will take your feedback into account as we refine our research.
Thank you for your detailed review and thoughtful suggestions. We appreciate the time you’ve taken to provide feedback on our work, and we are pleased that the additional experiments have strengthened the paper. Below, we summarized your feedback into three main comment points and provide point-by-point responses to each.

# Response to Comment 1：

Thank you for the suggestion. We agree that including the VLFM + motion planning results will provide a more direct and fair comparison. We will include these results in the revised version of the paper to ensure a more "apples to apples" comparison, rather than just stating that VLFM is not comparable. This addition will help to better illustrate the differences and improvements brought by our approach.

# Response to Comment 2：

Thank you very much for your willingness to raise the score. We acknowledge that the percentage improvements over VLFM are not very large, but our results still push the state of the art forward, and as you mentioned, they seem big enough to merit publication.  Additionally, we appreciate your recognition of the novelty of the G+A approach. As mentioned in our response to other reviewers, we tested other models like BLIP and BLIP-2 in our ablation study, and the results indicate that applying our method to more advanced VLMs leads to further improvements in accuracy. Agreeing with your suggestions for our future work, we plan to integrate more advanced VLMs to further improve our robust navigation approach.

# Response to Comment 3：

Our intuition is that this training helps the PointNav policy recognize patterns, make predictions, and optimize decisions that are more aligned with the ObjNav objective. In contrast, off-the-shelf motion planning algorithm  ​​, which hasn’t been exposed to as many scenarios, might not perform as effectively. For example, in the VLFM paper, PointNav was trained on the HM3D dataset, enabling it to learn locomotion tailored to the dataset’s specific scenarios. This specialization explains its strong performance in subsequent navigation tasks within HM3D.
Thank the authors for the rebuttal. I am still not fully convinced by the real world experiment which I think is very important to validate the effectiveness of the proposed method. Only one experiment in one scenario without comparing to existing methods cannot convince me of that. Therefore, I am still keeping my original rating.
I thank the authors for doing the extra experiments, they make the paper stronger. I suggest you include the VLFM + motion planning results to make it more of an apples to apples comparison rather than just saying that VLFM is not comparable. 

I will raise my score pending reviewer discussion. That said, the improvements compared to VLFM + motion planning aren't very large (3-4% SR and 10-15% SPL) but they seem big enough to merit publication. The G+A approach is somewhat novel, although the other reviewers are right to question how specific to it is to CLIP. The entire affordance part also had rather marginal improvements as seen in the ablations (going from 1 -> 3 had no effect). Based on how limited CLIP language understanding tends to be, I had also not expected this to work well. If it is a good idea, maybe it will work better with some future VLM. 

As a tangent, it is not obvious what VLFM gained by training a custom PointNav policy instead of using an off-the-shelf motion planning algorithm like the authors. The VLFM paper does not seem to be clear on that. Any standard motion planner should be able to provide a near optimal (shortest) path for a simple robot like here (I assume it is point or sphere). It is curious that the original VLFM w/ trained-PointNav actually performed better than the version with the (optimal?) motion planner by the authors. Do the authors have any intuition of why it is more suitable for the ObjNav objective (or conversely why theirs is less so)?