Title: GAMap: Zero-Shot Object Goal Navigation with Multi-Scale Geometric-Affordance Guidance
Venue: NeurIPS 2024
Accepted: yes
Keywords: zero-shot object goal navigation
Abstract: Zero-Shot Object Goal Navigation (ZS-OGN) enables robots to navigate toward objects of unseen categories without prior training. Traditional approaches often leverage categorical semantic information for navigation guidance, which struggles when only partial objects are observed or detailed and functional representations of the environment are lacking. To resolve the above two issues, we propose \textit{Geometric-part and Affordance Maps} (GAMap), a novel method that integrates object parts and affordance attributes for navigation guidance. Our method includes a multi-scale scoring approach to capture geometric-part and affordance attributes of objects at different scales. Comprehensive experiments conducted on the HM3D and Gibson benchmark datasets demonstrate improvements in Success Rates and Success weighted by Path Length, underscoring the efficacy of our geometric-part and affordance-guided navigation approach in enhancing robot autonomy and versatility, without any additional task-specific training or fine-tuning with the semantics of unseen objects and/or the locomotions of the robot.

GAMap: Zero-Shot Object Goal Navigation with Multi-Scale Geometric-Affordance Guidance

Abstract

Zero-Shot Object Goal Navigation (ZS-OGN) enables robots or agents to navigate toward objects of unseen categories without object-specific training. Traditional approaches often leverage categorical semantic information for navigation guidance, which struggles when only objects are partially observed or detailed and functional representations of the environment are lacking. To resolve the above two issues, we propose Geometric-part and Affordance Maps (GAMap), a novel method that integrates object parts and affordance attributes as navigation guidance. Our method includes a multi-scale scoring approach to capture geometric-part and affordance attributes of objects at different scales. Comprehensive experiments conducted on HM3Dand Gibson benchmark datasets demonstrate improvements in Success Rate and Success weighted by Path Length, underscoring the efficacy of our geometricpart and affordance-guided navigation approach in enhancing robot autonomy and versatility, without any additional object-specific training or fine-tuning with the semantics of unseen objects and/or the locomotions of the robot. Our project is available at https://shalexyuan.github.io/GAMap/ .

1 Introduction

Zero-Shot Object Goal Navigation (ZS-OGN) is a pivotal research domain in embodied AI and robotics, enabling robots to navigate towards the objects of unseen categories without training or fine-tuning on these objects [1, 2, 3, 4, 5]. This capability is crucial for real-world robots, such as home service robots and blind guiding robots, allowing them to interact with diverse objects in real-world scenarios, thereby enhancing their autonomy and versatility.

Prior works on ZS-OGN either leverage deep neural networks to directly map RGB-D observations to actions learned from paired training data [6, 7, 8, 9, 10, 11] or utilize map-based navigation methods [12, 13, 14, 15, 16, 17, 3]. However, deep neural network approaches often struggle due to their dependence on extensive annotated data, resulting in poor generalization to unseen environments [18, 16], while map-based navigation methods instead offer an alternative. Map-based navigation methods track categorical semantics and topological information observed by the agent to select promising exploration locations [17]. With the advent of foundation models, the studies [3, 17, 19] have exploited the reasoning capabilities of Large Language Models (LLMs) to strategically select

Fail to detect chair,

Locate chair, move towards it

explore selected frontier

Pass by the target object, fail to find chair

waypoints by analyzing commonsense, such as object co-occurrence relationships, to navigate robots towards the target. However, LLM-based approaches require converting visual and semantic information into categorical descriptions, which leads to a loss of spatial and visual information [9]. Vision Language Models (VLMs) enhance semantic reasoning capabilities, but still rely on maps that encompass only categorical information [16]. The primary limitation of exclusively relying on categorical information is that such maps treat objects as monolithic entities, disregarding local geometric features. This becomes particularly problematic when only the target object is partially observed, leading to incorrect categorical information and potential errors in waypoint selection.

We argue that using a categorical map for robot exploration is suboptimal, as it discards intricate geometric details and functional representations of the environment, as illustrated in Figure 1. Drawing inspiration from human cognitive processes - where distinctive geometric parts are often identified first when locating an object in an unfamiliar environment [20, 21] - we propose Geometric-part and Affordance Maps (GAMap), a zero-shot approach, for the geometric parts and affordance attribute driven semantic navigation to explore and find the target object in an unseen environment. Specifically, given a target object, our proposed method starts by using an LLM to infer the object's geometric parts and potential affordance attributes, aiming at providing a detailed understanding of both the object's physical structure and its functional properties. Given depth observations, GAMap maintains a 2D projection of obstacles and explored areas. Instead of relying on object detection and prompt engineering of LLMs to select the next area to explore, our approach employs a pre-trained CLIP [22] to score observations based on their similarity to the reasoned geometric parts and affordance attributes, guiding the exploration process. To construct the proposed GAMap, which requires obtaining scores for geometric parts at different scales, we further propose a Multi-scale Geometric and Affordance score (GA score). Such an integration addresses a notable limitation in existing approaches that compute similarity at a single scale, which usually results in the oversight of fine-grained details of an object, as such details are potentially essential for an accurate identification of geometric parts or affordance attributes of objects with different sizes.

Our proposed method is evaluated on HM3D [23] and Gibson [24] benchmarks and achieves significant improvements in Success Rate (SR) of 26.4% on HM3D [23] and 23.7% on Gibson [24] compared to previous approaches. Additionally, we achieve substantial gains in Success weighted by Path Length (SPL) of 37% on Gibson, highlighting the efficiency and effectiveness of our proposed geometric parts and affordance guided navigation. The contributions of our method are mainly summarized as follows:

We propose a novel Geometric-part and Affordance Map (GAMap) for ZS-OGN using object part and affordance attributes as guidance. To the best of our knowledge, this is the first work to study the integration of these attributes in ZS-OGN.

Recognizing that geometric parts and affordance attributes often relate to multiple scales of an object, we propose a Multi-scale Geometric and Affordance score, which allows GAMap to be constructed in real-time, better capturing these attributes at different scales.

We achieve state-of-the-art performance on two navigation benchmark datasets without any training or fine-tuning with the semantics of unseen objects and/or the locomotions of the robot, which demonstrates the effectiveness of our method in unseen environments.

2 Related Work

Semantic Mapping. In the context of object goal navigation, it is crucial to transform observations into structured information for effective decision-making. Frontier-based methods [3, 16, 19] utilize categorical semantic information near frontiers to select exploration areas. Additionally, graph-based mapping methods [17, 25, 18] predict waypoints from RGB-D images or simplified maps to create topological representations of the environment. Most of the aforementioned works rely on semantic segmentation or object detection to build semantic maps, which are constrained by the pre-defined semantic classes and thus fail to capture the full semantic richness of environments [18, 26]. To overcome these limitations, recent approaches like VLMaps [18] have introduced open-vocabulary semantic maps, enabling natural language indexing and expanding the scope of semantic mapping. In addition, previous works [27, 28] attempt to utilize attributes and long descriptions for object perception. While these methods have advanced the navigation field, they often overlook object parts, treating objects as monolithic entities and leading to errors when these objects are partially observed. Inspired by human cognitive processes [21, 20], where distinctive geometric parts are identified first in unseen environments, we propose Geometric-part and Affordance Maps (GAMap). Unlike previous methods focusing solely on categorical information, GAMap integrates geometric parts and affordance attributes, providing a richer and more functional representation of the environment.

Zero-shot Object Goal Navigation. In the context of object goal navigation, the aim is to efficiently explore a new environment while searching for a target object that is not directly visible. Previous research relies heavily on visual context via imitation [29, 6] or reinforcement learning [7, 8, 9] to guide robots. These approaches often require extensive data collection and annotation for training, which limits their practical application in real-world environments. Thus, the focus in object goal navigation has been shifting towards zero-shot object navigation, which aims to equip robots with the ability to adapt to unseen objects and environments without the need for training [12, 30, 13, 14]. Clip-Nav [31] utilizes CLIP [22] to execute vision-and-language navigation in a zero-shot scheme, whilst CoW [32] employs CLIP for object goal navigation. Recently, Frontier-based Exploration (FbE) [33] is widely adopted in navigation by moving the robot to the frontier between known and unknown spaces [32, 34, 35, 36, 37], leading to promising performance compared to learning-based exploration methods [38, 39]. More recently, ESC [3] leverages the reasoning ability of LLMs to select frontiers using pre-defined rules. Chen et al . [15] explore frontier selection by jointly considering the shortest path to frontiers and the relevance scoring between objects for exploration. To enable more robust and reliable exploration and waypoint selection, Wu et al . [17] propose a Voronoi-based scene graph for waypoint selection. Unlike the above methods that use the reasoning ability of LLMs to select frontiers, VLFM [16] introduces a value map to score frontiers based on the categorical similarity between the observation and the target object. In contrast to prior work, for the first time, we explore robot navigation using geometric parts and affordance attributes as guidance. This approach integrates detailed geometric parts and functional properties of objects, offering a more comprehensive strategy for navigation.

3 Method

We first formalize the ZS-OGN problem in Section 3.1. Then, we detail our method, as shown in Figure 2, from four phases: attribute generation in Section 3.2, multi-scale attribute scoring in Section 3.3, GAMap generation in Section 3.4, and exploration policy in Section 3.5. Initially, our method generates geometric parts and affordance attributes for the target object. During the exploration, the method computes a multi-scale attribute score from the RGB observations collected by the agent. These scores are then mapped onto a 2D geometric parts and affordance map, which is pivotal in guiding the exploration process. Subsequently, the agent selects the location with the highest score for further exploration.

Depth Observation

RGB Observation

Target Object:

'

Chair

'

Affordance Prompt

Multi-scale Visual Embeddings

CLIP

Text

Encoder

Cosine

Similarity

GA Scores

GA Map

Average

CLIP

Visual

Encoder

Partition

𝑒

!

𝑒

"

Attribute Embeddings

𝑒

#

!

$#

"

…

𝑁

!

+𝑁

"

3D Local Point Cloud

Projection

vo.o

→

0

3.1 Problem Formulation

In ZS-OGN, the robot must navigate to a target object g i that has never been encountered before in an unseen environment s i , without any training on g i and s i . A navigation episode can be defined as E i = { g i , s i , p 0 } , where p 0 denotes the robot's initial location, the subscript i refer to the i th episode. The robot receives a color image I t , a depth image D t , and its pose, i.e ., position ( x t , y t ) and orientation θ t , at each exploration step t . We denote these readings as an observation O t = { I t , D t , x t , y t , θ t } . The agent accumulates pose readings over time to determine its relative position p t . Based on the readings at each step, the robot needs to select an action a t from the action space. A navigation episode is marked as successful if the robot executes the STOP action within a pre-defined distance to the target object. In this work, we approach the navigation task as a sequence of decisions made by the robot. The process starts at the initial time step t = 0 and ends at the final time step T . This final step is either when the robot executes the STOP action or when a pre-defined maximum number of exploration steps is reached.

3.2 Attribute Generation

We focus on two types of attributes essential for object recognition: Affordance and Geometric-part attributes. Affordance attributes refer to the potential actions that an object facilitates [40], which are crucial for understanding how an agent might interact with different objects within an environment. Geometric-part attributes, on the other hand, describe the shape and spatial characteristics of an object, aiding in its visual identification and differentiation from other objects.

To extract these attributes, we employ an LLM to reason about the target object's characteristics. Specifically, we utilize GPT-4V [41] for the attributes generation. We initiate this process by setting the system prompt as: 'I am a highly intelligent question-answering bot, and I answer questions from a human perspective. ' Subsequently, we employ two tailored prompts to extract the desired attributes. For affordance attributes, we design prompt as: 'For the target object <target object g i >, please provide < N a > affordance attributes that to the most reflect its characteristics. ' to query N a number of affordance attributes. For geometric parts, the prompt is: 'Summarize < N g > geometric part visual features of the <target object g i > which are typically used to identify why it is a <target object g i >.' to get N g number of geometric attributes. Once the set of affordances attributes { A n } N a n =1 and geometric attributes { G n } N g n =1 have been identified, the next stage involves coupling these attributes with the agent's observations by computing Geometric-part and Affordance scores (GA scores) through multi-scale visual features, as detailed in the following.

3.3 Multi-scale Attribute Scoring

The key idea of Multi-scale Attribute Scoring is to quantify the relevance of observed areas' geometric and affordance characteristics in locating the target object. These scores determine the agent's subsequent exploration decisions, guiding it to the areas most likely to contain the target object.

To correlate the scoring with both the global frame and localized patches, ensuring that finer details and smaller components are effectively scored, we partition the observed image into patches of equal size across different scales. Specifically, given an RGB observation image I t , with height H and width W , the partitioning process is as follows. At level k , the image is partitioned into 4 ( k -1) equal parts with each patch at level k of size H 2 ( k -1) × W 2 ( k -1) . The patches at level k can be represented as: I k t = { P h,w | 1 ≤ h ≤ 2 ( k -1) , 1 ≤ w ≤ 2 ( k -1) } where P h,w denotes the patch located at the h th row and w th column of the partitioned image at level k . All patches from all levels are then resized to the same dimensions for further processing. We utilize CLIP [32] to calculate the visual embeddings for all patches from all levels. For each patch P k h,w at level k , its visual embedding v k h,w is computed using the image encoder of CLIP. Simultaneously, each attribute embedding e is computed using the text encoder of CLIP, given the attribute descriptions, { A n } N a n =1 and { G n } N g n =1 , generated from an LLM. For each patch P k h,w , we calculate the cosine similarity between its visual embedding v k h,w and the attribute embedding e to obtain a similarity score S k,e h,w as follows:

Due to the hierarchical partitioning of the image, we accumulate the scores across all scales for each pixel location and then take the average to obtain the final score for the image. Specifically, let L be the number of levels, and the accumulated score for the pixel at position ( p, q ) in the image I t is calculated by summing the scores from all levels and then averaging:

where h ( p, q, k ) and w ( p, q, k ) map the pixel position ( p, q ) in the image to the corresponding patch indices at level k . The scores are then used to generate a Geographic and Affordance Map (GAMap) for the explored area. The process for generating the GAMap is detailed in the next section.

3.4 Geographic and Affordance Map

At the core of our approach is the Geographic and Affordance Map. This map assigns a GA score to each pixel within the explored area to quantify the relevance of different regions in locating the target object, associating the areas with the highest values as the most promising for further exploration.

We define the GAMap at time step t as M t ∈ R ˆ H × ˆ W × C , where ˆ H and ˆ W are the dimensions of the 2D projection grid map, and C is the number of attributes, i.e ., C = N a + N g with N a and N g representing the number of attributes and parts, respectively. To construct the GAMap from the RGB-D observation I t and the depth data D t , we back-project every pixel from D t to reconstruct the point cloud following [18]:

where K is the intrinsic matrix of the depth camera, and D t ( p, q ) is the depth value of the pixel at the coordinate ( p, q ) . To transform the point cloud into the world coordinate frame, we use X world = T W · X , where T W is the transformation matrix from the camera coordinate to the world coordinate. The 3D points are then projected onto the ground plane to determine the corresponding positions on the grid map. We assume perfect calibration between the depth and RGB cameras, allowing us to project each image pixel's score to its corresponding grid cell in the map. Given that multiple 3D points may project to the same grid location, we retain the maximum value for each channel that falls into the same grid cell as the score of this cell:

where M t ( ˆ h, ˆ w,e ) represent the score of attribute e in the 2D grid at position ( ˆ h, ˆ w ) , S ( p, q, e ) is the score for the pixel at ( p, q ) for the attribute e , and cell M t ( ˆ h, ˆ w ) denotes the grid cell on the GAMap. When the robot moves to a new position, resulting in overlapping observations with the previously explored regions, the GA scores for each pixel in the overlapping region are updated. The updated GAMap is computed by taking the maximum of the current attribute score and the previous attribute score for each cell:

where M ( ˆ h o , ˆ w o , e ) is the GAMap constructed from the previous step, and the subscript o represents the overlapped grid cell.

3.5 Exploration Policy

To determine the next area to explore, the robot selects the region with the highest GA score, calculated as the average of all attribute channels. To enable efficient exploration, only areas near the frontier with the highest scores are selected. Once the area with the highest GA score is identified, a heuristic search algorithm, i.e ., the Fast Marching Method (FMM) [42], is employed to find the shortest path from the robot's current location to the selected area. The robot then generates the appropriate actions to navigate along this path. At each step, the GAMap is updated based on new observations. We repeat this process until the robot either identifies and reaches the target object or the episode ends.

4 Experiment

Datasets. HM3D [23] is a dataset consisting of 3D data from real-world indoor spaces, along with semantic annotations, serving as a foundational resource for the Habitat 2022 ObjectNav Challenge [23]. This comprehensive dataset includes 142,646 object instance annotations, organized into 40 distinct classes across 216 environments, covering a total of 3,100 individual rooms. We follow the validation settings from [3, 32] to evaluate our proposed method. Gibson [24] was developed by Al-Halah et al . [43]. The dataset comprises 5 validation scenes across 6 object categories, and we adhere to the standard evaluation protocol [44, 19, 30, 2] to use all 5 validation scenes for evaluation.

Metrics. Success Rate (SR, % ) [45] focuses on the agent's accuracy in reaching the designated target, where a higher value indicates better performance. SR is computed based on whether the robot successfully stops within 0.1m of the target object g i :

where K is the number of episodes, d ( g i , p T i ) is the distance between the target object g i and the robot's final position p T i in the i th episode, and 1 {·} is an indicator function. Success weighted by Path Length (SPL, % ) [45] evaluates success relative to the shortest possible path, normalized by the actual path taken by the agent, measuring the efficiency of the agent's success in reaching a goal, defined as:

where L i is the actual path length traveled by the robot in the i th episode and L ∗ i is the shortest possible path length to the target in the same environment.

4.1 Baselines

We compare our method against several ZS-OGN approaches, including the state-of-the-art methods: Random Exploration : takes random actions to explore the environment. Nearest FbE [33]: explores the environment by selecting the nearest frontier. SemExp [2]: utilizes a category semantic map and trains a local navigation policy for exploration. PixNav [9]: trains models for navigation by selecting pixels as intermediate goals. PONI [44]: uses potential functions to select frontiers for exploration. ZSON [30]: employs categorical information to train a model for object-based navigation tasks. CoW

[32]: uses CLIP for categorical information extraction and explores using the nearest frontier-based exploration. ESC [3]: utilizes a categorical semantic map and commonsense reasoning for target object exploration. L3MVN [19]: uses an LLM to reason about the next exploration area based on a trained detection head for semantic map construction. VLFM [16]: employs BLIP-2 [46] and categorical information of the target object to evaluate and select frontiers for exploration. VoroNav [17]: uses a Voronoi-based decomposition strategy for navigation. SemUtil [15]: considers the shortest path distance to frontiers and the relevance scoring between objects for exploration.

4.2 Results and Analysis

We compare our method with existing approaches across four categories: those utilizing both locomotion and semantic training [2, 30], those employing only locomotion training [9, 16], those using only semantic training [19, 44], and those that do not incorporate any training [3, 32, 17, 15]. Locomotion training involves learning-based methods for navigation, while semantic training requires training or finetuning a perception module to construct a semantic map. The results and comparisons are shown in Table 1.

On the HM3D dataset, our method achieved a SR of 53.1 % and a SPL of 26.0 % . This represents a significant improvement over the best method [17] that does not use locomotion and semantic training, with a 26.4% increase in SR. Although the SPL (26.0 % ) of our method is lower than that of VLFM (30.4 % ), this discrepancy can be attributed to the fact that VLFM's local policy planning is trained. Considering the different path planning methods adopted in our approach and VLFM, we construct a com-

0

1

2

3

N_g (Number of Geometric Attributes)

0

1

2

3

N_a (Number of Affordance Attributes)

SR and Time Cost

55.4

27.8

16.7

0.0

18.2

1.0

0.8

0.6

0.4

0.2

0.0

Decrease on Success Rate

parative experiment with VLFM. Specifically, we kept the VLFM value map generation process unchanged and replaced its path planning method with FMM instead of the trained policy. The detailed results are shown in the second last row in Table 1. The comparison reveals that our model outperforms VLFM-based mapping method by 4.32% and 10.17% in SR and SPL on HM3D. On the Gibson dataset, GAMap attained an SR of 85.7 % and an SPL of 55.5 % , which marks a substantial

improvement over methods that do not utilize locomotion and semantic training [17], with a 23.7% increase in SR and a 37.0% increase in SPL.

5 Ablative Study

5.1 Effectiveness of Affordance and Geometric-part Attributes

We analyzed the effectiveness of the number of affordances and geometric parts in our proposed method. Ablation studies were conducted by varying N a and N g to assess performance gain versus time loss, as well as the contribution of each attribute type. We evaluated on the official mini-validation split of the HM3D dataset.

The heatmap, as shown in Figure 3, illustrates the SR with different combinations of N a and N g . The color changes from dark purple to yellow indicate the increased percentage in SR. Red sold and dashed lines with labels indicate the time cost associated with each combination of N a and N g . Increasing the number of geometric parts ( N g ) from 0 to 3 results in a significant improvement in SR across all levels of N a . For example, when N a = 0 , increasing N g from 0 to 3 raises the SR more than increasing N a from 0 to 3 when N g = 0 . This demonstrates the substantial contribution of geometric parts to navigation performance. To more clearly demonstrate the specific impacts of N a and N g on various performance metrics, we have converted Figure 3 to Table 6 in the Appendix. Similarly, increasing the number of affordance attributes ( N a ) also improves SR, though the effect is slightly less pronounced than that of geometric parts. The best performance is achieved when both N a and N g are maximized, with both set to 3. This suggests a synergistic effect, where the combination of both attributes leads to optimal navigation performance. However, it requires an 18.2% increase in time cost.

5.2 Effectiveness of Different Scaling Levels

We analyzed the effectiveness of different scaling levels ( L ). Figure 4 presents changes in SR and SPL when using different numbers of scales and the time required for processing on the mini-validation split of HM3D.

Increasing the number of scaling levels from 1 to 4 leads to notable changes in both SR and SPL. At the highest scale level of 4, the SR improves by approximately 10%, and the SPL increases by about 20% than L = 1 . However, there is a trade-off between performance improvement and time cost. The time required increases with the number of scales, as indicated by the red line in the figure. The time cost starts at 0.66 seconds for a single scale and rises progressively, reaching 0.78 seconds at the fourth scale level. This increase in time cost suggests that while higher scaling levels improve both SR and SPL, they also

1

2

3

4

0

10

20

30

40

50

SR and SPL Values

46.2

48.6

50.25

50.67

21.4

23.1

25.3

25.6

Analyze on Number of Scales

SR

SPL

Time

0.66

0.68

0.70

0.72

0.74

0.76

0.78

Time

0.66

0.67

0.71

0.78

Scales

demand more computational resources and time. This emphasizes the need to balance performance and computational efficiency when determining the optimal scaling level for practical applications.

5.3 Effectiveness of Different Methods for Calculating GA Scores

We analyzed the proposed patch-based method by comparing it with the gradient-based method. The visualization of the GA score for different methods is shown in Figure 5, which illustrates the GA score of the armrest, backrest, and seat attributes of a target object chair. Observations indicate that the gradient-based method often attends to irrelevant areas. For example, the ceiling of the room has a higher GA score, which is incorrect. In contrast, the patch-based method more accurately focuses

on relevant areas, such as the armrest, backrest, and seat of the chair, validating its effectiveness over the gradient-based method. Moreover, as observed from Table 2, the gradient-based method is also slower than the patch-based method. One reason for this is that the gradient-based method requires back-propagation of the gradient.

Based on different methods, we further explored the effectiveness of various pretrained encoders, as shown in Table 2. We compared SR and computation time required by three types of encoders: CLIP (ours), BLIP, and BLIP-2. Although using more powerful encoders such as BLIP and BLIP-2 leads to better performance, they require significantly more time than CLIP, while the performance gain is limited. This trade-off makes it less valuable for us to use a BLIP-based encoder.

Moreover, we analyzed different ways of aggregating GA scores using patch-based methods. Given the multiple levels of patches, we aimed to find the best method to aggregate the final GA score from multilevel GA scores.

Backrest

Armrest

Seat

Gradient Based

Patch Based

We compared 'Max Value' and 'Average Value' (ours). In the 'Max Value' method, the maximum GA score across different scale levels is used. The 'Average Value' method, which we propose, calculates the average GA score across levels. Note that the gradient-based method directly gives the score, so it is not analyzed. As shown in Table 2, using the average to aggregate the score gives us the best performance.

5.4 Effectiveness of Different Methods for Updating GA Scores

Different methods of updating GA scores and their impact on navigation performance are shown in Table 3. In the 'Replacement' method, the previous value is disregarded and overwritten with the new one. The 'Average' method calculates the new value as the average of the previous and current values. Our approach, 'Max', retains the maximum scores between the previous and new values, which memorizes the most salient score in a specific direction, as the agent could observe an object from different perspectives during exploration, thus potentially finding the optimal perspective.

Our findings indicate that the 'Max' method consistently enhances performance compared to the other two methods across all three datasets. In Table 3, the 'Max' method achieves SR of 50.25 % and SPL of 25.3 % . The 'Average' method results in a 3% decrease in SR and a 10% decrease in SPL, indicating a moderate impact on performance. The 'Replacement' method shows the most significant performance drop, with a 4% decrease in SR and a 17% decrease in SPL. These results highlight that the 'Max' method is the most effective in maintaining and enhancing navigation performance, as it better captures and retains the most relevant object attributes from different perspectives.

5.5 Effectiveness of Geometric and Affordance Guidance Navigation

We evaluated the effectiveness of the proposed part and affordance guidance navigation by analyzing three types of errors: detection error, planning error, and exploration error. 1) Detection error happens

when the agent either misses the goal or incorrectly believes it has detected the goal. 2) Planning error arises when the agent either recognizes the target but cannot reach it or gets stuck without spotting the goal, reflecting the path-planning ability of the system. 3) Exploration error occurs when the agent fails to see the goal object due to issues other than planning or detection, assessing its ability to approach the goal. Table 4 shows the comparison of these errors using our proposed method versus using categorical semantic information as the guidance for navigation. Note that our proposed method significantly decreases the errors in all three categories.

5.6 Effectiveness of Multi-scale Approach

Furthermore, we conduct an experiment to verify whether VLM [47] models can directly capture enough multi-scale information. We randomly selected a scene and compared the ability of GPT-4V and our

proposed CLIP with multi-scale scoring method to identify the target object. As shown in Figure 6, we input an image with a sofa located in a distant corner as the target object and compared the subsequent movement trajectories of the two algorithms. As illustrated in Figure 6, our method successfully captures the small sofa back in the far corner, leveraging geometric parts and affordance attributes to guide the exploration process. In contrast, GPT-4V failed to identify the object.

GPT-4V

Time

6 Conclusion

In this work, we introduced the Geometric-part and Affordance Maps (GAMap) for zero-shot object goal navigation, leveraging geometric parts and affordance attributes to guide exploration in unseen environments. Our method employs LLMs for detailed attribute inference and VLMs for multi-scale scoring, capturing object intricacies at various scales. Comprehensive experiments on HM3D and Gibson datasets exhibit significant improvements in SR and SPL over previous methods. These results highlight the effectiveness of our approach in enhancing navigation efficiency without any task-specific training or fine-tuning.

References

Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural slam. In International Conference on Learning Representations , 2020.

Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems , 33:4247-4258, 2020.

Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, and Xin Eric Wang. Esc: exploration with soft commonsense constraints for zero-shot object navigation. In Proceedings of the 40th International Conference on Machine Learning , pages 42829-42842, 2023.

Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 7641-7649, 2024.

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations , 2023.

Haresh Karnan, Garrett Warnell, Xuesu Xiao, and Peter Stone. Voila: Visual-observation-only imitation learning for autonomous navigation. In International Conference on Robotics and Automation , pages 2497-2503. IEEE, 2022.

Gregory Kahn, Adam Villaflor, Bosen Ding, Pieter Abbeel, and Sergey Levine. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In IEEE International Conference on Robotics and Automation , pages 5129-5136. IEEE, 2018.

Jan Wöhlke, Felix Schmitt, and Herke van Hoof. Hierarchies of planning and reinforcement learning for robot navigation. In IEEE International Conference on Robotics and Automation , pages 10682-10688. IEEE, 2021.

Wenzhe Cai, Siyuan Huang, Guangran Cheng, Yuxing Long, Peng Gao, Changyin Sun, and Hao Dong. Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill. In IEEE International Conference on Robotics and Automation , pages 52285234. IEEE, 2024.

Dhruv Shah, Michael Robert Equi, Bła˙ zej Osi´ nski, Fei Xia, Brian Ichter, and Sergey Levine. Navigation with large language models: Semantic guesswork as a heuristic for planning. In Conference on Robot Learning , pages 2683-2699. PMLR, 2023.

Hao Huang, Shuaihang Yuan, Congcong Wen, Yu Hao, and Yi Fang. 3d-trans: 3d hierarchical transformer for shape correspondence learning. In International Conference on Automation, Robotics and Applications , pages 536-540. IEEE, 2024.

Qianfan Zhao, Lu Zhang, Bin He, Hong Qiao, and Zhiyong Liu. Zero-shot object goal visual navigation. In IEEE International Conference on Robotics and Automation , pages 2025-2031. IEEE, 2023.

Vishnu Sashank Dorbala, James F Mullen Jr, and Dinesh Manocha. Can an embodied agent find your 'cat-shaped mug'? llm-based zero-shot object navigation. IEEE Robotics and Automation Letters , 2023.

Qianfan Zhao, Lu Zhang, Bin He, and Zhiyong Liu. Semantic policy network for zero-shot object goal visual navigation. IEEE Robotics and Automation Letters , 2023.

Junting Chen, Guohao Li, Suryansh Kumar, Bernard Ghanem, and Fisher Yu. How to not train your dragon: Training-free embodied object goal navigation with semantic frontiers. In Robotics: Science and Systems , 2023.

Naoki Harrison Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, and Bernadette Bucher. Vlfm: Vision-language frontier maps for zero-shot semantic navigation. In Workshop on Language and Robot Learning: Language as Grounding , 2023.

Pengying Wu, Yao Mu, Bingxian Wu, Yi Hou, Ji Ma, Shanghang Zhang, and Chang Liu. Voronav: Voronoi-based zero-shot object navigation with large language model. arXiv preprint arXiv:2401.02695 , 2024.

Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In IEEE International Conference on Robotics and Automation , pages 10608-10615. IEEE, 2023.

Bangguo Yu, Hamidreza Kasaei, and Ming Cao. L3mvn: Leveraging large language models for visual target navigation. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3554-3560. IEEE, 2023.

Shimon Ullman. High-level vision: Object recognition and visual cognition . MIT press, 2000.

MGazzaniga, R Ivry, and G Mangun. Cognitive science . Springer, 2002.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2021.

Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 9068-9079, 2018.

Obin Kwon, Jeongho Park, and Songhwai Oh. Renderable neural radiance map for visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9099-9108, 2023.

Hao Huang, Shuaihang Yuan, CongCong Wen, Yu Hao, and Yi Fang. Noisy few-shot 3d point cloud scene segmentation. In IEEE International Conference on Robotics and Automation , pages 11070-11077. IEEE, 2024.

Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183 , 2022.

Songhao Han, Le Zhuo, Yue Liao, and Si Liu. Llms as visual explainers: Advancing image classification with evolving visual descriptions. arXiv preprint arXiv:2311.11904 , 2023.

David Silver, James Bagnell, and Anthony Stentz. High performance outdoor navigation from overhead data using imitation learning. Robotics: Science and Systems , 1, 2008.

Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. Advances in Neural Information Processing Systems , 35:32340-32352, 2022.

Vishnu Sashank Dorbala, Gunnar A Sigurdsson, Jesse Thomason, Robinson Piramuthu, and Gaurav S Sukhatme. Clip-nav: Using clip for zero-shot vision-and-language navigation. In Workshop on Language and Robotics at CoRL , 2022.

Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, and Shuran Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23171-23181, 2023.

Brian Yamauchi. A frontier-based approach for autonomous exploration. In Proceedings of IEEE International Symposium on Computational Intelligence in Robotics and Automation , pages 146-151. IEEE, 1997.

Shuaihang Yuan, Muhammad Shafique, Mohamed Riyadh Baghdadi, Farshad Khorrami, Anthony Tzes, and Yi Fang. Zero-shot object navigation with vision-language foundation models reasoning. In International Conference on Automation, Robotics and Applications , pages 501-505. IEEE, 2024.

Congcong Wen, Yisiyuan Huang, Hao Huang, Yanjia Huang, Shuaihang Yuan, Yu Hao, Hui Lin, Yu-Shen Liu, and Yi Fang. Zero-shot object navigation with vision-language models reasoning. arXiv preprint arXiv:2410.18570 , 2024.

Halil Utku Unlu, Shuaihang Yuan, Congcong Wen, Hao Huang, Anthony Tzes, and Yi Fang. Reliable semantic understanding for real world zero-shot object goal navigation. arXiv preprint arXiv:2410.21926 , 2024.

Shuaihang Yuan, Halil Utku Unlu, Hao Huang, Congcong Wen, Anthony Tzes, and Yi Fang. Exploring the reliability of foundation model-based frontier selection in zero-shot object goal navigation. arXiv preprint arXiv:2410.21037 , 2024.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. In International Conference on Learning Representations , 2019.

Austin Myers, Ching L Teo, Cornelia Fermüller, and Yiannis Aloimonos. Affordance detection of tool parts from geometric features. In IEEE International Conference on Robotics and Automation , pages 1374-1381. IEEE, 2015.

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

James A Sethian. A fast marching level set method for monotonically advancing fronts. proceedings of the National Academy of Sciences , 93(4):1591-1595, 1996.

Ziad Al-Halah, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17031-17041, 2022.

Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah, Jitendra Malik, and Kristen Grauman. Poni: Potential functions for objectgoal navigation with interactionfree learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18890-18900, 2022.

Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757 , 2018.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023.

Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. In Workshop on Vision-Language Models for Navigation and Manipulation at ICRA , 2024.

A Difference with Existing Works

As shown in Table 5, the key difference between our method and other methods is that we leverage geometric parts and affordance information to represent the environment, in addition to using objectlevel category information as previous methods do. Furthermore, we utilize multi-scale feature representation to capture local features, enhancing the overall accuracy and robustness.

B Experiment Setup

In our experiment, we adopt GPT-4V as the LLM to generate the geometric and affordance attributes. We set N a to 1 and N g to 3 for the experiments on the HM3D and Gibson datasets. For the partition process, we use three scaling levels in all our experiments: the first level is the original image, the second level has 4 equal-sized patches, and the third level has 16 equal-sized patches. We use CLIP as the pre-trained visual and text encoder. Following the standard evaluation protocol [19], we use 2000 episodes on the validation split of HM3D to report the results. Similarly, we follow this method [19]to produce the results on the Gibson dataset. We use a Titan XP GPU for the experiment evaluation, and the entire evaluation process takes around 44 hours.

C Effectiveness of Affordance and Geometric-part Attributes

We provide the quantitative result for Figure 3 in the main paper. Time is measured in seconds.

D Result Visualizations

In this section, we visualize the navigation paths on both the Gibson and HM3D datasets, as shown in Figures 7 and 8. Part of the visualization code is adapted from L3MVN [19].

E Time Complexity

To validate the efficiency of our method, we compare the FPS of our method to SemExp [2], L3MVN [16], and VLFM [19] in navigation tasks on the HP3D dataset. The experimental results are shown in Table 7. It can be observed that SemExp has the highest FPS, indicating the fastest processing speed. This is because it uses a detection head and does not employ a foundation model. However, SemExp has the lowest SR and SPL, indicating that despite its fast processing speed, it performs poorly in navigation accuracy and path efficiency. In contrast, L3MVN has the second-highest FPS as it uses a lightweight foundation model. Although its processing speed is not as fast as SemExp, it shows improved navigation accuracy and path efficiency, achieving an SR of 76.1% and an SPL of 37.7%. On the other hand, VLFM has a lower FPS of only 2, but it significantly improves SR and SPL, reaching 84.0% and 52.2%, respectively. This indicates that although VLFM has a slower processing speed, it has considerable advantages in navigation accuracy and path efficiency. Our model has the same FPS as VLFM, both at 2, but further improves SR and SPL, reaching 85.7% and 55.5%, respectively. This demonstrates that our method maintains high navigation accuracy and path efficiency while providing comparable processing speed to VLFM. These experimental results verify that our proposed method achieves a good balance between time and accuracy.

F Real-world Experiments

In our real-world experiment, we will evaluate the performance of four zero-shot object goal navigation algorithms, including L3MVN [19], COW [32], ESC [3], and VLFM [16], within a standard indoor apartment environment consisting of two bedrooms, two bathrooms, one kitchen, and one living room. The experiment adopts a four-wheeled robot. Specifically, we use a JetAuto-Pro from Hiwonder equipped with an Intel Realsense D435i camera as our robot agent to navigate the environment and locate specific target objects, including a bed, toilet, table, sofa, and chair, without any prior knowledge of their locations. To ensure a fair comparison, the starting position of the robots was kept consistent across all trials for each algorithm. The video demo can be found on our project page: https://shalexyuan.github.io/GAMap/ .

8

G Limitation and Future Work

While our proposed method, Geometric-Affordance Maps (GAMap), has demonstrated significant improvements in zero-shot object goal navigation, it is important to acknowledge its limitations. Our approach relies heavily on the visual processing power of vision-language models, i.e ., CLIP. The effectiveness of GAMap also depends on the accuracy of geometric parts and affordance attributes inferred by LLMs. Although the multi-scale scoring method enhances attribute detection, it introduces additional complexity and computational overhead.

To address these limitations and further advance this research field, future work should optimize the integration of LLMs and VLMs to reduce computational overhead, potentially through techniques like model distillation. Enhancing the accuracy of geometric and affordance attribute inference is crucial, and more powerful foundational models could improve this accuracy. Additionally, exploring better methods for aggregating these attributes is also an interesting research direction.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We show the claim in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitation is included

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not have assumptions

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include it in the appendix

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release our code upon the decision of the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We follow the standard testing details which are stated in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The standard evaluation not include the error bar.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This is included in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: It is discussed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper has no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All assets are referenced.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is studying the problem of zero-shot object goal navigation. To solve the problems in traditional approaches, the authors propose a novel method, named Geometric and Affordance Maps (GAMap), which incorporates both geometric and affordance attributes as navigation guidance. A multi-scale scoring approach is proposed to cope with objects at different scales. Experiments on HM3D and Gibson datasets show the effectiveness of the proposed method for the metrics of success rates and success weighted by path length.\n\nSTRENGTHS:\n1. The paper is studying the problem of zero-shot object goal navigation, which I think is an important problem in object-goal navigation. \n\n2. The proposed method incorporating both geometric and affordance attributes is sensible to me. \n\n3. There are extensive analysis in the experiment section of the paper, which provides several insights.\n\nWEAKNESSES:\n1. Although the authors have demonstrated the difference of their work compared with previous ones, I am still not convinced about the key novelty of the work. In terms of organisation of the paper, it would be better if the authors could provide a table to better compare the difference of their work with existing works. \n\n2. There seems no real world experiments. I am not fully convinced about the effectiveness of the proposed method unless results of real robot experiments are provided.\n\n3. The authors have not removed the checklist instruction blocks.\n\n4. In Line 214, there is an issue with the citations. The paper needs further proofread.\n\nQUESTIONS:\nIt would be good if the authors could address my concerns in the weakness part. I will make my final decision after reading the authors' rebuttal and other reviewers' comments.\n\nLIMITATIONS:\nThe authors have discussed about limitation in appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach for the task of zero-shot object goal navigation. The approach first generates a geometric and affordance description of the goal object. This description is then matched with the visual embedding of the RGB frame (at multiple scales) in the CLIP embedding space to compute a similarity score. They project this score on an orthographic 2D map. During navigation, the agent explores the frontiers/unexplored regions with the highest score. They demonstrate the efficacy of their method on HM3D and Gibson object-goal benchmark.\n\nSTRENGTHS:\n- The authors present excellent ablations in the paper, justifying each design decision with ablations. They show ablations for using affordance and geometric attributes, multiple scaling levels, different visual encoders, and even different methods for calculating and merging GA scores! \n- The results (success-rate) on HM3D and Gibson object-goal navigation benchmark are better than the next best method (VLFM) (1.1% on ML3D, and 1.7% on Gibson).\n\nWEAKNESSES:\n- The idea of using attributes and longer descriptions for objects instead of just object label has been well studied [A, B] and I would have expected the authors to make connections to this popular line of work on effectively utilising CLIP representations. \n- I also thought that the 23.7% increase in SPL, and 37.0% in SPL claim is quite misleading specially because the only difference between them and VLFM is that VLFM uses a learned policy for navigation. It has been well established that using FMM for navigation works well enough for indoor navigation\n\n[A] Visual Classification via Description from Large Language Models, ICLR 2023. \n[B] LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions, 2023\n[C] Navigating to Objects in the Real World, 2022.\n\nQUESTIONS:\n- The authors argue that comparison to VLFM is unfair because they use a trained policy compared to FMM to achieve high SPL on HM3D. Even then, they obtain better SPL on Gibson. Can the authors comment on why a trained policy will help HM3D, but not Gibson? \n- I didn't fully understand Figure 3. What does decrease in success rate mean? Is it relative or absolute decrease? Is it percentage or absolute points?\n\nLIMITATIONS:\nThe authors don't discuss any limitation of their work in the manuscript. \n- Maybe the authors should add a small section discussing potential limitations such as lack of real-robot experiments in the paper. \n- assumptions about perfect odometry / localization to build a map. \n\nNot a limitation, but a suggestion to improve the manuscript: \n- I thought that the writing could be clearer. Some of the sentences were overly complicated. For example Line 42-43 - \"proposed method initiates with the LLM inferring the object’s geometric part attributes and potential\naffordances attributes, delving into a detailed understanding of the object’s physical structure and\nfunctional properties\". The paper will benefit from more succinct and clearer writing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an algorithm for the object-goal navigation/exploration problem by using a 2D navigation map containing geometric and affordance scores for the target object. Geometric and affordance features for the target object are extracted by asking an LLM to list them for the object, and these words are then used with a multi-scale CLIP setup to get the resulting \"GA\" scores that are projected into a 2D map and accumulated over time.\n\nSTRENGTHS:\n* The object navigation problem is important for robotics and embodied AI applications\n* The paper is mostly well-written and the experiments have ablations\n* The approach appears novel if somewhat narrow.\n\nWEAKNESSES:\n* Narrowness of contribution: Projecting CLIP-like open-vocab scores into a 2D navigation map has been done in earlier work like VLFM [17]. The novelty here appears to be that you ask an LLM for multiple (geometric, affordance) properties of the object to query CLIP with. \n\n* Motivation of baselines: The results only seem about 1% better than the recent VLFM paper, which you dismiss because it contains \"locomotion\" training. However, no mention is made how your robot does motion planning. Since your results measure path length, you have to use something to get to a high-scoring frontier without colliding. The digitized environments are also just referred to as \"data sets\", but there has to be a simulator running on top of them to simulate some type of robot moving through the environment. Some details on this may help motivate why you dismiss VLFM. \n\n* Reproducibility: The results are all with (and contributions hinges on) GPT4, but the back-end of GPT4 changes substantially over time and there is afaik no way to run a specific version of GPT4 (only major versions). This means the numbers in the paper may not be reproducible at the time of publication. It would be helpful to also include results with a specific version of an open-source LLM to at least have something reproducible.\n\n--------\nAfter rebuttal: The authors ran more experiments which addressed weaknesses #2 and #3. The quantitative improvements weren't as good as originally claimed but good enough. They even threw in a real robot experiment (video would also have been nice!)..The contribution is still a bit narrow but given the SOTA results on ObjNav it merits publication..\n\nQUESTIONS:\n* Please clarify what kind of path planning and robot model you use, and in more detail why VLFM is not comparable.\n\n* Even assuming the VLFM \"locomotion\" is not comparable, could you not just use their 2D map component with your path planning? Both are 2D maps with scores.\n\nLIMITATIONS:\nAdequate except for the points above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Geometric and Affordance Maps to tackle zero-shot object-goal navigation. It focuses on how to leverage VLMs to navigate toward objects of unfamiliar categories without prior training. The authors use GPT-4 to generate multiple attributes that describe affordance and geometry, and acquire attribute embeddings by CLIP text encoder. Besides, they propose a multi-scale scoring approach which divides each RGB image into multiple patches processed by CLIP visual encoder, to acquire multi-scale visual embeddings. Cosine similarity can be computed between those embeddings, resulting in a map used for navigation (e.g., applying Fast Marching Method based on the map). Experiments on HM3D and Gibson demonstrate improvements in success rates and success weighted by path length (SPL). The authors also provide an analysis on error pattern, which shows that detection error (missing the target object) is significantly reduced.\n\nSTRENGTHS:\nThe paper is clearly written. The proposed approach to leverage LLMs and VLMs to tackle zero-shot object-goal navigation is clean and simple.\n\nWEAKNESSES:\n1. The overall approach is limited by LLMs and VLMs. The analysis in Sec. 5.9 (Table 4) implies that the improvement mostly results from the reduced detection error. Since off-the-shelf LLMs and VLMs are used, the technical contribution of the paper seems to be limited by this fact. Although geometric and affordance attributes are proved effective in Figure 3, the conclusion might be limited to the specific VLM used in this paper, namely CLIP.\n2. It is better to replace Figure 3 with a table. It is not easy to tell the exact effect of $N_a$ and $N_g$.\n3. Some sentences in Sec. 5.1 are with typos (perhaps in a rush). For example, \"Gibson is generated by Al-Halah et al. [36] for agents with configuration A\". And the citations in L214 are messy.\n\nQUESTIONS:\n1. Can the authors compare the proposed method with PIVOT[1], which can also leverage VLMs to tackle robotic navigation? PIVOT focuses more on VQA, while this paper focuses on the similarity between the text and visual embeddings generated by CLIP. CLIP may not capture enough details, and thus the authors may need to introduce a multi-scale scoring approach. It is interesting to see whether VLM models can directly capture enough multi-scale information.\n2. Can the authors also compare the time cost with baselines, like SemExp and VLFM?\n\n\n[1] PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs\n\nLIMITATIONS:\nThe authors have addressed the limitations in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}