GAMap: Zero-Shot Object Goal Navigation with Multi-Scale Geometric-Affordance Guidance
Authors: Shuaihang Yuan
Hao Huang
Yu Hao
Congcong Wen
Anthony Tzes
Yi Fang
Venue: NeurIPS 2024 poster (2024)
Keywords: zero-shot object goal navigation
Primary area: robotics

Abstract: Zero-Shot Object Goal Navigation (ZS-OGN) enables robots to navigate toward objects of unseen categories without prior training. Traditional approaches often leverage categorical semantic information for navigation guidance, which struggles when only partial objects are observed or detailed and functional representations of the environment are lacking. To resolve the above two issues, we propose \textit{Geometric-part and Affordance Maps} (GAMap), a novel method that integrates object parts and affordance attributes for navigation guidance. Our method includes a multi-scale scoring approach to capture geometric-part and affordance attributes of objects at different scales. Comprehensive experiments conducted on the HM3D and Gibson benchmark datasets demonstrate improvements in Success Rates and Success weighted by Path Length, underscoring the efficacy of our geometric-part and affordance-guided navigation approach in enhancing robot autonomy and versatility, without any additional task-specific training or fine-tuning with the semantics of unseen objects and/or the locomotions of the robot.

GAMap: Zero-Shot Object Goal Navigation with Multi-Scale Geometric-Affordanc Guidance
Shuaihang Yuan* 1,244 Hao 24 Yu 2 Congcong Wen
Anthony Ties Yi Fang 1,2,334
NYUAD Center for Artificial Intelligence and Robotics (CAIR), Abu Dhabi, UAE. New York University Abu Dhabi, Electrical Engineering, Abu Dhabi 129188, UAE New York University, Electrical & Computer Engineering Dept., Brooklyn, NY 11201, USA. Embodied AI and Robotics (AIR) Lab, NYU Abu Dhabi, UAE.
Abstract
Zero-Shot Object Goal Navigation (ZS-OGN) enables robots or agents to navigate toward objects of unseen categories without object-specific training. Traditional approaches often leverage categorical semantic information for navigation guidance, which struggles when only objects partially observed detailed and functional representations of the environment are lacking. To resolve the above two issues, we propose Geometric-part and Affordance Maps (GAMap), a novel method that integrates object parts and affordance attributes as navigation guidance. Our method includes multi-scale scoring approach capture geometric-pe and affordance attributes of objects at different scales. Comprehensive experiments conducted on HM3D and Gibson benchmark datasets demonstrate improvements Success Rate and Success weighted by Path Length, underscoring the efficacy of our geometric- part and affordance- guided navigation approach in enhancing robot autonomy and versatility, without any additional object-specific training or fine-tuning with the semanties of unseen objects and/or the locomotions of the robot. Our project is available at htttt::::.. / wwwsshalllllae lewyuan gi thub. 1o/GAMap/)
Introduction
Zero-Shot Object Goal Navigation (ZS-OGN) is a pivotal research domain in embodied Al and robotics, enabling robots to navigate towards the objects of unseen categories without training or fine-tuning on these objects I 2 3 5. This capability crucial for real-world robots, such as home service robots and blind guiding robots, allowing them to interact with diverse objects in real-world scenarios, thereby enhancing their autonomy and versatility.
Prior works on ZS-OGN either leverage deep neural networks to directly map RGB-D observations to actions learned from paired training data 66 7 10 or utilize map-based navigation methods
However, deep neural network approaches often struggle due to their dependence on extensive annotated data, resulting poor generalization unseen environments while map-based navigation methods instead offer an alternative. Map-based navigation methods track categorical semantics and topological information observed by the agent to select promising exploration locations [1]. With the advent of foundation models, the studies 3 17 have exploited the reasoning capabilities of Large Language Models (LLMs) to strategically select
[3
13
14 15
16
7
3
18
16],
"Equal contribution. CCoresponning author: Yi Fang <yfang @yy.euun....
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Attributo "hhlr bask" high
Altrituta "chuii log" back
scoee: soplone this anas
Hight accou,, pursioi in direction
Propore
frontior
trrget object,
Figure I: The leftmost RGB image shows the same observation for both methods. Our method (top row) effectively identifies the geometric part of the chair back, which is missed by the traditional method (bottom row). Consequently, GAp successfully guides the agent the target object, while the traditional method fails. The red circles highlight the areas where the chair is located, and the GA score is high, indicating the effectiveness of our approach in localizing relevant regions.
waypoints by analyzing commonsense, such as object co-occurrence relationships, to navigate robots towards the target. However, LLM--based approaches require converting visual and semantic information into categorical descriptions, which leads loss of spatial and visual information [9] Vision Language Models (VLMs) enhance semantic reasoning capabilities, but still rely on maps that encompass only categorical information [6]. The primary limitation of exclusively relying on categorical information is that such maps treat objects as monolithic entities, disregarding local geometric features. This becomes particularly problematic when only the target object partially observed, leading to incorrect categorical information and potential errors in waypoint selection.
We argue that using a categorical map for robot exploration is suboptimal, as it discards intricate geometric details and functional representations of the environment, as illustrated in Figure [ Drawing inspiration from human cognitive processes where distinctive geometric parts are often identified first when locating an object in an unfamiliar environment [20 21 we propose Geometric-pars and Affordance Maps (GAMap), zero-shot approach, for the geometric parts and affordance attribute driven semantic navigation to explore and find the target object in unseen environment. Specifically, given a target object, our proposed method starts by using an LLM to infer the object's geometric parts and potential affordance attributes, aiming providing detailed understanding of both the object's 's physical structure and its functional properties. Given depth observations, GAMap maintains 2D projection of obstacles and explored areas. Instead of relying on object detection and prompt engincering of LLMs to select the next area to explore, our approach employs pre-trained CLIP [22] score observations based their similarity to the reasoned geometric parts and affordance attributes, guiding the exploration process. To construct the proposed GAMap, which requires obtaining scores for geometric parts different scales, we further propose Multi-scale Geometric and Affordance score (GA score). Such an integration addresses a notable limitation in existing approaches that compute similarity oversight of fine-grade fine-grained details of an object, as such details are potentially essential for an accurate identification of geometric parts or affordance attributes of objects with different sizes.
single scale, which usually results in the
Our proposed method is evaluated on HM3D 23 and Gibson 24 benchmarks and achieves sig- nificant improvements in Success Rate (SR) of 26.4% on HM3D 23 and 23.7% on Gibson [24] compared to previous approaches. Additionally, we achieve substantial gains in Success weighted by Path Length (SPL) of 37% on Gibson, highlighting the efficiency and effectiveness of our proposed geometric parts and affordance guided navigation. The contributions our method are mainly summarized as follows:
1. We propose novel Geometric-part and Affordance Map (GAMap) for 7S-OGN using object part and affordance attributes as guidance. To the best of our knowledge, this the first work to study the integration of these attributes in ZS-OGN.

---

2. Recognizing that geometric parts and affordance attributes often relate to multiple scales of an object. we propose a Multi-scale Geometric and Affordance score. which allows GAMap to be constructed in real-time, better capturing these attributes at different scales.
3. We achieve state-of-the-an thtttheetttttte performance on two navigation benchmark datasets without any training or fine-tuning with the semantics of unseen objects and/or the locomotions of the robot, which demonstrates the effectiveness of our method in unseen environments.
2 Related Work
Semantic Mapping. In the context of object goal navigation, it is crucial to transform observations into structured information for effective crrision-maaing.... Frontier-based methods [3] 16 19] utilize
categorical semantic information near frontiers to select exploration areas. Additionally, graph-based mapping methods [7 [5] 18] predict waypoints from RGB-D images or simplified maps create topological representations of the environment. Most of the aforementioned works rely on semantic segmentation or object detection to build semantic maps, which are constrained by the pre-defined semantic classes and thus fail to capture the full semantic richness of environments [18 26 To
overcome these limitations, recent approaches like VI.Maps [8]] have introduced apen-vocabulary semantic maps, enabling natural language indexing and expanding the scope of semantic mapping. In addition, previous works [27 28) attempt to utilize attributes and long descriptions for object perception. While these methods have advanced the navigation field, they often overlook object parts, treating objects as monolithic entities and leading to errors when these objects are partially observed. Inspired by human cognitive processes [21 20 where distinctive geometric parts are identified first in unseen environments, we propose Geometric- paatt and Affordance Maps (GAMap). Unlike previous methods focusing solely categorical information, GAMap integrates geometric parts and affordance attributes, providing a richer and more functional representation of the environment.
Zero-shot Object Goal Navigation. In the context of object goal navigation, the aim is to efficiently explore a new environment while searching for target object that is not directly visible. Previous research relies heavily on visual context via imitation [29] 6 or reinforcement learning 7 S 9 to guide robots. These approaches often require extensive data collection and annotation for training, which limits their practical application in real-world environments. Thus, the focus in object goal navigation has been shifting towards zero-shot object navigation, which aims to equip robots with the ability to adapt unseen objects and environments without the need for training 12 30 [3 14]
Clip-Nav [3I] utilizes CLIP 22 to execute vision-and- and-language navigation in a zero-shot scheme, whilst CoW [32 employs CLIP for object goal navigation. Recently, Frontier-based Exploration (FbE) 33 is widely adopted in navigation by moving the robot to the frontier between known and 37), leading to promising performance compared learning-based 39. More recently, ESC (3 leverages the reasoning ability of LLMS
unknown spaces 332 34
35 36.
exploration methods 38
to select frontiers using pre-defined rules. Chen et al. [5] explore frontier selection by jointly considering the shortest path to frontiers and the relevance scoring between objects for exploration. To enable more robust and reliable exploration and waypoint selection, Wu el al. [17 propose Voronoi- basdd scene graph for waypoint selection. Unlike the above methods that use the reasoning ability of LLMs to select frontiers, VLFM [16] introduces a value map to score frontiers based on the categorical similarity between the observation and the target object. In contrast to prior work, for the first time, we explore robot navigation using geometric parts and affordance attributes as guidance. This approach integrates detailed geometric parts and functional properties of objects, offering more
comprehensive strategy for navigation.
3 Method
We first formalize the ZS-OGN problem in Section 3.1 Then, we detail our method, as shown
in Figure 2 from four phases: attribute generation in Section 3.2 multi-scale attribute scoring in
Section 3.3) GAMap generation in Section [3.4] and exploration policy in Section [3.5] Initially, our method generates geometric parts and affordance attributes for the target object. During the exploration, the method computes multi-scale attribute score from the RGB observations collected by the agent. These scores are then mapped onto 2D geometric parts and affordance map, which pivotal in guiding the exploration process. Subsequently, the agent selects the location with the highest score for further exploration.
3

---

CoeTe
Pintin
Dreede
vdl
DES
Huzt Oteeradios
Large Node
OtgeCt Object "Char"
SElooose Foonet
Ambuts Entendin
Canner Poza
Maz
Figure 2: Pipeline of the GAMap generation. Geometric parts and affordance attributes are generated by an LLM. The RGB observation partitioned into multiple scales, and CLIP visual encoder generates multi-scale visual embeddings. GA scores are computed using cosine similarity between attribute text embeddings from a CLIP text encoder and the multi-scale visual embeddings. These scores averaged and projected onto 2D grid to form the GAMap.
3.1 Problem Formulation
In ZS-OGN, the robot must navigate to a target object that has never been encountered before in unseen environment 8;, without any training 9: and 8;. A navigation episode be defined as & = {9., 8,, Po). where Po denotes the robot's ss initial location, the subscript refer to the 8th episode. The robot receives a color image I, a depth image D., and its pose, ie., position (x1,3e) Vr) and orientation 9,, at each exploration step t. We denote these readings = D., It. yr, } The agent accumulates pose readings over time to determine its relative position Pt. Based on the readings at each step, the robot needs to select action a, from the action space. A navigation episode is marked as successful if the robot executes the STOP action within a pre-defined distance the target object. In this work, approach the navigation task of decisions made by the robot. The process starts at the initial time step = 0 and ends at the final time step T. This final step is either when the robot executes the STOP action or when pre-defined maximum number of exploration steps is reached.
observation
sequence
3.2 Attribute Generation
We focus on two types of attributes essential for object recognition: Affordance and Geometric-part attributes. Affordance attributes refer the potential actions that object facilitates [40), which crucial for understanding how an agent might interact with different objects within an environment. Geometric-pan attributes, on the other hand. describe the shape and spatial characteristies of an object, aiding in its visual identification and differentiation from other objects.
To extract these attributes, employ LLM to reason about the target object's characteristics. Specifically, we utilize GPT-4V [41 for the attributes generation. We initiate this process by setting the system prompt as: "I am highly intelligent question- -natttrrnnn bot, and I answer questions from human perspective." Subsequently,
employ two tailored prompts to extract the desired attributes. For affordance attributes, we design prompt "For the target object <target object g;>, please provide <N.> alfordance altribules that the most reflect its characteristics: to query N, number of affordance attributes. For geometric parts, the prompt is: "Summarize <N₂> geometric part visual features of the <target object g;> which are typically used to identify why it is a <target object g.>. get N, number of geometric attributes. Once the of affordances attributes ) and geometric attributes ( have been identified, the next stage involves coupling these attributes with the agent's i observations by computing Geometric- part and Affordance scores (GA scores) through multi-scale visual features, as detailed in the following.

---

3.3 Multi-scale Attribute Scoring
The key idea of Multi-scale Attribute Scoring is to quantify the relevance of observed areas' geometric and affordance characteristics in locating the target object. These scores determine the agent's subsequent exploration decisions, guiding it to the areas most likely to contain the target object.
To correlate the scoring with both the global frame and localized patches, ensuring that finer details and smaller components are effectively scored, we partition the observed image into patches of equal size across different scales. Specifically, given an RGB observation image I,, with height H and width W. the partitioning process is as follows. At level k, the image is partitioned into 4(k-1) equal parts with each patch level l of size 2 X 21--15.
W The patches at level it can be represented as:
1 = {Ph.. 1 h S 2((-11), 11 w S 2(kk1)) where Pr,w denotes the patch located at the Ath row and whth column of the partitioned image at level k. All patches from all levels are then resized to the same dimensions for further processing. We utilize CL.IP [32] to calculate the visual embeddings for all patches from all levels. For each patch Pft at level k its visual embedding " is computed using the image encoder of CLIP. Simultancously, each attribute embedding e is computed using the text encoder of CLIP, given the attribute descriptions, (An) and {G,, ( generated from an LLM. For each patch Pk N, we calculate the cosine similarity between its visual embedding "i and the attribute embedding to obtain a similarity score skke as follows:
sk.e h,
(I)
1 ||lle||
Due to the hierarchical partitioning of the image, we accumulate the scores across all scales for each pixel location and then take the average to obtain the final score for the image. Specifically, let L be the number of levels, and the accumulated score for the pixel at position (p, q) in the image Ie is calculated by summing the scores from all levels and then averaging:
S(p, q, = L b.ip-a.k),.6(....k
Σ
she
where (p, g, k) and w(p, q, k) map the pixel position (p. q) in the image to the corresponding patch indices at level k. The scores are then used to generate a Geographic and Affordance Map (GAMap) for the explored area. The process for generating the GAMap is detailed in the next section.
3.4 Geographic and Affordance Map
At the core of our approach is the Geographic and Affordance Map. This map assigns GA score to each pixel within the explored area to quantify the relevance of different regions in locating the target object, associating the areas with the highest values as the most promising for further exploration.
We define the GAMap at time step f as Mr E &WWCC where A and W. are the dimensions of the 2D projection grid map, and C' is the number of attributes, i.e., C = N₂ + Ng with N, and N2 representing the number of attributes and parts, respectively. To construct the GAMap from the RGB-D observation /, and the depth data D,, we back-project every pixel from D, to reconstruct the point cloud following [8]
X D.(pg) K- (p.g.11" 1)"
where K is the intrinsic matrix of the depth camera, and Dr(p, q) is the depth value of the pixel at the coordinate (p, q). To transform the point cloud into the world coordinate frame, we use X Xworld = Tw X, where Tw is the transformation matrix from the camera coordinate to the world coordinate. The 3D points are then projected onto the ground plane to determine the corresponding positions on the grid map. We assume perfect calibration between the depth and RGB cameras, allowing us to project each image pixel's score to its corresponding grid cell in the map. Given that multiple 3D points may project to the same grid location, we retain the maximum value for each channel that falls into the same grid cell as the score of this cell:
M,(h, w. e) = max { {S e) (p. 4) E cell Mc(h, w)
5

---

where (h, tù, e) represent the score of attribute e in the 2D grid at position (h. u). S(p 9. e) is the score for the pixel at (p, 9) for the attribute e, and cell M, u) denotes the grid cell on the GAMap. When the robot moves to a new position, resulting in overlapping observations with the previously explored regions, the GA scores for each pixel in the overlapping region are updated. The updated GAMap is computed by taking the maximum of the current attribute score and the previous attribute score for each cell:
M (ho úh e) = max (Mi Wo c), Mho Wa wo,e)) e))
(5)
where (h:: o. e) is the GAMap constructed from the previous step, and the subscript o represents the overlapped grid cell.
3.5 Exploration Policy
To determine the next area to explore, the robot selects the region with the highest GA score, calculated as the average of all attribute channels. To enable efficient exploration, only areas near the frontter with the highest scores are selected. Once the area with the highest GA score is identified, a heuristic search algorithm, i.e., the Fast Marching Method (FMM) 42), is employed to find the shortest path from the robot's s current location to the selected area. The robot then generates the appropriate actions to navigate along this path. At each step, the GAMap is updated based new observations. We repeat this process until the robot either identifies and reaches the target object or the episode ends.
4 Experiment
a dataset consisting of 3D data from real-world indoor spaces, along with semantic annotations, serving foundational resource for the Habitat 2022 ObjectNav Challenge [23]. This comprehensive dataset includes 142,646 object instance annotations, organized into 40 distinct classes across 216 environments, covering a total of 3, 1,000 individual rooms. We follow the validation settings from 3 32 to evaluate our proposed method. Gibson (24 was developed by
Datasets. HM3D 23
Al-Halah al. [43] The dataset comprises 5 validation scenes across 6 object categories, and
adhere to the standard evaluation protocol 44
19 30 2
to use all validation scenes for evaluation.
Metrics. Success Rate (SR, %) 45 focuses on the agent's accuracy in reaching the designated target, where a higher value indicates better performance. SR computed based on whether the robot successfully stops within 0. Im of the target object g::
1 SR(T) = K E
(6)
where K is the number of episodes, d(gi, pT:) is the distance between the target object gi and the robot's final position Pr; in the 7th episode, and ((-) is an indicator function. Success weighted by Path Length (SPL, %) [45) evaluates success relative to the shortest possible path, normalized by the actual path taken by the agent, measuring the efficiency of the agent's success in reaching a goal, defined
K
SPL(T) = K
1
Σ
(1(g) )50.)
max L., LT)
LE
where L; is the actual path length traveled by the robot in the ht episode and L is the shortest possible path length to the target in the same environment.
4.1 Baselines
We compare our method against several ZS-OGN approaches, including the state-of-the-ar art methods: Random Exploration: takes random actions to explore the environment. Nearest FbE [3]: explores the environment by selecting the nearest frontier. SemExp 22] utilizes a category semantic map and trains a local navigation policy for exploration. PixNav [9: trains models for navigation by selecting
pixels as intermediate goals. PONI 44 uses potential functions to select frontiers for exploration.
ZSON [30]: employs categorical information to train model for object-based navigation tasks. CoW

---

32 : uses CLIP for categorical information extraction and explores using the nearest trontier-based exploration. ESC A utilizes categorical semantic map and commonsense reasoning for target object exploration. L3MVN [9] uses an L.M to reason about the next exploration area based on a trained detection head for semantic map construction. VLFM [6]] employs BL.IP-2 [46] and categorical information of the target object to evaluate and select frontiers for exploration. VoroNav [77] uses a Voronoi-based decomposition strategy for navigation. StmUtil [15]: considers the shortest path distance to frontiers and the relevance scoring between objects for exploration.
Table Comparison of navigation performance between different methods on HM3D and Gibson datasets, measured by SR and SPL. metrics. This table highlights the performance of our proposed method, demonstrating improvements over existing methods on both datasets.
Training
HIM3
Gibson
Method
Venue
Zero-shot
Locomotion
Semantic
SR
SPL
SR
SPL
SemLxxp
NeurlPS 20
37.9
18.8
65.2
33.6
2SON 30
NeurlPS 22
25.5
12.6
31.3
12.0
PixNav
9
ICRA 24 CoRL.
X
X
37.9
20.5
VLFM
x
52.5
30.4
84.0
52.2
PONI
CVPR 22
x
X
V
73.6
41.0
23.7
12.3
41.7
21.4
L3MVX 19
IROS 23
50.4
23.1
76.1
37.7
Random
CoW 33
CVPR 23
32,0
18.1
ESC
3
ICMI. 23
38.5
22.0
SemUtil
RSS 23
69.3
40.5
VoroNav
ICML
42.0
26.0
VLFM Value Map FMM [6
CoRL
50.9
23.6
82.8
48.5
GAMap
Pruposed
26.0
85.7
55.5
4.2 Results and Analysis
We compare our method with existing ap- proaches across four categories: those uti- lizing both locomotion and semantic training
SR and Time Cost
N.a (Number of Affordance Attributes]
2 30, those employing only locomotion train- ing [6]. those using only semantic training 19 44 and those that do not incorporate any training 3
Decrease on Success Rate
17 15). Locomotion training methods for navigation, while semantic training requires training or fine- tuning a perception module to construct a se- mantic map. The results and comparisons are shown in Table
32
-157
involves learning-bast
On the HM3D dataset, method achieved SR of 53.1% and a SPL of 26.0%. This repre- sents significant improvement over the best method [17 that does not use locomotion and semantic training, with 26.4% increase in SR. Although the SPI. (26.0%) of our method is lower than that of VLFM (30.4%). this discrep- ancy can be attributed to the fact that VLFM's local policy planning is trained. Considering the different path planning methods adopted in our approach and VLFM, we construct com- parative experiment with VLFM. Specifically, we kept the VLFM value map generation process unchanged and replaced its path planning method with FMM instead of the trained policy. The detailed results are shown in the second last row in Table [] The comparison reveals that our model outperforms VL-FM-aased mapping method by 4.32% and T0.17% 17% in SR and SPL on HM3D. On the Gibson dataset, GAMap attained SR of 85.7% and SPL of 55.5%, which marks a substantial
N_g (Number of Geometric Attributes)
Figure 3: Heatmap showing the increase and de- crease in the percentage of SR and time cost for varying the numbers of N and Ng. Darker colors indicate greater decrease SR, and red solid and dashed lines represent the associated time cost.
7

---

improvement over methods that do not utilize locomotion and semantic training [77. with 23.7% increase in SR and a 37.0% increase in SPL.
5
Ablative Study
5.1 Effectiveness of Affordance and Geometrie-part Attributes
We analyzed the effectiveness of the number of affordances and geometric parts in proposed method. Ablation studies were conducted by varying Na and Ng to assess performance gain versus time loss, well as the contribution of each attribute type. We evaluated on the official mini vali-valiation split of the HM3D dataset.
The heatmap, shown in Figure 3 illustrates the SR with different combinations of N₂ and Ng, The color changes from dark purple to yellow indicate the increased percentage SR. Red sold and dashed lines with labels indicate the time cost associated with each combination of Na and g. Increasing the number of geometric parts (N₂,) from 0 results in significant improvement in SR across all levels of N.. For example, when Na = O, increasing N, from 0 to 3 raises the SR more than increasing No from to 3 when Ng = 0. This demonstrates the substantial contribution of geometric parts to navigation performance. To more clearly demonstrate the specific impacts of Na and Ng on various performance metrics, we have converted Figure 3 to Table in the Appendix. Similarly, increasing the number of affordance attributes (N,) also improves SR, though the effect is slightly less pronounced than that of geometric parts. The best performance is achieved when both N₂ and Ng are maximized, with both set to 3. This suggests synergistic effect, where the combination of both attributes leads to optimal navigation performance. However, it requires an 18.2% increase in time cost.
5.2 Effectiveness of Different Scaling Levels
We analyzed the effectiveness of differ- ent scaling levels (L). Figure 4 presents changes in SR and SPI. when using differ- ent numbers of scales and the time required for processing on the mini-validation split of HM3D.
Analyce on Number of Scales
50
SR
48.6
46.2
SPL
Time
0.76
Increasing the number of scaling levels from 1 to 4 leads to notable changes in both SR and SPL. At the highest scale level of 4, the SR improves by approximately 10%, and the SPL. increase by about 20% than
and SPL Wales
074
20-
0.72
21.4
L. = 1. However, there is a trade-off be- tween performance improvement and time cost. The time required increases with the number of scales, as indicated by the red line in the figure. The time cost starts at 0.66 seconds for single scale and rises progressively, reaching 0.78 seconds at the fourth scale level. This increase time cost suggests that while higher scaling lev- els improve both SR and SPI., they also demand more computational resources and time. This emphasizes the need balance performance and computational efficiency when determining the optimal scaling level for practical applications.
10
0.66
0.67
0.66
Figure 4: Changes in SR, SPL., and processing time across different scaling levels on the mini-validation split of HM3D. Increasing scales improves SR and SPL but also increases processing time.
5.3 Effectiveness of Different Methods for Calculating GA Scores
We analyzed the proposed patch-based method by comparing it with the gradient-based method. The visualization the GA score for different methods shown in Figure which illustrates the GA score of the armrest, backrest, and seat attributes of target object chair. Observations indicate that the gradient-based method often attends irrelevant areas. For example, the ceiling of the room has higher GA score, which is incorrect. In contrast, the patch-based method more accurately focuses

---

on relevant areas, such as the armrest, backrest, and seat of the chair, validating its effectiveness over the gradient- based method. Moreover, as observed from T the gradient-bad based method is also slower than the patch-based method. One reason for this is that the gradient-based method requires back propaaappaatinnn of the gradient.
Based on different methods, we further explored the effectiveness of various pre- trained encoders, as shown in Table 2 We
Armrest
Backrest
Grodient Boned
compared SR and computation time re- quired by three types of encoders: CLIP (ours), BLIP, and BLIP-2. Although using more powerful encoders such as BLIP and BLIP-2 leads to better performance, they require significantly more time than CLIP, while the performance gain is limited. This trade-off makes it less valuable for us to
Patch Be sed
use a BLIP-based encoder.
Figure 5: Comparison of GA score visualization be- tween gradient-based and patch-based methods for the armrest, backrest, and seat attributes of target chair. The gradient- based method (top row) often attends to irrelevant areas, such as the ceiling, while the patch- based method (bottom row) accurately focuses on the relevant areas.
Moreover, we analyzed different ways of aggregating GA scores using patch-bat based methods. Given the multiple levels of patches, we aimed find the best method to aggregate the final GA score from multi- level GA scores.
We compared "Max Value" and "Average Value" (ours). In the "Max Value" method, the maximum GA score across different scale levels used, The "Average Value" method, which we propose, calculates the average GA score across levels. Note that the gradient-based method directly gives the score, it is not analyzed. As shown in Table 2 using the average to aggregate the score gives us the best performance.
Table 2: Analysis of different GA scoring methods.
Table 3: Impact of different GA score updating methods.
Method
Encoder
Ave.T
Maxt
Time!
CL.IP
50.25
49.75
0.7
Patch
BLIP
51.20 (te
50.67cte 19)
1.2 (1715)
Method
SPL
BLIP-2
51.60 (9))
51.10 (h22
1.6 (11289)
Max
50.25
0.253 0.226 0.209 (14179)
Average
48. 48.30 (2000)
CLIP
49.78 (1-15))
0.9 (1719)
Replacement
47.78 (1449)
Gradient
BI.IP
50.67 (tee 199
1.5 (11149)
5.4 Effectiveness of Different Methods for Updating GA Scores
Different methods of updating GA scores and their impact on navigation performance are shown in Table In the "Replacement" method, the previous value is disregarded and overwritten with the new one. The "Average" method calculates the new value as the average of the previous and current values. Our approach, "Max", retains the maximum scores between the previous and new values, which memorizes the most salient score in specific direction, as the agent could observe an object from different perspectives during exploration, thus potentially finding the optimal perspective.
Our findings indicate that the "Max" method consistently enhances performance compared to the other two methods across all three datasets. In Table the "Max" method achieves SR of 50.25% and SPL of 25.3%. The "Average" method results in 3% decrease in SR and 10% decrease in SPL, indicating a moderate impact on performance. The "Replacement" method shows the most significant performance drop, with a 4 4% decrease in SR and a 17% decrease in SPL. These results highlight that the "Max" method is the most effective in maintaining and enhancing navigation performance, as it better captures and retains the most relevant object attributes from different perspectives.
5.5 Effectiveness of Geometric and Affordance Guidance Navigation
We evaluated the effectiveness of the proposed part and affordance guidance navigation by analyzing three types of errors: detection error, planning error, and exploration error. 1) Detection error happens

---

when the agent either misses the goal or incorrectly believes it has detected the goal. 2) Planning ennor arises when the agent either recognizes the target but cannot reach or gets stuck without spotting the goal, reflecting the path-planning ability of the system. 3) Exploration error occurs when the agent fails see the goal object due issues other than planning or detection, assessing its ability to approach the goal. 4 shows the comparison of these errors using our proposed method versus using categorical semantic information as the guidance for navigation. Note that our proposed method significantly decreases the errors in all three categories.
5.6 Effectiveness of Multi-scale Approach
Furthermore, conduct an experiment to verify whether VLM [47] mod- els can directly capture enough multi-scale informa- tion. We randomly selected scene and compared the
Table 4: Comparison of errors using categorical semantic guidance versus geometric parts and affordance guidance.
Method
Detection Error (%) Planning Error (%)
Exploration Error (%)
Categorical
16,95
19.1
15.65
GA
9.75 (142.59)
11.34 (14088)
12. (11839))
ability of GPT-4V and our proposed CLIP with multi-scale scoring method to identify the target object. As shown in Figure 6, we input an image with sofa located in a distant corner as the target object and compared the subsequent movement trajectories of the two algorithms. As illustrated in Figure 6 our method successfully captures the small sofa back in the far corner, leveraging geometric parts and affordance attributes guide the exploration process. In contrast, GPT-4V failed identify the object.
Goal sofa
Multiscale CLIP
Goal sofa
GPT GPT-4V -VV
Time
Figure 6: The top row images shows our proposed method, where the multi-scale approach effectively captures objects at all scales, such as the sofa back in the background. The bottom row of images shows the results GPT-4V.
6 Conclusion
In this work, we introduced the Geometric- part and Affordance Maps (GAMap) for zero-shot object goal navigation, leveraging geometric parts and affordance attributes to guide exploration in unseen environments. Our method employs LLMs for detailed attribute inference and VLMs for multi-scale scoring, capturing object intricacies at various scales. Comprehensive experiments on HM3D and Gibson datasets exhibit significant improvements in SR and SPI. over previous methods. These results highlight the effectiveness of our approach in enhancing navigation efficiency without any task-specific training fine-tuning.
10

---

Acknowledgements
Authors appreciate the support provided by the NYUAD Center for Artificial Intelligence and Robotics (CAIR), funded by Tamkeen under the NYUAD Research Institute Award CG010.
References
[1] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhut- dinov. Learning explore using active neural slam. In International Conference Learning Representations, 2020.
[2] Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhut- dinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33:4247 11:4247-4255 4258, 2020.
[3] Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, and Xin Eric Wang. Ese: exploration with soft commonsense constraints for zero-shot object navigation. In Proceedings of the 40th International Conference on Machine Learning, pages 42829-42842, 2023.
[4] Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning vision-ande and-aaddddnnee and-language navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641-7649, 2024.
[5] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023.
[6] Haresh Karnan, Garrett Warnell, Xuesu Xiao, and Peter Stone. Voila: Visual-observaaionnonli imitation learning for autonomous navigation. In International Conference Robotics and Automation, pages 2497-2503, IEEE, 2022.
[7] Gregory Kahn, Adam Villaflor, Bosen Ding, Picter Abbeel, and Sergey Levine. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In IEEE International Conference on Robotics and Automation, pages 5129-5136. IEEE, 2018
[8] Jan Wöhlke, Felix Schmitt, and Herke van Hoof. Hicrarchies of planning and reinforcement learning for robot navigation. In JEEE International Conference on Robotics and Automation, pages 10682-10688. JEEE, 2021.
[9] Wenzhe Cai, Siyuan Huang, Guangran Cheng, Yuxing Long, Peng Gao, Changyin Sun, and Hao Dong. Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill. In IEEE International Conference on Robotics and Automation, pages 5228- 5234. IEEE, 2024.
[10] Dhruv Shah, Michael Robert Equi, Blažej Osiński, Fei Xia, Brian Ichter, and Sergey Levine. Navigation with large language models: Semantic guesswork as heuristic for planning. In Conference on Robot Learning, pages 2699-2699. PMLR, 2023.
[11] Hao Huang, Shuaihang Yuan, Congcong Wen, Yu Hao, and Yi Fang. 3d-trans: 3d hierarchical transformer for shape correspondence learning. In International Conference on Automation, Robotics and Applications, pages 33-5540- IEEE, 2024.
[12] Qianfan Zhao, Lu Zhang, Bin He, Hong Qiao, and Zhiyong Liu. Zero-shot object goal visual navigation. In IEEE International Conference on Robotics and Automation, pages 2025-200 IEEE, 2023.
[13] Vishnu Sashank Dorbala, James Mullen Jr, and Dinesh Manocha. Can an embodied agent find your "cat-shaped mug"? llm-based zero-shot object navigation. IEEE Robotics and Automation Letters, 2023.
[14] Qianfan Zhao, Lu Zhang Bin He, and Zhiyong Liu. Semantic policy network for zero-shot object goal visual navigation. IEEE Robotics and Automation Letters, 2023.

---

[15] Junting Chen, Guohao Li, Suryansh Kumar, Bernard Ghanem, and Fisher Yu. How to not train your dragon: Training free embodied object goal navigation with semantic frontiers. In Robotics: Science and Systems, 2023.
[16] Naoki Harrison Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang. and Bernadette Bucher. Vifm: Vision-language frontier maps for zero-shot semantic navigation. In Workshop on Language and Robot Learning: Language as Grounding. 2023.
[17] Pengying Wu, Yao Mu, Bingxian Wu, Yi Hou, Ji Ma, Shanghang Zhang, and Chang Liu. Voronav: Voronoi-based zero-shot object navigation with large language model. arXiv preprint arXiv:2401.0269) 2024.
[18] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In IEEE International Conference on Robotics and Automation, pages (0608-10615. IEEE, 2023.
[19] Bangguo Yu, Hamidreza Kasaei, and Ming Cao. L3mvn: Leveraging large language models for visual target navigation. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3554-3560. IEEE, 2023.
[20] Shimon Ullman. High-level vision: Object recognition and visual cognition. MIT press, 2000.
[21] M Gazzaniga, R Ivry, and G Mangun. Cognitive science. Springer, 2002.
[22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8788 8763. PMLR, 2021.
[23] Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexan- der Clegg. John M Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habirat-matterpor 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[24] Fei Xia, Amir Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2018
[25] Obin Kwon, Jeongho Park, and Songhwai Oh. Renderable neural radiance map for visual navigation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9099-9108, 9108, 2023.
[26] Hao Huang, Shuaihang Yuan, CongCong Wen, Yu Hao, and Yi Fang. Noisy few-shot 3d point cloud scene segmentation. In IEEE International Conference on Robotics and Automation, pages 11070-11077. IEEE, 2024.
[27] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183. 2022.
[28] Songhao Han, Le Zhuo, Yue Liao, and Si Liu. Lims visual explainers: Advancing image classification with evolving visual descriptions. arXiv preprint arXiw:2311.11904 2023.
[29] David Silver. James Bagnell, and Anthony Stentz. High performance outdoor navigation from overhead data using imitation learning. Robotics: Science and Systems, 2008
[30] Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, and Dhruv Batra. Zson: Zero-shot object-goal navigation using multimodal goal embeddings. Advances in Neural Information Processing Systems, 35:32340-32355 2022.
[31] Vishnu Sashank Dorbala, Gunnar A Sigurdsson, Jesse Thomason, Robinson Piramuthu, and Gaurav S Sukhatme. Clip-nav: Using clip for zero-shot vision-and-language navigation. In Workshop Language and Robotics at CoRL, 2022.
12

---

[32] Samir Yitzhak Gadre, Mitchell Wortsman, Gabriel Ilbarco, Ludwig Schmidt, and Shuran Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23171-23181, 2023.
[33] Brian Yamauchi. A frontier-based approach for autonomous exploration. In Proceedings of IEEE International Symposium Computational Intelligence in Robotics and Automation, pages 146-151. IEEE, 1997.
[34] Shuaihang Yuan, Muhammad Shafique, Mohamed Riyadh Baghdadi, Farshad Khorrami, An- thony Tzes, and Yi Fang. Zero-shot object navigation with vision-language foundation models reasoning. In International Conference on Automation, Robotics and Applications, pages 501-505. IEEE, 2024.
[35] Congcong Wen, Yisiyuan Huang, Hao Huang, Yanjia Huang. Shuaihang Yuan, Yu Hao, Hui Lin, Yu-Shen Liu, and Yi Fang. Zero-shot object navigation with vision- lisionllanguage models reasoning. arXiv preprint urXiv:2410. 18570, 2024.
[36] Halil Utku Unlu, Shuaihang Yuan, Congcong Wen, Hao Huang, Anthony Tzes, and Yi Fang. Reliable semantic understanding for real world zero-shot object goal navigation. arXiv preprint arXiv:2410.21926, 2024.
[37] Shuaihang Yuan, Halil Utku Unlu, Hao Huang, Congcong Wen, Anthony Tzes, and Yi Fang. Exploring the reliabillity of foundation model-based frontier selection in zero-shot object goal navigation. arXiv preprint arXiv:2410.21037, 2024.
[38] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXiv:100.066477 2017.
[39] Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near- pear-perfect pointgoal navigators from 2.5 billion frames. International Conference on Learning Representations, 2019.
[40] Austin Myers, Ching L Teo, Cornelia Fermüller, and Yiannis Aloimonos. Affordance detection of tool parts from geometric features. In IEEE International Conference on Robotics and Automation, pages 1374-1381. IEEE, 2015.
[41] OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[42] James A Sethian. A fast marching level set method for monotonically advancing fronts. proceedings of the National Academy of Sciences, 93(4):1591-1595 1996.
[43] Ziad Al-Halah, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 17031-17041, 2022.
[44] Santhosh Kumar Ramakrishnan, Devendra Singh Chaplot, Ziad Al-Halah, Jitendra Malik, and Kristen Grauman. Poni: Potential functions for objectgoal navigation with interaction- free learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18890-18900, 2022.
[45] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi. Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv: 1807.0677 2018.
[46] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. /2597, 2023.
[47] Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. In Workshop on Vision Laiguage Models for Navigation and Manipulation at ICRA, 2024.
13

---

A Difference with Existing Works
As shown in 5 the key difference between our method and other methods that we leverage geometrie parts and affordance information represent the environment, in addition using object- level category information as previous methods do. Furthermore, we utilize multi-scale feature representation to capture local features, enhancing the overall accuracy and robustness.
Table 5: The differences between our work and existing methods.
Training
Method
Mapping
Multi-Scale
Zero-shot
Locomotion
Semantic
SemExp
Categorical
X
ZSON 30
Categorical
X
X
PixNav
9
Language-Grounde Grounded
VLFM
16
Grounded
X
PONI 44
Categorical
L3MVN
19
Categorical
X
CoW
32
Categorical
X
ESC
Categorical
X
X
VoroNav
Categorical
X
GAMap
Affordance+Gemmottiic
B Experiment Setup
In our experiment, we adopt GPT-4V as the LLM to generate the geometric and affordance attributes. We set Na to and Ng to for the experiments process, we use three scaling levels in all our experiments: the first level is the original image, the second level has 4 equal-sized patches, and the third level has 16 equal-sized patches. We CLIP as the pre-trained visual and text encoder. Following the standard evaluation protocol [19], we use 2000 episodes the validation split of HM3D report the results. Similarly, follow this method to produce the results on the Gibson dataset. We use Titan XP GPU for the experiment evaluation, and the entire evaluation process takes around hours.
the HM3D and Gibson datasets. For the partition
C Effectiveness of Affordance and Geometric-part Attributes
provide the quantitative result for Figure in the main paper. Time is measured seconds.
Table 6: Quantitative results for the effectiveness of affordance and geometric-pan part attributes.
N.
Ng
SR
Time
N
N
SR
Time
0
2
0
42.6
0.152
0
42.1
0.094
2
46.4
0.175
44.7
0.152
49.1
0.210
0
3
47.5
0.175
2
3
50.3
0.248
0
41.2
0.094
3
44.0
0.175
45.7
0.152
3
47.2
0.210
1
48.8
0.175
3
2
49.2
0.248
50.2
0.210
3
50.3
0.290
D Result Visualizations
In this section, we visualize the navigation paths on both the Gibson and HM3D datasets, as shown in
Figures and Part of the visualization code is adapted from L3MVN [9].
14

---

E Time Complexity
To validate the efficiency of our method, we compare the FPS of our method to SemExp [2], L3MVN [16], and VLFM [19] in navigation tasks the HP3D dataset. The experimental results are shown Table 7 can observed that SemExp has the highest FPS, indicating the fastest processing speed. This is because it uses detection head and does not employ foundation model. However, SemExp has the lowest SR and SPL, indicating that despite its fast processing speed, it performs poorly in navigation accuracy and path efficiency. In contrast, L3MVN has the second -econd- FPS as it uses a lightweight foundation model. Although its processing speed is not as fast as SemExp, shows improved navigation accuracy and path efficiency, achieving SR of 76. % and SPL of 37.7%. On the other hand, VLFM has a lower FPS of only 2, but significantly improves SR and SPL., reaching 84.0% and 52.2%, respectively. This indicates that although VLFM has slower processing speed, has considerable advantages in navigation accuracy and path efficiency. Our model has the same FPS VLFM, both at 2, but further improves SR and SPL, reaching 85.7% and 55.5%, respectively. This demonstrates that our method maintains high navigation accuracy and path efficiency while providing comparable processing speed VLFM. These experimental results verify that our proposed method achieves good balance between time and accuracy.
Table 7: Comparison of different method's FPS on the HM3D dataset.
HM3D
Method
FPS
SR
SPI.1
SemExp
2
37.9
18.8
VLFM
16
3
52.5
30.4
L3MVN
19
2
50.4
23.1
GAMap
2
53.1
26.0
F Real-world Experiments
In our real-world experiment, we will evaluate the performance of four zero-shot object goal navi- gation algorithms, including L3MVN [19], COW (32], ESC (3. and VLFM (16), within a standard indoor apartment environment consisting of two bedrooms, two bathrooms, one kitchen, and one living room. The experiment adopts wheelwe robot. Specifically, Hiwonder equipped with an Intel Realsense D435i camera as our robot agent to navigate the environ- ment and locate specific target objects, including bed, toilet, table, sofa, and chair, without any prior knowledge of their locations. To ensure fair comparison, the starting position of the robots was kept consistent across all trials for each algorithm. The video demo be found on project page: https: / httpssaaaleaaaaaee www.:eeeuuunnnituu...e github. 10/ [o/GMMpp]
a JetAuto-Pro from
Figure 7: Visualizations of the last observation frame, navigation path, and GAMap.
15

---

Figure 8: Visualizations of last observation frame, navigation path, and GAMap.
G
Limitation and Future Work
While our proposed method, Geometric-Afffrdancce Maps (GAMap), has demonstrated significant improvements in zero-shot object goal navigation, it is important to acknowledge its limitations. Our approach relies heavily on the visual processing power of vision-lange Lssoun-ang models, i.e., CLIP. The effectiveness of GAMap also depends on the accuracy of geometrie parts and affordance attributes inferred by LLMs. Although the multi-scale scoring method enhances attribute detection, introduces additional complexity and computational overhead.
To address these limitations and further advance this research field, future work should optimize the integration of L.LMs and VLMs to reduce computational overhead, potentially through techniques like model distillation. Enhancing the accuracy of geometric and affordance attribute inference is crucial, and more powerful foundational models could improve this accuracy. Additionally, exploring better methods for aggregating these attributes is also an interesting research direction.
16

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We show the claim in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitation is included
17

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not have assumptions
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We include it in the appendix
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release our code upon the decision of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (bttps://nips.cr co/
public/ guides/ /wwdee//ooeeeuuineeeeee CodeeuumieSeeeee ssionPolicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https / //nips cc/ http:/cccccblcccuieeeeeeeeeeeeee Code Sumii as wisssooooiiyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
19

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We follow the standard testing details which stated in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The standard evaluation not include the error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This is included in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
20

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://neurrps... ottpr////iiiteprr///ooo.o.oooooo http::///aaarrre.ooooo...c////aa
Answer: [Yes]
Justification: We reviewed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: It is discussed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper has no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
21

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets are referenced.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperrs thcode com / da t. sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
22

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
23


Reviews:
[{"review_id": "IR2LuVYtGG", "reviewer": "Reviewer_suJv", "summary": "The paper proposes Geometric and Affordance Maps to tackle zero-shot object-goal navigation. It focuses on how to leverage VLMs to navigate toward objects of unfamiliar categories without prior training. The authors use GPT-4 to generate multiple attributes that describe affordance and geometry, and acquire attribute embeddings by CLIP text encoder. Besides, they propose a multi-scale scoring approach which divides each RGB image into multiple patches processed by CLIP visual encoder, to acquire multi-scale visual embeddings. Cosine similarity can be computed between those embeddings, resulting in a map used for navigation (e.g., applying Fast Marching Method based on the map). Experiments on HM3D and Gibson demonstrate improvements in success rates and success weighted by path length (SPL). The authors also provide an analysis on error pattern, which shows that detection error (missing the target object) is significantly reduced.", "questions": "1. Can the authors compare the proposed method with PIVOT[1], which can also leverage VLMs to tackle robotic navigation? PIVOT focuses more on VQA, while this paper focuses on the similarity between the text and visual embeddings generated by CLIP. CLIP may not capture enough details, and thus the authors may need to introduce a multi-scale scoring approach. It is interesting to see whether VLM models can directly capture enough multi-scale information.\n2. Can the authors also compare the time cost with baselines, like SemExp and VLFM?\n\n\n[1] PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs", "limitations": "The authors have addressed the limitations in the appendix.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly written. The proposed approach to leverage LLMs and VLMs to tackle zero-shot object-goal navigation is clean and simple.", "weaknesses": "1. The overall approach is limited by LLMs and VLMs. The analysis in Sec. 5.9 (Table 4) implies that the improvement mostly results from the reduced detection error. Since off-the-shelf LLMs and VLMs are used, the technical contribution of the paper seems to be limited by this fact. Although geometric and affordance attributes are proved effective in Figure 3, the conclusion might be limited to the specific VLM used in this paper, namely CLIP.\n2. It is better to replace Figure 3 with a table. It is not easy to tell the exact effect of $N_a$ and $N_g$.\n3. Some sentences in Sec. 5.1 are with typos (perhaps in a rush). For example, \"Gibson is generated by Al-Halah et al. [36] for agents with configuration A\". And the citations in L214 are messy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "---\n# Response to Weakness 1\nThank you to the reviewer for the feedback on our paper. We understand your concerns and would like to clarify and respond to them as follows:\n\n1. We would like to first highlight the main contributions of this paper: the Geometric and Affordance Map (GAMap) and the Multi-scale Geometric and Affordance Scoring Method. These innovations primarily address the challenges in semantic scene understanding during robotic open-vocabulary exploration in unknown environments, particularly those caused by occlusion or suboptimal viewing angles.\n\n2. In the analysis of Section 5.9 (Table 4), we indeed show significant improvements in reducing detection errors with our approach. These improvements are primarily attributed to the aforementioned GAMap and the multi-scale geometric and affordance scoring method. Specifically, as shown in Table 1 of the original paper, our method (CLIP+GAMap) significantly outperforms CoW (CLIP) with an increase in performance from 32 to 53.1. Additionally, our method (CLIP+GAMap) outperforms VLFM (BLIP-2), showing a performance improvement of 3.2 in SR. Furthermore, as demonstrated in the ablation studies in Table 2 of the original paper, the performance of BLIP-2+GAMap is better than that of BLIP-2.\nTherefore, we can draw the conclusion that the improvements in reducing detection errors are primarily attributed to the aforementioned GAMap and multi-scale geometric and affordance scoring method. Also, our GAMap and and multi-scale geometric and affordance scoring can be broadly applicable and not limited to a specific VLM, as evidenced by the performance improvement on different VLMs, such as CLIP, BLIP, and BLIP-2.\n\nWe sincerely appreciate the valuable feedback from the reviewer. We believe that through these clarifications and explanations, we can better showcase the technical contributions and innovations of our work.\n\n---\n# Response to Weakness 2\n\nThank you for your suggestion. We agree that converting Figure 3 into a table can more clearly demonstrate the specific contributions of Na and Ng to various performance metrics. We have redrawn the table as shown in ***Table b*** of the attached PDF. We will include this table in the revised version of the paper to better present our experimental results.\n\n---\n# Response to Weakness 3\n\nWe appreciate the reviewer's careful examination and understanding of our paper. We have revised Sec 5.1, correcting all spelling errors and citation format issues. \n\n---\n# Response to Question 1\n\nThank you to the reviewer for the valuable suggestions. Since PIVOT has not released its code, we have chosen to compare using the VLM employed in their paper, namely GPT4v. Given the difficulty of conducting a comprehensive quantitative comparison within a week, we have attempted to address whether VLM models can directly capture enough multi-scale geoemtric and affordance information through a qualitative example.\n\nThe experiment was designed as follows: we randomly selected a scene and compared the ability of GPT4v and our proposed CLIP with multi-scale scoring method to identify the target object. As shown in ***Figure a*** of the attached PDF, we input an image with a sofa located in a distant corner as the target object and compared the subsequent movement trajectories of the two algorithms.\nAs illustrated in the figure, our method successfully captures the small sofa back in the far corner, leveraging geometric part and affordance attributes to guide the exploration process. In contrast, GPT-4V failed to identify the object.\n\n---\n# Response to Question 2\n\nThank you for the reviewer’s suggestion. We selected three baselines, namely SemExp, L3MVN, and VLFM, and compared their FPS and performance metrics on navigation tasks in the HP3D dataset. The experimental results are shown in ***Table d*** of the attached PDF.\n\nIt can be observed that SemExp has the highest FPS, indicating the fastest processing speed. This is because it uses a detection head and does not employ a Foundation model. However, SemExp has the lowest SR and SPL, indicating that despite its fast processing speed, it performs poorly in navigation accuracy and path efficiency. In contrast, L3MVN has the second-highest FPS as it uses a lightweight foundation model. Although its processing speed is not as fast as SemExp, it shows improvements in navigation accuracy and path efficiency, achieving an SR of 76.1 and an SPL of 37.7. On the other hand, VLFM has a lower FPS of only 2, but it significantly improves SR and SPL, reaching 84.0 and 52.2, respectively. This indicates that although VLFM has a slower processing speed, it has considerable advantages in navigation accuracy and path efficiency. Our model has the same FPS as VLFM, both at 2, but further improves SR and SPL, reaching 85.7 and 55.5, respectively. This demonstrates that our method maintains high navigation accuracy and path efficiency while providing comparable processing speed to VLFM.\n\nThese experimental results verify that our proposed method achieves a good balance between time and accuracy. This indicates the excellent overall performance and practical application of our method. We will include these detailed experimental results and comparative analyses in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wGVVghg1hL", "reviewer": "Reviewer_KUEn", "summary": "The authors propose an algorithm for the object-goal navigation/exploration problem by using a 2D navigation map containing geometric and affordance scores for the target object. Geometric and affordance features for the target object are extracted by asking an LLM to list them for the object, and these words are then used with a multi-scale CLIP setup to get the resulting \"GA\" scores that are projected into a 2D map and accumulated over time.", "questions": "* Please clarify what kind of path planning and robot model you use, and in more detail why VLFM is not comparable.\n\n* Even assuming the VLFM \"locomotion\" is not comparable, could you not just use their 2D map component with your path planning? Both are 2D maps with scores.", "limitations": "Adequate except for the points above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The object navigation problem is important for robotics and embodied AI applications\n* The paper is mostly well-written and the experiments have ablations\n* The approach appears novel if somewhat narrow.", "weaknesses": "* Narrowness of contribution: Projecting CLIP-like open-vocab scores into a 2D navigation map has been done in earlier work like VLFM [17]. The novelty here appears to be that you ask an LLM for multiple (geometric, affordance) properties of the object to query CLIP with. \n\n* Motivation of baselines: The results only seem about 1% better than the recent VLFM paper, which you dismiss because it contains \"locomotion\" training. However, no mention is made how your robot does motion planning. Since your results measure path length, you have to use something to get to a high-scoring frontier without colliding. The digitized environments are also just referred to as \"data sets\", but there has to be a simulator running on top of them to simulate some type of robot moving through the environment. Some details on this may help motivate why you dismiss VLFM. \n\n* Reproducibility: The results are all with (and contributions hinges on) GPT4, but the back-end of GPT4 changes substantially over time and there is afaik no way to run a specific version of GPT4 (only major versions). This means the numbers in the paper may not be reproducible at the time of publication. It would be helpful to also include results with a specific version of an open-source LLM to at least have something reproducible.\n\n--------\nAfter rebuttal: The authors ran more experiments which addressed weaknesses #2 and #3. The quantitative improvements weren't as good as originally claimed but good enough. They even threw in a real robot experiment (video would also have been nice!)..The contribution is still a bit narrow but given the SOTA results on ObjNav it merits publication..", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "---\n# Response to Weakness 1\n\nWe acknowledge the contributions of existing works, such as VLFM, in utilizing CLIP-like models for navigation. VLFM primarily considers semantic relevance between all objects in the scene and the target object to generate value maps, which is indeed effective. \nDifferent from VLFM we propose **Geometric and Affordance Map (GAMap)** and the **Multi-scale Geometric and Affordance Scoring** to address the challenges in semantic scene understanding during open-vocabulary robotic exploration in unknown environments, caused by occlusion or partial viewing angles. Our approach tackles this issue on two levels: first, by representing objects using affordance and geometric part information; second, by dividing the robot's observed images into multi-scale patches. By correlating these patches with geometric parts and affordance information, the robot can more accurately infer and locate target objects from partial observations. In contrast, previous methods, which recognize object-level categories typically obtained from a relatively complete view of an object, struggle to identify the object from partial observations, especially when only a small part of the object is observed by the robot. \n\nIn addition to geometric part information, we propose leveraging affordance information for navigation. Affordances describe the possible actions that can be performed with an object, and affordances remain identifiable even from partial views, offering robustness to variations in appearance due to lighting or occlusions. Affordance for navigation has seldom been explored in previous literature. In our work, more concretely, affordance information (e.g., it can provide support) enhances the confidence in the existence of a chair in the area to be explored, thus increasing the exploration efficiency.\n\n---\n# Response to Weakness 2\n\nThank you for the detailed review and feedback. We understand your concerns. Below is our detailed response and clarification.\n\nWe mentioned that VLFM includes \"locomotion\" training because it employs a distributed deep reinforcement learning algorithm to train a PointNav policy to help the robot navigate to a designated waypoint. The authors trained their PointNav policy using scenes from the HM3D dataset with 4 GPUs, each with 64 workers, for 2.5 billion steps, taking around 7 days.\nIn our work, we utilize the heuristic search algorithm FMM for motion planning, as mentioned in Line 200 of the original paper, to find the shortest path from the robot's current location to the frontier. Additionally, we adopted the Habitat simulator but did not use any of its provided path planning algorithms. Instead, we implement FMM for path planning.\n\nWe compared the accuracy of VLFM and our method using the same motion planning algorithm, as suggested in your Question 2. For detailed results, please refer to our response to Question 2.\n\n---\n# Response to Weakness 3\n\nThank you for highlighting concerns regarding the reproducibility of our work. To address this, we selected LLAMA and LLAMA2 as open-source LLMs and conducted navigation experiments on HM3D and Gibson datasets. The results are shown in ***Table e*** of the attached PDF. We will include this table in the final version of the paper and release all code on GitHub.\n\n---\n# Response to Question 1\n\nThank you for your valuable feedback. Below is our detailed response to the concerns you raised:\n\n**Path Planning**\n\nOur method uses FMM to find the shortest path from the robot's current location to the designated target point.\n\n**Robot Model**\n\nWe employed a simulation environment based on the Habitat simulator and did not use provided path planning method.\n\n**Why VLFM is not comparable**\n\nAs mentioned in our response to Weakness 2, VLFM uses a trained PointNav policy for robot navigation. This training process requires a large amount of training data and computational resources and is complex and time-consuming. Specifically, it involves approximately 7 days of training on 4 GPUs, each with 64 workers, using the training split of the HM3D dataset.\n\nIn contrast, our method uses FMM, which does not need training, simpler to implement, and more computationally efficient. \n\nTo ensure a fair comparison, we designed a comparative experiment according to the suggestion mentioned in your Question 2. For detailed results, please refer to our response to Question 2.\n\n---\n# Response to Question 2\n\nThank you for the constructive suggestions. We also believe this is an excellent baseline to better demonstrate the effectiveness of our method. Following your suggestion, we use the Vision-Langauge Frontier Map proposed by VLFM for frontier selection and use FMM for path planning. The experimental results are shown in ***Table b*** of the attached PDF. It can be observed that, after removing the PointNav motion planning module, the VLFM variant shows performance decline in SR and SPL on both datasets. Our model shows improvements of 4.32% and 10.17% in SR and SPL on HM3D, and 3.50% and 14.43% on Gibson, respectively, compared to this new version of VLFM.\n\nThese results indicate that with the same path planning algorithm, our model still shows significant improvements in navigation performance compared to VLFM. This demonstrates that our proposed GAMap is more effective for robotic navigation tasks than the value map generated by VLFM. This further validates the innovation and superiority of our model.\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CGwKS1nXwW", "reviewer": "Reviewer_13Wq", "summary": "The paper presents an approach for the task of zero-shot object goal navigation. The approach first generates a geometric and affordance description of the goal object. This description is then matched with the visual embedding of the RGB frame (at multiple scales) in the CLIP embedding space to compute a similarity score. They project this score on an orthographic 2D map. During navigation, the agent explores the frontiers/unexplored regions with the highest score. They demonstrate the efficacy of their method on HM3D and Gibson object-goal benchmark.", "questions": "- The authors argue that comparison to VLFM is unfair because they use a trained policy compared to FMM to achieve high SPL on HM3D. Even then, they obtain better SPL on Gibson. Can the authors comment on why a trained policy will help HM3D, but not Gibson? \n- I didn't fully understand Figure 3. What does decrease in success rate mean? Is it relative or absolute decrease? Is it percentage or absolute points?", "limitations": "The authors don't discuss any limitation of their work in the manuscript. \n- Maybe the authors should add a small section discussing potential limitations such as lack of real-robot experiments in the paper. \n- assumptions about perfect odometry / localization to build a map. \n\nNot a limitation, but a suggestion to improve the manuscript: \n- I thought that the writing could be clearer. Some of the sentences were overly complicated. For example Line 42-43 - \"proposed method initiates with the LLM inferring the object’s geometric part attributes and potential\naffordances attributes, delving into a detailed understanding of the object’s physical structure and\nfunctional properties\". The paper will benefit from more succinct and clearer writing.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors present excellent ablations in the paper, justifying each design decision with ablations. They show ablations for using affordance and geometric attributes, multiple scaling levels, different visual encoders, and even different methods for calculating and merging GA scores! \n- The results (success-rate) on HM3D and Gibson object-goal navigation benchmark are better than the next best method (VLFM) (1.1% on ML3D, and 1.7% on Gibson).", "weaknesses": "- The idea of using attributes and longer descriptions for objects instead of just object label has been well studied [A, B] and I would have expected the authors to make connections to this popular line of work on effectively utilising CLIP representations. \n- I also thought that the 23.7% increase in SPL, and 37.0% in SPL claim is quite misleading specially because the only difference between them and VLFM is that VLFM uses a learned policy for navigation. It has been well established that using FMM for navigation works well enough for indoor navigation\n\n[A] Visual Classification via Description from Large Language Models, ICLR 2023. \n[B] LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions, 2023\n[C] Navigating to Objects in the Real World, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Weakness 1\n\nThank you to the reviewer for the valuable suggestions. \n\nWhile previous work has effectively utilized attributes and longer descriptions for object classification [A, B], our approach uniquely explores the role of attributes in navigation, particularly the use of affordance attributes. We demonstrate the distinct advantages of leveraging affordance in navigation, especially for dealing with partial observations obtained by a robot. Utilizing affordance attributes in exploration and navigation provides the robot with valuable hints, allowing it to prioritize exploring areas with a higher likelihood of containing the target objects with the desired affordance. We hope this explanation more clearly demonstrates the innovation and effectiveness of our method.\n\nWe hope this will more clearly demonstrate the innovation and effectiveness of our method. \n\n---\n\n# Response to Weakness 2\n\nThank you to the reviewer for the thorough review and feedback on our work. The percentages in Table 1 of the original text were compared against methods without locomotion and semantic training, which indeed may cause misunderstandings. We will remove the percentage improvements in the revised version.\n\nTo ensure a fairer comparison with VLFM, we redesigned a baseline according to reviewer KUEn's suggestion, where VLFM uses FMM for navigation instead of the trained policy. The specific results can be found in ***Table b*** of the attached PDF. It can be observed that our model shows improvements in SR and SPL on HM3D by 4.32% and 10.17%, respectively, and on Gibson by 3.50% and 14.43%, respectively, compared to this baseline. This further validates the effectiveness and advantages of our method.\n\nAdditionally, we compared our method with other FMM-based methods, as shown in Table 1 of the original paper, demonstrating the effectiveness of our proposed GAMap and multi-scale geometric and affordance scoring method.\n\n---\n\n# Response to Question 1\n\nThank you to the reviewer for the valuable suggestions. There are two main reasons for this:\n\n1. According to the description in section IV.D of the VLFM paper, it seems like the policy was trained only on the HM3D dataset and not on the Gibson dataset.\n\n2. The HM3D dataset is more complex and diverse compared to the Gibson dataset. A trained policy can learn complex environmental features and navigation strategies from extensive training data and scenes. Therefore, using a trained policy shows more significant improvement on the HM3D dataset than on the Gibson dataset.\n\nWe will include these detailed explanations in the revised version of the paper. Once again, thank you for your review and feedback.\n\n---\n# Response to Question 2\n\nThank you to the reviewer for the detailed review of our work. \nFigure 3 illustrates the impact of different numbers of geometric attributes and affordance attributes on time and performance. Specifically, we use the performance of three geometric attributes and affordance attributes as baselines, and the performance is measured using percentage (%).\n\nTo more clearly convey the Effectiveness of Affordance and Geometric Attributes, we have converted Figure 3 to a table, as shown in ***Table c*** of the attached PDF. We will replace it in the revised version of the paper to ensure readers can accurately understand the experimental results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pywlBChOUs", "reviewer": "Reviewer_fGDQ", "summary": "The paper is studying the problem of zero-shot object goal navigation. To solve the problems in traditional approaches, the authors propose a novel method, named Geometric and Affordance Maps (GAMap), which incorporates both geometric and affordance attributes as navigation guidance. A multi-scale scoring approach is proposed to cope with objects at different scales. Experiments on HM3D and Gibson datasets show the effectiveness of the proposed method for the metrics of success rates and success weighted by path length.", "questions": "It would be good if the authors could address my concerns in the weakness part. I will make my final decision after reading the authors' rebuttal and other reviewers' comments.", "limitations": "The authors have discussed about limitation in appendix.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is studying the problem of zero-shot object goal navigation, which I think is an important problem in object-goal navigation. \n\n2. The proposed method incorporating both geometric and affordance attributes is sensible to me. \n\n3. There are extensive analysis in the experiment section of the paper, which provides several insights.", "weaknesses": "1. Although the authors have demonstrated the difference of their work compared with previous ones, I am still not convinced about the key novelty of the work. In terms of organisation of the paper, it would be better if the authors could provide a table to better compare the difference of their work with existing works. \n\n2. There seems no real world experiments. I am not fully convinced about the effectiveness of the proposed method unless results of real robot experiments are provided.\n\n3. The authors have not removed the checklist instruction blocks.\n\n4. In Line 214, there is an issue with the citations. The paper needs further proofread.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Weakness 1\n\nThank you to the reviewer for the thorough review and feedback on our work. First, we would like to clarify the two key innovations of our work:\n\nOur main contribution is the proposed **Geometric and Affordance Map (GAMap)** and the **Multi-scale Geometric and Affordance Scoring** to address the challenges in semantic scene understanding during open-vocabulary robotic exploration in unknown environments, caused by occlusion or partial viewing angles. Specifically, our approach tackles this issue on two levels: first, by representing objects using affordance and geometric part information; second, by dividing the robot's observed images into multi-scale patches. By correlating these patches with geometric parts and affordance information, the robot can more accurately infer and locate target objects from partially observed attributes. In contrast, previous methods, which recognize object-level categories typically obtained from a relatively complete view of an object, struggle to identify the object from partial observations, especially when only a small part of the object is observed by the robot. \nIn addition to geometric part information, we propose leveraging affordance information for navigation. Affordances describe the possible actions that can be performed with an object, and affordances remain identifiable even from partial views, offering robustness to variations in appearance due to lighting or occlusions. Affordance for navigation has seldom been explored in previous literature. In our work, more concretely, affordance information (e.g., it can provide support) enhances the confidence in the existence of a chair in the area to be explored, thus increasing the exploration efficiency.\n\n---\n\nWe agree with your suggestion to use a table to illustrate the differences between our work and existing works. For this purpose, we have created ***Table a***, as shown in the attached PDF. The key difference between our method and other methods is that we leverage geometric part information and affordance information to represent the environment, in addition to using object-level category information as previous methods do. Furthermore, we utilize multi-scale feature representation to capture local features, enhancing the overall accuracy and robustness of our approach.\n\nBy clarifying these innovations and adding the comparative table, we hope to more clearly demonstrate our work's key novelties and further validate our research contributions.\n\n---\n# Response to Weakness 2\n\nThank you to the reviewer for the thorough review and feedback on our work. Due to the time constraints of the rebuttal period, we have only completed initial testing in an indoor environment for the task of zero-shot object goal navigation on a real-world robot. The test results are shown in the attached PDF in ***Figure b***.  As shown in clockwise order in ***Figure b***, at the beginning, the robot is placed at a random position, and the target object is a trash can. The robot begins to scan the environment using geometric parts and affordance information. However, the robot does not find any trash can-related geometric parts or affordance attributes in the bedroom. Then, the robot moves to the living room for further exploration. Here, it continues to utilize affordance information to prioritize the exploration area that more likely contains the target object. Next, the robot moves towards the kitchen and identifies the trash can there, marked with a red circle in the image, indicating the successful recognition of the target.\n\n---\n# Response to Weakness 3\n\nThank you for your thorough review. We will remove the checklist instruction blocks in the revised version.\n\n---\n# Response to Weakness 4\n\nThank you for your detailed suggestions. We will thoroughly proofread the entire paper and address issues such as the citation problem in Line 214. Our aim is to ensure accurate citations, fluent language, and clear structure throughout the paper. We will include these revisions in the updated version of the paper to enhance its quality and credibility.\n\n---\n# Response to Questions\n\nThank you very much for your comprehensive feedback and suggestions on our paper. We have provided point-by-point responses to your concerns in the weakness part and will include these clarifications in the final version of the paper. These include further explanations of the core innovations, a comparison table with existing work, and real-world experimental results. Additionally, we will thoroughly proofread the entire paper to address formatting, citation, and textual issues, thereby improving the overall quality.\n\n---\n\nWe aim to address all your concerns comprehensively, demonstrating the innovation and effectiveness of our method. We have also responded to the other reviewers' comments in detail, and we kindly request your further review. We believe we have thoroughly addressed all the feedback and hope this will lead to an improved evaluation. Thank you again for your time and constructive comments on this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

We appreciate all reviewers for their thorough reviews and valuable suggestions on our paper. First, we address three common concerns raised by all the reviewers here, and then we provide point-by-point responses to each reviewer's specific comments.

---

## 1. Novelty of the Paper
The main contributions of this paper are twofold: the **Geometric and Affordance Map (GAMap)** and the **Multi-scale Geometric and Affordance Scoring**. These contributions primarily address the challenges in semantic scene understanding during open-vocabulary robotic exploration in unknown environments, caused by occlusion or partial viewing angles. Our approach tackles this issue on two levels: first, by representing objects using affordance and geometric part information; second, by dividing the robot's observed images into multi-scale patches. By correlating these patches with geometric parts and affordance information, the robot can more accurately infer and locate target objects from partial observations. 

For instance, when a robot observes only a patch of a chair back, our method can infer the geometric part information (e.g., this is a chair back), thus identifying the object as a chair. In contrast, previous methods, which recognize object-level categories typically obtained from a relatively complete view of an object, struggle to identify the object from partial observations, especially when only a small part of the object is observed by the robot. 

In addition to geometric part information (e.g., this is a chair back), we propose leveraging affordance information (e.g., it can provide support) for navigation. Affordances describe the possible actions that can be performed with an object, and affordances remain identifiable even from partial views, offering robustness to variations in appearance due to lighting or occlusions. Affordance for navigation has seldom been explored in previous literature. In our work, more concretely, affordance information (e.g., it can provide support) enhances the confidence in the existence of a chair in the area to be explored, thus increasing the exploration efficiency.

Extensive experiments and ablation studies demonstrate the effectiveness of the aforementioned contributions, as shown in Table 1 to Table 4 and Figure 3 to Figure 5 of the original paper. Through these clarifications and explanations, we hope that our technical contributions and innovations are more clearly presented.

## 2.Comparison with VLFM
VLFM first generates an object-level language-grounded value map based on the semantic relevance of all the objects in the environment w.r.t the category of the target goal, and then uses this map to identify the most promising frontier to explore. In contrast, our approach leverages multi-scale geometric part and affordance information to construct a Geometric and Affordance Map (GAMap) for enhanced semantic scene understanding to guide robot exploration. In particular, our GAMap reveals the geometric part and affordance relevance of each object in the environment w.r.t the geometric part and affordance attributes of the target goal.  The differences between VLFM and our method make our method more robust in identifying partially visible objects.

Another difference lies in the path planning method. After obtaining the 2D value map, VLFM employs a distributed deep reinforcement learning algorithm, i.e., Variable Experience Rollout (VER), to train a Point Goal Navigation (PointNav) policy to help the robot navigate to a designated waypoint. The authors trained their PointNav policy using scenes from the training split of the HM3D dataset with 4 GPUs, each having 64 workers, for 2.5 billion steps, which took around 7 days. In contrast, after generating the GAMap, our method employs a heuristic search algorithm, i.e., Fast Marching Method (FMM), to find the shortest path from the robot’s current location to the designated waypoint. ***Unlike PointNav, which requires about a week of pre-training, our FMM does not require any training process.*** This makes our approach easier to apply to various environments with different conditions, as the trained Point Goal Navigation might not easily transfer to unseen environments. Additionally, FMM is simpler to implement and computationally more efficient.

Considering the different path planning methods adopted in our approach and VLFM, and based on **Reviewer KUEn**'s suggestion, we redesigned a comparative experiment with VLFM. Specifically, we kept the VLFM value map generation process unchanged and replaced its path planning method with FMM instead of the trained policy. The detailed results are shown in ***Table b*** of the attached PDF. The comparison reveals that our model outperforms this baseline by 4.32% and 10.17% in SR and SPL on HM3D, and by 3.50% and 14.43% in SR and SPL on Gibson, respectively. We hope this comparative experiment further validates the effectiveness and advantages of our GAMap compared to the vision-language frontier map proposed in VLFM.

## 3. Modification of Figure 3

To more clearly demonstrate the specific impacts of Na and Ng on various performance metrics, we have converted Figure 3 to a table, as shown in ***Table c*** of the attached PDF. We will include this table in the revised version of the paper.

---
Through the above explanations and experiments, we hope to address the major concerns raised by the reviewers and further demonstrate our contributions. We will continue to provide detailed responses to each reviewer's specific comments and reflect these improvements in the revised version. Once again, we thank all reviewers for their valuable feedback and suggestions.

Meta-review: This paper received divergent initial opinions with two reviewers in support of acceptance and two negative reviewers.  The main concerns by the negative reviewers revolved around narrowness and reproducibility of the contribution, missing time cost comparisons for the method and baselines, missing comparisons to prior work using VLMs for navigation, and missing experiments with real-world robot navigation.

The rebuttal responded to reviewer concerns, and one initially negative reviewer raised their opinion to accept.  Another negative reviewer acknowledged that some of their concerns were alleviated but remained opposed to acceptance citing the lack of convincing real-world robot experiments as the rationale.  That concern was not shared by any of the other reviewers, and in fact the authors do describe a real-world experiment in their rebuttal (albeit at a small scale).  Thus, the AC does not find a basis to overrule the reviewer consensus in favor of acceptance.  The AC recommends acceptance and strongly encourages the authors to incorporate clarifications and revisions to improve the final manuscript.
Decision: Accept (poster)