# ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models

Bingchen Gong, Diego Gomez, Abdullah Hamdi, Abdelrahman Eldesokey, Ahmed Abdelreheem, Peter Wonka, Maks Ovsjanikov

► To cite this version:

Bingchen Gong, Diego Gomez, Abdullah Hamdi, Abdelrahman Eldesokey, Ahmed Abdelreheem, et al.. ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models. ICCV 2025 - International Conference on Computer Vision, Oct 2025, Honolulu, United States. 10.48550/arXiv.2412.06292 . hal-05434062

HAL Id: hal-05434062

https://hal.science/hal-05434062v1

Submitted on 29 Dec 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
arXiv:2412.06292v1 [cs.CV] 9 Dec 2024

# ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models

Bingchen Gong¹

Diego Gomez¹

Abdullah Hamdi²

Abdelrahman Eldesokey³

Ahmed Abdelreheem³

Peter Wonka³

Maks Ovsjanikov¹

¹École Polytechnique

²Visual Geometry Group, University of Oxford

³King Abdullah University of Science and Technology (KAUST)

gongbingchen@gmail.com

<figure><img src="image_2.png" /><figcaption>Figure 1. **Zero-shot 3D Keypoint Detection.** Without any ground truth labels or supervised training, our method leverages the point-level reasoning embedded within MLLMs to extract and name salient keypoints on 3D models. The figure illustrates how our approach achieves competitive performance compared to CLIP-DINOiser [46] baselines, highlighting the potential of integrating language models with vision tasks for enhanced 3D shape understanding.</figcaption></figure>

## Abstract

We propose a novel zero-shot approach for keypoint detection on 3D shapes. Point-level reasoning on visual data is challenging as it requires precise localization capability, posing problems even for powerful models like DINO or CLIP. Traditional methods for 3D keypoint detection rely heavily on annotated 3D datasets and extensive supervised training, limiting their scalability and applicability to new categories or domains. In contrast, our method utilizes the rich knowledge embedded within Multi-Modal Large Language Models (MLLMs). Specifically, we demonstrate, for the first time, that pixel-level annotations used to train recent MLLMs can be exploited for both extracting and naming salient keypoints on 3D models without any ground truth labels or supervision. Experimental evaluations demonstrate

that our approach achieves competitive performance on standard benchmarks compared to supervised methods, despite not requiring any 3D keypoint annotations during training. Our results highlight the potential of integrating language models for localized 3D shape understanding. This work opens new avenues for cross-modal learning and underscores the effectiveness of MLLMs in contributing to 3D computer vision challenges. Our code is available at https://sites.google.com/view/zerokey.

## 1. Introduction

Multimodal Large Language Models (MLLMs) have been shown to seamlessly integrate visual and text representations with great success [3, 25–27, 32, 54]. Models like GPT-4 with vision capabilities, OFA [44], and Flamingo [3] have demonstrated the ability to process visual and textual
data, leading to advancements in vision-language under-
standing. Furthermore, related efforts have been made to
endow MLLMs with 3D or spatial reasoning [17, 47, 53].
When trained at scale, these MLLMs enable tackling a wide
array of complex tasks, ranging from comprehensive im-
age or 3D shape description to answering detailed questions
about visual content.

Unfortunately, despite this tremendous progress, existing MLLMs still struggle with tasks that require precise points or pixel-level reasoning, such as localization, counting, or salient keypoint detection [7, 36, 52]. This type of reasoning focuses on understanding and interpreting visual input at a fine-grained level using text. In general, we can observe an increased level of difficulty when going from complete objects to object parts and, finally, to specific points or small regions. Even advanced models like GPT-4o [32] and Claude Sonnet 3.5 [4], which have demonstrated impressive capabilities in various computer vision tasks, still struggle with point-level understanding.

This challenge arises because point-level reasoning requires models to accurately identify and analyze local details, such as landmarks on a face, joints in a human pose, or intricate components in a 3D shape. However, the architectures, training methods, and dataset annotations of most existing approaches are primarily designed to capture global visual properties and potentially link them to textual descriptions. Since text captions are most often crawled from the Internet, and thus rarely contain precise point or region-level annotations, point-level reasoning capabilities of most MLLMs remain very limited.

Addressing this problem necessitates a shift in how mod-
els are designed and trained. One potential solution is to use
more powerful visual encoders trained on real-world data
encompassing a variety of global and local features. Inter-
estingly, scaling alone to more crawled data or model com-
plexity does not seem to alleviate this problem [33]. In con-
trast, very recently, a family of MLLMs has been introduced
that was trained with specially-designed auxiliary training
data and tasks aimed at pixel-level image annotations along-
side global image captions [11]. When trained with this ad-
ditional data source, the resulting model (dubbed Molmo)
exhibits impressive localized reasoning abilities (counting,
localization, etc.) surpassing other MLLMs.

Inspired by these recent developments, we propose in-
vestigating MLLMs endowed with point-level reasoning in
the context of 3D shape understanding and specifically for
zero-shot keypoint detection. Given a 3D model from an
arbitrary category, our main task is to localize and name
salient keypoints on this model. Traditional methods for
3D keypoint detection heavily rely on annotated 3D datasets
and extensive supervised training [5, 45, 51]. This severely
limits their applicability to specific data-rich object cate-
gories. At the same time, recent general-purpose 3D LLMs

are typically trained with explicit objectives that align 3D
data representations with vision-language models [17], en-
abling global tasks like classification. However, since they
are based on standard vision-language models with limited
localization capabilities, none of these methods can accu-
rately perform 3D keypoint localization based on a language
prompt, see the supplementary material for more details.

Even in the presence of a powerful MLLM, we note that
localization or keypoint detection from images to 3D envi-
ronments presents a significant challenge. Since both detec-
tions and point-level captions tend to be noisy, developing
a robust approach requires reliable multi-view aggregation,
backprojection, and filtering mechanisms. In this paper, we
propose a comprehensive zero-shot 3D keypoint detection
method that exploits the point-level reasoning capabilities
of recent MLLMs [11] while integrating 3D consistency
and being completely category agnostic. To the best of our
knowledge, ours is the first robust method for zero-shot 3D
keypoint detection.

The applications of this approach are vast. This pa-
per demonstrates two significant applications: Schelling
points analysis and point describability and consistency.
Our evaluations demonstrate that, despite not having access
to ground truth data, our method can predict points sim-
ilar to those identified by human annotators by selecting
appropriate text prompts. Additionally, we developed two
text-image baselines for detecting 3D keypoints from text
prompts and conducted comparisons with these baselines.

To summarize, our main contributions are as follows:

* Identify the challenge of zero-shot point-level reasoning in 3D computer vision.
* Propose the first zero-shot 3D keypoint detection method that operates without 3D annotations or training and applies to arbitrary shape categories, establishing a baseline for zero-shot 3D keypoint detection in this new area.
* Evaluate several MLLMs in our context, highlighting the importance of auxiliary point-level training to solve the task at hand.

## 2. Related Work

2.1. Multimodal Large Language Models

The remarkable success of vision-language models like
CLIP [35] and large language models like GPT-3 [6] has
motivated the integration of multimodal capabilities into
large language models for visual understanding. Multi-
modal Large Language Models (MLLMs) have been ex-
tended to process images alongside text, achieving remark-
able success in vision-language tasks [3, 25–27, 32, 54].
Notable models such as GPT-4 with vision capabilities,
MiniGPT-4 [54], LLaVA [26, 27], Flamingo [3], and BLIP
[25] have demonstrated advanced abilities in integrating vi-
sual and textual information, enabling tasks ranging from
<figure><img src="image_3.png" /><figcaption>Figure 2. **ZeroKey Pipeline.** Our proposed ZeroKey employs MLLM Molmo for zero-shot keypoint detection on 3D objects by 1) rendering multiple views for a given shape, 2) leveraging MLLM reasoning in each view using point-specific prompts, and 3) aggregating the results through clustering, eliminating the need for annotated training data for 3D keypoint detection.</figcaption></figure>

detailed image captioning to complex visual reasoning. These models leverage large-scale pre-training on extensive image-text datasets to capture both global semantics and fine-grained details in 2D images. Despite their success with 2D data, extending these capabilities to 3D data remains a significant challenge due to the elevated cost and complexity of collecting large amounts of high-quality 3D data. Consequently, there is no dominant MLLM capable of processing 3D data with natural language understanding. To address this gap, recent efforts have focused on aligning models trained on small amounts of 3D data with large vision-language models. For instance, ULIP [48] seeks to align 3D models with vision-language representations to enhance cross-modal understanding. Similarly, Hong et al. [17] propose training a 3D Large Language Model by leveraging a pre-trained vision-language model, aiming to bridge the modality gap and facilitate better 3D comprehension through multimodal learning. While these “3D LLMs” are promising, the granularity we are targeting in work at the point-level 3D reasoning is still beyond their current capabilities [17, 48].

## 2.2. Keypoint Detection

Keypoint detection has been extensively studied in 3D due to its significance in shape representation, matching, and abstraction applications [8, 12, 20, 39, 41, 43, 49]. However, the definition of a keypoint has remained task-specific and subjective. Even human annotations of keypoints often differ, as highlighted in [9, 50]. Recent work [16] has shown that image diffusion models can develop an unsupervised understanding of keypoints as salient features. Additionally, it was demonstrated in [40] that the CLIP foundation model [35] can reasonably interpret keypoints with appropriate visual prompts. This paper investigates the understanding of keypoints as foundational elements of objects and shapes

within recent multimodal large language models (MLLMs).

## 2.3. Lifting from 2D to 3D

A number of studies have explored multi-view approaches to enhance 3D data analysis by leveraging 2D representations [10, 13, 15, 21, 22, 24, 28, 30, 31, 42]. A key challenge in this area is combining features from different viewpoints to represent local 3D points or voxels while preserving essential geometric details. Some methods address this by averaging features across views [22, 24], propagating labels between views [42], or learning from reconstructed neighboring points [21]. Others organize points within a unified grid structure [29] or merge multi-view features with 3D voxel information [10, 18]. VointNet [14] operates within the Voint cloud framework, retaining the original point cloud’s compactness and descriptive 3D properties. It employs learned aggregation of multi-view features applied independently to each point. Recently, there’s a growing trend of utilizing foundational models with zero-shot understanding capabilities—such as the Segment Anything Model [23]—to achieve zero-shot 3D comprehension by lifting 2D predictions into 3D space [1, 2, 19, 38]. Our work differs from the Zero-shot 3D segmentation in that our final goal is a fine-grained single 3D point representing an unseen text query on an unseen new 3D object. As we show below, existing vision-language models struggle with this task, thus necessitating a novel adapted solution.

## 3. Motivation

Localization and naming of points in an image or a 3D shape is an extremely challenging problem. Solving such a problem requires a localized understanding of the visual data and tight integration between text. Vision-Language Models (VLMs) are thus a strong candidate to tackle this problem. However, as mentioned above, most such models
are trained using global or object-level semantics which are
insufficient for point localization tasks. Furthermore, the re-
cent MLLMs that incorporate 3D data [17, 47, 53] are typi-
cally trained with explicit alignment against pre-trained tra-
ditional vision-language models, and thus inherit their lim-
itations in point-level reasoning. As a result, such models
are unable to solve the zero-shot keypoint detection prob-
lem, which is the main focus of our work.

Existing methods for the 3D keypoint detection problem
typically formulate the problem as either a supervised learn-
ing task, by exploiting the ground truth annotations, e.g., in
the KeypointNet dataset [50]. Alternatively, several few-
shot approaches have been developed, e.g., [45], which are
based on *transferring* known keypoints to new instances.

Rather than training an entirely new 3D model, our goal is to exploit the improved point-level reasoning capabilities of very recent VLMs and demonstrate their utility in 3D shape understanding tasks. We base our approach on the recent open-source MOLMO model [11], which incorporated a specially designed pixel-level annotation task into its training. Thanks to a well-chosen design and this additional curated dataset the model has been shown to perform significantly better than even the largest closed-source models in localized image understanding.

Our key motivation is to evaluate the effectiveness of
such a model in the context of *3D shape* understanding and
specifically for zero-shot keypoint detection. We first intro-
duce the problem of zero-shot keypoint detection and nam-
ing. Then, we develop a pipeline based on prompting the
model and multi-view aggregation. We show that the re-
sulting approach leads to unprecedented localized 3D shape
understanding, opening the door to a range of applications.
In this paper, we present the very first baseline for solving
the zero-shot 3D keypoint detection problem. We leverage
the powerful 2D spatially-aware VLM, Molmo [11], to gen-
erate precise 2D keypoint predictions from several points of
view. These points are back-projected into a 3D shape, and
after aggregation, we obtain a 3D keypoint prediction.

4. Method

**Problem setting and overview:** We consider the Zero-Shot Keypoint Detection problem for 3D shapes. Specifically, given a 3D shape, we aim to automatically generate a set of salient points corresponding to the shape's semantic parts. Our solution comprises three main components: first, we prompt a MLLM with the shape, asking the model to generate a list of names for possible candidate keypoints. Then, for each candidate, we ask the model to detect the precise coordinates of the point in a given image. Finally, we back-project those detected points into 3D and aggregate them to get the location of 3D keypoints in a given shape.

<figure><img src="image_4.png" /><figcaption>Figure 3. Comparing the ground truth KeypointNet dataset annotations (in red) to our method’s predictions (in blue). This figure showcases our results on the KeypointNet dataset, illustrating the effectiveness of our approach in keypoint detection. The close alignment of the red and blue dots demonstrates the effectiveness of our approach in accurately detecting keypoints and highlights its precision in point-level reasoning.</figcaption></figure>

4.1. Generating Text Candidates for Salient Points

To detect points using a MLLM, we first need to assign
names to these points. The definition of keypoints can vary
depending on the application. However, these points are
typically common to a given shape, salient, and carry signif-
icant semantic meaning. This information can be retrieved
from MLLMs by prompting the model with the shape class
in text form or by using an image of the shape when the
MLLM supports multimodal input. Given the shape class
label *c*, we prompt the MLLM to generate a list of candidate
keypoint names: $\mathcal{K} = \{k_1, k_2, \dots, k_N\}$, where *N* is the
number of keypoints generated. For example, for the class
"airplane," the MLLM might generate keypoint names such
as "nose," "wing tip," and "tail." In our experiments, we
utilize GPT-4o [32] as our MLLM backbone for this step.
We prompt it with a front-facing rendering of the 3D model
along with the request text: "*List possible salient key points (in text).*" The GPT-4o model is instructed to provide a can-
didate list of possible names for keypoints in JSON format.

4.2. Prompting Molmo to Detect 2D Keypoints

To detect the precise 2D coordinates of each candidate key-
point, we utilize Molmo [11], a state-of-the-art MLLM ca-
pable of localizing points in images based on natural lan-
guage prompts. Given a set of candidate keypoint names
$K = \{k_1, k_2, ..., k_N\}$ and a set of images (views) of the 3D shape $V = \{V_1, V_2, ..., V_M\}$, we prompt Molmo to detect the location of each keypoint in each image. We define the detection function as:

$$ \mathbf{p}_{i,j} = \text{Molmo}(V_j, k_i), i = 1, ..., N; j = 1, ..., M, \quad (1) $$

where $\mathbf{p}_{i,j} \in \mathbb{R}^2$ are the 2D coordinates of keypoint $k_i$ in image $V_j$ as predicted by Molmo. The prompt to Molmo consists of the image $V_j$ and the instruction to localize the keypoint $k_i$. For example:

> "Point to the left wing tip in this image."

This leverages Molmo's capability to understand natural language instructions and perform point-level localization.

### 4.3. Zero-Shot 3D Keypoint Detection

After obtaining the 2D keypoint detections from Molmo, we reconstruct the 3D coordinates of each keypoint in shape by back-projecting the 2D points into 3D space and aggregating the results from multiple views. Fig. 4 shows aggregated points from different numbers of views.

**Back-Projection of 2D Keypoints.** Given the camera projection matrices $\{C_j\}_{j=1}^M$ corresponding to each image $V_j$, we can back-project the 2D points $\mathbf{p}_{i,j}$ into 3D space. Assuming a pinhole camera model, each camera projection matrix $C_j$ maps 3D points $\mathbf{P} \in \mathbb{R}^3$ to 2D points $\mathbf{p} \in \mathbb{R}^2$:

$$ \mathbf{p}_{i,j} \sim C_j \mathbf{p}_{i,j}, \quad i = 1, ..., N; \quad j = 1, ..., M, \quad (2) $$

where $\sim$ denotes equality up to scale. To back-project $\mathbf{p}_{i,j}$ into 3D space, we compute the ray $\mathbf{r}_{i,j}(t)$ corresponding to $\mathbf{p}_{i,j}$:

$$ \mathbf{r}_{i,j}(t) = C_j^{-1}(\mathbf{p}_{i,j}, t), \quad (3) $$

where $t$ represents the depth along the ray. We intersect the ray with the 3D shape $S$ to find the 3D point $\mathbf{P}_{i,j}$:

$$ \mathbf{P}_{i,j} = \text{Intersect}(\mathbf{r}_{i,j}, S). \quad (4) $$

During the rendering process, we cache the depth $t$ for each pixel and utilize that information for back-projection, eliminating the need for expensive intersection calculations.

**Aggregation of 3D Keypoints.** For each of the $N$ keypoint $k_i$, we collect the set of back-projected 3D points from all views: $\mathcal{P}_i = \{\mathbf{P}_{i,j} | j = 1, ..., M\}$. To obtain the final 3D keypoint location $\hat{\mathbf{P}}_i$, we aggregate the set $\mathcal{P}_i$. A straightforward method is to compute the mean of the points:

$$ \hat{\mathbf{P}}_i = \frac{1}{|\mathcal{P}_i|} \sum_{j=1}^{M} \mathbf{P}_{i,j}. \quad (5) $$

Naive summarization often produces unsatisfactory results. This occurs because back-projection can become unstable when the angle of the ray-mesh intersection is sharp.

<figure><img src="image_5.png" /><figcaption>Figure 4. The number of rendered views versus the detected key-points after aggregation. This figure shows how varying the number of rendered views affects the total number of keypoints detected by ZeroKey. The prompt here is "corner of the table". As the number of views increases, ZeroKey will detect more key-points that suit the description of the prompt.</figcaption></figure>

Molmo [11] can also generate multiple keypoints for one prompt, resulting in ambiguity and inconsistencies in the summarized data.

**Filtering and Refinement.** To handle sharp angular intersections and stabilize the mapping process, we back-project a $h \times h$ patch $\mathbf{S}_{i,j}$ in the region centered at $\mathbf{P}_{i,j}$. We exclude all pixels that do not intersect with the mesh, then compute the mean of the back-projected points within the patch to obtain a stable back-projection of $\mathbf{P}_{i,j}$. Below is the revised back projection from Equation. 4:

$$ \mathbf{P}_{i,j} = \frac{1}{|\mathbf{S}_{i,j}|} \sum_{p,q \in \mathbf{S}_{i,j}} \text{Intersect}(\mathbf{r}_{p,q}, S). \quad (6) $$

where $\mathbf{S}_{i,j}$ represents the set of all pixels within the patch whose rays intersect with the mesh $S$, and $|\mathbf{S}_{i,j}|$ denotes the number of such pixels.

Patchify back projection effectively addresses the issue of sharp angular intersections. However, the predictions from Molmo can still yield multiple points for each prompt and may appear noisy across different views. One approach to improve this is to extract the confidence weighting from Molmo's feature map and apply it to the predictions. Nonetheless, normalizing the confidence scores across various views remains a challenge. In our method, we treat Molmo as a black box and employ hierarchical density-based spatial clustering on the predicted points obtained from multiple views. Specifically, we apply HDBSCAN to the set of predicted points $\mathcal{P}_i$ obtained from multiple views to identify dense clusters that represent consistent predictions across views. HDBSCAN operates by defining the mutual reachability distance $d_m$ between two points $\mathbf{P}_{i,j}$ and $\mathbf{P}_{i,l}$ as:

$$ d_m = \max(\text{core}_k(\mathbf{P}_{i,j}), \text{core}_k(\mathbf{P}_{i,l}), d(\mathbf{P}_{i,j}, \mathbf{P}_{i,l})), \quad (7) $$

where $\text{core}_k(\mathbf{P}_{i,j})$ is the core distance of point $\mathbf{P}_{i,j}$, defined as the distance to its $k$-th nearest neighbor, and $d(\mathbf{P}_{i,j}, \mathbf{P}_{i,l})$ is the Euclidean distance between points $\mathbf{P}_{i,j}$ and $\mathbf{P}_{i,l}$. By utilizing the mutual reachability distance and setting correct
<figure><img src="image_6.png" /><figcaption>Figure 5. We ask Molmo to describe the green point, and using this as a prompt ZeroKey predicts the blue point. We show that salient points, given by the Schelling Points paper [9], are more easily describable and consistently retrievable than arbitrary points. Some arbitrary points even lead to ZeroKey being unable to find any suitable points.</figcaption></figure>

minPts *k*, we effectively filter out the noise and identify the
most stable clusters in the predicted points.

5. Applications

Powerful VLMs, such as Molmo [11], have strong 2D spa-
tial reasoning abilities. We find that this reasoning gen-
eralizes to some degree to 3D. In this section, we explore
some applications that explore the 3D knowledge encoded
in Molmo.

5.1. Schelling Points on 3D Surface

Schelling points, as introduced by Schelling [37], are spe-
cific focal points that people are likely to choose indepen-
dently due to their prominence. In visual perception, these
keypoints stand out due to distinct features. We hypothesize
that the salient points people select often have clear, widely
understood names in language. Therefore, an effective vi-
sual language model (VLM) should be able to better predict
such salient points by leveraging its knowledge of both lan-
guage and vision.

To illustrate this idea, we conducted experiments using
a dataset [9] that captures human-selected Schelling points.
We utilized ZeroKey to generate a set of salient points for
various images. The prompts guiding ZeroKey were pro-
vided by ChatGPT, simulating natural language descrip-
tions that humans might use. This approach aimed to repli-
cate the human process of identifying and naming keypoints
based on visual and linguistic salience. An illustration of
this application is available in the supplementary material.

We find in practice that the prompts given by ChatGPT align
as expected with the most salient points reported by [9].

5.2. Point Describability and Consistency

During our experiments, we observed that certain keypoints
were more challenging to retrieve than others. This was
raised when performing our quantitative analysis on the
KeypointNet dataset [50], which provides ground truth key-
points for various 3D models. Assigning a concise and de-
scriptive text prompt to each ground truth point proved to
be part of the challenge. We noticed a correlation: when
a point was easy to name succinctly by a human observer,
ZeroKey was able to retrieve it more effectively. Based on
this intuition, we propose to study how describable different
points are and how consistently ZeroKey can retrieve them.

To quantify how “describable” a point is, we leverage
the distributions reported in the Schelling Points paper [9].
Points with high-density values in the distributions reported
by [9] are considered more describable due to their inherent
semantic significance. We refer to such points as salient
points. The experiment is executed as shown in Alg. 1.

The qualitative results of our experiments are showcased
in Fig. 5. For this experiment we ask Molmo to describe the
green dot, using a simple prompt. We then give the prompt
generated by Molmo to ZeroKey to predict a 3D location
of the original point. We can see that for salient points,
ZeroKey is able to leverage the prompt given by Molmo
to relocate the original queried point with good accuracy.
However, for other “random” non-salient points this tasks
is much more complicated. ZeroKey points to similar parts
**Algorithm 1** Shape Selection and Saliency Prediction

**Require:** Random shape *S*, and a point *p* ∈ *S*. ▷ Choose a point *p* ∈ *S* with 1) the highest reported saliency value; or 2) without semantic meaning (saliency value is 0).

**Ensure:** Absolute error shows that if *p* is the saliency point
1: Place a red marker at *p* on *S*, render several views.
2: Ask Molmo to describe the *p* with a short name *l*, re-
taining the most repeated prompt.
3: Render new views of *S* without any marker. ▷ Query
these views with prompt *l*.
4: Infer a 3D point prediction $\hat{p} \leftarrow \text{ZeroKey}(S, l)$.
5: Store the absolute error $e \leftarrow |\hat{p} - p|$.

<figure><img src="image_7.png" /><figcaption>Figure 6. We show through a quantitative study that “salient” points are retrieved with higher accuracy than “non-salient” ones, regardless of the distance threshold used.</figcaption></figure>

of the object, such as the airplane wing; or seemingly unre-
lated parts, like for the teddy bear. Moreover, we observed
a fail state where for some non-salient points Molmo was
unable to give a good prompt, resulting in ZeroKey affirm-
ing that no points matching the prompt were present in the
scene. Quantitative results can be found in Fig. 6. In this
figure, we see that the accuracy of retrieving a point from
a Molmo description using Molmo is much higher for a se-
mantic meaningful point than an arbitrary point.

6. Experiments

6.1. Setup and Dataset

We evaluate our method using the KeypointNet dataset. Our
evaluation strategy follows the approach of You et al. [51],
which computes the Intersection over Union (IoU) between
predicted keypoints and ground-truth keypoints from the
KeypointNet dataset, using varying distance thresholds. A
match is counted if the geodesic distance between a ground-
truth keypoint and a predicted keypoint is less than the spec-
ified threshold. We focus on the same three classes from
the dataset during our evaluation: airplane, chair, and ta-
ble. The labels in KeypointNet only provide integer IDs for

<figure><img src="image_8.png" /><figcaption>Figure 7. We compare against baselines CLIP-DINOiser and Red-
Circle. While both baselines identify some prominent regions,
they fall in accurately localizing keypoints according to the text
prompt. In contrast, our method effectively detects and precisely
locates keypoints.</figcaption></figure>

each keypoint. To detect keypoints from open vocabulary
textual descriptions and evaluate them using KeypointNet,
we manually annotated the labels of keypoint classes in the
dataset. We then used this annotated text as a textual prompt
for zero-shot 3D keypoint detection. Please refer to our sup-
plementary for the exact prompt we used in each category.

6.2. Zero-Shot 3D Keypoint Detection Baseline

**RedCircle.** The Red Circle method [40] is one of the simplest and most straightforward ways to highlight points in the image based on a text query by randomly sampling points and highlighting them with a red circle and picking the one point with the highest CLIP similarity to the text query. We lift the prediction of this method to 3D using the same lifting procedure used in our method to compare 3D Zero-shot keypoint detection.

**CLIP-DINOiser.** The CLIP-DINOiser [46] combines the strengths of CLIP and DINO to perform zero-shot object localization and segmentation based on text query in images. To adapt the CLIP-DINOiser for zero-shot 3D keypoint detection, we first apply the method to each view of the 3D scene to obtain 2D keypoint predictions corresponding to the text prompt. For each image, the CLIP-DINOiser produces a heatmap indicating the regions of interest related to the query. We select the point with the highest activation in the heatmap as the 2D keypoint prediction in that view. We then lift these 2D keypoints to 3D using the same back-projection technique described in our method.

**GPT-40.** In this Baseline, we replace our model with GPT-40 to observe variations in performance. Notably, we highlight that MLLMs trained solely on image-level tasks fail to provide useful signals for keypoint detection. In the GPT-40 version, we replace Molmo model in our method with the latest GPT-40 (2024-08-06) and evaluate its performance on point-level tasks. This comparison highlights the necessity of specialized training or architectural features that support fine-grained reasoning in visual contexts.
<table><caption>Table 1. Comparison of IoU between the predicted and ground-truth keypoints from KeypointNet using different methods across various geodesic distance thresholds.</caption><thead><tr><th rowspan="2">Method</th><th colspan="10">IoU (in %)</th></tr><tr><th>0.001</th><th>0.01</th><th>0.02</th><th>0.03</th><th>0.04</th><th>0.05</th><th>0.06</th><th>0.07</th><th>0.08</th><th>0.09</th><th>0.10</th></tr></thead><tbody><tr><td>Distance Thresholds</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>RedCircle [40]</td><td>0.21</td><td>0.34</td><td>0.64</td><td>1.16</td><td>1.90</td><td>3.05</td><td>4.81</td><td>7.55</td><td>11.06</td><td>14.92</td><td>18.50</td></tr><tr><td>GPT-4o</td><td>0.18</td><td>0.48</td><td>1.20</td><td>2.61</td><td>4.19</td><td>6.04</td><td>8.48</td><td>10.89</td><td>14.03</td><td>17.03</td><td>20.73</td></tr><tr><td>CLIP-DINOiser [46]</td><td>0.73</td><td>1.41</td><td>3.00</td><td>4.94</td><td>7.31</td><td>9.80</td><td>12.66</td><td>15.52</td><td>18.52</td><td>21.76</td><td>25.56</td></tr><tr><td><strong>ZeroKey (Ours)</strong></td><td><strong>3.69</strong></td><td><strong>5.84</strong></td><td><strong>10.91</strong></td><td><strong>17.58</strong></td><td><strong>23.72</strong></td><td><strong>29.81</strong></td><td><strong>35.52</strong></td><td><strong>40.82</strong></td><td><strong>45.72</strong></td><td><strong>50.24</strong></td><td><strong>54.64</strong></td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Figure 8. Comparison of the performance across different configurations: (blue) our original method; (green) results with a Global Text prompt; (red) results using GPT-4 as the MLLM backbone; and (yellow) results without HDBSCAN clustering. This figure shows the impact of each modification on the overall performance.</figcaption></figure>

## 6.3. Quantitative and Qualitative Analysis

In our evaluation of KeypointNet, our Zero-Shot method significantly outperforms the VLM-based baselines across all distance thresholds. It achieves IoU levels that are comparable to those of reference-based Few-Shot methods and supervised methods specifically tailored for this dataset. Qualitative results are shown in Table 1. In Fig. 3 we show side by side comparisons between some ground truth keypoints and our Zero-Shot prediction.

## 6.4. Ablation Studies

In this ablation study, we modify different components of our method to gain deeper insights into the point-level reasoning abilities of MLLMs. Specifically, we compare the effects of various text prompts, experiment with different MLLM backbones, and evaluate the impact of aggregating information from different numbers of views.

**Global Text prompt.** We evaluate the effectiveness of a point-specific text prompt. The original method utilizes prompts tailored to specific points in the image, providing precise guidance to the model. In the ablated version, we replace these point-specific text prompts with a global prompt: “Point to all salient points in this image.” We then compare the detected set of keypoints with the ground

<figure><img src="image_10.png" /><figcaption>Figure 9. Renderings from the ablation study where the MLLM backbone is replaced with GPT-4o. As an MLLM trained solely on image-level tasks, it struggles with point-level reasoning, leading to less accurate and consistent keypoints compared to Molmo.</figcaption></figure>

truth to assess how this change affects the model's ability to localize specific points. This experiment helps us understand the importance of prompt specificity in guiding the MLLM's attention to particular regions of the image.

**Aggregation ablation.** We explore aggregating information from multiple views to enhance the model's understanding of the scene. Our original method aggregates keypoint information from 26 viewpoints to improve accuracy. In the 6-views variant, we reduce the number of viewpoints to 6. In another variant, we compare the performance of direct average against HDBSCAN clustering for aggregating detections. By evaluating these different aggregation strategies, we aim to determine how the number of views and the method of combining information influence the model's ability to accurately identify keypoints.

The ablation results shown in Fig. 8 illustrate the relative performance compared with ZeroKey. The ability to achieve 80% performance with only 6 views indicates that our method efficiently aggregates points. The poor performance of direct averaging suggests the necessity of HDBSCAN clustering. Fig. 9 demonstrates that GPT-4o fails to precisely locate the keypoint. This shows the need for an MLLM backbone trained with point-level tasks for precise keypoint detection.

## 7. Conclusion and Future Work

In this paper, we presented a novel zero-shot approach for 3D keypoint detection by harnessing the capabilities of recent MLLMs trained with enhanced localized reasoning capabilities. Unlike traditional methods that rely heavily on annotated 3D datasets and extensive supervised training,
our method leverages the pixel-level annotations inherent in MLLMs to both extract and name salient keypoints without any ground truth labels or supervision. Our experimental evaluations demonstrate that our approach achieves remarkable performance. We believe that this work opens the door to entirely novel applications by providing a bridge between 3D data and MLLMs with localized reasoning. This can help solve downstream tasks such as shape manipulation, deformation, and processing, which typically require significant domain knowledge. Furthermore, this link can also help to enhance the power of MLLMs by providing feedback thanks to multiview consistency from 3D data, thus potentially helping to improve the robustness of their pixel-level understanding.

**Acknowledgments.** Parts of this work were supported by the ERC Consolidator Grant 101087347 (VEGA) and the ANR AI Chair AIGRETTE. This work was also supported by the KAUST Center of Excellence for Generative AI, under award number 5940, and the KAUST Ibn Rushd Postdoc Fellowship program.

## References

[1] Ahmed Abdelreheem, Abdelrahman Eldesokey, Maks Ovsjanikov, and Peter Wonka. Zero-shot 3d shape correspondence. In *SIGGRAPH Asia 2023 Conference Papers*, pages 1–11, 2023. 3

[2] Ahmed Abdelreheem, Ivan Skorokhodov, Maks Ovsjanikov, and Peter Wonka. Satr: Zero-shot semantic segmentation of 3d shapes. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 15166–15179, 2023. 3

[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. *Advances in neural information processing systems*, 35:23716–23736, 2022. 1, 2

[4] AI Anthropic. Claude 3.5 sonnet model card addendum. *Claude-3.5 Model Card*, 2024. 2

[5] Souhaib Attaiki and Maks Ovsjanikov. Ncp: Neural correspondence prior for effective unsupervised shape matching. *Advances in Neural Information Processing Systems*, 35:28842–28857, 2022. 2

[6] Tom B Brown. Language models are few-shot learners. *arXiv preprint arXiv:2005.14165*, 2020. 2

[7] Declan Campbell, Sunayana Rane, Tyler Giallanza, Nicolò De Sabbata, Kia Ghods, Amogh Joshi, Alexander Ku, Steven M Frankland, Thomas L Griffiths, Jonathan D Cohen, et al. Understanding the limits of vision language models through the lens of the binding problem. *arXiv preprint arXiv:2411.00238*, 2024. 2

[8] Nenglun Chen, Lingjie Liu, Zhiming Cui, Runnan Chen, Duygu Ceylan, Changhe Tu, and Wenping Wang. Unsupervised learning of intrinsic structural representation points. In

*Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 9121–9130, 2020. 3

[9] Xiaobai Chen, Abulhair Saparov, Bill Pang, and Thomas Funkhouser. Schelling points on 3d surface meshes. *ACM Transactions on Graphics (TOG)*, 31(4):1–12, 2012. 3, 6, 12, 13

[10] Angela Dai and Matthias Nießner. 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation. In *Proceedings of the European Conference on Computer Vision (ECCV)*, pages 452–468, 2018. 3

[11] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. *arXiv preprint arXiv:2409.17146*, 2024. 2, 4, 5, 6

[12] Clara Fernandez-Labrador, Ajad Chhatkuli, Danda Pani Paudel, Jose J Guerrero, Cédric Demonceaux, and Luc Van Gool. Unsupervised learning of category-specific symmetric 3d keypoints from point sets. In *Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16*, pages 546–563. Springer, 2020. 3

[13] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 1–11, 2021. 3

[14] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem. Voint cloud: Multi-view point cloud representation for 3d understanding. In *The Eleventh International Conference on Learning Representations*, 2023. 3

[15] Abdullah Hamdi, Faisal AlZahrani, Silvio Giancola, and Bernard Ghanem. Mvtn: Learning multi-view transformations for 3d understanding. *International Journal of Computer Vision*, 2024. 3

[16] Eric Hedlin, Gopal Sharma, Shweta Mahajan, Xingzhe He, Hossam Isack, Abhishek Kar, Helge Rhodin, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised keypoints from pretrained diffusion models. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 22820–22830, 2024. 3, 13

[17] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. *Advances in Neural Information Processing Systems*, 36:20482–20494, 2023. 2, 3, 4

[18] Ji Hou, Angela Dai, and Matthias Nießner. 3d-sis: 3d semantic instance segmentation of rgb-d scans. In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition*, pages 4421–4430, 2019. 3

[19] Rui Huang, Songyou Peng, Ayca Takmaz, Federico Tombari, Marc Pollefeys, Shiji Song, Gao Huang, and Francis Engelmann. Segment3d: Learning fine-grained class-agnostic 3d segmentation without manual labels. *European Conference on Computer Vision (ECCV)*, 2024. 3

[20] Tomas Jakab, Richard Tucker, Ameesh Makadia, Jiajun Wu, Noah Snavely, and Angjoo Kanazawa. Keypointdeformer:
Unsupervised 3d keypoint discovery for shape control. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 12783–12792, 2021. 3
[21] Maximilian Jaritz, Jiayuan Gu, and Hao Su. Multi-view pointnet for 3d scene understanding. In *Proceedings of the IEEE International Conference on Computer Vision Workshops*, pages 0–0, 2019. 3
[22] Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 3d shape segmentation with projective convolutional networks. In *proceedings of the IEEE conference on computer vision and pattern recognition*, pages 3779–3788, 2017. 3
[23] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023. 3
[24] Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, and Caroline Pantofaru. Virtual multi-view fusion for 3d semantic segmentation. In *European Conference on Computer Vision (ECCV)*, pages 518–535. Springer, 2020. 3
[25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In *International conference on machine learning*, pages 19730–19742. PMLR, 2023. 1, 2
[26] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In *NeurIPS*, 2023. 2
[27] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 26296–26306, 2024. 1, 2
[28] Zhengzhe Liu, Xiaojuan Qi, and Chi-Wing Fu. 3d-to-2d distillation for indoor scene parsing. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 4464–4474, 2021. 3
[29] Yecheng Lyu, Xinming Huang, and Ziming Zhang. Learning to segment 3d point clouds in 2d image space. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 12255–12264, 2020. 3
[30] Jinjie Mai, Abdullah Hamdi, Silvio Giancola, Chen Zhao, and Bernard Ghanem. Egoloc: Revisiting 3d object localization from egocentric videos with visual queries. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 45–57, 2023. 3
[31] Jinjie Mai, Wenxuan Zhu, Sara Rojas, Jesus Zarzar, Abdullah Hamdi, Guocheng Qian, Bing Li, Silvio Giancola, and Bernard Ghanem. Tracknerf: Bundle adjusting nerf from sparse and noisy views via feature tracks, 2024. 3
[32] OpenAI. Gpt-4 technical report, 2023. 1, 2, 4
[33] Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, and Zuxuan Wu. Synthesize diagnose and optimize: Towards fine-grained vision-language understanding. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 13279–13288, 2024. 2
[34] Zekun Qi, Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge, Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodied interaction. In *Computer Vision – ECCV 2024: 18th European Conference*, Milan, Italy, September 29–October 4, 2024, *Proceedings*, Part XLIII, page 214–238, Berlin, Heidelberg, 2024. Springer-Verlag. 12

[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language super-vision. In *ICML*, pages 8748–8763. PMLR, 2021. 2, 3
[36] Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. *arXiv preprint arXiv:2407.06581*, 2024. 2
[37] Thomas C Schelling. *The Strategy of Conflict: with a new Preface by the Author*. Harvard university press, 1980. 6
[38] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe. Mask3D: Mask Transformer for 3D Semantic Instance Segmentation. In *International Conference on Robotics and Automation (ICRA)*, 2023. 3
[39] Ruoxi Shi, Zhengrong Xue, Yang You, and Cewu Lu. Skeleton merger: an unsupervised aligned keypoint detector. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 43–52, 2021. 3
[40] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? Visual prompt engineering for vlms. In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 11987–11997, 2023. 3, 7, 8, 13
[41] Supasorn Suwajanakorn, Noah Snavely, Jonathan J Tompson, and Mohammad Norouzi. Discovery of latent 3d key-points via end-to-end geometric reasoning. *Advances in neural information processing systems*, 31, 2018. 3
[42] Brian H Wang, Wei-Lun Chao, Yan Wang, Bharath Hariharan, Kilian Q Weinberger, and Mark Campbell. Ldls: 3d object segmentation through label diffusion from 2d images. *IEEE Robotics and Automation Letters*, 4(3):2902–2909, 2019. 3
[43] Hanyu Wang, Jianwei Guo, Dong-Ming Yan, Weize Quan, and Xiaopeng Zhang. Learning 3d keypoint descriptors for non-rigid shape matching. In *Proceedings of the European Conference on Computer Vision (ECCV)*, pages 3–19, 2018. 3
[44] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In *International conference on machine learning*, pages 23318–23340. PMLR, 2022. 1
[45] Thomas Wimmer, Peter Wonka, and Maks Ovsjanikov. Back to 3d: Few-shot 3d keypoint detection with back-projected 2d features. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 4154–4164, 2024. 2, 4, 13
[46] Monika Wysoczanska, Oriane Siméoni, Michaël Ramamon-jisoa, Andrei Bursuc, Tomasz Trzcinski, and Patrick Pérez. Clip-dinoiser: Teaching clip a few dino tricks for open-
vocabulary semantic segmentation. In Proc. ECCV, 2024.
1, 7, 8, 13

[47] Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiang-miao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. In *European Conference on Computer Vision*, pages 131–147. Springer, 2024. 2, 4

[48] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 1179–1189, 2023. 3

[49] Zi Jian Yew and Gim Hee Lee. 3dfeat-net: Weakly supervised local 3d features for point cloud registration. In *Proceedings of the European conference on computer vision (ECCV)*, pages 607–623, 2018. 3

[50] Yang You, Yujing Lou, Chengkun Li, Zhoujun Cheng, Liangwei Li, Lizhuang Ma, Cewu Lu, and Weiming Wang. Keypointnet: A large-scale 3d keypoint dataset aggregated from numerous human annotations. *arXiv preprint arXiv:2002.12687*, 2020. 3, 4, 6

[51] Yang You, Wenhai Liu, Yanjie Ze, Yong-Lu Li, Weiming Wang, and Cewu Lu. Ukpgan: A general self-supervised keypoint detector. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2022. 2, 7

[52] Chenhui Zhang and Sherrie Wang. Good at captioning bad at counting: Benchmarking gpt-4v on earth observation data. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops*, pages 7839–7849, 2024. 2

[53] Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tiejun Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. *arXiv preprint arXiv:2310.06773*, 2023. 2, 4

[54] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. *arXiv preprint arXiv:2304.10592*, 2023. 1, 2
# ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models

## Supplementary Material

<figure><img src="image_11.png" /><figcaption>Figure 10. Qualitative comparison of keypoint localization using ShapeLLM on out-of-distribution 3D airplane shape. ShapeLLM struggles to accurately identify the shape and point to keypoints on this unseen pointcloud data.</figcaption></figure>

### A. 3D Multimodal Language Models

Recent general-purpose 3D language models (LLMs) are usually trained with specific objectives that align 3D data representations with vision-language models, allowing for global tasks like classification. However, these models are based on standard vision-language frameworks that have limited localization capabilities. Consequently, they are unable to accurately perform 3D keypoint localization based on language prompts. We evaluated the KeypointNet model against the state-of-the-art ShapeLLM [34]. The qualitative results are shown in Fig. 10. We observed that 3D LLMs are often sensitive to the alignment and rotation of the input point cloud and do not generalize well to new datasets. Even with perfectly aligned point clouds, we found that ShapeLLM still struggled to accurately understand out-of-distribution shapes and localize keypoints when the shapes were not seen during its training phase. These limitations highlight the challenges faced by current 3D language models in precise localization and generalization to unseen data in zero-shot settings.

<figure><img src="image_12.png" /><figcaption>Figure 11. Superposition of the keypoints predicted by ZeroKey (blue) and the Schelling point distribution [9] (red).</figcaption></figure>

### B. Schelling Points

We make the hypothesis that the salient points selected by individuals in visual scenes often correspond to objects or features with clear, identifiable names in natural language. This leads us to believe that an effective Visual Language Model (VLM) should be capable of predicting such salient points by leveraging its combined understanding of language and vision.

To test this hypothesis, we conducted experiments using the dataset from Chen et al. [9], which captures human-selected Schelling points in various visual scenarios. In the context of coordination games, Schelling points refer to prominent locations or choices that people commonly select without prior communication, relying on mutual expectations and shared cultural references.

In Fig. 11, we showcase a visualization that superposes the distributions reported by the Schelling Points paper [9] to a set of salient points generated by ZeroKey, where the candidate keypoints were provided by ChatGPT using the following prompt:

> "List possible salient keypoints (in text)."

Besides text prompt, we also input renderings of the 3D
<table><caption>Table 2. Comparison of IoU between the predicted and ground-truth keypoints from KeypointNet using different methods across various geodesic distance thresholds. The IoU of our method is highlighted in bold.</caption><thead><tr><th rowspan="2">Method</th><th colspan="10">IoU (in %)</th></tr><tr><th>0.001</th><th>0.01</th><th>0.02</th><th>0.03</th><th>0.04</th><th>0.05</th><th>0.06</th><th>0.07</th><th>0.08</th><th>0.09</th><th>0.10</th></tr></thead><tbody><tr><td>Distance Thresholds</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>HARRIS-3D</td><td>0.15</td><td>0.76</td><td>2.19</td><td>3.96</td><td>6.16</td><td>8.88</td><td>11.91</td><td>15.13</td><td>18.63</td><td>22.10</td><td>25.69</td></tr><tr><td>SIFT-3D</td><td>0.29</td><td>1.05</td><td>2.62</td><td>4.38</td><td>6.65</td><td>9.42</td><td>12.38</td><td>15.65</td><td>19.39</td><td>22.71</td><td>26.15</td></tr><tr><td>ISS</td><td>0.49</td><td>1.10</td><td>2.69</td><td>4.67</td><td>7.27</td><td>10.14</td><td>13.29</td><td>16.56</td><td>20.02</td><td>23.58</td><td>27.20</td></tr><tr><td>USIP</td><td>0.83</td><td>1.70</td><td>3.25</td><td>5.24</td><td>8.00</td><td>11.76</td><td>15.98</td><td>20.63</td><td>25.56</td><td>30.20</td><td>33.67</td></tr><tr><td>D3FEAT</td><td>2.36</td><td>3.86</td><td>7.27</td><td>10.37</td><td>13.37</td><td>17.12</td><td>21.41</td><td>26.12</td><td>30.39</td><td>34.82</td><td>38.41</td></tr><tr><td>UKPGAN</td><td>3.95</td><td>6.54</td><td>12.27</td><td>17.86</td><td>22.42</td><td>26.55</td><td>30.28</td><td>34.32</td><td>38.46</td><td>42.65</td><td>46.49</td></tr><tr><td>FPS</td><td>6.34</td><td>10.55</td><td>17.68</td><td>23.18</td><td>26.91</td><td>30.27</td><td>33.57</td><td>36.85</td><td>40.47</td><td>44.42</td><td>48.38</td></tr><tr><td>FSKD</td><td>7.00</td><td>7.94</td><td>11.17</td><td>16.74</td><td>23.99</td><td>31.14</td><td>38.14</td><td>43.97</td><td>49.30</td><td>53.62</td><td>57.03</td></tr><tr><td>B2-3D (62 views)</td><td>12.41</td><td>20.29</td><td>35.55</td><td>46.25</td><td>52.74</td><td>57.72</td><td>61.61</td><td>64.19</td><td>66.56</td><td>68.55</td><td>70.57</td></tr><tr><td>B2-3D (26 views)</td><td>8.60</td><td>14.39</td><td>25.10</td><td>33.31</td><td>39.29</td><td>44.11</td><td>47.85</td><td>50.83</td><td>53.17</td><td>55.11</td><td>56.90</td></tr><tr><td>StablePoints [16]</td><td>3.66</td><td>5.80</td><td>9.94</td><td>13.32</td><td>16.58</td><td>19.91</td><td>23.48</td><td>27.26</td><td>31.28</td><td>34.89</td><td>38.22</td></tr><tr><td>RedCircle [40]</td><td>0.21</td><td>0.34</td><td>0.64</td><td>1.16</td><td>1.90</td><td>3.05</td><td>4.81</td><td>7.55</td><td>11.06</td><td>14.92</td><td>18.50</td></tr><tr><td>GPT-4o</td><td>0.18</td><td>0.48</td><td>1.20</td><td>2.61</td><td>4.19</td><td>6.04</td><td>8.48</td><td>10.89</td><td>14.03</td><td>17.03</td><td>20.73</td></tr><tr><td>CLIP-DINOiser [46]</td><td>0.73</td><td>1.41</td><td>3.00</td><td>4.94</td><td>7.31</td><td>9.80</td><td>12.66</td><td>15.52</td><td>18.52</td><td>21.76</td><td>25.56</td></tr><tr><td><strong>ZeroKey (Ours)</strong></td><td><strong>3.69</strong></td><td><strong>5.84</strong></td><td><strong>10.91</strong></td><td><strong>17.58</strong></td><td><strong>23.72</strong></td><td><strong>29.81</strong></td><td><strong>35.52</strong></td><td><strong>40.82</strong></td><td><strong>45.72</strong></td><td><strong>50.24</strong></td><td><strong>54.64</strong></td></tr></tbody></table>

scene to ChatGPT to obtain prompts simulates natural language descriptions that humans are likely to use. This process can simulate the way humans might describe these scenes in natural language, capturing the linguistic cues that people are likely to use when identifying salient features.

We see in this image that some of the salient points [9] do correspond to the predictions made by ZeroKey. Our findings reveal that several of the salient points identified by humans correspond to the predictions made by ZeroKey. These results support our hypothesis that salient points in visual scenes are closely tied to language and that VLMs that harness this connection can more effectively predict human-selected points of interest.

## C. More Quantitative Analysis

In this supplementary document, we present additional quantitative evaluations. We compared our Zero-Shot method with both supervised and few-shot methods trained on KeypointNet. Our results demonstrate that the Zero-Shot method can even outperform some of the supervised methods. The IoU levels achieved by our method are comparable to those of the reference-based Few-Shot methods and the supervised methods specifically designed for this dataset. Qualitative results are provided in Table 2. Our method is evaluated using 26 rendered views, and we have also included the results from the same number of views of the few-shot method B2-3D as a reference.

In addition to the supervised and few-shot methods, we also compared our approach with an unsupervised baseline called StablePoint [16]. This method learns keypoints by optimizing text embeddings from latent diffusion models. The main idea is to identify text embeddings that guide the generative model to consistently focus on compact regions within images, which are then used as keypoints. Unlike our

approach, this baseline does not retrieve keypoint according to a text prompt and operates fully unsupervised. However, it does necessitate specifying the exact number of keypoints we wish to detect. In our experiments, we set the number of keypoints to match the number of ground-truth points in the KeypointNet dataset.

## D. Keypoint Descriptions for Categories

In this supplementary material, we present a comprehensive text prompt outlining key points for 16 common object categories used in KeypointNet. Each object category is defined by a specific set of semantically meaningful manually annotated key points that capture its structural and functional characteristics. These key points are carefully selected to represent distinctive geometric features and important structural elements of each object. However, in KeypointNet, these key points are not accompanied by a text description; instead, they are assigned a unique semantic index. Some of these labels are derived from the source referenced in [45]. This information may also be useful for others aiming to detect key points based solely on textual descriptions.
<table><thead><tr><th>Object</th><th>Index</th><th>Description</th><th>Short Description</th></tr></thead><tbody><tr><td rowspan="11">Table</td><td>0</td><td>corner of the table top</td><td>different corners</td></tr><tr><td>1</td><td>corner of the table top</td><td>different corners</td></tr><tr><td>2</td><td>corner of the table top</td><td>different corners</td></tr><tr><td>3</td><td>corner of the table top</td><td>different corners</td></tr><tr><td>4</td><td>end of the table leg</td><td>end of the legs</td></tr><tr><td>5</td><td>end of the table leg</td><td>end of the legs</td></tr><tr><td>6</td><td>end of the table leg</td><td>end of the legs</td></tr><tr><td>7</td><td>end of the table leg</td><td>end of the legs</td></tr><tr><td>8</td><td>end of the stretcher between the table legs</td><td>-</td></tr><tr><td>9</td><td>end of the stretcher between the table legs</td><td>-</td></tr><tr><td>10</td><td>end of the stretcher between the table legs</td><td>-</td></tr><tr><td>11</td><td>end of the stretcher between the table legs</td><td>-</td></tr><tr><td rowspan="10">Mug</td><td>0</td><td>mug rim</td><td>-</td></tr><tr><td>1</td><td>mug rim</td><td>-</td></tr><tr><td>2</td><td>mug rim</td><td>-</td></tr><tr><td>3</td><td>mug rim</td><td>-</td></tr><tr><td>4</td><td>mug base rim</td><td>-</td></tr><tr><td>5</td><td>mug base rim</td><td>-</td></tr><tr><td>6</td><td>mug base rim</td><td>-</td></tr><tr><td>7</td><td>mug base rim</td><td>-</td></tr><tr><td>8</td><td>upper attachment point for the mug handle</td><td>top of the handle</td></tr><tr><td>9</td><td>center of the mug handle</td><td>handle</td></tr><tr><td>10</td><td>lower attachment point for the mug handle</td><td>bottom of the handle</td></tr><tr><td rowspan="17">Bottle</td><td>0</td><td>lower rim of the bottle</td><td>-</td></tr><tr><td>1</td><td>lower rim of the bottle</td><td>-</td></tr><tr><td>2</td><td>lower rim of the bottle</td><td>-</td></tr><tr><td>3</td><td>lower rim of the bottle</td><td>-</td></tr><tr><td>4</td><td>edge between the bottle body and the bottle neck</td><td>-</td></tr><tr><td>5</td><td>edge between the bottle body and the bottle neck</td><td>-</td></tr><tr><td>6</td><td>edge between the bottle body and the bottle neck</td><td>-</td></tr><tr><td>7</td><td>edge between the bottle body and the bottle neck</td><td>-</td></tr><tr><td>8</td><td>bottle cap center</td><td>-</td></tr><tr><td>9</td><td>edge of the bottle cap</td><td>-</td></tr><tr><td>10</td><td>edge of the bottle cap</td><td>-</td></tr><tr><td>11</td><td>edge of the bottle cap</td><td>-</td></tr><tr><td>12</td><td>edge of the bottle cap</td><td>-</td></tr><tr><td>13</td><td>lower edge of the bottle neck</td><td>-</td></tr><tr><td>14</td><td>lower edge of the bottle neck</td><td>-</td></tr><tr><td>15</td><td>lower edge of the bottle neck</td><td>-</td></tr><tr><td>16</td><td>lower edge of the bottle neck</td><td>-</td></tr><tr><td>17</td><td>lower edge of the bottle neck</td><td>-</td></tr></tbody></table>
<table><thead><tr><th>Object</th><th>Index</th><th>Description</th><th>Short Description</th></tr></thead><tbody><tr><td></td><td>0</td><td>bow of a boat</td><td>front</td></tr><tr><td></td><td>1</td><td>lower part of the end of the bow of a boat (starboard)</td><td>bottom front</td></tr><tr><td></td><td>2</td><td>upper part of the end of the bow of a boat (starboard)</td><td>-</td></tr><tr><td></td><td>3</td><td>upper part of the end of the bow of a boat (port)</td><td>-</td></tr><tr><td></td><td>4</td><td>lower part of the end of the bow of a boat (port)</td><td>bottom front</td></tr><tr><td></td><td>5</td><td>lower corner of the boat stern (starboard)</td><td>back</td></tr><tr><td></td><td>6</td><td>upper corner of the boat stern (starboard)</td><td>back left corner</td></tr><tr><td></td><td>7</td><td>upper corner of the boat stern (port)</td><td>back left corner</td></tr><tr><td></td><td>8</td><td>lower corner of the boat stern (port)</td><td>back</td></tr><tr><td></td><td>9</td><td>mast step of a sailboat</td><td>-</td></tr><tr><td></td><td>10</td><td>mast top of a sailboat</td><td>-</td></tr><tr><td rowspan="12">Vessel</td><td>11</td><td>lower front corner of the deckhouse (starboard)</td><td>right front of the cabin</td></tr><tr><td>12</td><td>lower front corner of deckhouse (port)</td><td>left front of the cabin</td></tr><tr><td>13</td><td>lower aft corner of the deckhouse (starboard)</td><td>-</td></tr><tr><td>14</td><td>lower aft corner of the deckhouse (port)</td><td>-</td></tr><tr><td>15</td><td>upper front corner of the deckhouse (starboard)</td><td>-</td></tr><tr><td>16</td><td>upper front corner of the deckhouse (port)</td><td>-</td></tr><tr><td>17</td><td>upper aft corner of the deckhouse (starboard)</td><td>-</td></tr><tr><td>18</td><td>upper aft corner of the deckhouse (port)</td><td>-</td></tr><tr><td>19</td><td>foremost elevation point on the deck of the ship</td><td>-</td></tr><tr><td>20</td><td>lower corner of the windshield of the ship (starboard)</td><td>-</td></tr><tr><td>21</td><td>lower corner of the windshield of the ship (port)</td><td>-</td></tr><tr><td>22</td><td>upper corner of the windshield of the ship (starboard)</td><td>-</td></tr><tr><td>23</td><td>upper corner of the windshield of the ship (port)</td><td>-</td></tr><tr><td rowspan="20">Chair</td><td>0</td><td>upper corner of the back of the chair</td><td>top corners of the back rest</td></tr><tr><td>1</td><td>upper corner of the back of the chair</td><td>top corners of the back rest</td></tr><tr><td>2</td><td>lower corner of the back of the chair</td><td>different corners of the seat</td></tr><tr><td>3</td><td>lower corner of the back of the chair</td><td>different corners of the seat</td></tr><tr><td>4</td><td>front corner of the seat of the chair</td><td>different corners of the seat</td></tr><tr><td>5</td><td>front corner of the seat of the chair</td><td>different corners of the seat</td></tr><tr><td>6</td><td>front corner of the armrest of the chair</td><td>end of both armrests</td></tr><tr><td>7</td><td>back corner of the armrest of the chair</td><td>end of both armrests</td></tr><tr><td>8</td><td>front corner of the armrest of the chair</td><td>end of both armrests</td></tr><tr><td>9</td><td>back corner of the armrest of the chair</td><td>end of both armrests</td></tr><tr><td>10</td><td>lower end of the front leg of the chair</td><td>tip of all legs</td></tr><tr><td>11</td><td>lower end of the back leg of the chair</td><td>tip of all legs</td></tr><tr><td>12</td><td>lower end of the front leg of the chair</td><td>tip of all legs</td></tr><tr><td>13</td><td>lower end of the back leg of the chair</td><td>tip of all legs</td></tr><tr><td>14</td><td>lower end of the leg of the chair</td><td>tip of all legs</td></tr><tr><td>15</td><td>top of the chair cylinder</td><td>cylinder</td></tr><tr><td>16</td><td>bottom of the chair cylinder</td><td>cylinder</td></tr><tr><td>17</td><td>lower end of the front leg of the chair</td><td>-</td></tr><tr><td>18</td><td>lower end of the back leg of the chair</td><td>-</td></tr><tr><td>19</td><td>lower end of the back leg of the chair</td><td>-</td></tr><tr><td>20</td><td>lower end of the front leg of the chair</td><td>-</td></tr></tbody></table>
<table><thead><tr><th>Object</th><th>Index</th><th>Description</th><th>Short Description</th></tr></thead><tbody><tr><td></td><td>0</td><td>lower end of the front leg of the bed</td><td>-</td></tr><tr><td></td><td>1</td><td>lower end of the headboard leg of the bed</td><td>-</td></tr><tr><td></td><td>2</td><td>lower end of the headboard leg of the bed</td><td>-</td></tr><tr><td></td><td>3</td><td>lower end of the front leg of the bed</td><td>-</td></tr><tr><td></td><td>4</td><td>foot corner of the mattress</td><td>-</td></tr><tr><td></td><td>5</td><td>head corner of the mattress</td><td>-</td></tr><tr><td></td><td>6</td><td>head corner of the mattress</td><td>-</td></tr><tr><td></td><td>7</td><td>foot corner of the mattress</td><td>-</td></tr><tr><td></td><td>8</td><td>foot corner of the upper mattress of the bunk bed</td><td>-</td></tr><tr><td rowspan="10">Bed</td><td>9</td><td>head corner of the upper mattress of the bunk bed</td><td>-</td></tr><tr><td>10</td><td>head corner of the upper mattress of the bunk bed</td><td>-</td></tr><tr><td>11</td><td>foot corner of the upper mattress of the bunk bed</td><td>-</td></tr><tr><td>12</td><td>upper corner of the bed frame of the bunk bed</td><td>-</td></tr><tr><td>13</td><td>upper corner of the bed frame of the bunk bed</td><td>-</td></tr><tr><td>14</td><td>upper corner of the bed frame of the bunk bed</td><td>-</td></tr><tr><td>15</td><td>upper corner of the bed frame of the bunk bed</td><td>-</td></tr><tr><td>16</td><td>upper corner of the footboard of the bed</td><td>-</td></tr><tr><td>17</td><td>upper corner of the headboard of the bed</td><td>-</td></tr><tr><td>18</td><td>upper corner of the headboard of the bed</td><td>-</td></tr><tr><td>19</td><td>upper corner of the footboard of the bed</td><td>-</td></tr><tr><td></td><td>0</td><td>corner of the tail of the skateboard</td><td>-</td></tr><tr><td></td><td>1</td><td>corner of the tail of the skateboard</td><td>-</td></tr><tr><td></td><td>2</td><td>corner of the nose of the skateboard</td><td>-</td></tr><tr><td></td><td>3</td><td>corner of the nose of the skateboard</td><td>-</td></tr><tr><td rowspan="6">Skateboard</td><td>4</td><td>wheel of the skateboard</td><td>wheel</td></tr><tr><td>5</td><td>wheel of the skateboard</td><td>wheel</td></tr><tr><td>6</td><td>wheel of the skateboard</td><td>wheel</td></tr><tr><td>7</td><td>wheel of the skateboard</td><td>wheel</td></tr><tr><td>8</td><td>nose of the skateboard</td><td>end</td></tr><tr><td>9</td><td>tail of the skateboard</td><td>end</td></tr><tr><td></td><td>0</td><td>tip of the knife</td><td>tip of the blade</td></tr><tr><td></td><td>1</td><td>butt of the knife</td><td>bottom</td></tr><tr><td rowspan="5">Knife</td><td>2</td><td>shoulder of the knife</td><td>-</td></tr><tr><td>3</td><td>heel of the knife</td><td>-</td></tr><tr><td>4</td><td>end of the crossguard of the knife</td><td>-</td></tr><tr><td>5</td><td>end of the crossguard of the knife</td><td>-</td></tr><tr><td>0</td><td>top button of the cap</td><td>top</td></tr><tr><td rowspan="5">Cap</td><td>1</td><td>front of the visor of the cap</td><td>front of the visor</td></tr><tr><td>2</td><td>corner of the visor of the cap</td><td>-</td></tr><tr><td>3</td><td>corner of the visor of the cap</td><td>-</td></tr><tr><td>4</td><td>center of the back of the cap</td><td>middle of the strap</td></tr><tr><td>5</td><td>center of the visor attachment of the cap</td><td>-</td></tr><tr><td rowspan="6">Laptop</td><td>0</td><td>corner of the laptop body</td><td>bottom left corner</td></tr><tr><td>1</td><td>hinge of the laptop</td><td>bottom left corner of the screen</td></tr><tr><td>2</td><td>hint of the laptop</td><td>bottom right corner of the screen</td></tr><tr><td>3</td><td>corner of the laptop body</td><td>bottom right corner</td></tr><tr><td>4</td><td>corner of the laptop screen</td><td>top left corner of the screen</td></tr><tr><td>5</td><td>corner of the laptop screen</td><td>top right corner of the screen</td></tr></tbody></table>
<table><thead><tr><th>Object</th><th>Index</th><th>Description</th><th>Short Description</th></tr></thead><tbody><tr><td rowspan="20">Motorcycle</td><td>0</td><td>headlight of the motorcycle</td><td>headlight</td></tr><tr><td>1</td><td>end of the handlebar of the motorcycle</td><td>right part of the steering wheel</td></tr><tr><td>2</td><td>end of the handlebar of the motorcycle</td><td>left part of the steering wheel</td></tr><tr><td>3</td><td>seat of the motorcycle</td><td>seat</td></tr><tr><td>4</td><td>axle of the front wheel of the motorcycle</td><td>front wheel</td></tr><tr><td>5</td><td>axle of the back wheel of the motorcycle</td><td>back wheel</td></tr><tr><td>6</td><td>bottom part of the frame of the motorcycle</td><td>bottom of the engine</td></tr><tr><td>7</td><td>axle of the front wheel of the motorcycle</td><td>front wheel</td></tr><tr><td>8</td><td>axle of the back wheel of the motorcycle</td><td>back wheel</td></tr><tr><td>9</td><td>front part of the front wheel of the motorcycle</td><td>front of the front wheel</td></tr><tr><td>10</td><td>back part of the back wheel of the motorcycle</td><td>back of the back wheel</td></tr><tr><td>11</td><td>top of the fuel tank of the motorcycle</td><td>top of the tank</td></tr><tr><td>12</td><td>front end of the front fender of the motorcycle</td><td>top of the front wheel</td></tr><tr><td>13</td><td>back end of the rear fender of the motorcycle</td><td>top of the back wheel</td></tr><tr><td rowspan="20">Airplane</td><td>0</td><td>nose of the airplane</td><td>nose of the airplane</td></tr><tr><td>1</td><td>cockpit of the airplane</td><td>top of the cockpit</td></tr><tr><td>2</td><td>tail of the airplane</td><td>back end</td></tr><tr><td>3</td><td>the end of the left wing of the airplane</td><td>tip of all the wings</td></tr><tr><td>4</td><td>wing root of the left wing of the airplane</td><td>different corners of the intersection between the body and the wings</td></tr><tr><td>5</td><td>the end of the right wing of the airplane</td><td>tip of all the wings</td></tr><tr><td>6</td><td>wing root of the right wing of the airplane</td><td>different corners of the intersection between the body and the wings</td></tr><tr><td>7</td><td>wingtip of the left horizontal stabilizer of the airplane</td><td>tip of all the wings</td></tr><tr><td>8</td><td>wingtip of the right horizontal stabilizer of the airplane</td><td>tip of all the wings</td></tr><tr><td>9</td><td>top of vertical stabilizer of the airplane</td><td>-</td></tr><tr><td>10</td><td>wingtip of the left diagonal stabilizer of the airplane</td><td>tip of all the wings</td></tr><tr><td>11</td><td>wingtip of the right diagonal stabilizer of the airplane</td><td>tip of all the wings</td></tr><tr><td>12</td><td>wing root of the left wing of the airplane</td><td>different corners of the intersection between the body and the wings</td></tr><tr><td>13</td><td>wing root of the right wing of the airplane</td><td>different corners of the intersection between the body and the wings</td></tr><tr><td>14</td><td>inner turbine of the left wing of the airplane</td><td>center of all turbines</td></tr><tr><td>15</td><td>inner turbine of the right wing of the airplane</td><td>center of all turbines</td></tr><tr><td>16</td><td>outer turbine of the left wing of the airplane</td><td>center of all turbines</td></tr><tr><td>17</td><td>outer turbine of the right wing of the airplane</td><td>center of all turbines</td></tr><tr><td>18</td><td>outer turbine of the right wing of the airplane</td><td>-</td></tr></tbody></table>
<table><thead><tr><th>Object</th><th>Index</th><th>Description</th><th>Short Description</th></tr></thead><tbody><tr><td></td><td>0</td><td>front right wheel of the car</td><td>front right wheel</td></tr><tr><td></td><td>1</td><td>rear right wheel of the car</td><td>back right wheel</td></tr><tr><td></td><td>2</td><td>rear left wheel of the car</td><td>back left wheel</td></tr><tr><td></td><td>3</td><td>front left wheel of the car</td><td>front left wheel</td></tr><tr><td></td><td>4</td><td>corner of the front bumper of the car</td><td>right part of the front bumper</td></tr><tr><td></td><td>5</td><td>corner of the rear bumper of the car</td><td>left part of the back bumper</td></tr><tr><td></td><td>6</td><td>corner of the rear bumper of the car</td><td>left part of the back bumper</td></tr><tr><td></td><td>7</td><td>corner of the front bumper of the car</td><td>right part of the front bumper</td></tr><tr><td></td><td>8</td><td>top corner of the windshield of the car</td><td>top of right window</td></tr><tr><td></td><td>9</td><td>top corner of the windshield of the car</td><td>top right corner of back windshield</td></tr><tr><td rowspan="10">Car</td><td>10</td><td>top corner of the rear window of the car</td><td>left corner of back windshield</td></tr><tr><td>11</td><td>top corner of the rear window of the car</td><td>top left corner of front windshield</td></tr><tr><td>12</td><td>bottom corner of the windshield of the car</td><td>bottom left corner of front windshield</td></tr><tr><td>13</td><td>bottom corner of the windshield of the car</td><td>bottom right corner of front windshield</td></tr><tr><td>14</td><td>bottom corner of the rear window of the car</td><td>bottom right corner of back windshield</td></tr><tr><td>15</td><td>bottom corner of the rear window of the car</td><td>bottom left corner of back windshield</td></tr><tr><td>16</td><td>right headlight of the car</td><td>front right headlight</td></tr><tr><td>17</td><td>right taillight of the car</td><td>right back headlight</td></tr><tr><td>18</td><td>left taillight of the car</td><td>left back headlight</td></tr><tr><td>19</td><td>left headlight of the car</td><td>front left headlight</td></tr><tr><td>20</td><td>right side mirror of the car</td><td>right rear view</td></tr><tr><td>21</td><td>left side mirror of the car</td><td>left rear view</td></tr><tr><td rowspan="8">Guitar</td><td>0</td><td>head of the guitar</td><td>top of the headstock</td></tr><tr><td>1</td><td>connection of the body and neck of the guitar</td><td>middle of the fretboard</td></tr><tr><td>2</td><td>corner of the body of the guitar</td><td>top left corner of the body</td></tr><tr><td>3</td><td>corner of the body of the guitar</td><td>top right corner of the body</td></tr><tr><td>4</td><td>corner of the body of the guitar</td><td>bottom left corner of the body</td></tr><tr><td>5</td><td>corner of the body of the guitar</td><td>bottom right corner of the body</td></tr><tr><td>6</td><td>strap pin on the bottom of the body of the guitar</td><td>bottom of the body</td></tr><tr><td>7</td><td>waist of the body of the guitar</td><td>middle left of the body</td></tr><tr><td>8</td><td>waist of the body of the guitar</td><td>middle right of the body</td></tr></tbody></table>
<table><thead><tr><th>Object</th><th>Index</th><th>Description</th><th>Short Description</th></tr></thead><tbody><tr><td rowspan="23">Bathtub</td><td>0</td><td>top outer corner of the bathtub</td><td>-</td></tr><tr><td>1</td><td>top outer corner of the bathtub</td><td>-</td></tr><tr><td>2</td><td>top outer corner of the bathtub</td><td>-</td></tr><tr><td>3</td><td>top outer corner of the bathtub</td><td>-</td></tr><tr><td>4</td><td>bottom outer corner of the bathtub</td><td>-</td></tr><tr><td>5</td><td>bottom outer corner of the bathtub</td><td>-</td></tr><tr><td>6</td><td>bottom outer corner of the bathtub</td><td>-</td></tr><tr><td>7</td><td>bottom outer corner of the bathtub</td><td>-</td></tr><tr><td>8</td><td>top inner corner of the bathtub</td><td>-</td></tr><tr><td>9</td><td>top inner corner of the bathtub</td><td>-</td></tr><tr><td>10</td><td>top inner corner of the bathtub</td><td>-</td></tr><tr><td>11</td><td>top inner corner of the bathtub</td><td>-</td></tr><tr><td>12</td><td>bottom inner corner of the bathtub</td><td>-</td></tr><tr><td>13</td><td>bottom inner corner of the bathtub</td><td>-</td></tr><tr><td>14</td><td>bottom inner corner of the bathtub</td><td>-</td></tr><tr><td>15</td><td>bottom inner corner of the bathtub</td><td>-</td></tr><tr><td>16</td><td>center of the short side upper edge of the bathtub</td><td>-</td></tr><tr><td>17</td><td>center of the short side upper edge of the bathtub</td><td>-</td></tr><tr><td>18</td><td>center of the short side lower edge of the bathtub</td><td>-</td></tr><tr><td>19</td><td>center of the short side lower edge of the bathtub</td><td>-</td></tr><tr><td>20</td><td>center of the short side upper edge of the bathtub</td><td>-</td></tr><tr><td>21</td><td>center of the short side upper edge of the bathtub</td><td>-</td></tr><tr><td>22</td><td>center of the short side lower edge of the bathtub</td><td>-</td></tr><tr><td>23</td><td>center of the short side lower edge of the bathtub</td><td>-</td></tr><tr><td rowspan="8">Helmet</td><td>0</td><td>top of the helmet</td><td>top</td></tr><tr><td>1</td><td>front of the helmet peak</td><td>front</td></tr><tr><td>2</td><td>back of the helmet peak</td><td>back</td></tr><tr><td>3</td><td>left side of the helmet peak</td><td>left</td></tr><tr><td>4</td><td>right side of the helmet peak</td><td>right</td></tr><tr><td>5</td><td>right bottom of the helmet</td><td>bottom right</td></tr><tr><td>6</td><td>left bottom of the helmet</td><td>bottom left</td></tr><tr><td>7</td><td>bottom of the visor of the helmet</td><td>-</td></tr><tr><td>8</td><td>bottom of the back of the helmet</td><td>bottom back</td></tr></tbody></table>