Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features
Authors: Chengkai Hou
Zhengrong Xue
Bingyang Zhou
Jinghan Ke
Lin Shao
Huazhe Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Keypoint Detection  Deformable Object  3D Point Cloud
Primary area: machine_vision

Abstract: Detecting 3D keypoints with semantic consistency is widely used in many scenarios such as pose estimation, shape registration and robotics. Currently, most unsupervised 3D keypoint detection methods focus on the rigid-body objects. However, when faced with deformable objects, the keypoints they identify do not preserve semantic consistency well. In this paper, we introduce an innovative unsupervised keypoint detector Key-Grid for both the rigid-body and deformable objects, which is an autoencoder framework. The encoder predicts keypoints and the decoder utilizes the generated keypoints to reconstruct the objects. Unlike previous work, we leverage the identified keypoint in formation to form a 3D grid feature heatmap called grid heatmap, which is used in the decoder section. Grid heatmap is a novel concept that represents the latent variables for grid points sampled uniformly in the 3D cubic space, where these variables are the shortest distance between the grid points and the “skeleton” connected by keypoint pairs. Meanwhile, we incorporate the information from each layer of the encoder into the decoder section. We conduct an extensive evaluation of Key-Grid on a list of benchmark datasets. Key-Grid achieves the state-of-the-art performance on the semantic consistency and position accuracy of keypoints. Moreover, we demonstrate the robustness of Key-Grid to noise and downsampling. In addition, we achieve SE-(3) invariance of keypoints though generalizing Key-Grid to a SE(3)-invariant backbone.

Representing objects through a set 3D keypoints is one of the most popular and intuitive approaches to compress and comprehend 3D objects [3]]66] Effectively exhibiting their utility, 3D keypoints have contributed to the success of a number of downstream tasks, including pose estimation, shape as well various kinds of Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features
Chengkai 1.3 Zhengrong Xue!.2, 12, Bingyang Zhou', Jinghan Ke¹,
Lin Shao", Huazhe Xu2.2.7 Shhanghai Qizhi Institute
TTingguu University Speking University
The University of Hong Kong University of Science and Technology of China National Unversity of Singapore
"Shnnhaii Al Lab
Abstract
Detecting 3D keypoints with semantic consistency is widely used in many see- narios such as pose estimation, shape registration and robotics. Currently, most unsupervised 3D keypoint detection methods focus on rigid-body objects. However, when faced with deformable objects, the keypoints they identify do not preserve semantic consistency well. In this paper, we introduce keypoint detector Key-Grid for both rigid-body and deformable objects, which is autoencoder framework. The encoder predicts keypoints and the decoder utilizes the generated keypoints to reconstruct the objects. Unlike previous work, leverage the identified keypoint information form 3D grid feature heatmap called grid heatmap, which used in the decoder section. Grid heatmap is novel concept that represents the latent variables for grid points sampled uniformly in the 3D cubic space, where these variables are the shortest distance between the grid points and the "skeleton" connected by keypoint pairs. Meanwhile, incorporate the information from each layer of the encoder into the decoder model. We conduct extensive evaluation of Key-Grid achieves the state-of-the-an performance on the semantic consistency and position accuracy of keypoints. Moreover, we demonstrate the robustness of Key-Grid noise and downsampling. In addition, we achieve SE-(3) invariance of keypoints though generalizing Key-Grid SE(3)-invariant backbone.
innovative unsupervised
list of benchmark datasets. Key-Grid
Introduction
registration, object tracking in computer vision 15 27
robotic manipulation tasks 40 16..
To make the detected keypoints as capable and accessible as possible, the research community is now concentrating on the unsupervised learning of semantically consistent keypoints on 3D point clouds. The implication of semantic consistency is typically twofold: the keypoints should be located the semantically salient parts of the objects; they are also desired to be aligned within the same category under large shape variations among diverse 3D object instances. To achieve these objectives, previous works (26) [1 46) often adopt autoencoder frameworks to facilitate self-supervised training, where the encoders serving the keypoint predictor backbone networks [2] 22 generalizable to the shape variation, and the decoders reconstruct the input shape conditioned on the predicted keypoints. Since neural networks are in general better compression than generation 20
"Corresponding to:
huazhe xu@mail tsinghua. edu.cn
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Figure 1: Examples of the keypoints detected by Key-Grid. The detected keypoints preserve semantic consistency under: (Top) large intra-category shape variations of higid-body objects from the ShapeNetCoreV2 V2 [ dataset; (Bottom) dramatic deformations of soft-body objects from the ClothesNet [49] dataset.
the primary challenge of this pipeline lies in reconstructing the entire point cloud from few estimated
keypoints. Thus, the state-of-the-ar (SOTA) detectors (26 9 lay emphasis on leveraging different priors on 3D structures (e.g., "skeleton" in Skeleton Merger [26]. and "cage" KeypointDeformer (9) so that the 3D object shapes can be more reasonably approximated by the information from the detected keypoints alone.
While maintaining semantic consistency is already demanding under the shape variations of rigid- body objects (e.g., those in the ShapeNetCoreV2 [2] dataset), would be even more challenging deformable objects are also taken into consideration. For instance, when detecting keypoints on the trousers, if one detected keypoint is located at one of the trouser legs, then the keypoint is desired to follow the motion of the trouser leg in the process of the trousers being folded (shown in Figure []] Note that the shape variation caused by deformation so dramatic that even the outline of the object shape has been completely altered, indicating shift in the spatial and geometric structure of the keypoints. Despite the difficulty, with a growing interest deformable object manipulation 24 25 in robotics as well as the emergence of large-scale datasets for deformable objects [49] 6 in computer vision, it of increasing significance to develop a keypoint detector that is equally effectively when faced with deformable objects.
In this work, we present Key-Grid, an unsupervised keypoint detector on 3D point clouds aiming for semantic consistency under the shape variations of both rigid-body and deformable objects. In accordance with the prevailing practice, Key-Grid uses an autoencoder framework. In response to the potentially shifted geometric structure of the keypoints brought by deformations, we propose to embed the information of the predicted keypoints into a dense 3D feature heatmap. To be more specific, we first uniformly sample large number of grid points in the shape of 3D array. Then, we assign a feature to each grid by calculating the shortest distance from the grid point to the connected lines of all the keypoint pairs (i.e., the "skeleton" [26] of the keypoints) and multiplying it by the weight of the connected lines. Finally, when the decoder attempts to reconstruct the point cloud, coarse-to-fine features are extracted from the dense grid feature heatmap, where the extracted point coordinates are line with the hierarchical point sets in the PointNet++ [22] modules the encoder. Intuitively, the grid heatmap can be viewed dense extension of the "skeleton" where its undefined blank spaces are smoothly extrapolated. Functionally, the grid heatmap constitutes a continuous feature landscape across the entire 3D space, providing richer and more stable geometric descriptions of the object. This could be vitally beneficial when the object undergoes intense shape variations, such cloth deformations.
Empirically, the experimental results show that Key-Grid not only achieves SOTA performance for rigid-body objects in the popular ShapeNetCoreV2 [2) dataset but also surpasses the previous SOTA 99 by 8.0% and 9.1% for objects with dropping and dragging deformation respectively the recently proposed ClothesNet [49] dataset. Meanwhile, Key-Grid is found to be robust to noise or downsampling operations. Additionally, we also show that Key-Grid can easily extended to an SE(3))-eqivariam version when it is integrated with the USEEK 40 framework. We are committed to releasing the code.
Related works
Deformable object datasets. While there an increasing number of large-scale 3D dataset repositories such as Shap 21 PARTNET [18), SAPIEN (37), and ThingilOK [50), only

---

a few datasets contain deformations from the same model. Among them, the deformations from the same piece of clothing are diverse and have practical research significance. For instance, Deep Fashion3D [6] contains around 2,000 3D models reconstructed from 563 real garment instances in different poses. subset of ClothesNet [49] contains around 3,000 3D garment meshes which can be directly loaded into differentiable simulations such as DiffclothAl [45] to generate various deformations after operating like dropping, folding, or dragging.
Unsupervised keypoint detection. There have been various approaches proposed for supervised estimation of 3D keypoints using manually annotated keypoints [35 14 13 42). In contrast to supervised methods, our approach is unsupervised, meaning it does not rely on manually annotated keypoints. Thus, we review methods that adopt unsupervised approach for estimating 3D keypoints. USIP 12 is the first detector that identifies 3D keypoints in an unsupervised manner by minimizing the chamfer distance between detected keypoints in randomly transformed object pairs. Canonical Capsules
(29 is a similar approach that feeds pairs of randomly translated copies of the same object into a network to detect keypoints. Following USIP, SC3K [51] also uses a random rotation to create two transformed visions of objects and generate the corresponding keypoints by mapping the keypoints of each version back to the original object. Another way to estimate 3D keypoints is proposed by Chen et al. (3]. They encode the point cloud as a set of local feature and input into a novel structure model generate the possibility of keypoints. Recently, Fernandez et al. W proposed a novel method that distinguishes between Node branches and Pose and coefficient branches to find the optimal keypoints of an object.
However, these methods do not consider the geometric structure information that keypoints can represent. When they encounter irregular shape of objects, such as airplanes and cars, the keypoints they identify will lose the crucial information about the object. Skeleton Merger [26] generates the skeletons of an object connected by keypoints and uses the Composite Chamfer Distance (CCD) to make these skeletons close to the original point cloud. This makes the detected keypoints to represent the important structural information of the object. Yuan et al. [46) propose similar way that generates keypoints by utilizing skeletons from two objects within the same category to reconstruct mutually. USEEK 44) utilizes teacher-student network, where the teacher module is based on Skeleton Merger and the student module employs a SE(3)-invarian) backbone network, SPRIN 43.. LAKe- Net [32) uses detected keypoints to achieve the shape completion by locating them to generate a surface skeleton and refining the shape of the object. KeypointNet [30] learns category-specifte 3D keypoints using depth and 2D position information from pair of 2D images. For keypoint detection on deformable objects, KeypointDeformer [9] aligns the shape of the source object to the target object by utilizing the difference in keypoints positions between them and propose a novel loss function that encourages keypoints to distribute well and keep semantic consistency.
3 Method
In this section, we propose Key-Grid, an unsupervised keypoint detector on 3D point clouds based on the autoencoder architecture. Figure 2 shows the overview of Key-Grid. In the following section, we provide detailed explanation of the key ingredients in the Key-Grid: keypoint locations in an input point cloud; a grid heatmap is a 3D feature map used to capture the geometric structure of deformable objects by computing the shortest distance from points uniformly sampled in 3D cubic space to the "skeleton" generated by keypoint pairs; a decoder that leverages the grid heatmap and the information in each layer of the encoder to reconstruct point clouds.
encoder that predicts
3.1 Encoder: Keypoint Detection
In Key-Grid, each keypoint is regarded as the weighted sum of all the points in the point cloud. Given an input point cloud X E IR Nx3 the goal of the encoder is to produce a weight matrix W E RKN such that the matrix multiplication of W and X directly gives the predicted K keypoints E Rx,,
(1) To be more specific, the encoder consists of L PointNet++ (22 layers. The i-th PointNet++ layer in
K= W. X
the encoder generates a hierarchically down-sampled point cloud (i) E Nx3 and its corresponding feature F(5) E Nx where i E {1, L). The last-layer feature passing through a Softmax activation function gives the weight matrix:
W Softmax
(2)
3

---

Point Cloud
Encoder
Keypoint
PointNete+
[1024 96)
(64 512)
[16 1024)
Point Score Rhxd
Matrix Muitiply
Farthest point keypoint loss
Similarity loss
Decoder
MLP
MLP
MLP
Multiplication
Reconstruction
Combination
Points Heatmap
Points Heatmap
Points Heatmap
Copy
Figure 2: Pipeline of Key-Grid. In the encoder section, given a point cloud, we detect the keypoints by utilizing the PointNet++. Then, we utilize the detected keypoints to form the grid heatmap. In the decoder section, we use each layer the PointNet++ and the grid heatmap to reconstruct the input point cloud. "MLP" stands for multi-layer perceptron, which contains Batch-norm and ReI.U.
Additionally, following the practice in Skeleton Merger [26]. the encoder outputs an additional head to predict the weights of C K K(K 1)/2 skeleton segments, i.e., the edges between each pair of keypoints. Formally, (k,, k,) denotes the skeleton segment connecting the keypoints k; E IR3 and kj R and 8ij denotes the weight of the skeleton segment (k,, kj).
3.2 Grid Heatmap: Dense 3D Feature Map
The proposed grid heatmap is a dense 3D feature map designed to densely represent the 3D shape only through the information from the predicted keypoints. Ideally, for describing the object in an implicit manner similar to the Occupancy Networks [17]. the feature on each grid desired to reflect the distance from the grid point the 3D object shape. Since the ground-truth object shape is the reconstruction target in our problem setting, we need to find ways to approximate the object shape using the predicted keypoints. In Key-Grid, the object shape is represented by the weighted connected lines of all the keypoint pairs. For each individual grid point, we calculate the distances from this point all the skeleton segments, and take the maximum of these distances as the feature of this grid. Intuitively, this gives a dense feature field whose value the smallest at the geometric center of the object, and gradually increases along with the grid point coordinate moving outside the outline of the object shape. In the following, we illustrate the detailed procedures for establishing such grid heatmap.
adopt the skeleton (26) approximation, where
To begin with, we uniformly sample a 3D array of grid points P E MxMxXM x3 in the normalized cubic 3D space, where M denotes the number of points we sample each side of the cube. In Key-Grid, we set M = 16, giving 4096 gird points. Next, we calculate the distance between the grid points and all the line segments in the skeleton. When the projection of grid point onto the skeleton line falls in the range of the skeleton segment, the distance is defined as the distance between the grid point and the projection point. Otherwise, the distance is directly defined as the distance between the grid point and the nearest endpoint of the skeleton segment. Formally, the distance de (p) between grid point E 3 and a skeleton segment (k,, kj) connecting the keypoints k; and k, is defined as:
pp k,
if 5 0 if0 < t<t
(p)(p)
(11
tjk: + tk,)
(3)
pp k,1
ift
where
(p k,) (k, k,)
(4)
ki k; 11
Then, the feature of each grid point D(p) is the maximum of the weighted distances from this point to all the skeleton segments:
D(p) max (si exp exp (d. (p)/o?))
(5)

---

where siy refers to the learnable weight of the skeleton segment (k, ky) produced by the encoder, and o is a fixed hyperparameter. And the grid heatmap H is the 3D array consisting of the features of all the grid points:
H (D((ay)))))) 1... E R MXXXXXXX
(6)
where P Pzyz E I denotes the extracted grid point coordinate from P indexed by (z, y, 2). Concep- tually, the grid heatmap H contains the complete geometric information of the keypoints K. As dense and continuous feature field rather than a set of sparse points, the grid heatmap is expected to force the encoder to predict precise keypoints and bring benefits to the reconstruction process.
In the grid heatmap, we measure the value of (p)) using the maximum point-to- skeletons distance instead of minimum distance because the maximum distance can better distinguish the spatial locations, especially what is inside or outside an object. For instance, Figure B shows that if we choose the minimum distance between the sampled grid points and the keypoints, the grid point p1 inside the pants will have the same value dman as the grid point po outside the pants. However, if we take the maximum value dmazz the grid point p1 inside the pants will have a smaller value than the grid point Po outside the pants.
Po
3.3 Decoder: Point Cloud Reconstruction
As an inverse process of the encoder, the decoder tries to reconstruct the entire point cloud by gradually augmenting finer geometric details in hierarchical manner. Unlike previous methods [26 9 where the keypoint-related 3D structures are used only once as the input of the encoder, we propose to integrate increasingly finer grid heatmap features with the layers of the decoder. In addition, the features in the encoder are directly copied and fused into the corresponding layer the decoder in U-Net [ fashion to further assist the reconstruction process.
Figure
Example of distance defi- nition on the grid heatmap.
Formally, the feature of the (L i + 1)-th layer in the decoder is composed by the following three components:
(f.-+11) = H(X dec
(4) mmn) + F(t) @ PFoj (1--1)
(0-1)
dec
where H(X (i)
N denotes the extracted grid features indexed by X RN3 F R XF denotes the corresponding feature of the same number of points copied from the encoder, Proj(F(L--)
x
(i-1))
x(me) denotes the feature projected from the former layer of the decoder, and enc
dec
@ denotes element-wise concatenation. Note that H(X())) and F(o are already aligned on the first dimension and thus ready for concatenation, the remaining challenge how to design a feature
projection mechanism F Frarg Proj(Fori. oris Xori, Xtare) so that the former-layer features can be aligned with the current-layer features.
To solve this problem, we propose to represent the target feature Flarg as the weighted sum of the spatially neighboring features in the original feature Fori- More specifically, for every point coordinate x E R-3 in the target point cloud Xtarg we find its Nneig neighbors in the original point cloud X
Neei (x (xn) = { (x x
on ocis (NN)
(8)
The target feature of the coordinate is defined as:
F Ftarg(x) Exex (xxx) w(x, x (x) Ex (xIx) w(x, x)
(9)
where wxx,x) denotes the inverse of the squared distance between x and x:
1
w(xxx)= = ||x x 112
(10)
3.4 Training Objectives
Our approach adopts end-to-end method called Key-Grid to identify the keypoints, by leveraging the grid heatmap to reduce the discrepancy between the reconstructed point cloud and the input point cloud. In this section, we will introduce two types of loss function to optimize Key-Grid.
5

---

Similarity loss. The common way to evaluate the similarity between a reconstructed point cloud X, and target point cloud X is by employing the Chamfer distance metric ( 48 By minimizing the Chamfer distance between them, we can optimize the reconstruction process to closely match the target point cloud, thereby achieving a higher level similarity between the two point clouds. Thus, we utilize the Chamfer distance between the reconstructed point cloud X, and the target point cloud X to approximate this similarity loss Lsim-
Farthest point keypoint loss. To ensure that the aligned keypoints distribute well on the surface of objects and represent the geometric structure of objects, we use the farthest point keypoint loss 9 far to allocate the keypoints. Firstly, we randomly select an initial point from the point cloud X. Then, each iteration, it selects the point that farthest from all the previously selected points. This process continues until J points have been selected. Finally, we can obtain different sets of sampled points Q = {qi, q2: ... } E R x3 These sampled farthest points Q are regarded as a prior estimation of the distribution of keypoints. We define this loss minimizing the Chamfer Distance between the predicted keypoints K and the sampled points Q. overrall loss function is
Lover Ossm Sii Offarbsar
(11)
where for and asim are scalar loss coefficients.
4
Experiments
In this section, compare the performance of Key-Grid over the existing SOTA approaches both rigid-body and deformable object datasets. Meanwhile, we conduct the robustness analysis, ablation studies, and show Key-Grid can be easily extended to -E(3iiaquivarriat version.
4.1 Setup
Datasets. We use the ShapeNetCore and the ClothesNet datasets 2
49 evaluate the
performance of Key-Grid. For the ShapeNetCoreV2 dataset [2]. it contains 51,300 rigid-body objects of 55 different categories. In this paper, we only use categories that are manually annotated with semantic correspondence labels in the KeypointNet dataset 42]. For the ClothesNet dataset [4), we take three types of deformations on different type of clothing objects: dropping, dragging, and folding. For each deformation of different garment categories, there are 128 samples during the deformation process. We test the performance of Key- on real-world 3D scans of clothing from the Deep Fashion3D V2 dataset [6] and conduct the additional experiments on SUN3D dataset [3 to further illustrate Key-Grid capability in coping with real and large-scale dataset.
Baselines. We compare Key-Grid with the current SOTA approaches: KeypointDeformer (KD) 99] Skeleton Merger (SM) 26) and SC3K [5T] that all detect the 3D keypoints in an unsupervised way. Compared to SM (26), SC3K [11, and Key-Grid, KD [ not only requires point clouds as input but relies on object meshes to perform shape transformations from the source shape to the target shape. Therefore, regarding KD [9] baseline is actually unfair to our method. However, since it is the current SOTA method for keypoint detection on deformable objects, we choose as baseline to compare with our method.
Evaluation metrics. For the ShapeNetCore V2 dataset 2 we use the Dual Alignment Score (DAS) 26 to assess the degree of keypoints semantic consistency for each category. Meanwhile, to verify the accuracy of detected keypoint locations, we compute the mean Intersection over Union (mloU) metric 333 with threshold of 0.1 to evaluate the difference between the detected keypoints and the ground truth provided by the KeypointNet dataset [42]. For the ClothesNet dataset [49], due the absence of manually annotated keypoints, only DAS to evaluate the semantic consistency of keypoint detection on objects with three type of deformations. Higher values for both metrics correspond to better model performance.
4.2 Results
For the Seape dataset (2, we adopt Key-Grid and other baselines to detect ten keypoints and use the DAS and mloU to evaluate their performances the 13 categories of rigid-body objects in the Table For objects with straight-line geometrical structures such as "Airplane", "Vessel", "Knife", and "Guitar", skeletons formed by connecting keypoints easily represent their

---

DAST
mloU?
Category
KD
SM
SC3K
Key-Grid (Ours)
KD
SM
SC3K
Key-Grid (Ours)
Airplane
73.4
77.8
82.9
84.7
53.3
73.5
74.9
24.4
Bed
69.5
61.3
64.9
70.6
40.7
59.9
19.9
51.2
Bottle
60.3
59.9
62.7
66.2
34.0
36.5
38.0
38.7
Cup
53.1
59.7
11.3
Car
80.2
79.5
75.2
84.5
53.8
53.3
33.2
56.1
Chair
84.3
76.8
87.0
93.4
62.0
60.0
39.8
65.4
Guitar
62.9
63.2
65.7
69.1
52.6
49.7
68.0
56.2
Helmet
55.3
57.0
58.6
62.5
13.5
16.7
17.6
39.2
Knife
60.4
Motorbike
61.5
57.8
59,4
63.7
45.7
39.7
39.7
48.2
Mug
69.8
67.2
75.3
79.4
12.6
23.6
18.4
24.3
Table
72.6
70.0
76.0
78.6
60.9
46.4
66.5
Vessel
73.9
72.4
76.0
74.6
51.2
58.9
47.5
52.4
Average
67.7
65.8
69.7
72.8
40.7
45.6
38.7
49.0
Table Comparative DAS and mloU score: Key-Grid vs. State-of-the-Art Approaches on the ShapeNetCooee dataset. t means better performance. The results are calculated for 10 keypoints and the DAS value of SM and SC3K reported in [51] and [26). The mloU values of each method are the results we reproduced based on their official code. Colorbox and underlined respectively represent the first and second best performance in all tables of this paper.
Drop Clothes SC3K
Drag Clothes
Category
Key-Grid (Ours)
KD
SM
SC3K
Key-Grid (Ours)
Hat
96.7
17.7
35.7
100.0
50.0
42.7
42.7
51.5
Long Pants
57.5
46.4
77.1
83.7
87.5
90.6
81.3
99.0
Jacket
78.6
90.2
66.7
45.9
67.7
Long Dress
70.8
73.2
69.6
75.9
61.5
69.8
50.0
61.5
Short Dress
90.2
72.6
57.1
91.1
20.8
66.7
57.3
76.0
Musk
94.3
95.5
91.7
100.0
29.2
43.8
26.0
47.9
Polo
65.8
48.2
64.3
73.2
55.2
44.8
38.5
62.8
Scarf
87.5
75.0
54.5
91.1
65.6
$8.3
Shirt
74.1
55.4
84.8
87.5
59.4
45.8
45.8
60.2
Short
Skirt
89.4
37,5
92.9
95.5
52.1
53.1
21.9
59.3
Tie
79.2
87.5
37.5
79.5
55.2
56.3
33.3
64.6
Vest
96.7
76.8
96.4
100.0
67.7
50.0
62.5
89.6
Average
818
67,5
68.4
89.8
57.9
58.6
48.6
67.7
Table 2: Comparative DAS score: Key-Grid vs. State-of-the-Art Approaches on the ClothesNet dataset. We demonstrate the DAS values for 8 keypoints recognized by different methods 13 types of clothing under the dropping and dragging deformations.
geometrical structures, which is why SM outperforms Key-Grid for these categories on the mloU metric. However, Key-Grid's performance these categories is comparably optimal, achieving higher average scores for both DAS and mloU compared to other baselines. Additionally, we present various supervised networks and self-supervised methods to compare Key-Grid on the KeypointNet dataset [42] based on mIoU metric in Table 3 Several standard networks, PointNet [2I]. SpiderCNN [39]. and PointConv (36, are trained to predict keypoint probabilities in supervised manner. We can observe that Key-Grid demonstrates superior accuracy in keypoint localization compared to other self- supervieed methods and outperforms some supervised approaches that utilize PointNet and SpiderCNN backbones.
Airplane
Chair
Car
Average
PointNel 21
45.4
23.8
15.3
28.2
SpiderCNN 39
55.0
49.0
38.7
47.6
PointConv
36
93.5
86.0
83.6
87.0
SM 26
79.4
68.4
63.2
70.3
SC3K [51]
82.7
38.5
34.9
52.0
Key-Grid
80.9
75.2
69.3
75.1
Table Comparative mloU score: Key-Grid vs. Supervised Keypoint Approaches on the Key- pointNet dataset. The results of the mIoU score are calculated for 10 keypoints on the KeypointNet dataset. The results supervised keypoint detection method are reported in [26].

---

KD
SC3
Key
(d)
Figure 4: Different methods on the Hat and Long Pant categories during the dropping and dragging processes. (a) and (b): Keypoint detection of Hat under the dropping and dragging deformation. (c) and (d): Keypoint detection of Long Pant under the dropping and dragging deformation. We use lines connect keypoints of the same color, representing the positional changes of the same keypoints in the deformation process of the objects.
Fold Clothes
Calegory
Normal Placement
SK(3) Transformation
SC3K
Key-Grid (Ours)
SM
SC3K
Key-Grid (Ours)
Shitt
81.6
793
53.6
92.0
80.4
75.9
51.9
90.2
Table 4: Comparison of DAS values for Folded Clothes under Normal Placement and SE(3) Transformation. For deformations with large changes, such folding, Key-Grid has more noticeable advantage whether the clothes are placed normally or undergo SE(3) transformation.
For the ClothesNet dataset [49] aim to make the eight detected keypoints maintaining semantic consistency in objects under the deformation process of the same category. Table @aale[ssoou shows Key- Grid performs better on the deformations of dropping and dragging than other methods. Key-Grid outperforms other baselines in the drop deformation for all categories, except for "Tie", while also demonstrating superior performance in the drag deformation for all categories, except for "Long Dress" and "Scarf". Additionally, we also show the semantic consistency of keypoints identified by different methods during the folding process of shirts and pants Table Table 4 illustrates that for objects with significant deformations, such as folding, Key-Grid signiticanlly outperforms other methods in terms of keypoint semantic consistency. Figure and Figure 3 show the visualization results of keypoint detection under three deformations of clothing using different methods. In Figure for the dropping and dragging deformation process on the "Hat" and "Long Pant" categories, compared to other methods, keypoints identified by Key-Grid distribute evenly objects and keep great semantic consistency. The keypoint positions do not change with the deformation the objects. In Figure 5(a) and (b), we can observe that when facing folding deformation, keypoints detected by KD and SM have redundancy phenomenon, which means multiple keypoints are identified at same location. Compared to the SC3K method, keypoints identified by Key-Grid not only capture essential geometric details of deformable objects but also ensure semantic coherence throughout the folding process. Figure 5)c) shows the grid heatmaps are evidently better accurately reflecting the geometric structure of the folded clothes than the skeleton structures proposed in SM. Thus, we think that precise representation of the object structure using keypoints forces the network to generate precise keypoints, which makes Key-Grid perform better than previous methods.
For the Deep Fashion3D V2 dataset (6], we select three shirts with different deformations for keypoint recognition. Key-Grid successfully learns eight semantically consistent 3D keypoints for these objects, as shown in Figure(iid) even with obvious deformations in the sleeves of the clothes. The hyperparameters used in this experiment are consistent with those employed on the ClothesNet dataset.
For the SUN3D dataset 38 Figure d) presents the visualization results of 20 keypoints on the
different local geometric scenes identified by Key-Grid. Despite the influence of real-distributed noise and occlusion on keypoint detection results, Key-Grid can still recognize keypoints with important geometric information in real-level data, such as corners of buildings and center positions of scenes.

---

Skeleton Structure
Deep Fash3D V2
SuNaD
Grid Heatmap
Key Grid
(a)
(b)
(c)
(d)
Figure 5: Keypoints detected on the Fold Clothes, the Deep Fash3D V2 dataset and the SUN3D
dataset. (a) and (b): Eight keypoints identified by different methods during the folding process of clothes. The lines connect the keypoints with the same color, which means the positions these keypoints change in the deformation process. (c): Grid Heatmaps and Skeleton Structures on the fold clothes. the skeleton structures, we use purple dots to connect the keypoints identified by SM to construct the skeleton. In the grid heatmap, with yellow indicating smaller values. The yellow dots capture the geometric structure of the folded clothes. (d): Keypoints detected by Key-Grid on the Deep Fash3D V2 and the SUN3D dataset.
colors to represent the values of ((p).
DAS DAS Score
Score
Gaussian
Downssapling
(a)
Noise
Downsampling
[c]
(d) SE(3) Transformation
Occluded Point Cloud
Outlier- Point Clouds
View Point Clouds
(e)
Figure 6: Robustness Analysis of Key-Grid and Visualization Results of the Equiv-Equi Keypoints. (a) and (b): DAS value under Gaussian noise and downsampling. (c): Visualization results of Key-Grid under these situations. The Gaussian noise scale is 0.06 and the downsample rate is 8x, respectively. (d): Keypoints the folded pants undergoing SE(3) transformations. Key:Key detection on the on occluded, side view, and outlier-laden point clouds.
4.3 Robustness Analysis
In this section, focus evaluating the robustness of Key-Grid on the noisy and down-sampled point clouds folded shirts. To obtain the noisy point clouds, we add Gaussian noise with varying variances to the point clouds. And utilize the Farthest Point Sampling method to downsample the original point clouds, which has also been used in previous works (19 47).
Figure a)), (b) and (c) show the DAS and visualization results of detected keypoints for the noisy and wown-sampled point clouds. The results indicate that both SC3K and Key-Grid experience a decrease in DAS as the noise level increases. Meanwhile, when downsampling the original point cloud by

---

factor of 16, the DAS of Key-Grid and SC3K will significantly decrease. However, when Key-Grid is subjected sigmificant noise interference and high-level downsampling, the DAS of Key-Grid is still better than SC3K, which has not been subjected to any interference. The visualization results also show that keypoints detected by Key-Grid exhibit strong robustness even when subjected to Gaussian noise or downsampling.
Additionally, we also present the visualization of keypoints identified from occluded objects, outlier- laden objects, and side views of objects under the folding deformation in Figure 6 e). Key-Grid shows robust performance on both occluded point clouds and those captured from side views. Meanwhile, even when point clouds contain outliers, keypoints detected by Key-Grid maintain strong semantic consistency. Thus, we can conclude that Key-Gri Grid effectively identifies keypoints in comparatively low-quality or partial point clouds, similar to its performance with complete point clouds.
4.4 Squii--quivariant Keypoints
It is significant that Key-Grid is capable of identifying keypoints of objects undergoing SE(3) trans- formations. We analyze the capability of Key-Grid to identify keypoints of objects undergoing SE(3) transformations. However, if using a SPRIN module [3 which is SOTA SE(3)-
backbone to replace PointNet++ [22] directly, the training process of Key-Grid does not converge. Thus, to achieve Key-Grid to accurately identify the keypoints of objects undergoing SE(3) transformations, we regard Key-Grid as the teacher network USEEK [40]. The student network of USEEK is the SPRIN module. Figure () shows that the keypoints of folded pants recognized by USEEK invariant under an assortment of random SE(3) transformations. We also consider other baselines as the teacher model of USEEK. Table demonstrates that Key-Grid outperforms other baselines in terms of identifying the SE(3)-Equivarian keypoints with semantic consistency.
4.5
Ablation Studies
Strategy of decoder. In the decoder section, Key-Grid combines the information from each layer of the encoder with the grid heatmap to reconstruct the original point cloud. In order to illustrate the importance of encoder information and grid heatmap for the reconstruction process, Key-Grid respectively uses one of these two strategies the decoder section. Table 5 that when using both input streams to reconstruct the point cloud, the keypoints detected by Key-Grid exhibit better semantic consistency.
Loss ablation. To emphasize the importance of each loss, we conduct the evaluation of the proposed approach by systematically excluding each loss individually. The results are illustrated in Table 5 We can observe that the DAS of Key-Grid decreases when any of the loss functions is excluded from the training process. The similarity loss contributes comparatively low, but regardless of whether Key-Grid is applied on the ClothesNet dataset or the ShapeNetCoreV2 dataset, it still can improve the semantic consistency of the keypoints identified by Key-Grid. The contribution of the farthest point keypoint loss is comparatively higher than the similarity loss. Without the farthest point keypoint loss. the DAS of Key-Grid will decrease significantly.
ShapeNetCCrre22
NlothesNet
Airplane
Chair
Folded Shirt
Folded Pant
Key-Grid
84.7
93.4
92.0
100.0
No Encoder Information
73.4
82.6
81.7
90.8
No Grid Heatmap
73.2
81.7
79.5
87.5
No Similarity
71.5
78.6
77.7
84.5
No Farthest Point
15.6
13.7
17.9
17.0
Table 5: DAS of ablation study on the ShapeNetCoree2 and ClothesNet dataset.
5 Conclusion
In this paper, propose Key-Grid that detects keypoints for both rigid-body and deformable objects. uses the detected keypoints to build grid heatmap and incorporate it into the reconstruction process of point clouds. We evaluate the quality of keypoints multiple datasets and analyze the robustness of Key-Grid. Meanwhile, we embed the Key-Grid into the USEEK framework to produce SE(3)- equivariant keypoints. Extensive experiments shows that Key-Grid can detect the keypoints with great semantic consistency and precise location.
10

---

References
[1] Alper Canberk, Cheng Chi, Huy Ha, Benjamin Burchfiel, Eric Cousineau, Siyuan Feng, and Shuran Song. Cloth funnels: aunoncalizee-aalggmeenr for multi-purpose garment manipulation. 2023 IEFE International Conference on Robotics and Automation (ICRA), pages 5872-5879, IEEE, 2023.
[2] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat. Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv; 1512.03012, 2015.
[3] Nenglun Chen, Lingjie Liu, Zhiming Cui, Runnan Chen, Duygu Ceylan, Changhe Tu, and Wenping Wang. Unsupervised learning of intrinsic structural representation points. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9121-9130, 2020.
[4] Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, and Xi Zhou. Joint 3d face reconstruction and dense alignment with position map regression network. In Proceedings of the European conference on computer vision (ECCV), pages 534-551, 2018.
[5] Yisheng He, Wei Sun. Haibin Huang. Jianran Liu, Haoqiang Fan, and Jian Sun. Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1632-11641, 2020.
[6] Zhu Heming, Cao Yu, Jin Hang. Chen Weikai, Du Dong, Wang Zhangye, Cui Shuguang, and Han Xiaoguang. Deep fashion3d: A dataset and benchmark for garment reconstruction from single images. In Computer Vision - ECCV 2020, pages 512-530. Springer International Publishing. 2020.
Sagar Imambi, Kolla Bhanu Prakash, and GR Kanagachidambareean, Pytorch. Programming with TensorFlow: Solution for Edge Computing Applications, pages 87 804104, 2021.
[8] Aaron S Jackson, Adrian Bulat, Vasileios Argyriou, and Georgios Tzimiropoulos. Large pose 3d face reconstruction from a single image via direct volumetric cnn regression. Proceedings of the IEFF international conference on computer vision, pages 1031-1039, 2017.
[9] Tomas Jakab, Richard Tucker, Ameesh Makadia, Jiajun Wu, Noah Snavely, and Angjoo Kanazawa. Keypointdeformer: Unsupervised 3d keypoint discovery for shape control. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[10] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980. 2014.
[11] Avisek Lahiri, Arnav Kumar Jain, Sanskar Agrawal, Pabitra Mitra, and Prabir Kumar Biswas. Prior guided gan based semantic inpainting. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 13696-13705, 2020.
[12] Jiaxin Li and Gim Hee Lee. Usip: Unsupervised stable interest point detection from 3d point clouds. In Proceedings of the IEEE/CVF international conference on computer vision, pages 361-370 2019.
[13] Shifeng Lin, Zunran Wang, Yonggen Ling, Yidan Tao, and Chenguang Yang. E2ck: End-to-end regression network based on keypoint for 6d pose estimation. IEEE Robotics and Automation Letters, 7(3):6526-66) 6533, 2022.
[14] Yiqun Lin, Lichang Chen, Haibin Huang, Chongyang Ma, Xiaoguang Han, and Shuguang Cui. Task-aware sampling layer for point-wise analysis. IEEE Transactions on Visualization and Computer Graphics, 2022.
[15] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. Pose guided person image generation. Advances in neural information processing systems, 30, 2017.
[16] Lucas Manuelli, Wei Gao, Peter Florence, and Russ Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. In The International Symposium of Robotics Research, pages 132-157. Springer, 2019.
[17] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowazin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1460-44700 2019.
[18] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 909-918. 2019.
11

---

[19] Seyed Saber Mohammadi, Yiming Wang. and Alessio Del Bue. Pointview- 3d shape classification with multi-view point clouds. In 2021 IEEE International Conference Image Processing (ICIP), pages 3103-3107. IEEE, 2021.
[20] James O' Neill. An overview of neural network compression. arXiv preprint arXiv:2006. 2020.
[21] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference Computer Vision and Pottern Recognition, pages 652-660, 2017.
[22] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnct++: Deep hicrarchical feature metric space, Advances in neural information processing systems, 30, 2017.
learning point
[23] Olaf Ronneherger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assister CiseveentionMICCCAA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[24] Haochen Shi, Huazhe Xu, Samuel Clarke, Yunzhu Li, and Jiajun Wu. Robocook: Long-horizen elasto- plastic object manipulation with diverse tools. arXiv preprint 3rXiv:2306.14444 14447, 2023.
[25] Haochen Shi, Huazhe Xu, Zhiao Huang, Yunzhu Li, and Jiajun Wu. Robocraft: Learning to see, simulate, and shape clasto- -passtc objects with graph networks, arXiv preprint (xXiv:225.002909 2022.
[26] Ruoxi Shi, Zhengrong Xue, Yang You, and Cewu Lu. Skeleton merger: an unsupervised aligned keypoint detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 43-52. 2021.
[27] Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliere, and Nicu Sebe. Deformable gans for pose- based human image generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3408 3408-3416, 2018
[28] Shih- Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin, A-nerf: Articulated neural radiance fields for lcarning human shape, appcarance, and pose. Advances in Neural Information Processing Systems, 34: 14:12-1222291 2021.
[29] Weiwei Sun, Andrea Tagliasacchi, Boyang Deng, Sara Sabour, Soroosh Yazdani, Geoffrey E Hinton, and Kwang Moo Yi. Canonical capsules: Self-supervised capsules in canonical pose. Advances in Neurul information processing systems, 34:24993-25005, 2021.
[30] Supasorn Suwajanakorn, Noah Snavely, Jonathan J Tompson, and Mohammad Norouzi. Discovery of latent 3d keypoints via end-to-end geometric reasoning. Advances in neural information processing systems, 31, 2018.
[31] Levente Tamas and Lucian Cosmin Goron. 3d semantic interpretation for robot perception inside office environments. Engineering Applications of Artificial Intelligence. 32:76 87, 2014.
[32] Junshu Tang. Zhijun Gong, Ran Yi, Yuan Xie, and Lizhuang Ma. Lake-net: Topology-awar point cloud completion by localizing aligned keypoints. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 1726-1735, 2022.
[33] Leizer Teran and Philippos Mordohai. 3d interest point detection via discriminative learning. In Computer Vision- ECCV 2014: 13th European Conference, Zurich. Switzeriand. September 6-12, 2014, Proceedings. Part I 13. pages 159-173. Springer. 2014.
[34] Hanyu Wang, Jianwei Guo, Dong-Ming Yan, Weize Quan, and Xiaopeng Zhang. Learning 3d keypoint descriptors for non-rigid shape matching. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3-19, 2018
[35] Guangshun Wei, Long Ma, Chen Wang, Christian Desrosiers, and Yuanfeng Zhou, Multi-task joint learning of 3d keypoint saliency and correspondence estimation. Computer-Aided Design, 141:103105, 2021.
[36] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 9621-9630. 2019.
[37] Fanbo Xiang. Yuzhe Qin, Kaichun Mo. Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang. Yifu Yuan. He Wang, et al. Sapien: A simulated part based interactive environment. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 11097-11107, 2020.
12

---

[38] Jianxiong Xiao, Andrew Owens, and Antonio Torralba. Sun3d: A database of big spaces reconstructed using sfm and object labels. In Proceedings the IEEE international conference computer vision, pages 1625-1632, 2013.
[39] Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning point sets with parameterized convolutional filters. In Proceedings of the European conference on computer vision (ECCV), pages 87-102, 2018.
[40] Zhengrong Xue, Zhecheng Yuan, Jiashun Wang. Xueqian Wang, Yang Gao, Huazhe Xu. Useek: Unsupervised se cquivvaritt 3d keypoints for generalizable manipulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2023.
[41] Yang You. Wenhai Liu, Yanjie Ze. Yong-Lu Li, Weiming Wang. and Cewu Lu. Ukpgan: A general self-supervised keypoint detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17042- 17051, 2022.
[42] Yang You, Yujing Lou, Chengkun Li, Zhoujun Cheng, liangwei Li, Lizhuang Ma, Cewu Lu, and Weiming Wang. Keypointnet: A large-scale 3d keypoint dataset aggregated from numerous human annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13647- 13656, 2020.
[43] Yang You. Yujing Lou, Ruoxi Shi, Qi Liu, Yu- lo-Winng Tai, Lizhuang Ma, Weiming Wang, and Cewu Lu. Prin/sprin: On extracting point-wise rotation invariant features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1 (4(12):9489-995
2021.
[44] Tao Yu, Kaiwen Guo, Feng Xu, Yuan Dong Zhaoqi Su, Jianhui Zhao, Jianguo Li, Qionghai Dai, and Yebin Liu. Bodyfusion: Real-time capture of human motion and surface geometry using single depth camera. In Proceedings of the IEEE International Conference Computer Vision, pages 910-919, 2017.
[45] Xinyuan Yu, Siheng Zhao, Siyuan Luo, Gang Yang, and Lin Shao. Diffelothai: Differentiable cloth simulation with intersection-fre frrce frictional contact and differentiable two-way coupling with articulated rigid bodies. In 2023 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS). IEEE, 2023.
[46] Haocheng Yuan, Chen Zhao, Shichao Fan, Jiaxi Jiang, and Jiaqi Yang. Unsupervised learning of 3d semantic keypoints with mutual reconstruction. arXiv preprint arXiv:2203.10212. 2022.
[47] Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. 2018 international conference on 3D vision (3DV), pages 728-737. IEEE, 2018.
[48] Yifan Zhao, Lingjin Xu, Lin Li, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-ane aware transformers. Proceedings of IEEE/CVF International Conference on Computer Vision, pages 12236-12245 12245, 2021.
[49] Bingyang Zhou, Haoyu Zhou, Tianhai Liang, Qiaojun Yu, Siheng Zhao, Yuwei Zeng, Jun Lv, Siyuan Luo, Qiancai Wang, Xinyuan Yu, et al. Clothesnet: An information-rich 3d garment model repository with simulated clothes environment. arXiv preprint arXiv:2308.09987, 2023.
[50] Qingnan Zhou and Alec Jacobson. Thingi10k: A dataset of 10,000 3d-printing models. arXiv preprint arXiv: 1605.04797, 2016.
[51] Mohammad Zohaib and Alessio Del Buc. Sc3k: Self-supervised and coherent 3d keypoints estimation from rotated, noisy. and decimated point cloud data. arXiv preprint arXiv:2308.00 054400 2023.
[52] Mohammad Zohaib, Matteo Taiana, Milind Gajanan Padalkar, and Alessio Del Bue. 3d key-points estimation from single-view rgb images. In International Conference Image Analysis and Processing, pages 27-38. Springer, 2022.
13

---

Appendix Materials
A
Implementation Details
We use the PyTorch framework to train our method. The entire network is optimized on the Adam optimizer [0] with learning rate of 0.1. For all the experiments, the batch size is set 8. Our method is trained for 100 epochs, and for the first 20 epochs of training, we do not use similarity loss function which measures the similarity between the reconstructed point clouds and the original point clouds.
For all the experiments, the input point cloud contains N 2048 points. The number of encoder layers L in the PointNet++ network [22] is 4. For the ShapeNetCore dataset [2), we set the number of keypoints K to be 10. Thus, when calculating the farthest point keypoint loss function, we choose the number of the farthest points J as 14. For the ClothesNet [49] and Deep Fash 3D dataset [6), we and 12, respectively. The hyperparameter K and J are 20 and 24 in the SUN3D
set K and J dataset [38]. the process of building the grid heatmap, the number of points M uniformly selected on each edge of the cubic space is 16. For the decoder, the hyperparameter N Neig for the number of neighboring features applied in aligning the feature from the former layer to the current layer is set to
In the loss function, a for and Qaim are both
B Efficiency of Key-Grid
Table 6 presents the time and memory consumption of Key-Grid and SM when inferring batch comprising 32 samples on a single 1080 Ti GPU. According to Table 6 Key-Grid achieves equal efficiency to the baseline method (SM) in terms of inference speed but consumes acceptably larger memory size.
Method
Memory (MiB)
Time (s)
SM
4531
0.6
Ours
6081
0.6
Table 6: Analysis of Key-Grid efficiency.
C Visualization Results
In this section, we primarily present additional visualization results about the experimental part to illustrate the effectiveness of Key-Grid.
Number of keypoints. We vary the number of unsupervised keypoints discovered by Key-Grid on the folded pants. Figure shows that these different numbers keypoints on the folded pants maintain good semantic consistency during the deformation process. And the number of keypoints increases, we find that these keypoints do not overlap with each other, they still evenly distribute on the surface of the objects, and each keypoint represents the geometric information of the objects.
Ablation study. We show the visualization results of the ablation study in Figure g From the Figure 8 we find that in the decoder section, if only the encoder information the grid heatmap information to reconstruct the original point cloud, the positions of the keypoints on the folded pants not stable. When we do not use the similarity loss function in the decoder section, the keypoints will be placed in the wrong prior positions and the network predicts the redundant keypoints. Moreover, when we do not apply the farthest point keypoint loss function to train the network, the performance of keypoint recognition will be poor, as the keypoints will cluster around the center of the object.
Gaussian noise to point cloud. Figure shows the visualization results of Key-Grid for different noisy point clouds, which add the Gaussian noise different scales to the input point cloud. We can observe that even when Gaussian noise is added the original point cloud with magnitude of 0.08,
14

---

Unsupervised Keypoints
Unsupervised Keypoints
10 Unsupervised Keypoints
12 Unsupervised Keypoints
Figure Different number detected keypoints on the folded pants. Key-Grid identifies different numbers (6, 8, 10, 12) of keypoints on the folded pants, respectively.
the positions of the keypoints detected by Key-Grid in these noisy point clouds are still close to those found in the original point cloud, Thus, can conclude that Key-Grid exhibits robustness when dealing with the noisy point clouds.
Downsampling point cloud. This paragraph presents the performance of Key-Grid on the down- sampled point clouds shown in Figure TO For decimating the original point cloud, we use the Farthest Point Sampling method for downsampling. We discover that Key-Grid successfully estimate the keypoints on the downsampled point clouds. When downsampling the original point cloud by factor of 16, resulting in a point cloud with 128 sample points, except for the blue keypoints which will be shifted downwards, the positions of the other keypoints will remain close to their positions detected in the original point cloud.
Deep Fash3D V2 dataset. We show the additional visualization results of keypoints detected by Key-Grid on the Deep Fash3D V2 dataset [6]. Figure 13 illustrates that when faced with a wider variety of clothing cuff deformations, the keypoints detected by Key-Grid achieves great semantic consistency on the world scanned objects and represents the important information about these objects. Thus, think that Key-Grid performs well in the world scanned objects.
ShapeNetCore V2 and ClothesNet dataset. For ShapeNetCoreV2 [2] and ClothesNet 449 dataset, we present the visualization results of keypoint recognition wider range of object categories using SM and Key-Grid In Figure we separately present the visualization results of keypoint recognition on the ShapeNetCore V2 dataset using SM and Key-Grid. Compared the redundancy phenomenon that many keypoints locate in the same region generated by SM (26), the keypoints detected by Key-Grid distribute separately on surface the object, and these keypoints also can represent important geometric information of the object. For the ClothesNet dataset, Figure 13 demonstrates the keypoint detection SM (26) and Key-Grid in the deformation process wider range of object categories, including drop and drag deformations. In the deformation process, compared to SM (26).
15

---

Key-Grid
No Encoder Information
No Grid Heatmap
No Similarity Loss
No Farthest Point Keypoint Loss
Figure 8: Visualization results of the ablation study on the folded pants.
Noise 0.00
Noise 0.02
Noise 0.04
Nobse 0.08
Figure 9: Performance of Key-Grid on the noisy point clouds. We indicate the level of Gaussian noise added underneath each noisy point cloud, "Noise 0.00" is the original point cloud.
the keypoints identified by our method keep good semantic consistency and distribute evenly the surface of the object. Thus, for the rigid-body and deformable objects, Key-Grid exhibits great performance keypoint detection.
D
Video Results
In the supplementary materials, we also provide keypoint recognition results for the folding clothes the ClothesNet dataset using different methods. The video we provide include the entire process of deformation for shirts and pants. From this video, we can conclude that Key-Grid identifies the keypoints that maintain good semantic consistency during the deformation process. These keypoints also carry the geometric feature information of the deformable objects.
16

---

sampled points
1024 sampled points
sampled points
128 samoled points
Figure 10: Performance of Key-Grid on the downsampled point clouds. The input point clouds are downsampled for different scales, as mentioned at the bottom of each point cloud. "2048 sampled points" is the input point cloud.
Figure 11: Additional visualization results on the Deep Fash3D V2 dataset. Key-Grid recognizes the keypoints in these three garments with different deformation.
E Limitation and Social Imapet
Following previous works, we need to manually set the total number of predicted keypoints before- hand. In future research, will propose adaptive keypoint detection method that can generate varying numbers of keypoints with accurate positions for different samples. Key-Grid predicts the keypoints on the publicly available datasets. Thus, we think Key-Grid has very limited potential negative societal impacts.
17

---

(a) Keypoints detected by SM on the rigid-body objects
(b) Keypoints detected by Key-Grid on the rigid-body objects
Figure 12: Keypoint detection on the ShapeNetCoreV2 dataset. The keypoints are detected by SM and Key-Grid the "Bed", "Guitar", "Car", and "Motorcycle" category of objects in the ShapeNetCore V2 dataset. Each category shows four samples.
18

---

(a) Keypoints detected by SM on the deformable objects
(b) Keypoints detected by Key-Grid on the deformable objects
Figure 13: Keypoint detection the ClothesNet dataset. For the drop and drag deformation processes, we select the "Long Dress", "Tie", and "Vest" categories of objects to visualize the keypoints identified by SM and Key-Grid. We lines connect the keypoints the same position during the deformation process.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction section include the contributions and scope of the study.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In the appendix, we describe the limitations of our study.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
Our article does not involve theoretical proofs and results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In the experimental section, we disclose the datasets required for our study and provide the hyperparameters required for each dataset in the appendix, to facilitate the reproducibility of our experimental results for the readers.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release the code facilitate the reproducibility of our work for the readers. The code is already provided in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
coblic//uuides/Coeettbbmsss otw.ooeeessboiiisooooooooiioog for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppplccggua.aeaeaeccuuea
Stteeee//ooSSSeeissSioica www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We disclose the necessary details required for the experiments in both the experimental section and the appendix to assist readers in understanding our experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We do repeat the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the appendix, we discuss the memory usage and inference time of our method.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tthisssattsss//nuurpppss..
Answer: [Yes]
Justification:
Our research adheres the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:[Yes]
Justification:
Our paper discusses socictal impacts in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our paper does not have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Our paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "kDqIv3fBs0", "reviewer": "Reviewer_YtUf", "summary": "Novel PointNet-based autoencoder method  called KeyGrid, that predicts semantic keypoints on objects, even when objects are subject to deformations. Similar to previous approaches, keypoints are produced as a linear combination of inputs points, according to a learned weight matrix.\n\nThe key novelty over related works using PointNet++-based auto-encoders such as SkeletonMerger (SM) [26] is to pass a densely sampled grid of features (3d “point heatmap”) at each decoder layer (which are hierarchical). In this heatmap grid, each cell holds the maximum of the weighted distances to the 'keypoint skeleton segments'. \nAlso, KeyGrid appears to pass richer encoder information to the decoder than the SM work, although the differences here are less clear. \n\nThe method is shown to outperform previous baselines such as SK3D and SM on ShapeNetV2 dataset (more right objects) and on ClothesNet (deformable objects).", "questions": "- Compared to SM[26], is the keypoint heatmap the main difference, how about the use of hierarchical decoder. \n- Wouldn't a richer shape descriptor heatmap do even better than a max function? (e.g. K channels, distances to the keypoints, etc). \n- How do you get keypoint correspondences across shapes for your model, do they emerge naturally? \n- What are the method results on KeypointNet dataset? Can you provide some additional comparisons to supervised baselines on your datasets, if there are such? \n- Can this method be applied on datasets where shapes are significantly occluded (e.g. viewed from a specific direction)?", "limitations": "Limitations have been sufficiently addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Beats KeypointDeformer (KD), SkeletonMerger (SM), SC3K on almost all categories on the ClothesNet dataset and ShapeNetCoreV2. Deformable objects keypoints seem to improve more. \n- Ablations in Tab. 4 indicate the importance on 4 (among dozens) of semantic classes how encoder-info, a grid heatmap, farthest point sampling loss, and similarity (i.e. chamfer loss) each boost performance.", "weaknesses": "# Significance\n- Is the hierarchical decoder setup (excluding the heatmap grid itself) novel, esp compared to SkeletonMerger [26]? Right now there are no specific claims or ablations related to this. \n- The primary novelty relative to SM [26] seems to be the use of keypoint grid heatmaps in the decoder, storing max distances from grid cells to the weighted keypoint segments. However, this is just one possible shape descriptor feature, and intuitively it should be quite a weak and ad hoc feature, even if the authors gave some intuition why it may be better than the min function. Would other shape feature with yet more information be more useful could (e.g. why not have K channels with distances to all segments or to the keypoints as opposed to just taking the max, etc)? In general it would help if the paper provided more intuition or experimental results giving more insight on the information contained in this heatmap. \n- This class of approaches seems generally limited to cases where entire point clouds were visible. How would you handle cases when some parts of the objects were occluded, such as in real-world scenes? This has not beed discussed. \n\n# Related work \nCould discuss other dense representations for correspondence like the pointmaps used in “DUSt3R: Geometric 3D Vision Made Easy” https://arxiv.org/pdf/2312.14132 )\n\n# Clarity\n- I found Sec 3.3 particularly difficult to follow and understand. A rewrite, or further details in the appendix clarifying better how these quantities are 'composed' and aligned would help. \n- Fig 1 illustrates points heatmap in the decoder by simply showing a point cloud. I was expecting pictures like in Fig 4c. \n- L13, L58 Unclear where keypoint pairs come from, when discussed for first time. \n- No discussion of how correspondences are established, e.g. as shown in Figure 3. \n- The loss terms in Equation 11 are not mathematically defined, would be helpful in the Appendix. \n\n# Experimental results\n  - KeypointDeformer, SkeletonMerger, and SC3K are tested on KeypointNet dataset (https://arxiv.org/pdf/2002.12687 ). Why do the authors omit it here? Other works such as SC3K state “We use KeypointNet dataset [36 ] in our experiments, considering that this is the standard and most recent dataset used for keypoints estimation”. If the results are not as good on that dataset, that is okay, as long as we provide them and state that our method is specifically designed for datasets with deformable/soft objects. This would provide confidence that the authors ran their eval of official source code correctly on the two new datasets. \n  - Why no comparisons with supervised baselines?\n  - Figure 4D: Sun3D keypoints are so small in the figure it appears as if they are randomly distributed\n\n# Some Grammar & Style Nits:\n- L453 SC3K bibtex should be ICCV 2023, not Arxiv “\n- L148 typo “giving a total of 4096 gird points.” -> “…4096 grid points”\n- L229 “mean Intersection over Unions” -> “mean Intersection over Union”\n- Grammar: Figure 3: caption: “Different methods on the Hat and Long Pant during” ->  prefer “Different methods on the Hat and Long Pant categories during”\n- Table 3 caption is too close to Table 3, whitespace seems too shrunken\n- L75 “Deformable object dataset” -> “Deformable object datasets”\n- Table 1 caption “codes” -> should be singular, not plural (“the results we reproduced based on their official codes” -> “the results we reproduced based on their official code.”)\n- Color scheme is jarring in Tables 1-4, prefer tango colors that have a more attractive colormap (https://sobac.com/sobac/tangocolors.htm)\n- L3 extraneous article: “focus on the rigid body objects” -> “focus on rigid body objects”\n- L6 extraneous article: “for both the rigid-body and deformable objects” -> “for both rigid body and deformable objects”\n- L300 “the importance of encoder information and grid heatmap for reconstruction process,” -> “...for the reconstruction process,”\n- L302: “Table 4 show that the keypoints detected by Key-Grid which utilizes both two strategies to reconstruct the point cloud have better semantic consistency.” -> “Table 4 shows that when using both input streams to reconstruct the point cloud, the keypoints detected by Key-Grid exhibit better semantic consistency”\n- L9-10 Instead of “Unlike previous work, we leverage the identified keypoint information to form a 3D grid feature heatmap 1called grid heatmap, which is used in the decoder section”, prefer to say something like “..to form a 3D grid feature heatmap which we refer to as a grid heatmap. A grid heatmap is…” \n- L14 “Into the decoder section” -> “into the decoder model”? Ambiguous what “section” refers to\n- L51 “aiming at the semantic consistency” -> “aiming for semantic consistency”, e.g. “an unsupervised keypoint detector on 3D point clouds aiming for semantic consistency under shape variations of both rigid-body and deformable objects.”\n- L74 section should be named “Related Work” not “Related Works”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing thoughtful and detailed feedback, which greatly enhances the quality of our article.  Based on your comments regarding related work, clarity, and grammar & style nits, we will revise our paper in the came ready section.\nRegarding your inquiries about experimental results and significance sections, we address each one individually. Additionally, we provide the new visualization results of keypoint detected by Key-Grid on the SUN3D dataset in the new submission material.\n\n$\\color{Indigo}Q1$:  Compared to SM, is the keypoint heatmap the main difference, how about the use of hierarchical decoder.\n\n$\\color{red}A$: The main distinction between SM and Key-Grid lies in the decoder design. We incorporate information from the encoder into the decoder to reconstruct point clouds through hierarchical decoding. Additionally, we integrate grid heatmap information during the reconstruction process. Therefore, we consider the hierarchical decoder and the grid heatmap as the two significant innovations of our method. In our paper, we conduct ablation experiments on the hierarchical decoder in Section 4.5. In Table 4 from our paper, the label “No Encoder Information” means the ablation of the hierarchical decoder.  We observe that when we do not use a hierarchical decoder, our method's performance will decrease on both the deformable and rigid objects.\n\n\n$\\color{Indigo}Q2$:  Wouldn't a richer shape descriptor heatmap do even better than a max function? (e.g. K channels, distances to the keypoints, etc).\n\n$\\color{red}A$: In the following table, we display the impact of grid heatmaps with different distance definitions on keypoint recognition in deformable objects. These definitions are the K-channel distances from each grid point to all segments or keypoints. For the folding deformation, the distance defined in our paper achieves great performance on the semantic consistency of keypoints. We think that adopting a K-channel distance will result in some overlap of distance information between the point clouds outside the deformable objects and those inside. This condition reduces the ability of the grid heatmap to depict the structure of the deformable objects.\n\n|                                      | Folded Shirt | Folded Pant |\n|--------------------------------------|----------------|--------------|\n| Distance to keypoints     | 86.4           | 93.1         |\n| Distance to segment       |   88.7         | 95.8       |\n| Our                                | 92             | 100          |\n\n$\\color{Indigo}Q3$:  How do you get keypoint correspondences across shapes for your model, do they emerge naturally?\n\n$\\color{red}A$:  The positions of keypoints are outputted in an orderly manner by Key-Grid and other baselines(SC3K, SM, and KD).  In our article's visualization of keypoints, we use different colors to represent keypoints outputted in different orders. During the entire deformation process, if keypoints of the same color (with the same output order) maintain their positions unchanged, we refer to these keypoints as having good semantic consistency.  The output of keypoints with good semantic consistency can trace various positions that contain essential geometric information in objects when objects undergo deformation. This is crucial for robot manipulation of deformable objects and understanding the deformation process of objects.  In our paper, Key-Grid naturally outputs keypoints with semantic consistency.\n\n$\\color{Indigo}Q4$:  What are the method results on KeypointNet dataset? Can you provide some additional comparisons to supervised baselines on your datasets, if there are such?\n\n$\\color{red}A$: Thank you for pointing out this issue. In the following table, we present various supervised networks and self-supervised methods to compare Key-Grid on the KeypointNet dataset based on mIoU metric. Several standard networks, PointNet [1], SpiderCNN [2], and PointConv [3], are trained on KeypointNet to predict the probability of each point being a keypoint in a supervised manner. We can observe that compared with other self-supervised methods, Key-Grid recognizes the more accurate location of keypoint, and it performs even better than some supervised methods using PointNet and SpiderCNN as backbones.\n\n|           | Airplane | Chair | Car   | Average |\n|-----------|----------|-------|-------|---------|\n| PointNet  | 45.4     | 23.8  | 15.3  | 28.2    |\n| SpiderCNN | 55.0     | 49.0  | 38.7  | 47.6    |\n| PointConv | 93.5     | 86.0  | 83.6  | 87.0    |\n| SM        | 79.4     | 68.4  | 63.2  | 70.3    |\n| SC3K      | 82.7     | 38.5  | 34.9  | 52.0    |\n| Key-Grid      | 80.9     | 75.2  | 69.3  | 75.1    |\n\n$\\color{Indigo}Q5$: Can this method be applied on datasets where shapes are significantly occluded (e.g. viewed from a specific direction)?\n\n$\\color{red}A$:  Thanks for your insightful suggestion. Keypoints detected by Key-Grid estimated on the deformation of objects from side views are depicted in Figure 1 from the new submission material. Meanwhile, we also show the keypoint detection results when objects' shapes are occluded in Figure 1 from the new submission material. We find that Key-Grid exhibits robustness on the occluded point clouds and point clouds obtained from different views.\n\n[1] Pointnet: Deep learning on point sets for 3d classification and segmentation. CVPR 2017\n\n[2] Spidercnn: Deep learning on point sets with parameterize convolutional filters. ECCV 2018\n\n[3] Pointconv: Deep convolutional networks on 3d point clouds. CVPR 2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8LgRi3qcfQ", "reviewer": "Reviewer_U8aQ", "summary": "The authors propose a novel unsupervised method to detect 3D point clouds key points by producing an intermediary heatmap based on a grid and points distances from the skeleton and connected key points. They achieve state-of-the-art accuracy and semantic consistency and easily achieve Se-(3) invariance with minimal adaptation of the method.", "questions": "Perhaps I missed this point, but how does the author ensure that the key points are semantically the same across different folding/deformation of the objects? ( and give them the same colour)", "limitations": "Perhaps the only limitation I can pick up is the consistency across more than two deformations of an object. Would the method perform well over long-term deformations? It ties with showing one or two failure cases of the method.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The method's description, results, and a very in-depth ablation study, together with many supplementary materials, make this paper pleasant to read and easy to understand, with impressive results. Although simple at first glance, the method has the merit of being solid and achieving good results. Great results section and analysis allow the reader to understand in depth the method and results.", "weaknesses": "A small neat-picking addition would be to add one or two visual examples where the method does not work super well, to contrast with the excellent results presented in the paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We address your concerns as follows: we provide demonstrations of our method on some less successful objects and investigate whether keypoints maintain semantic consistency across various deformations of objects.\n\n$\\color{Indigo} Q1$：A small neat-picking addition would be to add one or two visual examples where the method does not work super well, to contrast with the excellent results presented in the paper\n\n$\\color{red}A$:   We present the visual results of the drag deformation for 'Mask' and 'Skirt' in  Figure 4 of our submission material, which exhibit lower DAS values than other categories on the ClothesNet dataset. Although the semantic consistency of keypoints detected on Masks and Skirts is relatively lower than other categories, keypoints identified by Key-Grid uniformly distribute across the object's surface, located at positions rich in geometric information compared to other methods. For instance, keypoints detected by Key-Grid are evenly distributed along the straps of the mask.\n\n \n$\\color{Indigo} Q2$：Perhaps I missed this point, but how does the author ensure that the key points are semantically the same across different folding/deformation of the objects? \n\n$\\color{red}A$:  Figure 2 of our submission material shows  the visualization results of keypoints detected by Key-Grid on long pants under the dropping, pulling, and dragging deformation. We can observe that keypoints detected by Key-Grid keep great semantic consistency during this long-term deformation process. At the same time, we observe that keypoints identified by Key-Grid are uniformly distributed across deformable objects, with their positions containing substantial geometric information, such as the hem and waist of long pants.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C6O5JAZbVZ", "reviewer": "Reviewer_p6XE", "summary": "This paper presents Key-Grid, an unsupervised keypoint detection network designed for 3D point clouds. Unlike previous methods that emphasize leveraging various priors on 3D structures, this paper converts keypoints into a grid heatmap. This heatmap forms a continuous feature landscape across the entire 3D space, providing richer and more stable geometric descriptions of objects, particularly for deformable objects. Meanwhile, this paper achieves state-of-the-art (SOTA) results on the ShapeNetCoreV2 and ClothesNet datasets.", "questions": "If the authors address the issues mentioned in the weaknesses, I am willing to increase the rating.", "limitations": "Yes, the authors discussed the limitations and potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed grid heatmap is effective in the task of 3D keypoint detection and represents a novel approach to introducing 3D priors for deformable objects.\n\n2. The authors conducted extensive experiments, achieving SOTA performance on both rigid and deformable objects.\n\n3. The overall design follows the mainstream architecture of 3D keypoint detection, providing convenience for subsequent research in this field.", "weaknesses": "Q1. My main concern is whether it can be empirically or theoretically justified why the grid heatmap can represent a dense extension of the skeleton, and why the current method of generating the grid heatmap is reasonable. Could using the density of the 'skeleton' or the density of the point cloud directly represent the grid heatmap instead?\n\nQ2.  In the ablation study (Table 4), the accuracy drop after \"No Grid Heatmap\" is minimal, yet the Grid Heatmap is the most significant contribution of this paper. Could the authors explain this phenomenon and provide ablation studies across more categories? \n\nQ3. Can this paper adaptively select the number of detected keypoints for different categories of objects?\n\nQ4. The quality of figures 1 and 2 could be enhanced. For example, the information content of figure 1 is too minimal to serve as a teaser.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We address your concerns as follows: We explain the rationale of the grid heatmap from a theoretical perspective and highlight its advantages over the skeleton structure proposed by SM. Subsequently, we demonstrate  that the grid heatmap is more effective for keypoint detection than skeleton structure on deformable objects through experimental validation.  Furthermore, we clarify some confusions, such as the importance of the grid heatmap to the Key-Grid and whether Key-Grid can adaptively generate keypoints.\n\n$\\color{Indigo}Q1$：My main concern is whether it can be empirically or theoretically justified why the grid heatmap can represent a dense extension of the skeleton, and why the current method of generating the grid heatmap is reasonable. Could using the density of the 'skeleton' or the density of the point cloud directly represent the grid heatmap instead?\n\n$\\color{red}A$: Skeleton structures proposed by SM does not accurately represent the skeletal structure of deformed objects. For example, in Figure 2 of our paper, if you connect the red keypoint with the black keypoint,  this skeleton does not rationally represent the structure of the folded pants. Unlike SM, which directly uses the lines connecting keypoints as the object's skeleton, we characterize the overall skeletal structure of deformed objects more delicately by depicting the distances between grid points and keypoints pair line. Figure 4(C) of our paper shows the visualization results of the grid heatmaps and skeleton structures proposed by SM， denoting that the grid heatmap provides a more accurate depiction of the underlying skeletal structure of deformed objects. We also report the performance of Key-Grid which replaces the grid heatmap with the skeleton on the fold deformation in the following table. We can conclude that the grid heatmap helps Key-Grid identify keypoints with more semantic relevance on the fold deformation of clothes.\n\n|              | Folded Shirt | Folded Pant |\n|--------------|----------------|--------------|\n| Grid Heatmap |     92.0       |     100.0      |\n| Skeleton  Structure   |     83.9       |     92.7     |\n\n\n$\\color{Indigo}Q2$:  In the ablation study (Table 4), the accuracy drop after \"No Grid Heatmap\" is minimal, yet the Grid Heatmap is the most significant contribution of this paper. Could the authors explain this phenomenon and provide ablation studies across more categories?\n\n\n$\\color{red}A$: Key-Grid utilizes the farthest point keypoint loss to pre-select several initial positions for keypoints. Subsequently, we employ the Grid Heatmap and leverage encoder information for hierarchical decoding to assist the model in finalizing these pre-selected positions. In other words, the Grid Heatmap fundamentally aids the model in selecting keypoint positions rather than directly generate their positions. Therefore, when removing the grid heatmap, its overall impact on the model's effectiveness is less significant compared to removing the farthest point keypoint loss, which provides initial candidate keypoint positions.  However, in Table 4 from our paper,  “No Grid Heatmap”   actually has a greater impact on reducing the semantic relevance of keypoints compared to  “No Encoder Information”, which also illustrates the significant role of grid heatmap in the decoder. In the following table, we present an ablation study of more diverse categories of deformable and rigid objects under the DAS metric and also show the average impact of ablating different modules on the performance of Key-Grid. \n\n\n|                         | Table (Rigid) | Motorbike (Rigid) | Hat (Drop) | Long pant (Drop) | Hat (Drag) | Long pant (Drag) | Average  |\n|-------------------------|---------------|-------------------|------------|------------------|------------|------------------|--------------|\n| Key-Grid                    | 78.6          | 63.7              | 100.0        | 83.7             | 51.5       | 99.0             | 79.4          |\n| No Grid Heatmap         | 63.8          | 51.3              | 87.9       | 70.6             | 36.7       | 86.1             | 66.1(-13,3)          |\n| No Encoder Information  | 65.3          | 51.8              | 89.1       | 74.9             | 39.8       | 88.9             | 68.3(-11.1)           |\n\n\n$\\color{Indigo}Q3$:  Can this paper adaptively select the number of detected keypoints for different categories of objects?\n\n$\\color{red}A$: Thank you for your suggestion. Currently, in Key-Grid, we need to manually set the total number of predicted keypoints beforehand. In  future research, we will propose an adaptive keypoint detection method that can generate varying numbers of keypoints with accurate positions for different samples.\n\n$\\color{Indigo}Q4$:  The quality of figures 1 and 2 could be enhanced. For example, the information content of figure 1 is too minimal to serve as a teaser.\n\n$\\color{red}A$: Thanks for your advice. In the come ready version, we will submit the new version of Figure 1 and Figure 2. The new version of Figure 1 not only provides the visualizations of keypoints detected on both deformable and rigid objects but also adds the grid heatmap and skeleton structure visualizations like Figure 4(c) in our article, which demonstrates that grid heatmap can more accurately represent the geometric information of the object.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8grmkLqxbz", "reviewer": "Reviewer_LZF7", "summary": "This paper presents a novel unsupervised network for keypoint detection. The method can be applied to handle both rigid and deformable objects. It follows an autoencoder framework where the encoder predicts keypoints and the decoder utilizes the generated keypoints to reconstruct the objects. The main contribution lies in the grid based representation which is suitable to capture the shifted geometric structure of the keypoints brought by deformations.", "questions": "I have listed some questions in the weakness. The first question is the most important.\n\nIn L 14, 'Meanwhile, we incorporate the information from each layer of the encoder into the decoder section', Is this your contribution? \n\nI recommend to move Fig 6 in the appendix to the main text for clarity.", "limitations": "The author has discussed potential limitations in the paper", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1 This paper presents an unsupervised method for consistent keypoint detection.  \"Unsupervised\" means the method can be applied in larger-scale datasets without 3D annotations.\n2 The paper is easy to follow. The writing is good and clear.\n3 Extensive experiments show that this method achieves sota performance on many public datasets", "weaknesses": "1 The visualized results presented in this paper all feature uniform and complete point clouds. Can this method handle partial point clouds, such as those obtained from back-projecting depth maps? Can it deal with point clouds that contain severe outliers, not just those with some Gaussian noise? This is very important for practical use.\n\n2 In 289, However, if using a SPRIN module [41] which is a SOTA SE(3)-invariant backbone 289 to replace PointNet++ [22] directly, the training process of Key-Grid does not converge. Why? How about using other backbones? like Vector Neurons?\n\nDeng, C., Litany, O., Duan, Y., Poulenard, A., Tagliasacchi, A., & Guibas, L. J. (2021). Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 12200-12209).\n\n3 This proposed representation is based on Euclidean distance between the grid point towards each line segment, right? Why not first establish a graph and then aggregate features along the graph? I saw many papers using graph to model deformable objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your great suggestion. In the rebuttal phase, we address the reviewers' concerns about the effectiveness of our method on partial point clouds, such as those obtained from back-projecting depth map sand point clouds containing outliers. We also provide a detailed explanation of the convergence issues with the SE(3)-invariant model and highlight the advantages of using grid heatmaps over graphs.\n \n\n\n$\\color{Indigo}Q1$:  The visualized results presented in this paper all feature uniform and complete point clouds. Can this method handle partial point clouds, such as those obtained from back-projecting depth maps? Can it deal with point clouds that contain severe outliers, not just those with some Gaussian noise? This is very important for practical use.\n\n$\\color{red}A$:  In Figure 1(a) and (b), we present the visualization results of keypoints detected by Key-Grid on both point clouds with outlines and partial point clouds sampling from the depth map. We observe that Key-Grid identifies keypoints with strong semantic consistency, even when dealing with point clouds containing ten outliers in Figure 1(a). For the point clouds obtained from depth maps, we first generate depth maps of objects from multiple-angle photographs and then sample point clouds based on the depth maps. We find that  when facing the low-quality/partial point clouds, Key-Grid identifies keypoints whose locations are similar to those detected in the complete point clouds.\n\n\n\n$\\color{Indigo}Q2$:  In 289, However, if using a SPRIN module which is a SOTA SE(3)-invariant backbone to replace PointNet++ [22] directly, the training process of Key-Grid does not converge. Why? How about using other backbones? Like Vector Neurons?\n\n$\\color{red}A$:  Figure 3 in our new submission material provides the training loss of Key-Grid using SPRIN and Vector Neurons instead of PointNet++ to detect keypoints on the fold pants. By examining the curves, we observe that Key-Grid does not converge by using SPRIN and Vector Neurons in the training phase. We think the SE(3) backbone model is not converging, primarily because of the stringent training conditions. Our designed loss function fails to provide sufficient information to enable the SE(3) model to converge. Like USEEK, to achieve the convergence of the SE(3) model, the authors compute the difference between features outputted by SPRIN and features from the pre-trained PointNet++  in SM as a loss function to optimize the SE(3) model.  If the SE(3) model is directly used as the backbone in SM and Key-Grid, we only optimize it by locating the keypoints instead of providing a standard feature for the SE(3) model to learn from. Therefore, we think this is the main reason why the SE(3) model fails to converge.\n\n$\\color{Indigo}Q3$:  This proposed representation is based on Euclidean distance between the grid point towards each line segment, right? Why not first establish a graph and then aggregate features along the graph? I saw many papers using graph to model deformable objects.\n\n$\\color{red}A$: Thank you for your valuable perspective. Currently, applying graph networks to deformable objects typically involves constructing a graph structure from point clouds, where nodes represent points on the object and edges capture mesh information between these points. Subsequently, graph neural networks input graph structures and output the required points [1]. However, the role of the grid heatmap is to use keypoints to build the skeletal structure of the deformable object, aiding the network in reconstructing the original point cloud. This process is contrary to the current application of graph neural networks for deformable objects. Therefore, we think that replacing grid heatmaps with graph structures is not suitable for our approach.\n\n[1] Learning language-conditioned deformable object manipulation with graph dynamics. arXiv:2303.01310, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks to the esteemed reviewers for your insightful feedback, which has significantly enhanced the quality of our paper. Based on your suggestions, we provide the corresponding visual results in the new submission material.

**Figure 1(a)**: In response to Reviewer YtUf and Reviewer LZF7's inquiries regarding our method's capability to handle the occluded, partial, and outlier-laden point clouds, we show the visualization results of Key-Grid's performance on these types of point clouds. 

**Figure 1(b)** : Regarding the Reviewer LZF7's inquiry about whether Key-Grid can handle point clouds obtained from depth maps, we generate a depth map from multi-angle images and then illustrate keypoints identified by Key-Grid on the point cloud sampled from this long pant depth map.

**Figure 2**: For Reviewer U8aQ's concern about the effectiveness of Key-Grid on objects undergoing  long-term deformation, we demonstrate the KeyGrid's capability to capture the keypoints with high semantic consistency during the long-term deformation processes containing dropping, pulling, and dragging. In response to the YtUf reviewer's concern about the visualization result of keypoints in the Sun3D dataset is not clear, we improve the visibility of the keypoints and present additional visualization results with more samples from the Sun3D dataset.

**Figure 3**: Regarding the LZF7 reviewer's concern about the non-convergence of Key-Grid with the SE(3)-invariant backbone, we present the training loss curves of the Key-Grid method with the PointNet+/SPRIN/Vector Neurons models.

**Figure 4**: In response to the U8aQ reviewer's request to show some examples where our method does not work well, we select the visualizations of keypoint detected on the Skirt and Mask under the drag deformations.

Meta-review: The paper proposes a novel approach for 3d keypoint estimation via unsupervised learning. It initially received discordant ratings and comments from the reviewers (BR, BR, BA, SA). During the rebuttal and discussion stage, some main concerns from the reviewers have been resolved and a stronger consensus towards acceptance emerged. 3 reviewers suggested to accept (6,6,7) and one reviewer kept the initial BR score but mentioned in their comments that they are satisfied with the rebuttal and open to raise their score. Overall, the AC suggests to accept this paper as their final recommendation.
Decision: Accept (poster)