Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features

Authors: chengkai  hou
zhengrong  xue
bingyang  zhou
jinghan  ke
lin  shao
huazhe  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 03:01:25

Abstract: Detecting 3D keypoints with semantic consistency is widely used in many scenarios such as pose estimation, shape registration and robotics. Currently, most unsupervised 3D keypoint detection methods focus on the rigid-body objects. However, when faced with deformable objects, the keypoints they identify do not preserve semantic consistency well. In this paper, we introduce an innovative unsupervised keypoint detector Key-Grid for both the rigid-body and deformable objects, which is an autoencoder framework. The encoder predicts keypoints and the decoder utilizes the generated keypoints to reconstruct the objects. Unlike previous work, we leverage the identified keypoint in formation to form a 3D grid feature heatmap called grid heatmap, which is used in the decoder section. Grid heatmap is a novel concept that represents the latent variables for grid points sampled uniformly in the 3D cubic space, where these variables are the shortest distance between the grid points and the “skeleton” connected by keypoint pairs. Meanwhile, we incorporate the information from each layer of the encoder into the decoder section. We conduct an extensive evaluation of Key-Grid on a list of benchmark datasets. Key-Grid achieves the state-of-the-art performance on the semantic consistency and position accuracy of keypoints. Moreover, we demonstrate the robustness of Key-Grid to noise and downsampling. In addition, we achieve SE-(3) invariance of keypoints though generalizing Key-Grid to a SE(3)-invariant backbone.

Main paper: 
 introduction: 
 Representing objects through a set of 3D keypoints is one of the most popular and intuitive approaches to compress and comprehend 3D objects [31; 26] . Effectively exhibiting their utility, 3D keypoints have contributed to the success of a number of downstream tasks, including pose estimation, shape registration, object tracking in computer vision [15; 27; 4; 8; 44; 28; 34] , as well as various kinds of robotic manipulation tasks [40; 1; 16] .To make the detected keypoints as capable and accessible as possible, the research community is now concentrating on the unsupervised learning of semantically consistent keypoints on 3D point clouds. The implication of semantic consistency is typically twofold: the keypoints should be located at the semantically salient parts of the objects; they are also desired to be aligned within the same category even under large shape variations among diverse 3D object instances. To achieve these objectives, previous works [26; 9; 41; 3; 46] often adopt autoencoder frameworks to facilitate self-supervised training, where the encoders serving as the keypoint predictor are backbone networks [21; 22] generalizable to the shape variation, and the decoders reconstruct the input shape conditioned on the predicted keypoints. Since neural networks are in general better at compression than generation [20] ,Figure 1 : Examples of the keypoints detected by Key-Grid. The detected keypoints preserve semantic consistency under: (Top) large intra-category shape variations of rigid-body objects from the ShapeNetCoreV2 [2] dataset; (Bottom) dramatic deformations of soft-body objects from the ClothesNet [49] dataset. the primary challenge of this pipeline lies in reconstructing the entire point cloud from a few estimated keypoints. Thus, the state-of-the-art (SOTA) detectors [26; 9] lay emphasis on leveraging different priors on 3D structures (e.g., "skeleton" in Skeleton Merger [26] , and "cage" in KeypointDeformer [9] ) so that the 3D object shapes can be more reasonably approximated by the information from the detected keypoints alone.While maintaining semantic consistency is already demanding under the shape variations of rigidbody objects (e.g., those in the ShapeNetCoreV2 [2] dataset), it would be even more challenging if deformable objects are also taken into consideration. For instance, when detecting keypoints on the trousers, if one detected keypoint is located at one of the trouser legs, then the keypoint is desired to follow the motion of the trouser leg in the process of the trousers being folded (shown in Figure 1 ). Note that the shape variation caused by deformation is so dramatic that even the outline of the object shape has been completely altered, indicating a shift in the spatial and geometric structure of the keypoints. Despite the difficulty, with a growing interest in deformable object manipulation [1; 24; 25] in robotics as well as the emergence of large-scale datasets for deformable objects [49; 6] in computer vision, it is of increasing significance to develop a keypoint detector that is equally effectively when faced with deformable objects.In this work, we present Key-Grid, an unsupervised keypoint detector on 3D point clouds aiming for semantic consistency under the shape variations of both rigid-body and deformable objects. In accordance with the prevailing practice, Key-Grid uses an autoencoder framework. In response to the potentially shifted geometric structure of the keypoints brought by deformations, we propose to embed the information of the predicted keypoints into a dense 3D feature heatmap. To be more specific, we first uniformly sample a large number of grid points in the shape of a 3D array. Then, we assign a feature to each grid by calculating the shortest distance from the grid point to the connected lines of all the keypoint pairs (i.e., the "skeleton" [26] of the keypoints) and multiplying it by the weight of the connected lines. Finally, when the decoder attempts to reconstruct the point cloud, coarse-to-fine features are extracted from the dense grid feature heatmap, where the extracted point coordinates are in line with the hierarchical point sets in the PointNet++ [22] modules of the encoder. Intuitively, the grid heatmap can be viewed as a dense extension of the "skeleton" where its undefined blank spaces are smoothly extrapolated. Functionally, the grid heatmap constitutes a continuous feature landscape across the entire 3D space, providing richer and more stable geometric descriptions of the object. This could be vitally beneficial when the object undergoes intense shape variations, such as cloth deformations.Empirically, the experimental results show that Key-Grid not only achieves SOTA performance for rigid-body objects in the popular ShapeNetCoreV2 [2] dataset but also surpasses the previous SOTA [9] by 8.0% and 9.1% for objects with dropping and dragging deformation respectively in the recently proposed ClothesNet [49] dataset. Meanwhile, Key-Grid is found to be robust to noise or downsampling operations. Additionally, we also show that Key-Grid can be easily extended to an SE(3)-equivariant version when it is integrated with the USEEK [40] framework. We are committed to releasing the code. a few datasets contain deformations from the same model. Among them, the deformations from the same piece of clothing are diverse and have practical research significance. For instance, Deep Fashion3D [6] contains around 2,000 3D models reconstructed from 563 real garment instances in different poses. A subset of ClothesNet [49] contains around 3,000 3D garment meshes which can be directly loaded into differentiable simulations such as DiffclothAI [45] to generate various deformations after operating like dropping, folding, or dragging. Unsupervised keypoint detection. There have been various approaches proposed for supervised estimation of 3D keypoints using manually annotated keypoints [35; 5; 14; 52; 13; 42] . In contrast to supervised methods, our approach is unsupervised, meaning it does not rely on manually annotated keypoints. Thus, we review methods that adopt an unsupervised approach for estimating 3D keypoints. USIP [12] is the first detector that identifies 3D keypoints in an unsupervised manner by minimizing the chamfer distance between detected keypoints in randomly transformed object pairs. Canonical Capsules [29] is a similar approach that feeds pairs of a randomly translated copies of the same object into a network to detect keypoints. Following USIP, SC3K [51] also uses a random rotation to create two transformed visions of objects and generate the corresponding keypoints by mapping the keypoints of each version back to the original object. Another way to estimate 3D keypoints is proposed by Chen et al. [3] . They encode the point cloud as a set of local feature and input it into a novel structure model to generate the possibility of keypoints. Recently, Fernandez et al. [11] proposed a novel method that distinguishes between Node branches and Pose and coefficient branches to find the optimal keypoints of an object.However, these methods do not consider the geometric structure information that keypoints can represent. When they encounter irregular shape of objects, such as airplanes and cars, the keypoints they identify will lose the crucial information about the object. Skeleton Merger [26] generates the skeletons of an object connected by keypoints and uses the Composite Chamfer Distance (CCD) to make these skeletons close to the original point cloud. This makes the detected keypoints to represent the important structural information of the object. Yuan et al. [46] propose a similar way that generates keypoints by utilizing skeletons from two objects within the same category to reconstruct mutually. USEEK [40] utilizes a teacher-student network, where the teacher module is based on Skeleton Merger and the student module employs a SE(3)-invariant backbone network, SPRIN [43] . LAKe-Net [32] uses detected keypoints to achieve the shape completion by locating them to generate a surface skeleton and refining the shape of the object. KeypointNet [30] learns category-specific 3D keypoints using depth and 2D position information from a pair of 2D images. For keypoint detection on deformable objects, KeypointDeformer [9] aligns the shape of the source object to the target object by utilizing the difference in keypoints positions between them and propose a novel loss function that encourages keypoints to distribute well and keep semantic consistency. 
method: 
 In this section, we propose Key-Grid, an unsupervised keypoint detector on 3D point clouds based on the autoencoder architecture. Figure 2 shows the overview of Key-Grid. In the following section, we provide a detailed explanation of the key ingredients in the Key-Grid: an encoder that predicts keypoint locations in an input point cloud; a grid heatmap is a 3D feature map used to capture the geometric structure of deformable objects by computing the shortest distance from points uniformly sampled in 3D cubic space to the "skeleton" generated by keypoint pairs; a decoder that leverages the grid heatmap and the information in each layer of the encoder to reconstruct point clouds.In Key-Grid, each keypoint is regarded as the weighted sum of all the points in the point cloud. Given an input point cloud X ∈ R N ×3 , the goal of the encoder is to produce a weight matrix W ∈ R K×N , such that the matrix multiplication of W and X directly gives the predicted K keypoints K ∈ R K×3 :EQUATIONTo be more specific, the encoder consists of L PointNet++ [22] layers. The i-th PointNet++ layer in the encoder generates a hierarchically down-sampled point cloud X (i) enc ∈ R Ni×3 and its corresponding feature F (i) enc ∈ R Ni×Fi , where i ∈ {1, 2, ..., L}. The last-layer feature passing through a Softmax activation function gives the weight matrix:EQUATIONFigure 2 : Pipeline of Key-Grid. In the encoder section, given a point cloud, we detect the keypoints by utilizing the PointNet++. Then, we utilize the detected keypoints to form the grid heatmap. In the decoder section, we use each layer of the PointNet++ and the grid heatmap to reconstruct the input point cloud. "MLP" stands for multi-layer perceptron, which contains Batch-norm and ReLU.Additionally, following the practice in Skeleton Merger [26] , the encoder outputs an additional head to predict the weights of C K 2 = K(K -1)/2 skeleton segments, i.e., the edges between each pair of keypoints. Formally, (k i , k j ) denotes the skeleton segment connecting the keypoints k i ∈ R 3 and k j ∈ R 3 , and s ij denotes the weight of the skeleton segment (k i , k j ).The proposed grid heatmap is a dense 3D feature map designed to densely represent the 3D shape only through the information from the predicted keypoints. Ideally, for describing the object in an implicit manner similar to the Occupancy Networks [17] , the feature on each grid is desired to reflect the distance from the grid point to the 3D object shape. Since the ground-truth object shape is the reconstruction target in our problem setting, we need to find ways to approximate the object shape using the predicted keypoints. In Key-Grid, we adopt the skeleton [26] approximation, where the object shape is represented by the weighted connected lines of all the keypoint pairs. For each individual grid point, we calculate the distances from this point to all the skeleton segments, and take the maximum of these distances as the feature of this grid. Intuitively, this gives a dense feature field whose value is the smallest at the geometric center of the object, and gradually increases along with the grid point coordinate moving outside the outline of the object shape. In the following, we illustrate the detailed procedures for establishing such a grid heatmap.To begin with, we uniformly sample a 3D array of grid points P ∈ R M ×M ×M ×3 in the normalized cubic 3D space, where M denotes the number of points we sample on each side of the cube. In Key-Grid, we set M = 16, giving 4096 gird points. Next, we calculate the distance between the grid points and all the line segments in the skeleton. When the projection of grid point onto the skeleton line falls in the range of the skeleton segment, the distance is defined as the distance between the grid point and the projection point. Otherwise, the distance is directly defined as the distance between the grid point and the nearest endpoint of the skeleton segment. Formally, the distance d ij (p) between a grid point p ∈ R 3 and a skeleton segment (k i , k j ) connecting the keypoints k i and k j is defined as:EQUATIONwhereEQUATIONThen, the feature of each grid point D(p) is the maximum of the weighted distances from this point to all the skeleton segments:EQUATIONwhere s ij refers to the learnable weight of the skeleton segment (k i , k j ) produced by the encoder, and σ is a fixed hyperparameter. And the grid heatmap H is the 3D array consisting of the features of all the grid points:EQUATIONwhere P xyz ∈ R 3 denotes the extracted grid point coordinate from P indexed by (x, y, z). Conceptually, the grid heatmap H contains the complete geometric information of the keypoints K. As a dense and continuous feature field rather than a set of sparse points, the grid heatmap is expected to force the encoder to predict precise keypoints and bring benefits to the reconstruction process. In the grid heatmap, we measure the value of D(p) using the maximum point-toskeletons distance instead of minimum distance because the maximum distance can better distinguish the spatial locations, especially what is inside or outside an object. For instance, Figure 3 shows that if we choose the minimum distance between the sampled grid points and the keypoints, the grid point p 1 inside the pants will have the same value d min as the grid point p 0 outside the pants. However, if we take the maximum value d max , the grid point p 1 inside the pants will have a smaller value than the grid point p 0 outside the pants.As an inverse process of the encoder, the decoder tries to reconstruct the entire point cloud by gradually augmenting finer geometric details in a hierarchical manner. Unlike previous methods [26; 9] where the keypoint-related 3D structures are used only once as the input of the encoder, we propose to integrate increasingly finer grid heatmap features with the layers of the decoder. In addition, the features in the encoder are directly copied and fused into the corresponding layer in the decoder in a U-Net [23] fashion to further assist the reconstruction process.Formally, the feature of the (L -i + 1)-th layer in the decoder is composed by the following three components:EQUATIONwhere H(X (i) enc ) ∈ R Ni×1 denotes the extracted grid features indexed byEQUATIONenc ∈ R Ni×Fi denotes the corresponding feature of the same number of points copied from the encoder, Proj(F(L-i) dec , X (i-1) enc , X (i)enc ) denotes the feature projected from the former layer of the decoder, and ⊕ denotes element-wise concatenation. Note that H(X (i) enc ) and F (i) enc are already aligned on the first dimension and thus ready for concatenation, the remaining challenge is how to design a feature projection mechanism F targ = Proj(F ori , X ori , X targ ) so that the former-layer features can be aligned with the current-layer features.To solve this problem, we propose to represent the target feature F targ as the weighted sum of the spatially neighboring features in the original feature F ori . More specifically, for every point coordinate x ∈ R 3 in the target point cloud X targ , we find its N neig neighbors in the original point cloud X ori :EQUATION)The target feature of the coordinate x is defined as:EQUATIONwhere ω(x, x ′ ) denotes the inverse of the squared distance between x and x ′ :EQUATION)Our approach adopts an end-to-end method called Key-Grid to identify the keypoints, by leveraging the grid heatmap to reduce the discrepancy between the reconstructed point cloud and the input point cloud. In this section, we will introduce two types of loss function to optimize Key-Grid.Similarity loss. The common way to evaluate the similarity between a reconstructed point cloud X r and a target point cloud X is by employing the Chamfer distance metric [9; 48] . By minimizing the Chamfer distance between them, we can optimize the reconstruction process to closely match the target point cloud, thereby achieving a higher level of similarity between the two point clouds. Thus, we utilize the Chamfer distance between the reconstructed point cloud X r and the target point cloud X to approximate this similarity loss L sim .Farthest point keypoint loss. To ensure that the aligned keypoints distribute well on the surface of objects and represent the geometric structure of objects, we use the farthest point keypoint loss [9] L f ar to allocate the keypoints. Firstly, we randomly select an initial point from the point cloud X.Then, at each iteration, it selects the point that is farthest from all the previously selected points. This process continues until J points have been selected. Finally, we can obtain different sets of sampled pointsQ = {q 1 , q 2 , • • • , q J } ∈ R J×3. These sampled farthest points Q are regarded as a prior estimation of the distribution of keypoints. We define this loss as minimizing the Chamfer Distance between the predicted keypoints K and the sampled points Q.Our overrall loss function isEQUATIONwhere α f ar and α sim are scalar loss coefficients. 
experiments: 
 In this section, we compare the performance of Key-Grid over the existing SOTA approaches on both rigid-body and deformable object datasets. Meanwhile, we conduct the robustness analysis, ablation studies, and show Key-Grid can be easily extended to an SE(3)-equivariant version.Datasets. We use the ShapeNetCoreV2 and the ClothesNet datasets [2; 49] to evaluate the performance of Key-Grid. For the ShapeNetCoreV2 dataset [2] , it contains 51,300 rigid-body objects of 55 different categories. In this paper, we only use categories that are manually annotated with semantic correspondence labels in the KeypointNet dataset [42] . For the ClothesNet dataset [49] , we take three types of deformations on different type of clothing objects: dropping, dragging, and folding. For each deformation of different garment categories, there are 128 samples during the deformation process. We test the performance of Key-Grid on real-world 3D scans of clothing from the Deep Fashion3D V2 dataset [6] and conduct the additional experiments on SUN3D dataset [38] to further illustrate Key-Grid capability in coping with real and large-scale dataset.Baselines. We compare Key-Grid with the current SOTA approaches: KeypointDeformer (KD) [9] , Skeleton Merger (SM) [26] and SC3K [51] that all detect the 3D keypoints in an unsupervised way. Compared to SM [26] , SC3K [51] , and Key-Grid, KD [9] not only requires point clouds as input but relies on object meshes to perform shape transformations from the source shape to the target shape. Therefore, regarding KD [9] as a baseline is actually unfair to our method. However, since it is the current SOTA method for keypoint detection on deformable objects, we choose it as a baseline to compare with our method.Evaluation metrics. For the ShapeNetCoreV2 dataset [2] , we use the Dual Alignment Score (DAS) [26] to assess the degree of keypoints semantic consistency for each category. Meanwhile, to verify the accuracy of detected keypoint locations, we compute the mean Intersection over Union (mIoU) metric [33] with a threshold of 0.1 to evaluate the difference between the detected keypoints and the ground truth provided by the KeypointNet dataset [42] . For the ClothesNet dataset [49] , due to the absence of manually annotated keypoints, we only use DAS to evaluate the semantic consistency of keypoint detection on objects with three type of deformations. Higher values for both metrics correspond to better model performance.For the ShapeNetCoreV2 dataset [2] , we adopt Key-Grid and other baselines to detect ten keypoints and use the DAS and mIoU to evaluate their performances on the 13 categories of rigid-body objects in the Table 1 . For objects with straight-line geometrical structures such as "Airplane", "Vessel", "Knife", and "Guitar", skeletons formed by connecting keypoints easily represent their geometrical structures, which is why SM outperforms Key-Grid for these categories on the mIoU metric. However, Key-Grid's performance on these categories is comparably optimal, achieving higher average scores for both DAS and mIoU compared to other baselines. Additionally, we present various supervised networks and self-supervised methods to compare Key-Grid on the KeypointNet dataset [42] based on mIoU metric in Table 3 . Several standard networks, PointNet [21] , SpiderCNN [39] , and PointConv [36] , are trained to predict keypoint probabilities in a supervised manner. We can observe that Key-Grid demonstrates superior accuracy in keypoint localization compared to other self-supervised methods and outperforms some supervised approaches that utilize PointNet and SpiderCNN as backbones. For the ClothesNet dataset [49] , we aim to make the eight detected keypoints maintaining semantic consistency in objects under the deformation process of the same category. Table 2 shows Key-Grid performs better on the deformations of dropping and dragging than other methods. Key-Grid outperforms other baselines in the drop deformation for all categories, except for "Tie", while also demonstrating superior performance in the drag deformation for all categories, except for "Long Dress" and "Scarf". Additionally, we also show the semantic consistency of keypoints identified by different methods during the folding process of shirts and pants in Table 4 . Table 4 illustrates that for objects with significant deformations, such as folding, Key-Grid significantly outperforms other methods in terms of keypoint semantic consistency. Figure 4 and Figure 5 show the visualization results of keypoint detection under three deformations of clothing using different methods. In Figure 4 , for the dropping and dragging deformation process on the "Hat" and "Long Pant" categories, compared to other methods, keypoints identified by Key-Grid distribute evenly on objects and keep great semantic consistency. The keypoint positions do not change with the deformation of the objects.In Figure 5 (a) and (b), we can observe that when facing folding deformation, keypoints detected by KD and SM have redundancy phenomenon, which means multiple keypoints are identified at the same location. Compared to the SC3K method, keypoints identified by Key-Grid not only capture essential geometric details of deformable objects but also ensure semantic coherence throughout the folding process. Figure 5 (c) shows the grid heatmaps are evidently better at accurately reflecting the geometric structure of the folded clothes than the skeleton structures proposed in SM. Thus, we think that precise representation of the object structure using keypoints forces the network to generate precise keypoints, which makes Key-Grid perform better than previous methods.For the Deep Fashion3D V2 dataset [6] , we select three shirts with different deformations for keypoint recognition. Key-Grid successfully learns eight semantically consistent 3D keypoints for these objects, as shown in Figure 5 (d), even with obvious deformations in the sleeves of the clothes. The hyperparameters used in this experiment are consistent with those employed on the ClothesNet dataset.For the SUN3D dataset [38] , Figure 5 (d) presents the visualization results of 20 keypoints on the different local geometric scenes identified by Key-Grid. Despite the influence of real-distributed noise and occlusion on keypoint detection results, Key-Grid can still recognize keypoints with important geometric information in real-level data, such as corners of buildings and center positions of scenes. In this section, we focus on evaluating the robustness of Key-Grid on the noisy and down-sampled point clouds of folded shirts. To obtain the noisy point clouds, we add Gaussian noise with varying variances to the point clouds. And we utilize the Farthest Point Sampling method to downsample the original point clouds, which has also been used in previous works [19; 47] . factor of 16, the DAS of Key-Grid and SC3K will significantly decrease. However, when Key-Grid is subjected to significant noise interference and high-level downsampling, the DAS of Key-Grid is still better than SC3K, which has not been subjected to any interference. The visualization results also show that keypoints detected by Key-Grid exhibit strong robustness even when subjected to Gaussian noise or downsampling.Additionally, we also present the visualization of keypoints identified from occluded objects, outlierladen objects, and side views of objects under the folding deformation in Figure 6 (e). Key-Grid shows robust performance on both occluded point clouds and those captured from side views. Meanwhile, even when point clouds contain outliers, keypoints detected by Key-Grid maintain strong semantic consistency. Thus, we can conclude that Key-Grid effectively identifies keypoints in comparatively low-quality or partial point clouds, similar to its performance with complete point clouds.It is significant that Key-Grid is capable of identifying keypoints of objects undergoing SE(3) transformations. We analyze the capability of Key-Grid to identify keypoints of objects undergoing SE(3) transformations. However, if using a SPRIN module [43] which is a SOTA SE(3)-invariant backbone to replace PointNet++ [22] directly, the training process of Key-Grid does not converge. Thus, to achieve Key-Grid to accurately identify the keypoints of objects undergoing SE(3) transformations, we regard Key-Grid as the teacher network of USEEK [40] . The student network of USEEK is the SPRIN module. Figure 6(d) shows that the keypoints of folded pants recognized by USEEK are invariant under an assortment of random SE(3) transformations. We also consider other baselines as the teacher model of USEEK. Table 4 demonstrates that Key-Grid outperforms other baselines in terms of identifying the SE(3)-Equivariant keypoints with semantic consistency.Strategy of decoder. In the decoder section, Key-Grid combines the information from each layer of the encoder with the grid heatmap to reconstruct the original point cloud. In order to illustrate the importance of encoder information and grid heatmap for the reconstruction process, Key-Grid respectively uses one of these two strategies in the decoder section. Table 5 shows that when using both input streams to reconstruct the point cloud, the keypoints detected by Key-Grid exhibit better semantic consistency.Loss ablation. To emphasize the importance of each loss, we conduct the evaluation of the proposed approach by systematically excluding each loss individually. The results are illustrated in Table 5 .We can observe that the DAS of Key-Grid decreases when any of the loss functions is excluded from the training process. The similarity loss contributes comparatively low, but regardless of whether Key-Grid is applied on the ClothesNet dataset or the ShapeNetCoreV2 dataset, it still can improve the semantic consistency of the keypoints identified by Key-Grid. The contribution of the farthest point keypoint loss is comparatively higher than the similarity loss. Without the farthest point keypoint loss, the DAS of Key-Grid will decrease significantly.  
conclusion: 
 In this paper, we propose Key-Grid that detects keypoints for both rigid-body and deformable objects.It uses the detected keypoints to build grid heatmap and incorporate it into the reconstruction process of point clouds. We evaluate the quality of keypoints on multiple datasets and analyze the robustness of Key-Grid. Meanwhile, we embed the Key-Grid into the USEEK framework to produce SE(3)equivariant keypoints. Extensive experiments shows that Key-Grid can detect the keypoints with great semantic consistency and precise location. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer:[Yes] Justification: Our paper discusses societal impacts in the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper does not have such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: Our paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We use the PyTorch framework [7] to train our method. The entire network is optimized on the Adam optimizer [10] with a learning rate of 0.1. For all the experiments, the batch size is set to 8. Our method is trained for 100 epochs, and for the first 20 epochs of training, we do not use a similarity loss function which measures the similarity between the reconstructed point clouds and the original point clouds.For all the experiments, the input point cloud contains N = 2048 points. The number of encoder layers L in the PointNet++ network [22] is 4. For the ShapeNetCoreV2 dataset [2] , we set the number of keypoints K to be 10. Thus, when calculating the farthest point keypoint loss function, we choose the number of the farthest points J as 14. For the ClothesNet [49] and Deep Fash 3D dataset [6] , we set K and J as 8 and 12, respectively. The hyperparameter K and J are 20 and 24 in the SUN3D dataset [38] . In the process of building the grid heatmap, the number of points M uniformly selected on each edge of the cubic space is 16. For the decoder, the hyperparameter N neig for the number of neighboring features applied in aligning the feature from the former layer to the current layer is set to 3. In the loss function, α f ar and α sim are both set to 1.Table 6 presents the time and memory consumption of Key-Grid and SM when inferring a batch comprising 32 samples on a single 1080 Ti GPU. According to Table 6 , Key-Grid achieves equal efficiency to the baseline method (SM) in terms of inference speed but consumes an acceptably larger memory size. In this section, we primarily present additional visualization results about the experimental part to illustrate the effectiveness of Key-Grid.We vary the number of unsupervised keypoints discovered by Key-Grid on the folded pants. Figure 7 shows that these different numbers of keypoints on the folded pants maintain good semantic consistency during the deformation process. And as the number of keypoints increases, we find that these keypoints do not overlap with each other, they still evenly distribute on the surface of the objects, and each keypoint represents the geometric information of the objects.Ablation study. We show the visualization results of the ablation study in Figure 8 . From the Figure 8 , we find that in the decoder section, if we only use the encoder information or the grid heatmap information to reconstruct the original point cloud, the positions of the keypoints on the folded pants are not stable. When we do not use the similarity loss function in the decoder section, the keypoints will be placed in the wrong prior positions and the network predicts the redundant keypoints. Moreover, when we do not apply the farthest point keypoint loss function to train the network, the performance of keypoint recognition will be poor, as the keypoints will cluster around the center of the object.Gaussian noise to point cloud. Figure 9 shows the visualization results of Key-Grid for different noisy point clouds, which add the Gaussian noise of different scales to the input point cloud. We can observe that even when Gaussian noise is added to the original point cloud with a magnitude of 0.08, the positions of the keypoints detected by Key-Grid in these noisy point clouds are still close to those found in the original point cloud. Thus, we can conclude that Key-Grid exhibits robustness when dealing with the noisy point clouds.Downsampling point cloud. This paragraph presents the performance of Key-Grid on the downsampled point clouds shown in Figure 10 . For decimating the original point cloud, we use the Farthest Point Sampling method for downsampling. We discover that Key-Grid successfully estimate the keypoints on the downsampled point clouds. When downsampling the original point cloud by a factor of 16, resulting in a point cloud with 128 sample points, except for the blue keypoints which will be shifted downwards, the positions of the other keypoints will remain close to their positions detected in the original point cloud.Deep Fash3D V2 dataset. We show the additional visualization results of keypoints detected by Key-Grid on the Deep Fash3D V2 dataset [6] . Figure 13 illustrates that when faced with a wider variety of clothing cuff deformations, the keypoints detected by Key-Grid achieves great semantic consistency on the world scanned objects and represents the important information about these objects. Thus, we think that Key-Grid performs well in the world scanned objects.ShapeNetCoreV2 and ClothesNet dataset. For ShapeNetCoreV2 [2] and ClothesNet [49] dataset, we present the visualization results of keypoint recognition in a wider range of object categories using SM and Key-Grid. In Figure 12 , we separately present the visualization results of keypoint recognition on the ShapeNetCoreV2 dataset using SM and Key-Grid. Compared to the redundancy phenomenon that many keypoints locate in the same region generated by SM [26] , the keypoints detected by Key-Grid distribute separately on the surface of the object, and these keypoints also can represent the important geometric information of the object. For the ClothesNet dataset, Figure 13 demonstrates the keypoint detection of SM [26] and Key-Grid in the deformation process of a wider range of object categories, including drop and drag deformations. In the deformation process, compared to SM [26] ,8: Visualization results of the ablation study on the folded pants. the keypoints identified by our method keep good semantic consistency and distribute evenly on the surface of the object. Thus, for the rigid-body and deformable objects, Key-Grid exhibits great performance on keypoint detection.In the supplementary materials, we also provide keypoint recognition results for the folding clothes in the ClothesNet dataset using different methods. The video we provide include the entire process of deformation for shirts and pants. From this video, we can conclude that Key-Grid identifies the keypoints that maintain good semantic consistency during the deformation process. These keypoints also carry the geometric feature information of the deformable objects. Following previous works, we need to manually set the total number of predicted keypoints beforehand. In future research, we will propose an adaptive keypoint detection method that can generate varying numbers of keypoints with accurate positions for different samples. Key-Grid predicts the keypoints on the publicly available datasets. Thus, we think Key-Grid has very limited potential negative societal impacts.Figure 13 : Keypoint on the ClothesNet dataset. For the drop and drag deformation processes, we select the "Long Dress", "Tie", and "Vest" categories of objects to visualize the keypoints identified by SM and Key-Grid. We use lines to connect the keypoints at the same position during the deformation process.• The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: In the appendix, we discuss the memory usage and inference time of our method.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: Our research adheres to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper proposes a novel approach for 3d keypoint estimation via unsupervised learning. It initially received discordant ratings and comments from the reviewers (BR, BR, BA, SA). During the rebuttal and discussion stage, some main concerns from the reviewers have been resolved and a stronger consensus towards acceptance emerged. 3 reviewers suggested to accept (6,6,7) and one reviewer kept the initial BR score but mentioned in their comments that they are satisfied with the rebuttal and open to raise their score. Overall, the AC suggests to accept this paper as their final recommendation.

Reviews: review: paper_summary: paper_summary: Novel PointNet-based autoencoder method  called KeyGrid, that predicts semantic keypoints on objects, even when objects are subject to deformations. Similar to previous approaches, keypoints are produced as a linear combination of inputs points, according to a learned weight matrix.

The key novelty over related works using PointNet++-based auto-encoders such as SkeletonMerger (SM) [26] is to pass a densely sampled grid of features (3d “point heatmap”) at each decoder layer (which are hierarchical). In this heatmap grid, each cell holds the maximum of the weighted distances to the 'keypoint skeleton segments'. 
Also, KeyGrid appears to pass richer encoder information to the decoder than the SM work, although the differences here are less clear. 

The method is shown to outperform previous baselines such as SK3D and SM on ShapeNetV2 dataset (more right objects) and on ClothesNet (deformable objects).
strength_weakness: strength_weakness: # Significance
- Is the hierarchical decoder setup (excluding the heatmap grid itself) novel, esp compared to SkeletonMerger [26]? Right now there are no specific claims or ablations related to this. 
- The primary novelty relative to SM [26] seems to be the use of keypoint grid heatmaps in the decoder, storing max distances from grid cells to the weighted keypoint segments. However, this is just one possible shape descriptor feature, and intuitively it should be quite a weak and ad hoc feature, even if the authors gave some intuition why it may be better than the min function. Would other shape feature with yet more information be more useful could (e.g. why not have K channels with distances to all segments or to the keypoints as opposed to just taking the max, etc)? In general it would help if the paper provided more intuition or experimental results giving more insight on the information contained in this heatmap. 
- This class of approaches seems generally limited to cases where entire point clouds were visible. How would you handle cases when some parts of the objects were occluded, such as in real-world scenes? This has not beed discussed. 

# Related work 
Could discuss other dense representations for correspondence like the pointmaps used in “DUSt3R: Geometric 3D Vision Made Easy” https://arxiv.org/pdf/2312.14132 )

# Clarity
- I found Sec 3.3 particularly difficult to follow and understand. A rewrite, or further details in the appendix clarifying better how these quantities are 'composed' and aligned would help. 
- Fig 1 illustrates points heatmap in the decoder by simply showing a point cloud. I was expecting pictures like in Fig 4c. 
- L13, L58 Unclear where keypoint pairs come from, when discussed for first time. 
- No discussion of how correspondences are established, e.g. as shown in Figure 3. 
- The loss terms in Equation 11 are not mathematically defined, would be helpful in the Appendix. 

# Experimental results
  - KeypointDeformer, SkeletonMerger, and SC3K are tested on KeypointNet dataset (https://arxiv.org/pdf/2002.12687 ). Why do the authors omit it here? Other works such as SC3K state “We use KeypointNet dataset [36 ] in our experiments, considering that this is the standard and most recent dataset used for keypoints estimation”. If the results are not as good on that dataset, that is okay, as long as we provide them and state that our method is specifically designed for datasets with deformable/soft objects. This would provide confidence that the authors ran their eval of official source code correctly on the two new datasets. 
  - Why no comparisons with supervised baselines?
  - Figure 4D: Sun3D keypoints are so small in the figure it appears as if they are randomly distributed

# Some Grammar & Style Nits:
- L453 SC3K bibtex should be ICCV 2023, not Arxiv “
- L148 typo “giving a total of 4096 gird points.” -> “…4096 grid points”
- L229 “mean Intersection over Unions” -> “mean Intersection over Union”
- Grammar: Figure 3: caption: “Different methods on the Hat and Long Pant during” ->  prefer “Different methods on the Hat and Long Pant categories during”
- Table 3 caption is too close to Table 3, whitespace seems too shrunken
- L75 “Deformable object dataset” -> “Deformable object datasets”
- Table 1 caption “codes” -> should be singular, not plural (“the results we reproduced based on their official codes” -> “the results we reproduced based on their official code.”)
- Color scheme is jarring in Tables 1-4, prefer tango colors that have a more attractive colormap (https://sobac.com/sobac/tangocolors.htm)
- L3 extraneous article: “focus on the rigid body objects” -> “focus on rigid body objects”
- L6 extraneous article: “for both the rigid-body and deformable objects” -> “for both rigid body and deformable objects”
- L300 “the importance of encoder information and grid heatmap for reconstruction process,” -> “...for the reconstruction process,”
- L302: “Table 4 show that the keypoints detected by Key-Grid which utilizes both two strategies to reconstruct the point cloud have better semantic consistency.” -> “Table 4 shows that when using both input streams to reconstruct the point cloud, the keypoints detected by Key-Grid exhibit better semantic consistency”
- L9-10 Instead of “Unlike previous work, we leverage the identified keypoint information to form a 3D grid feature heatmap 1called grid heatmap, which is used in the decoder section”, prefer to say something like “..to form a 3D grid feature heatmap which we refer to as a grid heatmap. A grid heatmap is…” 
- L14 “Into the decoder section” -> “into the decoder model”? Ambiguous what “section” refers to
- L51 “aiming at the semantic consistency” -> “aiming for semantic consistency”, e.g. “an unsupervised keypoint detector on 3D point clouds aiming for semantic consistency under shape variations of both rigid-body and deformable objects.”
- L74 section should be named “Related Work” not “Related Works”
questions: questions: - Compared to SM[26], is the keypoint heatmap the main difference, how about the use of hierarchical decoder. 
- Wouldn't a richer shape descriptor heatmap do even better than a max function? (e.g. K channels, distances to the keypoints, etc). 
- How do you get keypoint correspondences across shapes for your model, do they emerge naturally? 
- What are the method results on KeypointNet dataset? Can you provide some additional comparisons to supervised baselines on your datasets, if there are such? 
- Can this method be applied on datasets where shapes are significantly occluded (e.g. viewed from a specific direction)?
limitations: limitations: Limitations have been sufficiently addressed.
paper_summary: paper_summary: The authors propose a novel unsupervised method to detect 3D point clouds key points by producing an intermediary heatmap based on a grid and points distances from the skeleton and connected key points. They achieve state-of-the-art accuracy and semantic consistency and easily achieve Se-(3) invariance with minimal adaptation of the method.
strength_weakness: strength_weakness: A small neat-picking addition would be to add one or two visual examples where the method does not work super well, to contrast with the excellent results presented in the paper
questions: questions: Perhaps I missed this point, but how does the author ensure that the key points are semantically the same across different folding/deformation of the objects? ( and give them the same colour)
limitations: limitations: Perhaps the only limitation I can pick up is the consistency across more than two deformations of an object. Would the method perform well over long-term deformations? It ties with showing one or two failure cases of the method.
paper_summary: paper_summary: This paper presents Key-Grid, an unsupervised keypoint detection network designed for 3D point clouds. Unlike previous methods that emphasize leveraging various priors on 3D structures, this paper converts keypoints into a grid heatmap. This heatmap forms a continuous feature landscape across the entire 3D space, providing richer and more stable geometric descriptions of objects, particularly for deformable objects. Meanwhile, this paper achieves state-of-the-art (SOTA) results on the ShapeNetCoreV2 and ClothesNet datasets.
strength_weakness: strength_weakness: Q1. My main concern is whether it can be empirically or theoretically justified why the grid heatmap can represent a dense extension of the skeleton, and why the current method of generating the grid heatmap is reasonable. Could using the density of the 'skeleton' or the density of the point cloud directly represent the grid heatmap instead?

Q2.  In the ablation study (Table 4), the accuracy drop after "No Grid Heatmap" is minimal, yet the Grid Heatmap is the most significant contribution of this paper. Could the authors explain this phenomenon and provide ablation studies across more categories? 

Q3. Can this paper adaptively select the number of detected keypoints for different categories of objects?

Q4. The quality of figures 1 and 2 could be enhanced. For example, the information content of figure 1 is too minimal to serve as a teaser.
questions: questions: If the authors address the issues mentioned in the weaknesses, I am willing to increase the rating.
limitations: limitations: Yes, the authors discussed the limitations and potential negative societal impact.
paper_summary: paper_summary: This paper presents a novel unsupervised network for keypoint detection. The method can be applied to handle both rigid and deformable objects. It follows an autoencoder framework where the encoder predicts keypoints and the decoder utilizes the generated keypoints to reconstruct the objects. The main contribution lies in the grid based representation which is suitable to capture the shifted geometric structure of the keypoints brought by deformations.
strength_weakness: strength_weakness: 1 The visualized results presented in this paper all feature uniform and complete point clouds. Can this method handle partial point clouds, such as those obtained from back-projecting depth maps? Can it deal with point clouds that contain severe outliers, not just those with some Gaussian noise? This is very important for practical use.

2 In 289, However, if using a SPRIN module [41] which is a SOTA SE(3)-invariant backbone 289 to replace PointNet++ [22] directly, the training process of Key-Grid does not converge. Why? How about using other backbones? like Vector Neurons?

Deng, C., Litany, O., Duan, Y., Poulenard, A., Tagliasacchi, A., & Guibas, L. J. (2021). Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 12200-12209).

3 This proposed representation is based on Euclidean distance between the grid point towards each line segment, right? Why not first establish a graph and then aggregate features along the graph? I saw many papers using graph to model deformable objects.
questions: questions: I have listed some questions in the weakness. The first question is the most important.

In L 14, 'Meanwhile, we incorporate the information from each layer of the encoder into the decoder section', Is this your contribution? 

I recommend to move Fig 6 in the appendix to the main text for clarity.
limitations: limitations: The author has discussed potential limitations in the paper

Comments: title: Final Rating
rebuttal response
Final Rating
Thanks for your respone
Additional comments
comment: Thank you for your feedback on our paper, particularly regarding the applicability of the Key-Grid to partial and noisy point clouds. We will include this section in the next version of the paper. We greatly appreciate your consideration of the paper as ready for "weak accept".  We notice that the rating is not updated. We kindly hope that you update the rating of our paper to reflect the improvements made to the paper.
Thank you for your detailed review and comments on our paper. We will revise our paper according to your  suggestions. We are also pleased to hear that you believe our paper has reached the ’weak accept‘ stage.  However, we observe that the rating is not updated. We would be grateful if you could revise the rating to better reflect the improvements made to the paper.
Thank you for the response. The rebuttal is very good and has addressed my major concerns, and I raise my rating to "weak accept."
The authors have provided helpful answers and results that largely address my questions. I also saw the newly attached figure, which shows performance under some occlusion - which is ok even if the occlusion amount shown is quite small. 

In light of this, I am willing to raise my rating to 'weakly accept'.
Thank you for the response. The rebuttal has addressed my main concerns, and I am willing to raise my rating to "weakly accept."
In my view, using the distance from a point to keypoints to represent the deformable object’s structure is less effective than using the distance from a point to the skeleton. For example, in the main paper of Figure 2, if we use the distance from a point to keypoints, in the tail section of the pants (i.e., the area between the red and blue points), the value near the red point differs from the value at the center of this section. However, the geometric information represented by these two points should be consistent. 
By using the distance from a point to the skeleton, the point values are derived based on the skeleton spreading outward from the center. This results in different values for points near the skeletons compared to those at the object’s edges, reflecting their distinct geometric information. Therefore, we believe that using the distance from a point to the skeleton more accurately characterizes the structure of deformable objects than using the distance to keypoints.
Thank you for the response and the additional experiments. My main concerns regarding the grid heatmap representation and ablation study have been resolved. The quality of the Figures 1 and 2 has also improved. I tend to accept this paper. However, I still have another question: can the distances between key points or the density of key points express the dense extension of the deformable object?