SNAKE: Shape-aware Neural 3D Keypoint Field
Authors: Chengliang Zhong
Peixing You
Xiaoxue Chen
Hao Zhao
Fuchun Sun
Guyue Zhou
Xiaodong Mu
Chuang Gan
Wenbing Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D Keypoint Detection
Implicit Representation

Abstract: Detecting 3D keypoints from point clouds is important for shape reconstruction, while this work investigates the dual question: can shape reconstruction benefit 3D keypoint detection? Existing methods either seek salient features according to statistics of different orders or learn to predict keypoints that are invariant to transformation. Nevertheless, the idea of incorporating shape reconstruction into 3D keypoint detection is under-explored. We argue that this is restricted by former problem formulations. To this end, a novel unsupervised paradigm named SNAKE is proposed, which is short for shape-aware neural 3D keypoint field. Similar to recent coordinate-based radiance or distance field, our network takes 3D coordinates as inputs and predicts implicit shape indicators and keypoint saliency simultaneously, thus naturally entangling 3D keypoint detection and shape reconstruction. We achieve superior performance on various public benchmarks, including standalone object datasets ModelNet40, KeypointNet, SMPL meshes and scene-level datasets 3DMatch and Redwood. Intrinsic shape awareness brings several advantages as follows. (1) SNAKE generates 3D keypoints consistent with human semantic annotation, even without such supervision. (2) SNAKE outperforms counterparts in terms of repeatability, especially when the input point clouds are down-sampled. (3) the generated keypoints allow accurate geometric registration, notably in a zero-shot setting. Codes and models are available at https://github.com/zhongcl-thu/SNAKE.

SNAKE: Shape-aware Neural 3D Keypoint Field
Chengliang Zhong!:4 1,2,3
Peixing You'
Xiaoxue Chen'
Hao Zhao
Fuchun Sun
Guyue Zhou²
Xiaodong Mu' Chuang Gan
Wenbing Huang
Xi' an Research Institute of High-Tech THHAA, Tsinghua University
3 AARR, Tsinghua University Peking University "Intel Labs MIT
T "Gaoling School of Artificial Intelligence, Renmin University of China
Beijinng Key Laboratory of Big Data Management and Analysis Methods
zhongel 190mails. tsinghua edu. en; hao zhao@intel.com www.iinnhuuuuuu....o.mmm edu.
fcsun@tsinghua.com
Abstract
Detecting 3D keypoints from point clouds is important for shape reconstruction, while this work investigates the dual question: can shape reconstruction benefit 3D keypoint detection? Existing methods either seek salient features according to statistics of different orders or learn to predict keypoints that are invariant to transformation. Nevertheless, the idea of incorporating shape reconstruction into 3D keypoint detection is under-explored. We argue that this is restricted by former problem formulations. To this end, novel unsupervised paradigm named SNAKE is proposed, which is short for shape-aware neural 3D keypoint field. Similar to re- cent coordinate-based radiance distance field, our network takes 3D coordinates as inputs and predicts implicit shape indicators and keypoint saliency simultane- ously, thus naturally entangling 3D keypoint detection and shape reconstruction. We achieve superior performance on various public benchmarks, including standalone object datasets ModelNet40, KeypointNet. SMPL meshes and scene-level datasets 3DMatch and Redwood. Intrinsic shape awareness brings several ad advantages as follows. (1) SNAKE generates 3D keypoints consistent with human semantic annotation, even without such supervision. (2) SNAKE outperforms counterparts in terms of repeatability, especially when the input point clouds are down- sample. (3) the generated keypoints allow accurate geometric registration, notably in a zero-shot setting. Codes are available at http:://gtttbb.oomm//ooggttutttf
1
Introduction
2D sparse keypoints play a vital role in reconstruction 32 recognition [22] and pose estimation with scale invariant feature transform (SIFT) 19 being arguably the most important pre- Deep Learning (DL) computer vision algorithm. Altough dense alignment using photometric or featuremetric losses is also successful in various domains 2 36 8) sparse keypoints are usually preferred due to compactness in storage/computatioon and robustness to illumination/rotation. Just like their 2D counterparts, 3D keypoints have also drawn of attention from the community both
43
Dre [3 35 and DL [5 1 38 literature, with various applications in reconstruction
45 41
and recognition 34
However, detecting 3D keypoints from raw point cloud data is very challenging due to sampling sparsity. No matter how we obtain raw point clouds (e.g., through RGB-D cameras [4)], stereo 4. or LIDAR (TO), they are only a discrete representation of the underlying 3D shape. This fact drives to explore the question of whether jointly reconstructing underlying 3D shapes helps 3D
"Correspondngg author: Fuchun Sun.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Salieney Prob. Prob)(n)
Occupancy Prob.
Profu P)
Keypoint Coord. (1.
Saliency Prob. (P)
Coord, Recon.
Saliency Decoder
Occupancy Decoder
Keypoint Network
Saliency Network
Revonsiructiom Network
Peature Volume
Encoder
Encoder
Encoder
Poshine
Chanfer Diszance
Input
Input
Input
Inpur Query ((t.3)
64 USIP
UKPGAN
Id SNAKE (Durs)
Figure I: A comparison between existing 3D keypoint detection formulations and our newly proposed one. (a) USIP-like methods directly predict keypoint coordinates from input point clouds P. (b) UKPGAN-like methods predict saliency scores for P. Using chamfer distance, it reconstructs P coordinates based the saliency scores and latent features. (c) Our SNAKE formulation predicts saliency probabilities and shape indicators simultaneously for each continuous query point q instead of discrete point clouds P. Sub-networks used for keypoint detection and reconstruction are shown in yellow and red, although they have different formulations. Here, the occupied points are those on the input surface.
keypoint detection. To our knowledge, former methods have seldom visited this idea. Traditional 3D keypoint detection methods are built upon some forms of first-order (e.g., density in intrinsic shape signature [44) second-order (e.g., curvature in mesh saliency [14] statistics, including
sophisticated reformulation like heat diffusion [33. Modern learning-base based methods rely upon the idea of consistency under geometric transformations, which can be imposed on either coordinate like
USIP [15] or saliency value like D3Feat M1. The most related method that studies joint reconstruction and 3D keypoint detection is recent one named UKPGAN (38), yet it reconstructs input point cloud coordinates using an auxiliary decoder instead of the underlying shape manifold.
Why is this promising idea under-explored in the literature? We argue the reason is that former problem formulations are not naturally applicable for reconstructing the underlying shape surface. Existing paradigms are conceptually illustrated in Fig- [ USIP-like methods directly output keypoint coordinates while UKPGAN- methods generate saliency values for input point clouds. In both cases, the representations are based upon discrete point clouds. By contrast, problem using coordinate-basse networks, as inspired by the recent success of neural radiance fields 21 17 29 and neural distance fields 23 31. As shown in Fig. De, our model predicts keypoint saliency value for each continuous input query point coordinate ())
reformulate the
A direct advantage of this new paradigm is the possibility of tightly entangling shape reconstruction and 3D keypoint detection. As shown in Fig. [tc besides the keypoint saliency decoder, we attach parallel shape indicator decoder that predicts whether the query point q is occupied. The input to decoders is feature embedding generated by trilinearly sampling representations conditioned on input point clouds P. Imagine feature embedding at the wing tip of an airplane, if it be used to reconstruct the sharp curvature of the wing tip, it can be naturally detected as a keypoint with high repeatability. As such, our method is named as shape-awar neural 3D keypoint field, or SNAKE.
Shape awareness, as the core feature of our new formulation, brings several advantages. (1) High repeatability. Repeatability is the most important metric for keypoint detection, i.e., an algorithm should detect the same keypoint locations in two-view point clouds. If the feature embedding can successfully reconstruct the same chair junction from two-view point clouds, they expected to generate similar saliency scores. (2) Robustness to down-sampling. When input point clouds are sparse, UKPGAN- frameworks can only achieve reconstruction up to the density of inputs. In contrast, our SNAKE formulation can naturally reconstruct the underlying surface up to any resolution because exploits coordinate-bant based networks. (3) Semantic consistency. SNAKE reconstructs the shape across instances of the same category, thus naturally encouraging semantic consistency although no semantic annotation is used. For example, intermediate representations need to be similar for successfully reconstructing different human bodies because human shapes are intrinsically similar.
To summarize, this study has the following two contributions:

---

We propose new network for joint surface reconstruction and 3D keypoint detection based upon implicit neural representations. During training, we develop several self-supervised losses that exploit the mutual relationship between two decoders. During testing, we design a gradient-based optimization strategy for maximizing the saliency of keypoints.
Via extensive quantitative and qualitative evaluations on standalone object datasets Model- Net40, KeypointNet, SMPL meshes, and scene-level datasets 3DMatch and Redwood, we demonstrate that shape-aware formulation achieves state-of-the- performance under three settings: (1) semantic consistency; (2) repeatability; (3) geometric registration.
2 Related Work
3D Keypoint Detector As discussed in the introduction, 3D keypoint detection methods can be mainly categorized into hand-crafted and learning-baseke Popular hand-crafted approaches +4 30 28 employ local geometric statistics to generate keypoints, These methods usually fail to detect consistent keypoints due to the lack of global context, especially under real-world disturbances, such as density variations and noise. USIP [5] is a pioncering learning- baanddg---addd---e 3D keypoint detector that outperforms traditional methods by a large margin. However, the detected keypoints are not semantically salient, and the number of keypoints is fixed. Fernandez et al. [9] exploit the symmetry prior to generate semantically consistent keypoints. But this method is category speeiffc,,, limiting the generalization to unseen categories and scenes. Recently, UKPGAN 38 makes use of reconstruction to find semantics- aware 3D keypoints. Yet, it recovers explicit coordinates instead of implicit shape indicators. As shown in Fig. [ different from these explicit keypoint detection methods, we propose new detection framework using implicit neural fields, which naturally incorporates shape reconstruction.
Implicit Neural Representation Our method exploits implicit neural representations to parameterize continuous 3D keypoint field, which is inspired by recent studies of neural radiance fields [17 21 29]
and neural distance fields (23 31 16 42). Unlike explicit 3D representations such as point clouds, voxels, or meshes, implicit neural functions can decode shapes continuously and learn complex shape topologies. To obtain fine geometry. ConvONet [24) proposes to use volumetric embeddings to get local instead of global features (20) of the input. Recently, similar local geometry preserving networks show great success for the grasp pose generation [12] and articulated model estimation [11. They utilize the synergies between their main tasks and 3D reconstruction using shared local representations and implicit functions. Unlike [T] [22 that learn geometry as an auxiliary task, novel losses tightly couple surface occupancy and keypoint saliency estimates.
3 Method
This section presents SNAKE, a shape-aware implicit network for 3D keypoint detection. SNAKE conditions implicit decoders (for shape and keypoint saliency) on shared volumetric feature embeddings, which is shown in Fig. [rrrmmwookk. To encourage repeatable, uniformly scattered, and sparse keypoints, we employ several self-supervised loss functions which entangle the predicted surface occupancy and keypoint saliency, as depicted in the middle panel of Fig. [2] During inference, query points with high saliency are further refined by gradient-based optimization since the implicit keypoint field is continuous and differentiable, which is displayed in
Fig. 2 inferencee
3.1 Network Architecture
Point Cloud Encoder As fine geometry is essential to local keypoint detection, we adopt the ConvONets (24), which can obtain local details and scale to large scenes, as the point cloud encoder denoted fe... for SNAKE. Given input point cloud P E Xx3
encoder firstly processes it C with the PointNet++ 25 (or alternatives like (46) ) to get feature embedding E IR N NCC, where N and Cj are respectively the number of points and the dimension of the features. Then, these features are projected and aggregated into structured volume Z' E R xX D where H, W and D are the number of voxels in three orthogonal axes. The volumetric embeddings serve as input to the 3D UNet [6] to further integrate local and global information, resulting in the output G E RC2xx XxxxXxxxxx where C2 is the output feature dimension. More details can be found in the Appendix.
Shape Implicit Decoder As shown in the top panel of Fig. each point g E R3 from a query set Q is encoded into Ce dimeensional vector ge via luyti perceptron that is denoted the positional
3

---

Prob,(Q1P)
Recen.
Input
Lacal Go (6,)
Point Clond Enceder
Orrugancy Decoder
Marching
Cabes
Triltnzar Samppingg
Lass
Sprnate
Positional Encoding
Kespaint
Queey
Encoded Cound.
Framework
4, [4.)
(QPP)
Keypoint Detertion
w D)
Similarity
(TQIT) Te SROT
(riP)(QPP))
Prok (4 (P)
Prod, (P)
(A) Repeatability
(b) Surface Constraint
(c) Sparsity
Loss Functions for Keypoint Field
Suurface Occupancy Field
SNAKE Network
Object-scale Keypoints
Gradicats
Oijs PriP)
Query
Saliency
80
Sampling
Keypuint Optimization
Inference
Result
Scene-scale Keypoints
Figure 2: Framework: We use an implicit network decode the surface occupancy and keypoint saliency probability simultaneously. Green arrows indicate the mutual relationships between the geometry and saliency field. Through marching cubes and non-man maximum suppression (NMS). it could respectively recover the shape and detect keypoints from the input. Loss functions for keypoint filed: Three loss functions try to make the generated keypoint repeatable, located on the underlying surface, and sparse. Inference: We design gradient-based -based optimization method to extract keypoints from the saliency field. Result: The object-scale and scene-scale keypoints after inference are displayed.
encoder for... i.e. qe = fe (g). Then, the local feature G, is retrieved from the feature volume G according to the coordinate of q via trilinear interpolation. The generated ge and G3 are concatenated and mapped to the surface occupancy probability Probo(qlP) E [0. 1] by the occupancy decoder fe.. as given in Eq. ( q is on the input surface, the Prob, (ql P) would be 1, otherwise be 0. In our formulation, the points inside the surface are also considered unoccupied.
fe. (ge, Gg) + Pr)b)(g)P)
(I)
Keypoint Implicit Decoder Most of the process here is the same as in shape implicit decoder, except for the last mapping function. The goal of keypoint implicit decoder fe, is estimate the saliency of the query point g conditioned on input points P, which is denoted as Pr) E [0, 1] and formulated by:
Sa, (ge; Gg) Prob. P).
Here, saliency of the query point y is the likelihood that it is keypoint.
3.2 Implicit Field Training
The implicit field jointly optimized for surface occupancy and saliency estimation by several self- supervised losses. In contrast to former arts [I] with similar architecture that learn multiple tasks separately, we leverage the geometry knowledge from shape field to enhance the performance of keypoint field, as shown in the green arrows of Fig. Specifically, the total loss is given by:
L, + L, + L... + L.:
where L. encourages the model to learn the shape from the sparse input, Ly, L. and L., respectively help the predicted keypoint to be repeatable, located on the underlying surface and sparse

---

Surface Occupancy Loss The binary cross-entropy loss (вCE between the predicted surface occu- pancy (rob,(g) P) and the ground srtnh label Probg is used for shape recovery. The queries Q are randomly sampled from the whole volume size H X W x D. The average over all queries is as follows:
1 Σ [Q1
C.
=
|BCE( (Prob(rrbc(g))) Prob. (q) P)),
(4)
REQ
where [Q] is the number of queries Q.
Repeatability Loss Detecting keypoints with high repeatability is essential for downstream tasks like
registration between two-view point clouds. That indicates the positions of keypoint are covariant to
the rigid transformation of the input. To achieve a similar goal, 21) keypoint detection methods (27] 7 433 enforce the similarity of corresponding local salient patches from multiple views. Inspired by them, we enforce the similarity of local overlapped saliency fields from two-view point clouds. Since the implicit field is continuous, we uniformly sample some values from local field to represent the local saliency distribution. Specifically, as shown in the top and the middle part of Fig. 2 we build several local 3D Cartesian grids {Q: [2 with resolution of H1 x Wi X Dr and size of 1/U. We empirically set the resolution of Q₂ to be almost the same as the feature volume G. As non-occupied regions are uninformative, the center of Qi is randomly sampled from the input. Then, we perform random rigid transformation 7 on the P and Qi to generate P and TQ;- Similar to [27), the cosine similarity, denoted as cosim, is exploited for the corresponding saliency grids of Qi and TQ;
Σ cosim ((rr,,(..))))))) Crrrob..TQQIT (TQIP)). SEN
C, =
(5)
Surface Constraint Loss As discussed in [5]. 3D keypoints are encouraged to close to the input. They propose loss to constrain the distance between the keypoint and its nearest neighbor from the input. Yet, the generated keypoints are inconsistent when given the same input but with a different density. Thanks to the shape decoder, SNAKE can reconstruct the underlying surface of the input, which is robust to the resolution change. Hence, we use the surface occupancy probability to represent the inverse distance between the query and the input. As can be seen in Fig. 2g (surface constraint), we enforce the saliency of the query that is far from input P close to O, which is defined as
Cm
Σ
(1 (robblllp))))) (rlb. qEQ
(6)
Q1
Sparsity Loss Similar to 2D keypoint detection methods [27.. we design a sparsity loss to avoid the trivial solution (Prob,[Q)P)=)
in (5 55 6). As can be seen in Fig. 2 the goal is to maximize the local peakiness of the local saliency grids. As the sailency values of non-occupied points are enforced to 0 by Lm. we only impose the sparsity loss on the points with high surface occupancy probability. Hence, we derive the sparsity loss with the help of decoded geometry by
c. --1--X Σ maxx Prob, (P) ( mean Prob, (P.
T. VEN
P) where Q7 = {ala E Qi, (P) (hr J. Urr E (0, 0.5] is a constant, and n is the number of grids. It is noted that the spatial frequency of local peakiness is dependent on the grid size 1/U, see section 4.4 Since the network is not only required to find sparse keypoints, but also expected to recover the object shape. would generate high saliency at the critical parts of the input, like joint points of a desk and corners of house, as shown in the Fig. resuut..
3.3 Explicit Keypoint Extraction
The query point g whose saliency is above a predefined threshold thr E (0, 1) would be selected as a keypoint at the inference stage. Although SNAKE can obtain the saliency of any query point, a higher resolution query set results in high computational cost. Hence, shown in Fig. [2 inference, we build a relatively low-resolution query sets Qinter which are evenly distributed in the input space and further refine the coordinates of Qinter by gradient-based optimization this energy function:
E E(Quufer: P)
Σ
1 1- 1 (g)
(
[Qinerr] qE
Specifically, details of the explicit keypoint extraction algorithm are summarized in Alg !
5

---

Algorithm Optimization for Explicit Keypoint Extraction
Require: P, Qinter. Jo Soyor Se So.- papeer-prraeters: X, J. thro. thr
Get initial (QinfrQinter! according to E..
Filter get new query Qinfer' {1/g Vinfer; Probo
thro)-
for to do
Evaluate energy function E(Qinfer', P).
Update coordinates with gradient descent: Qinter' = Qinfer' - AV Qiates E EiQinter' P).
VViintee
end for
Sample final keypoints Qx = -tglg E Qinter', Prob, (glp) > thrs).
4 Experiment
In this section, evaluate SNAKE under three settings. First, we compare keypoint semantic consistency across different instances the same category, using both rigid and deformable objects. Next, keypoint repeatability of the same instance under disturbances such SE(3) transformation, noise and downsample is evaluated. Finally, we inspect the point cloud registration task on the 3DMatch benchmark, notably in a zero-shot generalization setting. Besides, ablation study is done to verify the effect of each design choice in SNAKE. The implementation details and hyper-parameters for SNAKE three settings can be found in the Appendix.
4.1 Semantic Consistency
Datasets The KeypointNet [39] dataset and meshes generated with the SMPL model [18] are utilized. KeypointNet has numerous
3D keypoints for 16 object categories from ShapeNet [3]. The training set covers all categories that contain 5500 instances. Following (38), we evaluate 630 unseen instances from airplanes, chairs, and tables. SMPL is a skinned vertex-based deformable
model that accurately captures body shape variations natural human poses. We use the same strategy in (38) to generate both training and testing data.
Metric Mean Intersection over Union (mloU) adopted to show whether the keypoints across intra-class instances have the same semantics or not. For KeypointNet, predicted keypoint is considered the same as numan-annotated semantic point if the geodesic distance between them under some threshold. Due to the lack of human-labbled keypoints keypoint consistency in a pair of human models. A keypoint the first model is regarded semantically consistent if the distance between its corresponding point and the nearest keypoint in the second model is below some threshold.
SMPL, compare the
Evaluation and Results We compare SNAKE with random detection, hand-crafted detectors;
ISS [44) Harris-3D (30 and SIFT-3D 288, and basea unsupervised detectors: USIP [15]
and UKPGAN [8). As USIP has not performed semantic consistency evaluations, we train the model with the code they provided. We fol- low the same protocols in (88 to filter the key- points via NMS with a Fuclidean radius of 0.1. Quantitative results are provided in Fig. (a.c). KeypointNet [39] dataset.
Figure 3: Comparison with human annotations on
SNAKE obtains higher mloU than other methods under most thresholds on KeypointNet and SMPL. Qualitative results in Fig. show our keypoints make good alignment with human annotations. Fig. provides qualitative comparisons of semantically consistent keypoints rigid and deformable objects. Owing to entangling shape reconstruction and keypoint detection, SNAKE can extract aligned representation for intra-class instances. Thus, our keypoints better outline the object shapes and are more semantically consistent under large shape variations. As shown in the saliency field projected slices, we can get symmetrical keypoints, although without any explicit constraint like the one used in [38]. Here, a projected slice is obtained by taking the maximum value of a given field along the projection direction.

---

UKPGAN
Ours
Sallency Field
IS8
USIP
UKPGAN
Ours
Saliency Field
Projected Slice
Projected Slice
Figure 4: Semantic consistency of keypoints on rigid and deformable objects. Our keypoints more evenly scattered on the underlying surface of objects, more symmetrical, and more semantically consistent under significant shape variations when compared to other methods. The saliency field projected slice shows that SNAKE decodes well-aligned saliency values for keypoints in different instances but with similar semantics, such the wingtip of the airplane and the leg of the human. Here, small saliency shown in bright red and gets darker with a larger value.
KespolncNet
ModelNet40
100
303
302
LEPONE
43
20
23
0.02
0.38 8:0 0o 0.08 5.10
Gusebaic Thrcalod
di Madsebvet
Contsomok 1atz Mode No400 lov0
MuddlNosO 12m0.24
SMPL
Redwesod
40
www.aaaaaaaa 7012
SPT30
3.04
6.14 6.1
032
0.92 Connaeygle Bate Redroed te-d
0.00
6:38 9.00 U.0 0.10
(D
Figure 5: Quantitative results on four datasets. Keypoint semantic consistency (a)(e) on KeypointNet and SMPL. Relative repeatability for two-view point clouds with different distance threshold (b), downsample rate (c), Gaussian noise (0, Tnoise) (d) on ModelNet40. The results of (h) are tested on Redwood with the same settings in (b)) The specific numerical results can be found in Appendix.
4.2 Repeatability
Datasets ModelNet40 37 is synthetic object-level dataset that contains 12,311 pre-aligned shapes from 40 categories, such as plane, guitar, and table. We adopt the official dataset split strategy. 3DMatch [41] and Redwood [5] are RGB-D reconstruction datasets for indoor scenes. Following (15), we train the model on 3DMatch and test it on Redwood to show the generalization performance. The training set contains around 19k samples and the test set consists of 207 point clouds.
Metric adopt the relative repeatability proposed in USIP [5] as the evaluation metrie. Given two point clouds captured from different viewpoints, keypoint in the first point cloud is repeatable if its

---

Original
St Downsample
sid. Noise
Original
Downsampl:
std. (m) Noisc
I8S
USIP
Ours
Shape Recon,
Figure 6: Visualization of keypoints under some disturbances object-level 37 and scene-level [5] datasets compared to hand-crafted [4] and explicit representation based [B] methods. Downsample rate is 8x and the Gaussian noise scale (a) is 0.06. The shape reconstruction via marching cubes for our occupancy field is also given. Visualization repeatability can be found in the Appendix.
distance to the nearest keypoint in the other point cloud is below a threshold c. Relative repeatability means the number of repeatable points divided by the total number of detected keypoints.
Evaluation and Results Random detection, traditional methods and USIP are chosen as our baselines. Since UKPGAN does not provide pre-trained models on these two datasets, we do not report its results in Fig. F but make an additional comparison on KeypointNet, which is illustrated in the next paragraph. We use NMS to select the local peaky keypoints with a small radius (0.01 normalized distance on ModelNet40 and 0.04 meters on Redwood) for ours and baselines. We generate 64 keypoints in each sample and show the performance under different distance thresholds c, downsample rates, and Gaussian noise scales. We set a fixed e of 0.04 normalized distance and 0.2 meters on the ModelNer40 and Redwood dataset when testing under the last two cases. As shown in Fig. 5- (b,f), SNAKE outperforms state-of-the-art at most distance thresholds. We do not surpass USIP on Redwood in the lower thresholds. Note that it is challenging get higher repeatability
Redwood because the paired inputs have very small overlapping regions. Fig. (H..d.g.h) show the repeatability robustness to different downsample rates (d.r.) and Gaussian noise N (0, σ)) levels. SNAKE gets the highest repeatability in most cases because the shape-aware strategy helps the model reason about the underlying shapes of the objects/scenes, which makes keypoints robust the input variations. Fig. provides visualization of object-level and scene-level keypoints of the original and disturbed inputs. SNAKE can generate more consistent keypoints than other methods under drastic input changes.
We have tried to train UKPGAN (official implementation) on ModelNer40 and 3DMatch datasets from scratch but observed divergence under default hyper-parameters. As such, provide experiment to compare their repeatability on the KeypointNet dataset, on which UKPGAN provided pre-trained model. We randomly perform SE(3) transformation the test point clouds generate the second view point clouds. Then, we select top-32 salient keypoints with NMS (radius=0.03) in each sample and report the keypoint repeatability under different distance thresholds e, downsample rates, and Gaussian noise scales. The results are summarized in Table [ which show that SNAKE achieves significant gains over UKPGAN in most cases. More discussions can be found in the Appendix.
Table 1: Relative repeatability (%) with differ- ent distance thresholds on the KeypointNet dataset.
Table 2: Relative repeatability (%) when input point clouds are disturbed (==0.03). Here, ori. means the original input.
0.03
0.05
0.07
0.09
0.10
ori.
d.r=4
d.r=8
o=0.02
o=0.03
UKPGAN
0.199
0.454
0.661
0.81
0.864
UKPGAN
0.199
0.570
0.427
0.608
0,558
Ours
0.643
0.806
0.892
0.936
0.948
Ours
0.643
0.594
0.525
0.626
0.536
8

---

Un6
Ur
U-10
Opsiiation
Figure 7: (a) SNAKE fails to predict semantically consistent keypoints without the occupancy decoder. (b) Saliency field slice with a different grid size of (1/U) (c) The impact the optimization step.
4.3 Zero-shot Point Cloud Registration
Datasets We follow the same protocols in 38 to train the model on KeypointNet and then directly test it on 3DMatch [1 dataset, evaluating how well two-view point clouds can be registered. The test set consists of scenes which include some partially overlapped point cloud fragments and the ground truth SE(3) transformation matrices.
Metric To evaluate geometric registration, we need both keypoint detectors and descriptors. Thus, we combine an off-the-shelf and state-of-the-ar descriptor D3Feat [ with our and other keypoint detectors. Following [38], we compute three metrics: Feature Matching Recall, Inlier Ratio, and Registration Recall for a pair of point clouds.
Evaluation and Results As baselines, we choose random detection, ISS, SIFT-3D, UKPGAN, and D3Feat. Note that D3Feat is a task-specific learning-based detector trained on the 3DMatch dataset, thus not included in this zero-shot comparison. Ours and UKPGAN are trained on the synthetic object dataset KeypointNet only. The results are reported under different numbers of keypoints (i.e., 2500, 1000, 500, 250, 100). The NMS with radius of 0.05m is used for D3Feat, UKPGAN, and ours. shown Table SNAKE outperforms other methods consistently under three metrics. For registration recall and inlier ratio, we achieve significant gains over UKPGAN and other traditional keypoint methods. Notably, when the keypoints are high numbers, SNAKE even outperforms D3Feat which has seen the target domain. Local shape primitives like planes, corners, curves may be shared between objects and scenes, so our shape-aware formulation allows superior generalization from objects to scenes.
Table 3: Registration result on 3DMatch. We combine the off-the-shelf descriptor D3Feat []] and
different keypoint detectors to perform two-view point cloud registration.
Feature Matching Recall (8)
Registration Recall
Inlier Ratio (%)
Detector D3Feat
Descriptor
2500
1000
500
250
100
1500
[000]
500
250
100
1500
1000
500
250
100
D3Feat
93.6
94.3
93.3
84.4
82.3
79.3
67.2
40.6
42.7
44.1
45.0
Random
D3Feat
95.1
94.5
92.8
90.0
81.2
83.0
80.0
77.0
65.5
38.8
38.6
33.6
28.9
23.6
17.3
ISS
D3Teat
95.2
93.4
90.1
81.0
83.5
79.2
76.0
64.3
37.2
38.2
33.5
28.8
23.9
17.4
SIFT
D3Feat
94.9
94.0
93.0
91.2
81.3
84.0
79.9
76.1
60.9
38.6
38.4
33.6
28.8
23.3
17.4
UKPGAN
DSFeat
94.7
94.2
93.5
92.6
85.9
82.8
81.4
77.1
69.7
47.4
38.8
35.5
34.0
33.1
27.7
Ours
D3Feat
94.7
92.9
74.6
41.3
37.0
33.5
4.4 Ablation Study
Loss Function Table reports the performance w.r.t. designs of loss functions. (Row 1) If the surface occupancy decoder is removed, the surface constraint cannot be performed according to Eq. so they are removed simultaneously. Although the model could detect significantly repeatable keypoints on ModelNet40 377, it fails to give semantically consistent keypoints on KeypointNet (39]. Fig. 7 shows that SNAKE is unable to output symmetric and meaningful keypoints without the shape-aware technique. That indicates the repeatability could not be the only criterion for keypoint detection if an implicit formulation adopted. (Row 2-4) Each loss function for training keypoint field is vital for keypoint detection. Note that the model gives a trivial solution (0) for the saliency field and cannot extract distinctive points when removing the sparsity loss.
Grid Size and Volumetric Resolution The grid size 1/U controls the number of keypoints because
C. enforces the model to predict single local maxima per grid of size (1/U)². Fig. 7 shows

---

Table Ablations for the designs of loss function. occ. = occupancy, sur. = surface, rep. = repeatability, spa. sparsity and m. relative repeatability.
Table 5: Impact of different local grid size used in the L. and L, on ModelNet4O.
6
8
10
r.. (%) (e=0.04) 0.79 0.85
0.79
0.77
(%) on [7
mloU (%)
39
Threshold
0.04
0.05
0.06
0.08
0.09
0.1
Table 6: Impact of different global volumetric resolution ModelNet40.
w/o occ. & sur.
0.92
0.94
0.95
0.22
0.25
0.28
w/o sur.
0.28
0.36
0.42
0.31
0.35
0.39
rep.
0.34
0.30
0.35
0.39
H(= W D)
32
48
64
80
w/o spa.
0
0
0
0
0
IT. (%) (c=0.04)
0.62
0.79
0.85
0.78
w/ all
0.85
0.89
0.90
0.30
0.37
0.42
different saliency field slices obtained from the same input with various 1/U. When U is small, SNAKE outputs fewer salient responses, and more for larger values of U. We also give the relative repeatability results ModelNet40 under distance threshold
0.04 in Table indicating that U = 6 gives the best results. From Table [6] we can see that higher resolution improves performance. However, the performance drops when it reaches the resolution of 80. The potential reason is as such: the number of queries in a single grid increases when the resolution becomes higher, as mentioned in
In this case, finer details make the input cosine similarity too long and contain spurious values.
Optimization Step and Learning Rate Fig. 7c shows the importance of optimization (see Alg. I for refining keypoint coordinates on the ModelNet40 dataset. It is noted that too many optimization steps will not bring more gains but increase the computational overhead. number of update steps to 10. The learning rate for optimization is also key the final result. When the learning rate is set to 0.1, 0.01, 0.001 and 0.0001, the relative repeatability (%) on ModelNet40 dataset with the same experimental settings Table 0.002, 0.622, 0.854 and 0.826, respectively. In addition, the comparison of computation cost of baselines and ours can be found in the Appendix.
this paper, we set the
5 Conclusion and Discussion
We propose SNAKE, method for 3D keypoint detection based on implicit neural representations. Extensive evaluations show our keypoints are semantically consistent, repeatable, robust to downsam- ple, and generalizable to unseen scenarios. Limitations. The optimization for keypoint extraction during inference requires considerable computational cost and time, which may not be applicable for use in scenarios that require real-time keypoint detection. Negative Social Impact. The industry may use the method for pose estimation in autonomous robots. Since our method is not perfect, may lead to wrong decision making and potential human injury.
Acknowledgments
This research jointly supported by following projects: the Scientific Innovation 2030 Major Project for New Generation of AI under Grant N0.2020AA Ministry of Science and Technology the People's Republic China; the Key Field R&D Program of Guangdong Province (No. 00-2001100014 B0101410002); Gino-n Collaborative Research Project Crossmodal Learning (NSFC 1162113600S/DFR SFB/TRRI69) the National Natural Science Foundation of China (No.6200613)) Beijing Outstanding Young Scientist Program (No. BJJWZYJHO120 1009900020099) We would like to thank Pengfei for discussions about implicit field learning. We would also like to thank the anonymous reviewers for their insightful comments.
References
[1] Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, and Chiew-Lan Tai. D3feat: Joint learning of dense detection and description of 3d local features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6359-6367, 2020.
[2] Simon Baker and lain Matthews. Lucas-kanade 20 years on: unifying framework. Interna- tional journal of computer vision, 56(3))221-255, 2004.
10

---

[3] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese. Manolis Savva, Shuran Song, Hao Su, et Shapenet: An information-rice 3d model repository. arXiv preprint arXiv: 1512.0301,, 2015.
[4] Xuelian Cheng, Yiran Zhong, Mehrtash Harandi, Yuchao Dai, Xiaojun Chang, Hongdong Li, Tom Drummond, and Zongyuan Ge. Hierarchical neural architecture search for deep stereo matching. Advances in Neural Information Processing Systems, 33:22158 22169,, 2020.
[5] Sungjoon Choi, Qian-Yi Yi Zhou, and Vladlen Koltun. Robust reconstruction of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5556-5565, 2015.
[6] Özgün Çięek, Ahmed Abdulkadir, Soeren Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention, pages 424-432. Springer, 2016.
[7] Daniel Tonone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE conference computer vision and pattern recognition workshops. pages 224-236, 2018.
[8] Jakob Engel, Jurgen Sturm, and Daniel Cremers. Semi-dens, visual odometry for monocular camera. In Proceedings of the IEEE international conference on computer vision, pages 1449 1456, 2013.
[9] Clara Fernandez- Fermandez-Labrador Ajad Chhatkuli, Danda Pani Paudel, Jose J Guerrero, Cédric Demon- ceaux, and Luc Van Gool. Unsupervised learning of category-speciff symmetric 3d keypoints from point sets. In European Conference on Computer Vision, pages 546-563. Springer, 2020.
[10] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 12(11):22311127, 2013.
[11] Zhenyu Jiang, Cheng--Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. arXiv preprint arXiv:2202.08227, 2022.
[12] Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, and Yuke Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. Robotics: science and systems, 2021.
[13] Andrew E Johnson and Martial Hebert. Using spin images for efficient object recognition in cluttered 3d scenes. IEEE Transactions on pattern analysis and machine intelligence, 21(5):433-449 1999.
[14] Chang Ha Lee, Amitabh Varshney, and David W Jacobs. Mesh saliency. In ACM SIGGRAPH 2005 Papers, pages 659-666. 2005.
[15] Jiaxin Li and Gim Hee Lee. Usip: Unsupervised stable interest point detection from 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 361-370, 2019.
[16] Pengfei Li, Yongliang Shi, Tianyu Liu, Hao Zhao, Guyue Zhou, and Ya-Qin Zhang. Semi- supervised implicit scene completion from sparse lidar. arXiv preprint arXiv:2 2021.
[17] Lingji Liu, Jiatao Gu, Kyaw 7aw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651-15663, 2020.
[18] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. ACM Trans. Graph., 34(6), oct 2015.
[19] David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.

---

[20] Lars Mescheder, Michael Oechsle. Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction function space. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4460 4470, 2019.
[21] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference computer vision, pages 405-421. Springer, 2020.
[22] David Nister and Henrik Stewenius. Scalable recognition with vocabulary tree. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 006), volume 2, pages 2161-2168. Icee, 2006.
[23] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[24] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference on Computer Vision, pages 523 923-540.30 Springer, 2020.
[25] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[26] Hossein Rahmani, Arif Mahmood, Q Du Huynh, and Ajmal Mian. Hope: Histogram of oriented principal components of 3d pointclouds for action recognition. In European conference on computer vision, pages 742-757, Springer, 2014.
[27] Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel. R2d2: Reliable and repeatable detector and descriptor. Advances in Neural Information Processing Systems, 32, 2019.
[28] Blaine Rister, Mark A Horowitz, and Daniel L Rubin. Volumetric image registration from invariant keypoints. IEEE Transactions on Image Processing, 26( 26(10)4900-4910, 2017.
[29] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33:20154 20166, 2020.
[30] Ivan Sipiran and Benjamin Bustos. Harris 3d: a robust extension of the harris operator for interest point detection on 3d meshes. The Visual Computer, 27(11):963-977 2011.
[31] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
[32] Noah Snavely, Steven M Seitz, and Richard Szeliski. Modeling the world from internet photo collections. International journal of computer vision, (012):189-210 2008
[33] Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multi- scale signature based on heat diffusion. In Computer graphics forum, volume 28, pages 1383-1392. Wiley Online Library, 2009.
[34] Supasorn Suwajanakorn, Noah Snavely, Jonathan J Tompson, and Mohammad Norouzi. Discov- ery of latent 3d keypoints via end-to-end geometric reasoning. Advances in neural information processing systems, 31, 2018.
[35] Federico Tombari, Samuele Salti, and Luigi Di Stefano. Performance evaluation of 3d keypoint detectors. International Journal of Computer Vision, 102(1):198-225) 2013.
[36] Philippe Weinzaepfel, Jerome Revaud, Zaid Harchaoui, and Cordelia Schmid. Deepflow: Large displacement optical flow with deep matching. In Proceedings of the IEFE international conference computer vision, pages 1385-1392, 2013.
12

---

[37] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1912-1920, 2015.
[38] Yang You, Wenhai Liu, Yanjie Ze. Yong-Lu Li, Weiming Wang, and Cewu Lu. Ukpgan: A general self-supervised keypoint detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[39] Yang You, Yujing Lou, Chengkun Li, Zhoujun Cheng, Liangwei Li, Lizhuang Ma, Cewu Lu, and Weiming Wang. Keypointnet: A large-scale 3d keypoint dataset aggregated from numerous human annotations. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 13647-13656, 2020.
[40] Aviad Zabatani, Vitaly Surazhsky, Erez Sperling, Sagi Ben Moshe, Ohad Menashe, David H Silver, Zachi Karni, Alexander M Bronstein, Michael M Bronstein, and Ron Kimmel. Intel Te sr300 coded light depth camera. IEEE transactions on pattern analysis and machine intelligence, 42(10))2333-2 233--2345, 2019.
[41] Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas A. Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 199-208, 2017.
[42] Hao Zhao, Rene Ranftl, Yurong Chen, and Hongbin Zha. Transferable end-to-end room layout 2021.
estimation via implicit encoding. arXiv preprint
[43] Chengliang Zhong, Chao Yang, Fuchun Sun, Jinshan Qi, Xiaodong Mu, Huaping Liu, and Wenbing Huang. Sim2real object-centric keypoint detection and description. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 5440-5449, 2022.
[44] Yu Zhong. Intrinsic shape signatures: A shape descriptor for 3d object recognition. 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, pages 689-696. IEFE, 2009.
[45] Qian Yi Zhou, Jaesik Park, and Vladlen Koltun. Fast global registration. In European conference on computer vision, pages 766-782. Springer, 2016.
[46] Yi Zhou, Chenglei Wu, Zimo Li, Chen Cao, Yuting Ye, Jason Saragih, Hao Li, and Yaser Sheikh. Fully convolutional mesh autoencoder using efficient spatially varying kernels. Advances in Neural Information Processing Systems, 33:9251-9262 2020.
13


Reviews:
[{"review_id": "dcARkPno-5", "reviewer": "Reviewer_5p2u", "summary": "The paper is well written, the motivation and technical details are clearly presented. The idea of estimating saliency field from sparse keypoints seem novel, and it is shown to be effective to produce repeatable and consistent keypoint detection result.", "questions": "Additional visualization of repeatibility of keypoints under SE3 transformation as well as different sparsity of points can make the contribution of the paper more clear. The plots in Figure 5 alone is not visual enough to show the quality of the proposed method in terms of repeatibility.", "limitations": "Keypoint extraction during inference requires considerable computational cost, not suitable for real-time on-device applications.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe paper is well written, the motivation and technical details are clearly presented. The idea of estimating saliency field from sparse keypoints seem novel, and it is shown to be effective to produce repeatable and consistent keypoint detection result.\n\nWeakness:\n\nKeypoint extraction in inference time requires iterative gradient descent and query the implicitly defined saliency field. Thus the computational cost is high and is not suitable for real-world applications at its current form.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "WFiXaP9Q0Xz", "reviewer": "Reviewer_2TwX", "summary": "The paper presents an unsupervised method to predict 3d keypoints from point cloud. Several novel losses are proposed to enforce repeatability, Surface Constraint, and Sparsity. They also achieve superior performance on various public benchmarks. They also use the novel two head to model occupancy and saliency separately to better disentangle these two tasks and let them serve as independently different functions.", "questions": "Are there any other newer methods to compare with? The UKPGAN seems to be in the year 2020.\nCan you elaborate more on discrete space and continuous space?\nCan you elaborate more of the difference when compared with \"R2d2: Reliable and repeatable detector and descriptor\"?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths：\nI like the intuition that starts from continuous instead of discrete space. \nThe proposed architectures and losses are novel as far as I see. \nThe presentation is very well.\nIt achieves the SOTA performance on several datasets.\nWeaknesses：\nLack of related work discussion: FULLY CONVOLUTIONAL MESH AUTOENCODER USING EFFICIENT SPATIALLY VARYING KERNELS\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "PDx5_h8F0Ns", "reviewer": "Reviewer_xHwu", "summary": "The authors introduced a method to detect keypoints from point clouds. They leverage a deep learning model to learn an implicit function which is a mapping to provide each location a probability of being a keypoint. To make the method shape-aware, the authors use deep learning models to learn the kepoint field along with the learning of an occupancy field to explore whether the shape information can improve the keypoint detection. The contribution lies in the way of combining the learning of implicit fields and keypoint fields. The authors evaluate the effectiveness by comparing it with the latest methods under the widely used benchmark.", "questions": "1. First of all, I do not think it makes any sense to learn the probability of keypoints as a field. Keypoints should be located on surfaces, they are not floating in the space. Specifically, the authors aim to detect keypoints from input point clouds, the keypoints to be determined should be some of the input points. It should be much easier to learn the probability of keypoints among the discrete input points than in the continuous whole space.\n \n2. The motivation of improving the performance with surface reconstruction does not make any sense either. Surface reconstruction is a harder task than keypoint detection, since point clouds have been given as input which provides a range as a good constraint to find solutions for kepoint detection. We already know the structure of shapes represented by point clouds and why we have to learn an implicit function to reconstruct shapes. I believe it just makes this problem more complex to get resolved.\n \n3. The authors claim that the proposed method is a novel unsupervised method. This is also not correct, since this method requires ground truth occupancy information as a supervision.\n \n4. The proposed method is also not novel. UKPGAN has explored the feasibility of combining shape reconstruction and keypoint detection together, although UKPGAN reconstructs the input point cloud rather than learning an implicit function to represent the same shape. The minor difference here is what representation is used to represent 3D shapes.\n \n5. Learning occupancy field results in using an additional surface constraint. This term should not be there due to the learning of the occupancy field. Fig.2(b) may be confusing, I think the close to 0 line should be parallel to the x axis rather than y axis.\n \n6. The term of repeatability loss is hard to understand. Why do we need a term like this? For shapes, they are well aligned, why do we need to consider the rigid transformation? Another question is about the cosine similarity, the probabilities are scalars, right? Is there any reason to use cosine similarity to evaluate the difference between two numbers? \n \n7. The authors use occupancy probability to represent the inverse distance between the query and the input. I do not think it works here. Since occupancy probability always gets smaller from 1 to 0 when moving a query from inside to outside while the inverse distance gets smaller to larger before going across the surface, and then gets smaller after that.\n \n8. Explicit keypoint extraction via optimization is also an operation that I do not understand. An intuitive idea of extracting keypoints is to use points of the input point clouds as queries to predict the occupancy values since keypoints can only locate on the surface. Why do we have to extract keypoints via updating the locations of randomly sampled queries in the 3D space?\n \n9. In the visualization of saliency field slices, how to select the slices to visualize? \n \n10. How to determine the keypoint number? I noticed that the numbers of keypoints produced by different methods are different even in the same case, such as the visual comparison in Fig.6. Why do not use the same number of points to perform the comparison？\n \n11. The reason why I do not think the reconstruction can improve the performance of keypoint detection is the results in Table 2. The results without surface reconstruction are better than the results with surface reconstruction. I do not think this is a good support for the argument.\n \n12. With the ground truth occupancy supervision, it is not a fair comparison with other methods.\n", "limitations": "Yes, the authors addressed the limitations and potential negative societal impact of their work.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The visualization is good.\n2. The paper is easy to follow.\n \n \nWeaknesses:\n1. The motivation is not convincing at all.\n2. The experimental results cannot justify the effectiveness of the method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "C-c4H4cWr2x", "reviewer": "Reviewer_ZdeR", "summary": "The paper presents a novel unsupervised method SNAKE to detect 3D keypoints from point clouds based on implicit neural representations. The key idea is to combine shape reconstruction and keypoint detection during training. Experiments show that jointly learning 3D shapes and key points improves semantical consistency, better repeatability under disturbance, and accurate geometric registration under zero-shot settings. \n", "questions": "- L224 states that UKPGAN is not involved in the experiments due to the absence of pretrained model. Since their code is publicly available, I wonder if training from scratch is possible? If not, I wonder if it is possible to compare alternative datasets? \n- The registration experiments (sec 4.3) rely on D3Feat descriptors for the detected keypoints. I am aware that this descriptor is commonly used in the UKPGAN’s experiment. I am interested in understanding the bottleneck of the problem. Since D3Feat detector + D3Feat descriptor still serves as an upper bound in this experiment, I wonder whether it is possible that certain detector requires specifically designed descriptors in order to work perfectly in the registration problem. D3Feat descriptor may not provide the best features for SNAKE or UKPGAN. \n", "limitations": "Yes. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Though the idea of combining reconstruction and saliency prediction is not new (UKPGAN also reconstructs shape), this paper takes advantage of implicit representation and shows advantages over the GAN-based method. The proposed method is simple but effective. The proposed four loss functions are intuitive and ablations show that they are important.\n+ The experiments are mostly thorough and show good qualitative and quantitative results compared to competitive baselines. \n+ The paper is well written and the figures are easy to understand. \n- Missing comparison with UKPGAN in section 4.2. UKPGAN is a competitive baseline and its code is publicly available. \n\nPost-rebuttal:\nMy final rating is weak accept. Thanks to the authors and reviewers for their effort. The rebuttal mostly answers my questions. I think the paper is novel and has enough difference from UKPGAN -- the method does not have a GAN and the experiment results are better. However, I do think D3Feat descriptor may not be the best way to evaluate the proposed method in experiments since it is designed for D3Feat detector and may not be discriminative enough for different feature detectors. Since the prior works are using this protocol, I won't criticize too much for following it.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept