Correspondence-Free Point Cloud Registration with SO(3)-Equivariant Implicit Shape Representations
Authors: Minghan Zhu
Maani Ghaffari
Huei Peng
Venue: CoRL2021 Poster (2021)
Keywords: point cloud registration
implicit shape model
equivariant neural network
representation learning

Abstract: This paper proposes a correspondence-free method for point cloud rotational registration. We learn an embedding for each point cloud in a feature space that preserves the SO(3)-equivariance property, enabled by recent developments in equivariant neural networks. The proposed shape registration method achieves three major advantages through combining equivariant feature learning with implicit shape models. First, the necessity of data association is removed because of the permutation-invariant property in network architectures similar to PointNet. Second, the registration in feature space can be solved in closed-form using Horn's method due to the SO(3)-equivariance property. Third, the registration is robust to noise in the point cloud because of the joint training of registration and implicit shape reconstruction. The experimental results show superior performance compared with existing correspondence-free deep registration methods.  

Correspondence-Free Point Cloud Registration with S0(3)-Equivariant Implicit Shape Representations
Minghan Zhu', Maani Ghaffari", Huei Peng'
'Department of Mechanical Engineering, University of Michigan, United States DDepartment of Naval Architecture and Marine Engineering, University of Michigan, United States (minghanz, maanigj, hpeng) Coumich.ee
This paper proposes a correspondence-free method for point cloud ro- tational registration. We learn an embedding for each point cloud in a feature space that preserves the S0(3)- property, enabled by recent develop- ments in equivariant neural networks. The proposed shape registration method achieves three major advantages through combining equivariant feature learning with implicit shape models. First, the necessity of data association removed because of the permutation-invarian property in network architectures similar PointNet. Second, the registration in feature space can be solved in closed-form using Horn's 's method due to the SO(3)-cquivariance property. Third, the registra- tion is robust to noise in the point cloud because of the joint training of registration and implicit shape reconstruction. The experimental results show superior perfor- mance compared with existing correspondence- free deep registration methods.
Abstract:
Point cloud registration, implicit shape model, equivariant neural net- work, representation learning
Keywords:
Introduction
Point clouds are a major form of 3D information representation in perception with numerous ap- plications today, including intelligent robots and self-driving cars. The registration of two different point clouds capturing the same underlying object or scene refers to estimating the relative trans- formation that aligns them. Point cloud registration enables information aggregation across several observations, which is essential for many tasks such as mapping, localization, shape reconstruction, and object tracking. Given perfect point-to-point pairings between the point-cloud-pat the relative transformation can be solved in closed-form using Hor'' 's method [1].
However, such a pairing is not available or is corrupted in real-world problems, posing significant challenge for the point cloud registration task, namely data association. ICP matches the closest points in Euclidean space and iteratively updates the matching using the current pose estimation [2]. In the deep learning era, the strong fcature learning capacity of neural networks enables better description and discrimination of points, improving their matching. Overall, these methods can be categorized as correspondence-basee registration.
An alternative approach, called correspondence- registration, is to completely circumvent the matching of points [3] by learning global representation for the entire point cloud and estimating the transformation by aligning their representations. Similar to correspondence--asee registration, correspondence-free registration can also be with or without the involvement of deep learning. The main obstacle in this approach is that the mapping between the input Euclidean space and the output feature space realized by neural networks is nonlinear and obscure. Thus it is common to rely on iterative optimization methods with local lincarization such as Gauss-Newton [4]. The performance of these approaches deteriorates when the initial pose difference gets larger. A method called global if the registration result is independent of initial pose, or local otherwise [5]. Most existing correspondence- free methods are local.
In this paper, we employ equivariant neural networks and implicit shape learning into correspondence-free registration, achieving two major advantages. First, because of the equivari-
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

SOX S0Ssquiivaria encoder
Implicit shape deveder
SVD
Point cloud
Registration space
Occupancy field
Figure 1: Overview of the correspondence-free free rotational registration network. The point cloud input is of shape R and the encoded feature is of shape R N is the number of points, and C' is the dimension of features. Occupancy field is function e(p) 10,1]. RR² mapping any 3D coordinate an occupancy value (sec Sec. 3.2 for details). The rotation estimated by aligning the features using Hom's 's method [1].
ance property, the feature space preserves the same rotation operation as the Euclidean input space, allowing us to solve the feature-space registration in a closed form using Horn's 's method. As a result, our method is global, i.e., invariant the initial pose difference. Second, we improve the registra- tion robustness against noise in point clouds to better handle challenges in real-world applications through training the features for both the implicit shape reconstruction and registration tasks. The implicit shape learning encourages the network to learn features describing the underlying geometry rather than specifie points sampled on the geometry; thus, the features are better prepared for reg- istration by including the registration in the training loop. In this work, we focus on the rotational registration task.
In particular, the main contribution of this paper are:
1. We propose correspondence-fres point cloud registration method via -elev-vriuaar representation learning and closed-form pose estimation !
2. The proposed method is global, delivering consistent registration regardless of initial rota- tion error.
3. The proposed method is robust against imperfections in input point clouds, including noise, sampling, and density differences.
Preliminaries on Equivariance and Vector Neurons
Symmetry (or equivariance) in neural network is an important ingredient for efficient learning and generalization, It also allows us to build a clear connection between the changes in the input space and the output space. For example, convolutional neural networks are translation-equivarinnt It makes a network generalize to image shifting and enables huge parameter saving. However, other symmetries are less explored. For example, most point cloud processing networks (e.g., Point- Net [6], DGCNN (7]) are not equivariant to rotations.
We leverage the recently proposed Vector Neuron layers [8] to enable rotation -etttiii-eu feature learning for point clouds. For the sake of completeness, we briefly introduce the preliminary on cquivariance and the design of layers.
X is equivariant commutes, i.e., g. f(x) = f( x),, Vr E X. For example, applying a translation on a 2D image and then going through a convolution layer is identical processing the original image with a convo- lution layer and then shifting the output feature map. Therefore convolution layers are translation- equivariant.
A function
set of transformations G, if for any 9 E G. f and
'Code will be available https: /etg://ggtttt http:////..ngggan////iii.eee

---

Vector Neurons extend the equvariance property to (3(3) rotations for point cloud neural networks. The key idea is augment the scalar feature each feature dimension to vector in E and redesign the linear, nonlinear, pooling, and normalization layers accordingly. In regular network, the feature matrix with feature dimension C' corresponding to set N points V E NR In Vector Neuron networks, the feature matrix is of form V E TR. NxCx3 The mapping between layers can be written as f R RaxxC,x3 + x++++ x3 where the layer index. Following this design, the representation of (3) rotations in feature space is traight-forward, g(ft) V := VR, where g ((It) denotes the rotation operation the feature space, parameterized by the 3-by-3 rotation matrix R SO(3). In the following, we ignore the first dimension N of V for simplicity.
The linear layer for Vector Neurons is similar a traditional MLP: fin(V) WV, where W E RCa1xCC It is easy to see that such a mapping is (0(3)
( fim (x) = WVR R fin (R( V)
(1)
Designing nonlinearities for Vector Neuron networks is less trivial. A naive ReLu on all elements in the feature matrix will break the equivariance. Instead, vectorized version of ReLu is designed as the truncation of vectors along a hyperplane. Specifically, given an input feature matrix V, predict a canonical direction k- UV E _ where U E 1R Then the ReLU defined as:
e'
U U
if k) 2 0 otherwise
(2)
(u.
A |AT
for each row R1x3 of V. This design preserves SO(3) equivariance because A and V will rotate together, and (VR, kR) (V, k). meaning that rotations do not change the activation. For the design of pooling layers and normalization layers of Vector Neurons, please refer the original paper for details [8].
3 Methodology
We employ S0(3) neural network to construct a feature space preserving the same rotation representations as to the input space, thus simplifying solving the rotation feature align- ment. Furthermore, deep implicit representation learning and end-to-end training of registration improve the robustness of the features against noise and allow registration between two different scans of the same geometry.
3.1 (3)
point cloud global feature extractor
We choose PointNet as the feature extractor backbone. The permutation permutation-invariiat property implies that two point clouds correspond to the same feature embeddings if they only differ in the permuta- tion of points. We replace the layers in PointNet with the corresponding Vector Neuron version [8]. Denote the feature extraction network f, a point cloud R Nx3 and its rotated and permuted copy P' = MR,, where M E R N X is permutation matrix and R SO(3) is 3-by-3 rotation matrix. Then we have
Q' (P= = J(MP) S(P) QR
(3)
where Q = /(P) E RC33 the feature of P extracted by the encoder. Equation (3) guaranteed by the -ouu)--quivariinnce and permuaation-invaiianc properties, which is essential for our feature- space registration.
A Vector Neuron network is designed to take inputs of R xCx33 while the point cloud input is P. R. x3 Simply reshaping P' to Il Nx1x3 with C2 does work, because a linear layer with weight W E R x will make the resulted feature WP& E I. Nxx1x3 linearly dependent in the third dimension for each point. Following [8], we use one DGCNN edge-convolution [7] layer to initialize the R N Nxxxx x3 feature from raw R Nx3 point cloud using the neighboring points. See [8] for more details. Furthermore, we explore replacing the k-nearest neighbor operation with a random sampling of points in a ball with certain radius in the edge convolution layer. The motivation is to alleviate the sensitivity of the feature against point cloud density (see Sec. 4.2.3). However, the evaluations in this paper use K-NN by default unless specified.
3

---

3.2 Deep implicit representation learning
Being able register two rotated copies of the same point cloud is not enough for practical use. In real applications, the measurements have noise, and the individual points captured by different scans generally do not correspond the same physical points.
Our solution is to repurpose the global feature (for registration ultimately) for implicit shape repre- sentation. Following the Occupancy Network [9], build encoder-decoder network. The en- coder the aforementioned
feature extractor f. taking a point cloud P E IR Nx3 as input. The decoder takes the encoded shape feature Q = f(P) IR Cxx3 and a queried position E I as input, predicting the occupancy value of that position v(p: Q) E [0, 1J. Notice that need not be coming from P. In practice, we sample n querying points for each input calculate the implicit shape reconstruction (occupancy value prediction) loss:
Loce
Lemos entropy (p(ps; [(P)), Ug (p:)),
(4)
where (p)(p) E {0, J is the ground truth occupancy value of a query point p. The sampling strategy of the querying points is consistent with the Occupancy Network [9]. In this way, we implicitly encourage the feature f(P)), to be close as long as P1, P2... are point clouds of the same geometry.
3.3 Point cloud registration through feature alignment
With the above network design, we can further relax the connection between the two point clouds P' and P in (3). The features Q and Q'are approximately are not permutations of each other, as long as they are sampled on the same geometry. We say is approximate because trained implicit shape reconstruction network can only represent the implicit shape approximately. Therefore, we also include the registration step in the training loop, directly encouraging the accurate registration of point clouds with noise. As shown below, the registration has a closed-form solution and is differentiable, allowing end-to-end training of features.
even when P' and pr
One can see that (3) is exactly in the form of an orthogonal Procrustes problem. The learned features Q and Q' can be regarded as two pseudo point clouds, and each row of them automatically matched. This problem has standard closed-form solution. First, calculate the cross-covariance matrix = Q" Q' followed by its Singular Value Decomposition (SVD) as = USVT. The optimal rotation matrix is given as
A -= VAUT.
(5)
where A diag(L. 1, det((VUT) E 3333 We design the registration loss as the mean squared error between the residual rotation matrix and the identity matrix, i.c. chordal distance, as
Leeg
R
Is 1
(6)
where Res is calculated from (5), and ||-|| denotes the Frobenius norm.
4 Experiments
4.1
Overview
Our network trained on the ModelNet40 dataset [10]. It is composed of 12311 CAD models from 40 categories of objects. Following the official split, 9843 models are used for training. and 2468 models are used for testing. The models are preprocessed using the tool provided by Stutz and Geiger [11] to obtain watertight and simplified models that are centered and scaled to unit cube. Given watertight models, the occupancy value of any 3D coordinate can be determined depending on whether it is inside or outside the model.
We also evaluate our method on a real-world indoor dataset: 7Scenes [12], which is a subset of 3DMatch [13]. We choose it to be consistent and comparable with one of our baselines, Feature- Metric Registration (FMR) [14]. 7Scenes consists of sequences of RGBD seans of indoor envi- ronments. The RGB information is not used in the experiments.

---

Table I: Rotational registration error given rotated copies of point clouds. Tested using ModelNet40 official
lest set. The best are shown in bold. The second best are shown in italic. All values are in degrees.
Max initial ritation angle
0
30
60
90
120
150
180
Calegorie
Global
Methods
Rotation error atter registration
Correspondenes-
PCR-Na
7.06
9.50
27.38
68.22
109.61
129.29
133.49
N
FMR[14]
0.00
0.45
5.29
21.95
42.26
66.39
79.43
Iree
Ours
0.02
0.02
0.02
0.02
0.02
0.02
Correspondence-
N
RPM-Net [6]
0.26
0.27
0.42
1.57
2.85
3.42
4.02
based
Y
DepGMRIS5
0.02
0.02
0.02
0.02
0.02
0.02
0.02
The training of the network is composed of two stages. The network is first trained for occupancy value prediction only, after which the network is trained for the occupancy prediction and registration tasks jointly. In the first stage. 1024 points are randomly sampled on each mesh model as the input point cloud with batch size 24. In the second stage, a pair of point clouds are sampled on the same shape model with a different pose and number of points, The registration loss between the pair is calculated. together with the occupancy loss of each of the point clouds. The occupancy value is queried at 2048 points in the unit cube. The batch size of such pairs is 10 in this stage.
The dimension of the learned global feature is set C' 342, corresponding dimensions for a traditional feature vector, close to the dimension chosen by previous literature including PointNetLK [3], PCRNet [15], and FMR [14]. The network is trained on single Tesla V100 GPU.
342 3 1026
Representative correspondence- free methods PCRNet and FMR are our basclines. PointNetLK also falls in this category, which we omit here because FMR follows a very similar approach to Point- NetLK and can be viewed its improved version. We also included results from some representa- tive correspondence-baat bassed methods to give readers a better context of both types of approaches. The experiments the baselines based on their official open-source implementation and pretrained weights.
In all following experiments, random initial rotations are generated using the axis-angle represen- tation, of which the angle sampled from a uniform distribution with the specified maximum, and the rotation axis is also randomly picked. The evaluation metric is the mean isotropic rotation error as in [16].
4.2 Evaluation on synthetic data: ModelNet40
4.2.1 Registration with rotated copies of point clouds
We first tested the registration of two points clouds that are only different by rotation and permu- tation. Both point clouds have 1024 points sampled from the mesh model. The results are shown in Table I. We can see that as local methods, both PCR-Net and FMR only work properly when the initial rotation is small because they rely on iterative refinements. In comparison, our method provides consistent and almost perfect registration independent of the initial rotation angle.
Both correspondence- methods, RPM-Net and DeepGMR, work well in all initial conditions. DeepGMR delivers similarly perfect accuracy as ours, both of which are global methods. The error of RPM-Net slightly increases as the initial rotation gets larger while remaining small overall, probably because the inputs are noise-free, and perfectly matching pairs of points exist in the data, allowing the correspondence-fndinn process to work nicely.
4.2.2 Registration with rotated copies of point clouds corrupted with Gaussian noise
In practice, the pair of point clouds for registration will not be identical to each other. Therefore we test the case where the source and target point clouds are both corrupted by noise. We add a Gaussian noise with = 0.01 to all points before putting them through the networks. The rest of the setup the same as in Sec. 4.2. As shown Table 2, PCR-Net and FMR show a similar trend of increasing error when the initial angle gets larger. Our method, though showing a slightly larger registration error, still outperforms the baseline most columns. Most importantly, the error remains consistent across different initialization.
5

---

Table 2: Rotational registration of noisy point clouds ModelNee40..
Max initial rotation ungle
30
120
150
180
Calegories
Global
Methods
Rotation
registration
Corresonddenee-
N
PCR-Nei[IS]
7.21
9.30
27.40
69.96
108.4;
125.55
132,40
FMR[14]
1.56
2.01
5.78
19.95
45.73
67.65
82.05
free
Ours
2.84
2955
2.79
3.58
3.39
3.40
3.26
Correspondence-
RPM-Net[G]
0.22
0.24
3.14
23.14
52.20
84.31
109.59
based
Y
DeePGMR15]
1.73
1.74
1.76
1.77
1.75
1.74
1.74
Table 3: Rotational registration error of point clouds with different densities ModelNet40.
Max initial rotation angle
30
60
120
130
180
Categories
Global
Methads
Rotation
after registration
Correspondenc-
N
PCR Ne[
7.37
9.70
27.04
65.95
109.22
126.43
130.37
FMR[14]
1.95
6.91
23.75
45.14
free
Y
Ours
16.16
15,20
16,59
15.74
16,93
17.16
16,50
Corresoondence
0.52
based
Y
DeepGMR[S]
19.60
19.50
19.67
19.98
19.33
19.24
19.74
As the point-wise correspondence corrupted by noise, RPM-Net deteriorates fast as the initial rotation grows. DeepGMR performs the best, showing the benefit of global methods and that its component-wise correspondence more robust noise. also shows small gap between our correspondence--ree method and the state-of- thettht--rttt correspondence-batt global method, motivat- ing future improvements.
4.2.3 Registration with point clouds with different densities
To further test the robustness of the proposed method against variations in the point clouds, we tested the registration performance when given two point clouds of different densities. It is practical challenge when the sensor changes or one conducts trame-to-mode alignments. Here we sample 1024 and 512 points for the pair of point clouds, respectively. The result is shown in Table 3. We do observe an increase error our method this experiment. possible reason is that DGCNN layer for feature initialization at the beginning of the encoder is sensitive to the point cloud density since takes the relative coordinate of neighboring points as inputs. Therefore, we also experimented with replacing the k-nearest neighbor with random sampling in ball centered at each point when collecting neighboring points for the edge-convolution of that point. A more detailed discussion is in the supplementary material.
With either of the strategies, the registration error is slightly larger than that of the baselines when the initial rotation angle is small but is much smaller when the initial rotation is arbitrary. We notice methods. RPM-Net outperforms DeepGMR the initial rotation is small but performs much worse otherwise. Between the two global methods, DeepGMR shows a slightly larger error than ours.
a similar trend in the correspondence-basee
4.3 Evaluation on real-world data: 7Scenes
The model evaluated on 7Scenes is first trained on ModelNet40 with the two stages mentioned in Sec. 4.1, and then finetuned on 7Scenes with the registration loss only. The train/test split is consistent with the work of Huang et al. [14].
The experiments on 7Scenes consist of two parts. the first part, the pair of point clouds are from the same frame but sampled differently. In the second part, the pair of point clouds are from adjacent frames and only overlaps partially. They reflect challenges in real-world applications since any change of the viewpoint of the sensor could result in different samples of points and visible parts. Due to the page limit, we only show the result of the first part here, and the second part is in the supplementary material.
4.3.1 Registration with point clouds that are sampled differently
The original dense RGBD scan has more than 100,000 points. 1024 points are sampled randomly to form an input point cloud. The pair of point clouds then go through random rotation to form the final input. In this case, the points in one point cloud are neither noisy version nor a subset of

---

Table 4: Rotational registration error of point clouds sampled differently 7Scenes.
Max initial ritation angle
30
60
90
120
150
180
Methods
error aller regisiration
FMR[14]
1,591
1,405
6.614
19.829
39,496
68.463
84.597
Ours
4.701
4.319
4,932
6.070
5,474
5.189
5.388
the other. To our knowledge, we are one of the first to report registration results under this setting in correspondence-ffe methods [3, 14, 15]. Table 4 shows that our method is working properly on real-world scans of point clouds that are different samplings of the underlying geometry.
Overall, our proposed rotational registration method can estimate consistent results independent of initial rotations. It outperforms the correspondence-free baselines in all evaluation settings when the initial rotational angle arbitrary. Experiments on synthetic and real-world data are conducted, covering conditions with noisy inputs, varying density, different sampling, and partial overlapping. However, our method still shows some error increase when challenging inputs are presented, moli- vating future improvements.
5 Related Work and Discussion
5.1 Correspondence-baset point cloud registration
major challenge here is to recover the corresponding pairs of points from pair point clouds. ICP simply matches the closest points together, solves the transformation, and iteratively rematches the closest points after aligning the two point clouds using the estimated transformation [2]. Point-to- line [17]. point-to-plane [18]. plane-to-plane [19], and Generalized-ICE [20] build local geometric structures to the loss formulation.
Finding correspondences require strong feature descriptor for points, on which deep learning ap- proaches show expertise. Through metric learning, a feature descriptor can be learned such that matching points are close to each other in the feature space, while non- mi--matching points are far away. Approaches following this idea include PPFNet [21]. 3DSmoothNet [22], SpinNet [23], and FCGF [24]. Good point matching leads to accurate pose estimation when solving the orthogonal Procrustes problem. Therefore, the error in pose estimation can used to supervise point matching and fea- ture learning. DCP [25]. RPM-Net [16]. DGR [26]. and 3DRegNet [27] are some of the methods leveraging ground truth pose to supervise point feature learning. Neural networks are designed to better pick the keypoints (e.g., USIP [28] and SampleNet [29]). Keypoint selection and feature learning may also be considered jointly. Representative works include 3DFeat-Net [30], D3Feat [31], DeepICP [32], and PRNet [33].
The main remaining challenge that such matching pairs may not exist in the input point clouds in the first place, because point clouds are sparse and point may not be captured repeatedly by differ- ent scans. Soft matching (or many to many) [34, 35] is proposed to address this problem, but is at best an approximation of the underlying true matching using sparse samples, which can deteriorate the performance when the sparsity increases. DeepGMR [5] learns latent Gaussian Mixture Models (GMMs) for point clouds and establishes simponent-to-compone correspondences.
5.2 Correspondence-free point cloud registration
Correspondence-free registration treats a point cloud as a whole rather than a collection of points, requiring a global representation for an entire point cloud. In this way, the limitation brought by the sparsity as mentioned in Sec. 1. is circumvented. CVO [36, 37] represents a point cloud as function in a reproducing kernel Hilbert space, transforming the registration problem to maximizing the inner product of two functions. A series of deep-learning based methods attempt to extract global feature embedding for a point cloud and solve the registration by aligning the global features. Examples include PointNetLK [3], PCRNet [15], and Feature- Metric Registration [14]. A limitation of lobal-featuue-bbsee registration is that the nonlinearity of the feature extraction networks leaves us few clues about the structure and properties of the feature space. Therefore these methods rely on iterative local optimization such as the Gauss-Newton algorithm. Consequently, these methods require good initialization to achieve decent results. Our method differentiates from previous work
7

---

in that the feature extraction network is SO(3) equivariant, enabling well-behaved optimization in the feature space (detailed in Sec. 3).
5.3 Group equivariant neural networks
Neural networks today mainly preserve symmetry against translation, limiting the capacity of net- works deal with broader transformations presented in data. One line of work forms kernels as some steerable functions so that the rotations in the input space can be transformed into rotations in the output space [38, 39, 40]. However, this strategy can limit the expressiveness of the feature extractor since the form of the kernels is constrained.
Another strategy is to lift the input space higher-dimennional space where the group is con- tained so that equivariance is obtained naturally [41, 42]. However, for each lifted input to K group elements, we need to integrate over the entire group for computing its convolution. While being mathematically sound, this approach involves the use of group exp and log maps (for mapping from and to the Lie algebra) and a Monte Carlo approximation to compute the convolution integral over a fixed set of group elements [42, Sections 4.2-4.4).
Cohen and Welling [43] studied the problem of learning the irreducible representations of commuta- tive Lie groups. Kondor and Trivedi [41] studied the generalization of equivariance and convolution in neural networks to the action of compact groups. The latter also discussed group convolutions and their connection with Fourier analysis [44, 45]. In this context, the convolution theorem naturally extends to the group-valued functions and the representation theory [46].
new design of SO(3) equivariant neural networks is proposed by Deng et al. [8]. This approach allows incorporating the SO(3) equivariance property into existing network architectures such as PointNet and DGCNN by replacing the linear, nonlinear, pooling, and normalization layers with their vector newron version. Our equivariant feature learner builds on this simple and useful idea.
5.4 Deep implicit shape modeling
Implicit shape modeling represents shape as field on which function such as an occupancy or signed distance function is defined. The surface is typically defined by the equation f(z, y, 2) = for some constant c. In recent years, deep models are developed to model such implicit function fields, where network is trained to predict the function value given query input point coordinate. Some representative works include DeepSDF [47], OccNet [9], NeRF [48].
Implicit functions model a shape continuously, up to the infinite resolution, which offers opportu- nities registration eircumvent the challenge in data association due to sparsity point clouds. DI-Fusion [49] leverages deep implicit models to do surface registration and mapping by optimizing the position of the query points to minimize the absolute SDF value. iNeRF [50] performs RGB- only registration through deep implicit models by minimizing the pixel residual between observed images and neural-rendered images. In this work, we also exploit the continuous nature of the deep implicit model but by using the learned latent feature instead of the decoded function field.
6 Conclusion
In this paper, we presented a correspondence-fiee rotational registration method for point clouds. The method is built upon the developments in equivariant neural networks and implicit shape repre- sentation learning. We construct a feature space where the rotational relations in Euclidean space are preserved, in which registration can be done efficiently. We circumvent the need for data association and solve the rotation in closed form, achieving low errors independent of initial rotation angles. Furthermore, leverage implicit representation learning and end-to-end registration enhance the network robustness against non-ideal cases where the points are not exactly corresponding.
We conducted experiments synthetic and real-world data, showing the network's 's ability to handle various input imperfections while also motivating future work for improved performance. Further- more, there are two open issues unsolved. The first problem is on generalizing the (033)-equivariant neural network to SE(3) so that the registration of rotation and translation can be solved together. The second problem is on generalizing the registration to handle outdoor LiDAR scans. These problems are left for future studies.
8

---

Acknowledgment
Toyota Research Institute provided funds to support this work. Funding for M. Ghaffari was in part provided by NSF Award No. 2118818.
References
[1] B. K. P. Hom, H. M. Hilden, and S. Negahdaripour. Closed-form solution of absolute orientation using orthonormal matrices. J. Opt. Soc. Am. A, 50)):127-1133,, Jul 1988. 10. 13.4//0AAA,..0000 URL http://josaa. osa. otgg/bastraeeeeeeeeeeeeeeeeeeeee
[2] P. J. Besl and N. D. McKay. method for registration of 3-d shapes. IEEE Trans. Puttern Anal. Mach. Intell., 14())239-256, Feb 1992. ISSN 1939-3539. 1oi:10.1109/34.22111
[3] Y. Aoki, H. Goforth, R. A. Srivatsan, and S. Lucey. PointNeiLK: Robust & efficient point cloud registra- tion using pointnet. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., June 2019.
R. W. Wedderburn. Quasi-likelihood functions, generalized linear models, and the gauss newton method. Biometrika, 61(3):439 447, 1974.
[5] W. Yuan, B. Eckart, K. Kim, V. Jampani, D. Fox, and J. Kautz. DeepGMR: Learning latent Gaussian mixture models for registration. In Proc. European Conf. Comput. Vis., pages 733-750. Springer, 2020.
C. Qi, H. Su, K. Mo, and L. Guibas. Pointnet: Deep learning point for 3d classification and segmentation. arXiv preprint arXiv:1612.00593, 2016.
[7] Y. Wang. Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph can for learning point clouds. ACM Transactions Graphics (TOG), 38(5):1-12 12, 2019.
[8] C. Deng, O. Litany. Y. Duan, A. Poulenard, A. Tagliasacchi, and L. Guibas. Vector neurons: A general framework for -quiiaaqiaa networks. arXiv preprint arXiv:2104. /2229, 2021.
[9] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proc. IEEE Conf. Comput. Vis. Pattern Recog,, pages 4460-4470, 2019.
[10] 7. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proc. IEFE Conf. Comput. Vis. Pattern Recog., pages 1912-1920, 2015.
[11] D. Stutz and A. Geiger. Learning 3d shape completion under weak supervision. CoRR, abs/ 805.07290, 2018. URL http://arxiv. arg/abs// awwpg//b///80....m .72990.
[12] Shotton, Glocker, C. Zach, Izadi, Criminisi, gression forests camera relocalization in rgb-d images. In Proc. IEEE Conf. Comput. Vis. Pat- tern Recog. June
Fitzgibbon. Scene coordinate re-
June 2013. URL http:: microsoft. com/ www.au//rreeerrcppppiactaa ccene-corddinate-rrggeessss foreessssoo--orrrtt------------- cemeerererloolllizttiooo--ieeeee
imagges./.
[13] A. Zeng. S. Song, M. Nießner, M. Fisher. J. Xiao, and T. Funkhouser: 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. Proc. IEEE Conf. Comput. Vis. Puttern Recog. 2017.
[14] X. Huang, G. Mei, and J. Zhang. Feature-metric registration: fast semi-superrvsed approach for robust point cloud registration without correspondences. In Proc. IEEE Conf. Comput. Vis. Pattern Recog, pages 11366 10066-113444 11374, 2020.
[15] V. Sarode, registration network using pointnet encoding. ArXiv, abs/1908. htss/998.009966 2019.
Goforth, Aoki, R. A. Srivatsan, Lucey, and Choset. Pernet: Point cloud
[16] Z. J. Yew and G. H. Lee. Rpm-net: Robust point matching using learned features. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 11824-11833, 2020.
[17] A. Censi. An ICP variant using point-to- metric. Proc. IEEE Int. Conf, Robot, and Automation, pages 19 25. IEEE, 2008
[18] Y. Chen and G. G. Medioni. Object modeling by registration of multiple range images. Image Vision Comput., 10(3):155- 1992.

---

[19] N. J. Mitra, N. Gelfand, H. Pottmann, and L. Guibas. Registration of point cloud data from geometric optimization perspective. In Proceedings of the 2004 Eurogruphics/ACM SIGGRAPH Symposium on Geometry Procexsing, SGP '04, page 22-31, New York, NY, USA, 2004, Association for Computing Machinery. 1o::10..145/105444 /4/.11/44/15521055755555
[20] A. Segal, D. Hachnel, and S. Thrun. Generalized-ICP. In Robotics: science and systems, volume 2, page 435, Seattle, WA, 2009.
[21] H. Deng, T. Birdal, and S. llie. Ppfiet: Global context aware local features for robust 3d point matching. In Proc. IEEE Conf. Comput. Vis. Pattern Recog,, pages 195-205, 2018.
[22] Z. Gojeic, C. Zhou, J. D. Wegner, and A. Wieser. The perfect match: 3d point cloud matching with smoothed densities. In Proc. IEEE Conf. Comput. Vis. Pattern Recog,, pages 5545-5554, 2019.
[23] S. Ao, Q. Hu, B. Yang, A. Markham, and Y. Guo. Spinnct: Learning a general surface descriptor for 3d point cloud registration. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2021.
[24] C. Choy, J. Park, and V. Koltun. Fully convolutional geometric features. In Proc. IEEE Int. Conf. Comput. Vis., 2019.
[25] Y. Wang and J. M. Solomon. Deep closest point: Learning representations for point cloud registration. In Proc. IEEE Int. Conf. Comput. Vis., pages 3523-3532, 2019.
[26] C. Choy, Dong, and Koltun. Deep global registration. Proc. IEEE Conf. Comput. Vis. Pattern Recog,, 2020.
[27] G. D. Pais, Ramalingam, V. M. Govindu, J. C. Nascimento, R. Chellappa, and P. Miraldo. Net: A deep neural network for 3d point registration. Proc. IEFE Conf. Comput. Vis. Pattern Recog., pages 7193-7203, 2020.
[28] Li and H. Lee. Usip: Unsupervised stable interest point detection from 3d point clouds. In Proc. IEEE Int. Conf. Comput. Vis., pages 361-370, 2019.
[29] I. Lang, A. Manor, and Avidan. SampleNet: Differentiable point cloud sampling. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 7578-7588, 2020.
[30] Z. J. Yew and G. H. Lee. 3dfeat-net: Weakly supervised local 3d features for point cloud registration. In ECCV, 2018
[31] X. Bai, Z. Luo, L. Zhou, H. Fu, L. Quan, and C.-L. Tai. D3Feat: Joint learning of dense detection and description of 3d local features. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., pages 6359-6367, 2020.
[32] W. Lu, G. Wan, Y. Zhou, X. Fu, P. Yuan, and S. Song. Deepvep: An end-to-end deep neural network for point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12 21, 2019.
[33] Y. Wang and J. M. Solomon. Pmne: Sclf-supervised lcarning for partial-to-partial reg- istration. In H. Wallach, H. Larochelle, A. Beygelzimer, and R. Garnett, editors, Advances Neural Information Processing Systems 32, pages 8814-8826. Curran Associates, Inc., 2019. URL http://paperrs http://appprr.nin..com nips wwtp:erreipp...p.ppppeeeeeepeeee cc/paper/ 9085-
FAlché-Buc, E. Fox,
.df..
[34] S. Gold, A. Rangarajan, C.-P. Lu, Pappu, and E. Mjolsness. New algorithms for 2D and 3D point matching: Pose estimation and correspondence. Pattern recognition, 31(8):0019-103 1998.
[35] S. Granger and X. Pennec. Multi-scale EM-ICP: A fast and robust approach for surface registration. In Proc. European Conf. Comput. Vis., pages 418-432, 2002.
[36] M. Ghaffari, W. Clark, A. Bloch, R. M. Eustice, and J. W. Grizzle. Continuous direct sparse visual odometry from RGB-D images. In Proc. Robot: Sei. Syst. Conf., Freiburg, Germany, June 2019.
[37] R. Zhang, T.-Y. Lin, C. E. Lin, S. A. Parkison, W. Clark, W. Grizzle, R. M. Eustice, and M. Ghaffari. A new framework for registration of semantic point clouds from stereo and RGB-D cameras. Proc. IEEE Int. Conf. Robot, and Automation, pages 1-8, 2021.
[38] T. Cohen and M. Welling. Steerable arXiv preprint wwwiv:1612..0...0 2016.
[39] C. Esteves, C. Menn-Blacchette, A. Makadia, and K. Daniilidis. Learning so (3) equivariant representa- tions with spherical cans. In Proc. European Conf. Comput. Vis., pages 52 68, 2018.
10

---

[40] F. B. Fuchs, D. E. Worrall, V. Fischer. and M. Welling. Se (3)-trasfformmrs: 3d roto-translation equivariant attention networks. arXiv preprint arXiv:2006. 20t.:2000.115003.. 2020.
[41] R. Kondor and S. Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference Machine Learning, pages 2747-2755 PMLR, PMLR, 2018.
[42] M. Finzi, S. Stanton, P. Izmailov, and A. Wilson. Generalizing convolutional neural networks for equiv- ariance to Lie groups on arbitrary continuous data. International Conference on Machine Learning. pages 3165 3176. PMLR, 2020.
[43] T. Cohen M. Welling. Learning irreducible representations commutative groups. Inter- national Conference on Machine Learning, pages 1755-1763. PMLR, 2014.
[44] G. Chirikjian and A. B. Kyatkin. Engineering applications of noncommutative harmonic analysis: with emphasis on rotation and motion groups. CRC Press, 2001. doi: 11t....01///1/1/1111 1220041667.
[45] G. Chirikjian. Stochastic models, information theory, and Lie groups, volume' 2: Analytic methods and modern applications, volume 2. Springer Science & Business Media, 2009.
[46] B. Hall. groups. Lie algebras, and representations: an elementary introduction, volume 222. Springer, 2015.
[47] Park, P. Florence, J. Straub, R. Newcombe, and Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. IEEE Conf. Comput. Vis. Puttern Recog, pages 165-174, 2019.
[48] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proc. Luropean Conf. Comput. Vis., pages 405-421. Springer, 2020.
[49] J. Huang. S.-S. Huang. H. Song, and S.-M. Hu. Di-fusion: Online implicit 3d reconstruction with deep priors. arXiv preprint arXiv:2012.0555,, 2020.
[50] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez. P. Isola, and T-Y.. Lin. inerf: Inverting neural radiance fields for pose estimation. arXiv preprint arXiv:2012.05877, 2020.


Reviews:
[{"review_id": "bfkLOqKfujK", "reviewer": "Reviewer_nLiN", "summary": "This paper presents a method for registering a pair of point clouds without correspondences (data association). The key idea is to use the so-called vector neurons to embed the original 3D point clouds into a 3D feature space that is equivariant to 3D rotations, and invariant to permutations. Precisely, if the original point cloud is rotated by an arbitrary rotation and permuted by an arbitrary permuation, then the learned features will be rotated by the same rotation, but will not be affected by the permutation. Using such an architecture, estimating the rotation can be performed in feature space using Horn's method in closed form.\n\nTo train the features, the paper proposes to use an encoder-decoder architecture with the encoder being the vector neurons and the decoder being an implicit shape representation for computing the losses more robustly. The proposed method is tested on ModelNet40, compared with two baselines, and shown to achieve generally better performance in the presence of noise and density variances.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The SO(3)-equivariant network using vector neurons is a very interesting idea. I like this idea because, as written in the summary, it discards the permutation/ordering of the point cloud, but preserves the rotation so that registration can be done in the feature space.  In some sense, I think the network can be seen as simultaneously learning keypoints and correspondences, because the output features of the network, Q and Q' as in eq. (3), can be seen as a set of corresponding keypoints, where the keypoints do not necessarily belong to the original point cloud (kind of being hallucinated by the network). But I am OK with the authors calling this method \"correspondence-free\".\n\n- This paper is very easy to understand, with very few typos. The preliminaries and methodology sections well explain the key novelty of the paper.\n\nWeaknesses\n- The major weakness of this paper is that experiments are too toy and simple to convince me that the proposed method is useful for any practical robotics or computer vision applications. The authors write in conclusions that \"generalizing to deal with translation\" and \"generalizing to large-scale problems\" are two directions of future work. However, I cannot agree with accepting a paper which proposes a new method without extensive evaluation on practical problems, and I think those two generalizations must be done before this paper can be accepted.  \n- To be more specific, firstly, the ModelNet40 dataset is a toy dataset that only contains single objects without complex cluttered scenes typically encountered in real applications. Second, the experiments in this paper do NOT consider point clouds with partial overlapping, which is too ideal to be useful. In fact, Table 3 shows that the proposed method already gives poor performance under different densities. So I have reason to suspect this method will perform even worse under partial overlapping and cluttered scenes. Third, I think it is also important to compare the proposed method to correspondence-based methods.  Indeed, [Ref1] (see Table 1) shows that strong correspondence-based approaches like TEASER [Ref2] already achieves almost 100% recall on ModelNet40 and the problem without partial overlapping is considered more or less a solved problem.  Therefore, I suggest to test on the more realistic 3DMatch dataset [Ref3], with comparison to more baseline approaches to demonstrate the usefulness of the proposed method.\n- I think Section 3.2 Deep implicit representation learning should be presented in more details for the reader to fully understand how the loss function is computed.\n\nTechnical suggestions\nI want to provide a few known ideas that may help the proposed method.\n- It is shown in [Ref2] that robust point cloud registration can be solved in a decoupled fashion (scale, rotation and then translation) by designing invariant measurements. Therefore, the idea in [Ref2] already gives one approach for generalizing the proposed method to handle translations.\n\n- As I mentioned, the proposed method can be seen as simultaneously learning keypoints and correspondences. Therefore, using this correspondence-based perspective, it is straightforward to apply robust estimation methods to register the learned features. Precisely, currently the authors use Horn's method to estimate rotation from Q and Q' in eq. (3). However, Horn's method is known to be sensitive to outliers (wrong matches in Q and Q'). So I believe the authors could try using a robust method to estimate R from Q and Q', such as RANSAC (available in Open3D) or TEASER [Ref2], which perhaps can help get more accurate estimates.\n\n\n[Ref1] Yuan, Wentao, Benjamin Eckart, Kihwan Kim, Varun Jampani, Dieter Fox, and Jan Kautz. \"Deepgmr: Learning latent gaussian mixture models for registration.\" In European Conference on Computer Vision, pp. 733-750. Springer, Cham, 2020.\n\n[Ref2] Yang, Heng, Jingnan Shi, and Luca Carlone. \"Teaser: Fast and certifiable point cloud registration.\" IEEE Transactions on Robotics 37, no. 2 (2020): 314-333.\n\n[Ref3] Zeng, Andy, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. \"3dmatch: Learning local geometric descriptors from rgb-d reconstructions.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1802-1811. 2017.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this paper proposes an interesting and novel technical idea that may have good impact.\n\nHowever, the experiments are significantly lacking to convince me the practical value of the proposed method. In its current form, the paper cannot be accepted. \n\n**Post rebuttal**\n\nThe authors added new experiments on 7Scenes, which addressed my major concern. Although the experiments are still not particularly strong, considering the idea of this paper is interesting, I'd like to increase my score to Weak accept.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See weaknesses.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}, {"review_id": "QA02oluHja-", "reviewer": "Reviewer_sqvt", "summary": "This paper proposes to do point cloud registration in learned feature space. The features are extracted using a neural network that retains SO(3)-equivariance in points' features. Then the registration is done in the feature space and then propagated to every point. Experiments are done on the point clouds of the synthetic dataset ModelNet40. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\n- The idea of doing point cloud registration in feature space is novel. This could be inspiring for the rest of the research community.\n- The experiment results seem to be strong and outperform the baselines by a significant margin.\n\nWeakness:\n- The extracted deep features could be very unstable, especially when there is occlusion in the point cloud.\n- The authors only evaluated two simple real-data effects: random noise in the point cloud and different densities. However, these two cases are both way too simple and easy, because if the point occupancy grid is used as representation, both cases do not change occupancy much. One very important experiment is missing: registration of $\\textit{partial}$ 3D point scans, especially when the two scans have different partial pieces. This is a case where the proposed algorithm can fail completely because partial scans can dramatically change the deep features, therefore the registration step in the feature space can fail easily. The authors have to evaluate this case to support their claim in the paper.\n- Since the authors used occupancy grid as point representation, the proposed method is unable to deal with large-scale point scans such as depth scans of the entire room, because the grid resolution will be severely limited.\n- What is the loss function used to train the deep features? This is not clear from the paper.\n\nSome other issues on the paper:\n- Fig 2. visualization of equivariant features is unnecessary. \n- The related work section is too long. Most of the sentences in this section can be more concise.\n- \"Occupancy Network\" is first mentioned in line 108, but is not cited until line 141. This is very confusing.\n- In Fig 1., the author mentioned the output to be \"Occupancy Field\", but where is it defined in the text?\n- How is \"Rotation Error\" defined in Tables 1, 2, and 3? Is it rotation angle difference, in terms of radians or degrees? If so, why isn't rotation axis difference evaluated? Or it is some combination of rotation axis and angle? It's not clear from the paper.\n \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I'm not convinced that the proposed approach is actually useful, especially because the authors used point occupancy as representation. This results in the proposed approach being unable to deal with large-scale scans.\n\nBesides, the presentation of this paper is very poor. A lot of necessary details are missing from the paper and are very hard to follow. It seems that this paper is written in a rush without careful check of logic and soundness.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "The authors can address the items under \"Weakness\" and \"Some other issues\" in my main review section.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "CiQ3syUQGR", "reviewer": "Reviewer_ZnCf", "summary": "This paper describes a correspondence-free method for pointcloud registration.  The main idea is to learn an SO(3)-equivariant (and permutation-invariant) mapping from the (entire) input pointcloud into an abstract feature space.  The fact that the learned feature mapping is SO(3)-equivariant means that the desired rotation R aligning the pointclouds can be gotten as the optimal rotation aligning their *features*; this latter problem admits a simple direct closed-form solution using Horn’s method.\n\nIn detail, the paper shows how to “upgrade” the (permutation-invariant) PointNet to be SO(3)-equivariant by replacing its layers with the [SO(3)-equivariant] Vector Neurons proposed in reference [5].  This modified PointNet is then used as the encoder in an encoder-decoder network (based on OccupancyNet) that is designed to learn a (global) implicit shape descriptor for the pointcloud.  The idea behind the use of an implicit shape representation is to account for small nuisance variations in the sampled pointclouds (due to both measurement noise, and the fact that in general two scans of the same object will not sample *exactly* the same sets of points on the object).  The overall result is a network design that learns an implicit shape representation that is SO(3)-equivariant *by construction*.\n\nExperimentally, the paper evaluates the proposed method in comparison with PCR-Net and Feature-Metric Registration (FMR) [two correspondence-free deep pointcloud registration methods] on three tasks requiring the registration of pairs of pointclouds sampled from CAD models taken from the ModelNet40 dataset.  Overall, the results show that (as expected) the accuracy of the registrations computed by the SO(3)-equivariant approach proposed in this work is *independent* of the magnitude of the rotation aligning the input pointclouds.  This is *not* true for PCR-Net and FMR, whose performance significantly degrades with increasing alignment angle because they make use of (local) iterative refinement to align the input pointclouds.  In absolute terms, the proposed method performs very well in the presence of large-magnitude rotations and Gaussian measurement noise, although its performance does seem to degrade significantly when comparing two pointclouds of differing density.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "My overall impression of this work is very positive: the problem under consideration is certainly well-motivated, the proposed approach is very elegant, and the empirical results show that it achieves the desired result [namely, SO(3)-equivariance] -- as in fact it must, since the proposed method inherits this property *by construction*.  I would be very interested in using this method in my own work (which I think is about the highest praise a reader can bestow).  \n\nThe paper as a whole is also remarkably well-written; I particularly appreciated its clear and concise review of equivariance and Vector Neurons (the latter of which I had not seen before) and the thorough review and contextualization of related work in Section 5.\n\nIn my view the paper’s only substantial weakness (which it already acknowledges) is the significant degradation in performance in aligning pointclouds of different sampling densities.  I do not completely agree with the paper’s assertion (in lines 179-180) that pointcloud registration errors of ~20 degrees are “practical” -- I can easily imagine applications (e.g. object manipulation or localization) where such errors would be too large for the returned result to be practically useful.  [I would also expect this different-density scenario to arise fairly regularly, since it simply corresponds to scanning the same object from two different distances.]  Nevertheless, I still think that the SO(3)-equivariant design reported in this work is a very solid technical contribution that is likely to be of interest to the robotics community, despite this specific shortcoming.\n\nA couple of other minor comments:\n\n* I notice that in the results reported in Table 5.3, FMR seems to perform very well as long as the initial rotational offset is less than ~30 degrees.  Given that the proposed method is capable of generating registrations accurate to within about 20-25 degrees across *all* initial offsets, perhaps one way of improving robustness in the differing-densities case would be to treat the proposed method as an *initialization*, and then apply a subsequent (finer) refinement using FMR or something similar?  (Admittedly this lacks the elegance of having a single end-to-end SO(3)-equivariant method, but as a practical approach it might be very effective at addressing the (uniformly) higher errors reported in the differing-density case.)\n\n* In lines 104-106, the paper mentions that this method is among the first to address the problems of measurement noise and inexact correspondences between the input pointclouds in deep correspondence-free registration methods.  If this is correct, I would encourage the authors to emphasize this point more strongly (e.g. in the abstract and introduction), as it strikes me as an additional significant feature of the method.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation for accepting this paper is based upon the following considerations:\n\n* It addresses a fundamental problem in robotic perception (pointcloud alignment), which is likely to be of interest to a very broad subset of the community\n* It develops a solution to this problem that provides a novel set of capabilities (DNN-based correspondence-free poincloud registration that is *also* SO(3)-equivariant) with respect to prior work\n* The design process that leads to the proposed method is both mathematically rigorous and technically very interesting, and is based on principles [e.g. geometric analysis of the design of feature maps, equivariance, etc.] that can be generalized far beyond the specific problem and algorithm considered in this work.  I expect that many readers would find this independently interesting, and I could easily imagine that the presentation of these ideas in a nice \"case study\" like this paper could catalyze their application to many other problems.\n\nFinally, I want to mention that I rated this paper's impact at \"3\" only because the criteria for 4 strikes me as unreasonably high (even the fraction of *published* papers that have \"a major impact in robotics or machine learning\" is vanishingly small -- well under 5%), and there was no intermediate rating between \"incremental\" and \"major\".  This should in no way be construed as a criticism of the paper's merit -- it's a very fine contribution, and I would like to see it accepted.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "The only issue that I would specifically encourage the authors to address during the response and revision period is to try to improve the performance of the method on the differing-density case: I think this would greatly strengthen the paper, as well as (more importantly) making the method significantly more useful."}}, {"review_id": "AAivyT6wUZG", "reviewer": "Reviewer_b5A4", "summary": "The paper proposes a correspondence-free Deep Neural Network (DNN)-based method for point cloud registration. The network exploits the \"Vector Neurons\" architecture, a SO(3)-equivariant DNN for point cloud processing, to extract a global feature vector from an input point cloud. A decoder network is used to reconstruct the original input from the global feature and a query position. Finally, the rotation between two point clouds is estimated by comparing their global feature vectors in a closed-form solution.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper is well written, well organized, and in general easy to read.\n+ The proposed approach is very interesting, and addresses a relevant problem.\n+ Different methods for point cloud registration are well presented and discussed.\n+ The results show great improvement over existing correspondence-free DNN-based approaches, especially when the initial rotation between the point clouds is large.\n\nWeaknesses:\n- The paper does not provide a strong argument or evidence in support of using correspondence-free over correspondence-based methods. The argument that \"matching (point) pairs may not exist in the input point clouds\" (line 210) is weak, since robust estimators such as RANSAC can effectively overcome this issue. Recent correspondence-based point cloud registration methods [1, 2, 3] that combine DNN-based feature extractors with RANSAC achieved outstanding results and robustness to initial misalignment. A strong evidence, such as a comparison of the proposed approach with recent correspondence-based methods, or at least a strong argument on why correspondence-free methods should be preferred over correspondence-based ones, should be provided.\n- The proposed method only estimates the rotation between the two point clouds, while most existing approaches estimate both rotation and translation.\n- The proposed method is evaluated only on the synthetic ModelNet dataset, an experimental evaluation on a real-world dataset, such as 3DMatch or KITTI/KITTI-360, would give a better insight into the possible applications of the approach.\n\n[1] Bai, X., Luo, Z., Zhou, L., Fu, H., Quan, L., & Tai, C. L. (2020). D3feat: Joint learning of dense detection and description of 3d local features.\n\n[2] Cattaneo, D., Vaghi, M., & Valada, A. (2021). LCDNet: Deep Loop Closure Detection and Point Cloud Registration for LiDAR SLAM.\n\n[3] Huang, S., Gojcic, Z., Usvyatsov, M., Wieser, A., & Schindler, K. (2021). PREDATOR: Registration of 3D Point Clouds with Low Overlap.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the proposed method is interesting, a strong evidence supporting the claim that correspondence-free methods should be preferred over correspondence-based ones is missing. Therefore, my recommendation is weak reject.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- A strong evidence, such as a comparison of the proposed approach with recent correspondence-based methods, or at least a strong argument on why correspondence-free methods should be preferred over correspondence-based ones, should be provided.\n- An experimental evaluation on a real-world dataset, such as 3DMatch or KITTI/KITTI-360, would give a better insight into the possible applications of the approach."}}, {"review_id": "A6jFSn_Xkag", "reviewer": "Reviewer_4hsK", "summary": "This paper proposes a new encoder-decoder approach for correspondence-free point cloud registration with rotations. The encoder is a combination of PointNets which provides permutation invariance and the recently proposed Vector Neurons which provides SO3 equivariance. The decoder is an implicit function (occupancy network) that takes an encoding and a query position and predicts the occupancy value. Two point clouds are registered by treating their encodings as a pair of pseudo point clouds that are already pairwise matched, so that the registration problem admits a closed form solution.  Compared to two other baselines, when the initial rotational difference is large, the proposed method shows significantly lower error.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n \n- The paper is generally well written and proposes a novel point cloud rotational registration method.\n- The improvements in rotational error seem quite significant particularly for large initial rotation angles.\n\nWeaknesses\n\n- Many details on network architecture and training are omitted and delegated to the appendix, making some sections a bit hard to follow.\n- In Table 2 and Table 3, the error seems higher than the baselines for small initial rotation angles. How can this be fixed?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Point cloud registration has many applications in Robotics. Though a robotics application is not shown in this paper, the network design is particularly interesting in the way it addresses permutation and rotational invariance, and builds on implicit shape representations. The improvements seem convincing overall. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- Figure 1 and its caption could be improved to be more informative. \n- Overall, some technical details on architecture design are missing section 3.\n- Modern autodiff engines allow Eqn 4 to be implemented and treated as a \"layer\" in a network. It was not clear whether the feature alignment step is part of the training or outside it?\n- In Table 2 and Table 3, the error seems higher than the baselines for small initial rotation angles. How can this be fixed?"}}]

Author rebuttal:

Meta-review: The paper has obtained 5 expert reviews, which is above average for CORL and will hopefully provide useful inputs to the authors. 

The reviewers appreciated the following aspects: The paper is well written, proposes a novel and elegant registration method, and the results confirm such a method has lower errors under large rotations. The reviewers particularly appreciated the idea of mapping the problem into a rotation-equivariant and permutation-invariant feature space where registration becomes straightforward.

The main weaknesses are:
- For the paper to have an impact on the robotics community, the experiments should include evaluations on real-world datasets (such as 3DMatch and KITTI), and evaluate on partial/occluded point clouds. It is also recommended to include comparisons against correspondence-based methods (LCDNet, PREDATOR, TEASER or similar). 
- The approach seems to have slightly higher errors (compared to baselines) when the initial rotation angles are small. While this is not necessarily a problem, a discussion on why this is the case and how to resolve this issue is desirable. Similarly, the performance of the approach quickly degrades with the sampling density, which may limit real-world applicability. 
- The approach may not scale to large scenes due to the discretization and the size of the grid map. A discussion about scalability is in order.
- The discussion is hard to follow in some points (e.g., details on the architecture design in Section 3). This point should be easy to address.

*Post-rebuttal comments*:
The reviewers and area chair have carefully read the revised paper and rebuttal and there has been an interesting discussion about the merits of the paper. The reviewers mostly agree that the paper has an interesting contribution and advances the literature on deep-learning-based correspondence-free methods. The reviewers and AC also appreciated the additional experiments: they definitely strengthened the paper. On the downside, one reviewer is still concerned that the 7scenes evaluation is not fully satisfactory since it might still hide issues in the registration of point clouds with low overlap. While I share the same concern, I think no paper is perfect and I consider this paper interesting for the CoRL audience. A second issue that I consider very important is to provide a more transparent contextualization with respect to correspondence-based methods: while I agree that learning-based correspondence-free methods are less mature, it is important for the paper to acknowledge that, and to provide suitable pointers to the non-expert reader. Similarly, I would have liked to see a comparison against non-learning-based correspondence-free methods, such as ICP and variants. Since these are relatively minor changes, I trust the authors will add a few sentences to clarify these points in the final submission, therefore I vote to accept this paper. 

Minor comments:
- In the literature review, including ICP in the "correspondence-based" methods feels wrong. Correspondence-based methods take the correspondences as input or establish correspondences using descriptors. It should be included in the correspondence-free methods instead.
- Please rephrase the sentence “To our knowledge, we are one of the first to report registration results under this setting in correspondence-free methods” to mention “learning-based correspondence-free methods” (going back to the previous point, ICP has been extensively used for correspondence-free registration).




Decision: Accept (Poster)