3D Meta-Registration: Meta-learning 3D Point Cloud Registration Functions
Authors: Yu Hao
Yi Fang
Venue: ICLR 2022 Submitted (2022)

Abstract: Learning robust 3D point cloud registration functions with deep neural networks has emerged as a powerful paradigm in recent years, offering promising performance in producing spatial geometric transformations for each pair of 3D point clouds. However, 3D point cloud registration functions are often generalized from extensive training over a large volume of data to learn the ability to predict the desired geometric transformation to register 3D point clouds. Generalizing across 3D point cloud registration functions requires robust learning of priors over the respective function space and enables consistent registration in presence of significant 3D structure variations. In this paper, we proposed to formalize the learning of a 3D point cloud registration function space as a meta-learning problem, aiming to predict a 3D registration model that can be quickly adapted to new point clouds with no or limited training data. Specifically, we define each task as the learning of the 3D registration function which takes points in 3D space as input and predicts the geometric transformation that aligns the source point cloud with the target one. Also, we introduce an auxiliary deep neural network named 3D registration meta-learner that is trained to predict the prior over the respective 3D registration function space. After training, the 3D registration meta-learner, which is trained with the distribution of 3D registration function space, is able to uniquely parameterize the 3D registration function with optimal initialization to rapidly adapt to new registration tasks. We tested our model on the synthesized dataset ModelNet and FlyingThings3D, as well as real-world dataset KITTI. Experimental results demonstrate that 3D Meta-Registration achieves superior performance over other previous techniques (e.g. FlowNet3D).

Under review as a conference paper at ICLR 2022
3D META-REGISTRATION: META-LEARNING POINT CLOUD REGISTRATION FUNCTIONS
3D
Anonymous authors Paper under double-blind review
ABSTRACT
Learning robust 3D point cloud registration functions with deep neural networks has emerged as a powerful paradigm recent years, offering promising perfor- mance in producing spatial geometric transformations for each pair of 3D point clouds. However, 3D point cloud registration functions are often generalized from extensive training over large volume of data to learn the ability predict the desired geometric transformation to register 3D point clouds. Generalizing across 3D point cloud registration functions requires robust learning of priors over the respective function space and enables consistent registration in presence of signif- icant 3D structure variations. In this paper, we proposed to formalize the learning of a 3D point cloud registration function space as a meta-learning problem, aiming predict 3D registration model that can be quickly adapted to new point clouds with no or limited training data. Specifically, we define each task as the learning of the 3D registration function which takes points in 3D space as input and predicts the geometric transformation that aligns the source point cloud with the target one. Also, we introduce an auxiliary deep neural network named 3D registration meta-learner that is trained to predict the prior over the respective 3D registration function space. After training, the 3D registration meta-learner, which is trained with the distribution of 3D registration function space, is able to uniquely param- eterize the 3D registration function with optimal initialization to rapidly adapt to registration tasks. We tested our model on the synthesized dataset ModelNet and TriiingT,,,,,D,, as well as real-world dataset KITTI. Experimental results demonstrate that 3D Meta-Registration achieves superior performance other previous techniques (e.g. FlowNet3D).
INTRODUCTION
The point cloud registration is defined as a process to determine the spatial geometric transforma- tions (i.e. rigid and non-rigid transformation) that can optimally register the source point cloud towards the target one. In comparison to classical registration methods Besl & McKay (1992); Yang
let al. (2015): Myronenko et al. (007), learning-based registration methods LiahoddsLiu et al. (l.999))); Balakr- ishnan et al. (2018) usually leverage neural network- based structure to directly predict the desired transformation for given pair of source and target point clouds.
Recently based on the PointNet Qi et al. (2017a) structure, Liu et al. proposed FlowNet3D Liu al. 2019) to learn the points flow field register two point clouds together. Balakrishnan et al. proposed VoxelMorph Balakrishnan et al. (018)) for aligning two volumetric 3D shapes. These methods achieved impressive performance for the registration task of 3D shapes/scenes. In com- parison to iterative registration methods, learning-based methods have advantages in dealing with a large number of datasets since learning-basse methods can transfer the registration pattern from one dataset to another one. However, there are two main challenges for the learning-based methods. Firstly, sarning-based networks often require large volume of data and a long learning period to acquire the ability to predict the desired geometric transformation register 3D point clouds. Secondly, The generalization capacity can be greatly degraded the distribution of the dataset in practice differs from the training dataset.
As shown in Figure T previous learning based methods tend to learn a single 3D registration func- tion (learnrr) for any pair of source and target point clouds. In this way, the 3D registration leamne

---

Under review as a conference paper at ICLR 2022
Previous Learning
Our
Methods
Target Clouds
Source Clouds
Registration Learner 1
Aligmment Loss
Target Clouds
Source Clouds
Registration Learner
Target Clouds
Source Cloads
Registration Meta-
Registration Learner 2
Aligmment Los 2
Learner
Turget
Source
Registration Learner n
Aligument Loss
Allgmment
Clouds
Clouds
Tn
Geoometric Transform
Figure I: Comparison of the pipeline between previous learning methods and our solution for point cloud registration. Previous learning- Barning-baseed methods regard 3D registration as a task for the given point clouds. In comparison, we define each task learning unique 3D registration function (learner) for each pair of given 3D objects.
performs well from extensive training on a large number of labeled data. In contrast, we proposed to formalize the learning of 3D registration function space as meta-learning problem. More specif- ically, we define each task as learning unique 3D registration function (learner) for each given 3D> object. Besides, prior over the respective 3D registration function space. The meta-learner is responsible for pro- viding the optimal initialization of 3D registration learner. In general, our meta-learning- -based approach gains competitive advantages over existing generalization methods that our method can uniquely parameterize the 3D registration function for each pair of shapes provide 3D point cloud registration.
design auxiliary deep neural network
meta-learner that predict the
As shown in Figure our model includes modules: 3D registration learner and 3D registration meta-lcarner. We observe that different regions of a 3D shape vary in their geometric structures which makes it more sense that we have region-conditioned transformation instead of the shape- conditioned one. In this way, the proposed 3D registration learner includes shape embedding and region aware flow regression which uses multiple non-linear multi-layer perceptron (MLP)-based function to predict the transforms and weights for different regions respectively. The 3D registration meta-learner includes two parts. The first part a variational auto-encoder that maps shape repre- sentations to a distribution over 3D registration tasks, which gives the priors over the 3D registration function space and the second part is to sample from the 3D registration function space to predict optimal initialization for the parameters of the 3D shape registration function. Our contributions are listed as below:
In this paper, to the best of knowledge, it is the first time to formalize learning of 3D point cloud registration function space as a meta-learning problem 3D computer vision. Under this circumstance, the 3D registration model can be quickly adapted to new point clouds with no or limited labeled training data.
In this paper, we propose a novel variational encoder that maps shape representations to distribution 3D registration tasks, contributing robust learning of priors over the 3D registration function space.
In this paper, we observe that different regions of 3D shape vary in their geometric struc- tures which enables us to propose a region-aware flow regression module instead of the shape-conditioned one.
In this paper, compared 3D Meta-Registrane to other ofate-of-the-an ones widely used benchmark datasets and demonstrated superior registration performance over both seen and unseen data.
2

---

Under review as a conference paper at ICLR 2022
Source nx
3D Registration Meta learner
Sampling
Tarpet n'
Embedding
Task Distribution Estimator
Learner's Initialization Predictor
Sourcee
3D Registration Learner
Conv
Set
Conv
Set
(2
How nX3
Fbw
Fuse
Tarpe: n> 3
nx
PlW
Shape Embedding
Region Aware Flow Regression
Figure 2: Main Pipeline. Our model includes two networks: 3D registration learner and 3D reg- istration meta- beta-learner. The registration learner includes shape embedding and region aware flow regression. 3D registration meta-learner include task distribution estimator and learner's initializa- tion predictor. Pn denotes the n-th region probability score.
RELATED WORKS
2.1 POINT CLOUD REGISTRATION
In comparison to classical methods Besl & McKay [9922; Yang et al. (2015): Myronenko et al. 2007), learning-bai based methods have significant advantages in dealing with large number of datasets by transferring the "knowledge" from registering training pairs to testing pairs. Based the feature learning structure proposed by PointNet Qi al. (2017a), Aoki et al. Aoki et al. 2019 proposed PointNetLK for rigid point cloud registration by leveraging Lucas & Kanade algo- 2019), Wang al. proposed Deep Closest Point Wang & Solomon (2019a) for learning rigid point cloud registration and PR-Net Wang & Solomon (001bb for learning partial shapes registration. For non-rigid point cloud registration, Liu al. proposed FlowNet33 Liu et al. (2019) to learn the points flow field for non non-rigid point cloud registration. Wang et proposed FlowNet3DD++ WWangg Wang et al. (2020) on top of Net3N by adding geometric constraints and angular alignment to dramatically improve the alignment performance.
rithm. Based DGCNN Wang
2.2 META-LEARNING METHODS
Meta-learning Andrychowicz et al. (2016) Schmidhuber/ (1992): Hospedales et al. (2000) refers to subfield of machine learning that learning new concepts and skills much faster and more efficiently given only small amount of dataset. Parameters prediction Finn et (001 2018 Lee & Choil (0188) is one of the strategies in meta-learning, which refers to a network trained to predict the parameters of another network so that the first network can encode the related information to the second network which makes the overall network more flexible and adaptive to a particular task. Recently, meta-learning approaches are widely used in computer vision tasks. MANN was proposed by Santoro et al. [Santoro et al. (2016) to use explicit storage buffer which is easier for the network to rapidly incorporate new information and not to forget in the future. Ravi al. Ravi Larochelle (016) use the LSTM to learn update rule for training neural network in few-shot learning. Snell et al. Snell et al. (2017) proposed Prototypical networks for few-shot classification task which map the sample data of each class into metric space by calculating the euclidean distance of prototype representations each class. In the field of 3D computer vision, Littwin et al. Littwin Wolf (209)) firstly deep neural network to map the input point cloud to the parameters of another
3

---

Under review as a conference paper at ICLR 2022
network for the 3D shape representation task. Yang al. Yang et al. (2020) proposed Meta3D which an external memory is used to store image features and their corresponding volumes. In this paper, first propose meeaaleerning based method with 3D registration meta-learner, which can learn to learn the registration pattern more efficiently.
3 METHODS
We introduce our approach in the following sections. In section 3.1 we state the problem of learning- based registration. Section 3.2 illustrates the 3D registration learner. The 3D registration meta-
leamer is explained in section 33
The definition of the loss function is discussed in section 3.4
3.1 PROBLEM STATEMENT
We need to define the optimization task firstly. For given training dataset D {(S,, G.)), where S;, G2 C R S, is the source point cloud and G2 the target point cloud. We as- sume the existence of parametric function (e(S,, G,) ø using neural network structure, where is the transformation function (point flow this paper) which deforms source point cloud towards the target point cloud. We call go 3D registration learner in this paper and 0 is the set of weights in the 3D registration learner. For previous learning-ber based network structure, the 0 is optimized using stochastic gradient descent based algorithm for given dataset:
Opptimal arg min BrsmiinBss....DD((S... go, G.))]]
(1)
where L represents similarity measure.
In comparison. assume that the 3D registration learner includes two of parameters: and Om. 0, is pre-trained from training dataset, but 0m is predicted by another parametric function fo which is called 3D registration meta-learner in this paper. Similarly, we have the desired transfor- mation function @ 0(m). (S,, G:) and we have On (S, Gt). For given training data set, we have:
g goptimal
optimal
arg 6,00
Gr C100 G.))I]
3.2 3D REGISTRATION LEARNER
The 3D registration learner includes two modules: shape embedding 3.2.1 and region aware flow
regression 3.2.2. We discuss them the following subsections.
3.2.1 SHAPE EMBEDDING
For a given pair of input point clouds, we firstly learn to extract the shape features that captures representative and deformation-insensitiv geometric information. Let (Se, G2) denotes the source and target point clouds and S, G2 C Rे Yæ E S,, we denote the feature of æ as bg E R Following the recent architecture from PointNet+ Qi et (b017b) and FlowNet3D Liu al. (2019). the first set convolution 91 3 + IR is non-linear MLP b-based function 91:
Maxpool (1;)) StAzyeS.
(3)
where is predefined distance and Maxpool is element-wise
pooling function.
Vr E S,, we further concatenate the coordinates x with the learned feature lN and we denote it as learn the feature of each point in the target point cloud G,. The
[z, Lz] E [e++3)) Similarly, shape descriptor for source point cloud S, is: {[x, J.leuss and the shape descriptor for target point cloud G; is: [[x, [=]).
4

---

Under review as a conference paper at ICLR 2022
3.2.2 REGION AWARE FLOW REGRESSION
Based on the learned shape descriptors for both source point cloud [[r,,]]ess, and target point cloud {1z, JEEG, from previous section, in this section we introduce two more set convolution structures for point flow regression. We define the second set convolution 92 [ + [
learn the relation information between descriptors of source and target point clouds. 92 non-linear MLP-based function. Vz Ss, we denote relation tensor Pr as:
= Maxpool ((2 (2 My,
(4)
where [.] denotes concatenation.
Based on the learned relation feature {[x,, P=llues, from source and target descriptors, define set of region aware decoders {9s. 1..,,,.. where K the pre-defined region numbers dataset D and f : (c+3) + R is non-learn MI.P-based function.
We note the pre- trained weights in 93 0, and the meta-learned weights in 03 0m The weights for 33 is the element-wise summation of O, and Om. Vx E S,, we have the estimated region probability score & and flow & for each region. Notice that we use a softmax function normalize the output to the estimated region probability score Pz.k- We have:
Pz,ki U2
(3, Pnl)
(5)
Then we use the estimated region probability score Px, as the weight to further balance among the point flow Un for each region. We define the final point flow Vg as:
Vr
Uz,kUz,k
(6)
Therefore, the transformed source shape S for + Ux JaEs
3.3 3D REGISTRATION META-
The 3D registration meta-learn includes two parts: task distribution estimator (3.3. and learner's
initialization predictor 3.3.2) We discuss them in the following subsections.
3.3. TASK DISTRIBUTION ESTIMATOR
We leverage a variational auto-encoder (VAE) network to learn the task distribution from each pair of input shapes. Instead of learning the detailed features of 3D shapes, learn the most general global shape information via VAE. More specifically, we use a multi-layer MLP-based function fi : 3 + R learn the mean and variance of the task distribution for source and target point clouds. We denote the weights in Si as σi. We denote μ as the mean of task distribution space, and a as the standard deviation of task distribution space. V(S,, Gt).
n, f[ Vil)x
We further sample L ~ N(a, o), and I..
3.3.2 LEARNER'S INITIALIZATION PREDICTOR
To enable the 3D registration meta-learner to learn the pattern of registration, we introduce a second multi-layer MIP based architecture f2 R2" - RR," where ur indicates the dimension of all the weights included 01, 02 03. We denote the weights in f2 as σ2. The meta-learned weights in the 3D registration learner are predicted from the following. V(S., Gt), we have:
0 0g = f2((L)
(8)
5

---

Under review as a conference paper at ICLR 2022
Therefore, for any given pair of source and target point clouds, be accordingly adjusted by adding the meta-learned weights of 6m = { registration meta-learner together with the pre-trained weights of Oz.
3D registration leamer can ...,,,, 0% from the 3D
3.4 LOSS FUNCTION
For the target point cloud G2 and transformed source point cloud Sy. we define the loss function in this section. Assuming that we have ground truth flow field {vz) we use the simple L1 loss be- tween predicted flow field and ground truth flow field with cycle consistency term regularization. We note uy as the predicted flow from S to S. The loss function defined as:
1 1
G.)
Elle, aES,
-Il Aullu + o X,
(9)
where denotes the set cardinality and A a pre-defined terms. X2 the balance term for the KL divergence regularization
to balance the two
EXPERIMENT
In this section, describe the dataset and experimental settings in section 44. and section 4.2 In section [4.3 we perform an ablation study on FlyingThingss dataset to demonstrate our model's performance. In section 4.4] we conduct a series of experiments to verify the effect of our model in the registration of unseen data set from different categories on ModelNet40 dataset. We demonstrate the effectiveness of our proposed method on large-scale synthetic dataset (FlyingThings3D) in section 4.5) and in section 4.6 we show our model's generalization capacity on registration of real Lidar scans from KITTI.
4.1 DATASET PREPARATION
ModelNet40: This dataset contains 12311 pre-processed CAD models from 40 categories. For each 3D point object, we uniformly sample 2048 points from surface. For each source shape S; we generate the transformed shapes G, by applying a rigid transformation defined by the rotation matrix which is characterized by rotation angles along the x-y-z-axis, where each value is uniformly sam- pled from [0, 45) unit degree, and the translation which uniformly sampled from [-0.5, 0.5). For the testing dataset of ModelNet40 with noise, we prepare three types of noise added on both source and target point set. To prepare the position drift (P.D.) noise, we applied zero-mean and 0.05 standard deviation Gaussian to each sample from the point set. To prepare the data incompleteness (D.I.) noise, we randomly choose a point and keep its nearest 1536 points from both the source and target point set. To prepare the data outlier (D.O.) noise, randomly add 148 points outliers, sampled from a [-10, 10) x |-10, 10] [-10, 10] uniform distribution.
FlyingTThings31 The Flying Things3D dataset is an open-source collection, which consists of more than 39000 stereo RGB images with disparity and optical flow ground truth. By following the process procedures provided by FlowNet3D, truth using the disparity map and optical map rather than using RGB images.
generate 3D point clouds and registration ground
Another dataset used in this paper is the KITTI scene flow dataset, which consists of 200 training scenes and 200 test scenes. Following previous work FlowNet33, we use the pre- preepreeeseed point clouds data which is generated using the original disparity map and ground truth flow.
KITTI:
4.2 SETTINGS
In our model, the batch size is set to 16 and we use Adam optimizer as our optimizer. For the 3D registration learner, the first set convolution includes 6 MLPs with dimensions of (32, 32, 64, 64, 64, 128). The second set convolution includes 6 MLPs with dimensions of (128, 128, 256, 256, 256, 512). The third set convolution includes 6 MLPs with dimensions of (256,256, 122,,2,66221111 22,256,,266 and region-aware decoders includes fully connected layers with dimensions of (128,3). 3). For the 3D reg- istration meta-learner, we use 3 MLPs with dimensions of (32, 64, 128) with a fully connected layer with dimension (256) for learning the mean of VAE and one fully connected layer with dimension

---

Under review as a conference paper at ICLR 2022
Table 1: Ablation study. End-point-error (EPE) and estimation accuracy(ACC) with 0.05 and 0.1 threshold are used for evaluation of point cloud registration performance to compare our model with different settings the FlyingThings!! dataset.
Models
EPE
ACC(0.05)
ACC(0.1)
Learner with Weight-Settingg
0.1694
25.37%
57.85%
Learner with
0.1503
27.88%
60.22%
Learner with Weight-Setting C
0.1453
28.67%
61.65%
Learner with Weight-Setinn-DD
0.1437
29.92%
62.21%
Table 2: Results on the 20 unseen testing categories of the ModelNet40 dataset with noise.
Method
Noise
EPE
ACC (0.05)
ACC (0.1)
w/o meta-lcarner
0.3648
10.01%
45.18%
w/ meta-leamnr
0.2342
34.01%
75.56%
w/o meta-lcarne
P.D.
0,4392
6.75%
34.47%
w/ meta-leamer
P.D.
0.3128
20.92%
61.30%
w/o meta-lcarner
D.I.
0,3749
11.67%
46.07%
w/ meta- leammer
D.I.
0,2689
24.04%
65.84%
w/o meta-learner
D.O.
0.5738
4.81%
24.05%
w/ meta-leamer
D.O. 0.4156
10.49%
40.91%
(256) for learning the standard deviation of VAE. Then we have three fully connected layers with dimensions of 128, 512, 2048, and the number of all parameters in the convolution layers. We use the ReLU activation function and implement batch normalization for every MLP layer except the last output layer. We the learning rate as 0.001 with exponential decay of 0.7 every 20000 steps. For the balance terms in loss function, X1 is set to 0.3 and X2 is set to in this paper.
For evaluation point cloud registration performance, we use 3D endpoint error (EPE) and point cloud registration estimation accuracy (ACC) with different thresholds as introduced in Flow The 3D EPE measures the 1.2 distance between the point cloud registration estimation and ground truth. A lower EPE value indicates better estimation of the point flow field. ACC measures the portion of the points with estimated flow field error which less than one selected threshold among all the points.
4.3 ABLATION STUDY
Experiment setting. For this experiment, we use the FlyingThings33 dataset for demonstration. We test four settings of model for comparison of different weight generation methods. In the weight-setting-A, without our meta-learning strategy, the network parameters of the 3D registration learner are optimized from the training dataset. In the weight-setting-B the network parameters of the 3D registration learner are meta-learned by the 3D registration meta-learner. Note that in the weight-setting-B,
directly use MLPs learn the function prior which further send the prediction of the learner as input to predict the parameters of the 3D registration Learner. In the Beight-setting-CC instead of directly using the MLPs map the input shape the function prior, we add variational auto-encoder( VAB) to learn the task distribution of the input 3D shape, which can be further sampled as the function prior over task distribution for the learner predictor to estimate the optimal weights of the 3D registration learner. In the weight-setting-D D, use the region aware decoder to estimate the probability score and point flow for each region which is further used to weighted fuse the multiple transformations to get the final point flow. The quantitative results demonstrated in T
Results. From the first two rows, we notice that the results of weight-setting-B with our meta learning strategy are better than the results of weight-setiing-A with all weights predicted from the registration model learning module. This indicates that meta-learning method can learn the prior over 3D registration function space and benefit the 3D registration learner from good meta-learned initialized weights. Moreover, indicated in row 2 and 3, the experimental results of the weight- setting-C are better than the weight-setting--. B, which indicates that the task distribution generated by the VAE in the 3D registration meta-learner can gather the prior the 3D registration function
7

---

Under review as a conference paper at ICLR 2022
FlowNet3D
Ours
FlowNet3D Ours
Selected point cloud registration qualitative results on the FlyingThings3D dataset. Red points represent the source point cloud. Green points represent the target point cloud. Blue points represent the transformed source point cloud.
Figure 3:
space and further sampled for the learner predictor to predict the optimal parameters of the 3D) registration learner. The weight-setting-DD achieves 0. 0.1437 EPE which indicates that the proposed region aware decoder with weighted fusion can further improve the performance. In the rest of the experiments, we use this setting our default model.
4.4 EXPERIMENTS ON UNSEEN CATEGORY DATASET
Experiment sefting. We use the first 20 categories in the ModelNet40 dataset as our training dataset. We use the last 20 categories in ModelNet40 ing dataset. We compare our model with meta-learner and without meta-learner. Specifically, we use the last 20 categories with different noise in ModelNet40 testing dataset. We do the exact fair com-
Table 3: Results on the Flx ying SingThing33D test dataset.
our test-
Method
EPE
ACC (0.05)
ACC (0.1).
ICP
0,5019
7.62%
7.62%
FlowNer3D
0.1694
25.37%
57.85%
FlowNet3D++
0.1553
28.50%
60.39%
MeteorNet
0.209
52.12%
HPL.FowNee
0.1453
29.46%
61.91%
Ours
0.1437
29.92%
62.21%
parison show the effect of our proposed meta-learner in dealing with the unseen dataset. We list the quantitative results in Table
Results. As shown in Table 2 for all these three types of noise (P.D, D.I, and D.O noise), with the meta-learner our model achieves much lower EPE and higher accuracy compared to the model without meta-learner on the unseen 20 testing categories. This result indicates that our model with meta-learner can less affected by all these different noise patterns in comparison with the model without meta-learner.
4.5 EXPERIMENTS ON FLYINGTHINGS3]] DATASET
Experiment setting: From nearly
39000 models in the FlyingThings30 dataset, we randomly select 20000 models for training and 2000 models for testing. We sample 2048 points from each point cloud. To show the efficiency of our proposed model, we compare it with ICP(iterative closest point) and three settings of FlowNet3D. De refers to the optimal model in Liu et al. (019)). We also compare method with state-of-th -ott-e--rtttttttt-- methods FlowNet3D++ Wang al. 2020
Table 4: Results on the KITTI dataset.
Method
EPE
ACC (0.05)
ACC (0.1)
FlowNet3D
0.2113
8.79%
32,85%
SPLATFlowNet
0.1988
21.74%
53.91%
MeteorNet
0.2510
FlowNet3D++
0.2530
Ours
0.1456
24.63%
55.44%
Ours(seff-supervisee)
0.1196
29.49%
61.19%
and HPLFlowNet Gu
jet al. (2019). We list the quantitative results in Table and we demonstrate selected qualitative results for in Figure 3
8

---

Under review as a conference paper at ICLR 2022
RGB Image
Point Clonds
FlowNer3D
Figure 4: Qualitative results of point cloud registration on the KITTI dataset. Red points represent the source point cloud. Green points represent the target point cloud. Blue points represent the transformed source point cloud.
Results. As shown in Table 3 our method achieves lower EPE (0. 1437) compared to FlowNet3D (0.1694) and ICP (0.5019). As to the registration estimation accuracy (ACC), our method achieves significantly better results with 29.92% for the threshold 0.05 and 62.21% for the thresh- old 0.1, which is better than 25.37% for the threshold 0.05 and 57.85% for the threshold 0. achieved by FlowNet3D. Moreover, our proposed method outperforms the state-of-the-art meth- ods FlowNer3D++ and HPLFlowNet. In addition, from the qualitative results shown in Figure 3 notice that the alignment of source and target point clouds more accurate for our method. For example, we can clearly see the gap between green (target) and blue (transformed source) points for the left case in the result of FlowNet3D.
4.6 EXPERIMENTS ON KITTI DATASET
Experiment setting. The KITTI scene flow dataset includes 200 stereo images. Following the evaluation standards claimed in FlowNet3D for a fair comparison, we use the first 150 images from the KITTI scene flow dataset as the testing dataset to evaluate our model. Note that our model is only trained on the training dataset of Flying Things3D.
Results. The results shown in Table 4 demonstrates that our model trained on Flying Things31 achieves lower EPE and better ACC on KITTI comparison the results achieved by FlowNet3ID, SPLATFlowNet BBalkkisshann et al. (2018). For the qualitative results shown in Figure 4 we can clearly see that our registration result for the two cars on the left side is better since all the blue points are almost overlapped with the green points. In comparison, the result of FlowNet3D shows a gap between blue and green points.
CONCLUSION
This paper introduces novel setaa-learinng-bsee approach to our research community for point cloud registrations. contrast to recent proposed learning-based methods, our method leverages 3D registration meta-learner to learn the prior over 3D registration function space, the 3D reg- istration meta-learner can accordingly predict an optimal structure for the 3D registration learner to address the desired transformation for registration. To the best of our knowledge, method firstly leveraged a meta-learning based structure for this task and we achieved superior point cloud registration results the unseen dataset.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp. 3981-3989 2016.
Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, and Simon Lucey. Pointnetlk: Robust & efficient point cloud registration using pointnet. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7163-7172, 2019.
Guha Balakrishnan, Amy Zhao, Mert Sabuncu, John Guttag, and Adrian V Dalca. An unsu- pervised learning model for deformable medical image registration. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9252-9260, 2018.
Paul J Besl and Neil D McKay, Method for registration of 3-d shapes. In Sensor Fusion IV: Control Paradigms and Data Structures, volume 1611, pp. 886-607. International Society for Optics and Photonics, 1992.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Ad- vances Neural Information Processing Systems, pp. 9516 (216--9577,7 2018.
Xiuye Gu, Yijie Wang, Chongruo Wu, Yong Jae Lee, and Panqu Wang. Hplflownet: Hierarchical permutohedral lattice flownet for scene flow estimation on large-scale point clouds. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3254-3263, 2019.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: survey, arXiv preprint arXiv:2004.05439, 2020.
Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. arXiv preprint
2018.
Gidi Littwin and Lior Wolf. Deep meta functionals for shape representation, In Proceedings of the IEEE International Conference on Computer Vision, pp. 1824-1833. 2019.
Xingyu Liu, Charles Qi, and Leonidas Guibas. Flownet3d: Learning scene flow in 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 939,537, 2019.
Andriy Myronenko, Xubo Song, and Miguel A Carreira-Perpinán. Non-rigid point set registration: Coherent point drift. In Advances in Neural Information Processing Systems, pp. 1009-1016, 2007.
Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 1(2)44 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnct++: Deep hicrarchical feature learning point sets in metric space. In Advances in Neural Information Processing Systems, pp. 5099-5108, 2017b.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with ing, pp. 1842-1850. 2016.
neural networks. In International conference on machine leamn
Jürgen Schmidhuber. Learning to control weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139 1992.
10

---

Under review as a conference paper at ICLR 2022
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Yue Wang and Justin M Solomon. Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3523- 3532, 2019a.
Yue Wang and Justin M Solomon. Pmet: Self-super learning for sirtial-to-partiaal registration. arXiv preprint 1910. 2019b.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael Bronstein, and Justin M Solomon. Dynamic graph enn for learning point clouds. ACM Transactions on Graphics (TOG), 38(5): 1-12, 2019.
2019.
Zirui Wang. Shuda Li, Henry Howard-Jenking Victor Prisacariu, and Min Chen. Flownet3d++: Geometric losses for deep scene flow estimation. In The IEEE Winter Conference on Applications of Computer Vision, pp. 91-98, 2020.
Jiaolong Yang, Hongdong Li, Dylan Campbell, and Yunde Jia. Go-iep: A globally optimal solution to 3d iep point-set registration. IEFE transactions on pattern analysis and machine intelligence, 38(11):22444 38(11))2241-2254 2015.
Shuo Yang, Min Xu, and Hongxun Yao. Meta3d: Single-view 3d object reconstruction from shape priors in memory. arXiv preprint arXiv:2003.0371,, 2020.


Reviews:
[{"review_id": "y4PrvZCDKnX", "reviewer": "Reviewer_eJC8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Presentation is in general deficient; often the text seems vague. I will give some examples next. I believe that concise explanations and providing examples could help in communicating ideas in this paper. \n\nI feel the abstract could be improved to be more precise. For example, when saying \"we define each task\", results unclear what are those tasks or why is it not a single task. \n\nThe following affirmation seems vague to me \"In comparison to iterative registration methods, learning-based methods have advantages in dealing with a large number of datasets since learning-based methods can transfer the registration pattern from one dataset to another one.\" I feel we can also transfer patterns (parameters?) from different datasets when using traditional methods. \n\nI am not sure about the meaning of optimal in the following sentence: \"The meta-learner is responsible for providing the optimal initialization of a 3D registration learner\". My intuition tells me that an optimal initialisation does not require any adjustment; hence no need for a 3D registration learner. \n\nI am not sure what the text is trying to comunicate in the following sentence: \"... our meta-learning-based approach gains competitive advantages over existing generalization methods that our method can uniquely parameterize the 3D registration function for each pair of shapes to provide 3D point cloud registration.\"\n\n\n\nOn experimentation over the KITTI dataset:\n----------------------------------------------------------\nI think the text should explain the point cloud pair generation (source, target) from the stereo vision information. How many pairs were considered?\n\nI recommend report also quantitative results for the KITTI dataset.\n\n\nOther comments\n-----------------------\nI suggest rewriting section 2.2 (META-LEARNING METHODS) to contrast the proposed approach against existent ones. My impression of the review here is a list of related approaches but without comparing them with the propositions here. Beyond the particular problem here (3D registration), what do other methods have in common with the proposed one? What is new?\n\nWhat is the notation \"~\" in Eq. (1)?\n\nWhy theta_m is not in Eq. (2)?\n\nWhat is the difference between theta_m and sigma?\n\nWhat is the self-supervised variant in Table 4?\n", "summary_of_the_paper": "This paper addresses point cloud registration from a meta-learning perspective to quickly adapt with limited training data. The main idea is using a meta-learner is to initialise a 3D registration learner. The meta-learner predicts a prior registration that can rapidly adapt to new registration problems. Experimental results on several datasets (ModelNet, FlyingThings3D, and KITTI) showed superior performance over FlowNet3D.\n", "summary_of_the_review": "I found the presentation deficient and my impression is that despite the results seeming promising, this paper requires substantial improvements to be ready for another review process. \n\nThis paper seems correct in methodology however I did not check everything, in part of difficulties related to the presentation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nEhj5HM_aiz", "reviewer": "Reviewer_xfRw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The objective of the work: This paper proposes a learning method to improve the generalization ability for the point cloud registration tasks.\n\nStrong points: This overall idea is a hot topic in the point cloud area. The concept of this meta-learning is interesting.\n\nWeak points: (1). The paper claimed to improve the generalization ability, while the experiments are extremely lacked to support this claim. All the other recent point cloud registration methods (e.g, FMR[1], DeepGMR[2], RGM[3], PntLK++[4]) achieve very well in this generalization setting. (2) The paper claimed to solve the point cloud registration problem. However, the experiments have not been compared with the recent state-of-the-art point cloud registration methods, such as FMR, DeepGMR, RGM, PntLK++. The current experimental results in Table 2 seems worse than the state-of-the-art registration methods. \n\n[1]. Huang, Xiaoshui, Guofeng Mei, and Jian Zhang. \"Feature-metric registration: A fast semi-supervised approach for robust point cloud registration without correspondences.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n[2]. Yuan, Wentao, et al. \"Deepgmr: Learning latent gaussian mixture models for registration.\" European Conference on Computer Vision. Springer, Cham, 2020.\n[3]. Fu, Kexue, et al. \"Robust Point Cloud Registration Framework Based on Deep Graph Matching.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[4]. Li, Xueqian, Jhony Kaesemodel Pontes, and Simon Lucey. \"PointNetLK Revisited.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n", "summary_of_the_paper": "This paper proposes a learning method to estimate the transformation matrix directly using neural networks. This paper tried to leverage the meta-learning strategy to improve the generalization ability. ", "summary_of_the_review": "The overall idea is interesting. However, the experiments are much lacking behind to support the claims, and the current experiments have not compared with the state-of-the-art registration methods. I think this work is ongoing work and need to do more investigation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8xjbYx53uGK", "reviewer": "Reviewer_t8Fa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe idea looks new to me. The results are promising but are still not solid to support the claim (well addressing the generalization problem).\n \nWeakness:\nThough this work is about 3D point cloud registration, many important works about 3D point cloud registration are missing. For example, feature-based methods like FCGF, D3Feat, and end-to-end methods such as Global Point Registration, PREDATOR, PR-Net, RPM-net，Deep closet point, etc.\n \nThe proposed method does not show its superior performance on generalization in the experiments. The only experiment for evaluation of generalization capability is the KITTI test, where the model was trained on flyingthings3d and tested on KITTI.  I think more tests should be conducted on more datasets and compared with some state-of-the-art methods such as PREDATOR and RPM-Net.", "summary_of_the_paper": "This paper presents a new architecture for point cloud registration. Different from existing methods, the proposed architecture consists of two stages. The first one is called meta learner, which is used to predict a task distribution and sample the key parameters for the second stage - 3D registration learner.\nBy separating the network into two different parts, the learned model may be better at generalization.\n \nExperiments were conducted on ModelNet40, FlyingThings3D, and KITTI datasets. The results show the proposed method outperforms some existing methods, including FlowNet3D, FlowNet3D++, and HPLFlowNet.", "summary_of_the_review": "1. The proposed idea is new.\n2. Many related works are missing.\n3. Experiments are not solid to support the idea.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3wcMPcFvPWE", "reviewer": "Reviewer_woQH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The design of this proposed network is fairly interesting, but I am not convinced that all claims are properly substantiated. Additionally, the paper itself isn't very polished and leaves me with several concerns regarding its effectiveness for the proposed applications.\n\nFirst, 3d point cloud registration typically refers to finding a single rigid transformation in SE(3) that best aligns two point sets. The title mentions point cloud registration, but the method itself and the methods it is compared against are scene flow methods. Scene flow differs from 3d point cloud registration in that each point is assigned a different transformation in SE(3), typically in a 1-to-1 manner as a 3D analogue of optical flow. Given the non-standard use of technical terms, I question how this method is to be used. If the main comparative work is FlowNet3D and FlowNet3D++, it would seem that this paper should be more aligned with other scene flow works than registration (e.g. you also compare against ICP, which *is* a 3d point cloud registration algorithm). \n\nThe qualitative results are not all that convincing to me, and I don't know whether or not the result was cherry-picked or demonstrates the typical performance difference. Much of the details I would need to reimplement this method (including explanations of metric and experimental procedure) just refer to FlowNet3D. I think to make the paper more self-contained, it would be nice to include a little more detail instead of requiring reading FlowNet3D to understand your evaluation.\n\nTraining on FlyThings3D and testing on KITTI, showing good results, is encouraging to me that your method is actually achieving some superior generalizability performance with its meta-learning design. However, I am not quite sure if your ablations prove your claims since one could attribute the superior performance to just having a higher capacity network when the meta-learner is enabled? Also, why do you elementwise add the estimated meta parameters to the learned parameters? It would nice to see an ablation where you either fully predict the parameters (a traditional hypernetwork approach), or concatenate instead of adding, or use the predicted parameters just for layer normalization (which is more commonly seen in approaches like conditional batchnorm, or FiLM, or even in things like AdaIn or StyleGAN). \n\nThe text itself needs some work-- the citations seem to have a latex problem and break up the text, disrupting the flow and making everything hard to parse. Some of the math exposition is a little imprecise. For example, in Equation 1, shouldn't it be argmax if \\mathcal{L} is a similarity measure? Also, S_i and G_i are *not* subsets of R^3. A subset of R^3 is a single 3D point. S_i and G_i, since they are 3d point clouds, are a set of points in R^3. Typically, I would expect to see something like S_i \\in R^{N \\times 3}. \n\nOther small things: uniformly sampling rotation angles on each three rotation axes independently results in a biased sampling of SO(3). Axis-angle representation is better suited for unbiases SO(3) sampling of rotation angles under 45 degrees. In Table-1, you should have understandable shorthand instead of A,B,C,D-- I find this really hard to cross-reference. ", "summary_of_the_paper": "A scene flow estimation network is proposed with a meta-learning hypernetwork type design: the top level estimates the parameters of the bottom level network, which combines with meta-learned parameters to enable better generalization properties.", "summary_of_the_review": "Overall, I think this paper just has too many problems for me to recommend acceptance. I'm not fully convinced by the approach to meta-learn in function space-- I did not see a convincing argument explain how exactly this is necessary (e.g. a toy experiment to demonstrate its necessity would be nice)-- and the ablation results seem to be explainable by just having a higher capacity and therefore higher performing network. Lastly, I am a little confused whether this is a scene flow or a point cloud registration paper-- it would appear to be the former though the title and intro seem to indicate the former.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper applies a metalearning strategy to point cloud registration, which refines 3D registration networks to improve performance on specific datasets/settings.  Reviews for this paper recognized its potential interest but uniformly highlighted that the work is lacking in polish---both from an expository perspective and in terms of experiments.  Questions included whether the experiments truly support the claim of generalization, and whether the work would be better considered as a method for scene flow.  Authors did not rebut these points, so I am recommending rejection.
Decision: Reject