CoFiNet: Reliable Coarse-to-fine Correspondences for Robust PointCloud Registration
Authors: Hao Yu
Fu Li
Mahdi Saleh
Benjamin Busam
Slobodan Ilic
Venue: NeurIPS 2021 Poster (2021)
Keywords: Point cloud Matching
Point Cloud Registration
Coarse-to-fine Mechanism
Deep Neural Networks

Abstract: We study the problem of extracting correspondences between a pair of point clouds for registration. For correspondence retrieval, existing works benefit from matching sparse keypoints detected from dense points but usually struggle to guarantee their repeatability. To address this issue, we present CoFiNet - Coarse-to-Fine Network which extracts hierarchical correspondences from coarse to fine without keypoint detection. On a coarse scale and guided by a weighting scheme, our model firstly learns to match down-sampled nodes whose vicinity points share more overlap, which significantly shrinks the search space of a consecutive stage. On a finer scale, node proposals are consecutively expanded to patches that consist of groups of points together with associated descriptors. Point correspondences are then refined from the overlap areas of corresponding patches, by a density-adaptive matching module capable to deal with varying point density. Extensive evaluation of CoFiNet on both indoor and outdoor standard benchmarks shows our superiority over existing methods. Especially on 3DLoMatch where point clouds share less overlap, CoFiNet significantly outperforms state-of-the-art approaches by at least 5% on Registration Recall, with at most two-third of their parameters.

CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration
Hao Yu
Fu Li
Mahdi Saleh
Benjamin Busam
Slobodan 3,
Technical University of Munich National University of Defense Technology
Siemens AG, München
yuao.yu, fu 1i, m. masahh, b. busam,, slobodan. ilii))tum.dd. @tum.
Abstract
We study the problem of extracting correspondences between pair of point clouds for registration. For correspondence retrieval, existing works benefit from matching sparse keypoints detected from dense points but usually struggle to guarantee their repeatability. To address this issue, we present CoFiNet Coarse-to- Fine Network which extracts hierarchical correspondences from coarse to fine without keypoint detection. On a coarse scale and guided by a weighting scheme, our model firstly learns to match down-sampled nodes whose vicinity points share more overlap, which significantly shrinks the search space of a consecutive stage. On a finer scale, node proposals are consecutively expanded to patches that consist of groups points together with associated descriptors. Point correspondences are then refined from the overlap areas of corresponding patches, by density-adaptiin matching module capable to deal with varying point density. Extensive evaluation of CoFiNet both indoor and outdoor standard benchmarks shows superiority over existing methods. Especially on 3DI Lotachh where point clouds share less overlap, CoFiNet significantly outperforms state-of-the-ae approaches by at least 5% on Registration Recall, with at most two-third of their parameters. [Code]
Introduction
Correspondence search a core topic of computer vision and establishing reliable correspondences key to success in many fundamental vision tasks, such as tracking, reconstruction, flow estimation, and particularly, point cloud registration. Point cloud registration aims at recovering the transforma- tion between a pair of partially overlapped point clouds. It is fundamental task in a wide range of real upplications. including scene reconstruction, autonomous driving, simultancous localization and mapping (SLAM), etc. However, due to the unordered and irregular properties of point clouds, extracting reliable correspondences from them has been a challenging task for a long time. From early-stage hand-crafted methods [1, 2, 3, 4, 5] recently emerged deep learning- based approaches [6, 7, 8, 9, 10, 1, 12], many works contributed to improving the reliability of correspondences.
We can broadly categorize recent deep learning- based point cloud registration methods into three categories. The first [13, 14, 15] follows the idea of ICP [16], where they iteratively find dense correspondences and compute pose estimation. The second [17, 18] includes the correspondence-feee methods based on the intuition that the feature distance between two well-aligned point clouds should be small. Such methods encode the whole point cloud as a single feature and iteratively optimize the relative pose between two frames by minimizing the distance corresponding features. Though achieving reasonable results synthetic object datasets [19], both of them struggle on large-scale real benchmarks [20, 6], as the first suffers from low correspondence precision and high computational complexity, while the second lacks robustness to noise and partial overlap.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Coarse: Correspondence Proposal Block
a. Context Aggregation
b. Correspondence Proposal
Similnty
Coreesondenees
? e ....
Sel-att leff-tttnnion
Cross atteentoon
Self-ai anlf-atuntion
Coarse Scate
(P)F)
09.50
Sinkhorn
Slack Entnes
Confidence Metrix
19, F31
Correspondence
Proposal
88.00
(6.61 61
Finer: Correspondence Refinement Block
Finer Scale
Shared
Patch Grouping
b. Rofinoment
(r,F)
Corresspondences
3 @ &
coreespordunois
Correspondence
Refinement Block
chared
2100
Figure I: Left: Overview of CoFiNet. From top to bottom: 1) Dense points are down-sampled to uniformly distributed nodes, while associated features are jointly learned. 2) Correspondence Proposal Block (Top Right): Features are strengthened and used to calculate the similarity matrix. Coarse node correspondences are then proposed from the confidence matrix. 3) Strengthened features are decoded to dense descriptors associated with each input point. 4) Correspondence Refinement Block (Bottom Right): Coarse node proposals are first expanded to patches via grouping. Patch correspondences are then refined to point level by our proposed density-adaptive matching module, whose details can be found in Fig. 2.
Differently, the last category of methods [6, 7, 9, 10, 11, 12] tackles point cloud registration in a two-stage manner. They firstly learn local descriptors of down-sampled sparse points (nodes) for matching, and afterward use robust pose estimators, e.g., RANSAC [21], J, for recovering the relative transformation. Their two-stage strategy makes them achieve state-of-the-art performance on large- scale real benchmarks [20, 6]. Uniform sampling [7, 8] and keypoint detection [22, 10, 11, 12] are two common ways introduce sparsity. Compared to uniform sampling that samples points randomly, keypoint detection estimates the saliency of points and samples points with strong geometry features, which significantly reduces the ambiguity of matching. However, the sparsity by nature challenges repeatability, i.c., sub- samplann increases the risk where a certain point loses its corresponding point in the other frame, which constrains the performance of detection-based methods [22, 10, 11, 12].
Recently, a coarse-to-fine mechanism leveraged by our 2D counterparts [23, 24, 25] to avoid direct keypoint detection, which shows superiority over the However, 3D point cloud matching, where keypoint detectors usually perform worse, existing deep learning-basee based methods do not yet exploit such coarse-to-fine strategy. To fill the gap, we focus on leveraging the coarse-to-fine mechanism to eliminate the side effects of detecting sparse keypoints.
art detection-based method [26].
Nevertheless, designing such a coarse-to-fine fine pipeline in point cloud matching is non-trivial, mainly due to the inherent unordered and irregular nature point clouds. To this end, we propose weighting matching module for correspondence
scheme for coarse node matching and a censity-adaptis refinement, which enables CoFiNet to extract coarse-to-fine correspondences from point clouds. More specifically, on a coarse scale, the weighting scheme proportional to local overlap ratios guides the model propose correspondences of nodes whose vicinity areas share more overlap, which effectively squeezes the search space of the consecutive refinement. On a finer scale, the adnsity-adaptive matching module refines coarse correspondence proposals to point level by solving a differentiable optimal transport problem [26] with awareness to varying point density, which shows more robustness on irregular points. An overview of our proposed method can be found in Fig. 1. Our main contributions summarized as follows:
A detection-free learning framework that treats point cloud registration as a coarse-to- fine correspondence problem, where point correspondences are consecutively refined from coarse proposals that are extracted from unordered and irregular point clouds.

---

A weighting scheme that, on a coarse scale, guides our model to learn to match uniformly down-sampled nodes whose vicinity areas share more overlap, which significantly shrinks the search space for the refinement.
A differentiable density-adaptiv matching module that refines coarse correspondences to point level based on solving an optimal transport problem with awareness to point density, which is more robust to the varying point density.
To the best of our knowledge, we are the first deep learning-basse work that incorporates a coarse- to-fine mechanism in correspondence search for point cloud registration. Extensive experiments are conducted on both indoor and outdoor benchmarks to show our superiority. Notably, CoFiNet surpasses the state-of-the-art with much fewer parameters. Compared to [12], we only use around two-third and one-fourth of parameters on indoor and outdoor benchmarks, respectively.
2 Related Work
Learned local descriptors. Early networks proposed to learn local descriptors for 3D correspon- dence search mainly take uniformly distributed local patches input. As pioneer, Zeng et al. [6] propose the 3DMatch benchmark, on which they exploit a Siamese network [27] that consumes voxel grids of TDFs (Truncated Distance Fields) match local patches. PPFNet [7] directly consumes raw points augmented with point-pair features (PPF) by leveraging PointNet [28] as its encoder. PPF-FoldNet [8] leverages only PPF, which is naturally ivaatin-invvrrian, as its input and further incorporates a FoldingNet [29] architecture to enable the unsupervised training of rotation-invarinn descriptors. Gojeic al. [9] propose a network to consume the smoothed density value (SDV) representation aligned to the local reference frame (LRF) to eliminate the rotation-variang of learned descriptors. To extract better geometrical features, Graphite [11] utilizes graph neural networks for local patch description. SpinNet [30] utilizes LRF for patch alignment and 3D cylindrical convolution layers for feature extraction, achieving the best generalization ability to unseen datasets. However, patch-based methods usually suffer from low computational efficiency, as typically shared activations of adjacent patches are not reused. To address this, FCGF [31] makes the first attempt by using sparse convolutions [32] to compute dense descriptors of the whole point cloud in single pass, which leads to 600x speed-up while still being able to achieve comparable performance to patch- patch-based methods.
Learned 3D keypoint detectors. USIP [22] learns to regress the position of the most salient point in each local patch in a self-supervieed manner. However, it suffers from degenerated cases when the number of desired keypoints is relatively small. D3Feat [10] exploits a fully convolutional encoder- decoder architecture for joint dense detection and description. However, it does not consider overlap relationships and shows low robustness on low-overlap scenarios. In addition to jointly estimating salient scores and learning local descriptors, PREDATOR [12] also predicts dense overlap scores that indicate the confidence whether points are on the overlap regions. Keypoints will be sampled under the condition of both saliency and overlap scores. Though it surpasses existing methods by large margin on both 3DMatch[6] and 3DLoMatch(12) the precision of estimated scores and the repeatability of sampled keypoints constrain its performance.
Coarse-to-fine correspondences. As witnessed in 2D image matching, many recent works [23, 24, 25] leverage a coarse-to-fin mechanism to climinate the inherent repeatability problem in keypoint detection and thus boost the performance. DRC-Net [23] utilizes 4D cost volumes to enumerate all the possible matches and establishes pixel correspondences in a coarse-to-fine manner. Concurrently with DRC-Net, Patch2Pix [24] first establishes patch correspondences and then regresses pixel correspondenee correspondences according to matched patches. In similar coarse-to- to-rse-to-fine manner with Patch2Pixel, LoFTR [25] leverages Transformers [33], together with an optimal transport matching layer [26], to match mutual-nearest patches on the coarse level, and then refines the corresponding pixel of the patch center the finer level.
3 Methodology
Problem statement. Given a pair of unordered point sets with n points x; E R and Y with m points V/ E RR3, we aim at recovering the rigid transformation TX SE(() between them. For simpler notation, we define their coordinate matrices as Px E R and Py E Rm respectively. We follow the path of extracting correspondences first and then estimating the relative pose, where we mainly focus on the former. For this purpose, propose CoFiNet that takes pair of point clouds as
3

---

input and outputs point correspondences, which can be leveraged to estimate the rigid transformation by RANSAC [21].
3.1 Coarse-scale Matching
Point encoding. On the coarse level, our target is matching uniformly down-sampled nodes whose vicinity areas share more overlap. To achieve this goal, we first adopt shared KPConv [34] encoders to down-sample raw points to uniformly distributed nodes P E R x3 and P E TRm x3 while jointly learning their associated features F'x E IR"" xb and Fy E IR xb Demonstration of down-sampled nodes can be found in 1) of Fig. 1. Please refer Appendix for more details of network architecture.
Attentional feature aggregation. As illustrated in Fig. 1, the Correspondence Proposal Block (CPB) takes input the down-sampled nodes and associated features. In CPB (a), following [26, 12], the attention [33] mechanism is leveraged to incorporate more global contexts to the learned features. Following [12]. we adopt sequence of self-, cross- and self self-attention modules, which interactively aggregates global contexts across nodes from the same and the other frame in pair of point clouds. Below we briefly introduce the cross-attention module. Given (F'x x F'y), ) akin database retrieval, the former is linearly projected by learnable matrix Wq E R query Q as Q = F'x W Q. while the latter is similarly projected to key K and value V by learnable matrices WK and Wv. v. respectively. The attention matrix A is represented as A QK"/ /vb, whose rows are then normalized by softmax. M A V. which represents the linear
The message M flows from F'y to Fxx is formulated combination of values weighted by the attention matrix. In the cross-attention module, contexts are aggregated bidirectionally, from Fxx to F'y and from Fyy to F'x. For computational efficiency, we replace the graph- craph-based based module [35] leveraged in [12] with the self-attention module in [26], which has the same architecture as the cross- coss-attention module but takes the features from the same point cloud, c.g., (F'x, F'x), as input.
Correspondence proposal. As shown in CPB (b) of Fig. 1, leverage strengthened features F'x and Fyy to calculate the similarity matrix. Down-sampled nodes whose vicinity areas share enough overlap are matched. However, there can be two cases where a node fails to match: 1) The major portion of its vicinity areas is occluded in the other frame. 2) Though most of its vicinity areas are visible in the other frame, there does not exist a node whose vicinity areas share sufficient overlap with its. Thus, for the similarity matrix, we expand with slack row and column with m' and nt' slack entries, respectively [36]. So that nodes fail to match other nodes could match their corresponding slack entries, i.e., having maximum scores there. Similar to [26], we compute the similarity matrix using an inner product, which can be presented as:
F F
s'
S' E "((n"+++))((""
(I)
ZT
where all slack entries are set to the same learnable parameter 2. On S' we run the Sinkhorn Algorithm [37,36 38, 39], seeking an optimal solution for the optimal transport problem. After that, each entry j") in the obtained matrix represents the matching confidence between the node i' and node j' from P'x and P'y, respectively. To guarantee a higher recall, we adopt a threshold Ta for likely correspondences whose confidence scores are above тę- We define the obtained coarse node correspondence set as C' = {PP'x (i'), (j'))) with |C|| d' where |.| denotes the set cardinality. Furthermore, we set the other threshold Tm to guarantee that é' 2 Tm When c' < Tm we gradually decrease To to extract more coarse node correspondences.
3.2 Point-level Refinement
Node decoding. On the finer scale, we aim at refining coarse correspondences from the preceding stage to point level. Those refined correspondences can then be used for point cloud registration. We first stack several KPConv [34] layers to recover the raw points, Px and Py, while jointly learning associated dense descriptors, Fx E Raxe and Fy E IR" associated feature + f E IR. as illustrated in 3) of Fig. Then, as demonstrated in 4) of Fig. 1, obtained dense descriptors, together with raw points and coarse correspondences are fed into the Correspondence Refinement Block (CRB), where coarse proposals are expanded to patches that are then refined to point correspondences.
xe, xe, We thereby assign to each point an

---

b. Density Adaptive Matching
Context Aggregation
b.1. Similariiy Matrix
Normalization
X.N b.3. Confidence
Normallzza
COS .F.
Row Normalization
Column
exp(-
07
apeaz)
valid score dritries
muled scare entries
slack cnntrics
Figure 2: Illustration of our proposed density matching module. The input is pair of patches truncated by k. a) We use the context aggregation part from the Correspondence Proposal Block to condition both patches and to strengthen features. b.1) The similarity matrix is computed. Slack entries are initialized with 0 and muted entries corresponding to repeatedly sampled points are set to -00 b.2) N iterations of the Sinkhorn Algorithm are performed. We drop the slack row and column for row and column normalization, respectively. b.3) We obtain the confidence matrix, whose and column- normalized respectively. For correspondences,
first & rows and & columns pick the maximum confidence score in every row and column to guarantee a higher precision.
Point-to-nods grouping. For refinement, we need to expand nodes in coarse correspondences to patches consisting of groups of points and associated descriptors. Accordingly, we use a point-to-node grouping strategy [40, 41, 22] to assign points to their nearest nodes in geometry space. If point has multiple nearest nodes, a random one will be picked. We demonstrate this procedure in CRB (a) of Fig. 1. The advantages of point-to-nodh over k-nearest neighbor search radius-based ball query two-fold: 1) Every point will be assigned to exactly one node, while some points could be left out in other strategies. 2) It can automatically adapt to various scales [22]. After grouping, nodes with their associated points and descriptors form patches, upon which we can extract point correspondences. For certain node P (t',, its associated point set G.P. and feature set G can be denoted as:
G. = (p E Px 1Ip (x)II S 1p-1pp-e P #1).
G = If Fx + + with p G.
where |I-II = |1-112 represents the Euclidean norm. In the point-to-node grouping, we expand the coarse node correspondence set C' to its corresponding patch correspondence set, both in geometry space Cp [G G and feature space CF {G G.))).
Density-adaptic matching. Extracting point correspondences from a pair of overlapped patches in some way analogous to matching two smaller scale point clouds from a local perspective. Thus, directly leveraging the CPB in Fig. with imput (G., G.) and (GP, G could theoretically tackle the problem. However, simply utilizing CPB to extract point corréspondences would lead to bias towards slack rows and columns, i.e., the model learns to predict more points as occluded. Reasons for this are two-fold: 1) For computational efficiency, similar to radius-based ball query, in point-to-node grouping, we need to truncate the number of points to unified number k for every patch. If patch contains less than points, like [42], a fixed point or randomly sampled points will be repeated as a supplement. 2) On a coarse level, our model learns to propose corresponding nodes with overlapped vicinity areas. However, after expansion, proposed patches can supplemented by some occluded points, which introduces biases in the training of refinement. To address the issue, we propose a aensity-adaptive matching module that refines coarse correspondences to point level by solving optimal transport problem with awareness to point density. We denote the truncated patches G and demonstrate our proposed density-adaptive matching module in Fig. 2. Notably, both during and after normalization, the exponent projection of any muted entries is always equal to 0, which eliminates the side effects caused by the repeated sampling of points. The final point correspondence set Ć is represented as the union of all the obtained correspondence sets (()) C can be directly leveraged by RANSAC[2]] for registration.
3.3 Loss Functions
Our total loss L = Le + >Ly is calculated as the weighted sum of the coarse-scale L. and the fine-scale Ly, where À is used to balance the terms. We detail the individual parts hereafter.
5

---

Coarse scale. On the coarse scale, we leverage weighting scheme proportional to the overlap ratios over patches coarse supervision. Given pair of down-sampled nodes (x')") and Pyy (j"). with their expanded patch representation geometry space, G.P. and G we can compute the ratio of points in Gø that are visible in point cloud Py as:
((t) Ip E G. E Py s.t.
TY qll <
(3)
G
where is the distance threshold. Similarly, we can calculate the ratio of points in G that have correspondences in G as:
(') (p E E s.t. TY all < TpH
(4)
Based on Eq. 3 and Eq. 4, we define the weight matrix W E (
(m'++)) as:
< n' A m
S n' Aj = m 1,
W'('))=
(5)
r(i)),
-= n' A S m
0,
otherwise.
Finally, we define the coarse scale loss as:
C. Eig
W (") log(S' (f,J'))
(6)
Eg W (7'J") Finer scale. On the finer point level, for the 7th truncated patch correspondence (GP, GJ.) s.t. (GP, GP E Cp, we define the binary matrix ( E (+++1)x(+++) as:
(i,))) - { 0, otherwise,
1. I G (1) -G00) <Tp.
[
and
(1,k +1) (1,
max(0, Σ (1,1),
Ei [1,
j=l
(8)
( + 1.j) max(0, 1- Σ B (i.)).
Vj
Additionally, further set the rows and columns of ( which correspond to repeatedly sampled points to 0 to eliminate their side effects during training. B (k 1, k 1) is also set to 0. Therefore, by defining the confidence matrix in b.3 of Fig. 2 as (1)), the loss function on the finer scale reads as: Cj - (1) (1, (Sg) (1,5)
Eis ( (i,5)
where we define 0 log(()= = 0.
4 Results
We evaluate our model on three challenging public benchmarks, including both indoor' and outdoor scenarios. Following [12], for indoor scenes, evaluate model both 3DMatch [6], where point cloud pairs share 30% overlap, and 3DLoMatch [12], where point cloud pairs have 10% -30% overlap. In line with existing works [10, 12], evaluate for outdoor scenes odometryKITTI [20]. Please refer to Appendix for more details of implementation and datasets.
As PREDATOR fixed a bug after our submission, we update their latest results. We also update ours according the rebuttal.

---

True: 5
True: 5
True:
False: 245
False: 24
False: 178
IR: 2.0%
IR: 17.2%
IR: 28.8%
Alignment
PREDATOR
Ours-Coarse
Ours-Fime
Figure 3: Qualitative results on Inlier Ratio. We compare our point correspondences (the last column) with correspondences (the third column) and correspondences from PREDATOR (the second column) on a hard case from 3DLoMatch. The first column provides the ground truth alignment, which shows that overlap very limited. The significantly larger inlier ratio can be observed from the incorrect (red) and correct (green) correspondence connections.
4.1 3DMatch and 3DLoMatch
We compare our proposed CoFiNet to other state-of-the-art approaches including 3DSN [9], FCGF [31], D3Feat [10] and PREDATOR [12] Tab. and Fig. 4. Comparisons to Spinlm- ages [1], SHOT [2], FPFH [4] and 3DMatch [6] are also included in Fig. 4. For efficiency and qualitative results, please refer to Appendix.
Table 1: R both 3DMatch and 3DLoMatch datasets under different numbers of samples. We also show the number of utilized parameters of all the approaches in the last column. Best performance highlighted in bold while the second best is marked with underline.
3DMatch
3Dtaatth
Samples
5000
2500
1000
500
250
5000
2500
1000
500
250
Params |
Registration Recall(%)
3DSN[9]
78.4
76.2
71.4
67.6
50.8
33.0
29.0
23.3
17.0
11.0
10.2M
RCGH|31]
85.1
84.7
83.3
81.6
71.4
40.1
41.7
38.2
35.4
26.8
8.76M
D3Feat] 10]
81.6
84.5
83.4
82.4
77.9
37.2
42.7
46.9
43.8
39.1
27.3M
PREDATOR[12]
89.0
89.9
90.6
88.5
86.6
59.8
61.2
62.4
60.8
58.1
7.43M
ColiNet(ours)
89.3
88.9
88.4
87.4
87.0
67.5
66.2
64.2
63.1
61.0
5.48M
Feature Matching Recall
3DSN[91
95.0
94.3
92.9
90.1
82.9
63.6
61.7
53.6
45.2
34.2
10.2M
FCGF|31]
97.4
97.3
97.0
96.7
96.6
76.6
75.4
74.2
71.7
67.3
8.76M
D3Feat[10]
95.6
95.4
94.5
94.1
93.1
67.3
66.7
67.0
66.7
66.5
27.3M
PREDATOR[12]
96.6
96.6
96.5
96.3
96.5
78.6
77.4
76.3
75.77
75.3
7.43M
CoFiNet(owrs)
98.3
83.1
83.5
83.3
83.1
82.6
5.48M
Inlier Ratio(%) f
3DSN[9]
36.0
32.5
26.4
21.5
16.4
11.4
10.1
8.0
6.4
4.8
10.2M
FCGF
56.8
42.5
34.1
21.4
20.0
17.2
14.8
11.6
8.76M
D3Feat[10]
39.0
40.4
41.5
41.8
13.2
13.1
14.0
14.6
15.0
27.3M
PREDATOR[12]
58.4
54.1
49.3
26.7
28.1
28.3
27.5
25.8
7.43M
CoFiNet(ours)
49.8
51.2
51.9
52.2
52.2
24.4
25.9
26.7
26.8
26.9
5.48M
Metries. We adopt three typically-used metrics, namely Registration Recall (RR), Feature Match- ing Recall (FMR) and Inlier Ratio (IR), to show the superiority of CoFiNet
existing approaches. Specifically, 1) the Registration Recall the fraction of point cloud pairs whose error of transforma- tion estimated by RANSAC is smaller than a certain threshold. e.g., RMSE 0.2m, compared to the ground truth. 2) The Feature Matching Recall indicates the percentage of point cloud pairs whose Inlier Ratio is larger than certain threshold, e.g., 5%. 3) The Inlier Ratio is the fraction of correspondences whose residual error in geometry space is less than a threshold, e.g., T1 10cm, under the ground truth transformation. Metric details given in Appendix.
? As PREDATOR computes Inlier Ratio on correspondence set different to the one used for registration, we give more results in 4.1 and Tab. for fair comparison.

---

Correspondence sampling. We follow [10, 12] and report performance with different numbers of samples. However, as CoFiNet avoids keypoint detection and directly outputs point correspondences. we cannot strictly follow [10, 12] to sample different numbers of interest points. For a fair comparison, we instead sample correspondences in our experiments but keep the same number as them. Corre- spondences are sampled with probability proportional to a global confidence Cgiobal = Cfine" Ccoorse For certain point correspondence refined from patch correspondence (G. Gj.), we define Cfine as its fine-level confidence score and Ccoarse as S' ,i'',
Inlier Ratio. As the main contribution of CoFiNet is that we adopt the coarse-to- to-e mechanism to avoid keypoint detection, while existing methods struggle to sample repeatable keypoints for matching, we first check the Inlier Ratio of CoFiNet, which is directly related to the quality of extracted correspondences. We show quantitative results in Tab. and qualitative results Fig. As shown in Tab. 1, on Inlier Ratio, CoFiNet outperforms all the previous methods except PREDATOR [12] on 3DLoMatch and only performs worse than PREDATOR [12] and FCGF[31] on 3DMatch. Notably, when the sample number is 250, we perform the best on both datasets, since detection tetection-based methods face a more severe repeatability problem this case. By contrast, as our method leverages a coarse- to-fine mechanism and thus avoids keypoint detection, it is more robust to the aforementioned case. Furthermore, the fact that sampling fewer correspondences leads to a higher Inlier Ratio indicates that our learned scores are well-calibrated, i.e., higher confidence scores indicate more reliable correspondences.
Table 2: Registration results without RANSAC [21]. Relative poses are directly solved based on extracted correspondences by singular value decomposition (SVD). Best performance is highlighted in bold while the second best is marked with an underline.
3DMatch
3DLoMttch
# Samples
5000
2500
1000
500
250
5000
2500
1000
500
250
Registration Recall w/o RANSAC (%) f
FCGF[31]
28.5
27.9
25.7
23.2
21.2
2.3
1.7
1.3
1.1
1.1
D3Fca(10]
24.3
24.0
23.0
22.4
19.1
1.1
1.4
1.1
1.0
1.0
PREDATOR[12]
48.7
51.8
54.3
53.5
53.0
6.1
8.1
10.1
11.4
11.3
CoHiNtt(uur)
63.2
63.4
63.8
64.9
64.6
19.0
20.4
21.0
20.9
21.6
Reliability of our correspondences. Though Inlier Ratio is an important metric of correspondence quality, it is naturally affected by the distance threshold 71- To better illustrate the reliability of correspondence correspondences extracted by CoFiNet and show our superiority over existing methods, we conduct another experiment and show related results in Tab. In this experiment, we directly solve the relative poses using singular value decomposition (SVD) based on extracted correspondences, without the assistance of the robust estimator RANSAC [21]. As we can see, for FCGF [31] and D3Feat [10], though they can work on 3DMatch, they fail on 3DLoMatch, where point cloud share less overlap and thus reliable correspondences are harder to obtain. Compared with PREDATOR [12], on both 3DMatch and 3DLoMatch, our proposed CoFiNet performs much better, which indicates that we propose more reliable correspondences on both datasets.
Feature Matching Recall and Regis- tration Recall. On Feature Match- ing Recall, CoFiNet significantly out- performs all the other methods on both 3DMatch and 3DLoMatch. Espe- cially on 3DtoMatth, which is more challenging due to the low-overlap scenarios, our proposed method sur- passes others with large margin of more than 4%. It indicates that CoFiNet is more robust to different scenes, i.e., we find at least 5% inlier correspondences for more test cases. Additionally, we also follow [10, 12] to show the Feature Matching Recall in relation to 72 and Tj on 3DMatch in
200
Festure Matching Recall
11.05
0.1t
Sprimages
60
60-
40
40
MEDOTON
0.1
0.00
0.2
0.16
0.2
0.11
0.01
0.08
0.12
0.16 0.20
Inlie:
Thresholdd
Inlier Distance Threshold (m)
Figure 4: Feature Matching Recall in relation to: 1) Inlier Ratio Threshold (72) and 2) Inlier Distance Threshold (Ti) on 3DMatch.
8

---

Fig. 4, which further shows our superiority over other methods. When referring to the most important metric Registration Recall which better reflects the final performance on point cloud registration. though we perform slightly worse than PREDATOR [12], we significantly outperform others on 3DMatch. When evaluated on 3DLoMatch, our proposed approach significantly surpasses all the others, which shows the advantages of our method in scenarios with less overlap. Moreover, we also compare the number of parameters used different methods in the last column of Tab. which shows that CoFiNet uses the least parameters while achieving the best performance.
Table 3: Inlier Ratio and Registration Recall on the same correspondence set. For CoFiNet, coarse correspondences are extracted based on thresholds and non-mutual selection is used on the finer scale. Best performance is highlighted in bold while the second best is marked with an underline.
3DMatch
3DLoMatch
Samples
5000
2500
1000
500
250
5000
2500
1000
500
250
Registration Recall(S)
PREDATOR[ 22 (mntual)
86.6
86.4
85.3
85.6
84.3
61.8
61.8
61.6
58.4
56.2
PREDATOR[ 22 (non-mutual)
89.0
89.9
90.6
88.5
86.6
59.8
61.2
62.4
60.8
58.1
CoFiNet(ours)
89.3
88.9
88.4
87.4
87.0
67.5
66.2
64.2
63.1
61.0
Inlier Ratio(%) f
PREDATOR (mutual)
58.0
58.4
57.1
54.1
49.3
26.7
28.1
28.3
27.5
25.8
PREDATOR[12 (non-mutual)
46.6
48.3
47.2
44.1
38.8
19.3
21.6
22.1
21.3
19.7
CoFiNct(ours)
49.8
51.2
51.9
52.2
52.2
24.4
25.9
26.7
26.8
26.9
Inlier Ratio and Registration Recall on the same correspondence set. In Tab. 1, PREDA- TOR [12] reports Inlier Ratio on a correspondence set that is different to the one used for registration, while CoFiNet uses the same. PREDATOR uses correspondences extracted by mutual selection to report Inlier Ratio, while computing Registration Recall on a correspondence set obtained by non-mutual selection. As we target registration, consider it meaningless to evaluate correspondence set that is not used for pose estimation. Thus, to make a fair comparison, we compare and PREDDATOR(non-nen
CoFiNet with both PREDATOR two points x and y are considered as a correspondence when x match to and y match to x, while in non-mutual selection, the correspondence is extracted when match y or y match to In Tab. compared to non-mutual, mutual selection rejects some outliers, and thus increases Inlier Ratio of PREDATOR. However, as meanwhile filters out some inlier correspondences, when combined with RANSAC [21], Registration Recall usually drops. Since our task is registration, PREDATOR (non- mutual) with higher Registration Recall preferred over itself with mutual selection. In this case, CoFiNet achieves higher Inlier Ratio than PREDATOR on both datasets.
in Tab. 3. In mutual selection,
Influence of the number of coarse correspon- dences. As illustrated in Tab. 4, on both 3DMatch and 3DLo- Match, when sampled only with Tex a higher threshold results in fewer coarse correspondences and meanwhile a higher Inlier Ratio, which indicates the learned confidence scores are well-calibratdd on the coarse level. However, Registration Recall drops at the same time. the number of correspondences for refinement is decreased, and thus fewer point correspondences are leveraged in RANSAC for pose estimation. The last row is the strategy used in our paper. Except for Te, also 7m to guarantee that CoFiNet samples least 7m coarse correspondences on each point cloud pair, as described before. This strategy slightly sacrifices Inlier Ratio but brings significant improvements on Registration Recall.
Table 4: Ablation study of the number of coarse correspondences, tested with # Samples=2500. # Coarse indicates the average number of sampled coarse correspondences. Best performance is highlighted in bold.
3DMatch RR (%)
3DLoMatch RR (5:)
To
Tu
IR (%)
Coarse
IR (%:
# Coarse
0.05
49.4
87.4
575
27.3
62.8
260
0.10
51.1
88.1
335
29.8
62.3
128
0.15
55.5
85.5
222
32.7
58.9
74
0.20
200
51.2
88.9
230
25.9
66.2
203
Importance of individual modules. As shown in Tab. 5, in the first experiment, we directly use the coarse correspondence set C' for point cloud registration. Unsurprisiggy, it performs worse

---

all the metries, indicating that CoFiNet benefits from refinement. Then, we ablate the weighting scheme which is proportional to overlap ratios and guides the coarse matching of down- sampled nodes. We replace it with a binary mask similar to the one used the finer level. Results show that it leads to worse performance, which proves that coarse matching of nodes benefits from our designed weighting scheme. Finally, we do the last ablation study on the density-adaptic matching module. Results indicate that on both 3DMatch and 3DLoMatch, with the density-adaptive matching module, CoFiNet better adapts to the irregular nature of point clouds.
Table 5: Ablation study of individual modules, tested with # Samples=2500. Best performance is highlighted in bold.
3DMatch
3DLoMatch
RR (%)T
FMR (%)
IR (%) f
RR (%)
FMR (%)
IR (%)]
Full CoFiNet
88,9
98.3
51.2
66.2
83,5
25.9
w/o refinement
79.6
96.5
44.3
41.2
81.4
21.3
w/o weighting
87.4
97.3
50.0
61.5
80.5
23.5
w/o adasiteee
88.3
97.9
49.3
65.1
82.7
24.7
4.2 KITTI
Metrics. We follow [12] and use 3 metrics, namely, the Relative Rotation Error (RRE), which is the geodesic distance between estimated and ground truth rotation matrices, the Relative Translation Error (RTE), which is the Euclidean distance between the estimated and ground truth translation, and the Registration Recall (RR) mentioned before. More details are provided in Appendix.
Comparisons to existing ap- proaches. On KITTI, we com- pare CoFiNet 3DFeat-net [43],
Table 6: Quantitative comparisons on KITTI. Best performance is highlighted in bold.
FCGF [31], D3Feat [10] and
PREDATOR [12]. Quantitative
Method
RTE(cm))
RRE())
RR(%)1
Params
results can be found in Tab. 6, while qualitative results given in Appendix. On RTE and RRE, we stay in the middle, but for RR, together with [10, 12], we perform the best. Notably, we achieve such performance by us- ing only 5.48M parameters and training for 20 epochs compared the best performing model [12], which uses over 20M parameters and is trained for 150 epochs. This experiment indicates that our model can deal with outdoor scenarios.
3DFeat-Net [43]
25.9
0.57
96.0
0.32M
FCGF [31]
9.5
0.30
96.6
8.76M
D3Fcat [10]
7.2
0.30
99.8
27.3M
PREDATOR [12]
6.8
0.27
99.8
22.8M
CoFiNet(ours)
8.5
0.41
99.8
5.48M
5 Conclusion
In this paper, we present a deep neural network that leverages a coarse-to-fin strategy to extract correspondences from unordered and irregularly sampled point clouds for registration. Our proposed model is capable of directly consuming unordered point sets and proposing reliable correspondences without the assistance of keypoints. To tackle the irregularity of point clouds, we first propose weighting scheme proportional to local overlap ratios. It guides the model to match nodes that have overlapped vicinity areas, which significantly shrinks the search space of the following refinement. On finer level, we then adopt a beensit--ddattive matching module, which eliminates the side effects from repeated sampling and enables our model deal with density varying points. Extensive experiments on both indoor and outdoor benchmarks validate the effectiveness of our proposed model. We stay on par with the state-of-the-ae approaches on 3DMatch and KITTI, while surpassing them on 3DLoMatch using model with significantly fewer parameters. Limitations and broader impact are discussed in Appendix.
coarse scale,
Acknowledgments
We appreciate the valuable discussion with Zheng Qin and Hao Wu. We would like to thank Dr. Kai Wang for paper revision. Hao Yu is supported by China Scholarship Council (CSC).
10

---

References
[1] Andrew E. Johnson and Martial Hebert. Using spin images for efficient object recognition in cluttered 3d scenes. IEEE Transactions pattern analysis and machine intelligence. (1(5):433-444 1999.
[2] Federico Tombari, Samuele Salti, and Luigi Di Stefano. Unique signatures of histograms for local surface description. In European conference computer vision, pages 356-369. Springer, 2010.
[3] Federico Tombari, Samuele Salti, and Luigi Di Stefano. Unique shape context for 3d data description. In Proceedings of the ACM workshop on 3D object retrieval, pages 57-62, 2010.
[4] Radu Bogdan Rusu, Nico Blodow, and Michael Beetz. Fast point feature histograms (fpfh) for 3d registration. In 2009 IEEE international conference on robotics and automation, pages 3212-3217. IEEE, 2009.
[5] Radu Bogdan Rusu, Zoltan Csaba Marton, Nico Blodow, and Michael Beetz. Persistent point feature histograms for 3d point clouds. In Proc 10th Int Conf Intel Autonomous Syst (IAS-10), Baden-Bade Baden, Germany, pages 119-128, 2008.
[6] Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1802- 1811, 2017.
[7] Haowen Deng, Tolga Birdal, and Slobodan Ilie. Ppfnet: Global context aware local features for robust 3d point matching. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 195 205, 2018
[8] Haowen Deng, Tolga Birdal, and Slobodan Ili.. Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors. In Proceedings of the European Conference Computer Vision (ECCV), pages 602-618, 2018.
[9] Zan Gojcic, Caifa Zhou, Jan D Wegner, and Andreas Wieser. The perfect match: 3d point cloud matching with smoothed densities. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5545-5554, 2019.
[10] Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, and Chiew-Lan Tai. D3feat: Joint learning of dense detection and description of 3d local features. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 6359-6367, 2020.
[11] Mahdi Saleh, Shervin Dehghani, Benjamin Busam, Nassir Navab, and Federico Tombari. Graphite: Graph-induced feature extraction for point cloud registration. In 2020 International Conference on 3D Vision (3DV), pages 241-2311 241-251, 2020.
[12] Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler. Predator: Registration of 3d point clouds with low overlap. arXiv preprint arXiv:2011. 2020.
[13] Yue Wang and Justin M Solomon. Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3523-3533 2019.
[14] Yue Wang and Justin M Solomon. Prnet: Self-supervised learning for partial-to-p to--trtial registra- tion. arXiv preprint 19X0v:1110.220 12240, 2019.
[15] Zi Jian Yew and Gim Hee Lec. Rpm-net: Robust point matching using learned features. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11 11824-1 11833, 2020.
[16) Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586-606. International Society for Optics and Photonics, 1992.

---

[17] Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, and Simon Lucey. Pointnetlk: Robust & efficient point cloud registration using pointnet. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7163-7172, 2019.
[18] Xiaoshui Huang, Guofeng Mei, and Jian Zhang. Feature-metric registration: A fast semi- supervised approach for robust point cloud registration without correspondences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognilion, pages 1 11366-11374, 2020.
[19] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[20] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231-1237, 2013.
[21] Martin Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381-395, 1981.
[22] Jiaxin Li and Gim Hee Lee. Usip: Unsupervised stable interest point detection from 3d point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 361-370, 2019.
[23] Xinghui Li, Kai Han, Shuda Li, and Victor Prisacariu. Dual-resolution correspondence networks. Advances in Neural Information Processing Systems, 33, 2020.
[24] Qunjie Zhou, Torsten Sattler, and Laura Taixe Patch2pix: chipolar-guides pixel-level correspondences. arXiv preprint arXiv:2012.01909, 2020.
[25] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. arXiv preprint arXiv:2104.0066 2021.
[26] Paul-Fdouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938-4947, 2020.
[27] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. Advances in neural information processing systems, 6:737-744, 1993.
[28] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[29] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2018
2018
[30] Sheng Ao, Qingyong Hu, Bo Yang, Andrew Markham, and Yulan Guo. Spinnet: Learning a general surface deseriptor for 3d point cloud registration. arXiv preprint arXiv:2011.12149, 2020.
[31] Christopher Choy, Jaesik Park, and Vladlen Koltun. Fully convolutional geometric features, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8958-8966. 2019.
[32] Christopher Choy, Jun JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3075-3084, 2019.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv: 7706.000777777 03762, 2017.
12

---

[34] Hugues Thomas, Charles R Qi, Jean-Emmanue Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6411-66 1421- 2019.
[35] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
[36] Benjamin Busam, Marco Esposito, Simon Che'Rose, Nassir Navab, and Benjamin Frisch. A stereo vision approach for cooperative robotic movement therapy. Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 127-135, 2015.
[37] Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343-34) 348, 1967.
[38] Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. Advances in neural information processing systems, 26:2292-2300, 2013.
[39] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends(R) in Machine Learning, 11(5-6):355-60) 2019.
[40] Teuvo Kohonen. The self-organizing map. Proceedings of the IEEE, 78(9):464-1144 1990.
[41] Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 9397-9406 2018.
[42] Charles R Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint 0rXiv:1706.02415 2017.
[43] Zi Jian Yew and Gim Hee Lee. 3dfeat-net: Weakly supervised local 3d features for point cloud registration. In Proceedings of the European Conference Computer Vision (ECCV), pages 607-623, 2018.
13


Reviews:
[{"review_id": "y_Vr-CMcJ8U", "reviewer": "Reviewer_1Jc9", "summary": "The paper proposed to study the problem of extracting correspondences for 3D point cloud registration. The paper presented CoFiNet (Coarse-to-Fine Network) to extract hierarchical correspondences from coarse to fine without detecting keypoints.  The model firstly learns to match down-sampled nodes and proposes node correspondences. On a finer scale, node proposals are consecutively expanded to patches that consist of groups of points together with associated descriptors. Patch correspondences are then refined to point level by a density-adaptive matching module. The proposed method has been evaluated on both indoor and outdoor standard benchmarks.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper extended the coarse-to-fine strategy in 2D vision tasks to 3D point cloud registration. The related work did not cite and discuss other category of point cloud registration approaches such as the correspondence-free approaches.\n\nQuality: In the paper, the proposed coarse-to-fine pipeline works at a coarse and finer scale. Could this coarse-to-fine strategy be applied to multiple scales? \n\nClarity: While the paper presented interesting ideas, the clarity is a major concern. For example, the differentiable density-adaptive matching module, which is a key component of the proposed framework, has not been made clear. \n\nSignificance: 3d point cloud registration is an important task in computer vision. The paper extended the coarse-to-fine strategy in 2D vision tasks to 3D.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper did not address the limitations of the work. The paper aims at tackling partial overlap while it did not provide experiments with respect to different ratios of overlapping between input point clouds. It is interesting and critical to analyze the robustness with respect to different ratios of overlapping. The scalability issue has also not been analyzed in the paper. Could the proposed approach be applied to large scale point clouds with low overlapping ratios?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "tZGNnq9FK_Z", "reviewer": "Reviewer_5BJR", "summary": "The authors present a coarse-to-fine approach to extract correspondences that can be used for a RANSAC-based point cloud registration.\nThey demonstrate on standardized datasets that they achieve a higher or similar inlier ratio, registration recall and feature matching recall than current state of the art with fewer parameters.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Interesting approach to employ a coarse-to-fine approach for refining matching points which improves on the two-stage RANSAC-based methods.\n\nThe paper is well-written though parts could be made clearer.\n\n\nSome questions:\n\nI'm a bit unclear on the changes you've made in the CPB block. Could you expand on how your correspondence proposal block compares to the one used in [24], is there a difference between their dustbins and your slack variables? And have you compared how the performance depends on the number of self- + cross-attention stacks you use (currently you use self- + cross- + self-attention)?\n\nYou've examined the use of a uniform sampling procedure as well as one based on the predicted scores but did you also try a voxel-based sampling procedure or similar to maintain the larger distribution of points that, as you discuss in the supplementary material, is necessary for a reliable registration?\nCould this be the reason why the RR metric is similar with both the 'prob' and 'rand' sampling strategies while the IR is clearly in favor of the 'prob' approach?\n\nThe results on KITTI seems good, is the RTE and RRE calculated for the correctly registered point cloud or for all?\n\nOne comment is that the W and the B matrices used in the loss calculations probably should be defined in the paper not only verbally but similarly to what is done in the supplementary. It could also be clearer if the patch-to-patch correspondences are used or the point-to-point correspondences inside of two patches are compared since  $\\tilde{S}^{(l)}$ is not defined.\n\nCould you clarify the exact nature of the RANSAC procedure, is the GT transformation involved in selecting the correct transformation or is it only based on the number of inliers as would be done in a real-world scenario?\n\n\nOn the whole, it's a good paper that takes the concept of coarse-to-fine refinement from the 2D domain to the 3D case which results in a higher inlier ratio.\n\nUpdate after rebuttal:\n\nThe authors addressed most issues from the reviews well. My already positive assessment remain unchanged.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, as much as can be expected for the topic of this paper.", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "nwvXuJDknQr", "reviewer": "Reviewer_em8a", "summary": "This paper proposes a coarse-to-fine network to extract the correspondences for point cloud registration.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The overall framework is similar to PREDATOR, but replace the sparse convolution with KPCov. The attention module follows PREDATOR's transformer. The encoder module is the coarse step and the decoder module is the fine step in this submission. Based on my understanding, the contribution is incremental and the performance achieves very limited improvement compared to PREDATOR. \n\nQuality: This paper is technically sound but I am still worried about the efficiency compared to PREDATOR. The authors have not discussed the weakness of their work. From Figure 3, we can find the final correspondences are located in the very limited regions. Why other regions are all outliers in this work. I am very worried about the robustness since most of the information are discarded roughly in the coarse step.\n\nClarity: The overall organization is well but could be improved. For example, I disagree with the claim in the abstract \"Existing works widely adopt keypoint detection before establishing correspondences but struggle to propose repeatable keypoints\". The recent point cloud registration methods, such as pointnetlk(CVPR2019), DGR (CVPR2020), FMR(CVPR2020), PREDATOR (CVPR2021) and PointDSC(CVPR2021), all do not adopt the keypoint detection. \n\nSignificance: This work is an increment of PREDATOR by replacing the sparse convolution with KPCov. The significance of the scientific finding is very limited.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "m6cCOV-RgSs", "reviewer": "Reviewer_ZezU", "summary": "The paper presents a novel learning method for point cloud registration by extracting hierarchical correspondences in a coarse to fine manner. There are two modules:\n\nCoarse scale:  a network is first utilized to downsample and extract feature from input  point cloud. The features are then strengthened and used to compute a score matrix for coarse correspondence proposals.\n\nFine scale: coarse correspondence proposals are expanded to patch correspondences via grouping and refined with a density-adaptive matching module.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- It is a novel idea to treat point cloud registration as a coarse-to-fine correspondence problem. The approach is free of keypoint detection and thus keypoint repeatability issue, which enables robust registration.\n\n- The method achieves state-of-the-art results on both indoor and outdoor benchmarks with fewer parameters. The experimental results are detailed.\n- The paper is well-written.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- In the coarse scale, how the number of sample points affects the results?\n- It would be better to add some examples of failure cases.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "_Y88Nd9Qav3", "reviewer": "Reviewer_z7KQ", "summary": "The paper proposes an approach for finding correspondences for robust point cloud registration without detecting keypoints. The method is based on the hierarchical coarse-to-fine processing of the point clouds that is then used to find the sought rigid transformation by using RANSAC.", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Keypoints do not have to be detected to find the rigid transformation. This improves the repeatability of the solution. \n- The proposed weighting scheme helps in finding the correspondences efficiently.\n- The experiments show that the proposed method leads to SOTA results on the 3DMatch and 3DLocMatch datasets. To Table 3, it is not SOTA on Kitti, thus, I suggest removing sentence \"Results show that CoFiNet achieves state-of-the-art performance on\n14 all evaluated datasets\" from the abstract. \n\nWeaknesses:\n- The paper (at least for me) was quite challenging to understand. Aside from the topic being far from me, I would suggest significantly improving the text since it is unclear in many places. For example, Fig.1 is supposed to help in understanding the method, however, it does not really help. \n\nSince I am not too familiar with the topic I can't judge if the method is new. \nThe results seem good compared to recent approaches and, thus, I am suggesting acception. \n\nSuggestion:\n- It would be interesting to see the performance if a modern robust estimator is used, e.g. [a,b], not RANSAC.\n\n[a] Raguram, Rahul, et al. \"USAC: A universal framework for random sample consensus.\" IEEE transactions on pattern analysis and machine intelligence 35.8 (2012): 2022-2038.\n\n[b] Barath, Daniel, et al. \"MAGSAC++, a fast, reliable and accurate robust estimator.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"ethical_concerns": "-", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This papers introduces CoFiNet, a coarse-to-fine methods to extract hierarchical correspondences between 3D point clouds, which relaxes the need for keypoints detection. Experiments are conducted on datasets with high (3DMatch) and low (3DLoMatch) level of correspondence. 

The paper initially received three accept and two reject recommendation. The main concerns pointed out by reviewers were the lack of novelty of the proposed method, and the modest improvement compared to recent baselines (e.g. PREDATOR).
After rebuttal, the initially positive reviewers were convinced by the rebuttal and recommended acceptance. Especially, R5BJR who did a detailed review supported paper acceptance. On the other hand, Rem8a and R1Jc9 still considered the novelty insufficient and the experiments not convincing enough, and stuck on their rejection recommendation.

The AC carefully read the submission and authors' feedback. The AC considers that the idea of adapting keypoint-free coarse-to-fine refinement methods from the 2D to 3D is interesting. Although closely based on prior works, the adaptations and contributions are overall convincing, i.e. the weighting scheme and the differentiable density-adaptive matching. The experiments also show the relevance of the approach compared to recent baselines, e.g. PREDATOR. 
The AC thus recommends acceptance, but highly encourages the authors to carefully take into account and reviewers' comments and there feedback to improve the final version of the paper. Especially, the clarity of the paper should be improved to clarify and highlight the contribution and the positioning of the proposed method with respect to related works. The submission has been discussed with the senior area chair who agrees with the recommendation.

Decision: Accept (Poster)