CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registration
Year: 2022

Abstract:
We study the problem of extracting correspondences between a pair of point clouds for registration. For correspondence retrieval, existing works benefit from matching sparse keypoints detected from dense points but usually struggle to guarantee their repeatability. To address this issue, we present CoFiNet Coarse-to-Fine Network which extracts hierarchical correspondences from coarse to fine without keypoint detection. On a coarse scale and guided by a weighting scheme, our model firstly learns to match down-sampled nodes whose vicinity points share more overlap, which significantly shrinks the search space of a consecutive stage. On a finer scale, node proposals are consecutively expanded to patches that consist of groups of points together with associated descriptors. Point correspondences are then refined from the overlap areas of corresponding patches, by a density-adaptive matching module capable to deal with varying point density. Extensive evaluation of CoFiNet on both indoor and outdoor standard benchmarks shows our superiority over existing methods. Especially on 3DLoMatch where point clouds share less overlap, CoFiNet significantly outperforms state-of-the-art approaches by at least 5% on Registration Recall, with at most two-third of their parameters. [Code]

heading: 1 Introduction
text: Correspondence search is a core topic of computer vision and establishing reliable correspondences is a key to success in many fundamental vision tasks, such as tracking, reconstruction, flow estimation, and particularly, point cloud registration. Point cloud registration aims at recovering the transformation between a pair of partially overlapped point clouds. It is a fundamental task in a wide range of real applications, including scene reconstruction, autonomous driving, simultaneous localization and mapping (SLAM), etc. However, due to the unordered and irregular properties of point clouds, extracting reliable correspondences from them has been a challenging task for a long time. From early-stage hand-crafted methods [1, 2, 3, 4, 5] to recently emerged deep learning-based approaches [6, 7, 8, 9, 10, 11, 12], many works contributed to improving the reliability of correspondences.
We can broadly categorize recent deep learning-based point cloud registration methods into three categories. The first [13, 14, 15] follows the idea of ICP [16], where they iteratively find dense correspondences and compute pose estimation. The second [17, 18] includes the correspondence-free methods based on the intuition that the feature distance between two well-aligned point clouds should be small. Such methods encode the whole point cloud as a single feature and iteratively optimize the relative pose between two frames by minimizing the distance of corresponding features. Though achieving reasonable results on synthetic object datasets [19], both of them struggle on large-scale real benchmarks [20, 6], as the first suffers from low correspondence precision and high computational complexity, while the second lacks robustness to noise and partial overlap.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Differently, the last category of methods [6, 7, 8, 9, 10, 11, 12] tackles point cloud registration in a two-stage manner. They firstly learn local descriptors of down-sampled sparse points (nodes) for matching, and afterward use robust pose estimators, e.g., RANSAC [21], for recovering the relative transformation. Their two-stage strategy makes them achieve state-of-the-art performance on largescale real benchmarks [20, 6]. Uniform sampling [7, 8] and keypoint detection [22, 10, 11, 12] are two common ways to introduce sparsity. Compared to uniform sampling that samples points randomly, keypoint detection estimates the saliency of points and samples points with strong geometry features, which significantly reduces the ambiguity of matching. However, the sparsity by nature challenges repeatability, i.e., sub-sampling increases the risk where a certain point loses its corresponding point in the other frame, which constrains the performance of detection-based methods [22, 10, 11, 12].
Recently, a coarse-to-fine mechanism is leveraged by our 2D counterparts [23, 24, 25] to avoid direct keypoint detection, which shows superiority over the state-of-the-art detection-based method [26]. However, in 3D point cloud matching, where keypoint detectors usually perform worse, existing deep learning-based methods do not yet exploit such a coarse-to-fine strategy. To fill the gap, we focus on leveraging the coarse-to-fine mechanism to eliminate the side effects of detecting sparse keypoints.
Nevertheless, designing such a coarse-to-fine pipeline in point cloud matching is non-trivial, mainly due to the inherent unordered and irregular nature of point clouds. To this end, we propose a weighting scheme for coarse node matching and a density-adaptive matching module for correspondence refinement, which enables CoFiNet to extract coarse-to-fine correspondences from point clouds. More specifically, on a coarse scale, the weighting scheme proportional to local overlap ratios guides the model to propose correspondences of nodes whose vicinity areas share more overlap, which effectively squeezes the search space of the consecutive refinement. On a finer scale, the density-adaptive matching module refines coarse correspondence proposals to point level by solving a differentiable optimal transport problem [26] with awareness to varying point density, which shows more robustness on irregular points. An overview of our proposed method can be found in Fig. 1. Our main contributions are summarized as follows:
• A detection-free learning framework that treats point cloud registration as a coarse-to-fine correspondence problem, where point correspondences are consecutively refined from coarse proposals that are extracted from unordered and irregular point clouds.
• A weighting scheme that, on a coarse scale, guides our model to learn to match uniformly down-sampled nodes whose vicinity areas share more overlap, which significantly shrinks the search space for the refinement.
• A differentiable density-adaptive matching module that refines coarse correspondences to point level based on solving an optimal transport problem with awareness to point density, which is more robust to the varying point density.
To the best of our knowledge, we are the first deep learning-based work that incorporates a coarseto-fine mechanism in correspondence search for point cloud registration. Extensive experiments are conducted on both indoor and outdoor benchmarks to show our superiority. Notably, CoFiNet surpasses the state-of-the-art with much fewer parameters. Compared to [12], we only use around two-third and one-fourth of parameters on indoor and outdoor benchmarks, respectively.

heading: 2 Related Work
text: Learned local descriptors. Early networks proposed to learn local descriptors for 3D correspondence search mainly take uniformly distributed local patches as input. As a pioneer, Zeng et al. [6] propose the 3DMatch benchmark, on which they exploit a Siamese network [27] that consumes voxel grids of TDFs (Truncated Distance Fields) to match local patches. PPFNet [7] directly consumes raw points augmented with point-pair features (PPF) by leveraging PointNet [28] as its encoder. PPF-FoldNet [8] leverages only PPF, which is naturally rotation-invariant, as its input and further incorporates a FoldingNet [29] architecture to enable the unsupervised training of rotation-invariant descriptors. Gojcic et al. [9] propose a network to consume the smoothed density value (SDV) representation aligned to the local reference frame (LRF) to eliminate the rotation-variance of learned descriptors. To extract better geometrical features, Graphite [11] utilizes graph neural networks for local patch description. SpinNet [30] utilizes LRF for patch alignment and 3D cylindrical convolution layers for feature extraction, achieving the best generalization ability to unseen datasets. However, patch-based methods usually suffer from low computational efficiency, as typically shared activations of adjacent patches are not reused. To address this, FCGF [31] makes the first attempt by using sparse convolutions [32] to compute dense descriptors of the whole point cloud in a single pass, which leads to 600x speed-up while still being able to achieve comparable performance to patch-based methods.
Learned 3D keypoint detectors. USIP [22] learns to regress the position of the most salient point in each local patch in a self-supervised manner. However, it suffers from degenerated cases when the number of desired keypoints is relatively small. D3Feat [10] exploits a fully convolutional encoderdecoder architecture for joint dense detection and description. However, it does not consider overlap relationships and shows low robustness on low-overlap scenarios. In addition to jointly estimating salient scores and learning local descriptors, PREDATOR [12] also predicts dense overlap scores that indicate the confidence whether points are on the overlap regions. Keypoints will be sampled under the condition of both saliency and overlap scores. Though it surpasses existing methods by a large margin on both 3DMatch[6] and 3DLoMatch[12], the precision of estimated scores and the repeatability of sampled keypoints constrain its performance.
Coarse-to-fine correspondences. As witnessed in 2D image matching, many recent works [23, 24, 25] leverage a coarse-to-fine mechanism to eliminate the inherent repeatability problem in keypoint detection and thus boost the performance. DRC-Net [23] utilizes 4D cost volumes to enumerate all the possible matches and establishes pixel correspondences in a coarse-to-fine manner. Concurrently with DRC-Net, Patch2Pix [24] first establishes patch correspondences and then regresses pixel correspondences according to matched patches. In a similar coarse-to-fine manner with Patch2Pixel, LoFTR [25] leverages Transformers [33], together with an optimal transport matching layer [26], to match mutual-nearest patches on the coarse level, and then refines the corresponding pixel of the patch center on the finer level.

heading: 3 Methodology
text: Problem statement. Given a pair of unordered point sets X with n points xi ∈ R3 and Y with m points yj ∈ R3, we aim at recovering the rigid transformation T X
Y ∈ SE(3) between them. For simpler notation, we define their coordinate matrices as PX ∈ Rn×3 and PY ∈ Rm×3, respectively. We follow the path of extracting correspondences first and then estimating the relative pose, where we mainly focus on the former. For this purpose, we propose CoFiNet that takes a pair of point clouds as
input and outputs point correspondences, which can be leveraged to estimate the rigid transformation by RANSAC [21].

heading: 3.1 Coarse-scale Matching
text: Point encoding. On the coarse level, our target is matching uniformly down-sampled nodes whose vicinity areas share more overlap. To achieve this goal, we first adopt shared KPConv [34] encoders to down-sample raw points to uniformly distributed nodes P′X ∈ Rn ′×3 and P′Y ∈ Rm ′×3, while jointly learning their associated features F′X ∈ Rn ′×b and F′Y ∈ Rm
′×b. Demonstration of down-sampled nodes can be found in 1) of Fig. 1. Please refer to Appendix for more details of network architecture.
Attentional feature aggregation. As illustrated in Fig. 1, the Correspondence Proposal Block (CPB) takes as input the down-sampled nodes and associated features. In CPB (a), following [26, 12], the attention [33] mechanism is leveraged to incorporate more global contexts to the learned features. Following [12], we adopt a sequence of self-, cross- and self-attention modules, which interactively aggregates global contexts across nodes from the same and the other frame in a pair of point clouds. Below we briefly introduce the cross-attention module. Given (F′X, F ′ Y), akin to database retrieval, the former is linearly projected by a learnable matrix WQ ∈ Rb×b to query Q as Q = F′XWQ, while the latter is similarly projected to key K and value V by learnable matrices WK and WV, respectively. The attention matrix A is represented as A = QKT / √ b, whose rows are then normalized by softmax. The message M flows from F′Y to F ′ X is formulated as M = A ·V, which represents the linear combination of values weighted by the attention matrix. In the cross-attention module, contexts are aggregated bidirectionally, from F′X to F ′ Y and from F ′ Y to F ′ X. For computational efficiency, we replace the graph-based module [35] leveraged in [12] with the self-attention module in [26], which has the same architecture as the cross-attention module but takes the features from the same point cloud, e.g., (F′X, F ′ X), as input.
Correspondence proposal. As shown in CPB (b) of Fig. 1, we leverage strengthened features F̃′X and F̃ ′ Y to calculate the similarity matrix. Down-sampled nodes whose vicinity areas share enough overlap are matched. However, there can be two cases where a node fails to match: 1) The major portion of its vicinity areas is occluded in the other frame. 2) Though most of its vicinity areas are visible in the other frame, there does not exist a node whose vicinity areas share sufficient overlap with its. Thus, for the similarity matrix, we expand it with a slack row and column with m′ and n′ slack entries, respectively [36]. So that nodes fail to match other nodes could match their corresponding slack entries, i.e., having maximum scores there. Similar to [26], we compute the similarity matrix using an inner product, which can be presented as:
S′ = F̃′XF̃′ TY z zT z  , S′ ∈ R(n′+1)×(m′+1), (1) where all slack entries are set to the same learnable parameter z. On S′ we run the Sinkhorn Algorithm [37, 38, 39], seeking an optimal solution for the optimal transport problem. After that, each entry (i′, j′) in the obtained matrix represents the matching confidence between the node i′ and node j′ from P′X and P ′ Y, respectively. To guarantee a higher recall, we adopt a threshold τc for likely correspondences whose confidence scores are above τc. We define the obtained coarse node correspondence set as C′ = {(P′X(i′),P′Y(j′))}, with |C′| = c′, where |·| denotes the set cardinality. Furthermore, we set the other threshold τm to guarantee that c′ ≥ τm. When c′ < τm, we gradually decrease τc to extract more coarse node correspondences.

heading: 3.2 Point-level Refinement
text: Node decoding. On the finer scale, we aim at refining coarse correspondences from the preceding stage to point level. Those refined correspondences can then be used for point cloud registration. We first stack several KPConv [34] layers to recover the raw points, PX and PY, while jointly learning associated dense descriptors, FX ∈ Rn×c and FY ∈ Rm×c. We thereby assign to each point p an associated feature p↔ f ∈ Rc, as illustrated in 3) of Fig. 1. Then, as demonstrated in 4) of Fig. 1, obtained dense descriptors, together with raw points and coarse correspondences are fed into the Correspondence Refinement Block (CRB), where coarse proposals are expanded to patches that are then refined to point correspondences.
Point-to-node grouping. For refinement, we need to expand nodes in coarse correspondences to patches consisting of groups of points and associated descriptors. Accordingly, we use a point-to-node grouping strategy [40, 41, 22] to assign points to their nearest nodes in geometry space. If a point has multiple nearest nodes, a random one will be picked. We demonstrate this procedure in CRB (a) of Fig. 1. The advantages of point-to-node over k-nearest neighbor search or radius-based ball query are two-fold: 1) Every point will be assigned to exactly one node, while some points could be left out in other strategies. 2) It can automatically adapt to various scales [22]. After grouping, nodes with their associated points and descriptors form patches, upon which we can extract point correspondences. For a certain node P′X(i
′), its associated point set GPi′ and feature set G F i′ can be denoted as: G P i′ = {p ∈ PX ∣∣‖p−P′X(i′)‖ ≤ ‖p−P′X(j′)‖,∀j′ 6= i′}, GFi′ = {f ∈ FX
∣∣f ↔ p with p ∈ GPi′ }, (2) where ‖·‖ = ‖·‖2 represents the Euclidean norm. In the point-to-node grouping, we expand the coarse node correspondence set C′ to its corresponding patch correspondence set, both in geometry space CP = {(GPi′ ,GPj′)} and feature space CF = {(GFi′ ,GFj′)}.
Density-adaptive matching. Extracting point correspondences from a pair of overlapped patches is in some way analogous to matching two smaller scale point clouds from a local perspective. Thus, directly leveraging the CPB in Fig. 1 with input (GPi′ ,G F i′) and (G P j′ ,G F j′) could theoretically tackle the problem. However, simply utilizing CPB to extract point correspondences would lead to a bias towards slack rows and columns, i.e., the model learns to predict more points as occluded. Reasons for this are two-fold: 1) For computational efficiency, similar to radius-based ball query, in a point-to-node grouping, we need to truncate the number of points to a unified number k for every patch. If a patch contains less than k points, like in [42], a fixed point or randomly sampled points will be repeated as a supplement. 2) On a coarse level, our model learns to propose corresponding nodes with overlapped vicinity areas. However, after expansion, proposed patches can be supplemented by some occluded points, which introduces biases in the training of refinement. To address the issue, we propose a density-adaptive matching module that refines coarse correspondences to point level by solving an optimal transport problem with awareness to point density. We denote the truncated patches as G̃ and demonstrate our proposed density-adaptive matching module in Fig. 2. Notably, both during and after normalization, the exponent projection of any muted entries is always equal to 0, which eliminates the side effects caused by the repeated sampling of points. The final point correspondence set C̃ is represented as the union of all the obtained correspondence sets C̃(l). C̃ can be directly leveraged by RANSAC[21] for registration.

heading: 3.3 Loss Functions
text: Our total loss L = Lc + λLf is calculated as the weighted sum of the coarse-scale Lc and the fine-scale Lf , where λ is used to balance the terms. We detail the individual parts hereafter.
Coarse scale. On the coarse scale, we leverage a weighting scheme proportional to the overlap ratios over patches as coarse supervision. Given a pair of down-sampled nodes P′X(i ′) and P′Y(j ′), with their expanded patch representation in geometry space, GPi′ and G P j′ , we can compute the ratio of points in GPi′ that are visible in point cloud PY as:
r(i′) = |{p ∈ GPi′ |∃q ∈ PY s.t. ‖T
X
Y(p)− q‖ < τp}| |GPi′ | , (3)
where τp is the distance threshold. Similarly, we can calculate the ratio of points in GPi′ that have correspondences in GPj′ as:
r(i′, j′) = |{p ∈ GPi′ |∃q ∈ GPj′ s.t. ‖T
X
Y(p)− q‖ < τp}| |GPi′ | . (4)
Based on Eq. 3 and Eq. 4, we define the weight matrix W′ ∈ R(n ′+1)×(m′+1) as:
W′(i′, j′) =  min(r(i′, j′), r(j′, i′)), i′ ≤ n′ ∧ j′ ≤ m′, 1− r(i′), i′ ≤ n′ ∧ j′ = m′ + 1, 1− r(j′), i′ = n′ + 1 ∧ j′ ≤ m′, 0, otherwise.
(5)
Finally, we define the coarse scale loss as: Lc = − ∑ i′,j′ W ′ (i′, j′) log(S′ (i′, j′))∑
i′,j′ W ′ (i′, j′)
. (6)
Finer scale. On the finer point level, for the lth truncated patch correspondence (G̃Pi′ , G̃Pj′) s.t. (GPi′ ,G P j′) ∈ CP, we define the binary matrix B̃(l) ∈ R(k+1)×(k+1) as:
B̃(l)(i, j) = { 1, ‖TXY(G̃Pi′ (i))− G̃Pj′(j)‖ < τp, 0, otherwise,
∀i,∀j ∈ [1, k] , (7)
and
B̃(l)(i, k + 1) = max(0, 1− k∑
j=1
B̃(l)(i, j)), ∀i ∈ [1, k] ,
B̃(l)(k + 1, j) = max(0, 1− k∑
i=1
B̃(l)(i, j)), ∀j ∈ [1, k] .
(8)
Additionally, we further set the rows and columns of B̃(l) which correspond to repeatedly sampled points to 0 to eliminate their side effects during training. B̃(l)(k+1, k+1) is also set to 0. Therefore, by defining the confidence matrix in b.3 of Fig. 2 as S̃(l), the loss function on the finer scale reads as:
Lf = − ∑
l,i,j B̃ (l) (i, j) log(S̃(l) (i, j))∑ l,i,j B̃ (l) (i, j) , (9)
where we define 0 · log(0) = 0.

heading: 4 Results
text: We evaluate our model on three challenging public benchmarks, including both indoor1 and outdoor scenarios. Following [12], for indoor scenes, we evaluate our model on both 3DMatch [6], where point cloud pairs share > 30% overlap, and 3DLoMatch [12], where point cloud pairs have 10% ~30% overlap. In line with existing works [10, 12], we evaluate for outdoor scenes on odometryKITTI [20]. Please refer to Appendix for more details of implementation and datasets.
1As PREDATOR fixed a bug after our submission, we update their latest results. We also update ours according to the rebuttal.

heading: 4.1 3DMatch and 3DLoMatch
text: We compare our proposed CoFiNet to other state-of-the-art approaches including 3DSN [9], FCGF [31], D3Feat [10] and PREDATOR [12] in Tab. 1 2 and Fig. 4. Comparisons to SpinImages [1], SHOT [2], FPFH [4] and 3DMatch [6] are also included in Fig. 4. For efficiency and qualitative results, please refer to Appendix.
Metrics. We adopt three typically-used metrics, namely Registration Recall (RR), Feature Matching Recall (FMR) and Inlier Ratio (IR), to show the superiority of CoFiNet over existing approaches. Specifically, 1) the Registration Recall is the fraction of point cloud pairs whose error of transformation estimated by RANSAC is smaller than a certain threshold, e.g., RMSE < 0.2m, compared to the ground truth. 2) The Feature Matching Recall indicates the percentage of point cloud pairs whose Inlier Ratio is larger than a certain threshold, e.g., τ2 = 5%. 3) The Inlier Ratio is the fraction of correspondences whose residual error in geometry space is less than a threshold, e.g., τ1 = 10cm, under the ground truth transformation. Metric details are given in Appendix.
2As PREDATOR computes Inlier Ratio on a correspondence set different to the one used for registration, we give more results in 4.1 and Tab. 3 for a fair comparison.
Correspondence sampling. We follow [10, 12] and report performance with different numbers of samples. However, as CoFiNet avoids keypoint detection and directly outputs point correspondences, we cannot strictly follow [10, 12] to sample different numbers of interest points. For a fair comparison, we instead sample correspondences in our experiments but keep the same number as them. Correspondences are sampled with probability proportional to a global confidence cglobal = cfine · ccoarse. For a certain point correspondence refined from patch correspondence (G̃Pi′ , G̃ P j′), we define cfine as its fine-level confidence score and ccoarse as S′(i′, j′).
Inlier Ratio.2 As the main contribution of CoFiNet is that we adopt the coarse-to-fine mechanism to avoid keypoint detection, while existing methods struggle to sample repeatable keypoints for matching, we first check the Inlier Ratio of CoFiNet, which is directly related to the quality of extracted correspondences. We show quantitative results in Tab. 1 and qualitative results in Fig. 3. As shown in Tab. 1, on Inlier Ratio, CoFiNet outperforms all the previous methods except PREDATOR [12] on 3DLoMatch and only performs worse than PREDATOR [12] and FCGF[31] on 3DMatch. Notably, when the sample number is 250, we perform the best on both datasets, since detection-based methods face a more severe repeatability problem in this case. By contrast, as our method leverages a coarseto-fine mechanism and thus avoids keypoint detection, it is more robust to the aforementioned case. Furthermore, the fact that sampling fewer correspondences leads to a higher Inlier Ratio indicates that our learned scores are well-calibrated, i.e., higher confidence scores indicate more reliable correspondences.
Reliability of our correspondences. Though Inlier Ratio is an important metric of correspondence quality, it is naturally affected by the distance threshold τ1. To better illustrate the reliability of correspondences extracted by CoFiNet and show our superiority over existing methods, we conduct another experiment and show related results in Tab. 2. In this experiment, we directly solve the relative poses using singular value decomposition (SVD) based on extracted correspondences, without the assistance of the robust estimator RANSAC [21]. As we can see, for FCGF [31] and D3Feat [10], though they can work on 3DMatch, they fail on 3DLoMatch, where point cloud share less overlap and thus reliable correspondences are harder to obtain. Compared with PREDATOR [12], on both 3DMatch and 3DLoMatch, our proposed CoFiNet performs much better, which indicates that we propose more reliable correspondences on both datasets.
Feature Matching Recall and Registration Recall. On Feature Matching Recall, CoFiNet significantly outperforms all the other methods on both 3DMatch and 3DLoMatch. Especially on 3DLoMatch, which is more challenging due to the low-overlap scenarios, our proposed method surpasses others with a large margin of more than 4%. It indicates that CoFiNet is more robust to different scenes, i.e., we find at least 5% inlier correspondences for more test cases. Additionally, we also follow [10, 12] to show the Feature Matching Recall in relation to τ2 and τ1 on 3DMatch in
Fig. 4, which further shows our superiority over other methods. When referring to the most important metric Registration Recall which better reflects the final performance on point cloud registration, though we perform slightly worse than PREDATOR [12], we significantly outperform others on 3DMatch. When evaluated on 3DLoMatch, our proposed approach significantly surpasses all the others, which shows the advantages of our method in scenarios with less overlap. Moreover, we also compare the number of parameters used in different methods in the last column of Tab.1, which shows that CoFiNet uses the least parameters while achieving the best performance.
Inlier Ratio and Registration Recall on the same correspondence set. In Tab. 1, PREDATOR [12] reports Inlier Ratio on a correspondence set that is different to the one used for registration, while CoFiNet uses the same. PREDATOR uses correspondences extracted by mutual selection to report Inlier Ratio, while computing Registration Recall on a correspondence set obtained by non-mutual selection. As we target at registration, we consider it meaningless to evaluate on a correspondence set that is not used for pose estimation. Thus, to make a fair comparison, we compare CoFiNet with both PREDATOR(mutual) and PREDATOR(non-mutual) in Tab. 3. In mutual selection, two points x and y are considered as a correspondence when x match to y and y match to x, while in non-mutual selection, the correspondence is extracted when x match to y or y match to x. In Tab. 3, compared to non-mutual, mutual selection rejects some outliers, and thus increases Inlier Ratio of PREDATOR. However, as it meanwhile filters out some inlier correspondences, when combined with RANSAC [21], Registration Recall usually drops. Since our task is registration, PREDATOR(nonmutual) with higher Registration Recall is preferred over itself with mutual selection. In this case, CoFiNet achieves higher Inlier Ratio than PREDATOR on both datasets.
Influence of the number of coarse correspondences. As illustrated in Tab. 4, on both 3DMatch and 3DLoMatch, when sampled only with τc, a higher threshold results in fewer coarse correspondences and meanwhile a higher Inlier Ratio, which indicates the learned
confidence scores are well-calibrated on the coarse level. However, Registration Recall drops at the same time, as the number of correspondences for refinement is decreased, and thus fewer point correspondences are leveraged in RANSAC for pose estimation. The last row is the strategy used in our paper. Except for τc, we also use τm to guarantee that CoFiNet samples at least τm coarse correspondences on each point cloud pair, as described before. This strategy slightly sacrifices Inlier Ratio but brings significant improvements on Registration Recall.
Importance of individual modules. As shown in Tab. 5, in the first experiment, we directly use the coarse correspondence set C′ for point cloud registration. Unsurprisingly, it performs worse on
all the metrics, indicating that CoFiNet benefits from refinement. Then, we ablate the weighting scheme which is proportional to overlap ratios and guides the coarse matching of down-sampled nodes. We replace it with a binary mask similar to the one used on the finer level. Results show that it leads to a worse performance, which proves that coarse matching of nodes benefits from our designed weighting scheme. Finally, we do the last ablation study on the density-adaptive matching module. Results indicate that on both 3DMatch and 3DLoMatch, with the density-adaptive matching module, CoFiNet better adapts to the irregular nature of point clouds.

heading: 4.2 KITTI
text: Metrics. We follow [12] and use 3 metrics, namely, the Relative Rotation Error (RRE), which is the geodesic distance between estimated and ground truth rotation matrices, the Relative Translation Error (RTE), which is the Euclidean distance between the estimated and ground truth translation, and the Registration Recall (RR) mentioned before. More details are provided in Appendix.
Comparisons to existing approaches. On KITTI, we compare CoFiNet to 3DFeat-net [43], FCGF [31], D3Feat [10] and PREDATOR [12]. Quantitative results can be found in Tab. 6, while qualitative results are given in Appendix. On RTE and RRE, we stay in the middle, but for RR, together with [10, 12], we perform the best. Notably, we achieve such performance by us-
ing only 5.48M parameters and training for 20 epochs compared to the best performing model [12], which uses over 20M parameters and is trained for 150 epochs. This experiment indicates that our model can deal with outdoor scenarios.

heading: 5 Conclusion
text: In this paper, we present a deep neural network that leverages a coarse-to-fine strategy to extract correspondences from unordered and irregularly sampled point clouds for registration. Our proposed model is capable of directly consuming unordered point sets and proposing reliable correspondences without the assistance of keypoints. To tackle the irregularity of point clouds, on a coarse scale, we first propose a weighting scheme proportional to local overlap ratios. It guides the model to match nodes that have overlapped vicinity areas, which significantly shrinks the search space of the following refinement. On a finer level, we then adopt a density-adaptive matching module, which eliminates the side effects from repeated sampling and enables our model to deal with density varying points. Extensive experiments on both indoor and outdoor benchmarks validate the effectiveness of our proposed model. We stay on par with the state-of-the-art approaches on 3DMatch and KITTI, while surpassing them on 3DLoMatch using a model with significantly fewer parameters. Limitations and broader impact are discussed in Appendix.

heading: Acknowledgments
text: We appreciate the valuable discussion with Zheng Qin and Hao Wu. We would like to thank Dr. Kai Wang for paper revision. Hao Yu is supported by China Scholarship Council (CSC).

Review summaries:
The paper proposed to study the problem of extracting correspondences for 3D point cloud registration. The paper presented CoFiNet (Coarse-to-Fine Network) to extract hierarchical correspondences from coarse to fine without detecting keypoints.  The model firstly learns to match down-sampled nodes and proposes node correspondences. On a finer scale, node proposals are consecutively expanded to patches that consist of groups of points together with associated descriptors. Patch correspondences are then refined to point level by a density-adaptive matching module. The proposed method has been evaluated on both indoor and outdoor standard benchmarks.

The paper presents a novel learning method for point cloud registration by extracting hierarchical correspondences in a coarse to fine manner. There are two modules:  Coarse scale:  a network is first utilized to downsample and extract feature from input  point cloud. The features are then strengthened and used to compute a score matrix for coarse correspondence proposals.  Fine scale: coarse correspondence proposals are expanded to patch correspondences via grouping and refined with a density-adaptive matching module.