CoFiNet: Reliable Coarse-to-fine Correspondences for Robust PointCloud Registration

1 Introduction . Correspondence search is a core topic of computer vision and establishing reliable correspondences is a key to success in many fundamental vision tasks , such as tracking , reconstruction , flow estimation , and particularly , point cloud registration . Point cloud registration aims at recovering the transformation between a pair of partially overlapped point clouds . It is a fundamental task in a wide range of real applications , including scene reconstruction , autonomous driving , simultaneous localization and mapping ( SLAM ) , etc . However , due to the unordered and irregular properties of point clouds , extracting reliable correspondences from them has been a challenging task for a long time . From early-stage hand-crafted methods [ 1 , 2 , 3 , 4 , 5 ] to recently emerged deep learning-based approaches [ 6 , 7 , 8 , 9 , 10 , 11 , 12 ] , many works contributed to improving the reliability of correspondences . We can broadly categorize recent deep learning-based point cloud registration methods into three categories . The first [ 13 , 14 , 15 ] follows the idea of ICP [ 16 ] , where they iteratively find dense correspondences and compute pose estimation . The second [ 17 , 18 ] includes the correspondence-free methods based on the intuition that the feature distance between two well-aligned point clouds should be small . Such methods encode the whole point cloud as a single feature and iteratively optimize the relative pose between two frames by minimizing the distance of corresponding features . Though achieving reasonable results on synthetic object datasets [ 19 ] , both of them struggle on large-scale real benchmarks [ 20 , 6 ] , as the first suffers from low correspondence precision and high computational complexity , while the second lacks robustness to noise and partial overlap . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Differently , the last category of methods [ 6 , 7 , 8 , 9 , 10 , 11 , 12 ] tackles point cloud registration in a two-stage manner . They firstly learn local descriptors of down-sampled sparse points ( nodes ) for matching , and afterward use robust pose estimators , e.g. , RANSAC [ 21 ] , for recovering the relative transformation . Their two-stage strategy makes them achieve state-of-the-art performance on largescale real benchmarks [ 20 , 6 ] . Uniform sampling [ 7 , 8 ] and keypoint detection [ 22 , 10 , 11 , 12 ] are two common ways to introduce sparsity . Compared to uniform sampling that samples points randomly , keypoint detection estimates the saliency of points and samples points with strong geometry features , which significantly reduces the ambiguity of matching . However , the sparsity by nature challenges repeatability , i.e. , sub-sampling increases the risk where a certain point loses its corresponding point in the other frame , which constrains the performance of detection-based methods [ 22 , 10 , 11 , 12 ] . Recently , a coarse-to-fine mechanism is leveraged by our 2D counterparts [ 23 , 24 , 25 ] to avoid direct keypoint detection , which shows superiority over the state-of-the-art detection-based method [ 26 ] . However , in 3D point cloud matching , where keypoint detectors usually perform worse , existing deep learning-based methods do not yet exploit such a coarse-to-fine strategy . To fill the gap , we focus on leveraging the coarse-to-fine mechanism to eliminate the side effects of detecting sparse keypoints . Nevertheless , designing such a coarse-to-fine pipeline in point cloud matching is non-trivial , mainly due to the inherent unordered and irregular nature of point clouds . To this end , we propose a weighting scheme for coarse node matching and a density-adaptive matching module for correspondence refinement , which enables CoFiNet to extract coarse-to-fine correspondences from point clouds . More specifically , on a coarse scale , the weighting scheme proportional to local overlap ratios guides the model to propose correspondences of nodes whose vicinity areas share more overlap , which effectively squeezes the search space of the consecutive refinement . On a finer scale , the density-adaptive matching module refines coarse correspondence proposals to point level by solving a differentiable optimal transport problem [ 26 ] with awareness to varying point density , which shows more robustness on irregular points . An overview of our proposed method can be found in Fig . 1 . Our main contributions are summarized as follows : • A detection-free learning framework that treats point cloud registration as a coarse-to-fine correspondence problem , where point correspondences are consecutively refined from coarse proposals that are extracted from unordered and irregular point clouds . • A weighting scheme that , on a coarse scale , guides our model to learn to match uniformly down-sampled nodes whose vicinity areas share more overlap , which significantly shrinks the search space for the refinement . • A differentiable density-adaptive matching module that refines coarse correspondences to point level based on solving an optimal transport problem with awareness to point density , which is more robust to the varying point density . To the best of our knowledge , we are the first deep learning-based work that incorporates a coarseto-fine mechanism in correspondence search for point cloud registration . Extensive experiments are conducted on both indoor and outdoor benchmarks to show our superiority . Notably , CoFiNet surpasses the state-of-the-art with much fewer parameters . Compared to [ 12 ] , we only use around two-third and one-fourth of parameters on indoor and outdoor benchmarks , respectively . 2 Related Work . Learned local descriptors . Early networks proposed to learn local descriptors for 3D correspondence search mainly take uniformly distributed local patches as input . As a pioneer , Zeng et al . [ 6 ] propose the 3DMatch benchmark , on which they exploit a Siamese network [ 27 ] that consumes voxel grids of TDFs ( Truncated Distance Fields ) to match local patches . PPFNet [ 7 ] directly consumes raw points augmented with point-pair features ( PPF ) by leveraging PointNet [ 28 ] as its encoder . PPF-FoldNet [ 8 ] leverages only PPF , which is naturally rotation-invariant , as its input and further incorporates a FoldingNet [ 29 ] architecture to enable the unsupervised training of rotation-invariant descriptors . Gojcic et al . [ 9 ] propose a network to consume the smoothed density value ( SDV ) representation aligned to the local reference frame ( LRF ) to eliminate the rotation-variance of learned descriptors . To extract better geometrical features , Graphite [ 11 ] utilizes graph neural networks for local patch description . SpinNet [ 30 ] utilizes LRF for patch alignment and 3D cylindrical convolution layers for feature extraction , achieving the best generalization ability to unseen datasets . However , patch-based methods usually suffer from low computational efficiency , as typically shared activations of adjacent patches are not reused . To address this , FCGF [ 31 ] makes the first attempt by using sparse convolutions [ 32 ] to compute dense descriptors of the whole point cloud in a single pass , which leads to 600x speed-up while still being able to achieve comparable performance to patch-based methods . Learned 3D keypoint detectors . USIP [ 22 ] learns to regress the position of the most salient point in each local patch in a self-supervised manner . However , it suffers from degenerated cases when the number of desired keypoints is relatively small . D3Feat [ 10 ] exploits a fully convolutional encoderdecoder architecture for joint dense detection and description . However , it does not consider overlap relationships and shows low robustness on low-overlap scenarios . In addition to jointly estimating salient scores and learning local descriptors , PREDATOR [ 12 ] also predicts dense overlap scores that indicate the confidence whether points are on the overlap regions . Keypoints will be sampled under the condition of both saliency and overlap scores . Though it surpasses existing methods by a large margin on both 3DMatch [ 6 ] and 3DLoMatch [ 12 ] , the precision of estimated scores and the repeatability of sampled keypoints constrain its performance . Coarse-to-fine correspondences . As witnessed in 2D image matching , many recent works [ 23 , 24 , 25 ] leverage a coarse-to-fine mechanism to eliminate the inherent repeatability problem in keypoint detection and thus boost the performance . DRC-Net [ 23 ] utilizes 4D cost volumes to enumerate all the possible matches and establishes pixel correspondences in a coarse-to-fine manner . Concurrently with DRC-Net , Patch2Pix [ 24 ] first establishes patch correspondences and then regresses pixel correspondences according to matched patches . In a similar coarse-to-fine manner with Patch2Pixel , LoFTR [ 25 ] leverages Transformers [ 33 ] , together with an optimal transport matching layer [ 26 ] , to match mutual-nearest patches on the coarse level , and then refines the corresponding pixel of the patch center on the finer level . 3 Methodology . Problem statement . Given a pair of unordered point sets X with n points xi ∈ R3 and Y with m points yj ∈ R3 , we aim at recovering the rigid transformation T X Y ∈ SE ( 3 ) between them . For simpler notation , we define their coordinate matrices as PX ∈ Rn×3 and PY ∈ Rm×3 , respectively . We follow the path of extracting correspondences first and then estimating the relative pose , where we mainly focus on the former . For this purpose , we propose CoFiNet that takes a pair of point clouds as input and outputs point correspondences , which can be leveraged to estimate the rigid transformation by RANSAC [ 21 ] . 3.1 Coarse-scale Matching . Point encoding . On the coarse level , our target is matching uniformly down-sampled nodes whose vicinity areas share more overlap . To achieve this goal , we first adopt shared KPConv [ 34 ] encoders to down-sample raw points to uniformly distributed nodes P′X ∈ Rn ′×3 and P′Y ∈ Rm ′×3 , while jointly learning their associated features F′X ∈ Rn ′×b and F′Y ∈ Rm ′×b . Demonstration of down-sampled nodes can be found in 1 ) of Fig . 1 . Please refer to Appendix for more details of network architecture . Attentional feature aggregation . As illustrated in Fig . 1 , the Correspondence Proposal Block ( CPB ) takes as input the down-sampled nodes and associated features . In CPB ( a ) , following [ 26 , 12 ] , the attention [ 33 ] mechanism is leveraged to incorporate more global contexts to the learned features . Following [ 12 ] , we adopt a sequence of self- , cross- and self-attention modules , which interactively aggregates global contexts across nodes from the same and the other frame in a pair of point clouds . Below we briefly introduce the cross-attention module . Given ( F′X , F ′ Y ) , akin to database retrieval , the former is linearly projected by a learnable matrix WQ ∈ Rb×b to query Q as Q = F′XWQ , while the latter is similarly projected to key K and value V by learnable matrices WK and WV , respectively . The attention matrix A is represented as A = QKT / √ b , whose rows are then normalized by softmax . The message M flows from F′Y to F ′ X is formulated as M = A ·V , which represents the linear combination of values weighted by the attention matrix . In the cross-attention module , contexts are aggregated bidirectionally , from F′X to F ′ Y and from F ′ Y to F ′ X . For computational efficiency , we replace the graph-based module [ 35 ] leveraged in [ 12 ] with the self-attention module in [ 26 ] , which has the same architecture as the cross-attention module but takes the features from the same point cloud , e.g. , ( F′X , F ′ X ) , as input . Correspondence proposal . As shown in CPB ( b ) of Fig . 1 , we leverage strengthened features F̃′X and F̃ ′ Y to calculate the similarity matrix . Down-sampled nodes whose vicinity areas share enough overlap are matched . However , there can be two cases where a node fails to match : 1 ) The major portion of its vicinity areas is occluded in the other frame . 2 ) Though most of its vicinity areas are visible in the other frame , there does not exist a node whose vicinity areas share sufficient overlap with its . Thus , for the similarity matrix , we expand it with a slack row and column with m′ and n′ slack entries , respectively [ 36 ] . So that nodes fail to match other nodes could match their corresponding slack entries , i.e. , having maximum scores there . Similar to [ 26 ] , we compute the similarity matrix using an inner product , which can be presented as : S′ = F̃′XF̃′ TY z zT z , S′ ∈ R ( n′+1 ) × ( m′+1 ) , ( 1 ) where all slack entries are set to the same learnable parameter z . On S′ we run the Sinkhorn Algorithm [ 37 , 38 , 39 ] , seeking an optimal solution for the optimal transport problem . After that , each entry ( i′ , j′ ) in the obtained matrix represents the matching confidence between the node i′ and node j′ from P′X and P ′ Y , respectively . To guarantee a higher recall , we adopt a threshold τc for likely correspondences whose confidence scores are above τc . We define the obtained coarse node correspondence set as C′ = { ( P′X ( i′ ) , P′Y ( j′ ) ) } , with |C′| = c′ , where |·| denotes the set cardinality . Furthermore , we set the other threshold τm to guarantee that c′ ≥ τm . When c′ < τm , we gradually decrease τc to extract more coarse node correspondences .

Summary: The paper presents a novel learning method for point cloud registration by extracting hierarchical correspondences in a coarse to fine manner. There are two modules:  Coarse scale:  a network is first utilized to downsample and extract feature from input  point cloud. The features are then strengthened and used to compute a score matrix for coarse correspondence proposals.  Fine scale: coarse correspondence proposals are expanded to patch correspondences via grouping and refined with a density-adaptive matching module.