Title: A Consistency-Aware Spot-Guided Transformer for Versatile and Hierarchical Point Cloud Registration
Venue: NeurIPS 2024
Accepted: yes
Keywords: Point cloud registration

Rigid transformation estimation

Feature matching

Correspondence

Deep learning
Abstract: Deep learning-based feature matching has shown great superiority for point cloud registration in the absence of pose priors. Although coarse-to-fine matching approaches are prevalent, the coarse matching of existing methods is typically sparse and loose without consideration of geometric consistency, which makes the subsequent fine matching rely on ineffective optimal transport and hypothesis-and-selection methods for consistency. Therefore, these methods are neither efficient nor scalable for real-time applications such as odometry in robotics. To address these issues, we design a consistency-aware spot-guided Transformer (CAST), which incorporates a spot-guided cross-attention module to avoid interfering with irrelevant areas, and a consistency-aware self-attention module to enhance matching capabilities with geometrically consistent correspondences. Furthermore, a lightweight fine matching module for both sparse keypoints and dense features can estimate the transformation accurately. Extensive experiments on both outdoor LiDAR point cloud datasets and indoor RGBD point cloud datasets demonstrate that our method achieves state-of-the-art accuracy, efficiency, and robustness.

A Consistency-Aware Spot-Guided Transformer for Versatile and Hierarchical Point Cloud Registration

Abstract

1 Introduction

In summary, our main contributions are as follows:

A consistency-aware spot-guided Transformer (CAST) with multi-scale feature fusion for much tighter coarse matching with a focus on geometric consistency.

A spot-guided cross-attention module with a consistency-aware matching confidence criterion that can maintain local consistency without interfering with irrelevant areas.

A consistency-aware self-attention module based on sparse sampling from the compatibility graph to enhance global consistency during feature aggregation.

A lightweight and scalable sparse-to-dense matching module involving both sparse keypoints and dense features to achieve lower registration errors without optimal transport and hypothesis-and-selection pipelines.

2 Related Work

3D Keypoint Detectors. Detection-based methods have been widely studied in image matching but less developed for 3D point clouds. Existing 3D keypoint detectors are mainly hand-crafted, which

extract salient points based on unique geometric features such as specific curvatures [21] or principal directions [22]. However, they suffer from noisy, sparse, and non-uniform real-world point clouds with large-scale transformations. Recent advances include learning-based detectors such as USIP [23] that predicts repeatable keypoints by minimizing a probabilistic chamfer loss, and HRegNet [24] that further utilizes weighted farthest point sampling to select sparse keypoints from the predicted ones for hierarchical registration. 3DFeat-Net [25] extracts patch-wise descriptors with saliency scores for keypoint selection in a weakly supervised manner by minimizing a weighted feature alignment triplet loss. D3Feat [9] adopts a fully convolutional network to predict point-wise descriptors with hand-crafted saliency scores by minimizing a self-supervised detection loss.

3 Method

In this section, we present the proposed consistency-aware spot-guided Transformer (CAST) with a lightweight sparse-to-dense fine matching module for accurate and efficient point cloud registration.

3.1 Overview

where R ∈ SO (3) and t ∈ R 3 are the rotation and the translation between X and Y , respectively.

Ƹ

Ƹ

Similarity Matrix

MLP

Correspondences

෡

𝐑,

𝐭

𝑭 𝟏/𝟖

×𝑳

𝑭 𝟏/𝟐

Self-Attention

Cross-Attention

Linear

Cross-Attention

Consistency-Aware

Self-Attention

Spot-Guided

Cross-Attention

…

Matching Scores

×

×

𝑭 𝟏/𝟒

෡

𝑭𝑿

𝑺

෡

𝑭𝒀

𝑺

ෝ

𝒐𝒀

ෝ

𝒐𝑿

෡

𝑭 𝑿

(𝒍)

෡

𝑭 𝒀

(𝒍)

෩

𝑭 𝒀

(𝒍)

෩

𝑭 𝑿

(𝒍)

Upsample&Fuse

Downsample&Fuse

Spot Selection

Signs:

Transformer-based Coarse Matching

𝑭 𝑿

𝟏/𝟖

𝑭 𝒀

𝟏/𝟖

𝑿

𝒀

𝑭 𝑿

𝟏/𝟒

𝑭 𝒀

𝟏/𝟒

Keypoint Detector

and Descriptor

Nearest

Assignment

Grouping

…

𝐱𝒊

𝐲𝒊

Keypoints

𝑿 𝟏/𝟒

, 𝑭

𝑿

𝟏/𝟒

𝒀 𝟏/𝟒

, 𝑭

𝒀

𝟏/𝟒

Compatibility

Single-Head Attention

Keypoint-to-Patch

Correspondences

…

𝐱𝒊

ෝ

𝒚𝒊

෡

𝒅𝒊

Confidence Weights

Sparse-to-Dense Fine Matching

Sparse Matching

Dense Matching

Weighted

SVD

෡

𝐑𝟎,

𝐭 𝟎

Weighted

SVD

Local

Attention

Alignment

…

…

# # #

# # # #

3

For efficient fine matching, we extract a keypoint from the neighborhood of each semi-dense node in X 1 / 4 , and predict its virtual correspondence in Y 1 / 4 based on the lightweight single-head attention. Then we utilize compatibility graph embedding to predict the confidence of these keypoint correspondences as weights in Eq. 1 for initial pose estimation. Finally, a lightweight local attention module for dense points X 1 / 2 and Y 1 / 2 predicts dense correspondences to refine the pose.

3.2 Consistency-Aware Spot-Guided Attention

To tackle the sparsity and looseness of coarse matching, we focus on feature aggregation among semi-dense features F 1 / 4 leveraging both local and global geometric consistency. To be specific, the self-attention only attends to salient nodes sampled from a global compatibility graph, while the cross-attention only attends to nodes sampled based on local consistency, which are referred to as consistency-aware self-attention and spot-guided cross-attention , respectively.

Preliminaries. Transformers stacked by alternate self-attention and cross-attention have showcased advanced performance in coarse feature matching. When D -dimensional features F A attends to F B , the output of vanilla attention is formulated as:

where W Q , W K , W V are learnable linear transformations to generate queries, keys, and values. When F A , F B related to coordinates P A , P B are from the same point cloud, it becomes self-attention that requires positional encoding to embed spatial information. To encode the 3D relative positions, we equip the rotary positional embedding [35] ˜ R ( · ) with learnable weights b 1 , · · · , b D/ 2 ∈ R 1 × 3 :

When applying ˜ R ( · ) to vanilla self-attention, the output is formulated as:

Architecture. As both spot-guided cross-attention and consistency-aware self-attention are sparse attention lacking of abundant global context, we propose to enhance the semi-dense features via multi-scale feature fusion with coarse features. Hence, the architecture of our coarse matching module is designed as a sequence of blocks for attention-based multi-scale feature aggregation. For each

Vanilla Global Attention     Spot-Guided Attention

Consistency-Aware

Self-Attention

Spot-Guided

Cross-Attention

Matching Matrix

Compatibility Graph

Generalized

Degrees

Matching

Scores

Consistency-Aware

Matching Confidence

Spot Selection

Spot

block with both semi-dense features F 1 / 4 and coarse features F 1 / 8 as inputs, we first feed F 1 / 8 into a self-attention module (Eq. 5) and a cross-attention module (Eq. 3). Then F 1 / 4 and F 1 / 8 are fused into each other based on nearest up-sampling and distance-based interpolated down-sampling [18]:

Finally, ˆ F 1 / 4 is fed into a consistency-aware self-attention module and a spot-guided cross-attention module at the end of each block. Before these sparse attention modules, we need to match the semi-dense features and evaluate the geometric consistency as a clue to select sparse yet instructive tokens. Given semi-dense features ˆ F ( l ) X , ˆ F ( l ) Y in the l -th block, the matching score is formulated as:

Then the correspondence of each node can be obtained as the node from another point cloud with the highest matching score, forming a correspondence set C ( l ) = { ( x S i , y S i ) : x S i ∈ X 1 / 4 , y S i ∈ Y 1 / 4 } . An insight about the consistency among correspondences is that the distance between two points is invariant after transformation. Hence, geometric compatibility is adopted as a simple yet effective measure of consistency [32], which is based on the length difference between pairwise line segments. Given a pre-defined threshold σ c , the pair-wise geometric compatibility of C ( l ) is formulated as:

The compatibility matrix B c = [ β ij ] M ′ × N ′ is also considered as the adjacency matrix of a weighted undirected graph known as the compatibility graph, where each vertex is a pair of correspondence and the edge connectivity corresponds to the compatibility between two correspondences. Intuitively, we adopt the generalized degree of a pair of correspondence in the graph as a measure of global consistency, which quantifies the connectivity of a vertex as the sum of edge weights connected to it.

Consistency-Aware Self-Attention. Intuitively, the correspondences of less salient nodes can be effectively located based on the geometric relationships between them and the salient ones. Hence, compared with global self-attention that attends to all nodes, attending to only salient nodes is more efficient and effective to encode the geometric context for matching. We propose the consistency-aware self-attention that samples sparse salient nodes to be attended to based on both geometric consistency and feature similarity. Given the correspondence set C ( l ) with a compatibility graph, we perform two-stage sampling by ranking the generalized degrees and matching scores, respectively. The first-stage graph sampling using generalized degrees can obtain sufficient consistent correspondences as proposals. The second-stage sampling based on matching scores can further obtain sparse salient nodes from these proposals. Finally, semi-dense features ˆ F ( l ) X , ˆ F ( l ) Y only attend to features of salient nodes from the same point cloud for feature aggregation according to Eq. 5.

Spot-Guided Cross-Attention. As shown in Figure 2, global cross-attention tends to aggregate features from many irrelevant regions with similar patterns, leading to false correspondences. Inspired by local consistency that the correspondences of adjacent 3D points remain close to each other, we design the spot-guided cross-attention as depicted in Figure 2. For each node x S i such that ( x S i , y S i ) ∈ C ( l ) , we select a subset N s ( x S i ) from its neighborhood N ( x S i ) as seeds, and construct a region of interest for it as S ( x S i ) = ⋃ x S k ∈N s ( x S i ) N ( y S k ) , namely its spot . N s ( x S i ) selects x S i and only its neighbors with reliable correspondences. We propose a consistency-aware matching confidence criterion to rank the neighbors, which is formulated as the product of the matching score and the normalized generalized degree in the compatibility graph. This criterion incorporates feature similarity and geometric consistency to properly measure the reliability of correspondences for seed selection. Finally, semi-dense features attend to their spots for feature aggregation according to Eq. 3. Under the guarantee of local consistency, the spots are likely to cover the true correspondences, providing guidance for feature aggregation without interfering with irrelevant areas.

3.3 Sparse-to-Dense Fine Matching

Given a coarse correspondence set ˆ C = { ( x S j , y S j ) : x S j ∈ X 1 / 4 , y S j ∈ Y 1 / 4 } selected as mutual nearest neighbors from the final coarse matching scores (Eq. 2), we propose a lightweight sparse-todense fine matching module for hierarchical pose estimation without optimal transport, maintaining scalability and efficiency. For sparse matching, we first search k -nearest neighbors (kNN) of semidense nodes X 1 / 4 among dense points X 1 / 2 to group patches, then we use an attentive keypoint detector [36] to predict a repeatable keypoint with a descriptor from each patch. Each keypoint of point cloud X is assigned to its nearest node, and each node y S j ∈ Y 1 / 4 with a correspondence in ˆ C groups a patch P ( y S j ) of keypoints via kNN. Then, a keypoint x i assigned to x S j will correspond to the patch P ( y S j ) , forming a pair of keypoint-to-patch correspondence. Finally, we utilize a shared singlehead attention layer for each keypoint-to-patch correspondence to predict virtual correspondences for keypoints. Denote the descriptor of x i as d X i , the virtual correspondence ˆ y i with feature ˆ d Y i is predicted from keypoints y i 1 , · · · , y i k with topk descriptor similarity in P ( y S j ) as:

where d Y i 1 , · · · , d Y i k are descriptors of y i 1 , · · · , y i k , and W Q and W K are learnable weights. Inspired by PointDSC [32], we construct a compatibility graph B (Eq. 8) of sparse keypoint correspondences { ( x i , ˆ y i ) } for spatial consistency filtering via compatibility graph embedding:

where E ( l ) is the correspondence-wise embedding of the l -th layer with learnable weights W ( l ) Q , W ( l ) K , W ( l ) V . Finally, the embedding is fed into an MLP to classify if a correspondence is an inlier. The predicted inlier confidences serve as the weights of keypoint correspondences for pose estimation formulated as Eq. 1, which can be analytically solved by weighted Kabsch algorithm [37]. It is noteworthy that the above process is really lightweight and scalable to large-scale registration tasks.

After aligning two point clouds based on sparse matching, we propose to refine the transformation based on dense matching. We still utilize local attention (Eq. 9) to predict the correspondences of dense points X 1 / 2 from its neighbors in Y 1 / 2 within a radius R d , and we simply set the confidence weight of a correspondence with a distance d as w = [1 -d/R d ] + . By solving Eq. 1 again with both sparse and dense correspondences, we can achieve more accurate pose estimation efficiently.

3.4 Loss Functions

Our loss function needs to supervise four modules, i.e. , keypoint detection, coarse matching, keypoint matching, and dense registration. For keypoint detection, we utilize the probabilistic chamfer loss [23] L p to minimize the distances between the closest keypoints from the source and target point clouds after alignment under the ground-truth transformation. Please refer to [23] for details.

Coarse Matching. Given the ground-truth coarse correspondence set C with an overlap ratio o ij for each correspondence ( i, j ) ∈ C , we propose a spot matching loss L s and a coarse matching loss L c formulated as weighted cross entropy losses to supervise the layer-wise coarse matching scores P ( l ) ( l = 1 , 2 , · · · , L ) and the final coarse matching scores P , respectively:

where N X and N Y are sets of semi-dense nodes in point clouds X and Y without correspondences, respectively. Two nodes are considered as a pair of coarse correspondence only when their groundtruth overlap ratio is greater than 0. Assuming that the patch centered at a point p ∈ R 3 is a spherical neighborhood of radius r , the overlapping ratio o ij of patches centered at p i ∈ X S and q j ∈ Y S with ground-truth translation t ∈ R 3 and rotation R ∈ SO (3) can be calculated by:

Keypoint Matching. As our keypoint matching module follows a three-stage pipeline including similarity calculation, correspondence prediction, and consistency filtering, it is reasonable to supervise these stages with three losses, respectively. Only valid keypoint-to-patch correspondences are supervised during training, i.e. , the distance of the keypoint x and its closest point p x in the patch C x is less than a pre-defined threshold R p > 0 , and the points whose distances from x are greater than a pre-defined threshold R n ≥ R p form a non-empty set N x ⊂ C x . We formulate the keypoint matching loss L f as an InfoNCE loss [38] with symmetric learnable weights W , which aims at maximizing the similarity between descriptors d x and d p x of true correspondences ( x, p x ) and minimizing the similarity between descriptors d x and d n x of false correspondences ( x, n x ) , n x ∈ N x .

For correspondence prediction, we adopt a L 2 loss L k = E ( x, ˆ y ) ∥ R x + t -ˆ y ∥ 2 for the predicted correspondences ˆ y of keypoints x from all valid keypoint-to-patch correspondences ( x, C x ) . For consistency filtering, we simply utilize a binary entropy loss L i to supervise the confidence scores of all keypoint correspondences. The binary ground-truth label of a keypoint correspondence ( x, ˆ y ) is 1 if and only if it is an inlier, i.e. , ∥ R x + t -ˆ y ∥ 2 is less than a threshold R f > 0 .

,

Dense Registration. Given the translation ˆ t and rotation ˆ R estimated by dense registration, we adopt a translation loss L t = ∥ ˆ t -t ∥ 2 and a rotation loss L r = ∥ ˆ R T R -I 3 × 3 ∥ F for supervision. Finally, we formulate our loss as L = L p + λ s L s + λ c L c + λ f L f + λ k L k + λ i L i + λ t L t + λ r L r where λ c , λ s , λ f , λ k , λ i , λ t , λ r are balancing weights.

4 Experiments

In this section, we evaluate our method on both outdoor LiDAR point cloud datasets KITTI [39], nuScenes [40], and the indoor RGBD point cloud dataset 3DMatch [16]. Our network is trained using an AdamW [41] optimizer with a batch size of 1, an initial learning rate of 1e-4, and a weight decay of 1e-4. The step scheduler decrease the learning rate to 90% every five steps, with gradients clipped at a norm of 0.5 during back propagation. Despite the complexity of the loss function, only one stage is needed for training. Our model is trained on an NVIDIA RTX 3090 GPU with an Intel Xeon CPU @2.90GHZ for 5, 40, and 3 epochs on 3DMatch, KITTI, and nuScenes, respectively, and we set λ f = λ i = 1 , λ r = 20 , λ t = 5 , λ s = 0 . 1 , and λ c = 0 . 2 , λ k = 1 for KITTI and nuScenes, λ c = 1 , λ k = 10 for 3DMatch.

Source Frame

with Keypoints

Target Frame

with Keypoints

Correspondences

4.1 Outdoor Scenarios: KITTI and NuScenes

KITTI [39] is a popular benchmark for autonomous driving. Following [9], we use sequences 0 to 5 for training, 6 to 7 for validation, and 8 to 10 for testing, and select only point cloud pairs at least 10m away from each other with ICP-refined [42] GPS localization results as ground truth. NuScenes [40] is another large-scale outdoor autonomous driving benchmark including 850 scenes for training and validation and 150 for testing. Following [24], we select each LiDAR keyframe with the second keyframe after it as a pair of point clouds. We use three metrics for evaluation [20]: relative translation error (RTE), relative rotation error (RRE), and registration recall (RR).

Our results on KITTI are detailed quantitatively in Table 1 and qualitatively in Figure 3. Table 1 shows that the proposed CAST outperforms various learning-based methods, including descriptor-based [25, 8, 9, 19, 20], coarse-to-fine correspondence-based [11, 12, 13] including the latest ones with iterative matching [29, 30], a recent graph-based [43] and an end-to-end [44] baselines. Specifically, CAST achieves a RR of 100.0% and the lowest RTE of 2.5cm, which is 60.3% improvement over the state-of-the-art DiffusionPCR [30], highlighting its superior robustness and accuracy.

As for RRE, CAST slightly underperforms some coarse-to-fine methods [12, 13, 29, 30], primarily due to numerical errors in SVD-based pose estimation that usually produces non-orthonormal rotation matrices. RRE is set as 0 when trace ( ˆ R T R ) > 3 , a condition met frequently across all methods. Consequently, geodesic distance-based RRE may not accurately reflect the actual performance.

For a more challenging LiDAR benchmark nuScenes, we compare CAST with both traditional [42, 45, 15] and learning-based algorithms [26, 27, 46, 31, 24] in Table 2. We do not include the coarse-to-fine methods since none have been trained or tested on nuScenes. Most of the results are borrowed from [24] while HRegNet [24] is re-evaluated with their open source codes. Our method achieves the lowest translation error of 0.12m and the lowest rotation error of 0.20° while maintaining the best RR of 99.9%, showcasing state-ofthe-art robustness and accuracy.

4.2 Indoor Scenarios: 3DMatch and 3DLoMatch

On the 3DMatch benchmark, our method achieves state-of-the-art RR of 95.2%. On the more challenging 3DLoMatch, CAST achieves a high RR of 75.1%, outperforming all descriptors and non-iterative correspondence-based methods [47, 11, 12, 28, 48, 49] except OIF-Net [13] using more than 1000 sampled points. As our method typically detects about 1000 sparse keypoints and establishes less than 250 keypoint correspondences on 3DLoMatch, it is fair to compare CAST with other methods using only 250 sample points. However, CAST outperforms the state-of-the-art non-iterative correspondence-based methods OIF-Net [13] using less than 1000 points. Notably, our method achieves such superior performance only with the lowest runtime, while RANSAC remains efficient due to our high inlier ratio. Although PEAL [29] and DiffusionPCR [30] show higher RR on 3DLoMatch, their iterative feature matching with overlap priors is extremely time-consuming (10 times of ours), while PEAL even requires extra information from 2D images.

4.3 Ablation Studies

We select indoor datasets for ablation studies of coarse matching as they are more challenging. Here we evaluate the RR over the whole dataset rather than the average RR of eight sequences reported in Table 3, which is more reasonable for a dataset with significant variances of sequence lengths. Besides, we assess two extra metrics to directly measure the performance of coarse matching: patch inlier ratio (PIR), the fraction of patch matches with actual overlap; and patch matching recall (PMR), the fraction of point cloud pairs with PIR above 20%. Results from five experiments in Table 4 demonstrate the effects of the proposed multi-scale feature fusion (MS), spot-guided cross-attention (SG), consistency-aware self-attention (CA), and the overlap head for overlap score prediction (OV). The first experiment ablating CA and replacing SG with linear cross-attention, suffers performance degradation in all metrics due to inconsistency. The second experiment improves all metrics based on SG, while the last one achieves the best performance via CA, showcasing their effectiveness. Figure 2 visualizes the vanilla global cross-attention and our spot-guided cross-attention. Instead of interacting with many similar yet irrelevant regions for misleading feature aggregation, SG can effectively select instructive areas to attend to according to local consistency. Compared to the last experiment, the third one verifies the effectiveness of multi-scale feature fusion, while the fourth one demonstrates the necessity of overlap prediction.

Additionally, we conducted five ablation studies on KITTI for a better understanding of our fine matching, since pose errors are better metrics to reflect accuracy. The second experiment using only sparse keypoints for registration highlights the effectiveness of dense registration, while the third one shows the effect of learnable dense correspondences compared to nearest neighbors. The last three experiments report the performance of sparse registration by

ablating the keypoint detector, the learnable sparse correspondences, and the compatibility graph embedding, each demonstrating their necessity for accuracy. Despite these variations, all studies maintain a 100% RR, showing the robustness of coarse matching.

5 Conclusion

In this paper, we present a novel consistency-aware spot-guided Transformer to achieve compact and consistent coarse matching for point cloud registration. At the coarse matching stage, our consistencyaware self-attention enhances the feature representations with sparse sampling from the geometric compatibility graph. Additionally, our spot-guided cross-attention leverages local consistency to guide the cross-attention to confident spots without interfering with irrelevant areas. Based on these semi-dense and consistent coarse correspondences, a lightweight and scalable sparse-to-dense fine matching module empowered by local attention can achieve accurate pose estimation without optimal transport or hypothesis-and-selection pipelines. Our method has showcased state-of-the-art accuracy, robustness, and efficiency for point cloud registration across different 3D sensors and scenarios, which paves the way for large-scale real-time applications such as SLAM.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (62203383/62088101).

References

Renlang Huang, Minglei Zhao, Jiming Chen, and Liang Li. Kdd-loam: Jointly learned keypoint detector and descriptors assisted lidar odometry and mapping. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 8559-8565, 2024.

Weitong Hua, Zhongxiang Zhou, Jun Wu, Huang Huang, Yue Wang, and Rong Xiong. Rede: End-to-end object 6d pose robust estimation using differentiable outliers elimination. IEEE Robotics and Automation Letters , 6(2):2886-2893, 2021.

Jianyuan Wang, Yiran Zhong, Yuchao Dai, Stan Birchfield, Kaihao Zhang, Nikolai Smolyanskiy, and Hongdong Li. Deep two-view structure-from-motion revisited. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8953-8962, 2021.

Radu Bogdan Rusu, Nico Blodow, Zoltan Csaba Marton, and Michael Beetz. Aligning point cloud views using persistent feature histograms. In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 3384-3391. IEEE, 2008.

Radu Bogdan Rusu, Nico Blodow, and Michael Beetz. Fast point feature histograms (fpfh) for 3d registration. In 2009 IEEE International Conference on Robotics and Automation , pages 3212-3217. IEEE, 2009.

Samuele Salti, Federico Tombari, and Luigi Di Stefano. Shot: Unique signatures of histograms for surface and texture description. Computer Vision and Image Understanding , 125:251-264, 2014.

Zan Gojcic, Caifa Zhou, Jan D Wegner, and Andreas Wieser. The perfect match: 3d point cloud matching with smoothed densities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5545-5554, 2019.

Christopher Choy, Jaesik Park, and Vladlen Koltun. Fully convolutional geometric features. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8958-8966, 2019.

Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, and Chiew-Lan Tai. D3feat: Joint learning of dense detection and description of 3d local features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6359-6367, 2020.

Jiahuan Yu, Jiahao Chang, Jianfeng He, Tianzhu Zhang, Jiyang Yu, and Feng Wu. Adaptive spot-guided transformer for consistent local feature matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21898-21908, 2023.

Hao Yu, Fu Li, Mahdi Saleh, Benjamin Busam, and Slobodan Ilic. Cofinet: Reliable coarse-to-fine correspondences for robust point cloud registration. Advances in Neural Information Processing Systems , 34:23872-23884, 2021.

Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, Slobodan Ilic, Dewen Hu, and Kai Xu. Geotransformer: Fast and robust point cloud registration with geometric transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Fan Yang, Lin Guo, Zhi Chen, and Wenbing Tao. One-inlier is first: Towards efficient position encoding for point cloud registration. Advances in Neural Information Processing Systems , 35:6982-6995, 2022.

Minhao Li, Zheng Qin, Zhirui Gao, Renjiao Yi, Chenyang Zhu, Yulan Guo, and Kai Xu. 2d3d-matr: 2d-3d matching transformer for detection-free registration between images and point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14128-14138, 2023.

Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM , 24(6):381-395, 1981.

Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1802-1811, 2017.

Haowen Deng, Tolga Birdal, and Slobodan Ilic. Ppfnet: Global context aware local features for robust 3d point matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 195-205, 2018.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 652-660, 2017.

Sheng Ao, Qingyong Hu, Bo Yang, Andrew Markham, and Yulan Guo. Spinnet: Learning a general surface descriptor for 3d point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11753-11762, 2021.

Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler. Predator: Registration of 3d point clouds with low overlap. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4267-4276, 2021.

Hui Chen and Bir Bhanu. 3d free-form object recognition in range images using local surface patches. Pattern Recognition Letters , 28(10):1252-1262, 2007.

Yu Zhong. Intrinsic shape signatures: A shape descriptor for 3d object recognition. In IEEE International Conference on Computer Vision Workshops, ICCV workshops , pages 689-696. IEEE, 2009.

Jiaxin Li and Gim Hee Lee. Usip: Unsupervised stable interest point detection from 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 361-370, 2019.

Fan Lu, Guang Chen, Yinlong Liu, Lijun Zhang, Sanqing Qu, Shu Liu, Rongqi Gu, and Changjun Jiang. Hregnet: A hierarchical network for efficient and accurate outdoor lidar point cloud registration. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Zi Jian Yew and Gim Hee Lee. 3dfeat-net: Weakly supervised local 3d features for point cloud registration. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 607-623, 2018.

Yue Wang and Justin M Solomon. Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3523-3532, 2019.

Jiahao Li, Changhao Zhang, Ziyao Xu, Hangning Zhou, and Chi Zhang. Iterative distance-aware similarity matrix convolution with mutual-supervised point elimination for efficient point cloud registration. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIV 16 , pages 378-394. Springer, 2020.

Hao Yu, Zheng Qin, Ji Hou, Mahdi Saleh, Dongsheng Li, Benjamin Busam, and Slobodan Ilic. Rotationinvariant transformer for point cloud matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5384-5393, 2023.

Junle Yu, Luwei Ren, Yu Zhang, Wenhui Zhou, Lili Lin, and Guojun Dai. Peal: Prior-embedded explicit attention learning for low-overlap point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17702-17711, 2023.

Zhi Chen, Yufan Ren, Tong Zhang, Zheng Dang, Wenbing Tao, Sabine Süsstrunk, and Mathieu Salzmann. Diffusionpcr: Diffusion models for robust multi-step point cloud registration. arXiv preprint arXiv:2312.03053 , 2023.

Christopher Choy, Wei Dong, and Vladlen Koltun. Deep global registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2514-2523, 2020.

Xuyang Bai, Zixin Luo, Lei Zhou, Hongkai Chen, Lei Li, Zeyu Hu, Hongbo Fu, and Chiew-Lan Tai. Pointdsc: Robust point cloud registration using deep spatial consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15859-15869, 2021.

Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6411-6420, 2019.

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning , pages 5156-5165. PMLR, 2020.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024.

Fan Lu, Guang Chen, Yinlong Liu, Zhongnan Qu, and Alois Knoll. Rskdd-net: Random sample-based keypoint detector and descriptor. Advances in Neural Information Processing Systems , 33:21297-21308, 2020.

Wolfgang Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography , 32(5):922-923, 1976.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In IEEE Conference on Computer Vision and Pattern Recognition , pages 3354-3361. IEEE, 2012.

Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:1903.11027 , 2019.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

P.J. Besl and Neil D. McKay. A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence , 14(2):239-256, 1992.

Xiyu Zhang, Jiaqi Yang, Shikun Zhang, and Yanning Zhang. 3d registration with maximal cliques. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1774517754, 2023.

Jiuming Liu, Guangming Wang, Zhe Liu, Chaokang Jiang, Marc Pollefeys, and Hesheng Wang. Regformer: An efficient projection-aware transformer network for large-scale point cloud registration. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 8417-8426, 2023.

Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Fast global registration. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 766-782. Springer, 2016.

Xiaoshui Huang, Guofeng Mei, and Jian Zhang. Feature-metric registration: A fast semi-supervised approach for robust point cloud registration without correspondences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11366-11374, 2020.

Zi Jian Yew and Gim Hee Lee. Regtr: End-to-end point cloud correspondences with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6677-6686, 2022.

Sheng Ao, Qingyong Hu, Hanyun Wang, Kai Xu, and Yulan Guo. Buffer: Balancing accuracy, efficiency, and generalizability in point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1255-1264, 2023.

Suyi Chen, Hao Xu, Ru Li, Guanghui Liu, Chi-Wing Fu, and Shuaicheng Liu. Sira-pcr: Sim-to-real adaptation for 3d point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14394-14405, 2023.

Haiping Wang, Yuan Liu, Zhen Dong, and Wenping Wang. You only hypothesize once: Point cloud registration with rotation-equivariant descriptors. In Proceedings of the 30th ACM International Conference on Multimedia , pages 1630-1641, 2022.

Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2930-2937, 2013.

Kevin Lai, Liefeng Bo, and Dieter Fox. Unsupervised feature learning for 3d scene labeling. In 2014 IEEE International Conference on Robotics and Automation (ICRA) , pages 3050-3057. IEEE, 2014.

Julien Valentin, Angela Dai, Matthias Nießner, Pushmeet Kohli, Philip Torr, Shahram Izadi, and Cem Keskin. Learning to navigate the energy landscape. In 2016 Fourth International Conference on 3D Vision (3DV) , pages 323-332. IEEE, 2016.

Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (ToG) , 36(4):1, 2017.

Jianxiong Xiao, Andrew Owens, and Antonio Torralba. Sun3d: A database of big spaces reconstructed using sfm and object labels. In Proceedings of the IEEE International Conference on Computer Vision , pages 1625-1632, 2013.

Maciej Halber and Thomas Funkhouser. Fine-to-coarse global registration of rgb-d scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1755-1764, 2017.

Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques , pages 303-312, 1996.

Heng Yang, Jingnan Shi, and Luca Carlone. Teaser: Fast and certifiable point cloud registration. IEEE Transactions on Robotics , 37(2):314-333, 2021.

Zhi Chen, Kun Sun, Fan Yang, and Wenbing Tao. Sc2-pcr: A second order spatial compatibility for efficient and robust point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13221-13231, 2022.

Yifei Zhang, Hao Zhao, Hongyang Li, and Siheng Chen. Fastmac: Stochastic spectral sampling of correspondence graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17857-17867, 2024.

François Pomerleau, M. Liu, Francis Colas, and Roland Siegwart. Challenging data sets for point cloud registration algorithms. The International Journal of Robotics Research , 31(14):1705-1711, December 2012.

M. Leordeanu and M. Hebert. A spectral technique for correspondence problems using pairwise constraints. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1 , pages 1482-1489, 2005.

A Appendix

In this appendix, we will first detail our neural network architecture with hyper-parameters in Sec. A.1. Then we introduce the related datasets in Sec. A.2 and provide more quantitative experimental results of our method with detailed explanation of evaluation metrics in Sec. A.3. Additionally, more qualitative results are illustrated in Sec. A.4. Finally, we will discuss the limitations and broader impacts in Sec. A.5 and Sec. A.6, respectively.

A.1 Neural Network Architectures and Hyper-parameters

Feature Pyramid Network. CAST utilizes a fully convolutional feature pyramid network as the backbone, which follows an encoder-decoder architecture based on KPConv [33] operations. Details of our network architecture are illustrated in Figure 4, which remain the same as [12] including five encoder layers and three decoder layers. Note that the backbone for indoor datasets 3DMatch [16] is slightly different, which only comprises four encoder layers and two decoder layers.

Coarse Matching Module. Figure 5 illustrates the details of our consistency-aware spot-guided attention blocks for coarse matching. Both coarse features and semi-dense features extracted from the backbone are first projected to 128 dimensions and then pass through three consistency-aware spot-guided attention blocks (Figure 1). Each attention module uses 4 heads with ReLU activation. Compared to vanilla attention with a quadratic increase in the size of the attention matrix with respect to the input length, the linear attention [34] is much more efficient in global context aggregation by replacing the softmax operator with the product of two kernel functions:

where ϕ ( · ) = elu ( · ) + 1 . For spot-guided cross-attention, each node x S i selects 4 seeds as N s ( x S i ) from the neighborhood N ( x S i ) with 12 nodes based on consistency-aware matching confidence. For consistency-aware self-attention, we first scale the generalized degrees to [0 , 1] , and sample 48 nodes with highest matching scores from nodes with scaled degrees greater than 0.3 as keys to be attended.

Fine Matching Modules. For keypoint detection and description, we utilize an attentive keypoint detector [36] to extract a keypoint with a descriptor from each local patch, which contains k = 32 nearest neighbors among dense points x i 1 , · · · , x i k ∈ X 1 / 2 with features f i 1 , · · · , f i k of a patch

KPConv (1, 64)

ResBlock (64, 128)

StridedResBlock (128, 128)

ResBlock (128, 256)

ResBlock (256, 256)

StridedResBlock (256, 256)

ResBlock (256, 512)

ResBlock (512, 512)

StridedResBlock (512, 512)

ResBlock (512, 1024)

ResBlock (1024, 1024)

StridedResBlock (1024, 1024)

ResBlock (1024, 2048)

ResBlock (2048, 2048)

Nearest Up-Sampling

Linear (3072, 1024)

C

Nearest Up-Sampling

C

Linear (1536, 512)

Nearest Up-Sampling

C

Linear (768, 256)

KPConv (in/2, in/2)

Linear (in, in/2)

Linear (in/2, in*2)

+

Linear (in, in*2)

Linear (in, in/4)

KPConv (in/4, in/4)

Linear (in/4, in)

+

Linear (in, in/4)

Max-Pool

KPConv (in/4, in/4)

Linear (in/4, in)

+

LeakyReLU

+

Add

C

Concat

ResBlock (in, in*2)

ResBlock (in, in)

StridedResBlock (in, in)

FC

Multi-Head

Self-Attention

Add & Norm

FC

FC

Multi-Head

Self-Attention

Multi-Head

Self-Attention

Add & Norm

Add & Norm

Feed Forward

Add & Norm

Feed Forward

Add & Norm

Multi-Head

Cross-Attention

Multi-Head

Cross-Attention

FC

Add & Norm

Add & Norm

Feed Forward

Feed Forward

Linear Attention

𝐗 𝐶

𝐘 𝑆

𝐘 𝐶

𝐗 𝑆

𝐅 𝑋

𝐶

𝐅 𝑌

𝐶

𝐅 𝑋

𝑆

𝐅 𝑌

𝑆

𝐐

𝐊

𝐕

𝐐

𝐊

𝐕

Add & Norm

Rotary

Positional

Encoding

𝐐

𝐊

𝐕

𝐐

𝐊

𝐕

Add & Norm

Add & Norm

Add & Norm

FC

Add & Norm

FC

3

4

1

2

3

Multi-Scale

Fusion

FC

FC

Add & Norm

Feed Forward

Multi-Head

Cross-Attention

Add & Norm

Feed Forward

Add & Norm

𝐐

𝐊

𝐕

FC

Multi-Head

Self-Attention

Add & Norm

Add & Norm

FC

Add & Norm

Feed Forward

Multi-Head

Cross-Attention

Add & Norm

Feed Forward

Add & Norm

𝐐

𝐊

𝐕

𝐐

𝐊

𝐕

𝐐

𝐊

𝐕

4

Sampling

Spot

Selection

Linear Attention

FC

FC

2

1

node x S i ∈ X S . The details of the keypoint detector and descriptor are illustrated in Figure 6, which are based on only shared MLPs with [ x i j , ∥ x i j -x S i ∥ 2 , f i j ] , j = 1 , · · · , k as inputs. For keypoints x K 1 , · · · , x K M ′ with uncertainties σ 1 , · · · , σ M ′ and y K 1 , · · · , y K N ′ with uncertainties σ 1 , · · · , σ N ′ from point cloud X and Y , respectively, we adopt the probabilistic chamfer loss in [23] for training, please refer to their paper for details.

To establish keypoint-to-patch correspondences based on coarse correspondences, each keypoint is assigned to its nearest node when their distance is below a threshold R k . Then the keypoint corresponds to the corresponding patch of this node containing k p keypoints. As mentioned in Sec. 3, we leverage a single-head attention layer to predict the virtual correspondence of each keypoint based on only k s keypoints

keypoints

MLP(260,64,64)

MLP(64,64,64,1)

attentive

softmax

scores

softplus

uncertainty

groupedfeatures

maxpool

concatenation

MLP(576,64,64)

maxpool

product

descriptor

sum

in its corresponding patch with the highest similarity (Eq. 9). Finally, spatial consistency filtering is performed via three graph compatibility embedding layers with embedding dimension D e = 64 in Eq. 10. After sparse matching and registration, we further refine the transformation based on dense matching, which searches k d nearest neighbors within a distance threshold R d for each dense point to predict its virtual dense correspondence. Specifically, we set k s = 4 , k d = 6 in our implementation, while other related hyper-parameters are listed in Table 6.

Weighted Kabsch Algorithm. The weighted Kabsch algorithm [37], also known as the Proscrutes algorithm, provides the closed-form solution for the point cloud registration problem (Eq. 1). Given a predicted correspondence set ˆ C = { ( x k , y k ) : k = 1 , · · · , n } , the optimal rigid transformation ˆ R ∈ SO (3) , ˆ t ∈ R 3 can be estimated via two steps:

Step 1. Centralize the point clouds by subtracting away their weighted centroids:

Step 2. Estimate the transformation. A 3 × 3 weighted covariance matrix can be computed as

With its singular value decomposition H = UΣV T , the optimal estimate of pose is given by:

A.2 Data and Benchmarks

3DMatch. 3DMatch [16] is a widely used RGBD point cloud dataset for 3D reconstruction, including 62 scenes from 7-Scenes [51], RGB-D Scenes V2 [52], Analysis-by-Synthesis [53], BundleFusion [54], SUN3D [55] and Halber et al. [56] with their licenses in Table 7, where 46 scenes are used for training, 8 scenes for validation and 8 scenes for testing. Input point clouds are generated by fusing 50 consecutive depth frames collected by RGBD cameras using TSDF volumetric fusion [57]. Different from the original 3DMatch [16] that only consists of point cloud pairs with >30% overlaps, [20] also includes point cloud pairs with overlaps between 10% and 30% for training, and it sets two benchmarks for performance evaluation, i.e. , 3DMatch consisting of point cloud pairs with >30% overlaps, and 3DLoMatch consisting of point cloud pairs with low overlap ratios between 10% and 30%. Following [12], we utilize the voxel grid down-sampling of 2.5cm voxel size for data preprocessing, which picks the centroid as a down-sampled point when multiple points fall into a common voxel grid. We leverage the data and evaluation protocols in [20] for training and testing.

KITTI. KITTI [39] is a classic benchmark under the NonCommercial-ShareAlike 3.0 License for a variety of computer vision tasks of autonomous driving, ranging from LiDAR-based or vision-based or multi-sensors based odometry, object detection and tracking, optical flow estimation, point cloud registration, etc . KITTI comprises of 11 sequences scanned by a Velodyne HDL-64 3D laser scanner in driving scenarios. Following the data splitting method in [8], we use sequences 0 to 5 for training, 6 to 7 for validation, and 8 to 10 for testing. Besides, we directly leverage the source code of [9] to

select point cloud pairs which are at least 10m away from each other, which leads to 1,358 training pairs, 180 validation pairs, and 555 testing pairs. Moreover, as the ground truth transformations provided by GPS are less accurate, we follow [9] to refine them via standard ICP [42] in 500 iterations. Following [9], we utilize the voxel down-sampling of 0.3m voxel size for data preprocessing.

NuScenes. NuScenes [40] is an outdoor autonomous driving datasets under CC BY-NC-SA 4.0 license. It is the first large-scale dataset to provide data from the entire sensor suite of an autonomous vehicle, consisting of 850 scenes for training and validation and 150 scenes for testing. Following [24], we select the first 700 scenes from the 850 scenes for training and the others for validation. The information about the point cloud pairs with ground-truth transformations is downloaded from the source codes of [24] which selects each LiDAR keyframe with the second keyframe after it as a pair. For data preprocessing, we apply 0.3m voxel grid down-sampling.

A.3 Evaluation Metrics with Extra Quantitative Results

The outdoor datasets KITTI [39] and nuScenes [40] commonly use three metrics for evaluation [20]: (1) relative translation error (RTE), the Euclidean distance between the estimated and ground-truth translation vectors; (2) relative rotation error (RRE), the geodesic distance between the estimated and ground-truth rotation matrices on SO (3) ; and (3) registration recall (RR), the percentage of point cloud pairs with RTE < 2m and RRE < 5°. Given the ground-truth rotation R ∈ SO (3) and translation t ∈ R 3 , as well as the estimated rotation ˆ R and translation ˆ t , the RRE and RTE are formulated as

The indoor benchmarks 3DMatch [16] and 3DLoMatch [20] commonly leverage three metrics for evaluation: registration recall (RR), inlier ratio (IR), and feature matching recall (FMR). To keep with existing methods [20], we exclude the consecutive point clouds when evaluating the RR.

Registration Recall refers to the percentage of point cloud pairs whose root mean square error (RMSE) is less than a pre-defined threshold τ = 0 . 2 m. Given point clouds X , Y with a groundtruth correspondence set C = { ( x i , y j ) : x i ∈ X , y j ∈ Y } , estimated rotation ˆ R ∈ SO (3) , and estimated translation ˆ t ∈ R 3 , the RMSE of X , Y is formulated as

Inlier Ratio refers to the percentage of estimated correspondences whose distance is less than a pre-defined threshold τ 1 = 0 . 1 m. Given point clouds X , Y with an estimated correspondence set ˆ C = { ( x i , y j ) : x i ∈ X , y j ∈ Y } , ground-truth rotation R ∈ SO (3) , and ground-truth translation t ∈ R 3 , the IR of X , Y is formulated as

Feature Matching Recall refers to the fraction of point cloud pairs with an IR > τ 2 = 5 %.

We adopt RR for evaluation as it directly measures the performance on the target task of point cloud registration, while FMR and IR are not suitable to quantize our matching performance for three reasons: (1) our method detects only sparse keypoints for fine matching, hence it is not fair to be compared with other methods based on dense descriptors or correspondences when using the same number of sampled points for evaluation; (2) our method directly supervises the differential pose estimator to learn virtual correspondences with confidence scores instead of explicit feature matching and outlier rejection, hence our method inevitably retains many outliers which are acceptable for registration but not likely to maintain advanced matching performance; (3) feature matching recall and inlier ratio only reflect the matching performance which is not decisive for either accuracy (relative pose errors) or robustness (registration recall) especially for RANSAC-free methods [12, 13, 28, 49].

Feature Matching Performance. Nevertheless, this appendix presents IR and FMR in Table 8 to demonstrate the feature matching performance of our method. For IR evaluation, we scale the inlier confidences predicted by compatibility graph embedding to [0 , 1] and discard the correspondences with confidences < 0 . 1 . As the number of keypoint correspondences after filtering is always less than 1000, we only report the IR regarding ≤ 1000 correspondences. Even without filtering, the number of keypoint correspondences is always less than 2500, hence we only report the FMR regarding ≤ 2500 correspondences. Enjoying the merits of our coarse matching, keypoint detection, and compatibility graph embedding, our method achieves the highest inlier ratio compared to all sorts of baselines. As for FMR, our method performs on par with DiffusionPCR [30] on 3DMatch and better than CoFiNet [11] on 3DLoMatch. However, our method performs worse than other coarse-to-fine methods [12, 13, 28, 29, 49, 30] on 3DLoMatch, since it remains challenging to extract enough keypoint correspondences in extremely low overlapping cases due to sparsity.

Indoor Registration Performance. We demonstrate the accuracy of CAST for indoor RGB-D point cloud registration by comparing it with various point cloud registration methods [15, 58, 59, 31, 32, 43, 60] in Table 9. All of the registration methods leverage the prevalent FCGF descriptor [8], and FastMAC [60] uses a sampling ratio of 50%. For a fair comparison, we follow the evaluation strategy of MAC [43] to re-compute the registration recall of our method, which is formulated as the fraction of point cloud pairs with RTE < 30cm and RRE < 15°. Our method achieves the highest registration recall and the lowest registration errors, suggesting its robustness and accuracy.

Generalization Studies. To extensively evaluate the generalizability of the proposed CAST in unseen domains, we conduct a generalization experiment from the outdoor dataset KITTI [39] to another outdoor dataset ETH [61]. Note that the KITTI and ETH datasets use Velodyne-64 3D LiDAR and Hokuyo 2D LiDAR, respectively, leading to very different appearances and distributions of point clouds. Hence, our generalization study is practical in applications and solid to demonstrate the generalizability of different methods. For fairness, all methods adopt 30cm for voxel down-sampling, and all methods involving RANSAC set the maximum iterations to be 50000 and the confidence to be 0.999 as the convergence criteria. To enhance the robustness, our method is combined with RANSAC estimating an initial pose from 250 coarse correspondences to reject the outliers during fine matching, and utilizes the global registration in GeoTransformer [12] to refine the pose estimate.

We present the translation errors, rotation errors, and the registration recalls in Table 11. Our method achieves satisfying accuracy and robustness, showcasing better generalizability than the coarse-to-fine baseline GeoTransformer [12] and other point-wise descriptors [8, 1, 20]. We also compare the our learnable compatibility graph embedding (CGE) with spectral matching (SM) [62] for outlier rejection in our method. With RANSAC filtering out severe outliers in advance, spectral matching can lead to better performance than learning-based CGE in unseen domains. Notably, all point-wise methods including CAST exhibit lower registration recalls in generalization studies than patch-wise local descriptor SpinNet [19] and BUFFER [48] incorporating patch-wise and point-wise features. This is mainly because they adopt a feature pyramid network architecture to learn features with abundant global context, which is detrimental for generalization [19]. Furthermore, we conduct an unsupervised domain adaptation (UDA) experiment for CAST, which tunes the network by learning to align a point cloud to itself after random rotation and cropping. The results indicate that our model can easily adapt to an unseen domain and achieve robust and accurate performance after one epoch's unsupervised tuning (only 20 minutes on an NVIDIA RTX3090 GPU).

Experiment Statistical Significance. Finally, Table 10 reports the standard deviations (1-sigma) of our evaluation metrics, which are assumed to be Gaussian distributed. Despite the randomness from voxel down-sampling and RANSAC, the performance of our method remains stable. Notably, the runtime of some methods such as [8, 19, 50] in Table 3 are quite different from results in [48], since we report the average runtime including data preprocessing, feature extraction, feature matching, and pose estimation, while the source codes of these methods save some intermediate results such as descriptors to avoid repeated calculation of the same point cloud in different pairs, which leads to unfair runtime comparison.

A.4 Qualitative Results

Figure 3, Figure 7, and Figure 8 provide qualitative results about the registration performance on outdoor datasets KITTI [39], nuScenes [40], and the indoor dataset 3DMatch [16], respectively.

A.5 Limitation

The main limitation of the proposed CAST is the sub-optimal performance in low overlapping scenarios such as the 3DLoMatch benchmark compared to state-of-the-art methods, which may be ascribed to two aspects. (1) There is no effective outlier rejection for coarse matching as it is difficult to search inliers from patch correspondences based on geometry consistency due to low resolution. (2) Due to the sparsity and non-uniformity of keypoints, the inlier ratio of keypoint correspondences still falls short of what is required for robust pose estimation without a hypothesis-and-selection pipeline. Nevertheless, this RANSAC-free lightweight fine matching pipeline can achieve satisfying performance in outdoor scenarios. Considering the superior PIR and PMR of our coarse matching, we may directly exploit dense feature matching to enhance the robustness in low overlapping point cloud registration scenarios as a future work.

A.6 Broader Impacts

We present a novel consistency-aware spot-guided Transformer based on sparse attention to extract consistent coarse correspondences from point clouds. In addition, we propose a lightweight fine matching module for versatile and hierarchical point cloud registration, benefiting from the efficiency of sparse keypoint matching and the accuracy of dense registration. Different from existing methods, our fine matching is based on flexible local attention instead of optimal transport heavily relying on patch-to-patch correspondences, thus allowing independent deployment without coarse matching. Besides, the sparsity of keypoints ensures the efficiency of spatial consistency filtering.

Enjoying these merits, this work not only achieves superior accuracy, efficiency, and robustness in point cloud registration, but also paves the way to various large-scale real-time applications, such as SfM, SLAM, autonomous driving, or any other where point cloud registration plays a role. For examples, the reconstruction of indoor scenes and objects from unlabeled 3D scans could benefit from our work, which can precisely recover the rigid transformation between different scans. Additionally, our fine matching may independently construct a real-time LiDAR-based or RGBD camera-based odometry system for SLAM or SfM, as it is capable of efficient and reliable local data association and accurate pose estimation between two large-scale point clouds with a strong pose prior, while our coarse matching could be utilized in place recognition and global re-localization in SLAM.

As our work aims at tackling a fundamental problem in 3D computer vision, we do not anticipate a direct negative impact. Potential negative outcomes might occur in real applications where our method is involved.

Source Frame

with Keypoints

Target Frame

with Keypoints

Correspondences

RMSE: 3.30m

RMSE: 2.92m

RMSE: 3.71m

RMSE: 0.16m

RMSE: 4.24m

RMSE: 4.87m

RMSE: 4.31m

RMSE: 0.09m

RMSE: 0.21m

RMSE: 0.21m

RMSE: 0.30m

RMSE: 0.16m

RMSE: 2.90m

RMSE: 3.95m

RMSE: 3.94m

RMSE: 0.06m

RMSE: 0.29m

RMSE: 2.94m

RMSE: 1.53m

RMSE: 0.07m

CoFiNet

GeoTransformer

RoITr

Ours (CAST)

Ground-Truth

1

1

1

1

1

1

1

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction clearly state the claims made, including the motivations, contributions, and the performance of our approach.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have detailed the limitations of our work in Sec. A.5 of the appendix, and we also point out the scope of our claims made, including the benchmarks and sensors (Sec. A.2) and the computational efficiency (Sec. 4), etc .

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include any theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This paper fully discloses all the information needed to reproduce the main experimental results, including the detailed system architecture in Sec. 3 and some modular architectures along with the hyper-parameters in Sec. A.1, the training and evaluation settings in Sec. 4, and datasets with metrics in Sec. A.2 and Sec. A.3, respectively.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have submitted the data and codes in the supplementary material with detailed instructions on data access and preparation as well as guidelines to reproduce all experimental results. The paper will provide public access to the codes upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: This paper fully specifies all the training and evaluation settings, including the optimizer and the learning rate scheduler in Sec. 4, hyper-parameters in Sec. A, data splits and preparation in Sec. A.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In this paper, all metrics are assumed to be normally distributed, whose empirical standard deviations (1-sigma) in repeated experiments are reported in Table 10 to demonstrate the experiment statistical significance. The randomness mainly comes from voxel down-sampling and RANSAC.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Sec. 4, we have detailed the type of CPU and GPU of our device, and the runtime of the proposed method and nearly all baselines on our computer.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This research conforms with the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broader impacts are discussed in Sec. A.6 of the appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: This paper involves existing benchmarks and utilizes existing methods for evaluation, whose papers are properly cited with licenses. The related URL are included in our codes.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: This paper provides details of our model in both main part (Sec. 3) and the appendix (Sec. A.1). Our source codes have been submitted in the supplementary material while no new datasets are proposed.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed a coarse-to-fine matching approach for point cloud registration, where a consistency-aware spot-guided transformer is proposed to ensure the coarse matches to be geometric consistent and around the overlap regions. This method is compared with SOTA PCR methods on three benchmarks and the experiment results validate its superior performance in terms of reregistration accuracy, success rate and time cost.\n\nSTRENGTHS:\n1. Extensive experiments and SOTA results. The proposed method is evaluated on three benchmarks including two outdoor and one indoor cases, where challenging cases such as low overlapping data are also included. It achieves the SOTA performance in terms of registration accuracy, success rate,  time cost. \n\n2. The idea of incorporating geometric consistency to increase the distinctiveness of the learned feature is novel. For pure 3D descriptor methods, such as SPINNet, YOHO, only rotation-equivalence is considered. With the help of geometric consistency, the proposed method showed superior feature matching performance in Table 3. \n\n2. The wiriting are clear and easy to follow.\n\nWEAKNESSES:\nI don’t see any significant weakness of this work. Some minor comments are here:\n\n1.  In line 123, “third level” and “fourth level” need to be explained more or illustrated in the Figure 1. \n\n2.  Please clear state the proposed contribution or difference with respect to the existing work. For instance, the “spot”, cross-attention, self-attention are also discussed in DiffusionPCR, GeoTransformer, and geometric consistency is widely applied by many PCR method, such as PointDSC, RANSAC, Spectral Matching [1].\n\n3. Even though the additional experiment is not encouraged during this process, I would recommend the generalizability of the proposed method can be discussed. I can imagine the proposed method should have better generalizability than existing correspondence-based methods, due to the help of geometric consistency. But, how about it compared to description-based methods (SpinNet, YOHO) with RANSAC/PointDSC ? Trained on indoor dataset and test on outdoor cases?  \n\n[1] A spectral technique for correspondence problems using pairwise constraints. ICCV 2015.\n\nQUESTIONS:\nIn table 3, the performance of CAST+RANSAC is better than CAST means the proposed coarse-to-fine matching approach is not strictly geometric consistent, which arises a question that instead of introducing the geometric consistency to the feature matching process, why not making it as a separate and following process, just like RANSAC or Spectral Matching ?\n\nLIMITATIONS:\nI didn’t see any limitation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a consistency-aware, spot-guided Transformer, adapted from 2D image matching techniques, to minimize interference from irrelevant areas. It incorporates a consistency-aware self-attention module that enhances matching capabilities by using edge length constraints to filter correspondences. Additionally, a lightweight fine matching module is designed for both sparse keypoints and dense features, enabling accurate transformation estimation. The method performs well on outdoor datasets; however, its effectiveness diminishes in scenarios with low overlap.\n\nSTRENGTHS:\nThe motivation to make learning-based registration more efficient and scalable for real-time applications is both interesting and important. I appreciate the introduction of the edge equality constraint into the registration pipeline, which guides the correspondence search in a more efficient manner compared to COTReg [1].\n\nThe method performs well on outdoor datasets; however, its effectiveness diminishes in scenarios with low overlap.\n\nWEAKNESSES:\nThe method section of this paper is challenging to follow. It would be beneficial to introduce each method sequentially, aligned with the flow depicted in the figure—for example, starting with self-attention, followed by cross-attention, and then linear cross-attention. \n\nThe captions of Figure 1 should clearly label the coarse and fine matching modules, as they are not marked in the current figures, making it difficult to understand. Furthermore, it's unclear which network layer corresponds to \"the third level\" mentioned in line 123—is it part of the encoder or decoder? It is also hard to distinguish between spot attention and other forms in Figure 1. \n\nThe methodology for obtaining overlap scores is not adequately explained and is not illustrated in Figure 1. \n\nThe performance on datasets with low overlap seems subpar, which is concerning since handling cases of low partial overlap is crucial.\n\nThe authors claim that existing methods are not efficient or scalable for real-time applications like odometry in robotics; however, only time comparisons are provided in Table 3, and these do not clearly demonstrate an advantage. Additionally, there is no assessment of performance on large-scale datasets. Therefore, the authors should provide more comprehensive evaluations, including comparisons of other metrics such as FLOPS.\n\nThis paper lacks visualizations for the spot-guided attention, which are essential for demonstrating the method's effectiveness.\n\nThe manuscript lacks critical references such as COTReg [1] by Mei et al., which discusses geometric consistency and should be cited. Comparisons with state-of-the-art methods cited in references [2] and [3] are also necessary to validate the proposed method's efficacy.\n\n[1] Mei, Guofeng, et al. \"COTReg: Coupled optimal transport based point cloud registration.\" arXiv preprint arXiv:2112.14381 (2021).\n[2] Zhang, Yifei, et al. \"FastMAC: Stochastic Spectral Sampling of Correspondence Graph.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[3] Huang, Tianyu, et al. \"Scalable 3d registration via truncated entry-wise absolute residuals.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\nThe paper does not specify if the loss supervises the overlap scores, it does not require it?\n\nThe term \"partial tokens\" mentioned in line 180 needs clarification. \nDetails on computing the coarse matching for spot-guided attention are also needed.\n\nSuggestion: \n1. Ensure consistency in font sizes across Tables 3 and 4.\n2. The terminology in line 123 could be improved by using \"superpoints\" or \"nodes\" instead of \"patches,\" as the latter suggests a collection of points rather than individual points or nodes.\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript mainly focuses on the learning based feature matching of point cloud registration. The authors propose a consistency-aware spot-guided transformer and a lightweight fine matching module. Experiments on both indoor and outdoor benchmarks prove the effectiveness of the designs. However, this article requires significant revisions and improvements in the narrative of the method introduction. It needs to provide a clearer sequence of the entire process, the inputs and outputs of each module, and the specific implementations of key designs.\n\nSTRENGTHS:\n1. It is reasonable to improve the attention architecture by utilizing geometric consistency constraints. \n\n2. Effective registration result improvements have been achieved on both indoor and outdoor datasets.\n\nWEAKNESSES:\n1. Figure 1 should be presented with greater clarity and precision. Currently, the coarse matching module and fine matching module are not labeled in the figure, making it difficult to correspond the depicted process with the title description.\n2. In section 3.2 of the Methods part, the definition of “spot” by the authors is not clear. “Spot” seems to be a key concept in this section, yet it is only briefly described in line 145. Is “spot” referring to a patch? How does it guide the cross-attention?\n3. The logic in the Methods section is not sufficiently clear, requiring repeated and careful reading. The inputs and outputs of each module are not explicitly stated. For instance, what are the inputs and outputs of the Sparse-to-Dense Fine Matching module in section 3.3? This module is also not represented in the flowchart.\n4. What is the sequence of the Consistency-Aware Self-Attention and Spot-Guided Cross-Attention in the process? Figure 1 shows that self-attention is performed first, so it is recommended to follow the same sequence in the detailed introduction.\n5. In Formula 4, is the pre-defined threshold σc set differently for datasets of varying sizes?\n6. How generalizable is the method proposed in this paper? If trained on the 3DMatch dataset, how would the results be when tested on outdoor datasets such as ETH?\n7. The method designed in this paper seems reasonable and effective. It is suggested that the authors make the code open-source for others to learn from.\n\nQUESTIONS:\nPlease see Weaknesses.\n\nLIMITATIONS:\nPlease see Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper focuses on feature matching for point cloud registration. To this end, it aims to improve the effectiveness of the coarse-to-fine matching mechanism by designing a consistency-aware spot-guided Transformer (CAST). More specifically, the proposed method incorporates a spot-guided cross-attention module to avoid interfering with irrelevant areas, and a consistency-aware self-attention module to enhance matching capabilities. Extensive experiments on both indoor and outdoor datasets are conducted to validate the proposed model.

STRENGTHS:
1. The overall writing is satisfied;

2. The experiments on outdoor benchmarks look good

WEAKNESSES:
1. One major issue of this paper is the novelty of this paper. It looks like the proposed method roughly follows the pipeline of GeoTransformer, but introduces outlier rejection techniques and more attention layers. Moreover, it is also necessary to show the size of the proposed model (in terms of the number of parameters)

2. The second major concern lies in the performance on indoor benchmarks including 3DMatch and 3DLoMatch. As aforementioned, with many modules added to GeoTransformer, the proposed method even fails to outperform it. The advantages of the proposed method on indoor scenarios should be better demonstrated.

QUESTIONS:
See weaknesses.

LIMITATIONS:
Limitations have been included as a part of the Appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12