A Consistency-Aware Spot-Guided Transformer for Versatile and Hierarchical Point Cloud Registration
Authors: Renlang Huang
Yufan Tang
Jiming Chen
Liang Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Point cloud registration
Rigid transformation estimation
Feature matching
Correspondence
Deep learning
Primary area: machine_vision
TL;DR: We propose a novel consistency-aware spot-guided Transformer for versatile and hierarchical point cloud registration, achieving state-of-the-art accuracy, efficiency, and robustness on both outdoor and indoor benchmarks.

Abstract: Deep learning-based feature matching has shown great superiority for point cloud registration in the absence of pose priors. Although coarse-to-fine matching approaches are prevalent, the coarse matching of existing methods is typically sparse and loose without consideration of geometric consistency, which makes the subsequent fine matching rely on ineffective optimal transport and hypothesis-and-selection methods for consistency. Therefore, these methods are neither efficient nor scalable for real-time applications such as odometry in robotics. To address these issues, we design a consistency-aware spot-guided Transformer (CAST), which incorporates a spot-guided cross-attention module to avoid interfering with irrelevant areas, and a consistency-aware self-attention module to enhance matching capabilities with geometrically consistent correspondences. Furthermore, a lightweight fine matching module for both sparse keypoints and dense features can estimate the transformation accurately. Extensive experiments on both outdoor LiDAR point cloud datasets and indoor RGBD point cloud datasets demonstrate that our method achieves state-of-the-art accuracy, efficiency, and robustness.

A Consistency-Aware Spot-Guided Transformer for Versatile and Hierarchical Point Cloud Registration
Renlang Huang
Yufan Tang Jiming Chen College of Control Science and Engineering Zhejiang University, Hangzhou 310027, China
Liang Li"
{renlanghuang, tyfan, cjm, liang. 1i azju. edu. cn
Abstract
Deep learning-bast -aaning--asedddd feature matching has shown great superiority for point cloud registration in the absence of pose priors. Although coarse-to-fine matching ap- proaches are prevalent, the coarse matching of existing methods is typically sparse and loose without consideration of geometric consistency, which makes the sub- sequent fine matching rely on ineffective optimal transport and hypothesis-ande selection methods for consistency. Therefore, these methods neither efficient nor scalable for real-time applications such as odometry in robotics. To address these issues, we design a consistency-awarg spot-guided Transformer (CAST), which incorporates a spot-guided cross-attention module to avoid interfering with irrelevant areas, and consistency-awarr self-attention module to enhance match- ing capabilities with geometrically consistent correspondences. Furthermore, a lightweight fine matching module for both sparse keypoints and dense features can estimate the transformation accurately. Extensive experiments on both outdoor LiDAR point cloud datasets and indoor RGBD point cloud datasets demonstrate that our method achieves state-of-the-ar accuracy, efficiency, and robustness. Our code is available at hhttps: htt///gituuu thub com Ren Ben.angguangg/CCSSS
Introduction
Point cloud registration is a fundamental yet crucial task for variety of 3D vision and robotic applications, such as simultancous localization and mapping (SLAM) (I. object pose estimation 2 and structure from motion (SfM) 3. Aiming at aligning two partially overlapped point clouds, the typical approach involves a two-stage pipeline: data association which establishes reliable point correspondences, and pose estimation. However, establishing these correspondences has been challenging due to the noisy, irregular, non-uniform, and textureless nature of 3D point clouds.
Feature matching has long been the mainstream of data association without pose priors. Extensive research has made advances in distinctive local feature representations, ranging from hand-crafted 6 to recent learning-basee descriptors 7 g 9 Although the emerging learning- based descriptors significantly improve the reliability of correspondences, the inlier ratio still falls short of what is required for robust and efficient pose estimation. Recently, coarse-to-fine matching is a thriving framework for 2D-2D 100, 3D-3D III [3, and even 2D-3D [14] data association. It has been consensus that Transformers stacked by alternate self-attention and cross attention modules are effective for coarse matching, which are inspired by human visual processes. Typically. humans may first scan through the point clouds to identify and match salient landmarks across different point clouds reliably. For less salient points, the geometric relationships between them and those salient landmarks would be utilized to revisit their potential correspondences. The correspondences will eventually be established for the entire point cloud after several iterations of this process.
descriptors
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Unfortunately, existing coarse matching approaches tend to be sparse and loose without consideration of geometric consistency, An important reason for looseness is that global Pross-attention inevitably attends to similar yet irrelevant areas, resulting in misleading feature aggregation and consequent inconsistent correspondences that undermine both robustness and accuracy. As a result, the hypothesis- and-selection pipeline such as RANSAC [1] is commonly used for outlier rejection, which is typically inaccurate and inefficient, especially for numerous samples with low inlier ratio. Furthermore, the sparsity necessitates the use of complicated fine matching such as optimal transport-based algorithms to establish reliable dense correspondences. Due to iterative dense matrix operations for patch-to- patch correspondences established by coarse matching, these fine matching methods are neither efficient nor scalable for real-tine time-time large-scale applications such as odometry.
To this end, we attempt to design an efficient and scalable yoarse-to-fing matching network based on consistency-aware semi-dense coarse correspondences. Inspired by ASTR [10] for 2D) feature matching, we leverage local consistency to direct the cross-attention of each point exclusively to corresponding patches of its confident neighbors, which referred to as spot-guided cross- attention. Unlike [10], we propose novel consistency-awant matching confidence criterion sample reliable neighbors based on both feature similarity and geometric compatibility. Additionally, we design consistency-awart
self-attention module to enhance the distinctiveness of coarse feature representations via aggregation with salient nodes from the compatibility graph. Notably, both spot- guided cross- coss-attentiin and consistency-awee aware self-attention are efficient sparse attention mechanisms.
For scalability to real-time applications such as odometry with pose priors, we propose lightweight fine matching module allowing independent deployment without coarse matching. The scalability credited to flexible point-to-patch local matching instead of optimal transport heavily relying on patch-to-patch correspondences. In addition, our fine matching adopts sparse-to-dense registration pipeline, benefiting from the efficiency of sparse keypoint matching and the accuracy of dense registration. Furthermore, efficient compatibility graph embedding module is leveraged for outlier rejection as a substitute for inefficient eypothesis-anddselees
pipelines.
In summary, our main contributions are as follows:
A consistency-awant spot-guided Transformer (CAST) with multi-scale feature fusion for much tighter coarse matching with focus on geometric consistency.
A spot-guides aooss-attetioon module with consistency- awarre matching confidence crite- rion that can maintain local consistency without interfering with irrelevant areas.
A consisteney-awe awaar self-attention module based on sparse sampling from the compatibility graph to enhance global consistency during feature aggregation.
A lightweight and scalable sparse-to-dense matching module involving both sparse key- points and dense features to achieve lower registration errors without optimal transport and hypothesis-ande and-sclection pipelines.
2
Related Work
3D Feature Deseriptors. Feature matching plays a crucial role in point cloud registration, enabling the establishment of reliable correspondences without pose priors. Early methods use hand-crafted descriptors based on signatures [ or histograms [4] (S) to represent local geometric features. Recently, learning-based 3D descriptors have showcased greater performance than hand-crafted ones, which are usually trained in a self-supervised manner by maximizing the similarity between descriptors of true correspondences and minimizing the similarity otherwise. 3DMatch (16] and PerfectMatch ( leverage 3D CNNs to learn local patch-wise descriptors from 3D patches converted into voxels of truncated distance function (TDF) values and smoothed density value (SDV) representations. respectively. PPFNet [I7] extracts global context-aware patch-wise descriptors based on PointNet [18). FCGF 8 employs a sparse 3D convolutional encoder-decoder network for dense descriptor learning. SpinNet [19 proposes a 3D cylindrical convolution network to extract notation-invariall patch-wise descriptors. Predator (20 utilizes graph convolution and cross-attention to enhance the descriptors and predict the overlapping regions for robust performance in low overlap scenarios.
3D Keypoint Detectors. Detection-based methods have been widely studied image matching but less developed for 3D point clouds. Existing 3D keypoint detectors are mainly hand-crafter which

---

extract salient points based on unique geometric features such as specific curvatures [2] or principal directions [22]. However, they suffer from noisy, sparse, and non-uniform real-world point clouds with large-scale transformations. Recent advances include learning-baser detectors such USIP 23 that predicts repeatable keypoints by minimizing probabilistic chamfer loss, and Nete [2 that further utilizes weighted farthest point sampling to select sparse keypoints from the predicted ones for hierarchical registration. 3DFeat-Net (25) extracts patch-wist wise descriptors with saliency scores for keypoint selection in weakly supervised manner by minimizing weighted feature alignment triplet loss. D3Feat [9] adopts a fully convolutional network to predict point-wise descriptors with hand-crafted saliency scores by minimizing a self-supervised detection loss.
3D Correspondence Learning. DCP 126 predicts soft correspondences from learned features and estimates the pose by differential SVD layer. IDAM 27 designs iterative distance-aware similarity matrix convolution for iterative pairwise matching and pose estimation. Recently, coarse- to-fine correspondence learning has been regarded promising approach. The pioneering work CoFiNet [I]] exploits a group of self-attention and cross-attention for coarse feature matching and the optimal transport for fine matching. GeoTransformer [12] proposes geometric structure embedding for self-attention, and the local-to-global registration (LGR) for consistent pose estimation.
point cloud Transformer based on point pair features, while PEAL [29] and DiffusionPCR [30] use overlap priors and diffusion models for iterative feature matching, respectively. For outlier rejection, RANSAC [15] remains popular despite its inefficiency. DGR [31] predicts correspondence- wise confidence scores via 6D convolutional network, while PointDSC [3] designs a consistency-guict non-local feature embedding to sample consistent correspondences for neural spectral matching and pose estimation.
RolTr (28) improves the coarse-to-Pi framework with a
3 Method
In this section, we present the proposed consistency-awarr spot-guided Transformer (CAST) with a lightweight sparse-to-dense fine matching module for accurate and efficient point cloud registration.
3.1 Overview
Given two partially overlapped point clouds X = [x, E = = 1, M) and Y = [y, E = R I, ,N.. the point cloud registration problem can be formulated as solving the optimal rigid transformation between X, Y by minimizing the weighted sum of point-to-pon errors of predicted correspondence set with a confidence weight Wy for each correspondence Yx):
min R,t
Σ (X
wk [ + t y
where R E SO(3) and E R are the rotation and the translation between X and Y, respectively.
As depicted in Figure 1 CAST follows a coarse-to-fine feature matching and registration architec- ture, including a feature pyramid network, a consisteney-awant spot spot-guided attention based coarse matching module, and a sparse-to-dense fine matching module. We first utilize KPConv-bar
fully convolutional network (33 extract multi-scale features. We denote feature maps of the decoder with
the size of L/K as F { F//3,, which correspond to nodes 1/ and 1/kk down-sampled from X, Y. respectively. For coarse matching, we first adopt an efficient linear cross-attention [4 module to enhance F1/4, Then both semi-dense features F1/4 and coarse features F1/8 are fed into a consistency-aware spot-guided attention-based coarse matching module to improve the feature distinctiveness. The similarity matrix S E IR IRM'xx M'NN" between these enhanced semi-dense features Fx E IRM xD Fy E N is computed based on inner product: S = F FT. Furthermore, we fed Fx and Fy into point-wise MLP to predict the overlap scores, which encode the likelihood of a node having a correspondence. We perform dual-softmax on S to obtain the final matching scores:
P
oü
NxY softmax (Skz); softmax (Sxx)s. kE(L.--.,M)
(2)
ké(1,-...N)
where ax and oy are predicted overlap scores of the i-th node of X1/1 and the j-th node of 1/4 respectively. We use the mutual nearest neighbor scheme to select confident coarse correspondences.
3

---

Transformer-based Matching
Sparse-to- Fine Matching
Singen Hard Atte aiton
Matching Scores,
Saif Ait.Atttttionn
Gross Attantion
Signs
Compatiileg
UpsamplefFuee
Downsampleddusse
9,
Groght
Spot
Sampling
Keppointt to Patcr
Comsipandancee
Weighted
R.t
Similarity Matrix
Coe enty nanc--aaare Salf Artintion
MUP
Noureet
K Grospnin
Sparse Matching
Dense Matching
Uneer Cless Coos-Attentio
Spok Guided
Local
Kopalita
Alignment
Knypoint Detector
RRg. Co
and Descripter
Weighted SVD
Overlap
y014
Figure 1: Overview of CAST. The feature pyramid network down-samples the point clouds and learns features multiple resolutions. The coarse matching module extracts consistency-awee semi-dense correspondences via group alternate consistency-awarc self-attentio modules and spot-guided cross-attention modules with multi-scale feature fusion. Finally, the fine matching module predicts correspondences for both sparse keypoints and dense features and estimates the transformation.
For efficient fine matching, we extract keypoint from the neighborhood of each semi-dense node in |/4 and predict its virtual correspondence in 1/4 based on the lightweight single-head attention. Then utilize compatibility graph embedding predict the confidence of these keypoint correspon- dences as weights in Eq. foor initial pose estimation. Finally, a lightweight local attention module for dense points 1 and 1/2 predicts dense correspondences to refine the pose.
3.2
Consistency-Awan Spot-Guided Attention
To tackle the sparsity and looseness of coarse matching, we focus feature aggregation among semi-dense features F1/4 leveraging both local and global geometric consistency. To be specific, the self-attention only attends salient nodes sampled from global compatibility graph, while the cross-attention only attends to nodes sampled based on local consistency, which are referred to as consistency-awarr self- oef--attentonn and spot-guided attes-attenoon respectively.
Preliminaries. Transformers stacked by alternate self-attention and cross-attention have showcased advanced performance coarse feature matching. When D-dimensional features FA attends F B. the output of vanilla attention is formulated as:
F = softmax FAWO(FBWR)) FE FBWv.
where W WK. Wv are learnable linear transformations to generate queries, keys, and values. When F A. Fa ø related to coordinates A. P. B are from the same point cloud, becomes self-attention that requires positional encoding to embed spatial information. To encode the 3D relative positions, we equip the rotary positional embedding (35] R(-) with learnable weights b,,
bD/2 E glx3,
R(bip)
[oos sin 0
(p))
R(0)
vp Y
(4)
sin
cos
R(bp/2P)
When applying R(-) (-) to vanilla self-attention, the output is formulated as:
VD (
FA = softmax
F&Wv.
(5)
both spot-guided cross-attention and consistency-aware self-attention are sparse attention lacking of abundant global context, we propose to enhance the semi-dense features via multi-scale feature fusion with coarse features. Hence, the architecture of our coarse matching module is designed sequence of blocks for attention-basee multi-scale feature aggregation. For each
Architecture.

---

Compuiii Graph
Matcc Martric
Matching Scores
Generalized Degrees
Graph Sampling
Matching Confidence
Corsisteny--Annt
......r
Spot Selection
Consistency Award
Self- Attertion
Spot Guid ed Cross- Attention
Spot
Vanilla Global Attention
Spot Guided Attention
Figure 2: Illustration of consistency-aware self-attention and spot-guided cross-attention (Left), as well as visualization of the global cross-attention and spot-guided cross-attention (Right). For the left part, the green nodes are query nodes, while the red ones with correct correspondences (green dot lines) are reliable neighbors, and the blue one with a false correspondence (red dot line) an unreliable neighbor. The self-attention (black lines) only attends salient nodes while the cross-attention (black lines) only attends to spots (nodes within black circles).
block with both semi-dense features F1/4 and coarse features F1/8 as inputs, first feed FL/8 into self-attention module (Eq. 5) and cross-attention module (Eq. 3). Then F1/4 and F1/8 are fused into each other based on nearest up-sampling and distance- based interpolated down-sampling [8]]
1/4 F1/ + MLP(Nearest Up-sampling(F*/*))
(6)
F1/8 1/ + MLP(Interpolated Down (ammling((P//))
Finally, F1/4 is fed into a consistency-award self-attention module and a spot-guided cross-attention need to match the
module the end of each block. Before these sparse attention modules, semi-dense features and evaluate the geometric consistency as a clue to select sparse yet instructive
tokens. Given semi-dense features ((()
x in the 1-th block, the matching score is formulated as:
(((0)
P(O softmax (S softmax (S )j, ( = $2 (T))).
(7)
kE(L .M (E(1, .N'
Then the correspondence of each node can obtained as the node from another point cloud with the highest matching score, forming correspondence set (()) . x
X1/4 . Y1/4).
An insight about the consistency among correspondences that the distance between two points is
invariant after transformation. Hence, geometric compatibility is adopted as simple yet effective measure of consistency (32), which is based on the length difference between pairwise line segments. Given a pre-defined threshold c. the pair-wise geometric compatibility of c()) formulated as:
Bij [1 101-dt,/00 / dij ||Ix xx lly yflla|-
The compatibility matrix B, [8a] M is also considered the adjacency matrix of weighted undirected graph known as the compatibility graph. where each vertex is a pair of correspondence and the edge connectivity corresponds to the compatibility between two correspondences. Intuitively, we adopt the generalized degree of pair of correspondence in the graph as a measure of global consistency, which quantifies the connectivity of a vertex as the sum of edge weights connected
Consistency-Award Self-Attention. Intuitively, the correspondences of less salient nodes can be effectively located based on the geometric relationships between them and the salient ones. Hence, compared with global self-attention that attends to all nodes, attending to only salient nodes is more efficient and effective encode the geometric context for matching. We propose the consistency -aware aware self-attention that samples sparse salient nodes to attended to based on both geometric consistency and feature similarity. Given the correspondence set C()) with a compatibility graph, we perform two-stage sampling by ranking the generalized degrees and matching scores, respectively. The first-stage graph sampling using generalized degrees can obtain sufficient consistent correspondences proposals. The second-stage sampling based on matching scores further obtain sparse salient nodes from these proposals. Finally, semi-dense features F ((()) only attend to features of salient nodes from the same point cloud for feature aggregation according to Eq. 5
5

---

Spot-Guided Cross-Attention. As shown in Figure global cross-attentiin tends to aggregate features from many irrelevant regions with similar patterms, leading to false correspondences. Inspired by local consistency that the correspondences of adjacent 3D points remain close to each other, we design the spot-guided cross-attention as depicted Figure 2 For each node x such that (xF,, yf) E C(t), we select a subset (x²") from its neighborhood N(xx) as seeds, and construct a region of interest for it as 5(x²)) UxfeN, (y namely its spot. N,,x selects x/ matching confidence criterion to rank the neighbors, which is formulated as the product of the matching score and the normalized generalized degree the compatibility graph. This criterion incorporates feature similarity and geometric consistency properly measure the reliability of correspondences for seed selection. Finally, semi-dense features attend to their spots for feature aggregation according to Eq. 3 Under the guarantee of local consistency, the spots likely cover the true correspondences, providing guidance for feature aggregation without interfering with irrelevant areas.
and only its neighbors with reliable correspondences. We propose a consistency-awant
3.3 Sparse-to-Dense Fine Matching
Given a coarse correspondence set é = nearest neighbors from the final coarse matching scores (Eq. 2 we propose lightweight sparse-to- dense fine matching module for hierarchical pose estimation without optimal transport, maintaining scalability and efficiency. For sparse matching, we first search k-nearest neighbors (kNN) of semi- dense nodes X 1/ among dense points X/2 to group patches, then
x
X E Y!/1) selected as mutual
attentive keypoint detector [66 to predict a repeatable keypoint with a descriptor from each patch. Each keypoint of point cloud X is assigned to its nearest node, and each node y E Y1/ with a correspondence in groups patch P(y-) of keypoints via kNN. Then, keypoint x, assigned to x will correspond to the patch Plyf). forming a pair of keypoint-to-patch correspondence. Finally, we utilize a shared single- head attention layer for each seypoint-to-pater correspondence to predict virtual correspondences for keypoints. Denote the descriptor of x; d. the virtual correspondence ý; with feature dj is predicted from keypoints Y Ni. with top-k descriptor similarity in Ply)) as:
k Σ
y: softmax d Wold Wk) Na oy softmax (d, We W Da (9)
where dY dy are descriptors of yi,: Yi. and W and W are learnable weights. Inspired by PointDSSC 32,, we construct compatibility graph B (Fq. B of sparse keypoint correspondences {() for spatial consistency filtering via compatibility graph embedding:
E(++1) softmax (
1 E
Q E
RT
OB
E
SD
W
E MLP((X,, ,. df D), V.
(10)
where E()) is the correspondence-wisse embedding of the 1-th layer with learnable weights w wQ K w Finally, the embedding is fed into an MLP to classify if a correspondence an inlier. The predicted inlier confidences serve the weights of keypoint correspondences for pose estimation formulated as Eq. [ which can be analytically solved by weighted Kabsch algorithm [37]. It is
Q
noteworthy that the above process is really lightweight and scalable large-scale registration tasks.
After aligning two point clouds based on sparse matching, we propose to refine the transformation based dense matching. We still utilize local attention (Eq. (9) to predict the correspondences of dense points 1/2 from its neighbors in 1122 within a radius R,, and we simply set the confidence weight of a correspondence with a distance as w = [1 - d/ Ra]+. By solving Eq. agaain with both sparse and dense correspondences, we can achieve more accurate pose estimation efficiently.
3.4 Loss Functions
Our loss function needs to supervise four modules, iee, keypoint detection, coarse matching, keypoint matching, and dense registration. For keypoint detection, we utilize the probabilistic chamfer loss 2 L. to minimize the distances between the closest keypoints from the source and target point clouds after alignment under the ground-truth transformation. Please refer to (3) for details.

---

Coarse Matching, Given the ground-trute coarse correspondence set C with an overlap ratio of for each correspondence (i.j) E we propose a spot matching loss L, and a coarse matching loss f.e formulated as weighted cross entropy losses to supervise the layer-wise coarse matching scores P( (I = 1,2, L) and the final coarse matching scores P, respectively:
C. = -
1 L
E
oi log p!! is
(11)
E (ij)ec Oit (i.))C
C. Eiiseec Oij
Oij log Púj (K.))E
Σ
Σ kENx
log( -
Σ RENY
log( ( -1 (12)
[W
Wr
where Nxx and My are sets of semi-dense nodes in point clouds X and Y without correspondences, respectively. Two nodes are considered as a pair of coarse correspondence only when their ground- truth overlap ratio is greater than O. Assuming that the patch centered at a point E IR3 is spherical neighborhood of radius r, the overlapping ratio os of patches centered at p: E X and qj E Y with ground-truth translation t E R and rotation R SO(3) can be calculated by:
2T (r2 h2)dhh
3d
ds
Oig
4xr
=1
d == max R| + t +- 2rl.2r). 16r 4r
(13)
Keypoint Matching. As our keypoint matching module follows a three-stage pipeline including similarity caleulation, correspondence prediction, and consistency filtering, reasonable to super- vise these stages with three losses, respectively. Only valid scypoint-to-patee correspondences supervised during training, i.e., the distance of the keypoint æ and its closest point pa in the patch Cz is less than pre-defined threshold Rp > 0, and the points whose distances from greater than pre-defined threshold R,, 2 R2 form non-empty set N₂ C Cz. We formulate the keypoint matching loss Ly an InfoNCE loss (38) with symmetric learnable weights W, which aims at maximizing the similarity between descriptors de and dp., of true correspondences (x. Pw) and minimizing the similarity between descriptors d, and dnS of false correspondences (ær, n2);; E Na.
exp(d]] Wdp.)
fj = - -B(z,eeNN)) log exp(d] dp..) ) + Em.EN, exp (d_ (d_ Wdn.))
(14)
For correspondence prediction, adopt T2 loss La ((.,6) Rr t Úl1 for the predicted correspondences ý of keypoints from all valid keypoint-to-patch correspondences (I, Cz). For consistency filtering, we simply utilize a binary entropy loss L, to supervise the confidence scores of
all keypoint correspondences. The binary ground-trus truth label of keypoint correspondence (z, () is 1
and only if it is an inlier, i.e., Rr + Øl1 is less than a threshold R > 0.
Dense Registration. Given the translation i and rotation R estimated by dense registration, we
adopt a translation loss L, = ||t t/|2 and a rotation loss L, = || R - Iaxa || for supervision. Finally, we formulate our loss as L = Cp + X.C, + Xohe + Ay + Au + Acc + XcGe + Xr where de, As Aj de, Xi, de, X, are balancing weights.
4 Experiments
In this section, we evaluate our method on both outdoor LiDAR point cloud datasets KITTI [39]. nuScenes [40]. and the indoor RGBD point cloud dataset 3DMatch [T6]. Our network is trained using an AdamW A1l optimizer with batch size of 1, an initial learning rate of le-4, and a weight decay of 1e-4. The step scheduler decrease the learning rate to 90% every five steps, with gradients clipped at norm of 0.5 during back propagation. Despite the complexity of the loss function, only one stage needed for training. Our model trained on an NVIDIA RTX 3090 GPU with an Intel Xeon CPU @2.90GHZ for 5, 40, and epochs on 3DMatch, KITTI, and nuScenes, respectively, and we set X X; = A, = 20, d = 5, A = 0.1, and A = 0.2, A = for KITTI and nuScenes,
A. 1, d 10 for 3DMatch.
7

---

Source Frame with Keypoints
Target Frame with Keypoints
Spare Keypoint
Correspondences
Pose Estimation
Figure 3: Qualitative registration results on KITTI dataset. We show three examples in three columns. The first two rows present the raw point clouds and highlight the 3D keypoints with low uncertainty in red. Our keypoints are typically located in sharp corners and edges of buildings, pillars, and vehicles. The third row shows the predicted sparse keypoint correspondences with high scores, while the last row presents the aligned point clouds after pose estimation. Although few outliers colored in red have not been filtered out, their distances are acceptable for accurate registration.
4.1 Outdoor Scenarios: KITTI and NuScenes
KITTI 39 is popular benchmark for autonomous driving. Following 9., we use sequences O for testing, and select only point cloud pairs at least 10m away from each other with ICP-refined [42] GPS localization results ground truth.
to 5 for training, to for validation, and
NuScenes [40] another large-scale outdoor autonomous driving benchmark including 850 scenes for training and validation and 150 for testing. Following (24), we select each LiDAR keyframe with the second keyframe after it as a pair point clouds. use three metrics for evaluation 20 relative translation error (RTE), relative rotation error (RRE), and registration recall (RR).
Table 1: Registration performance on KITTI odometry dataset.
Model
Publication
RTE (em)
RRE (9)
RR (%)
3DFeat-Net
ECCV 25
0.25
FCGF
ICCV 2019
g
9.5
0.30
96.6
D3Fcat
CVPR 2020
9
7.2
0.30
99.8
SpinNet
CVPR 2021
19
9.9
0.47
99.1
Predator
CVPR 2021
20
6.8
0.27
99.8
CoFiNet
NeurIPS 2021
0.41
99.8
Geo Geo Transformer
CVPR 2022
12
6.8
0,24
99.8
OIF-Net
NeurlPS 2022
13
6.5
0.23
99.8
PEAL
CVPR 2023
29
6.8
0.23
99.8
DiffusionPCR
CVPR 2024
0.23
99.8
MAC
CVPR 2023
0.40
99.5
RegFormer
ICCV 2023
44
8.4
0.24
99.8
CAST
2.5
0.27
100.0
Our results KITTI are detailed quantitatively in Table qualitatively in Figure T shows that the proposed CAST outperforms various learning-based methods, including bescriptor-based
[1] 12 [3] including the latest ones with [43] and an end-to-en [44] baselines. Specifically, CAST achieves RR of 100.0% and the lowest RTE of 2.5cm, which is 60.3% improvement over the state-of-the-art DiffusionPCR [30], highlighting its superior robustness and accuracy.
25 8,
19, 20), coarse-to-fine correspondence-basee
iterative matching 22 30), a recent graph-based
primarily due to numerical errors SVD-based pose estimation that usually produces non orthoommooraal rotation matrices. RRE is set as when trace R.) > 3, condition met frequently across all methods. Consequently, geodesic based RRE may not accurately reflect the actual performance.
As for RRE, CAST slightly underperforms some coarse-to to- fine methods 12
13
29
30

---

Table 3: Evaluation results on indoor RGBD point cloud datasets.
Dataset
3DMatch
3DLoMatch
Average
Registration Recall (%)
Registration Recall (%)
Time (s) All
Samples
5000
2500
1000
250
5000
2500
1000
500
250
descriptor bar-ebseed
PerfectMatch
78.4
76.2
71.4
67.6
50.8
33.0
29.0
23.3
17.0
11.0
FCGF g
85.1
84.7
83.3
81.6
71.4
40.1
41.7
38.2
35.4
26.8
0.271
D3Feat 9
81.6
84.5
83.4
82.4
77.9
37.2
42.7
46.9
43.8
39.1
0,289
SpinNet [9
88.6
86.6
85.5
83.5
70.2
59.8
54.9
48.3
39.8
26.8
90.804
YOHO [5
90.8
90.3
89.1
88.6
84.5
65.2
65.5
63.2
56.5
48.0
13.529
Predator 20
89.0
89.9
88.5
86.6
59.8
61.2
62.4
60.8
0.759
REGTR
92.0
64.8
0,382
correspondence saice--bseed
CoFiNet 11]
89.3
88.9
87.4
87.0
67.5
64.2
63.1
61.0
0.306
GeoTransformer
12
92.0
91.8
91.8
91.4
91.2
75.0
74.8
74.2
74.1
73.5
0.192
OIF-Net 13
92.4
91.9
91.8
92.1
91.2
76.1
75.4
75.1
74.4
73.6
0.555
RolTr 28
91.9
91.7
91.8
91.4
91.0
74.7
74.8
74.8
74.2
73.6
0.457
PEAL 29
94.4
94.1
93.4
79.2
79.0
78.8
78.5
77.9
2.074
BUFFER
48
92.9
71.8
0.290
SIRA-PCR
49
93.6
93.9
93.9
92.7
92.4
73.5
73.9
73.0
73.4
71.1
0,291
DiffusionPCR 30
94.4
94.3
94.5
94.0
93.9
80.0
80.4
79.2
78.8
78.8
1.964
CAST
95.2
75.1
0.182
For more challenging LiDAR benchmark nuScenes, we compare CAST with both tra-
Table Registration performance on nuScenes.
ditional 42 45
15 and learning- based algo-
Method
RTE (m)
RRE (?)
RR (%)
rithms 26 27
46
31 24 in Table We do
Point-to Po-Point ICP
42
0.25
0.25
18.8
include the coarse-to-fime methods since none have been trained or tested on nuScenes. Most of the results are borrowed from [24] while HRegNet 24 is re-evaluated with their open source codes. Our method achieves lowest translation error of 0. 0. and the low- est rotation error of 0.20" while maintaining the best RR of 99.9%, showcasing state-of- the-art robustness and accuracy.
Point-to-Plane ICP
42
0.15
0.21
36.8
FGR 45
0.71
1.01
32.2
RANSAC 15
0.21
0.74
60.9
DCP 26
1.09
2.07
56.8
IDAM 27
0.47
0.79
88.0
FMR 46
0.60
1.61
92.1
DGR
0.21
0.48
HRegNet 24
0.18
0.45
99.9
CAST
0.12
0.20
99.9
4.2 Indoor Scenarios: 3DMatch and 3DLoMatch
Our approach is also evaluated on indoor benchmarks 3DMatch [16] and 3DLoMatch 20 which consist of point cloud pairs with overlaps >30% and 10% 30%, respectively. In Table we use registration recall [16] as our evaluation metric, and the runtime of all methods in Pytorch implementation with 5000 points on our device with an Intel CPU 1p-177113 17-12800HX @ 2. 2.30GHZ and an
NVIDIA RTX 3080Ti GPU for fairness, except 77] in Tensorflow implementation and 20
using the results reported in their papers [3] 30 due to the absence of source codes. Our method along with other sparse matching baselines
48 directly uses all points for evaluation. To enhance
the robustness in low overlapping cases, our method is combined with RANSAC estimating initial pose from only 250 coarse correspondences to reject the outliers during fine matching.
On the 3DMatch benchmark, our method achieves state-of-the-ari RR of 95.2%. On the more challenging 3DLoMatch, CAST achieves high RR of 75.1% outperforming all descriptors and non-iterative correspondence-basee
48 49] except OIF-Net (13) using more than 1000 sampled points. As our method typically detects about 1000 sparse keypoints and establishes less than 250 keypoint correspondences with other methods using only 250 sample points. However, CAST outperforms the state-of-the-ad non-iterative correspondence-baat based methods OIF-Net [13] using less than 1000 points. Notably, our method achieves such superior performance only with the lowest runtime, while RANSAC remains efficient due to our high inlier ratio. Although PEAL 29 and DiffusionPCR 30 show higher RR on 30thhM their iterative feature matching with overlap priors is extremely time-consuming (10 times of ours), while PEAL even requires extra information from 2D images.
methods 47 11
12
28
3DLoMatch, it is fair to compare CAST

---

Table 4: Ablation studies of coarse matching modules on indoor datasets.
3DMatch PMR (%)
3Dooattch PMR (%)
MS
SG
CA
OV
PIR (%)
RR (%)
(%)
RR (%)
77.56
95.87
94.45
40.82
70.58
72.07
77.95
96.61
94.92
42.55
72.77
74.57
/
V
69.58
96.67
94.14
32.59
65.02
73.00
4
/
V
V
73.56
97.17
95.07
35.25
68.33
74.91
/
V
79.79
97.17
96.01
44.41
75.24
76.59
4.3 Ablation Studies
We select indoor datasets for ablation studies of coarse matching as they are more challenging. Here we evaluate the RR over the whole dataset rather than the average RR of eight sequences reported sequence lengths.
in Table 3] which more reasonable for dataset with significant variances Besides, we assess two extra metrics to directly measure the performance of coarse matching: patch inlier ratio (PIR), the fraction of patch matches with actual overlap; and patch matching recall (PMR), the fraction of point cloud pairs with PIR above 20%. Results from five experiments in Table 4 demonstrate the effects of proposed multi-scale feature fusion (MS), spot-guided cross-attention (SG), consistency-awee awaree self-attention (CA), and the overlap head for overlap score prediction (OV). The first experiment ablating CA and replacing SG with linear cross-attention, suffers performance degradation in all metrics due to inconsistency. The second experiment improves all metries based on SG, while the last one achieves the best performance via CA, showcasing their effectiveness. Figure visualizes the vanilla global cross-attention and our spot-guided musss-atention. Instead of interacting with many similar yet irrelevant regions for misleading feature aggregation, SG can effectively select instructive areas to attend to according to local consistency. Compared to the last experiment, the third one verifies the effectiveness of multi-scale feature fusion, while the fourth one demonstrates the necessity of overlap prediction.
Additionally, we conducted five abla- tion studies on KITTII for better un- derstanding of our fine matching, since pose errors are better metrics to reflect accuracy. The second experiment using only sparse keypoints for registration highlights the effectiveness of dense registration, while the third shows the effect of learnable dense correspon- dences compared to nearest neighbors. The last three experiments report the performance of sparse registration by ablating the keypoint detector, the learnable sparse correspondences, and the compatibility graph embedding, each demonstrating their necessity for accuracy. Despite these variations, all studies maintain a 100% RR, showing the robustness of coarse matching.
Table 5: Ablation studies of fine matching on KITTI.
RTE (cm)
RRE (*)
RR (%)
ours
2.51
0.27
100.00
ours w/o dense registration w/o virtual dense ours keypoint detection
3.13
0.30
100.00
2.85
0.28
100.00
3.58
0.30
100.00
ours w/o virtual sparse corr.
3.25
0,30
100.00
ours w/o graph embedding
5.01
0.30
100.00
Conclusion
In this paper, we present a novel consistency-awant spot-guided Transformer to achieve compact and consistent coarse matching for point cloud registration. At the coarse matching stage, our consistency- aware self-attention enhances the feature representations with sparse sampling from the geometric compatibility graph. Additionally, our spot-gu goo-gg cross-attentio leverages local consistency to guide the ants-ttttnnion to confident spots without interfering with irrelevant areas. Based on these semi-dent deied and consistent coarse correspondences, lightweight and scalable sparse-to toesee fine matching module empowered by local attention can achieve accurate pose estimation without optimal transport or pooothesis-andsselccti robustness, and efficiency for point cloud registration across different 3D sensors and scenarios, which paves the way for large-scale real-time applications such as SLAM.
pipelines. Our method has showcased stafe-of-the-ar accuracy,
10

---

Acknowledgments
This work is supported by the National Natural Science Foundation of China (62203383/62088100)
References
[1] Renlang Huang, Minglei Zhao, Jiming Chen, and Liang Li. Kdd-loam: Jointly learned keypoint detector and descriptors assisted lidar odometry and mapping. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 8559-8565, 2024.
Weitong Hua, Zhongxiang Zhou, Jun Wu, Huang Huang, Yue Wang, and Rong Xiong. Rede: End-to-end object 6d pose robust estimation using differentiable outliers elimination. IEEE Robotics and Automation Letters, 6(2))2886-2893, 2021.
[3] Jianyuan Wang, Yiran Zhong, Yuchao Dai, Stan Birchfield, Kaihao Zhang, Nikolai Smolyanskiy, and Hongdong Li. Deep two-view mrructurefrrommmttion revisited. In Proceedings of Be IEEE/CVF Conference Computer Vision and Puttern Recognition. pages 8953-8962, 2021.
Radu Bogdan Rusu, Nico Blodow, Zoltan Csaba Marton, and Michael Beetz. Aligning point cloud views using persistent feature histograms. 2008 IEEF/RSI International Conference on Intelligent Robots and Systems, pages 3384-3391. IEFE, 2008.
[5] Radu Bogdan Rusu, Nico Blodow, and Michacl Beetz. Fast point feature histograms (fpfh) for 3d registration. In 2009 IEEE International Conference Robotics and Automation, pages 3212-3217. IEEB, 2009.
Samuele Salti, Federico Tombari, and Luigi Di Stefano. Shot: Unique signatures of histograms for surface and texture description. Computer Vision and Image Understanding, 125:251-264, 2014.
[7] Zan Gojeic, Caifa Zhou, Jan D Wegner, and Andreas Wieser. The perfect match: 3d point cloud matching with smoothed densities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5545-5554, 2019.
[8] Christopher Choy, Jaesik Park, and Vladlen Koltun. Fully convolutional geometric features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8958-8966, 2019.
Xuyang Bai, Zixin Luo, Lci Zhou, Hongbo Fu, Long Quan, and Chiew- Tai. D3feat: Joint learning of dense detection and description of 3d local features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6359-6367, 2020.
[10] Jiahuan Yu, Jiahao Chang, Jianfeng He, Tianzhu Zhang. Jiyang Yu, and Feng Wu. Adaptive spot spot-guided transformer for consistent local feature matching. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21898-21908, 2023.
[11] Hao Yu, Li, Mahdi Salch, Benjamin Busam, and Slobodan Ilie. Cofinet: Reliable coarse-to-fine correspondences for robust point cloud registration. Advances Neural Information Processing Systems, 07223872-23884 2021.
[12] Zheng Qin, Hao Yu, Changjian Wang. Yulan Guo, Yuxing Peng. Slobodan Ilic, Dewen Hu, and Kai Xu. Geotransformer: Fast and robust point cloud registration with geometric transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[13] Fan Yang, Lin Guo, Zhi Chen, and Wenbing Tao. One-inlier is first: Towards efficient position encoding for point cloud registration. Advances in Neural Information Processing Systems, 35:6982 6995, 2022.
[14] Minhao Li, Zheng Qin, Zhirui Gao, Renjiao Yi, Chenyang Zhu, Yulan Guo, and Kai Xu. 2d3d-matr: 2d-3d matching transformer for detection free registration between images and point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 14128 14138, 2023.
[15] Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications image analysis and automated cartography. Communications of the ACM, 24(6))381-395, 1981.
[16] Andy Zeng. Shuran Song, Matthias NieBner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1802-1811. 2017.
11

---

[17] Haowen Deng, Tolga Birdal, and Slobodan Ilie. Ppfnet: Global context aware local features for robust 3d point matching. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 195-205, 2018.
[18] Charles Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference Computer Vision and Pottern Recognition, pages 652-660, 2017.
[19] Sheng Ao, Qingyong Hu, Bo Yang, Andrew Markham, and Yulan Guo. Spinnet: Learning a general surface descriptor for 3d point cloud registration. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 1753-11762, 2021.
[20] Shengyu Huang. Zan Gojeic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler. Predator: Registration of 3d point clouds with low overlap. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4267-42777 066-4427,,7 2021.
[21] Hui Chen and Bir Bhanu. 3d free-form object recognition in range images using local surface patches. Pattern Recognition Letters, 122--1222,2222,, 2007.
[22] Yu Zhong. Intrinsic shape signatures: A shape deseriptor for 3d object recognition. In IEEE International Conference on Computer Vision Workshops, ICCV workshops, pages 689-696. IEFE, 2009.
[23] Jiaxin Li and Gim Hee Lec. Usip: Unsupervised stable interest point detection from 3d point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 361-370, 2019.
[24] Fan Lu, Guang Chen, Yinlong Liu. Lijun Zhang, Sanqing Qu. Shu Liu, Rongqi Gu, and Changjun Jiang. Hregnet: A hierarchical network for efficient and accurate outdoor lidar point cloud registration. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2023.
[25] Zi Jian Yew and Gim Hee Lee. 3dfeat-net: Weakly supervised local 3d features for point cloud registration. In Proceedings of the European Conference on Computer Vision (ECCV), pages 607 623, 2018.
[26] Yue Wang and Justin Solomon. Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3523-355 3532, 2019,
[27] Jiahao Li, Changhao Zhang, Ziyao Xu, Hangning Zhou, and Chi Zhang. Iterative distance-aware similarity matrix convolution with nutual-superriise point elimination for efficient point cloud registration. In Computer Esiun-ECCV 2020: 16th Eurupean Conference, Glaxgow; UK. August 23-28, 2020, Proceedings. Part XXIV 16, pages 378-394. Springer. 2020.
[28] Hao Yu, Zheng Qin, Ji Hou, Mahdi Saleh, Dongsheng Li, Benjamin Busam, and Slobodan Ilic. Rotation- invariant transformer for point cloud matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[29] Junle Yu, Luwei Ren, Yu Zhang, Wenhui Zhou, Lili Lin, and Guojun Dai. Peal: Prior-embedded explicit attention learning for low-overlap point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17702-17711, 2023.
[30] Zhi Chen, Yufan Ren, Tong Zhang, Zheng Dang. Wenbing Tao, Sabine Süsstrunk, and Mathieu Salz- mann. Diffusionper: Diffusion models for robust multi-step point cloud registration. arXiv preprint arXiv:2312.03053. 2023.
[31] Christopher Choy, Wei Dong, and Vladlen Koltun. Deep global registration. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 25 2514-2523, 2020.
[32] Xuyang Bai, Zixin Luo, Lei Zhou, Hongkai Chen, Lei Li, Zeyu Hu, Hongbo Fu, and Chiew- Tai. Pointdsc: Robust point cloud registration using deep spatial consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15859-15869, 2021.
[33] Hugues Thomas, Charles R Qi, Sraan-Emmnuull Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 641 1-6420, 2019.
[34] Angelos Katharopoulos. Apoory Vyas, Nikolaos Pappas, and François Fleuret. Transformers are runs: Fast autoregressive transformers with linear attention. In International Conference Machine Learning, pages 5156-5165. PMLR, 2020.
[35] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 668:127063, 2024.
12

---

[36] Fan Lu. Guang Chen, Yinlong Liu, Zhongnan Qu, and Alois Knoll, Rskdd-net: Random sample-based keypoint detector and descriptor. Advances in Neural Information Processing Systems, 33:21297-21338.. 2020.
[37] Wolfgang Kabsch. A solution for the best rotation to relate two sets of vectors. Acia Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 32(5):922-923, 1976.
[38] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint wwXiv:187.0034440 2018
[39] Andreas Geiger, Philip Lenz, and Raquel Urtasun, Are we ready for autonomous driving? kitti vision benchmark suite. In JEEE Conference on Computer Vision and Pattern Recognition, pages 3354-3361.
IEEE, 2012.
[40] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint 1903. 1027, 2019.
[41] Ilya Loshchilov Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018
[42] P.J. Besl and Neil D. McKay. method for registration of 3-d shapes. IEFF Transactions on Pattern Analysis and Machine Intelligence. 14(2):339- 256, 1992.
[43] Xiyu Zhang, Jiaqi Yang, Shikun Zhang, and Yanning Zhang. 3d registration with maximal cliques. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17745- 17754, 2023.
[44] Juuming Liu, Guangming Wang, Zhe Liu, Chaokang Jiang, Mare Pollefeys, and Hesheng Wang. Regformer: An efficient projection-aware transformer network for large-scale point cloud registration. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8417 1417-8426, 2023.
[45] Qian-Ya Yi Zhou, Jaesik Park, and Vladlen Koltun. Fast global registration. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 1I 14, pages 766-782. Springer, 2016.
[46] Xiaoshui Huang, Guofeng Mei, and Jian Zhang. Feature-metric registration: A fast semi-supervised approach for robust point cloud registration without correspondences. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1336-11374. 2020.
[47] Zi Jian Yew and Gim Hee Lee. Regtr: End-to-end point cloud correspondences with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6677-6686. 2022.
[48] Sheng Ao. Qingyong Hu, Hanyun Wang. Kai Xu, and Yulan Guo. Buffer: Balancing accuracy, efficiency. and generalizability in point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 1255-1264 2023.
[49] Suyi Chen, Hao Xu, Ru Li, Guanghui Liu, Chi-Wing Fu, and Shuaicheng Liu. Sira-per: Sim-to-real adaptation for 3d point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14394- 14405, 2023.
[50] Haiping Wang, Yuan Liu, Zhen Dong. and Wenping Wang. You only hypothesize once: Point cloud registration with rotation- equivariant descriptors. In Proceedings of the 30th ACM International Conference on Multimedia, pages 1630- 1641, 2022.
[51] Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2930-2933 2013.
[52] Kevin Lai, Liefeng Bo, and Dieter Fox. Unsupervised feature learning for 3d scene labeling. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3050-3057 30500 IEEE, 2014.
[53] Julien Valentin, Angela Dai, Matthias Nießnner, Pushmeet Kohli, Philip Torr, Shahram Izadi, and Cem Keskin. Learning to navigate the energy landscape. In 2016 Fourth International Conference on 3D Vision (3DV), pages 323-332. IEEF, 2016.
[54] Angela Dai, Matthias NieBner, Michael Zollhöfer. Shahram Izadi, and Christian Theobalt. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly fly surface reintegration. ACM Transactions Graphics (ToG), 36(4):1, 2017.
13

---

[55] Jianxiong Xiao, Andrew Owens, and Antonio Torralba. Sun3d: A database of big spaces reconstructed using sfm and object labels. In Proceedings the IEEE International Conference Computer Vision, pages 1625-1632, 2013.
[56] Maciej Halber and Thomas Funkhouser. Fine-to-coarse global registration of rgb-d In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 1755-1764, 2017.
[57] Brian Curless and Mare Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pages 303-312, 1996.
[58] Heng Yang, Jingnan Shi. and Luca Carlone. Teaser: Fast and certifiable point cloud registration. JEEE Trunsactions on Robotics, 37(2))314-333 2021.
[59] Zhi Chen, Kun efficient and robust point cloud registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13221-13232 2022.
Yang, Wenbing Tao. Sc2-per: second order spatial compatibility for
[60] Yifei Zhang, Hao Zhao, Hongyang Li, and Siheng Chen. Fastmac: Stochastic spectral sampling of correspondence graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17857-17865 2024.
[61] François Pomerleau, M. Liu, Francis Colas, and Roland Siegwart. Challenging data sets for point cloud registration algorithms. The International Journal of Robotics Research, 17554)11705-1711, December 2012.
[62] M. Leordeanu and M. Hebert. spectral technique for correspondence problems using pairwise constraints. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume pages 1482- 1489, 2005.
14

---

A Appendix
this appendix, we will first detail our neural network architecture with hyper-parameters' Sec. A.1 Then introduce the related datasets in Sec. A.2 of our method with detailed explanation of evaluation metrics Sec. A.3 Additionally, more qualitative results illustrated in Sec. A.4 Finally, will discuss the limitations and broader impacts Sec. A.5 and Sec. A.6 respectively.
A.2 and provide more quantitative experimental results
A.1 Neural Network Architectures and Hyper-parameters
Feature Pyramid Network. CAST utillzes a fully convolutional feature pyramid network as the backbone, which follows mncoder-decoder architecture based on KPConv [33] operations. Details of our network urchitecture are illustrated in Figure which remain the same [12] including five encoder layers and three decoder layers. Note that the backbone for indoor datasets 3DMatch [16] is slightly different, which only comprises four encoder layers and two decoder layers.
Coarse Matching Module. Figure iillustrates the details of consistency aware spot-guided attention blocks for coarse matching. Both coarse features and semi-dense features extracted from the backbone first projected to 128 dimensions and then pass through three consistency-aware spot-guided attention blocks (Figure [ Each attention module uses heads with ReLU activation. Compared vanilla attention with a quadratic increase in the size of the attention matrix with respect to the input length, the linear attention (34) is much more efficient in global context aggregation by replacing the softmax operator with the product of kernel functions:
Linear attention Q, K, V) = (())))())
(15)
where <(-) elu(-) + 1. For spot-guided cross-attention, each node x selects 4 seeds as N. from the neighborhood (V(x)) with 12 nodes based on consistency-awant matching confidence. For consistency-awer aware self-attention, first scale the generalized degrees to [0, 1]. and sample 48 nodes with highest matching scores from nodes with scaled degrees greater than 0.3 as keys to be attended.
Fine Matching Modules. For keypoint detection and description, we utilize an attentive keypoint detector [36] to extract keypoint with descriptor from each local patch, which contains k = 32 nearest neighbors among dense points X,, Xix X1/2 with features fi," Six of patch
Strided BesdedRReeBlock (128, 128)
Strided dReddeesBBBB Blook (238, 258)
StridedResBloct Block (512, 512)
StridedFee sBllockkkkkkkkkkk (1024, 1024)
KPConv (1, 64)
ResBlock (8))
ResBlock 256)
ResBlock 256)
ResBlock (256, 512)
ReaBlock (512, 512)
ResBlock (512, 1024)
ReaBlock (1024, 1024)
ResBlock (1024, 2048)
ResBlock (2048, 2048)
Nearest pp-Sampling
Linear (3072. 1024)
Nearest Up Sampling
Linear (1536, 512)
Nearest Up Sampling
Linear (768, 256)
(in, in*2)
(in, in)
StridedResBlock (in, in)
GroupNorm
Linear (in/2, 2)
(in14,
(in/4, in)
LeakyReLU
(in/2, in/2)
(in, in
(in/4, In/4)
KPConv (in/4, in/4)
Concat
Linear (in, in/2)
Linear in/4)
Linear (in, in/4)
Figure The detailed architecture of the KPConv-basee feature pyramid network.
15

---

FC
FC
Add & Norm
Add & Norm
Add Norm
Add & Norm
Feed Forward
Feed Forward
Add & Norm
Add & Norm
Add Norm
Add & Norm
Feed Forward
Multi Scale
Feed Forward
Multi-Head Cross Attention
Multi-Head Cross Attention
Spot
e
Selection
Q1 K1
Add Norm
Add & Norm
Multi-Head
Multi-Head Attention
Add & Norm
Add & Norm
Cross Attention
Feed Forward
Feed Forward
Qf Kt
v1
Kt Qf
Add Norm
Add & Norm
Add Norm
Salient Node
Add & Norm
Feed Forward
Feed Forward
Multi-Head Self Attention
Sampling
Multi-Head Self Attention
Qt
Qt
Add & Norm
Rotary Positional Encoding
Add & Norm
Multi-Head Head Self- Gelf-Attention
Multi-Head Self-Attentoon
FC
FC
Add
Add
Q1
K
Q1
Kt
vf
x3
Linear Attention
Linear Attention
F
X
Y
FG
F
X
YS
Figure The detailed architecture of the consistency-awarc spot-guided Transformer with multi-scale feature fusion for coarse feature matching.
node x X The details of the keypoint detector and descriptor are illustrated in Figure 6 which are based on only shared MLPs with [x,, ||x, x; ft,] . 1,... k as inputs. For keypoints xf ..... x with uncertainties arr M' and y , N' with uncertainties d O' from point cloud X and Y, respectively, we adopt the probabilistic chamfer loss in [23 for training, please refer to their paper for details.
To establish keypoint-to- torpiinttoo-ptch correspondences based on coarse correspondences, each keypoint assigned to its nearest node when their distance is below threshold R4- Then the keypoint corresponds to the correspond- ing patch of this node containing kp keypoints. As mentioned in Sec. we leverage a single-head attention layer to predict the vir- tual correspondence of each key- point based only k, keypoints its corresponding patch with the highest similarity (Eq. 9 Finally, spatial consistency filtering is performed via three graph compatibility embedding layers with embedding dimension D. 64 in Eq. 10 After sparse matching and registration, we further refine the transformation based on dense matching, which searches ki nearest neighbors within a distance threshold R& for each dense point predict its virtual dense correspondence. Specifically, we set k2 4, ka = 6 in our implementation, while other related harer-parameters listed in Table
keypoints
MLP (250, 64, 64)
MLP (64, 54, 64,
attentive
softmax
scores
Somelu
H
uncertainty
grouped features
maxpool
concatenation
MLP (576, 64, 64)
maxpool
product
descriptor
sum
Figure 6: Attentive keypoint detector and descriptor.
Weighted Kabsch Algorithm. The weighted Kabsch algorithm 37. also known the Proscrutes algorithm, provides the closed-form solution for the point cloud registration problem (Eq. [ Given predicted correspondence set Č {(x+. k = 1, ... n). the optimal rigid transformation
R E SO(3), é e 3 can be estimated via two steps:
16

---

Table 6: setting for three datasets in our implementation.
Hyper-parameters
Explanation
3DMatch
KITTI
nuScenes
coarse overlap radius (Eq. 13
0.075m
1.2m
1.2m
de
coarse compatibility threshold (Eq.
0.15m
1.8m
1.8m
Rs
keypoint to node distance threshold
0.1m
1.8m
1.8m
Odd
fine compatibility threshold
0.1m
1.0m
1.0m
Rd
dense matching radius positive matching threshold (Eq.
0.15m
0.75m
1.0m
R
0.05m
0.45m
0.45m
Rn
negative matching threshold (Eq. number of keypoints in patch
0.06m
0.6m
0.6m
kp
16
24
24
Step 1. Centralize the point clouds by subtracting away their weighted centroids:
-
Ek= wkYk
Ek= k= Wkk
= = - yk yk y,
X
(16)
Ek=
Step Estimate the transformation. A x 3 weighted covariance matrix can be computed
H
Σ W k=1
(17)
With its singular value decomposition
H
UEVT T the optimal estimate of pose is given by:
1
0
R
V 0
1
0
UT. t- -x.
(18)
0
0
det(VUT)
A.2 Data and Benchmarks
3DMatch. 3DMatch [16] is a widely used RGBD point cloud dataset for 3D reconstruction,
including 62 scenes from 7-Scenes [51]] RGB-D Scenes V2 [52], maalyis-by--Sntten [53]. BundleFusion (54). SUN3D (55) and Halber et al. [56] with their licenses in Table 7 where 46 scenes are used for training, scenes for validation and 8 scenes for testing. Input point clouds
generated by fusing 50 consecutive depth frames collected by RGBD cameras using TSDF volumetric
fusion [57]. Different from the original 3DMatch [16] that only consists of point cloud pairs with >30% overlaps, (20] also includes point cloud pairs with overlaps between 10% and 30% for training,
and sets two benchmarks for performance evaluation, i.e. 3DMatch consisting of point cloud pairs with >30% overlaps, and 3DLoMatch consisting of point cloud pairs with low overlap ratios between 10% and 30%. Following 12 we utilize the voxel grid down- nown-samplig of 2.5cm voxel size for data preprocessing, which picks the centroid as a nown-sampled point when multiple points fall into common voxel grid. We leverage the data and evaluation protocols in 20 for training and testing.
Table Raw data in the 3DMatch [6 dataset and their licenses.
Datasets
License
7-Scenes [5]]
Non-commercial use only
Analysis-by basss-yb-yyythhii
53
CC BY-NC-SA4.0 4.0
BundleFusion [54]
CC BY-NC-SA (License not stated) CC BY-NC-SA 4.0
RGB-D Scenes v2 55
SUN3D 55
Halber et al.
56
CC BY-NC-SA 4.0
KITTI. KITTI 39 is classic benchmark under the Caammerrial-SSaarAAiiiii 3.0 License for variety of computer vision tasks of autonomous driving, ranging from LiDAR-based or vision-based or multi-sensors based odometry, object detection and tracking, optical flow estimation, point cloud registration, etc. KITTI comprises of 11 sequences scanned by Velodyne HDL-64 3D laser scanner in driving scenarios. Following the data splitting method in (8, we use sequences to 5 for training, 6 to 7 for validation, and 8 to 10 for testing. Besides, directly leverage the source code of [9] to
17

---

select point cloud pairs which are at least 10m away from each other, which leads to 1,358 training pairs, 180 validation pairs, and 555 testing pairs. Moreover, as the ground truth transformations provided by GPS are less accurate, we follow [|] to refine them via standard ICP [42 in 500 iterations. Following [9], we utilize the voxel down- samplign of 0.3m voxel size for data preprocessing.
NuScenes. NuScenes 440 is an outdoor autonomous driving datasets under CC BY-NC-SA 4.0 license. is the first large-scall scale dataset to provide data from the entire sensor suite of an autonomous vehicle, consisting of 850 scenes for training and validation and 150 scenes for testing. Following [24], we select the first 700 scenes from the 850 scenes for training and the others for validation. The
information about the point cloud pairs with ground-truth transformations is downloaded from the
source codes of [24] which selects each LiDAR keyframe with the second keyframe after it as a pair. For data preprocessing, we apply 0.3m voxel grid down-sampling.
A.3 Evaluation Metries with Extra Quantitative Results
The outdoor datasets KITTI 39 and nuScenes 40 commonly use three metrics for evaluation 20 (1) relative Iranslation error (RTE), the Euclidean distance between the estimated and ground-truth translation vectors; (2) relative rotation error (RRE), the geodesic distance between the estimated and ground-truth rotation matrices SO(3); and (3) registration recall (RR), the percentage of point cloud pairs with RTE<2m and RRE<5.. Given the ground-truth rotation R. E SO(3) and translation t E R as well as the estimated rotation R. and translation t, the RRE and RTE are formulated as
trace R RR)-
RRE arccos
RTE || t||2-
(19)
2
The indoor benchmarks 3DMatch 16 and 3DLoMatch [20 commonly leverage three metrics for evaluation: registration recall (RR), inlier ratio (IR), and feature matching recall (FMR). To keep with existing methods 200] we exclude the consecutive point clouds when evaluating the RR.
Registration Recall refers the percentage of point cloud pairs whose root mean square error (RMSE) is less than a pre-defined threshold т = 0.2m. Given point clouds X, Y with a ground- truth correspondence set C = (z) x; E X, E Y}, estimated rotation R E SO(3), and estimated translation i. E RR3, the RMSE of X, Y is formulated as
RMSE(X, Y)-
1 C
Σ
Rx, +i--t- 111.
(20)
(XI
Inlier Ratio refers the percentage of estimated correspondences whose distance is less than pre-defined threshold
0.1m. Given point clouds X, Y with an estimated correspondence set Č == [(x;, y;) E X, Y), ground-truth rotation R SO(3), and ground-truth translation t E 3 the IR of X, Y is formulated as
IR(X, Y)
Σ (EC
1[[Rx;
y, < 72).
(21)
C
Feature Matching Recall
refers to the fraction of point cloud pairs with an IR> 72 5%
We adopt RR for evaluation as it directly measures the performance on the target task of point cloud registration, while FMR and IR are not suitable to quantize our matching performance for three reasons: (1) our method detects only sparse keypoints for fine matching, hence is fair to be compared with other methods based on dense descriptors or correspondences when using the same number of sampled points for evaluation; (2) our method directly supervises the differential pose estimator to learn virtual correspondences with confidence scores instead of explicit feature matching and outlier rejection, hence our method inevitably retains many outliers which are acceptable for registration but not likely to maintain advanced matching performance; (3) feature matching recall and inlier ratio only reflect the matching performance which not decisive for either accuracy (relative pose errors) or robustness (registration recall) especially for RANSAC-fre methods 22
B 28 49
18

---

Table 8: Inlier ratios and feature matching recalls on indoor datasets 3DMatch.
Inlier Ratio (%)
Benchmark Samples
3DMatch
3DLoMatch
5000
2500
1000
500
5000
2500
1000
250
descriptor- sur---eedd
PerfectMatch
36.0
32.5
26.4
21.5
16.4
11.4
10.1
8.0
6.4
4.8
FCGF S
56.8
54.1
48.7
42.5
34.1
21.4
20.0
11.6
D3Feat
39.0
38.8
40.4
41.5
41.8
13.2
13.1
14.0
14.6
15.0
SpinNet 19
47.5
44.7
39.4
33.9
27.6
20.5
19.0
16.3
13.8
11.1
YOHO
50
64.4
60.7
55.7
46.4
41.2
25.9
23.3
22.6
18.2
15.0
Predator
58.0
57.1
54.1
26.7
28.1
28.3
27.5
25.8
correspondence bes-bd
CoFiNet
49.8
51.2
51.9
52.2
52.2
24.4
25.9
26.7
26.8
26.9
GeoTransformer
12
71.9
76.0
82.2
85.1
45.3
46.2
52.9
57.7
OIF-Net [3
62.3
65.2
66.8
67.1
67.5
27.5
30.0
31.2
32.6
33.1
RoITr 28
82.6
82.8
83.0
83.0
83.0
54.3
54.6
55.1
55.2
55.3
PEAL 29
74.8
81.3
86.0
87.9
89.2
49.1
54.1
60.5
63.6
65.0
SIRA-PCR
49
70.8
78.3
83.7
85.9
87.4
43.3
49.0
55.9
58.8
60.7
DiffusionPCR
30
75.0
81.6
86.3
88.2
89.4
49.7
55.4
61.8
64.5
66.2
CAST
91.2
91.5
93.1
66.3
66.3
66.5
Feature Matching Recall (%)
Benchmark Samples
3DMatch
3DLoMatch
5000
2500
1000
500
250
5000
2500
1000
500
250
descriptor-bee bhppor--bssdd burdb
PerfectMatch 7
95.0
94.3
92.9
90.1
82.9
63.6
61.7
45.2
34.2
FCGF
97.4
97.3
97.0
96.7
96.6
76.6
75.4
74.2
71.7
67.3
D3Feat 9
95.6
95.4
94.5
94.1
93.1
67.3
66.7
67.0
66.7
66.5
19
97.6
97.2
96.8
95.5
94.3
75.3
74.9
72.5
70.0
63.6
YOHO 50
98.2
97.6
97.5
97.7
96.0
79.4
78.1
76.3
73.8
69.1
Predator 20
96.6
96.6
96.5
96.3
96.5
78.6
77.4
76.3
75.7
75.3
saannc-hhseed -aseddhee
CoFiNet
98.1
98.2
83.5
82.6
GooTransformer
12
97.9
97.9
97.9
97.9
97.6
88.3
88.6
88.8
88.6
88.3
correspondencee
OIF-Net
13
98.1
98.1
97.9
98.4
98.4
84.6
85.2
85.5
86.6
87.0
RolTr
28
98.0
98.0
97.9
98.0
97.9
89.6
89.6
89.5
89.4
89.3
PEAL 29
98.5
98.6
98.6
98.7
98.7
89.1
89.2
89.0
89.0
88.8
SIRA-PCR
49
98.2
98.4
98.5
89.0
88.9
88.6
87.7
DiffusionPPR
30
98.3
98.3
98.3
98.3
98.3
86.3
85.9
86.0
86.1
85.9
CAST
98.3
98.3
98.4
98.3
83.1
83.6
85.5
84.7
Feature Matching Performance. Nevertheless, this appendix presents IR and FMR in Table & demonstrate the feature matching performance our method. For IR evaluation, we scale the inlier confidences predicted by compatibility graph embedding to j0, 1] and discard the correspondences with confidences < 0.1. As the number of keypoint correspondences after filtering always less than 1000, we only report the IR regarding <1000 correspondences. Even without filtering, the number of keypoint correspondences is always less than 2500, hence we only report the FMR regarding <2500 correspondences. Enjoying the merits of coarse matching. keypoint detection. and compatibility graph embedding, our method achieves the highest inlier ratio compared to all sorts of baselines. As for FMR, our method performs par with DiffusionPCR [30) on 3DMatch and better than CoFiNet [1] on 3DLoMatch. However, our method performs worse than other coarse-to- to-fine methods [3 28 29 49 30) 3DLoMatch, since it remains challenging to extract enough keypoint correspondences in extremely low overlapping cases due to sparsity.
Indoor Registration Performance. We demonstrate the accuracy of CAST for indoor RGB-D point cloud registration by comparing with various point cloud registration methods [5 g in Table |
32 43
All of the registration methods leverage the prevalent FCGF descriptor and FastMAC (60) uses a sampling ratio of 50%. For a fair comparison, we follow the evaluation
60
to re-compute the registration recall of our method, which is formulated as the fraction point cloud pairs with RTE<30cm and RRE<15". Our method achieves the highest registration recall and the lowest registration errors, suggesting its robustness and accuracy.
strategy of MAC
4
19

---

Table 9: Registration results on indoor RGBD point cloud datasets.
3DMatch RTE (cm)
3DLoMatch RTE (cm)
Methods
(%)
RRE ()
RR (%)
RRE (9)
RANSAC-IM T5
88.42
9.42
3.05
9.77
14.87
7.01
RANSAC-4M 15
91.44
8.38
2.69
10.44
15.14
6.91
TEASER++ 58
85.77
8.66
2.73
46.76
12.89
4.12
SC3-PCR 59
93.16
6.51
2.09
58.73
10.44
3.80
DGR 3J
88.85
7.02
2.28
43.80
10.82
4.17
PointDSC [32
91.87
6.54
2.10
56.20
10.48
3.87
MAC 4
93.72
6.54
2.02
59.85
9.75
3.50
FastMAC [60]
92.67
6.47
2.00
58.23
10.81
3.80
CAST
96.48
5.64
1.71
76.13
8.47
2.75
Table 10: Empirical standard deviations of the evaluation metrics of CAST in repeated experiments.
Dataset
3DMatch FMR
KITTI RTE
nuScenes
Metrics
RR
IR
PIR
PMR
RR
RRE
RR
RTE
RRE
STD
0.4%
0.4%
0.5%
0.3%
0.2%
0.0%
0.1cm
0.01
0.0%
0.1cm
0.01
Generalization Studies. To extensively evaluate the generalizability of the proposed CAST in unseen domains, we conduct a generalization experiment from the outdoor dataset KITTI [39 to another outdoor dataset ETH [6]]. Note that the KITTI and ETH datasets use Velodyne-64 64 3D LiDAR and Hokuyo 2D LiDAR, respectively, leading to very different appearances and distributions of point clouds. Hence, our generalization study is practical generalizability of different methods. For fairness, all methods adopt 30cm for voxel down-sampling, and all methods involving RANSAC set the maximum iterations be 50000 and the confidence be 0.999 as the convergence criteria. To enhance the robustness, our method is combined with RANSAC estimating an initial pose from 250 coarse correspondences to reject the outliers during fine matching, and utilizes the global registration in Geo Transforrrrr 12 to refine the pose estimate.
applications and solid to demonstrate the
We present the translation errors, rotation errors, and the registration recalls in Table [1] Our method achieves satisfying accuracy and robustness, showcasing better generalizability than the coarse-to-fing baseline GeoTransformer [12] and other point-wise descriptors [8] [ 20). We also compare the
our learnable compatibility graph embedding (CGE) with spectral matching (SM) [62) for outlier rejection in our method. With RANSAC filtering out severe outliers in advance, spectral matching can lead to better performance than learning- based CGE in unseen domains. Notably, all point-wise methods including CAST exhibit lower registration recalls in generalization studies than patch-wist local descriptor SpinNet [19 and BUFFER [48] incorporating patch-wit and point-wise features. This mainly because they adopt feature pyramid network architecture to learn features with abundant global context, which is detrimental for generalization [19]. Furthermore,
conduct an unsupervised domain adaptation (UDA) experiment for CAST, which tunes the network by learning to align point cloud to itself after random rotation and cropping. The results indicate that model can casily adapt to an unseen domain and achieve robust and accurate performance after one epoch's unsupervised tuning (only 20 minutes NVIDIA RTX3090 GPU).
Experiment Statistical Signiticance. Finally, Table rerprrts reports the standard deviations (1-sigma) of our evaluation metrics, which are assumed to be Gaussian distributed. Despite the randomness from voxel nown-sampling and RANSAC, the performance of our method remains stable. Notably, the runtime of some methods such as 8 19 50 in TableBaan are quite different from results in [48), since we report the average runtime including data preprocessing. feature extraction, feature matching, and pose estimation, while the source codes of these methods save some intermediate results such as descriptors to avoid repeated calculation of the same point cloud in different pairs, which leads to unfair runtime comparison.
20

---

Table 11: Results of generalization from KITTI to ETH.
Methods
RTE (em)
RRE (?
RR (%)
FCGF
8
9.08
0.94
45.86
Predator
11.72
1.38
65.64
SpinNet
19
6.05
0.98
99.44
TCKDD
1
9.61
0.88
92.43
Geo Transfrmrerr
12
5.97
0.73
91.87
BUFFER 48
6.02
0.71
100.00
CAST (CGE)
6.85
0.65
97.76
CAST (SM)
6.66
0.61
98.04
CAST + UDA
5.25
0.56
99.58
A.4
Qualitative Results
Figure 3 Figure 7 and Figure
8 provide qualitative results about the registration performance on outdoor datasets KITTI 39 nuScenes [4], and the indoor dataset 3DMatch [16], respectively.
A.5 Limitation
The main limitation of the proposed CAST is the sub-optimal performance in low overlapping scenarios such the 3DLoMatch benchmark compared to fate-of-the-arr methods, which may be ascribed to two aspects. (1) There is no effective outlier rejection for coarse matching as it difficult to search inliers from patch correspondences based geometry consistency due low resolution.
(2) Due to the sparsity and non- uniformity of keypoints, the inlier ratio of keypoint correspondences still falls short of what is required for robust pose estimation without a hypothesis sel-selettion pipeline. Nevertheless, this RANSAC-free lightweight fine matching pipeline can achieve satisfying performance in outdoor scenarios. Considering the superior PIR and PMR of coarse matching, we may directly exploit dense feature matching to enhance the robustness in low overlapping point cloud registration scenarios future work.
A.6 Broader Impacts
We present a novel consistency-awe aware spot-guided Transformer based on sparse attention to extract consistent coarse correspondences from point clouds. In addition, propose lightweight fine matching module for versatile and hierarchical point cloud registration, benefiting from the efficiency of sparse keypoint matching and the accuracy of dense registration. Different from existing methods, our fine matching is based on flexible local attention instead of optimal transport heavily relying on patch-to-patch correspondences, thus allowing independent deployment without coarse matching. Besides, the sparsity of keypoints ensures the efficiency of spatial consistency filtering.
Enjoying these merits, this work not only achieves superior accuracy, efficiency, and robustness in point cloud registration, but also paves the way to various large-scale real-time applications, such SfM, SLAM, autonomous driving, any other where point cloud registration plays role. For examples, the reconstruction indoor scenes and objects from unlabeled 3D scans could benefit from work, which can precisely recover the rigid transformation between different scans. Additionally, our fine matching may independently construct a real-time odometry system for SLAM SfM, it capable of efficient and reliable local data association and accurate pose estimation between two large-scale point clouds with a strong pose prior, while our coarse matching could be utilized in place recognition and global re-localization in SLAM.
-based or RGBD camera- based
As our work aims at tackling a fundamental problem in 3D computer vision, we do not anticipate a direct negative impact. Potential negative outcomes might occur in real applications where our method is involved.
21

---

Source Frame with Keypoints
Target Frame with Keypoints
Sparse Keypoint Correspondences
Pose stimaation
Figure 7: Qualitative registration results nuScenes dataset. We show three examples in three columns to demonstrate the effectness of CAST in keypoint extraction, matching, and pose estimation.
RMSE: 3.30m
RMSE: 2.92m
RMSE: 3.71m
RMSE: 0.16m
RMSE: 4.24m
RMSE: 4.87m
AMSE: 4.31m
RMIS:: 0.09m
RMSE: 0.211
RMSE: 0.21m
RMSE: 0.30m
RMSE: 0.16m
RMSE: 2.90m
RMSE: 3.95m
RMSE: 3.94m
RMSE: 0.06m
RISS: 0.29m
RMSE: 2.94m
RMSE: 1.53m
RMSE: 0.07m
CoFiNet
GeoTransformer
RolTr
Ours (CAST)
Ground-Trun
Figure 8: Qualitative registration results of CoFiNet [T]. GeoTransformer [2]. RolTr (28] and CAST compared with the ground truth alignment on 3DMatch dataset. We present five examples in five rows, which demonstrate the robustness and accuracy of our method.
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction clearly state the claims made, including the motivations, contributions, and the performance of our approach.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have detailed the limitations of our work in Sec. A.5 of the appendix, and we also point out the scope our claims made, including the benchmarks and sensors (Sec. A.2) and the computational efficiency (Sec. 4
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [NA]
Justification:
This paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
This paper fully discloses all the information needed to reproduce the main experimental results, including the detailed system architecture in Sec. 3 and some modular architectures along with the
Justification:
in Sec. A. the training and evaluation
settings in Sec. 4 and datasets with metries in Sec. and Sec.
A.3 respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
24

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have submitted the data and codes in the supplementary material with detailed instructions on data access and preparation as well as guidelines to reproduce all experimental results. The paper will provide public access to the codes upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https
/
CobmeSumeeesac / ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This paper fully specifies the training and evaluation settings, including optimizer and the learning rate scheduler in Sec. hyper-parameters in Sec. data splits and preparation in Sec. A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In this paper, all metrics are assumed to be normally distributed, whose empirical standard deviations (1-sigma) in repeated experiments are reported in To demonstrate the experiment statistical significance. The randomness mainly comes from voxel down-sampling and RANSAC.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Sec. we have detailed the type of CPU and GPU our device, and the runtime of the proposed method and nearly all baselines computer.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics atttps:/nneurr: hctpppbbliccfthhcccccueeeeeeeeee
Answer: [Yes]
Justification:
This research conforms with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts discussed in Sec. Af the appendix.
26

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual- should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
This paper involves existing benchmarks and utilizes existing methods for evaluation, whose papers are properly cited with licenses. The related URL are included in our codes.
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
27

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets.papersstt paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper provides details of our model in both main part (Sec. 3 and the appendix (Sec.A..) Our source codes have been submitted in the supplementary material while datasets are proposed.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
28

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "3DwrPptTNc", "reviewer": "Reviewer_D9ok", "summary": "This paper focuses on feature matching for point cloud registration. To this end, it aims to improve the effectiveness of the coarse-to-fine matching mechanism by designing a consistency-aware spot-guided Transformer (CAST). More specifically, the proposed method incorporates a spot-guided cross-attention module to avoid interfering with irrelevant areas, and a consistency-aware self-attention module to enhance matching capabilities. Extensive experiments on both indoor and outdoor datasets are conducted to validate the proposed model.", "questions": "See weaknesses.", "limitations": "Limitations have been included as a part of the Appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The overall writing is satisfied;\n\n2. The experiments on outdoor benchmarks look good", "weaknesses": "1. One major issue of this paper is the novelty of this paper. It looks like the proposed method roughly follows the pipeline of GeoTransformer, but introduces outlier rejection techniques and more attention layers. Moreover, it is also necessary to show the size of the proposed model (in terms of the number of parameters)\n\n2. The second major concern lies in the performance on indoor benchmarks including 3DMatch and 3DLoMatch. As aforementioned, with many modules added to GeoTransformer, the proposed method even fails to outperform it. The advantages of the proposed method on indoor scenarios should be better demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and valuable suggestions. Below, we offer detailed responses to each of your comments and questions. If there are any points where our answers don't fully address your concerns, please let us know, and we will respond as quickly as possible.\n\n- **Weakness 1: Novelty**\n\nOur method roughly follows the pioneering coarse-to-fine matching framework CoFiNet (NeurIPS21'), but there are significant improvements besides performance. We believe this is a novel way incorporating both local and global consistency in feature aggregation rather than outlier removal as existing registration methods such as SC2-PCR and PointDSC. And it is also important that our method makes learning-based registration more efficient and scalable for real-time applications such as odometry and SLAM.\n\nEver since the publishment of CoFiNet (NeurIPS21'), many works roughly follow the pipeline and revise some of the modules, including GeoTransformer(CVPR22'), OIF-Net (NeurIPS22'), PEAL (CVPR23'), etc. Although these works widely adopt self-attention and cross-attention for coarse matching, however, CoFiNet and GeoTransformer adopt global cross-attention for feature aggregation, which inevitably attends to similar yet irrelevant areas, resulting in misleading feature aggregation and inconsistent correspondences. Additionally, these methods focus on matching among very coarse nodes without considering geometric consistency, which is not tight enough for fine matching. Instead, **we focus on feature aggregation among semi-dense features leveraging both local and global geometric consistency to tackle the sparsity and looseness of coarse matching**. To be specific, our consistency-aware self-attention only attends to salient nodes sampled from a global compatibility graph, while our spot-guided cross-attention only attends to nodes selected based on local consistency. **We believe this is a novel way to introduce geometric consistency to feature aggregation rather than outlier rejection such as PointDSC and SC2-PCR.**\n\nMore importantly, **our overall design using a complex coarse matching module and a very lightweight fine matching module is motivated by the efficiency, accuracy and scalability requirements from real-time applications such as LiDAR odometry and SLAM**. In a LiDAR odometry system, it is a consensus that the real-time frame-to-map registration is essential for accuracy, instead of frame-to-frame registration as we do in this task. However, all of the existing coarse-to-fine methods fail to achieve real-time performance. Moreover, their fine matching tightly couples with coarse matching, which requires node-based partitioning and really time-consuming optimal transport for patch-to-patch correspondences. Therefore, existing methods are not scalable to SLAM. We believe the key is only using a lightweight fine matching process, because coarse matching is not necessary in odometry with a small pose deviation. Hence, **we design a sparse-to-dense fine matching pipeline allowing independent deployment without coarse matching to meet these requirements**, which can efficiently establish virtual correspondences for any point and their neighbors in the other point cloud. Moreover, we validate the feasibility of this scheme using a learnable inlier classifier instead of existing time-consuming pose estimators (RANSAC, PointDSC, MAC, etc). Nevertheless, coarse matching is essential for place recognition and global re-localization in SLAM without real-time requirement, hence we introduce the consistency to it to make the whole system robust for large-scale pose deviation.\n\nFinally, **extensive experiments validates the accuracy, robustness, and efficiency on both indoor and outdoor scenes.** It is noteworthy that our model can easily adapts to an unseen domain and achieve robust and accurate performance after a short period of unsupervised tuning (only 20min for a epoch!), which is detailed in **part 3 in author rebuttal**. The model sizes of popular methods are also reported as follow:\n| Method | Model Size (M) |\n| :---: | :---: |\n| REGTR | 11.85 |\n| CoFiNet | **5.48** |\n| Predator | 7.42 |\n| GeoTransformer | 9.83 |\n| RoITr | 10.10 |\n| CAST | 8.55 |\n\n- **Weakness 2: Performance on indoor datasets**\n\nWe update our results on 3DLoMatch and discuss the performance of our method in low overlapping scenes in **part 1 of author rebuttal**. Please refer to it, thank you for your comment!\n\nIt is noted that on indoor benchmark 3DMatch, our method achieves the highest RR of 95.2%. On 3DLoMatch, our method achieves 75.1% RR, surpassing all descriptors and all non-iterative correspondence-based methods except OIF-Net. As for DiffusionPCR and PEAL showing higher RR than others, they iteratively use a variant of GeoTransformer with overlap priors for multi-step matching, which is extremely time-consuming (10x runtime of ours), and PEAL even uses extra information from 2D images. Notably, our method achieves such superior performance only with the lowest time consumption.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lIsrWsOdTZ", "reviewer": "Reviewer_LKCi", "summary": "This manuscript mainly focuses on the learning based feature matching of point cloud registration. The authors propose a consistency-aware spot-guided transformer and a lightweight fine matching module. Experiments on both indoor and outdoor benchmarks prove the effectiveness of the designs. However, this article requires significant revisions and improvements in the narrative of the method introduction. It needs to provide a clearer sequence of the entire process, the inputs and outputs of each module, and the specific implementations of key designs.", "questions": "Please see Weaknesses.", "limitations": "Please see Weaknesses.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. It is reasonable to improve the attention architecture by utilizing geometric consistency constraints. \n\n2. Effective registration result improvements have been achieved on both indoor and outdoor datasets.", "weaknesses": "1. Figure 1 should be presented with greater clarity and precision. Currently, the coarse matching module and fine matching module are not labeled in the figure, making it difficult to correspond the depicted process with the title description.\n2. In section 3.2 of the Methods part, the definition of “spot” by the authors is not clear. “Spot” seems to be a key concept in this section, yet it is only briefly described in line 145. Is “spot” referring to a patch? How does it guide the cross-attention?\n3. The logic in the Methods section is not sufficiently clear, requiring repeated and careful reading. The inputs and outputs of each module are not explicitly stated. For instance, what are the inputs and outputs of the Sparse-to-Dense Fine Matching module in section 3.3? This module is also not represented in the flowchart.\n4. What is the sequence of the Consistency-Aware Self-Attention and Spot-Guided Cross-Attention in the process? Figure 1 shows that self-attention is performed first, so it is recommended to follow the same sequence in the detailed introduction.\n5. In Formula 4, is the pre-defined threshold σc set differently for datasets of varying sizes?\n6. How generalizable is the method proposed in this paper? If trained on the 3DMatch dataset, how would the results be when tested on outdoor datasets such as ETH?\n7. The method designed in this paper seems reasonable and effective. It is suggested that the authors make the code open-source for others to learn from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and valuable suggestions. Below, we offer detailed responses to each of your comments and questions. If there are any points where our answers don't fully address your concerns, please let us know, and we will respond as quickly as possible.\n\n- **Weakness 1.3.4 about writing.**\n\nThank you for your valuable suggestions. **We have carefully revised our writing and figures of Method Section and released them in the supplementary pdf of author rebuttal and general comments, respectively, and it would be appreciated if you can read this version**. \n- - Specifically, we adjust our Figure 1 to clearly label the coarse and fine matching modules with as well as submodules, and include more details to achieve greater clarity and precision.\n- - The inputs and outputs of each module are explicitly stated in our revised version and depicted in the revised Figure 1. For instance, the inputs of the sparse matching are coarse correspondences $\\hat{C}=\\\\{(x_j^S, y_j^S): x_j^S\\in X^{1/4}, y_j^S\\in Y^{1/4}\\\\}$ and the semi-dense nodes $X^{1/4},Y^{1/4}$ with features $F^{1/4}$ from FPN, while the outputs of the single-head attention and the compatibility graph embedding (submodules) are virtual correspondences and their confidence weights, respectivel, which are also the outputs of sparse matching. Then these virtual correspondences with weights along with dense features $F^{1/2}$ are the inputs of dense matching, which estimates the pose for coarse alignment and further local matching, then it outputs the final pose estimate.\n- - Finally, we have added an extra *Architecture* subsection to introduce the pipeline and kept the writing in the same same order as the figure, for instance, the consistency-aware self-attention is before the spot-guided cross-attention. The architecture of our coarse matching module is designed as a sequence of blocks for attention-based multi-scale feature aggregation. For each block with both semi-dense features $F^{1/4}$ and coarse features $F^{1/8}$ as inputs, we first feed $F^{1/8}$ into a self-attention module and a cross-attention module. Then $F^{1/4}$ and $F^{1/8}$ are fused into each other. Finally, semi-dense features are fed into a consistency-aware self-attention module and later a spot-guided cross-attention module at the end of each block. I believe our revised version can address most of your concerns about writing and presentation.\n\nI believe our revised version can address most of your concerns about writing and presentation.\n\n- **Weakness 2 about the description of spot.**\n\nWe have carefully revised this part and you can check it in the supplementary pdf (Figure 2) of author rebuttal and general comments. Here I would like to briefly describe the definition and effect of spot. Before spot-guided cross-attention, we compute a coarse correspondence set by matching, denoted as $C^{(l)}$. For each node $x_i^S$ such that $(x_i^S,y_i^S)\\in C^{(l)}$, we select a subset $\\mathcal{N}_s(x_i^S)$ as seeds from its neighborhood $\\mathcal{N}(x_i^S)$ and construct a region of interest for it as $\\mathcal{S}(x_i^S)=\\bigcup _{x_k^S\\in \\mathcal{N}_s(x_i^S)} \\mathcal{N}(y_k^S)$, namely its **spot**. $\\mathcal{N}_s(x_i^S)$ selects $x_i^S$ and only its neighbors with reliable correspondences, and we propose a consistency-aware matching confidence criterion to rank the neighbors. **Effect:** As shown in Figure 2 of the supplementary pdf, global cross-attention tends to aggregate features from wrong regions under the disturbance of many similar yet irrelevant regions, leading to false correspondences. Our formulation of spot is inspired by local consistency that the correspondences of adjacent 3D points remain close to each other. Hence, the spots defined above are likely to cover the true correspondences of query nodes, providing guidance for feature aggregation without interfering with irrelevant areas.\n\n- **Weakness 5:** In Formula 4, is the pre-defined threshold $\\sigma_c$ set differently for datasets of varying sizes?\n\nYes, the threshold $\\sigma_c$ for evaluating the consistency should be set differently for datasets of varying sizes. For simplification, we set it as the 6 times of the initial voxel size for down-sampling.\n\n- **Weakness 6:** How generalizable is the method proposed in this paper? If trained on the 3DMatch dataset, how would the results be when tested on outdoor datasets such as ETH?\n\nWe present our results of generalizability and discuss the potential in realistic applications in **part 3 of author rebuttal**. Although our method trained on 3DMatch fails to deploy on ETH due to the out of memory error, we evaluate the generalizability of different methods when trained on KITTI and tested on ETH. Note that these datasets use Velodyne-64 3D LiDAR and Hokuyo 2D LiDAR, respectively, leading to very different appearance of point clouds, hence we believe it is solid to show the generalizability of different methods. Our method showcases better generalizability than GeoTransformer, due to the help of consistency. But it falls behind SpinNet and BUFFER using RANSAC mainly owing to the point-wise FPN backbone and lightweight yet learnable fine matching. Moreover, we conduct an UDA experiment, indicating that our model easily adapts to an unseen domain and achieves robust and accurate performance after a short period of unsupervised tuning (only 20min for a epoch!). For applications, we believe generalizing or quickly adapting from an outdoor LiDAR dataset to another is a more realistic setting than generalizing from an RGBD camera dataset 3DMatch to a LiDAR dataset ETH as many papers.\n\n- **Weakness 7:** The method designed in this paper seems reasonable and effective. It is suggested that the authors make the code open-source for others to learn from.\n\nThank you for your suggestion! Our source codes have been submitted as the supplementary material before rebuttal, and we will make it open source once the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "asirBeD6PT", "reviewer": "Reviewer_UeYt", "summary": "This paper introduces a consistency-aware, spot-guided Transformer, adapted from 2D image matching techniques, to minimize interference from irrelevant areas. It incorporates a consistency-aware self-attention module that enhances matching capabilities by using edge length constraints to filter correspondences. Additionally, a lightweight fine matching module is designed for both sparse keypoints and dense features, enabling accurate transformation estimation. The method performs well on outdoor datasets; however, its effectiveness diminishes in scenarios with low overlap.", "questions": "The paper does not specify if the loss supervises the overlap scores, it does not require it?\n\nThe term \"partial tokens\" mentioned in line 180 needs clarification. \nDetails on computing the coarse matching for spot-guided attention are also needed.\n\nSuggestion: \n1. Ensure consistency in font sizes across Tables 3 and 4.\n2. The terminology in line 123 could be improved by using \"superpoints\" or \"nodes\" instead of \"patches,\" as the latter suggests a collection of points rather than individual points or nodes.", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The motivation to make learning-based registration more efficient and scalable for real-time applications is both interesting and important. I appreciate the introduction of the edge equality constraint into the registration pipeline, which guides the correspondence search in a more efficient manner compared to COTReg [1].\n\nThe method performs well on outdoor datasets; however, its effectiveness diminishes in scenarios with low overlap.", "weaknesses": "The method section of this paper is challenging to follow. It would be beneficial to introduce each method sequentially, aligned with the flow depicted in the figure—for example, starting with self-attention, followed by cross-attention, and then linear cross-attention. \n\nThe captions of Figure 1 should clearly label the coarse and fine matching modules, as they are not marked in the current figures, making it difficult to understand. Furthermore, it's unclear which network layer corresponds to \"the third level\" mentioned in line 123—is it part of the encoder or decoder? It is also hard to distinguish between spot attention and other forms in Figure 1. \n\nThe methodology for obtaining overlap scores is not adequately explained and is not illustrated in Figure 1. \n\nThe performance on datasets with low overlap seems subpar, which is concerning since handling cases of low partial overlap is crucial.\n\nThe authors claim that existing methods are not efficient or scalable for real-time applications like odometry in robotics; however, only time comparisons are provided in Table 3, and these do not clearly demonstrate an advantage. Additionally, there is no assessment of performance on large-scale datasets. Therefore, the authors should provide more comprehensive evaluations, including comparisons of other metrics such as FLOPS.\n\nThis paper lacks visualizations for the spot-guided attention, which are essential for demonstrating the method's effectiveness.\n\nThe manuscript lacks critical references such as COTReg [1] by Mei et al., which discusses geometric consistency and should be cited. Comparisons with state-of-the-art methods cited in references [2] and [3] are also necessary to validate the proposed method's efficacy.\n\n[1] Mei, Guofeng, et al. \"COTReg: Coupled optimal transport based point cloud registration.\" arXiv preprint arXiv:2112.14381 (2021).\n[2] Zhang, Yifei, et al. \"FastMAC: Stochastic Spectral Sampling of Correspondence Graph.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[3] Huang, Tianyu, et al. \"Scalable 3d registration via truncated entry-wise absolute residuals.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and valuable suggestions. Below, we offer detailed responses to each of your comments and questions. If there are any points where our answers don't fully address your concerns, please let us know, and we will respond as quickly as possible.\n\n- **Weakness 1-3 about writing.**\n\n**We have carefully revised our writing and figures of Method Section in the general comments, and it would be appreciated if you can read it**. Specifically, we adjust our Figure 1 to clearly label the coarse and fine matching modules and include more details such as the prediction of overlap scores. We explicitly describe our pipeline in both Figure 1 and an extra Architecture subsection in the same order. I believe our revised version can address most of your concerns.\n\n- **Weakness 4** about performance in low overlapping scenes.\n\nWe update our results on 3DLoMatch and discuss the performance of our method in 3DLoMatch in **part 1 of author rebuttal**. Please refer to it, thank you!\n\n- **Weakness 5** about our claim of potential in large-scale applications.\n\nThe FLOPs of our method is significantly lower than GeoTransformer and RoITr, indicating its efficiency. Although FLOPs of other methods are lower, their runtime is much more. We believe the runtime on RTX3080Ti is more direct than FLOPs only focusing on deep models.\n\n| Method | FLOPs (G) |\n| :---: | :---: |\n| CoFiNet | **23.46** |\n| Predator | 36.89 |\n| GeoTransformer | 271.23 |\n| RoITr | 422.14 |\n| CAST | 102.16 |\n\nBy the way, \"large-scale applications\" in our manuscript may be misleading, thank you for pointing it out! Instead of large-scale datasets, it mainly stands for SLAM in our context, which requires frame-to-map registration for accuracy, instead of frame-to-frame registration as we do in this task. Compared with our task, the frame-to-map registration is large-scale. Here we would like to clarify why our design has greater potential in this application.\n\nAt present, existing coarse-to-fine methods fail to achieve real-time performance in SLAM. Moreover, their fine matching tightly couples with coarse matching, which requires node-based partitioning and really time-consuming optimal transport. Therefore, they are not scalable to SLAM. We believe the key is only using a lightweight fine matching process, because coarse matching is not necessary in odometry with a small pose deviation. Hence, **we design a sparse-to-dense fine matching pipeline allowing independent deployment without coarse matching to meet these requirements**, which can efficiently establish virtual correspondences for any point and their neighbors in the other point cloud. Moreover, we validate its feasibility using a learnable inlier classifier instead of existing time-consuming pose estimators (RANSAC, MAC, etc). Nevertheless, coarse matching is essential for place recognition and global re-localization in SLAM without real-time requirement, hence we introduce the consistency to it to make the whole system robust for large-scale pose deviation.\n\n- **Weakness 6:** lacks visualizations for the spot-guided attention.\n\nWe add visualizations of vanilla global cross-attention and our spot-guided cross-attention of nodes from both salient areas and non-salient areas, as is shown in **Figure 2 in the supplementary pdf**, which demonstrates its effects to select instructive areas to attend to according to local consistency, while vanilla cross-attention may be attracted by many irrelavant areas.\n\n\n- **Weakness 7:** The manuscript lacks critical references. Comparisons with state-of-the-art methods cited in references [2] and [3] are also necessary.\n\nThank you for pointing it out. We add the mentioned reference in our revised version in general comments. Comparisons with these SOTA registration methods are shown in **Table 1 in author response**, which further validate the efficacy of our method. For fairness, all of these methods use the FCGF descriptor rather than coarse-to-fine methods. Rather than focusing on feature extraction or matching as the baselines included in our manuscript, these methods focus on how to search a consensus set for robust pose estimation, using a pipeline more complex than our sparse-to-dense fine matching and LGR in GeoTransformer. We believe this is why existing papers only compare these registration methods with each other using the same descriptors or correspondences.\n\n- **Q1:** The paper does not specify if the loss supervises the overlap scores, it does not require it?\n\n**A:** Our coarse matching loss $\\mathcal{L}_c$ in Equation (12), Line 233 simultaneously supervises the coarse matching and the overlap scores, since the final matching score is the product of feature similarities and overlap scores, and we also include the negative log-likelihood terms of overlap scores whoose labels are \"0\". Besides, we add an ablation study by ablating this overlap head as is shown in **Table 2 in the supplementary pdf**, which demonstrates its importance.\n\n- **Q2:** The term \"partial tokens\" needs clarification. Details on computing the coarse matching for spot-guided attention are also needed.\n\n**A:** \"Partial tokens\" means instead of attending to all nodes, our attention only select some of them to attend to. To be specific, the consistency-aware self-attention only attends to salient nodes sampled from a global compatibility graph, while the spot-guided cross-attention only attends to nodes selected based on local consistency, i.e., the spot. We have carefully revised our writing and figures of **Method Section in the general comments and supplementary pdf**, which clearly provides the details of how to match the semi-dense nodes and evaluate their consistency in **Subsection Architecture**.\n\n- **Suggestion**\n\nThank you for pointing it out! We will ensure consistency in font sizes across Tables 3 and 4. And we also replace the terminology \"patches\" in line 123 or any similar cases with \"nodes\", which is more appropriate.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wP2hU6aFt6", "reviewer": "Reviewer_L6BY", "summary": "This work proposed a coarse-to-fine matching approach for point cloud registration, where a consistency-aware spot-guided transformer is proposed to ensure the coarse matches to be geometric consistent and around the overlap regions. This method is compared with SOTA PCR methods on three benchmarks and the experiment results validate its superior performance in terms of reregistration accuracy, success rate and time cost.", "questions": "In table 3, the performance of CAST+RANSAC is better than CAST means the proposed coarse-to-fine matching approach is not strictly geometric consistent, which arises a question that instead of introducing the geometric consistency to the feature matching process, why not making it as a separate and following process, just like RANSAC or Spectral Matching ?", "limitations": "I didn’t see any limitation.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Extensive experiments and SOTA results. The proposed method is evaluated on three benchmarks including two outdoor and one indoor cases, where challenging cases such as low overlapping data are also included. It achieves the SOTA performance in terms of registration accuracy, success rate,  time cost. \n\n2. The idea of incorporating geometric consistency to increase the distinctiveness of the learned feature is novel. For pure 3D descriptor methods, such as SPINNet, YOHO, only rotation-equivalence is considered. With the help of geometric consistency, the proposed method showed superior feature matching performance in Table 3. \n\n2. The wiriting are clear and easy to follow.", "weaknesses": "I don’t see any significant weakness of this work. Some minor comments are here:\n\n1.  In line 123, “third level” and “fourth level” need to be explained more or illustrated in the Figure 1. \n\n2.  Please clear state the proposed contribution or difference with respect to the existing work. For instance, the “spot”, cross-attention, self-attention are also discussed in DiffusionPCR, GeoTransformer, and geometric consistency is widely applied by many PCR method, such as PointDSC, RANSAC, Spectral Matching [1].\n\n3. Even though the additional experiment is not encouraged during this process, I would recommend the generalizability of the proposed method can be discussed. I can imagine the proposed method should have better generalizability than existing correspondence-based methods, due to the help of geometric consistency. But, how about it compared to description-based methods (SpinNet, YOHO) with RANSAC/PointDSC ? Trained on indoor dataset and test on outdoor cases?  \n\n[1] A spectral technique for correspondence problems using pairwise constraints. ICCV 2015.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and valuable suggestions. Below, we offer detailed responses to each of your comments and questions. If there are any points where our answers don't fully address your concerns, please let us know, and we will respond as quickly as possible.\n\n- **Weakness 1:** In line 123, “third level” and “fourth level” need to be explained more or illustrated in the Figure 1.\n\nFor feature extraction based on KPConv, we need to down-sample the raw point clouds with a series of voxel sizes $v, 2v, 4v, 8v$ before each encoder layer, resulting in the first level, the second level, the third level and the fourth level points. As the feature maps in the decoder of these points are with the size of $1/k$ of that of the input point cloud $X$ where $k=1,2,4,8$, respectively, we will introduce the notation $X^{1/k}$ and $F_X^{1/k}$ to symbolize the points and their features to make our maniscript more easy to understand. **We have carefully revised our writing and figures of Section.3 Method and released this version as a general response, and it would be appreciated if you can read our revised overview part**.\n\n- **Weakness 2**: Please clear state the proposed contribution or difference with respect to the existing work.\n\nExisting work widely adopts self-attention and cross-attention for coarse matching, however, CoFiNet and GeoTransformer adopt global cross-attention for feature aggregation, which inevitably attends to similar yet irrelevant areas, resulting in misleading feature aggregation and inconsistent correspondences. DiffusionPCR follows an iterative matching scheme that explicitly selects overlapped regions to attend to, leading to much longer inference time. Additionally, these methods focus on matching among very coarse nodes without considering geometric consistency, which is not tight enough for fine matching. Instead, we focus on feature aggregation among semi-dense features leveraging both local and global geometric consistency to tackle the sparsity and looseness of coarse matching. To be specific, our consistency-aware self-attention only attends to salient nodes sampled from a global compatibility graph, while our spot-guided cross-attention only attends to nodes selected based on local consistency, i.e., the spot.\n\nAs for geometric consistency, existing work only uses it to search consistent correspondences (outlier rejection) from a given correspondence set for robust pose estimation, such as PointDSC, spectral matching, etc. However, our method leverages geometric consistency in feature aggregation during coarse matching, instead of outlier rejection. For fine matching, we adopt the geometric consistency for inlier prediction, which is similar to PointDSC but without time-consuming hypothesis-and-verification pipelines. Therefore, our attention-based modules and the way using geometric consistency are very different from existing work.\n\n- **Weakness 3**: I would recommend the generalizability of the proposed method can be discussed.\n\nWe present our results of generalizability and discuss the potential in realistic applications in **part 3 of author rebuttal**. Please refer to it, thank you for your suggestion!\n\nAlthough our method trained on 3DMatch fails to deploy on ETH due to the out of memory error, we evaluate the generalizability of different methods when trained on KITTI (outdoor) and tested on ETH (outdoor). Note that these datasets use Velodyne-64 3D LiDAR and Hokuyo 2D LiDAR, respectively, leading to very different appearance of point clouds, hence we believe it is solid to show the generalizability of different methods. Our method achieves satisfying accuracy and robustness, showcasing better generalizability than GeoTransformer, due to the help of consistency. But it falls behind SpinNet and BUFFER using RANSAC mainly owing to the point-wise FPN backbone and lightweight yet learnable fine matching. Furthermore, we conduct an unsupervised domain adaptation experiment, indicating that our model easily adapts to an unseen domain and achieves robust and accurate performance after a short period of unsupervised tuning (only 20min for a epoch!). For applications, we believe generalizing or quickly adapting from an outdoor LiDAR dataset to another is a more realistic setting than generalizing from an RGBD camera dataset 3DMatch to a LiDAR dataset ETH as many papers.\n\n- **Q:** Instead of introducing the geometric consistency to the feature matching process, why not making it as a separate and following process?\n\n**A:** Our overall design using a complex coarse matching module and a lightweight fine matching module is motivated by the efficiency, accuracy and scalability requirements from real-time applications such as LiDAR odometry and SLAM. In a LiDAR odometry system, it is a consensus that the real-time frame-to-map registration is essential for accuracy, instead of frame-to-frame registration as we do in this task. However, all of the existing coarse-to-fine methods fail to achieve real-time performance. We believe the key is only using a lightweight fine matching process, because coarse matching is not necessary in odometry with a small pose deviation. Existing hypothesis-and-verification pipelines such as RANSAC and PointDSC are robust but time-consuming. Hence, we propose a lightweight fine matching module allowing independent deployment without coarse matching to meet these requirements. Nevertheless, coarse matching is essential for place recognition and global re-localization in SLAM without real-time requirement, hence we introduce the consistency to it to make the whole system robust for large-scale pose deviation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

Thank you for dedicating time to review our paper. We thank the reviewers **L6BY,LKCi,UeYt** for your appreciations of our idea using consistency in feature aggregation, and thank the reviewer **UeYt** for pointing out that we make learning-based registration more efficient and scalable for real-time applications. We thank all reviewers for highlighting our superiority on outdoor scenes, especially reviewers **LKCi,L6BY** also highlighting our superiority on indoor scenes, and we thank reviewers **L6BY,D9ok** for compliments on writing. Meanwhile, reviewers **LKCi,UeYt** concerns about some details of writing, while reviewers **UeYt,D9ok** concerns about the performance on indoor low overlapping scenes. Generalizability tests are suggested by reviewers **LKCi,L6BY**. Here we address these common issues:

## 1. Performance in low overlapping cases
We update the evaluation results in **Table 1 in the supplementary pdf**, as we notice the calculation of RR in our original version is not the same as other methods. In the orginal manuscript we report the RR over the whole dataset but other methods use the average RR of eight sequences in their codes, hence we use the same evaluation protocol for fairness.

On 3DMatch, our method **achieves the highest RR of 95.2%**. On 3DLoMatch, our method achieves **75.1% RR, surpassing all descriptors and all non-iterative correspondence-based methods except OIF-Net.** As CAST typically detects about 1000 keypoints and establishes <250 keypoint correspondences on 3DLoMatch after consistency filtering, it is fair to compare with other methods using only 250 points. Our method outperforms OIF-Net using 1000,500,250 points, indicating its efficacy in low overlapping scenes. Though PEAL and DiffusionPCR show higher RR on 3DLoMatch, they iteratively use a variant of GeoTransformer with overlap priors, which is extremely time-consuming (10x runtime of ours), and PEAL even uses priors from 2D images. Notably, our method achieves such superior performance only with the lowest time consumption. 

As suggested by reviewer **UeYt**, we compare CAST with SOTA registration methods using FCGF descriptor cited in FastMAC and TEAR(CVPR24') to further validate its efficacy. For fairness, we use their RR, i.e., the fraction of point clouds with RTE<30cm and RRE<15°. As reported in Table 1, our method achieves the highest RR and the lowest registration errors, suggesting its robustness and accuracy.

**Table 1: Comparison on 3DM (3DMatch) and 3DLM (3DLoMatch).**
|  | RR (%) | RTE (cm) | RRE (°) | RR (%) | RTE (cm) | RRE (°) |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|  | 3DM | 3DM | 3DM | 3DLM | 3DLM | 3DLM |
RANSAC-4M | 91.44 | 8.38 | 2.69 |10.44 | 15.14 | 6.91 |
TEASER++ | 85.77 | 8.66 | 2.73 | 46.76 | 12.89 | 4.12 |
SC$^2$-PCR | 93.16 | 6.51 | 2.09 | 58.73 | 10.44 | 3.80 |
DGR | 88.85 | 7.02 | 2.28 | 43.80 | 10.82 | 4.17 |
PointDSC | 91.87 | 6.54 | 2.10 | 56.20 | 10.48 | 3.87 |
MAC | 93.72 | 6.54 | 2.02 | 59.85 | 9.75 | 3.50
FastMAC | 92.67 | 6.47 | 2.00 | 58.23 | 10.81 | 3.80 |
CAST | **96.48** | **5.64** | **1.71** | **76.13** | **8.47** | **2.75** |

## 2. Revision of the manuscript and Figure 1
**We have carefully revised our writing and figures of Method Section and released them in the general comments and the supplementary pdf below, and it would be appreciated if you can read this version**. I believe it can address most reviewers' concerns about writing.

## 3. Generalizability
For fair comparison with existing methods trained on 3DMatch (indoor) and evaluated on ETH (outdoor), we need to set the voxel size very small as they do during down-sampling, but our model with multi-scale feature aggregation runs out of memory on RTX3090. This can not be solved even when we scale the point clouds to 1/8 of the original size. Therefore, we only evaluate the generalizability when trained on KITTI (outdoor) and tested on ETH in Table 2, where RR on ETH represents the fraction of point cloud pairs with RTE<0.3m and RRE<2°. Our method achieves satisfying accuracy and robustness, showcasing better generalizability than GeoTransformer, due to the help of consistency. But it falls behind SpinNet and BUFFER using RANSAC.

**Table 2**
|  | RTE (cm) | RRE (°) | RR (%) |
| :---: | :---: | :---: | :---: |
| FCGF | 6.13 | 0.80 | 39.55 |
| Predator | 7.88 | 0.87 | 71.95 |
| SpinNet |**3.63** | 0.62 | 99.44 |
| GeoTransformer | 8.01 | 0.89 | 93.55 |
| BUFFER | 3.85 | **0.57** | **99.86** |
| CAST | 6.86 | 0.66 | 97.05 |
| CAST + UDA 1epoch | 5.29 | 0.58 | 99.44 |
| CAST + UDA 2epoch | 4.96 | **0.57** | **99.86** |

Patch-wise features like SpinNet and BUFFER have good generalizability due to local characteristics that are inherently more robust than point-wise features extracted from FPN, such as FCGF, Predator, and coarse-to-fine methods. Moreover, we uses a lightweight learnable pipeline for fine matching without robust pose estimators, which is not likely to achieve high robustness in lower inlier ratio cases. Hence, we believe that the subpar generalizability mainly owes to the backbone and fine matching. Nevertheless, coarse-to-fine methods significantly outperform descriptors when trained and evaluated on the same dataset.

Furthermore, we conduct an unsupervised domain adaptation experiment, which mixes the KITTI point clouds with ground-truth poses and the ETH point clouds without ground-truth poses for fine-tuning. For ETH, we augment a point cloud via random rotation and cropping and learn to align it with itself before augmentation. **The results indicate that our model easily adapts to an unseen domain and achieves robust and accurate performance after a short period of unsupervised tuning (only 20min for a epoch!)**. For applications, we believe generalizing or quickly adapting from an outdoor LiDAR dataset to another is a more realistic setting than generalizing from an RGBD camera dataset 3DMatch to a LiDAR dataset ETH as many papers.

Meta-review: This paper received mixed ratings of two Borderline Accepts, Reject, and Accept from four reviewers. During the rebuttal phase, Reviewer UeYt (who suggested Reject) believed that its concerns have been addressed, but did not change the rating. After reading the reviewers' comments, the authors' responses and discussions, the AC believes the work can be accepted. Meanwhile, the authors are encouraged to improve the writing of this paper and include these important information provided during rebuttal into the camera ready version.
Decision: Accept (poster)