Multi-scale Consistency for Robust 3D Registration via Hierarchical Sinkhorn Tree
Authors: Chengwei Ren
Yifan Feng
Weixiang Zhang
Xiao-Ping Zhang
Yue Gao
Venue: NeurIPS 2024 poster (2024)
Keywords: Point Cloud Registration
Coarse-to-fine Mechanism
Correspondence Retrieval
Multi-scale Consistency
Primary area: machine_vision
TL;DR: We present Hierarchical Sinkhorn Tree, a pruned tree structure designed to hierarchically measure the multi-scale consistency of coarse correspondences for point cloud registration.

Abstract: We study the problem of retrieving accurate correspondence through multi-scale consistency (MSC) for robust point cloud registration. Existing works in a coarse-to-fine manner either suffer from severe noisy correspondences caused by unreliable coarse matching or struggle to form outlier-free coarse-level correspondence sets. To tackle this, we present Hierarchical Sinkhorn Tree (HST), a pruned tree structure designed to hierarchically measure the local consistency of each coarse correspondence across multiple feature scales, thereby filtering out the local dissimilar ones. In this way, we convert the modeling of MSC for each correspondence into a BFS traversal with pruning of a K-ary tree rooted at the superpoint, with its K nearest neighbors in the feature pyramid serving as child nodes. To achieve efficient pruning and accurate vicinity characterization, we further propose a novel overlap-aware Sinkhorn Distance, which retains only the most likely overlapping points for local measurement and next level exploration. The modeling process essentially involves traversing a pair of HSTs synchronously and aggregating the consistency measures of corresponding tree nodes. Extensive experiments demonstrate HST consistently outperforms the state-of-the-art methods on both indoor and outdoor benchmarks.

Multi-scale Consistency for Robust 3D Registration via Hierarchical Sinkhorn Tree
Chengwei Ren
Yifan Feng
Weixiang Zhang² Xiao-Ping Zhang 1.2- Yue G
( ('Shenzhen Ubiquitous Data Enabling Key Lab, "Shenzhen International Graduate School, BNRist, THUIBCS, School Software). Tsinghua University
rew22@@mals.tinnguu.. edu.en, evanfeng97 gmail.com, zhang-wx22 @aiilk..inghua..du.ee. xpzhang @ ier,.org, gaoyue tsinghua.edu.cn
Abstract
We study the problem of retrieving accurate correspondence through multi-scale consistency (MSC) for robust point cloud registration. Existing works in coarse- to-fine manner either suffer from severe noisy correspondences caused by unreliable coarse matching or struggle to form outlier-free coarse-level correspondence sets. To tackle this, we present Hierarchical Sinkhorn Tree (HST), pruned tree structure designed hierarchically measure the local consistency of each coarse correspon- dence across multiple feature scales, thereby filtering out the local dissimilar ones. In this way, we convert the modeling of MSC for each correspondence into BFS traversal with pruning of K-ary tree rooted at the superpoint, with its K nearest neighbors in the feature pyramid serving as child nodes. To achieve efficient prun- ing and accurate vicinity characterization, we further propose novel overlap-aware Sinkhorn Distance, which retains only the most likely overlapping points for local measurement and next level exploration. The modeling process essentially involves traversing a pair HSTs synchronously and aggregating the consistency measures of corresponding tree nodes. Extensive experiments demonstrate HST consistently outperforms the state-of-the aar methods on both indoor and outdoor benchmarks.
Introduction
Point cloud registration crucial task in 3D computer vision that in- volves aligning a pair of partially overlapped point clouds to create a unified scene representation. The most common approaches 3 follow the two stage technical roadmap, i.e., matching and trans- formation. They first form a set high confident correspondences through repeatable feature descriptors, and then a robust estimator is utilized to calculate the rigid transformation. Although extensively studied over the past decades, the task remains challenging due limited overlap, severe noise, etc.
Coarse
soale
Putadie Match
Sinkhorn Distance
Sinkhorn
Distance
Fine scale
Recent advances 4 5 6 7 have made substantial progress in learning-basse based methods. The key idea is train shared network to extract point-wise features and establish reliable correspondences based on them. Inspired by the progress image registration counter- parts 8 10
Figure 1: Illustration of the proposed HST for measuring MSC. extracts local patches at multi-scales in the feature pyramid and then calculates their similarity layer by layer.
coarse-to- fine strategy 2 13 is leveraged to avoid keypoint detection and unrepeatable correspondence searching. It has demonstrated superior performance over the state-of-the oftee-of-the-at art methods. They establish sparse superpoint correspondences on the downsampled input point clouds (coarse level) and then refine them to
point level to yield dense point
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input
Coarse Level (
Fine Level
MSC Modeling Module
Course Matching
BTS
ovelap
ppanll
Patch Pentune
Tran Transformaa cformattion
Ov Overlap swars Sink Dist,
Fine Motching
Coarse Matches
C
MSC
Grouping
D P
Patch Overlap Prediction (POP)
Overlap-aware Sinkhom Distance (Overlap SD)
Overlap
Overlag SD
Pninod
x
Low
Overag--ware
Marginal
Price
Optinal Transpor
Points
Solver
esp
MI.P
dusthin
Overlap Score
Fültered
Overlap SD
Potential
Overlap Score
space
layerWich
EdgeComn
Attension
Overlap
Hierarchical Sinkhorn Tree
Figure 2: Overview of the proposed method. The overall architecture (Top) is designed in a coarse- to-fine manner, which extracts coarse correspondences and then refines them to point level. Our proposed method contains the following parts: 1. Local exploration (Leftt) extracts local patch pairs for each putative match in multiple scales from the feature pyramid (Sec. 3.2.1 Patch Overlap Prediction module (Bottom Middle) is then adopted to predict the overlap points between patch pairs (Sec. 3.2.2). Overlap-awar Sinkhorn Distance (Bottom Right) measures the patch similarity by focusing on potential overlap points (Sec. 3.2.3). Finally, we repeat the above operations across the layers construct the Hierarchical Sinkhorn Tree (Left) in a BFS way. The process of modeling MSC essentially involves traversing pair of HSTss and then aggregating the consistency measures of the corresponding tree nodes (Sec. 3.2.4).
correspondences (fine level). Therefore, the accuracy of coarse correspondences directly impacts that of fine correspondences [12 14] Intuitively, pair of incorrect coarse matches may introduce potential outlier fine correspondences, thereby influencing the final transformation estimation.
14 15 attempts to nip it in the bud during coarse matching. They adapt the Transformer [16] to unordered point cloud representation mitigate matching ambiguity. series of embeddings are proposed facilitate learning discriminative geometric consistency in both explicit and implicit manners for more accurate matching. The second 17
Prior works try alleviate this problem from two perspectives. The first 13
20 22 ignores the inaccuracy in coarse stage and directly filters out the outlier correspondences at fine stage, known as outlier removal methods. Such methods distinguish between inlier and outlier by developing well-designed compatibilities characterize the affinity relationship between correspondences in geometric or feature space. Though the above techniques have achieved surprising performances, both of them still suffer from noise and low overlap. Specifically, the first only considers coarse-level feature interactions, lacking exploration of finer-scale local consistency. Erroneous correspondences are more likely to occur, especially in low-overlapping scenarios. The second suffers from inaccurate outlier filtering when confronted with a set of high outlier rates correspondences caused by severe noise. Consequently, the outliers, which arise primarily from inaccurate matching in the coarse stage, would inevitably be involved in the transformation estimation step, ultimately resulting in failed registration.
23
to
To tackle the above problems, we propose the Hierarchical Sinkhorn Tree with overlap- Sinkhorn distance model the multi-scale consistency (MSC) of correspondences for more accurate coarse-level matching. Multi-scale consistency analysis, due to its innate capability to analyze patterns different levels of detail, has been variously applied to matching- 25
26 27 and retrieval- (8 29)
related vision tasks. However, in point cloud registration, where the irregular nature of data hinders the promotion of corresponding image MSC analysis methods, leveraging them for more reliable correspondence retrieval remains unexplored. Given that MSC characterizes the local consistency of matches across multiple feature scales, which effectively addresses the challenge of inaccurate matches at the coarse stage, aim to tackle them both. The key idea is to hierarchically evaluate vicinity geometric similarity of each correspondence at multiple scales using Sinkhorn distance 30) and filter out highly dissimilar coarse matches. Specifically, employ Local Exploration to

---

extract local patches at each correspondence's next decoder layer. Subsequently, the overlap-award Sinkhorn distance (over- lap SD) measures the patch differences (see Fig. 3 by focusing the Sinkhorn operation on potential overlap points within the patch, disregarding the non-overlapping ones. This significantly enhances the robustness to noise points while achieving com- putational savings. Following this, a subset of the most likely overlapping points is retained for further local exploration and overlap SD measurement. This repetitive process refines the multi-scale local consistency of each correspondence as the scale becomes finer. Finally. we aggregate the consistency mea- sures across all scales and utilize probabilistic search to select the most reliable coarse correspondences robustly. An overview of our method can be found in Fig.
SD
Hoe
SD: 0,678
Figure A toy example illustrat- ing the Sinkhom Distance (SD) be- tween local patches. The patch pair from inlier correspondence (green) maintains lower SD metric.
To sum up, our main contributions are three-fold:
To the best of our knowledge, we are the first to introduce multi-scale consistency into point cloud registration task to mitigate the effects of low overlap and high noise.
We propose a method for modeling multi-scale consistency called HST, which character- izes the similarity of potential overlapping points in the vicinity areas layer by layer and aggregates them into multi-scale consistency.
We introduce an overlap-aware aware Sinkhorn Distance to focus optimal transport processes only on potential overlapping points, significantly enhancing the robustmess of consistency calculations while reducing solution complexity.
Extensive experiments on both indoor and outdoor benchmarks demonstrate our scene-agnostic superiority. HST significantly outperforms the state-of-the-art methods on Registration Recall on the challenging 3Dth benchmark.
2
Related Work
Correspondence- point cloud registration. Early correspondence- bbseeedeeeeeeeddeddeeee registration meth- ods follow the detect-and- -nantransformm pipelines. A series of local geometric descriptors []] 32 33 are proposed detect repeatable salient points of point cloud pairs. Then point-level correspondences 5 6 are established to recover the transformation between point clouds through robust estimator
20 21 22]
Recently, detector-fre methods 12
1 15 to avoid unrepeatable keypoint detection
are proposed. They introduce a coarse-to- to-fine manner derived from 2D image matching [
[
I0 [1]
to shrink the correspondence searching space. It first extracts full and reliable matches on coarse- resolution features and then refines them on the corresponding local patch from finer resolution. It's becoming prevalent due to its excellent matching accuracy and computational efficiency. Our method follows the technical roadmap of fetector-free methods and focuses on improving the correspondence reliability of the coarse phase.
advance 2D image matching by leveraging coarse-to-fing fashion to avoid unrepeatable keypoint detection while increasing matching reliability, known as detector- free methods. DualRC-Net [8] extracts coarse feature maps to form complete correlations and generate pixel-level matches with the help of fine features. Patch2pix 9 refines coarse patch matches by regressing fine pixel matches from local regions. Loftr [10] and CasMTR [I] introduce an attention mechanism (16) to search for accurate low-resolution correspondences and then establish dense matching based on informative high-resolution local patches.
Coarse-to-fie matching. Recent works R 9 10
Multi-scale consistency. The core idea of multi-scale consistency is evaluate the similarity of matches across multiple feature scales. The criterion under this for evaluating an inlier match that each spatial resolution of feature pyramids should maintain certain degree of similarity. BiseNet [34] utilizes an auxiliary loss to supervise the consistency between multi-scale features and ground-truth to enhance the feature representation. MSCAN (28 leverages a multi-scale attention module to capture features of different scales and aggregates them into a global descriptor for robust image retrieval. ACMM (25 conducts the multi-scale geometric consistency to refine depth maps at each scale, reaching satisfying performance.
3

---

3 Method
3.1 Problem Statement
the goal is to recover a rigid transformation T(R. t) that aligns the two point clouds with rotation R E SO(3) and translation R We utilize the KPConv-FPN [33 the backbone and leverage geometric self- and cross-attention [13] to estimate C' pairs of coarse correspondences for registration in a coarse-to- fine manner 12 13. Our goal remove the outlier correspondences from the above estimated correspondences for further accurate and robust registration. The key idea is to hierarchically model the vicinity similarity at multiple scales between each correspondence.
Given two point clouds X = (x, E R | I..N, and Y (y, E i i
3.2 Hierarchical Sinkhorn Tree
We first propose Multi-scale Consistency (MSC) for coarse level correspondences, which assumes that each correspondence contains similar local features different down-lup-sampling scales. I.e., given
a pair of points x and y with their local patch features L scales be {Fx (1), Fx (2) ..... Fx., Fx (1)
and {Fy (1) Fy Fx (D), then x and y is an putative inlier correspondence only they satisfy:
Oa. LE [I, Σ dF (4) Fx
(1)
where dist(-) is the pre-defined metric for measuring the patch difference and 0₂ is the threshold.
We further propose the Hierarchical Sinkhorn Tree with Overlap-awar Sinkhorn Distance model the MSC of correspondences. It first applies Local Exploration (Sec. (.2.1] to extract local patch at next finer scale from the feature pyramid. Then the Overlap- Sinkhorn Distance (Sec. (32.3 picks the overlap points from patches using Patch Overlap Prediction (Sec. 3.2.2 module and performs Sinkhorn Distance with overlap-aware marginal initialization on them. These two steps are performed layer-wisely in a hierarchical tree way called the Hierarchical Sinkhorn Tree (Sec. 3.2.4.
3.2.1 Local Exploration
varies point density by altering the size of grid cells at each layer form feature pyramids. Morcover, the features from FPN decoder provide sufficient information to analyze the vicinity similarity between correspondences due the skip connections. Consequently, we perform nearest-neighbor exploration [35] on the match's 'ss neighboring points from subsequent decoder layer.
KPConv-FPN 33
Given point cloud X (!) from the 1-th layer of decoder, and its next layer point cloud ((+1),, where 0 1 T, and L is the number of decoder. We have k-nearest neighbor (k-NN) search obtain the local patch (1++)) from the next layer for point (!)
p((-1) = argtopk 200 exar x 111))2))
3.2.2 Overlap Points Prediction
After the above local exploration, each correspondence obtains pair of local patch features from the first decoder layer. We then utilize them to predict overlap points between each patch pair for subsequent Overlap-aware Sinkhorn Distance Computation. We extract overlap information from local patches using a simple yet effective Patch Overlap Prediction (POP) module. It takes input as the coarse-graine coarse-grained features containing global overlap information from coarse matching module and the fine-grained features from the current decoder layer, aiming to integrate features across granularity and predict fine-grained overlap between patches. consists of two components: borderless EdgeConv for aggregating local features, and cross-altention 6 36) for interacting cross-patch information.
Though EdgeConv [7] effectively captures local information by constructing wisnt-wise k-NN graphs within the patch, it restricts the search within the patch. pushes the vicinity towards the interior of the patch when encountering boundary points, affecting the smoothness of local descriptions. Therefore, we have lifted this restriction, allowing the k-NN search to include points outside the patch, and we refer to it as borderless EdgeConv. Furthermore, strengthen each point

---

feature as fx Co(cat [ fx) for integrating overlap features into patches, Here, fg is the feature from the current decoder layer and f is the nearest upsampled feature from the last transformer block. &o nonlinear layer activated by Softplus and each decoder layer's point features are mapped to the same dimension, allowing for the sharing parameters of subsequent modules across scales. Then the edge feature eij between point pair x, and x; with features f and fg in patch fx (). Here, he denotes nonlinear layer activated by LeakyReL.U, & denotes the k-th layer of EdgeConv. Finally, the point-wise feature is calculated as fx = he (uaat|ff (0) fx ,xx" 1), and fx (8), is max-pooled point feature from k-th layer. (1)
can be calculated ii ha(cat [cat[[x - (22(k11)
(k)
Feature-based cross-attention is then used to achieve information interaction between two local patches corresponding to a pair of node correspondences. Given the features Fx and Fy after EdgeConv of two patches X and Y, the output feature of Fx after cross-attention is:
f E |X|
Or (F/W),
Fx, Fy,
(3)
where a4j Softmax W (K/WK) /vde) is the attention score between Fx and Fy.
The co-contextual feature of Fy can be calculated in the same way. Now the output features FX and Fx contain sufficient information to achieve overlap prediction. We utilize the 0-1 scaled cosine similarity between points of two patches the overlap score:
Fx FY F Fy
1)/2.
I
3.2.3 Overlap-ware Sinkhorn Distance
Overlap Points Filtering Conducting a direct search for optimal transport across all points within patches may precipitate erroneous matches, consequently inducing inaccuracies Sinkhorn distance computation. To mitigate this issue, we advocate limiting the computation of the Sinkhorn distance to the most probable overlapping points. The rationale is that the selectively filtered overlapping points exhibit a higher likelihood of successful matching, thereby mitigating the susceptibility to distance measurement errors. Therefore, we only retain points with high overlap scores for further Sinkhorn iteration to improve the robustness of optimal transport solver
Specifically, adopt dynamic A 38 39 strategy select the points with top overlap score adaptively. Intuitively, the number of potential overlap points should vary across the patches due to factors like geometry, overlap area, etc. Patch with higher overlap scores should retain more overlapping points. Therefore, we select the top-g predicted scores and sum them up to represent the roughly estimated overlap points number, i.e., k = max Ei= s. 1}. Then points with top-k overlap scores are retained as overlap points. Rows and columns devoid of overlap points are discarded, and only the remaining entries are involved in the subsequent Sinkhorn operation. This strategy effectively enhances the robustness while reducing computational overhead.
009,, S₁
S 528
53=0.8, 54 0.7.
k
10.7 7 + 0.8 0.8 0.9]
$1
=3
Dynamic top-k
$2
$3
S4
Filtered Overlap Score S'
Overlap Score S
Figure 4: Overlap Points Filtering with Dynamic Top-k Illustration
Overlap-aware Marginal Prior The initialization of the Sinkhorn algorithm typically sets the marginal distributions, μ and as uniform distributions. This initialization strategy presupposes equal importance among points to be allocated. However, it is suboptimal and potentially unreasonable in most cases. The importance of individual points ideally varies according factors such as their positions, features, etc. With the availability of estimated overlap scores between two patches, it is prudent to incorporate this as prior during initialization. With the help of POP for estimating the overlap score, a more informed initialization of the Sinkhorn algorithm can be achieved.
Specifically, with the overlap score denoted s. we discard the rows and columns where without top-k scores to yield the filtered overlap score s'. Then we apply row- and column- to s' to adjust the marginal vectors, weighting them according to the overlap scores:
Poo E/e s
و'g
Vow
Liml
Lij=l s
(5)
5

---

(!) Overlap-aware Sinkhorn Distance Given pair of patches X and Y with their overlap score the filtered overlap points can be computed as Xov =
{{ {x top-k (i,j) (s) and Yor is computed in the same way. Similar to [4)] we then calculate the cost matrix C = - Fx... (F T/ /vd based on their features. Following the dustbin setting as in [40
41 to handle unmatched pairs, we augment the cost matrix C E R xxoovx/\Yo. to C E ( + + Y ov I+1) by appending a new row and column with learnable parameter 2. To incorporate overlap-aware marginal prior, we append the sum of one marginal to another to serve as the dustbin. Specifically, the unnormalized Poow can be updated
as Mov == Mav E Vou and similarly for Voe. Then normalization is applied in Eq. 5 to ensure that the sum equals 1. The formulation of the Overlap-aware Sinkhorn Distance is as follows:
oY |+1 (Hons t=1
Xov min
Dov
Σ
City Tij j=1
Σ
(6)
s.t. 71 = Moo, T | = Vou:
where T the i, j-h element from the assignment matrix T E R(lno]|+1)x((es of optimal transport problem. The above problem can be solved efficiently via the Sinkhorn Algorithm [42 30
on GPUs. Moreover, it is differentiable [40], enabling the prop-propagttion of the supervision signal from the transport result to the Patch Overlap Prediction module.
3.2.4 Hierarchical Sinkhorn Tree for Multi-scale Consistency Modeling
Though the above proposed overlap SD effectively models the similarity between patches, it solely explores the neighborhood features at single scale while still neglecting multi-scale information. To gather features at various scales for characterizing informative MSC, we propose the Hierarchical Sinkhorn Tree to traverse the feature hierarchy by repeating the above exploring and measuring steps.
Specifically, for modeling single-scale consistency, we conduct local exploration of correspondence the next level of the feature pyramid to form local patches and then measure their similarity using overlap SD. Notably, besides the measurement computation, overlap SD also matches potential overlapping points across patches. Moreover, due the presence of dustbin strategy, these points are further filtered. If the assignment results T generated by Sinkhorn algorithm can help select more accurate overlap points for finer- greingr local exploration, enables a more precise characterization of MSC. Therefore, we retain the most likely overlapping matches via mutual top-k selection (dropping dustbin). These matches continue exploring the feature at the next level with overlap SD. Repeating these two steps achieves the complete modeling of MSC across all scales. The whole modeling process essentially involves a Breath First Search (BFS) traversal with pruning of a K-ary tree rooted at the superpoint, with its k-NN in the feature pyramid serving as child nodes. HST is the subtrees pruned by overlap SD. Modeling MSC is essentially synchronously traversing pair of HSTss and aggregating the overlap SD of corresponding tree nodes. We take the mean of overlap SD for each layer and then perform a weighted sum across scales to obtain the final MSC. Following (6 [4], more robust probabilistic selection strategy is then adopted to form output correspondence set. The probability for all putative matches is p = Softmax(1 (1/ -m)), where m is the 0-1 normalized MSCs and is the temperature parameter that controls the soft assignment.
3.3
Loss Functions
The total loss L is the sum of each layer's loss. And each layer-wise loss L()) is composed of the overlap loss (0)) the overlap-aware matching loss (()) and the Overlap-aware circle loss c!!) i.e.,
(()) (1) + (1) + Loc.
Overlap loss
To supervise the overlap prediction, we minimize the cross entropy loss between the
predicted overlap score s and the ground truth 5 with radius T.. Here [.] is the Iversion bracket.
s
C.
Biy log (sig) + (1 (e,)log (1 Bt)), Bi = [x)((x)) -sll22 <To
(7)

---

Table 1: Results on the 3DMatch and 3DLoMatch datasets.
3DMatch
RR (%) f
FMR (%)
IR
(%)
t
Samples
5000
2500
1000
500
250
5000
2500
1000
500
250
5000
2500
1000
500
250
FCGF 32
85.1
84.7
83.3
81.6
71.4
97.4
97.3
97.0
96.7
96.6
56.8
54.1
48.7
42.5
34.1
D3Feat
81.6
84.5
83.4
82.4
77.9
95.6
95.4
94.5
94.1
93.1
39.0
38.8
40.4
41.5
41.8
SpinNet
86.6
85.5
83.5
70.2
97.6
97.2
96.8
95.5
94.3
47.5
44.7
39.4
33.9
27.6
Predator 6
89.0
89.9
90.6
88.5
86.6
96.6
96.6
96.5
96.3
96.5
58.0
58.4
57.1
54.1
49.3
CoFiNet 12
89.3
88.9
88.4
87.4
87.0
98.1
98.3
98.1
98.2
98.3
49.8
51.2
51.9
52.2
52.2
YOHO 43
90.8
90.3
89.1
88.6
84.5
98.2
97.6
97.5
97.7
96.0
64.4
60.7
55.7
46.4
41.2
GeoTR 13
92.0
91.8
91.8
91.4
91.2
97.9
97.9
97.9
97.9
97.6
71.9
75.2
76.0
82.2
85.1
RIGA
89.3
88.4
89.1
87.7 89.0
97.9
97.8
97.7
97.7
97.6
68.4
69.7
70.6
70.9
71.0
REGTR 45
92.0
OIF [4
92.4
91.9
91.8
92.1
91.2
98.1
98.1
97.9
98.4
98.4
62.3
65.2
66.8
67.1
67.5
RolTr 46
91.9
91.7
91.8
91.4
91.0
98.0
98.0
97.9
98.0
97.9
82.6
82.8
83.0
83.0
83.0
HST (Ours)
92.9
92.1
98.8
98.5
98.6
98.5
75.9
80.5
83.3
83.6
3DLoMatch
RR (%)
FMR (%)
IR (%)
Samples
5000
2500
250
5000
2500
1000
500
250
5000
2500
1000
500
250
FCGF 32
40.1
41.7
38.2
35.4
26.8
76.6
75.4
74.2
71.7
67.3
21.4
20.0
17.2
14.8
11.6
D3Feat
42.7
46.9
43.8
39.1
67.3
66.7
67.0
66.7
66.5
13.2
13.1
14.0
14.6
15.0
SpinNet
5
59.8
54.9
48.3
39.8
26.8
75.3
74.9
72.5
70.0
63.6
20.5
19.0
16.3
13.8
11.1
Predator
6
59.8
61.2
62.4
60.8
58.1
78.6
77.4
76.3
75.7
75.3
26.7
28.1
28.3
27.5
25.8
CoFiNet 12
67.5
66.2
64.2
63.1
61.0
83.1
83.5
83.3
83.1
82.6
24.4
25.9
26.7
26.8
26.9
YOHO 43
65.5
63.2
48.0
79.4
76.3
73.8
25.9
23.3
22.6
18.2
15.0
GeoTR 13
75.0
74.8
74.2
74.1
73.5
88.3
88.6
88.8
88.6
88.3
43.5
45.3
46.2
52.9
57.7
RIGA 44
65.1
64.7
64.5
64.1
64.8
85.1
85.0
85.1
85.1 84.3
32.1
33.4
34.3
34.5
34.6
REGTR 45
64.8
OIF 14
76.1
75.4
75.1
74.4
73.6
84.6
85.2
85.5
86.6
87.0
27.5
30.0
31.2
32.6
33.1
RolTr 46
74.7
74.8
74.8
74.2
73.6
89.6
89.6
89.5
89.4
89.3
54.3
54.6
55.1
55.2
55.3
HST (Ours)
77.8
77.4
76.9
75.5
74.0
88.8
88.8
89.0
88.8
89.0
41.7
48.0
54.5
56.6
58.3
To supervise the optimal transport result of Overlap-aware Sinkhorn Distance, we minimize the negative log- likelihoeooo loss as in (40) on the assignment matrix T. Given set OV representing the overlap point pairs within the overlap radius Too as the ground truth matching, I and J denoting the unmatched points, Lom is defined as:
Overlap matching loss
Com
Σ (n.))EVV
log Tr.
Σ log Tx,m+ - Σ log Tu+1.y TEI
(8)
VEJ
Overlap-aware circle loss Inspired by (34), extend the coarse matching loss 13 to the feature
hierarchy to further supervise the overlap prediction at each layer, i.c., Loc (Cx + Lx) /2, and
L
log
Σ
Map
-
Σ
(An
A
EYEE
GY
We provide details on the individual terms and further particulars in the supplementary material.
Experiments
We evaluate our proposed Hierarchical Sinkhorn Tree on both indoor 3DMatch M. 3DLoMatch (6] dataset, and outdoor KITTI odometry [47] dataset. More details about the datasets, evaluation metrics, and implementation are provided in the supplementary material.
4.1 Indoor 3DMatch & 3DLoMatch
Metrics. We adopt metrics evaluate our method. Our main metric the Registration Recall
(RR), the fraction of correctly aligned point cloud pairs. Following the settings in 6 13]. the
7

---

Table 2: Registration results w/ and w/o RANSAC on 3DMatch and 3DLoMatch. The number of samples for RANSAC-50K, weighted SVD, and LGR are 5000, 250, and all respectively. The units for metrics are RR (%), RRE ()), RTE (m), and time (s).
Model
Estimator
3DMtth
3DLoMatch
Time(s)
RR
RRE
RTE
RR
RRE
RTE
Predator 6
RANSAC-50K
89.0
2.02
0.064
62.5
3.04
0.093
3.915
CoFiNet 12
RANSAC-50K
89.3
2.44
0.067
67.5
5.44
0.155
1.746
GeoTrans 13
RANSAC-50K
92.0
1.87
0.065
75.0
2.94
0.090
1.992
OIF-PCR 14
RANSAC-50K
92.4
1.86
0.064
76.1
3.04
0.092
RoReg 48
RANSAC-50K
92.9
1.84
0.063
70.3
3,09
0.093
RolTr 46
RANSAC-50K
92.4
1.87
0.063
75.7
2.84
0.091
2.270
HST (Ours)
RANSAC-50K
93.5
1.83
0.063
77.8
2.80
0.088
2.160
Predator 66
weighted SVD
50.0
3.89
0.122
6.4
>10
>1
0.073
CoFiNet 22
weighted SVD
64.6
2.92
0.087
21.6
6.34
0.154
0.264
Geo
13
weighted SVD
86.5
2.13
0.070
59.9
3.88
0.105
0.245
HST (Ours)
weighted SVD
88.1
2.10
0.068
62.1
3.78
0.105
0.279
CoFiNet 12
LGR
87.6
2.23
0.071
64.8
3.49
0.124
0.279
GeoTrans [1
LGR
91.5
1.91
0.068
74.0
2.95
0.090
0.260
Roltr [46]
LGR
91.5
1.80
0.065
73.7
2.88
0.090
0.311
HST (Ours)
LGR
93.2
1.70
0.059
77.3
2.71
0.084
0.296
PEAL-3d 15
Iterative LGR
94.1
1.75
0.061
78.8
2.80
0.087
HST (Ours)
Iterative LGR
94.4
1.72
0.061
79.9
2.72
0.084
registration considered correct the root mean square error (RMSE) is under 0.2m. We also report the feature matching recall (FMR) and inlier ratio (IR). IR is defined the fraction of point pairs' distance less than 0.1m under correct transformation and FMR is the fraction of point pairs' IR larger than 5%. Relative Rotation Error (RRE) and Relative Translation Error (RTE) the metrics to measure the difference between the predicted transformation and the ground-trun transformation.
Comparisons to the state-of-the-art methods. We compare our proposed HST recent state-of- the-art methods including: FCGF (32), D3Feat
[4. SpinNet [3]. Predator [6], CoFiNet 112, YOHO
43], Geo Transformer (GeoTR) 13 RIGA 44 REGTR 45 OIF-PCR (OIF) 14 and RolTr [46),
see in Tab. | We first report the RR, FMR, and IR results using the RANSAC 17 estimator under different numbers of sampled correspondences. Following 66 13 14],
RANSAC for 50k iterations to estimate the final transformation. Following [14], we only report individual results of REGTR 445 not distinguishing the number of sampling points because it estimates the final pose based on all the superpoints instead of sampling. HST outperforms all the previous rescriptor-based and end-to-end methods on both 3DMatch and 3DLoMatch. It surpasses the closest competitor by 1.1 pp and 1.7 pp respectively on RR, reflecting the superiority in actual high and low overlap alignment. When the correspondence number varies, HST consistently maintains a significant lead over the others on the boards. For FMR and IR, HST still shows outstanding performance. It reaches the best or second best most data points, indicating its stability when confronting limited correspondences.
We further compare the registration results replacing RANSAC with other estimators including weighted SVD 49). local-to-global registration (LGR) [K]. and Iterative LGR [15 in Tab. 2 In addition to RR, we further introduce RRE (*), RTE (m), and time (s) to evaluate the estimated error and latency of the methods. First, when replacing with weighted SVD, all methods suffer severe performance degradation and some even fail to align while HST still achieved the best performance across all the metrics. When using LGR, HST consistently outperforms all the state-of-the-an
methods by a large margin. It improves the previous best (13) by 1.7% on 3DMatch I and 3.6% on 3DLoMatch [6] while with the smallest RRE and RTE. The most recent advances [15] refine the registration in an iterative update way. We adapt HST to iterative registration built upon PEAL [I5] with 3d overlap prior and utilize LGR multiple times to gradually refine the final transformation. The results demonstrate that HST achieving competitive performance although we have not optimized for the sulti-step pipeline, which once again proves the effectiveness of our method. The above results demonstrate that our method performs well even without RANSAC to filter out outliers in

---

Table 3: Comparisons to the baseline w/o and w/ fine-tuning on 3DExMatch using different estimators.
Method
Estimator
samples
w/o tuning
w/ tuning RRE( RTE(cm)
RR(%)
RRE(") RTE(cm)
RR(%)
GeoTrans HST (Ours)
LGR LGR
29.8
3.83
0.110
53.8
3.73
0.107
32.5
3.78
0.111
59.0
3.79
0.106
GeoTrans HST (Ours)
RANSAC
250
28.8
4.56
0.123
47.0
4.39
0.121
RANSAC
250
31.5
4.42
0.122
49.8
4.43
0.119
GeoTrans
RANSAC
500
30.5
4.25
0.121
49.9
4.55
0.113
HST (Ours)
RANSAC
500
32.5
4.16
0.118
52.6
4.03
0.117
GeoTrans HST (Ours)
RANSAC
1000
31.7
4.23
0.118
52.4
4.03
0.117
RANSAC
1000
33.6
4.03
0.114
57.5
4.13
0.115
GeoTrans HST (Ours)
RANSAC
2500
31.9
4.09
0.116
55.1
3.89
0.116
RANSAC
2500
34.7
3.81
0.111
59.0
3.89
0.112
GeoTrans
RANSAC
5000
31.0
4.04
0.117
56.1
4.07
0.116
HST (Ours)
RANSAC
5000
34.4
3.87
0.114
60.3
3.93
0.111
fine-level correspondences. indicates that HST is capable of forming reliable and robust coarse matching sets, which effectively assist subsequent fine matching to achieve accurate registration.
Comparisons under extreme low overlap. We design of experiments targeting overlap ratios of less than 10% and compare the performance with Tranrrrnform [33 to test the robustness of HST when facing extremely low overlap. However, the currently available preprocessed datasets, 3DMatch [1] and 3DLoMatch [6] (> 30% and 10 30%, respectively), do not include samples with such extremely low overlap 10%). Therefore, we access the 3DMatch raw dataset [1] and collect new set of point cloud pairs with overlap ratio under 10%, which we refer to as 3DExtremeLoMMe (3DExMatch) dataset, containing total of 1,343 samples. We first 3DMatch directly on 3DEx.Match, and the results can be found in the left part of Tab.
the model pre-trained on
We then randomly divide the 3DExMatch into training, validation, and test sets with proportions of 60% (805 samples), 10% (134 samples), and 30% (404 samples), respectively. We fine-tune the pretrained Geo'Transformar [3] and HST both for 3 epochs the training set, and then evaluate them on the test set. The results can be found in right part of Tab. 3 Both results clearly demonstrate that HST maintains strong performance even under low overlap conditions, confirming the effectiveness of our method.
Comparisons to the outlier-rejectoon methods. We further compare HST with other state-of-the- art outlier-rejection methods to gain more insights into handling outliers. For fairness, we replace HST directly with GC-RANSAC [18]. MAC (22), and FastMAC [50]. Results both 3DMatch and 3DLoMatch can found the following Tab. 4 All three methods, along with HST, showed performance improvements compared the vanilla Geofrranffrmmrr, with HST demonstrating the most significant enhancement. However, on 3DLoMatch, the improvements from these three methods were less pronounced, and some even showed a potential negative impact. In contrast, HST maintained better robustness, highlighting its superior effectiveness over previous outlier rejection methods in scenarios with low overlap and high noise.
Table 4: Registration results compare with state-of-the-art outlier-rejection methods.
Benchmark
3DMatch RRE(")
3DLoMatch
RR(%)
RTE(cm)
RR (%)
RRE(")
RTE(cm)
vanilla 13
91.5
1.91
0.068
74.0
2.95
0.090
GC-R IC-RANSAC 18
92.1
1.78
0.068
73.4
2.96
0.088
MAC 22
92.2
1.99
0.067
74.4
2.85
0.086
FastMAC
500
91.9
1.73
0.062
74.2
2.86
0.087
HST (Ours)
93.2
1.70
0.059
77.3
2.71
0.084

---

Table 6: Ablation study for each component. Tested with RANSAC # Samples=5000.
Model
3DMatch FMR(%)
3DLoMatch FMR(%)
Time(s)
IR (6)
RR(%)
IR(%)
RR(%)
Full (# Depth=2, scales)
75.9
98.8
93.5
41.7
88.8
77.8
2.160
vanilla SD
66.0
98.1
90.4
37.9
86.4
73.3
w/o Overlap Filtering
70.5
98.1
91.6
39.7
86.5
75.1
w/o Overlap-aware Initial
71.1
98.6
92.9
39.3
88.4
77.0
w/o Patch Overlap Pred
69.8
98.2
92.0
39.4
86.8
75.6
HST # Depth=1
74.3
98.2
92.8
40.5
88.1
76.8
2.075
HST Depth=0
71.0
98.1
92.4
39.9
87.5
74.6
2.039
4.2
Outdoor KITTI Odometry
Metrics. Following 16 13,, 3 metrics are adopted to evaluate our methods: Relative Ro- tation Error (RRE), Relative Translation Error (RTE), and Registration Recall (RR).
Table 5: Registration results on KITTI odometry.
Method
|REE()) RTE(cm). RR(%)T
FCGF 32
0.30
9.5
96.6
D3Feat
0.30
7.2
99.8
Registration Results. We compare our
SpinNet S
0.47
9.9
99.1
method with recent state-of-the-art methods
Predator 66
0.28
6.8
99.8
in Tab. [ including FCGF (32), D3Feat
CoFiNet 12
0.41
8.2
99.8
4] SpinNet [5), Predator (6), CoFiNet [2).
GeoTR 33
0.24
6.8
99.8
GeoTR [13] OIF [14, and PEAL [15] Our
OIF 14
0.23
6.5
99.8
method outperforms all the baselines for all metrics. It indicates that HST is effective in
PEAL 15
0.23
6.8
99.8
HST (Ours)
0.23
6.3
99.8
handling both indoor and outdoor scenes.
4.3 Ablation Study
Importance of individual modules. Tab. 6 shows the results of the ablation studies of each component on 3DMatch and 3DLoMatch. We first replace our proposed overlap-aware Sinkhorn Distance with vanilla Sinkhorn Distance, i.e., removing overlap filtering and overlap-aware initializa- tion. Results indicate that all the metrics decrease significantly, proving that HST benefits from our designed scheme. Then we ablate these two components and the POP module individually. Removing either will cause severe performance degradation, indicating each component beneficial for patch difference measurement. Finally, we ablate the depth of HST, i.e., the number of scales used. The results show that performance degrades as the depth reduces, with optimal performance achieved when all scales are explored. This confirms that robust coarse matching relies accurate MSC.
Robustness study Fig. shows the results of adding zero-mean Gaussian noise with standard deviation of 0.01 3DLoMatch [6] dataset, and grad- ually increasing the proportion of noise points to test the robustness of the model. demonstrates that HST
RST
GRo ColfiNet degrodattonn
Registration Recall (R.R) (%)
maintains the most stable performance compared to
13 [2], indicating its superior noise resistance.
92
-4
Conclusion
-16
0%
3%
30%
15%
20%
30%
35%
40%
45%
50%
In the paper, we present a simple but effective Hierar- chical Sinkhorn Tree (HST) model the multiscale geometric consistency for robust point cloud registration. We hierarchically explore the neighbor- hoods of each correspondence in their feature pyramids, and devise novel overlap-awar Sinkhorn Distance to compute the vicinity similarity. Subsequently, the most likely overlapping points are retained to continue local exploration. The modeling process essentially involves a BFS traversal of k-ary tree rooted at the coarse-level point. Pruning is performed during traversal using the overlap- aware Sinkhorn distance to obtain subtrees, which is the so-called HST. Extensive experiments show HST consistently outperforms the -oee-----hherrr methods on both indoor and outdoor benchmarks.
Noāse Polnt Ratio
Figure 5: Details of robustness study.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by Shenzhen Key Laboratory of Ubiquitous Data Enabling (No. Z 10610015) and by Tsinghua Shenzhen International Graduate Pengrui Endowed Professorship Scheme of Shenzhen Pengrui Foundation.

---

References
[1] Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1802- 1811, 2017.
[2] Haowen Deng, Tolga Birdal, and Slobodan llie. Ppfnet: Global context aware local features for robust 3d point matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 195-205, 2018.
[3] Haowen Deng, Tolga Birdal, and Slobodan Ilie. Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors. In Proceedings of the European conference on computer vision (ECCV), pages 602-618, 2018
[4] Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, and Chiew-Lan Tai. D3feat: Joint learning of dense detection and description of 3d local features. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6359-6367, 2020.
[5] Sheng Ao, Qingyong Hu, Bo Yang, Andrew Markham, and Yulan Guo. Spinnet: Learning a general surface descriptor for 3d point cloud registration. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 11753-11762, 2021.
[6] Shengyu Huang, Zan Gojeic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler. Predator: Registration 3d point clouds with low overlap. In Proceedings of the IEEE/CVF Conference computer vision and pattern recognition, pages 4267-4276 2021.
[7] Zi Jian Yew and Gim Hee Lee. 3dfeat-net: Weakly supervised local 3d features for point cloud registration. In Proceedings of the European conference computer vision (ECCV), pages 607-623, 2018.
[8] Xinghui Li, Kai Han, Shuda Li, and Victor Prisacariu. Dual-resolution correspondence networks. Advances in Neural Information Processing Systems, 53:17346-13357, 2020.
[9] Qunjie Zhou, Torsten Sattler, and Laura Leal-Taixe. Patch2pix: Spipolar-guides pixel-level correspondences. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2021.
[10] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8922-8931, 2021.
[11] Chenjie Cao and Yanwei Fu. Improving transformer-based image matching by cascaded capturing spatially informative keypoints. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12129-12139, 2023.
[12] Hao Yu, Fu Li, Mahdi Saleh, Benjamin Busam, and Slobodan Ilic. Cofinet: Reliable coarse- to-fine correspondences for robust pointcloud registration. Advances Neural Information Processing Systems, 0122387222888-- 2021.
[13] Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11143-1115 2022.
[14] Fan Yang, Lin Guo, Zhi Chen, and Wenbing Tao. One-inlier is first: Towards efficient position encoding for point cloud registration. Advances in Neural Informaion Processing Systems, 15:6982-6995, 2022.
[15] Junie Yu, Luwei Ren, Yu Zhang, Wenhui Zhou, Lili Lin, and Guojun Dai. Peal: Prior-embedded explicit attention learning for low-overlap point cloud registration. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 17702-17711, 2023.
12

---

[16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[17] Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381-3) 1981.
[18] Daniel Barath and Jiří Matas. Graph-cut ransac. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6733-6741, 2018.
[19] Heng Yang, Jingnan Shi, and Luca Carlone. Teaser: Fast and certifiable point cloud registration. IEEE Transactions on Robotics, 37(2):11--333 2020.
[20] Xuyang Bai, Zixin Luo, Lei Zhou, Hongkai Chen, Lei Li, Zeyu Hu, Hongbo Fu, and Chiew-Lan Tai. Pointdsc: Robust point cloud registration using deep spatial consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15859, 2021.
[21] Zhi Chen, Kun Sun, Fan Yang, and Wenbing Tao. Sc2-per: A second order spatial compatibility for efficient and robust point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13221-13231, 2022.
[22] Xiyu Zhang, Jiaqi Yang, Shikun Zhang, and Yanning Zhang. 3d registration with maximal cliques. In Proceedings the IEEE/CVF Conference Computer Vision and Pattenn Recogni- sion, pages 17745-17754, 2023.
[23] Junha Lee, Seungwook Kim, Minsu Cho, and Jaesik Park. Deep hough voting for robust global registration. In Proceedings of the IEEE/CVF international conference computer vision, pages 15994-16003, 2021.
[24] Jiaqi Yang, Zhiqiang Huang, Siwen Quan, Zhaoshuai Qi, and Yanning Zhang. Sac-cot: Sample consensus by sampling compatibility triangles in graphs for 3-d point cloud registration. IEEE Trunsactions on Geoscience and Remote Sensing, 10:1- 2021.
[25] Qingshan Xu and Wenbing Tao. Multi-scale geometric consistency guided multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5492- 2019.
[26] Zhengfa Liang, Yulan Guo. Yiliu Feng, Wei Chen, Linbo Qiao, Li Zhou, Jianfeng Zhang, and Hengzhu Liu. Stereo matching using multi-level cost volume and multi-scale feature constancy. IEEE Iransactions on pattern analysis and machine intelligence, 43(1):300-315, 2019.
[27] Qingshan Xu, Weihang Kong, Wenbing Tao, and Marc Pollefeys. Multi-scale geometric consistency guided and planar prior assisted view stereo. JEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4945-4966 2022.
[28) Yihang Lou, Yan Bai, Shiqi Wang, and Ling- Yu Duan. Multi-scale context attention network for image retrieval. In Proceedings of the 26th ACM international conference on Multimedia, pages 1128-1136. 2018.
[29] Chia- Wang, Yu-Chee Tseng, Ting-Hui Chiang, and Yan-Ann Chen. Learning multi-scale representations with single-stream network for video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6165-6175, 2023.
[30] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[31] Zan Gojeic, Caifa Zhou, Jan D Wegner, and Andreas Wieser. The perfect match: 3d point cloud matching with smoothed densities. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
[32] Christopher Choy, Jaesik Park, and Vladlen Koltun. Fully convolutional geometric features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9958-8966. 2019.
13

---

[33] Hugues Thomas, Charles R Qi, Jean-Emmanue Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEF/CVE international conference on computer vision, pages 2019.
[34] Changqian Yu, Jingbo Wang. Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 325-341, 2018.
[35] Jiaxin Li, Ben M Chen, and Gim Hee Lee. So-net: Self-organizing network for point cloud analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9397-9406, 201 8.
[36] Yue Wang and Justin M Solomon. Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3523-3532, 2019..
[37] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5):1-12, 2019.
[38] Zheng Ge, Songtao Liu, Zeming Li, Osamu Yoshie, and Jian Sun. Ota: Optimal transport assignment for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 303-312, 2021.
[39] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv: 100.0-3000 2021.
[40] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9938-4947, 2020.
[41] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 224 236, 2018.
[42] Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343-33 348, 1967.
[43] Haiping Wang, Yuan Liu, Zhen Dong, and Wenping Wang. You only hypothesize once: Point cloud registration with rotation rotuiion- descriptors. In Proceedings of the 30th ACM International Conference on Multimedia, pages 1630- 2022.
[44] Hao Yu, Ji Hou, Zheng Qin, Mahdi Saleh, Ivan Shugurov, Kai Wang. Benjamin Busam, and Slobodan Ilie. Riga: Rotation-invariial and globally- descriptors for point cloud registration. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2024.
[45] Zi Jian Yew and Gim Hee Lee. Regtr: End-to-end point cloud correspondences with transformers. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6677-6686, 2022.
[46] Hao Yu, Zheng Qin, Ji Hou, Mahdi Salch, Dongsheng Li, Benjamin Busam, and Slobodan llie. Rotation-invarian transformer for point cloud matching. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 3384-5393, 2023.
[47] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3361. IEFF, 2012.
[48] Haiping Wang, Yuan Liu, Qingyong Hu, Bing Wang, Jianguo Chen, Zhen Dong, Yulan Guo, Wenping Wang, and Bisheng Yang. Roreg: Pairwise point cloud registration with oriented descriptors and local rotations. IEEE Transactions on Pattern Analysis and Machine Intelligence 2023.
14

---

[49] K Somani Arun, Thomas S Huang, and Steven D Blostein. Least-squares fitting of two 3-d point sets. IEEE Transactions on pattern analysis and machine intelligence, (5):698-700, 1987.
[50] Yifei Zhang, Hao Zhao, Hongyang Li, and Siheng Chen. Fastmac: Stochastic spectral sampling of correspondence graph. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pages 17857-17867, 2024.
15

---

A Appendix supplemental material
In this supplemental material, we first provide detailed introductions about our utilized datasets in Sec. A.T Metries utilized for evaluation in our experiments are demonstrated in Sec. A.2 Detailed loss and further particulars are provided in Sec.
AA.]] We further provide more implementation details
in Sec. A.4 More results are provided in Sec. A.5 Limitations and broader impact are discussed in
Sec. AA6 and Sec. A.7 Finally, we demonstrate more qualitative results registration on 3DMatch,
3DLoMatch in Sec. A.8
A.1 Datasets
3DMatch 3DLoMatch 3DMatch [ contains 62 scenes collected from SUN3D (2], 7-Scenes
3. RGBD Scenes v.2 [4. Seallyis-by-SSyttheee [5] BundleFusion [6] and Halbel et al. 7 among which 46 scenes are used for training, 8 scenes for validation and 8 scenes for testing. Scenes are captured from various indoor environments with sensors like Microsoft Kinect. The licenses for each component are listed as follows Tab. 7 Following 9 10,, we use training data preprocessed by [8 and evaluate on 3DMatch and 3DLoMatch benchmarks. The original 3DMatch only contains point clouds with overlap > 30%. 3DLoMatch additionally includes a subset of point cloud pairs with an overlap ratio between 10% and 30%, specifically designed to evaluate performance in low overlap scenarios.
Table 7: Datasets in 3DMatch
and their licenses.
Datasets
License
SUN3D [24]
CC BY-NC-SA 4.0
7-Scenes [17]
Non-commercial use only
RGB-D Scenes v.2 [11]
(License not stated)
Saelysis-by-Syntheest [21]
CC BY-NC-SA 4.0
BundleFusion [5]
CC BY-NC-SA 4.0
Halbel et al. [8]
CC BY-NC-SA 4.0
KITTI Odometry KITTI is a vision benchmark especially focusing on outdoor scenes. consists of 11 sequences of outdoor driving scenarios scanned by Velodyne HDL-64 3D laser scanner. We follow 12 13 g and adopt 0-5 for training, 6-7 for validation and 8-10 for testing. The ground truth transformations
refined utilizing ICP following [12] 13 & due to the noisy GPS data. we only use point cloud pairs that are at least 10m away for evaluation, i.e., 1358 pairs for training, 180 for validation, and 555 for testing following [12. is published under the Haaamerrial-ShaaAAlliii 3.0 License.
A.2 Evaluation Metrics
Relative Rotation Error Relative Rotation Error (RRE) the geodesic distance in degrees between estimated and ground-truth rotation matrices. is calculated by measuring the difference between the estimated and true rotation vectors as follows:
trace (R. R -1
RRE = arccos
(10)
2
Relative Translation Error Relative Translation Error (RTE) is the Euclidean distance between estimated and ground truth translation vectors. is calculated by measuring the difference between the estimated and true translation matrices as follows:
RTE || 2|12-
(11)
Registration Recall Registration Recall (RR) is the fraction of correctly registered point cloud pairs. In the 3DMatch and 3DLoMatch benchmarks, two fragments considered registered correctly if
16

---

the transformation Root Mean Squared Error (RMSE) is smaller than 0.2m:
RMSE
Σ (pi, JEC
|T (p%.) - Th l1
(12)
RR
TRMSE, 0.2.m]
(13)
where T is the predicted transformation and C are the ground truth correspondences. [-] is the Iversion bracket. Moreover, we align with the evaluation protocol in 3DMatch [. the immediately adjacent point clouds excluded due their high overlap ratio.
In the KITTI Odometry benchmark, the registration is correct when RRE 50 and RTE < 2m:
M Σ 2=1
RR
IRRE, < 5" A RTE, < 2m].
(14)
M
Inlier Ratio
Inlier Ratio (IR) the fraction of inlier correspondences among all the putative correspondences. A correspondence is considered an inlier when the distance between two points is smaller than = 10 em under the ground-truth transformation T:
IR
Σ
(P:x;)
quill ll <n>,
(15)
C
(prg.aa,) JEC
where C is the putative correspondence set.
Feature Matching Recall Feature Matching Recall (FMR) is the fraction of point cloud pairs whose IR is larger than certain threshold 0.05. It measures the likelihood that the optimal transformation between point clouds can be recovered using robust estimators. is calculated as:
M
1 M
FMR
Σ
IR,
ml
(16)
A.3 Detailed Loss
Overlap-aware circle loss Each layer-wise overlap-aware circle loss [10] is calculated as the mean of losses point cloud pairs X and Y from the I-th layer of feature pyramid, i.c.. Loo (!) (CX (I) + CY (1) /2. The anchor patch set A is collected as the patches in X have at least one positive patch in Y. The positive patch set ef is defined as the patches that share at least 10% overlap. The negative patch sete the patches share overlap. Then the individual loss for X is defined as:
[
L
1
Σ
log[1 +
Σ GE
Σ
Biik (bn (An-))))
(17)
L
0 Eet
where de = | hiy 1|2 is the distance in the feature space, X = (o²) and of are the overlap
ratio between G.X and G. The positive and negative weights are computed as 383 = y (d) 4,) and gi.k =7 [A, d=). The hyper- parametrr for margin are set to Ap 0.1 and A. = 1.4
following [10 The loss LY (1) for Y are calculated in the same way.
Inspired by 14 introducing segmentation loss to multiple feature scales to enhance the feature representation, extend the coarse (superpoint) matching loss to the feature hierarchy to further
17

---

supervise the overlap prediction at each layer. The overall overlap-ware circle loss is calculated as Loc Ei= wi "), where wy is the weight for the (-th layer. For stable training and better performance, we decay the weight by half with each successive layer, i.e., Loc Ei Loc !
A.4 Implementation Details
Our proposed method is implemented and evaluated in Pytorch 13 and we train it on a single RTX 3090 GPU with an AMD EPYC 9654 CPU. Specifically, the network is trained with Adam optimizer [16] for 40 epochs on 3DMatch and 80 epochs on KITTI with batch size
and weight decay of 10-6, The learning rate initializes from 10-1 and decays exponentially by 0.05 every epoch on 3DMatch and every 4 epochs on KITTI, respectively. The matching radius 7 is set as 5cm for indoor 3DMatch/3DLLa Lotthh and 60cm for outdoor KITTI generate overlapping matches. We randomly sample 128 pairs of ground-truth coarse matches during training. The overall Multi- scale Consistency (MSC) is calculated as the weighted sum of each layer's mean overlap-aware Sinkhorn distance (overlap SD) ( and the weights also decay by half as the layer increases, i.e., MSC = Σ'= (1)s The k that controls the vicinity size for k-NN exploration is set as 16. We run the Sinkhorn Algorithm [17] for 100 iterations to calculate the overlap-awald aware Sinkhorn distance and solve the optimal transport for fine matching.
Coarse Matching The coarse matching is achieved by calculating the Gaussian correlation matrix [10 as the matching score S. Given the features Fx and Fyy from the last Geometric Transformer block 10]. the matching score calculated as:
Sij = exp(p F 12,1)))
(18)
Then dual-softmax normalization operator [18 19 10 is utilized on reduce ambiguous matches while converting it to the probability P of mutual matching [1)]:
Pij = softmax(S(i, )) softmax(S(--1)))
(19)
Finally, we select the top-k largest entries as the coarse correspondence set:
Č (xx,,,,,)) (i,5) top-k.,(Piss))
(20)
Fine Matching At fine level matching, each correspondence refined to point level for establishing finer correspondences. For each coarse correspondence x; and yj. we adopt the point-to-node grouping 10) to extract the local patches X, and Y, from the dense point layer. We then compute the feature similarity matrix s with their corresponding features Fx, and Fy, of two patches:
S =
Fx, F (F xxx, //Vd
(21)
where d the feature dimension.
Then the feature similarity matrix is augmented by learnable row and column entry that serves the dustbin as in 20
10. The fine matching is considered an optimal transport problem solving by the (Sinkhorn
Sinkhorn Algorithm 21). We then apply 100 iterations of row- and column-normalizatio Algorithm) to compute the final assigned matching score S by discarding the dustbin entries. The output point-level correspondences from patches X; and Y, are then obtained by selecting the mutual top-k entries of the score S:
C= [[x vj) (i.f) e mutual top-k, (5
(22)
A.5 Additional Experimental Results
A.5.1
Scene-wise Results
We present the scene-wise Registration Recall (RR) results in Tab. 8 and Tab. The results on 3DMatch []] demonstrate method outperforms baselines in most of the scenes. On the hard case
18

---

like Home 2, our method improves the previous best by 3.2 pp. indicating its better registration accuracy. While on 3DLoMatch [8], our method surpasses the strongest baseline [TO] almost all scenarios and has achieved nearly all of the best performances. This indicates that our HST shows even greater performance improvement low-overlap scenarios compared high-overlap scenarios, proving its superior robustness in handling severe conditions.
Table 8: Scenc-wise registration results on 3DMatch.
Model
3DMatch Home_2 Hotel_ Hotel_2 Hotel_3
Kitchen Home_
Study
Lab
Mean
Registration Recall (%) f
3DSN 22
90.6
90.6
65.4
89.6
82.
80.8
68.4
60.0
78.4
FCGF 13
98.0
94.3
68.6
96.7
91.0
84.6
76.1
71.1
85.1
D3Feat 12
96.0
86.8
67.3
90.7
88.5
80.8
78.2
64.4
81.6
Predator
8
97.6
97.2
74.8
98.9
96.2
88.5
85.9
73.3
89.0
CoFiNet
96.4
99.1
73.6
95.6
91.0
84.6
89.7
84.4
89,3
GeoTrans
10
96.9
97.7
81.1
98.0
89.7
88.5
88.9
88.9
91.5
HST (Ours)
98.2
98.1
84.3
97.3
94.9
92.3
91.5
88.9
93.2
Table 9: Scene-wise registration results 3DLoMatch.
Model
3DLoMatch Kitchen Home_1 Home_2 Hotel_ Hotel_ Hotel_3
Study
Lab
Mean
Registration Recall (%) f
3DSN 22
51.4
25.9
44.1
41.1
30.7
36.6
14.0
20.3
33.0
FCGF 13
60.8
42.2
53.6
53.1
38.0
26.8
16.1
30.4
40.1
D3Feat 12
49.7
37.2
47.3
47.8
36.5
31.7
15.7
31.9
37.2
Predator
8
71.5
58.2
60.8
77.5
64.2
61.0
45.8
39.1
59.8
CoFiNet
99
76.7
66.7
64.0
81.3
65.0
63.4
53.4
49.6
67.5
GeoTraas 10
85.9
73.5
72.5
89.5
73.2
73.2
66.7
75.7
74.0
HST (Ours)
89.1
76.7
74.8
91.0
75.4
71.4
64.1
75.7
77.3
A.5.2
Additional Ablation Studies
In this subsection, we provide more ablation studies the type of Local Exploration, and the type of Patch Feature Transformation. Here, we provide only the Registration Recall results for comparison, primary focus is the actual effectiveness of point cloud registration. In many cases, the distribution, location, and number of correspondences can greatly affect the Inlier Ratio (IR) and Feature Matching Recall (FMR) metries. It common for the IR and FMR to decrease significantly even with better registration results (higher RR), which can impact the ablative analysis of each component. The similar observations have also been made by { 9 and (23
Table 10: Registration results of additional ablation studies.
Ablation Part
3DMatch RR(%)
3DLoMatch RR(%)
P2N grouping 10
93.2
77.2
k-NN grouping
93.5
77.8
EdgeConv
93.5
77.5
Self-attention
25
93.3
77.2
Borderless EdgeCony
93.5
77.8
The type Local Exploration. We evaluate two different types of local exploration for extracting local patches: the Point-to-node (P2N in the table) grouping [26] 9], and the k-NN grouping
19

---

use. Point-to-node grouping constructs local patches by associating each point with its nearest node (superpoint). can be easily extended hierarchical local exploration by replacing nodes with lower spatial resolution point clouds from adjacent layers. The results in Tab. TO show that the Point-to-node grouping strategy has a slightly lower performance than k-NN grouping. In fact, there is not much difference in performance between the two grouping methods. Our experience believe that the two patches extracted by k- NN from same source point cloud pair may have closer distributions, which beneficial for subsequent overlap prediction. Therefore, we choose k- NN for local exploration.
The type of Patch Feature Transformation. We then provide an ablation study on the patch feature transformation method. We compare the results between the vanilla EdgeCony [24], self-attention [25], and our expanded version EdgeConv (borderless EdgeConv) in Tab. [0] Registration results show EdgeConv-basee methods perform better than self-attention. Due to the introduction of points outside the patch to smooth the feature description and the construction of k-NN graph, the borderless one has slight performance improvement compared to vanilla EdgeConv 3DLoMatch.
A.6 Limitations.
The limitations of our proposed HST are 2-fold: 1) HST still involves evaluating and selecting generated coarse correspondences. In some extreme low-overlap noisy cases, when the quality of putative correspondences searched by the coarse stage model is poor, can still negatively impact registration and lead to failure. A feasible solution is to subsequently integrate HST into the coarse correspondence generation process to produce more accurate match set directly. 2) Assessing patch similarity purely from geometric perspective is limited due noise, locality, etc. If more priors or features, such as color, normal maps, and semantic labels, could be introduced subsequently to characterize neighborhood features, it would enable more accurate coarse correspondence searching. We think it is a promising topic for further improving the robustness of registration.
A.7 Broader Impact.
We study the problem of retrieving accurate correspondence through multi-scale consistency (MSC) for robust point cloud registration and present novel method to model the MSC called HST. HST makes a first attempt towards injecting coarse correspondence filter into the coarse-to-fin pipeline, allowing for more accurate and robust point cloud registration. It contributes to various applications such autonomous driving and robotics. For example, Simultaneous Localization and Mapping (SLAM) tasks can benefit from our method by enabling more robust unified scene reconstruction. Also, our method can help more precise scene understanding in autonomous driving as it is capable of forming a reliable correspondence set for aligning point cloud scans at different timestamps. However potential negative impacts may occur it the fundamental of various computer vision tasks. For example, our method may fail in some severe environments like no overlapping area leading to wrong scene representations.
A.8 Qualitative Results
We provide qualitative results on 3DMatch, 3DLoMatch, and Outdoor KITTI Odometry in Fig. 6 The column (a) and (b) are the input source and target point clouds for registration. Column (c) shows the estimated transformation from our proposed HST while column (d) is the ground truth alignment.
20

---

References
[1] Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1802-
1811, 2017.
[2] Jianxiong Xiao, Andrew Owens, and Antonio Torralba. Sun3d: A database of big spaces reconstructed using sfm and object labels. In Proceedings of the IEFE international conference on computer vision, pages 1625-1632, 2013.
[3] Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2930-2937, 2013.
[4] Kevin Lai, Liefeng Bo, and Dieter Fox. Unsupervised feature learning for 3d scene labeling. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3050-3057. IEEE, 2014.
[5] Julien Valentin, Angela Dai, Matthias Nießner, Pushmeet Kohli, Philip Torr, Shahram Izadi, and Cem Keskin. Learning to navigate the energy landscape. 2016 Fourth International Conference on 3D Vision (3DV), pages 323-332. IEEE, 2016.
[6] Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reinte- gration. ACM Transactions on Graphics (ToG), 36(4):1, 2017.
[7] Maciej Halber and Thomas Funkhouser. Fine-to-coarse global registration of rgb-d scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1755-1764, 2017.
[8] Shengyu Huang, Zan Gojeic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler. Predator: Registration 3d point clouds with low overlap. In Proceedings of the IEEE/CVF Conference computer vision and pattern recognition, pages 4267-4276, 2021.
[9] Hao Yu, Fu Li, Mahdi Saleh, Benjamin Busam, and Slobodan Ilie. Cofinet: Reliable coarse- to-fine correspondences for robust pointeloud registration. Advances Neural Information Processing Systems, 34:23872- -238844 2021.
[10] Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11143-11152, 2022.
[11] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361. IEEE, 2012.
[12] Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, and Chiew-Lan Tai. D3feat: Joint learning of dense detection and description of 3d local features. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6359-6367, 2020.
[13] Christopher Choy, Jaesik Park, and Vladlen Koltun. Fully convolutional geometric features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9958-8966, 2019.
[14] Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 325-341, 2018.
[15] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances neural information processing systems, 32, 2019.
21

---

[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[17] Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. Advances in neural information processing systems, 26, 2013.
[18) Ignacio Rocco, Mírcea Cimpoi, Relja Arandjelović, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Neighbourhood consensus networks. Advances in neural information processing systems, 31, 2018.
[19] Jiaming Sun, Zehong Shen. Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8922-8931, 2021.
[20] Paul-Fdouard Sarlin, Daniel DeTone, Tomasz Malisiewiez, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938 4947, 2020.
[21] Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343-33 348, 1967.
[22] Zan Gojeic, Caifa Zhou, Jan D Wegner, and Andreas Wieser. The perfect match: 3d point cloud matching with smoothed densities. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5545-5554, 2019.
[23] Fan Yang, Lin Guo, Zhi Chen, and Wenbing Tao. One-inlier first: Towards efficient position encoding for point cloud registration. Advances in Neural Information Processing Systems, 35:6982-6999 2022.
[24] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions Graphics (tog), 38(5):1-12, 2019.
[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[26] Jiaxin Li and Gim Hee Lee. Usip: Unsupervised stable interest point detection from 3d point clouds. In Proceedings of the IEEE/CVF international conference on computer vision, pages 361-370, 2019.
22

---

3DMatch
3DMatch
3Dth
3DLoMatch
KITTI
KITTI
aa) Target (b) Source
(c) Estimation
(d) GT
Figure 6: Qualitative registration results on 3DMatch, 3DLoMatch, and KITTI Odometry.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see the Abstract and Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see the supplemental material Sec. A.6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please the Method section and the supplemental material Sec. A.4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open Access to Data and Code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Please see the supplemental material Sec.
A.4
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee 7/nips.com / casssiinnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please see the Method section and the supplemental material Sec. A.4 Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Given that the baseline methods compared do not report error bars, align our evaluation with theirs to ensure fair comparisons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see the supplemental material A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:////.uurr..ccpppaaa//aaaaaa
Answer: [Yes]
Justification:
We have read the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please the supplemental material Sec. A.7
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for Existing Assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use open-source indoor benchmarks 3DMatch and 3DLoMatch as well as the outdoor KITTI Odometry benchmark. Licenses please refer to the supplemental material Sec.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: It will be included in the camera-ready version.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "XZHTSJeJWB", "reviewer": "Reviewer_AXAW", "summary": "This paper introduces the Hierarchical Sinkhorn Tree (HST) for reliable correspondence identification in point cloud registration. The core idea is to hierarchically evaluate the local consistency of each correspondence at multiple feature scales using Sinkhorn distance, thereby filtering out the locally dissimilar correspondences. Specifically, Local Exploration is first employed to extract local patches at each correspondence’s next decoder layer. Then, the overlap-aware Sinkhorn distance is used to evaluate the patch differences, filtering out the non-overlapping local patches. The filtered overlapping points are further utilized for additional local exploration and overlap Sinkhorn distance measurements. Finally, the consistency measures across all scales are aggregated for robust transformation estimation. Extensive experiments on public benchmark datasets verify the effectiveness of the proposed method.", "questions": "See Weaknesses.", "limitations": "See Weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The authors’ introduction of the concept of multi-scale consistency (MSC) for robust registration is both interesting and promising. Developing a Hierarchical Sinkhorn Tree to model MSC is also a novel approach that achieves significant performance gains.\n\n(2) Overall, this manuscript is well-structured and clearly written. The authors have effectively organized their ideas, making the content easy to follow and understand.\n\n(3) The authors provide sufficient comparisons and ablation studies to verify the effectiveness of the proposed mechanism.", "weaknesses": "(1) Could you please explain Equation (2)? I understand that you aim to gather the k-nearest neighbor points from the next layer to form the local patch  X_i^{(l)} . However, I find it difficult to grasp this idea from Equation (2). For instance, what does  \\|x_i^l, x_j^{l+1}\\|  represent? Do you mean the distance between  x_i^l  and  x_j^{l+1} ? If so, how does applying the argmin operator on K yield the desired local patch? It’s confusing and not intuitive.\n\n(2) Some relevant outlier rejection-focused registration methods should be cited, such as:\n[1] Robust Outlier Rejection for 3D Registration with Variational Bayes. CVPR’2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first would like to thank the reviewer for providing us valuable comments and suggestions.\n\n**Response to Weakness 1**: Thank you for your question regarding Eq. (2) about the k-NN local exploration part. We apologize for the typo in Eq. (2) and for any misunderstandings it may have caused.\nYour are correct about the understanding of Eq. (2). Its purpose is to search for the k-NN of any given point $x_i^{l}$ from the $l$-th layer in the next layer to form the corresponding local patch. We provide the following explanations in hopes of resolving your confusion.  \n(1) The term $||x_i^l, x_j^{l+1} ||$ represents the Euclidean distance between the given $l$-th layer's point $x_i^l$ and the $(l+1)$-th layer's point $x_j^{l+1}$. So the operator $||\\cdot||$ we used here is actually the L2-norm of coordinates' difference. We will add a subscript 2 to the $||\\cdot||$ operator to avoid any potential misunderstandings, i.e., revise $|| x_i^l, x_j^{l+1} ||$ to $|| x_i^l, x_j^{l+1} ||_2$.  \n(2) We mistakenly used $argmax$ instead of $argmin$ operator in the formula. The set $K$ actually represents the collection of Euclidean distances between all points in $(l+1)$-th layer and the given point in $l$-th layer. We should use the $argmin$ operator on $K$ to search for the k nearest points to form the local patch.  \n(3) Our initial intention is to use a more mathematical $argmin$ operator to represent the k-NN search instead of directly using $kNN(\\cdot)$. However, such an expression might be difficult to understand and could lead to ambiguities, such as the definition of the set $S$ in the formula. Therefore, we take a compromise approach by replacing the $argmin$ operator with the $argtopk$ operator to make it more intuitive and easier to understand.  \nThe following is the revised formula and will be included in the revised version.  \n\nGiven $\\mathbf{x}_i^{(l)}$, its local patch $\\mathbf{P}_i^{(l+1)}$ is defined as:  \n$$\n\\mathbf{P}_i^{(l+1)}=\\operatorname{argtopk} _{\\mathbf{x}_j^{(l+1)}\\in\\mathbf{X}^{(l+1)}}(-||\\mathbf{x}_i^{(l)},\\mathbf{x}_j^{(l+1)}||_2).  \n$$\n  \n**Response to Weakness 2**: Thanks for your suggestion about citing relevant outlier rejection-focused registration methods. We will cite approaches like GC-RANSAC [1], MAC [2], VBReg [3], FastMAC [4], etc., and summarize them as fine-level outlier removal methods in our revised version.  \n[1] Barath, Daniel, and Jiří Matas. \"Graph-cut RANSAC.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.  \n[2] Zhang, Xiyu, et al. \"3D registration with maximal cliques.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.  \n[3] Jiang, Haobo, et al. \"Robust outlier rejection for 3d registration with variational bayes.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.  \n[4] Zhang, Yifei, et al. \"FastMAC: Stochastic Spectral Sampling of Correspondence Graph.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8DNP9Lpwqj", "reviewer": "Reviewer_Sd2B", "summary": "This paper studies the problem of correspondence retrieval for point cloud registration. To this end, this paper proposes the Hierarchical Sinkhorn Tree, which is a pruned tree structure designed to hierarchically measure the local consistency of each coarse correspondences. To validate the proposed methods, the authors conducted experiments on both indoor and outdoor benchmarks.", "questions": "See the weaknesses part.", "limitations": "Limitations have been discussed in the Appendix.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written, well-organized, and easy to follow;\n\n2. The idea of introducing tree structure into a coarse-to-fine mechanism is somehow interesting.", "weaknesses": "1. Although incorporating a tree structure into a coarse-to-fine mechanism is interesting, the idea sounds more like repeating the coarse-to-fine matching strategy used by CoFiNet and GeoTransformer. I suggest emphasizing this core contribution in the rebuttal phase;\n\n2. In the main comparison (Tab. 1), the proposed method does not significantly outperform existing methods, which limits the value of the model.\n\n3. As the proposed model leverages a tree structure to repeat the coarse-to-fine procedure, the efficiency (in terms of running speed) is a major concern. The authors should include related comparisons with state-of-the-art approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and feedback. We hope that our responses can address your concerns.  \n\n**Response to Weakness 1**: Thanks you for your valuable comments. First, we would like to emphasize that modeling multi-scale consistency (MSC) is non-trivial, even with the help of a coarse-to-fine strategy for modeling the neighborhood. We understand your concern that our method might simply be applying the coarse-to-fine procedure across multiple scales. But our proposed method is not simply a repetition of the coarse-to-fine strategy, as merely repeating such an operation would result in an $O(n^l)$ complexity, where $l$ is the number of scales. This level of complexity is unacceptable, posing a major challenge in practical applications. Additionally, this naive approach could introduce a large number of non-overlapping noisy points, negatively impacting the model's robustness and performance, and could even degrade the quality of the correspondence set compared to the original.  \nTo address these issues, we proposed a series of designs to improve the model's feasibility and performance, which distinguishes it from the mentioned models. First, we design a lightweight Patch Overlap Prediction module to efficiently predict overlap. Next, we propose Overlap-aware Sinkhorn Distance with Overlap Points Filtering and Overlap-aware Marginal Prior to effectively retain the most likely overlapping points based on the prediction, filtering out a large number of potential non-overlapping points. Finally, we perform the above methods with local exploration layer-wisely to prune the K-ary tree rooted at the superpoint, thereby constructing the HST to model the MSC of correspondences. Our extensive experiments on both indoor and outdoor benchmarks have demonstrated the scene-agnostic effectiveness of our proposed method.  \nFinally, we would like to emphasize our key contributions as follows: 1) To the best of our belief, our work is the first to introduce MSC into point cloud registration tasks to mitigate the effects of low overlap and high noise. 2) We propose a method for modeling MSC called HST, which introduces a pruned tree structure to efficiently characterize the similarity of potential overlapping points in the vicinity areas layer by layer and aggregates them into MSC. 3) We introduce an overlap-aware Sinkhorn Distance with Overlap Points Filtering and Overlap-aware Marginal Prior to focus optimal transport processes only on potential overlapping points, significantly enhancing the robustness of consistency calculations while reducing solution complexity.  \nThe above discussions will be included in the revised version.\n\n**Response to Weakness 2**: Thank you for your comments regarding the comparative performance. As mentioned by other reviewers and shown in Tab 1, HST gains significant improvements over the state-of-the-art methods on both indoor and outdoor benchmarks. The improvements are even greater on 3DLoMatch, where point cloud pairs share fewer overlaps, suggesting that our model demonstrates better robustness and performance under low-overlap, high-noise scenarios. To validate this, we further design a set of experiments targeting overlap ratios of less than 10% and compare the performance with GeoTransformer. Please note that the currently available preprocessed datasets, 3DMatch (overlap > 30%) and 3DLoMatch (10% < overlap < 30%), do not include samples with such extremely low overlap (overlap<10%). Therefore, we access the 3DMatch raw dataset and collect a set of point cloud pairs with overlap < 10%, which we refer to as 3DExtremeLoMatch (3DExMatch) dataset, containing a total of 1,343 samples. We first test the model pre-trained on 3DMatch directly on 3DExMatch, and the results can be found in the following table:  \n|Estimator|Method|RR(\\%)|RRE($^\\circ$)|RTE(m)|\n|---|---|:---:|:---:|:---:|\n|LGR|GeoTrans|29.8|3.83|0.110|\n||HST|32.5|3.78|0.111|\n|RANSAC-250|GeoTrans|28.8|4.56|0.123|\n||HST|31.5|4.42|0.122|\n|RANSAC-500|GeoTrans|30.5|4.25|0.121|\n||HST|32.5|4.16|0.118|\n|RANSAC-1000|GeoTrans|31.7|4.23|0.118|\n||HST|33.6|4.03|0.114|\n|RANSAC-2500|GeoTrans|31.9|4.09|0.116|\n||HST|34.7|3.81|0.111|\n|RANSAC-5000|GeoTrans|31.0|4.04|0.117|\n||HST|34.4|3.87|0.114|\n\nWe then randomly divide the 3DExMatch into training, validation, and test sets with proportions of 60% (805 samples), 10% (134 samples), and 30% (404 samples), respectively. We fine-tune GeoTransformer and HST both for 3 epochs on the training set, and then evaluate the models on the test set. The results can be found in the following table. Both results clearly demonstrate that HST can significantly outperform other methods even under low overlap conditions, further confirming the effectiveness of our method.  \n|Estimator|Method|RR(\\%)|RRE($^\\circ$)|RTE(m)|\n|---|---|:---:|:---:|:---:|\n|LGR|GeoTrans|53.8|3.73|0.107|\n||HST|59.0|3.79|0.106|\n|RANSAC-250|GeoTrans|47.0|4.39|0.121|\n||HST|49.8|4.43|0.119|\n|RANSAC-500|GeoTrans|49.9|4.55|0.113|\n||HST|52.6|4.03|0.117|\n|RANSAC-1000|GeoTrans|52.4|4.03|0.117|\n||HST|57.5|4.13|0.115|\n|RANSAC-2500|GeoTrans|55.1|3.89|0.116|\n||HST|59.0|3.89|0.112|\n|RANSAC-5000|GeoTrans|56.1|4.07|0.116|\n||HST|60.3|3.93|0.111|\n\n**Response to Weakness 3**: Thank you for your insightful feedback regarding the efficiency of our proposed method. We share your concern on this matter. In fact, we have compared the running time of our method with state-of-the-art approaches under different estimators in the manuscript. Please refer to the last column of Table 2. The results indicate that our method achieves significant performance improvements with only a slight, acceptable increase in latency due to the introduction of new modules. We will highlight this comparison more prominently in the revised version. \n\nThank you again for your valuable feedback. We hope our responses address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YcLElEgLW2", "reviewer": "Reviewer_WyrU", "summary": "This paper presents a method to enhance the performance of GeoTransformer by filtering outlier correspondences at the coarse level using a multi-scale, overlap-guided Sinkhorn algorithm. It introduces an overlap-aware Sinkhorn Distance designed to detect potential overlapping points, thereby enhancing the robustness of consistency calculations and reducing the complexity of solutions.", "questions": "1. How about if directly use a outlier removal method such as MAC, FastMac, GCRANSAC to replace the Hierarchical Sinkhorn Tree, what is the performance?\n2. The authors claim that the proposed method mitigates the effects of low overlap and high noise as its main contribution. However, it appears unable to address the challenges faced by the GeoTransformer in cases of extremely low overlap during registration. It would be beneficial to know how the method performs when the overlap ratio between two point clouds is less than 10%. \n3. Equation (2) appears to be incorrect and requires further explanation.", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a coarse-level outlier removal method that can be integrated into the GeoTransformer and trained end-to-end with the correspondence search. This approach is intriguing to me, despite being incremental.\n\n2. It proposes a method for modeling multi-scale consistency named HST, which characterizes the similarity of potential overlapping points in neighboring areas, layer by layer, and aggregates these into a multi-scale consistency framework.\n\n3. The numerical results achieve state-of-the-art performance on both indoor and outdoor benchmarks.", "weaknesses": "The method appears to be an incremental enhancement, adding a multi-scale local consistency module to the GeoTransformer, which increases both model and time complexity. The registration results still heavily depend on the coarse matching that generates initial correspondences via the GeoTransformer. If these initial correspondences are of poor quality, then performance suffers.\n\nThe paper lacks an ablation study on the parameter selection for the top-q predicted scores and the overlap points, which could provide deeper insights into the model's sensitivity to these parameters.\n\nAdditionally, it would be beneficial to explore the impact of replacing the Hierarchical Sinkhorn Tree with a direct outlier removal method such as MAC. Understanding the performance differences could highlight the advantages or disadvantages of the proposed method in handling outliers.\n\nThe overlap score-guided Sinkhorn algorithm for registration, which was also used in OCFNet, should be cited, even though the application method differs. Additionally, this method functions as an outlier removal technique, similar to approaches like FastMac [2]. Therefore,  many sota outlier remove methods need to be compared.\n\nThe authors claim that the proposed method mitigates the effects of low overlap and high noise as its main contribution. However, it appears unable to address the challenges faced by the GeoTransformer in cases of extremely low overlap during registration. It would be beneficial to know how the method performs when the overlap ratio between two point clouds is less than 10%.\n\n[1] Mei, Guofeng, et al. \"Overlap-guided coarse-to-fine correspondence prediction for point cloud registration.\" 2022 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2022.\n[2]YZhang, Yifei, et al. \"FastMAC: Stochastic Spectral Sampling of Correspondence Graph.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first would like to thank the reviewer for giving us valuable comments.\n\n**Response to the comments on our work (first paragraph of Weaknesses)**: Thank you for your insightful comments on our work. We understand your concern that our method builds upon the foundation laid by existing work. However, we would like to emphasize that modeling the multi-scale consistency (MSC) is non-trivial due to the challenges in ensuring both effectiveness and efficiency. To achieve this, we proposed a series of methods to improve performance while maintaining low complexity, such as a lightweight overlap prediction module, Overlap-aware SD with Overlap Filtering and Overlap-aware Marginals, pruned HST, and others. Our extensive experiments on both indoor and outdoor benchmarks have demonstrated their scene-agnostic effectiveness. HST shows significant improvement compared to SOTAs, maintaining robust performance even under low overlap (please refer to the 3DLoMatch results in Tab. 1 & 2, as well as our response to Question 2 below), while introducing only a slight, acceptable time overhead.  \nAs for your concern that we might suffer from poor initial correspondences, we believe that by modeling MSC, our method improves the quality of the correspondence set and thus suffers less performance degradation compared to others. All current coarse-to-fine-based methods heavily depend on coarse matching, and if the coarse correspondences are poor, they will face performance loss. Please refer to our response to Weakness 2. Under extremely low overlap, GeoTransformer shows significant performance degradation, whereas HST suffers less.  \n\n\n**Response to Question 1**: Thanks for your suggestions about comparing HST with outlier rejection methods. We are also very interested in the comparison. For fairness, we replace HST directly with GC-RANSAC, MAC, and FastMAC. Results on both 3DMatch and 3DLoMatch can be found in the following table.\n\n|3DM\\|3DLM|RR(\\%)|RRE($^\\circ$)|RTE(m)|RR(\\%)|RRE($^\\circ$)|RTE(m)|time(s)|\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|vanilla|91.5|1.91|0.068|74.0|2.95|0.090|0.260|\n|GC-RANSAC|92.1|1.78|0.068|73.4|2.96|0.088|0.324|\n|MAC|92.2|1.99|0.067|74.4|2.85|0.086|0.475|\n|FastMAC|91.9|1.73|0.062|74.2|2.86|0.087|0.282|\n|HST|93.2|1.70|0.059|77.3|2.71|0.084|0.296|\n\n**Response to Question 2**: Thank you for your constructive suggestions to test the HST under extremely low overlap. We design a set of experiments targeting overlap ratios of less than 10% and compare with GeoTransformer. Please note that the currently available preprocessed datasets, 3DMatch (overlap>30%) and 3DLoMatch (10%<overlap<30%), both do not include samples with such extremely low overlap. Therefore, we access the 3DMatch raw dataset and collect a set of pairs with overlap<10%, which we refer to as 3DExtremeLoMatch (3DExMatch), containing a total of 1,343 samples. We first test the model pre-trained on 3DMatch directly on 3DExMatch, and the results can be found in the following table:\n\n|Estimator|Method|RR(\\%)|RRE($^\\circ$)|RTE(m)|\n|---|---|:---:|:---:|:---:|\n|LGR|GeoTrans|29.8|3.83|0.110|\n||HST|32.5|3.78|0.111|\n|RANSAC-250|GeoTrans|28.8|4.56|0.123|\n||HST|31.5|4.42|0.122|\n|RANSAC-500|GeoTrans|30.5|4.25|0.121|\n||HST|32.5|4.16|0.118|\n|RANSAC-1000|GeoTrans|31.7|4.23|0.118|\n||HST|33.6|4.03|0.114|\n|RANSAC-2500|GeoTrans|31.9|4.09|0.116|\n||HST|34.7|3.81|0.111|\n|RANSAC-5000|GeoTrans|31.0|4.04|0.117|\n||HST|34.4|3.87|0.114|\n\nWe then randomly divide the 3DExMatch into training, validation, and test sets with proportions of 60% (805 samples), 10% (134 samples), and 30% (404 samples), respectively. We fine-tune GeoTransformer and HST both for 3 epochs on the training set and then evaluate on the test set. The results are as follows:\n\n|Estimator|Method|RR(\\%)|RRE($^\\circ$)|RTE(m)|\n|---|---|:---:|:---:|:---:|\n|LGR|GeoTrans|53.8|3.73|0.107|\n||HST|59.0|3.79|0.106|\n|RANSAC-250|GeoTrans|47.0|4.39|0.121|\n||HST|49.8|4.43|0.119|\n|RANSAC-500|GeoTrans|49.9|4.55|0.113|\n||HST|52.6|4.03|0.117|\n|RANSAC-1000|GeoTrans|52.4|4.03|0.117|\n||HST|57.5|4.13|0.115|\n|RANSAC-2500|GeoTrans|55.1|3.89|0.116|\n||HST|59.0|3.89|0.112|\n|RANSAC-5000|GeoTrans|56.1|4.07|0.116|\n||HST|60.3|3.93|0.111|\n\n**Response to Question 3**: Thank you for your feedback on Eq. (2). We apologize for the typo and for any misunderstandings it may have caused.  \nOur purpose of Eq. (2) is to find the k-NN of any given point $x_i^{l}$ in its next layer to form the local patch. We provide the following explanations in hopes of resolving your confusion.  \n(1) We mistakenly used $argmax$ instead of $argmin$ operator in the formula. We should use the $argmin$ on $K$ to find the k nearest points to form the local patch.  \n(2) Our initial intention is to use a more mathematical $argmin$ operator to represent the k-NN search instead of directly using $kNN(\\cdot)$. However, such an expression might be hard to understand and could lead to ambiguities, such as the definition of the set $S$ in the formula. Therefore, we take a compromise approach by replacing the $argmin$ with the $argtopk$ operator to make it more intuitive.  \nThe following is the revised Eq. (2) and will be included in the revised version.  \nGiven $\\mathbf{x}_i^{(l)}$, its local patch $\\mathbf{P}_i^{(l+1)}$ is defined as:  \n$$\n\\mathbf{P}_i^{(l+1)}=\\operatorname{argtopk} _{\\mathbf{x}_j^{(l+1)}\\in\\mathbf{X}^{(l+1)}}(-||\\mathbf{x}_i^{(l)},\\mathbf{x}_j^{(l+1)}||_2)\n$$\n\n**Lack of ablation on top-q**: We apologize for the lack of ablation on top-q for overlap points filtering. The performance of HST is not sensitive to the selection of q. The RR of varying q can be found in the following table.\n|Top-q|5|10|15|25|30|35|40|45|50|\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|3DMatch|91.2|92.4|92.6|92.9|92.5|92.7|93.1|92.5|91.5|91.8|\n|3DLoMatch|75.4|76.5|76.8|77.2|77.6|77.1|77.3|77.3|76.8|76.4|\n\n**Citing other overlap score-guided Sinkhorn method**: Thank you for your suggestion. We will cite OCFNet in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WmObJyGsBd", "reviewer": "Reviewer_hZ2Q", "summary": "This paper proposed a hierchical skinhorn tree approach to extract the correspondences that are consistent across multiple feature scales for the point cloud registration task. Besides, an overlap-aware module is proposed to better locate the correspondences around the overlap regions. The proposed methods are evaluated on two benchmarks and its results show superior performance than existing SOTA methods in terms of success rate and registration accuracy.", "questions": "I don’t have any question.", "limitations": "Multi-scale consistency is a quite strong constrains, which will lead to no match or erroneous matches when the learned multi-scale features are less descriptiveness or the test data are extremely low overlap cases. Moreover, the optimization of multi-scale consistency requires a careful fine tuning of hyperparameters, such as #levels, grid size.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It applied the mulit-scale consistency concept to the point cloud registration task. Previous criteria for extreacting correspondences are either based on nearest distance in feature space or geometric consistency in Euclidean space. Here, it extends the former criteria in a multi-scale level. And its experiment results validates its effectiveness. \n\n2. It extends the skinhorn algorithm to the multi scale level to model the multi-scale consistency and proposed a BFS way to optimize the assignment problem in multi-scale levels.", "weaknesses": "1. The use of wrong terminology. In Eq1, it seems describe how to perform feature matching across multiple scales, not describing “inlier correspondence”, which is defined as two points of a match are close enough under rigid transformation.   \n\n2.  In Eq 5, the variable Tij is not explained. Even though its definition can be found in the Superglue paper, which is a binary assignment variable indicating whether it is a good or bad match, it is never a bad thing to explain more.\n\n3. For sec 3.2.4, it is better to illustrate how to optimize the proposed HST in BFS way with a detailed figure or a piece of pseudo code.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your valuable comments and suggestions.  \n\n**Response to Weakness 1**: Thank you for pointing out this issue. We apologize for any misunderstanding caused by the term \"inlier correspondence\" in Eq. (1). It should be revised as \"putative inlier correspondences\". As you mentioned, the definition of inlier correspondence refers to points in two point clouds that are sufficiently close given a ground truth transformation. Actually our original intention in using 'inlier correspondence' here was to convey that, since the ground truth transformation is not provided beforehand, we often have to design a method to estimate the inlier correspondence set. A common approach is to do feature matching, as you mentioned. Therefore, in Eq. (1) we aim to describe how we could solve this problem from the Multi-scale Consistency (MSC) perspective, so the \"inlier correspondence\" here should be revised as \"putative inlier correspondence\", representing the inlier correspondence inferred by MSC. We will fix it to \"putative inlier correspondence\" in the revised version.  \n\n**Response to Weakness 2**: Thanks for your feedback regarding the lack of explanation of $T_{ij}$. Here $T_{ij}$ is the element from the assignment matrix $T \\in R^{(|\\mu_{ov}|+1)\\times(|\\nu_{ov}|+1)}$ of optimal transport problem, indicating the pairwise matching (overlap) score from source to target point cloud. The row and column sum of the matrix $T$ are respectively subject to the augmented marginal distributions $\\mu_{ov}$ and $\\nu_{ov}$ to ensure the allocation constraints. $T$ is the optimization variable in the calculation of the Overlap-aware Sinkhorn Distance and can be efficiently solved by the Sinkhorn algorithm. We will include the above explanation in the revised version.  \n\n**Response to Weakness 3**: Thank you for your suggestions on better illustrating the optimization of HST. We will include a detailed diagram in the revised version that illustrate the forward computation process and the backward gradient flow to thoroughly demonstrate the optimization process of HST.  \n\n**Response to Limitations** : Thank you for your insightful feedback on our work!  \nWe understand your concern that Multi-scale Consistency (MSC) may hinder the model's ability to learn multi-scale descriptive features because it requires the features of two point clouds to exhibit similarity across multiple scales. In fact, we also observe this issue in our experiments. Therefore, we introduce a layer-wise overlap-aware circle loss (please refer to lines 253-255 in the paper) to help maintain the descriptiveness of features at each layer. This ensures that the backbone does not learn overly uniform features while still preserving MSC.  \nAs for the concern that MSC might fail in test data with extremely low overlap, we design a set of experiments targeting overlap ratios of less than 10% and compare the performance with GeoTransformer to validate the robustness of HST when facing extremely low overlap. Please note that the currently available preprocessed datasets, 3DMatch (overlap > 30%) and 3DLoMatch (10% < overlap < 30%), do not include samples with such extremely low overlap (overlap<10%). Therefore, we access the 3DMatch raw dataset and collect a set of point cloud pairs with overlap < 10%, which we refer to as 3DExtremeLoMatch (3DExMatch) dataset, containing a total of 1,343 samples. We first test the model pre-trained on 3DMatch directly on 3DExMatch, and the results can be found in the following table:  \n|Estimator|Method|RR(\\%)|RRE($^\\circ$)|RTE(m)|\n|---|---|:---:|:---:|:---:|\n|LGR|GeoTrans|29.8|3.83|0.110|\n||HST|32.5|3.78|0.111|\n|RANSAC-250|GeoTrans|28.8|4.56|0.123|\n||HST|31.5|4.42|0.122|\n|RANSAC-500|GeoTrans|30.5|4.25|0.121|\n||HST|32.5|4.16|0.118|\n|RANSAC-1000|GeoTrans|31.7|4.23|0.118|\n||HST|33.6|4.03|0.114|\n|RANSAC-2500|GeoTrans|31.9|4.09|0.116|\n||HST|34.7|3.81|0.111|\n|RANSAC-5000|GeoTrans|31.0|4.04|0.117|\n||HST|34.4|3.87|0.114|\n\nWe then randomly divide the 3DExMatch into training, validation, and test sets with proportions of 60% (805 samples), 10% (134 samples), and 30% (404 samples), respectively. We fine-tune GeoTransformer and HST for 3 epochs on the training set, and then evaluate the models on the test set. The results can be found in the following table. Both results clearly demonstrate that HST maintains strong performance even under low overlap conditions, confirming the effectiveness of our method.\n|Estimator|Method|RR(\\%)|RRE($^\\circ$)|RTE(m)|\n|---|---|:---:|:---:|:---:|\n|LGR|GeoTrans|53.8|3.73|0.107|\n||HST|59.0|3.79|0.106|\n|RANSAC-250|GeoTrans|47.0|4.39|0.121|\n||HST|49.8|4.43|0.119|\n|RANSAC-500|GeoTrans|49.9|4.55|0.113|\n||HST|52.6|4.03|0.117|\n|RANSAC-1000|GeoTrans|52.4|4.03|0.117|\n||HST|57.5|4.13|0.115|\n|RANSAC-2500|GeoTrans|55.1|3.89|0.116|\n||HST|59.0|3.89|0.112|\n|RANSAC-5000|GeoTrans|56.1|4.07|0.116|\n||HST|60.3|3.93|0.111|\n\nRegarding the potential need to adjust hyperparameters like the number of levels and grid size for optimization, we believe that the adjustment of the number of levels primarily involves balancing performance and latency. As shown in the ablation study in Table 4, increasing the number of levels enhances performance but also slightly increases inference time. If the application scenario requires more real-time processing, the number of levels can be reduced to accelerate the model without significantly sacrificing performance. Similar to other coarse-to-fine methods, such as GeoTransformer, hyperparameters like grid size also require fine-tuning based on the specific scenario and the source of the point clouds.  \n\nAll the above discussions will be added in the revised version. We hope the above responses address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We first would like to thank all the reviewers for providing insightful comments and we are immensely grateful for your thorough feedback on our manuscript. It is encouraging that the reviews found  

* Our paper is well-written  
  - "The authors have effectively organized their ideas, making the content easy to follow and understand." - Reviewer AXAW  
  - "This paper is well-written, well-organized, and easy to follow" - Reviewer Sd2B  

* Our paper presents a novel and interesting idea  
  - "The authors’ introduction of the concept of multi-scale consistency (MSC) for robust registration is both interesting and promising. Developing a Hierarchical Sinkhorn Tree to model MSC is also a novel approach that achieves significant performance gains." - Reviewer AXAW  
  - "The idea of introducing tree structure into a coarse-to-fine mechanism is somehow interesting." - Reviewer Sd2B  
  - "This approach is intriguing to me" - Reviewer WyrU  

* Experiments show superior performance and verify the effectiveness of the proposed method
  - "The authors provide sufficient comparisons and ablation studies to verify the effectiveness of the proposed mechanism." - Reviewer AXAW  
  - "The numerical results achieve state-of-the-art performance on both indoor and outdoor benchmarks." - Reviewer - WyrU  
  - "The proposed methods are evaluated on two benchmarks and its results show superior performance than existing SOTA methods in terms of success rate and registration accuracy." - Reviewer hZ2Q  

---

We have carefully read all the comments and responded to them in detail. All of those will be addressed in the final version.  
We summarize the main concerns of the reviews with the corresponding response as follows.  

1. **About the performance under extremely low overlap.**  
To test the robustness of HST when facing extremely low overlap, we design a set of experiments targeting overlap ratios of less than 10% and compare the performance with GeoTransformer using different estimators. However, we found that currently available preprocessed datasets, 3DMatch (overlap > 30%) and 3DLoMatch (10% < overlap < 30%), do not include samples with such extremely low overlap (overlap<10%). Consequently, we accessed the raw 3DMatch dataset and collected a new set of point cloud pairs with overlap < 10%, which we refer to as the 3DExtremeLoMatch (3DExMatch) dataset, comprising a total of 1,343 samples. We conducted two types of experiments: one where we directly evaluated the pre-trained model and another where we evaluated the model after fine-tuning on the partitioned 3DExMatch dataset. The empirical results demonstrate that our proposed HST outperforms GeoTransformer in both settings, particularly after fine-tuning. This indicates that, even under extremely low overlap conditions, our method does not fail due to a lack of matches or excessive mismatches. Instead, by incorporating MSC modeling, HST can form a higher-quality correspondence set, resulting in improved performance under severe conditions and less performance degradation compared to other methods.  

2. **About the comparison with outlier rejection-based methods.**  
We have conducted experiments comparing HST with outlier rejection-based methods to gain more insights into handling outliers. For a fair comparison, we replace HST directly with the state-of-the-art approaches GC-RANSAC, MAC, and FastMAC, and evaluate them on both 3DMatch and 3DLoMatch datasets. All three methods, along with HST, showed performance improvements compared to the vanilla GeoTransformer, with HST demonstrating the most significant enhancement. This suggests that when the quality of the correspondence set is high, the registration process benefits from outlier rejection methods. However, on 3DLoMatch, the improvements from these three methods were less pronounced, and GC-RANSAC even showed a potential negative impact. In contrast, HST maintained better robustness, highlighting its superior effectiveness over previous outlier rejection methods in scenarios with low overlap and high noise. This further validates the efficacy of our MSC modeling in handling outliers from coarse correspondences.  

Thanks again for your efforts in the review. We appreciate all the valuable feedback that helped us to improve our submission.

Meta-review: The paper is about introducing multi-scale consistency for point cloud registration. This is an under-explored problem due to the characteristics of point clouds (i.e., the format being irregular). This paper deals with this problem effectively with the newly proposed hierarchical Sinkhorn tree (HST). Most reviewers have agreed on the importance of the topic, as well as the novelty and good performance of the proposed method. There were some concerns (e.g., incremental contribution, marginal improvements, and processing time), but they were all clearly resolved by the authors' responses. The AC has personally reviewed the paper in detail and agreed on the above points.
Decision: Accept (poster)