Learning to Register Unbalanced Point Pairs
Authors: Kanghee Lee
Junha Lee
Jaesik Park
Venue: Submitted to ICLR 2023 (2023)
Keywords: Point Cloud Registration

Abstract: Point cloud registration methods can effectively handle large-scale, partially overlapping point cloud pairs. Despite its practicality, matching the unbalanced pairs in terms of spatial extent and density has been overlooked and rarely studied. We present a novel method, dubbed UPPNet, for Unbalanced Point cloud Pair registration. We propose to incorporate a hierarchical framework that effectively finds inlier correspondences by gradually reducing search space. The proposed method first predicts subregions within target point cloud that are likely to be overlapped with query. Then following super-point matching and fine-grained refinement modules predict accurate inlier correspondences between the target and query. Additional geometric constraints are applied to refine the correspondences that satisfy spatial compatibility. The proposed network can be trained in an end-to-end manner, predicting the accurate rigid transformation with a single forward pass. To validate the efficacy of the proposed method, we create a carefully designed benchmark, named KITTI-UPP dataset, by augmenting the KITTI odometry dataset. Extensive experiments reveal that the proposed method not only outperforms state-of-the-art point cloud registration methods by large margins on KITTI-UPP benchmark, but also achieves competitive results on the standard pairwise registration benchmark including 3DMatch, 3DLoMatch, ScanNet, and KITTI, thus showing the applicability of our method on various datasets. The source code and dataset will be publicly released.

Under review as a conference paper at ICLR 2023
LEARNING TO REGISTER UNBALANCED POINT PAIRS
Anonymous authors
Paper under double-blind review
ABSTRACT
Point cloud registration methods can effectively handle large-scale, partially over- lapping point cloud pairs. Despite its practicality, matching the unbalanced pairs in terms of spatial extent and density has been overlooked and rarely studied. We present a novel method, dubbed UPPNet, for Unbalanced Point cloud Pair registra- tion. We propose to incorporate a hicrarchical framework that effectively finds inlier correspondences by gradually reducing search space. The proposed method first predicts subregions within target point cloud that are likely to be overlapped with query. Then following super-point matching and fine-grained refinement modules predict accurate inlier correspondences between the target and query. Additional geometric constraints are applied to refine the correspondences that satisfy spatial compatibility. The proposed network can be trained in an end-to-end manner, predicting the accurate rigid transformation with single forward pass. To validate the efficacy of the proposed method, we create a carefully designed benchmark, named KITTI-UPP dataset, by augmenting the KITTI odometry dataset. Extensive experiments reveal that the proposed method not only outperforms state-of-the-art point cloud registration methods by large margins on KITTI-UPP benchmark, but also achieves competitive results on the standard pairwise registration benchmark including 3DMatch, 3DLoMatch, ScanNet, and KITTI, thus showing the appli- cability of our method on various datasets. The source code and dataset will be publicly released.
INTRODUCTION
Point cloud registration is a task that aims to recover 3D rigid transformation between two possibly overlapping point cloud fragments. The rapid advance of commodity 3D sensors gives rise to the ne- cessity of efficient point cloud registration algorithms for numerous real-world applications, including 3D reconstruction for virtual-, augmented-, and mixed reality applications, and the navigation systems of autonomous vehicles or robotic agents. Recent work has made remarkable progress in developing learning-based point cloud registration algorithms for tackling real-world 3D scans (Geiger et al., 2012; Zeng et al., 2017) with high-ress regouuteoottion feature extraction (Choy et al., 201 2019b; Bai et al., 2020) under presence of low ratio of the inlier correspondences (Choy et al., 2020b; Bai et al., 2021; Lee al., 2021) or small overlap region between point pairs (Huang et al., 2021).
However, the imbalance issue in terms of spatial extent and point density between the input point clouds is often overlooked, despite its practical utility in the problems such incremental mapping, or the registration of partial observations and the holistic environment. For instance, there are sensible solutions for registering pair of 3D LiDAR scans, but registering single LiDAR scan and large-scale 3D map still remains challenging. A viable solution is to apply a global localization approach (Uy & Lee, 2018: Komorowski, 2021; Du al., 2020; Zhang & Xiao, 2019; Liu al., 2019), but the existing methods cast the problem as retrieval task and assume that the 3D map is given as a set of overlapping 3D seans rather than a holistic map, which is not generally applicable to the unbalanced point pairs.
Recent feature-based pairwise point cloud registration methods are equipped with matchability detection (Bai et al., 2020), overlap detection (Huang et al., 2021), hierarchical correspondence prediction (Yu et al., 2021), which are possibly advantageous in registering unbalanced point clouds. However, empirically found that they collapse in registering unbalanced point clouds. Point cloud description and matching in the modern feature-basee based registration methods tend to be distracted by similar geometric structures that often appear in the larger point cloud.

---

Under review as a conference paper at ICLR 2023
To this end. propose UPPNet. the first neural architecture which is designed to be efficient for large-scale Unbalanced Point cloud Pair registration. UPPNet is a hierarchical framework that effectively finds inlier correspondences by gradually reducing search space. In the coarsest level, submap proposal module proposes the subregions that are likely to be overlapped with the query by utilizing global geometric context. Then, the coarse-to-fine matching module predicts accurate point-level correspondences by utilizing attention- context aggregation and solving optimal transport problems. The subsequent structured matching module filters out outlier correspondences that violate spatial compatibility.
To evaluate our method, we create a carefully designed benchmark, namely KITTI-UPP dataset, for matching point cloud pairs under the diverse spatial extent and point density imbalance by augmenting the KITTI odometry dataset (Geiger et al., 2012). The experiment shows that our method improves the Registration Recall on the KITTI-UPP dataset by over 19.6% than offe-of-the-ar registration pipelines when the target point cloud is 11.1 times spatially larger and 11.7 times denser than the query point cloud. Furthermore, we evaluate the proposed method under the unbalanced indoor environments using ScanNet (Dai et al., 2017) dataset and show that the proposed method can be generalized for indoor RGB-D scans. Finally, to demonstrate the applicability of the proposed method for partially overlapped point cloud pairs, we evaluate our method on the standard pairwise registration benchmarks, 3DMatch, 3DLoMatch (Zeng et al., 2017; Huang al., 2021), and KITTI odometry (Geiger et al., 2012) datasets. The proposed method achieves competitive registration accuracy with the modern pairwise registration methods (Bai et al., 2020; Huang al., 2021: Yu et al., 2021). An overview of our method can be found in Figure 1.
Our main contributions are summarized as follows:
We propose novel hierarchical framework that gradually reduces the search space via submap proposal module and coarse-to-fine matching modules, which can effectively handle unbalanced point cloud registration tasks.
We introduce a new benchmark, namely KITTI-UPP dataset, by carefully augmenting the large-scale outdoor LiDAR dataset (Geiger al., 2012).
Our method can be trained in an end-to-end manner and demonstrates strong generalization ability on wide range of spatial and point density. It outperforms the previous state-of-the- art methods by 19.6% Registration Recall on challenging benchmark.
Our method achieves competitive registration accuracy in both unbalanced indoor RGB-D registration (Dai et al., 2017) and the standard pairwise registration benchmarks (Zeng et al., 2017; Huang et al., 2021; : Geiger et al., 2012), showing the applicability of the method.
2 RELATED WORK
Point cloud registration. Given partially overlapping point cloud pairs, point cloud registration aims to estimate the rigid transformation parameters that align the input point clouds. Typical pipelines start with the feature extraction and matching stage to produce set of putative correspondences, followed by robust model fitting algorithm to estimate the pose parameters from given correspondences.
Traditional local descriptors for point clouds (Johnson & Hebert, 1999; Rusu al., 2008; 2009; Tombari et al., 2010; Salti et al., 2014) encode local geometry using hand-crafted features such as surface normal curvature. Recent learnable feature descriptors train network extract geometric features in data-driven manner. Qi et al. (2017) incorporate shared MLP and permutation-invant aggregation operations process unordered and irregular point cloud data. Deng et al. 20:));b extended Qi et al. (2017) by combining point pair feature to extract global context-aware feature descriptors. Choy et al. (2019h) adopted the fully convolutional networks to process point cloud data by utilizing sparse convolution (Choy al., 2019a) and achieved bate-of-theeart feature matching accuracy in large-scale real-world datasets. Bai et al. (2020) and Huang et al. (2021) incorporate keypoint and overlap confidence prediction for robust feature matching between low overlapping point pairs.
The set of putative correspondences that is built by matching local feature descriptors tends to contain a high portion of outliers. Hence, the pose estimation algorithm should be robust against the existence of outliers. RANdom SAmple Consensus (RANSAC) (Fischler & Bolles, 1981) and
2

---

Under review as a conference paper at ICLR 2023
its variants (Fitzgibbon, 2003; Raguram et al., 2012; Hoseinnezhad & Bab- Hab-HHdiashar, 2011; Chum & Matas, 2005) are one of the most popular methods. Recent studies of learning-basee outlier rejection methods apply for the problem of two-view correspondences (Moo Yi et al., 2018: Zhang et al., 2019; Brachmann et al., 2017) and 3D correspondences (Choy et al., 2020b;a). Lee et al. (2021) utilizes Hough voting in 61) parameter space and learnable refinement module, resulting in robust and efficient registration pipeline for the large-scale point cloud. Bai et al. (2021) incorporate the spatial compatibility to filter out noisy correspondences. Yu al. (2021) proposed to avoid keypoint detection by incorporating hierarchical correspondence extraction modules and Lu et al. (2021) present specialized pipeline for large-scale LiDAR scans. However, none of those methods are explicitly designed for the unbalanced point cloud registration and we empirically found that those methods collapse under the extreme imbalance in terms of spatial extent and point density.
Global localization. The early global localization algorithms are based on the 2D images (Chen et al., 2017: Sarlin et al., 2019; Sattler al., 2017), where input images are matched against the 31) map reconstructed with Structure from Motion (SIM). They often cast visual localization task as retrieval problem, where the query images are described with global descriptors, and the most similar point features are retrieved from the database. Vector of Locally Aggregated Descriptors (VLAD) (Jégou et al., 2010; Arandjelovic & Zisserman, 2013) is one of the most widely used approaches. aggregates the local descriptors into single global descriptor via clustering and ís designed to be very low dimensional. NetVLAD (Arandjelovic et al., 2016) has extended it to learnable method.
Recently, 3D point cloud-based global localization methods are developed. Yin et al. (2017; 2018) projects point cloud onto a spherical image and then extracts a global feature descriptor via 2D CNN. In Yin al. (2019), is combined with localization module, which utilizes particle filter and ICP for 3D pose estimatiin. Uy & Lee (2018) apply NetVLAD to point cloud data using PointNet Qi al. (2017) as backbone network. Based on this, Zhang& & Xiao (2019) employs per-point attention to enhance the global descriptor. Liu et al. (2019) applies a graph neural network on hand-crafter point features to encode local context and aggregate it into global descriptors using the NetVLAD layer. Du et al. (2020) jointly optimizes both local and global feature extractors and solves both global retrieval and local registration in a single forward pass. Komorowski (2021) utilizes sparse convolutional networks to overcome the limitation of the local context of PointNet.
One feasible solution to tackle the unbalanced point cloud registration is to device the task into two separate sub-tasks; 1) the nearest frame retrieval and 2) local registration. After the target frame is retrieved, the relative pose between the query and the target frame is predicted by applying the conventional pairwise registration methods. These methods, however, suffer from several drawbacks. First, there are significant overheads in both the computational cost and the memory footprint to represent scene with set of multiple overlapping local frames. Second, they are likely not to generalize when point cloud pairs have a domain difference. For instance, considering the registration between the reconstructed point cloud of a large-scale scene and a local scan, the global localization methods fail to be directly adopted due to the severe density difference. Finally, the success of the registration strongly depends on the localization results. They can not recover when the coarse alignment via localization fails.
3
METHOD
This section describes the proposed UPPNet, designed for registering unbalanced point cloud pairs. To handle spatial and density imbalance, we introduce a hierarchical framework consisting of three levels of matching stages, submap matching (Section 3.3), super point matching (Section 3.4), and point matching (Section 3.5). The overview of the proposed pipeline is illustrated in Figure 1.
3.1 PROBLEM DEFINITION
Given a pair of possibly overlapping point clouds, X E R=x3 Y R our goal is to estimate the optimal rigid transformation R t* that minimizes the geometric error defined as follows:
R t* arg min
|Ry, +t x,112
(1)
R.t
(KJ)ECC
where R E SO(3), t E R33 are rotational matrix and translation vector, C is a set of true correspon- dences between X and Y. In this paper, are interested in the case where V(X) >> V(Y), and
3

---

Under review as a conference paper at ICLR 2023
Submap Matching
Y (querry)
(map)
O" R
Couseor Aggregation
c Ancwriow-bated
( ('
Sutvepiio Proporal & Groupsngg 65 t Si)
Agaregom Pretures
Fy
stared
(X',
Super Point Matching
Matching modale
Submap Matching
(P.R)
Sinkhorn
A Gronping
Super Point Matching
Slock ennnes
P
L.g
shanal
Point Matching
Point Matching
Strectored Matching
Cerepynomees
Node
Maeding mochie
Lpp
R,E
(Left) Overview of UPPNet. Given an unbalanced point cloud pair, UPPNet hierarchically reduce the search space by utilizing multi-level features. (Top right) Super point features get strengthen via Atlention-based Context aggregation module, shown at @., and aggregated into global descriptor using Generalized mean pooling, shown at C We select the submaps with top-k similarity. (Middle right) We build similarity matrix using super point features in each selected submap. We solve the optimal transport problem on the similarity matrix, and the super point correspondences
Figure 1:
estimated, shown at ( (Botton right) Super point correspondences refined to point correspondences by utilizing node grouping, shown at @,, and matching module. ! Structured matching filters the noisy correspondences that do not satisfy spatial compatibility. Modules in UPPNet, including feature extraction and correspondence estimation, can be trained in an end-to-end manner with the three losses L., Lg. and Lp-
>> m, where VV-)) denotes the spatial volume of the bounding box that tightly covers the input point cloud. In other words, there is spatial and density imbalance between X and Y. From here on, we will refer to the reference point cloud that spans larger spatial extent with higher density (X) as map, and a query point cloud with a small and sparse point cloud (Y) as query. Furthermore, we use the terms spatial imbalance factor, Ps V(Y)' and point imbalance factor, Pp ;. to denote the relative imbalances between two point clouds terms of spatial extent and number of points.
v(X)
Note that we can also calculate the density imbalance factor by Pa Pa
3.2 FEATURE EXTRACTION
The proposed UPPNet begins with the feature extraction to encode the geometric context of map and query point clouds. To achieve this goal, we adopt a shared U-shaped network implemented with KPConv (Thomas et al., 2019) to extract the multi-level features. Given an input point cloud X E x with initial feature F E Raad the feature extractor fo(-) outputs the pointwise feature Fx E Raxd that encode local geometric context, denoted (X, Fx) fo (X, FiR). At the most coarse resolution, i.e., at the end of the encoder, we obtain downsampled points with corresponding feature vectors, and we call downsampled points as super points and denotes the point coordinates and the features with X' E Rnn x3 and Fx.. For each super point x', the input point cloud X can be partitioned into groups by assigning each point to its closest super point as in Yu et al. (2021):
G E
x111 < ||
x111 i]}
4

---

Under review as a conference paper at ICLR 2023
where we denote the corresponding super point feature map F',,
3.3 SUBMAP MATCHING
Building submaps. The first stage of our hierarchical registration pipeline is the submap proposal that builds the submap candidates that are likely to be overlapped with the query point cloud. To do so, divide the map into L overlapping submaps. A submap S₁ is subset of superpoints of the map, i.c., S; c X', that lie in i-th submap region S, S, = {x' E X' :''' E S. Each (cubic-shaped) submap region S, with its center C2 E R (which we call center point) and edge length of o E IR+ is formally defined as S = {s IR3 max s - c, S r/2). In this work, we evenly place L overlapping submap regions {8.) i=1 cover all the superpoints of the map X' with carefully chosen overlapping ratio je, satisfying [S, S = ll and ||c, Czll = e y for any adjacent, overlapping submap regions S, and Sj. We set μ = 0.5 in our experiments. The edge length of the submap regions, V E R$ , is defined by the spatial size of the query Y: Specifically, we first compute the furthest distance between two points in Y along i-th axis as s = max,.aa, mo.. R + thus giving {s.)_ of which respective elements represent width (81). height (82), and length (83) of Y. The edge length of submap regions is defined as the largest edge length of Y: v = max,3;, The overview of the submap proposal and grouping is shown in Figure (b).
Attention-basee context aggregation. The super point feature maps of query and submaps are then strengthened via attention- bassddd---dddd-- context aggregation module (Yu et al., 2021; Huang al., 2021: Sarlin et al., 2020) to incorporate the global geometric context. As in Huang et al. (2021), we construct a bipartite graph between query and map point cloud regarding the super points as nodes, and we apply a sequence of self-, cross-, and self-attention layers to fuse the global context between two point clouds and augment each super point feature. Given super point feature maps (F'x, 1F'x.'' Fy), we linearly project the source feature F'x query, Q WqF's x and target feature Fyy to key and value as K WKFYy, V = F respectively, where WQ. WK. and Wy are the learnable parameters. To calculate the message M that flows in the graph, we attention-based aggregation
V, where is the channel dimension of super point features. When calculating self-attention messages, we take (F'x Fxx) and (F'y, F'y) as input and flow messages to corresponding subgraphs.
as follows: M QK"
vb
Global feature aggregation. After augment super point features, we aggregate super point features into a global descriptor for each submap. The super point features are lifted into a higher dimension by linear layer and then aggregated into global descriptor via Generalized Mean Pooling (GeM) (Radenović et al., 2018). GeM has shown its strength in localization tasks for various modality (Tolias et al., 2015; Komorowski, 2021). Concisely, GeM aggregates the super point features F's, for submap S; and builds the global descriptor FGP as FG) ( Σx; es, (F%, (k)))) 1/a where S. (k) is k-th element of the global descriptor, and is a learnable parameter.
Global feature matching. With the aggregated global descriptors for each submap (F and query F&D, we calculate the similarity matrix using I.2 distance between two global descriptors:
GD
F GD
GD 12,
GD
LGGD
GD
mx1
where dGD is a one dimensional vector because the query point cloud is described with a single global descriptor FGD, whereas the map X is described with global descriptors FCD for each submap S.. On train time, we apply loss function on dDD with groundtruth supervision. On inference time, we pick k submaps with the top-k similarity values, where & is a hyperparameter.
3.4 SUPER POINT MATCHING
After the candidate submaps are proposed with the consideration of the global geometric context, we perform the super point matching. For each proposed submap, the super points belonging the submap S, are retrieved with corresponding super point features. F's, E RlS, xa' Then we leverage super point features F's,, F'y of submap and query calculate the similarity matrix Ss, using inner product:
Ss, =FF F
Ss, R S.(xn'
5

---

Under review as a conference paper at ICLR 2023
Neighborhood constraint. The similarity matrix Ss, effectively convey the geometric information with moderate receptive field size. Following Lu et al. (2021), we incorporate additional geometric context, called neighborhood constraint, to handle challenging cases. As described in Eq. 2, each point x E X are assigned to the closest super point. For each super point x E X', we aggregate the neighbor features of points within the partition G; using max pooling and denote the aggregated features as neighbor feature FG.. Now we calculate another similarity matrix Ns., F Fy. called neighbor similarity matrix. The final similarity matrix is defined as addition of two similarity matrices augmented with additional row and column for slack entries to handle unmatched super points as suggested in Yu et al. (2021); Sarlin al. (2020):
3s. Ss. Nss
z)
3s. E ([5(+++))(()++)
(5)
2
2]
We then utilize Sinkhorn algorithm (Sinkhorn & Knopp. 1967) solve optimal transport problem on 2s, and obtain Zs.. Each entry (i,j) in Zs, indicates normalized probability whether (i, j) correspondence is a true correspondence. By thresholding 2s. by predefined value Tz, we obtain the set of predicted super point correspondences, C", and they passed to the final point matching module to point correspondences,
Structured matching, We use spatial compatibility (Bai et al., 2021; Lee et al., 2021) to further reject the outlier correspondences. It considers two correspondences (pt,9) and (p;.9) are likely to be inliers if their spatial distance Id (pt, P,)) d (g:: q;]l is smaller than predefined threshold. Extending this concept, for each correspondence, we compute the compatibility score with respect to the number of their compatible correspondences in the whole correspondence set. For example, whole correspondences set is fcl, c2, c3) and compatibility score of cl will be 2 when c2 and c3 satisfy spatial compatibility with cl. With the compatibility score, we consider correspondences with a low score as outliers and filter out them. In our hierarchical framework, we apply this strategy to super point matching module and point matching module extract only reliable correspondence pairs. Furthermore, KNN search is used to guarantee as many inliers as possible for the initial correspondence set. For more details about the structured matching, please refer Appendix.
3.5 POINT MATCHING
Given predicted super point correspondence, we refine it to point-level correspondences for the final rigid transformation parameter estimation. We expand a single super point correspondence pair of point patches with neighboring points around the super points in each point cloud. We use the point-to-super point grouping as described in Eq. and pass the selected point groups to the matching module as in Section 3.4. After applying the threshold to the similarity matrix and structured matching, we get the final set of peint-level correspondences between two input point clouds. We then run RANSAC on the correspondence set to estimate the rigid transformation parameter. Note that there are multiple sets of correspondences since we perform the matching module for each proposed submap in parallel. We select the one with the highest inlier ratio among the multiple transformation candidates as our final prediction.
3.6 LOsS
We train our network using the loss function defined as L L. XgC. + App where the total loss is the weighted sum of submap matching loss Lg, super point matching loss Cg. and point matching loss Cp. Specifically, define point matching loss Cp as:
- Eis ((i.j) j) log 2(i
(6)
Σ,, (i,
where 2 is the predicted similarity matrix after solving optimal transport using Sinkhorn algorithm, 2 is the binary matrix that indicates groundtruth. If (i.j) correspondence is true correspondence then 2ii = 1, and (i, = O otherwise. For L.g and L. we apply the same formulation, but we calculate the overlap ratio between two patches around the super points as ground-truth soft label values. This fact indicates that we only provide point-wise binary matrix for the supervision, and the similarity matrix for super point matching and submap matching are calculated with 2. Additional supervision rather than the rigid transformation matrix between query and map point clouds is not required. More details calculating ground-truth similarity matrices are provided in Appendix.

---

Under review as a conference paper at ICLR 2023
Query
Registration results
Query
Registration results
(a) KITTI-UPP
(b) ScanNet
Figure 2: Qualitative results on the KITTI-UPP dataset (a) and ScanNet dataset (b). Left figure for each dataset shows query point cloud and right one shows our registration result on unbalanced point cloud pairs. The green lines indicate inlier correspondences and the red lines indicate the outlier correspondences.
4 EXPERIMENT
4.1 EXPERIMENTAL SETTINGS
KITTI-UPP dataset. To the best of our knowledge, there are no available benchmarks and public datasets targeting the large-scale unbalanced point cloud registration task. To validate the effectiveness of our approach, we introduce KITTI-UPP, a carefully designed dataset for large-scale unbalanced point cloud registration tasks. We build KITTI-UPP by aggregating sequential LiDAR frames for each scene provided by the KITTI Odometry benchmark (Geiger al., 2012). To control spatial and density imbalance factor between map and the query as defined in Section 3.1, we tune two parameters when selecting KITTI frames for aggregation: range and hop. The range determines how many frames we use to construct a map, indicating the map size. If the range value is set to 500, we aggregate 500 consecutive LiDAR frames to build a single map. The hop indicates the frame jump for the LiDAR frame aggregation. If we set the hop value as 10, every 10th frame is used for the aggregation. In this manner, the hop controls the density of the aggregated point cloud.
In our experiment, we set 300 and 10 as the default values for range and hop, respectively, to train UPPNet and baseline approaches. We then utilize KITTI-UPP scenes made with other hops and ranges that are unseen during the training to analyze the registration performance on the various spatial and density imbalance factors. Note that we ensure that any query is not included in the map. When we aggregate every 10th frames to construct a map, i.c., {10- i|0 I i < rrange }}. then
select the query frames from a index set {10-E i 5 10 < is not used in both query and map.
range
} to ensure that the same frame
10
ScanNet benchmark. To validate the robustness of our proposed method in large-scale indoor ScanNet (Dai et al., 2017) benchmark which contains 2M RGB-D scans of over 707 unique indoor scenes. To evaluate the methods under an unbalanced environment, the provided reconstructed mesh of ScanNet as map and register single RGB-D frame with the map, thus being able to omit the labor-heavy process for generating maps as in KITTI- UPP dataset construction. For the unbalanced point cloud registration task. An example pair is illustrated in Figure 2. Specifically, we use a subset of ScanNet which consists of 12/2/5 scenes for traniing/alliaation/eee
environments,
splits respectively where each scene contains a varying number of scans ranging from 8 to 113.
Evaluation metric. We use the standard metric assess the pairwise registration accuracy using Rotational Error (RE): arecos Tr (R and relative Translational Error (TE): |tt i|l where R. are the predicted rotation matrix and a translation vector, R, È are the ground-truth. We also use relative Inlier Ratio (IR) and Registration Recall (RR) for the evaluation. IR is defined as the ratio of correspondences whose geometric distances are below the predefined threshold (T)) when transformed with the ground-truth transformation For registered pair having RE and TE less than the predefined thresholds (TR. Tt), we regard as a successful registration and calculate the RR of successful registration over the entire dataset. For indoor datasets, report three standard metries,
7

---

Under review as a conference paper at ICLR 2023
KITTI
KITTI-PPP
KITI
KITTI-PP
100
100
99.3
97.8
95.6
94.5
99.3
94.2
82.5
Registratio Recail (%)
80
86.2
19.6
Registration Recall
60
60
67.6
62.9
40
40
20
20
Point imbalance factor
Spatial imbalance factor
HRegNet
FGR-ICP
RANSACHICP
- D3Fest
Predator
-
CoFiNet
Ours
MDV Predator
MinkLoc3DV2+CooFiiet
Figure 3: Evaluation results on KITTI and KITTI-UPP benchmark with various spatial (ps) and point (Pp) imbalance factors. The regions on the left side of the gray lines indicate the balanced pairwise registration environment the standard KITTI dataset on which the previous pairwise registration methods mainly handle. For the experiments with various point imbalance factors (left), we fix the range value to 100 (p.- = 2.7) and change the hop value. For the experiments with various spatial imbalances (right), fix the hop value to 25 (pp- 6.9) and change the range value.
Table Quantitative registration results on 3DMatch (Zeng et al., 2017), 3DLoMatch (Huang et al., 2021), 1), and ScanNet (Dai et al., 2017).
3DMatch (Zeng al., 2017)
3DI aaMch (Huang et al., 2021)
ScanNet Dai al. (2017)
RR (%)
FMR (%)
IR (5%
RR (%)
FMR (8)
IR (%)
RR (%)
FMR (%) IR (%)
FCGF (Choy al., 2019b)
85.1
97.4
56.8
40.1
76.6
21.4
D3Feat (Bai al., 2020)
81.6
95.6
39.0
37.2
67.3
13.2
Predator (Huang al., 2021)
89.0
96.6
58.0
59.8
78.6
26.7
CoFiNet (Yu et al., 2021)
89.3
98.1
49.8
67.5
83.1
26.7
50.0
84.2
$1.0
Ours
93.0
96.8
72.4
82.3
39.7
71.0
64.1
namely Registration Recall (RR), Inlier ratio (IR) and Feature Matching Recall (FMR) following the literature. The precise definition of evaluation metrics can be found in the Appendix.
4.2 COMPARISON
Outdoor Experiment. We compare our UPPNet with the state-of-the-ar pairwise registration methods (Bai et al., 2020; Huang et al., 2021; Yu et al., 2021) in Figure 3. We select those methods as baselines since they are equipped with keypoint detector (Bai et al., 2020), overlap prediction module (Huang et al., 2021), or coarse-to-fine registration module (Yu et al., 2021), which are likely to be favorable for the unbalanced point cloud registration. For a fair comparison, we finetune all baseline methods our KITTI-UPP dataset, where the baselines were pretrained with KITTI odometry dataset (Geiger et al., 2012). All details of the training and evaluation setup for the baseline models can be found in Appendix. As shown in Figure 3, our method achieves the best results in all metrics. Moreover, our method achieves the best generalization ability w.r.t the various spatial extent and density imbalances as shown in Figure 3, indicating the advantages of our method. To verify the effectiveness of UPPNet, we conduct additional experiments; incorporating retrieval method with baseline models. In this experiment, we combine pairwise registration methods with MinkLoc3Dv2 (Komorowski. 2022) that shows best performance on 3D retrieval task. For fair comparison, We train the MonkLoc3Dv2 (Komorowski, 2022) on our KITTI-UPP dataset for 200 epochs. Even though pairwise registration methods benefit from MinkLoo33Dv2 (Komorowski, 2022), UPPNet still outperforms baseline models by large margins. We conclude that the method of generating global descriptor in MinkLo:3Dv2 (Komorowski, 2022) is not suitable for our challenging scenario which is under extreme imbalance in terms of spatial extent and point density; aggregating the features of all points rather than reliable points in contrast to UPPNet.
Indoor Experiment. We report unbalanced point cloud registration results on large-scale indoor environments, e.g., ScanNet (Dai et al., 2017). In this experiment, compare our UPPNet with CoFiNet (Yu et al., 2021). We take two methods of ours and CoFiNet pretrained on 3DMatch (Zeng et al., 2017) and finetune Sea Registration results are summarized in Table 1; our method
8

---

Under review as a conference paper at ICLR 2023
Table 2: Ablation study on (Left) k-nearest-neighbors, (Right) submap proposal and structured
matching modules. The range value and hop value are set to 500 (p. 11.1) and 23 (pp = 11.7).
#k
Recall (%)
TE (m)
RF (F(()
IR (%)
Submap
Structured
Recall (%)
super point
point
65.5
0.691
1.613
2
75.3
0.675
1.599
70.2
X
80.0
78.5
0,668
1.477
22.0
74.5 x
8
81.5
0.602
1.379
19.7
73.8
16
82.5
0.612
1.314
16,9
V
x 76.4
32
81.5
0.602
1.327
14.0
82.5
achieves 21.1% Registration Recall improvements over CoFiNet, which reveals the robustness of our method for indoor unbalanced point cloud registration as well.
4.3 ANALYSIS
To study the effectiveness of the proposed UPPNet, conduct extensive ablation experiments and report the results in Table 2. In Table 2. we conduct an ablation study on the core design choices of method: k-nearest-neighbor values and usage of submap proposal and structured matching modules. As shown Table 2, k = 16 is the optimal configuration for considering all metrics. In addition, we found that both submap proposal and structured matching modules bring significant improvement in registration recall, where the best performance is achieved when all modules are enabled.
Moreover, we evaluate our method on three popular benchmarks, namely 3DMatch (Zeng et al., 2017), 3DtoMatch (Huang et al., 2021), and KITTI (Geiger al., 2012), to demon- strate the applicabiliity of the proposed method the balanced but partially overlapping setup. We compare our method with the state-of-the- based registration meth- ods (Bai et al., 2020; Choy et al., 2019b; Yu et al., 2021; Huang et al., 2021) on 3DMatch and 3DLoMatch benchmarks. For the KITTI odometry benchmark, we select HRegNet (Lu et al., 2021) and CoFiNet (Yu et al., 2021) as base- lines since they exhibit strong performance on KITTI benchmark (Geiger al., 2012). For these experiments, we follow the training and evaluation settings of Yu et al. (2021). As reported in Table and Figure our method exhibits competitive results from the previous state-of-the oofte-o--he-rrt art registration methods on both indoor and outdoor datasets, even with the extremely low overlap scenario. This result suggests that our method not specialized for the KITTI-UPP benchmark but also applies to the indoor and low overlap environment, For more details on training and evaluation for the 3DMatch dataset, please refer to the Appendix.
Table 3: Model runtime comparisons on KITTI- UPP dataset. Range and hop values are the same with Table
Recall (%)
Time (s)
D3FFat (Bai et al,, 2020)
22.9
0.44
Predator (Huang et al., 2021)
36.7
0.19
art ceature-matchinn
CoFiNet Yu et (2021)
62.9
0.62
Ours(k=4) Ours(k=16)
78.5
1.03
82.5
2.28
Finally, we measure the latency of our method and report the breakdown of elapsed times for each module on the KITTI-UPP benchmark in comparison with CoFiNet (Yu et al., 2021). As reported in Table 3, our method takes 0.4 seconds more than CoFiNet but improves registration recall by 15.6%.
5 CONCLUSION
In this paper, we presented a neural architecture for unbalanced point cloud registration with extreme spatial scale and point density discrepancy. We proposed a hierarchical framework that finds inlier correspondences effectively by gradually reducing search space to tackle this problem. Our proposed method can handle scale differences by finding subregions that are likely to overlapped with the query point cloud and estimating the correspondences via coarse-to-fine to-fine matching module to the selected subregions. Finally, structured matching applied to prune the noisy correspondences further. Our method outperforms the state-of-the-art methods by large margin in extensive experiments the challenging dataset.
Limitations. We use RANSAC get the final rigid transformation parameter with the estimated cor- respondences. Leveraging differentiable model estimator, e.g., weighted Procrustes method (Choy al., 2020a), would be an interesting future research direction.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Relja Arandjelovic and Andrew Zisserman. All about vlad. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 1578-1585, 2013. 3
Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5297-5307, 2016. 3
Xuyang Bai, Zixin Luo, Lei Zhou, Hongbo Fu, Long Quan, and Chiew-Lan Tai. D3feat: Joint the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 6359-6367, 2020. 1, 2, 8, 9, 16, 17, 18
learning of dense detection and description of 3d local features. In Proceedings the
Xuyang Bai, Zixin Luo, Lei Zhou, Hongkai Chen, Lei Li, Zeyu Hu, Hongbo Fu, and Chiew-Lan Tai. Pointdsc: Robust point cloud registration using deep spatial consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15859 15869, 2021. 3,
Paul Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pp. 606. International Society for Optics and Photonics, 1992. 16
Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother. DDaa-diiffernniable ransac for camera localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6684-6692, 2017.
Zetao Chen, Adam Jacobson, Niko Sünderhauf, Ben Upcroft, Lingqiao Liu, Chunhua Shen, lan Reid, and Michael Milford. Deep learning features at scale for visual place recognition. In 2017 IEEE International Conference on Robotics and Automation (ICRA). pp. 3223-33 2223-3230. IEEE, 2017. 3
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3075-3084, 2019a.
Christopher Choy, Jaesik Park, and Vladlen Koltun. Fully convolutional geometric features. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8958-8966, 2019b. 1, 2,8.9, 15, 16, 17
Christopher Choy, Wei Dong, and Vladlen Koltun. Deep global registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2514-2523, 2020a. 3, 9
Christopher Choy, Junha Lee, Rene Ranftl, Jaesik Park, and Vladlen Koltun. High-dimensional convolutional networks for geometric pattern recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020b. 1,3
Ondrej Chum and Jiri Matas. Matching with
sample consensus. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), volume 1, pp. 220-226. IEEE, 2005. 3
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 5828-5839, 2017. 2, 7,
Haowen Deng, Tolga Birdal, and Slobodan Ilic. Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors. In Proceedings of the European Conference on Computer Vision (ECCV), pp.
2018a. 8a.
Haowen Deng, Tolga Birdal, and Slobodan Ilie. Ppfnet: Global context aware local features for robust 3d point matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 195-205, 2018b. 2
Juan Du, Rui Wang, and Daniel Cremers. Dh3d: Deep hierarchical 3d descriptors for robust large- scale 6dof relocalization. In European Conference on Computer Vision, pp. 744-762. Springer, 2020. 1,3
10

---

Under review as a conference paper at ICLR 2023
Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381-395, 1981. 2, 16, 17, 18
Andrew W Fitzgibbon. Robust registration of and 3d point sets. Image and vision computing, 21
(13-14::1145-1153, 2003. 3
Andreas Geiger, Philip Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. 2012 IEEE Conference Computer Vision and Pattern Recognition, pp. 3354-3361, 2012. 2, 7, 8, 9, 15,
Reza Hoseinnezhad and Alireza hab-Hadiashar. An m-estimator for high breakdownroblle estimation in computer vision. Computer Vision and Image Understanding. (15(8):1145-1133 2011. 3
Shengyu Huang, Zan Gojeie, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler. Predator: Registration of 3d point clouds with low overlap. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4267-42766 June 2021. 1, 2, 5, 8, 16, 17, 18, 19
Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez. Aggregating local descriptors into a compact image representation. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3304-3311. IEEE, 2010.
A. E. Johnson and M. Hebert. Using spin images for efficient object recognition in cluttered 3d scenes. PAMI, 21(5), 1999. 2
Jacek Komorowski. Minkloc3d: Point cloud based large-scale place recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications Computer Vision, pp. 1790-1799, 2021. 1, 3,
Jacek Komorowski. Improving point cloud based place recognition with manking-based loss and large batch training. arXiv preprint arXiv:2203.00972. 2022. 8
Junha Lee, Seungwook Kim, Minsu Cho, and Jaesik Park. Deep hough voting for robust global registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 1,
Zhe Liu, Shunbo Zhou, Chuanzhe Suo, Peng Yin, Wen Chen, Hesheng Wang, Haoang Li, and Yun-Hui Liu. Lpd-net: 3d point cloud learning for large-scale place recognition and environment analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2831-2840, 2019. 1,3
Fan Lu, Guang Chen, Yinlong Liu, Lijun Zhang, Sanqing Qu, Shu Liu, and Rongqi Gu. Hregnet: A hierarchical network for large-scale outdoor lidar point cloud registration. In Proceedings the IEEF/CVF International Conference on Computer Vision, pp. 16014-16023, 2021. 3, 6, 9, 16, 17, 18
Kwang Moo Yi, Eduard Trulls, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua. Learning to find good correspondences. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2666-2674, 2018. 3
Jaesik Park, Qian- Yi Zhou, and Vladlen Koltun. Colored point cloud registration revisited. In Proceedings of the IEEE international conference on computer vision, pp. 143-152, 2017. 17, 18
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-peeformance deep learning library. Advances in neural information processing systems, 32, 2019. 16
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017. 2,3

---

Under review as a conference paper at ICLR 2023
Filip Radenović, Giorgos Tolias, and Ondiej Chum. Fine-tuning emn image retrieval with no human annotation. IEEE transactions on pattern analysis and machine intelligence, 41(7):1655- 2018. 5
Rahul Raguram, Ondrej Chum, Mare Pollefeys, Jiri Matas, and Jan Jan-Michael Frahm. Usac: universal framework for random sample consensus. IEEE transactions on puttern analysis and machine intelligence, 35(8):2022-2038, 2012.
R. B. Rusu, N. Blodow, Z. C. Marton, and M. Beetz. Aligning point cloud views using persistent feature histograms. In IROS, 2008.
R. B. Rusu, N. Blodow, and M. Beetz. Fast point feature histograms (FPFH) for 3D registration. In ICRA, 2009. 2, 16
S. Salti, F. Tombari, and L. di Stefano. SHOT: Unique signatures of histograms for surface and texture description. Computer Vision and Image Understanding, 125, 2014. 2
Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymezyk. From coarse to fine: Robust hierarchical localization at large scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12716 12725, 2019. 3
Paul-Edouard Sarlin, Daniel Donone, Tomasz Malisiewiez, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4938-4947, 2020. 5,
Torsten Sattler, Akihiko Torii, Josef Sivic, Mare Pollefeys, Hajime Taira, Masatoshi Okutomi, and Tomas Pajdla. Are large-scale 3d models really necessary for accurate visual localization? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637-1646, 2017. 3
Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343-385 348, 1967. 6
Hugues Thomas, Charles R Qi. -mmmanell Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6411-6420, 2019.
4, 16.6 19
Giorgos Tolias, Ronan Siere, and Hervé Jégou. Partieular object retrieval with integral max-pooling of cnn activations. arXiv preprint arXiv: 2015. 5
Federico Tombari, Samuele Salti, and Luigi Di Stefano. Unique signatures of histograms for local surface description. In European conference on computer vision, pp. 356-369. Springer, 2010. 2
Mikaela Angelina Uy and Gim Hee Lee. Pointnetvlad: Deep point cloud based retrieval for large- scale place recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4470-4479 2018 8. 1,3
Huan Yin, Xiaqing Ding, Li Tang, Yue Wang, and Rong Xiong. Efficient 3d lidar based loop closing using deep neural network. In 20/7 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 481 486. IEFF, 2017.
Huan Yin, Li Tang, Xiaqing Ding, Yue Wang, and Rong Xiong. Locnet: Global localization in 3d point clouds for mobile vehicles. In 2018 IEEE Inielligent Vehicles Symposium (IV), pp. 728-733. IEEE, 2018.
Huan Yin, Yue Wang, Xiaqing Ding, Li Tang, Shoudong Huang, and Rong Xiong. 3d lidar-based global localization using siamese neural network. IEEE Transactions on Intelligent Transportation Systems, 21(4):1380-1392 2019.
Hao Yu, Fu Li, Mahdi Saleh, Benjamin Busam, and Slobodan Ilc.. Cofinet: Reliable coarse-to-fie correspondences for robust pointcloud registration. Advances in Neural Information Processing Systems, 34, 2021. 1, 2, 3, 4, 5, 6, 8, 9, 14, 16, 17, 18, 19
12

---

Under review as a conference paper at ICLR 2023
Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In CVPR, 2017. 2, 16
Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao. Learning two-view correspondences and geometry using order-aware network. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5845-5854, 3
Wenxiao Zhang and Chunxia Xiao. Pcan: 3d attention map learning using contextual information for point cloud based retrieval. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattem Recognition, pp. 12436-12445, 2019. 1,3
Qian Yi Zhou, Jaesik Park, and Vladlen Koltun. Fast global registration. In Eur. Conf. Comput. Vis., pp. 766-782. Springer, 2016. 16, 17, 18
13

---

Under review as a conference paper at ICLR 2023
A APPENDIX
In this supplementary material, we provide the detailed algorithm of the structured matching procedure in Section A. the description of how we form the data splits of our KITTI-UPP dataset for training and evaluation in Section A.2, additional details on the experiments in Section A.3, the additional quantitative results in Section A.4 with the implementation details of baseline methods, equations for the loss terms in Section A.5, details of architectural configuration in Section A.6, and finally qualitative results in Section A.7.
A.1 STRUCTURED MATCHING
We provide additional information on our structured matching. As in Section3.4 44, the initial corre- spondence set is extracted from similarity matrix Z. Simple strategy is to consider an entry (i, j)
with high confidence score in as a valid correspondence as in (Yu et al., 2021):
= ((i.)) > Tz}
(7)
However, this strategy is prone to missing inlier correspondences in our unbalanced setting; that is difficult to extract consistent features from the same region of the map and query as the inlier correspondences often have low confidence scores. An alternative solution is to select confident correspondences relatively for each point rather than using an absolute threshold value. We utilize K-nearest-neighi neighbor (KNN) search in feature space and retrieve the top-k candidate correspondences for each point having the highest inlier confidence values:
C {(i,,)) ((i,))) Topk (2(1,))
(8)
This approach can guarantee select k candidates for each point even if their confidence scores are low. Although more inlier correspondences can be obtained by KNN search, selecting extra correspondences leads a high number of outliers. To cope with these challenges, we combine spatial compatibility with KNN search for outlier rejection. We first check whether each correspondence satisfies spatial compatibility with others, as illustrated in Figure 4(b). Assume that we have N correspondences [Z yn ) where Tn. Un
R is pair of 3D points, i.e., n-th correspondence. R N.N encodes relative distances of the correspondences such that S.j 1 1[]((z,, 2;) d(y,, y₁)l < 8] where @ is distance threshold; score of 1 is assigned to S if a pair of correspondences j are spatially consistent and scores of 0 is given otherwise. In this toy example, {cl, c3, c7) are inlier correspondences and {c2, c4, c6, c8) are outlier correspondences. Although we can reject c6 by sampling the correspondences with at least one compatible correspon- dence, we still suffer from the outliers {c2, c4, c8). Notably, large-scale map would have numerous repetitive structures, making it challenging to distinguish hard negative correspondences {c2, c4, c8} using only their associated features, i.e., through first order matching. Instead, we can select corre- spondences with high compatibility scores, as described in Section 3.4, by counting the number of compatible correspondences for each correspondence. Furthermore, we can calculate computibility coherence between two correspondences by calculating how many correspondences are compatible with both of them. The calculation of compatibility coherence scores of some correspondence i is formulated similarities (dot- (rotupr between spatial consistency of correspondence and those of other correspondences: C, = SS.. E R.V which counts the number of spatially-conse consiisent matches that correspondence i has in common with others. In other words, compatibility coherence score of correspondences and j amounts to the number of spatially-consister in common. As an specific example, correspondences cl and c3 in Figure 4(a) have two common apatiall-coonssseen correspondences of c5 and c7 which makes C1,3 = C3,1 = 2 while correspon- dences c6 and c8 do not have any common apatially-coniisten correspondences so C6 = Cs.6 O. In our experiments, we use the compatibility coherence because it shows better performance in Table We can get compatibility coherence by simply multiplying the compatibility matrix in Figure 4(b). Then, we compute the average coherence value per correspondence. Finally, thresholding these values with indicates which correspondences to keep for the final correspondence set {cl, c3, c5, c7). In our case, we set this threshold as the mean value of the average coherence values, as shown in Figure 4(c). As a result, by combining this strategy with KNN search, we firstly can obtain as many inliers possible and efficiently filter out the outliers through structured matching. Experimentally. combining these two approaches is the key to solving the noisy correspondence problem caused by feature ambiguity common in unbalanced point pairs.
The spatial compatibility matrix
correspondences they have
14

---

Under review as a conference paper at ICLR 2023
0
0.25
O
0
2
0
0
2
O
0.75
0.25
2: 0.46
U
O
0
2
0
2
0
0
0
0.75
U
0
0
0
0.0
Aug
0.75
CH
0
0
0
0.25
(b)
(c)
Figure 4: (a) Example of initial correspondences where green and red lines indicate inliers and outliers. (b) Spatial compatibility matrix. (c) Compatibility coherence matrix.
Table 4: Ablation study on compatibility matrix of score and similarity. The range value and hop value are 500 (p. 11.1) and 25 (pp 11.7).
Scale Imhalance Factor (p.):
11,1
Recall (%)
(m)
(E)
IR(%)
Ps
Ours + compatibility score Ours compatibility coherence
81.1
0.553
1.377
16.3
11.7
82.5
0.612
1,314
16.9
A.2 DATASET
We follow the data split strategy that FCGF (Choy al., 2019b) uses for KITTI odometry dataset (Geiger et al., 2012) for our KITTI-UPP dataset. We use sequences (I5 for training, 6-7 for validation, and 8-10 for testing. For building the input point cloud pair, the query point clouds are sampled from the original frames of the KITTI odometry dataset, and the map point clouds are created by aggregating the LiDAR frames while managing two parameters: range and hop as described in Section 4.1. During training, we set the range and hop value to 300 and 10 to build maps. The query frames are at least 10m apart from each other, and we pick the map which the closest one to each query among the generated map point clouds. For evaluation, evaluate the generalization ability the methods. Especially for the test split, we carefully designed the input point cloud pairs that the query would not be included in the frames that are used to generate the map. Through this procedure, we yield 1,358 pairs for training, 180 for validation, and 275 for testing.
use varying range and hop values to
15

---

Under review as a conference paper at ICLR 2023
A.3 ADDITIONAL DETAILS ON THE EXPERIMENTS
Evaluation metric. We use the standard metric to assess the pairwise registration accuracy using Rotational Error (RE): arccos Tr (R and relative Translational Error (TE): |t - i|]2, where R. are the predicted rotation matrix and translation vector, R, E are the ground-truth. We also use relative Inlier Ratio (IR) and Registration Recall (RR) for the evaluation. IR is defined as the ratio of correspondences whose geometric distances are below the predefined threshold (T)) when transformed with the ground-trute transformation. For a registered pair having RE and TE less than the predefined thresholds (TR, Te). regard it successful registration and calculate the RR of successful registration over the entire dataset.
2
method with PyTorch (Paszke al., 2019) and KP- Conv (Thomas et al., 2019) for efficient 3D kerne--point convolution. We use a U-shaped network equipped with an encoder and decoder, with three layers of KPConv and KPConv transposed layers. The feature dimensions of the point feature and super point feature are 32 and 256, respectively. Input point clouds are downsampled with a Im voxel size on KITTI-UPP, 30cm on KITTI (Geiger et al., 2012), and 2.5cm on 3DMatch (Zeng et al., 2017) and 3DLoMatch (Huang et al., 2021). We set Ag >p L. and train the network for 200 epochs with Adam optimizer and initial learning rate of le-4. On inference time, k is set to 16 for KNN search. The detailed architectural configuration can be found in Figure 7. All experiments are performed on a single Nvidia Tesla V100 GPU and Intel Xeon Gold 6230R CPU @ 2. 2.10GHz.
Implementation details. We implement
Pairwise registration on 3DMatch and 3DLoMatch. To evaluate our method on 3DMatch and 3DLoMatch dataset, we use the checkpoint of CoFiNet (Yu et al., 2021) pretrained on the 3DMatch dataset since the network architecture of our approach for the super point and point description is compatible with CoFiNet
Following the recent literature (Choy et al., 2019b; Bai et al., 2020; Huang et al., 2021; Yu et al., 2021), use three metrics to assess the registration performance of the methods:
Registration Recall (RR): The ratio of successful registration over the entire dataset. We consider the registration with RMSE; 0.2 (m) as successful for 3DMatch and 3DLoMatch datasets.
Inlier Ratio (IR): The fraction of correct correspondences among the estimated correspon- dence set. We consider correspondences matched with smaller than 10cm residual correct ones.
Feature Matching Recall (FMR): The fraction of point pairs with an inlier ratio higher than the predefined value over the entire dataset. We use 5% as the threshold for 3DMatch and LMtchh datasets.
A.4 ADDITIONAL QUANTITATIVE RESULTS
More baselines. We provide the additional comparison of our UPPNet with the state-of-the- art pairwise registration methods (Bai et al., 2020; Huang et al., 2021; Yu et al., 2021; Lu et al., 2021) and classical methods (Zhou et al., 2016; Fischler & Bolles, 1981) in Table 5 and Table 6 with varying spatial and point imbalance factors. To compare with classical methods, we evaluate RANSAC (Fischler & Bolles, 1981) and FGR (Zhou et al.. 2016) equipped with classical descriptor FPFH (Rusu et al., 2009) on our KITTI-UPP dataset. For RANSAC, we report the results with 4M iterations. They mostly fail on our challenging dataset. We further refine the registration results of both methods by applying post-processing with ICP (Besl & McKay, 1992). For state-of-the-ane methods, we select Predator (Huang et al., 2021), CoFiNet (Yu et al., 2021), D3Feat (Bai et al., 2020), HRegNet (Lu et al., 2021) for the baseline methods as they are likely to be favorable for the unbalanced point cloud registration. For a fair comparison, we train the Predator, CoFiNet and ours for 200 epochs our KITTI-UPP dataset. For D3Feat and HRegNet, the pretrained model trained on the KITTI Odometry dataset with 200 epochs is finetuned on our KITTI- UPP dataset with additional 50 epochs.
16

---

Under review as a conference paper at ICLR 2023
Table 5: Evaluation results on KITTI-UPP benchmark under various point imbalance factors (pp) and scale imbalance factors (p,). We evaluate the methods by changing hop value while we fix range value to (a) 100 (p.- 2. 2.7), (b) 200 (p.- = 4.7).
Seale Inshulance Factor 2.7
Aug. Scale Inbulance Factor 4.7
Py.
(E(m)) RE IR/R)T
TB:
RE(") IRC))
(2016)
4.4
1,009
FGR Zhou el (2016)
0.660
PGR-ICP et al. (2017)
11.112
1.661
0.0
HUR Park (2017)
0.713
2.541
RANSAC Fischler& Rolles (1981)
17.5
0.646
0.0
RANSAC Fischler & Bolles (1981)
1.203
2,095
0.0
RANSACHICP Park et all (2017)
28.4
0.500
1.347
0.0
RANSAC+ICP Piek ct (2017)
0.637
2,030
[[RegNet La al. (2.02))
23
5.1
0.500
1.456
1.5
IRRegNe! Lu etal. (2021)
4.5
0.7
0.225
1.335
0.2
HRogNet Lu al. (2001)
1.8
1.309
1.593
0.6
Hitaghet La et al. (2021)
0.11
D3Feat Bai al. (2020)
80.0
0..09
1.252
8.9
D3Feat Bai et al. (2020)
57.5
0.753
1.717
5.4
Piodater Husig ct (2021)
76.0
0.463
1942
11.7
Produte Huang et al. (2021)
54.9
0.530
1.409
6.9
CoFiNet Yo ala (2021)
92.4
0.440
0.981
13.5
CoffiNet eu al. (2021)
61.1
0.576
1.334
12.0
0.359
0.762
Our
94.2
20,6
PGR Zhou (2016)
18
0.741
1.607
0.0
FGR Zhou al. (2016)
0.829
0.0
FGR-ICP
(2017)
3.6
2.312
FOR+ICP
0.552
RANSAC Pischler Bolles (1981)
4.7
1.047
1.689
0.0
RANSAC Fischler Bolles (1961)
1.100
2,035
0.0
RANSAC+ICP Park ct (2017)
17.8
0.502
1.537
0.0
RANSAC+CCP ct (2017)
0.872
2,006
0.0
HRRggNei et al. (2021)
HReyNel La et (2021)
1.5
1.5
0.397
1,073
0.454
2.341
0.5
0.4
HRagNee Lu ct (2021)
HRigNet et (2021)
69
0.4
0.4
0.991
1.971
1,150
3,969
0.1
0.1
D3Feat Bai al. (2020)
74.6
0.689
1.697
7.
D3Feat Bui et al. (2020)
46.0
0.915
2,084
4.3
Piodater Husin ot (2021)
69.5
0.544
1.221
8.8
Produce Huang ct (2021)
56.4
0.631
1.516
5.4
CoFiNet Ya al. (2021)
88.4
11.551
1.289
13.1
CoFiNet Yo al. (2012)
77.5
0.671
1.673
12.0
Ours
97.8
0.407
0.590
21.4
Oun
93.1
0.518
1,132
20.2
FGR Zhou (2016)
1.8
018339
1.655
0.0
FUR Zhou (20116)
PGR-ICP et al. (2017)
3.6
01.225
2,272
0.0
PRRERP Park (2017)
0.409
2.626
0.0
RANSAC Fischler Rolles (1981)
3,6
0.568
1813
RANSAC Fichler Rulles (1961)
RANSAC+ECP
(2017)
0.765
1.767
RANSAC+ICP
(2017)
0.912
HRogNNe La et al. (2021)
1.4 01.569
0.690
0.7
HRagNee! Lu et (2021)
922
0,4
1.186
1.618
0.1
HRRegNet Lu et al. (2021)
1.5
1,495
1.731
0.2
HRoggNet Lu al. (2021)
0.0
D:FForl Bai ct all (2020)
58.9
0.747
1.82
6.3
D3Fcat Bai ct (2020)
$1.3
0.926
2.152
3.5
Predator Uluang al. (2021)
63.6
0.552
1.398
7.5
Predasee Uuung et al. (2021)
52.7
0.758
1.787
4.5
CoFiNet Ya al. (2022)
86.5
116235
1.339
12.8
CoFiNet Ya et al. (2011)
70.1
0.698
1.635
11.4
Ours
95.6
11412
0.902
21.7
Ous
90.5
0.515
1.005
19.2
FGR Zhou all (2016)
1.1
1.287
1.598
0.0
FOR Zhou al. (2016)
0.0
FGR-ICP al. (2017)
2.9
0.39%
2,349
0.0
FGR+ICP Park (2017)
0.728
2.659
0.0
RANSAC Fiscler Rolles (1981)
1.5
1.307
RANSAC Fachler Bolles
RANSACHCCP
al. (2017)
0.714
1918
RANSACHICP
(2017)
HRogNet La et (2071)
6.9
1.5
0.529
0.673
0.5
HRegNe! et (2021)
13.0
[RRegNet Lu et. (2021)
0.0
HRRggNe Lu al. (2021)
0.0
Dal Bai et (2020)
35.6
0.870
1.891
4.3
D3feat et (2020)
15.6
1.086
2.165
2.2
Predator Husag ct (2021)
46.9
0.60%
1.000
5.2
Producee Huang et (2021)
33.8
0.727
1.919
3.0
CoFiNe: Yo al. (2021)
76.0
0.706
486
113
CoFiNet Ya al. (2001)
56.0
0.806
1.721
9.6
Ours
94.5
0.494
1.004
20.4
Ous
84.4
0.595
1.183
16.8
(a)
(b)
More results with varying imbalance factors. To evaluate the generalizability, we compare our UPPNet with the baseline methods under various scales and densities in Table 5 and Table 6. In Table 5a, we report the results with the fixed range value 100 under different hop value 25 (pp 2.3), 10 (pp = 3.7), 5 (pp = 4.8), and 1 (0p = 6.9). Note that the range value 100 with hop value 25 is the least challenging data which is most similar to balanced dataset e.g., KITTI odometry dataset. Likewise, the results in Table 5b, 6a, 6b are reported under various hop value 25, 10, 5, and 1 with fixed range value 200, 300, and 400, respectively. As shown in Table 5a-6b, performance of D3Feat (Bai et al., 2020) and CoFiNet (Yu et al., 2021) drops dramatically as the spatial extent and density of map become larger. On the other hand, our UPPNet can maintain the high performance at all evaluation metrics and outperform the baseline methods by a large margin, with robust structured matching and our hierarchical framework.
Registration results with different overlap ratios. To verify the robustness of our method with respect to the overlap ratio between query and map, we conduct experiments under varying overlap ratios well as in presence of spatial and density imbalance. In this experiment, we set the range and hop values to 100 and 25 respectively. Following FCGF (Choy et al., 2019b), each pair is at least 10m apart. We compute overlap ratio for each pair and assign each pairs to three different groups; pairs with < 70%. 70-85%, and > 85% overlap, and denote each group as low, mid, and high overlaps respectively. The results are shown in Figure 5 well as the results of two basclines methods, CoFiNet (Yu et al., 2021) and Predator (Huang et al., 2021) As shown in the figure, our method consistently outperforms other baselines by large margin for all overlap ratios of low, mid, and high, gaining notable registration recall improvement of 26% over CoFiNet. We note that our method is not only effective presence of spatial density imbalance but it is also robust under different overlap ratios.
17

---

Under review as a conference paper at ICLR 2023
Table 6: Evaluation results on KITTI-UPP benchmark under various point imbalance factors (pp) and scale imbalance factors (p,). We evaluate the methods by changing hop value while we fix range value to (a) 300 (p.- 6.8), and (b) 400 (p. 9.0).
Seall Inhhllance Factor (m)) RE
Scale Inbalance
My
IRCS)
(2016)
PGR-ICP (2017)
HUR Park (2017)
RANSAC Fischler Rolles (1981)
0.7
1.425
1714
00
RANSAC Fischler Bolles (1981)
1.102
1835
0.0
RANSACHICP
(2017)
5.1
0.920
23355
0.0
RANSAC+ICP
(2017)
0.800
HRegNet Lu (2021)
[[RegNet La et. (2001)
7.0
0.0
0.0
HitagNet
IRRggNe! Lu etall (2021)
(2023)
89
0.11
0.0
D3Feat Bai (2020)
43.3
0.870
1.366
4.3
D3Feat Bai et (20200)
52.7
0.990
2,025
3.3
Piodator Huseg ct (2021)
49.1
01634
166
5.1
Produte Huang et (2021)
58.9
0.686
1.732
3.8
CoFiNet Yo (2021)
76.4
0.715
1.555
11.6
ColfiNet eu (2021)
67.6
0.737
1.787
11.3
1.1%6
(2016)
0.0
FGR
(2016)
FGR-ICP
(2017)
0.0
FOR+ICP
RANSAC Bolles (1981)
0.7
1.921
RANSAC Fischler Bolles (1961)
RANSAC+ICP
(2017)
2.5
0.640
2.362
0.0
KANSAC+CCP
(2017)
0.931
HRogNel La (2021)
0.0
HRogNee Lu et (2021)
13.5
0.11
HRRggNei en (2021)
0.0
HRagNet et (2021)
0.0
D3Feat Bai (2020)
31.3
0.902
2.162
3.1
D3Feat Bai et (2020)
19.6
1.001
2.148
2.6
Piodator Husing ot (2021)
45.5
(.721
1,859
4.1
Produnce Huang ct (2021)
54.5
0.764
1841
3.1
CoFiNet Ya al. (2022)
64.7
01.34
1.118
11.1
CoFiNet Ya (2022)
58,9
0.782
19990
10.3
Ours
85.1
0.592
1.375
18.0
Oun
79.6
0.608
1,291
168
PGR-ICP (2016)
0.0
HUR Zhou (20116)
(2017)
0.0
PRRERC Park
RANSAC Fischer Rolles
0.4
1.424
20077
RANSAC Fichler Rollex
RANSAC+ECP
(2017)
2.9
2213
RANSAC+ICP
HRog/Ne La et (2021)
HRRggNet Lu (2021)
0.0
0.0
HiRggNe! Lu et (2021)
HiRogNei Lu et (2021)
18.8
0.01
0.0
D:FForl Bai ct (2020)
21.8
0.875
2.119
2.4
DaFca Bai ct (2020)
13.5
1.127
2,056
188
Predator Uluang et (2021)
35.6
0.687
1.787
3.2
Predasee Uuung et al. (2021)
25.1
0.786
1.938
2.2
CoFiNet Yu et (2022)
61.1
0.795
1,395
10.4
CoFiNet Ya al. (2012)
52.4
0:837
1.929
9.6
Ours
81.
0,611
1.266
16.7
Ous
78,5
0.617
1,333
15.2
FGR Zhou all (3016)
0.0
FOR Zhou al. (2016)
0.0
FGR-ICP
(2017)
0.0
FGR+CCP Park
RANSAC Fiscler Rolles
0.0
RANSAC Fachler Ballss
RANSAC-CCP
(2017)
1.5
1,072
2.433
RANSACHICP
(2017)
HRogNet La at. (2011)
0.0
HRegNel et (2021)
[[RegNet Lu at. (2021)
0.0
HiRggNe: Lu et (2021)
D3f Bai at. (2020)
10.9
1.080
1.349
1.
Dafeat et (2020)
1.137
18854
1.1
Predator Husag ct (2021)
18.5
0.7922
1999
2.0
Produsoe Huang et al. (2011)
13.5
0.929)
2.128
1.3
CoFiNe: Yo al. (2021)
48.0
0.569
2.121
8.2
CoFiNet Yo (2001)
42.5
0.813
2.029
6.8
Ours
70.9
0.651
1.256
13.8
Ous
71.6
0.674
1.460
12.9
(a)
(b)
Ours
CoFiNet
Predator
0.8
0.2
Low
Mid
High
Figure 5: Evaluation results on KITTI-UPP benchmark with different overlap ratios. For the experiments with different overlap ratios, we fix the range value to 100 (p. 2.3) and the hop value to 25 (pp- = 2.3).
Registration results with ground-truth submap. To validate the effectiveness of our proposed submap matching and structured matching modules, we analyze the registration performance of all methods when the ground-trutt -trth submap is provided. Note that this setting is advantageous for the
18

---

Under review as a conference paper at ICLR 2023
baseline methods since they optimized for balanced point pairs, and refer to these results as the theoretical upperbounds of the baselines methods since we can assume the combination of a highly accurate localization algorithm with the baseline methods. As shown in Figure 6, thanks to the robustness of the submap matching module in establishing reliable submap correspondences, our model without gt submaps achieves relatively small performance drops compared to others (Yu et al., 2021; Huang et al., 2021). Moreover, we highlight that the proposed structured matching effectively helps our method achieve the best results, outperforming the upper bounds of other baselines by large margins even without ground truth submaps.
A.5 Loss
In this section, we provide concise equations calculate the loss terms, C.. Lg. and Cp. As we described in Section 3.6 in our main manuscript, the point matching loss Lp is defined as:
hp Ei, ((i,j) log Z(i
(9)
Σns (i.
where 2 is the predicted similarity matrix after solving optimal transport using the Sinkhorn algorithm, 2 is the binary matrix that indicates ground-truth. For coarse scale super point and submap level matching, we follow (Yu et al., 2021) )) and calculate the overlap ratio between two super points, or submaps, calculate soft-labeled supervision. For super point matching, we calculate the overlap ratio between two super points as:
(( = E By E G.Y. GJ S.t. Ry + < Tgl
(10)
GG
where R. Ê is the groundtruth rotation and translation and T, is the predefined distance threshold. If the superpoint (i',j") is not an inlier correspondence, we assign it to the slack entry in the similarity matrix. For this, we calculate the ratio of points in Ge which are overlapped with point cloud Y:
r(t' [x E G By E Y, Y,t. RRy + +t-xx x|| Tyl
(11)
|Gpl
and finally build the groundtruth similarity matrix 2g E Rn pn''+xxm'+ +1 as:
min mninr((',, r(( r(7)
if n' +1Aj < m + if < n' + +1 = m +1
(12)
r-r
if = n' + +1 < <m +1
otherwise
Then we calculate the loss similar with point matching loss.
C, Ex (1, log (1', (")
(13)
Ev (0)))))
Note that unlike the point matching loss fp. super point matching loss is supervised with the continuous soft labeled supervision E.e Similarly, we can calculate the submap matching loss C.:
C, = -E 2.(1', log 2.,
(14)
Ev (1')5
where 2, the ground-truth similarity matrix at submap level, and it can be calculated in a similar way.
A.6 NETWORK ARCHITECTURE
UPPNet adopt a shared U-shaped network based on KPConv (Thomas et al., 2019) to extract multi- level features. Detailed architecture is demonstrated in Figure 7. Compared to CoFiNet (Yu et al., 2021), additional layers for the submap matching module are added to our network. We generate a global descriptor through Generalized Mean Pooling (GeM). As the global descriptor of the query is a single vector, the self-attention module is applied only for global descriptors of the map, such as in Figure 7.
19

---

Under review as a conference paper at ICLR 2023
Recietration Rocall
predator
8
DoFiNet wi gt
8
Ours w/
50
Density factor
Figure 6: Registration Recall in relation to (Left) point imbalance factor (pp) and (Right) scale imbalance factor (p,). The dotted lines indicate the theoretical upper bounds of each method measured by providing a ground truth submap. Our method shows the best registration recall in all settings, and the margin to the upper bound is smaller than the baselines.
Vilu aady 2000
Ik Oo
Iib Ower
ICEE Yit grou
EF: UX Rasaaiie 2P13
Ik
Resblock
Resblock (b).
Self Attention, N
Cross Attention,
Source
Paareil 54
dn 1 CoNeon
20 Pi Sopipe
21 scrgerg
Iozz 1
992 Voooaeeee
Sduccli
100
E1 S
Targes
encoder
encoder
Figure 7: Network architecture of UPPNet.
A.7 ADDITIONAL QUALITATIVE RESULTS
In Figure 8, provide additional qualitative results of our method along with the baseline methods on the KITTI-UPP test dataset.
20

---

Under review as a conference paper at ICLR 2023
(a) Input
(b) D3Feat
(c) CoFINet
(d) Ours
(e) Ground truth
Figure Registration results on KITTI-UPP dataset. (Green) A query. (Blue) map.
21


Reviews:
[{"review_id": "jOPItuwdgU", "reviewer": "Reviewer_CTpR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper defines a new task: matching unbalanced points cloud pairs in terms of spatial extent and density.\n\nThough the task is somewhat similar with registering low-overlap point cloud pairs for spatial extent, this paper adds a factor - density.\n\nThis paper proposes three steps/modules to solve the problem: 1) Submap partition and matching to find top-K submaps of the large database for the query; 2)  coarse-level super-point matching; and 3) fine-grained point matching.\n\nResults on the KITTI-UPP and several standard benchmark datasets show the effectiveness of the proposed method.\n\n\n", "summary_of_the_review": "Though this paper proposes to solve a new task, some important baseline experiments are missing [please refer to weaknesses], this brings a question -- can we combine the best practice of a state-of-the-art nearest frame retrieval method [e.g., https://github.com/jac99/MinkLoc3Dv2] and a state-of-the-art registration method [Geometric Transformer for Fast and Robust Point Cloud Registration] to solve the proposed task?", "strength_and_weaknesses": "[Strength]\n\n1. This paper studies a new registration problem;\n2. This paper is well-written and easy to follow;\n3. Experiment results are good;\n\n[Weaknesses]\n\n1. [One feasible solution..., above section 3].\nAuthors say that it is suboptimal to divide the proposed task into two subtasks : 1) the nearest frame retrieval and 2) local registration.  However, authors still follow this pipeline by first performing KNN submap retrieval and then do local registration.\nThe difference could be authors use an attention-based context aggregation module to extract better global features.\nAccording to the above analysis, I would expect the following three experiments:\n1) Substitute the proposed submap retrieval [Sec.3.3] with a state-of-the-art nearest frame retrieval method [e.g., https://github.com/jac99/MinkLoc3Dv2], and keep the remaining modules to test the proposed method;\n2) Use a state-of-the-art nearest frame retrieval method for state-of-the-art methods to find a local submap for a query, and then test the performance of state-of-the-art methods;\n3) Use the proposed submap retrieval [Sec.3.3]  for baseline methods to find a local submap for a query, and then test the performance of state-of-the-art methods;\n\nThe above three experiments (ablations) can help readers get a clear picture of the proposed method and identifying what is actually working.\n\n2. It is blurry how the proposed method is tested on the standard 3DMatch and original KITTI datasets. Are you using K=1 and skip the submap matching module?\n\n3. For comparison with respect to state-of-the-art methods (Table 1), please report the original figures of these methods. For example, \n [Predator, RR] 90.6 for 3DMatch and 62.4 for 3DLoMatch.\nFurthermore, please also compare the state-of-the-art method [Geometric Transformer for Fast and Robust Point Cloud Registration], it got 92.5 for 3DMatch and 74.2 for 3DLoMatch.\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "JF8X65APcW", "reviewer": "Reviewer_g5jK", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a neural architecture named UPPNet to solve the registration problem of a local 3D scan and the global 3D map. The proposed UPPNet mainly consists of three modules: submap matching, super-point matching, and point matching. The submap matching module aims to retrieval the most similar set of target points from the global 3D map for the source local scan. For the source local scan and the retrieved target point cloud, the super-point matching and point matching modules are progressively performed to achieve pair-wise registration.", "summary_of_the_review": "This paper has major issues in the experimental part.", "strength_and_weaknesses": "#Strenths#\nThis paper is well written and easy to follow.\n#Weakness#\nThe major faults of this paper lie in the experiment part, which can be summarized as:\n(1)\tThe starting point of this paper is inappropriate. Since this work considers the registration between a local 3D scan and a global 3D map, this experimental setup actually lies in the category of relocalization. However, no experiment is provided to compare the proposed UPPNet with existing relocalization methods.\n(2)\tThe authors conduct extensive experiments to compare the proposed UPPNet with the pair-wise registration methods. However, the experimental setup in this paper is unfair and different from the original setting of pair-wise registration. Pair-wise registration is mainly used to align a captured 3D scan with another 3D scan instead of the reconstructed 3D map.\n(3)\tThe authors mention the limitations of the paradigm of retrieval-to-registration in related work. However, there are no specific experiments to demonstration these claims. In fact, the proposed UPPNet is an end-to-end implementation of this paradigm and does not really addressed these limitations.\n\nAdditional comparative experiments are expected to demonstrate the motivation of this work, such as:\n(1) Compare the registration performance of the proposed UPPNet with relocalization methods (such as DH3D [1]) rather than pair-wise registration methods.\n(2) Conduct extensive experiments to compare the proposed UPPNet with the state-of-the-art combinations of retrieval-then-registration (such as MinkLoc3D + Geotransformer[2]).\n\nIn addition, in the implementation details in Appendix, the authors claim that the voxel size is set to 1m. I have two questions about this:\n(1) Is this setting also the same on the indoor 3DMatch/3DLoMatch dataset? On the indoor dataset, only the registrations with RMSE < 0.2m are considered for the calculation of Registration Recall. If the voxel size is set to 1m, I think it is difficult to achieve such a high RR in Table 1.\n(2) In original pair-wise registration papers, the values of voxel sizes on 3DMatch and KITTI are 0.03m and 0.3m, respectively. According to the visualization results, this setting is reasonable. But in this paper, in order to reduce memory consumption and improve computation efficiency, the voxel size is set to 1m, which not only loses a lot of geometric information, but also is extremely unfair to the pair-wise registration methods.\n\n[1] Du J, Wang R, Cremers D. Dh3d: Deep hierarchical 3d descriptors for robust large-scale 6dof relocalization. ECCV, 2020.\n[2] Qin Z, Yu H, Wang C, et al. Geometric transformer for fast and robust point cloud registration. CVPR, 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "2zJDt_8fEkg", "reviewer": "Reviewer_fyXh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new point cloud registration method considering an unbalanced pair of point clouds. The main assumption is that one of them contains much fewer points than the other and represents only a part of the scene in interest. There are several spatial levels as well as the corresponding matching modules in the proposed method. The points are downsampled to super points, which are then again grouped into several submaps. The point and super-point features are calculated by a KPConv-based network, and they are aggregated using attention to calculate those for submaps. These features are used in a hierarchical matching process comprised of submap matching, super point matching, and finally point matching. An additional RANSAC procedure is applied to the point correspondences to refine the results. Experiments show that the proposed method beats the existing methods by a large margin for a new dataset created considering the particular problem setting proposed in the paper, and it is competitive for existing datasets.", "summary_of_the_review": "This paper proposes an important problem that often arises in practical scenarios, and achieves good performance in the experiment. However, I also have a doubt about its method design (regarding the treatment of Y). Moreover, many important details are missing or confusingly described in the paper.\n\n[After rebuttal] I'm satisfied with the answers regarding my original concerns. However, I agree with g5jK regarding the fairness of the experiments, which is a non-negligible concern. Accordingly, I decrease my score to 5.", "strength_and_weaknesses": "The main motivation of the paper, unbalanced point cloud pairs, is a plausible problem setting that can be important for practical scenarios. Although many corners of the proposed method are inspired by existing techniques in the literature, they are combined in a way that the above problem can be solved effectively. The proposed method provides good performance in the experiment which is also a good point.\n\nHowever, I also think that the proposed method has a limitation in its design. It seems that Y (the query) is assumed to be almost a subset of X. Unlike X, Y has only a single submap, which is matched with various submaps of X, as if we are searching for a part of X that corresponds to Y. However, a more general situation is when X and Y are only partially overlapped, and a large part of Y is also absent in X. Of course, I see that the authors are considering unmatched super points for both directions in (6), but I still somewhat doubt the choice of having a single submap for Y. It seems that this design simplifies the problem too much. I'd like to hear the authors' justification regarding this part.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject