FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators
Authors: Haiping Wang
Yuan Liu
Bing WANG
YUJING SUN
Zhen Dong
Wenping Wang
Bisheng Yang
Venue: ICLR 2024 poster (2024)
Keywords: Image-to-point cloud registration
cross-modality feature extraction
diffusion models
Primary area: visualization or interpretation of learned representations
TL;DR: FreeReg extracts cross-modality features from pretrained diffusion models and monocular depth estimators for accurate image to point cloud registration.

Abstract: Matching cross-modality features between images and point clouds is a fundamental problem for image-to-point cloud registration. However, due to the modality difference between images and points, it is difficult to learn robust and discriminative cross-modality features by existing metric learning methods for feature matching. Instead of applying metric learning on cross-modality data, we propose to unify the modality between images and point clouds by pretrained large-scale models first, and then establish robust correspondence within the same modality. We show that the intermediate features, called diffusion features, extracted by depth-to-image diffusion models are semantically consistent between images and point clouds, which enables the building of coarse but robust cross-modality correspondences. We further extract geometric features on depth maps produced by the monocular depth estimator. By matching such geometric features, we significantly improve the accuracy of the coarse correspondences produced by diffusion features. Extensive experiments demonstrate that without any task-specific training, direct utilization of both features produces accurate image-to-point cloud registration. On three public indoor and outdoor benchmarks, the proposed method averagely achieves a 20.6 percent improvement in Inlier Ratio, a $3.0\times$ higher Inlier Number, and a 48.6 percent improvement in Registration Recall than existing state-of-the-arts. The code and additional results are available at \url{https://whu-usi3dv.github.io/FreeReg/}.

Published as a conference paper at ICLR 2024
FREEREG: IMAGE-TO-POINT CLOUD REGISTRATION LEVERAGING PRETRAINED DIFFUSION MODELS AND MONOCULAR DEPTH ESTIMATORS
Haiping Wangh Yuan Liu Bing Wang
Yujing Sun?
Zhen Dongh!
Wenping Wang Bisheng Yangh:
'LISMARS & Hubei Luojia Laboratory, Wuhan University
Thhe university of Hong Kong
'The Hong Kong Polytechnic University *Texas A&M University
[hpwang, dongzherwhu, oshyang)@win edu.. edu. bingwang@pol edu. hk yjsunêcs .ecelccs.kk.... hku.hk
yuanlyeconnect. hku. wenpingê waw.nngettaeeeee.... edd
ABSTRACT
Matching cross-modality features between images and point clouds is funda- mental problem for image-to-poing cloud registration. However, due to the modal- ity difference between images and points, it is difficult to learn robust and dis- criminative cross-modality features by existing metric learning methods for fea- matching. Instead of applying metric learning propose to unify the modality between images and point clouds by pretrained large-scale models first, and then establish robust correspondence within the same modality. We show that the intermediate features, called diffusion features, ex- tracted by depth-to-image diffusion models are semantically consistent between images and point clouds, which enables the building of coarse but robust cross- modality correspondences. We further extract geometric features on depth maps produced by the monocular depth estimator. By matching such geometric fea- tures, we significantly improve the accuracy of the coarse correspondences pro- duced by diffusion features. Extensive experiments demonstrate that without any training on the I2P registration task, direct utilization of both features produces accurate image-to-point cloud registration. On three public indoor and outdoor benchmarks, the proposed method averagely achieves a 20.6% improvement in Inlier Ratio, 3.0x higher Inlier Number, and 48.6% improvement in Regis- tration Recall than existing state-of-the-arts The code and additional results are available httpt:t:///// /tttps:/wwhu-ssse
rross-modality data, we
qithub..hh.... io/FreeReq/
INTRODUCTION
Image-to-point cloud (I2P) registration requires estimating pixel-to-point correspondences between images and point clouds to estimate the SE(3) pose the image relative to the point cloud. It is a prerequisite for many tasks such Simultancous Localization and Mapping (Zhu et al. 2022 3D reconstruction (Dong et al. 2020), segmentation
Guo et al. 2020), and visual localization (Sarlin
al. 2023).
To establish pixel-to-poim correspondences, we have to match features between images and point clouds. However, it is difficult to learn robust cross modality features for images and point clouds.
Most existing methods (Feng et al.] 2019 Wang et al. 2021 [Pha et al.] 2020] Tiang & Saripalli 2022 al. 2023) resort metric learning methods like contrastive loss, triplet loss InfoCE loss to force the alignment between the 2D and 3D features of the same object. However, due to the inherent data disparities that images capture appearances while point clouds represent structures, di- rectly aligning cross- cross-modal data inevitably leads to poor convergence, Consequently, cross cross-modality metric learning suffers from poor feature robustness (Wang et al. 2021 and limited generalization
ability (Li et al. 2023
"Equal contribution.
1 "ooresponning Authors.

---

Published as a conference paper at ICLR 2024
Image Manching
FreeReg wil treining fine-tuning
Inpur
I2P
Depui Emination
Munch
Image Rendering
6a) Cross-wodallty Imput
(b) Extracted feutures (c) Extimaned conequndence
Matching
Reconsracted PC
Inpat PC
Exirting: cress-ausdel mutching
Ounc Align Cloud (PC) image unalulity
Ohrs: Aligu image point cloud medallly
(d) Freettng can handle large sienpoint change, lov overlap. and sparse point donsity.
Figure I: Left: FreeReg unifies the modalities of images and point clouds, which enables mono modality matching to build coss-modalltty correspondences. Right: FreeReg does not require any training on the 12P task and is able to register RGB images to point clouds in both indoor and outdoor scenes, even for challenging cases with small overlaps, large viewpoint changes, and sparse point density.
In this paper, we propose a novel method, called FreeReg, build robust cross- modality correspon- dences between images and point clouds with the help of recent large-scale diffusion models (Rom- bach et al. 2022 Zhang & Agrawala 2023 Mou et al. 2023) and monocular depth estimators (Bhat et al. 2023 Yin et al. (023). FreeReg avoids the difficltt cross-modality metric learning and does even not require training on the I2P task. As shown in Fig. ! the key idea is to unify the modality between images and point clouds by these large-scale pretrained models so FreeReg allows robust correspondence estimation within the same modality for cross- coos-modallty matching.
In order convert point clouds to the image modality, straightforward way is to project points onto an image plane to get a depth map and then convert the depth map to an image by a depth-to- to--tagee diffusion model ControlNet (Zhang & Agrawala 2023) However, as shown in Fig. 2 (I), depth map may correspond to multiple possible images so that the generated image from the point cloud would have completely different appearance from the input image, which leads incorrect match- ing results even with SoTA image matching methods (Sarlin et al. 2020 DeTone el al.) 2018 Sun al. p. To address this problem, we propose match the semantic features between the gen- erated images and the input image because the generated images show strong semantic consistency with the input image in spite of different appearances. Inspired by recent diffusion based semantic correspondence estimation methods (Tang et al.. 2023 Zhang et al. 2023). we utilize the interme- diate feature maps in the depth-to-image ControlNet to match between depth maps and images. As shown in Fig- 2 (II), we visualize the diffusion features of the depth map and the RGB image. Then, we utilize the nearest neighbor (NN) matcher with mutual check [Wang et al., 2022a) to establish correspondences between them. We find that such semantic features show strong consistency even though they are extracted on depth maps and images separately, making it possible to build robust mods-ttyy correspondences. However, the semantic features are related to large region of the image. Such a large receptive field leads to coarse-grained features and only sparse correspondences in feature matching.
We further improve the accuracy of our cross-modality correspondences with the help of the monoc- ular depth estimators (Bhat al. 2023). Recent progress in monocular depth estimators enables metric depth estimation on a single-view image. However, directly matching features between the point cloud and the estimated depth maps from the input image leads to poor performance as shown in Fig. 2 (III). The main reason is that the predicted depth maps are plausible but still contain large distortions in comparison with the input point cloud. The distortions prevent us from estimating robust correspondences. Though the global distortions result in noisy matches, the local geome- try of the estimated depth maps still provides useful information to accurately localize keypoints and densely estimate fine-gn fine-grained correspondences. Thus, we combine the local geometric fea- tures (Choy et al.] 2019) extracted on the estimated depth maps with the semantic features extracted from diffusion models as the cross-modality features, which enable dense and accurate correspon- dence estimation between images and point clouds, as shown in Fig. (IV).
In summary, FreeReg has the following characteristics. 1) FreeReg combines coarse- maase-grained mantic features from diffusion models and fine-grained geometric features from depth maps for accurate mross-modality feature matching. 2) FreeReg does not require training on the I2P task,
2

---

Published as a conference paper at ICLR 2024
Input Image
I. Naive: Matching input und generated images
FreeReg-De Diffision Feuture
Estimated
Inpur Point Cloud
III. FreeReg-G: Geomerric Feature
I.. FreeReg: Diffusion Feature Geometric Poature
Figure 2: To unify the modalities of point clouds (PCs) and images, I: straightforward way is generate RGB images from point clouds by depth-to-image diffussio models. However, the generated images usually have large appearance differences from the query images. II: We find that intermediate features of diffusion models show strong semantic consistency between RGB images and depth maps, resulting in sparse but robust correspondences. III: We further convert RGB images to point clouds by a monocular depth estimator and extract geometric features match between the input and the generated point clouds, yiclding dense but noisy correspondences. IV: We propose fuse both types of features to build dense and accurate correspondences.
which avoids the unstable and notorious metric learning to align local features of point clouds and images. 3) FreeReg significantly outperforms existing fully-supervised cross-modality registration baselines (Pham et al. 2020 Li et al.) 2023). Specifically, on the indoor 3DMatch and ScanNet datasets and the outdoor KITTI-DC dataset, FreeReg roughly achieves over 20% improvement in Inlier Ratio, 3.0x more Inlier Number, and 48.6% improvement in Registration Recall.
RELATED WORK
mmage-tto-pint cloud registration. In order establish correspondences between images and point
clouds for pose recovery, most existing methods (Li et al. 2015 Xing et al.) 2018 |eng et al.) 2019
Lai et al.
2021 Wang al.] 2021 Pham al. 2020 Liu et [020 Diang & Saripalli 2022
ct al.
2023 rely on metric learning to align local features of images and point clouds (Feng et al.
2019 Pham et 2020 Lai al.
2021 Jiang Saripalli 2022), , or depth maps (Liu
2020
Wang et al. 2021 Li et al. 20 However, these methods often require cross-modal registration
training data [Pham al. 2020] Wang et al.] [02T] Diang Saripalli [022] [i et al. [023 Kim
et al.
2023 and show limited generalization ability (Pham et al. 2020 Wang et al. 2021
al.)
2023
Ren et al. 2022 Yao et al. 2023) due to the difficulty in the cross- modd-modayttyy metric learning-
In contrast, FreeReg does not require any training and fine-tuning on the I2P registration task and
exhibits strong generalization ability to both indoor and outdoor scenes.
Some other methods directly solve image-to-point cloud registration
optimization prob-
lem (David et al. 2004 Campbell al.] 2019) Arar et al. 2020 Wang et al. 2023a Zhou et al.
2023 which regresses poses by progressively aligning keypoints & Lee 202 Ren al. 2022
Campbell et al. 2019). pole structures (Wang et al. [2022), semantic boundaries (Liao et al.] 2023
cost volumes (Wang al. 2023a) of RGB images and depth maps. However, these methods heavily rely on an accurate initial pose (Wang et al. 2021 Liao et al.] [0033 to escape from local minima optimizations. FreeReg does not require such strictly accurate initialization because FreeReg matches features to build correspondences to handle large pose changes.
Diffusion feature extraction. Recently, category of research (Ho et al. 2020 Song al. 2020a b)
Karras et al. 2022 Song & Ermon] 2019 Dhariwal & Nichol] 2021 Liu et al. 2023), known as dif- fusion models, has demonstrated impressive generative capabilities. Based on that, with the advent of classifier-fii free guidence (Ho & Salimans 2022) and billions of text-to-image training data (Schuh-
mann al. 2022), , a latent diffusion model, specifically stable diffusion (Rombach al. 2022) has shown remarkable text-to-image generation capabilities. Building upon this, existing methods have Stable Diffusion internal representations (Diffusion Feature) (Kwon et al. 2022 Tuuaayyan et al. [2023) in various domains such as segmentation Amit
demonstrated the exceptional performance
et al.
2021
Baranchuk et al. [021]] 22021/Cher Chen et al. a 2022b Diang et al.] 2018]] | Tan et al.)
2022 Wolleb
al.
2022 detection (Chen et al. 2022aa,, depth estimation (Duan et al. 2023 3] Saxena et al.
2023b
a). These methods only extract diffusion features on RGB images utilizing Stable Diffsuion.
Our method extracts diffusion features RGB and depth maps based on recent finetuned diffusion
3

---

Published as a conference paper at ICLR 2024
Diffusion Features of RGB
Fused Features of
Salve SE(3) Pone ee
Zoe-Depth
Geometric Features of RGB
RGB
Match &
Input image
Estimated Correspondences
Dapeh View
PC Diffusion Features PC
Fused
RGB Vlew
Features PC
Estinated SE(3)
Rolanive Pose
Geometric Features of PC
Input Point Cloud
Estimated SE(3) Relative Pose
Figure 3: FreeReg pipeline. Given a point cloud (PC) and partially overlapping RGB image, FreeReg extracts diffusion features and geometric features for the point cloud and the image. These two features fused and matched to establish pixel-to-point correspondences, which compute the SE(3) relative pose hetween the image and the point cloud.
models ControlNet (hhng & Agrawala, 2023
T2IAdaptor (Mou al. 2023 which efficiently leverage depth, semantic maps, and sketches to guide stable diffusion in image generation.
Diffusion feature for matching. Some recent works utilize diffusion features for representation learning (K et al. 2022) and semantic matching (Luo et al.) 2023a Tang et al. 2023 Hedlin
instances and cate- gories. In comparison, our method shows the effectiveness of diffusion features in learning cross- modality features for image-to-point cloud registration.
al. 2023] Zhang
2023) among RGB images capturing objects
Monocular depth estimator Monocular depth estimation inherently suffers from scale ambigu-
ity (Chen al. 2016
2020
Xian et al. 2018
2020). With more and more monocular depth training
data (Guizilini et al.
2023
Antequera et al.
2020 Wilson et al. 2023), recent works (Bhat et al.
2021
2022 Jun al.
2022
Li et [022
Yang et al. [221]] Yin et 2021 2019 Yuan al.
2022 Guizllni et al. [023]
Yin et al. 2023
learn scene priors to regress depth values real metric
space and show impressive results. employ a SoTA metric depth estimator Zoe-Depth (Bhat
et al.
2023
to recover point clouds in the same metrics corresponding to the RGB images.
METHOD
Let E RHHxxxx x3 be an RGB image and E Nxx be a point cloud. We first project P to depth map E "'''''" on camera pose, which is calculated from the depth LiDAR sensor center and orientation. More details about this projection are given in the supplementary material. FreeReg aims to match the cross-modality features extracted on / and D establish correspondences and solve the relative pose between them. The pipeline of FreeReg is illustrated in Fig, 3 Specifically, We extract diffusion features (Sec. (.2) and geometric features (Sec. (3.3) for feature matching and then estimate the I2P transformation estimation from the matching results. We begin with brief review diffusion methods, which we utilize to extract cross-modality features.
3.1 PRELIMINARY: STABLE DIFFUSION AND CONTROLNET
The proposed cross-modality features are based ControlNet Zhang & Agrawala 2023
(CN)
so we briefly review the related details of ControlNet in this section. Diffusion models contain forward process and reverse process, both of which are Markov chains. The forward process gradually adds noise to the input image in many steps and finally results in pure structure-less noise. The corresponding reverse process gradually denoises the noise sep-by-step to gradually recover the structure and generate the image. Stable Diffusion (Rombach al. 2022) (SD) is widely- used diffusion model mainly consisting
UNet which takes noisy RGB images as input and predicts the noise. The original Diffusion model only allows text-to-image generation. Recent ControlNet (Zhang & Agrawala 2023). as shown in Fig. (b), adds an additional encoder to process depth maps and utilizes the extracted depth features guide the reverse process of SD, enabling SD to generate images coherent the input depth map from a pure Gaussian noise. FreeReg, we utilize CN and SD extract moos-modaality features for feature matching.

---

Published as a conference paper at ICLR 2024
ControlNet
Densify
Intermediate feature maps at step't
Pure
Noise
Noise step't
Stable Diffusion
Stable Diffusion
interative denoising
(a) Diffusion features of RGB image
(b) Diffusion features of depth map
Layer
10
(c) Diffusion feature visualization
Figure 4: Diffusion feature extraction on (a) images and (b) depth maps. (c) Visualization of diffusion features.
3.2
DIFFUSION FEATURES ON
DATA
Directly generating image from the input depth map suffers from appearance inconsistency with the input image, which results in inaccurate feature matching. Instead of generating an explicit image,
the intermediate feature maps of stable diffusion models for cross-modality feature matching. The overview shown in Fig.
RGB diffusion feature. As shown in Fig. a). perform the forward process of SD (Rombach et al. 2022) to add noise to the input RGB image, which results in a noisy image on a predefined step The noisy image is fed to the UNet of the SD and the intermediate feature maps the UNet decoder are used as the diffusion feature for the input RGB image.
Depth diffusion feature. Given the depth maps, first densify them using traditional erosion and dilation operations (Ku et al. 2018). As shown in Fig. (b), we propose to feed the depth map to CN Zhhag & Agrawala 2023) as a condition to guide the reverse process of SD, With such
condition, SD gradually denoise pure Gaussian noise until the same predefined step and then we use the feature maps in the SD UNet decoder as the depth diffusion features. alternative way is to directly treat the depth map leads to poor performance as shown in the supplementary material.
RGB image for diffusion feature extraction, which however
Layer selection. The remaining problem is about which layer to be used for feature extraction. Visualization of extracted diffusion features on RGB images and depth maps are given in Fig. 4(c. It can be observed that the features of early upsampling layers with layer index / < 6 show strong consistency between RGB and depth data. Features of later upsampling layers with an index larger than 6 show more fine-ga ine-grained details like textures that no longer exhibit consistency. Therefore, use features of early layers 0,4,6 as our diffusion features. To reduce the feature dimension on each layer, we apply Principal Component Analysis (PCA) reduce the feature dimension 128. The resulting diffusion features of RGB image and depth map D are F and FP respectively, both of which are obtained by concatenating the features from different layers and L2 normalized.
3.3 GEOMETRIC FEATURES ON
DATA
The above diffusion feature extracted from large region on the image, which struggles to capture fine-grained local details and estimates only sparse correspondences shown in (b/e). To im- prove the accuracy these correspondences, we introduce a so-called geometric feature, leveraging the monocular depth estimator Zoe- Zoe-Depth (Bhat al.) 2023
Specifically, we utilize Lee-Depth to generate per-pixel depth D for the input RGB image / and recover point cloud from the generated depth map. Then, we employ pre-trained point cloud feature extractor FCGF (Choy et al.) 2019) to extract per-point features, which serve as the geometric features of their corresponding pixels in the image I. We construct geometric features for pixels of the depth map D in the same way. As illustrated in Fig. [5] (c/f). solely matching geometric features produces many outlier correspondences due large distortion in the single-view depth
5

---

Published as a conference paper at ICLR 2024
Features
Correspondences
(a) Image PC
(b) Diffusion
(c) Geametric
Fused
Diffussin
i
Geumetric
Fused
Figure 5: Visualization of features and estimated correspondences. (a) Input images and point clouds. (b), (c), and (d) show the visualization of diffusion, geometric, and fused feature maps respectively. (e). (f), and (g) show the pixel-to-point correspondences estimated by nearest neighbor (NN) matcher using diffusion, geometric, fused features respectively. Diffusion features estimate reliable but sparse correspondences. Geometric features yield dense matches but with more outliers. Fused features strike a balance hetween accuracy and preserving fine-grained details, resulting in accurate and dense matches.
estimation. However, these geometric features provide local descriptions of the geometry, which are more localized and enable more accurate correspondence in cooperation with the diffusion features.
3.4 FUSE BOTH FEATURES FOR I2P REGISTRATION
Fuse features. In this section, we propose to fuse the two types of features to enable accurate correspondence estimation, as shown in Fig. Note that we uniformly sample a dense grid of keypoints on both the depth map and the image. Then, we extract the above diffusion features and geometric features on the keypoints. Both features are normalized by their L2 norm before the fusion. Specifically, we follow (Zhang et al. 2023) to fuse two kinds of features on each keypoint in or D by
F= Fa (1 w))F..
fusion weight, [ means concatenating feature dimension, and F is the resulting FreeReg feature.
Pixel-to-point correspondences. Given sets of fused features F! on RGB image and FD on depth map D, we conduct nearest neighborhood (NN) matching with mutual nearest check (Wang 2022a find of putative correspondences. Note that the pixel from the depth map D in each match corresponds to 3D point in point cloud P.
Image-to-point cloud registration. To solve SE(3) poses of RGB image I relative to P. typical approach is to conduct the Perspective-n-Poin (PnP) algorithm (Lepetit et al.] (209) on the estab- lished pixel-to-point correspondences. However, have estimated depth map corresponding to RGB using Zoe- Depth (Bhat et al.] 2023). Thus, we can convert the tixel-to-poing correspondences 3D point-to-poins correspondences, and estimate the SE(3) relative pose using the Kabsch al- gorithm (Kabsch) 1976). In the supplementary material, we empirically show that using the PnP algorithm leads more accurate pose estimation but fails in many cases, while the Kabsch algo- rithm works in more cases but the estimated transformations exhibit larger errors.
EXPERIMENTS
4.1 EXPERIMENTAL PROTOCOT.
Datasets. We evaluate the proposed method on three widely used datasets: (1) The 3DMatch Zeng
al,, 2017) testset comprises RGB images and point clouds (called /2P pairs) from indoor scenes. The point clouds used here are collected by an Asus Xtion depth sensor. We manually exclude the I2P pairs with very small overlaps resulting in 1210 12P pairs with over 30% overlaps. (2) The
ScanNet (Dai et al.) 2017) testset consists of 4,660 I2P pairs from 31 indoor scenes with more than 30% overlap. To further increase the difficulty, we downsampled the input point clouds using a voxel
size of 3cm, which leads highly sparse point clouds. (3) The Kitri-DC (hrig al.. 2017 testset has 342 I2P pairs from 4 selected outdoor scenes. The sparse point clouds come from 64-line
LiDAR scan. The distance between each I2P pair is less than 10 meters.

---

Published as a conference paper at ICLR 2024
Table 1:
Cross-modality registration performance of different methods, "InvCP." means Inverse
Camera Projection (Li & Lee 2021).
Method
LCD
SG
Deepl2P InvCP.
CN+SG PnP
12P-Matr PnP
FrecReg-D PnP
FreeReg-G Kahsch
FreeReg PnP
FreeReg
SE(3) Solver
PnP
PuP
Kabsch
FMR[%)
40.1
50.3
64.7
90.6
91.9
90.7
94.6
94.6
IR(%)
35.1
11.1
18.4
24.9
39.6
31.4
47.0
47.0
3DMatch
IN(#)
4.3
10.9
49.0
60.8
49.4
82.8
82.8
RR(S))
1.8
6.5
28.2
33.2
50.4
40.0
63.8
FMR(%)
55.1
53.2
64.1
87.0
95.3
96.4
98.5
98.5
IR(%)
30.7
13.4
18.3
14.3
45.7
40.5
56.8
56.8
ScanNef
IN(#)
5.0
4.7
9.1
24.8
61.5
84.5
114.4
114.4
RK(%)
1.2
5.5
8.5
42.3
69.4
57.6
78.0
FMR(%)
73.4
94.2
100.0
94.4
99.7
99.7
IR(%)
18.1
34.4
59.4
41.2
58.3
58.3
Kitti-DC
IN(#)
12.6
51.1
103.6
93.6
132.9
133.9
RR(5)
8.2
20.9
20.4
68.1
43.3
70.5
67.5
Metries. Following Choy et al. Feature Matching Recall (FMR) the fraction I2P pairs with more than 5% correct estimated correspondences. A correspondence is regarded as correctly matched if its ground truth 3D distance smaller than Te- Tr is set 0.3m for 3DMMatc//SaanNet and 3m for Kitti-DC. (2) Inlier Ratio (IR)
2019
Wang et al. 2023c b). we adopt four evaluation metrics: (1)
is the average correct correspondence proportions among all I2P pairs. (3) Inlier Number (IN) is
the average number of correct correspondences on each 12P pair. and (4) Registration Recall (RR)
the percentage of correctly-alignes I2P pairs with rotation and translation errors less than TR and T: respectively. (TH, Tς) is set to (20%, 0.5m) for 3DMatch/Scan.e
and (10%, 3m) for Kitti-DC. We provide additional results under different threshold conditions in the supplementary material.
Baselines. We compare FreeReg with fully supervised registration baselines. The image regis- tration method SuperGlue (SG) (Sarlin et al. 2020) is modified to match RGB images and point clouds. LCD (Pham et al. (0200) learns to construct I2P cross-modality descriptors utilizing metric learning. Deep12P (Li & Leel 2021] resolve I2P registration by optimizing an accurate initial pose. We implement a cross- cross-modality feature extraction method 12P-Matr following a concurrent work 2023), where the official codes are released yet. Mcanwhile,
2D3D-Matr (Li et al. FreeReg with P2-Net (Wang et al. 202T and 2D3D-Man [Li et al.] 2023) under their experimen- tal protocol (i al. 2023) the supplementary material, where FreeReg also achieves the best registration performance. We also adopt baseline as mentioned in Fig. 2 (T) that first utilizes Con- trolNet (Zhagg & Agrawala 2023) to generate an RGB image from the target point cloud and then conducts SuperGlue (Sarlin et al.) 2020) to match the input and the generated image (CN+SG). For our method, we report the results using only the diffusion feature (FreeReg-DD D, i.c w 1), only the geometric feature (FreeReg-GG G, i.c. 0), and the fused feature (FreeReg, i.c. 0.5 by default) for matching. More implementation details and analysis are provided in the supplementary material.
compare
4.2 RESULTS ON THREE BENCHMARKS
The quantitative results of FreeReg and basclines on the three cross-modality registration bench- marks are given in Table Some quantitative results are shown Fig.
Correspondence quality is reflected by FMR, IR, and IN. For LCD and 12P-Matr, utilizing a met- ric learning method directly align cross-modality features leads poor performance. CN+SG suffers from the appearance difference between generated images and the input images and thus fails to build reliable correspondences. For FreeReg, using solely diffusion features (FreeReg-D) or geometric features (FreeRee-GG) can already yield results superior to the baselines. Utilizing both features, FreeReg achieves the best correspondence quality and outperforms baselines by a large margin with 54.0% in FMR, 20.6% in IR, and 3.0x higher IN. Note that, unlike baseline methods, FreeReg does not even train on the 12P task.
Registration quality is indicated by RR. Benefited by the high-quality correspondences, FreeReg significantly outperforms the baseline methods by a 48.6% RR and FreeReg- D/G by a 22.9%, 22..9%16... 46.4% RR. Moreover, FreeReg utilizing Kabsch significantly surpasses PnP indoor 3DMatch/ScanNet but is 3% lower than PnP on the outdoor Kitti-DC. The main reason that Zoe- Zoe-Depth performs better these two indoor datasets with average 0.27m error but worse the KITTI with an

---

Published as a conference paper at ICLR 2024
(a) Input RGB & PC
(b) 12t-Matr
(c) FreeReg-D
(e) FreeReg-G
(c) FreeReg
Figure 6: Visualization of correspondences. (a) Input RGB images and point clouds for registration. (b)
Estimated correspondences 12P-Matr. c//d/e Estimated corresoondenees from FreeReg-D FreeReg-G / FreeReg.
average 3.4m error. In the supplementary material, we further provide more analysis and find that PnP achieves more accurate results while Kabsch provides plausible results more cases.
4.3 MORE ANALYSIS
We conduct comprehensive ablation experiments on FreeReg in the following. More analysis about how to fine-tune FreeReg for performance gains, runtime analysis and acceleration strategies of FreeReg, FreeReg performance under different thresholds, more qualitative comparisons and abla- tion studies are provided in the supplementary material.
4.3.1 ABLATING DIFFUSION FEATURE EXTRACTION
In this section, we evaluate a validation scene "bundleffusion-officc""" (BFO) which is not in- cluded in the testset to tune hyperparameters in diffusion feature layer selection and diffusion step f selection. Subsequently,
report their performances on the 3DMatch dataset.
Diffusion layer selection. In table 2 (a-i), we report the size of output feature maps layers the UNet of Stable Diffusion. The feature map size is divided into three levels, i.e. small group (8 11, layers 0-2), medium group (16 x 22, layers 3-5), and large group (32 x 44, layers 6-8). We select the layers with the best registration performance and reasonable matching quality from each level on BFO, specifically layers 0, 4, and 6, to construct our Diffusion Features. Then, in table 2 (j-m), we ablate the layer selection in constructing diffusion features. can be seen that concatenating features of 0,4,6 layers significantly improves the correspondence quality and registration performance. The results from 3DMatch further validate the effectiveness of our choice. More ablation studies on the diffusion layer selection are provided in the supplementary material.
Diffusion step selection. In Table 3 we aim to determine the diffusion step The experimental sults demonstrate that the Diffusion Features from = 150 achieve the best registration performance on BFO. Results on 3DMatch confirm its effectiveness.
4.3.2 ABLATING FEATURE FUSION WEIGHT
8

---

Published as a conference paper at ICLR 2024
Table Layer selection diffusion feature extraction. "Feature map" means the size of the feature map in the form of "channelx lenghh
length".
Feature Map
BFO
3DMatch
ID
Layer
(chamel w)
FMRR%)
IR(%)
IN(#)
RR/%)
FMR
IR(%)
IN(#)
RR
1280
88.9
42.7
18.9
14.4
89.5
39.7
17.6
16.7
(b)
1280 x 8 x 11
91.5
42.1
19.1
12.4
86.9
39.7
18.1
15.8
(c)
1280
86.3
42.9
21.2
14.4
84.2
39.7
20.2
16.9
1250 16 22
87.6
42.7
45.9
23.5
88.4
41.0
47.2
23.0
(e)
4
1280 x 16 22
91.5
36.0
31.7
24.2
92.1
35.3
32.9
26.0
(f)
1250 x 16 22
89.5
35.4
28.1
22.9
91.7
35,5
28.9
25.6
(g)
1280 32
640 32
90.8
19.9
34.3
14.4
85.2
19.6
35.1
22.1
())
640 x 32 44
88.9
17.2
28.4
9.8
82.9
16.8
27.5
17.3
(
[0.4]
256
93.5
44.6
25.9
25.5
92.5
41.3
34.7
26.5
256 32
53.9
34.0
38.5
62.2
32,9
(1)
[4.6]
256 32 x 44
91.5
36.4
45.8
32.0
91.4
35.6
56.2
30.7
(m)
[0.4.6]
384 32 44
94.8
42.3
58.2
35.9
91.9
39.6
60.8
33.2
Table 3: Determining f in diffusion feature extraction.
BFO
3DMalch
FMR(%)
IR(%)
IN(#)
RR(%)
FMR( %)
IR(%)
IN(#)
RR( %)
94.1
40.0
55.8
33.3
91.7
39.4
60.4
31.4
200
92.8
41.0
58.1
35.3
91.8
39.8
61.4
31.2
150
94.8
42.3
58.2
35,9
91.9
39.6
60.8
33,2
100
92.8
41.3
57.3
35.3
91.8
38.8
59.3
31.6
50
92.8
40.0
54.6
32.7
92.0
38.1
57.3
30.6
We ablate the fusion weight w to fuse diffusion and ge- ometric features in Table. 4 based on the baseline model FreeReg. It can be seen that FreeReg achieves the best registration performance when is over, we find that relying more on diffusion features, i.e., 0.6 achieves much similar result the default
Table Determining the fusion weight to fuse diffusion and geometric features.
w FMR(%)
% IN(#) RR( %)
0.5. More-
1.0
91.9
39.6
60.8
52.6
0.7
94.8
45.1
74.1
58.5
0.6
95.3
47.1
81.7
62.3
0.5
94.6
47.0
82.8
63.8
FreeReg. While relying more on geometric features, i.e.,
0.4
93.8
42.9
73.5
w = 0.4 causes sharp performance drop a 8.7% lower IR and a 2.5% lower RR. This demonstrates the robust- ness of the proposed diffusion features.
0.3
91.8
37.5
61.9
0.0
90.7
31.4
49.4
50.4
4.4 LIMITATIONS
The main limitation is that FreeReg requires about 9.3s and 12.7G GPU memory match single I2P pair on a 4090 GPU, yielding much higher RR but longer time usage than baselines LCD (0.6s, 3.5G, 12P-Matr (1.7s, 2.7G), and CN+SG (6.4s, 11.6G). The reason is that need backward process steps ControlNet to denoise the pure noises to reach a specific step for feature extraction. In the supplementary material, show how accelerate FreeReg by 50% with only a ~ 1.4% RR drop. Meanwhile, though we show the superior performance of using diffusion features for I2P registration, we manually select layers and denoising steps in the diffusion feature extraction, which could be improved by future works to automatically select good features.
multiple
CONCLUSION
We propose an I2P registration framework called FreeReg. The key idea of FreeReg is the utilization of diffusion models and monocular depth estimators for cross-modality feature extraction. Specif- ically, we leverage the intermediate representations of diffusion models to construct multi-modal diffusion features that show strong consistency across RGB images and depth maps. We further introduce so-called geometric features to capture distinct local geometric details on RGB images and depth maps. Extensive experiments demonstrate that FreeReg shows strong generalization and robustness in the I2P task. Without any training or fine-tuning achieves a 20.6% improvement in Inlier Ratio, a 3.0x higher Inlier Number, and a 48, 48.6% improve- ment in Registration Recall on three public indoor and outdoor benchmarks.
I2P registration task, FreeReg

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This research is jointly sponsored by the National Key Research and Development Program of China 20022022FF330041000 the Open Fund of Hubei Luojia Laboratory (No. 2201000054), the Na- tional Natural Science Foundation of China (No.42301520), and the Innovation and Technology Commission of the HKSAR Government under the InnoHK initiative and Ref. T45-205/21-N of Hong Kong RGC.
REFERENCES
Tomer Amit, Tal Shaharbany, Eliya Nachmani, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint arX1v:2112.00390, 2021.
Manuel López Antequera, Pau Gargallo, Markus Hofinger, Samuel Rota Bulò, Yubin Kuang, and Peter Kontschieder. Mapillary planet-scale depth dataset. In ECCV, 2020.
Moab Arar, Yiftach Ginger, Dov Danon, Amit H Bermano, and Daniel Cohen-Or. Unsupervised multi-modal image registration via geometry preserving image-to-image translation. In CVPR, 2020.
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. arXiv preprint arXiv:21 2021.
Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adap- tive bins. In CVPR, 2021.
Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Localbins: Improving depth estimation by learning local distributions. ECCV, 2022.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302./2288, 2023.
Gary Bradski. The opencv library. Dr. Dobb's 's Journal, 25(1 1):
2000.
Dylan Campbell, Lars Petersson, Laurent Kneip, Hongdong Li, and Stephen Gould. The alignment of the spheres: Globally-optine spherical mixture alignment for camera pose estimation. In CVPR, 2019.
Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. arXiv preprint 2XXi::221.00788. 2022a.
Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David Fleet. A generalist framework panoptic segmentation images and videos. arXiv preprint arXiv:2210.063666. 2022b.
Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception the wild. NeurlPS, 2016.
Weifeng Chen, Shengyi Qian, David Fan, Noriyuki Kojima, Max Hamilton, and Jia Deng. Oasis: A large-scale dataset for single image 3d in the wild. In CVPR, 2020.
Christopher Choy, Jaesik Park, and Vladlen Koltun. Fully convolutional geometric features. In ICCV, 2019.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
Philip David, Daniel Dementhon, Ramani Duraiswami, and Hanan Samet. Softposit: Simultancous pose and correspondence determination. IJCV, 59:259 2844, 2004.
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In CVPRW, 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.
10

---

Published as a conference paper at ICLR 2024
Zhen Dong, Fuxun Liang, Bisheng Yang, Yusheng Xu, Yufu Zang, Jianping Li, Yuan Wang, Wenxia Dai, Hongchao Fan, Juha Hyyppä, et al. Registration of large-scale terrestrial laser scanner point clouds: A review and benchmark. ISPRS J., 163:327-342, 2020.
Yiqun Duan, Xianda Guo, and Zheng Zhu. Diffusiondepthh Diffusion denoising approach for monocular depth estimation. arXiv preprint urXiv:2303.05022: 2023.
Mengdan Feng, Sixing Hu, Marcelo H Ang, and Gim Hee Lee. 2d3d-matchnet: Learning to match keypoints across 2d image and 3d point cloud. In ICRA, 2019.
David Forsyth and Jean Ponce. Computer vision: a modern approach. prentice hall professional technical reference, 2002.
Vitor Guizilini, Igor Vasiljevic, Dian Chen, Rares Ambrus, and Adrien Gaidon. Towards zero-shot scale-aware monocular depth estimation. arXiv preprint arXiv:2306. 2023.
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey. IEEE TPAMI, 43(12):4338 43688,44444440 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion. arXiv preprint arXiv:2305. 15581, 2023.
Jonathan Ho and Tim Salimans. arXiv:2207.12590 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Peng Jiang and Srikanth Saripalli. Contrastive learning of features between images and lidar. In CASE, 2022.
Peng Jiang, Fanglin Gu, Yunhai Wang, Changhe Tu, and Baoquan Chen. Difnet: Semantic segmen- tation by diffusion networks. In NeurlPS, 2018.
Jinyoung Jun, Jae-Han Han Lee, Chul Lee, and Chang-Su Kim. Depth map decomposition for monocular depth estimation. In ECCY, 2022.
Wolfgang Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallo- graphica Sec.A, 32(5):922-923, 1976.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In NewrIPS, 2022.
Minjung Kim, Junsco Koo, and Gunhee Kim. Ep2p-loc: End-to-end 3d point 2d pixel localization for large-scale visual localization. In ICCV, pp. 21527-21537, 2023.
Jason Ku, Ali Harakeh, and Steven L Waslander. In defense of classical image processing: Fast depth completion the cpu. In CRV, 2018.
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have semantic latent space. In ICLR, 2022.
Baiqi Lai, Weiquan Liu, Cheng Wang, Xuesheng Bian, Yanfei Su, Xiuhong Lin, Zhimin Yuan, Siqi Shen, and Ming Cheng. Learning cross-domain descriptors for 2d-3d matching with hard triplet loss and spatial transformer network. In ICIG, 2021.
Kevin Lai. Liefeng Bo, and Dieter Fox. Unsupervised feature learning for 3d scene labeling. In ICRA, 2014.
Vincent Lepetit, Francese Moreno-Noguer, and Pascal Fua. Epnp: An accurate o (n) solution to the p n p problem. IJCV, 81:155-166, 2009.
11

---

Published as a conference paper at ICLR 2024
Jiaxin Li and Gim Hee Lee. Deepi2p: Image-to-ponte -o-poonn cloud registration via deep classification. In CVPR, 2021.
Minhao Li, Zheng Qin, Zhirui Guo, Renjiao Yi, Chengyang Zhu, and Kai Xu. 2d3d-matr: 2d-3d matching transformer for detection-free registration between images and point clouds. In ICCV, 2023.
Yangyan Li, Hao Su, Charles Ruizhongtai Qi, Noa Fish, Daniel Cohen-Or, and Leonidas J Guibas. Joint embeddings of shapes and images via enn image purification. ACM TOG, 34(6):1-12, 2015.
Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation. arXiv preprint arXiv:2204.0098) 2022.
Youqi Liao, Jianping Li, Shuhao Kang, Qiang Li, Guifang Zhu, Shenghai Yuan, Zhen Dong, and Bisheng Yang. Se-calib: Semantic edges based lidar-camert boresight online calibration in urban scenes. IEEE TGRS, 2023.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Liu Liu, Dylan Campbell, Hongdong Li, Dingfu Zhou, Xibin Song, and Ruigang Yang. Learn- ing 2d-3d correspondences to solve the blind perspective-n-ponte problem. arXiv preprint arXiv:2003.06652, 2020.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview consistent images from a single-view image. arXiv preprint arXiv:2309.03443 2023.
Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. arXiv preprint arXiv:2305. 14334, 2023a.
Simian Luo, Yiqin Tan, Longho Huang, Jian Li, and Hang Zhao. Latent consistency models: Syn- thesizing high-resolution images with few-step inference. 2023b.
Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman. 1 year, 1000 km: The oxford robotcar dataset. I/RR, 36(1):3-15,3 2017.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprini 23X.:2332..08533 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.07193. 2023.
Quang-Hieu Pham, Mikaela Angelina Uy, Binh-Son Hua, Due Thanh Nguyen, Gemma Roig, and Sai-Kit Yeung. Lcd: Learned cross-domain descriptors for 2d-3d matching. In AAAI, 2020.
Zheng Qin, Hao Yu, Changjian Wang. Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In CVPR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 7768-8763. PMLR, 2021.
Siyu Ren, Yiming Zeng, Junhui Hou, and Xiaodong Chen. Corri2p: Deep image-to-poing cloud registration via dense correspondence. IEEE T-CSVT, 33(3): 199-1128,, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In CVPR, 2020.
12

---

Published as a conference paper at ICLR 2024
Paul-Edouard Sarlin, Daniel DeTone, Tsun-Yi Yi Yang. Armen Avetisyan, Julian Straub, Tomasz Mal- isiewicz, Samuel Rota Bulò, Richard Newcombe, Peter Kontschieder, and Vasileios Balntas. Ori- enternet: Visual localization in 2d public maps with neural matching. In CVPR, 2023.
Saurabh Saxena, Charles Hermaann Junhwa Hur, Abhishek Kar, Mohammad Norouzi, Deqing Sun, and David J Fleet. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation. arXiv preprint (rxXiv:330601922 2023a.
Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J Fleet. Monocular depth estima- tion using diffusion models. arXiv preprint arXiv:2302. 14816, 2023b.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. In NeurlPS, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint wrXiv:2010002502, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 I. 13456, 2020b.
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In CVPR, 2021.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng. Zhongdao Wang. and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. CVPR, 2020.
Haoru Tan, Sitong Wu, and Jimin Pi. Semantic diffusion network for semantic segmentation. In NeurlPS, 2022.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. arXiv preprint arXiv:2306.088 03881, 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-ime -mmee translation. In CVPR, 2023.
Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant cnns. In 3DV, 2017.
Bing Wang, Changhao Chen, Zhaopeng Cui, Jie Qin, Chris Xiaoxuan Lu, Zhengdi Yu, Peijun Zhao, Zhen Dong, Fan Zhu, Niki Trigoni, et al. P2-net: Joint description and detection of local features for pixel and point matching. In ICCV, 2021.
Guangming Wang, Yu Zheng, Yanfeng Guo, Zhe Liu, Yixiang Zhu, Wolfram Burgard, and Hesheng Wang. End-to-end 2d-3d registration between image and lidar point cloud for vehicle localization. arXiv preprint ar ar36v:2366. 11346, 2023a.
Haiping Wang, Yuan Liu, Zhen Dong, and Wenping Wang. You only hypothesize once: Point cloud registration with onation-equivariin descriptors. In ACM MM, 2022a.
Haiping Wang, Yuan Liu, Zhen Dong, Yulan Guo. Yu-Shen Liu, Wenping Wang. and Bisheng Yang. Robust multiview point cloud registration with reliable pose graph initialization and history reweighting. In CVPR, 2023b.
Haiping Wang, Yuan Liu, Qingyong Hu, Bing Wang. Jianguo Chen, Zhen Dong, Yulan Guo, Wen- ping Wang, and Bisheng Yang. Roreg: Pairwise point cloud registration with oriented descriptors and local rotations. IEEE TPAMI, 2023c.
Yuan Wang. Yuhao Li. Yiping Chen, Mingjun Peng. Haiting Li, Bisheng Yang, Chi Chen, and Zhen Dong. Automatic registration of point cloud and panoramic images urban scenes based on pole matching. JAG, 1 10:1003, 2022b.
13

---

Published as a conference paper at ICLR 2024
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khan- delwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Ar- goverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00493 2023.
Julia Wolleb, Robin Sandkühler, Florentin Bieder, Philippe Valmaggia, and Philippe C Cattin. Dif- fusion models for implicit image segmentation ensembles. In MIDL, 2022.
Ke Xian, Chunhua Shen, Zhiguo Cao, Hao Lu, Yang Xiao, Ruibo Li, and Zhenbo Luo. Monocular relative depth perception with web stereo data supervision. In CVPR, 2018.
Ke Xian, Jianming Zhang, Oliver Wang, Long Mai, Zhe Lin, and Zhiguo Cao. Structure-guider ranking loss for single image depth prediction. In CVPR, 2020.
Xiaoxia Xing, Yinghao Cai, Tao Lu, Shaojun Cai, Yiping Yang, and Dayong Wen. 3dtnet: Learning local features using 2d and 3d cues. In 3DV, 2018.
Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe, and Elisa Ricci. Transformer-base attention networks for continuous pixel-wise prediction. In ICCV, 2021.
Gongxin Yao, Yixin Xuan, Yiwei Chen, and Yu Pan. Cli2p: Coarse-to-fine cross-modal correspon- dence learning for image-to-point cloud registration. arXiv preprimt arXiv:330.007142, 2023.
Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan. Enforcing geometric constraints of virtual normal for depth prediction. ICCV, 2019.
Wei Yin, Yifan Liu, and Chunhua Shen. Virtual normal: Enforcing geometric constraints for accu- rate and robust depth prediction. IEEE TPAMI, 44(10);7282-7295, 2021.
Wei Yin, Zhing, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from a single image. arXiv preprint TrXi1:2307.10984, 2023.
Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. New erfs: Neural window fully-comnected crfs for monocular depth estimation. arXiv preprint arXiv:2203.01502. 2022.
Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In CVPR, 2017.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsual Yang. tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. arXiv preprint 230v:2305..1347, 2023.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05544 2023.
Junsheng Zhou, Baorui Ma, Wenyuan Zhang, Yi Fang, Yu-Shen Liu, and Zhizhong Han. Differ- entiable registration of images and lidar point clouds with voxelpoint-to-pixll matching. arXiv preprint 2rX10:2312.04000, 2023.
Qian-Y Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for 3d data processing. arXiv preprint
2018.
Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Os- wald, and Mare Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In CVPR, 2022.
14

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1
IMPLEMENTATION DETAILS OF FREEREG
A.1.1 DEPTH PROJECTION AND DENSIFY
We project the point cloud to a given camera pose following a classic method (Forsyth Ponce 2002 and set null pixels to zeros. We adopt a CPU-based depth completion method IDC-DC (Ku ct aL. 2018) to densify sparse depth maps. We adopt fill-in_fstt method with a kernel of DIAMOND KERNEL. for ScanNet data and fill. in inmultiise method with the default settings for Kitti-DC.
A.1.2 DIFFUSION FEATURE
Depth. We adopt ControlNet (Zhang Agrawala 2023 for diffusion feature extraction on depth
maps. The depth maps are normalized to 0 ~ 255 following (Zhang & Agrawalal 2023 We
use the pre-trained model of ControlNet conditioning on depth map We use DDIM Sampling
strategy (Song et al., 2020a with diffusion steps set to range(begue (eegin 1000, end 0, step
-50) and ddin eta set to 1.0. We follow (Zhang Agrawala
2023 to adopt Classifier-Free
Guidance (Ho & Salimans
2022
c' 1Iuss, wl((s' C,),
(2)
where is the estimated noise step t', is so-called unconditional guidance scale, C is the text- prompt for conditional sampling and C, is the text-prompt for unconditional sampling. We set w to 4.0 in implementation. We use depth maps conditions for both conditional sampling process ((z",t,C) and unconditional sampling process U(z*, f, Cu). C is "best quality, a photo of a room, furniture, household items" for indoor 3DMatch and ScanNet datasets, "a vehicle camera photo of street view, trees, cars, people, house, road, sky" for outdoor Kitti-DC, C, is set to "lowres, bad anatomy, bad hands, cropped, worst quality". We set di to 1.0 for all SD UNet the step f 150.
RGB. We use the pre-traine pre-trained Stable Diffusion v1.5 model for diffusion feature extraction on RGB images. The image is normalized to -1 before being red to SD, same as (Rombach al. 2022
Zhang & Agrawala 2023). We use the internal features of SD U-net at step i = 150.
A.1.3 ZOE-DEPTII
We adopt Zoe-Depth (Bhat et al. 2023 for monocular metric depth estimation. We utilized the Zoe-Depth model pretrained on the Mi2-NYU-Depth--2e for the indoor 3DMatch and ScanNet datasets. We use the Zoe-Depth model pretrained on M12+ 111=-YU--Dpthh-22KKT Kitti-DC dataset. Note that the training set has no overlap with our testsets.
for the outdoor
A.1.4 FCGF
We use FCGF (Choy et al.
2019h for 3D geometric feature extraction point clouds recovered from
Zoe-Depth outputs or input point clouds. We use the FCGF (Choy et al. 2019) model pretrained on the 3DMatch (Zeng et al.
2017 training set for the indoor 3DMatch and ScanNet datasets and the Uhrig et al. 2017 for the outdoor KC dataset.
model pretrained on KITTI
Following (Choy et al. 2019 we downsample point clouds by a voxel size of 0.025m for indoor datasets and 0.3m for outdoor datasets before FCGF extraction. However, since FCGF is constructed on downsampled point clouds, not every pixel in the depth maps has a corresponding FCGF feature. Therefore, for query pixel, we first project it into 3D space and retrieve its nearest FCGF feature in 3D space as its FCGF feature. However, the distance of this point its nearest FCGF feature
M12 pt
nodel httpp//......oaoooo.o.././aaaaaa M12 NK
15

---

Published as a conference paper at ICLR 2024
exceeds Tg. set its geometric feature to zero vector. Tg is set to 0.5m for indoor datasets and 5m for outdoor datasets.
A.1.5 FEATURE FUSION
The RGB images and depth maps are resized to 512 x 704 for indoor datasets and 512 x 1280 for outdoor datasets. The output feature map of is 16x smaller than the original input. The Diffusion Feature maps for indoor and outdoor images are 32 x 44 and 32 x 80, respectively. We interpolate features on these feature maps construct diffusion features.
A.1.6 SOLVING RELATIVE TRANSFORMATION
Given two sets of fused features F' on RGB image I and FD on depth map D, we conduct Nearest Neighbor (NN) feature matching with a mutual nearest check (Wang et al. 2022a) to find set of correspondences between and D. Note that each pixel in D corresponds
3D point in P. We {((p E Z+ .9 R))), where p pixel
thus obtain set of pixel-to-pon tootootttttoooooo correspondences coordinate in / and is point in point cloud P. Based on these correspondences, we employ the Kabsch algorithm recover the I2P relative pose when we have Zoe-depth estimations. Otherwise, we use the Perspective-n-Point (PnP) method.
The Kabsch algorithm is formulated by
(R. t)
Σ - Rproj(p K!) t||2, RESO(3), (P.O)EC
arg min
where (ro v), d. K) = K ju, v, 1)" * converts 2D pixel coordinates (u, E Z+² to 3D point based on depth d and intrinsic matrix
The PnP algorithm is formulated by
(R, = arg min RESO (F-9)EC
Σ
- 11--proj tg.j-((g K!, R, f)11,,
(4)
where (g. K', RR,t) projects a 3D point to a 2D pixel coordinate according to the intrinsic matrix K! and I2P relative pose (R, t) (Forsyth & Ponce 2002
To estimate the SE(3) relative transformation with PnP, we use PnP-RANSAC implemented in OpenCV (Bradski] 2000) with 50k iterations and the distance tolerance of 10.0. For Kabsch-based pose estimation, we adopt the Kabsch-RANSS implemented in Open3D (Zoou et al. 2018 with 50k iterations and the distance tolerance 0.2/4m for indoor/outdoor datasets.
A.2 IMPLEMENTATION DETAILS OF BASELINE METHODS
2020) is a cross-modality registration baseline published in AAAI 2020. We utilize the official codes and models. We extract LCD-2D/3D features on the same feature pix- els/points as FreeReg for a fair comparison. LCD models are trained on the indoor 3DMatch Zeng
LCD (Pham
et al.
2017) dataset and almost fail on the outdoor Kitti-DC dataset.
DeepI2P & Lee 2021 is cross-registration baseline published in CVPR 2021. We utilize the official codes and models pretrained on the Oxford RobotCar dataset (Maddern et al.) 2017)) However, does not generalize well to both the indoor 3DMatch/SeanNer datasets and the outdoor Kitti-DC dataset.
Corr12P (Ren et al. 2022 is a modality baseline published in TCSVT 2022. The official codes are released but pretrained model is available. We thus train it following the official guid- ance. The training process takes about 14 hours for 25 epoches. The trained model achieved an average rotation error of degrees their dataset. However, the model almost failed our KITTI- DC test data because Corrl2 relies on the intensity information of the point cloud as input, which is not available in our test data and set them ones.
SuperGlue (Sarlin et al. 2020 (SG) is a widely adopted image-matching method. We employed the official implementation and indoor/outdoor models estimate correspondences between an RGB
16

---

Published as a conference paper at ICLR 2024
image and depth image. A total of 1024 key points were detected every image, and the corre- spondence confidence threshold was set at 0.01 to retain more correspondences to solve PnP.
P2-Net (Wang et al. 2021 is I2P registration baseline published in ICCV 2021. However, the P2-Net fails to converge on the 3DMatch trainset as stated in 2D3D-Matr (Li et al. 2023).
I2P-Matr extracts features from RGB images and point clouds separately. Then, 2D3D-Mat subse-
quently utilizes the nearest neighborhood matcher with a mutual nearest check for feature matching.
Same as (Li al. 2023), we utilize ResNet (He et al. 2016) and FPN (Lin et al. 2017) construct siamese network as the feature extractor for both RGB images and depth maps projected from point
clouds. We adopt Circlelosss (Sun al. 2020 Li et al. 2023 to supervise the training, and the training strategy is consistent with that of 2D3D Matr (Li ct al. 2022 We trained I2P-Matr on the 3DMatch (Zeng et al. 2017) trainset for 20 epochs, utilizing a learning rate of Ie-3, and applying a decay factor of 0.98 after each epoch. For the test, we adopt the same data pre-processing strategies and evaluation settings as FreeReg.
Superrluue et al. 2020 (CTL-SG) is a combined baseline. We adopt the same settings as Sec. A.1.2 for ControlNet.
ControlNet(Zhann & Agrawala 2023
A.3 COMAPRISON WITH CONCURRENT WORK 2D3D-MATR
In this section, we provide a comparison with the concurrent work 2D3D-Matr (Li et al. 2023 on their evaluation dataset RGBD- Since their codes have not been released yet, we re- port their results from their paper. 2D3D-Matr (Li et al. (023) is a concurrent work of FreeReg. MatD-Matr adopts a feature training methodology similar to 12P-Matr, and further learns to estimate correspondence construction based on dense matcher similar to Granroosfrrmer (Qin et al.) 2022)
yielding better correspondences and transformation solutions.
Experimental protocol. Following (Li al. 2023), we conducted evaluation on scene_ll 11 14 from the RGBD- Scenes - v2 dataset (Lii et al. 2014). The dataset preparation is identical to 2D3D- Matr. For evaluation. adopt three metrics same 2D3D-Matr: (1) Inlier Ratio (IR), the ratio of pixel-to-poine toint matches whose 3D distance is below 5cm over all constructed matches. (2) Feature Matching Recall (FMR), the ratio of maage-t--point-llsn (3) Registration Recall (RR), the ratio of
pairs whose inlier ratio is above 10%. pairs whose RMSE between the point clouds transformed by the ground truth and the predicted transformation is below T 10cm. For fair comparison, we follow (Li et al. (023) to adopt a traditional coarse-to-fing strategy similar to 2D3D-Matr establish dense correspondences. Specifically, we initially utilized k-nearest neighbor mutual matcher on the diffusion features to construct coarse correspondences. Subsequently, we built dense correspondences on local patches of the established correspondences utilizing geometric features and nearest neighbor (NN) matcher. Note that FreeReg requires no training on the I2P task, whereas 2D3D-Matr was trained on scenes 1-8 of the RGBD-Scenes-v2 dataset.
The results are shown in Table. 5 FreeReg achieves 30.9% IR close to 2D3D-Mas and better registration quality than 2D3D-Matr with RR of 57.3%. We notice FreeReg with the Kabsch method for transformation estimation performs much worse results than PnP in this experimental setting because the estimated depth maps from Zoe-Depth have a large global scale difference from real depth values. More analysis about the Kabsch and PnP can be found in Sec. A.4.4
Results.
A.4 MORE ABLATIONS AND RESULTS
A.4.1 USE OTHER LARGE VISION BACKBONES RATHER THAN DIFFUSION NETWORKS.
In FreeReg, we utilize pre-trained diffusion models, specifically Stable Diffusion and ControlNet, to establish cross-modality consistent features between images and point clouds. A possible alternative method could be employing some other pre-trained large-scale vision foundation backbones( (VFB), such as Dino-v2 (Qquab et al. 2023) and CLIP (Radford et al., 202I), for cross- feature construction. However, these VTBs are designed and trained to process RGB images, while point clouds encode geometric information rather than RGB information, containing distinct modality differences from RGB images. Thus neglect the modality differences between point clouds and images. Specifically, we first project the input point cloud into a depth map and expand it to three channels. Subsequently, use Dino-v2 (Oquab al. 2023). CLIP (Radford et al. 2021
17

---

Published as a conference paper at ICLR 2024
Table 5: Quantitative results RGBD-Scene-v2
Method
Scene-11
Scene-12
Scene-14 Feature Matching Recall (%)
Scene-13
Mean
FCGF-2D3D
11.1
30.4
51.5
15.5
27.1
P2-Net
48.6
65.7
82.5
41.6
59.6
Predator-2D3D
86.1
89.2
63.9
24.3
65,9
2DtD-Matr
98.6
98.0
88.7
77.9
90.8
FreeReg
91.9
93.4
93.1
49.6
82.0
Inlier Ratio (%)
FCGF-2D3D
11.8
P2-Net
9.7
12.8
17.0
9.3
12.2
Predator-
17.7
19.4
17.2
8.4
15.7
2D3D-Matr
32.8
34.4
39.2
23.3
32,4
FreeReg
36.6
34,5
34.2
18.2
30,9
Registration Recall (%)
FCGF-2D3D
26.4
41.2
37.1
16.8
30.4
P2-Net
41.2
38.4
Predator-2D3D
21.6
13.7
30.2
2D3D-MMtt
63.9
53.9
56.4
FreeReg+Kabsie
38.7
51.6
30.7
15.5
34.1
FreeReg+PnP 74.2
72.5
54.5
27.9
$7.3
Table 6: Results on 3DMatch utilizing features from other VFBs
Methods
FMR (%)
IR (%)
IN (#)
RR (%)
Dino-v2 (Oquab et al. 2023
64.3
13.0
8.8
9.8
CLIP Radford al. 22T
21.9
3.3
2.4
Dino-SI ayer Zhang al.
2023
35.5
8.9
12.6
14.8
ATTF
Zhang el al. 2023
41.7
10.9
15.5
17.4
FreeReg (Ours)
94.6
47.0
82.8
63.8
or ATTF Zhang et al. 2023 to extract features from the depth map and input RGB for matching.
Table 6 we report the performance of features from other VFBs I2P registration. The results indicate that existing visual models struggle to overcome significant modality differences between images and point clouds, achieving only less than 13% IR and less than 20% RR on 3DMatch.
A.4.2 USING STABLE DIFFUSION TO EXTRACT DEPTH FEATURES
In FreeReg, employ CN to extract diffusion features on depth maps. An alternative approach would to di- rectly feed the noised depth map to SD for diffusion fea- ture extraction. In table 7 we observe that using CN boosts FreeReg by 2x IN and 14.0% on RR than di- rectly feeding it to SD.
Table 7: Extract depth diffusion features with ControlNet (CN) or not.
CN FMR(% %) IR( %) IN(#) RR(%)
83.2
28.4
30.3
19.2
91.9 39.6
60.8
33.2
A.4.3 RESULTS UTILIZING DIFFERENT DIFFUSION LAYER FEATURES
In table 8 utilize features of different diffusion layers construct cross-modality diffusion features. In combinations (a/b), it can be observed that employing the later layers indexed over leads sharp performance drop. Combinations (c-j.u) combine different layers from the three groups mentioned in Sec.4.3.1 of the main paper, i.c., small group (with size of 8 x 11, layers 0-2), medium group (with size of x 22, layers 3-5), and large group (with size of 32 44, layers 6-8). These combinations yield similar registration performances. Specifically, the combination [2,4,6] achieves the optimal FMR, [0,3,6] achieves the best IR, and [0,5,6] achieves the highest RR albeit with a significantly lower IR and IN. Combinations (h-p) fuse features of different layers within the same group and combinations (q-t) utilize more layers. These combinations do not yield performance improvement. We empirically select the layer combination [0.4.6] which exhibits ideal correspondence and registration performance. The layer selection could improved by future works automatically select good features.
18

---

Published as a conference paper at ICLR 2024
Table 8: Results bundlefusion-ooficee utilizing different diffusion layer features.
Layer
Feature Dimension
FMR(%)
IR(%)
IN(
RR(%)
(a)
10,4,71
384 x 32 x 44
92.2
37.2
56.1
31.4
(b)
[0,4,8]
384 x 32 x 44
92.2
36.6
51.8
32.7
(C)
[1,4,6]
384 x 32 x 44
93.5
41.5
56.2
35.3
(d) [2,4,6]
384 x 32
x 44
95.4
41.7
56.8
34.6
(e) [0,3,6]
384 x 32 x 44
94.8
43.6
60.4
34.6
( [0,5,6]
384 x 32 x 44
93.5
40.7
53.6
36.6
( [1,3,6]
384 x 32 x 44
93.5
41.8
60.2
35.3
(h) [1.5.6]
384 x 32 x 44
92.2
40.9
53.6
32.0
( [2,3,6]
384 x 32 x 44
91.5
43.2
58.5
34.6
0 [2,5,61
384 x 32 44
94.8
41.0
54.0
34.0
(K) 10,1.6
384 32 44
92.2
43.6
59.8
34.6
() [0,2,6]
384 x 32 x 44
94.8
44.2
61.0
33.3
(m) [1.2,6]
384 x 32 x 44
93.5
43,9
59.1
31.4
(n) [3,4,6]
384 x 32 x 44
93.5
40.5
57.9
35.3
(0) [3,5,61
384 x 32 X 44
92.8
40.0
55.0
34.6
(p) [4,5,6]
384 x 32 44
94.8
37.2
46.4
34.0
9
[0,12,6
512 x 32
44
92.8
45.3
59.7
33.5
(1) [3,4.5.6]
512 x 32
44
92.8
40.4
53.0
34.0
(8) 10.1,4.6]
512 x 32
x 44
93.5
43.8
57.7
34.6
(1) 10,3,..61
512 x 32
x 44
92.2
43.3
59.6
35.9
(0)
[0,4.6]
384 x 32
x 44
94.8
42.3
58.2
35.9
Table 9: Results of PnP and Kabsch algorithms.
3DMatch
Registration Recall (9)
Zoe-ADE (m)
RE(")
TE(m)
(5".0.1m)
(10".0.2m)
0..3m
(205,
(255.0.5m)
FreeReg PnP FreeReg Kabsch
78.4
9,4
22.3
31.4
40.0
40.2
22.5
0.659
3.5
20.5
41.6
63.8
65.3
ScanNer
Registration Recall (St)
Zoe-ADE (m)
RE(")
TB(m)
(5%.0.1m)
(10' 0.2mm
(15%.0.3m)
20050.5m)
(25 (0.5m)
FreeReg + PnP FreeReg Kabseh
51.9
11.7
32.9
46.2
57.6
57.9
0.23
0.429
8.2
34.0
$8.2
78.0
78.5
Kitti-DC
Registration Recall (%)
Zoe-ADE (m)
RE(")
TE(m)
(37,1m)
(5%.2m)
(7*.3m)
(10
(10' (m)
FreeReg PaP 3.40
22.3
3.150
39.3
S7.6
67.0
70.5
75.1
FreeReg Kabsch
6.2
2.559
5.3
26.0
55.0
67.5
80.1
A.4.4 COMPARISON BETWEEN PNP AND KABSCH
In this section, we focus on comparing PnP algorithm with the Kabsch algorithm. In Table 9 we report the performance of FreeReg different RR thresholds using PnP and Kabsch algorithms. We also provide the average depth error (ADE) of Zoe- Coe-Depth estimation, the average rotation error (RE), and the translation error (TE) of the estimated I2P transformations on the four datasets.
It can be seen that under strict thresholds, PnP achieves a higher RR, whereas Kabsch experiences sharp performance drop. Using the Kabsch method yields more correct transformations in larger RR thresholds than PnP, and has lower average transformation errors on all thresholds. The main reason that the Zoe- DeptD predictions are absolutely accurate, especially terms of scales. has 0.27m depth error in indoor datasets and a 3.4m depth error in outdoor scenes which leads to a bias in the Kabsch estimation.
For further analysis, we provide the recall under different rotation and translation errors on the three datasets Fig.7) can be seen that on the ScanNet dataset with relatively low estimate depth error for Zoe-Depth the Kabsch algorithm estimates more accurate I2P registrations, which is similar to PnP even under strict thresholds. However, on 3Dmatch and KITTI-DC with larger depth emrors for Zoe- ReetDh the translation accuracy of Kabsch estimations is much lower. Nevertheless, Kabsch leverage Zoe-depth to constrain the transformation estimation thus successfully registering more I2P pairs under loose RR threshold.
19

---

Published as a conference paper at ICLR 2024
3DMatch
ScanNet
Kitti
10
1.-
0.8-
Recall
Recall
Recull
0.4
0.4
0.2
kabsch
kabsch pnp
kabsch
pnp
pnp
0.0
0.0
0.0
Rotation Error ()
Rotation Error ()
40
80
Rotation Error i
1.0
0.7
0.8
0.6
Recall
0.5-
Recoll
0.6
Recall
0.4
0.3
0.4
0.4
kabsch
02
kabsch
kabsch
0.1
pnp
pnp
0.0
00
0.0
01 02
0.
011
0s
Translation Error (m)
Translation Error (in)
Translation Error (m)
Figure Rotation and translation errors on three benchmarks.
Table 10: Mono-modality registration performance: "RGB" means registering RGB images. "DPT"
means 3D registration on point clouds recovered from depth maps.
3DMatch
ScanNet
Method
RGB
DPT
FMR(%)
IR(%)
IN(#)
RR(%)
FMR %
IR(%)
IN(#)
RR(
LCD
94.6
48.9
101.0
99.7
53.0
140.0
SG
99.6
83.8
161,9
92.6
117.8
12P-Matr
94.7
46.1
140.8
99.1
45.1
146.3
FreeReg
98.3
65.3
185.6
74,2
99.9
72.8
249.2
87.9
LCD
76.9
18.8
46.0
44.9
92.6
32.2
79.9
71.8
ROGF
95.4
54.5
121.3
84.7
99.8
75.5
230.8
95.3
12P-Matr
96.1
43.9
140.4
80.9
98.7
55.3
174.3
91.6
FreeReg
96,4
55.7
131.8
87.0
99.5
73.8
202.5
94.7
A.4.5 RESULTS ON MONO-MODALITY BENCHMARKS
FreeReg features are capable of performing mono-modalin registration on RGB or 3D data. The qualitative results given in Table 10 Given RGB image pairs, FreeReg employs Stable Diffu- sion (Rombach et al.] 2022 to construct diffusion features FreeReg-D and uses Zoe-Depth to estimate depth maps of query RGB images for further constructing geometric features for FreeReg.
When given depth map pairs, FreeReg uses ControlNet (Zhang & Agrawala 2023 to extract dif- fusion features for matching in FreeReg-D, and further combine them with Geometrie Features extracted on depth maps in FreeReg.
RGB Registration. FreeReg (FreeRegg significantly outperforms fully-supervised mrods-modality registration baseline LCD (Pham et al. 2020 and I2P-Matr by 18.1%/23.5% and 1.8 x /1.5x on IR and IN when registering RGB images. Compared with fully-supervised RGB registration baseline SuperGlue (Sarlin et al. 2020). features of FreeReg are distinctive enough to construct 1.6x more inlier correspondences. However, FreeReg under-performs SuperGilue by 19.2% on IR. This is mainly because SuperGlue applies transformer for matching. whereas FreeReg constructs correspondences by simple NN matcher. terms of the final registration results, FreeReg is able to produce better registration results due its utilization of Zoe- -Dpth..
3D Registration. For 3D registration on depth map pairs from 3DMatch and ScanNet, FreeReg significantly outperforms cross- modslltyy baseline LCD (Pham al. 2020) and I2P-Matr. In com- parison with 3D-registration baseline FCGF (Choy et al., 2019), FreeReg has comparable results with better performance the 3D Match dataset but worse results the ScanNet dataset.
20

---

Published as a conference paper at ICLR 2024
Table 11: FreeReg performances before after fine-tuning (-Ft).
3DMatch
FMR (%)
IR (%)
IN (#)
RR (%)
FréeReg
94.6
47.0
82.8
63.8
FreeReg-Fi
95.8 (+1.2)
53.7 (+6.7)
95.4 (+12.6)
70.1 (+6.3)
Scan
FMR (%)
IR (%)
IN (#)
RR (%)
FreeReg
98.5 98.9 (+0.4)
56.8
114.4
78.0
FreeReg-Fi
62.4 (+5.6)
126.1 (+11.7)
81.4 (+3.4)
KITTI-DC
FMR (%)
IR (%)
IN (#)
RR (%)
FreeReg
99.7
58.3
132.9
70.5
FreeReg-Fi
100 (+0.3)
62.8 (44.5)
156.1 (+23.2)
81.9 (+11.4)
Table 12: Comparison 2D3D- Matr and FreeReg with fine-tuning.
Method
Scene-11
Scene-12
Scene-13
Scene-14
Mean
Feature Matching Recall (%)
2D3D-Matr 98.6
98.0
88.7
77.9
90.8
FreeReg 91.9
93.4
93.1
49.6
82.0
FreeReg-Ft
93.5
95.6
95.0
63.3
86.9
Inlter Ratto ( (%)
2D3D-Matr
32.8
34.4
39.2
23.3
32.4
FreeReg
36.6
34.5
34.2
18.2
30.9
FreeReg-Ft
40.3
40.9
46.2
25.9
38.3
Registration Recall (%)
2D3D-Matr
63.9
53.9
58.8
49.1
56.4
FreeReg
74.2
72.5
54.5
27.9
57.3
FreeReg-Ft
80.6
82.4
73.3
38.1
68.6
A.4.6
MORE QUALITATIVE RESULTS.
We provide more qualitative results in Fig. The estimated cross-modality correspondences can be utilized to warp local image patches onto the point cloud illustrated Fig. T0
A.5 MORE ANALYSIS ON FREEREG
A.5.1 HOW TO FINE-TUNE FREEREG FOR IMPROVED PERFORMANCES?
We found that ControlNet's pre-training model trained on depth maps from monocular depth estimation rather than depth data collected from sensors (Zhang & Agrawala 2023 Then, argue that if sensor-captured RGB-D data available, ControlNet fine-tuning can significantly im- prove FreeReg's performance. We conducted depth-to-RGB ControlNet tnn-ttunig for 1 hour Then, on test-set, FreeReg-Fr achieved IR improvements of 6. 37%55.%%/44 and RR improvements 6.3%/3 /3.4%/11..... as shown in Table 11 Comparing with 2D3D-Matr in Table 12 FreeReg-Fi achieved 5.9% increase in IR and 12.2% increase in RR.
using sensor-captun sensor-captured RGB-D data from the train-set of 3DMatch/ScanNe//IITI SDMMachSSaaNN//KKTTTTa
Table 13: FreeReg acceleration with fewer DDIM sampling iterations.
DDIM-Iters (#)
FMR (%)
IR (%)
IN (#)
RR (%)
Time (s)
93.6
47.8
72.1
62.4
4.7
10
95.0
46.6
83.0
63.5
6.4
15
94.5
47.1
82.4
63.0
8.1
20 (Ours)
94.6
47.0
82.8
63.8
9.3
21

---

Published as a conference paper at ICLR 2024
Figure 8: Semantic properties of FreeReg features.
A.5.2 HOw TO ACCELERATE FREEREG?
Here we provide two insights for FreeReg speed-up.
ControlNet acceleration. The most time-consuming part of FreeReg registration the sampling process of ControlNet. As described in Sec. [...2] adopt DDIM Sampling strategy (Song al. 2020a) with diffusion steps set to sange(beging = 1000, end = 0, step = -50) for ControlNet execution, i.e., 20 sampling iterations by defaults. We enlarge the sampling step interval for fewer sampling iterations. As shown in table when the number of the DDIM iterations is reduced only 5 samples, FreeReg achieves registration time reduction of 50%, with only 1.4% decrease in RR. This could be further optimized with the emergence of diffusion sampling speed-up strategies (Lo al. 2023b).
The features of FreeReg are directly derived from the pre-trained large model Stable Diffusion, providing powerful descriptive capabilities but is slow. We thus can employ FreeReg as a teacher model to guide the training of a lightweight but strong student image/point cloud feature extractor through knowledge distillation.
Knowledge distillation.
A.5.3 ARE FREEREG FEATURES SUITABLE FOR OTHER CROSS-MODALITY TASKS?
Besides cross-modality consistency, we argue that FreeReg features semantic-aware. In Fig. we present some visualizations of point cloud FreeReg features, revealing strong semantic relevance. This suggests the potential applicability of these features for downstream tasks requiring semantics.
22

---

Published as a conference paper at ICLR 2024
Input RGB PC
Diffusion Feat. RGB&PC
Geometric Feat. RGB&PC
(d) Fused Feat. of RGB&PC
Estimated Correspondences
Figure 9: Additional qualitative results. (a) Input RGB images and depth maps for registration. (b/cld) Diffusion / Geometric Fused Feature maps of the input RGB images and depth maps. (e) Estimated corre- spondences.
23

---

Published as a conference paper at ICLR 2024
(a) Input RGB PC
(b) Ours
(c)
(a) Input RGB & PC
(b) Ours
(c) GT
Figure 10: Patch warping results. Based the estimated FreeReg correspondencess warp the 32 32 local RGB patches to their estimated corresponding positions point cloud. (a)The imput RGB and depth images. (byThe RGB patch warping results are based on the established correspondenees FreeReg, (c) The ground truth RGB image of the input depth map.
24


Reviews:
[{"review_id": "YF0uAyRIcr", "reviewer": "Reviewer_BaLi", "summary": "The paper studies the image-to-point-cloud registration problem. The idea is to utilize a diffusion model and ControlNet to generate diffusion features from input point cloud. For images and point clouds, the final features used for matching are composed of both a diffusion part and a geometric part in a weighted average fashion. The latter is extracted from FCGF to serve as geometric features. The pixel-to-point correspondences are then obtained by a NN matching with mutual check. Experiments show the empirical improvements in I2P matching in several benchmark datasets.", "questions": "1. Can the method work with only diffusion features from RGB and point clouds? e.g. Can the weighting between F_d and F_g be either 1 or 0?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- High-quality writing: The content is well-written, with a focus on clarity, coherence, and precision.\n\n- Improved results over baselines: The results outperform standard models, demonstrating significant enhancements in performance and accuracy.\n\n- Efficient feature distillation and cross-modality matching: Large model features are distilled effectively, facilitating feature matching across different modes for improved system performance.", "weaknesses": "- The paper primarily focuses on the design of an improved feature that serves as a unifying element for both the image and depth map domains. While this feature has shown remarkable efficacy within this specific context, it may not be directly transferable to other cross-modality problems. Adapting it to different cross-modality tasks would necessitate careful and tailored design to ensure its successful application.\n\n- Efficiency is indeed a concern as pointed out in the limitation section, as feature extraction via stable diffusion and ControlNet is a costly computation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Xqkgh6HIC", "reviewer": "Reviewer_zNJ2", "summary": "This paper introduces a image-to-point cloud registration framework. The key idea is to generate RGB image from point cloud and reconstruct depth image from RGB so that correspondences can be established between images of the same modality. Though the image generation of both directions are well studied, a naive implementation does not work well. For this reason, the authors first generate depth image from point cloud and then use intermediate feature maps in the depth-to-image ControlNet to establish semantic correspondence with the original image. At the same time, a depth map is generated from the original image and local geometric features extracted from the depth map are combined with the semantic features for better correspondences. Experiments are conducted on three datasets, including both indoor and outdoor scenes.", "questions": "1. Is there any way that can are readily to be tried for speed improvement?\n\n2. The residual rotation error is quite high. From a practical point of view, is the rotation error of 10 or 20 degrees a good threshold for recall?  What's the requirements of rotation accuracy in different application areas?\n\n3. The authors \"analyze\" the limitation of straightforward/direct implementations in the 4th and 5th paragraph of Introduction, but did not provide any experimental results to support supporting the conclusion.  For the task of this paper and the fusion of two kinds of features, these straightforward may also work well.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The idea of first generating images and point clouds from the other modality and then find correspondence in the same modality is interesting and the authors find practical ways to implement this idea.\n\n2.The performance is promising even without training on the target task with ground-truth correspondence.\n\n3. The paper is well written and the adequate ablation studies are conducted.", "weaknesses": "1. As mentioned by the author, inference speed is a limit of the proposed method and 11s per image is quite slow.  I hope that the author can provide their thoughts for further improvement of speed.\n\n2. Another limitation is that the performance is only comparable with the concurrent work 2D3D-MATR on the dataset RGBD-Scene-v2, while  I think that this is not a big problem and the proposed method has its own value. \n\n3. The residual rotation error seems quite large. I'd like to know what's the initial rotation error before registration?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9CB4GdTEoP", "reviewer": "Reviewer_fhKS", "summary": "Overall, the core idea of the paper is interesting, which considers leveraging diffusion networks to achieve feature enhancing in the task of image-point-cloud registration.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Overall, the core idea of the paper is interesting, which considers leveraging diffusion networks to achieve feature enhancing in the task of image-point-cloud registration.", "weaknesses": "1. The idea of using diffusion networks is interesting, but the way to use diffusion networks is to some extent trivial. It is not well-motivated  why you choose to use diffusion networks instead of any other pretrained feature extractor, such as those vision foundation backbones (DINO, SAM, CLIP, ...)?\n\n2. In Related Work (Image-to-point cloud registration), \"In contrast, FreeReg does not require task-specific\ntraining and finetuning and exhibits strong generalization ability to both indoor and outdoor scenes\". It seems this description is not solid, as you feature extractors (diffusion networks and depth estimation networks) are already trained on some datasets.\n\n3. The baseline comparison is a little bit weak. More recent and related works should be considered. And in Table S1, the proposed approach seems not solid outperform the counterpart 2D3D-Matr.\n[1] Corri2p: Deep image-to-point cloud registration via dense correspondence. TCSVT 2022.\n[2] EP2P-Loc: End-to-End 3D Point to 2D Pixel Localization for Large-Scale Visual Localization. ICCV 2023.\n[3] CFI2P: Coarse-to-Fine Cross-Modal Correspondence Learning for Image-to-Point Cloud Registration. arXiv 2023.\n[4] CoFiI2P: Image-to-Point Cloud Registration withCoarse-to-Fine Correspondences for Intelligent Driving. arXiv 2023.\n[5] End-to-end 2D-3D Registration between Image and LiDAR Point Cloud for Vehicle Localization. arXiv 2023.\n\n\n4. In Table 4, what is the performance under w=0 and w=1.0 ?\n\n5. The run time speed/memory comparison with other models is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)