Generative Point Cloud Registration
Authors: Haobo Jiang
Jin Xie
jian Yang
liang yu
Jianmin Zheng
Venue: ICML 2025 poster (2025)
Keywords: point cloud registration
Primary area: applications->computer_vision
TL;DR: We propose a novel Generative Point Cloud Registration paradigm, which bridges advanced 2D generative models with 3D matching tasks to enhance registration performance.

Abstract: In this paper, we propose a novel 3D registration paradigm, Generative Point Cloud Registration, which bridges advanced 2D generative models with 3D matching tasks to enhance registration performance. Our key idea is to generate cross-view consistent image pairs that are well-aligned with the source and target point clouds, enabling geometric-color feature fusion to facilitate robust matching. To ensure high-quality matching, the generated image pair should feature both 2D-3D geometric consistency and cross-view texture consistency. To achieve this, we introduce Match-ControlNet, a matching-specific, controllable 2D generative model. Specifically, it leverages the depth-conditioned generation capability of ControlNet to produce images that are geometrically aligned with depth maps derived from point clouds, ensuring 2D-3D geometric consistency. Additionally, by incorporating a coupled  conditional denoising scheme and coupled prompt guidance, Match-ControlNet further promotes cross-view feature interaction, guiding texture consistency generation. Our generative 3D registration paradigm is general and could be seamlessly integrated into various registration methods to enhance their performance. Extensive experiments on 3DMatch and ScanNet datasets verify the effectiveness of our approach.

Generative Point Cloud Registration
Haobo Jiang'
Jin Xie
Jian Yang?
Liang Yu*
Jianmin Zheng'
Abstract
Conventional 3D Registration
Generative 3D Registration
Soure
In this paper, we propose novel 3D registra- tion paradigm, Generative Point Cloud Registra- tion, which bridges advanced 2D generative mod- els with 3D matching tasks to enhance registra- tion performance. Our key idea to generate cross-view consistent image pairs that are well- aligned with the source and target point clouds, enabling geometry-colon feature fusion to facil- itate robust matching. To ensure high-quality matching, the generated image pair should fea- ture both 2D-3D geometric consistency and cross- view texture consistency. To achieve this, we in- troduce Match-Controlie a matching-spectie controllable 2D generative model. Specifically, it leverages the depth-conditioned generation ca- pability of ControlNet to produce images that are geometrically aligned with depth maps de- rived from point clouds, ensuring 2D-3D geomet- ric consistency. Additionally, by incorporating a coupled conditional denoising scheme and cou- pled prompt guidance, Match-ControlNe further promotes cross-view feature interaction, guiding texture consistency generation. Our generative 3D registration paradigm is general and could be seamlessly integrated into various registration methods to enhance their performance. Extensive experiments on 3DMatch and ScanNet datasets verify the effectiveness of our approach. [Code]
Geometry only Maanty-oolyMMtttttti Matoriin & EPooeEttinttoon Extination
Match- Match- Canth-CCntrreeer
Pooe Extimation
Generated Images
Figure I. Paradigm comparison of generative point cloud registration with conventional methods. Unlike geometry-only matching in previous methods, our approach introduces Match- ControlNet, a matching-specinf 2D generative model that gener- ates cross-view images pairs from point cloud data, providing rich color cues for enhanced geometric matching and pose estimation.
challenges like low overlap and noisy points still hinder its adoption in broader real-world scenarios.
Existing 3D registration methods be roughly catego- rized into traditional approaches and data-driven deep meth- ods. The traditional approaches include optimization- baaedd fine alignment methods (Besl & McKay, 1992; Yang et al., 2013), which iteratively perform least-squares pose opti- mization for precise aligmment, and handcrafted descriptor- based coarse alignment methods (Rusu al., 2008: 2009), which capture local geometry to establish correspondences for sypothesize-and-veed methods, whether end-to-ent (Yew & Lee, 2020; 2022) or descriptor-based (Huang et al., 2021; Qin et al., 2022; Jiang et al., 2023a), exploit the power of deep neural networks to learn discriminative deep 3D features for robust matching. These deep methods significantly enhance the quality of es- timated correspondences and improve registration accuracy.
registration. Deep registration
1. Introduction
Point cloud registration a problem finding the optimal rigid transformation, comprising 3D rotation and a 3D translation, which aligns the source and target point clouds precisely. It plays an important role in various downstream computer vision applications, such 3D reconstruction, Li- DAR SLAM, and object localization. However, real-world
Despite the impressive performance achieved by current point cloud registration methods, their robustness remains limited in challenging scenarios that contain low overlap, repetitive patterns, or noisy points. Recent RGB-D registra- tion studies (Yuan et al., 2023; Mu et al., 2024) have shown that incorporating rich texture and semantic cues from RGB images would significantly enhance the distinctiveness of point cloud descriptors, leading to improved matching accu- racy. However, in geometry-only point cloud registration, the RGB images corresponding to the point clouds are un- available, and existing methods rely solely on 3D geometric information for correspondence estimation and pose calcu-
'Nayyang Technological University, Singapore 'Nanjing Uni- versity, China Nankai University, China Aliibba Group, China. Correspondence to: Jianmin Zheng <ASJMZheng@ etu.
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Generative Point Cloud Registration
lation. This raises an interesting question: "Can we still leverage color information to enhance geometry-only point descriptors for enhanced 3D registration."
specific, pairwise image generation. It incorporates depth-conditioned generation, coupled conditional de- noising, and coupled prompt guidance to ensure that the generated image pairs maintain 2D-3D geometric consistency and cross-view texture consistency.
Motivated by this question and inspired by the recent suc- cesses of generative Al models (Ho et al., 2020; Song et al., 2020; Yang et al., 2023; Li et al., 2025; Rombach et al., 2022; Zhang et al., 2023; Jiang et al.. 2023b: Wang et al., 2024), we introduce Generative Point Cloud Registration, new 3D matching paradigm that bridges the task gap be- tween the 2D generative models and 3D matching tasks to enhance registration performance. Our key idea is to gen- erate cross-view image pairs that are well-aligned with the corresponding source and target point clouds. These images provide rich color information to complement geometric features, enabling more robust matching (see Fig. 1). Un- like prevalent 2D generative models that focus single image generation, our matching-speciffe image generation is pairwise. Importantly, to ensure high matching quality, the generated cross-view image pair should feature two key properties: 2D-3D geometric consistency and cross-view texture consistency. To achieve this, we introduce Match- ControlNet, a matching-specifice controllable 2D generative model.
Our Generative Point Cloud Registration framework is general and plug-and-play. Benefiting from our ef- fective zero-shot geometric-color feature fusion and XYZ-RGB fusion schemes, it can be integrated with var- ious 3D registration approaches to provide free-lunch color information, enhancing their performance.
2. Related Work
Traditional 3D Registration Methods. Traditional point cloud registration methods are typically categorized into coarse and fine registration approaches. Iterative Closest Point (ICP) (Besl & McKay, 1992), prominent fine regis- tration method, iteratively computes nearest-neighbor cor- respondences and performs least-squares optimization for pose estimation. Go-ICP (Yang et al., 2013) enhances ICP's robustness to initialization errors through a branch-and- bound (BnB) global search. Trimmed ICP (Chetverikov et al., 2002) further improves robustness by optimizing over minimal subsets to handle outliers. Additional variants like (Sharp et al., 2002: Fitzgibbon, 2003; Bae & Lichti, 2008; Gressin et al., 2013; Deng al., 2018) also demon- strate promising precision in fine alignment. Coarse regis- tration methods generally combine handcrafted geometric descriptors with robust pose estimators, such as RANSAC. (Johnson & Hebert, 1999) develops the spin image- based shape descriptors for surface matching and object recog- nition. USC (Tombari et al., 2010) improves the feature descriptors using shape context-awart unique local ref- erence frame to improve matching accuracy. SHOT (Salti et al., 2014) introduces 3D histogram based feature us- ing normal vectors to describe surface. PFH (Rusu et al., 2008) and FPFH (Rusu et al., 2009) constructs a discrimi- native and efficient local descriptor based on the oriented histogram with pairwise 3D representations. Other notable coarse methods, including (Mohamad al., 2014; 2015; Xu et al., 2019; Huang et al., 2017; Ge, 2017), have also achieved impressive registration precision.
leverages ControlNet's 's depth- conditioned generation capabilities to produce images geo- metrically aligned with depth maps (derived from the point cloud pairs), ensuring 2D-3D geometric consistency. Ad- ditionally, by incorporating coupled conditional denoising and coupled prompt guidance, Match-Come ControlNet enables effective cross- view image feature interaction, achieving mu- tual texture message passing and thereby enhancing cross- view texture consistency. Finally, we propose zero-shot geometric-color fusion mechanism that leverages pretrained large vision models (e.g., DINOv2 (Oquab et al., 2023) and Stable Diffusion (Rombach et al., 2022)) to extract discrimi- native zero-shot features of generated images for enhancing geometric descriptors via weighted concatenation.
It should be pointed out that Generative Point Cloud Registration framework can operate in both zero-shot and few-shot settings (with minimal fine-tuning samples), each providing valuable color information to enhance preci- sion. Moreover, our framework is general and can be inte- grated with various 3D registration methods to enhance their matching accuracy. Experiments on 3DMatch and ScanNet datasets validate the effectiveness of our proposed method. To summarize, contributions are as follows:
Learning-based Deep Registration Methods. Deep regis- tration methods primarily consist end-to-end approaches and deep deseriptor- based methods. For end-to-end ap- proaches, DCP (Wang & Solomon, 2019) introduces dif- ferentiable soft correspondences for SVD-basee pose es- timation. RPM-Net (Yew & Lee, 2020) incorporates the Sinkhorn layer and an annealing strategy to mitigate outlier inference. CEMNet and LatentCEM (Jiang et al., 202 la:b) formulate the 3D registration task as a Markov decision pro- cess, and introduce reinforcement learning-driven learning
We propose a new Generative Point Cloud Registration paradigm, aimed generating cross-view image pairs for both source and target point clouds, thereby provid- ing rich color information for effective geometric-color feature fusion and improved matching quality.
Unlike conventional single-image generation, we de- velop an effective Match- ControoCotll for matching-
2

---

Generative Point Cloud Registration
1. Zero-sbot Grometrie-Coler Festure Fusion
comprising rotation R SO(3) and translation t E R to align them precisely. The optimal rigid transformation is typically computed by solving:
Descriptors
seg, POST
Crees-vjew Image Generation
Pretrained Lange Fivion Models for. 30..
min R.t
Σ R- q"12 EEC
(I)
Macch Cow.rolohheee De
A.XYZ-GGA RGB Pesion
Cermepondrrce Eatimation
where C* denotes the ground-truth correspondences between source and target point clouds. However, C* is generally unknown in practical usage, and we need estimate set of putative correspondences through finding feature nearest neighbor among the pointwise geometric descriptors.
Color
Cloud
Registration
3.1. Motivation
Figure 2. Pipeline of Generative Point Cloud Registration. Given a source and a targel point cloud, we first apply Match-ControlNet generate their corresponding images. Next, employ either zero-shot geometric-colon feature fusion or XYZ-RGB fusion to create color-enhanced geometric descriptors, enabling high-quality correspondence estimation and robust pose estimation.
Recent RGB- D point cloud registration methods (Yuan et al., 2023; Mu et al., 2024) have demonstrated that RGB im- ages can significantly enhance geometry-only descriptors by providing rich color and semantic information. This enhancement facilitates the construction of higher-quality correspondences, leading to more robust registration.
framework for planning- based trial-and-error pose search- ing (Liu et al., 2024; 2023; Jiang et al., 2021c; 2022). RegTR (Yew & Lee, 2022) designs an effective transformer- based correspondence regression module, addressing large- scale indoor scene registration an end-to-end manner. For deep descriptor-basee methods, 3DMatch (Zeng al., 2017) employs Siamese 3D CNN to extract local geometric fea- tures for patchwise matching. FCGF (Choy et al., 2019) develops a fully convolutional network to learn dense 3D fea- tures for pointwise matching. Predator (Huang et al., 2021) introduces& a cross-attention transformer between point cloud pairs for overlap perception and robust registration. Geo- Transformer (Qin et al., 2022) integrates geometric embed- dings into the transformer, enhancing feature discrimination. Rol'r (Yu et al., 2023) designs iuation-invarian trans- formation to further improve the rotational robustness of geometric descriptors. Other methods (Wang et al., 2023; Bai et al., 2020; Li & Harada, 2022; Li et al., 2020; Choy et al., 2020; Chen et al., 2023; Fu et al., 2021; Ao et al., 2023) also demonstrate impressive performance in 3D regis- tration. Beyond traditional and barning-basse frameworks, this work introduces a new paradigm: Generative Point Cloud Registration. By integrating advanced 2D generative models with the 3D registration domain, our approach gener- ates complementary color information for input point cloud pairs, producing color-enhanced geometric descriptors to improve precision.
However, in the context of 3D matching tasks, focus geometry-only registration using pure point clouds, as the relevant RGB data is unavailable. To overcome this limitation, we introduce Generative Point Cloud Registra- rion, a general framework designed to generate high-quality RGB data for both point clouds, enabling geometric-color feature fusion for enhanced matching. Notably, unlike the conventional single-image generation focused by prevalent 2D generative models (Rombach et al., 2022; Zhang et al., 2023), our matching-specific image generation is pairwise, which should satisfy two key criteria:
(i) 2D-3D Geometric Consistency: should preserve the geometric structure and spatial layout of their respective point clouds to ensure accurate pixel-to- point correspondences and avoid introducing noise;
The generated images
(ii) Cross-view Texture Consistency: The generated image pair should maintain consistent textures for correspondences. Otherwise, inconsistent textures would reduce feature simi- larity of correspondences, leading to mismatches.
3.2. Zero-Shot Geometric Consistency Generation
In this section, we first address the 2D-3D geometric consis- tency generation through ControlNet (Zhang et al., 2023), a variant of Stable Diffusion (Rombach et al., 2022) with spatially localized image conditions (c.g. Canny edge maps and depth maps).
3. Approach
Stable Diffusion. Stable Diffusion is a widely used latent diffusion model for text-to-image generation. It operates within the latent space of a pretrained autoencoder, where a denoiser (x t, c) (conditioned on the timstamp t and tokenized text prompt c) gradually refines the noisy latent
Problem Setting, Given a pair of source and target point clouds P- = {p; E 3 | i = 1,....Ny .) and Q = {q, E R3 = 1,...,M,, point cloud registration seeks to re- cover their rigid transformation T (R, t) E SE(3),
3

---

Generative Point Cloud Registration
and target image denoising generation processes into a joint denoising pass, thereby enabling mutual texture message passing and promoting texture consistency (see Fig. 3).
Marco Phatch-Conie Controllee
Independent Goneration sol Fauturs Intrrction
Conmonnee
Based on this motivation, we establish Match-ControoNNe an improved ControlNet variant for matching-speciffe, con- ditional image generation. Following Sec. 3.2, we still take depth maps derived by point clouds as conditional images so as to inherit ControlNet's 2D-3D geometric consistency generation capability. Additionally, we introduce two key designs: coupled conditional denoising and coupled prompt guidance to achieve the cross-view texture consistency gen- eration. The details of these two designs are as follows:
Corsiitunt Coumetyy
Constotert Testore
Conatrtent Guomatry
Comuisenn Tusture
Figure 3. Instead of independently performing ControlNet to gen- erate source and target images, our Match- ControlNet integrates their denoising generation processes into a unified framework, fa- cilitating feature interaction (i.e., mutual texture message passing) and enhancing their cross-view view texture consistency.
feature X, to clean one for image decoding. The denoiser fol- lows a UNet architecture with an encoder, middle block, and skip-connected decoder, incorporating stacked transformer and residual modules. Each transformer module utilizes intra-image self-attention for contextual understanding and prompt-to-image cross-attention to guide generation.
Coupled Conditional Denoising. To achieve mutual tex- ture message passing. a straightforward approach is to build two denoisers and incorporate an additional cross-denoiser attention module to facilitate their message passing. How- ever, running two denoisers simultaneously is inefficient, and such significant architectural changes would require extensive model fine-tuning.
-orirel 2D-3D Geometric Consistency. Con- trolNet further equips the denoiser of Stable Diffusion with a learnable encoder copy for encoding the conditional image c, forming a conditional denoiser: (x;; t, c, cy). Conse- quently, the encoded condition features concatenated with the original encodings of noisy latent representations x, for conditional feature decoding via skip connections. Notably, ControlNet allows the use of depth maps as conditional inputs to generate RGB images that preserve geometric structures well-aligned with the provided depth prior. This capability perfectly aligns with our objective and motivates us to convert the source and target point clouds into their corresponding depth maps, Dp and De IR. Xxxx1 via the intrinsic matrix. Then, each produced depth map can be independently used condition ControlNet to produce the geometrically consistent image.
To enable effective cross-view view message passing without any finetuning (i.e., zero-shot), we propose an efficient coupled conditional denoising scheme for joint, interactive source and target image generations. Specifically,
expand the noisy latent representation x; (()) with shape [H' W", d) to coupled one x,"º with the extended shape |2H', W", dJ. Also, we vertically concatenate the source and target depth maps into coupled one Dpo E 22 x1, and further emply the ControlNet's zero convolutions to encode it into the condition features dpe E 1g2H" x xW xd Consequently, without any architectural modifications or parameter fine- tuning, the original conditional denoiser can be directly employed for our coupled conditional denoising:
Ea (x, P2: (t,,t,c,dp)) - x
(3)
3.3. Zero-Shot Texture Consistency Generation
forming the denoising Markov chain: xP
x,0 Here, the initial coupled latent representation x7.
Although the original ControlNet can generate source and target image pairs that are geometrically well-aligned with the given point cloud pair, the texture details of the corre- sponding regions between the generated image pair often differ as shown in Fig. 3 (left). This texture inconsistency primarily arises because the denoising processes for the source and target images operate independently as follows:
is sampled from a standard Gaussian distribution (/ (O I).
To further clarify the cross-view texture message passing during our coupled denoising process, we formulate the self- attention mechanism within the denoiser as SA(x," (x)
e (x e W (x? P W"),
softmax
(4)
Vd
és (x(' it, c, dp) + +x
(2)
where W W and W ddxd denote the projection matrices for queries, keys and values, respectively. Eq. illustrates that by coupling the source and target noisy latent representations, each feature element can establish long- range dependencies with all feature elements from both the source and target feature maps, allowing effective cross- view feature interaction and texture- aware message passing for promoting texture consistency generation.
Eo (x,,)) + x211
with each unaware of the colors produced by the other. Here, x/ E H''''''x denote the noisy latent rep- resentations corresponding to source and target images: dp,d, dg E H <W' xd represent the encoded features of depth maps Dp and De via optimized zero convolutions of ControlNet. This insight motivates us to combine source
4

---

Generative Point Cloud Registration
Coupled Prompt Guidance. Although the aforementioned coupled denoising generative mechanism has provided the essential components for texture consistency generation, the denoiser still fails to produce the image pair with expected cross-view texture consistency. The core reason is that the denoiser does not know what kind of image the user expects to generate, and we need to tell it what to do. Our impor- tant finding that when we use a specific prompt (named coupled prompt) as below to tell our Match- ControlNer to produce the vertically stacked images with consistent layout and elements:
Zero-ahoe paro-ahot
Finetune
Finetune
Finetuned
Figure 4. Compared to the zero-shot Match- ControlNer (top), the finetuned
can tend to achieve higher 2D-3D geometric consistency and the cress-view texture consistency. information from generated source and target images, Ip and Le. We provide two geometric-color fusion schemes:
"Generate two vertically stacked images that are captured from the different viewpoints in a same scene. The images should feature the same environment-whether indoor or outdoor, like living room, office, street, or natural land- scape-with very subtle differences between them. Overall, the layout and key elements remain the same.".
Zero-Shot Geometric-Color Feature Fusion. Inspired by the powerful RGB representations of large vision mod- els, we utilize them to directly extract zero-shot seman- tic features from the generated images. Specifically, we employ two widely-used vision foundation models: DI- NOv2 (Oquab et al., 2023) and Stable Diffusion (Rombach et al., 2022) for image encoding, achieving corresponding feature maps. These feature maps are then projected into the point cloud space using the camera intrinsic matrix, yield- ing pointwise color descriptors: (fpe*) } and (fqrh) for both source and target point clouds. Finally, we apply sim- ple weighted concatenation to combine the RGB descriptors with the geometric descriptors, resulting in fused descriptors as follows:
the denoiser can naturally be guided to recover consistent textures without any model fine-tune tuning. To the best of our knowledge, we are the first to uncover and utilize this inher- ent capability of the pre-trained ControlNet for zero-shot pairwise image generation enjoying both 2D-3D geometric consistency and cross-view texture consistency.
3.4. Few-Shot Consistency Fine-tuning
Although our zero-shot Conttcc--ootrolNeet above has demon- strated promising pairwise consistency generation capabili- ties, Fig. 4 shows that some corresponding regions may still exhibit geometric or texture inconsistency issues. To miti- gate it, we further propose a few-shot finetuning mechanism to improve the consistency generation quality of our Match- ControlNet. It's noted that we only finetune the learnable encoder copy of Match-Controlie eters to preserve the powerful generation ability of Stable Diffusion. Specifically, we first collect set of RGB-GC depth pairs, {(Ip, Dr), (Io, } (j denotes the sample in- dex), as the training data. For each sample pair, we then concatenate its depth maps and RGB images into coupled ones (Ipo, )) Finally, we use the loss function below to finetune the denoiser:
fpp = : ; - w) PCA((F.. E teo (6)
where - denotes the feature concatenation operator; w is the fusion weight and fgeo represents the geometric de- scriptors; PCA(-) denotes the principal component analysis function to compress the feature dimension of the color de- scriptor to fit that of the geometric descriptor. The same fusion scheme is also applied to the target point clouds. Notably, this zero-shot geometric-color fusion approach is general and can be applied a variety geometric descrip- tors, whether traditional or deep descriptors.
rather than the all param-
XYZ-RGB Fusion. This fusion scheme directly projects the generated source and target RGB images into the point cloud space. The resulting RGB values are then concatenated with the point coordinates of the point clouds, forming 6D color source and target point clouds, as shown in Fig. (left). These color point clouds are subsequently used as inputs to the color point cloud registration method, like ColorPCR (Mu et al., 2024), for 3D registration.
C C= B (. [||- (xP ,,, [|t-@@(xP"tt..doooll
(5)
where x represents the diffused latent representation of the coupled image Ipg: è denotes the token sequence of our coupled prompt. Our experiments show that even with a limited number of samples (~33), few-shot finetuning can effectively improves the quality of consistency generation.
4. Experiments
3.5. Chometric-Coloo Fused Point Descriptor
4.1. Experimental Setting
In this section, we focus on how to enhance the geometric representations of point clouds with the fuee-lunch color
Implementation Details. During the few-shot fine-tuning stage, we randomly select 3,000 sample pairs from the Scan-
5

---

Generative Point Cloud Registration
Table Comparison the methods on rotation, translation, and Chamfer distance on ScanNet (Dai al., 2017) benchmark dataset.
Rotation (deg)
Translation (om)
Chamfer (mum)
Accuracy 1
Error
Accuracy t
Error
Accuracy 1
Error
Methods
45
Mean
Med.
25
Mean
Med.
5
10
Mean
Med.
FPEH (Rusu al., 2009)
41.4
56.7
73.3
39.2
7.1
17.5
35.1
50.9
79.5
23.5
32.3
48.0
53.0
159.6
6,5
Lepard (Li & Harada, 2022)
63.3
75.5
84.1
24.9
3.3
31.3
56.4
72.3
48.4
8.1
51.6
69.1
73.0
89.2
0,9
RegTR (Yow & Lec, 2022)
72.5
83.8
94.1
10.2
2.3
44.3
65.6
80.0
27.7
5.8
61.0
76.6
80.9
54.0
0.5
RolTr (Yu et al., 2023)
70.0
77.5
83.7
62.3
75.1
45.6
58.8
72.4
75.4
94.1
0.6
FCGF (Choy et al., 2019)
78.9
84,2
87.5
19.4
1.5
55.3
70.7
79.7
37.8
4.3
67.3
78.2
80.3
100.7
0.4
Generative PCGEDINO,s
81.0
86.2
89.4
16.5
1.4
57.3
72.6
80.9
33.9
4.0
68.9
79.5
81.5
97.4
0.3
Generative FCGFSD
82,9
90.0
94.4
56.4
73.0
82.7
21.7
4.1
67.7
80.9
83.7
66.0
0.4
Improvement
4.0
6.9
11.0
0.1
2.0
2.3
3.0
16.1
0.3
2.7
3.4
34.7
0.1
Predator (Huang al., 2021)
64.3
75.2
82.6
26.3
3.2
30.1
54.8
69.2
48.7
8.4
50.8
66.9
70.6
93.2
1.0
Generative PNNOa
67.0
78.0
87.2
19.0
3.0
30.7
56.0
70.3
41.4
8.1
52.0
67.8
71.3
79.2
0,9
Generative PredatorSD
70.7
81.3
88.7
17.0
33.0
59.4
73.3
7.5
54.7
70.8
74.2
72.5
0.8
Improvement
6.4
6.1
9.3
0.4
2.9
4.6
4.1
12.1
0.9
3.9
3.9
3.6
20.7
0.2
GeoTirans QQin et al., 2022)
71.5
78.0
83.4
26.2
2.0
48.4
65.2
74.6
51.9
5.2
62.0
72.5
75.0
97.3
0.5
Generative Branss DINO,,
74.3
81.0
87.6
19.7
50.8
67.4
76.0
41.8
4.9
63.7
73.9
76.2
86.2
0.4
Generative GeoTTrans 8D
77.2
84.0
89.9
16.5
68.7
78.4
35.6
65.2
76.1
78.7
71.0
0,4
Improvement"
5.7
6.0
6.5
9.7
0.2
2.9
3.5
3.8
16.3
0.4
3.2
3.6
3.7
26.3
0.1
Match-Controlin Cet
Overlap: 17.5% Inller Ratio: 0.5%
Overlap: 17.5% Inller Radio: 26.7%
Target
Sdurce
Predator
Generative Predator
The visualization of the generated RGB image pairs and the formed color source and target point clouds; Right: In low-overlap cases, the original Predator struggles with registration. By contrast, the Generative Predator, enhanced with generated color information, successfully align them well.
Figure 5. Left:
Net training set (Dai et al., 2017) for model fine-tuning. Following the default fine-tuning configuration of Con- trolNet (Zhang et al., 2023), we adopt the AdamW opti- mizer (Loshchilov, 2017) with learning rate of le-5 and set the training epoch 10. The code for this project is conducted implemented in PyTorch, and all experiments on server equipped with an Intel i5 2.2 GHz CPU and a TITAN RTX GPU. In our experiments, we integrate our zero-shot geometric-color fusion (Sec. 3.5) with three preva- lent deep geometric descriptors: FCGF (Choy et al., 2019), Predator (Huang et al., 2021), and Geo Transformer (Qin al., 2022), resulting in corresponding color-enhanced variants: Generative FCGF, Generative Predator, and Generative Geo Trans for method evaluation. Additionally, to validate our XYZ-RGB fusion scheme, we replace the real color point clouds (with actual RGB values) used by ColorPCR (Mu et al., 2024) with our generated color point clouds (with synthesized RGB values), forming Generative ColorPCR for 3D matching.
Chamfer error, including the accuracy across varying thresh- olds and mean/median errors, for performance evaluation.
4.2. Comparison with Existing Methods
Evaluation on ScanNet. We first perform model evaluation on a widely-used, large-scale indoor benchmark dataset, ScanNet (Dai et al., 2017). We follow the official data split to divide this dataset into the training, validation, and test- ing subsets, and construct view pairs by sampling image pairs that are 50 frames apart. Compared to the 20-frame separation used in (El Banani et al., 2021; Yuan et al., 2023), our approach with a 50-frame separation further reduces the overlap ratio (i.e., lower overlap), thereby increasing the registration difficulty. We compare our method against with one representative traditional deseriptor: FPFH (Rusu et al., 2009), one scene-level end-to-end registration net- work: RegTR (Yew & Lee, 2022), and five deep descriptors: FCGF (Choy et al., 2019), Predator (Huang et al., 2021). GeoTrans (Qin et al., 2022), Lepard (Li & Harada, 2022),
Evaluation Metric. Following (El Banani et al., 2021; Yuan et al., 2023), we use rotation error, translation error, and
and RolTr (Yu et al., 2023). We adopt RANSAC-50k as the pose estimator for FPFH, Lepard and RolTr, and select
6

---

Generative Point Cloud Registration
Table Comparison the methods on rotation, translation, and Chamfer distance on 3DMatch (Zeng et al.. 2017) benchmark dataset.
Rotation (deg)
Translation (om)
Chamfer (num)
Accuracy 1
Error
Accuracy t
Error
Accuracy t
Error
Methods
10
45
Mean
Med.
25
Mean
Med.
5
10
Mean
Med.
FPFH (Rusu al., 2009)
69.1
82.9
91.2
15.0
3.1
25.8
53.9
75.1
37.4
9.1
52.5
74.2
79.2
57.6
0.9
Lepard (Li & Harada, 2022)
84.3
91.0
94.1
11.1
2.1
43.1
75.2
88.9
21.8
5.8
72.1
88.3
90.5
45.3
0.4
RegTR (Yow & Lec, 2022)
86.2
92.1
97.2
5.7
1.6
55.0
77.6
88.9
18.8
4.6
75.4
88.2
91.3
40.0
0.3
RolTr (Yu et al., 2023)
86.3
91.1
93.8
11.1
51.2
77.4
89.1
20.5
4.9
75.2
88.5
90.6
50.1
0,4
FCGF (Choy et al., 2019)
90.4
93.7
94.8
9.4
1.4
53.4
79.3
91.0
19.2
4.7
76.7
90.8
92.4
40.3
0.4
Generative PCGEDINO,s
91.5
94.3
95.3
8.5
1.4
53.6
79.3
91.5
18.1
4.6
77.5
91.1
92.7
41.1
0.4
Generative FCGFSD
94.3
96.7
98.1
4.5
54.3
81.5
93.1
12.5
4.7
78.2
92.9
94.6
37.7
0.4
Improvement
3.9
3.0
3.3
4.9
0.0
0.9
2.2
2.1
6.7
0.0
2.1
2.2
2.6
0.0
Predator (Huang al., 2021)
85.0
91.5
94.2
10.5
2.0
42.1
72.5
87.1
22.6
5.8
71.2
85.8
88.6
45.0
0.5
Generative PNoOa
88.1
94.8
96.9
6.2
188
44.7
73.9
88.4
15.5
5.6
72.4
87.7
90.8
33.1
0.4
Generative PredatorS
88.6
94.6
97.0
45.7
74.5
89.1
15.7
5.5
73.3
88.3
90.9
40.4
0.4
Improvement
3.6
3.1
2.8
4.6
0.2
3.6
2.0
2.0
7.1
0.3
2.7
2.5
2.3
11.9
0.1
Geo'Tran (Qin et al., 2022)
88.9
91.8
93.3
12.0
1.4
59.8
81.0
90.1
24.6
4.0
79.2
89.0
90.6
53.3
0.3
Generative GeoTrans DINO,,
90.2
93.2
95.2
8.9
1.4
61.0
83.1
90.4
16.9
3.9
80.4
89.7
91.7
36.9
0.3
Generative GeoTTrans SD
91.5
94.3
96.2
7.6
1.4
61.3
82.9
90.9
17.2
3,9
81.5
90.1
92.3
37.3
0.3
Improvement" ?
2.6
2.5
2.9
4.4
0.0
1.5
2.7
0.8
7.7
0.7
2.3
1.1
1.7
16.4
0.0
SC2PCR (Chen et al., 2022). RANSAC-50k and LGR for (Generative) FCGF, (Generative) Predator, and (Generative) GeoTrans (Qin et al., 2022), to validate the robustness of our generative 3D registration paradigm across different pose estimators. Table 1 demonstrates that enhanced by the free-lunch color information generated by our Match- ControlNet, all generative versions of FCGF, Predator, and GeoTrans achieve significant performance improvements, such as 6.9% 1 of Generative FCGF on 45 @ Rotation met- ric. These confirm the generality and effectiveness of our proposed generative point cloud registration paradigm. Ad- ditionally, we find that compared to the DINOv2 image encoding, Stable Diffusion can capture more discriminative representations and achieve higher precisions.
geometric consistency and cross-view texture consistency (geo+tex) through our Contrl-ControNee results in higher registration accuracy. This improvement is due to the addi- tional benefit of consistent textures and colors, which further facilitate accurate correspondence identification. Addition- ally, we observe that the generated images with only 2D-3D geometric consistency can also bring performance gain in some criteria. We attribute it to that DINOv2 and Stable Dif- fusion can extract powerful semantic representations, miti- gating the feature inconsistency of correspondences caused by the texture difference and thereby aiding correspondence identification. Furthermore, age pair for given source and target point clouds in Fig. 5 (left). It shows that our Match-Controlle producing high-quality image pairs with consistent 2D-3D geometry and cross-vies texture.
visualize the generated im-
is capable of
Evaluation on 3DMatch. We next evaluate our method on 3DMatch (Zeng et al., 2017), another widely-used bench- mark dataset for 3D registration. We follow (El Banani et al., 2021; Yuan al., 2023) as in ScanNet to produce the pair- wise samples. Also, we increase the view separation from 20 to 40, resulting in point cloud pairs with lower overlap to increase the registration challenge. Table. 2 demonstrates that by incorporating FCGF, Predator, and Geo'Trans into our generative point cloud registration framework, their gen- erative variants also consistently achieve the performance gain, validating the efffectiveness of our proposed paradigm.
Zero-Shot vs Finetuning. We further investigate the perfor- mance of settings. As shown in the second block of Table 3, both ap- proaches yield substantial improvements over FCGF, More- over, because the finetuned Match ControlNer benefits from task-specific training, it consistently achieves higher regis- tration accuracy than the zero-shot version. Notably, even few-shot finetuning with as few as 1K samples yields clear performance gains. Increasing the number of finetuning samples (e.g., to 3K or 5K) provides additional improve- ments; however, models trained on 3K or 5K samples show comparable registration accuracy in practice. Hence, we adopt 3K samples as our default finetuning configuration.
both zero-shot and finetuned
4.3. Ablation Studies and Analysis
Effectiveness of Match-ControoNNt. We first evaluate the performance contribution of our developed Match- ControlNer: (i) The top block of Table demonstrates that, compared to using generated image pairs with only 2D- 3D geometric consistency (geo), incorporating both 2D-3D
Zero-Shot Geometric-Cole conduct ablation studies on the zero-shot geometric-color feature fusion described in Eq. 6. As shown in the fourth block of Table 3, Generative Transans exhibits varying
Feature Fusion. We next
7

---

Generative Point Cloud Registration
Table 3, Ablation studies on 3DMatch (Zeng et al., 2017) dataset. (*) denotes the default configuration.
Rotation (deg)
Translation (cm)
Chamfer (mwn)
Accuracy
Error]
Accuracy
Error
Accuracy
Error
Methods
10
45
Mean
Med.
10
25
Mean
Med.
5
10
Mean
Med.
FCGF
90.4
93.7
94.8
9.4
1.4
53.4
79.3
91.0
19.2
4.7
76.7
90.8
92.4
40.3
0.4
Generative FCG
(geo)
92.4
96.1
97.8
5.2
1.5
53.3
79.2
91.7
13.5
4.8
75.6
91.1
93.1
35.8
0.4
Generative FCGEE (geo + tex)
94.3
96.7
98.1
4.5
1.4
54.3
81.5
93.1
12.5
4.7
78.2
92.9
94.6
37.7
0.4
Generative FCGF (zero-shot)
92.4
96.1
97.3
5.4
1.5
54.3
80.3
92.4
13.0
4.6
77.3
92.1
94.0
33.7
0.4
Generative FCGESD (finetuning)
94.3
96.7
98.1
4.5
54.3
81.5
93.1
12.5
78.2
92.9
94.6
37.7
0.4
Finetune (0samples=1000)
93.3"
96.8
98.1
4.6
1.4
$4.2
80.5
92.5
12.4
4.7
78.2
92.5
94.7
"32.8
0.4
Finetune (05000
96.7
98.1
4.5
81.5
12.5
4.7
78.2
92.9
94.6
37.7
0.4
Finetune (#sapples=5000)
93.6
97.2
98.0
4.4
1.5
54.0
80.8
92.7
11.9
4.5
77.7
92.9
94.2
32.3
0.4
ColorPCR (Mu et 2024)
79.9
84.6
88.9
16.5
1.8
48.3
69.6
82.2
41.8
5.2
66.6
80.6
83.3
81.1
0.5
Generative ColorPCR
83.6
89.8
93.2
12.0
1.9
47.3
73.3
86.7
28.3
5.3
70.3
85.7
88.2
59.1
0.4
Improvement
3.7
5.2
4.3
4.5
3.7
13.5
0.1
3.7
5.1
22.0
0.1
Color feut. dim. drgh 16
90.8
93.9
96.3
7.6
1.4
62.2
83.4
90.7
18.1
3,9
81.5
89.7
91.7
41.8
0.3
Color feat. dim. drgb = 32
91.5
94.3
96.2
7.6
1.4
61.3
82.9
90.9
17.2
3.9
81.5
90.1
92.3
37.3
0.3
Color feat. dim, drgb = 64
91.5
94.3
96.2
7.6
1.4
61.3
82.9
90.9
17.2
3.9
81.5
90.1
92.3
37.3
0.3
Color fent. dim. drgh 128
91.0
94.3
96.0
8.1
1.4
62.0
83.4
91.6
18.1
3.8
81.5
90,4
92.4
41.2
0.3
Fusion weight
93.3
96.6
6.9
77.0
86.5
20.8
4.7
74.6
85.9
88.5
50.5
0.4
Fusion weight 0.25
90.7
95.5
97.2
6.1
1.5
57.9
81.5
89.8
16.5
4.2
78.6
89.2
92.0
40.4
0.3
Fusion weight = 0.50*
91.5
94.3
96.2
7.6
1.4
61.3
82.9
90.9
17.2
3.9
81.5
90.1
92.3
37.3
0.3
Fusion weight w 0.75
89.2
92.7
93.9
10.9
1.4
60.5
81.7
90.6
22.6
4.0
79.5
89.3
91.4
49.5
0.3
Fusion weight 1.0
89.0
91.8
93.3
12.0
1.4
59.9
81.1
90.2
24.6
4.0
79.3
89.1
90.8
53.3
0.3
tive ColorPCR with the synthesized color even outperforms the original ColorPCR with the real color. This advantage is attributed not only to the high-quality pairwise image generation provided by our Match-ControlNet but also to several key benefits of our generated XYZ-RGB data over real XYZ-RGB data: (i) Mitigating calibration errors: As shown in Fig. 6 (left), some real RGB-D data would suffer from calibration errors, which may lead to misalignment in the colored point clouds. By contrast, our framework. benefiting from the powerful 2D-3D consistency generation ability, effectively reduces such calibration errors, producing higher-qualine colored point clouds and enabling more ac- curate matching. (ii) Mitigating lighting challenges: Fig. 6 (right) shows that RGB data from real-world conditions can degrade under poor lighting, negatively impacting RGB-D matching performance. Our generative point cloud registra- tion framework, however, can generate images with consis- tent lighting conditions, independent of real-world lighting issues, thus enhancing the overall lighting robustness.
Figure Our Match ControlNer effectively mitigates calibration errors and lighting challenges commonly encountered in real- weal-world RGB-D data, thereby improving the matching precision of color point cloud registration methods.
registration performance under different color feature di- mensions, drgb E {16, 32, 64, 128}. We observe that a very small color feature dimension (e.g., drgs = 16) degrades performance due to limited semantic representational capac- ity, while excessively large dimensions do not yield signif- icant performance gains. Therefore, balance inference efficiency with registration precision, we set dryb 64 as our default setting. Additionally, in the fifth block of Table 3, we investigate performance variations with different fusion weights w E {0.0, 0.25, 0.50, 0.75, 1.0), where a larger w places more emphasis on the geometric descriptors (see Eq. 6). Our results indicate that both overly high w (which overemphasizes geometry) and overly low w (which overem- phasizes color) lead degraded registration accuracy. By contrast, balanced weight (e.g., = 0.50) achieves higher performance. As result, we adopt = 0.50 as our default hyperparameter configuration.
5. Conclusion
We have introduced a novel 3D registration paradigm, gen- erative point cloud registration, which effectively lever- ages advanced 2D generative models to augment geometry- only 3D registration. To this end, we developed Match- ControlNet, a matching-specific variant of ControlNet de- signed to synthesize paired RGB images for both source and target point clouds. By integrating generation from ControlNet, coupled conditional denoising,
XYZ-RGB Fusion. We finally evaluate the effectiveness of the XYZ-RGB fusion (see Sec. 3.5). The third block in Table 3 demonstrates that, on the 3DMatch dataset, Genera-
8

---

Generative Point Cloud Registration
and coupled prompt guidance, these generated RGB image pairs preserve both 2D-3D geometric consistency and cross- view texture consisteney, thereby facilitating high-quality 3D matching. Notably, our generative framework is general and can be incorporated into a variety of registration meth- ods to improve their performance. Extensive experiments demonstrate the effectiveness of the proposed paradigm.
Choy, C., Park, J., and Koltn, V. Fully convolutional geometric features. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8958- 8966, 2019.
Choy, C., Dong, W., and Koltun, V. Deep global registration. In CVPR, 2020.
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., and Nießner, M. Scannet: Richly-annotated recon- structions of indoor scenes. In CVPR, 2017.
Impact Statement
This paper presents work whose goal is to advance research in point cloud registration. While potential societal implica- tions may exist, we do not find any that necessitate explicit discussion here.
Deng. H., Birdal, T., and Ilie, S. Ppfnet: Global context aware local features for robust 3d point matching. In CVPR, 2018.
Acknowledgments
El Banani, M., Gao, I.., and Johnson, J. Unsuperviser&&r: Unsupervised point cloud registration via differentiable rendering. In CVPR, 2021.
This research is supported by the RIE2025 Industry Align- ment Fund Industry Collaboration Projects (IAF-ICP) (Award E0026,, administered by S*STAR, as well as supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL).
Fitzgibbon, A. W. Robust registration of 2D and 3D point sets. Image and vision computing, 2003.
Fu, K., Liu, S., Luo, X., and Wang, M. Robust point cloud registration framework based on deep graph matching. In CVPR, 2021.
References
Ao, S., Hu, Q. Wang, H., Xu, K., and Guo, Y. Buffer: Balancing accuracy, efficiency, and generalizability in point cloud registration. In CVPR, 2023.
Ge, X. Automatic markerless registration of point clouds with semantic-keyponn- 4-points congruent sets. ISPRS Journal of Photogrammetry and Remote Sensing, 2017.
Bac, K.-H. and Lichti, D. D. A method for automated registration of unorganised point clouds. Journal of Pho- togrammetry and Remote Sensing, 2008
Gressin, A., Mallet, C., Demantké, J., and David, N. To- wards 3D lidar point cloud registration improvement us- ing optimal neighborhood knowledge. ISPRS journal of photogrammetry and remote sensing, 2013.
Bai, X., Luo, Z., Zhou, L., Fu, H., Quan, L.,, and Tai, C.LL D3feat: Joint learning of dense detection and description of 3d local features. Proceedings of the IEFF/VVF Conference on Computer Vision and Pattern Recognition, pp. 6359-6367, 2020.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. NeurIPS, 2020.
Huang, J., Kwok, T.-H,, and Zhou, C. V4PCS: Volumet- ric 4PCS algorithm for global registration. Journal of Mechanical Design, 2017.
Besl, P. and McKay, N. D. Method for registration of 3-D shapes. Sensor fusion IV: control paradigms and data structures, volume 1611, pp. 586-606, 1992.
Huang, S., Gojeic, Z., Usvyatsov, M., Wieser, A., and Schindler, K. Predator: Registration of 3d point clouds with low overlap. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 4267-4277 2021.
Chen, S., Xu, H., Li, R., Liu, G., Fu, C.-W., and Liu, S. Sira- per: Sim-to-real adaptation for 3d point cloud registration. In ICCV, 2023.
Chen, Z., Sun, K., Yang, F., and Tao, W. Sc2-per: A second order spatial compatibility for efficient and robust point cloud registration. CVPR, 2022.
Jiang, H., Shen, Y., Xie, J., Li, J., Qian, J., and Yang, J. Sampling network guided cross- cross-entropy method for unsu- pervised point cloud registration. In ICCV, 2021a.
Chetverikov, D., Svirko, D.,. Stepanov, D., and Krsek, P. The trimmed iterative closest point algorithm. In Ob- ject recognition supported by user interaction for service robots, 2002.
Jiang, H., Xie, J., Qian, J., and Yang, J. Planning with learned dynamic model for unsupervised point cloud reg- istration, arXiv preprint arXiv:2 108.63.0 2021b.

---

Generative Point Cloud Registration
Jiang, H., Xie, J., and Yang. J. Action candidate based clipped double q-learning for discrete and continuous action tasks. In AAAI, 2021c
Mu, J., Bie, L., Du, S., and Gao, Y. Colorper: Color point cloud registration with multi-stage geometric-color fu- sion. In CVPR, 2024.
Jiang, H., Li, G., Xie, J., and Yang, J. Action candidate driven clipped double q-learning for discrete and continu- ous action tasks. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F, EI- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0711 2023.
Jiang. H., Dang, Z., Wei, Z., Xie, J., Yang. J., and Salz- mann, M. Robust outlier rejection for 3d registration with variational bayes. In CVPR, 2023a.
Qin, Z., Yu, H., Wang, C., Guo, Y., Peng, Y., and Xu, K. Geometric transformer for fast and robust point cloud registration. In CVPR, 2022.
Jiang, H., Salzmann, M., Dang. Z.., Xie, J., and Yang, Se (3) diffusion model-based point cloud registration for robust 6d object pose estimation. NeurIPS, 2023b.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Johnson, A. E. and Hebert, M. Using spin images for effi- cient object recognition in cluttered 3d scenes. IEEE Transactions on pattern analysis and machine intelli- gence, 1999.
Rusu, R. B., Blodow, N., Marton, Z. C., and Beetz, M. Aligning point cloud views using persistent feature his- tograms. In IROS, 2008
Rusu, R. B., Blodow, N., and Beetz, M. Fast point feature histograms (fpfh) for 3d registration. In ICRA, 2009.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Salti, S., Tombari, F., and Di Stefano, L. Shot: Unique sig-
natures of histograms for surface and texture description. Computer Vision and Image Understanding. 2014.
Li, J. Zhang, C., Xu, Z., Zhou, H., and Zhang, C. Iter- ative distance-aware similarity matrix convolution with mutual -uutual-supervied point climination for efficient point cloud registration. In ECCV, 2020.
Sharp, G. C., Lee, S. W., and Wehe, D. K. ICP registration using invariant features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002.
Li, M., Yang. T., Kuang, H., Wu, J., Wang. Z., Xiao, X., and Chen, C. Controlnet++: Improving conditional controls with efficient consistency feedback. In ECCV. 2025.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er-
mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint urXiv:2011.13455 2020.
Li, Y. and Harada, T. Lepard: Learning partial point cloud matching in rigid and deformable scenes. In CVPR, 2022.
Tombari, F., Salti, S., and Di Stefano, L. Unique shape context for 3d data description. In Proceedings of the ACM workshop on 3D object retrieval, 2010.
Liu, S., Zhou, Y., Song, J., Zheng, T., Chen, K., Zhu, T., Feng, Z., and Song, M. Contrastive identity-aware learn- ing for multi-agent value decomposition. In AAAI, 2023.
Wang, H., Liu, Y., Wang, B., Sun, Y., Dong, Z., Wang, W. and Yang, B. Freereg: Image-to-pon to-point cloud registration leveraging pretrained diffusion models and monocular depth estimators. ICLR, 2024.
Liu, S.. Song. J., Zhou, Y., Yu, N., Chen, K., Feng, Z., and Song, M. Interaction pattern disentangling for multi-agent reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Wang, W., Mei, G., Ren, B., Huang, X., Poiesi, F., Van Gool, L., Sebe, N., and Lepri, B. Zero-shot point cloud registra- tion. arXiv preprint arXiv:23 230333, 2023.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint 171 2017.
Wang, Y. and Solomon, J. M. Deep closest point: Learning representations for point cloud registration. In ICCV, 2019.
Mohamad, M., Rappaport, D., and Greenspan, M. General- ized 4-points congruent sets for 3D registration. In 3DV, 2014.
Xu, Y., Boerner, R., Yao, W., Hoegner, L., and Stilla, U. Pair- wise coarse registration of point clouds urban scenes using voxel-based planes congruent sets. ISPRS journal of photogrammetry and remote sensing, 2019.
Mohamad, M., Ahmed, M. T., Rappaport, D., and Greenspan, M. Super generalized 4pcs for 3D registration. In 3DV, 2015.
10

---

Generative Point Cloud Registration
Yang, J. Li, H., and Jia, Y. Go-ICP: Solving 3D registration efficiently and globally optimally. In ICCV, 2013.
Yang. L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., and Yang, M.-H. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 2023.
Yew, Z. J. and Lee, G. H. RPM-Net: Robust point matching using learned features. In CVPR, 2020.
Yew, Z. J. and Lee, G. H. Regtr: End-to-end point cloud correspondences with transformers, In CVPR, 2022.
Yu, H., Qin, Z., Hou, J., Saleh, M., Li, D., Busam, B., and llie, S. Rotation-invarian transformer for point cloud matching, In CVPR, 2023.
Yuan, M., Fu, K., Li, Z., Meng. and Wang, M. Pointmbf: A multi-scale bidirectional fusion network for unsuper- vised rgb-d point cloud registration. In ICCV, 2023.
Zeng, A., Song, S., Nicßner, M., Fisher, M., Xiao, J., and Funkhouser, T. 3dmatch: Learning local geometric de- scriptors from rgb-d reconstructions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1802-1811, 2017.
Zhang, I.., Rao, A., and Agrawala, M. Adding conditional control text-to-image diffusion models. In ICCV, 2023.

---

Generative Point Cloud Registration
A. More Quantitative Analysis
To further validate the effectiveness our Generative Point Cloud Registration paradigm, we integrate the handcrafted geometric descriptor, FPFH, into our framework, forming Generative FPFH. As shown in Table 4, this generative variant achieves sigmificant performance improvement over the baseline FPFH on the 3DMatch benchmark dataset, regardless of encoding used.
whether DINOv2 encoding
Table 4. Comparison of the methods rotation, translation, and Chamfer distance on 3DMatch (Zeng al., 2017) benchmark dataset.
Rotation (deg)
Translation (cm)
Chamfer (mm)
Accuracy 1
Error
Accuracy
1
Error
Accuracy 1
Error
Methods
45
Mean
Med.
10
25
Mean
Med.
5
Mean
Med.
FPEH (Rusu et al., 2009)
69.1
82.9
91.2
15.0
3.1
25.8
53.9
75.1
37.4
9.1
52.5
74,2
79.2
57.6
0.9
Generative FIFH
83.7
91.4
95.9
7.9
2.1
33.9
64.0
80.5
24.7
6.9
62.7
80.6
83.2
50.0
0.6
Generative FPFHSD
94.7
96.9
2.0
85.5
18.5
85.7
88.8
41.2
Improvement 1
19.1
11.8
5.7
8.9
1.1
11.8
15.0
10.4
18.9
2.5
14.4
11.5
9.6
16.4
0.3
B. More Visualization Results of Match-ControlNet
In 7, we present additional visualization results the RGB image pairs generated by our ChutchCControNNet along with the corresponding colorized source and target point clouds. Furthermore, Fig.8 visualizes the image pairs generated by our zero-shot Match- Cotto-,ontrolNee, while Fig. 9 showcases image pairs produced by the finetuned Match-ControlNet.
Match- ControlNett
Match- Conttooonn
Match- ControlNet
Match ControlNet
Figure 7. More visualization results of the generated RGB image pairs and the formed color source and target point clouds.
12

---

Generative Point Cloud Registration
Zero-Shot Conto--onntroleet Generation
Figure Source and target image generation via zero-shot Couth--CnntrolNet without any finetuning.
13

---

Generative Point Cloud Registration
Finetuned Match-ControlNet Generation
Figure 9, Source and target image generation via finetuned Match-Controlle ControlNet.
14


Reviews:
[{"review_id": "slM9lML5jg", "reviewer": "Reviewer_synA", "summary": "This paper proposes a new 3D Registration method, Generative Point Cloud Registration, which connects advanced 2D generative models with 3D matching tasks to improve registration performance. The key idea in this paper is to generate cross-view consistent image pairs that are well aligned with source and target point clouds, so as to achieve geometric-color feature fusion to promote robust matching. Experimental results show that it can be seamlessly integrated into various registration methods to enhance their performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Discussion on applicability in outdoor LiDAR scenes.**  \n**A1:** We sincerely appreciate this insightful comment. Our current Match-ControlNet indeed targets leveraging depth maps rather than LiDAR data for image generation. Compared to forward-facing depth maps, outdoor LiDAR point clouds provide omnidirectional (360-degree) scans that cannot be directly represented as conventional single-viewpoint depth maps due to their inherent multi-directional nature. In this rebuttal, we demonstrate the feasibility of partially projecting LiDAR points from predefined viewpoints to produce sparse depth maps for Match-ControlNet generation, yielding promising preliminary generation results as illustrated in the figures (https://anonymous.4open.science/r/rebuttal-688D/outdoor_vis.pdf). For future work, we plan to comprehensively address omnidirectional scans by employing multi-view depth maps or equirectangular images (a 2D image representation of LiDAR data) to adapt our Match-ControlNet to LiDAR point clouds. We will include the above discussion into our revised manuscript.\n\n**Q2: Consistency of texture information under low overlap?**  \n**A2:** Our texture consistency under low overlap can be largely ensured by following four aspects: **(i) Extensive pre-training knowledge:** Our Match-ControlNet effectively unlocks the intrinsic zero-shot texture consistency capabilities of foundation models, which are derived from large-scale data pre-training. We believe this extensive training provides a strong foundation for ensuring texture consistency across diverse real-world challenges, including low-overlap scenarios.\n **(ii) Joint image-level and prompt-level texture consistency enhancement:** Our coupled conditional denoising mechanism and the coupled prompt guidance jointly prompt the texture consistency by incorporating both image-level texture consistency interaction and the prompt-level texture consistency guidance, significantly enhancing the texture coherence in low-overlap settings; **(iii) Effective texture consistency fine-tuning mechanism:** We introduce a few-shot consistency fine-tuning strategy (see Sec. 3.4) that requires only a small number of samples to further enhance the generation robustness of Match-ControlNet, particularly in challenging scenarios; \n**(iv) Extensive validation under low-overlap conditions:** To validate the robustness of our method under low-overlap conditions, we significantly increased the viewpoint separation (from 20 to 50 degrees on ScanNet and from 20 to 40 degrees on 3DMatch) as detailed in Line 311 (right column) and Line 362 (left column). Both quantitative and qualitative results (Fig. 7, first image; Fig. 8, third column) demonstrate reliable texture consistency. Additionally, we tested our method on in-the-wild, low-overlap data captured by a mobile phone. The resulting generations (https://anonymous.4open.science/r/rebuttal-688D/wild_vis.pdf) further confirm the robustness of our approach (please refer to A4@P29t for more details).\n\n\n**Q3: Reference Issue.**  \n**A3:**  We thank the reviewer for pointing out the missing FREEREG reference. We will include and thoroughly discuss this work in the revised manuscript. \n\n**Q4: Code availability.**  \n**A4:**  We will release our implementation publicly upon acceptance to facilitate reproducibility and enable broader verification by the community.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The proposed method is of great significance in the field of point cloud registration, and an innovative idea of point cloud registration is proposed.", "theoretical_claims": "There is no Theoretical Claim.", "experimental_designs_or_analyses": "The experimental setting was adequate and reasonable, and the only deficiency was that the analysis was not carried out in the outdoor scene, regardless of whether the results were good or bad.", "relation_to_broader_scientific_literature": "This paper presents a new idea in the field of point cloud registration. The effect of point cloud registration is enhanced by generative method.", "essential_references_not_discussed": "Also starting from the generative model registration problem, this paper lacks references to FREEREG (ICLR24).\n\n[1] Wang H, Liu Y, Wang B, et al. Freereg: Image-to-point cloud registration leveraging pretrained diffusion models and monocular depth estimators[J]. arXiv preprint arXiv:2310.03420, 2023.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The experimental results are excellent, demonstrating strong performance on both the 3DMatch and ScanNet datasets.\n\n2. The writing of the paper is well-executed, and the content is easy to understand.\n\n3. This paper has some innovation. From the perspective of generation, it provides a new idea for point cloud registration field. But from another point of view, there are many methods to enhance the point cloud registration effect through the texture and color information of 2D images, such as Colorpcr (CVPR24), PointMBF (CVPR24), PEAL (CVPR23).\n\nWeaknesses:\n\n1. The applicability of the method in this paper is poor, and I personally understand that the method cannot be applied to outdoor LiDAR scenes. Because this paper generates 2D images from depth maps. In the follow-up, we can add some experiments to verify whether it is effective for outdoor scenes, and consider whether this problem can be solved in future work.\n\n2. No code is provided for verification.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I have a small question, as shown in Figure 5, how to ensure that the same texture and color information is generated in the overlapping area when faced with low overlap? If this point cannot be guaranteed, I think performance should not improve significantly or even decrease at low overlap.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N5yeWIXaqb", "reviewer": "Reviewer_evgg", "summary": "This paper introduces a novel approach to point cloud registration by leveraging generative models to synthesize 2D images from 3D point clouds, enabling better feature extraction and matching for registration tasks. Traditional methods primarily rely on 3D feature matching, which often struggles in scenarios with low overlap, noise, and incomplete data. The authors propose Match-ControlNet, a framework that utilizes generative 2D diffusion models (such as Stable Diffusion) to improve geometric and texture consistency for robust point cloud alignment.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The approach heavily relies on the quality of the generated 2D images, which could introduce artifacts or inconsistencies in certain scenarios?**  \n**A1:** The image generation quality is indeed crucial to the overall performance. Notably, our Match-ControlNet successfully unlocks the generalizable zero-shot consistency generation ability inherently learned through the large-scale pretraining of Stable Diffusion and ControlNet (see Sec.3.2 and Sec.3.3). \nThis large-scale pretraining on consistency generation can effectively help mitigate potential artifacts and ensures strong consistency across a wide range of scenarios. On top of that, our coupled denoising, coupled prompt guidance, and few-shot consistency fine-tuning strategy further enhance generation reliability. These components are extensively validated through comprehensive real-world benchmark experiments, especially in challenging scenarios with low overlap, occlusions, and cluttered environments. \n\n**Q2: About computational overhead.**   \n**A2:** While image generation and feature fusion introduce additional computational cost, they substantially enhance the quality  of geometric descriptors, leading to significantly improved registration robustness. To better balance performance and efficiency, our future work will explore single-step denoising techniques and knowledge distillation mechanisms to accelerate both image generation and feature fusion. \n\n**Q3: Few-shot fine-tuning requirement.**  \n**A3:** Our method already demonstrates impressive performance in zero-shot scenarios (as shown in Fig. 8). The few-shot fine-tuning, requiring only a minimal amount of image pairs (~3K samples), further improves accuracy. We believe that this lightweight requirement is both practical and consistent with common industry practices.\n\n**Q4: Potential sensitivity to viewpoint selection?**  \n**A4:** We thank the reviewer for raising this important point. In practice, our method exhibits strong robustness to viewpoint selection. Notably, our extensive experiments on the ScanNet and 3DMatch benchmarks already cover a wide range of diverse and challenging viewpoint configurations. Across these varying viewpoint conditions, our method consistently achieves significant improvements in registration performance, demonstrating its reliability under viewpoint selection. We attribute this robustness to the powerful generalization ability of Match-ControlNet, which benefits from the large-scale pretraining of foundation models, as well as our coupled conditional denoising design. We will incorporate this discussion into the revised manuscript. \n\n**Q5: About real-world applicability in autonomous driving or robotics.**  \n**A5:** We appreciate the reviewer’s concern. \n**(i)** While 3DMatch and ScanNet are standard benchmarks, it is important to note that they are constructed from real-world RGB-D scans and have been widely adopted in robotics and embodied AI research, especially ScanNet, which is a common benchmark for indoor robotic perception; \n**(ii)** Furthermore, these datasets capture realistic and challenging conditions, such as low overlap, occlusions, and cluttered layouts, which are highly representative of deployment scenarios in indoor robotics; \n**(iii)** Regarding autonomous driving scenarios, as shown in the figure (https://anonymous.4open.science/r/rebuttal-688D/outdoor_vis.pdf), our Match-ControlNet can generate cross-view consistent images from partial LiDAR scan-based sparse depth maps, demonstrating promising generation effectiveness in outdoor self-driving scenes (please refer to A1@synA for more details);\n**(iv)** Moreover, we tested our method on in-the-wild, low-overlap data (from the author's room) captured by a mobile phone. The resulting generations (https://anonymous.4open.science/r/rebuttal-688D/wild_vis.pdf) further confirm the robustness of our approach in the real-world, unconstrained environment (please refer to A4@P29t for more details). We will further expand the discussion on applications to autonomous driving and robotics in our revision.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The key contributions of Generative Point Cloud Registration are closely related to several areas in the broader scientific literature, particularly in 3D vision, generative modeling, and point cloud processing.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n$\\cdot$ Novel Paradigm: The paper introduces a new approach to point cloud registration by leveraging generative 2D models to enhance 3D matching tasks, bridging a gap between 2D and 3D data processing.\n\n$\\cdot$ Match-ControlNet: The proposed Match-ControlNet improves geometric and texture consistency between generated image pairs, which aids in better point cloud registration.\n\n$\\cdot$ Enhanced Feature Fusion: The work integrates both zero-shot geometric-color feature fusion and XYZ-RGB fusion, providing additional visual cues for more accurate correspondence estimation.\n\n$\\cdot$ Generalization and Plug-and-Play Nature: The framework can be integrated with various 3D registration methods without requiring significant modifications.\n\n$\\cdot$ Empirical Validation: Extensive experiments on benchmark datasets (3DMatch, ScanNet) demonstrate improved registration accuracy compared to existing methods.\n\n$\\cdot$ Addressing Low-Overlap Issues: The proposed method shows strong performance in challenging cases with low overlap and noisy point clouds.\n\n\nWeaknesses:\n\n$\\cdot$ Dependency on Generative Models: The approach heavily relies on the quality of the generated 2D images, which could introduce artifacts or inconsistencies in certain scenarios.\n\n$\\cdot$ Computational Overhead: Generating 2D images using generative models such as Stable Diffusion and performing additional feature fusion may introduce extra computation costs.\n\n$\\cdot$ Few-Shot Fine-tuning Requirement: While the method offers a zero-shot solution, performance improvements through fine-tuning indicate that additional labeled data might still be necessary for optimal results.\n\n$\\cdot$ Potential Sensitivity to Viewpoint Selection: The quality of the generated images depends on the viewpoint chosen for rendering, which might impact registration performance in complex 3D scenes.\n\n$\\cdot$ Limited Real-World Evaluation: The datasets used (3DMatch, ScanNet) are standard benchmarks, but real-world performance in applications like autonomous driving or robotics remains to be seen.\n\nOverall, the paper presents a promising direction for improving point cloud registration by incorporating generative models, though computational efficiency and generalization to diverse real-world scenarios might require further investigation.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PRK7bdesKs", "reviewer": "Reviewer_z9of", "summary": "The paper proposes a new perspective on Point Cloud Registration: Generative Point Cloud Registration. Compared to traditional methods or purely geometry-based learning methods, the paper incorporates image generative models. The input is a point cloud pair with unknown pose, and the output is the transformation between them.\n\nSpecifically, the proposed Match-ControlNet generates corresponding images for each point cloud, and then uses image features as additional information to assist existing point cloud matching methods in finding better correspondences. \n\nThe authors design Match-ControlNet's condition images and prompts to ensure geometric and texture consistency. The authors add this method on top of multiple baselines and experimentally demonstrate that Generative Point Cloud Registration can improve the performance of baselines.\n\n## update after rebuttal\nI carefully reviewed the materials provided by the author during the rebuttal phase. With these additional details, I believe the experimental integrity of the paper has improved. Initially, I had concerns about the completeness of the experimental section. However, after the author's explanations and provision of additional materials, my concerns have been adequately addressed, justifying a higher rating for the manuscript.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Lack of joint distribution modeling of multi-view images?**  \n**A1:** We respectfully clarify that our coupled denoising mechanism has implicitly modeled the joint distribution of multi-view images (i.e., cross-view images in our task). \nFormally, the likelihood over the cross-view image pair can be expressed as $\\mathbb{E}[p_\\theta({x}^{P}, {x}^Q)]$ (we omit conditional variables for simplicity). \nBy concatenating the cross-view images into a unified variable \n${x}^{PQ} = Cat([{x}^P, x^Q])$, we can model the joint distribution of cross-view images via the likelihood over the concatenated representation: $\\mathbb{E}[p_\\theta({x}^{PQ})]$. As such, we can derive a variational lower bound on the data log-likelihood for optimization: L=E_q [log p_θ(x_0^{PQ} | x_1^{PQ}) - ∑_{t>1} D_KL(q(x_{t-1}^{PQ} | x_t^{PQ}, x_0^{PQ}) || p_θ(x_{t-1}^{PQ} | x_t^{PQ})) - D_KL(q(x_T^{PQ}| x_0^{PQ}) || p(x_T^{PQ}) )].\n\nThrough the reparameterization trick, maximizing this lower bound is equivalent to minimizing our training loss in Eq. 5. This derivation demonstrates that our coupled denoising implicitly models the joint distribution of cross-view images, thereby effectively enforcing cross-view consistency generation. We will clarify this in our revision. \n\n**Q2: The proposed method is a workaround and exhibits instability? Absence of explicit attention-based modules like Zero123++?**  \n**A2:**\nWe respectfully clarify that our method is not a workaround, but an efficient and elegant design that enables effective cross-view information fusion and consistency learning with minimal overhead. Unlike methods like Zero123++, which introduce an additional complex attention modules for fusion, our attention design is native. We employ a novel coupled denoising mechanism that naturally extends the inherent intra-image self-attention to both inter- and intra-image attention, without any architectural modifications (see Eq. 4). We note that this design effectively unlocks the zero-shot consistency generation ability of Stable Diffusion, acquired through extensive large-scale pretraining, thereby enhancing both generation quality and cross-view consistency. As evidenced by extensive experiments, our method consistently delivers stable results across diverse scenarios (see Table 5, 6, and Fig. 5). Moreover, we tested on low-overlap, in-the-wild data (from the author's room) captured by a mobile phone. The results (https://anonymous.4open.science/r/rebuttal-688D/wild_vis.pdf) further confirm its stability in unconstrained conditions (please see A4@P29t for details).\n\n**Q3: Real vs. generated images comparison.**  \n**A3:** As shown in Table 3 (third block) in our manuscript, we had compared ColorPCR using ground-truth images with our generative ColorPCR that utilizes generated images. Surprisingly, our generated images can even achieve superior performance across most metrics. We attribute this improvement to that our generated images effectively mitigate calibration errors and lighting challenges inherent in real-world data, as illustrated in Fig. 6. \n\n**Q4: Comparison with image-only methods (e.g., DUSt3R).**  \n**A4:** We would like to emphasize that our paper targets the point cloud matching rather than the image matching. Therefore, comparisons with image-only methods actually fall outside the scope of our research. Below, we report the score of SOTA image-only method, DUSt3R, on 3DMatch. It shows that, due to the absence of geometric and scale information inherent in point clouds, DUSt3R demonstrates limited precision, highlighting the importance of point cloud features for accurate registration.\n\n| Methods               | Rot@5  | Rot@10 | Rot@45 | Mean | Med. | Trans@5 | Trans@10 | Trans@25 | Mean | Med. |\n|-----------------------|------------------|----------|----------|---------------|---------------|----------------------|----------|----------|---------------|---------------|\n| DUSt3R                | 50.9             | 64.2     | **98.5** | 10.0          | 4.9           | 6.6                  | 21.2     | 61.7     | 23.3          | 19.7          |\n| Generative FCGF  | **94.3**         | **96.7** | 98.1     | **4.5**        | **1.4**        | **54.3**              | **81.5** | **93.1** | **12.5**        | **4.7**         |\n\n**Q5: Variance analysis across generated images.**  \n**A5:** Thank you for your suggestion. We repeated the evaluations on 3DMatch using varying random seeds (123, 1234, 12345) for Match-ControlNet. The mean and variance of Generative FCGF are 93.7 ± 0.6 for Rot@5 and 54.1 ± 0.1 for Trans@5, which still exhibits a significant performance gain over the baseline, validating the robustness of our method. \n\n**Q6: More visualization results of point cloud overlap after matching.**  \n**A6:** We provide more registration visualisation results in figure (https://anonymous.4open.science/r/rebuttal-688D/reg_vis.pdf), qualitatively showing our excellent precision. We will include them into our revised version.", "claims_and_evidence": "Yes, I believe introducing image information would be helpful for point cloud matching.", "methods_and_evaluation_criteria": "I see methodological flaws in the proposed approach. While the authors introduced requirements like 2D-3D Geometric Consistency and Cross-view Texture Consistency, the method essentially processes a stack of images using ControlNet. This alone does not robustly ensure multi-view consistency. Achieving consistency across views fundamentally requires accurately modeling the joint distribution of multiple images—a key aspect seemingly overlooked here. Instead, the current method relies on fine-tuning ControlNet on general image diffusion models, yet fails to introduce explicit constraints to enforce consistency. As a result, I remain doubtful about the reliability of the generated multi-view outcomes.\n\nFurthermore, since the approach presented in the paper often serves as a workaround in various image diffusion-based applications, it exhibits certain instability.\n\nFor instance, works like Zero123++ accomplish multi-view information fusion by implementing novel modules within attention mechanisms—elements conspicuously missing from this paper.", "theoretical_claims": "No Theoretical Claims", "experimental_designs_or_analyses": "1. I remember ScanNet has images, and the experiment lacks a comparison between the improvement of using real multi-view consistent images as additional information over the baseline and the magnitude of improvement using generated images over the baseline.\n2. For methods like DUSt3R, although not used for point cloud registration problems, they can obtain the relative pose between two viewpoints through images alone. I'm curious about how the proposed method would compare with these methods? In other words, when both images and point clouds are available, how important are the point cloud features for the registration task?\n3. I think the authors should add information about how different image generation results affect the estimated pose for the same pair of point clouds, or provide the mean and variance of improvements over the baseline after multiple executions.", "relation_to_broader_scientific_literature": "- DUSt3R: Geometric 3D Vision Made Easy[CVPR2024]\n    - This paper discusses the problem of pose estimation using only images\n- Zero-1-to-3: Zero-shot One Image to 3D Object[ICCV2023]\n- Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model\n    - The above two papers discuss how to better inject multi-view consistency into Diffusion", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "This paper combines the generative capabilities of ControlNet with Point Cloud Registration in an application-oriented article. Its strength lies in providing a new perspective on the Registration problem, using generative methods to compensate for the lack of image-point cloud pairs. Its weakness is that the proposed method resembles more of an experimental report, and neither theoretically nor through rigorous experimental analysis can it explain why this approach enables ControlNet to achieve multi-view texture consistency.", "other_comments_or_suggestions": "I hope to show more visualization results of point cloud overlap after matching, rather than just the images generated by ControNet.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aYZOCRoKK8", "reviewer": "Reviewer_P29t", "summary": "This work introduces a novel method for point cloud registration that aims to generate geometry-consistent RGB image pairs from paired point sets. These generated RGB image pairs are then used to enhance the performance of point-based registration methods. The proposed approach incorporates two key innovations: a newly designed coupled conditional denoising technique and a prompt guidance mechanism, both of which are developed to enforce cross-view texture consistency. Extensive experiments conducted on the ScanNet and 3DMatch datasets demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Clarification on rigid transformations.**  \n**A1:** Thank you for your valuable suggestion. In the revised introduction, we will explicitly state that our work focuses exclusively on the rigid point cloud registration problem to set clear expectations for the readers. \n\n**Q2: The discrepancy between the noticeable improvement in mean errors and the lack of significant enhancement in median error in Table 2 requires further explanation.**  \n**A2:** Thank you for the insightful comment. We clarify that this discrepancy arises because mean and median errors capture different aspects of performance. Specifically, the mean error reflects the overall precision and is sensitive to outliers. The noticeable improvement in mean error stems from a substantial reduction in large-error cases, indicating that our method can effectively handle challenging scenarios. Instead, the median error reflects the model’s performance on the majority of cases, which was already strong and thus shows minimal change. We will clarify this distinction in the revised manuscript to avoid confusion and better highlight the strengths of our approach. \n\n**Q3: Comparison with newer state-of-the-art.**  \n**A3:** We appreciate the reviewer’s valuable suggestion. Following the reviewer’s recommendation, we have conducted an additional comparison with a recent state-of-the-art method, PARE-Net (ECCV 2024), and included the results in the table below. The new results show that our proposed Generative FCGF (SD) consistently outperforms PARE-Net, confirming our superior registration performance. We will include these comparisons in our revised version.\n\n| Methods               | Rot@5  | Rot@10 | Rot@45 | Mean | Med. | Trans@5 | Trans@10 | Trans@25 | Mean | Med. |\n|-----------------------|------------------|----------|----------|---------------|---------------|----------------------|----------|----------|---------------|---------------|\n| PARE-Net (ECCV'2024)                | 75.6             | 82.1     | 86.5 | 21.0          | 2.1           | 40.6                 | 63.5     | 77.4     | 47.4          | 6.4          |\n| Generative FCGF  | **94.3**         | **96.7** |  **98.1**     | **4.5**        | **1.4**        | **54.3**              | **81.5** | **93.1** | **12.5**        | **4.7**         |\n\n**Q4: The concern is that ControlNet may have been trained on easier datasets like ScanNet, which could simplify the generation task. It's important to test the proposed method on more challenging datasets to confirm its generalizability.**  \n**A4:** Thank you for the thoughtful comment. **(i)** We would like to clarify that, although the training dataset for the depth-conditioned ControlNet model has not been publicly released, the depth maps used in that model were generated by the MiDaS depth estimation model. By contrast, our experiments utilize real depth maps captured directly by depth sensors. Despite the significant domain gap between the MiDaS-estimated and sensor-captured depth maps, our method still achieves impressive generation quality, highlighting its robustness and generalization capability; **(ii)** To further validate the generalizability and robustness of our Match-ControlNet in more challenging scenarios, we conducted an additional experiment where we casually captured low-overlap photos of a cluttered, unconstrained indoor environment (i.e., the author's room) using a mobile phone. We then estimated their depth maps using DUSt3R for subsequent Match-ControlNet generation (without any model fine-tuning). The figure (https://anonymous.4open.science/r/rebuttal-688D/wild_vis.pdf) demonstrates that, even under these challenging in-the-wild conditions with a different depth-map source, our method still achieves impressive cross-view consistency and generation quality. These results confirm the practical effectiveness and excellent generalizability of our approach.", "claims_and_evidence": "The claimed contributions are well-supported by the proposed method and experimental results.", "methods_and_evaluation_criteria": "The newly designed coupled conditional denoising technique and prompt guidance mechanism are technically sound and interesting. The evaluation metrics are reasonable.", "theoretical_claims": "No theoretical proof in the paper.", "experimental_designs_or_analyses": "experimental designs or analyses are solid.", "relation_to_broader_scientific_literature": "The proposed method can serve as a plug-and-play module for various existing point-based registration models.", "essential_references_not_discussed": "No. The literature review looks comprehensive.", "other_strengths_and_weaknesses": "Strengths:\nThe idea of using generated image pairs to enhance the registration performance of point-based models is highly innovative. Both quantitative and qualitative results demonstrate the effectiveness of the proposed method, showcasing its potential to improve registration accuracy.\n\nWeaknesses:\n1. Point set registration encompasses both rigid and non-rigid transformations. However, this paper focuses exclusively on rigid transformations. This should be clearly stated early in the introduction to set appropriate expectations for readers.  \n2. In Table 2, the proposed method shows a noticeable improvement in mean errors but no significant enhancement in median error. This discrepancy warrants further explanation or analysis to clarify the underlying reasons.  \n3. The baseline models primarily consist of older methods published several years ago. To better highlight the advancements of the proposed method, it would be beneficial to include evaluations against more recent state-of-the-art models.  \n4. There is a concern that ControlNet might have been trained on datasets such as ScanNet or similar datasets, which could reduce the difficulty of the generation task. It would be valuable to verify whether the proposed method remains effective on more challenging datasets to ensure its generalizability.", "other_comments_or_suggestions": "No", "questions_for_authors": "See the weakness section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All the reviewers agree that the paper provided novel perspective for addressing 3D point clouds registration problem, the method is interesting and outperforms various state-of-the-art. During rebuttal, more clarification and explanations and additional experimental results have been provided. All the reviewers are convinced that the paper has made important contribution to the field. After the interactions with the authors, two reviewers raised their overall recommendation scores.
Decision: Accept (poster)