SE(3) Diffusion Model-based Point Cloud Registration for Robust 6D Object Pose Estimation
Authors: Haobo Jiang
Mathieu Salzmann
Zheng Dang
Jin Xie
Jian Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: 6D object pose estimation
Point cloud registration
Diffusion probabilistic model
TL;DR: Present a diffusion model over SE(3) manifold for point cloud registration-based 6D object pose estimation.

Abstract: In this paper, we introduce an SE(3) diffusion model-based point cloud registration framework for 6D object pose estimation in real-world scenarios. Our approach formulates the 3D registration task as a denoising diffusion process, which progressively refines the pose of the source point cloud to obtain a precise alignment with the model point cloud. Training our framework involves two operations: An SE(3) diffusion process and an SE(3) reverse process. The SE(3) diffusion process gradually perturbs the optimal rigid transformation of a pair of point clouds by continuously injecting noise (perturbation transformation). By contrast, the SE(3) reverse process focuses on learning a denoising network that refines the noisy transformation step-by-step, bringing it closer to the optimal transformation for accurate pose estimation. Unlike standard diffusion models used in linear Euclidean spaces, our diffusion model operates on the SE(3) manifold. This requires exploiting the linear Lie algebra $\mathfrak{se}(3)$ associated with SE(3) to constrain the transformation transitions during the diffusion and reverse processes. Additionally, to effectively train our denoising network, we derive a registration-specific variational lower bound as the optimization objective for model learning. Furthermore, we show that our denoising network can be constructed with a surrogate registration model, making our approach applicable to different deep registration networks. Extensive experiments demonstrate that our diffusion registration framework presents outstanding pose estimation performance on the real-world TUD-L, LINEMOD, and Occluded-LINEMOD datasets.

SE(3) Diffusion Model-based Point Cloud Registration for Robust 6D Object Pose Estimation
Haobo Jiang', Mathieu Salzmann 2.3+ Zheng Dang", Jin Xie and Jian Yang"*
1PCA Lab, Nanjing University of Science and Technology, China
CVLa,, EPFL, Switzerland SClearSpace, Switzerland
(Jiang. hao bo, csixie, cs j @nnusst....... edu. en (zheng. dang, mathieu. sal salzmann] @epfl. ch
Abstract
In this paper, we introduce an SE(3) diffusion model-based point cloud registration framework for 6D object pose estimation in real-world scenarios. Our approach formulates the 3D registration task as denoising diffusion process, which pro- gressively refines the pose of the source point cloud to obtain precise alignment with the model point cloud. Training our framework involves two operations: An SE(3) diffiusion process and SE(3) reverse process. The SE(3) diffusion process gradually perturbs the optimal rigid transformation of a pair of point clouds by continuously injecting noise (perturbation transformation). By contrast, the SE(3) reverse process focuses on learning a denoising network that refines the noisy trans- formation step-by-step. bringing closer to the optimal transformation for accurate pose estimation. Unlike standard diffusion models used linear Euclidean spaces, diffusion model operates the SE(3) manifold. This requires exploiting the linear Lie algebra se(3) associated with SE(3) to constrain the transformation transitions during the diffusion and reverse processes. Additionally, effectively train our denoising network, we derive registration-specific variational lower bound the optimization objective for model learning. Furthermore, we show that our denoising network can be constructed with a surrogate registration model, making our approach applicable to different deep registration networks. Extensive experiments demonstrate that our diffusion registration framework presents out- standing pose estimation performance on the real-world TUD-L, LINEMOD, and Occluded-LLNNEMMOD datasets. The code is available at https///ittubb.com/iingg HB//Diffusioneeg
Introduction
Accurately estimating the 6D pose of object, comprising both its spatial position and orientation, is a critical task in the field of computer vision, and has been extensively applied various domains such
robotics grasping [10 52 68
augmented reality [32 33), and autonomous navigation [7 [6] [58].
While substantial efforts have been dedicated to developing methods for 6D pose estimation based on
RGB or RGB-D data 26 42 52
38 63
49
the advancements of 3D sensors (such Kinect and
liDAR) and 3D registration techniques has promoted the emergence of point cloud registration-basee pose estimation promising direction.
"Corresponding authors.
Haobo Jiang, Jin Xie, Jian Yang are with PCA Lab, Key Lab Intelligent Perception and Systems for High- Sigh-Dinnsional Information Ministry of Education, and Jiangsu Key Lab Image and Video Understanding for Social Security, School of Computer Science and Engincering, Nanjing University of Science and Technology, China.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Nevertheless, the state-of-the arr methods for object-level 3D registration 44 62 15]
primarily
focus on synthetic data and struggle yield precise registration on real-world 6D pose estimation
datasets such as TUD-I 20 LINEMOD [18] and Occluded-LINNMMOO 6 Unlike the controlled geometric structures, manually defined transformations, and artificial partial overlap present in synthetic data, real-world pose estimation datasets present significant challenges, including large rotations and translations, natural noise interference, and severe occlusions, considerably exacerbating the registration difficulty. Prior attempts to tackling these challenges include [T. which introduces general match normalization layer to regularize the feature distribution of the source and model point clouds thus facilitating feature matching, and [60, 48), which design RANSAC-based outlier rejection strategy to enhance the registration robustness. While these initial attempts constitute significant steps towards making object-level point cloud registration practical, much work remains to be done to fully address all the aforementioned real-world challenges.
In this paper, inspired by the remarkable success of diffusion models in generative Al [19 45 46 47
we propose to adapt them to the 3D registration field, and introduce an SE(3) diffusion model-based registration method for robust 6D object pose estimation in the real world. Our approach formulates 3D registration as a denoising diffusion process in SE(3) (special Euclidean group), aiming to progressively refine the pose of the source point cloud to achieve precise alignment with the model point cloud. Training our model then involves two key operations: An SE(3) diffusion process and an SE(3) reverse process. The SE(3) diffusion process progressively converts the optimal transformation between the source and model point clouds into a noise one by continuously injecting perturbation transformations. Conversely, the SE(3) reverse process learns denoising network to gradually refine the noise transformation to the optimal one. Our diffusion model operates on the SE(3) manifold, which poses the challenge of extending the diffusiinn/evverse formulas from linear Euclidean space to a nonlinear manifold. To address this challenge, we exploit the linear Lie algebra se(3) associated with SE(3) to conduct linear diffusion/reverse computations and map the se(3) results to SE(3) to obtain the desired diffused/reveese transformations.
To effectively train our denoising network, we exploit Bayesian formalism and derive a registration- specific variational lower bound as optimization objective for model optimization. Furthermore, we reformulate our denoising network with surrogate registration model, which makes approach applicable to different deep registration methods such as [54] 662.. During the inference stage, the learned denoising network progressively refines identity transformation to approach the optimal one, given the source and model point clouds as conditioning signals.
Compared to previous registration methods, our diffusion registration framework offers the following two advantages. First, the diffusion process generates a diverse set of poses for the source point cloud, allowing for more comprehensive model training. This increased pose diversity facilitates the model's ability to handle wider range of rotation/translation and improves its generalization capabilities. Second, by incorporating the Bayesian posterior in the reverse process, guides the update of the source point cloud pose at each reverse step. This guidance mitigates the risk of getting trapped in local optima, leading to more robust and accurate pose estimation results. Our empirical results provide substantial evidence supporting these two advantages.
approach effectively
To summarize, our main contributions are as follows: (i) We introduce a novel SE(3) diffusion model-based 3D registration framework for robust 6D object pose estimation, where the optimal transformation is estimated via progressive denoising process. (ii) To train our denoising network, we follow a Bayesian approach and establish a 3D eegistration-specific variational lower bound as optimization objective for our SE(3) diffusion model. Furthermore, we reformulate the denoising network with a surrogate registration model, enabling the integration of different deep registration models into framework. (iii) To the best of our knowledge, we are the first to successfully adapt the diffusion model from linear Euclidean space to the SE(3) point cloud registration task for 6D object pose estimation. Our extensive experiments real-world datasets, including TUD-L (20). LINEMOD [8) and Occluded-LINEMOD 66, confirm the effectiveness of our framework.
2 Related Work
Point Cloud Registration. Point cloud registration aims to estimate the rigid transformation between a pair of point clouds. Existing methods can be roughly divided into optimization-batt techniques and deep learning-based ones. Iterative Closest Point (ICP) 4 alternately searches for the closest correspondences and estimates transformation until convergence. Go-ICP (6] uses the branch- and-bound algorithm to improve the robustness of ICP to initialization. Robust ICP [64] designs
2

---

SF(3) Diffusion (8R3) Interpolation SR3) Perturhatio)
K,M
SE(3) Reverse (Shrroguee reginirration))
Po(H M)
Hr
H
H
Ho
HHH
H
Figure I: Probabilistic graphical model of our SE(3) diffusion model-based registration framework.
Welsch's s function-based robust metric to obtain an outlier-robust alignment evaluation for optimiza- tion. Sparse ICP (5) reformulates ICP using sparsity-inducinn norms. Many other ICP variants have been proposed, such as 8, 43 14
2 [7], exhibiting promising registration performance. Nevertheless, the current main research effort focuses on deep learning models. this context, DCP [54] constructs pseudo correspondences for SVD-based transformation estimation using deep feature similarity. PRNet (55) leverages keypoint identification and Gumbel softmax for establishing more reliable correspondences. PointNetLK [ and FMR [21) introduce the Lucas & Kanada (LK) algorithm [ and inverse compositional (IC) algorithm [ into deep models for transformation refinmmen
feature aligmment. CEMNet [23 uses a planning- based cross- cross-entropy method for transformation
optimization. RPMNet RGM 15] and RTENet 44 leverage Sinkhorn optimization, neighbor- hood structure consistency, and graph matching, respectively, for robust outlier removal. Many other
deep registration models such as [28 36 30 29 (7 have been developed and achieve impressive registration performance. However, these registration models mainly focus on synthetic datasets, such as ModelNet40 (56). As shown in [Л] they still yield limited precision on real-world 6D object pose estimation datasets such TUD-L (20), LINEMOD
[18), and Occluded-LINEMOOT (6).
6D Object Pose Estimation. Estimating the 6D pose (orientation and position) of objects has gained increasing attention in recent years. Early approaches directly estimated object pose from extracted image features through regression or classification. PoseCNN [57] decouples 6D pose estimation into center-based translation regression and quaternion-based rotation regression. SSD-6D [24] extends the single-shot object detector to cover the full 6D object space for pose inference. Trabelsi er al. [51) propose multi-attentional network for iterative pose refinement using the appearance and flow information. DeepIM (31) and LatentFusion [37] learn pose estimation by minimizing the error between the rendered model and the observation. More recent research has focused on a two-stage estimation pipeline, where 2D keypoints are extracted to solve for 6D pose using the PnP algorithm [33. Rad et al. [42] take the predicted 2D projections of the corners of 3D bounding boxes as the keypoints, while Zhao et al. [65] manually designate keypoints over the surface of the object model. PVNet[41] establishes pixel-wise voting network for keypoint estimation using
49 [2] also present promising pose estimation performance. Recently, thanks the advances in 3D registration techniques, the community has witnessed a growing interest in 3D registration- based 6D) object pose estimation, which recovers the object's 6D pose by estimating the rigid transformation between the object (source) and the model point clouds. This is what we achieve here, but, in contrast to most existing methods which focus on synthetic data, we tackle the challenging scenario of working with real point clouds.
predicted pixel-wise voting vectors. Other methods 66 40 50
52
3 Approach
3.1
Revisiting the Euclidean Diffusion Model
Diffusion models, as generative models, aim to generate new data by progressively denoising noisy inputs 19
45 46 477. Their training phase involves diffusion process and a reverse process.
The diffusion process successively injects Gaussian noise into the data sample Xy ~ Pdata (Pdate indicates the distribution of the training data) so gradually transform X, into the noise data XT (O, I) (standard Gaussian distribution), thus forming a Markov chain Xa - XI + - XT- As demonstrated in 19), the random variable x, g(x, x;-1) == ~N(x;; /I B, 3.I) can also be expressed in a closed form X, ~ {(x Xo). which can be formulated as follows:
X Va + VI ātE, (O.
(I)
3

---

Here, the diffusion coefficients át ==0 a2 = II (11)))..... and 8g indicates the noise coefficient
determined by a linear schedule 19 or cosine schedule [35]. Then, the reverse process learns denoising network, parameterized normal distribution po (x,-1 x,) = N (x:-1; M4 (x,, , ,,,T), to progressively denoise the noisy data XT into the clean one Xo. forming reverse Markov chain XT XT-1
+ Xo- Here, Mg (x,,t indicates the parameterized mean of the normal distribution. The following variational lower bound the log likelihood over the training data is then
derived as the optimization objective for training the denoising network:
Exo-PPaata [xo)) 2 Egl DKL (q (x,_1 X Xo) llp (x,-1 x,)) log pa (Xo x1)] (2)
1>1
Based on Bayes' formula, the random variable x following the posterior distribution q (xx-1 Xe, xo) in Eq. 2 can be represented as
1at-18 1-ā,
Xo + Var āi-1) Xt + v BiE,
(3)
1 āi
where the random variable E ~ ~N(O,)) I) and the variance scale 3. = I-6s
1- Br.
3.2 SE(3) Diffusion Registration Model for 6D Object Pose Estimation
In the context of 6D object pose estimation based on point clouds, our objective to determine the rigid transformation between a partially-scanned source point cloud X' = {x, R and complete model point cloud M [m, E 3 so as to align their overlapping regions precisely. Jj=1
This transformation encompasses rotation matrix R E SO(3) and translation vector t E R representing the orientation of the object and its spatial position, respectively. To obtain a partial source point cloud, following [11), we mask input depth map to restrict it to the object, and convert the resulting depth values into 3D points using the known camera intrinsic parameters. By contrast, the model point cloud is generated by uniformly sampling a complete mesh model. We denote the optimal transformation Ho E S3)3) and the identity transformation H. E S1E(3) as
R O
t
I 0 1 1
0
Ho
H
(4)
both of which are represented by 4 x 4 homogeneous transformation matrices.
Inspired by the progressive generation of diffusion models, we adopt the concept of denoising diffusion to tackle the point cloud registration task, specifically in the SE(3) space, and thus propose an SE(3) diffusion model-based 3D registration framework for robust 6D object pose estimation. While [27 (33 have introduced variations of the diffusion model on SE(3) or SO(3) manifolds, they not suitable for our 3D registration task. The training phase of our framework thus involves an SE(3) diffusion process and an SE(3) reverse process. Given a pair of source and model point clouds, the former continuously disturbs their optimal transformation by injecting noise into it, while the latter aims to learn a denoising network to progressively convert the noisy transformation to the optimal one. As such, during inference, the learned denoising network can be leveraged recover the transformation between the source and model point clouds through a progressive denoising process. The training phase and inference phase of our algorithm are detailed in Algorithms I and [ Below, we provide a detailed explanation of our SE(3) diffusion process and SE(3) reverse process.
3.2.1 SE(3) Diffusion Process
In accordaacce with the standard diffusion model described Sec. 3.1 our SE(3) diffusion process progressively disturbs the optimal transformation H, of point clouds X' and M into a noisy trans- formation Hy by injecting perturbation transformations (noise) into thus forming a diffusion Markov chain: H. + H, - ................................ - Hy. However, our SE(3) diffusion process features two critical differences from the conventional one: (i) Our diffusion process operates on the nonlinear SE(3) manifold, unlike the standard diffusion process which acts in linear Euclidean space. Consequently, the linear diffusion operations provided in Eq cannot be directly applied to our model. (ii) The standard diffusion process assumes the noise variable X ~ (O, I) to be centered around the zero vector. However, in our model for 3D registration, we require the noise transformation Hr to be centered around the identity transformation H.. To account for these differences while generating the

---

Algorithm/ 1 Training phase
Algorithm 2 Inference phase
repeat
for 1 = T.... 1 do
Sample X, MM. Ho Pdata
Sample t ~ Uniform (1..................... TD, H. = Register H2-00 = So(K', M), where A; T(X, H₂)
Exp(yv] - a (V; Ho, Hi)
Estimate H = Exp(Ao Log (H, (o, +
Register H = ( M)
\ Log (H,))
Optimize loss L, = loss (oss(H,,00) H)
end for Return: H
until converged
desired diffusion Markov chain, we propose to interpolate the transformations between Ho and H on the SE(3) manifold, incorporating noise perturbations. Formally, our interpolation-base SE(3)
diffusion formula formulates H, ~ qH, Hb) at time step t (1 S $ T) as
H Exp(rv1 a())) F F(vār; Ho. HH),
M~~~00 N00,1)
(5)
Perturbation
Interpolation
In essence, this formalism utilizes interpolation function F( Cee Ho, HI) to derive intermediate transformation that lies between the optimal transformation H. and the identity transformation H. Subsequently, augment this interpolated transformation with randomly-sample perturbation transformation (noise), denoted as Exp(ry/i - o2g), to yield the diffused transformation H,. Below, will introduce these two components in detail.
Transformation Interpolation Function F( (v H., The interpolation function SE(33) x
SE(3) [0 - (3) aims to compute an intermediate transformation between the optimal transformation Ho and the identity one H. Following the setting of diffusion coefficients in Eq. [ set the interpolation weight transformation H, time step & voa and, consequently, the interpolation weight on H is 1 Vāz. As the time step increases, the interpolation weight vā: on Ho decreases gradually, resulting in transition of the interpolated transformation from Ho H. However, due to the non-linearity of the SE(3) manifold, direct linear interpolation methods, such as weighted averages, cannot be applied interpolate intermediate transformations. To address this, we exploit the Lie algebra se(3) associated to SE(3). The Lie algebra se(3) is a linear 6D vector space corresponding the tangent space of SE(3) at the identity transformation. Transferring any transformation in SE(3) to a 6D vector element in se(3) can be achieved using the logarithm map Log : (3) + R conversely, the exponential map Exp: E - SE) transforms a 6D vector in se(3) back to the SE(3) manifold. Therefore, we can first project the SE(3) transformation to se(3), perform linear interpolation in this tangent space, and then convert the interpolated vector back to SE(3) to obtain the interpolated transformation. Formally, our transformation interpolation function F is thus expressed as
F(V Ci Ho, HI) = Exp((1 a.) Hog(H
which starts by calculating the relative transformation HH,," from the optimal transformation Hg to the identity transformation H. and mapping it to the linear Lie algebra se(3) through the logarithmic map. In this linear space, we then scale the vector Log(HIIH by the interpolation weight vag, and map this weighted vector back into the corresponding weighted relative transformation using the exponential map. The weighted relative transformation quantifies the offset of the interpolated transformation relative to Hy. Finally, multiplying the optimal transformation H, by the computed transformation offset yields the desired interpolated transformation at time step t.
Perturbation Transtormation Exp(ryI ā,e). Following the standard diffusion formula in E we introduce random noise (i.e., a perturbation transformation) into the interpolated transfor- mation F(vā;; Ho, H)) at each time step to randomize our SE(3) diffusion process. As indicated in Eq. [. conventional Euclidean diffusion models typically draw such noise from Gaussian dis- tribution in Euclidean space. However, formulating a Gaussian distribution on the SE(3) manifold non-trivial. To tackle this issue, we utilize the Lie algebra once more. Specifically, we randomly sample a 6D noise vector e E R2 from (0, I) over IR which can be interpreted as an element in se(3). Then, we scale the noise vector by the factor T Or as Eq. [ to control the magnitude of the perturbation at different time steps. Finally, this scaled noise vector is converted back to SE(3)
5

---

using the exponential map to obtain the corresponding perturbation transformation. Please refer to Appendix A for more details about our perturbation transformation.
3.2.2 SE(3) Reverse Process
With the diffusion Markov chain Ho + H1 -... Hr generated by the SE(3) diffusion process in Sec. 3.2.1. the objective of the SE(3) reverse process is train a denoising network pa (H,-1 & = 7 7(x, C.H,H. H.),)) to progressively refine the noisy transformation towards the optimal one, thus forming reverse Markov chain Hy. H design our denoising network to predict a probability distribution over the transformation H given the model point cloud M and the transformed source point cloud X; = ( H2) (R,x; + tz). Here, R₃ and tt denote the rotation matrix and translation vector of transformation H,. To effectively train our denoising network, we derive the following registration-speclne variational lower bound of the log likelihood over the training samples the optimization objective:
Ho- In the context of 3D registration, we
Po (HoT X,
M. Mo [In pe Ho
X X) 2 EE
H HETOO HoM,HooPPaatt
(HET Ho)
=E
Hapon HoM.Hootaatte
In (Ho M)
DKL(O (H Ho)l((HH))
Residual term
Prior matching term
T
DKL H., Ho
Xe M))
Denoising matching term
where Pdtta indicates the distribution of the training data, including the pairs of source and model point clouds and their ground-trune rigid transformations. In the following, we elaborate each loss term of the derived variational lower bound. Please refer to Appendix A for a detailed derivation.
Denoising Matching Term. This is the fundamental loss term for training denoising network. q(H,1 H,, Ho) represents the posterior distribution of H,-1 conditioned on H, and H,, while Po(H, (H,-1 M) denotes the prior distribution over H,- predicted by our denoising network. In contrast to the prior distribution, the posterior has access to the optimal transformation Hg, enabling it infer more reliable distribution for H;-1 through Bayes' formula. As a result, can treat this posterior distribution as ground-truth signal to supervise the prior distribution prediction of the denoising network by minimizing their Kullback-Liiber (KL) divergence. Inspired by the Bayesian posterior in Eq. 3 the random transformation H following the posterior distribution, H (HH Hg, Ho), can be expressed as
Hoost Exp
Var-1 Bt
Log (Ho) + Va (1 āz-1)
Log (H.) + v B.e).
1 02
A
where we employ the logarithm map to convert the SE(3) transformations H, and H, to their corresponding 6D vector representations in the linear Lie algebra se(3). This allows us to perform a linear combination with posterior coefficients Àg and Y₁. as in Eq. 3 The randomness of HP arises from the addition of a random variable e, which follows a Gaussian distribution (O, over RRC The resulting vector is then converted back from se(3) to SE(3) using the exponential map, yielding 1-11 Analogously, the random transformation following the prior distribution,
H ~ po (H, M), can be represented as
H. Horior Exp (Log (Ho( + BBE).
where (X,, M) denotes the parameterized mean of our prior distribution, and v 3. represents the random term that is identical to that of the posterior. As such, minimizing the KL divergence between the prior and posterior distributions is equivalent to minimizing the error between Log (po and Ao (oo) + >1 Log(H, H,), i.e..
(eeeee,,,, (X, M))
L, (8) loss A Log (og) A Log Log(
(10)

---

The loss function 10 indicates that, at each time step t, given the point clouds 2; - ( H,) and M, the mean Hig (2,, M) needs be predicted as the transformation Exp(Ag (og + X log(H,) rather than the relative transformation HgH, between X; and M. Consequently, existing deep registration models such as [54, 62 may not be directly applicable for parameterizing the mean of our prior distribution. Moreover, designing specific network that takes X₂ and M as input but predicts non-relative transformation also is a non-trivial task. To handle it, we consider that the transformation Exp(Ao Log(Ho) + X log (H.)) can rewritten as
Exp (Ao Log(H,, + X1 Log H,) Exp Da Log( HH,H. Ho) A (H.))
(11)
H.-0
This inspires us to reformulate μel ue( M) using surrogate registration model fo( M) as
pa(K;, = Exp (Ao ( M)H,, a Log(H.)).
(12)
Then, minimizing the loss function TO is equivalent optimizing the surrogate registration model M. to predict the relative transformation H H,H,, between the point clouds X, and M. Therefore, different deep registration models can potentially serve as surrogate registration mod- els in our denoising network. Finally, we optimize the surrogate registration network by minimizing the L1 distance between the source points transformed using the ground-truth transformation H and the predicted one H = M), This is written as
(0) = loss H,- = N
Σ
NN
H
x
x
(13)
where E (2.... and x indicates the i-th point in the transformed source point cloud X,
Residual and Prior Matching Terms. To maximize the probability (Ho) M), the mean (o M) Exp (Ao Log(fe M)H, + Log(H1)) should be optimized align closely
with Ho. As such, the optimization target of (X M) is H,H,', and the loss function can be represented as (0) in 13 In addition, as the prior matching term does not require learning any parameters, it can be regarded as a constant and hence can be omitted.
4 Experiments
4.1 Experimental Settings
Implementation Details. We set the numbers of points in the source and model point clouds to N = 512 and M = 1024 through random sampling. For the SE(3) diffusion process, we adopt cosine schedule 335 to determine the diffusion coefficients {B,).. The number of diffusion steps T is set to 200, and the scaling coefficient y for the perturbation transformation is set to 0.1. For the SE(3) reverse process, the number of reverse steps in the training phase is set 200, while that in the inference phase is set to 5 to accelerate the inference speed of the diffusion registration. We use the ADAM optimizer (55 with a learning rate of 0.001 to optimize the loss function [3 for 20 epochs with batch size of 32, and employ Py PyTorch (39 to implement our framework. All experiments are conducted on a server equipped with an Intel i5 2.2 GHz CPU and one TITAN RTX GPU.
Evaluation Metries. Following M. we evaluate the model performance by quantifying the rotation and translation errors between the predicted rotation and translation R and É, and the ground-trui ones R and t² The evaluation metrics are defined as
Tr R R R
RE(Î.
= arccos
TE(t) = ||t t 11
(14)
2
As in 12 we summarize these errors via mean average precision (mAP) under varying thresholds.
4.2 Comparison with Existing Methods
Evaluation on TUD-L.. We first evaluate our approach on the TUD-I dataset 20 a real-world dataset comprising three household objects. The compared methods encompass four representative
7

---

TUD-L
LINEMOD
Docluded-LINEMOD
Models
50
10
lem
2cm
5
10a
lem
2cm
5
10
lcm
2cm
ICP 4
0.02
0.02
0.01
0.14
0.00
0.01
0.04
0.27
0.01
0.01
0.07
0.36
FGR 66
0,00
0.01
0.04
0.25
0.00
0.00
0,05
0.31
0.00
0.00
0.08
0.43
TEASER 59
0.13
0.17
0.03
0.22
0.01
0.03
0.03
0.21
0.01
0.02
0.04
0.26
S4PCS
0.30
0.50
0.05
0.40
0.02
0.09
0.04
0.31
0.01
0.03
0.06
0.31
IDAM 28
0.03
0.05
0.02
0.08
0.00
0.01
0.03
0.16
0.00
0.02
0.07
0.26
FMR 22
0.02
0.09
0.02
0.06
0.00
0.01
0.07
0.17
0.00
0.00
0.09
0.18
RGM [5
0.00
0.00
0.02
0.03
0.00
0.00
0.07
0.15
0.00
0.00
0.09
0.22
RIENct [4
0.00
0.00
0.06
0.11
MN-IDAM 11
0.36
0.46
0.23
0.47
0.01
0.07
0.13
0.38
0.02
0.08
0.15
0.44
MN-DCP
0.70
0.81
0.71
0.86
0.10
0.27
0.26
0.60
0.07
0.19
0.24
0.57
DCP [54]
0.23
0.62
0.04
0.26
0.06
0.22
0.11
0.27
0.03
0.12
0.11
0.27
DCf
0.65
0.85
0.73
0.94
0.22
0.51
0.65
0.82
0.10
0.29
0.38
0.57
Improvement f
0.42
0.23
0.69
0.68
0.16
0,29
0.54
0.55
0.07
0.17
0.27
0.30
RPMNet 62]
0.73
0.97
0.89
0.94
0.05
0.18
0.22
0.45
0.03
0.13
0.22
0.40
Dift-RPMNet
0.90
0.98
0.98
0.99
0.18
0.47
0.51
0.72
0.12
0.29
0.36
0.52
Improvement
0.17
0.01
0.09
0.05
0.13
0.29
0.29
0.27
0.09
0.16
0.14
0.12
Table 1: Quantitative comparisons on TUD-L 220. LINEMOD 18]. and Occlude--LNEEMOD 6
traditional approaches: ICP 4 FGR [66], TEASER [60] and S4PCS (34] alongside eight state-of-
the-art learning- deep registration models: DCP (54) IDAM (28), FMR [2), RPMNet [62],
RGM 15]. RIENet A4, MN-IDAM [II and MN-DCP I.. Although Sec. 3.2.2 revealed that
many deep registration models are theoretically applicable to establish our denoising network, the empirical results in Table I indicate that some models such IDAM, FMR, RGM, and RIENet fail to produce meaningful results in real-world challenges, suggesting their limited capability in learning high-quality denoising network through loss function optimization (see Eq 33p. Consequently, we opt to employ DCP and RPMNet, which demonstrate promising performance, to establish our denoising network and generate their corresponding diffusion variants: Diff-DCP and Diff-RPMNet. (left block) shows that the real-world challenges posed by TUD-L lead to limited performance exhibited by the compared traditional and deep methods. In contrast, Diff-RPMNet achieves the highest registration accuracy across all rotation and translation criteria. Furthermore, both Diff-DCP and Diff-RPMNNet outperform their respective baselines, DCP and RPMNet, by a substantial margin, particularly impressive for the 5 @ mAP (42%t) and 5cm mAP (69%1) improvements achieved by Diff-DCP. Such superior performance can be primarily attributed to two factors: (i) The Bayesian posterior in Eq/ effectively orchestrates the pose change of the source point cloud at each reverse step, mitigating premature convergence to local optima; (ii) The diffusion process generates a diverse range of poses for the source point cloud, facilitating more comprehensive model training. Table P (top block) also strongly confirms these two factors: The baseline DCP model, trained using samples generated by our diffusion process (DiffAug), presents a remarkable precision improvement (factor (ii).. Furthermore, when equipped with the reverse process (Rev., its performance further enhanced (factor (i)). Some qualitative results are provided in Fig 2 Please see Appendix B for additional qualitative results.
Evaluation on LINEMOD and Sccluded-LINEMMOL We further assess the performance of our method on two widely-used real-world 6D object pose estimation datasets: LINEMOD [[]] and Occluded-LINEMOR 66. The former dataset encompasses 15 texture-less household objects situated in cluttered scenes, while the latter a subset comprising 8 texture-less objects with varying degrees of occlusion. As shown in Table [ (middle and right blocks), our Diff-DCP and Biff-RPMNet consistently outperform the compared methods across nearly all rotation and translation mAP criteria, yielding remarkable improvements over their respective baselines, DCP and RPMNet.
Ablation Studies and Analysis
Diffusion Process. (1) We first test the performance variations under different noise schedules. namely the linear schedule [19 and cosine schedule [35], using Diff-DCP for our ablation study. As shown in Table 2 (second block), the cosine schedule tends to yield higher precision on the more challenging LINEMOD and Occluded-LINEMOD datasets, whereas the linear schedule performs better on the comparatively easier TUD-L dataset. We attribute this discrepancy to the fact that the transformation denoising during reverse process in challenging datasets (e.g., LINEMOD) necessitates
8

---

Inputs
RPMNN
DiI-RPMNNtt (Ours)
TUDLL
LINEMOD
O-LINTMOD
Greanfrrc
b 1F.3Ptthie (Ouas)
RRNNN
Figure 2: Qualitative comparisons TUD-L 20
LINEMOD IB and Occluded- 6
TUD-L
LINEMOD
Occluded-LINEMON
Models
5
10"
ccm
2cm
5
10"
lcm
2cm
5"
10a
lem
2cm
DCP
0.23
0.62
0.04
0.26
0.06
0.22
0.11
0.27
0.03
0.12
0.11
0.27
DCP+DiffAug
0.48
0.82
0,56
0.88
0.16
0.42
0.52
0.71
0.07
0.24
0.33
0.51
DCP-Dif[Aug+Red
0.65
0.85
0.73
0.94
0.22
0.51
0.65
0.82
0.10
0.29
0.38
0.57
Linear schedule
0.78
0.93
0.93
0.97
0.16
0.44
0.56
0.75
0.10
0.26
0.35
0.54
Cosine schedule*
0.65
0.85
0.73
0.94
0.22
0.51
0.65
0.82
0.10
0.29
0.38
0.57
Random infer. Deterministic infer."
0.63
0.86
0.68
0.92
0.22
0.50
0.62
0.80
0.11
0.29
0.38
0.55
0.65
0.85
0.73
0.94
0.22
0.51
0.65
0.82
0.10
0.29
0.38
0.57
Truin, steps Train, steps 100
0.61
0,65
0.87
0.19
0.47
0.54
0.73
0.10
0.27
0.35
0.55
0.70
0.92
0.87
0.95
0.18
0.46
0.80
0.12
0.28
0.42
0.57
Train, steps 200"
0.65
0.85
0.73
0.94
0.22
0.51
0.65
0.82
0.10
0.29
0.38
0.57
Table 2: Ablation studies TUD-L (00). LINEMOD [18. and Occluded-LINEMOU 6
the rich sample diversity offered by the cosine schedule. In contrast, an abundance of sample diversity on TUD-L would lead to reduced sample efficiency (many generated pose samples of the source point cloud are useless), thus resulting in degradation of model performance. (2) Additionally, we investigate the registration precision of Diff-DCP across different training steps. Table 2 (bottom block) illustrates that, compared to a small number of steps, such as 50, employing a larger number of diffusion steps, such as 100 and 200, leads to higher precision. This observation stems from the fact that greater number of diffusion steps significantly enhances sample diversity. Notably, Diff-DCP with I' = 200 cannot achieve the best performance on TUD-L. This further validates claim that too rich sample diversity on the easier TUD-L does not bring higher performance due to low sample efficiency.
Reverse Process. (1) We evaluate our model using two infer- ence strategies: deterministic inference and random inference. In deterministic inference, the denoised transformation each time step is equal to the predicted mean from Eq. [1] while in
TUD-L
Transform. error (MAE)
0.6
LINEMOD
O-LINEMOD
random inference, the mean is slightly perturbed by the sampled noise described in Eq. 9 As shown in Table (third block), deterministic inference without noise tends to exhibit greater stability and achieve lower estimation errors. (2) To verify the effectiveness of the Bayesian posterior (Eq. S in guiding the pose change of the source point cloud at each reverse step, plot the mean error changes of the estimated transformations at different denoising time steps. Fig. confirms that, scheduled by the Bayesian posterior, the denoised transformation grad- ually approaches to the optimal transformation. Furthermore, the table in Fig. indicaaess that employing a larger number of reverse steps gradually increases the inference time. Therefore, in our implementation, we set the inference step to 5 to improve the registration efficiency.
0.2
0.0
Reverse steps
#steps
3
4
5
Sec.
0.04
0.07
0.11
0.14
0.17
Figure 3: Variations in prediction error and runtime for different time steps during the reverse process.
Conclusion
In this paper, we have proposed novel and effective SE(3) diffusion model-based point cloud registration framework for robust 6D object pose estimation in real-world scenarios. The framework

---

formulates point cloud registration as denoising diffusion process, enabling progressive refinement of the pose of the source point cloud for precise alignment with the model point cloud. To facilitate the diffusion and reverse processes over the SE(3) manifold, we have introduced the Lie algebra se(3) associated with SE(3) to constrain the transformation transitions. Furthermore, we have derived cegistrationsseciffee variational lower bound to effectively optimize our denoising network. By reformulating our denoising network with a surrogate registration model, different deep registration networks can theoretically be employed within approach. Our extensive experiments challeng- ing real-world datasets have validated the effectiveness of our framework. We discuss the broader impact, limitations, and future work in Appendix C.
7
Acknowledgments
The authors would like to thank the editor and the anonymous reviewers for their critical and constructive comments and suggestions. This work was supported by the National Science Fund of China under (Grant Nos. U1713208, 62361 666670, 62276144) and the Swiss Innovation Agency (Innosuisse).
References
[1] Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, and Simon Lucey. PointnetLK: Robust & efficient point cloud registration using pointnet. In CVPR, 2019.
[2] Kwang-Ho Bac and Derek D Lichti. A method for automated registration of unorganised point clouds. Journal of Photogrammetry and Remote Sensing. 2008
[3] Simon Baker and lain Matthews. Lucas-kanade 20 years on: A unifying framework. Interna- lional journal of computer vision, 2004.
[4] Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor Sessor IV: control paradigms and data structures, 1992.
[5] Sofien Bouaziz, Andrea Tagliasacchi, and Mark Pauly. Sparse iterative closest point. In Computer graphics forum, 2013.
[6] Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6d object pose estimation using 3d object coordinates. In ECCV, 2014.
[7] Xiaozii Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving, In CVPR, 2017.
[8] Dmitry Chetverikov, Dmitry Svirko, Dmitry Stepanov, and Pavel Krsek. The trimmed iterative closest point algorithm. In Object recognition supported by user interaction for service robots, 2002.
[9] Christopher Choy, Wei Dong, and Vladlen Koltun. Deep global registration. In CVPR, pages 2514-2523, 2020.
[10] Alvaro Collet, Manuel Martinez, and Siddhartha S Srinivasa. The moped framework: Object recognition and pose estimation for manipulation. The international journal of robotics research, 2011.
[11] Zheng Dang. Lizhou Wang, Yu Guo, and Mathieu Salzmann. Learning- based point cloud registration for 6d object pose estimation in the real world. In ECCV, 2022.
[12] Zheng Dang, Kwang Moo Yi, Yinlin Hu, Fei Wang, Pascal Fua, and Mathieu Salzmann. Eigendecompositionn-f training of deep networks with zero cigenvalue-basee losses. In ECCV, 2018
[13] Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981.
10

---

[14] Andrew W Fitzgibbon. Robust registration of 2D and 3D point sets. Image and vision computing. 2003.
[15] Kexue Fu, Shaolei Liu, Xiaoyuan Luo, and Manning Wang. Robust point cloud registration framework based on deep graph matching. In CVPR, 2021.
[16] Andreas Geiger, Phillip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
[17] Adrien Gressin, Clément Mallet, Jérôme Demantké, and Nicolas David. Towards 3D lidar point cloud registration improvement using optimal neighborhood knowledge. ISPRS journal of photogrammetry and remote sensing, 2013.
[18] Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilie, Stefan Holzer, Gary Bradski, Kurt Kono- lige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3d objects heavily cluttered scenes. ACCV, 2013.
[19] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
[20] Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kchl, Anders GlentBuch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, et al. Bop: Benchmark for 6d object pose estimation. ECCV, 2018.
[21] Xiaoshui Huang, Guofeng Mei, and Jian Zhang. Feature-metric registration: fast semi- supervised approach for robust point cloud registration without correspondences. In CVPR, 2020.
[22] Xiaoshui Huang, Guofeng Mei, and Jian Zhang. Feature-metric registration: A fast semi- supervised approach for robust point cloud registration without correspondences. CVPR. 2020.
[23] Haobo Jiang, Yaqi Shen, Jin Xie, Jun Li, Jianjun Qian, and Jian Yang. Sampling network guided cross-entropy method for unsupervised point cloud registration. In ICCV, 2021.
[24] Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, and Nassir Navab. Ssd-6d: Making igb-based 3d detection and 6d pose estimation great again. In ICCV, 2017.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1121v:1412.6980, 2014.
[26] Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic. Cosypose: Consistent multi- view multi-object 6d pose estimation. In ECCV, 2020.
[27] Adam Leach, Sebastian M Schmon, Matteo T Degiacomi, and Chris G Willcocks. Denoising diffusion probabilistic models on so (3) for rotational alignment. In ICLR Workshop, 2022.
[28] Jiahao Li, Changhao Zhang. Ziyao Xu, Hangning Zhou, and Chi Zhang. Iterative distance- similarity matrix convolution with muperies point climination for efficient point cloud registration. In ECCV, 2020.
[29] Xiang Li, Lingjing Wang, and Yi Fang. PC-Net: Unsupervised point correspondence learning with neural networks. In 3DV, 2019.
[30] Xiang Li, Lingjing Wang, and Yi Fang. Unsupervised partial point set registration via joint shape completion and registration. arXiv preprint arXiv:2009.0521 0000000 2020.
[31] Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox. Deepim: Deep iterative matching for 6d pose estimation. In ECCV, 2018
[32] Eric Marchand, Hideaki Uchiyama, and Fabien Spindler. Pose estimation for augmented reality:
a hands-on survey. IEEE transactions on visualization and computer graphics, 2015.
[33] Eitan Marder-Eppsterr Project tango. In ACM SIGGRAPH. 2016.

---

[34] Nicolas Mellado, Dror Aiger, and Niloy J Mitra. Super 4pes fast global pointeloud registration via smart indexing. In Computer graphics forum, 2014.
[35] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
[36] Dias Pais, Srikumar Ramalingam, Venu Madhav Govindu, Jacinto C Nascimento, Rama Chellappa, and Pedro Miraldo. 3DRegNet: A deep neural network for 3D point registration. In CVPR, 2020.
[37] Keunhong Park, Arsalan Mousavian, Yu Xiang, and Dieter Fox. Latentfusion: End-to-end differentiable reconstruction and rendering for unseen object pose estimation. In CVPR, 2020.
[38] Kiru Park, Timothy Patten, and Markus Vincze. Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation. In ICCV, 2019.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killcen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perforreform deep learning library. NeurlPS, 2019.
[40] Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G Derpanis, and Kostas Daniilidis. 6-dof object pose from semantic keypoints. In ICRA, pages 20112211120001 1-2018, 2017.
[41] Sida Peng, Yuan Liu, Qixing Huang, Xiaowei Zhou, and Hujun Bao. Pvnet: Pixel-wise voting network for 6dof pose estimation. In CVPR, 2019.
[42] Mahdi Rad and Vincent Lepetit. Bb8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth. ICCV, 2017.
[43] Gregory C Sharp, Sang W Lee, and David K Wehe. ICP registration using invariant features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002.
[44] Yaqi Shen, Le Hui, Haobo Jiang, Jin Xie, and Jian Yang. Reliable inlier evaluation for unsupervised point cloud registration. In AAAI, 2022.
[45] Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. ICML, 2015.
[46] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurIPS, 2019.
[47] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
[48] Lei Sun and Lu Deng. Trivoc: Efficient voting-based consensus maximization for robust point cloud registration with extreme outlier ratios. IEEE Robotics and Automation Letters, 2022.
[49] Martin Sundermeyer, Zoltan-Csaba Marton, Maximilian Durner, Manuel Brucker, and Rudolph Triebel. Implicit 3d orientation learning for 6d object detection from rgb images. In ECCV, 2018.
[50] Bugra Tekin, Sudipta N Sinha, and Pascal Fua. Real-time seamless single shot 6d object pose prediction. In CVPR, 2018.
[51] Ameni Trabelsi, Mohamed Chaabane, Nathaniel Blanchard, and Ross Beveridge. A pose proposal and refinement network for better 6d object pose estimation. In CVPR, 2021.
[52] Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birchfield. Deep object pose estimation for semantic robotic grasping of household objects. In CoRL, 2018.
[53] Julen Urain, Niklas Funk, Jan Peters, and Georgia Chalvatzaki. Se di)ddiffusionf::lld:: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In ICRA, 2023,
12

---

[54] Yue Wang and Justin M Solomon. Deep closest point: Learning representations for point cloud registration. In ICCV, 2019.
[55] Yue Wang and Justin M Solomon. PRNe:: Self-supervised learning for partial-to-partial registration. arXiv preprint arXiv:1910.12240, 2019.
[56] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: deep representation for volumetric shapes. CVPR, 2015.
[57] Yu Xiang. Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convo- lutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199, 2017.
[58] Danfei Xu, Dragomir Anguelov, and Ashesh Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. In CVPR, 2018.
[59] Heng Yang and Luca Carlone. A polynomial-time solution for robust registration with extreme outlier rates. arXiv preprint ar arX:v: 2019.
[60] Heng Yang, Jingnan Shi, and Luca Carlone. Teaser: Fast and certifiable point cloud registration. IEEE Transactions on Robotics, 2020.
[61] Jiaolong Yang, Hongdong Li, and Yunde Jia. Go-ICP: Solving 3D registration efficiently and globally optimally, In ICCV, 2013.
[62] Zi Jian Yew and Gim Hee Lee. Rpm-net: Robust point matching using learned features. In CVPR, 2020.
[63] Sergey Zakharov, Ivan Shugurov, and Slobodan Ilie. Dpod: 6d pose object detector and refiner. ICCV, 2019.
[64] Juyong Zhang, Yuxin Yao, and Bailin Deng. Fast and robust iterative closest point. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
[65] Zelin Zhao, Gao Peng, Haoyu Wang. Hao-Shu Fang, Chengkun Li, and Cewu Lu. Estimating 6d pose from localizing designated surface keypoints. arXiv preprint arXiv; 1812.0381,,,388, 2018.
[66] Qian- Yi Zhou, Jacsik Park, and Vladlen Koltun. Fast global registration. In ECCV, 2016.
[67] Jing Zhu and Yi Fang. Reference grid-assisted network for 3D point signature learning from point clouds. WACV, 2020.
[68] Menglong Zhu, Konstantinos G Derpanis, Yinfei Yang, Samarth Brahmbhatt, Mabel Zhang, Cody Phillips, Matthieu Lecce, and Kostas Danillidis. Single image 3d object detection and pose estimation for grasping. In ICRA, 2014.
13


Reviews:
[{"review_id": "cccl4o67qo", "reviewer": "Reviewer_5ZGT", "summary": "The paper proposes and SE(3) diffusion model for pose estimation i.e. rotation R and translation t for the use case of registration two pointclouds. The paper addresses an important problem in 3D vision and which has applications in Robotics, AR/VR. The author claim that the convential diffusion models won't work for this task of SE(3) diffusion and proposes several changes to constaint the transformation transitions during the diffusion and reverse processes. Results are shown on competing and challenging baselines.", "questions": "Please see the remarks in the weakness section. I am willing to improve my score if the authors address my concerns about 1. more complicated benchmarks i.e comparisons to categorical 6D pose and size estimation or in the wild pose estimation, discussion/comments about parameter tuning for the author's approach, discussion/comparison to timing i.e. speed vs accuracy tradeoff relation to some of the state of the art, fast 6D pose and size estimation approaches.", "limitations": "The author's work focuses on point cloud registration only and requires a depth map or depth sensor and wouldn't work in a monocular i.e. RGB only case", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The technical contributions look sound and the author addresses an important and challenging problem in 3D vision. Adapting conventional diffusion models to SE(3) using a thorough mathematical formulation is convincing. Results show strong improvement on quantitative metrics.", "weaknesses": "1. I wonder why the authors didn't compare or show experiments on more challenging categorical 6D pose and size estimation benchmarks nor discuss those in related works [1][2][3]. Is it a limitation of current work? This should at least be discussed in literature review for the work to have more merit since conventional instance-based pose estimation which assumes known CAD models has mostly been solved achieving higher accuracies. Might this be one of the reasons the author gets smaller improvements in some of the metrics?\n\n2. Relating to my point above, the author only evaluates their approach on a single and very simpler benchmark. I understand this is what the authors are proposing i.e. point cloud registration but adding more benchmarks (within the point cloud registration works such as in the wild pose estimation etcetera) could further reinforce the author's results since it is hard to fully validate the efficacy of the results when evaluated on a single benchmark. Recently, the shift has been more toward the wild pose estimation[4] and zero-shot pose estimation and I wonder if this work can directly be useful to some of the other more complicated problems as well. \n\n3. One of the pain points of traditional ICP is parameter tuning which can improve results although takes a lot of hand-tuning. Does the author's approach suffer from the same problem? like a number of denoising steps and etc. \n\n4. Diffusion models are slow to train and infer. Can the authors comment on their inference speeds vs accuracy i.e. if they use a smaller denoising timestep? In essence, I saw no comparisons to state-of-the-art timing results for 6D pose estimation i.e. [2][3], or a direct discussion of it. Timing is crucial for robotics applications and the work's strong improvement might be downplayed if this is very slow and not real-time.\n\n\n[1] Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation, Wang et al.\n[2] CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation Irshad et al.\n[3] ShAPO : Implicit Representations for Multi Object Shape Appearance and Pose Optimization, Irshad et al.\n[4] Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Dataset, Fu et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Instance-based pose estimation has been solved? Why not compare categorical benchmarks? \n\n**A1**: Thanks for your valuable comments to help improve the quality of our paper. \n\n(1) Although the current RGB(D)-based instance-level pose estimation methods have achieved relatively good performance, they usually require the assumption of access to high-quality RGB images. As such, in more challenging lighting conditions (e.g., low-light conditions or color-varying light conditions), their instance-level estimation precisions would be significantly degraded. As shown in the table below, when we use the Gamma transformation (gamma=5,7,9) to darken the RGB image, the estimation precisions of the SOTA GDR-Net would degrade significantly. Thus, realizing lighting-robust, instance-level pose estimation is still an unresolved and meaningful task for wider applications of 6D pose estimation.\n\n(2) To alleviate the aforementioned dilemma, our paper thus focuses on how to utilize geometry-rich point-cloud data (without any RGB information) to achieve lighting-robust, instance-level pose estimation. The table below verifies that our method presents excellent robustness to challenging lighting conditions.\nHowever, as our method exploits the object-model alignment for instance-level estimation, the inherent intra-class shape variation in category-level estimation would confuse such alignment and thereby reduce the pose accuracy. As such, our current method mainly focuses on instance-level datasets for evaluation, rather than categorical datasets. In the future, following your suggestion, we will add discussions of papers [1, 2, 3] in our related work. \n\nMoreover, by integrating NOCS [R1] (normalized object coordinate space) into our model, the NOCS-enhanced SE(3) diffusion registration framework has the potential to address category-level estimation tasks. Specifically, in the surrogate registration model, we can take NOCS as the category-level model point cloud (PC), which can be used to construct category-level correspondences with the source PC for pose and scale estimations. This enables our SE(3) reverse process to gradually denoise the object pose and scale in a category-level setting. It's noted that our SE(3) diffusion process can directly adapt to category-level tasks without any modification. \n\n[R1] Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation, CVPR’2019.\n\n|   | 5$^{\\circ}$@mAP|10$^{\\circ}$@mAP |1cm@mAP|2cm@mAP|\n|-----|---|---|---|---|\n| GDR-Net (Gamma=5) | 0.10 |0.23|0.06|0.14\n| GDR-Net (Gamma=7) | 0.02 |0.08|0.02|0.05\n| GDR-Net (Gamma=9) | 0.01 |0.04|0.01|0.02\n| Diff-RPMNet (ours) | 0.18|0.47|0.51|0.72\n| Diff-DCP (ours) |  **0.22**|**0.51**|**0.65**|**0.82**\n\n**Q2**:  Only evaluate on a single and very simpler benchmark? Is our method useful for wild or zero-shot pose estimation?\n\n**A2**: (1) We hope to clarify that our method was not solely evaluated on a single and simpler benchmark. Instead, following the SOTA method [R2], we conducted extensive comparisons on three challenging and widely-used 6D estimation benchmarks (TUD-L, LINEMOD, and Occluded-LINEMOD). Although they are instance-level datasets, using pure point-cloud data to achieve lighting-robust, instance-level estimation is still an unresolved yet important problem for wider applications of 6D pose estimation. \n\n(2) It’s a highly interesting idea to extend our framework to wild/zero-shot pose estimation. \nSince our current model focuses on using 3D registration to realize lighting-robust, instance-level estimation, we need to make some modifications for enabling our model to handle wild/zero-shot estimations. Specifically, inspired by NOCS [R1], we can further exploit the vector quantization to compress the different instance models into a set of primitive keypoint representations. As such, these primitive representations can be viewed as the primitive model for constructing correspondences with the unseen object PC to estimate its pose and scale.\n\n[R2]: Learning-based Point Cloud Registration for 6D Object Pose Estimation in the Real World. ECCV’2022\n\n**Q3**: Whether our method suffer from a lot of hand-tuning, such as the number of denoising steps, like in ICP?\n\n**A3**: Our framework can achieve good performance without a lot of hand-tuning like ICP. In contrast, our ablation studies in Table 2 show that different configurations can consistently bring significant performance gains over the baseline. Particularly, for the number of denoising steps, Fig.3 shows that more steps bring lower errors across all datasets. These ablation results effectively validate the robustness of our method to these hyperparameters, and indicate that our method can be free from heavy hand-tuning like ICP for achieving good performance. \n\n**Q4**: Diffusion models are slow to train and infer? Comment on their inference speeds vs accuracy?\n\n**A4**: (1) For training time, to ensure a fair comparison, we take the same training epochs as compared methods for model training. Particularly, during each epoch, given a training sample {X, M, H_0}, the compared methods directly use it for training, while our method generates one diffusion sample {X_t, M, H_t} for training (please refer to Alg. 1). Thus, their numbers of training samples utilized in each epoch is identical. This indicates that our diffusion model shares a comparable training time with other methods and thus our diffusion model is not slow to train.\n\n(2) For inference time, Fig.3 shows that it depends on the number of denoising steps, and a smaller timestep would accelerate the inference speed while decreasing the accuracy. For a good balance, we set denoising steps to 5, which can achieve excellent precisions with tolerable speed (~0.17s). In the future, as discussed in our future work, we will convert our SE(3) diffusion registration from the point-cloud space to the compact feature space for increasing our inference speed."}, {"review_id": "tpYIhJrT9A", "reviewer": "Reviewer_VaQE", "summary": "This paper proposes a SE(3) diffusion model-based point cloud registration framework for robust 6D object pose estimation, which formulates point cloud registration as a denoising diffusion process and enables progressive refinement of the transformation between the source point cloud and the model point cloud. Experiments are conducted on TUD-L, LINEMOD, and Occluded-LINEMOD datasets, and state-of-the-art results are achieved on these datasets.", "questions": "See Weaknesses.", "limitations": "The authors mention the limitation of the proposed method in supplementary material.", "rating": 4, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The paper is overall well written and easy to follow.\n2. Experiments are conducted on three different datasets and comparisons are done to different methods. High performance is achieved.\n", "weaknesses": "1. Since this paper aims for 6D object pose estimation, the compared methods should mainly focus on 6D object pose estimation methods obviously, not just some point cloud registration methods.\n2. The proposed method is trained and tested on 6D object pose estimation datasets, However, the compared methods are point cloud registration methods. Did the authors retrain these learning-based point cloud registration models on the datasets in order to fairly compare with these learning-based methods? I am concerned that this is the key to the improved performance.\n3. The comparisons are done to a few methods in the current manuscript, I suggest more state-of-the-art methods should be included for comparison, e.g., point cloud registration methods (Predator[1], Cofinet[2], Geotransformer[3], and so on) and 6D object pose estimation (Fs-net[4], Ove6d[5], Gpv-pose[6] and so on).  \n[1] Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler. Predator: Registration of 3d point clouds with low overlap. In CVPR, 2021.  \n[2] Hao Yu, Fu Li, Mahdi Saleh, Benjamin Busam, and Slobodan Ilic. Cofinet: Reliable coarse-to-fine correspondences for robust point cloud registration. NeurIPS, 2021.  \n[3] Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In CVPR, 2022.  \n[4] Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, Shen Linlin, and Ales Leonardis. Fs-net: Fast shape-based network for category-level 6d object pose estimation with decoupled rotation mechanism. In CVPR, 2021.  \n[5] Dingding Cai, Janne Heikkila, and Esa Rahtu. Ove6d: Object viewpoint encoding for depth-based 6d object pose estimation. In CVPR, 2022.  \n[6] Yan Di, Ruida Zhang, Zhiqiang Lou, Fabian Manhardt, Xiangyang Ji, Nassir Navab, and Federico Tombari. Gpv-pose: Category-level object pose estimation via geometry-guided point-wise voting. In CVPR, 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Compared methods should focus on 6D object pose estimation methods obviously, not just point cloud registration methods.\n\n**A1**: Thanks for your diligent comments to help improve our work. Considering that the RGB(D)-based pose estimation methods would suffer from limited robustness to challenging lighting conditions (e.g., low-light conditions or color-varying light conditions), our work thus focuses on how to utilize geometry-rich point-cloud data (without any RGB information) to achieve lighting-robust, instance-level pose estimation. Based on this task setting, point cloud registration methods are the suitable and popular 6D pose estimation methods, where the orientation and position of the object can be effectively recovered by registering the scanned object point cloud to the model one. \nHowever, the conventional RGB(D)-based methods are unable to adapt to this setting due to the requirement for additional RGB information. Therefore, to ensure a fair comparison, following [R1], we compare our method with those registration-based pose estimation methods rather than conventional RGB(D)-based methods. In the future, following your suggestion, we will add some RGB(D)-based 6D pose estimation methods (e.g., your listed papers below) for performance comparisons in our revised version.\n\n[R1]: Learning-based Point Cloud Registration for 6D Object Pose Estimation in the Real World. ECCV’2022\n\n**Q2**: Did the authors retrain these learning-based point cloud registration models on the datasets in order to fairly compare with these learning-based methods? I am concerned that this is the key to the improved performance.\n\n**A2**: To ensure a fair comparison, all compared learning-based point cloud registration models are retrained on the training data of 6D object pose estimation datasets for performance comparisons. Particularly, we employ their official codes for implementation and meticulously fine-tune them to enhance their performance on real-world 6D object pose estimation datasets. It should be noted that due to the unavailability of source code for MN-DCP and MN-IDAM, we reference their reported experimental outcomes from the original papers for performance comparisons. \n\n**Q3**: The comparisons are done to a few methods in the current manuscript, I suggest more state-of-the-art methods should be included for comparison.\n\n**A3**: Thanks for your suggestion. We will incorporate the methods you've mentioned for comparisons in our future version. Furthermore, we hope to clarify that the listed registration methods [1, 2, 3] primarily center around scene-level registration rather than object-level. Due to the huge scale difference between the scene and object point clouds, adapting their network architecture and hyperparameters to effectively extend them for object-level registration and object pose estimation is a non-trivial task. Hence, our initial paper version does not engage in a performance comparison with them. In the future, we will make our best effort to fine-tune their network architecture and hyperparameters to better suit object-level registration for pose estimation. Additionally, we note that among these methods, Predator [1] offers a set of hyperparameters specifically for object-level registration, and we retrain it on the TUD-L dataset for comparisons. The table below shows that our method can achieve significant performance advantage over Predator with a lower time cost. \n\n  Moreover, considering that the RGB(D)-based instance-level pose estimation methods still suffer from limited robustness to challenging lighting conditions (e.g., low-light conditions or color-varying light conditions), our paper thus focuses on improving the lighting robustness of instance-level methods via point cloud registration. Hence, we mainly compare our method with the instance-level methods rather than the categorical methods as you listed. In our future version, we will include your suggested methods into our experimental comparisons. \n\n|     |  5$^{\\circ}$@mAP | 10$^{\\circ}$@mAP  | 1cm@mAP  | 2cm@mAP  | Times (Sec.)\n|-----|---|---|---|---|---|\n| Predator |  0.42|\t0.71|\t0.66|\t0.81|\t0.62 | 0.62\n| Diff-DCP (ours) |  0.65 |\t0.85 |\t0.73|\t0.94|\t**0.13**\n| Diff-RPMNet (ours) |  **0.90**|\t**0.98**|\t**0.98**|\t**0.99**| \t0.17\n\n\n\n"}, {"review_id": "f68x4afhuN", "reviewer": "Reviewer_WNiN", "summary": "This paper introduces a point cloud alignment method based on a diffusion model on SE(3). For this, the forward and reverse diffusion processes are performed in the lie group se(3). The method is evaluated on challenging real datasets, showing significant improvements over its baselines.", "questions": "Q1: 179: There are several different ways for interpolating rotations; I've often seen quaternions used for this. Is there a motivation for using the exponential map over other representations? Similarly for the perturbations.\n\nQ2: 268: How are 512 points sampled if the objects are smaller than that (especially in real scenes)?\n\n(Suggestion for future work, no effect on rating, no need to respond): The authors might want to look at SYMSOL [R1], data synthetic dataset especially for dealing with pose ambiguities. \n\n[R1] Murphy, Kieran A., et al. \"Implicit-PDF: Non-Parametric Representation of Probability Distributions on the Rotation Manifold.\", ICML 2021", "limitations": "Limitations and impact are adequately discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Relevant Application and Challenging Method: The paper addresses the highly relevant topic of 6DoF pose estimation using diffusion models, which is an area of significant interest. Applying diffusion models to complex parameter spaces like SE(3) poses a challenging problem that has yet to be fully resolved.\n\nExperimental setup: The method is evaluated on challenging real datasets that are often used in the literature regarding 6DoF pose estimation and refinement. The comparison with a significant number of existing methods is fair and comprehensive. Multiple scenarios and thresholds are evaluated, allowing for a better insight into the strengths and weaknesses of the different methods.\n\nResults: The results validate the method, showing significant improvements over the baseline methods. Several ablation studies allow for insights into the method's parameterizations.\n\nMethod: The method can be used with different backbones, increasing its impact as an add-on method that potentially strengthens any existing baseline.\n\nWriting: The paper is well written and guides the reader through the traditionally notation-heavy math of the diffusion model. The ideas are well motivated and explained. \n\n**Rebuttal** The reviewer appreciates the clarifications and additional experiments provided in the rebuttal and maintains the initial rating.", "weaknesses": "Some related work on diffusion models over SE(3) / SO(3) could be cited. Examples include:\n\n[R2] Leach et al., Denoising diffusion probabilistic models on so(3) for rotational alignment, ICLRW 2022.\n[R3] Urain et al., SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion, ICRA 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: 179: There are several different ways for interpolating rotations; I've often seen quaternions used for this. Is there a motivation for using the exponential map over other representations? Similarly for the perturbations. \n\n**A1**: Thanks for your valuable and positive comments. (1) There are indeed some methods for interpolating 3DoF rotations, such as using quaternion or matrix representations. However, our task needs to employ more complex 6DoF transformation interpolation. In this context, the utilization of the 6DoF exponential map for transformation interpolation has been demonstrated to generate smooth and shortest interpolation path (i.e., geodesic path) between two transformations. Instead, decoupling 6DoF transformation interpolation into separate 3DoF rotation interpolation and 3DoF translation interpolation would suffer from non-smooth, unstable interpolation traces, resulting in the low-quality training sample for denoising-network training (see Alg. 1). The experimental results on TUD-L dataset in table below also empirically validates that using 6DoF exponential map for transformation interpolation can achieve higher estimation precisions than the 3DoF quaternion+3DoF translation interpolation. Therefore, we prefer using the exponential map over other representations for transformation interpolation.\n\n(2) For 6DoF transformation perturbations, both 6DoF exponential-map perturbation and 3DoF quaternion+3DoF translation perturbation are valid options. In our paper, to maintain content coherence, we still employ the exponential map representation for transformation perturbation. \n\n|   | 5$^{\\circ}$@mAP|10$^{\\circ}$@mAP |1cm@mAP|2cm@mAP|\n|-----|---|---|---|---|\n| Diff-RPMNet w/ 3DoF quaternion+3DoF translation | 0.63 |0.93|0.95|0.98\n| Diff-RPMNet w/ 6DoF exponential map | **0.90**|**0.98**|**0.98**|**0.99**\n\n\n**Q2**: 268: How are 512 points sampled if the objects are smaller than that (especially in real scenes)?\n\n**A2**: If the number of scanned object points is less than 512, some points will be sampled repeatedly to meet the required point count. For example, if we want to sample a point set of size 5 from 3 points {p1, p2, p3}, the resulting sampled points would be {p1, p1, p2, p2, p3}, where points {p1, p2} are sampled repeatedly.\n\n**Q3**: Some related work on diffusion models over SE(3) / SO(3) could be cited. \n\n**A3**: Thanks for pointing them out. We will incorporate them into our related work section for a thorough discussion in the revised version. \n"}, {"review_id": "pRY1SxlPjI", "reviewer": "Reviewer_NS5b", "summary": "The authors propose an approach to address the problem of 6D pose estimation on real-world data based on a denoising diffusion process.\nThey introduce a novel diffusion process on the SE(3) manifold, leveraging Lie algebra se(3) to shift the process from the linear Euclidean to the nonlinear SE(3) space. \nFurthermore, a registration-specific variational lower bound is derived as optimization objective for model-based learning.\nBy reformulating the transformation following the posterior of the denoising matching term, the prior mean can be adapted to incorporate a surrogate registration model, enabling the application of existing deep registration models.\nThe experimental results show that not all registration models can deal with real-world data and emphasize the significant effectiveness of the novel SE(3) diffusion process.\n", "questions": "Where do the evaluation results for the related methods in Table 1 come from?\nDid you evaluate the methods yourself?\nIf so, which code base did you use, or where did you find these results?\nMost methods do not provide them in their papers.\n\nWhy are only the qualitative results for RPMNet listed in the Appendix and not also for DCP?\nSince DCP was used in the ablation studies, it would also be interesting to see some visual results of it.\n\n\nAppendix A.1 $\\rightarrow$ equation 5 $\\rightarrow$ penultimate term: distribution $p$ is used in the numerator, but should it not be distribution $q$?", "limitations": "The paper addresses the broader implications in terms of societal and academic impact.\nMoreover, it discusses the limitations of their work and indicated possible future directions based on these limitations.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:   \nA novel diffusion process on the nonlinear SE(3) space for point cloud registration is introduced.\n\nQuality:  \nThe paper is well written in terms of language and organization, and a solid mathematical foundation for the proposed diffusion process is provided.\nThe extensive experiments highlight the superior performance of the approach compared to the respective baselines and a variety of other related work.\nThe ablation studies demonstrate the impact of the approach and the influence of different configurations on multiple datasets.\n\nClarity:  \nDespite the emphasis on the mathematical background, all mathematical derivations are explained concisely and comprehensibly.\nThe figure of the framework overview and the provided pseudo codes of the training and inference algorithms allowing to grasp the underlying concepts easily. \n\nSignificance:  \nThe novel denoising diffusion process significantly outperforms the baselines on real-world data, showing the effectiveness of the approach compared to related work.\n", "weaknesses": "I could not find any major weaknesses. The authors:\n- Explain the method well\n- Conduct extensive experiments and compare their approach to multiple related works on different datasets (only real-world datasets, but this is intended)\n- Provide quantitative and qualitative results and analyzed which part of their approach contribute to the improvements\n- Include ablation studies of the diffusion process (forward & backward) to further demonstrate the impact of their approach\n- The Appendix contains the mathematical derivations in detail, additional visualization, and a section on broader impact, limitations and future work\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Where do the evaluation results for the related methods in Table 1 come from? Did you evaluate the methods yourself? If so, which code base did you use, or where did you find these results? Most methods do not provide them in their papers.\n\n**A1**: Thanks for your positive score and encouraging comments. Due to the unavailability of source code for MN-DCP and MN-IDAM, we reference their reported experimental outcomes from the original papers for performance evaluation. The remaining results such as  FMR, RGM, RIENet, DCP, and RPMNet are evaluated by ourselves. To ensure a fair comparison, we employ their official codes for implementation and meticulously fine-tune them to enhance their performance on real-world 6D object pose estimation datasets. \n\n**Q2**: Why are only the qualitative results for RPMNet listed in the Appendix and not also for DCP? Since DCP was used in the ablation studies, it would also be interesting to see some visual results of it.\n\n**A2**: Thanks for your suggestion. We have presented some qualitative comparisons of DCP and Diff-DCP on TUD-L, LINEMOD and Occluded-LINEMOD datasets in Fig.1 of the uploaded rebuttal material (Please refer to the attached PDF file). In the future, following your suggestion, we will add these visualization results into our qualitative evaluations in our revised version. \n\n**Q3**: In equation 5 of Appendix A.1, distribution p is used in the numerator, but should it not be distribution q?\n\n**A3**: Thanks for pointing this typo out. We will correct it by replacing \"p\" with \"q\" in our revised paper.\n"}, {"review_id": "sBiPOkJtS8", "reviewer": "Reviewer_9j1P", "summary": "The paper tackles model-based 6D object pose estimation using SE(3) diffusion model-based point cloud registration. Point cloud registration is trained with diffusion and denoising processes on SE(3), which gradually perturb the optimal pose and learn a denoising network to refine the noisy transformation progressively. The SE(3) optimization objective is derived from a 3D registration-specific variational lower bound and the denoising network is trained with a surrogate registration model. The perturbation and interpolation on SE(3) are performed with Lie algebra. Experiments on TUD-L, LINEMOD, and Occluded-LINEMOD datasets show the validity of the method.", "questions": "1) Could the authors provide the training time cost comparison of the proposed method and the compared methods? Does it require significantly more time for training compared to existing methods?\n\n2) How about performing the interpolation and perturbation of the transformation on so(3) + R^3 (which is also a linear vector space) instead of se(3)? ", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper reformulates 3D point cloud registration as a diffusion and denoising framework, and the results validate the effectiveness of the framework.\n\n2) The paper is fairly well-written and easy to follow.", "weaknesses": "1) The title of the paper says the method is \"robust\". However, it seems there is a dilemma in sample diversity and sample efficiency for the proposed method. That means a fixed set of hyper-parameters might not work well for different kinds of datasets. In other words, the method is not as robust as claimed in the title.\n\n2) Only mAP is used for evaluating 6D pose estimation, other popular evaluation metrics such as ADD and the BOP metrics should be considered for a more comprehensive understanding of the method's performance.\n\n3) [a] also learns a SE(3) diffusion model. Maybe it should be discussed as a related work. \n\n[a] SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion. ICRA 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: The title of the paper says the method is \"robust\". However, it seems there is a dilemma in sample diversity and sample efficiency for the proposed method. That means a fixed set of hyper-parameters might not work well for different kinds of datasets. In other words, the method is not as robust as claimed in the title.\n\n**A1**: Thank you for your insightful comments and constructive feedback. The “robust” claimed in the title mainly refers to that our SE(3) diffusion registration models can present excellent estimation robustness on real-world pose estimation challenges (e.g., full-range transformation and severe occlusions). Additionally, Table 2 shows that a fixed set of hyper-parameter settings can consistently bring significant performance gain across different datasets, which strongly supports the algorithm robustness of our method. \n\n**Q2**: Only mAP is used for evaluating 6D pose estimation, other popular evaluation metrics such as ADD and the BOP metrics should be considered for a more comprehensive understanding. \n\n**A2**: Thanks for your suggestion. We will add more evaluation metrics (such as ADD and the BOP metrics) in our revised version. In this rebuttal response, we give the results of methods DCP, Diff-DCP, RPMNet and Diff-RPMNet over ADD, and four BOP metrics (MSSD, AD, ADI, PROJ) on LINEMOD benchmark as below. It can be observed that our SE(3) diffusion registration methods (Diff-DCP and Diff-RPMNet) can consistently outperform their baseline methods (DCP and RPMNet) by a large margin on all metrics. We will add these comparison results in our revised version. \n\n|     |  ADD | MSSD  | AD  | ADI  | PROJ |\n|-----|---|---|---|---|---|\n| DCP |  0.226 |\t0.378|\t0.430|\t0.893|\t0.021\n| Diff-DCP |  0.640|\t0.739|\t0.801|\t0.967|\t0.340\n| RPMNet | 0.341| 0.417|\t0.521|\t0.897|\t0.071\n| Diff-RPMNet|  0.609|\t0.710|\t0.779|\t0.959|\t0.267\n\n\n\n**Q3**: [a] also learns a SE(3) diffusion model. Maybe it should be discussed as a related work. \n\n**A3**: Thanks for pointing it out. [a] proposes an SE(3) diffusion model for robotic grasping tasks and presents promising results. Since it’s a concurrent paper with us, we didn’t notice and discuss it in our original paper. We will discuss it as a related work in our revised version. \n\n**Q4**: Training time cost comparison of the proposed method and the compared methods? Does it require significantly more time for training?\n\n**A4**: We hope to clarify that to ensure a fair comparison, we take the same training epochs as compared methods for performance evaluations. Particularly, during each epoch, given a training sample {X, M, H_0}, the compared methods directly use it for training, while our method generates one diffusion sample {X_t, M, H_t} for training (please refer to Alg. 1). Thus, their numbers of training samples utilized in each epoch are also identical. This indicates that our diffusion model shares a comparable training time with other methods and it doesn't require significantly more time for training. \n\n**Q5**: How about performing the interpolation and perturbation of the transformation on so(3) + R^3 instead of se(3)?\n\n**A5**: As shown in table below, compared to se(3) diffusion, so(3)+R^3 diffusion presents some performance degradation on TUD-L dataset. Their performance differences primarily stem from the disparity in quality of training samples generated by se(3) diffusion and so(3)+R^3 diffusion. Specifically, se(3) diffusion is demonstrated to generate smooth and shortest interpolation path (i.e., geodesic path) between two transformations, which can provide high-quality training samples to train the denoising network for a more reliable se(3) reverse process (Please refer to Alg. 1). Instead, so(3)+R^3 diffusion trace would generate non-smooth and unstable diffusion path, which would generate large amounts of low-quality training samples and thereby degrade the pose-correction ability of the trained denoising network. \n\n|   | 5$^{\\circ}$@mAP|10$^{\\circ}$@mAP |1cm@mAP|2cm@mAP|\n|-----|---|---|---|---|\n| Diff-RPMNet w/ so(3)+R^3 | 0.63 |0.93|0.95|0.98\n| Diff-RPMNet w/ se(3) | **0.90**|**0.98**|**0.98**|**0.99**\n\n\n"}]

Author rebuttal:
To address Q2 raised by Reviewer-NS5b, we have included some qualitative comparisons of DCP and Diff-DCP on TUD-L, LINEMOD, and Occluded-LINEMOD datasets in the attached PDF file.

Meta-review: This paper obtained discordant ratings after the review stage. Certain reviewers appreciated the novelty of the point cloud registration approach proposed and the solidity of the results that it obtained. Some other reviewers highlighted the need for clarifications of certain methodological issues and the need for more experimental evaluations. Despite the discordant ratings were not completely resolved during discussion, a clear consensus emerged towards acceptance among the reviewers, with four out of five reviewers eventually suggesting leaning towards acceptance and only one reviewer sticking on a 4 rating. In particular, the comments of the reviewers who expressed a stronger signal for acceptance were deemed particularly meaningful by the AC. The final recommendation is thus to accept this paper.
Decision: Accept (poster)