Title: SE(3) Diffusion Model-based Point Cloud Registration for Robust 6D Object Pose Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: 6D object pose estimation

Point cloud registration

Diffusion probabilistic model
Abstract: In this paper, we introduce an SE(3) diffusion model-based point cloud registration framework for 6D object pose estimation in real-world scenarios. Our approach formulates the 3D registration task as a denoising diffusion process, which progressively refines the pose of the source point cloud to obtain a precise alignment with the model point cloud. Training our framework involves two operations: An SE(3) diffusion process and an SE(3) reverse process. The SE(3) diffusion process gradually perturbs the optimal rigid transformation of a pair of point clouds by continuously injecting noise (perturbation transformation). By contrast, the SE(3) reverse process focuses on learning a denoising network that refines the noisy transformation step-by-step, bringing it closer to the optimal transformation for accurate pose estimation. Unlike standard diffusion models used in linear Euclidean spaces, our diffusion model operates on the SE(3) manifold. This requires exploiting the linear Lie algebra $\mathfrak{se}(3)$ associated with SE(3) to constrain the transformation transitions during the diffusion and reverse processes. Additionally, to effectively train our denoising network, we derive a registration-specific variational lower bound as the optimization objective for model learning. Furthermore, we show that our denoising network can be constructed with a surrogate registration model, making our approach applicable to different deep registration networks. Extensive experiments demonstrate that our diffusion registration framework presents outstanding pose estimation performance on the real-world TUD-L, LINEMOD, and Occluded-LINEMOD datasets.

SE(3) Diffusion Model-based Point Cloud Registration for Robust 6D Object Pose Estimation

Abstract

In this paper, we introduce an SE(3) diffusion model-based point cloud registration framework for 6D object pose estimation in real-world scenarios. Our approach formulates the 3D registration task as a denoising diffusion process, which progressively refines the pose of the source point cloud to obtain a precise alignment with the model point cloud. Training our framework involves two operations: An SE(3) diffusion process and an SE(3) reverse process. The SE(3) diffusion process gradually perturbs the optimal rigid transformation of a pair of point clouds by continuously injecting noise (perturbation transformation). By contrast, the SE(3) reverse process focuses on learning a denoising network that refines the noisy transformation step-by-step, bringing it closer to the optimal transformation for accurate pose estimation. Unlike standard diffusion models used in linear Euclidean spaces, our diffusion model operates on the SE(3) manifold. This requires exploiting the linear Lie algebra se (3) associated with SE(3) to constrain the transformation transitions during the diffusion and reverse processes. Additionally, to effectively train our denoising network, we derive a registration-specific variational lower bound as the optimization objective for model learning. Furthermore, we show that our denoising network can be constructed with a surrogate registration model, making our approach applicable to different deep registration networks. Extensive experiments demonstrate that our diffusion registration framework presents outstanding pose estimation performance on the real-world TUD-L, LINEMOD, and Occluded-LINEMOD datasets. The code is available at https://github.com/JiangHB/DiffusionReg.

1 Introduction

Accurately estimating the 6D pose of an object, comprising both its spatial position and orientation, is a critical task in the field of computer vision, and has been extensively applied in various domains such as robotics grasping [10, 52, 68], augmented reality [32, 33], and autonomous navigation [7, 16, 58]. While substantial efforts have been dedicated to developing methods for 6D pose estimation based on RGB or RGB-D data [26, 42, 41, 52, 38, 63, 49], the advancements of 3D sensors (such as Kinect and LiDAR) and 3D registration techniques has promoted the emergence of point cloud registration-based pose estimation as a promising direction.

Nevertheless, the state-of-the-art methods for object-level 3D registration [44, 62, 15] primarily focus on synthetic data and struggle to yield precise registration on real-world 6D pose estimation datasets such as TUD-L [20], LINEMOD [18], and Occluded-LINEMOD [6]. Unlike the controlled geometric structures, manually defined transformations, and artificial partial overlap present in synthetic data, real-world pose estimation datasets present significant challenges, including large rotations and translations, natural noise interference, and severe occlusions, considerably exacerbating the registration difficulty. Prior attempts to tackling these challenges include [11], which introduces a general match normalization layer to regularize the feature distribution of the source and model point clouds thus facilitating feature matching, and [60, 48], which design RANSAC-based outlier rejection strategy to enhance the registration robustness. While these initial attempts constitute significant steps towards making object-level point cloud registration practical, much work remains to be done to fully address all the aforementioned real-world challenges.

In this paper, inspired by the remarkable success of diffusion models in generative AI [19, 45, 46, 47], we propose to adapt them to the 3D registration field, and introduce an SE(3) diffusion model-based registration method for robust 6D object pose estimation in the real world. Our approach formulates 3D registration as a denoising diffusion process in SE(3) (special Euclidean group), aiming to progressively refine the pose of the source point cloud to achieve a precise alignment with the model point cloud. Training our model then involves two key operations: An SE(3) diffusion process and an SE(3) reverse process. The SE(3) diffusion process progressively converts the optimal transformation between the source and model point clouds into a noise one by continuously injecting perturbation transformations. Conversely, the SE(3) reverse process learns a denoising network to gradually refine the noise transformation to the optimal one. Our diffusion model operates on the SE(3) manifold, which poses the challenge of extending the diffusion/reverse formulas from linear Euclidean space to a nonlinear manifold. To address this challenge, we exploit the linear Lie algebra se (3) associated with SE(3) to conduct linear diffusion/reverse computations and map the se (3) results to SE(3) to obtain the desired diffused/reverse transformations.

To effectively train our denoising network, we exploit a Bayesian formalism and derive a registrationspecific variational lower bound as optimization objective for model optimization. Furthermore, we reformulate our denoising network with a surrogate registration model, which makes our approach applicable to different deep registration methods such as [54, 62]. During the inference stage, the learned denoising network progressively refines an identity transformation to approach the optimal one, given the source and model point clouds as conditioning signals.

Compared to previous registration methods, our diffusion registration framework offers the following two advantages. First, the diffusion process generates a diverse set of poses for the source point cloud, allowing for more comprehensive model training. This increased pose diversity facilitates the model's ability to handle a wider range of rotation/translation and improves its generalization capabilities. Second, by incorporating the Bayesian posterior in the reverse process, our approach effectively guides the update of the source point cloud pose at each reverse step. This guidance mitigates the risk of getting trapped in local optima, leading to more robust and accurate pose estimation results. Our empirical results provide substantial evidence supporting these two advantages.

To summarize, our main contributions are as follows: (i) We introduce a novel SE(3) diffusion model-based 3D registration framework for robust 6D object pose estimation, where the optimal transformation is estimated via a progressive denoising process. (ii) To train our denoising network, we follow a Bayesian approach and establish a 3D registration-specific variational lower bound as optimization objective for our SE(3) diffusion model. Furthermore, we reformulate the denoising network with a surrogate registration model, enabling the integration of different deep registration models into our framework. (iii) To the best of our knowledge, we are the first to successfully adapt the diffusion model from linear Euclidean space to the SE(3) point cloud registration task for 6D object pose estimation. Our extensive experiments on real-world datasets, including TUD-L [20], LINEMOD [18] and Occluded-LINEMOD [6], confirm the effectiveness of our framework.

2 Related Work

Point Cloud Registration. Point cloud registration aims to estimate the rigid transformation between a pair of point clouds. Existing methods can be roughly divided into optimization-based techniques and deep learning-based ones. Iterative Closest Point (ICP) [4] alternately searches for the closest correspondences and estimates transformation until convergence. Go-ICP [61] uses the branchand-bound algorithm to improve the robustness of ICP to initialization. Robust ICP [64] designs a

X

,

M

H

T

H

t

H

t

-

1

H

0

p

θ

(

H

t

-

1

| X

t

,

M

)

q

(

H

t

H

t

1

)

SE(3) Reverse

(

Surrogate registration

)

SE(3) Diffusion

(

SE(3) Interpolation + SE(3) Perturbation

)

|

-

Welsch's function-based robust metric to obtain an outlier-robust alignment evaluation for optimization. Sparse ICP [5] reformulates ICP using sparsity-inducing norms. Many other ICP variants have been proposed, such as[8, 43, 14, 2, 17], exhibiting promising registration performance. Nevertheless, the current main research effort focuses on deep learning models. In this context, DCP [54] constructs pseudo correspondences for SVD-based transformation estimation using deep feature similarity. PRNet [55] leverages keypoint identification and Gumbel softmax for establishing more reliable correspondences. PointNetLK [1] and FMR [21] introduce the Lucas & Kanada (LK) algorithm [3] and inverse compositional (IC) algorithm [3] into deep models for transformation refinement via feature alignment. CEMNet [23] uses a planning-based cross-entropy method for transformation optimization. RPMNet [62], RGM [15] and RIENet [44] leverage Sinkhorn optimization, neighborhood structure consistency, and graph matching, respectively, for robust outlier removal. Many other deep registration models such as [28, 9, 36, 30, 29, 67] have been developed and achieve impressive registration performance. However, these registration models mainly focus on synthetic datasets, such as ModelNet40 [56]. As shown in [11], they still yield limited precision on real-world 6D object pose estimation datasets such as TUD-L [20], LINEMOD [18], and Occluded-LINEMOD [6].

1 1 6D Object Pose Estimation. Estimating the 6D pose (orientation and position) of objects has gained increasing attention in recent years. Early approaches directly estimated object pose from extracted image features through regression or classification. PoseCNN [57] decouples 6D pose estimation into center-based translation regression and quaternion-based rotation regression. SSD-6D [24] extends the single-shot object detector to cover the full 6D object space for pose inference. Trabelsi et al. [51] propose a multi-attentional network for iterative pose refinement using the appearance and flow information. DeepIM [31] and LatentFusion [37] learn pose estimation by minimizing the error between the rendered model and the observation. More recent research has focused on a two-stage estimation pipeline, where 2D keypoints are extracted to solve for 6D pose using the PnP algorithm [13]. Rad et al. [42] take the predicted 2D projections of the corners of 3D bounding boxes as the keypoints, while Zhao et al. [65] manually designate keypoints over the surface of the object model. PVNet[41] establishes a pixel-wise voting network for keypoint estimation using predicted pixel-wise voting vectors. Other methods [6, 40, 50, 52, 49, 52] also present promising pose estimation performance. Recently, thanks to the advances in 3D registration techniques, the community has witnessed a growing interest in 3D registration-based 6D object pose estimation, which recovers the object's 6D pose by estimating the rigid transformation between the object (source) and the model point clouds. This is what we achieve here, but, in contrast to most existing methods which focus on synthetic data, we tackle the challenging scenario of working with real point clouds.

3 Approach

3.1 Revisiting the Euclidean Diffusion Model

Diffusion models, as generative models, aim to generate new data by progressively denoising noisy inputs [19, 45, 46, 47]. Their training phase involves a diffusion process and a reverse process. The diffusion process successively injects Gaussian noise into the data sample x 0 ∼ p data ( p data indicates the distribution of the training data) so as to gradually transform x 0 into the noise data x T ∼ N ( 0 , I ) (standard Gaussian distribution), thus forming a Markov chain x 0 → x 1 →··· → x T . As demonstrated in [19], the random variable x t ∼ q ( x t | x t -1 ) := N ( x t ; √ 1 -β t x t -1 , β t I ) can also be expressed in a closed form x t ∼ q ( x t | x 0 ) , which can be formulated as follows:

Here, the diffusion coefficients ¯ α t = ∏ t s =0 α s = ∏ t s =0 (1 -β s ) , and β s indicates the noise coefficient determined by a linear schedule [19] or a cosine schedule [35]. Then, the reverse process learns a denoising network, a parameterized normal distribution p θ ( x t -1 | x t ) := N ( x t -1 ; µ θ ( x t , t ) , β t I ) , to progressively denoise the noisy data x T into the clean one x 0 , forming a reverse Markov chain x T → x T -1 → ··· → x 0 . Here, µ θ ( x t , t ) indicates the parameterized mean of the normal distribution. The following variational lower bound of the log likelihood over the training data is then derived as the optimization objective for training the denoising network:

Based on Bayes' formula, the random variable x t -1 following the posterior distribution q ( x t -1 | x t , x 0 ) in Eq. 2 can be represented as where the random variable ε ∼ N ( 0 , I ) and the variance scale ˜ β t = 1 -¯ α t -1 1 -¯ α t β t .

3.2 SE(3) Diffusion Registration Model for 6D Object Pose Estimation

In the context of 6D object pose estimation based on point clouds, our objective is to determine the rigid transformation between a partially-scanned source point cloud X = { x i ∈ R 3 } N i =1 and a complete model point cloud M = { m j ∈ R 3 } j M =1 so as to align their overlapping regions precisely. This transformation encompasses a rotation matrix R ∈ SO (3) and a translation vector t ∈ R 3 , representing the orientation of the object and its spatial position, respectively. To obtain a partial source point cloud, following [11], we mask an input depth map to restrict it to the object, and convert the resulting depth values into 3D points using the known camera intrinsic parameters. By contrast, the model point cloud is generated by uniformly sampling a complete mesh model. We denote the optimal transformation H 0 ∈ SE (3) and the identity transformation H ∈ SE (3) as

both of which are represented by 4 × 4 homogeneous transformation matrices.

Inspired by the progressive generation of diffusion models, we adopt the concept of denoising diffusion to tackle the point cloud registration task, specifically in the SE(3) space, and thus propose an SE(3) diffusion model-based 3D registration framework for robust 6D object pose estimation. While [27, 53] have introduced variations of the diffusion model on SE(3) or SO(3) manifolds, they are not suitable for our 3D registration task. The training phase of our framework thus involves an SE(3) diffusion process and an SE(3) reverse process. Given a pair of source and model point clouds, the former continuously disturbs their optimal transformation by injecting noise into it, while the latter aims to learn a denoising network to progressively convert the noisy transformation to the optimal one. As such, during inference, the learned denoising network can be leveraged to recover the transformation between the source and model point clouds through a progressive denoising process. The training phase and inference phase of our algorithm are detailed in Algorithms 1 and 2. Below, we provide a detailed explanation of our SE(3) diffusion process and SE(3) reverse process.

3.2.1 SE(3) Diffusion Process

In accordance with the standard diffusion model described in Sec. 3.1, our SE(3) diffusion process progressively disturbs the optimal transformation H 0 of point clouds X and M into a noisy transformation H T by injecting perturbation transformations (noise) into it, thus forming a diffusion Markov chain: H 0 → H 1 →··· → H T . However, our SE(3) diffusion process features two critical differences from the conventional one: (i) Our diffusion process operates on the nonlinear SE(3) manifold, unlike the standard diffusion process which acts in linear Euclidean space. Consequently, the linear diffusion operations provided in Eq 1 cannot be directly applied to our model. (ii) The standard diffusion process assumes the noise variable x T ∼ N ( 0 , I ) to be centered around the zero vector. However, in our model for 3D registration, we require the noise transformation H T to be centered around the identity transformation H . To account for these differences while generating the

desired diffusion Markov chain, we propose to interpolate the transformations between H 0 and H on the SE(3) manifold, incorporating noise perturbations. Formally, our interpolation-based SE(3) diffusion formula formulates H t ∼ q ( H t | H 0 ) at time step t (1 ≤ t ≤ T ) as

In essence, this formalism utilizes an interpolation function F ( √ ¯ α t ; H 0 , H ) to derive an intermediate transformation that lies between the optimal transformation H 0 and the identity transformation H . Subsequently, we augment this interpolated transformation with a randomly-sampled perturbation transformation (noise), denoted as Exp( γ √ 1 -¯ α t ε ) , to yield the diffused transformation H t . Below, we will introduce these two components in detail.

Transformation Interpolation Function F ( √ ¯ α t ; H 0 , H ) . The interpolation function F : SE (3) × SE (3) × [0 , 1] → SE (3) aims to compute an intermediate transformation between the optimal transformation H 0 and the identity one H . Following the setting of diffusion coefficients in Eq. 1, we set the interpolation weight on transformation H 0 at time step t to √ ¯ α t and, consequently, the interpolation weight on H is 1 - √ ¯ α t . As the time step t increases, the interpolation weight √ ¯ α t on H 0 decreases gradually, resulting in a transition of the interpolated transformation from H 0 to H . However, due to the non-linearity of the SE(3) manifold, direct linear interpolation methods, such as weighted averages, cannot be applied to interpolate intermediate transformations. To address this, we exploit the Lie algebra se (3) associated to SE(3). The Lie algebra se (3) is a linear 6D vector space corresponding to the tangent space of SE(3) at the identity transformation. Transferring any transformation in SE(3) to a 6D vector element in se (3) can be achieved using the logarithm map Log : SE (3) → R 6 ; conversely, the exponential map Exp : R 6 → SE (3) transforms a 6D vector in se (3) back to the SE(3) manifold. Therefore, we can first project the SE(3) transformation to se (3), perform linear interpolation in this tangent space, and then convert the interpolated vector back to SE(3) to obtain the interpolated transformation. Formally, our transformation interpolation function F is thus expressed as which starts by calculating the relative transformation H H -1 0 from the optimal transformation H 0 to the identity transformation H , and mapping it to the linear Lie algebra se (3) through the logarithmic map. In this linear space, we then scale the vector Log( H H -1 0 ) by the interpolation weight 1 - √ ¯ α t , and map this weighted vector back into the corresponding weighted relative transformation using the exponential map. The weighted relative transformation quantifies the offset of the interpolated transformation relative to H 0 . Finally, multiplying the optimal transformation H 0 by the computed transformation offset yields the desired interpolated transformation at time step t .

Perturbation Transformation Exp( γ √ 1 -¯ α t ε ) . Following the standard diffusion formula in Eq. 1, we introduce random noise ( i.e. , a perturbation transformation) into the interpolated transformation F ( √ ¯ α t ; H 0 , H ) at each time step to randomize our SE(3) diffusion process. As indicated in Eq. 1, conventional Euclidean diffusion models typically draw such noise from a Gaussian distribution in Euclidean space. However, formulating a Gaussian distribution on the SE(3) manifold is non-trivial. To tackle this issue, we utilize the Lie algebra once more. Specifically, we randomly sample a 6D noise vector ε ∈ R 6 from N ( 0 , I ) over R 6 , which can be interpreted as an element in se (3). Then, we scale the noise vector by the factor γ √ 1 -¯ α t as in Eq. 1 to control the magnitude of the perturbation at different time steps. Finally, this scaled noise vector is converted back to SE(3)

using the exponential map to obtain the corresponding perturbation transformation. Please refer to Appendix A for more details about our perturbation transformation.

3.2.2 SE(3) Reverse Process

With the diffusion Markov chain H 0 → H 1 →··· → H T generated by the SE(3) diffusion process in Sec. 3.2.1, the objective of the SE(3) reverse process is to train a denoising network p θ ( H t -1 | X t = T ( X , H t ) , M )) to progressively refine the noisy transformation towards the optimal one, thus forming a reverse Markov chain H T → H T -1 →··· → H 0 . In the context of 3D registration, we design our denoising network to predict a probability distribution over the transformation H t -1 given the model point cloud M and the transformed source point cloud X t = T ( X , H t ) = { R t x i + t t } . Here, R t and t t denote the rotation matrix and translation vector of transformation H t . To effectively train our denoising network, we derive the following registration-specific variational lower bound of the log likelihood over the training samples as the optimization objective:

where p data indicates the distribution of the training data, including the pairs of source and model point clouds and their ground-truth rigid transformations. In the following, we elaborate on each loss term of the derived variational lower bound. Please refer to Appendix A for a detailed derivation.

Denoising Matching Term. This is the fundamental loss term for training our denoising network. q ( H t -1 | H t , H 0 ) represents the posterior distribution of H t -1 conditioned on H 0 and H t , while p θ ( H t -1 | X t , M ) denotes the prior distribution over H t -1 predicted by our denoising network. In contrast to the prior distribution, the posterior has access to the optimal transformation H 0 , enabling it to infer a more reliable distribution for H t -1 through Bayes' formula. As a result, we can treat this posterior distribution as ground-truth signal to supervise the prior distribution prediction of the denoising network by minimizing their Kullback-Leibler (KL) divergence. Inspired by the Bayesian posterior in Eq. 3, the random transformation H post t -1 following the posterior distribution, H post t -1 ∼ q ( H t -1 | H t , H 0 ) , can be expressed as

where we employ the logarithm map to convert the SE(3) transformations H 0 and H t to their corresponding 6D vector representations in the linear Lie algebra se (3). This allows us to perform a linear combination with posterior coefficients λ 0 and λ 1 , as in Eq. 3. The randomness of H post t -1 arises from the addition of a random variable ε , which follows a Gaussian distribution N ( 0 , I ) over R 6 . The resulting vector is then converted back from se (3) to SE(3) using the exponential map, yielding H post t -1 . Analogously, the random transformation following the prior distribution, H prior t -1 ∼ p θ ( H t -1 | X t , M ) , can be represented as

where µ θ ( X t , M ) denotes the parameterized mean of our prior distribution, and √ ˜ β t ε represents the random term that is identical to that of the posterior. As such, minimizing the KL divergence between the prior and posterior distributions is equivalent to minimizing the error between Log( µ θ ( X t , M )) and λ 0 Log( H 0 ) + λ 1 Log( H t ) , i.e.,

The loss function 10 indicates that, at each time step t , given the point clouds X t = T ( X , H t ) and M , the mean µ θ ( X t , M ) needs to be predicted as the transformation Exp( λ 0 log( H 0 ) + λ 1 log( H t )) rather than the relative transformation H 0 H -1 t between X t and M . Consequently, existing deep registration models such as [54, 62] may not be directly applicable for parameterizing the mean of our prior distribution. Moreover, designing a specific network that takes X t and M as input but predicts a non-relative transformation also is a non-trivial task. To handle it, we consider that the transformation Exp( λ 0 Log( H 0 ) + λ 1 log( H t )) can be rewritten as

This inspires us to reformulate µ θ ( X t , M ) using a surrogate registration model f θ ( X t , M ) as

Then, minimizing the loss function 10 is equivalent to optimizing the surrogate registration model f θ ( X t , M ) to predict the relative transformation H t → 0 = H 0 H -1 t between the point clouds X t and M . Therefore, different deep registration models can potentially serve as surrogate registration models in our denoising network. Finally, we optimize the surrogate registration network by minimizing the L 1 distance between the source points transformed using the ground-truth transformation H t → 0 and the predicted one ˆ H t → 0 = f θ ( X t , M ) . This is written as

where t ∈ { 2 , ..., T } and x t i indicates the i -th point in the transformed source point cloud X t .

Residual and Prior Matching Terms. To maximize the probability p θ ( H 0 | X 1 , M ) , the mean µ θ ( X 1 , M ) = Exp( λ 0 Log( f θ ( X 1 , M ) H 1 ) + λ 1 Log( H 1 )) should be optimized to align closely with H 0 . As such, the optimization target of f θ ( X 1 , M ) is H 0 H -1 1 , and the loss function can be represented as L 1 ( θ ) in 13. In addition, as the prior matching term does not require learning any parameters, it can be regarded as a constant and hence can be omitted.

4 Experiments

4.1 Experimental Settings

Implementation Details. We set the numbers of points in the source and model point clouds to N = 512 and M = 1024 through random sampling. For the SE(3) diffusion process, we adopt a cosine schedule [35] to determine the diffusion coefficients { β t } . The number of diffusion steps T is set to 200, and the scaling coefficient γ for the perturbation transformation is set to 0.1. For the SE(3) reverse process, the number of reverse steps in the training phase is set to 200, while that in the inference phase is set to 5 to accelerate the inference speed of the diffusion registration. We use the ADAMoptimizer [25] with a learning rate of 0.001 to optimize the loss function 13 for 20 epochs with a batch size of 32, and employ PyTorch [39] to implement our framework. All experiments are conducted on a server equipped with an Intel i5 2.2 GHz CPU and one TITAN RTX GPU.

Evaluation Metrics. Following [11], we evaluate the model performance by quantifying the rotation and translation errors between the predicted rotation and translation ˆ R and ˆ t , and the ground-truth ones R ∗ and t ∗ . The evaluation metrics are defined as

As in [12, 11], we summarize these errors via mean average precision (mAP) under varying thresholds.

4.2 Comparison with Existing Methods

Evaluation on TUD-L. We first evaluate our approach on the TUD-L dataset [20], a real-world dataset comprising three household objects. The compared methods encompass four representative

traditional approaches: ICP [4], FGR [66], TEASER [60], and S4PCS [34], alongside eight state-ofthe-art learning-based deep registration models: DCP [54], IDAM [28], FMR [22], RPMNet [62], RGM [15], RIENet [44], MN-IDAM [11] and MN-DCP [11]. Although Sec. 3.2.2 revealed that many deep registration models are theoretically applicable to establish our denoising network, the empirical results in Table 1 indicate that some models such as IDAM, FMR, RGM, and RIENet fail to produce meaningful results in real-world challenges, suggesting their limited capability in learning a high-quality denoising network through loss function optimization (see Eq.13). Consequently, we opt to employ DCP and RPMNet, which demonstrate promising performance, to establish our denoising network and generate their corresponding diffusion variants: Diff-DCP and Diff-RPMNet . Table 1 (left block) shows that the real-world challenges posed by TUD-L lead to limited performance exhibited by the compared traditional and deep methods. In contrast, Diff-RPMNet achieves the highest registration accuracy across all rotation and translation criteria. Furthermore, both Diff-DCP and Diff-RPMNet outperform their respective baselines, DCP and RPMNet, by a substantial margin, particularly impressive for the 5 ◦ @mAP (42% ↑ ) and 5cm@mAP (69% ↑ ) improvements achieved by Diff-DCP. Such superior performance can be primarily attributed to two factors: (i) The Bayesian posterior in Eq.8 effectively orchestrates the pose change of the source point cloud at each reverse step, mitigating premature convergence to local optima; (ii) The diffusion process generates a diverse range of poses for the source point cloud, facilitating more comprehensive model training. Table 2 (top block) also strongly confirms these two factors: The baseline DCP model, trained using samples generated by our diffusion process ( DiffAug ), presents a remarkable precision improvement (factor (ii)). Furthermore, when equipped with the reverse process ( Rev. ), its performance is further enhanced (factor (i)). Some qualitative results are provided in Fig.2. Please see Appendix B for additional qualitative results.

Evaluation on LINEMOD and Occluded-LINEMOD. We further assess the performance of our method on two widely-used real-world 6D object pose estimation datasets: LINEMOD [18] and Occluded-LINEMOD [6]. The former dataset encompasses 15 texture-less household objects situated in cluttered scenes, while the latter is a subset comprising 8 texture-less objects with varying degrees of occlusion. As shown in Table 1 (middle and right blocks), our Diff-DCP and Diff-RPMNet consistently outperform the compared methods across nearly all rotation and translation mAP criteria, yielding remarkable improvements over their respective baselines, DCP and RPMNet.

5 Ablation Studies and Analysis

Diffusion Process. (1) We first test the performance variations under different noise schedules, namely the linear schedule [19] and cosine schedule [35], using Diff-DCP for our ablation study. As shown in Table 2 (second block), the cosine schedule tends to yield higher precision on the more challenging LINEMOD and Occluded-LINEMOD datasets, whereas the linear schedule performs better on the comparatively easier TUD-L dataset. We attribute this discrepancy to the fact that the transformation denoising during reverse process in challenging datasets ( e.g. , LINEMOD) necessitates

RPMNet

Diff-RPMNet (Ours)

Inputs

TUD-L

LINEMOD

O-LINEMOD

Ground truth

Diff-RPMNet (Ours)

RPMNet

LINEMOD

O-LINEMOD

Table 2: Ablation studies on TUD-L [20], LINEMOD [18], and Occluded-LINEMOD [6]. the rich sample diversity offered by the cosine schedule. In contrast, an abundance of sample diversity on TUD-L would lead to reduced sample efficiency (many generated pose samples of the source point cloud are useless), thus resulting in a degradation of model performance. (2) Additionally, we investigate the registration precision of Diff-DCP across different training steps. Table 2 (bottom block) illustrates that, compared to a small number of steps, such as T = 50 , employing a larger number of diffusion steps, such as 100 and 200, leads to higher precision. This observation stems from the fact that a greater number of diffusion steps significantly enhances sample diversity. Notably, Diff-DCP with T = 200 cannot achieve the best performance on TUD-L. This further validates our claim that too rich sample diversity on the easier TUD-L does not bring higher performance due to low sample efficiency.

Reverse Process. (1) We evaluate our model using two inference strategies: deterministic inference and random inference. In deterministic inference, the denoised transformation at each time step is equal to the predicted mean from Eq. 12, while in random inference, the mean is slightly perturbed by the sampled noise as described in Eq. 9. As shown in Table 2 (third block), deterministic inference without noise tends to exhibit greater stability and achieve lower estimation errors. (2) To verify the effectiveness of the Bayesian posterior (Eq. 8) in guiding the pose change of the source point cloud at each reverse step, we plot the mean error changes of the estimated transformations at different denoising time steps. Fig. 3 confirms that, scheduled by the Bayesian posterior, the denoised transformation gradually approaches to the optimal transformation. Furthermore, the table in Fig. 3 indicates that employing a larger number of reverse steps gradually increases the inference time. Therefore, in our implementation, we set the inference step to 5 to improve the registration efficiency.

6 Conclusion

In this paper, we have proposed a novel and effective SE(3) diffusion model-based point cloud registration framework for robust 6D object pose estimation in real-world scenarios. The framework

1

2

3

4

5

Reverse steps

0.0

0.2

0.4

0.6

Transform. error (MAE)

TUD-L

LINEMOD

O-LINEMOD

#steps

1

2

3

4

5

Sec.

0.04

0.07

0.11

0.14

0.17

formulates point cloud registration as a denoising diffusion process, enabling progressive refinement of the pose of the source point cloud for precise alignment with the model point cloud. To facilitate the diffusion and reverse processes over the SE(3) manifold, we have introduced the Lie algebra se (3) associated with SE(3) to constrain the transformation transitions. Furthermore, we have derived a registration-specific variational lower bound to effectively optimize our denoising network. By reformulating our denoising network with a surrogate registration model, different deep registration networks can theoretically be employed within our approach. Our extensive experiments on challenging real-world datasets have validated the effectiveness of our framework. We discuss the broader impact, limitations, and future work in Appendix C.

7 Acknowledgments

The authors would like to thank the editor and the anonymous reviewers for their critical and constructive comments and suggestions. This work was supported by the National Science Fund of China under (Grant Nos. U1713208, 62361166670, 62276144) and the Swiss Innovation Agency (Innosuisse).

References

Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, and Simon Lucey. PointnetLK: Robust & efficient point cloud registration using pointnet. In CVPR , 2019.

Kwang-Ho Bae and Derek D Lichti. A method for automated registration of unorganised point clouds. Journal of Photogrammetry and Remote Sensing , 2008.

Simon Baker and Iain Matthews. Lucas-kanade 20 years on: A unifying framework. International journal of computer vision , 2004.

Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures , 1992.

Sofien Bouaziz, Andrea Tagliasacchi, and Mark Pauly. Sparse iterative closest point. In Computer graphics forum , 2013.

Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother. Learning 6d object pose estimation using 3d object coordinates. In ECCV , 2014.

Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In CVPR , 2017.

Dmitry Chetverikov, Dmitry Svirko, Dmitry Stepanov, and Pavel Krsek. The trimmed iterative closest point algorithm. In Object recognition supported by user interaction for service robots , 2002.

Christopher Choy, Wei Dong, and Vladlen Koltun. Deep global registration. In CVPR , pages 2514-2523, 2020.

Alvaro Collet, Manuel Martinez, and Siddhartha S Srinivasa. The moped framework: Object recognition and pose estimation for manipulation. The international journal of robotics research , 2011.

Zheng Dang, Lizhou Wang, Yu Guo, and Mathieu Salzmann. Learning-based point cloud registration for 6d object pose estimation in the real world. In ECCV , 2022.

Zheng Dang, Kwang Moo Yi, Yinlin Hu, Fei Wang, Pascal Fua, and Mathieu Salzmann. Eigendecomposition-free training of deep networks with zero eigenvalue-based losses. In ECCV , 2018.

Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM , 1981.

Andrew W Fitzgibbon. Robust registration of 2D and 3D point sets. Image and vision computing , 2003.

Kexue Fu, Shaolei Liu, Xiaoyuan Luo, and Manning Wang. Robust point cloud registration framework based on deep graph matching. In CVPR , 2021.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR , 2012.

Adrien Gressin, Clément Mallet, Jérôme Demantké, and Nicolas David. Towards 3D lidar point cloud registration improvement using optimal neighborhood knowledge. ISPRS journal of photogrammetry and remote sensing , 2013.

Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In ACCV , 2013.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS , 2020.

Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders GlentBuch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, et al. Bop: Benchmark for 6d object pose estimation. In ECCV , 2018.

Xiaoshui Huang, Guofeng Mei, and Jian Zhang. Feature-metric registration: A fast semisupervised approach for robust point cloud registration without correspondences. In CVPR , 2020.

Xiaoshui Huang, Guofeng Mei, and Jian Zhang. Feature-metric registration: A fast semisupervised approach for robust point cloud registration without correspondences. In CVPR , 2020.

Haobo Jiang, Yaqi Shen, Jin Xie, Jun Li, Jianjun Qian, and Jian Yang. Sampling network guided cross-entropy method for unsupervised point cloud registration. In ICCV , 2021.

Wadim Kehl, Fabian Manhardt, Federico Tombari, Slobodan Ilic, and Nassir Navab. Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again. In ICCV , 2017.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic. Cosypose: Consistent multiview multi-object 6d pose estimation. In ECCV , 2020.

Adam Leach, Sebastian M Schmon, Matteo T Degiacomi, and Chris G Willcocks. Denoising diffusion probabilistic models on so (3) for rotational alignment. In ICLR Workshop , 2022.

Jiahao Li, Changhao Zhang, Ziyao Xu, Hangning Zhou, and Chi Zhang. Iterative distance-aware similarity matrix convolution with mutual-supervised point elimination for efficient point cloud registration. In ECCV , 2020.

Xiang Li, Lingjing Wang, and Yi Fang. PC-Net: Unsupervised point correspondence learning with neural networks. In 3DV , 2019.

Xiang Li, Lingjing Wang, and Yi Fang. Unsupervised partial point set registration via joint shape completion and registration. arXiv preprint arXiv:2009.05290 , 2020.

Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang, and Dieter Fox. Deepim: Deep iterative matching for 6d pose estimation. In ECCV , 2018.

Eric Marchand, Hideaki Uchiyama, and Fabien Spindler. Pose estimation for augmented reality: a hands-on survey. IEEE transactions on visualization and computer graphics , 2015.

Eitan Marder-Eppstein. Project tango. In ACM SIGGRAPH . 2016.

Nicolas Mellado, Dror Aiger, and Niloy J Mitra. Super 4pcs fast global pointcloud registration via smart indexing. In Computer graphics forum , 2014.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML , 2021.

G Dias Pais, Srikumar Ramalingam, Venu Madhav Govindu, Jacinto C Nascimento, Rama Chellappa, and Pedro Miraldo. 3DRegNet: A deep neural network for 3D point registration. In CVPR , 2020.

Keunhong Park, Arsalan Mousavian, Yu Xiang, and Dieter Fox. Latentfusion: End-to-end differentiable reconstruction and rendering for unseen object pose estimation. In CVPR , 2020.

Kiru Park, Timothy Patten, and Markus Vincze. Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation. In ICCV , 2019.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS , 2019.

Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G Derpanis, and Kostas Daniilidis. 6-dof object pose from semantic keypoints. In ICRA , pages 2011-2018, 2017.

Sida Peng, Yuan Liu, Qixing Huang, Xiaowei Zhou, and Hujun Bao. Pvnet: Pixel-wise voting network for 6dof pose estimation. In CVPR , 2019.

Mahdi Rad and Vincent Lepetit. Bb8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth. In ICCV , 2017.

Gregory C Sharp, Sang W Lee, and David K Wehe. ICP registration using invariant features. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2002.

Yaqi Shen, Le Hui, Haobo Jiang, Jin Xie, and Jian Yang. Reliable inlier evaluation for unsupervised point cloud registration. In AAAI , 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML , 2015.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. NeurIPS , 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Lei Sun and Lu Deng. Trivoc: Efficient voting-based consensus maximization for robust point cloud registration with extreme outlier ratios. IEEE Robotics and Automation Letters , 2022.

Martin Sundermeyer, Zoltan-Csaba Marton, Maximilian Durner, Manuel Brucker, and Rudolph Triebel. Implicit 3d orientation learning for 6d object detection from rgb images. In ECCV , 2018.

Bugra Tekin, Sudipta N Sinha, and Pascal Fua. Real-time seamless single shot 6d object pose prediction. In CVPR , 2018.

Ameni Trabelsi, Mohamed Chaabane, Nathaniel Blanchard, and Ross Beveridge. A pose proposal and refinement network for better 6d object pose estimation. In CVPR , 2021.

Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birchfield. Deep object pose estimation for semantic robotic grasping of household objects. In CoRL , 2018.

Julen Urain, Niklas Funk, Jan Peters, and Georgia Chalvatzaki. Se (3)-diffusionfields: Learning smooth cost functions for joint grasp and motion optimization through diffusion. In ICRA , 2023.

Yue Wang and Justin M Solomon. Deep closest point: Learning representations for point cloud registration. In ICCV , 2019.

Yue Wang and Justin M Solomon. PRNet: Self-supervised learning for partial-to-partial registration. arXiv preprint arXiv:1910.12240 , 2019.

Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In CVPR , 2015.

Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199 , 2017.

Danfei Xu, Dragomir Anguelov, and Ashesh Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. In CVPR , 2018.

Heng Yang and Luca Carlone. A polynomial-time solution for robust registration with extreme outlier rates. arXiv preprint arXiv:1903.08588 , 2019.

Heng Yang, Jingnan Shi, and Luca Carlone. Teaser: Fast and certifiable point cloud registration. IEEE Transactions on Robotics , 2020.

Jiaolong Yang, Hongdong Li, and Yunde Jia. Go-ICP: Solving 3D registration efficiently and globally optimally. In ICCV , 2013.

Zi Jian Yew and Gim Hee Lee. Rpm-net: Robust point matching using learned features. In CVPR , 2020.

Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic. Dpod: 6d pose object detector and refiner. In ICCV , 2019.

Juyong Zhang, Yuxin Yao, and Bailin Deng. Fast and robust iterative closest point. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Zelin Zhao, Gao Peng, Haoyu Wang, Hao-Shu Fang, Chengkun Li, and Cewu Lu. Estimating 6d pose from localizing designated surface keypoints. arXiv preprint arXiv:1812.01387 , 2018.

Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Fast global registration. In ECCV , 2016.

Jing Zhu and Yi Fang. Reference grid-assisted network for 3D point signature learning from point clouds. In WACV , 2020.

Menglong Zhu, Konstantinos G Derpanis, Yinfei Yang, Samarth Brahmbhatt, Mabel Zhang, Cody Phillips, Matthieu Lecce, and Kostas Daniilidis. Single image 3d object detection and pose estimation for grasping. In ICRA , 2014.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles model-based 6D object pose estimation using SE(3) diffusion model-based point cloud registration. Point cloud registration is trained with diffusion and denoising processes on SE(3), which gradually perturb the optimal pose and learn a denoising network to refine the noisy transformation progressively. The SE(3) optimization objective is derived from a 3D registration-specific variational lower bound and the denoising network is trained with a surrogate registration model. The perturbation and interpolation on SE(3) are performed with Lie algebra. Experiments on TUD-L, LINEMOD, and Occluded-LINEMOD datasets show the validity of the method.\n\nSTRENGTHS:\n1) The paper reformulates 3D point cloud registration as a diffusion and denoising framework, and the results validate the effectiveness of the framework.\n\n2) The paper is fairly well-written and easy to follow.\n\nWEAKNESSES:\n1) The title of the paper says the method is \"robust\". However, it seems there is a dilemma in sample diversity and sample efficiency for the proposed method. That means a fixed set of hyper-parameters might not work well for different kinds of datasets. In other words, the method is not as robust as claimed in the title.\n\n2) Only mAP is used for evaluating 6D pose estimation, other popular evaluation metrics such as ADD and the BOP metrics should be considered for a more comprehensive understanding of the method's performance.\n\n3) [a] also learns a SE(3) diffusion model. Maybe it should be discussed as a related work. \n\n[a] SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion. ICRA 2023.\n\nQUESTIONS:\n1) Could the authors provide the training time cost comparison of the proposed method and the compared methods? Does it require significantly more time for training compared to existing methods?\n\n2) How about performing the interpolation and perturbation of the transformation on so(3) + R^3 (which is also a linear vector space) instead of se(3)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an approach to address the problem of 6D pose estimation on real-world data based on a denoising diffusion process.\nThey introduce a novel diffusion process on the SE(3) manifold, leveraging Lie algebra se(3) to shift the process from the linear Euclidean to the nonlinear SE(3) space. \nFurthermore, a registration-specific variational lower bound is derived as optimization objective for model-based learning.\nBy reformulating the transformation following the posterior of the denoising matching term, the prior mean can be adapted to incorporate a surrogate registration model, enabling the application of existing deep registration models.\nThe experimental results show that not all registration models can deal with real-world data and emphasize the significant effectiveness of the novel SE(3) diffusion process.\n\nSTRENGTHS:\nOriginality:   \nA novel diffusion process on the nonlinear SE(3) space for point cloud registration is introduced.\n\nQuality:  \nThe paper is well written in terms of language and organization, and a solid mathematical foundation for the proposed diffusion process is provided.\nThe extensive experiments highlight the superior performance of the approach compared to the respective baselines and a variety of other related work.\nThe ablation studies demonstrate the impact of the approach and the influence of different configurations on multiple datasets.\n\nClarity:  \nDespite the emphasis on the mathematical background, all mathematical derivations are explained concisely and comprehensibly.\nThe figure of the framework overview and the provided pseudo codes of the training and inference algorithms allowing to grasp the underlying concepts easily. \n\nSignificance:  \nThe novel denoising diffusion process significantly outperforms the baselines on real-world data, showing the effectiveness of the approach compared to related work.\n\nWEAKNESSES:\nI could not find any major weaknesses. The authors:\n- Explain the method well\n- Conduct extensive experiments and compare their approach to multiple related works on different datasets (only real-world datasets, but this is intended)\n- Provide quantitative and qualitative results and analyzed which part of their approach contribute to the improvements\n- Include ablation studies of the diffusion process (forward & backward) to further demonstrate the impact of their approach\n- The Appendix contains the mathematical derivations in detail, additional visualization, and a section on broader impact, limitations and future work\n\nQUESTIONS:\nWhere do the evaluation results for the related methods in Table 1 come from?\nDid you evaluate the methods yourself?\nIf so, which code base did you use, or where did you find these results?\nMost methods do not provide them in their papers.\n\nWhy are only the qualitative results for RPMNet listed in the Appendix and not also for DCP?\nSince DCP was used in the ablation studies, it would also be interesting to see some visual results of it.\n\n\nAppendix A.1 $\\rightarrow$ equation 5 $\\rightarrow$ penultimate term: distribution $p$ is used in the numerator, but should it not be distribution $q$?\n\nLIMITATIONS:\nThe paper addresses the broader implications in terms of societal and academic impact.\nMoreover, it discusses the limitations of their work and indicated possible future directions based on these limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a point cloud alignment method based on a diffusion model on SE(3). For this, the forward and reverse diffusion processes are performed in the lie group se(3). The method is evaluated on challenging real datasets, showing significant improvements over its baselines.\n\nSTRENGTHS:\nRelevant Application and Challenging Method: The paper addresses the highly relevant topic of 6DoF pose estimation using diffusion models, which is an area of significant interest. Applying diffusion models to complex parameter spaces like SE(3) poses a challenging problem that has yet to be fully resolved.\n\nExperimental setup: The method is evaluated on challenging real datasets that are often used in the literature regarding 6DoF pose estimation and refinement. The comparison with a significant number of existing methods is fair and comprehensive. Multiple scenarios and thresholds are evaluated, allowing for a better insight into the strengths and weaknesses of the different methods.\n\nResults: The results validate the method, showing significant improvements over the baseline methods. Several ablation studies allow for insights into the method's parameterizations.\n\nMethod: The method can be used with different backbones, increasing its impact as an add-on method that potentially strengthens any existing baseline.\n\nWriting: The paper is well written and guides the reader through the traditionally notation-heavy math of the diffusion model. The ideas are well motivated and explained. \n\n**Rebuttal** The reviewer appreciates the clarifications and additional experiments provided in the rebuttal and maintains the initial rating.\n\nWEAKNESSES:\nSome related work on diffusion models over SE(3) / SO(3) could be cited. Examples include:\n\n[R2] Leach et al., Denoising diffusion probabilistic models on so(3) for rotational alignment, ICLRW 2022.\n[R3] Urain et al., SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion, ICRA 2022.\n\nQUESTIONS:\nQ1: 179: There are several different ways for interpolating rotations; I've often seen quaternions used for this. Is there a motivation for using the exponential map over other representations? Similarly for the perturbations.\n\nQ2: 268: How are 512 points sampled if the objects are smaller than that (especially in real scenes)?\n\n(Suggestion for future work, no effect on rating, no need to respond): The authors might want to look at SYMSOL [R1], data synthetic dataset especially for dealing with pose ambiguities. \n\n[R1] Murphy, Kieran A., et al. \"Implicit-PDF: Non-Parametric Representation of Probability Distributions on the Rotation Manifold.\", ICML 2021\n\nLIMITATIONS:\nLimitations and impact are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a SE(3) diffusion model-based point cloud registration framework for robust 6D object pose estimation, which formulates point cloud registration as a denoising diffusion process and enables progressive refinement of the transformation between the source point cloud and the model point cloud. Experiments are conducted on TUD-L, LINEMOD, and Occluded-LINEMOD datasets, and state-of-the-art results are achieved on these datasets.\n\nSTRENGTHS:\n1. The paper is overall well written and easy to follow.\n2. Experiments are conducted on three different datasets and comparisons are done to different methods. High performance is achieved.\n\nWEAKNESSES:\n1. Since this paper aims for 6D object pose estimation, the compared methods should mainly focus on 6D object pose estimation methods obviously, not just some point cloud registration methods.\n2. The proposed method is trained and tested on 6D object pose estimation datasets, However, the compared methods are point cloud registration methods. Did the authors retrain these learning-based point cloud registration models on the datasets in order to fairly compare with these learning-based methods? I am concerned that this is the key to the improved performance.\n3. The comparisons are done to a few methods in the current manuscript, I suggest more state-of-the-art methods should be included for comparison, e.g., point cloud registration methods (Predator[1], Cofinet[2], Geotransformer[3], and so on) and 6D object pose estimation (Fs-net[4], Ove6d[5], Gpv-pose[6] and so on).  \n[1] Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, Andreas Wieser, and Konrad Schindler. Predator: Registration of 3d point clouds with low overlap. In CVPR, 2021.  \n[2] Hao Yu, Fu Li, Mahdi Saleh, Benjamin Busam, and Slobodan Ilic. Cofinet: Reliable coarse-to-fine correspondences for robust point cloud registration. NeurIPS, 2021.  \n[3] Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In CVPR, 2022.  \n[4] Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, Shen Linlin, and Ales Leonardis. Fs-net: Fast shape-based network for category-level 6d object pose estimation with decoupled rotation mechanism. In CVPR, 2021.  \n[5] Dingding Cai, Janne Heikkila, and Esa Rahtu. Ove6d: Object viewpoint encoding for depth-based 6d object pose estimation. In CVPR, 2022.  \n[6] Yan Di, Ruida Zhang, Zhiqiang Lou, Fabian Manhardt, Xiangyang Ji, Nassir Navab, and Federico Tombari. Gpv-pose: Category-level object pose estimation via geometry-guided point-wise voting. In CVPR, 2022.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe authors mention the limitation of the proposed method in supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes and SE(3) diffusion model for pose estimation i.e. rotation R and translation t for the use case of registration two pointclouds. The paper addresses an important problem in 3D vision and which has applications in Robotics, AR/VR. The author claim that the convential diffusion models won't work for this task of SE(3) diffusion and proposes several changes to constaint the transformation transitions during the diffusion and reverse processes. Results are shown on competing and challenging baselines.\n\nSTRENGTHS:\nThe technical contributions look sound and the author addresses an important and challenging problem in 3D vision. Adapting conventional diffusion models to SE(3) using a thorough mathematical formulation is convincing. Results show strong improvement on quantitative metrics.\n\nWEAKNESSES:\n1. I wonder why the authors didn't compare or show experiments on more challenging categorical 6D pose and size estimation benchmarks nor discuss those in related works [1][2][3]. Is it a limitation of current work? This should at least be discussed in literature review for the work to have more merit since conventional instance-based pose estimation which assumes known CAD models has mostly been solved achieving higher accuracies. Might this be one of the reasons the author gets smaller improvements in some of the metrics?\n\n2. Relating to my point above, the author only evaluates their approach on a single and very simpler benchmark. I understand this is what the authors are proposing i.e. point cloud registration but adding more benchmarks (within the point cloud registration works such as in the wild pose estimation etcetera) could further reinforce the author's results since it is hard to fully validate the efficacy of the results when evaluated on a single benchmark. Recently, the shift has been more toward the wild pose estimation[4] and zero-shot pose estimation and I wonder if this work can directly be useful to some of the other more complicated problems as well. \n\n3. One of the pain points of traditional ICP is parameter tuning which can improve results although takes a lot of hand-tuning. Does the author's approach suffer from the same problem? like a number of denoising steps and etc. \n\n4. Diffusion models are slow to train and infer. Can the authors comment on their inference speeds vs accuracy i.e. if they use a smaller denoising timestep? In essence, I saw no comparisons to state-of-the-art timing results for 6D pose estimation i.e. [2][3], or a direct discussion of it. Timing is crucial for robotics applications and the work's strong improvement might be downplayed if this is very slow and not real-time.\n\n\n[1] Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation, Wang et al.\n[2] CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation Irshad et al.\n[3] ShAPO : Implicit Representations for Multi Object Shape Appearance and Pose Optimization, Irshad et al.\n[4] Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Dataset, Fu et al.\n\nQUESTIONS:\nPlease see the remarks in the weakness section. I am willing to improve my score if the authors address my concerns about 1. more complicated benchmarks i.e comparisons to categorical 6D pose and size estimation or in the wild pose estimation, discussion/comments about parameter tuning for the author's approach, discussion/comparison to timing i.e. speed vs accuracy tradeoff relation to some of the state of the art, fast 6D pose and size estimation approaches.\n\nLIMITATIONS:\nThe author's work focuses on point cloud registration only and requires a depth map or depth sensor and wouldn't work in a monocular i.e. RGB only case", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}