Title: Generative Category-level Object Pose Estimation via Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Category-Level Object Pose Estimation

Diffusion Model
Abstract: Object pose estimation plays a vital role in embodied AI and computer vision, enabling intelligent agents to comprehend and interact with their surroundings. Despite the practicality of category-level pose estimation, current approaches encounter challenges with partially observed point clouds, known as the multihypothesis issue. In this study, we propose a novel solution by reframing categorylevel object pose estimation as conditional generative modeling, departing from traditional point-to-point regression. Leveraging score-based diffusion models, we estimate object poses by sampling candidates from the diffusion model and aggregating them through a two-step process: filtering out outliers via likelihood estimation and subsequently mean-pooling the remaining candidates. To avoid the costly integration process when estimating the likelihood, we introduce an alternative method that distils an energy-based model from the original score-based model, enabling end-to-end likelihood estimation. Our approach achieves state-of-the-art performance on the REAL275 dataset, surpassing 50% and 60% on strict 5 ◦ 2cm and 5 ◦ 5cm metrics, respectively. Furthermore, our method demonstrates strong generalization to novel categories without the need for fine-tuning and can readily adapt to object pose tracking tasks, yielding comparable results to the current state-of-the-art baselines. Our checkpoints and demonstrations can be found at https://sites.google.com/view/genpose.

GenPose: Generative Category-level Object Pose Estimation via Diffusion Models

Abstract

Object pose estimation plays a vital role in embodied AI and computer vision, enabling intelligent agents to comprehend and interact with their surroundings. Despite the practicality of category-level pose estimation, current approaches encounter challenges with partially observed point clouds, known as the multihypothesis issue. In this study, we propose a novel solution by reframing categorylevel object pose estimation as conditional generative modeling, departing from traditional point-to-point regression. Leveraging score-based diffusion models, we estimate object poses by sampling candidates from the diffusion model and aggregating them through a two-step process: filtering out outliers via likelihood estimation and subsequently mean-pooling the remaining candidates. To avoid the costly integration process when estimating the likelihood, we introduce an alternative method that trains an energy-based model from the original scorebased model, enabling end-to-end likelihood estimation. Our approach achieves state-of-the-art performance on the REAL275 dataset and demonstrates promising generalizability to novel categories sharing similar symmetric properties without fine-tuning. Furthermore, it can readily adapt to object pose tracking tasks, yielding comparable results to the current state-of-the-art baselines. Our checkpoints and demonstrations can be found at https://sites.google.com/view/genpose .

1 Introduction

Object pose estimation is a fundamental problem in the domains of robotics and machine vision, providing a high-level representation of the surrounding world. It plays a crucial role in enabling intelligent agents to understand and interact with their environments, supporting tasks such as robotic manipulation, augmented reality, and virtual reality [1; 2; 3]. In the existing literature, categorylevel object pose estimation [4; 5; 6] has emerged as a more practical approach compared to instancelevel pose estimation since the former eliminates the need for a 3D CAD model for each individual instance. The main challenge of this task is to capture the general properties while accommodating the intra-class variations, i.e ., variations among different instances within a category [7; 5; 8].

(a) Symmetric

(b) Partial Observation

𝑣𝑖𝑒𝑤

!

…

…

…

…

𝑣𝑖𝑒𝑤

"

𝑣𝑖𝑒𝑤

#

𝑣𝑖𝑒𝑤

$

𝑣𝑖𝑒𝑤

!

𝑣𝑖𝑒𝑤

"

𝑣𝑖𝑒𝑤

#

𝑣𝑖𝑒𝑤

$

To address this task, previous studies have explored regression-based approaches. These approaches involve either recovering the object pose by predicting correspondences between the observed point cloud and synthesized canonical coordinates [4; 7], or estimating the pose in an end-to-end manner [9; 6]. The former approaches leverage category prior information, such as the mean point cloud, to handle intra-class variations. As a result, they have demonstrated strong performance across various metrics. On the other hand, the latter approaches do not rely on the coordinate mapping process, which inherently makes them faster and amenable to end-to-end training.

Despite the promising results of previous approaches, a fundamental issue inherent to regressionbased training persists. This issue, known as the multi-hypothesis problem , arises when dealing with a partially observed point cloud. In such cases:

Multiple feasible pose hypotheses can exist, but the network can only be supervised for a single pose due to the regression-based training.

Figure 1(a) illustrates this problem by showcasing the feasible ground truth poses of symmetric objects ( e.g ., bowls). Moreover, partially observed point clouds of an object may appear similar from certain views ( e.g ., a mug with an obstructed handle may look the same from certain views, as shown in Figure 1(b)), further exacerbating the multiple hypothesis issue. Previous studies have proposed ad-hoc solutions to tackle this issue, such as designing special network architectures [10; 11] or augmenting the ground truth poses [4; 5] for symmetric objects. However, these approaches cannot fundamentally resolve this issue due to the lack of generality. Consequently, an ideal object pose estimator should model a pose distribution instead of regressing a single pose when presented with a partially observed point cloud.

To this end, we formulate category-level object pose estimation as conditional generative modeling that inherently models the distribution of multiple pose hypotheses conditioned on a partially observed point cloud. To model the conditional pose distribution, we employ score-based diffusion models, which have demonstrated promising results in various conditional generation tasks [12]. Specifically, we estimate the score functions, i.e ., the gradients of the log density, of the conditional pose distribution perturbed by different noise levels. These trained gradient fields can provide guidance to iteratively refine a pose, making it more compatible with the observed point cloud. In the testing phase, we estimate the object pose of a partial point cloud by sampling from the conditional pose distribution using an MCMC process incorporated with the learned gradient fields. As a result, our method can propose multiple pose hypotheses for a partial point cloud, thanks to stochastic sampling.

However, it is essential to note that a sampled pose might be an outlier of the conditional pose distribution, meaning it has a low likelihood, which can severely hurt the performance. To address this issue, we sample a group of pose candidates and then filter out the outliers based on the estimated likelihoods. Unfortunately, estimating the likelihoods from score-based models requires a highly time-consuming integration process [13], rendering this approach impractical. To overcome this challenge, we propose an alternative solution that trains an energy-based model from the original score-based model for likelihood estimation. Following training, the energy network is guaranteed to estimate the log-likelihood of the original data distribution up to a constant. By ranking the candidates according to the energy network's output, we can filter out candidates with low ranks ( e.g ., the last 40%). Finally, the output pose is aggregated by mean-pooling the remaining pose candidates.

Our experiments demonstrate several superiorities over the existing methods. Firstly, without any ad-hoc network or loss design for symmetry objects, our method achieves state-of-the-art performance and surpasses 50% and 60% on the strict 5 ◦ 2cm and 5 ◦ 5cm metrics, respectively, on the REAL275 dataset. Besides, our method can directly generalize to novel categories without any fine-tuning to some degree due to its prior-free nature. Our methods can be easily adapted to the object pose tracking task with few modifications and achieve comparable performance with the SOTA baselines.

Our contributions are summarized as follows:

We study a fundamental problem, namely multi-hypothesis issue , that has existed in category-level object pose estimation for a long time and introduce a generative approach to address this issue.

We propose a novel framework that leverages the energy-based diffusion model to aggregate the candidates generated by the score-based diffusion model for object pose estimation.

Our framework achieves exceptionally state-of-the-art performance on existing benchmarks, especially on symmetric objects. In addition, our framework is capable of object pose tracking and can generalize to objects from novel categories sharing similar symmetric properties.

2 Related Works

2.1 Category Level Object Pose Estimation

Category-level object pose estimation [4; 14; 5; 15] aims to predict pose of various instances within the same object category without requiring CAD models. To this end, NOCS [4] introduce normalized object coordinate space for each object category and predicts the corresponding object shape in canonical space. Object pose is then calculated by pose fitting using the Umeyama [16] algorithm. CASS [17], DualPoseNet [9], and SSP-Pose [18] end-to-end regress the pose and simultaneously reconstruct the object shape in canonical space to improve the precision of pose regression. SARNet [10] and GPV-Pose [11] specifically design symmetric-aware reconstruction to enhance the performance of predicting the translation and size of symmetrical objects while regressing the object pose. However, direct regression of pose or object coordinates in canonical space fail when there is considerable intra-class variation. To overcome this, FS-Net [6] introduces an online box-cage-based 3D deformation augmentation method, and RBP-Pose [19] proposes a nonlinear data augmentation method. On the other hand, SPD [7], SGPA [5], CR-Net [20], and DPDN [8] introduce category-level object priors ( e.g ., mean shape) and learn how to deform the prior to get the object coordinates in canonical space. These methods have demonstrated strong performance. Overall, the existing category-level object pose estimation methods are regression-based. However, considering symmetric objects ( e.g ., bowls, bottles, and cans) and partially observed objects ( e.g ., mugs without visible handles), there are multiple plausible hypotheses for object pose. Therefore, the problem of object pose estimation should be formulated as a generation problem rather than a regression problem.

2.2 Score-based Generative Models

In the realm of estimating the gradient of the log-likelihood pertaining to specified data distribution, a pioneering approach known as the score-based generative model [21; 22; 23; 24; 25; 13; 26] was originally introduced by [22]. In an effort to provide a feasible substitute objective for scorematching, the denoising score-matching (DSM) technique[21] has further put forward a viable proposition. To enhance the scalability of the score-based generative model, [23] has introduced a sliced score-matching objective, which involves projecting the scores onto random vectors prior to their comparison. They have also presented annealed training for denoising score matching[24] and have introduced improved training techniques to complement these methods [25]. Additionally, they have extended the discrete levels of annealed score matching to a continuous diffusion process, thereby demonstrating promising results in the domain of image generation [13]. Recent studies have delved further into exploring the design choices of the diffusion process [27], maximum likelihood training [26], and deployment on the Riemann manifold [28]. These advancements have showcased encouraging outcomes when applying score-based generative models to high-dimensional domains, thus fostering their broad utilization across various fields, such as object rearrangement [29], medical imaging [30], point cloud generation [31], scene graph generation [32], point cloud denoising [33], depth completion [34], and human pose estimation [35]. These studies have formulated perceptionrelated problems as either conditional generative modeling or in-painting tasks, thereby harnessing the power of score-based generative models to tackle these challenges. In contrast to these approaches, we incorporate the score-based diffusion model with an additional energy-based diffusion model [36; 37] for filtering out the outliers so as to improve the performance by aggregating the remaining candidates.

3 Method

Task Description: In this work, we aim to estimate 6D object pose from the partially observed point cloud. The learning agent is given a training set with paired object poses and point clouds D = { ( p i , O i ) } n i =1 , where p i ∈ SE (3) and O i ∈ R 3 × N denote a 6D pose and a partially observed 3D point cloud with N points respectively. Given an unseen point cloud O ∗ , the goal is to recover the corresponding ground-truth pose p ∗ .

Overview: We formulate the object pose estimation as a conditional generative modeling problem and train a score-based diffusion model Φ θ using the dataset D . Further, an energy-based diffusion model Ψ ϕ is trained from the score-based model Φ θ for likelihood estimation. During test time, given an unseen point cloud O ∗ , we first sample a group of pose candidates { ˆ p 1 , ˆ p 2 , ... ˆ p K } via the score-based diffusion model Φ θ . Then we sort the candidates { ˆ p i } K i =1 in descending order according

(I) Training

(II) Inference

Φ

!

Ψ

"

{ $

𝒑

!

}

!"#

$

Partial Point

Cloud

𝑂

∗

(a) Candidates Generation

…

…

(b) Likelihood Estimation

Reverse ODE Process

(c) Pose Aggregation

Mean  Pooling

{Ψ

&

( $

𝒑

!

, 𝜀|𝑂)}

!"#

$

Ranking by Pose Energies

remove

outliers

Top

𝛿 / 𝐾

…

Φ

!

Ψ

"

Examples

ScoreNet

EnergyNet

𝒑

!

, 𝑂

!

!"#

'

score

matching

score

matching

∇Ψ

1

𝒑 2

#

𝒑 2

(

𝒑 2

)

𝒑 2

*

{ $

𝒑

!

}

!"#

$

Partial Point

Cloud

𝑂

∗

𝒩(0,𝜎

+,-

𝐼

(

)

𝜀

to the energy outputs and filter out the last 1 -δ %candidates ( e.g ., δ = 60% ). Finally, we aggregate the remaining candidates by mean-pooling to obtain the estimated pose ˆ p .

3.1 Sampling Pose Candidates via Score-based Diffusion Model

To address the multi-hypothesis issue, an ideal pose estimator should be capable of being trained on multiple feasible ground truth poses given the same point cloud. Also, the ideal pose estimator should be able to output all possible pose hypotheses of the given point cloud during inference.

To this end, we propose to tackle object pose estimation in a conditional generative modeling paradigm. We assume dataset D is sampled from an implicit joint distribution D = { ( p i , O i ) ∼ p data ( p , O ) } . We aim to model the conditional pose distribution p data ( p | O ) during training, and sample pose hypotheses of an unseen point cloud O ∗ from p data ( p | O ∗ ) during test-time.

Specifically, we employ a score-based diffusion model [24; 23; 13] to estimate the conditional distribution p data ( p | O ) . We adopt Variance-Exploding (VE) Stochastic Differential Equation (SDE) proposed by [13] to construct a continuous diffusion process { p ( t ) } 1 t =0 indexed by a time variable t ∈ [0 , 1] where p (0) ∼ p data ( p | O ) denotes the ground truth pose of the point cloud O . As the t increases from 0 to 1, the time-indexed pose variable p ( t ) is perturbed by the following SDE:

where σ min = 0 . 01 and σ max = 50 are hyper-parameters.

During training, we aim to estimate the score function of the perturbed conditional pose distribution ∇ p log p t ( p | O ) of all t , where the p t ( p | O ) denotes the marginal distribution of p ( t ) :

Notably, when t = 0 , p 0 ( p (0) | O ) = p data ( p (0) | O ) is exactly the data distribution.

Thanks to the Denoising Score Matching (DSM) [21], we can obtain a guaranteed estimation of ∇ p p t ( p | O ) by training a score network Φ θ : R |P| × R 1 × R 3 × N → R |P| via the following objective:

where ϵ is a hyper-parameter that denotes the minimal noise level. When minimizes the objective in Eq. 3, the optimal score network satisfies Φ θ ( p , t | O ) = ∇ p log p t ( p | O ) according to [21].

After training, we can approximately sample pose candidates { ˆ p i } K i =1 from p data ( p | O ) by sampling from p ϵ ( p | O ) , as lim ϵ → 0 p ϵ ( p | O ) = p data ( p | O ) . To sample from p ϵ ( p | O ) , we can solve the following Probability Flow (PF) ODE [13] where p (1) ∼ N ( 0 , σ 2 max I ) , from t = 1 to t = ϵ :

where the score function log p t ( p | O ) is empirically approximated by the estimated score network Φ θ ( p , t | O ) and the ODE trajectory is solved by RK45 ODE solver [38].

In practice, the score network Φ θ ( p , t | O ) is implemented without any ad-hoc design for the symmetric objects: The point cloud O is encoded into a global feature by PointNet++ [39], the input pose p is encoded by feed-forward MLPs and the time variable t is encoded by a commonly used projection layer following [13]. The pose p is represented as a 9-D variable [ R | T ] , where R ∈ R 6 and T ∈ R 3 denote rotation and translation vectors, respectively. Due to the discontinuity of quaternions and Euler angles in Euclidean space, we employ the continuous 6-D rotation representation [ R x | R y ] following[6; 40]. We defer full details into Appendix A.

3.2 Aggregating Pose Candidates via Energy-based Diffusion Model

Though we can sample pose candidates { ˆ p i } K i =1 from the conditional pose distribution p data ( p | O ) via the learned score model Φ θ , we still need to determine a final output estimation ˆ p ∈ SE (3) . In other words, the pose candidates need to be aggregated into a single estimation.

An initial approach to aggregating candidates is mean pooling. However, when sampling candidates from p data ( p | O ) , there is no guarantee that the sampled candidates will have a high likelihood. This means that outlier poses in low-density regions are still likely to be included in the mean-pooled pose, which can negatively impact its performance. Therefore, we propose estimating the data likelihoods { p data (ˆ p i | O ) } K i =1 to rank the candidates and then filter out low-ranking candidates.

Unfortunately, estimating likelihood via the score model itself requires a time-consuming integration process [13], which is impractical for real-time applications:

where { p ( t ) } t ∈ [0 , 1] is the same forward diffusion process as in Eq. 1.

To this end, we propose to train an energy-based model Ψ ϕ : R |P| × R 1 × R 3 × N → R 1 that enables an end-to-end surrogate estimation of the data likelihood by supervising the energy-induced gradient ∇ p Ψ ϕ ( p , t | O ) with the denoising score-matching objective in Eq. 3:

In this way, the optimal energy model holds ∇ p Ψ ∗ ϕ ( p , t | O ) = ∇ p log p t ( p | O ) , or equivalently, Ψ ∗ ϕ ( p , t | O ) = log p t ( p | O ) + C where C is a constant. Although the optimal energy model and the ground truth likelihood differ by a constant C , this does not prevent it from functioning as a good surrogate likelihood estimator for ranking the candidates:

Nevertheless, training an energy-based diffusion model from Eq. 6 is known to be difficult and time-inefficient due to the need to calculate the second-order derivations in the objective function. Following [36], we parameterize the energy-based model as Ψ ϕ ( p , t | O ) = ⟨ p , Φ ϕ ( p , t | O ) ⟩ to alleviate the training burden. We defer the full training details to Appendix A.

With the trained energy model, we sort the candidates into a sequence ˆ p τ 1 ≻ ˆ p τ 2 ... ≻ ˆ p τ K where:

Subsequently, we filter out the last 1 -δ %candidates and obtain ˆ p τ 1 ≻ ˆ p τ 2 ... ≻ ˆ p τ M where δ ∈ (0 , 1) is a hyper parameter and M = ⌊ δ · K ⌋ .

Finally, we aggregate the remaining candidates { ˆ p τ i = ( ˆ T τ i , ˆ R τ i ) } i M =1 by averaging the rotations { ˆ R τ i } i M =1 and the translations { ˆ T τ i } i M =1 respectively, to obtain the output pose ˆ ˆ p = ( ˆ T, ˆ R ) . In specific, the translations are pooled by vanilla averaging ˆ T = ∑ i M =1 ˆ T τ i M . To obtain the averaged rotation, we initially translate the rotations into quaternions { ˆ q τ i } i M =1 . Following [41], the average quaternion can then be found by the following maximization procedure:

By definition, the solution of Eq. 9 is the eigenvector of the 4 × 4 matrix ∑ i M =1 A ( q τ i ) M corresponding to the maximum eigenvalue, which can be efficiently solved by QUEST algorithm [42].

3.3 Discussion

Despite addressing the multi-hypothesis issue, our method offers several additional advantages:

No Ad-hoc Design: Unlike previous approaches, we do not incorporate any ad-hoc designs or tricks into the network architecture for symmetric objects. Both the score and energy models are implemented using commonly used feature extractors ( e.g ., PointNet++[39]) and feed-forward MLPs. Surprisingly, our method performs exceptionally well in handling symmetric objects (refer to Sec 4.4) and achieves state-of-the-art (SOTA) performance on existing benchmarks (refer to Sec 4.2).

Prior-Free: Our method eliminates the requirement of category-level canonical prior, freeing us from designing a shape-deformation module to incorporate the canonical prior. This also provides the potential to generalize our method to objects from unseen categories sharing similar symmetric properties. In Sec 4.4, we present experiments to demonstrate this potential.

Capable of Pose Tracking: Thanks to the closed-loop generation process of the diffusion models, we can adapt the single-frame pose estimation framework to pose tracking tasks by warm-starting from the previous predictions. The adapted object pose tracking framework differs from the single-frame estimation framework only in the candidates' sampling. For each frame that receives the point cloud observation O cur, we initialize the PF-ODE in Eq.4 with p (0 . 1) ∼ N (ˆ p prev , σ 2 (0 . 1) I ) , where ˆ p prev ∈ SE (3) denotes the estimated pose of the previous frame. Subsequently, we solve this modified PF-ODE from t = 0 . 1 to t = ϵ using the gradient fields Φ θ ( p , t | O cur ) to sample candidates for pose tracking. By following the same aggregation procedure, we can obtain the pose estimation ˆ p cur for the current frame. We summarise the tracking framework in Appendix D. In Sec.4.5, our adapted pose-tracking method outperforms the state-of-the-art object pose-tracking baseline in most metrics.

4 Experiments

4.1 Experimental Setup

Datasets. Our method is trained and evaluated on two common category-level object pose estimation datasets, namely CAMERA and REAL275 [4], following the NOCS [4] convention for splitting the data into training and testing sets. These datasets include 6 daily objects: bottle, bowl, camera, can, laptop, and mug. CAMERA dataset is a synthetic dataset generated using mixed-reality methods. It comprises real background images with synthetic rendered foreground objects and consists of 275K training images and 25K test images. REAL275 dataset is a real-world dataset that employs the same 6 object categories as the CAMERA dataset. It includes 3 unique instances per category in both the training and test sets and consists of 7 scenes for training with 4.3K images and 6 scenes for testing with 2.75K images. Each scene in the REAL275 dataset contains more than 5 objects.

Metrics. Following NOCS [4], we report the mean Average Precision (mAP) in n ◦ and m cm to evaluate the accuracy of object pose estimation. Here, n and m denote the prediction error of rotation and translation, respectively, where the predicted rotation error is less than n ◦ and the predicted translation error is less than m cm. Specifically, we use 5 ◦ 2 cm , 5 ◦ 5 cm , 10 ◦ 2 cm , and 10 ◦ 5 cm as our evaluation metrics. Similar to NOCS, for symmetric objects (bottles, bowls, and cans), we ignore the

rotation error around the symmetry axis. For the 'mug' category, we consider it a symmetric object when the handle is not visible and a non-symmetric object when the handle is visible.

Implementation Details. For a fair comparison, we employed the instance mask generated by MaskRCNN [43] during the inference phase, which was identical to the one used in previous work. Both the Energy-based diffusion model for pose ranking and the score-based diffusion model for pose generation shared the same network structure. The input pointcloud consisted of 1024 points. During the training phase, we used the same data augmentation techniques as FS-Net [6], which are widely adopted in category-level object pose estimation tasks. All experiments were conducted on a single RTX3090 with a batch size of 192. All the experiments are implemented using PyTorch [44].

4.2 Comparison with State-of-the-Art Methods

Table 1 provides a comprehensive comparison between our method and the state-of-the-art approaches on the REAL275 [4] dataset, highlighting a significant advancement in performance. In Table 1, 'Ours' represents the aggregated pose using K as 50 and δ as 60%. 'Ours( K =10)' and 'Ours( K =50)' indicate that we set K as 10 and 50, respectively, and select a pose from candidates with the minimum distance to ground truth pose for evaluation. We deal with all categories by a single model.

As shown in Table 1, our approach demonstrates a remarkable improvement over the current SOTA method, GPV-Pose[11], exceeding it by more than 20% in both the rigorous 5 ◦ 2 cm and 5 ◦ 5 cm evaluation metrics. Significantly, for the first time, we achieve results on the REAL275 dataset, surpassing the impressive thresholds of 50% and 60% in the 5 ◦ 2 cm and 5 ◦ 5 cm metrics, respectively. These exceptional outcomes further support the efficacy of our approach.

Even when compared to methods that take RGB-D data and category prior as input, our method maintains a notable advantage. Without relying on any additional information, our method achieves a performance boost of over 10% in the metric of 5 ◦ 5 cm , surpassing the SOTA method, DPDN [8].

In addition, the results that evaluate the pose nearest to the ground truth indicate that condition generative models exhibit tremendous potential for category-level object pose estimation. Another crucial metric is the number of learnable parameters of the network, which significantly impacts the feasibility of model deployment. Table 1 reveals that our method achieves a remarkable accuracy for predicting poses with the fewest network parameters, further improving its deployability.

4.3 Ablation Studies

Number of Pose Candidates and Proportion of Selected Poses. Using the 10 ◦ 2 cm metric, Table 2 demonstrates the impact of two factors on performance: the number of generated pose candidates M , and the proportion δ of selected poses out of the total pose candidates.

The results show a notable improvement when M increases from 10 to 50. This enhancement can be attributed to the increasing number of sampling times, resulting in a pose candidate set that aligns more closely with the predicted distribution. However, the improvement becomes marginal when M is further increased from 50 to 100. This can be attributed to the predictions that approach the upper limit of the

aggregation approach. After considering the trade-off between performance and overhead, we ultimately decided to adopt K = 50 . In view of the inherent challenge in training an energy model that precisely aligns with the actual distribution, it becomes apparent that employing a small delta value, as demonstrated in Table 2, does not yield an optimal outcome. Consequently, we regard the energy model primarily as a detector for outliers. The most successful results of this paper were achieved using a delta value of 60%, which effectively mitigated the presence of relatively insignificant noise originating from the sampling process.

Effectiveness of the Energy-based Likelihood Estimator Ψ ϕ . In this part, we report the results of three distinct ranking methodologies, namely 'Random', 'Energy', and 'GT', employed to sort pose candidates { ˆ p i } K i =1 . 'Random' applies a stochastic shuffling to { ˆ p i } K i =1 , providing a baseline approach. 'Energy' uses our trained energy model to rank { ˆ p i } K i =1 . 'GT' ranks { ˆ p i } K i =1 in ascending order of the distances to the ground truth pose, serving as an upper limit for all ranking methods.

The absence of mean pooling in Table 3 implies that one pose candidate is chosen randomly for evaluation. We observe that utilizing an energybased likelihood estimator as a sampler enhances the sort-

ing of pose candidates and effectively eliminates outliers. This improvement is significant compared to random sampling. Although doesn't reach the upper bound, we offer a novel and practical approach for ranking and aggregating poses sampled from estimated distribution and demonstrate its potential.

To better understand the energy model's impact, we further investigated the correlation between pose error and energy output. As depicted in Figure 3, there is a general negative correlation between the output and the error of the energy model. However, the energy model excels at distinguishing poses with significant error differences but performs poorly when distinguishing poses with low errors. For instance, the energy model assigns similar values ( e.g ., translation, < 2 cm ) or even incorrect values ( e.g ., rotation, < 10 ◦ ) for poses with low errors. Additionally, the energies associated with low-error poses ( e.g ., < 10 ◦ , < 2 cm ) are higher than those of high-error poses ( e.g ., > 20 ◦ , > 4 cm ).

High error

High error

Low error

Low error

Rotation

Translation

900

6

800

4

Energy

Energy

700

2

600

0

500

400

0

5

10

15

20

25

30

35

40

45

0

1

2

3

4

5

6

7

8

9

Degree

bottle (symmetric)

SO(3) distribution

laptop (asymmetric)

SO(3) distribution

bottle

laptop

60°

30°

。0

-30°

-90°

0

90°

180°

270°

Roll

90°

09-

0°

60°

30°

。0

-30°

90°

180°

270°

Roll

06

-60

0°

4.4 Analysis on Symmetric Objects

Performance. Our approach outperforms the state-of-theart method RBP-Pose [19], especially for symmetrical objects, as shown in Table 4. Importantly, our method does not rely on specific network structures or loss designs for these categories. Notably, for 'mug' category, where asymmetry is present when the handle is visible and symmetry

when it is not, our method effectively addresses this challenge and achieves significant performance improvements compared to previous methods. To visually depict the disparity between symmetric and asymmetric objects, Figure 4 visually demonstrates the accuracy of our method in predicting rotation distributions for both symmetric and asymmetric objects.

Toward Cross-category Object Pose Estimation. Our method demonstrates a surprisingly crosscategory generalizability, as it directly predicts the distribution of object poses, eliminating the reliance on object category prior. Table 5 verifies the remarkable ability of our approach to generalize across categories sharing similar symmetric properties.

Our approach demonstrates consistent and robust performance when tested on unseen categories with similar symmetrical attributes, whereas the baseline performance exhibits a significant decline. We hypothesize that the specific out-ofdistribution (OOD) generalization ability of our method arises from the learned feature space of the point cloud. For example, although the bowl

category is considered OOD, the extracted features from point clouds in the bowl category may exhibit similarities or closeness to seen categories, to some extent. In other words, a bowl may share visual characteristics with items such as cans, bottles, or mugs, as identified by the PointNet++ of the

w/o bowl

w/o bottle

w/o can

bottle

bottle

bottle

bowl

bowl

bowl

camera

camera

camera

can

can

can

laptop

laptop

laptop

bnu

ScoreNet. To validate this hypothesis, we conducted a t-SNE [47] analysis on the point cloud feature space of the ScoreNet. Specifically, we extracted features from the point clouds of objects in the seen and unseen test set and visualized the t-SNE results. As shown in Figure 5, the results demonstrate that features from cans and bottles tend to intermingle, aligning with the accurate observation that both cans and bottles exhibit symmetrical cylindrical shapes. Meanwhile, features from the bowl category show proximity to features from mugs.

4.5 Category-level Object Pose Tracking

Table 6 shows the performance of category-level object pose tracking of our tracking method and baselines on REAL275 datasets. For the first frame, we utilized a perturbed ground truth pose as the initial object pose, following CAPTRA [48] and CATRE [49]. Despite being directly transferred from a single-image prediction method, our approach has achieved performance comparable to the state-of-the-art method, CATRE. Remarkably, we significantly outperform CATRE in the metrics of 5 ◦ 5 cm and mean rotation error. This highlights the expansibility of our pose estimation method for more nuanced downstream tasks. Despite our lower FPS compared to CATRE, our method still adequately fulfills usage requirements, particularly for robot operations.

5 Conclusion and Discussion

In this work, we study a fundamental problem, namely the multi-hypothesis issue , that has existed in category-level object pose estimation for a long time. To overcome this issue, we propose a novel object pose estimation framework that initially generates the pose candidates via a score-based diffusion model. To aggregate the candidates, we leverage an energy-based diffusion model to filter out the outliers, which avoids the costly integration process of the score-based model. Our framework achieves exceptionally state-of-the-art performance on existing benchmarks, especially on symmetric objects. In addition, our framework is capable of object pose tracking and can generalize to objects from novel categories sharing similar symmetric properties.

Limitations and Future Works: Although our framework achieves real-time object pose tracking at approximately 17 FPS, the efficiency of single-frame object pose estimation is still limited by the costly sampling process of the score-based model. In the future, we may leverage the recent advances in accelerating the sampling process of the diffusion models to speed up the inference [52; 53]. In addition to passively filtering out the outliers, we may incorporate reinforcement learning to train an agent that actively increases the likelihood of the estimated pose [54].

Boarder Impact: This work opens the door to leveraging energy-based diffusion models to improve the generation results, which might facilitate more research in border communities.

Acknowledgments and Disclosure of Funding

This work is supported by the National Natural Science Foundation of China - General Program (Project ID: 62376006), National Youth Talent Support Program (Project ID: 8200800081), and Beijing Municipal Science & Technology Commission (Project ID: Z221100003422004).

References

Eric Marchand, Hideaki Uchiyama, and Fabien Spindler. Pose estimation for augmented reality: a hands-on survey. IEEE transactions on visualization and computer graphics , 22(12):26332651, 2015.

Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, and Dieter Fox. Self-supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 3665-3671. IEEE, 2020.

Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birchfield. Deep object pose estimation for semantic robotic grasping of household objects. arXiv preprint arXiv:1809.10790 , 2018.

He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2642-2651, 2019.

Kai Chen and Qi Dou. Sgpa: Structure-guided prior adaptation for category-level 6d object pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2773-2782, 2021.

Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, Linlin Shen, and Ales Leonardis. Fs-net: Fast shape-based network for category-level 6d object pose estimation with decoupled rotation mechanism. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1581-1590, 2021.

Meng Tian, Marcelo H Ang, and Gim Hee Lee. Shape prior deformation for categorical 6d object pose and size estimation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16 , pages 530-546. Springer, 2020.

Jiehong Lin, Zewei Wei, Changxing Ding, and Kui Jia. Category-level 6d object pose and size estimation using self-supervised deep prior deformation networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX , pages 19-34. Springer, 2022.

Jiehong Lin, Zewei Wei, Zhihao Li, Songcen Xu, Kui Jia, and Yuanqing Li. Dualposenet: Category-level 6d object pose and size estimation using dual pose network with refined learning of pose consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3560-3569, 2021.

Haitao Lin, Zichang Liu, Chilam Cheang, Yanwei Fu, Guodong Guo, and Xiangyang Xue. Sar-net: shape alignment and recovery network for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6707-6717, 2022.

Yan Di, Ruida Zhang, Zhiqiang Lou, Fabian Manhardt, Xiangyang Ji, Nassir Navab, and Federico Tombari. Gpv-pose: Category-level object pose estimation via geometry-guided pointwise voting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6781-6791, 2022.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Qiyu Dai, Jiyao Zhang, Qiwei Li, Tianhao Wu, Hao Dong, Ziyuan Liu, Ping Tan, and He Wang. Domain randomization-enhanced depth simulation and restoration for perceiving and grasping specular and transparent objects. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX , pages 374-391. Springer, 2022.

Guanglin Li, Yifeng Li, Zhichao Ye, Qihang Zhang, Tao Kong, Zhaopeng Cui, and Guofeng Zhang. Generative category-level shape and pose estimation with semantic primitives. In Conference on Robot Learning , pages 1390-1400. PMLR, 2023.

Shinji Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence , 13(04):376-380, 1991.

Dengsheng Chen, Jun Li, Zheng Wang, and Kai Xu. Learning canonical shape space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11973-11982, 2020.

Ruida Zhang, Yan Di, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Ssp-pose: Symmetry-aware shape prior deformation for direct category-level object pose estimation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 7452-7459. IEEE, 2022.

Ruida Zhang, Yan Di, Zhiqiang Lou, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Rbp-pose: Residual bounding box projection for category-level pose estimation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part I , pages 655-672. Springer, 2022.

Jiaze Wang, Kai Chen, and Qi Dou. Category-level 6d object pose estimation via cascaded relation and recurrent reconstruction networks. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 4807-4814. IEEE, 2021.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4), 2005.

Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence , pages 574-584. PMLR, 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems , 32, 2019.

Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems , 33:12438-12448, 2020.

Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems , 34:14151428, 2021.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364 , 2022.

Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modeling. arXiv preprint arXiv:2202.02763 , 2022.

Mingdong Wu, Fangwei Zhong, Yulong Xia, and Hao Dong. TarGF: Learning target gradient field for object rearrangement. arXiv preprint arXiv:2209.00853 , 2022.

Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005 , 2021.

Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In European Conference on Computer Vision , pages 364-381. Springer, 2020.

Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gerard Medioni, and Leonid Sigal. Energy-based learning for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13936-13945, 2021.

Shitong Luo and Wei Hu. Score-based point cloud denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4583-4592, 2021.

Ruizhi Shao, Zerong Zheng, Hongwen Zhang, Jingxiang Sun, and Yebin Liu. Diffustereo: High quality human reconstruction via diffusion-based stereo using sparse cameras. arXiv preprint arXiv:2207.08000 , 2022.

Hai Ci, Mingdong Wu, Wentao Zhu, Xiaoxuan Ma, Hao Dong, Fangwei Zhong, and Yizhou Wang. Gfpose: Learning 3d human pose prior with gradient fields. arXiv preprint arXiv:2212.08641 , 2022.

Tim Salimans and Jonathan Ho. Should EBMs model the energy or the score? In Energy Based Models Workshop - ICLR 2021 , 2021.

Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. arXiv preprint arXiv:2302.11552 , 2023.

John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics , 6(1):19-26, 1980.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5745-5753, 2019.

F Landis Markley, Yang Cheng, John L Crassidis, and Yaakov Oshman. Averaging quaternions. Journal of Guidance, Control, and Dynamics , 30(4):1193-1197, 2007.

IY Bar-Itzhack and Yaakov Oshman. Attitude determination from vector observations: Quaternion estimation. IEEE Transactions on Aerospace and Electronic Systems , (1):128-136, 1985.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose: Predicting probabilistic relative rotation for single objects in the wild. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI , pages 592-611. Springer, 2022.

Kieran Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Implicit-pdf: Non-parametric representation of probability distributions on the rotation manifold. arXiv preprint arXiv:2106.05965 , 2021.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Yijia Weng, He Wang, Qiang Zhou, Yuzhe Qin, Yueqi Duan, Qingnan Fan, Baoquan Chen, Hao Su, and Leonidas J Guibas. Captra: Category-level pose tracking for rigid and articulated objects from point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13209-13218, 2021.

Xingyu Liu, Gu Wang, Yi Li, and Xiangyang Ji. Catre: Iterative point clouds alignment for category-level object pose refinement. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II , pages 499-516. Springer, 2022.

Chen Wang, Roberto Martín-Martín, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, and Yuke Zhu. 6-pack: Category-level 6d pose tracker with anchor-based keypoints. In 2020 IEEE International Conference on Robotics and Automation (ICRA) , pages 10059-10066. IEEE, 2020.

Xinke Deng, Junyi Geng, Timothy Bretl, Yu Xiang, and Dieter Fox. icaps: Iterative categorylevel object pose and shape estimation. IEEE Robotics and Automation Letters , 7(2):1784-1791, 2022.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469 , 2023.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 , 2022.

Hai Ci, Mickel Liu, Xuehai Pan, Fangwei Zhong, and Yizhou Wang. Proactive multi-camera collaboration for 3d human pose estimation. arXiv preprint arXiv:2303.03767 , 2023.

A Implementation Details

Architecture of the Score Network The detailed architecture of the score network Φ θ is illustrated in Figure 6. We utilize PointNet++ [39] to extract the global geometry feature F O of the partially observed point cloud O ∗ . And the sampled pose p and timestep t features are embedded as F p and F t , respectively, using a Multi-Layer Perceptron (MLP). Then F O , F p and F t are concatenated to obtain the global feature F , and three parallel branches are employed to predict the scores of R x , R y , and T individually, where [ R x | R y ] ∈ R 6 and T ∈ R 3 denote rotation and translation vectors, respectively. And [ R x | R y ] is a continuous rotation representation proposed by [40] to address the discontinuity of quaternions and Euler angles in Euclidean space. As introduced in [40], the mapping from SO(3) to the 6D representation of rotation is:

The mapping form the 6D representation to SO(3) is:

Here N ( · ) denotes a normalization function.

Architecture of the Energy Network The energy network Ψ ϕ shares exactly the same architecture with the score network Φ θ . The inputs are first fed into Φ ϕ to obtain a score-shaped vector Φ ϕ ( p , t | O ) ∈ R |P| . Then, the output energy is calculated by the dot product between the input pose and the score-shaped vector Ψ ϕ ( p , t | O ) = ⟨ p , Φ ϕ ( p , t | O ) ⟩ ∈ R 1 .

PointNet++

…

MLP

…

Partial Point

Cloud

𝑂

∗

𝑡

MLP

…

1

×

256

1

×

128

1

×

1024

1

×

1408

⨁

FC256

FC3

FC256

FC3

FC256

FC3

𝜕 log 𝑝

"#$#

𝜕𝑅

%

𝒑

𝜕 log 𝑝

"#$#

𝜕𝑅

&

𝜕 log 𝑝

"#$#

𝜕𝑇

ℱ

𝒑

ℱ

𝒕

ℱ

𝑶

ℱ

B Qualitative Comparison on REAL275

Figure 2 illustrates the qualitative comparison results between our method and RBP-Pose [19] on the REAL275 dataset. The images are accompanied by red boxes highlighting objects that exhibit noticeable differences in the predicted results. Additionally, the bottom-right corner of each image provides an enlarged view of the highlighted object, showing the ground truth pose as well as the poses estimated by RBP-Pose and our approach. Our method demonstrates a significant performance improvement compared to RBP-Pose, particularly in the case of objects such as mugs. Notably, in the fourth row of the figure, it can be observed that our method achieves highly accurate poses even when only a small portion of the mug handle is visible. This success can be attributed to the fact that, during the training process, a unique pose exists when the mug handle is visible. However, when the mug handle becomes occluded, a multi-hypothesis problem arises, which our generative formulation effectively handles.

C More Results and Analysis

C.1 Per-category Results

Figure 8 demonstrates a quantitative comparison between our method and the state-of-the-art depthbased approach, RBP-Pose [19], for various object categories at different thresholds. The results

GT

RBP-Pose

Ours

沃

电

clearly indicate that our method outperforms RBP-Pose in all metrics, despite the fact that we do not incorporate augmentation specifically designed for symmetric objects during the training phase, unlike RBP-Pose. Our approach exhibits significant improvements, particularly in regions with stringent threshold requirements. This emphasizes the superior performance of our generative category-level object 6D pose estimation approach in effectively addressing the multi-hypothesis challenges posed by symmetric objects and partial observations, thereby enabling its successful application in robot manipulation tasks demanding precise object pose prediction.( e.g ., pouring liquids.)

C.2 Results on CAMERA

Table 7 illustrates a quantitative comparison between our method and the baselines on the CAMERA [4] dataset. The results clearly demonstrate the remarkable performance enhancement achieved by our method. When compared to approaches that rely solely on depth data as network input, as well as those that utilize RGB-D and shape priors as network input, our method consistently outperforms them, surpassing the current state-of-the-art performance. Notably, our method exhibits a particularly pronounced advantage when stricter accuracy requirements are imposed, such as the 5 ◦ 2 cm metric. In this case, our method outperforms the current SOTA method, RBP-Pose, by an impressive margin of 6.4%. This significant improvement highlights the efficacy of our approach.

RBP-Pose

Ours

Rotation

Translation

Rotation

Translation

100

100

80

80

Average Precision

Average Precision

60

60

bottle

bottle

40

bowl

40

bowl

camera

camera

can

can

20

laptop

20

laptop

mean

mean

Q

0

0

5

10

0

2

4

6

0

5

10

0

2

4

6

Degree

Degree

C.3 Real World Experiments

We have also successfully integrated our approach with robot manipulation capabilities, as demonstrated through various experiments conducted with the UFACTORY xArm6 equipped with RealSense D435. The demonstrations can be found in the supplementary video or on the project website. As shown in Figure 9, we illustrate the following three tasks:

Pouring Task. This task involves transferring the contents ( e.g ., water) from one container to another. The demonstration highlights the potential of combining our approach with heuristic strategies, enabling functional robot operations.

Stacking Task. In this task, we focused on piling up objects of the same category, like organizing scattered bowls on a tabletop. This demonstrates the precision of the estimated object pose, as accurate knowledge of object poses is crucial for completing this task.

Handover Task. This task involved either receiving objects from human hands to perform tasks or passing objects to person. The demonstration exemplified one form of human-robot interaction empowered by our method.

(a) Pouring

(b) Stacking

(c) Handover

D Details of Object Pose Tracking

Our pose estimation framework can be adapted to pose tracking with minor modifications. With the learned score network Φ θ , the tracking algorithm is summarised as follows:

Algorithm 1 Our Object Pose Tracking Framework

1: Initialisation: Score network Φ θ , energy network Ψ ϕ , initial point cloud O 0 , the initial ground truth pose reference p 0 and the tracking horizon T .

2: for t = 1 to T do

3: Receive current observation O t

4: z 1 , z 2 , ... z K ∼ N ( p , 0 . 1 I

t -1 2 ) ▷ Sample initial poses around the previous prediction.

5: Sample candidates { ˆ p i } K i =1 from the initialization ( z 1 , z 2 , ... ) via the following PF-ODE:

6: ˆ p τ 1 ≻ ˆ p τ 2 ... ≻ ˆ p τ K where ˆ p τ i ≻ ˆ p τ j ⇐⇒ Ψ ϕ (ˆ p τ i , ϵ | O ) > Ψ ϕ (ˆ p τ j , ϵ | O ) ▷ 7: Estimate the current pose p t = ( ∑ i M =1 ˆ T τ i M , ˆ q t ) where M = ⌊ δ · K ⌋ and:

Ranking

8: end for

E Ethics Statement and Boarder Impact

Our method has the potential to develop the home-assisting robot, thus contributing to social welfare. We evaluate our method in synthesized or human-collected datasets, which may introduce data bias. However, similar studies also have such general concerns. We do not see any possible major harm in our study.

Peer reviews:
comments: SUMMARY:
This paper proposes a novel approach for generative object pose estimation based on diffusion models.

STRENGTHS:
1. Formulating pose estimation as a diffusion process is novel. It shows excellent ability to solve the multi-hypothesis issues in pose estimation caused by symmetry and partial observation. Moreover, it can be easily extended to solve the object pose tracking problem.

2. This paper proposes an efficient framework to estimate the object pose using diffusion models. It trains a score-based diffusion model to sample pose hypotheses, and an additional energy-based diffusion model to estimate the likehood of each hypothesis.

3. The paper is well written with nice figures, clear equations and proficient writing style. 

4. Experimental results on several benchmarks demonstrate the efficiency of the proposed method.

WEAKNESSES:
In Table-1, this paper conducts comparison with category-level object pose estimation methods that predict 9 DoF object pose, consisting of 3D rotation, 3D translation and 3D size. However, the 3D size is not considered by the proposed method. Though the metrics are focused on rotation and translation, it still could cause a potential unfair comparison.

QUESTIONS:
As a likehood is estimated for each pose hypothesis, maybe an alternative way to get the final estimation is fusing them with a weighting scheme. For example, could it be possible to design a weighted version of Eq-9?

LIMITATIONS:
The limitation of inference speed and future work is included in the last section of the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-23

comments: SUMMARY:
This paper proposes a novel conditional generative approach for category-level pose estimation to solve the multi-hypothesis problem. The proposed method utilize the energy-based diffusion model to aggregate the candidates generated by the score-based diffusion model. Extensive experiments have been conducted to show the effectiveness of the proposed method on the REAL275 dataset under category-level pose estimation/tracking and novel category pose estimation evaluation protocols.

STRENGTHS:
1. The author states the method clearly, and the notation is easy to follow.
2. The author proposed a novel multi-hypothesis method and shows promising results compared to regression and correspondence methods. 
3. The ablation study is convincing.

WEAKNESSES:
1. I wonder if scoreNet and energyNet are needed for each class. Does a single model infer all classes? 
2. I wonder about the inference time of object pose estimation. Is the object pose estimation inference time linearly increased according to the number of hypotheses?
3. Is there anything to consider about symmetries properties when augmentation in the training process?

QUESTIONS:
1. If it is not the model that predicts all classes, but each model, how did you compare the parameters of the model (Table 1)?

LIMITATIONS:
Covered the limitation in the main paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mines and formulates ambiguity in the task of object pose estimation, proposing to use a diffuse generative model to generate multiple hypotheses, which are then aggregated through additional scoring and ranking by another scorer net. The model achieves significant improvements with less supervision and parameters, has good cross-category generalization results, and performs well on tracking and stimulation tasks.\n\nSTRENGTHS:\n- I like the observation of the symmetric ambiguity and generative multi-hypothesis formulation.\n- The improvement has been dramatically boosted, and it can beat all SOTAs even with fewer supervision signals and parameters.\n- Thank the authors for testing the method on many datasets and tasks.\n\nWEAKNESSES:\n- (Conditional generation) Adding objects as conditions should further improve in-distribution test performance, right?\n(Redundancy of model designs) EnergyNet can also be used for sampling, so why is ScoreNet needed?\n- (Model design choices) How about using other models instead of EnergyNet to estimate, e.g., normalizing flows can give exact likelihood? There has been much multi-hypothesis work in human pose estimation/motion generation [a-c]. Essentially, this framework does not explore or mine features for object pose estimation. In other words, this model can also be used in other fields, and models in other fields can also be used in this field. Regarding the problem setup, there seem to be no many special things about monocular estimation except the mentioned symmetric ambiguity; however, the dramatic improvements in straightforwardly applying generative models encourage us to think about the underlying principles.\n- (Mean pooling aggregation) Why must mean pooling to be used for aggregation? It looks like the mode of the largest cluster corresponds to the most likely solution, which is probably better than the mean. Additionally, some other works also study empirical aggregations, e.g., clustering [a], weighting [b] and [c]. Is it because quaternions are not easy to work on? What about converting to other forms (e.g., coordinates)? I expect to see comparisons or authors’ discussions on this.\n- (Less yet better?) Can you elaborate on your speculation for the L265-267? Why does your method require less supervision information and a lighter model yet achieve better results? This guess will be very interesting.\n- (Best hypothesis) I don't quite understand the last line in the table in Tab. 3; why is mean pooling needed since GT is accessible? What did I miss?\n- (Cross-category generalization) I think the authors should emphasize the limitations when claiming generalization gains. I don't particularly understand that even the generative model also has unstable, bad, and unreliable performance for OOD point cloud conditions that have not been seen in the training set. And comparing the baselines in Tab. 5 is unfair; it is better to let them also train and test on the same split. I hope the author could kindly elaborate on this.\n- (Missing related works) Should add related work of multi-hypothesis generation [a-e].\n- (Visualizations of multiple pose hypotheses) While there is a dimensionality reduction visualization in Fig. 3, it is suggested to add the predicted multiple pose hypotheses along with the estimated energies to illustrate the method's effectiveness qualitatively. I also hope that the author can provide visualizations of the results, showing the advantages and failures of your method compared to the baseline under different settings such as D and RGB-D (Sup. Fig. 2 is better to also show input depth and point cloud data). This helps readers better understand the proposed method.\n- (Reproducibility) While this cannot be forced or required, if the author commits to publicizing the code, it will have a greater impact and be more helpful to the field.\n\nReferences:\n- [a] C. Li & G. Lee. Weakly supervised generative network for multiple 3D human pose hypotheses. BMVC’20.\n- [b] B. Biggs et al. 3D multibodies: Fitting sets of plausible 3D models to ambiguous image data. NeurIPS’20.\n- [c] W. Shan et al. Diffusion-based 3D human pose estimation with multi-hypothesis aggregation. CVPR’23.\n- [d] G. Chliveros et al. Robust multi-hypothesis 3D object pose tracking. ICVS’13.\n- [e] F. Michel et al. Global hypothesis generation for 6D object pose estimation. CVPR’17.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYeah.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo settle the multi-hypothesis issue in category-level 6D pose estimation, this paper formulates the focused task as conditional generative modeling and proposes a novel method based on diffusion models, which utilizes a score-based diffusion model to sample pose candidates with an energy-based one followed to rank those candidates. The proposed method achieves the state-of-the-art results on REAL275 dataset for category-level pose estimation, and is further extended for the task of pose tracking.\n\nSTRENGTHS:\n- The authors propose a new perspective for category-level 6D pose estimation by formulating the task as conditional generative modeling and realizing it via diffusion models.\n\n- The proposed method achieves the state-of-the-art results on REAL275 dataset for both tasks of category-level pose estimation and pose tracking. Cross-category experiments and real-world applications are also conducted to verify its generalization ability.\n\n- The paper is well written and presented with detailed illustrations and thorough experiments.\n\nWEAKNESSES:\n- The proposed method is not competitive in terms of inference time with the use of generative models.\n\n- The original definition of category-level task includes the estimation of object sizes, which could not be learned in the proposed method.\n\n- Some methods are not compared with in Table 1 for pose estimation, e.g., [1][2][3], and Table 6 for pose tracking, e.g., [1][4][5].\n\nReference:\n\n[1] Sparse Steerable Convolutions: An Efficient Learning of SE(3)-equivariant Features for Estimation and Tracking of Object Poses in 3d Space.\n\n[2] CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation.\n\n[3] ShAPO: Implicit Representations for Multi-Object Shape, Appearance and Pose Optimization.\n\n[4] ICK-Track: A Category-Level 6-DoF Pose Tracker Using Inter-Frame Consistent Keypoints for Aerial Manipulation.\n\n[5] BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models.\n\nQUESTIONS:\n- What’s the special designs of the proposed method for category-level pose estimation, compared to instance-level pose estimation? Could it be applied to the instance-level task?\n\n- Some typos.\n    - ’w‘ in Eq. (1) and $\\dot{\\sigma}$ in Eq. (4) are not explained.\n    - In Line167, should '$log p_t(p|O)$' be '$\\Delta_p log p_t(p|O)$'?\n    - In Line213, \"symmetric objects ??\" -> \"symmetric objects\".\n    - In Line278,281,284, should all the 'M' be K\"?\n    - In References, paper [12] and paper [13] refer to the same papers.\n\nLIMITATIONS:\nThe authors have discussed the limitations of their work in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a diffusion-based model for category-level object pose estimation. Different from the existing deterministic approaches which treat the object pose estimation as a regression problem, the proposed diffusion model alternatively formulated it as a generation problem. In this way, it could tackle the multiple pose hypotheses issue comes from symmetric objects and partial observation.  The proposed approach achieves state-of-the-art performance on existing benchmarks and could generalize to unseen symmetric categories.\n\nSTRENGTHS:\na. The paper introduces a novel diffusion-based framework for category-level pose estimation. The framework consists of two diffusion models. The first diffusion model generates a set of pose candidates for a given point cloud during the inference stage. The second diffusion model uses an energy-based approach to rank the candidates and filter out those with low rankings. This paper claims to be the first to propose a solution for pose estimation using diffusion models.\n\nb. The diffusion-based model presented in the paper demonstrates state-of-the-art performance on existing benchmarks. It surpasses the performance of previous approaches by a significant margin. Additionally, the authors argue that their framework has the potential to generalize to objects of unseen categories that have not been studied by other approaches.\n\nWEAKNESSES:\na. The chosen representation of the pose parameter in this paper consists of a 9-dimensional vector, where 6 dimensions are allocated for rotation and 3 dimensions for translation. Notably, the omission of the scale parameter from this representation raises an important inquiry regarding the reasons behind its exclusion and the means by which it can be obtained during inference. Further clarification is required to address these concerns.\n\nb. The current approach employed by the authors involves utilizing mean pooling to derive the final pose estimation from the filtered pose candidates. Given that the energy-based diffusion model is capable of estimating the data likelihood for each pose candidate, an alternative consideration arises: would adopting weighted averaging be a more suitable approach for calculating the final pose estimation? It would be insightful to explore the potential benefits of incorporating weighted averaging as a means to enhance the accuracy of the pose estimation.\n\nc. Additional information clarifying the specifics of the pose tracking implementation would be appreciated. Considering that the framework incorporates two diffusion models, it appears intriguing for me to comprehend how the pose tracking component can maintain such computational efficiency and deliver results in real-time.\n\nd. Some typos. (Line 213)\n\nQUESTIONS:\nSome questions are listed in the weakness section.\n\nLIMITATIONS:\nThe authors have discussed the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}