Generative Category-level Object Pose Estimation via Diffusion Models
Authors: Jiyao Zhang
Mingdong Wu
Hao Dong
Venue: NeurIPS 2023 poster (2023)
Keywords: Category-Level Object Pose Estimation
Diffusion Model
TL;DR: To address the multi-hypothesis issue, We propose a diffusion-based framework for category-level object pose estimation.

Abstract: Object pose estimation plays a vital role in embodied AI and computer vision, enabling intelligent agents to comprehend and interact with their surroundings. Despite the practicality of category-level pose estimation, current approaches encounter challenges with partially observed point clouds, known as the multihypothesis issue. In this study, we propose a novel solution by reframing categorylevel object pose estimation as conditional generative modeling, departing from traditional point-to-point regression. Leveraging score-based diffusion models, we estimate object poses by sampling candidates from the diffusion model and aggregating them through a two-step process: filtering out outliers via likelihood estimation and subsequently mean-pooling the remaining candidates. To avoid the costly integration process when estimating the likelihood, we introduce an alternative method that distils an energy-based model from the original score-based model, enabling end-to-end likelihood estimation. Our approach achieves state-of-the-art performance on the REAL275 dataset, surpassing 50% and 60% on strict 5 ◦ 2cm and 5 ◦ 5cm metrics, respectively. Furthermore, our method demonstrates strong generalization to novel categories without the need for fine-tuning and can readily adapt to object pose tracking tasks, yielding comparable results to the current state-of-the-art baselines. Our checkpoints and demonstrations can be found at https://sites.google.com/view/genpose.

GenPose: Generative Category-level Object Pose Estimation via Diffusion Models
Jiyao Zhang 12.3*, Mingdong Wu! Hao Dong 1.3t
Center on Frontiers of Computing Studies, School of Computer Science, Peking University Beijing Academy of Artificial Intelligence National Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University
jiyaozhangOstu.pae pku. edu. en, [wmingd, hao dao.dong))oe @pku. edu. cn
Abstract
Object pose estimation plays a vital role in embodied Al and computer vision, enabling intelligent agents comprehend and interact with their surroundings. Despite the practicality of category-level pose estimation, current approaches encounter challenges with partially observed point clouds, known as the multi- hypothesis issue. In this study, we propose a novel solution by reframing category- level object pose estimation conditional generative modeling, departing from traditional point-to-point regression. Leveraging score-based diffusion models, we estimate object poses by sampling candidates from the diffusion model and aggregating them through a two-step process: filtering out outliers via likelihood estimation and subsequently mean-pooling the remaining candidates. To avoid the costly integration process when estimating the likelihood, we introduce an alternative method that trains an energy-basee based model from the original score- based model, enabling end-to-end likelihood estimation. Our approach achieves state-of-the-art performance on the REAL275 dataset and demonstrates promising generalizability to novel categories sharing similar symmetric properties without fine-tuning. Furthermore, can readily adapt to object pose tracking tasks, yielding comparable results to the current state-of-the-ar baselines. Our checkpoints and demonstrations can be found at https: /topps:/ssite /sitess.... google. www.oooooooooooooooo com/view/ genpose.
Introduction
Object pose estimation is fundamental problem in the domains of robotics and machine vision, pro- viding a high-level representation of the surround- ing world. plays crucial role in enabling intel- ligent agents to understand and interact with their environments, supporting tasks such as robotic manipulation, augmented reality, and virtual real- ity [1; 2: 3]. the existing literature, category- level object pose estimation [4; 5; 6] has emerged as more practical approach compared to instance- level pose estimation since the former eliminates the need for 3D CAD model for each individual instance. The main challenge of this task is to cap- ture the general properties while accommodating the intra-class variations, i.e., variations among different instances within category [7; 5: 8).
views
viewn
view
vtew
vlewa
view;
view
viewa
Symmetric
(b) Partial Observation
Figure I: Multi-hypothesis issue in category- level object pose estimation comes from (a) sym- metric objects and (b) partial observation.
equal contribution, f: corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

To address this task, previous studies have explored regression-based approaches, These approaches involve either recovering the object pose by predicting correspondences between the observed point cloud and synthesized canonical coordinates [4; 7]. or estimating the pose in an end-to-end manner [9; 6]. The former approaches leverage category prior information, such as the mean point cloud, to handle intra-class variations. As result, they have demonstrated strong performance across various metrics. On the other hand, the latter approaches do not rely on the coordinate mapping process, which inherently makes them faster and amenable to end-to-end training.
Despite the promising results of previous approaches. a fundamental issue inherent to regression- based training persists. This issue. known as the multi-hypotheess problem, arises when dealing with a partially observed point cloud. In such cases:
Multiple feasible pose hypotheses can exist, but the network can only be supervised for single pose due to the regression-based training.
Figure 1(a) illustrates this problem by showcasing the feasible ground truth poses of symmetric objects (e.g., bowls). Moreover, partially observed point clouds of an object may appear similar from certain views (e.g.. mug with obstructed handle may look the same from certain views, as shown in Figure 1(b)), further exacerbating the multiple hypothesis issue. Previous studies have proposed ad-hoc solutions tackle this issue, such as designing special network architectures [10; 11] or augmenting the ground truth poses [4; 5] for symmetric objects. However, these approaches cannot fundamentally resolve this issue due the lack of generality. Consequently, an ideal object pose estimator should model a pose distribution instead of regressing a single pose when presented with a partially observed point cloud.
To this end, we formulate category-level object pose estimation as conditional generative modeling that inherently models the distribution of multiple pose hypotheses conditioned partially observed point cloud. To model the conditional pose distribution, we employ score-basee diffusion models, which have demonstrated promising results in various conditional generation tasks [12]. Specifically, we estimate the score functions, i.e., the gradients of the log density, of the conditional pose dis- tribution perturbed by different noise levels. These trained gradient fields can provide guidance to iteratively refine a pose, making it more compatible with the observed point cloud. In the testing phase, we estimate the object pose of a partial point cloud by sampling from the conditional pose distribution using an MCMC process incorporated with the learned gradient fields. As a result, our method can propose multiple pose hypotheses for a partial point cloud, thanks to stochastic sampling.
However, it is essential to note that a sampled pose might be an outlier of the conditional pose distribution, meaning it has low likelihood, which can severely hurt the performance. To address this issue, we sample group of pose candidates and then filter out the outliers based on the estimated likelihoods. Unfortunately, estimating the likelihoods from score-based models requires highly time-consuming integration process [13], rendering this approach impractical. To overcome this challenge, we propose an alternative solution that trains an energy-based model from the original score-based model for likelihood estimation. Following training. the energy network is guaranteed to estimate the log- likeliieldoood of the original data distribution up a constant. By ranking the candidates according to the energy network's 's output, we can filter out candidates with low ranks (e.g.. the last 40%). Finally, the output pose is aggregated by mean-pooling the remaining pose candidates.
Our experiments demonstrate several superiorities over the existing methods. Firstly, without any ad-hoc network or loss design for symmetry objects, our method achieves tute-of-theeart performance and surpasses 50% and 60% on the strict 5 and 5 5cm metrics, respectively, on the REAL275 dataset. Besides, our method can directly generalize novel categories without any fine-tuning to some degree due to its prior-free nature. Our methods can be easily adapted to the object pose tracking task with few modifications and achieve comparable performance with the SOTA baselines.
Our contributions are summarized as follows:
We study a fundamental problem, namely multi- huptti-hypothesis issue, that has existed in category- object pose estimation for a long time and introduce a generative approach to address this issue. We propose a novel framework that leverages the energy- based diffusion model to aggregate the candidates generated by the score-based diffusion model for object pose estimation.
Our framework achieves exceptionally state-of-the-an performance on existing benchmarks, especially on symmetric objects. In addition, our framework is capable of object pose tracking and can generalize to objects from novel categories sharing similar symmetric properties.

---

2 Related Works
2.1 Category Level Object Pose Estimation
Category-Ievel object pose estimation [4; 14; 15] aims to predict pose of various instances within the same object category without requiring CAD models. To this end. NOCS [4] introduce normalized object coordinate space for each object category and predicts the corresponding object shape in canonical space. Object pose is then calculated by pose fitting using the Umeyama [16] algorithm. CASS [17]. DualPoseNet [9], and SSP-Pose [18] end-to-end regress the pose and simultaneously reconstruct the object shape in canonical space to improve the precision of pose regression. SAR- Net [10] and GPV-Pose [11] specifically design symmetric-aware reconstruction to enhance the performance of predicting the translation and size of symmetrical objects while regressing the object pose. However, direct regression of pose or object coordinates in canonical space fail when there is considerable intra-class variation. To overcome this, FS-Net [6] introduces an online box-cage-basee 3D deformation augmentation method, and RBP-Pose [19] proposes a nonlinear data augmentation method. On the other hand, SPD [7], SGPA [5]. CR-Net [20], and DPDN [8] introduce category-level object priors (e.g., mean shape) and learn how to deform the prior to get the object coordinates in canonical space. These methods have demonstrated strong performance. Overall, the existing category- level object pose estimation methods are regression- based. However, considering symmetric objects (e.g., bowls, bottles, and cans) and partially observed objects (e.g., mugs without visible handles), there are multiple plausible hypotheses for object pose. Therefore, the problem of object pose estimation should be formulated as generation problem rather than regression problem.
2.2 Score-based Generative Models
In the realm of estimating the gradient of the log- ligellidddood pertaining to specified data distribution, pioneering approach known the score-based generative model [21; 22: 23; 24; 25; 13; 26] was originally introduced by [22]. In an effort to provide a feasible substitute objective for score- matching, the denoising score-matching (DSM) technique(21] has further put forward viable proposition. To enhance the scalability of the score-based generative model, [23] has introduced sliced score-matching objective, which involves projecting the scores onto random vectors prior to their comparison. They have also presented annealed training for denoising score matching(24) and have introduced improved training techniques to complement these methods [25]. Additionally, they have extended the discrete levels of annealed score matching to a continuous diffusion process, thereby demonstrating promising results in the domain of image generation [13]. Recent studies have delved further into exploring the design choices of the diffusion process [27], maximum likelihood training [26], and deployment on the Riemann manifold [28]. These advancements have showcased encouraging outcomes when applying score-based based generative models to high high-dimensional domains, thus fostering their broad utilization across various fields, such as object rearrangement [29], medical imaging [30], point cloud generation [31], scene graph generation [32], point cloud denoising [33], depth completion [34], and human pose estimation [35]. These studies have formulated perception- related problems either conditional generative modeling or in-painting tasks, thereby harnessing the power of score-based generative models to tackle these challenges. In contrast to these approaches, we incorporate the score-based diffusion model with an additional energy-based diffusion model [36; 37] for filtering out the outliers so as to improve the performance by aggregating the remaining candidates.
3 Method
Task Description: In this work, we aim to estimate 6D object pose from the partially observed point cloud. The learning agent is given training set with paired object poses and point clouds D = {(p,. O,) where P; E SE(3) and O E gx denote a 6D pose and a partially observed 3D point cloud with N points respectively. Given unseen point cloud O+, the goal is to recover the corresponding ground-trunt pose p".
Overview: We formulate the object pose estimation as conditional generative modeling problem and train a score-based diffusion model & using the dataset D. Further, an energy-based diffusion model V & is trained from the score-based model d for likelihood estimation. During test time, given an unseen point cloud O*, we first sample a group of pose candidates (p). P2.... ) via the score-based diffusion model b.. Then we sort the candidates (p, )KK in descending order according
3

---

(
Top B R
matching
ouiliers
P
P2
P1
P4
Candidates
Partial Point Cloud
Reverse ODE Process
ScoreNet
Ranking by Pase Energies
(a) Candidates Generation
Moan Pooling
matching
Partial Point
Cloud O
W
V4
4
Pose Candidates (Peli
EnergyNet
Pose Energies
Examples
(b) Likelihood Estimation
Aggregation
(1) Training
(II) Inference
Figure 2:
Overview. (I) A score- based diffusion model D and an energy- diffusion model V à is trained via denoising score-matching. (II) a) We first generate pose candidates {p, 1 from the score-based model and then b) compute the pose energies &e c(O") for candidates via the energy-based model. e) Finally, rank the candidates with the energies and then filter out low- ranking candidates. The remaining candidates are aggregated into the final output by mean-pooling.
to the energy outputs and filter out the last 8% candidates (e.g.. s, 60%). Finally, we aggregate the remaining candidates by mean-pooling obtain the estimated pose p.
3.1 Sampling Pose Candidates via Score-based Diffusion Model
To address the multi-hypothesis issue, an ideal pose estimator should be capable of being trained on multiple feasible ground truth poses given the same point cloud. Also, the ideal pose estimator should be able to output all possible pose hypotheses of the given point cloud during inference.
To this end, we propose to tackle object pose estimation conditional generative modeling paradigm. We assume dataset D is sampled from an implicit joint distribution D {(p,, O₃) Paata (p, O)}- We aim to model the conditional pose distribution Ptaa(pp0) during training, and sample pose hypotheses of unseen point cloud O from Pdaa (p (r) during test-time.
Specifically, we employ score-based diffusion model [24; 23; 13] to estimate the conditional distribution Paata (p)O). We adopt Variance-Explodin (VE) Stochastic Differential Equation (SDE) proposed by [13] to construct continuous diffusion process {p(t)) indexed by a time variable t E [0, 1J where p(0) Pdaa (plO) denotes the ground truth pose of the point cloud O. As the / increases from 0 to 1, the time-indexed pose variable p(t) is perturbed by the following SDE:
(t) di
dw, (1) = miin T
dp
Omiin
where Gmin = 0.01 and mmax = 50 are hyper-parameters
During training, we aim to estimate the score function of the perturbed conditional pose distribution Vp log (p(O) of i, where the pt (p)O) denotes the marginal distribution of p(t):
(.(p(t)O)) = / N (V(((t)()))))))))))(<))) (())))))(()).))))) (t)I) Po (o(p()))) dp(0)
(2)
Notably, when L = 0, po(p()) = Para (p(0)0) is exactly the data distribution.
Thanks to the Denoising Score Matching (DSM) [21], can obtain guaranteed estimation of V ppP: (p(O) by training score network b TR x R x DaxN - R!Pl via the following objective:
A(O)
p(0) p())
((0) E.-((E)))
P(0) (P00)11 P(O) (pt=A
$ Lo
(1)2
(3)
where is a hyper-parameter that denotes the minimal noise level. When minimizes the objective in Eq. 3, the optimal score network satisfies Le(p, OO) Po (plO)) according to [21].

---

After training, we can approximately sample pose candidates {p.] from Puata (p(a())) 0) by sampling
from p. (plO), as lim,-o Pe (p) Paum (plu(pple O). To sample from p. (p O). we can solve the following Probability Flow (PF) ODE [13] where 11 ~ (0, o max J). from / 1 to =
dp -o (O)(()V
(4)
dt
where the score function log Pr (p) Q) is empirically approximated by the estimated score network Lo (p, ()O) and the ODE trajectory is solved by RK45 ODE solver [38].
In practice, the score network Lp #/O) is implemented without any ad-hoc design for the symmetric objects: The point cloud O is encoded into global feature by PointNet++ [39], the input pose p is encoded by feed-forward MLPs and the time variable is encoded by a commonly used projection layer following [13]. The pose p is represented as a 9-D variable [R]T], where R and T E R denote rotation and translation vectors, respectively. Due to the discontinuity of quaternions and Euler angles in Euclidean space, we employ the continuous 6-D rotation representation [R., RR,)
following[6; 6; 40J. We defer full details into Appendix A.
3.2 Aggregating Pose Candidates via Energy-basel Diffusion Model
Though we can sample pose candidates {p, [ from the conditional pose distribution Paata (p O) via the learned score model 4p, we still need to determine final output estimation ğ E (SE(3). In other words, the pose candidates need to be aggregated into single estimation.
An initial approach to aggregating candidates is mean pooling. However, when sampling candidates from Pdata (pl O). there is no guarantee that the sampled candidates will have a high likelihood. This means that outlier poses in low-density regions are still likely to be included in the wean-ppoled pose, which can negatively impact its performance. Therefore, we propose estimating the data likelihoods (Pamu (P; OO) to rank the candidates and then filter out low-ranking candidates.
Unfortunately, estimating likelihood via the score model itself requires time-consuming integration process [13], which is impractical for real-time applications:
log Paata (pl 0) se lo log pi (p(1)0)) [. d()=(() V Do(p(1),(0dd (5)
di
where (p(())effo,) is the same forward diffusion process in Eq. 1.
To this end, we propose to train energy-based model V. RI x R x 3 + I that enables an end-to-end surrogate estimation of the data likelihood by supervising the amergy-induced gradient Vp (p, with the denoising score-matching objective in Eq. 3:
((0)= E,-(((.))) { ((t)EE
p(0) p(t) 22] |1
p(0)~pa(plo0 p(t)-rgs (O)
V pCO Po(p(t))
ot))
(6)
In this way, the optimal energy model holds Vp V 1|0) Vp log (p)O), or equivalently, (p, t|O)= log Pt 0) + C where C' is a constant. Although the optimal energy model and the ground truth likelihood differ by constant C, this does not prevent it from functioning as good surrogate likelihood estimator for ranking the candidates:
V (P..
T (', = logp((,;1O) log logp<(p,10)
(7)
Nevertheless, training an energy-based diffusion model from Eq. 6 is known to be difficult and time-inefficient due to the need to calculate the second-order derivations in the objective function.
Following [36], parameterize the energy-based model V (p, (10)
(pp to
alleviate the training burden. We defer the full training details to Appendix A.
With the trained energy model, we sort the candidates into sequence P,, Pr.... > PTX where:
P.
Pr
V Volo,, (0) V Volp.,, (())
(8)
Subsequently, we filter out the last - 5% candidates and obtain P,, > Pr.... > Prxe where 8 E (0, is a hyper parameter and M [6- K].
5

---

Finally, we aggregate the remaining candidates (pr, = (Fr.. Rr,) by averaging the rotations {Rr, I. and the translations {T,, k respectively, to obtain the output pose (T. R). In specific, the translations are pooled by vanilla averaging T Σ T. To obtain the averaged M
rotation, we initially translate the rotations into quaternions { J. Following [41], the average quaternion can then be found by the following maximization procedure:
Di MM
A(Â..) M
arg max q
9ESO(3) T
9, A(q-)= q..
ΣNe Ala..) corresponding
By definition, the solution of Eq. 9 is the eigenvector of the 4 x 4 matrix to the maximum eigenvalue, which can be efficiently solved by QUEST algorithm [42].
3.3 Discussion
Despite addressing the multi-hypothesis issue, our method offers several additional advantages:
No Ad-hoc Design: Unlike previous approaches, we do not incorporate any ad-hoc designs or tricks into the network architecture for symmetric objects. Both the score and energy models are implemented using commonly used feature extractors (e.g.. PointNet++((3)) and feed-forward MLPs. Surprisingly, our method performs exceptionally well in handling symmetric objects (refer to Sec 4.4) and achieves state-of-the-art (SOTA) performance existing benchmarks (refer to Sec 4.2).
Prior-Frees Our method eliminates the requirement of category-level canonical prior, freeing us from designing a -efoommeooontooon module to incorporate the canonical prior. This also provides the potential to generalize our method to objects from unseen categories sharing similar symmetric properties. In Sec 4.4, we present experiments to demonstrate this potential.
Thanks to the closed-loop generation process of the diffusion models, we can adapt the single-frame pose estimation framework to pose tracking tasks by warm-starting from the previous predictions. The adapted object pose tracking framework differs from the single-frame estimation framework only in the candidates' sampling. For each frame that receives the point cloud observation Ocur. we initialize the PF-ODE in Eq.4 with p(0.1) ~ pre. (0.1))),, where Pprev E SE(3) denotes the estimated pose of the previous frame. Subsequently, we solve this modified PF-ODE from t = 0.1 to
Capable of Pose Tracking:
using the gradient fields o (uuur) to sample candidates for pose tracking. By following the same aggregation procedure, we can obtain the pose estimation Peur for the current frame. We summarise the tracking framework in Appendix pose-tracking method outperforms the ofte-oof-hhe-rt object pose-tracking baseline in most metrics.
Sec.4.5, our adapted
Experiments
4.1 Experimental Setup
Datasets. Our method trained and evaluated on two common category-level object pose estimation datasets, namely CAMERA and REAL275 [4], following the NOCS [4] convention for splitting the data into training and testing sets. These datasets include daily objects: bottle, bowl, camera, can, laptop, and mug. CAMERA dataset is a synthetic dataset generated using mixed-reality methods. It comprises real background images with synthetic rendered foreground objects and consists of 275K training images and 25K test images. REAL275 dataset is real-world datasct that employs the same object categories as the CAMERA dataset. It includes 3 unique instances per category in both the training and test sets and consists of 7 scenes for training with 4.3K images and scenes for testing with 2.75K images. Each scene in the REAL275 dataset contains more than 5 objects.
Metrics. Following NOCS [4], we report the mean Average Precision (mAP) in n² and m cm to evaluate the accuracy object pose estimation. Here, n and m denote the prediction error of rotation and translation, respectively, where the predicted rotation error is less than n' and the predicted translation error less than m cm. Specifically, we use 5:2m, 5 5em, 1092cm, and 1095cm as our evaluation metries. Similar to NOCS, for symmetric objects (bottles, bowls, and cans), we ignore the

---

Table Quantitative comparison of category-level object pose estimation on REAL275 dataset. We summarize the results reported in the original paper for the bascline method. 1 represents a higher value indicating better performance, while represents lower value indicating better performance. Data refers to the format of the input data used by the method, and Prior indicates whether the method requires category prior information. '.' indicates that the metrics are not reported in the original paper. K represents the number of hypotheses.
Method
Data
Prior
5"2cm
555mm
10"2emf
1055emt
Parameters(M1)
NOCS [4]
RGB-D
9.5
13.8
26.7
CASS [17]
RGB-D
23.5
50.8
47.2
DualPoseNet [9]
RGB-D
29.3
35.9
50.0
66.8
67.9
SPD [7]
RGB-D
19.3
21.4
43.2
54.1
18.3
CR-Net [20]
RGB-D
27.8
34.3
47.2
60.8
SGPA [5]
RGB-D
35.9
39.6
61.3
70.7
Deterministic
DPDN [8]
RGB-D
46.0
50.7
70.4
78.4
FS-Net [6]
D
19.9
33.9
69.1
41.2
GPV-Pose [11]
DDD
DUDD
32.0
42.9
55.0
73.3
SAR-Net [10]
DDDD
31.6
42.3
50.4
68.3
6.3
SSP-Posc [18]
34.7
44.6
77.8
RBP-Pose [19]
D
38.2
48.1
63.1
79.2
Ours
DI
D
52.1
60.9
72.4
84.0
4.4
Prohabilistic
Our((K=10)
71.5
75.9
85.2
90.8
2.2
Our((K{50)
D
82.0
84.5
92.8
95.9
2.2
rotation error around the symmetry axis. For the "mug" category, we consider it a symmetric object when the handle is not visible and non-symmetric object when the handle is visible.
Implementation Details. For a fair comparison, we employed MaskRCNN [43] during the inference phase, which was identical to the one used in previous work. Both the Energy-based diffusion model for pose ranking and the score-based di diffusion model for pose generation shared the same network structure. The input pointcloud consisted of 1024 points. During the training phase, we used the same data augmentation techniques as FS-Net [6]. which are widely adopted in category-level object pose estimation tasks. All experiments were conducted single RTX3090 with a batch size of 192. All the experiments are implemented using Py PyTorch [44].
instance mask generated by
4.2 Comparison with State-of-the-Art Methods
Table provides a comprehensive comparison between our method and the state-of-the-art approaches the REAL2275 [4] dataset, highlighting significant advancement in performance. In Table 1, "Ours" represents the aggregated pose using as 50 and 8 as 60%. "Ours( K==0)" and "Ours( K K=50)" indicate that set K as 10 and 50, respectively, and select pose from candidates with the minimum distance to ground truth pose for evaluation. We deal with all categories by single model.
As shown in Table I, approach demonstrates remarkable improvement over the current SOTA method, GPV-Pose[L1], exceeding by more than 20% both the rigorous 5 and 5cm evaluation metrics. Significantly, for the first time, achieve results the REAL275 dataset, surpassing the impressive thresholds of 50% and 60% in the 5 2cm and 5 5cm metrics, respectively. These exceptional outcomes further support the efficacy of our approach.
Even when compared to methods that take RGB-D data and category prior as input, our method maintains notable advantage. Without relying any additional information, our method achieves performance boost of over 10% in the metric 5e5m, surpassing the SOTA method, DPDN [8].
In addition, the results that evaluate the pose nearest to the ground truth indicate that condition generative models exhibit tremendous potential for category-level object pose estimation. Another crucial metric is the number of learnable parameters of the network, which significantly impacts the feasibility of model deployment. Table reveals that our method achieves a remarkable accuracy for predicting poses with the fewest network parameters, further improving its deployability.

---

4.3 Ablation Studies
Number of Pose Candidates and Proportion of Selected Poses. Using the 20 metric, Table demonstrates the impact of two factors on performance: the number of generated pose candidates M, and the proportion ó of selected poses out of the total pose candidates.
The results show a notable improvement when M increases from 10 to 50. This enhancement can be attributed to the increasing number of sampling times, resulting in pose candidate set that aligns more closely with the predicted distribution. However, the improvement becomes marginal when M is further increased from 50 to 100. This can be attributed to the predictions that approach the upper limit of the aggregation approach. After considering the trade-off between performance and overhead, we ultimately decided adopt K 50. In view of the inherent challenge in training an energy model that precisely aligns with the actual distribution, it becomes apparent that employing small delta value, as demonstrated in Table 2, does not yield an optimal outcome. Consequently, we regard detector for outliers. The most successful results of this paper
Table 2: Ablation on number of pose candi-
dates K and proportion of selected poses 8.
5
20%
40%
60%
80%
100%
K
10
65.3
67.7
67.9
67.7
65.0
50
70.2
71.8
72.4
71.9
69.7
100
70.8
72.2
72.6
72.5
70.2
the energy model primarily were achieved using a delta value of 60%, which effectively mitigated the presence of relatively insignificant noise originating from the sampling process.
Effectiveness of the Energy-based Likelihood Estimator L In this part, we report the results of three distinct ranking methodologies, namely "Random". "Energy", and "GT", employed to sort pose candidates [p, "Random" applies stochastic shuffling to {p; 1 providing baseline approach. "Energy" uses trained energy model to rank (f, Jk "GT" ranks (f.) J in ascending order of the distances to the ground truth pose, serving as an upper limit for all ranking methods.
The absence of mean pooling in Table 3 implics that one pose candidate is chosen ran- domly for evaluation. We ob- serve that utilizing energy- based likelihood estimator as GT(upper bound)
Table 3: Effectiveness of the energy-based likelihood estimator V
Ranking
Mean pooling
5
5
10m2cm
10"5cm
Random
13.5
49.1
21.5
76.3
Random
49.4
58.6
68.5
80.7
Energy
52.1
60.9
72.4
84.0
62.1
66.8
80.9
86.9
sampler enhances the sort-
ing of pose candidates and effectively eliminates outliers. This improvement is significant compared random sampling. Although doesn't t reach the upper bound, we offer a novel and practical approach for ranking and aggregating poses sampled from estimated distribution and demonstrate its potential.
To better understand the energy model's impact, we further investigated the correlation between pose error and energy output. As depicted in Figure 3, there a general negative correlation between the output and the error of the energy model. However, the energy model excels distinguishing poses with significant error differences but performs poorly when distinguishing poses with low errors. For instance, the energy model assigns similar values (e.g., translation, <2cm) or even incorrect values (e.g., rotation, <10") for poses with low errors. Additionally, the energies associated with low-error poses (e.g., <10?, <2em) are higher than those of high-error poses (e.g., 200% Acm).
Rotation
Translation
800
Energy
Energy
700
600
High error
Low error
High
Degree
Centimeter
Figure 3: Energy-Error Correlation Analysis. Points for raw scatters plot, curves for mean error.

---

bottle (symmetric)
SO(3) distribution
laptop
bottle
90"
1807
WF
laplop (asymmetric)
SO(3) distribution
The predicted conditional rotation distribution of symmetric and asymmetric objects. The left figure an example from the REAL275 dataset, while the right figure shows the distribution of the generated rotations for the symmetrical bottle and the asymmetric laptop. The rotation distribution visualized by plotting yaw as latitude, pitch as longitude, and roll as the color, following the approach inspired by [45; 46]. The ground truth is represented by the center of the open circle, and the dot shows the result of generating rotations 50 times.
Figure 4:
4.4 Analysis Symmetric Objects
Performance. Our approach outperforms the ffut-oof-tee art method RBP-Pose [19], es- pecially for symmetrical ob- jects, as shown in Table 4. Importantly, our method does not rely on specific network structures loss designs for these categories. Notably, for "mug' category, where asym- metry is present when the han- dle is visible and symmetry when it is not, our method effectively addresses this challenge and achieves significant performance improvements compared to previous methods. To visually depict the disparity between symmetric and asymmetric objects, Figure 4 visually demonstrates the accuracy of our method in predicting rotation distributions for both symmetric and asymmetric objects.
Table 4: Per-category results of our method and RBP-Pose.
5c2
5 5"5em
Category
Symmetric
RBP-Pose
Ours
RBP-Pose
Ours
camera
1.3
2,9
1.5
3.2
laptop
41.3
63.4
75.2
91.4
average
21.3
32.2(11.91)
38.4
47.3(k9:)
bottle
38.7
52.6
43.5
60.9
bowl
75.4
85.4
81.7
92.6
can
53.5
72.5
67.1
80.4
average
55.9
70.2(143r)
64.1
78.0(1399)
18.9
35.7(1681)
19.4
36.4(170:)
Toward Cross-category Object Pose Estimation. Our method demonstrates a surprisingly cross- category generalizability, it directly predicts the distribution of object poses, eliminating the reliance on object category prior. Table verifies the remarkable ability of our approach to generalize across categories sharing similar symmetric properties.
Our approach demonstrates consistent and robust perfor- mance when tested on un- seen categories with sim- ilar symmetrical attributes, whereas the baseline per- formance exhibits a signifi- cant decline. We hypothe- size that the specific out-of- distribution (OOD) general- ization ability of our method arises from the learned feature space of the point cloud. For bowl
Table 5: Cross category evaluation on REAL275. The left and right sides of '/" respectively indicate the performance when the testing category is seen and unseen in training data.
Category
Method
5"2cm
5'5cm
10"2cm
10
SAR-Ne:[10]
58.1/36.4
66.0/77.3
83.7/59.4
93.6/81.5
bowl
RBP-Pose[19]
75.4/0.0
81.7/6.9
92.1/0.1
100.0/30.7
Ours
85.4/64.5
92.6/72.5
93.1/87.2
100.0/98.6
SAR-Ne:[00
43.5/11.7
54.0/23.0
61.3/33.6
79.8/68.0
bottle
RBP-Pose[199
38.7/4.3
43.5/5.8
76.4/24.7
89,8/ 99.8/29.7
Ours
52.6/39.0
60.9/53.2
81.4/73.6
92.7/94.6
SAR-Ne:
32.2/7.3
62.2/52.3
52.5/12.1
92.9/87.9
RBP-Pose(19)
53.5/0.8
67.1/210
78.8/2.6
96.3/61.7
Ours
72.5/62.5
80.4/74.0
88.8/81.6
99.8/99.7
example, although category is considered OOD, the extracted features from point clouds in the bowl category may exhibit similarities or closeness seen categories, to some extent. In other words, bowl may share visual characteristics with items such cans, bottles, mugs, as identified by the PointNet++ of the

---

bottlo
bottl Wae
bottle font samera
Font
w/o bowl
w/o bottle
w/o can
Figure 5: Visualization of the features. 'w/o' denotes the unseen category during training process.
ScoreNet. To validate this hypothesis, we conducted a t-SNE [47] analysis on the point cloud feature space of the ScoreNet. Specifically, we extracted features from the point clouds of objects in the and unseen test set and visualized the t-SNE results. As shown in Figure 5, the results demonstrate that features from and bottles tend to intermingle, aligning with the accurate observation that both cans and bottles exhibit symmetrical cylindrical shapes. Meanwhile, features from the bowl category show proximity features from mugs.
4.5 Category-level Object Pose Tracking
Table shows the performance of category-level object pose tracking of our tracking method and baselines REAL275 datasets. For the first frame, utilized perturbed ground truth pose the initial object pose, following CAPTRA [48] and CATRE [49]. Despite being directly transferred from single-image prediction method, approach has achieved performance comparable the state-of-the method, CATRE. Remarkably, we significantly outperform CATRE in the metrics of 5 5em and mean rotation error. This highlights the expansibility of pose estimation method for more nuanced downstream tasks. Despite our lower FPS compared to CATRE, our method still adequately fulfills usage requirements, partieularly for robot operations.
Table 6: Results of category-level object pose tracking REAL275. The results averaged
over all 6 categories. The best performance is in bold and the second best is underscored.
Method
Oracle ICP[48]
6-PACK[50]
iCaps[51]
CAPTRA[48]
CATRE[49]
Ours
Input Init. Speed(FPS)t
RGBD GT.
RGBD
RGBD
D
D
GT. Pert.
Det. and Seg.
GT. Pert.
GT. Pert.
GT. Pert.
12.66
89.21
17.18
5emm
0.7
33.3
31.6
62.2
57.2
71.5
Rerr (?)
40.3
16.0
9.5
5.9
6.8
4.2
(emr(
7.7
3.5
2.3
7.9
1.2
1.5
5
Conclusion and Discussion
In this work, we study a fundamental problem, namely the multi-hypothesis issue, that has existed in category-level object pose estimation for long time. To overcome this issue, propose novel object pose estimation framework that initially generates the pose candidates via a score-based diffusion model. To aggregate the candidates, leverage energy-based diffusion model filter out the outliers, which avoids the costly integration process of the score-based model. Our framework achieves exceptionally state-of-the-art performance existing benchmarks, especially symmetric objects. In addition, our framework is capable of object pose tracking and can generalize to objects from novel categories sharing similar symmetric properties.
Limitations and Future Works: Although our framework achieves real-time object pose tracking at approximately 17 FPS, the efficiency of single-frame object pose estimation is still limited by the costly sampling process of the score-based model. In the future, we may leverage the recent advances in accelerating the sampling process of the diffusion models to speed up the inference [52; 53]. In addition to passively filtering out the outliers, we may incorporate reinforcement learning to train an agent that actively increases the likelihood of the estimated pose [54].
Boarder Impact: This work opens the door to leveraging energy- diffusion models to improve the generation results, which might facilitate more research in border communities.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China General Program (Project ID: 62376006), National Youth Talent Support Program (Project ID: 8200800081) 1), and Beijing Municipal Science & Technology Commission (Project ID: Z22 100003422004).
References
[1] Eric Marchand, Hideaki Uchiyama, and Fabien Spindler. Pose estimation for augmented reality: hands-on survey. IEEE transactions visualization and computer graphics, 22(12):2633- 2651, 2015.
[2] Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, and Dieter Fox. Self-supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 3665-3671. IEEE, 2020.
[3] Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birchfield. Deep object pose estimation for semantic robotic grasping of household objects. arXiv preprint arXiv: 18X9:1109.11790 2018.
[4] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin. Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2642-2651, 2019.
[5] Kai Chen and Qi Dou. Sgpa: couuctrre-guided prior adaptation for category- 6d object pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2773-2782, 2021.
[6] Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, Linlin Shen, and Ales Leonardis. Fs-net: Fast shape-based network for category-level 6d object pose estimation with decoupled rotation mechanism. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1581-1590, 2021.
[7] Meng Tian, Marcelo H Ang, and Gim Hee Lee. Shape prior deformation for categorical 6d object pose and size estimation. In Computer Micon-CCVV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 530-546. Springer, 2020.
[8] Jiehong Lin, Zewei Wei, Changxing Ding, and Kui Jia. Category-level 6d object pose and size estimation using self-supervised deep prior deformation networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 19-34. Springer, 2022.
[9] Jichong Lin, Zewei Wei, Zhihao Li, Songcen Xu, Kui Jia, and Yuanqing Li. Dualposenet: Category-level 6d object pose and size estimation using dual pose network with refined learning of pose consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3560-3569, 2021.
[10] Haitao Lin, Zichang Liu, Chilam Cheang, Yanwei Fu, Guodong Guo, and Xiangyang Xue. Sar-net: shape alignment and recovery network for category-level object pose and size estimation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 6707-6717, 2022.
[11] Yan Di, Ruida Zhang, Zhiqiang Lou, Fabian Manhardt, Xiangyang Ji, Nassir Navab, and Federico Tombari. Gpv-pose: Category-level object pose estimation via ccometry-guides point- wise voting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 6781-6791, 2022.
[12] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.

---

[13] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprini arXiv:2011. 13456, 2020.
[14] Qiyu Dai, Jiyao Zhang, Qiwei Li, Tianhao Wu, Hao Dong, Ziyuan Liu, Ping Tan, and He Wang. Domain randomization-ennance depth simulation and restoration for perceiving and grasping specular and transparent objects. Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX, pages 374-391. Springer, 2022.
[15] Guanglin Li, Yifeng Li, Zhichao Ye, Qihang Zhang, Tao Kong, Zhaopeng Cui, and Guofeng Zhang. Generative category-level shape and pose estimation with semantic primitives. In Conference Robot Learning, pages 1390-1400. PMLR, 2023.
[16] Shinji Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence, 13(04))376-380 1991.
[17] Dengsheng Chen, Jun Li, Zheng Wang, and Kai Xu. Learning canonical shape space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11933-11982 2020.
[18] Ruida Zhang, Yan Di, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Ssp-pose: Symmetry-aware shape prior deformation for direct category-level object pose estimation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7452-7459. IEEE, 2022.
[19] Ruida Zhang, Yan Di, Zhiqiang Lou, Fabian Manhardt, Federico Tombari, and Xiangyang Ji. Rbp-pose: Residual bounding box projection for category-level pose estimation. In Com- puter Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part I. pages 655-672. Springer, 2022.
[20] Jiaze Wang, Kai Chen, and Qi Dou. -ategory-level object pose estimation via cascaded relation and recurrent reconstruction networks. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 4807-4814 14. IEEE, 2021.
[21] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1677 2011.
[22] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
[23] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. Uncertainty in Artificial Intelligence, pages 574-584. PMLR, 2020.
[24] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[25] Yang Song and Stefano Ermon. Improved techniques for training score-based based generative models. Advances in neural information processing systems, 03:12438-12444 2020.
[26) Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415- 1428, 2021.
[27] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00 003640 2022.
[28] Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modeling. arXiv preprint 2rXi:22202.02763. 2022.
12

---

[29] Mingdong Wu, Fangwei Zhong, Yulong Xia, and Hao Dong. TarGF: Learning target gradient field for object rearrangement. arXiv preprint arXiv:2209.00853 2022.
[30] Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score-based generative models. arXiv preprini
2021.
[31] Ruojin Cai, Guandao Yang. Hadar Averbuch-Elor, Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In European Conference on Computer Vision, pages 364-381. Springer, 2020.
[32] Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gerard Medioni, and Leonid Sigal. Energy-based learning for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13936-13943 2021.
[33] Shitong Luo and Wei Hu. Score-based point cloud denoising. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 4583-4599 2021.
[34] Ruizhi Shao, Zerong Zheng. Hongwen Zhang, Jingxiang Sun, and Yebin Liu. Diffustereo: High quality human reconstruction via diffusion-bbsed stereo using sparse cameras. arXiv preprint arXiv:2207.088 0,00000 2022.
[35] Hai Ci, Mingdong Wu, Wentao Zhu, Xiaoxuan Ma, Hao Dong, Fangwei Zhong, and Yizhou Wang. Gfpose: Learning 3d human pose prior with gradient fields. arXiv preprint arXiv:2212.08644 2022.
[36] Tim Salimans and Jonathan Ho. Should EBMs model the energy the score? In Energy Based Models Workshop ICLR 2021, 2021.
[37] Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fer- gus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. arXiv preprint vrXiv:2302.11552 2023.
[38] John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26, 1980.
[39] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[40] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5745-5753 2019.
[41] F Landis Markley, Yang Cheng, John I. Crassidis, and Yaakov Oshman. Averaging quaternions. Journal of Guidance, Control, and Dynamics, 30(4):1 193-1197, 2007.
[42] IY Bar-Itzhack and Yaakov Oshman. Attitude determination from vector observations: Quater- nion estimation. IEEE Transactions on Aerospace and Electronic Systems, (1):128-136, 1985.
[43] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[44] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, systems, 32, 2019.
deep learning library. Advances in neural information processing
[45] Jason Y Zhang, Deva Ramanan, and Shubham Tulsiani. Relpose: Predicting probabilistic relative rotation for single objects in the wild. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI, pages 191-611. Springer, 2022.
13

---

[46] Kieran Murphy. Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Implicit-pdf, Non-paramettic representation of probability distributions on the rotation manifold. arXiv preprini
2021.
[47] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11). 2008.
[48] Yijia Weng, He Wang. Qiang Zhou, Yuzhe Qin, Yueqi Duan, Qingman Fan, Baoquan Chen, Hao Su, and Leonidas J Guibas. Captra: Category-level pose tracking for rigid and articulated objects from point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13209-13218, 2021.
[49] Xingyu Liu, Gu Wang, Yi Li, and Xiangyang Ji. Catre: Iterative point clouds alignment for category-level object pose refinement. In Computer Vision-ECCH 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II, pages 499-516 Springer, 2022.
[50] Chen Wang, Roberto Martín-Martín, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, and Yuke Zhu. 6-pack: Category-Ievee 6d pose tracker with anchor-based keypoints, 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10059-10066. IEEE, 2020.
[51] Xinke Deng, Junyi Geng, Timothy Bretl, Yu Xiang, and Dieter Fox. icaps: Iterative category- level object pose and shape estimation. IEEE Robotics and Automation Letters, 7(2):1784-177) 2022.
[52] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint wrXiv:2303.01469 2023.
[53] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.
[54] Hai Ci, Mickel Liu, Xuehai Pan, Fangwei Zhong, and Yizhou Wang. Proactive multi-camera collaboration for 3d human pose estimation. arXiv preprint arXiv:2303.03767 2023.
14

---

A Implementation Details
Architecture of the Score Network The detailed architecture of the score network b is illustrated in Figure We utilize PointNet++ [39] to extract the global geometry feature Fo of the partially observed point cloud O And the sampled pose and timestep ł features are embedded as Fp and Fr. respectively, using a Multi-Layer Perceptron (MLP). Then Fo. Fp and Fr are concatenated to obtain the global feature F. and three parallel branches are employed to predict the scores of Rz, Ry and T individually, where [R, |Ry] E IR and E R& denote rotation and translation vectors, respectively. And [Rg [Ry] is continuous rotation representation proposed by [40] to address the discontinuity of quaternions and Euler angles in Euclidean space. As introduced in [40], the mapping from SO(3) to the 6D representation of rotation is:
9GS ([a1 a2 as]) = [1
az]
(10)
The mapping form the 6D representation to SO(3) is:
fas (la1 a2]) [b1
b2 ba] if 1
(11)
N (a1)
b,
N( (b1 aa.2a))b))
if
(12)
by x b2
if i 3
Here N(.) denotes a normalization function.
Architecture of the Energy Network ture with the score network Φy. The inputs first fed into L to obtain score-shaped vector (p #|0) E RPP Then, the output energy is calculated by the dot product between the input pose and the score-shaped vector Li (p. t/)) (p, o (p, E RR
The energy network V shares exactly the same architec-
Pagaa (p.c(0)
MLP
aR_
Ix256
MLP
|--II
log Patca (p,c dR.
Lx128
1x1408
Fo
log Paata (0.)
PointNet
Partial Point Cloud 0*
dy
1x1024
Figure 6: Architecture of the score network og. denotes sampled 6D object poses. O' denotes partially observed 3D point cloud condition. denotes timestep. denotes the concatenation operator.
B Qualitative Comparison on REAL275
Figure 2 iillsstrates the qualitative comparison results between our method and RBP-Pose [19] on the REAL275 dataset. The images accompanied by red boxes highlighting objects that exhibit noticeable differences in the predicted results. Additionally, the bottom-right corner of each image provides enlarged view of the highlighted object, showing the ground truth pose well as the poses estimated by RBP-Pose and our approach. Our method demonstrates a significant performance improvement compared RBP-Pose, particularly in the of objects such mugs. Notably, in the fourth row of the figure, it can be observed that our method achieves highly accurate poses even when only small portion of the mug handle is visible. This success can be attributed to the fact that, during the training process, a unique pose exists when the mug handle is visible. However, when the mug handle becomes occluded, a hypothsssss problem arises, which our generative formulation effectively handles.
C More Results and Analysis
C.1 Per-category Results
Figure demonstrates quantitative comparison between our method and the state-of-the-art depth- based approach, RBP-Pose [19], for various object categories at different thresholds. The results
15

---

GT
RBP-Pose
Ours
Figure 7: Qualitative comparison with RBP-Pose [19] on REAL275. The left column represents the ground truth pose, the middle column represents the results of RBP-Pose, the right column represents the results of our approach.
clearly indicate that our method outperforms RBP-Pose in all metrics, despite the fact that we do incorporate augmentation specifically designed for symmetric objects during the training phase, unlike RBP-Pose. Our approach exhibits significant improvements, particularly regions with stringent threshold requirements. This emphasizes the superior performance of our generative category-level object 6D pose estimation approach in effectively addressing the multiihyypohhesis challenges posed by symmetric objects and partial observations, thereby enabling its successful application in robot manipulation tasks demanding precise object pose prediction. pouring liquids.)
C.2 Results on CAMERA
Table illustrates quantitative comparison between our method and the baselines on the CAM- ERA [4] dataset. The results clearly demonstrate the remarkable performance enhancement achieved by our method. When compared to approaches that rely solely on depth data network input, as well as those that utilize RGB-D and shape priors network input, our method consistently outperforms them, surpassing
current state-of-the-an performance. Notably, our method exhibits particularly pronounced advantage when stricter accuracy requirements are imposed, such as the 5*2cm metric. In this case, our method outperforms the current SOTA method, RBP-Pose, by an impressive margin of 6.4%. This significant improvement highlights the efficacy of our approach.
16

---

Rotation
Translation
Rotation
Translation
100
100
80
Average Precision
60
60
RRP-PPos
Auarage
bottle
bolt
40
40
Mpttp
leptop
10
10
Cantimeter
Cantimetor
Figure 8: Per-category quantitative comparison with RBP-Pose [19] on REAL275. The left represents the results of RBP-Pose, while the right represents the results of our approach.
Table 7: Quantitative comparison of category-level object pose estimation on CAMERA [4] dataset. We summarize the results reported in the original paper for the baseline method. t represents a higher value indicating better performance, while 4 represents a lower value indicating better performance. Data refers the format of the input data used by the method, and Prior indicates whether the method requires category prior information. indicates that the metrics are not reported the original paper. represents the number of hypotheses."
Method
Data
Prior
5"2emf
595emt
10"2cm²
10"5em"
Parameters(MI))
NOCS
RGB-D
32.3
40.9
48.2
64.6
DualPoseNet [9]
RGB-D
64.7
70.7
77.2
84.7
67.9
SPD [7]
RGB-D
54.3
59.0
73.3
81.5
18.3
CR-Net [20]
RGB-D
72.0
76.4
81.0
87.7
SGPA [5]
RGB-D
70.7
74.5
82.7
88
Deterministic
GPV-Pose [1]]
D
72.1
79.1
89.0
SAR-Net [10]
D
66.7
70.9
75.3
80.3
6.3
SSP-Pose [18]
D
64.7
75.5
87.5
RBP-Pose
79.6
82.1
89.5
Ours
79.9
84.4
84.6
89.6
4.4
Probabilistic
Ours(K=10)
D
90.8
93.0
93.4
95.7
2.2
Ours( (=5))
D
95.5
96.4
97.2
98.2
2.2
C.3 Real World Experiments
We have also successfully integrated our approach with robot manipulation capabilities, as demonstrated through various experiments conducted with the UFACTORY xArmó equipped with RealSense D435. The demon- strations can be found in the supplementary video or on the project website. As shown in Figure 9, we illustrate the following three tasks:
Pouring Task. This task involves transferring the con- tents (e.g., water) from container to another. The demonstration highlights the potential of combining our approach with heuristic strategies, enabling functional robot operations.
Pouring
(h) Stacking
Stacking Task. In this task, we focused on piling up objects of the same category, like organizing scattered bowls on tabletop. This demonstrates the precision of the estimated object pose, as accurate knowledge of object poses is crucial for completing this task.
Handover
Handover Task. This task involved either receiving objects from human hands to perform tasks or passing ob- jects to person. The demonstration exemplified one form of human-robed interaction empowered by our method.
Figure 9: Pose estimation for robot ma- nipulation tasks. We demonstrate three types of tasks.
17

---

D Details of Object Pose Tracking
Our pose estimation framework can be adapted to pose tracking with minor modifications. With the learned score network o the tracking algorithm is summarised as follows:
Algorithm 1 Our Object Pose Tracking Framework
1: Initialisation: Score network D energy network v d. initial point cloud Ov. the initial ground truth pose reference Po and the tracking horizon T.
2: for /= = 1 to 7 do
3: Receive current observation O₂
4: 21, ZK -V (Pt-1, 101-,0.1277 0.1
Sample initial poses around the previous prediction. 5: Sample candidates {f,) k from the initialization (z1, (2..... via the following PF-ODE:
dp dt (t) (0,,)
(13)
6: Pr PT.... > DTX where p,, Pr, - Lo cl0) > V Vo(D,, cl0)
Ranking
-i= Tr.
7: Estimate the current pose
qe) where = = = K and:
M
Σ Li=1 A(q..)
q. arg max q"
M q. A(q.)) = - T
(14)
gE8O(3)
8: end for
E
Ethics Statement and Boarder Impact
Our method has the potential develop the home-assisting robot, thus contributing social welfare. We evaluate our method in synthesized or human-collected datasets, which may introduce data bias. However, similar studies also have such general concerns. We do not see any possible major harm in our study.
18


Reviews:
[{"review_id": "flXkVxuJpv", "reviewer": "Reviewer_7PCs", "summary": "This paper proposes a diffusion-based model for category-level object pose estimation. Different from the existing deterministic approaches which treat the object pose estimation as a regression problem, the proposed diffusion model alternatively formulated it as a generation problem. In this way, it could tackle the multiple pose hypotheses issue comes from symmetric objects and partial observation.  The proposed approach achieves state-of-the-art performance on existing benchmarks and could generalize to unseen symmetric categories.", "questions": "Some questions are listed in the weakness section.", "limitations": "The authors have discussed the limitations of this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "a. The paper introduces a novel diffusion-based framework for category-level pose estimation. The framework consists of two diffusion models. The first diffusion model generates a set of pose candidates for a given point cloud during the inference stage. The second diffusion model uses an energy-based approach to rank the candidates and filter out those with low rankings. This paper claims to be the first to propose a solution for pose estimation using diffusion models.\n\nb. The diffusion-based model presented in the paper demonstrates state-of-the-art performance on existing benchmarks. It surpasses the performance of previous approaches by a significant margin. Additionally, the authors argue that their framework has the potential to generalize to objects of unseen categories that have not been studied by other approaches.", "weaknesses": "a. The chosen representation of the pose parameter in this paper consists of a 9-dimensional vector, where 6 dimensions are allocated for rotation and 3 dimensions for translation. Notably, the omission of the scale parameter from this representation raises an important inquiry regarding the reasons behind its exclusion and the means by which it can be obtained during inference. Further clarification is required to address these concerns.\n\nb. The current approach employed by the authors involves utilizing mean pooling to derive the final pose estimation from the filtered pose candidates. Given that the energy-based diffusion model is capable of estimating the data likelihood for each pose candidate, an alternative consideration arises: would adopting weighted averaging be a more suitable approach for calculating the final pose estimation? It would be insightful to explore the potential benefits of incorporating weighted averaging as a means to enhance the accuracy of the pose estimation.\n\nc. Additional information clarifying the specifics of the pose tracking implementation would be appreciated. Considering that the framework incorporates two diffusion models, it appears intriguing for me to comprehend how the pose tracking component can maintain such computational efficiency and deliver results in real-time.\n\nd. Some typos. (Line 213)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: inquiry regarding the reasons about exclusion of scale parameters and the means by which it can be obtained during inference.**\n\n**A1:** Thanks for pointing this out! We would like to clarify that our work primarily focus on the estimation of a 6D object pose. Nonetheless, our method possesses the capability to derive a 9D object pose given the point cloud and the segmentation mask. This process involves several key steps:\n - Transforming the object's point cloud into the canonical space, utilizing estimated 6D object pose. \n - Denoising the caninocal point cloud via an off-the-shelf outlier removal algorithm.\n - Calculating the axis-aligned bounding box of the denoised point cloud to get the object size. \n\nAnd in our real world experiments (can be seen on the project page or supplementary video), the 9D object pose is calculated via the aforementioned method and be used for robotic manipulation. We acknowledge that estimating 3D sizes is also important and willing to provide more comprehensive evaluations if required.\n\n> **Q2: would adopting weighted averaging be a more suitable approach for aggregation?**\n\n**A2:** Thanks for your insightful suggestion! We concede that there might be other aggregation techniques superior to simply averaging. Nonetheless, the primary concern in this work is dealing with outliers. The presence of outliers invariably skews aggregated results, regardless of the chosen method, be it K-means or Mean Pooling. While some research addresses the multi-hypothesis challenge, they overlook this foundational problem. To our best knowledge, this is the first work that leverages energy-based diffusion model to remove outliers, which is also the key technical novelty of this work.\n\nWe conducted experiments to compare mean-pooling with other aggregation methods (i.e., likelihood weighting). As shown in **Table 1** of the PDF, mean-pooling consistently outperforms weighted averaging. We hypothesize that the energy model is better at distinguishing poses with significant differences in errors but performs poorly when distinguishing poses with low errors. As a result, some poses with higher errors might overshadow those with lower errors in energy weighting. To validate this hypothesis, we further explored the relationship between the pose error and the energy output. Results in **Figure 1** of the PDF demonstrate that the energy model assigns similar (right, <2 cm), or even incorrect values (left, <10 degrees) for poses with low errors. Meanwhile, the energies of low-error poses (e.g., <10 degrees, <2 cm) are higher than those of high-error poses (e.g., >20 degrees, >4 cm).\n\n\n> **Q3: clarifying the specifics of the pose tracking implementation.**\n\n**A3:** \nWe apologize for any confusion caused. We have summarized our tracking algorithm in **Algorithm 1** of the PDF for your reference. Our algorithm is adept at maintaining both computational efficiency and high performance. This is primarily attributed to the benefit of warm-starting from previous predictions. Since the difference in object positions between adjacent frames is typically minimal, the ODE process requires fewer time steps to arrive at a reliable estimation from the previous one, thereby boosting efficiency.\n\n> **Q4: Some typos. (Line 213)**\n\n**A4:** Thanks for pointing out the typos, and we will address all the typos in the revised version.\n"}, {"review_id": "RrLtlenwBN", "reviewer": "Reviewer_4bwh", "summary": "To settle the multi-hypothesis issue in category-level 6D pose estimation, this paper formulates the focused task as conditional generative modeling and proposes a novel method based on diffusion models, which utilizes a score-based diffusion model to sample pose candidates with an energy-based one followed to rank those candidates. The proposed method achieves the state-of-the-art results on REAL275 dataset for category-level pose estimation, and is further extended for the task of pose tracking.", "questions": "- What’s the special designs of the proposed method for category-level pose estimation, compared to instance-level pose estimation? Could it be applied to the instance-level task?\n\n- Some typos.\n    - ’w‘ in Eq. (1) and $\\dot{\\sigma}$ in Eq. (4) are not explained.\n    - In Line167, should '$log p_t(p|O)$' be '$\\Delta_p log p_t(p|O)$'?\n    - In Line213, \"symmetric objects ??\" -> \"symmetric objects\".\n    - In Line278,281,284, should all the 'M' be K\"?\n    - In References, paper [12] and paper [13] refer to the same papers.", "limitations": "The authors have discussed the limitations of their work in the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The authors propose a new perspective for category-level 6D pose estimation by formulating the task as conditional generative modeling and realizing it via diffusion models.\n\n- The proposed method achieves the state-of-the-art results on REAL275 dataset for both tasks of category-level pose estimation and pose tracking. Cross-category experiments and real-world applications are also conducted to verify its generalization ability.\n\n- The paper is well written and presented with detailed illustrations and thorough experiments.\n\n", "weaknesses": "- The proposed method is not competitive in terms of inference time with the use of generative models.\n\n- The original definition of category-level task includes the estimation of object sizes, which could not be learned in the proposed method.\n\n- Some methods are not compared with in Table 1 for pose estimation, e.g., [1][2][3], and Table 6 for pose tracking, e.g., [1][4][5].\n\nReference:\n\n[1] Sparse Steerable Convolutions: An Efficient Learning of SE(3)-equivariant Features for Estimation and Tracking of Object Poses in 3d Space.\n\n[2] CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation.\n\n[3] ShAPO: Implicit Representations for Multi-Object Shape, Appearance and Pose Optimization.\n\n[4] ICK-Track: A Category-Level 6-DoF Pose Tracker Using Inter-Frame Consistent Keypoints for Aerial Manipulation.\n\n[5] BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: The proposed method is not competitive in terms of inference time.**\n\n**A1:** Thanks for pointing it out! We acknowledge that the sampling process of the diffusion model does introduce a notable computational overhead. Consequently, our current approach for estimating 6D object pose from individual images does indeed lack competitive inference speed.To address this concern, we have demonstrated within the main body of our paper the adaptation of our method into a final framework for 6D object pose estimation, achieving a swift execution rate (18FPS) while maintaining high performance. Additionally, we are actively exploring a promising avenue for further research, which involves expediting the sampling process [1][2]. This area constitutes our future work.\n\n> **Q2: Object sizes could not be learned in the proposed method.**\n\n**A2:** Thanks for brining it up! To clarify, although our method is primarily geared towards 6D object pose estimation, it can directly produce a 9D object pose when provided with a point cloud and segmentation mask. We first map the object point cloud to canonical space using its estimated 6D pose. Then, we refine the canonical point cloud using a standard outlier removal algorithm. Finally, we determine the 3D scales by computing the axis-aligned bounding box of the refined point cloud. The 9D object pose computation process is deployed in the real-world experiments found on our project page and in our supplementary video. We understand the importance of estimating 3D scales and are open to offering a more exhaustive evaluation if needed.\n\n\n> **Q3: Some methods are not compared with in Table 1 for pose estimation, e.g., [1][2][3], and Table 6 for pose tracking, e.g., [1][4][5].**\n\n**A3:** We sincerely appreciate your valuable feedback, which has shed light on additional baseline methods. According to your suggestion, we have meticulously presented a comparative analysis between our proposed approach and the methods referenced in your comments. \n\nRegarding the primary task of category-level 6D object pose estimation, our method maintain a substantial lead in terms of performance:\n| Method | $5^{\\circ}2$cm$\\uparrow$ | $5^{\\circ}5$cm$\\uparrow$ | $10^{\\circ}2$cm$\\uparrow$ | $10^{\\circ}5$cm$\\uparrow$ |\n|---|---|---|---|---|\n|            [3]          |   36.6    |   43.4   |   52.6    |    63.5    |\n| CenterSnap[4]  |      -       |   29.1   |       -      |   64.3    |\n|     ShAPO[5]     |      -       |   48.8   |       -      |    66.8    |\n|          Ours        | **52.4** | **61.2** | **72.8**| **84.2** |\n\nFor object tracking task, despite being directly transferred from a single-image prediction method, our approach has achieved comparable performance with the baselines:\n|Method| $5^{\\circ}5$cm$\\uparrow$ | $r_{error}\\downarrow$ | $t_{error}\\downarrow$ |\n|---|---|---|---|\n|                        [3]                         |    54.5   |   5.2    |   1.9    |\n|                ICK-Track[6]                 | **84.4** |   4.5    |   3.1    |\n| BundleTrack[8] w/o Pose Graph |    39.9   |   9.2    |   2.4    |\n|                      Ours                       |    71.5   | **4.2** | **1.5** |\n|              BundleTrack[7]               |    87.4   |   2.4    |   2.1    |\n\nNotably, the bundle track method achieved enhanced performance by employing multi-frame images for global optimization. However, it is noteworthy that this procedure holds generality. When subjected to a fair comparison utilizing only a pair of image frames, our approach demonstrates superiority over the bundle track method. \n\nWe intend to include these particular baseline comparisons, as well as the references, in the revised version of our manuscript.\n\n> **Q4: What’s the special designs of the proposed method for category-level pose estimation, compared to instance-level pose estimation? Could it be applied to the instance-level task?**\n\n**A4:** Thank you for addressing this. For details on the special design, please refer to Q1 in the Common Responses. Our framework is specifically designed to address the multi-hypothesis challenge in category-level pose estimation. However, it can also be adapted for instance-level tasks by conditioning on the target object's CAD model. It's worth noting that this adapted method might not offer significant benefits over other instance-level methods.\n\n### Thank you for the detailed response regarding typos. We will revise the paper accordingly. Below are the detailed explanations: \n\n> **T1: 'w' in Eq. (1) and \\sigma in Eq. (4) are not explained.**\n\n**A1:** The $dw$ is the standard Wiener process[3] (a.k.a., Brownian motion). The $\\dot{\\sigma}(t)$ is the derivation of $\\sigma(t)$: $\\dot{\\sigma}(t) = \\sigma_{min} \\cdot (\\ln \\sigma_{max} -\\ln \\sigma_{min}) (\\frac{\\sigma_{max}}{\\sigma_{min}})^t$\n\n> **T2: In line 167, should `logp_t(p|O)' be \\delta `log p_t(p|O)'?**\n\n**A2:** Yes, it shoud be $\\delta_{p} \\log p_t(p|O)$.\n\n> **T3: In Line278,281,284, should all the 'M' be K\"?**\n\n**A3:** Yes! It should be $K$.\n\n[1] Song Y,  et al. Consistency models. ICML 2023.\n\n[2] Salimans T, et al. Progressive distillation for fast sampling of diffusion models. ICLR 2022.\n\n[3] Lin J, et al.  Sparse Steerable Convolutions: An Efficient Learning of SE(3)-equivariant Features for Estimation and Tracking of Object Poses in 3d Space. Advances in Neural Information Processing Systems, 2021.\n\n[4] Irshad M Z, et al. CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation. ICRA 2022.\n\n[5] Irshad M Z,  et al. ShAPO: Implicit Representations for Multi-Object Shape, Appearance and Pose Optimization. ECCV, 2022.\n\n[6] Sun J, et al. ICK-Track: A Category-Level 6-DoF Pose Tracker Using Inter-Frame Consistent Keypoints for Aerial Manipulation. IROS, 2022.\n\n[7] Wen B, et al. BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models. IROS, 2021.\n"}, {"review_id": "mGfrsaXbYf", "reviewer": "Reviewer_W8Ay", "summary": "This paper mines and formulates ambiguity in the task of object pose estimation, proposing to use a diffuse generative model to generate multiple hypotheses, which are then aggregated through additional scoring and ranking by another scorer net. The model achieves significant improvements with less supervision and parameters, has good cross-category generalization results, and performs well on tracking and stimulation tasks.", "questions": "See Weaknesses.", "limitations": "Yeah.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- I like the observation of the symmetric ambiguity and generative multi-hypothesis formulation.\n- The improvement has been dramatically boosted, and it can beat all SOTAs even with fewer supervision signals and parameters.\n- Thank the authors for testing the method on many datasets and tasks.", "weaknesses": "- (Conditional generation) Adding objects as conditions should further improve in-distribution test performance, right?\n(Redundancy of model designs) EnergyNet can also be used for sampling, so why is ScoreNet needed?\n- (Model design choices) How about using other models instead of EnergyNet to estimate, e.g., normalizing flows can give exact likelihood? There has been much multi-hypothesis work in human pose estimation/motion generation [a-c]. Essentially, this framework does not explore or mine features for object pose estimation. In other words, this model can also be used in other fields, and models in other fields can also be used in this field. Regarding the problem setup, there seem to be no many special things about monocular estimation except the mentioned symmetric ambiguity; however, the dramatic improvements in straightforwardly applying generative models encourage us to think about the underlying principles.\n- (Mean pooling aggregation) Why must mean pooling to be used for aggregation? It looks like the mode of the largest cluster corresponds to the most likely solution, which is probably better than the mean. Additionally, some other works also study empirical aggregations, e.g., clustering [a], weighting [b] and [c]. Is it because quaternions are not easy to work on? What about converting to other forms (e.g., coordinates)? I expect to see comparisons or authors’ discussions on this.\n- (Less yet better?) Can you elaborate on your speculation for the L265-267? Why does your method require less supervision information and a lighter model yet achieve better results? This guess will be very interesting.\n- (Best hypothesis) I don't quite understand the last line in the table in Tab. 3; why is mean pooling needed since GT is accessible? What did I miss?\n- (Cross-category generalization) I think the authors should emphasize the limitations when claiming generalization gains. I don't particularly understand that even the generative model also has unstable, bad, and unreliable performance for OOD point cloud conditions that have not been seen in the training set. And comparing the baselines in Tab. 5 is unfair; it is better to let them also train and test on the same split. I hope the author could kindly elaborate on this.\n- (Missing related works) Should add related work of multi-hypothesis generation [a-e].\n- (Visualizations of multiple pose hypotheses) While there is a dimensionality reduction visualization in Fig. 3, it is suggested to add the predicted multiple pose hypotheses along with the estimated energies to illustrate the method's effectiveness qualitatively. I also hope that the author can provide visualizations of the results, showing the advantages and failures of your method compared to the baseline under different settings such as D and RGB-D (Sup. Fig. 2 is better to also show input depth and point cloud data). This helps readers better understand the proposed method.\n- (Reproducibility) While this cannot be forced or required, if the author commits to publicizing the code, it will have a greater impact and be more helpful to the field.\n\nReferences:\n- [a] C. Li & G. Lee. Weakly supervised generative network for multiple 3D human pose hypotheses. BMVC’20.\n- [b] B. Biggs et al. 3D multibodies: Fitting sets of plausible 3D models to ambiguous image data. NeurIPS’20.\n- [c] W. Shan et al. Diffusion-based 3D human pose estimation with multi-hypothesis aggregation. CVPR’23.\n- [d] G. Chliveros et al. Robust multi-hypothesis 3D object pose tracking. ICVS’13.\n- [e] F. Michel et al. Global hypothesis generation for 6D object pose estimation. CVPR’17.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: why is mean pooling needed since GT is accessible? What did I miss?**\n\n**A1:** To clarify, the ground truth (GT) is not accessible during test time. That's why we employ another energy-based diffusion model to aggregate these candidates into a final output in the absence of the GT.\n\nIn our experiments, we emphasize the performance without the GT label in Table 1 of the main text. Additionally, we assess the upper limit of our approach by evaluating the best candidate (i.e., the one closest to the GT), in line with what other multi-hypothesis studies do[1].\n\n> **Q2: EnergyNet can also be used for sampling, so why is ScoreNet needed?**\n\n**A2:** Thanks for the good question. There are two main reasons:\n - The score-based model significantly outperforms the energy-based model in candidate generation. We conducted an ablation study to compare the performance of using the score and energy models for candidate generation, respectively. As shown in **Table 2** of the PDF, 'score + energy' markedly outperforms 'energy + energy' across all metrics. This might be due to the limited capabilities of using the 2nd-order derivatives of an energy model to parameterize the score function.\n\n - Furthermore, using the energy-based model to sample candidates requires calculating the second-order derivatives, which is time-inefficient.\n\n> **Q3: Why must mean pooling to be used for aggregation?**\n\n**A3:** Thank you for your valuable suggestion. Due to the character limit, please refer to **Q3** in the Common Responses.\n\n> **Q4: Why does your method require less supervision information and a lighter model yet achieve better results?**\n\n**A4:** Thank you for highlighting this interesting point. We hypothesize that the performance gain arises from the enlarged computational graph induced by the denoising process. During inference, the pose candidates are generated from a denoising process that involves hundreds of inferences from the score network. Although the score network has only **2.2M** parameters, the expanded computational graph due to the denoising process can grow to encompass hundreds of millions of parameters.\n\n> **Q5: This framework does not explore or mine features for object pose estimation. What is the underlying principles of the dramatic improvements?**\n\n**A5:** Thank you for bring this up. Sorry for any confusion regarding our unique design features.  \n - (Special Design) Due to the character limit, please refer to **Q1** in Common Response.\n\n - (Underlying Priciples) Moreover, we concur that understanding the principles behind the significant improvements is valuable. We hypothesize that these improvements might stem from the substantially expanded computational graph of the denoising process and the energy model's capacity to eliminate outliers. We delve into these hypotheses in **Q3** and **Q4**, respectively. We will incorporate these analyses in our subsequent revision.\n\n> **Q6: The authors should emphasize the limitations when claiming generalization gains. Comparing the baselines in Tab. 5 is unfair**\n\n**A6:** We apologize for the oversight in clarifying the limitations. The limitations in cross-category generalization stem from conditional generation, as statistical methods can't guarantee out-of-distribution (OOD) generalization. For instance, if trained on REAL275 without cameras, our method may not perform well on camera objects due to their distinct geometry. We'll update Sections 4.4 and 5 to highlight these limitations more clearly.\n\nWe are still retraining all the baselines using the same split as ours (see Table 5 of the PDF). We will update the final results as soon as possible during the discussion period.\n\n> **Q7: Adding objects as conditions should further improve in-distribution test performance, right?**\n\n**A7:** We agree that adding such prior (e.g., a 'mean point cloud') as input conditions might further improve the in-distribution test performance, which is also observed by previous caninocal-based methods. \nNonetheless, such design choice would also hurt the OOD performance when encountered with objects from novel categories, due to the large shape variations between the seen and unseen categories. \n\n> **Q8: How about using other models instead of EnergyNet to estimate likelihoods, e.g., normalizing flows?**\n\n**A8:** Thanks for the good question. We agree that flow-based model could also be an alternative approach for exact likelihood estimation. Nevertheless, normalizing flows have to use specialized architecture to build a normalized probability model, which limits their capability. Besides, estimating exact likelihood from normalizing flows requires calculating the determinant of Jacobian matrix of z to x mapping, which is time-consuming. On the other side, diffusion-based methods have achieved SOTA performance on NLL test[2].\nIn light of this, we choose to use EnergyNet to estimate the likelihood. We will include the rationale of choosing the likelihood estimator\n\n> **Q9: Should add related work of multi-hypothesis generation [a-e].**\n\n**A9:** Thanks for your advice! We will include these related works in the future revision.\n\n> **Q10: Add the predicted multiple pose hypotheses along with the estimated energies. Showing the advantages and failures of your method**\n\n**A10:** Thanks for the good suggestion! We promise to revise Fig.2 and Fig.3 accordingly and provide qualitative results of advatanges/failure cases in the future revision. However, due to the page limit of PDF, we do not demonstrate them at the rebuttal stage.\n\n> **Q11: Releasing the codes will have a greater impact and be more helpful to the field.**\n\n**A11:** We promise to release the code upon acceptance. We are also open to deploying our model on HuggingFace to validate its effectiveness, if required.\n\n[1] Shan et al. Diffusion-Based 3D Human Pose Estimation with Multi-Hypothesis Aggregation.\n\n[2] Song et al. Score-based generative modeling through stochastic differential equations.\n\n"}, {"review_id": "RISW9OenRY", "reviewer": "Reviewer_sHLy", "summary": "This paper proposes a novel conditional generative approach for category-level pose estimation to solve the multi-hypothesis problem. The proposed method utilize the energy-based diffusion model to aggregate the candidates generated by the score-based diffusion model. Extensive experiments have been conducted to show the effectiveness of the proposed method on the REAL275 dataset under category-level pose estimation/tracking and novel category pose estimation evaluation protocols.", "questions": "1. If it is not the model that predicts all classes, but each model, how did you compare the parameters of the model (Table 1)?", "limitations": "Covered the limitation in the main paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author states the method clearly, and the notation is easy to follow.\n2. The author proposed a novel multi-hypothesis method and shows promising results compared to regression and correspondence methods. \n3. The ablation study is convincing.", "weaknesses": "1. I wonder if scoreNet and energyNet are needed for each class. Does a single model infer all classes? \n2. I wonder about the inference time of object pose estimation. Is the object pose estimation inference time linearly increased according to the number of hypotheses?\n3. Is there anything to consider about symmetries properties when augmentation in the training process?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: I wonder if scoreNet and energyNet are needed for each class. Does a single model infer all classes?**\n\n**A1:** Apologies for the confusion. To clarify, we only require one set of both the score and energy models for all classes. Notably, neither model is conditioned on the class label or categorical priors, such as a mean point cloud. We will revise the paper to make this point clearer.\n\n\n> **Q2:  I wonder the inference time of object pose estimation. Is the object pose estimation inference time linearly increased according to the number of hypotheses?**\n\n**A2:** Thank you for raising this point. We concur that inference time is a critical concern. Due to the time-consuming sampling process of our diffusion model, our model achieves only 3FPS when handling the 6D object pose estimation based on single-frame images. However, to counter this challenge, we introduced a tracking algorithm. By leveraging well-initialized values from the previous frame, we significantly accelerate the sampling process, enabling the tracking frame rate to reach **18FPS**. Furthermore, we are actively exploring avenues to expedite the sampling process [1][2], marking an area of our future work.\n\nThe inference time for our single-frame object pose estimation doesn't scale linearly with the number of pose hypotheses. This is because the hypotheses are sampled in batch-wise parallel (on the GPU) using the reverse ODE process.  Below, we list the inference times for varying numbers of pose hypotheses:\n\n| | 1 | 10 | 20 | 30 | 40 | 50 |\n|---|---|---|---|---|---|---|\n|   Single Frame Pose Estimation (s) |  0.231 |   0.285 |   0.292   | 0.301 | 0.316 | 0.328 |\n|               Pose Tracking (s)               | 0.036 | 0.043   |  0.046\t |0.049\t | 0.052\t | 0.054 |\n\n[1] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. ICML 2023\n[2] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. ICLR 2022.\n\n> **Q3: Is there any symmetric augmentation in the training process?**\n\n**A3:** To clarify, we didn't incorporate specific designs, like tailored loss functions or symmetry-focused data augmentation, for symmetrical objects during training. Even so, we noted significant performance improvements, particularly with symmetric objects (as detailed in Sec 4.4 of the main text). We credit this success to our conditional generation formulation, which we believe provides valuable insights for the wider research community.\n\n> **Q4: If it is not the model that predicts all classes, but each model, how did you compare the parameters of the model (Table 1)**\n\n**A4:** As mentioned in **Q1**, we only train a single model to infer all categories in our paper. \n\n**We hope our rebuttal could address your concerns. Please let us know whether you have further questions. We are sincerely waiting for discussion with you.**\n"}, {"review_id": "F5b3VleWQn", "reviewer": "Reviewer_sAr7", "summary": "This paper proposes a novel approach for generative object pose estimation based on diffusion models. \n", "questions": "As a likehood is estimated for each pose hypothesis, maybe an alternative way to get the final estimation is fusing them with a weighting scheme. For example, could it be possible to design a weighted version of Eq-9? ", "limitations": "The limitation of inference speed and future work is included in the last section of the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Formulating pose estimation as a diffusion process is novel. It shows excellent ability to solve the multi-hypothesis issues in pose estimation caused by symmetry and partial observation. Moreover, it can be easily extended to solve the object pose tracking problem.\n\n2. This paper proposes an efficient framework to estimate the object pose using diffusion models. It trains a score-based diffusion model to sample pose hypotheses, and an additional energy-based diffusion model to estimate the likehood of each hypothesis.\n\n3. The paper is well written with nice figures, clear equations and proficient writing style. \n\n4. Experimental results on several benchmarks demonstrate the efficiency of the proposed method.", "weaknesses": "In Table-1, this paper conducts comparison with category-level object pose estimation methods that predict 9 DoF object pose, consisting of 3D rotation, 3D translation and 3D size. However, the 3D size is not considered by the proposed method. Though the metrics are focused on rotation and translation, it still could cause a potential unfair comparison.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **(6D -> 9D)Q1: In Table-1, this paper conducts comparison with category-level object pose estimation methods that predict 9 DoF object pose, consisting of 3D rotation, 3D translation and 3D size. However, the 3D size is not considered by the proposed method. Though the metrics are focused on rotation and translation, it still could cause a potential unfair comparison.**\n\n**A1:**  Thanks for pointing it out! To clarify, although our method is primarily geared towards 6D object pose estimation, it can directly produce a 9D object pose when provided with a point cloud and segmentation mask. We first map the object point cloud to canonical space using its estimated 6D pose. Then, we refine the canonical point cloud using a standard outlier removal algorithm. Finally, we determine the 3D scales by computing the axis-aligned bounding box of the refined point cloud. The 9D object pose computation process is deployed in the real-world experiments found on our project page and in our supplementary video. We understand the importance of estimating 3D scales and are open to offering a more exhaustive evaluation if needed.\n\n\n> **(Mean Pooling v.s. Weighted Mean Pooling)Q2: As a likehood is estimated for each pose hypothesis, maybe an alternative way to get the final estimation is fusing them with a weighting scheme. For example, could it be possible to design a weighted version of Eq-9?**\n\n**A2:** Thanks for your insightful suggestions! We concede that there might be other aggregation techniques superior to simply averaging. Nonetheless, the primary concern in this work is dealing with outliers. The presence of outliers invariably skews aggregated results, regardless of the chosen method, be it weighed Mean Pooling or Mean Pooling. While some research addresses the multi-hypothesis challenge, they overlook this foundational problem. To our best knowledge, this is the first work that leverages energy-based diffusion model to remove outliers, which is also the key technical novelty of this work.\n\nNevertheless, we conducted experiments to compare mean-pooling with other aggregation methods (i.e., likelihood weighting). As shown in **Table 1** of the PDF, mean-pooling consistently outperforms weighted averaging. We hypothesize that the energy model is better at distinguishing poses with significant differences in errors but performs poorly when distinguishing poses with low errors. As a result, some poses with higher errors might overshadow those with lower errors in energy weighting. To validate this hypothesis, we further explored the relationship between the pose error and the energy output. Results in **Figure 1** of the PDF demonstrate that the energy model assigns similar (right, <2 cm), or even incorrect values (left, <10 degrees) for poses with low errors. Meanwhile, the energies of low-error poses (e.g., <10 degrees, <2 cm) are higher than those of high-error poses (e.g., >20 degrees, >4 cm).\n"}]

Author rebuttal:
We are grateful to all reviewers for recognizing the merit of our idea, experiments, and presentation:
 - "I appreciate the observation of symmetric ambiguity and the generative multi-hypothesis formulation." (W8Ay)
 - "The improvement has seen a significant boost." (W8Ay)
 - "This surpasses the performance of prior approaches by a considerable margin." (7PCs)
 - "We've also conducted cross-category experiments and real-world applications to validate its generalizability." (4bwh)
 - "The paper is meticulously written with comprehensive illustrations and thorough experiments." (4bwh)

**(Q1: Special Design)** However, we notice that some reviewers may be confused about our special design on object estimation tasks (**4bwh**, **W8Ay**). In this work, we address the multi-hypothesis issue, which is one of the **key features** of monocular object pose estimation. Moreover, to eliminate outliers resulting from conditional generation, our **key design** feature is the use of an energy-based diffusion model to filter them out. To highlight this primary objective, our networks consist solely of commonly used architectures. This aspect notably paves the way for future explorations of specialized architectural designs within our framework. We will clarify this further in our revised introduction. 

| | Previous works | This work | 
| --- | --- | ---|
| Training Paradigm | Regression-based | Conditional Generative Modeling|
| Key Challenge |  Handling Intra-class Variations |  Removing Outliers |

**(Q2: 3D Scale Estimation)** We also observed feedback from reviewers noting our omission of 3D scale estimation in the primary text (**7PCs**, **4bwh**, **sAr7**). To clarify, although our method is primarily geared towards 6D object pose estimation, it can directly produce a 9D object pose when provided with a point cloud and segmentation mask. We first map the object point cloud to canonical space using its estimated 6D pose. Then, we refine the canonical point cloud using a standard outlier removal algorithm. Finally, we determine the 3D scales by computing the axis-aligned bounding box of the refined point cloud. The 9D object pose computation process is deployed in the real-world experiments found on our project page and in our supplementary video. We understand the importance of estimating 3D scales and are open to offering a more exhaustive evaluation if needed.
 
**(Q3: Alternative Aggregation Methods)** Additionally, some reviewers have suggested alternative aggregation methods like weighted mean-pooling (**7PCs**, **W8Ay**, **sAr7**). We concede that there might be other aggregation techniques superior to simply averaging. Nonetheless, the primary concern in this work is dealing with outliers. The presence of outliers invariably skews aggregated results, regardless of the chosen method, be it K-means or Mean Pooling. While some research addresses the multi-hypothesis challenge, they overlook this foundational problem. To our best knowledge, this is the first work that leverages energy-based diffusion model to remove outliers, which is also the key technical novelty of this work.

To address reviewers' concerns, we conducted experiments to compare mean-pooling with other aggregation methods (i.e., likelihood weighting). As shown in **Table 1** of the PDF, mean-pooling consistently outperforms weighted averaging. We hypothesize that the energy model is better at distinguishing poses with significant differences in errors but performs poorly when distinguishing poses with low errors. As a result, some poses with higher errors might overshadow those with lower errors in energy weighting. To validate this hypothesis, we further explored the relationship between the pose error and the energy output. Results in **Figure 1** of the PDF demonstrate that the energy model assigns similar (right, <2 cm), or even incorrect values (left, <10 degrees) for poses with low errors. Meanwhile, the energies of low-error poses (e.g., <10 degrees, <2 cm) are higher than those of high-error poses (e.g., >20 degrees, >4 cm).

We sincerely hope our work contributes to the Machine Learning + 3D Vision research community. Below we reply to reviewers’ questions point-by-point. Thanks again for your valuable comments and suggestions!


Meta-review: This paper proposes a method to apply diffusion models to category-level object pose estimation. Experimental results show that the proposed method outperforms state-of-the-art methods. All reviewers acknowledged the novelty, clarity, and promising experimental results. In the weaknesses section, several concerns were raised regarding inference time and limitations of object size estimation, which were addressed in the authors' rebuttal. As a result, several reviewers raised their final scores, and we reached a consensus to accept the paper. It is strongly recommended that the final version of the paper include the discussion and experimental results provided in the rebuttal.
Decision: Accept (poster)