SlowGuess/NeurIPS_2023_OCR huggingface dataset:
No Ad-hoc Design: Unlike previous approaches, we do not incorporate any ad-hoc designs or tricks into the network architecture for symmetric objects. Both the score and energy models are implemented using commonly used feature extractors (e.g., PointNet++[39]) and feed-forward MLPs. Surprisingly, our method performs exceptionally well in handling symmetric objects (refer to Sec 4.4) and achieves state-of-the-art (SOTA) performance on existing benchmarks (refer to Sec 4.2).
Prior-Free: Our method eliminates the requirement of category-level canonical prior, freeing us from designing a shape-deformation module to incorporate the canonical prior. This also provides the potential to generalize our method to objects from unseen categories sharing similar symmetric properties. In Sec 4.4, we present experiments to demonstrate this potential.
Capable of Pose Tracking: Thanks to the closed-loop generation process of the diffusion models, we can adapt the single-frame pose estimation framework to pose tracking tasks by warm-starting from the previous predictions. The adapted object pose tracking framework differs from the single-frame estimation framework only in the candidates' sampling. For each frame that receives the point cloud observation  $O_{\mathrm{cur}}$ , we initialize the PF-ODE in Eq.4 with  $\pmb{p}(0.1) \sim \mathcal{N}(\hat{\pmb{p}}_{\mathrm{prev}}, \sigma^2(0.1)\mathbf{I})$ , where  $\hat{\pmb{p}}_{\mathrm{prev}} \in \mathrm{SE}(3)$  denotes the estimated pose of the previous frame. Subsequently, we solve this modified PF-ODE from  $t = 0.1$  to  $t = \epsilon$  using the gradient fields  $\Phi_\theta(\pmb{p}, t|O_{\mathrm{cur}})$  to sample candidates for pose tracking. By following the same aggregation procedure, we can obtain the pose estimation  $\hat{\pmb{p}}_{\mathrm{cur}}$  for the current frame. We summarise the tracking framework in Appendix D. In Sec.4.5, our adapted pose-tracking method outperforms the state-of-the-art object pose-tracking baseline in most metrics.
Datasets. Our method is trained and evaluated on two common category-level object pose estimation datasets, namely CAMERA and REAL275 [4], following the NOCS [4] convention for splitting the data into training and testing sets. These datasets include 6 daily objects: bottle, bowl, camera, can, laptop, and mug. CAMERA dataset is a synthetic dataset generated using mixed-reality methods. It comprises real background images with synthetic rendered foreground objects and consists of 275K training images and 25K test images. REAL275 dataset is a real-world dataset that employs the same 6 object categories as the CAMERA dataset. It includes 3 unique instances per category in both the training and test sets and consists of 7 scenes for training with 4.3K images and 6 scenes for testing with 2.75K images. Each scene in the REAL275 dataset contains more than 5 objects.
Metrics. Following NOCS [4], we report the mean Average Precision (mAP) in  $n^{\circ}$  and  $m$  cm to evaluate the accuracy of object pose estimation. Here,  $n$  and  $m$  denote the prediction error of rotation and translation, respectively, where the predicted rotation error is less than  $n^{\circ}$  and the predicted translation error is less than  $m$  cm. Specifically, we use  $5^{\circ}2cm$ ,  $5^{\circ}5cm$ ,  $10^{\circ}2cm$ , and  $10^{\circ}5cm$  as our evaluation metrics. Similar to NOCS, for symmetric objects (bottles, bowls, and cans), we ignore the
Table 1: Quantitative comparison of category-level object pose estimation on REAL275 dataset. We summarize the results reported in the original paper for the baseline method.  $\uparrow$  represents a higher value indicating better performance, while  $\downarrow$  represents a lower value indicating better performance. Data refers to the format of the input data used by the method, and Prior indicates whether the method requires category prior information.  $-$ " indicates that the metrics are not reported in the original paper. K represents the number of hypotheses.
Method	Data	Prior	5°2cm↑	5°5cm↑	10°2cm↑	10°5cm↑	Parameters(M)↓
Deterministic	NOCS [4]	RGB-D	×	-	9.5	13.8	26.7	-
CASS [17]	RGB-D	×	19.5	23.5	50.8	58.0	47.2
DualPoseNet [9]	RGB-D	×	29.3	35.9	50.0	66.8	67.9
SPD [7]	RGB-D	✓	19.3	21.4	43.2	54.1	18.3
CR-Net [20]	RGB-D	✓	27.8	34.3	47.2	60.8	-
SGPA [5]	RGB-D	✓	35.9	39.6	61.3	70.7	-
DPDN [8]	RGB-D	✓	46.0	50.7	70.4	78.4	-
FS-Net [6]	D	×	19.9	33.9	-	69.1	41.2
GPV-Pose [11]	D	×	32.0	42.9	55.0	73.3	-
SAR-Net [10]	D	✓	31.6	42.3	50.4	68.3	6.3
SSP-Pose [18]	D	✓	34.7	44.6	-	77.8	-
RBP-Pose [19]	D	✓	38.2	48.1	63.1	79.2	-
Probabilistic	Ours	D	×	52.1	60.9	72.4	84.0	4.4
Ours(K=10)	D	×	71.5	75.9	85.2	90.8	2.2
Ours(K=50)	D	×	82.0	84.5	92.8	95.9	2.2
rotation error around the symmetry axis. For the "mug" category, we consider it a symmetric object when the handle is not visible and a non-symmetric object when the handle is visible.
Implementation Details. For a fair comparison, we employed the instance mask generated by MaskRCNN [43] during the inference phase, which was identical to the one used in previous work. Both the Energy-based diffusion model for pose ranking and the score-based diffusion model for pose generation shared the same network structure. The input pointcloud consisted of 1024 points. During the training phase, we used the same data augmentation techniques as FS-Net [6], which are widely adopted in category-level object pose estimation tasks. All experiments were conducted on a single RTX3090 with a batch size of 192. All the experiments are implemented using PyTorch [44].
4.2 Comparison with State-of-the-Art Methods
Table 1 provides a comprehensive comparison between our method and the state-of-the-art approaches on the REAL275 [4] dataset, highlighting a significant advancement in performance. In Table 1, "Ours" represents the aggregated pose using  $K$  as 50 and  $\delta$  as  ${60}\%$  . "Ours  $\left( {K = {10}}\right)$  " and "Ours  $\left( {K = {50}}\right)$  " indicate that we set  $K$  as 10 and 50,respectively,and select a pose from candidates with the minimum distance to ground truth pose for evaluation. We deal with all categories by a single model.
As shown in Table 1, our approach demonstrates a remarkable improvement over the current SOTA method, GPV-Pose[11], exceeding it by more than  $20\%$  in both the rigorous  $5^{\circ}2cm$  and  $5^{\circ}5cm$  evaluation metrics. Significantly, for the first time, we achieve results on the REAL275 dataset, surpassing the impressive thresholds of  $50\%$  and  $60\%$  in the  $5^{\circ}2cm$  and  $5^{\circ}5cm$  metrics, respectively. These exceptional outcomes further support the efficacy of our approach.
Even when compared to methods that take RGB-D data and category prior as input, our method maintains a notable advantage. Without relying on any additional information, our method achieves a performance boost of over  $10\%$  in the metric of  $5^{\circ}5cm$ , surpassing the SOTA method, DPDN [8].
In addition, the results that evaluate the pose nearest to the ground truth indicate that condition generative models exhibit tremendous potential for category-level object pose estimation. Another crucial metric is the number of learnable parameters of the network, which significantly impacts the feasibility of model deployment. Table 1 reveals that our method achieves a remarkable accuracy for predicting poses with the fewest network parameters, further improving its deployability.
Number of Pose Candidates and Proportion of Selected Poses. Using the  $10^{\circ}2cm$  metric, Table 2 demonstrates the impact of two factors on performance: the number of generated pose candidates  $M$ , and the proportion  $\delta$  of selected poses out of the total pose candidates.
The results show a notable improvement when  $M$  increases from 10 to 50. This enhancement can be attributed to the increasing number of sampling times, resulting in a pose candidate set that aligns more closely with the predicted distribution. However, the improvement becomes marginal when  $M$  is further increased from 50 to 100. This can be attributed to the predictions that approach the upper limit of the
Table 2: Ablation on number of pose candidates  $K$  and proportion of selected poses  $\delta$ .
δ
K	20%	40%	60%	80%	100%
10	65.3	67.7	67.9	67.7	65.0
50	70.2	71.8	72.4	71.9	69.7
100	70.8	72.2	72.6	72.5	70.2
aggregation approach. After considering the trade-off between performance and overhead, we ultimately decided to adopt  $K = 50$ . In view of the inherent challenge in training an energy model that precisely aligns with the actual distribution, it becomes apparent that employing a small delta value, as demonstrated in Table 2, does not yield an optimal outcome. Consequently, we regard the energy model primarily as a detector for outliers. The most successful results of this paper were achieved using a delta value of  $60\%$ , which effectively mitigated the presence of relatively insignificant noise originating from the sampling process.
Effectiveness of the Energy-based Likelihood Estimator  $\Psi_{\phi}$ . In this part, we report the results of three distinct ranking methodologies, namely "Random", "Energy", and "GT", employed to sort pose candidates  $\{\hat{p}_i\}_{i=1}^K$ . "Random" applies a stochastic shuffling to  $\{\hat{p}_i\}_{i=1}^K$ , providing a baseline approach. "Energy" uses our trained energy model to rank  $\{\hat{p}_i\}_{i=1}^K$ . "GT" ranks  $\{\hat{p}_i\}_{i=1}^K$  in ascending order of the distances to the ground truth pose, serving as an upper limit for all ranking methods.
The absence of mean pooling in Table 3 implies that one pose candidate is chosen randomly for evaluation. We observe that utilizing an energy-based likelihood estimator as a sampler enhances the sort
Table 3: Effectiveness of the energy-based likelihood estimator  ${\mathbf{\Psi }}_{\phi }$  .
Ranking	Mean pooling	5°2cm	5°5cm	10°2cm	10°5cm
Random	×	13.5	49.1	21.5	76.3
Random	✓	49.4	58.6	68.5	80.7
Energy	✓	52.1	60.9	72.4	84.0
GT(upper bound)	✓	62.1	66.8	80.9	86.9
ing of pose candidates and effectively eliminates outliers. This improvement is significant compared to random sampling. Although doesn't reach the upper bound, we offer a novel and practical approach for ranking and aggregating poses sampled from estimated distribution and demonstrate its potential.
To better understand the energy model's impact, we further investigated the correlation between pose error and energy output. As depicted in Figure 3, there is a general negative correlation between the output and the error of the energy model. However, the energy model excels at distinguishing poses with significant error differences but performs poorly when distinguishing poses with low errors. For instance, the energy model assigns similar values (e.g., translation,  $<2cm$ ) or even incorrect values (e.g., rotation,  $<10^{\circ}$ ) for poses with low errors. Additionally, the energies associated with low-error poses (e.g.,  $<10^{\circ}, <2cm$ ) are higher than those of high-error poses (e.g.,  $>20^{\circ}, >4cm$ ).
Figure 3: Energy-Error Correlation Analysis. Points for raw scatters plot, curves for mean error.
bottle (symmetric)
laptop (asymmetric)
Figure 4: The predicted conditional rotation distribution of symmetric and asymmetric objects. The left figure is an example from the REAL275 dataset, while the right figure shows the distribution of the generated rotations for the symmetrical bottle and the asymmetric laptop. The rotation distribution is visualized by plotting yaw as latitude, pitch as longitude, and roll as the color, following the approach inspired by [45; 46]. The ground truth is represented by the center of the open circle, and the dot shows the result of generating rotations 50 times.
SO(3) distribution
4.4 Analysis on Symmetric Objects
Performance. Our approach outperforms the state-of-the-art method RBP-Pose [19], especially for symmetrical objects, as shown in Table 4. Importantly, our method does not rely on specific network structures or loss designs for these categories. Notably, for 'mug' category, where asymmetry is present when the handle is visible and symmetry
when it is not, our method effectively addresses this challenge and achieves significant performance improvements compared to previous methods. To visually depict the disparity between symmetric and asymmetric objects, Figure 4 visually demonstrates the accuracy of our method in predicting rotation distributions for both symmetric and asymmetric objects.
Toward Cross-category Object Pose Estimation. Our method demonstrates a surprisingly cross-category generalizability, as it directly predicts the distribution of object poses, eliminating the reliance on object category prior. Table 5 verifies the remarkable ability of our approach to generalize across categories sharing similar symmetric properties.
Our approach demonstrates consistent and robust performance when tested on unseen categories with similar symmetrical attributes, whereas the baseline performance exhibits a significant decline. We hypothesize that the specific out-of-distribution (OOD) generalization ability of our method arises from the learned feature space of the point cloud. For example, although the bowl
category is considered OOD, the extracted features from point clouds in the bowl category may exhibit similarities or closeness to seen categories, to some extent. In other words, a bowl may share visual characteristics with items such as cans, bottles, or mugs, as identified by the PointNet++ of the
Table 4: Per-category results of our method and RBP-Pose.
Category	Symmetric	5°2cm	5°5cm
RBP-Pose	Ours	RBP-Pose	Ours
camera	1.3	2.9	1.5	3.2
laptop	×	41.3	63.4	75.2	91.4
average	21.3	32.2(11.9↑)	38.4	47.3(8.9↑)
bottle	38.7	52.6	43.5	60.9
bowl	75.4	85.4	81.7	92.6
can	✓	53.5	72.5	67.1	80.4
average	55.9	70.2(14.3↑)	64.1	78.0(13.9↑)
mug	-	18.9	35.7(16.8↑)	19.4	36.4(17.0↑)
Table 5: Cross category evaluation on REAL275. The left and right sides of  $\prime$  ’ respectively indicate the performance when the testing category is seen and unseen in training data.
Category	Method	5°2cm	5°5cm	10°2cm	10°5cm
bowl	SAR-Net[10]	58.1/36.4	66.0/47.3	83.7/59.4	93.6/81.5
RBP-Pose[19]	75.4/0.0	81.7/6.9	92.1/0.1	100.0/30.7
Ours	85.4/64.5	92.6/72.5	93.1/87.2	100.0/98.6
bottle	SAR-Net[10]	43.5/11.7	54.0/23.0	61.3/33.6	79.8/68.0
RBP-Pose[19]	38.7/4.3	43.5/5.8	76.4/24.7	89.8/29.7
Ours	52.6/39.0	60.9/53.2	81.4/73.6	92.7/94.6
can	SAR-Net[10]	32.2/7.3	62.2/52.3	52.5/12.1	92.9/87.9
RBP-Pose[19]	53.5/0.8	67.1/21.0	78.8/2.6	96.3/61.7
Ours	72.5/62.5	80.4/74.0	88.8/81.6	99.8/99.7
Figure 5: Visualization of the features. 'w/o' denotes the unseen category during training process.
ScoreNet. To validate this hypothesis, we conducted a t-SNE [47] analysis on the point cloud feature space of the ScoreNet. Specifically, we extracted features from the point clouds of objects in the seen and unseen test set and visualized the t-SNE results. As shown in Figure 5, the results demonstrate that features from cans and bottles tend to intermingle, aligning with the accurate observation that both cans and bottles exhibit symmetrical cylindrical shapes. Meanwhile, features from the bowl category show proximity to features from mugs.
4.5 Category-level Object Pose Tracking
Table 6 shows the performance of category-level object pose tracking of our tracking method and baselines on REAL275 datasets. For the first frame, we utilized a perturbed ground truth pose as the initial object pose, following CAPTRA [48] and CATRE [49]. Despite being directly transferred from a single-image prediction method, our approach has achieved performance comparable to the state-of-the-art method, CATRE. Remarkably, we significantly outperform CATRE in the metrics of  $5^{\circ}5cm$  and mean rotation error. This highlights the expansibility of our pose estimation method for more nuanced downstream tasks. Despite our lower FPS compared to CATRE, our method still adequately fulfills usage requirements, particularly for robot operations.
Table 6: Results of category-level object pose tracking on REAL275. The results are averaged over all 6 categories. The best performance is in bold and the second best is underscored.
Method	Oracle ICP[48]	6-PACK[50]	iCaps[51]	CAPTRA[48]	CATRE[49]	Ours
Input Init.	RGBD GT.	RGBD GT. Pert.	RGBD Det. and Seg.	D GT. Pert.	D GT. Pert.	D GT. Pert.
Speed(FPS)↑	-	10	1.84	12.66	89.21	17.18
5°5cm↑	0.7	33.3	31.6	62.2	57.2	71.5
Rerr(°)↓	40.3	16.0	9.5	5.9	6.8	4.2
terr(cm)↓	7.7	3.5	2.3	7.9	1.2	1.5
In this work, we study a fundamental problem, namely the multi-hypothesis issue, that has existed in category-level object pose estimation for a long time. To overcome this issue, we propose a novel object pose estimation framework that initially generates the pose candidates via a score-based diffusion model. To aggregate the candidates, we leverage an energy-based diffusion model to filter out the outliers, which avoids the costly integration process of the score-based model. Our framework achieves exceptionally state-of-the-art performance on existing benchmarks, especially on symmetric objects. In addition, our framework is capable of object pose tracking and can generalize to objects from novel categories sharing similar symmetric properties.
Limitations and Future Works: Although our framework achieves real-time object pose tracking at approximately 17 FPS, the efficiency of single-frame object pose estimation is still limited by the costly sampling process of the score-based model. In the future, we may leverage the recent advances in accelerating the sampling process of the diffusion models to speed up the inference [52; 53]. In addition to passively filtering out the outliers, we may incorporate reinforcement learning to train an agent that actively increases the likelihood of the estimated pose [54].
Boarder Impact: This work opens the door to leveraging energy-based diffusion models to improve the generation results, which might facilitate more research in border communities.
This work is supported by the National Natural Science Foundation of China - General Program (Project ID: 62376006), National Youth Talent Support Program (Project ID: 8200800081), and Beijing Municipal Science & Technology Commission (Project ID: Z221100003422004).
[1] Eric Marchand, Hideaki Uchiyama, and Fabien Spindler. Pose estimation for augmented reality: a hands-on survey. IEEE transactions on visualization and computer graphics, 22(12):2633-2651, 2015.
[2] Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl, and Dieter Fox. Self-supervised 6d object pose estimation for robot manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3665-3671. IEEE, 2020.
[3] Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birchfield. Deep object pose estimation for semantic robotic grasping of household objects. arXiv preprint arXiv:1809.10790, 2018.
[4] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2642-2651, 2019.
[5] Kai Chen and Qi Dou. Sgpa: Structure-guided prior adaptation for category-level 6d object pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2773-2782, 2021.
[6] Wei Chen, Xi Jia, Hyung Jin Chang, Jinming Duan, Linlin Shen, and Ales Leonardis. Fs-net: Fast shape-based network for category-level 6d object pose estimation with decoupled rotation mechanism. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1581-1590, 2021.
[7] Meng Tian, Marcelo H Ang, and Gim Hee Lee. Shape prior deformation for categorical 6d object pose and size estimation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 530-546. Springer, 2020.
[8] Jiehong Lin, Zewei Wei, Changxing Ding, and Kui Jia. Category-level 6d object pose and size estimation using self-supervised deep prior deformation networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 19-34. Springer, 2022.
[9] Jiehong Lin, Zewei Wei, Zhihao Li, Songcen Xu, Kui Jia, and Yuanqing Li. Dualposenet: Category-level 6d object pose and size estimation using dual pose network with refined learning of pose consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3560-3569, 2021.
[10] Haitao Lin, Zichang Liu, Chilam Cheang, Yanwei Fu, Guodong Guo, and Xiangyang Xue. Sar-net: shape alignment and recovery network for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707-6717, 2022.
[11] Yan Di, Ruida Zhang, Zhiqiang Lou, Fabian Manhardt, Xiangyang Ji, Nassir Navab, and Federico Tombari. Gpv-posed: Category-level object pose estimation via geometry-guided pointwise voting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6781-6791, 2022.
[12] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
[46] Kieran Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Implicit-pdf: Non-parametric representation of probability distributions on the rotation manifold. arXiv preprint arXiv:2106.05965, 2021.
[47] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[48] Yijia Weng, He Wang, Qiang Zhou, Yuzhe Qin, Yueqi Duan, Qingnan Fan, Baoquan Chen, Hao Su, and Leonidas J Guibas. Captra: Category-level pose tracking for rigid and articulated objects from point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13209-13218, 2021.
[49] Xingyu Liu, Gu Wang, Yi Li, and Xiangyang Ji. Catre: Iterative point clouds alignment for category-level object pose refinement. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part II, pages 499-516. Springer, 2022.
[50] Chen Wang, Roberto Martin-Martin, Danfei Xu, Jun Lv, Cewu Lu, Li Fei-Fei, Silvio Savarese, and Yuke Zhu. 6-pack: Category-level 6d pose tracker with anchor-based keypoints. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10059–10066. IEEE, 2020.
[51] Xinke Deng, Junyi Geng, Timothy Bretl, Yu Xiang, and Dieter Fox. ics: Iterative categorylevel object pose and shape estimation. IEEE Robotics and Automation Letters, 7(2):1784-1791, 2022.
[52] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
[53] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.
[54] Hai Ci, Mickel Liu, Xuehai Pan, Fangwei Zhong, and Yizhou Wang. Proactive multi-camera collaboration for 3d human pose estimation. arXiv preprint arXiv:2303.03767, 2023.
Architecture of the Score Network The detailed architecture of the score network  $\Phi_{\theta}$  is illustrated in Figure 6. We utilize PointNet++ [39] to extract the global geometry feature  $\mathcal{F}_O$  of the partially observed point cloud  $O^*$ . And the sampled pose  $\pmb{p}$  and timestep  $t$  features are embedded as  $\mathcal{F}_{\pmb{p}}$  and  $\mathcal{F}_t$ , respectively, using a Multi-Layer Perceptron (MLP). Then  $\mathcal{F}_O$ ,  $\mathcal{F}_{\pmb{p}}$  and  $\mathcal{F}_t$  are concatenated to obtain the global feature  $\mathcal{F}$ , and three parallel branches are employed to predict the scores of  $R_x$ ,  $R_y$ , and  $T$  individually, where  $[R_x|R_y]\in \mathbb{R}^6$  and  $T\in \mathbb{R}^3$  denote rotation and translation vectors, respectively. And  $[R_x|R_y]$  is a continuous rotation representation proposed by [40] to address the discontinuity of quaternions and Euler angles in Euclidean space. As introduced in [40], the mapping from SO(3) to the 6D representation of rotation is: