SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
g _ {G S} \left(\left[ \begin{array}{l l l} \mathbf {a} _ {1} & \mathbf {a} _ {2} & \mathbf {a} _ {3} \end{array} \right]\right) = \left[ \begin{array}{l l} \mathbf {a} _ {1} & \mathbf {a} _ {2} \end{array} \right] \tag {10}
$$
The mapping form the 6D representation to SO(3) is:
$$
f _ {G S} \left(\left[ \begin{array}{l l} \mathbf {a} _ {1} & \mathbf {a} _ {2} \end{array} \right]\right) = \left[ \begin{array}{l l l} \mathbf {b} _ {1} & \mathbf {b} _ {2} & \mathbf {b} _ {3} \end{array} \right] \tag {11}
$$
$$
b _ {i} = \left[ \left\{ \begin{array}{l l} N (\mathbf {a} _ {\mathbf {1}}) & \text {i f} i = 1 \\ N (\mathbf {a} _ {\mathbf {2}} - (\mathbf {b} _ {\mathbf {1}} \cdot \mathbf {a} _ {\mathbf {2}}) \mathbf {b} _ {\mathbf {1}}) & \text {i f} i = 2 \\ \mathbf {b} _ {\mathbf {1}} \times \mathbf {b} _ {\mathbf {2}} & \text {i f} i = 3 \end{array} \right] \right. \tag {12}
$$
Here  $N(\cdot)$  denotes a normalization function.
Architecture of the Energy Network The energy network  $\Psi_{\phi}$  shares exactly the same architecture with the score network  $\Phi_{\theta}$ . The inputs are first fed into  $\Phi_{\phi}$  to obtain a score-shaped vector  $\Phi_{\phi}(\pmb{p}, t|O) \in \mathbb{R}^{|\mathcal{P}|}$ . Then, the output energy is calculated by the dot product between the input pose and the score-shaped vector  $\Psi_{\phi}(\pmb{p}, t|O) = \langle \pmb{p}, \Phi_{\phi}(\pmb{p}, t|O) \rangle \in \mathbb{R}^1$ .
Figure 6: Architecture of the score network  $\Phi_{\theta}$ .  $p$  denotes sampled 6D object poses.  $O^{*}$  denotes partially observed 3D point cloud condition.  $t$  denotes timestep.  $\oplus$  denotes the concatenation operator.
B Qualitative Comparison on REAL275
Figure 2 illustrates the qualitative comparison results between our method and RBP-Pose [19] on the REAL275 dataset. The images are accompanied by red boxes highlighting objects that exhibit noticeable differences in the predicted results. Additionally, the bottom-right corner of each image provides an enlarged view of the highlighted object, showing the ground truth pose as well as the poses estimated by RBP-Pose and our approach. Our method demonstrates a significant performance improvement compared to RBP-Pose, particularly in the case of objects such as mugs. Notably, in the fourth row of the figure, it can be observed that our method achieves highly accurate poses even when only a small portion of the mug handle is visible. This success can be attributed to the fact that, during the training process, a unique pose exists when the mug handle is visible. However, when the mug handle becomes occluded, a multi-hypothesis problem arises, which our generative formulation effectively handles.
C More Results and Analysis
C.1 Per-category Results
Figure 8 demonstrates a quantitative comparison between our method and the state-of-the-art depth-based approach, RBP-Pose [19], for various object categories at different thresholds. The results
RBP-Pose
Ours
Figure 7: Qualitative comparison with RBP-Pose [19] on REAL275. The left column represents the ground truth pose, the middle column represents the results of RBP-Pose, the right column represents the results of our approach.
clearly indicate that our method outperforms RBP-Pose in all metrics, despite the fact that we do not incorporate augmentation specifically designed for symmetric objects during the training phase, unlike RBP-Pose. Our approach exhibits significant improvements, particularly in regions with stringent threshold requirements. This emphasizes the superior performance of our generative category-level object 6D pose estimation approach in effectively addressing the multi-hypothesis challenges posed by symmetric objects and partial observations, thereby enabling its successful application in robot manipulation tasks demanding precise object pose prediction.(e.g., pouring liquids.)
C.2 Results on CAMERA
Table 7 illustrates a quantitative comparison between our method and the baselines on the CAMERA [4] dataset. The results clearly demonstrate the remarkable performance enhancement achieved by our method. When compared to approaches that rely solely on depth data as network input, as well as those that utilize RGB-D and shape priors as network input, our method consistently outperforms them, surpassing the current state-of-the-art performance. Notably, our method exhibits a particularly pronounced advantage when stricter accuracy requirements are imposed, such as the  $5^{\circ}2cm$  metric. In this case, our method outperforms the current SOTA method, RBP-Pose, by an impressive margin of  $6.4\%$ . This significant improvement highlights the efficacy of our approach.
Figure 8: Per-category quantitative comparison with RBP-Pose [19] on REAL275. The left represents the results of RBP-Pose, while the right represents the results of our approach.
Table 7: Quantitative comparison of category-level object pose estimation on CAMERA [4] dataset. We summarize the results reported in the original paper for the baseline method.  $\uparrow$  represents a higher value indicating better performance, while  $\downarrow$  represents a lower value indicating better performance. Data refers to the format of the input data used by the method, and Prior indicates whether the method requires category prior information.  $-\cdot$  indicates that the metrics are not reported in the original paper. K represents the number of hypotheses."
Method	Data	Prior	5°2cm↑	5°5cm↑	10°2cm↑	10°5cm↑	Parameters(M)↓
Deterministic	NOCS [4]	RGB-D	×	32.3	40.9	48.2	64.6	-
DualPoseNet [9]	RGB-D	×	64.7	70.7	77.2	84.7	67.9
SPD [7]	RGB-D	✓	54.3	59.0	73.3	81.5	18.3
CR-Net [20]	RGB-D	✓	72.0	76.4	81.0	87.7	-
SGPA [5]	RGB-D	✓	70.7	74.5	82.7	88.4	-
GPV-Pose [11]	D	×	72.1	79.1	-	89.0	-
SAR-Net [10]	D	✓	66.7	70.9	75.3	80.3	6.3
SSP-Pose [18]	D	✓	64.7	75.5	-	87.4	-
RBP-Pose [19]	D	✓	73.5	79.6	82.1	89.5	-
Probabilistic	Ours	D	×	79.9	84.4	84.6	89.6	4.4
Ours(K=10)	D	×	90.8	93.0	93.4	95.7	2.2
Ours(K=50)	D	×	95.5	96.4	97.2	98.2	2.2
C.3 Real World Experiments
We have also successfully integrated our approach with robot manipulation capabilities, as demonstrated through various experiments conducted with the UFACTORY xArm6 equipped with RealSense D435. The demonstrations can be found in the supplementary video or on the project website. As shown in Figure 9, we illustrate the following three tasks:
Pouring Task. This task involves transferring the contents (e.g., water) from one container to another. The demonstration highlights the potential of combining our approach with heuristic strategies, enabling functional robot operations.
Stacking Task. In this task, we focused on piling up objects of the same category, like organizing scattered bowls on a tabletop. This demonstrates the precision of the estimated object pose, as accurate knowledge of object poses is crucial for completing this task.
Handover Task. This task involved either receiving objects from human hands to perform tasks or passing objects to person. The demonstration exemplified one form of human-robot interaction empowered by our method.
Figure 9: Pose estimation for robot manipulation tasks. We demonstrate three types of tasks.
D Details of Object Pose Tracking
Our pose estimation framework can be adapted to pose tracking with minor modifications. With the learned score network  $\Phi_{\theta}$ , the tracking algorithm is summarised as follows:
Algorithm 1 Our Object Pose Tracking Framework
1: Initialisation: Score network  $\Phi_{\theta}$ , energy network  $\Psi_{\phi}$ , initial point cloud  $O_0$ , the initial ground truth pose reference  $\pmb{p}_0$  and the tracking horizon  $T$ .
2: for  $t = 1$  to  $T$  do
3: Receive current observation  $O_{t}$
4:  $z_{1},z_{2},\ldots z_{K}\sim \mathcal{N}(\pmb{p}_{t - 1},0.1^{2}I)$  ▷ Sample initial poses around the previous prediction.
5: Sample candidates  $\{\hat{p}_i\}_{i=1}^K$  from the initialization  $(z_1, z_2, \ldots)$  via the following PF-ODE:
$$
\frac {d \boldsymbol {p}}{d t} = - \sigma (t) \dot {\sigma} (t) \Phi_ {\theta} (\boldsymbol {p}, t | O _ {t}) \tag {13}
$$
6:  $\hat{\pmb{p}}_{\tau_1} \succ \hat{\pmb{p}}_{\tau_2} \dots \succ \hat{\pmb{p}}_{\tau_K}$  where  $\hat{\pmb{p}}_{\tau_i} \succ \hat{\pmb{p}}_{\tau_j} \iff \Psi_\phi(\hat{\pmb{p}}_{\tau_i}, \epsilon|O) > \Psi_\phi(\hat{\pmb{p}}_{\tau_j}, \epsilon|O)$ $\triangleright$  Ranking
7: Estimate the current pose  $\pmb{p}_t = \left(\frac{\sum_{i=1}^M \hat{T}_{\tau_i}}{M}, \hat{\pmb{q}}_t\right)$  where  $M = \lfloor \delta \cdot K \rfloor$  and:
$$
\hat {\boldsymbol {q}} _ {t} = \underset {\boldsymbol {q} \in S O (3)} {\arg \max } \boldsymbol {q} ^ {T} \left(\frac {\sum_ {i = 1} ^ {M} A \left(\hat {\boldsymbol {q}} _ {\tau_ {i}}\right)}{M}\right) \boldsymbol {q}, A \left(\hat {\boldsymbol {q}} _ {\tau_ {i}}\right) = \hat {\boldsymbol {q}} _ {\tau_ {i}} \hat {\boldsymbol {q}} _ {\tau_ {i}} ^ {T} \tag {14}
$$
E Ethics Statement and Boarder Impact
Our method has the potential to develop the home-assisting robot, thus contributing to social welfare. We evaluate our method in synthesized or human-collected datasets, which may introduce data bias. However, similar studies also have such general concerns. We do not see any possible major harm in our study.