SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 11: Shape mAP result of safe category.
Fscore@80%	Fscore@90%	CD1@5%	CD1@1%	IoU@25%	IoU@50%
A-SDF-GT [38]	83.86	82.17	84.09	72.80	83.75	41.31
A-SDF-GT-2 [38]	83.86	82.05	84.09	71.90	83.52	40.86
Ours-BG	89.01	87.28	90.64	79.68	83.28	15.60
Ours	90.62	87.31	92.51	78.53	82.25	29.65
Table 12: Shape mAP result of oven category.
Fscore@80%	Fscore@90%	CD1@5%	CD1@1%	IoU@25%	IoU@50%
A-SDF-GT [38]	37.43	20.14	53.24	17.52	15.24	0.68
A-SDF-GT-2 [38]	38.23	21.05	53.58	19.34	18.66	0.57
Ours-BG	57.44	52.97	57.09	42.59	29.17	0.17
Ours	58.66	54.60	59.12	40.87	30.76	1.28
Table 13: Shape mAP result of storage category.
Fscore@80%	Fscore@90%	CD1@5%	CD1@1%	IoU@25%	IoU@50%
A-SDF-GT [38]	57.68	33.22	79.75	25.26	19.23	0.00
A-SDF-GT-2 [38]	57.57	32.08	81.11	25.48	16.38	0.00
Ours-BG	73.30	51.98	79.97	29.69	5.57	0.00
Ours	73.67	47.16	78.91	28.74	14.61	0.00
Table 14: Shape mAP result of table category.
Fscore@80%	Fscore@90%	CD1@5%	CD1@1%	IoU@25%	IoU@50%
A-SDF-GT [38]	84.62	63.59	89.74	59.23	72.69	12.95
A-SDF-GT-2 [38]	86.03	71.28	90.13	65.51	77.05	8.33
Ours-BG	64.42	63.02	65.78	63.00	59.50	4.94
Ours	65.81	64.05	69.12	64.07	57.62	5.40
Table 15: Shape mAP result of microwave category.
Fscore@80%	Fscore@90%	CD1@5%	CD1@1%	IoU@25%	IoU@50%
A-SDF-GT [38]	63.88	39.76	72.97	29.70	7.15	0.00
A-SDF-GT-2 [38]	71.03	52.85	75.88	44.48	13.82	0.12
Ours-BG	73.19	68.83	73.64	62.41	20.88	0.09
Ours	70.66	65.69	74.59	56.36	19.92	0.23
Table 16: Shape mAP result of refrigerator category.
Fscore@80%	Fscore@90%	CD1@5%	CD1@1%	IoU@25%	IoU@50%
A-SDF-GT [38]	65.85	41.10	77.38	24.41	77.72	48.04
A-SDF-GT-2 [38]	67.64	41.10	75.48	22.84	79.06	48.38
Ours-BG	86.40	78.12	87.43	45.74	87.22	73.74
Ours	90.30	83.25	92.47	42.96	92.29	73.16
Table 17: Shape mAP result of washing machine category.
Fscore@80%	Fscore@90%	CD1@5%	CD1@1%	IoU@25%	IoU@50%
A-SDF-GT [38]	53.96	43.57	69.89	43.80	10.86	0.00
A-SDF-GT-2 [38]	74.38	62.93	80.05	55.96	11.10	0.24
Ours-BG	77.10	74.12	78.47	67.02	10.00	0.00
Ours	74.74	69.19	77.57	62.03	8.96	0.00
I Effect of the refiner on kinematic estimation
The effect of the refiner  $\mathcal{R}$  on kinematic estimation is shown in Fig. 14. We conduct evaluations on joint parameter estimation. These evaluations focus on the intersection of true positive detections from both the model with and without the refiner with various mAP thresholds. Except for the joint axis orientation error at mAP@90%, the refiner improves the joint parameter estimation.
J Additional qualitative results on synthetic dataset
We show additional qualitative results on SAPIEN [58] dataset in Fig. 15.
K Additional qualitative results on real-world data
Additional qualitative results on real-world data are shown in Fig. 16. For the washing machine example in the top left corner of the figure, a semi-automatic segmentation tool on Microsoft PowerPoint was used for foreground segmentation by specifying the rough foreground region, as https://remove.bg could not properly segment the foreground in this example.
L Qualitative results from novel viewpoint
Corresponding to Fig. 5 and Fig. 7 in 4, we visualize the qualitative results from novel viewpoint on SAPIEN [58] dataset and BMVC [36] dataset in Fig. 17 and Fig. 18, respectively.
M Failure cases and limitations
Uncertainty in closed parts One failure case arises when our model attempts to estimate the orientation of a part opening while it is in its closed state, as illustrated in Fig. 19 (a). We show quantitative evaluation on axis direction error for closed parts in Table 20. We regard a part as closed when its ground truth motion amount is less than a threshold. For revolute joints, we use  $5^{\circ}$  and  $10^{\circ}$  as thresholds, and  $5\%$  and  $10\%$  of the fully opened amount for prismatic joint. As seen in the table, the model successfully estimates reasonable joint direction for prismatic parts, while for revolute
Table 18: Shape mAP result of box category.
Fscore@80%	Fscore@90%	CD1@5%	CD1@1%	IoU@25%	IoU@50%	Params.
ND=16	73.25	68.27	74.76	58.64	40.74	10.57	14.33M
ND=8	71.88	65.38	73.68	54.79	35.31	8.20	9.05M
ND=6, NR=2	74.22	68.80	75.71	58.61	40.06	9.80	10.25M
Table 19: Ablation on the refiner  $\mathcal{R}$  for shape mAP and the number of parameters.
Figure 14: Effect of the refiner to joint parameter estimation.
<5°/5%	<10°/10%
Revolute	Prismatic	Average	Revolute	Prismatic	Average
Ours	25.96°(3.15°)	2.04°(2.60°)	24.46°(3.08°)	18.09°(2.80°)	2.37°(2.60°)	16.79°(2.78°)
Ours-BG	30.56°(3.39°)	3.36°(2.42°)	28.85°(3.27°)	22.27°(2.90°)	2.56°(2.43°)	20.64°(2.85°)
Table 20: Evaluation of joint axis direction error for closed parts. The part is regarded as closed if its ground truth motion amount is less than the threshold. For the prismatic part, we normalize the motion amount by its fully opened amount for thresholding to consider the different scales of parts. For reference, we denote the corresponding error of open parts in the parenthesis.
parts, the error is larger. Two contributing factors are: (1) our model does not explicitly model the uncertainty, particularly in ambiguous cases like the joint axis direction shown in Fig. 19 (a). The prismatic part's joint direction has less diversity due to the physical constraint by the base part, while several joint directions are possible for the revolute joint given the base part pose, thus the opening direction is ambiguous to the model. (2) The model could not detect small geometric cues from noisy depth maps, such as handles or knobs, which can provide valuable information about opening directions. These cases can be improved in several ways. For instance, we could extend the model to treat joint parameter ambiguity as a probability distribution, providing multiple joint parameter candidates sampled from the estimated distribution. This approach can still be practical, limiting the
Figure 15: Additional qualitative results on SAPIEN [58] dataset.
kinematics parameter space to a few options for initial interaction steps in robotics applications or allowing humans to choose the best prediction in AR/VR applications during a human-in-the-loop process. Previous work [1, 11] has shown the effectiveness of this probability-based approach. Another approach would be to integrate 2D and 3D features into the detection backbone [54, 26, 7]. Incorporating 2D features would enhance the encoder's ability to capture small yet informative features, like handles or knobs, that may be missed by a sparse point cloud representation alone.
Unseen categories Another failure case is that our model struggles to reconstruct instances from unseen categories, particularly those with shapes and part structures with small samples or absent in training data. Fig. 19 (b) shows the failure case with a dehumidifier as an unseen category, which has an unseen drawer shape and part structure; the base part only has one bottom drawer. This limitation can be improved by incorporating more data from recent datasets, such as [13, 34], which offers scanned real-world objects of various shapes, sizes, and part structures. However, it's important to note that generalization to unseen categories is beyond the scope of this paper. Moreover, in terms of part structure, it's also important to note that previous works can only handle very limited part structures, such as one base part and one articulated part [16] and predefined part count per category [38]. In contrast, our model can handle various part counts with a single model.
False positives and negatives Due to the detection-based nature of our model and also the domain gap to real-world data, false negatives or positives still occur, as shown in Fig. 19 (c) and (d). One practical improvement would be including real-world data [13, 34] in training to minimize the domain gap, hence boosting the model's generalization capabilities. Another approach for reducing false positives especially with physically implausible configurations, is to include regularization losses, such as physical visolation loss [61] for learning physical constraints. It's important to note that the kinematics-aware part fusion improves both false positives and negatives, as discussed in Sec. 4.3.
Objects with many joints The proposed approach works reasonably for complex articulated objects such as those with more than five joints, as shown in Fig. 20 (a) when all parts are visible from the given viewpoint. However, from certain viewpoints, we also observe that a single instance is reconstructed as two separate instances, each with fewer joints, as shown in the first row of Fig. 20
Current pose
Fully opened
Closed
Current pose
Depth
Figure 16: Additional qualitative results on real-world data.
Current pose
Fully opened
(b). We attribute this to the current dataset consisting of a small number of CAD models with  $>5$  joints, while and the majority of instances have fewer joints. Also, when some parts are only partially visible, our method tends to make inaccurate pose estimations for such parts, such as the right stacked three drawers indicated by the red box in the second row of Fig. 20 (b).
Latency of KPF module While the KPF module improves detection performance, query oversampling, part proposal fusing, and kIoU calculation add extra running time for inference. We visualize the latency and accuracy trade-off in Fig. 21. We change the number of query oversampling from one to ten, and measure the per image processing speed in seconds. We randomly sample 100 samples from the test split for the evaluation and latency is measured on a V100 GPU. Although detection performance in shape mAP improves with more query oversampling, the latency degrades. Note that the result of w/o KPF applies 3D cuboid NMS with GPU parallelization, while the KPF module is implemented on CPU without any parallelization. We believe the latency of the KPF module can be significantly improved by optimizing the implementation, such as utilizing GPU parallelization.
A-SDF-GT-2
Figure 17: Qualitative results on SAPIEN [58] from novel viewpoint.
Input
Figure 18: Qualitative results on BMVC [36] from novel viewpoint.
(a) Ambiguous opening direction
RGB
Depth
(c) False positive
Figure 19: Failure cases on real-world data.
(b) Unseen category
RGB
Depth
(d) False negative
(a) Successful cases of objects with  $>5+$  joints
GT
(b) Failure cases of objects with  $>5+$  joints
Figure 20: Successful and failure cases of articulated objects with more than five joints.
Figure 21: Latency and accuracy trade-off for different numbers of query oversampling (QO) in KPF module.