SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $G_{l|i} = \{j\in G_l\mid \delta_{li} = 1\}$  and  $\eta_{ij} = \| \mathbf{z}_i - \mathbf{z}_j\|$  denotes the L2 distance between the part-to-instance association embeddings of the  $i$ -th and  $j$ -th parts. The first term enforces the distance between the two embeddings of two different parts belonging to the same instance below  $\tau_{\mathbf{z}}^{\prime}$  and the second term ensures the distance between embeddings of parts belonging to different instances is larger than  $3\tau_{\mathbf{z}}^{\prime}$ . However, computing all combinations of triplets for the second term is operationally complex for part-wise supervision. To streamline this, we instead opt to maximize the difference between the upper bound and the lower bound of inter- and intra-instance distances of the embeddings. In practice, we replace max and min with their soft approximations defined as  $\mathrm{LogSumExp}(\eta_{ij})$  and  $-\mathrm{LogSumExp}(-\eta_{ij'})$  for smooth gradient propagation. During inference, we use a threshold  $\tau_{\mathbf{z}} = \frac{1}{2}\big(3\tau_{\mathbf{z}}' + \tau_{\mathbf{z}}'\big)$  to determine if two parts belong to the same instance based on the distance between their embeddings.
The total loss we minimize is  $\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{part}} + \mathcal{L}_{\mathrm{part}}^{\prime} + \mathcal{L}_{\mathrm{instance}}$ . At training time, we use the same part prediction MLPs to predict part parameters at every layer in the decoder. We compute the  $\mathcal{L}_{\mathrm{total}}$  for each layer independently and sum all the losses. We only use the part parameter predicted from the last decoder layer at test time. Full loss formulation can be found in the appendix.
3.8 Implementation detail
The implementation of the transformer encoder-decoder architecture and the optimizer configuration are based on the publicly available code of 3DETR [37]. We employ a masked encoder architecture for  $\mathcal{E}$  from [37]. The input point cloud to the encoder consists of  $N_{P} = 32768$  points, which are subsampled to  $N_{P^{\prime}} = 2048$  points. The number of queries is set to  $N_{q} = 128$  during training. At test time, we set it to  $N_{q} = 512$ , and with independent runs for query oversampling as  $N_{Q} = 10$ . The number of decoder layers for  $\mathcal{D}$  and  $\mathcal{R}$  are set to  $N_{\mathcal{D}} = 6$  and  $N_{\mathcal{R}} = 2$ , respectively. For the evaluation of synthetic data, the foreground masks are inferred by a pretrained segmentation model using a ResNext50 [59] encoder and a DeepLabV3Plus [6] segmentation head with RGBD input. To evaluate real-world data, we automatically generate foreground masks using https://remove.bg, following [56]. The weights for the matching cost  $\mathcal{C}_{\mathrm{match}}$  were set to  $\lambda_1 = 8$ ,  $\lambda_{2} = 10$ ,  $\lambda_{3} = 1$ ,  $\lambda_{4} = 5$ . We set  $\lambda_{\mathrm{intra}} = 0.1$  for the intra-instance loss term of the instance loss  $\mathcal{L}_{\mathrm{instance}}$ . We use the AdamW [31] optimizer with a base learning rate of 9e-4 and employ a cosine scheduler with nine epochs for warm-up. The training was performed using two A100 GPUs, each 40GB of GPU memory, and with a batch size of 26 for each GPU. The training took approximately two days. Please refer to the appendix for further details regarding the training process, hyperparameters, and network architecture.
Datasets We evaluate our method on both synthetic and real-world data. We use the SAPIEN [58] dataset for synthetic data evaluation, following recent works on articulated shape reconstruction [24, 56, 15]. We select ten categories with representative articulation types, a sufficient number of CAD models, and various part structures across categories. We then randomly construct room
A-SDF-GT-2
Category	Joint type	Part count	Instance count
Rev.	Pris.	Most freq.	Min	Max	Train	Test
Dishwasher	✓	✓	1	1	2	4997	1032
Trashcan	✓	✓	1	1	2	5037	1017
Safe	✓	1	1	1	4795	1029
Oven	✓	1	1	3	4951	1006
Storage	✓	✓	1	1	14	4907	973
Table	✓	✓	1	1	9	4790	999
Microwave	✓	✓	1	1	2	4832	934
Fridge	✓	2	1	3	5089	924
Washing	✓	1	1	1	5042	995
Box	✓	1	1	4	4823	979
Table 2: Overview of synthetic data from SAPIEN [58] dataset. Rev. and Pres. denote revolute and prismatic joints, respectively.
Figure 5: Qualitative results on SAPIEN [58] dataset.
Input
Ours
Figure 6: Visualization on real-world data.
geometry (wall and floor) and place one to four instances per scene. Each instance is generated by applying random horizontal flips, random anisotropic resizing of side lengths, and random articulation of CAD models. The camera pose is sampled randomly, covering the upper hemisphere of the scene. Instances with severe truncation from the view frustum or occlusion with other instances are ignored during training and evaluation, ensuring that least one instance is visible in a view. For the training split, we randomize the textures of parts and room meshes. We use the original textures from the SAPIEN dataset for the test split. We generated 188,726 images for training and validation. Due to computational and time constraints, we used 20,000 images for training and kept the rest for validation usage. Also, we generated 4,000 images for the test split. Image size is  $360 \times 640$  in height and width. The data overview is shown in Table 2. More details on data preparation can be found in the appendix.
For real-world data, we use the BMVC [36] dataset for quantitative evaluation. We use cabinet class which includes both prismatic and revolute parts and has the same part count and similar object shape as those used in our training and baseline models. The data contains two sequences of RGBD frames, capturing the same target objects with different part poses from various camera poses. We also test our method on RGB images taken with an iPhone X and depth maps generated from partial front views of the scene using Nerfacto [50].
Metrics For shape evaluation, following detection-based object reconstruction studies [40, 51, 16], we report mAP combined with standard shape evaluation metrics denoted as metric@threshold. We use F-Score [52], Chamfer distance (CD), and volumetric IoU (IoU) with multiple matching thresholds. More details on shape metrics can be found in the appendix. The shape mAP evaluation includes all the predicted parameters except for part kinematic parameters, serving as a comprehensive proxy for their quality. For part-wise detection with part pose and size, we use the average L2 distance
GT
Figure 7: Qualitative results on the BMVC [36] dataset.
between the corresponding eight corners of bounding boxes between the ground truth and prediction as a matching metric of mAP, similar to the ADD metric [17], denoted as mAP@threshold. The distance is normalized by the ground truth part's diameter, with a proportion of the diameter used as the threshold. Since IoU-based metrics can yield values close to zero for thin structured parts, even when they are reasonably proximate and slightly off the ground truth, and do not consider part orientation, we use the above matching metrics to better analyze part-wise detection. For part kinematics evaluation, we evaluate the absolute error on joint state (State), Orientation Error (OE) for joint direction, and Minimum Distance (MD) for rotation origin following [22] for the detected parts with matched ground truth.
Baselines By default, we denote the proposed method that takes the foreground mask as 'Ours', and the model taking unsegmented input as 'Ours-BG'. To the best of our knowledge, no prior work operates on exactly the same problem setting as ours. For shape reconstruction, we benchmark against the state-of-the-art category-level object reconstruction method A-SDF [38], the closest to our approach. We follow the most recent work setup [15] in evaluation. A-SDF requires instance segmentation, pose, and scale to project the object from the camera to the object-centric frame. We implement it using all the required data with ground truth, denoted as A-SDF-GT following [15]. Given A-SDF's assumption of a fixed part count per model, as per a similar setup of [22], we train it on the dataset's most frequent part count per category. To account for multiple part counts in a category, for categories with more than one part count, we train the second model for the next most frequent count, resulting in a maximum of two models per category, termed A-SDF-GT-2. When the input instance has an untrained part count at test time, the model trained on the most frequent part count is used for A-SDF-GT-2. In the mAP evaluation, we give GT correspondence between the prediction and ground truth; thus, a false positive happens only when the shape metric is lower than the threshold for A-SDF. For kinematic evaluation, we compare our method with the state-of-the-art single-view part kinematic estimation method OPD [22]. Note that we input OPD with an RGBD image to align input modality and modify it to output joint state. More details on baselines can be found in the appendix.
4.1 Shape reconstruction
We show the shape mAP result in Table 1. Our method outperforms A-SDF [38] with ground truth in all metrics, and Ours-BG outperforms in the majority of metrics. We show the qualitative results in Fig. 5. Our models effectively reconstruct multiple instances with diverse part counts and structures, outperforming A-SDF which struggles with reconstructing articulated parts. We attribute our method's superior performance to its part-level representation, which facilitates easier learning of various part structures. Moreover, our shape-decoder is less affected by shape variations arising from the combination of part structures and poses. The qualitative results from novel viewpoints and additional visualizations can be found in the appendix.
4.2 Kinematic estimation
OPD [22] performs 2D detection evaluated by 2D IoU, while ours on 3D by L2 distance between 3D bounding box corners. To make kinematics estimation results of OPD and ours comparable, we experimentally choose the matching threshold of our mAP as  $70\%$  so that a similar number of detected parts with OPD's result with 2D segmentation mAP@50%. Then, we select the intersection of true positive detected parts by OPD and ours. The result is shown in Table 3. Our methods outperform OPD significantly. We attribute the reason to our method operating on 3D point clouds
$\mathrm{QO + PF}$  w/o kIoU
$QO + PF$  w/ kIoU (KPF)
Figure 8: Qualitative results on kinematics-aware part fusion (KPF).
Figure 9: Qualitative results on kinematics-aware IoU (kIoU).
(b) Physically implausible false positive
Figure 10: Failure cases.
while OPD is on 2D. Thus, our method is more robust to various textures and lighting conditions and makes it easier to reason about 3D kinematics. Note that our focus here is kinematics estimation after the detection step. Thus, superiority in detection performance is not our focus.
In the following ablation studies, we validate each proposed component in a challenging setting where the input to the encoder has an unsegmented background using Ours-BG.
Kinematics-aware part fusion We show quantitative results in Table 4. Besides mAP, we also show precision considering false positives. QO denotes test-time query oversampling, and PF indicates part fusion. As a baseline, we first disable all components (w/o QO, PF, kIoU) using the same number of queries  $N_{q} = 128$  during training and add each component one by one. Introducing our proposed kIoU on top of QO and PF outperforms the baseline in shape reconstruction mAP and part detection while preserving similar precision. We visualize the effectiveness of the KPF module in Fig. 8. In the provided comparison, 'w/o KPF' denotes disabling QO, PF and kIoU. We see that KPF enhances the detection and pose estimation of small parts. We also show the qualitative results on the proposed kIoU in Fig. 9. Applying QO and PF alone leads to false positives of thin parts, which kIoU effectively reduces. It indicates that the proposed KPF module improves overall detection performance while suppressing false positives.
Anisotropic scaling and end-to-end training for shape learning We validate the effect of anisotropic scale normalization and end-to-end training in shape mAP evaluation. In Table 5, w/o AS denotes using isotropic scaling by normalizing the maximum side length to one instead of all sides. w/o SF denotes not passing shape feature h but training shape decoder  $\mathcal{O}$  separately. Disabling each component degrades performance. Especially disabling anisotropic scaling significantly drops the performance, as the single shape decoder is tasked to decode various sizes of target shapes.
Ratio of decoder layers in the refiner  $\mathcal{R}$  We investigate the relationship between the proportion of decoder layers in  $\mathcal{R}$  and performance in shape mAP. The result is shown in Fig. 11. We vary the ratio of decoder layers in the refiner  $N_{\mathcal{R}} / N_{\mathcal{D} + \mathcal{R}}$  from  $0\%$  to  $75\%$ . Allocating a portion  $(25\%)$  of decoder layers to the refiner improves performance with the same number of decoder layers while reducing excessive decoder layers from the decoder degrades performance.
4.4 Real-world data
We verify the generalizability of our approach, which is trained only on synthetic data to real-world data. Here, we include foreground masks as inputs to mitigate the domain gap from background.
State ↓	OE ↓	MD ↓
OPD [22]	19.23°/17.10cm	29.68°	38.11cm
Ours-BG	4.30°/5.27cm	3.97°	6.43cm
Ours	3.85°/4.06cm	3.66°	6.58cm
Table 3: Joint parameter estimation results.
mAP@90↑	Precision↑	F-Score@90%↑
w/o QO, PF, kIoU	38.87	52.29	67.43
w/o PF, kIoU	36.48	32.65	65.99
w/o kIoU	40.78	49.64	66.24
All (Ours-BG)	41.09	51.64	68.8
Table 4: Ablation on KPF module.
Fscore@80%	Fscore@90%	CD1@5%	CD1@1%	IoU@25%	IoU@50%
w/o AS	59.37	38.83	69.84	24.10	25.72	5.76
w/o SF	71.04	60.49	73.83	43.14	25.17	0.64
All (Ours-BG)	74.22	68.80	75.71	58.61	40.06	9.80
We present the quantitative results on the BMVC [36] dataset in Table 6. As only one instance is present in the scene, we evaluate with shape metrics without mAP. The CD value is multiplied by 100. Our method outperforms A-SDF [38]. We observe reasonable generalization as shown in Fig. 7. Furthermore, we also show the qualitative result in Fig. 6 for scenes containing multiple instances. We observe reasonable generalization to real-world data.
While our method makes significant strides in the daily articulated object reconstruction task, it does have several limitations. (1) Our method cannot handle cases where the instance boundary is not well defined, such as a door directly attached to a room. 3D reconstruction of room geometry with part-level shape reconstruction of articulated parts is an interesting future direction for scene reconstruction task. (2) The proposed approach currently does not consider the physical constraints between parts during training. Although the KPF module removes the redundant parts as post-processing, physically implausible false positives still occur as shown in Fig. 10 (a) for parts without overlapping trajectories. Introducing regularization, such as a physical violation loss [61], for such implausible configurations would alleviate this problem. (3) Our method does not explicitly consider the ambiguity in the opening direction of a part. As shown in Fig. 10 (b), the model fails to estimate the correct axis direction w.r.t. the base part for the closed part of the oven. This is because the dataset includes parts with different opening directions but similar shapes when closed. Such ambiguity can be addressed by explicit uncertainty modeling as in [1, 11], and integrating finer 2D visual cues for 3D reasoning [54, 26, 7] for localizing knobs and handles, which are informative for opening direction. Further discussions on limitations can be found in the appendix.
We presented an end-to-end trainable part-level shape reconstruction method for multiple articulated objects from a single RGBD image. We have demonstrated that our method successfully tackles the major limitation of previous works, which are unable to handle objects with various part counts using a single model, by employing a novel detect-then-group approach. We have also shown that the proposed kinematic part fusion (KPF) module effectively handles small parts as challenging targets while suppressing false positives for detection-based reconstruction. Our method outperformed the state-of-the-art baselines in shape reconstruction and kinematics estimation on the SAPIEN [58] dataset. Furthermore, on the BMVC [36] dataset and casually captured images, we demonstrated that the model trained on synthetic data reasonably generalizes to real-world data.
Figure 11: Ablation on refiner  $\mathcal{R}$
Table 5: Ablation on shape learning.
F-Score ↑	CD ↓	IoU ↑
A-SDF-GT-2 [38]	83.25	1.73	12.26
Ours	97.51	0.56	27.96
Table 6: Shape reconstruction results on the BMVC [36] dataset.
We appreciate the members of the Machine Intelligence Laboratory for constructive discussion and their insightful feedback during the research meetings. This work was partially supported by JST Moonshot R&D Grant Number JPMJPS2011, CREST Grant Number JPMJCR2015 and Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo.
[55] Watson, J., Aodha, O.M., Turmukhambetov, D., Brostow, G.J., Firman, M.: Learning stereo from single images. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)
[56] Wei, F., Chabra, R., Ma, L., Lassner, C., Zollhöfer, M., Rusinkiewicz, S., Sweeney, C., Newcombe, R., Slavcheva, M.: Self-supervised neural articulated shape and appearance models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
[57] Weng, Y., Wang, H., Zhou, Q., Qin, Y., Duan, Y., Fan, Q., Chen, B., Su, H., Guibas, L.J.: Captra: Category-level pose tracking for rigid and articulated objects from point clouds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR). pp. 13209-13218 (2021)
[58] Xiang, F., Qin, Y., Mo, K., Xia, Y., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang, H., et al.: Sapien: A simulated part-based interactive environment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11097-11107 (2020)
[59] Xie, S., Girshick, R., Dollar, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1492-1500 (2017)
[60] Yang, G., Vo, M., Neverova, N., Ramanan, D., Vedaldi, A., Joo, H.: Banmo: Building animatable 3d neural models from many casual videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2863-2873 (2022)
[61] Zhang, C., Cui, Z., Zhang, Y., Zeng, B., Pollefeys, M., Liu, S.: Holistic 3d scene understanding from a single image with implicit representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
[62] Zhou, Y., Barnes, C., Lu, J., Yang, J., Li, H.: On the continuity of rotation representations in neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
[63] Zuffi, S., Kanazawa, A., Berger-Wolf, T., Black, M.J.: Three-d safari: Learning to estimate zebra pose, shape, and texture from images" in the wild". In: Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR). pp. 5359-5368 (2019)
[64] Zuffi, S., Kanazawa, A., Jacobs, D.W., Black, M.J.: 3d menagerie: Modeling the 3d shape and pose of animals. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 6365-6373 (2017)
A Details on kinematics-aware part fusion (KPF)
In Algorithm 1, we present the details of kinematics-aware part fusion. We gather part proposals through  $N_{Q}$  independent inference runs, achieved by randomly sampling query positions  $\{\mathbf{q}_n\}$  with the furthest point sampling (FPS). The 'model' in the algorithm represents the decoder  $\mathcal{D}$  and the refiner  $\mathcal{R}$ , and  $\mathbf{y}_{\mathrm{fg}}$  represents objectness, defined as  $1 - \mathrm{bg}$ .
After part proposals are generated, we apply Non-Maximum Suppression (NMS) with 3D bounding box IoU and threshold by objectness  $1 - \mathbf{y}_{\mathrm{bg}}$  before applying NMS with kinematics-aware IoU (NMS-kIoU) for performance reasons. PF-kIoU denotes the part fusion (PF) process using kIoU, which is based on the Weighted Box Fusion (WBF) approach as described in [4]. The procedure is further detailed in Algorithm 2.
Unlike the conventional WBF, our algorithm fuses all parameters in the part proposal rather than only the 2D bounding box parameters. Furthermore, our algorithm iteratively runs until convergence, as demonstrated from line 11 to line 15 in Algorithm 1.  $C$  represents clusters of overlapping part proposals identified by kIoU. We employ a simple arithmetic weighted average for all elements except the rotation matrix in a part proposal for fusion, denoted as weighted-average.
For the rotation matrix  $\mathbf{R} \in \mathrm{SO}(3)$ , we initially derive a weighted average for the  $3 \times 3$  matrices, labeled as  $\mathbf{R}_{\mathrm{wa}} \in \mathbb{R}^{3 \times 3}$ . We then compute the weighted-averaged rotation matrix by minimizing the Frobenius norm to  $\mathbf{R}_{\mathrm{wa}}$ . This is accomplished as  $\mathbf{R} = \operatorname*{argmin}_{\mathbf{R}} \| \mathbf{R} - \mathbf{R}_{\mathrm{wa}} \|_F$ , following the 3D rotation library RoMa [4].
The value of  $\mathbf{y}_{\mathrm{fg},n'}'$  denotes the scaled objectness as a confidence score, considering the number of independent inferences  $N_{Q}$ . If the number of part proposals in a cluster is fewer than the independent inference runs  $N_{Q}$ , it indicates that only a limited number of independent inferences predict it. In such cases, we adjust the corresponding confidence score  $\mathbf{y}_{\mathrm{fg},n'}'$  by scaling it down by the ratio of the number of part proposals in the cluster ( $|C[n']|$ ) to the number of independent inferences  $T = N_{Q}$ . Conversely, we scale up the confidence score if a cluster contains more part proposals than the independent inferences  $N_{Q}$ . For the initial run of part fusion (line 13 of Algorithm 1), we set  $T = N_{Q}$  to fuse  $N_{Q}$  inferences. We assign  $T = 1$  for subsequent runs, assuming we apply part fusion to a single inference run fused by the previous part fusion.
Hyperparameters The hyperparameter  $\tau_{\mathrm{IoU}}$  is set to 0.25, following the original code of 3DETR [37]. We set  $\tau_{\mathrm{obj}}$  to 0.25, representing the uniform probability of the three joint types plus background class, as outlined in Section 3. Additionally, we simplify the hyperparameter by setting  $\tau_{\mathrm{obj\_final}}$  equal to  $\tau_{\mathrm{obj}}$ . The hyperparameter  $\tau_{\mathrm{kloU}}$  is assigned a value of 0.5, double the  $\tau_{\mathrm{IoU}}$  value, to impose a stricter overlapping threshold and ensure similar trajectory part proposals are clustered together. We experimentally determine the values for  $\tau_{\mathrm{scaled}}$  and  $\tau_{\mathrm{count}}$ , setting them to 0.1 and 3, respectively.
B Full Loss Formulation
During training, we use the same multi-layer perceptron (MLP) for part prediction at each decoder layer in the decoder  $\mathcal{D}$ . We calculate the total loss,  $\mathcal{L}_{\mathrm{total}}$ , as defined in Section 3.7, independently for each layer and sum up these losses. The comprehensive loss formulation is as follows:
$$
\mathcal {L} _ {\text {t o t a l}} = \sum_ {k = 1} ^ {N _ {\mathcal {D}}} \mathcal {L} _ {\text {p a r t}, k} + \mathcal {L} _ {\text {p a r t}, k} ^ {\prime} + \mathcal {L} _ {\text {i n s t a n c e}, k}. \tag {4}
$$
C Architecture
C.1 Detection backbone