SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuki Kawana<sup>1</sup>, Tatsuya Harada<sup>1,2</sup>  
<sup>1</sup>The University of Tokyo, <sup>2</sup>RIKEN AIP {kawana, harada}@mi.t.u-tokyo.ac.jp
We propose an end-to-end trainable, cross-category method for reconstructing multiple man-made articulated objects from a single RGBD image, focusing on part-level shape reconstruction and pose and kinematics estimation. We depart from previous works that rely on learning instance-level latent space, focusing on man-made articulated objects with predefined part counts. Instead, we propose a novel alternative approach that employs part-level representation, representing instances as combinations of detected parts. While our detect-then-group approach effectively handles instances with diverse part structures and various part counts, it faces issues of false positives, varying part sizes and scales, and an increasing model size due to end-to-end training. To address these challenges, we propose 1) test-time kinematics-aware part fusion to improve detection performance while suppressing false positives, 2) anisotropic scale normalization for part shape learning to accommodate various part sizes and scales, and 3) a balancing strategy for cross-refinement between feature space and output space to improve part detection while maintaining model size. Evaluation on both synthetic and real data demonstrates that our method successfully reconstructs variously structured multiple instances that previous works cannot handle, and outperforms prior works in shape reconstruction and kinematics estimation.
Estimating object shape, pose, size, and kinematics from a single frame of partial observation is a fundamental challenge in computer vision. Understanding such properties of daily articulated objects has various applications in robotics and AR/VR.
Shape reconstruction of daily articulated objects is a challenging task. These objects exhibit a range of shapes resulting from different local part poses. More importantly, they display significant intra- and inter-category diversity in part configurations, including variations in part counts and structures. These factors together contribute to an exponentially increasing shape variation. Previous works have addressed this issue by either limiting a single model to target objects with a single articulated part [15] or employing multiple category-level models [28] to accommodate varying part counts. These approaches first detect each instance, and then model the target shape in an instance-level latent space, primarily employing A-SDF [38] for shape learning. A-SDF maps the target shape into an instance-wise latent space, and then a shape decoder outputs the entire shape of the instance. However, this approach is limited when dealing with varying part counts and structures, as the shape decoder must handle an exponentially increasing number of shape variations due to different part layout combinations [43, 9] in addition to local part poses. Consequently, addressing this variety with a single model remains a complex and unsolved task.
In this paper, we address this complexity through our novel detect-then-group approach. Our key observation is that daily articulated objects consist of similar part shapes. For example, regardless
Figure 1: Our detection-based approach estimates part-level shape, pose, and kinematics as joint parameters. It also recovers parts-to-instance associations to handle multiple instances with various part structures and counts.
of the number of refrigerator doors, each door typically has a similar shape, and the base part may share similarities with those from other categories, such as storage units. By detecting each part and then grouping them into multiple instances, we provide a scalable and generalizable approach for handling diverse part configurations of daily articulated objects in a scene.
Based on this concept, we propose an end-to-end detection-based approach for part-level shape reconstruction. Building upon 3DETR [37] as an end-to-end trainable detector backbone, given a single RGBD image with an optional foreground mask, our model outputs part shape, pose, joint parameters, parts-to-instance association, and instance category. Our approach employs a novel detect-then-group approach. It first detects parts and applies simple clustering of parts into instances based on learned part embeddings' distance, in contrast to the previous works using additional instance detection module [28, 15]. An overview of our approach is shown in Fig 1. However, we found that detection-based shape reconstruction is prone to false positives for articulated objects' thin and small parts with little overlap, which is hard to remove by NMS. Also, articulated objects often have parts of varied sizes and scales, making training with a single-shape decoder challenging. Additionally, increasing model size by end-to-end training from detection to reconstruction makes simply enlarging the model size to improve performance undesirable. To address these challenges, we propose:(1) kinematics-aware part fusion to reduce false positives and improve detection accuracy; (2) anisotropic scale normalization for various part sizes and scales in shape learning; (3) and an output space refiner module coupled with a model-size balancing strategy with decoder for improved performance while keeping the model size. We evaluate our method on the photorealistically rendered SAPIEN [58] dataset, and our approach outperforms state-of-the-art baselines in shape reconstruction and joint parameter estimation. Furthermore, the model trained only on synthetic data generalizes to real-world data, outperforming the state-of-the-art methods on the BMVC [36] dataset.
Our contributions can be summarized as follows: (1) a novel part-level end-to-end shape reconstruction method for articulated objects from a single RGBD image; (2) a novel detect-then-group approach that simplifies the pipeline; (3) addressing detection-based reconstruction challenges with kinematics-aware part fusion, anisotropic scale normalization, and a refiner module coupled with model-size balancing; (4) superior performance on the SAPIEN [58] dataset, with the ability to generalize to real-world data from the BMVC [36] dataset.
Articulated shape reconstruction A number of works have focused on human subjects using single image input [46, 45] and utilize category-specific templates to recover deformation from a canonical shape [29, 64, 3, 63, 25]. Recent research has delved into recovering articulated shapes with unknown kinematic chains from sequences of a target in motion [41, 60]. These works make category-level assumptions about kinematic structures, with targets in observations sharing common kinematic structures. Their main focus is on natural objects such as humans and animals. In contrast, our interest is in reconstructing the shape of multi-category, daily man-made articulated objects with diverse kinematic structures using a single model. Recent years have seen the emergence of methods specifically targeting man-made articulated objects [56, 23, 53], and taking single-frame input [38, 28, 15, 24]. However, these models are constrained by either a predefined number of parts per category or the necessity of multiple models for each combination of categories and part counts. Consequently, they are unable to scale to a wide array of real-world articulated objects with varying part counts using a single model. Our approach addresses this limitation.
Figure 2: Overview of the pipeline. The input RGBD image is projected to a colored point cloud. The encoder  $\mathcal{E}$  extracts scene features and a downsampled point cloud. The decoder  $\mathcal{D}$  outputs a set of part proposals  $\{X_{n}\}$  from part queries  $\{\mathbf{q}_n\}$ . The refiner  $\mathcal{R}$  estimates the residual of part pose and size  $\Delta B$  and joint parameter  $\Delta A$  for refined  $\{B, A\}$ . At test time, the inference is run  $N_{Q}$  times independently to densely sample part proposals as  $\{X_{n'}\}$ . KPF removes false positives in  $\{X_{n'}\}$  by using kinematics-aware IoU (kIoU) to refine the prediction further. The part shape is reconstructed by the implicit shape decoder  $\mathcal{O}$ .
Figure 3: Architecture of refiner  $\mathcal{R}$
Figure 4: Illustration of (a) false positives, (b) Convex hull for kIoU, and (c) comparison of 3D box IoU and kIoU for overlapping parts.
Pose and kinematic estimation of articulated objects predominantly, existing research on pose and kinematic estimation of articulated objects has focused on estimation from sequences [16, 21, 57, 49], necessitating multiple frames of moving targets or interaction with the environment before estimation. Estimation from a single image has also been explored [27, 36, 28], but these methods are limited to predefined part structures. A few recent studies have proposed approaches without assumptions on part structure [22, 13]. However, these methods target single instances, requiring instance detection before part-level estimation. Moreover, their focus is limited to detection, pose and kinematic estimation, whereas our work aims for shape reconstruction in an end-to-end manner.
Detection-based reconstruction A large body of work exists combining detection and reconstruction for multiple rigid objects in diverse settings, such as indoor scenes [51, 39, 40, 61, 54, 14], tabletop environments [19, 20], and road scenes [2, 33]. Recent works target daily articulated objects [28, 16]. However, these methods predominantly rely on an instance-level detection approach. In contrast, our work pivots towards part-level detection to effectively handle a wide variety of part structures of real-world articulated objects.
3.1 Problem setting
Our method takes a point cloud  $P$  with  $N_P$  3D points and color feature  $F$  lifted from a single RGBD image  $I$  of articulated objects and camera intrinsic  $K$  as input. It outputs a set of parts  $G = \{i \in [N]\}$  and non-overlapping subsets of all parts  $\{G_l\}_{l=1}^{N_G}$  as  $N_G$  instances, where  $G = \bigcup_{l} \{G_l\}_{l=1}^{N_G}$ . Note that we use the shorthand  $\{*\}_i$  to denote an ordered set  $\{*\}_i^N$  for brevity. For each part  $i$ , we estimate 6D pose and size  $B \in \mathrm{SE}(3) \times \mathbb{R}^3$ , part shape  $\mathcal{O}$  as an implicit representation and kinematic parameters  $M$ . Our shape representation is an implicit function defined as  $\mathcal{O}: \mathbb{R}^3 \to [0,1]$  with isosurface threshold  $\tau_{\mathcal{O}}$ , where  $\{\mathbf{x} \in \mathbb{R}^3 \mid \mathcal{O}(\mathbf{x}) > \tau_{\mathcal{O}}\}$  indicates inside the shape. The kinematic parameter  $M$  consists of joint type  $y \in \{0,1,2\}$  which represents fixed, revolute, and prismatic types, joint direction  $\mathbf{a} \in \mathbb{S}^2$ , 1D joint state  $d_{\mathrm{current}}$  and  $d_{\mathrm{max}}$  for the current pose and fully opened pose from
the canonical pose. We also predict the revolute origin  $\mathbf{v} \in \mathbb{R}^3$  for the revolute part. We define the joint parameter as  $A = \{\mathbf{a}, d_{\mathrm{current}}, d_{\mathrm{max}}[, \mathbf{v}]\}$ , thus  $M = \{y, A\}$ . For each instance  $l$ , we estimate the instance parameter  $J$  which consists of category  $u$  and part association defined as  $\delta_{li} = \mathbb{1}(i \in G_l)$  where  $\mathbb{1}$  is an indicator function.
3.2 Detection backbone
Our detection backbone consists of a transformer-based encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$  based on 3DETR [37]. The encoder comprises recursive self-attention layers encoding 3D points  $P$  and color feature  $F$  into downsampled 3D point cloud  $P'$  of  $N_{P'}$  points and  $D_{F'}$ -dimensional scene feature  $F'$ . Query locations  $\{\mathbf{q}_n\}_{n=1}^{N_q} \in \mathbb{R}^3$  are randomly sampled using furthest point sampling (FPS) from  $P'$ . The decoder is composed of transformer decoder layers  $\{\phi_k\}_{k=1}^{N_{\mathcal{D}}}$ , considering cross-attention between queries and  $F'$  and self-attention among queries. The decoder iteratively refines query features  $\{\mathbf{e}_{n,k+1}\} = \phi_k(\{\mathbf{e}_{n,k}\}, F')$ . Lastly, a set of part prediction MLPs decodes each refined query feature to produce output values. For clarity, the query index  $n$  is omitted when possible.
3.3 Part representation
Part pose and size We predict part pose and size  $B$  as a set of part center  $\mathbf{c} \in \mathbb{R}^3$ , rotation  $\mathbf{R} \in \mathrm{SO}(3)$  and size  $\mathbf{s} \in \mathbb{R}^3$  for each query. We predict  $\mathbf{c}$  as an offset  $\Delta \mathbf{c}$  from  $\mathbf{q}$ , added to the query coordinates, i.e.,  $\mathbf{c} = \mathbf{q} + \Delta \mathbf{c}$ .
Part shape We employ a shared-weight, single implicit shape decoder  $\mathcal{O}$  for performing part-wise shape reconstruction by taking point clouds around the detected regions where parts are identified. Given the diversity in shape and pose, and anisotropic scaling of the parts we focus on, it is challenging to learn shape bias with a single shape decoder. Therefore, we propose anisotropically normalizing the side lengths of the shape decoder's input and output to a unit scale to perform reconstruction. We define the input point cloud  $P_{\mathcal{O}}$  as follows:
$$
P _ {\mathcal {O}} = \left\{\left(\mathbf {R S}\right) ^ {- 1} (\mathbf {p} - \mathbf {c}) \mid \mathbf {p} \in P, \max  \left(\left| \left(\mathbf {R S}\right) ^ {- 1} (\mathbf {p} - \mathbf {c}) \right|\right) \leq 0. 5 \right\}. \tag {1}
$$
where  $\mathbf{S}$  denotes diagonal matrix of scale  $\mathbf{s}$ . We define the output occupancy value at  $\mathbf{x} \in \mathbb{R}^3$  as  $o_{\mathbf{x}} = \mathcal{O}((\mathbf{RS})^{-1}(\mathbf{x} - \mathbf{c}) \mid P_{\mathcal{O}}, \mathbf{h})$ , where  $\mathbf{h} \in \mathbb{R}^{D_{\mathrm{h}}}$  is a part shape feature modeled by a part prediction MLP. Given that the input point cloud  $P_{\mathcal{O}}$  includes background, the geometry of the target part shape can be ambiguous. To address this issue, we train the detector backbone and shape decoder end-to-end, by inputting  $\mathbf{h}$  as shape geometry to the shape decoder so that  $\mathbf{h}$  informs the shape decoder of the foreground target shape. We utilize a shape decoder architecture with a lightweight local geometry encoder [42] to spatially associate input points  $P_{\mathcal{O}}$  with output occupancy values  $o_{\mathbf{x}}$  which are both defined in normalized space.
Part kinematics We predict a 4-dimensional vector  $\mathbf{y}$  as a probability distribution over part joint types. This includes a 'background' or 'not a part' type for instances where predicted part proposals might not contain a part. The revolute origin  $\mathbf{v} = \mathbf{q} + \Delta \mathbf{v}$  is predicted similarly to the part center  $\mathbf{c}$ , with an offset  $\Delta \mathbf{v}$ . Joint states  $d_{\mathrm{current}}$  and  $d_{\mathrm{max}}$  are modeled by separate part prediction MLPs for revolute and prismatic types, and we only supervise the output corresponding to the ground truth joint type. A single MLP is used for joint direction  $\mathbf{a}$  for both revolute and prismatic types.
3.4 Instance representation
We form a collection of parts as a single instance  $G_{l} = \{i\in G\mid \forall i,j\in G,i\neq j\Rightarrow \| \mathbf{z}_{i} - \mathbf{z}_{j}\| <  \tau_{\mathbf{z}}\}$  where the distance between the  $D_{\mathbf{z}}$ -dimensional embeddings,  $\mathbf{z}_i$  and  $\mathbf{z}_j$ , of each part pair within the group is kept below a specified threshold  $\tau_{\mathbf{z}}$ . We predict an  $N_{\mathbf{u}}$ -dimensional vector  $\mathbf{u}$  as a probability distribution over instance categories per part. At test time, we predict the instance category by taking the category prediction with the highest confidence from the category predictions of the parts belonging to the same instance.
3.5 Refiner  $\mathcal{R}$
Query features are iteratively refined in the decoder  $\mathcal{D}$  in the feature space [37]. Increasing the number of decoder layers can enhance performance at the expense of a larger model size [37]. However, it is
crucial to avoid increasing the model size to enable efficient end-to-end training from detection to shape reconstruction. To improve model performance while maintaining the model size, we found cross-refining both output space and feature space to be effective. We introduce a refiner  $\mathcal{R}$ , which has an identical architecture with the decoder  $\mathcal{D}$ . The refiner  $\mathcal{R}$  serves to refine the prediction in output space by reallocating a portion of decoder layers from decoder  $\mathcal{D}$  to  $\mathcal{R}$ . Assuming there are  $N_{\mathcal{D} + \mathcal{R}}$  decoder layers in the original model, we reallocate  $N_{\mathcal{R}}$  decoder layers to the refiner  $\mathcal{R}$ . Consequently,  $N_{\mathcal{D}} = N_{\mathcal{D} + \mathcal{R}} - N_{\mathcal{R}}$  layers are used for query feature refinement in decoder  $\mathcal{D}$ . The refiner  $\mathcal{R}$  refines part pose and size  $B'$  and joint parameter  $A'$  from  $\mathcal{D}$  by predicting residuals  $\Delta B$  and  $\Delta A$  to produce refined prediction  $B$  and  $A$ , respectively. The architecture of the refiner  $\mathcal{R}$  is shown in Fig. 3.
3.6 Kinematics-aware part fusion (KPF)
Articulated objects often consist of small and thin parts. Especially for unsegmented inputs, only a small portion of the point cloud represents such parts after subsampling. To ensure a sufficient number of queries cover such parts and their surrounding context, at test time, we randomize the sampling positions of the queries and independently carry out inference and NMS for the input  $N_{Q}$  times, obtaining a collection of densely sampled part proposals from  $N_{Q}$  inference runs. This process is termed Query Oversampling (OQ). However, OQ can increase false positives and degrade detection performance. To mitigate this, we propose Part Fusion (PF), inspired by Weighted Box Fusion (WBF) [48], to merge overlapping part proposals, thereby reducing false positives and improving detection accuracy. Unlike WBF, which fuses only 2D bounding parameters, PF fuses all the predicted parameters of part proposals, with an average weight defined as objectness  $1 - \mathbf{y}_{\mathrm{bg}}$ . However, using IoU with 3D bounding boxes as overlapping metrics yields overly small values for thin structured parts, even when they are proximate. This results in PF failing to merge redundant parts, leading to false positives, as illustrated in Fig. 4 (a). To overcome this challenge, we propose a kinematics-aware IoU (kIoU) based on the observation that a redundant part pair exhibits significantly overlapping trajectories. To calculate kIoU, we construct a convex hull for each part in the pair using the 24 vertices of three bounding boxes based on size and canonical pose, current pose, and fully opened pose using the predicted part pose and size  $B$  and joint parameter  $A$ , as show in Fig. 4 (b). Then, we calculate the IoU between the two hulls. The comparison between the 3D bounding box IoU and kIoU is depicted in Fig. 4 (c). Our overall method, termed Kinematic-aware Part Fusion (KPF), includes: (1) executing inference and NMS using kIoU to gather part proposals multiple times, (2) conducting PF using kIoU to update these proposals iteratively, and (3) removing proposals with low objectness. Note that KPF is non-differentiable and is disabled during training, with  $N_{Q} = 1$ . For further details, please refer to the appendix.
3.7 Set matching and training loss
Set matching We base our end-to-end training of detection to reconstruction by utilizing 1-to-1 matching between part proposals and ground truth, using bipartite matching [5] for loss calculation. Similarly to [37], we define a matching cost between a predicted part and a ground truth part as  $C_{\text{match}} = \lambda_1 \| \mathbf{B} - \mathbf{B}^{\mathrm{GT}} \|_1 + \lambda_2 \| \mathbf{c} - \mathbf{c}^{\mathrm{GT}} \|_1 - \lambda_3 \mathbf{y}_y + \lambda_4 (1 - \mathbf{y}_{\mathrm{bg}})$ , where  $\mathbf{B}$  defines eight vertices of cuboid defined by part pose and size  $B$ ,  $\mathbf{y}_y$  defines the joint type probability given the ground truth label  $y$ , and  $1 - \mathbf{y}_{\mathrm{bg}}$  defines the foreground probability. Deviating from [37], we use the L1 distance of eight vertices of a cuboid instead of the GIoU [44] of cuboids in the first term to avoid the costly calculation of enclosing hull for 3D rotated cuboids.
Training losses For each pair of prediction and ground truth, we define part loss as
$$
\begin{array}{l} \mathcal {L} _ {\text {p a r t}} = \frac {1}{N} \sum_ {i} ^ {N} \| \mathbf {B} _ {i} - \mathbf {B} _ {i} ^ {\mathrm {G T}} \| _ {1} + \| I - \mathbf {R} _ {i} ^ {T} \mathbf {R} _ {i} ^ {\mathrm {G T}} \| _ {F} ^ {2} + \mathbb {E} _ {\mathbf {x} \sim \mathbb {R} ^ {3}} \mathrm {B C E} \left(o _ {\mathbf {x}, i}, o _ {\mathbf {x}, i} ^ {\mathrm {G T}}\right) + \| d _ {\max , i} - d _ {\max , i} ^ {\mathrm {G T}} \| _ {1} \tag {2} \\ + \left\| d _ {\text {c u r r e n t}, i} - d _ {\text {c u r r e n t}, i} ^ {\mathrm {G T}} \right\| _ {1} - \mathbf {a} _ {i} ^ {T} \mathbf {a} _ {i} ^ {\mathrm {G T}} + \operatorname {P L} (\mathbf {v} _ {i}, \mathbf {v} _ {i} ^ {\mathrm {G T}}, \mathbf {a} _ {i} ^ {\mathrm {G T}}) + \operatorname {C E} (\mathbf {y} _ {i}, \mathbf {y} _ {i} ^ {\mathrm {G T}}) + \operatorname {C E} (\mathbf {u} _ {i}, \mathbf {u} _ {i} ^ {\mathrm {G T}}) \\ \end{array}
$$
All loss terms have equal weights. The first term is a disentangled L1 loss described in [47] to optimize B. This loss is replicated three times by using only one of the predicted three components (R,c,s) for B, while replacing the other three with their ground truth values. We also found that directly optimizing rotation, as in the second term of the loss, leads to smaller rotation loss during training. BCE and CE denote binary cross-entropy and cross-entropy loss, respectively. PL denotes
F-Score@80%↑	F-Score@90%↑	CD@5%↑	CD@1%↑	IoU@25%↑	IoU@50%↑
A-SDF-GT [38]	65.49	47.69	74.60	39.76	36.62	10.81
A-SDF-GT-2 [38]	68.81	52.55	75.91	43.58	38.59	10.43
Ours-BG	74.22	68.80	75.71	58.61	40.06	9.80
Ours	74.77	68.38	77.39	56.53	41.35	11.63
Table 1: Shape mAP results on SAPIEN [58] dataset.
the point-line distance between the revolute origin  $\mathbf{v}_i$  and the ground truth joint axis defined by  $\mathbf{v}_i^{\mathrm{GT}}$  and  $\mathbf{a}_i^{\mathrm{GT}}$ . For unrefined prediction  $B'$  and  $A'$ , we define the same loss as  $\mathcal{L}_{\mathrm{part}}$  except for  $o_{\mathbf{x}}$ ,  $\mathbf{y}_i$  and  $\mathbf{u}_i$  denoted as  $\mathcal{L}_{\mathrm{part}}'$ . During training, we use ground truth  $B$  for Eq.1 to avoid noisy prediction adversely affecting shape learning.
We also define instance loss  $\mathcal{L}_{\mathrm{instance}}$  for learning part-to-instance association with modified improved triplet loss [8] defined as:
$$
\mathcal {L} _ {\text {i n s t a n c e}} = \lambda_ {\text {i n t r a}} \sum_ {i \in G} \frac {1}{| G _ {l | i} |} \sum_ {j \in G _ {l | i} \backslash i} [ \eta_ {i j} - \tau_ {\mathbf {z}} ^ {\prime} ] _ {+} + \frac {1}{N} \sum_ {i \in G} [ \max  _ {j \in G _ {l | i} \backslash i} \eta_ {i j} - \min  _ {j ^ {\prime} \in G \backslash G _ {i}} \eta_ {i j ^ {\prime}} + 3 \tau_ {\mathbf {z}} ^ {\prime} ] _ {+} \tag {3}
$$