Title: Detection Based Part-level Articulated Object Reconstruction from Single RGBD Image
Venue: NeurIPS 2023
Accepted: yes
Keywords: articulated objects

shape reconstruction

3D reconstruction
Abstract: We propose an end-to-end trainable, cross-category method for reconstructing multiple man-made articulated objects from a single RGBD image, focusing on part-level shape reconstruction and pose and kinematics estimation. We depart from previous works that rely on learning instance-level latent space, focusing on man-made articulated objects with predefined part counts. Instead, we propose a novel alternative approach that employs part-level representation, representing instances as combinations of detected parts. While our detect-then-group approach effectively handles instances with diverse part structures and various part counts, it faces issues of false positives, varying part sizes and scales, and an increasing model size due to end-to-end training. To address these challenges, we propose 1) test-time kinematics-aware part fusion to improve detection performance while suppressing false positives, 2) anisotropic scale normalization for part shape learning to accommodate various part sizes and scales, and 3) a balancing strategy for cross-refinement between feature space and output space to improve part detection while maintaining model size. Evaluation on both synthetic and real data demonstrates that our method successfully reconstructs variously structured multiple instances that previous works cannot handle, and outperforms prior works in shape reconstruction and kinematics estimation.

Detection Based Part-level Articulated Object Reconstruction from Single RGBD Image

Abstract

We propose an end-to-end trainable, cross-category method for reconstructing multiple man-made articulated objects from a single RGBD image, focusing on part-level shape reconstruction and pose and kinematics estimation. We depart from previous works that rely on learning instance-level latent space, focusing on man-made articulated objects with predefined part counts. Instead, we propose a novel alternative approach that employs part-level representation, representing instances as combinations of detected parts. While our detect-then-group approach effectively handles instances with diverse part structures and various part counts, it faces issues of false positives, varying part sizes and scales, and an increasing model size due to end-to-end training. To address these challenges, we propose 1) test-time kinematics-aware part fusion to improve detection performance while suppressing false positives, 2) anisotropic scale normalization for part shape learning to accommodate various part sizes and scales, and 3) a balancing strategy for cross-refinement between feature space and output space to improve part detection while maintaining model size. Evaluation on both synthetic and real data demonstrates that our method successfully reconstructs variously structured multiple instances that previous works cannot handle, and outperforms prior works in shape reconstruction and kinematics estimation.

1 Introduction

Estimating object shape, pose, size, and kinematics from a single frame of partial observation is a fundamental challenge in computer vision. Understanding such properties of daily articulated objects has various applications in robotics and AR/VR.

Shape reconstruction of daily articulated objects is a challenging task. These objects exhibit a range of shapes resulting from different local part poses. More importantly, they display significant intraand inter-category diversity in part configurations, including variations in part counts and structures. These factors together contribute to an exponentially increasing shape variation. Previous works have addressed this issue by either limiting a single model to target objects with a single articulated part [15] or employing multiple category-level models [28] to accommodate varying part counts. These approaches first detect each instance, and then model the target shape in an instance-level latent space, primarily employing A-SDF [38] for shape learning. A-SDF maps the target shape into an instance-wise latent space, and then a shape decoder outputs the entire shape of the instance. However, this approach is limited when dealing with varying part counts and structures, as the shape decoder must handle an exponentially increasing number of shape variations due to different part layout combinations [43, 9] in addition to local part poses. Consequently, addressing this variety with a single model remains a complex and unsolved task.

In this paper, we address this complexity through our novel detect-then-group approach. Our key observation is that daily articulated objects consist of similar part shapes. For example, regardless

Input

Output

·Part shape

·Part-to-instance

Optional

·Part6DPose

association

·Part size

Instance

Jointtype

category

Jointdirection

Foregroundmask

Jointstate

Revoluteorigin

RGBDimage

Detect-then-group

inembeddingspace

of the number of refrigerator doors, each door typically has a similar shape, and the base part may share similarities with those from other categories, such as storage units. By detecting each part and then grouping them into multiple instances, we provide a scalable and generalizable approach for handling diverse part configurations of daily articulated objects in a scene.

Based on this concept, we propose an end-to-end detection-based approach for part-level shape reconstruction. Building upon 3DETR [37] as an end-to-end trainable detector backbone, given a single RGBD image with an optional foreground mask, our model outputs part shape, pose, joint parameters, parts-to-instance association, and instance category. Our approach employs a novel detect-then-group approach. It first detects parts and applies simple clustering of parts into instances based on learned part embeddings' distance, in contrast to the previous works using additional instance detection module [28, 15]. An overview of our approach is shown in Fig 1. However, we found that detection-based shape reconstruction is prone to false positives for articulated objects' thin and small parts with little overlap, which is hard to remove by NMS. Also, articulated objects often have parts of varied sizes and scales, making training with a single-shape decoder challenging. Additionally, increasing model size by end-to-end training from detection to reconstruction makes simply enlarging the model size to improve performance undesirable. To address these challenges, we propose:(1) kinematics-aware part fusion to reduce false positives and improve detection accuracy; (2) anisotropic scale normalization for various part sizes and scales in shape learning; (3) and an output space refiner module coupled with a model-size balancing strategy with decoder for improved performance while keeping the model size. We evaluate our method on the photorealistically rendered SAPIEN [58] dataset, and our approach outperforms state-of-the-art baselines in shape reconstruction and joint parameter estimation. Furthermore, the model trained only on synthetic data generalizes to real-world data, outperforming the state-of-the-art methods on the BMVC [36] dataset.

Our contributions can be summarized as follows: (1) a novel part-level end-to-end shape reconstruction method for articulated objects from a single RGBD image; (2) a novel detect-then-group approach that simplifies the pipeline; (3) addressing detection-based reconstruction challenges with kinematics-aware part fusion, anisotropic scale normalization, and a refiner module coupled with model-size balancing; (4) superior performance on the SAPIEN [58] dataset, with the ability to generalize to real-world data from the BMVC [36] dataset.

2 Related Work

Articulated shape reconstruction A number of works have focused on human subjects using single image input [46, 45] and utilize category-specific templates to recover deformation from a canonical shape [29, 64, 3, 63, 25]. Recent research has delved into recovering articulated shapes with unknown kinematic chains from sequences of a target in motion [41, 60]. These works make category-level assumptions about kinematic structures, with targets in observations sharing common kinematic structures. Their main focus is on natural objects such as humans and animals. In contrast, our interest is in reconstructing the shape of multi-category, daily man-made articulated objects with diverse kinematic structures using a single model. Recent years have seen the emergence of methods specifically targeting man-made articulated objects [56, 23, 53], and taking single-frame input [38, 28, 15, 24]. However, these models are constrained by either a predefined number of parts per category or the necessity of multiple models for each combination of categories and part counts. Consequently, they are unable to scale to a wide array of real-world articulated objects with varying part counts using a single model. Our approach addresses this limitation.

Coloredpoint

PointcloudP

Partshape

Camera intrinsicK

cloud {P,F}

EncoderE

Shape

2D to3D

Scene

Downsampled

decoder0

projection

featureF'

pointcloudP'

Shape

FPS

No

feature

Pose+

Partposeand

Jointparam.

Query

{hi}

size{Bi}

shapes

estimation

Optional

{qn}

Part

Eux3

proposals

parameters{M}

Kinematic

KPF

RGBDimageI

ForegroundmaskW

RefinerR

Kinematics-aware

parameters{i3

Instance

Partstoinstance

partfusion

association

Output{Xn/B，An}

Refined {Bn,An}

Non-refined {B, A}

Center

RefinerR

Residual

Positional

{Cn +qn}

{Bn,△An}

encoding

ΦND

MLPs

Query

Φ1+ND

PNR

MLPs

feature

Query{qn}

{enND)

Scene

featureF'

dmax

(a)False positives（

(b)Convexhull (c)loU =21%,kloU =77%

Pose and kinematic estimation of articulated objects predominantly, existing research on pose and kinematic estimation of articulated objects has focused on estimation from sequences [16, 21, 57, 49], necessitating multiple frames of moving targets or interaction with the environment before estimation. Estimation from a single image has also been explored [27, 36, 28], but these methods are limited to predefined part structures. A few recent studies have proposed approaches without assumptions on part structure [22, 13]. However, these methods target single instances, requiring instance detection before part-level estimation. Moreover, their focus is limited to detection, pose and kinematic estimation, whereas our work aims for shape reconstruction in an end-to-end manner.

Detection-based reconstruction A large body of work exists combining detection and reconstruction for multiple rigid objects in diverse settings, such as indoor scenes [51, 39, 40, 61, 54, 14], tabletop environments [19, 20], and road scenes [2, 33]. Recent works target daily articulated objects [28, 16]. However, these methods predominantly rely on an instance-level detection approach. In contrast, our work pivots towards part-level detection to effectively handle a wide variety of part structures of real-world articulated objects.

3 Methods

3.1 Problem setting

Our method takes a point cloud P with N P 3D points and color feature F lifted from a single RGBD image I of articulated objects and camera intrinsic K as input. It outputs a set of parts G = { i ∈ [ N ] } and non-overlapping subsets of all parts { G l } N G l =1 as N G instances, where G = ⋃ l { G l } N G l =1 . Note that we use the shorthand {∗ i } to denote an ordered set {∗ i } N i =1 for brevity. For each part i , we estimate 6D pose and size B ∈ SE (3) × R 3 , part shape O as an implicit representation and kinematic parameters M . Our shape representation is an implicit function defined as O : R 3 → [0 , 1] with isosurface threshold τ O , where { x ∈ R 3 | O ( x ) > τ O } indicates inside the shape. The kinematic parameter M consists of joint type y ∈ { 0 , 1 , 2 } which represents fixed, revolute, and prismatic types, joint direction a ∈ S 2 , 1D joint state d current and d max for the current pose and fully opened pose from

the canonical pose. We also predict the revolute origin v ∈ R 3 for the revolute part. We define the joint parameter as A = { a , d current , d max [ , v ] } , thus M = { y, A } . For each instance l , we estimate the instance parameter J which consists of category u and part association defined as δ li = ✶ ( i ∈ G l ) , where ✶ is an indicator function. 3.2 Detection backbone

Our detection backbone consists of a transformer-based encoder E and decoder D based on 3DETR [37]. The encoder comprises recursive self-attention layers encoding 3D points P and color feature F into downsampled 3D point cloud P ′ of N P ′ points and D F ′ -dimensional scene feature F ′ . Query locations { q n } N q n =1 ∈ R 3 are randomly sampled using furthest point sampling (FPS) from P ′ . The decoder is composed of transformer decoder layers { ϕ k } N D k =1 , considering cross-attention between queries and F ′ and self-attention among queries. The decoder iteratively refines query features { e n,k +1 } = ϕ k ( { e n,k } , F ′ ) . Lastly, a set of part prediction MLPs decodes each refined query feature to produce output values. For clarity, the query index n is omitted when possible.

3.3 Part representation

Part pose and size We predict part pose and size B as a set of part center c ∈ R 3 , rotation R ∈ SO (3) and size s ∈ R 3 for each query. We predict c as an offset ∆ c from q , added to the query coordinates, i.e., c = q +∆ c .

Part shape We employ a shared-weight, single implicit shape decoder O for performing part-wise shape reconstruction by taking point clouds around the detected regions where parts are identified. Given the diversity in shape and pose, and anisotropic scaling of the parts we focus on, it is challenging to learn shape bias with a single shape decoder. Therefore, we propose anisotropically normalizing the side lengths of the shape decoder's input and output to a unit scale to perform reconstruction. We define the input point cloud P O as follows:

where S denotes diagonal matrix of scale s . We define the output occupancy value at x ∈ R 3 as o x = O (( RS ) -1 ( x -c ) | P O , h ) , where h ∈ R D h is a part shape feature modeled by a part prediction MLP. Given that the input point cloud P O includes background, the geometry of the target part shape can be ambiguous. To address this issue, we train the detector backbone and shape decoder end-to-end, by inputting h as shape geometry to the shape decoder so that h informs the shape decoder of the foreground target shape. We utilize a shape decoder architecture with a lightweight local geometry encoder [42] to spatially associate input points P O with output occupancy values o x , which are both defined in normalized space.

Part kinematics We predict a 4 -dimensional vector y as a probability distribution over part joint types. This includes a 'background' or 'not a part' type for instances where predicted part proposals might not contain a part. The revolute origin v = q +∆ v is predicted similarly to the part center c , with an offset ∆ v . Joint states d current and d max are modeled by separate part prediction MLPs for revolute and prismatic types, and we only supervise the output corresponding to the ground truth joint type. A single MLP is used for joint direction a for both revolute and prismatic types.

3.4 Instance representation

We form a collection of parts as a single instance G l = { i ∈ G | ∀ i, j ∈ G,i = j ⇒∥ z i -z j ∥ < τ z } , where the distance between the D z -dimensional embeddings, z i and z j , of each part pair within the group is kept below a specified threshold τ z . We predict an N u -dimensional vector u as a probability distribution over instance categories per part. At test time, we predict the instance category by taking the category prediction with the highest confidence from the category predictions of the parts belonging to the same instance.

3.5 Refiner R

Query features are iteratively refined in the decoder D in the feature space [37]. Increasing the number of decoder layers can enhance performance at the expense of a larger model size [37]. However, it is

̸

crucial to avoid increasing the model size to enable efficient end-to-end training from detection to shape reconstruction. To improve model performance while maintaining the model size, we found cross-refining both output space and feature space to be effective. We introduce a refiner R , which has an identical architecture with the decoder D . The refiner R serves to refine the prediction in output space by reallocating a portion of decoder layers from decoder D to R . Assuming there are N D + R decoder layers in the original model, we reallocate N R decoder layers to the refiner R . Consequently, N D = N D + R -N R layers are used for query feature refinement in decoder D . The refiner R refines part pose and size B ′ and joint parameter A ′ from D by predicting residuals ∆ B and ∆ A to produce refined prediction B and A , respectively. The architecture of the refiner R is shown in Fig. 3.

3.6 Kinematics-aware part fusion (KPF)

Articulated objects often consist of small and thin parts. Especially for unsegmented inputs, only a small portion of the point cloud represents such parts after subsampling. To ensure a sufficient number of queries cover such parts and their surrounding context, at test time, we randomize the sampling positions of the queries and independently carry out inference and NMS for the input N Q times, obtaining a collection of densely sampled part proposals from N Q inference runs. This process is termed Query Oversampling (OQ). However, OQ can increase false positives and degrade detection performance. To mitigate this, we propose Part Fusion (PF), inspired by Weighted Box Fusion (WBF) [48], to merge overlapping part proposals, thereby reducing false positives and improving detection accuracy. Unlike WBF, which fuses only 2D bounding parameters, PF fuses all the predicted parameters of part proposals, with an average weight defined as objectness 1 -y bg. However, using IoU with 3D bounding boxes as overlapping metrics yields overly small values for thin structured parts, even when they are proximate. This results in PF failing to merge redundant parts, leading to false positives, as illustrated in Fig. 4 (a). To overcome this challenge, we propose a kinematics-aware IoU (kIoU) based on the observation that a redundant part pair exhibits significantly overlapping trajectories. To calculate kIoU, we construct a convex hull for each part in the pair using the 24 vertices of three bounding boxes based on size and canonical pose, current pose, and fully opened pose using the predicted part pose and size B and joint parameter A , as show in Fig. 4 (b). Then, we calculate the IoU between the two hulls. The comparison between the 3D bounding box IoU and kIoU is depicted in Fig. 4 (c). Our overall method, termed Kinematic-aware Part Fusion (KPF), includes: (1) executing inference and NMS using kIoU to gather part proposals multiple times, (2) conducting PF using kIoU to update these proposals iteratively, and (3) removing proposals with low objectness. Note that KPF is non-differentiable and is disabled during training, with N Q = 1 . For further details, please refer to the appendix.

3.7 Set matching and training loss

Set matching We base our end-to-end training of detection to reconstruction by utilizing 1-to-1 matching between part proposals and ground truth, using bipartite matching [5] for loss calculation. Similarly to [37], we define a matching cost between a predicted part and a ground truth part as C match = λ 1 ∥ B -B GT ∥ 1 + λ 2 ∥ c -c GT ∥ 1 -λ 3 y y + λ 4 (1 -y bg ) , where B defines eight vertices of cuboid defined by part pose and size B , y y defines the joint type probability given the ground truth label y , and 1 -y bg defines the foreground probability. Deviating from [37], we use the L1 distance of eight vertices of a cuboid instead of the GIoU [44] of cuboids in the first term to avoid the costly calculation of enclosing hull for 3D rotated cuboids.

Training losses For each pair of prediction and ground truth, we define part loss as

All loss terms have equal weights. The first term is a disentangled L1 loss described in [47] to optimize B . This loss is replicated three times by using only one of the predicted three components ( R , c , s ) for B , while replacing the other three with their ground truth values. We also found that directly optimizing rotation, as in the second term of the loss, leads to smaller rotation loss during training. BCE and CE denote binary cross-entropy and cross-entropy loss, respectively. PL denotes

the point-line distance between the revolute origin v i and the ground truth joint axis defined by v GT i and a GT i . For unrefined prediction B ′ and A ′ , we define the same loss as L part except for o x , y i and u i denoted as L ′ part . During training, we use ground truth B for Eq.1 to avoid noisy prediction adversely affecting shape learning.

We also define instance loss L instance for learning part-to-instance association with modified improved triplet loss [8] defined as:

where G l | i = { j ∈ G l | δ li = 1 } and η ij = ∥ z i -z j ∥ denotes the L2 distance between the part-to-instance association embeddings of the i -th and j -th parts. The first term enforces the distance between the two embeddings of two different parts belonging to the same instance below τ ′ z and the second term ensures the distance between embeddings of parts belonging to different instances is larger than 3 τ ′ z . However, computing all combinations of triplets for the second term is operationally complex for part-wise supervision. To streamline this, we instead opt to maximize the difference between the upper bound and the lower bound of inter- and intra-instance distances of the embeddings. In practice, we replace max and min with their soft approximations defined as LogSumExp ( η ij ) and -LogSumExp ( -η ij ′ ) for smooth gradient propagation. During inference, we use a threshold τ z = 1 2 (3 τ ′ z + τ ′ z ) to determine if two parts belong to the same instance based on the distance between their embeddings.

The total loss we minimize is L total = L part + L ′ part + L instance. At training time, we use the same part prediction MLPs to predict part parameters at every layer in the decoder. We compute the L total for each layer independently and sum all the losses. We only use the part parameter predicted from the last decoder layer at test time. Full loss formulation can be found in the appendix.

3.8 Implementation detail

The implementation of the transformer encoder-decoder architecture and the optimizer configuration are based on the publicly available code of 3DETR [37]. We employ a masked encoder architecture for E from [37]. The input point cloud to the encoder consists of N P = 32768 points, which are subsampled to N P ′ = 2048 points. The number of queries is set to N q = 128 during training. At test time, we set it to N q = 512 , and with independent runs for query oversampling as N Q = 10 . The number of decoder layers for D and R are set to N D = 6 and N R = 2 , respectively. For the evaluation of synthetic data, the foreground masks are inferred by a pretrained segmentation model using a ResNext50 [59] encoder and a DeepLabV3Plus [6] segmentation head with RGBD input. To evaluate real-world data, we automatically generate foreground masks using https://remove.bg, following [56]. The weights for the matching cost C match were set to λ 1 = 8 , λ 2 = 10 , λ 3 = 1 , λ 4 = 5 . We set λ intra = 0 . 1 for the intra-instance loss term of the instance loss L instance. We use the AdamW [31] optimizer with a base learning rate of 9e-4 and employ a cosine scheduler with nine epochs for warm-up. The training was performed using two A100 GPUs, each 40GB of GPU memory, and with a batch size of 26 for each GPU. The training took approximately two days. Please refer to the appendix for further details regarding the training process, hyperparameters, and network architecture.

4 Experiments

Datasets We evaluate our method on both synthetic and real-world data. We use the SAPIEN [58] dataset for synthetic data evaluation, following recent works on articulated shape reconstruction [24, 56, 15]. We select ten categories with representative articulation types, a sufficient number of CAD models, and various part structures across categories. We then randomly construct room

Input

GT

A-SDF-GT-2

Ours

Input

Ours

geometry (wall and floor) and place one to four instances per scene. Each instance is generated by applying random horizontal flips, random anisotropic resizing of side lengths, and random articulation of CAD models. The camera pose is sampled randomly, covering the upper hemisphere of the scene. Instances with severe truncation from the view frustum or occlusion with other instances are ignored during training and evaluation, ensuring that least one instance is visible in a view. For the training split, we randomize the textures of parts and room meshes. We use the original textures from the SAPIEN dataset for the test split. We generated 188,726 images for training and validation. Due to computational and time constraints, we used 20,000 images for training and kept the rest for validation usage. Also, we generated 4,000 images for the test split. Image size is 360 × 640 in height and width. The data overview is shown in Table 2. More details on data preparation can be found in the appendix.

For real-world data, we use the BMVC [36] dataset for quantitative evaluation. We use cabinet class which includes both prismatic and revolute parts and has the same part count and similar object shape as those used in our training and baseline models. The data contains two sequences of RGBD frames, capturing the same target objects with different part poses from various camera poses. We also test our method on RGB images taken with an iPhone X and depth maps generated from partial front views of the scene using Nerfacto [50].

Metrics For shape evaluation, following detection-based object reconstruction studies [40, 51, 16], we report mAP combined with standard shape evaluation metrics denoted as metric@threshold. We use F-Score [52], Chamfer distance (CD), and volumetric IoU (IoU) with multiple matching thresholds. More details on shape metrics can be found in the appendix. The shape mAP evaluation includes all the predicted parameters except for part kinematic parameters, serving as a comprehensive proxy for their quality. For part-wise detection with part pose and size, we use the average L2 distance

Input

GT

A-SDF-GT-2

Ours

Input

GT

A-SDF-GT-2

Ours

between the corresponding eight corners of bounding boxes between the ground truth and prediction as a matching metric of mAP, similar to the ADD metric [17], denoted as mAP@threshold. The distance is normalized by the ground truth part's diameter, with a proportion of the diameter used as the threshold. Since IoU-based metrics can yield values close to zero for thin structured parts, even when they are reasonably proximate and slightly off the ground truth, and do not consider part orientation, we use the above matching metrics to better analyze part-wise detection. For part kinematics evaluation, we evaluate the absolute error on joint state (State), Orientation Error (OE) for joint direction, and Minimum Distance (MD) for rotation origin following [22] for the detected parts with matched ground truth.

Baselines By default, we denote the proposed method that takes the foreground mask as 'Ours', and the model taking unsegmented input as 'Ours-BG'. To the best of our knowledge, no prior work operates on exactly the same problem setting as ours. For shape reconstruction, we benchmark against the state-of-the-art category-level object reconstruction method A-SDF [38], the closest to our approach. We follow the most recent work setup [15] in evaluation. A-SDF requires instance segmentation, pose, and scale to project the object from the camera to the object-centric frame. We implement it using all the required data with ground truth, denoted as A-SDF-GT following [15]. Given A-SDF's assumption of a fixed part count per model, as per a similar setup of [22], we train it on the dataset's most frequent part count per category. To account for multiple part counts in a category, for categories with more than one part count, we train the second model for the next most frequent count, resulting in a maximum of two models per category, termed A-SDF-GT-2. When the input instance has an untrained part count at test time, the model trained on the most frequent part count is used for A-SDF-GT-2. In the mAP evaluation, we give GT correspondence between the prediction and ground truth; thus, a false positive happens only when the shape metric is lower than the threshold for A-SDF. For kinematic evaluation, we compare our method with the state-of-the-art single-view part kinematic estimation method OPD [22]. Note that we input OPD with an RGBD image to align input modality and modify it to output joint state. More details on baselines can be found in the appendix.

4.1 Shape reconstruction

We show the shape mAP result in Table 1. Our method outperforms A-SDF [38] with ground truth in all metrics, and Ours-BG outperforms in the majority of metrics. We show the qualitative results in Fig. 5. Our models effectively reconstruct multiple instances with diverse part counts and structures, outperforming A-SDF which struggles with reconstructing articulated parts. We attribute our method's superior performance to its part-level representation, which facilitates easier learning of various part structures. Moreover, our shape-decoder is less affected by shape variations arising from the combination of part structures and poses. The qualitative results from novel viewpoints and additional visualizations can be found in the appendix.

4.2 Kinematic estimation

OPD [22] performs 2D detection evaluated by 2D IoU, while ours on 3D by L2 distance between 3D bounding box corners. To make kinematics estimation results of OPD and ours comparable, we experimentally choose the matching threshold of our mAP as 70% so that a similar number of detected parts with OPD's result with 2D segmentation mAP@50%. Then, we select the intersection of true positive detected parts by OPD and ours. The result is shown in Table 3. Our methods outperform OPD significantly. We attribute the reason to our method operating on 3D point clouds

Input

w/oKPF

Input

w/oKPF

GT

W/KPF

GT

W/KPF

(a)Ambiguousopeningdirection

QO+PF w/ kloU (KPF)

(b)Physicallyimplausiblefalsepositive

Figure 10: Failure cases.

while OPD is on 2D. Thus, our method is more robust to various textures and lighting conditions and makes it easier to reason about 3D kinematics. Note that our focus here is kinematics estimation after the detection step. Thus, superiority in detection performance is not our focus.

4.3 Ablation studies

In the following ablation studies, we validate each proposed component in a challenging setting where the input to the encoder has an unsegmented background using Ours-BG.

Kinematics-aware part fusion Weshow quantitative results in Table 4. Besides mAP, we also show precision considering false positives. QO denotes test-time query oversampling, and PF indicates part fusion. As a baseline, we first disable all components (w/o QO, PF, kIoU) using the same number of queries N q = 128 during training and add each component one by one. Introducing our proposed kIoU on top of QO and PF outperforms the baseline in shape reconstruction mAP and part detection while preserving similar precision. We visualize the effectiveness of the KPF module in Fig. 8. In the provided comparison, 'w/o KPF' denotes disabling QO, PF and kIoU. We see that KPF enhances the detection and pose estimation of small parts. We also show the qualitative results on the proposed kIoU in Fig. 9. Applying QO and PF alone leads to false positives of thin parts, which kIoU effectively reduces. It indicates that the proposed KPF module improves overall detection performance while suppressing false positives.

Anisotropic scaling and end-to-end training for shape learning We validate the effect of anisotropic scale normalization and end-to-end training in shape mAP evaluation. In Table 5, w/o AS denotes using isotropic scaling by normalizing the maximum side length to one instead of all sides. w/o SF denotes not passing shape feature h but training shape decoder O separately. Disabling each component degrades performance. Especially disabling anisotropic scaling significantly drops the performance, as the single shape decoder is tasked to decode various sizes of target shapes.

Ratio of decoder layers in the refiner R We investigate the relationship between the proportion of decoder layers in R and performance in shape mAP. The result is shown in Fig. 11. We vary the ratio of decoder layers in the refiner N R /N D + R from 0% to 75%. Allocating a portion (25%) of decoder layers to the refiner improves performance with the same number of decoder layers while reducing excessive decoder layers from the decoder degrades performance.

4.4 Real-world data

We verify the generalizability of our approach, which is trained only on synthetic data to real-world data. Here, we include foreground masks as inputs to mitigate the domain gap from background.

We present the quantitative results on the BMVC [36] dataset in Table 6. As only one instance is present in the scene, we evaluate with shape metrics without mAP. The CD value is multiplied by 100. Our method outperforms A-SDF [38]. We observe reasonable generalization as shown in Fig. 7. Furthermore, we also show the qualitative result in Fig. 6 for scenes containing multiple instances. We observe reasonable generalization to real-world data.

5 Limitations

While our method makes significant strides in the daily articulated object reconstruction task, it does have several limitations. (1) Our method cannot handle cases where the instance boundary is not well defined, such as a door directly attached to a room. 3D reconstruction of room geometry with part-level shape reconstruction of articulated parts is an interesting future direction for scene reconstruction task. (2) The proposed approach currently does not consider the physical constraints between parts during training. Although the KPF module removes the redundant parts as postprocessing, physically implausible false positives still occur as shown in Fig. 10 (a) for parts without overlapping trajectories. Introducing regularization, such as a physical violation loss [61], for such implausible configurations would alleviate this problem. (3) Our method does not explicitly consider the ambiguity in the opening direction of a part. As shown in Fig. 10 (b), the model fails to estimate the correct axis direction w.r.t. the base part for the closed part of the oven. This is because the dataset includes parts with different opening directions but similar shapes when closed. Such ambiguity can be addressed by explicit uncertainty modeling as in [1, 11], and integrating finer 2D visual cues for 3D reasoning [54, 26, 7] for localizing knobs and handles, which are informative for opening direction. Further discussions on limitations can be found in the appendix.

6 Conclusion

We presented an end-to-end trainable part-level shape reconstruction method for multiple articulated objects from a single RGBD image. We have demonstrated that our method successfully tackles the major limitation of previous works, which are unable to handle objects with various part counts using a single model, by employing a novel detect-then-group approach. We have also shown that the proposed kinematic part fusion (KPF) module effectively handles small parts as challenging targets while suppressing false positives for detection-based reconstruction. Our method outperformed the state-of-the-art baselines in shape reconstruction and kinematics estimation on the SAPIEN [58] dataset. Furthermore, on the BMVC [36] dataset and casually captured images, we demonstrated that the model trained on synthetic data reasonably generalizes to real-world data.

Acknowledgments

We appreciate the members of the Machine Intelligence Laboratory for constructive discussion and their insightful feedback during the research meetings. This work was partially supported by JST Moonshot R&D Grant Number JPMJPS2011, CREST Grant Number JPMJCR2015 and Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo.

References

Abbatematteo, B., Tellex, S., Konidaris, G.: Learning to generalize kinematic models to novel objects. In: Proceedings of the Conference on Robot Learning (CoRL). pp. 1289-1299 (2020)

Beker, D., Kato, H., Morariu, M.A., Ando, T., Matsuoka, T., Kehl, W., Gaidon, A.: Monocular differentiable rendering for self-supervised 3d object detection. In: European Conference on Computer Vision (ECCV). pp. 514-529 (2020)

Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., Black, M.J.: Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 561-578 (2016)

Brégier, R.: Deep regression on manifolds: a 3D rotation case study. In: International Conference on 3D Vision (3DV) (2021)

Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European Conference on Computer Vision (ECCV). pp. 213-229 (2020)

Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 801-818 (2018)

Chen, Z., Li, Z., Zhang, S., Fang, L., Jiang, Q., Zhao, F.: Deformable feature aggregation for dynamic multi-modal 3d object detection. In: European Conference on Computer Vision (ECCV). pp. 628-644 (2022)

Cheng, D., Gong, Y., Zhou, S., Wang, J., Zheng, N.: Person re-identification by multi-channel parts-based cnn with improved triplet loss function. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

Conant, J., Michaels, T.: On the number of tilings of a square by rectangles. European Journal of Combinatorics 18 , 21-34 (2014)

Denninger, M., Winkelbauer, D., Sundermeyer, M., Boerdijk, W., Knauer, M., Strobl, K.H., Humt, M., Triebel, R.: Blenderproc2: A procedural pipeline for photorealistic rendering. Journal of Open Source Software 8 (82), 4901 (2023)

Desingh, K., Lu, S., Opipari, A., Jenkins, O.C.: Factored pose estimation of articulated objects using efficient nonparametric belief propagation. In: 2019 International Conference on Robotics and Automation (ICRA). pp. 7221-7227 (2019). https://doi.org/10.1109/ICRA.2019.8793973

Garland, M., Heckbert, P.S.: Surface simplification using quadric error metrics. In: Proceedings of the 24th annual conference on Computer graphics and interactive techniques. pp. 209-216 (1997)

Geng, H., Xu, H., Zhao, C., Xu, C., Yi, L., Huang, S., Wang, H.: Gapartnet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)

Gkioxari, G., Malik, J., Johnson, J.: Mesh r-cnn. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019)

Heppert, N., Irshad, M.Z., Zakharov, S., Liu, K., Ambrus, R.A., Bohg, J., Valada, A., Kollar, T.: Carto: Category and joint agnostic reconstruction of articulated objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)

Heppert, N., Migimatsu, T., Yi, B., Chen, C., Bohg, J.: Category-independent articulated object tracking with factor graphs. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 3800-3807 (2022)

Hinterstoisser, S., Lepetit, V ., Ilic, S., Holzer, S., Bradski, G., Konolige, K., Navab, N.: Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In: Proceedings of the Asian Conference on Computer Vision (ACCV) (2013)

Iakubovskii, P.: Segmentation models pytorch. https://github.com/qubvel/ segmentation_models.pytorch (2019)

Irshad, M.Z., Kollar, T., Laskey, M., Stone, K., Kira, Z.: Centersnap: Single-shot multi-object 3d shape reconstruction and categorical 6d pose and size estimation. In: IEEE International Conference on Robotics and Automation (ICRA) (2022)

Irshad, M.Z., Zakharov, S., Ambrus, R., Kollar, T., Kira, Z., Gaidon, A.: Shapo: Implicit representations for multi object shape appearance and pose optimization. In: European Conference on Computer Vision (ECCV) (2022)

Jain, A., Lioutikov, R., Chuck, C., Niekum, S.: Screwnet: Category-independent articulation model estimation from depth images using screw theory. In: IEEE International Conference on Robotics and Automation (ICRA). pp. 13670-13677 (2021)

Jiang, H., Mao, Y., Savva, M., Chang, A.X.: OPD: Single-view 3D openable part detection. In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)

Jiang, Z., Hsu, C.C., Zhu, Y.: Ditto: Building digital twins of articulated objects from interaction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

Kawana, Y., Mukuta, Y., Harada, T.: Unsupervised pose-aware part decomposition for manmade articulated objects. In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)

Kulkarni, N., Gupta, A., Fouhey, D.F., Tulsiani, S.: Articulation-aware canonical surface mapping. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 452-461 (2020)

Li, H., Zhang, Z., Zhao, X., Wang, Y., Shen, Y., Pu, S., Mao, H.: Enhancing multi-modal features using local self-attention for 3d object detection. In: European Conference on Computer Vision (ECCV). pp. 532-549 (2022)

Li, X., Wang, H., Yi, L., Guibas, L.J., Abbott, A.L., Song, S.: Category-level articulated object pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3706-3715 (2020)

Liu, L., Xu, W., Fu, H., Qian, S., Yu, Q., Han, Y., Lu, C.: Akb-48: a real-world articulated object knowledge base. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 14809-14818 (2022)

Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: A skinned multiperson linear model. ACM transactions on graphics (TOG) 34 (6), 1-16 (2015)

Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH). p. 163-169 (1987)

Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: Proceedings of the International Conference on Learning Representations (ICLR) (2018)

Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Ojea, J.A., Goldberg, K.: Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics (2017)

Manhardt, F., Kehl, W., Gaidon, A.: Roi-10d: Monocular lifting of 2d detection to 6d pose and metric shape. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2069-2078 (2019)

Mao, Y., Zhang, Y., Jiang, H., Chang, A.X., Savva, M.: Multiscan: Scalable rgbd scanning for 3d environments with articulated objects. In: Advances in Neural Information Processing Systems (NeurIPS) (2022)

Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4460-4470 (2019)

Michel, F., Krull, A., Brachmann, E., Ying Yang, M., Gumhold, S., Rother, C.: Pose estimation of kinematic chain instances via object coordinate regression. In: Proceedings of the British Machine Vision Conference (BMVC). pp. 181.1-181.11 (2015)

Misra, I., Girdhar, R., Joulin, A.: An end-to-end transformer model for 3d object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2021)

Mu, J., Qiu, W., Kortylewski, A., Yuille, A., Vasconcelos, N., Wang, X.: A-sdf: Learning disentangled signed distance functions for articulated shape representation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13001-13011 (2021)

Nie, Y., Han, X., Guo, S., Zheng, Y., Chang, J., Zhang, J.J.: Total3dunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

Nie, Y., Hou, J., Han, X., Niessner, M.: Rfd-net: Point scene understanding by semantic instance reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

Noguchi, A., Iqbal, U., Tremblay, J., Harada, T., Gallo, O.: Watch it move: Unsupervised discovery of 3D joints for re-posing of articulated objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

Peng, S., Niemeyer, M., Mescheder, L., Pollefeys, M., Geiger, A.: Convolutional occupancy networks. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)

Reading, N.: Generic rectangulations. European Journal of Combinatorics 33 (4), 610-623 (2012)

Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union: A metric and a loss for bounding box regression. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR). pp. 658-666 (2019)

Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In: The IEEE/CVF International Conference on Computer Vision (ICCV) (October 2019)

Saito, S., Simon, T., Saragih, J., Joo, H.: Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In: CVPR (2020)

Simonelli, A., Bulo, S.R., Porzi, L., López-Antequera, M., Kontschieder, P.: Disentangling monocular 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1991-1999 (2019)

Solovyev, R., Wang, W., Gabruseva, T.: Weighted boxes fusion: Ensembling boxes from different object detection models. Image and Vision Computing (2021)

Sturm, J., Stachniss, C., Burgard, W.: A probabilistic framework for learning kinematic models of articulated objects. Journal of Artificial Intelligence Research 41 , 477-526 (2011)

Tancik, M., Weber, E., Ng, E., Li, R., Yi, B., Kerr, J., Wang, T., Kristoffersen, A., Austin, J., Salahi, K., et al.: Nerfstudio: A modular framework for neural radiance field development. arXiv preprint arXiv:2302.04264 (2023)

Tang, J., Chen, X., Wang, J., Zeng, G.: Point scene understanding via disentangled instance mesh reconstruction. In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)

Tatarchenko, M., Richter, S.R., Ranftl, R., Li, Z., Koltun, V., Brox, T.: What do single-view 3d reconstruction networks learn? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3405-3414 (2019)

Tseng, W.C., Liao, H.J., Yen-Chen, L., Sun, M.: Cla-nerf: Category-level articulated neural radiance field. In: IEEE International Conference on Robotics and Automation (ICRA). pp. 8454-8460 (2022)

Tyszkiewicz, M.J., Maninis, K.K., Popov, S., Ferrari, V.: Raytran: 3d pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 211-228 (2022)

Watson, J., Aodha, O.M., Turmukhambetov, D., Brostow, G.J., Firman, M.: Learning stereo from single images. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)

Wei, F., Chabra, R., Ma, L., Lassner, C., Zollhöfer, M., Rusinkiewicz, S., Sweeney, C., Newcombe, R., Slavcheva, M.: Self-supervised neural articulated shape and appearance models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

Weng, Y., Wang, H., Zhou, Q., Qin, Y., Duan, Y., Fan, Q., Chen, B., Su, H., Guibas, L.J.: Captra: Category-level pose tracking for rigid and articulated objects from point clouds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR). pp. 13209-13218 (2021)

Xiang, F., Qin, Y., Mo, K., Xia, Y ., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y ., Wang, H., et al.: Sapien: A simulated part-based interactive environment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11097-11107 (2020)

Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1492-1500 (2017)

Yang, G., Vo, M., Neverova, N., Ramanan, D., Vedaldi, A., Joo, H.: Banmo: Building animatable 3d neural models from many casual videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2863-2873 (2022)

Zhang, C., Cui, Z., Zhang, Y., Zeng, B., Pollefeys, M., Liu, S.: Holistic 3d scene understanding from a single image with implicit representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

Zhou, Y., Barnes, C., Lu, J., Yang, J., Li, H.: On the continuity of rotation representations in neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

Zuffi, S., Kanazawa, A., Berger-Wolf, T., Black, M.J.: Three-d safari: Learning to estimate zebra pose, shape, and texture from images" in the wild". In: Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR). pp. 5359-5368 (2019)

Zuffi, S., Kanazawa, A., Jacobs, D.W., Black, M.J.: 3d menagerie: Modeling the 3d shape and pose of animals. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 6365-6373 (2017)

A Details on kinematics-aware part fusion (KPF)

In Algorithm 1, we present the details of kinematics-aware part fusion. We gather part proposals through N Q independent inference runs, achieved by randomly sampling query positions { q n } with the furthest point sampling (FPS). The 'model' in the algorithm represents the decoder D and the refiner R , and y fg represents objectness, defined as 1 -bg.

After part proposals are generated, we apply Non-Maximum Suppression (NMS) with 3D bounding box IoU and threshold by objectness 1 -y bg before applying NMS with kinematics-aware IoU (NMS-kIoU) for performance reasons. PF-kIoU denotes the part fusion (PF) process using kIoU, which is based on the Weighted Box Fusion (WBF) approach as described in [4]. The procedure is further detailed in Algorithm 2.

Unlike the conventional WBF, our algorithm fuses all parameters in the part proposal rather than only the 2D bounding box parameters. Furthermore, our algorithm iteratively runs until convergence, as demonstrated from line 11 to line 15 in Algorithm 1. C represents clusters of overlapping part proposals identified by kIoU. We employ a simple arithmetic weighted average for all elements except the rotation matrix in a part proposal for fusion, denoted as weighted-average.

For the rotation matrix R ∈ SO (3) , we initially derive a weighted average for the 3 × 3 matrices, labeled as R wa ∈ R 3 × 3 . We then compute the weighted-averaged rotation matrix by minimizing the Frobenius norm to R wa. This is accomplished as R = argmin R ∥ R -R wa ∥ F , following the 3D rotation library RoMa [4].

The value of y ′ fg ,n ′ denotes the scaled objectness as a confidence score, considering the number of independent inferences N Q . If the number of part proposals in a cluster is fewer than the independent inference runs N Q , it indicates that only a limited number of independent inferences predict it. In such cases, we adjust the corresponding confidence score y ′ fg ,n ′ by scaling it down by the ratio of the number of part proposals in the cluster ( | C [ n ′ ] | ) to the number of independent inferences T = N Q . Conversely, we scale up the confidence score if a cluster contains more part proposals than the independent inferences N Q . For the initial run of part fusion (line 13 of Algorithm 1), we set T = N Q to fuse N Q inferences. We assign T = 1 for subsequent runs, assuming we apply part fusion to a single inference run fused by the previous part fusion.

Hyperparameters The hyperparameter τ IoU is set to 0.25, following the original code of 3DETR [37]. We set τ obj to 0.25, representing the uniform probability of the three joint types plus background class, as outlined in Section 3. Additionally, we simplify the hyperparameter by setting τ obj_final equal to τ obj. The hyperparameter τ kIoU is assigned a value of 0.5, double the τ IoU value, to impose a stricter overlapping threshold and ensure similar trajectory part proposals are clustered together. We experimentally determine the values for τ scaled and τ count, setting them to 0.1 and 3, respectively.

B Full Loss Formulation

During training, we use the same multi-layer perceptron (MLP) for part prediction at each decoder layer in the decoder D . We calculate the total loss, L total, as defined in Section 3.7, independently for each layer and sum up these losses. The comprehensive loss formulation is as follows:

C Architecture

C.1 Detection backbone

Our encoder and decoder architectures strictly follows the ones proposed in 3DETR [37]. We employ the masked-encoder from [37] as the encoder architecture for improved performance. We also employ the Fourier positional encoding, as proposed in [37], for the query embedding prior to input to the decoder. Please refer to [37] for comprehensive details and visualizations of the architectures.

Algorithm 1 Kinematic-aware part fusion (KPF)

Input:

Subsampled point cloud P , scene feature F

′ ′

Algorithm 2 Part Fusion with kIoU (PF-kIoU)

C.2 Part prediction MLPs

We adapt the box prediction MLPs outlined in [37] as our part prediction MLPs. Our part prediction MLPs retain the same architecture and are referred to as 'MLPs' in Fig. 3 of the decoder architecture. We employ 13 MLPs, each dedicated to a specific part proposal parameter. These parameters include the center offset from the query position ∆ c , rotation R 6D, size s , current pose joint state for prismatic and revolute types d pris current and d rev current , fully-opened pose joint state for prismatic and revolute types d pris max and d rev max , joint axis a , revolute origin offset from the query position ∆ v , joint type y , category u , shape feature h , and part-to-instance embedding z . We utilize the 6D rotation representation [62] as the MLP's output representation R 6D, subsequently converted to a 3 × 3 rotation matrix.

C.3 Refiner

Our refiner architecture closely follows the decoder architecture, with MLPs outputting residual parameters. It follows the same architecture of the part prediction MLPs from the decoder, except for the dropout layers. We employ nine MLPs for residual predictions of center, rotation, size, prismatic and revolute types' current pose joint state, prismatic and revolute types' fully-opened pose joint state, joint axis, and revolute origin. For the rotation matrix and joint axis, the MLPs output 6D rotation representation which are then converted into 3 × 3 rotation matrices. These are updated by matrix product. The remaining parameters are updated through the addition of residuals. The same positional encoding is applied to the part center c to generate an embedding feature, which is then added to the corresponding refined query embedding e N D from the decoder's final layer. This forms the new query feature input for the refiner, as depicted in Fig. 3.

C.4 Shape decoder

Our shape decoder is based on the ConvONet architecture [42]. It consists of a local point encoder, a volume encoder, and an implicit shape decoder. The local point encoder quantizes input points to multiple 2D (tri-planes) or 3D grids and encodes points within each grid. The volume encoder then takes these locally encoded points and further encodes them into a 3D voxel feature. To decode the occupancy value at a 3D point x , we sample the local feature h local , x by interpolation on the 3D voxel feature at x . The implicit shape decoder outputs the occupancy value o x at x , given h local , x as input.

Our approach differs from the original ConvONet in that we concatenate the shape feature per part proposal h to the local feature to form h ′ local , x = h local , x ⊕ h . This serves as the learned part shape prior for the implicit shape decoder that reduces shape ambiguity due to the unsegmented per-part point cloud P O , as outlined in Eq. 1. We utilize the lightweight tri-plane architecture proposed in [42] for the volume encoder architecture. For more detailed information on the ConvONet architecture, please refer to [42].

C.5 Foreground segmentation model

Our segmentation model incorporates a ResNext50 [59] architecture for the encoder with RGBD input and DeepLabV3Plus [6] for the segmentation head. We use the off-the-shelf implementation from [18].

D Implementation details

D.1 Architecture

Our approach strictly follows the hyperparameters of the masked-encoder described in [37]. We employ three self-attention transformer layers with self-attention masks between points in the point cloud, and each point only attends to others within a specified radius. The input point cloud to the encoder consists of N P = 32768 points, which are subsequently subsampled to N P ′ = 2048 points in the encoder. We set the scene feature dimension D F a value of 256, following [37].

For the decoder, we maintain the hyperparameters from [37], except for the number of decoder layers N D . The decoder layers for the decoder D and the refiner R are set to N D = 6 and N R = 2 , respectively. We match the query embedding dimension with the scene feature dimension for addition

and set the number of queries to N q = 128 during training. For testing, N q is set to 512, unless stated otherwise, with independent runs for query oversampling (QO) for kinematics-aware part fusion (KPF) with N Q = 10 independent inference runs.

As for the part prediction Multi-Layer Perceptrons (MLPs), we follow the box prediction MLP hyperparameters in [37], except for the number and output dimension of each MLP. The MLPs in the refiner for residual prediction strictly follow the part prediction MLPs hyperparameters in the decoder but without dropout. We set the dimension for shape feature h = 128 , and the dimension for part-to-instance association embedding z = 32 .

With respect to the ConvONet [42] architecture in the shape decoder, the local point encoder consists of five MLPs of width 128, and the volume encoder has three MLPs with widths of 16, 32, and 64. We employ the tri-plane version of the volume encoder for volume representation, using only the xy and yz planes to save GPU memory. In the implicit shape decoder, we employ four fully-connected layers with a hidden dimension of 128 and a leaky ReLU activation.

D.2 Training details

We set the weights for the matching cost C match at λ 1 = 8 , λ 2 = 10 , λ 3 = 1 , λ 4 = 5 . Each loss term in the total loss L total carries equal weight. We use the AdamW [31] optimizer with a base learning rate of 9e-4 with a cosine scheduler down to a learning rate of 1e-6. The warm-up period is set to nine epochs, and mixed precision is used during training.

All models were trained on two A100 GPUs, each with 40GB GPU memory. We set the batch size to 26 per GPU, and it took approximately two days to complete 500 epochs. Weight decay was set at 0.1, with gradient clipping with an L2 norm of 0.1, as per [37].

We implemented on-the-fly sampling of 3D points and corresponding occupancy values to train the shape decoder. Half of these points are sampled uniformly from the space [ -0 . 5 , 0 . 5] 3 . Additionally, a quarter of the points are sampled around the surface with a Gaussian-distributed random offset along the surface normal direction, with a standard deviation of 0.1. The remaining quarter is sampled with a standard deviation of 0.01. Each part has 128 points sampled for occupancy values.

In training the foreground segmentation model, we used the AdamW optimizer and a cosine scheduler, identical to the approach used in training the previous models. This model was also trained on two A100 GPUs with 40GB GPU memory and a batch size of 26 per GPU. Training took approximately one day for 500 epochs.

We add noise to the depth map during training, following the approach in [32]. We incorporate a depth map filter for the model tested on real-world data to mitigate the flying pixel effect on the object edge as suggested by [55]. This filter is also applied during testing for real-world data. Following the projection of the depth map to 3D points, we introduced random scaling within a range of ± 15%, and random rotation along the surface normal direction within ± 30 ◦ . The point cloud was zero-centered for both training and testing. Color augmentation during training follows the original code of [37]. For the model using a foreground mask, we augment the mask by randomly displacing the foreground pixels around the border and performing successive dilation and erosion to simulate noise on inferred masks during training.

D.3 Mesh generation

We sample occupancy values on 64 3 voxel grids per part, then extract the surface mesh using the marching cubes method [30] at an isosurface level τ O = 0 . 3775 = Sigmoid ( -0 . 5) . For surface mesh reconstruction evaluation, we store an instance mesh as a union of part meshes and apply quadratic decimation [12] to reduce the number of faces to 10000. For volumetric IoU evaluation, where we evaluate the union of IoU for each part, we stored a part mesh and applied quadratic decimation to reduce the number of faces to 5000.

E Dataset Details

E.1 SAPIEN [58] dataset

We choose ten categories offering prismatic and revolute joints as representative articulation types, adequate CAD models, and various part structures. The CAD models are randomly divided into training + validation splits (referred to as the trainval split) and test split, maintaining an approximate 8:2 ratio. No overlapping CAD model IDs exist between these splits.

All CAD models are manually aligned such that they stand upward along the y-axis and face the z-axis. We use BlenderProc [10] to generate synthetic data. We randomly create room geometry (wall and floor) in each scene and place one to four instances.

For the trainval split, we randomize the texture of the part mesh and room geometries. The textures for the part mesh are randomly applied from https://www.blenderkit.com, using the downloading script of BlenderProc [10]. We employ 2923 textures for this split. However, for the test split, we use the original texture from the SAPIEN [58] dataset.

The room geometry textures are downloaded from https://ambientcg.com using the BlenderProc script [10] and randomly applied. We use 353 textures for the trainval split and 85 textures for the test split, ensuring no overlap.

For environmental lighting, HDR maps downloaded from https://polyhaven.com are used. We employ 113 HDR maps for the trainval split and 29 for the test split, with no overlap.

Instances are created by applying random horizontal flips, random anisotropic resizing of side lengths, and random articulation of CAD models. The maximum joint state of the revolute joint defaults to 135 ◦ unless it touches the ground. We set the maximum joint state value to avoid ground contact if it does. We use the CAD model's maximum prismatic joint state value for prismatic joints, adjusted for rescaling.

This paper considers instances comprising one base part and at least one articulated part attached to it. Therefore, we fix articulated parts attached to another articulated part and ignore their joints, such as small prismatic buttons and rotating knobs on doors.

The scene is populated with instances by first sampling one to four categories uniformly, with replacement. Then, a CAD model is randomly sampled for each selected category, and its part pose, size, and yaw rotation are randomly augmented. The size distribution is shown in Fig. 13. The yaw rotation is limited to ± 90 ◦ , ensuring instances face roughly forward. Instances are placed randomly in the scene, maintaining a minimum and maximum distance of 0.1m and 0.8m from other instances, respectively. This setup prevents instances from being scattered in the room.

We allow up to 10 sampling of camera poses in the same scene, limiting the camera position to the union of convex hulls of instances projected on the x-z plane (same as the floor) and a circular sector whose center is at the scene center and the arc is an angle between ± 80 ◦ . This restriction, coupled with the yaw range limitation, discourages the self-occlusion of parts by an instance facing backward to cameras. We illustrate this idea in Fig. 13.

We use the BlenderProc API to sample camera poses against instances with a minimum distance to a camera from an instance of 0.5m. In total, we render 4,000 frames for the test split and 188,726 for the trainval split, randomly sampling 20,000 frames for training and allocating the rest for validation usage.

We exclude instances whose 2D silhouette is occluded more than 50% or truncated by 25%. Each frame is ensured to have at least one visible instance. The image size is 360 × 640 (height and width), and the field of view is set to 86.64 ◦ .

For the generation of watertight mesh for the implicit shape ground truth in shape learning, we follow the instructions from [35] to generate part-wise watertight meshes. We then apply quadratic decimation [12] to the generated part mesh, reducing the face count to approximately 10,000 when combined as an instance.

3

2

0

Dishwasher

Trashcan

Safe

Oven

Storage

Table

Microwave

Fridge

Washing

Box

Category

E.2 BMVC[36] dataset

We use the cabinet class data from this dataset, which includes one prismatic part and one revolute part. The original data comprises two sequences of RGBD frames, with different part poses for each sequence and varied camera poses. The first sequence contains 1108 frames, while the second has 1119 frames. The image size is 480 × 640 (height and width).

Before feeding the data to the model, we apply a depth map filter to suppress the flying pixel effect on the object edge, as detailed in [55]. We extract the foreground point cloud using the silhouette of the projected CAD model on the image.

E.3 Images taken by iPhone X

We record the partial front view of the scene as a short clip, extracting approximately 300 frames to train the neural rendering model Nerfacto [50] for depth map generation. Default training hyperparameters are used for Nerfacto training. The image size is 540 × 960 (height and width). For our evaluation, foreground masks are automatically extracted using https://www.remove.bg unless specified otherwise. Note that we only use the point cloud lifted from a single RGBD image as input, not recovered from all frames.

F Details on baselines

F.1 A-SDF [38]

Data preparation We follow the description in the original A-SDF paper [38] for data preparation. After normalizing the global pose and scale, we ensure the base part's position does not change regardless of random articulation during training. In addition, for categories with multiple parts, we maintain consistent part ordering. We automate this by assigning part positions in canonical poses to a cell in the 5 × 5 × 5 voxel, which discretizes the positions. The 1D flattened cell positions of the voxel determine the part order. To limit the training time, we randomly sample a maximum of 1500 instances per category for the training split.

Instance's yaw

angle range

Camera position

Instance B

sampling area

Instance A

Camera

pose 1

Camera

pose 3

Camera

pose 2

Model The model implementation uses publicly shared author code. The original A-SDF only targets revolute parts. We normalize the prismatic joint state by the maximum joint state to fall within the [0 , 1] range. We then multiply this range by 135, ensuring joint states for revolute and prismatic parts are within the same range to extend the model for prismatic parts.

A-SDF is a category-level approach that works with a fixed number of parts, which becomes a problem when evaluated on dataset containing instances with varied part counts in a category. Similarly to our approach, OPD [22] can handle multiple part counts without assuming the predefined part number. They evaluate their method against baselines targeting a fixed number of parts by training the baselines for a category's most common part count.

We follow this protocol but aim to favor the baseline A-SDF more by training up to two models per category. The first model is for the most common part count, and the second, if applicable, is for the next most common part count. The second model is used when a category has more than two different part counts in the test split.

This setting differs from OPD's method, which only uses the most common part count. During testing, we use the model with the most frequent part count for instances with untrained part counts. We call the baseline model trained for the most frequent part count A-SDF-GT and the one using the next most frequent part count A-SDF-GT-2. The baselines and per category part counts, excluding the base part, are summarized in Table 8.

Training We employ the publicly shared author code for training. During training, each part's articulation varies randomly from 0 ◦ to 135 ◦ , sampled at 5 ◦ intervals as suggested in [38]. We dynamically sample 3D points and their corresponding signed distance values from the randomly articulated watertight meshes of parts during training. The number of sample points and the ratio of uniformly sampled points to points near the mesh surface follow the original A-SDF paper [38]. A-SDF model training uses part label supervision and takes approximately ten days on a single V100 or A100 GPU per model.

Mesh generation For mesh generation, we use the publicly shared author code as well. A-SDF assumes no background and normalized global pose and size for input. During testing, we use the ground truth instance segmentation mask for each instance in a scene to isolate instance-wise foreground points. We also normalize the depth map to the input space using the ground truth pose and size. The surface mesh is extracted using the provided implementation, which samples signed distance values on 64 3 voxel grids and extracts surface mesh using marching cubes [30]. Additionally,

we apply the quadratic decimation [12] to limit the number of faces to 10000. We then project the generated mesh back to the original scale and pose in the scene using the ground truth values.

F.2 OPD [22]

Data Following the original paper [22] and the publicly shared author code and dataset, we generate ground truth data. The original data includes a semantic label of part type, one of drawer, lid, or door. However, our synthetic dataset does not have corresponding labels. Thus we replace the semantic label with the joint type. Consistent with the original paper, we exclude the fixed type part from the ground truth, targeting only the revolute and prismatic parts for detection.

Model We employ the OPDRCNN-C model from [22], which predicts kinematic parameters in camera coordinates. For input modality alignment, the model uses RGBD input. We have adapted the model to produce a continuous 1D joint state; instead of supervising the joint state prediction head with binary joint state supervision, we use continuous 1D joint state supervision. We duplicate the MLP head to separately predict joint states for revolute and prismatic joints. We supervise only the output corresponding to the ground truth joint type. Furthermore, we guide the joint axis prediction with reference to the floor's normal direction, as we experimentally found this improves detection performance.

Training We follows the author's implementation for training code, maintaining the same settings and hyperparameters for the OPDRCNN-C model.

Testing As described in Sec. 4, we match the ground truth using part segmentation evaluation with an IoU threshold of 50% following the author's implementation. We then evaluate the predicted joint parameters against the matched ground truth.

G Category-wise shape mAP

We show category-wise shape mAP results in Table 9, 10, 11, 13, 14, 15, 16, 17, 18.

H Additional results on the refiner R

We evaluate the effectiveness of the refiner R in enhancing performance while maintaining a comparable model size. Table 19 shows the quantitative results on the shape mAP and the number of learnable parameters of the Ours-BG model, with the number of decoder layers in the decoder N D = 6 and in the refiner N R = 2 . We compare this model against the model without a refiner and having the same total number of decoder layers N D = 8 , and another model with a doubled number of decoder layers N D = 16 without the refiner. The result shows that using the refiner achieves comparable performance in shape mAP with the model with a doubled number of decoder layers N D = 16 with a smaller model size.

I Effect of the refiner on kinematic estimation

The effect of the refiner R on kinematic estimation is shown in Fig. 14. We conduct evaluations on joint parameter estimation. These evaluations focus on the intersection of true positive detections from both the model with and without the refiner with various mAP thresholds. Except for the joint axis orientation error at mAP@90%, the refiner improves the joint parameter estimation.

J Additional qualitative results on synthetic dataset

We show additional qualitative results on SAPIEN [58] dataset in Fig. 15.

K Additional qualitative results on real-world data

Additional qualitative results on real-world data are shown in Fig. 16. For the washing machine example in the top left corner of the figure, a semi-automatic segmentation tool on Microsoft PowerPoint was used for foreground segmentation by specifying the rough foreground region, as https://remove.bg could not properly segment the foreground in this example.

L Qualitatative results from novel viewpoint

Corresponding to Fig. 5 and Fig. 7 in 4, we visualize the qualitatative results from novel viewpoint on SAPIEN [58] dataset and BMVC [36] dataset in Fig. 17 and Fig. 18, respectively.

M Failure cases and limitations

Uncertainty in closed parts One failure case arises when our model attempts to estimate the orientation of a part opening while it is in its closed state, as illustrated in Fig. 19 (a). We show quantitative evaluation on axis direction error for closed parts in Table 20. We regard a part as closed when its ground truth motion amount is less than a threshold. For revolute joints, we use 5 ◦ and 10 ◦ as thresholds, and 5% and 10% of the fully opened amount for prismatic joint. As seen in the table, the model successfully estimates reasonable joint direction for prismatic parts, while for revolute

2

Revolute joint state error mAP@90%

Prismatic joint state error mAP@90%

Orientation Error mAP@90%

Minimum Distance mAP@90%

2.5 -

4.0

6-

3.5

2.0

5

3.0

2.5

Degree

Degree

1.5

3

1.5

1.0

2

1.0

0.5

0.5

0.0

0

0.0

0

w/ refine

w/o refine

w/ refine

w/o refine

w/ refine

w/o refine

w/ refine

w/o refine

Model Names

Model Names

Model Names

Model Names

Revolute joint state error mAP@80%

Prismaticjoint state error mAP@80%

OrientationError mAP@80%

MinimumDistance mAP@80%

5

6

3.5

4.0

6

4

5

3.0

5

Degree

3

Degree

4

2.5

3

2

1.5

2

1.0

1

0.5

1

w/ refine

w/o refine

w/ refine

w/o refine

0.0

w/ refine

w/o refine

0

w/ refine

w/o refine

Model Names

Model Names

Model Names

Model Names

Revolute joint state error mAP@70%

Prismatic joint state error mAP@70%

Orientation Error mAP@70%

Minimum Distance mAP@70%

5

6

6

5

4

Degree

Degree

Degree

3

3

3

3

1

1

0

0

w/refine

w/o refine

w/ refine

w/o refine

w/ refine

w/o refine

w/ refine

w/o refine

Model Names

Model Names

Model Names

Model Names

Revolute joint state error mAP@60%

Prismatic joint state error mAP@60%

Orientation Error mAP@60%

Minimum Distance mAP@60%

6

6

6

5

5

5

5

4

Degree

Degree

4

3

3

3

2

1

1

1

0

0

0

w/ refine

w/o refine

w/ refine

w/o refine

w/ refine

w/o refine

w/ refine

w/o refine

Model Names

Model Names

Model Names

Model Names

Table 20: Evaluation of joint axis direction error for closed parts. The part is regarded as closed if its ground truth motion amount is less than the threshold. For the prismatic part, we normalize the motion amount by its fully opened amount for thresholding to consider the different scales of parts. For reference, we denote the corresponding error of open parts in the parenthesis.

parts, the error is larger. Two contributing factors are: (1) our model does not explicitly model the uncertainty, particularly in ambiguous cases like the joint axis direction shown in Fig. 19 (a). The prismatic part's joint direction has less diversity due to the physical constraint by the base part, while several joint directions are possible for the revolute joint given the base part pose, thus the opening direction is ambiguous to the model. (2) The model could not detect small geometric cues from noisy depth maps, such as handles or knobs, which can provide valuable information about opening directions. These cases can be improved in several ways. For instance, we could extend the model to treat joint parameter ambiguity as a probability distribution, providing multiple joint parameter candidates sampled from the estimated distribution. This approach can still be practical, limiting the

Input

GT

A-SDF-GT-2

Ours

kinematics parameter space to a few options for initial interaction steps in robotics applications or allowing humans to choose the best prediction in AR/VR applications during a human-in-the-loop process. Previous work [1, 11] has shown the effectiveness of this probability-based approach. Another approach would be to integrate 2D and 3D features into the detection backbone [54, 26, 7]. Incorporating 2D features would enhance the encoder's ability to capture small yet informative features, like handles or knobs, that may be missed by a sparse point cloud representation alone.

Unseen categories Another failure case is that our model struggles to reconstruct instances from unseen categories, particularly those with shapes and part structures with small samples or absent in training data. Fig. 19 (b) shows the failure case with a dehumidifier as an unseen category, which has an unseen drawer shape and part structure; the base part only has one bottom drawer. This limitation can be improved by incorporating more data from recent datasets, such as [13, 34], which offers scanned real-world objects of various shapes, sizes, and part structures. However, it's important to note that generalization to unseen categories is beyond the scope of this paper. Moreover, in terms of part structure, it's also important to note that previous works can only handle very limited part structures, such as one base part and one articulated part [16] and predefined part count per category [38]. In contrast, our model can handle various part counts with a single model.

False positives and negatives Due to the detection-based nature of our model and also the domain gap to real-world data, false negatives or positives still occur, as shown in Fig. 19 (c) and (d). One practical improvement would be including real-world data [13, 34] in training to minimize the domain gap, hence boosting the model's generalization capabilities. Another approach for reducing false positives especially with physically implausible configurations, is to include regularization losses, such as physical visolation loss [61] for learning physical constraints. It's important to note that the kinematics-aware part fusion improves both false positives and negatives, as discussed in Sec. 4.3.

Objects with many joints The proposed approach works reasonably for complex articulated objects such as those with more than five joints, as shown in Fig. 20 (a) when all parts are visible from the given viewpoint. However, from certain viewpoints, we also observe that a single instance is reconstructed as two separate instances, each with fewer joints, as shown in the first row of Fig. 20

RGB

Currentpose

RGB

Current pose

Depth

Fully opened

Depth

Fully opened

RGB

Current pose

RGB

Currentpose

Depth

Closed

Depth

Closed

RGB

Current pose

RGB

Current pose

Depth

Closed

Depth

Closed

(b). We attribute this to the current dataset consisting of a small number of CAD models with >5 joints, while and the majority of instances have fewer joints. Also, when some parts are only partially visible, our method tends to make inaccurate pose estimations for such parts, such as the right stacked three drawers indicated by the red box in the second row of Fig. 20 (b).

Latency of KPF module While the KPF module improves detection performance, query oversampling, part proposal fusing, and kIoU calculation add extra running time for inference. We visualize the latency and accuracy trade-off in Fig. 21. We change the number of query oversampling from one to ten, and measure the per image processing speed in seconds. We randomly sample 100 samples from the test split for the evaluation and latency is measured on a V100 GPU. Although detection perfomance in shape mAP improves with more query oversampling, the latency degrades. Note that the result of w/o KPF applies 3D cuboid NMS with GPU parallelization, while the KPF module is implemented on CPU without any parallelization. We believe the latency of the KPF module can be significantly improved by optimizing the implementation, such as utilizing GPU parallization.

Input

GT

A-SDF-GT-2

Ours

Input

GT

A-SDF-GT-2

Ours

Input

GT

A-SDF-GT-2

Ours

Dehumidifier

RGB

Currentpose

RGB

Currentpose

Depth

Fully opened

Depth

Fully opened

(a) Ambiguous opening direction

(b) Unseen category

RGB

Currentpose

RGB

Currentpose

Depth

Fully opened

Depth

Fully opened

(c) False positive

(d) False negative

Input

GT

Ours

(a) Successful cases of objects with >5+ joints

Input

GT

Ours

(b) Failure cases of objects with >5+ joints

W/ KPF QOx10

68.5

W/KPFQO×8

%

0

68.0

w/oKPF

9

W/KPF Q0x6

W/ KPF QOx1

67.5

W/oKPF-

W/ KPF Q0×2

W/ KPF Q0×2

W/KPF QOx4

S

67.0

W/ KPF QO×6

W/KPFQO×8

66.5

W/KPF-QOx1

W/KPF QOx10

0

10

20

30

40

50

Latency (second)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel task focused on the reconstruction of multiple articulated objects, considering part-level shape, pose, joint parameters, and part-instance association, using only a single RGBD image. The authors propose an effective detect-and-group strategy that harnesses the part-level representations to detect, reconstruct, and predict parameters for articulated objects with diverse structures. Additionally, to enhance the detection performance, the paper introduces an oversampling and fusion strategy during inference. Moreover, the authors incorporate anisotropic size normalization and a refinement module to improve reconstruction quality and enhance part pose/motion prediction.\n\nSTRENGTHS:\n* The proposed task of reconstructing any number of articulated objects is both novel and valuable for exploring potential downstream robotics applications.\n* Utilizing part-level representations and employing the detect-then-group strategy are intuitive and effective when handling articulated objects with diverse structures.\n* The proposed end-to-end method builds upon 3DETR by incorporating part-level detection and pose/motion prediction. Furthermore, the authors introduce an instance-loss to aid in grouping within the part latent space. \n* The experimental settings are reasonable and allow for meaningful comparisons with previous work in the fields of articulated object reconstruction and motion prediction. The results demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n* Due to the task's focus on reconstructing part shapes, it’s crucial to show some novel viewpoint of the recons trued shapes. For example, the reconstructed shapes of some base parts are hard to see the improvement only with quantitative results. Additionally, all visualizations in the paper appear to align with the input RGBD viewpoint, limiting the comprehensive understanding of the reconstructed shapes.\n* If I understand correctly, for the statistics in table 2, the instance number includes the same articulated objects with various part states, or it’s hard to explain why there are so many instances in each category (much more than the object number in SAPIEN dataset)\n* For the evaluation, it seems that in the test set, the part state are randomly initialized. Then most doors or drawers are actually open. However, in the real case, most time, the movable parts are closed. It’s better to have separate evaluation for the models with different motion states\n* The paper lacks statistics on the number of articulated objects present in the input image, and it would be intriguing to include evaluation results that consider the number of objects to better understand the performance of part-instance association.\n* The quantitative improvements achieved with QO, PF, and kIoU metrics do not appear to be substantial. Are there any qualitative results available to provide additional insights or demonstrate the effectiveness of the proposed module?\n\nQUESTIONS:\n* Why the number of images in the validation set is much more than the number of images in the train set and test set? \n* For the number of objects in each image, what’s the statistics in the train set and test set? \n* For the model, why “ours-BG” sometimes outperform “ours”? “Ours” should have extra information of the foreground mask, right? Is this caused by the imperfect performance of the foreground mask?\n\nLIMITATIONS:\nThe authors mention the limitation in the supplement.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new method for 3D semantic instance reconstruction at the part level from a single RGB-D image. This method follows a top-down manner by first detecting object parts using 3DETR, where each instance part's bounding box will be predicted. The point cloud located inside each bounding box contains the part surface geometry, which will be normalized into a uniformly-scaled canonical system for the following part shape reconstruction. The key contribution of this paper is the kinematics-aware part fusion module, which follows a bottom-up manner to constitute complete instance shapes from parts.\n\nThe paper writing could be improved, e.g., the first paragraph in the introduction section is actually elaborating on related work. An illustration in Sec 3.6 would be more accessible for the audience to follow.\n\nThe experiment metrics and designs are extensive but lack some comparisons with the state-of-the-art.\n\nSTRENGTHS:\nIn my view, the major strengths can be concluded as follows:\n\n1. Kinematics-aware part fusion (KPF). The authors follow a top-down and bottom-up manner to first detect part-level geometries from the point cloud. Then they use KPF to construct instance shapes from parts. These two processes are trained end-to-end jointly to improve the entire performance.\n\n2. It shows good qualitative and quantitative performance in both synthetic data and synthetic data, with an acceptable generalization ability, even though they only train their model with synthetic data.\n\nWEAKNESSES:\nThe weaknesses of this paper are also obvious.\n\n1. This paper follows a similar top-down pipeline as previous works. (e.g., RfD-Net). In L34, the authors argued that previous methods \"are either limited to a single instance or require a separate instance detector, making them not end-to-end trainable\". I do not think this makes sense, as this paper also requires an instance detector. Besides, relying on a separate instance detector does not mean they can not be trained end-to-end.\n\n2. As this task is not new and this paper follows a similar pipeline, it means the contribution of this paper only comes from the module design. To me the major contribution lies on KPF module. But from the ablation study, it seems KPF does not contribute the most.\n\n3. Section 4.2, comparing with OPD by adjusting the IoU threshold is not rigorous. There is no theoretical guarantee that this comparison is fair. Since OPD is an RGB-based method while this paper uses RGB-D information, a fair comparison would be to add depth information into OPD or to remove the depth channel in your method.\n\nQUESTIONS:\nHere are some suggestions to improve this work, with which I would consider raising my score.\n\n1. A comprehensive analysis and comparison between the key module of this method (e.g., KPF) and the state-of-the-art (A-SDF is a bit old).\n2. The introduction should be about telling the story of the paper's motivation, key observations and contributions. Listing many related works there made it difficult to follow.\n\nLIMITATIONS:\nThis paper well discussed the limitations and failure cases in the supplemental.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a detection-based reconstruction method for articulated objects along with estimating part-level 6D object poses, sizes, and joint parameters. The paper uses 3DETR as the backbone predicts all of the above quantities while treating this problem as a supervised learning approach given labeled synthetic data. The approach trains only on synthetic data and transfers to real world dataset.\n\nSTRENGTHS:\nThe paper effectively uses a detection backbone i.e. 3DETR to reconstruct multiple articulated objects from a single view RGB-D observation. Sim-to-real transfer uses only synthetic data pretraining is a strong result. The use of set matching (from instance segmentation) literature for articulated part reconstruction is interesting. Joint estimation and shape reconstruction quantitative results are significantly better than baselines. I believe breaking the problem in this way i.e. part level reconstruction omits the need for model selection (i.e. training one model per category such as different models for glasses (2 joints) and refrigerators (1 joint) which is a major limitation in previous works.\n\nWEAKNESSES:\n1. The paper mentions CARTO, which is super relevant and recent CVPR'23 work but doesn't directly compare to it in terms of both detection and reconstruction. Is there a reason for it? Rather the paper takes individual joint parameter baseline i.e. OPD and shape reconstruction baseline i.e. A-SDF and compares them separately to those. While these are relevant, CARTO is the most relevant to this work in terms of system-level joint detection and shape reconstruction. \n\n2. While the work offers a solution for handling an arbitrary number of parts in the image, the qualitative results show simple examples i.e. same types of joints and 2 joints mostly. Did the authors test their approach on more complicated geometries i.e. varying number of joints or articulated objects with more joints >5+.\n\n3. I didn't see a discussion to run the speed of the model. The most relevant works which the author discusses i.e. CARTO (which builds upon CenterSnap and ShaPO) are all fast approaches. Is 3DETR equally fast and did the author consider adding that backbone i.e. CenterSnap backbone which offers a faster solution + equally good model in terms of accuracy? This is crucial for real-time applications like robotics or grasping etc.\n\nQUESTIONS:\nPlease see the questions raised above in the weakness section. To summarize, a comparison to strong and relevant system-level baselines, more qualitative results, and discussion/comparison to speed vs accuracy tradeoff would be useful to answer.\n\nLIMITATIONS:\nThe authors can only handle cases where instance boundaries are defined so it would fail in scenarios where a door would be attached to a room let's say.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for man-made articulated objects reconstruction from a single RGBD image across different object categories. The method is based on a detect-then-group pipeline, using kinematics-aware fusion for addressing false negatives.\n\nSTRENGTHS:\nThe proposed method addresses the challenging problem of man-made articulated object reconstruction from a single RGBD image, estimating at the same time each part's pose. \n\nThe overall architecture has a somewhat complicated structure with numerous encoder and decoder modules employed. Nevertheless, the design choices are sufficiently motivated in the text and the loss function is quite straightforward given the overall architecture. The experimental evaluation considers both a synthetic (SAPIEN) as well as a real-world (BMVC) dataset. The proposed method achieves improved performance with respect the state-of-the-art on SAPIEN. The ablative study shows the relative contribution of important components. \n\nRegarding reproducibility, sufficient details are provided making it easier to reproduce the results (considering also the supplemental material).\n\nWEAKNESSES:\nThe method seems to be closely related to [20]. I think a more detailed discussion about the relation between the two methods should be included in the Related work section. Related to this, it is not clear why the experimental evaluation does not consider comparison also with [20]. This is important, considering the few baselines available for this task. Alternatively, methods considering multi-view reconstruction on SAPIEN could also be considered to give additional insight on the performance of the proposed method. On a side note, the considered metrics could also include the joint state error.\n\n### Minor comments\n- L.45: define acronym NMS\n- L.127: not clear\n- Fig. 2 is not referenced in text\n- Although prior work regarding human subjects is briefly visited in the related work, similar work for animals is briefly mentioned but not specified. Works like [R1], [R2] and [R3] could be included to make this aspect of related work stronger.\n\n[R1] Ntouskos, V., Sanzari, M., et al. (2015). Component-wise modeling of articulated objects. ICCV\n\n[R2] Zuffi, S., Kanazawa, A., et al. (2017). 3D menagerie: Modeling the 3D shape and pose of animals. CVPR\n\n[R3] Jiang, L., Lee, C., Teotia, D., & Ostadabbas, S. (2022). Animal pose estimation: A closer look at the state-of-the-art, existing gaps and opportunities. Computer Vision and Image Understanding, 103483.\n\nQUESTIONS:\n- L.126: why P_O both conditions O and is given also as an argument?\n\nLIMITATIONS:\nLimitations are discussed in the text", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper proposes an end-to-end trainable method for reconstructing multiple articulated objects from a single RGB-D image, consisting of detecting parts, reconstructing part-level shapes, and estimating poses, bounding boxes as well as kinematic parameters. The parts are grouped into instances later. The authors propose anisotropic scale normalization for shape reconstruction to accommodate various part sizes and scales. Besides, the authors also propose test-time kinematic-aware part fusion (similar to non-maximum suppression) to reduce false positives when multiple detected results are generated and needed to be merged. Evaluation on both synthetic and real data demonstrates the effectiveness of the proposed method.

STRENGTHS:
- The paper is clearly written and easy to follow.
- The authors propose a clean pipeline for reconstructing articulated objects that can be trained end-to-end and inferred straightforwardly without composing multiple networks.

WEAKNESSES:
Minor typos:
- L38: "a end-to-end" -> "an end-to-end"
- L219-L220: "and use cosine scheduler to learning rate of 1e-6" sounds a little strange to me.

QUESTIONS:
1. How do the authors choose the revolute origin? It seems that any point on the revolute axis can be chosen as the origin. I think this question also applies to prismatic joints.
2. Is there any explanation why anisotropic scaling is better than isotropic one?
3. L235-L236: Do you actually mean 20,000 validation images and 168,726 training images? Otherwise, it is strange why more images are used for validation instead of training.

LIMITATIONS:
The authors has addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-09