Detection Based Part-level Articulated Object Reconstruction from Single RGBD Image
Authors: Yuki Kawana
Tatsuya Harada
Venue: NeurIPS 2023 poster (2023)
Keywords: articulated objects
shape reconstruction
3D reconstruction

Abstract: We propose an end-to-end trainable, cross-category method for reconstructing multiple man-made articulated objects from a single RGBD image, focusing on part-level shape reconstruction and pose and kinematics estimation. We depart from previous works that rely on learning instance-level latent space, focusing on man-made articulated objects with predefined part counts. Instead, we propose a novel alternative approach that employs part-level representation, representing instances as combinations of detected parts. While our detect-then-group approach effectively handles instances with diverse part structures and various part counts, it faces issues of false positives, varying part sizes and scales, and an increasing model size due to end-to-end training. To address these challenges, we propose 1) test-time kinematics-aware part fusion to improve detection performance while suppressing false positives, 2) anisotropic scale normalization for part shape learning to accommodate various part sizes and scales, and 3) a balancing strategy for cross-refinement between feature space and output space to improve part detection while maintaining model size. Evaluation on both synthetic and real data demonstrates that our method successfully reconstructs variously structured multiple instances that previous works cannot handle, and outperforms prior works in shape reconstruction and kinematics estimation.

Detection Based Part-level Articulated Object Reconstruction from Single RGBD Image
Yuki Kawana', Tatsuya Harada 1:2
The University of Tokyo, "RIKEN AIP (kawana, harada] @mi u-tooyyo ac.yyo..c...... jc.jp
Abstract
We propose an end-to-end trainable, cross-category method for reconstructing multiple man-made articulated objects from a single RGBD image, focusing on part-level shape reconstruction and pose and kinematics estimation. We depart from previous works that rely on learning instance-level latent space, focusing on man-made articulated objects with predefined part counts. Instead, we propose novel alternative approach that employs part-level representation, representing instances as combinations of detected parts. While our detect-then-group approach effectively handles instances with diverse part structures and various part counts, it faces issues of false positives, varying part sizes and scales, and an increasing propose
model size due to end-to-end training. To address these challenges, I) test-time kinematics-awar part fusion to improve detection performance while suppressing false positives, 2) anisotropic scale normalization for part shape learn- ing to accommodate various part sizes and scales, and 3) a balancing strategy for cross-refinement between feature space and output space to improve part detection while maintaining model size. Evaluation on both synthetic and real data demon- strates that our method successfully reconstructs variously structured multiple instances that previous works cannot handle, and outperforms prior works shape reconstruction and kinematics estimation.
Introduction
Estimating object shape, pose, size, and kinematics from single frame of partial observation is fundamental challenge in computer vision. Understanding such properties of daily articulated objects has various applications in robotics and AR/VR.
Shape reconstruction daily articulated objects is a challenging task. These objects exhibit a range of shapes resulting from different local part poses. More importantly, they display significant intra- and inter-category diversity in part configurations, including variations in part counts and structures. These factors together contribute to an exponentially increasing shape variation. Previous works have addressed this issue by either limiting a single model to target objects with a single articulated part [15] employing multiple category-level models (88 to accommodate varying part counts. These approaches first detect each instance, and then model the target shape in an instance- latent space, primarily employing A-SDF (8 for shape learning. A-SDF maps the target shape into an instance-wise latent space, and then shape decoder outputs the entire shape of the instance. However, this approach is limited when dealing with varying part counts and structures, the shape decoder must handle an exponentially increasing number of shape variations due to different part layout combinations 44 addition local part poses. Consequently, addressing this variety with a single model remains a complex and unsolved task.
In this paper, address this complexity through novel detect-then-grout approach. Our key observation that daily articulated objects consist of similar part shapes. For example, regardless
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Input
Output
Part shape
Part-to instance association Instance category
Optional
Part 6D Pose
Part size
Joint type
Joint direction
Foreground mask
Joint
Revolute origin
RGBD image
Detect-then theen--p embedding space
Our detection-basee approach estimates part-level shape, pose, and kinematics as joint parameters. It also recovers parts-to-instance associations handle multiple instances with various part structures and counts.
Figure 1:
of the number of refrigerator doors, each door typically has similar shape, and the base part may share similarities with those from other categories, such as storage units. By detecting each part and then grouping them into multiple instances, we provide a scalable and generalizable approach for handling diverse part configurations of daily articulated objects in scene.
Based on this concept, we propose an end-to-end detection based approach for part-level shape reconstruction. Building upon 3DETR [37 single RGBD image with an optional foreground mask, our model outputs part shape, pose, joint parameters, parts-to- tiisstto-isttnce association, and instance category. Our approach employs novel detect-then-group approach. It first detects parts and applies simple clustering of parts into instances based learned part embeddings' distance, in contrast to the previous works using additional instance detection module [28] [5]. An overview of our approach is shown in Fig T However, we found that detection-bare -based shape reconstruction prone false positives for articulated objects' thin and small parts with little overlap. which is hard to remove by NMS. Also, articulated objects often have parts of varied sizes and scales, making training with a single-shape decoder challenging. Additionally, increasing model size by end-to-end training from detection to reconstruction makes simply enlarging the model size improve performance undesirable. To address these challenges, we propose:(1 (1) kinematics-awart
end-to-end trainable detector backbone, given
part fusion to reduce false positives and improve detection accuracy; (2) anisotropic scale normalization for various part sizes and scales shape learning; (3) and an output space refiner module coupled with model-size balancing strategy with decoder for improved performance while keeping model size. evaluate our method on the photorealistically rendered SAPIEN (88) dataset, and our approach outperforms state-of-the- baselines in shape reconstruction and joint parameter estimation. Furthermore, the model trained only on synthetic data generalizes to real-world data, outperforming the state-of-the the-art methods on the BMVC (36] dataset.
Our contributions can be summarized as follows: (1) novel part-level end-to-end shape recon- struction method for articulated objects from single RGBD image; (2) novel detect-then- group approach that simplifies the pipeline: (3) addressing detection-baas -aseed reconstruction challenges with kinematics-ane aware part fusion, anisotropic scale normalization, and refiner module coupled with model-size balancing: (4) superior performance on the SAPIEN 58 dataset, with the ability to generalize to real-world data from the BMVC [36] dataset.
2 Related Work
Articulated shape reconstruction A number of works have focused on human subjects using and utilize category-specifie templates to recover deformation from Recent research has delved into recovering articulated shapes with unknown kinematic chains from sequences of target motion [1 60). These works make category-level assumptions about kinematic structures, with targets in observations sharing common kinematic structures. Their main focus on natural objects such humans and animals. In contrast, our interest is in reconstructing the shape of multi-category, daily man-made articulated objects with diverse kinematic structures using single model. Recent years have seen the emergence of methods specifically targeting man man-made articulated objects [56] 23 53), and taking single-frame frame input [38 28 15 24). However, these models are constrained by either predefined number of parts per category the necessity of multiple models for each combination of categories and part counts. Consequently, they are unable to scale to wide array of real-world articulated objects with varying part counts using a single model. Our approach addresses this limitation.
single image input 46
45
canonical shape 29
3
63
25

---

Point cloud
Colored point cloud {P,F)
Part shape
Camera Intrinai
Encoder E
Shape
20 3D projection
Scene
Downsampled
decoder 0
feature F*
polint cloud P
Shape
FPS
Ng
feature
Pose +
Part pose and shapes
param. estimation
Query [q
Part (hr)
size (Bd)
Optional
Decoder 2
proposals
[x,?]
parameters {M,
Kinematic
KPF
RGBD image
Foreground mask W
Refiner 32
Kinematlcs-
Instance
parameters Vi)
fusion
assaciation
Figure 2: Overview of the pipeline. The input RGBD image is projected to a colored point cloud. The encoder & extracts scene features and downsampled point cloud. The decoder 'D outputs of part proposals (X,, from part queryies {q²}. }. The refiner R. estimates the residual of part pose and size AB and joint parameter AA for refined { B, A). At test time, the inference is run No times independently to densely sample part proposals as [x J. KPF removes false positives in {X,r} by using kinematics-aware IoU (kloU) refine the prediction further. The part shape is reconstructed by the implicit shape decoder O.
Gerput (% AK
Roffaed
deument
Desooier
Conter (Ac, qui
fulliner 2t
Rounddar My
Poctioral escoding
dauxx
this
MMLP
Cuary (Cung
tW's
(a) positives (b) Convex hull (c) loU 21%, kloU 77%
(4a)
Soone
Figure 4: Illustration of (a) false positives, (b) Convex hull for kloU, and (c) comparison of 3D box IoU and kloU for overlapping parts.
festure
Figure 3: Architecture of refiner R
Pose and kinematic estimation of articulated objects predominantly, existing research on pose and kinematic estimation of articulated objects has focused estimation from sequences [16] 21 necessitating multiple frames of moving targets or interaction with the environment before estimation. Estimation from single image has also been explored [27 36 28), but these methods are limited to predefined part structures. A few recent studies have proposed approaches without However, these methods target single instances, requiring instance detection before part-level estimation. Moreover, their focus limited to detection, pose and kinematic estimation, whereas our work aims for shape reconstruction in end-to-end manner.
57 49
assumptions on part structure 22 13
Detection-based reconstruction A large body of work exists combining detection and reconstruc- tion for multiple rigid objects in diverse settings, such as indoor scenes [5T] 39 40 tabletop environments
61
[4] [4],
[19 20), and road scenes 33). Recent works target daily articulated objects [2]] [6]. However, these methods predominantly rely on an instance-level detection approach. In contrast, our work pivots towards part-level detection to effectively handle wide variety of part
structures of real-world articulated objects.
3 Methods
3.1
Problem setting
Our method takes a point cloud P with Np 3D points and color feature F lifted from a single RGBD image / of articulated objects and camera intrinsic K input. outputs set of parts G
[N]}
and non-overlapping subsets of all parts {Gi) Ng as NG instances, where G = U.(G) 1=1- Note that we use the shorthand {*:} to denote an ordered set {*+), for brevity. For each part we estimate 6D pose and size B SE(3) x R3, part shape O as an implicit representation and kinematic parameters M. Our shape representation
an implicit function defined as R + [0. 1] with isosurface threshold TO. where (x R O(x) тO) indicates inside the shape. The kinematic parameter M consists of joint type y E (0, 1,2) which represents fixed, revolute, and prismatic types, joint direction E S?, ID joint state deuneut and dmax for the current pose and fully opened pose from
3

---

the canonical pose. We also predict the revolute origin v E R for the revolute part. We define the joint parameter as AA fa. deurents dmax v]}. thus M = {y, A}. For each instance I, we estimate the instance parameter J which consists of category u and part association defined as 6;; = 1 E Gi), where 1 is an indicator function.
3.2
Detection backbone
Our detection backbone consists of a transformer based encoder & and decoder 'D based on 3DETR
The encoder comprises recursive self-attention layers encoding 3D points P and color feature F into downsampled 3D point cloud P of Np points and Dpr-dimensional scene feature F'. Query [q>Jne E I are randomly sampled using furthest point sampling (FPS) from P'. The decoder is composed of transformer decoder layers { considering cross-attention between queries and F' and seff-attention among queries. The decoder iteratively refines query features {on, ox (fe((n, F*). Lastly, of part prediction MI.Ps decodes each refined query feature to produce output values. For clarity, the query index n. is
37
locations
omitted when possible.
3.3 Part representation
Part pose and size We predict part pose and size B as set of part center E R3, rotation R E SO(3) and size s E R$ for each query. We predict as an offset Ac from q, added to the query coordinates, i.e., q Ac.
Part shape We employ Shared-weight single implicit shape decoder O for performing part-wise shape reconstruction by taking point clouds around the detected regions where parts are identified. Given the diversity in shape and pose, and anisotropic scaling of the parts we focus on, is challenging to learn shape bias with single shape decoder. Therefore, we propose anisotropically normalizing the side lengths of the shape decoder's input and output to unit scale to perform reconstruction. We define the input point cloud Po as follows:
Po = {RS)) (p- c) p P. max( (RS)- c)) S 0.5).
(1)
where S denotes diagonal matrix of scale We define the output occupancy value at E R3 as Ox = ((RS) (x c) Po. h). where E Ph is a part shape feature modeled by part prediction MLP. Given that the imput point cloud Po includes background, the geometry of the target part shape can be ambiguous. To address this issue, we train the detector backbone and shape decoder end-to-end, by inputting h as shape geometry to the shape decoder so that h informs the shape decoder of the foreground target shape. We utilize shape decoder architecture with a lightweight local geometry encoder [42] to spatially associate input points Po with output occupancy values ox. which are both defined in normalized space.
Part kinematics We predict a 4-dimensional vector y as a probability distribution over part joint types. This includes 'background' 'not part' type for instances where predicted part proposals might not contain part. The revolute origin v + Av is predicted similarly to the part center with an offset Av. Joint states deurrent and dmax are modeled by separate part prediction MLPs for revolute and prismatic types, and we only supervise the output corresponding to the ground truth joint type. single MLP used for joint direction for both revolute and prismatic types.
3.4 Instance representation
We form a collection of parts as a single instance G, = E where the distance between the Dy-dimensional embeddings, 2; and Zy. of each part pair within the group is kept below a specified threshold т.. We predict an Nj,-dimensional vector u as a probability distribution
112--,1|<< }
instance categories per part. At test time, predict the instance category by laking the category prediction with the highest confidence from the category predictions of the parts belonging the same instance.
3.5 Refiner R
Query features are iteratively refined in the decoder in the feature space (37 Increasing the number of decoder layers enhance performance at the expense of larger model size [3]. However, it is

---

crucial to avoid increasing the model size to enable efficient end-to-end training from detection to shape reconstruction. To improve model performance while maintaining the model size, we found cross-refining both output space and feature space to be effective. We introduce refiner R. which has an identical architecture with the decoder DD The refiner R. serves to refine the prediction in output space by reallocating a portion of decoder layers from decoder 'D to R. Assuming there are ND+R decoder layers in the original model, we reallocate NR decoder layers to the refiner R. Consequently, ND- ND+R NR layers are used for query feature refinement in decoder D. The refiner R refines part pose and size B' and joint parameter A' from D by predicting residuals AB and A A to produce refined prediction B and A, respectively. The architecture of the refiner R is shown in Fig.
3.6 Kinematics-aware part fusion (KPF)
Articulated objects often consist of small and thin parts. Especially for unsegmented inputs, only a small portion of the point cloud represents such parts after subsampling. To ensure a sufficient number of queries cover such parts and their surrounding context, at test time, we randomize the sampling positions of the queries and independently carry out inference and NMS for the input No times, obtaining collection of densely sampled part proposals from No inference runs. This process termed Query Oversampling (0Q). However, OQ can increase false positives and degrade detection performance. To mitigate this, we propose Part Fusion (PF), inspired by Weighted Box Fusion (WBF) [88]. to merge overlapping part proposals, thereby reducing false positives and improving detection accuracy. Unlike WBF, which fuses only 2D bounding parameters. PF fuses all the predicted parameters of part proposals, with an average weight defined as objectness - Yog. However, using IoU with 3D bounding boxes as overlapping metrics yields overly small values for thin structured parts, even when they are proximate. This results in PF failing to merge redundant parts, leading to false positives, as illustrated in Fig. 4 (a). To overcome this challenge, we propose kinematics aware IoU (kloU) based on the observation that a redundant part pair exhibits significantly overlapping trajectories. To calculate kloU, we construct convex hull for each part in the pair using the 24 vertices of three bounding boxes based on size and canonical pose, current pose, and fully opened pose using the predicted part pose and size B and joint parameter A, as show in Fig. (b). Then, we calculate the IoU between the two hulls. The comparison between the 3D bounding box IoU and kloU is depicted in Fig. (c). Our overall method, termed Kinematic-
Part Fusion (KPF),
includes: (1) executing inference and NMS using kloU to gather part proposals multiple times, (2) conducting PF using kloU to update these proposals iteratively, and (3) removing proposals with low objectness. Note that KPF is non- con-difeerentaalle and is disabled during training, with NQ = 1. For further details, please refer to the appendix.
3.7
Set matching and training loss
Set matching We base our end-to-end training of detection to reconstruction by utilizing 1-to-1 matching between part proposals and ground truth, using bipartite matching [5] for loss calculation. Similarly to (37), we define a matching cost between a predicted part and a ground truth part as Cmatch A |B BGT || | c II + >a yog), where B defines eight vertices of cuboid defined by part pose and size B. y, defines the joint type probability given the ground truth label y, and 1 Yhe defines the foreground probability. Deviating from (37, we use the L1 distance
of eight vertices of a cuboid instead of the GloU 44 of cuboids in the first term to avoid the costly calculation of enclosing hull for 3D rotated cuboids.
Training losses
For each pair of prediction and ground truth, we define part loss as
Epan
1 N
Σ
B. B |||||
R R 1 +Ex (CB((xx,s xx) ||dmax, GCT Il
(2)
GT +IIdeurrent, current. - a: + PL(V., o GT + CE CT) CE(U,, u
All loss terms have equal weights. The first term is a disentangled L1 loss described in 47 to optimize B. This loss replicated three times by using only one of the predicted three components (R,c,s) for B, while replacing the other three with their ground truth values. We also found that directly optimizing rotation, as in the second term of the loss, leads to smaller rotation loss during training. BCE and CE denote binary cross-entropy and cross-cntropy loss, respectively. PL denotes
5

---

F-Score 80% f
F-Score900% f
CD@5%
CD@
IoU @25% f
IoU 50%%
A-SDF-GT
65.49
47.69
74.60
39.76
36.62
10.81
A-SDE-G1-2
68.81
52.55
75.91
43.58
38,59
10.43
Ours- BG
74.22
68.80
75.71
58.61
40.06
Ours
74.77
68.38
77.39
56.53
41.35
11.63
Table Shape mAP results on SAPIEN
dataset.
the point-line distance between the revolute origin v; and the ground truth joint axis defined by v
and aTT For unrefined prediction B' and A', we define the same loss as 4annt except for ox. Y; and denoted as par During training, we use ground truth B for Eq adversely affecting shape learning.
i to avoid noisy prediction
We also define instance loss Linstance for learning part-to-instance association with modified improved triplet loss 8 defined as:
Linuance
Xintra
Σ iEG
Σ [ns - -el+ +
1 N
E LEG
max "h min My + +3//++ (3) JEG
Gtyjl
j'EGKG
JEGIIA
where Gijs E G1 6Gs = 13 and 7kj = | 2 denotes the L.2 distance between the part-to-instance association embeddings of the i-th and j-th parts. The first term enforces the distance between the two embeddings of two different parts belonging to the same instance below =y. and the second term ensures the distance between embeddings of parts belonging to different instances is larger than 3 However, computing all combinations of triplets for the second term is operationally complex for part-wise supervision. To streamline this, we instead opt to maximize the difference between the upper bound and the lower bound of inter- and intra-instance distances of the embeddings. In practice, we replace max and min with their soft approximations defined as LogSumExpl (1s)) and LoggSuEE -7)y') for smooth gradient propagation. During inference, we use threshold Tz (3r + T to determine if two parts belong the same instance based on the distance between their embeddings.
The total loss minimize is Liail Epan + Span Linstance- At training time, prediction MLPs to predict part parameters at every layer in the decoder. We compute the Liotal for each layer independently and sum all the losses. We only use the part parameter predicted from the last decoder layer at test time. Full loss formulation can be found in the appendix.
the same part
3.8 Implementation detail
The implementation of the transformer encoder-decoder architecture and the optimizer configuration are based on the publicly available code of 3DETR [37]. We employ a masked encoder architecture for & from 37]. The input point cloud to the encoder consists of Np 32768 points, which are subsampled to Np = 2048 points. The number of queries set to N₂ = 128 during training. At test time, set it to Ng 512, and with independent runs for query oversampling
NQ 10. The number of decoder layers for 'D and R are set to ND = 6 and NR = 2, respectively. For the evaluation of synthetic data, the foreground masks inferred by pretrained segmentation model using a ResNext50) (59) encoder and a Deepl V3Plus [6] segmentation head with RGBD input. To evaluate real-world data, automatically generate foreground masks using https://remove.bg following (56). The weights for the matching cost Cmatch were set to λ = 8, A2 = 10, \3 = 1, 5. We set Xintra 0.1 for the intra-instance loss term of the instance loss Linstance. We use the AdamW 31 optimizer with a base learning rate of 9e-4 and employ a cosine scheduler with nine epochs for warm-up. The training was performed using two A100 GPUs, each 40GB of GPU memory, and with batch size of 26 for each GPU. The training took approximately two days. Please refer to the appendix for further details regarding the training process, hyperparameters, and network architecture.
4 Experiments
Datasets We evaluate our method on both synthetic and real-world data. We use the SAPIEN 58 dataset for synthetic data evaluation, following recent works articulated shape reconstruction 24 56 15). We select ten categories with representative articulation types, sufficient number of CAD models, and various part structures across categories. We then randomly construct room

---

Input
A-SDF-GT-2
Ours
Figure 5: Qualitative results SAPIEN
58
dataset.
Joint type
Part count
Instance count
Category
Rev.
Pris.
Most freq.
Max
Train
Test
Dishwasher
2
4997
1032
Trashcan
2
5037
1017
Safe
4795
1029
Oven
3 4951
1006
Storage
14
4907
973
Table
9
4790
999
Microwave
2
4832
Frige
2
3
5089
924
Washing
5042
995
Box
4
4823
979
Table 2: Overview of synthetic data from SAPIEN [58] dataset. Rev. and Pres. denote revolute and prismatic joints, respec- tively.
Input
Ours
Figure 6: world data.
Visualization on real-
geometry (wall and floor) and place to four instances per scene. Each instance is generated by applying random horizontal flips, random anisotropic resizing of side lengths, and random articulation of CAD models. The camera pose is sampled randomly, covering the upper hemisphere of the scene. Instances with severe truncation from the view frustum or occlusion with other instances are ignored during training and evaluation, ensuring that least one instance visible in view. For the training split, we randomize the textures of parts and room meshes. We use the original textures from the SAPIEN dataset for the test split. We generated 188.726 images for training and validation. Due to computational and time constraints, we used 20,000 images for training and kept the rest for validation usage. Also, we generated 4,000 images for the test split. Image size 360 640 in height and width. The data overview is shown in Table 2 More details on data preparation can be found in the appendix.
For real-world data, we use the BMVC 36 dataset for quantitative evaluation. We use cabinet class which includes both prismatic and revolute parts and has the part count and similar object shape as those used in our training and baseline models. The data contains two sequences of RGBD frames, capturing the same target objects with different part poses from various camera poses. We also test our method on RGB images taken with an iPhone X and depth maps generated from partial front views of the
using Nerfacto (50]
Metrics For shape evaluation, following detection-baan object reconstruction studies 40 16] we report mAP combined with standard shape evaluation metrics denoted as metric @ threshold. use F-Score [52), Chamfer distance (CD), and volumetric loU (IoU) with multiple matching thresholds. More details on shape metrics can be found in the appendix. The shape mAp evaluation includes all the predicted parameters except for part kinematic parameters, serving as a comprehensive proxy for their quality. For part-wise detection with part pose and size, use the average L2 distance

---

Input
GT
A-SDF-GT-2
Input
A-SDF-GT-2
Ours
Figure 7: Qualitative results on the BMVC 36 dataset.
between the corresponding eight corners of bounding boxes between the ground truth and prediction as a matching metric mAP, similar to the ADD metric [[7] denoted as mAP @trreshold. The distance is normalized by the ground truth part's diameter, with proportion of the diameter used as the threshold. Since loU-based metrics can yield values close to zero for thin structured parts, when they reasonably proximate and slightly off the ground truth, and do not consider part orientation, we use the above matching metrics to better analyze part-wise detection. For part kinematics evaluation, evaluate the absolute error joint state (State), Orientation Error (OE) for joint direction, and Minimum Distance (MD) for rotation origin following 22 for the detected parts with matched ground truth.
Baselines By default, we denote the proposed method that takes the foreground mask as 'Ours', and the model taking unsegmented input as 'Ours-BG'. To the best of our knowledge, no prior work operates on exactly the same problem setting ours. For shape reconstruction, we benchmark against the state-of-the-art category- object reconstruction method A-SDF [38], the closest to approach. We follow the most recent work setup (13) in evaluation. A-SDF requires instance segmentation, pose, and scale to project the object from the camera to the object-centric frame. We implement using all the required data with ground truth, denoted as A-SDF-GT following (15) Given A-SDF's assumption of fixed part count per model, as per a similar setup of [22). we train it the dataset's 's most frequent part count per category. To account for multiple part counts in category, for categories with more than one part count, we train the second model for the next most frequent count, resulting in a maximum of two models per category, termed A-SDF-GT- When the input instance has an untrained part count at test time, the model trained on the most frequent part count is used for A-SDF-GT-2. In the mAP evaluation, we give GT correspondence between the prediction and ground truth; thus, a false positive happens only when the shape metric is lower than the threshold for A-SDE For kinematic evaluation, we compare our method with the state-of-the-ae single- view part kinematic estimation method OPD [22]. Note that we input OPD with an RGBD image align input modality and modify it to output joint state. More details on baselines can be found in the appendix.
4.1 Shape reconstruction
We show the shape mAP result in Table | Our method outperforms A-SDF 38 with ground truth in all metrics, and Ours-BG outperforms in the majority of metrics. We show the qualitative results in Fig. 5 Our models effectively reconstruct multiple instances with diverse part counts and structures, outperforming A-SDF which struggles with reconstructing articulated parts. We attribute our method's superior performance to its part-level representation, which facilitates easier learning of various part structures. Moreover, our shape-decodes is less affected by shape variations arising from the combination of part structures and poses. The qualitative results from novel viewpoints and additional visualizations can found in the appendix.
4.2 Kinematic estimation
OPD performs 2D detection evaluated by 2D IoU, while ours on 3D by L2 distance between 3D bounding box corners. To make kinematics estimation results of OPD and ours comparable, we experimentally choose the matching threshold of our mAP as 70% so that similar number of
detected parts with OPD's result with 2D segmentation mAP@50% Then, we select the intersection of true positive detected parts by OPD and ours. The result is shown in Table 3 Our methods
outperform OPD significantly. We attribute the reason to our method operating on 3D point clouds

---

Input
w/o KPF
Input
w/o KPF
W/ KPF
GT
w/ KPF
Figure 8: Qualitative results on kinematics-aware part fusion (KPF).
Ambiguous opening
QO+PF w/o kloU
QO+PF w/ kloU (KPF)
(b) Physically implzusible false positive
Figure Qualitative results on kinematics-awwe IoU (kIoU).
Figure 10: Failure cases.
while OPD on 2D. Thus, our method is more robust to various textures and lighting conditions and makes it easier to reason about 3D kinematics. Note that our focus here is kinematics estimation after detection step. Thus, superiority in detection performance not our focus.
4.3 Ablation studies
In the following ablation studies, we validate each proposed component challenging setting where the input the encoder has an unsegmented background using Ours-BG.
Kinematics-aware part fusion We show quantitative results in Table Besides mAP, we also show precision considering false positives. QO denotes test-time query oversampling, and PF indicates part fusion. As a baseline, we first disable all components (w/o QO. PF, kloU) using the same number of queries N₂ 128 during training and add each component by one. Introducing proposed kloU on top of QO and PF outperforms the baseline shape reconstruction mAP and part detection while preserving similar precision. We visualize the effectiveness of the KPF module in Fig. S In the provided comparison, 'w/o KPF' denotes disabling QO, PF and kloU. We see that KPF enhances the detection and pose estimation of small parts. We also show the qualitative results on the proposed kloU in Fig. 9 Applying QO and PF alone leads to false positives of thin parts, which kloU effectively reduces. It indicates that the proposed KPF module improves overall detection performance while suppressing false positives.
Anisotropic scaling and end-to-end training for shape learning We validate the effect of anisotropic scale normalization and end-to-end training in shape mAP evaluation. In Table [5] w/o AS denotes using isotropic scaling by normalizing the maximum side length one instead of all sides, w/o SF denotes not passing shape feature h but training shape decoder O separately. Disabling each component degrades performance. Especially disabling anisotropic scaling significantly drops the performance, as the single shape decoder is tasked to decode various sizes of target shapes.
Ratio of decoder layers in the refiner R. We investigate the relationship between the proportion of decoder layers in R and performance in shape mAP. The result is shown Fig. [1 We vary the ratio of decoder layers in the refiner NR/ ND+R from 0%% to 75%. Allocating portion (25%) of decoder layers to the refiner improves performance with the same number of decoder layers while reducing excessive decoder layers from the decoder degrades performance.
4.4 Real-world data
We verify the generalizability of our approach, which is trained only on synthetic data to real-world data. Here, we include foreground masks inputs to mitigate the domain gap from background.

---

State 1
OE
MD 4
mAP
Precision
"SScoreu191Pe
OPD [22
19.23*/17.106
29.68
38.11cm
Ours-BG Ours
4.30%/5.27cm
3.97
6.43cm
wio PF. kloU
36.48
32.65
65.
kloU
40.78
49.64
66.24
3.85m/4
3.66
6.58cm
All (Oars-BG)
41.09
51.64
Table 3: Joint parameter estimation results.
Table 4: Ablation KPF module.
F Fscore 80% 80%
bscore9905
CDT@5%
C 1%
@5%%
loU@50%
w/o AS
59.37
38.83
69.84
24. 10
25.72
5.76
w/o SF
71.04
60.49
73.83
43. 14
25.17
0.64
All (Ours-BG)
74.22
68.80
75.71
58.61
40.06
9.80
Table 5: Ablation on shape learning.
We present the quantitative results on the BMVC 36 dataset in Table [6] As only one instance is present in the scene, we evaluate with shape metrics without mAP. The CD value is multiplied by 100. Our method outperforms A-SDF [38] We observe reasonable generalization as shown in Fig. 7 Furthermore, also show the qualitative result in Fig. for scenes containing multiple instances. We observe reasonable generalization to real-world data.
Limitations
While our method makes significant strides the daily articulated object reconstruction task, does have several limitations. (1) Our method cannot handle cases where the instance boundary is not well defined, such door directly attached to a room. 3D reconstruction of room geometry with part-level shape reconstruction of articulated parts is an interesting future direction for scene reconstruction task. (2) The proposed approach currently does not consider the physical constraints between parts during training. Although the KPF module removes the redundant parts post- processing, physically implausible false positives still occur as shown in Fig. T0 (a) for parts without overlapping trajectories. Introducing regularization, such implausible configurations would alleviate this problem. (3) Our method does not explicitly consider the ambiguity in the opening direction of part. As shown in Fig. (b),,,, the model fails to estimate the correct axis direction w.r.t. the base part for the closed part of the oven. This is because the dataset includes parts with different opening directions but similar shapes when closed. Such ambiguity can be addressed by explicit uncertainty modeling as I1, and integrating finer 21) visual cues for 3D reasoning [54] 26 7 for localizing knobs and handles, which are informative for opening direction. Further discussions on limitations can be found in the appendix.
physical violation loss [61], for such
6 Conclusion
We presented an end-to-end trainable part-level shape reconstruction method for multiple articulated objects from single RGBD image. We have demonstrated that our method successfully tackles the major limitation of previous works, which are unable to handle objects with various part counts using single model, by employing novel detect-then-group approach. We have also shown that the proposed kinematic part fusion (KPF) module effectively handles small parts as challenging targets while suppressing false positives for detection-based reconsiruction. Our method outperformed the of-e-of--hhear baselines in shape reconstruction and kinematics estimation on the SAPIEN (58] dataset. Furthermore, the BMVC (36) dataset and casually captured images, demonstrated that the model trained on synthetic data reasonably generalizes to real-world data.
F.Score@ 90%
68
F-Score
CD J
IoU
67
A-SDF-GT-2 38
83.25
1.73
12.26
66
Ours
97.51
0.56
27.96
65
25
50
Table 6: Shape reconstruction results on the
Ratio of decoder layers Nc/Nu
BMVC [36] dataset.
Figure 11: Ablation on refiner R.
10

---

Acknowledgments
We appreciate the members of the Machine Intelligence Laboratory for constructive discussion and their insightful feedback during the research meetings. This work was partially supported by JST Moonshot R&D Grant Number JPMJPS2011, CREST Grant Number JPMJCR2015 and Basic Research Grant (Super AI) of Institute for Al and Beyond of the University Tokyo.
References
[1] Abbatematteo, B., Tellex, S., Konidaris, G.: Learning to generalize kinematic models to novel objects. In: Proceedings of the Conference on Robot Learning (CoRI.). pp. 1289-1299 (2020)
[2] Beker, D., Kato, H., Morariu, M.A., Ando, T., Matsuoka, T., Kehl, W., Gaidon, A.: Monocular differentiable rendering for self-supervised 3d object detection. In: European Conference on Computer Vision (ECCV). pp. 54-5529 (2020)
[3] Bogo, F., Kanazawa, A., Lassner, C.. Gehler, P., Romero. J., Black, M.J.; Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In: Proceedings the European Conference Computer Vision (ECCV). pp. 561-578 (2016)
[4] Brégier, R.: Deep regression on manifolds: a 3D rotation case study. In: International Confer- ence on 3D Vision (3DV) (2021)
[5] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European Conference on Computer Vision (ECCV). pp. 213-229 (2020)
[6] Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F,, Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European Conference Computer Vision (ECCV). pp. 801-818 (2018)
[7] Chen, Z., Li, Z.., Zhang, S., Fang, L.,, Jiang, Q., Zhao, F.: Deformable feature aggregation for dynamic multi-modal 3d object detection. In: European Conference on Computer Vision (ECCV). pp. 628-644 (2022)
[8] Cheng, D., Gong, Y., Zhou, S., Wang, J., Zheng, N.: Person re-identification by multi-channel parts-based cnn with improved triplet loss function. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) (2016)
[9] Conant, J., Michaels, T.: On the number of tilings square by rectangles. European Journal of Combinatorics 18, 21-34 (2014)
[10] Denninger, M., Winkelbauer, D., Sundermeyer, M., Boerdijk, W., Knauer, M., Strobl, K.H., Humt, M., Triebel, R.: Blenderproc2: procedural pipeline for photorealistic rendering. Journal of Open Source Software 8(82), 4901 (2023)
[11] Desingh, K., Lu, S., Opipari, A., Jenkins, O.C.: Factored pose estimation of articulated objects using efficient nonparametric belief propagation. In: 2019 International Conference on Roboties and Automation (ICRA). pp. 7221-7227 (2019). https://doi.co org///......0....o
[12] Garland, M., Heckbert, P.S.: Surface simplification using quadric error metrics. In: Proceedings of the 24th annual conference Computer graphics and interactive techniques, pp. 209-216 (1997)
[13] Geng, H., Xu, H., Zhao, C., Xu, C., Yi, L., Huang. S., Wang, H.: Gapartnet: Cross-category Nomaii--gnneralizable object perception and manipulation via generalizable and actionable parts. In: Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR) (2023)
[14] Gkioxari, G., Malik, J., Johnson, J.: Mesh r-cnn. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019)
[15] Heppert, N., Irshad, M.Z., Zakharov, S., Liu, K., Ambrus, R.A.. Bohg, J., Valada, A., Kollar, T.: Carto: Category and joint agnostic reconstruction of articulated objects. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) (2023)
[16] Heppert, N., Migimatsu, T., Yi, B., Chen, C., Bohg. J.: Category-independeee articulated object tracking with factor graphs. In: TEEE/RS] International Conference on Intelligent Robots and Systems (IROS). pp. 3800-3807 (2022)

---

[17] Hinterstoisser, S., Lepetit, V., Ilie, S., Holzer, S., Bradski, G., Konolige, K., Navab, N.: Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In: Proceedings of the Asian Conference on Computer Vision (ACCV) (2013)
[18] lakubovskii, P.:
Segmentation models pytorch. https: http:/ggtttub... http / qubvel/
segmentation, models.dooss.ooooo wwtpmmdspptttrcciieeccpp (2019)
[19] Irshad, M.Z., Kollar, T., Laskey, M., Stone, K., Kira, Z.: Centersnap: Single-shot multi-object
3d shape reconstruction and categorical 6d pose and size estimation. In: IEEE International Conference on Robotics and Automation (ICRA) (2022)
[20] Irshad, M.Z., Zakharov. S., Ambrus, R., Kollar, T., Kira, Z., Gaidon, A.: Shapo: Implicit repre- sentations for multi object shape appearance and pose optimization. European Conference on Computer Vision (ECCV) (2022)
[21] Jain, A., Lioutikov, R., Chuck, C., Niekum, S.: Screwnet: Category-inddpeeddmt articulation model estimation from depth images using screw theory. In: IEEE International Conference on
Robotics and Automation (ICRA). pp. 13670-13677 (2021)
[22] Jiang. H., Mao, Y., Savva, M., Chang, A.X.: OPD: Single-view 3D openable part detection. In: Proceedings of the European Conference Computer Vision (ECCV) (2022)
[23] Jiang, Z., Hsu, C.C., Zhu, Y: Ditto: Building digital twins of articulated objects from interaction.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
[24] Kawana, Y., Mukuta, Y., Harada, T.: Unsupervised pose-aware part decomposition for man- made articulated objects. In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)
[25] Kulkarni, N., Gupta, A., Fouhey, D.F., Tulsiani, S.: Articulation-aware canonical surface mapping. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pp. 452-461 (2020)
[26] Li, H., Zhang, Z., Zhao, X., Wang, Y., Shen, Y., Pu, S., Mao, H.: Enhancing multi-modal features using local self-attention for 3d object detection. In: European Conference on Computer Vision (ECCV). pp. 532-549 (2022)
[27] Li, X., Wang, H., Yi, L., Guibas, L.J., Abbott, A.L., Song, S.: Category-level articulated object pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3706-3715 (2020)
[28] Liu, L., Xu, W., Fu, H., Qian, S., Yu, Q., Han, Y., Lu, C.: Akb-48: real-world articulated object knowledge base. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 14809-14818 (2022)
[29] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: A skinned multi- person linear model. ACM transactions on graphics (TOG) 34(6), 1-16 (2015)
[30] Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), p. 163-169 (1987)
[31] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: Proceedings of the
International Conference on Learning Representations (ICLR) (2018)
[32] Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Ojea, J.A., Goldberg, K.: Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics (2017)
[33] Manhardt, F., Kehl, W., Gaidon, A.: Roi-10d: Monocular lifting of 2d detection to 6d pose and
metric shape. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2069-2078 (2019)
[34] Mao, Y., Zhang, Y., Jiang, H., Chang, A.X., Savva, M.: Multiscan: Scalable rgbd scanning for 3d environments with articulated objects. In: Advances in Neural Information Processing Systems (NeurIPS) (2022)
[35] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4460-4470 (2019)
12

---

[36] Michel, F., Krull, A., Brachmann, E., Ying Yang. M., Gumbold, S., Rother, C.: Pose estimation of kinematic chain instances via object coordinate regression. In: Proceedings of the British Machine Vision Conference (BMVC). pp. 181.1-181.11 (2015)
[37] Misra, I., Girdhar, R., Joulin, A.: An end-to-end transformer model for 3d object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2021)
[38] Mu, J., Qiu, W., Kortylewski, A., Yuille, A., Vasconcelos, N., Wang, X.: A-sdf: Learning disentangled signed distance functions for articulated shape representation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13001-13011 (2021)
[39] Nie, Y., Han, X., Guo, S., Zheng, Y., Chang, J., Zhang, J.J.: Total3dundderstanding Joint layout, object pose and mesh reconstruction for indoor scenes from a single image. In: IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) (2020)
[40] Nie. Y., Hou, J., Han, X., Niessner, M.: Rfd-net: Point scene understanding by semantic instance reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
[41] Noguchi, A., Iqbal, U., Tremblay, J., Harada, T., Gallo, O.: Watch it move: Unsupervised
discovery of 3D joints for re-posing of articulated objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
[42] Peng, S., Niemeyer, M., Mescheder, L., Pollefeys, M., Geiger, A.: Convolutional occupancy networks. In: Proceedings of the European Conference Computer Vision (ECCV) (2020)
[43] Reading, N.: Generic rectangulations. European Journal of Combinatorics 33(4), 610-623 (2012)
[44] Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized inter- section over union: A metric and a loss for bounding box regression. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR). pp. 658-666 (2019)
[45] Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: Pifu: Pixel-aligned im- plicit function for high-resolution clothed human digitization. In: The IEEE/CVF International Conference on Computer Vision (ICCV) (October 2019)
[46] Saito, S., Simon, T., Saragih. J., Joo, H.: Pifuhd: Multi-level pixel-aligned implicit function for high-resolution 3d human digitization. In: CVPR (2020)
[47] Simonelli, A., Bulo, S.R., Porzi, L., Lnpez--nte M., Kontschieder, P.: Disentangling
monocular 3d object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1991-1999 (2019)
[48] Solovyev, R., Wang. W., Gabruseva, T.: Weighted boxes fusson: Ensembling boxes from different object detection models. Image and Vision Computing (2021)
[49] Sturm, J., Stachniss, C., Burgard, W.: A probabilistic framework for learning kinematic models of articulated objects. Journal of Artificial Intelligence Research 41, 477-526 (2011)
[50] Tancik, M., Weber, E., Ng, E., Li, R., Yi, B., Kerr, J., Wang, T., Kristoffersen, A., Austin, J., Salahi, K., et al.: Nerfstudio: A modular framework for neural radiance field development. (2023)
arXiv preprint arXiv:2302.0422
[51] Tang, J., Chen, X., Wang. J., Zeng, G.: Point scene understanding via disentangled instance mesh reconstruction. In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)
[52] Tatarchenko, M., Richter, S.R., Ranftl, R., Li, Z., Koltun, V., Brox, T.: What do single-view 3d reconstruction networks learn? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3405-3414 (2019)
[53] Tseng, W.C., Liao, H.J., Yen-Chen, L., Sun, M.: Cla-nerf: Category-level articulated neural radiance field. In: IEEE International Conference Roboties and Automation (ICRA). pp. 8454-8460 (2022)
[54] Tyszkiewicz, M.J., Maninis, K.K., Popov, S., Ferrari, V.: Raytran: 3d pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers. In: Proceedings of the European Conference Computer Vision (ECCV). pp. 211-228 (2022)
13

---

[55] Watson, J., Aodha, O.M., Turmukhambetov, D., Brostow, G.J,, Firman, M.: Learning stereo from single images. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)
[56] Wei, F., Chabra, R., Ma, L., Lassner, C., Zollhöfer, M., Rusinkiewicz, S., Sweeney, C., New- combe, R., Slavcheva, M.: Self-supervised neural articulated shape and appearance models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
[57] Weng. Y., Wang, H., Zhou, Q., Qin, Y., Duan, Y., Fan, Q., Chen, B., Su, H., Guibas, LJJ:: Captra: Category-level pose tracking for rigid and articulated objects from point clouds. In: Proceedings of the IEEE/CVF International Conference Computer Vision (CVPR). pp- 13209-13218 (2021)
[58] Xiang, F., Qin, Y., Mo, K., Xia, Y., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang. H., et al.: Sapien: A simulated part-based interactive environment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1097--11007 (2020)
[59] Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1492-1500 1500 (2017)
[60] Yang, G., Vo, M., Neverova, N., Ramanan, D., Vedaldi, A., Joo, H.: Banmo: Building animatable 3d neural models from many casual videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2863-2873 (2022)
[61] Zhang, C., Cui, Z., Zhang, Y., Zeng. B., Pollefeys, M., Liu, S.: Holistic 3d scene understanding from a single image with implicit representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
[62] Zhou, Y. Barnes, C., Lu, J., Yang, J., Li, H.: On the continuity of rotation representations in neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
[63] Zuffi, S., Kanazawa, A., Berger- Berger-Wolf, T., Black, M.J.: Three-d safari: Learning to estimate zebra pose, shape, and texture from images" in the wild". In: Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR). pp. 5359-5366 (2019)
[64] Zuffi, S., Kanazawa, A., Jacobs, D.W., Black, M.J.: 3d menagerie: Modeling the 3d shape and pose of animals. In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR). pp. 6365-6373 (2017)
14

---

A Details on kinematics-aware part fusion (KPF)
In Algorithm [. we present the details of Kinematics-aware part fusion. We gather part proposals through No independent inference runs, achieved by randomly sampling query positions {qn} J with the furthest point sampling (FPS). The 'model' in the algorithm represents the decoder 'D and the refiner/ R, and Yig represents objectness, defined as 1-bg. bg.
After part proposals are generated, we apply Non-Maximum Suppression (NMS) with 3D bounding box IoU and threshold by objectness Ybg before applying NMS with kinematics-aware IoU (NMS-kloU) for performance reasons. PF-kloU denotes the part fusion (PF) process using kloU, which is based on the Weighted Box Fusion (WBF) approach as described in 4 The procedure is further detailed in Algorithm 2
Unlike the conventional WBF, algorithm fuses all parameters in the part proposal rather than only the 2D bounding box parameters. Furthermore, our algorithm iteratively runs until convergence, demonstrated from line 11 to line 15 in Algorithm[[] C' represents clusters of overlapping part proposals identified by kloU. We employ a simple arithmetic weighted average for all elements except the rotation matrix in part proposal for fusion, denoted weighted-average.
For the rotation matrix R E SO(3), we initially derive weighted average for the 3x3 matrices, labeled as Rwa E _3x33 We then compute the weighted-averaged rotation matrix by minimizing the Frobenius norm to Rwa. This is accomplished as R argming R Rwall F. following the 3D rotation library RoMa
The value of Y'g.. denotes the scaled objectness as a confidence score, considering the number of independent inferences Ng. If the number of part proposals in cluster fewer than the independent inference runs No. it indicates that only a limited number of independent inferences predict it. In such cases, adjust the corresponding confidence score Vig.ne by scaling down by the ratio of the number of part proposals in the cluster (C[n'') to the number of independent inferences T = No. Conversely, we scale up the confidence score if a cluster contains more part proposals than the independent inferences No. For the initial run part fusion (line 13 of Algorithm [}. we set 7 No to fuse NQ inferences. We assign T = 1 for subsequent runs, assuming we apply part fusion to single inference run fused by the previous part fusion.
Hyperparameters The hyperparameter ToU is set to 0.25, following the original code of 3DETR (37]. We set Tobj to 0.25, representing the uniform probability of the three joint types plus background class, as outlined in Section 3 Additionally, we simplify the hyperparameter by setting final equal Tobj. The hyperparameter TaoU assigned a value of 0.5, double the ToU value, to impose stricter overlapping threshold and ensure similar trajectory part proposals are clustered together. We experimentally determine the values for Tscaled and Teount. setting them to 0.1 and respectively.
B
Full Loss Formulation
During training, we use the same multi-layer perceptron (MLP) for part prediction each decoder layer in the decoder D. We calculate the total loss, Liotal, as defined in Section 3.7 independently for each layer and sum up these losses. The comprehensive loss formulation is as follows:
No Σ Epant + Spant + Linstunce, k-
Liotal
(4)
C Architecture
C.1
Detection backbone
We employ the masked-encoder from (37 as the encoder architecture for improved performance. We also employ the Fourier positional encoding, as proposed in (37), for the query embedding prior to input to the decoder. Please refer to (37] for comprchensive details and visualizations of the architectures.
Our encoder and decoder architectures strictly follows the ones proposed 3DETR (37
15

---

Algorithm Kinematic-aware part fusion (KPF)
Input: Subsampled point cloud P', scene feature F
Output: Set of detected parts = (Xi)
1: X' <- 0
2: for No times do
3:
{qn) t- FPS((P)
4:
X" <- ((q F*)
5:
X' 4- NMS(X", ToU)
6:
X" + [Xn, n [X'], Vfg.n > Tabi)
7:
X' + NMS- NMS-kloU(d', Tiou)
8: X + X" +X
9: end for
10: count f- 0
11: repeat
12: Xola <- X
13: d'+ (Xala)
14: count + count +
until X' Xold count Tcount
16: X' (X, n E [2], Vig.n > Tobj_linal)
return X'
Algorithm 2 Part Fusion with kloU (PF-kloU)
Input: Set of part proposals X'
Output: List of updated part proposals X"
1: C' t Empty list, X' + Empty list
2: Sort in desceding order by objectness
3: for vX E X' do
4: match <- False
5: for vX' X do
6: kloU, > Tktou then
7: matched <- True
8:
n' 6- index of X' X
9:
Append X Cin')
10:
[''n'] + weighted-averagee([[h)))
11:
Break
12: end if
end for
14: if not match then
Append X X' and 16: end if
end for
18: for Vn' E [X']] do
Ytan f- Yig.n' [CIn'll
end for
21: X' p [X,,In' E [X']. sian > Tscated
22: return X²
16

---

C.2 Part prediction MLPs
We adapt the box prediction MLPs outlined in [37 as our part prediction MLPs. Our part prediction MLPs retain the same architecture and are referred to as 'MLPs' in Fig. 3 of the decoder architecture. We employ 13 MIPs, each dedicated to a specific part proposal parameter. These parameters include the center offset from the query position Åc, rotation R&D, size current pose joint state for prismatic
and revolute types deure and deu current fully-opened pose joint state for prismatic and revolute types d and desy joint axis a, revolute origin offset from the query position joint type y. category u, shape feature h, and part-to-instance embedding 2. We utilize the 6D rotation representation [62 as the MLP's output representation RED. subsequently converted to x33 rotation matrix.
C.3 Refiner
Our refiner architecture closely follows the decoder architecture, with ML.ss outputting residual parameters. It follows the same architecture of the part prediction MLPs from the decoder, except for the dropout layers. We employ nine MLPs for residual predictions of center, rotation, size, prismatic and revolute types' current pose joint state, prismatic and revolute types' fully-opened pose joint state, joint axis, and revolute origin. For the rotation matrix and joint axis, the MLPs output 6D rotation representation which are then converted into 3x3 rotation matrices. These are updated by matrix product. The remaining parameters are updated through the addition of residuals. The same positional encoding is applied to the part center с to generate an embedding feature, which is then added to the corresponding refined query embedding eNp from the decoder's final layer. This forms the new query feature input for the refiner, as depicted in Fig. 3
C.4 Shape decoder
Our shape decoder is based on the ConvONet architecture 42 It consists of a local point encoder, a volume encoder, and an implicit shape decoder. The local point encoder quantizes input points to multiple 2D (tri-planes) or 3D grids and encodes points within each grid. The volume encoder then takes these locally encoded points and further encodes them into 3D voxel feature. To decode the occupancy value at a 3D point x, we sample the local feature hocal by interpolation on the 3D voxel feature The implicit shape decoder outputs the occupancy value 0x at x. given hiocal,x as input.
Our approach differs from the original ConvONet in that we concatenate the shape feature per part proposal h to the local feature to form h/ h + h. This serves the learned part shape prior for the implicit shape decoder that reduces shape ambiguity due to the unsegmented per-part point cloud Po. outlined in Eq. [
We utilize the lightweight tri-plane architecture proposed in [42 for the volume encoder architecture. For more detailed information on the ConvONet architecture, please refer
42
C.5 Foreground segmentation model
Our segmentation model incorporates a ResNext50 (59 architecture for the encoder with RGBD input and DeePlaabV3Plus 6 from 18
for the segmentation head. We use the off-the-sheif implementation
D Implementation details
D.1 Architecture
Our approach strictly follows the hyperparameters of the masked-encoder described in [37]. We employ three self-attention transformer layers with self-attention masks between points in the point cloud, and each point only attends to others within a specified radius. The input point cloud to the encoder consists of Np 32768 points, which are subsequently subsampled to Npr 2048 points in the encoder. We set the scene feature dimension Dr a value of 256, following [37]
For the decoder, we maintain the hyperparameters from 37 except for the number of decoder layers ND. The decoder layers for the decoder and the refiner are set to No = 6 and N = respectively. We match the query embedding dimension with the scene feature dimension for addition
17

---

and set the number of queries to Ng 128 during training. For testing, Ng is set to 512, unless stated otherwise, with independent runs for query oversampling (QO) for kinematics-aware part fusion (KPF) with No 10 independent inference runs.
As for the part prediction Multi-Layer Perceptrons (MLPs), follow the box prediction MIL hyperparameters in (37], except for the number and output dimension of each MLP. The MLPs in the refiner for residual prediction strictly follow the part prediction MLPs hyperparameters in the decoder but without dropout. We set the dimension for shape feature h 128, and the dimension for part-to-instance association embedding 32.
With respect to the ConvONet [42] architecture in the shape decoder, the local point encoder consists of five MLPs of width 128, and the volume encoder has three MLPs with widths of 16, 32, and 64. We employ the tri-plane version of the volume encoder for volume representation, using only the xy and yz planes to save GPU memory. In the implicit shape decoder, we employ four fully-connected layers with a hidden dimension of 128 and a leaky ReL.U activation.
D.2 Training details
We set the weights for the matching cost Cmatch at λ = 8, A2 10, 1, 5. Each loss term in the total loss Greaal carries equal weight. We use the AdamW (31) optimizer with a base learning rate of 9 with a cosine scheduler down learning rate of le-6. The warm-up period is set to nine epochs, and mixed precision is used during training.
All models were trained on two A100 GPUs, each with 40GB GPU memory. We set the batch size to 26 per GPU, and it took approximately two days to complete 500 epochs. Weight decay was set at 0.1, with gradient clipping with an 1.2 norm of 0.1, as per (37]
We implemented on-the-fly sampling of 3D points and corresponding occupancy values to train the shape decoder. Half of these points are sampled uniformly from the space [-0.5, 0.5]3. Additionally, a quarter the points are sampled around the surface with a Gaussian-distributee random offset along the surface normal direction, with a standard deviation of 0.1. The remaining quarter is sampled with a standard deviation of 0.01. Each part has 128 points sampled for occupancy values.
In training the foreground segmentation model, we used the AdamW optimizer and a cosine scheduler, identical to the approach used in training the previous models. This model was also trained on two A100 GPUs with 40GB GPU memory and batch size of 26 per GPU, Training took approximately one day for 500 epochs.
We add noise to the depth map during training, following the approach in 32. We incorporate a depth map filter for the model tested on real-world data mitigate the flying pixel effect on the object edge as suggested by (55). This filter is also applied during testing for real-world data. Following the projection of the depth map to 3D points, we introduced random scaling within range of 15%, and random rotation along the surface normal direction within +:30% The point cloud was zero-centered for both training and testing. Color augmentation during training follows the original code of [37]. For the model using a foreground mask, we augment the mask by randomly displacing the foreground pixels around the border and performing successive dilation and erosion simulate noise on inferred masks during training.
D.3
Mesh generation
We sample occupancy values on 64 voxel grids per part, then extract the surface mesh using the marching cubes method [30) at an isosurface level To = 0.3775 = Sigmoid(-0.55 For surface mesh reconstruction evaluation, we store an instance mesh as union of part meshes and apply quadratic decimation [12] to reduce the number of faces to 10000. For volumetric loU evaluation, where we evaluate the union of IoU for each part, we stored part mesh and applied quadratic decimation to reduce the number of faces to 5000.
18

---

E
Dataset Details
E.1 SAPIEN 588 dataset
We choose ten categories offering prismatic and revolute joints as representative articulation types, adequate CAD models, and various part structures. The CAD models are randomly divided into training + validation splits (referred to as the trainval split) and test split, maintaining an approximate 8:2 ratio. No overlapping CAD model IDs exist between these splits.
All CAD models are manually aligned such that they stand upward along the y-axis and face the z-axis. We use BlenderProc [T0] to generate synthetic data. We randomly create room geometry (wall and floor) in each scene and place one to four instances.
For the trainval split, we randomize the texture of the part mesh and room geometries. The textures for the part mesh are randomly applied from https://www.. https:/wwww.beedderii...c blenderkit.com, using the downloading script of BlenderProc [TO]. We employ 2923 textures for this split. However, for the test split,
the original texture from the SAPIEN 58 dataset.
The room geometry textures are downloaded from https:/ambbietteg.o com using the BlenderProc script [10] and randomly applied. We use 353 textures for the trainval split and 85 textures for the test split, ensuring overlap.
For environmental lighting, HDR maps downloaded from https:/ppoyyaave.no hom are used. We employ 113 HDR maps for the trainval split and 29 for the test split, with no overlap.
Instances are created by applying random horizontal flips, random anisotropic resizing of side lengths, and random articulation of CAD models. The maximum joint state of the revolute joint defaults to unless it touches the ground. We set the maximum joint state value to avoid ground contact if it does. We use the CAD model's maximum prismatic joint state value for prismatic joints, adjusted for rescaling.
This paper considers instances comprising one base part and at least one articulated part attached to it. Therefore, we fix articulated parts attached to another articulated part and ignore their joints, such as small prismatic buttons and rotating knobs on doors.
The scene is populated with instances by first sampling one to four categories uniformly, with replacement. Then, a CAD model is randomly sampled for each selected category, and its part pose, size, and yaw rotation are randomly augmented. The size distribution is shown in Fig. 13 The yaw rotation is limited to +-90%, ensuring instances face roughly forward. Instances are placed randomly in the scene, maintaining a minimum and maximum distance of 0.1 and 0.8m from other instances, respectively. This setup prevents instances from being scattered in the room.
We allow up to 10 sampling of camera poses in the same scene, limiting the camera position to the union of convex hulls of instances projected on the x-z plane (same as the floor) and a circular sector whose center is at the scene center and the are is an angle between =80" This restriction, coupled with the yaw range limitation, discourages the self-occlusion of parts by an instance facing backward to cameras. We illustrate this idea in Fig. 13
We the BlenderProc API to sample camera poses against instances with a minimum distance to camera from an instance of 0.5m. In total, we render 4,000 frames for the test split and 188,726 for the trainval split, randomly sampling 20,000 frames for training and allocating the rest for validation usage.
We exclude instances whose 2D silhouette is occluded more than 50% or truncated by 25%. Each frame is ensured to have at least one visible instance. The image size is 360 x 640 (height and width), and the field of view is set to 86.64.
For the generation of watertight mesh for the implicit shape ground truth in shape learning, we follow the instructions from 335 to generate part-wise watertight meshes. We then apply quadratic decimation [12] to the generated part mesh, reducing the face count to approximately 10,000 when combined as an instance.
19

---

Dishwasher
Trashcan
Safe
Oven
Storage
Table
Microwave
Frige
Washing
Box
Trainval
33
47
24
14
266
60
10
33
13
20
Test
11
4
68
16
9
Table 7: Number of CAD models from SAPIEN synthetic dataset.
588
dataset used in trainval and test splits for our
Size
Dishwasher
Washcan
Safe
Oven
Storage
Microwave
Fridge
Washing
Category
Figure 12: Size distribution per category after size augmentation applied to CAD models.
E.2 BMVC 36 dataset
We the cabinet class data from this dataset, which includes one prismatic part and revolute part. The original data comprises two sequences of RGBD frames, with different part poses for each sequence and varied camera poses. The first sequence contains 1108 frames, while the second has 1119 frames. The image size is 480 x 640 (height and width).
Before feeding the data to the model, we apply depth map filter to suppress the flying pixel effect on the object edge, as detailed in [55]. We extract the foreground point cloud using the silhouette of the projected CAD model on the image.
E.3 Images taken by iPhone X
We record the partial front view of the scene as short clip, extracting approximately 300 frames to train the neural rendering model Nerfacto [50] for depth map generation. Default training hyper- parameters are used for Nerfacto training. The image size is 540 x 960 (height and width). For our evaluation, foreground masks are automatically extracted using https://www.removt
unless specified otherwise. Note that we only use the point cloud lifted from a single RGBD image as input, not recovered from all frames.
Details on baselines
F11
A-SDF 38
Data preparation We follow the description in the original A-SDF paper 38 for data preparation. After normalizing the global pose and scale, we ensure the base part's position does not change regardless of random articulation during training. In addition, for categories with multiple parts, we maintain consistent part ordering. We automate this by assigning part positions in canonical poses to a cell in the 5 x 5 5 voxel, which discretizes the positions. The ID flattened cell positions of the voxel determine the part order. limit the training time, we randomly sample a maximum of 1500 instances per category for the training split.
20

---

Instance's yaw angle range
Camera position sampling area
Instance B
Instance A
Camera pose
Camera pose 3
Camera pose 2
Figure 13: Illustration of yaw rotation range of an instance and camera position sampling area.
Model The model implementation uses publicly shared author code. The original A-SDF only targets revolute parts. We normalize the prismatic joint state by the maximum joint state to fall within the [0, range. We then multiply this range by 135, ensuring joint states for revolute and prismatic parts are within the same range to extend the model for prismatic parts.
A-SDF is category-level approach that works with fixed number of parts, which becomes problem when evaluated on dataset containing instances with varied part counts in category. Similarly to our approach, OPD [22 can handle multiple part counts without assuming the predefined part number. They evaluate their method against baselines targeting fixed number of parts by training the baselines for category's 's most common part count.
We follow this protocol but aim to favor the baseline A-SDF more by training up two models per category. The first model is for the most common part count, and the second, if applicable, is for the next most common part count. The second model is used when category has more than two different part counts in the test split.
This setting differs from OPD's method, which only uses the most common part count. During testing, we use the model with the most frequent part count for instances with untrained part counts. We call the baseline model trained for the most frequent part count A-SDF-GT and the using the next most frequent part count A-SDF-GT-2, The baselines and per category part counts, excluding the base part, summarized in Table
Training We employ the publicly shared author code for training. During training, each part's articulation varies randomly from Oo 135", sampled at 5"intervals as suggested in (38). We dynamically sample 3D points and their corresponding signed distance values from the randomly articulated watertight meshes of parts during training. The number of sample points and the ratio of uniformly sampled points points near the mesh surface follow the original A-SDF paper (38). A-SDF model training uses part label supervision and takes approximately ten days on single V100 or A 1100 GPU per model.
Mesh generation For mesh generation, we use the publicly shared author code as well. A-SDF assumes no background and normalized global pose and size for input. During testing, we use ground truth instance segmentation mask for each instance in a scene to isolate instance-wise foreground points. We also normalize the depth map to the input space using the ground truth pose and size. The surface mesh is extracted using the provided implementation, which samples signed distance values on 643 voxel grids and extracts surface mesh using marching cubes [30]. Additionally,
21

---

Dishwasher
Trashcan
Oven
Storage
Tablc
Microwave
Frige
Washing
Box
Most freq. Next Most freq.
2
4
Table 8: The most frequent and the next most frequent part count for each category that baseline A-SDF models are trained on.
apply the quadratic decimation [1] to limit the number of faces to 10000. We then project the generated mesh back to the original scale and pose in the scene using the ground truth values.
F.2
OPD 12
Data Following the original paper 22 and the publicly shared author code and dataset, we generate ground truth data. The original data includes semantic label of part type, one of drawer, lid, door. However, our synthetic dataset does not have corresponding labels. Thus we replace the semantic label with the joint type. Consistent with the original paper, we exclude the fixed type part from the ground truth, targeting only the revolute and prismatic parts for detection.
Model We employ the OPDRCNN-C model from 22 which predicts kinematic parameters in camera coordinates. For input modality alignment, the model uses RGBD input. We have adapted the model produce continuous 1D joint state: instead of supervising the joint state prediction head with binary joint state supervision, we use continuous ID joint state supervision. We duplicate the ML. head to separately predict joint states revolute and prismatic joints. We supervise only the output corresponding to the ground truth joint type. Furthermore, we guide the joint axis prediction with reference to the floor's normal direction, as we experimentally found this improves detection performance.
Training We follows the author's implementation for training code, maintaining the same settings and hyperparameters for the OPDRCNN-C model.
Testing As described in Sec. we match the ground truth using part segmentation evaluation with an IoU threshold of 50% following the author's implementation. We then evaluate the predicted joint parameters against the matched ground truth.
G
Category-wise shape mAP
We show category wise shape mAP results in Table
TO
13
14
15
16
17
18
H
Additional results on the refiner R
We evaluate the effectiveness of the refiner R in enhancing performance while maintaining compara- ble model size. Table 19 shows the quantitative results on the shape mAP and the number of learnable parameters of the Ours-BG model, with number decoder layers in the decoder Np = 6 and in the refiner NR 2. We compare this model against the model without refiner and having the same total number of decoder layers Np = 8, and another model with a doubled number of decoder layers No 16 without the refiner. The result shows that using the refiner achieves comparable performance in shape mAP with the model with doubled number of decoder layers Np 16 with smaller model size.
Fscore 80%
Fscore 6900%
CDI@5%
CDI @ 1%
loU@25%
loU lo0@50%
A-SDF-GT
38
59.71
43.14
62.79
29.64
29.64
0.00
A A-SDF-GT-2 38
58.40
41.49
61.36
26.23
26.89
0.22
Ours-Bo BG
85.73
83,93
85.94
80.72
41.75
0.15
Ours
88.28
84.67
91.14
76.00
42.01
0.33
Table Shape mAP result of dishwasher category.
22

---

Fscore @ 80%
Fscore 90%
CDI @55%
CDI@ 1%%
IoU@255%
ToU@50%%
A-SDF-GT 138
78.76
61.84
82.73
55.08
12.95
0.00
A-SDF-GT-2
81.45
70.83
84.13
62.78
21.24
0.00
Ours-Bo BG
76.41
72.21
78.67
71.15
16.78
0.01
Ours
78.23
74.63
81.03
72.32
19.08
0.00
Table 10: Shape mAP result of trashcan category.
Acoree@805
I'soree 90%
CDT 695%
CDI @ 1%
150@
10%%
SDF-GT [38
69.19 19
48.34
73,42
40.21
37.00
5.15
A-SDF-GT-2
69.53
49.83
73.31
41.24
38.14
5.61
Ours-BG
59.25
55.52
59.45
44.15
46.43
3.27
Ours
56.69
53.21
57.47
43.40
46.03
6.28
Table 11: Shape mAP result of safe category.
Fscore 80%
Fscore 90%
@DI@5%
CD1 1%
IoU@25%
loU@50%
A-SDF-GT 388
83.86
82.17
84.09
72.80
83.75
41.31
A-SDF-GT-2 [8
83.86
82.05
84.09
71.90
83.52
40.86
Ours-BG
89.01
87.28
90.64
79.68
83.28
15.60
Ours
90.62
87.31
92.51
78.53
82.25
29.65
Table 12: Shape mAP result of oven category.
Escore
"score@9UU5
@
CDI@ 1%
@@2%
A- A-SDF-GT
38
37.43
20.14
53.24
17.52
15.24
0.68
A A-SDF-GT-2 38
38.23
21.05
53.58
19.34
18.66
0.57
Ours-BG
57.44
52.97
57.09
42.59
29.17
0.17
Ours
58.66
54.60
59.12
40.87
30.76
1.28
Table 13: Shape mAP result of storage category.
"score@8800
"score@9055
CDI@5%
CD1@ 1%
IoU 155%
100@50%
A-SDF-GT 38
57.68
33.22
79.75
25.26
19.23
0.00
A-SDF-GT-2 38
57.57
32.08
81.11
25.48
16.38
0.00
Ours-BG
73.30
51.98
79.97
29.69
5.57
Ours
73.67
47.16
78.91
28.74
14.61
0.00
Table 14: Shape mAP result of table category.
Escore 80%
Fscore 90%
CDI@5%
CD1 @ 1%
loU@25%
100%00%
A-SDF-GT 38
84.62
63.59
89,74
$9.23
72.69
12,95
A A-SDF-GT-2 38
86.03
71.28
90.13
65.51
77.05
8.33
Ours-BG
64.42
63.02
65.78
63.00
59.50
4.94
Ours
65.81
64.05
69.12
64.07
$7.62
5.40
Table 15: Shape mAP result of microwave category.
Fscore
Fscore @9%%
CDI@5%
CDI@ 1%
loU@25%
loU@50%
A-SDF-GT 38
63,88
39.76
72.97
29.70
7.15
0.00
A-SDF-GT-2 38
71.03
52.85
75.88
44.48
13.82
0.12
Ours-BG
73.19
68.83
73.64
62.41
20.88
0.09
Ours
70.66
65.69
74.59
56.36
19.92
0.23
Table 16: Shape mAP result of refrigerator category.
23

---

Fscore @ 80%
CDI@5%
CDI@
IoU@25%
100@50%
A AT 38
65.85
41.10
77.38
24.41
77.72
48.04
A-SDF-GT-2 38
67.64
41.10
75.48
22.84
79.06
48.38
Ours-BG
86.40
78.12
87.43
45.74
87.22
73.74
Ours
90.30
83.25
92.47
42,96
92.29
73.16
Table 17: Shape mAP result of washing machine category.
Fscore @ 80%
Fscore @ 90%
CDI @5%%
CD1 1%
loU@25%%
IoU 100%
A-SDE-GTT [38
53.96
43.57
69.89
43.80
10.86
A-SDF-GT-2 38
74.38
62.93
80.05
55.96
11.10
0.24
Ours-BG
77.10
74.12
78.47
67.02
10.00
0.00
Ours
74.74
69.19
77.57
62.03
8.96
0.00
Table 18: Shape mAP result of box category.
I Effect of the refiner on kinematic estimation
The effect of the refiner R on kinematic estimation is shown in Fig. 14 joint parameter estimation. These evaluations focus on the intersection of true positive detections from both the model with and without the refiner with various mAP thresholds. Except for the joint axis orientation error at mAP@90% the refiner improves the joint parameter estimation.
We conduct evaluations on
Additional qualitative results on synthetic dataset
We show additional qualitative results SAPIEN 58 dataset in Fig.
15
K
Additional qualitative results on real-world data
Additional qualitative results on real-world data are shown Fig. 16 For the washing machine example in the top left corner of the figure, semi-automatic segmentation tool Microsoft PowerPoint was used for foreground segmentation by specifying the rough foreground region, as http://remove.bbg could not properly segment the foreground in this example.
L Qualitatative results from novel viewpoint
Corresponding to Fig. and Fig. in we visualize the qualitatative results from novel viewpoint on SAPIEN (88 dataset and BMVC (36] dataset Fig.
and Fig. respectively.
M Failure cases and limitations
Uncertainty in closed parts One failure case arises when our model attempts to estimate the orientation of a part opening while in its closed state, as illustrated in Fig. 19 (a). We show quantitative evaluation when its ground truth motion amount is less than a threshold. For revolute joints, we use and 10 as thresholds, and 5% and 10% of the fully opened amount for prismatic joint. As seen in the table, the model successfully estimates reasonable joint direction for prismatic parts, while for revolute
axis direction error for closed parts in Table 20 We regard part closed
Fscore@
@'co
@D1
CD1@1%
IoU@25%
e50%
Params.
No = 16
73.25
68.27
74.76
58.64
40.74
10.57
14,33M
No 8
71.88
73.68
54.79
35.31
8.20
9.05M
No 6. No.
74.22
75.71
40.06
10.25M
Table 19: Ablation on the refiner R. for shape mAP and the number of parameters.

---

Rovolute joint state ertor mARg
Prismatic (oint state aror mARg@00%
Oriertation terttttiinn Error mAki 30ts
Kinimum Distance mAPad00
2.3
23
Medel
Model
Model Narros
maPgg80
Prismatic:
mAPg@60%
mAinggos
whndin
M0WM SOUTE
Revolute jont state entrr mARgsTOS
Prismatik jont state ertr MARgTON
Onietation Emror m07g
Kinimum Distance mAPg10%%
joint 60R
Prismatic: mAPO600%
mAPgeO's
Madel Nameo
Vedul/auree
Modid Narros
Figure 14: Effect of the refiner to joint parameter estimation.
<5%5
<10%/0%
Revolute
Prismatic
Average
Revolute
Prismatic
Average
Ours
(3.15)
2.04"(2.60' (2.60")
24.46
(3.08")
18.09 (2.80")
2..60(2.60')
16. (2.77. 78")
Ours-BG
(3.36))).39 (2.42))
28.85%)).279)
22.27
2.56%(2.43))
20.64)(2.859)
Table 20: Evaluation of joint axis direction enror for closed parts. The part regarded as closed its ground truth motion amount is less than the threshold. For the prismatic part, we normalize the motion amount by fully opened amount for thresholding to consider the different scales of parts. For reference, we denote the corresponding error of open parts in the parenthesis.
parts, the error is larger. Two contributing factors are: (1) our model does not explicitly model the uncertainty, particularly in ambiguous cases like the joint axis direction shown in Fig. 19 (a). The prismatic part's joint direction has less diversity due to the physical constraint by the base part, while several joint directions are possible for the revolute joint given the base part pose, thus the opening direction is ambiguous to the model. (2) The model could not detect small geometric cues from noisy depth maps, such handles or knobs, which can provide valuable information about opening directions. These cases can be improved in several ways. For instance, we could extend the model to treat joint parameter ambiguity as probability distribution, providing multiple joint parameter candidates sampled from the estimated distribution. This approach still be practical, limiting the
25

---

Input
GT
A-SDF-GT-2
Ours
Figure 15: Additional qualitative results SAPIEN
58 dataset.
kinematics parameter space few options for initial interaction steps in robotics applications allowing humans to choose the best prediction in AR/VR applications during human-in-the-luee process. Previous work [1] [1 has shown the effectiveness of this probability-base
approach. Another approach would be to integrate 2D and 3D features into the detection backbone [54] 26 77].
Incorporating 2D features would enhance the encoder's ability capture small yet informative features, like handles or knobs, that may be missed by sparse point cloud representation alone.
Unseen categories Another failure case that our model struggles to reconstruct instances from unseen categories, particularly those with shapes and part structures with small samples absent in training data. Fig. (b) shows the failure case with a dehumidifier as an unseen category, which has unseen drawer shape and part structure; the base part only has one bottom drawer. This limitation can be improved by incorporating more data from recent datasets, such as [13 34]. which offers scanned real-world objects of various shapes, sizes, and part structures. However, it's important to note that generalization to unseen categories is beyond the scope of this paper. Moreover, in terms of part structure, it's also important to note that previous works can only handle very limited part structures, such as one base part and one articulated part [16] and predefined part count per category [88 In contrast, our model can handle various part counts with a single model.
False positives and negatives Due the detection- based nature of our model and also the domain gap to real-world data, false negatives or positives still occur, as shown in Fig. 19 (c) and (d). One practical improvement would including real-world data [33 34 in training minimize the domain gap, hence boosting the model's generalization capabilities. Another approach for reducing false positives especially with physically implausible configurations, such physical visolation loss [6 for learning physical constraints. It's important to note that the kinematics-awarc part fusion improves both false positives and negatives, as discussed Sec.
include regularization losses,
Objects with many joints The proposed approach works reasonably for complex articulated objects such as those with more than five joints, shown in Fig. 20 (a) when all parts are visible from the given viewpoint. However, from certain viewpoints, we also observe that a single instance reconstructed as two separate instances, each with fewer joints, shown in the first row of Fig. 20
26

---

RGB
Current pose
RGB
Current pose
Depth
Fully opened
Depth
Fully opened
RGB
Current pose
RGB
Current pose
Depth
Closed
Depth
Closed
RGB
Current pose
RGB
Current pose
Depth
Closed
Depth
Closed
Figure 16: Additional qualitative results on real-world data.
(b). We attribute this to the current dataset consisting of small number of CAD models with >5 joints, while and the majority of instances have fewer joints. Also, when some parts are only partially visible, method tends make inaccurate pose estimations for such parts, such the right stacked three drawers indicated by the red box in the second row of Fig. (b).
Latency of KPF module While the KPF module improves detection performance, query oversam- pling, part proposal fusing, and kloU calculation add extra running time for inference. We visualize the latency and accuracy trade-off in Fig. 21 We change the number of query oversampling from one to ten, and measure the per image processing speed in seconds. We randomly sample 100 samples from the test split for the evaluation and latency is measured on V100 GPU. Although detection perfomance in shape mAP improves with more query oversampling, the latency degrades. Note that the result of w/o KPF applies 3D cuboid NMS with GPU parallelization, while the KPF module is implemented on CPU without any parallelization. We believe the latency of the KPF module can be significantly improved by optimizing the implementation, such utilizing GPU parallization.
27

---

Input
GT
A-SDF-GT-2
Ours
Figure 17: Qualitatative results on SAPIEN
58
from novel viewpoint.
Input
A-SDF-GT-2
Ours
Input
GT
A-5DF-GT-2
Ours
Figure 18: Qualitatative results on BMVC 36 from novel viewpoint.
28

---

Dehumldifier
RGB
Current pose
RGB
Current pose
Depth
Fully opened
Depth
Fully opened
(a) Ambiguous opening direction
(b) Unseen category
Current pose
Current pase
Depth (c) False positive
Fully opened
Depth
Fully opened
(d) False negative
Figure 19: Failure cases on real-world data.
29

---

Input
GT
Ours
(a) Successful cases of objects with >5+ joints
Input
GT
Ours
(b) Failure cases of objects with >5+ joints
Figure 20: Successful and failure cases of articulated objects with more than five joints.
J// KPF Q0X10
68.5
/// KPF QOXB
KPF QOX4
F-Score@90%
68.0
w/o KPF
J// KPF QOX6
w/ KPF QOx1
67.5
J// KPF
w/ KPF QOX2
W/ KPF Q0X4
'// KPF QOX2
67.0
W/ KPF Q0X6
W/ KPF QOX8
66.5
J/ KPF Q0x1
w/ KPF Q0x10
10
20
30
40
50
Latency (second)
Figure 21: Latency and accuracy trade-off for different numbers of query oversampling (QO) KPF module.
30


Reviews:
[{"review_id": "FY7dopv94h", "reviewer": "Reviewer_VtGS", "summary": "The paper proposes an end-to-end trainable method for reconstructing multiple articulated objects from a single RGB-D image, consisting of detecting parts, reconstructing part-level shapes, and estimating poses, bounding boxes as well as kinematic parameters. The parts are grouped into instances later. The authors propose anisotropic scale normalization for shape reconstruction to accommodate various part sizes and scales. Besides, the authors also propose test-time kinematic-aware part fusion (similar to non-maximum suppression) to reduce false positives when multiple detected results are generated and needed to be merged. Evaluation on both synthetic and real data demonstrates the effectiveness of the proposed method.", "questions": "1. How do the authors choose the revolute origin? It seems that any point on the revolute axis can be chosen as the origin. I think this question also applies to prismatic joints.\n2. Is there any explanation why anisotropic scaling is better than isotropic one?\n3. L235-L236: Do you actually mean 20,000 validation images and 168,726 training images? Otherwise, it is strange why more images are used for validation instead of training.", "limitations": "The authors has addressed the limitations.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is clearly written and easy to follow.\n- The authors propose a clean pipeline for reconstructing articulated objects that can be trained end-to-end and inferred straightforwardly without composing multiple networks.", "weaknesses": "Minor typos:\n- L38: \"a end-to-end\" -> \"an end-to-end\"\n- L219-L220: \"and use cosine scheduler to learning rate of 1e-6\" sounds a little strange to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper!\n\n## How to choose revolute origin?\nAs pointed out, any point on the line formed by GT revolute origin and GT joint axis is the correct revolute origin. Thus for evaluation, we measure the minimum distance (MD) between the GT axis line and the predicted revolute origin. During training, we simply minimized the L2 distance between the GT revolute origin and prediction, following OPD. As a prismatic joint does not have a GT origin, we only optimize the predicted revolute origin if the matched GT part’s kinematic class is revolute type.\n\n## Why is anisotropic scaling better than isotropic one?\nAlthough we have explained the reason in the main paper's L308 in Section 4.3 and L121-122 in Section 3.3 of the main paper, let us recap here again for better understanding. With isotropic scaling, the shape decoder needs to learn variations in shapes with different width, height, and depth ratios. Anisotropic scaling normalizes all three sides to unit length, reducing the variation of shapes that the shape decoder needs to learn and making it easier for the shape decoder to be optimized during the training.\n\n\n## Validation data size\nPlease refer to the global comment for a detailed response.\n\n## Writing errors\nWe appreciate the reviewer for pointing this out. As responded in the global comment, we promise to fix this in camera ready. \n\n\n"}, {"review_id": "ZK5NbQFEOF", "reviewer": "Reviewer_Y464", "summary": "The paper proposes a method for man-made articulated objects reconstruction from a single RGBD image across different object categories. The method is based on a detect-then-group pipeline, using kinematics-aware fusion for addressing false negatives.\n", "questions": "- L.126: why P_O both conditions O and is given also as an argument?", "limitations": "Limitations are discussed in the text", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method addresses the challenging problem of man-made articulated object reconstruction from a single RGBD image, estimating at the same time each part's pose. \n\nThe overall architecture has a somewhat complicated structure with numerous encoder and decoder modules employed. Nevertheless, the design choices are sufficiently motivated in the text and the loss function is quite straightforward given the overall architecture. The experimental evaluation considers both a synthetic (SAPIEN) as well as a real-world (BMVC) dataset. The proposed method achieves improved performance with respect the state-of-the-art on SAPIEN. The ablative study shows the relative contribution of important components. \n\nRegarding reproducibility, sufficient details are provided making it easier to reproduce the results (considering also the supplemental material).", "weaknesses": "The method seems to be closely related to [20]. I think a more detailed discussion about the relation between the two methods should be included in the Related work section. Related to this, it is not clear why the experimental evaluation does not consider comparison also with [20]. This is important, considering the few baselines available for this task. Alternatively, methods considering multi-view reconstruction on SAPIEN could also be considered to give additional insight on the performance of the proposed method. On a side note, the considered metrics could also include the joint state error.\n\n### Minor comments\n- L.45: define acronym NMS\n- L.127: not clear\n- Fig. 2 is not referenced in text\n- Although prior work regarding human subjects is briefly visited in the related work, similar work for animals is briefly mentioned but not specified. Works like [R1], [R2] and [R3] could be included to make this aspect of related work stronger.\n\n[R1] Ntouskos, V., Sanzari, M., et al. (2015). Component-wise modeling of articulated objects. ICCV\n\n[R2] Zuffi, S., Kanazawa, A., et al. (2017). 3D menagerie: Modeling the 3D shape and pose of animals. CVPR\n\n[R3] Jiang, L., Lee, C., Teotia, D., & Ostadabbas, S. (2022). Animal pose estimation: A closer look at the state-of-the-art, existing gaps and opportunities. Computer Vision and Image Understanding, 103483.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and questions!\n\n## Joint state error\nWe show the joint state error compared with OPD in Table 3 in the main paper, denoted as “State.” We have also added the joint state error comparison against A-SDF-GT-2 in the table below. Note that A-SDF-GT-2 can only be evaluated against GT instances with its learned part counts. Therefore, we evaluated A-SDF-GT-2 and ours only on the GT instances with the part counts that A-SDF-GT-2 learned.\n\n\n### Joint state error evaluation\n|               | Revolute (deg) | Prismatic (cm) |\n|---------------|----------------|----------------|\n| A-SDF-GT-2    | 25.49          | 13.97          |\n| Ours          | **4.62**       | **4.80**       |\n\n\n\n\n## Why is P_O both conditions O and is also given as an argument?\nWe appreciate the reviewer for pointing this out. Actually $P_\\mathcal{O}$ is in the right place but $\\mathbf{p}$ should be $\\mathbf{x}$. The correct equation is as follows:\n\n$o_{\\mathbf{x}} = {\\mathcal{O}}((\\mathbf{R}\\mathbf{S})^{-1}(\\mathbf{x}-\\mathbf{c}) \\mid P_{\\mathcal{O}},\\mathbf{h})$\n\n$\\mathbf{x}$ is in world coordinates, and $(\\mathbf{R}\\mathbf{S})^{-1}(\\mathbf{x}-\\mathbf{c})$ projects it into the local coordinate of the part to sample occupancy value at world coordinates $\\mathbf{x}$. We will fix the equation in camera ready.\n\n## Comparison with PPD [1]\nPlease refer to the global comment onthe comparison with state-of-the-art. \n\n## Additional related works\nWe appreciate an idea to improve our related work section. We will include them in the camera ready.\n\n## Writing errors\nAs responded in the global comment, we promise to fix this in camera ready. \n\n[1] Kawana et al. Unsupervised Pose-aware Part Decomposition for Man-made Articulated Objects, ECCV 2022.\n"}, {"review_id": "FWQb2hshfd", "reviewer": "Reviewer_MJHC", "summary": "The paper presents a detection-based reconstruction method for articulated objects along with estimating part-level 6D object poses, sizes, and joint parameters. The paper uses 3DETR as the backbone predicts all of the above quantities while treating this problem as a supervised learning approach given labeled synthetic data. The approach trains only on synthetic data and transfers to real world dataset. ", "questions": "Please see the questions raised above in the weakness section. To summarize, a comparison to strong and relevant system-level baselines, more qualitative results, and discussion/comparison to speed vs accuracy tradeoff would be useful to answer. ", "limitations": "The authors can only handle cases where instance boundaries are defined so it would fail in scenarios where a door would be attached to a room let's say.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper effectively uses a detection backbone i.e. 3DETR to reconstruct multiple articulated objects from a single view RGB-D observation. Sim-to-real transfer uses only synthetic data pretraining is a strong result. The use of set matching (from instance segmentation) literature for articulated part reconstruction is interesting. Joint estimation and shape reconstruction quantitative results are significantly better than baselines. I believe breaking the problem in this way i.e. part level reconstruction omits the need for model selection (i.e. training one model per category such as different models for glasses (2 joints) and refrigerators (1 joint) which is a major limitation in previous works.", "weaknesses": "1. The paper mentions CARTO, which is super relevant and recent CVPR'23 work but doesn't directly compare to it in terms of both detection and reconstruction. Is there a reason for it? Rather the paper takes individual joint parameter baseline i.e. OPD and shape reconstruction baseline i.e. A-SDF and compares them separately to those. While these are relevant, CARTO is the most relevant to this work in terms of system-level joint detection and shape reconstruction. \n\n2. While the work offers a solution for handling an arbitrary number of parts in the image, the qualitative results show simple examples i.e. same types of joints and 2 joints mostly. Did the authors test their approach on more complicated geometries i.e. varying number of joints or articulated objects with more joints >5+.\n\n3. I didn't see a discussion to run the speed of the model. The most relevant works which the author discusses i.e. CARTO (which builds upon CenterSnap and ShaPO) are all fast approaches. Is 3DETR equally fast and did the author consider adding that backbone i.e. CenterSnap backbone which offers a faster solution + equally good model in terms of accuracy? This is crucial for real-time applications like robotics or grasping etc. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! \n\n## Targeting articulated objects with >5+ joints\nThe trained model works reasonably for complex target instances like >5+ joints, as shown in Fig. 2 of the attached material, especially when all parts are clearly visible from the given view. However, from a certain viewpoint, we also observe that the single instance is reconstructed as two separate instances (Fig.3 of the attached material, first row) with fewer joints for each instance. We attribute this to the current dataset consisting of a small number of CAD models with >5+ joints, and the majority of instances have fewer joints. Also, when some parts are only partially visible, our method tends to make inaccurate pose estimations for such parts (Fig.3 of the attached material, second row, the right stacked three drawers indicated by red box). We add these cases as limitations in camera ready.\n\n## Comparison with CARTO\nPlease refer to the global comment on the comparison with state-of-the-art.\n\n## Real-time application\nCurrently, improvement in detection-to-reconstruction speed compared to the state-of-the-art methods is not our focus in this paper. Combination of real-time DETR-based 3D detectors like MonoDTR [1], tuning the number of queries based on speed vs. accuracy trade-off, employing hierarchical isosurface sampling for fast meshing as CARTO, re-implementing current single processed, CPU-based implementation of KPF module with multiprocessed or GPU-based implementation can be possible extensions for speed improvement, and we leave it for future work. We will include quantitative speed measurement in the camera ready.\n\n[1] Huang et al. MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer, CVPR 2022.\n\n\n"}, {"review_id": "JNEAROmWUM", "reviewer": "Reviewer_HLDh", "summary": "This paper presents a new method for 3D semantic instance reconstruction at the part level from a single RGB-D image. This method follows a top-down manner by first detecting object parts using 3DETR, where each instance part's bounding box will be predicted. The point cloud located inside each bounding box contains the part surface geometry, which will be normalized into a uniformly-scaled canonical system for the following part shape reconstruction. The key contribution of this paper is the kinematics-aware part fusion module, which follows a bottom-up manner to constitute complete instance shapes from parts.\n\nThe paper writing could be improved, e.g., the first paragraph in the introduction section is actually elaborating on related work. An illustration in Sec 3.6 would be more accessible for the audience to follow.\n\nThe experiment metrics and designs are extensive but lack some comparisons with the state-of-the-art.\n", "questions": "Here are some suggestions to improve this work, with which I would consider raising my score.\n\n1. A comprehensive analysis and comparison between the key module of this method (e.g., KPF) and the state-of-the-art (A-SDF is a bit old).\n2. The introduction should be about telling the story of the paper's motivation, key observations and contributions. Listing many related works there made it difficult to follow.", "limitations": "This paper well discussed the limitations and failure cases in the supplemental.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In my view, the major strengths can be concluded as follows:\n\n1. Kinematics-aware part fusion (KPF). The authors follow a top-down and bottom-up manner to first detect part-level geometries from the point cloud. Then they use KPF to construct instance shapes from parts. These two processes are trained end-to-end jointly to improve the entire performance.\n\n2. It shows good qualitative and quantitative performance in both synthetic data and synthetic data, with an acceptable generalization ability, even though they only train their model with synthetic data.", "weaknesses": "The weaknesses of this paper are also obvious.\n\n1. This paper follows a similar top-down pipeline as previous works. (e.g., RfD-Net). In L34, the authors argued that previous methods \"are either limited to a single instance or require a separate instance detector, making them not end-to-end trainable\". I do not think this makes sense, as this paper also requires an instance detector. Besides, relying on a separate instance detector does not mean they can not be trained end-to-end.\n\n2. As this task is not new and this paper follows a similar pipeline, it means the contribution of this paper only comes from the module design. To me the major contribution lies on KPF module. But from the ablation study, it seems KPF does not contribute the most.\n\n3. Section 4.2, comparing with OPD by adjusting the IoU threshold is not rigorous. There is no theoretical guarantee that this comparison is fair. Since OPD is an RGB-based method while this paper uses RGB-D information, a fair comparison would be to add depth information into OPD or to remove the depth channel in your method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed review!\n\n## Effectiveness of the KPF module and comprehensive analysis\nThe primary source of improvement in shape reconstruction accuracy is our part-level reconstruction approach enabling the reconstruction of articulated objects with various part counts, which is the main contribution of this paper. In addition, we also observed qualitative improvement using the KPF module for challenging targets like distant and small part shapes, which we detailed in the global comment.\n\n## Comparison with more recent state-of-the-art baseline\nWe have added another baseline based on AKBNet from CVPR 2022 as a more recent state-of-the-art method besides A-SDF, as detailed in the global comment.\n\n## Improving introduction\nWe show the revised introduction idea below and promise to revise the introduction further in the camera ready.\n```\nEstimating object shape, pose, size, and kinematics from a single frame of partial observation is a fundamental challenge in computer vision. Understanding such properties of daily articulated objects like refrigerators and drawers has various applications in robotics and AR/VR.\n\nShape reconstruction of daily articulated objects is a challenging task. First, the objects have various shapes resulting from different local part poses. More importantly, they have intra- and inter-category diverse part configurations regarding part counts and structures. A combination of those factors results in exponentially increasing shape variation. Previous works handles only fixed part count by a single model [1] or use multiple category-level models for different part counts after instance-level detection [2] by modeling the target shape in instance-level latent space. Handling those varieties with a single model is a complex and unsolved task.\n\nIn this paper, we address this complexity through the novel detect-then-group approach. Our key observation is that the daily articulated objects consist of similar part shapes. For example, regardless of the number of refrigerator doors, each door can have a similar shape, and the base part may have similar shapes to those from other categories, such as storage. Detecting each part and then grouping them into multiple instances is a scalable and generalizable approach for diverse part configurations of daily articulated objects in a scene.\n\nBased on this idea, we propose an end-to-end detection-based approach for part-level shape reconstruction. <the rest of the text will follow the second and the third paragraphs of the introduction>\n```\n\n## Improving the figure for the KPF module\nWe promise to revise the current figure in Section 3.6 in the camera ready for better accessibility.\n\n## Using an instance detector does not mean end-to-end trainable\nWe agree that some methods, like RfD-Net, operate end-to-end from detection to reconstruction. Therefore, we promise to rephrase the original sentence not to imply that detection-based necessarily means not end-to-end trainable up to shape reconstruction.\n\n## Contribution of the paper\nAs a high-level idea, we base our approach on detector-based reconstruction approaches. However, we have developed these ideas and made novel progress. As reviewers **mcMG** and **MJHC** agree, introducing a detect-then-group approach that can handle daily articulated objects with an arbitrary number of parts, a major limitation of previous works, is novel. This presents a new setting for shape reconstruction of daily articulated objects and stands as the significant contribution of this paper against the recent previous works.\n\n## Fairness of comparison against OPD\nAs written in L275 of the main paper, we have included depth alongside RGB as an additional channel for OPD's input, ensuring a fair comparison. Addressing concerns about the IoU threshold's fairness, we've also conducted an additional evaluation with a favorable setting for OPD, which is presented in the table below. While we maintained the same evaluation settings as in the main paper, we selected the pair of IoU threshold values for OPD and ours from all possible combinations (50% to 90% with 10% steps for both OPD and ours) this time. We chose a pair of IoU thresholds in a way that gave the best metric values for OPD. In the table below, IoU thresholds for OPD and ours are 70% and 80% for the prismatic joint state, respectively, and 90% for the rest of the joint parameters for both OPD and ours. Despite these adjustments, our method still outperforms OPD.\n\n### Additional joint state evaluation\n|       | State (revolute/prismatic)                     | OE         | MD     |\n|-------|---------------------------|---------------|--------------|\n| OPD   | 16.52°/16.46cm            | 10.81°        | 34.68cm      |\n| Ours-BG | **3.34°**/**5.45cm**    | **1.96°**     | **5.15cm**   |\n\n\n\n[1] Heppert et al. CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects, CVPR 2023.\n\n[2] Liu et al. AKB-48: A Real-World Articulated Object Knowledge Base, CVPR 2022.\n\n\n\n\n"}, {"review_id": "phNTEVN58t", "reviewer": "Reviewer_mcMG", "summary": "The paper presents a novel task focused on the reconstruction of multiple articulated objects, considering part-level shape, pose, joint parameters, and part-instance association, using only a single RGBD image. The authors propose an effective detect-and-group strategy that harnesses the part-level representations to detect, reconstruct, and predict parameters for articulated objects with diverse structures. Additionally, to enhance the detection performance, the paper introduces an oversampling and fusion strategy during inference. Moreover, the authors incorporate anisotropic size normalization and a refinement module to improve reconstruction quality and enhance part pose/motion prediction.", "questions": "* Why the number of images in the validation set is much more than the number of images in the train set and test set? \n* For the number of objects in each image, what’s the statistics in the train set and test set? \n* For the model, why “ours-BG” sometimes outperform “ours”? “Ours” should have extra information of the foreground mask, right? Is this caused by the imperfect performance of the foreground mask?\n", "limitations": "The authors mention the limitation in the supplement.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed task of reconstructing any number of articulated objects is both novel and valuable for exploring potential downstream robotics applications.\n* Utilizing part-level representations and employing the detect-then-group strategy are intuitive and effective when handling articulated objects with diverse structures.\n* The proposed end-to-end method builds upon 3DETR by incorporating part-level detection and pose/motion prediction. Furthermore, the authors introduce an instance-loss to aid in grouping within the part latent space. \n* The experimental settings are reasonable and allow for meaningful comparisons with previous work in the fields of articulated object reconstruction and motion prediction. The results demonstrate the effectiveness of the proposed method.\n", "weaknesses": "* Due to the task's focus on reconstructing part shapes, it’s crucial to show some novel viewpoint of the recons trued shapes. For example, the reconstructed shapes of some base parts are hard to see the improvement only with quantitative results. Additionally, all visualizations in the paper appear to align with the input RGBD viewpoint, limiting the comprehensive understanding of the reconstructed shapes.\n* If I understand correctly, for the statistics in table 2, the instance number includes the same articulated objects with various part states, or it’s hard to explain why there are so many instances in each category (much more than the object number in SAPIEN dataset)\n* For the evaluation, it seems that in the test set, the part state are randomly initialized. Then most doors or drawers are actually open. However, in the real case, most time, the movable parts are closed. It’s better to have separate evaluation for the models with different motion states\n* The paper lacks statistics on the number of articulated objects present in the input image, and it would be intriguing to include evaluation results that consider the number of objects to better understand the performance of part-instance association.\n* The quantitative improvements achieved with QO, PF, and kIoU metrics do not appear to be substantial. Are there any qualitative results available to provide additional insights or demonstrate the effectiveness of the proposed module?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and questions!\n\n## Novel viewpoint of the reconstructed shapes\nWe promised to add novel views in camera ready. For reference, we show the novel viewpoint of the reconstructed shape in Fig. 5 and Fig. 7 of the main paper in Fig. 4 of the attached material. Our method qualitatively better reconstructs the shape of occluded regions than the baseline.\n\n\n## The instance number in Table 2\nWe augment each CAD model from the SAPIEN dataset in terms of three side lengths and randomize the part poses as explained in Section 4, L230. Randomizing the three side lengths changes the original shape significantly. Thus we count it as a unique instance in Table 2. We list the number of original CAD models for each category per split in Table 1 of the appendix. We also show the size distribution after the augmentation in Fig. 1 of the appendix.\n\n\n## Evaluation for closed parts\nWe follow the relevant previous works’ [1,2,3,4,5] standard experimental settings of uniformly randomized articulation for the test set. We qualitatively confirmed that our approach reasonably works for closed or nearly closed parts, as visualized in Fig. 7 (right) of the main paper, Fig 6. (top row) of the appendix, and Fig 3 (bottom, three stacked drawers on the left) of the attached material. We will add further analysis regarding the closed parts in the camera ready.\n\n\n## Evaluation results that consider the number of instances\nWe have evaluated shape mAP in terms of different numbers of instances in a scene, as shown in Fig. 5 of the attached material. As the number of instances in a scene grows, F-Score decreases. We attribute this to the camera being often distant from instances when a view contains multiple instances. Thus it becomes harder to understand the fine details of the part geometry accurately.\n\n\n\n## Qualitative results for the KPF module\nPlease refer to the global comment for the detail.\n\n## Larger validation set than the training set\nAlso, please refer to the global comment for the detail as well.\n\n\n## Statistics of the number of objects in each image\nThe average number of instances for the training set is 2.154, and 2.158 for the test set. \n\n\n\n\n\n\n## Why “Ours-BG” sometimes outperform “Ours”?\nAs pointed out, the imperfect performance of the foreground mask can be one reason. Another reason could be that sometimes background context near the instance would help detect an instance and estimate the part pose. In some cases, the background context of the floor might be informative for estimating the rotation of the part pose based on the tilt of the floor, and the position of the floor might help to estimate the center of the part. That being said, \"Ours\" without background outperforms \"Ours-BG\" with background in the majority of the metrics, as in the majority of cases when there is a background, less number of queries cover the foreground objects, becoming disadvantageous for detection and detailed understanding of part shapes and poses.\n\n[1] Heppert et al. CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects, CVPR 2023.\n\n[2] Liu et al. AKB-48: A Real-World Articulated Object Knowledge Base, CVPR 2022.\n\n[3] Kawana et al. Unsupervised Pose-aware Part Decomposition for Man-made Articulated Objects, ECCV 2022.\n\n[4] Jiang et al. OPD: Single-view 3D Openable Part Detection, ECCV 2022.\n\n[5] Mu et al. A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation, ICCV 2021.\n\n\n"}]

Author rebuttal:
# Global comment
We thank all the reviewers for their thoughtful feedback. We are encouraged that the reviewers having identified our paper making a good contribution (**mcMG**, **Y464**, **MJHC**, **VtGS**), and the proposed to be interesting (**MJHC**), intuitive and effective (**mcMG**), working on a novel task (**mcMG**), and addressing the major limitations of previous works (**MJHC**). We are also glad that all reviewers agree that the experiments demonstrate effectiveness and good performance on both synthetic and real data, significantly better than the baseline (**MJHC**), with reasonable experimental settings for meaningful comparisons with previous works (**mcMG**). Below, we write responses to the concerns common to several reviewers.


## Comparison to the state of the art (**Y464**, **MJHC**, **HLDh**):
To the best of our knowledge, no prior work operates in exactly the same setting as ours, which can handle cross-category, multiple-articulated objects with various part counts and structures. The closest work is the system-level (handling multiple articulated objects) reconstruction approach CARTO [1], which targets cross-category multiple articulated objects with a **single** articulated part. We were unable to directly compare against it due to the lack of an official code at the time of submission, and it was only made publicly available several days ago.

To alleviate this problem, we used the SOTA system-level baseline setting from the paper [1]: A-SDF with the ground truth (denoted as A-SDF-GT in our paper) for detection. This baseline provides an upper-bound system-level performance in detection and is used as a comparable baseline against CARTO in the paper [1]. To address concerns from fellow reviewers to consider another baseline (**Y464**) and to compare against a more recent approach than A-SDF from ICCV 2021 (**HLDh**), we added the new baseline based on the idea from AKBNet [2] (CVPR 2022), denoted as AKBNet-GT in the table below. AKBNet uses A-SDF for shape reconstruction and improves the accuracy of shape reconstruction by using the motion amount estimated by an additional, improved pose encoder during shape reconstruction. As the official AKBNet code for the encoder has not been released, we use the ground truth motion amount during evaluation instead. As it uses a category-level shape decoder, we use the same setting with A-SDF-GT-2 in the main paper that trains up to two most frequent part counts per category. Our approach still outperforms AKBNet-GT in the majority of metrics.

To avoid unfair comparison, we did not include another recent work, PPD [3], as our baseline, suggested by fellow reviewer **Y464**. This is because PPD focuses explicitly on shape **abstraction** but not accurate shape reconstruction with unsupervised learning. In contrast, our approach is fully-supervised, and targets shape reconstruction.



### Shape mAP evaluation
|               | Fscore@80% | Fscore@90% | CD1@5%   | CD1@1%   | IoU@25% | IoU@50% |
|---------------|------------|------------|---------|---------|--------|--------|
| AKBNet-GT   | 72.67      | 58.73      | **79.17** | 49.92   | **41.61** | 11.26  |
| Ours          | **74.77**   | **68.38**      | 77.39   | **56.53**   | 41.35  | **11.63** |



## Effectiveness of the KPF module (**mcMG**, **HLDh**):
We have detailed the effectiveness of the KPF module qualitatively through additional ablation studies in Appendix H. The KPF module improves detection and pose estimation, especially for small and distant parts, as visualized in Fig. 3 in the appendix while suppressing false positives, as shown in the precision score in Table 4 of the main paper. As explained in L159-161 in Section 3.6 of the main paper, the reason for this improvement is that the query oversampling allows more queries to cover better small parts represented as a small number of points in the input point cloud. 

We added qualitative visualization of the KPF module in Fig. 1 of the attached material. We observe that the KPF module also improves detection for occluded parts with a small number of points in the input point cloud due to the occlusion (indicated by the bottom red box in the figure). Furthermore, we have qualitatively demonstrated the effectiveness of kIoU in the KPF module in Fig. 4 of the appendix. Query oversampling (QO) and part fusion (PF) alone with standard 3D box IoU results in false positives for thin parts like doors due to the small overlap between parts, but kIoU effectively suppresses the false positives by considering their overlapping trajectories, as explained in L171 in Section 3.6.




## Larger number of the validation set than the training set (**mcMG**, **VtGS**):
We generated 188,726 images for training and validation purposes. Due to our limited computational resources and time budget for the experiments in parallel, we used only 20,000 images for training. One can include the remaining images in the validation set to the training set to make the sizes of the validation and training sets comparable, depending on available computational resources and time budget.


## Writing improvement: typos, grammatical errors, missing reference to the figure, unclear sentence (**VtGS**, **Y464**):
We appreciate the reviewers pointing out the areas to improve in the manuscript. We promise to fix those issues in the camera-ready version.

[1] Heppert et al. CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects, CVPR 2023.

[2] Liu et al. AKB-48: A Real-World Articulated Object Knowledge Base, CVPR 2022.

[3] Kawana et al. Unsupervised Pose-aware Part Decomposition for Man-made Articulated Objects, ECCV 2022.





Meta-review: In this paper, the authors propose a new method for part-level articulated object reconstruction from single RGBD images. After rebuttal, all reviewers find this paper making solid contribution to our field and recommend acceptance. The AC agrees with the assessment.
Decision: Accept (poster)