SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our encoder and decoder architectures strictly follows the ones proposed in 3DETR [37]. We employ the masked-encoder from [37] as the encoder architecture for improved performance. We also employ the Fourier positional encoding, as proposed in [37], for the query embedding prior to input to the decoder. Please refer to [37] for comprehensive details and visualizations of the architectures.
Algorithm 1 Kinematic-aware part fusion (KPF)
Input: Subsampled point cloud  $P^{\prime}$  , scene feature  $F^{\prime}$    
Output: Set of detected parts  $\mathcal{X} = \{X_i\}$    
1:  $\mathcal{X}\gets \emptyset$    
2: for  $N_{Q}$  times do   
3:  $\{\mathbf{q}_n\} \leftarrow \mathrm{FPS}(P')$    
4:  $\mathcal{X}'\gets \mathrm{model}(\{\mathbf{q}_n\} ,F')$    
5:  $\mathcal{X}'\gets \mathrm{NMS}(\mathcal{X}',\tau_{\mathrm{IoU}})$    
6:  $\mathcal{X}'\gets \{X_n\mid n\in [\mathcal{X}'],\mathbf{y}_{\mathrm{fg},n} > \tau_{\mathrm{obj}}\}$    
7:  $\mathcal{X}'\gets \mathrm{NMS - kIoU}(\mathcal{X}',\tau_{\mathrm{IoU}})$    
8:  $\mathcal{X}\gets \mathcal{X}' + \mathcal{X}$    
9: end for   
10: count  $\leftarrow 0$    
11: repeat   
12:  $\mathcal{X}_{\mathrm{old}}\gets \mathcal{X}$    
13:  $\mathcal{X}\gets \mathrm{PF - kIoU}(\mathcal{X}_{\mathrm{old}})$    
14: count  $\leftarrow$  count + 1   
15: until  $\mathcal{X} = \mathcal{X}_{\mathrm{old}}$  or count  $= \tau_{\mathrm{count}}$    
16:  $\mathcal{X}\gets \{X_n\mid n\in [\mathcal{X}],\mathbf{y}_{\mathrm{fg},n} > \tau_{\mathrm{obj\_final}}\}$    
17: return X
Algorithm 2 Part Fusion with kIoU (PF-kIoU)
Input: Set of part proposals  $\mathcal{X}$    
Output: List of updated part proposals  $\mathcal{X}'$    
1:  $C\gets$  Empty list,  $\mathcal{X}'\gets$  Empty list   
2: Sort  $\mathcal{X}$  in descending order by objectness   
3: for  $\forall X\in \mathcal{X}$  do   
4: match  $\leftarrow$  False   
5: for  $\forall X^{\prime}\in \mathcal{X}^{\prime}$  do   
6: if kIoU(X,X') >  $\tau_{\mathrm{kIoU}}$  then   
7: matched  $\leftarrow$  True   
8:  $n^\prime \leftarrow$  index of  $X^{\prime}$  in  $\mathcal{X}'$    
9: Append  $X$  to  $C[n']$    
10:  $\mathcal{X}'[n']\gets$  weighted-average(C[n'])   
11: Break   
12: end if   
13: end for   
14: if not match then   
15: Append  $X$  to  $\mathcal{X}'$  and  $C$    
16: end if   
17: end for   
18: for  $\forall n'\in [\mathcal{X}]$  do   
19:  $\mathbf{y}_{\mathrm{fg},n'}'\gets \mathbf{y}_{\mathrm{fg},n'}\frac{|C[n']|}{T}$    
20: end for   
21:  $\mathcal{X}' = \{X_{n'}'|n'\in [\mathcal{X}],\mathbf{y}_{\mathrm{fg},n'}' > \tau_{\mathrm{scaled}}\}$    
22: return  $\mathcal{X}'$
C.2 Part prediction MLPs
We adapt the box prediction MLPs outlined in [37] as our part prediction MLPs. Our part prediction MLPs retain the same architecture and are referred to as 'MLPs' in Fig. 3 of the decoder architecture. We employ 13 MLPs, each dedicated to a specific part proposal parameter. These parameters include the center offset from the query position  $\Delta c$ , rotation  $\mathbf{R}_{6D}$ , size s, current pose joint state for prismatic and revolute types  $d_{\mathrm{current}}^{\mathrm{pris}}$  and  $d_{\mathrm{current}}^{\mathrm{rev}}$ , fully-opened pose joint state for prismatic and revolute types  $d_{\mathrm{max}}^{\mathrm{pris}}$  and  $d_{\mathrm{max}}^{\mathrm{rev}}$ , joint axis a, revolute origin offset from the query position  $\Delta v$ , joint type y, category u, shape feature h, and part-to-instance embedding z. We utilize the 6D rotation representation [62] as the MLP's output representation  $\mathbf{R}_{6D}$ , subsequently converted to a  $3\times 3$  rotation matrix.
C.3 Refiner
Our refiner architecture closely follows the decoder architecture, with MLPs outputting residual parameters. It follows the same architecture of the part prediction MLPs from the decoder, except for the dropout layers. We employ nine MLPs for residual predictions of center, rotation, size, prismatic and revolute types' current pose joint state, prismatic and revolute types' fully-opened pose joint state, joint axis, and revolute origin. For the rotation matrix and joint axis, the MLPs output 6D rotation representation which are then converted into  $3 \times 3$  rotation matrices. These are updated by matrix product. The remaining parameters are updated through the addition of residuals. The same positional encoding is applied to the part center c to generate an embedding feature, which is then added to the corresponding refined query embedding  $\mathbf{e}_{N_{\mathcal{D}}}$  from the decoder's final layer. This forms the new query feature input for the refiner, as depicted in Fig. 3.
C.4 Shape decoder
Our shape decoder is based on the ConvONet architecture [42]. It consists of a local point encoder, a volume encoder, and an implicit shape decoder. The local point encoder quantizes input points to multiple 2D (tri-planes) or 3D grids and encodes points within each grid. The volume encoder then takes these locally encoded points and further encodes them into a 3D voxel feature. To decode the occupancy value at a 3D point  $\mathbf{x}$ , we sample the local feature  $\mathbf{h}_{\mathrm{local},\mathbf{x}}$  by interpolation on the 3D voxel feature at  $\mathbf{x}$ . The implicit shape decoder outputs the occupancy value  $o_{\mathbf{x}}$  at  $\mathbf{x}$ , given  $\mathbf{h}_{\mathrm{local},\mathbf{x}}$  as input.
Our approach differs from the original ConvONet in that we concatenate the shape feature per part proposal  $\mathbf{h}$  to the local feature to form  $\mathbf{h}_{\mathrm{local},\mathbf{x}}^{\prime} = \mathbf{h}_{\mathrm{local},\mathbf{x}} \oplus \mathbf{h}$ . This serves as the learned part shape prior for the implicit shape decoder that reduces shape ambiguity due to the unsegmented per-part point cloud  $P_{\mathcal{O}}$ , as outlined in Eq. 1. We utilize the lightweight tri-plane architecture proposed in [42] for the volume encoder architecture. For more detailed information on the ConvONet architecture, please refer to [42].
C.5 Foreground segmentation model
Our segmentation model incorporates a ResNext50 [59] architecture for the encoder with RGBD input and DeepLabV3Plus [6] for the segmentation head. We use the off-the-shelf implementation from [18].
D.1 Architecture
Our approach strictly follows the hyperparameters of the masked-encoder described in [37]. We employ three self-attention transformer layers with self-attention masks between points in the point cloud, and each point only attends to others within a specified radius. The input point cloud to the encoder consists of  $N_P = 32768$  points, which are subsequently subsampled to  $N_{P^{\prime}} = 2048$  points in the encoder. We set the scene feature dimension  $D_{\mathbf{F}}$  a value of 256, following [37].
For the decoder, we maintain the hyperparameters from [37], except for the number of decoder layers  $N_{\mathcal{D}}$ . The decoder layers for the decoder  $\mathcal{D}$  and the refiner  $\mathcal{R}$  are set to  $N_{\mathcal{D}} = 6$  and  $N_{\mathcal{R}} = 2$ , respectively. We match the query embedding dimension with the scene feature dimension for addition
and set the number of queries to  $N_{q} = 128$  during training. For testing,  $N_{q}$  is set to 512, unless stated otherwise, with independent runs for query oversampling (QO) for kinematics-aware part fusion (KPF) with  $N_{Q} = 10$  independent inference runs.
As for the part prediction Multi-Layer Perceptrons (MLPs), we follow the box prediction MLP hyperparameters in [37], except for the number and output dimension of each MLP. The MLPs in the refiner for residual prediction strictly follow the part prediction MLPs hyperparameters in the decoder but without dropout. We set the dimension for shape feature  $\mathbf{h} = 128$ , and the dimension for part-to-instance association embedding  $\mathbf{z} = 32$ .
With respect to the ConvONet [42] architecture in the shape decoder, the local point encoder consists of five MLPs of width 128, and the volume encoder has three MLPs with widths of 16, 32, and 64. We employ the tri-plane version of the volume encoder for volume representation, using only the xy and yz planes to save GPU memory. In the implicit shape decoder, we employ four fully-connected layers with a hidden dimension of 128 and a leaky ReLU activation.
D.2 Training details
We set the weights for the matching cost  $\mathcal{C}_{\mathrm{match}}$  at  $\lambda_1 = 8, \lambda_2 = 10, \lambda_3 = 1, \lambda_4 = 5$ . Each loss term in the total loss  $\mathcal{L}_{\mathrm{total}}$  carries equal weight. We use the AdamW [31] optimizer with a base learning rate of 9e-4 with a cosine scheduler down to a learning rate of 1e-6. The warm-up period is set to nine epochs, and mixed precision is used during training.
All models were trained on two A100 GPUs, each with 40GB GPU memory. We set the batch size to 26 per GPU, and it took approximately two days to complete 500 epochs. Weight decay was set at 0.1, with gradient clipping with an L2 norm of 0.1, as per [37].
We implemented on-the-fly sampling of 3D points and corresponding occupancy values to train the shape decoder. Half of these points are sampled uniformly from the space  $[-0.5, 0.5]^3$ . Additionally, a quarter of the points are sampled around the surface with a Gaussian-distributed random offset along the surface normal direction, with a standard deviation of 0.1. The remaining quarter is sampled with a standard deviation of 0.01. Each part has 128 points sampled for occupancy values.
In training the foreground segmentation model, we used the AdamW optimizer and a cosine scheduler, identical to the approach used in training the previous models. This model was also trained on two A100 GPUs with 40GB GPU memory and a batch size of 26 per GPU. Training took approximately one day for 500 epochs.
We add noise to the depth map during training, following the approach in [32]. We incorporate a depth map filter for the model tested on real-world data to mitigate the flying pixel effect on the object edge as suggested by [55]. This filter is also applied during testing for real-world data. Following the projection of the depth map to 3D points, we introduced random scaling within a range of  $\pm 15\%$ , and random rotation along the surface normal direction within  $\pm 30^{\circ}$ . The point cloud was zero-centered for both training and testing. Color augmentation during training follows the original code of [37]. For the model using a foreground mask, we augment the mask by randomly displacing the foreground pixels around the border and performing successive dilation and erosion to simulate noise on inferred masks during training.
D.3 Mesh generation
We sample occupancy values on  $64^{3}$  voxel grids per part, then extract the surface mesh using the marching cubes method [30] at an isosurface level  $\tau_{\mathcal{O}} = 0.3775 = \mathrm{Sigmoid}(-0.5)$ . For surface mesh reconstruction evaluation, we store an instance mesh as a union of part meshes and apply quadratic decimation [12] to reduce the number of faces to 10000. For volumetric IoU evaluation, where we evaluate the union of IoU for each part, we stored a part mesh and applied quadratic decimation to reduce the number of faces to 5000.
E Dataset Details
E.1 SAPIEN [58] dataset
We choose ten categories offering prismatic and revolute joints as representative articulation types, adequate CAD models, and various part structures. The CAD models are randomly divided into training + validation splits (referred to as the trainval split) and test split, maintaining an approximate 8:2 ratio. No overlapping CAD model IDs exist between these splits.
All CAD models are manually aligned such that they stand upward along the y-axis and face the z-axis. We use BlenderProc [10] to generate synthetic data. We randomly create room geometry (wall and floor) in each scene and place one to four instances.
For the trainval split, we randomize the texture of the part mesh and room geometries. The textures for the part mesh are randomly applied from https://www.blenderkit.com, using the downloading script of BlenderProc [10]. We employ 2923 textures for this split. However, for the test split, we use the original texture from the SAPIEN [58] dataset.
The room geometry textures are downloaded from https://ambientcg.com using the BlenderProc script [10] and randomly applied. We use 353 textures for the trainval split and 85 textures for the test split, ensuring no overlap.
For environmental lighting, HDR maps downloaded from https://polyhaven.com are used. We employ 113 HDR maps for the trainval split and 29 for the test split, with no overlap.
Instances are created by applying random horizontal flips, random anisotropic resizing of side lengths, and random articulation of CAD models. The maximum joint state of the revolute joint defaults to  $135^{\circ}$  unless it touches the ground. We set the maximum joint state value to avoid ground contact if it does. We use the CAD model's maximum prismatic joint state value for prismatic joints, adjusted for rescaling.
This paper considers instances comprising one base part and at least one articulated part attached to it. Therefore, we fix articulated parts attached to another articulated part and ignore their joints, such as small prismatic buttons and rotating knobs on doors.
The scene is populated with instances by first sampling one to four categories uniformly, with replacement. Then, a CAD model is randomly sampled for each selected category, and its part pose, size, and yaw rotation are randomly augmented. The size distribution is shown in Fig. 13. The yaw rotation is limited to  $\pm 90^{\circ}$ , ensuring instances face roughly forward. Instances are placed randomly in the scene, maintaining a minimum and maximum distance of  $0.1\mathrm{m}$  and  $0.8\mathrm{m}$  from other instances, respectively. This setup prevents instances from being scattered in the room.
We allow up to 10 sampling of camera poses in the same scene, limiting the camera position to the union of convex hulls of instances projected on the x-z plane (same as the floor) and a circular sector whose center is at the scene center and the arc is an angle between  $\pm 80^{\circ}$ . This restriction, coupled with the yaw range limitation, discourages the self-occlusion of parts by an instance facing backward to cameras. We illustrate this idea in Fig. 13.
We use the BlenderProc API to sample camera poses against instances with a minimum distance to a camera from an instance of  $0.5\mathrm{m}$ . In total, we render 4,000 frames for the test split and 188,726 for the trainval split, randomly sampling 20,000 frames for training and allocating the rest for validation usage.
We exclude instances whose 2D silhouette is occluded more than  $50\%$  or truncated by  $25\%$ . Each frame is ensured to have at least one visible instance. The image size is  $360 \times 640$  (height and width), and the field of view is set to  $86.64^{\circ}$ .
For the generation of watertight mesh for the implicit shape ground truth in shape learning, we follow the instructions from [35] to generate part-wise watertight meshes. We then apply quadratic decimation [12] to the generated part mesh, reducing the face count to approximately 10,000 when combined as an instance.
Dishwasher	Trashcan	Safe	Oven	Storage	Table	Microwave	Fringe	Washing	Box
Trainval	33	47	24	14	266	60	10	33	13	20
Test	9	11	6	4	68	16	3	9	4	6
Table 7: Number of CAD models from SAPIEN [58] dataset used in trainval and test splits for our synthetic dataset.
Figure 12: Size distribution per category after size augmentation applied to CAD models.
E.2 BMVC [36] dataset
We use the cabinet class data from this dataset, which includes one prismatic part and one revolute part. The original data comprises two sequences of RGBD frames, with different part poses for each sequence and varied camera poses. The first sequence contains 1108 frames, while the second has 1119 frames. The image size is  $480 \times 640$  (height and width).
Before feeding the data to the model, we apply a depth map filter to suppress the flying pixel effect on the object edge, as detailed in [55]. We extract the foreground point cloud using the silhouette of the projected CAD model on the image.
E.3 Images taken by iPhone X
We record the partial front view of the scene as a short clip, extracting approximately 300 frames to train the neural rendering model Nerfacto [50] for depth map generation. Default training hyperparameters are used for Nerfacto training. The image size is  $540 \times 960$  (height and width). For our evaluation, foreground masks are automatically extracted using https://www.remove.bgb unless specified otherwise. Note that we only use the point cloud lifted from a single RGBD image as input, not recovered from all frames.
F Details on baselines
F.1 A-SDF [38]
Data preparation We follow the description in the original A-SDF paper [38] for data preparation. After normalizing the global pose and scale, we ensure the base part's position does not change regardless of random articulation during training. In addition, for categories with multiple parts, we maintain consistent part ordering. We automate this by assigning part positions in canonical poses to a cell in the  $5 \times 5 \times 5$  voxel, which discretizes the positions. The 1D flattened cell positions of the voxel determine the part order. To limit the training time, we randomly sample a maximum of 1500 instances per category for the training split.
Figure 13: Illustration of yaw rotation range of an instance and camera position sampling area.
Model The model implementation uses publicly shared author code. The original A-SDF only targets revolute parts. We normalize the prismatic joint state by the maximum joint state to fall within the  $[0,1]$  range. We then multiply this range by 135, ensuring joint states for revolute and prismatic parts are within the same range to extend the model for prismatic parts.
A-SDF is a category-level approach that works with a fixed number of parts, which becomes a problem when evaluated on dataset containing instances with varied part counts in a category. Similarly to our approach, OPD [22] can handle multiple part counts without assuming the predefined part number. They evaluate their method against baselines targeting a fixed number of parts by training the baselines for a category's most common part count.
We follow this protocol but aim to favor the baseline A-SDF more by training up to two models per category. The first model is for the most common part count, and the second, if applicable, is for the next most common part count. The second model is used when a category has more than two different part counts in the test split.
This setting differs from OPD's method, which only uses the most common part count. During testing, we use the model with the most frequent part count for instances with untrained part counts. We call the baseline model trained for the most frequent part count A-SDF-GT and the one using the next most frequent part count A-SDF-GT-2. The baselines and per category part counts, excluding the base part, are summarized in Table 8.
Training We employ the publicly shared author code for training. During training, each part's articulation varies randomly from  $0^{\circ}$  to  $135^{\circ}$ , sampled at  $5^{\circ}$  intervals as suggested in [38]. We dynamically sample 3D points and their corresponding signed distance values from the randomly articulated watertight meshes of parts during training. The number of sample points and the ratio of uniformly sampled points to points near the mesh surface follow the original A-SDF paper [38]. A-SDF model training uses part label supervision and takes approximately ten days on a single V100 or A100 GPU per model.
Mesh generation For mesh generation, we use the publicly shared author code as well. A-SDF assumes no background and normalized global pose and size for input. During testing, we use the ground truth instance segmentation mask for each instance in a scene to isolate instance-wise foreground points. We also normalize the depth map to the input space using the ground truth pose and size. The surface mesh is extracted using the provided implementation, which samples signed distance values on  $64^3$  voxel grids and extracts surface mesh using marching cubes [30]. Additionally,
Dishwasher	Trashcan	Safe	Oven	Storage	Table	Microwave	Fringe	Washing	Box
Most freq.	1	1	1	2	1	1	1	2	1	1
Next Most freq.	n/a	2	n/a	n/a	2	n/a	2	1	n/a	4
we apply the quadratic decimation [12] to limit the number of faces to 10000. We then project the generated mesh back to the original scale and pose in the scene using the ground truth values.
F.2 OPD [22]
Data Following the original paper [22] and the publicly shared author code and dataset, we generate ground truth data. The original data includes a semantic label of part type, one of drawer, lid, or door. However, our synthetic dataset does not have corresponding labels. Thus we replace the semantic label with the joint type. Consistent with the original paper, we exclude the fixed type part from the ground truth, targeting only the revolute and prismatic parts for detection.
Model We employ the OPDRCNN-C model from [22], which predicts kinematic parameters in camera coordinates. For input modality alignment, the model uses RGBD input. We have adapted the model to produce a continuous 1D joint state; instead of supervising the joint state prediction head with binary joint state supervision, we use continuous 1D joint state supervision. We duplicate the MLP head to separately predict joint states for revolute and prismatic joints. We supervise only the output corresponding to the ground truth joint type. Furthermore, we guide the joint axis prediction with reference to the floor's normal direction, as we experimentally found this improves detection performance.
Training We follows the author's implementation for training code, maintaining the same settings and hyperparameters for the OPDRCNN-C model.
Testing As described in Sec. 4, we match the ground truth using part segmentation evaluation with an IoU threshold of  $50\%$  following the author's implementation. We then evaluate the predicted joint parameters against the matched ground truth.
G Category-wise shape mAP
We show category-wise shape mAP results in Table 9, 10, 11, 13, 14, 15, 16, 17, 18.
H Additional results on the refiner  $\mathcal{R}$
We evaluate the effectiveness of the refiner  $\mathcal{R}$  in enhancing performance while maintaining a comparable model size. Table 19 shows the quantitative results on the shape mAP and the number of learnable parameters of the Ours-BG model, with the number of decoder layers in the decoder  $N_{\mathcal{D}} = 6$  and in the refiner  $N_{\mathcal{R}} = 2$ . We compare this model against the model without a refiner and having the same total number of decoder layers  $N_{\mathcal{D}} = 8$ , and another model with a doubled number of decoder layers  $N_{\mathcal{D}} = 16$  without the refiner. The result shows that using the refiner achieves comparable performance in shape mAP with the model with a doubled number of decoder layers  $N_{\mathcal{D}} = 16$  with a smaller model size.
Table 8: The most frequent and the next most frequent part count for each category that baseline A-SDF models are trained on.
Fscore@80%	Fscore@90%	CD1@5%	CD1@1%	IoU@25%	IoU@50%
A-SDF-GT [38]	59.71	43.14	62.79	29.64	29.64	0.00
A-SDF-GT-2 [38]	58.40	41.49	61.36	26.23	26.89	0.22
Ours-BG	85.73	83.93	85.94	80.72	41.75	0.15
Ours	88.28	84.67	91.14	76.00	42.01	0.33
Table 9: Shape mAP result of dishwasher category.
Fscore@80%	Fscore@90%	CD1@5%	CD1@1%	IoU@25%	IoU@50%
A-SDF-GT [38]	78.76	61.84	82.73	55.08	12.95	0.00
A-SDF-GT-2 [38]	81.45	70.83	84.13	62.78	21.24	0.00
Ours-BG	76.41	72.21	78.67	71.15	16.78	0.01
Ours	78.23	74.63	81.03	72.32	19.08	0.00
Table 10: Shape mAP result of trashcan category.
Fscore@80%	Fscore@90%	CD1@5%	CD1@1%	IoU@25%	IoU@50%
A-SDF-GT [38]	69.19	48.34	73.42	40.21	37.00	5.15
A-SDF-GT-2 [38]	69.53	49.83	73.31	41.24	38.14	5.61
Ours-BG	59.25	55.52	59.45	44.15	46.43	3.27
Ours	56.69	53.21	57.47	43.40	46.03	6.28