SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our model is evaluated on three various application scenarios with four datasets: 1) SAPIEN [68] for articulated objects, 2) OGC-DR and its single-view counterpart OGC-DRSV [61] for furniture arrangements; 3) KITTI-SF [48] for vehicular traffic. Following previous works [61, 32, 51], we evaluate the rigid segmentation performance on the following seven measures: Average Precision (AP), Panoptic Quality (PQ), F1-score (F1), Precision (Pre), and Recall (Rec) at an Intersection over Union threshold of 0.5, in addition to the mean Intersection over Union (mIoU) and Rand Index (RI). In terms of multi-body motion estimation, we report End-Point-Error 3D (EPE3D) in the main body, and other metrics (e.g., Outlier) are provided in Supp. More information about the datasets and our evaluation protocol can be found in Supp.
4.2 Pilot Studies on The Two Heads
We first conduct experiments on SAPIEN to verify the generalizability of our two-head structure, including the point-level invariance of the segmentation head and the part-level equivariance of the motion head.
Figure 4: Results of pilot studies. (a)(b) are the experimental results of our segmentation head, while (c) is the exploration outcomes of the motion head.
Can the point-level invariance of our segmentation head help generalize to open-set motion? On SAPIEN, each articulated target has four frames with various part-level rigid orientations and locations. By training the segmentation head on a subset of the  $1^{st}$  frames (with the canonical pose in [32]) and testing them on all frames, we compare its performance with the segmentation results using the same parameters of the non-equivariant counterpart KPCov [62] and global equivariant EPN. As shown in Figure 4(a)(b), the SE(3)-equivariance can essentially boost the generalization even given the global since there are some targets that only contain two rigid parts. By introducing point-level invariance, both precision and recall can be further improved.
Can the part-level equivariance of our motion head help generalize to category-agnostic parts? Note that the category of targets for training, validation, and testing on SAPIEN is completely disjoint. By optimizing the motion head on the training set and testing its performance on the validation data, we ensure that the targets do not overlap in categories so their parts are entirely category-agnostic. To exclude the interference of part segmentation, we directly provide the ground-truth rigid partition as the mask. By comparing the decrease in performance on the validation data when using part-level equivariance vs. not using it, as illustrated in Figure 4(c), it is observed that the mean angular error of our model only increases by less than  $2^{\circ}$ , while that without part-level equivariance surges from  $5.69^{\circ}$  to  $20.77^{\circ}$ . This demonstrates the strong generalizability of our model to category-agnostic parts.
Table 1: Ablation studies on SAPIEN.
Seg. Head	Scene Flow	Mot. Head	Metrics
SE(3)Feat.	Point-levelFlexibility	Current Past	Consensus	AP↑	PQ↑	F1↑	Pre↑	Rec↑	mIoU↑	RI↑
45.2	44.2	58.9	53.8	65.1	60.9	71.2
✓	51.7	50.0	65.8	64.7	67.0	61.6	72.3
✓	✓	55.3	52.8	68.3	65.9	70.0	62.3	72.7
✓	✓	✓	54.8	52.0	67.6	66.0	69.3	63.5	73.8
✓	✓	✓	✓	57.0	51.6	67.3	63.8	71.1	63.1	73.2
✓	✓	✓	✓	✓	63.8	61.3	77.3	84.2	71.3	63.7	75.4
We perform detailed ablation studies on SAPIEN to explore the effects of three main components: segmentation head, motion head, and scheme of scene flow updating.
Segmentation Head. Two crucial design elements significantly enhance our segmentation head: 1) the use of SE(3)-equivariant features, and 2) point-level flexibility for invariant representations. By simply introducing the global equivariant features, AP increases from  $45.2\%$  to  $51.7\%$ . This demonstrates the importance of SE(3)-equivariance in modeling rigid transformations. The proposed point-level flexibility further boosts the performance to  $55.3\%$ , suggesting that our point-level invariance is highly advantageous to the multi-body task.
Scene Flow Updating Scheme. For the updated choice of scene flow, we empirically test three options: 1) completely relying on the past flow extracted by an unsupervised flow network, 2) ignoring the past flow and directly using the current estimation, and 3) combining them together with a decay
Figure 5: Qualitative comparison between our unsupervised results and other methods (including supervised ones) on SAPIEN. This provides a glimpse into the qualitative performance of our approach, and more results can be found in Supp.
factor. It is observed that the extreme choices of 1) or 2) result in minor performance differences (AP:  $55.3\%$  vs.  $54.8\%$ ). The significant improvement comes from the weighted combination in option 3), achieving the AP of  $57.0\%$ .
Motion Head. The motion head contributes to the consensus score for the removal of scene flow noise. From the last row of Table 1, the consensus between the motion head and scene flow significantly boosts the final segmentation performance in most of the metrics. By controlling the point-level learning weight during the training process, our motion head can assist in optimizing the segmentation head.
Table 2: Rigid segmentation and motion estimation results on SAPIEN. * indicates that we evaluate these metrics upon the officially released model; - means that the metric is unavailable.
AP↑	PQ↑	F1↑	Pre↑	Rec↑	mIoU↑	RI↑	EPE3D↓
Supervised Methods	PointNet++ [55]	-	-	-	-	-	51.2	65.0	-
MeteorNet [46]	-	-	-	-	-	45.7	60.0	-
DeepPart [71]	-	-	-	-	-	53.0	67.0	5.95
MBS [32]	49.4*	52.6*	67.6*	61.4*	75.2*	67.3	77.0	5.03
OGC-sup [61]	66.1	48.7	62.0	54.6	71.7	66.8	77.1	-
Ours-sup	73.5	57.8	71.1	65.6	77.7	72.6	81.4	3.86
Unsupervised Methods	TrajAffn [52]	6.2	14.7	22.0	16.3	34.0	45.7	60.1	-
SSC [51]	9.5	20.4	28.2	20.9	43.5	50.6	65.9	-
WardLinkage [66]	17.4	26.8	40.1	36.9	43.9	49.4	62.2	-
DBSCAN [19]	6.3	13.4	20.4	13.9	37.9	34.2	51.4	-
NPP [28]	-	-	-	-	-	51.5	66.0	21.22
OGC [61]	55.6	50.6	65.1	65.0	65.2	60.9	73.4	-
Ours	63.8	61.3	77.3	84.2	71.3	63.7	75.4	5.47
4.4 Results & Comparisons
4.4.1 SAPIEN
SAPIEN is a challenging dataset of articulated objects since its training, validation and testing sets comprise completely disjoint categories of objects. This domain gap in the data split complicates the precise segmentation of rigid parts. Adapting our model to the supervised formulation is straightforward by optimizing the segmentation with ground-truth training labels. Therefore, we also report the supervised performance of the proposed framework. As shown in Table 2, our segmentation performance sets a new benchmark across all metrics in both supervised and unsupervised settings.
In line with the default setting of [32], we estimate motion between both consecutive frames and nonadjacent ones. Table 2 reports its EPE3D performance and comparison: our unsupervised performance of 5.47 EPE3D on motion estimation is on par with the best existing supervised method (MBS), while our supervised EPE3D achieves 3.86, marking a relative error reduction of approximately  $23\%$ .
As depicted in Figure 1, owing to the lightweight two-head structure, our model encompasses only a small number of parameters (0.25M) and incurs a low cost in computational complexity (0.92G floating point operations, FLOPs). We also show a sneak peek of the qualitative results (Figure 5). Our unsupervised method is comparable (and often times better) than other supervised methods. More results are shared in the Supp.
Table 3: Rigid segmentation results on OGC-DR and OGC-DRSV.
AP↑	PQ↑	F1↑	Pre↑	Rec↑	mIoU↑	RI↑
Supervised Methods	OGC-sup [61]	90.7 / 86.3	82.6 / 78.8	87.6 / 85.0	83.7 / 82.2	92.0 / 88.0	89.2 / 83.9	97.7 / 97.1
Ours-sup	92.8 / 89.3	86.9 / 82.6	91.0 / 87.9	88.8 / 85.5	93.2 / 90.4	91.2 / 86.6	98.7 / 97.9
Unsupervised Methods	TrajAffn [52]	42.6 / 39.3	46.7 / 43.8	57.8 / 54.8	69.6 / 63.0	49.4 / 48.4	46.8 / 45.9	80.1 / 77.7
SSC [51]	74.5 / 70.3	79.2 / 75.4	84.2 / 81.5	92.5 / 89.6	77.3 / 74.7	74.6 / 70.8	91.5 / 91.3
WardLinkage [66]	72.3 / 69.8	74.0 / 71.6	82.5 / 80.5	93.9 / 91.8	73.6 / 71.7	69.9 / 67.2	94.3 / 93.3
DBSCAN [19]	73.9 / 71.9	76.0 / 76.3	81.6 / 81.8	85.8 / 79.1	77.8 / 84.8	74.7 / 80.1	91.5 / 93.5
OGC [61]	92.3 / 86.8	85.1 / 77.0	89.4 / 83.9	85.6 / 77.7	93.6 / 91.2	90.8 / 84.8	97.8 / 95.4
Ours	93.9 / 88.1	87.0 / 80.0	91.1 / 86.1	87.0 / 80.8	95.6 / 92.2	92.4 / 86.7	98.1 / 96.6
4.4.2 OGC-DR & Single-view Counterpart
OGC-DR is a dataset for indoor furniture arrangements, where the training, validation, and testing instances are distinct from one another. OGC-DRSV, the single-view version of OGC-DR, presents a challenge due to the incomplete nature of the furniture caused by occlusion, making it difficult to identify consistent rigidity. As demonstrated in Table 3, the proposed framework constantly surpasses the state-of-the-art models across all measures. Remarkably, even under the most challenging condition, i.e., unsupervised single-view, our AP still achieves the value of  $88.1\%$ . This underscores the robustness of our model in handling incomplete observations. The performance on motion estimation can be found in Supp.
Table 4: Rigid segmentation results on KITTI-SF. Our model still achieves competitive results even though the data setting is inconsistent with the model's assumption.
Method Category	Method	AP↑	PQ↑	F1↑	Pre↑	Rec↑	mIoU↑	RI↑
Supervised Methods	OGC-sup [61]	62.4	52.7	65.1	63.4	67.0	67.3	95.0
Ours-sup	65.1	56.3	68.6	69.4	67.8	69.5	95.7
Unsupervised Methods	TrajAffn [52]	24.0	30.2	43.2	37.6	50.8	48.1	58.5
SSC [51]	12.5	20.4	28.4	22.8	37.6	41.5	48.9
WardLinkage [66]	25.0	16.3	22.9	13.7	69.8	60.5	44.9
DBSCAN [19]	13.4	22.8	32.6	26.7	42.0	42.6	55.3
OGC [61]	54.4	42.4	52.4	47.3	58.8	63.7	93.6
Ours	53.6	44.4	55.1	56.3	54.0	61.5	93.4
4.4.3 KITTI-SF
Strictly speaking, KITTI-SF is not a multi-body rigid dataset, as the background in its point clouds may be deformable. This is inconsistent with the assumption of our framework, which posits that the feature is only equivariant for rigid transformations. However, as shown in Table 4, our framework still delivers competitive performance in rigid segmentation despite this inconsistency.
This paper introduces a part-level SE(3)-equivariant framework for modeling multi-body rigid motion. The two heads for rigid segmentation and motion estimation are meticulously designed based on their inherent invariant and equivariant characteristics. The relationship among scene flow, rigid segmentation, and multi-body transformation is then exploited to derive an unsupervised optimization strategy. Our approach achieves state-of-the-art results on multiple datasets while significantly reducing the required parameters and computations.
Limitations & Future Work. Our model is predicated on an assumption: the part-level motion should be a rigid transformation. Therefore, as shown in Table 3, if the observation of part-level movements is non-rigid (such as occluded single views), it would suffer a performance decrease. In the future, we would like to develop a model that is more robust to its observation.
[64] K. Wang and S. Shen, "Estimation and propagation: Scene flow prediction on occluded point clouds," IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 12 201-12 208, 2022.
[65] L. Wang, H. Liu, Y. Liu, J. Kurtin, and S. Ji, "Learning hierarchical protein representations via complete 3d graph networks," in The Eleventh International Conference on Learning Representations, 2022.
[66] J. H. Ward, "Hierarchical grouping to optimize an objective function," Journal of the American Statistical Association, vol. 58, no. 301, pp. 236-244, 1963.
[67] M. Weiler, F. A. Hamprecht, and M. Storath, "Learning steerable filters for rotation equivariant cnns," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 849-858.
[68] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al., "Sapien: A simulated part-based interactive environment," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11097-11 107.
[69] X. Xu, L.-F. Cheong, and Z. Li, "3d rigid motion segmentation with mixed and unknown number of models," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 1, pp. 1-16, 2019.
[70] B. Yang, J. Wang, R. Clark, Q. Hu, S. Wang, A. Markham, and N. Trigoni, “Learning object bounding boxes for 3d instance segmentation on point clouds,” Advances in neural information processing systems, vol. 32, 2019.
[71] L. Yi, H. Huang, D. Liu, E. Kalogerakis, H. Su, and L. Guibas, "Deep part induction from articulated object pairs," ACM Transactions on Graphics (TOG), vol. 37, no. 6, pp. 1-15, 2018.
[72] T. Yin, X. Zhou, and P. Krahenbuhl, "Center-based 3d object detection and tracking," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 11784-11793.
[73] H.-X. Yu, J. Wu, and L. Yi, "Rotationally equivariant 3d object detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1456-1464.
[74] J.-X. Zhong, K. Zhou, Q. Hu, B. Wang, N. Trigoni, and A. Markham, “No pain, big gain: classify dynamic point cloud sequences with static models by fitting feature-level space-time surfaces,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8510–8520.