Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation
Authors: Jia-Xing Zhong
Ta-Ying Cheng
Yuhang He
Kai Lu
Kaichen Zhou
Andrew Markham
Niki Trigoni
Venue: NeurIPS 2023 poster (2023)
Keywords: Dynamic Point Cloud Analytics
Multi-body Motion
TL;DR: Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation

Abstract: A truly generalizable approach to rigid segmentation and motion estimation is fundamental to 3D understanding of articulated objects and moving scenes. In view of the closely intertwined relationship between segmentation and motion estimates, we present an SE(3) equivariant architecture and a training strategy to tackle this task in an unsupervised manner. Our architecture is composed of two interconnected, lightweight heads. These heads predict segmentation masks using point-level invariant features and estimate motion from SE(3) equivariant features, all without the need for category information. Our training strategy is unified and can be implemented online, which jointly optimizes the predicted segmentation and motion by leveraging the interrelationships among scene flow, segmentation mask, and rigid transformations. We conduct experiments on four datasets to demonstrate the superiority of our method. The results show that our method excels in both model performance and computational efficiency, with only 0.25M parameters and 0.92G FLOPs. To the best of our knowledge, this is the first work designed for category-agnostic part-level SE(3) equivariance in dynamic point clouds.

Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation
Jia-Xing Zhong, Ta-Ying Cheng, Yuhang He, Kai Lu, Kaichen Zhou Andrew Markham, Niki Trigoni
Department of Computer Science, University of Oxford
{jiaxing- zzong, ta-ying. cheng, yuhang. he, kai lu, rui mri.zhou)@cs..oo ox. ax.ac.uk {andrew. handrew.markham, niki.triae trigonii@cs...... ww.pox.ac....................... ac.uk
Abstract
A truly generalizable approach to rigid segmentation and motion estimation is fundamental to 3D understanding of articulated objects and moving scenes. In view of the closely intertwined relationship between segmentation and motion estimates, we present SE(3) equivariant architecture and training strategy tackle this task in an unsupervised manner. Our architecture is composed of two interconnected, lightweight heads. These heads predict segmentation masks using point-level invariant features and estimate motion from SE(3) equivariant features, all without the need for category information. Our training strategy unified and can be implemented online, which jointly optimizes the predicted segmentation and motion by leveraging the interrelationships among scene flow, segmentation mask, and rigid transformations. We conduct experiments on four datasets to demonstrate the superiority of our method. The results show that our method excels in both model performance and computational efficiency, with only 0.25M parameters and 0.92G FL.OPs. To the best our knowledge, this is the first work designed for category-agnostic part-level SE(3) equivariance in dynamic point clouds.
Introduction
Comprehending point cloud motion is critical for vari- ous 3D vision tasks in dynamic scenarios, e.g., tracking, animation, simulation, and manipulation. Many types of 3D motion can be described as composition of multi-body rigid movements, such as articulated ob-
Ours-sup
70-
OCCCspp
#PARAMS:
jects 68), and vehicular traffic scenes (25). Specifically, the setting of multi-body rigid motion requires all mov- ing parts (i.e., bodies) to undergo only translation and
60-
Ours-unsup
OCC-unsu
rotation, without any type of deformation. The process of modeling these multi-body rigid movements typically involves two primary portions 32 1) the identifica- tion distinct moving bodies (i.e., rigid segmentarion), and 2) the calculation of individual movements for each identified body (i.e., motion estimation).
1.0
4,0
600.0
FLOPs
Figure 1: Comparison of FLOPs, parame- ter number, and AP on SAPIEN, Supervised methods are marked in O.
Being the first modeling portion, rigid segmentation significantly differs from traditional semantic segmentation tasks that concentrate on category infor- semantics, rigid segmentation becomes category-agnostic about moving parts as rigid bodies are not limited to a specific set of shapes and naturally cannot be assigned certain category labels. As regards motion
mation [16] 3 30 31 70). Based multi-body motion instead of nategory-dependee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

estimation, individual pose variations would occur arbitrarily, including transformations unseen in the training data. This constitutes an open set of pose changes.
Considering the relative inaccessibility and the constrained generalizability of manual annotations, Song and Yang [61] have proposed seminal work for unsupervised rigid segmentation. Nevertheless, the unsupervised paradigm remains a formidable challenge because of the generalizable requirement in multi-body movements and the interdependent nature between rigid segmentation and motion estimation. Due to the aforementioned category- rigid prior and open-set pose changes in multi-body motion, a model is required to generalize well to transformations of appearance, location, and orientation. Moreover, rigid segmentation and multi-body motion estimation are highly coupled the calculation of movements for each body is based on the output of a rigid partition, while an estimate of multi-body motion facilitates further segmentation. In absence of precise supervision, it difficult to independently obtain effective training signals for either rigid segmentation or motion estimation.
To tackle the difficulties of generalizability and interdependence, this paper presents a three-fold contribution in the form of architecture, training strategy, and experimental evaluation:
1. Architecture. We design part-level alization to open-set motion and cetegory-agnossiee moving bodies. The SE(3) equiiiaraie allows features of model to keep coherence with the rigid transformations of point cloud inputs, thereby enhancing the model's 's robustness to such spatial transformations (i.e., rigid motion) and therefore is more generalizable for open-set pose changes of global rigid targets Tailing the SF(3)--quivariince is two lightweight heads for segmentation and motion estimation. The former, unlike previous segmentation networks using global SE(3) equivariance, exhibits point- level local flexibility to SE(3)-invariance. The latter, robustly employing the probability of part consistency, utilizes part-level "softly" matching operations to handle SE(3)--quivariance without requiring knowledge of part categories. As a result of integrating two lightweight heads into a unified network, rather than utilizing large independent networks, our model is characterized by small number of parameters (0.2) M) and low computational complexity (0.92G FLOPs). To the best of our knowledge, this is the first work designed for antegory-agnostee part-level SE(3) -quiivaraae in dynamic point clouds.
network that demonstrates strong gener-
[3]
14 67
63
2. Training Strategy. We leverage the interdependence between rigid segmentation and motion estimation and present a unified training strategy to jointly optimize their outputs in an online fashion. Based our proposed two-head network, simultaneously filter out the noisy flow predictions and refine the estimates of rigid motion by exploiting the interrelation among scene flow, segmentation mask, and rigid transformation. Superior previous arts, our online training process free from complicated components and manual intervention, e.g., the alternation of Markov Chain Monte Carlo proposals and Gibbs sampling updates [28], the offline repetition of training multiple segmentation networks from scratch [61], or the continual optimization of an accessory neural network of flow estimation 32).
3. Experiments. Experiments on four datasets demonstrate the efficacy of our model in performance, as well as its efficiency in parameters and computational complexity. We conduct comprehensive experiments on four datasets (SAPIEN [68] OGC-DR [6], OGC-DRSV [6]. and KITTI-SF [48]) across three application scenarios (articulated objects, furniture arrangements, and vehieular traf- fic). Noticeably, as shown in Figure D our performance on the SAPIEN dataset of articulated objects surpasses state-of-the-an results w.r.t. all evaluation metrics, achieving the relative gain of at least 14.7% AP in rigid segmentation and at least 23.3% in motion estimation, with only 0.25M parameters and 0.92G FLOPs. The code is available at https:/gtitubb.om//xzorrrr zhhoo wwwrsooooooooooooooddrrreoeee for-acddemic purposc/Multiody. SE3
Related Works
Deep Learning on Dynamic Point Clouds. Deep neural networks for point cloud video mod- eling aim to understand our dynamic 3D surroundings. These networks enable the resolution of several downstream tasks, such as video-level classification
20
22
44
23
21
74 29 frame-level
prediction [57 36 50], object-level detection 6 40
54
58 or tracking 26
56 72 part-level
mobility parsing 600 59 (1], point-level segmentation
22
23
20), and scene flow estima-
tion [8 64
In contrast to the above dynamic tasks that usually assume contiguous
2

---

sequential input, multi-body rigid motion may be captured between discrete frames unique challenge.
32 presenting a
3D Motion Segmentation & Multi-body Rigid Motion. Although the systematic formulation of multi-body rigid motion modeling is a relatively recent development [32], a related problem. Le., motion segmentation, has been long sought after. Motion segmentation aims to group points with similar motion patterns from the input of scene flow, using such techniques as factorization
[2 69]. clustering (33, graph optimization 47 35 4 and deep learning (39 28 7 However, motion segmentation does not explicitly take into account multi-frame multi-body consistency. To address this issue, Huang al. (32 present the seminal fully-supervised work for modeling multi- body rigid motion. Song and Yang [6]] further attempt to predict the mask of rigid segmentation via three object geometry losses in unsupervised setting. Following the same unsupervised paradigm, our approach has distinct motivations: 1) to achieve high generalizability in the presence of low-quality training signals, and 2) to simplify the training process through online optimization.
SE(3) Equivariant Networks for Point Clouds. Equivariant networks have superior discriminative and expressive ability for various data structures (e.g., graphs 55 65], images [15] 49) under the transformation of some symmetry groups Among them, the equivariance of a Special Fuclidean
group (3) (SE(3) has drawn increasing attention 3D point cloud processing [3
67 63
43 8 53). A vast majority of these researches focus on global, while the part-level local equivariance under-explored. Recently, 773 has utilized part-level SE(3) equivariance for supervised bounding box detection. Concurrently with our work, Lei et al. 41 and Feng et al. [24] apply part-level equivariance sategory-speciife tasks of object segmentation and human body parsing, respectively. Differently, we leverage part-level equivariance to the unsupervised category-agnostic problem.
3 Methodology
3.1 Background: SELD)=equivariinc///vvaaaaan
Discretization
Given point cloud X E RRax3 of n points and rigid transformation T E SE(3): Rax3 + R=x3 a neural network mapping inputs to a feature domain ó R=x3 Fis is defined as if 6(T X) T o (X), vT E SE(3), where o means performing an SE(3) transformation. Likewise, SE(3)-invariance is expressed as @(T X) = ((X). To reduce the computational cost of equivariant networks, [12) discretizes the rotation space into icosahedral group g of 60 rotational angles (6g) = 60). As a further extension, Equivariant Point Network (EPN) [ operates the SE(3) discretization on point clouds. For point x: R in the input X, the feature extractor of EPN outputs the per-point representation f(x) R where D is the feature dimension. By definition, fz) is SEu))-eqivvrrian to its neighbors of the convolution kernel's receptive field:
1) Rotation equivariance within the icosahedral group: f(g o x) go f(x), Vig E g.
2) Translation invariance w.r.t. arbitrary translation t: [tt xF) f()).
By relaxing the strictly rotational equivariance into the equivariance w.r.t. the 60 icosahedral angles in G. EPN is proven to be robust to many downstream tasks, such as 6D pose estimation 43 and place recognition [45]. Due to its high robustness, we choose EPN as our SE(3) equivariant backbone of feature extraction. In principle, our training strategy is versatile, and other networks with per-point SE(3) eqquivaraa representations can also serve as its feature extractor.
3.2 Problem Statement
We define set of point clouds P from K frames that are not necessarily consecutive P. P2,, -.. ........PP.... }k where each frame Ps = {p) E R = ..2, 1, N} contains N three-dimennionnl points. All frames of P represent the same target which is segmented into independently moving rigid parts. For given frame Pt., the point-part rigid mask is defined as indicates that the point P) belongs to the tth rigid part; M
M2 E {0, 1 M
indicates that does not. The rigid motion of the th part between two frames (Px. P,) is denoted
'The term "target" refers to objects, environments, or scenes associated with multi- body rigid motion, such articulated objects, arrangements of indoor furniture, scenes of vehicular traffic.
3

---

Outputs
P2
Point Cloud Pair
SDL3) Features
Equivuriant Motion Extimation Hend
SE(3)
Equivariant Network
Segmentation Outputs
Joint
Invariant Segmentation Head
PS
Optintzation
Towahedione
Eigeeomment
Teitill Scene Flom
An overview of network structure and training strategy. We feed a pair of point clouds Pk. Pt into the SE(3) equivariant networks to obtain a pair of SE(3) icosahedronic features. These features are then fed into the two proposed heads for motion segmentation and invariant segmentation. The outputs, combined with scene flow, are used for a joint optimization that can be done in an unsupervised manner.
Figure 2:
as Th SE(3). This rigid transformation is specified by rotation matrix R SO(3) and translation vector th E R Note that the targets in the dataset belong various categories, including classes possibly unseen in the training data.
Given the point cloud set P. we aim to segment each frame Pa with rigid-part masks M.. and obtain part-level rigid transformation Tí between frames. The former task called rigid segmentation. while the latter is named motion estimation. In the supervised setting, ground-truth segmentation masks and rigid motions (or scene flow) are provided in the training data. the unsupervised formulation, neither segmentation motion information is available, and training input solely consists of point cloud frames.
3.3 Network Architecture
As shown in Figure 2 our network takes pair of point cloud frames as the input and outputs the predictions of rigid segmentation and motion estimation. There are three main components in the proposed framework, i.e., a feature extractor, an invariant segmentation head, and an equivariant motion estimation head.
Per-point Feature Extractor: For an input frame Pk. the feature extractor of EPN outputs per-point -quiiiaaua representations Fg E [R following the notations in Section 3.1 and 3.2
The corresponding feature f T xD point pf can be viewed as concatenation different representations w.r.t. 9j E G over the rotation group dimension:
f (0(p: 91), 0(kk- 92), 0(p 91g - -1), 0(pi. 9()))
(1)
where 0 is a D-dimensional encoder based on stack of convolution kernels. The prediction module of vanilla EPN is designed for global SE(3)-equivariance for all input points. Differently, our unsupervised multi-bodi body task requires the model's s ability to handle part-level local equivariance, especially under low- low-quality training signals. For this purpose, we further devise two heads for rigid segmentation and motion estimation.
Point-level Invariant Segmentation Head. Rigid segmentation is an SE(3)-invariant task, the predicted mask should remain consistent for the same points across various poses and positions. Tradi- tional eqqiivaaaa structures assume that all input points undergo the same rigid transformation. which is not in alignment with the multi-body setting. To encode distinct transformations for individ- ual input points, the equivariant features 0(p) 9j) are aggregated into an invariant representation ii across the dimension of the rotation group, as depicted in Figure
uf Σ w(p 93 (p))(()))))
where w(p). Jo, selection probability of the discrete rotation 9j in G, derived through 1 x convolution. The weighted sum uf. is invariant to rigid motion given point with its neighbors

---

Invariant Sugmentation Head
Equivariant Molion Head
Medale
N
Multi-Layer Aggregative
.V XXX
x
Frame Curretaiinn
:::3
Features Oat
Regrvaise
laveriant Modole
Equivariant Module
Part-lere SEA3)
X 1 x
Rg
N'
(6)
Fentures V1
x x
Module
Equivariant Feature Module
Hardsuarr Product
M2
Probebilitic SEG
Part-lesel SE43)
f
Ixt
Fasteres
Fratures
Can-
Venteres
Eqpic.
xx
Figure An overview of Segmentation and Motion heads. The invariant segmentation head comprises an invariant module that sums 0(-) and w(.) to obtain invariant representation which is then aggregated through multiple layers to obtain a segmentation mask. Pairs segmentation masks My. Mi are fed into the equivariant motion head along with per-point equivariant features F& to obtain correlations features for predicting the final transformations of points.
in the convolution receptive field. As
Jillustrates, by fusing such invariant representations Mk(1) E I.P,.... ,..,, (k())) E IR from & layers in the feature extractor, the segmentation head outputs a soft prediction Mr E [0, 1| of the rigid mask.
Part-level Equivariant Motion Estimation Head. Part-level -Eeeeequuiattianee is desirable for motion analysis, especially rotation estimation. Based the noisy predictions (Mr, M.) of the frames (k,1) 1) from the head rigid segmentation, the motion head supposed to handle these uncertain calegory-agnostic parts. Figure 3 demonstrates the operational scheme of our motion estimation head. First of all, the part-level SE(3) feature V&, E R xDx w.r.t. the rotation Jj E g of single frame k is obtained from the per-point equivariant representations F& and the predicted rigid mask Md:
Vh.j = {mt 0(pk. 9j), mit 0(pk: 9j).
m N-1
0(p N .g)), m O(p) .g)))
(3)
where mi is the element corresponding to point p in Mr and is the broadcast operation of Hardamard product. Afterward, Va.j over the rotation group G is concatenated as V₂ E D followed by permutation-invarian operation Nx xDx 1 (e.g., max pooling) over all the points to produce part-level equivariant features Vj (V2).. Between two frames (k, I), the part-level rotation correlated feature Cы E R x x is defined as:
CR = V.V""
4
where T is matrix transposition. Cs is calculated upon "softly matching" within each consistent rigid part, while the specific category labels can be agnostic the model. Based on the correlated feature Cki. the motion head estimates rotation R and translation i of each rigid part s. More implementation details can be found in Supp.
3.4 Unsupervised Training Strategy
Ideally, the relation among scene flow ōạz multi-body transformation Te is as follows:
I Nx3 rigid segmentation ME of the th part, and
= Ps 84 UT (M) Pt)).
(5)
wherein the operation of union set is denoted U. and signifies the removal of a point pt if it does not the rigid 8. Nevertheless, their predictions f M₁ and du may exhibit considerable amount of interdependent noise during the unsupervised training process. To mitigate the noise in one prediction by leveraging the other two, we propose an online training strategy without interrupting the end-to-ena optimization of network, as shown in Figure
5

---

Cold Start & Scene Flow rgg Updating. Scene flow creates relation between motion and segmen- tation, As dense vector field that maps points to points, can directly assist point-level segmentation. At the same time, scene flow also contains movement information for motion estimation. We initialize by collecting noisy scene flow from an unsupervised flow estimator (e.g., FlowStep33D [8 D, and calculate rudimentary estimates of rigid masks and transformation by minimizing the discrepancy between their derived displacement and scene flow. As the accuracy of their estimates Mc. and T improves, the scene flow is online corrected during a training epoch:
8 8R1 = -m + (1- (1-a) (Mī Pc)] Pk),
(6)
where a E [0, 1] is decay factor to control the updating rate. In this manner, improved scene flow is capable of providing enhanced supervision learn segmentation masks and motion estimates.
Motion & Flow Segmentation MA- Although scene flow encodes point-level supervisory signals for segmentation, accurately computing flow between non-adjacent frames is challenging. pointed out by Song and Yang [61]. To alleviate the influence of miscaleulated scene flow, optimize our segmentation head based on the consensus between the motion head and updated scene flow. Given the outputs ki th of our motion head, and the interpolation point pi = pj. + 6%
derived from scene flow, the consensus score 3%(4) of the point pi, w.r.t the rigid between the frames (k: is defined as:
exp(--] + tt P'Il)).
where T is a temperature coefficient to set the sharpness of 3 and || is 22-norm. Intuitively, a low (e(() indicates large disagreement between scene flow and the motion head, of which the mask should have small learning weight in the segmentation loss:
seg
1
ΣΣ
18 1, Th pb)ll:
(8)
NS
where Tú is computed based on the scene flow and segmentation, as described in the next paragraph.
we employ the Segmentation & Flow - Motion T Following previous works 32
61
weighted-K Kabschh algorithm [37 27 determine part-level rigid transformation T given the estimates of scene flow and rigid masks. Further, the motion head is optimized by the rotation component of T and estimates corresponding translation by minimizing our probability-bug based part-level distance. The implementation details of this distance and our motion loss can be found in Supp.
Experiments
4.1
Datasets & Metrics
Our model evaluated on three various application scenarios with four datasets: 1) SAPIEN 688
for articulated objects, 2) OGC-DR and its minge-view counterpart OGC-DRSV 66 for furniture arrangements; 3) KITTI-SF [48 for vehicular traffic. Following previous works 66 32 5T, evaluate the rigid segmentation performance (AP), Panoptic Quality (PQ). F1-score (F1), Precision (Pre), and Recall (Rec) at an Intersection over Union threshold of 0.5, in addition to the mean Intersection over Union (mIoU) and Rand Index (RT). In terms of multi-body motion estimation, we report Lond-Point-Eroo 3D (EPE3D) in the main body, and other metrics (e.g., Outlier) are provided in Supp. More information about the datasets and our evaluation protocol can be found in Supp.
we
the following seven measures: Average Precision
4.2 Pilot Studies on The Two Heads
We first conduct experiments on SAPIEN to verify the generalizability of our two-head structure, including the point-level invariance of the segmentation head and the part-level equívariance of the motion head.

---

Trale
10
Val
15.2
65-
10
149
65-
12.9
82
Trai A1.
61.5
Irats
Yol
Trais
50
235
EPM
Ouz
KPCcee
EPN
Ouu
wio wwo.aarieed Eestrainnee
Oura
(6) 5r
Recallg
(c) Angle
Figure 4: Results of pilot studies. (a)(b) are the experimental results of our segmentation head, while (c) is the exploration outcomes of the motion head.
Can the point-level invariance of our segmentation head help generalize to open-set motion? On SAPIEN, each articulated target has four frames with various part-level rigid orientations and locations. By training the segmentation head subset of the 1st frames (with the canonical pose in (3) and testing them on all frames, we compare its performance with the segmentation results using the same parameters of the non-equivariant counterpart KPConv [62] and global equivariant EPN. As can essentially boost the generalization even given the global since there are some targets that only contain two rigid parts. By introducing point-level invariance, both precision and recall can be further improved.
shown in Figure (b), the
Can the part-level equivariance of motion head help generalize to category-agnostic parts? Note that the category of targets for training, validation, and testing on SAPIEN is completely disjoint. By optimizing the motion head the training and testing its performance on the validation data, we ensure that the targets do not overlap in categories so their parts are entirely category-agnostic. To exclude the interference of part segmentation, we directly provide the ground -truth rigid partition as the mask. By comparing the decrease in performance on the validation data when using part-level equivariance vs. using it, illustrated in Figure e). it observed that the mean angular error of our model only increases by less than 2", while that without part-level equivariance surges from 5.69 70. This demonstrates the strong generalizability of our model category-agnostte parts.
Table 1: Ablation studies SAPIEN.
Seg. Head
Scene Flow
Mot. Head
Metries
SF(3) Point-level
Current Past
Consensus
AP
PQ1
F1
Pret
Rect
mloUt
RIT
Feat. Flexibility
45.2
44.2
58.9
53.8
65.1
60.9
71.2
51.7
50.0
65.8
64.7
67.0
61.6
72.3
55.3
52.8
68.3
65.9
70.0
62.3
72.7
54.8
52.0
67.6
66.0
69.3
63.5
73.8
57.0
51.6
67.3
63.8
71.1
63.1
73.2
v
63.8
61.3
77.3
84.2
71.3
63.7
75.4
4.3 Ablation Studies
We perform detailed ablation studies on SAPIEN to explore the effects of three main components: segmentation head, motion head, and scheme of scene flow updating.
Segmentation Head. Two crucial design elements significantly enhance our segmentation head: 1) the use of SE(3)--quivarian features, and 2) point-level flexibility for invariant representations. By simply introducing the global equivariant features, AP increases from 45.2% to 51.7%. This demonstrates the importance of -Euiaaiquie in modeling rigid transformations, The proposed point-level flexibility further boosts the performance to 55.3%, suggesting that our point-level invariance highly advantageous to the multi-body task.
Scene Flow Updating Scheme. For the updated choice of scene flow, we empirically test three options: 1) completely relying on the past flow extracted by an unsupervised flow network, 2) ignoring the past flow and directly using the current estimation, and 3) combining them together with decay

---

MBS
OGC
OGC-sup
Ours-unsup
GT
Figure 5: Qualitative comparison between our unsupervised results and other methods (including supervised ones) SAPIEN. This provides a glimpse into the qualitative performance of our approach, and more results can be found Supp.
factor. It is observed that the extreme choices of 1) or 2) result in minor performance differences (AP: 55.3% vs. 54.8%). The significant improvement comes from the weighted combination in option 3), achieving the AP of 57.0%.
Motion Head. The motion head contributes to the consensus score for the removal of scene flow noise. From the last row of Table the consensus between the motion head and scene flow significantly boosts the final segmentation performance in most of the metrics. By controlling the point-level learning weight during the training process, our motion head can assist in optimizing segmentation head.
Table 2: Rigid segmentation and motion estimation results on SAPIEN. indicates that we evaluate these metrics upon the officially released model; means that the metric is unavailable.
APT
PQ
FIf
Pref
Rect
mloU
RIt
EPE3D
PointNet++
55
51.2
65.0
MeteorNet
46
45.7
60.0
Supervised Methods
DeepPart
71
53.0
67.0
5,95
MBS 32
49.4"
52.6*
67.6
61.4
75.2*
67.3
77.0
5.03
OGC-sup
61
66.1
48.7
62.0
54.6
71.7
66.8
77.1
Ours-sup
73.5
57.8
71.1
65.6
77.7
72.6
81.4
3.86
TrajAffn
52
6.2
14.7
22.0
16.3
34.0
45.7
60.1
SSC [1
9.5
20.4
28.2
20.9
43.5
50.6
65.9
WardLinkage
[66
17.4
26.8
40.1
36.9
43.9
49.4
62.2
Unsupervised
DBSCAN
[19
6.3
13.4
20.4
13.9
37.9
34.2
51.4
Methods
NPP
28
51.5
66.0
21.22
OGC 61
55.6
50.6
65.1
65.0
65.2
60.9
73.4
Ours
63.8
61.3
77.3
84.2
71.3
63.7
75.4
5.47
4.4 Results & Comparisons
4.4.1 SAPTEN
SAPIEN is challenging dataset of articulated objects since its training, validation and testing sets comprise completely disjoint categories of objects. This domain gap in the data split complicates the precise segmentation of rigid parts. Adapting model the supervised formulation is straightfor- ward by optimizing the segmentation with ground-truth training labels. Therefore, we also report the supervised performance the proposed framework. As shown Table 2 our segmentation performance sets a new benchmark across all metrics in both supervised and unsupervised settings.
In line with the default setting of 322 estimate motion between both consecutive frames and non- adjacent ones. Table[rrepore its EPE3D performance and comparison: our unsupervised performance of 5.47 EPE3D motion estimation is par with the best existing supervised method (MBS), while our supervised EPE3D achieves 3.86, marking relative error reduction of approximately 23%.
As depicted in Figure T owing to the lightweight two-head structure, our model encompasses only a small number of parameters (0.25M) and incurs a low cost in computational complexity (0.92G floating point operations, FLOPs). We also show sneak peek of the qualitative results (Figure 5). Our unsupervised method is comparable (and often times better) than other supervised methods. More results shared in the Supp.

---

Table 3: Rigid segmentation results on OGC-DR and OGC-DRSV.
APT
Rect
mloUt
Supervised Methods
OGC-sup [
90.7 86,3
82.6/ 78.8
87.6/85.0
83.7/82.2
92.0/ 88.0
89.2/83.9
91.7 /7.1
Ours-sup
92.8/89.3
86.9/82.6
91.08879
85.8/85.5
93.2 90.4
91.2/ 86.6
98.7 97.9
TrajAIIn 52
46.7/43.8
57.8/548
46.8/45.9
SSC 50
74.5/70.3
79.2/75.4
84.2/81.5
9
77.3/74.7
74.6/70.8
91.5/91.3
Unsupervised Methous
WardLinkage
66
72.3/69.
74.0/71.
82.5 80.5
93.9/91.8
73.6/71.7
69.9/67.2
DBSCAN
19
73.9/719
76.0/76.3
81.//818
85.8 79.1
77.8/84.8
70.7/80.1
91.5/
OGC
61
92.3/86.8
85.1/77.0
89.4 83.9
85.6/
93.6/91.2
90. 8/ 90.8/848
97. 97.8195.4
93.9/ 88.1
87.0/80.0
87.07
95.6/92.2
4.4.2 OGC-DR & Single-view Counterpart
OGC-DR is dataset for indoor furniture arrangements, where the training, validation, and testing instances are distinct from one another. OGC-DRSV, the single-view version of OGC-DR, presents challenge due the incomplete nature of the furniture caused by occlusion, making it difficult to identify consistent rigidity. As demonstrated in Table the proposed framework constantly surpasses the state-of-the-art models across all measures. Remarkably, even under the most challenging condition, Le., unsupervised single-view, our AP still achieves the value of 88.1%. This underscores the robustness of our model handling incomplete observations. The performance on motion estimation can be found in Supp.
Table Rigid segmentation results on KITTI-SE Our model still achieves competitive results even though the data setting inconsistent with the model's assumption.
Method Category
Method
APt
PQ
F11
Pre
Rect
mloUt
RIt
Supervised
OGC-sup [6]
62.4
52.7
65.1
63.4
67.0
67.3
95.0
Methods
Ours-sup
65.1
56.3
68.6
69.4
67.8
69.5
95.7
TrajAffn [52
24.0
30.2
43.2
37.6
50.8
48.1
58.5
SSC 51
12.5
20.4
28.4
22.8
37.6
41.5
48.9
Unsupervised Methods
WardLinkage
666
25.0
16.3
22.9
13.7
69.8
60.5
44.9
DBSCAN
19
13.4
22.8
32.6
26.7
42.0
42.6
55.3
OGC 61
54.4
42.4
52.4
47.3
58.8
63.7
93.6
Ours
53.6
44.4
55.1
56.3
54.0
61.5
93.4
4.4.3 KITTI-SF
Strictly speaking, KITTI-SF is not multi-body rigid dataset, as the background its point clouds may be deformable. This is inconsistent with the assumption of our framework, which posits that the feature is only equivariant for rigid transformations. However, as shown in Table our framework still delivers competitive performance in rigid segmentation despite this inconsistency.
Conclusion
This paper introduces a part-level (B(3)-equivarian framework for modeling multi-body rigid motion. The two heads for rigid segmentation and motion estimation are meticulously designed based on their inherent invariant and equivariant characteristics. The relationship among scene flow, rigid segmentation, and multi-body transformation is then exploited to derive an unsupervised optimization strategy. Our approach achieves saate-fftthe-att results multiple datasets while significantly reducing the required parameters and computations.
assumption: the part-level motion should be a rigid transformation. Therefore, as shown in Table 3 if the observation of peve- movements is non-rigid (such as occluded single views), it would suffer performance decrease. In the future, we would like to develop a model that more robust to its observation.
Limitations & Future Work. Our model is predicated

---

Acknowledgements. Our research is supported by Amazon Web Services in the Oxford-Singapon Machine Collaboration Programme and by the ACE-OPS project (EP/S030832/1). We are grateful all of the anonymous reviewers for their valuable comments.
References
[1] S. A. Baur, J. Emmerichs, F. Moosmann, P. Pinggera, B. Ommer, and A. Geiger, "Slim: Self-supervised lidar scene flow and motion segmentation." in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. pp. 13 126-1161 13 136.
[2] A. Behl, D. Paschalidou, Donne, and Geiger. "Pointflownet: Learning representations for rigid motion estimation from point clouds," in Proceedings IEEE Conf. on Computer Vision and Puttern Recognition
(CVPR), June 2019.
[3] J. Behley, M. Garbade, Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall, "Semantickitti: dataset for semantic scene understanding of lidar sequences," " Proceedings of the IEEF/CVF international conference on compuder vision, 2019, pp. 9297-9307.
[4] T. Birdal and S. Ilie, "Cad priors for accurate and flexible instance reconstruction," in Proceedings of the IEEE international conference computer vision, 2017, pp. 133-142.
[5] J. Brandstetter, R. Hesselink, E. van der Pol, E. J. Bekkers, and M. Welling. "Geometric and physical quan- tities improve (3) equivariant message passing," " in International Conference on Learinng Representations, 2021.
[6] H. Caesar, Bankiti, H. Lang, Vora, V. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, "nuscenes: A multimodal dataset for autonomous driving." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 11 621-11 631.
[7] H. Cao, Y. Lu, C. Lu, B. Pang. G. Liu, and A. Yuille, "Asap-net: Attention and structure aware point cloud sequence segmentation." British Machine Vision Conference (BMVC), 2020.
[8] E. Chatzipantazis, S. Pertigkiozoglou, E. Dobriban, and K. Daniilidis, "Se (3)-equivariant attention networks for shape reconstruction in function space," arXiv preprint wxXx:22244002394 2022.
[9] H. Chen, S. Liu, W. Chen, H. Li, and R. Hill, "Fquivariant point network for 3d point cloud analysis," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 14 514 14523.
[10] Y. Chen, M. Li, L. Lan, H. Su, Y. Yang, and C. Jiang. "A unified newton barrier method for multibody dynamics," " ACM Transactions on Graphics (TOG), vol. 41, no. 4, pp. 1-14, 2022.
[11] Cheng and H. Ko, "Bi-poinfflowee: Bidirectional learning for point cloud based scene flow estimation," in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII. Springer, 2022, pp. 108-124.
[12] T. Cohen, Weiler, B. Kicanaoglu, and Welling, "Gauge equivariant convolutional networks and the icosahedral can," in International conference on Machine learning. PMLR, 2019, pp. 1321-1330, 1330.
[13] T. Cohen and M. Welling, "Group equivariant convolutional networks." in International conference on machine learning. PMLR, 2016. pp. 2990-2999.
[14] T. S. Cohen, Geiger, Köhler, and M. Welling, "Spherical cnns," International Conference on Learning Representations, 2018
[15] T. S. Cohen and M. Welling, "Steerable cnns," in International Conference on Learning Representations, 2016.
[16] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. NieBner, "Scannet: Richly-an annotated 3d reconstructions of indoor scenes." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 5828-5839.
[17] C. Deng. O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and L.. Guibas, "Vector neurons: A general framework for so (3)-cquivariant networks," in Proceedings of the IEEE/CVE International Conference on Computer Vision, 2021, pp. 12 200-12
[18] L. Ding. S. Dong, T. Xu, X. Xu, J. Wang, and J. Li, "Fh-net: A fast hierarchical network for scene flow estimation on real-world point clouds," in Computer Vision ECtenn-ECVAAAC 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXIX. Springer, 2022, pp. 213-229.
[19] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, "A density- algorithm for discovering clusters in large spatial databases with noise," in Proc. 2nd Int. Conf. on Knowledge Discovery and Data Mining (KDD '96
J. Simoudis, J. Han, and U. Fayyad, Eds. AAAI Press, 1996, pp. 226-231.
[20] H. Fan, Y. Yang, and M. Kankanhalli, "Point 4d transformer networks for spatio temporal modeling in point cloud videos." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021. pp. 14 204 14313.
10

---

[21]
"Point spatio-temporal transformer networks for point cloud video modeling." IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2. pp. 2181-2192, 2022.
[22] H. Fan, X. Yu, Ding. Y. Yang. and M. Kankanhalli, "Pstnet: Point spatio-temporal convolution on point cloud sequences," in International Conference on Learning Representations, 2021.
[23] H. Fan, X. Yu, Y. Yang, and M. Kankanhalli, "Deep hicrarchical representation of point cloud videos via spatio-temporal decomposition." IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44,
no, 12, pp. 9918-9930, 2021.
[24] H. Feng, P. Kulits, S. Liu, M. J. Black, and V. Abrevaya, "Generalizing neural human fitting to unseen poses with articulated se (3) equivariance," arXiv preprint arXiv:2304.1055 2023.
[25] A. Geiger, M. Lauer, C. Wojek, C. Stiller, and R. Urtasun, "3D Traffic Scene Understanding From Movable Platforms," IEEE Transactions on Puttern Analysis and Machine Intelligence, vol. 36, no. 5, pp. 1012-1025, 2014.
[26] S. Giancola, J. Zarzar, and B. Ghanem, "Leveraging shape completion for 3d siamese tracking," in Proceedings of the IEEESCVF conference on computer vision and pattern recognition, 2019, pp. 1359- 1368.
(27] Z. Gojeie, C. Zhou, J. D. Wegner, L. J. Guibas, and T. Birdal, "Learning multiview 3d point cloud registration." in Proceedings of the IEEE/CVF conference on computer vision and puttern recognition. 2020, pp. 1759-1769.
[28] D. S. Hayden, J. Pacheco, and J. W. Fisher, "Nonparametric object and parts modeling with lie group dynamics," in Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognation, 2020, pp. 7426-7435.
[29] Y. He, L. Chen, J. Xic, and L. Chen, "Learning 3D Semantics From Posc-Noisy 2D Images with Hicrarchi- cal Full Attention Network," in Computer Vision ECCV 2022 Workshops. Springer Nature Switzerland, 2023, pp. 726-742.
[30] Q Hu, B. Yang. S. Khalid. W. Xiao, N. Trigoni, and A. Markham, "Towards semantic segmentation of ucban-scale 3d point clouds: A dataset, benchmarks and challenges." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 4977-4987.
[31] Q. Hu, B. Yang, L. Xie, S. Rosa, Y. Guo, Z. Wang, N. Trigoni, and A. Markham, "Randla-net: Efficient semantic segmentation large-scale point clouds," Proceedings of the IEEE/CVE conference on computer vision and pattern recognition. 2020, pp. 11 108 11 117.
[32] J. Huang, H. Wang. T. Birdal, M. Sung. F. Arrigoni, S.-M. Hu, and J. Guibas, "Multibodyssne: Multi- body segmentation and motion estimation via 3d sean synchronization." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2021. pp. 7108-7118.
[33] J. Huang, S. Yang. Z. Zhao, Y.-K. Lai, and S.-M. Hu, "Clusterslam: A slam backend for simultaneous rigid body clustering and motion estimation," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 5875-5884.
[34] S. Huang, Z. Gojeic, Huang, A. Wieser, and K. Schindler, "Dynamic 3d scene analysis by point cloud accumulation," in Computer Vision-EECV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVIII. Springer, 2022, pp. 674-690.
[35] H. Isack and Y. Boykov, "Energy-basee geometric mulli-mode fitting." " International journal of computer vision, vol. 97, no. 2. pp. 123- 227,147, 2012.
[36] B. Jiang. Y. Zhang, X. Wei. X. Xue, and Y. Fu, "Learning compositional representation for 4d captures with neural ode." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2021, pp. 5340-5350.
[37] W. Kabsch, "A solution for the best rotation to relate two sets of vectors," Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, vol. 32, no. 5, pp. 922299222999 923, 1976.
[38] Y. Kittenplon, Y. C. Eldar, and D. Raviv, "FlowStep3D: Model Unrolling for Self- Self-Super Scene Flow Estimation," IEEE/CVE International Conference on Computer Vision (CVPR), 2021.
[39] F. Kluger, E. Brachmann, H. Ackermann, C. Rother, M. Y. Yang, and B. Rosenhahn, "Consac: Robust multi-model fitting by conditional sample consensus," " in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 4634-4643.
[40] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, "Pointpillars: Fast encoders for object detection from point clouds," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 19697-22 705.
[41] J. Lei, C. Deng, K. Schmeckpeper, L. Guibas, and K. Daniilidis, "Efem: Equivariant neural field expectation maximization for 3d object segmentation without scene supervision." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 4902 12.
11

---

[42] T. Li. V. Kallem. D. Singaraju, and R. Vidal, "Projective factorization of multiple rigid-body motions," in 2007 IEEE Conference on Computer Vision and Patiem Recognition. IEEE, 2007, pp. 1-6.
[43] X. Li, Y. Weng, L. Yi, J. Guibas, A. Abbott, S. Song, and H. Wang, "Leveraging se (3) equivariance for self-supervised category-level object pose estimation from point clouds," Advances in Neural Information Processing Systems, vol. 34, pp. 15 370-15 381, 2021.
[44] X. Li, Q. Huang. Z. Wang. Z. Hou, and T. Yang. "Sequentialpoinnnet: A strong parallelized point cloud sequence network for 3d action recognition," arXiv preprint arXiv:21 1.00492, 2021.
[45] C. E. Lin, J. Song. R. Zhang, M. Zhu, and M. Ghaffari, "Se(3)-equivarian point cloud-based place recognition," in Proceedings of The 6th Conference on Robot Learning, ser. Proccedings of Machine Learning Research, K. Liu, D. Kulic, and J. Ichnowski, Eds., vol. 205. PMLR, 14-18 Dec 2023, pp. 1520-1530.
[46] X. Liu, M. Yan, and J. Bohg. "Meteornet: Deep learning on dynamic 3d point cloud sequences," in Proceedings of the IEEE/CVF International Conference Computer Vision, 2019, pp. 9246-9996999 9255.
[47] I.. Magri and A. Fusiello, "Fitting multiple heterogeneous models by multi-class cascaded t-linkage," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. 2019, pp. 7460-7468.
[48] M. Menze and A. Geiger. "Object scene flow for autonomous vehicles," in Proceedings of the IEEE conference computer vision and pattern recognition, 2015, pp. 3061-3070.
[49] T. W. Mitchel, N. Aigerman, V. G. Kim, and M. Kazhdan, "Möbius convolutions for spherical cnns," in ACM SIGGRAPH 2022 Conference Proceedings, 2022, pp. pp. 1-9.
[50] M. Niemeyer. L. Mescheder, M. Oechsle, and A. Geiger, "Oecupancy flow: 4d reconstruction by learning particle dynamics," in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 5379-5389.
[51] U. M. Nunes and Y. Demiris, "3d motion segmentation articulated rigid bodies based on RGB-D data." in British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018. BMVA Press, 2018, p. 255.
[52] P. Ochs, J. Malik, and T. Brox, "Segmentation of Moving Objects by Long Term Video Analysis." IEFE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2014.
[53] O. Puny, M. Atzmon, E. J. Smith, Misra, Grover, H. Ben- Ben-Hamu, and Y. Lipman, "Frame averaging for invariant and equivariant network design," in International Conference on Learning Representations, 2022.
[54] C. R. Qi. W. Liu, C. Wu, H. Su, and L. J. Guibas, "Frustum pointnets for 3d object detection from rgb-d data," " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.
918 9277
[55] C. R. Qi. L. Yi. H. Su, and L. J. Guibas, "Pointnet++: Deep hierarchical feature learning on point sets in metric space," Advances in neural information processing systems, vol. 30, 2017.
[56] H. Qi, C. Feng, Z. Cao, F. Zhao, and Y. Xiao, "P2b: Point-to-box network for 3d object tracking in point clouds." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 6329-6338.
[57] D. Rempe, T. Birdal, Y. Zhao, Z. Gojeic, S. Sridhar, and L. J. Guibas, "Caspr: Learning canonical spatiotemporal point cloud representations." Advances in neural information processing systems, vol. 33, pp. 13 688-13 701, 2020.
[58] S. Shi, X. Wang, and H. Li, "Pointrenn: 3d object proposal generation and detection from point cloud," in Proceedings of the IEFE/CVF conference on computer vision and pottern recognition, 2019, pp. 770-779.
[59] Y. Shi, X. Cao, F. Lu, and B. Zhou, "P" 3-net: Part mobility parsing from point cloud sequences via learning explicit point correspondence,"" in Proceedings of the AAAI Conference Artificial Intelligence, vol. 36, no. 2, 2022. pp.
[60] Y. Shi, X. Cao, and B. Zhou, "Self-supervised learning of part mobility from point cloud sequence," in Computer Graphics Forum, vol. 40, no. 6. Wiley Online Library, 2021, pp. 104-116 116.
[61] 7. Song and B. Yang. "Oge: Unsupervised 3d object segmentation from rigid dynamics of point clouds," in Advances in Neural Information Processing Systems, 2022.
[62] H. Thomas, C. R. Qi, J.-E. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas, "Kpconv: Flexible and deformable convolution for point clouds," in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6411 11- 1-6400.
[63] H. Wang. Y. Liu, Z. Dong, and W. Wang, "You only hypothesize once: Point cloud registration with ootation-equivarte descriptors," in Proceedings of the 30th ACM International Conference on Multimedia, 2022. pp. (630-1641.
12

---

[64] K. Wang and S. Shen, "Estimation and propagation: Scene flow prediction occluded point clouds."
IEEE Robotics and Automation Letters, vol. no. 4, pp. 201-12 2022.
[65] L. Wang, H. Liu, Y. Liu, J. Kurtin, and S. Ji, "Learning hierarchical protein representations via complete
3d graph networks," in The Eleventh International Conference on Learning Representations, 2022.
[66] H. Ward, "Hierarchical grouping to optimize an objective function." Journal of American Statistical Association, vol. 58, no. 301, pp. 234,244, 1963.
[67] M. Weiler, F. A. Hamprecht, and M. Storath, "Learning stecrable filters for rotation equivariant enns." in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 849 858.
[68] F. Xiang. Y. Qin, K. Mo. Y. Xia. H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al., "Sapien: A simulated part- based interactive environment." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020, pp. 11 097-11 107.
[69] X. Xu, L.-F. Cheong, and Z. Li, "3d rigid motion segmentation with mixed and unknown number of models," IEEE Transactions Pattern Analysis and Machine Intelligence, vol. 43, no. 1, pp. 1-16, 2019.
[70] B. Yang. J. Wang. R. Clark, Q. Hu, Wang, A. Markham, and N. Trigoni, "Learning object bounding boxes for 3d instance segmentation on point clouds," Advances in neural information processing systems,
vol. 32, 2019.
[71] L. Yi, H. Huang, D. Liu, Kalogerakis, H. Su, and Guibas, "Deep part induction from articulated object pairs." ACM Trunsactions on Graphics (TOG), vol. 37, no. 6, pp. 1-15, 2018.
[72] T. Yin, X. Zhou, and P. Krahenbuhl, "Center-based 3d object detection and tracking." " in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 784-11 793.
[73] H.-X. Y., J. Wu, and L. Yi. "Rotationally equivariant 3d object detection," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022, pp. 1456- 1464.
[74] J.-X. Zhong, K. Zhou, Q. Hu, B. Wang. N. Trigoni, and A. Markham, "No pain, big gain: classify dynamic point cloud sequences with static models by fitting feature-level space-time surfaces," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 8510-8520.
13


Reviews:
[{"review_id": "h43kILts60", "reviewer": "Reviewer_sjc7", "summary": "This paper presents a method that estimates the part segmentation, and the rigid motions of the parts, for an articulated object (with multiple rigid parts), given as input a sequence of point clouds of that object. The method works by first computing SE3-equivariant features for the points, and then processing with an SE3-invariant head for segmentation estimation (pooling across rotation groups), and then processing with an SE3-equivariant head for rigid motion estimation (pooling across points within estimated segments). The method also initializes a scene flow estimate from a pre-trained off-the-shelf model, and then iteratively updates this to match the estimated rigid motion. Finally, the model uses the weighted-Kabsch algorithm to estimate transformations for the parts according to the flow and segmentation, which yields optimization targets that encourage the estimates to be self-consistent.", "questions": "Do the two point clouds correspond? I suppose not, because then motion can be computed just from taking a difference of the positions.\n\n\nThe problem statement in 3.2 suggests a sequence, but later (e.g., in 3.3) it seems like the sequence length might just be 2. Is the sequence length ever not 2?\n\n\nThe w(p^i_k,g_j) used in equation 2 are described as \"derived through a 1x1 convolution\". What does this mean exactly?\n\nThe paper talks about \"fusing such invariant representations u^i_{k(1)},... from h layers\" -- how is this fusing done exactly?\n\nThe paper says \"the segmentation head outputs a probabilistic prediction M^k of the rigid mask\" -- what makes it probabilistic exactly? This idea of \"probabilistic\" masks is repeated many times through the paper, and at some point there are also \"probabilistic SE3 features\" (actually only in Figure 3). In my experience, the word \"probabilistic\" is usually used to indicate something non-deterministic, but in this case it seems like it might actually be deterministic. What exactly is meant by a probabilistic mask or probabilistic feature? \n\nThe paper says \"the motion head is supposed to *handle these uncertain category-agnostic parts*\" (italics in the paper). Why are those words in italics? What is meant by \"handle\" here? It seems like a special usage of the word. \n\nEquation 3 looks like it might just be depicting an element-wise product. Is this correct? It would be great to write this more efficiently. \n\nThe paper says \"the specific category labels can be agnostic to the model\". What does it mean for category labels to be agnostic to a model? \n\nThe paper says \"Based on the correlated feature Ckl, the motion head estimates rotation R^s_{kl} and translation t^s_{kl} of each rigid part s.\" This is a critical part of the model. How are the correlations used exactly, what is the predictor, how are the rotation and translation outputs represented, and how are the outputs supervised?\n\nUnder equation 6, it says \"In this manner, improved scene flow is capable of providing enhanced supervision to learn segmentation masks and motion estimates.\" I don't see how this follows from the equation. The equation only produces an updated scene flow, using a convex combination of the previous estimate and the rigid estimate. I see no \"supervision\" (or \"enhanced supervision\") applied to segmentation masks here.\n\nIn the final paragraph of the method, it says \"the motion head is optimized by the rotation component of T^s_{kl}, and estimates corresponding translation by minimizing our probabilistic part-level distance.\" What does it mean to optimize the motion head by rotation? What does it mean to estimate translation by minimizing \"probabilistic part-level distance\"? It seems like these are critical details in the procedure.", "limitations": "Looks OK", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper makes a well-motivated case for using a combination of equivariant and invariant features to jointly solve rigid part segmentation and motion estimation. The proposed network has fewer parameters and runs faster than the baselines, and also achieves higher accuracy, in both the supervised setting and the unsupervised setting. Jointly optimizing the motion, segmentation, and scene flow, makes sense.", "weaknesses": "I found this paper very difficult to follow. I am quite sure that the content in the main text is not a complete description of a reproducible method. Nearly every section defers to the supplementary material for additional information. In my reading of the guidelines, the supplementary material is for supporting the paper, not for describing key details of the method. The method in the paper should make sense on its own.\n\n Because my main complaint is on clarity, I'll put a lot of questions, but please interpret some of the questions as reflecting weaknesses, because these are things that should be clear in the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We are grateful for your acknowledgment of the motivation, novelty, and contributions of our model in terms of its parameters, complexity, and accuracy. While we understand your concern regarding the clarity of certain sections, we would like to address each of your concerns in detail below and assert that these corrections can indeed be addressed for the camera-ready version. We hope our answers, along with the support of Reviewer mTMF and ZttT that the explanation of the paper is \"clear\" and \"detailed\", can alleviate your concerns and give more merit based on the impact and originality of our method.\n\n**Q1:** Do the two point clouds correspond? I suppose not, because then motion can be computed just from taking a difference of the positions.\n\n**A1:** No, we do not assume that these point clouds correspond to one another. The correspondence between points in the two frames is unknown.\n\n**Q2:** The problem statement in 3.2 suggests a sequence, but later (e.g., in 3.3) it seems like the sequence length might just be 2. Is the sequence length ever not 2?\n\n**A2:** The length of an input sequence can exceed 2. For simplicity, we only describe how to process a pair of point clouds, but the method can be extended to a sequence longer than two frames, following common practice. In accordance with previous works [32, 61, 68], the sequence is input into a model as a pair of either any two frames (for the SAPIEN dataset) or consecutive frames (for the OGC dataset) at each iteration. We will further describe this at the start of 3.3 in the revision.\n\n**Q3:** The $w(p^i_k,g_j)$ used in equation 2 are described as \"derived through a 1x1 convolution\". What does this mean exactly?\n\n**A3:** This notation follows the paper of the original EPN [9]: the convolution of the $j^{th}$ icosahedral angle $g_j$ takes a point $p^i_k$ as its input and outputs the selection probability $w$. In other words, the value of $w$ is obtained from a 1x1 convolution, which is implemented through a point-wise mapping of the features from the last layer for a dimensional change.\n\n**Q4:** The paper talks about \"fusing such invariant representations $u^i_{k(1)},...$ from $h$ layers\" -- how is this fusing done exactly?\n\n**A4:** We apologize for not including a detailed description of our fusing process in the main body of the paper. Due to page limitations, our initial approach was to explain the high-level end-to-end idea of our method and have conformed to putting the details in the original supplementary material. Specifically, the fusing process is illustrated in Figure 2 of Supp. The multi-layer invariant representations are fused through an hourglass structure, where we downsample the features with Set Abstraction (SA) and then upsample with Feature Propagation (FP) modules [55]. Subsequently, this is followed by an attention-based decoder, as practiced in OGC [61]. We will make sure to include a brief description in Sec 3.3 in the main body to make it more readable. \n\n**Q5:** The paper says \"the segmentation head outputs a probabilistic prediction $M^k$ of the rigid mask\" -- what makes it probabilistic exactly? This idea of \"probabilistic\" masks is repeated many times through the paper, and at some point there are also \"probabilistic SE(3) features\" (actually only in Figure 3). In my experience, the word \"probabilistic\" is usually used to indicate something non-deterministic, but in this case it seems like it might actually be deterministic. What exactly is meant by a probabilistic mask or probabilistic feature?\n\n**A5:** We use the term “probabilistic” to refer to the fact that the segmentation head outputs a prediction of the rigid mask in the form of a probability distribution. Instead of a \"hard\" binary rigid partition, the mask represents the probability of each point “softly” belonging to different rigidities. Similarly, the term “probabilistic SE(3) features” refers to features that correspond to a probability distribution over SE(3) transformations, rather than a fixed rigid transformation. We apologize for any confusion and will change this term to  **soft prediction** in Sec 3.3.\n\n**Q6:** The paper says \"the motion head is supposed to _handle these uncertain category-agnostic parts_\" (italics in the paper). Why are those words in italics? What is meant by \"handle\" here? It seems like a special usage of the word.\n\n**A6:** We use italics to emphasize that the motion head can deal with parts of the data that are uncertain (estimated from the segmentation head, without ground-truth partition) and do not belong to a specific category. By “handle”, we mean that the motion head is designed to process these uncertain parts in order to produce a meaningful output.\n\n**Q7:** Equation 3 looks like it might just be depicting an element-wise product. Is this correct? It would be great to write this more efficiently.\n\n**A7:** Thank you for the question. The symbol $\\cdot$ means a broadcasting element-wise product over the $D$-dimensional feature channel, which is not exactly the same as a standard element-wise operation. Since $\\hat{m}_k^i$ and $\\theta (p_k^i,g_j)$ have different sizes, a standard element-wise operation cannot be directly applied.\n\n**Q8:** The paper says \"the specific category labels can be agnostic to the model\". What does it mean for category labels to be agnostic to a model?\n\n**A8:** This sentence occurs on Line 181, which means that our model does not require any category labels of the parts (e.g., 'Lense', 'Temple', 'Nose Pad' for eyeglasses), and the ground truth of part-level correspondence between frames is unknown to this model. We will change the wording to convey this idea more clearly.\n\n**Our response to your Q9-Q11 will be uploaded once the function of interactive comment is available.**"}, {"review_id": "sKLWb2erNy", "reviewer": "Reviewer_86bd", "summary": "Modeling multi-body rigid movements involves - rigid segmentation which is category-agnostic and motion estimation which is open-set. This work proposes a part-level SE(3) equivariant network that takes in point-clouds corresponding to video frames and can effectively estimate category-agnostic open-set motion.\n", "questions": "Please address the issues mentioned in the weakness section", "limitations": "The authors have discussed limitations of their proposed approach.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The work leverages the point level feature equivariance from existing works and uses it to obtain invariant representations for the rigid segmentation and then further uses it to estimate motion using an equivariant motion head.  \n\nThe approach is able to provide state of the art performance with much lesser flops.\n\nIt makes note of the interdependence between the tasks of rigid segmentation and motion estimation and uses that interdependence for unsupervised training. ", "weaknesses": "The paper writing could be better so that the high-level idea is more easily understood. The contributions of the work could be summarized in fewer words in the introduction section making the work much easier to follow. \n\nSE(3) has been used multiple times in the paper including the title but it has not been described in the paper anywhere. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate your recognition of our improvements on feature equivariance, the high performance and low FLOPs, and our use of interdependence for unsupervised training. Please find our responses to each concern below.\n\n**Q1:** The paper writing could be better so that the high-level idea is more easily understood. The contributions of the work could be summarized in fewer words in the introduction section making the work much easier to follow.\n\n**A1:** We really appreciate your advice to improve the readability of this paper. Currently, the main summary for each contribution is highlighted in italic format. We realize that this format may be hard to read, and will reformat the section by describing the details first and having shorter bullet points towards the end of Introduction.\n\n**Q2:** SE(3) has been used multiple times in the paper including the title but it has not been described in the paper anywhere.\n\n**A2:** Thank you for bringing this to our attention and we apologize for not including a detailed description of SE(3) in our paper. SE(3) stands for the **S**pecial **E**uclidean group in three dimensions (**3**D), which is the group of rigid body transformations in three-dimensional space. We will make sure to include a definition and explanation of SE(3) in Sec 3.1."}, {"review_id": "v329TUaGbZ", "reviewer": "Reviewer_mTMF", "summary": "This paper proposes a part-level SE(3)-equivariant framework for multi-body rigid motion modeling. The two heads that take in the SE(3) features, segmentation and motion estimation, are carefully crafted based on their inherit invariant and equivariant characteristics. The relationship of scene flow, rigid segmentation, and multi-body transformation is then exploited to derive an unsupervised optimization strategy. It achieves state-of-the-art in multiple datasets while significantly reducing the parameters and training computations required.", "questions": "Please refer to the weaknesses section, refute views or answer questions or improve the paper.", "limitations": "The limitations has been discussed.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The structure of this paper is very clear, providing a comprehensive explanation of the research problem, motivation, background, principles, methods, and experiments. The text and figures have been meticulously polished.\n\nThis work effectively utilizes SE(3) equivariance in rigid segmentation and multi-body motion estimation, and is outstanding in terms of experimental results and model size.", "weaknesses": "Rigid segmentation and multi-body motion estimation are intrinsically coupled. I wonder if this coupling might also have negative effects on the model, for instance, inaccurate segmentation leading to poorer scene flow estimation results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback and are grateful for your recognition of our writing and contributions presented in our paper, such as the utilization of SE(3) equivariance, experiments, and model size. We have carefully considered your main concern and have provided detailed responses below.\n\n**Q1:** Rigid segmentation and multi-body motion estimation are intrinsically coupled. I wonder if this coupling might also have negative effects on the model, for instance, inaccurate segmentation leading to poorer scene flow estimation results.\n\n**A1:** We agree that rigid segmentation and multi-body motion estimation are intrinsically coupled, and this coupling can have both positive and negative effects on the model.\n\n(1) It is natural that inaccurate segmentation would lead to poor scene flow estimation. To further explore the influence of inaccurate segmentation, we conduct additional experiments on the validation set of SAPIEN. By adding different levels of noise to the predictive segmentation, we compare the performance change in unsupervised motion estimation:\n\n| Noise Level | EPE3D$\\downarrow$ | AccS$\\uparrow$ | AccR$\\uparrow$ | Outl$\\downarrow$ |\n|-------------|-------------------|----------------|----------------|------------------|\n| 5%          | 4.34              | 15.0           | 32.2           | 84.1             |\n| 15%         | 6.79              | 11.6           | 26.1           | 87.6             |\n| 25%         | 9.29              | 9.28           | 21.5           | 89.9             |\n\nIt is observed that poor segmentation would lead to low performance on scene flow estimation results.\n\n(2) Experiments also demonstrate that poor scene flow could result in performance degradation in rigid segmentation. Please refer to **A2** in our response to Reviewer ZttT for more details.\n\n(3) To mitigate the negative effect of this coupling, this paper proposes some corrective schemes, such as consensus $\\beta^{s(i)}_{kl}$ (Lines 205 - 210) and smoothly updating scene flow (Lines 194 - 200). In the unsupervised setting, the benefits of coupling rigid segmentation and multi-body motion estimation outweigh the potential drawbacks. Experiments show that our model is able to leverage the complementary information provided by each task to improve its performance on both.\n\nFor future work, we encourage the community to improve our model to alleviate the side effect of the coupling under the unsupervised setting. Thank you again for your insightful comment."}, {"review_id": "0MVTdbjlvT", "reviewer": "Reviewer_ZttT", "summary": "This paper proposes a unified framework to simultaneously handle rigid segmentation and motion estimation. Specifically, an probabilistic SE(3)-equivariant network is presented to obtain transformation-invariant features, which is followed by two heads for rigid segmentation and motion estimation. Due to the interdependent nature between the above two tasks, the authors utilize a unified training strategy to jointly optimize the two heads. Extensive experiments on four datasets show that this method can yield good performance with low computational complexity.\n", "questions": "please see the weakness.", "limitations": "-In related works, deep learning on dynamic point clouds is too highly summarized. The authors can present more details.\n\n-The authors can visualize more segmentation results and intermidiate results so that the proposed method can be better understood.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-Significance and originality\n\nUnsupervised rigid segmentation and motion estimation are challenging. Taking the generalization requirement into account, the authors introduce a probabilistic part-level SE(3)-equivariant feature encoder to capture transformation-invariant features, which is different from the global SE(3) equivariance. And it is simple but makes sense.\nA unified training strategy to jointly optimize the outputs of two heads in an online fashion is well motivated. Although many other tasks also employ joint optimization, this is the first one for this task.\n\n-Clarity\n\nThe paper contains enough details and definitions of the proposed contributions.\n\n-Experiment\n\nQuantitative results show a clear improvement compared to previous methods. \n", "weaknesses": "-Time cost of the proposed methods and other methods.\n\n-The author should systematically analyze the robustness on initial scene flow. Does the performance drop a lot with poor initial scene flow results? (add some noise or with insufficiently trained scene flow results).\n\n-This paper relies on rigidity constraint and works well on finite datasets. The SAPIEN and OGC-DR datasets are without background and well-separated. The authors can conduct some experiments on large outdoor scenes, which are more complicated.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We are grateful for your recognition of the significance and originality of our training strategy based on part-level SE(3)-equivariance, clarity, and experiments. We have carefully reviewed each of your concerns and all the additional experiments would be added to the supplementary materials accordingly.\n\n**Q1:** Time cost of the proposed methods and other methods.\n\n**A1:** We conduct experiments to compare the training and the testing time costs on the whole dataset of SAPIEN, reported as follows: \n\n| Model | Fully-supervised Training (Hours) | Unsupervised Training (Hours) | Inference (Hours) |\n|-------|--------------------|-----------------------|-----------|\n| MBS   |        18.7         |          ---         |    3.2      |\n| OGC   |        4.9         |          21.8         |    0.6   |\n| Ours  |        4.5         |          17.3         |    0.8  |\n\nOur inference time is slightly longer than that of the OGC model due to the additional processing time required by the EPN to handle the 60-angle icosahedral group. However, our model has a strong ability to capture rigid information, which allows it to converge with fewer training iterations and results in a shorter overall training time.\n\n**Q2:** Does the performance drop a lot with poor initial scene flow results? (add some noise or with insufficiently trained scene flow results).\n\n**A2:** We conduct additional experiments to analyze the influence of initial scene flow on the performance of segmentation. By adding various intensities of zero-centered Gaussian noise to scene flow, the segmentation result on SAPIEN is reported as follows:\n\n|  Variance of Gaussian Noise   | AP$\\uparrow$ | PQ$\\uparrow$ | F1$\\uparrow$ | Pre$\\uparrow$ | Rec$\\uparrow$ | mIoU$\\uparrow$ | RI$\\uparrow$ |\n|----------------------------|--------------|--------------|--------------|---------------|---------------|----------------|--------------|\n|0 (w/o Noise)|63.8|61.3|77.3|84.2|71.3|63.7|75.4|\n|0.01|62.7|60.1|76.2|82.0|71.1|63.7|75.1|\n|0.025|60.3|57.4|73.0|75.6|70.6|62.6|74.8|\n|0.05|53.8|32.9|46.5|36.5|64.2|56.7|68.1|\n|0.1|49.2|27.1|40.3|30.2|60.5|53.4|65.2|\n\nOur model has been observed to be robust to noisy scene flow within a variance range of 0.01 to 0.025. However, as the intensity of the noise increases to 0.05, there is a noticeable decline in performance. \n\n**Our response to your Q3-Q5 will be uploaded once the function of interactive comment is available.**\n\n"}, {"review_id": "yFNnzGvque", "reviewer": "Reviewer_S5HY", "summary": "This paper studies the problem of rigid part segmentation and motion segmentation. The proposed network first uses a global SE(3) equivariant backbone to extract point features, then a part segmentation head will predict invariant zero-order segmentation and a motion head will predict one-order transformations. The network can be trained on pairs of point clouds before and after deforming with noisy scene flow. \n\nThe multibody equivariance is approximate because of 1.) descretization in EPN, 2.) the global backbone can’t handle deformation.", "questions": "See weakness.", "limitations": "The limitation is explicitly highlighted in the paper, but the reviewer hasn't found the social impact claim.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper exploits strong geometry inductive bias — Equivariance, into the segmentation problem, which should be highlighted and emphasized more to the community.\n- The system is tested across different settings from articulated objects to outdoor scenes and the experiments are convincing.", "weaknesses": "- The main weakness of this paper is that the “equivariance” is not exact, i.e. because there are pooling and information aggregation in the point cloud feature backbone when the target is deformed (not necessarily non-rigid, but rigid transformed, articulated, this is a word from equivariant learning theory), the overall point cloud changes in such cases, but the information can be propagated across rigid parts so the features of each point do not transform purely following its parts transformation. This is because without knowing the mask, the group action of SE(3)xSE(3)x…xSE(3) on the point cloud can not be defined.\n- Another weakness of this paper is that the system should take in at least a pair of point as input during inference (as far as the reviewer currently understand, please correct him if this is wrong). So the task is relatively easy because by comparing two point clouds (even by brutal-force enumerating and classical geometric algorithms ) the segmentation of rigid moved parts may be trivially found in most cases. And the system seems only to focus on moving parts rather than semantic or instance parts that may not be moved.\n- The reviewer is also curious why the author chooses to use EPN as the backbone, is there a specific reason for such a design? Note EPN is only approximate equivariance due to the discretization. In other words, will such a framework still work for other backbones like TFN or VNN?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough feedback and are grateful for your recognition of our exploitation of applying equivariance to segmentation and our convincing experiments. We have carefully considered each of your concerns and have provided detailed responses to each one below.\n\n**Q1:** The “equivariance” is not exact, i.e. because there are pooling and information aggregation in the point cloud feature backbone when the target is deformed (not necessarily non-rigid, but rigid transformed, articulated, this is a word from equivariant learning theory), the overall point cloud changes in such cases, but the information can be propagated across rigid parts so the features of each point do not transform purely following its parts transformation. This is because without knowing the mask, the group action of SE(3)xSE(3)x…xSE(3) on the point cloud can not be defined.\n\n**A1:** Thank you for helping us improve our work. We agree that per-point features cannot transform strictly along with part-level transformation, resulting in an inexactly defined local \"equivariance\". However, the inexact local equivariance is proven to be sufficiently effective in practice from existing work [73]. Specifically, EON [73] utilizes object-level equivalence in fully-supervised object detection, of which the Region Context Aggregation is not perfectly equivariant. In this paper, we explore the feasibility of part-level equivariance in unsupervised multi-body segmentation, and it performs well on various datasets. We appreciate your insight and will include a discussion on the inexactness of equivariance to Sec 5 (Limitations).\n\n**Q2:** Another weakness of this paper is that the system should take in at least a pair of point as input during inference (as far as the reviewer currently understand, please correct him if this is wrong). So the task is relatively easy because by comparing two point clouds (even by brutal-force enumerating and classical geometric algorithms) the segmentation of rigid moved parts may be trivially found in most cases. And the system seems only to focus on moving parts rather than semantic or instance parts that may not be moved.\n\n**A2:** Thank you for your valuable comment. We would like to clarify that our model is capable of taking a single frame as input **during inference** and can segment parts that may not be moved. In the training stage, our model indeed requires at least two frames since scene flow serves as a supervisory signal in the unlabeled setting. However, the segmentation head can work without the help of motions during inference. In the experiments, static cars/trucks parked at the side of the road frequently occur on KITTI-SF (Table 4 in the main body of this paper), KITTI-Det & SemanticKITTI (**A3** in our response to Reviewer ZttT), and our model still performs comparably or better than state-of-the-art methods. We will include additional notes in Figure 2 to provide further clarification on the input data.\n\n**Q3:** The reviewer is also curious why the author chooses to use EPN as the backbone, is there a specific reason for such a design? Note EPN is only approximate equivariance due to the discretization. In other words, will such a framework still work for other backbones like TFN or VNN?\n\n**A3:** (1) The choice of using EPN as the backbone in our paper was made based on its robustness (Lines 121 - 125). By regressing the residual after selecting the rotational anchor, instead of directly regressing the entire rotation, vanilla EPN exhibits robust global equivariance, which is adopted in some existing unsupervised tasks like 6D pose estimation [43]. Designed for unsupervised learning, our model also adopts EPN as the backbone. (2) Theoretically, our framework should still work with other backbones such as TFN or VNN, as long as its per-point equivariant feature is accessible. Experimentally, it would require further investigation to determine their robustness to the part-level uncertainty and the unsupervised setting, which would be beyond the scope of this paper. We appreciate your suggestion and will make sure such explorations are included in future work. \n"}, {"review_id": "w5mrglG7OF", "reviewer": "Reviewer_XLXX", "summary": "The paper describes a method to simultaneously segment rigid parts and estimate part-level transformations for multi-body point cloud inputs. Given the input point cloud, equivariant features are first extracted and aggregated to invariant features for part segmentation. To estimate the transformations between two point clouds, an equivariant motion estimation head is applied to pairs of point clouds to obtain SE(3) motions. The entire model is trained in an unsupervised fashion using the consensus between 3D scene flow, motion segmentation, and transformations. Experiments show that the model is generalizable in two aspects: open-set motion and category-agnostic part segmentation. The authors test the model on SAPIEN, OGC, and KITTI datasets, and show superior performance in comparison to past baselines, with much better accuracy and much fewer FLOPs, even surpassing the supervised method.", "questions": "1. Does the current training scheme leverage the cycle consistency among multiple inputs? If not how could this be possibly incorporated into the algorithm?\n2. How does the method generalize to different point densities/distributions? What is the maximum number of points could the method process?\n3. Some typos:\n- Line 144, 'There are mainly comprised of...'\n- Line 237, 'even given the global since...'", "limitations": "Limitations are discussed in the last paragraph of the main paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The identification of two generalization problems, i.e. invariant segmentation and equivariant motion estimation, is insightful. The use of equivariant backbones and their invariant counterpart is well-suited for such a problem.\n2. The use of initial scene flow to bootstrap (or cold-start) the learning process and to get rid of local minima is novel. By utilizing the consistency between the output attributes which is inherent in the multi-body registration task, the method could be learned in an unsupervised manner.\n3. Experiments are extensive and the results are compelling. The method is tested on four different datasets ranging from articulated objects to outdoor driving scenes, justifying the wide applicability. In the meanwhile, the fact that the algorithm is able to achieve a much higher accuracy with a much lower cost is impressive and set up a strong waypoint for future works.", "weaknesses": "1. The unsupervised training strategy lacks clarity. What is the loss function? Is it eq.(8) plus eq.(7) in the supplementary with some balancing factor? This should be clarified in the main paper. In Line 213, the authors mention the weighted-Kabsch algorithm, what is the 'weight' here?\n2. The generalization of the segmentation head is questionable -- this is my most important doubt about the paper. Although the generalization to category-agnostic part segmentations is demonstrated through experiments, the reason why an equivariant network backbone could achieve that is not explained. What adds to the confusion is that the method is trained in an unsupervised manner, which means that no canonical segmentation labels are defined. Given a novel multi-body shape that is not seen from the training set (e.g. a cabinet with four drawers), how could the model give a unique index to each of the drawers in a consistent fashion (i.e. the drawer index stays unchanged across multiple articulation states)? What is the maximum number of parts supported in the algorithm? Is it predefined or optimized/given for each test sample? The doubt also links to the experiment section -- In Fig.4 (a,b), the gap between training and validation is not shown, and it's unclear whether Fig.4 (c) is based on ground truth or predicted segmentation.\n3. The experiments are not explained very clearly. In Sec4.2, why 'point-level invariance of segmentation' helps 'open-set motion' generalization, and why 'part-level equivariant of motion' helps 'category-agnostic part' generalization? Shouldn't they be swapped? It is preferable to move Sec4.4 ahead of Sec4.3.\n4. More visualizations and challenging cases should be shown. There are only 3 visualizations provided in the paper+supplementary, on SAPIEN datasets. These visualizations only contain segmentation labels with two parts. To make the results more convincing, the author should provide more visualizations on all the datasets they've tested on, with transformations shown, and on more challenging cases, e.g., large scenes with >4 rigid parts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate your recognition of our use of equivariant backbones & their invariant counterparts, our solution to the cold-start problem, and our extensive and compelling experiments. We would like to address each of your concerns in detail below.\n\n**Q1:** The unsupervised training strategy lacks clarity. What is the loss function? Is it eq.(8) plus eq.(7) in the supplementary with some balancing factor? This should be clarified in the main paper. In Line 213, the authors mention the weighted-Kabsch algorithm, what is the 'weight' here?\n\n**A1:** Thank you for bringing the clarity issues to our attention. We would like to address them as follows:\n\n(1) We apologize for your confusion caused by a missing term $\\hat M_k^s$ in Equation 8, of which the correct unsupervised segmentation loss should be $l_{seg} = \\frac{1}{NS} \\sum^N_i \\sum^S_s ||\\beta_{kl}^s \\hat{M}\\_k^s ({p^\\prime}^i_l  - \\hat{\\mathbf{T}}^s_{kl} \\circ p^i_k)||$. By minimizing $l_{seg}$, the predictive part-level mask $\\hat M_k^s$ can be optimized in a differentiable fashion. \n\n(2) We mainly utilize three tricks to balance the optimization process between the segmentation head (Eq. 8) and the motion head (Supp-Eq. 7): a) smoothly updating scene flow (Lines 194 - 200) mitigates the sharp fluctuation of motion estimation; b) the consensus $\\beta^{s(i)}_{kl}$ (Lines 205 - 210) alleviates the influence of predictive errors from either head; c) Two heads are alternately updated per batch to prevent their gradients interfering each other, and their initial learning rates are set as segmentation : motion = 2.0e-4 : 2.0e-5. \n\n(3) The weight in the Kabsch algorithm refers to the weight assigned to the deviation of each point. Following [32, 61], we use the predictive mask of rigid segmentation as the point-wise weight. \n\nTo address these concerns, we will take the following actions:\n\n(1) Rectifying Equation 8 and providing a description of the training strategy after Line 211.\n\n(2) Adding additional elaboration in Sections 2.1 and 2.2 within the supplementary material.\n\n(3) Introducing the pseudo-code for the weighted-Kabsch algorithm [37, 27] to the supplementary material and releasing our open-source implementation for public accessibility upon the publication of our work.\n\n**Q2:** Although the generalization to category-agnostic part segmentations is demonstrated through experiments, the reason why an equivariant network backbone could achieve that is not explained. What adds to the confusion is that the method is trained in an unsupervised manner, which means that no canonical segmentation labels are defined. Given a novel multi-body shape that is not seen from the training set (e.g. a cabinet with four drawers), how could the model give a unique index to each of the drawers in a consistent fashion (i.e. the drawer index stays unchanged across multiple articulation states)? What is the maximum number of parts supported in the algorithm? Is it predefined or optimized/given for each test sample? In Fig.4 (a, b), the gap between training and validation is not shown, and it's unclear whether Fig.4 (c) is based on ground truth or predicted segmentation.\n\n**A2:** (1) Segmentation Consistency: The segmentation head can take a single frame as its input, and the output part-level label indicates the partition inside a frame. The segmentation indexes are not necessarily consistent between the two frames. Following the common practice in [32, 61], we utilize the Hungarian-matching algorithm based on the IoU score between two masks to determine consistent parts. After the Hungarian-matching algorithm, the same rigid index across different frames is assigned together, and the correlated features $C_{kl}$ can be obtained for part-level equivariance in the motion head. We will include this description in Section 3.3 in the main body.\n\n(2) Maximum Number of Parts: This upper bound of part number is predefined as a sufficiently large number. Following the settings of OGC, the maximum is set as 8, 8, 8 and 15 on SAPIEN, OGC-DR, OGC-DRSV and KITTI-SF, respectively.\n\n(3) Explanation of Fig.4: a) The gap between training and validation is as follows: \n\n| Model  | Prec@50$\\uparrow$ (Training) | Recall@50$\\uparrow$ (Training) | Prec@50$\\uparrow$ (Validation) | Recall@50$\\uparrow$ (Validation) |\n|--------|------------------------------|--------------------------------|--------------------------------|----------------------------------|\n| KPConv | 71.9                         | 55.5                           | 48.6                           | 61.9                             |\n| EPN    | 73.7                         | 58.1                           | 56.2                           | 65.7                             |\n| Ours   | 74.9                         | 61.8                           | 61.7                           | 72.0                             |\n\n\nThe performance gap between training and validation for our model is significantly smaller than that of other models, providing further evidence of its generalizability. \n\nb) For a fair comparison, the evaluation of our motion head excludes the inference from part segmentation, and both experiments in Fig.4 (c) are based on ground-truth segmentation. \n\n**Our response to your Q3-Q7 will be uploaded once the function of interactive comment is available.**"}]

Author rebuttal:
We thank all reviewers for their valuable comments and suggestions. We would like to highlight some strengths of our paper pointed out by reviewers, including a **well-motivated** (sjc7) and **original** (Ztt7) idea, **extensive and convincing experiments with impressive results surpassing state-of-the-art methods** (86bd, Ztt7, S5HY, XLXX), and **clear and detailed explanation of the motivation, contributions, method, and experiments** (mTMF, ZttT). We provide individual feedback for every review to address specific concerns.

We also provide additional visualizations in the attached pdf file. The file includes (a) challenging scenes (multiple parts) on SAPIEN, with successful and failure cases (errors as circled in green on the final two rows); (b) visualizations on **all of the other datasets** (OGC-DR, OGC-DRSV, and KITTI-SF). All of the results ("Ours") are obtained from our model in the **unsupervised setting**, while "GT" means the ground-truth segmentation.

Meta-review: This paper presents a method that estimates the part segmentation and the rigid motions of the parts, for an articulated object (with multiple rigid parts), given as input a sequence of point clouds of that object. All reviewers agree on the extensive experiments and the good empirical performance. Rsjc7 and R86bd raised concerns regarding clarity and the method being not self-contained. The rebuttal submitted by the authors provided additional experiments and visualizations and addressed to the raised concerns. The authors are encouraged to make the paper self-contained and rigorously describe their method to make it accessible to a wide audience.
Decision: Accept (poster)