Title: Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Dynamic Point Cloud Analytics

Multi-body Motion
Abstract: A truly generalizable approach to rigid segmentation and motion estimation is fundamental to 3D understanding of articulated objects and moving scenes. In view of the closely intertwined relationship between segmentation and motion estimates, we present an SE(3) equivariant architecture and a training strategy to tackle this task in an unsupervised manner. Our architecture is composed of two interconnected, lightweight heads. These heads predict segmentation masks using point-level invariant features and estimate motion from SE(3) equivariant features, all without the need for category information. Our training strategy is unified and can be implemented online, which jointly optimizes the predicted segmentation and motion by leveraging the interrelationships among scene flow, segmentation mask, and rigid transformations. We conduct experiments on four datasets to demonstrate the superiority of our method. The results show that our method excels in both model performance and computational efficiency, with only 0.25M parameters and 0.92G FLOPs. To the best of our knowledge, this is the first work designed for category-agnostic part-level SE(3) equivariance in dynamic point clouds.

Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation

Abstract

A truly generalizable approach to rigid segmentation and motion estimation is fundamental to 3D understanding of articulated objects and moving scenes. In view of the closely intertwined relationship between segmentation and motion estimates, we present an SE(3) equivariant architecture and a training strategy to tackle this task in an unsupervised manner. Our architecture is composed of two interconnected, lightweight heads. These heads predict segmentation masks using point-level invariant features and estimate motion from SE(3) equivariant features, all without the need for category information. Our training strategy is unified and can be implemented online, which jointly optimizes the predicted segmentation and motion by leveraging the interrelationships among scene flow, segmentation mask, and rigid transformations. We conduct experiments on four datasets to demonstrate the superiority of our method. The results show that our method excels in both model performance and computational efficiency, with only 0.25M parameters and 0.92G FLOPs. To the best of our knowledge, this is the first work designed for category-agnostic part-level SE(3) equivariance in dynamic point clouds.

1 Introduction

Comprehending point cloud motion is critical for various 3D vision tasks in dynamic scenarios, e.g ., tracking, animation, simulation, and manipulation. Many types of 3D motion can be described as a composition of multi-body rigid movements, such as articulated objects [68], and vehicular traffic scenes [25]. Specifically, the setting of multi-body rigid motion requires all moving parts ( i.e ., bodies) to undergo only translation and rotation, without any type of deformation. The process of modeling these multi-body rigid movements typically involves two primary portions [32]: 1) the identification of distinct moving bodies ( i.e ., rigid segmentation ), and 2) the calculation of individual movements for each identified body ( i.e ., motion estimation ).

1.0

4.0

600.0

40

50

60

70

……

FLOPs

AP

%

MBS

OGC-unsup

OGC-sup

Ours-unsup

0.25M

Ours-sup

0.5M

10M

#PARAMS: M

G

# Being the first modeling portion, rigid segmentation significantly differs from traditional semantic segmentation tasks that concentrate on category information [16, 3, 30, 31, 70]. Based on multi-body motion instead of category-dependent semantics, rigid segmentation becomes category-agnostic about moving parts as rigid bodies are not limited to a specific set of shapes and naturally cannot be assigned certain category labels. As regards motion

estimation, individual pose variations would occur arbitrarily, including transformations unseen in the training data. This constitutes an open set of pose changes .

Considering the relative inaccessibility and the constrained generalizability of manual annotations, Song and Yang [61] have proposed a seminal work for unsupervised rigid segmentation. Nevertheless, the unsupervised paradigm remains a formidable challenge because of the generalizable requirement in multi-body movements and the interdependent nature between rigid segmentation and motion estimation. Due to the aforementioned category-agnostic rigid prior and open-set pose changes in multi-body motion, a model is required to generalize well to transformations of appearance, location, and orientation. Moreover, rigid segmentation and multi-body motion estimation are highly coupled -the calculation of movements for each body is based on the output of a rigid partition, while an estimate of multi-body motion facilitates further segmentation. In absence of precise supervision, it is difficult to independently obtain effective training signals for either rigid segmentation or motion estimation.

To tackle the difficulties of generalizability and interdependence, this paper presents a three-fold contribution in the form of architecture, training strategy, and experimental evaluation:

1. Architecture. We design a part-level SE(3)-equivariant network that demonstrates strong generalization to open-set motion and category-agnostic moving bodies. The SE(3)-equivariance allows features of a model to keep coherence with the rigid transformations of point cloud inputs, thereby enhancing the model's robustness to such spatial transformations ( i.e ., rigid motion) [13, 17, 9, 14, 67] and therefore is more generalizable for open-set pose changes of global rigid targets [63, 43, 8, 53]. Tailing the SE(3)-equivariance is two lightweight heads for segmentation and motion estimation. The former, unlike previous segmentation networks using global SE(3) equivariance, exhibits pointlevel local flexibility to SE(3)-invariance. The latter, robustly employing the probability of part consistency, utilizes part-level 'softly' matching operations to handle SE(3)-equivariance without requiring knowledge of part categories. As a result of integrating two lightweight heads into a unified network, rather than utilizing large independent networks, our model is characterized by a small number of parameters ( 0 . 25 M) and low computational complexity (0.92G FLOPs). To the best of our knowledge, this is the first work designed for category-agnostic part-level SE(3)-equivariance in dynamic point clouds.

2. Training Strategy. We leverage the interdependence between rigid segmentation and motion estimation and present a unified training strategy to jointly optimize their outputs in an online fashion. Based on our proposed two-head network, we simultaneously filter out the noisy flow predictions and refine the estimates of rigid motion by exploiting the interrelation among scene flow, segmentation mask, and rigid transformation. Superior to previous arts, our online training process is free from complicated components and manual intervention , e.g ., the alternation of Markov Chain Monte Carlo proposals and Gibbs sampling updates [28], the offline repetition of training multiple segmentation networks from scratch [61], or the continual optimization of an accessory neural network of flow estimation [32].

3. Experiments. Experiments on four datasets demonstrate the efficacy of our model in performance, as well as its efficiency in parameters and computational complexity. We conduct comprehensive experiments on four datasets (SAPIEN [68], OGC-DR [61], OGC-DRSV [61], and KITTI-SF [48]) across three application scenarios (articulated objects, furniture arrangements, and vehicular traffic). Noticeably, as shown in Figure 1, our performance on the SAPIEN dataset of articulated objects surpasses state-of-the-art results w.r.t. all evaluation metrics, achieving the relative gain of at least 14.7% AP in rigid segmentation and at least 23.3% in motion estimation, with only 0.25M parameters and 0.92G FLOPs. The code is available at https://github.com/jx-zhong-for-academicpurpose/Multibody_SE3.

2 Related Works

Deep Learning on Dynamic Point Clouds. Deep neural networks for point cloud video modeling aim to understand our dynamic 3D surroundings. These networks enable the resolution of several downstream tasks, such as video-level classification [20, 22, 44, 23, 21, 74, 29], frame-level prediction [57, 36, 50], object-level detection [6, 40, 54, 58] or tracking [26, 56, 72], part-level mobility parsing [60, 59, 71], point-level segmentation [22, 23, 7, 20], and scene flow estimation [18, 64, 34, 11, 2]. In contrast to the above dynamic tasks that usually assume contiguous

sequential input, multi-body rigid motion may be captured between discrete frames [32], presenting a unique challenge.

3D Motion Segmentation & Multi-body Rigid Motion. Although the systematic formulation of multi-body rigid motion modeling is a relatively recent development [32], a related problem, i.e ., motion segmentation, has been long sought after. Motion segmentation aims to group points with similar motion patterns from the input of scene flow, using such techniques as factorization [10, 42, 69], clustering[33], graph optimization [47, 35, 4], and deep learning [39, 28, 71, 1]. However, motion segmentation does not explicitly take into account multi-frame multi-body consistency. To address this issue, Huang et al . [32] present the seminal fully-supervised work for modeling multibody rigid motion. Song and Yang [61] further attempt to predict the mask of rigid segmentation via three object geometry losses in an unsupervised setting. Following the same unsupervised paradigm, our approach has distinct motivations: 1) to achieve high generalizability in the presence of low-quality training signals, and 2) to simplify the training process through online optimization.

SE(3) Equivariant Networks for Point Clouds. Equivariant networks have superior discriminative and expressive ability for various data structures ( e.g ., graphs [5, 65], images [15, 49]) under the transformation of some symmetry groups. Among them, the equivariance of a Special Euclidean group (3) (SE(3)) has drawn increasing attention to 3D point cloud processing [13, 17, 9, 14, 67, 63, 43, 8, 53]. A vast majority of these researches focus on global, while the part-level local equivariance is under-explored. Recently, [73] has utilized part-level SE(3) equivariance for supervised bounding box detection. Concurrently with our work, Lei et al . [41] and Feng et al . [24] apply part-level equivariance to category-specific tasks of object segmentation and human body parsing, respectively. Differently, we leverage part-level equivariance to the unsupervised category-agnostic problem.

3 Methodology

3.1 Background: SE(3)-equivariance/invariance & Discretization

Given a point cloud X ∈ R n × 3 of n points and a rigid transformation T ∈ SE(3) : R n × 3 → R n × 3 , a neural network mapping inputs to a feature domain ϕ : R n × 3 →F is defined as SE(3)-equivariant if ϕ ( T ◦ X ) = T ◦ ϕ ( X ) , ∀ T ∈ SE(3) , where ◦ means performing an SE(3) transformation. Likewise, SE(3)-invariance is expressed as ϕ ( T ◦ X ) = ϕ ( X ) . To reduce the computational cost of equivariant networks, [12] discretizes the rotation space into an icosahedral group G of 60 rotational angles ( |G| = 60 ). As a further extension, Equivariant Point Network (EPN) [9] operates the SE(3) discretization on point clouds. For a point x ∈ R 3 in the input X , the feature extractor of EPN outputs the per-point representation f ( x ) ∈ R |G|× D , where D is the feature dimension. By definition, f ( x ) is SE(3)-equivariant to its neighbors of the convolution kernel's receptive field:

Rotation equivariance within the icosahedral group: f ( g ◦ x ) = g ◦ f ( x ) , ∀ g ∈ G .

Translation invariance w.r.t. arbitrary translation t : f ( t ◦ x ) = f ( x ) .

By relaxing the strictly rotational equivariance into the equivariance w.r.t. the 60 icosahedral angles in G , EPN is proven to be robust to many downstream tasks, such as 6D pose estimation [43], and place recognition [45]. Due to its high robustness, we choose EPN as our SE(3) equivariant backbone of feature extraction. In principle, our training strategy is versatile, and other networks with per-point SE(3)-equivariant representations can also serve as its feature extractor.

3.2 Problem Statement

We define a set of point clouds P from K frames that are not necessarily consecutive as P = { P 1 , P 2 , · · · , P K -1 , P K } , where each frame P k = { p i k ∈ R 3 | i = 1 , 2 , · · · , N -1 , N } contains N three-dimensional points. All frames of P represent the same target, 1 which is segmented into S independently moving rigid parts. For a given frame P k , the point-part rigid mask is defined as M k ∈ { 0 , 1 } N × S . M is k = 1 indicates that the point p i k belongs to the s th rigid part; M is k = 0 indicates that it does not. The rigid motion of the s th part between two frames ( P k , P l ) is denoted

SE(3)

Equivariant

Network

Invariant

Segmentation

Head

Equivariant Motion Estimation Head

P1

P2

Rotation Outputs

+

Initial Scene Flow

P1

P2

Segmentation Outputs

+

Joint

Optimization

SE(3) Features

= Icosahedronic

Equivariant Feature

as T s kl ∈ SE(3) . This rigid transformation is specified by a rotation matrix R s kl ∈ SO(3) and a translation vector t s kl ∈ R 3 . Note that the targets in the dataset belong to various categories, including classes possibly unseen in the training data.

Given the point cloud set P , we aim to segment each frame P k with rigid-part masks M k , and obtain part-level rigid transformation T s kl between frames. The former task is called rigid segmentation , while the latter is named motion estimation . In the supervised setting, ground-truth segmentation masks and rigid motions (or scene flow) are provided in the training data. In the unsupervised formulation, neither segmentation nor motion information is available, and our training input solely consists of point cloud frames.

3.3 Network Architecture

As shown in Figure 2, our network takes a pair of point cloud frames as the input and outputs the predictions of rigid segmentation and motion estimation. There are three main components in the proposed framework, i.e ., a feature extractor, an invariant segmentation head, and an equivariant motion estimation head.

Per-point Feature Extractor. For an input frame P k , the feature extractor of EPN outputs per-point SE(3)-equivariant representations F k ∈ R N ×|G|× D , following the notations in Section 3.1 and 3.2. The corresponding feature f i k ∈ R |G|× D of a point p i k can be viewed as a concatenation of different representations w.r.t. g j ∈ G over the rotation group dimension:

where θ is a D -dimensional encoder based on a stack of convolution kernels. The prediction module of vanilla EPN is designed for global SE(3)-equivariance for all input points. Differently, our unsupervised multi-body task requires the model's ability to handle part-level local equivariance , especially under low-quality training signals. For this purpose, we further devise two heads for rigid segmentation and motion estimation.

Point-level Invariant Segmentation Head. Rigid segmentation is an SE(3)-invariant task, as the predicted mask should remain consistent for the same points across various poses and positions. Traditional SE(3)-equivariant structures assume that all input points undergo the same rigid transformation, which is not in alignment with the multi-body setting. To encode distinct transformations for individual input points , the equivariant features θ ( p i k , g j ) are aggregated into an invariant representation u i k across the dimension of the rotation group, as depicted in Figure 3:

where w ( p i k , g j ) ∈ [0 , 1] is a selection probability of the discrete rotation g j in G , derived through a 1 × 1 convolution. The weighted sum u i k is invariant to rigid motion given a point with its neighbors

in the convolution receptive field. As Figure 3 illustrates, by fusing such invariant representations u i k (1) ∈ R D , · · · , u i k ( h ) ∈ R D ′ from h layers in the feature extractor, the segmentation head outputs a soft prediction ˆ M k ∈ [0 , 1] N × S of the rigid mask.

Part-level Equivariant Motion Estimation Head. Part-level SE(3)-equivariance is desirable for motion analysis, especially rotation estimation. Based on the noisy predictions ( ˆ M k , ˆ M l ) of the frames ( k, l ) from the head of rigid segmentation, the motion head is supposed to handle these uncertain category-agnostic parts . Figure 3 demonstrates the operational scheme of our motion estimation head. First of all, the part-level SE(3) feature V k,j ∈ R N × D × S w.r.t. the rotation g j ∈ G of a single frame k is obtained from the per-point equivariant representations F k and the predicted rigid mask ˆ M k :

where ˆ m i k is the element corresponding to a point p k i in ˆ M k , and · is the broadcast operation of Hardamard product. Afterward, V k,j over the rotation group G is concatenated as V k ∈ R N ×|G|× D × S , followed by a permutation-invariant operation σ : R N ×|G|× D × S → R |G|× D × S ( e.g ., max pooling) over all the points to produce part-level equivariant features ˜ V k = σ ( V k ) . Between two frames ( k, l ) , the part-level rotation correlated feature C kl ∈ R |G|×|G|× S is defined as:

where T is matrix transposition. C kl is calculated upon 'softly matching' within each consistent rigid part, while the specific category labels can be agnostic to the model. Based on the correlated feature C kl , the motion head estimates rotation ˆ R s kl and translation ˆ t s kl of each rigid part s . More implementation details can be found in Supp .

3.4 Unsupervised Training Strategy

Ideally, the relation among scene flow δ kl ∈ R N × 3 , rigid segmentation M s k of the s th part, and multi-body transformation T s kl is as follows:

wherein the operation of union set is denoted as ⋃ , and ⋆ signifies the removal of a point p k i if it does not in the rigid s . Nevertheless, their predictions ˆ T s kl , ˆ M k and ˆ δ kl may exhibit a considerable amount of interdependent noise during the unsupervised training process. To mitigate the noise in one prediction by leveraging the other two , we propose an online training strategy without interrupting the end-to-end optimization of a network, as shown in Figure 2.

Cold Start & Scene Flow ˆ δ kl Updating. Scene flow creates a relation between motion and segmentation. As a dense vector field that maps points to points, it can directly assist point-level segmentation. At the same time, scene flow also contains movement information for motion estimation. We initialize by collecting noisy scene flow from an unsupervised flow estimator ( e.g ., FlowStep3D [38]), and calculate rudimentary estimates of rigid masks and transformation by minimizing the discrepancy between their derived displacement and scene flow. As the accuracy of their estimates ˆ M s k and ˆ T s kl improves, the scene flow is online corrected during a training epoch:

where α ∈ [0 , 1] is a decay factor to control the updating rate. In this manner, improved scene flow is capable of providing enhanced supervision to learn segmentation masks and motion estimates.

Motion & Flow → Segmentation ˆ M s k . Although scene flow encodes point-level supervisory signals for segmentation, accurately computing flow between non-adjacent frames is challenging, as pointed out by Song and Yang [61]. To alleviate the influence of miscalculated scene flow, we optimize our segmentation head based on the consensus between the motion head and updated scene flow . Given the outputs ˆ R s kl , ˆ t s kl of our motion head, and the interpolation point p ′ i l = p i k + δ i kl derived from scene flow, the consensus score β s ( i ) kl of the point p i k w.r.t. the rigid s between the frames ( k, l ) is defined as:

where τ is a temperature coefficient to set the sharpness of β s ( i ) kl , and || || is ℓ 2 -norm. Intuitively, a low β s ( i ) kl indicates a large disagreement between scene flow and the motion head, of which the mask should have a small learning weight in the segmentation loss:

where ˆ T s kl is computed based on the scene flow and segmentation, as described in the next paragraph.

Segmentation & Flow → Motion ˆ T s kl . Following previous works [32, 61], we employ the weighted-Kabsch algorithm [37, 27] to determine part-level rigid transformation ˆ T s kl given the estimates of scene flow and rigid masks. Further, the motion head is optimized by the rotation component of ˆ T s kl , and estimates corresponding translation by minimizing our probability-based part-level distance. The implementation details of this distance and our motion loss can be found in Supp .

4 Experiments

4.1 Datasets & Metrics

Our model is evaluated on three various application scenarios with four datasets: 1) SAPIEN [68] for articulated objects, 2) OGC-DR and its single-view counterpart OGC-DRSV [61] for furniture arrangements; 3) KITTI-SF [48] for vehicular traffic. Following previous works [61, 32, 51], we evaluate the rigid segmentation performance on the following seven measures: Average Precision ( AP ), Panoptic Quality ( PQ ), F1-score ( F1 ), Precision ( Pre ), and Recall ( Rec ) at an Intersection over Union threshold of 0.5, in addition to the mean Intersection over Union ( mIoU ) and Rand Index ( RI ). In terms of multi-body motion estimation, we report End-Point-Error 3D ( EPE3D ) in the main body, and other metrics ( e.g ., Outlier) are provided in Supp . More information about the datasets and our evaluation protocol can be found in Supp .

4.2 Pilot Studies on The Two Heads

We first conduct experiments on SAPIEN to verify the generalizability of our two-head structure, including the point-level invariance of the segmentation head and the part-level equivariance of the motion head.

50

55

60

65

70

75

61.8

72.0

Ours

Train

Val

55.5

61.9

KPConv

Train

Val

6.4

10.2

58.1

65.7

EPN

Val

7.6

Train

(b) Recall@50

5.69

2.85

20.77

4.61

0

5

10

15

20

25

w/o Part-level Equivariance

Ours

Train

Train

Val

Val

15.08

1.76

°

(c) Mean Angle Error

40

45

50

55

60

65

70

75

80

KPConv

Train

Val

71.9

48.6

23.3

EPN

Val

Train

73.7

56.2

17.5

Ours

Train

Val

74.9

61.7

13.2

(a) Prec@50

%

%

Can the point-level invariance of our segmentation head help generalize to open-set motion?

On SAPIEN, each articulated target has four frames with various part-level rigid orientations and locations. By training the segmentation head on a subset of the 1 st frames (with the canonical pose in [32]) and testing them on all frames, we compare its performance with the segmentation results using the same parameters of the non-equivariant counterpart KPConv [62] and global equivariant EPN. As shown in Figure 4(a)(b), the SE(3)-equivariance can essentially boost the generalization even given the global since there are some targets that only contain two rigid parts. By introducing point-level invariance, both precision and recall can be further improved.

Can the part-level equivariance of our motion head help generalize to category-agnostic parts?

Note that the category of targets for training, validation, and testing on SAPIEN is completely disjoint. By optimizing the motion head on the training set and testing its performance on the validation data, we ensure that the targets do not overlap in categories so their parts are entirely category-agnostic. To exclude the interference of part segmentation, we directly provide the ground-truth rigid partition as the mask. By comparing the decrease in performance on the validation data when using part-level equivariance vs . not using it, as illustrated in Figure 4(c), it is observed that the mean angular error of our model only increases by less than 2 °, while that without part-level equivariance surges from 5 . 69 ° to 20 . 77 °. This demonstrates the strong generalizability of our model to category-agnostic parts.

4.3 Ablation Studies

We perform detailed ablation studies on SAPIEN to explore the effects of three main components: segmentation head, motion head, and scheme of scene flow updating.

Segmentation Head. Two crucial design elements significantly enhance our segmentation head: 1) the use of SE(3)-equivariant features, and 2) point-level flexibility for invariant representations. By simply introducing the global equivariant features, AP increases from 45 . 2% to 51 . 7% . This demonstrates the importance of SE(3)-equivariance in modeling rigid transformations. The proposed point-level flexibility further boosts the performance to 55 . 3% , suggesting that our point-level invariance is highly advantageous to the multi-body task.

Scene Flow Updating Scheme. For the updated choice of scene flow, we empirically test three options: 1) completely relying on the past flow extracted by an unsupervised flow network, 2) ignoring the past flow and directly using the current estimation, and 3) combining them together with a decay

GT

Ours-unsup

OGC-sup

OGC

MBS

factor. It is observed that the extreme choices of 1) or 2) result in minor performance differences (AP: 55 . 3% vs . 54 . 8% ). The significant improvement comes from the weighted combination in option 3), achieving the AP of 57 . 0% .

Motion Head. The motion head contributes to the consensus score for the removal of scene flow noise. From the last row of Table 1, the consensus between the motion head and scene flow significantly boosts the final segmentation performance in most of the metrics. By controlling the point-level learning weight during the training process, our motion head can assist in optimizing the segmentation head.

4.4 Results & Comparisons

4.4.1 SAPIEN

SAPIEN is a challenging dataset of articulated objects since its training, validation and testing sets comprise completely disjoint categories of objects. This domain gap in the data split complicates the precise segmentation of rigid parts. Adapting our model to the supervised formulation is straightforward by optimizing the segmentation with ground-truth training labels. Therefore, we also report the supervised performance of the proposed framework. As shown in Table 2, our segmentation performance sets a new benchmark across all metrics in both supervised and unsupervised settings.

In line with the default setting of [32], we estimate motion between both consecutive frames and nonadjacent ones. Table 2 reports its EPE3D performance and comparison: our unsupervised performance of 5 . 47 EPE3D on motion estimation is on par with the best existing supervised method (MBS), while our supervised EPE3D achieves 3 . 86 , marking a relative error reduction of approximately 23% .

As depicted in Figure 1, owing to the lightweight two-head structure, our model encompasses only a small number of parameters (0.25M) and incurs a low cost in computational complexity (0.92G floating point operations, FLOPs). We also show a sneak peek of the qualitative results (Figure 5). Our unsupervised method is comparable (and often times better) than other supervised methods. More results are shared in the Supp .

4.4.2 OGC-DR & Single-view Counterpart

OGC-DR is a dataset for indoor furniture arrangements, where the training, validation, and testing instances are distinct from one another. OGC-DRSV, the single-view version of OGC-DR, presents a challenge due to the incomplete nature of the furniture caused by occlusion, making it difficult to identify consistent rigidity. As demonstrated in Table 3, the proposed framework constantly surpasses the state-of-the-art models across all measures. Remarkably, even under the most challenging condition, i.e ., unsupervised single-view, our AP still achieves the value of 88 . 1% . This underscores the robustness of our model in handling incomplete observations. The performance on motion estimation can be found in Supp .

4.4.3 KITTI-SF

Strictly speaking, KITTI-SF is not a multi-body rigid dataset, as the background in its point clouds may be deformable. This is inconsistent with the assumption of our framework, which posits that the feature is only equivariant for rigid transformations. However, as shown in Table 4, our framework still delivers competitive performance in rigid segmentation despite this inconsistency.

5 Conclusion

This paper introduces a part-level SE(3)-equivariant framework for modeling multi-body rigid motion. The two heads for rigid segmentation and motion estimation are meticulously designed based on their inherent invariant and equivariant characteristics. The relationship among scene flow, rigid segmentation, and multi-body transformation is then exploited to derive an unsupervised optimization strategy. Our approach achieves state-of-the-art results on multiple datasets while significantly reducing the required parameters and computations.

Limitations & Future Work. Our model is predicated on an assumption: the part-level motion should be a rigid transformation. Therefore, as shown in Table 3, if the observation of part-level movements is non-rigid (such as occluded single views), it would suffer a performance decrease. In the future, we would like to develop a model that is more robust to its observation.

Acknowledgements. Our research is supported by Amazon Web Services in the Oxford-Singapore Human-Machine Collaboration Programme and by the ACE-OPS project (EP/S030832/1). We are grateful to all of the anonymous reviewers for their valuable comments.

References

S. A. Baur, D. J. Emmerichs, F. Moosmann, P. Pinggera, B. Ommer, and A. Geiger, 'Slim: Self-supervised lidar scene flow and motion segmentation,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 13 126-13 136.

A. Behl, D. Paschalidou, S. Donne, and A. Geiger, 'Pointflownet: Learning representations for rigid motion estimation from point clouds,' in Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , June 2019.

J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall, 'Semantickitti: A dataset for semantic scene understanding of lidar sequences,' in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 9297-9307.

T. Birdal and S. Ilic, 'Cad priors for accurate and flexible instance reconstruction,' in Proceedings of the IEEE international conference on computer vision , 2017, pp. 133-142.

J. Brandstetter, R. Hesselink, E. van der Pol, E. J. Bekkers, and M. Welling, 'Geometric and physical quantities improve e (3) equivariant message passing,' in International Conference on Learning Representations , 2021.

H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, 'nuscenes: A multimodal dataset for autonomous driving,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 11 621-11 631.

H. Cao, Y. Lu, C. Lu, B. Pang, G. Liu, and A. Yuille, 'Asap-net: Attention and structure aware point cloud sequence segmentation,' in British Machine Vision Conference (BMVC) , 2020.

E. Chatzipantazis, S. Pertigkiozoglou, E. Dobriban, and K. Daniilidis, 'Se (3)-equivariant attention networks for shape reconstruction in function space,' arXiv preprint arXiv:2204.02394 , 2022.

H. Chen, S. Liu, W. Chen, H. Li, and R. Hill, 'Equivariant point network for 3d point cloud analysis,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 14 51414 523.

Y. Chen, M. Li, L. Lan, H. Su, Y. Yang, and C. Jiang, 'A unified newton barrier method for multibody dynamics,' ACM Transactions on Graphics (TOG) , vol. 41, no. 4, pp. 1-14, 2022.

W. Cheng and J. H. Ko, 'Bi-pointflownet: Bidirectional learning for point cloud based scene flow estimation,' in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVIII . Springer, 2022, pp. 108-124.

T. Cohen, M. Weiler, B. Kicanaoglu, and M. Welling, 'Gauge equivariant convolutional networks and the icosahedral cnn,' in International conference on Machine learning . PMLR, 2019, pp. 1321-1330.

T. Cohen and M. Welling, 'Group equivariant convolutional networks,' in International conference on machine learning . PMLR, 2016, pp. 2990-2999.

T. S. Cohen, M. Geiger, J. Köhler, and M. Welling, 'Spherical cnns,' in International Conference on Learning Representations , 2018.

T. S. Cohen and M. Welling, 'Steerable cnns,' in International Conference on Learning Representations , 2016.

A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, 'Scannet: Richly-annotated 3d reconstructions of indoor scenes,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 5828-5839.

C. Deng, O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and L. J. Guibas, 'Vector neurons: A general framework for so (3)-equivariant networks,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 12 200-12 209.

L. Ding, S. Dong, T. Xu, X. Xu, J. Wang, and J. Li, 'Fh-net: A fast hierarchical network for scene flow estimation on real-world point clouds,' in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX . Springer, 2022, pp. 213-229.

M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, 'A density-based algorithm for discovering clusters in large spatial databases with noise,' in Proc. 2nd Int. Conf. on Knowledge Discovery and Data Mining (KDD'96 ) , E. Simoudis, J. Han, and U. Fayyad, Eds. AAAI Press, 1996, pp. 226-231.

H. Fan, Y. Yang, and M. Kankanhalli, 'Point 4d transformer networks for spatio-temporal modeling in point cloud videos,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 14 204-14 213.

--, 'Point spatio-temporal transformer networks for point cloud video modeling,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 2, pp. 2181-2192, 2022.

H. Fan, X. Yu, Y. Ding, Y. Yang, and M. Kankanhalli, 'Pstnet: Point spatio-temporal convolution on point cloud sequences,' in International Conference on Learning Representations , 2021.

H. Fan, X. Yu, Y. Yang, and M. Kankanhalli, 'Deep hierarchical representation of point cloud videos via spatio-temporal decomposition,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 12, pp. 9918-9930, 2021.

H. Feng, P. Kulits, S. Liu, M. J. Black, and V. Abrevaya, 'Generalizing neural human fitting to unseen poses with articulated se (3) equivariance,' arXiv preprint arXiv:2304.10528 , 2023.

A. Geiger, M. Lauer, C. Wojek, C. Stiller, and R. Urtasun, '3D Traffic Scene Understanding From Movable Platforms,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 36, no. 5, pp. 1012-1025, 2014.

S. Giancola, J. Zarzar, and B. Ghanem, 'Leveraging shape completion for 3d siamese tracking,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 13591368.

Z. Gojcic, C. Zhou, J. D. Wegner, L. J. Guibas, and T. Birdal, 'Learning multiview 3d point cloud registration,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 1759-1769.

D. S. Hayden, J. Pacheco, and J. W. Fisher, 'Nonparametric object and parts modeling with lie group dynamics,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 7426-7435.

Y. He, L. Chen, J. Xie, and L. Chen, 'Learning 3D Semantics From Pose-Noisy 2D Images with Hierarchical Full Attention Network,' in Computer Vision - ECCV 2022 Workshops . Springer Nature Switzerland, 2023, pp. 726-742.

Q. Hu, B. Yang, S. Khalid, W. Xiao, N. Trigoni, and A. Markham, 'Towards semantic segmentation of urban-scale 3d point clouds: A dataset, benchmarks and challenges,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 4977-4987.

Q. Hu, B. Yang, L. Xie, S. Rosa, Y. Guo, Z. Wang, N. Trigoni, and A. Markham, 'Randla-net: Efficient semantic segmentation of large-scale point clouds,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 11 108-11 117.

J. Huang, H. Wang, T. Birdal, M. Sung, F. Arrigoni, S.-M. Hu, and L. J. Guibas, 'Multibodysync: Multibody segmentation and motion estimation via 3d scan synchronization,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 7108-7118.

J. Huang, S. Yang, Z. Zhao, Y.-K. Lai, and S.-M. Hu, 'Clusterslam: A slam backend for simultaneous rigid body clustering and motion estimation,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 5875-5884.

S. Huang, Z. Gojcic, J. Huang, A. Wieser, and K. Schindler, 'Dynamic 3d scene analysis by point cloud accumulation,' in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVIII . Springer, 2022, pp. 674-690.

H. Isack and Y. Boykov, 'Energy-based geometric multi-model fitting,' International journal of computer vision , vol. 97, no. 2, pp. 123-147, 2012.

B. Jiang, Y. Zhang, X. Wei, X. Xue, and Y. Fu, 'Learning compositional representation for 4d captures with neural ode,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 5340-5350.

W. Kabsch, 'A solution for the best rotation to relate two sets of vectors,' Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography , vol. 32, no. 5, pp. 922-923, 1976.

Y. Kittenplon, Y. C. Eldar, and D. Raviv, 'FlowStep3D: Model Unrolling for Self-Supervised Scene Flow Estimation,' IEEE/CVF International Conference on Computer Vision (CVPR) , 2021.

F. Kluger, E. Brachmann, H. Ackermann, C. Rother, M. Y. Yang, and B. Rosenhahn, 'Consac: Robust multi-model fitting by conditional sample consensus,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 4634-4643.

A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, 'Pointpillars: Fast encoders for object detection from point clouds,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 12 697-12 705.

J. Lei, C. Deng, K. Schmeckpeper, L. Guibas, and K. Daniilidis, 'Efem: Equivariant neural field expectation maximization for 3d object segmentation without scene supervision,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 4902-4912.

T. Li, V. Kallem, D. Singaraju, and R. Vidal, 'Projective factorization of multiple rigid-body motions,' in 2007 IEEE Conference on Computer Vision and Pattern Recognition . IEEE, 2007, pp. 1-6.

X. Li, Y. Weng, L. Yi, L. J. Guibas, A. Abbott, S. Song, and H. Wang, 'Leveraging se (3) equivariance for self-supervised category-level object pose estimation from point clouds,' Advances in Neural Information Processing Systems , vol. 34, pp. 15 370-15 381, 2021.

X. Li, Q. Huang, Z. Wang, Z. Hou, and T. Yang, 'Sequentialpointnet: A strong parallelized point cloud sequence network for 3d action recognition,' arXiv preprint arXiv:2111.08492 , 2021.

C. E. Lin, J. Song, R. Zhang, M. Zhu, and M. Ghaffari, 'Se(3)-equivariant point cloud-based place recognition,' in Proceedings of The 6th Conference on Robot Learning , ser. Proceedings of Machine Learning Research, K. Liu, D. Kulic, and J. Ichnowski, Eds., vol. 205. PMLR, 14-18 Dec 2023, pp. 1520-1530.

X. Liu, M. Yan, and J. Bohg, 'Meteornet: Deep learning on dynamic 3d point cloud sequences,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 9246-9255.

L. Magri and A. Fusiello, 'Fitting multiple heterogeneous models by multi-class cascaded t-linkage,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 7460-7468.

M. Menze and A. Geiger, 'Object scene flow for autonomous vehicles,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 3061-3070.

T. W. Mitchel, N. Aigerman, V. G. Kim, and M. Kazhdan, 'Möbius convolutions for spherical cnns,' in ACM SIGGRAPH 2022 Conference Proceedings , 2022, pp. 1-9.

M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger, 'Occupancy flow: 4d reconstruction by learning particle dynamics,' in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 5379-5389.

U. M. Nunes and Y. Demiris, '3d motion segmentation of articulated rigid bodies based on RGB-D data,' in British Machine Vision Conference 2018, BMVC 2018, Newcastle, UK, September 3-6, 2018 . BMVA Press, 2018, p. 255.

P. Ochs, J. Malik, and T. Brox, 'Segmentation of Moving Objects by Long Term Video Analysis,' IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) , 2014.

O. Puny, M. Atzmon, E. J. Smith, I. Misra, A. Grover, H. Ben-Hamu, and Y. Lipman, 'Frame averaging for invariant and equivariant network design,' in International Conference on Learning Representations , 2022.

C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, 'Frustum pointnets for 3d object detection from rgb-d data,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 918-927.

C. R. Qi, L. Yi, H. Su, and L. J. Guibas, 'Pointnet++: Deep hierarchical feature learning on point sets in a metric space,' Advances in neural information processing systems , vol. 30, 2017.

H. Qi, C. Feng, Z. Cao, F. Zhao, and Y. Xiao, 'P2b: Point-to-box network for 3d object tracking in point clouds,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 6329-6338.

D. Rempe, T. Birdal, Y. Zhao, Z. Gojcic, S. Sridhar, and L. J. Guibas, 'Caspr: Learning canonical spatiotemporal point cloud representations,' Advances in neural information processing systems , vol. 33, pp. 13 688-13 701, 2020.

S. Shi, X. Wang, and H. Li, 'Pointrcnn: 3d object proposal generation and detection from point cloud,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 770-779.

Y. Shi, X. Cao, F. Lu, and B. Zhou, 'Pˆ 3-net: Part mobility parsing from point cloud sequences via learning explicit point correspondence,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 2, 2022, pp. 2244-2252.

Y. Shi, X. Cao, and B. Zhou, 'Self-supervised learning of part mobility from point cloud sequence,' in Computer Graphics Forum , vol. 40, no. 6. Wiley Online Library, 2021, pp. 104-116.

Z. Song and B. Yang, 'Ogc: Unsupervised 3d object segmentation from rigid dynamics of point clouds,' in Advances in Neural Information Processing Systems , 2022.

H. Thomas, C. R. Qi, J.-E. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas, 'Kpconv: Flexible and deformable convolution for point clouds,' in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 6411-6420.

H. Wang, Y. Liu, Z. Dong, and W. Wang, 'You only hypothesize once: Point cloud registration with rotation-equivariant descriptors,' in Proceedings of the 30th ACM International Conference on Multimedia , 2022, pp. 1630-1641.

K. Wang and S. Shen, 'Estimation and propagation: Scene flow prediction on occluded point clouds,' IEEE Robotics and Automation Letters , vol. 7, no. 4, pp. 12 201-12 208, 2022.

L. Wang, H. Liu, Y. Liu, J. Kurtin, and S. Ji, 'Learning hierarchical protein representations via complete 3d graph networks,' in The Eleventh International Conference on Learning Representations , 2022.

J. H. Ward, 'Hierarchical grouping to optimize an objective function,' Journal of the American Statistical Association , vol. 58, no. 301, pp. 236-244, 1963.

M. Weiler, F. A. Hamprecht, and M. Storath, 'Learning steerable filters for rotation equivariant cnns,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 849-858.

F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al. , 'Sapien: A simulated part-based interactive environment,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 11 097-11 107.

X. Xu, L.-F. Cheong, and Z. Li, '3d rigid motion segmentation with mixed and unknown number of models,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 43, no. 1, pp. 1-16, 2019.

B. Yang, J. Wang, R. Clark, Q. Hu, S. Wang, A. Markham, and N. Trigoni, 'Learning object bounding boxes for 3d instance segmentation on point clouds,' Advances in neural information processing systems , vol. 32, 2019.

L. Yi, H. Huang, D. Liu, E. Kalogerakis, H. Su, and L. Guibas, 'Deep part induction from articulated object pairs,' ACM Transactions on Graphics (TOG) , vol. 37, no. 6, pp. 1-15, 2018.

T. Yin, X. Zhou, and P. Krahenbuhl, 'Center-based 3d object detection and tracking,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 11 784-11 793.

H.-X. Yu, J. Wu, and L. Yi, 'Rotationally equivariant 3d object detection,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 1456-1464.

J.-X. Zhong, K. Zhou, Q. Hu, B. Wang, N. Trigoni, and A. Markham, 'No pain, big gain: classify dynamic point cloud sequences with static models by fitting feature-level space-time surfaces,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 8510-8520.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a method to simultaneously segment rigid parts and estimate part-level transformations for multi-body point cloud inputs. Given the input point cloud, equivariant features are first extracted and aggregated to invariant features for part segmentation. To estimate the transformations between two point clouds, an equivariant motion estimation head is applied to pairs of point clouds to obtain SE(3) motions. The entire model is trained in an unsupervised fashion using the consensus between 3D scene flow, motion segmentation, and transformations. Experiments show that the model is generalizable in two aspects: open-set motion and category-agnostic part segmentation. The authors test the model on SAPIEN, OGC, and KITTI datasets, and show superior performance in comparison to past baselines, with much better accuracy and much fewer FLOPs, even surpassing the supervised method.\n\nSTRENGTHS:\n1. The identification of two generalization problems, i.e. invariant segmentation and equivariant motion estimation, is insightful. The use of equivariant backbones and their invariant counterpart is well-suited for such a problem.\n2. The use of initial scene flow to bootstrap (or cold-start) the learning process and to get rid of local minima is novel. By utilizing the consistency between the output attributes which is inherent in the multi-body registration task, the method could be learned in an unsupervised manner.\n3. Experiments are extensive and the results are compelling. The method is tested on four different datasets ranging from articulated objects to outdoor driving scenes, justifying the wide applicability. In the meanwhile, the fact that the algorithm is able to achieve a much higher accuracy with a much lower cost is impressive and set up a strong waypoint for future works.\n\nWEAKNESSES:\n1. The unsupervised training strategy lacks clarity. What is the loss function? Is it eq.(8) plus eq.(7) in the supplementary with some balancing factor? This should be clarified in the main paper. In Line 213, the authors mention the weighted-Kabsch algorithm, what is the 'weight' here?\n2. The generalization of the segmentation head is questionable -- this is my most important doubt about the paper. Although the generalization to category-agnostic part segmentations is demonstrated through experiments, the reason why an equivariant network backbone could achieve that is not explained. What adds to the confusion is that the method is trained in an unsupervised manner, which means that no canonical segmentation labels are defined. Given a novel multi-body shape that is not seen from the training set (e.g. a cabinet with four drawers), how could the model give a unique index to each of the drawers in a consistent fashion (i.e. the drawer index stays unchanged across multiple articulation states)? What is the maximum number of parts supported in the algorithm? Is it predefined or optimized/given for each test sample? The doubt also links to the experiment section -- In Fig.4 (a,b), the gap between training and validation is not shown, and it's unclear whether Fig.4 (c) is based on ground truth or predicted segmentation.\n3. The experiments are not explained very clearly. In Sec4.2, why 'point-level invariance of segmentation' helps 'open-set motion' generalization, and why 'part-level equivariant of motion' helps 'category-agnostic part' generalization? Shouldn't they be swapped? It is preferable to move Sec4.4 ahead of Sec4.3.\n4. More visualizations and challenging cases should be shown. There are only 3 visualizations provided in the paper+supplementary, on SAPIEN datasets. These visualizations only contain segmentation labels with two parts. To make the results more convincing, the author should provide more visualizations on all the datasets they've tested on, with transformations shown, and on more challenging cases, e.g., large scenes with >4 rigid parts.\n\nQUESTIONS:\n1. Does the current training scheme leverage the cycle consistency among multiple inputs? If not how could this be possibly incorporated into the algorithm?\n2. How does the method generalize to different point densities/distributions? What is the maximum number of points could the method process?\n3. Some typos:\n- Line 144, 'There are mainly comprised of...'\n- Line 237, 'even given the global since...'\n\nLIMITATIONS:\nLimitations are discussed in the last paragraph of the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of rigid part segmentation and motion segmentation. The proposed network first uses a global SE(3) equivariant backbone to extract point features, then a part segmentation head will predict invariant zero-order segmentation and a motion head will predict one-order transformations. The network can be trained on pairs of point clouds before and after deforming with noisy scene flow. \n\nThe multibody equivariance is approximate because of 1.) descretization in EPN, 2.) the global backbone can’t handle deformation.\n\nSTRENGTHS:\n- The paper exploits strong geometry inductive bias — Equivariance, into the segmentation problem, which should be highlighted and emphasized more to the community.\n- The system is tested across different settings from articulated objects to outdoor scenes and the experiments are convincing.\n\nWEAKNESSES:\n- The main weakness of this paper is that the “equivariance” is not exact, i.e. because there are pooling and information aggregation in the point cloud feature backbone when the target is deformed (not necessarily non-rigid, but rigid transformed, articulated, this is a word from equivariant learning theory), the overall point cloud changes in such cases, but the information can be propagated across rigid parts so the features of each point do not transform purely following its parts transformation. This is because without knowing the mask, the group action of SE(3)xSE(3)x…xSE(3) on the point cloud can not be defined.\n- Another weakness of this paper is that the system should take in at least a pair of point as input during inference (as far as the reviewer currently understand, please correct him if this is wrong). So the task is relatively easy because by comparing two point clouds (even by brutal-force enumerating and classical geometric algorithms ) the segmentation of rigid moved parts may be trivially found in most cases. And the system seems only to focus on moving parts rather than semantic or instance parts that may not be moved.\n- The reviewer is also curious why the author chooses to use EPN as the backbone, is there a specific reason for such a design? Note EPN is only approximate equivariance due to the discretization. In other words, will such a framework still work for other backbones like TFN or VNN?\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nThe limitation is explicitly highlighted in the paper, but the reviewer hasn't found the social impact claim.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a unified framework to simultaneously handle rigid segmentation and motion estimation. Specifically, an probabilistic SE(3)-equivariant network is presented to obtain transformation-invariant features, which is followed by two heads for rigid segmentation and motion estimation. Due to the interdependent nature between the above two tasks, the authors utilize a unified training strategy to jointly optimize the two heads. Extensive experiments on four datasets show that this method can yield good performance with low computational complexity.\n\nSTRENGTHS:\n-Significance and originality\n\nUnsupervised rigid segmentation and motion estimation are challenging. Taking the generalization requirement into account, the authors introduce a probabilistic part-level SE(3)-equivariant feature encoder to capture transformation-invariant features, which is different from the global SE(3) equivariance. And it is simple but makes sense.\nA unified training strategy to jointly optimize the outputs of two heads in an online fashion is well motivated. Although many other tasks also employ joint optimization, this is the first one for this task.\n\n-Clarity\n\nThe paper contains enough details and definitions of the proposed contributions.\n\n-Experiment\n\nQuantitative results show a clear improvement compared to previous methods.\n\nWEAKNESSES:\n-Time cost of the proposed methods and other methods.\n\n-The author should systematically analyze the robustness on initial scene flow. Does the performance drop a lot with poor initial scene flow results? (add some noise or with insufficiently trained scene flow results).\n\n-This paper relies on rigidity constraint and works well on finite datasets. The SAPIEN and OGC-DR datasets are without background and well-separated. The authors can conduct some experiments on large outdoor scenes, which are more complicated.\n\nQUESTIONS:\nplease see the weakness.\n\nLIMITATIONS:\n-In related works, deep learning on dynamic point clouds is too highly summarized. The authors can present more details.\n\n-The authors can visualize more segmentation results and intermidiate results so that the proposed method can be better understood.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a part-level SE(3)-equivariant framework for multi-body rigid motion modeling. The two heads that take in the SE(3) features, segmentation and motion estimation, are carefully crafted based on their inherit invariant and equivariant characteristics. The relationship of scene flow, rigid segmentation, and multi-body transformation is then exploited to derive an unsupervised optimization strategy. It achieves state-of-the-art in multiple datasets while significantly reducing the parameters and training computations required.

STRENGTHS:
The structure of this paper is very clear, providing a comprehensive explanation of the research problem, motivation, background, principles, methods, and experiments. The text and figures have been meticulously polished.

This work effectively utilizes SE(3) equivariance in rigid segmentation and multi-body motion estimation, and is outstanding in terms of experimental results and model size.

WEAKNESSES:
Rigid segmentation and multi-body motion estimation are intrinsically coupled. I wonder if this coupling might also have negative effects on the model, for instance, inaccurate segmentation leading to poorer scene flow estimation results.

QUESTIONS:
Please refer to the weaknesses section, refute views or answer questions or improve the paper.

LIMITATIONS:
The limitations has been discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
Modeling multi-body rigid movements involves - rigid segmentation which is category-agnostic and motion estimation which is open-set. This work proposes a part-level SE(3) equivariant network that takes in point-clouds corresponding to video frames and can effectively estimate category-agnostic open-set motion.

STRENGTHS:
The work leverages the point level feature equivariance from existing works and uses it to obtain invariant representations for the rigid segmentation and then further uses it to estimate motion using an equivariant motion head.  

The approach is able to provide state of the art performance with much lesser flops.

It makes note of the interdependence between the tasks of rigid segmentation and motion estimation and uses that interdependence for unsupervised training.

WEAKNESSES:
The paper writing could be better so that the high-level idea is more easily understood. The contributions of the work could be summarized in fewer words in the introduction section making the work much easier to follow. 

SE(3) has been used multiple times in the paper including the title but it has not been described in the paper anywhere.

QUESTIONS:
Please address the issues mentioned in the weakness section

LIMITATIONS:
The authors have discussed limitations of their proposed approach.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method that estimates the part segmentation, and the rigid motions of the parts, for an articulated object (with multiple rigid parts), given as input a sequence of point clouds of that object. The method works by first computing SE3-equivariant features for the points, and then processing with an SE3-invariant head for segmentation estimation (pooling across rotation groups), and then processing with an SE3-equivariant head for rigid motion estimation (pooling across points within estimated segments). The method also initializes a scene flow estimate from a pre-trained off-the-shelf model, and then iteratively updates this to match the estimated rigid motion. Finally, the model uses the weighted-Kabsch algorithm to estimate transformations for the parts according to the flow and segmentation, which yields optimization targets that encourage the estimates to be self-consistent.\n\nSTRENGTHS:\nThis paper makes a well-motivated case for using a combination of equivariant and invariant features to jointly solve rigid part segmentation and motion estimation. The proposed network has fewer parameters and runs faster than the baselines, and also achieves higher accuracy, in both the supervised setting and the unsupervised setting. Jointly optimizing the motion, segmentation, and scene flow, makes sense.\n\nWEAKNESSES:\nI found this paper very difficult to follow. I am quite sure that the content in the main text is not a complete description of a reproducible method. Nearly every section defers to the supplementary material for additional information. In my reading of the guidelines, the supplementary material is for supporting the paper, not for describing key details of the method. The method in the paper should make sense on its own.\n\n Because my main complaint is on clarity, I'll put a lot of questions, but please interpret some of the questions as reflecting weaknesses, because these are things that should be clear in the work.\n\nQUESTIONS:\nDo the two point clouds correspond? I suppose not, because then motion can be computed just from taking a difference of the positions.\n\n\nThe problem statement in 3.2 suggests a sequence, but later (e.g., in 3.3) it seems like the sequence length might just be 2. Is the sequence length ever not 2?\n\n\nThe w(p^i_k,g_j) used in equation 2 are described as \"derived through a 1x1 convolution\". What does this mean exactly?\n\nThe paper talks about \"fusing such invariant representations u^i_{k(1)},... from h layers\" -- how is this fusing done exactly?\n\nThe paper says \"the segmentation head outputs a probabilistic prediction M^k of the rigid mask\" -- what makes it probabilistic exactly? This idea of \"probabilistic\" masks is repeated many times through the paper, and at some point there are also \"probabilistic SE3 features\" (actually only in Figure 3). In my experience, the word \"probabilistic\" is usually used to indicate something non-deterministic, but in this case it seems like it might actually be deterministic. What exactly is meant by a probabilistic mask or probabilistic feature? \n\nThe paper says \"the motion head is supposed to *handle these uncertain category-agnostic parts*\" (italics in the paper). Why are those words in italics? What is meant by \"handle\" here? It seems like a special usage of the word. \n\nEquation 3 looks like it might just be depicting an element-wise product. Is this correct? It would be great to write this more efficiently. \n\nThe paper says \"the specific category labels can be agnostic to the model\". What does it mean for category labels to be agnostic to a model? \n\nThe paper says \"Based on the correlated feature Ckl, the motion head estimates rotation R^s_{kl} and translation t^s_{kl} of each rigid part s.\" This is a critical part of the model. How are the correlations used exactly, what is the predictor, how are the rotation and translation outputs represented, and how are the outputs supervised?\n\nUnder equation 6, it says \"In this manner, improved scene flow is capable of providing enhanced supervision to learn segmentation masks and motion estimates.\" I don't see how this follows from the equation. The equation only produces an updated scene flow, using a convex combination of the previous estimate and the rigid estimate. I see no \"supervision\" (or \"enhanced supervision\") applied to segmentation masks here.\n\nIn the final paragraph of the method, it says \"the motion head is optimized by the rotation component of T^s_{kl}, and estimates corresponding translation by minimizing our probabilistic part-level distance.\" What does it mean to optimize the motion head by rotation? What does it mean to estimate translation by minimizing \"probabilistic part-level distance\"? It seems like these are critical details in the procedure.\n\nLIMITATIONS:\nLooks OK", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-22"}