Title: Banana: Banach Fixed-Point Network for Pointcloud Segmentation with Inter-Part Equivariance
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3D deep learning

equivariant network

pointcloud segmentation

multi-body system
Abstract: Equivariance has gained strong interest as a desirable network property that inherently ensures robust generalization. However, when dealing with complex systems such as articulated objects or multi-object scenes, effectively capturing inter-part transformations poses a challenge, as it becomes entangled with the overall structure and local transformations. The interdependence of part assignment and per-part group action necessitates a novel equivariance formulation that allows for their co-evolution. In this paper, we present Banana, a Banach fixed-point network for equivariant segmentation with inter-part equivariance by construction. Our key insight is to iteratively solve a fixed-point problem, where point-part assignment labels and per-part SE(3)-equivariance co-evolve simultaneously. We provide theoretical derivations of both per-step equivariance and global convergence, which induces an equivariant final convergent state. Our formulation naturally provides a strict definition of inter-part equivariance that generalizes to unseen inter-part configurations. Through experiments conducted on both articulated objects and multi-object scans, we demonstrate the efficacy of our approach in achieving strong generalization under inter-part transformations, even when confronted with substantial changes in pointcloud geometry and topology.

Banana : Bana ch Fixed-Point N etwork for Pointcloud Segmentation with Inter-Part Equiv a riance

Abstract

Equivariance has gained strong interest as a desirable network property that inherently ensures robust generalization. However, when dealing with complex systems such as articulated objects or multi-object scenes, effectively capturing interpart transformations poses a challenge, as it becomes entangled with the overall structure and local transformations. The interdependence of part assignment and per-part group action necessitates a novel equivariance formulation that allows for their co-evolution. In this paper, we present Banana , a Banach fixed-point network for pointcloud segmentation with inter-part equivariance by construction . Our key insight is to iteratively solve a fixed-point problem, where pointpart assignment labels and per-part SE(3) -equivariance co-evolve simultaneously. We provide theoretical derivations of both per-step equivariance and global convergence, which induces an equivariant final convergent state. Our formulation naturally provides a strict definition of inter-part equivariance that generalizes to unseen inter-part configurations. Through experiments conducted on both articulated objects and multi-object scans, we demonstrate the efficacy of our approach in achieving strong generalization under inter-part transformations, even when confronted with substantial changes in pointcloud geometry and topology.

1 Introduction

From articulated objects to multi-object scenes, multi-body systems, i.e. shapes composed of multiple parts where each part can be moved separately, are prevalent in various daily-life scenarios. However, modeling such systems presents considerable challenges compared to rigid objects, primarily due to the infinite shape variations resulting from inter-part pose transformations with exponentially growing complexities. Successfully modeling these shapes demands generalization across potential inter-part configurations. While standard data augmentation techniques can potentially alleviate such problems, exhaustive augmentation can be highly expensive especially given the complexity of such systems compared to rigid objects. Meanwhile, recent rigid-shape analysis techniques have made significant progress in building generalization through the important concept of equivariance [39, 14, 79, 2, 81, 71, 20, 11, 15, 3, 36, 42, 55]. Equivariance dictates that when a transformation is applied to the input data, the network's output should undergo a corresponding transformation. For example, if a is rotated by 90 degrees, the segmentation masks should also rotate accordingly. However, to the best of our knowledge, no existing work has managed to extend the formulation of equivariance to inter-part configurations.

In this paper, we address the challenge of achieving inter-part equivariance in the context of part segmentation, a fundamental task in analysis, which is also the key to generalizing equivaraince from single object to multi-body system. Extending equivariance to multi-body systems presents notable challenges in both formulation and realization. When dealing with a shape that allows for inter-part motions, the model must exhibit equivariance to both global and local state changes, which can only be defined by combining part assignment and per-part transformations. For example, to model an

Equivariance

rigid trans.

per-part trans.

for

X

T E SE(3)

X'

X

Ay E SE(3)P

X'

segmentation?

equiv.to

seg

seg

seg

seg

equiv.to

X

rigid trans.

per-part trans.

T E SE(3)

Ay E SE(3)P

equiv.to

y

Segmentation

defines

equivariance

Globalequivariance

Inter-partequivariance

oven's inter-part states, we first need to differentiate between the parts corresponding to the door and the body and then define the movement of each individual part. An intriguing "chicken or the egg" problem arises when attempting to build inter-part equivariance without access to a provided segmentation, where segmentation is necessary to define equivariance, but segmentation itself is the desired output that we aim to produce.

Our key insight to tackle this seeming dilemma is to model inter-part equivariance as a sequential fixed-point problem by co-evolving part segmentation labels and per-part SE(3) -equivariance. We provide theoretical derivations for both the per-step behavior and global convergence behavior, which are crucial aspects of a fixed-point problem. We show that our formulation establishes per-step equivariance through network construction, which then induces an overall inter-part equivariance upon convergence. Thus, by having equivariant per-step progression and global convergence, our formulation naturally gives rise to a strict definition of inter-part equivariance through iterative inference that generalizes to unseen part configurations. We further bring our formulation to concrete model designs by proposing a novel part-aware equivariant network with a weighted messagepassing paradigm. With localized network operators and per-part SE(3) -equivaraint features, the network is able to guarantee per-step inter-part equivariance as well as facilitate stable convergence during the iterative inference. We test our framework on articulated objects to generalize from static rest states to novel articulates and multi-object scenes to generalize from clean synthetic scenes to cluttered real scans. Our model shows strong generalization in both scenarios even under significant changes in pointcloud geometry or topology.

To summarize, our key contributions are

To the best of our knowledge, we are the first to provide a strict definition of inter-part equivariance for pointcloud segmentation and introduce a learning framework with such equivariance by construction .

Wepropose a fixed-point framework with one-step training and iterative inference and show that the per-step equivariance induces an overall equivariance upon convergence.

We design a part-aware equivariant message-passing network with stable convergence.

Experiments show our strong generalization under inter-part configuration changes even when they cause subsequent changes in pointcloud geometry or topology.

2 Related Work

Equivariant pointcloud networks . Existing works in equivariant 3D learning mainly focus on rigid SE(3) transformations. As a well-studied problem, it has developed comprehensive theories [39, 14, 79, 2, 81] and abundant network designs [71, 20, 11, 15, 3, 36, 42, 55], which benefit a variety of 3D vision and robotics tasks ranging from pose estimation [44, 46, 54, 61, 89], shape reconstruction [11, 10], to object interaction and manipulation [19, 27, 60, 63, 80, 82]. A few recent works have extended SE(3) equivariance to part level [86, 41, 50]. [86] employs a local equivariant feature extractor for object bounding box prediction, showing robustness under object-level and scene-level pose changes. [41] learns an SE(3) -equivaraint object prior and applies it to object

seg

seg

seg

seg

equiv-1l/

equiv.1l

equiv.1

to

to

to

to

to

Training with ground-truth

Iterativeinferencewith

segmentation

per-stepequivariance

detection and segmentation in scenes with robustness to scene configuration changes. [50] learns part-level equivariance and pose canonicalization from a collection of articulated objects with a twostage coarse-to-fine network. However, all these works are purely heuristics-based, and none has provided a strict definition of inter-part equivariance.

Multi-body systems . From small-scale articulated objects to large-scale multi-object scenes, there is a wide range of 3D observations that involve multiple movable parts or objects, exhibiting various configurations and temporal variations. This has sparked a rich variety of works dedicated to addressing the challenges in multi-body systems, studying their part correspondences [56, 49, 8, 48, 43], segmentations [85, 76, 67, 31, 30, 70, 7, 12], reconstructions [34, 37, 32, 53, 40], or rearrangements [78, 68], to name just a few. However, as we know the system is acted via products of group actions; a more structured network that exploits such motion prior by construction is desired. Only a few works have studied this problem. [86] exploits global and local gauge equivariance in 3D object detection via bottom-up point grouping but without a systematical study of inter-object transformations. [41] further exploits equivariance of object compositions in scene segmentation, but as an EM-based approach, it requires exhaustive and inefficient enumerations and no global convergence is guaranteed. In contrast, our approach offers a theoretically sound framework that achieves multibody equivariance by construction, which enables robust analysis for various tasks and provides a more structured and reliable solution compared to existing methods.

Iterative inference . From recurrent neural networks [23, 29] to flow [16, 59, 17, 38, 83] or diffusionbased generative models [65, 66, 28, 64, 51, 88], iterative inference has played many diverse yet essential roles in the development of computer vision. Though naturally adapted to sequential or temporal data, iterative inference can also be applied to static learning problems such as image analysis [22, 21, 73, 26] and pointcloud segmentation [84, 47]. Recent studies also show that iterative inference presents stronger generalizability than one-step forward predictions [62, 6], with explanations of their relations to the "working memory" of human minds [4, 5] or human visual systems [45, 35]. In our work, we employ iterative inference on static pointclouds to co-evolve two intertwined attributes: part segmentation and inter-part equivaraince, in order to address a seemingly contradictory situation.

3 Inter-Part Equivariance

We start by defining inter-part equivariance on pointclouds with given part segmentations (Sec. 3.1). Then, we will introduce how to extend this definition to unsegmented pointclouds using a fixedpoint framework (Sec. 3.2) with one-step training and iterative inference. The iterations have per-step equivariance by network construction, which is shown to induce an overall equivariance upon convergence (Sec 3.3). Finally, we will also explain how to eliminate part orders for instance segmentation (Sec 3.4).

3.1 Multi-Body Systems

Let X ∈ R N × 3 be a pointcloud with P parts. Segmentations on X are represented as point-part assignment matrices y ∈ [0 , 1] N × P which sum up to 1 along the P dimension. As the action of SE(3) P on X and its resulting equivariance is subject to the part decompositions, we define the action of SE(3) P on (pointcloud, segmentation) pairs ( X , y ) . When y is a binary mask, the P

parts of X are P disjoint sub-pointclouds and each SE(3) component of SE(3) P acts on one part separately. More concretely, for any transformation A = ( T 1 , · · · , T P ) ∈ SE(3) P , we define A ( X , y ) := ( X ′ , y ) where the n -th point x n is transformed to x ′ n by

where R p and t p are the rotation and translation components of T p ∈ SE(3) . For soft segmentation masks, we view each point as probabilistically assigned to one of the P parts.

Equivariance . Similar to SE(3) -equivariance on rigid shapes f ( TX ) = T f ( X ) , ∀ T ∈ SE(3) , we define the inter-part equivariance on ( X , y ) pairs by saying that a function f is SE(3) P -equivariant if ∀ A ∈ SE(3) P ,

Note that if the function outputs f ( X , y ) are no longer (pointcloud, segmentation) pairs, the SE(3) P -action on its outputs needs to be specified. A special case is when SE(3) P acts trivially on f ( X , y ) by f ( A ( X , y )) = f ( X , y ) , which defines the common-sense 'invariance'. For a pointcloud segmentation network with per-point part-label outputs y ∈ [0 , 1] N × P , we desire it to be SE(3) P -invariant under the above definitions, that is f ( A ( X , y )) ≡ y , ∀ A ∈ SE(3) P .

3.2 Banach Fixed-Point Iterations

'Chicken or the egg' . The "chicken or the egg" nature of segmentation with inter-part equivariance immediately emerges from the above definition: to enforce SE(3) P -equivariance for a neural network f ( · ; Θ) , the part segmentation y is required as input, which is also exactly the desired output of the network (Fig. 1). That this,

To resolve this dilemmatic problem, we approach Eq. 3 as a fixed-point equation on function f ( · , y ; Θ) w.r.t. y . And instead of using single-step forward predictions, we solve it with Banach fixed-point iterations (Fig. 2).

Training . At training time, we aim to optimize the network weights Θ such that the labeled s ( X i , y i ) , i ∈ I from the dataset become fixed points of the function f ( · , Θ) :

However, there exists a trivial solution which is the identity function f ( · , y ; Θ) ≡ y , meaning that any arbitrary segmentation can satisfy the objective for any pointclouds. Thus, to avoid such degenerated cases, we have to limit the network expressivity. More specifically, we limit the Lipschitz constant L of the network w.r.t. y which is defined by

If f degenerates to the identical mapping, its Lipschitz L = 1 , which violates the above constraint. When L < 1 , f is a contractive function and has a unique fixed point based on the Banach fixed-point theorem. Our training objective is to align this fixed point with the ground-truth segmentation.

Iterative inference . At inference time, given an input pointcloud X 0 , the learnable parameters Θ ∗ fixed in f and we look for a segmentation y that satisfies

We solve this equation with Banach fixed-point iterations [1]

where y (0) is a random initialization of the segmentation on X 0 . For a contractive function f , the uniqueness of y ∗ induces a well-defined mapping from pointclouds to segmentations f ∗ Θ : X 0 → y ∗ .

3.3 Equivariance

For an input pairs ( X , y ) with known segmentation y , we employ an SE(3) -equivariant backbone [15] on each individual part and construct an SE(3) P -equivariant network, which will be further explained in Section 4. This guarantees the inter-part equivariance at training time and at each timestep during inference, but most importantly we desire the overall SE(3) P -equivariance at the convergence point of the Banach iteration. Here we prove two properties:

Self-coherence of convergent state . At each inference timestep k , we have SE(3) P -equivariance subject to y ( k ) , that is, ∀ A ∈ SE(3) P ,

Because of the continuity of f and the compactness of [0 , 1] N × P , we can take limits w.r.t. k on both sides, which gives

This shows the self-coherence of the convergent point between f ∗ Θ and y ∗ .

Generalization to novel inter-part states . Suppose the network has seen ( X , y gt ) in the training set and learned f Θ ( X , y gt ; Θ ∗ ) = y gt with the loss function in Eq. 4 minimized to zero. Now apply an inter-part transformation A ∈ SE(3) P and test the network on ( X ′ , y gt ) := A ( X , y gt ) . We would like to show f ∗ Θ ( X ′ ) = f ∗ Θ ( X ) = y gt. First of all, we know that y gt itself is a fixed point of f ( X ′ , · ; Θ) , which is due to the equivariance w.r.t. y gt at training time:

Now if f is a contractive function on y , we have the uniqueness of Banach fixed-points and thus y gt is the only fixed point for f ( X ′ , · ; Θ) , implying that the iterations in Eq. 7 must converge to y gt . Equivariance of the per-step iteration and the final convergent state are illustrated in Fig. 2 right.

In less ideal cases when the training loss is not zero but a small error ε , we view it as the distance ∥ y (1) -y (0) ∥ between the ground-truth y (0) = y gt and the one-step iteration output y (1) . The distance between the actual fixed-point y ∗ and the ground-truth y gt is then bounded by

where L is the Lipschitz constant of the network.

3.4 Part Permutations

A pointcloud segmentation represented by y ∈ [0 , 1] N × P inherently carries an order between the P parts, which is the assumption in semantic segmentation problems. But in various real-world scenarios, it is common to encounter situations where multiple disjoint parts are associated with the same semantic label, without any clear or coherent part orderings. In order to tackle this challenge, we also extend our method to encompass the instance segmentation problem without part orderings. For simplicity, here we assume that all parts can be permuted together by S P . In practice, permutations only occur among the parts within each semantic label, for which we can easily substitute the S P below with its subgroup and the conclusion still holds. We define an equivalence relation on [0 , 1] N × P by

which gives us a quotient space [0 , 1] N × P / S P and each equivalent class ˆ y in this quotient space represents an instance segmentation without part ordering. We can define a metric on ˆ y by

which makes ( [0 , 1] N × P / S P , d ) a complete metric space, on which the Banach fixed-point theorem holds and so are the iterations and convergence properties. Proofs for the well-definedness of d and the completeness of the quotient space are shown in the supplementary material.

Input points

Per-point equiv. features

Per-point inv.

Point-part

Updated per-

and labels

NxC×3

features

assign. score

point labels

(N × 3, N × P)

N× C × 1

N × P × 1

N × P

Weighted

VN-Inv

conv

Weighted

Pool

message

passing

Part

decomp.

Seg-weighted

Pool

Global inv.

Per-part inv.

message passing

latent code

latent code

NXCx1

C×1

C ×1× P

4 Part-Aware Equivariant Network

In this section, we present our part-aware equivariant network for segmentation label updates (Fig.3). We utilize a SE(3) -equivariant framework called Vector Neurons (VN) [15] as the backbone to encode per-point features V ∈ R N × C × 3 , ensuring equivariance within each part. Subsequently, these features are transformed into invariant representations, eliminating relative poses to facilitate inter-part information propagation and global aggregations.

Segmentation-weighted message passing . Key to our part-aware equivariant network is a messagepassing module weighted by the input segmentation y (Fig. 3 left). Intuitively, given a point x n ∈ X with latent feature V n , for its neighborhood points x m ∈ N , we only allow information propagation between x n and x m if they belong to the same part. When the part segmentation y is a soft mask, we compute the probability p nm of x n and x m belonging to the same part by p nm = y n y t m , y n , y m ∈ [0 , 1] 1 × P and weight the local message passing with

Proof of inter-part equivariance for this module can be found in the supplementary material.

Network architecture . Fig. 3 right shows the overall network architecture for segmentation label updates. Given an input pair ( X , y ) , we first extract a per-point SE(3) -equivariant local feature V ∈ R N × C × 3 within each part masked by y . V is then passed through a sequence of weighted convolutions with the massage passing defined in Eq. 14. To enable efficient inter-part information exchange, after each message-passing layer, we compute a global invariant feature through a perpoint VN-invariant layer, followed by a global pooling and a concatenation between per-point and global features. After the point convolutions, the per-point equivariant features V n are converted to invariant features S n with inter-part poses canceled. We then apply a global pooling on S n to obtain a global shape code, which is decomposed into P part codes Q 1 , · · · , Q P as in [13]. Finally, we compute a point-part assignment score for all ( S n , Q p ) pairs (resulting in an N × P tensor) and apply a softmax activation along its P dimension to obtain the updated segmentation y ′ . For instance segmentation without part permutations (Sec. 3.4), we modify the computation of the part feature Q by replacing the global feature decomposition with point feature grouping based on y .

For pointcloud networks with set-invariance across all points, directly restricting the upper bound of the network Lipschitz using weight truncations [72, 24, 87] will greatly harm network expressivity as it limits the output range on each point. On the other hand, the space of all possible segmentations [0 , 1] N × P has extremely high dimensionality, making the computation of Lipschitz regularization losses [69] rather inefficient. Therefore, we do not explicitly constrain the Lipschitz constant of our network. Nevertheless, we confine all operations involving y to local neighborhoods per point, as shown in Eq. 14. This practical approach generally ensures a small Lipschitz constant for the network in most scenarios. The stability of fixed-point convergence will be studied in Sec. 5.3.

5 Experiments

5.1 Articulated-object part segmentation

We begin by evaluating our method on articulated-object part segmentation using on four categories of the Shape2Motion dataset [74]: washing machine, oven, eyeglasses, and refrigerator. To demon-

Training

Inferenceiterations

state

=

strate the generalizability of our approach, we train our model on objects in a single rest state, such as ovens with closed doors (as depicted in Fig. 5, left). This training setup aligns with many synthetic datasets featuring static shapes [9, 52]. Subsequently, we assess the performance of our model on articulated states, where global and inter-part pose transformations are applied, replicating real-world scenarios. We evaluate our model on both unseen articulation states of the training instances (Tab. 1, left) and on unseen instances (Tab. 1, right). In both cases, the joint angles are uniformly sampled from the motion range for each category. We compare our network to the most widely adopted object-level part segmentation networks PointNet [57] and DGCNN [75] without equivariance, plus VNN [15] with global SE(3) -equivariance. We also compare to a VNN variant, named VNN-Inv, which first computes per-point local SE(3) -equivariant features and then converts them to invariant features and applies invariant message passing. This is a strategy adopted by [18] for human segmentation. All IoUs are computed for semantic parts. In the case of eyeglasses and refrigerators, which have two parts sharing the same semantic label, we train our network with three motion part inputs but output two semantic part labels. This is feasible because our weighted message passing (Eq. 14) is agnostic to the number of parts.

Fig. 4 demonstrates our iterative inference process on novel states after training on the rest states. Fig. 5 shows our segmentation predictions on the four categories. The inter-part pose transformations can cause significant changes in pointcloud geometry and topology, yet our method can robustly generalize to the unseen articulations with inter-part equivariance.

5.2 Multi-Object Scans

We also test our instance segmentation framework (without part orders) on multi-object scans.

Segmentation transfer on DynLab . DynLab [30] is a collection of scanned laboratory scenes, each with 2-3 rigidly moving solid objects captured under 8 different configurations with object positions randomly changed. We overfit our model to the first configuration of each scene and apply it to the 7 novel configurations, transfering the segmentation from the first scan to the others via inter-object equivariance.

We compare our method to a variety of multi-body segmentation methods, including motion-based co-segmentation methods (DeepPart [85], NPP [25], MultiBodySync [30]), direct segmentation prediction models (PointNet++ [58], MeteorNet [49]), geometry-based point grouping (Ward-linkage [77]), and pre-trained indoor instance segmentation modules (PointGroup [33]). The baseline setups follow [30]. Tab. 2 shows the segmentation IoUs. One thing to note is that motion-based co-segmentation can also be viewed as learning inter-object equivariance via Siamese training, yet they cannot reach the performance of equivariance by construction.

Training

—Inferenceonunseenstates

Inference onunseenstates+unseeninstances

sample

VNN

Ours

VNN

Ours

Method

PointNet++ [58]

39.4

±

7.1

MeteorNet [49]

71.8

±

9.7

DeepPart [85]

66.3

±

17.2

NPP [25]

71.6

±

7.7

Ward-linkage [77]

88.6

±

5.8

PointGroup [33]

72.4

±

12.5

MultiBodySync [30] 94.0

±

3.1

Ours

95.5

±

4.6

Synthetic

Inferenceonrealscans

trainingsamples

Z

SO(3)

n

Pile

Synthetic to real chair scans . We train our model using a synthetic dataset constructed from clean ShapeNet chair models [9] with all instances lined up and facing the same direction (Fig. 6 left). We then test our model on the real chair scans from [41] with diverse scene configurations (Fig. 6 right). The configurations range from easy to hard, including: Z (all chairs standing on the floor), SO(3) (chairs laid down), and Pile (chairs piled into clutters). Remarkably, our model with interobject equivariance demonstrates successful generalization across all these scenarios, even in the most challenging Pile setting with cluttered objects.

5.3 Ablation Studies

Network operators and convergence . We show the importance of our locally confined message passing and SE(3) -features (Eq. 14) to the convergence of test-time iterations. As opposed to extracting part-level equivariant features, one can also apply pose canonicalizations to each part individually and employ non-equivariant learning methods in the canonical space. For SE(3) -transformations with a translation component in R 3 and a rotation component in SO(3) , the former can be canonicalized by subtracting the part centers and the latter by per-part PCA. Thus we compare our full model with two ablated versions with part-level equivariance replaced by canonicalization: PCA , where we canonicalize both the part translations and rotations and simply employ a non-equivariant DGCNN [75] for the weighted message passing; VNN , where we only canonicalize the part translations but preserve the SO(3) -equivariant structures in the network.

Tab. 3 shows the segmentation IoUs of the full model and the ablated versions on the Shape2Motion oven category. espite canonicalizations being agnostic to per-part SE(3) transformations, they rely on absolute poses and positions, which breaks the locality of network operations, resulting in a significant increase in the Lipschitz constant of the network. Consequently, when any local equivariant operator is replaced by canonicalization, the network performance experiences a drastic drop.

To further examine the convergence ranges, we test the three models under different y (0) initializations by adding random noises ξ ∼ U (0 , 1) to y gt according to y (0) = (1 -α ) y gt + αξ, α ∈ [0 , 1] . Fig. 7 shows the performance changes of the three models with gradually increased noise levels

on y (0) . The PCA rotation canonicalization has a theoretically unbounded Lipschitz constant, making the network unable to converge stably even within a small neighborhood of y gt . The SO(3) -equivariant VNN with translation canonicalization perfectly overfits to the ground-truth fixed point but exhibits a rapid decline in performance as the initial y (0) deviates from y gt . In contrast, our SE(3) -equivariant network demonstrates stable performance across different noise levels.

Lipschitz constraints. . We also provide a study of different Lipschitz constraining methods under different norms in Tab. 4. For weight truncation [72], we use the l ∞ -Lipschitz as it is the loosest constraint and set the per-layer Lipschitz upper bound to be 0.99999. For regularization losses, we set the overall network Lipschitz threshold to 0.99. The adversarial sampling [69] loss only works with l 2 -norm. We reduce the total number of points from 2048 to 1024 and the GNN neighbors from 40 to 20 due to the large memory consumption for the loss computations. Based on our observations, not having an explicit Lipschitz constraint and using the SE(3)-equivariant message passing work best for the current situation, yet none of these existing Lipschitz constraining methods are helpful to the network performance. We also plot the l 2 -norm regularization losses in the supplementary material, showing that they are zero almost everywhere.

6 Conclusions

In this work, we propose Banana , which provides both theoretical insights and experimental analysis of inter-part equivariance. While equivariance is typically examined from an algebraic perspective, we approach it from an analysis standpoint and obtain a strict formulation of inter-part equivariance at the convergent point of an iteration sequence. Based on our theoretical formulation, we propose a novel framework that co-evolves segmentation labels and per-part SE(3) -equivariance, showing strong generalization under both global and per-part transformations even when these transformations result in subsequent changes to the pointcloud geometry or topology. Experimentally, we show that our model can generalize from rest-state articulated objects to unseen articulations, and from synthetic toy multi-object scenes to real scans with diverse object configurations.

Limitations and future work . While our local segmentation-weighted message passing reduces the Lipschitz constant of the network, practically providing stability during test-time iterations, it is important to note that it is not explicitly bounded or regularized. Lipschitz bounds and regularizations for set-invariant networks without compromising network expressivity would be an interesting and important problem for future study. In addition, we study the full SE(3) P action on multi-body systems, but in many real-world scenarios, different parts or objects may not move independently due to physical constraints. Limiting the SE(3) P -action to its subset of physically plausible motions can potentially increase our feature-embedding conciseness and learning efficiency.

Broader Impacts . Our study focuses on the general 3D geometry and its equivariance properties. Specifically, we investigate everyday appliances in common scenes, for which we do not anticipate any direct negative social impact. However, we acknowledge that our work could potentially be misused for harmful purposes. On the other hand, our research also has potential applications in areas that can benefit society, such as parsing proteins in medical science and building home robots for senior care.

Acknowledgments and Disclosure of Funding

We gratefully acknowledge the following grants: a TRI University 2.0 grant, a Vannevar Bush Faculty Fellowship, and a gift from the Adobe Corporation awarded to Stanford University; and NSF FRR 2220868, NSF IIS-RI 2212433, NSF TRIPODS 1934960, NSF CPS 2038873 awarded to the University of Pennsylvania.

References

Praveen Agarwal, Mohamed Jleli, Bessem Samet, Praveen Agarwal, Mohamed Jleli, and Bessem Samet. Banach contraction principle and applications. Fixed Point Theory in Metric Spaces: Recent Advances and Applications , pages 1-23, 2018.

Jimmy Aronsson. Homogeneous vector bundles and g-equivariant convolutional neural networks. Sampling Theory, Signal Processing, and Data Analysis , 20(2):10, 2022.

Serge Assaad, Carlton Downey, Rami Al-Rfou, Nigamaa Nayakanti, and Ben Sapp. Vn-transformer: Rotation-equivariant attention for vector neurons. arXiv preprint arXiv:2206.04176 , 2022.

Alan Baddeley. Working memory. Science , 255(5044):556-559, 1992.

Alan Baddeley. Working memory: Theories, models, and controversies. Annual review of psychology , 63:1-29, 2012.

Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Logical extrapolation without overthinking. arXiv preprint arXiv:2202.05826 , 2022.

Stefan Andreas Baur, David Josef Emmerichs, Frank Moosmann, Peter Pinggera, Björn Ommer, and Andreas Geiger. Slim: Self-supervised lidar scene flow and motion segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13126-13136, 2021.

Aseem Behl, Despoina Paschalidou, Simon Donné, and Andreas Geiger. Pointflownet: Learning representations for 3d scene flow estimation from point clouds. arXiv preprint arXiv:1806.02170 , 2, 2018.

Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 , 2015.

Evangelos Chatzipantazis, Stefanos Pertigkiozoglou, Edgar Dobriban, and Kostas Daniilidis. Se (3)-equivariant attention networks for shape reconstruction in function space. arXiv preprint arXiv:2204.02394 , 2022.

Haiwei Chen, Shichen Liu, Weikai Chen, Hao Li, and Randall Hill. Equivariant point network for 3d point cloud analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14514-14523, 2021.

Yujin Chen, Matthias Nießner, and Angela Dai. 4dcontrast: Contrastive learning with dynamic correspondences for 3d scene understanding. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII , pages 543-560. Springer, 2022.

Zhiqin Chen, Kangxue Yin, Matthew Fisher, Siddhartha Chaudhuri, and Hao Zhang. Bae-net: Branched autoencoder for shape co-segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8490-8499, 2019.

Taco S Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homogeneous spaces. Advances in neural information processing systems , 32, 2019.

Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12200-12209, 2021.

Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516 , 2014.

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803 , 2016.

Haiwen Feng, Peter Kulits, Shichen Liu, Michael J Black, and Victoria Abrevaya. Generalizing neural human fitting to unseen poses with articulated se (3) equivariance. arXiv preprint arXiv:2304.10528 , 2023.

Jiahui Fu, Yilun Du, Kurran Singh, Joshua B Tenenbaum, and John J Leonard. Robust change detection based on neural descriptor fields. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 2817-2824. IEEE, 2022.

Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks. Advances in Neural Information Processing Systems , 33:1970-1981, 2020.

Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 1440-1448, 2015.

Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 580-587, 2014.

Christoph Goller and Andreas Kuchler. Learning task-dependent distributed representations by backpropagation through structure. In Proceedings of International Conference on Neural Networks (ICNN'96) , volume 1, pages 347-352. IEEE, 1996.

Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning , 110:393-416, 2021.

David S Hayden, Jason Pacheco, and John W Fisher. Nonparametric object and parts modeling with lie group dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7426-7435, 2020.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

Carolina Higuera, Siyuan Dong, Byron Boots, and Mustafa Mukadam. Neural contact fields: Tracking extrinsic contact with tactile sensing. arXiv preprint arXiv:2210.09297 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):17351780, 1997.

Jiahui Huang, He Wang, Tolga Birdal, Minhyuk Sung, Federica Arrigoni, Shi-Min Hu, and Leonidas J Guibas. Multibodysync: Multi-body segmentation and motion estimation via 3d scan synchronization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 71087118, 2021.

Shengyu Huang, Zan Gojcic, Jiahui Huang, Andreas Wieser, and Konrad Schindler. Dynamic 3d scene analysis by point cloud accumulation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVIII , pages 674-690. Springer, 2022.

Ajinkya Jain, Rudolf Lioutikov, Caleb Chuck, and Scott Niekum. Screwnet: Category-independent articulation model estimation from depth images using screw theory. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 13670-13677. IEEE, 2021.

Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and Pattern recognition , pages 4867-4876, 2020.

Zhenyu Jiang, Cheng-Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5616-5626, 2022.

Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B Issa, and James J DiCarlo. Evidence that recurrent circuits are critical to the ventral streams execution of core object recognition behavior. Nature neuroscience , 22(6):974-983, 2019.

Oren Katzir, Dani Lischinski, and Daniel Cohen-Or. Shape-pose disentanglement using se (3)-equivariant vector neurons. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part III , pages 468-484. Springer, 2022.

Yuki Kawana, Yusuke Mukuta, and Tatsuya Harada. Unsupervised pose-aware part decomposition for 3d articulated objects. arXiv preprint arXiv:2110.04411 , 2021.

Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems , 31, 2018.

Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning , pages 2747-2755. PMLR, 2018.

Jiahui Lei and Kostas Daniilidis. Cadex: Learning canonical deformation coordinate space for dynamic surface representation via neural homeomorphism. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Jiahui Lei, Congyue Deng, Karl Schmeckpeper, Leonidas Guibas, and Kostas Daniilidis. Efem: Equivariant neural field expectation maximization for 3d object segmentation without scene supervision. arXiv preprint arXiv:2303.15440 , 2023.

Jiahan Li, Shitong Luo, Congyue Deng, Chaoran Cheng, Jiaqi Guan, Leonidas Guibas, Jian Peng, and Jianzhu Ma. Directed weight neural networks for protein structure representation learning. arXiv preprint arXiv:2201.13299 , 2022.

Ruibo Li, Guosheng Lin, and Lihua Xie. Self-point-flow: Self-supervised scene flow estimation from point clouds with optimal transport and random walk. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15577-15586, 2021.

Xiaolong Li, Yijia Weng, Li Yi, Leonidas J Guibas, A Abbott, Shuran Song, and He Wang. Leveraging se (3) equivariance for self-supervised category-level object pose estimation from point clouds. Advances in Neural Information Processing Systems , 34:15370-15381, 2021.

Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint arXiv:1604.03640 , 2016.

Cheng-Wei Lin, Tung-I Chen, Hsin-Ying Lee, Wen-Chin Chen, and Winston H Hsu. Coarse-to-fine point cloud registration with se (3)-equivariant representations. arXiv preprint arXiv:2210.02045 , 2022.

Fangyu Liu, Shuaipeng Li, Liqiang Zhang, Chenghu Zhou, Rongtian Ye, Yuebin Wang, and Jiwen Lu. 3dcnn-dqn-rnn: A deep reinforcement learning framework for semantic parsing of large-scale 3d point clouds. In Proceedings of the IEEE international conference on computer vision , pages 5678-5687, 2017.

Xingyu Liu, Charles R Qi, and Leonidas J Guibas. Flownet3d: Learning scene flow in 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 529-537, 2019.

Xingyu Liu, Mengyuan Yan, and Jeannette Bohg. Meteornet: Deep learning on dynamic 3d point cloud sequences. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 92469255, 2019.

Xueyi Liu, Ji Zhang, Ruizhen Hu, Haibin Huang, He Wang, and Li Yi. Self-supervised category-level articulated object pose estimation with part-level se (3) equivariance. arXiv preprint arXiv:2302.14268 , 2023.

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2837-2845, 2021.

Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 909-918, 2019.

Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13001-13011, 2021.

Haoran Pan, Jun Zhou, Yuanpeng Liu, Xuequan Lu, Weiming Wang, Xuefeng Yan, and Mingqiang Wei. So (3)-pose: So (3)-equivariance learning for 6d object pose estimation. arXiv preprint arXiv:2208.08338 , 2022.

Adrien Poulenard and Leonidas J Guibas. A functional approach to rotation equivariant non-linearities for tensor field networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13174-13183, 2021.

Gilles Puy, Alexandre Boulch, and Renaud Marlet. Flot: Scene flow on point clouds guided by optimal transport. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII , pages 527-544. Springer, 2020.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 652-660, 2017.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning , pages 1530-1538. PMLR, 2015.

Hyunwoo Ryu, Jeong-Hoon Lee, Hong-in Lee, and Jongeun Choi. Equivariant descriptor fields: Se (3)-equivariant energy-based models for end-to-end visual robotic manipulation learning. arXiv preprint arXiv:2206.08321 , 2022.

Rahul Sajnani, Adrien Poulenard, Jivitesh Jain, Radhika Dua, Leonidas J Guibas, and Srinath Sridhar. Condor: Self-supervised canonicalization of 3d pose for partial shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16969-16979, 2022.

Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems , 34:6695-6706, 2021.

Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA) , pages 6394-6400. IEEE, 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Ziyang Song and Bo Yang. Ogc: Unsupervised 3d object segmentation from rigid dynamics of point clouds. arXiv preprint arXiv:2210.04458 , 2022.

Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Diffuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis. arXiv preprint arXiv:2303.14207 , 2023.

Dávid Terjék. Adversarial lipschitz regularization. arXiv preprint arXiv:1907.05681 , 2019.

Hugues Thomas, Ben Agro, Mona Gridseth, Jian Zhang, and Timothy D Barfoot. Self-supervised learning of lidar segmentation for autonomous indoor navigation. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 14047-14053. IEEE, 2021.

Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219 , 2018.

Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems , 31, 2018.

Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu. Cnn-rnn: A unified framework for multi-label image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2285-2294, 2016.

Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8876-8884, 2019.

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog) , 38(5):1-12, 2019.

Yuqi Wang, Yuntao Chen, and ZHAO-XIANG ZHANG. 4d unsupervised object discovery. Advances in Neural Information Processing Systems , 35:35563-35575, 2022.

Joe H Ward Jr. Hierarchical grouping to optimize an objective function. Journal of the American statistical association , 58(301):236-244, 1963.

Qiuhong Anna Wei, Sijie Ding, Jeong Joon Park, Rahul Sajnani, Adrien Poulenard, Srinath Sridhar, and Leonidas Guibas. Lego-net: Learning regular rearrangements of objects in rooms. arXiv e-prints , pages arXiv-2301, 2023.

Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Coordinate independent convolutional networks-isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint arXiv:2106.06020 , 2021.

Thomas Weng, David Held, Franziska Meier, and Mustafa Mukadam. Neural grasp distance fields for robot manipulation. arXiv preprint arXiv:2211.02647 , 2022.

Yinshuang Xu, Jiahui Lei, Edgar Dobriban, and Kostas Daniilidis. Unified fourier-based kernel and nonlinearity design for equivariant networks on homogeneous spaces. In International Conference on Machine Learning , pages 24596-24614. PMLR, 2022.

Zhengrong Xue, Zhecheng Yuan, Jiashun Wang, Xueqian Wang, Yang Gao, and Huazhe Xu. Useek: Unsupervised se (3)-equivariant 3d keypoints for generalizable manipulation. arXiv preprint arXiv:2209.13864 , 2022.

Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4541-4550, 2019.

Xiaoqing Ye, Jiamao Li, Hexiao Huang, Liang Du, and Xiaolin Zhang. 3d recurrent neural networks with context fusion for point cloud semantic segmentation. In Proceedings of the European conference on computer vision (ECCV) , pages 403-417, 2018.

Li Yi, Haibin Huang, Difan Liu, Evangelos Kalogerakis, Hao Su, and Leonidas Guibas. Deep part induction from articulated object pairs. arXiv preprint arXiv:1809.07417 , 2018.

Hong-Xing Yu, Jiajun Wu, and Li Yi. Rotationally equivariant 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1456-1464, 2022.

Bohang Zhang, Du Jiang, Di He, and Liwei Wang. Rethinking lipschitz neural networks and certified robustness: A boolean function perspective. In Advances in Neural Information Processing Systems , 2022.

Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5826-5835, 2021.

Minghan Zhu, Maani Ghaffari, and Huei Peng. Correspondence-free point cloud registration with so (3)equivariant implicit shape representations. In Conference on Robot Learning , pages 1412-1422. PMLR, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers an important problem in learning on point clouds -- the equivariance under the SE(3) group. \nNamely, the authors address the requirement of inter-part equivariance, essential for handling real-world scenarios, where an object can consist of multiple moving parts, or a scene can contain multiple objects that undergo different rigid transformations. \n\nThe key observation is that segmentation is necessary to define such per-part equivariance, which the authors are the first to do in a strict manner. \nFurthermore, the proposed fixed-point framework with one-step training and iterative inference is used to demonstrate that the per-step equivariance induces an overall equivariance upon convergence. \nThe developed inter-part equivariant message-passing network with stable convergence is experimentally shown to have strong generalization under different scene configurations, even those changing the point cloud geometry/topology. \n\nOverall, the paper provides a sound theoretical framework for a complex practical problem. \n\n---------------------------------------\nPOST REBUTTAL\n\nThe authors comprehensively addressed my questions and concerns.\nGiven the provided answers and rebuttal overall, I maintained my positive assessment of the paper.\n\nSTRENGTHS:\nS1. The paper is very well-written, and the structure is sound. The illustrations are clear and instructive, which helps to comprehend the presented theory. \n\n  \n\nS2. The proposed view of inter-part equivariance as a co-evolving interplay between geometry and segmentation is novel and compelling. \n\n  \n\nS3. The experimental validation supports well the claims stated in the contribution.\n\nWEAKNESSES:\nW1. A discussion on the (time-)complexity of the proposed method is missing.  \n\n- It is unclear how efficient the iterative inference is. \n- Besides, a comparison of the proposed model complexity and the baselines is missing. \n\n  \n\nW2. The details of the hyperparameter choice, including the iterative inference part, are missing.  \n\n- Crucially, the optimal number of iterations $\\textbf{k}$ is unspecified, nor is its effect on the complexity with regards to W1.  \n\n- The same applies to the motivation of the chosen size of the radius of the ball query, $r$, and the maximum number of points in the local neighborhood, $k$. \n\n  \n\nW3. Experiment Section 5.1: the selected Shape2Motion shapes have at most 3 parts with two different semantic labels.  \n\n- I wonder how the proposed method would perform on Shape2Motion shapes with more than 2 different part labels, e.g., motorbike or bicycle.\n\nQUESTIONS:\nQ1. The first part of the title is often associated with negative connotations in English; see, e.g., [0]. The authors might want to revise it so as not to cause unnecessary misunderstanding. \n\nQ2. How can the proposed method be used to perform point cloud classification? \n\nQ3. The declaration of $φ$ in (14) is missing. \n\nQ4. What is the number of runs in the experiment statistics presented in the tables in Section 5? Especially important for Table 2.\n\nQ5. Can the authors explain the large standard deviation of the results in Table 1? Why does the proposed model, in most cases, perform better on Unseen states + unseen instances than Unseen states (see Table 1)?\n\nQ6. Ablation studies: Was the inherent ambiguity of PCA-based canonicalization (see, e.g., [1]) taken care of? Could authors elaborate on the details of this experiment? \n\nQ7. In the theory of the VN framework [2], which the authors' method is based on, the pose-effect cancelation is achieved by means of the inner product of the equivariant features, which also cancels the effect of reflections. Could the authors run a simple experiment showing if their method is actually E(3)-equivariant and discuss it? \n\nQ8. Further noise-stability analysis of the method, where the noise is applied to the input point cloud (not just the segmentation mask), would be beneficial. \n\nQ9. It would make the paper more accessible to a broader audience if the authors included an informal motivation of using the Banach fixed-point theorem and iterations. \n\n \nA non-exhaustive list of typos: \n\n- Line 148 \" and we\" --> \", we\" \n\n- Line 277 \"espite\" --> \"despite\" \n\n-------------------------------------------------\n\n[0]  \"banana.\" Farlex Dictionary of Idioms. 2015. Farlex, Inc 14 Jun. 2023 https://idioms.thefreedictionary.com/banana \n\n \n\n[1] Li et al. (2021), A Closer Look at Rotation-Invariant Deep Point Cloud Analysis \n\n  \n\n[2] Deng et al. (2021), Vector neurons: A general framework for SO(3)-equivariant networks\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and broader societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a method for object part segmentation of point clouds that is equivariant/invariant to SE(3) part transformations. The core of their method is a neural network with points and segmentation as input and segmentation as output. The network is assumed to be contractive and is then used to perform Banach fixed point iterations towards the correct segmentation during inference. The Banach fixed point network is embedded in a larger architecture using Vector Neurons. For implementation, a message passing network is used, which uses the input segmentation for determining message strength. \n\nThe proposed method is evaluated for articulated object part segmentation (on Shape2Motion) and for segmentation of multi-object scans (on DynLab), where it compares favorably against baselines.\n\nSTRENGTHS:\n- The theoretical framework of Banach iterations is elegant and an interesting perspective on iterative networks during inference. In practice, similar ideas on point clouds have been there before, understanding such networks as reweighting functions for iterative reweighting least square schemes, EM iterations, etc. However, I think there is value in this specific perspective, especially since it comes with a convergence guarantee under Lipschitz constraints. To my knowledge, this is a novel contribution.\n- The presented algorithm seems to solve the given tasks well, outperforming the given baselines clearly (however, there are some concerns regarding missing baselines, see below).\n- The paper is nicely presented and easy to follow, given the slightly more complex nature.\n- The result of generalization from synthetic to real chairs is strong\n\nWEAKNESSES:\n- The first part of the paper is all about the consequences of a Lipschitz-constraint network and then there is no Lipschitz constraint in practice. The authors claim that the local message passing serves as some type of Lipschitz regularization. However, there is no reference given for this claim and I don't know one either.  I think it would make the paper stronger if this aspect would be supported by experiments or theory, which is isn't.  The authors only ablate on model performance under different noise levels, which does not give a full picture about convergence.\n- In articulated object part segmentation,  the paper seems to leave out important comparisons to previous work, e.g., [1,2,3]. All three papers are referenced in related work but not compared against. It would be good if the authors would provide a comparison or a discussion, why the comparison is not necessary.\n\nI think this is a very interesting paper with a new perspective on iterative networks. However, the above points are a bit concerning, which is why I am not fully convinced to give a better score.\n\n[1] Kawana et al.: Unsupervised pose-aware part decomposition for 3d articulated objects.\n\n[2] Kawana et al.: Neural star domain as primitive representation\n\n[3] Chen et al.: BAE-NET: Branched Autoencoder for Shape Co-Segmentation\n\nQUESTIONS:\n- I suspect that equation 1 wants to express linear blending of part transforms. However, I don't think it is correct: (1) sum of matrix $\\mathbf{R}$ and vector \\mathbf{t}, (2) $\\mathbf{t}_p$ not weighted by $\\mathbf{y}_{np}$ . Could the authors clarify?\n- Please provide further evidence that the network indeed behaves contractive or a theoretical justification for the Lipschitz regularization via local message passing.\n- Please discuss or provide missing comparisons with previous methods for part segmentation on Shape2Motion.\n\nLIMITATIONS:\nThe authors discuss limitations and societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an equivariant network for part-based (or multi-object) point cloud segmentation. The approach is equivariant to separate SE(3) transformations of each part/object. This is ensured by introducing a Banach fixed-point network. The network takes the point-could and the current segmentation as input, and iterates till convergence. The proposed part-aware equivariant network employs the Vector Neurons method to achieve equivariance for the individual parts. A segmentation-weighted message passing then adds communication between the different parts. Experiments are performed on objects from the Shape2Motion and DynLab datasets.\n\nSTRENGTHS:\n-\tNovel work based on very interesting and elegant ideas.\n-\tTheoretically sound.\n-\tVery well written and clear.\n-\tGood illustrations.\n-\tComparison with several methods.\n\nWEAKNESSES:\n1.\tThe experimental evaluation is limited to very small-scale datasets. It is not clear how the method would scale to larger datasets and more complex scenes, e.g. in an automotive setting, or from terrestrial Lidar scans. It would be good if the authors could discuss this in more detail.\n\n2.\tIt seems that the authors always train on the exact instance which is also encountered during inference. How would the method perform if it encounters a new type of, e.g., oven, after being trained on a dataset of different ovens. \n\n3.\tI did not find how the other methods in table 1 and 2 were trained. Is data augmentation used? How would data augmentation impact the performance of e.g. PintNet++ or MeteorNet?\n\n4.\tI did not find discussion on inference and training time.\n\n5.\tIt would be very interesting to see some plots of the IoU w.r.t. the number of iterations, to understand the convergence behavior of the model.\n\n\nIn summary, I think that this is a very interesting and solid work. I find no significant weaknesses. Although it would be appreciated if the authors can answer and address my comments in a rebuttal to further strengthen the paper.\n\nQUESTIONS:\nSee weaknesses. \n\nMoreover:\nWhich other tasks could be suitable for this method?\n\nLIMITATIONS:\nThey are well addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-26"}

comments: SUMMARY:
The paper propose a Banach, an approach for part-based point-cloud segmentation. In particular, the authors propose an approach to enforce equivariance of the part-segmentation, by construction.  They propose a fixed-point framework with one-step training and iterative inference. They propose a part-aware segmentation network.

STRENGTHS:
- The results are very convincing - the proposed approach seem to significantly outperform previous approaches

WEAKNESSES:
I do not see any direct weaknesses but I have very little knowledge about this field.

QUESTIONS:
- what is the run-time of the approach? How does it compare to previous approaches?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a neural network architecture that is equivariant to transformations in SE(3) for each object part independently. The network is part of a fixed-point framework where the network is trained with a single step but during testing an iterative approach is used that converges to the desired segmentation.\n\nSTRENGTHS:\n- The paper presents a fresh idea to solve the problem of part segmentation that takes into consideration the different movable parts that compose an object and makes the network equivariant to transformations of those.\n\n- The paper provides theoretical derivations that motivate the solution proposed in the paper.\n\n- The paper shows the effectiveness of the solution by providing several experiments.\n\nWEAKNESSES:\nAlthough I like the paper, I believe the evaluation could be improved by including other types of equivariant networks:\n\n- For example, neural networks based on equivariant operations such as group convolutions or steerable convolutions have by construction equivariance wrt the object parts too. Although these methods would allow information flow between object parts, most of the object parts would remain equivariant to transformations of those. \n\n- Moreover, network architectures that work only with the intrinsic information of the shape should also be included. Graph Convolution networks or equivariant mesh convolutions would also maintain equivariance of object parts. These networks are commonly used to segment people in different poses, which is a related problem to the one addressed in the paper.\n\n- As the paper states, there is no guarantee that L < 1. The paper states that weight truncation could restrict the upper bound of L but harm the expressivity of the network. An experiment where this is studied would improve the paper.\n\n- Lastly, I do not like the acronym Banana since it is not really an acronym of the title. There is no need to include an acronym on the paper.\n\nQUESTIONS:\nSee the weaknesses section.\n\nLIMITATIONS:\nThe limitations are well addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}