Banana: Banach Fixed-Point Network for Pointcloud Segmentation with Inter-Part Equivariance
Authors: Congyue Deng
Jiahui Lei
Bokui Shen
Kostas Daniilidis
Leonidas Guibas
Venue: NeurIPS 2023 spotlight (2023)
Keywords: 3D deep learning
equivariant network
pointcloud segmentation
multi-body system

Abstract: Equivariance has gained strong interest as a desirable network property that inherently ensures robust generalization. However, when dealing with complex systems such as articulated objects or multi-object scenes, effectively capturing inter-part transformations poses a challenge, as it becomes entangled with the overall structure and local transformations. The interdependence of part assignment and per-part group action necessitates a novel equivariance formulation that allows for their co-evolution. In this paper, we present Banana, a Banach fixed-point network for equivariant segmentation with inter-part equivariance by construction. Our key insight is to iteratively solve a fixed-point problem, where point-part assignment labels and per-part SE(3)-equivariance co-evolve simultaneously. We provide theoretical derivations of both per-step equivariance and global convergence, which induces an equivariant final convergent state. Our formulation naturally provides a strict definition of inter-part equivariance that generalizes to unseen inter-part configurations. Through experiments conducted on both articulated objects and multi-object scans, we demonstrate the efficacy of our approach in achieving strong generalization under inter-part transformations, even when confronted with substantial changes in pointcloud geometry and topology.

Banana: Banach Fixed-Point Network for Pointcloud Segmentation with Inter-Part Equivariance
Congyue Deng
Kostas Daniilidis Leonidas Guibas' Stanford University University of Pennsylvania Archimedes, Athena RC {congyue, willshen, guibas}@cs. stanford.
Jiahui Lei
Bokui Shen'
(leijh, kostas)0cis upenn .eu edu
Abstract
Equivariance has gained strong interest desirable network property that in- herently ensures robust generalization. However, when dealing with complex sys- tems such as articulated objects multi-object scenes, effectively capturing inter- part transformations poses a challenge, as it becomes entangled with the overall structure and local transformations. The interdependence of part assignment and per-part group action necessitates a novel equivariance formulation that allows for their co-evolution. In this paper, we present Banana, Banach fixed-point network for pointcloud segmentation with inter-part equivariance by construc- tion. Our key insight is to iteratively solve fixed-point problem, where point- part assignment labels and per-part (egu))-ruvaai co-evolve simultaneously. We provide theoretical derivations of both per-step equivariance and global con- vergence, which induces an equivariant final convergent state. Our formulation naturally provides a strict definition of inter-part equivariance that generalizes unseen inter-part configurations. Through experiments conducted on both articu- lated objects and multi-object scans, we demonstrate the efficacy of our approach in achieving strong generalization under inter-part transformations, even when confronted with substantial changes in pointeloud geometry and topology.
Introduction
From articulated objects to multi-object scenes, multi-body systems, i.e. shapes composed of multi- ple parts where each part can be moved separately. are prevalent in various daily-life scenarios. How- ever, modeling such systems presents considerable challenges compared rigid objects, primarily due to the infinite shape variations resulting from inter-part pose transformations with exponentially growing complexities. Successfully modeling these shapes demands generalization across poten- tial inter-part configurations. While standard data augmentation techniques can potentially alleviate such problems, exhaustive augmentation can be highly expensive especially given the complexity of such systems compared to rigid objects. Mcanwhile, recent rigid-shape analysis techniques have made significant progress in building generalization through the important concept of equivariance [39, [4., 79, 2, 81, 711. 2 IIII (155, B, 66, 4Z, 555. Equivariance dictates that when transformation applied to the input data, the network's output should undergo a corresponding transformation. For example, if is rotated by 90 degrees, the segmentation masks should also rotate accordingly. However, to the best of our knowledge, no existing work has managed to extend the formulation of equivariance inter-part configurations.
In this paper, we address the challenge of achieving inter-part equivariance in the context of part segmentation, fundamental task in analysis, which is also the key to generalizing equivaraince from single object to multi-body system. Extending equivariance to multi-body systems presents notable challenges in both formulation and realization. When dealing with shape that allows for inter-pat iiter-part motions, the model must exhibit equivariance to both global and local state changes, which can only be defined combining part assignment and per-part transformations. For example, to model an
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Equivariance for segnentation?
rigid trans. T SE(3)
per-part trans. A, SD
X
X
equic TO
X
equiil to
X
rigid trans. T SE(3)
per-part trans. Ay E SE(3)"
equik to
Segnentation defines equivariance
Global equivariance
Inter-part equivariance
Figure 1: "Chicken the egg" problem with inter-part equivariance. Left: Rigid SE(3)- cquivariance. rigid transformation is applied to the input, the output segmentation transforms ac- cordingly. Right: Inter-part equivariance. Analogously, under per-part transformations. But such a definition of equivariance requires part segmentations, which is also exactly the desired network output, resulting in a "chicken or the egg" problem.
want the input and output be coherent
oven's 's inter-part states, we first need to differentiate between the parts corresponding to the door and the body and then define the movement of each individual part. An intriguing "chicken or the egg" problem arises when attempting to build inter-part equivariance without access to provided segmentation, where segmentation necessary to define equivariance, but segmentation itself the desired output that we aim to produce.
Our key insight to tackle this seeming dilemma is to model inter-part equivariance sequen- tial fixed-point problem by co-evolving part segmentation labels and per-part SE(())-quivvriance. We provide theoretical derivations for both the per-step behavior and global convergence behav- ior, which are crucial aspects of a fixed-point problem. We show that our formulation establishes per-step equivariance through network construction, which then induces an overall inter-part equiv- ariance upon convergence. Thus, by having equivariant per-step progression and global convergence, our formulation naturally gives rise strict definition of inter-par equivariance through iterative inference that generalizes to unseen part configurations. We further bring our formulation to con- crete model designs by proposing novel part-aware equivariant network with a weighted message- passing paradigm. With localized network operators and per-part -equvvvrra features, the network is able to guarantee per-step inter inter-part equivariance as well as facilitate stable convergence during the iterative inference. We test our framework on articulated objects generalize from static rest states to novel articulates and multi-object scenes to generalize from clean synthetic scenes to cluttered real scans. Our model shows strong generalization in both scenarios even under significant changes in pointcloud geometry or topology.
To summarize, our key contributions are
To the best of our knowledge, we are the first to provide strict definition of inter-part equivariance for pointcloud segmentation and introduce a learning framework with such equivariance by construction.
We propose fixed-point framework with one-step training and iterative inference and show that the per-step equivariance induces an overall equivariance upon convergence.
We design part-aware equivariant message-passing passing network with stable convergence.
Experiments show our strong generalization under inter-part configuration changes even when they cause subsequent changes pointcloud geometry or topology.
2 Related Work
Equivariant pointcloud networks. Existing works in equivariant 3D learning mainly focus on rigid SE(3) transformations. As well-studied problem, it has developed comprehensive theories [69, 14. 79, 2. XI]] and abundant network designs [[Z]] 20, III. [15,, B, 36, 55], which benefit
variety of 3D vision and robotics tasks ranging from pose estimation [44
BE 54, bl 89], shape 5%
MI X2) A few recent works have extended SE(3) equivariance to part level [86, A1L 500 [&G] employs local equivariant
reconstruction [IЛI, IT], to object interaction and manipulation [19,
feature extractor for object bounding box prediction, showing robustness under leject-level and scene-level pose changes. [I]] learns (esuivreqiv object prior and applies it to object

---

equi
equis
equiv
equiv
equiv
to
to
to
to
Training with ground-trath
Iterative inforencee with per-itep equivariance
segmentation
Figure 2: Single-state training and novel-state iterative inference. Left: Training with ground- truth segmentation both network input and output. Right: Equivariant Banach iterations. At each step k; the network is equivariant its current segmentation input y()) by construction. At the end of the iterations, it converges to its fixed point with an induced overall equivariance.
detection and segmentation in scenes with robustness to scene configuration changes. [SO] learns part-level equivariance and pose canonicalization from a collection of articulated objects with two- stage coarse-te to- fine network. However, all these works are purely muristics-bbsede and none has provided strict definition of inter-part equivariance.
Multi-body systems. From small-scale articulated objects to large-scale multi-object scenes, there a wide range of 3D observations that involve multiple movable parts objects, exhibiting various configurations and temporal variations. This has sparked a rich variety of works dedicated to address- ing the challenges multi-body systems, studying their part correspondences
[56., BM% S. HR. [.)
42 5.3, HI], or rearrangements [788, 68], name just few. However, as know the system is acted via products of group actions; more structured network that exploits such motion prior by construction is desired. Only a few works have studied this problem. [X6] exploits global and local gauge equivariance in 3D object detection via bottom-up point grouping but without systematical study of inter-object transforma- tions. [II]] further exploits equivariance of object compositions in scene segmentation, F.M-bssed approach, it requires exhaustive and inefficient enumerations and no global convergence guaranteed. In contrast, our approach offers a theoretically sound framework that achieves multi- body equivariance by construction, which enables robust analysis for various tasks and provides more structured and reliable solution compared to existing methods.
segmentations [8 7R, 67, (TT, 30 70, Z [2], reconstructions [6_]]
37
as an
Iterative inference. From recurrent neural networks [23 2X] to flow [16. 59. 17 38, K3 or diffusion- based generative models [65. 6f. 28. 1e 51 КX] iterative inference has played many diverse yet essential roles in the development of computer vision. Though naturally adapted to sequential or temporal data, iterative inference can also be applied to static learning problems such as image anal- ysis [22), 21, 7A, 26] and pointcloud segmentation [84, A7]]. Recent studies also show that iterative inference presents stronger generalizability than one-step forward predictions [62, BJ), with explana- tions of their relations to the "working memory" of human minds [H [S]] or human visual systems E5. 555 In our work, we employ iterative inference on static pointclouds to co-evolve two inter- twined attributes: part segmentation and inter-part equivaraince, in order to address seemingly contradictory situation.
Inter-Part Equivariance
We start by defining inter-part equivariance on pointclouds with given part segmentations (Sec. B.J.). Then, we will introduce how to extend this definition to unsegmented pointclouds using fixed- point framework (Sec. (.2) with one-step training and iterative inference. The iterations have per-step equivariance by network construction, which is shown to induce an overall equivariance upon convergence (Sec (.3) Finally, we will also explain how to eliminate part orders for instance segmentation (Sec (.4)..
3.1 Multi-Body Systems
Let X E Nx3 be pointcloud with P parts. Segmentations on are represented as point-part assignment matrices y N which sum up to along the P dimension. As the action of SF P on and resulting equivariance is subject to the part decompositions, we define the action of SF(3)P on (pointeloud, segmentation) pairs yX. When is binary mask, the P
3

---

parts of X are P disjoint sub-pointcloude and each SE(3) component of SE(3)* P acts on one part
separately. More concretely, for any transformation A (T..... Tp) SE(3) ?. we define y) = yX) where the n-th point x, is transformed to x", by
P
x, = Σ Yapl (x Rx + tp).
(1)
where R2 and tp are the rotation and translation components of TT p SE(3). For soft segmentation masks, we view each point as probabilistically assigned to one of the P parts.
Equivariance. Similar to -uuiiaaqqi on rigid shapes f(TX) = Tf(X), VT E SE(3), we define the inter-part equivariance on (X, y) pairs by saying that a function f is SE(3) equiiariant if A E SE(3)*.
(A((X)))) A)(
Note that if the function outputs /(X, y) are no longer (pointeloud, segmentation) pairs, the SE(3) action on its outputs needs to be specified. A special case is when SE(3) acts trivially on f(X,y) by (XA(X, y)) f(X,y), y), which defines the common-sense "invariance". For a pointcloud seg- mentation network with per-point part-label outputs E [o, 1] NNPx we desire it to be SE(3) invariant under the above definitions, that y(((X,)) y, AA SE(3)
3.2 Banach Fixed-Point Iterations
"Chicken or the egg". The "chicken the egg" nature of segmentation with inter-part equivariance immediately emerges from the above definition: to enforce work O); the part segmentation y required as input, which is also exactly the desired output of the network (Fig. (I.. That this,
for neural net-
f(X, y.
To resolve this dilemmatic problem, we approach Eq. as fixed-point equation on function f(:y;; 9) w.t.. And instead of using single-step forward predictions, we solve it with Banach fixed-point iterations (Fig. ()
Training. At training time, we aim to optimize the network weights θ such that the labeled s yi), I from the dataset become fixed points of the function ():
θ, = arg min
Σ FE
()) - y:ll-
T
However, there exists a trivial solution which is the identity function f(-.y: Q) y. meaning that any arbitrary segmentation can satisfy the objective for any pointclouds. Thus, to avoid such degen- crated cases, we have to limit the network expressivity. More specifically, we limit the Lipschitz constant L of the network w.r.t. y which is defined by
(1; - f(X, y2: 0) S L - Y2ll. VX, Vy1.
(5)
If degenerates to the identical mapping, its Lipschitz I. 1, which violates the above constraint. When L < 1, ff is contractive function and has unique fixed point based on the Banach fixed fxee-point theorem. Our training objective is to align this fixed point with the ground-truth segmentation.
Iterative inference. At inference time, given input pointcloud Xo. the learnable parameters O. fixed in ff and we look for segmentation y that satisfies
e. .. (y) = f(xo, y; 0.)
(6)
We solve this equation with Banach ixed-point iterations
lim (k) lim 60000
X (0)
where y (0) random initialization of the segmentation on Xp. For contractive function f. the
uniqueness of induces a well-defined mapping from pointclouds to segmentations fō Xo + y

---

3.3 Equivariance
For an input pairs yX with known segmentation y. we employ an -quuv--quivariant backbone [15] each individual part and construct (F(3))-equivarii) network, which will be further explained in Section EL 'Tis guarantees the inter-part equivariance at training time and at each timestep during inference, but most importantly we desire the overall SE(3) -eqivvariacce the convergence point of the Banach iteration. Here we prove two properties:
Self-coherence of convergent state. At each inference timestep k, we have -equivarqqa subject to y (()), that is, AA E SE(3)
S(A(Xo,
(()), ,), 4k: E,
(8)
Because of the continuity of f and the compactness of [0, 1JxxP, we can take limits w.r.f. & on both sides, which gives
f(A(Xo. y"): 9) = f(Xo,y' () = (Xo).
(9)
This shows the self-coherence of the convergent point between fā and y*.
Generalization to novel inter-part states. Suppose the network has seen (X,yg) in the training set and learned fe Ng. Ygt: Ox) = Yg with the loss function in Eq. minimized to zero. Now apply an inter-part transformation A E SE(3) R and test the network on (X', yg)) == A(X,y.)) We would like to show få (X') fe (X) Ygt- First of all, we know that Ygt itself is fixed point of f(X', , (), which is due to the equivariance w.e.t Ygt at training time:
f(X' 0) = ) f(x yg.
(10)
Now if is contractive function we have the uniqueness of Banach fixed-points and thus Yu is the only fixed point for f(X', : O), implying that the iterations in Eq. 0 must converge to Ygt- Equivariance of the per-step iteration and the final convergent state are illustrated in Fig. D right.
In less ideal cases when the training loss not zero but small error E, we view as the distance (ly (0)) between the ground-truth (0) Ygt and the one-step iteration output y1), The distance between the actual fixed-point y* and the ground-truth Yg: is then bounded by
ly Vell
(x+(+) (x)
Σ k=
by -100
1 1-L
(11)
where L is the Lipschitz constant of the network.
3.4 Part Permutations
pointcloud segmentation represented by y e [0, 1| inherently carries an order between the P parts, which is the assumption in semantic segmentation problems. But in various real-world scenarios, it common to encounter situations where multiple disjoint parts are associated with the same semantic label, without any clear or coherent part orderings. In order to tackle this challenge, we also extend our method to encompass the instance segmentation problem without part orderings. For simplicity, here we assume that all parts can be permuted together by Gp. In practice, permuta- tions only occur among the parts within each semantic label, for which we can easily substitute 6p below with its subgroup and the conclusion still holds. We define equivalence relation on [0, NxP by
Y1 Y2
? 3o 510
(12)
which gives quotient space [0, 1|NXP//S GGp and each equivalent class ý in this quotient space represents an instance segmentation without part ordering. We can define a metric on y by
d d(S min
lyio -2ll.
(13)
dEGp
which makes (1o, Nx HNNxP/Gp. d) complete metric space, which the Banach fixed- fixed-point theorem holds and so are the iterations and convergence properties. Proofs for the well-definedness of d and the completeness of the quotient space are shown in the supplementary material.
5

---

liper peizes (N &S
Preegatst
Prinpprrt NxPx1
Updlated label XXP
-N CX
Scctane MxCx3
Weighted
Weigred anoot
decomp
Seg-weighred
Pasl
Glstel breer onde
Prepurt bnnett code
mussoage paswing
AxC*1
MKCK3
CX1
CXIX
Figure Part-aware equivariant network. Left: Segmentation-weigher message passing. Right: Overall architecture for segmentation label update. Color gradients on the tensors indicate the usage of segmentation labels,
4 Part-Aware Equivariant Network
In this section, we present our part-aware equivariant network for segmentation label updates (Fig E) We utilize -equivaaava framework called Vector Neurons (VN) [TSS] the backbone to encode per-point features V E R xxxx, ensuring equivariance within each part. Subsequently,
these features are transformed into invariant representations, eliminating relative poses to facilitate inter-part information propagation and global aggregations.
part-aware equivariant network is message- passing module weighted by the input segmentation y (Fig. B left). Intuitively, given point x,, E X with latent feature V,, for its neighborhood points Xm E N, only allow information propagation between and Xm if they belong to the same part. When the part segmentation y is soft mask, we compute the probability Prom of Xn, and Xm. belonging to the same part by Pam Ynym m Yas Ym [0, 1]2xP" and weight the local message passing with
Segmentation-weighhe message passing. Key
Prom (p( Vn. V.. MEN
Σ
Σ mEN
Pro
(14)
Proof of inter-pan part equivariance for this module can be found in the supplementary material. Network architecture. Fig. right shows the overall network architecture for segmentation label updates. Given an input pair y), we first extract per-point SE(3) (E3uivcquivariian local feature V NxCx3 within each part masked by y. is then passed through sequence of weighted convolutions with the massage passing defined in Eq. [4 To enable efficient inter-part information exchange, after each pessage-passing layer, we compute a global invariant feature through a per- point VN-invariant layer, followed by a global pooling and a concatenation between per-point and global features. After the point convolutions, the per-point equivariant features V, are converted to invariant features S,, with inter-part poses canceled. We then apply a global pooling on S, to obtain a global shape code, which is decomposed into part codes Q..... Q as in [[3]) Finally, we compute point-part assignment score for all (Sn, Qp) pairs (resulting in an N x P tensor) and apply a softmax activation along its P dimension to obtain the updated segmentation y'. For instance segmentation without part permutations (Sec. 4d,, modify the computation of the part feature Q by replacing the global feature decomposition with point feature grouping based on y.
For pointeloud networks with set-invariance across all points, directly restricting the upper bound of the network Lipschitz using weight truncations [r2] 24, 87] will greatly harm network expressivity as it limits the output range each point. On the other hand, the space of all possible segmentations
[0, XP has extremely high dimensionality, making the computation of Lipschitz regularization losses [69] rather inefficient. Therefore, we do not explicitly constrain the Lipschitz constant of our network. Nevertheless, we confine all operations involving y to local neighborhoods per point, shown in Eq. IAl This practical approach generally ensures small Lipschitz constant for the network in most scenarios. The stability of fixed-point convergence will studied in Sec.
5 Experiments
5.1 Articulated-object part segmentation
We begin by evaluating our method on articulated-ohject part segmentation using on four categories of the Shape2Motion dataset [/2]]: washing machine, oven, eyeglasses, and refrigerator. To demon-

---

Setting
Unseen states
Unseen states unseen instances
Washing machine
Eye- glasses
Refrige- rator
Washing machine
Eye-
Refrige-
Category
Oven
Oven
glasses
rator
PointNet
5
46.18 18 a 1s4 44.08 a ast 38,90 11841 38.37
112.22
46.15
1.18 45. 29
a 9.54 39.21 a:1 39.00 1:213
DGCNN 7
46.78 4:37 44.30
m184 33.35
#2096 39.70
14.05
46.60
14m 46.69
122.6 34,96
22244 40. 40.13 #1381
VNN [T
47.35
1.9
53.64
11327 57.08
+1555 52.36
11.39
47.09
1M4
51.01
=1.00 62.98
11009 48.49 +11.5
VNN-Inv
46.58
62.32
37.24
126
60.78
+15.77
46.66
67.35
16.50
38.19
10226 63.18
Ours
82.32
81.91
1101 77.78
11445 77.26
a 1.79 84.99
11:06
82.84
8.3 78.65
1033 73.93 1:400
Table 1: Shape2Motion results. Numbers are segmentation IoU multiplied by 100.
Training state
Inference iterations
Figure 4: Banach iterations on Shape2Motion. The network is trained on rest-state objects (left) and tested on novel articulation states.
strate the generalizability of our approach, we train our model on objects in a single rest state, such as ovens with closed doors (as depicted in Fig. 5L left). This training setup aligns with many synthetic datasets featuring static shapes [9. 52]. Subsequently, we assess the performance of our model on ar- ticulated states, where global and inter-part pose transformations scenarios. We evaluate our model on both unseen articulation states the training instances (Tab. II left) and unseen instances (Tab. I1 right). In both cases, the joint angles are uniformly sam- pled from the motion range for each category. We compare our network to the most widely adopted object-level part segmentation networks PointNet [57]] and DGCNN [[ZS] without equivariance, plus We also compare to a VNN variant, named VNN-Inv, which first computes per-point local (E(()--eqiiaariant features and then converts them to invariant features and applies invariant message passing. This is a strategy adopted by [[8] for human seg- mentation. All IoUs are computed for semantic parts. In the case of eyeglasses and refrigerators. which have two parts sharing the same semantic label, we train our network with three motion part inputs but output two semantic part labels. This is feasible because our weighted message passing (Eq. [T4 is agnostic to the number of parts.
applied, replicating real-world
VNN [05] with global (E((3)-cquivarian)
Fig. A demonstrates our iterative inference process novel states after training on the rest states. Fig. 5 shows our segmentation predictions on the four categories. The inter-part pose transfor- mations can cause significant changes in pointcloud geometry and topology, yet our method can robustly generalize to the unseen articulations with inter-part equivariance.
5.2 Multi-Object Scans
We also test our instance segmentation framework (without part orders) on multi-object scans.
Segmentation transfer on DynLab. DynLab [30] is collection of scanned laboratory scenes, each with 2-3 rigidly moving solid objects captured under 8 different configurations with object positions randomly changed, We overfit our model to the first configuration of each scene and apply it to the 7 novel configurations, transfering the segmentation from the first scan to the others via inter-object cquivariance.
We compare our method to a variety of multi- body segmentation methods, including motion- based co- co-segmentation methods (DeepPart [85], NPP [25]], MultiBodySyne [30I), direct segmentation pre-
diction models (PointNet++ [58], MeteorNet [19)), geometry-based point grouping
[ZZ]), and pre-trained indoor instance segmentation modules (PointGrouu [33]]. The baseline se- tups follow [[0]] Tab. Q shows the segmentation loUs. One thing to note is that motion-based co-segmentation can also be viewed as learning inter-object equivariance via Siamese training, yet they cannot reach the performance of equivariance by construction.
7

---

Training sample
Inference unseen states
Inference on unseen states unseen instances
VNN
Ours
VNN
Ours
Figure 5: Shape2Motion part segmentation results. Global poses are aligned here for visualiza- tion but we do not assume it in our inference.
Syathetic trsining samples
Inference on real scans
Method
Seg. loU
PointNet++ [58]
39.4
MeteorNet 19
71.8
DeepPart 88
66.3 17.2
NPP [
7.
SO(3)
Ward-linkage [
88.6
PointGroup [GS]]
72.4 12.5
MultiBodySyne [G0]
94.0 3.1
Pile
Ours
95.5
Table 2: DynLab segmentation.
Figure 6: Chair scan segmentation.
Synthetic real chair scans. We train our model using a synthetic dataset constructed from clean ShapeNet chair models [9] with all instances lined up and facing the same direction (Fig. B left). We then test our model on the real chair scans from [[T with diverse scene configurations (Fig. right). The configurations range from easy to hard, including: Z (all chairs standing on the floor), SO(3) (chairs laid down), and Pile (chairs piled into clutters). Remarkably, our model with inter- object equivariance demonstrates successful generalization across all these scenarios, even in the most challenging Pile setting with cluttered objects.
5.3 Ablation Studies
Network operators and convergence. We show the importance of our locally confined message passing and (E(3)-features (Eq. [ the convergence of test-time iterations. As opposed to extracting part-level equivariant features, one can also apply pose canonicalizations to each part individually and employ non-equivariant learning methods transformations with translation component in R and rotation component in SO(3). the for- mer can canonicalized by subtracting the part centers and the latter by per-part PCA. Thus we compare our full model with two ablated versions with part-level equivariance replaced by canoni- calization: PCA, where we canonicalize both the part translations and rotations and simply employ a non-equivariant DGCNN [75] for the weighted message passing; VNN, where we only canonicalize the part translations but preserve the -qquiiaaaaa structures in the network.
the canonical space. For SF(3)-
Tab. Bl shows the segmentation IoUs of the full model and the ablated versions on the Shape2Motion oven category. espite canonicalizations being agnostic to per-part SE(3) transformations, they rely on absolute poses and positions, which breaks the locality of network operations, resulting in sig- nificant increase the Lipschitz constant of the network. Consequently, when any local equivariant operator is replaced by canonicalization, the network performance experiences a drastic drop.
To further examine the convergence ranges, we test the three models under different y0)) initializa- tions by adding random noises ş 2/(0, 1) to yg according to (()) (1 a) + ag, [o, 1). Fig. Ø shows the performance changes of the three models with gradually increased noise levels

---

10
Method
Equiv. Pose cano.
Seg.
IoU
PCA
rot.+trans.
31.22 7.65
PCA
VNN Ours
SO(3)
trans.
27.86+ 13.07
04
SE(3)
82.84 1: 8.13
0.0
0.4
0.0
Table 3: Ablations of per-part equivariant operators replaced by pose canonicalization. Numbers are loU multiplied by 100.
Initializaatonn noise level
Figure 7: Model performances with ini- tializations of different noise levels.
Lipschitz
Weight trunc. [A
Reg. loss (rand.)
Reg. loss (rand.)
Reg. loss (adv.) [69]
None
constraint
Norm
60
100
loU (x (x0))
_3.99_24 31.3
71.29 11807
43.99 2240
80.68 1:00
Table 4: Network performances under different Lipschitz constraints.
on ( The PCA rotation canonicalization has theoretically unbounded Lipschitz constant, mak- ing the network unable to converge stably even within a small neighborhood of Yg. The SO(3)- equivariant VNN with translation canonicalization perfectly overfits to the ground-truth fixed point but exhibits rapid decline in performance as the initial y() deviates from Ygt. In contrast, our -l3uiv-equivariant network demonstrates stable performance across different noise levels.
Lipschitz constraints.. We also provide a study of different Lipschitz constraining methods under different norms in Tab. E For weight truncation [[22]. we the lo -Lisschizz constraint and set the per-layer Lipschitz upper bound to be 0.99999. For regularization losses, we set the overall network Lipschitz threshold to 0.99. The adversarial sampling [69]] loss only works with 12-norm. We reduce the total number of points from 2048 to 1024 and the GNN neighbors from 40 20 due to the large memory consumption for the loss computations. Based our observations, not having an explicit Lipschitz constraint and using the SE(3)-equivariant message passing work best for the current situation, yet none of these existing Lipschitz constraining methods are helpful to the network performance. We also plot the 12-norm regularization losses in the supplementary material, showing that they are zero almost everywhere.
is the loosest
6 Conclusions
In this work, we propose Banana, which provides both theoretical insights and experimental analysis of inter-part equivariance. While equivariance is typically examined from an algebraic perspective, we approach it from analysis standpoint and obtain strict formulation of inter-part equivariance the convergent point of an iteration sequence. Based on our theoretical formulation, we propose novel framework that co-evolves segmentation labels and per-part -E(3)-qquivariance, showing strong generalization under both global and per-part transformations even when these transforma- tions result in subsequent changes the pointeloud geometry or topology. Experimentally, show that our model can generalize from rest-state articulated objects to unseen articulations, and from synthetic toy multi-object scenes to real scans with diverse object configurations.
Limitations and future work. While our local segmentation weighted message passing reduces the Lipschitz constant of the network, practically providing stability during test-time iterations, it is important to note that not explicitly bounded or regularized. Lipschitz bounds and regulariza- tions for set-invariant networks without compromising network expressivity would be an interesting and important problem for future study. In addition, we study the full SE(3)" action on multi-body systems, but in many real-world scenarios, different parts or objects may not move independently due to physical constraints. Limiting the SE(3)" -actiin to its subset of physically plausible motions can potentially increase our feature-embedding conciseness and learning efficiency.
Broader Impacts. Our study focuses on the general 3D geometry and its equivariance properties. Specifically, we investigate everyday appliances in common scenes, for which we do not anticipate any direct negative social impact. However, we acknowledge that our work could potentially be misused for harmful purposes. On the other hand, our research also has potential applications in areas that can benefit society, such as parsing proteins in medical science and building home robots for senior care.

---

Acknowledgments and Disclosure of Funding
We gratefully acknowledge the following grants: a TRI University 2.0 grant, a Vannevar Bush Fac- ulty Fellowship. and a gift from the Adobe Corporation awarded to Stanford University; and NSF FRR 2220868, NSF IIS-RI 2212433, NSF TRIPODS 1934960, NSF CPS 2038873 awarded to the University of Pennsylvania.
References
[1] Praveen Agarwal, Mohamed Jelli, Bessem Samet, Praveen Agarwal, Mohamed Jelli, and Bessem Samet. Banach contraction principle and applications. Fixed Point Theory Metric Spaces: Recent Advances and Applications, pages 1-23, 2018.
[2] Jimmy Aronsson. Homogeneous vector bundles and g-equivariant convolutional neural networks, Sam- pling Theory, Signal Processing, and Data Analysis, 20(2):10, 2022.
[3] Serge Assaad, Carlton Downey, Rami Al-Rfou, Nigamaa Nayakanti, and Ben Sapp. Vn- Vn-transformer: Rotation-equivarian attention for vector neurons. arXiv preprint arXiv:2206.04117 2022.
[4] Alan Baddeley. Working memory. Science, 255(50444:556-555, 1992.
Alan Baddelcy. Working memory: Theories, models, and controversies. Annual review of psychology, 63:1-29, 2012.
[6] Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithu synthesis with recurrent networks: Logical extrapolation without overthinking. arXiv preprint arXiv:220.008826 2022.
[7] Stefan Andreas Baur, David Josef Emmerichs, Frank Moosmann, Peter Pinggera, Bjöm Ommer, and Andreas Geiger. Slim: Self-supervised lidar scene flow and motion segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13126-1311 2021.
[8] Aseem Behl, Despoina Paschalidou, Simon Donné, and Andreas Geiger. Pointflownet: Learning repre- sentations for 3d scene flow estimation from point clouds. arXiv preprint urXiv:1806.021700 2, 2018.
[9] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 1512.03012, 2015.
[10] Evangelos Chatzipantaziss Stefanos Portigkiozoglou, Edgar Dobriban, and Kostas Daniilidis. Se
(3) eqquivarri attention networks for shape reconstruction in function space. arXiv preprint arXiv:2204.02394 2022.
[11] Haiwei Chen, Shichen Liu, Weikai Chen, Hao Li, and Randall Hill. Equivariant point network for 3d point cloud analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14514-14523, 2021.
[12] Yujin Chen, Matthias Nieliner, and Angela Dai. 4dcontrast: Contrastive learning with dynamic correspon- dences for 3d scene understanding. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 23-27, 2022, Proceedings. Part XXXII, pages 543-560. Springer, 2022.
[13] Zhiqin Chen, Kangxue Yin, Matthew Fisher, Siddhartha Chaudhuri, and Hao Zhang. Bae-net: Branched autoencoder for shape co-segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8490 8499, 2019.
[14] Taco S Cohen, Mario Geiger, Maurice Weiler. A general theory of equivariant cnns on homogeneous spaces. Advances in neural information processing systems, 32, 2019.
[15] Congyue Deng, Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12200-12209, 2021.
[16] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-lincar independent components estimation. arXiv preprint arXiv:1400.80 1410.8516, 2014.
[17] Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 00005 2016.
10

---

[18] Haiwen Feng, Peter Kulits, Shichen Liu, Michael J Black, and Victoria Abrevaya. Generalizing neural human fitting to unseen poses with articulated (3) equivariance. arXiv preprint arXiv:2304.1052 10528.. 2023.
[19] Jiahui Fu, Yilun Du, Kurran Singh, Joshua B Tenenbaum. and John J Leonard. Robust change detection based neural descriptor fields. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2817-2824. IEEE, 2022.
[20] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se transffooorr 3d roto coto-translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-1981, 2020.
[21] Ross Girshick. Fast r-c... In Proceedings of the IEEE international conference on computer vision, pages 1440 1448, 2015.
[22] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.
[23] Christoph Goller and Andreas Kuchler. Learning task-dependent distributed representations by backprop- agation through structure. In Proceedings of International Conference on Neural Networks (ICNN'96) volume 1, pages 347 352... IEEE, 1996.
[24] Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning. 110:393-416 2021.
[25] David Hayden, Jason Pacheco, and John W Fisher. Nonparametrie object and parts modeling with lie group dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 7426-7435. 2020.
[26] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961 119129992222 2017.
[27] Carolina Higuera, Siyuan Dong, Byron Boots, and Mustafa Mukadam. Neural contact fields: Tracking extrinsic contact with tactile sensing, arXiv preprint urXiv:2210.09299, 2022.
[28] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[29] Sepp Hachreiter and Jürgen Schmidhuher. Long short-term memory. Newrol computation, 978) 1780, 1997.
[30] Jiahui Huang, He Wang, Tolga Birdal, Minhyuk Sung, Federica Arrigoni, Shi-Min Hu, and Leonidas Guibas. Multibodysyne: Multi-hody segmentation and motion estimation via 3d scan synchronization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7108-
7118, 2021.
[31] Shengyu Huang, Zan Gojeic, Jiahui Huang, Andreas Wieser, and Konrad Schindler. Dynamic 3d scene analysis by point cloud accumulation. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23 27, 2022, Proceedings, Part XXXVIII, pages 674 690. Springer, 2022.
[32] Ajinkya Jain, Rudolf Lioutikov, Caleb Chuck, and Scott Niekum. Screwnet: Category-independent artic- ulation model estimation from depth images using screw theory. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 13670- 13677. IEEE, 2021.
[33] Li Jiang, Hengshuang Zhao, Shaoshuai Shi, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and Pattern recognition, pages 4867-488 4876, 2020.
[34] Zhenyu Jiang, Cheng-Chie Chun Hsu, and Yuke Zhu. Ditto: Building digital twins of articulated objects from interaction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 5616-5626, 2022.
[35] Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias Issa, and James DiCarlo. Evidence that recurrent circuits are critical to the ventral streams execution of core object recognition behavior. Nature neuro- science, 22(6))974-92) 97,3 2019.
[36] Oren Katzir, Dani Lischinski, and Daniel Cohen-On Or. Shape-pose disentanglement using se equivarrit vector neurons. In Computer 23-27, 2022, Proceedings, Part III, pages 468 484. Springer, 2022. 2022..
2022: 17th European Conference. Tel Aviv, Israel, October

---

[37] Yuki Kawana, Yusuke Mukuta. and Tatsuya Harada. Unsupervised pose-aware part decomposition for 3d articulated objects. arXiv preprint 10.04:2110.0441, 2021.
[38] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
[39] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning, pages 2747-2755. PMLR, 2018.
[40] Jiahui Lei and Kostas Daniilidis. Cadex: Learning canonical deformation coordinate space for dynamic surface representation via neural homeomorphism. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2022.
[41] Jiahui Lei, Congyuc Deng, Karl Schmeckpeper, Leonidas Guibas, and Kostas Danilliiss. Efem: Equiv- ariant ncural field expectation maximization for 3d object segmentation without seene supervision. arXiv preprint arXiv:2303. 15440, 2023.
[42] Jiahan Li, Shitong Luo, Congyue Deng, Chaoran Cheng, Jiaqi Guan, Leonidas Guibas, Jian Peng, and Jianzhu Ma. Directed weight neural networks for protein structure representation learning. arXiv preprint arXiv:2201. 13299, 2022.
[43] Ruibo Li, Guosheng Lin, and Lihua Xic. Self-point-flow: Self-supervised scene flow estimation from point clouds with optimal transport and random walk. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15577-15586, 2021.
[44] Xiaolong Li, Yijia Weng, Li Yi, Leonidas J Guibas, A Abbott, Shuran Song, and He Wang. Leveraging se (3) equivariance for self-supervised category-level object pose estimation from point clouds. Advances in Neural Information Processing Systems, 15355-11-1,,,,, 2021.
[45] Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint 1604.
2016.
[46] Cheng-War Wei Lin, Tung-I Chen, Hsin- Ying Lee, Wen-Chin Chen, and Winston H Hsu. Coarse-to fine point cloud registration with se (3)-equivariant representations. arXiv preprint arXiv:2210.02045 2022.
[47] Fangyu Liu. Shuaipeng Li. Liqiang Zhang. Chenghu Zhou, Rongtian Ye, Yuebin Wang. and Jiwen La. 3denn-dqn-ran: A deep reinforcement Icarning framework for semantic parsing of large-scale 3d point clouds. In Proceedings of the IEEE international conference on computer vision, pages 6678-5687, 2017.
[48] Xingyu Liu, Charles R Qi. and Leonidas Guibas. Flownet3d: Learning scene flow in 3d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 529-537, 2019.
[49] Xingyu Liu, Mengyuan Yan, and Jeannette Bohg. Meteornet: Deep learning dynamic 3d point cloud sequenees. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9246- 9255, 2019.
[50] Xueyi Liu, Ji Zhang, Ruizhen Hu, Haibin Huang, He Wang. and Li Yi. Self-supervised category- level articulated object pose estimation with part-level se (3) equivariance. arXiv preprint arXiv:2302. 142668 2023.
[51] Shitong Luo and Wei Hu. Diffuusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[52] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi. Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scat seale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 909-918. 2019.
[53] Jiteng Mu, Weichao Qiu, Adam Kortylewski, Alan Yuille, Nuno Vasconcelos, and Xiaolong Wang. A-sdf: Learning disentangled signed distance functions for articulated shape representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13001-13011 2021.
[54] Haoran Pan, Jun Zhou, Yuanpeng Liu, Xuequan Lu, Weiming Wang. Xuefeng Yan, and Mingqiang Wei. So (3)-p So equivarraee learning for 6d object pose estimation. arXiv preprint arXiv:2008.08338 2022.
12

---

[55] Adrien Poulenard and Leonidas J Guibas. A functional approach to rotation equivariant non- non-linean linearities for tensor field networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13174-13183, 2021.
[56] Gilles Puy, Alexandre Boulch, and Renaud Marlet. Flot: Scene flow point clouds guided by optimal transport. In Computer Vision-ECCV 2020: 16th Europaan Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII, pages 527-544. Springer, 2020.
[57] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[58] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++:: Deep hierarchical feature learning point
metric space. Advances in neural information processing systems, 30, 2017.
[59] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning. pages 1530- 1538. PMLR, 2015.
[60] Hyunwoo Ryu, Jeong-Hoon Lee, Hong-in Lee, and Jongeun Choi. Equivariant descriptor fields: Se (3)-eqvvvaiiant energy-based models for end- no--to-end visual robotic manipulation learning. arXiv preprint urXiv:2206.00 083221, 2022.
[61] Rahul Sajnani, Adrien Poulenard, Jivitesh Jain, Radhika Dua, Leonidas Guibas, and Srinath Sridhar. Condor: Self-supervised canonicalization of 3d pose for partial shapes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16969-16979, 2022.
[62] Avi Schwarzschild, Eitan Borgnia, Arjun Gupta. Furong Huang, Uzi Vishkin. Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems, 34:6695 6706, 2021.
[63] Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B Tenenhaum, Alherto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-cquivariaant ohject representations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 6394-6400.
IEEE, 2022.
[64] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 20X0020010002502. 2020.
[65] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[66] Yang Song, Jaseha Sohl-Dickste Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Paole. Score-hased generative modeling through stochastic differential equations. arXiv preprint 1/Xiv:201..13456, 2020.
[67] Ziyang Song and Bo Yang- Oge: Unsupervised 3d object segmentation from rigid dynamics of point clouds. arXiv preprint arXiv:2210.04458, 2022.
[68] Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thics, and Matthias Nießner. Diffuscene: Scene graph denoising diffusion prohabilistic model for generative indoor scene synthesis. arXiv preprint ar0iv:2303.. 14207, 2023.
[69] Dávid Terjék. Adversarial lipschitz regularization. arXiv preprint 1o0vv: 2019.
[70] Hugues Thomas, Ben Agro, Mona Gridseth, Jian Zhang, and Timothy D Barfoot. Self-supervised learning of lidar segmentation for autonemous indoor navigation. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 14047-14053. IEEE, 2021.
[71] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-an translation translation-qquivariann neural networks for 3d point clouds. arXiv preprint Xiv: 182.1802.002222 08219, 2018.
[72] Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
[73] Jiang Wang, Yi Yang. Junhua Mao, Zhiheng Huang. Chang Huang, and Wei Xu. Cnn-ran: A unified framework for lubti image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2016.
13

---

[74] Xiaogang Wang, Bin Zhou, Yahao Shi, Xiaowu Chen, Qinping Zhao, and Kai Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8876-8884 2019.
[75] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Aom Transactions On Graphics (tog), 38(5):1-12. 2019.
[76] Yuqi Wang. Yuntao Chen, and ZHAO-XIANG ZHANG. 4d unsupervised object discovery. Advances in Neural Information Processing Systems, 35:35563 35575, 2022.
[77] Joe H Ward Jr. Hierarchical grouping to optimize an objective function. Journal of the American statistical association, 58(301):2362244 1963.
[78] Qiuhong Anna Wei, Sijie Ding, Jeong Joon Park, Rahul Sajnani, Adrien Poulenard, Srinath Sridhar, and Leonidas Guibas. Lego-net: Learning regular rearrangements of objects in rooms. arXiv e-prints, pages 2xX1, 2023.
[79] Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Coordinate independent convolu- tional networks-isometry and gauge equivariant convolutions on riemannian manifolds. arXiv preprint arXiv:2106. 060.00000000 2021.
[80] Thomas Weng. David Held, Franziska Meier, and Mustafa Mukadam. Neural grasp distance fields for robot manipulation. arXiv preprint urXiv:2211.026477 2022.
[81] Yinshuang Xu, Jiahui Lei, Edgar Dobriban, and Kostas Danilidis.. Unified fourier-based kernel and nonlinearity design for equivariant networks on homogeneous spaces. In International Conference on Machine Learning, pages 24596 24614. PMLR, 2022.
[82] Zhengrong Xue, Zhecheng Yuan, Jiashun Wang, Xueqian Wang, Yang Gao, and Huazhe Xu. Useek: Unsu- pervised (3)-euiivrii 3d keypoints for generalizable manipulation. arXiv preprint arXiv:2209.1338 13864, 2022.
[83] Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Point- flow: point cloud generation with continuous normalizing flows. Proceedings of the IEEE/CVF international conference on computer vision, pages
2019.
[84] Xiaoqing Ye, Jiamao Hexiao Huang. Liang Du, and Xiaolin Zhang. recurrent neural networks with context fusion for point cloud semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 403-417, 2018.
[85] Li Yi, Haibin Huang, Difan Liu, Evangelos Kalogerakis, Hao Su, and Leonidas Guibas. Deep part induc- tion from articulated object pairs. arXiv preprint arXiv: 1188888097777 2018.
[86] Hong-Xing Yu, Jiajun Wu, and Li Yi. Rotationally equivariant 3d object detection. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1456-1464, 2022.
[87] Bohang Zhang, Du Jiang. Di He, and Liwei Wang. Rethinking lipschitz neural networks and certified robustness: A boolean function perspective. In Advances in Neural Information Processing Systems, 2022.
[88] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and complction through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-5833 2021.
[89] Minghan Zhu, Maani Ghaffari, and Huei Peng. Correspondence-ffee free point cloud registration with so (3)- equivariant implicit shape representations. In Conference on Robot Learning, pages 1422. PMLR. 2022.
14


Reviews:
[{"review_id": "RbRfZjApWy", "reviewer": "Reviewer_iMm7", "summary": "The paper proposes a neural network architecture that is equivariant to transformations in SE(3) for each object part independently. The network is part of a fixed-point framework where the network is trained with a single step but during testing an iterative approach is used that converges to the desired segmentation.", "questions": "See the weaknesses section.", "limitations": "The limitations are well addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a fresh idea to solve the problem of part segmentation that takes into consideration the different movable parts that compose an object and makes the network equivariant to transformations of those.\n\n- The paper provides theoretical derivations that motivate the solution proposed in the paper.\n\n- The paper shows the effectiveness of the solution by providing several experiments.", "weaknesses": "Although I like the paper, I believe the evaluation could be improved by including other types of equivariant networks:\n\n- For example, neural networks based on equivariant operations such as group convolutions or steerable convolutions have by construction equivariance wrt the object parts too. Although these methods would allow information flow between object parts, most of the object parts would remain equivariant to transformations of those. \n\n- Moreover, network architectures that work only with the intrinsic information of the shape should also be included. Graph Convolution networks or equivariant mesh convolutions would also maintain equivariance of object parts. These networks are commonly used to segment people in different poses, which is a related problem to the one addressed in the paper.\n\n- As the paper states, there is no guarantee that L < 1. The paper states that weight truncation could restrict the upper bound of L but harm the expressivity of the network. An experiment where this is studied would improve the paper.\n\n- Lastly, I do not like the acronym Banana since it is not really an acronym of the title. There is no need to include an acronym on the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback and for appreciating our work! Here’re our responses to your questions and comments and we hope that they could address your concerns as well:\n\n**Equivariant convolutions.**\n\nThe VNN baseline we compare to is also using a graph convolution network backbone. These types of methods face the dilemma of local neighborhood size – when the neighborhoods are too small, it’s more agnostic of part motions but the local features are less expressive, and the contrary when the neighborhoods are large. We also add a baseline comparison with invariant-feature message passing (PDF Table 2, also discussed below), which is an extreme case where local equivariant features are only extracted in the first layer.\n\n**Intrinsic methods.**\n\nWe agree that intrinsic methods are highly relevant to our task and will add discussions of these works to the paper. Compare to intrinsic methods, our method has the following superiorities:\n- Rigid part motions preserve geometric distances in many cases, however, this doesn’t hold when topological changes exist, e.g. oven door closed to oven door opened, where the intrinsic operators such as the Laplacian cannot stay constant (PDF Figure 3). And most intrinsic methods (either spectral methods like the functional map, or spatial methods like the graph convolution) are developed based on the Laplacian operator.\n- Another limitation of intrinsic methods is that they cannot do part segmentation with duplicated parts (like the chairs example in Figure 6, main paper), as the same local geometries can only be classified as the same labels. In other words, our method can give “equivariant” (order 1) outputs, but intrinsic methods can only give “invariant” (order 0) outputs.\n- Also, implementing intrinsic operators on pointclouds is much more non-trivial than on meshes, especially when noises are also considered.\n\nIn addition, we also add a baseline network which first computes per-point local SE(3)-equivariant features, converts them to invariant features, and applies invariant message passing (PDF Table 2), which is a strategy adopted for human segmentation in [1].\n\n[1] Feng, Haiwen, et al. \"Generalizing Neural Human Fitting to Unseen Poses With Articulated SE (3) Equivariance.\" arXiv preprint arXiv:2304.10528 (2023).\n\n**Network Lipschitz.**\n\nWe have added a paragraph of (informal) theoretical explanation of why the SE(3)-equivariant message passing is helpful to small network Lipschitz constants. The PDF file is directly sent to AC as it contains TeX equations that cannot be rendered by MathJax here in the text box.\n\n\nIn addition, the convergence itself actually doesn’t rely on the network being contractive with $L<1$, as $[0,1]^P$ is a compact convex space and the Brouwer fixed-point theorem guarantees the existence of fixed-points for any continuous functions, which can be found by Newton fixed-point iterations. However, we developed our theory on top of the Banach fixed-point theorem/iteration for the uniqueness of the fixed point, which is for the proof of equivariance.\n\n\nWe also provide a study of different Lipschitz constraining methods under different norms (PDF Table 1 and Figure 1). In practice, none of these existing Lipschitz constraining methods are helpful to the network performance (PDF Table 1). We also plot the $l_2$-norm regularization losses (PDF Figure 1) and they are zero almost everywhere.\n\n\nDirectly evaluating the network Lipschitz is also non-practical, as the space $[0, 1]^P$ is of very high dimension and the sampling in it is very inefficient. This inefficient sampling also makes the behavior of the Lipschitz regularization losses less understandable and controllable, as they’re not regularizing the entire space but only on some sparsely sampled points, which may bring unexpected natures to the space.\n\nOverall, based on our observations, not having an explicit Lipschitz constraint and using the SE(3)-equivariant message passing work best for the current situation. But we also agree that, as we discussed in our limitation section, Lipschitz bounds and regularizations for set-invariant networks (which norm to use and how to constrain) would be an interesting and important problem for future study.\n\n**Acronym “Banana”.**\n\nIn fact, the acronym “Banana” comes from a meme in some math departments where students jokingly call “Banach space” the “Banana space”. But we are very sorry for not being aware of its negative connotations in English and we will consider changing or removing it."}, {"review_id": "ewYDaEroYF", "reviewer": "Reviewer_DYXN", "summary": "The paper propose a Banach, an approach for part-based point-cloud segmentation. In particular, the authors propose an approach to enforce equivariance of the part-segmentation, by construction.  They propose a fixed-point framework with one-step training and iterative inference. They propose a part-aware segmentation network. ", "questions": "- what is the run-time of the approach? How does it compare to previous approaches? ", "limitations": "yes ", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The results are very convincing - the proposed approach seem to significantly outperform previous approaches", "weaknesses": "I do not see any direct weaknesses but I have very little knowledge about this field. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback and for appreciating our work! Here’re our responses to your questions and comments and we hope that they could address your concerns as well:\n\n**Inference and training time.**\n\nIn our experiments, we set k=20 iterations for evaluation. But in practice, we plot the IoU w.r.t. the number of iterations (PDF Figure 2) and observe that the network prediction converges within ~k=5 iterations. We will add these information to our paper.\n\nThe training time is the same as the standard training frameworks as it only takes a single-step prediction with the ground-truth labels. If one wants to further incorporate Lipschitz regularization losses into the training, it will take more time for the loss computation, especially if it uses adversarial sampling which needs to compute the network gradients and sample iterations. But currently we are not incorporating such regularizations.\n"}, {"review_id": "a15ZjdTPTL", "reviewer": "Reviewer_tamA", "summary": "The paper proposes an equivariant network for part-based (or multi-object) point cloud segmentation. The approach is equivariant to separate SE(3) transformations of each part/object. This is ensured by introducing a Banach fixed-point network. The network takes the point-could and the current segmentation as input, and iterates till convergence. The proposed part-aware equivariant network employs the Vector Neurons method to achieve equivariance for the individual parts. A segmentation-weighted message passing then adds communication between the different parts. Experiments are performed on objects from the Shape2Motion and DynLab datasets.", "questions": "See weaknesses. \n\nMoreover:\nWhich other tasks could be suitable for this method?", "limitations": "They are well addressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "-\tNovel work based on very interesting and elegant ideas.\n-\tTheoretically sound.\n-\tVery well written and clear.\n-\tGood illustrations.\n-\tComparison with several methods.\n", "weaknesses": "1.\tThe experimental evaluation is limited to very small-scale datasets. It is not clear how the method would scale to larger datasets and more complex scenes, e.g. in an automotive setting, or from terrestrial Lidar scans. It would be good if the authors could discuss this in more detail.\n\n2.\tIt seems that the authors always train on the exact instance which is also encountered during inference. How would the method perform if it encounters a new type of, e.g., oven, after being trained on a dataset of different ovens. \n\n3.\tI did not find how the other methods in table 1 and 2 were trained. Is data augmentation used? How would data augmentation impact the performance of e.g. PintNet++ or MeteorNet?\n\n4.\tI did not find discussion on inference and training time.\n\n5.\tIt would be very interesting to see some plots of the IoU w.r.t. the number of iterations, to understand the convergence behavior of the model.\n\n\nIn summary, I think that this is a very interesting and solid work. I find no significant weaknesses. Although it would be appreciated if the authors can answer and address my comments in a rebuttal to further strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback and for appreciating our work! Here’re our responses to your questions and comments and we hope that they could address your concerns as well:\n\n**1. Generalization to larger datasets and more complex scenes.**\n\nFor larger scenes, we believe a major difficulty would be the network architecture. Currently, we are using the Vector Neuron framework for equivariance and the message passing for encouraging small network Lipschitz. But in larger scenes, more complex network structures are usually important, e.g. transformers, down/up-sampling, voting schemes, and how to incorporate equivariance and Lipschitz properties into these complex networks is yet underexplored.\n\n**2. Inference on novel instances.**\n\nIn Table 1 right and Figure 5 right in the main paper, our method is tested on novel instances which are not seen in the training set. We also show in Figure 6 (main paper) that after training on the clean synthetic samples, our method can be applied to real scans with some noise. We visualize more paired training and test instances of different states only for a better illustration of the equivariance properties.\n\n**3. Data augmentation.**\n\nWe further compare our methods to the baselines under different data augmentation settings: no augmentation, global pose augmentation $SE(3)$, and per-part pose augmentation $SE(3)^P$. The results are in Table 3 in the attached PDF file.\n\n**4. Inference and training time. 5. Convergence behavior of the model.**\n\nIn our experiments, we set k=20 iterations for evaluation. But in practice, we plot the IoU w.r.t. the number of iterations (PDF Figure 2) and observe that the network prediction converges within ~k=5 iterations. We will add this information to our paper.\n\nThe training time is the same as the standard training frameworks as it only takes a single-step prediction with the ground-truth labels. If one wants to further incorporate Lipschitz regularization losses into the training, it will take more time for the loss computation, especially if it uses adversarial sampling which needs to compute the network gradients and sample iterations. But currently, we are not incorporating such regularizations.\n\n**Moreover: Which other tasks could be suitable for this method?**\n\nWe think one task that may be very suitable for our method is tracking. As the changes between consecutive frames are small in tracking, one can probably use the segmentations from timestep $t$ as an initialization for the iterations at timestep $t+1$, which might be very helpful for the convergence. And the equivariance properties, on the other hand, may help the tracking network achieve better inter-frame consistency."}, {"review_id": "0XUZ6lt5Pk", "reviewer": "Reviewer_T7gc", "summary": "The authors introduce a method for object part segmentation of point clouds that is equivariant/invariant to SE(3) part transformations. The core of their method is a neural network with points and segmentation as input and segmentation as output. The network is assumed to be contractive and is then used to perform Banach fixed point iterations towards the correct segmentation during inference. The Banach fixed point network is embedded in a larger architecture using Vector Neurons. For implementation, a message passing network is used, which uses the input segmentation for determining message strength. \n\nThe proposed method is evaluated for articulated object part segmentation (on Shape2Motion) and for segmentation of multi-object scans (on DynLab), where it compares favorably against baselines.", "questions": "- I suspect that equation 1 wants to express linear blending of part transforms. However, I don't think it is correct: (1) sum of matrix $\\mathbf{R}$ and vector \\mathbf{t}, (2) $\\mathbf{t}_p$ not weighted by $\\mathbf{y}_{np}$ . Could the authors clarify?\n- Please provide further evidence that the network indeed behaves contractive or a theoretical justification for the Lipschitz regularization via local message passing.\n- Please discuss or provide missing comparisons with previous methods for part segmentation on Shape2Motion.", "limitations": "The authors discuss limitations and societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The theoretical framework of Banach iterations is elegant and an interesting perspective on iterative networks during inference. In practice, similar ideas on point clouds have been there before, understanding such networks as reweighting functions for iterative reweighting least square schemes, EM iterations, etc. However, I think there is value in this specific perspective, especially since it comes with a convergence guarantee under Lipschitz constraints. To my knowledge, this is a novel contribution.\n- The presented algorithm seems to solve the given tasks well, outperforming the given baselines clearly (however, there are some concerns regarding missing baselines, see below).\n- The paper is nicely presented and easy to follow, given the slightly more complex nature.\n- The result of generalization from synthetic to real chairs is strong", "weaknesses": "- The first part of the paper is all about the consequences of a Lipschitz-constraint network and then there is no Lipschitz constraint in practice. The authors claim that the local message passing serves as some type of Lipschitz regularization. However, there is no reference given for this claim and I don't know one either.  I think it would make the paper stronger if this aspect would be supported by experiments or theory, which is isn't.  The authors only ablate on model performance under different noise levels, which does not give a full picture about convergence.\n- In articulated object part segmentation,  the paper seems to leave out important comparisons to previous work, e.g., [1,2,3]. All three papers are referenced in related work but not compared against. It would be good if the authors would provide a comparison or a discussion, why the comparison is not necessary.\n\nI think this is a very interesting paper with a new perspective on iterative networks. However, the above points are a bit concerning, which is why I am not fully convinced to give a better score.\n\n[1] Kawana et al.: Unsupervised pose-aware part decomposition for 3d articulated objects.\n\n[2] Kawana et al.: Neural star domain as primitive representation\n\n[3] Chen et al.: BAE-NET: Branched Autoencoder for Shape Co-Segmentation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback and for appreciating our work! Here’re our responses to your questions and comments and we hope that they could address your concerns as well:\n\n**I suspect that equation 1 wants to express linear blending of part transforms. However, I don't think it is correct: Could the authors clarify?**\n\nYes, it is a typo. Thanks for pointing it out! It should be $\\sum_{p=1}^P \\mathbf{y}_{np}(\\mathbf{x}_n\\mathbf{R}_p+\\mathbf{t}_p)$. We made this mistake because we were writing $(\\mathbf{R}_p, \\mathbf{t}_p)$ together as $\\mathbf{T}_p$ in the beginning and changed it in the last minute but forgot to change the expressions.\n\n**Please provide further evidence that the network indeed behaves contractive or a theoretical justification for the Lipschitz regularization via local message passing.**\n\nWe have added a paragraph of (informal) theoretical explanation of why the SE(3)-equivariant message passing is helpful to small network Lipschitz constants. The PDF file is directly sent to AC as it contains TeX equations that cannot be rendered by MathJax here in the text box.\n\n\nIn addition, the convergence itself actually doesn’t rely on the network being contractive with $L<1$, as $[0,1]^P$ is a compact convex space and the Brouwer fixed-point theorem guarantees the existence of fixed-points for any continuous functions, which can be found by Newton fixed-point iterations. However, we developed our theory on top of the Banach fixed-point theorem/iteration for the uniqueness of the fixed point, which is for the proof of equivariance.\n\n\nWe also provide a study of different Lipschitz constraining methods under different norms (PDF Table 1 and Figure 1). In practice, none of these existing Lipschitz constraining methods are helpful to the network performance (PDF Table 1). We also plot the $l_2$-norm regularization losses (PDF Figure 1) and they are zero almost everywhere.\n\n\nDirectly evaluating the network Lipschitz is also non-practical, as the space $[0, 1]^P$ is of very high dimension and the sampling in it is very inefficient. This inefficient sampling also makes the behavior of the Lipschitz regularization losses less understandable and controllable, as they’re not regularizing the entire space but only on some sparsely sampled points, which may bring unexpected natures to the space.\n\nOverall, based on our observations, not having an explicit Lipschitz constraint and using the SE(3)-equivariant message passing work best for the current situation. But we also agree that, as we discussed in our limitation section, Lipschitz bounds and regularizations for set-invariant networks (which norm to use and how to constrain) would be an interesting and important problem for future study.\n\n**Please discuss or provide missing comparisons with previous methods for part segmentation on Shape2Motion.**\n\nThe assumptions and training/test setups of [1, 2, 3] are different from ours in the following aspects:\n-  [1, 2, 3] are unsupervised **co-segmentation**, ([1] uses a GAN loss, and [2, 3] follows the slot-attention-like techniques), but we are following the standard supervised training/test frameworks for semantic/part segmentations.\n- Most importantly, [1, 2, 3] require the training data to have objects at **all articulation states**, but we train our method only on limited states (e.g. only the rest state) and show its generalization to unseen states.\n- Another minor point is that [1, 2, 3] also needs watertight meshes/implicits for their training, but our method can work on pure pointcloud data.\n\n[2, 3] doesn’t apply directly to articulated objects, but similar ideas are incorporated in [4] with part-level SE(3)-equivariance for articulated object segmentation (also discussed in the related work section) – and the two arguments above hold for [4] as well. Also note that although [4] shows that they feed their segmentation and pose predictions back to their feature extractor in their pipeline figure, they only do a two-step coarse-to-fine prediction and are not actually using an iterative framework – and also they only leverage per-part SE(3)-equivarinace but don’t provide any theoretical insights for inter-part equivariance.\n\nWe will add these discussions to our paper.\n\n[1] Kawana et al.: Unsupervised pose-aware part decomposition for 3d articulated objects.\n\n[2] Kawana et al.: Neural star domain as primitive representation\n\n[3] Chen et al.: BAE-NET: Branched Autoencoder for Shape Co-Segmentation\n\n[4] Liu, Xueyi, et al. \"Self-Supervised Category-Level Articulated Object Pose Estimation with Part-Level SE (3) Equivariance.\" arXiv preprint arXiv:2302.14268 (2023)."}, {"review_id": "fsphwTkBzn", "reviewer": "Reviewer_4UMu", "summary": "This paper considers an important problem in learning on point clouds -- the equivariance under the SE(3) group. \nNamely, the authors address the requirement of inter-part equivariance, essential for handling real-world scenarios, where an object can consist of multiple moving parts, or a scene can contain multiple objects that undergo different rigid transformations. \n\nThe key observation is that segmentation is necessary to define such per-part equivariance, which the authors are the first to do in a strict manner. \nFurthermore, the proposed fixed-point framework with one-step training and iterative inference is used to demonstrate that the per-step equivariance induces an overall equivariance upon convergence. \nThe developed inter-part equivariant message-passing network with stable convergence is experimentally shown to have strong generalization under different scene configurations, even those changing the point cloud geometry/topology. \n\nOverall, the paper provides a sound theoretical framework for a complex practical problem. \n\n---------------------------------------\nPOST REBUTTAL\n\nThe authors comprehensively addressed my questions and concerns.\nGiven the provided answers and rebuttal overall, I maintained my positive assessment of the paper.", "questions": "Q1. The first part of the title is often associated with negative connotations in English; see, e.g., [0]. The authors might want to revise it so as not to cause unnecessary misunderstanding. \n\nQ2. How can the proposed method be used to perform point cloud classification? \n\nQ3. The declaration of $φ$ in (14) is missing. \n\nQ4. What is the number of runs in the experiment statistics presented in the tables in Section 5? Especially important for Table 2.\n\nQ5. Can the authors explain the large standard deviation of the results in Table 1? Why does the proposed model, in most cases, perform better on Unseen states + unseen instances than Unseen states (see Table 1)?\n\nQ6. Ablation studies: Was the inherent ambiguity of PCA-based canonicalization (see, e.g., [1]) taken care of? Could authors elaborate on the details of this experiment? \n\nQ7. In the theory of the VN framework [2], which the authors' method is based on, the pose-effect cancelation is achieved by means of the inner product of the equivariant features, which also cancels the effect of reflections. Could the authors run a simple experiment showing if their method is actually E(3)-equivariant and discuss it? \n\nQ8. Further noise-stability analysis of the method, where the noise is applied to the input point cloud (not just the segmentation mask), would be beneficial. \n\nQ9. It would make the paper more accessible to a broader audience if the authors included an informal motivation of using the Banach fixed-point theorem and iterations. \n\n \nA non-exhaustive list of typos: \n\n- Line 148 \" and we\" --> \", we\" \n\n- Line 277 \"espite\" --> \"despite\" \n\n-------------------------------------------------\n\n[0]  \"banana.\" Farlex Dictionary of Idioms. 2015. Farlex, Inc 14 Jun. 2023 https://idioms.thefreedictionary.com/banana \n\n \n\n[1] Li et al. (2021), A Closer Look at Rotation-Invariant Deep Point Cloud Analysis \n\n  \n\n[2] Deng et al. (2021), Vector neurons: A general framework for SO(3)-equivariant networks", "limitations": "The authors adequately addressed the limitations and broader societal impact.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1. The paper is very well-written, and the structure is sound. The illustrations are clear and instructive, which helps to comprehend the presented theory. \n\n  \n\nS2. The proposed view of inter-part equivariance as a co-evolving interplay between geometry and segmentation is novel and compelling. \n\n  \n\nS3. The experimental validation supports well the claims stated in the contribution.", "weaknesses": "W1. A discussion on the (time-)complexity of the proposed method is missing.  \n\n- It is unclear how efficient the iterative inference is. \n- Besides, a comparison of the proposed model complexity and the baselines is missing. \n\n  \n\nW2. The details of the hyperparameter choice, including the iterative inference part, are missing.  \n\n- Crucially, the optimal number of iterations $\\textbf{k}$ is unspecified, nor is its effect on the complexity with regards to W1.  \n\n- The same applies to the motivation of the chosen size of the radius of the ball query, $r$, and the maximum number of points in the local neighborhood, $k$. \n\n  \n\nW3. Experiment Section 5.1: the selected Shape2Motion shapes have at most 3 parts with two different semantic labels.  \n\n- I wonder how the proposed method would perform on Shape2Motion shapes with more than 2 different part labels, e.g., motorbike or bicycle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThanks for your feedback and for appreciating our work! Here’re our responses to your questions and comments and we hope that they could address your concerns as well:\n\n**W1. Time complexity of the proposed method is missing.**\n\nIn our experiments, we set k=20 iterations for evaluation. But in practice, we plot the IoU w.r.t. the number of iterations (PDF Figure 2) and observe that the network prediction converges within ~k=5 iterations. We will add this information to our paper.\n\nThe training time is the same as the standard training frameworks as it only takes a single-step prediction with the ground-truth labels. If one wants to further incorporate Lipschitz regularization losses into the training, it will take more time for the loss computation, especially if it uses adversarial sampling which needs to compute the network gradients and sample iterations. But currently we are not incorporating such regularizations.\n\n**W2. Details of the hyperparameter choice.**\nAs mentioned above, for the experiments in the main paper, we set k=20 iterations for inference, but on average the network converges within ~k=5 iterations.\n\nThe input pointcloud contains 2048 points and is scaled to [-1, 1]. For the message passing layers, we set a neighborhood radius r=0.3 with maximum number of 40 points. An ablation study on the radius is shown in Table 4 of the attached PDF. When the radius is too small, the network fails to extract useful local features; and when the radius is 0.3~0.4 the network performance is relatively consistent.\n\nWe will add these details to the paper.\n\n**W3. Experiment Section 5.1: the selected Shape2Motion shapes have at most 3 parts with two different semantic labels.**\n\nWe demonstrate our method on these object categories mostly because their part motions are more obvious, which helps us to show our “inter-part equivariance”. For categories like motorbike or bicycle, the part motions are relatively small (e.g. wheels rotating). We believe for several parts, our method can also work well. But if it's tens of parts like scene segmentation, the dimension of the iteration space $[0,1]^P$ greatly increases and it may cause difficulties.\n\n**Q1. Acronym “Banana”.**\n\nIn fact, the acronym “Banana” comes from a meme in some math departments where students jokingly call “Banach space” the “Banana space”. But we are very sorry for not being aware of its negative connotations in English and we will consider changing or removing it.\n\n**Q2. How can the proposed method be used to perform point cloud classification?**\n\nAs the fixed-point iteration must be always in the same domain $[0, 1]^P$, the direct input of our framework can only be the segmentation labels. However, if we don’t restrict the whole pipeline to be end-to-end, one can first do segmentation and apply another standard part-aware equivariant network to do other tasks like classification.\n\n**Q3. The declaration of $\\varphi$ in (14) is missing.**\n\nThanks for pointing it out! $\\varphi$ is an MLP representing the edge function in the message passing. We will add the declaration to the paper.\n\n**Q4. Number of runs.**\n\nThe error bars are not w.r.t. different random seeds, but are the standard deviation across different instances in the datasets.\n\n**Q5. The large standard deviation in Table 1. Better results on Unseen states + unseen instances than Unseen states.**\n\nFor the large standard deviation, we notice that unlike many standard segmentation networks where prediction errors often exist locally (e.g. mislabeling a small region), our errors usually happen in a way that it converges to a totally different fixed point, e.g. an oven with its door slightly open, the network converges to a state where another wall of the oven body is labeled as “door” and the door is labeled as “body”. In other words, our errors usually exist globally instead of locally.\n\nThe better performances on unseen instances seem to also happen with all the baseline methods. Our guess is that it may be due to the shape biases between the two sets. For example, in the washing machine category, the majority of the instances have oval doors and the minority have rectangular doors, and there’re more rectangular doors in the test split than in the training split.\n\n**Q6. Inherent ambiguity of PCA-based canonicalization.**\n\nYes, the inherent ambiguity of PCA is eliminated in the canonicalization with sign-flipping based on the mean at each PCA direction. The infinite Lipschitz upper bound of PCA comes from the possibility that a small change in the part assignment $\\mathbf{y}$ can result in a substantial change in the PCA directions. For example, imagine you have a part whose shape is a perfect sphere, now if any point around this part is added to this part (with weight $\\mathbf{y}_n = \\varepsilon$), the point’s direction will immediately become a principle direction of the part – such direction change caused by one single point label change is an example of unbounded Lipschitz.\n\n**Q7. E(3)-equivariance.**\n\nYes, the VN framework we adopt is indeed E(3)-equivariant. A sketchy draft discussing the reflection-equivariance and potentially how to break such symmetry can be found at: https://arxiv.org/pdf/2210.16646.pdf\n\n**Q8. Noise-stability analysis of the method.**\n\nWe show the noise-stability analysis of our method in PDF Figure 4. We apply Gaussian noises to the input pointclouds with different standard deviations ranging from 0 to 0.05 (the value 0.05 is adopted from [1]). In Figure 6 of the main paper, we also show some qualitative results of our method trained on clean synthetic pointclouds and tested on noisy real scans. \n\n[1] Mescheder, Lars, et al. \"Occupancy networks: Learning 3d reconstruction in function space.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\n**Q9. Informal motivation for using the Banach fixed-point theorem and iterations.**\n\nThanks for the suggestion! We will add some motivations to the paper.\n"}]

Author rebuttal:
We thank all reviewers for their constructive feedbacks, and are glad that they find our work presenting novel and compelling ideas as well as convincing results.

Here we provide a brief summary of our response, including additional theoretical explanations and experimental evaluations. Detailed responses are replied to each reviewer separately. We attach a PDF file presenting tables/figures with brief explanations (for more detailed discussions please refer to our text response), and a PDF file directly sent to AC containing the equations that cannot be rendered in the text boxes. We will also add these contents as well as fix the typos in our paper revision.

- Network Lipschitz (iMm7, T7gc)
  - Theoretical explanations. (The file is directly sent to AC through an anonymous link as it contains equations that cannot be rendered by MathJax directly in the text boxes.)
  - Network performance under different Lipschitz constraining methods. [PDF file Table 1 and Figure 1]

- Algorithm complexity (tamA, DYXN)
  - Training and inference time compared to other methods (tamA, DYXN).
  - IoU w.r.t. the number of iterations (tamA). [PDF file Figure 2]

- Evaluations and comparisons (iMm7, tamA, T7gc, 4Umu)
  - Hyperparameters for neighborhood radius (4Umu). [PDF file Table 4]
  - Discussions of equivariant convolutions. Comparison to an invariant message passing strategy adopted for human segmentation (iMm7). [PDF file Table 2]
  - Discussions of intrinsic methods (iMm7). [PDF file Figure 3]
  - Data augmentation for the baseline methods (tamA). [PDF file Table 3]
  - Network stability w.r.t. different noise levels on the input pointclouds (4Umu). [PDF file Figure 4]
  - Discussions of other articulated-object segmentation methods (T7gc).

- Other discussions, including answers to questions, clarifications, and the acronym of the paper title (iMm7, tamA, T7gc, 4Umu)


Meta-review: This paper addresses an important problem of learning equivariance for complex systems such as articulated objects or multi-object scenes. It presents an equivariance formulation that allows for their co-evolution, and proposes a Banach fixed-point network for equivariant segmentation with inter-part equivariance by construction. The paper also gives theoretical results for per-step equivariance and global convergence. Experiments verify the proposed method. Reviewers suggest inclusion of other types of equivariant networks and more technical discussions. It seems that the proposed method has the challenge of being used by other architectures for large-scale learning. The authors are suggested to include these additional discussions and results in the final version. Congratulations!
Decision: Accept (spotlight)