Approximately Piecewise E(3) Equivariant Point Networks
Authors: Matan Atzmon
Jiahui Huang
Francis Williams
Or Litany
Venue: ICLR 2024 poster (2024)
Keywords: E(3) equivariant networks
Primary area: learning on graphs and other geometries & topologies
TL;DR: A network design for functions satisfying bounded approximation error of piecewise $E(3)$ equivariance

Abstract: Integrating a notion of symmetry into point cloud neural networks is a provably effective way to improve their generalization capability. Of particular interest are $E(3)$ equivariant point cloud networks where Euclidean transformations applied to the inputs are preserved in the outputs. Recent efforts aim to extend networks that are equivariant with respect to a single global $E(3)$ transformation, to accommodate inputs made of multiple parts, each of which exhibits local $E(3)$ symmetry.
In practical settings, however, the partitioning into individually transforming regions is unknown a priori.
Errors in the partition prediction would unavoidably map to errors in respecting the true input symmetry. Past works have proposed different ways to predict the partition, which may exhibit uncontrolled errors in their ability to maintain equivariance to the actual partition. To this end, we introduce APEN: a general framework for constructing approximate piecewise-$E(3)$ equivariant point networks. Our framework offers an adaptable design to guaranteed bounds on the resulting piecewise $E(3)$ equivariance approximation errors.
Our primary insight is that functions which are equivariant with respect to a finer partition (compared to the unknown true partition) will also maintain equivariance in relation to the true partition. Leveraging this observation, we propose a compositional design for a partition prediction model. It initiates with a fine partition and incrementally transitions towards a coarser subpartition of the true one, consistently maintaining piecewise equivariance in relation to the current partition.
As a result, the equivariance approximation error can be bounded solely in terms of (i) uncertainty quantification of the partition prediction, and (ii) bounds on the probability of failing to suggest a proper subpartition of the ground truth one.
We demonstrate the practical effectiveness of APEN using two data types exemplifying part-based symmetry: (i) real-world scans of room scenes containing multiple furniture-type objects; and, (ii) human motions, characterized by articulated parts exhibiting rigid movement. Our empirical results demonstrate the advantage of integrating piecewise $E(3)$ symmetry into network design, showing a distinct improvement in generalization accuracy compared to prior works for both classification and segmentation tasks

Published as a conference paper at ICLR 2024
APPROXIMATELY PIECEWISE E(3) EQUIVARIANT POINT NETWORKS
Matan Atzmon' Jiahui Huang" Francis Williams' 1 Or Litany
NVIDIA 2 Technion
{mat zmon, jiahuih, fwilliams, olitany) @nvidia.con
ABSTRACT
Integrating a notion of symmetry into point cloud neural networks is provably effective way to improve their generalization capability. Of particular interest are E(3) equivariant point cloud networks where Euclidean transformations applied to the inputs are preserved the outputs. Recent efforts aim to extend networks that are equivariant with respect to single global E(3) transformation, to accommo- date inputs made of multiple parts, each of which exhibits local E(3) symmetry. practical settings, however, the partitioning into individually transforming regions is unknown a priori. Errors in the partition prediction would unavoidably map to errors in respecting the true input symmetry. Past works have proposed different ways to predict the partition, which may exhibit uncontrolled errors in their ability maintain equivariance the actual partition. To this end, we introduce APEN: a general framework for constructing approximate piecewise-E(() equivariant point networks. Our framework offers an adaptable design guaranteed bounds on the resulting piecewise E(3) equivariance approximation errors. Our primary insight is that functions which are equivariant with respect to a finer partition (compared to the unknown true partition) will also maintain equivariance in relation to the true partition. Leveraging this observation, we propose a compositional design for partition prediction model. It initiates with a fine partition and incrementally transitions towards a coarser subpartition ing piecewise equivariance in relation to the current partition. As result, the equivariance approximation eтror can be bounded solely in terms of (i) uncertainty quantification of the partition prediction, and (ii) bounds the probability of failing to suggest a proper subpartition of the ground truth one. We demonstrate the practical effectiveness of APEN using two data types exemplifying part-based symmetry: (i) real-world scans of room scenes containing multiple furniture-tyrr objects; and, (ii) human motions, characterized by articulated parts exhibiting rigid movement. Our empirical results demonstrate the advantage of integrating piecewise E(3) symmetry into network design, showing distinct improvement in generalization accuracy compared to prior works for both classification and segmentation tasks.
the true one, consistently maintain-
INTRODUCTION
In recent years, there has been an ongoing research effort on the modeling of neural networks for 3D recognition tasks. Point clouds, simple and prevalent 3D input representation, have received substantial focus, leading to point networks: specialized neural network architectures operating on point clouds (Qi al. 2017, Zaheer al., 2017). Since many point cloud recognition tasks can be characterized as equivariant functions, modeling them with an equivariant point network has been shown to be an effective approach. Indeed, equivariant modeling can simplify a learning problem: knowledge learned from input, automatically propagates all input's 's symmetries(Bietts et al.) 2021 Elesedy & Zaidi 2021 Tahmaaebi & Jegelka (203)).
One important symmetry exhibited in point clouds is the Euclidean motions, E(3), consisting of all the possible rigid motions in space. Building on the demonstrated success of E(3) equivari- ant point networks in prior research (Thomas et al. 2018), recent efforts have been dedicated to

---

Published as a conference paper at ICLR 2024
extending E(3) symmetry to model piecewise rigid motions symmetry well (Yu et al. 2022
Lei et al. 2023 Deng et al.] 2023). This extension is valuable since some recognition tasks can be better characterized piecewise E(3 equivariant functions. To support this claim, we turn to the task of instance segmentation within a scene, illustrated by a 2D toy example in the right inset. In the leftmost column, we visualize segmentation predictions by distinct colors. In the middle column, we observe the expected invariant predictions under global Euclidean motion of the entire scene. Finally, in the right column, we showcase invariant predictions under piecewise deformation that allows individual objects to move independently in rigid manner, decoupled from the overall scene's motion.
Incorporating piecewise F(3) symmetry to point networks presents several challenges. The primary hurdle is the unknown partitioning of the input point cloud into its moving parts. While having such a partition makes it possible to implement equivariant design using a E(3) equivariant siamese network across parts (Atzmon et al. 2022). this is often infeasible in real-world applications. For instance, in the segmentation task shown in the inset, the partition is inherently tied to the model's segmentation predictions. Thus, in cases where the underlying partition is not predefined but rather predicted by a (non- dennddeeeerttdd model, any suggested piecewise equivariant model will introduce an approximation error in satisfying the equivariance constraint. We will use the term equivariance approximation error to refer to the error that arises when a function is unable to satisfy the piecewise E33 equivariance constraint (w.r.t. the true unknown partition); see Definition I This equivariance approximation error is inherent unless the partition prediction remains perfectly consistent under the input symmetries. This implies it must be invariant the very partition it seeks to identify. So far in the literature, less attention has been given to piecewise equivariant network designs that offer means to control the network's equivariance approximation error. For example, [Liu et al.) (2023) suggests an initial partition prediction model based on input points' global E(3) invariant and equivariant features. Yu et al. (2022), local-context invariant features are used for the partition prediction model. In both cases, it is unclear how failures in the underlying partition prediction model will affect the equivariance approximation error. Notably, the concurrent work of Deng et al. (2023) also partition
observes the equivariance approximation error. Their work suggests an optimization-basse prediction model based on (approximately) contractive steps, striving to achieve exact piecewise equivariance; errors in the partition prediction model arising from expanding steps and their impact on the resulting equivariance approximation error are not discussed.
In this paper, we propose novel framework for the design of approximately piecewise equivariant networks, called APEN. Our goal is to suggest a practical design that can serve as a backbone for piecewise E33 equivariant tasks, while identifying how elements in the design control the piecewise equivariance approximation error. Our framework is built on the following simple fact. Let G and G' be two symmetry groups for which each symmetry in G' is also in G. i.e., G' G. Then, any G equiv- ariant function is also a G' equivariant function. Thus, we can have an exact piecewise equivariant model, as long as the model partition a proper subpartition of the (unknown) ground-truth one. The right inset illustrates this fact: the piecewise equivariant predictions of vote targets, marked as black dots, are accurate for subpartition of the ground-truth truth partition (left column), whereas an equivariant approximation error arises for a partition that includes a bad part consisting of points mixed from two different parts in the ground truth partition (i.e., the red
dots in the right column). This observation may lead to the following simple model for partition prediction drawing a random partition from the distribution of non-degenerated partitions of size k (i.c., all k parts get at least one point). For such a model, the probability of drawing a bad part reaches 0 increases. In turn, the probability of drawing a bad partition can be used to bound the equivariance approximation error of piecewise equivariant function, as good sub- sabr--attiions induce no equivariance approximation error. Importantly, this approach alleviates the need for additional constraints on the underlying model function to control the equivariance approximation error.
However, this approach needs to be pursued with caution, as increasing the complexity of the possible partitions reduces the expressivity of the resulting piecewise equivariant point network model class. This caveat is especially relevant to the common design using a shared (among parts) E(3) equivariant backbone. Indeed, at the limit where each point belongs to a distinct part, the only shared backbone E(3) equivariant functions are constant. To mitigate potential expressivity issues, our APEN framework employs a compositional network architecture. This architecture comprises
2

---

Published as a conference paper at ICLR 2024
sequence of piecewise equivariant layers, with the complexity of their underlying partition decreasing gradually. Each layer is defined as a piecewise E() equivariant function, which not only predicts layer-specific features but also parametrizes prediction of a coarser partition. This coarser partition serves as the basis for the subsequent layer's piecewise F(3) symmetry group. The goal of this "bottom-up" approach is to allow the network to overcome the issue of ambiguous predictions in earlier layers by learning to merge parts that are likely to transform together, resulting in a simpler partition in the subsequent equivariant layer. Importantly, this design also provides bounds for the piecewise equivariant approximation error of each layer, resulting solely from two sources in the design: (i) uncertainty in the partition prediction model, and (ii) the probability of drawing a bad partition.
We instantiated our APEN framework for two different recognition tasks: classification and part seg- mentation. We conducted experiments using datasets comprising of (i) articulated objects consisting of human subjects performing various sequence movements (Bogo et al. 2017), and (ii) real-world room scans of furniture-type objects (Huang al. 202 The results validate the efficacy of our framework and support the notion of potential benefits in incorporating piecewise E(3) deformations to point networks.
METHOD
2.1 BACKGROUND: EQUIVARIANT POINT NETWORKS
We will consider point networks as functions h U W. where U and W denote the vector spaces for the input and output domains, respectively. The input vector space U takes the form == =2xx with n denoting the number of points in the input point cloud, is the point embedding space dimension (usually d 3), and 2 per-point features: spatial location and an oriented normal vector. Depending on the task at hand, classification, or segmentation, the output vector space W can be W = R or W Ruxe To incorporate symmetries into point network, we consider group G. along with its action g on the vector spaces U and W. Of particular interest in our work is the Euclidean motions group G F((d) defined by rotations, reflections and translations in d-dimensional space. The group action on X é U is defined by g X X + 1t"", with g = (R. t) being an element in E(d) while the action the output Y W varies depending on the task (e.g.. g- Y Y for classification). An important property for our networks h to satisfy is equivariance with respect to G:
hgg X) g. )
Vg G
We consider the typical case of networks h which follow an encoder-decoder structure, i.e., h do e. The encoder U -0+1 + V transforms an input into learnable latent representation V. In our case, V is an F(3) equivariant latent space, up to order type I, of the form V Ra+bx3 with a, being positive integers. The decoder d V + W decodes the latent representation to produce the expected output response which can be invariant or equivariant to the input. Both e and d are modeled as a composition of multiple invariant equivariant layers. Having covered the basics of cquivariant point networks, we will now proceed to describe our proposed framework, starting with the formulation of piecewise E(d) equivariant layer.
2.2 PIECEWISE E(d) EQUIVARIANCE LAYER
We start this section by describing the settings for which we model piecewise E(d) equivariant layer. Let X E U be the input to the layer. Our assumption is that the partition prediction is modeled as (conditional) probability distribution, Qz (E+)" over the k parts partitions X can exhibit.
. 277" {0. nxk with Z1 1, denote Here ΣΦ denotes the probability simplex. Let Z [=T,,.,,, a realization of partition from Qz i.e., Z ~ Qzix-
Let Ž be the unknown ground truth partition of X. An important quantity of interest
X(Q)= = Pz-Qzx (3 (315 < i,j nn (ZZ" );z (22").) ))))
(2)
Note that in fact g (R, 0) on the input normals features.
3

---

Published as a conference paper at ICLR 2024
measuring the probability of drawing a "bad" partition from Q, i... a non--roper subpartition of Z. In that context, a reference partition prediction model is simple which is defined by uniform draw of a partition satisfying 1" Zez > 0 for each j E [A:]. An important property of Qimple is X(Qsimple) ~ as k + To better understand this claim about X simmpl),) one can consider the sequential process generating random & parts partition. Clearly, larger values of k result in each part containing fewer points. Since the probability of drawing the next point from mixed ground-truth parts is independent of lt determined solely by the number of input points and the ground-trute partition, the probability that the next drawn point generated bad part lowers increases. turn, X( Xiiimp)) can serve as useful bound for the resulting equivariance approximation error. Consequently, we opt for model Q that satisfies lim sup X( (Q) X(Qsimple), where the last limit is taken with respect to a hyper-parameter in the design of Q.
More precisely, we suggest the following characterization for Q. Let 8: (Ex)" + IR., satisfying
8(Q) ~ O, whenever Q Qv
with Qu e {0, (0, no n (Ex)". That is, 8 measures the uncertainty in the model's prediction. Our design requirement is that
limsup X(Q) = X(Qsimple). as 6(Q) - 0.
In other words, we suggest constraining Q model to behave in a way such that, as it becomes more certain in how it draws its predictions, the probability of drawing "bad" partition converges to be no worse than the one of the simple model. The functional 5 measures the uncertainty of Q, and is considered as one of the design choices in the modeling of Q. In turn, will used to bound the equivariance approximation error. Fig. T illustrates the qualitative behavior of 6.
Figure 1: The func-
We defer the discussion on how we provide a model for Q supporting these ideas for later. Instead, we start by describing how Qz x is incorporated to model a piecewise equivariant layer.
tional bound 6. Green colors indicate values close to O.
Fixed partition. To facilitate discussion, first assume that Z is fixed, and will start by describing a piecewise (d) equivariant layer with respect to Z partition. Let G = E(d) x.. x (()) be the product consisting of k copies of the Euclidean motions group. For g (g1"". G, define
g. (x, Z) Σ (g) ()-x)) (Ze,
(5)
where 9j X X= X + 1,,4, { is the standard basis in Re I is the vector of all ones in Rd
and @ denotes the Hadamard product between two matrices.
One appealing way to model piecewise F(d)) equivariant function, s ;: :U> {0, 1) 1m U², which
also respects the inherited order symmetry of the part's assignments. is by employing an (d))
equivariant backbone th U - U' shared among the parts (Atzmon et al. 2022 Deng et al. 2023). taking the form:
+ (x 2)
Σ (x Ze,13) o Ze,1".
(6)
The following lemma, whose proof can be found in the Appendix, verifies these properties for vp. U' be a function as in Eq. 6). Let E and ox(:) permutation
Lemma 1. Let th Ux {0, 1) nokk on [k]. Then,
v(g- (), 2)= g (1(((,XX,) Z), Z): (Xx, Z') 1/ (x
for any X T U. Z E (0, and Z' Z,o(i)-
Note that one consider augmenting the design of Eq. (6) with function over orderless repre- sentation of parts Edd invariant features (Maron et al. 2020). Equipped with the construction in Eq. (6), we will now move on to the case where Z is uncertain.
4

---

Published as a conference paper at ICLR 2024
the possible Z. Some simple options for marginalization are i) Φ1 () / X, E 2) as implemented in Atzmon et al. (2022); ii) Φu X) = E (x, 2); and iii) om X) & (X, Z. ). where (Z,) ) earg max, Q( kyy Unfortunately, however, all of these options
Uncertain partition. Incorporating Qzjx into a layer can be done by marginalizing
merely an approximation of piecewise ((d) equivariant function. The scheme di relies on scaling, which can be an arbitrarily bad approximation to the input's geometry. The scheme on relies on the averaging of equivariant point features, which not stable under realization of particular partition Q. Similarly, om is also not equivariant under all possible realizations of Z. However, the equivariance approximation error dm induces can be controlled, as we discuss next.
Bounding the equivariant approximation error. In this work, advocate for layers of the form Φm. The motivation for doing so is that it enables a uniform control over the equivariant approximation error as a function of Q, crucially, without relying on bounding the variation of Φ. This advantage is especially prominent for neural networks, as existing techniques for bounding network's bounded variation, e.g., by controlling the network's Lipshitz constant, impose additional complexity the network architecture and may hinder the training process (Anil et al. 2019). On the other hand, as we will see in the next section, the approximation error Q induces can be controlled explicitly by choice of hyper- parer--arameterss in the parametrization of Q.
The next definition captures our suggested characterization for an approximation error of desired piecewise E(d) equivariant layer:
Definition 1. Let do U + U' be a bounded function with | s M. Let 5: (Ex) - R., satisfying Eq. 3 and Eq. 4 w.rt. Q. Then, à is a (G,Q) equivariant function if and only if for any given
X U, the following is satisfied
Eazx 1 (g.
(6(X). x): Z)|l & (A(Qsimptc) 5(Q)) M
for all g G. We denote the set of (G, Q) equivariant functions by Fo-
The above characterization for the equivariance approximation error can be seen as resulting from two different sources of properties in the partition prediction model: (i) an intrinsic source, as captured by 8, which measures the uncertainty of the model Q, and (ii) an extrinsic source, determined by measure independent from Q captured by X. In addition, the above definition generalizes the notion of exact equivariant function classes. For instance, consider satisfying že, 1 for some fixed j: setting 5 yields that FQ coincides with the class of global E(d) equivariant functions.
To conclude this section, we verify in the following theorem that our construction of Φ indeed falls under the suggested characterization of approximate piecewise F(d) equivariant functions. Proof details are in the Appendix.
Theorem 1. Let Φ: U + U' be of form
((X)
Σ (X (. Z,e,1², F=1
(8)
where (Z.) = Carg max, ((zl));)' and th : U¹ an E(d) equivariant backbone. Then,
Φ Fq.
2.3 Q PREDICTION
So far, we have treated Q as a given input to the layer. In fact, we suggest that Q results from a piecewise equivariant prediction of prior layer. Exceptional is the first layer, for which Q = Qsimple- Given layer output of the form in Eq. (8). we will next describe how Queed is inferred. Note that Q still denotes the given input partition prediction model.
Modeling considerations. As a first attempt, one might consider parametrizing Queed as the softmax of a per-point Q piecewise invariant layer prediction. However, this approach introduces several difficulties, causing it to be unfeasible. Firstly. it is unclear how to supervise during training to predict good sub- partitions of the ground truth partition. Secondly, network optimization could be tricky, since the domain of possible partition solutions has high dimensional combinatorial structure,
5

---

Published as a conference paper at ICLR 2024
especially due design bias for large number of parts in early network layers. Lastly, there is need to model the merging of parts in the imput partition to generate a coarser one.
To address these challenges, propose geometric approach to model Qured, Our suggestion is to set Quead as the assignment scores resulting from the partitioning (i.e., clustering) in R of Q piecewise equivariant per-point predictions. Notably, this suggestion falls under the well-known attention layer (Vaswani et al. 2017, Locatello et al. 2020 Liu et al.) 2023) following query, key, and value structure with 6(2 X being the values and queries, part centers being the keys, and the prediction Q is proportional to the matching score of query to key. One of the advantages of this approach is that Q emerges as an orderlexs prediction with respect to possible parts assignments, thus simplifying the optimization domain. However, it is not clear how this model can (i) control the resulting 5 5(Qure) by means of its design; and (ii) support the merging of parts to constitute a prediction of coarser partition. To this end, we suggest that the part center (keys) predictions are set as the minimizers of an energy that is invariant to Q piecewise ((d) deformations of ((X) (values). We formalize this idea in the next paragraph.
Q Prediction. Let Y [y1."",,.]] Raad denote the first equivariant per-point prediction in o( () e U'. Let [M]]] [m]]] e dRa denote the underlying predicted part centers with which the score of Queed is defined. We define [mj]] as the minimizers of an energy consisting of the negative log- leeellkeihoodd of a Gaussian Mixture Model and regularization term that constraints the KL. distance between all pairs of Gaussians to be greater than some threshold. Let P(Y; (Hg, Ty; (Hz,, )
denote the mixture distribution, parametrized by a. Then, the log-likelihood is log aa) Σi log(E) j-1 n (y:: ()) where ((; ((, denotes the density of an isotropic Gaussian random variable, centered μe with variance I²I. Note that is fixed and is considered hyper-parameter. Then, [M]]]] are defined as
(H),, mT) arg min log P(Y:a) -
DKL
(9)
jag
In turn, the prediction Q.7aa is defined as
Qpred N(yGuj,
(10)
V mmz
Importantly, the above construction yields that as o - 0: i) X(Qured) + A (Q since each random partition is a minimizer of the likelihood functional, and ii) ((Qure)) - 0. In addition, a also controls the sensitivity of Gaussians merge (under a fixed coefficient 7). where larger values encourage Gaussians to explain wider distribution of values y; Thus, setting an increasing sequence of a values across layers supports the gradual coarsening of partitions design. Lastly, note that differentiating the prediction of Qpeed w.r.t. its inputs is not trivial; these details are covered in the next section.
2.4 IMPLEMENTATION DETAILS
Network architecture. We start by sharing the details about the construction of the layer vb in
Eq. 66) given a known partition Z. For that end, we used Frame Averaging (FA) (Puny al. 2022)
with shared pointnet (Qi et al. 2017) network, á. We define our shared equivariant backbone by
4/b (/ @ Ze,17) (x (X @ Ze,17)) F(X0Ze,13)
where F(X @ Ze,13) 1 is the same PCA based construction for an E(d) frame suggested in Puny al. (2022), and (-) the FA symmetrization operator. Then, (x(X, Z) is defined exactly as in Eq. T Since this construction needs to support layers with a relatively large number of parts k. we implement the network th using the sparse linear layers from Choy al. (0019).
In all our experiments, we implemented the encoder as compo- sition of L layers, OG o... Φ1, with L = 4; see Fig. 2 Qsimple is set as the input to Φ1. In fact, Qsimple can be further regulated than the naive suggestion. In practice, we set Qsimple by a Voronoi partition resulting from i furthest point samples from the input X. The exact analysis of X(Qsimpl) ) as a function of n and l is out of the scope of this work we only rely on Eq.
Great
Figure 2: APEN network design.

---

Published as a conference paper at ICLR 2024
Q prediction. For finding minimizer of Eq. 9 used slight modification of the well-known EM algorithm (Dempster et al. (197) that supports the merging of centers closer than the threshold T. Note that during training, the backward calculation requires the derivative of 0Q Since the EM is āe
in an iterative algorithm, this might unnecessarily increase the computational graph of the backward computation. To mitigate this, use the following construction, based on implicit differentiation (Atzmon et al. [ Bai et al., 2019). Let & be minimizer Eq. (9) that is detached from the computational graph and Y. Then, Y; áx) where s (Y; = Va log P(Y;a) ax), known in the literature as the score function (Bishop & Nasrabadi] 2006). We define
a = à a (a)
(11)
where (a) Var (x(Y;))) the fisher information matrix (Bishop & Nasrabadi 2006 caleu- lated at Ō. Importantly, 7 only depends on and does not involve second derivative calculations. be easily verified that minimizer Eq. (9) and that da M(rggmnn, (E(o,, Y))) where BY found in the Appendix.
BY
F.()) denotes the energy defined in Eq. This is summarized in Alg.
Training details. Our framework requires supervision in order to train Qured to approximate the ground-truth partition. To that end, compute the ground-truth Yer e 1ndd to supervise the parts center vote predictions Yi E TR axd of the 1thh layer. We utilize the given segmentation information, to calculate Yor ZCT -X, X, where Z {0, 1] nxk are the ground-truit truth assignments of X Raxd and C' e dx is calculated as the center of the minimal bounding box encompassing each of the input parts. Then, standard L1 loss is added optimization.
loS8A Σ
Y Yarl-
E
3 EXPERIMENTS
We evaluate our method on two types of datasets that fit piecewise E(3) symmetry: (i) scans of human subjects performing various sequences of movements (Loper al., 2015 Bogo al. 2017]] Mahmood et al. (019)), and (ii) real-world rooms scans of furniture-type objects (Huang et al. 2021a). In all of our experiments, we used the ground-truth segmentation maps to extract Yer supervision as described in Sec. 2.4
PointNct
Ours
Onc-shot Example
DGCNN
(Syntheicc)
Large Training Sct
Ours
One-shot Example
Unseen Samples
Figure 3: Qualitative results for me-shot generalization on DynLab dataset (Huang et al. 2021a).
7

---

Published as a conference paper at ICLR 2024
3.1 HUMAN SCANS
We start by evaluating our framework for the task of point part segmentation, a basic computer-vision task with many downstream applications. man body parts segmentation, where the goal is to assign each of the input scan points to part chosen from predefined list. In our case, the list consists of 24 body parts. To evaluate different aspects of our framework, we use three different train/test splits. The first consists of random 190%/10%% train/test split of 41, 461 human models from the SMPL dataset (Loper et al., 2013) consisting of 10 different human subjects as in (Huang et al.) 2021b). This experiment acts a sanity test and ensure our method does not underperform compared to baselines. The second and third splits use the scans from the Dynamic FAUST (DFAUST) dataset (Bogo al. 2017), consist- ing of 10 to 12 different sequences of motions (e.g.. jumping jacks, punching. etc.) for each of the 10 human subjects. In the second split, we divide the data by a random choice a different action sequences for each human. This experiment ensures method can generalize knowledge of action sequences seen in training from one human subject to other human subjects at test time. Finally, in the third split choose the same sequence of movements (e.g., the one-leg jump sequence) to be removed from the training set and be placed as the test set. The last test evaluates the effect of the piecewise E(3) prior, implemented in our method, to generalize to unseen movement.
Specifically, we consider hu-
GT
Baseline
Qurs
GT
Bazeline
Ours
Figure 4: Human body part segmentation.
In Tab. D we report the mean loU(%) score for all 3 tests. As baseline models, opt for PointNet
Method
random
unscen random seq.
unsecn soq.
PointNet
84.4
78.5
$0.1
QQi et al. 2017) and DGCNN (Wang et al. 2019)
DGCNN
82.2
70.3
79.5
as order invariant point networks. For E(3) invari- ant networks, our baselines selection includes Vector
VN
42.4
24.8
33.3
VN-T
63.5
50.9
50.0
83.5
78.1
76.7
Neurons (VN) (Deng et al. 202T). VN-Transformer (VN-T) (Assaad et al.) (023)) Frame/Averaging (FA) (Puny et a (022), and Equivariant Point Network (EPN) (Chen et al. (021) backbone as implemented in the human body part segmentation network de-
EPN
89.6
77.8
84.1
Ours
94.2
92.2
93.5
Table 1:
Mean IoU(%) test score for human body parts segmentation.
scribed in Feng et al. (002)). Fig. 4 shows qualitative test results of unseen random seq. pose (first row) and an unseen random pose (second row). We conclude from the results that (i) our framework is valid backbone with similar expressive power common point network baselines, (ii) our framework utilizes piecewise F33) equivariance to gain better generalization across human subjects than baseline approaches and, (iii) piecewise E'(3) equivariant prior can help to generalize to unseen movements.
Lastly, test the versatility of our framework, we evaluate on point cloud classification task. On that hand, we consider the DFaust subset of AMASS (Mahmood et al. 2019), consisting of 9 human subjects. We define the task of classifying model to a subject. For testing, we use an "out of distribution" test set from PoscPrior (Akhter & Black 2015). The results from this experiment support the usability of our framework for classification tasks as well. The detailed report can be found in the Appendix, including all the syper-parameter used for the experiments in this section.
3.2 ROOM SCANS
In this section, we test the potential of our framework for one-shot generalization. To that end, we employ a dataset scenes capturing a real-world room where the furniture in the room has been positioned differently in each of the 8 scans for each scene. Within each scan, there are 3 to 4 labeled furniture-type objects, including the floor. The task objective is to assign each input point to one of the object instances composing the scene. In addition to the difficulty of segmenting moving objects in the scene, solutions to this task must handle noise and sampling artifacts arising from the scanning procedure. For instance, scans of objects occasionally contain holes or exhibit ghost geometry. Here
8

---

Published as a conference paper at ICLR 2024
we compare two alternative solutions this this task: (1) we only train our method using single scan, and test its generalization to the other seven scans of the same scene. (2) We train baseline networks on the large-scale synthetic shape segmentation dataset from Huang et al. (2021a), which randomly samples independent motions for multiple objects taken from ShapeNet CChng et al. 2015).
In Tab. 2 we report the mean (o)(%) test score for each of the scenes. Fig. shows qualitative results for 2 rooms. Despite only training on a single scan, our model outperforms baselines trained on a large synthetic dataset in 7 out of the 8 test scenes. These results suggest potential advantages of using piecewise E(3) equivariant architectures in single shot setting over the use of large-scale synthetic data. Furthermore, to make baseline approaches work, we employed a RANSAC algorithm to identify the ground plane, with an inlier distance threshold of 0.02 and 1000 RANSAC iterations. In contrast, our method requires no preprocessing since the network can treat the floor as would for any other part of the input data.
Method
Scene
Scene
Scene
Scene 4
Scene 5
Scene
Scene
Scene
PointNet
33.0 8.5
50.2 a 4.3
31.1 a 3.4
38.3 4.3
36.7 6.1
45.2 22.0
57.4 a 1,5
36.6 a 4.6
DGCNN
3.6
38.8 + 10.8
41.8 = 4.9
31.0 2.7
48.9
35.1
$9.5
35.4 6.3
VN
2.8
18.6 1.5
24.7 0.8
15.2
2.14
17.6 1.7
25.6 1.0
23.0 1.2
Ours
88.0 13.0
98.2 0.7
97.4 1.5
96,3
93.2 3.9
2.
83.3 a 13.3
92.2 = 1.8
PointNet (Synthteic)
76.6 22.4
97.3 2.1
91.2 4.8
89.7
91.9 5.1
95.1 1.0
66.6
83.2 4.0
DGCNN (Synthetic)
77.5 a 22.3
93.7 a: 10.9
97.1 a 0.7
84.4 13.0
89.1 16.6
95.6 1.1
76.2 a: 10.6
90.6 a 6.2
VN (Synthteic)
65.5 a 18.7
93.7 a 4.9
80.7 a 17.6
59.3 11.0
92.5 4.9
82.5 a 15.0
77.4 6.1
62.0 a 12.9
Table 2: One-shot generalization on real-world scans from the Dynlab dataset (Huang al.
202 la).
4 RELATED WORK
Global Equivariance. We introduce a novel method for piecewise E(3) equivariance in point
networks. Euclidean group symmetry has been studied in point networks mainly in describing architectures that accommodate global transformations (Chen et al.
2019 Thomas et al. 2018
Fuchs et al., 2020 Chen et al. 202T Deng et al. [021] Assaad et al.
2023 Zisling & Shari
2022
Katzir et al. 2022; Poulenard & Guibas, 2021 Puny et al.] 2022)) ). These was shown to perform
well in various applications inc Including reconstruction (Deng et al. [202 Chatzipantazis et al. 2022
Chen et al. 2022). pose estimation (Li et al. 2021 Lin et al. 2023 Pan et al. 2022] Sajnani et al.
2022 Zhu al. 2022), and robot manipulation (Simeonov et al. 2022 |Higuera et al. 2023) Xue ct al.
2023) tasks. Some works have dealt with respecting the symmetry by manipulating their input representation (Deng et al. 2018: Zhang et al.) 2019 Gojeie et al. 2011). popular line of work
utilizes the theory of spherical harmonics to achieve equivariance (Worrall et al. 2017] Esteves et al.
2018 Liu et al., 2018:: Weiler et al. 2018 Cohen et al. [2018).
Object-Level and Part- Based Equivariance Several works have studied the equivariance of parts. EON YYu et al. 2022) and EFEM (Lei et al., (023) both studied object-level equivariance in scenes. EON used a manually tumed 'suspension' to compute an equivariant object frame in which the context aggregated. In EFEM, instance segmentation is achieved by training a shape prior using shape collection, and employing it to refine scene regions. Instead, we do not assume prior knowledge of the underlying partition. Equívariance for per-part pose estimation in articulated shape was devised in Liu et al. (2023). Yet their self- superviee approach relies on part grouping according to features that are invariant to global rotations which may result in unknown errors when local transformations are introduced. Part-based equivariance was also studied for segmentation in Deng et al. (2023). relying on an intriguing fixed-poine convergence procedure.
CONCLUSION
We presented APEN, a point network design for approximately piecewise F(3) equivariant models. We implemented APEN networks to tackle recognition tasks such as point cloud segmentation, and classification, demonstrating superior generalization over common baselines. On the theoretical side, our work lays the ground for an analysis of piecewise equivariant networks in terms of their equivariance approximation error. The bounds we present in this study serve as merely initial insights on the possibility of controlling the equivariance approximation error, and further analysis of our suggested bounds is marked as an interesting future work. Further extending this framework for other 3D tasks, e.g., generative modeling and reconstruction is another interesting research venue.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors would like to thank Jonah Philion for the insightful discussions and valuable comments. Or Litany is Taub fellow and is supported by the Azrieli Foundation Early Career Faculty Fellowship.
REFERENCES
Ijaz Akhter and Michael J Black. Pose-conditioned joint angle limits for 3d human pose reconstruction. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 1446-1455, 2015.
Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation. In International Conference on Machine Learning. pp. 291-301. PMLR, 2019.
Serge Assaad, Carlton Downey, Rami Al-Rfou, Nigamaa Nayakanti,, and Ben Sapp. Vn-transformer. Rotation-equivarian attention for vector neurons, 2023.
Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, Haggai Maron, and Yaron Lipman. Controlling neural level sets. Advances in Neural Information Processing Systems, 32, 2019.
Matan Atzmon, Koki Nagano, Sanja Fidler, Sameh Khamis, and Yaron Lipman. Frame averaging for equivariant shape space learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 631 631-641, 2022.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
Alberto Bietti, Luca Venturi, and Joan Bruna. On the sample complexity of learning with geometric stability. arXiv preprint arXiv:2106.07148, 2021.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
Federica Bogo, Javier Romero, Gerard Moll, and Michael J. Black. Dynamic FAUST: Register- ing human bodies in motion. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), July 2017.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint 11Xiv:1512.03012, 2015.
Evangelos Chatzipantazis, Stefanos Pertigkiozoglou, Edgar Dobriban, and Kostas Daniilidis. Se (3) (3)-cquivariant attention networks for shape reconstruction in function space. arXiv preprint 023399 2022.
Chao Chen, Guanbin Li, Ruijia Xu, Tianshui Chen, Meng Wang, and Liang Lin. Clusternet: Deep hierarchical cluster network with rigorously rotation-invariant representation for point cloud analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4994-5002, 2019.
Haiwei Chen, Shichen Liu, Weikai Chen, Hao Li, and Randall Hill. Equivariant point network for 3d point cloud analysis. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14514-14523, 2021.
Yunlu Chen, Basura Fernando, Hakan Bilen, Matthias Nießner, and Efstratios Gavves. 3d equivariant graph implicit functions. In European Conference Computer Vision, pp. 485-502. Springer, 2022.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3075-3084, 2019.
10

---

Published as a conference paper at ICLR 2024
Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv:1801.10130 2018.
arXiv preprint
Arthur Dempster, Nan M Laird, and Donald Rubin. Maximum likelihood from incomplete data via the algorithm. Journal of the royal statistical society: series B (methodological), 39(1): 1-22, 1977.
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12200-12209, 2021.
Congyue Deng, Jiahui Lei, Bokui Shen, Kostas Danillidis, and Leonidas Guibas. Banana: Banach fixed-point network for pointcloud segmentation with inter-part equivariance. arXiv preprint arXiv:2305.16314. 2023.
Haowen Deng, Tolga Birdal, and Slobodan llie. Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors. In Proceedings of the European conference on computer vision (ECCV), pp. 602-618, 2018.
Bryn Elesedy and Sheheryar Zaidi. Provably strict generalisation benefit for equivariant models. In International Conference on Machine Learning. pp. 2959-299999999 2969. PMLR, 2021.
Carlos Esteves, Christine Blanccleeeett, Ameesh Makadia, and Kostas Danilidis. Learning so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference Computer Vision (ECCV), pp. 52-68, 2018.
Haiwen Feng, Peter Kulits, Shichen Liu, Michael J Black, and Victoria Fernandez Abrevaya. Gener- alizing neural human fitting unseen poses with articulated se (3) equivariance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7977-7988, 2023.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se (3)-transformer:: 3d roto- translation equivariant attention networks. Advances neural information processing systems, 33: 1970-1981, 2020.
7an Gojeic, Caifa Zhou, Jan D) Wegner, and Andreas Wieser. The perfect match: 3d point cloud matching with smoothed densities. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 5545-5554, 2019.
Carolina Higuera, Siyuan Dong, Byron Boots, and Mustafa Mukadam. Neural contact fields: Tracking extrinsic contact with tactile sensing. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 22576-12582 IEEE, 2023.
Jiahui Huang, He Wang. Tolga Birdal, Minhyuk Sung, Federica Arrigoni, Shi-Min Hu, and Leonidas J Guibas. Multibodysyne: Multi-body segmentation and motion estimation via 3d sean synchroniza- tion. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 7108-7118, 2021a.
Qixing Huang, Xiangru Huang, Bo Sun, Zaiwei Zhang, Junfeng Jiang, and Chandrajit Bajaj. Arapreg: An as-rigid-an as possible regularization loss for learning deformable shape generators. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5815-5825, 2021b.
Oren Katzir, Dani Lischinski, and Daniel Cohen-Or. Shape-pose disentanglement using se (3)- equivariant vector neurons. In European Conference on Computer Vision, pp. 468 484. Springer, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6980. 2014.
Jiahui Lei, Congyue Deng, Karl Schmeckpeper, Leonidas Guibas, and Kostas Daniilidis. Efem: Equivariant neural field expectation maximization for 3d object segmentation without scene supervision, 2023.
Xiaolong Li, Yijia Weng, Li Yi. Leonidas Guibas, A. Lynn Abbott, Shuran Song, and He Wang. Leveraging se(3) equivariance for self-supervised category-level object pose estimation, 2021.
11

---

Published as a conference paper at ICLR 2024
Cheng- Wei Lin, Tung-I Chen, Hsin- Ying Lee, Wen-Chin Chen, and Winston H Hsu. Coarse-to- fine point cloud registration with se (3)equiiaariant representations. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 2833-28444 IEEE, 2023.
Min Liu, Fupin Yao, Chiho Choi, Ayan Sinha, and Karthik Ramani. Deep learning 3d shapes using alt-az anisotropic 2-sphere convolution. In International Conference on Learning Representations, 2018.
Xueyi Liu, Ji Zhang, Ruizhen Hu, Haibin Huang, He Wang, and Li Yi. Self-supervised category-level articulated object pose estimation with part-level se(3) equivariance, 2023.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33:1 1525 11538, 2020.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6): 248:1 1--488866 16, October 2015.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5442-5451, 2019.
Haggai Maron, Or Litany, Gal Chechik, and Ethan Fetaya. On learning of symmetric elements. In International conference on machine learning, pp. 6734-6744. PMLR, 2020.
Haoran Pan, Jun Zhou, Yuanpeng Liu, Xuequan Lu, Weiming Wang. Xuefeng Yan, and Mingqiang Wei. So (3)-pose: So equequraaae learning for 6d object pose estimation. In Computer Graphics Forum, volume 41, pp. 371-381. Wiley Online Library, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
Adrien Poulenard and I Leonidas J Guibas. A functional approach to rotation equivariant non-linearities for tensor field networks. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 13174-13183, 2021.
Omri Puny, Matan Atzmon, Edward J. Smith, Ishan Misra, Aditya Grover, Heli Ben-Hamu, and Yaron Lipman. Frame averaging for invariant and equivariant network design. In International Conference on Learning Representations, 2022. URL CRLhttps: /ttpa/::////eeeeeee/ net fformm?
1d (ddzI0yy IUyj555XR
Charles R Qi. Hao Su. Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Rahul Sajnani, Adrien Poulenard, Jivitesh Jain, Radhika Dua, Leonidas J Guibas, and Srinath Sridhar. Condor: Self-supervised canonicalization of 3d pose for partial shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16969-16979, 2022.
Anthony Simeonov, Yilun Du, Andrea Tagliasacchi. Joshua Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-equivrriant object representations for manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pp.
IEEE, 2022.
Behrooz Tahmasebi and Stefanie Jegelka. The exact sample complexity gain from invariances for kernel regression on manifolds, 2023.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-ant and caasllaton-eeuivariiii neural networks for 3d point clouds. arXiv preprint arXiv: 88888000022299 2018
12

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data, 2018.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5028-5037, 2017.
Zhengrong Xue, Zhecheng Yuan, Jiashun Wang, Xueqian Wang, Yang Gao, and Huazhe Xu. Useek: Unsupervised se (3)-equivariant 3d keypoints for generalizable manipulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 1715-1722. IEEE, 2023.
Hong- -ing Yu, Jiajun Wu, and Li Yi. Rotationally equivariant 3d object detection. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, 2022.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets. arXiv preprint arXiv:1703.06111 2017.
Zhiyuan Zhang, Binh-Son Hua, David W Rosen, and Sai-Kit Yeung. Rotation invariant convolutions for 3d point clouds deep learning. In 2019 International conference on 3d vision (3DV), pp. 204-213 IEFE, 2019.
Minghan Zhu, Maani Ghaffari, and Huei Peng. Correspondence-fres point cloud registration with so (3)-equivariant implicit shape representations. In Conference on Robot Learning, pp. 1412-1422. PMLR, 2022.
Hedi Zisling and Andrei Sharf. Vnt-net: Rotational invariant vector neuron transformers, 2022.
A APPENDIX
A.1 PROOFS
A.1.1
PROOF OF LEMMA 1
Proof. (Lemma Let X E U, Z E {0, 1 and E G. Then,
b (9g-(X,22 (x, Z)
Σ ( (X,2) oe, -e,1-
Σ 3=1
(9) 1=1
X) (Ze, @ Ze,1 Σ 9j 2 (X Ze,1,) Ze,1
where the last equality follows from the fact the th is Edd) equivariant and the second equality from the fact that Ze, o Ze,, 0 for j j'. Lastly, for any permutation (+),), have,
&
k
(x (X
Zea Ze, Σ j-l
Sx o 1), Ze,
13

---

Published as a conference paper at ICLR 2024
A.1.2 PROOF OF THEOREM
Proof. (Theorem Let d: U U' be of the form
x(x)
th (Xo Z,e,1%) o Z,e,1', Σ 5=1
(12)
where (Z.)s;: max Q(2)X),, and w/ : U U³ an E(d) equivariant backbone.
Let A 10 Z. Then,
R Q(A) s [1 ((e,-ee,,))) (1 (1-66 -1(())))
1=1
i=1
where j(t). arg max, Qij. Then, we
12 8(Q) - [ Qis()))
Clearly 8 satisfies conditon
Now, Let Q satisfying condition w.r.t. A. (ZZ")
Let B
31siij n s.t. (zzT).
Then,
( - (Bn (Bn (Bn n AB nA
Note that for BC n A there no equivariance approximation error. For (Bn A), and (BC n A) we can bound using 6(Q). Lastly, Z ε (Bn A means Z, is "bad" partition, thus X(Q) s X simple)) To conclude, union bound composed of the decomposition above to get that,
Eazx 100 (x) 9 (Ø (6(X), 2) (A(Qsimple) 8(Q))
A.2 Q PREDICTION
In this section we provide an empirical validation to the expected behavior of X(Qsimple) as & n. To that end, we examine groups.
2D toy example, featuring n = 14 points partitioned Figure shows this toy example, with distinct colors denoting the ground truth partition. Figure shows a plot X(Q) values for k & [1. 14]. The green line shows X(Q) for the simple Q model, defined by uniform draw of & parts partition, where each part includes at least point. The red line shows X(Q) for a Q model, defined by a Voronoi partition with centers drawn randomly proportionally &s furthest point sampling. Note that as expected, X(Q) as k + n..
Next, we provide in Alg. detailed description of our predic- tion algorithm.
Figure 5: 2D toy example consist- ing of 14 points, partitioned into 3 parts.
A.3 ADDITIONAL IMPLEMENTATION DETAILS
A.3.1 ARCHITECTURE
We start by describing our concrete construction for the encoder, e and d used in our experiments. The network consists of APEN layers of the form,
APEN (n, Gins bin Gouts bout) na + (nux(uuutxxaaa
Then, the encoder consists of the following blocks:
14

---

Published as a conference paper at ICLR 2024
0.5
Simple
Voronoi
2
4
5
6
7
8
9
10
11
12
13
14
Figure 6: The probability drawing bad partition, > ((Qsimple) ) as - n, for 2D toy example with = 14 points.
APEN(n, 07,51
APEN (n. 17,5, 17,5)
APEN(2, 0,2,17,5) ~ APEN(n, 21,
The decoder consists the following block for the segmentation task:
APEN(n, (n, 65, 21, 24,
and for the classification task:
APEN(1, 65, 25,
Each APEN block is built equivariant backhone, implemented with Frame Averaging. In turn, the backbone symmetrize a pointnet network t). We now describe its details.
The network consists of layers of the form
FC(n, dins dout): : X v (XW 16") MaxPool(n, dmn X 1 1[max Xe,]
where X E uxd,, W e R.daxdon b Rh are the learnable parameters, 1 E R is the vector of all ones, [.] is the concatenation operator, e; is the standard basis in IR and v is the ReLU activation. We used the following architecture for the first APEN layer:
FC(n, 96) 1 FC(n, 96, 128) La FC(n, 128, 160) La FC(n, 160, 192) 4
FC(n, 192, 224) La MaxPool(n,, 224) La [L1 Lz, Ls. LA. Ls. Lol 4
FC(n, 1024, 256) 4 FC(n, 256, 256) L FC(n, 128,32).
15

---

Published as a conference paper at ICLR 2024
Algorithm Q prediction
Input: T T
0 merge threshold and merge frequency
i 0 (1)) + random furthest point sample of points from Y
J + R
while i << max iter do
74 E. (
Hj + Ei Eeni Tig
To Y,
T, Enr
if i modd f == 0 then
(j.5') + arg min (KL(((H)))NN(:)))
(13))())1500))
d (K(((())))))(
while d< do
TJ - Tt TT
T - (i,')
DK((G()))NGG)) (1,,))))
arg
d (=Dr((((:)))WW
end while
end if
i i- i+1
end while
(Pys TT)) + (g)
(n):
(Mj
Que Nuuan
Output: Quedl (differential) minimizer of E(Y)
For the second and third,
F2,n,32,96) 4 FC(n, 96, 128) 4/3 FC(n, 128,160) 4 FC(n, 160, 192) 4
FC(n, 192,224) 4 MaxPool(n,222) 224) Le [L1 L2, Ls, Ls, Ls] 4 1024,1024,250 256) l FC(n, 256, 256) 4 FC(n, 128,32).
And lastly,
32,9,32,96) 3 FC(n, 96, 128) 1/3 FC(n, 128, 160) f FC(n, 160, 192) 4 FC(n, 192,224) 1 MaxPool(n, 224) Fgg [L, L2, Ls, L1, Ls, Ls] 4
FC(n,1024 256) The FC(n, 256,256) 1/2 FC(n, 128, 128).
A.3.2 HYPER PARAMETERS AND TRAINING DETAILS
We set a; = (0.002, 0.005, 0.008, 0.1). The number of iterations the EM was networks using the ADAM (Kingma & Bal 2014) optimizer, setting the batch size to 8. We set fixed learning rate 0.001. All models were trained for 3000 epochs. Training was done on a single Nvidia V-100 GPU, using PYTORCH deep learning framework (Paszke et al. 2019)
trained our
A.4 ADDITIONAL RESULTS
In this section, we present visualizations of the learned partitions Qurel across layers in the APEN encoder. Figure 7 shows the learned APEN encoder layers partitions from the experiment in section
16

---

Published as a conference paper at ICLR 2024
Figure 7: APEN encoder's learned partitions, Q extracted from two test-set examples in the human body segmentation experiment. In each group of clements, the leftmost column shows Quead partitions, with subsequent layers' partitions ordered left-to-right, culminating in the rightmost column that shows the encoder's last layer partition.
Figure 8: APEN encoder's learned partitions, Qued, extracted from the shot segementation experiment. In the top row, layer partitions of a single training example are shown, while the bottom row shows layer partitions of unseen test example. The leftmost column shows Queed partitions, with subsequent layers' partitions ordered left-to-right, culminating in the rightmost column that shows the encoder's last layer partition.
3.1 while Figure g shows partitions from the experiment in section 3.2 Each input point assigned distinctive colors according to max, Qurea It is worth noting that progressing from left to right, the predicted partitions tend to become coarser, behavior encouraged by setting the hyper-parameter
d 01.
A.5 SUBJECT CLASSIFICATION EXPERIMENT
Method
PointNet
DGCNN
VN
Ours
Accuracy (%)
18.5
32.1
71.4
Table 3: Subject classification accuracy comparison.
Here we provide the results the point cloud classification experiment described in the main text. Fig. shows several typical examples from the considered split. Note the relatively large difference the distribution of poses. Tab. logs the quantitative evaluation, validating our framework's superiority in this case well.
17

---

Published as a conference paper at ICLR 2024
Test Samples Figure 9: Training and test set visualization for the subject classification task.
Train Samples
18


Reviews:
[{"review_id": "nSHgYx1iiI", "reviewer": "Reviewer_gDD2", "summary": "This study aims to extend global E(3)-equivariant point networks by introducing support for multiple components, each subject to its local E(3) symmetries. Prior approaches have unbounded errors when their partition predictions diverge from the actual data. To address this, this paper proposes a framework called APEN to construct approximate piecewise-E(3) equivariant point networks, which offers guaranteed bounds on the resulting approximation errors.\n\nIn particular, the APEN model starts with a detailed initial partition and gradually merges these partitions into larger segments in successive layers. The efficacy of this method is validated through tests on 3D scans of interior spaces and human movement patterns.", "questions": "I think I don't fully understand this work so I can't make a good judgment yet. It would be greatly appreciated if the authors could clarify the following questions:\n- Page 2, what exactly is defined as \"equivariant approximation error\"? What is the relationship between this and general approximation error? The sentence \"this simple model enables bounding the equivariance approximation error solely by the probability of drawing a 'bad' partition. Crucially, this bound is independent of any required restriction on the resulting piecewise equivariant model function bounded variation.\" is difficult to follow in the introduction without more context, before read all the theorems and technical details afterwards.\n- Equation 2, as I understand it, as long as the two partitions are inconsistent, | $||ZZ^T - Z_{\\ast}Z_{\\ast}^T|| > 0$, even if $Z$ is a finer partition of $Z_{\\ast}$? Is that correct? If not, can you explain why?\n- Theorem 1, is it possible the $\\arg\\max$ would give a set rather than a single value (i.e. ties)? Would that cause any problems?\n- Table 1, can the authors discuss why the performance of vector neurons are so bad, even compared to baselines?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper tackles an important problem: modeling local symmetries for point networks. Previous work provided no guarantees on the approximation error when the predicted partitions are inconsistent with the true partitions.", "weaknesses": "- The paper is a bit hard to follow in general. I think the introduction should provide more high-level pictures and intuitions without talking about technical details. Also see questions.\n- The baselines used are PointNet, DGCNNs, and VectorNeurons, while this work focuses on local symmetries. It would be useful to discuss how previous methods that extend global E(3) point networks to handle local symmetries perform on these tasks. While the authors likely don't need to run experiments for the rebuttal, it would be good to hear their rationale for choosing these particular baselines over other more directly relevant work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ue77I2QFXr", "reviewer": "Reviewer_FhAC", "summary": "This paper introduces a general framework called APEN for constructing an approximately piecewise-E(3) equivariant neural network for point clouds. The model is then used for classification and part-segmentation tasks on two datasets: scans of human objects performing various sequences of movements and room scans of furniture-type objects.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Errors that arise when maintaining equivariance of point clouds are unavoidable in practice. The model presented in this paper can be employed to control these unavoidable errors. To the best of my knowledge, there have been very few works in the literature that can address this problem. This model can also be seen as a generalization of classical global E(3)-equivariant point networks when the error is set to zero.\n- The presented model is constructed in such a way that it can ensure equivariance of partitions of the point clouds, which is more challenging than achieving global equivariance.\n- The paper includes a detailed theoretical analysis with useful intuitive explanations.\n- The model performs well in the two experiments.\n\nIn general, I believe this is a noteworthy paper.", "weaknesses": "- The theoretical explanations in the paper are somewhat challenging to comprehend. For example, why the indicator $\\lambda(Q)$ in Eq. (2) can be used to measure the probability of drawing a bad partition from a non-proper subpartition of $Z_*$. The rationale behind defining $Q_{simple}$ in that manner and the behavior of the number $\\lambda(Q_{simple})$ as $k$ tends to $n$ also need further clarification.\n- The approximation error defined in Definition 1 depends on $M$ which is an unknown and possibly large number. Therefore, it is unclear how this approximation error can be employed to control the errors occurring in experiments. For example, given a positive number $\\epsilon$ how can we design the model in a way that the approximation error does not exceed $\\epsilon$?\n- PointNet and DGCNN are non-equivariant models. Therefore, it may not be fair to compare the proposed model with them. Instead, it would be interesting to assess the efficiency of the proposed model compared to other equivariant models for point clouds, with suitable modifications to make them equivariant for partitions of the point clouds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "86v9YJLQxJ", "reviewer": "Reviewer_vmat", "summary": "The paper proposes a new neural network design that approximates piecewise E(3) equivariance. This design allows to segment better deformable or movable objects such as human body parts or objects in a 3D scene.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I believe this paper tackles a relevant problem, piecewise network equivariance. A solution for this problem will partially solve some open problems in 3D scene understanding such as 3D scan segmentation or instance segmentation by design of the network.\n\nMoreover, the solution proposed is simple and easy to implement: Initialize the network with an initial partition, use E(3) equivariant layers to process the parts, and then use a clustering algorithm to determine the new set of parts from the predicted features.", "weaknesses": "Although I liked the paper and the proposed solution, I believe the paper was very difficult to follow and needs an improved evaluation. In the following paragraphs, I listed some of the things that were unclear to me, maybe I did not fully understand these parts, and some points on how to improve the evaluation of the paper:\n\n- It was not clear to me the statements at the end of section 2.3. How $\\sigma \\rightarrow 0$ makes $\\lambda (Q^{pred}) \\rightarrow \\lambda (Q^{simple})$ ? There is no control over the features that come out from the previous layers and the clustering obtained can be worse than $Q^{simple}$ as defined later in the paper, farthest point sampling with KNN. I suppose here $Q^{simple}$ refers to the random uniform partition and not the Voronoi partition used later. If this is true, later I would not refer to the Voronoi partition as $Q^{simple}$.\n\n- Moreover, how sigma is defined to achieve the desired number of parts (number of group truth parts)? How robust is the proposed method to different choices of $\\sigma$?\n\n- Maybe I missed it, but I think $\\delta$ is never properly defined. Moreover, Fig.1 is not helpful unless more context is given. What Fig. 1 illustrates?\n\n- In the Training details paragraph, $Y$ is used to refer to part centers but in the Q Prediction paragraph is used to refer to per-point features. In this same paragraph, the centers are referred as $\\mu$ instead. Moreover, $\\pi$ is never defined. From the formulas, I suppose it refers to the per-point weights of each of the Gaussians in the GMM.\n\n- In Training details, it is not clear how the loss would work. $Y_{GT} \\in R^{n \\times d}$, but $Y_l \\in R^{k \\times d}$ if they are the part centers as the text indicates. Moreover, is each layer supervised by this loss or only the last one?\n\n- In network architecture, it is not clear which network architecture is used. First, it indicates that PoinNet is used but later SpareConv layers are used. From the appendix, I believe all layers are PointNet layers, but I don't understand where the SparseConv layers come into play.\n\n- Regarding evaluation, I believe it needs significant improvement before publication. Here is a list of possible experiments that would help improve the evaluation:\n\t- Comparison with different choices of sigma in the layers.\n\t- Visualization of the intermediate partition assignments will help understand the behavior of the model.\n\t- Experiments on other datasets, such as ShapeNet part segmentation and real 3D scans such as ScanNet.\n\t- Several relevant baselines missing. Only one global rotation equivariant network is used in the comparisons, Vector Neurons. It would be necessary to compare to network architectures that are locally equivariant such as E2PN, LieConv, or simple graph convolution networks. It is not relevant to show improvement over basic point architectures such as PointNet or DGCNN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)