$\mathrm{SE}(3)$-Equivariant Attention Networks for Shape Reconstruction in Function Space
Authors: Evangelos Chatzipantazis
Stefanos Pertigkiozoglou
Edgar Dobriban
Kostas Daniilidis
Venue: ICLR 2023 poster (2023)
Keywords: shape reconstruction
equivariance
neural fields
attention
3D vision
point clouds

Abstract: We propose a method for 3D shape reconstruction from unoriented point clouds. Our method consists of a novel SE(3)-equivariant coordinate-based network (TF-ONet), that parametrizes the occupancy field of the shape and respects the inherent symmetries of the problem. In contrast to previous shape reconstruction methods that align the input to a regular grid, we operate directly on the irregular point cloud. Our architecture leverages equivariant attention layers that operate on local tokens. This mechanism enables local shape modelling, a crucial property for scalability to large scenes. Given an unoriented, sparse, noisy point cloud as input, we produce equivariant features for each point. These serve as keys and values for the subsequent equivariant cross-attention blocks that parametrize the occupancy field. By querying an arbitrary point in space, we predict its occupancy score. We show that our method outperforms previous SO(3)-equivariant methods, as well as non-equivariant methods trained on SO(3)-augmented datasets. More importantly, local modelling together with SE(3)-equivariance create an ideal setting for SE(3) scene reconstruction. We show that by training only on single, aligned objects and without any pre-segmentation, we can reconstruct novel scenes containing arbitrarily many objects in random poses without any performance loss. 


Published as a conference paper at ICLR 2023
3E ATTENTION NETWORKS FOR SHAPE RECONSTRUCTION IN FUNCTION SPACE
Evangelos Chatzipantazis', Stefanos Pertigkiozoglon"
Edgar Dobriban University of Pennsylvania
University of Pennsylvania
{vaghat, pstefano)$seass upenn.
dobriban@whartee upenn, edu
Kostas Daniilidis
University of Pennsylvania
kostas@cis www.n.uennnncmmmmm edu
ABSTRACT
We propose a method for 3D shape reconstruction from unoriented point clouds. Our method consists of novel SE(3)-equivariant coordinate- basee network (TF- ONet), that parametrizes the occupancy field of the shape and respects the inherent symmetries of the problem. In contrast previous shape reconstruction methods that align the input to regular grid, we operate directly on the irregular point cloud. Our architecture leverages equivariant attention layers that operate on local tokens. This mechanism enables local shape modelling, a crucial property for scalability to large scenes. Given an unoriented, sparse, noisy point cloud as input, we produce equivariant features for each point. These serve as keys and values for the subsequent equivariant cross-altention blocks that parametrize the occupancy field. By querying arbitrary point in space, predict its occupancy score. We show that our method outperforms previous -qqiivaaaa methods, as well as non-equivariant methods trained on (0(3)-augmented) datasets. More importantly, local modelling together with SE(())-qqivvarianc create an ideal setting for SB(3) scene reconstruction. We show that by training only single, aligned objects and without any pre-segmentation, we can reconstruct novel scenes containing arbitrarily many objects in random poses without any performance loss.
INTRODUCTION
Figure I: (Above): A scene-level point cloud produced by individual SE(3) (E(3)-ttaanste transformations of three sparse object point clouds. (Below): Our equivariant reconstruction. The whole scene given as input to network. The network, trained only single objects in canonical poses and agnostic to the number, position and orientation the objects able to reconstruct the scene accurately.
With the advent of range sensors in robotics and in medical applications, research in shape recon- struction from point clouds has seen an increasing activity (Berger et al. (007)). The performance of classical optimization methods tends to degrade when point clouds become sparser, noisier, un- oriented, or untextured. Deep learning methods have been proven useful in encoding shape priors, and solving the reconstruction problem end to end (Riegler al. 2017). Many of these deep learn- ing methods operate on meshes (Wang Zhang 2022 Gong et al.) 2019), voxels (Riegler et al. 2017). and point clouds (Qi al. 2016). While voxels easy manipulate, shape resolution is limited by memory. On the other hand, meshes can guarantee watertight reconstructions, but
"Equal Contribution

---

Published as a conference paper at ICLR 2023
they only handle predefined topology. Point clouds are lightweight in terms of memory, but they discard topology. Recently proposed deep learning methods represent the geometry via a learned occupancy map. signed distance function (SDF). In partieular, the seminal works Mescheder ) Park et al. (219)) inspired many follow-up works (Chen & Zhang [019b) [Genova al. 2020 Sitzmann al. 2019). Such representations can encode arbitrary topologies with an
let al.
effectively infinite resolution.
According to Kendall, "Shape is the geometry of an object modulo position, orientation, and scale" (Kendall 1989). While intensive research in the field (Niemeyer & Geiger 202T Peng et al. 2020) Niemeyer et al. 2020 has led increasingly better results. very few of these methods incorporate reconstruction methods build on the convolutional occupancy network (Peng et al. 2020), while most -Quuieeqivaaiint ar- chitectures (Zhu et al. (021), and their extensions to SE(3) with GraphOnet (Chen et al. (022), use the equivariant modules from Vector Neurons (Deng et al.) 202T). We propose TF-Onet, novel SE(3)-equivariant coordinate-based network for shape reconstruction. Motivated by the SE(3)- transformer (Fuchs et al. 2020), we design a two-level network that uses equivariant attention mod- ules. The first level, acting as encoder, extracts local features from the point cloud by applying self-attention in local neighborhoods around each point. The second level, a cross-attention occu- pancy network, takes input the extracted point features and the coordinates of query point in space, and outputs the value of the occupancy function at the specified query point.
symmetries as an inductive bias for learning. Most cunslation-equivariaa
Even unique objects consist of smaller primitive parts, whose subsets subsequently composed to form large collections of objects. This property extends naturally to scenes that are created by a composition of objects. Our method performs local shape modeling by leveraging the expressivity of equivariant local attention modules and generalizes to novel scenes with novel configurations of objects from classes unseen during training. This property distinguishes our method from similar equivariant works that either use global features (Deng et al. (02T) or per-point features that encode long-range dependencies by using subsampling to expand their receptive field (Chen et al., 2022). Additionally, as we describe in Section 3.3 the use of the Tensor Field framework allows our method to utilize higher order representations in contrast to the previous works which use Vector Neurons and thus are constrained to only use type-0 (scalars) and type-1 (vectors) representations. In Section we provide experimental evidence showcasing how these differences benefit our method in the reconstruction of single objects in arbitrary poses and in the reconstruction of novel scenes.
4,
Our contributions can be summarized as follows:
pordinatee-asedd attention network for learning occupancy fields from sparse point clouds and use it for surface reconstruction.
We propose TF-Onet, novel -E())eaquivaiian,,
Experimentally. we outperform other equivariant coordinate-based networks (Vector Neurons, GraphOnet) and non-equivariant networks (Oecupancy Networks, Convolutional Occupancy Net- works, IFNet, NeuralPull) trained with augmentations.
The most compelling property of our method is that equivariance and local shape modeling allows our network to produce high-quality reconstructions of novel scenes while being trained only on single-aligned objects. These scenes contain arbitrary number of objects in random poses. We show quantitative Saaiiveeaaaaaa 5aland qualitative performanec gap over previous methods in synthetic dataset of randomly placed objects (Seismic dataset). We also show qualitative results on the more challenging Matterport3D (Chang et al., 2017) containing real scenes with unseen object classes.
RELATED WORK
In this section, we discuss previous work on surface reconstruction from input point clouds. We focus on methods that reconstruct the surface of an object by using either an occupancy function or
a SDF For oriented point clouds (with known normal vectors), the occupancy function or the SDF can be constructed by classical methods that do not require learning (Alexa et al. 2003] Kazhdan &l Hoppe 2013). These methods tend to fail in the presence of noise, or when the input point cloud sparse. To surpass such limitations, Mescheder et al. (2019); Chen & Zhang (2019a) proposed
to learn the occupancy function for each input point cloud. Similarly, Park et al. (2019) proposed to learn to infer the SDF of the object's surface from sparse set of SDF values. A limitation of
the above methods is that they use global feature vector- or code- to represent the whole object
(or scene), which limits their ability to generalize to novel scenes objects. More recent methods
2

---

Published as a conference paper at ICLR 2023
Output Occuganry Field
I
Multi-Head SR(R) Attrritionn
Multi-Head StCl)-Attention
Point features
Input Pointctoud
Tinear
Unneor
Linear
Linear
linwar
Wils-
Query Features
Weds Wi,0s
gggse
Contforrennion
[renee aetection)
10 X
Our method consists of two networks E, T. First, to each point T. in the point cloud, we assign simple local feature fi of type-1 (rotating as a vector) which can be the relative position of the point Z. to the centroid of its local neighborhood. Similarly, to each query point q E R assign local feature fq. Given point cloud consisting of pairs (Z,, ft),, the network & applies SE(3)-equivariiat attention to assign to each point E learned feature ff. Finally, given the learned feature-augmented point cloud and the query-feature pair (7. fq), the network T applies SH(3)- equivariant attentonn output the scalar value of the occupancy function at point
Figure 2:
leverage the similarities between local patches of different objects by learning either a combination of local and global features (Genova et al., 2020 Erler et al. 22220), or only local ones (Jiang al.,
2020 Chabra et al. 2020 Tretschk et al. 2020; Boulch & Marlet, 2022, Williams et al.. 2022 Chen
et al. 2022). The composition of these local features allows the reconstruction of scenes containing
a variety of different objects. To define the local neighborhoods for which the local features are
extracted, Jiang et al. (2020); [Cabra et al.] (2020); Tretschk et al. (2020) voxelize the space into regular grid and learn a feature representation for each voxel, while Boulch & Marlet (2022); Chen et al. (2022) learns local feature representation for each input point. In our work, follow the latter approach by using local attention layers (Bahdanau et al. 2015) that dynamically change the attention weights of each point to its neighbors. Attention layers were popularized with the intro- duction of the transformer architecture (Vaswani et al.] 2017 7017. and were later applied in computer vision tasks such as image classification (Dosovitskiy et al. 202T; Wu et al.) (02T). Specifically for point cloud processing tasks, |aan et al. (2021) used both local and global attention for object detection, and Yu et al. (202T) used a transformer for point cloud completion.
There is a large body of work on incorporating known symmetries into the learning process, trac- ing back to Fukushimal (I980); LeCun et al. (1989) with the use of convolutional layers to build translation-cquivariaee
networks. More recent works extend this idea to discrete (Cohen & Welling 2016) and continuous (Weiler et al. 2018b) rotationally invariant architectures. These methods have also been applied beyond Euclidean domains, to spheres (Esteves al., 2018), graphs (Maron ctt al. 2019 Brandstetter et al. 2022), meshes (de Haan et al. 2021), and general manifolds (Co- hen et al. 2019: Weiler et al.] (021). For transformer architectures, Fuchs al. (2020) propose an SE(3)-equivariant transformer by using the results of Tensor Field Networks (Thomas et al. 2018).
Romero & Cordonnier (202T) proposed a framework for constructing linear self-attention layers that are equivariant to arbitrary discrete groups. Operating on point clouds, an -E(3)-equivariant network (Chen et al. 202T) performs pose estimation and classification using SE(3) convolutions.
For the problem of surface reconstruction from point clouds, Convolutional Occupancy Networks (Peng et al. (020) and later methods (Lionar et al., [222]] Tang et al., [22T]] Boulch & Marlet, 2022) use convolutional layers to achieve caanslation-eqivvriianee
Finally, Vector Neurons (Deng et al.) 2021) propose an architecture for -quui-aaui reconstruction from point clouds, extended by GraphOnet Chen et al. (2022) for the SE(3) -Equivaanva case.
3 METHOD
3.1 LEARNING THE OCCUPANCY FIELD
Consider 3D point cloud P ((T,, (()) where F. E R denotes the spatial location of point and fi R.M is an (optional) associated feature. If the features include the normal vectors, the point cloud is called oriented. Otherwise it is called unoriented. We denote the matrix of coordinates by X [ [N] 3 and the matrix of features by F [f...... : ffwl MMXN
Given a point cloud, our goal is to infer the shape from which it was sampled. We represent this shape with occupancy function o" R3 10 whose 1-level set, (% E R² (7) 1).
3

---

Published as a conference paper at ICLR 2023
encodes the volume occupied by the object and whose boundary encodes the surface of the ob- ject. We approximate this function by learning an operator T that acts on an input point cloud and produces a function ôp : 12 + [0. 1] that describes the occupancy score 6p(4) of each position T E RR$. Following Mescheder et al. (211)) we parametrize T by a coordinate-basel neural net- work. In other words, given point cloud P (X, F), and an arbitrary point in space, we predict T[X, F)( (ÿ) E [0, 1]. In contrast to Mescheder et al. (2019), our architecture works directly on the irregular point grid, extracting a local signature describe the point cloud, and constraining the model to achieve SE(3) (E(3)-equivarianc,, thus respecting the symmetries of the problem.
more common, especially in robotics applications, receive featureless point clouds (for ex- ample, from Lidar sensors). Our paper focuses on sparse, noisy, and featureless (in particular, unoriented) point clouds. Since the performance of 'T depends on the expressivity of the input fea- tures F, we first design feature extractor (or encoder) & that produces a point cloud along with ()) features, E (X, F). Overall, (g( = ()) 0). Our design of ' E is founded on two basic principles: local shape modelling and equivariance. Objects can be seen a composition of local parts and local surfaces. At this level, even dissimilar objects may share structure. Local shape modelling leverages this compositionality to learn from multiple shapes in a more effective way. This is particularly helpful for reconstructing novel object classes.
On the other hand, respective of the rigid transformations of the shape. This can further reduce sample complexity. Together, these two properties create an ideal setting for scene reconstruction. Scenes composed of objects which appear in different orientations and positions. Without leveraging compositionality and equivariance, every new configuration of objects would be seen novel scene by the network, and thus learning would be hindered by the combinatorial explosion of these combinations. We discuss how we impose the above properties on T, E in the next sections.
restricts models to produce consistent occupancy maps ir-
We note two more important properties of point cloud processing. First, there is no canonical order- ing on the points, so a simultancous permutation of the columns of X, F describes the same point cloud (permutation equivariance) and second, the number N can vary between point clouds. Thus the input has a set structure, irregularly embedded in a Euclidean domain. Two main architectures that deal with such inputs are Point Cloud Convolutions (Wu et al., (019) and Attention modules (Yu et al.) 2021) l). We opt for the latter in our design because point cloud convolutions, when con- strained to be SE())--quivariant, result in very few free parameters (Thomas et al., 2018) and in particular they eliminate the angular degrees of freedom from learning. Thus the performance is highly dependent on ad-hoc non-linearities (Poulenard & Guibas 2021). Attention modules the other hand are inherently non-linear and the attention kernel can be viewed as a modulation to the angular profile of the weights, thus having more degrees of freedom (Fuchs al. 2020
3.2 USING ATTENTION FOR LOCAL SHAPE MODELLING
Preliminaries on Attention:
We propose to construct our networks E, T as a composition of cquivariant attention modules. Given Nout query tokens Qi Rde and Nin key-value pairs K E Rdg X V E X an attention module can be described by the formula:
Nin
exp (Q)
A(Q;, K,V) Vsoftmax (KTiiaxxxxXT))
V3, E [Nout].
exp (Q Kz)
(QQ..G)) where subscripts index columns. For each query token Qi, the output A is linear combination of the values V,, modulated by the similarity aQQ,, K2) of the query Q, to the corresponding key K,. as imposed by the attention kernel a. In self-attention layers the queries, keys and values are functions of the same set of features, while in cross-attention the query features are distinct and only the keys and values are functions of the same features. We design the feature extractor as composition of L self oeff-attention layers, and the occupancy operator T as composition of L' cross-attention layers.
Self-attention feature extractor E: We associate each point with a query token with the same functional dependency
Q =Wf.
Moreover, we design the self-attention module so that not all queries have access to all the keys, but query attends only keys depending its position. In particular, if T, E (Z.)), with A(.)) the
4

---

Published as a conference paper at ICLR 2023
Ev
Figure 3: (Up): Input point cloud, input query field (type-l), 1), output occupancy field (type-0) and (Down) their roto-tramslations. Here C describes the action on both the query field (ğ. S', q)) depicted above and the point cloud field Fx, (described in the main text); T, are equivariant which is equivalent to the diagram being commutative, i.e., Cn = T Cy and & Ly C. c.
nearest neighbors of T4., we assign to the pair (ii Tfy) key-value token. The functional form of keys and values is
Vi T. (v( E.).. K = -x -
The standard way incorporate relative positional encoding is via concatenation or addition Vaswani et al. 2017). Our functional form Wx(Z, - Æ can be understood as a more gen- eral way to encode the positions of the tokens. This generalization will be important in our case when we impose extra constraints to satisfy rotational equivariance.
The self-attention module of the encoder & at layer 2 gets feature-augmented point cloud (X, F) and produces a new feature-augmented pointcloud (X, (X) where F' is computed as:
F")
exp (W) E.ff)]
F)
E
- E.)ff,
(1)
(EE)(E.) E2,EN(a.) exp (W)J)) ( E:Js))
Since the input point cloud X includes only the locations of the points without additional features, we design function 8 that associates hand-designed (not learned) features to the points. In this work, [']] = S(X) where S(X); = J. 1/IW(a;;) (x)eN((a)) JJ is the relative position from the neighborhood's 'ss centroid. Then, E = _ o... o o g
Cross-attention occupancy network T: We design the occupancy operator T composition of L' cross-attention layers. The input query to T can be any point q E I to which we assign a token Q(7. fg) Wofg. The output of T is the occupancy value of that query location. The keys and values in T are constructed from the output the feature extractor { = (F$ == [f {) In particular, if ž E (2), we create a key-value pair for (q. with the form K 7, $5 (T, T3 and Vi 7. f = Wy (ā, 0.f.- Given the cross-attention module T outputs new feature-augmented point g (Ø. 11+1). These new features are computed as:
point g² = (ā. f?),
exp I( ( 0ff (1) EnyENs(A exp (W))))(WW - Wile, Dsf (2)
T|X 1F)
Σ
Since the input query includes only its location q and not additional features, first we associate with fixed (not learned) feature fę == S' S'(X
[x( ExtENx 4y where Nx (g) M(rr mingex || 1.12)). Then after passing the feature augmented point to the remaining layers of T we get the output (7. ft'+1) which corresponds to the occupancy value at point q. Thus the self-attention and cross-attention modules process the features the point cloud and the 3d field respectively, without changing the topology and by performing local attention.
3.3 EQUIVARIANT ATTENTION FOR SHAPE RECONSTRUCTION
Shape reconstruction should be independent the coordinate system used, including of the posi- tion of the origin and the orientation of the coordinate axes. One way to capture this geometric consistency is via (qqiv--evvra,,,,,,, which can formulated in the language group theory. In setup, neither E nor T satisfy this property without additional constraints on (Wq. WK. Wv) and ( (W' Q. W W(). In this section we formulate and solve these constraints. We will assume familiarity with equivariance, and defer more extensive discussion to the Appendix.
5

---

Published as a conference paper at ICLR 2023
Equivariance constraints: In our formulation, the equivariance constraint is that the occupancy field should be a rype-O (or scalar) field i.c., a simultaneous roto-translation of the point cloud and the query should result in invariant prediction. Formally, for all (T, R) E SE(3), q R$.
TIE(RX + (NT)] T) (()
(3)
This constraint has to be satisfied from imput to output, not at every layer. For intermediate layers we can relax the constraint and output more expressive vector or tensor fields, with pre-specified transformation properties, so that the whole composition of layers results in scalar field.
Per-layer equivariance constraints. We can think of feature- augmented point cloud (X.F) as a feature field in IR with finite support. Then, both &, T process fields in R& at each layer. We need to specify how these fields transform under a roto-translation of X. Different transformation laws in the layers correspond to different constraints on their weights. We design the layers transform as,
(RX + 0xT, de (1) = ps (R) TIE(RX + Dx + TT, - (
(4)
(5)
for all (T, R) E SF(3), where Pe (R), p are SO represtiprssntnaai i.e., square invertible ma- trices satisfying p{R, R2) = (()) (R))) for all R₁, R₂ E &SO(3). The layers E0, To have also been designed to admit this transformation law since:
(RX + @NT) = (x)
T" + @N ( + RT (T((((
ie., (B( = (R) R (proof Appendix AA.1O). These per-layer constraints in Eqs. are
sufficient to solve Eq. provided that p. +4'+1(12) = T. These transformation laws have the form of
the induced representation of SE(3) via SO(3) (Fulton & Harris 2013) (Appendix (A.844..
Per-layer constraints on the weights: Observe that the transformation laws above describe fields whose features stay invariant under a translation of their domains (i.e., the point cloud and IR$)) but when this domain rotates they are multiplied by square matrix p(R)) Such fields are called p- p-fields. It is clear from Eq. [ 2 that due to the use of relative positional encoding, each feature translation invariant; and thus each field is translation equivariant. We focus on rotations next, discussing constraints on E, and adapting them to T. We prove in the Appendix that to solve Eq, it suffices to satisfy for all R E SO(3) and Fe, J. # the following constraints on the weights:
(R) = -R)''(R 1+1 (R)W -z)) W (R I:)) (R) P (R(E -R))
(R)W (G, Z.).
This reduces the per-layer constraints in Eq. Ato constraints on the weights of each layer. By solving these constraints, we uncover the free parameters in each layer.
Feature types and irreducibles: To solve equation it is necessary to exploit the structure of the matrices Pee p'y, which is an object of study of representation theory (Appendix A.8 Specifically,
according to Peter- Peter-Weyl theorem we can block diagonalize a representation p: SO(3) - d as
p(R) Q² (R)]
where Q is a change of basis matrix and D2 SO(3) + (22++))((6+111 A E N is the k-th irre- of SO(3). Those D&(R) matrices produced by the decomposition are called k-th Wigner-D matrices. If a field decomposes to single irreducible Dr, i.e., p = D& for some k E N. it is called a type-k field. We already discussed a type-0 (or scalar) field p( R)) Do (t) = 7, the occupancy field, as well as a type-1 (or vector) field p(R) = Di (R) = R on the input point cloud (X, ((X). In practice our feature fields on each point are composed of multiple copies (or maltiplicities) of irreducibles of different types that form complex -- fields i.e., f Ok S where indexes the type and m, its multiplicity.
ducible representation i.e.. non -ecmmmoooooabble into smaller one one-of
Layer Parametrization Given n' = DREK + m, Sk.mp and fl+1 = OMEKY ! mg
1+1 mm we
denote by w block of the matrix W that maps a k-type to a k' t'ytype (there are many such blocks depending the multiplicities). For clarity, we drop the layer index from the matrices, since all

---

Published as a conference paper at ICLR 2023
Otet
NeuraPPul
IF-Net
ComCOnet
VNN
GrachOnet
Durs
Input
OccNet
VNN
ConvOnet
S90aa 1091//00 1,800
GraphOnet
TF-Onet (Ours)
(a) Single Object Reconstruction
(b) Synthetic Scene Reconstruction
Figure 4: (a) Qualitative results for single object surface reconstruction for models trained and evaluated in three modes: training and testing on aligned shapes (T/T), training on aligned shapes and testing on rotated ones (3)SO(3)), training and testing on rotated shapes (SO(3) (3))SS))))) (b) Scene reconstructions from the Seismic dataset, using models trained on aligned single objects.
constraints are similar. By requiring that the attention kernel a scalar field of the positions, we find in Appendix AA the sufficient conditions
w k (x) DR (R)
W - Dz)) = DE (R)W F,)
W (R( = D2 ( -.). H.).
The solution of the first equation follows from Schur's Lemma (Appendix (..8.I). The next two have been studied in /Weiler et al. (2018)); Thomas et al. (2018). Finally, the inner product in the attention kernel
out some of the parameters in the keys:
W w I2 W 0, k + k
W - T.) = Σ 2 2- k-- O (1IE - -:11:0v 2C11 (1
+
WK' =;)== = J.K (1|æF- -x,1:0x CCj 12,-2.1 W OK
k
where (z)(()) = =IE- (6) Q Jm a =/|==|| and Q4t E (22 are slices from the Clebsch-Gordae matrices Q y S² 1R2++1 is the j-th real-valued spherical harmonic and ym (+)())) its m-th coordinate. The remaining free parameters are the, Φ which we parametrize with small MLPs. (See Appendix A..7 for an example of these solutions)
The constraints for the cross- antsenattenionn module T are similar Eq. 7 However, we can further restrict to type-1 features for the keys in the first layer, i.e., [Wx] O, for + 1, without any loss. The reason is that the input query field is of type- In Fig. 3 we visualize the equivariance constraint on T. & by using commutative diagram. For additional expressivity, we perform equivariant multi- headed attention by splitting the channels, i.c., the multiplicities of the irreducible representations. We also use skip connections and equivariant layer normalization as in Thomas al. (2018). but defer the details to the Appendix. See Fig. 2 for an overview the method.
Irreducible types for shape reconstruction: In deep networks, the difference between stacked channels and those forming a p-field is that the latter are equipped with transformation laws to mix channels, providing geometric meaning to the representations. We select the irreducibles to describe our intuition about the geometric entities we look for. If we want to learn a feature that behaves like 3D Euclidean vector under rotation (for example normal vector) we need construct a type-I feature. If we want a feature that behaves like symmetric matrix under rotations (such as the inertial matrix), we need 6-dimensional channel that comprises of one type-0 (scalar) channel and five channels that compose type-2 (traceless symmetric matrix) feature. We incorporate such geometric entities in a distinct way from previous $O(3)-based methods in surface reconstruction, which only handle vector fields (Deng et al. 2021 Zhu et al. 202 Chen et al. 2022). 2) In particular, type-2 features can be useful in our problem to define the bending of the local surface in the receptive field of the query. Thus they useful to infer whether the query is inside the shape. Our intuitions are supported by the experimental performance gain compared to Deng et al. (202T): [Chnn et al. (022) and the ablation study the types of representations used in network, shown in Table
7

---

Published as a conference paper at ICLR 2023
Some Reconstrucioo (Seismic)
Noise Ablation 50(3)/50(3)
Sparsity Ablation (0(3)/SO(3)
100
10
10
F-Srore (7-2%)
8
F-Suree
TFDret
-0-
70
Ey Coweee:
60
Tro
One
VNN Conriroe GraphChre. TFChnt
0.005
0.807
Inl 0.000
09
200
400
prints
Conchct
(a)
(b)
Figure 5: (a) Quantitative results of reconstruction on the Seismic dataset (synthetic scenes) for
models trained on single aligned objects. (b) Ablation study on the effect of point cloud density and
noise. All models are trained on 300 points with added normal noise of standard deviation 0.005 (black dashed line in the figure) and are evaluated on different sparsity and noise settings.
Table 1: Chamfer-L1 distance, F-Score and IoU achieved by different methods on single object reconstruction from sparse point clouds (300 points) sampled from ShapeNet. We evaluate our method (TF-Onet) on three different versions: (E:0-1.D:0-1) the encoder and the decoder use up to type-I representations, (E:0-2.D:0-1) the encoder uses up to type-2 and the decoder uses up to type- representations, (E:0-2.D:0-2) the encoder and the decoder use up to type- representations
CHAMPER-..T
13IT
F-Smaa (
PMTT
1/1
17S05T
5001)900)
172
1:0031
8013/S003
1/903
5013/8013
1.7
1/003
SC33/S0RJ
ONET
0.1
56.4%
11.4%
39.3%
59.7%
40.4%
03.0%
27.8%
30.9%
58.2%
CONVÖNET
6,083
0.16
0.12
71.7%
43.9%
58.9%
92.4%
73.8%
35.0%
37.28
55.8%
71.2%
VXX
0.14
0.14
0.15
56.5%
56.5%
55.88
61.2%
K1.2%
79.6%
69.35
69.3%
65.3%
TFNST
0,835
0.17
015
888
43.34
359
01.6%
13.3%
34.3%
37.3%
07.9%
0.18
0.17
30.45
50.3%
30355
13.05
65.75
GRAMIUNET
0.105
0.105
0.004
0.15
67.8
0722
85.75
58.7%
35.75
35.25
73.25
75.2%
0.105
6519
8599
33.0%
73.8%
750%
TRONER (00-1
0383
00.2%
00.2%
00995
10.9%
37.4%
73.2%
77.3%
T9ONER 2.000)))))))
0.083
0.093
0.993
71.2%
71.2%
71.1%
91.3%
91.3%
91.3%5
78%
75%
78%
4 EXPERIMENTS
We perform experiments with surface reconstruction from unoriented sparse and noisy input point clouds. We show the importance of (euivaaqui by evaluating objects in various poses and positions in space. Additionally, we show how local shape modeling and equivariance allows our method to train only on single aligned objects and generalize to scenes containing multiple objects in arbitrary locations and orientations.
4.1 SINGLE OBJECT RECONSTRUCTION FROM SPARSE POINT CLOUD
In our first experiment, we train and evaluate our model on sparse point clouds sampled from single objects in ShapeNet (Chang et al. 2015). For each input shape, we first uniformly sample 300 points from the ground truth mesh, and then we add normal noise with zero mean and standard deviation of 0.005. We note that this experimental setting of 300 points is more challenging compared to previous works that used denser point clouds containing 3000 points.
First we train and evaluate the original objects from ShapeNet, where both the training and test data points are in their canonical position (the I/1 case). Additionally, we follow Deng et al. (2021) and evaluate on test data points transformed by random SO(3) rotations. We evaluate models that were trained either on aligned training data (the I I/)O(3) case). or on training data augmented by S3(3) rotations (the SO(3)/SO(S case). In the case of our method (TF-Onet), as shown in T we experiment with different choices for the type of the intermediate representations used by our encoder and decoder. By adjusting the number of channels we make sure that all of our models have the same number of learnable parameters. We compare with the Occupancy Network (Onet) (Mescheder et al.
2019 non-equivariant network that extracts global feature; with the Convolutional Occupancy Network (ConvOnet) (Peng et al. 2020) and the Implicit Feature Network (IF-Net) (Chibane al. 2020 two translation equivariant networks that extract local features; with Neural Pull (Baorui et al.
2021 method that focuses on reconstruction from dense point clouds by performing only test time optimization; with Vector Neurons (VNN) (Deng et al., 202T). SO(3) equivariant network that extracts global feature; and with GraphOnet (Chen et al. 2022). a S3( equivariant network that subsamples the input point cloud to extract per point features with long range dependencies. We note that in contrast to our method both VNN and GraphOnet can only use up to type-1 equivariant features.
In Table I present the F-Score (Tatarchenko et al., 2019), the Chamfer-LI distance (Fan et al. 2017) and the Intersection over Union (IoU) of the reconstruction, for models trained on the aligned and 50(3) (augmmaeee datasets. We refer to section A. of the Appendix for more detailed de-
8

---

Published as a conference paper at ICLR 2023
Inouts
Erens
Qutnuts
Quina
(a)
Figure 6: Examples of scene reconstructions using our method, trained only on aligned single objects from ShapeNet. (a) Reconstruction of synthetic scenes from the seismic dataset, (b) Reconstructions of realistic scenes from Matterport3D. (Chang et al. (017)
scription of the evaluation metrics. Our method achieves consistent performance regardless of the rotation of the training or testing data points. Additionally, the best performance is achieved when up to type-2 representations in both the encoder and the decoder (E:0-2.D:0-2). When the testing data are transformed by random S3(3) rotations, our method consistently outperforms the compared methods, regardless of whether it is trained rotated aligned training data. In figure 4a we show qualitative comparisons between the methods. Our model achieves consistently high quality reconstruction all settings LL/L,1//O00 cloud density and the added noise affects the reconstruction, we train in the original setting and eval- uate on point clouds with different densities and levels of noise. shown in figure 5b our method outperforms previous methods on sparser point clouds, and scales better to denser point clouds.
(3) Finally, to study how the point
4.2 SCENE RECONSTRUCTION WITH SINGLE OBJECT TRAINING
In this section, we evaluate the ability of our model to reconstruct novel scenes with many different objects, while trained only single objects. Due to -E())--qui,,,,,nc,, performance is consistent and independent of the pose and the position of the objects. Additionally, our method performs computations in local neighborhoods that usually contain points from single object. These two factors allow us to reconstruct scenes that contain objects in arbitrary poses and positions, without the need to train similar scenes, or segment into separate objects and reconstruct each We construct a dataset of synthetic rooms with multiple objects from ShapeNet in arbitrary locations and poses, similarly to the dataset constructed in Peng et al. (2020), but with the addition of random SO(3) rotations on each object. We call this dataset the Seismic dataset.
In figure Sa we show quantitative comparison between our method (TF-Onet [E:0-2.D:0-2]) and previous methods that lack either the local shape modeling or the equivariant property. Figure 4b shows a qualitative comparison between the reconstructions achieved by these methods, while Fig- shows more examples of the reconstruction achieved by our model for scenes from the seismic dataset. Methods that perform global shape modeling (Onet, VNN) and are trained on single objects fail to generalize to scenes containing multiple objects. On the other hand, methods that perform lo- cal shape modeling but are not equivariant to SE(3) transforms (ConvOnet), can reconstruct objects in novel poses, but with reduced quality. Finally, the SE(3) equivariant GraphOnet, that per- point features with long range dependencies, cannot generalize well on novel scenes when it is only trained single objects. Our method benefits from both equivariance and local shape modeling, and is able to generalize to novel scenes achieving quality similar to that on single object reconstruction. In Figure [bb] we also show examples of reconstructions of realistic scenes captured from the Mat- terport3D dataset (Chang et al.] 2017). These scenes contain between 6000 to 13000 points. While our method has only been trained with aligned single objects from ShapeNet represented as point clouds of 300 points, it successfully reconstructs complicated realistic scenes containing multiple objects in arbitrary positions and poses, even from unseen classes.
5 CONCLUSION
coordinate- based model for shape reconstruction, consisting of two attention-based networks. By incorporating equivariance and local shape modeling, our method leverages the compositional structure of objects and scenes. These two properties allow our model to train single-aligned objects and reconstruct novel objects and scenes. We evaluate our method against state-of-the-an augmentations and compare favorably in the single shape reconstruction category. Additionally, using our model trained on single aligned objects, we show that it can reconstruct novel scenes with quality similar to single object reconstruction, task where other methods fail.
We proposed novel
and non-equivann -oueeuuivaraant methods trained with

---

Published as a conference paper at ICLR 2023
6 ACKNOWLEDGEMENTS
We gratefully acknowledge financial support by the following grants: NSF FRR 2220868, NSF IIS- RI 2212433, NSF TRIPODS 1934960, NSF CPS 2038873, ARL DCIST CRA 191 and ONR N00014-22-1-2677.
ARO MURI 1911NF-00 ONR
REFERENCES
M. Alexa, J. Behr, Cohen-Or, Fleishman, Levin, and C.T. Silva. Computing and rendering point set surfaces. IEEE Transactions Visualization and Computer Graphics, 9(1):3 2003. doi: 20.1 1175093.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.064400, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Con- ference Track Proceedings, 2015. URL http: .rr.
org abs 1409 0473
Ma Baorui, Han Zhizhong, Liu Yu-Shen, and Zwicker Matthias. Neural-pull: Learning signed distance functions from point clouds by learning pull space onto surfaces. In International Conference on Machine Learning (ICML), 2021.
Matthew Berger, Andrea Tagliasacchi, Lee M Seversky, Pierre Alliez, Gael Guennebaud, Joshua A Levine, Andrei Sharf, and Claudio T Silva. A survey of surface reconstruction from point clouds. In Computer Graphics Forum, volume 36, pp. 301-329. Wiley Online Library, 2017.
Alexandre Boulch and Renaud Marlet. Poco: Point convolution for surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6302-631- June 2022.
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Geomet- ric and physical quantities improve c(3) equivariant message passing. In International Confer- Learning Representations, 2022. URL https: htttp:://opereeviieee net wttprr/wwwwweeeeeeeuuuummmaae forum?id=
xwr XwrBgOBeVI
Rohan Chabra, Jan E. Lenssen, Eddy Ilg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In ECCV 2020: 16th European Conference, pp. 608 608-625, 2020.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017.
Angel X. Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rien 3D Model Repository. Technical Report 1rXiv:1512.0300 [cs.GR], Stanford University - Princeton University - Toyota Technological Institute at Chicago, 2015.
Haiwei Chen, Shichen Liu, Weikai Chen, Hao Li, and Randall Hill. Equivariant point network for 3d point cloud analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14514-14523, 2021.
Yunlu Chen, Basura Fernando, Hakan Bilen, Mathias NieBner, and Efstratios Gavves. equivari- ant graph implicit functions, 2022. URL /
17178
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019a.
10

---

Published as a conference paper at ICLR 2023
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5939-59999999 5948, 2019b.
Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2020.
Christopher Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: uni- fied approach for single and multi-view 3d object reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learming, pp. 2990-2999, 2016.
Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolu- tional networks and the icosahedral CNN. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 1321-1330, 2019.
Pim de Haan, Maurice Weiler, Taco Cohen, and Max Welling. Gauge equivariant mesh cnns: Anisotropic convolutions on geometric graphs. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL httts::/cppenevr/ ew net/ mott:forrummddd-sppppee 111
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi. and Leonidas J. Guibas. Vector neurons: A general framework for so(3)--quivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12200-12209 October 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openreview. htt.:///ooommmmm-..mm....
Philipp Erler, Paul Guerrero, Stefan Ohrhallinger, Niloy J. Mitra, and Michael Wimmer. Points2Surf: Learning implicit surfaces from point clouds. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 108-124, Cham, 2020. Springer International Publishing.
Carlos Esteves, Christine Allen-Blanchen Ameesh Makadia, and Kostas Danilidis. Learning so(3) equivariant representations with spherical cnns. In The European Conference Computer Vision (ECCV), September 2018.
Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point set generation network for 3d object reconstruction from single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Fabian B. Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling, Se(3)-transformers: 3d roto- translation equivariant attention networks. In Advances in Neural Information Processing Systems 34 (NeurIPS), 2020.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193-22)) 202. 1980.
William Fulton and Joe Harris. Representation theory: a firrstt course, volume 129, Springer Science & Business Media, 2013.
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Samaa, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In The IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 4856-4866 06 2020. doi: 10. 10.11999CVPP2266000000001011

---

Published as a conference paper at ICLR 2023
Shunwang Gong, Lei Chen, Michael Bronstein, and Stefanos Zafeiriou. Spiralnet++: A fast and highly efficient mesh convolution operator. In Proceedings of the IEEE/CVF International Con- ference Computer Vision Workshops, pp. 0-0, 2019.
Brian C Hall. Lie groups, Lie algebras. and representations: an elementary introduction, volume 10. Springer, 2003.
Chiyu Max Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3d scenes. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. Arans Graph., 32(3), jul 2013. ISSN 0730-0301. doi: 10.1 00.1145/24872282487733 10. 0011155447722
htt.h:httt://.tt org
2487
David G Kendall. A survey of the statistical theory of shape. Statistical Science, 4(2):87-99, 1989.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICIR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Yann LeCun, Berhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hub- bard, and Lawrence Jackel. Backpropagation applied handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Stefan Lionar, Daniil Emtsev, Dusan Svilarkovic, and Songyou Peng. Dynamic plane convolutional occupancy networks. In Winter Conference on Applications of Computer Vision (WACV), 2021.
William Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 1634)163,,6,, 1987.
Haggai Maron, Heli Ben-Hamu, Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In 7th International Conference on Learning Representations, ICLR 2019, New Or- leans, LA, USA, May 6-9, 2019, 2019.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11453111444 2021.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3504-3515, 2020.
Xuran Pan, Zhuofan Xia, Shiji Song, Li Erran Li, and Gao Huang. 3d object detection with point- former. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition
(CVPR), pp. 7463-7472, June 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In European Conference Computer Vision, pp. 523 523-540. Springer, 2020.
Adrien Poulenard and Leonidas J. Guibas. A functional approach to rotation equivariant non- linearities for tensor field networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ( (CVPR), pp. 13174-13183 June 2021.
12

---

Published as a conference paper at ICLR 2023
Charles R. Qi, Hao Su, Kaichun Mo. and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation, 2016. URL http:// http://axivv.orro www.xiiw..gggaaooooo wttp:rggaabba..... 00593 cite arxiv:1612.00592
Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representa- tions at high resolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3577-3586, 2017.
David W. Romero and Jean-Baptiste Cordonnier. Group equivariant stand-alone self-attention for vision. International Conference on Learning Representations, 2021. URL https:
//openreview wwtp://ttttwo/o/ooooooo (ttp://oorrrmmmdddk..rrooooeee
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-ane cessing Systems, 32, 2019.
neural scene representations. Advances in Neural Information Pro-
Jiapeng Tang, Jiabao Lei, Dan Xu, Feiying Ma, Kui Jia, and Lei Zhang. Sa-convonet: Sign-agnostic optimization convolutional occupancy networks. In Proceedings of the IEEE/CVE Interna- fional Conference on Computer Vision, 2021.
Maxim Tatarchenko, Stephan R. Richter, Rene Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. What do single-view 3d reconstruction networks learn? In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and -enuuiation-uuuaariaatt neural networks for 3d point clouds. arXiv preprint arXiv://82.0082/9, 2018.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Carsten Stoll, and Chris- tian Theobalt. PatchNets: Patch-Based Generalizable Deep Implicit 3D Shape Representations. European Conference on Computer Vision (ECCV), 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017.
He Wang and Juyong Zhang. A survey of deep learning-basse mesh processing. Communications in Mathematics and Statistics, pp. 1-32, 2022.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In NeurlPS, 2018a.
Maurice Weiler, Fred A. Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 849-858. IEEE Computer Society, 2018b.
Maurice Weiler, Patrick Forré, Erik Verlinde, and Max Welling. Coordinate independent convolu- tional networks isometry and gauge equivariant convolutions riemannian manifolds. CoRR, abs/2106.06020, 2021. URL UtL.htppp:://rrxtv.rrr///bs//// 0000//00b002220000000000 0602C
Francis Williams, Zan Gojeic, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, and Or Litany. Neural fields as learnable kernels for 3d reconstruction. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Paltern Recognition (CVPR), pp. 18500-18510, June 2022.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22-31, October 2021.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointcomv: Deep convolutional networks on 3d point clouds. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
13

---

Published as a conference paper at ICLR 2023
Xumin Yu, Yongming Rao, Ziyi Wang. Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry- awwae transformers. In ICCV, 2021.
Minghan Zhu, Maani Ghaffari, and Huei Peng. Correspondence-free point cloud registration with 50(3)-equivarian implicit shape representations. In Sth Annual Conference Robot Learning, 2021. URL https: // opwprevee.......eeomeee/e net http:/ooormmmmd-k00o0od Ta
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 EVALUATION METRICS
In table [ we present the F-Score, Chamfer-LI distance and Intersection over Union (IoU) of the reconstruction achieved by various models. For the F-Score and Chamfer-LI distance, we compute the reconstructed mesh using the Marching Cubes algorithm {Lorensen & Cline 1987). Then, we uniformly sample 100,000 points from the reconstructed mesh, and compare them with the "ground truth" points sampled similarly from the "ground truth" mesh. The standard deviation of the results for both metrics due to the randomness introduced by the sampling of the 100,000 points is smaller than 10- As defined in Tatarchenko et al. (2019), the F-Score here is the harmonic mean of the precision and the recall of the reconstruction. Precision is the percentage of reconstructed points that lie within a certain distance T of the ground truth points, and recall is the percentage of ground truth points that lic within distance of the reconstructed points. We compute the F-Score with r equal to 1% of the side length of the reconstructed volume, and with T equal to 2% of the side length of the reconstructed volume. For the IoU metric, uniformly sample 100,000 points from the whole space, and compare the predicted occupancy to the ground truth occupancy. The IoU is then computed by taking the ratio of points that are occupied according to both the predicted and the ground truth occupancy to the points that are occupied according to either the predicted or the ground truth oceupancy.
A.2 MODEL. ARCHITECTURE, TRAINING AND TESTING DETAILS
For the self-attention feature extractor network &, and for the cross-attention occupancy network T we use another two. All multi-headed SE(3) atten- tion layers use eight heads. For each point हे of the point cloud, we define the neighborhood. (Z,) as its k nearest neighbors, where k is chosen to be 5% of the size of the point cloud (c.g.. for 300 points, 15)) Each feature map in the layers of & T uses irreducibles up to type-2.
ten multi-headed SE(3) attention layers,
We train our model on the ShapeNet (Chang et al. 2015) subset constructed in Choy et al. 2016 We use the Adam (Kingma & Ba 2015) optimizer with learning rate that starts at 2 10 and linearly decreases to reach the value of TO We train for 200,000 iterations using batch size of 64. During training, we take as input point cloud of 300 points and as ground truth the occupancy value of 2048 points that are sampled uniformly inside box that bounds the object. As training loss, we use the binary cross-entropy loss between the predicted and the ground truth occupancy of the 2048 randomly sampled points.
During inference, recalling that the output of our model for one query point is in the range of [o, 1]. classify query point occupied if the value of the learned occupancy function is above 0.2. After we query the occupancy of points throughout the space, we reconstruct the object's mesh using the Marching Cubes algorithm (Lorensen & Cline 1987).
A.3 EXTRACTING DIFFERENT TYPES OF REPRESENTATIONS FEATURES
An important quality of our method is its ability to use different types of representations as its intermediate features. These different types determine the way that the features transform as the input rotates. These transformation laws affect the shape properties that each type of features can encode. For example type-0 features can encode properties of the shape that are invariant to the rotation of the shape, type-1 features encode properties that must rotate as 3d vectors (e.g. the normal vectors at the surface) and type-2 features can encode properties that rotate as symmetric matrices (e.g. the inertial matrix of the shape).
A type feature consists of a 27 + 1 dimensional vector that rotates according to the corresponding Wigner-D D matrix D; 5O(3) + (22++1)x
In figure we visualize the different types of features extracted by encoder for different cases of inputs. The difference between the type of features can be clearly observed in the case of the symmetric sphere where features of higher type / correspond to higher frequencies on the sphere.
15

---

Published as a conference paper at ICLR 2023
type m
type
type m=
m --
m=
m=0 =1
Figure 7: Visualization of each dimension of type-l feature extracted by our TF-Onet encoder (with = 0, 2). type-l feature corresponds to a - 1 dimensional vector where each dimension is denoted with index (-l,....), l). by our method for the corresponding input.
The last right column shows the final reconstruction achieved
A.4 RECONSTRUCTIONS OF SCANNED SCENES
We further investigate the generalization of our method to novel scenes by reconstructing real objects of different domains and levels of clutter (Fig. [8.. The input is a sparse, unoriented point cloud scanned from scenes consisting of arbitrary number of randomly placed objects. The model is agnostic to the number of objects in the scene and is trained only on synthetic single objects. The model outputs high-quality reconstructions even in difficult settings with objects very close each other. We observe small artifacts in very cluttered scenes, localized in regions of incidence between objects (e.g. mugs in line of Fig.
Figure 8: (Left): Reconstructions of real scans of randomly placed objects produced by our model which is trained only on single synthetic objects. (Middle): Input scene-level point cloud. (Right); RGB image of the scene.
A.5 ROBUSTNESS TO CLUTTER
We evaluate the model the Seismic dataset (Sec. 4.2 which contains scenes with varying levels of clutter. Fig. we measure the Intersection over Union (IoU) of the reconstruction versus the minimum distance between any objects in the scene (normalized with respect to the room size).
16

---

Published as a conference paper at ICLR 2023
The minimum distances range from around 6% 20% of the room. The performance of the model relatively stable across all settings with a maximum performance decrease of 0.025 IoU.
loU versus Minimum Distance
0.02
0.80
0.78
0.76
0.74
0.72
0.10
0.16
min distance room size
Figure 9: Intersection over Union (IoU) of the reconstructions from the Seismic dataset (Sec. 4.2 with respect to the minimum distance of any two objects in the scene.
A.6 LIMITATIONS AND FUTURE WORK
possible limitation of this work is the inherent memory overhead of the attention modules used in the network. In our setting, this overhead is lowered due to the locality of the attention operation but it can further minimized with additional optimization of the attention modules. The optimiza- tion of the attention modules is active research direction that can also benefit our method. An additional possible direction for future work is to extend our method to the problem scene com- pletion from partial observations. In this task, the model is required to hallucinate and reconstruct unobserved areas of a scene and thus requires a combination the scene configuration. While significant methodological extension is required for method to tackle such a task we believe that the core principles of equivariance via higher-order representations and local shape modeling can provide useful tools.
local and global information about
A.7 WEIGHT PARAMETRIZATION
Here, present the parametrization of the weights of the queries, keys, and values that appear as the solutions of Equation 7 (and written analytically Equation 8). We focus on the specific case where both the input and output feature maps p-fields consisting of up to type-2 irreducibles. Then, each feature the feature maps can be decomposed into
M
N
fm
Jkmm fout
mn
1==
=0= may
For clarity, start with multiplicities 1 i.e., Mr N2. 1 (and omit the index of the layer). Then we extend to the general case.
The matrix We appearing in the query tokens Q fia) WiJin has the form:
17

---

Published as a conference paper at ICLR 2023
Fout
0,00
in
0,1
fout
w 1.1
I3
fou
0
2 ls
2
We
Similarly, we present the form of the function of matrices WK. that appear in the key tokens as K (ū fin) (T, F:)fm To simplify the notation we use o instead of ž H, and | To write WK in matrix form we first need to define:
( e (Z) =
(1
* (+)
learned
fixed
The functions 0f matrices defined in Section g
R
R are parametrized by MLPs. Cf (2) E 1(2++1)((2X-))
are fixed
Given the definition above we can write WK as:
L0 20-o(110 8 (f)
(1=((1æ)) @ (+)
0.2 (2-2(1
C222
1::0 (||
(+)
+ (= & CL
C12.(()
fi
2.0 2- (1 (2)
(1 & c2
S (1
CZ
Wk(#)
The form of the matrices Wv appearing in the value tokens V(ž fi) Wv i, is similar to Wx k above.
In the general case, if the input representation contains Mr copies of type-k irreducibles need to stack the corresponding blocks of Wo. WK. M₁ times in the column dimension. Similarly, the output representation contains N₁ copies of type-k' irreducibles need to stack the corresponding blocks Ne times in the row dimension. Specifically,
for Wi and k k':
k.k ((.1) T26+1
l.k ((LMa 12k+1
k.k 128411
w kik
(N 122+1 11
w
(Nar 12k+1
lok
for WK and [ 1K6 - -kl. u= |K k/:
( (a 2 Cf'L*
K ( @ CK.
$ kk.k I-u e aak .
[ ( 1) a
( @ Cfi
A.8 PRELIMINARIES
We recall some basic notions from group and representation theory, see e.g., Fulton & Harris 2013). group (G. is set together with a binary operator "."; "."; axioms:
x + G that satisfies the following
18

---

Published as a conference paper at ICLR 2023
Associativity: g. (h- (g- h) f for all g, h, E G
Idnniity there exists an element e E G such that e 9 -
Inverse: for all G, there exists g E G such that g g = =e.
Given G, we say that each group element g G acts the space X via action Ly X X
Lg satisfy the following two properties:
If is the identity element of then Le [x] x: for all G;
Lig Ln Lygh h for all g, h E G.
If for any 2, E X there exists g E G such that Lg [x] = then we call X a homogeneous space for the group G.
When X - V is a vector space, we can define the group action using linear group representation (V. where p G + GV(V) is a map from group to the general linear group GV((V. This means that using the linear operator p(g) on V. we can define the group action Ly on the vector linear group representation if p
space V as Lg [r] p(g)xz for all æ E V.g G. Then (V. is group homomorphism, i.e., p(g h) = p(g)p(h) for all g. G, and p(e) = Ty is the identity operator over V. To simplify the notation, when the vector space V that the group acts on is easily inferred from the context, we will use p denote the representation (V,
Given a set of actions Lg XX - X for g E G, and a set actions Tg Y - Y for g E G, we say that map f X + Y is (G, L, eqqivaaaaa if for every y E G:
[>][(=)] = [(T,, (x)) for all g E G, E X.
If f is linear and equivariant (with respect to (G, I, T)), then it is called an intertwiner (with respect (G, T)).
If there exists a subspace W C V such that for all g E G and w E W, we have that p(g) E W, then is a G-invariant subspace of V, and pW,p) is a subrepresentation tions (V, p) have at least two subrepresentations: (0, and pV, If representation has no other subrepresentations, then it is called irreducíble. Otherwise called reducible. A fundamental result is the following (Fulton & Harris 2013).
(V. All representa-
A.8.1 SCHUR'S 'S LEMMA
Let (V. Pv). PW, ) be irreducible representations of G acting on V and W, respectively.
If V and W are not isomorphic, then there are no nontrivial intertwiners between them.
If V W are finiteddimensional vector spaces over C, and pv pw. then all intertwin- ers are scalar multiples of the identity.
Now we list number of fundamental examples, see e.g., Fulton Harris (2013) Hall 2003).
A.8.2 THE TRANSLATION GROUP +)):
R equipped with the addition operator "+" is group that is isomorphic the group of translations
in the 3D space.
A.8.3 THE SPECIAL ORTHOGONAL GROUP SO(3):
SO(3) the group of orthogonal matrices with determinant +-1 equipped with multiplication: and is isomorphic to the group of all 3D rotations about the origin. SO(3) is a compact group and as a result of the Peter-Wee Weyl theorem, its linear representations can be decomposed into a direct sum of unite-dimensional unitary, irreducible representations. Specifically linear representation of SO(3) decomposes as:
(()) = QT
+ (g) J20
Q for all g G,
19

---

Published as a conference paper at ICLR 2023
where Q is change of basis matrix and J 0, o... D, are +)) (2. 1) matrices known as the Wigner D-matrices. The Wigner D-matrices are the irreducible representations of (3)3). In the context of the features of neural net, the representations (viewed as vectors) that transform according to Dy are called type-J vectors.
A.8.4 THE SPECIAL EUCLIDEAN GROUP SE(3):
SE(3) the group of proper rigid transformations of 3D space, and is isomorphic to the semidirect product (IR3, +) x (3).). An element of SE(3) can be represented as (I" where T is an element of the group of translations and is an element of the group S0(3) of 3D rotations. For two elements (Ti, R2), (T2, R2) SE(3), the group law is defined as:
RT) (T2, R2) = (Ti + R,
Since every point in R3 can be transformed into any other point with a proper rigid transformation, we have that Rd is homogeneous space for SE(3).
In addition to the action of SE(3) on vectors in R3, we can also define the action of SE(3) on func- tions R33 R-M. for any given integer M 0. This action is called the induced representation π Ind So(3) P of SE(3). It acts on f as follows:
(T( = P(R) -T)).
where (T SE(3), E R and p representation of SO(3). Especially in the context of neural nets where functions are viewed as feature fields, a function that transforms according to π S0d3) is called p-field, and if p corresponds to the 1-th irreuucible representation of SO(3), it is also called a field of type-l.
A.9 ARCHITECTURE DETAILS
A feature-augmented point cloud P (X,F) where X == [....-N [N]] E RaxN and F [fi---fN fxl E R for some M E N can be associated with a 3d field f R + RM of fi- nite support by writing f(z) = Σ fvö(Z Z.). This association will be useful to define the group action on the feature-augmented point cloud and will unify the inputs and outputs of T, & in the sense that both of them process fields in R and output fields in R3 We will interchangeably use (X, F') and f for the point cloud in the next sections. We will call f the point cloud function when the distinction is not clear.
The main module in our architecture is an multi-head SE(3)-equivariin attention module followed by a skip connection and an equivariant layer normalization step.
attention block. Each block consists of
A.9.1 MULTI-HEAD SE(3)- EQQIVARIAAT ATTENTION MODULE:
This module consists of multiple heads that implement either self-attention (input features for keys, values and queries are the same) cross-attention (input features for keys and values are the same and different from the inputs for the queries). We will first describe the self-attention variant of the module and then describe the changes that are required for the cross-altention version.
The self-attention SF((3) as discussed above) defined by f(ø) = > f,ö(ī - J Each one of the fi vectors can be decom- posed into irreducible representations of SO(3) appearing with different multiplicities. This means that a single vector f: can be decomposed as f. = Dizo Dmzo to the my-th multiplicity of the irreducible component of type-l. Since perform self-attention, the keys, values and queries are computed using the same input function f. Specifically, for pairs of compute the keys K(Ø H, fy) WK(F, f.)f the queries Q fx) = Wofi, and the values V T., T, f,) = Wy Wa. Wv must satisfy the conditions described in Section 3.3
module takes as input a function / (describing the point cloud
where fi.m, corresponds
key and query points (.).
E.).f- To ensure equivariance, WK,
Suppose that for the computed key and query features, the I-th irreducible appears with multiplicity Mt, and for the computed value features the k-th irreducible appears with multiplicity Nç. For each
20

---

Published as a conference paper at ICLR 2023
irreducible, split its multiplicities across the different heads of the attention block. Assuming that we have H heads in total, each head h receives keys K Fi; containing irreducibles with multiplicities M and receives values (T) H;, fi) containing irreducibles appearing with multiplicities N H. After this split, the output of the self-attention for each head can be computed
and queries Q(h) (T, fi) f5) (,,
F5)
E ax ( (EE((E)
where
exp (Q)
ax (2)
ET,EN(E.) exp [Q K G)
Similar to the keys, values and queries, the output can have irreducibles and multiplicities that differ from the input and decompose as:
SA (b) X
SA (M) fX,
72
where SA (1) [X, Many is the ni multiplicity of the k-th irreducible.
After the application of the self- self-attention layer, concatenate the output of all the heads cre- SA[X, f] A SA(h) [X, f]. Then we pass the concatenated output through a linear SE((3- equivariant layer to take SA out [X, = WpSA[X, .]. Since this linear layer also needs to be equivariant, it must follow the same constraints as the query matrix Wo- By Schur's is lemma, it follows that Wp can only mix feature vectors that correspond to the same irreducibles.
For implementing cross-attention, we use the same process as with self-attention, with the only difference that the inputs for the queries are different from the inputs for the keys and the values. As result, the output of the cross-attention for single head h is computed as:
CA (K) [X, f(
Σ ax (othh (g. K ( (I,,
(2).
E
A.9.2 SKIP CONNECTION:
The skip connection concatenates the output features of the multi-headed (B0(33))qqvvaiant module with the features of the input query. To respect the type of each feature, this concatenation must happen between features corresponding to the same irreducible.
Suppose that foun the output of the multi-head attention, decomposing into irreducibles and their multiplicities rou = θ On, Jk. Similarly suppose that fy is the input query, decomposing into irreducibles and their multiplicities fa θ Am, fa.(l.m.) The application of the skip connection gives an output
JOUE [ n%
]
pakip +
A fqi(k.me) mi
e
To keep the output at the same types and multiplicities as specified by the user (i.e., to be a Pour field) we only concatenate multiplicities from the input that exist in the output. Also, after the skip connection we apply an cquivariant linear layer between irreducibles of the same type to project the features the correct dimensions.
A.9.3 EQUIVARIANT LAYER NORM:
Suppose that fi.m denotes the m-th type-l irreducible of the input vector f. As proposed in Fuchs al. (2020), for each vector fi.m: we apply layer normalization and a nonlinearity on the norm of fi m, leaving its direction unchanged. Thus, the equivariant normalization layer can be written as:
frm
EqLayerNorm(f)u. -ReLU
LayerNorm
+
||fG,ml]|
[
21

---

Published as a conference paper at ICLR 2023
where LayerNorm corresponds to layer normalization, proposed in Ba al. (2016).
We use the SE(3) -qqiivaria attention block to construct both the network & that assigns to each point in the point cloud a Iearned feature, and the network T. that outputs the occupancy value of a query point in space. The network consists of ten -e(3)-equivariant self-attention blocks, and T consists of two -Eqiivaquivarrant cross-attention blocks. Figure shows diagram of this architecture.
In both networks, features up to type-2. Additionally, for each type we set the multiplicity of the corresponding irre- ducible to 32. Finally, for the computation of the local neighborhood ((7) around each point Ø. use the k nearest neighbors, where k is chosen to be 5% of the size of the point cloud (c.g., for 300 points, k=15).
blocks with eight heads and intermediate representations that contain
Although it is possible for T to output a single scalar that corresponds to the occupancy value at queried point, we observe in the experiments an increase in performance when T outputs 32 scalar values that we then pass through a simple MLP to get the final occupancy value.
A.10 PROOFS ON EQUIVARIANCE FROM SEC 3.3
Input-output equivariance (Eq. 3) We will formalize the equivariance constraints in the language of group theory. We have map â that takes as input point cloud X E R3*N and outputs an occupancy field 6(X) : IR' + IR First we need to define the actions of SE(3) on the input point cloud and the occupancy map which we denote by Lin. Lout respectively. Those have the following form for (T, R) E SE(3):
(Tn,(T,R) [X] RX + GNT
(9)
(6
(10)
where @v vectors denotes concatenation column-wise. The first equation describes N standard representations of SE(3) and the second the induced representation of SE(3) via SO(3) with p(R) = I, i.e., the map ô is scalar (or type-0) field.
For completeness we show that Lin indeed describes an action of SE(3) on X. Letting 0x0) be the identity element of SE(3), can check that
fm. [X] = 1X + @x0 = X. Also, for any (Ti, Ri), (T2, R2) from SE(3), we can check that
R,jj( RR, R,) [X]] R2(Bi @xTii + BNT2=
R2RRx R2(@NTi) + @N72
R2 + N R + T2)
-in, + 22 R. [X]
(T).
Using the vector space isomorphism associating X E R3xN with the unrolled vector vec(X) E R3N, we can view this action as the direct sum of N standard representations. Moreover, Lout is also an action, and in fact an induced representation of SE(3) via SO(3) with p(R) = 1, SO(3), according to the definition A.8.4
Now that have the input and output actions, we can define the equivariance constraint for our problem. Informally, a simultaneous roto-translation of the point cloud and the query results in an invariant prediction. Formally,
o(Cm, [X]) = Lout,
(11)
[ō(Lm, [X)][(] (o (())
V +
(12)
Jō(RX + BN (xT)) (o(X) (T-T)), Vie R
(13)
Jō(RX- + o, Tx] + T) ( V
(14)
Now recall that we have parametrized 6 as composition of a feature extractor & and an occu- pancy network T. i.e. [ = (T) Thus we arrive to Eq. 3
T(E(RX + [
(((()))
(15)
22

---

Published as a conference paper at ICLR 2023
2. Per-layer equivariance (Eq. 4 5)
Next, need to prove that the per-layer equivariance
constraints Eq. 4 are sufficient to satisfy the input-output equivariance constraint from Eq. 3 provided that PT the oceupancy network T are:
L R = I, R E 5O(3). We recall that the forms of the feature extractor & and
E[X] = e..... E 00 [ [x]
(16)
E|[X] (X))(X)
(17)
T|X, ( = T T T'OT' o F
(18)
T [X, F( = ((,
(19)
All 111,,,8 ,,,....."""""".."........ _ take as input and produce as output a feature field on the point cloud and satisfy the constraints in Eq. 4 Similarly, all T . T take as input a feature field on the point cloud and feature field in R.3 and pass the point cloud feature field unaltered to the next layer. They transform the feature field in Is' and satisfy the constraints in Eq. 5 Thus, by composing these constraints in Eq. 45
we immediately find:
ch...o 0² o Ed [RX+ + @N ONT, (TTT,P((R)))) (RX + @N @NT, 1+ (8)
(20)
T .......7 .T T
(R T, + (
where p4' (R) = as discussed. The equations above show that if every layer transforms an input p-field to an output p-field, then the composition also transforms an input p-field to an output p-field. Thus, it remains to show that the fixed layers Eo go namely ((X) and (g. (" (X, g)) respectively are p-fields as well. Then, the equations in Eq. 20 would not only apply from the first layer to the output, but from the input to the output. In other words, we need to check that the features produced by S, S" do not translate when the point cloud translates but they do rotate (under suitable representations) when the point cloud rotates.
2.1. Input point cloud field and input query field are type-1 fields: For the i-th point at position E, we construct feature f; as input to the first self atf-nattenion layer of E. We select the feature fa as the relative position of the i-th point, H. to the centroid of a neighborhood constructed from its nearest neighbors in the point cloud in Euclidean norm, i.e.,
:== E.
Σ (EN((X)
X= + (
(
where
( - S 1
Ix 11
(1) x
the neighborhood of the i-th point in the point cloud. Also,
is sorting of the points
in the point cloud in increasing Euclidean distance the i-th point, i.e, || xf [l2 S < | - x lll. In case of ties, we assign numbers randomly. Due to the use of < " (instead of the strict inequality symbol " < "), in the definition of the neighborhoods, if there are ties for x()) we include all tied points in the neighborhood.
Now we study how the features fi transform when the points in the point cloud transform via Lin,(T,R) discussed in the first step. Since
Cin,(T,1o)X (Cm, (X]:) e [RT, + +T},
we find:
(x Cin,rr,n)()]:
Σ
W (Cin,(TTT)
jENt (fm,,),,9)1))
=RE, +T)
Σ
(Rē + T),
(X)
SEN((X)
23

---

Published as a conference paper at ICLR 2023
where we used the claim, proved below, that
N²t (Iin, = (X).
(21)
Thus, (( (x)x; further equals:
Σ R F4 (F(( JENT(X)
Ty = RS(X), for (T, R) SE(3).
We now prove 21 When all H. are mapped as J4 (T.R). +T the Euclidean distance between any two points preserved, i... [[ + T) (Rã, T || Ey112. Thus, if before the transformation || #
5) 2l2 dR. then after the transformation we also have [,,),,TT,)) [X]]=
[X] I& 2l = dx. This is because all nearest neighbors preserve their distances, and thus sorting returns the same indices up random tie breaking. Thus,
(r,
j E =
| 55l12 Σ dx II(RE: + T) (Rã, T2) dx
ii R) [1] - Lin,(T,R) R²) (15sl2 S ds
(T [X]; - (T.. (xlil22 S [|C.,,(TxxxX) (T,.(T.R)) (x) ll - N (X
The first and fourth equivalence hold because we include all tied neighbors in the neighborhood. Thus, the neighborhood is defined by the distance dg, and not by the identity the k-neighbors.
Thus fi = S(X); transforms according to the standard representation of SO(3) when each point in the point cloud transforms according to the standard representation of SE(3). If we view the features as function the point cloud extended to the homogeneous space R = SE(3)/SO(3),
i... f(x) = E' f,0(z -x:), then this function transforms according the induced representation as:
N T)) Σ (RE) (T- -2)).
cland) [f( Rf (E-
Recall that functions transforming according to the above law are called type-1 fields. We will keep the name, but use matrix notation instead of the Dirac notation. By concate- nating the features column-wise, we find the map, described in the main text as S, i.c., S(RX + @x7) = Rf RS(X).
Now we turn the second network, T. For each point q E R33 whose occupancy value we wish to find, we first construct feature := S'' g) and then use the pair (q. f as the query to the first cross-attention layer of T. We will show that, when the query q and the point cloud transform according to the standard representation of SF(3), this input feature f also transforms according to the standard representation of (O(3).
We again construct the feature ft as the relative position between q and the centroid of its neighbor- hood N (g). We consider 2 (7) to be the same as the neighborhood of its closest-in Euclidean distance pponnt in the point cloud, and write (if the closest point is defined uniquely)
N2
.ak min (11 (X) (X).
We discuss at the end of this step the case where the nearest neighbors are tied. At the moment, let the unique closest point be = arg min (2122). Then, the query feature becomes:
S/ S = ( JENx (
Σ z
=
Σ プz
X X=
N
(22)
(X) (X)
24

---

Published as a conference paper at ICLR 2023
The proof that
=
the same the one for S before. Viewed again as function defined on R23 the map q (X. constitutes type-1 field. The transformation law of this field is depicted in Fig. 3
2.2. Remark: When there are ties, i.e., the set arg min (l7 Tbll2) contains more than one point, we form all neighborhoods
N (g) (X) arg min (g-x,112): [E[N]
Then, we consider a query token for each pair (q", fq), where q" q and fÿ are constructed in using the neighborhood. Not (X) E X
22
Ja transforms as a type-1 field, then, after the roto-translation of the point cloud and the query. closest neighbor. However, since those points equivalent as
could identify the same nearest neighbors, we will process the whole set fields independently, producing a set of fields in the output. A different order of selection of nearest neighbors after the rolo-translation corresponds to permutation of the set of the output fields. Since attention modules are permutation equivariant, this permutation propagates to the output.
We only need to discuss how to combine these outputs on the tokens q" that correspond to the same position & in the occupancy field. Since the output is scalar field (as we prove next), we can take the maximum across the same channels to construct a new scalar field that is also invariant to any permutation. The idea is that taking the maximum corresponds both the usual non-linearities (such as the sigmoid) and the thresholding operations that follow are increasing functions of their inputs. Thus, when the network predicts that the position of the query is "occupied",, even based on one neighborhood, the position is likely to be considered occupied.
"OR" operation, since
From per-layer constraints 4 layer of &, T separately. Now we need to prove that Eq. 6 provide sufficient conditions on the weights to satisfy the per-layer constraints 4 5 Every layer is composed of a multi-headed attention layer, skip connection and normalization layer. We prove the result for the self-attention and cross-attention layers (focusing on one attention head for simplicity).
constraints on the weights 6 We focus now on each
will be clear from the proof that concatenation of the heads and subsequent linear transformation mixing channels that correspond to the same irreducible preserves equivariance, as well as a skip connection that concatenates irreducibles of the same type.
We start with the self- aelf-attention layers in E described in A.9.I We drop the layer index / for clarity and denote the actions ple, P by Pins Pout.
At the i-th token. we have
SA|X, =
Σ (E
f)) Wr(E,, IDff
V(8,
where the attention kernel ax takes the form:
exp
a Ex(
ΣjENA(X) exp (Q( [() exp
ΣjENs(X) exp (Wo)))) -Wx(1)-#:))) iE [N
Then, we are given
(R) Pont (R) W F,))pm ( R) = Pout W) - -F,) Wv(R(T, - -..) m (R) = Pout Wy) I.)
(23)
and we need prove:
SA(RX P Pout Pout (R)SA(X, F).
25

---

Published as a conference paper at ICLR 2023
For the attention layer, have for X e [ and for each output token i [N]:
SAIRX + Pm
(24)
Σ
TQX+E + (m.)) K(Rē, + + RT, + + T.
(EX
Wv((RE, + T) - (RT. - Pm)
= Σ
aRx+ aRX+ QPtn (i)) Wx (R(T, (1)) - pz (R)
(X)
where used 21 for the invariant neighborhoods, (RX @N7) (X). Now using the constraints for the matrices 23 we find for each term j E N, (X) that the individual terms in the sum above equal:
a aRX+SNT Pow
E,ff, Pout (R))xx*xxx)) Poout (R) Pout (R))x((3) [r) T,
(R) (ooner Pout -
(25)
where in the second equation we used that the attention kernel gives scalar output. Now, the attention kernel from the last equation transforms as
aRx+0xx T (R)WQJs, Pout (R) K (ā) T:)s
exp [(out (R) (Pont (R)W K -
E EJEN exp [Pout (R) ( (Pout (R)Wx (3) I:Jds]
exp ((W((( ( (Wx - i.ffs)
(x) exp (Wef.)" (Wx(T; -Z,,,, = ax(Q K(),
T f)),
(26)
where we used again the properties the invariant neighborhoods and that Pout is a unitary repre- sentation, i.e., Pout (R)"
"R)"PPn Pous
for all R E SO(3).
Using 25
26 and
we find:
SAIRX + (m)
Σ
(x &))v J.}B
Pout
JENT(X)
Pout (R)SA(X,FF
Thus each self-attention layer in is equivariant. The proof for each cross-attention layer in T similar. Again, note that if the closest neighbor of the query is not uniquely defined, then-as discussed above-we output the entire set of fields for every equivalent neighborhood. Then, the output is also set of fields and a roto-translation of the input will only result in permutation of these fields. Then, the "max" operation the output will eliminate the permutation, making the output equivariant.
4. From the constraints on the weights (Eq. 6 their solutions (Eq. Again, we focus on each layer separately we did in the previous section. The goal is to solve Eq.
By the Peter- Peter-Weyl theorem, Pin decomposes into unitary, irreducible representations of SO(3), pos- sibly with multiplicities. Thus,
) Qin, Re SO(3),
Pin (I) = Q'i (+)D.(
VTAI
where in our case Qin I, + for matrices denotes concatenation along the diagonal and [ indexes the irreducible types and my indexes the multiplicity of the (-th irreducible. Then, for each F in the point cloud (where we suppress the index for clarity), we have fin
θ θ finm We also consider an output field transforming according to Pout (R) = Q (+2 D.. (R))QQQuut Qout = I in case that decomposes fout D + k. Recall that each of the matrices Wq. WK. K, Wy are of dimension E. (2k: 1) x Σ my (21 1).
26

---

Published as a conference paper at ICLR 2023
1. For WQ, we have for all R E SO(3):
Pout WR (H)
D (R)JWQ Wol D((R) 4 L.my D& (R) WoD:(R)
R,
(27)
By Schur's Lemma AA88.1 for each block [Wolt.m. that multiplies fimm to create four (after adding all contributions), we have, for some constants C,mi
R.nk.
if if = k.
[ Iime =
(28)
121+1
Intuitively, the solution above says that the query cannot transform irreducible type different irreducible type (e.g., a scalar to vector), but only mix channels that correspond to the same irreducible.
2. For Wv, the constraint is that for all SO(3), the following set of equivalent statements holds:
Wv(R(T, Fim) Pm (R)
Pout (R) - T:) =
Wv(R( - D.()) = +
E..)
1,my
k,n
W - I:)) D.(R) = (R) - Iz )
k.n.
The solution, as discussed in the main text and solved in Weiler al. ct al. (2018) is:
(2018a):
Thomas
1+
V.((...m (1 - #11:0)) ( b 33-28 5
[Wv - M.Ma
Σ
|| -.
J=
where ( = YI (i) Qu E ((2+++)x(2++1)) are the slices from the QKi Maosch-Gordan matrices, Y, 52 (2J+1), is the J-th real spherical harmonic, (fm(f) = [+)( m is its m-th coordinate, and è = =/||=||.
3. For WK. we have the same equation as for Wv above:
Wx (R(Z, - P: R Pont R) (7,-
- F.).
but in addition we can constrain the blocks that transform to an irreducible that does not appear the input be zero without impacting the result. The reason is the form of Wq in Eq. 28 In particular, if l', k are different irreducibles in the input and output respectively, then the blocks [W x 1k m,,, that transform type-l' to type-k & only contribute as terms to the total inner product in the attention kernel as follows.
For all (1, my):
(Woltim Di(R),i,
E Di(R))i my
The above inner product is zero if ( + due to Eq. If & K. where I. is the set of irreducibles appearing in the input (i.e. the range of () then the inner product zero for all I, my. So, we might well choose:
28
[W - =1)) k. m. = 0, if I..
since parametrizing those blocks will not contribute the result. For the rest of the blocks we have, similar as before,
14
[ Fim]lime
Σ
(
(k.mc) (K(.J..m.))
(1
E11:0)c
| F.
J=
27

---

Published as a conference paper at ICLR 2023
The parametrization of the weights of the cross-attention module T is similar to Eq. 7 The only difference is that we can further reduce the number of parameters without impacting the result by using only the type-1 features of the keys in the first layer, i.e. [W 1. = O. for k * 1. This is due to the same equation involving the inner product between the key and the query we saw above, and the fact that the input query field is of type-1. In Fig- 3 we visualize the equivariance constraint on T, & by using a commutative diagram.
6. Additional Equivariant Layers: Clearly, the concatenation of multiple attention heads with the same irreducible types, as well as skip connection layers defined in section A.9 preserve equivariance. Since they add the multiplicities of each irreducible type independently. they only change the output representation by increasing the multiplicities of its irreducibles.
The subsequent mixing of channels of the same irreducible type by a linear map performed in the multi-headed attention module corresponds to the same operation that the query performs, thus also preserves equivariance.
Finally, the equivariant layer-norm layer also operates on each irreducible type independently. Since the irreducibles are unitary transforms .ireformmsii...... mm ll = [D l22 for all fi,m and all D; any non-linear operation on the norm of a type-/ vector produces a type-0 vector. Since fi.m S.m,//lf,,m. fm ll2 is again a type-/ vector, the final result is a type-l vector.
A Attention as a set operation: In addition to -quuivaauie, our model inherits the properties of the standard attention layers. Thus, it is equivariant to any permutation of the points in the point cloud, and to the order of the queries. Moreover, the number of output tokens can vary. We use this property for scene reconstruction, by reconstructing scenes of a variable number of points (and point clouds) even by training on single objects of a fixed number of points. The input (key-value) tokens can also change during inference, which we can use to adapt the neighborhoods dynamically during inference. This can counter that super-sampling a point cloud may reduce the receptive field.
8. On independent SO(3) rotations. Our local attention neighborhoods and equivariant reconstruc- tions are particularly important properties when reconstructing scenes. Here the point clouds can be independently placed in arbitrary positions. of our model in a particular scene to the performance in another scene, where the same point clouds have been independently roto-translated For a single object, the true occupancy function should be the same under a simultaneous roto -rootrraslltionn of the point cloud and the query. Our equivariant pipeline respects this property, by outputting a scalar field. Thus, the performance of our model is consistent independently of the SE(3) transformations of single point cloud.
natural to ask if we can connect the performance
Following similar approach for scenes with multiple objects, one can associate an independent copy of S0(3) acting on each point cloud, i.c., action X' + [x*[ for all objects i E 1. We can ask there is a transformation of the query q. that, after the rotation of the point clouds, results in the same occupancy value. Also, can this implemented without knowing the segmentation function that assigns the points to their point clouds? A reasonable first approach would be to transform the query with the rotation matrix R, used transform its closest neighbor.
Unfortunately, this transformation does not correspond to a group action. To understand this, con- sider two unit spheres at positions (0, 0, 0) and (1, 0, 0) in dime-dimmnsional space, and the query point 7 (- 10, 0, 0). Then, consider the product group action L1 = L .r L(o.0.0). that rotates the first sphere around the z-axis by 180 degrees- 11 = (0, 0, x) x)) and fixes the second sphere- ri = Next, consider the reverse action L2 = L0..0, If we first multiply the corresponding group elements together, we find L = L1 o La Le.e = T. If we then find the nearest sphere and apply the identity action the query, it will remain at (-10, 0, 0). On the other hand, if we first find the nearest neighbor-in this case, the first sphere-and apply the corresponding action, then 0, If we then apply the second action by finding again the nearest neighbor-nov the second sphere-the identity action will be applied and the query point stays at (10, 0, 0). Thus, this transformation does not satisfy the "ccoppatibllity" property and thus it is not group action.
the query will rotate around the z-axis by 180-degrees, moving
Even though the conditions are not met for all query points in space, there are subsets of I that are closed under these transformations and for which these transformations indeed form group actions. We will consider equivariance only in those subsets. We construct them geometrically for two point
28

---

Published as a conference paper at ICLR 2023
clouds for simplicity. Consider the point clouds X1, X2 rotating around points C12 C2 respectively. We construct the equivariant zone for the point cloud X1. First take the point in X2 that has the maximum distance (in Euclidean norm), say d2. its center of rotation Cz. Form the sphere S2 around X2, with a radius d2 and center c2. All of X2 is contained in S2. Connect the centers ç1, C2. and denote the point of intersection of this segment with S2 p. Then, draw the segment between p and e₁ and name the distance of e₁ to the middle point of this segment D Every point in the sphere Si of radius DJ and center ej in the equivariant zone of X,, called Z1.
We prove that in the equivariant zone, our model is equivariant, for any independent rotation of the point clouds. By construction, every query q in the first equivariant zone Z1 has its closest neighbor in the point cloud X1. This holds for any rotation of any point cloud. Then, by definition, the action on the query is the same rotation R, that transformed the points X₁. Since the point cloud and the query both rotate with the same transformation, the query neighborhood constructed within the cross-attention layers is learingg to the same closest neighbor-as as we proved for the single object case.
Now, suppose each point in X, has its nearest neighbors in X1, for any rotation of the point clouds; this is reasonable for sufficiently dense separated point clouds. Then these point cloud neighborhoods are also invariant after the individual rotations. Under these conditions, a direct product action of SO(3)-s the first equivariant zone is viewed by the attention network as a simultaneous rotation of the point cloud X₁ and the query q. This is because ((()) contains only points from X1, the attention module uses only key-value tokens from points in X1. Further, the neighborhood of the query is invariant simultancous rotation of X and Thus, we proved in the single-object case, the occupancy value prediction for this transformed query is invariant to the transformation. Thus, for all queries in an equivariant zone, equivariance to the direct product action of SO(3) holds, i.e., for all T1.72," -. ry SO(3), and for all q E Zj, with j E [2].
the point clouds with simultancous action of R₁ the query in
T((,, (x),, fr [X2), . ,r, x R = T X2,'" ,N)).
29


Reviews:
[{"review_id": "ewpwJETyDA", "reviewer": "Reviewer_Ut5F", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript describes a novel method to reconstruct continuous surfaces of scenes or objects from sparse point clouds. The method applies recent advances in SE(3)-equivariant transformer architectures to this problem. The transformer uses an encoder-decoder architecture. The encoder is an SO(3)-equivariant transformer which computes a per-point feature vector based on a local neighborhood of points (the encoder is translation invariant because only relative positions of neighbors are used). Different parts of the feature vector can be interpreted as being scalar-valued (as in a traditional network), vector-valued (as in Vector Neurons), or tensor-valued for arbitrarily high-order tensors. Because of the geometric interpretability of these representations, an SE(3) transformation of the input points induces a corresponding transformation of the representations output by the encoder. The decoder then takes as input the point cloud, the features output by the encoder, and a query point anywhere in 3D, and outputs an occupancy value. This network then implicitly defines a continuous occupancy field. The decoder is also SE(3)-equivariant, such that passing a transformed point cloud to the network results in an occupancy field transformed by the same SE(3) transformation. The manuscript demonstrates how this can be used to infer shapes in novel poses that were not seen during training, which has implications for both training efficiency and composability of representations.", "summary_of_the_review": "This paper is a good example of a step change in performance; the proposed networks process input point clouds in a fundamentally different way (i.e. they are SE(3)-equivariant) from prior works, and the result is a large gap in performance in scenarios where this is important (e.g. inference of shapes in a different pose than they appeared in training. The novelty may be limited, but the empirical contribution is nevertheless compelling.\n", "strength_and_weaknesses": "Ultimately, this manuscript presents a straightforward application of an existing technique (SE(3) Transformers) to an existing problem (occupancy modeling). The novelty is somewhat limited, but the work is timely and does contribute novel and valuable experiments to the literature. Occupancy modeling is a great application for SE(3)-equivariant networks, which remove the need for rotational data augmentation to handle arbitrary rotations. Even more compelling, when combined with local feature extraction, the proposed method can be applied to infer scenes comprising novel combinations of familiar shapes, which would otherwise require data augmentation that generates an immense number of combinations of shapes and poses. The experiments clearly demonstrate the value of the approach compared to a variety of baselines.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "eK-oLvIHWN", "reviewer": "Reviewer_dRvs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents an SE(3)-equivariant transformer architecture for occupancy prediction from point clouds. The network uses two types of attention layers, one attention layer on edges in the point graph and one cross attention layer to aggregate information from local neighborhoods in the output layers.\nFurther, the network utilizes Wigner-D harmonic bands of degree 2, allowing the network to model certain symmetries in the data.\nThe network is evaluated on I/I, I/SO(3) and SO(3)/SO(3) setups on ShapeNet, a synthetic scene dataset based on ShapeNet and Matterport3D, a real scene dataset.", "summary_of_the_review": "I lean towards rejecting this work, based on the above mentioned problems. I am not fully convinced by the experiments. There are many methods doing 3D reconstruction with neural networks now and I have seen much more detailed reconstructions [1, 2, 3] against which this method is not compared. The authors claim that their SE(3)-equivariant formulation is very beneficial for scene reconstruction but at the same time, they only show very few examples of 3D reconstructions on real datasets and do not provide a quantitative evaluation or ablation studies.\n\n[1] Chen et al.: 3D Equivariant Graph Implicit Functions, ECCV 2022\n\n[2] Jiang et al.: Local Implicit Grid Representations for 3D Scenes, CVPR 2020\n\n[3] Chabra et al.: Deep Local Shapes, ECCV 2020\n\nMinor notation problems:\n- sometimes, x_i is used to describe the relative position (Fig2), sometimes to describe the absolute position (3.1)\n- neighboring point pairs are sometimes denoted as x_i, x_j, and sometimes as x,y", "strength_and_weaknesses": "Strengths:\n- Utilizing harmonic bands of degree > 1 makes sense to allow the network to model symmetries within the input. To my knowledge, it is first applied to 3D reconstruction in this work. Similar ideas have been seen in 2D before (harmonic networks).\n- I like that the SE(3) equivariant formulations are well grounded in group theoretic considerations of irreducable representations.\n- The paper is technically sound\n- The paper is well written\n\nWeaknesses:\n- The novelty of the work is limited. There have been SE(3)-equivariant networks in a very similar setting before [1]. The notable addition to this work are the additional harmonic bands.\n- The paper leaves out some of those work in its comparisons.\n- In general, it is not evaluated if the additional harmonic bands (type-2) improve the results. The claim by the authors is not supported in experiments or theory. An ablation study, showing the benefits, is needed. Also, the a qualitative evaluation of these feature maps would be interesting. It might be that one can actually see different behaviour on symmetric objects.\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MxYAUkXsUZ", "reviewer": "Reviewer_YvKA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel architecture to obtain local shape modelling and SE(3) equivariance for the task of 3D shape reconstruction. These properties are obtained by generalizing the attention mechanism and imposing specific equivariance constraints on the layer weights. The method can be trained on single aligned objects, and at inference, time can naturally work both with single roto-translated objects and also with complex scenes.", "summary_of_the_review": "Given that I am not high-confident about the math and parts of the explanation are confusing, I cannot assess the correctness of the method. I think the method requires substantial clarification before publication, and there are unjustified lacks in the experiments and no mention of limitations or future works. For this reason, I think the paper requires some work and cannot vote for acceptance yet.\nLooking forward to the rebuttal; I will be happy to raise my score if it addresses my concerns.", "strength_and_weaknesses": "STRENGTH\n===========\nS1) GENERALIZATION: The method can generalize to complex scenes without seeing them at training time. I am not aware of other works with the same capability. This is a particularly compelling property.\n\nS2) SIGNIFICANCE: The design of equivariant networks is an active research field which collects significant interest from Computer Vision, Graphics, and ML communities. The new technique proposed in the paper can open novel research directions with a consequent tangible impact.\n\nWEAKNESSES\n============\nW1) CLARITY: I am not completely familiar with all the involved math, and not all passages are clear to me. Often, the maths heavily breaks the text (e.g., after Eq. 5), notation is pretty heavy (e.g., Eq. 6), and some background concepts are introduced during the method explanation interrupting the narration (e.g., after Eq. 6). In this sense, I highly recommend to revise the text, grouping symbols and concepts definition before method presentation, and well separating mathematical derivation from the textual narration.\n\nW2) APPLICATIONS AND LIMITATIONS: The proposed method is tested only on the shape reconstruction task. Other reported networks have shown results on different applications and challenges. There is no mention of possible limitations, and I do not see methodological limits to applying the formulated attention also on other tasks (e.g., segmentation, classification, matching) with minor modification. Testing on different tasks (and also domains) is standard for novel feature extractors (e.g., Vector Neurons).\n\n\nMINORS\n=======\nQ1) Is it possible to manipulate the Query Tokens to obtain a modification of the output shape?\nQ2) How the model performs in scenes where target objects are not separated in space (i.e., they are close, as objects in clutter)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2fcFWwRFSbf", "reviewer": "Reviewer_i4in", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes SE(3)-equivariant coordinate-based model for shape reconstruction. The model consists of a self-attention model to encoder the points in the point cloud and a cross-attention model to output the occupancy of any query point. Both models are made SE(3) equivariant. The local attention and the SE(3) equivariance allows compositional generalization. \n", "summary_of_the_review": "This paper makes a solid contribution to the problem of shape reconstruction from point cloud. The proposed self-attention and cross-attention modules are interesting and useful. ", "strength_and_weaknesses": "Strengths: \n\n(1) Both the self-attention and cross attention models are well designed and prove to be effective. \n\n(2) The design of SE(3) equivariant attention is based on rigorous group representation consideration. \n\n(3) The experimental results on compositional generalization are impressive. \n\nWeakness: \n\nThe difference and novelty relative to existing equivariant attention networks should be explained more carefully. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster