An Intuitive Multi-Frequency Feature Representation for SO(3)-Equivariant Networks
Authors: Dongwon Son
Jaehyung Kim
Sanghyeon Son
Beomjoon Kim
Venue: ICLR 2024 poster (2024)
Keywords: Equivariant networks
SO(3) Equivariance
Fourier features
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a frequency-based rotation equivariant feature representation for 3D data.

Abstract: The usage of 3D vision algorithms, such as shape reconstruction, remains limited because they require inputs to be at a fixed canonical rotation. Recently, a simple equivariant network, Vector Neuron (VN) has been proposed that can be easily used with the state-of-the-art 3D neural network (NN) architectures. However, its performance is limited because it is designed to use only three-dimensional features, which is insufficient to capture the details present in 3D data. In this paper, we introduce an equivariant feature representation for mapping a 3D point to a high-dimensional feature space. Our feature can discern multiple frequencies present in 3D data, which, as shown by Tancik et al. (2020), is the key to designing an expressive feature for 3D vision tasks. Our representation can be used as an input to VNs, and the results demonstrate that with our feature representation, VN captures more details, overcoming the limitation raised in its original paper.

Published as a conference paper at ICLR 2024
AN INTUITIVE MULTI-FREQUENCY FEATURE REPRE- SENTATION FOR S0 NETWORKS
Dongwon Son, Jachyung Kim, Sanghyeon Son, Beomjoon Kim
Department of AI, KAIST {dongwon. son, kim kimjaehyung, ssh.88on, kimaeeaao......kktakt.tt........ kr
ABSTRACT
The usage of 3D vision algorithmss, such as shape reconstruction, remains limited because they require imputs be fixed canonical rotation. Recently, simple equivariant network, Vector Neuron (VN) (Deng et al., 2021) has been proposed that can be easily used with the state-of-the-ard 3D neural network (NN) archi- tectures. However, its performance is limited because it is designed to use only three-dimensional features, which is insufficient to capture the details present 3D data. In this paper, we introduce an equivariant feature representation for mapping a 3D point to a high-dimensional feature space. Our feature can discern multiple frequencies present in 3D data, which, shown by Tancik al. (2020), is the key to designing an expressive feature for 3D vision tasks. Our represen- tation can be used as input VNs, and the results demonstrate that with feature representation, VN captures more details, overcoming the limitation raised in its original paper.
Ground truth
OccNet
VN-OceNet
Ours
Figure 1: EGAD (Morrison et al., 2020) meshes constructed from the embeddings given by different models based on OceNet (Mescheder et al., 2019) canonical poses. As already noted in their original paper, VN-OccNet (3rd column), the VN version of OccNet, fails to capture the details present in the ground-truth shapes and does worse than OceNet (2nd column). Using our feature representation, VN-OccNet qualitatively performs better than OccNet (4th column). Note that each of these shapes consists of multiple frequencies in some parts of the object, the shape changes abruptly, while in some parts, it changes very smoothly.
INTRODUCTION
SO(3) equivariant neural networks (NN) change the output accordingly when the point cloud input rotated without additional training. For instance, given a point cloud rotated by, say, 90 degrees,

---

Published as a conference paper at ICLR 2024
an SO(3) equivariant encoder output embedding rotated by 90 degrees without ever being trained on the rotated input. Pioneering works such as TFN and SE(3) transformers (Thomas et al., 2018: Fuchs al., 2020) use tools from quantum mechanics to design equivariant network, but they are difficult to understand and require a specific architecture that is incompatible with recent architectural advancements in 3D vision.
More recently, Vector neuron (VN) (Deng et al., 2021) has been proposed as a more accessible alternative. By extending a neuron, which typically outputs scalar, output three-dimennionall vector and modifying typical operations in NNs (e.g. ReLU activation function, max pooling, etc.), VN guarantees equivariance. Further, because it uses the existing operations in NNs, can easily be implemented within well-establiseed point processing networks like PointNet (Li et al., 2018) and DGCNN (Wang et al., 2019). However, their key limitation is the low dimensionality of the features: unlike TFN or Srans-transformmes which use nigh-dimensional features, a feature in vector neurons confined 3D space. This limits the expressivity of the features and cannot capture the details present in 3D data, as depicted in Figure 3.
In this work, we propose an equivariant feature representation for mapping a 3D point to a high- dimensional space. The key challenge here is designing an algorithm that computes expressive yet rotation equivariant features from 3D point clouds. As shown by Mildenhall et al. (2021) and Tancik et al. (2020), for 3D data, the effectiveness of features heavily depends on its ability to represent different frequencies present in the input, as 3D shapes typically are multi-frequency signals (e.g. Figure 1). So, one way is use Fourier basis that consists of sinusoids of various frequencies as done in NeRF (Mildenhall et al., 2021). However, this does not guarantee cquivariance.
Based on the observation that a rotation matrix can be written as sinusoids whose frequencies are determined by its eigenvalues (Fulton & Harris, 2013), we instead propose to construct a mapping D SO(3) + (O(n) and use D to define our feature representation in way that is provably equivariant. At high-level, our idea is to describe the orientation of a given point i E R from a basis axis, say 8, denoted R (it), and then use D map R to a higher dimensional feature space such that it defines the same amount of rotation from basis axis in R Then, we simply apply D(R (1)) to the chosen basis axis in R." to get our feature representation of i7. This guarantees equivariance because when we rotate 77, the corresponding feature would also rotate by the same amount. Figure 2 demonstrates this intuition.
We propose a set of mathematical conditions for constructing D that achieves this, and prove that our feature representation is equivariant. Further, we show this rotation-matrix-le based feature repre- sentation can be written as sinusoids, much like in Fourier bases, whose frequency is determined by the maximum eigenvalues of D(R). Based on this result, we propose an algorithm that can con- struct & that consists of sinusoids of frequency n=1
8 (a)
((R)
a
<RR
< R 3
Figure 2: Intuition of our equivariant feature representation, v. that maps a point 2D to 3) for an illustrative purpose. (Left) The basis axis in 2D is & = [0. 1]. and = R$ (ū)E, with f its amount of rotation from é. Our D is constructed so that it defines the same amount of rotation, 6, but from a basis in the 3D space, which in this case is chosen to be é = [0, 0, 1). The feature representation of a is given by v())=
We call our representation Frequency-based Equivariant feature Representation (FER). We use it as an input to VN instead of 3D points and integrate VN into standard point processing architectures, PointNet and DGCNN, and show TER-VN-PPintNet and FER-VN-DGCNN achieve ofite-of-th-aase art performance among equivariant networks in various tasks, including shape clas- sification, part segmentation, normal estimation, point completion, and shape compression. No- tably, unlike the standard VN which performs worse than the non-equivariant counterpart in the point completion and compression tasks at a canonical pose, we show that FER-VN outperforms both of them by capturing the -igh-frequences present in the 3D shape as illustrated in Figure 3.
3D (i.e.
((R((M)) As you can see, when I changes, rotates both å and (ū) by the same amount. Note that the description of magnitude neglected for brevity.
2

---

Published as a conference paper at ICLR 2024
2 RELATED WORKS
2.1 (O(3) EQUIVARIAT NETWORKS
For 3D inputs, several equivariant methods have been proposed (Weiler et al., 2018; Thomas et al., 2018: Fuchs et al., 2020; Brandstetter et al., 2021; Anderson et al., 2019). These models use an equivariant feature representation called spherical harmonics to map 3D points to an n dimensional feature space where rotations described using Wigner-D matrices. While this representation is equivariant, this Wigner-D matrix is from quantum mechanics originally used to describe quantum states (Sakurai & Commins, 1995), and it is difficult to understand why they work, and what the intuitions behind the hyperparameters are. For instance, what does the number of dimensions of representations imply? Does it relate the intuition that we need to discern multiple frequencies? For readers without background in quantum mechanics, these questions are hard to answer. Fur- thermore, these methods are confined to using specific architectures, making difficult leverage state-of-the-ar 3D vision architectures. One of our goals in this work is to offer a more accessible and intuitive alternative to derive high-dimensional frequency- -ased equivariant feature represen- tation.
Recent advancements focus on creating more adaptable SO(3) -equiivaaaa neural networks, espe- cially for point cloud processing (Deng et al., 2021; Puny et al., 2021; Kaba et al., 2023). Frame averaging method (Puny et al., 2021) proposes to achieve equivariance by averaging over a group through symmetrization, but its of Principal Component Analysis (PCA) makes it sensitive to noisy and missing points, a phenomenon prevalent in 3D sensors. Kaba et al. (2023) introduces method for equívariant structure by training extra network that maps rotated inputs canonical pose. However, a small error in this network could have a large negative impact on the overall performance. Among these, Vector Neurons Deng et al. (2021) stands because it achieves cquivariance by simply modifying the existing operations in NNs. Its key limitation however is the confinement the 3D feature space. Our work offers a high dimensional frequency- -ased represen- tation that can be used with VNs.
2.2 FREQUENCY-BASHH FEATURE REPRESENTATION
Frequency-basse bbase sinusoidal feature representation has been widely adopted in 3D vision. Perhaps the most well-known usage is in NeRF (Mildenhall et al., 2021), where it maps a viewing position and angle into high-dimensional, multi-frequency sinusoidal feature representation. Similarly, Tancik et al. (2020) shows that using sinusoidal feature representation achieves significantly better performance in shape compression task than using 3D coordinate as an input to the network. The reason this works better than the coordinate-basee representation is that, for a complex shape, some parts are very non-smooth and change rapidly with respect to the changes in the position on the shape, while some parts are smooth (e.g. shapes in Figure 1). Using sinusoidal feature representa- tion, such as Fourier basis, at different frequencies helps because they can discern both smooth and non-smooth changes. However, the conventional Fourier basis is not SO(3) equivariant. Our feature representation guarantees equivariance and captures multiple frequencies
3 FREQUENCY-BASED EQUIVARIANT FEATURE REPRESENTATION
Our feature representation v 3 + Rd defined as
(( ((E))()
where ū the input point, is a basis in the dimensional space, Rū) is rotation matrix defining the orientation measured from z-axis (i.e. [0, 0, 1)) to û. We will show that by construction, & is provably equivariant, and that it consists of sinusoids whose maximum frequency is ""=1 / Let us first describe the conditions and construction of D that make this possible. All proofs for theorems and propositions in this section are in the appendix.
First and foremost, we would like D(R) to be an element of S0(n). Second, we would like to map every element of R SO(3) unique element in SO(n) so that changes in SO(3) are uniquely
3

---

Published as a conference paper at ICLR 2024
captured in SO(n). This gives us the following set of conditions for D.
vR1. R2 E SO(3), if R & R2, then ((R)) # D(R2),
vR E SO(3), D maps R to single matrix D E _
VR SO(3), D()) D(R)" ( = 1,
VR & 50(3), det ((R)) =.
The first two conditions ensure that D uniquely maps for all E SO(3). The last two conditions ensure that D(R) belongs to SO(n) (Stillwell, 2008).
To construct D that satisfies these conditions, we use the exponential form of rotation matrix. In R3, given R S0(3) expressed in axis-angle representation with i R as its rotation axis and angle A E R. we have exp(@r F) where w F =E= w; Fs. W6 as 8th element of w, and F = [Fi, F2, Fal.
000
8
80
0
F =
0 P/2
F3
1
01
Intuitively, F2, F3 describe the rotation about the z,y. axes respectively (Fulton & Harris, 2013). For instance, when à [1, 0, 0j. then R is determined solely by Fj.
We generalize this representation n dimensional space, where we use J J2. J. Rax", so that
D(R) (tu J)
33)
where 0 and wi denote the angle and axis of rotation of R E 50(3), respectively. Intuitively, just like Fi, F-2 and F3 represent axes of rotation about æ, y and axes in IR J1, J2 and Ja represent the angles about axes (a)), (()). and (2).. The key here is that they describe the same amount of rotation angle even when they are in different spaces. For instance, if we set 0 [0, 0, 0g] and à = [0, 0, 1], then the only contributing term for D(R) is JzJ Based on this observation, we will construct J instead of constructing D directly. We have the following helpful theorem.
Theorem 1. If J3 E Ruxn Vi E {1,2,3] satisfies -j, = J,", [J1. Js. [J2 JI, (Ja, Ji] J2 where [A, B] AB BA, and exp(2mm.J, Inx, Vm E Z, where the space of integers, then D(R) = cxp(pet (é: J) E (O(n) and satisfies the conditions Equation 2.
design algorithm for constructing J. But before doing so, let us show the relationship between the eigenvalues of D(R) and the frequency of th. This will provide a way to design J so that to has the maximum possible frequency for a given n. The following propositions and theorem will help do that.
Given these conditions,
Proposition Suppose J1, J2, Ja satisfy the conditions in Theorem 1. Then, J1, J2, and Ja have the same eigenvalues and multiplicities. In particular, the eigenvalues are A == t-ki, -(k 1)i.....-in 0, i..,..., ki} for some non-negane non-negative integer Further, if X an eigenvalue of Jg with multiplicity m. then > is also an eigenvalue of Jr with the same multiplicity m.
Proposition Suppose J1, J2, J3 satisfy the conditioon Theorem 1. Then, Ow I have the same eigenvalues Jr, A = {-kfi, -(k 1)0i..... .... -i, 0, ... kkii
Theorem 2. Suppose D satisfies the conditions in Equation 2, and J1, J2, Ju satisfy the conditions in Theorem I. VR E S0(3) whose angle of rotation is 9 and rotation axis is Ws, we have
D(R) exp(B@ J) Σ
bAbk (sin(20) cos(()))
where by is the eigenvector of tw JJ that corresponds to eigenvalue X.
Note the correlation between the frequency of D(R) and the magnitude of X. If we have large X, then D(R) consists of sinusoids with large magnitude, and vice-versa. The following proposition gives insight to why high-dimensional features necessary.
4

---

Published as a conference paper at ICLR 2024
Proposition 3. The maximum value of l in Proposition 2 is 1=11]
Therefore, for all R SO(3), the maximum frequency D(R) can have is -/-1//2 This shows that if our feature space is say, dimension 3, then the maximum frequency it can attain is just 1/2m.
Finally, we have the following equivariance theorem for FER.
Theorem 3. Consider the mapping (ū)) (ū))e, v rotational equivariant if is the eigenvector corresponding to the zero eigenvalue of J3, and D satisfies all the conditions in equation
The reason that we have the condition for è is that space to û, R (ū). to express its orientation. Note that we could have chosen instead to use R (ū) or R (ū) in vb. In such cases, would have choose é eigenvector with the zero eigenvalues of J1 or Jn respectively.
using the angle from the z-axis in 3D
We now have all the ingredients for constructing D. We will ensure construct D to satisfy all the conditions in Theorem and " ii included in J so that it can capture the maximum frequency with the given dimension n. Algorithm gives pseudocode for doing this.
Algorithm CONSTRUCT J1,
Algorithm SAMPLEJ/3
Require: n
Require:
I: J3 & SAMPLE/3(n)
I: A t randomly fill 1 matrix
2: J1,J2 + CREATESEARCHSPPCE(.))
J3 f- A A-A A
3: J1, + argmin [I[J. J2] - Jallp
3: U <
JIEJ,
4: A DIAGONAL
([-
n-
- Jil)
4: return [J1,J2,
J3 + UAU*
6: return J3
Given n, the algorithm begins by constructing Ja by calling SAMPLEEJ- Algorithm 2 begins with a construction of a random skew-symmetric matrix (L112), and set its eigenvalues to A [- - li] while keeping the eigenvectors unchanged (L3-5). Given that the returned Jy has eigenvalues A and is skew-symmetric, we have that (xp(2kx.J. Inxn by Proposition 4 in the appendix.
We then return to Algorithm 1 to construct J1 and J2, such that they satisfy conditions in Theorem 1. We now want to find J1 and J2 such that they satisfy
[Jg. J1] = J2. [J2, Jal = J1, J J1 J1 [J1, J2l = J3.
(4)
Note that if we satisfy these conditions. J2 + J. = O would automatically satisfied. To find Ji and J2 that satisfy them, first define the space of J1 and Jz that satisfies the first three conditions, by realizing that these equations define an under-determined linear system of equations whose unknowns the last condition, we solve the non-linear optimization problem using the Cross-Entropy Method (CEM) (De Boer al., 2005) (L3).
the elements of J1 and J2 (for details, Appendix F) (L2). Then, satisfy
EXPERIMENT
We conduct our experiments on both $O(3)-invarian) task and SO(3)-equivariant task. We use three tasks adopted from Deng et al. (2021): point cloud classification (invariant), segmentation (invari- ant), and point cloud completion (equivariant in the encoder, invariant in the decoder). Further, we evaluate them on three more SO(3) equivariant tasks: shape compression, the normal estimation, and point cloud registration, adopted from Mescheder et al. (2019), Puny et al. (2021), and Zhu et al. (2022). In all subsections, we call our approach FER-VN.
5

---

Published as a conference paper at ICLR 2024
4.1 POINT CLOUD COMPLETION
The objective here is to reconstruct the shape, expressed with a neural implicit function, from partial and noisy input point cloud. The ar- chitecture and training details can found Appendix E.2.2.
Method
V/1
USO(()
80(3Y50(3)
OccNett (Mescheder et al. (2019))
71.4
30.9
58.2
VN OceNet (Deny al. (2021))
69.3
69.3
68.8
FER- FER-VV-OccNee (Ours)
71.9
71.9
71.9
Volumetric mloU on ShapeNet recon- struction with neural implicit. These results are av- erage category mean IoU over 9 classes. T/T indi- cates train and test with canonical poses, (3) indicates train with canonical poses and test with different SO(3) rotations, and 50(3)/5((3 indi- cates train and test with different SO(3) rotations. Bold is the best performance.
Table 1:
Dataset: We use ShapeNet consisting of 13 major classes, following the categorization in Deng et al. (2021). The imput point cloud P is comprised of 300 points, sampled from each model's surface and perturbed with noise ~ N(0, 0.005). Query points for ShapeNet are uniformly sampled within the Axis-Aligned Bounding Box (AABB).
Table 1 shows the volumetric mean IoU for reconstruction. As already noted in the original paper (Deng et al., 2021). VN-OccNet performs worse than OccNet at canonical poses. FER-VN OccNet on the other hand, outperforms both approaches by utilizing frequency-basee features. This evident in Figure 3, where, unlike Vector Neurons, our method captures details present in cars (wheels, side mirrors) and chair's s legs. We further demonstrate that FER enhances model robustness across various sample sizes, detailed Appendix G.I.
Results:
(500))
5009500
50C//5009
1
Yenr
Nerren
Natwork
Figure 3: Reconstructions of meshes from point cloud inputs across three models: the original OccNet Mescheder et al. (2019) (bottom), VN-OccNet Deng et al. (2021) (middle), and our proposed model (top).
4.2
SHAPE COMPRESSION
Here, the task is to compress 3D shape into embedding, and then reconstruct the same shape when just given an embedding. The purpose of this experiment is to investigate how well different models capture details, and in particular, whether our network can learn capture high-frequency details and compress them to an embedding.
Dataset: We use EGAD dataset (Morrison al., 2020) comprised of 2281 shapes. EGAD dataset provides the categorization of the shapes by shape complexity level from 0 to 25, measured by the distribution of curvature across vertices. We train our model on the given dataset, and save the embeddings. We then reproduce original shapes based on just the embeddings. We use OceNet as the basis model. The training point cloud is sampled following the strategy in section 4.1. The query points sampled both from the surface and AABB at a 9:1 ratio, with surface points perturbed by the noise 8 (0, 0.025). For all models, we train with various different rotations.
Figure 4 shows the results. As can as the complexity of the shape increases, the performances of VN drop significantly, while that of ours drops at slower rate, which again demonstrates the effectiveness of frequency- baseed representation. Figure shows the quali- tative result at the canonical pose. Our model is able to reconstruct the high-frequency details of the shapes, while VN-OccNet and standard OceNet that use coordinate-basse inputs smooth
Results:

---

Published as a conference paper at ICLR 2024
15
IENVN--c:NNe-- SE-0ccNat50311
T0U (55)
80
70-
PER YN Docket 50131
SO(3)
50
5
10 Complesccy
20
10
15
Complexity
Figure 4: The left graph shows the volumetric IoU of OceNet, VN-OceNet, and FER- FER-VN-OOe VN-OeeNet across the complexity level in the EGAD training set. We apply rotational augmentation during both training and test time. The right graph shows FER-VNsOceNet's IoU improvement over VN-OccNet.
or miss the detail, again showing the effectiveness of the frequency-basee representation. Dimen- sional analysis reveals that FER enhances detail accuracy with reduced inference impact, detailed in Appendices G.3, G.2, and G.6. Additionally, we demonstrate improved IoU of our model with consistent compression ratios in Appendix G.4 and report advancements over recent VN-based model in Appendix G.7.
4.3 NORMAL ESTIMATION
The goal of this equivariant task is predicting the normal direction of point cloud. The Model- Net40 (Wu et al., 2015) and ShapeNet (Chang al., 2015) datasets are used. For each ob- ject, we sampled 512 random surface points. As shown in Table 2, our method outperforms both Vector Neurons and a preceding rotation- equivariant method by Puny al. (2021).
ShopeNet
SO00S003)
PuintNc: al. (207Ta)
0.214
0.204
PainNee (Deu (2021))
0,216
0.219
PA.Niinaa (Puny at (2021)
0.308
0.315
0.52
0.156
pan vx Pointeer (Dans)
0.195
0,198
0.125
0.126
DOCNN (Wong al. (2009)
0.212
0.170
0.301
0.109
VN DGUNN (Degg et (20219)
0.152
0.152
0.022
0.089
EN-DOUNN (Pom et (2021))
0.150
0.155
0183
0/001
PER FER-YN-DGCNN (Ouns)
0.143
0.142
Table 2: Test normal estimation results on the ShapeNet and ModelNet40 dataset. Numbers in- dicate the evaluation metric adopted from Puny et al. (2021) 1 - (n" n) where is prediction ñ and is ground-truth. Bold is the best perfor- mance.
4.4 POINT CLOUD REGISTRATION
The objective of point cloud registration is to align two sets of points Pi and P2 that come from the same shape. Registration algorithms such as ICP typically try to find the correspondences between the points in Pi and P2 but they are prone to noise. Zhu et al. (2022) proposes a different approach, which solves for the latent codes Z1, Z2 E Cx3 using VNs. Following this, use the encoders of various models from the shape completion task (section 4.1), find the orientation in the feature space, and from this determine R. The details can be found in Appendix E.2.4.
Dataset: We use ShapeNet Chang et al. (2015). We generate two point clouds sampled from the same shape, with one being randomly rotated from its canonical pose. Unlike the studies by Zhu et al. (2022) and Yuan et al. (2020), which employ the ModelNet40 dataset and generate denser point clouds of 1,000 or 500 points, we opt for sparser point cloud of 300 points using ShapeNet. Our experiments feature three test configurations: 1) P2 is created by copying Pi (Copy), 2) Pa consists of different points from the same shape from P1 (Distinct sample); 3) P2 consists of different set of points and density from P1 where the number of points for P2 varies between 50 and 300 points(Varying density). We evaluate our methodology against three baseline approaches: VN-EquivReg (Zhu et al., 2022), DeepGMR (Yuan et al., 2020) following the setting of (Zhu et al., 2022), and Iterative Closest Point (Bes) & McKay, 1992). Due to the symmetrical character- istics of ShapeNet objects and the absence of point correspondence information, we use Chamfer Distance (CD) as our evaluation metric and skip the training with pose error in Zhu et al. (2022).
Our hypothesis posits that our feature's ability to capture input details results in consistent feature encoding and more robust point cloud registration. Table validates this hypothesis. While all learning- based methods provided, our in the other two settings where we use different point samples and have different density in Pa from Pt. Figure 5 shows the qualitative results. VN-EquivReg struggles to distinguish
Results:
accurate in recovering the original rotation when rotated copy is

---

Published as a conference paper at ICLR 2024
Tixt
Volk Naxes
Das
Roeated point cloud
Capy
Distinct sample
Varying demsity
KPP
0,02059
DeapKMMK (Yomn et (20130))
000000
0301574
Se---EuiiiRg Brg (Zhu at al (2022))
6,00000
000560
0.01077
FRR-V Fra-vn-FquisRR Rog (Oers)
0,00000
(UN0347
0,00714
No retason
0100000
0000510
0:00611
Table 3: Registration results on the ShapeNet dataset. The metric is Chamfer Distance. Bold is the best performance.
Figure 5: Qualitative results of point cloud reg- istration. Red is P1 and blue is P2 the "Dis- tinct sample" setting.
Methods
+/SOi3)
(0(3)/50(3)
5/2
s/80())
SOCS/SO03)
Point mech inpuis
Natter
ShellKet (Zhong etal. (2019003
PuintNcs- (Qk al. (2017b))
DOCNN (Warg cial. (2019)
Pa (Li at (2018)
PounNe: 7207
9033
03.1
X5.3
91.5
01.5
41.2
28.4
33.8
1955
1966
578
$8.6
54.5
65.0
14.7
Neihher
SheliNet (Zhang
PountNate- (Q) ct (20017b))
DOCNN (Wang et (2019))
PoinsCNN (1 al. (2018g)
PoimNee (Qr @i (2017a)1
(2019b))
35.0
09.1
48.1
47.2
$4.7
78.6
77.1
71.4
62.3
76,7
VK. Poire (Dreg a al (20213)
77.5
17.2
VT. NN-Tom (Degg
DGCSN (Degg
(2021))
50.5
VN-DGGNN
(2021))
81.4
Rotztion
PER:
(Ours)
$2.1
Rotation appicaaan
DDKNN (Duna
90.5
50.5
$0.5
equivariant
PUR-Vx-DGCNN (Chus)
SVNet DGUNN (Sc (30220)
90.3
5413
5000
SVNN-DGONN eS (2022))
814
81.4
Sphencca CNN [Esteves ct (2018))
oNNAUNN et (2018))
TFN (Thmma al. (2018)
SPCNN et (2019))
(Zhong
88.5
85.9
85.5
01.4
76.7
K7M
65.3
54.8
67.6
55.7
56.0
30.1
Rotation invoinnt
RI dl-Frameook (1i et (2021))
R12 (Zhing ct (2015)3)
GC-Conv (Zhang at. (2020))
(Thomss al. (2018))
79.2
76.8
77.1
753
77.3
79.4
76.2
75.3
Rocation imaiiantt
ChuserNea (Chen si (301
(Poulcuaa (2019))
83.1
$6.6
671
$76
TetrSpphre (Meleyk ct al. (3022))
PaRINet (Chea Cong (20122)
838
82.3
83.8
82.1
GC Coar (Zang (20250)
99.0
59.1
59.
RI-Fomowook (E. ctal. (20211)
894
69.4
69.3
PuRINer (Ches Cong (5022)
91.4
91.4
91.4
Table 5: Test part segmentation results on the
ShapeNet dataset. These results are average cat- egory mean IoU over 16 classes. Bold is the best performance and underlined is the next best per- formance.
Table 4: Test classification accuracy on the Mod- elNet40 dataset. Bold is the best performance and underlined the next best performance.
between similar features, like an airplane's 's tail and head, or the shade and base of lamp. In contrast, our method successfully captures finer details, such as the airplane's 's tailplane and the lamp's 's cone- shaped shade. We additionally illustrate that FER improves model robustness against diverse initial perturbations, described in Appendix G.5.
4.5 POINT CLOUD CLASSIHICATION AND SEGMENTATION
Dataset: For both tasks, we adopted the experimental data setup from Deng et al. (2021). The ModelNer40 dataset (Wu et al., 2015), used for object classification, comprises 40 classes with 12,311 CAD models-9843 are designated for training and the remainder for testing. For the object part segmentation task, the ShapeNet dataset (Chang al., 2015) was used, containing over 30,000 models spanning 16 shape classes.
The results of the object Table contains the results of the object classification experi- ment, showing the classification accuracy for three different augmentation strategies. Our method- ology was benchmarked against multiple ixtation-equivarian and -invariant methods. The methods are grouped by whether it is equivariant, invariant, or neither. In the case of rotation-equivariant methods, they originally maintain the orientation on the feature space but achieve invariance via an additional invariant layer. Even though our method is primarily votation-equivarinn after the PaRINet (Chen & Cong, 2022) which is specifically designed for invariance, especially in z/SO(3) and (0(3)/S0(3) setups.
Results:
ours is only
CONCLUSION
In this work, we propose FER, frequency-based equivariant feature representation tailored for 3D data. Our approach relies on the fact that rotation matrices can be written as sinusods, whose maximum frequency is determined by the dimensionality of the rotation matrix. This was made possible by defining a mapping function D that maps 3D rotation to SO(n) space, such that the basis axes in 3D are preserved in the dimensional space. When used with VN and state-of-the
8

---

Published as a conference paper at ICLR 2024
point processing networks such as PointNet and DGCNN, experimental results demonstrate that it captures details that previous methods fail to capture in various different 3D vision tasks.
REPRODUCIBILITY STATEMENT
Appendix E contains the details of our architecture and hyperparameters necessary to reproduce results. Also, our code is available at sttpss//gitt http:/ggttttutttt com/ PeEemcyltt frepuunccce..
ACKNOWLEDGMENTS
This work was supported by Institute of Information communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-0005. Artificial In- telligence Graduate School Program (KAIST)), Reinforcement Learning Techniques for Contact- Rich Robotic Manipulation of Everyday Objects), (No. 2022-0-00612, Geometric and Physical Commonsense Reasoning based Behavior Intelligence for Embodied AT).
0t.202220-000111 Development of Goal-Oriented
REFERENCES
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems, 32, 2019.
P.J. Besl and Neil D. McKay, A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2):239-255 1992. doi: 19.1009/34.211991
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Ge- ometric and physical quantities improve (3) equivariant message passing. arXiv preprint arXiv:2110.02905. 2021.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint
2015.
Chao Chen, Guanbin Li, Ruijia Xu, Tianshui Chen, Meng Wang, and Liang Lin. Clusternet: Deep hierarchical cluster network with rigorously
representation for point cloud anal- ysis. In Proceedingy of the IEEE/CVF conference computer vision and pattern recognition, pp.
2019.
Ronghan Chen and Yang Cong. The devil is in the pose: Ambiguity-fre 3d rotation-invarian learning via pose-aware convolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7472-7481, 2022.
Yunlu Chen, Basura Fernando, Hakan Bilen, Matthias Nießner, and Efstratios Gavves. 3d equivari- ant graph implicit functions. In European Conference on Computer Vision, pp. 485-502. Springer, 2022.
Pieter- Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134: 2005.
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. Vector neurons: A general framework for so (3)-equivrriant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12200-12209, 2021.
Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis. Learning so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 52-68, 2018.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se transsfooos::::::: 3d roto- translation equivariant attention networks. Advances in neural information processing systems, 33: 33:1970-1991, 2020.

---

Published as a conference paper at ICLR 2024
William Fulton and Joe Harris. Representation theory: first course, volume 129, Springer Science & Business Media, 2013.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Couraaeaau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre
Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Ar- ray programming with NumPy. Nature, 585(7825)335-3362, September 2020. doi: 10. 10.1038/ $41586-020-2649-2 URL https: ht/. 11tp://1111111111 s41586 0200000002222222 2649-2.
Sékou-Qumar Kaba, Arnab Kumar Mondal, Yan Zhang, Yoshua Bengio, and Siamak Ravanbakhsh. Equivariance with learned canonicalization functions. In International Conference on Machine Learning, pp. 15546-15566. PMLR, 2023.
Xianzhi Li, Ruihui Li, Guangyong Chen, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. A rotation-invariant framework for deep point cloud analysis. IEEE transactions on visualization and computer graphics, 28(1))4503-45514 2021.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolu- tion on x-transformed points. Advances in neural information processing systems, 31, 2018.
Min Liu, Fupin Yao, Chiho Choi, Ayan Sinha, and Karthik Ramani. Deep learning 3d shapes using alt-az anisotropic 2-sphere convolution. In International Conference on Learning Representa- tions, 2018.
Pavlo Melnyk, Andreas Robinson, Märten Wadenbäck, and Michael Felsberg. Tetrasphere: A neural descriptor for o (3)-invariant point cloud classification. arXiv preprint arXiv:2211 144456, 2022.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pp. 4460-4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
D. Morrison, P. Corke, and J. Leitner. Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation. IEEE Robolics and Automation Letters, 5(3):4368-4375, 2020.
Adrien Poulenard, Marie-Julie Rakotosaona, Yann Ponty, and Maks Ovsjanikov. Effective rotation- invariant point cnn with spherical harmonics kernels. In 2019 International Conference on 3D Vision (3DV), pp. 47-56. IEEE, 2019.
Omri Puny, Matan Atzmon, Edward Smith, Ishan Misra, Aditya Grover, Heli Ben- GamuH and Yaron Lipman. Frame averaging for invariant and equivariant network design. In International Conference on Learning Representations, 2021.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 652-660, 201 7a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++:: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017b.
Yongming Rao, Jiwen Lu, and Jie Zhou. Spherical fractal convolutional neural networks for point cloud recognition. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 452-460, 2019.
Jun John Sakurai and Eugene D Commins. Modern quantum mechanies, revised edition, 1995.
10

---

Published as a conference paper at ICLR 2024
Peter H Schönemann. generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1-10, 1966.
Stanford University. The stanford 3d scanning repository. https:///gaaahicssstaattooo edu/data/ Satattt/Dsaarrep///. 2023, Accessed: 2023-11-17.
John Stillwell. Naive lie theory. Springer Science & Business Media, 2008.
Zhuo Su, Max Welling, Matti Pietikäinen, and Li Liu. Svnet: Where so (3) equivariance meets binarization point cloud representation. In 2022 International Conference on 3D Vision (3DV), pp. 547-556. IEEE, 2022.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-7547, 2020.
Nathaniel Thomas, Tess E. Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick F. Riley. Tensor field networks: Rotation- and translation-equiiarrann neural networks for 3d point clouds. ArXiv, abs/1 802.08219, 2018. URL https: /api www.antetii.... cashantcccchllr...
org/ Corpus 3457605.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5): 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Informa- fion Processing Systems, 31, 2018.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Wentao Yuan, Benjamin Eckart, Kihwan Kim, Varun Jampani, Dieter Fox, and Jan Kautz. Deepgmr: Learning latent gaussian mixture models for registration. In European Conference on Computer Vision, pp. 733-750 Springer, 2020.
Zhiyuan Zhang, Binh-Son Hua, David W Rosen, and Sai-Kit Yeung. Rotation invariant convolutions for 3d point clouds deep learning. In 2019 International conference on 3d vision (3DV), pp. 204- 213. IEEE, 2019a.
Zhiyuan Zhang. Binh-Son Hua, and Sai-Kit Yeung. Shellnet: Efficient point cloud convolutional neural networks using concentric shells statistics. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1607-1616. 2019b.
Zhiyuan Zhang, Binh-Son Hua, Wei Chen, Yibin Tian, and Sii-Kit Yeung. Global context aware convolutions for 3d point cloud understanding. In 2020 International Conference on 3D Vision (3DV), pp. 210-219. IEEE, 2020.
Minghan Zhu, Maani Ghaffari, and Huci Peng. Correspondence-fuee point cloud registration with so (3)-quuivrriant implicit shape representations. In Conference Robot Learning, pp. 1412-1422. PMLR, 2022.
11

---

Published as a conference paper at ICLR 2024
A PROOF FOR THEOREM 1 IN THE MAIN PAPER
Theorem 1. If J, E R Vi E (1,2,3} satisfies -J; = J. [2] J3. [J2, Js] J1, Wa, J2 where [A AB BA, and exp(2mmJi) Inxns Vm E where is the space of integers, then D(R)= exp(O0 E (O(n) and satisfies the conditions Equation 2.
In this section, the proof of Theorem and Proposition from the main paper is described. Here, the terms including F [Fi, F2, Fs), J= [J1. Jsl. D(R) are all from Section 4.
VR1, R₂ E S0(3), if R1 # R2, then D(R1) * (R22) (Lemma 8)
VR E 80(3), D maps R to a single matrix D(R) E R"** (Lemma 1)
VR E 8O(3), D(R) D(R)" D(R) = 7 (Lemma
VR SO(3), det D R)) = (Lemma 10)
Additionally, prove the compatibility of D(R) which is necessary to prove the equivariance of (T))
D(R = D R2) for all R1, Rg E SO(3) (Lemma 4)
In the proposed solution of D(R), the axis-angle representation is used. However, given a rotation matrix R. there are infinite ways to make corresponding axis-angle representation. To be a high- dimensional rotation, there should be only one ( corresponding to R. So, Lemma shows the uniqueness of ((R) given single rotation matrix R.
Lemma 1. Suppose D satisfies the conditions in Equation 2, and JI. Ja satisfy the conditions Theorem I. VR E SO(3), D maps R a single matrix D(I)) E RRax.
Proof. Let's assume that two different anis-angle representations (w), 01), (42 02) of R mapped into two different (RR), which are exp(0)@) J) + exp(02@2 J).
R = cxp(f101 F) exp(02002 F) holds. From comparing each element of cxp(0141 F) = exp(0142 F), exp(0)41 .) exp(0202 .) should be satisfied. So a contradiction has been reached the assumption.
To show the compatibility of D, understanding the multiplication of two matrix exponentials is important because D is defined as the matrix exponential, Ceceer--Cmmbell-HHHne formula is expression of solution for Z the equation ex e = e² which is described in Lemma 2.
Lemma 2. (Baker-Canne HhaubeellHHuudorffff formula)
exp (X) exp (Y) exp (Z)
YX. The solution for Z the equation is series in X and Y and iterated commutator thereof. The first few terms of this series are
where X,Y E Raxx and the commutator [. is defined as
Z = 1X,
1 XX,IX 12
1 12
1 +
The important thing here is that the formula is only affected by the commutator, and not affected by the dimension the matrix. Lemma says that i set three matrices satisfy a cyclic commutator condition, then the solution for Z is linear combination of these three matrices and their coefficients
are independent of the dimension of the matrix. By using the fact that F [Fi, F2, F3] and J [J1, J2, Ja] both satisfies the cyclic commutator condition of Lemma 3.
Lemma 3. Given a set of matrices
(A1, A2, As) C RmXm (m N 3)
12

---

Published as a conference paper at ICLR 2024
that satisfies
[A,, A2] = A3, (1,, As] = A1, [A,, A1] = A2, where X XY - YX
ä A = (a + a2 + az a E R
exp(ă e)exp( A) exp(2) e R
Then
== = b) Á
where
9: g R R3
independent of the dimension of the matrices
Proof. Vū, 5 IRs, we can define a function f R XR + RR where s(ā, b) = |a2b3- - -abb,, azh1
ajbs, 01b2 a2b1l
a A, A) (a1b2 a2b1) (azb3 azb2)A1 (asbi a2b3)
= =
The important thing here that f is independent of the dimension of the matrices m.
let exp(ī- A) exp(( A) exp(z) s.t.
RR mm
1 a. Ā, A.b A +...
z= a. A A
a A,F A +
2
12
= (ā + 1+ 1
-+,
(a,
+
1o (ā, f(ā. b)) + +...). Ā
1
2
9ā b) À
Since every term of 2 consists of iterative commutators, the result of every iterative commutator will
be E. Á where č E IR3, Since Es only related to f.g: R3 x R R33 is independent of m.
Lemma 4. Suppose D satisfies the conditions in Equation 2, and J1, J2, Ja satisfy the conditions in Theorem I. D(Ri) = D(R, R2), for all Ra, R2 E S0(3)
Proof. For any rotation matrix R SO(3), there exist an axis-angle representation with axis and angle 0,
R exp(@@ F)
where {F1, F2, F&} is the basis for 3 x 3 skem-symeetric matrix defined in the prerequisites, and satisfies Lemma 3 with function g.
Vā, & E RR, exp(ā expp(T F) = explo(a, b). 17)
Also, proposed high-dimensional rotation D
D(R) exp(O@ J)
where satisfies Lemma 3 with a function g.
Vā, b R², exp(ā J) exp(b J) exp(g(ā, .J)
Since function from Lemma 3 independent of the dimension of matrices, the function used in R and D are identical.
Since R, R2 E SO(3), we can get axis-angle representations that are axes W, W2g W3 E R and angles 0,, O2 Og of each Rı, R2, and Rí Ra.
13

---

Published as a conference paper at ICLR 2024
R1 - exp(0101 F), R2 - exp(0202 F), R2 = exp(As00s F)
R2 @xp exxx(02222 F) exp(s(01 02w2) F) exp(O300s F)
From Lemma 1,
exp(g(0](1, 0202) J) = exp(0s03 J)
(())) = @ J) exp(0222 J)
= =expgg(0 0242) J) exp J) = D(R,
To prove the injectiveness of D, we will first show that J1, J2. J3 share n eigenvalues in Lemma 5. By using this, we will prove Lemma 8, which means that D is injective.
Lemma S. J1 J3 E RRaxn that satisfies commatator relationship [J] = Ju where j. k) are cyclic permutations of 21,,,3), they share the same eigenvalues including both the values and their multiplicities.
Proof. Without loss of generality, we will show there exists a similarity transform between J1 and Jz. With scalar 0 R
exp(0J3)JJ1 eapl1epp(-11 exp(-0Js) (1 [L+0J, 0J3 + 1 02 + J.I 0J3 +
] 0 ....)
2!
02
= J1 + 0JJa1 - Jz)z) + (Jzh
2JzJ1JJ + JIJz) +...
2!
J1 0[J,, J1]
Jss [J3,
Jil]
[J3,
[Ja, [Js, Jıl]] +...
2!
31
[J 0
0
12
+....
JI cos 0 J2 sin
So, if 0 = 7/2, exp(OJ3)JJ exp(-0J5) 0J3a) = J2 holds. Since exp(0J3) exp(-0.j) = I, There exists a similarity transform between J1 and J2. So, J1 and J2 share the eigenvalues and multiplicities. By doing the same way, it can be proved that J1, J2 and J3 share the same eigenvalues and multiplicities.
In Lemma 6, J+ and J_ is defined. They used in Proposition find the eigenvalues of J3, since if there exists an eigenvalue X of J3 s.t. Jzū = XT, then J3J_ = (A + i)J_ 7 and JzJ+ā (A +) hold. So applying J4 makes the eigenvalue one step bigger smaller, or the corresponding eigenvector is in the nullspace of J4.
Lemma 6. Suppose J1, J2, Jy satisfy the conditions in Theorem Define
J =+ J1 J1+
J_ J-= iJ2
Then J+, J_ share the same eigenvalues include multiplicity with J; and satisfies (J3, J1] -iJ
Proof. By proposition 2, eigenvalues of J1 and J2 the same, with eigenvalues. If X is eigenvalue J, and Jz and there will be corresponding eigenvectors in D2 E Rn for each J1, J2.
JJ λιo
J20 = A
JA = (11+ii2)) = (1))
J_ (J1 i12))1 (
14

---

Published as a conference paper at ICLR 2024
A1 is also eigenvalue of J_. So, J4, J_ share the same eigenvalues with J. The same procedure can be applied when A1 has a multiplicity bigger than 1.
Also, by applying commutator on J+.
[Ja, J+l = [J3, (J1 iJ2)] (77
J2 F FiJ+
Proposition 1. Suppose J1, J2, J3 satisfy the conditions in Theorem 1. Then, J1, J2, and J3 have the same eigenvalues and multiplicities. In particular, the eigenvalues are = {-ki, - non-negative integer k. Further, if X is eigenvalue of J; with
1)i..... 0, ki} for multiplicity m, then -X is also an eigenvalue of Jj with the same multiplicity m.
Proof. Sharing eigenvalues J1, J2, and J3 have same eigenvalues and multiplicities, proved by Lemma 5.
Integer eigenvalue coefficients. Since Jj E {J1, J3} is a real diagonalized into J, P(A)P-I where P E C" is matrix and A Cax" is diagonal matrix whose diagonal elements A are the eigenvalues of J,
matrix, can be
Then, from the condition of J: exp(2kTJ.) Laxm Vi (1,2,3)
Texp(2kxA1)
exp(2kenJ:) =
p-1 Inxn
exp(2kstAn)
Texp(2kxA1)
p-1 Ln-nn Laxn
exp(2kmAn)
So, exp(2kTA,) = should be satisfied for vk E, Vi E (1,2,..., nn.. Therefore,
= 3m E Z
Existence of null vector. From Lemma 5 and 6, J1, J2, Js. J4, shares the eigenvalues. Let's choose one eigenvalue \; and the corresponding eigenvector of Jg. Then,
Jz \42%
J3J_;; = (1s.==
= iJ v; + J- (Aro)) = (A; +i)J_0;
Here if J_ o Ō then >r + i is eigenvalue of Ja and J_v; is the corresponding eigenvector. Then, O. then A; 2 eigenvalue of J/.- This procedure can be repeated, but since J3 is a finite matrix, its eigenvalue is also finite. So we can always find J_ (_((m) v;) Ō where /(m) U; + Ō.
by doing the same way, if J_ (J_
Consecutive eigenvalues Then, let's define m as the dimension of the null space of Jg. Then we always find the real orthonormal basis U1, ...... Om of the null space of Ja since Js is real matrix. Let's choose vj I" and it satisfies Jæu, 0. Then if J_ vy o then is eigenvalue of Ja and J_ is the corresponding eigenvector.
Jz(J_ = iJ3
Apply conjugate: (J( = -iJ
So -i is also an eigenvalue of J3 and J4 is the corresponding eigenvector. By applying this procedure repeatedly, we can find the first n. 2 0 E ž that satisties J_ (1g+1) = Õ, and we can find the continuous interval of eigenvalues
Aj (-nzi, -i,0, .nyi)
15

---

Published as a conference paper at ICLR 2024
with corresponding eigenvectors
(J Ug. J4 Uj Ugy _
Completeness For each Aj. we can find J_ (nyt)) J- -- (n)())))))))) where J_ (n)) Ô. So J_ (m)) by is a nonzero null vector J_ while also an eigenvector of J3 corresponding to the eigenvalue nzl.
there is another nonzero eigenvalue x2 outside of Ao A2 and corresponding eigenvector Uz, then Ur is independent with v1, "2,""" ...
1m- We can always find J_ (mx) Un. which nonzero null vector of J/_ while also an eigenvector of J3 corresponding to the eigenvalue A + ng1. So we found a nonzero null vector of J_ that is independent with existing k null vectors of J-, and contradiction since the
makes the dimension of null space of J- is bigger than k. However dimension of null space of J3 is k and J_ shares the eigenvalues with J3.
Lemma 7. Suppose D satisfies the conditions in Equation 2, and J1, J2, J3 satisfy the conditions in Theorem I.
- for any R E S0(3)
Proof. From Lemma
D(R)D(R-)) D(RR-1) = Inxn: D(R D(R) D(R R) Inxn
D =
Lemma 8. Suppose D satisfies the conditions in Equation 2, and J1, J2, J2 J3 satisfy the conditions in Theorem 1. Then, vR,, Rr E 8O(3), if Ri + R2, then D(R,) + D(R2)
Proof. Let's prove a rotation matrix which is mapped to Lnxn is only 1ax3
Let's assume that there exists another rotation matrix & lax3 which mapped to Iaxn. Then we can find an angle representation of R which the axis w = [wi, W2, wy] and the angle # Ô. Let's define J = @ J. Then (RR) exp(eJ) = Inxn.
From Proposition eigenvalues of is A {-ki,.... 0, kij, E Z including multi- plicities. Then eigenvalues of exp(6.J) is (c where A, A. The eigenvalue of Inxa is only 1, so every @b should be Then 6 should be 2km, but makes 13x3, so is contradiction.
If there is two distinct R, R2 mapped same RRxn, then R, (RR,)
D(R, (RR)) Inxo from Lemma 7. But Rí R2 is not I, there is contradiction. Therefore D is injective.
Lemma 9. Suppose D satisfies the conditions in Equation 2, and Jr, J2, J3 satisfy the conditions in Theorem 1. Then, VR
D(R)T D(R) =
Proof.
exp(ō F) = exp(ū) = Tax3 E 3
So is obviously one possible anis-angle representation of Taxa, and D(Iaxa) = cxp(ū-J) Inxo- From Lemma 1, D(Iax3) = Inxn unique.
Lemma 10. Suppose satisfies the conditions in Equation 2, and Jn Jz, J3 satisfy the conditions in Theorem 1. VR de( =1
16

---

Published as a conference paper at ICLR 2024
Proof.
1 det((((RR)))
- =ed (
Therefore, del(D(R)) = 1 VR E 50(3).
Let's assume (e)()(R)) for all R E (0 Then, for R1, R2 SO(3),
1 ded((((R)))dd
det R2))
which is contradictory. Therefore det D 1 for all 50(3).
B PROOF OF THEOREM 2 IN THE MAIN PAPER
Theorem 2. Suppose D satisfies the conditions in Equation 2, and Jı, J2, Ja satisfy the conditions Theurem I. VR SO(3) whose angle of rotation 0 and rotation axis w. we have
D(R) exp(0@ J) Σ
E,6% (sin(X0) iicos()0)
AEA
where bx is the eigenvector of 0l .J that corresponds to eigenvalue X.
Proof. Since Jj and J are real skew-symmetric matrices, they are diagonalizable. Further, by proposition 2, J has the same eigenvalues as J, This allows us to write exp(0@ exp(P(A)P-1) = P Pexp(A) (xsp())-1 where P E Cnxx is matrix and A Caxn is diagonal matrix of the cigenvalues of J multiplied by 0. So, D(R) can be expressed with
Tep(( ki0)
coo( k0) tisinn
D(R) = P
p P
p
exp()10)
cos(k0)+isiu((e
From this, can write
D(R) = cxp(fi -) PAP-1- -PPP-P1 Σ b, (sin(k;0) + cos(k;;))
where 5y the th eigenvector of ④
C PROOF OF THEOREM 3 IN THE MAIN PAPER
Theorem 3. Consider the mapping (ū) (ū)) i rotational equivariant if is the eigenvector corresponding to the zero eigenvalue of Js. and D satisfies all the conditions in equation?
Proof. We recall the definition of vb
(Φt) = x D(R
Based on this definition, the theorem would be true if
(RRT)= D(E (RE))
=<(1==]) (R#))e
equivalent
D(R) D(R)
17

---

Published as a conference paper at ICLR 2024
By definition, R (k) maps [0,0,1] to lo for some unit vector k. So, we have
R R R$
and
R (Rt) 8 Rd
Because RR* = Rx (R(E)) 22
[R) [ -2 =
(R) (
=
So (2) R [ must be a rotation about the z-axis. This means that there exists some [R (Ra) R2(7), and R2(1) means rotating angle about the z-axis.
y R where [R (+:)
So [R (Ri)] = [RR (+) R2 (7) holds. Considering (RE).
((iT") ((I (R&)))
* = D(RR) R+))
since compatibility of D(R) holds from Lemma
From the definition of D. ((R² exp(7J3) holds. Also, considering o is the eigenvector of Ja corresponding the zero eigenvalue, Jzu 0. From the definition of the matrix exponential exp(>Ja) = I + (7Jz) + v ((E)) ((() (R(IE)) ((R)v(T))
e...(())))))) = satisfies. So
D PROOFS OF REQUIRED PROPOSITIONS
Proposition Suppose Jis J2, J3 satisfy the conditions in Theorem 1. Then, J1, J2, and Ja have the same eigenvalues and multiplicities. In particular, the eigenvalues are A = {-ki, - (k 1)i..... 0, i..,,... ki} for some non-negative integer Further, if an eigenvalue of Jg with multiplicity m. then > is also an eigenvalue of J; with the same multiplicity m.
Proof. Sharing eigenvalues J1, J2, and Jg have same eigenvalues and multiplicities, proved by Lemma 5.
Integer eigenvalue coefficients, Since J₁ E {J1, J3} is real kymmetymettric matrix, it can be diagonalized into J P-A)P-I where P E C matrix and A C is diagonal matrix whose diagonal elements A; are the eigenvalues of J.
Then, from the condition of Ję exp(2kxJ.) Inxi; Vi (1,2,3)
rexp(2kxA1)
0
exp(2kenJ.)) =
P-1 1 Inxa
22p
Texp(2kxx1)
p-1 --'Inxx P Inxn
exp(2km)n)
So, exp(2kTX:) = should be satisfied for Vk Z,Vi E (1,2.....
..... nn. Therefore,
X mi, 3m Z
18

---

Published as a conference paper at ICLR 2024
Existence of null vector. From Lemma 5 and 6, J2, Ja. J-.J- shares the eigenvalues. Let's choose one eigenvalue A and the corresponding eigenvector v of J3. Then,
Jz X,4%
13J_ ([J3, [-] [1 iJ + J- (A,0;) (A, +2)J_
Here if J-u # then A, + is cigenvalue of J3 and J_v, is the corresponding cigenvector. Then, by doing the same way. if (J-- * Ó, then A; + 2i is eigenvalue of J3- This procedure can be repeated, but since J3 is finite matrix, its eigenvalue is also finite. So we can always find J- J_ (J( v.) = 0 where (m)) U/ 0.
Consecutive eigenvalues Then, let's define the dimension of the null space of J3. Then can always find the real orthonormal basis U1, U2,-- Cm of the null space of J3 since J3. is a real matrix. Let's choose by I= and it satisfies Jæú = 0. Then if J- * then is eigenvalue of J3 and Uy the corresponding eigenvector.
Ja 0y) iJ3
Apply conjugate: Ja(J+ = -iJ3
So -i is also an eigenvalue of J3 and Jz is the corresponding eigenvector. By applying this procedure repeatedly, we can find the first nj 2 0 E that satisfies J_ (15t1) Õ. and we can find the continuous interval of eigenvalues
Aラ = { -n,,,
,nj}
with corresponding eigenvectors
U Uソ. J+ Ug J-yyJ-11p1 Ug: {
Completeness For each Aj. we can find J_ (n, (1,+1) vj = J- (J_ (n)22)))) = where J. (3),, * Ö. So J_ (12) by is nonzero null vector of J_ while also an eigenvector of Js corresponding to the eigenvalue nyd.
there another nonzero eigenvalue Xe outside of A2 and corresponding eigenvector U/ then ug is independent with U1 "2. Pm- We can always find J_ (m²) Ugg which nonzero null vector of J_ while also eigenvector of J3 corresponding to the eigenvalue Au: n,, So we found nonzero null vector of J_ that independent of existing è null vectors of J_, and makes the dimension of null space of J_ is bigger than k. However, it is contradiction since the dimension of null space of J3 is and J_ shares the eigenvalues with J3.
Proposition 2. Suppose J1, J3 satisfy the conditions in Theorem 1. Then, Ows I have the same eigenvalues as Jr, = {-k0i -(k
-ii 0, fi....... .kf.) .kii}
Proof. Given rotation R E S0(3), we can always find an euler angle representation a, 8,B,y where R = Rr (a) (B)) Ra means the rotation along k-axis, and a, B, n E IR.
Let's think about R2(1) first. From 5.
exp(7da)Jiexp(- Jj cos = ( J
exp(7J3)J2 (x = J2 cos - J, sin (R.
exp():33)J3 exp( -J3)) = //3 J3 (R=)
By using these terms,
exp() J) exp(- = (z)
Here exp(7J3) exp((0,0,)) J) D ())2())) means the rotation of 7 angle z-axis.
19

---

Published as a conference paper at ICLR 2024
By doing the same thing the x-axis and y-axis, we get below.
exp(aJi) exp(312) exp(r expp(-7))) exppp(--)))) exp(-BJ2) exp(-aJ1)
= (3)
Since compatibility holds in D(R), ((R)
((R((())R(())(((()
exp(aJ1) exp(3J2) (xp(3)))) exp(rJa) So, (Re)) J J= D(R) JD(R)"
Then, we can always find rotation R from 0.00, 1] to w and satisfies à J= D(R) So a J shares eigenvalues with Jz. From Proposition 1, the eigenvalues of Ja {-ki, -k -i,0,,,,,,,,,....... kij. Multiplying 0 to matrix also makes the eigenvalues to be multiplied by 0, so the eigenvalues of 0ā are ( 1-k0i,-(k 1)0i, ... 01, 0, ... k0iJ.
Proposition 3. The maximum value of k in Proposition 2 is n-1
Proof. From Proposition J3 E R=xa always have the eigenvalues of 1-ki,-(4 I)i..... -,, i...,, ki} For to the maximum, the multiplicity should except zero eigen- value, and the multiplicity of zero eigenvalue is I when n is even, and 2 when is odd. So, the maximum value is 1=1
Proposition 4. Ifa skew- matrix A E Rax= s.t. A = = has the eigenvalues X m,i where: 1,2,...,7 and m, &, then exp(2k7 = Inxn where k E Z.
Proof. Since A is a real skew-symmetric matrix, it can be diagonalized into A P(A)P-1 where P E C is matrix and A Caxn is a diagonal matrix whose diagonal elements X m,j are the eigenvalues of A.
exp((2m kmi)
exp(2kTAA)
p- Plaxn P-1
Laxn
exp(2mmkkl))
E ARCHITECTURE DETAILS
E LAYERS
In this section, we clarify the difference between the Vector Neurons and introduce new layers. We built our architecture on Vector Neuron's implementation. For all kinds of layers, the main difference is in the dimension of the vector-list feature V. In Vector Neurons, V E RCxx In our case, we employ augmented V (C(++)) where C' is the channel count and 3 + n is the dimension of an (O(3) -quuivrquiaariant layers of the Vector Neurons. A linear layer, for example, is redefined only by changing the vector dimension 3 to 3 + n:
vector with an added feature size of n. This easily adapts to the existing
V' Jun (V: W) WV (3x((3+
It is worth noting that number of learnable parameter is the same with original VN (i.e., number of elements in W) while the proposed one achieve significantly better performance than that. We adopt the input edge convolution layer, linear layer, pooling layer, and invariant layer from Vector Neurons by simply changing the vector dimension. To this end, we also further introduce layers we devised for method.
20

---

Published as a conference paper at ICLR 2024
E.1.1 FEATURE AUGMENTATION
For any imput layer that subject the SO(3) transformation, we begin with the feature augmenta- tion layer. Here, we define the concatenation of the original vector ī E R2 and its high-dimensional representation (0)) R". Given multi- multi-frequency feature dimensions such as 3+5 or 3+5+7, we iterate over each dimension and expand the feature by concatenating features with different frequen- cies:
(11=3,5,7... (/
where d/s (7) is our feature representation given dimension of the feature i, which is constructed from I by Algorithm 1. Within definition of in equation 1, we have (<(]iz]] for the scale, and use simple MLP with two fully connected layers of dimension 16 and ReLU. The input and output of this MLP is I to adjust the scale factor for each point.
1.1. NON-LINEAR LAYERS
We introduce a new non-linear layer based on the magnitude scaling of a vector, which is SO(3) equivariant. We first take the Euclidean norm of each vector of the vector list feature V which gives us vector qE _C, The i-th element qi is given by:
qi = V1, E R, i i=1....,
The vector q is processed through compact neural network featuring two MLP layers with bias, separated by 2D ReLU activation layer, denoted as fo Eº + RC, + q². Subsequently, non-linear layer is formulated as the element-wise scaling of each vector V, by its corresponding b, expressed as:
Vis5 = df Vis for i= I...., C' and j == I,..., (3 + n)
E.1 NORMALIZATION LAYERS
We found that the normalization layers introduced in Deng et al. (2021) do not help in improving the performance. We thus do not incorporate those layers in our architectures.
E.2
ARCHITECTURAL DETAIL FOR EACHI TASK
E.2.1 POINT CLOUD CLASSIHICATION AND SEGMENTATION
We utilize PointNet (Qi al. (2017a)) and DGCNN (Wang al. (2019)) as the underlying archi- tectures for both classification and segmentation tasks. Our approach seamlessly integrates with their existing Vector Neurons implementations. By incorporating an initial feature augmentation layer 1 (z²) E (3+m) and substituting the activation function as indicated in E.I, further mod- ifications to other layers are required. We utilize n 5 for all models. We reduced our number of channels C/8 where C is the number of channels the baseline models (Qi et al. (2017a), Wang et al. (2019)). This is more challenging than C/3 by Deng et al. (2021). We adopt all other hyperparameters from Deng et al. (2021).
E.2.2 NEURAL IMPLICIT RECONSTRUCTION
In the Occupancy Network (Mescheder et al. (2019)), the encoder produces a latent code RC from the input point cloud P E N3 and the decoder estimates the likelihood
query point I E 3 being occupied, conditioned on this latent code z. Following the approach used in Deng al. (2021), we substitute the standard PointNet encoder with its FER-VNS WER-VN-PPintNet variant. This modi- fied encoder outputs a vector- list feature Z E RC¹ x(++m), Our decoder function is defined in oth- ers of three invariant compositions of high-dimen nigh-dimensional features V (æ) E (3++)), Z E RC' x(3+n); O (z), Z), (T) FER- In Z)) 10
where FER-VN-In(2 Z) is our FER-VN- variant of the invariant layer (section E. which maps SO(3)
equivariant feature Z E (3-nn) to SO(3) invariant feature _ E C², We use C' = 513 for the Occupancy Network and C" = 171 for both VN-OceNet and FER-VE SERR-VN-cccNer following Deng et al. (2021) resulting in the same number of learnable parameters. We train the network for 300k iterations with learning rate of 0.0001 and batch size of 64, selecting the models based on the best validation mloU scores following Mescheder al. (2019). We utilize 5 for all models.
21

---

Published as a conference paper at ICLR 2024
E.2.3 NORMAL ESTIMATION
For this task, we leverage the VN-PointNeeDDCNN framework with an output head specifically tailored for point cloud normal estimation, as presented in Puny et al. (2021). Given that point cloud normal estimation an -Qu)eequivariaa task, we preserve the -euiv-equiv of fea- tures up to the final output layers. Specifically, we replace the invariant layer, max-pooling layer, and succeeding standard MI.P layers from the architecture detailed in E.2.1. We instead incorporate four intermediate SO(3) +Qu3i-qquivarian layers with activations. The concluding layer outputs feature vector of size 1, represented as n E xxx1xx where denotes the input point cloud's number of points and 3 signifies each point's normal vector. However, due to our method's nigher-dimensione. features, the output dimension becomes
R (1*xx)x)++) To align the output dimensions with those of the label, we employ a high-frequency representation of the ground-trus normal vectors V/ ngt E R.Nx3 denoted (ngt) R (3+n) during the training. The loss function minimized is:
n4 + min (1 f..l2 |, + Ru11²)
r2a
where N is the total number of points in the point cloud, and 7ū, and fú, are the predicted and ground- truth figh-freeuency representation of normal vectors respectively for the i-th point in the input point cloud. The first term estimates the accuracy of the predicted normal direction, and the second term regularizes the length of the vector to 1. We adopt the default hyperparameters provided Puny et al. (2021) including the number of channels. We utilize 5 for all models.
E.2.4 POINT CLOUD REGISTRATION
Point cloud registration aims to align two sets of N points P1.P2 E Nx33 that come from the same shape. This is formally addressed by the Orthogonal Procrustes problem Schönemann (1966), which finds the analytical solution However, it requires the knowledge of the correspondences between the points in P1, P2 which is usually hard to achieve. Zhu et al. (2022) proposes different approach, which solves for the latent codes Z1, Zo E CRC3 of Vector Neurons instead of the explicit point clouds when the correspon- dence between points not known. Due to the initial layer's edge convolution in Vector Neurons, the channel dimension of Z achieves permutation invariance, which facilitates correspondence free feature registration. Following this approach, we utilize our encoder from the shape complction task (section 4.1) to find R in the feature space. To acquire R from the rotation in high-dimension D(R) exp(# (J) where G R3 solve the following optimization problem:
rotation matrix SO(3) that minimizes | R P2l
W arg min
Σ
1Z.s(exp(a JAY Z2;el1
(3,5,77,...
where Zid and Ze. are features corresponding the augmented dimension i E (,,5,7,... 7,... of high-dimensional features Z1, Zg E (3++)) We use the Cross-Entropy Method (CEM) (De Boer et al. (2005)) as solver.
F ALGORIHHMIC DETAILS TO GET J1, J2, J3
This section is for explaining details in Algorithm and 2 to obtain J1, J2, Js.
22

---

Published as a conference paper at ICLR 2024
Constraints, In theorem 1, have eight constraints to get J1, J2, J3.
[11, J3
(5)
[J2, J3l = J1
(6)
[1] J2
J1 + +J-
J2 JI =
J/ + J =
(10)
exp(2rkJ1) exp(2xkJ2) =
(11)
(12)
exp(2nkJz) =
(13)
Eigenvalue of J3 = f-1
11
7 1
[1}
(14)
2
2
These constraints redundant, first analyze the dependency between conditions.
equation can be derived from equation 10, equation 8, and equation 7.
equation 13 can be derived from equation 14 and other constraints with Lemma 5.
equation 11 and equation 12 can be derived from Proposition 4.
Our strategy get feasible J1, J2, J3 consists of three steps.
1. Sample J3 satisfying equation 10 and equation 14.
2. Construct linear bases of J/1 and J/ with equation 6, equation 7, and equation
3. Perform non-linear optimization with equation 5.
Here is more detailed description of the procedures described in the paper. For step 2, we pick three equations (equation equation 7, and equation 8) that can be converted into linear relations with elements of J1 and Jz. This relation can be expressed as:
Iaxn X J3 JJ & Inxn
- In2xn
In2x
-J' & Laxa + Inxn 8 Jz
VI)( VFC(J2) =
(15)
((vEC(J)) (FF
8vEC(J1)
where A x23n @ denotes Kronecker product and VEC(.) is vectorization of matrix. We have 3n2 equations and 2n variables, so this can have no solution. But, we found out that there is redundancy within equations, where some equations are expressed by a linear combination of others. For example, J1 Jf
have n² equations, but it contains both J1 (i, + (1) = 0 and ii Ji(,j) 0 where (i. means 7th row and jth column elements of J1. Additionally, we assume that there exists solution satisfying all conditions Thcorem 1, so there should be non-empty null space of A. Practically,
EIG function in the Numpy library (Harris et al. (2020)) and always check the nullity of A within Algorithm
G ADDITIONAL EXPERIMENT
In this appendix, we delve into a series of experiments designed to validate the efficacy our feature representation method across multiple contexts. By integrating our representation, we demonstrate notable enhancements capturing intricate shape details with minimal impact on inference effi- ciency, particularly in shape compression tasks (G.2) and spherical shape regression (G.6). Through- out dimensional analysis to investigate how each dimensional feature contributes to the shape com- pression tG33, get clearer picture of what's going Finally, experiments reveal that FER leads to heightened model robustness in tasks such as point cloud completion (G.1) and registration (G.5).
23

---

Published as a conference paper at ICLR 2024
FERNN-GGcheel
80
80
40
20-
200
150
300
350
400
45.0
500
550
600
Figure 6: Volumetric mloU on ShapeNet reconstruction with the different number of sampled input points. Every model trained with the ShapeNet dataset where each data consists of 300 points.
G.1 POINT CLOUD COMPLETION WITH DIFFERENT NUMBER OF SAMPLED POINTS
We experiment to assess the impact of the number of points. We train models with a dataset where each data consists of 300 points. We then evaluate the models using 200 to 600 input points during test time. As illustrrated in the following figure 6, our method is more robust to changes in the number of points compared to both the Vector Neuron (VN)-based and vanilla Occupancy networks.
The reason why performance goes down as the number of points increases over 350 input points that these methods extract point-wise features from point cloud. VN uses K-Nearest- Neigh for each point-wise feature initialization, which processes the relative coordinates of adjacent points. Relative coordinates are sensitive changes in point density. Such changes alter the scale of relative coordinates compared to what was observed during training. This change will decrease the performance not only when the number of points gets smaller but also larger.
G.2 SHAPE COMPRESSION WITH DIFFERENT DIMENSIONALITY
We conduct additional experiments the shape compression task to address the impact of the di- mensionality of our feature representation the computational cost and performance. Below the results for processing 300 points for a single object for encoding and 100,000 query points for decoding. The results are averaged over 300 predictions.
Model type
IoU (%)
inference time encoder
inference time decoder
VN-OccNet VER-VVN-Occeet (n=8)
73.4
3.44 0.05
9.57 # 0.10 ms
81.0
3.47 # 0.06 ms
9.64 + 0.16 ms
FER-VNNOccNeet (n=15)
81.9
6.55 0.26
9.87 + 0.21 ms
Table 6: Shape completion performance and inference time of encoder and decoder for VN-OccNet and FER- VN-OccNet with different dimensionality of feature representation.
As the table 6 shows, incorporating our 8-dimensional feature representation CF-R-VN-OcceNet (n=8)) boosts performance (a 7.6% increase in IoU) with negligible impact encoder inference time compared to the VN-OccNet baseline. However, expanding the feature representation to 15 (n=15)) doubles the computational time of the encoder with only a marginal performance improvement. On the other hand, the decoder inference time remained relatively stable across models, underscoring the efficiency integrating FER into the network.
dimensions
G.3 SHAPE COMPRESSION OF EACH DIMENSIONAL FEATURE
We have an additional experiment on shape completion that compares which features are responsible for completing which part of the shape. Our composite feature is a concatenation of n 3, 5, 7, di- mensional features (See Appendix 1. for how we make the concatenation). The composite feature then processed through three-layer MI.P to predict occupancy values in occupancy network.
24

---

Published as a conference paper at ICLR 2024
Figure 7:
Shape compression result of our model by using the data from Stanford University (2023). means the dimensionality of features, and they are performed individually. The greater the mag- nitude of the gradient, the more yellow it is.
To visualize the contribution of each feature dimension the completed shape, compute the gra- dient magnitude of the occupancy predictions with respect to our feature representation of the surface points on the reconstructed mesh for each n. More concretely, we have (3(2)), (2), t/ (x), (1) where r E R is a surface point and v/n (x) (IR - R²) is our feature presentation with number of dimensions. These four features are concatenated and processed by MLP predict occupancy o= [00,, Φ5, 27, Φ9) where f is MLP. Then, gradient used for visualization is 20/00, Oo 00/01 00/00n. 0o/Ov/.
The visualization in figure 7 indicates that the lower-dimensional features, particularly for 3 and n 5, predominantly capture the more expansive and volumetric components of the meshes, such the torso and back of these objects. In contrast, the higher-dimennsonaal features n = 9 tend to focus on finer and more intricate details, such as facial features. These results offer valuable insights into the network's ability to differentiate between broad structural elements and detailed features of the meshes, reinforeing the effectiveness frequency- based feature mappings in SO(3) equivariant learning context.
7 and
G.4 SHAPE COMPRESSION WITH IDENTICAL COMPRESSION RATIO
We conduct an additional experiment where we match the embedding size of all the methods, thus having the same compression ratio, and try to evaluate their reconstruction (i.e. decompression) performance. In this table 7, we observe that our method achieves the highest IoU while having the same compression ratio.
Method OccNet
Input size
Embedding size
Compression ratio
IoU (%)
300 x 900
513
57%
67.5
VN-OecNet
300 900
171 x 3= 513
513
57%
73.4
FER-OccNet (n=3+5)
300 3 900
64 512
57%
77.3
Table 7: Shape completion performance measured by IoU for each model with the same compression ratio.
G.S POINT REGISTRATION WITH DIFFERENT INITIAL PERTURBATIONS
In the point registration application, the conventional Iterative Closest Points (ICP) method is known to sensitive to initial guesses. That is, when the initial guess is far from the ground truth, ICP often fails to predict good results. On the other hand, the equivariance-basee point registration method Zhu et al. (2022) is demonstrated for its robustness over initial guess. We hypothesize that the proposed FER-VN- DE--ved method also adopts the same characteristic,
conduct the same
25

---

Published as a conference paper at ICLR 2024
experiment in Zhu al. (2022) demonstrate this. To do so, in this experiment, compare the performance of point registration for three different methods (FER-VN, VN, ICP) by changing the initial perturbation range.
Max angle
30
60
90
120
150
180
FER-VN (ours)
0.00348
0.00350
0.00349
0.00349
0.00350
0.00350
0.00349
VN
0.00649
0.00649
0.00649
0.00649
0.00649
0.00649
0.00649
ICP
0.00310
0.00312
0.00340
0.00508
0.00773
0.00966
0.01123
GT
0.00310
Table Point registration performance measured by Chamfer Distance for each model by changing the range of initial perturbation. Max angle denotes the initial perturbation is sampled from uni- form distribution with bound [0, Max angle] in degree.
As the table 8 demonstrates, our method is also robust to the initial perturbation, while ICP fails when we apply large perturbation. Notably, the performance with ours is significantly better than the one with VN and it almost reaches the performance of ground truth. Note that add Gaussian noise to the input point clouds, so the ground truth CD not zero.
G.6 SPHERICAL SHAPE REGRESSION
To test the effectiveness of controlling the frequency of the feature representation, we employ simple spherical shape regression experiment. We first pick shape from EGAD (Morrison et al. (2020)). Then, we represent the shape's surface by changing the distance of points on sphere. Basically, we train model to predict how far a point from the center when given direction 3D space. To train our model, chose random directions and used ray casting to get the right distances for those directions.
feature dimension
(GT)
3
3+3
3+5 (low frequency)
3+5 (max frequency)
3+5+7 (max frequency)
Figure 8: Toy experiment to show the effectiveness of controlling the frequency of the augmented features. One shape from EGAD (Morrison et al. (2020)) is regressed to networks based on the controlled features with different frequencies.
First, we train the model without feature augmentation. As the reconstructed shape labeled 3 in figure 8. the detained features are smooth out. When we increase the dimension of the representation by repeating the point coordinate twice to make the dimension of the representation 6 (3+3), still, the reconstruction quality bad. This implies that just making the feature high-dimensional is useful, and we need additional factors to increase expressiveness.
We observe how the frequency in the augmented feature is depicted by the eigenvalues of J, As- suming that utilizing only three dimensions does not adequately capture the shape details, we opt for the addition of 5-dimensional features. From Proposition 3, the eigenvalues including multi- plicity will be either A {-i, O, 0, i) or A2 (-2i, 2i, -i, i, 21}. Based on Theorem 2, higher magnitude eigenvalues render & more sensitive to the input feature, thereby capturing more details. Fig. 8 demonstrates the impact of the frequency of the augmented feature the shape details. A low frequency, where the 5-dimensional augmented feature used J3 with eigenvalues of A1, shows negli- gible improvement. However, employing the maximum frequency, with the augmented feature using J3 having eigenvalues of A2, results a significant enbancement compared to the low- www-freuunccy augmented feature.
26

---

Published as a conference paper at ICLR 2024
G.7 APPLICATION OF FER ON VECTOR NEURON-BASER METHODS
Our feature representation, FER, can be integrated into any of VN-based methods. We applied our method to of the recent VN-based methods, GraphONet Chen et al. (2022). We select meshes from Stanford University (2023), and generate a dataset with 300 surface points and 1024 query points to evaluate occupancy. All meshes are normalized in scale, and we use query points generated by adding small Gaussian noise to the surface points. We train GraphONet baseline and FER- GraphONet, which variant of GraphONet by integrating our proposed feature representation (FER). We apply feature dimension of 8 (3+5). To evaluate representation power, we adopt the test setting the occupancy network, where reconstruction quality evaluated on the meshes used in training. For the evaluation metric, we use volumetric intersection over union (IoU).
Method GraphONet 56.0
IoU (%)
FER-Graph GeRRGrraph
57.6
Table 9:
Shape reconstruction result measured by volumetric intersection over union (IoU). Our method is applied to GraphONet, and shows improved performance.
As shown in table 9, we observe enhanced performance, reinforcing the effectiveness of our ap- proach. The result for GraphONet is 56.0% and 57.6% for FER-GraphONet, which shows 1.6% improvement. The results demonstrate that our proposed feature representation further improves performance when used with tate-of-thhe-rr methods.
27


Reviews:
[{"review_id": "r5J9XX9CHe", "reviewer": "Reviewer_Jvrz", "summary": "In this paper the authors propose a representation for point cloud data that can be used with the existing method [1] to improve expressivity while maintinaing equivariance to 3D rotations. The representation is constructed by transforming points to an axis-angle representation, then lifting that representation to a random subspace of SO(n). The subspace is defined by three matrices J_1, J_2, and J_3, where J_1 is a random vector in the Lie algebra of SO(n), and J_2 and J_3 (also in the Lie algebra) are matrices whose Lie bracket is close to J_1. The final SO(n) representation of a given point is the exponential map of the linear combination of J_1, J_2, and J_3, where the combination coefficients are determined by the axis-angle representation.\n\n[1] Deng, Congyue, et al. \"Vector neurons: A general framework for so (3)-equivariant networks.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "questions": "*Questions:*\n- What is meant by “$R^z(\\hat{u})$ is a rotation matrix defining the orientation measured from the $z$-axis to $\\hat{u}$.” Is it the 2D rotation about the axis $z \\times \\hat{u}$ that aligns the $z$-axis and $\\hat{u}$?\n- It seems like the map is from SO(3) to a subspace of SO(n) is that right\n- Is the axis arbitrarily selected?\n- Createsearchspace from Algo 2 is not described in the text, how does this work?\n\n*Possible typos:*\n- “Theorem 1. theoremIf” → “Theorem 1. If” \n- Are $\\hat{u}$ and $\\overrightarrow{u}$ used interchangeably in section 3? It would be clearer if the notation were consistent\n- “intuitively, just like F1, F2 and F3 represent angles” → “intuitively, just like F1, F2 and F3 represent axes of rotation”\n- Should “axes ψ(ˆx), ψ(ˆy), and ψ(ˆz)” be “axes ψ(F_1), ψ(F_2), and ψ(F_3)”? If not what does $\\hat{.}$ mean here?\n- “ effective-yet-rotation-equivariant” →  expressive yet rotation equivariant\n- In section D, readers are referred to the proof of Thm 1 for the proof of Prop 1, but the proof of Thm 1 does not prove Prop 1.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "*Originality:* The proposed approach appears novel. Although other approaches have used a Fourier like representation to represent 3D objects [1,2,3], this approach differs from those in that 1) the representation is intentionally high-dimensional, 2) only a subspace of the high-dimensional space is considered.\n\n*Quality:* In the experiments, proposed method performs on-par with or better than most baselines.\n\n*Clarity:* (see weakness)\n\n*Significance:* Designing expressive equivariant representations is a challenging and important problem in the field.\n\n\n[1] Esteves, Carlos, et al. \"Learning so (3) equivariant representations with spherical cnns.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n\n[2] Cohen, Taco S., et al. \"Spherical cnns.\" arXiv preprint arXiv:1801.10130 (2018).\n\n[3] Kondor, Risi, Zhen Lin, and Shubhendu Trivedi. \"Clebsch–gordan nets: a fully fourier space spherical convolutional neural network.\" Advances in Neural Information Processing Systems 31 (2018).", "weaknesses": "*Quality:* \n* The authors present a very critical view of methods that use spherical harmonics which I think is not only inappropriate but also incorrect. Spherical harmonics can be seen analogous to the Fourier basis which have been applied in signal processing problems for centuries, and the authors rely on for intuition building. Also, the authors state that spherical harmonics come from quantum mechanics, but the use of spherical harmonics predates the \"discovery\" of quantum mechanics by more than a century. \n* There does not appear to be any discussion of the impact of choosing a higher dimensional input representation space. How does the choice of $n$ impact performance? What is the additional computational cost?\n* The notation is inconsistent (see questions)\n* Some proofs are missing (see questions)\n\n*Clarity:* I found the presentation to be difficult to follow. The paper uses advanced math concepts such as the Lie bracket without motivating or even naming them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MflIochfta", "reviewer": "Reviewer_MkmG", "summary": "The paper introduces a frequency-based input representation for Vector-Neuron for SO-3 equivariant network. It provides compelling evidence to support the claim of keeping equivariance with this input representation. Both visualized and numerical results demonstrate the effectiveness of the proposed method, thus validating its ability to achieve equivariant representations.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents sufficient mathematical proofs, which provide robust support for the proposed representation. This strong evidence reinforces the credibility of the approach.\n- The multi-frequency design demonstrates its effectiveness in terms of both visualization and benchmark performance. \n- The writing style and figures in the paper are great as they effectively elucidate the design and motivation behind the proposed representation. \n- The experiments conducted in the study convincingly demonstrate the effectiveness of the proposed method across various downstream tasks.", "weaknesses": "- Lack of ablation studies\n- (This is not a weakness) Does author try to extend the method on scene (indoor) ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RZoeuWpnDh", "reviewer": "Reviewer_79hQ", "summary": "The paper targets the challenging 3D representation problem and a new equivariant feature representation which can map 3D points to high-dimensional feature space is presented. The experimental results have been validated on several tasks like point-cloud completion, shape compression, normal estimation, point cloud registration, point cloud classification and segmentation. Reasonable results have been reported with a comparision with the existing work.", "questions": "Please address the questions in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A new 3D representation is presented which can discern multiple frequencies in the 3D data. Also, it provides theretical justification of the proposed approach. \n2. The proposed approach is valiated on several tasks like point-cloud completion, shape compression, normal estimation, point cloud registration, point cloud classification and segmentation. Competitive performances have been reported with a comparison of the existing baselines.", "weaknesses": "1. For the experiments on point-cloud completion, as the experimental setup discussed, it samples 300 points for the evaluation. How about the performance of different number of sampled points?\n2. For the experiments on the test classification and part segmentation, it seems the proposed algorithm does not have obvious performance gain over the existing work like PaRINet. What is the main reason under this result?\n3. A minor suggestion, the baselines are usually introduced in the year before 2022. Is it possible to report more recent results for the comparison in the experimental section, like the work published in 2023?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "747ORqdqxN", "reviewer": "Reviewer_XaCw", "summary": "The paper introduces an equivariant feature representation to map per 3D point to a high-dimensional feature vector, which is supposed to be more capable to represent rich point information. Extensive experiments are conducted to verify the effectiveness in multiple tasks: point cloud completion, 3D shape compression, normal estimation, point cloud registration, and point cloud classification and segmentation. The results are very positive.", "questions": "Details in weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The primary strength of the proposed component lies in its capability to map 3D points to high-dimensional feature space with the property of SO(3)-equivariant, while the existing work Vector Neuron(VN) can only deal with three-dimensional features. The construction of the frequency-based transformation function D in equation (1) looks very reasonable and provable. \n\nSecondly, the effectiveness of the proposed module has been extensively evaluated on multiple tasks and datasets, which makes this work very solid and convincing.", "weaknesses": "There are some minor questions.\n\n1. The proposed component is always applied together with the existing VN module. Is it possible to work by its alone? If so, how to extend it? \n\n2. Since the construction of transformation function D is related to different frequencies. However, in the evaluation, there is a lack of concrete experiments to deeply analyse how such frequency-based function can help the network to learn more discriminative features. Although most of downstream tasks have excellent performance, it is unclear what type of features are learned while keeping the SO(3) equivariance. \n\n3. All experiments are conducted on very small-scale point clouds like objects. Is it possible to scale up the proposed method on larger scale 3D point clouds such as room-level ScanNet/S3DIS datasets or even urban-level SensatUrban dataset? If not, what are the potential reasons?\n\n4. In Section 2.2, the last line states that \"Our feature representation can xxxx, which is not equivariant, so that it is equivariant\". It's suggested to clarify the point?\n\n5. In page 4, the section \"Theorem 1.\", you may need to remove the word \"theorem\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s0MrcYqsdq", "reviewer": "Reviewer_HjVh", "summary": "**UPDATE**  \nAfter reading the rebuttal, I now better understand and appreciate the contribution. While I still believe this is a testimony to the much needed improvement in writing, I believe the work should be accepted. \n\n\nThe work proposes an SO(3) equivariant pointcloud network by extending vector neurons representation from a list of 3-dim vectors to a list of k-dim vectors. To accommodate this  increase in dimensionality a rotation operator construction is developed that can map a rotation in SO(3) to k dim.", "questions": "- which experiment is meant to demonstrate that the lack of details is due to the 3 dim feature vector? Take the non-equivariant network occNet. This network maintains better details and still has N dim. I therefore am not convinced that the issue is with the dim of the features. \n- I understanding that the motivation in the proposed lifting to k-dim is to keep it simpler than other tensor networks, but how is the proposed representation compare to it? is it less expressive? i would be glad to see a discussion", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The work demonstrates improvement in detail preservation compared to the baseline VN. Quantitatively, it manages to bridge the reconstruction gap reported in VN with the non-equivariant baseline, occNet. Qualitatively, it can be seen the the proposed method is able to preserve some details such as the vehicle side mirror.\n- The paper compares performance with the baseline VN on many additional applications including compression, normal estimation, classification, segmentation and registration. In all of them if demonstrates superior performance.", "weaknesses": "- The paper's presentation is one of the main reasons for my low rating. It is possible that I did not understand some important insights, but given my relatively broad background in the field, this suggests that the authors did not do an adequate job of presenting and conveying their ideas to the reader. Key issues include the lack of a clear motivation, insufficient highlighting of limitations in existing solutions, inadequate consideration of the suggested design choices, and—most importantly—an outdated comparison with relevant literature\n- The description of the limitation of VN is not clear. VN embeds features in a high dimensional space. But instead of using a n-dim vector, they use a nx3 matrix to keep the rotation operation in its same form as in the input space. It is thus my understanding that the representation capacity shouldn’t be smaller than any other global-embedding network which maps inputs to a high n-dim space. Therefore, i couldn’t quite understand the authors claim about VN having “3dim” features. First, I don’t think this is an accurate description of the dimensionality of VN, but more importantly i don’t understand how the fact that the vectors in the list live in 3 dimensions is causing issues with representing geometric details. I’ll try to give an intuitive example. Imagine a very dense pointcloud that captures fine details of the shape. Such a pointcloud may have 10^6 points and these points may live in 3D. My point is, the fact that the points live in R^3 isn’t itself a problem for capturing these details. Instead, what is known to cause issues with capturing fine details in neural fields is global representations. To fight this, many methods focus on partitioning the shape or scene into smaller regions and representing each of them with a local function, like switching from occnet to conv-occnet. In fact, there have been several follow up works to VN that try to do that. Other works tried to improve the encoder too. These works are not mentioned in the submission but should be discussed and compared with: [1,2,3,4].\n- It is perhaps my misunderstanding, but it seems the lifting to R^N replaces the principal axes in R^3 with a 3-dim subspace in R^N. Why then is this helpful?\n- The compression experiment is a bit unclear to me. There’s no report of the compression ratio — how much is the original pointcloud compressed wrt the embedding? It seems to measure reconstruction on the train set rather than compression.\n\n**Minor**:\n\n- The presentation should be more accurate. Sentences like “Equivariant neural networks (NN) change the output accordingly when the point cloud input is rotated without additional training.” are not accurate.  Equivariant NN are more general than that. Here the authors refer specifically to point cloud networks that are rotation equivariant.\n- why is z/z not shown for part segmentation?\n\nTypos:\n* theoremIf\n\n**References**: \n\n[1] VN-Transformer: Rotation-Equivariant Attention for Vector Neurons\n\n[2] VNT-Net: Rotational Invariant Vector Neuron Transformers\n\n[3] 3D Equivariant Graph Implicit Functions\n\n[4] On the universality of rotation equivariant point cloud networks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)